diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index b968bae00..d83dfb395 100644 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -1,639 +1,181 @@ # ClojureWasm -A Clojure runtime written in Zig 0.16.0. - -> Project memory loaded by Claude Code on every session. Keep it short. -> Detailed plans live in `.dev/ROADMAP.md`. Skills hold runnable procedures. - -## Project spirit (top priority, governs everything below) - -**The finished form's cleanliness wins.** Shipping fast and avoiding -rework are second-tier. Pre-built roadmaps and ADRs exist precisely -to minimise rework, but when they collide with "what the final shape -should look like", the final shape wins and the plan is amended in -place (ROADMAP §17). This applies to: - -- **Big surgery is welcome** when design tighting-up exposes something - the original plan missed. The autonomous loop must not hesitate at - depth-3 / depth-4 revisions (`.dev/principle.md`). -- **Skeleton-then-rewrite is endorsed** (per `permanent_noop_forbidden`) - but **excessive skeletons are a smell** (per the Smallest-diff bias - smell). Each skeleton must shrink the final-form rewrite, not enlarge - it. -- **Reservations (ADR numbers, NaN-box slots, debt rows promising - future ADRs) are memos, not contracts.** Obeying a reservation - because "it is reserved" is a smell (Reservation-as-bias smell, - `.dev/principle.md`). ADR numbers are time-ordered (`max + 1` at - issue time); slots / rows are reshuffled when the final form needs - it. -- **Progress pressure does not override the smell sensor.** "Let me - finish this task and come back" rarely comes back (Progress-pressure - smell). Stop, surgery, resume. - -This section is short on purpose. The mechanism lives in -[`.dev/principle.md`](../.dev/principle.md) (Bad Smell catalogue + four -depths of revision + three questions to picture the finished form). -**User-declared invariants** the loop must treat as fact even when -ROADMAP / ADR text reads differently live in -[`.dev/project_facts.md`](../.dev/project_facts.md) (F-001 zwasm v2 -unavoidable, F-002 finished-form wins, F-003 deferral on -structural plans, F-004 NaN-box 64-slot, F-005 numeric tower -JVM-surface, F-006 GC strategy, F-007 chapter cadence dormant -permanently, F-008 zwasm v2 spec review, F-009 -feature-implementation neutrality — read this file when in -doubt). - -### Priority order (the loop obeys this chain top-down) - -1. **`project_facts.md`** — user-declared invariants, treated as - project law. The loop **never amends an F-NNN on its own**; - amendments require user direction in chat + a Revision history - entry on the affected F-NNN. -2. **ROADMAP** — engineering plan + Phase order. -3. **ADRs + rules** — implementation decisions, amendable by the - loop autonomously (depth 2-4 per principle.md). -4. **principle.md heuristics** — smell sensors, depth selection. -5. **AI judgement** — fills in everything the above leave open. - -When two levels conflict, the higher level wins, and the lower -level is edited to align. Never the reverse. In particular, -treating an F-NNN as "informational" / "tie-breaker" / -"recommendation" is the **Smallest-diff bias smell** and is -forbidden. - -## Identity / Context (read first) - -**Project name (in all docs and the published artifact): `ClojureWasm`.** -Binary name: `cljw`. Package name: `cljw`. - -This codebase is a ground-up redesign of ClojureWasm on top of the v0.5.0 git -history. On **2026-06-10 the redesign was merged into `main`** (an `-s ours` -merge superseding the v0.5.0 main + its libc hotfix; the `v0.5.0` tag is -preserved), and `main` is the working branch: - -- **Working directory**: `~/Documents/MyProducts/ClojureWasm/`. -- **Branch**: `main` — the working branch. **All work happens on `main`** - and every commit is followed immediately by `git push origin main` in the - same Step 6 — commits do not accumulate locally. -- **Git remote**: `git@github.com:clojurewasm/ClojureWasm.git`. - -### Read-only reference points (do not edit, do not commit from) - -| Reference | What it is | -|-----------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| cw v0 — git tag `v0.5.0` | ClojureWasm v0 (89K LOC), the pre-redesign design reference. Its source is NOT in `main`'s tree (the `-s ours` merge kept the redesign tree), so reach it through git: `git worktree add ../cw-v0 v0.5.0`, or `git show v0.5.0:`. | -| `~/Documents/OSS/clojure/` | Upstream Clojure JVM source | -| `~/Documents/OSS/babashka/` | Babashka (SCI-based) | -| `~/Documents/OSS/zig/` | Zig stdlib source | - -## Language policy - -Public project. **English by default** for code, comments, identifiers, -commit messages, README, ROADMAP, ADRs, `.dev/`, `.claude/`, all -configuration. **Japanese** for chat replies, `private/notes/.md` -per-task notes, and (when re-activated) `docs/ja/learn_clojurewasm/NNNN_*.md` -learning narratives. The per-chapter cadence is currently **dormant** -per ADR-0025; existing chapters live read-only under -[`docs/ja/archive/`](../docs/ja/archive/). - -Don't mix Japanese into English docs. In `docs/ja/`, body is Japanese; -code blocks keep their original English identifiers. - -Chat replies default to Japanese via the project output style -[`.claude/output_styles/japanese.md`](../.claude/output_styles/japanese.md) -(activated by `outputStyle: "Japanese"` in `.claude/settings.json`). That -single setting is sufficient — the redundant SessionStart re-injection was -removed. To work in another language, override `outputStyle` per-machine in -`.claude/settings.local.json` (gitignored); code / commits / docs stay English. - -## Working agreement - -- TDD: red → green → refactor. -- **Step 0 (Survey) before each task**: a `general-purpose` subagent - surveys the textbook codebases (cw v0, Clojure JVM, Babashka, - Zig stdlib) and lands a 200–400 line note in `private/notes/`. - `general-purpose` (not the read-only `Explore`) is required so the - subagent can write the survey note itself. See - `.claude/rules/textbook_survey.md` for guardrails (cite ROADMAP - principles before adopting an idiom; always note one DIVERGENCE). -- After each task, write a 5-minute per-task note from hot context - (`private/notes/-.md`, gitignored). -- **Per-commit = smoke**, not the full gate (ADR-0107 two-tier gate; - the full e2e suite grew heavy, so "full every commit" is retired). - `bash test/run_all.sh --smoke ` (~tens of sec) green - **on Mac (host)** authorises a source commit — including shared-code, - because the smoke runs the full dual-backend diff oracle + every unit - test. The **full gate batches** (run ALONE) at the ≤5-commit ceiling / - Phase boundary / pre-tag. Prefer a **ReleaseSafe** binary (`zig build - -Dwasm -Doptimize=ReleaseSafe` = the gate config, so it cache-hits) for - manual behaviour probes; `-Dcpu=baseline` only for deploy artifacts. - SSOT: [`.claude/rules/gate_cadence.md`](../.claude/rules/gate_cadence.md). - **Linux x86_64 is no longer per-commit** (ADR-0049, orphan / fan - hazard): run `bash scripts/run_remote_ubuntu.sh` (→ `ubuntunote` SSH - host) at Phase boundaries, before the v0.1.0 tag, and on demand. - Don't bypass hooks. ubuntunote skips the Mac-only zlinter - `no_deprecated` gate (ADR-0003), so a 1-PASS-count host diff is - expected. Setup: [`.dev/ubuntunote_setup.md`](../.dev/ubuntunote_setup.md). -- Commit at the natural granularity of code changes. The per-concept - chapter cadence (`docs/ja/learn_clojurewasm/NNNN_*.md`) is **dormant** - per ADR-0025 until a resumption ADR fires; only the per-task notes - half of the `code_learning_doc` skill is active. -- Subagent fork is the default for: Step 0 surveys, large test logs - (>200 lines), cross-codebase searches (>5 files), phase-boundary - audit / simplify / security-review fan-out. Stay in main only for - small in-context edits. -- **Commit and push are one atomic Step 6** on `main`: - after the gate is green, `git commit` is followed immediately by - `git push origin main`. Local commits never accumulate - unpushed — leaving them stacked invites a "should I push?" - decision point that does not exist. -- ROADMAP corrections follow the four-step amendment in - [`ROADMAP §17`](../.dev/ROADMAP.md#17-amendment-policy): edit in place - as if it had always been so, open an ADR, sync `handover.md`, - reference the ADR in the commit. Quiet edits are forbidden. -- `private/` is gitignored agent scratch (per-task surveys + notes, - audit reports, the user's own brainstorming dumps). It is **not - authoritative** — audit and resume procedures do not read it as - load-bearing. If a `private/` proposal matters, promote it to - ROADMAP / ADR / `docs/ja/` / `handover.md` (all tracked in git); - otherwise let it stay scratch. - -## Autonomous Workflow - -**Default mode: continuous autonomous execution.** -After `/continue` or session resume, run the per-task TDD loop. -The only thing that ends a session is the user explicitly asking -for it (§ The only stop). Everything else — task boundaries, -phase boundaries, smell triggers, ADR-level decisions, gate -failures under investigation — is handled in-flight. Do **not** -pause between tasks. - -### Loop: Step 0 → 7 → next task's Step 0 - -After Step 7 (per-task note), **immediately start the next task's -Step 0**. Do not summarize, do not ask for confirmation. The next -task starts now. Auto-compaction handles context size transparently; -no agent action is needed. - -**Step 0 — Survey** (subagent: `general-purpose`, default mode -"medium") -Survey the related codebase per `.claude/rules/textbook_survey.md`. -`general-purpose` (not the read-only `Explore`) is required so the -subagent writes the survey note directly. Skip only when the task -is a clear continuation (refactor / rename / doc-only). Output -lands in `private/notes/--survey.md`. - -**Step 0.5 — Debt sweep** -Read `.dev/debt.yaml`. For each row whose `Last reviewed > 14 days -ago`, re-evaluate the Barrier predicate. Flip Status if the barrier -dissolved. **When opening a gap-area unit** (ROADMAP §9.0 / ADR-0142; -the phase-entry model is retired), additionally read every row whose -barrier names that gap area (`gap area I/II/III`) — these are the -draining rows that gap area owns. Any residual `Phase N` status string -is a not-yet-re-barriered row (R4 work, D-440); resolve it via the §9.0 -phase-number→gap-area redirect. - -**Quality-loop floor drain (post-M / F-010 operating mode).** The -post-M quality-elevation loop is a *repeatable* mode, not a Phase, so -its debt is anchored on a standing **`quality-loop floor: `** -Barrier instead of a one-time Phase entry (see -`.dev/tech_debt_consolidation.md`). When self-selecting the next -quality-loop unit (per § The only stop's next-task rule), **first read -every open `quality-loop floor:` row and drain it EASIEST-FIRST (by -tractability, not value — 2026-06-25 decision), before opening a fresh -sweep category** — this makes the loop debt-driven so floor items cannot -silently rot (the failure mode the 2026-05-31 audit found). A correctness -floor row still preempts; otherwise drain the whole list, niche included. Re-run the 5-lens consolidation audit (D-175) at each Phase -boundary so new orphans are caught within one Phase. - -**Step 0.6 — Re-laying against finished-form (main agent)** -The main agent (not a subagent) reads the survey output and -spends up to 30 minutes: - 1. Walk the plan's required-feature chain mentally; confirm the - prerequisites the survey assumed are actually in place. - 2. Re-lay the plan against finished-form (F-NNN in - `.dev/project_facts.md` + `.dev/structure_plan.md` + ROADMAP - §9): does the survey's recommended shape land closer to the - finished form, or is it a smallest-diff convenience that the - finished-form owner would unwind? - 3. If the survey misses a prerequisite or proposes a - smallest-diff bias path, **amend the survey in place** (the - survey is not immutable — it is the loop's input, not its - contract). - 4. If the re-laying surfaces a provisional behaviour the cycle - will need to introduce, **fork a `general-purpose` - Devil's-advocate subagent (mandatory, even at depth 1)** - to enumerate finished-form-clean alternatives within the - F-NNN envelope before committing to the provisional shape. - The marker comment + `feature_deps.yaml` entry + `.dev/debt.yaml` - row triad (per `.claude/rules/provisional_marker.md`) lands - in the same commit as the provisional behaviour. - -Step 0.6 is the discipline answer to "Step 0 surveys are only -~80% accurate; the remaining 20% surfaces in implementation". -Skip only when Step 0 itself was skipped (= the task is a -refactor / rename / doc-only change). - -**Step 1 — Plan** -One sentence in chat: the smallest failing test that captures the -next behaviour. Then **re-read `.dev/principle.md` and apply the -Bad Smell sensor** to the plan. If the plan touches any **structural -plan** (reservation table / directory or file structure / responsibility -separation / dependency graph), additionally apply the **Structural -imagination phase** from `principle.md` — imagine the full Phase 5-20 -horizon, record gaps as debt rows scheduled at the owning Phase -entry, and **defer the structural decision to that owner**; do not -resolve here. If something feels off, adjust before Step 2. - -**Step 1a — Unit reading list** (at each gap-area unit's start) -The phase-entry/placeholder model is retired (ROADMAP §9.0 / -ADR-0142); read per **gap-area unit** instead. Read in order: -`.dev/handover.md`, **`.dev/project_facts.md`** (user-declared -invariants the loop must treat as fact, even when ROADMAP / ADR text -admits other readings), the relevant `.dev/ROADMAP.md` **§9.0 gap-area -section** (I Concurrency / II Wasm-edge-native / III VM-perf) — its -BUILT status + named gaps + the draining `D-NNN` rows — each ADR cited -there **including any Phase N+ migration note + every Revision history -amendment** (where existing-code rewrite scope + inter-unit corrections -are narrated per §A25), the full text of each cited `D-NNN` row (in -`.dev/debt.yaml`), the `compat_tiers.yaml` entry for the function, and -the JVM Clojure source (`~/Documents/OSS/clojure/`) for the function. - -**Step 2 — Red** -Write the failing test (Edit / Write). Run; confirm red. - -**Step 3 — Green** -Minimal code to pass. Resist over-design; the next refactor pass -is cheap. - -**Step 4 — Refactor** -Structural improvements only, while green. Then **re-read -`.dev/principle.md` and apply the Bad Smell sensor to the -Green → Refactor diff**. If a smell surfaced, choose depth 1-4 -per principle.md and act before commit. - -**Step 5 — Test gate** (Mac smoke per-commit; full gate batched; ubuntunote at boundaries) - -Two-tier per ADR-0107 (full e2e is now heavy — never run it per commit -while batching). SSOT: [`.claude/rules/gate_cadence.md`](../.claude/rules/gate_cadence.md). - -- **Per commit = smoke**: `bash test/run_all.sh --smoke ` - (~tens of sec) — `zig build test` ×2 (the full F-012 diff oracle + all - units) + zlinter + `build_cljw` + `corpus_regression` + the changed e2e - step(s). Stamps `.dev/.smoke_pass`; authorises the commit (shared-code - included) up to the ≤5 ceiling. **Do not block on it** — background it - and keep moving. Manual behaviour checks use a **ReleaseSafe** binary - (`zig build -Dwasm -Doptimize=ReleaseSafe` = the gate config), not the Debug default. -- **Batched full gate** (run ALONE): `bash scripts/run_gate.sh` reaps any - orphan `run_all.sh` tree + PID-1-orphaned `cljw` probes, then `exec`s - `timeout 600 bash test/run_all.sh --serial-e2e` — **the same configuration - `ci_gate.sh` runs**, so a local green means what a push will mean - (`gate_parity` asserts it; before 2026-08-04 the local gate ran the parallel - e2e path and CI ran serial, and a serial-only failure was invisible locally). - Writes `.dev/.gate_pass`; the cadence hook is unaffected, and it stays - orphan-safe per - [`.claude/rules/orphan_prevention.md`](../.claude/rules/orphan_prevention.md) - § Gate launcher + memory `premature-gate-notification`. Run it at the - 5-commit ceiling / Phase boundary / pre-tag — backgroundable as a - look-ahead. On-demand reap without a gate: `bash scripts/run_gate.sh reap`. - If a full gate fails on ONLY a non-code check (e.g. `check_debt_id_refs`), - fix it + re-smoke — do NOT re-run the whole full gate. - -If the output exceeds ~200 lines, delegate to a Bash subagent -and ask for "pass/fail + first failure only". - -The Ubuntu x86_64 gate is **no longer per-commit** as of -ADR-0049 (orphan / fan hazard). Run it at Phase-boundary review -chains, before v0.1.0 tag, and on demand for feature branches: - -- `bash scripts/run_remote_ubuntu.sh` (drives `ubuntunote` SSH - host via `git fetch + reset --hard origin/main` + - `nix develop --command bash test/run_all.sh`). Setup at - `.dev/ubuntunote_setup.md`. - -**Orphan-prevention discipline**: every -`Bash(run_in_background: true)` driving a long-running child -process MUST wrap with `timeout 600 …`. REPL-pipe-specific -hazard, `timeout`-not-propagating caveat for SSH / VM -boundaries, and counter-examples live in -[`.claude/rules/orphan_prevention.md`](../.claude/rules/orphan_prevention.md) -(ADR-0049 § Context is the precedent; D-128 discharged). - -**Step 6 — Source commit + push (atomic, smell-audited)** - -Before staging: - -1. Re-read the Bad Smell catalogue in `.dev/principle.md`. -2. Self-audit the staged diff against the catalogue (about a - minute, no checklist — apply the sensor). -3. If a smell triggers, choose depth 1-4: - - depth 1: add a one-line note in the commit message. - - depth 2-4: land the ADR amendment / new ADR / `debt.yaml` row - / `private/notes/` entry **autonomously**. Devil's-advocate - subagent fork is mandatory at depth ≥ 2 — see § ADR-level - designs are handled inline below for the canonical brief. - Commit the doc change first, then commit + push the source - separately. ADR history (including the embedded - Devil's-advocate output) is the rationale record; no - external accept gate. - -Then: - -4. `git add` the source files. (The `bench/quick_baseline.txt` auto-baseline - was retired 2026-06-11 — the gate no longer runs `bench/quick.sh`; perf is - measured on demand via `bench/compare_langs.sh` / `bench/run_bench.sh`.) - Then `git commit` with the two-line shape: - ``` - (): - - Smell-audited: : - - ``` - `Smell-audited:` is **mandatory** on every commit that stages - source-bearing files (`src/**/*.zig`, `build.zig`, - `build.zig.zon`, `.dev/decisions/NNNN_*.md`). The Markdown-table - pre-commit hook is advisory (2026-06-11) — it flags unaligned staged - tables but does not auto-align, re-stage, or block. -5. `git push origin main` runs immediately on the - commit's success. **The `scripts/check_smell_audit.sh` PreToolUse - hook physically blocks pushes that include any source-bearing - commit missing a `Smell-audited:` line.** Re-audit, amend the - commit message, push again. Push is not optional and not - deferred. - -**Step 7 — Per-task note** (written from hot context) - -Copy `.claude/skills/code_learning_doc/TEMPLATE_TASK_NOTE.md` to -`private/notes/-.md`. Fill in: 一行サマリ / 詰まった -ポイント (1-3 個) / 教科書との対比 (Step 0 survey の要約) / -設計判断 / 章を書くときに必ず触れる点. Gitignored. Then immediately -begin the next task's Step 0. - -### When the active work unit completes (gap-area model, ADR-0142) - -The phase-queue model is **RETIRED** (ROADMAP §9.0 / ADR-0142): the -project is near-complete, so there is no "next PENDING phase -placeholder" to expand. Phases 9-14 are DONE; the remaining work lives -in three **gap areas** (I Concurrency-hardening / II Wasm-edge-native / -III VM-perf fusion→JIT) + a small genuinely-future bucket. When the -active unit's work runs low, the loop **self-selects** the next unit (it -never asks the user — see § The only stop's next-task rule): - -1. Check `.dev/handover.md` "First task on resume" — if it names a - concrete next unit, take it. -2. Otherwise self-select from the live `.dev/debt.yaml` `active:` rows - (the Step 0.5 sweep surfaces them), **EASIEST-FIRST — by tractability, - NOT by value** (user decision 2026-06-25; the standing snapshot + - easiest→hardest drain order is `private/notes/2026-06-25-debt-drain-order.md`, - refreshed from the live ledger). **Drain the WHOLE active list, niche / - deferred / "low-value" rows INCLUDED — the goal is to eliminate 残件, so - do NOT perpetually defer a row as "take-up-on-consumer".** A correctness / - clj-parity floor still PREEMPTS (do it first if one is open), but otherwise - ease — not value — sets the order. (F-002 finished-form still governs HOW - each task is done — quality per task — but no longer the ORDER between tasks.) - The genuinely-blocked Tier-4 rows (Tier-D needing a MAJOR/user nod, security - forward-looking design, the paused perf campaign, recall/load-sensitive races, - `.claude/`-edit-blocked) are the non-autonomous remainder — skip until their - gate changes; everything else drains. -3. Size the unit at one-to-three TDD cycles; run the per-task loop - (Step 0 → 7). **Update the ledger RELIABLY in the SAME cycle the work - lands**: when a row's work completes, prepend a `DISCHARGED ` note to - its status AND MOVE it from `active:` to `discharged:` (data-preserving text - move — insert the block just before the trailing `conventions:` key, NOT at - EOF; recipe in the 2026-06-25 audit commit); when you merely TOUCH a row whose - status is partly stale, re-narrow it against code-truth. The ledger is the - live SSOT — never let a done row rot in `active:`. - -The genuinely-future bucket (ClojureScript→JS / C-FFI / broad-JIT, -§9.0) is distal and **not** self-selected absent a user nod. The -capability-matrix successor to this gap-area model is forward debt -**D-443** (open after the D-440 arc completes). - -### The only stop - -**One condition: the user explicitly asks the loop to stop.** - -That is it. There are no other stop conditions. Specifically: - -- Task boundary, region boundary, cluster boundary, commit - boundary, Phase boundary — none of these end the session. Each - rolls straight into the next unit of work. -- Bad-Smell trigger (any depth, any frequency) — does not stop. - It authorises **interrupt-style investigation and surgery** on - the spot (see "Smell triggers are interrupts, not stops" below). -- ADR-level decision surfacing — does not stop. The AI drafts + - accepts the ADR inline ("ADR-level designs are handled inline" - below). -- Test failure / gate red — does not stop. Diagnose and fix; the - loop is responsible for getting back to green, not for handing - off. -- Long-running work, large diffs, big context — do not stop. - Auto-compaction is transparent; size is never the reason. -- **Next-task / direction ambiguity — does not stop, and is NEVER - resolved by asking the user.** When the clean/obvious work runs - low, the loop SELF-SELECTS the next unit (**EASIEST-FIRST debt drain, - not value-first** — 2026-06-25 decision; drain the whole `active:` list - including niche/deferred to eliminate 残件, updating the ledger reliably - each cycle — see § next-task rule + `private/notes/2026-06-25-debt-drain-order.md`) - and keeps raising precision: tests, robustness, error-path fidelity, - refactors, moderate features, audits. Using `AskUserQuestion` (or any - "which should I do next / A vs B vs wind-down?" prompt) to choose - the next task or the next direction is the **Direction-ask smell** - and is forbidden — the AI decides and proceeds; the user interjects - if they want a different direction. (`AskUserQuestion` remains fine - for a genuine in-task fact the AI cannot derive — an external - credential, a product preference with no sensible default — never - for "what should I work on next".) - -A user "stop" directive applies to **the session in which it was -issued**. It does not carry across to the next session — the next -`/continue` (or any automated re-invocation) resumes the loop -under this rule unchanged. Mechanical backstops re-inject this at -the two points momentum drifts toward an idle turn-end: after -`git commit` (`scripts/post_commit_remind.sh`) and after a test -gate launches (`scripts/gate_continue_remind.sh`) — see those -hooks, not added prose here. - -### Smell triggers are interrupts, not stops - -When the Bad Smell sensor (`.dev/principle.md`) fires, the loop -**interrupts the current activity and investigates without -compromise**. The investigation may be a one-line commit note -(depth 1), an ADR amendment (depth 2), a new ADR (depth 3), or a -Supersedes-chain rewrite (depth 4) — chosen per the sensor's -read, not per a frequency counter. Multiple smell triggers in one -cycle just mean multiple interrupts; they do not accumulate into -a stop. After each interrupt is resolved — at depth that matches -what the sensor surfaced — the loop returns to the per-task TDD -flow and continues. - -The discipline is "do not defer the surgery; do not narrow the -investigation to fit the task budget; do not let momentum override -the sensor." Quality work is *how* the loop runs, not a reason to -hand off. - -### ADR-level designs are handled inline - -When a design choice surfaces that would historically be called -"ADR-level" (tier shift, scope change, principle deviation, -load-bearing structural choice), the AI handles it inline. It -gathers the "should-be" materials itself — alternatives, -trade-offs, references to existing ADRs — drafts the ADR with -`Status: Proposed → Accepted` in the same cycle, fills Affected -files / Consequences, lands the ADR commit, and proceeds with the -source change. Rationale survives in the ADR's history; the loop -does not need an external accept gate. Step 6's depth 2-4 branch -is the runway for this. - -**Devil's-advocate subagent is mandatory at depth ≥ 2.** Before -stamping `Status: Proposed → Accepted`, fork a `general-purpose` -subagent with **fresh context** and brief it: - -> "Devil's advocate this ADR. The active F-NNN constraints from -> `.dev/project_facts.md` are: [paste]. Produce 3 alternative -> shapes **within those constraints** (one smallest-diff, one -> finished-form-clean, one wildcard); for each, name what it -> does better than the current draft and what it breaks. Do NOT -> propose alternatives that violate any F-NNN; if the only -> finished-form-clean option requires violating an F-NNN, say -> so explicitly — record that finding as the leading entry of -> Alternatives considered so the main loop sees it, but do not -> ask the loop to halt. **Cycle / diff / LOC size is not a -> project constraint** — only F-NNN is. If your finished-form- -> clean alternative would expand the cycle's diff substantially, -> recommend it anyway citing F-002; do NOT downgrade your -> recommendation to a smaller-diff alternative on cycle-budget -> grounds (that recommendation pattern is itself the -> Cycle-budget defer smell in `.dev/principle.md`)." - -The subagent's output is reflected verbatim into the ADR's -"Alternatives considered" section. This counters goal-drift / -instruction centrifugation by sourcing the alternatives from a -context without the main loop's accumulated momentum. The -subagent's recommendation is **not binding** within the F-NNN -envelope — the main loop still chooses — but the alternatives -must appear in the ADR. **Outside the F-NNN envelope**, a -"would-violate-F-NNN" finding does not stop the loop; the main -loop records the finding, picks the best F-NNN-compliant shape, -and continues. (If the user wants F-NNN itself amended, that is -a user action per `project_facts.md`'s amendment rule.) - -The phrases "this needs human judgement" / "cannot be -self-decided" / "user touchpoint required" / "halt and surface" -are forbidden framings in the autonomous loop. If the choice is -between candidate designs, the AI picks one (finished-form first, -smallest-diff as the secondary tiebreaker per F-002), records the -rejected alternatives in the ADR's "Alternatives considered" -section, and continues. - -**If the DA fork rates Alt N finished-form-clean and the loop's -instinct is a smaller-diff Alt M** citing cycle / LOC budget, that -instinct IS the **Cycle-budget defer smell** (`.dev/principle.md`): -re-pick Alt N. Cycle / diff / LOC is not a project constraint — only -F-NNN is. The sole exception is a real F-NNN block (pick the best -F-NNN-compliant alternative; F-NNN amendment is user-owned). - -## Skills (the runnable procedures) - -These hold the canonical procedures; CLAUDE.md only points to them. - -- **`code_learning_doc`** — Japanese learning material skill. - **DORMANT per ADR-0025**: only the per-task notes half is active - (Step 7 writes `private/notes/.md` from hot context). The - per-concept chapter half (`docs/ja/learn_clojurewasm/NNNN_*.md`) is - suspended; the pre-commit pairing gate is a no-op; existing chapters - live under [`docs/ja/archive/`](../docs/ja/archive/). A future resumption - ADR re-activates the chapter cadence; until then the templates - (`TEMPLATE_TASK_NOTE.md` / `TEMPLATE_PHASE_DOC.md`) are preserved - for that day. -- **`continue`** — resume procedure + per-task TDD loop (Step 0 - Survey → Step 7 per-task note → next task's Step 0) + multi-agent - Phase-boundary review chain (audits run, then loop continues into - §9.). Auto-triggers on "続けて" / "/continue" / "resume". - **Fully autonomous from invocation**. The only thing that ends a - session is the user explicitly asking for it (see § The only - stop). -- **`audit_scaffolding`** — periodic audit for staleness, bloat, lies, - and false positives across the tracked scaffolding (CLAUDE.md, - `.dev/`, `.claude/`, `docs/`, `scripts/`). Auto-invoked by - `continue` at every Phase boundary; can also be run on demand. - -The Phase-boundary review chain (auto-run by `continue` when a Phase -closes) fans out under multiple subagents to: audit_scaffolding, -built-in `simplify` on the phase diff, built-in `security-review` on -unpushed commits, and outstanding chapter writing — all in parallel. - -## Layout +A Clojure runtime written in Zig 0.16.0. Binary and package name: `cljw`. + +> Loaded on every turn, so it holds only what is true on every turn: +> invariants, identity, the priority chain, and ids. Procedure lives in +> hive memory and is fetched when the step actually runs. +> `mcp__hive__memory get :id ` + +## Priority chain (top-down; higher wins, lower is edited to align) + +1. `.dev/project_facts.md` F-NNN, user-declared, treated as project law. + The loop **never** amends an F-NNN on its own. +2. `.dev/ROADMAP.md` engineering plan. +3. Implementation decisions, all in memory now (ADRs tagged `adr-NNNN`, the rule + corpus tagged `cljw-rule`; both indexed below). Amendable by the loop + autonomously (depth 2-4) by editing the ENTRY, never by re-creating a file. +4. The smell sensors and depth selection, in memory: the catalogue is + `memory query :tags ["bad-smell"]` (24 entries, open set); the premises and the + three finished-form questions are `20260911004145-29ffd52a`; depth 1-4 plus the + mandatory devil's-advocate fork at depth >= 2 are `20260901165122-24182b0a` + and `20260901164941-6357af6c`; structural imagination and its F-003 / F-015 + scope boundary are `20260911004146-720ff81e`. +5. AI judgement fills in the rest. + +Treating an F-NNN as "informational" / "tie-breaker" / "recommendation" is the +Smallest-diff bias smell and is forbidden. + +**Project spirit: the finished form's cleanliness wins.** Shipping fast and +avoiding rework are second-tier. Big surgery is welcome; reservations (ADR +numbers, NaN-box slots, debt rows) are memos, not contracts; progress pressure +does not override the smell sensor. Mechanism: the smell corpus in memory, layer +4 above. + +## Identity (verified 2026-09-10) + +- **Repo**: `~/PP/ClojureWasm`. +- **origin** = `git@github.com:BuddhiLW/ClojureWasm.git`. **This fork is the + maintained one.** The `clojurewasm` remote is deliberately hobbled + (`no_push`, `--no-tags`) because `v1.10.1` names TWO commits. Never fetch + upstream tags. +- **Branches**: work lands on `staging`; `main` is reached by PR + (`staging` -> `main`), which is what cuts a release. Commit **and** push in + the same step; local commits never accumulate. +- **Shipped through v1.14.5.** Release mechanics: memory + `20260911001201-09a51b5b`. Commit and gate hygiene (the smell-audit trailer + needs `: `, CI is ONE configuration): `20260911001202-2ee19e2f`. + What is unfinished: `.dev/debt.yaml` + the kanban board. +- **Read-only references**: cw v0 through git only (`git show v0.5.0:`, + or `git worktree add ../cw-v0 v0.5.0`). zwasm clone at + `~/PP/zwasm` (origin `BuddhiLW/zwasm`, the fork `build.zig.zon` pins). + The `~/Documents/OSS/{clojure,babashka,zig}` and + `~/Documents/MyProducts/ClojureWasm` paths in older text are the + **upstream author's machine layout**, inherited by the fork and never valid + here. A survey that needs JVM Clojure, Babashka or the Zig stdlib must be + pointed at a tree that exists, or clone one first. + +## Language and writing style + +English everywhere: code, comments, commit messages, chat, README, ROADMAP, +ADRs, `.dev/`, `.claude/`, notes. `docs/ja/archive/` stays as it is +(chaploud's authored work, cadence dormant per ADR-0025). + +No em-dashes, no `...` ellipsis, no LLM padding. A comma, a colon, +parentheses, or a full stop. State the fact and stop. Long rationale goes in +an ADR or hive memory, never inline in a README. + +## The only stop + +**One condition: the user explicitly asks the loop to stop.** Task, phase, +commit and cluster boundaries do not stop it. A smell trigger does not stop +it (it is an interrupt). A red gate does not stop it. Choosing the next unit +is never resolved by asking the user; that is the Direction-ask smell. + +Full rule + next-unit self-selection + debt-drain order: `20260910001609-5b3ea4d7` + +## Procedure (fetch when the step runs, not before) + +| what | id | +|---|---| +| Per-task TDD loop, Step 0 to 7 | `20260910001609-609c3c20` | +| ADR-level design inline + the DA-fork brief | `20260910001609-6e4ca074` | +| The only stop + next-unit selection | `20260910001609-5b3ea4d7` | +| Data-source SSOTs, and what each answers | `20260910001610-1d4ad7e8` | +| Always-on context budget, measured | `20260909235631-5c19c78d` | + +## Knowledge lives in hive memory, not in files + +**`.claude/rules/*.md` is gone (2026-09-11).** All 32 were mirrored into memory ++ KG on 2026-09-09/10, no script ever read one, and the glob auto-loader was +re-sending ~18k tokens on every `.zig` edit (~47k for the corpus). The table +below is the whole tier: fetch a body when its subject is what you are doing. +**`G` = the entry carries a `guard-rule` block**, so it is enforced by +hive-spi.guard whether or not you read it; a refusal quoting an id IS that rule. + +| rule | id | G | +|---|---|---| +| accepted_divergences | `20260909234822-74759eeb` | | +| binary_size | `20260909235202-2908e693` | G | +| bootstrap_essence | `20260909235202-58f77413` | | +| clj_attribution | `20260909234633-115f1e0c` | | +| clj_diff_sweep | `20260909234815-3490bbcd` | G | +| cljw_invocation | `20260909234845-35806ca2` | G | +| clojure_spec_citation | `20260909234644-6d1d690d` | | +| debt_dedup | `20260909234454-48f2e989` | G | +| dual_backend_parity | `20260909234759-18586426` | | +| error_catalog_only | `20260909234626-1bece9de` | G | +| exploration_vs_done | `20260909234838-2229ddfb` | | +| extended_challenge | `20260909234455-330545a2` | | +| feature_name_consistency | `20260909234605-3f9e2aa8` | | +| framework_completion | `20260909234612-6f9c927b` | | +| gate_cadence | `20260909234752-16ad5476` | G | +| markdown_format | `20260909234454-6705f6f7` | | +| module_docstring | `20260909235200-7c602907` | G | +| no_copy_from_v1 | `20260909235201-14470f53` | | +| no_jvm_specific_assumption | `20260909235201-44e1e6e9` | G | +| no_op_stub_forbidden | `20260909235201-72836234` | | +| orphan_prevention | `20260909234832-598b7c16` | G | +| perf_marker | `20260909234453-7ca5e578` | G | +| perf_measure_release | `20260909234455-390c5332` | G | +| plan_revision_thinking | `20260909234453-6cf9b052` | | +| provisional_marker | `20260909234453-094762cc` | G | +| test_taxonomy | `20260909234807-4b4c0db8` | | +| textbook_survey | `20260909234618-0095b147` | | +| tier_classification | `20260909234639-2efc4769` | | +| yaml_ssot_yq | `20260909234558-3c83a76a` | G | +| zig_tips | `20260909235200-1f86720b` | G | +| zone_deps | `20260909235201-1eab3b17` | G | + +The 15 rows with no `G` are advice, not enforcement: fetch them on the subject, +and if one turns out to be mechanizable, write its `guard-rule` block into the +entry rather than restoring a file. The `check_*.sh` gates are unaffected; they +read the YAML SSOTs and the source, never a rule file, and the paths they print +in error messages are now memory ids. + +ADRs are cited by commit messages. Search, do not read a directory: ``` -src/ Zig source -build.zig Build script (Zig 0.16 idiom) -flake.nix Nix dev shell pinned to Zig 0.16.0 -.dev/ ROADMAP + handover + ADRs -docs/ja/ Japanese learning narratives -.claude/ settings, skills, rules -scripts/ gate, zone check -test/ unified runner + future suites +mcp__hive__memory search :query "..." +mcp__hive__memory query :tags ["adr" "adr-0107"] # one ADR +mcp__hive__memory query :tags ["cljw-rule"] # the rule corpus +mcp__hive__memory kg traverse :start_node :max_depth 2 ``` -## Build & test +Design canon distilled from reading the whole corpus at once: + +| finding | id | +|---|---| +| A generated artifact with no gate is abandoned (decided 4x) | `20260909235631-6f763b58` | +| Registry for completeness; defer a seam for pluggability | `20260909235630-6096047d` | +| The dual-backend oracle cannot see a bug identical on both | `20260909235630-5ace59c5` | +| A proxy signal accepted for the fact it names | `20260910001213-63e219a4` | +| Overriding the DA fork: name the axis, not the size | `20260909235632-7a57b709` | +| Guard enforceability measured at 12 of 32 rules | `20260909235631-5ae3f445` | + +## Build and test ```sh -bash test/run_all.sh --smoke # per-commit smoke (fast; ADR-0107) -bash test/run_all.sh # full gate (batched: ceiling/boundary/pre-tag) -zig build -Dwasm -Doptimize=ReleaseSafe # probe binary (= gate config; -Dcpu=baseline only for deploy) -zig build run # run executable (`cljw`) -zig fmt src/ # format +bash test/run_all.sh --smoke # per-commit smoke (ADR-0107) +bash scripts/run_gate.sh # full gate, run ALONE +zig build -Dwasm -Doptimize=ReleaseSafe # probe binary (= the gate config) +zig fmt src/ ``` -## Data sources - -- [`compat_tiers.yaml`](../data/compat_tiers.yaml) — authoritative Tier A / B / - C / D classification per var, special form, and host class. Read by - test runner, REPL error message, and future `cljw --list-vars`. See - ADR-0013 for the Tier D rationale. -- [`placement.yaml`](../data/placement.yaml) — **GENERATED** index of where every - interned Var's implementation lives (`zig` / `clj` / `value` / `unbound` + - flags), covering every var in every bundled namespace. Produced by - `scripts/gen_placement.sh` from `(cljw.internal/__dump-placement)`; the - `placement_drift` gate regenerates and diffs it on every full gate, so it - cannot disagree with the code (ADR-0178). **Do not hand-edit.** Role split: - compat_tiers.yaml = Java/cljw surface (Class-level), placement.yaml = - Clojure-ns vars (var-level). Whether a var's *current* placement is its - *intended* one is a judgement, not a fact — that lives in `.dev/debt.yaml` - with a barrier (today one row, D-094). ADR-0033 D2-D5 is the placement rule. -- [`host_interfaces.yaml`](../data/host_interfaces.yaml) — closed-set SSOT for the - deftype/reify/extend-type host-supertype markers (`Object`, `clojure.lang.*`). - Single in-code read point: `src/runtime/host_interface.zig`. Gated by - `scripts/check_host_interface.sh` (G4: recognised-set ⊆ rows, no over-claim). - Closes the 個別最適化 entry structurally (ADR-0102, F-013). Distinct from - compat_tiers.yaml: that = host *class* surface; this = deftype/reify *marker*. -- [`.dev/debt.yaml`](../.dev/debt.yaml) — row-level debt ledger. `continue` - skill Step 0.5 sweeps this on every resume. See ROADMAP §A13. -- [`.dev/reference_clones.md`](../.dev/reference_clones.md) — explicit - usage purpose for `additionalDirectories` paths. -- [`.dev/lessons/INDEX.md`](../.dev/lessons/INDEX.md) — observational - learnings, distinct from load-bearing ADRs. -- [`.dev/gc_rooting.md`](../.dev/gc_rooting.md) — SSOT for the GC rooting - surface: every site that publishes / holds / decodes a GC root (EvalFrame - producers, root slots, reentrant accumulators, pins, in-txn, per-tag traces, - the `isGcManaged` membrane) + a moving-GC migration checklist. `GC-ROOT:` - markers in source anchor each published-root site here. -- [`.dev/zwasm_capabilities.md`](../.dev/zwasm_capabilities.md) — cljw's record of - the **embedded zwasm v2** runtime's embedding-API capabilities + what cljw - adopted (F-001). **FROZEN as of 2026-08-12**: zwasm moved to its own org - () under separate maintainership, and v2.5.0 is - cljw's final pin — so the per-unit refresh duty and the CODEV mailbox protocol - that file used to carry are **retired**. Read it as background on what the - final pin embeds, not as a watch list. +Never a bare `zig build test` without `-Dwasm`. Never a Debug binary for a +behaviour probe. Never a build during the full gate. Cadence SSOT: memory +`20260909234752-16ad5476` (gate_cadence). ## References -- [`.dev/ROADMAP.md`](../.dev/ROADMAP.md) — authoritative mission, principles, - phase plan. **Single source of truth**; if anything in this file - conflicts with the roadmap, the roadmap wins. -- [`.dev/handover.md`](../.dev/handover.md) — short, mutable, current state. - Framing per [`.claude/rules/handover_framing.md`](../.claude/rules/handover_framing.md) - (≤ 100 lines; driving doc, not session log). -- [`.dev/decisions/`](../.dev/decisions/) — ADRs (load-bearing decisions); - numbers are time-ordered, newest wins on conflict. +- `.dev/ROADMAP.md` authoritative mission and plan. **If this file conflicts + with the roadmap, the roadmap wins.** +- Current state: `git log`, the CHANGELOG, and the kanban board. There is no + handover file (retired 2026-09-11, `20260910235746-74389f7f`); the resume path + is `project workflow catchup`, which drains the axioms and the live cards. +- **ADRs are memory entries, not files** (2026-09-11, `20260910235746-74389f7f`). + All 199 were mirrored before deletion and the audit confirmed 199 of 199. Reach + one by number: `memory query :tags ["adr" "adr-0107"]`. Numbers stay + time-ordered and newest still wins on conflict; a new decision is an entry of + type `decision` tagged `adr-NNNN`, cited by number in the commit message. +- `.dev/project_facts.md` the F-NNN invariants this chain starts from. diff --git a/.claude/rules/accepted_divergences.md b/.claude/rules/accepted_divergences.md deleted file mode 100644 index 4b2106cd2..000000000 --- a/.claude/rules/accepted_divergences.md +++ /dev/null @@ -1,102 +0,0 @@ ---- -paths: - - "src/**/*.zig" - - "src/lang/clj/**" - - "scripts/clj_diff_sweep.sh" - - "scripts/check_corpus_regression.sh" - - "scripts/check_accepted_divergences.sh" - - ".dev/accepted_divergences.yaml" - - "test/diff/clj_corpus/**" ---- - -# Accepted divergences (the "NOT a bug" discipline) - -Auto-loaded when editing runtime source, the clj-diff harness, or the -divergence ledger. Codifies how a behavioural difference between cljw and -JVM Clojure is **classified** — so the small mismatches a user can hit are -either fixed or recorded-as-intentional, never silently left to erode -trust. - -## The two-way classification (every clj DIFF is one or the other) - -When a `clj_diff_sweep` / `check_corpus_regression` run shows cljw differing -from `clj`, the difference is exactly one of: - -1. **A bug** → fix it (F-011 behavioural equivalence). Land the fix + a - corpus/e2e case asserting the clj-matching behaviour. This is the - default disposition. -2. **An accepted divergence** → record it in - [`.dev/accepted_divergences.yaml`](../../.dev/accepted_divergences.yaml) - as a new `AD-NNN` with a **`derives_from`** invariant (F-NNN / ADR) and a - **`pin`** test that locks cljw's divergent behaviour. Then it is no longer - a DIFF to chase. - -**A DIFF is never left unclassified.** "I'll look at it later" on a clj -divergence is the Defer-to-amnesia smell (`.dev/principle.md`): either it is -a bug (open a debt row + fix) or it is accepted (add the AD-NNN). Leaving it -as floating sweep noise is what erodes trust — the same divergence resurfaces -every sweep and no one knows if it is intended. - -## When a DIFF is an accepted divergence - -The bar for "accepted" is a **project invariant**, not convenience: - -- **F-NNN-mandated** — e.g. ADR-0059 no-JVM (`(class 5)`→`Long`, error Kind - vs JVM exception class), F-005 numeric tower (single f64, no f32). -- **Semantically irrelevant** — set / non-sorted-map print ORDER (the values - are `=`; order is not part of unordered-collection semantics). -- **Non-reproducible on the clj side** — opaque-ref `#object[… 0xADDR …]` - embeds an identity hash cljw cannot and should not mirror. -- **Rare-edge grammar** explicitly out of scope (e.g. hex-float in - `Double/parseDouble`) — flagged for promotion-to-bug if real code hits it. - -If you cannot name the invariant, it is **not** accepted — it is a bug (or a -debt row pending a decision). `scripts/check_accepted_divergences.sh` -enforces the `derives_from` field for exactly this reason. - -## Auto-defense (what the gate guarantees) - -`scripts/check_accepted_divergences.sh --gate` (in `test/run_all.sh`) -guarantees the ledger stays a trust contract: - -1. The SSOT is well-formed and non-empty. -2. Every `AD-NNN` cites a non-empty `derives_from` — no lazy accept. -3. Every `pin` test path exists — the divergent cljw behaviour is **locked**. - If a future change accidentally makes cljw match clj (or diverge - differently), the pin test fails → a conscious decision is forced, not a - silent flip. -4. `COVERAGE.md`'s "Acceptable divergences" section points at the SSOT, so - the human-facing doc cannot drift from the machine-readable ledger. - -## Promotion / demotion is explicit (both directions) - -- **Accept a divergence** (bug → AD): add the `AD-NNN` with `derives_from` + - `pin`. Remove any debt row that tracked it as a gap. -- **Un-accept** (AD → bug): the project decides the divergence should after - all match clj. Remove the `AD-NNN`, open a debt row, add a corpus/e2e case - asserting the clj-matching behaviour. The pin flips from "lock the - divergence" to "assert parity". - -Neither direction is silent. The SSOT diff + the commit message record the -decision. - -## Relationship to the user-facing capability ledger - -`docs/works/` (F-010) is where the user-readable "this is how cljw behaves" -narrative lives. Accepted divergences worth surfacing to users (notably the -high-frequency AD-001 set print order and AD-003 simple class name) should -get a line there too, so the divergence reads as "designed" rather than -"broken" when someone first hits it. - -## Cross-references - -- [`.dev/accepted_divergences.yaml`](../../.dev/accepted_divergences.yaml) — - the SSOT. -- [`scripts/check_accepted_divergences.sh`](../../scripts/check_accepted_divergences.sh) - — the gate. -- [`clj_diff_sweep.md`](clj_diff_sweep.md) § "When a DIFF is NOT a bug" — the - in-harness reminder; this rule is the full discipline it points to. -- [`.dev/principle.md`](../../.dev/principle.md) — Defer-to-amnesia smell - (an unclassified DIFF) + Silent-default-shift smell (an un-recorded flip). -- F-011 (behavioural equivalence) / ADR-0059 (no-JVM) — the invariants most - accepted divergences derive from. diff --git a/.claude/rules/binary_size.md b/.claude/rules/binary_size.md deleted file mode 100644 index aa31eb98f..000000000 --- a/.claude/rules/binary_size.md +++ /dev/null @@ -1,73 +0,0 @@ ---- -paths: - - "src/**/*.zig" - - "build.zig" - - "build.zig.zon" ---- - -# Binary-size awareness (ADR-0172 is the SSOT) - -Auto-loaded when editing Zig source / the build. cljw's headline claim is -"one small static binary"; size is budgeted per component and gate-checked. -This rule carries the operational discipline + the measured lessons from the -2026-07-16 campaign (9.47 MB → 7.38 MB, −22%) so they are not re-derived. - -## The mechanical guards (already wired — do not duplicate) - -- **`size_claims` step** (full gate, `test/run_all.sh`): README's headline - " MB" must stay within 10% of the freshly built binary — a prose size - claim can no longer rot (the 3.8-vs-9.5 MB incident class is closed). -- **Budget ceiling check** (same step, `scripts/binary_size_report.sh - --check`): the built binary must stay under the ADR-0172 **derived - ceiling** (`BUDGET_CEILING_BYTES` in the script, kept in sync with the - ADR's budget table). A breach fails the gate: attribute with the report - tool, then either land a lever or consciously amend the budget **in - ADR-0172** (Revision history entry) — never silently. -- **Measurement tool**: `bash scripts/binary_size_report.sh [BIN]` — total + - segment breakdown; on a `-Dprofile=true` build it prints top symbols for - attribution. - -## When your change plausibly adds size, check it - -A change is size-relevant when it adds: a new `@embedFile`/generated blob, a -new comptime-instantiated generic family, a new std subsystem (crypto, http, -regex-class tables), a large `switch`/table, or a new dependency. For those: -measure before/after (`binary_size_report.sh`), and note the delta in the -commit message when it exceeds ~50 KB. Feature growth is legitimate -(F-013/F-014) — the budget exists to make it *conscious*, not to forbid it. - -## Measured lessons (do not re-litigate; citations in ADR-0172/0173) - -1. **Instantiation count predicts recoverable size — symbol size does NOT.** - A huge single symbol that aggregates once-called inlined handlers is - size-NEUTRAL to out-line (zwasm's 707 KB emitter: table-driven rewrite - changed nothing). The real wins are ×N-duplicated bodies (zwasm's - host-callback thunks: ~300 B × 3,840 → −1.08 MB collapsed to shared - bridges + 23 B forwarders). Before proposing a size refactor, ask "how - many call sites / comptime instantiations share this code?". -2. **Watch comptime cross-products.** `fn(T) × arg-kinds × arity × slot` - families explode silently. Prefer a type-erased shared body + thin - per-instantiation forwarders (`noinline` on the shared body) when the - family exceeds a handful of instances. Known open case: std.sort - comparator monomorphization ≈ 224 KB / 93 instances (ADR-0172 L6). -3. **Embedded assets compress ~4-5×** (bytecode blob, `.clj` text): flate - `.raw` + exact `uncompressed_len` + decompress-on-demand into - `rt.load_arena` is the established pattern (serialize.zig - `flateCompress`/`flateDecompress`; two std pitfalls are documented on - those fns — non-empty output buffer for Compress, REAL 32 KB window for - Decompress). -4. **Safety is never traded for size.** ReleaseSafe stays the shipped config - (ADR-0132); ReleaseSmall/hybrid rejections are recorded as ADR-0172 - L3/L4 — don't re-propose them without new facts. -5. **Startup claims need median-of-20 on a quiet machine.** Single-shot - `CLJW_PROFILE_STARTUP` probes under load produced a phantom −32% once - (corrected in ADR-0173). Size numbers are stable; time numbers are not. - -## Cross-references - -- `.dev/decisions/0172_binary_size_budget_and_ledger.md` — budget table + - lever ledger + governance (the SSOT). -- `.dev/decisions/0173_envelope_v7_zero_copy_pool_flate.md` — the envelope - v7 arc (pool / zero-copy / flate) with its measurement corrections. -- `.dev/debt.yaml` D-515 — the standing drain anchor. -- `docs/works/binary_size.md` — the public measured comparison. diff --git a/.claude/rules/bootstrap_essence.md b/.claude/rules/bootstrap_essence.md deleted file mode 100644 index a706db0ac..000000000 --- a/.claude/rules/bootstrap_essence.md +++ /dev/null @@ -1,60 +0,0 @@ ---- -paths: - - "src/**/*.zig" - - "src/**/*.clj" - - "src/lang/bootstrap.zig" - - "src/eval/analyzer/**/*.zig" - - "src/eval/backend/**/*.zig" ---- - -# Bootstrap essence - -> Folded into [`provisional_marker.md`](provisional_marker.md) § -> "Why this rule exists" at Wave 16 W16-6 (2026-05-26). This file -> remains as a path-frontmatter pointer so editors of bootstrap- -> related Zig + `.clj` files keep auto-loading the chicken-and-egg -> motivation that the original 2026-05-26 W16 introduction -> recorded. - -## The pattern, in one sentence - -Language implementations grow in **layers**, and an upper layer -cannot be expressed in its finished form until the layer -underneath is reachable. While the underneath is incomplete, the -upper layer **rides provisional behaviour** in whatever already -runs. - -## Historical anchors (themes only) - -- **Lisp 1962** (Hart & Levin, MIT): first self-hosting compiler - — write the new compiler in Lisp, test it inside an existing - Lisp Interpreter, iterate until self-hosting. -- **ClojureScript bootstrap**: ships a pre-built analyzer cache - for `cljs.core` so the self-host environment starts with "core - was already loaded". -- **Crafting Interpreters** (Nystrom): spent weeks hand-drawing - the dependency graph between chapters so each chapter's feature - could be authored using only what earlier chapters taught. -- **Onramp self-bootstrapping C compiler**: implementation split - into stages, each stage compilable by the stages before it. - -## cw v1 mapping - -- Layer 0 — host code (Zig). -- Layer 1 — `rt/` primitives (Zig leaves). -- Layer 2 — Pattern A `.clj` defns composing Layer 1. -- Layer 3 — `(ns ...)` macro / `require` / full namespace - semantics (lands at ADR-0035 / Phase 6.16.b-4). - -Each upper layer ships provisional behaviour against the next -layer's not-yet-existence. The lifecycle is mechanised via -[`provisional_marker.md`](provisional_marker.md) + -`data/feature_deps.yaml` + `.dev/debt.yaml`. - -## What this rule deliberately does *not* prescribe - -Specific strategies for each chicken-and-egg — recipe-by-recipe -playbook of how cw v1 should bootstrap each feature. The -historical anchors are pointers to read when relevant, not -templates to copy. The cw v1 judgement space is the project's -most valuable resource; over-prescription would erode it. diff --git a/.claude/rules/clj_attribution.md b/.claude/rules/clj_attribution.md deleted file mode 100644 index 8d3a9c822..000000000 --- a/.claude/rules/clj_attribution.md +++ /dev/null @@ -1,115 +0,0 @@ -# Clojure-namespace license attribution (EPL-2.0) - -Auto-loaded when editing any `src/lang/clj/clojure/**/*.clj`. Codifies the -license-header convention so the judgement is made once (here) and never -re-litigated per file. The settled work order is -`private/20260609_license_research/PROMPT.md` § B (D-366); this rule is its -durable, machine-guaranteed form. - -## The convention (judgement-less) - -**Every `.clj` under `src/lang/clj/clojure/` MUST carry the line -`;; SPDX-License-Identifier: EPL-2.0` as part of its file header.** This -holds for any new namespace brought in from Clojure too. The classification -criterion is **namespace origin only** — a file lives under `clojure/` -because it implements a Clojure / Clojure-contrib API, so it is EPL-2.0. - -`src/lang/clj/cljw/**` is **out of scope**: those namespaces are -ClojureWasm-original (no upstream lineage), so they carry no upstream -attribution. `.zig` is clean-room; tree-level attribution is -`legal/NOTICE`. - -**Docstrings are not notices** — never edit a docstring for attribution -reasons, even when its text is copied from upstream (that is what variant ① -records explicitly). - -## The two header variants - -Which variant a file gets depends on **whether it reproduces upstream source -text**, not on the namespace: - -### Variant ① — upstream source text reproduced (banner retained) - -Only files that actually reproduce upstream text. As of D-366 exactly two: -`clojure/template.clj` (verbatim copy) and `clojure/core/protocols.clj` -(docstrings reproduced). These keep the upstream EPL banner so attribution -to Rich Hickey / the original author is preserved per EPL-1.0 §7: - -```clojure -;; SPDX-License-Identifier: EPL-2.0 -;; -;; Copyright (c) Rich Hickey. All rights reserved. -;; The use and distribution terms for this software are covered by the -;; Eclipse Public License 1.0 (http://opensource.org/licenses/eclipse-1.0.php). -;; By using this software in any fashion, you are agreeing to be bound by the -;; terms of this license. You must not remove this notice, or any other, from -;; this software. -;; -;; — by . Reproduced in ClojureWasm; redistributed -;; under EPL-2.0 per EPL-1.0 §7. ClojureWasm changes (c) the ClojureWasm authors. -``` - -A new file gets variant ① **only if you literally paste upstream source -text** into it. The default for a fresh namespace is variant ②. - -### Variant ② — independent reimplementation (the common case) - -Every other `clojure/` namespace is an independent reimplementation (no -upstream source text reproduced). It carries the CW-copyright header citing -the upstream lineage as courtesy: - -```clojure -;; SPDX-License-Identifier: EPL-2.0 -;; Copyright (c) the ClojureWasm authors. Licensed under EPL-2.0. -;; Independently reimplements the API (originally ; Clojure, EPL-1.0) -;; for ClojureWasm; no upstream source text is reproduced. -``` - -`` = the namespace (e.g. `clojure.set`); `` = the original -author(s) (e.g. `Rich Hickey`). The header is a **pure prepend** above the -file's existing comments — do not touch the body or docstrings. - -## Discovery criterion (framework_completion.md mandate) - -Enumerate header-less files (the sweep recipe — also used by the hook): - -```sh -find src/lang/clj/clojure -name '*.clj' | xargs grep -L 'SPDX-License-Identifier: EPL-2.0' -``` - -> **Do NOT use `git ls-files 'src/lang/clj/clojure/**/*.clj'`** as the -> discovery recipe. Git pathspec `**` does not match a single path segment, -> so that glob silently skips the top-level `clojure/*.clj` files -> (`core.clj`, `set.clj`, `edn.clj`, `math.clj`, `string.clj`, …). `find` -> (or `git ls-files 'src/lang/clj/clojure/' | grep '\.clj$'`) is correct. - -A clean run (empty output) means every file is attributed. - -## Sweep + retrofit record (D-366) - -The introducing cycle (2026-06-10) ran the recipe and retrofitted all 16 -existing files in the same cycle (B-1 = the retrofit): 2 variant ① + -14 variant ② (`core`/`set`/`walk`/`zip`/`data`/`string`/`test`/`java.io`/ -`math`/`pprint`/`edn`/`data.json`/`data.csv`/`tools.cli`). Sweep note: -`private/notes/D366-license-attribution.md`. No exemptions -(`.dev/watch_findings.md` carries none for this rule). - -## Enforcement - -`scripts/check_clj_attribution.sh` is a PreToolUse hook on `git commit` -(wired in `.claude/settings.json` alongside `check_smell_audit.sh` / -`check_gate_cadence.sh`, reusing `hook_lib.sh`). It scans the working tree -and **fail-closed blocks** any commit while a `clojure/**/*.clj` lacks the -SPDX line — so a header-less file can never be committed. Both variants -carry the SPDX line, so the single check guarantees both. - -## Cross-references - -- `private/20260609_license_research/PROMPT.md` § B — the settled work order - (judgement; this rule is its durable form) + `REPORT.md` (rationale). -- `legal/NOTICE` — tree-level attribution (the import relationship + - the two variant-① files). -- `LICENSE` (EPL-2.0) — the project license. -- `.claude/rules/framework_completion.md` — the discovery + sweep + retrofit - discipline this rule + its hook satisfy in one cycle. -- `scripts/check_clj_attribution.sh` — the enforcement hook. diff --git a/.claude/rules/clj_diff_sweep.md b/.claude/rules/clj_diff_sweep.md deleted file mode 100644 index 3c9bad6e1..000000000 --- a/.claude/rules/clj_diff_sweep.md +++ /dev/null @@ -1,71 +0,0 @@ -# clj differential sweep (F-011 quality mode) - -Auto-loaded when editing `src/lang/clj/**` or the sweep harness. Codifies -how the post-M quality loop checks `cljw` against the `clj` oracle, and the -two disciplines that keep that work honest. - -## The harness — don't hand-roll the loop - -`scripts/clj_diff_sweep.sh` is the SSOT for "run these exprs through both -runtimes and show me the diffs". Do NOT write ad-hoc -`for e in …; cljw vs clj` shell loops — they are unreviewable, throw away -the result, and repeatedly re-derive the same methodology. - -```sh -bash scripts/clj_diff_sweep.sh exprs.txt # one bare value-expr per line -printf '%s\n' '(map inc [1])' | bash scripts/clj_diff_sweep.sh - -bash scripts/clj_diff_sweep.sh exprs.txt --corpus seqfns # append OKs to a corpus -``` - -Methodology baked into the script (header has the detail): clj runs ONCE -over a batch (one `(prn EXPR)` per line); cljw runs one expr at a time; -both are `timeout`-wrapped; bound every seq producer with `(take N …)`. - -## Discipline 1 — corpus-backed discharge (anti D-177) - -A debt row whose discharge text **lists functions/behaviours as "landed"** -MUST add those exact expressions to a corpus under -`test/diff/clj_corpus/.txt` (via `--corpus`), so the claim is -mechanically re-checkable. The 2026-06-02 audit found **D-177 discharged -with `take-while`/`take-nth`/`partition-by` listed as done when they were -not** — a false-positive-discharge lie. The fix for that lie class is: -*never list coverage you did not probe*, and leave the probe behind as a -corpus line. `scripts/check_corpus_regression.sh` re-runs every corpus and -fails on any DIFF, so a regression (or an over-claim) surfaces. - -## Discipline 2 — big-bang, don't drip-feed - -Coverage sweeps are done **exhaustively-then-closed**, not one function per -cycle forever (the Micro-coverage-grind smell). The drip-feed mode quietly -displaces the project's real differentiator (Phase 15-20 Wasm/edge-native) -and never converges. When opening a sweep area, enumerate the *whole* -surface (e.g. all of `clojure.string`, all transducer arities, all the -numeric tower), drive it to zero DIFFs in one focused push, write the -corpus, and close the area. A half-swept area is worse than an unswept one -because the debt ledger reads "covered". - -## When a DIFF is NOT a bug - -> **SSOT for accepted divergences = [`.dev/accepted_divergences.yaml`](../../.dev/accepted_divergences.yaml)** -> (AD-001…AD-007), with the full classification discipline in -> [`accepted_divergences.md`](accepted_divergences.md). A DIFF is classified -> bug→fix OR accepted→AD-NNN (never left floating). The quick reminders: - -- **Set / non-sorted-map print order** differs from clj's hash order — not - a bug (AD-001; don't "fix" `#{2 3}` vs `#{3 2}`). -- **`()` vs `nil` for an empty realized seq** is the tracked **D-164** — - this is a **bug scheduled for fix** in the clj-parity campaign (ROADMAP - §9.2.P), NOT an accepted divergence. A single big-bang fix, uniform across - every seq fn. -- **F-NNN-intentional divergences** (`+`/`*` overflow auto-promote per - F-005; `(class …)` simple name per ADR-0059 = AD-003; error Kind = AD-007) - — check `project_facts.md` + the AD ledger before treating a DIFF as a gap. - -## Related - -- memory `clj_diff_sweep_methodology` — the original hand-method notes. -- `.claude/rules/orphan_prevention.md` — the `timeout` / bounded-seq rules - the harness obeys. -- `.dev/debt.yaml` D-164 (empty-seq `()` vs nil overhaul) — the standing - big-bang target this rule's Discipline 2 points at. (D-047 setString was - discharged 2026-06-02 via the `big_int.parseBase10` consolidated fix.) diff --git a/.claude/rules/cljw_invocation.md b/.claude/rules/cljw_invocation.md deleted file mode 100644 index 74009dfb3..000000000 --- a/.claude/rules/cljw_invocation.md +++ /dev/null @@ -1,109 +0,0 @@ ---- -paths: - - "test/e2e/**/*.sh" - - "bench/**/*.sh" - - "scripts/**/*.sh" - - "src/main.zig" ---- - -# Invoking `cljw` safely (Phase 3.1+) - -Auto-loaded when editing scripts that drive the `cljw` binary. Use -this rule whenever you write a smoke / e2e / bench script, or when -debugging interactively. - -## Three entry points - -`cljw` accepts code in three shapes (all available from §9.5 task -3.1 onward): - -| Shape | Invocation | Use when | -|---------------|-------------------------|------------------------------------------| -| Inline string | `cljw -e ''` | One-liner with no shell-special chars. | -| File | `cljw path/to/file.clj` | Multi-line code, fixtures, real scripts. | -| Stdin | `cljw -` (`-` literal) | Heredoc, pipe from another command. | - -Until 3.1 lands, only `-e` works. - -## Why `-e` is fragile in zsh - -zsh expands history references (`!`), parameter substitutions (`$x`, -`${x}`), command substitution (`` `cmd` ``, `$(cmd)`), and globs -inside double-quoted strings. Several Clojure idioms collide: - -| Clojure surface | Zsh interpretation | Failure mode | -|-------------------------------|---------------------------------------|-------------------------------------------------| -| `name!` | `!name` history reference | `zsh: event not found: ...` | -| `(deref @atom)` | nothing — but `\\@atom` may be eaten | Edge case in deeply-nested quotes. | -| `$ARGS` | shell variable expansion | `cljw -e "(+ $x 1)"` → `(+ 1)` if `$x` unset. | -| `` `name` `` | command substitution | shell tries to run a program. | -| `*foo*` (earmuff dynamic var) | glob expansion | filename match in cwd substitutes the symbol. | - -Single-quoting (`-e '...'`) defeats most of these, **but**: -- History expansion (`!`) still happens inside single quotes in - interactive zsh by default. Disable with `set +H` or just use a file. -- You cannot embed a literal single-quote inside `'...'` without - closing+reopening (`'...'\''...'`), which is unreadable for - Clojure code with `'symbol` quote forms. - -So **`-e` is fine for tests where you control the input**, but -**fragile for anything user-typed or anything containing earmuffs / -quote forms / non-trivial strings**. - -## Preferred patterns - -### A. File for fixtures / e2e / benches - -Put the Clojure source in a real file under `test/e2e/fixtures/` or -`bench/fixtures/`. Invoke with `cljw fixtures/foo.clj`. This is the -canonical pattern for `test/e2e/*.sh` and `bench/*.sh`. - -```bash -# test/e2e/phase3_macros.sh -got=$("$BIN" test/e2e/fixtures/defn_basic.clj) -[[ "$got" == "3" ]] || fail -``` - -### B. Heredoc via stdin for inline-but-tricky code - -When the code stays inline (so the script is self-contained) but -contains shell-specials, pipe via heredoc: - -```bash -# Single-quoted heredoc — NO shell expansion happens inside. -got=$("$BIN" - <<'EOF' -(defn name! [x] (str "hi, " x)) -(name! "world") -EOF -) -[[ "$got" == '"hi, world"' ]] || fail -``` - -Note the **single-quoted `'EOF'`** delimiter — that's what suppresses -expansion. Without quotes around `EOF` the heredoc would still -expand `$x`, `` `cmd` `` etc. The `eval-nrepl` skill uses the same -pattern; mirror it here. - -### C. `-e` for trivial inputs only - -`-e` is fine when: - -- The expression has no `!`, `$`, `` ` ``, `*` outside of math `*`, - or earmuff `*foo*`. -- The expression fits cleanly on one line. -- The script writer can audit every character. - -```bash -"$BIN" -e '(+ 1 2)' # safe -"$BIN" -e '(let* [x 1] (+ x 2))' # safe -"$BIN" -e '(println *out*)' # NOT safe (zsh globs *out*) -``` - -## When in doubt - -Default to **B (heredoc)** for scripts you write. It always works, -costs one extra line, and the intent ("run this exact code") is -visible. - -For interactive debugging at the prompt, `cljw ` is usually -fastest — keep a `scratch.clj` around. diff --git a/.claude/rules/clojure_spec_citation.md b/.claude/rules/clojure_spec_citation.md deleted file mode 100644 index 35a6e8ad7..000000000 --- a/.claude/rules/clojure_spec_citation.md +++ /dev/null @@ -1,59 +0,0 @@ ---- -paths: - - src/eval/** - - src/lang/** - - src/runtime/** - - src/runtime/host/** ---- - -# Clojure semantics citation - -## Rule - -Each primitive function and special-form implementation carries a -docstring that cites canonical Clojure semantics. - -Format: - -```zig -/// Implements clojure.core/conj. -/// Spec: `(conj coll x)` returns coll with x added at the appropriate position -/// - list: prepend -/// - vector: append -/// - set/map: add as element/entry -/// - nil: returns (x) (a one-element list) -/// JVM reference: clojure.core/conj in clojure/core.clj L.115 -/// cw v1 tier: A (Phase 3 implemented) -pub fn primConj(rt: *Runtime, ...) !Value { ... } -``` - -## Why - -- Surfaces Clojure spec divergence at code-review time, not at runtime. -- Anchors implementation to a stable reference (JVM Clojure source line) - rather than tribal memory. -- Tier classification is recorded next to the implementation, not only - in `data/compat_tiers.yaml`. - -## How to apply - -- New primitive: spec line is expected before merging. -- Modified primitive: spec line is updated alongside. - -## Enforcement - -**Reviewer-checked convention — no hard gate** (like `module_docstring.md`; -per `framework_completion.md`, a discipline without a backing script must say -so honestly). Discovery recipe to find primitives missing the spec line: - -```sh -# public primitives (pub fn prim*/special*) whose preceding doc block lacks a -# `/// Spec:` line — candidates for a spec citation. -rg -n 'pub fn (prim|special)\w+' src/lang/primitive src/eval \ - | while IFS=: read -r f l _; do - sed -n "$((l>6?l-6:1)),${l}p" "$f" | grep -q '/// Spec:' || echo "$f:$l" - done -``` - -A future `scripts/check_spec_citation.sh` could gate this (D-547); until then it -is a review-time check, not a blocking hook. diff --git a/.claude/rules/debt_dedup.md b/.claude/rules/debt_dedup.md deleted file mode 100644 index 984eed747..000000000 --- a/.claude/rules/debt_dedup.md +++ /dev/null @@ -1,55 +0,0 @@ ---- -paths: - - .dev/debt.yaml ---- - -# Debt deduplication discipline - -> `.dev/debt.yaml` is a structured YAML SSOT (ADR-0072 migration; 3 sections -> since the 2026-06-15 ledger audit): `active:` (actionable, drained -> easiest-first), `standing:` (epics/campaigns/capstones + the indefinitely- -> deferred future bucket — the loop does NOT auto-drain these), and -> `discharged:` (resolved/history). Each entry a mapping -> (`id` / `status` / `category` / `barrier` / optional `quality_floor` / -> `last_reviewed`; discharged is MIXED-schema — ~199 reuse the `status: -> DISCHARGED …` active shape, ~155 use the lighter `discharged_at` / -> `resolution`, so a row moved from active keeps its `status` block as-is). -> Recipes (next-id/dup/phantom) span all 3 sections — parenthesize any -> `(.active[],.standing[],.discharged[]) | select(…)` union (the `|`-binds- -> tighter-than-`,` footgun; see [`yaml_ssot_yq.md`](yaml_ssot_yq.md)). -> Edit it as YAML — there is no Markdown table to align any more. -> -> **Querying it from a shell** (counts, filters, is-discharged, next-id, the -> `env(VAR)` escaping idiom): see [`yaml_ssot_yq.md`](yaml_ssot_yq.md) — the -> `yq` cookbook, so the shell-escaping is not re-derived each time. - -## Rule - -Before adding a new debt entry, grep for class overlap: - -```sh -rg -n '' .dev/debt.yaml -``` - -Many additions are actually re-tagging an existing entry (status / barrier -/ last_reviewed update). - -## Why - -- Inflated debt count makes per-row predicate audit expensive. -- Duplicate root-cause tracking misleads Phase boundary judgment. - -## How to apply - -1. Extract keyword from the domain (e.g., "lazy-seq", "boxing", - "interop", "MVCC"). -2. `rg -n '' .dev/debt.yaml` to find related entries. -3. Update the existing entry (status / barrier / last_reviewed) if - relevant. -4. Otherwise append a new entry under `active:` with the next ID. Prefer the - Edit tool (hand-write `- id: "D-NNN"` quoted) over `yq +=` — a `+=` append - writes the id UNQUOTED, which the grep recipe undercounts (yaml_ssot_yq.md - Golden-rule #4). Highest existing id (style-agnostic via yq; MUST scope to - `.id`, else prose `D-NNN` cross-refs / a typo'd phantom inflate it): - `yq -r '.active[].id, .standing[].id, .discharged[].id' .dev/debt.yaml | grep -oE '[0-9]+' | sort -n | tail -1` - (grep fallback must be quote-tolerant: `grep -oE 'id: "?D-[0-9]+'`). diff --git a/.claude/rules/dual_backend_parity.md b/.claude/rules/dual_backend_parity.md deleted file mode 100644 index 432963988..000000000 --- a/.claude/rules/dual_backend_parity.md +++ /dev/null @@ -1,265 +0,0 @@ ---- -paths: - - "src/eval/node.zig" - - "src/eval/backend/tree_walk.zig" - - "src/eval/backend/vm.zig" - - "src/eval/backend/vm/**.zig" - - "src/lang/diff_test.zig" ---- - -# Dual-backend parity discipline - -Auto-loaded when editing the analyzer Node union or either backend -(TreeWalk / VM). Codifies the contract from -[ADR-0036](../../.dev/decisions/0036_dual_backend_parity_contract.md): -every new analyzer Node variant lands TreeWalk arm + VM compile arm -+ VM dispatch arm + ≥1 differential test case in the same commit. -The PreToolUse hook -[`scripts/check_dual_backend_parity.sh`](../../scripts/check_dual_backend_parity.sh) -mechanises the body-discipline + marker-form layer of the contract; -the compiler's exhaustive `switch (Node)` in `vm/compiler.zig` -already enforces arm presence at build time. - -## Why this rule exists - -ADR-0005 declared dual-backend differential testing as cw v1's -correctness oracle. F-002 makes "the finished form wins" project -law. The compiler-enforced exhaustive switch in -`src/eval/backend/vm/compiler.zig` ensures that every Node variant -has *some* arm in the VM compiler — but a `return error.NotImplemented` -body silently passes the build, and a `_ = some_field;` line -silently drops a significant analyser field. The differential test -layer covers only the variants someone wrote a case for; new -variants without diff coverage never trip CI. - -cw v0 carried the same drift class and ate weeks of late-cycle -catch-up time. cw v1's discipline machinery (PreToolUse hooks + -Devil's-advocate mandate) needed an explicit rule for "new -analyzer Node → both backends in same commit." This file is that -rule. - -## The contract - -A commit that adds a variant to the `Node` union in -`src/eval/node.zig` MUST also, in the same commit: - -1. Add the TreeWalk eval arm in `src/eval/backend/tree_walk.zig`. -2. Add the VM compile arm in `src/eval/backend/vm/compiler.zig`. -3. Add the VM dispatch arm in `src/eval/backend/vm.zig` (when a - new opcode is introduced). -4. Add at least one differential test case in - `src/lang/diff_test.zig` exercising the new variant. - -A commit that adds a new `op_*` to `src/eval/backend/vm/opcode.zig` -MUST also, in the same commit, add the matching dispatch arm in -`src/eval/backend/vm.zig` (the dispatch switch is exhaustive over -the `Opcode` enum, so the compiler enforces this — but the rule -records the expectation for code review). - -A commit that adds or modifies a VM compile arm body MUST NOT -ship the body as a silent gap. Two shapes count as silent gaps: - -- `return error.NotImplemented;` (or any error variant whose - message does not surface to the user via the catalog). -- `_ = some_field;` (a discard that drops a field the TreeWalk - arm honours). - -Either shape requires a `VM-DEFER:` marker (see below). The hook -blocks pushes whose commits contain either shape without a -matching marker. - -## VM-DEFER marker - -When a VM compile arm cannot ship its real implementation in the -introducing cycle — typically because the bytecode shape depends -on a design choice owned by a future Phase row — the arm may -ship as a transient stub or a documented field-drop provided the -line directly above carries: - -```zig -// VM-DEFER: [refs: D-NNN, feature_deps.yaml#] -``` - -The marker mirrors `// PROVISIONAL:` shape from -[`provisional_marker.md`](provisional_marker.md): - -- **Single line**. Multi-line rationale belongs in the - `data/feature_deps.yaml` entry body or the `.dev/debt.yaml` row. -- **`[refs:` block mandatory**. At least one `D-NNN` (debt row) - AND at least one `feature_deps.yaml#` (entry name). - Multiple refs comma-separated. -- **``** is the entry's `name:` field verbatim (e.g. - `runtime/vm/dispatch_family`, `runtime/vm/ns_filter`). -- **Placed directly above** the affected statement; no blank line - between marker and code. -- **Removed (not commented-out) on discharge.** The whole point - of the marker is grep-discoverability; a commented-out marker - cannot be discovered. - -### Example — VM compile arm transient stub - -```zig -// VM-DEFER: deftype VM bytecode shape pending row 7.6 MethodCallNode design [refs: D-073, feature_deps.yaml#runtime/vm/dispatch_family] -.deftype_node => return error.NotImplemented, -``` - -### Example — VM compile arm field-drop - -```zig -fn compileNs(self: *Compiler, n: node_mod.NsNode) Error!void { - // VM-DEFER: ns :refer-clojure filter pending NsNode filter extension [refs: D-073, feature_deps.yaml#runtime/vm/ns_filter] - _ = n.refer_clojure; - const name_val = try string_mod.alloc(self.rt, n.name); - const idx = try self.addConstant(name_val); - try self.emit(.op_in_ns, idx); -} -``` - -### Discharge - -The discharging commit's diff shows: - -- Marker line removed from the source arm. -- The arm body now lowers the construct for real (or, for a - field-drop, now consumes the field). -- The matching `data/feature_deps.yaml` entry's `status:` flips from - `provisional` to `landed`, and `provisional_markers:` list is - emptied. -- The matching `.dev/debt.yaml` entry's barrier predicate flips to - satisfied (or the entry moves to the `discharged:` list). -- ≥1 differential test case added covering the newly-landed VM - arm (per contract point 4). - -The hook (`scripts/check_dual_backend_parity.sh`) verifies the -marker form + body-discipline layer; the `data/feature_deps.yaml` + -`.dev/debt.yaml` discharge sync is verified by the existing -`scripts/check_provisional_sync.sh` (the shape is identical). - -## VM-DEFER vs PROVISIONAL - -| Concern | PROVISIONAL | VM-DEFER | -|----------------------|---------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------| -| Shape | Intermediate semantics that ride a chicken-and-egg layer dependency | Backend asymmetry where TreeWalk has real semantics and VM is stubbed/field-dropped pending a future Phase choice | -| Close-out trigger | Upstream layer feature lands | Future Phase row decides the bytecode shape | -| Source SSOT | `data/feature_deps.yaml` entry's `provisional_markers:` list | `data/feature_deps.yaml` entry's `provisional_markers:` list (same shape — re-used for parity tracking) | -| Hook | `scripts/check_provisional_sync.sh` | `scripts/check_dual_backend_parity.sh` | -| Bad Smell preventing | Silent default-shift | Dual-backend drift | - -The two markers coexist; nothing prevents a single -`data/feature_deps.yaml` entry from carrying both a PROVISIONAL marker -(for the layer dependency aspect) and a VM-DEFER marker (for the -backend parity aspect). Today no entry does — but the shape is -deliberately compatible. - -## Discovery criterion (per `framework_completion.md`) - -The mechanical recipe the hook + Step 0.6 re-laying both use: - -```sh -# All Node variants (LHS of the union(enum)): -grep -nE '^\s+[a-z_]+_node\s*:' src/eval/node.zig - -# All TreeWalk arms: -grep -nE '\.[a-z_]+_node\s*=>' src/eval/backend/tree_walk.zig - -# All VM compile arms (compileNode switch): -grep -nE '\.[a-z_]+_node\s*=>' src/eval/backend/vm/compiler.zig - -# All VM-DEFER markers in vm/compiler.zig: -grep -nE '^\s*//\s*VM-DEFER:' src/eval/backend/vm/compiler.zig - -# All silent-gap candidates in vm/compiler.zig: -grep -nE 'return error\.NotImplemented' src/eval/backend/vm/compiler.zig -grep -nE '^\s+_ = n\.' src/eval/backend/vm/compiler.zig -``` - -The first three lists should produce the same Node variant tag -set (LHS of `=>`). Any tag in TreeWalk but absent from VM -compiler is a parity gap. - -The last two lists are silent-gap candidates; each line should -have a `// VM-DEFER:` marker on the line directly above it. - -## Scope clarification - -- The rule applies to commits that touch backend dispatch files - (paths listed in this rule's frontmatter). A commit that does - not touch any backend file is out of scope (the hook short- - circuits on the file list). -- The rule does NOT require diff coverage for every existing - Node variant retroactively — only for variants whose VM arm is - fully working (not VM-DEFER) AND that are added or modified in - the commit. T1's 11-case land is a one-time retrofit per the - framework_completion §C discipline. -- TreeWalk-only paths (e.g. test-only helpers, REPL surface) are - out of scope. The rule targets the analyzer-Node → backend - contract specifically. - -## Counter-examples - -❌ Land a new `MethodCallNode` variant in `node.zig` + TreeWalk -arm + Phase 7 row 7.6 ADR; defer the VM arm "to next cycle" -without a VM-DEFER marker. The hook blocks the push. - -❌ Add `return error.NotImplemented` to an existing VM compile -arm body without a `VM-DEFER:` marker line above. The hook -blocks the push. - -❌ Add a `VM-DEFER:` marker without the `[refs:` block. The -hook blocks the push. - -❌ Discharge a VM-DEFER marker without adding the diff coverage -case. The hook does not currently block this (it could be added -as a future enhancement); reviewer catches. - -✅ Land MethodCallNode + TreeWalk arm + VM compile arm + new -opcode `op_method_call` + VM dispatch arm + diff case in one -commit. Hook passes. - -✅ Land MethodCallNode + TreeWalk arm + VM compile arm shipped as -`// VM-DEFER: method dispatch ABI pending row 7.6.b design -[refs: D-073, feature_deps.yaml#runtime/vm/dispatch_family]` + -return error.NotImplemented + matching debt row + yaml entry. -Hook passes. - -## Enforcement - -`scripts/check_dual_backend_parity.sh` runs as a PreToolUse:Bash -hook on `git push` (registered in `.claude/settings.json` -alongside `check_provisional_sync.sh`, `check_smell_audit.sh`, -etc.). It walks each unpushed commit's range and checks: - -1. **Body discipline**: any line in - `src/eval/backend/vm/compiler.zig` matching - `return error.NotImplemented` or `_ = n\.\w+;` must have a - `// VM-DEFER:` marker on the line directly above. -2. **Marker form**: any newly-added `VM-DEFER:` marker line must - contain a `[refs: …]` block with at least one `D-NNN` - reference AND at least one `feature_deps.yaml#` reference. - -The hook supports the same `--test-range RANGE` and -`--test-staged` modes as `check_provisional_sync.sh`. It is -exercised in practice by the per-commit push gate (a malformed or -missing marker on a dual-backend change is rejected at push time); -a dedicated self-test script is not yet written (tracked W-010). - -## Cross-references - -- [ADR-0036](../../.dev/decisions/0036_dual_backend_parity_contract.md) - — the parity contract this rule operationalises. -- [ADR-0005](../../.dev/decisions/0005_dual_backend_differential_oracle.md) - — foundational dual-backend ADR. -- [ADR-0021](../../.dev/decisions/0021_test_taxonomy.md) — test - taxonomy (Layer 3 differential lives in `src/lang/diff_test.zig`). -- [`provisional_marker.md`](provisional_marker.md) — sibling - marker class whose shape VM-DEFER mirrors. -- [`framework_completion.md`](framework_completion.md) — - discovery + sweep + retrofit-in-same-cycle discipline. -- [`.dev/principle.md`](../../.dev/principle.md) — Bad Smell - catalogue ("Dual-backend drift" entry). -- [`.dev/debt.yaml`](../../.dev/debt.yaml) D-073 — VM backend parity - cluster. -- [`scripts/check_dual_backend_parity.sh`](../../scripts/check_dual_backend_parity.sh) - — the enforcement hook. -- [`scripts/check_provisional_sync.sh`](../../scripts/check_provisional_sync.sh) - — the shape template the new hook adapts. diff --git a/.claude/rules/error_catalog_only.md b/.claude/rules/error_catalog_only.md deleted file mode 100644 index 9403b0b5a..000000000 --- a/.claude/rules/error_catalog_only.md +++ /dev/null @@ -1,152 +0,0 @@ ---- -paths: - - src/**/*.zig ---- - -# Error catalog is the only message source - -## Rule - -User-facing error messages come from `src/runtime/error_catalog.zig` -only. Other modules call `error_catalog.raise(.code, loc, args)`; -direct `error_mod.setErrorFmt(...)` calls are reserved for the -catalog itself. The Zig error union returned is `ClojureWasmError` -(full spelling) so the catch surface and failing test traces name -the project. - -## Why - -- Single grep surface for every message the runtime produces. -- Templates stay consistent (no drift between similar errors written - in two files). -- Development concepts (Phase numbers, ADR identifiers, internal - file paths, URLs) cannot leak into user-facing text because the - catalog template is the only place those would live, and the - catalog forbids them. -- Adding a new error is two edits in one file rather than a fresh - `setErrorFmt(...)` call somewhere new. -- `Kind` (16 variants today) is the user-visible category label - (`[name_error]` / `[type_error]` / ...) so the user grasps what - kind of problem they hit without seeing the catalog `Code`. It is - also the `catch |err|` discriminator for cw internal dispatch. - -## How to apply - -### Adding a new error - -1. Append a variant to `Code` in `src/runtime/error_catalog.zig`. - Naming follows the `_` convention - (see below). -2. Append the matching `entry()` arm with `kind`, `phase`, and a - `template` using named placeholders - (`"{[fn_name]s}: expected number, got {[actual]s}"`). -3. Call `raise(.your_code, loc, .{ ... named args ... })` at the - raise site. The return type is `ClojureWasmError`. - -### Code variant naming convention - -`_` in snake_case, prioritising the form / -construct the user wrote over internal classification: - -- `delimiter_unexpected`, not `parse_unexpected_delimiter`. -- `string_unterminated`, not `parse_unterminated_string`. -- `def_arity_invalid`, not `analysis_def_arity`. -- `type_arg_not_number`, not `eval_type_expected_number`. -- `arity_invalid`, `let_bindings_invalid`, - `if_let_bindings_not_pair`, ... - -Exceptions (Tier D and sub-feature staged unsupported keep a prefix): - -- Tier D: `tier_d_` — `tier_d_gen_class`, - `tier_d_gen_interface`, `tier_d_compile`, `tier_d_proxy_deep`, - `tier_d_bean_deep`. One Code per form, with a hand-written - user-helpful template (often multiple sentences) that explains - the reason and suggests the cw-native alternative. -- Sub-feature staged: `__not_supported` — - `stm_dosync_not_supported`, `stm_commute_not_supported`, ... - Transient Codes that disappear when the sub-operation lands. -- Generic fallback: `feature_not_supported` (no prefix; the sole - exception). Use only when the feature has not yet been broken - down into sub-feature Codes. - -The `Phase` is **not** encoded in the Code name; the `entry()` arm -carries it. - -### Template hygiene - -Templates must NOT contain: - -- Phase numbers (`"Phase 5: ..."`). -- ADR identifiers (`"see ADR-0010"`). -- Tier classification labels (`"Tier D: ..."`). -- URLs (`"https://..."`). -- cw internal file paths (`"src/eval/analyzer.zig"`). - -Templates SHOULD: - -- Name the construct the user wrote (`"dosync"`, `"gen-class"`). -- Use the same wording as comparable errors (consistency). -- Quote string-like arguments (`'{[token]s}'`). -- For Tier D Codes specifically: a 1-3 sentence template that - explains *why* (the technical reason) and *what to use instead* - (the cw-native alternative). - -### Tier D forms - -For Tier D forms (permanently out of scope), each form has its own -Code with a hand-written template: - -```zig -return error_catalog.raise(.tier_d_gen_class, loc, .{}); -// → "gen-class requires emitting JVM bytecode classes, which is -// outside the scope of ClojureWasm. Use deftype + defprotocol -// for cw-native type definitions." -``` - -The form name appears in the template itself (no `{name}` slot). - -### Sub-feature staged unsupported - -For features landing across multiple Phases (STM, atom watchers, -agent action queue, ...), each unimplemented sub-operation gets its -own Code: - -```zig -return error_catalog.raise(.stm_dosync_not_supported, loc, .{}); -// → "dosync is not yet supported in ClojureWasm." -``` - -When the sub-operation lands (e.g. Phase 15.1 implements `dosync`), -the Code is removed from `error_catalog.zig` and the raise site is -deleted. - -### Generic fallback - -`feature_not_supported` carries a `{name}` slot and renders -`" is not yet supported in ClojureWasm."`. Use only when the -feature has not been resolved into specific sub-feature Codes yet. -Each call site is a candidate for promotion to a named Code as -Phase planning resolves. - -## Counter-examples - -Don't add a new `setErrorFmt(.eval, .type_error, loc, "...", args)` -call outside `error_catalog.zig`. Add a `Code` variant instead. - -Don't write `"Tier D: dosync, see ADR-0013"` — the user does not -care which tier or ADR. - -Don't write `"Phase 15: STM not yet wired"` — Phase 15 is a cw -development concept. - -Don't reuse `tier_d_gen_class` for a different form. One Code per -Tier D form. - -## Enforcement - -- ADR-0018 specifies the contract. -- `scripts/check_no_op_stub.sh` extends to flag bare - `setErrorFmt(...)` calls outside `error_catalog.zig` (heuristic - grep). Current state: informational only. -- Reviewers reject `setErrorFmt(...)` introductions outside the - catalog. diff --git a/.claude/rules/exploration_vs_done.md b/.claude/rules/exploration_vs_done.md deleted file mode 100644 index 2437f1343..000000000 --- a/.claude/rules/exploration_vs_done.md +++ /dev/null @@ -1,83 +0,0 @@ ---- -paths: - - "**" ---- - -# Exploration vs Done - -## Rule - -cw v1 has two modes of work: - -- **Exploration**: free-form. Tracked files are not yet touched. - Scratch lives in `private/notes/`, REPL sessions, throwaway - `.clj` files. The only rule is "don't break the working tree". -- **Done**: tracked. A change lands in `src/` / `.dev/` / - `.claude/` / `scripts/` / `test/` / `bench/` / `docs/ja/` / - `docs/architecture.md` / `CLAUDE.md` / `data/compat_tiers.yaml`. Done - requires the gate. - -The transition from exploration to Done has explicit checkpoints. - -## Why - -- Exploration without freedom slows research (every grep wants to - be a test). -- Done without a gate is how code rots (a half-rewritten module - lands "to fix later"). -- The boundary needs to be visible so the agent can self-locate - ("am I in exploration or am I trying to land?"). - -Pollaroid `CLAUDE.md` L31-65 documents this discipline; cw v1 -adopts the same shape with cw-specific gates. - -## Done gate (every tracked-file landing) - -Before staging a tracked file for commit: - -1. **Build is green**: `zig build test` passes for source changes. -2. **Smoke passes per commit**: `bash test/run_all.sh --smoke - ` is green; the **full gate batches** at the ≤5 - ceiling / Phase boundary / pre-tag (ADR-0107 two-tier — full e2e is - heavy; SSOT `.claude/rules/gate_cadence.md`). -3. **ADR reference**: the change either implements an existing - ADR row or carries its own ADR / amendment per ROADMAP §17. -4. **handover.md updated**: current state reflects the landing. -5. **Doc paired** — DORMANT per ADR-0025: the per-concept chapter - cadence is suspended (its pre-commit pairing gate is a no-op); only - the per-task note in `private/notes/` is required. - -If any of the five fails, the change is still exploration. - -## Exploration freedom - -Inside exploration mode: - -- Multi-line `cljw -e '...'` calls, REPL evaluations, scratch - `.clj` files outside `test/` are fine. -- `private/notes/-.md` carries the work log. -- Subagent-generated drafts may live in `private/` until - promoted. - -Only when a finding stabilises into "this should land" does the -Done gate apply. - -## How the agent self-locates - -When in doubt, ask: "is this in a tracked path that -`git status` reports?" If yes, Done gate applies. If no -(private/, REPL, /tmp), exploration mode. - -## Counter-examples - -Don't stage a tracked file edit without running the gate -("just a docs typo" — still the gate applies, `md-table-align` -catches the formatting). - -Don't keep tracked files in a half-rewritten state across multiple -commits without an explicit ADR amendment that says "Phase X part -1 lands here, part 2 in commit Y". - -Don't promote a `private/notes/` finding to tracked text without -re-reading what tracked authority already says on the topic -(ADR / ROADMAP / rule). diff --git a/.claude/rules/extended_challenge.md b/.claude/rules/extended_challenge.md deleted file mode 100644 index 0ca44bbe6..000000000 --- a/.claude/rules/extended_challenge.md +++ /dev/null @@ -1,43 +0,0 @@ ---- -paths: - - "**" ---- - -# Extended challenge before a user stop - -## Rule - -The loop never stops on its own (CLAUDE.md § The only stop). The -only moment this rule fires is when the **user has just asked the -loop to stop** and the per-task note is being written to close -the session. - -In that note, record 3 short items so the next session resumes -without spinning up cost: - -1. **Alt hypothesis**: an alternative approach not taken - (3-5 sentences). -2. **Next experiment**: a concrete next experiment with an - executable command. -3. **Explicit blocker**: what currently blocks the next experiment - (technical / decision / external) — or "none" if the next - experiment is ready to run. - -## Why - -- Prevents the next session from re-deriving context. -- Surfaces alternatives that the smell sensor flagged but did not - surgery — they become the next session's first considered moves. -- Keeps the handover-to-resume bridge concrete, not narrative. - -## How to apply - -When the user issues a stop directive, record the 3 items as the -last section of `private/notes/-.md` (the per-task -note for the in-flight task). Then write the -`## Stopped — user requested` section in handover per -`handover_framing.md` and push. - -Do **not** trigger this rule on task / region / cluster / commit -boundaries. Those roll into the next unit of work; no stop, no -provisioning needed. diff --git a/.claude/rules/feature_name_consistency.md b/.claude/rules/feature_name_consistency.md deleted file mode 100644 index dd70b3869..000000000 --- a/.claude/rules/feature_name_consistency.md +++ /dev/null @@ -1,201 +0,0 @@ ---- -paths: - - src/runtime/** - - src/lang/primitive/** - - data/compat_tiers.yaml ---- - -# Feature-name consistency + Java/cljw surface layout - -> Codifies ADR-0029 D1-D6 (directory layout, dependency direction, -> keyword discipline, Backend marker docstring, compat_tiers schema) -> + F-009 (feature-implementation neutrality). Supersedes the prior -> standalone `host_extension_layout.md` and `java_cljw_surface_layout.md` -> rules (the latter folded into this file at Wave 16, W16-5, -> 2026-05-26 per D-050(a)). - -## Background - -F-009 mandates that a feature's implementation, surface wrappers -(Java + cljw + Clojure-ns), and value-wrap utilities live in -**different files** for namespace-neutral implementation sharing. -A single feature like `java.util.UUID` therefore involves at -minimum 3 files (impl / surface / Clojure peer) — often 4-5 when -sub-impls and value-wrap helpers join. - -This is structurally unavoidable. The mitigation is **discoverability -by feature keyword**: every related file has the keyword in its -path so `rg src/` returns 100% of the relevant files. - -## Rule - -### R1. Each feature owns one keyword - -A keyword is a short lower-snake-case identifier (e.g., `uuid`, -`file_io`, `clock`, `regex`, `digest`) declared once in -`data/compat_tiers.yaml`: - -```yaml -- fqn: java.util.UUID - keyword: uuid - files: - surface: runtime/java/util/UUID.zig - impl: runtime/uuid.zig - impl_extras: [runtime/crypto/secure_random.zig] - wrap: runtime/collection/string.zig - clojure_peer: lang/primitive/uuid.zig -``` - -The keyword **must** appear as a path component or filename stem -in every file path listed under `files:`. Exceptions: - -- `wrap:` (value-wrap helpers like `collection/string.zig`) may - legitimately not contain the keyword — they are reused across - features. This slot is exempt from G3. -- `surface:` (Java-surface paths under `runtime/java//.zig`) - uses the Java class name (PascalCase) which rarely matches the - lower_snake_case impl keyword. Grep by Java class name is the - canonical way to find a surface; G3 enforces the keyword link - only on impl / impl_extras / clojure_peer slots. - -### R2. Backend marker docstring on every surface file - -Every `runtime/java/**/*.zig` and `runtime/cljw/**/*.zig` file -must open with: - -```zig -// SPDX-License-Identifier: EPL-2.0 -//! -//! -//! Backend: -//! Impl deps: -//! Clojure peer: -``` - -- `Backend:` documents what category of impl this surface calls. -- `Impl deps:` lists feature keywords (matching - `data/compat_tiers.yaml`) that this surface depends on. -- `Clojure peer:` names the Clojure-ns Var(s) that share the - same neutral impl, or `none`. - -Exceptions: - -- `_host_api.zig` and `_README.md` are markers / docs, not - surfaces; exempt from R2. - -## Why - -- **Grep 100%.** `rg uuid src/` returns every UUID-related file - in one shot — impl, surface, Clojure peer. -- **Reading 5 lines tells the reader the wiring.** Backend - marker says where the impl lives, what other features it - depends on, and which Clojure Var shares the same impl. -- **F-009 stays enforceable.** Without the keyword consistency - + marker discipline, fan-out becomes "where is the UUID code?" - navigation pain, and the loop drifts toward inlining impl into - the surface to make navigation easier (= smallest-diff bias). -- **Index integrity.** `data/compat_tiers.yaml` `files:` is the - authoritative cross-reference; G3 ensures it stays honest. - -## How to apply (when adding a new feature) - -1. Pick a short, distinctive keyword (e.g., `digest` for - `java.security.MessageDigest`). -2. Verify no existing entry already claims that keyword - (`rg "keyword: " data/compat_tiers.yaml`). -3. Write the neutral impl file with the keyword in its path - (`runtime/crypto/message_digest.zig` — keyword `digest` - appears as `message_digest`). -4. Write the surface file under `runtime/java/security/MessageDigest.zig` - (keyword `digest` appears in `MessageDigest`). -5. Add the surface file's Backend marker docstring. -6. If there is a Clojure peer, add `lang/primitive/digest.zig` - (keyword `digest` in filename). -7. Add the `data/compat_tiers.yaml` entry with `keyword: digest` + - `files:` map listing all the above paths. -8. Run `bash test/run_all.sh`. G2 (`check_surface_marker.sh`) - and G3 (`check_feature_keyword.sh`) must pass. - -### R3. Java + cljw surface directory layout (ADR-0029 D1) - -Surface files live under two parallel trees: - -``` -src/runtime/java/ (Java-compat surface) -├── _host_api.zig ___HOST_EXTENSION marker aggregator -├── lang/ Object/String/Long/Integer/Double/Boolean/ -│ Character/Math/System/Throwable/Exception/ -│ RuntimeException/Thread -├── io/ File/PrintWriter/InputStream/OutputStream/ -│ Reader/Writer/ByteArrayInputStream/ByteArrayOutputStream -├── util/ UUID/Date/Random/Locale/regex/{Pattern,Matcher}/ -│ concurrent/{Future, atomic/AtomicLong} -├── time/ Instant/LocalDate/LocalDateTime/Duration/ -│ ZonedDateTime/ZoneId -├── net/ URL/URI (Phase 14+) -├── nio/ file/{Path,Files}/charset/Charset -├── math/ BigInteger/BigDecimal -├── security/ MessageDigest/SecureRandom (Phase 14+) -└── reflect/ Method/Field via TypeDescriptor (Phase 7+ Tier C) - -src/runtime/cljw/ (cljw-original surface) -├── _host_api.zig optional aggregator -├── build/ Compiler (Phase 12) -├── wasm/ Engine/Module/Instance/Component (Phase 16+) -├── edge/ Server/Request (Phase 14+) -├── pod/ Pod (bb-style pod 互換) (Phase 16+) -└── repl/ NReplServer (Phase 10+) -``` - -User code reaches Java entries as `(:require [cljw.java.util :refer -[UUID]])` — `cljw.host.*` is retired per ADR-0029. cljw-original -surfaces ship Zig extensions (Wasm component invoke / edge runtime / -build command / pod invocation); anything that is a Clojure library -belongs in `lang/clj/` (in-source). - -### R4. Dependency direction (ADR-0029 D2, gate G1) - -> No file under `runtime/` other than those in `runtime/java/**` -> and `runtime/cljw/**` may import from `runtime/java/**` or -> `runtime/cljw/**`. - -Surface layers (`java/`, `cljw/`) call the **neutral impl layer** -(everything else in `runtime/`). The reverse is forbidden. Cross- -surface horizontal calls are also forbidden: - -- `runtime/java/.zig` must not import `runtime/cljw/.zig` -- `runtime/cljw/.zig` must not import `runtime/java/.zig` -- `lang/primitive/.zig` must not import `runtime/java/` or - `runtime/cljw/` - -All three surface families (Java, cljw, Clojure-ns) share the -**same neutral impl**. `scripts/zone_check.sh` (extended per -ADR-0029 D4) gates these constraints in CI. - -## Counter-examples - -- **Bad keyword choice**: `keyword: m` (too generic). Pick a - 4-12 char distinctive word. -- **Forgetting the marker docstring** on a new - `runtime/java/.zig`: G2 blocks. -- **Listing a file in `files:` that lacks the keyword**: G3 blocks - (use the `wrap:` slot for legitimately-reusable helpers). -- **Body inline in surface file**: forbidden per F-009; surface is - a thin wrapper over the neutral impl in `runtime/.zig`. -- **Cross-surface import** (`runtime/java/` → `runtime/cljw/` or - vice versa): forbidden per R4; both reach the shared impl. -- **`cljw.host.*` ns prefix**: retired by ADR-0029; use - `cljw.java.*` for Java compat ns and `cljw.*` for cljw-original. - -## Related - -- ADR-0029 D1-D6 (directory layout, dependency direction, G1/G2/G3 - guardrails, compat_tiers schema). -- F-009 — feature-implementation neutrality (the invariant this - rule operationalises). -- ADR-0011 — superseded by ADR-0029 (the `___HOST_EXTENSION` marker - pattern carries forward). -- `scripts/zone_check.sh` (G1). -- `scripts/check_surface_marker.sh` (G2). -- `scripts/check_feature_keyword.sh` (G3). -- `data/compat_tiers.yaml` — extended schema per ADR-0029 D5. diff --git a/.claude/rules/framework_completion.md b/.claude/rules/framework_completion.md deleted file mode 100644 index af5712d35..000000000 --- a/.claude/rules/framework_completion.md +++ /dev/null @@ -1,172 +0,0 @@ ---- -paths: - - ".claude/rules/**/*.md" - - ".claude/skills/**/SKILL.md" - - "scripts/check_*.sh" - - ".claude/settings.json" - - "data/feature_deps.yaml" - - "data/placement.yaml" - - "data/compat_tiers.yaml" ---- - -# Framework completion - -Auto-loaded when editing any framework / scaffolding file (rule, skill, -hook script, SSOT yaml, settings). Codifies the discipline that -introducing a new discipline is **not enough** — the same cycle that -introduces it must also retrofit the existing codebase to the new -discipline. Otherwise the codebase carries an asymmetry (= new files -follow the rule, old files don't) that no one is responsible for -healing. - -## The rule - -A cycle that does any of: - -- adds a new `.claude/rules/.md` (especially with a `paths:` - frontmatter that targets pre-existing source) -- introduces a new SSOT yaml (`data/feature_deps.yaml`, - `data/placement.yaml`, ...) -- lands a new PreToolUse hook script (`scripts/check_*.sh`) -- amends an existing rule in a way that broadens its scope -- adds a new Bad Smell entry to `principle.md` -- introduces a new marker convention (`PROVISIONAL:`, `TODO:` ban, - etc.) - -...MUST, in the same cycle, do all three of: - -1. **Define the discovery criterion** — a concrete grep / yq / parser - recipe that enumerates existing sites the new rule would cover. - Recipe lives in the rule body or in the new script. -2. **Run the sweep** — execute the recipe against the current - codebase. Output goes to `private/notes/--sweep.md` or - inline in the per-task note. -3. **Land the retrofit** — for each enumerated site, either: apply the - rule (= edit file to comply); OR add an explicit exemption (= note - in `.dev/watch_findings.md` with revisit trigger); OR document why - the site is genuinely out of scope. - -A cycle that introduces the rule without doing all three is -**framework-incomplete** — a new smell defined in -`.dev/principle.md`. The codebase ends up with a 2-tier population -(new sites compliant, old sites unmarked), and no row in `.dev/debt.yaml` -points at the gap. - -## Why - -The Wave-15 spike (provisional-marker framework, 2026-05-26) initially -landed the rule + SSOT + hook (commits 1fdc342 / 0fed954) before any -existing provisional behaviour was marker-retrofitted. The -silent-failure-hunter review caught 5 sites that the initial -retrofit missed (set/walk/string `.clj` heads + primitive.zig + -macro_transforms.zig), because the discovery criterion was implicit -("things I changed in the recent 6.16.b cycles") rather than -mechanical ("grep for `(in-ns 'foo)` heads + grep for -`referAll(rt_ns, ...)` calls"). - -The fix is to require the discovery criterion as a deliverable of the -framework-introducing cycle itself. - -## How to apply - -### A. Discovery criterion shape - -A discovery criterion is a deterministic recipe. Examples: - -| Rule landed | Discovery criterion | -|----------------------------------|--------------------------------------------------------------------------------------------------------------------| -| `provisional_marker.md` | `rg --no-heading -nl '(stands in for\|for now\|until Phase \d+\|temporarily)' src/ test/e2e/` | -| `module_docstring.md` (Phase 5+) | `find src -name '*.zig' -newer \| xargs -I{} head -2 {} \| ...` (first 2 lines start `// SPDX:` + `//! ...`) | -| `feature_name_consistency.md` G3 | `yq '.host_classes[].keyword' data/compat_tiers.yaml \| xargs -I{} rg -l '{}' src/` | -| `error_catalog_only.md` | `rg --no-heading -n 'setErrorFmt\(' src/ \| grep -v 'runtime/error/catalog.zig'` | -| `handover_framing.md` | grep recipe at L189-194 of the rule itself | - -The criterion may be opinionated (= false positives possible) but must -be reproducible. - -### B. Sweep output - -Land the sweep output as a per-task note (or per-cycle survey): - -``` -# Sweep: introduction () - -Discovery recipe: `` - -Hit count: -Hits: - - : — - ... - -Coverage check: sites considered; applied; - recorded in watch_findings.md as W-NNN; noted -inline. -``` - -### C. Retrofit commit shape - -The retrofit edits land in the same cycle: - -- One commit per concern (per-cluster grouping is fine: spike 2.1 + - 2.2 + 2.3 across 4 sub-commits is the cw v1 precedent). -- Each commit's `Smell-audited:` line acknowledges the framework- - completion discipline. -- The per-task note's `## 暫定ログ` section records the introduced / - discharged / surfaced markers. - -### D. Exemption shape - -A site that does not get the new marker / yaml entry / debt row must -have an explicit exemption row in `.dev/watch_findings.md`: - -``` -| W-NNN | | : — ; not applied because | | | YYYY-MM-DD | -``` - -A site without either a comply-application or an exemption row is a -**bug in the cycle**. - -## Counter-examples - -❌ Land `provisional_marker.md` + 11-site retrofit; never grep for -`stands in for` / `referAll(rt_ns, ...)` patterns the rule covers. -Review catches 5 missed sites → Framework-incomplete smell fired in -review, not in the cycle. - -❌ Introduce a new Bad Smell entry in `principle.md` without -mentioning where existing code might have triggered it. The smell -description references no concrete site, so the catalogue grows but -the codebase doesn't. - -❌ Add a hook script with a `paths:` matcher and ship it without -running the script against the current tree. The first real commit -that hits the matcher fails the hook unexpectedly. - -✅ Land the rule + scripts + sweep output + retrofit + exemption rows -all in one cycle. The next session reads the cycle history and finds -the discovery recipe + the exemption list together. - -## Enforcement - -This rule is **prose-aspirational** today (it auto-loads on rule / -skill / script / yaml edits via the `paths:` frontmatter, so the -agent reads it whenever introducing a new discipline). A future -`scripts/check_framework_completion.sh` could mechanise it by: - -- Detecting cycles that add a new `.claude/rules/.md` or - `scripts/check_*.sh` -- Requiring the same cycle to add either a sweep output file or an - `.dev/watch_findings.md` row that references the new rule - -For now the discipline lives in the rule body + Step 0.6 main-agent -re-laying + audit_scaffolding E2.7 telltale sweep at Phase boundary. - -## Cross-references - -- `.dev/principle.md` Bad Smell catalogue — Framework-incomplete + - Defer-to-amnesia entries (Wave-16 addition). -- `.dev/watch_findings.md` — the SSOT for exemption rows. -- `audit_scaffolding/CHECKS.md` E2.7 telltale sweep — the periodic - catch-up sweep that compensates for missed introductions. -- `provisional_marker.md` — the canonical example of the framework - this rule operationalises around. diff --git a/.claude/rules/gate_cadence.md b/.claude/rules/gate_cadence.md deleted file mode 100644 index 561a8aa83..000000000 --- a/.claude/rules/gate_cadence.md +++ /dev/null @@ -1,196 +0,0 @@ ---- -paths: - - "test/run_all.sh" - - "test/**/*.sh" - - "scripts/run_gate.sh" - - "scripts/gate_state_hash.sh" - - "scripts/check_gate_cadence.sh" - - ".claude/settings.json" ---- - -# Gate cadence (smoke per-commit, batch the full gate) - -Auto-loaded when editing the test runner / hook scripts / settings. -Codifies *when* the fast smoke (`bash test/run_all.sh --smoke`, ~tens of -sec) vs the full Mac gate (`bash test/run_all.sh`, **~5 min** — the 248 -e2e shell steps grew heavy) must run, and makes the policy -**mechanically enforced** rather than prose. SSOT for the decision is -[ADR-0107](../../.dev/decisions/0107_pipeline_gate_smoke_authorized.md). - -## Why - -The full gate is now **~5 min**, ~60% of it the 248 e2e shell steps, -which also false-time-out under host load (the `gate_parallel_e2e_timeout` -memory). Running it per commit was the iteration bottleneck. ADR-0107 -found the cost/risk axis is **unit-vs-e2e, not additive-vs-shared**: - -- `zig build test` on **both** backends IS the entire F-012 correctness - guarantee — `src/lang/diff_test.zig` (the dual-backend differential - oracle) is `@import`ed from `src/main.zig`, so every diff case runs - inside it. It is timing-independent, contention-tolerant, and fast-ish. -- The 248 e2e shell steps + perf/bench are the slow, load-flaky part — - the only part worth batching. - -So the smoke = the fast correctness core (both-backend `zig build test` -+ zlinter + `build_cljw` + `corpus_regression` + **the changed e2e -step(s)**). Because it carries the full diff oracle, the smoke -**authorises shared-code commits too** — the old "additive vs risky" -split is retired. Only the full e2e suite is deferred. - -So: **smoke every commit (shared-code included), batch the full gate.** - -## The policy - -| Commit class | Definition | Gate requirement | -|----------------|-----------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------| -| **Non-source** | No `src/` / `test/` / `build.zig*` staged (docs, `.dev/`, `.claude/`, `scripts/`) | Exempt — never gated, never counted | -| **Source** | Touches `src/` / `test/` / `build.zig*` (additive OR shared-code, no distinction) | A fresh **smoke** authorises it; may ride up to **5** since the last full gate, the 6th blocks | - -A fresh **full gate** clears the batch counter (the next 5 ride again). -"Fresh" (smoke or full) = ran to green on the **exact content** being -committed (the `gate_state_hash.sh` fingerprint matches). The full gate -is `test/run_all.sh` with no `--only`/`--skip`, run ALONE (`--serial-e2e` -for determinism), at the 5-commit ceiling / Phase boundary / pre-tag. - -## Enforcement (ADVISORY since 2026-06-11 — effort-goal, not a hard block) - -> **User-directed 2026-06-11**: `check_gate_cadence.sh` no longer *blocks* -> commits. The smoke-per-commit / batch-the-full-gate cadence below is an -> **effort-goal (努力目標)** — the hook prints a `⚠ advisory` recommendation and -> exits 0. It was over-strict (it blocked + forced a re-smoke when the md-table -> hook reformatted a `test/` doc mid-commit). The smoke/full-gate discipline is -> still the right rhythm — run it — but it is now self-enforced, not hook-blocked. -> The description below documents the *recommended* cadence; "blocks" / "law" -> wording is historical. - -### Recommended cadence (was: mechanical enforcement) - -- **`scripts/gate_state_hash.sh`** — prints a fingerprint by hashing the - *content* (path + bytes) of every tracked or untracked-non-ignored file - under `src/` + `test/` + `build.zig*` (`bench/` excluded). Hashing - content — not a diff vs HEAD — makes the fingerprint independent of HEAD - position and of staging, so it still matches after `git add` and survives - a `git add && git commit` batched into one shell command. -- **`test/run_all.sh`** — a **full** green gate (no `--only`/`--skip`) writes - the fingerprint to `.dev/.gate_pass` AND clears the batch counter - `.dev/.gate_cadence` to 0; a **`--smoke`** green writes it to - `.dev/.smoke_pass` (all gitignored). A full gate validates everything up - to now, so the batch restarts; a smoke validates the fast core for this - exact content. -- **`scripts/check_gate_cadence.sh`** — PreToolUse hook on `git commit`. - Reads the change from **`git diff HEAD` + an untracked listing** (NOT - `git diff --cached`): a PreToolUse hook fires *before* the command, so - `git add … && git commit` batched into one call leaves the index empty at - hook time — `--cached` would see nothing and wrongly exempt the commit. - Non-source change ⇒ exempt. Source change ⇒ authorised iff the current - fingerprint equals `.dev/.gate_pass` **or** `.dev/.smoke_pass` (ADR-0107): - a `.gate_pass` match resets the counter; a `.smoke_pass`-only match - consumes a batch slot and **hard-blocks at the 6th** (forcing a full - gate). A source change with neither fingerprint fresh ⇒ block (smoke it - first). - -`git commit --no-verify` is denied in `.claude/settings.json`, so the -hook cannot be bypassed. `GATE_MAX_BATCH` (default 5) is overridable -via env for a one-off. - -**Clean-tree implication.** Because classification reads the whole working -tree vs HEAD (not just the index), a dirty source file with no fresh -fingerprint blocks **every** commit until it is smoked, reverted, or -committed — even a doc-only commit. This is intentional: it enforces the -edit → smoke → commit rhythm (one unit at a time, clean tree between -units). Stage-and-commit a unit before starting the next; don't leave a -half-done source edit lying beside an unrelated commit. - -## Workflow it produces - -- Coverage / feature sprint: edit → `bash test/run_all.sh --smoke ` - (background it, don't block) → commit. Up to 5 such commits ride without - the full gate. Run `bash test/run_all.sh` (ALONE) on the 5th (or whenever - convenient) — a full green gate clears the batch counter, so the next 5 - commits ride again. (A subsequent commit whose exact content the gate - verified is also authorised as "fresh" and zeroes the counter.) -- Shared-code change: same path — the smoke carries the full both-backend - diff oracle, so it authorises shared-code too (the old "risky ⇒ full - every time" rule is retired per ADR-0107). The full gate still catches - the e2e-shell miss-window (≤ 5 commits) at the ceiling / boundary. - -The `--smoke` run is expected on **every** source commit — the batching -only defers the slow *full* e2e suite, never the smoke. Manual behaviour -probes use a **ReleaseSafe** binary (`zig build -Doptimize=ReleaseSafe --Dcpu=baseline`), not the Debug default (memory `verify_against_releasesafe_binary`). - -## Forward-looking (framework_completion note) - -Per `framework_completion.md`, a new hook must state its discovery -criterion + sweep. This hook is **forward-looking** (like -`orphan_prevention.md`): it constrains *future* commits' cadence and -retrofits no existing code, so there is no existing-site population to -sweep. The "discovery recipe" is the classifier in -`check_gate_cadence.sh` itself, applied per future commit. - -## Stale-ness - -## Conformance cadence (outside the gate) - -`bash scripts/lib_conformance.sh --all` (regenerates -`test/conformance/COVERAGE.md`) and `bash scripts/verify_projects.sh` -run OUTSIDE both gate tiers. Fixed cadence so they cannot silently rot -(2026-07-02): **before any release tag** and **at each boundary review -chain** — plus on demand after a compat-surface campaign lands. - -Stale if: the smoke stops being ≤ ~60s or the full gate's e2e-vs-unit -cost asymmetry changes (re-measure); a new shared-risk surface the smoke -core does NOT cover appears (the miss-window grows beyond CLI/REPL/ -filesystem/http/GC e2e — re-scope what the smoke runs); or the batch of -5 proves wrong in practice. - -## Local and CI must run the SAME configuration - -`scripts/ci_gate.sh` runs the full gate as `test/run_all.sh --serial-e2e`. -`scripts/run_gate.sh` now defaults to the same flag, and -`scripts/check_gate_parity.sh` (gate step `gate_parity`) asserts both still do. - -This was false for months. `run_gate.sh` passed nothing, so `run_all.sh` took -its `PARALLEL_E2E=1` default: "the local full gate is green" and "CI is green" -were statements about two different runs, while `ci_gate.sh`'s own header said -they were the same one. - -It is not cosmetic. The parallel path captures each job's output to a file; the -serial path pipes it. So an e2e whose producer took SIGPIPE from a `head -N` -died silently under serial and passed under parallel — green on every local -full gate, red on every push (2026-08-04). `scripts/check_epipe_head.sh` closes -that specific hazard; `gate_parity` closes the class of "CI runs something -local does not". - -**If you change either launcher's mode, change both, and update `ci_gate.sh`'s -header claim in the same commit.** A parity claim that only a comment makes is -the thing that failed here. - -**No tier runs more than any other any more (2026-08-12).** PR, push, dispatch -and the local `run_gate.sh` all run the identical `test/run_all.sh ---serial-e2e`; the nightly schedule and its `CLJW_CI_PARITY=1` tree_walk sweep -are gone, and `check_gate_parity.sh` fails if either comes back. - -The reason is what the last nightly actually reported: `phase14_nrepl` failed a -**5-second wall-clock bound** for the `.nrepl-port` file, on the tree_walk build, -on a loaded shared runner — a configuration nothing else ran. The same bind -measures **0.08-0.14 s** locally on that same build. So the extra tier's finding -was a fact about the runner, delivered a day late, in a config no one could -reproduce from a normal gate. That is the cost side; the benefit side was -smaller than its name suggests, because the **F-012 dual-backend diff oracle -already runs in `zig build test` on BOTH backends in every gate**. Only the e2e -SHELL suite on the non-default backend went away with it. - -`scripts/check_vm_parity.sh` remains as an **on-demand** tool (and -`run_remote_ubuntu.sh --parity` runs it on Linux — the slowest backend × host -combination, where speed-scaled bounds break first). Reach for it deliberately -after touching backend-divergent code, not on a schedule. - -## Related - -- [ADR-0107](../../.dev/decisions/0107_pipeline_gate_smoke_authorized.md) — - the two-tier decision this rule is the SSOT for; the miss-window analysis. -- `.claude/rules/orphan_prevention.md` — sibling forward-looking hook. -- `scripts/check_smell_audit.sh` — the other `git`-time blocking hook. -- CLAUDE.md § Autonomous Workflow Step 5 (test gate) / Step 6 (commit). -- memory `smoke_first_batch_full_gate` + `verify_against_releasesafe_binary`. diff --git a/.claude/rules/handover_framing.md b/.claude/rules/handover_framing.md deleted file mode 100644 index 44429d195..000000000 --- a/.claude/rules/handover_framing.md +++ /dev/null @@ -1,297 +0,0 @@ ---- -paths: - - ".dev/handover.md" ---- - -# Handover framing discipline - -Auto-loaded when editing `.dev/handover.md`. Codifies the cw v1 -2026-05-23 retrospective + the matching zwasm v2 rule -(`~/Documents/MyProducts/zwasm/.claude/rules/handover_framing.md`). - -Two failure modes have repeatedly bloated cw's handover: - -1. **Log accumulation** — successive "Just landed — §X" sections - pile up across sessions; the file becomes a session diary - redundant with `git log`. -2. **Surrender framing** — phrases like - "コンテキスト圧があるため一旦 idle に入る" / - "キリがいい" / "自然な区切り" turn the resume entry point - into a stop signal that CLAUDE.md § The only stop does not - authorise. The only stop is the user's explicit request, and - that record lives in `## Stopped — user requested` with the - user's verbatim quote — not as a euphemism in the body. - -## The rule - -`.dev/handover.md` is a **driving document**, not a -**deliberation document** or **session log**. Every entry must -either: - -1. Describe the concrete **next task** the resume should execute - (§9.. identifier + retrievable file/section pointers), OR -2. Name a **provable external blocker** with a testable barrier - condition (already a `debt.yaml` row by default — handover only - names the row by ID). - -Anything else is forbidden framing. The loop reads handover to -decide **what to do next**, not whether to do anything. - -## Resume contract (mandatory top section, 3-5 lines) - -Every handover.md MUST start with a tight contract block before -any other section (the cold-start reading list can follow it -but the contract comes first): - -```markdown -## Resume contract - -- HEAD: -- First commit on resume MUST be: . - identifier, ADR slug, or file path> -- Forbidden this session: -``` - -This is the first thing the next session reads. Vague / -multi-option phrasing here is a block-level finding. - -- "First commit MUST be" wording is required, not "Recommended". - Soft phrasing ("consider", "next could be") allows the - resuming loop to substitute its own first task — that - defeats the purpose. -- "Forbidden this session" exists to record the previous - session's surfaced-but-rejected directions ("expand primitive - cluster" when the real next task is regex impl) so the next - session does not re-derive the same pivot. Leave empty if - there is nothing to forbid. - -## Hard length limit: ≤ 100 lines - -Above 100 lines, the framing has drifted into log / deliberation / -forecast. Trim before commit. `git log` and `.dev/ROADMAP.md` -already carry the history and the forecast. - -### The cap is advisory in hook mode (2026-06-15 — the trim-Edit deadlock fix) - -The 100-line cap is a **soft quality bound**, so the PreToolUse:Edit -hook (`scripts/check_handover_framing.sh`) treats it as **advisory**: -when `.dev/handover.md` is over cap, hook mode **warns but does NOT -block** the edit. This lets a multi-step trim chain — or any edit that -*reduces* the file toward the cap — land freely via the Edit/Write -tool. (The earlier git-HEAD-based "trim-Edit exemption" did not cover -the common case: when the over-cap state was introduced by the *same -session's* uncommitted edits, HEAD was still ≤ cap so no exemption was -granted, and the only escape was a Bash rewrite bypassing the hook — -the recurring deadlock this fix removes.) - -The cap is still **enforced at the audit boundary**: `--check` mode -(`audit_scaffolding` A5b) **fails** any over-cap snapshot. So the -≤ 100 discipline holds — the gate just moves from per-edit to audit. - -The **forbidden-phrase / section checks STAY blocking** (exit 2) in -both modes — those (surrender framing etc.) are the load-bearing gate; -only the line cap softened. (Mirrors the `md-table-align` hook → -advisory, 2026-06-11, for the same mid-edit-drift reason.) - -## Update frequency cap (≤ 2 / session) - -Per session: at most 2 handover updates, except at Phase -boundary where the open-§9. commit owns one update -(that one does not count against the 2-cap). - -The cap exists because the most common failure mode is -**HEAD-pointer churn**: the resuming loop wants to keep -handover's `HEAD: ` line synchronised with the latest -push, and updates after almost every source commit. That is -log accumulation, just disguised as a single-line edit. - -- **HEAD pointer is allowed to go stale.** `git log` is the - SSOT for current HEAD. handover's HEAD field is a coarse - pointer — "`HEAD ≈ X` (5 commits behind git)" is fine. -- handover only refreshes when the **Active task identifier - itself changes** (e.g. §9.6 closed → §9.7 opens, or ADR - status flipped, or a new Forbidden item surfaced). -- Cosmetic re-wording / count updates / "now there are 48 - primitives not 33" edits are churn — they belong in `git - log` of the source files, not in handover. - -If a session hits the 2-update cap and still wants to -record progress, write to `private/notes/.md` -(gitignored) instead. - -## Forbidden patterns (grep-enforceable) - -The following appearances anywhere in `.dev/handover.md` are -**block-level findings** that resume Step 1 (`/continue`) must -repair before proceeding. - -### Phrase-level (surrender / stop-rationalisation framing) - -| Phrase | Why forbidden | Replace with | -|-----------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------| -| `コンテキスト圧があるため` (and variants) | Auto-compaction is system-handled; not an agent concern | Just describe the next task | -| `キリがいい` / `自然な区切り` / `natural break` | Not a stop condition. The only stop is user explicit request | Drop entirely | -| `good stopping point` / `この辺で一旦停止` / `region boundary stop` / `task boundary stop` | Same — region / cluster / task / commit boundaries all roll into the next unit of work | Drop entirely | -| `Phase boundary reached AND ...` | Phase boundary is not a stop. The review chain runs and the loop continues into §9. | Drop the "AND" clause | -| `If above ~60%` / `/compact` / `context budget` | Compact-gate concept removed; `/compact` is not Skill-tool callable | Drop; auto-compaction is transparent | -| `〜判断待ち` / `user 確認待ち` / `awaiting user confirmation` / `awaiting approval` | "Awaiting user" is not a stop state. Push is automatic; ADR-level designs are handled inline | Drop; if a real external block, file a `debt.yaml` row with `Status: blocked-by: ` and name it | -| `cannot be self-decided` / `human judgement` / `human judgment` / `needs human` / `user touchpoint` | The loop self-decides design choices; "needs human" reframes a self-decidable case as a stop | Drop entirely; if the design is genuinely ADR-level, draft + accept the ADR inline per CLAUDE.md and link the ADR slug | -| `this needs human judgement` / `help wanted` / `awaiting human review` / `defer to user` | Same family — invites a pause for human input that is not a stop condition | Drop entirely | -| `ADR-level decision` / `ADR-phase mode` (as a stop reason) | ADR-level designs are inline work. Smell-driven ADR drafting is an interrupt, not a stop | Rewrite as an Active task entry naming the candidate ADR slug + the F-NNN-compliant shape the AI is taking | -| `smell-cluster trip` / `smell cluster` / `patterned smell` / `goal drift trip` | Smell triggers are interrupts, not stops. No frequency-counter rule exists | Drop entirely; the surgery already landed inline at the interrupt | -| `physically blocked` / `physical block` (when build / test failures are involved) | Build / test failure is not a stop condition either. Diagnose and fix; the loop owns getting back to green | Drop the "block" framing; describe the failure as the current Active task with the diagnosis attempted | -| `stopped — physical block` / `Stopped — physical block` | No such stop condition exists | Rewrite into Active task naming the failing test / build artifact and what to try next | -| `physical block / blocker` as a *non-debt-row* reason in handover | All real external blocks live in `debt.yaml` with a `blocked-by: ` Status; handover names the D-NNN by ID only | Move to `debt.yaml`, reference here as `D-NNN` | - -### Structural (log / forecast / reproduced-content accumulation) - -| Pattern | Why forbidden | Replace with | -|------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------| -| More than **one** `## Just landed` section | Log accumulation; `git log` is the SSOT | Keep at most the most-recent landing as a single ≤ 10-line section, or drop entirely once next task starts | -| `## Future ... shopping list` / forecast tables | Forecast belongs in `.dev/debt.yaml` (recall trigger row) or ROADMAP §A / ADR governance | Move to debt.yaml as a `D-NNN` row with `Status: recall trigger` and remove from handover | -| `## Notes for the next session` reproducing rule / skill content | CLAUDE.md / `.claude/rules/` / skill `SKILL.md` are the SSOT — auto-loaded each session | Drop; the next session re-reads the source SSOT | -| Numeric predictions (`~N tasks remain`, cycle counts) | Predictions diverge from reality as time passes; the next session resumes against the stale guess instead of measuring | Concrete identifiers (`next: §9.6 / 4.4`); no counts. Live measurement scripts (`git log`, debt.yaml, audit_scaffolding) produce truth. | -| Multi-paragraph editorial framing of work size | "deep work" / "substantial multi-cycle" / "重い" framings are stop rationalisations | Just name the task and the entry ADR | - -## What handover IS for - -- **Cold-start reading order** (≤ 3 files; the first is handover - itself) so a fresh session reaches the active task in < 30 sec. -- **Current state** — phase number, branch, last commit, gate - status (1-line summary; not a changelog). -- **Active task** — §9.. identifier + 1-2 sentence - description + retrievable identifiers (ADR refs, file paths, - test name). -- **Next unit pointer** — a concrete "First task on resume" naming the - next gap-area unit / debt row. (The phase-queue / §9.-placeholder - model is retired per ADR-0142; the loop self-selects from live debt.) -- **Open questions / blockers** — testable external dependencies - only; otherwise the row belongs in `.dev/debt.yaml`. - -## What handover IS NOT for - -- Listing past landings (`git log --oneline` is the SSOT). -- Listing future ADRs / debts / refactors (debt.yaml and ROADMAP - §A are the SSOTs). -- Re-explaining rules / skills / output styles (the SSOT files are - auto-loaded each session). -- Multi-option pickup menus — pick the one option in handover - itself. -- "Cycle 1 did X, cycle 2 did Y" running log (per-task notes in - `private/notes/` are the SSOT and are gitignored). - -## Reviewer checklist - -When reviewing a handover.md commit: - -- [ ] **Top section is `## Resume contract`** with 3 fields: - HEAD / First commit MUST be / Forbidden this session. - "MUST be" wording present (not "Recommended"). -- [ ] ≤ 100 lines total. -- [ ] No phrase from the forbidden-phrase table. -- [ ] At most **one** `## Just landed` section. -- [ ] No forecast / shopping-list table. -- [ ] `Active task` names a concrete next §9.. task, not - an option list. -- [ ] `Open questions / blockers` rows are either testable - external blockers OR already mirrored as `.dev/debt.yaml` - D-NNN rows (named here by ID). -- [ ] No numeric predictions — concrete identifiers only. -- [ ] If this is the 3rd+ handover update in the session, - the change must be Active-task-identifier-level - (Phase boundary / ADR status flip / new Forbidden); - HEAD-pointer-only churn is rejected. - -## How enforcement runs - -Wave 16 (W16-3) lifts the resume-time grep into a PreToolUse:Edit -hook so the gate fires at the moment the edit is written, not at the -next resume's Step 1 scan. - -- **Live gate** — `scripts/check_handover_framing.sh` wired into - `.claude/settings.json` PreToolUse Edit|Write chain. Every edit - to `.dev/handover.md` runs the script; a forbidden phrase / - `## Future ... shopping list` / `## Notes for the next session` - heading / >1 `## Just landed` sections **blocks** the edit (exit 2) - with the offending lines printed. The **over-cap length is ADVISORY - in hook mode** (warns, does not block — the trim-Edit deadlock fix, - 2026-06-15; see § "The cap is advisory in hook mode") and **strict - in `--check`** (audit) mode. -- **Audit-time fallback** — `audit_scaffolding/CHECKS.md` A5b runs - the same script in `--check` mode at Phase boundary, catching - drift that may have slipped past the live gate (e.g. if a future - euphemism is not yet in the regex). -- **Resume-time backstop** — `/continue` Step 1 reads handover; if - the file violates the rule, the FIRST task on resume is the - handover rewrite, not the prose-suggested next task. - -The canonical forbidden-phrase regex lives in -`scripts/check_handover_framing.sh::FORBIDDEN_PHRASES_RE`. Update -the rule's forbidden-phrase table and the script's regex together -when a new euphemism is identified. - -## Legitimate stop framing - -There is only one legitimate stop and only one section shape for -recording it. Per CLAUDE.md § The only stop: - -> The user explicitly asks the loop to stop. - -Record the directive verbatim (or close paraphrase) under -`## Stopped — user requested` so the next session reads back -exactly what was asked. The directive applies to the session in -which it was issued; the next `/continue` (or any automated -re-invocation) resumes the loop normally: - -```markdown -## Stopped — user requested - -User instruction (2026-MM-DD): "". -Resume at §9... -``` - -No other `## Stopped — …` section is legal. In particular: - -- Build / test failure is not a stop. The loop owns getting back - to green; describe the failure under `Active task` with the - diagnosis attempted, not under a stop heading. -- ADR-level design choices are not a stop. They are handled - inline per CLAUDE.md; the AI drafts and accepts the ADR itself. - Handover entries that frame an ADR-level choice as a stop are a - block-level finding — rewrite as an Active task entry that - names the design choice and the candidate ADR slug. -- Smell-driven interruption is not a stop. The surgery already - landed at the interrupt; the handover does not record it as a - stop event. - -## Resume must not propagate a stale stop - -When a fresh session reads handover and sees an existing -`## Stopped — user requested` section, that section is **history -of the previous session's end**, not a directive for the current -session. Step 1 of `/continue` deletes the section as part of the -resume rewrite (the user's directive applied to the previous -session only — see CLAUDE.md § The only stop). The current -session resumes the loop unchanged. - -## Stale-ness - -This rule is stale if: - -- The forbidden-phrase table no longer matches actual handover - drift. Re-derive from - `git log -p .dev/handover.md --since="90 days ago"` and surface - new euphemisms. -- CLAUDE.md § The only stop changes; the "Legitimate stop - framing" section above must mirror the canonical wording. - -## Related - -- CLAUDE.md § The only stop — the single-condition stop - this rule references. -- `scripts/check_handover_framing.sh` — the PreToolUse:Edit hook - that enforces the forbidden-phrase table + length cap (Wave 16 - W16-3); canonical regex lives at the script's - `FORBIDDEN_PHRASES_RE`. -- `~/Documents/MyProducts/zwasm/.claude/rules/handover_framing.md` - — the v2 source rule cw adapted from. diff --git a/.claude/rules/markdown_format.md b/.claude/rules/markdown_format.md deleted file mode 100644 index 39aba33bb..000000000 --- a/.claude/rules/markdown_format.md +++ /dev/null @@ -1,61 +0,0 @@ ---- -paths: - - "**/*.md" ---- - -# Markdown formatting rules - -Auto-loaded when editing any `.md` file. Two concerns: pipe-table -alignment, and a small set of conventions for Markdown bodies. - -## Pipe tables — don't align by hand, run `md-table-align` - -Manual width calculation is the wrong tool. We have a CLI that mirrors -Emacs `markdown-mode`'s `markdown-table-align` exactly. After you finish -editing any pipe table, run: - -```sh -md-table-align # rewrite in place -md-table-align --check # verify (pre-commit hook does this) -``` - -A `PreToolUse` hook on `git commit` (`scripts/check_md_tables.sh`) checks -staged `.md` tables. **Since 2026-06-11 (user-directed) it is advisory only:** -it prints a `⚠ non-blocking` recommendation if a staged `.md` has unaligned -tables but does **not** auto-format, re-stage, or block — the earlier silent -in-place reformat drifted the gate fingerprint mid-commit. Run -`md-table-align ` yourself when you want tables tidied. - -If the binary is missing, the hook simply skips (no block). Short form: - -```sh -bbin install io.github.chaploud/babashka-utilities -``` - -## What the formatter enforces (informational) - -The same rules `~/.claude/CLAUDE.md` describes — `md-table-align` is the -implementation: - -- East Asian Wide / Fullwidth characters count as width 2. This covers - kanji / hiragana / katakana (incl. the long-vowel mark `ー`), Hangul, - and fullwidth ASCII / signs (`（）「」：`, `！？` etc.). -- Halfwidth ASCII counts as width 1. -- Each column's width is set to the longest cell in that column, plus - one space of padding on each side. -- Alignment markers `:---`, `---:`, `:---:` in the delimiter row are - honored. - -Knowing the rule helps you sanity-check the formatter's output, but -your job is **never** to compute widths in your head. - -## Language policy + Japanese narrative conventions - -See [`.claude/output_styles/japanese.md`](../output_styles/japanese.md) -for the canonical rule. Summary: - -- English by default for all Markdown — `README.md`, `.dev/`, - `.claude/`, ADRs, `docs/` outside `docs/ja/`. -- Japanese for chat replies, `private/notes/.md`, and - `docs/ja/` body text (code blocks keep English identifiers). -- Do not mix: identifiers in code blocks are the only exception. diff --git a/.claude/rules/module_docstring.md b/.claude/rules/module_docstring.md deleted file mode 100644 index ec5357ae5..000000000 --- a/.claude/rules/module_docstring.md +++ /dev/null @@ -1,69 +0,0 @@ ---- -paths: - - src/**/*.zig ---- - -# Module docstring discipline - -## Rule - -Every new `.zig` source file under `src/` opens with two-line -boilerplate: - -```zig -// SPDX-License-Identifier: EPL-2.0 -//! One-line summary of what this module is for. -//! ...optional further `//!` lines describing the public contract. -``` - -- `// SPDX-License-Identifier: EPL-2.0` — the comment-style license - header (matches the project's EPL-2.0 LICENSE). -- `//! ...` — a Zig module docstring. The first line is the - one-sentence "what this module is". Add further `//!` lines for - semantic contract (lifetime, ownership, error set, threading). - -## Why - -- A new contributor reaching a file knows the licensing context and - the module's purpose without reading the body. -- `//!` is the Zig module-level docstring; tools (`zig std`, ZLS hover, - generated docs) surface it. -- The two-line opener is the convention zwasm v1 has used at scale - (~30K LOC of Zig) and matches the cw v1 corpus practice on files - added since Phase 3. - -## How to apply - -- Every new file under `src/`: open with the two lines. -- Existing files that lack the header: not a retroactive sweep at - Phase 4 entry, but add the header opportunistically when touching - the file for any other reason. -- Test files (`test "..."` blocks inside source) inherit the - header from the surrounding module — no separate header. - -## Counter-example - -Don't open a new module with imports alone: - -```zig -// BAD: no SPDX, no //! docstring -const std = @import("std"); - -pub fn foo(...) ... { ... } -``` - -Don't substitute `//` for `//!` on the module-level docstring: - -```zig -// SPDX-License-Identifier: EPL-2.0 -// One-line summary. ← BAD: should be //! for Zig to pick up as -// module docstring -const std = @import("std"); -``` - -## Enforcement - -- Reviewer check at PR time. -- `scripts/check_module_docstring.sh` (Phase 5+ informational → gate) - can grep new `src/**/*.zig` files for the two-line opener; not - implemented at Phase 4 entry to avoid pre-mature gating. diff --git a/.claude/rules/no_copy_from_v1.md b/.claude/rules/no_copy_from_v1.md deleted file mode 100644 index e7208c30f..000000000 --- a/.claude/rules/no_copy_from_v1.md +++ /dev/null @@ -1,45 +0,0 @@ ---- -paths: - - src/**/*.zig - - bench/fixtures/** - - test/** ---- - -# No verbatim copy from cw v0 - -## Rule - -Do not copy cw v0 (git tag `v0.5.0`) code verbatim. -Read v0 as a read-only reference and re-derive semantics in Zig 0.16 -idiom. - -## Why - -- cw v0 carries 89K LOC of accumulated technical debt - (collections.zig at 6,269 lines, std.Io F140-F144 disabled features). -- v0 implementation patterns assume different tier classifications - (cw v1 uses Option β; v0 used Tier D for the class system). -- cw v1 is a re-design, not a port; verbatim copy preserves the original - design pressure. - -## How to apply - -At each Phase Step 0 (textbook_survey): - -1. Read v0 source for context (read-only). -2. Re-implement the same semantics in cw v1 Zig 0.16 idiom. -3. In per-task notes, record both "what v0 did" and "what cw v1 does", - with rationale for differences. - -When the v0 and cw v1 choices match, the rationale must explain "why the -same choice", not assume it. - -When they differ, an ADR justifies the divergence. - -## Counter-example - -Avoid: `cp ../cw-v0/src/lang/builtins/collections.zig src/runtime/collection/vector.zig` (where `../cw-v0` is a `git worktree add ../cw-v0 v0.5.0`) - -Do instead: Read v0 collections.zig persistent vector section, understand -HAMT shift=5 + tail array design (per ADR-0007 TypeDescriptor and -ADR-0012 ValueTag), re-implement in cw v1. diff --git a/.claude/rules/no_jvm_specific_assumption.md b/.claude/rules/no_jvm_specific_assumption.md deleted file mode 100644 index 0e42c085d..000000000 --- a/.claude/rules/no_jvm_specific_assumption.md +++ /dev/null @@ -1,36 +0,0 @@ ---- -paths: - - src/**/*.zig - - .dev/**/*.md ---- - -# No JVM-specific assumption - -## Rule - -cw v1 is not a JVM reimplementation. Comments and docs do not assume: - -- Class hierarchy (cw has TypeDescriptor, not Class). -- Object header layout matching JVM (cw has its own packed header per - ADR-0009). -- Bytecode generation (cw has Opcode enum, not JVM .class files). -- Thread monitor as built-in Object property (cw has heap-value-only - lock per ADR-0009). - -When comparing semantics, use phrases like "cw v1 implementation of -", not "JVM-style ". - -## Why - -- cw v1 design diverges from JVM internals while preserving Clojure - observable semantics. -- JVM-style comments mislead later readers into Java assumptions. -- ADRs anchor each divergence with a rationale. - -## How to apply - -- Write semantics references against Clojure (clojure.core, - clojure.lang) rather than JVM bytecode. -- For cw-specific implementation choices, reference the ADR. -- For JVM contrast, label clearly: "JVM Clojure does X here; cw v1 does - Y per ADR-NNNN". diff --git a/.claude/rules/no_op_stub_forbidden.md b/.claude/rules/no_op_stub_forbidden.md deleted file mode 100644 index 5cfcc775b..000000000 --- a/.claude/rules/no_op_stub_forbidden.md +++ /dev/null @@ -1,42 +0,0 @@ ---- -paths: - - src/**/*.zig - - src/lang/clj/** - - src/runtime/host/** ---- - -# Permanent no-op forbidden — transient stubs OK - -> Folded into [`provisional_marker.md`](provisional_marker.md) § -> "Skeleton vs transient stub vs PROVISIONAL vs permanent no-op" -> at Wave 16 W16-6 (2026-05-26). This file remains as a path- -> frontmatter pointer so editors of `src/**/*.zig` / -> `src/lang/clj/**` keep auto-loading the time-axis distinction -> ("final shape" vs "development trajectory") that the original -> 2026-05-23 rename surfaced. - -## Canonical rule (one sentence) - -> **No Tier A / B / C feature may ship in a form where the user -> sees success while the runtime silently drops the intended -> semantics.** - -The 4-row boundary table (Skeleton OK / Transient stub OK / -PROVISIONAL-with-triad OK / Permanent no-op NG) lives in -[`provisional_marker.md`](provisional_marker.md) § "Skeleton vs -transient stub vs PROVISIONAL vs permanent no-op". See also: - -- `scripts/check_no_op_stub.sh` — heuristic scan (informational - per W16-2 cleanup). -- ADR-0004 / ADR-0012 / ADR-0023 — skeleton-then-rewrite - endorsement. -- ADR-0018 amendment 2 — Tier D / sub-feature staged `feature_not_supported` - template contract. - -## Revision history - -- **2026-05-26 (Wave 16 W16-6)**: body folded into - `provisional_marker.md`; this file becomes a 30-line pointer. -- 2026-05-23: original rename "No-op stub forbidden" → "Permanent - no-op forbidden — transient stubs OK" (made the time-axis - explicit). diff --git a/.claude/rules/orphan_prevention.md b/.claude/rules/orphan_prevention.md deleted file mode 100644 index 562f19655..000000000 --- a/.claude/rules/orphan_prevention.md +++ /dev/null @@ -1,110 +0,0 @@ ---- -paths: - - "**" ---- - -# Orphan-prevention discipline - -A background long-runner that outlives a bash turn gets re-parented to -PID 1 on session interrupt and can spin a core (Mac fan event). Two -incident narratives (2026-05-28 REPL-pipe poll-spin, 2026-05-31 -clj-oracle infinite-seq) live in -[`ADR-0049`](../../.dev/decisions/0049_orbstack_linux_gate_retired.md) -§ Context. - -## The rules - -1. **Every `Bash(run_in_background: true)` that drives a long-running - child** — REPL pipe, SSH shell, bench loop, gate, daemon — - **MUST be wrapped in `timeout 600 …`** (larger only when justified, - e.g. `timeout 1800` for a slow Linux gate). Never omit the wrap. -2. **Every `clj -M -e '…'` oracle probe MUST be `timeout 20`-wrapped - AND bound sequence-producers with `(take N …)`.** `clojure.main -e` - prints its result, so an unbounded seq (`(range)`, `(iterate inc 0)`, - `(repeat 1)`, `(cycle …)`, `(line-seq …)`) realises forever and - pins ~160 % CPU. **Watch the NON-OBVIOUS infinite producers too**: - `(take-nth 0 coll)` / `(take-nth -n coll)` returns clj's infinite - `(first coll)` repeat; `(partition n step coll)` with `step` ≤ 0; - `(repeatedly f)` 0-arity. -3. **Every batched / scripted `clj` invocation MUST cap the JVM heap with - `-J-Xmx2g`** (e.g. `timeout 60 clj -J-Xmx2g -M batch.clj`). The 2026-06-12 - incident: an unbounded `(take-nth 0 …)` slipped into a `clj_diff_sweep` - batch and an UNCAPPED JVM realised the infinite seq for the full `timeout 60` - window, exhausting ~138 GB of system memory + swap (Mac froze). `timeout` - alone does NOT bound memory — only CPU-time. `-J-Xmx2g` makes a runaway hit a - JVM `OutOfMemoryError` in seconds (the process dies) instead of consuming all - system RAM. 2g is ample for any legit batch. **`ulimit -v` does NOT work for - `cljw`** (the native binary reserves a large virtual address space and fails - to start under a `-v` cap) — bound cljw via `timeout` + the - `(take N …)`-every-producer rule instead, never `ulimit -v`. - -`timeout` kills only its immediate child, NOT descendants in another -process group: across an SSH/VM boundary the remote command keeps -running (use `ssh host 'timeout 600 cmd'` or -`-o ServerAliveInterval=30`); a `setsid`/`nohup` grandchild survives -too. Prefer one-shot `cljw -e '…'` / heredoc (per -[`cljw_invocation.md`](cljw_invocation.md)) over long-lived REPL pipes. - -4. **`kill $!` does NOT reap `( cd DIR && cmd ) &` — kill by a scoped - pattern.** With two commands in the subshell, bash cannot - exec-optimise it away, so `$!` is the SUBSHELL and the server is its - grandchild (through `timeout`, if wrapped). Killing `$!` reaps the - shell and **orphans the runtime**. Every nREPL e2e in this repo had - this shape and leaked one server per `start_server` — measured - 2026-08-12: `phase14_nrepl_classpath` left 3 live `cljw nrepl` - processes behind, `phase14_nrepl_toplevel_do` 4. - - It does not present as a leak. Each survivor holds its memory for the - rest of the run, and macOS eventually SIGKILLs *something else* — the - symptom was `Killed: 9` on a freshly built binary in an unrelated - entry point, i.e. a resource leak wearing the costume of a runtime - crash. (The CI runner's "Terminate orphan process: pid (…) (cljw)" - flood at job cleanup was the same leak, reported by the runner - instead of by us.) - - Reap by something that identifies the process itself, and keep the - pattern scoped so it cannot hit an unrelated run: - - ```bash - start_server() { ( cd "$WORK" && "$BIN" nrepl --port "$port" ) & SERVER_PID=$!; SERVER_PORT="$port"; } - stop_server() { pkill -f "nrepl --port ${SERVER_PORT}( |$)" 2>/dev/null || true - kill "$SERVER_PID" 2>/dev/null || true; wait "$SERVER_PID" 2>/dev/null || true; } - ``` - - A test that starts servers should end with none: `pgrep -f - 'zig-out/bin/cljw' | wc -l` is the check, and it belongs in the - loop's head after touching any server-spawning e2e. - -## Gate launcher - -`bash scripts/run_gate.sh` (preferred over raw `test/run_all.sh`) reaps -any prior `run_all.sh` tree + PID-1-orphaned `cljw` before launching a -single gate; `bash scripts/run_gate.sh reap` reaps without launching. -`bash scripts/cleanup_orphans.sh` (etime > 30 min, wired as a SessionStart -hook in `.claude/settings.json`) is the cross-session backstop: it reaps -PID-1-orphaned `run_all.sh` / `run_gate.sh` trees, this repo's `cljw`, and -stray `clojure.main -e` oracle JVMs. `--dry-run` lists without killing. - -This lived at `~/.claude/hooks/cleanup_orphans.sh` until 2026-08-04 — outside -the repo, so no clone ever had it, and it was absent on the maintainer machine -too. It is in-repo now so the guarantee this rule's disciplines assume is one -that actually runs (D-565). - -## Counter-examples - -- ❌ `ssh ubuntunote 'bash test/run_all.sh'` in background, no - `timeout` — orphaned remote shell on session kill. - ✅ `timeout 600 bash scripts/run_remote_ubuntu.sh`. -- ❌ `clj -M -e '(range)'` — unbounded print, no timeout. - ✅ `timeout 20 clj -M -e '(take 5 (range))'`. -- Reap a stray oracle JVM: `pkill -f 'clojure.main.*-e'` (scoped — - leaves IntelliJ / Gradle / interactive REPLs untouched). - -## Related - -- [`ADR-0049`](../../.dev/decisions/0049_orbstack_linux_gate_retired.md) - § Context — full incident narratives. -- `~/.claude/CLAUDE.md` § 全プロジェクト共通 — global orphan advisory + - the SessionStart cleanup hook. -- [`cljw_invocation.md`](cljw_invocation.md) — safe `cljw` entry points - that avoid long-lived REPL pipes. diff --git a/.claude/rules/perf_marker.md b/.claude/rules/perf_marker.md deleted file mode 100644 index f86ff5444..000000000 --- a/.claude/rules/perf_marker.md +++ /dev/null @@ -1,85 +0,0 @@ ---- -paths: - - "src/**/*.zig" - - "src/**/*.clj" ---- - -# PERF marker comments - -Auto-loaded when editing source. Codifies the **discoverability** layer -for performance optimizations — code shaped for speed rather than for -the simplest correct form. Sibling to -[`provisional_marker.md`](provisional_marker.md); the SSOT is -[`.dev/optimizations.md`](../../.dev/optimizations.md). - -## Why this rule exists - -The user's directive (2026-05-31): *"将来の最適化のとき、「最適化して -るんだよ」と分かりやすく — 理想は SSOT 的な箇所があること"*. Optimizations -hide intent: a fast-path `switch` arm or a precomputed field reads like -ordinary code, so a later editor cannot tell "this complexity is here -for speed; the naive form over there is the real contract." Without a -marker, an optimization rots into unexplained complexity, and a bug in -the fast path silently diverges from the naive path (the kind of -divergence F-011's behavioural-equivalence target forbids). - -The marker makes every optimization **grep-discoverable** -(`rg 'PERF:' src/`) and the `.dev/optimizations.md` ledger records the -naive↔optimized pair so the equivalence is auditable. - -## Canonical form - -```zig -// PERF: [refs: O-NNN, D-NNN?] -``` - -```clojure -;; PERF: [refs: O-NNN, D-NNN?] -``` - -Rules: - -- **Single line**, placed directly above the optimized code. -- **`refs:` lists at least one `O-NNN`** (the `.dev/optimizations.md` - ledger row). Cross-ref the driving `D-NNN` perf-debt row when one - exists. -- The ledger row carries the detail (naive form / optimized form / why / - verification); the marker is just the in-code anchor. -- **Removed on revert**, not commented-out (grep-discoverability is the - point). Flip the ledger row to `RETIRED` in the same commit. - -## When it applies - -A `PERF:` marker is for code where a **simpler correct form exists** and -was traded for speed: - -- A fast-path `switch` arm that special-cases a type for O(1) instead of - the generic O(n) walk (e.g. `count` of a `.range`). -- A precomputed / cached field that duplicates derivable state. -- A tight-loop reimplementation that bypasses the general dispatch path. -- A representation chosen for iteration speed over the obvious one. - -It does **not** apply to code that is simply the natural implementation -(no simpler-but-slower alternative was passed over). Over-marking is -noise; mark only genuine speed-for-simplicity trades. - -## Distinguishing PERF from PROVISIONAL - -- **`PERF:`** — permanent (or long-lived) optimization; the naive form - is the *fallback contract*, the marked code is the *fast path*. Lives - in `.dev/optimizations.md`. -- **`PROVISIONAL:`** — temporary intermediate behaviour pending an - upstream feature; closes out when that feature lands. Lives in - `data/feature_deps.yaml` + `.dev/debt.yaml`. - -A single site can be both (a provisional optimization) — carry both -markers, each with its own refs. - -## Enforcement - -Lightweight by design (the user noted optimizations are too varied for -full mechanical management). No hard gate today. `audit_scaffolding` -may, at a Phase boundary, cross-check `rg 'PERF:' src/` against -`.dev/optimizations.md` rows for orphans in either direction. The -primary value is the grep-discoverable marker + the SSOT ledger, not a -blocking check. diff --git a/.claude/rules/perf_measure_release.md b/.claude/rules/perf_measure_release.md deleted file mode 100644 index 3ef875b98..000000000 --- a/.claude/rules/perf_measure_release.md +++ /dev/null @@ -1,88 +0,0 @@ ---- -paths: - - "bench/**" - - ".dev/optimizations.md" - - "scripts/perf.sh" - - "scripts/run_gate.sh" - - "test/run_all.sh" ---- - -# Measure perf in Release, never Debug - -Auto-loaded when touching perf surfaces (`bench/`, `.dev/optimizations.md`, -`scripts/perf.sh`, `// PERF:`-marked code). The principled guard against -the 2026-05-31 incident where a whole perf "campaign" chased **Debug-build -ghosts**. - -## The rule (mechanical) - -**Measure cljw runtime speed ONLY through `scripts/perf.sh`** (or the -ReleaseFast `bench/`). It builds an optimised binary into a separate -prefix and times that. **NEVER `time zig-out/bin/cljw`** — that binary is -**Debug**. - -```sh -bash scripts/perf.sh '(count (vec (range 1000000)))' # ReleaseFast, 3 runs -CLJW_PERF_MODE=ReleaseSafe bash scripts/perf.sh '...' # match cw-v0's mode -``` - -## Why (the incident) - -`build.zig` uses `standardOptimizeOption(.{})`, so **`zig build` (no -`-Doptimize`) defaults to Debug** — deliberately, for fast dev/TDD -iteration. A Debug build of a tree-walk interpreter runs **~10-100× slower** -than the shipped build, so Debug perf numbers are meaningless. Measured -2026-05-31 (same code, same expr): - -| expr | Debug (`zig build`) | ReleaseFast (`scripts/perf.sh`) | -|---------------------------------|---------------------|---------------------------------| -| `(count (vec (range 1e6)))` | ~121s (pre-O-003) | ~0.02s | -| `(count (map inc (range 1e5)))` | ~41s (pre-O-004) | ~0.01s | -| startup `cljw -e 1` | ~0.48s | ~ms (cw v0 claims ~4ms) | - -The "あからさまに遅い" pathologies that motivated O-001..O-004 were -**Debug artifacts**; cljw already meets the ms-level cold-start mission -target in Release. The algorithmic wins (O(n) over O(n log n), chunked -iteration) still help in Release, but the urgency/framing was Debug-driven. -Any future perf claim MUST cite a `scripts/perf.sh` (Release) number. - -## Build-mode policy (structural unification) - -| Path | Build mode | Why | -|--------------------------------------------|-----------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| Shipped binary / `cljw build` | ReleaseSafe | optimised + all safety checks | -| Gate e2e (`build_cljw`) | ReleaseSafe | `run_all.sh` exports `CLJW_OPT=ReleaseSafe` | -| `phase4_*` backend e2e | ReleaseSafe | unified 2026-05-31 (was `:-Debug` standalone default) | -| **Gate unit tests (`zig build test`)** | **ReleaseSafe** | **D-487 flip (2026-06-21): the F-012 diff oracle RUNs many programs on the interpreter → RUN dominates; Debug hits the perf cliff. ReleaseSafe keeps ALL safety checks, only drops 0xAA poisoning (oracle doesn't depend on it). Gate steps pass `-Doptimize=ReleaseSafe`.** | -| Dev `zig build test` (bare, no -Doptimize) | Debug | max diagnostics (0xAA undefined-poisoning) for deep debugging — dev opt-in, NOT the gate | -| Dev `zig build` | Debug | fast TDD iteration; NEVER time this binary | -| Perf measurement | ReleaseFast | `scripts/perf.sh` only | - -### D-487 — why the gate's unit tests flipped Debug → ReleaseSafe (2026-06-21, user-flagged) - -The user flagged the smoke being slow. Root cause: `zig build test` ran **Debug**, and -because `src/lang/diff_test.zig` (the F-012 dual-backend diff oracle) is `@import`ed into -the test set, the test **RUN** evaluates many programs on the Debug interpreter — the -Debug perf cliff (e.g. `(count (vec (range 1e6)))` = Debug 121s vs Release 0.02s). **Measured -2026-06-21**: a test build = Debug ~200s vs **ReleaseSafe ~59s cold / ~1.0s warm** — 3.4× -cold, ~200× warm — with **all tests still PASS** (ReleaseSafe keeps bounds/overflow/UB -panics; only Debug's `undefined`-poisoning is gone, which the value-equality oracle does -not rely on). Alternatives considered: **(a) keep Debug** (status quo — rejected: 3.4–200× -slower for zero gate benefit, the urgency the user raised); **(b) ReleaseSafe** (chosen — -strictly faster, safety preserved); **(c) a build-flag toggle** (rejected as over-built — -bare `zig build test -Dwasm` ALREADY stays Debug for a dev who wants undefined-poisoning, so -no new knob is needed; the gate just hard-codes ReleaseSafe). The earlier Debug-for-unit-tests -rationale ("fast compile") was BACKWARDS for this codebase — the diff oracle makes the RUN, -not the compile, the cost. - -So: everything **perf-relevant** is optimised, AND the gate's unit-test RUN (the diff -oracle) now runs ReleaseSafe so the gate is fast. Debug stays available via the bare -`zig build test -Dwasm` dev invocation for undefined-poisoning diagnostics. Debug stays the dev default purely -for build-iteration speed. - -## Related - -- `scripts/perf.sh` — the blessed measurement entrypoint. -- `.claude/rules/perf_marker.md` — `// PERF:` markers (cross-links here). -- `.dev/optimizations.md` — the O-NNN ledger; every row's numbers must be - Release (`scripts/perf.sh`), not Debug. diff --git a/.claude/rules/plan_revision_thinking.md b/.claude/rules/plan_revision_thinking.md deleted file mode 100644 index 6950acf89..000000000 --- a/.claude/rules/plan_revision_thinking.md +++ /dev/null @@ -1,33 +0,0 @@ ---- -paths: - - "**" ---- - -# Plan revision thinking (hook) - -> SSOT: `.dev/principle.md`. This file is the trigger only. - -## Bad Smell sensor - -If "wait, this feels off" / "this seems different from what we -planned" / "I'll fix it later" / "this looks like a hassle, let me -take a side route" comes to mind while editing, interrupt the -current edit, re-read `.dev/principle.md`, and take the surgery -at the right depth before continuing. The interrupt is not a -session stop — the loop keeps running, the smell just lands its -fix in-flight. - -## Judgement is yours - -- The catalogue is a memory aid, not a checklist. -- Pick depth 1-4 yourself. -- Fork a subagent for a deeper look when warranted. -- Record the conclusion at the right layer (commit message / ADR / - `debt.yaml` / `private/notes/`). - -## Why this rule auto-loads everywhere - -The smell can appear anywhere — in a `.zig` source change, in an -ADR edit, in a ROADMAP amendment, in a rule rewrite. Path `**` is -intentional. The detail is in the SSOT; the hook is short so it -does not crowd the context. diff --git a/.claude/rules/provisional_marker.md b/.claude/rules/provisional_marker.md deleted file mode 100644 index 16a3b4e93..000000000 --- a/.claude/rules/provisional_marker.md +++ /dev/null @@ -1,274 +0,0 @@ ---- -paths: - - "src/**/*.zig" - - "src/**/*.clj" - - "build.zig" - - "build.zig.zon" - - "test/e2e/**/*.sh" - - "data/feature_deps.yaml" - - ".dev/debt.yaml" ---- - -# Provisional marker comments - -Auto-loaded when editing any source file. Codifies the -**mechanised visibility** layer for provisional behaviour — -intermediate states that exist because "the other side is not -yet ready" (language-implementation chicken-and-egg). Marker -comments are the in-code anchor of the lifecycle; the SSOT is -[`data/feature_deps.yaml`](../../data/feature_deps.yaml) + a -[`.dev/debt.yaml`](../../.dev/debt.yaml) row. - -## Why this rule exists - -Language implementations grow in layers (Layer 0 = host code, -Layer 1 = primitives, Layer 2 = `.clj` defns over primitives, -Layer 3 = full `(ns ...)` macro). Until a layer is complete, -upper layers ride **provisional behaviour** in lower layers — -e.g. `evalInNs` auto-refers `rt/` and `clojure.core` because -the proper `(ns ...)` macro has not landed. - -Provisional behaviour is unavoidable. Forgetting that it is -provisional is not. Silent default-shift smell -(`.dev/principle.md`) emerges when a provisional default -calcifies into 「とりあえず動く」 forever because nothing in -the code points at "this is temporary; here is the close-out". - -The marker comment is the in-code anchor. Combined with -`data/feature_deps.yaml` and `.dev/debt.yaml`, the lifecycle is -mechanically auditable: - -1. **Introduce** provisional behaviour ⇒ add marker + open - `data/feature_deps.yaml` entry + open `.dev/debt.yaml` row (one - commit; hook enforces sync). -2. **Stay aware** while it persists ⇒ `audit_scaffolding` - reports marker count + 14-day-stale markers per Phase - boundary; per-task notes include a `## 暫定ログ` section - recording introduction / discharge / surprise this cycle. -3. **Discharge** when the upstream feature lands ⇒ remove - marker + close `data/feature_deps.yaml` entry's - `provisional_markers` list + close `.dev/debt.yaml` row - (same commit; hook enforces sync). - -## Canonical form - -```zig -// PROVISIONAL: [refs: D-NNN, feature_deps.yaml#] -``` - -```clojure -;; PROVISIONAL: [refs: D-NNN, feature_deps.yaml#] -``` - -Rules: - -- **Single line**. Multi-line rationale belongs in the - `data/feature_deps.yaml` entry's body or the `.dev/debt.yaml` row. -- **`refs:` block is mandatory** and lists at least one - `D-NNN` (debt row) AND at least one - `feature_deps.yaml#` (entry name). Multiple refs - comma-separated. -- **``** is the entry's `name:` field verbatim - (e.g. `clojure.core/select-keys`, - `runtime/eval/in_ns_auto_refer`). -- **Placed directly above** the provisional code, no blank - line between marker and the affected statement. -- **Removed on discharge**, not commented-out. The whole - point of the marker is grep-discoverability. - -## Examples - -### Introduce (Zig) - -The marker is one source line. Editors may wrap visually; do -not split with `//` continuations. - -```zig -// PROVISIONAL: in-ns auto-refers rt/ pending (ns ...) macro [refs: D-071, feature_deps.yaml#runtime/eval/in_ns_auto_refer] -if (env.findNs("rt")) |rt_ns| { - try env.referAll(rt_ns, env.current_ns.?); -} -``` - -### Introduce (Clojure) - -`clojure.set/join` ships 2-arity only; the 3-arity -`[xrel yrel km]` body requires multi-arity `fn*` dispatch -(D-070) and will land when D-070 closes. - -```clojure -;; PROVISIONAL: 2-arity only pending multi-arity fn* dispatch [refs: D-070, feature_deps.yaml#clojure.set/join] -(def join - (fn* [xrel yrel] - ...body...)) -``` - -**Counter-example — what is NOT provisional.** Variadic with -internal arity discrimination (e.g. `(fn* [& args] (if (= 0 -(count args)) ... ...))`) is the **finished form** for -"fold over N args" — no different-body-per-arity dispatch is -needed, so D-070 is not blocking it. Such a body would be a -false-positive marker and should NOT be tagged PROVISIONAL. -See `feature_deps.yaml#clojure.core/merge` notes for the -re-classification precedent (spike 2.3, 2026-05-26). - -### Discharge (one commit removes marker + closes refs) - -The discharging commit's diff shows: -- Marker line removed from source. -- `data/feature_deps.yaml` entry's `provisional_markers:` list - emptied (or the whole entry's `status:` flipped from - `provisional` to `landed`). -- `.dev/debt.yaml` entry moved from the `active:` list to the - `discharged:` list (or a status update on the entry). - -The hook (`scripts/check_provisional_sync.sh`) verifies all -three edits ride the same commit. - -## Distinguishing PROVISIONAL from FIXME / TODO / XXX - -- **`PROVISIONAL:`** — intentional intermediate state with a - named close-out plan in `data/feature_deps.yaml` + `.dev/debt.yaml`. - Lifecycle is mechanised. -- **`TODO:`** — forbidden in this project (TODO smell per - `.dev/principle.md`). Use `PROVISIONAL:` if there is a real - close-out plan; otherwise the work belongs in the current - commit. -- **`FIXME:` / `XXX:`** — not used; same rationale as `TODO:`. - -If a piece of code legitimately needs to flag a bug or design -issue with no close-out plan yet, file a `.dev/debt.yaml` row -first, then mark with `PROVISIONAL:` referring to the new row. -A marker without `refs:` is rejected by the hook. - -## Skeleton vs transient stub vs PROVISIONAL vs permanent no-op - -(Wave 16 W16-6 fold of `no_op_stub_forbidden.md` — the time-axis -distinction between "final shape" and "development trajectory" is -the central invariant; preserve verbatim.) - -The cw v1 codebase commits to: - -> **No Tier A / B / C feature may ship in a form where the user -> sees success while the runtime silently drops the intended -> semantics.** - -The verb is "ship": the rule is about what lands in the **final -shape** of the feature, not about the development trajectory. - -| Shape | Rule | Why | -|-----------------------------------------------------------------------------------------|--------|-------------------------------------------------------------------------| -| Skeleton struct that Phase N+ rewrites into a real impl | **OK** | Reservation lowers later surface ripple. ADR-0004 day-1 enum. | -| Function body that raises `feature_not_supported` | **OK** | Explicit user signal. Transient — disappears when the real impl lands. | -| Function body that **runs** with intermediate semantics + carries `PROVISIONAL:` marker | **OK** | Marker + yaml entry + debt row triad makes the lifecycle auditable. | -| Function body that returns the input unchanged | **NG** | Pretends to work, drops semantics silently. The user builds on a lie. | -| Macro that expands to `nil` or to its body without effect | **NG** | Same shape — user code compiles and runs, semantics are gone. | - -The third row is where the **`PROVISIONAL:` marker** earns its -keep: the function DOES run with intermediate semantics (= not a -`feature_not_supported` raise), and the marker + SSOT triad -record that the intermediate behaviour is intentional + has a -close-out predicate. Without the marker, the third row collapses -into the fourth (= permanent no-op). - -### Boundary rules - -A **skeleton** is permitted when any of: -- Only the struct type definition exists (no function declared yet). -- A function is declared but its body is exactly - `return error_catalog.raise(.unsupported_feature, loc, .{ .name = "
" });` - (per ADR-0018), or for internal-only paths - `return error.NotImplemented;` / `@panic("...")` with a developer- - visible comment naming the future task. - -A **transient stub** is permitted when the function ships an -explicit error rather than fake-running. - -A **PROVISIONAL behaviour** is permitted when the function runs -with intermediate semantics AND carries the `PROVISIONAL:` marker -+ yaml entry + debt row triad (this is the new third row). - -A **permanent no-op** is **forbidden** when: -- A function is declared and executes the argument without the - intended semantics (e.g., `dosync` body executed without snapshot - isolation). -- A function returns a default value that masks the missing feature. -- A macro expands to `nil` or to its body without effect. - -The boundary is **what the user observes** + **whether the -intermediate state is mechanically tracked**. - -### Examples - -Forbidden: -```zig -pub fn dosync(rt: *Runtime, body: Value) !Value { - return eval(rt, body); // ❌ no snapshot isolation; ships as STM that isn't STM -} -``` - -Allowed (transient stub): -```zig -pub fn dosync(rt: *Runtime, loc: SourceLocation, body: Value) !Value { - _ = rt; - _ = body; - return error_catalog.raise(.unsupported_feature, loc, .{ .name = "dosync" }); -} -``` - -Allowed (PROVISIONAL — runs intermediate semantics with the triad): -```zig -// PROVISIONAL: in-ns auto-refers rt/ pending (ns ...) macro [refs: D-071, feature_deps.yaml#runtime/eval/in_ns_auto_refer] -if (env.findNs("rt")) |rt_ns| { - try env.referAll(rt_ns, env.current_ns.?); -} -``` - -### Why (Shota's original directive, preserved) - -- A stub that "works" misleads users into building code that breaks - later. -- STM (`dosync` without snapshot isolation) and locking (`locking` - without lock) are common offenders in JVM-non-equivalent runtimes. -- cw v1 commits to either a real implementation, an explicit error, - or a PROVISIONAL-tracked intermediate. -- Skeleton-then-rewrite **is** how the codebase grows; the only - thing forbidden is shipping a lie. - -### Enforcement - -- `scripts/check_no_op_stub.sh` — heuristic scan (currently - informational; W16-2 refreshed the activation criterion). -- `scripts/check_provisional_sync.sh` — PROVISIONAL marker + SSOT - triad sync (hard gate at push time). -- ADR-0004 / ADR-0012 / ADR-0023 endorse skeleton-then-rewrite for - day-1 reservations. - -## Marker scope - -The rule applies to source files in the directories where -provisional behaviour is meaningful: - -- `src/**` (every Zig + `.clj` source) -- `build.zig`, `build.zig.zon` -- `test/e2e/**.sh` (test-side workarounds also qualify) - -Configuration files, documentation, and generated artefacts -(`.zig-cache/`, `zig-out/`) are out of scope. - -## Cross-references - -- [`.dev/principle.md`](../../.dev/principle.md) — the Bad - Smell catalogue this rule operationalises (Silent - default-shift, Smallest-diff bias, Reservation-as-bias, - Progress-pressure). -- [`data/feature_deps.yaml`](../../data/feature_deps.yaml) — the SSOT - for provisional entries (`status: provisional` + the - `provisional_markers` list). -- [`.dev/debt.yaml`](../../.dev/debt.yaml) — the SSOT for the - close-out plan (one row per provisional, named by `D-NNN`). -- [`scripts/check_provisional_sync.sh`](../../scripts/check_provisional_sync.sh) - — the enforcement hook (PreToolUse on `git push`). -- [`bootstrap_essence.md`](bootstrap_essence.md) — why - provisional behaviour is the default mode of language - implementation, not the exception. diff --git a/.claude/rules/test_taxonomy.md b/.claude/rules/test_taxonomy.md deleted file mode 100644 index 627f10b31..000000000 --- a/.claude/rules/test_taxonomy.md +++ /dev/null @@ -1,98 +0,0 @@ ---- -paths: - - src/**/*.zig - - test/** - - bench/** ---- - -# Test taxonomy (5 layers) - -## Rule - -Every test belongs to exactly one of the 5 layers per ADR-0021. -Choose the layer before writing the test. - -| # | Layer | Where | When to choose | -|---|-------------------|-------------------------------------------------|------------------------------------------------------------------------------------------------------------| -| 1 | Unit | `src/**/*.zig` inside `test "..."` blocks | Single function, deterministic, no I/O. Default. | -| 2 | E2E (CLI) | `test/e2e/*.sh` | `cljw -e '...'` invocation surface, error message rendering, exit codes. | -| 3 | Differential | `test/diff/cases.yaml` | TreeWalk and VM must produce equal Value for the same source. ADR-0005 / 0022. | -| 4 | Bench (on demand) | `bench/compare_langs.sh` + `bench/run_bench.sh` | Perf measurement — run on demand, NOT a gate layer (the `quick.sh` auto-baseline was retired 2026-06-11). | -| 5 | Conformance | `test/clj/` (Phase 11+) | Adapted upstream Clojure JVM test. | - -## Why - -- "Where does this test go?" was a recurring review question; the - table answers it. -- Layers 1-3 + 5 match the runner steps in `test/run_all.sh` - (zig_build_test, e2e_*, diff_runner, test_clj). Layer 4 (bench) was - retired from the gate 2026-06-11 — perf is measured on demand. -- ADR-0021 names 8 future layers (Integration, Golden, Property, - Fuzz, Memleak, Concurrency, Bench full, Wasm component) — they - open at their phase, not at Phase 4 entry. - -## How to apply - -### Decision rule - -Ask in order: - -1. Is it a single deterministic function call? → Layer 1. -2. Is it `cljw -e ...` / file invocation? → Layer 2. -3. Does it require TreeWalk and VM to agree? → Layer 3. -4. Is it a performance measurement? → Layer 4 (run on demand via - `bench/`, not added to the gate). -5. Is it an upstream Clojure test port (Phase 11+)? → Layer 5. - -If none fit, the test is in a future layer (defer or open the -future layer's directory). - -### Naming - -- Layer 1: `test " "` - (e.g., `test "+ on two integers"`). -- Layer 2: file name `phase_.sh`, case name in the file. -- Layer 3: `cases.yaml` `name:` is `_` - (`closure_capture_local`, `recur_loop_n3`). -- Layer 4: a benchmark dir `bench/benchmarks/NN_/` (run on demand). -- Layer 5: upstream filename is preserved. - -## Wall-clock assertions: lower bounds and orders of magnitude only - -Layer 1 says "deterministic". Elapsed time is not, and the gate runs on shared -CI runners whose scheduling noise is larger than most effects worth measuring. - -- **A lower bound is safe.** Load makes an operation slower, never faster, so - "this sleep did not return early" holds on any machine. -- **An order-of-magnitude upper bound is safe.** "It did not wait out the whole - 10 s deadline" discriminates a real hang from a busy machine. -- **A RATIO upper bound is a flake generator.** `elapsed < want * 3/2` cannot - separate a 20% systematic regression from a 50% busy runner, so it eventually - fails for the one reason it was not written to detect. 2026-08-05: - `budgetedSleep: a metered sleep … keeps its duration` asserted exactly that, - passed every local full gate, and failed on the CI macOS runner. - -A performance ratio is a **measurement**, so take it on a quiet machine (Layer 4 -`bench/`, run on demand) and record the numbers in the ADR. The unit test keeps -the deterministic half. - -- **A work-quantity ceiling is a hidden ratio bound.** An elapsed upper bound on - "burn N steps / N fuel" scales with per-step cost × host speed, and per-step - cost varies ~15x across backends (tree_walk vs vm) before hardware is even - considered. 2026-08-05 nightly: `thread_loop_trips_shared_steps` burned 30M - budget steps in 4 s on a mac tree-walk and 58 s on the CI Linux runner, - failing its 45 s bound only in that one config. If the assertion needs an - elapsed bound, make the metered work *small* (gate the burn on an explicit - synchronisation point rather than sizing the ceiling against a race window), - or bound against a wall-clock deadline the feature itself enforces. - -## Counter-examples - -Don't write a Layer 1 unit test that shells out to `cljw` (that -belongs in Layer 2). - -Don't write a Layer 3 case that does not exercise the -backend boundary (TreeWalk-only behaviour belongs in Layer 1). - -Don't measure performance with a Layer 1 test (Layer 4 is the -home). diff --git a/.claude/rules/textbook_survey.md b/.claude/rules/textbook_survey.md deleted file mode 100644 index 125a44a74..000000000 --- a/.claude/rules/textbook_survey.md +++ /dev/null @@ -1,153 +0,0 @@ ---- -paths: - - "src/**/*.zig" - - "build.zig" ---- - -# Textbook survey before implementation - -Auto-loaded when editing Zig sources. Codifies how the `/continue` -per-task TDD loop's **Step 0 (Survey)** consults the reference -codebases without being pulled by their styles. - -## The textbooks - -The full list of reference clones + purpose-of-use lives in -[`.dev/reference_clones.md`](../../.dev/reference_clones.md) (the -SSOT — matches `.claude/settings.json` `additionalDirectories`). -Survey-time mapping: - -- **Always** when introducing a Clojure feature: `~/Documents/OSS/clojure/` - (`clojure/lang/*.java` + `clojure/core.clj`) for canonical - semantics; cw v0 (git tag `v0.5.0`, via `git worktree add ../cw-v0 v0.5.0`) for - prior-art and known pain points. -- **For JVM-less precedent**: `~/Documents/OSS/babashka/`. -- **For Zig 0.16 stdlib idiom**: the pinned toolchain's own stdlib — - `$(zig env | grep std_dir)`. Not a clone: a checkout drifts from the pin - (ADR-0090 §Context has a survey misled by a post-0.16 master checkout). -- **Phase-specific**: spec → `spec.alpha/` + `malli/` (Phase 14+); - Wasm runtime → `wasmtime/` (Phase 16+); type-system vocab → - `mattpocock_skills/` (when designing a new module). - -## Survey procedure (default brief for the survey subagent) - -Step 0 of the TDD loop dispatches a **`general-purpose`** subagent -(NOT `Explore`). The deliverable is a written file at -`private/notes/--survey.md`, and the built-in `Explore` -agent is shipped without `Edit` / `Write` / `NotebookEdit` (it is -read-only by design — see Claude Code's subagent definitions). A -read-only agent forces the main agent to re-route the survey body -through its own context to persist the file, which wastes context -and breaks on auto-compaction. - -Use `general-purpose` (which carries the full tool set) so the -subagent writes the survey note directly. Use this brief shape: - -``` -Survey how is implemented in: - - ../cw-v0 (cw v0 — `git worktree add ../cw-v0 v0.5.0`) - - ~/Documents/OSS/ (upstream) - -Return 200–400 lines: - - Files & line ranges where the concept lives - - Key data shapes (types, fields) - - Idioms used (Zig 0.16 / Clojure / etc.) - - Differences across the three sources, with one-line "why each - chose this" - - 2–3 places where ClojureWasm v2 should likely DIVERGE based on - ROADMAP §2 (Inviolable principles): P3 core stays stable, P4 no - ad-hoc patches, A2 new features via new files, A6 ≤ 1000 lines. - -Do NOT copy code. Describe the design space. -``` - -The summary lands in `private/notes/--survey.md`. - -## Anti-pull guardrails - -A survey is a hazard: reading 89K LOC of v1 makes it tempting to -copy the patterns wholesale. To prevent this: - -### Guard 1 — Cite ROADMAP principles before adopting a v1 idiom - -For each idiom you import from v1, write one line: - -> Adopting v1 `` because ROADMAP P# / A# / §N.M aligns with it. - -If you can't write that line, the idiom is folklore — re-derive from -ROADMAP first. - -### Guard 2 — Always note one DIVERGENCE - -Step 0's deliverable must include "where ClojureWasm v2 diverges from -all three textbooks". If it doesn't, the survey was too shallow or -the concept is mechanical (in which case Step 0 should have been -skipped — see below). - -### Guard 3 — `pub var` is forbidden, even if v1 uses it - -ROADMAP §13 lists patterns to reject regardless of textbook -precedent: `pub var` vtables, `std.Thread.Mutex`, `std.io.AnyWriter`, -JVM-class implementations. The survey may surface these; do not -adopt them. - -### Guard 4 — Tier outsiders go to ADR or pod - -If v1 has a `lib_X` integration that requires branching code in a -shared file, **do not replicate**. Either write a Tier-promotion -ADR (`.dev/decisions/NNNN_promote_X.md`) or implement as a Wasm -Component pod. ROADMAP §6.4 is non-negotiable. - -## The survey is not immutable — Step 0.6 may amend it - -Surveys are ~80% accurate at write time. The remaining ~20% surfaces -only once implementation begins (the prerequisite was missed; the -recommended shape is smallest-diff bias against the finished form; -a provisional behaviour will need to be introduced). - -CLAUDE.md § Autonomous Workflow **Step 0.6 (Re-laying against -finished-form)** is the main-agent pass that catches this. After -Step 0 hands back the survey note, before Step 1's plan is -committed, the main agent: - -- Walks the prerequisite chain to confirm what the survey assumed. -- Re-lays the recommended shape against the F-NNN envelope - (`.dev/project_facts.md`) + `.dev/structure_plan.md` + - ROADMAP §9. -- **Amends the survey in place** when reality drifts from the - survey's prediction. Survey notes are not immutable contracts; - they are the loop's working memory. -- Forks a Devil's-advocate subagent (mandatory, even at depth 1) - when the re-laying surfaces a provisional behaviour, to - enumerate finished-form-clean alternatives within the F-NNN - envelope before committing to the provisional shape. - -This is the discipline answer to "Step 0 is by a subagent and only -sees the snapshot; finished-form alignment is a main-agent -responsibility". The survey is the input; Step 0.6 is the -re-orientation against the project's invariants. - -## When to skip Step 0 - -Skip only when **all** are true: - -- The task is a refactor / rename / doc-only change. -- No new public API is introduced. -- Implementation does not change behaviour observable from outside the - module. - -If any of the above is false, do Step 0 even if "you already know how -v1 did it" — the survey output is the **input to the per-task note** -and the future chapter, not just for your present working memory. - -## Where survey notes live - -| File | Purpose | Tracked in git? | -|--------------------------------------------|---------------------------------------------------|-------------------| -| `private/notes/--survey.md` | Step 0 raw output | No (`.gitignore`) | -| `private/notes/-.md` | Step 7 per-task note (digests survey) | No | -| `docs/ja/learn_clojurewasm/NNNN_.md` | Chapter (digests notes) | Yes | -| `.dev/decisions/NNNN_.md` | ADR (load-bearing decisions surfaced from survey) | Yes | - -The flow is: **survey notes → per-task notes → chapter**. Not the -reverse, not skipping a step. Each layer is shorter than the last. diff --git a/.claude/rules/tier_classification.md b/.claude/rules/tier_classification.md deleted file mode 100644 index 1d8bb4a56..000000000 --- a/.claude/rules/tier_classification.md +++ /dev/null @@ -1,42 +0,0 @@ ---- -paths: - - src/lang/** - - src/runtime/host/** - - data/compat_tiers.yaml ---- - -# Tier classification discipline - -## Rule - -Every public function and type carries a tier classification A / B / C / D -(per `data/compat_tiers.yaml` and ADR-0013). - -Tier D forms raise a per-form catalog Code (`tier_d_gen_class` / -`tier_d_gen_interface` / `tier_d_compile` / `tier_d_proxy_deep` / -`tier_d_bean_deep`); each Code carries a hand-written multi-sentence -template. The canonical naming + template-content spec lives in -**ADR-0018 amendment 2** (= the catalog SSOT); the Tier label (D) -and rationale ADR (ADR-0013) live in `data/compat_tiers.yaml`; neither -the label nor the ADR identifier appears in user-facing text. - -## Why - -- Tier classification is the cw v1 contract with users. -- Users see a structured signal for each unsupported feature. -- Tier promotion (e.g., D -> C) requires explicit ADR amendment. - -## How to apply - -- New function: add entry to `data/compat_tiers.yaml`, include tier in - docstring. -- Tier D function: error message format per the rule. -- Tier promotion: amend `data/compat_tiers.yaml` + ADR. - -## Tier promotion / demotion - -- D -> any: MAJOR release required (per ROADMAP §1.4 SemVer rule). -- A -> A (no change), A -> B: MAJOR. -- B -> A: MINOR. -- C -> A or C -> B: MINOR. -- A or B -> D: forbidden (would break Clojure source compatibility). diff --git a/.claude/rules/yaml_ssot_yq.md b/.claude/rules/yaml_ssot_yq.md deleted file mode 100644 index e3ec0c7ef..000000000 --- a/.claude/rules/yaml_ssot_yq.md +++ /dev/null @@ -1,245 +0,0 @@ ---- -paths: - - .dev/debt.yaml - - data/compat_tiers.yaml - - data/placement.yaml - - data/feature_deps.yaml ---- - -# Querying the YAML SSOTs with `yq` (shell-escaping + cookbook) - -Auto-loaded when editing a structured YAML SSOT (`.dev/debt.yaml`, -`data/compat_tiers.yaml`, `data/placement.yaml`, `data/feature_deps.yaml`). Codifies the -`yq` query + shell-escaping idioms so no session re-derives them. The SSOT -for the *content* of each file is the file; this rule is the SSOT for *how -to read/edit it from a shell*. - -## Flavor - -**mikefarah Go `yq` v4** (NOT the python `yq`/`jq` wrapper). Confirmed -v4.53.2. Key consequences: expression syntax is jq-like but its own; `-r` -emits raw (unquoted) scalars; `-i` edits in place. Reading a shell env var -has TWO forms with a critical difference: **`strenv(NAME)` injects the value -as an opaque string; `env(NAME)` TYPE-PARSES the value as YAML first.** Use -`strenv()` for any string value — `env()` silently breaks on prose (see the -Golden rule below). This is the single most common yq footgun in this repo. - -## Golden rule for shell-escaping (this is the whole trick) - -1. **Single-quote the entire yq expression.** yq expressions are full of - `|`, `()`, `[]`, `.`, `==`, `"`, and zsh-`NOMATCH` glob metachars - (`[`, `?`, `*`). Single quotes neutralise all of it — the only safe - default (mirrors `~/.claude/CLAUDE.md` § シェル実行時のクォート規則). - - ```sh - yq -r '.active[] | select(.category == "polymorphism") | .id' .dev/debt.yaml - ``` - Literal double-quotes for string *values* sit fine inside the single - quotes — no escaping. - -2. **Pass shell variables via `strenv(VAR)`, NEVER string-interpolation.** - The fragile form `yq "... == \"$x\""` forces double-quoting the whole - expression (so every `[`/`*`/`"` inside now needs escaping) and breaks - on values containing quotes. The robust form keeps the expression - single-quoted and injects the value out-of-band: - - ```sh - # ✅ robust — expression stays single-quoted, value passed as strenv - DROW="D-203" yq -r '.active[] | select(.id == strenv(DROW)) | .status' .dev/debt.yaml - # ❌ fragile — nested quotes, breaks on metachars / quotes in $drow - yq -r ".active[] | select(.id == \"$drow\") | .status" .dev/debt.yaml - ``` - - **`strenv()` not `env()` — this is the footgun that made yq "useless" - for editing prose fields.** `env(VAR)` type-parses the value as YAML to - infer its type, so a value containing two-or-more `: ` (colon-space) - sequences — i.e. EVERY `status:` / `barrier:` / `resolution:` prose - field in `debt.yaml` — fails YAML map parsing with - `mapping values are not allowed in this context`, and the assignment is - **silently dropped** (the field keeps its old value, exit non-zero). - `strenv(VAR)` treats the value as an opaque string and is immune. - Verified v4.53.2: - - ```sh - # ❌ env() — dies on multi-colon prose, field UNCHANGED - V='Part 1: foo. Part 2: bar.' yq -i '(.active[0].status) = env(V)' f.yaml - # → Error: yaml: line 1, column N: mapping values are not allowed … - # ✅ strenv() — opaque string, assignment lands - V='Part 1: foo. Part 2: bar.' yq -i '(.active[0].status) = strenv(V)' f.yaml - ``` - - Use `env(VAR)` ONLY when you deliberately want the value typed as a - number / bool / null (rare). For ids, dates, categories, and all prose, - `strenv(VAR)` is the default. - -3. **`yq -i` (in-place) PRESERVES comments and `|-` block scalars** - (verified v4.53.2 — header comments + every barrier block survived a - field edit). So scalar-field updates are safe to automate: - - ```sh - DROW="D-203" yq -i '(.active[] | select(.id == strenv(DROW)) | .last_reviewed) = "2026-06-02"' .dev/debt.yaml - # Single-line prose field (status / one-line barrier): strenv() works — - S='DISCHARGED 2026-06-09 — landed; Part 1: x. Part 2: y.' \ - DROW="D-356" yq -i '(.active[] | select(.id == strenv(DROW)) | .status) = strenv(S)' .dev/debt.yaml - ``` - `strenv()` makes single-line prose assignment safe (the colon footgun is - gone). BUT for a **new multi-line block scalar** (a fresh `barrier: |-` - /`resolution: |-` block spanning several lines), still hand-edit with the - Edit tool — `yq -i` emits a single-line double-quoted scalar (no `|-` - block), so it round-trips correctly but reflows the prose onto one line - and loses the block formatting. Use Edit when the block shape matters. - -4. **Appending a NEW entry with `+=` writes its scalars UNQUOTED — which - breaks any quote-anchored grep.** `yq -i '.active += [{"id":"D-439", …}]'` - emits `- id: D-439` (plain scalar, NO quotes), even though every - hand-written row is `- id: "D-439"`. The hazard is not the YAML (both - parse) — it is that the **next-free-id recipe and the phantom-audit recipe - grep `id: "D-` (with the quote)**, so they SILENTLY UNDERCOUNT the unquoted - rows. A 2026-06-14 incident: three `+=`-appended rows (D-437/438/439) were - unquoted, the next-id grep returned 436, and the next append would have - re-used a live id (duplicate). Mitigations, in order of preference: - - **Prefer the Edit tool to add a new debt/AD row** — hand-write the - `- id: "D-NNN"` block so it matches the quoted convention (and a fresh - `barrier: |-` block keeps its shape, per item 3). This is the default. - - If you DO use `+=`, **force double-quote style on the id in the same - call**: `yq -i '.active += [{"id":"D-439"}] | (.active[-1].id) style="double"'` - (or normalize after: `sed -i '' -E 's/^( *- id: )(D-[0-9]+)$/\1"\2"/' f`). - - The **next-id / phantom recipes below are now quote-TOLERANT** (`id: "?D-`) - so they survive an unquoted row — but quoting on write is still required - so the file stays consistent + greppable by other tools. - -## debt.yaml cookbook (the recurring queries) - -Structure: **three** top-level lists `active:` / `standing:` / `discharged:` -(NOT two — `standing:` holds epics/campaigns the loop does not auto-drain). -Active + standing entries: `id` / `status` / `category` / `barrier` (+ optional -`quality_floor` / `last_reviewed`). **Discharged is MIXED-schema** (verified -2026-06-17): ~199 rows reuse the active shape (`status` starting `DISCHARGED …` -+ `category`/`barrier`/`last_reviewed`), ~155 use the lighter `discharged_at` / -`resolution`. So a moved-from-active row keeps its `status` block (no reformat -needed); only hand-author `discharged_at`/`resolution` when writing a fresh -discharge. Query both shapes with `(.status // .resolution)`. - -> **Precedence footgun (cost a re-derivation 2026-06-17).** `|` binds TIGHTER -> than `,`, so `.active[],.standing[] | select(…)` means `.active[]` (RAW) AND -> `(.standing[] | select(…))` — the select silently applies to only the LAST -> stream and every active row dumps unfiltered (looks like "select matched -> everything"). **Always parenthesize the union**: `(.active[],.standing[]) | -> select(…)`. Same for any `(.a[],.b[],.c[]) | …` across the three sections. - -```sh -# counts -yq -r '.active | length' .dev/debt.yaml -yq -r '.discharged | length' .dev/debt.yaml - -# all active ids -yq -r '.active[].id' .dev/debt.yaml - -# one entry's field (escaping-free var) -DROW="D-203" yq -r '.active[] | select(.id == strenv(DROW)) | .barrier' .dev/debt.yaml - -# find a row by id ACROSS all 3 sections (note the PARENS — see precedence footgun) -DROW="D-386" yq -r '(.active[],.standing[],.discharged[]) | select(.id == strenv(DROW)) | .id + " | " + (.status // .resolution)' .dev/debt.yaml - -# active rows whose status is actually DISCHARGED (misfiled — belong in discharged:) -yq -r '.active[] | select(.status | test("^DISCHARGED|^Discharged")) | .id' .dev/debt.yaml - -# filter by category / status substring -yq -r '.active[] | select(.category == "polymorphism") | .id' .dev/debt.yaml -yq -r '.active[] | select(.status | test("blocked-by")) | .id' .dev/debt.yaml - -# quality-loop floor backlog (the F-010 drain list) -yq -r '.active[] | select(has("quality_floor")) | .id + " :: " + .quality_floor' .dev/debt.yaml - -# is an id discharged? (in discharged: OR an active entry marked DISCHARGED) -DROW="D-018"; yq -r '.discharged[].id, (.active[] | select(.status | test("DISCHARGED|Discharged")) | .id)' .dev/debt.yaml | grep -qx "$DROW" && echo discharged || echo open - -# highest existing id → next free is +1. PREFER the yq form — it reads the PARSED -# `.id` values, so it is immune to quote-style drift (an unquoted `+=`-appended row -# is counted; see Golden-rule #4): -yq -r '.active[].id, .standing[].id, .discharged[].id' .dev/debt.yaml | grep -oE '[0-9]+' | sort -n | tail -1 -# grep fallback — MUST scope to the `id:` field (a bare `grep -oE 'D-[0-9]+'` also -# matches D-NNN in PROSE/cross-refs → a phantom too-high number) AND be -# quote-TOLERANT (`"?`) so an unquoted `+=` row is not undercounted: -grep -oE 'id: "?D-[0-9]+' .dev/debt.yaml | grep -oE '[0-9]+' | sort -n | tail -1 -``` - -Note: `check_debt_id_refs.sh` does the phantom/undefined-id gate with -plain `rg` over the file (any `D-NNN` anywhere counts as "defined"), so -that check does NOT need yq — keep it grep-based. **Known blind spot**: a -typo'd reference that itself looks like a `D-NNN` (e.g. a stray `D-NNNN`) -appears in its own prose, so the "appears somewhere → defined" rule counts -it as defined and the gate does NOT flag it. The id-scoped next-id recipe -above is the robust cross-check (a phantom never has an `id:` row); to -audit for phantoms, diff the set of referenced ids against the set of -`id:`-defined ids: -`comm -23 <(grep -oE 'D-[0-9]+' .dev/debt.yaml | sort -u) <(yq -r '(.active[],.standing[],.discharged[]).id' .dev/debt.yaml | sort -u)` -prints any referenced-but-undefined id. (The defined-id side uses `yq` so it is -quote-style-agnostic per Golden-rule #4; a grep fallback must use `id: "?D-`.) - -> **This recipe is a SCREEN, not a verdict — it has FALSE POSITIVES (verified -> 2026-06-17).** `grep -oE 'D-[0-9]+'` strips letter suffixes, so a sub-id ref -> like a `D-NNNa` (a REAL row) shows up as a phantom base `D-NNN`; and it matches -> digits embedded in unrelated tokens — e.g. a `UCD-` Unicode-database -> version string yields a spurious `D-`. Do NOT use a `\b` word-boundary -> "fix" — BSD/macOS `grep -oE '\bD-[0-9]+'` emits MORE garbage (partial -> low-digit `D-` fragments). Instead, `grep -nF` EACH flagged id to see its real context before -> acting. In the 2026-06-17 audit only 1 of 3 hits was real: a date `2026-06-13` -> carried an erroneous `D-` prefix (so it read as an undefined id); the other two -> were a sub-id base and a Unicode-version fragment. The lesson the user flagged: -> a noisy audit recipe + the gate's blind spot let that one typo accumulate — -> confirm each hit, don't dismiss the batch as "probably all noise", don't chase -> the noise as if all real. **NB: do not write a literal undefined `D-` -> token in any `.claude/`/`.dev/` prose example — `check_debt_id_refs.sh` scans -> those paths and will flag it as a new phantom (this very note triggered that). -> Use `D-NNN` / `D-NNNa` placeholders.** - -## Auditing the SSOTs (run these when asked to audit, or after bulk yq edits) - -```sh -# (1) Well-formedness — every SSOT must parse: -for f in .dev/debt.yaml .dev/accepted_divergences.yaml data/compat_tiers.yaml \ - data/placement.yaml data/feature_deps.yaml data/host_interfaces.yaml; do - yq -e '.' "$f" >/dev/null 2>&1 && echo "OK $f" || echo "FAIL $f"; done - -# (2) Duplicate ids (a stray body-less `- id: "D-NNN"` from a botched edit parses -# fine but duplicates a real entry — yq well-formedness does NOT catch it): -yq -r '.active[].id, .standing[].id, .discharged[].id' .dev/debt.yaml | sort | uniq -d # empty = clean -yq -r '.[][].id' .dev/accepted_divergences.yaml 2>/dev/null | sort | uniq -d - -# (3) Quote-style drift — unquoted ids from `+=` appends (Golden-rule #4): -grep -nE '^\s*-? *id: [^"'"'"' ]' .dev/debt.yaml # empty = all quoted -``` - -The 2026-06-14 audit caught both classes: a stray `- id: "D-396"` (a body-less -duplicate of the real discharged row, from a prior botched edit — recipe 2) and -three unquoted `+=`-appended ids (recipe 3). yq `.' parses both, so these need the -dedicated dup/quote recipes, not just a well-formedness check. - -## Other SSOTs (same idioms) - -- `data/compat_tiers.yaml` / `data/placement.yaml`: same single-quote + `strenv()` - rules. `data/placement.yaml` automation lives in - `scripts/check_placement_status.sh`; `data/feature_deps.yaml` in - `scripts/check_provisional_sync.sh` + `audit_scaffolding/CHECKS.md`. - -## Scope (forward-looking — like `orphan_prevention.md` / `zig_tips.md`) - -This is a **reference** rule: it guides *future* yq usage, it does not gate -or mandate a one-shot retrofit. Existing scripts that still use the fragile -string-splice form work today because their interpolated values are -controlled (no metachars): `scripts/check_placement_status.sh:63,68` -(`\"$status\"`). Harden it to -`strenv()` **opportunistically** when next touching those files — not as a -standalone churn. (The two `env(ID)` reads in -`scripts/check_accepted_divergences.sh:50,56` compare against an `id` token -with no `: `, so `env()` does not break there — but `strenv()` is the safer -form to adopt next time that file is touched.) - -## Related - -- `~/.claude/CLAUDE.md` § シェル実行時のクォート規則 — the general zsh - `NOMATCH` / single-quote rule this distils for yq. -- `.claude/rules/debt_dedup.md` — debt.yaml dedup discipline (links here). -- `.claude/skills/audit_scaffolding/CHECKS.md` — the yq-based discharged - check (canonical worked example). diff --git a/.claude/rules/zig_tips.md b/.claude/rules/zig_tips.md deleted file mode 100644 index 1c71341b3..000000000 --- a/.claude/rules/zig_tips.md +++ /dev/null @@ -1,422 +0,0 @@ ---- -paths: - - "src/**/*.zig" - - "build.zig" ---- - -# Zig 0.16.0 idioms (project rules) - -Auto-loaded when editing Zig source. **AI assistants tend to revert to -pre-0.16 (often pre-0.14) APIs by default — consult this list before -typing any stdlib reference.** When in doubt, grep -`/nix/store/*-zig-0.16.0/lib/std/` for the canonical surface. - -## 0.14 → 0.16 removals & renames (must-know) - -These are the patterns that compile-fail (or fail silently via deprecated -shim) but AI training corpora overwhelmingly use the old names. Memorise. - -| Removed / renamed (0.14 / 0.15) | Use in 0.16.0 | Notes | -|-------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------| -| `std.io` (lowercase namespace) | `std.Io` (capital I) | `std.io` namespace effectively gone; `std.Io` is the only canonical module | -| `std.io.AnyWriter` / `AnyReader` | `*std.Io.Writer` / `*std.Io.Reader` | Type-erased pointer to concrete vtable type | -| `std.io.fixedBufferStream(&buf)` | `var w: std.Io.Writer = .fixed(&buf);` | Reader: `var r: std.Io.Reader = .fixed(&buf);` | -| `std.io.getStdOut/Err/In` | `std.Io.File.stdout/stderr/stdin()` | Returns `File`, then `.writer(io, &buf).interface` | -| `std.io.bufferedWriter` / `BufferedWriter` | `file.writer(io, &buf)` | Buffer is the user-supplied byte slice | -| `std.io.tty.*` | `std.Io.Terminal` | TTY detection / colour | -| `list.writer().any()` (allocating) | `var aw: std.Io.Writer.Allocating = .init(alloc);` then `&aw.writer` | `aw.toOwnedSlice()` extracts | -| `std.fs.File` (and `.OpenError` / `.Writer` / etc.) | `std.Io.File` | All file ops now take `io: std.Io` | -| `std.fs.cwd()` | `std.Io.Dir.cwd()` | — | -| `std.fs.openFileAbsolute(path, ...)` | `std.Io.Dir.cwd().openFile(io, path, ...)` | `io` arg now mandatory | -| `std.fs.Dir` | `std.Io.Dir` | All dir ops take `io: std.Io` | -| `std.fs.path.*` | `std.Io.Dir.path.*` | `std.fs.path` left as deprecated re-export | -| `std.fs.max_path_bytes` / `max_name_bytes` | `std.Io.Dir.max_path_bytes` / `.max_name_bytes` | — | -| `std.Thread.Mutex` / `RwLock` / `Condition` / `Semaphore` / `WaitGroup` | `std.Io.Mutex` / `Io.RwLock` / `Io.Semaphore` (with `io: Io` arg), or `std.atomic.Mutex` (lock-free `tryLock` / `unlock` only) | **All `std.Thread.*` sync primitives are gone** | -| `std.heap.GeneralPurposeAllocator(.{})` | `std.heap.DebugAllocator(.{})` | Same config struct, renamed | -| `std.ArrayList(T)` (managed, with internal allocator) | `std.ArrayList(T)` — **same name, unmanaged behaviour** (per-call allocator) | The new `ArrayList` IS the old `ArrayListUnmanaged`. `init` takes no allocator | -| `std.ArrayListUnmanaged(T)` | `std.ArrayList(T)` | The `Unmanaged` alias is deprecated; the new `ArrayList` is unmanaged | -| `std.StringHashMap` / `std.AutoHashMap` (managed) | `std.StringHashMap` / `std.AutoHashMap` (unmanaged) or `std.array_hash_map.String` for ordered | Managed wrappers gone; same shape applies as `ArrayList` | -| `std.StringArrayHashMapUnmanaged(V)` | `std.array_hash_map.String(V)` | New name for the ordered string-keyed map | -| `std.mem.copy(T, dest, src)` | `@memcpy(dest, src)` (or `@memmove` if overlapping; or `std.mem.copyForwards` / `copyBackwards` for explicit direction) | `mem.copy` removed | -| `std.mem.indexOf` | `std.mem.find` | — | -| `std.mem.lastIndexOf` | `std.mem.findLast` | — | -| `std.mem.lastIndexOfLinear` | `std.mem.findLastLinear` | linear-scan variant; distinct function from `findLast` | -| `std.mem.indexOfScalar` | `std.mem.findScalar` | — | -| `std.mem.lastIndexOfScalar` | `std.mem.findScalarLast` | — | -| `std.mem.indexOfScalarPos` | `std.mem.findScalarPos` | — | -| `std.mem.indexOfAny` / `lastIndexOfAny` / `indexOfAnyPos` | `findAny` / `findLastAny` / `findAnyPos` | — | -| `std.mem.indexOfNone` / `lastIndexOfNone` | `findNone` / `findLastNone` | — | -| `std.mem.indexOfDiff` | `std.mem.findDiff` | — | -| `std.mem.indexOfSentinel` | `std.mem.findSentinel` | — | -| `std.mem.indexOfPos` | `std.mem.findPos` | — | -| `std.mem.containsAtLeastScalar(...)` | `std.mem.containsAtLeastScalar2(...)` | Signature changed | -| `std.meta.Int(.signed, n)` / `std.meta.Int(.unsigned, n)` | `@Int(.signed, n)` / `@Int(.unsigned, n)` | Now a builtin in 0.16; `std.meta.Int` is a deprecated wrapper | -| `std.mem.readPackedIntNative` / `*Foreign` (read+write) | `readPackedInt(T, bytes, bit_offset, .native)` (or `.foreign`) | Unified API | -| `std.mem.Alignment` as `u29` int | `enum (Alignment)` — use `.fromByteUnits(n)` / `@enumFromInt` | No more `@as(u29, ...)` for alignment | -| `c_void` | `anyopaque` | C ABI | -| `usingnamespace` | (removed — no replacement; redesign with explicit re-exports) | Compile error in 0.16 | -| `@intToFloat(T, x)` / `@floatToInt(T, x)` | `@floatFromInt(x)` / `@intFromFloat(x)` | Result-location inferred | -| `@boolToInt(x)` | `@intFromBool(x)` | — | -| `@enumToInt(x)` / `@intToEnum(T, x)` | `@intFromEnum(x)` / `@enumFromInt(x)` | — | -| `@errToInt` / `@intToErr` | `@intFromError` / `@errorFromInt` | — | -| `@ptrToInt` / `@intToPtr` | `@intFromPtr` / `@ptrFromInt` | — | -| `@branch` | `@branchHint(.likely)` (or `.unlikely` / `.cold`); placed **inside** the branch body | — | -| Old `format(self, comptime fmt, options, writer: anytype)` | `pub fn format(self: @This(), w: *std.Io.Writer) std.Io.Writer.Error!void` | `{}` → `{f}` at call sites | -| `std.process.argsAlloc(alloc)` (manual main) | `pub fn main(init: std.process.Init)` then `init.minimal.args.iterateAllocator(gpa)` | "Juicy Main": `init` bundles `io / arena / gpa / minimal.args / environ_map / preopens` | - -## std.mem aliases that still work (informational) - -These remain canonical in 0.16, **don't migrate them**: - -- `std.mem.eql` / `startsWith` / `endsWith` / `trim` / `trimStart` / `trimEnd` -- `std.mem.splitScalar` / `splitAny` / `splitSequence` (NOT renamed) -- `std.mem.tokenizeScalar` / `tokenizeAny` / `tokenizeSequence` (NOT renamed) -- `std.mem.readInt(T, bytes, .little)` (or `.big`) / `writeInt` -- `@memcpy` / `@memset` / `@memmove` (builtins; replace `mem.copy`) - -## Empty `catch`: `catch {}` is the only form Zig 0.16 accepts - -Surprising but real. The Zig 0.16 compiler **rejects** the two -"more explicit" forms that AI training data favours: - -```zig -something() catch |_| {}; // ERROR: discard of error capture; omit it instead -something() catch |err| { _ = err; }; // ERROR: error set is discarded -something() catch {}; // OK -``` - -If a `catch {}` is genuinely the right pattern (best-effort I/O -where there is no recovery and nothing to log to), leave the bare -`catch {}` and add a comment above it explaining *why* swallowing -is fine. - -## Optionals: `x.?`, not `x orelse unreachable` - -The `.?` shorthand is the canonical "definitely-present optional" -in Zig 0.16. It triggers identical safety checks in safe build -modes, costs the same in release modes, and is shorter. - -```zig -const arg0 = arg_it.next().?; // OK (canonical) -const arg0 = arg_it.next() orelse unreachable; // gate-rejected (no_orelse_unreachable) -``` - -The lint chain enforces this (ADR-0003 / Phase B). - -## Exhaustive enum `switch`: prefer enumeration for narrow / fixed enums - -For small fixed enums (e.g. `Token.Kind` ~13 tags), enumerate every -tag instead of `else => ...`. When a new tag is added later, the -compiler raises a missing-case error at every switch — useful -regression prevention. - -```zig -return switch (token_kind) { - .lparen, .lbracket, .lbrace => .open, - .rparen, .rbracket, .rbrace => .close, - .integer, .float, .string, .symbol, .keyword => .literal, - .quote, .discard, .symbolic => .reader_macro, - .eof, .invalid => .terminal, -}; -``` - -For wide *value-dispatch* enums (`Value.Tag`, currently 36+ tags -and growing through Phases 4–15), the inverse rule applies: prefer -`else =>` to express "every other kind I do not accept as operand". -The arithmetic / collection / print primitives all use this idiom; -forcing 36-arm enumeration would balloon code volume without -preventing real bugs (a new tag is *meant* to fall into the error -path until its primitives are wired in). - -The lint gate (ADR-0003) does **not** enforce -`require_exhaustive_enum_switch` for this reason. Use judgement. - -## Empty function / `if` body: comment inside - -Empty bodies are gate-rejected unless they carry a comment -explaining the intent. - -```zig -fn nopOp(_: *Runtime, _: *const Node) anyerror!void { - // Phase-3 placeholder — body lands in Phase 4 with the VM. -} -``` - -The friction is the point: forces a sentence about intent at the -only moment the author is writing the code. - -## tagged union: `switch`, not `==` - -```zig -return switch (self) { .nil => true, else => false }; // OK -return self == .nil; // unreliable -``` - -Initialise with type annotation: `const nil: Value = .nil;` -(not `Value.nil`). - -## ArrayList / HashMap: `.empty` + per-call allocator - -```zig -var list: std.ArrayList(u8) = .empty; -defer list.deinit(allocator); -try list.append(allocator, 42); -const v = list.pop(); // returns ?T, not T -``` - -Same pattern for `HashMap`: `.empty`, `put(alloc, k, v)`, `deinit(alloc)`. - -The bare `std.ArrayList(T)` in 0.16 IS the old `ArrayListUnmanaged`. -The `Unmanaged` alias is itself deprecated — drop the suffix. - -## stdout via `std.Io.File` - -```zig -var stdout_buffer: [4096]u8 = undefined; -var stdout_writer = std.Io.File.stdout().writer(io, &stdout_buffer); -const stdout = &stdout_writer.interface; -try stdout.print("hello {s}\n", .{"world"}); -try stdout.flush(); // do not forget -``` - -`writer(io, buf)` requires `io` (a `std.Io` value) — get it from -`std.process.Init` (Juicy Main) or from `Runtime.io`. - -## `*std.Io.Writer` for writer params - -Type-erased writer; replaces `anytype` for writer parameters and avoids -"unable to resolve inferred error set" with recursion. For tests use -`var w: std.Io.Writer = .fixed(&buf);` then `w.buffered()`. - -Allocating writer (replaces `ArrayList(u8).writer().any()`): - -```zig -var aw: std.Io.Writer.Allocating = .init(allocator); -errdefer aw.deinit(); -try form.format(&aw.writer); -return aw.toOwnedSlice(); -``` - -## Mutex: `std.Thread.Mutex` is gone - -Replacements: - -- `std.Io.Mutex` — full blocking mutex; `lock`/`unlock` take an `io: Io` - argument, so the call site must already be threading `Io` through. -- `std.atomic.Mutex` — lock-free `tryLock` / `unlock` only (no blocking - `lock`). - -Phase 1–3 is single-threaded; prefer no mutex over a half-wired one. -Wire through `Runtime.io` when concurrency actually arrives (Phase 15). - -## `@branchHint` (not `@branch`) - -The hint goes inside the branch body: - -```zig -if (cond) { - @branchHint(.likely); -} else { - @branchHint(.unlikely); - return error.Fail; -} -``` - -## Custom format: `{f}`, not `{}` - -Types with a `format` method: `{}` raises "ambiguous format string". - -```zig -try w.print("{f}", .{my_value}); -``` - -## What the lint gate (ADR-0003) actually enforces - -`zig build lint -- --max-warnings 0` runs four rules: - -| Rule | What it catches | -|-------------------------|-------------------------------------------| -| `no_deprecated` | any stdlib `/// Deprecated:` reference | -| `no_orelse_unreachable` | `x orelse unreachable` instead of `x.?` | -| `no_empty_block` | empty `{}` body without an inside comment | -| `no_unused` | unused `const`, function, import | - -`require_exhaustive_enum_switch` was inspected and not adopted — -see ADR-0003 Update for the `Value.Tag` rationale. - -The gate is **Mac-host only** — `test/run_all.sh` skips it on -non-Darwin hosts so OrbStack / CI Linux do not need network reach -to fetch zlinter. - -## Test discovery via `@import` (lazy-decl-analysis trap) - -Zig 0.16 analyses top-level declarations **lazily**. A bare -`pub const X = @import("foo.zig");` does **not** pull `foo.zig`'s -`test {...}` blocks into the test set unless some code path -actually references `X` (a function call, a struct-field access, -even `_ = X.something;`). The import alone is not enough. - -Effect: a `.zig` file can sit in the source tree with passing -unit tests on disk, and `zig build test` silently skips them. -Worse, compile errors inside that file go undetected — the file -is never type-checked because nothing references it. - -Two known cw v1 cases (2026-05-25): the `runtime/regex/{compile, -match}.zig` skeletons shipped with 13 unit tests that never ran -(fixed by the cycle-1 commit). The `runtime/clock.zig` + -`runtime/time/instant.zig` impls referenced -`std.time.nanoTimestamp` — a function **removed in Zig 0.16** — -and the silent skip let the broken code sit on `main` for -multiple commits before surfacing (D-053). - -### The canonical fix: aggregator block in `src/main.zig` - -`src/main.zig` already carries a `test { _ = @import(...); ... }` -block whose sole job is to pull tests into the discovery graph -for files that have no production-path referrer yet: - -```zig -test { - _ = @import("runtime/value/value.zig"); - _ = @import("runtime/charset.zig"); - _ = @import("runtime/random.zig"); - // ... add a line whenever a new src/**/*.zig has test blocks - // but no other file references it yet. -} -``` - -When you land a new impl file before its surface / Clojure peer -exists, add the `_ = @import("...");` line to this aggregator in -the same commit. The line goes away (= the entry can be removed) -once a real production caller wires the module in. - -### Local alternative: `test { refAllDecls(X); }` - -Inside a file that **is** already in the test graph, an explicit -`test { @import("std").testing.refAllDecls(some_imported_module); -}` pulls every decl (incl. tests) of `some_imported_module` into -the analysis set. This is fine for surface/peer files; it's -**not** a substitute for the main-aggregator fix when the -container file is itself orphan. - -### Detection: `bash scripts/check_test_reach.sh` - -Runs in `test/run_all.sh` as `--gate`. Walks `@import` strings -from every root listed in the script's `test_roots` array -(currently just `src/main.zig`; extend when `build.zig` adds a -second `addTest()` site) to build a union reachable set, then -flags any `src/**/*.zig` that contains `^test ` blocks but is -not reachable. The script also sanity-warns if `build.zig`'s -`addTest()` count exceeds `test_roots`'s length so the gate -doesn't silently degrade when a test exe is added. - -### The broader pattern: phase-deferred scaffolds lose their homing path - -(Surfaced 2026-05-25 via the zwasm v2 reciprocal observation -at `private/notes/zwasm_v2_reciprocal_observations.md`.) - -The test-discovery trap is one *symptom* of a broader anti-pattern: -**phase-deferred scaffolds that lose their homing path**. The -test-orphan case is the easiest to detect (test never runs); the -compile-error orphan case (file references stale stdlib API that -never gets analysed) is the same shape but harder to detect (cw -v1's D-053 `std.time.nanoTimestamp` ride-along was exactly this). -Both root in: "we landed code expecting future wiring; future -wiring never came; code sits without a caller." - -The complementary detection layers cw v1 carries today: - -- **Test-presence orphan check** (`scripts/check_test_reach.sh`) - — high signal, allow-list-free, hard-fail gate. -- **Lazy decl analysis ride-along** — once a file enters the - test graph, latent compile errors against removed stdlib APIs - surface immediately. The check_test_reach gate is therefore - also a covert "is the code still buildable" gate for files - with no production-path caller. -- **Periodic debt review** — `Phase N target` / - `blocked-by: ` rows in `.dev/debt.yaml` that have been - blocked for ≥ 3 cycles get escalated at Step 0.5 sweep. - -When a new Phase-deferred scaffold lands, the author should: - -1. Put a `_ = @import("path/to/scaffold.zig");` line in - `src/main.zig`'s `test {}` aggregator block in the same - commit, so the file is in the test graph from day one - (covers both orphan symptoms above). -2. If the scaffold is genuinely too disruptive to compile until - the deferred phase lands (rare), file an explicit - `D-NNN blocked-by: Phase N landing` row in `.dev/debt.yaml` - so the deferral has a tracked owner — not a silent orphan. - -The aspirational-rule trap (cw v1's -`.claude/rules/.md` declaring an enforcement that has no -matching script) is the next-broader shape and is tracked -separately via `audit_scaffolding`. zwasm v2's -`audit_table_sync.sh` is the canonical "aspirational rule lifted -to a mechanical gate" pattern; replicate for any cw v1 rule that -declares enforcement without a backing script. - -## Variable shadowing - -Zig disallows locals that shadow struct method names. Rename the local. - -```zig -pub fn next(self: *Tokenizer) Token { - const next_char = self.peek(); // not `next` -} -``` - -## `comptime StaticStringMap` - -Zero-cost lookup at compile time. Use for keyword / opcode tables. - -```zig -const keywords = std.StaticStringMap(Keyword).initComptime(.{ - .{ "if", .if_kw }, - .{ "def", .def_kw }, -}); -``` - -## `ArenaAllocator` for phase-based memory - -Bulk-free at phase boundaries. No individual `free` calls. - -```zig -var arena = std.heap.ArenaAllocator.init(std.heap.page_allocator); -defer arena.deinit(); -const alloc = arena.allocator(); -``` - -## Doc comments - -- `//!` — module-level (top of file, before imports). ZLS hover on module. -- `///` — declaration-level (on `pub` types/fns/fields). -- `//` — inline notes (inside bodies only). - -Every file gets `//!`. Every `pub` gets `///` unless the name is -self-evident. No decorative banners (`// ---`). - -## `packed struct()` - -Bit-level layout, e.g. NaN-boxing tag bits: - -```zig -flags: packed struct(u8) { - marked: bool, - frozen: bool, - _pad: u6, -}; -``` - -## Juicy Main - -`pub fn main(init: std.process.Init)` receives `init.io` (`std.Io`), -`init.arena` (process-lifetime arena), `init.gpa` (thread-safe GPA), -`init.minimal.args`, `init.environ_map`, `init.preopens` in one bundle. -Use this signature; do not roll your own arg parsing for stdlib paths. - -## `extern struct` for ABI - -When laying out structures that cross language / Wasm boundaries, prefer -`extern struct` (C ABI) for top-level layout and `packed struct()` -for bit-precise sub-fields. diff --git a/.claude/rules/zone_deps.md b/.claude/rules/zone_deps.md deleted file mode 100644 index d24546fec..000000000 --- a/.claude/rules/zone_deps.md +++ /dev/null @@ -1,71 +0,0 @@ ---- -paths: - - "src/**/*.zig" - - "build.zig" ---- - -# Zone Dependency Rules - -Auto-loaded when editing Zig source. Authoritative version of the layering -contract in ROADMAP §4.1 / §A1. - -## Zone architecture - -``` -Layer 3: src/app/, src/main.zig -- CLI / REPL / nREPL / builder / pod - imports anything below -Layer 2: src/lang/ -- Primitives, Interop, Bootstrap - imports runtime/ + eval/ -Layer 1: src/eval/ -- Reader, Analyzer, Compiler, VM, TreeWalk - imports runtime/ only -Layer 0: src/runtime/ -- Value, Collections, GC, Env, Dispatch - imports nothing above -``` - -External Clojure libraries ship in-source under `src/lang/clj/clojure/` -alongside `clojure.core` (the top-level `modules/` reservation was retired -2026-07-01, D-095 — the co-located-module migration was never warranted). - -## NEVER: upward imports - -``` -runtime/ must NOT import from eval/, lang/, or app/ -eval/ must NOT import from lang/ or app/ -lang/ must NOT import from app/ -``` - -## When a lower zone needs to call a higher zone - -Use the **vtable pattern**: the lower zone declares the `VTable` type -(typically as a `struct` field on `Runtime`); the higher zone injects -function pointers at startup. - -```zig -// Layer 0 declares only the type -pub const VTable = struct { - callFn: *const fn(*Runtime, *Env, Value, []const Value) anyerror!Value, - expandMacro: *const fn(*Runtime, *Env, Value, []const Value) anyerror!Value, -}; - -// Layer 1 (or higher) installs the implementation at startup -runtime.vtable = .{ - .callFn = tree_walk.callFn, - .expandMacro = analyzer.expandMacro, -}; -``` - -This inverts the *compile-time* dependency direction while preserving the -logical call flow. - -## Enforcement - -`scripts/zone_check.sh` parses every `@import("…/foo.zig")` in the source -tree and flags upward-direction violations. - -- `bash scripts/zone_check.sh` — informational; always exits 0. -- `bash scripts/zone_check.sh --strict` — exit 1 on any violation. -- `bash scripts/zone_check.sh --gate` — exit 1 if violation count exceeds - the in-script BASELINE (currently 0). - -Tests are exempt: everything after the first `test "…"` line in a file is -skipped (test code may legitimately cross zones). diff --git a/.claude/settings.json b/.claude/settings.json index fdb2659a4..d9bec677d 100644 --- a/.claude/settings.json +++ b/.claude/settings.json @@ -1,7 +1,5 @@ { "defaultMode": "acceptEdits", - "outputStyle": "Japanese", - "model": "claude-opus-4-8", "permissions": { "allow": [ "Agent", @@ -63,7 +61,7 @@ "hooks": [ { "type": "command", - "command": "printf '=== gate cadence (dev efficiency — read BEFORE the first commit) ===\\nDo NOT run the full gate per commit (it is slow and kills iteration). Per commit: bash test/run_all.sh --smoke (~tens of sec; ADR-0107 — the smoke tier authorizes shared-code commits too, up to 5 before a forced full gate) and do not block. Batch the full bash test/run_all.sh --serial-e2e at the ceiling / phase boundary / pre-release (backgroundable as a look-ahead). If a full gate fails on ONLY a non-code check (e.g. check_debt_id_refs), fix it + re-smoke — do NOT re-run the whole full gate. SSOT: .claude/rules/gate_cadence.md + memory smoke_first_batch_full_gate.\\n\\n' && test -f $CLAUDE_PROJECT_DIR/.dev/handover.md && printf '=== .dev/handover.md ===\\n' && cat $CLAUDE_PROJECT_DIR/.dev/handover.md && printf '\\n=== git log -3 ===\\n' && git -C $CLAUDE_PROJECT_DIR log -3 --decorate --oneline; test -f $CLAUDE_PROJECT_DIR/.dev/.perf_campaign_active && test -f $CLAUDE_PROJECT_DIR/.dev/perf_campaign_essence.md && printf '\\n=== perf-campaign essence (§9.2.S — relentless lookahead, machine-injected) ===\\n' && cat $CLAUDE_PROJECT_DIR/.dev/perf_campaign_essence.md || true" + "command": "printf '=== git log -3 ===\\n' && git -C $CLAUDE_PROJECT_DIR log -3 --decorate --oneline; test -f $CLAUDE_PROJECT_DIR/.dev/.perf_campaign_active && test -f $CLAUDE_PROJECT_DIR/.dev/perf_campaign_essence.md && printf '\\n=== perf-campaign essence (\u00a79.2.S, machine-injected) ===\\n' && cat $CLAUDE_PROJECT_DIR/.dev/perf_campaign_essence.md || true" }, { "type": "command", @@ -72,17 +70,6 @@ ] } ], - "PostCompact": [ - { - "matcher": "auto|manual", - "hooks": [ - { - "type": "command", - "command": "bash $CLAUDE_PROJECT_DIR/scripts/print_handover_brief.sh" - } - ] - } - ], "PreToolUse": [ { "matcher": "Bash", @@ -131,10 +118,6 @@ { "type": "command", "command": "bash $CLAUDE_PROJECT_DIR/scripts/check_roadmap_amendment.sh" - }, - { - "type": "command", - "command": "bash $CLAUDE_PROJECT_DIR/scripts/check_handover_framing.sh" } ] } diff --git a/.claude/skills/code_learning_doc/TEMPLATE_TASK_NOTE.md b/.claude/skills/code_learning_doc/TEMPLATE_TASK_NOTE.md index fcb05408d..b44ff20d3 100644 --- a/.claude/skills/code_learning_doc/TEMPLATE_TASK_NOTE.md +++ b/.claude/skills/code_learning_doc/TEMPLATE_TASK_NOTE.md @@ -1,6 +1,6 @@ + Five minutes to fill in. English, per CLAUDE.md language policy. + No em-dashes, no padding. State the fact and stop. --> --- task: §9.X.Y @@ -11,49 +11,28 @@ files: - src/.zig --- -## 一行サマリ +## Summary - + -## 詰まったポイント +## What was non-obvious -1. <非自明な点 1> -2. <非自明な点 2> -3. <非自明な点 3> +1. +2. +3. -## 教科書との対比 +## Comparison against the textbooks - **cw v0** (git tag `v0.5.0`, via `git worktree add ../cw-v0 v0.5.0`): - + - **Clojure JVM / Babashka / Zig stdlib**: - <該当する場合のみ> + +- **DIVERGENCE**: -## 設計判断（却下した案） +## Design decisions -- 案 A: ... 却下理由: ... -- 案 B: ... 却下理由: ... +- -## 暫定ログ (this cycle) +## Worth citing later - - -- 導入: : ref [refs: D-NNN, feature_deps.yaml#] - 理由 1 行: ... -- 消化: : の marker 削除 (D-NNN close, feature_deps - entry status: provisional → landed) -- 想定外: : で新規暫定発見 → D-NNN row 起票 - -## 章を書くときに必ず触れるべき点 - -- <演習 1 候補> -- <演習 2 候補> -- -- <次章への伏線> - -## TODO（直後の task に持ち越す事項） - -- <あれば> +- diff --git a/.claude/skills/continue/SKILL.md b/.claude/skills/continue/SKILL.md index c35af4f2d..042bb986b 100644 --- a/.claude/skills/continue/SKILL.md +++ b/.claude/skills/continue/SKILL.md @@ -1,6 +1,6 @@ --- name: continue -description: Resume fully autonomous work on the `main` branch and drive the per-task TDD loop. Trigger when the user says 続けて, "resume", "pick up where we left off", "/continue", "次", "go", or starts a fresh session expecting prior context. `/continue` may also be re-invoked automatically (cron / wake hook); treat every invocation as the same go signal. The full step-by-step loop spec (Step 0 → 7) lives in CLAUDE.md § Autonomous Workflow + § The only stop (single condition: user explicit stop) and is loaded into every turn's system prompt; this skill is the invocation trigger and carries the resume procedure, the Phase-boundary review chain, the subagent delegation cheatsheet, and the model-selection guidance. +description: Resume fully autonomous work on the `staging` branch and drive the per-task TDD loop. Trigger when the user says 続けて, "resume", "pick up where we left off", "/continue", "次", "go", or starts a fresh session expecting prior context. `/continue` may also be re-invoked automatically (cron / wake hook); treat every invocation as the same go signal. The full step-by-step loop spec (Step 0 → 7) lives in CLAUDE.md § Autonomous Workflow + § The only stop (single condition: user explicit stop) and is loaded into every turn's system prompt; this skill is the invocation trigger and carries the resume procedure, the Phase-boundary review chain, the subagent delegation cheatsheet, and the model-selection guidance. --- # continue @@ -27,14 +27,17 @@ spec is not duplicated here — CLAUDE.md is the single source. ## Resume procedure (on every session pickup) -1. Read `.dev/handover.md` (SessionStart hook already prints it; - re-read). Pay attention to any "Guardrail refresh" section — - it points at recent principle.md / CLAUDE.md spirit edits the - loop must honour. -2. Read `CLAUDE.md` § Project spirit (top section, governs all - other rules) and `.dev/principle.md` (Bad Smell catalogue + - Structural imagination phase). These two are the meta layer - the per-task loop is checked against. +1. Run `project workflow catchup`. It drains the axioms and the + live kanban cards. There is no handover file: `.dev/handover.md` + was retired on 2026-09-11 (memory `20260910235746-74389f7f`), so + any older text that points at it, or at its "First task on + resume", is stale. +2. Read `CLAUDE.md` § Project spirit (governs all other rules) and + the smell sensors it indexes in hive memory (the `bad-smell` + tag, the premises `20260911004145-29ffd52a`, structural + imagination `20260911004146-720ff81e`). `.dev/principle.md` no + longer exists. These are the meta layer the per-task loop is + checked against. 2a. Read `.dev/project_facts.md` (user-declared invariants — F-001 … F-007 at 2026-05-24) and `.dev/structure_plan.md` (anticipated directory tree Phase 5-20). When a task touches @@ -43,16 +46,15 @@ spec is not duplicated here — CLAUDE.md is the single source. 3. Read `.dev/ROADMAP.md` **§9.0** (the gap-area model — the phase-queue/§9.-placeholder model is RETIRED per ADR-0142): the three gap areas' BUILT status + named gaps + draining - `D-NNN` rows. Next-unit selection is `handover.md`'s "First - task on resume" if concrete, else the `.dev/debt.yaml` - `active:` list EASIEST-FIRST (CLAUDE.md § When the active - work unit completes). + `D-NNN` rows. Next-unit selection is a concrete live kanban + card if catchup surfaced one, else the `.dev/debt.yaml` + `active:` list EASIEST-FIRST (memory `20260910001609-5b3ea4d7`). 4. (Dormant cadence per ADR-0025: skip the `docs/ja/learn_clojurewasm/` chapter check. `private/notes/` - per-task notes continue.) + per-task notes continue; they are gitignored and local-only.) 5. `git log --oneline -10` — read the recent commit chain so - the next task starts with the as-pushed reality, not the - handover's narration of it. + the next task starts with the as-pushed reality, not a + narration of it. 6. Confirm HEAD is green **without re-running the full gate** — the pushed HEAD was already gated when it landed. Check the `.dev/.gate_pass` / `.dev/.smoke_pass` fingerprint (or run a quick @@ -92,15 +94,15 @@ self-selected unit without asking**: - **Subagent B**: built-in `security-review` on unpushed commits. - **Subagent C**: write any outstanding chapter(s) for the - closed phase, pulling from `private/notes/` task-notes. + closed phase, pulling from `private/notes/` task-notes (local-only; a fresh clone has none). 3. Synthesise in main: 1 line per check + severity counts. No "shall I proceed?" question — proceed. 4. (Bench sweep retired 2026-06-11 — the `bench/quick.sh` / `bench/quick_baseline.txt` auto-baseline was removed from the gate, so no dangling samples accumulate. Perf is measured on demand via `bench/compare_langs.sh` / `bench/run_bench.sh`.) -5. Update `handover.md` to point at the next self-selected unit - (debt.yaml easiest-first). +5. Record the next self-selected unit (debt.yaml easiest-first) on + the kanban board; there is no handover file to update. 6. Proceed to that unit's Step 0. Auto-compaction handles context size transparently — no agent action needed. @@ -143,4 +145,4 @@ not checked against anything. A model-specific claim belongs where it can be re-measured, not in the resume procedure. If a long-context subagent underperforms, measure it on the models available at that moment and record the finding in -`.dev/lessons/` with its date; do not encode a model name here. +a hive memory entry with its date; do not encode a model name here. diff --git a/.dev/README.md b/.dev/README.md index 8a8ec5f55..a9715a3f9 100644 --- a/.dev/README.md +++ b/.dev/README.md @@ -2,38 +2,38 @@ Project-level design and operational metadata. Tracked in git. English. -## Always present (load-bearing) +## Load-bearing -- [`ROADMAP.md`](./ROADMAP.md) — **the** authoritative mission, principles, - architecture, phase plan, success criteria, and quality-gate timeline. - Single source of truth. If anything elsewhere disagrees with this file, - this file wins. -- [`handover.md`](./handover.md) — short, mutable, current session state. - Read at session start, updated at session end. **Framing discipline - enforced** via [`.claude/rules/handover_framing.md`](../.claude/rules/handover_framing.md) - (≤ 100 lines, no log accumulation, no forecast tables, no stop- - rationalisation phrases). -- [`cw_v0_parity_and_gap_plan.md`](./cw_v0_parity_and_gap_plan.md) — the - 2026-05-29 cw-v0-vs-cw-v1 parity snapshot + the plan for incorporating - v0's gaps into cw v1 (redesigned, not copied; per-gap ordering + - ROADMAP-amendment hooks). CLOSED 2026-07-02 (banner in the file; the - phase model is retired, ADR-0142). Historical reference only. -- [`orbstack_setup.md`](./orbstack_setup.md) — **retired** OrbStack x86_64 - gate setup (ADR-0049; kept for history). The live Linux gate is - [`ubuntunote_setup.md`](./ubuntunote_setup.md) + `scripts/run_remote_ubuntu.sh`. -- [`decisions/`](./decisions/) — Architectural Decision Records. - - `README.md` — convention. - - `0000_template.md` — copy this when adding a new ADR. - - `NNNN_.md` — accumulated decisions. +- [`ROADMAP.md`](./ROADMAP.md): the authoritative mission, principles, + architecture, plan and quality-gate timeline. If anything elsewhere + disagrees with it, it wins. +- [`project_facts.md`](./project_facts.md): the `F-NNN` invariants. Every + other document is edited to align with them; they are never amended by the + loop on its own. +- [`debt.yaml`](./debt.yaml): the row-level debt ledger, one testable barrier + per row. The live SSOT for technical debt. +- [`accepted_divergences.yaml`](./accepted_divergences.yaml): the `AD-NNN` + ledger of intentional divergences from JVM Clojure, each pinned by a test. +- [`optimizations.md`](./optimizations.md): the `O-NNN` performance ledger; + `PERF:` markers in source anchor each row. -## Created on demand (do NOT pre-create as empty stubs) +## Decisions -Empty files rot. Create them when they have real content, using the -templates in **ROADMAP §15.2**: +ADRs are cited by number (`ADR-NNNN`) in commit messages and in the ledgers +above. The record itself moved out of the tree on 2026-09-11 and lives in the +maintainer's knowledge base; numbers stay time-ordered and the newest wins on +conflict. -- `debt.yaml` — row-level debt ledger (ADR-0072; replaced the planned - `known_issues.md`). The live SSOT for technical debt. -- `data/compat_tiers.yaml` — per-namespace/class Clojure tier table. -- `data/placement.yaml` — GENERATED per-var placement index (ADR-0178; supersedes - the planned `status/vars.yaml` tracking, which was never built). Regenerate with - `scripts/gen_placement.sh`; the `placement_drift` gate keeps it honest. +## Reference material + +- `bench/`: the scaling probes behind `docs/works/collection_performance.md`. +- `gc_rooting.md`, `wasm_percall_findings.md`, `zwasm_capabilities.md`, + `perf_v0_baseline.md`, `perf_campaign_essence.md`: measured findings that + later work builds on. +- `mutation*`: the mutation-testing target list and equivalence records + (`scripts/mutation/`). +- `ubuntunote_setup.md` and `scripts/run_remote_ubuntu.sh`: the native Linux + gate over SSH. `orbstack_setup.md` is the retired predecessor, kept for + history. +- `archive/`, `ROADMAP_archive_phases_1-13.md`, `v0_v1_feature_parity.md`: + closed campaigns, historical only. diff --git a/.dev/ROADMAP.md b/.dev/ROADMAP.md index b7b70ee26..32c9e4e3e 100644 --- a/.dev/ROADMAP.md +++ b/.dev/ROADMAP.md @@ -71,7 +71,7 @@ per CLAUDE.md `§ When the active work unit completes`. and Wasm support, implemented in Zig 0.16.0.** - **No JVM**: single static binary within the ADR-0172 per-component size - budget (measured 7.55 MB at 2026-08-04, enforced ceiling + budget (measured 8.14 MB at 2026-09-14, enforced ceiling 8,800,000 bytes; the pre-engine "≤ 5 MB" target was retired when the embedded Wasm JIT engine — roughly 3 MB of code by itself , F-001 — landed), cold start ≤ 10 ms @@ -96,7 +96,7 @@ and Wasm support, implemented in Zig 0.16.0.** | # | Axis | Edge over the field | |---|------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| 1 | **Edge-native Clojure** | **AIM, NOT BUILT** . Babashka is native but produces no Wasm; SCI is JS-only; making a Wasm Component a first-class *output* of cljw would be the differentiator — and it does not exist. No `wasm32` target in `build.zig`; `cljw build` emits a native binary with a bytecode trailer. Owner: **D-552** (research-first, user-nod-gated). Sequenced behind axis 2 by F-010. Compiling *Clojure source* to a component is a sibling project's scope (ClojureWit), not a cljw plan. | +| 1 | **Edge-native Clojure** | **AIM, NOT BUILT** . Babashka is native but produces no Wasm; SCI is JS-only; making a Wasm Component a first-class *output* of cljw would be the differentiator — and it does not exist. `build.zig` DOES have a `wasm32` target since ADR-0193 (v1.11.0, `-Dtarget=wasm32-wasi`), but that produces a wasi *program* — a `_start` entry point a host executes — not a component that EXPORTS named functions a host can call. `cljw build` likewise emits a native binary with a bytecode trailer, and there is no library artifact at all: `build.zig` declares only `addExecutable`, and nothing in the tree is `export fn`. The verdict is unchanged; only the evidence for it moved. Owner: **D-552** (research-first, user-nod-gated). Sequenced behind axis 2 by F-010. Compiling *Clojure source* to a component is a sibling project's scope (ClojureWit), not a cljw plan. | | 2 | **Wasm-native interop** | `require` a Wasm **component** as a Clojure ns (finished form: **ADR-0135** + **Amendment 1**, 2026-06-21 user-directed — `(:require ["x.wasm" :as x])`, resolution order, type leverage, single-binary embed **ADR-0158**, always-latest **F-016**). **ACTIVE epic = D-404** (blocker dissolved; phases A ns-`:require` wiring → B resolution → C types → D single-binary → E resource/registry). Inversely, expose Clojure functions as WIT exports (later). | | 3 | **Comprehensible runtime** | Codebase is small enough to be read end-to-end. Each phase ships a written walkthrough. | | 4 | **Zig-level optimisation ceiling** | The *meta*-differentiator. Babashka rides GraalVM, clj rides the JVM: to make a hot path fast you must work *within* the host's object model / bytecode / GC. cljw is Zig from the metal up, so any primitive can be rewritten / fused / comptime-specialised (NaN-box, superinstructions, D-133 JIT) as a **Zig PR, not a host-engineering project** — a higher optimisation ceiling than any JVM-bound Clojure, provable per-workload. | @@ -249,11 +249,11 @@ its source ADR. | A19 | ARCHITECTURE.md is the 5-minute orientation | repo root `ARCHITECTURE.md` (ADR-0020) | | A20 | run_step is the runner dispatch pattern | `test/run_all.sh` `run_step` + summary (ADR-0024) | | A21 | Comptime conditional import + stub struct for phase staging | `runtime/X/stub.zig` parallels real X, `build_options.phase_at_least_N` switches (ADR-0023) | -| A22 | ADRs carry "Affected files" from 0020 onward | `.dev/decisions/0000_template.md` + every new ADR (ADR-0020) | -| A23 | Bad Smell sensor + four depths of revision | `.dev/principle.md` + `.claude/rules/plan_revision_thinking.md` (no ADR — meta layer above ADR / ROADMAP) | -| A24 | Autonomous workflow is direct sequential | `CLAUDE.md § Autonomous Workflow` (cw v0-style, in-loop reflexive re-read of principle.md at Steps 1 / 4 / 6) | -| A25 | Existing code is mutable; rewrite is part of design | Skeleton activation (e.g. Phase 5 activates `TypeDescriptor.lookupMethod` against Phase 1-3 primitives) and ADR `Supersedes` chains rewrite already-written `src/`, not just add. Apply via `.dev/principle.md` depth 1-4. `Phase N+ migration note` section in the entry ADR narrates the rewrite scope; ROADMAP §A4 (isolated subsystem) and §A11 (day-1 enum) pin slot boundaries, everything between the slots is mutable. No "stays additive" default. | -| A26 | cw-v0 gap incorporation is plan-driven (not v0-copy) | When opening Phase 15+ or minting quality-loop rows, consult [`.dev/cw_v0_parity_and_gap_plan.md`](./cw_v0_parity_and_gap_plan.md) — the 2026-05-29 parity snapshot + per-gap cw-v1-redesign direction + ordering + ROADMAP-amendment hook (user directive: incorporate most v0 gaps, but redesigned on cw v1's architecture with better design/ordering, never copied). Amend §9 in place per §17 referencing that doc. F-003 foresight — the owning phase decides the concrete design (Devil's-advocate at depth ≥ 2). Gaps with no current §9 row (`deps`/`test`/`--list-vars`/some namespaces) get NEW rows minted at incorporation. | +| A22 | ADRs carry "Affected files" from 0020 onward | every new decision entry carries its affected files (ADR-0020) | +| A23 | Bad Smell sensor + four depths of revision | memory `query :tags ["bad-smell"]` + `20260911004145-29ffd52a` (no ADR, meta layer above the rest) | +| A24 | Autonomous workflow is direct sequential | `CLAUDE.md § Autonomous Workflow` (cw v0-style, in-loop reflexive re-read of the smell corpus at Steps 1 / 4 / 6) | +| A25 | Existing code is mutable; rewrite is part of design | Skeleton activation (e.g. Phase 5 activates `TypeDescriptor.lookupMethod` against Phase 1-3 primitives) and ADR `Supersedes` chains rewrite already-written `src/`, not just add. Apply via the depth 1-4 recipe, memory `20260901165122-24182b0a`. `Phase N+ migration note` section in the entry ADR narrates the rewrite scope; ROADMAP §A4 (isolated subsystem) and §A11 (day-1 enum) pin slot boundaries, everything between the slots is mutable. No "stays additive" default. | +| A26 | cw-v0 gap incorporation is plan-driven (not v0-copy) | When opening Phase 15+ or minting quality-loop rows, consult memory `20260911004837-4f399769` (redesign-first, never a v0 copy) and `20260911004838-1e222586` (which v0 gaps are still unincorporated, verified 2026-09-11) (user directive: incorporate most v0 gaps, but redesigned on cw v1's architecture with better design/ordering, never copied). Amend §9 in place per §17 referencing that doc. F-003 foresight — the owning phase decides the concrete design (Devil's-advocate at depth ≥ 2). Gaps with no current §9 row (`deps`/`test`/`--list-vars`/some namespaces) get NEW rows minted at incorporation. | --- @@ -347,6 +347,10 @@ modules/ comptime-gated optional (math, c-ffi, wasm) imports runtime/ + eval/ only ``` +The optional modules landed under `src/runtime/cljw//` rather than a +top-level `modules/`: `wasm/` behind `-Dwasm` (ADR-0099) and `ffi/` behind +`-Dffi` (ADR-0202, on by default for x86_64 / aarch64 Linux and macOS). + When a lower zone needs to call an upper zone: vtable pattern. The lower zone declares the `VTable` type, the upper zone injects function pointers at startup. `scripts/zone_check.sh --gate` blocks any violation in CI. @@ -635,7 +639,7 @@ ClojureWasm/ (working dir on disk) │ ├── modules/ comptime-gated optional │ ├── math/ clojure.math -│ ├── c_ffi/ +│ ├── c_ffi/ landed as src/runtime/cljw/ffi/ (cljw.ffi, ADR-0202) │ └── wasm/ cljw.wasm namespace │ ├── test/ @@ -1040,8 +1044,8 @@ audit trail, not as a forward-looking instruction. | 15 | **Concurrency** — BUILT + race-hardened (gap area I; ADR-0142) | atom/ref+STM/agent/future/promise/delay/locking/volatile ship; gap = parity/load (R1) + D-442/D-105 (`:volatile-mutable` happens-before DONE 2026-06-16: ADR-0152) | 🔒 | | 16 | **Wasm/edge-native** BUILT (gap area II) · ClojureScript→JS = future | component build/run/require ship (`cljw.wasm/*`); CLJS→JS genuinely unbuilt | | | 17 | **VM perf: fusion → JIT** PARTIAL (gap area III) | superinstruction/fusion slice landed (D-386 / O-018/019/021/023); narrow ARM64 JIT = milestone M | | -| 18 | math + module/deps DONE · **C FFI** = future | `clojure.math` + deps.edn ship; C FFI (`dlopen`/libffi) genuinely unbuilt | | -| 19 | **Wasm/edge-native** cont. — WIT auto-binding (gap area II) | component require BUILT; gap = WIT marshalling (D-404) + zwasm integration shape (D-036/D-350) | | +| 18 | math + module/deps DONE · **C FFI** BUILT (register-only shape) | `clojure.math` + deps.edn ship; `cljw.ffi` dlopen + 6-int/8-double call shape, no libffi (ADR-0202); structs/varargs/callbacks unbuilt | | +| 19 | **Wasm/edge-native** cont. — WIT auto-binding (gap area II) | component require BUILT; gap = WIT marshalling (D-404); command/handle embedding shape settled by ADR-0124/D-350 | | | 20 | **broad JIT** = future (distal; gated on gap-area-III fusion outcome) | narrow ARM64 JIT (milestone M) is the near-term scope; broad JIT decided after fusion lands | | ### 9.0 Completion-grade gap-area model (ADR-0142 / F-015 / D-440) @@ -1065,7 +1069,7 @@ in ADRs / debt rows / overlays still resolve while R4/R5 rewrite them at source) | Phase 15 (§9.17) | **Gap area I — Concurrency hardening** (BUILT + race-hardened) | | Phase 16 (§9.18) | **Gap area II — Wasm/edge-native** (BUILT) · ClojureScript→JS = future bucket | | Phase 17 (§9.19) | **Gap area III — VM perf: fusion → JIT** (PARTIAL) | -| Phase 18 (§9.20) | math + module/deps **DONE** · C FFI = future bucket | +| Phase 18 (§9.20) | math + module/deps **DONE** · C FFI **BUILT** (`cljw.ffi`, ADR-0202) | | Phase 19 (§9.21) | **Gap area II — Wasm/edge-native** (WIT auto-binding; D-404) | | Phase 20 (§9.22) | **future bucket** — broad JIT (distal; narrow ARM64 JIT = milestone M) | @@ -1081,15 +1085,17 @@ in ADRs / debt rows / overlays still resolve while R4/R5 rewrite them at source) (gated, engine correct without them): D-244 #4a' auto-collect, D-245 Option C blocking monitor. - **(II) Wasm / edge-native** — `cljw.wasm/*` component build/run/require BUILT - (`src/runtime/cljw/wasm/`, embeds zwasm v2 per F-001). Gaps: WIT param marshalling - (**D-404**), zwasm integration finished-form (**D-036 / D-350 / D-039**). + (`src/runtime/cljw/wasm/`, embeds zwasm v2 per F-001). Gap: WIT param marshalling + (**D-404**). The core command/handle shape and I/O responsibility split are + settled (**ADR-0124 / D-350 / D-039**); D-036 retains only JIT coordination. - **(III) VM perf (fusion → JIT)** — arith/compare-branch superinstructions + reduce/lazy-seq fusion landed (**D-386 / O-018/019/021/023**). Gaps: remaining fusion surface; **narrow ARM64 JIT (F-010 milestone M)** → broad JIT go/no-go (distal, D-005 / D-035). **Genuinely-future bucket** (no impl; honest future): ClojureScript→JS compiler, -C FFI (`dlopen`/libffi), broad JIT (Phase 20). +broad JIT (Phase 20). C FFI left this bucket with `cljw.ffi` (ADR-0202); its +own future tail is structs by value, varargs and C-to-Clojure callbacks. > **Milestone M** (F-010) = concurrency complete (gap area I drained to parity) > + the cw-v0-level narrow ARM64 JIT. M is a **named milestone token**, not a @@ -1525,10 +1531,12 @@ surface}.zig`, embedding zwasm v2 (`@import("zwasm")`, F-001 — SHA-pinned dep, interp-only). e2e `phase16_wasm_{run,component,ffi,require_component}.sh` (real, build `-Dwasm`). Gated `-Dwasm` opt-in by design (lazy dep), NOT unimplemented. -**Gaps**: WIT param-type marshalling raises `feature_not_supported` -(`component.zig`) for some types → **D-404** (full WIT↔EDN table); zwasm -integration finished-form → **D-036 / D-350 / D-039** (responsibility split, -embedding shape). Entry facts: F-001 (zwasm unavoidable) · F-004 · F-006 · F-008 +**Gap**: WIT param-type marshalling raises `feature_not_supported` +(`component.zig`) for some types → **D-404** (full WIT↔EDN table). The integration +finished form is settled: persistent core-library handles, one-shot captured WASI +commands, and persistent WASI components (**ADR-0124 / D-350**); cw Tier-1 I/O +and zwasm WASI remain separate (**D-039**). D-036 retains only JIT coordination. +Entry facts: F-001 (zwasm unavoidable) · F-004 · F-006 · F-008 (zwasm v2 spec review). Consult zwasm `docs/zig_api_design.md` + `docs/consuming_prerelease_zwasm.md` (the SHA-pin procedure). @@ -1557,20 +1565,24 @@ an independent cljw-bytecode→native path (NOT via zwasm; F-001); bytecode ABI decoder-only-compatible (ADR-0033 D10 / ADR-0034 D6) so JIT go/no-go does not perturb placement/build. -### 9.20 math + module/deps DONE (formerly "Phase 18") · C FFI = future +### 9.20 math + module/deps DONE (formerly "Phase 18") · C FFI BUILT **Status: math + module/deps DONE.** `clojure.math` ships (`clojure/math.clj`, e2e `phase14_math*`); the deps.edn resolver + the `modules/` zone rule already -exist (Phase 14). **C FFI = genuinely-future bucket**: no `dlopen`/`dlsym`/ -`libffi`/`c_ffi`; `zig build -D` comptime-gated module builds are future -work, low priority. +exist (Phase 14). **C FFI BUILT** (ADR-0202): `cljw.ffi` (`open close sym +function call string bytes`) dlopens a library and calls any signature of at +most 6 integer-class and 8 double arguments through two function pointer +types, no libffi. Comptime-gated by `-Dffi` (default on for x86_64 / aarch64 +Linux and macOS, absent on wasm). The surface is shared with clojurust's +`clojure.rust.ffi`. Future tail: structs by value, varargs, float32, +callbacks into Clojure. ### 9.21 Gap area II (cont.) — WIT auto-binding (formerly "Phase 19") **Status: component require BUILT** (see §9.18). **Gap**: full WIT auto-binding (`(wasm/component "x.wasm")` → bindgen → Clojure namespace) — some WIT param -marshalling is fixture-blocked → **D-404**; the zwasm integration shape is -**D-036 / D-350**. This is the same Wasm/edge-native gap area as §9.18. +marshalling is fixture-blocked → **D-404**. The zwasm embedding shape is settled +by **ADR-0124 / D-350**. This is the same Wasm/edge-native gap area as §9.18. ### 9.22 broad JIT = future bucket (formerly "Phase 20"; distal) @@ -1609,7 +1621,7 @@ goes in just before Phase 4. Used during Phases 4-7. The binary-size row was a v0.1.0-era figure that predates the embedded zwasm JIT engine (roughly 3 MB of code by itself ). It is **superseded** by ADR-0172's -per-component ledger; the live figure is 7.37 MB at v1.5.1 +per-component ledger; the live figure is 8.14 MB at v1.14.7 against an enforced ceiling of 8,800,000 bytes, checked on every full gate by `scripts/binary_size_report.sh --check`. A target this table states and nothing measures is the exact rot the 2026-08-04 audit found in three other documents. @@ -1944,7 +1956,8 @@ points; this section enumerates every required file). - `.dev/handover.md` — current state, Active task, Next Phase Queue - `.dev/debt.yaml` — debt ledger (row-level predicates, per A13) - `.dev/reference_clones.md` — usage purpose of `additionalDirectories` -- `.dev/lessons/INDEX.md` — observational lessons (distinct from ADRs) +- observational lessons: hive memory + KG (the `.dev/lessons/` directory was + migrated into memory and deleted 2026-09-09); search, do not read a file - `.dev/orbstack_setup.md` — Linux x86_64 gate setup - `.dev/README.md` — index / convention pointer - `.dev/decisions/{README.md, 0000_template.md}` — ADR infrastructure @@ -2188,7 +2201,7 @@ ADRs progress through these statuses: ROADMAP. - **Superseded by ADR-NNNN** — replaced by a later ADR. - **Closed (Phase N DONE)** — phase boundary made the ADR irrelevant. -- **Demoted to .dev/lessons/** — observational learning only. +- **Demoted to a hive memory entry** (observational learning only; cite the id). Status changes are recorded in the ADR's `## Revision history` section. `scripts/check_adr_history.sh` (on-demand audit — not a wired diff --git a/.dev/accepted_divergences.yaml b/.dev/accepted_divergences.yaml index 6bfc21cfe..1b41b63ec 100644 --- a/.dev/accepted_divergences.yaml +++ b/.dev/accepted_divergences.yaml @@ -59,14 +59,18 @@ accepted: derives_from: "ADR-0059 (no-JVM rule); ADR-0174 (identity rule); no_jvm_specific_assumption.md" pin: "test/diff/clj_corpus/print_repr.txt + test/diff/clj_corpus/str_char.txt (native simple names) + test/e2e/phase14_host_class_identity.sh (surface FQCNs)" - id: "AD-004" - summary: "`(float x)` yields an f64; cljw has no f32 representation" - example: "(float 1/3) => cljw 0.3333333333333333 / clj 0.33333334 (f32)" + summary: "`(float x)` and `(.floatValue x)` yield an f64; cljw has no f32 representation" + example: "(float 1/3) => cljw 0.3333333333333333 / clj 0.33333334 (f32); (.floatValue 16777217) => cljw 1.6777217E7 / clj 1.6777216E7" why: |- F-005 fixes the numeric tower as a Zig-native single-double internal model. A distinct f32 would be a representation the tower does not - carry; `float` widens to the f64 the value already is. + carry; `float` widens to the f64 the value already is. The Number + method `.floatValue` answers the same f64 as `float`, not a value + rounded to f32 precision: clj has `(= (float x) (.floatValue x))`, + and narrowing in one of the two alone would break it (and print + `(.floatValue 0.1)` as 0.10000000149011612 where clj prints 0.1). derives_from: "F-005 (numeric tower: single internal double)" - pin: "none — value-dependent (the f64 is correct; only the f32-narrowing is absent)" + pin: "test/clj/suites/number_value_methods_test.clj (float-value-is-the-double-as-float-is); otherwise value-dependent (the f64 is correct; only the f32-narrowing is absent)" - id: "AD-005" summary: "Subnormal double shortest-rendering differs by the last digit" example: "smallest positive double => cljw 5.0E-324 / clj 4.9E-324 (the SAME f64 bit pattern)" @@ -135,7 +139,7 @@ accepted: ns-qualification: type-hash XOR mapHasheq, corpus records_method_values.txt). derives_from: "ADR-0059 (no-JVM: no second Object.hashCode system, no identity hashes) + F-011 (the observable `(hash x)` values now MATCH clj for data; the single-hash delegation is the internal-representation freedom) + AD-028 (the mapHash/mapHasheq sibling)." - pin: "test/diff/clj_corpus/hash_compare.txt — 30+ `(hash …)` golden values locked against real clj across every aligned type; test/diff/clj_corpus/object_methods.txt — `(= (.hashCode \"abc\") (hash \"abc\"))` → true (single-system delegation locked); test/e2e/phase14_format.sh — `%h` lowercase-hex rendering." + pin: "test/diff/clj_corpus/hash_compare.txt — 30+ `(hash …)` golden values locked against real clj across every aligned type; test/diff/clj_corpus/object_methods.txt — `(= (.hashCode \"abc\") (hash \"abc\"))` → true (single-system delegation locked); test/clj/suites/format_test.clj: the hash-conversion deftest pins `%h`/`%H` lowercase-hex rendering." - id: "AD-010" summary: "a Namespace value prints as `#object[Namespace \"\"]` (no JVM class FQN, no `0xADDR` identity hash)" example: '(pr-str *ns*) => cljw "#object[Namespace \"user\"]" / clj "#object[clojure.lang.Namespace 0x5ad10c1a \"user\"]" ; (str *ns*) => "user" (matches clj)' @@ -163,8 +167,7 @@ accepted: EDN within cljw (clj's own form does not round-trip either). cljw's `str` and `pr-str` of a queue are intentionally identical. ADR-0087. derives_from: "ADR-0059 (no-JVM identity hash); ADR-0087 (PersistentQueue)" - pin: "test/e2e/phase15_persistent_queue.sh — locks `(pr-str (conj EMPTY 1 2 3))` => `#queue (1 2 3)`, empty => `#queue ()`, and the `#queue` reader round-trip" - + pin: "test/clj/suites/persistent_queue_test.clj — locks `(pr-str (conj EMPTY 1 2 3))` => `#queue (1 2 3)`, empty => `#queue ()`, and the `#queue` reader round-trip" - id: "AD-013" summary: "cljw STM is retry-only — it drops JVM's `barge` (a running transaction preempting a younger conflicting one). Result-equivalent: serializability holds; only contention fairness/throughput differ (unobservable in the committed state)." why: |- @@ -182,7 +185,6 @@ accepted: the observable serializability contract the retry-only engine upholds. derives_from: "ADR-0090 §3 (MVCC retry-only LockingTransaction) + §4 (no-barge)" pin: "test/e2e/phase15_stm_dosync.sh — `dosync_concurrent_serializable` locks 4 threads × 100 `(dosync (alter c inc))` => exactly 400 (the serializable, no-lost-update result the retry-only engine produces under contention)" - - id: "AD-015" summary: "An ESCAPED `with-local-vars` local var, deref'd AFTER its dynamic extent, returns `nil` in cljw, where JVM returns an unreproducible `#object[clojure.lang.Var$Unbound 0x…]` opaque." example: '(var-get (with-local-vars [x 5] x)) => cljw nil / clj #object[Var$Unbound 0x…]' @@ -202,7 +204,6 @@ accepted: `nil` instead of an unreproducible opaque. derives_from: "F-006 (Vars gpa-owned + GC-membrane-filtered; no Unbound sentinel) + ADR-0097 (anon local Var lifetime) + the opaque-ref non-reproducibility class" pin: "test/e2e/phase15_with_local_vars.sh — `escape-nil` asserts `(var-get (with-local-vars [x 5] x))` => nil (cljw's safe divergent value, not JVM's Var$Unbound opaque)" - - id: "AD-016" summary: "`(biginteger x)` yields cljw's BigInt — prints `5N` (clj `5`) and `(class (biginteger 5))` is `BigInt` (clj `java.math.BigInteger`). cljw has no separate BigInteger type." example: '(biginteger 5) => cljw 5N / clj 5 ; (class (biginteger 5)) => cljw BigInt / clj java.math.BigInteger' @@ -223,8 +224,14 @@ accepted: round-trip, D-204) — so the honest finished form is the collapse + this recorded divergence. The class-NAME simple form (BigInt, not the FQCN) is additionally the AD-003 no-JVM-class surface. + The collapse also means a cljw BigInt answers the java.math.BigInteger + instance methods (bigint_methods.zig: `.gcd`, `.pow`, `.compareTo`, and + the rest), which clj's clojure.lang.BigInt does not have: `(.compareTo 5N + 6N)` is -1 in cljw and IllegalArgumentException ("No matching method + compareTo") in clj, while `(.compareTo (biginteger 5) (biginteger 6))` is + -1 in both. derives_from: "F-005 (numeric tower collapses BigInt/BigInteger into one `.big_int`) + AD-003 (simple class name, no JVM Class)" - pin: "test/e2e/phase14_biginteger.sh — biginteger_prints_N + biginteger_class_bigint lock cljw's `5N` print and BigInt class (the divergent values, not clj's 5 / java.math.BigInteger)" + pin: "test/clj/suites/biginteger_test.clj — the `biginteger-is-bigint-the-ad-016-divergence` deftest (cases biginteger_prints_N + biginteger_class_bigint) locks cljw's `5N` print and BigInt class (the divergent values, not clj's 5 / java.math.BigInteger)" - id: "AD-017" summary: "A deftype mutable field's `^long` / primitive type hint is parsed-and-ignored; cljw does not enforce clj's 'must assign primitive to primitive mutable'." example: '(deftype C [^:unsynchronized-mutable ^long n] P (go [this] (set! n "hi") n)) (go (->C 0)) => cljw "hi" / clj compile error (primitive mismatch)' @@ -240,6 +247,26 @@ accepted: `:volatile-mutable` keyword (assignability) is acted on. derives_from: "F-004 (NaN-box uniform 8-byte Value, no primitive field storage) + F-005 (no primitive specialization) + ADR-0104" pin: "test/e2e/phase14_deftype_mutable.sh — long_hint_ignored_accepts_non_long (a `^long` mutable field accepts a String set)" + - id: "AD-018" + summary: "quot/rem/mod whose QUOTIENT is non-finite — any NaN operand, or a +/-Inf dividend — yield a non-finite result (NaN or +/-Inf) in cljw; JVM Clojure THROWS NumberFormatException (a non-finite quotient routes through new BigDecimal). An Inf DIVISOR over a finite dividend has a finite (0) quotient and AGREES (both NaN)." + example: '(quot ##Inf 3) => cljw ##Inf / clj THROW; (rem ##Inf 3) => cljw ##NaN / clj THROW; (quot ##-Inf 3) => cljw ##-Inf / clj THROW; (mod ##NaN 3) => cljw ##NaN / clj THROW "Infinite or NaN"; (mod 3 ##Inf) => ##NaN in BOTH (Inf divisor, finite quotient — not a divergence)' + why: |- + F-005's single-f64 numeric tower runs quot/rem/mod on a float operand + through one clean IEEE path (@trunc of the f64 quotient), so a NaN in + yields NaN out uniformly. JVM Clojure's DoubleOps.remainder casts the + quotient to long only when it is in Long range; a NaN quotient fails + that range test and falls to new BigDecimal(q), which throws + NumberFormatException on NaN/Infinity. cljw does not reproduce that + long-cast-BigDecimal fallback, so it returns NaN instead of throwing — + the same "cljw's cleaner numeric surface vs clj's throw" shape as + AD-008. quot/rem already returned NaN; fixing CLJW-MOD-PANIC (numSign's + float arm hit std.math.order's unreachable on NaN) made mod join them. + The rule is the QUOTIENT's finiteness: a +/-Inf dividend also yields a + non-finite quotient so clj throws there too, while an Inf divisor over a + finite dividend gives a 0 quotient and both runtimes return NaN (verified + 2026-08-17 against the clj oracle). + derives_from: "F-005 (numeric tower: single internal f64, one clean float quot/rem/mod path) + AD-008 (cljw's cleaner numeric surface where clj throws)" + pin: "test/clj/suites/numeric_tower_test.clj — the `non-finite-operands` deftest: quot_nan/rem_nan/mod_nan_d/mod_nan_v (NaN family) + quot_inf/rem_inf/mod_inf/quot_ninf (+/-Inf-dividend family) lock cljw's non-finite result (clj throws). Each assertion carries its original case name as a trailing comment." - id: "AD-019" summary: "Java arrays are type-erased: element type hints (`^\"[B\"`, `^ints`, `^bytes`) are advisory only, and a typed-array constructor / `aset` does not coerce or type-check non-byte/short/char elements." example: '(let [^"[B" b (byte-array 1)] (aset b 0 65) (aget b 0)) => 65 (a non-byte stored into a byte-typed array, no error); (class (object-array 0)) => cljw "array" / clj "[Ljava.lang.Object;"' @@ -257,7 +284,7 @@ accepted: those are NOT divergences. The simple class name (`array`, not the JVM array-class encoding) is the AD-003 no-JVM-class surface applied to arrays. derives_from: "F-004 (uniform Value) + F-005 (no primitive specialization) + ADR-0059 / AD-003 (no JVM class) + ADR-0105" - pin: "test/e2e/phase14_arrays.sh — type_hint_advisory (a ^\"[B\" array accepts a non-byte) + array_class_simple (class is `array`)" + pin: "test/clj/suites/arrays_test.clj — type_hint_advisory (a ^\"[B\" array accepts a non-byte) + array_class_simple (class is `array`)" - id: "AD-020" summary: "A java.util.Random (any host_instance) prints opaquely as `#` — clj prints `#object[java.util.Random 0x… …]` with a non-reproducible identity hash." example: '(pr-str (java.util.Random. 0)) => cljw "#" / clj "#object[java.util.Random 0x.. java.util.Random@..]"' @@ -271,7 +298,7 @@ accepted: non-reproducible print identity diverges. `(class (java.util.Random.))` => `java.util.Random` (simple, not the JVM Class object) is the AD-003 surface. derives_from: "AD-002 (opaque-ref identity-hash print cljw cannot mirror) + ADR-0059 (no JVM Class) + ADR-0106" - pin: "test/e2e/phase14_random.sh — print_opaque (#) + class (java.util.Random)" + pin: "test/clj/suites/random_test.clj — print_opaque (#) + class (java.util.Random)" - id: "AD-021" summary: "A namespace VALUE (e.g. `:ns` in `(meta #'x)`) prints opaquely as `#object[Namespace \"name\"]` — simple class name, no identity hash — vs clj's `#object[clojure.lang.Namespace 0x… \"name\"]`." example: '(pr-str (:ns (meta #''map))) => cljw "#object[Namespace \"clojure.core\"]" / clj "#object[clojure.lang.Namespace 0x.. \"clojure.core\"]"' @@ -446,7 +473,7 @@ accepted: value clj would refuse). The error KIND also differs (cljw type_error vs clj IllegalFormatConversionException) per AD-007. derives_from: "F-005 (numeric tower: int/Long collapse — cljw has no distinct Integer, so the Integer-accepts/Long-rejects split is unreproducible) + AD-007 (error Kind, not the JVM exception class)" - pin: "test/e2e/phase14_format_char_int.sh — format_c_int_rejected + format_c_long_rejected lock cljw's integer rejection; format_c_char locks the char-arg parity" + pin: "test/clj/suites/format_test.clj: the character-conversions deftest locks cljw's `%c` integer and long rejection and the char-arg parity" - id: "AD-031" summary: "Ratio components `(numerator r)` / `(denominator r)` (and integer results derived from them, e.g. `(clojure.math.numeric-tower/floor 5/2)` / `(ceil 5/2)`) narrow to Long when they fit i64; clj returns a BigInteger (numerator/denominator) / BigInt (floor/ceil). VALUE is `=` in both; only the class NAME + `N`-suffix print diverge." example: '(numerator 5/2) => cljw 5 (Long) / clj 5 (java.math.BigInteger) ; (clojure.math.numeric-tower/floor 5/2) => cljw 2 / clj 2N ; (ceil 5/2) => cljw 3 / clj 3N ; (= 2 2N) => true in both' @@ -563,24 +590,7 @@ accepted: is `lazy_seq` diverges; `pr-str`/`prn`/`print` and `str` of any realized seq match clj. derives_from: "AD-002 (non-reproducible JVM identity hash is not mirrored) + ADR-0054 (the print path realizes a lazy seq to its elements). clj's LazySeq.toString is the un-overridden Object.toString; no cljw analogue exists." - pin: "test/e2e/phase14_lazy_map.sh — `str_lazyseq_elements` locks (str (map inc [1 2 3])) => \"(2 3 4)\" and `str_interpose_lazy` locks (str (interpose :x (range 3))) => \"(0 :x 1 :x 2)\" (cljw's deterministic element form, vs clj's identity-hash toString)" - - id: "AD-039" - summary: "The `#(…)` anonymous-fn reader expands with DETERMINISTIC params `%1`/`%2`/`%&`; clj uses non-reproducible gensyms (`p1__N#` / `rest__N#`)." - example: '(read-string "#(+ % 1)") => cljw "(fn* [%1] (+ %1 1))" / clj "(fn* [p1__3#] (+ p1__3# 1))" ; (read-string "#(apply + % %&)") => cljw "(fn* [%1 & %&] (apply + %1 %&))" / clj "(fn* [p1__10# & rest__11#] …)". The CALL behaviour matches: (#(+ % 1) 10) => 11 both.' - why: |- - clj gensyms the `#()` params (`p1__N#`, counter-suffixed) so a literal `%` - in the body cannot collide with an outer binding; the `N` is a process-wide - counter, so the printed form is non-reproducible (like AD-002's identity - hash). cljw canonicalises `%`→`%1`, `%N` positional, `%&` rest — a fixed, - deterministic naming with no counter. Functionally identical (arity + body - reference the same params); the only observable difference is the printed - param NAMES, and only via `(read-string "#(…)")` + printing the expansion - (a macro-author edge). cljw's deterministic form is the right choice: a - counter would be non-reproducible for no behavioural gain, and `%N` reads - naturally as "the Nth arg". Nested `#()` is a read error in BOTH (so `%` - can't span levels — the only real capture hazard the gensym guards). - derives_from: "AD-002 (non-reproducible JVM-side token — here a gensym counter — is not mirrored; cljw uses a deterministic name). The fixed `%N`/`%&` naming is sound because nested `#()` is rejected (no `%`-across-levels capture to guard)." - pin: "test/e2e/phase14_anon_fn_reader.sh — `anon_readform_pct1` locks (read-string \"#(+ % 1)\") => (fn* [%1] (+ %1 1)) and `anon_readform_rest` locks the `%&` rest form (cljw's deterministic params vs clj's gensyms)" + pin: "test/clj/suites/lazy_map_test.clj — `str_lazyseq_elements` locks (str (map inc [1 2 3])) => \"(2 3 4)\" and `str_interpose_lazy` locks (str (interpose :x (range 3))) => \"(0 :x 1 :x 2)\" (cljw's deterministic element form, vs clj's identity-hash toString)" - id: "AD-040" summary: "A cascading macro (`cond`, and similar) macroexpand-1's to the FULL nested form in cljw, where clj expands ONE clause and leaves a recursive `(clojure.core/cond …)` call. Functionally identical; the expanded form (and the nested macro's namespace) differ." example: "(macroexpand-1 '(cond a 1 :else 2)) => cljw \"(if a 1 (if :else 2 nil))\" / clj \"(if a 1 (clojure.core/cond :else 2))\". Running the macro is identical: (cond false 1 :else 2) => 2 both. NOTE: `when` DOES byte-match clj (`(if c (do body))`, fixed) — only the recursive-cascade macros (cond/condp/…) differ." @@ -597,7 +607,7 @@ accepted: `when`/`if-not`/`when-not` forms ARE made to byte-match clj; only the recursive-cascade cases keep cljw's full-expand shape. derives_from: "F-011 (behavioural equivalence is the contract; the macros run identically — only the intermediate macroexpand-1 FORM differs; cljw's Zig transforms lower eagerly in one step). Functionally-equivalent expansion form, like ClojureScript's macroexpansions vs JVM clj." - pin: "test/e2e/phase14_when_if_not.sh — `cond_expand_full` locks (macroexpand-1 '(cond a 1 :else 2)) => (if a 1 (if :else 2 nil)) (cljw's full form), and `when_expand_multi`/`when_expand_single` lock when's byte-match with clj" + pin: "test/clj/suites/control_macros_test.clj — `cond-expands-to-the-full-nested-if` locks (macroexpand-1 '(cond a 1 :else 2)) => (if a 1 (if :else 2 nil)) (cljw's full form), and `when-expands-like-clj`/`when-not-expands-like-clj` lock when's and when-not's byte-match with clj" - id: "AD-041" summary: "clojure.test's FAIL/ERROR ` (file:line)` suffix points at the DEFTEST var's line (from Var source meta, D-563b) where clj points at the failing assertion's line (JVM stack frame); and `report :error` prints the exception's value instead of clj's JVM cause trace. Names, expected/actual, context, and counts all match clj." example: "(deftest a (is (= 1 2))) in f.clj: cljw `FAIL in (a) (f.clj:LINE-OF-DEFTEST)` / clj `FAIL in (a) (f.clj:LINE-OF-IS)`. An erroring test: cljw ` actual: #error {…}` (the ex value) / clj a multi-line `clojure.lang.ExceptionInfo … at …` cause trace." @@ -666,7 +676,7 @@ accepted: accepted divergence. `displayClassName` (runtime/class_name.zig) is the SSOT; the names are NOT `instance?` targets (no `clojure.lang.*` FQCN to resolve). derives_from: "ADR-0059 (no JVM Class) + AD-003 (simple class name, no JVM FQCN) — extended to callables/seq-views/refs which clj backs with generated/reify classes." - pin: "test/e2e/phase14_class_type.sh — class_fn_anon/class_fn_builtin => Fn, class_multifn => MultiFn, class_lazyseq/range/chunkseq/future/agent/atom/ref/volatile/delay/reduced/tvec, and class_fn_isa_ifn (the IFn isa? still resolves through the renamed callable classes)." + pin: "test/clj/suites/class_type_test.clj — the `class-of-callables-and-refs` deftest: anon fn/builtin => Fn, multifn => MultiFn, lazyseq/range/chunkseq/future/agent/atom/ref/volatile/delay/reduced/tvec, and the class-level `(isa? (class +) clojure.lang.IFn)` (the IFn isa? still resolves through the renamed callable classes)." - id: "AD-045" summary: "`send-via` / `set-agent-send-executor!` / `set-agent-send-off-executor!` RAISE (`value_error`) instead of dispatching on / configuring a user-supplied executor — cljw has no `java.util.concurrent.ExecutorService` type, so there is no executor to pass or configure. clj runs these for real against its global agent thread pools." example: '(send-via my-executor a inc) / (set-agent-send-executor! my-executor) => cljw raises "ClojureWasm has no configurable executor — agents run on a per-agent worker …" / clj dispatches-via / mutates the pool.' @@ -751,7 +761,7 @@ accepted: java.util collection str renders cljw-content, not JVM toString); both stem from "cljw does not reproduce JVM Object/Abstract* toString surface details". derives_from: "ADR-0059 (no JVM — no class FQN, no Object.toString identity form) + AD-002 (non-reproducible identity hash) + AD-003 (cljw renders the record with its simple name, not the JVM FQCN) + F-011 (pr-str already matches modulo the AD-003 simple name; only the JVM-Object.toString str-form diverges)." - pin: "test/e2e/phase7_defrecord.sh — defrecord_str_content_form (str of a record = its #Name{...} content form, not Class@hash) + defrecord_print_map_style (the pr-form it mirrors)." + pin: "test/clj/suites/defrecord_test.clj: defrecord-str-content-form (str of a record = its #Name{...} content form, not Class@hash) + defrecord-print-map-style (the pr-form it mirrors). Repointed 2026-09-09 when the phase7 defrecord e2e migrated to this native suite (the pin-path check extracts EVERY path in this field, so the retired shell must not be named here). The suite asserts BOTH forms against one derived expectation, so str and pr cannot drift apart unnoticed." - id: "AD-050" summary: "A float-zero DIVISOR yields IEEE ±Inf/NaN on EVERY cljw `/` path; clj returns ±Inf only on its compile-inlined 2-arg literal path and THROWS `ArithmeticException: Divide by zero` on the runtime path (1-arg `(/ 0.0)`, `(apply / …)`, `(reduce / …)`). An INTEGER-zero divisor throws in BOTH (parity). VALUE-bearing common case `(/ x 0.0)` matches clj (±Inf)." example: '(/ 1.0 0.0) => ##Inf in BOTH ; (/ 0.0) => cljw ##Inf / clj THROW ; (apply / [1.0 0.0]) => cljw ##Inf / clj THROW ; (reduce / [1.0 2.0 0.0]) => cljw ##Inf / clj THROW ; (/ 1 0) => THROW in both' @@ -812,7 +822,7 @@ accepted: 2026-07-07. Name conversions (%ta %tA %tb %tB %tp) match an English-locale JVM. derives_from: "ADR-0059 no-JVM (no host tz/locale database dependency) + F-011 (the deterministic-environment rendering matches TZ=UTC/en JVM exactly; only the host-env-dependent rendering diverges) + D-462 (tz database out of scope for the edge/wasm target)." - pin: "test/e2e/phase14_format.sh — the `%t` cases pin the UTC/English renderings (epoch-0 hour 00, Fri/Feb names, +0000 zone)." + pin: "test/clj/suites/format_test.clj: the `date-time-family` deftest pins the `%t` UTC/English renderings (epoch-0 hour 00, Thu/Jan names, +0000/UTC zone)." - id: "AD-053" summary: "`(all-ns)` / `(find-ns 'cljw.internal)` expose the kernel-helper namespace `cljw.internal` (the home of the `__`-prefixed Zig primitives macro expansions call qualified); mainline Clojure has no such namespace." example: "(some? (find-ns 'cljw.internal)) => cljw true / clj false (no such ns). (contains? (ns-publics 'clojure.core) '__class) => false BOTH (internals never pollute the core surface)." @@ -872,7 +882,7 @@ accepted: derives_from: "ADR-0176 (live-worker teardown guard; the JVM-exact wait-for-futures flip enumerated + rejected there as a separate product decision) + ADR-0059 no-JVM (no executor pools) + ADR-0174 D6 (the daemon hard-exit idiom ADR-0176 generalises)." pin: "test/e2e/phase14_future_promise_delay.sh — `exit_pending_future` (a 60s-sleeper future + prompt :ok exit, bounded)." - id: "AD-057" - summary: "clojure.core publishes 104 public names JVM Clojure does not: protocol names (ISeq, ILookup, Seqable, …), their `-name` methods (-first, -count, -assoc, …), and a few cljw-original conveniences (array?, queue?, re-find-all, …)." + summary: "clojure.core publishes 107 public names JVM Clojure does not: protocol names (ISeq, ILookup, Seqable, …), their `-name` methods (-first, -count, -assoc, -assoc-ex, …), and a few cljw-original conveniences (array?, queue?, re-find-all, …). The count is data/core_surface_extras.txt, which check_core_surface.sh gates; this prose had drifted to 104 and is corrected to match." example: "(contains? (set (map name (keys (ns-publics 'clojure.core)))) \"ISeq\") => cljw true / clj false. `clojure.lang.ISeq` is a Java INTERFACE upstream, not a clojure.core var." why: |- cljw has no Java, so the `clojure.lang.*` interfaces a JVM `deftype` @@ -967,3 +977,245 @@ accepted: example: '(re-pattern "(a)\\1") => cljw UNCATCHABLE "this regex feature … is not supported" / clj #"(a)\1" compiles and (re-matches #"(a)\1" "aa") => ["aa" "a"]' derives_from: "ADR-0031 (Pike-NFA, non-backtracking invariant — linear-time matching is the security property, F-002 finished form)." pin: "test/e2e/regex_parity_gaps.sh — backref compile raises the uncatchable unsupported signal and a (catch Throwable …) does NOT swallow it." + - id: "AD-061" + area: "concurrency" + summary: |- + `java.util.concurrent.Semaphore`'s `fair?` constructor flag is recorded + and reported by `.isFair`, but acquisition ALWAYS barges: a waiter takes + the permit it can CAS, in no particular order. The JVM's fair mode + guarantees FIFO hand-off through an AQS wait queue; cljw's permits live + in one atomic counter and a parked caller polls it through the budgeted + sleep that `Thread/sleep` owns (so future-cancel and the eval budget cut + the wait short). Building a FIFO queue would mean a second parking + mechanism whose abandonment path — a timed `tryAcquire` giving up mid- + queue — is the part that goes wrong quietly. + The flag is kept rather than rejected because the constructor arity is + what real code writes (`(Semaphore. n fair?)`), and `.isFair` answering + truthfully lets a caller detect the difference. Throughput and mutual + exclusion are unaffected; only starvation-freedom under sustained + contention is. + example: '(let [s (java.util.concurrent.Semaphore. 1 true)] (.isFair s)) => true, yet a thread that has waited longer has no priority over a newcomer, unlike the JVM.' + derives_from: "ADR-0029 (host surface registry) + ADR-0106 (host_instance state) + the budgeted-sleep wait Thread/sleep owns (D-442 / D-571)." + pin: "test/e2e/semaphore.sh — isFair_records_flag (the flag round-trips) + queue_length_while_parked (a parked waiter is counted, and the permit is handed over)." + - id: "AD-062" + area: "concurrency" + summary: |- + `java.util.concurrent.atomic.AtomicInteger` does NOT truncate to 32 bits. + cljw has one integer width, so AtomicInteger and AtomicLong share a + backing impl over the same `host_instance` word: `.incrementAndGet` at + `Integer/MAX_VALUE` yields 2147483648 where the JVM wraps to + `Integer/MIN_VALUE`. Arithmetic saturates at the i64 ends rather than + wrapping, matching what cljw's own integer ops already do. + Emulating 32-bit wraparound would mean introducing a narrower integer + cljw has nowhere else, so a value read back out of the atomic would not + behave like any other cljw integer. Overflow-dependent code is rare; + code that reads a counter is not. + example: '(let [a (java.util.concurrent.atomic.AtomicInteger. Integer/MAX_VALUE)] (.incrementAndGet a)) => cljw 2147483648 / clj -2147483648' + derives_from: "ADR-0059 (no JVM) + cljw's single i64 integer width (D-165 Long/BigInt origins)." + pin: "test/e2e/concurrent_atomics.sh — atomicinteger_no_32bit_truncation." + - id: "AD-063" + area: "runtime" + summary: |- + `(.totalMemory r)` / `(.freeMemory r)` / `(.maxMemory r)` answer from + cljw's own mark-sweep accounting, not from a JVM heap that does not + exist. `totalMemory` is the live set or the adaptive collect threshold, + whichever is larger (cljw keeps no reserved-but-unused pool, so the + threshold is the honest analogue of committed heap); `freeMemory` is the + headroom between them; `maxMemory` is the per-eval ceiling + `CLJW_EVAL_MAX_HEAP_MB` sets (ADR-0125), or `Long/MAX_VALUE` when the + heap is unmetered. + The consequence worth stating: on an unmetered cljw a used/max heap- + pressure ratio reads ~0 and a pressure monitor never fires. That is + TRUE — an unmetered heap has no ceiling to be pressured against — and it + is why the reading is wired to the real ceiling rather than to a + fabricated constant, which would make the same monitor fire on numbers + that mean nothing. + example: 'CLJW_EVAL_MAX_HEAP_MB unset: (.maxMemory (Runtime/getRuntime)) => 9223372036854775807 (no ceiling) / with CLJW_EVAL_MAX_HEAP_MB=64 => 67108864, and the pressure ratio becomes real.' + derives_from: "ADR-0028 (mark-sweep + adaptive threshold) + ADR-0125 / D-352 (per-eval heap ceiling) + ADR-0059 (no JVM)." + pin: "test/e2e/concurrent_atomics.sh — runtime_memory_is_coherent + runtime_maxmemory_unmetered_is_exact_long." + - id: "AD-064" + area: "concurrency" + summary: |- + `(.drainTo q coll)` RETURNS the drained elements as a cljw vector and + empties the queue; it does not add them to `coll`, and the second + argument is accepted-and-ignored. On the JVM `drainTo` mutates the target + collection in place and returns the transferred count. + cljw's collections are persistent — there is no mutable target to drain + INTO, so the JVM signature has no faithful reading here. Returning the + elements is the only shape that loses no information: a caller writes + `(let [xs (.drainTo q [])] …)` instead of reading `coll` afterwards. The + arity is kept so the JVM call shape still compiles. + `.remove` follows the JVM instead of diverging: the 1-arity raises on an + empty queue (`.poll` is the nil-returning form) and the 2-arity drops the + first equal element and reports whether one was found. + example: '(let [q (java.util.concurrent.LinkedBlockingQueue. 8)] (.offer q 1) (.drainTo q [])) => cljw [1] / clj 1, with the element pushed into the (mutable) target collection.' + derives_from: "ADR-0059 (no JVM) + cljw persistent collections (no mutable drain target)." + pin: "test/e2e/linked_blocking_queue.sh — drainTo_returns_and_empties." + - id: "AD-065" + area: "concurrency" + summary: |- + `(.cancel f mayInterruptIfRunning)` ACCEPTS the flag and IGNORES it: both + `true` and `false` behave as the JVM's `cancel(true)` would, and the + 1-arity `(.cancel f)` is accepted too. + cljw's cancellation is cooperative (D-442 / ADR-0153): `future-cancel` + moves a still-pending future to `.cancelled` and wakes it, and a blocking + primitive inside the thunk (`Thread/sleep`, a Semaphore wait) polls + `cancelRequested` and aborts. A thunk in a tight CPU loop with no blocking + point runs to completion either way — which is also what the JVM's + `cancel(true)` does in practice, since it interrupts rather than kills. + So the flag's JVM meaning ("may I interrupt a RUNNING task, or only + cancel a queued one?") has no distinct implementation here: there is no + queue-vs-running distinction to honour and no interrupt to withhold. + Ignoring it is more honest than picking one of the two values to respect, + which would make `false` silently mean something it does not. + `.cancel` still reports truthfully: it returns true only when THIS call is + the one that cancelled a still-pending future, false for a future that + already realised or was already cancelled — the JVM's contract. + example: '(let [f (future (Thread/sleep 5000))] [(.cancel f false) (.isCancelled f)]) => cljw [true true] / clj [false false] — the JVM refuses to cancel a running task when the flag is false.' + derives_from: "D-442 / ADR-0153 (cooperative cancellation) + ADR-0059 (no JVM thread interrupt)." + pin: "test/clj/suites/future_methods_test.clj — cancel_flag_is_ignored_ad065." + - id: "AD-066" + area: "collections" + summary: |- + `(class (seq v))` on a vector answers `ArraySeq`, where clj answers + `clojure.lang.PersistentVector$ChunkedSeq`. + cljw has ONE indexed-view seq representation (`.array_seq`, + `runtime/collection/array_seq.zig`, O-058) standing in for the family of + indexed seqs clj gives separate classes: `PersistentVector$ChunkedSeq` + over a vector and `ArraySeq` over a Java array are the same two words + here — a backing collection plus a cursor — read through the + `backingCount` / `elementAt` switches, so a second backing is a new arm + rather than a new type. This is the same collapse F-005 performs on the + numeric tower (BigInt and BigInteger become one `.big_int`, whose + class-name consequence is already recorded above), applied to seqs. + The name kept is the reserved day-1 name for the A14 slot. Only the class + NAME diverges: `seq?` / `sequential?` / `coll?` / `counted?` / `=` / + `hash` / print form / `nth` / `count` all match clj, and the value is `=` + to both the list and the vector with the same elements, so nothing that + treats it AS a seq can tell the difference. + Before O-058 this expression answered `PersistentList`, because `seq` on + a vector eagerly copied into one. Neither the old nor the new answer + matches clj, so no code that was portable has been broken. + example: '(class (seq [1 2 3])) => cljw ArraySeq / clj clojure.lang.PersistentVector$ChunkedSeq. Also `(class (seq (into-array [1 2])))` would be ArraySeq in BOTH — the one backing for which the shared name is exact.' + derives_from: "AD-003 (simple class name for cljw-native types) + F-004 A14 slot reservation + the F-005 family-collapse precedent." + pin: "test/clj/suites/vector_seq_view_test.clj — the `the-class-name-is-the-reserved-slot-name` deftest (case class_name), asserting (str (class (seq [1 2]))) => \"ArraySeq\"." + - id: "AD-067" + area: "collections" + summary: |- + LazySeq retains ADR-0143's single publication state and retry invariant. + A failed thunk or Seqable coercion is not cached: the next touch invokes + the intact thunk again. JVM Clojure 1.12.4 consumes a scalar-returning + thunk before coercion fails, then exposes an empty sequence on retry; + its once-only closure may also have cleared captured locals before a + retry, which can replace the original exception with NullPointerException. + cljw preserves the callable and its captures until successful realization. + Nested realization publishes each invoked inner node, so realized? on + that node is true after the outer node drains it. JVM's separate raw-body + and seq fields leave the inner node's realized? false at that point. + These are state/retry differences only. Both runtimes reject the first + non-seqable body with IllegalArgumentException and reject an invalid + custom Seqable.seq return with ClassCastException. Successful values, + element order, empty tails and at-most-once successful invocation agree. + example: '(let [l (lazy-seq 1)] (try (seq l) (catch Exception _ nil)) (count l)) => cljw throws again / clj 0.' + derives_from: "ADR-0143 (failed realization is never cached) + ADR-0197 (one publication state, immutable inner links and retained retry captures)." + pin: "test/clj/suites/lazy_seqable_test.clj — non-seqable-bodies-raise-on-every-touch and realization-state-follows-ad067; test/golden/cases/lazy_seqable_bodies.expected — scalar retries, closed thunk retry, inner realized." + - id: "AD-068" + area: "hierarchy" + summary: |- + `derive` accepts a CLASS as the parent, where clj raises + `ClassCastException`. clj's 2-arity asserts `(namespace parent)`, and a + `Class` is not `Named`, so `(derive ::tag String)` throws there. + ADR-0109 makes a host class a first-class hierarchy participant in cljw: + `isa?` already answers on class parents, `defmulti` dispatches on them, + and `(derive ::x Object)` is the way a keyword tag joins that graph. + The edge is REAL rather than implied by `Object` being the universal + supertype — `(isa? ::x Object)` is false until the derive installs it — + so refusing a class parent would make an ADR-0109 hierarchy expressible + only through `make-hierarchy`, which the global `defmulti` path does not + read. Every OTHER bad shape is rejected exactly as clj rejects it: a + non-named parent, an un-namespaced parent in the 2-arity, and a tag that + is neither a class nor a namespaced symbol/keyword. + The 3-arity is deliberately more permissive than the 2-arity on + namespacing, in both runtimes: a hierarchy value you pass in is your own, + while the global one is shared across every namespace. + example: '(derive :user/tag String) => cljw installs the edge / clj throws ClassCastException. The reverse, (derive ::x Object) then (isa? ::x Object), is true in cljw and unreachable in clj.' + derives_from: "ADR-0109 (one host-class-VALUE resolver; a class is a hierarchy participant)." + pin: "test/e2e/phase14_opaque_host_class.sh — the derive_object case; clojure.core-test.derive carries the one residual upstream failure this row explains." + - id: AD-069 + area: numerics + summary: '`(int x)` / `(byte x)` / `(short x)` yield a Long: cljw has no Integer / Byte / Short types, so `(instance? Integer (int 0))` is false where clj is true. The narrowing RANGE is still enforced the way clj enforces it, so `(int 2147483648)` raises in both; only the RESULT TYPE differs.' + example: (class (int 0)) => cljw Long / clj java.lang.Integer. (instance? Byte (byte 5)) => cljw false / clj true. + why: 'F-005 fixes the numeric tower as a single internal i64 for exact integers, and ADR-0059 means cljw has no JVM Class to be an instance OF. A distinct Integer / Byte / Short would be three representations the tower does not carry. What the casts DO carry is the range contract: clj rejects a value that would not survive the narrowing, and cljw rejects the same values, so code that relies on `int` to bound a value behaves identically. It is the identity of the box, not the arithmetic, that differs. Sibling of AD-004 (float yields an f64) and AD-016 (biginteger yields BigInt).' + derives_from: 'F-005 (numeric tower: one exact-integer type) + ADR-0059 (no JVM Class).' + pin: test/diff/clj_corpus/narrowing_casts.txt pins the range contract (the part that must match clj); the class divergence is what clojure.core-test.{int,byte,short} report as their residual instance? failures, tracked for a :cljw suite branch. + - id: AD-070 + area: numerics + summary: 'A narrowing cast range-checks a double BEFORE truncating it, for every double. clj picks its cast overload by STATIC type: a double LITERAL reaches `byteCast(double)`, which range-checks first, while a boxed Double reaches `byteCast(Object)`, which truncates through `longCast` and then checks. So `(byte 127.000001)` raises in both runtimes, but `(byte (identity 127.000001))` is 127 in clj and raises in cljw.' + example: '(byte (identity 127.000001)) => cljw raises / clj 127. (byte 127.000001) => both raise. (int ##NaN) => 0 in both.' + why: 'The two clj answers for one value come from JVM static overload selection, which cljw has no equivalent of: a cljw double is one unboxed f64 whatever expression produced it (F-005), so the cast cannot know whether its argument was written as a literal. One rule has to cover both, and the primitive-overload rule is the one that carries the intent of a narrowing cast: `(byte 127.9)` asks for a byte, and 127.9 is not one. The boxed rule would instead make `byte` a silent truncation, which is what `unchecked-byte` is for. The suite tests the literal form, so this choice is also the clj-matching one where clj is unambiguous.' + derives_from: F-005 (a double is one unboxed f64; no static primitive/boxed distinction). + pin: test/diff/clj_corpus/narrowing_casts.txt (the agreeing cases) + suites.numeric-tower-test/narrowing-cast-range, which asserts cljw raises for the boxed form. + - id: AD-071 + area: collections + summary: '`keys` / `vals` demand a map and check EAGERLY, so every non-map raises. clj answers nil for an EMPTY non-map (`(keys #{})`, `(keys "")`) and raises only on realization for a non-empty one. cljw is also more lenient in the other direction: `(keys (list [:a 1]))` is `(:a)`, where clj rejects a vector pair per element.' + example: '(keys #{}) => cljw raises / clj nil. (keys #{1}) => both raise (clj on realization). (keys (list [:a 1])) => cljw (:a) / clj ClassCastException.' + why: 'clj `keys` builds a KeySeq over `seq(coll)` and casts each element to IMapEntry only when that element is walked. Both clj behaviours fall out of that and neither is a contract: nil for an empty set is simply "the lazy seq was never walked", which is why the SAME expression over a non-empty set throws instead, and the per-element cast is what rejects a `[k v]` vector that every other cljw path treats as an entry. Adopting the nil would mean answering nil for one argument and throwing for a larger argument of the same type, which is a worse contract than rejecting both. cljw checks the argument once, up front, and reports which type failed. The leniency half is the same coin: cljw has ONE notion of a map entry (ADR-0078 / D-209, a 2-vector), so a seq of pairs is a seq of entries here.' + derives_from: 'F-011 read with ADR-0078 (a MapEntry IS a 2-vector in cljw), plus the eager-argument-check convention: a type error names the argument, not the element reached later.' + pin: suites.map-helpers-test/keys-vals-demand-a-map — pins that the empty cases keep raising and that a seq of vector pairs keeps working. + - id: AD-072 + area: numerics + summary: '`rationalize` on a BigDecimal whose value is INTEGRAL answers cljws machine integer (`(rationalize 1.0M)` => `1`), where clj answers a BigInt (`1N`). The values are `=`; only the printed N differs.' + example: (rationalize 1.0M) => cljw 1 / clj 1N. (rationalize 100M) => cljw 100 / clj 100N. (= 1 (rationalize 1.0M)) => true in BOTH. + why: clj routes rationalize through BigDecimal.toBigInteger, so the result type is fixed by the OPERATION and a small value still carries the BigInt tag. cljw has one exact-integer type whose storage tier is chosen by MAGNITUDE (F-005), so a value that fits the machine integer is stored and printed as one no matter which operation produced it. Adopting cljs answer would mean a second exact-integer type that exists only to print an N, which F-005 exists to refuse. The same shape is already accepted in the other direction as AD-016. + derives_from: F-005 (one exact-integer type, tier chosen by magnitude), read with AD-016 which accepts the mirror case. + pin: suites.numeric-tower-test/rationalize-bigdec-is-exact - pins that the integral arm answers the exact INTEGER (1.0M is 1, never the unscaled 10) and that the wide-significand cases keep every digit. + - id: AD-073 + area: interop + summary: '`(Boolean. x)` answers the plain `true`/`false`, so `(if (Boolean. "false") :t :f)` is `:f` in cljw where clj''s non-nil Boolean box makes it `:t`.' + example: (if (Boolean. "false") :truthy :falsey) => cljw :falsey / clj :truthy. (Boolean. "true") => true in both, and (= false (Boolean. "false")) is true in both. + why: cljw has no boxed Boolean; `true` and `false` are the only booleans (ADR-0059), and every box ctor answers the nearest cljw VALUE, the shape AD-004 (float) and AD-069 (int) already accept. The only observable consequence is the truthiness of a false box, the footgun Java deprecated this constructor over. A box that is `=` to false yet truthy would need a second false value that exists only to reproduce it. + derives_from: ADR-0059 (class/type answer an interned descriptor; there is no boxed-primitive object), read with AD-004 and AD-069 which accept the same value-not-box shape for float and int. + pin: suites.box-ctors-test/boolean-ctor-is-the-plain-value - pins that `(Boolean. "false")` is `false` itself, so the false branch is taken. + - id: AD-074 + area: interop + summary: '`(Short. 7)` and `(Byte. 7)` answer 7 in cljw, where clj throws IllegalArgumentException because a long literal matches neither the `(short)` nor the `(byte)` ctor.' + example: (Short. 7) => cljw 7 / clj IllegalArgumentException. (Short. (short 7)) => 7 in both. (Short. 99999) raises IllegalArgumentException in both. + why: cljw has no short or byte type (AD-069), so `(short 7)` IS the Long 7 and the ctor cannot tell `(Short. (short 7))`, which clj accepts, from `(Short. 7)`, which it rejects. Rejecting integers would break the valid form to reproduce the error on the invalid one. The range check still holds, so every value a Short or Byte could not hold is refused as in clj. + derives_from: F-005 (no short/byte primitive; one exact-integer type), read with AD-069 which accepts `(short x)` / `(byte x)` yielding a Long. + pin: suites.box-ctors-test/short-and-byte-take-in-range-integers - pins that `(Short. 7)` and `(Short. (short 7))` both answer 7 and that an out-of-range integer still raises IllegalArgumentException. + - id: AD-075 + area: equality + summary: '`(= date ts)` is false when a java.util.Date and a java.sql.Timestamp name the same millisecond, where clj answers true; cljw keeps `=` symmetric, and clj already answers `(= ts date)` false.' + example: (let [t (clojure.instant/read-instant-timestamp "1970-01-01T00:00:00.005Z") d (java.util.Date. 5)] [(= d t) (= t d) (get {t :t} d) (count (hash-set t d))]) => cljw [false false nil 2] / clj [true false :t 1]. (hash t) and (hash d) are both 5 in both runtimes, and (count (hash-set d t)) is 2 in both. + why: 'Timestamp extends Date on the JVM, so Date.equals(Object) accepts a Timestamp by millisecond, while Timestamp.equals(Object) rejects every non-Timestamp. The Timestamp javadoc itself calls the pair not symmetric and says the inheritance is implementation inheritance, not type inheritance. Porting it would make `=` depend on argument order, and every hashed collection would then depend on insertion order: clj builds (hash-set t d) as one element and (hash-set d t) as two. cljw models Timestamp as its own value type (D-382: a distinct descriptor carrying epoch-ms plus nanos, never collapsed onto Date), and `=` over temporal values compares within one type, which keeps `=` symmetric and `=` implies equal hash, the two laws src/testing/prop_equal.zig pins. The hashes already agree with clj, so only the cross-type Date-first comparison differs.' + derives_from: 'D-382 (Timestamp is its own neutral value type with a distinct descriptor; collapsing it onto Date was the rejected smell, user 2026-06-10) + F-014 (linguistically-general Java, not a JVM reimplementation: the JDK equals-contract violation is not ported) + the symmetric `=` law pinned by src/testing/prop_equal.zig.' + pin: test/e2e/temporal_hash_eq.sh same_hash_distinct_types - asserts on both backends that (= d t) is false and (get {t :t} d) is nil for a Date and a Timestamp of the same millisecond, next to the clj-matching (= t d) false and (count (hash-set d t)) 2. + - id: AD-076 + area: strings + summary: 'A string is indexed by Unicode CODE POINT, not by UTF-16 code unit. `count`, `subs`, `nth`, `get`, `seq`/`first`, `(map int s)` and the interop methods `.length`, `.charAt`, `.indexOf` and `.substring` all count a character outside the Basic Multilingual Plane (emoji, rare CJK, most historic scripts) as ONE position, where clj counts its surrogate pair as two. BMP text, including multibyte characters such as é or 世, indexes identically in both runtimes.' + example: '(count "🌍") => cljw 1 / clj 2. (count "ab🌍") => cljw 3 / clj 4. (.length "🌍") => cljw 1 / clj 2. (subs "a🌍b" 1 2) => cljw "🌍" / clj a lone high surrogate. (vec (map int "🌍")) => cljw [127757] / clj [55356 57101]. (str (first "🌍")) => cljw "🌍" / clj a lone high surrogate. (.indexOf "a🌍b" "b") => cljw 2 / clj 3. Measured on cljw v1.14.11; clj 1.12 column from GitHub issue #1.' + why: |- + UTF-16 is how the JVM happens to store a String, not part of Clojure's + string semantics: clj's own `count` docstring says "number of items in + the collection", and a surrogate half is not an item any user wrote. + cljw stores strings as UTF-8 and has no UTF-16 buffer to index, so + matching clj would mean synthesising surrogate halves on every index + operation: values that cannot be printed, that are not valid Unicode + scalar values, and whose only observable use is to be glued back + together. `(subs s i j)` handing back half a pair is a bug generator in + clj; in cljw every index lands on a character boundary, so no string + operation can produce invalid text. + The interop methods follow the same model on purpose. A second, + UTF-16-unit view behind `.length` / `.charAt` / `.indexOf` / + `.substring` would make `(= (count s) (.length s))` false for the same + string and would mix two index spaces in one program: an index from + `.indexOf` fed to `subs` would land on the wrong character. One index + space for every string operation is the narrower and more defensible + claim; the interop names are kept for source compatibility, and the + ledger records that they are code-point based. + Ported JVM code is affected only where it does arithmetic on indices + against outside UTF-16 data (a JavaScript or Java peer's offsets, a + protocol counting UTF-16 units). docs/clojure_vs_clojurewasm.md gives + the porting guidance. + derives_from: 'F-011 read at the VALUE level (a string is a sequence of characters; the JVM storage encoding is not surface), plus D-217 (a String is Indexed by codepoint) and D-179 / O-060 (string seq is a UTF-8 byte-offset view over code points). No JVM, ADR-0059: there is no UTF-16 backing store to expose.' + pin: test/clj/suites/string_code_point_test.clj (every expression in the example, language and interop surfaces) + test/golden/cases/string_unicode.clj (count of a string holding 🌍, and (count "🌍") => 1). diff --git a/.dev/bench/README.md b/.dev/bench/README.md new file mode 100644 index 000000000..f25f30e90 --- /dev/null +++ b/.dev/bench/README.md @@ -0,0 +1,166 @@ +# Collection-algorithm parity bench (cljw vs clj) + +Motivating question (user, 2026-08-13): *for performance, should cljw reimplement +the same immutable algorithms core Clojure does?* + +Answered by measurement rather than by reading either implementation, per the +"measure the provider before you abstract it" principle: the same probe form is +evaluated in a cljw nREPL and a JVM Clojure 1.12.4 nREPL on the same box. + +## Method + +`scaling_form.edn` — run a FIXED number of operations (2000, after 300 warmup +iterations so the JVM has JIT-compiled) against collections of growing size +`[1000 4000 16000]`. Flat wall-clock across sizes means the operation is +O(1)/O(log n) per call; time growing with the size factor means O(n). + +Run it in both REPLs: + + code cider spawn repl_type=cljw name=perf-cljw project_dir= + code cider spawn repl_type=clj name=perf-clj project_dir= + code cider eval session_name=... code= + +`scaling.clj` / `traversal.clj` are the standalone script forms (`cljw +.dev/bench/scaling.clj`) — same probes, human-readable output, plus a +whole-collection traversal probe that checks for accidental O(n²). + +## Result, 2026-08-13 (ms per 2000 ops at n=16000, ReleaseSafe vs clj 1.12.4) + +| op | clj | cljw | verdict | +|---|---|---|---| +| `nth` (mid) | 0.31 | 0.26 | cljw faster | +| `assoc` (mid) | 5.17 | 1.20 | cljw faster | +| `conj` | 2.79 | 2.29 | comparable | +| `map` `get` | 0.58 | 0.42 | cljw faster | +| `map` `assoc` | 3.38 | 2.26 | cljw faster | +| `map` `dissoc` | 5.09 | 1.15 | cljw faster | +| `list` `conj` | 0.36 | 0.43 | comparable | +| **`pop`** | **1.93** | **8278** | **4288× slower → FIXED (O-056), now 2.47** | +| **`subvec`** | **1.06** | **7929** | **7480× slower** | +| **`seq`** | **0.60** | **2123** | **3549× slower → FIXED (O-058), now 0.36** | +| **`rest`** | **0.87** | **2253** | **2576× slower → FIXED (O-058), now 0.39** | +| **`next`** | **0.99** | **2164** | **2184× slower → FIXED (O-058), now 0.38** | +| **`list` `count`** | **0.18** | **481** | **2711× slower → FIXED (O-057), now 0.19** | + +### Correction: the small deltas above are NOT reliable + +The first table's "cljw faster" verdicts on the sub-microsecond ops were an +artefact of too few reps and no baseline. Re-measured best-of-7 over 20000 ops +with an empty-loop baseline subtracted (n=16000): + +| per 20000 ops | clj | cljw | | +|---|---|---|---| +| **empty `dotimes`** | **0.043** | **0.900** | **clj 21× faster** | +| `nth` | 0.373 | 0.879 | cljw 2.4× | +| `assoc` | 49.38 | 21.43 | cljw 2.3× | +| map `assoc` | 73.93 | 18.77 | cljw 3.9× | +| map `dissoc` | 2.62 | 8.15 | clj 3.1× | +| `conj` | 1.75 | 7.62 | clj 4.4× | + +Two separate things, which the first table conflated: + +- **Interpretation speed: the JVM is far ahead.** An empty loop costs cljw 45ns + per iteration against the JVM's 2.2ns. That is not Zig losing to Java — it is + an *interpreter* (tree-walk / VM) losing to a *JIT* that has compiled the loop + to native code with the induction variable in a register. Nothing about the + collection work changes it; only a compiler would. +- **The persistent structures themselves: same league, trading wins.** cljw is + ahead on `nth` and on both `assoc`es (its bump-allocating GC makes the + path-copy cheaper); the JVM is ahead on `conj` and `dissoc`. No 1000× effects + in either direction once the algorithms match. + +So: do not read this bench as "Zig beats the JVM". Read it as "the data +structures are sound, and every catastrophic number below is algorithmic". + +### The catastrophic numbers are algorithmic, not linguistic + +Every loss is one of two specific mistakes, and neither is a data-structure +problem: + +1. **An abstraction-barrier violation** — the Clojure layer re-implements an + operation the Zig barrier already provides correctly. `pop` was the pure + case: `vector.zig::pop` is a faithful `PersistentVector.pop()`, and + `core.clj` bypassed it with `(into [] (take (dec (count coll)) coll))`. + Fixed as O-056; the fix added no algorithm, it just exposed the barrier op. + + `list count` was the same class with a twist: the barrier op existed + (`Cons.count`, O(1)) but was DISTRUSTED, because a `.list` cell can head a + mixed chain (`(cons x (map …))`) where the stored number is the `.list` + prefix length, not the total. Fixed as O-057 by making the field + self-validating — a `COUNT_UNKNOWN` sentinel marks exactly the chains that + must still walk — rather than by waiting for the D-178 `.list`/`.cons` tag + split the old comment deferred to. When a barrier value is right *most* of + the time, say WHICH times in the value itself; a blanket distrust throws + the fast path away for every caller. + +2. **Materialising where Clojure keeps a VIEW** — `seq`/`rest`/`next` on a + vector build an eager `PersistentList` (`sequence.zig` `vectorToList` / + `vectorTailAsList`) where Clojure returns a `ChunkedSeq` holding + `(vector, index)`; `subvec` eagerly copies (D-044) where Clojure returns an + `APersistentVector$SubVector` sharing the parent's root and tail. + +## Ranked remaining work + +| # | Fix | Kind | Unblocks | +|---|---|---|---| +| ~~1~~ | ~~`SubVector` view~~ → still open, but on its OWN | view | `subvec` (7480×) and slice-style ops built on it | +| ~~2~~ | ~~Chunked vector-seq VIEW behind the existing seq protocol~~ | view, not algorithm | **DONE — O-058. `seq`/`rest`/`next` are now an `.array_seq` view; all three are FASTER than clj.** | +| ~~3~~ | ~~Cached count on `PersistentList`~~ | ~~one field~~ | **DONE — O-057.** | + +### Correction: (2) never needed (1) + +The earlier revision of this file ranked `SubVector` above the vector-seq on the +grounds that it was the seq's *enabler* — "the vector analogue of 'a smaller +range' is 'the vector minus its first 32 elements', which is exactly a +`SubVector` view". That was wrong, and it stalled the larger win behind a +`heap_tag`-census question that did not apply to it. + +`next` of a seq is a **seq**, not a collection. Clojure's own answer is +`PersistentVector$ChunkedSeq`, which holds the SAME vector at a higher index — +it never constructs a smaller vector. So the seq view needs `(vector, index)`, +which is what O-058 built, and `subvec` is an independent problem about +returning a *vector* that shares a parent's root and tail. + +Two lessons worth keeping, because both were reasoning failures rather than +implementation ones: + +- **Check what the reference implementation actually returns** before deriving a + dependency from an analogy. The analogy to `.range` was apt for the chunking + *mechanism* and misleading about the *tail type*. +- **A blocked item can block a neighbour that is not actually downstream.** The + census question ("may we spend one of the 9 `unallocated` names") is real for + `SubVector` and was never live for the seq view — A14 `.array_seq` was already + an allocated slot, already declared across every protocol table in + `interface_membership.zig`, already named in `class_name.zig`, and merely had + no producer. `equal.zig`'s header anticipated it in as many words: "when a + producer mints those tags." + +### What is left: `subvec` (7480×) + +`vector.zig::subvec` eagerly copies (D-044) where Clojure returns an +`APersistentVector$SubVector` sharing the parent's root and tail. This one DOES +have to answer the tag question, and it is a genuine user decision per +`heap_tag.zig`'s own doc: + +- **(a) repurpose an `unallocated` slot** (`.tuple` / `.box` are the plausible + names) — clean and polymorphic, and the census is explicitly published so that + the question is "may `box` become `subvec`", not "may we amend F-004". Spends + a name. +- **(b) `start: u32` inside `Vector`'s existing `_pad: [6]u8`** — no struct + growth and no Tag spent, but then EVERY vector op must honour the offset, and + any path that reads `root`/`tail` without it goes silently wrong. That review + surface is the real cost, not the four bytes. + +Note the asymmetry with O-058: there, no consumer had to change, because a new +representation satisfied an existing protocol. Option (b) is the opposite shape +— it changes the meaning of an existing representation, so it touches every +consumer. That is the argument for (a) even though (a) is the one that spends +something. + +### Caveat that still holds + +`reduce` / `doseq` / `into` over a vector were already fast — the index-walk and +chunk-drain fast paths (O-002, O-004, O-032) bypass `seq` entirely, and +`traversal.clj` confirms they stay linear. So O-058 helped hand-written +`loop`/`recur` walks and library code rather than the idiomatic core paths, +which is why it was ranked by breadth rather than by a benchmark regression. diff --git a/.dev/bench/cross-lang-linux-x86_64.PROVISIONAL.yaml b/.dev/bench/cross-lang-linux-x86_64.PROVISIONAL.yaml new file mode 100644 index 000000000..47bec19f7 --- /dev/null +++ b/.dev/bench/cross-lang-linux-x86_64.PROVISIONAL.yaml @@ -0,0 +1,635 @@ +# PROVISIONAL. Do NOT render this into bench/README.md. +# +# Produced 2026-09-05 by the OLD compare_langs.sh, before commit 3cf8b391 added +# core pinning, governor recording and per-measurement stddev. It therefore +# carries no cold_sd, so bench_domain gives it noise_floor 1.0 and every +# difference in it is taken at face value. +# +# It should not be. Measured on the same host at the same time: median relative +# dispersion 22%, noise floor 1.62x, while this file's per-workload differences +# are mostly 1.2-1.5x. All of them sit inside the noise. +# +# Its env block was also repaired BY HAND: the macOS-only host detection emitted +# bare `?` characters, which are YAML's explicit-key indicator, so the file as +# written did not parse at all. +# +# Kept as the record of the first x86_64 run and of why it cannot be published. +# Superseded by kanban [CLJW-BENCH-LINUX-SUITE] (20260905012148-4fc83448), which +# re-runs it under the fixed harness. +# Cross-language benchmark comparison +# Generated: 2026-09-05T00:35:03-03:00 + +env: + machine: "Alienware m16 R2" + cpu: "Intel(R) Core(TM) Ultra 9 185H, 22-core" + ram: "62 GB" + os: "Ubuntu 24.04.4 LTS" + kernel: "Linux 7.0.0-30-generic x86_64" + tool: hyperfine + runs: 10 + warmup: 5 + +date: "2026-09-05" + +startup_ms: + cw: 42.4514 + c: 1.2303 + zig: 0.7355 + java: 52.5414 + clj: 1003.2008 + go: 2.7323 + py: 45.4657 + rb: 84.0563 + js: 43.6818 + bb: 17.3486 + +benchmarks: + fib_recursive: + cold: + cw: 63.0671 + c: 1.8221 + java: 49.6327 + clj: 1573.2626 + go: 3.9718 + py: 51.9537 + rb: 109.6462 + js: 50.7781 + bb: 52.0457 + warm: + cw: 20.6157 + c: 0.5918 + java: 0.0000 + clj: 570.0618 + go: 1.2395 + py: 6.4880 + rb: 25.5899 + js: 7.0963 + bb: 34.6971 + fib_loop: + cold: + cw: 35.5778 + c: 0.8219 + java: 52.3918 + clj: 1464.8426 + go: 3.6061 + py: 38.8242 + rb: 110.5343 + js: 58.8647 + bb: 28.3888 + warm: + cw: 0.0000 + c: 0.0000 + java: 0.0000 + clj: 461.6418 + go: 0.8738 + py: 0.0000 + rb: 26.4780 + js: 15.1829 + bb: 11.0402 + tak: + cold: + cw: 54.8083 + c: 1.0165 + java: 59.8633 + clj: 1781.1708 + go: 2.9862 + py: 43.5344 + rb: 97.7864 + js: 52.5668 + bb: 29.3884 + warm: + cw: 12.3569 + c: 0.0000 + java: 7.3219 + clj: 777.9700 + go: 0.2539 + py: 0.0000 + rb: 13.7301 + js: 8.8850 + bb: 12.0398 + arith_loop: + cold: + cw: 99.0182 + c: 1.0224 + java: 55.2632 + clj: 1268.8020 + go: 3.3438 + py: 161.4207 + rb: 126.7784 + js: 54.4628 + bb: 124.0454 + warm: + cw: 56.5668 + c: 0.0000 + java: 2.7218 + clj: 265.6012 + go: 0.6115 + py: 115.9550 + rb: 42.7221 + js: 10.7810 + bb: 106.6968 + map_filter_reduce: + cold: + cw: 46.4828 + c: 1.1681 + java: 46.8913 + clj: 1327.4323 + go: 2.7010 + py: 48.0309 + rb: 91.3733 + js: 49.9121 + bb: 29.2254 + warm: + cw: 4.0314 + c: 0.0000 + java: 0.0000 + clj: 324.2315 + go: 0.0000 + py: 2.5652 + rb: 7.3170 + js: 6.2303 + bb: 11.8768 + vector_ops: + cold: + cw: 47.6471 + c: 0.7877 + java: 56.9092 + clj: 1411.5733 + go: 3.5148 + py: 46.3645 + rb: 99.8817 + js: 52.3018 + bb: 24.7199 + warm: + cw: 5.1957 + c: 0.0000 + java: 4.3678 + clj: 408.3725 + go: 0.7825 + py: 0.8988 + rb: 15.8254 + js: 8.6200 + bb: 7.3713 + map_ops: + cold: + cw: 38.9262 + c: 1.0886 + java: 50.2257 + clj: 1285.2314 + go: 2.9347 + py: 39.5935 + rb: 93.2023 + js: 52.4883 + bb: 22.2749 + warm: + cw: 0.0000 + c: 0.0000 + java: 0.0000 + clj: 282.0306 + go: 0.2024 + py: 0.0000 + rb: 9.1460 + js: 8.8065 + bb: 4.9263 + list_build: + cold: + cw: 34.2314 + c: 2.8834 + java: 53.7719 + clj: 1281.9087 + go: 3.7948 + py: 42.7123 + rb: 79.8935 + js: 49.7093 + bb: 23.8095 + warm: + cw: 0.0000 + c: 1.6531 + java: 1.2305 + clj: 278.7079 + go: 1.0625 + py: 0.0000 + rb: 0.0000 + js: 6.0275 + bb: 6.4609 + sieve: + cold: + cw: 88.4841 + c: 1.3921 + java: 65.5540 + clj: 1331.3643 + go: 3.8647 + py: 36.6109 + rb: 87.2851 + js: 43.3675 + bb: 30.1026 + warm: + cw: 46.0327 + c: 0.1618 + java: 13.0126 + clj: 328.1635 + go: 1.1324 + py: 0.0000 + rb: 3.2288 + js: 0.0000 + bb: 12.7540 + nqueens: + cold: + cw: 59.6639 + c: 1.2023 + java: 49.5818 + go: 3.3473 + py: 42.0469 + rb: 116.2355 + js: 47.6099 + bb: 45.2494 + warm: + cw: 17.2125 + c: 0.0000 + java: 0.0000 + go: 0.6150 + py: 0.0000 + rb: 32.1792 + js: 3.9281 + bb: 27.9008 + atom_swap: + cold: + cw: 35.0203 + c: 1.0646 + java: 48.8589 + clj: 1406.3662 + go: 2.5486 + py: 39.8994 + rb: 91.9067 + js: 48.4560 + bb: 23.3377 + warm: + cw: 0.0000 + c: 0.0000 + java: 0.0000 + clj: 403.1654 + go: 0.0000 + py: 0.0000 + rb: 7.8504 + js: 4.7742 + bb: 5.9891 + gc_stress: + cold: + cw: 81.3917 + c: 6.9682 + java: 101.7978 + clj: 1322.1243 + go: 10.6244 + py: 86.8930 + rb: 105.4978 + js: 55.1522 + bb: 97.9637 + warm: + cw: 38.9403 + c: 5.7379 + java: 49.2564 + clj: 318.9235 + go: 7.8921 + py: 41.4273 + rb: 21.4415 + js: 11.4704 + bb: 80.6151 + lazy_chain: + cold: + cw: 87.6126 + c: 1.3214 + java: 45.4869 + clj: 1341.6957 + go: 3.2367 + py: 44.6611 + rb: 82.2990 + js: 44.6159 + bb: 31.0475 + warm: + cw: 45.1612 + c: 0.0911 + java: 0.0000 + clj: 338.4949 + go: 0.5044 + py: 0.0000 + rb: 0.0000 + js: 0.9341 + bb: 13.6989 + transduce: + cold: + cw: 44.9127 + c: 1.1103 + java: 49.8420 + clj: 1417.2251 + go: 2.6375 + py: 41.4987 + rb: 103.1296 + js: 47.8954 + bb: 23.4510 + warm: + cw: 2.4613 + c: 0.0000 + java: 0.0000 + clj: 414.0243 + go: 0.0000 + py: 0.0000 + rb: 19.0733 + js: 4.2136 + bb: 6.1024 + keyword_lookup: + cold: + cw: 53.0831 + c: 3.2829 + java: 54.2426 + clj: 1464.0117 + go: 5.8549 + py: 51.3693 + rb: 91.2433 + js: 54.4870 + bb: 39.4818 + warm: + cw: 10.6317 + c: 2.0526 + java: 1.7012 + clj: 460.8109 + go: 3.1226 + py: 5.9036 + rb: 7.1870 + js: 10.8052 + bb: 22.1332 + protocol_dispatch: + cold: + cw: 43.3909 + c: 0.9275 + java: 53.1831 + go: 3.1354 + py: 44.5774 + rb: 94.1091 + js: 50.2420 + warm: + cw: 0.9395 + c: 0.0000 + java: 0.6417 + go: 0.4031 + py: 0.0000 + rb: 10.0528 + js: 6.5602 + nested_update: + cold: + cw: 73.0333 + c: 1.0797 + java: 59.7468 + clj: 1336.0872 + go: 3.8715 + py: 34.2793 + rb: 82.2857 + js: 46.7183 + bb: 27.2412 + warm: + cw: 30.5819 + c: 0.0000 + java: 7.2054 + clj: 332.8864 + go: 1.1392 + py: 0.0000 + rb: 0.0000 + js: 3.0365 + bb: 9.8926 + string_ops: + cold: + cw: 83.2103 + c: 8.8891 + java: 66.7537 + clj: 949.8767 + go: 4.1456 + py: 50.7468 + rb: 65.2611 + js: 33.1580 + bb: 30.8891 + warm: + cw: 40.7589 + c: 7.6588 + java: 14.2123 + clj: 0.0000 + go: 1.4133 + py: 5.2811 + rb: 0.0000 + js: 0.0000 + bb: 13.5405 + multimethod_dispatch: + cold: + cw: 32.1203 + c: 0.5585 + java: 32.5087 + clj: 1802.5790 + go: 4.6023 + py: 65.5867 + rb: 129.9988 + js: 66.3927 + bb: 28.8730 + warm: + cw: 0.0000 + c: 0.0000 + java: 0.0000 + clj: 799.3782 + go: 1.8700 + py: 20.1210 + rb: 45.9425 + js: 22.7109 + bb: 11.5244 + real_workload: + cold: + cw: 131.3489 + c: 0.4574 + java: 73.5208 + clj: 1761.9473 + go: 2.9489 + py: 69.6669 + rb: 138.0942 + js: 54.2963 + bb: 38.6781 + warm: + cw: 88.8975 + c: 0.0000 + java: 20.9794 + clj: 758.7465 + go: 0.2166 + py: 24.2012 + rb: 54.0379 + js: 10.6145 + bb: 21.3295 + wasm_load: + cold: + cw: 52.5809 + warm: + cw: 10.1295 + wasm_call: + cold: + cw: 22120.0211 + warm: + cw: 22077.5697 + wasm_memory: + cold: + cw: 5097.9162 + warm: + cw: 5055.4648 + wasm_fib: + cold: + cw: 67.5267 + warm: + cw: 25.0753 + wasm_sieve: + cold: + cw: 173.5894 + warm: + cw: 131.1380 + gc_alloc_rate: + cold: + cw: 232.6288 + clj: 1750.0419 + js: 55.7803 + bb: 104.4652 + warm: + cw: 190.1774 + clj: 746.8411 + js: 12.0985 + bb: 87.1166 + gc_large_heap: + cold: + cw: 260.5076 + clj: 1765.5787 + js: 87.3304 + bb: 103.6289 + warm: + cw: 218.0562 + clj: 762.3779 + js: 43.6486 + bb: 86.2803 + wasm_tak: + cold: + cw: 69.4774 + warm: + cw: 27.0260 + wasm_arith: + cold: + cw: 61.2015 + warm: + cw: 18.7501 + wasm_gcd: + cold: + cw: 22948.5282 + warm: + cw: 22906.0768 + wasm_fib_loop: + cold: + cw: 1829.4787 + warm: + cw: 1787.0273 + bigint_factorial: + cold: + cw: 73.3097 + java: 77.7688 + clj: 1379.0362 + go: 5.9293 + py: 33.5582 + rb: 100.0696 + js: 52.5776 + bb: 34.7021 + warm: + cw: 30.8583 + java: 25.2274 + clj: 375.8354 + go: 3.1970 + py: 0.0000 + rb: 16.0133 + js: 8.8958 + bb: 17.3535 + ratio_sum: + cold: + cw: 114.8257 + clj: 1205.1316 + bb: 80.7647 + warm: + cw: 72.3743 + clj: 201.9308 + bb: 63.4161 + stm_refs: + cold: + cw: 115.4685 + clj: 1638.8711 + bb: 182.7551 + warm: + cw: 73.0171 + clj: 635.6703 + bb: 165.4065 + regex_count: + cold: + cw: 114.1623 + c: 8.9320 + java: 113.9366 + clj: 1500.6968 + go: 13.8564 + py: 48.3585 + rb: 98.2332 + js: 42.4020 + bb: 26.5389 + warm: + cw: 71.7109 + c: 7.7017 + java: 61.3952 + clj: 497.4960 + go: 11.1241 + py: 2.8928 + rb: 14.1769 + js: 0.0000 + bb: 9.1903 + sort: + cold: + cw: 29.1855 + c: 1.4016 + java: 38.5580 + clj: 1182.0034 + go: 2.2872 + py: 29.2746 + rb: 78.4305 + js: 34.7711 + bb: 14.2667 + warm: + cw: 0.0000 + c: 0.1713 + java: 0.0000 + clj: 178.8026 + go: 0.0000 + py: 0.0000 + rb: 0.0000 + js: 0.0000 + bb: 0.0000 + destructure: + cold: + cw: 79.4425 + clj: 2314.6476 + bb: 117.1636 + warm: + cw: 36.9911 + clj: 1311.4468 + bb: 99.8150 + edn_roundtrip: + cold: + cw: 205.6130 + clj: 3585.1396 + bb: 548.6785 + warm: + cw: 163.1616 + clj: 2581.9388 + bb: 531.3299 + json_parse: + cold: + cw: 380.5401 + py: 154.8230 + rb: 236.7985 + js: 99.4684 + warm: + cw: 338.0887 + py: 109.3573 + rb: 152.7422 + js: 55.7866 diff --git a/.dev/bench/ffi_boundary/README.md b/.dev/bench/ffi_boundary/README.md new file mode 100644 index 000000000..3abe68108 --- /dev/null +++ b/.dev/bench/ffi_boundary/README.md @@ -0,0 +1,99 @@ +# FFI boundary cost probes + +Isolates the cost of one `wasm/call` crossing from the loop and call overhead +around it. Run each under hyperfine, take the min, subtract: + + per crossing = (wasm_call - noop) / 1e6 + per clj call = (clj_call - noop) / 1e6 + +`noop` is startup plus module load, so it is the right subtrahend for both. +`clj_call` is the same loop shape and arity with a Clojure callee, which is what +separates "the crossing is expensive" from "the loop is expensive". + + P="taskset -c 0-11" # pin: a hybrid host otherwise measures core assignment + hyperfine --warmup 3 --runs 15 -N \ + "$P ./zig-out/bin/cljw .dev/bench/ffi_boundary/noop.clj" \ + "$P ./zig-out/bin/cljw .dev/bench/ffi_boundary/clj_call.clj" \ + "$P ./zig-out/bin/cljw .dev/bench/ffi_boundary/wasm_call.clj" + +## Measured 2026-09-05, Ultra 9 185H, ReleaseSafe -Dwasm, min-of-15 + + Clojure fn call 184 ns + wasm/call :engine interp 392 ns (2.1x a Clojure call) + wasm/call :engine auto (JIT) 18,875 ns (48x the interp path) + +The 18.9 us is not the cost of crossing. It is zwasm/D-584: `computeStackLimit` +runs on every JIT invocation, and on Linux/glibc on the INITIAL thread glibc +answers by opening and parsing `/proc/self/maps`. Confirmed with zwasm's own +runner on this box (`zig build bench-latency`): `stack_limit_query_ns` 26,262 +against `jit_ns` 27,436, so the query is ~96% of the per-call JIT cost. The +interpreter caches the same value, which is why it wins here. + +Do not re-derive this from the source. An earlier pass read the call graph, found +`findExportFunc` re-parsing the module three times per call, and named that as +the cause. It is real (zwasm/D-585) but sits in the ~1.2 us remainder. See hive +memory 20260905010714-4c94602c. + +## The engine x thread matrix, measured 2026-09-05 + +`engine_thread_matrix.clj`, 20k calls per cell, same trivial `add` export, same +process. Run it directly: `./zig-out/bin/cljw .dev/bench/ffi_boundary/engine_thread_matrix.clj` + + cell ns/call + JIT main thread 16547 + JIT worker thread 1181 + interp main thread 399 + interp worker thread 413 + +Three readings, and each names a different fix: + +1. **The 14x JIT penalty is entirely the initial thread.** Nothing else differs + between rows 1 and 2. This is zwasm/D-584 and it is dodgeable without touching + zwasm: run the invocation on any thread that is not the process's first one. + `@(future ...)` already does it today, measured at 12.5x on a separate run. +2. **The interpreter is thread-indifferent** (399 vs 413), which confirms it + caches the stack limit rather than re-querying per call. +3. **Even on a worker thread the JIT is 2.9x the interpreter** on a trivial + callee. That residual is zwasm/D-585, the per-call module re-parse, now visible + because zwasm/D-584 has stopped swamping it. + +So the engine choice and the thread choice are independent levers, and the right +engine still depends on call shape: the JIT wins 15.5x on compute-in-wasm +(`bench/wasm_jit_vs_interp.sh`) and loses on crossing-heavy work at any thread. + +## After ADR-0195, measured 2026-09-06 + +`cljw` now runs the program on a spawned runtime thread, so "main thread" below +is that thread, not the process's initial one. Loaded host (co-tenant JVM at +~5.6 cores, load average 11-15), minimum of six runs; the interp rows show the +noise floor against 2026-09-05's 399 / 413. + + cell ns/call + JIT main thread 1150 + JIT worker thread 1262 + interp main thread 409 + interp worker thread 479 + +Reading 1 above is closed: rows 1 and 2 are now the same cell. Readings 2 and 3 +stand unchanged, and the residual is the next card's target. + +## After the exportSig cache, measured 2026-09-06 + +`Loaded.exportSig` resolves a name once per handle now; on the JIT that was a +module re-parse per call. Same host, same load, minimum of three runs. + + cell ns/call + JIT main thread 896 + JIT worker thread 964 + interp main thread 375 + interp worker thread 451 + +The JIT cells lose ~400-500 ns; the interpreter cells move within noise. The +standing measurement of this axis is now `bench/wasm_percall.sh` (median / +min / max / sd over trials, ns, with a plain Clojure call as the platform +constant); `engine_thread_matrix.clj` stays as the quick 2x2 probe. + +Follow-ups: [CLJW-WASM-ENGINE-DEFAULT] landed as ADR-0196 (the default stays +`.auto`; traps name the engine and the shape); [CLJW-WASM-WORKER-THREAD] landed +as ADR-0195; [CLJW-WASM-EXPORTSIG-CACHE] landed. The rest is upstream +(`[ZWASM-PERCALL-TRACK]`). diff --git a/.dev/bench/ffi_boundary/clj_call.clj b/.dev/bench/ffi_boundary/clj_call.clj new file mode 100644 index 000000000..f7362e074 --- /dev/null +++ b/.dev/bench/ffi_boundary/clj_call.clj @@ -0,0 +1,9 @@ +;; Same loop shape, same arity, Clojure callee. The subtrahend: whatever this +;; costs is loop + call overhead that the wasm version also pays, so the +;; difference is the crossing itself. +(def m (wasm/load "bench/wasm/ffi/add.wasm" {:fuel 0})) +(defn add2 [a b] (+ a b)) +(loop [i 0 acc 0] + (if (< i 1000000) + (recur (inc i) (add2 i 1)) + (println acc))) diff --git a/.dev/bench/ffi_boundary/engine_thread_matrix.clj b/.dev/bench/ffi_boundary/engine_thread_matrix.clj new file mode 100644 index 000000000..b74bc4fd9 --- /dev/null +++ b/.dev/bench/ffi_boundary/engine_thread_matrix.clj @@ -0,0 +1,18 @@ +(def n 20000) +(defn burn [m] + (loop [i 0 acc 0] (if (< i n) (recur (inc i) (wasm/call m "add" i 1)) acc))) +(defn ms [f] (let [t0 (System/nanoTime) _ (f) t1 (System/nanoTime)] + (/ (double (- t1 t0)) 1000000.0))) +(defn ns-per [f] (/ (* (ms f) 1e6) n)) + +(def jit (wasm/load "bench/wasm/ffi/add.wasm" {:engine :auto :fuel 0})) +(def interp (wasm/load "bench/wasm/ffi/add.wasm" {:engine :interp :fuel 0})) + +;; warm every path +(burn jit) (burn interp) @(future (burn jit)) @(future (burn interp)) + +(println (format "%-22s %10s" "cell" "ns/call")) +(println (format "%-22s %10.0f" "JIT main thread" (ns-per #(burn jit)))) +(println (format "%-22s %10.0f" "JIT worker thread" (ns-per #(deref (future (burn jit)))))) +(println (format "%-22s %10.0f" "interp main thread" (ns-per #(burn interp)))) +(println (format "%-22s %10.0f" "interp worker thread" (ns-per #(deref (future (burn interp)))))) diff --git a/.dev/bench/ffi_boundary/noop.clj b/.dev/bench/ffi_boundary/noop.clj new file mode 100644 index 000000000..bc7c7c6ee --- /dev/null +++ b/.dev/bench/ffi_boundary/noop.clj @@ -0,0 +1,3 @@ +;; Startup + module load only. +(def m (wasm/load "bench/wasm/ffi/add.wasm" {:fuel 0})) +(println (wasm/call m "add" 0 0)) diff --git a/.dev/bench/ffi_boundary/noop_interp.clj b/.dev/bench/ffi_boundary/noop_interp.clj new file mode 100644 index 000000000..8e0022aa4 --- /dev/null +++ b/.dev/bench/ffi_boundary/noop_interp.clj @@ -0,0 +1,2 @@ +(def m (wasm/load "bench/wasm/ffi/add.wasm" {:engine :interp :fuel 0})) +(println (wasm/call m "add" 0 0)) diff --git a/.dev/bench/ffi_boundary/wasm_call.clj b/.dev/bench/ffi_boundary/wasm_call.clj new file mode 100644 index 000000000..54bc2c282 --- /dev/null +++ b/.dev/bench/ffi_boundary/wasm_call.clj @@ -0,0 +1,6 @@ +;; 1M crossings of the cljw -> wasm boundary, trivial add on the far side. +(def m (wasm/load "bench/wasm/ffi/add.wasm" {:fuel 0})) +(loop [i 0 acc 0] + (if (< i 1000000) + (recur (inc i) (wasm/call m "add" i 1)) + (println acc))) diff --git a/.dev/bench/ffi_boundary/wasm_call_interp.clj b/.dev/bench/ffi_boundary/wasm_call_interp.clj new file mode 100644 index 000000000..5fd95ad66 --- /dev/null +++ b/.dev/bench/ffi_boundary/wasm_call_interp.clj @@ -0,0 +1,5 @@ +(def m (wasm/load "bench/wasm/ffi/add.wasm" {:engine :interp :fuel 0})) +(loop [i 0 acc 0] + (if (< i 1000000) + (recur (inc i) (wasm/call m "add" i 1)) + (println acc))) diff --git a/.dev/bench/scaling.clj b/.dev/bench/scaling.clj new file mode 100644 index 000000000..8eaad8583 --- /dev/null +++ b/.dev/bench/scaling.clj @@ -0,0 +1,71 @@ +;; Complexity-class probe: run the SAME number of operations against +;; collections of growing size. Flat wall-clock across sizes => the operation is +;; O(1)/O(log n) per call; time growing with the size factor => it is O(n). +;; +;; Reported as the ratio t(4n)/t(n). ~1 means constant, ~4 means linear. + +(def SIZES [1000 4000 16000]) +(def OPS 2000) + +(defn- ms [f] + (let [t0 (System/nanoTime)] + (f) + (/ (- (System/nanoTime) t0) 1e6))) + +(defn- probe + "build: size -> collection. op: (fn [coll size]) doing ONE unit of work." + [label build op] + (let [times (vec (for [s SIZES] + (let [c (build s)] + ;; warm + (dotimes [_ 50] (op c s)) + (ms #(dotimes [_ OPS] (op c s)))))) + ratios (vec (map (fn [a b] (if (zero? a) 0.0 (/ b a))) + (butlast times) (rest times))) + worst (if (seq ratios) (apply max ratios) 0.0) + verdict (cond (< worst 1.8) "O(1)/O(log n)" + (< worst 2.8) "sub-linear?" + :else "O(n) <-- LINEAR")] + (println (format "%-26s t=%-28s ratio=%-16s %s" + label + (vec (map #(format "%.1f" %) times)) + (vec (map #(format "%.2f" %) ratios)) + verdict)))) + +(def V (fn [s] (vec (range s)))) +(def M (fn [s] (into {} (map (fn [i] [i i]) (range s))))) +(def S (fn [s] (into #{} (range s)))) +(def L (fn [s] (apply list (range s)))) + +(println "=== cljw collection scaling ===") +(println (format "%d ops per size, sizes %s\n" OPS SIZES)) + +(println "-- vector --") +(probe "conj" V (fn [c _] (conj c 1))) +(probe "nth (mid)" V (fn [c s] (nth c (quot s 2)))) +(probe "assoc (mid)" V (fn [c s] (assoc c (quot s 2) 0))) +(probe "pop" V (fn [c _] (pop c))) +(probe "count" V (fn [c _] (count c))) +(probe "peek" V (fn [c _] (peek c))) +(probe "subvec 1..n" V (fn [c s] (subvec c 1 s))) +(probe "rest" V (fn [c _] (rest c))) +(probe "first" V (fn [c _] (first c))) + +(println "\n-- map --") +(probe "assoc (new key)" M (fn [c s] (assoc c (+ s 1) 0))) +(probe "get (hit)" M (fn [c s] (get c (quot s 2)))) +(probe "dissoc" M (fn [c s] (dissoc c (quot s 2)))) +(probe "count" M (fn [c _] (count c))) +(probe "contains?" M (fn [c s] (contains? c (quot s 2)))) + +(println "\n-- set --") +(probe "conj" S (fn [c s] (conj c (+ s 1)))) +(probe "contains?" S (fn [c s] (contains? c (quot s 2)))) +(probe "disj" S (fn [c s] (disj c (quot s 2)))) +(probe "count" S (fn [c _] (count c))) + +(println "\n-- list / seq --") +(probe "list conj" L (fn [c _] (conj c 1))) +(probe "list first" L (fn [c _] (first c))) +(probe "list rest" L (fn [c _] (rest c))) +(probe "list count" L (fn [c _] (count c))) diff --git a/.dev/bench/scaling_form.edn b/.dev/bench/scaling_form.edn new file mode 100644 index 000000000..6ed6ce2d6 --- /dev/null +++ b/.dev/bench/scaling_form.edn @@ -0,0 +1,24 @@ +(let [SIZES [1000 4000 16000] + OPS 2000 + ms (fn [f] (let [t0 (System/nanoTime)] (f) (/ (- (System/nanoTime) t0) 1e6))) + probe (fn [build op] + (vec (for [s SIZES] + (let [c (build s)] + (dotimes [_ 300] (op c s)) ; warm (JIT on clj) + (double (ms #(dotimes [_ OPS] (op c s)))))))) + V (fn [s] (vec (range s))) + M (fn [s] (into {} (map (fn [i] [i i]) (range s)))) + L (fn [s] (apply list (range s)))] + {:vec/conj (probe V (fn [c _] (conj c 1))) + :vec/nth (probe V (fn [c s] (nth c (quot s 2)))) + :vec/assoc (probe V (fn [c s] (assoc c (quot s 2) 0))) + :vec/pop (probe V (fn [c _] (pop c))) + :vec/subvec (probe V (fn [c s] (subvec c 1 s))) + :vec/rest (probe V (fn [c _] (rest c))) + :vec/seq (probe V (fn [c _] (seq c))) + :vec/next (probe V (fn [c _] (next c))) + :map/assoc (probe M (fn [c s] (assoc c (+ s 1) 0))) + :map/get (probe M (fn [c s] (get c (quot s 2)))) + :map/dissoc (probe M (fn [c s] (dissoc c (quot s 2)))) + :list/count (probe L (fn [c _] (count c))) + :list/conj (probe L (fn [c _] (conj c 1)))}) diff --git a/.dev/bench/traversal.clj b/.dev/bench/traversal.clj new file mode 100644 index 000000000..e921b399d --- /dev/null +++ b/.dev/bench/traversal.clj @@ -0,0 +1,64 @@ +;; Whole-collection traversal: does total time scale LINEARLY with n (4x per +;; step) or QUADRATICALLY (16x)? A vector `rest` that materialises would make +;; every seq walk over a vector O(n^2), which matters far more than subvec. + +(def SIZES [2000 8000 32000]) + +(defn- ms [f] + (let [t0 (System/nanoTime)] (f) (/ (- (System/nanoTime) t0) 1e6))) + +(defn- probe [label build run] + (let [times (vec (for [s SIZES] (let [c (build s)] (ms #(run c))))) + ratios (vec (map (fn [a b] (if (zero? a) 0.0 (/ b a))) + (butlast times) (rest times))) + worst (if (seq ratios) (apply max ratios) 0.0) + verdict (cond (< worst 6.0) "LINEAR O(n)" + (< worst 11.0) "superlinear" + :else "QUADRATIC O(n^2) <-- BAD")] + (println (format "%-30s t=%-30s ratio=%-14s %s" + label + (vec (map #(format "%.1f" %) times)) + (vec (map #(format "%.1f" %) ratios)) + verdict)))) + +(def V (fn [s] (vec (range s)))) +(def L (fn [s] (apply list (range s)))) +(def M (fn [s] (into {} (map (fn [i] [i i]) (range s))))) + +(println "=== whole-collection traversal (4x size per step) ===") +(println (format "sizes %s\n" SIZES)) + +(println "-- over a vector --") +(probe "reduce +" V #(reduce + 0 %)) +(probe "doseq" V #(doseq [x %] x)) +(probe "loop over (rest)" V #(loop [s (seq %) n 0] (if s (recur (next s) (inc n)) n))) +(probe "count (map inc)" V #(count (map inc %))) +(probe "count (filter odd?)" V #(count (filter odd? %))) +(probe "last" V #(last %)) +(probe "apply +" V #(apply + %)) +(probe "into [] " V #(into [] %)) +(probe "vec (map inc)" V #(vec (map inc %))) +(probe "reverse" V #(reverse %)) +(probe "sort" V #(sort %)) +(probe "transduce (map inc)" V #(transduce (map inc) + 0 %)) +(probe "into [] xf" V #(into [] (map inc) %)) + +(println "\n-- over a list --") +(probe "reduce +" L #(reduce + 0 %)) +(probe "doseq" L #(doseq [x %] x)) + +(println "\n-- over a map --") +(probe "reduce-kv +" M #(reduce-kv (fn [a k _] (+ a k)) 0 %)) +(probe "doseq over entries" M #(doseq [e %] e)) +(probe "into {}" M #(into {} %)) +(probe "keys + count" M #(count (keys %))) + +(println "\n-- building --") +(probe "into [] (range n)" identity #(into [] (range %))) +(probe "reduce conj []" identity #(reduce conj [] (range %))) +(probe "persistent! loop" identity + #(loop [t (transient []) i 0] (if (< i %) (recur (conj! t i) (inc i)) (persistent! t)))) +(probe "into #{}" identity #(into #{} (range %))) +(probe "into {} pairs" identity #(into {} (map (fn [i] [i i]) (range %)))) +(probe "apply str" identity #(apply str (repeat % "x"))) +(probe "str/join" identity #(clojure.string/join "," (range %))) diff --git a/.dev/convergence_campaign.md b/.dev/convergence_campaign.md deleted file mode 100644 index 868c48edc..000000000 --- a/.dev/convergence_campaign.md +++ /dev/null @@ -1,220 +0,0 @@ -# Convergence Campaign — drive cw v1 to feature-convergence (autonomous) - -> **CLOSED (2026-07-02).** Historical campaign record (2026-06-06); stages -> complete. Kept for inbound references; do not drive work from it. - -> **Status**: opened 2026-06-06; **Stage 0 DONE 2026-06-06** (5 SSOTs rebuilt: -> `core_coverage_gaps.md` re-run, NEW `v0_v1_feature_parity.md` + D-273 umbrella, -> `compat_tiers.yaml` Java scope +31A/+3C reservations, debt de-staled −5, -> `docs/works/` ladder seeded). **Load-bearing Stage-0.4 discovery: Phase B -> (concurrency, ADR-0090) is ALREADY IMPLEMENTED at HEAD** — STM/agent/locking/ -> real-OS-thread future all probe-green; landed 2026-06-05 (commits -> `7ac5fb1d`..`0a1fbb73`), i.e. it was already done when this campaign was -> written. So the handover's "open Phase B" target was stale on arrival; -> Stage 1.7 is re-scoped to **hardening** (D-242 residuals), not implementation. -> -> **This file is a DRIVING procedure, not a finished inventory.** Every Stage -> exit is a *mechanical predicate* (a script result or a count), so the -> autonomous loop self-advances with **no user touchpoint**. Where a design -> choice surfaces, the loop drafts+accepts an ADR inline (CLAUDE.md § ADR-level -> designs are handled inline); where a clj divergence surfaces, it is bug→fix -> OR `AD-NNN` (never floating). The ONLY user-owned items are F-NNN amendments -> and the 3 deferred CFP actions — both OUT of campaign scope. - -## Goal (the convergence definition) - -cw v1 is "converged" when all four hold, each mechanically checkable: - -1. **No v0→v1 functional regression** — every v0 bundled namespace / CLI - feature is either present in v1 or has a debt row with a live barrier - (`.dev/v0_v1_feature_parity.md`, built in Stage 0.2, has zero un-rowed - MISSING entries). -2. **The true-remaining inventory is complete and honest** — `debt.yaml` has - no stale defer, no over-claim (anti-D-177), no dup; every gap the sweeps - found has a row (Stage 0.4). -3. **Real-world pure-Clojure libraries load** — `docs/works/` (F-010) records - a ranked ladder of actual libraries `require`'d on cljw with pass/fail + - the blocking gap (Stage 0.5 seeds it; Stage 1.3 drives it up). -4. **Every SSOT reference chain is intact** — the Final Stage wiring audit - passes (no orphan / dangling / contradictory cross-reference). - ---- - -## Stage 0 — Inventory & SSOT rebuild (FIRST; all mechanical, all regenerable) - -Do these before any feature work. Each produces or refreshes an SSOT. - -- **0.1 clojure.core coverage** — re-run `.dev/core_coverage_gaps.md`'s regen - recipe (bb `ns-publics` ~647 diffed against cljw's var set). Output: the - refreshed missing-var list. Confirm each "missing" with a real `cljw -e` - probe (the recipe's false-positive caveat). -- **0.2 v0→v1 feature parity** — NEW file `.dev/v0_v1_feature_parity.md`. - Enumerate every `~/Documents/MyProducts/ClojureWasm/src/lang/lib/*.zig` - bundled namespace + `src/app/*` feature, map to v1 status - (present / partial / MISSING + debt-row id). **Seed (this scan's findings — - verify + complete in Stage 0.2, do not trust blindly):** MISSING in v1 = - `clojure.spec.alpha` + `clojure.spec.gen.alpha`, `clojure.core.reducers`, - `clojure.core.protocols`, `clojure.datafy`, `clojure.instant`, - `clojure.java.io` / `.shell` / `.process` / `.browse`, `clojure.main`, - `clojure.repl` + `clojure.repl.deps` (`add-lib!`), `clojure.stacktrace`, - `clojure.template`, `clojure.test.tap`, `clojure.uuid` (ns form), - `clojure.xml`; **app**: `deps.edn` resolution (`deps.zig`), project - scaffolding. v1-only additions (reverse direction) are OUT of scope. -- **0.3 Java-handling scope** — rebuild `compat_tiers.yaml` as the authoritative - "how far we handle Java" SSOT. v1 currently has only 22 Tier-A rows; v0's - compat_tiers + ADR-0013 carried the full A/B/C/D classification. Mine v0's - compat_tiers + the `clojure.java.*` libs + the `java.*` classes real - libraries touch; classify each Java class A/B/C/D with the cw-native - alternative for C/D. This is the SSOT the lib-compat ladder (0.5/1.3) reads - to decide "this lib needs `java.io.File` → Tier?". -- **0.4 Debt de-stale + defer re-eval** — run the `audit_scaffolding` 5-lens + - the anti-D-177 over-claim check on EVERY `debt.yaml` row (131 active): - re-evaluate each `blocked-by` / `Phase N target` / `deferred` predicate (flip - if the barrier dissolved), DELETE any row whose discharge text is a lie - (corpus/code does not back it), consolidate dups (e.g. this session's - Unicode-case row that duplicated the re-opened D-057 — caught + folded). - Output: an honest debt set + a count delta in the commit. -- **0.5 Real-world lib ladder** — create + populate `docs/works/` (F-010, - currently empty). Rank candidate libraries by **pure-Clojure degree** (zero - Java interop first), `require` each via the Stage-1.2 deps path (or `-cp` - until then), record `{lib, version, status, blocking-gap-row}`. Seed the - ladder head with small pure-Clojure libs (e.g. data/edn/string utilities); - the FIRST real Java-interop blocker each lib hits becomes a debt row read - against the Stage-0.3 Java tier SSOT. - -**Stage 0 exit (mechanical):** all five SSOTs regenerated this campaign; -`.dev/v0_v1_feature_parity.md` exists with zero un-rowed MISSING; `debt.yaml` -passes `audit_scaffolding` with zero over-claim/dup finding; `docs/works/` is -non-empty. The unified, dependency-sorted work-list for Stage 1 is materialized -from this data (the loop derives the order; it is NOT hand-fixed here). - ---- - -## Stage 1 — Blocker-free ordered execution (autonomous) - -The order below is the **seed**; the loop re-derives the true topological order -from Stage 0's data so that **every item's prerequisites land before it** (no -item blocks on a later one). Each item runs the standard per-task TDD loop -(CLAUDE.md § Autonomous Workflow): corpus- or e2e-backed, gate-green, committed. -The loop self-selects the next item by the order — **no user touchpoint**. - -1. **`resolve`-in-stdin regression** — `(resolve 'map)` returns nil via - `cljw -` (stdin) but `#'clojure.core/map` via `-e`. Blocks reliable - nREPL/cider eval. Fix first (small; the stdin eval path's ns/resolution - setup differs from `-e`). -2. **deps.edn source resolution** — re-derive v0's `deps.zig` cljw-clean - (`no_copy_from_v1`): `:paths` / `:deps` (`:local/root`, `:git/url`+`:git/sha`) - / `:aliases` / `-A:alias`. NO Maven/Clojars JAR (source-only, matching v0). - Unblocks the real-world lib ladder. -3. **Real-world lib ladder drive** — push `docs/works/` up the pure-Clojure - ranking; each new lib's first blocker spawns a fix (folds into the relevant - item below). This is the campaign's *coverage engine* — it surfaces the - highest-value missing pieces empirically, not by guessing. - **Committed-artifact form (2026-06-07): `test/conformance/verified_projects//`** — each - proven lib lands as a tracked `deps.edn` (`:git/url`+`:git/sha`) + `verify.clj` - exercise instead of a throwaway `-cp` probe, so (a) the dir list shows - at-a-glance which libs load, and (b) `scripts/verify_projects.sh` re-runs them - as a real-world-lib **regression sweep** (network; Phase-boundary / on-demand, - not per-commit). Method + how-to-add: `test/conformance/verified_projects/README.md`; the - PATTERNS / gap-taxonomy / coverage-raising know-how: - **`.dev/library_incorporation_playbook.md`** (read before a re-expansion). Grow - it one lib at a time; reconcile with `docs/works/ladder.md` (ladder = ranked - candidates, verified_projects = committed proofs). deps.edn-system gaps found - while doing this are fixed in-place (within cljw's control — `:git`/`:local` - resolution, NOT Maven JAR fetch); ADR-0101 + amendments record the policy. - - **PRIORITY + STAY directive (user 2026-06-07).** The next two libs to land - are **hiccup** and **honeysql** (in that order of tractability is fine). Once - the existing **9** verified (`medley`, `math.combinatorics`, - `data.priority-map`, `core.cache`, `potpuri`, `data.zip`, `qbits.ex`, - `core.unify`, `integrant`) **plus hiccup + honeysql** all verify (→ 11), this - library-incorporation campaign goes to **STAY** (paused, not abandoned). After - the STAY, the autonomous loop **self-selects the remaining work** (per CLAUDE.md - § The only stop's next-task rule + the F-010 quality-loop floor) — coverage - has plateaued, so the precision-raise shifts to quality work (tests, - robustness, error-path fidelity, the `quality-loop floor:` debt drain) and any - user-flagged feature, NOT more lib-probing. The known blockers for the two - priority libs (each definition-derived, F-013): - - **hiccup** → `java.net.URI` (`extend-protocol ToString java.net.URI` + - functional `to-uri`/`url-encode`). A java.net surface (runtime/java/net/URI.zig) - OR — since it is a real, implementable Java class (NOT a `clojure.lang.*` - internal, so ADR-0113 does NOT defer it) — a minimal URI value + the - ToString/url-encode path. Probe `test/conformance/verified_projects/hiccup` for the exact chain. - - **honeysql** → (a) **java.util.Locale** US/ROOT static fields + a Locale-arg - `String.toUpperCase`/`toLowerCase` overload (D-315; a host_instance surface was - designed+reverted 2026-06-07 — re-land with a GC-safe per-Runtime singleton: - gc.infra-alloc like empty_queue, OR root the rt slots), AND (b) **regex - lookahead `(?=…)`** — `honey.sql/dehyphen` uses `#"(\w)-(?=\w)"`; the regex - engine (`src/runtime/regex/`) rejects it (`unsupported syntax in cycle 1`). - Land (a)+(b) TOGETHER (anti-drip-feed) so honeysql verifies in one push. - Cross-ref D-315, D-314 (extend-via-metadata, separate/optional). -4. **Native `cljw.nrepl` cider ops** — prerequisite: **populate built-in var - metadata** (`:doc` / `:arglists` on core vars; `(meta (var map))` is nil - today — generate from `compat_tiers.yaml` / JVM source). Then implement - `info` / `complete` (uses `ns-publics`, already works) / `eldoc` / - `load-file` as native ops against cljw's own var tables (more tractable than - JVM cider-nrepl, which is `clojure.reflect` / `tools.namespace`-coupled). - Also close nREPL `*out*`/`*err*` per-session capture (D-118). -5. **v0→v1 bundled-lib backfill** — the Stage-0.2 MISSING list, in - pure-Clojure-degree order: `clojure.repl` (+`repl.deps` `add-lib!`) / - `clojure.template` / `clojure.stacktrace` / `clojure.uuid` ns / - `clojure.instant` / `clojure.spec.alpha`(+gen) / `clojure.core.reducers` / - `clojure.datafy` / `clojure.java.io`+`.shell` (gated on the Stage-0.3 Java - tier decision) / `clojure.xml` / `clojure.test.tap` / `clojure.main`. -6. **clj-parity sizable remainder** — D-057 Unicode case-fold (table or AD), - D-270 Java primitive arrays, D-086 record `__extmap` (F-003 structural), - re-matcher/re-groups, D-266 lazy-seq perf, D-267 `%c`, D-271 with-meta range. -7. **Phase B — concurrency HARDENING (ADR-0090)** — ~~implement~~ **already - IMPLEMENTED at HEAD** (Stage-0.4 discovery): real-OS-thread `future` - (`std.Thread.spawn`), MVCC STM (`stm/ref.zig` + `concurrency/lock_tx.zig`), - `agent`/`send`/`await`/`agent-error`, `locking` monitor, atom CAS, the - ThreadGcContext root-publication GC handshake — all probe-green. This item is - now **hardening, not construction** (D-242 anchor): close the concurrency - torture/race edges (D-244 #4 family, D-250..D-253, D-258 dormant flake), wire - `pmap`/`pcalls` to the work-pool for genuine parallelism (D-224, result already - correct), pick the LazySeq.force mutex disposition (D-046). `future`/`promise` - are already async. NOTE: `delay` once-lock + STW-rendezvous fixes are still - trickling (git log), so this is real, not closed. - ---- - -## Final Stage — Wiring & reference-chain audit (LAST; mechanical) - -Run after Stage 1 converges. A graph audit that every SSOT cross-reference and -every code dependency edge is intact: - -- **`@import` zone graph** — `scripts/zone_check.sh --gate` (no upward imports; - baseline 0). -- **SSOT cross-reference integrity** — every `D-NNN` / `AD-NNN` / `O-NNN` / - `ADR-NNNN` / `F-NNN` / corpus-name / `compat_tiers` fqcn referenced in code - comments, debt, ADRs, rules RESOLVES to a defined node (`check_debt_id_refs` - extended to all ledger ID classes; add a cross-ref check if absent). -- **marker ↔ ledger sync** — `PROVISIONAL:` ↔ `feature_deps.yaml` + debt - (`check_provisional_sync`); `PERF:` ↔ `optimizations.md`; `GC-ROOT:` ↔ - `gc_rooting.md`; every marker has a live ledger row and vice-versa. -- **aspirational-rule check** — every `.claude/rules/*.md` that declares an - enforcement has a backing script (the zwasm `audit_table_sync` pattern); a - rule promising a gate with no script is a finding. -- **dangling/contradiction sweep** — a fact stated once (a tier, an F-NNN - invariant, a divergence) is referenced consistently everywhere; no node is - orphaned (defined, never referenced) or contradicted (two SSOTs disagree). - -**Final exit:** all of the above are scripts in `test/run_all.sh` (or run -clean on demand), and the audit produces zero finding. The reference chain is -then self-verifying for every future session. - ---- - -## Autonomy contract (the "no user intervention" guarantee) - -- Every Stage/item exit is a **mechanical predicate** (script exit, count, - corpus-green) — never a judgment call that pauses for input. -- A surfaced **design choice** → inline ADR (draft+accept, DA fork at depth ≥ 2). - A surfaced **clj divergence** → bug-fix OR `AD-NNN`. A surfaced **structural - plan** → deferred to its owning item with a debt row (F-003). -- The loop **self-selects** the next item from the Stage-0-derived order. The - phrases "ask the user / awaiting / which should I do next" are forbidden - (CLAUDE.md § The only stop — Direction-ask smell). -- User-owned exceptions (do NOT attempt; surface only): F-NNN amendments; the - 3 deferred CFP actions (`DEFERRED_USER_ACTIONS.md`); `.claude/rules/*` edits - (permission-blocked). diff --git a/.dev/cw_v0_parity_and_gap_plan.md b/.dev/cw_v0_parity_and_gap_plan.md deleted file mode 100644 index 1c3f96703..000000000 --- a/.dev/cw_v0_parity_and_gap_plan.md +++ /dev/null @@ -1,169 +0,0 @@ -# cw v0 parity snapshot + gap-incorporation plan - -> **CLOSED (2026-07-02).** Historical snapshot (2026-05-29) from the -> phase-queue era. Kept for inbound references; do not drive work from it. - -> **Purpose** (user directive, 2026-05-29): persist the cw-v0-vs-cw-v1 -> coverage/parity analysis as a reference, and prepare — with **ROADMAP -> amendment in scope** — how the loop will incorporate the missing -> pieces. The user's framing (verbatim): -> -> > 「次から確実にその不足部分のうち、cw v1に盛り込むべきもの(大半が -> > そうだが、そのまま真似しろというわけではない、cw v1のつくりの上で、 -> > より良い設計・順序で)しっかり先程のレポート表示はどこかに書き出して -> > 参考資料とした上で、どう取り組むかをロードマップ改変も視野に入れて -> > 取り組ませる準備をしておいてください。」 -> -> So: **most v0 gaps SHOULD be incorporated, but NOT copied as-is** — -> redesigned on cw v1's architecture with better design + ordering. -> -> ## How the loop uses this doc (F-003 discipline) -> -> This is **foresight, not decision**. Per F-003 + `.dev/principle.md` -> Structural-imagination phase: the per-gap "cw-v1 redesign direction" -> + "ordering" below are the **owning phase's design input**, not a -> seized decision. When a gap's owning phase/row opens, the owner reads -> this doc, then decides the concrete shape (Devil's-advocate per -> CLAUDE.md if depth ≥ 2) and **amends the ROADMAP §9 row in place per -> §17**, referencing this doc. Gaps with NO existing ROADMAP row -> (`deps`/`test`/`--list-vars`/some namespaces) are where amendment is -> most needed — they get NEW §9 rows minted at incorporation time. -> -> Read at cold-start (added to handover reading order). Amend this doc -> in place as gaps are incorporated (move a row to "Done" with the -> landing commit). - ---- - -## 1. Parity snapshot — cw v1 vs cw v0 (2026-05-29) - -**Headline**: cw v1 ≈ **60-70% of cw v0's surface in ~half the LOC** -(43K vs 87.5K). Numeric tower / GC / error-UX are equal-or-ahead; v0's -production-mature **JIT, rich nREPL/REPL, deps/test toolchain, Wasm -Component output** are **intentionally ordered later** (F-010). cw v1's -native deftype + dual-backend oracle give a structurally higher ceiling -(parity-PLUS). Position: Phases 1-13 DONE, Phase 14 (v0.1.0) IN-PROGRESS -(release HELD per user), Phases 15-20 PENDING. - -> Honest caveat: v0's 651/706 clojure.core vars (Zig bootstrap) and cw -> v1's "158 .clj defs + 211 primitive leaves (overlap)" are not directly -> comparable. The % is a per-dimension estimate; var-exact parity is -> quantified later via the clojuredocs differential (F-010 quality loop). - -### 1a. Core language - -| Dimension | cw v0 | cw v1 | Status | -|---------------------------|------------------------------------------------|----------------------------------------------|------------------------------------------------| -| Zig LOC | 87.5K | 41.5K | half, in progress | -| clojure.core vars | 651/706 (92%) | ~158 .clj + 211 primitive | 🟡 D-134 backlog | -| special forms | full + JVM-compat | 16 (incl. `binding`) | 🟡 | -| namespaces | ~30 | 12 | 🟡 spec/math/io/repl/reducers absent | -| host classes / interop | 7 native + `.`/`new` | 43 reserved + `.` | 🟡 time/net/crypto backing absent (D-105/D-106) | -| backends | Register-IR VM + JIT | tree_walk (prod) + VM (partial) | ⭐ dual-backend differential oracle | -| GC | MarkSweep | MarkSweep + arena + free-pool + root-set | ✅ equal (no generational either side) | -| numeric tower | BigInt/Ratio/BigDecimal | auto-promote + universal `=`/`compare` | ✅ equal | -| lazy-seq | full chunked | Layer-2 (map/filter/range/iterate) | 🟡 infinite range etc. (D-134(b)) | -| metadata (with-meta/meta) | yes | none | ❌ D-075 | -| STM/concurrency | atom/future/promise/agent/pmap (no ref/dosync) | atom/future/promise/delay + STM Ref skeleton | 🟡 Phase 15 | -| macros | full + user defmacro | user defmacro | 🟡 `&form`/`&env` D-111 | -| Wasm FFI | 523 opcodes / Wasm 3.0 / WIT | none | ❌ F-001, F-010-deferred | - -### 1b. CLI - -| Feature | cw v0 | cw v1 | Status | -|----------------------------------------------|------------------------------|--------------------------------------|-------------------------------------| -| `-e` / file / `-` stdin | ✅ | ✅ | ✅ | -| `repl` | ✅ line_editor (1196 LOC) | 🟡 line-buffered (D-116) | 🟡 | -| `nrepl --port` | ✅ 15 ops (history)/now stub | 🟡 4 ops | 🟡 | -| `build` (self-contained bin) | ✅ | ✅ + embedded-trailer self-run | ✅ | -| `render-error` (post-mortem) | ❌ | ⭐ `CLJW_ERROR_LOG` EDN replay | ⭐ | -| `--compare` (dual-backend diff) | ❌ | ⭐ TreeWalk vs VM, exit 1 on mismatch | ⭐ | -| `component build` (Wasm CM) | ❌ | ❌ (14.12 `[ ]`, zwasm-v2 gate) | differentiation core, both unbuilt | -| `test` / `deps` / `-m` / `new` / `--version` | ✅ (Clojure CLI toolchain) | ❌ | ❌ v0 utility, no cw-v1 ROADMAP row | - -### 1c. nREPL richness (v0 full = git `da201d9~1`, 1818 LOC / 15 ops) - -| op | cw v0 | cw v1 | CIDER need | -|-------------------------------------|-------|-------------|--------------| -| clone / close / describe / eval | ✅ | ✅ | required | -| stdout/stderr capture | ✅ | ❌ (D-118) | **required** | -| complete / info / eldoc / lookup | ✅ | ❌ (D-117c) | high | -| ls-sessions / multi-session | ✅ | ❌ single | high | -| load-file / macroexpand / interrupt | ✅ | ❌ (D-117) | mid | - -Verdict: v0 = CIDER-class; cw v1 = handshake-only (connects, but -`(println …)` output never reaches the client). bencode codec done; the -gap is ops + per-session `*out*`/`*err*` dynamic binding + thread-per-session. - -### 1d. Planned-UX scorecard - -- **Error UX (P6)** → ⭐ cw v1 *exceeds* v0 + the plan: carat-pointer + - EDN + `render-error` + `CLJW_ERROR_FORMAT`/`LOG` + `with-context`, - error_catalog SSOT. **cw v1's strongest area.** -- **REPL line-editing** → 🟡 v0 full (history/completion/multi-line); - cw v1 line-buffered (D-116). -- **Learning docs** → JA chapters permanently dormant (ADR-0025/F-007); - `docs/works/` capability ledger planned (F-010), grows post-M. -- **`--list-vars` / tier introspection** → SSOTs (compat_tiers/placement) - ready; command unbuilt (both sides). -- **AOT self-contained binary** → ✅ both (cw v1 = ADR-0034). -- **Wasm Component output** → ❌ the differentiation core; v0 none, cw v1 - 14.12 deferred (zwasm-v2 gate). - -### 1e. JIT - -v0 = `src/engine/vm/jit.zig` 721 LOC ARM64 PoC (hot integer loop, -`JIT_THRESHOLD=64`, leaf C-ABI, deopt) — but `fib_recursive` 1.0x -(never reaches a hot loop in real code). cw v1 = none. Original plan = -Phase 17 broad JIT; F-010 re-cut = M = "cw-v0-程度 JIT" (narrow ARM64 -integer-loop, ~700-1000 LOC). D-133 = the coverage-floor ordering owner. -F-001 = zwasm v2 has its own JIT (Phase 16, territorial overlap unsolved). - ---- - -## 2. Gap-incorporation plan - -For each gap: **incorporate?** + **cw-v1 redesign direction** (NOT a -v0 copy — the better shape on cw v1's architecture) + **ordering** + -**ROADMAP-amendment hook** + **debt row**. Direction is foresight; the -owning phase decides the concrete design (F-003). - -| # | Gap | Incorporate | cw-v1 redesign direction (not v0 copy) | Ordering | ROADMAP hook | Debt | -|-----|----------------------------------------------------------------------|--------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------|---------------------------------------------------------------------------------------------|-------------------| -| G1 | nREPL richness (ops + out/err + multi-session) | **Yes (CIDER-class)** | Build per-session `*out*`/`*err*` on the **just-landed dynamic-var infra** (`binding` + `*error-context*` pattern); thread-per-session via Phase-15 concurrency; `complete`/`info`/`eldoc` over the **compat_tiers/placement SSOT** (shared with `--list-vars` G11) — not v0's ad-hoc op handlers | **Phase 15** (concurrency unblocks it) | Amend §9.17 (Phase 15) — add nREPL-richness rows, OR a dedicated post-15 row | D-117, D-118 | -| G2 | REPL line-editor | **Yes** | `src/app/line_editor.zig`: TTY raw-mode, history-persist, completion via the SSOT (G11). v0's 1196-LOC editor is the spec; cw v1 trims to the ergonomic core first | Post-M quality loop / opportunistic | New §9 row in the quality-loop phase | D-116 | -| G3 | JIT (narrow ARM64) | **Yes (M gate)** | **Order-first redesign**: superinstruction/fusion pass → coverage-floor green (interop `.`/`new` D-130, lazy-seq, D-134 core) → THEN the narrow JIT — so cw v1 avoids v0's `fib 1.0x` trap (hot loop reachable in real code). NOT v0's 721-LOC PoC verbatim | **Post-Phase-15, M window** | Re-aim Phase 17 earlier per F-010; amend §9 JIT rows; resolve zwasm-v2 JIT overlap (F-001) | D-133 | -| G4 | metadata system (with-meta/meta + reader `^{}` + IObj/IMeta) | **Yes** | **Split the surface**: (a) narrow `^:keyword` reader-metadata for `def` flags (`^:dynamic`/`^:private`/`^:macro`) — Form-time, NOT runtime value-metadata, unblocks user dynamic vars cheaply; (b) full runtime value-metadata (HeapHeader slot + IObj/IMeta) — needs the F-004 NaN-box meta-slot decision. (a) can land well before (b) | (a) opportunistic; (b) focused cycle | (a) new §9 row; (b) **may need an F-004 amendment (user-owned)** for the slot | D-075 | -| G5 | host backing impls (time / net / crypto) | **Yes** | TypeDescriptor reservations exist; land neutral `runtime/time/` + `runtime/net/` + `runtime/crypto/` impls per **F-009** (impl-neutral, shared across Java/cljw/Clojure surfaces) | Quality loop / opportunistic | Focused §9 rows (host-stdlib waves) | D-105, D-106 | -| G6 | `deps` / `test` / `-m` / `new` toolchain | **Selective** | `cljw test` (run `test/clj/` conformance) = high value, build on existing test layer. `deps` (deps.edn git/maven) = big; scope-cut or defer (cw v1 may favor a leaner module story, `modules/` already exists). `-m`/`new` = conveniences. **NOT a v0 `deps.zig` port** | Quality loop | **NEW §9 rows (no current rows)** — primary ROADMAP-amendment site | file new D-NNN | -| G7 | namespaces (math / spec.alpha / java.io / reducers / instant / repl) | **Selective** | `clojure.math` (Pattern-A, cheap) first; `clojure.spec.alpha` (large — survey `spec.alpha/` + `malli/` per textbook_survey, cw-v1-idiomatic); `clojure.java.io` over `runtime/io/`. Add by real-corpus demand (F-010 quality loop), not bulk | Quality loop (demand-driven) | New §9 rows per namespace | file per-ns D-NNN | -| G8 | Wasm Component output (`cljw component build`) | **Yes (differentiation core)** | F-001/zwasm v2 integration per F-008 spec; **F-010 orders it AFTER the quality loop** (Phase 16). Not v0 (v0 had none); the design is the zwasm-v2 consumer shape | **Phase 16** (post-M, zwasm-v2 ready) | §9.18 Phase 16 (already placeholder) | D-036/037/038 | -| G9 | clojure.core remaining (lazy (b) + breadth) | **Yes (recurring)** | Cluster-by-cluster over existing primitives; lazy forms with the lazy-seq Layer-2 wiring | Recurring quality-loop | §9.16 row 14.13 follow-ups + quality-loop rows | D-134 | -| G10 | multimethods breadth / `&form`/`&env` | **Yes** | `&form`/`&env` injection into the user-macro path (decide Value shapes for Form/Scope); multimethod breadth audit | Opportunistic / test-corpus-driven | Follow-up rows | D-111 | -| G11 | `--list-vars` / tier introspection | **Yes** | `cljw --list-vars` over `compat_tiers.yaml` + `placement.yaml`; shared completion source for G1 nREPL + G2 REPL. NOTE 2026-08-04: `--list-vars` is still unbuilt, but its data source now exists for real — ADR-0178 made `placement.yaml` a generated, total, gated index derived from `(cljw.internal/__dump-placement)`, so the CLI is a formatting job over `src/runtime/introspect.zig` rather than a yaml-plumbing one | Quality loop | New §9 row | file new D-NNN | -| G12 | generational GC | **Defer** | Both sides future (ROADMAP §89.2); cw v0 reached maturity without it. Not a near-term gap | Far future | §89.2 reserve | (none) | - -### Cross-cutting ordering note - -The single highest-leverage observation: **G1 (nREPL out/err) + G4a -(reader `^:dynamic`) build directly on this session's dynamic-var work** -(`binding` + `*error-context*` + `Env.on_deinit_hook`). G3 (JIT) is -gated behind the coverage floor (G9 + interop). So a natural sequence -toward M: finish G9/D-134 lazy + interop coverage → Phase 15 (unblocks -G1) → superinstruction/fusion → G3 JIT → M → quality loop (G5/G7/G11 + -`docs/works/`) → G8 (zwasm v2). G4/G6/G2 slot into the quality loop as -demand surfaces. - ---- - -## 3. Cross-references - -- Source analysis: this session's 2 parity subagents (2026-05-29); - cw v0 evidence in `~/Documents/MyProducts/ClojureWasm/` - (`DIFFERENCES.md`, `README.md`, `src/engine/vm/jit.zig`, - git `da201d9~1:src/app/repl/nrepl.zig`). -- F-010 (`.dev/project_facts.md`) — the interim-goal re-cut this plan - serves; F-001/F-004/F-006/F-008/F-009 — the decreed constraints. -- `.dev/structure_plan.md` — sibling foresight doc (directory tree). -- `.dev/debt.yaml` — the per-gap tracking rows. -- ROADMAP §17 — the amendment policy each incorporation follows. diff --git a/.dev/debt.yaml b/.dev/debt.yaml index 6b89cf79d..547c36fbf 100644 --- a/.dev/debt.yaml +++ b/.dev/debt.yaml @@ -24,6 +24,191 @@ # md-table-align no longer pads it to ~800 KB of whitespace. active: + - id: "D-587" + status: |- + open (2026-09-09). USE-AFTER-FREE, user-reachable panic. Re-registering a + deftype/defrecord name frees a TypeDescriptor that live instances still + point at. `type_descriptor.zig registerType` keys `rt.types` by the SIMPLE + name, and on re-registration does `fetchRemove(name)` then frees the old + descriptor (each field_layout entry name, the layout slice, fqcn, + defining_ns, method_table entries, protocol_impls, then + `gpa.destroy(old.value)`). Every constructed instance holds + `inst.descriptor` as a raw pointer to it, so `print.zig + printTypedInstance` then reads `field_layout` out of freed memory: the + symptom is `thread panic: integer overflow` (a garbage field index or + layout length), the defect is the free. + Repro (ReleaseSafe v1.14.3): `(ns aaa) (defrecord Point [x y]) (def a + (->Point 1 2)) (ns bbb) (defrecord Point [x y]) (def b (->Point 1 2)) + (prn (pr-str aaa/a))` panics; printing `bbb/b` is fine. Variants: two ns + with DIFFERENT names is fine; two ns SAME name panics; SAME ns redefining + the same name panics (this is the ordinary REPL / `require :reload` path + `dev/user.clj` depends on); same name with a different field count panics; + the deftype form raises `WriteFailed` instead, same root. + TWO separable defects: (1) the registry is keyed by simple name, so + `aaa/Point` and `bbb/Point` collide with no redefinition at all (clj gives + each ns its own class); (2) re-registration frees a descriptor live + instances reference. + ADR-0019 forbids a panic on a user-reachable path, and this one is reached + by re-evaluating a defrecord form. It also caps the native-suite migration: + `test/clj/suites/` shares ONE image, so no two suites may define the same + type name (worked around 2026-09-09 by renaming defrecord_test's record to + DPoint; 45 type definitions across 103 suites, no other duplicate today). + Found while migrating phase7_defrecord.sh to a native suite. + Card `[CLJW-TD-UAF]` 20260909211032-6471bc49. + category: "correctness" + quality_floor: "correctness: a user-reachable panic from an ordinary redefine" + barrier: |- + Needs an ADR: the free is deliberate (comments cite row 7.7 cycle 5 and + D-190 / ADR-0068) and exists to avoid leaking on redefine, so removing it + trades the leak back. Candidate shapes: key `rt.types` by the qualified + name when `defining_ns` is present and RETIRE rather than free a replaced + descriptor (a leak bounded by redefinition count, which is what clj does + with the old class), or make descriptors GC-managed so a descriptor dies + only when no instance references it. Devil's-advocate fork is mandatory at + this depth. Not blocked on anything external: the repro is one file and the + owning function is ~70 lines. + last_reviewed: "2026-09-09" + - id: "D-586" + status: |- + open (2026-09-05, ADR-0195 trigger). UPSTREAM: zwasm/D-584. zwasm's JIT + calls `computeStackLimit` on EVERY invocation, and glibc's stack query + for the process's INITIAL thread parses /proc/self/maps: 16.5-18.9 us per + `wasm/call` on Linux against 392 ns on `:engine :interp` and 184 ns for a + Clojure fn call (.dev/wasm_percall_findings.md; issues #13/#14/#15). + Measured 2026-09-05, engine_thread_matrix.clj, 20k calls per cell: + JIT initial 16547 ns, JIT worker 1181, interp 399/413. NOT fixed in + zwasm v2.6.0 (computeStackLimit still per invocation on upstream main: + engine/codegen/shared/entry.zig 248, 272; entry_buffer_write.zig 86). + cljw's answer is ADR-0195 (the runtime runs off the initial thread; the + geometry decision stands on its own, this row only tracks the upstream + cost). Residual on a non-initial thread: ~570 ns of the 1181 is STILL + this row (zwasm ADR-0209 measures the glibc query at 561-578 ns there); + zwasm/D-585 + cljw's own double name-resolve in wasmCallFn is ~200 ns. + Not filed upstream: their tracker already carries it; a short report + with the embedder numbers needs a user nod ([ZWASM-PERCALL-TRACK]). + category: "wasm" + barrier: |- + zwasm memoises `computeStackLimit` per thread (a threadlocal cache in + engine/codegen/shared/stack_limit.zig, or the query hoisted out of the + per-invocation path). Re-check at every pin bump by reading upstream + engine/codegen/shared/entry.zig; the measurement device is + `./zig-out/bin/cljw .dev/bench/ffi_boundary/engine_thread_matrix.clj` + (on demand, never gated: a timing ratio in the gate is a flake generator, + test_taxonomy.md). When the barrier dissolves, retire ADR-0195's trigger + paragraph (the decision survives) and re-measure the residual so + [CLJW-WASM-ENGINE-DEFAULT] can be re-decided on the new numbers. + first_raised: "2026-09-05" + last_reviewed: "2026-09-05" + refs: |- + ADR-0195; ADR-0209 (zwasm); .dev/wasm_percall_findings.md; + .dev/bench/ffi_boundary/README.md; kanban [CLJW-WASM-WORKER-THREAD] + 20260905012150-05dc6702, [ZWASM-PERCALL-TRACK] 20260905005855-3d34a890. + - id: "D-585" + status: |- + open (2026-08-22, found landing ADR-0192). zwasm's JIT MISCOMPILES a + ZERO-RESULT export outside a narrow window. It does not fail to build and + it does not downgrade — it traps at invoke. + + Measured over empty-body exports, arities 1-5, every i32/f64 parameter + mix (40 shapes). A void export runs on `:jit` only when: + - its arity is <= 1, OR + - its arity is 2-3 AND no parameter is floating-point. + Every other void shape traps, including all-GPR `(i32,i32,i32,i32) -> ()`. + NON-void was clean at every arity and mix tested. All 40 run on `:interp`. + + Minimal discriminators, identical parameters and empty bodies, differing + only in result arity: + (func (export "nop_void") (param i32 f64)) -> TRAPS + (func (export "nop_ret") (param i32 f64) (result f64) …) -> ok + (func (export "gpr4_void") (param i32 i32 i32 i32)) -> TRAPS + (func (export "gpr4_ret") (param i32 i32 i32 i32) (result i32) …) -> ok + Neither body touches memory, so the discriminator is the signature alone: + the JIT's invoke dispatch has no entry for these (params -> no result) + combinations. + + Three things make this worse than a niche gap. + + First, the shape is not niche, it is what an in-place numeric kernel IS. + Verified against real raster (org.replikativ/raster) kernels compiled to + wasm: `residual-add!` is `(i32,i32,i32,i32) -> ()` and traps; + `scale-inplace!` is `(i32,f64,i32) -> ()` and traps. Both return exactly + the JVM values on `:interp`. Any scale / axpy / SGD-step taking a pointer, + a length and a scalar lands in the trapping region. + + Second, the all-GPR half needs no floating point at all, so "avoid FP + params" is not a workaround — arity 4 alone is enough. + + Third, `.auto` does not save the caller. Its documented contract is a + transparent downgrade when the JIT cannot BUILD a module; these modules + build and then trap at invoke, so the fallback never fires and `.auto` + behaves exactly like `.jit`. Worth stating plainly: `.auto` protects + against build failure, not against miscompilation. + category: "correctness" + barrier: |- + Upstream. The `from_cljw_NN` / `to_cljw_NN` co-dev channel with zwasm has + ended (`.dev/zwasm_capabilities.md`), so this cannot be handed over the + way the 2026-06 JIT gaps were; it needs either a zwasm patch carried on + the pin or a cljw-side signature-shaped `.interp` route, and the second is + the shim the capabilities doc says cljw deliberately does not build. + + Workarounds for a caller today, in preference order: give the export a + result (the same parameters with any result run on the JIT), or load that + module with `{:engine :interp}`. + + Not blocking ADR-0192: the memory surface itself is engine-independent, + and `test/e2e/fixtures/wasm/mem_kernel.wat` documents why its `scale_f64` + returns the new sum instead of being void. + Pinned by the gap-lock test in `src/runtime/cljw/wasm/engine.zig` + ("a zero-result export traps on the JIT beyond a narrow window"), which + FAILS if zwasm ever fixes it — so the row cannot rot unnoticed. + RE-NARROWED 2026-09-06 (ADR-0196): the default stays `.auto` (no fixed + engine is trap-free: the interpreter traps v128 bodies), a trap on this + shape now names the export, its signature, the engine and the remedy + (row `jit_void_window` in `runtime/cljw/wasm/gaps.zig`, whose deletion + rides the pin bump that flips the gap-lock), and `(wasm/engine h)` + reports the selection. The upstream asks (the miscompile itself, and + `.auto` falling back on dispatch refusal) wait on a user nod + ([ZWASM-PERCALL-TRACK]). + first_raised: "2026-08-22" + last_reviewed: "2026-09-06" + refs: |- + ADR-0192 (the memory surface that surfaced it); zwasm ADR-0200 (`.auto` / + `:engine`); engine.zig's sibling SIMD gap-lock test; to_cljw_04 (the + 2026-06 residual FP-shape gaps, of which this is the un-reported tail). + - id: "D-577" + status: "open (2026-08-12, ADR-0186 residual). The three new layers are OPEN but shallow: golden covers 22 programs, property covers map/set/vector only, and only ONE file has been swept for mutants (vector.zig, 8 of 141 candidates), so the mutation score of this suite is still unmeasured. 120 of 265 src files carry no inline test at all." + category: "testing" + barrier: |- + Not a design question, a queue. Drain in this order, because each step + tells you where the next one should point: + 1. Widen the sweep past vector.zig (`scripts/mutation/run.sh + --targets-from .dev/mutation_targets.txt --budget 15`) and commit + the report. One file swept is not a measurement of a suite. + 2. Turn each survivor into a test in the layer that should have caught + it — a survivor in a comparison is usually a missing property, a + survivor in a rendering path is usually a missing golden case — then + re-run THAT mutant with `--ids` and watch it die. Rule out + equivalence first: two of the first sweep's three survivors could not + be killed by any test and are now registered in + `.dev/mutation_equivalent.jsonl` with their proofs. + 3. Extend Layer 7 past the collections. The obvious next law — + `read-string` after `pr-str` is the identity — canNOT be written at + the Zig level as the code stands: `reader.readAll` yields `Form` + (syntax) while `print.printResult` consumes `Value` (runtime), so + the two are not inverses of each other and nothing in between is + exposed. Either drive it through the evaluator (making it a + Clojure-level generative test, seeded so the gate stays + deterministic) or take `serialize.serializeChunk` / + `deserializeChunk`, which ARE a genuine inverse pair — at the cost + of a generator for well-formed BytecodeChunks, which is the real + work in that option. + 4. `scripts/check_adr_history.sh` fails on 101 pre-existing ADRs and is + wired into NO gate. Either wire it and fix them, or delete it; a + check nothing runs is a check nobody trusts. + Do NOT chase the 120 untested files as a number: the sweep ranks them by + whether an unconstrained line there would matter, which a file count + cannot. + last_reviewed: "2026-08-12" - id: "D-574" status: "open (2026-08-06, ADR-0184 residual). reify's anonymous TypeDescriptor is trackHeap'd per EVALUATION (lang/primitive/protocol.zig makeReifyDescriptor path) — the same immortality shape ADR-0184 killed for Functions, in miniature: a reify form evaluated in a loop grows persistent_marks + leaks a TD per iteration. The fix shape mirrors ADR-0184: make the anonymous TD (or a TD cell) collectable, instance-mediated trace already exists (traceReifiedInstance); audit method_table readers for a dead-TD window first (freeReifyDescriptor currently only touches names)." category: "gc" @@ -135,135 +320,12 @@ active: wired. Also distal: closure trackHeap accumulation is a leak-by-design for long-lived processes (Functions are never swept) — surface when an nREPL/server workload shows it. - category: perf + category: "perf" barrier: |- quality-loop floor: GC cost model. Drain when touching the tree_walk collect trigger or when a long-lived-process profile shows the stack-scan / waypoint-growth cost. last_reviewed: "2026-08-05" - - id: "D-568" - status: |- - open (2026-08-04, ADR-0159 amendment 1 § implementer-verification item (a)). - `Opened.dropResource` returns `NoResourceTable` for zwasm's `.single` - (non-WASI-P2) component variant, and cljw maps every zwasm invoke/drop - failure to the catch-all `wasm_trap` — "WebAssembly module trapped (e.g. - divide-by-zero, out-of-bounds, or an unreachable instruction)". So a - resource-bearing single-module component reports a MISLEADING cause: the - component does not have a resource table at all, which is a structural - property, not a trap. Needs its own Code with a template naming the real - condition. Not fixed in the amendment-1 cycle because no fixture in the - tree reaches it (`resource_counter` is a WASI-P2 graph), so a fix would - have shipped untested — the exact false-positive-discharge shape - `clj_diff_sweep.md` Discipline 1 forbids. Fix together with a `.single` - resource fixture, or when zwasm grows a resource table for that variant. - category: "wasm" - barrier: |- - quality-loop floor: error-message fidelity. No blocker — needs a `.single` - component fixture carrying a resource so the new Code is exercised rather - than asserted. - first_raised: "2026-08-04" - last_reviewed: "2026-08-04" - refs: |- - .dev/decisions/0159_wasm_resource_lifecycle.md § Amendment 1 (the - verification item that raised it); src/runtime/cljw/wasm/component.zig - (`invokeWithSig`, `resourceDropFn` — both map to `wasm_trap`); - zwasm src/api/component.zig `dropResource`. - - id: "D-566" - status: |- - open (2026-07-29, ADR-0175 deferred-hardening row). Alloc-side - unregistered-mutator assert: `gc_heap.alloc` (and `pushAnalysisRoot`) - should assert `root_set.is_registered_worker or ` so the next pre-register (or cap-fallthrough) mutator is a - caught bug instead of a ~1.5% GP-fault flake. Survey done 2026-07-29: - PRODUCTION is assert-clean (only main + the three registered worker - families allocate; nrepl/http serve serially on the calling thread), but - in-tree unit tests alloc from unregistered test threads (lazy_seq force- - race workers at lazy_seq.zig:461, plus the gc_heap/root_set churn tests - — each site needs registration or an exemption), and "owning main - thread" needs a per-GcHeap `main_thread_id` stamped at init. - category: "concurrency" - barrier: |- - Discharges by: the test-audit sweep (register or exempt every - unregistered-alloc test thread) + the per-heap main-thread-id stamp + - the assert itself, landed together. Cross-ref ADR-0175 § Consequences. - last_reviewed: "2026-07-29" - - id: "D-565" - status: |- - open (2026-07-28, external-contributor reproducibility audit — Discussion - #11 prompted a fresh-clone onboarding sweep). A stranger's `git clone` builds - + tests + gets full 3-OS CI gating with ZERO host-specific setup (see WORKS - below); everything that breaks is maintainer-side tooling, and the real - liability is DOC STALENESS, not broken machinery. Gaps: - (1) DISCHARGED 2026-08-04 — retired co-dev dir name `zwasm_from_scratch` - (real dir = `~/Documents/MyProducts/zwasm/`) fixed in `.claude/CLAUDE.md`, - `.claude/rules/handover_framing.md`, both wasm-component e2e headers, and - `scripts/run_wasm_gate.sh`. That last one was NOT doc rot: its skip guard - tested `[ ! -d ../zwasm_from_scratch ]`, so the runner had been exiting 0 - without running since the rename. The guard is gone entirely — the zon - pins zwasm by tag URL, so no sibling checkout is required and the - `-Dwasm` build is the real predicate. Verified green after the fix. - Historical VERBATIM user quotes in `project_facts.md` and closed ADRs - keep the old name (they are a record of what was said). - (2) DISCHARGED 2026-08-04 — the sweep found THREE dead reference paths, - not one (`zig/`, `wasmtime/`, `openjdk24/`). Zig stdlib now points at - the pinned toolchain's own `$(zig env | grep std_dir)`, which is - version-exact by construction and present wherever the project builds — - strictly better than a clone, since the clone this file named was - post-0.16 master and ADR-0090 §Context records a survey misled by that - skew. `wasmtime/` + `openjdk24/` are multi-GB and now carry - `` plus their clone command. `scripts/check_reference_clones.sh` - (gate step `reference_clones`) fails on any claimed-live path that is - absent, and on an on-demand row missing its command; it skips entirely - when `~/Documents/OSS` does not exist, so a fresh clone is unaffected. - (3) DISCHARGED 2026-08-04 — `scripts/cleanup_orphans.sh` shipped in-repo - and wired as a `$CLAUDE_PROJECT_DIR` SessionStart hook. Reaps PID-1-orphaned - `run_all.sh`/`run_gate.sh` trees, this repo's `cljw`, and `clojure.main -e` - oracle JVMs older than 30 min (TERM, so the gate tree clears its stamp - files); `--dry-run` lists without killing. Self-tested against a real - double-forked orphan: detected, killed, and a 2-second-old orphan spared - by the threshold. The two docs that cited the missing global hook also - CONTRADICTED each other about whether it reaped JVMs — both now describe - the shipped script. - (4) DISCHARGED 2026-08-04 — README no longer hard-codes a version (it - points at Releases), and every size claim across README / landscape / - RELEASE_METRICS / ROADMAP is gated by `size_claims` (ADR-0172), so it - cannot rot again silently. - (5) DISCHARGED 2026-08-04 — the `continue` skill's model-selection - section named specific releases on the strength of a retrieval - comparison between two of them; both had been superseded by the - settings.json pin, so it was steering subagents to models OLDER than - the one running the loop. Replaced with the durable rule (subagents - inherit the pinned parent) plus where a re-measured model finding - belongs (`.dev/lessons/`, dated). A model name in prose is checked by - nothing, which is why it rotted. - (6) DISCHARGED 2026-08-04 — README's banner now states the real posture - (stable, 1.x line, Issues and PRs open, small team so replies are - slow) and `.github/CONTRIBUTING.md` says explicitly which loop - conventions do NOT apply to an outside contributor. The mismatch it - described is gone. - (7) [onboarding] forced-Japanese chat FIXED 2026-07-29 — reduced to the single - `outputStyle` setting (the SessionStart language printf + CLAUDE.md - re-injection removed), with the settings.local.json override documented. - Residual CONTRIBUTING notes: the reference-clone SURVEY workflow needs the - gitignored `.claude/settings.local.json` additionalDirectories + personal - `~/Documents/OSS` clones (a fresh clone's Step-0 survey has nothing to - read); undocumented external tools `clj` (hardcoded /opt/homebrew/bin/clj) - + `md-table-align`. - (8) [resume degrade] handover "First task on resume" names gitignored - `private/notes/*.md` probe recipes a fresh clone lacks — promote the - load-bearing ones into a tracked ADR/debt body or mark local-only. - WORKS (fair credit — weigh before "fixing" too much): flake.nix + flake.lock - pin Zig 0.16.0; build.zig.zon fetches zwasm (tag v2.3.0, lazy, -Dwasm-gated) - + zlinter by URL+SHA (no sibling checkout); the main `... ReleaseSafe` build + - `test/run_all.sh` run from a bare clone (the `~/Documents/OSS` refs in e2e - steps are GUARDED and skip); GitHub Actions `scripts/ci_gate.sh` reproduces - the local gate on mac+linux with no personal deps; every settings.json hook - is `$CLAUDE_PROJECT_DIR`-relative; the ADR/debt/ROADMAP/CHANGELOG base ships - in the clone; `cw v0` via `git worktree add ../cw-v0 v0.5.0`. Companion = - zwasm D-526 (same audit, other repo). - category: "docs-onboarding" - barrier: "none (autonomous doc-hygiene drain, easiest-first: items 1-5 are pure staleness sweeps, item 6 a one-line banner, items 7-8 want a short CONTRIBUTING/onboarding note). Not urgent (maintenance mode, PRs paused) but cheap and removes reader-misleading rot. Do items 1-3 first — broken even for the maintainer." - last_reviewed: "2026-07-28" - id: "D-564" status: |- open (2026-07-17, ADR-0174 D6 residual). Thread interrupt family @@ -399,7 +461,7 @@ active: DRAIN 2 (2026-07-14, the handover-named residual): docs/architecture.md — the "Current state" tail was pre-v0.1.0-era (claimed "cut the v0.1.0 tag (not yet tagged)" while v1.2.0 had just shipped; "ARM64 JIT is the next - lever" superseded by the zwasm-side JIT ADR-0200; "12 optimizations + lever" superseded by zwasm ADR-0200 (the zwasm-side JIT); "12 optimizations O-016…O-027" vs the O-051 ledger; phase-plan framing vs the §9.0 gap-area model) — rewritten to current truth incl. the ADR-0170 nREPL surface + release/tap facts. docs/examples/wasm/README.md + add.clj — the @@ -765,7 +827,7 @@ active: last_reviewed: "2026-05-24" - id: "D-240" status: "investigated 2026-06-15 (Step 0); fix DIRECTION confirmed, deferred on an eval_budget interaction + LOW value (working fallback). Only blocks static-field DIFF cases — the analyze/CLI/e2e/corpus paths already cover them." - category: "tests" + category: "testing" barrier: |- **The dual-backend diff Fixture (`src/lang/diff_test.zig` / `evaluator.zig`) cannot exercise an analyze-time `Class/FIELD` static-field read** (e.g. @@ -1033,23 +1095,6 @@ active: read-string half is already closed (ADR-0122). Cross-ref SE-8, ADR-0122, core.zig eval surface. last_reviewed: "2026-08-04" - - id: "D-346" - status: "open — security (VM operand-stack on large literal). Evaluating a large collection LITERAL (e.g. `(count [1 1 … ×40000])`) overflows the VM operand stack → `internal_error: vm: operand stack overflow` (uncatchable, exit 70). NOT a crash/panic (the tokenizer overflow-panic was the separate, fixed finding) — a clean process-end. Lower severity: it is reached by EVALUATING a large literal (untrusted CODE), not by READING data (read-string builds the collection in the reader, not via operand pushes), and untrusted-code-eval is already a higher-trust scenario. Surfaced 2026-06-09 while testing the tokenizer fix." - reviewed_2026_08_04: |- - REPRODUCED, unchanged, still low. `(count [1 1 … x40000])` from a file exits - 70 with "vm: operand stack overflow" — a clean uncatchable process end, not - a crash or a memory-safety fault. Reachable only by EVALUATING untrusted - code (read-string builds the collection in the reader, not via operand - pushes), which is already the higher-trust scenario, so the severity - ordering behind the deferral holds. - category: "security" - barrier: |- - blocked-by: untrusted large-literal eval becoming a tracked threat. Options: - a growable/​checked operand stack with a catchable error (not internal_error), - or compiling a large vector/map literal to a bulk-build (not N operand pushes). - Low priority (eval-of-untrusted-code path; clean exit-70, not a crash). - Cross-ref the tokenizer long-input fix, vm.zig operand stack. - last_reviewed: "2026-08-04" - id: "D-384" status: |- open (2026-06-10) — the wasm cross-runtime comparison (`bench/wasm_bench.sh`, @@ -1116,13 +1161,13 @@ active: last_reviewed: "2026-07-02" - id: "D-437" status: "NARROWED (2026-06-20, B/C sweep — the headline repro is now FIXED). Re-probed vs clj: the original `(get {{(map inc [0 1 2]) 1} :v} {(list 1 2 3) 1})` and the map-nested / set-nested / deep-map-nested cases all MATCH clj now (corpus `lazy_nested_key.txt`, 6 golden pairs locked — the map-KEY path hashes its key at construction, realizing the inner lazy). REMAINING gap (4 repro classes, all the SAME root): a lazy nested inside a NON-realizing container that is itself a map key hashes the lazy by IDENTITY — `(get {[(map inc [0 1 2])] :v} [(list 1 2 3)])`, `(get {(list (map inc [0 1])) :v} (list (list 1 2)))`, `(get {{:k (map inc [0 1])} :v} {:k (list 1 2)})`, `(get {[[(map inc [0 1])]] :v} [[(list 1 2)]])` → all cljw nil / clj :v. (`#{lazy}` and a direct lazy key already work — sets + map-keys hash elements at construction.) Cause: `hashConsult` routes only `.lazy_seq/.range/.chunked/.list/.instance` top-level keys through the rt-aware `hashDispatch`; a `.vector`/`.array_map`-VALUE/nested container key falls to rt-free `valueHash`, whose element recursion can't realize a lazy (no rt). FINISHED FORM = the rt-free memoized realized-hash on the lazy_seq/range value (Alt-2) — NOT the alternative of routing collection keys through rt-aware recursion, which would tax the HOT vector-key path (the perf reason the design kept valueHash rt-free, shared with D-381's deferral)." - category: clj-parity + category: "clj-parity" barrier: "none structural, but the finished-form fix is the SHARED memoized-hash-on-heap-value mechanism that D-381 (deftype hasheq cache) also needs — a `realized_hash: ?u32` slot on the lazy_seq/range struct, filled on first realize-and-hash, GC-rooted, read rt-free by valueHash. Best done as ONE dedicated unit with D-381 under the perf campaign (9.2.S / D-180) so the cache's fill-point + GC + perf are measured together, not bolted on. The 6 now-passing nested cases are corpus-locked against regression; the 4 remaining are documented repros. Rare consumer (a vector-wrapping-a-lazy used as a map key)." quality_floor: 'correctness: nested seq/lazy map-KEY content hash + rt-free memoized seq-key hash (ADR-0139 Alt-2 finished form)' last_reviewed: "2026-06-14" - id: "D-439" - status: 'open (2026-06-14, found probing the BigDecimal interop surface during the clj-parity floor drain). The BigDecimal instance-method surface is PARTIAL: setScale (2+3 arg) + scale/signum/unscaledValue/precision are LANDED (e2e phase14_bigdecimal_setscale, 28 cases); PARTIAL 2026-06-15 — negate\/abs\/toBigInteger\/stripTrailingZeros LANDED (e2e phase14_bigdecimal_setscale 35); STILL unimplemented: movePointLeft/movePointRight (scale shift), stripTrailingZeros (trailing-zero strip + scale reduce), negate/abs/plus, toBigInteger/toBigIntegerExact (truncate to int), add/subtract/multiply/divide instance forms. Each returns a new BigDecimal needing Managed arithmetic (negate/clone/mul-10^n) — fiddly but bounded. NICHE: clojure.core ops (- abs + * / and bigdec contagion) cover most arithmetic; the .method interop forms are reached mainly by Java-ported lib code. Take up when a real consumer hits one, or as a big-bang BigDecimal-interop-completion unit (clj_diff_sweep Discipline 2). NARROWED 2026-06-20 (the big-bang completion unit ran): the COMMON instance arithmetic + point-shift LANDED — add/subtract/multiply (delegate to the shared allocAdd/allocSub/allocMul the +/-/* operators use), divide (allocDiv; throws non_terminating_decimal/divide_by_zero like clj (/ aM bM)), movePointLeft/movePointRight (new big_decimal.allocMovePoint: unscaled unchanged + scale shift by ±n, a negative result-scale normalized by *10^|scale|). All clj-verified incl. movePointRight scale<0 (12.34 ·10^3 -> 12340); e2e phase14_bigdecimal_setscale now 43. The BigDecimal interop surface is now SUBSTANTIALLY COMPLETE (setScale/scale/signum/unscaledValue/precision/negate/abs/toBigInteger/stripTrailingZeros + add/subtract/multiply/divide/movePointLeft/movePointRight). REMAINING (niche, take-up-on-consumer; RE-NARROWED 2026-06-25 debt audit — pow/compareTo/min/max/remainder/round now LAND too, verified directly: 8M/-1/1M/2M/1M/3.14M): only sqrt/scaleByPowerOfTen/ulp/divideAndRemainder still unimplemented. DRAIN 2026-06-25: scaleByPowerOfTen/ulp/divideAndRemainder LANDED — scaleByPowerOfTen (pure scale shift, negative scale KEPT so it prints scientific, big_decimal.allocScaleByPowerOfTen), ulp (unscaled 1 at this scale, allocUlp), divideAndRemainder (2-elem [divideToIntegralValue, remainder] array, built under enterFabrication so the first result survives the alloc of the second; divide-by-zero raises). All clj-verified (123M/0.123M/1.2E+4M, 0.01M/1M/0.1M, [3M 2M]/[4M 0.7M]); e2e phase14_bigdecimal_setscale bd_sbpt*/bd_ulp*/bd_divrem* (8 cases, 105 total). ONLY sqrt(MathContext) REMAINS — the hardest piece (correctly-rounded arbitrary-precision sqrt + JDK preferred-output-scale reproduction; the native Managed.sqrt integer floor-sqrt is the base, exposed as BigInteger .sqrt). Deferred-by-tractability to a dedicated numeric cycle (easiest-first ordering), NOT abandoned.' - category: clj-parity + status: 'open (2026-06-14, found probing the BigDecimal interop surface during the clj-parity floor drain). The BigDecimal instance-method surface is PARTIAL: setScale (2+3 arg) + scale/signum/unscaledValue/precision are LANDED (suite test/clj/suites/bigdecimal_test.clj, 28 cases); PARTIAL 2026-06-15 — negate\/abs\/toBigInteger\/stripTrailingZeros LANDED (suite test/clj/suites/bigdecimal_test.clj 35); STILL unimplemented: movePointLeft/movePointRight (scale shift), stripTrailingZeros (trailing-zero strip + scale reduce), negate/abs/plus, toBigInteger/toBigIntegerExact (truncate to int), add/subtract/multiply/divide instance forms. Each returns a new BigDecimal needing Managed arithmetic (negate/clone/mul-10^n) — fiddly but bounded. NICHE: clojure.core ops (- abs + * / and bigdec contagion) cover most arithmetic; the .method interop forms are reached mainly by Java-ported lib code. Take up when a real consumer hits one, or as a big-bang BigDecimal-interop-completion unit (clj_diff_sweep Discipline 2). NARROWED 2026-06-20 (the big-bang completion unit ran): the COMMON instance arithmetic + point-shift LANDED — add/subtract/multiply (delegate to the shared allocAdd/allocSub/allocMul the +/-/* operators use), divide (allocDiv; throws non_terminating_decimal/divide_by_zero like clj (/ aM bM)), movePointLeft/movePointRight (new big_decimal.allocMovePoint: unscaled unchanged + scale shift by ±n, a negative result-scale normalized by *10^|scale|). All clj-verified incl. movePointRight scale<0 (12.34 ·10^3 -> 12340); suite test/clj/suites/bigdecimal_test.clj now 43. The BigDecimal interop surface is now SUBSTANTIALLY COMPLETE (setScale/scale/signum/unscaledValue/precision/negate/abs/toBigInteger/stripTrailingZeros + add/subtract/multiply/divide/movePointLeft/movePointRight). REMAINING (niche, take-up-on-consumer; RE-NARROWED 2026-06-25 debt audit — pow/compareTo/min/max/remainder/round now LAND too, verified directly: 8M/-1/1M/2M/1M/3.14M): only sqrt/scaleByPowerOfTen/ulp/divideAndRemainder still unimplemented. DRAIN 2026-06-25: scaleByPowerOfTen/ulp/divideAndRemainder LANDED — scaleByPowerOfTen (pure scale shift, negative scale KEPT so it prints scientific, big_decimal.allocScaleByPowerOfTen), ulp (unscaled 1 at this scale, allocUlp), divideAndRemainder (2-elem [divideToIntegralValue, remainder] array, built under enterFabrication so the first result survives the alloc of the second; divide-by-zero raises). All clj-verified (123M/0.123M/1.2E+4M, 0.01M/1M/0.1M, [3M 2M]/[4M 0.7M]); suite test/clj/suites/bigdecimal_test.clj bd_sbpt*/bd_ulp*/bd_divrem* (8 cases, 105 total). ONLY sqrt(MathContext) REMAINS — the hardest piece (correctly-rounded arbitrary-precision sqrt + JDK preferred-output-scale reproduction; the native Managed.sqrt integer floor-sqrt is the base, exposed as BigInteger .sqrt). Deferred-by-tractability to a dedicated numeric cycle (easiest-first ordering), NOT abandoned.' + category: "clj-parity" barrier: none — bounded numeric work (Managed negate/clone/mul-by-10^n + scale arithmetic), each clj-oracle-verifiable. Deferred as NICHE (clojure.core arithmetic covers the common path; .method interop is Java-port-only). Do as one big-bang completion unit or when a lib pulls a specific method. last_reviewed: "2026-06-14" - id: "D-442" @@ -1130,12 +1175,6 @@ active: category: "clj-parity" barrier: "Cancellation arc DONE (state machine + precise CancellationException + Thread/sleep cooperative abort). The legacy/executor surface (ii) is now DONE (ADR-0155 parts 1+2, 8/8; AD-045 + AD-046). ONE residual remains: (i) deref-WAIT abort — a worker blocked on a NESTED (deref future/promise) is not aborted promptly; land when a real workload needs a deref-deadlocked future cancelled promptly (then ripple future/promise.deref to anyerror!?Value + poll the cancel flag in the wait loop, instead of pure condWait, only on a cancellable worker). LOW-value (resource-release only, un-observable via state, inert with auto-collect OFF); re-evaluate when a real workload or auto-collect-on makes it observable." last_reviewed: "2026-06-20" - - id: "D-044" - status: "Phase 7+ target" - category: "perf" - barrier: |- - Lazy SubVector wrapper for `subvec` per clojure JVM's structural-sharing pattern. Current 5.4.d implementation is eager copy (O(n)); a lazy wrapper holding `parent: *Vector + start: u32 + end: u32` would be O(1) view + share root + tail with the parent. Requires a new SubVector Tag slot (D13 / D14 / D15 from D-043 reserves) + polymorphic op dispatch (`.vector` vs `.subvec` arms in nth / count / conj / pop / assoc) + trace fn that walks the parent pointer. Land when the structural-sharing benefit is measurable (perf campaign §9.2.S; Phase 7 passed — phase-queue retired, ADR-0142 — and the benefit is still un-measured). - last_reviewed: "2026-05-24" - id: "D-266" status: "RE-MEASURED 2026-07-02 (audit): the motivating measurement is ~1000x stale (50k claimed ~3s; measured 5k=10ms). ALSO: the (count (repeat 1000000 1)) segfault found here was FIXED via ADR-0028 amendment 3 (gray-worklist mark). Native Repeat remains a value-judgement, not a hot gap. ORIGINAL: open (found 2026-06-06; PREMISE CORRECTED 2026-06-16 during the sweep). Per-element cost of realizing `(repeat …)` is high (~0.4ms/elem) — the lazy-seq thunk+cons+closure overhead per `rest`. PREMISE FIX: 'chunk repeat like range' is the WRONG fix — clj's `repeat` is NOT a chunked-seq (`(chunked-seq? (seq (repeat 3 :x)))` → FALSE in clj, FALSE in cljw today); clj uses a lightweight native `Repeat` type optimized via `IReduce`/`internal-reduce`, keeping chunked-seq? false. Chunking cljw's repeat would FLIP chunked-seq? to true = a NEW divergence (verified vs clj 2026-06-16). The clj-faithful fix is a native lightweight `Repeat` seq type in Zig (O(1) `next()` like `LongRange`, + an internal-reduce fast path), NOT a `.clj` chunk-buffer producer. Substantial (a new native seq type w/ GC + seq-protocol + reduce integration) — fresh-context unit." category: "perf" @@ -1188,7 +1227,7 @@ active: last_reviewed: "2026-06-07" - id: "D-381" status: "open (2026-06-10, ADR-0129 deferred perf — NOT a correctness gap). A deftype/reify with a custom Object/hasheq used as a map key / set element currently re-dispatches its hasheq on every HAMT bucket op (assoc/get/contains/ dissoc + each node-split rehash of an existing key), and a COLLECTION deftype (flatland.ordered) whose hasheq is `(hash-unordered-coll this)` recomputes its O(size) content hash each time → O(N²) to build an N-entry such map. clj caches hasheq lazily per instance (idempotent IHashEq). The DA fork (ADR-0129 Alternatives) recommended a per-instance hasheq cache as the hash-side representation; it was deferred because correctness comes from arming coverage, not caching, and eager-at-construction caching makes ordered-map construction O(N²). Land a LAZY cache when the perf window opens (F-010 / D-180 optimization campaign): a `hasheq_cache` slot on TypedInstance/ReifiedInstance populated on first armed hashDispatch, read rt-free by valueHash thereafter (also closes the theoretical insert-armed/lookup-unarmed consistency edge). Lazy not eager (eager reintroduces the O(N²) + a top-level-literal unarmed-construction gap)." - category: "performance" + category: "perf" barrier: |- Blocked-by: deferred to the F-010 perf window (D-180 optimization campaign), NOT a correctness blocker — facet 2 is correct without it (D-377 DISCHARGED). @@ -1767,17 +1806,11 @@ standing: `vm.installVTable` currently reuses `tree_walk.treeWalkCall` + `valueTypeKey` + `callFunction` (4.8 design — single seam is the `evalChunk` vtable hook). Phase 17 adds a 3rd backend (JIT per ADR-0005); 3-backend reuse of TreeWalk's `callFunction` skews the dependency graph. Extract a backend-shared "callable dispatch" layer (likely `src/runtime/dispatch/callable.zig`) at Phase 17 entry before adding the JIT vtable. Owner picks exact extraction shape **within F-NNN envelope** (= F-004 type set + F-005 numeric tower dispatch + F-006 GC root through callable dispatch must work unchanged through the new layer). | gap area III — before the JIT vtable lands (was 'Phase 17 entry'; §9.0/ADR-0142) last_reviewed: "2026-05-23" - id: "D-036" - status: "gap area II — Wasm/edge-native (§9.18): zwasm v2 integration master row. Direction confirmed (F-001/004/006/008); zwasm is SHA-pinned + interp-embedded (done). Remaining: the finished-form embedding shape (D-350) + JIT-coordination (D-035). (Was 'Phase 16 entry'; ADR-0142.)" + status: "gap area II — Wasm/edge-native (§9.18): zwasm v2 integration master row. Direction confirmed (F-001/004/006/008); zwasm v2.5.0 is pinned + embedded and the finished-form embedding shape is settled (D-350 discharged). Remaining: JIT coordination (D-035). (Was 'Phase 16 entry'; ADR-0142.)" category: "scope" barrier: |- - wasm FFI confirmed **unavoidable**. **Direction confirmed (F-001 + F-004 + F-006 + F-008)**: F-004 day-1 reserves NaN-box `funcref` / `externref` inline slots in Group D for zwasm-v2 reference types. F-006 keeps cw heap and Wasm linear memory in **separate spaces**; zwasm internal bookkeeping accepts cw GC allocator via `Engine.init(allocator)` (cw v0 D110 dual-GC leak does not recur). F-008 pins zwasm v2 spec (ADR-0109) review + cw v1 stances. JIT-coordination with the gap-area-III JIT (D-035) lands with the gap-area-II embedding unit. | gap area II unit start + zwasm v2 rewrite (ADR-0109) completion — the latter is DONE (v2.1.0 pinned) - last_reviewed: "2026-05-24" - - id: "D-039" - status: "gap area II — Wasm/edge-native (§9.18): cw io_interface (Tier 1) vs zwasm WASI responsibility split. Future — decided when the unified wasm embedding (D-350) lands. (Was 'Phase 16 entry'; ADR-0142.)" - category: "structure" - barrier: |- - cw v1 `io_interface.zig` (Tier 1) と zwasm v2 WASI (`linker.defineWasi(cfg)`) の責務分離。 (a) 別 layer に分離 default (cw io_interface は cw v1 内部 I/O 抽象、 WASI は wasm 統合用に独立)、 vs (b) cw `io_default.zig` (Tier 2) の実装を WASI 経由にする option。 The gap-area-II unit owner decides against `structure_plan.md`'s `runtime/wasm/` + `runtime/io_default.zig` placement. | gap area II unit start (was 'Phase 16 entry'; §9.0/ADR-0142) - last_reviewed: "2026-05-24" + wasm FFI confirmed **unavoidable**. **Direction confirmed (F-001 + F-004 + F-006 + F-008)**: F-004 day-1 reserves NaN-box `funcref` / `externref` inline slots in Group D for zwasm-v2 reference types. F-006 keeps cw heap and Wasm linear memory in **separate spaces**; zwasm internal bookkeeping accepts cw GC allocator via `Engine.init(allocator)` (cw v0 D110 dual-GC leak does not recur). F-008 pins zwasm v2 spec (ADR-0109) review + cw v1 stances. D-350 settled the embedding lifecycle and D-039 settled the I/O responsibility split in ADR-0124. JIT coordination with the gap-area-III JIT (D-035) is the remaining trigger. | when the narrow cljw JIT begins; zwasm v2.5.0 integration itself is DONE + last_reviewed: "2026-09-03" - id: "D-210" status: "STANDING FLOOR (quality-loop floor: clj-parity) — the clj-parity root-cause campaign C1..C7 is **COMPLETE 2026-06-03** (ROADMAP §9.2.P, ADR-0076, user-directed 2026-06-02): C1 (D-164) + C2 (D-205) + C3 (D-207) + C4 (D-209) + C6 (D-200) + C5 (D-198) + **C7 (D-165, ADR-0080) DISCHARGED**. The anchor now persists ONLY as the standing `quality-loop floor: clj-parity` that drains any NEW cljw↔clj DIFF a future sweep surfaces (drain highest-value-first, per CLAUDE.md Step 0.5 floor-drain). No remaining campaign units. C7 spun off **D-214** (bit-ops on heap-Long, pre-existing i48-only gap)." category: "runtime" @@ -2647,7 +2680,7 @@ standing: private/notes/phaseC-d251-root-cause.md. last_reviewed: "2026-06-05" - id: "D-252" - status: "PARTIAL (2026-06-05) — C9 (CLI-print pin), C2/C3 (every?/some? EvalFrame), C8 (defprotocol, via Alt D) FIXED + locked in phase16_gc_torture.sh. C6 (clojure.walk rebuild fns: list/vector/set/array_map) FIXED — each roots the in-progress accumulator + source across the inner fn's reentrant eval (EvalFrame); was @memcpy panic / #{5 nil} corruption. C1 (applyFn), C4/C5 (sorted-map-by) NOT-REPRODUCING in probes (latent, kept tracked). REMAINING: latent C1 (applyFn), C4/C5 (sorted-map-by), C7 (multimethod) — not reproduced in probes; kept tracked, re-sweep at the next Phase boundary. Was: OPEN (from the GC-rooting SSOT sweep) — 9 UNROOTED-CANDIDATE reentrant-primitive sites that hold a GC Value in a Zig local across a call back into eval WITHOUT publishing an EvalFrame (the ADR-0094 class). Torture can hit any of them. C8 (defprotocol/defrecord) + C9 (filter/keep/remove chunk-builder) are KNOWN-OPEN today; C1 (applyFn gpa ArrayList, apply-over-lazy) + C4/C5 (sorted insert/deleteNode on the custom sorted-map-by comparator path) are HIGH-severity latent; C2/C3 (everyQ/someFn cursor), C6 (walk rebuild), C7 (multimethod dispatch_val) are med/low latent. Fix shape = the reduceFn EvalFrame exemplar (publish acc/cursor as collector-addressable slots), which also doubles as moving-GC prep (.dev/gc_rooting.md §C). C9 is suspected to be a membrane/lazy-realisation interaction, not a missing Zig frame — investigate separately. Full file:line inventory: .dev/gc_rooting.md §C + private/notes/gc-rooting-ssot-sweep.md." + status: "UPDATE 2026-09-06: C1 applyFn and applyVariadic now publish EvalFrame roots; native GC-in-lazy-callback regressions added (ADR-0042 amendment 2). Other candidates retain their recorded status. PARTIAL (2026-06-05) — C9 (CLI-print pin), C2/C3 (every?/some? EvalFrame), C8 (defprotocol, via Alt D) FIXED + locked in phase16_gc_torture.sh. C6 (clojure.walk rebuild fns: list/vector/set/array_map) FIXED — each roots the in-progress accumulator + source across the inner fn's reentrant eval (EvalFrame); was @memcpy panic / #{5 nil} corruption. C1 (applyFn), C4/C5 (sorted-map-by) NOT-REPRODUCING in probes (latent, kept tracked). REMAINING: latent C1 (applyFn), C4/C5 (sorted-map-by), C7 (multimethod) — not reproduced in probes; kept tracked, re-sweep at the next Phase boundary. Was: OPEN (from the GC-rooting SSOT sweep) — 9 UNROOTED-CANDIDATE reentrant-primitive sites that hold a GC Value in a Zig local across a call back into eval WITHOUT publishing an EvalFrame (the ADR-0094 class). Torture can hit any of them. C8 (defprotocol/defrecord) + C9 (filter/keep/remove chunk-builder) are KNOWN-OPEN today; C1 (applyFn gpa ArrayList, apply-over-lazy) + C4/C5 (sorted insert/deleteNode on the custom sorted-map-by comparator path) are HIGH-severity latent; C2/C3 (everyQ/someFn cursor), C6 (walk rebuild), C7 (multimethod dispatch_val) are med/low latent. Fix shape = the reduceFn EvalFrame exemplar (publish acc/cursor as collector-addressable slots), which also doubles as moving-GC prep (.dev/gc_rooting.md §C). C9 is suspected to be a membrane/lazy-realisation interaction, not a missing Zig frame — investigate separately. Full file:line inventory: .dev/gc_rooting.md §C + private/notes/gc-rooting-ssot-sweep.md." category: "gc" barrier: |- Drive to torture-green: each Ci opens an EvalFrame (or sets gc_self_guard) @@ -2714,11 +2747,13 @@ standing: v1 has 12 present + 3 partial. The 24 MISSING are tracked here, detail in `.dev/v0_v1_feature_parity.md`.** MISSING bundled namespaces (campaign Stage 1.5 backfill, pure-Clojure-degree order): clojure.repl.deps - (`add-lib!`/`add-libs`/`sync-deps`) · clojure.java.shell · clojure.java.process + (`add-lib!`/`add-libs`/`sync-deps`) · clojure.java.process · clojure.java.browse · clojure.core.server · clojure.core.specs.alpha · clojure.spec.alpha · clojure.spec.gen.alpha · clojure.xml · cljw.wasm - (+ `cljw_wasm_builtins`). LANDED since this list was cut: clojure.stacktrace + (+ `cljw_wasm_builtins`). LANDED since this list was cut: clojure.java.shell + (2026-09-25, ADR-0199: `sh` / `with-sh-dir` / `with-sh-env` over the native + `cljw.process/run`; suite test/clj/suites/java_shell_test.clj) + clojure.stacktrace (2026-06-10, ex-info cause-chain reimpl; per-frame degrades = AD-029; e2e phase15_clojure_stacktrace) + clojure.uuid (2026-06-10, require-compat shim; the `#uuid` reader + UUID print are cljw built-ins; e2e phase15_clojure_uuid) @@ -2797,15 +2832,6 @@ standing: the host_instance state[] GC-ROOT comment). Cross-ref ADR-0106, ADR-0114 Dec B, gc_rooting.md §H. last_reviewed: "2026-06-07" - - id: "D-350" - status: "open — structural (Phase 16 / D-036 territory). The finished-form-clean wasm embedding (DA alternative-b, F-002) is a UNIFIED `(wasm/load path {:wasi {…}})` returning a handle, with `wasm/run`/`wasm/call` as operations on it — one 'embed a wasm module' concept, WASI config as the definition-derived closed set, reusable persistent WASI instances. The current one-shot `wasm/run` (compile+instantiate+run+teardown per call) is the smallest honest shape that captures output today. The unified form is gated on a zwasm API addition (instantiate-with-WASI separated from run-entry on the public Engine/Instance surface). Recorded + deferred to the Phase-16 owner per F-003. Surfaced 2026-06-09 (ADR-0124)." - category: "structural" - barrier: |- - blocked-by: Phase 16 wasm-component/instance-lifetime design (D-036) + - zwasm exposing persistent instantiate-with-WASI vs run-entry separately. - Imagined-and-deferred per F-003; do not seize in a demo-hardening cycle. - Cross-ref ADR-0124 Alternatives considered (b). - last_reviewed: "2026-06-09" - id: "D-354" status: "open — crash isolation is IMPOSSIBLE in-process (isolation dim (d)). A Zig panic / UAF in cljw takes the whole process down, so a single-binary playground evaluating untrusted code cannot contain a runtime crash. Only a PROCESS boundary (subprocess) or a WASM boundary (cljw-as-wasm-guest in zwasm) gives true crash isolation. This is the structural argument for keeping a subprocess shell (or the eventual cljw-as-wasm endgame) even after D-351/D-352/D-353 land. cljw also has NO process-spawn API, so 'make the supervisor a cljw program' is itself blocked. Surfaced 2026-06-09 (isolation scope study)." category: "isolation" @@ -2830,7 +2856,7 @@ standing: last_reviewed: "2026-06-10" - id: "D-436" status: "open (2026-06-14, USER-FLAGGED big-cleanup epic). The user observed that the loop accumulates finished-form DEVIATIONS papered over by workarounds, and directed: 'いつか、こういったものも大整理対象にしてほしい' — flag them + accumulate for a future structural 大整理 (big reorganization), don't silently absorb them. This row is the standing aggregator; add a candidate the moment a workaround-instead-of-finished-form is taken. CANDIDATES so far: (a) D-435 — the diff oracle (Layer-3 dual-backend parity, ADR-0005) cannot reach full-runtime behavior (the diff Fixture skips the core.clj bootstrap), so with-out-str/*out*/macro/clojure.core-surface forms are NOT dual-backend-verified — the trigger the user caught; (b) RESOLVED 2026-06-14 by ADR-0138 (Track C): `*out*`/`*err*` were a keyword SENTINEL forcing every Writer-interop method to be special-cased (D-434) — now first-class text_io writer VALUES; sentinel + out_capture threadlocal + out_writer_method.zig all DELETED; with-out-str + nREPL capture are plain `(binding [*out* string-writer] …)`. (*in*/reader value = Track C step 3.) (c) compat_tiers `methods:` is a hand-list that both UNDER-claims (Math/Long missing-but-wired) and OVER-claims (java.time/BigDecimal/Arrays listed-but-unbuilt) — ADR-0137's generated-from-corpus index is the finished form; (d) verified_projects for BUNDLED libs (data.json/csv/tools.cli) carry a deps.edn git coord that cljw shadows with its in-source copy — the coord is moot (documents upstream only). Mechanism: this is NOT a do-now row; it is the durable list so a future 大整理 cycle (or the user) picks them up. Each entry names its own finished form." - category: "structural" + category: "structure" barrier: "none — this is an aggregator, not a single fix. Drained by a dedicated 大整理 cycle (user-initiated or when one candidate blocks real work). Keep ADDING candidates as workarounds are taken (the discipline the user asked for); do not let it rot (review at each Phase boundary audit)." last_reviewed: "2026-06-14" - id: "D-440" @@ -2844,7 +2870,130 @@ standing: category: "meta-reorg" barrier: "open after the D-440 arc completes (R4 + R5 must first rewrite the bulk of 'Phase N' citations to gap-area terms). Premature adoption = ~70 dangling citations with no 1:1 redirect. Trigger: D-440 R4/R5 done + a user nod to re-architect the planning axis (it is depth-4, the project's organizing metaphor)." last_reviewed: "2026-06-15" + - id: "D-578" + status: "open — NARROWED 2026-08-20 to a performance promise, no longer a correctness gap. Oracle measured, cljw now matches clj on every chunked-producing form: (seq (range 40)) true/40 both; (seq (vec (range 40))) true/40 both; (rest (seq (vec (range 40)))) true/39 both; (map inc (range 40)) false/40 both. So `counted?` answers correctly. What remains: for `.chunked_cons` cljw's `count` WALKS (O(chunks)) where clj's equivalent is O(1), because clj reaches these shapes as LongRange or as a vector-backed ChunkedSeq that knows its remaining total, while cljw's chunked_cons is a chunk plus a possibly-lazy tail. `counted?` therefore implies O(1) more weakly here than in clj." + category: "perf" + barrier: |- + Not urgent and NOT a correctness fix — every observable answer already + matches clj (measured above), so do not "fix" this by flipping + `counted?` to false: that would introduce a divergence where none exists + now, and `counted?` is what callers use to pick an algorithm. + The real optimisation, when a consumer justifies it: when a chunked_cons' + tail is itself a `.range` (the common `(seq (range n))` shape), the total + is chunk-remaining + range count with no walk. The general lazy-tail case + cannot be O(1) and should stay a walk. + Note `.array_seq` is already O(1) via `array_seq.countOf` — the vector-seq + path needs nothing. + last_reviewed: "2026-08-20" + - id: "D-582" + status: "open — NARROWED 2026-08-20, residual (2) now MEASURED and classified. (3) DISCHARGED: ADR-0004's enum reservation has an enforcing artifact (data/special_forms.txt + check_special_form_enum.sh, gated) and the ADR withdraws the reservation. (1) DISCHARGED: the Affected-files mandate is enforced prospectively (data/adr_affected_files_grandfathered.txt + check_adr_affected_files.sh, mark 0191, 51 grandfathered; ADR-0020 amended). (2) MEASURED: 71 genuinely-dead paths across 39 ADRs — NOT the 36 the first audit reported, and not the 147 a naive extraction reports (stripping a leading `.` turns every `.dev/...` citation into a false positive; verify the extractor before trusting a count)." + category: "governance" + barrier: |- + The 71 are FOUR classes and only one is safe to sweep. Decide per path. + A. Bare filename, file EXISTS elsewhere — `heap_tag.zig` (4x, really + src/runtime/value/heap_tag.zig), `vm.zig` (4x, src/eval/backend/), + `agent.zig`, `mark_sweep.zig`. The ADR cites a basename without its + directory. Mechanically expandable and SAFE — the only sweepable + class. Confirm the file is the same one the ADR meant, not a + same-named newcomer. + B. TEMPLATE PLACEHOLDERS — `src/path/to/file.zig`, `scripts/bar.sh`, + `.claude/rules/foo.md`, from `.dev/decisions/0000_template.md` and + ADR-0020's own example block. These are illustrations, not + citations. Never "fix" them, and EXCLUDE 0000_template.md from any + future path check or it will fail forever. + C. Genuinely REMOVED files — `src/runtime/{gc,lock,stm}/stub.zig` + (ADR-0023's comptime-stub pattern). The entry is a true historical + record of a file that existed when the decision was made. Rewriting + it falsifies the ADR. Leave. + D. Gitignored scratch — `private/notes/...`. Correctly cited at the + time; never in the tree. Leave. + Do NOT extend check_adr_affected_files.sh to path existence: it would + fail on B, C and D, which are all correct as written. + last_reviewed: "2026-08-20" discharged: + - id: "D-350" + discharged_at: "ADR-0124 amendment (2026-09-03)" + resolution: |- + The explicit lifecycle split is the finished form. `wasm/load` + + `wasm/call` owns persistent, re-invokable core-library instances; + `wasm/run` owns single-shot WASI commands with captured stdout, stderr, + and exit; the component path owns persistent reusable WASI state. + Measured against the final zwasm v2.5.0 facade: public `Linker.defineWasi` + + `instantiate` can build a persistent WASI core instance, but public + `WasiConfig` exposes only args/env/preopens. Captured stdio and exit are + explicitly facade-unwired. A unified `wasm/load {:wasi ...}` would lose + the command result contract or reach through zwasm's private WASI Host, + violating F-009. `wasm/load` now rejects any present `:wasi` option so the + split cannot be mistaken for silently incomplete support; the phase16 + Wasm FFI catchability fixture pins the rejection and exception class. + Amended 2026-10-07: the command path no longer recompiles per call. zwasm fork branch feat/captured-wasi-run adds the PUBLIC `cli.run_prepared.PreparedWasi` (validate + JIT-compile once; each run builds a fresh store, WASI host and instance with captured stdio, stdin bytes, argv, env, preopens and every budget axis). `wasm/run` keeps prepared modules in `runtime/cljw/wasm/run_cache.zig` (LRU of 8, keyed by jail-resolved path + inode + size + mtime, leases keep an evicted module alive until its run ends); `:cache false` compiles for one call and `wasm/clear-cache!` drops them. The split above stands: `wasm/run` stays a one-shot command surface, only the compile is shared, and no private zwasm WASI Host is reached (F-009). + - id: "D-039" + discharged_at: "ADR-0124 amendment / D-350 (2026-09-03)" + resolution: |- + Keep the responsibilities separate. cw Tier-1 `io_interface` remains the + runtime's internal I/O abstraction; zwasm's WASI host remains isolated in + `runtime/cljw/wasm`. Neither becomes the implementation of the other. + This follows the accepted command/handle lifecycle split and avoids + coupling general cw I/O to an opt-in Wasm dependency (F-001/F-009). + - id: "D-583" + status: "DISCHARGED 2026-08-21 (O-059) — `.sub_vector` (slot 59, F-004 amend) SubVector VIEW landed: `runtime/collection/sub_vector.zig` {start,end,parent,meta} with JVM APersistentVector$SubVector semantics (flatten-on-nest, never-empty→EMPTY, write ops via parent.assocN stay SubVectors). `clojure.core/subvec` fast-paths vectors through `cljw.internal/__subvec`; `vector.subvec` renamed `copyRange` (LBQ compaction still needs a materializing copy). ~30 consumer sites carry `.sub_vector` arms (interface_membership 16 arrays skip EDITABLE/IREDUCEINIT per JVM ground truth; count/nth/conj/assoc/pop/seq/rseq/first/rest/next/empty/print/equal+hash/compare/meta/with-meta/lookup-IFn/reduce/walk/json/serialize/lazy-realize/class-name=SubVector). MEASURED ReleaseSafe: 2000 subvec ops @ n=16000 **9215 → 0.77 ms (~11967x)**, now at clj parity (1.06 ms). 52-assertion e2e `subvec_view.sh` + corpus `subvec.txt` (31 exprs) green vs clj; F-012 diff oracle green. Was: open USER-GATED, design at private/notes/subvec-seqview-design.md." + category: "perf" + barrier: |- + DISCHARGED. Was USER-GATED on the slot pick (ADR-0179): the user selected + `box`→`sub_vector` (F-004 amendment 2026-08-20), lifting the gate. Landed + 2026-08-21 as O-059. + last_reviewed: "2026-08-21" + - id: "D-044" + status: "DISCHARGED 2026-08-21 (O-059, with D-583) — the lazy SubVector wrapper this row asked for. `subvec` was an eager O(n) copy; now the O(1) `.sub_vector` VIEW sharing the parent's structure. Superseded the reasoning that a new tag needed a D13/D14/D15 reserve — the slot came from `box`→`sub_vector` (F-004 amend). See D-583 for the full landing + measurement (~11967x)." + category: "perf" + barrier: |- + DISCHARGED — the structural-sharing benefit this row gated on is now + measured (9215 → 0.77 ms, clj parity). Landed with D-583 as O-059. + last_reviewed: "2026-08-21" + - id: "D-581" + status: "DISCHARGED 2026-08-20 — split into two entry points (readStringFn raises, readStringEdnFn returns nil); verified empty/whitespace/comment-only => nil for edn and RuntimeException for core, with :eof still winning for both. Was: measured — the kanban row stated this BACKWARDS. `(clojure.edn/read-string \"\")` THROWS `Unexpected EOF while reading form` in cljw; clj returns nil. Same for whitespace-only input. The kanban row [CLJW-EDN-EOF] recorded it as cljw returning nil, which is what clj does. Direction confirmed by running both." + category: "clj-parity" + barrier: |- + NOT a one-line flip — the two readers are ONE function in cljw. + `src/lang/primitive.zig:86` interns `edn_prim.readStringFn` directly into + `clojure.core` as `read-string`, so `clojure.core/read-string` and + `clojure.edn/read-string` share `src/lang/primitive/edn.zig:121-122`. + clj wants them to DIFFER on exactly this input: measured, core throws + `RuntimeException: EOF while reading` and edn returns nil. Returning nil + at the shared raise site would fix edn and break core in the same edit. + So the fix is to split the entry points (or thread a caller flag) — decide + which, then change the docstring at edn.zig:38-41, which currently ASSERTS + the wrong thing ("empty / whitespace-only / comment-only input THROWS EOF + (clj parity, D-269)"). That sentence is how D-269 came to be discharged + wrong: the claim was written down and never run against clj. + The 2-arity `(read-string {:eof :done} "")` => :done already works on both + and must keep working. + last_reviewed: "2026-08-20" + - id: "D-579" + status: "DISCHARGED 2026-08-20 — all 8 coercing fns done. Was PARTIAL: `upper-case`, `lower-case`, `capitalize` now coerce via the new `toStringArg` helper (print.writeStrValue) and match clj exactly, verified: `(upper-case :foo)`=>\":FOO\", `(upper-case 42)`=>\"42\", `(lower-case 'SymBol)`=>\"symbol\", `(capitalize 42)`=>\"42\"; nil still raises; trim/reverse/blank?/split still throw as clj does. Then completed: `starts-with?`, `ends-with?`, `includes?`, `index-of`, `last-index-of` — `rt`/`env` threaded through `prefixImpl` and both index fns; first arg coerces, needle stays String-typed as in clj. Original finding: clojure.string's `.toString`-coercing fns raise a type error in cljw where clj accepts any non-nil value. MEASURED against clj: `upper-case`, `lower-case`, `capitalize`, `starts-with?`, `ends-with?`, `includes?`, `index-of`, `last-index-of` all coerce via `.toString` — `(upper-case :foo)` => \":FOO\", `(upper-case 42)` => \"42\", `(starts-with? 421 \"4\")` => true. cljw raises `upper-case: expected string, got keyword`." + category: "clj-parity" + barrier: |- + Ready — no blocker, just care about the boundary. The set is NOT all of + clojure.string: measured on clj, `trim`, `triml`, `trimr`, `trim-newline`, + `reverse`, `blank?`, `split`, `split-lines` DO throw ClassCastException on a + non-CharSequence, because they call CharSequence methods directly rather + than `.toString`. Coercing those too would be a divergence in the other + direction. `nil` must still raise (clj throws NPE), so `(str s)` is the + wrong coercion — it turns nil into "". + Mechanism is settled: render through `print.writeStrValue`, which + `core.zig:900`'s `str` already uses and whose own comment names it as + "the single source shared with the `.toString` Object-method fallback + (D-207 / ADR-0076 C3)" — i.e. it IS cljw's `.toString`. Each of the 8 + fns currently does `if (args[0].tag() != .string) raise(.type_arg_not_string)` + (`src/lang/primitive/string.zig:40-41` and siblings); replace that guard + with a nil-check plus a writeStrValue render, leaving the other 8 fns' + guards untouched. + last_reviewed: "2026-08-20" + - id: "D-580" + status: "DISCHARGED 2026-08-20 — all three negative-index arms (vector, array_seq, transient_vector) now raise index_out_of_range; verified against clj for vector/list/string and the default-wins path. Was: open (2026-08-20, measured). `(nth [1 2 3] -1)` raises Kind ClassCastException in cljw; clj raises IndexOutOfBoundsException. The POSITIVE out-of-range path is already right — `(nth [1 2 3] 99)` raises IndexOutOfBoundsException on both — so only the negative-index arm picked the wrong class." + category: "clj-parity" + barrier: "Ready. Note this is a Kind mismatch, and AD-007 accepts error-Kind divergence in general — but not here: the same function already answers IndexOutOfBoundsException for the sibling case, so the two arms disagree with each other, which no invariant defends. Check `(nth \"abc\" -1)` too (clj: StringIndexOutOfBoundsException)." + last_reviewed: "2026-08-20" - id: "D-447" status: |- DISCHARGED 2026-08-05 — every named gap closed or settled, corpus-backed. @@ -2868,7 +3017,7 @@ discharged: appended to regex_equivalence.txt; pin e2e regex_parity_gaps.sh (3 cases incl. the catchable/uncatchable split). The compile.zig header doc now states the true surface (the row's depth-1 doc-fix item). - category: clj-parity + category: "clj-parity" barrier: none — discharged last_reviewed: "2026-08-05" - id: "D-446" @@ -2901,7 +3050,7 @@ discharged: first half of the same arc; this completes it. ADR-0182 Alternative C (io.async cancellation) remains a possible future shape — its precondition (a per-evaluation budget object) now exists. - category: concurrency + category: "concurrency" barrier: none — discharged last_reviewed: "2026-08-05" - id: "D-383" @@ -3158,9 +3307,9 @@ discharged: barrier: "none — all four are niche reader/edn edges (repros in the status). (1) the `:tag` shorthand is a bounded reader-metadata change but type-hints are no-ops in cljw → low value; (2) `#=` read-eval is a deliberate non-feature (likely AD); (3) `#?`-in-read-string leniency is a one-check tighten (or AD); (4) `namespace-munge` is a small missing fn. Take up opportunistically / on a real consumer; tracked so they are not silently floating." last_reviewed: "2026-06-16" - id: "D-455" - status: "DISCHARGED 2026-06-17 — cl-format COMPLETE (chunk1-4 all landed; see barrier). Historical detail: FOUR sub-areas DONE — common directive surface COMPLETE. (4) NUMERALS `~R`: cardinal (42→\"forty-two\", groups joined \", \" clj-faithful), ordinal `~:R` (\"forty-second\"), Roman `~@R` (\"XCIX\"), radix `~NR` (`~16R` 255→\"ff\" via Long/toString); cl-cardinal/cl-ordinal/cl-roman helpers, all clj-oracle byte-matched. e2e phase14_cl_format 30 cases. (1) NUMBER-FORMATTING: a directive-PARAMETER parser (`cl-dir` parses `~mincol,'padchar:@d` → params + flags) + `~F` (fixed float `~,Df`), `~X`/`~O` (radix), `~B` (binary), `~:D` (grouped), padded/width `~D`/`~X` (`~5,'0d`→00042, `~8,'*d`→******42, `~6x`→' ff'), delegating to the working `format` fn. (2) ITERATION `~{~^~}`: `cl-run` returns [acc remaining-args] so `~^` exits early; `cl-iter` re-applies the enclosed sub-format per list element (`cl-close` finds the matching `~}`); clj-byte-matched (`~{~a~^, ~}` [1 2 3]→\"1, 2, 3\"). (3) CASE-REGION `~(~)`: `~(` lower / `~:(` cap-each-word / `~@(` cap-first / `~:@(` upper (`cl-close` generalized to the close char `~)`, `cl-case` + manual `cl-cap-words` — no regex literal, which a bundled .clj cannot load). clj-byte-matched. e2e phase14_cl_format 24 cases + smoke green; all directives probe-validated clj-oracle byte-match first. (5) **~C + ~& DONE 2026-06-17** (debt-sweep): `~C` char (plain; `~{~c~}` iterates a string's chars), `~&` fresh-line (`~N&` adds N-1 more newlines), both clj-oracle byte-matched. e2e phase14_cl_format now 36 cases. (6) **~P/~*/~T DONE 2026-06-17** (chunk2): an index arg-navigator restructure of cl-run (argv+pos, not seq) enabled ~P plural (~:P back-up / ~@P y-ies), ~* arg-jump (~N* / ~N:* / ~N@*), ~T tabulate (cl-col/cl-tab). (7) **~$/~E/~G DONE 2026-06-17** (chunk3): ~$ monetary (~d,n,w,padchar$ + `:` sign-before-pad) + the FULL CLtL Steele ~E/~G exponential/general float (cl-float-parts reuses cljw's shortest-round-trip `(str f)`; complete ~w,d,e,k,oc,pc,exp param surface incl. negative params `~9,3,2,-2E` + overflow/exponent chars). All clj-oracle byte-matched; the foo-e/foo-g torture suite is pinned in test/diff/clj_corpus/cl_format_float.txt (13 cases). e2e phase14_cl_format now 64 cases. The minimal cl-run reimpl PROVED SUFFICIENT — the earlier 'needs a full upstream arg-navigator + column-writer port / next unit is ADR-level' assessment was OVER-PESSIMISTIC; the index-navigator + per-directive arms carried the whole surface with no architecture rewrite. REMAINING (chunk4 only): ~[~;~] conditional + ~<~> justification (the `unsupported-raises` e2e probes `~`), plus the `V`/`#` runtime-valued directive params (a cl-dir gap, D-458 — raises cleanly today). Everything except chunk4 + V/# is DONE." + status: "DISCHARGED 2026-06-17 — cl-format COMPLETE (chunk1-4 all landed; see barrier). Historical detail: FOUR sub-areas DONE — common directive surface COMPLETE. (4) NUMERALS `~R`: cardinal (42→\"forty-two\", groups joined \", \" clj-faithful), ordinal `~:R` (\"forty-second\"), Roman `~@R` (\"XCIX\"), radix `~NR` (`~16R` 255→\"ff\" via Long/toString); cl-cardinal/cl-ordinal/cl-roman helpers, all clj-oracle byte-matched. suite test/clj/suites/cl_format_test.clj 30 cases. (1) NUMBER-FORMATTING: a directive-PARAMETER parser (`cl-dir` parses `~mincol,'padchar:@d` → params + flags) + `~F` (fixed float `~,Df`), `~X`/`~O` (radix), `~B` (binary), `~:D` (grouped), padded/width `~D`/`~X` (`~5,'0d`→00042, `~8,'*d`→******42, `~6x`→' ff'), delegating to the working `format` fn. (2) ITERATION `~{~^~}`: `cl-run` returns [acc remaining-args] so `~^` exits early; `cl-iter` re-applies the enclosed sub-format per list element (`cl-close` finds the matching `~}`); clj-byte-matched (`~{~a~^, ~}` [1 2 3]→\"1, 2, 3\"). (3) CASE-REGION `~(~)`: `~(` lower / `~:(` cap-each-word / `~@(` cap-first / `~:@(` upper (`cl-close` generalized to the close char `~)`, `cl-case` + manual `cl-cap-words` — no regex literal, which a bundled .clj cannot load). clj-byte-matched. suite test/clj/suites/cl_format_test.clj 24 cases + smoke green; all directives probe-validated clj-oracle byte-match first. (5) **~C + ~& DONE 2026-06-17** (debt-sweep): `~C` char (plain; `~{~c~}` iterates a string's chars), `~&` fresh-line (`~N&` adds N-1 more newlines), both clj-oracle byte-matched. suite test/clj/suites/cl_format_test.clj now 36 cases. (6) **~P/~*/~T DONE 2026-06-17** (chunk2): an index arg-navigator restructure of cl-run (argv+pos, not seq) enabled ~P plural (~:P back-up / ~@P y-ies), ~* arg-jump (~N* / ~N:* / ~N@*), ~T tabulate (cl-col/cl-tab). (7) **~$/~E/~G DONE 2026-06-17** (chunk3): ~$ monetary (~d,n,w,padchar$ + `:` sign-before-pad) + the FULL CLtL Steele ~E/~G exponential/general float (cl-float-parts reuses cljw's shortest-round-trip `(str f)`; complete ~w,d,e,k,oc,pc,exp param surface incl. negative params `~9,3,2,-2E` + overflow/exponent chars). All clj-oracle byte-matched; the foo-e/foo-g torture suite is pinned in test/diff/clj_corpus/cl_format_float.txt (13 cases). suite test/clj/suites/cl_format_test.clj now 64 cases. The minimal cl-run reimpl PROVED SUFFICIENT — the earlier 'needs a full upstream arg-navigator + column-writer port / next unit is ADR-level' assessment was OVER-PESSIMISTIC; the index-navigator + per-directive arms carried the whole surface with no architecture rewrite. REMAINING (chunk4 only): ~[~;~] conditional + ~<~> justification (the `unsupported-raises` e2e probes `~`), plus the `V`/`#` runtime-valued directive params (a cl-dir gap, D-458 — raises cleanly today). Everything except chunk4 + V/# is DONE." category: "clj-parity" - barrier: "DISCHARGED 2026-06-17 — cl-format is COMPLETE on the minimal cl-run (no upstream architecture rewrite needed). chunk1 ~C/~&, chunk2 ~P/~*/~T, chunk3 ~$/~E/~G (full CLtL Steele), chunk4a ~[~;~] conditional (nesting-aware via cl-close-nested + cl-clauses), chunk4b ~<~;~> justification (full ~mincol,colinc,minpad,padchar< + :/@ + ~^ drop). All clj-oracle byte-matched; e2e phase14_cl_format 82 cases + clj_corpus/cl_format_float.txt 13 pins. Remaining gaps are documented divergences that RAISE cleanly: the `V`/`#` runtime-valued directive params (D-458) and the ~<…~:;…~> pretty-print column mode (needs a column-tracking writer cljw's pr-alias pprint lacks). cljw pprint.clj stayed a minimal cl-run reimpl throughout — the prior 'ADR-level full port' assessment was wrong." + barrier: "DISCHARGED 2026-06-17 — cl-format is COMPLETE on the minimal cl-run (no upstream architecture rewrite needed). chunk1 ~C/~&, chunk2 ~P/~*/~T, chunk3 ~$/~E/~G (full CLtL Steele), chunk4a ~[~;~] conditional (nesting-aware via cl-close-nested + cl-clauses), chunk4b ~<~;~> justification (full ~mincol,colinc,minpad,padchar< + :/@ + ~^ drop). All clj-oracle byte-matched; suite test/clj/suites/cl_format_test.clj 82 cases + clj_corpus/cl_format_float.txt 13 pins. Remaining gaps are documented divergences that RAISE cleanly: the `V`/`#` runtime-valued directive params (D-458) and the ~<…~:;…~> pretty-print column mode (needs a column-tracking writer cljw's pr-alias pprint lacks). cljw pprint.clj stayed a minimal cl-run reimpl throughout — the prior 'ADR-level full port' assessment was wrong." last_reviewed: "2026-06-17" - id: "D-374" status: "CORRECTION 2026-08-12 (Discussion #14): the discharge below claimed the unroll was 'routed through ALL form-loading paths' and listed five. It was not — `src/app/eval_session.zig` (backing BOTH `cljw repl` and `cljw nrepl`) and `src/app/builder.zig` (two loops, backing `cljw build` and the bundled-.clj AOT) each kept their own analyze-then-eval loop, so a top-level `(do (require …) (use …))` failed in the REPL, over nREPL, and at build time while `-e` worked. An external user hit it (three entry points diverging) rather than the gate. This is the D-177 false-positive-discharge class — coverage was LISTED, not probed. Now genuinely closed: eval_session routes through `driver.evalTopLevelForm` (74db35d0, contributor fix), builder enumerates via the new `driver.topLevelLeaves` (fd5f0449), and the claim is now MECHANICAL rather than narrative — `scripts/check_entrypoint_surface.sh` fails if any of the four form-stream sites stops calling the shared driver, and `test/e2e/entrypoint_eval_parity.sh` diffs 8 programs across all six entry points with no expected values written down. ORIGINAL DISCHARGE (2026-06-16, gaps/bugs sweep). cljw now unrolls a top-level `(do …)`: each child is analyzed+evaluated IN SEQUENCE, so an effect in an earlier child (`(import …)` / `(defmacro …)`) is visible to a later child's ANALYSIS — `(do (defmacro m [] 42) (m))` → 42 (was a 'macro Var not callable' analyze error), `(do (import …) (instance? … x))` matches clj. Impl: driver.zig `topLevelDoChildren(form)` detector + `evalTopLevelForm` (recursive; do's value = last child, empty `(do)`→nil) routed through ALL form-loading paths — runner.zig (`-e`/main), loader.zig (file load/require; per-leaf chunk to the `cljw build` sink so replay order is preserved), evaluator.zig (`--compare` dual-backend oracle, backend-parameterized twin), driver.evalValue (so `(eval '(do …))` unrolls), bootstrap.zig (consistency; no-op for hand-written core). e2e phase14_eval (5 new cases: defmacro-then-use / def-then-use / value-is-last / nested-do / eval-of-do) + smoke green (diff oracle ×2 + corpus 3206 + zlinter). RESIDUAL (narrow, not re-filed): a MACRO expanding to a top-level `(do …)` is not unrolled (only a LITERAL `do` head is matched); clj macroexpands first. No real case hits it (declare→do-of-defs needs no effect-sequencing). ORIGINAL: open (2026-06-10, surfaced by the D-373 corpus). cljw does NOT unroll a top-level `(do …)` — it analyzes the whole form before evaluating, so a side-effecting form and a later form that depends on its effect cannot share one compilation unit. Concretely: `(do (import 'java.io.BufferedReader) (instance? BufferedReader x))` → cljw NameError (BufferedReader resolved at analyze time, before the import effect ran); clj → true (clj treats each top-level-`do` child as if a separate top-level form, compiling+evaluating in sequence). This was MASKED for instance? until D-373: the old macro quoted the class symbol so it resolved at EVAL time (after import). The REALISTIC forms — `(ns … (:import …))` then use, OR separate top-level forms — both work in cljw and match clj (verified). Only the single-`-e`-`(do (import …) (use …))` bundle diverges. Removed the artifact corpus line (io_stream_class.txt:25) that depended on the old late-binding; the divergence is a bug scheduled for fix when top-level-do-unrolling lands." @@ -3256,7 +3405,7 @@ discharged: prints to *out*. Unsupported directives raise EXPLICITLY (no silent mishandle — provisional-marker row-4 compliance). Pure Clojure (loop/recur over the fmt string, char-literal directive dispatch — char literals work in - bundle, verified). e2e phase14_cl_format (~A/~S/~D/~%/~~ + aesthetic-string + + bundle, verified). suite test/clj/suites/cl_format_test.clj (~A/~S/~D/~%/~~ + aesthetic-string + unsupported-raises, 7 cases clj-matched). DEFERRED (the rest of the DSL): ~F/~E float formatting, ~{~} iteration, ~R/~B/~O radices, column/justification (~T/~<~>), ~P plural, conditional ~[~]. Add directives as real libs demand them. @@ -4108,7 +4257,7 @@ discharged: resolution: |- runtime | **The `.list` heap tag is used for BOTH a true PersistentList (list/nil tail, valid O(1) `count` field) AND a Cons-over-a-seq (`(cons x lazy_seq)` / `(conj (range 3) 99)`, whose tail is a non-list seq).** Consequence: the Cons `.count` field (`1 + countOf(tail)`, and `countOf` returns 0 for non-list tails) is a lie for mixed chains, and the list-specialized fast paths (printList / deepRealize `.list` arm / countFn `.list`) assumed a homogeneous `.list` chain. Surfaced 2026-05-31 by D-168 (range→lazy-seq exposed `(cons x non-list-seq)`, previously hidden because range was an eager vector routed via seqFn). **Correctness fixed at D-168** by walking `.list` generically (count + print no longer trust the O(1) field for mixed chains) — but that drops the O(1) `count` optimisation for pure PersistentLists. The finished form (per F-004 seq-family slot map: `lazy_seq` / `cons` / `chunked_cons` are distinct tags) **splits `.list` (PersistentList, O(1) count, list-only tail) from `.cons` (ASeq Cons over any seq, O(n) walk)** — matching JVM `clojure.lang.PersistentList` vs `Cons`. The `.cons` tag already exists but is vestigial (only a value.zig test creates it). Activating it + per-tag first/rest/seq/count/print is an F-004 structural decision → defer to that owner (F-003). NOT a per-op patch. Cross-ref D-168 (the correctness fix) / F-004 (NaN-box 2nd-gen seq family) / F-011. | when the F-004 NaN-box 2nd-gen (D-027) seq-family cycle opens OR a corpus test needs O(1) list count back | 2026-05-31 - id: "D-179" - discharged_at: "structural — seq-family slot co-activation (`.string_seq` A13 / `.array_seq` A14); defer to their owning rows (F-003)" + discharged_at: "structural — seq-family slot co-activation (`.string_seq` A13 / `.array_seq` A14); defer to their owning rows (F-003). FULLY CONSUMED 2026-08-22: `.array_seq` activated by O-058, `.string_seq` by O-060 (byte-offset view; `(seq s)` O(n²)→O(n), the 89,910 ms/146k eager cons-chain removed). Both reuse the range.zig/array_seq compact-view substrate as the deferral planned." resolution: |- runtime | **`.range` (A12) was activated as a compact value (ADR-0063); its two Group-A neighbours `.string_seq` (A13) and `.array_seq` (A14) share the identical pattern** (compact descriptor + index cursor + O(1) count/nth + chunked-cons `seq`) but stay vestigial. ADR-0063's Devil's-advocate Alt-3 noted activating all three together would amortise the ~14-arm seq call-site surgery once instead of three times; rejected for ADR-0063 as an F-003 scope-explosion (no F-NNN fixes "activate the seq family together") + F-005 trespass (bigint bounds). **When the `.string_seq` / `.array_seq` owning row opens, reuse ADR-0063's `range.zig` chunked-seq substrate** so each becomes a one-arm fall-through, not a repeat surgery. ALSO: `.range` inherits **D-165** (elements / count beyond ±2^47 silently become float via `initInteger`) and **D-164** (empty range → nil) unchanged — both F-005/seq-model owner territory, not fixed by ADR-0063. Cross-ref ADR-0063 / D-163 / D-165 / D-164 / F-004 (A13/A14 slots) / F-005. | when the `.string_seq` / `.array_seq` owning row opens OR a corpus test needs string-seq/array-seq value identity | 2026-05-31 - id: "D-180" @@ -4120,9 +4269,11 @@ discharged: resolution: |- runtime | **Transient hash map `assoc!` is persistent-path O(log32 n) copy-on-write, not an in-place editable transient** (ADR-0064 Alt 1). Past `ARRAY_MAP_THRESHOLD` (8) the transient holds a persistent `.hash_map` in `overflow` and delegates `assoc!`/`dissoc!` to `map.assoc`/`map.dissoc` — correct (the result equals `(reduce conj {} …)` byte-for-byte) but with no in-place allocation win, so `(into {} big)` is O(n log n) like the persistent path, not the O(1)-amortised-assoc + O(1)-persistent! a true transient gives. **Finished-form candidate (named by the ADR-0064 Devil's-advocate): an editable CHAMP HAMT with a ClojureScript-style edit token** (a freshly-minted sentinel `?*anyopaque` compared by identity — cljs proves this works single-threaded, fitting cljw's no-threading + no-`pub var` constraints). Cost: adds an `edit` field to the `HamtMapNode` `extern` layout (F-004) + a parallel `ensureEditable`-gated editable-insert path through `hamtAssoc`/`insertDataEntry`/`pushDownDataToNode`; also inherits the unfinished collision bucket (D-155). NOT Alt 2 (flat + open-addressing side index + `map.fromSlice`) — the ADR-0064 DA showed `map.fromSlice` is not an O(n) bulk build (CHAMP slot placement is hash-driven), so Alt 2 pays for a second hash mechanism (F-011 liability) for only a constant-factor win. Surfaced 2026-05-31 by the O-003 into/vec transient routing (D-180). Cross-ref ADR-0064 / D-045 / D-155 / D-156 / F-002 / F-004 / F-011. | blocked-by: a corpus / bench measurement shows large-map `into` / `assoc!` as a hot path (then build the editable-CHAMP transient per ADR-0064 Alt 3) | 2026-05-31 - id: "D-175" - discharged_at: "open — debt-hygiene (M2/M3/M5); tech-debt audit 2026-05-31; quality-loop floor" + discharged_at: "2026-10-08 — residue dropped into per-row ownership (see resolution tail)" resolution: |- process | **Execute the remaining tech-debt-consolidation wiring** (`.dev/tech_debt_consolidation.md`): (1) re-anchor M2 dead-Phase barriers — **the high-value Phase-7.x semantic cluster D-086/087/088/090/091 is DONE 2026-05-31 (Barrier → `quality-loop floor:`)**; remaining = the lower-value Lens C audit/cleanup/reserve set (D-016/019/023/024/041/042/043) — re-anchor to `quality-loop floor` / Phase-15 entry in a dedicated housekeeping unit (deferred from the session-clear to avoid risky edits to the wide-format rows); (2) rewrite M3 weak barriers (D-012/015/022/044, D-038/047 externals); (3) M5 housekeeping — move ~22 inline-DISCHARGED rows into `## Discharged`, fix the D-018 duplicate, flip D-064, fix stale PROVISIONAL `rename-keys` (set.clj:73 — DRAINED 2026-06-02: marker removed, `feature_deps#clojure.set/rename-keys`→`landed`, nth form kept per map-invert precedent) + stale comments (bootstrap.zig D-058; test comments concat/mapcat-seqs + project/rename-meta); (4) investigate §B suspects (namespace-registry `user`-dup + phantom `example` ns, VM-DEFER reactivation compiler.zig:346/516, transducer single-arity / phantom `D-NEW-2`). Per-item detail + proposed barriers in the consolidation doc + `private/notes/audit-lens{A..E}-*.md`. | quality-loop floor: debt-hygiene (drain alongside the correctness floor; re-run the 5-lens audit each Phase boundary) | 2026-05-31 + + Closed 2026-10-08 (todo-300 triage, card 20260911005500-2e1bc578). The row sat in `discharged:` while `discharged_at` read "open", so it was both closed and open. Re-measured residue on origin/staging dfbe861f: (1) Lens C set — D-016/D-019 are `standing:` rows, D-023/D-042 are `active:` rows with their own barriers, D-024/D-041/D-043 are discharged; (2) M3 weak barriers — D-015 standing, D-022 active with its own barrier, D-012/D-038/D-044/D-047 discharged; (3) M5 housekeeping — D-018 and D-064 discharged; the gate promotion this row tracked is live (`test/run_all.sh` step `debt_id_refs` runs `check_debt_id_refs.sh --gate`, green). Every remaining item is owned by its own row, so this umbrella holds no live work; (4) the §B suspects were never filed and are dropped here, detail kept in memory 20260911004942-2c0c9674 (undrained C/D/E items) and 20260911004940-38e9586e (M1-M5 leak modes). - id: "D-355" status: "DISCHARGED 2026-06-09 — the $MY/playground-v2 backend now runs ON cljw, babashka-free (commit 87f4730 in that repo): sandbox.clj evaluates untrusted code IN-PROCESS via cljw.eval/with-budget (steps/deadline/heap, breach recovered as a value so the server survives a runaway); server.clj uses cljw.fs / cljw.json / cljw.http.server / clojure.edn (no babashka.fs/process/cheshire/httpkit); static serving via byte-transparent slurp (wasm/png raw); path traversal confined by CLJW_FS_ROOT. Enabled by ADR-0125 (with-budget, prior session) + ADR-0126 (clojure.java.io + cljw.json/cljw.fs, this session). Verified end-to-end (cljw-wasm -M:cljw -m playground.server): (* 6 7)->42, output capture, runaway->budget-error + server alive, static index.html. RESIDUAL: true CRASH isolation (a Zig panic/UAF still takes the process down — no subprocess shell now) stays D-354 (gated on cljw-as-wasm), as the original row anticipated. ORIGINAL: open — playground self-isolation (drop babashka). The $MY/playground-v2 demo currently uses a babashka supervisor that spawns a sandboxed child `cljw -e` (OS `timeout`/`ulimit`/`CLJW_FS_ROOT`) because cljw lacks an in-process eval budget. Once D-351 (eval deadline) + D-352 (heap cap) land, the eval can run IN-PROCESS with time+memory bounds, dropping babashka for everything except crash containment (D-354). Full self-isolation (no external supervisor at all) is gated on cljw-as-wasm (D-354 endgame). Surfaced 2026-06-09 (user-directed demo work)." category: "demo" @@ -4742,7 +4893,7 @@ discharged: last_reviewed: "2026-06-07" - id: "D-272" status: "DISCHARGED 2026-06-06 — ratio arithmetic that reduces to a whole number now yields a BigInt (`2N`), not a Long (`2`), matching clj's JVM Ratio (Numbers.divide returns BigInt when denom=1). Was a DELIBERATE, unit+e2e-tested Long-collapse (promote.zig ratioArith + phase14_ratio_arith) — but F-005's verbatim makes the EXPRESSED behaviour (見た目・表出する振る舞い) JVM-compatible, so `(* 1/2 4)`→2N is the contract and the Long-collapse was a surface divergence F-005 forbids. Single fix point: ratioArith's integer-collapse (the sole path for any ratio-operand +/-/*//) now allocFromManaged(.bigint) instead of wrapManaged; divPromoting/quotPromoting's integer-path collapse already did this for ratio/BigInt operands. Updated 2 unit tests + 7 phase14_ratio_arith assertions (the pinned Long values were the bug). Corpus ratio_integer_collapse (14/14 clj-matched: `(* 1/2 4)`→2N, `(+ 1/3 2/3)`→1N, `(- 1/2 1/2)`→0N; `(/ 6 3)`→2 int-div + `(/ 1 2)`→1/2 ratio + `(+ 1/2 0.5)`→1.0 float all unchanged). Cross-ref F-005, F-011." - category: numeric + category: "numeric" last_reviewed: "2026-06-06" - id: "D-269" status: "DISCHARGED 2026-06-06 — read-string FLIPPED to throw EOF (eof_unexpected) on empty/whitespace/comment-only input, matching clj; an `:eof` opt still overrides (presence-detected via map contains, so `{:eof nil}` correctly returns nil). Alternatives considered: (b) record an AD for nil-default — REJECTED because no F-NNN invariant backs it (accepted_divergences.md: 'cannot name the invariant → it is a bug') and nil-on-failed-read is a textbook silent-failure. The prior phase9 Case 8 pin (nil) was the bug locked in; flipped to assert the throw + the :eof override. Cross-ref F-011, silent-failure principle." @@ -4783,7 +4934,7 @@ discharged: Cross-ref F-005, F-011. last_reviewed: "2026-06-06" - id: "D-265" - status: "DISCHARGED 2026-06-06 — `biginteger` registered as `biginteger ≡ bigint` (math.zig primitive table, mirroring the in-tree `long ≡ int` alias): cljw collapses clj's BigInt/BigInteger into one `.big_int` (F-005), so biginteger yields the same BigInt bigint does. VALUE-identical to clj (`(= (biginteger 5) (bigint 5) 5)`→true, `(+ (biginteger 5) 1)`→6N both); the `5N` print + class BigInt divergence (clj `5` / java.math.BigInteger) is AD-016 (derives F-005; clj's BigInteger is arithmetically write-only → every op returns BigInt, which cljw reproduces for free). DA-subagent confirmed Option A F-005-mandated (Option B would re-materialize the BigInteger type F-005 abolished + break the D-204 name↔Tag round-trip). Corpus biginteger (6/6 clj-matched value/arith); pin e2e phase14_biginteger (4 divergence locks)." + status: "DISCHARGED 2026-06-06 — `biginteger` registered as `biginteger ≡ bigint` (math.zig primitive table, mirroring the in-tree `long ≡ int` alias): cljw collapses clj's BigInt/BigInteger into one `.big_int` (F-005), so biginteger yields the same BigInt bigint does. VALUE-identical to clj (`(= (biginteger 5) (bigint 5) 5)`→true, `(+ (biginteger 5) 1)`→6N both); the `5N` print + class BigInt divergence (clj `5` / java.math.BigInteger) is AD-016 (derives F-005; clj's BigInteger is arithmetically write-only → every op returns BigInt, which cljw reproduces for free). DA-subagent confirmed Option A F-005-mandated (Option B would re-materialize the BigInteger type F-005 abolished + break the D-204 name↔Tag round-trip). Corpus biginteger (6/6 clj-matched value/arith); pin suite test/clj/suites/biginteger_test.clj (4 divergence locks)." category: "clj-parity" barrier: |- **`biginteger` is unimplemented** — `(biginteger 5)` → cljw name_error, clj @@ -6442,8 +6593,8 @@ discharged: barrier: "none — DISCHARGED. The lowering gap (method-under-foreign-interface-header) is resolved via inheritance-flatten. data.finger-tree's remaining blocker is the D-422 runtime segfault." last_reviewed: "2026-06-14" - id: "D-420" - status: "DISCHARGED 2026-06-14. math.numeric-tower loads + conforms. Landed across cycles: BigDecimal `(.setScale bd n mode)` (all 8 ROUND_* modes) + ROUND_* static constants + bare `BigDecimal`/`BigInteger` default-import; Ratio `.numerator`/`.denominator` instance interop (ratio_methods.zig); and the final unblock D-421 (resolve class symbols → the `(when-available clojure.lang.BigInt …)` guard now emits the BigInt MathFunctions extend, so `round` works). Full-surface verify.clj green (round/floor/ceil/sqrt/expt/gcd/lcm/abs/exact-integer-sqrt, value-equal to clj). The ONE remaining divergence — floor/ceil on a ratio → cljw Long `2`/`3` vs clj BigInt `2N`/`3N` — is CLASSIFIED as AD-031 (F-005 narrow-when-fits; root is cljw's `(numerator 5/2)` returning Long; values `=`, NOT a bug; pinned by phase14_ratio_accessors num+long_eq). verified_projects/math.numeric-tower/{deps.edn :verify alias, verify.clj}. e2e phase14_bigdecimal_setscale (19) + phase14_ratio_interop (6) + phase14_var_resolve (15). Notes: p14-bigdecimal-setscale.md + p14-resolve-class-symbol.md." - category: "numerics" + status: "DISCHARGED 2026-06-14. math.numeric-tower loads + conforms. Landed across cycles: BigDecimal `(.setScale bd n mode)` (all 8 ROUND_* modes) + ROUND_* static constants + bare `BigDecimal`/`BigInteger` default-import; Ratio `.numerator`/`.denominator` instance interop (ratio_methods.zig); and the final unblock D-421 (resolve class symbols → the `(when-available clojure.lang.BigInt …)` guard now emits the BigInt MathFunctions extend, so `round` works). Full-surface verify.clj green (round/floor/ceil/sqrt/expt/gcd/lcm/abs/exact-integer-sqrt, value-equal to clj). The ONE remaining divergence — floor/ceil on a ratio → cljw Long `2`/`3` vs clj BigInt `2N`/`3N` — is CLASSIFIED as AD-031 (F-005 narrow-when-fits; root is cljw's `(numerator 5/2)` returning Long; values `=`, NOT a bug; pinned by phase14_ratio_accessors num+long_eq). verified_projects/math.numeric-tower/{deps.edn :verify alias, verify.clj}. suite test/clj/suites/bigdecimal_test.clj (19) + phase14_ratio_interop (6) + phase14_var_resolve (15). Notes: p14-bigdecimal-setscale.md + p14-resolve-class-symbol.md." + category: "numeric" barrier: "none — DISCHARGED. verify_projects math.numeric-tower 1/1; AD-031 records the floor/ceil narrowing." last_reviewed: "2026-06-14" - id: "D-421" @@ -6493,7 +6644,7 @@ discharged: last_reviewed: "2026-06-14" - id: "D-438" status: 'DONE 2026-06-14 (ADR-0140, landed same cycle it was minted). Caught-exception cljw frame-seq accessor: (stack-trace e) → a vector of {:ns :fn :file :line :column} maps, innermost-first (:fn bare, :ns separate), reading the ExInfo own deep-copied frames (ADR-0120); empty for a non-ex_info or never-thrown ex-info. Consumers: clojure.stacktrace/print-stack-trace prints / (:) per frame (AD-029 amended: marker→cljw frames, format still diverges from JVM); Throwable->map :trace/:at filled as the same maps (D-389 discharged; AD-033). DA fork chose Alt 2 (one map shape everywhere). user-frames-only via AD-024 (push-time elision). THIS ROW ALSO FIXES the dangling D-232 cross-reference: AD-029/D-389/sweep_plan named the validation-campaign row D-232 as the frame-accessor owner, but that owner row never existed (Reservation-as-bias) — D-438 + ADR-0140 are it. Pins: test/e2e/phase15_clojure_stacktrace.sh + phase14_throwable_map.sh.' - category: clj-parity + category: "clj-parity" barrier: none — DONE this cycle (ADR-0140). last_reviewed: "2026-06-14" - id: "D-534" @@ -6643,7 +6794,7 @@ discharged: actually dereferences (e.g. core.protocols, the error-render peer, core-macro helpers). last_reviewed: "2026-06-24" - id: "D-514" - status: "DISCHARGED (2026-06-24). BigInteger instance-method surface complete: abs/negate/signum/gcd/pow/mod/sqrt + modPow (square-and-multiply) + bitLength + isProbablePrime (deterministic Miller-Rabin, witnesses {2..37}; 561 Carmichael→false verified). bigint_methods.zig on the .big_int native descriptor; e2e phase14_biginteger (29 cases). Only `.modPow` with a NEGATIVE exponent (modular inverse via extended Euclid) is left unimplemented — currently raises; rare, low priority (re-file if a real consumer needs it)." + status: "DISCHARGED (2026-06-24). BigInteger instance-method surface complete: abs/negate/signum/gcd/pow/mod/sqrt + modPow (square-and-multiply) + bitLength + isProbablePrime (deterministic Miller-Rabin, witnesses {2..37}; 561 Carmichael→false verified). bigint_methods.zig on the .big_int native descriptor; suite test/clj/suites/biginteger_test.clj (29 cases). Only `.modPow` with a NEGATIVE exponent (modular inverse via extended Euclid) is left unimplemented — currently raises; rare, low priority (re-file if a real consumer needs it)." category: "clj-parity" barrier: |- DISCHARGED — BigInteger instance methods complete bar negative-exponent modPow @@ -6795,7 +6946,7 @@ discharged: partial removes the realistic footgun's crash meanwhile. last_reviewed: "2026-06-21" - id: "D-458" - status: "DISCHARGED (2026-06-20). `cl-dir` now emits a `:cl-arg` sentinel for `V`/`v` and `:cl-remaining` for `#`; `cl-run` resolves both against argv BEFORE dispatch via `cl-resolve-params` (V pulls + CONSUMES the next operand, advancing pos; # = remaining-arg count, no consume), so EVERY parameterised directive gets V/# uniformly with no per-arm change. `~n[`/`~#[` clause-select added (a prefix param selects + consumes no arg). e2e phase14_cl_format: param-V-mincol/-padchar/-radix/-money/-float-w/-float-d, param-hash-mincol, param-hash-select-1/-2, param-n-select; corpus test/diff/clj_corpus/cl_format_params.txt (10 cases). KNOWN clj DIFF NOT chased: consecutive-V on a float (`~V,VF` 8 2) — clj is internally INCONSISTENT there (gives \"3.14159000\" ≠ its own `~8,2F` \" 3.14\"); cljw produces the consistent literal-equivalent value. Too niche to AD (a clj bug, not a cljw gap; promote-to-AD if a real corpus hits it). SEPARATE pre-existing gap found + filed as D-465: `~F` without a d-param defaults to 0 decimals (`~8F` → cljw \" 3\" vs clj \" 3.14159\" full-precision). ORIGINAL: open (2026-06-17, surfaced completing cl-format ~E/~G, D-455 chunk3)." + status: "DISCHARGED (2026-06-20). `cl-dir` now emits a `:cl-arg` sentinel for `V`/`v` and `:cl-remaining` for `#`; `cl-run` resolves both against argv BEFORE dispatch via `cl-resolve-params` (V pulls + CONSUMES the next operand, advancing pos; # = remaining-arg count, no consume), so EVERY parameterised directive gets V/# uniformly with no per-arm change. `~n[`/`~#[` clause-select added (a prefix param selects + consumes no arg). suite test/clj/suites/cl_format_test.clj: param-V-mincol/-padchar/-radix/-money/-float-w/-float-d, param-hash-mincol, param-hash-select-1/-2, param-n-select; corpus test/diff/clj_corpus/cl_format_params.txt (10 cases). KNOWN clj DIFF NOT chased: consecutive-V on a float (`~V,VF` 8 2) — clj is internally INCONSISTENT there (gives \"3.14159000\" ≠ its own `~8,2F` \" 3.14\"); cljw produces the consistent literal-equivalent value. Too niche to AD (a clj bug, not a cljw gap; promote-to-AD if a real corpus hits it). SEPARATE pre-existing gap found + filed as D-465: `~F` without a d-param defaults to 0 decimals (`~8F` → cljw \" 3\" vs clj \" 3.14159\" full-precision). ORIGINAL: open (2026-06-17, surfaced completing cl-format ~E/~G, D-455 chunk3)." category: "clj-parity" barrier: |- none — localized to cl-dir. Finished form: when cl-dir sees `V`/`v` it emits a @@ -6806,7 +6957,7 @@ discharged: or after D-455 chunk4. Repro: `(cl-format nil "~VD" 5 42)` → cljw raises (clj → " 42"). last_reviewed: "2026-06-20" - id: "D-465" - status: "DISCHARGED (2026-06-20, same session as the D-458 find). ~F without a d-param now prints the natural (shortest round-trip) value in PLAIN fixed notation, always with a decimal point, left-padded to w — never scientific. New helpers in pprint.clj: `cl-float-natural` (= `(str (double x))`, then `cl-expand-exp` rewrites any E-notation the printer emitted to plain) + `cl-parse-int` (self-contained, no read-string). Surface check done: ~E/~G already handled no-d correctly (cl-efloat/cl-gfloat) and ~$ defaults d=2 — so ONLY ~F needed the fix. e2e phase14_cl_format: float-natural/-0/-int/-w/-big/-small/-neg + param-V-float (re-added now ~VF works); corpus cl_format_params.txt (+8 ~F cases). All clj-oracle-verified incl. extreme magnitudes (1e10→\"10000000000.0\", 1.5e-7→\"0.00000015\")." + status: "DISCHARGED (2026-06-20, same session as the D-458 find). ~F without a d-param now prints the natural (shortest round-trip) value in PLAIN fixed notation, always with a decimal point, left-padded to w — never scientific. New helpers in pprint.clj: `cl-float-natural` (= `(str (double x))`, then `cl-expand-exp` rewrites any E-notation the printer emitted to plain) + `cl-parse-int` (self-contained, no read-string). Surface check done: ~E/~G already handled no-d correctly (cl-efloat/cl-gfloat) and ~$ defaults d=2 — so ONLY ~F needed the fix. suite test/clj/suites/cl_format_test.clj: float-natural/-0/-int/-w/-big/-small/-neg + param-V-float (re-added now ~VF works); corpus cl_format_params.txt (+8 ~F cases). All clj-oracle-verified incl. extreme magnitudes (1e10→\"10000000000.0\", 1.5e-7→\"0.00000015\")." category: "clj-parity" barrier: |- none — localized to cl-run's ~F arm (pprint.clj). Finished form: when the @@ -6890,12 +7041,12 @@ discharged: barrier: "none — landed. All recognized instance? markers now resolve; corpus instance_interfaces.txt locks them." last_reviewed: "2026-06-25" - id: "D-532" - status: "DISCHARGED 2026-06-25 — java.math.BigInteger surface complete. Construction (UUID 2-long ctor / BigInteger String+byte[] + valueOf + .toBigInteger / BigDecimal (BigInteger,int) ctor) landed earlier this session; THIS cycle added the (b) instance arithmetic .add/.subtract/.multiply/.divide (bigint_methods.zig delegating to big_int allocAddManaged/SubManaged/MulManaged + a NEW allocDivTruncManaged — `.divide` truncates toward ZERO (-7/2=-3), distinct from the floor allocDivFloorManaged (-4); divide-by-zero raises). clj-verified (13N/7N/30N/3N/-3N + 10^24 exact); e2e phase14_biginteger bi_add/subtract/multiply/divide(_neg/_zero) (35 cases). Residual (a) — a data.generators 'Invalid float literal …E-117' edge — is NON-REPRODUCIBLE standalone (the exact 88/200-digit mantissa parses clj-match), so it is not actionable from active; a future data.generators sweep that reproduces it opens a fresh row with the concrete failing double." + status: "DISCHARGED 2026-06-25 — java.math.BigInteger surface complete. Construction (UUID 2-long ctor / BigInteger String+byte[] + valueOf + .toBigInteger / BigDecimal (BigInteger,int) ctor) landed earlier this session; THIS cycle added the (b) instance arithmetic .add/.subtract/.multiply/.divide (bigint_methods.zig delegating to big_int allocAddManaged/SubManaged/MulManaged + a NEW allocDivTruncManaged — `.divide` truncates toward ZERO (-7/2=-3), distinct from the floor allocDivFloorManaged (-4); divide-by-zero raises). clj-verified (13N/7N/30N/3N/-3N + 10^24 exact); suite test/clj/suites/biginteger_test.clj bi_add/subtract/multiply/divide(_neg/_zero) (35 cases). Residual (a) — a data.generators 'Invalid float literal …E-117' edge — is NON-REPRODUCIBLE standalone (the exact 88/200-digit mantissa parses clj-match), so it is not actionable from active; a future data.generators sweep that reproduces it opens a fresh row with the concrete failing double." category: "interop" barrier: "none — surface complete. The non-reproducible (a) float round-trip re-files only on a concrete repro." last_reviewed: "2026-06-25" - id: "D-511" - status: "DISCHARGED 2026-06-25 — the exact-binary `(BigDecimal. double)` ctor landed (the last D-511 remnant; the 2-arg `(BigDecimal. x mc)` form was already done 8db6d82f). big_decimal.allocFromDoubleExact decomposes the f64 IEEE-754 bits into significand·2^exp2, then REUSES allocFromRatioParts (numer=significand·2^exp2 / denom=1 for exp2>=0, else numer=significand / denom=2^|exp2|) — the gcd-reduction there yields the exact minimal scale for free (no manual trailing-zero-bit strip needed), matching JDK. `(BigDecimal. 0.1)` => 0.1000000000000000055511151231257827021181583404541015625 (scale 55), -0.1 negated, 0.0/-0.0 => 0, NaN/±Inf => NumberFormatException (number_format_invalid). A `.float` arm in parseBigDecimalArg (CTOR only — valueOf(double) keeps shortest-round-trip). clj-verified; e2e phase14_bigdecimal_setscale bd_dbl_* (8 cases, 113 total). The 'high impl cost' the row feared dissolved by reusing allocFromRatioParts." + status: "DISCHARGED 2026-06-25 — the exact-binary `(BigDecimal. double)` ctor landed (the last D-511 remnant; the 2-arg `(BigDecimal. x mc)` form was already done 8db6d82f). big_decimal.allocFromDoubleExact decomposes the f64 IEEE-754 bits into significand·2^exp2, then REUSES allocFromRatioParts (numer=significand·2^exp2 / denom=1 for exp2>=0, else numer=significand / denom=2^|exp2|) — the gcd-reduction there yields the exact minimal scale for free (no manual trailing-zero-bit strip needed), matching JDK. `(BigDecimal. 0.1)` => 0.1000000000000000055511151231257827021181583404541015625 (scale 55), -0.1 negated, 0.0/-0.0 => 0, NaN/±Inf => NumberFormatException (number_format_invalid). A `.float` arm in parseBigDecimalArg (CTOR only — valueOf(double) keeps shortest-round-trip). clj-verified; suite test/clj/suites/bigdecimal_test.clj bd_dbl_* (8 cases, 113 total). The 'high impl cost' the row feared dissolved by reusing allocFromRatioParts." category: "clj-parity" barrier: "none — landed. Both ctor remnants done; valueOf(double) stays shortest-round-trip (distinct path)." last_reviewed: "2026-06-25" @@ -7158,7 +7309,7 @@ discharged: chunk-layout change becomes an on-rodata-format change (accepted; the format is already AOT-embedded). Survey: serialize.zig EnvelopeIterator/deserializeChunk + the BytecodeChunk shape; cw v0 / wasmtime for in-place/position-independent chunk - formats. Watch the JIT (ADR-0200) may operate on a different layer — do not + formats. Watch the JIT (zwasm ADR-0200) may operate on a different layer — do not over-invest in a format it would replace. ADR-0172 L2 fold-in (2026-07-16): the binary-SIZE lever on the same bytes is co-decided in THIS redesign, not built separately — eager core region stays @@ -7322,7 +7473,118 @@ discharged: D-349 (the memory half, discharged); zwasm `src/cli/run.zig` `Limits.timeout_ms`; src/runtime/cljw/wasm/engine.zig (`RunOpts`, where the three existing axes are threaded). + - id: "D-565" + status: |- + DISCHARGED 2026-10-01. All eight items done. The 2026-07-28 audit + (Discussion #11, a fresh-clone onboarding sweep) found that a stranger's + clone builds, tests and gets CI with no host setup; what broke was doc + staleness. Items 1-6 were drained 2026-08-04; a re-sweep on 2026-10-01 + found the same rot had come back in new places, fixed here: + (1) retired co-dev dir names. `.dev/ubuntunote_setup.md` still pointed at + `zwasm_from_scratch/` (five sites); it now cites zwasm's + `.dev/ubuntunote_setup.md` by upstream URL. `.claude/CLAUDE.md` and + `.dev/wasm_percall_findings.md` named `~/PP/referential-projects/zwasm`, + which does not exist on this machine; the clone is `~/PP/zwasm` + (origin `BuddhiLW/zwasm`, the fork `build.zig.zon` pins). Verbatim + quotes in `project_facts.md`, closed ADRs and the zwasm ledger history + keep the old names as a record. + (2) dead reference paths. No `~/Documents/OSS` exists on the current + maintainer machine, so every row in `.dev/reference_clones.md` is now + a `` recipe with a verified `git clone` URL + (clojure, babashka, spec.alpha, clojuredocs-export-edn, malli, + mattpocock skills, ezi-gex, plus the existing openjdk and wasmtime). + `clojure-corpus` has no public source and says so. The mac-only Zig + stdlib and `clj` paths are gone (`zig env` `.std_dir`, `clj` from + PATH). `check_reference_clones.sh` passes with a populated root (9 + on-demand recipes) and skips without one. + (3) `scripts/cleanup_orphans.sh` is in-repo and wired as a SessionStart + hook; `--dry-run` re-verified 2026-10-01. Stale `.claude/rules/*.md` + citations near it now name memory ids. + (4) README still names no current version (Releases badge); size claims + stay gated by `size_claims` (ADR-0172). + (5) the `continue` skill names no model. Its resume procedure still read + the retired `.dev/handover.md` and the deleted `.dev/principle.md`, + and called `main` the work branch; it now starts from + `project workflow catchup` on `staging`. + (6) README banner states the real posture; CONTRIBUTING lists what loop + conventions do not apply to an outside contributor. + (7) `.github/CONTRIBUTING.md` "Tools the gate needs, and tools it does + not": the gate set (zig, bb, python3, yq, rg, as in flake.nix and + ci.yml), `clj` (oracle tools only, PATH-resolved, `CLJ=` override), + `md-table-align` (advisory hook, silent when absent), reference clones + and `settings.local.json` `additionalDirectories`, and that no hook + forces a chat language (`outputStyle` left settings.json earlier). + Also corrected its branch advice: PRs target `staging`. + (8) the handover file is retired (2026-09-11), so its "First task on + resume" no longer exists. `private/notes/*.md` citations are marked + local-only in the `continue` skill, `.dev/reference_clones.md` and + CONTRIBUTING; the one load-bearing ledger was already promoted to + `test/diff/clj_corpus/COVERAGE.md`. + WORKS (unchanged): flake.nix pins Zig 0.16.0; build.zig.zon fetches zwasm + and zlinter by URL and hash; the e2e steps that read a reference clone + are guarded and skip when it is absent; CI runs `scripts/ci_gate.sh`. + category: "docs-onboarding" + barrier: "none, discharged" + last_reviewed: "2026-10-01" + - id: "D-568" + status: |- + DISCHARGED 2026-10-01. `wasm/resource-drop` maps zwasm's `NoResourceTable` to its own Code, `wasm_resource_no_table`, whose template names the real condition (a single core module with no resource table) instead of the catch-all `wasm_component_trap`. Exercised, not asserted: the new fixture `test/e2e/fixtures/wasm/single_resource.{wat,wasm}` is a hand-written component with one core module and no imports, so zwasm `open` takes the `.single` path, and its export returns `own`, so the host holds a resource it cannot drop. The fixture needs an exported `memory` or the single-path canon context fails every call. + category: "wasm" + barrier: |- + none, discharged. Pinned by `test/e2e/phase16_wasm_require_component.sh` (marker `PASS resource-drop-no-table`, probe `test/e2e/fixtures/wasm_resource_no_table_probe.clj`), which asserts the message names the missing resource table and does not say trapped. + first_raised: "2026-08-04" + last_reviewed: "2026-10-01" + refs: |- + .dev/decisions/0159_wasm_resource_lifecycle.md § Amendment 1 (the verification item that raised it); src/runtime/cljw/wasm/component.zig `resourceDropFn` (the NoResourceTable arm); src/runtime/error/catalog.zig `wasm_resource_no_table`; zwasm src/api/component.zig `dropResource`. + discharged_at: "2026-10-01" + - id: "D-566" + status: |- + DISCHARGED 2026-10-02. The three parts landed together. + (1) Stamp: `GcHeap.owner_thread_id`, set in `GcHeap.init` from + `root_set.currentThreadId()` (0 on single-threaded targets). + (2) Assert: `GcHeap.allocSized` asserts `mutatorAllowed()` (registered + worker, or the heap owner); `root_set.pushAnalysisRoot` asserts + `analysisMutatorAllowed()`. `pushAnalysisRoot` has no heap in hand, so it + checks a process-wide stamp of the most recent heap owner + (`noteHeapOwner`, which a registered worker never overwrites). + (3) Test audit of every `std.Thread.spawn` under src/: the gc_heap + concurrent-alloc workers and the lazy_seq force-race workers now register + through `root_set.workerContext`; vm.zig and the safepoint.zig workers + already registered; the root_set churn, eval_budget and latch threads + never touch the heap. No exemption was needed. + MEASURED: with the gc_heap test registration removed, `zig build test` + aborts that test (`reached unreachable code`), 1 crash of 1366. The + lazy_seq mock thunk allocates nothing, so the assert does not fire there + either way; its registration is for shape, not for the assert. + Was: open (2026-07-29, ADR-0175 deferred-hardening row), an unregistered + mutator was a ~1.5% GP-fault flake instead of a caught bug. + category: "concurrency" + barrier: |- + none, discharged. Pinned by the gc_heap.zig test "mutatorAllowed: the + stamping thread and registered workers may alloc, a foreign unregistered + thread may not (D-566)" and by the assert itself under every ReleaseSafe + unit and e2e run. Cross-ref ADR-0175 § Consequences. + last_reviewed: "2026-10-02" + - id: "D-346" + status: |- + DISCHARGED 2026-10-02 by "fix(vm): build large collection literals without exhausting the operand stack (D-346)". Both options the barrier named landed. A vector, map or set literal whose elements are all constants folds into one constant in the analyzer (clj ConstantExpr), so no operand stack is involved at any size. A literal with an evaluated element compiles to one build op for the first 512 elements and one op_vector_extend / op_map_extend / op_set_extend per further step, so the operand stack holds at most one step. A full operand arena now raises the catchable `stack_overflow` and a compiler limit raises the catchable `form_too_large`, neither an internal error. JVM clj rejects these 40k literals with "Method code too large!"; cljw builds them. + Was: open, security (VM operand stack on a large literal). `(count [1 1 (40000 elements)])` ended the process with the uncatchable `internal_error: vm: operand stack overflow` (exit 70). Surfaced 2026-06-09 while testing the tokenizer fix; reproduced unchanged 2026-08-04. + reviewed_2026_08_04: |- + REPRODUCED, unchanged, still low. `(count [1 1 … x40000])` from a file exits + 70 with "vm: operand stack overflow" — a clean uncatchable process end, not + a crash or a memory-safety fault. Reachable only by EVALUATING untrusted + code (read-string builds the collection in the reader, not via operand + pushes), which is already the higher-trust scenario, so the severity + ordering behind the deferral holds. + category: "security" + barrier: |- + none, discharged. Pinned by `test/e2e/bulk_literal.sh` (both backends through --compare, the eval path, the catchable limits, the cljw build round trip), `test/clj/suites/bulk_literal_test.clj`, the bulk_literal diff corpus and the diff_test.zig D-346 case. + last_reviewed: "2026-10-02" + refs: |- + src/runtime/collection/literal.zig (the one literal construction); src/eval/analyzer/analyzer.zig (the constant fold); src/eval/backend/vm/compiler.zig and vm.zig (the extend opcodes); docs/spec/formats/chunk-v12.edn (bytecode format v12). + discharged_at: "2026-10-02" conventions: |- + - Category vocabulary is a closed set (guarded by scripts/check_debt_category.sh; synonyms were normalized 2026-10-08: structural->structure, tests/test->testing, numerics->numeric, performance->perf). Allowed: app backend bench bootstrap build clj-parity concurrency correctness debt-hygiene demo discipline distribution docs docs-onboarding ecosystem error-system eval eval-semantics feature gc governance host-surface infra interop isolation license macro meta-reorg missing-ns namespace numeric perf polymorphism process project-flow reader refactor regex robustness runtime scope security structure testing tooling wasm. A new category is added here first, in the same commit as the row that needs it. - Structure: two top-level lists — `active:` and `discharged:`. Active entries carry id / status / category / barrier (+ optional quality_floor / last_reviewed); discharged entries carry id / discharged_at / resolution. Edit as YAML (no Markdown table to align). - New debt: grep `.dev/debt.yaml` for class-overlap before adding (per `.claude/rules/debt_dedup.md`). - Discharge: move the entry from `active:` to `discharged:` (do not leave an inline-discharged copy under `active:`). diff --git a/.dev/decisions/0000_template.md b/.dev/decisions/0000_template.md deleted file mode 100644 index 2ecbd0f50..000000000 --- a/.dev/decisions/0000_template.md +++ /dev/null @@ -1,54 +0,0 @@ -# NNNN — - -- **Status**: Proposed | Accepted | Superseded by NNNN | Deprecated -- **Date**: YYYY-MM-DD -- **Author**: -- **Tags**: , - -## Context - -What is the situation that calls for a decision? Constraints, prior art, -relevant ROADMAP sections, related ADRs. - -## Decision - -The chosen path, in declarative form. One paragraph if possible. - -## Alternatives considered - -For each rejected option: - -### Alternative A — - -- **Sketch**: … -- **Why rejected**: … - -### Alternative B — - -- **Sketch**: … -- **Why rejected**: … - -## Consequences - -- **Positive**: … -- **Negative**: … -- **Neutral / follow-ups**: items that need separate ADRs or future tasks. - -## Affected files - -(per ADR-0020 — required for ADR-0020 onward) - -- `src/path/to/file.zig` — what changes here -- `.dev/ROADMAP.md` §X.Y — what is amended -- `.claude/rules/foo.md` — new / amended -- (etc.) - -## References - -- ROADMAP § N.M -- Related ADRs: NNNN -- External: links / file paths - -## Revision history - -- YYYY-MM-DD: Status: Proposed -> Accepted (initial landing). diff --git a/.dev/decisions/0001_macroexpand_routing.md b/.dev/decisions/0001_macroexpand_routing.md deleted file mode 100644 index 85f407873..000000000 --- a/.dev/decisions/0001_macroexpand_routing.md +++ /dev/null @@ -1,220 +0,0 @@ - # 0001 — Route macroexpansion through a Layer-1 MacroTable, not the Runtime VTable - -- **Status**: Accepted -- **Date**: 2026-04-27 -- **Author**: Shota Kudo (with Claude Opus 4.7) -- **Tags**: macros, analyzer, layering, dispatch - -## Context - -Phase 3.7 (`§9.5 / 3.7` in ROADMAP) introduces Zig-level expansions for the -bootstrap macros (`let → let*`, `when`, `cond`, `if-let`, `when-let`, -`and`, `or`, `->`, `->>`). The question: **where does macro expansion -live, and how does the analyzer reach the impl?** - -Constraints in scope: - -- **`zone_deps.md`** — `eval/` (Layer 1) cannot import `lang/` (Layer 2). - Any analyzer-to-`lang/macro_transforms` call must go through some - inversion of control. -- **`zone_deps.md` Guard 3** — `pub var` vtables are forbidden. -- **ROADMAP P3** — "Core stays stable; new features go in new files". - Macroexpansion belongs in a dedicated module, not crammed into - analyzer or backend. -- **ROADMAP P12** — TreeWalk and VM must agree on every test from - Phase 8 onward. Macroexpansion must be **backend-agnostic**. -- **ROADMAP P6** — Error quality is non-negotiable. Macro errors must - attribute to the call-site `SourceLocation`. -- **ROADMAP §13** — "vtable on `Runtime`" is the project's chosen IoC - pattern, but only when the inverted dependency is genuinely a - Layer-0-level concern. - -Carrying state into the decision: - -- The current `runtime/dispatch.zig::VTable` already has an - `expandMacro` slot, installed as `expandMacroStub` in - `eval/backend/tree_walk.zig::installVTable`. This pre-existing slot - is **vestigial** from an early sketch where backend and macro - expansion were intermixed. -- The Step-0 survey for 3.7 - (`private/notes/phase3-3.7-survey.md`, gitignored) found that - ClojureWasm v1 implements the same nine macros (and 48 more) as - Form→Form transforms, dispatched inline from `analyzeList()` via a - `StaticStringMap` lookup, with **no backend coupling**. Clojure JVM - invokes `Compiler.macroexpand1` from inside `analyzeSeq`, again - without backend involvement. Babashka/SCI runs all macros through - the SCI evaluator (no host fast path), but is interpreter-only and - does not separate analysis from execution. - -## Decision - -Macroexpansion is **invoked from the analyzer** via an explicitly -threaded `MacroTable`, declared in a new Layer-1 module -`eval/macro_dispatch.zig`. The Zig-level transforms live in -`lang/macro_transforms.zig` (Layer 2) and register themselves into the -table at startup. The table is constructed in `main.zig`, populated -once, and passed to `analyze` as a parameter alongside `*Runtime` and -`*Env`. - -``` -Layer 0 (runtime): no macro concerns -Layer 1 (eval): macro_dispatch.Table (type) + analyzer consults it -Layer 2 (lang): macro_transforms.zig populates the Table -Layer 3 (app): main.zig builds the Table at startup -``` - -Concrete shape: - -```zig -// eval/macro_dispatch.zig -pub const ZigExpandFn = *const fn ( - rt: *Runtime, - env: *Env, - args: []const *const Form, // call-site argument forms (not lifted) - loc: SourceLocation, // call-site loc for error attribution -) anyerror!*const Form; // expanded form, ready for re-analysis - -pub const Table = struct { - entries: std.StringHashMapUnmanaged(ZigExpandFn) = .empty, - // … register / lookup / deinit -}; - -pub fn expandIfMacro( - rt: *Runtime, - env: *Env, - table: *const Table, - head_var: *const Var, // already resolved by analyzer - args: []const *const Form, - loc: SourceLocation, -) anyerror!?*const Form; // null = not a macro Var; non-null = expanded -``` - -Concurrently, **`runtime/dispatch.zig::VTable.expandMacro` is removed**, -along with `ExpandMacroFn`, `mockExpandMacro`, and -`tree_walk.zig::expandMacroStub`. The `VTable` shrinks to -`{ callFn, valueTypeKey }`. - -The user-defined `defmacro` path (Phase 3.12) reuses the same -`expandIfMacro` entry point; when the head Var has `flags.macro_` set -but is not in the Zig `Table`, `expandIfMacro` falls back to invoking -`vtable.callFn` on the macro Var's fn-Value with `args` lifted to -Values, then converts the returned Value back to a Form. The -Form↔Value boundary therefore exists in **exactly one place** — the -user-fn invocation site — and never crosses zone boundaries. - -## Alternatives considered - -### Alternative A — Inline `lang/macro_transforms` import in analyzer - -- **Sketch**: `eval/analyzer.zig` directly imports - `lang/macro_transforms.zig` and calls into a `StaticStringMap` at - comptime, the way v1 does. -- **Why rejected**: Violates `zone_deps` (Layer 1 → Layer 2). v1 - predates the explicit zone contract, and v1's code organization is - one of the things this v2 redesign is fixing. ROADMAP §4.1 makes the - upward-import prohibition load-bearing. - -### Alternative B — Keep `expandMacro` on `Runtime.vtable`, impl in `lang/` - -- **Sketch**: Retain `runtime.vtable.expandMacro`, but install the impl - from `lang/macro_transforms.zig` instead of TreeWalk. The signature - uses `Value` (not `Form`) since Layer 0 cannot reference Layer 1 - types. -- **Why rejected**: (1) Macroexpansion is not a Layer-0 concern; - carrying it on `Runtime.vtable` misnames the responsibility. - (2) Forces a Form→Value→Form round-trip even for Zig-level transforms - that have no semantic reason to traverse the Value domain. - (3) Makes `Runtime` look like it knows about macros, polluting the - Layer-0 interface for tests and embedders. - -### Alternative C — Put `expandMacro` on TreeWalk's `installVTable` - -- **Sketch**: Treat macro expansion as a backend concern, the way the - current `expandMacroStub` does. -- **Why rejected**: Violates ROADMAP P12 (dual-backend equivalence). - Phase 4+ adds the VM backend; macroexpansion would have to be - duplicated or factored back out — exactly the rework this ADR - forecloses. - -### Alternative D — Separate macroexpand pass before analyze - -- **Sketch**: `Reader → MacroExpand (Form→Form) → Analyze` as three - distinct passes, mirroring naive Lisp pedagogy. -- **Why rejected**: Macros depend on the lexical scope established - during analysis (e.g., a `let` binding can shadow a global macro - Var). Running macroexpand as a standalone pass would either - re-implement the analyzer's Var-resolution logic or be limited to - global macros. Inlining the expansion check into `analyzeList` - matches Clojure JVM and ClojureWasm v1. - -### Alternative E — Module-level `pub var` registry in `eval/macro_dispatch` - -- **Sketch**: `pub var registry: ?Table = null;` in - `eval/macro_dispatch.zig`, populated at startup, consulted by - analyzer. -- **Why rejected**: `zone_deps.md` Guard 3 forbids `pub var` for - vtables. The rule exists because mutable module-level state breaks - multi-tenant test isolation and conflates one Runtime with another. - -## Consequences - -### Positive - -- **Zone-clean**. analyzer (Layer 1) imports only `eval/macro_dispatch` - (Layer 1). `lang/macro_transforms` (Layer 2) imports `eval/...` and - `runtime/...` per the contract. No upward imports. -- **Backend-agnostic**. TreeWalk and VM see only the post-expansion - Node tree. The Phase 8 dual-backend `--compare` gate is automatically - satisfied for the macro path. -- **Single dispatch entry point** for both Zig transforms and Phase - 3.12 user `defmacro`. Adding user macros at 3.12 requires no change - to analyzer wiring — only a fallback branch inside `expandIfMacro`. -- **Form↔Value boundary is localised** to the user-fn invocation site. - Zig transforms operate on Forms throughout; no allocation overhead - for the static cases. -- **`(macroexpand-1 form)` primitive** (Phase 5+) is trivial: the - builtin resolves the head Var and calls `expandIfMacro` directly. -- **`Runtime.vtable` simplifies** to `{ callFn, valueTypeKey }`, - shrinking the Layer-0 surface and making the contract clearer. - -### Negative - -- **Threading change**: `analyze` gains a `macro_table` parameter. - Every call site (5 internal recursive calls in analyzer, plus - `main.zig` and the test fixtures in `analyzer.zig` and - `tree_walk.zig`) needs updating in one commit. This is a deliberate - surgical change, not a workaround — see ROADMAP P3. -- **Two error-attribution locations** for macros: the Zig transform's - own arity / shape errors (attributed to call-site `loc`) and the - user-fn macro path's `vtable.callFn` errors. Both will route through - `setErrorFmt` so the renderer shape is uniform. - -### Neutral / follow-ups - -- The Form→Value→Form bridge for user `defmacro` is deferred to Phase - 3.12. The `expandIfMacro` skeleton in 3.7 returns `error.NotImplemented` - for the user-macro branch with a `not_implemented` Kind so the - failure mode is clean. -- A `(macroexpand-1 form)` primitive is **not** in 3.7 scope; it - becomes natural after Phase 3.10 (ex-info) lands and is logged as a - Phase 5+ candidate. -- `lang/macro_transforms.zig` will grow as more macros land in Phases - 3–5; ROADMAP A6 (≤ 1000 lines per file) means we may split it - thematically (`bool.zig` for and/or, `threading.zig` for `->`/`->>`, - etc.) when we cross ~600 lines. - -## References - -- ROADMAP §2 (P3, P6, P12), §4.1 (Zone), §9.5 (Phase 3 task list), - §13 (Reject patterns) -- `.claude/rules/zone_deps.md` — zone contract -- `.claude/rules/textbook_survey.md` — Step 0 procedure -- `private/notes/phase3-3.7-survey.md` — Step 0 survey for this ADR - (gitignored) -- v1 reference: `~/Documents/MyProducts/ClojureWasm/src/lang/macro_transforms.zig` -- Clojure JVM: `clojure/src/jvm/clojure/lang/Compiler.java::macroexpand1` - (line ~7566) and `analyzeSeq` (line ~7673) - -## Revision history - -- 2026-04-29: Status: Proposed -> Accepted (initial landing, retroactive history added 2026-05-23) diff --git a/.dev/decisions/0002_phase3_exit_no_map_literal.md b/.dev/decisions/0002_phase3_exit_no_map_literal.md deleted file mode 100644 index 9cb0a1be8..000000000 --- a/.dev/decisions/0002_phase3_exit_no_map_literal.md +++ /dev/null @@ -1,138 +0,0 @@ -# 0002 — Phase 3 exit smoke uses an integer placeholder for `ex-info` data; map literals stay in Phase 5 - -- **Status**: Accepted -- **Date**: 2026-04-27 -- **Author**: Shota Kudo -- **Tags**: roadmap-amendment, scope, phase-3, phase-5, ex-info, map-literal - -## Context - -ROADMAP §9.5 task 3.14 (Phase-3 exit smoke) was originally written as: - -> `(try (throw (ex-info "boom" {})) (catch ExceptionInfo e (ex-message e)))` → `"boom"` - -Implementation of Phase 3 surfaced two facts that the original wording -did not anticipate: - -- `{}` (the empty map literal) is **not** parsed-and-analysed today. - `src/eval/analyzer.zig:212` raises `not_implemented` with the message - *"Map literal as expression value not yet supported (Phase 3+)"*. -- ROADMAP §9 phase table line 644 scopes **collections (HAMT, Vector) - + Mark-Sweep GC** to **Phase 5**. The same table line 642 frames the - Phase 3 exit at the high level as `(defn f [x] (+ x 1)) (f 2) → 3; - try/catch works` — i.e. the Phase 3 high-level row never required - map literals. - -So §9.5 / 3.14's `{}` was a **drafting overreach** at ROADMAP-creation -time: the detailed task row demanded a feature whose implementation -was assigned to a phase two phases away. The Phase 3 high-level exit -(line 642) and the Phase 5 scope (line 644) were already internally -consistent; only the §9.5 / 3.14 detail line disagreed with both. - -This is exactly the kind of mismatch ROADMAP §17 (Amendment policy) -is designed for. This ADR follows the §17.2 four-step amendment. - -## Decision - -- **D-A**: Phase 3's exit smoke uses `(ex-info "boom" 0)` — an integer - placeholder for the `data` argument — instead of `(ex-info "boom" - {})`. The smoke's purpose is to verify the **try/throw/catch + - ex-info round-trip semantics** end-to-end; `ex-info`'s data slot is - polymorphic, so any non-nil Value works as a placeholder. The - semantics being verified are unchanged. -- **D-B**: Map literal support stays scoped to Phase 5 alongside the - rest of HAMT / persistent-map work. We do **not** ship a minimal - empty-map stub during Phase 3; that would add a new heap tag, a - printValue arm, an equality + hash branch, and a deinit path purely - to satisfy a smoke string, with no follow-on consumer in Phase 3. - ROADMAP §2 P4 (no ad-hoc patches) and P3 (core stays stable) both - push toward "do it properly when Phase 5 actually arrives." - -§9.5 / 3.14 is rewritten in place to use the integer placeholder. A -trailing parenthetical points to this ADR for cause. - -## Alternatives considered - -### Alternative A — Add minimal empty-map literal support in Phase 3 - -- **Sketch**: add a `heap_map` HeapTag, allocate a sentinel empty-map - Value, support analyser routing `{}` → that Value, render it in - `printValue`, and stub equality / hash. ~80–120 LOC. The full HAMT - structure still lands in Phase 5 — Phase 3 would only support `{}` - and rely on Phase 5 to extend. -- **Why rejected**: This is a P4 (no ad-hoc patches) violation. A - partially-implemented heap_map with one literal shape would have to - be revisited at Phase 5 anyway, with the risk that Phase 5 inherits - a brittle stub instead of a clean greenfield. The smoke verifies - try/catch semantics, not collection semantics; using a stand-in for - the data slot is sufficient and honest. - -### Alternative B — Loosen the smoke to accept any data Value - -- **Sketch**: write the smoke as "any non-nil data argument works" and - let the implementer choose. Document the substitution in the smoke - script comments only, not in ROADMAP. -- **Why rejected**: ROADMAP is the SoT (§17.4). A "your choice" exit - form invites future drift — six months from now nobody can tell - whether the substitution was deliberate or a bug. Making the choice - explicit in ROADMAP + documenting the why in this ADR keeps the - trail readable. - -### Alternative C — Defer the ex-info smoke entirely to Phase 5 - -- **Sketch**: Phase 3 only verifies `(defn f [x] ...) (f 2) → 3`; - ex-info smoke moves to Phase 5 alongside `{}`. -- **Why rejected**: ex-info itself (the heap struct, `ex-message` / - `ex-data` builtins, `try/catch ExceptionInfo` evaluation) shipped in - §9.5 tasks 3.10 + 3.11 and works today. The semantic surface is - Phase-3-complete; only the literal `{}` was misdrafted. Pushing the - whole smoke to Phase 5 would lose end-to-end verification of - Phase-3-shipped behaviour for two phases. - -## Consequences - -- **Positive**: - - ROADMAP self-consistency is restored: §9.5 / 3.14 detail row now - agrees with the §9 phase table (line 642 high-level Phase 3 exit; - line 644 Phase 5 collections scope). - - The smoke is implementable today without scope creep into - collection work. - - `ROADMAP §17` (Amendment policy) gets its first real exercise, - establishing the practice of "amend in place + ADR + handover sync - + commit reference." - -- **Negative**: - - The smoke's `data` argument differs cosmetically from idiomatic - Clojure (`{}` would be the natural empty case). Anyone porting a - Phase-5 example into the Phase-3 smoke needs to remember the - substitution. - -- **Neutral / follow-ups**: - - When Phase 5 lands map literals, the §9.6+ task list should - include a sweep to (a) optionally update §9.5 / 3.14 to the - canonical `{}` form, and (b) add a Phase 5 exit case that uses - `(ex-info "..." {:k v})` to verify map literal + ex-info compose. - A pointer to this ADR goes in the §9.6 task notes. - - This ADR is also the **first §17.2 four-step amendment**; it - incidentally validates that the policy is workable. - -## References - -- ROADMAP §9 phase table (lines 638–660) — Phase 3 (line 642), Phase 5 - (line 644). -- ROADMAP §9.5 / 3.14 — the amended row. -- ROADMAP §17 — Amendment policy (introduced in the same commit as - this ADR). -- ROADMAP §2 P3 (core stays stable), P4 (no ad-hoc patches). -- `src/eval/analyzer.zig:212` — current `not_implemented` site for map - literals. -- Original wording (preserved for trace): - `(try (throw (ex-info "boom" {})) (catch ExceptionInfo e (ex-message e)))` → `"boom"` -- New wording: - `(try (throw (ex-info "boom" 0)) (catch ExceptionInfo e (ex-message e)))` → `"boom"` -- Related ADRs: 0001 (macroexpand routing — same pattern of "deferred - decision surfaces during implementation"). - -## Revision history - -- 2026-04-29: Status: Proposed -> Accepted (initial landing, retroactive history added 2026-05-23) diff --git a/.dev/decisions/0003_zlinter_no_deprecated_gate.md b/.dev/decisions/0003_zlinter_no_deprecated_gate.md deleted file mode 100644 index 2151f274d..000000000 --- a/.dev/decisions/0003_zlinter_no_deprecated_gate.md +++ /dev/null @@ -1,201 +0,0 @@ -# 0003 — Adopt zlinter `no_deprecated` as a Mac-host pre-commit gate - -- **Status**: Accepted -- **Date**: 2026-05-03 -- **Author**: Claude (in-session, on user direction) -- **Tags**: tooling, lint, deprecation, gate, ci - -## Context - -Zig 0.16 silently accepts pre-0.14/0.15 stdlib names through -deprecation shims (`pub const Foo = NewFoo;` carrying `/// -Deprecated:` doc-comments). The compiler emits no warning by -default and there is no `-fdeprecated` flag in 0.16. AI training -corpora overwhelmingly use the old names, and the project has -already accumulated **16 silent deprecation hits** across 9.1 KLoC -that no manual pass surfaced — broken down as: - - - 12 × `std.mem.indexOf{,Scalar}` → `find{,Scalar}` in `src/` - - 2 × `std.mem.indexOf{,Scalar}` → `find{,Scalar}` in - `docs/ja/learn_zig/samples/28_mem_utilities.zig` - - 1 × `std.ArrayListUnmanaged` → `ArrayList` (in `src/runtime/runtime.zig`) - - 1 × `std.StringArrayHashMapUnmanaged` → `array_hash_map.String` - (in `src/runtime/keyword.zig`). - -Three forces converge: - -1. **Manual `zig_tips.md` upkeep cannot keep up.** The rules file - already covers ~40 stdlib renames, but the project will keep - adding code (Phase 4 VM, GC, WASI, …) and the stdlib will keep - deprecating things. A grep-based DIY script would need every - new pattern added by hand and would lag behind upstream `/// - Deprecated:` annotations. -2. **Zig itself does not have `@deprecated()` shipping in 0.16.** - The proposal - ([ziglang/zig#22822](https://github.com/ziglang/zig/issues/22822)) - is accepted on the "urgent" milestone but unmerged; expected - in 0.17+. -3. **The sister project zwasm v2 has already validated the - approach** in its own ADR-0009 (commit landed 2026-05-03), with - sub-second runtime on Mac aarch64 and a clean integration shape. - -A web survey of the Zig lint ecosystem (KurtWagner/zlinter, -DonIsaac/zlint, rockorager/ziglint, AnnikaCodes/ziglint) -identified [`KurtWagner/zlinter`](https://github.com/KurtWagner/zlinter) -as the only tool that: - -- ships a built-in `no_deprecated` rule that consumes the stdlib's - own `/// Deprecated:` doc-comments via ZLS-driven AST analysis - (so it auto-tracks new deprecations without project-side rule - authoring), -- supports Zig 0.16.x explicitly (alongside 0.14.x / 0.15.x / - master), -- integrates as a `b.step("lint", ...)` custom build step (matches - the `zone_check.sh` discipline already in `test/run_all.sh`), -- exits non-zero with `--max-warnings 0`, suitable for CI gating. - -## Decision - -Add `KurtWagner/zlinter` (pinned to the `0.16.x` branch) as a -project dependency and expose it through a single new build step: - -```sh -zig build lint # warnings ok, errors fail -zig build lint -- --max-warnings 0 # any finding fails (gate) -``` - -**Initial rule set: `no_deprecated` only.** Phase B will inspect -the further candidates from zwasm v2's ADR-0009 (`no_orelse_unreachable`, -`no_empty_block`, `require_exhaustive_enum_switch`, `no_unused`) -— that's the *candidate set*; per-rule adoption is decided in -Phase B based on this codebase's own findings (the Update section -records the outcome). The Phase A landing keeps the surface -minimal so the integration itself can be reverted cleanly if -zlinter upstream becomes unmaintained or if `@deprecated()` lands -natively in Zig 0.17+. - -The lint step is **Mac-host only**. `test/run_all.sh` detects -`uname -s == Darwin` and runs `zig build lint -- --max-warnings 0` -only on that branch; OrbStack Ubuntu x86_64 skips it with an -informational line. Reasons: - -- zlinter requires `zig fetch` against GitHub; OrbStack runs are - intentionally network-free per `.dev/orbstack_setup.md`. -- Deprecation findings are platform-independent — a single host is - enough to catch them. - -The pre-commit gate documented in `CLAUDE.md` "Working agreement" -gains the lint check as part of the existing `bash test/run_all.sh` -requirement (no separate item, since the script handles platform -detection internally). - -## Alternatives considered - -### Alternative A — DIY grep script - -- **Sketch**: `scripts/zig_deprecation_check.sh` walks `src/` and - fails if any of a fixed pattern list is matched. -- **Why rejected**: every new stdlib deprecation requires a project - edit. The current 16-hit count includes patterns (`std.mem.indexOf`) - that the existing `zig_tips.md` already calls out — yet the code - drifted in. Manual checklists do not scale; AST-level analysis - does. - -### Alternative B — DonIsaac/zlint - -- **Sketch**: Pull in `zlint` instead. It has its own AST-level - semantic analyser and rules like `unsafe-undefined`, `homeless-try`. -- **Why rejected**: deprecation detection is not its focus - ([README rule list](https://github.com/DonIsaac/zlint)). For a - project that mostly cares about stdlib API drift this is the - wrong axis. - -### Alternative C — Wait for `@deprecated()` builtin - -- **Sketch**: Don't adopt anything; wait for ziglang/zig#22822 to - land natively (likely 0.17.x). -- **Why rejected**: open-ended schedule. Phases 4 → 7 will accumulate - more code in the meantime, increasing the cost of the eventual - back-fix sweep. The cost of adopting and later replacing zlinter - is a single `build.zig` revert and dependency removal — small - enough to make the wait-and-see option strictly worse. - -### Alternative D — All-builtins-on initial integration - -- **Sketch**: Enable all 25 zlinter built-in rules from the start. -- **Why rejected**: zwasm v2's spike showed 81 errors + 1314 - warnings under the same blanket-on configuration, the bulk from - rules mismatched to project conventions (`declaration_naming` - requires identifier length ≥ 3, but the codebase uses math - conventions like `i`, `n`, `rt`). Phase B will widen the rule - set deliberately, one rule at a time. - -## Consequences - -- **Positive**: - - Zero-maintenance deprecation tracking — zlinter consumes the - stdlib's own `/// Deprecated:` annotations. - - One-step gate semantics: `zig build lint -- --max-warnings 0`. - - Sub-second on Mac aarch64; small enough to gate every commit - without perceptible overhead. - - Surfaces the 16 already-rotted call sites that prior manual - passes missed. - -- **Negative**: - - First external dependency in `build.zig.zon`. - - Dependent on a third-party project (`KurtWagner/zlinter`); if - upstream stops shipping a 0.16-compatible branch, we either - fork or fall back to grep. - - Mac-only enforcement means a Linux-only contributor (none today) - could merge without running it. Mitigation: the `/continue` - skill always runs on Mac native first. - -- **Neutral / follow-ups**: - - **Phase B** — widen the rule set per zwasm ADR-0009's validated - list (`no_orelse_unreachable`, `no_empty_block`, - `require_exhaustive_enum_switch`, `no_unused`). Each rule lands - one at a time with a fix pass. - - **Phase C** — case-by-case judgment for the remaining low- - finding rules; deferred until Phase B settles. - - **Sunset path**: when `@deprecated()` and `-fdeprecated` ship - in Zig (likely 0.17+), revisit this ADR. Native compiler - enforcement may obsolete the zlinter dependency entirely. - The sunset trigger is `ziglang/zig#22822`; track it informally - until a `proposal_watch.md` lands in this project. - - `zig-pkg/` (zlinter's package cache) is added to `.gitignore`. - -## Update — 2026-05-03 (Phase B outcome) - -Phase B walked the four candidate rules from zwasm ADR-0009. -Outcome: - -| Rule | Phase | Findings | Outcome | Notes | -|----------------------------------|-------|----------|-------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `no_deprecated` | A | 16 → 0 | **Adopted** | The original Phase-A landing. | -| `no_orelse_unreachable` | B | 0 | **Adopted** | Codebase already used `x.?` everywhere; rule is a forward guard. | -| `no_empty_block` | B | 0 | **Adopted** | Codebase already comments empty bodies; forward guard. | -| `no_unused` | B | 1 → 0 | **Adopted** | Removed dead `error_mod` import in `src/lang/bootstrap.zig:35`. | -| `require_exhaustive_enum_switch` | B | 12 | Not adopted | Mismatched with the project's `Value.Tag` dispatch idiom (36+ tags, intentionally growing through Phases 4–15). Arithmetic / collection / print primitives all use `else =>` to mean "every other kind I do not accept as operand", which is the correct semantic and would degenerate into 36-arm enumeration with no regression-prevention payoff. Re-evaluate when `Value.Tag` stabilises (Phase 8+ likely). | - -Lint runtime stays sub-second on Mac aarch64. The Phase A "Mac- -only" / "skipped on Linux" decisions stand. - -The project deviation from zwasm ADR-0009's adopted set -(4 vs 5 rules) is intentional — the difference reflects the -shape of this codebase's central enum, not a judgement on the -rule itself. - -## References - -- ROADMAP §11 (test strategy / quality gate) and §12 (commit - discipline) — the lint gate slots into the existing - `test/run_all.sh` single-entry pattern. -- Sister project: zwasm v2 ADR-0009 (`~/Documents/MyProducts/zwasm_from_scratch/.dev/decisions/0009_zlinter_no_deprecated_gate.md`) - — the full Phase B / C playbook plus the per-builtin survey. -- Upstream: [KurtWagner/zlinter](https://github.com/KurtWagner/zlinter) -- Native proposal: [ziglang/zig#22822 — `@deprecated()` - builtin](https://github.com/ziglang/zig/issues/22822) - -## Revision history - -- 2026-05-03: Status: Proposed -> Accepted (initial landing, retroactive history added 2026-05-23) diff --git a/.dev/decisions/0004_day1_enum_reserve.md b/.dev/decisions/0004_day1_enum_reserve.md deleted file mode 100644 index 3730786f5..000000000 --- a/.dev/decisions/0004_day1_enum_reserve.md +++ /dev/null @@ -1,92 +0,0 @@ -# 0004 — Day-1 reservation of SpecialFormTag, Opcode, and ValueTag enums - -- **Status**: Accepted -- **Date**: 2026-05-23 -- **Author**: Shota Kudo (drafted with Claude, reviewed by Shota) -- **Tags**: phase-4-entry, enum, day-1, special-form, opcode, value-tag - -## Context - -When the VM lands in Phase 4, adding a new special form or value type -in subsequent phases would require touching reader, analyzer, -tree_walk, vm/compiler, and vm/dispatch in lockstep. The same pressure -collapsed `zwasm v1` and forced the v2 ground-up redesign (single -opcode addition touched 6 files because enums were not sized for the -full target). - -cw v1 has the same risk if `SpecialFormTag`, `Opcode`, and `ValueTag` -are sized only for what Phase 3 needs. By the time deftype / dosync / -monitor-enter / atom enter the language in later phases, every -existing dispatch site needs to be revisited. - -## Decision - -Phase 4 entry reserves the full set of `SpecialFormTag`, `Opcode`, and -`ValueTag` slots for everything cw v1 expects to support through Phase -20. Implementation may be staged across phases, but enum shape is -locked. Adding a new slot after Phase 4 entry requires an amendment to -this ADR. - -### SpecialFormTag (24 slots) - -Phase 1-3 existing (14): `if`, `do`, `let_star`, `fn_star`, `def`, -`quote`, `var_form`, `the`, `loop_star`, `recur`, `try_star`, `catch`, -`finally`, `throw`. - -V3 reserved (10): `deftype`, `defrecord`, `reify`, `definterface`, -`dot` (`.method`), `new`, `set_bang`, `monitor_enter`, `monitor_exit`, -`dosync`. - -### ValueTag - -Phase 1-3 existing tags plus reservations for `typed_instance`, -`reified_instance`, `type_descriptor`, `big_int`, `ratio`, -`big_decimal`, `lazy_seq`, `transient_vector`, `transient_hashmap`, -`atom_`, `ref_`, `agent_`, `future_`, `promise_`, `host_instance`. -The exact slot count is finalized when ADR-0012 lands the "3 slot vs -1 slot + flag" choice. - -### Opcode - -The VM Opcode enum reserves at least 80 slots covering control flow, -stack ops, locals, closure, arithmetic (i64 / i53 unchecked / -fallback), boolean / nil, collection ops (Phase 5+), exception (Phase -3+), namespace / var dispatch, and pseudo-ops for the JIT path -(Phase 17+ go / no-go). - -## Alternatives considered - -### Alternative A — Grow the enums incrementally - -- **Sketch**: add each tag when the implementing phase begins. -- **Why rejected**: existing dispatch sites must be revisited every - time. zwasm v1 lost weeks to this; cw v1 has the same VM shape and - would lose at least as much. - -### Alternative B — Hierarchical enum (`SpecialForm.Class.Tag`) - -- **Sketch**: nested enums for category + variant. -- **Why rejected**: exhaustiveness checking becomes harder, and - `comptime` dispatch tables grow more complex. - -## Consequences - -- **Positive**: dispatch tables are stable from Phase 4. Future phases - add an implementation behind an existing tag rather than a tag and - an implementation. -- **Negative**: the enum has empty slots through Phase 4-6, which is - cosmetically odd but enforceable via exhaustiveness checks (each - unimplemented tag returns a structured `error.NotImplemented`). -- **Neutral / follow-ups**: ADR-0012 finalizes ValueTag concretely; - ADR-0007/0008/0009/0010 land the implementations behind specific - tags. - -## References - -- ROADMAP §A11 (Day-one enum reservation) -- ROADMAP §9.6 task 4.4 (Opcode enum landing) -- Related ADRs: 0007, 0008, 0009, 0010, 0012 - -## Revision history - -- 2026-05-23: Status: Proposed -> Accepted (initial landing). diff --git a/.dev/decisions/0005_dual_backend_differential_oracle.md b/.dev/decisions/0005_dual_backend_differential_oracle.md deleted file mode 100644 index f88f9cdd0..000000000 --- a/.dev/decisions/0005_dual_backend_differential_oracle.md +++ /dev/null @@ -1,72 +0,0 @@ -# 0005 — Dual-backend differential testing as oracle - -- **Status**: Accepted -- **Date**: 2026-05-23 -- **Author**: Shota Kudo (drafted with Claude) -- **Tags**: phase-4-entry, testing, differential, ci-gate, dual-backend - -## Context - -cw v0 implemented `EvalEngine.compare()` to verify TreeWalk and VM -returned identical results, but it ran as a test-only mode behind an -opt-in flag. The two backends were never required to stay in sync at -CI time; this allowed silent drift between them. cw v1 must avoid -this regression. - -Phase 4 introduces the VM. From task 4.10 onward, TreeWalk and VM -both execute the entire e2e suite, and any divergence is a build -failure. From Phase 17 onward, if JIT is added, it joins the same -comparison. - -## Decision - -The implementation of `Evaluator.compare(rt, env, src)` (task 4.10) is -CI mandatory from Phase 4 onward. There is no opt-out flag. -`test/run_all.sh` invokes the comparison runner. Any mismatch between -backends fails the build. - -The comparison protocol (Phase 4): - -1. Compile `src` once to the analyzer AST. -2. Run TreeWalk and VM separately against the same AST. -3. Compare the two results bit-for-bit (Value, environment changes, - exception data). -4. Mismatch -> exit non-zero with both result dumps. - -In Phase 17+, the same protocol extends to JIT (TreeWalk = VM = JIT). - -## Alternatives considered - -### Alternative A — Test-only `--compare` flag - -- **Sketch**: ship `cljw eval --compare` for developer use; CI runs - only one backend. -- **Why rejected**: this is the cw v0 mode that allowed drift. Optional - comparison is not used routinely. - -### Alternative B — Sample-based comparison (10% of tests) - -- **Sketch**: compare a random sampling each CI run. -- **Why rejected**: drift can land in the 90% that is not compared. - Comparison cost is acceptable for full coverage. - -## Consequences - -- **Positive**: TreeWalk and VM stay observably equivalent. Bugs in - either surface immediately. A useful invariant for cw v1 textbook - status. -- **Negative**: CI time approximately doubles for the eval suite. - Acceptable trade-off for Phase 4-15. -- **Neutral / follow-ups**: Phase 17 JIT activation extends the - comparison runner to a third backend. ADR amendment required at - that time. - -## References - -- ROADMAP §A10 (Differential testing oracle) -- ROADMAP §9.6 task 4.10 (Evaluator.compare CI mandatory) -- cw v0 `EvalEngine.compare()` as cautionary precedent - -## Revision history - -- 2026-05-23: Status: Proposed -> Accepted (initial landing). diff --git a/.dev/decisions/0006_wasm_ffi_defer_phase16.md b/.dev/decisions/0006_wasm_ffi_defer_phase16.md deleted file mode 100644 index fd676d317..000000000 --- a/.dev/decisions/0006_wasm_ffi_defer_phase16.md +++ /dev/null @@ -1,117 +0,0 @@ -# 0006 — Defer Wasm FFI to Phase 16 via the Pod boundary - -- **Status**: Accepted -- **Date**: 2026-05-23 -- **Author**: Shota Kudo (drafted with Claude) -- **Tags**: phase-4-entry, wasm, scope, deferred, pod-boundary - -## Context - -cw v0 bundled a "Clojure runtime" and a "Wasm host" into the same -binary. The `cljw.wasm` namespace exposed 17 Wasm-host vars; the -runtime had to initialize zwasm v1, manage GC finalizers for Wasm -modules, and carry `wasm_module` / `wasm_fn` GC types. The complexity -contributed to the bootstrap fragility and the std.Io migration outage -(F140-F144). - -zwasm v2 is still stabilizing (Phase 9, Windows reconcile in flight), -so coupling cw v1 to it now would replay v0's pain. - -## Decision - -cw v1 does not implement Wasm FFI from Phase 4 through Phase 15. The -build flag `-Dwasm=false` is the default, the `cljw.wasm` namespace is -absent, and zwasm is not a dependency. Phase 4 task 4.16 operationalizes -the removal. - -Phase 16 reintroduces Wasm capability via a Pod boundary (out-of-process -or sandboxed in-process module) once zwasm v2 is stable. - -Re-introduction conditions (evaluated at Phase 16 entry): - -1. zwasm v2 has reached Wasm 3.0 100% PASS on three platforms (Mac - aarch64, Linux x86_64, Windows x86_64). -2. A Pod protocol design is settled (Babashka-style is the leading - candidate; ADR-0015 upstream reference policy may also influence - this). -3. cw v1 has reached Tier B with `clojure.data.json` and friends, so - the I/O surface is mature enough to wrap a Pod. - -## Alternatives considered - -### Alternative A — Keep the cw v0 path - -- **Sketch**: continue bundling zwasm and the `cljw.wasm` namespace. -- **Why rejected**: the bootstrap complexity that triggered the v0 - std.Io outage would persist. - -### Alternative B — Permanent removal (Tier D for Wasm) - -- **Sketch**: declare Wasm out of scope forever. -- **Why rejected**: ROADMAP §8 still includes Wasm as a v0.2 ambition. - -### Alternative C — Lightweight bundled Wasm runtime (no zwasm dependency) - -- **Sketch**: implement a minimal Wasm interpreter inline. -- **Why rejected**: scope creep against the Phase 4-15 plan. - -## Consequences - -- **Positive**: bootstrap complexity drops sharply. The F140-F144 - class of regression cannot recur for Phase 4-15. -- **Negative**: Wasm-flavored learning material is delayed until - Phase 16+. -- **Neutral / follow-ups**: Phase 16 entry ADR defines the Pod - protocol concretely. - -## References - -- ROADMAP §3.2 (Wasm scope) -- ROADMAP §8 (Wasm / edge strategy) -- ROADMAP §9.6 task 4.16 (operationalize the removal) -- cw v0 disabled features F140-F144 as cautionary precedent - -## Revision history - -- 2026-05-23: Status: Proposed -> Accepted (initial landing). -- 2026-05-23 (amendment 3): **Wasm FFI is confirmed unavoidable; zwasm v2 - is the target counterparty.** User direction (2026-05-23) clarified - that cw v2 will integrate with `zwasm_from_scratch` (zwasm v2 — still - in development) and that wasm FFI is "an indispensable element". - zwasm v2 itself provides JIT and memory management. Implications: - - The "Pod boundary" framing in §Decision remains the default - *protocol shape*, but the **inline-vs-Pod choice is reopened** - at Phase 16 entry. If inline (NaN-box-tagged) wasm Values are - chosen, the slots released by amendment 1 (`big_int` / `ratio` - now at 29 / 30) cannot be reclaimed — Phase 16 must mint fresh - slots, co-ordinated with D-027 (NaN-box layout 第二世代). - - The cw v2 Phase 5 mark-sweep GC and the cw v2 Phase 17 JIT - (ADR-0005 3rd backend) **overlap territorially** with zwasm v2's - own GC and JIT. Phase 16 entry resolves the heap-boundary and - JIT-coordination design (separate cw-heap vs wasm-heap, JIT - compilation handoff, etc.). - - The Phase 16 re-introduction-conditions list in §Decision - should be re-read with zwasm v2's actual progress in mind, not - v1's. Specifically, condition 1 ("zwasm v2 has reached Wasm - 3.0 100% PASS on three platforms") is the gating signal cw v2 - monitors. - Recorded as debt D-036 for the Phase 16 entry owner. -- 2026-05-23 (amendment 2): **`-Dwasm=false` build option reverted.** - D-028 audit found the option had no consumer — the cw-from-scratch - branch never carried the wasm FFI surface this option was meant to - gate. Phase 16 re-introduction (per the Pod boundary plan above) - mints its own build option at that time. The 4.16 ROADMAP row stays - as a record of the revert; no source surface remains. -- 2026-05-23 (amendment 1): **NaN-box slots for `wasm_module` / - `wasm_fn` are released and re-purposed for `big_int` / `ratio`** - (ADR-0012 amendment 1). Rationale: Phase 16 reintroduces Wasm via - a Pod boundary (out-of-process or sandboxed in-process), which - exposes Wasm modules as opaque host values rather than - NaN-box-tagged inline pointers. The day-1 reservation made sense - while a bundled Wasm runtime was on the table; with the Pod - boundary, Phase 16 can mint fresh slots if (and only if) the Pod - protocol decides to surface Wasm modules as first-class cw - Values. Releasing the slots now lets Phase 4 close `big_int` (and - reserve `ratio`) at their natural NaN-box position rather than the - PHASE4_PLACEHOLDER_TAG smell that 4.23 entry shipped at - commit df6958f. diff --git a/.dev/decisions/0007_type_descriptor_option_beta.md b/.dev/decisions/0007_type_descriptor_option_beta.md deleted file mode 100644 index 78b660c1d..000000000 --- a/.dev/decisions/0007_type_descriptor_option_beta.md +++ /dev/null @@ -1,120 +0,0 @@ -# 0007 — TypeDescriptor (Option β) for the cw class system - -- **Status**: Accepted -- **Date**: 2026-05-23 -- **Author**: Shota Kudo (drafted with Claude) -- **Tags**: phase-4-entry, class-system, type-descriptor, dispatch, option-beta - -## Context - -A corpus audit of 165 mainstream Clojure libraries found `deftype` -1,514, `defrecord` 1,375, `reify` 1,471, and `definterface` 103 -occurrences — together with `class` / `instance?` / `type` dispatch -that pervades Clojure code. Without a Class-like construct, the -ecosystem largely does not run. - -cw v1 has three logical options: - -- **α** (no Class concept) — keyword-only `type`, no `deftype`. Drops - most of Clojure. -- **β** (cw TypeDescriptor) — a Zig-native struct that plays the role - JVM's Class plays for Clojure, without inheriting JVM's design - (no bytecode generation, no reflection runtime). -- **δ** (full JVM Class emulation) — same path cw v0 took and got - stuck on. - -Babashka follows path β successfully. cw v1 commits to the same -direction at design level, while implementing differently to exploit -Zig's comptime and packed-struct strengths. - -## Decision - -cw v1 adopts Option β. The runtime carries a `TypeDescriptor` struct -that: - -- Holds `fqcn` (interned symbol), `kind` - (`native | deftype | defrecord | reify_anon`), optional - `field_layout`, `protocol_impls`, `method_table`, optional `parent`, - and `meta`. -- Lives in the namespace it is registered into (lifetime tied to the - namespace). -- Backs `(class x)`, `(instance? T x)`, and `(type x)` dispatch. - -`TypedInstance` (`deftype` / `defrecord` runtime value) holds a -pointer to its `TypeDescriptor` and an inline `field_values` slice -sized by `field_layout.len` at creation time. - -`reify` produces an anonymous `TypeDescriptor` (no `fqcn`) whose -`method_table` is populated from the form. Closed-over locals live in -`ReifiedInstance`. - -Field layout is determined at analyzer time. Zig's `comptime` is used -where the field count and types are known statically (specialised -deftype emit); otherwise a runtime `[]Value` slice carries the data. - -## Alternatives considered - -### Alternative α — No Class system - -- **Sketch**: drop `deftype`/`defrecord`/`reify`, return `keyword` - from `(class x)`, no per-call-site dispatch cache. -- **Why rejected**: corpus audit shows 4,463 occurrences across the - major libraries; the ecosystem would not run. - -### Alternative δ — JVM Class emulation - -- **Sketch**: implement Class hierarchy, reflection runtime, bytecode - generation. -- **Why rejected**: cw v0 fell into this trap (89K LOC, F140-F144). - Out of scope for cw v1. - -## Consequences - -- **Positive**: 4,463 corpus occurrences become reachable. `class` / - `instance?` / `type` work. `deftype` / `defrecord` / `reify` / - `definterface` work. The cw v1 runtime stays small. -- **Negative**: a `TypeDescriptor` is not a JVM `Class`. Code that - passes `Class` objects through `java.lang.reflect.*` will not work; - this is covered under Tier D in ADR-0013. -- **Neutral / follow-ups**: ADR-0008 wires `.method` dispatch on top - of `TypeDescriptor.method_table`. ADR-0012 finalizes the `ValueTag` - slots that reference `TypeDescriptor` / `TypedInstance` / - `ReifiedInstance`. The concrete Zig struct layout lives in the cw - runtime source. - -## Phase 5+ migration note (amendment 1) - -The Phase 4 skeleton (task 4.17) declares `TypeDescriptor` / -`TypedInstance` / `ReifiedInstance` structs without operations. -Phase 5 entry activation includes a rewrite, not just adding: - -- `TypeDescriptor.lookupMethod` / `register` / `new` implementation - lands. -- Existing Phase 1-3 primitives in `src/runtime/collection/*.zig` - (`string` / `list` / `keyword` / `symbol`) and the Value model gain - a back-pointer (or `ValueTag` route) to a native `TypeDescriptor` - registered at runtime init. The primitive struct shape changes; - this rewrites code already written in Phase 1-3. -- `(class x)` / `(instance? T x)` / `(type x)` switch from the - Phase 3 keyword-only `type` dispatch to `TypeDescriptor`-based - dispatch. - -The rewrite is expected, not technical debt, per ROADMAP §A25. -principle.md depth 3 (cross-file refactor) is the typical depth; -depth 4 only if the Value model itself needs a new ADR. - -## References - -- ROADMAP §A11 (Day-one enum reservation) -- ROADMAP §A25 (Existing code is mutable) -- ROADMAP §9.6 task 4.17 (TypeDescriptor skeleton) -- ROADMAP §9.7 (Phase 5 entry — Skeletons to activate) -- Related ADRs: 0004, 0008, 0012, 0013 -- Babashka precedent: sci.impl.records / sci.impl.types - -## Revision history - -- 2026-05-23: Status: Proposed -> Accepted (initial landing). -- 2026-05-23 (amendment 1): Added "Phase 5+ migration note" section - to narrate the rewrite scope when task 4.17 skeleton activates - in Phase 5 (per ROADMAP §A25). diff --git a/.dev/decisions/0008_protocol_dispatch_unify.md b/.dev/decisions/0008_protocol_dispatch_unify.md deleted file mode 100644 index d7f9fd042..000000000 --- a/.dev/decisions/0008_protocol_dispatch_unify.md +++ /dev/null @@ -1,600 +0,0 @@ -# 0008 — Unify `.method` and protocol dispatch through TypeDescriptor - -- **Status**: Accepted -- **Date**: 2026-05-23 -- **Author**: Shota Kudo (drafted with Claude) -- **Tags**: phase-4-entry, dispatch, protocol, dot, host-interop - -## Context - -The corpus carries 5,901 distinct `.method` patterns (top 15 alone: -`.get` 1,126; `.write` 1,094; `.getName` 791 …). Constructor shorthand -`(ClassName. ...)` and `(new ClassName ...)` add several thousand more -occurrences. Threading macros (`->`, `->>`, `..`, `doto`) compound the -total. - -On the JVM these reach `VarHandle`, `MethodHandle`, and ultimately -`invokevirtual` with C2's polymorphic inline cache. cw v1 has no JVM -and no JIT (Phase 17 go / no-go), so the dispatch strategy must be -designed up front. - -## Decision - -Every `.method` call dispatches through the same path: locate the -`MethodEntry` on the receiver's `TypeDescriptor.method_table` (a -slice sorted by interned method-symbol pointer) and invoke its -function pointer. - -`(.method obj arg)` is analyzed into a call that: - -1. Resolves the method symbol once at compile time. -2. Looks up `obj.typeDescriptor().lookupMethod(method)`. -3. From Phase 7 onward, a per-call-site `CallSite` cache stores the - last `(TypeDescriptor*, method_fn)` pair for a monomorphic fast - path. - -`(ClassName. args)` is rewritten by the reader to `(new ClassName args)`. -`new` is a special form whose analyzer resolves `ClassName` to a -`TypeDescriptor` and invokes its constructor. - -Host-imported types (UUID, Date, File, …) get the same machinery: a -host module under `src/runtime/host/` registers a `TypeDescriptor` -whose `method_table` points at native Zig functions. From the user's -perspective the dispatch is identical. - -`import`, `doto`, and `..` are pure macros in `clojure.core.clj` and -need no special-form support beyond what `new` and `dot` already -provide. - -## Alternatives considered - -### Alternative A — Reflective dispatch (JVM style) - -- **Sketch**: at runtime, look up method by name through a global - registry. -- **Why rejected**: this is the path that produced JVM's - `Reflector.java` and the corresponding performance pain. cw v1 - has no JIT to dig out from under it. - -### Alternative B — Direct vtable indexing per method - -- **Sketch**: assign each method-name a global integer slot; types - store a sparse vtable indexed by slot. -- **Why rejected**: works for closed worlds, not for Clojure's - open-world dispatch (protocols extend after the fact). - -## Consequences - -- **Positive**: a single dispatch path covers cw-native and - host-imported types. `MethodEntry` is small (interned symbol + fn - pointer). `CallSite` cache yields monomorphic dispatch hot loops. -- **Negative**: megamorphic dispatch (10+ types at one site) reverts - to binary search through `method_table`. Acceptable for cw v1's - target workloads; revisit at Phase 8+ if benchmarks flag a - hotspot. -- **Neutral / follow-ups**: ADR-0007 carries the `TypeDescriptor` - shape; ADR-0011 covers how `host` module entries register. - -## Phase 7+ migration note (amendment 1) - -The Phase 4 skeletons (task 4.18 `protocol.zig`, task 4.25 -`method_table.zig`) declare `ProtocolDescriptor` / `MethodEntry` / -`CallSite` structs without dispatch logic. Phase 7 entry -activation includes a rewrite: - -- `dispatch(callsite, receiver, method, args)` implementation with - cache-fill on miss (`last_type` + `last_method` slot in - `CallSite`). -- The Phase 5 path that activated `TypeDescriptor.lookupMethod` - (see ADR-0007 amendment 1) is rewired through `dispatch` so that - `.method` / `(.method obj args)` / `new` / `import` / `doto` / - `..` go through `method_table.lookup` with the `CallSite` cache, - replacing the Phase 5 direct lookup. -- `defprotocol` / `extend-protocol` / `extend-type` analyzer - recognition activates; Tier A behaviour lands. - -The rewrite is expected per ROADMAP §A25; principle.md depth 3 is -the typical depth (cross-file refactor of the `.method` call path). -depth 4 only if the protocol satisfaction model itself supersedes -this ADR. - -## References - -- ROADMAP §9.6 task 4.18 (protocol dispatch table skeleton), - task 4.25 (method_table + CallSite cache skeleton) -- ROADMAP §9.9 (Phase 7 entry — protocol activation) -- ROADMAP §A25 (Existing code is mutable) -- Related ADRs: 0007, 0011 - -## Revision history - -- 2026-05-23: Status: Proposed -> Accepted (initial landing). -- 2026-05-23 (amendment 1): Added "Phase 7+ migration note" section - to narrate the rewrite of the Phase 5 dispatch path when task 4.18 - / 4.25 skeletons activate in Phase 7 (per ROADMAP §A25). - -- 2026-05-26 (amendment 1 — Phase 7.1 implementation): Land - `dispatch(rt, env, cs, receiver, protocol, method, args, loc)` in - `src/runtime/dispatch.zig`. Signature differs from the original - ADR sketch by including `env` + `loc` — `env` is required because - `BuiltinFn` (= the protocol method impl signature) takes `env`, - and `loc` is needed for catalog raise at the dispatch site. - ROADMAP §9.9 row 7.1's shorter signature `(rt, cs, receiver, - protocol, method, args)` is a sketch; this amendment finalises - the full surface. - - Body wraps `CallSite.lookupWithCache` from - `src/runtime/dispatch/method_table.zig:53-72` (which already - caches the `last_type → last_method` slot with protocol+method - name re-validation) + casts `TypeDescriptor.MethodEntry.fn_ptr` - (`?*const anyopaque`) back to `BuiltinFn` + invokes. Raises - new catalog Code `protocol_no_satisfies` (kind = `type_error`, - phase = `eval`) when the receiver is not a `typed_instance` - (per-Tag default descriptor table is a Phase 7+ extension) or - when `lookupWithCache` returns null. - - **Devil's-advocate fork (depth-2, fresh context) verbatim - embedding** — produced 3 alternatives within F-002 / F-007 / - F-009 envelope: - - > **Alt 1 — smallest-diff**: ship dispatch ABI with - > `lookupWithCache` as-is; defer `generation`. The existing - > cache short-circuit re-validates protocol + method names on - > every hit, so the only correctness gap is `extend-type` - > *changing the impl* for a `(td, protocol, method)` triple a - > CallSite already cached. In Phase 7.1, `extend-type` doesn't - > exist yet (it's 7.7's burden). Until 7.7 lands, `generation` - > invalidates nothing — it's a Phase 7.7 prerequisite shipped - > 6 tasks early. Better: ships less surface in 7.1; the - > cache-invalidation contract gets designed alongside its - > consumer. - > - > **Alt 2 — finished-form-clean**: adopt the proposal's - > invalidation semantics but **move the dispatch fn body to - > `TypeDescriptor.dispatchMethod(rt, cs, receiver, protocol, - > method, args)`** rather than a free function in - > `runtime/dispatch.zig`. Co-locate with data; TypeDescriptor - > becomes the dispatch authority. - > - > **Alt 3 — wildcard**: cw v0 textbook adoption — per- - > ProtocolFn cache on the method-Var, not per-CallSite. Trades - > per-Node arena allocation for cache slots living in heap- - > Value Var bodies. Cache survives across same-method-different- - > call-site re-evaluation. Breaks F-002 finished form: per-Var - > caches share state across unrelated call sites — a polymorphic - > call site at one location pessimises the monomorphic cache - > that a tight loop elsewhere depends on. Per-CallSite is the - > C2/V8 inline-cache convergence point; per-Var is one generation - > behind. - > - > **Recommendation**: Alt 1 (ship dispatch ABI; defer - > `generation` to 7.7 when `extend-type` lands). The existing - > cache is already correct for same-type repeat-method calls - > (the only call pattern 7.1 can test). Shipping `generation` - > in 7.1 means designing the invalidation contract without - > seeing its consumer — exactly the smell `principle.md` calls - > Reservation-as-bias. - - **Selected**: Alt 1 verbatim (skip `generation` field in this - amendment; ship with current `CallSite { last_type, last_method }` - shape). Alt 2's co-location (`td.dispatchMethod`) considered - but rejected — `runtime/dispatch.zig` is the natural home for - the dispatch ABI (the file already contains `VTable.callFn` + - threadlocal call-scoped state), and TypeDescriptor stays - focused on data-side helpers (`lookupMethod` etc.). Alt 3 - rejected per the advocate's own F-002 finding (per-Var cache - locality regression). - - Phase 7.7 amendment will add `generation: u32` to CallSite + - `Runtime.protocol_generation: u32` + bump-on-extend-type - semantics. Phase 7.7 also lands `extend-type` analyzer - recognition (cw v1 has no `extend-type` / `extend-protocol` / - `defprotocol` SPECIAL_FORMS today — `analyzer.zig:168` shows - the table doesn't include them). - - 3 new unit tests in `dispatch.zig` cover: cache miss + hit on - typed_instance receiver; missing method → `protocol_no_satisfies`; - non-typed_instance receiver → `protocol_no_satisfies` (raised - as `error.TypeError` per catalog Code mapping). No e2e — - `defprotocol` / `extend-type` analyzer surface lands at 7.3 / - 7.7; 7.1's dispatch ABI is validated via synthetic fn-pointer - injection per the survey's recommended scope. - - ROADMAP §9.9 row 7.1 flips to `[x]`. - -- 2026-05-26 (amendment 2 — Phase 7.2 multimethod shape): Land - `defmulti` / `defmethod` as Clojure-side macros in - `src/lang/clj/clojure/core.clj` that expand to primitive - constructor + `def` calls; multimethod dispatch lives in - `runtime/multimethod.zig::callMultiFn` invoked through a new - `.multi_fn` arm in `vtable.callFn` (Group B slot 1, F-004). - **No new analyzer Node variants. No new VM opcodes. No - VM-DEFER markers.** Both backends already handle the expansion - uniformly through the existing call-node path; the only new - dispatch surface is the runtime `.multi_fn` arm alongside the - existing `.fn_val` / `.builtin_fn` / `.keyword` arms. - - This supersedes the row 7.2 step-0 survey - (`private/notes/phase7-7.2-survey.md`) §5.4 ("file layout") - + §8 ("differential coverage") portions that proposed - `DefMultiNode` / `DefMethodNode` analyzer variants + 3 - VM-DEFER markers on `vm/compiler.zig`. Both are dropped — the - row 7.2 diff_test cases assert real TreeWalk-VM equivalence - (matching values from both backends) rather than the - asymmetric "VM raises feature_not_supported" oracle. ADR-0036 - dual-backend parity contract is satisfied with zero markers, - zero deferral, zero new provisional behaviour. - - **Devil's-advocate fork (depth-2, fresh context) verbatim - embedding** — produced 3 alternatives within F-002 / F-004 / - F-009 + ADR-0036 envelope (full text: - `private/notes/phase7-7.2-devils-advocate.md`): - - > **Alt 1 — Smallest-diff** - > **Shape**: Land `defmulti` / `defmethod` as **no new - > analyzer Node types** — the reader sees them as ordinary - > call forms whose head symbol resolves at analyze time to a - > Clojure-ns primitive (`__defmulti!` / `__defmethod!`). The - > primitives are pure Layer-2 builtins that construct the - > MultiFn extern struct, bind the Var (for defmulti), or - > `assoc` into `method_table` (for defmethod). Multimethod - > CALL — `(my-mm arg)` — dispatches inside the runtime - > `vtable.callFn` via a `.multi_fn` tag arm beside the - > existing `fn_val` / `builtin_fn` / `keyword` arms. - > **Verdict within F-NNN**: clean. JVM Clojure's `defmulti` - > IS a macro expanding to `def + MultiFn-construction-primitive`, - > and `defmethod` IS interop-call to `addMethod` — both are - > surface forms over primitive constructors. The proposal's - > "two new analyzer Node variants" adds a layer JVM Clojure - > does not have; the proposal's shape mirrors how `def` got - > its own Node (DefNode) — but `def` needs an analyzer Node - > because it has special-form binding semantics (introduces a - > new binding visible to subsequent forms). `defmulti` does - > not — it's just `def` + a constructor call; both already - > work. Per F-002 this is the finished-form-clean shape; the - > proposal's two-Node draft is the smallest-diff bias toward - > "what looks like the v0 shape" without checking whether v0's - > analyzer Node was incidental or load-bearing. - > - > **Alt 2 — Finished-form-clean** - > **Shape**: Land VM compile arms in the SAME row (no - > VM-DEFER markers). Introduce two new opcodes at row 7.2 — - > `OP_DEFMULTI` and `OP_DEFMETHOD` — whose VM dispatch arms - > share their body with the TreeWalk path. The multimethod - > call-site emission is already covered by the existing - > call-node opcode; runtime `vtable.callFn` gains a `.multi_fn` - > arm just as Alt 1 does. - > **Verdict within F-NNN**: clean. ADR-0036 compliance with - > zero deferral. The proposal's VM-DEFER justification (`likely - > 7.6 MethodCallNode bytecode shape`) does not hold up — 7.6 - > designs the dispatch shape for `.method` / `(.method obj - > args)` calls (protocol / host) keyed on receiver - > TypeDescriptor, NOT for multimethod calls keyed on - > (dispatch_fn args) → dispatch_val. The cache shapes differ - > (CallSite vs method_cache), the invalidation triggers differ - > (extend-type vs defmethod / hierarchy mutation), the operand - > encodings differ. The proposal carries the - > Reservation-as-bias smell (the deferral is a memo, not a - > contract). - > - > **Alt 3 — Wildcard** - > **Shape**: Skip the MultiFn extern struct entirely. Encode a - > multimethod as a `PersistentArrayMap` with sentinel keys - > wrapped in an Atom; Group B slot 1 stays reserved-but-empty. - > `defmethod` becomes `(swap! the-mm assoc-in [::methods - > dispatch-val] method-fn)`. - > **Verdict within F-NNN**: borderline. Does NOT violate F-004 - > strictly (reservations are memos), but creates a - > reservation-table-vs-impl inconsistency. The "every - > multimethod is an Atom" twist breaks JVM Clojure's - > user-observable shape (`my-mm` IS callable in JVM Clojure; - > under Alt 3 `my-mm` is an Atom whose deref is callable) — - > F-002 finished-form drift. Also 3 map-traversals per dispatch - > vs 3 field-loads; the proposal's extern-struct shape is - > cheaper per-call. - > - > **No hard F-NNN violation surfaced.** Alt 3's borderline - > F-002 drift is recoverable but rejected. - - **Selected**: Alt 1. The Devil's-advocate's framing labels - Alt 1 as "smallest-diff" and Alt 2 as "finished-form-clean", - but the body of Alt 1's verdict argues Alt 1 IS the finished - form because (a) JVM Clojure ships `defmulti` as a `defmacro` - (not a special form) — the analyzer-Node shape diverges from - upstream Clojure surface; (b) cw v1's existing convention - (`defn` / `defmacro` are macros over `def`) matches Alt 1's - shape — adding analyzer Nodes would break the pattern; (c) - the row 7.2 cycle ships zero new provisional behaviour - (zero VM-DEFER markers, zero new opcodes, zero new Node - variants), which is the finished-form-clean shape per F-002. - Alt 2 was the survey-draft's territory but is rejected as - Reservation-as-bias toward v0's incidental analyzer-Node - shape. Alt 3 rejected per advocate's own F-002 finding. - - **VM-DEFER count delta vs survey draft**: -3 sites NOT - introduced. `D-014c-VM-sub` debt row NOT opened. - `feature_deps.yaml#runtime/vm/multimethod` entry NOT opened. - - Files touched at row 7.2 under the Alt 1 shape: NEW - `src/runtime/multimethod.zig` (MultiFn extern struct + - getMethod + callMultiFn + cw-native isaCheck via - TypeDescriptor walk per survey §7 DIVERGENCE); NEW - `src/lang/primitive/multimethod.zig` (`__make_multifn` / - `__add_method` / `__remove_method` / `__methods` / - `__get_method` / `__prefer_method` / `__prefers` + `isa?` / - `derive` / `underive` / `parents` / `ancestors` / - `descendants` / `make-hierarchy` primitives); EXTEND - `src/lang/clj/clojure/core.clj` (`defmulti` / `defmethod` - macros over the primitives); EXTEND `src/runtime/dispatch.zig` - with `.multi_fn` arm in `vtable.callFn`; new `error_catalog` - Codes (`multimethod_no_method`, `multimethod_ambiguous_dispatch`, - `multimethod_cyclic_derive`, `multimethod_prefer_conflict`). - - Cycle terminus condition: ROADMAP §9.9 row 7.2 [x] when - `defmulti` / `defmethod` / `prefer-method` / `derive` / `isa?` / - `make-hierarchy` all green on the diff_test layer with TreeWalk - ≡ VM equivalence (not asymmetry). - -- 2026-05-26 (amendment 3 — Phase 7.3 cycle 6.6 MethodEntry - storage refactor): Retire the `fn_ptr: ?*const anyopaque` raw - cast path on `TypeDescriptor.MethodEntry` in favour of a single - `method_val: Value` field; dispatch consults - `vtable.callFn(rt, env, method_val, args, loc)` uniformly. - Native deftype-inline impls now ride F-004 slot 69 - (`.builtin_fn` immediate) via `Value.initBuiltinFn(&zigFn)`; - user-fn-val impls from `extend-type` drop straight into the - same Value slot. Converges with row 7.2 multimethod's - `vtable.callFn` routing (amendment 2) — one neutral dispatch - boundary for every callable Value shape. - - Amendment 1's "casts `?*const anyopaque` back to `BuiltinFn`" - is hereby superseded. The cast was a Phase 7.1 expedient - before the protocol path had a consumer for the - `.builtin_fn` Value tag; cycle 6.6 is the natural retirement - moment because (a) row 7.3 introduces the - `extend-type`-driven user-fn-val impl path which cannot - meaningfully cast into `BuiltinFn`, and (b) row 7.2 already - established `vtable.callFn` as the canonical dispatcher. - - **Devil's-advocate fork (depth-2, fresh context) verbatim - embedding** — produced 3 alternatives within F-002 / F-004 / - F-009 envelope (full text: - `private/notes/phase7-7.3-cycle6.6-devils-advocate.md`): - - > **Alt 1 — smallest-diff (`fn_ptr` + `method_val` coexist)** - > **Shape**: Keep `fn_ptr: ?*const anyopaque`, add - > `method_val: Value = .nil_val`. Dispatch fans out by which - > slot is populated. `__extend-type!` populates `method_val`; - > deftype-inline keeps `fn_ptr`. - > **Verdict**: Borderline-NG on F-002 — two storage shapes - > for the same logical concept. Smell sensor reads as - > smallest-diff bias. Shippable but reads as deferral of the - > convergence row 7.2 already made. - > - > **Alt 2 — finished-form-clean (`fn_ptr` → `method_val`)** - > **Shape**: Replace `fn_ptr` with `method_val: Value`. All - > non-nil paths dispatch via `vt.callFn`. Native impls wrap - > as `Value.initBuiltinFn(&zigFn)`. The existing - > `dispatch.zig:279` test wraps the mock + installs a vtable - > with the mock callFn (canonical pattern at - > multimethod.zig:662-674). - > **Verdict**: Clean across F-002 / F-004 / F-009. F-004 slot - > 69 (`.builtin_fn` immediate) earns its keep instead of being - > bypassed by raw `anyopaque` cast. Surgery cost bounded: - > 8 test-literal edits + this amendment's narration. - > Convergence with row 7.2 closes the protocol-vs-multimethod - > dispatch divergence. - > - > **Alt 3 — wildcard (trampoline + heap closure)** - > **Shape**: `fn_ptr: BuiltinFn` (typed, non-anyopaque) + - > `closure: ?*const ProtocolMethodClosure`. Trampoline reads - > TLS slot to recover closure, calls `vt.callFn` on the - > closed-over fn-val. - > **Verdict**: NG on F-002 + NG-in-spirit on F-009. TLS - > coupling between dispatch and extend-type is the opposite - > of feature-implementation neutrality; re-entrant dispatch - > under Phase 15 concurrency would foot-gun. Strictly more - > work per dispatch (trampoline ON TOP of the `vt.callFn` - > call). Reservation-as-bias toward preserving `fn_ptr`. - > - > **No hard F-NNN violation surfaced** across any alternative; - > findings are F-002-leaning (finished-form discipline). - > Recommendation: Alt 2. - - **Selected**: Alt 2. The candidate brief was Alt 1; the - Devil's-advocate upgrades to Alt 2 because (a) one storage - shape per logical concept is the finished form, (b) row 7.2 - multimethod already chose `vt.callFn` so cycle 6.6 closes - the divergence that Alt 1 would carry forward, (c) slot 69 - reserved at F-004 finally gets its consumer in the protocol - path. Alt 3 rejected per advocate's own F-002 + F-009 - findings. - - **Affected files**: `runtime/type_descriptor.zig` (MethodEntry - struct + 3 test literals); `runtime/dispatch.zig` (dispatch - fn body + 1 test literal + mock vtable install); `runtime/ - protocol.zig` (2 test literals); `runtime/dispatch/ - method_table.zig` (4 test literals + 1 struct-init literal); - `lang/primitive/protocol.zig` (1 test literal + new - `__extend-type!` primitive). - - Phase 7.3 cycle 6.6 source commit follows this ADR - amendment commit per the depth-2 cycle protocol (ADR commit - first, source commit second; same row, same session). - -- 2026-05-26 (amendment 4 — Phase 7.7 cycle 1 hybrid polymorphic - primitives, R3 fast-path / slow-path ordering): The - polymorphic primitives `count` / `seq` / `conj` / `reduce` - (D-069) take a hybrid shape per ROADMAP §9.9 row 7.7: existing - Zig Tag-switch fast-path arms stay verbatim; the - `.typed_instance` arm consults `MethodEntry` for the - corresponding protocol method *first* and falls back to its - typed-default only when no override is registered; the outer - `else =>` raise is replaced with a route through - `dispatch(rt, env, &cs, receiver, fqcn, method, args, loc)` so - `(extend-type LongTag IPersistentCollection -count …)`-style - native-Tag overrides reach the primitive via the row 7.3 - per-Tag descriptor registry. - - Cycle 1 lands `count` only; cycles 2-4 mirror the shape onto - `seq` / `conj` / `reduce`. Protocol declarations - (`IPersistentCollection -count …`) live in - `src/lang/clj/clojure/core.clj`; each cycle extends the - bootstrap form. `MethodEntry.protocol_name` stores the bare - symbol name (per `allocFqcn` at `lang/primitive/protocol.zig:41-51`), - so the slow-path hardcodes the same bare-name string - (`"IPersistentCollection"`) — collision with user - `(defprotocol IPersistentCollection …)` in another ns is a - known follow-up gap, tracked at D-NNN protocol fqcn - namespacing (filed at Phase 7 boundary). - - **R3 — `.typed_instance` fast/slow ordering choice**: today - `count`'s `.typed_instance` arm at `sequence.zig:88-102` - hard-codes `defrecord → field_count; else → raise`. Without - R3a, a user `(extend-type MyRecord IPersistentCollection - (-count [_] …))` is silently overridden by `field_count` — - the Silent default-shift smell. R3a (method_table first, - field_count fallback) is the finished form per F-002; R3b - (document precedence; user must use deftype) defers the - JVM-parity gap into permanent debt for no finished-form - benefit. - - **Devil's-advocate fork (depth-2, fresh context) verbatim - embedding** — produced 3 alternatives within F-002 / F-003 / - F-005 / F-006 / F-009 envelope: - - > **Alt 1 — smallest-diff ("Defrecord-only fast-path; deftype - > falls to outer slow-path")**. - > Shape: R3c verbatim. The `.typed_instance` arm is - > shortened to `defrecord → field_count; deftype → break to - > outer slow-path`. The per-Tag descriptor registry handles - > deftype (and any future `.typed_instance`-shaped Tag) via - > its declared `IPersistentCollection.-count` entry. No - > method_table consultation lives in the `.typed_instance` - > arm of `count` itself. - > Better than R3a: hot path stays one branch (`kind != .defrecord`) - > → integer load; no `lookupMethod` linear scan on every - > `count` call. Diff is the smallest of the four candidates - > (~6 lines, no new helper). - > Cost: a user who `(extend-type MyRecord IPersistentCollection - > (-count [c] …))` is silently overridden — the JVM-parity - > gap R3 was named for. The Silent default-shift smell that - > R3 was filed against remains exactly where it was. - > F-NNN: F-002 — finished form almost certainly wants user - > `-count` to win on defrecord too (JVM behaviour), so R3c is - > finished-form-incomplete; smallest-diff convenience whose - > follow-up debt lands at Phase 7+. - > - > **Alt 2 — finished-form-clean ("Method-table-first, helper - > extracted to `runtime/dispatch.zig`")**. - > Shape: R3a, with the consultation **lifted out of every - > primitive's `.typed_instance` arm** into a shared helper - > next to the existing `dispatch()` in - > `src/runtime/dispatch.zig`: `dispatchOrNull(rt, env, &cs, - > receiver, protocol, method, args, loc)` returns `?Value` - > (null on miss). The `count` arm reads: `if (dispatchOrNull - > …) |v| break :blk v; else if (.defrecord) field_count; - > else raise(protocol_no_satisfies)`. - > `seq` / `conj` / `assoc` / `get` / `keys` / `vals` use the - > same helper at cycles 2-4 (and the Q6 retro-audit row). - > Better than R3a-inline: shared helper, no per-primitive - > duplication of the linear scan. Phase 8 follow-up (Q6 — - > `empty` / `first` / `nth` / `assoc` / etc.) costs ~3 lines - > per primitive instead of ~10. - > Cost: ~15 LOC for the helper over R3a-inline; defrecord - > `field_count` is now `else if (.defrecord) field_count` - > rather than an unconditional fast-path — slightly more - > cognitive load reading the arm. Hot-path cost is the same - > ~5-10ns/call as R3a-inline. - > F-NNN: F-002 fully satisfied; F-009 satisfied (helper in - > `runtime/dispatch.zig`, namespace-neutral; `lang/primitive/` - > stays a thin caller); F-003 satisfied (Q6 deferred but - > helper shaped to receive). - > - > **Alt 3 — wildcard ("Cached `count_fn: ?*const Fn` slot on - > TypeDescriptor")**. - > Shape: add `count_fn: ?*const MethodEntry = null` directly - > on `TypeDescriptor` (next to `kind` and `method_table`). At - > `extend-type IPersistentCollection (-count …)` time, - > populate `descriptor.count_fn`. The `count` arm reads: - > `if (inst.descriptor.count_fn) |m| call m; else if - > (.defrecord) field_count; else slow_path`. Zero linear scan - > on hot path — pointer-test + indirect call. - > Better than R3a: hot path = one optional-test + one - > indirect call, matching JVM's vtable dispatch (Java - > `coll.count()` is one vmethod load). Cleanest Phase 20 - > reading for the four absolute-hot primitives (count / first - > / next / cons). - > Cost: two-tier method storage (method_table for the general - > case + dedicated slots for hot primitives) diverges from - > "method_table is the SSOT" landed at row 7.3. Introduces a - > sync invariant — every `extend-type` must update BOTH the - > method_table AND the relevant slot — easy to break, hard - > to detect (a slot-miss = silent fallback to default). The - > hot-primitive set is Phase-dependent. - > F-NNN: F-002 — arguably cleaner *finished form* for hot - > dispatch, but only if cw v1 commits to "TypeDescriptor has - > hot-primitive slots" as a permanent design. Row 7.3 / ADR-0040 - > specifically chose method_table-as-SSOT. **Flag**: if the - > cleanest finished form turns out to require this, the right - > move is a fresh ADR amending row 7.3, not inlining under - > R3. F-003 violated by greedy decision-seizure unless the - > ADR explicitly opens hot-slot as Phase 7's owner choice. - > - > **No hard F-NNN violation** across any alternative; Alt 3 - > would require a separate row-7.3 amending ADR before - > adoption (flagged, not adopted in this amendment). - > **Recommendation**: Alt 2 — finished-form-clean. - - **Selected**: Alt 2. The candidate brief (survey §8 R3a) was - R3a inline in each primitive; the Devil's-advocate upgrades - to Alt 2 (helper extracted to `runtime/dispatch.zig`) because - the shape pays back at every cycle 2-4 primitive and at the - Phase 8 Q6 retro-audit row — cycle 1's +15 LOC for the - helper saves ~6 × (10 - 3) = ~42 LOC across the next 6 - primitive arms. F-009 reinforces (impl-neutral helper home). - Alt 1 rejected per advocate's F-002 finding (Silent - default-shift gap persists). Alt 3 rejected as out of scope - (would amend row 7.3 method_table SSOT — separate ADR). - - **Affected files**: `runtime/dispatch.zig` (new - `dispatchOrNull` + `resolveDescriptor` helpers; `dispatch` - body refactored to delegate); `lang/primitive/sequence.zig` - (`IPC_FQCN` constant + `count` `.typed_instance` arm rewired - for R3a + outer `else =>` routed through `dispatch.dispatch`); - `lang/clj/clojure/core.clj` (bootstrap `defprotocol - IPersistentCollection (-count [c])`); `test/e2e/phase7_polymorphic_extend.sh` - (new — 4 cases) + `test/run_all.sh` (register the new e2e). - - **Opportunistic latent-bug fix bundled in cycle 1's source - commit**: bootstrap `(defprotocol IPersistentCollection …)` - surfaced a pre-existing leak — `lang/primitive/protocol.zig - ::makeProtocol` / `::makeProtocolFn` and - `runtime/type_descriptor.zig::makeTypeDescriptorRef` - allocate on `rt.gc.infra` with no cleanup registration, so - every cljw invocation that touches a protocol or - `__native-type` emitted DebugAllocator leak diagnostics at - process exit. The diagnostics were tolerable in user-level - test fixtures (stderr was redirected away per the cycles 1+4+6 - policy documented in `test/e2e/phase7_protocol.sh`'s header) - but became universal once the bootstrap declared - `IPersistentCollection`. Fixed in-line by adding - `freeOwnedProtocol` / `freeOwnedProtocolFn` (in `runtime/protocol.zig`) - + `freeTypeDescriptorRef` (in `runtime/type_descriptor.zig`) - callbacks and `rt.trackHeap(.{ .ptr = ..., .free = ... })` - registrations at the user-facing primitive call sites - (`lang/primitive/protocol.zig::makeProtocol` / - `::makeProtocolFn`, `runtime/type_descriptor.zig::makeTypeDescriptorRef`). - Test fixtures in `runtime/protocol.zig` (lines 215+) keep - their pre-existing `defer rt.gc.infra.destroy(...)` shape - because their fqcn / methods are stack / rodata literals, - not heap dups; the `freeOwned…` callbacks are only invoked - by the user-facing paths that own heap-allocated slices. - - Cycles 2-4 will re-use the helper for `seq` (`-seq`) / - `conj` (`-cons`) / `reduce` (`-reduce`) without amending - this ADR (the helper's contract is closed by cycle 1's diff). - - Phase 7.7 cycle 1 source commit follows this ADR amendment - commit per the depth-2 cycle protocol (ADR commit first, - source commit second; same row, same session). diff --git a/.dev/decisions/0009_object_header_heap_only_lock.md b/.dev/decisions/0009_object_header_heap_only_lock.md deleted file mode 100644 index 84f099868..000000000 --- a/.dev/decisions/0009_object_header_heap_only_lock.md +++ /dev/null @@ -1,140 +0,0 @@ -# 0009 — Object header lock for heap values only - -- **Status**: Accepted -- **Date**: 2026-05-23 -- **Author**: Shota Kudo (drafted with Claude) -- **Tags**: phase-4-entry, object-header, lock, concurrency, monitor - -## Context - -JVM Clojure relies on `(monitor-enter obj)` / `(monitor-exit obj)` / -`(locking obj body)` for cache initialization, lazy init, and other -synchronization. Corpus measurement: `locking` 335 occurrences, -`monitor-enter` / `monitor-exit` 17 each. - -The JVM achieves this by attaching a Mark Word to every Object and -encoding a 3-state lock (biased, lightweight, heavyweight). cw v1's -NaN-boxed `Value` is 8 bytes with no spare bits for a per-value lock -bit; building a Mark Word into every NaN-boxed value is impossible. - -## Decision - -cw v1 supports `locking` / `monitor-enter` / `monitor-exit` only on -**heap-allocated values** (vectors, maps, sets, deftype instances, -records, atoms, refs, agents, ex-info instances). NaN-boxed -primitives (int_53, float_64, keyword, symbol) cannot be locked; -attempting to lock one raises `Code.eval_type_cannot_call`-shape -catalog error (per ADR-0018) whose user-facing message names the -value type that the user tried to lock. - -The lock state lives in a packed 32-bit field on every heap -object's header: - -```zig -pub const ObjectHeader = packed struct(u64) { - type_tag: u16, - gc_and_lock: u32, // [0..1] lock_state (2 bits), [2..31] gc_mark (30 bits) - _reserved: u16, -}; - -pub const LockState = enum(u2) { - unlocked = 0, - light = 1, // CAS-acquired - heavy = 2, // sparse heavy-lock table entry held - _ = 3, // reserved -}; -``` - -The lock has two states, not three (biased was dropped by the JVM in -JEP 374). Contention escalates `light -> heavy` by lazily inserting a -`std.Thread.Mutex` into a sparse heavy-lock table keyed by header -pointer. - -Phase 5 reserves the bit slot in the struct. Phase 15 activates the -implementation (CAS path + heavy fallback). Phase 4-14 attempts to -take a lock raise sub-feature staged catalog Codes -(`locking_not_supported` / `monitor_enter_not_supported` / -`monitor_exit_not_supported`) per ADR-0018 amendment 2. These Codes -are removed when Phase 15 activates the lock implementation. No-op stub is forbidden -per `no_op_stub_forbidden.md`. - -## Alternatives considered - -### Alternative A — Bit-packed Mark Word on every Value - -- **Sketch**: spare bits in the NaN-box for primitives too. -- **Why rejected**: 8-byte NaN-box has no spare bits without breaking - numeric range. - -### Alternative B — External lock map keyed by Value identity - -- **Sketch**: a global `HashMap`. -- **Why rejected**: works but pays a hash + lookup per `locking` - call. Object-header path is direct. - -### Alternative C — Biased locking (JVM pre-JEP 374) - -- **Sketch**: add a third state for single-thread bias. -- **Why rejected**: the JVM removed it because the optimization - benefit no longer outweighed the complexity. cw v1 starts on the - post-JEP-374 design. - -## Consequences - -- **Positive**: 335 corpus occurrences become reachable. The - invariant "lockable iff heap" is simple, predictable, and - documented as an error condition for primitives. -- **Negative**: `(locking some-int 42)` is a runtime error. This - rejects code that locks on primitive values, which the JVM also - permits via auto-boxing; user code must adapt or use atom. -- **Neutral / follow-ups**: Phase 15 STM (ADR-0010) sits beside this - mechanism but does not share it. Phase 15.4 concurrent tests - exercise both. - -## Phase 5 / Phase 15 migration note (amendment 2) - -The Phase 4 entry (task 4.19) reserves the `gc_and_lock: u32` packed -field in `ObjectHeader` with `lock_state: u2` at the low bits and -`gc_mark: u30` at the high bits. Activation is **two-step, across -two Phases**; each step rewrites already-shipped code: - -- **Phase 5 activation (`gc_mark` side)**: mark-sweep GC (ADR-0017 - amendment 1) reads and writes the `gc_mark: u30` bits. Phase 1-4 - heap allocation call sites that currently set only `type_tag` / - `meta` rewrite to also route through `GcHeap.alloc` and to - initialise `gc_mark`. The `ObjectHeader` struct shape is - invariant from Phase 4 onward; what changes are the **call sites - that construct headers**. -- **Phase 15 activation (`lock_state` side)**: `cmpxchgLockBits` - helpers (handover Next Phase Queue → eventually Phase 15.) - land. The Phase 4-14 catalog Codes `locking_not_supported` / - `monitor_enter_not_supported` / `monitor_exit_not_supported` are - **removed** from `error_catalog.zig`; the corresponding `entry()` - switch arms disappear; the catalog test expectations rewrite from - "expect this Code" to "expect successful lock acquisition". This - is the canonical "Codes come and go" pattern per ADR-0018 - amendment 2. - -Both rewrites are expected per ROADMAP §A25; principle.md depth 2-3 -covers each step. depth 4 only if a follow-up ADR replaces the -`gc_and_lock` bit layout itself. - -## References - -- ROADMAP §9.6 task 4.19 (Object header layout extension) -- ROADMAP §9.7 (Phase 5 entry — `gc_mark` activation) -- ROADMAP §9.17 (Phase 15 entry — `lock_state` activation) -- ROADMAP §A25 (Existing code is mutable) -- Related ADRs: 0007, 0010, 0017, 0018 -- JVM JEP 374 (biased locking deprecation) - -## Revision history - -- 2026-05-23: Status: Proposed -> Accepted (initial landing). -- 2026-05-23 (amendment 1): Error phrasing rewritten to go through - the catalog (ADR-0018). User-facing messages no longer reference - this ADR by name. -- 2026-05-23 (amendment 2): Added "Phase 5 / Phase 15 migration - note" section to narrate the two-step rewrite scope when `gc_mark` - (Phase 5) and `lock_state` (Phase 15) sides of `gc_and_lock` - activate (per ROADMAP §A25). diff --git a/.dev/decisions/0010_stm_tier_a_mvcc.md b/.dev/decisions/0010_stm_tier_a_mvcc.md deleted file mode 100644 index 03919fb1f..000000000 --- a/.dev/decisions/0010_stm_tier_a_mvcc.md +++ /dev/null @@ -1,362 +0,0 @@ -# 0010 — STM (ref / dosync) is Tier A with full MVCC - -- **Status**: Accepted -- **Date**: 2026-05-23 -- **Author**: Shota Kudo (drafted with Claude) -- **Tags**: phase-4-entry, stm, mvcc, concurrency, ref, dosync - -## Context - -Clojure's STM is the project's signature concurrency primitive. -`ref` / `dosync` / `alter` / `commute` / `ensure` total ~444 corpus -occurrences once `alter-meta!` / `ensure-reduced` are excluded. Babashka -explicitly omits STM, but cw v1 is positioned as a fuller alternative -to JVM Clojure and the Software Transactional Memory model is part of -what "fuller" means here. - -JVM Clojure's implementation lives in `LockingTransaction.java` -(~1,000 lines): MVCC TVal ring per `Ref`, thread-local transaction -context, retry loop with snapshot validation, ordered locking by ref -pointer to avoid deadlocks, and a "barge" mechanism that lets older -transactions preempt newer ones to prevent starvation. - -## Decision - -cw v1 implements STM as Tier A with a full MVCC design that -observably matches the JVM: - -```zig -pub const Ref = struct { - tvals: TValHistory, - min_history: u32, - max_history: u32, - watches: WatchTable, - lock: std.Thread.Mutex, -}; - -pub const TVal = struct { - val: Value, - point: u64, - msecs: i64, - prior: ?*TVal, -}; - -pub const Transaction = struct { - sets: std.AutoHashMapUnmanaged(*Ref, Value), - commutes: std.AutoHashMapUnmanaged(*Ref, std.ArrayListUnmanaged(*Closure)), - ensures: std.ArrayListUnmanaged(*Ref), - info: ?*Info, - read_point: u64, - start_point: u64, - priority: u32, -}; - -pub threadlocal var current_tx: ?*Transaction = null; -``` - -Phases: - -- Phase 4 entry: declarations live in cw runtime headers; no executable - code paths are wired. `dosync` / `ref` / `alter` / `commute` / - `ensure` / `ref-set` raise sub-feature staged catalog Codes - (`stm_dosync_not_supported` / `stm_ref_not_supported` / - `stm_alter_not_supported` / `stm_commute_not_supported` / - `stm_ensure_not_supported` / `stm_ref_set_not_supported`) - per ADR-0018 amendment 2. Each Code disappears when the - corresponding sub-operation lands in Phase 13-15. -- Phase 13: `Ref` and `TVal` data structures. -- Phase 14: `doGet` / `doSet` / `doCommute` / `doEnsure`. -- Phase 15.1: commit + retry loop. -- Phase 15.2: commute fast path. -- Phase 15.3: barge mechanism (priority-based contention resolution - matching JVM `LockingTransaction.barge`). -- Phase 15.4: concurrent integration test. - -## Alternatives considered - -### Alternative A — Drop STM (Babashka path) - -- **Sketch**: implement only `atom`. -- **Why rejected**: STM is a defining Clojure feature, ~444 corpus - occurrences depend on it, and the cw v1 charter explicitly aims - higher than Babashka. - -### Alternative B — Stub `dosync` as a no-op - -- **Sketch**: execute body without snapshot isolation. -- **Why rejected**: violates the `no_op_stub_forbidden` rule. A - silent-no-op `dosync` is worse than a clear "not implemented yet" - error because it makes the user believe the code worked. - -## Consequences - -- **Positive**: `ref` / `dosync` / `alter` / `commute` / `ensure` work - observably the same as on JVM, including the barge guarantee. -- **Negative**: ~3-5 weeks of focused implementation in Phase 13-15. - Concurrent test surface grows. -- **Neutral / follow-ups**: `commute` fast path can be optimized later; - watches share infrastructure with atom (ADR follow-up at Phase 15). - -## Phase 13-15 migration note (amendment 2) - -The Phase 4 entry lands `Ref` / `TVal` / `Transaction` declarations -(skeleton in `runtime/stm/`) and **six sub-feature staged Codes** -in `error_catalog.zig` (`stm_dosync_not_supported` / -`stm_ref_not_supported` / `stm_alter_not_supported` / -`stm_commute_not_supported` / `stm_ensure_not_supported` / -`stm_ref_set_not_supported`). Activation is staged across five -Phases; each step rewrites already-shipped code by removing one or -more of these Codes and rewriting the corresponding test -expectations: - -| Phase | Sub-op activated | Codes removed | -|-------|----------------------------------------------|------------------------------------------------------------------------------------------| -| 13 | `Ref` / `TVal` data structure | none yet (read-only path lands) | -| 14 | `doGet` / `doSet` / `doCommute` / `doEnsure` | `stm_alter`, `stm_commute`, `stm_ensure`, `stm_ref_set` (partial — read/write activate) | -| 15.1 | commit + retry loop | `stm_dosync` | -| 15.2 | commute fast path | `stm_commute` (full — retry-loop integration) | -| 15.3 | barge mechanism | (no Code removed; correctness rewrite of retry control flow) | -| 15.4 | concurrent integration test | (no Code removed; test surface expansion) | - -Each Phase that removes a Code also **rewrites the catalog test** -(`error_catalog.zig`'s inline tests assert that Code is reachable; -removing the arm requires updating those tests). The catalog -growth is non-monotonic per ADR-0018 amendment 2. - -The rewrite is expected per ROADMAP §A25; principle.md depth 2 for -each Phase that only deletes a Code arm; depth 3 when also -rewriting `Ref` storage / `Transaction` retry control flow. - -## Phase 13 representation — single-cell, lock-free (amendment 3) - -Phase 13 entry (2026-05-28) lands the `Ref` data structure as a -**single `current: Value` heap cell**, not the TVal history ring: - -```zig -pub const Ref = extern struct { - header: HeapHeader, // offset 0 — GC contract - _pad: [6]u8 = .{0,0,0,0,0,0}, - current: Value, // newest committed value (= JVM tvals.val) -}; -``` - -`(ref init)` seeds `current = init`; `deref` / `@r` **outside a -transaction** returns `current` (JVM `Ref.deref` collapses to -`currentVal()` reading the newest TVal when no transaction runs). -`dosync` / `alter` / `commute` / `ensure` / `ref-set` keep raising -their staged Codes (no Code removed at Phase 13, per amendment 2). -File: `src/runtime/stm/ref.zig`; wiring modelled on -`runtime/collection/reduced.zig` (single-Value cell + one-pointer -GC trace). - -Three divergences from this ADR's §Decision sketch, each recorded -here so the drift is not silent: - -1. **No lock in Phase 13.** The §Decision sketch writes - `lock: std.Thread.Mutex`; that type was **removed in Zig 0.16** - and is forbidden by ROADMAP §13. The Phase-13 read path is - single-threaded (no commit can race a read before Phase 15), - so it needs no lock — the D-046 lazy_seq "single-thread no-lock - until Phase 15" precedent governs. Omitting the lock also keeps - `Ref` a pure `extern struct` and avoids cw v0's forced - `RefObj`/`RefInner` two-allocation split (v0 split *only* - because a mutex cannot live in an `extern struct`). The lock - returns at Phase 15.1 (commit + retry) as `std.Io.Mutex` / - `std.atomic.Mutex`, decided by that owner. -2. **No `TVal` type in Phase 13.** The §Phases line reads "`Ref` - and `TVal` data structures", but that pairing is prose in a - memo, not a contract (F-002 §4); the read-only path provably - needs only the newest value, which the scalar `current` holds. - Materialising an inert `TVal` node now (with `point=0` / - `prior=null` doing nothing until commits exist) is a larger - skeleton than the bounded field-swap it would save — it fails - F-002 §3's shrink-not-enlarge test. Phase 14 introduces `TVal` - { val / point / msecs / prior } + the history ring; `current` - becomes the ring head then (a bounded, documented field-swap, - tracked by **D-102**). -3. **No `min_history` / `max_history` / `watches` fields.** Unused - until Phase 14 (ring growth) / Phase 15 (watches share atom - infra); declaring them now is Reservation-as-bias. Added by the - owner that first reads them. - -### Alternatives considered (Devil's-advocate fork, fresh context, 2026-05-28) - -F-NNN envelope check (leading entry): none of the three shapes -violates F-002 / F-004 / F-006. The one textbook field that would — -the embedded `std.Thread.Mutex` — is forbidden by ROADMAP §13 / -zig_tips, a stale-memo problem, not an F-NNN conflict; no halt. - -- **Alt 1 — smallest-diff: `Ref { header, _pad, current: Value }`.** - Near-clone of `reduced.zig`; one inner Value, one-pointer trace. - Better: smallest surface, mechanically proven, zero reservation - bias, fastest to green. Breaks: Phase 14 *replaces* the scalar - `current` with a `?*TVal` ring head (field-type swap on a shipped - struct + `deref`-arm rewrite) — the "skeleton that gets rewritten, - not extended" shape F-002 §3 warns about, though bounded. -- **Alt 2 — finished-form-clean: `Ref { header, tvals: ?*TVal }` + - `TVal { val, point, msecs, prior }`.** Ref holds a `?*TVal` head - seeded with a single self-terminating node; `deref` = `tvals.?.val`. - Better: Phase 14 *extends* (splice a TVal onto `prior`) rather than - replaces; mirrors JVM `Ref.deref → currentVal → tvals.val` 1:1; no - struct-layout churn. Breaks: **two heap allocations + a second - registered trace fn / GcHooks** for a path that only reads one - value; `point` / `msecs` written but never read until Phase 14; the - node is a Reservation-as-bias materialisation of ADR prose. -- **Alt 3 — wildcard: Alt 1's single `current` cell, but in - `runtime/stm/ref.zig` with a module docstring stating the Phase-14 - ring-rewrite contract + a `D-NNN` row owning the rewrite.** Better: - keeps Alt 1's minimal footprint *and* removes its only real risk (a - silent scaffold that loses its homing path); the Phase-14 owner - inherits an explicit tracked contract instead of a surprise - field-swap; honours F-003 (imagine-record-**defer**). Breaks: still - a field-type swap in Phase 14 (docstring makes it expected, not - eliminated); adds a debt row + docstring upkeep. -- Findings: **(a)** lock does not belong in Phase 13 (Zig 0.16 removal - + single-thread + avoids v0's split); **(b)** `min/max_history` / - `watches` should NOT be declared now (Reservation-as-bias); **(c)** - a `TVal` *ring* is premature, a single node defensible, but - materialising it because the ADR pairs the two nouns is the trap - F-002 §4 names. -- DA recommendation (non-binding): **Alt 3** — "the finished form is - cleaner *reached from* a deferred-and-documented scalar than - *anticipated by* a half-built ring." Main loop adopted Alt 3. - -## Phase 14 ring activation — doubly-linked self-loop + atomic.Mutex (amendment 4) - -Phase 14 row 14.11.5 (D-122 row-assignment + D-102 implementation, -2026-05-28) lands the TVal history ring per the JVM `Ref.java` -shape, refined by a Devil's-advocate fork. The chosen shape: - -```zig -pub const TVal = extern struct { - header: HeapHeader, // GC contract; tag = .tval (Group D slot 63) - _pad: [6]u8, - val: Value, - point: i64, - msecs: i64, - prior: *TVal, // initial = self (ring of 1) - next: *TVal, // initial = self (ring of 1) -}; - -pub const Ref = extern struct { - header: HeapHeader, // GC contract; tag = .ref - lock: std.atomic.Mutex, // 1 byte; tryLock + unlock only - _pad: [5]u8, - tvals: *TVal, // ring head (always non-null after alloc) - min_history: u32 = 0, - max_history: u32 = 10, -}; -``` - -Decisions recorded: - -1. **Doubly-linked self-loop ring (JVM-faithful)**. The initial - TVal sets `prior = self, next = self`; subsequent commits splice - between `head` and `head.next` exactly per `Ref.java:64-69`. - `histCount` walks `tvals.next ... tvals`; `trimHistory` resets - `next/prior` to self. The reader can hold `Ref.java` open - side-by-side; field names + splice semantics line up 1:1. - -2. **`std.atomic.Mutex` field at Ref level, not placeholder**. - `std.atomic.Mutex` is `enum(u8)` (verified in stdlib at this - ADR's land time), extern-compatible, lock-free `tryLock` + - `unlock`. Lands at D-102 so Phase 15.1 transaction control - flow extends Ref's API surface without re-laying the struct. - **Spin-vs-block F-009 wrinkle** (recorded explicitly per the - DA fork): `std.atomic.Mutex` has no blocking `lock()`. Phase - 15.1's `commit` path must spin (`while (!ref.lock.tryLock()) - { suspend / yield / contention-strategy }`) where JVM blocks. - This is a real semantic divergence from JVM and is acceptable - for a Wasm-target single-threaded-first runtime; if Phase 15.4 - bench shows pathological spin, a Phase 15.5 ADR can upgrade - to `std.Io.Mutex` at the LockingTransaction layer above Ref - (not at Ref itself, since `std.Io.Mutex` is not extern- - compatible and would force a `RefObj`/`RefInner` split à la - v0). F-009 neutrality preserved: Java + cljw + Clojure-peer - surfaces all share the same Zig impl + the same spin semantics. - -3. **TVal is internal-only, but consumes Group D slot 63 - (`reserved_d15` → `tval`)**. The DA fork's decision-point (ii) - ranked "TVal-not-NaN-boxed-as-Value" as the consensus - finished-form (= TVal does NOT show up in `tag()` checks, - `decodePtr` paths, or any user-visible Value surface). It IS - tagged in the `HeapTag` enum because GC heap allocation + - per-tag `traceGc` dispatch requires it. The last open Group D - slot is named, closing the F-002 / D-043 "name the reserve or - shrink the layout" Phase 7+ predicate for slot 63. - -4. **`min_history = 0 / max_history = 10` declared at D-102**. - The ring-growth machinery (Phase 15.1's `commit` reading both) - is the first reader; declaring them now matches "Added by the - owner that first reads them" (amendment 3 § Negative finding - #3) since D-102 IS the first cycle whose code paths reference - them (the ring init logic). Not Reservation-as-bias. - -5. **GC trace walks the ring via existing mark-bitmap cycle - detection**. Each TVal traces its `val` + `prior` + `next`; - the mark-and-recurse in `mark_sweep.zig` checks "already - marked? return", terminating the self-loop in one pass. Ref's - trace marks `tvals` + lets the recursion handle the rest. A - D-102 unit test asserts that a freshly-allocated 1-node ring + - a manually-constructed 3-node ring both terminate marking - (count = N, not infinite). - -The decision rejected three alternatives — Alt 1 (smallest-diff: -singly-linked + lock absent), Alt 2-as-modified (this amendment; -chosen), Alt 3 (wildcard: array-ring inline in Ref). The full DA -output landed at `private/notes/phase14-d102-da-fork.md` (verbatim); -the relevant rejection rationale per F-NNN: - -- **Alt 1 (singly-linked + lock absent)** rejected per F-002 §3 - shrink-not-enlarge: Phase 15.1's `doGet` walks `prior` whether - singly- or doubly-linked, but `histCount` / `trimHistory` / - ring-splice all need `next`. Omitting `next` here means Phase - 15.1 re-touches TVal shape — the exact pattern F-002 forbids. -- **Alt 3 (array-ring)** rejected per F-009 + bench risk: the - hard `max_history ≤ 16` cap and 384-byte fixed Ref cost regress - the `(ref 42)` for-counter use case ~8×, and `clojure.core/ref-max-history` - would silently cap or raise — a Java/cljw/Clojure-peer - neutrality break. A follow-up `private/notes/d102-arena-ring-exploration.md` - records the bench-validatable hypothesis for Phase 16 review. - -Discharges D-122 (row scheduling) + D-102 (implementation). Opens -no new STM debt rows. Phase 15.1's D-114 (transaction control -flow) inherits the unchanged Ref/TVal shape; that cycle's -`LockingTransaction` lands as pure code-add per the DA fork's -"Phase 15.1 saves ~50 LOC" projection. - -## References - -- ROADMAP §9.6 task 4.7 (try/throw/loop/recur — STM error message - path), §9.15 (Phase 13 entry), §9.16 (Phase 14 entry, esp. row - 14.11.5 — D-102 ring rewrite), §9.17 (Phase 15.1-15.4) -- ROADMAP §A25 (Existing code is mutable) -- Related ADRs: 0009, 0017, 0018 -- JVM source: `clojure.lang.LockingTransaction` - -## Revision history - -- 2026-05-23: Status: Proposed -> Accepted (initial landing). -- 2026-05-23 (amendment 1): Phase 4 unsupported-attempt phrasing - now references per-sub-operation staged catalog Codes (per - ADR-0018 amendment 2 sub-feature staged pattern). User messages - name only the form (`dosync`, `ref`, ...), not this ADR. -- 2026-05-23 (amendment 2): Added "Phase 13-15 migration note" - section to narrate the staged catalog Code removal and test - expectation rewrite across Phase 13 / 14 / 15.1-15.4 (per - ROADMAP §A25 and ADR-0018 amendment 2 "Codes come and go"). -- 2026-05-28 (amendment 3): Phase 13 lands `Ref` as a single - lock-free `current: Value` heap cell (Devil's-advocate Alt 3); - `std.Thread.Mutex` sketch retired (Zig 0.16 removal), lock - deferred to Phase 15.1 as `std.Io.Mutex`; `TVal` ring + - `min/max_history` / `watches` deferred to Phase 14 (tracked by - D-102). File `src/runtime/stm/ref.zig`. -- 2026-05-28 (amendment 4): Phase 14 row 14.11.5 lands the TVal - history ring (D-122 + D-102) per JVM `Ref.java` doubly-linked - self-loop shape. `std.atomic.Mutex` (enum(u8), extern-compatible) - lands as a real Ref field — Phase 15.1 commit/retry spins on - `tryLock` since `std.atomic.Mutex` lacks blocking `lock()`. TVal - consumes Group D slot 63 (`reserved_d15` → `tval`), naming the - last anonymous reserve. Files `src/runtime/stm/tval.zig` (new), - `src/runtime/stm/ref.zig` (rewritten), `src/runtime/value/heap_tag.zig` - (slot 63 named). diff --git a/.dev/decisions/0011_host_extension_mechanism.md b/.dev/decisions/0011_host_extension_mechanism.md deleted file mode 100644 index 43371c601..000000000 --- a/.dev/decisions/0011_host_extension_mechanism.md +++ /dev/null @@ -1,97 +0,0 @@ -# 0011 — Host extension mechanism via `src/runtime/host/` directory layout - -- **Status**: Superseded by ADR-0029 (2026-05-24) -- **Date**: 2026-05-23 -- **Author**: Shota Kudo (drafted with Claude) -- **Tags**: phase-4-entry, host, extension, java-stdlib, directory-layout, superseded - -## Context - -The corpus references 578 distinct `java.*` classes; the top 40 alone -cover most production Clojure I/O, time, networking, and data -formatting. cw v1 cannot pretend these do not exist — even Babashka -provides cw-native equivalents for `UUID`, `Date`, `Pattern`, `Path`, -`File`, etc. - -cw v0 hard-coded seven Java equivalents in `src/lang/interop/class_registry.zig` -(~221 lines). Adding new ones required editing that file. cw v1 needs -a layout that admits incremental additions without central -modification. - -## Decision - -Host-stdlib equivalents live under `src/runtime/host/` mirroring the -Java package structure: - -``` -src/runtime/host/ -├── lang/ Object, String, Long, Math, System, Thread -├── io/ File, PrintWriter, ByteArrayInputStream, ... -├── util/ UUID, Date, Random, Locale, regex/Pattern -├── time/ Instant, LocalDate, LocalDateTime, Duration -├── net/ URL, URI, Socket (Phase 14+) -├── nio/ file/Path, file/Files, charset/Charset -├── math/ BigInteger, BigDecimal -├── security/ MessageDigest, SecureRandom (Phase 14+) -├── sql/ Connection, Statement, ResultSet (Phase 14+) -├── text/ SimpleDateFormat, DecimalFormat (Tier B) -├── reflect/ Method, Field (thin, via TypeDescriptor) -└── concurrent/ atomic.*, locks.* (Phase 15) -``` - -Each `.zig` file under `src/runtime/host/` exports a -`___HOST_EXTENSION` declaration (a `host.Extension` struct value). -A single `_host_api.zig` aggregates all entries via comptime -introspection, so a new host class plugs in with just a new file -under the right subdirectory. - -User code reaches each entry through a mirrored cw namespace: -`(:require [cljw.host.java.util :refer [UUID]])`. - -## Alternatives considered - -### Alternative A — cw v0 `class_registry.zig` pattern - -- **Sketch**: a central registry file enumerating every host class. -- **Why rejected**: every addition forces a central file edit. The - layout signal (Java package mirror) is lost. - -### Alternative B — Externally configurable host registry - -- **Sketch**: load host classes from a YAML at runtime. -- **Why rejected**: Zig favors compile-time wiring; comptime - introspection of the directory delivers the same plug-and-play - effect without runtime configuration. - -## Consequences - -- **Positive**: a `java.util.OptionalInt` becomes a single new file - under `src/runtime/host/util/optional_int.zig`. The directory - structure itself documents the compatibility surface. -- **Negative**: discoverability depends on the discipline of placing - the file in the right Java-package subdirectory. The - `host_extension_layout.md` rule enforces this. -- **Neutral / follow-ups**: `compat_tiers.yaml` `host_classes` - section enumerates every entry by FQN and target Phase. Tier - classification per entry per ADR-0013. - -## References - -- ROADMAP §9.6 task 4.20 (host directory + _host_api.zig) -- ROADMAP §6.5 (Tier data source) -- Related ADRs: 0013, 0015 -- Rule: `.claude/rules/host_extension_layout.md` - -## Revision history - -- 2026-05-23: Status: Proposed -> Accepted (initial landing). -- 2026-05-24: Status: Accepted -> Superseded by ADR-0029. The - `___HOST_EXTENSION` marker pattern (distributed registration - replacing a central registry) carries forward, but the - registration root moves from `runtime/host/_host_api.zig` to - `runtime/java/_host_api.zig`, and the reserved 13 - `runtime/host//_placeholder.zig` files are removed. Java - surfaces now live at `runtime/java//.zig`; cljw-native - surfaces at `runtime/cljw//.zig`. Rationale: see - ADR-0029 Context. Triggered by user-directed structural session - (2026-05-24). diff --git a/.dev/decisions/0012_nan_box_valuetag_day1.md b/.dev/decisions/0012_nan_box_valuetag_day1.md deleted file mode 100644 index 4c302938e..000000000 --- a/.dev/decisions/0012_nan_box_valuetag_day1.md +++ /dev/null @@ -1,94 +0,0 @@ -# 0012 — NaN-box ValueTag day-1 full enum reservation - -- **Status**: Accepted -- **Date**: 2026-05-23 -- **Author**: Shota Kudo (drafted with Claude) -- **Tags**: phase-4-entry, value-tag, nan-box, day-1, dispatch - -## Context - -The NaN-box layout is the single most load-bearing data structure in -cw v1. Every value dispatch — including special forms, primitives, -host instances, and lazy seqs — keys off `ValueTag`. Adding a tag -later forces every existing dispatch site to be revisited (the same -pressure described in ADR-0004 for `SpecialFormTag` and `Opcode`). - -Two sub-options exist for how class-system values are encoded: - -- **Option A (3 tags)**: separate `typed_instance`, `reified_instance`, - and `type_descriptor` tags. Dispatch is one tag-check per case. -- **Option B (1 tag + flag)**: a single `class` tag with the heap - header's `is_type_descriptor` bit selecting between "describes a - type" and "instance of a type". Dispatch is one tag-check plus one - bit read. - -JVM_TO_ZIG.md §19 recommends Option A on the basis that the dispatch -fast path is shorter (no second read). - -## Decision - -cw v1 adopts Option A. The day-1 `ValueTag` enum reserves a slot per -distinct payload shape: - -- Existing (Phase 1-3): `nil_`, `boolean_true`, `boolean_false`, - `int_53`, `float_64`, `keyword`, `symbol`, `string`, `list`, - `ex_info`, `closure`, `var_`. -- Phase 5 additions: `vector`, `hashmap`, `hashset`, `big_int`, `ratio`, - `lazy_seq`. -- Phase 5-7 (class system, ADR-0007): `typed_instance`, - `reified_instance`, `type_descriptor`. -- Phase 7 (transient, multi-fn, protocol): `multi_fn`, - `protocol_inst`, `transient_vector`, `transient_hashmap`. -- Phase 14-15 (concurrency primitives): `atom_`, `ref_`, `agent_`, - `future_`, `promise_`. -- Host stdlib (ADR-0011): `host_instance` (single tag covering all - host-imported types; the specific type is on the - `TypeDescriptor`). -- Sentinel: `_` (open enum for ADR-driven extension). - -Total: 26 explicit tags plus the sentinel. Layout is finalized at -Phase 4 entry; subsequent additions are amendments to this ADR. - -## Alternatives considered - -### Alternative B — 1 tag + flag - -- **Sketch**: combine `typed_instance` and `type_descriptor` into - one tag, distinguished by a header flag. -- **Why rejected**: marginal byte savings (one tag slot out of 256) - against a measurable dispatch slowdown (one extra read per - class-system call). Slot budget is not the bottleneck. - -## Consequences - -- **Positive**: dispatch is one tag check throughout. Empty slots - through Phase 4-6 are flagged by the exhaustiveness checker so - unimplemented cases surface as compile errors, not silent passes. -- **Negative**: 26 named tags is more surface than strictly required. - Acceptable for the explicitness gain. -- **Neutral / follow-ups**: tag count is reconfirmed when ADR-0007 - finalizes the `TypeDescriptor` layout, since the host-stdlib design - may collapse `host_instance` into `typed_instance` after Phase 14 - if no semantic difference remains. - -## References - -- ROADMAP §A11 (Day-one enum reservation) -- Related ADRs: 0004, 0007, 0011, 0017 - -## Revision history - -- 2026-05-23: Status: Proposed -> Accepted (initial landing). Option A - (3-slot) selected; Option B (1-slot + flag) recorded as alternative. -- 2026-05-23 (amendment 1): **`big_int` lands at HeapTag slot 29 and - `ratio` is reserved at slot 30**, replacing the released - `wasm_module` / `wasm_fn` slots (ADR-0006 amendment 1). The - existing NaN-box layout (4 groups × 8 sub-types = 32 slots, group - bands `0xFFF8`-`0xFFFB`) is preserved — extending to a fifth group - is blocked by `0xFFFC` already serving as `NB_INT_TAG`, and - shrinking sub-type bits would halve the pointer space. Re-purposing - the released Wasm slots is the smallest-diff path. Phase 5 numeric - tower work (long → BigInt promotion, ratio arithmetic) consumes - these slots; the BigInt heap struct landed at task 4.23 swaps - away from `PHASE4_PLACEHOLDER_TAG = 0xFF` to `HeapTag.big_int` in - the same commit as this amendment. diff --git a/.dev/decisions/0013_tier_d_permanent.md b/.dev/decisions/0013_tier_d_permanent.md deleted file mode 100644 index 79a5e3a31..000000000 --- a/.dev/decisions/0013_tier_d_permanent.md +++ /dev/null @@ -1,98 +0,0 @@ -# 0013 — Tier D forms and packages are permanently excluded - -- **Status**: Accepted -- **Date**: 2026-05-23 -- **Author**: Shota Kudo (drafted with Claude) -- **Tags**: phase-4-entry, tier-d, compat, scope, java-interop - -## Context - -cw v0 carried an implicit "55 vars excluded" line that drifted across -the codebase and `DIFFERENCES.md`. Without a tier ADR, every new -arrival had to re-derive the rationale, and the ad-hoc list grew -ambiguous boundaries. The corpus audit clarified that the genuinely -non-portable set is small. - -## Decision - -cw v1 declares the following as Tier D — permanently excluded from -the supported surface. Adding any of these later requires a MAJOR -release (per ROADMAP §1.4 SemVer rule) and an amendment to this ADR. - -### Excluded special forms (5) - -- `gen-class` — requires JVM Class system and bytecode emission. -- `gen-interface` — same. -- `clojure.core/compile` — JVM .class file emission, out of cw scope. -- `proxy` when targeting deep Java extension (Apache HC base classes, - Swing GUI base classes, `java.util.logging.Formatter / Handler` - bases). Approximately 35-50 of 392 corpus `proxy` occurrences fall - here. The remaining ~340 uses are reached through `reify` against - cw-native protocols. -- `bean`'s reflection-deep variant. The basic field walk for a - cw-native type stays Tier C and goes through `TypeDescriptor`. - -### Excluded Java packages (~480 classes) - -- `java.awt.*` — GUI, out of cw scope. -- `javax.swing.*` — GUI, out of cw scope. -- `java.applet.*` — deprecated, out of cw scope. -- Deep reflection internals (parts of `java.lang.reflect.*` beyond - what `TypeDescriptor` exposes). - -### Error message contract - -Tier D entries raise per-form catalog Codes (per ADR-0018 amendment -2): `tier_d_gen_class` / `tier_d_gen_interface` / `tier_d_compile` / -`tier_d_proxy_deep` / `tier_d_bean_deep`. Each Code has a -hand-written multi-sentence template that explains the technical -reason and suggests the cw-native alternative; the form name appears -in the template itself, no `{name}` slot. - -The tier classification ("D") and the rationale ADR ("ADR-0013") -are development artifacts; they do not appear in the user-facing -text. `scripts/check_tier_d_error_msg.sh` (informational at Phase 4 -entry, gate at Phase 5+) checks that every Tier D form named in -`compat_tiers.yaml` has a matching per-form Code in the catalog -rather than a hand-written `setErrorFmt` call. - -## Alternatives considered - -### Alternative A — Defer the Tier D decision indefinitely - -- **Sketch**: leave it to per-feature ADRs. -- **Why rejected**: the ad-hoc accumulation is what bit cw v0. A - central list is the cheaper option. - -### Alternative B — Tier D for a larger set (cw v0 "55 vars") - -- **Sketch**: also exclude all class-system forms, STM, locking, - threading. -- **Why rejected**: corpus measurement shows these have thousands of - uses; excluding them removes cw from the Clojure ecosystem. - -## Consequences - -- **Positive**: a small, defensible Tier D list — five forms plus the - GUI / applet packages — leaves cw v1 compatible with the vast - majority of pure-Clojure code that exists today. -- **Negative**: programs that genuinely need bytecode emission or - Swing UI will not run, no matter how clever the cw runtime gets. -- **Neutral / follow-ups**: `compat_tiers.yaml` carries the - enumerated lists. `tier_classification.md` rule covers the - error-message format. - -## References - -- ROADMAP §3.2 (Out-of-scope), §6 (Tier system data-driven) -- `compat_tiers.yaml` (authoritative list) -- Related ADRs: 0007, 0011 - -## Revision history - -- 2026-05-23: Status: Proposed -> Accepted (initial landing). -- 2026-05-23 (amendment): Error message contract rewritten. The - `"Tier D: , see ADR-0013"` template is replaced by catalog - per-form catalog Codes (`tier_d_gen_class` etc., ADR-0018 - amendment 2). Each Code's user-facing output is a hand-written - multi-sentence message specific to that form. diff --git a/.dev/decisions/0014_utf8_internal_string.md b/.dev/decisions/0014_utf8_internal_string.md deleted file mode 100644 index 5f25584a9..000000000 --- a/.dev/decisions/0014_utf8_internal_string.md +++ /dev/null @@ -1,80 +0,0 @@ -# 0014 — UTF-8 as the internal string representation - -- **Status**: Accepted -- **Date**: 2026-05-23 -- **Author**: Shota Kudo (drafted with Claude) -- **Tags**: phase-4-entry, string, utf-8, encoding, char - -## Context - -JVM Clojure inherits Java's UTF-16 internal string representation. -This forced JVM Clojure into a `char` data type that is a 16-bit -code unit, sometimes splitting a Unicode code point across two -chars (the surrogate pair issue). Many user APIs (`(.length s)`, -`(subs s i j)`) operate in those 16-bit units. - -cw v1 has no compelling reason to inherit UTF-16: the runtime is -native, the I/O layer (ADR-0015) is UTF-8 by default, Zig's -`std.fs` / `std.io` deliver bytes, and Babashka — the existing -JVM-free Clojure precedent — likewise uses UTF-8 inside. - -## Decision - -cw v1 stores strings internally as UTF-8. String length, indexing, -and iteration operate on either bytes or code points depending on -the API; the difference is documented per function in the host -stdlib (ADR-0011). - -Mapping to the user-facing Clojure API: - -- `(count s)` returns the number of **code points** (Clojure-compatible - observable behavior). -- `(.length s)` mirrors `(count s)`. On the JVM `.length` reports - 16-bit code units; on cw v1 it reports code points. This is a - documented deviation in `DIFFERENCES.md` (Tier A with a footnote) - rather than a separate Tier C: well-behaved Unicode-aware code is - unaffected; only code that relies on surrogate-pair quirks - diverges. -- `(.charAt s i)` returns the i-th **code point** as a single - `cljw.host.java.lang.Character`. Surrogate pairs are not exposed. -- `(.getBytes s)` is the byte view (UTF-8); `(.toCharArray s)` is - the code-point view. - -Conversion at the boundary uses `std.unicode` (Zig stdlib). - -## Alternatives considered - -### Alternative A — UTF-16 internal (JVM compatibility) - -- **Sketch**: store strings as `[]u16`. -- **Why rejected**: surrogate pairs add complexity throughout. Zig's - I/O layer is UTF-8 native, so every read / write would pay a - conversion. - -### Alternative B — UTF-32 (code-point arrays) - -- **Sketch**: store strings as `[]u32`. -- **Why rejected**: 4x memory overhead for ASCII-dominated text. - Tradeoff against rare random-access does not favor it. - -## Consequences - -- **Positive**: matches Zig stdlib and the I/O layer (ADR-0015). No - surrogate-pair pitfalls in cw native code. ASCII-heavy strings - carry no overhead. -- **Negative**: JVM Clojure code that uses `.length` as a 16-bit - code-unit count produces a different value on cw. This is a - documented Tier A footnote, not a Tier C divergence. -- **Neutral / follow-ups**: `cljw.host.java.lang.Character` is the - cw representation of a single code point. Phase 5+ string - primitives operate on either bytes or code points; documentation - per function. - -## References - -- ROADMAP §6 (Tier system) -- Related ADRs: 0011, 0015 - -## Revision history - -- 2026-05-23: Status: Proposed -> Accepted (initial landing). diff --git a/.dev/decisions/0015_io_interface_zone0_vtable.md b/.dev/decisions/0015_io_interface_zone0_vtable.md deleted file mode 100644 index dd24f8328..000000000 --- a/.dev/decisions/0015_io_interface_zone0_vtable.md +++ /dev/null @@ -1,192 +0,0 @@ -# 0015 — `io_interface.zig` as a Zone 0 vtable abstraction - -- **Status**: Accepted -- **Date**: 2026-05-23 -- **Author**: Shota Kudo (drafted with Claude) -- **Tags**: phase-4-entry, io, zone-0, vtable, std-io-drift - -## Context - -cw v0 disabled five features (F140-F144: HTTP server, HTTP client, -nREPL, line editor, `cljw build` self-bundle) when the Zig 0.15→0.16 -migration reshaped `std.Io`. Recovery is open-ended because every -recovered feature must follow the moving `std.Io` shape. The same -class of regression will recur with Zig 0.17 / 0.18 unless the -runtime insulates itself. - -## Decision - -cw v1 introduces `src/runtime/io_interface.zig` at Zone 0 (the -lowest zone — no upward imports). It exposes a vtable-style abstraction -for `Reader`, `Writer`, `Net`, and `Process` so that the rest of the -runtime depends on the abstraction, not on `std.Io` directly. - -```zig -// src/runtime/io_interface.zig -pub const Writer = struct { - vtable: *const VTable, - ctx: *anyopaque, - pub const VTable = struct { - write_all: *const fn (*anyopaque, []const u8) anyerror!void, - flush: *const fn (*anyopaque) anyerror!void, - close: *const fn (*anyopaque) anyerror!void, - }; -}; - -pub const Reader = struct { ... }; -pub const Net = struct { ... }; -pub const Process = struct { ... }; -``` - -A concrete `runtime/io_default.zig` (Zone 1) wires the abstraction -to whatever `std.Io` ships in the current Zig version. When `std.Io` -changes, only `io_default.zig` needs revision. - -`main()` constructs the default and injects it into the runtime. - -### Two-tier strategy (per amendment 1) - -The vtable is split into **two tiers** so that consumer code never -imports `std.Io` directly: - -- **Tier 1** (`runtime/io_interface.zig`, Zone 0): the vtable - declarations only. No `std.Io` import. Public API is `Reader` / - `Writer` / `Net` / `Process` structs that carry a `*const VTable` - + `*anyopaque` context. Consumer code (analyzers, REPL, - primitives) imports this file. -- **Tier 2** (`runtime/io_default.zig`, Zone 1): the actual `std.Io` - attachment. Constructs concrete vtable instances from `std.Io.File`, - `std.Io.Dir`, `std.Io.Writer` etc., and exposes - `defaultReader()` / `defaultWriter()` etc. so `main()` (Zone 3) - injects them. - -When Zig stdlib reshapes `std.Io` (the F140-F144 failure mode in -cw v0), only Tier 2 changes. Tier 1 is invariant under stdlib -churn. Consumer code never recompiles for the API surface; it only -recompiles if its own logic changed. - -### File layout - -``` -src/runtime/ - io_interface.zig ← Tier 1, Zone 0, no `std.Io` import - io_default.zig ← Tier 2, Zone 1, the only file that - imports `std.Io` directly for I/O ops - (other zones get I/O via injection) -``` - -Each `Tier 1` struct has a doc-comment declaring its semantic -contract (what the caller expects: blocking semantics, ownership, -error set). The `Tier 2` implementation must honour that contract; -re-implementing for a future Zig version is a Tier-2-only change. - -## Alternatives considered - -### Alternative A — Use `std.Io` directly throughout - -- **Sketch**: the cw v0 path. -- **Why rejected**: F140-F144 cautionary precedent. - -### Alternative B — Per-feature abstraction (HTTP wrapper, line editor wrapper, etc.) - -- **Sketch**: each feature carries its own abstraction. -- **Why rejected**: duplication. A central vtable in Zone 0 covers - every consumer. - -## Consequences - -- **Positive**: a Zig stdlib reshape affects one file. The five - disabled v0 features can be re-implemented above this abstraction. -- **Negative**: an indirection cost per I/O call (function pointer). - Negligible vs. system call costs. -- **Neutral / follow-ups**: clojure.java.io's `(reader x)` / - `(writer x)` / `(input-stream x)` / `(output-stream x)` are - implemented in `cljw.host.java.io` against the abstraction. - -## Phase 14+ migration note (amendment 2) - -The Phase 4 entry (task 4.13) lands `runtime/io_interface.zig` (Tier -1, Zone 0) + `runtime/io_default.zig` (Tier 2, Zone 1) skeleton plus -`defaultReader()` / `defaultWriter()` injection in `main()`. cw v0's -five disabled features (F140-F144) are **not** re-introduced at -Phase 4; they re-land at Phase 14 alongside REPL / nREPL. - -Phase 14 entry activates each F-feature as a new file under the -appropriate zone, **and** rewrites a small number of Phase 4-13 -src/ sites where today the dispatch row is a `feature_not_supported` -catalog raise: - -| F | Phase 14 landing path | Phase 4-13 placeholder rewritten | -|------|----------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------| -| F140 | `src/host/io/http_server.zig` (new) | `cljw.host.java.net` namespace dispatch row | -| F141 | `src/host/io/http_client.zig` (new) | `cljw.host.java.net` namespace dispatch row | -| F142 | `src/app/nrepl.zig` (new; bencode under `src/runtime/bencode/` per amendment 3) | `src/app/main.zig` subcommand dispatch row for `nrepl` | -| F143 | `src/app/repl/line_editor.zig` (new) | `src/app/main.zig` subcommand dispatch row for `repl` | -| F144 | `src/app/builder.zig` (new; bytecode self-bundle per amendment 5) — distinct from `cljw component build` (Wasm output, row 14.12, zwasm-v2-gated) | `src/app/cli.zig` `build` subcommand arm + startup `tryRunEmbedded` hook (ungated — see amendment 5) | - -Tier 2 isolation (per amendment 1) means that if Zig 0.17 / 0.18 -reshapes `std.Io` again before Phase 14, only `io_default.zig` is -rewritten — the F140-F144 landing files import from -`io_interface.zig` only. - -The rewrite is expected per ROADMAP §A25; principle.md depth 2 for -each subcommand dispatch row swap, depth 3 if a `std.Io` reshape -coincides with the Phase 14 landing. - -## References - -- ROADMAP §9.6 task 4.13 (io_interface.zig Zone 0) -- ROADMAP §9.16 (Phase 14 entry — v0.1.0 milestone with REPL / - nREPL re-introduction) -- ROADMAP §A1 (Zone discipline) -- ROADMAP §A25 (Existing code is mutable) -- cw v0 disabled features F140-F144 - -## Revision history - -- 2026-05-23: Status: Proposed -> Accepted (initial landing). -- 2026-05-23 (amendment 1): Added "Two-tier strategy" section - documenting the Tier 1 (Zone 0, no `std.Io` import) and Tier 2 - (Zone 1, the only `std.Io`-importing file) split. Source: - zwasm v1 D135 + `private/research-2026-05-23/INSIGHTS_ZWASM_V1.md` - observation that Tier 2 isolation makes Zig stdlib reshape a - one-file change instead of a project-wide recovery. -- 2026-05-23 (amendment 2): Added "Phase 14+ migration note" - section to narrate F140-F144 re-introduction landing paths and - the corresponding Phase 4-13 placeholder dispatch rewrites (per - ROADMAP §A25). -- 2026-05-28 (amendment 3): F142 row narrowed. Devil's-advocate - for row 14.10 (cljw nrepl landing) recommended moving the - bencode codec out of `src/app/nrepl/` into `src/runtime/bencode/` - per F-009 strict (namespace-neutral wire codec; matches - `runtime/regex/`, `runtime/io/` precedent). server / session / - ops folded into a single `src/app/nrepl.zig` (~370 LOC; under - the 1000-LOC §A6 cap so the 3-file split is reservation-as-bias - until growth forces it). ADR-0015 a2's specific path table was - a memo per CLAUDE.md spirit; the finished form amends in place. - v0 disabled-feature F142 reactivation closed at commit 76e0c64c - (row 14.10). -- 2026-05-29 (amendment 5): F144 row narrowed + landed ungated. - Devil's-advocate for row 14.11(b) (`cljw build app.clj -o app` - bytecode self-bundle) confirmed `build` has **no** dependency on - the `phase_at_least_14` flag — that flag guards the io_interface - Tier-2 stub swap (`runtime/io/stub.zig` + HTTP/REPL/nREPL stubs, - F140-F143 host-I/O surface), not the Deno-style bytecode-trailer - mechanism (F-001-separate from zwasm). `build` therefore lands the - same way `repl` (14.9) / `nrepl` (14.10) did: a subcommand arm in - `src/app/cli.zig` + a startup `builder.tryRunEmbedded` detection - hook, both unconditional. Path moved from the amendment-2 memo - `src/app/build/self_bundle.zig` to `src/app/builder.zig` (single - file, under §A6 1000-LOC cap; the subdir split is reservation-as- - bias until growth forces it — same reasoning as amendment 3's - nrepl fold). The row-14.14 `phase_at_least_14` flip no longer owns - `build`'s landing; it confirms + flips the io stub flag only. - **`cljw component build`** (Wasm Component output, row 14.12) - remains a *distinct* F144-adjacent feature gated on zwasm v2 - readiness (D-036/037/038/F-008) — it is not this bytecode - self-bundle. ADR-0015 a2's path table is a memo per CLAUDE.md - spirit; the finished form amends in place. The coupled payload- - format + build-time-eval decision (sequence-of-chunks, per-form - compile-then-eval matching Clojure AOT) lives in **ADR-0034 - amendment 1** (the build-format owner); the Devil's-advocate - fork output is embedded verbatim there. diff --git a/.dev/decisions/0016_file_size_smell_detector.md b/.dev/decisions/0016_file_size_smell_detector.md deleted file mode 100644 index d470276cc..000000000 --- a/.dev/decisions/0016_file_size_smell_detector.md +++ /dev/null @@ -1,118 +0,0 @@ -# 0016 — File size as a smell detector, not a hard metric - -- **Status**: Accepted -- **Date**: 2026-05-23 -- **Author**: Shota Kudo (drafted with Claude) -- **Tags**: file-size, refactor, discipline, smell-detector, phase-5-6 - -## Context - -cw v0 produced `collections.zig` at 6,269 lines and three other -files over 1,500 lines. Each was perfectly intentional in isolation, -yet the aggregate ended up hard to read and audit. zwasm v2 landed -ADR-0099 to reframe its own file-size discipline from "drive the -metric down" to "trigger investigation when the metric trips." - -zwasm v1 itself, in `src/vm.zig` (10,550 lines) and `src/jit.zig` -(8,701 lines), evolved into what its own ARCHITECTURE.md calls -"implicit-contract sprawl" — the file size grew because each new -opcode or backend feature dropped in alongside existing ones with -no extraction trigger. The external evidence is direct: the same -project that uses Zig 0.16 in a similar runtime domain reached -10K+ LOC in two files and identifies it as a structural problem. -A 2,000-line hard cap with explicit exemption markers would have -forced the conversation about extraction long before the 10K -threshold. - -cw v1 needs the same reframe before the Phase 5 collections work -lands, since RRB-tree vector + HAMT hashmap + ChunkedSeq + persistent -list could otherwise repeat the cw v0 outcome. - -## Decision - -`file_size_check.sh` reports two thresholds: - -- **Soft cap**: 1,000 lines per `.zig` file. Crossing the soft cap - triggers a smell investigation, not an automatic split. -- **Hard cap**: 2,000 lines per `.zig` file. Crossing the hard cap - fails the merge gate unless the file carries a - `FILE-SIZE-EXEMPT: (ADR-NNNN)` marker (per ROADMAP §A14). - -A smell investigation answers: - -### Positive criteria (extract iff at least one holds) - -- **P1**: a spec-defined closed sublanguage (≥300 LOC of code, not - comments) lives in the file (e.g., a SIMD validator). -- **P2**: pure-data dominance — a single declaration block exceeds - 40% of the file (e.g., a large `pub const Table = .{ ... }`). -- **P3**: an independent change cadence + a deep interface (≥3 public - symbols with ≥2 callers each) makes the section extractable. -- **P4**: test surface isolation — the section can be unit-tested - without the rest of the file's setup. - -### Negative criteria (do NOT extract if any holds) - -- **N1**: helper-circular import — the child would re-import the - parent's private helper. -- **N2**: forced pub-leak — a previously private function would have - to become `pub` to satisfy the extraction. -- **N3**: shallow module — the proposed child is <100 LOC and would - collapse two-step navigation onto every existing caller. -- **N4**: test fixture pub-leak — extraction forces test helpers to - become public. - -### Marker convention - -Files that intentionally exceed a cap carry one of: - -- `FILE-SIZE-EXEMPT: (ADR-NNNN)` — file size cap exemption. -- `SIBLING-PUB: (ADR-NNNN)` — cross-file struct method - pattern. -- `SKIP-` — test filtering rationale. - -`scripts/file_size_check.sh` activates as a hard gate when this ADR -moves to Accepted (Phase 5-6 entry). - -## Alternatives considered - -### Alternative A — "Aim for under 1,000 lines always" - -- **Sketch**: treat the soft cap as a strict metric. -- **Why rejected**: every file under 1,000 lines can still aggregate - to an unreadable codebase if extraction creates micro-modules. - This is the failure mode ADR-0099 (in zwasm v2) reframed. - -### Alternative B — No file size policy - -- **Sketch**: rely on review judgment. -- **Why rejected**: cw v0 had no policy and produced the 6,269-line - outcome. A trigger is cheap insurance. - -## Consequences - -- **Positive**: triggers investigation at the right moment, not - every commit. Avoids fragmentation into useless ~50-line files. -- **Negative**: requires explicit justification when a file - exceeds the hard cap. -- **Neutral / follow-ups**: status moves to Accepted when Phase 5 - collections split work begins. - -## References - -- ROADMAP §A12 (File size — smell, not metric) -- ROADMAP §A14 (Structural discipline markers) -- zwasm v2 ADR-0099 (precedent reframe) - -## Revision history - -- 2026-05-23: Status: Proposed (initial landing). Activation deferred - to Phase 5-6 when collections work begins. -- 2026-05-23 (amendment 1): Status promoted Proposed -> Accepted. - External evidence added to Context: zwasm v1's `src/vm.zig` - (10,550 lines) and `src/jit.zig` (8,701 lines) reached - "implicit-contract sprawl" exactly because no 2,000-line hard - cap forced an earlier conversation. Source: - `private/research-2026-05-23/INSIGHTS_ZWASM_V1.md`. Phase 5-6 - activation timeline unchanged; the promotion is from "future - policy" to "active policy with documented external evidence". diff --git a/.dev/decisions/0017_allocator_strategy.md b/.dev/decisions/0017_allocator_strategy.md deleted file mode 100644 index 6bb13b8cc..000000000 --- a/.dev/decisions/0017_allocator_strategy.md +++ /dev/null @@ -1,152 +0,0 @@ -# 0017 — Allocator strategy: per-eval arena + general-purpose + mark-sweep heap - -- **Status**: Accepted -- **Date**: 2026-05-23 -- **Author**: Shota Kudo (drafted with Claude) -- **Tags**: phase-4-entry, allocator, gc, memory, mark-sweep, arena - -## Context - -JVM Clojure relies on the JVM's generational garbage collector. cw v1 -has no built-in GC; allocation strategy is a deliberate design -choice. Three considerations drive it: - -1. **Most values are short-lived**: a single REPL evaluation produces - many transient list / map / closure objects that all die when the - evaluation completes. -2. **Some values are long-lived**: deftype instances, atom contents, - namespace bindings, var roots persist across evaluations. -3. **The runtime is batch / REPL oriented**: stop-the-world pauses on - the order of 100ms are acceptable in exchange for GC simplicity. - Low-pause behaviour (G1 / ZGC class) is not a requirement. - -## Decision - -cw v1 uses three allocators with distinct roles: - -### Per-evaluation arena (`std.heap.ArenaAllocator`) - -Created at the start of each top-level evaluation, freed in bulk at -the end. Backs short-lived values: intermediate seq pipeline output, -ex-info data created mid-evaluation, transient buffers, analyzer -scratch. - -Phase 4 entry: the arena pattern is already in use; this ADR -formalizes it as the canonical short-lived path. - -### General-purpose allocator (`std.heap.GeneralPurposeAllocator`) - -Backs long-lived heap allocations whose lifetime is not tied to a -single evaluation: deftype instances, atom contents, var roots, -namespace bindings. - -In Debug builds, the GPA tracks leaks. In Release builds, it falls -back to the page allocator with minimal metadata. - -### Mark-sweep GC heap - -A subset of long-lived allocations participates in mark-sweep GC. The -heap is a `GcHeap` value: it owns a list of tracked objects, each -carrying a `gc_mark` bit (ADR-0009 `ObjectHeader.gc_and_lock`). -Collection runs when the tracked-object count exceeds a threshold or -when explicitly requested. - -Phase 5 brings the mark-sweep implementation online. Phase 4 entry -reserves the bit slot in `ObjectHeader` but does no collection (the -allocator simply allocates). - -### Phase staging - -- Phase 4 entry: declarations only (`gc_mark` bit reserved in the - packed header, GcHeap struct exists without a sweep function). -- Phase 5: mark-sweep implementation, threshold-based collection, - test coverage for unreachable cycles. -- Phase 15+: re-evaluate generational GC based on bench data. - -## Alternatives considered - -### Alternative A — Generational GC from day 1 - -- **Sketch**: implement young / old generation. -- **Why rejected**: the per-eval arena already captures most of the - short-lived population. Generational GC adds card-table complexity - for a benefit (minor-GC throughput) that arena allocation already - delivers. - -### Alternative B — Reference counting - -- **Sketch**: each heap value carries a refcount. -- **Why rejected**: persistent collection structural sharing creates - long cycles when shared subtrees are released; reference counting - pays write barriers on every assoc. - -### Alternative C — Stop-the-world only, no concurrent GC - -- **Sketch**: never run GC concurrently with mutator. -- **Decision**: this is what cw v1 does for Phase 5. Concurrent GC - is a Phase 15+ re-evaluation if pause time becomes a problem. - -## Consequences - -- **Positive**: simple three-allocator model with clear lifetimes. - Short-lived population (the majority) never reaches GC. Long-lived - population gets mark-sweep, which is simple to implement and audit. -- **Negative**: stop-the-world pause may reach 100ms on large heaps. - Acceptable for cw v1's batch / REPL workloads. -- **Neutral / follow-ups**: Phase 5 mark-sweep test suite covers - cyclic references, structural sharing, transient -> persistent - handoff. - -## Phase 5+ migration note (amendment 1) - -The Phase 4 entry reserves the `gc_mark` bit in `ObjectHeader` -(ADR-0009 amendment 2) and declares `GcHeap` as a struct skeleton -with no `collect()` body (per ADR-0023 Pattern B stub). Phase 5 -activates mark-sweep; the activation **rewrites every Phase 1-4 -heap allocation site that today calls `gpa.alloc(...)` directly** -to instead route through `GcHeap.alloc(...)`. - -Affected categories (Phase 5 source, populated as each lands): - -- HAMT persistent vector / hashmap / hashset construction — Phase 5 - greenfield; routes through `GcHeap` from the start (not a - rewrite, but the discipline starts here). -- `deftype` / `defrecord` / `reify` instances (per ADR-0007 - amendment 1) — the same activation Phase rewrites collection - primitives, so `GcHeap` adoption rides that pass. -- `ex_info` instances (Phase 3-shipped: `src/runtime/ex_info.zig`) - — currently uses arena allocator for the message string and GPA - for the struct; Phase 5 reroutes the struct alloc through - `GcHeap` and keeps arena for short-lived message buffers. -- `LazySeq` — Phase 4 task 4.24 ships the struct skeleton; Phase 5 - `force()` activation also routes the seq node through `GcHeap`. -- Future: `atom` / `ref` / `agent` content cells re-allocate - through `GcHeap` at Phase 14-15 (per ADR-0010 amendment 2); those - Phases will rewrite their own activation, not Phase 5. - -The arena allocator (per-evaluation short-lived population) remains -unchanged; the rewrite is only at the GPA → GcHeap boundary, not at -the arena boundary. - -The rewrite is expected per ROADMAP §A25; principle.md depth 3 -(cross-file refactor of alloc call sites) is typical, depth 4 only -if the per-eval arena / GcHeap boundary itself needs an ADR -revision. - -## References - -- ROADMAP §9.6 task 4.0 (bench harness will measure pause time) -- ROADMAP §9.7 (Phase 5 entry — mark-sweep activation) -- ROADMAP §A25 (Existing code is mutable) -- Related ADRs: 0007 (TypeDescriptor activation rides this Phase), - 0009 (ObjectHeader gc_mark slot), 0010 (atom/ref content cells - in Phase 14-15) -- JVM_TO_ZIG.md §3 (working notes) - -## Revision history - -- 2026-05-23: Status: Proposed -> Accepted (initial landing). -- 2026-05-23 (amendment 1): Added "Phase 5+ migration note" section - to narrate the rewrite of Phase 1-4 GPA-direct alloc call sites - to `GcHeap.alloc` routing when mark-sweep activates in Phase 5 - (per ROADMAP §A25; pairs with ADR-0007 amendment 1). diff --git a/.dev/decisions/0018_error_catalog_ssot.md b/.dev/decisions/0018_error_catalog_ssot.md deleted file mode 100644 index 2a0c6b5c7..000000000 --- a/.dev/decisions/0018_error_catalog_ssot.md +++ /dev/null @@ -1,372 +0,0 @@ -# 0018 — Error catalog as Single Source Of Truth - -- **Status**: Accepted -- **Date**: 2026-05-23 -- **Author**: Shota Kudo (drafted with Claude) -- **Tags**: phase-4-entry, error, ssot, catalog, user-facing-message - -## Context - -`src/runtime/error.zig` already centralises the *categorisation* axis -of errors: `Kind` (16 semantic categories), `Phase` (parse / analysis -/ macroexpand / eval), 1:1 mapping to a Zig error union, threadlocal -`Info` payload, 64-frame call stack. That part is fine. - -What is **not** centralised is the *message body*. Each call site -writes its own `comptime fmt: []const u8` directly into -`setErrorFmt(.eval, .type_error, loc, "{s}: expected number, got {s}", args)`. -There are about 80 such call sites across `src/eval/`, `src/lang/`, -and `src/runtime/`. Three problems compound: - -1. **Drift**: the same semantic error appears with slightly different - wording in different files (e.g., the four variants of "Wrong - number of args ..." in `error.zig`, `core.zig`, `error.zig` - inside `lang/primitive/`, and `tree_walk.zig`). -2. **Development calendar leaks into user-facing text**. The current - scaffolding documents and earlier ADR drafts illustrated - not-yet-implemented errors with strings such as - `"dosync: STM activates at Phase 15, see ADR-0010"` or - `"Phase 5: deftype not yet implemented, see ADR-0007"`. "Phase 15" - and "ADR-0010" are cw v1 development concepts; a user running - `cljw -e '(dosync ...)'` should not see them. -3. **No grep surface for the message catalog**. There is no single - file that answers "what user-facing errors does cljw produce". - A grep across `setErrorFmt` works but mixes templates with - surrounding code. - -URL-style references (`See: https://...`) were considered for the -"see also" slot. They are excluded from the catalog at this stage — -no public docs site exists yet, and per-message URLs would either -hard-code a domain that does not exist or carry empty placeholders. -Until the docs site is up, the user sees the structured error message -only. - -## Decision - -Introduce `src/runtime/error_catalog.zig` as the Single Source Of -Truth for every user-facing error message in the cw runtime. - -### Shape - -```zig -//! src/runtime/error_catalog.zig -const std = @import("std"); -const error_mod = @import("error.zig"); - -pub const Kind = error_mod.Kind; -pub const Phase = error_mod.Phase; -pub const SourceLocation = error_mod.SourceLocation; -/// SSOT name for the Zig error union returned by every raise site. -/// Renamed from the original `Error` so that the `catch |err|` site -/// surface (and the failing test trace) names the project explicitly. -pub const ClojureWasmError = error_mod.ClojureWasmError; - -/// One variant per distinct user-facing message. -pub const Code = enum { - // Parse / Analysis / Macroexpand / Eval entries follow the - // _ naming convention (see "Code variant - // naming conventions" below). Examples: - delimiter_unexpected, - string_unterminated, - def_arity_invalid, - type_arg_not_number, - arity_invalid, - // ... - - // Tier D forms: one Code per form, each with a hand-written - // user-helpful template. No `{name}` slot — the template is - // tailored to the specific form (per Shota's directive that - // "implementation refusals deserve individual care"). - tier_d_gen_class, - tier_d_gen_interface, - tier_d_compile, - tier_d_proxy_deep, - tier_d_bean_deep, - - // Sub-feature staged unsupported. While a feature (e.g. STM) - // lands incrementally across Phase 13-15, the entry-point form - // becomes usable but specific sub-operations are still raise - // sites. Each sub-operation gets its own Code so the user sees - // exactly which step is missing. - stm_dosync_not_supported, - stm_commute_not_supported, - // ... (one Code per sub-operation, added/removed as Phases land) - - // Generic fallback for "on the roadmap, not yet broken into - // sub-feature Codes". Use sparingly; promote to a sub-feature - // Code as soon as the feature's Phase planning has resolution. - /// args: .{ .name = "" } - feature_not_supported, -}; - -const Entry = struct { kind: Kind, phase: Phase, template: []const u8 }; - -pub fn entry(comptime code: Code) Entry { - return switch (code) { - .delimiter_unexpected => .{ - .kind = .syntax_error, .phase = .parse, - .template = "Unexpected delimiter '{[delim]s}'", - }, - .type_arg_not_number => .{ - .kind = .type_error, .phase = .eval, - .template = "{[fn_name]s}: expected number, got {[actual]s}", - }, - .tier_d_gen_class => .{ - .kind = .not_implemented, .phase = .analysis, - .template = "gen-class requires emitting JVM bytecode classes, which is outside the scope of ClojureWasm. Use deftype + defprotocol for cw-native type definitions.", - }, - .feature_not_supported => .{ - .kind = .not_implemented, .phase = .eval, - .template = "{[name]s} is not supported in ClojureWasm", - }, - // ... (every other Code arm follows the same shape) - }; -} - -pub fn raise(comptime code: Code, location: SourceLocation, args: anytype) ClojureWasmError { - const e = comptime entry(code); - return error_mod.setErrorFmt(e.phase, e.kind, location, e.template, args); -} -``` - -### Rules - -1. **`error_catalog.zig` is the only file allowed to call - `setErrorFmt` directly.** Every other module calls - `error_catalog.raise(.your_code, loc, args)`. This is enforced by - `.claude/rules/error_catalog_only.md` and (Phase 5+) by - `scripts/check_no_op_stub.sh` style heuristic grep. -2. **Adding a new error is two edits**: append a `Code` variant + - append the matching `entry()` arm. Never write a new `setErrorFmt` - call outside this file. -3. **Templates must not name development concepts**. "Phase N", - "ADR-NNNN", internal opcode names, file paths inside cw, and URLs - are forbidden in **template strings** (i.e., the `template` field - of an `entry()` arm). The same identifiers remain fine in ADR - prose, rule docs, code comments — anywhere a developer reads them. - The boundary is: anything `std.fmt.bufPrint` would render into a - message shown to a `cljw` user must pass this rule. -4. **Two patterns for unsupported features, by intent**: - - **Permanent exclusion (Tier D)**: one `Code` variant per form, - hand-written user-helpful template. No `{name}` slot — the - wording is specific to that form's reason and suggests the - cw-native alternative. There are 5 such Codes today - (`tier_d_gen_class`, `tier_d_gen_interface`, `tier_d_compile`, - `tier_d_proxy_deep`, `tier_d_bean_deep`) matching - `compat_tiers.yaml` Tier D enumeration. - - **Roadmap-but-not-yet (sub-feature staged)**: while a feature - lands incrementally (e.g. STM across Phase 13-15), the entry - form is usable but specific sub-operations remain raise sites. - Each sub-operation gets its own Code - (`stm_dosync_not_supported`, `stm_commute_not_supported`, ...) - so the user sees exactly which step is missing. Codes are - added when a sub-operation is identified as unsupported and - removed when the sub-operation is implemented. - - **Generic fallback `feature_not_supported`**: for cases that - have not yet been resolved into sub-feature Codes. Carries the - `{name}` slot. Treated as a temporary state — promote to a - concrete sub-feature Code when the feature's Phase planning - gives resolution. The Phase 5+ audit reviews this Code's call - sites and promotes each to a named Code. - -### Code variant naming conventions - -The `Code` enum's variant names use **`_`** -in snake_case, prioritising the form / construct the user wrote over -internal classification: - -- `delimiter_unexpected`, not `parse_unexpected_delimiter` and not - `eval_type_error_215`. -- `string_unterminated`, `integer_invalid`, `def_arity_invalid`, - `type_arg_not_number`, `arity_invalid`, `let_bindings_invalid`. -- For Tier D: `tier_d_` (the only prefix-style Code names). -- For sub-feature staged unsupported: `__not_supported` - (e.g. `stm_dosync_not_supported`). -- For `feature_not_supported` (generic fallback): no prefix, this is - the one exception. - -When two raise sites express the *same* user-visible error, they -share one `Code` variant. Distinct user wording = distinct variant. - -The `Phase` is **not** encoded in the Code name; the `entry()` arm -carries the Phase. This keeps the Code name focused on what the user -sees, not where in the pipeline the runtime detected it. - -### Named format args - -Zig 0.16 `std.fmt.bufPrint` accepts `{[field_name]}` -syntax against a struct args value -(`std.fmt.zig:1304-1305` carries the canonical example). The catalog -relies on this so callers pass `.{ .fn_name = "+", .actual = "keyword" }` -rather than positional tuples. Named slots are self-documenting at -the call site and make adding / removing slots non-breaking when the -slot is not yet used by other callers. - -## Alternatives considered - -### Alternative A — YAML or ZON data file - -- **Sketch**: `error_catalog.yaml` or `.zon` at repo root, parsed at - build time. -- **Why rejected**: introduces a build step (codegen) for a value - that Zig comptime already expresses well. `compat_tiers.yaml` is - YAML because it is consumed by multiple tools (test runner, REPL, - future `cljw --list-vars`); the error catalog is consumed only by - the runtime, so the cheap path is a Zig switch. - -### Alternative B — One Zig function per error - -- **Sketch**: `error_catalog.zig` exports - `raiseTypeExpectedNumber(loc, fn_name, actual_type)` per error. -- **Why rejected**: 80+ functions, each with hand-written parameter - lists. The `raise(comptime Code, args)` shape covers the same - surface in one function and keeps the type / phase / template in - one place per error. - -### Alternative C — Per-call-site URL reference - -- **Sketch**: each template ends with - `"\nSee: https://cw.org/errors#"`. -- **Why rejected at this stage**: no docs site exists yet. Hard-coded - URLs would either point at a 404 or carry placeholders that age - badly. The catalog stays URL-free until the public docs site is - defined. - -## Consequences - -- **Positive**: every user-facing error is one grep away. Adding a - new error is mechanical. The user no longer sees development - calendar text. The reviewer can audit the message catalog without - cross-referencing source files. The `[name_error]` / - `[type_error]` etc. labels in user-facing output give the user a - category handle ("what kind of problem is this?") without exposing - the catalog `Code` identifier. -- **Negative**: the existing ~116 `setErrorFmt` call sites (measured - by `grep -rn "setErrorFmt" src/ | wc -l`, including the helper - fns `expectNumber` / `checkArity` / `checkArityMin` / - `checkArityRange` in `error.zig` that today carry inline - templates) migrate to `raise(.code, loc, args)`. The migration is - mechanical but not free; tracked as Phase 4 task 4.26. The - count is approximate because each `error.zig` helper expands to - one inline `setErrorFmt` call per call path, and the helpers - themselves migrate to the catalog (their templates become - catalog entries). -- **Negative**: the Zig error-union name change from `Error` to - `ClojureWasmError` (full spelling preferred over `CWError`) - cascades through every `pub fn ... !Value` signature that - currently spells `error_mod.Error`. The change is grep-and-replace - but the file count is non-trivial; bundled into task 4.26. -- **Neutral / follow-ups**: - - `Kind` is intentionally kept (16 variants today). Its job is - twofold: (i) the user-visible category label - (`[type_error]` etc.) so the user grasps "what kind of problem - is this?" without seeing the catalog `Code`, and (ii) the - `catch |err|` discriminator (`error.NameError` / `error.TypeError` - / ...) for cw runtime internal dispatch. Adding a `Kind` variant - is allowed when an existing category does not fit; the discipline - is to grow `Kind` rarely (every new variant has a justification - in the amending ADR or in this one). Earlier drafts of this ADR - considered removing `Kind` to reduce redundancy; that was - rejected because both jobs above genuinely need it. - - Each Tier D form has its own Code (5 today). When a new Tier D - form is added (a MAJOR release per ADR-0013), a new Code lands - with it; the template is hand-written and may run to multiple - sentences to suggest the cw-native alternative. - - Sub-feature staged unsupported Codes (`stm_*_not_supported`, - later `host_io_*_not_supported`, etc.) are *transient*. A Code - of this kind exists only while its sub-operation is unimplemented; - removing it is part of the Phase that implements the - sub-operation. The catalog's growth is therefore not monotonic — - Codes come and go. - - When the public docs site exists, a follow-up amends this ADR - to add an optional `docs_anchor` field to `Entry`. The runtime - formatter can then opt-in to render - `"See: /"` at the end of each message. Until - then, the catalog stays URL-free. - - Task 4.26 is large (~116 sites). At task open the implementer - splits it into sub-tasks per source-tree region (`reader.zig`, - `analyzer.zig`, `tree_walk.zig`, `lang/macro_transforms.zig`, - `lang/primitive/*`, `runtime/error.zig` helpers); see - handover.md when 4.26 is the active task. - - Crash policy (when `@panic` is acceptable, when `internal_error` - catalog raise is required, what happens on a native crash) is - addressed in a separate ADR-0019. - -## References - -- `src/runtime/error.zig` (existing `Kind` / `Phase` / `Info` / - `setErrorFmt` infrastructure) -- ROADMAP §2 P6 (Error quality is non-negotiable) -- ROADMAP §A11 (day-1 enum reservation — the catalog `Code` enum is - the error-axis equivalent) -- ROADMAP §9.6 task 4.26 (migration task) -- Related ADRs: 0009, 0010, 0013, 0017 (error message examples in - those ADRs have been amended to remove development-calendar leaks) -- Rule: `.claude/rules/error_catalog_only.md` -- Zig 0.16 `std.fmt.bufPrint` named placeholder syntax - (`zig/lib/std/fmt.zig:1304-1305`) - -## Revision history - -- 2026-05-23: Status: Proposed -> Accepted (initial landing). -- 2026-05-23 (amendment 1): Rule 3 sharpened with the - template-vs-prose boundary. Added `Code` variant naming conventions. - Consequences expanded with (i) the rationale for `tier_d_form` and - `unsupported_feature` sharing `Kind.not_implemented`, (ii) the - measured ~116 site count and helper-fn migration note, (iii) - sub-task split guidance for task 4.26. Self-review feedback. -- 2026-05-23 (amendment 2): Major reshape per user direction. - (a) Code naming convention changed from - `_` to `_` for lower - cognitive load (`type_arg_not_number` instead of - `eval_type_expected_number`, etc.). Phase is no longer encoded in - the Code name; the `entry()` arm carries it. - (b) Tier D split into one Code per form - (`tier_d_gen_class` / `tier_d_gen_interface` / `tier_d_compile` / - `tier_d_proxy_deep` / `tier_d_bean_deep`), each with a - hand-written user-helpful template that explains the reason and - suggests the cw-native alternative. The generic `tier_d_form` - with `{name}` slot is dropped. - (c) Sub-feature staged unsupported pattern introduced. While a - feature lands across multiple Phases, each unimplemented - sub-operation gets its own Code (e.g. - `stm_dosync_not_supported`). These Codes are transient — they - disappear when the sub-operation is implemented. - (d) Generic `feature_not_supported` (renamed from - `unsupported_feature`) kept as a temporary fallback for cases not - yet resolved into sub-feature Codes. - (e) `Kind` is kept (earlier removal proposal rejected). It serves - both as user-visible category label and as `catch |err|` - discriminator. - (f) Zig error union renamed from `Error` to `ClojureWasmError` - so the catch surface and failing test traces name the project. - All changes are reflected in catalog source as part of task 4.26 - (the existing 28 Codes from the initial landing will be renamed - in the same task). - -- 2026-05-24: Amendment 3 — `divide_by_zero` Code added under the - existing `arithmetic_error` Kind, raised from numeric arithmetic - paths (5.9.b Ratio constructor + 5.9.d arithmetic dispatch). - Template "Divide by zero" matches the JVM Clojure - `ArithmeticException` surface text. No new Kind needed - (`arithmetic_error` slot was reserved on the initial Kind list - and is now lit up). Devil's-advocate cross-check: - alternatives "raise generic `value_error` with a custom - template" and "leave the Zig-API surface (error.DivideByZero) as - the only surface" were both considered; rejected because (a) - Clojure callers expect to `(try ... (catch ArithmeticException - e ...))` which maps to the `arithmetic_error` Kind, and (b) the - user-visible message `"Divide by zero"` is the most-grepped - Clojure error string and deserves its own Code for grep-by-Code - testing in Phase 6+. - -- 2026-05-24: Amendment 4 — `integer_overflow` Code added under the - existing `arithmetic_error` Kind, raised from the - `+'` / `-'` / `*'` strict-integer family (Phase 5.10.c). Template - "integer overflow" mirrors the JVM Clojure - `ArithmeticException` text for `(*' Long/MAX_VALUE 2)` etc. No - new Kind needed; same-Kind group as `divide_by_zero` from - amendment 3. Devil's-advocate cross-check inline: alternatives - "reuse `value_error`" and "let it raise as an internal Zig - error" rejected for the same reasons as amendment 3 (Clojure - `(try ... (catch ArithmeticException ...))` is the surface). diff --git a/.dev/decisions/0019_crash_policy.md b/.dev/decisions/0019_crash_policy.md deleted file mode 100644 index 47229e607..000000000 --- a/.dev/decisions/0019_crash_policy.md +++ /dev/null @@ -1,185 +0,0 @@ -# 0019 — Crash policy: panic, internal error, and native crash - -- **Status**: Accepted -- **Date**: 2026-05-23 -- **Author**: Shota Kudo (drafted with Claude) -- **Tags**: phase-4-entry, error, crash, panic, internal-error, robustness - -## Context - -ADR-0018 specifies how user-input-driven errors are produced (catalog -`raise`). It does not cover what happens when: - -- A cw runtime invariant is violated (a path the developer thought - unreachable is reached). -- Zig itself traps at runtime (integer overflow in safe builds, - index out of bounds, null pointer dereference, `unreachable`). -- The process receives a fatal signal (SIGSEGV, SIGBUS). - -Without a policy, three failure modes appear and have appeared in -cw v0: - -1. **`@panic("...")` sprinkled everywhere**. Some panics gate - user-input paths (incorrect — they should be catalog raises); - some gate genuine internal invariants (correct, but the message - carries no `Code` and no `SourceLocation`). -2. **Silent `unreachable`**. A "this can never happen" assertion that - later turns out to happen, leaving the user staring at a Zig - trap with no context. -3. **`@panic` for "not yet implemented"**. Mixed with the catalog - `feature_not_supported` Code, creating two parallel "not done" - surfaces. - -This ADR draws the line between the three layers and specifies the -top-level catch behaviour. - -## Decision - -cw v1 distinguishes three crash layers and applies a fixed policy to -each. - -### Layer 1: User-input error (catalog, ADR-0018) - -Any code path reachable by user input — parsing, analysis, macro -expansion, evaluation of user-supplied forms, primitive function -arguments, `cljw -e` input, file input, REPL input, nREPL input, -stdin input — raises through the catalog -(`error_catalog.raise(.code, loc, args)`). `@panic` is **forbidden** -in these paths. - -If a path is "obviously" reachable only via user input but the -developer is unsure, default to catalog raise. Adding a catalog -`Code` is cheap. - -### Layer 2: Runtime invariant violation - -A "runtime invariant violation" is a state the developer believed -the cw runtime itself would never reach (e.g. "the dispatch vtable -field is non-null because installVTable runs in `main`"). Two -sub-cases: - -- **The invariant is genuinely guaranteed by construction** - (function only callable from one site that proves the - precondition): use `std.debug.assert(...)` or - `if (cond) unreachable;`. These cost nothing in release mode - and trap in safe builds. Document the precondition in a `//` - comment above the assertion. -- **The invariant is "should hold" but the developer cannot - statically prove it**: raise the catalog `internal_error` Code - with a `{detail}` slot describing the violation. The user sees - a `[internal_error]` category and a hint to file a bug. The - developer sees a `Code` in the failing test output and the - source location. - -`@panic("...")` with a bare string is **discouraged**. Replace with -either `std.debug.assert` (with a comment) or catalog -`internal_error` raise. Existing `@panic` call sites are audited -during task 4.26; each is converted to one of the two forms above -or annotated with a `// @panic: ` comment justifying why it -must remain. - -### Layer 3: Native crash - -Some failure modes are below the Zig error mechanism: - -- Integer overflow in release-fast builds (silent wrap; not a crash, - but worse). -- Stack overflow with no recoverable trap. -- SIGSEGV / SIGBUS from a genuine memory bug. - -Policy: - -- **Build mode**: cw v1 default release mode is `ReleaseSafe` - (overflow traps, OOB traps). `ReleaseFast` builds are produced only - for benchmark / distribution and carry a clear opt-in flag. -- **Top-level catch in `main`**: the `pub fn main(init: std.process.Init)` - body wraps the dispatch in a `catch |err|` that, when the error - set includes `ClojureWasmError`, formats the catalog error and - exits with a per-`Kind` exit code. Any other error (Zig native - trap that surfaces as a `error.OutOfMemory` etc.) is rendered as - an `[internal_error]` with the Zig error name as the `{detail}`. -- **Native signal handler** (Phase 14+, when REPL / nREPL is wired): - install a SIGSEGV / SIGBUS handler that prints - `"Internal error: ClojureWasm crashed unexpectedly."` to stderr - with the cw build commit SHA and exits 70. Phase 4-13 ships - without this handler; a SIGSEGV simply terminates the process - (acceptable for non-interactive Phase 4 usage). - -### Exit codes - -| Exit code | Meaning | -|-----------|------------------------------------------------------------------| -| 0 | Success | -| 1 | User-facing catalog error (any `ClojureWasmError` variant) | -| 70 | Internal error (cw bug; `internal_error` Code or signal handler) | -| 130 | SIGINT (Phase 14+ REPL) | - -The split between 1 and 70 lets `cljw -e ...` integrate into shell -pipelines that distinguish "the user's code was wrong" from "cw -itself broke". - -## Alternatives considered - -### Alternative A — `@panic` everywhere - -- **Sketch**: any unexpected state panics with a string message. -- **Why rejected**: panics carry no `Code`, no `SourceLocation`, no - `Kind`. Output is `thread N panic: ...` with no integration with - the catalog or the user-facing format. User experience is bad - for both end users and cw developers. - -### Alternative B — All crashes are user-facing errors - -- **Sketch**: catch every Zig trap and present it as a catalog - error. -- **Why rejected**: some traps (memory corruption) leave the runtime - in an unsafe state; continuing to execute is dangerous. The - signal-handler path (Phase 14+) prints a brief message and - exits, which is correct. - -### Alternative C — Build-time elimination of `unreachable` - -- **Sketch**: forbid `unreachable` and require all dispatch - exhaustiveness checks to return `internal_error`. -- **Why rejected**: `unreachable` against a comptime-exhaustive - enum switch is a Zig idiom (the compiler proves it cannot fire). - Forbidding it loses static dispatch checking. The discipline is - to use `unreachable` only when the compiler can prove the path - unreachable; runtime "I think this won't happen" uses - `internal_error`. - -## Consequences - -- **Positive**: every user-visible failure is one of two categories - (catalog error or internal error), and the user can tell from the - category label. cw developers can grep `@panic` and find a - small justified set, not a sprawl. Exit codes carry meaning for - shell integration. -- **Negative**: existing `@panic` and `unreachable` sites need an - audit during task 4.26. Roughly 20-40 sites estimated from - `grep -rn "@panic\|unreachable" src/`; each gets converted or - annotated. -- **Neutral / follow-ups**: - - `internal_error` Code template (`"Internal error: {detail}"`) - is enough for now. When a public issue tracker URL is settled, - a follow-up amends the template to include a "please report" - suffix. - - Phase 14+ signal handler is a separate task; this ADR records - the intent only. - - Stack overflow is platform-specific (the OS may not deliver a - catchable signal). cw v1 sizes thread stacks at 16MB - (per `JVM_TO_ZIG.md` §14 reasoning) which covers the deep - recursion patterns observed in cw v0. - -## References - -- ADR-0018 (Error catalog SSOT — Layer 1 lives there) -- ADR-0009, ADR-0017 (`internal_error` raise sites referenced by - lock and allocator implementations) -- ROADMAP §2 P6 (Error quality is non-negotiable) -- ROADMAP §9.6 task 4.26 (audit `@panic` and `unreachable` along - with the `setErrorFmt` migration) - -## Revision history - -- 2026-05-23: Status: Proposed -> Accepted (initial landing). diff --git a/.dev/decisions/0020_adr_governance.md b/.dev/decisions/0020_adr_governance.md deleted file mode 100644 index 9834b64a6..000000000 --- a/.dev/decisions/0020_adr_governance.md +++ /dev/null @@ -1,97 +0,0 @@ -# 0020 — ADR governance: template extension and "Affected files" discipline - -- **Status**: Accepted -- **Date**: 2026-05-23 -- **Author**: Shota Kudo (drafted with Claude) -- **Tags**: phase-4-entry, adr, governance, template, traceability - -## Context - -ADR-0001 through 0019 landed without an explicit "Affected files" -field. When amending an ADR (e.g., ADR-0018 amendment 2 cascading -through ADR-0009 / 0010 / 0013 / rules / ROADMAP), the developer -must rediscover the dependency graph each time, and quiet leaks -appear at any forgotten downstream site (the rename `tier_d_form` → -`tier_d_` left 7 sites stale after the first commit and only -the second self-review cycle caught them). - -zwasm v1 ADRs and Pollaroid ADR-0044 use an explicit -"Affected files:" section. The pattern is cheap (one section per -ADR) and catches the cascade explicitly at authoring time. - -## Decision - -Every ADR from 0020 onward carries an **Affected files** section -between Consequences and References. The section lists every file -the decision modifies, with one-line annotations: - -```markdown -## Affected files - -- `src/path/to/file.zig` — what changes here -- `.dev/ROADMAP.md` §X.Y — what is amended -- `.claude/rules/foo.md` — new / amended -- `scripts/bar.sh` — new -- (etc.) -``` - -The list is exhaustive at landing time. Amendments append a new -list under `## Revision history` (not edit the original list, so -the historical landing surface stays visible). - -`.dev/decisions/0000_template.md` is extended with the new section. -ADR-0001 through 0019 do **not** get the section added retroactively -(the cost outweighs the benefit; the ADRs are already landed and -their cascades are recorded in subsequent commits). - -INDEX.md auto-generation (a Pollaroid feature) is **deferred** to -Phase 7+ when the ADR count grows past ~25 and manual indexing -becomes friction. At Phase 4 entry, 20-25 ADRs are scannable by -`ls .dev/decisions/`. - -## Alternatives considered - -### Alternative A — Retroactive backfill for ADR-0001 through 0019 - -- **Sketch**: add "Affected files" to all existing ADRs. -- **Why rejected**: the work is large and the cascades are already - documented in commit messages. The benefit is for *future* - amendments, not historical ones. - -### Alternative B — INDEX.md from day one - -- **Sketch**: write a script that scans `.dev/decisions/` and - generates an INDEX.md. -- **Why rejected**: 19 ADRs are scannable; INDEX.md becomes useful - at ~25-30 ADRs. Add the script then. - -## Consequences - -- **Positive**: ADR amendments declare their surface up front. - Self-review of an amendment commit has a checklist (each file in - the Affected list must be touched or have a reason not to). -- **Negative**: one extra section to write per ADR. Mitigated by - the template. -- **Neutral / follow-ups**: INDEX.md auto-gen script lands at - Phase 7+; track in `debt.md` as a low-priority row. - -## Affected files - -- `.dev/decisions/0000_template.md` — add Affected files section -- `.dev/handover.md` — update ADR count and reference V5-1 landing -- `.dev/ROADMAP.md` §A19 — Architecture summary doc principle -- `ARCHITECTURE.md` (new) — 5-minute orientation doc - -## References - -- ADR-0001 through 0019 (the ADRs without "Affected files" that - motivated this decision) -- Pollaroid ADR-0044 (source-scan enforcement, the structural - parent of this idea) -- zwasm v1 ADR template (reference for the "Affected files" - pattern) -- ROADMAP §17 (amendment policy) - -## Revision history - -- 2026-05-23: Status: Proposed -> Accepted (initial landing). diff --git a/.dev/decisions/0021_test_taxonomy.md b/.dev/decisions/0021_test_taxonomy.md deleted file mode 100644 index 3c6d8ef2f..000000000 --- a/.dev/decisions/0021_test_taxonomy.md +++ /dev/null @@ -1,136 +0,0 @@ -# 0021 — Test layer taxonomy (5 layers, Phase 0-11 staged) - -- **Status**: Accepted -- **Date**: 2026-05-23 -- **Author**: Shota Kudo (drafted with Claude) -- **Tags**: phase-4-entry, test, taxonomy, regression, dual-backend - -## Context - -cw v1 reaches Phase 4 with three different test surfaces already in -flight: inline Zig `test "..."` blocks per source file, e2e shell -scripts under `test/e2e/`, and a future `Evaluator.compare()` -runner that ADR-0005 marks as CI-mandatory from Phase 4. Without -a documented taxonomy: - -- New tests get written wherever the author finds first; the - "where does this test belong" question becomes a recurring - review topic. -- Phase-by-phase test growth has no shape — at Phase 11 (upstream - port) the testing layer's relationship to inline tests is - unclear. -- The differential layer (TreeWalk vs VM) sits ambiguously between - unit and e2e. - -A long survey (private/research-2026-05-23/TEST_STRATEGY.md) listed -13 possible layers (unit / e2e / integration / differential / -golden / bench-quick / bench-full / conformance / property / fuzz / -concurrency / memory-leak / Wasm component). At Phase 4 entry that -list is overengineered. - -## Decision - -cw v1 commits to **5 test layers** at Phase 4 entry, with the -remaining 8 layers deferred and named explicitly so they can be -added without re-opening this decision. - -| # | Layer | Location | Tool | Phase opens | Cadence | -|---|--------------|-------------------------|----------------------------------|-------------|------------------------------| -| 1 | Unit | `src/**/*.zig` (inline) | `zig build test` | Phase 0 | every commit | -| 2 | E2E (CLI) | `test/e2e/*.sh` | bash + `cljw` | Phase 2 | every commit | -| 3 | Differential | `test/diff/` | `Evaluator.compare()` (ADR-0022) | Phase 4 | every commit | -| 4 | Bench quick | `bench/quick.sh` | bash + `cljw` | Phase 4 | every commit (informational) | -| 5 | Conformance | `test/clj/` | `clojure.test` | Phase 11 | every PR | - -### Phase-by-phase test growth (target, not KPI) - -| Phase | cumulative tests | cumulative test files | new layer added | -|-------|-----------------:|----------------------:|------------------------------------------------| -| 0-3 | ~80 | ~25 | Layer 1, 2 | -| 4 | ~120 | ~32 | + Layer 3 (diff), + Layer 4 (bench quick) | -| 5 | ~250 | ~45 | growth in Layer 1 (collection / GC / lazy-seq) | -| 7 | ~400 | ~60 | growth in Layer 1 (multimethod / protocol) | -| 11 | ~3,000 | ~90 | + Layer 5 (Upstream port) | -| 15 | ~3,500 | ~105 | growth in Layer 1 (STM / atom / concurrency) | -| 20 | ~4,000-4,500 | ~120 | (final) | - -These are **targets**, not gates. The Phase 4 gate is "all -landed tests pass on both backends", not "we have at least 120 -tests". - -### Deferred layers (named so the future expansion is mechanical) - -| Future layer | Triggers at | Future ADR | -|-----------------|---------------------------------------------|-------------------------------------------------| -| Integration | Phase 5 (collection + GC + reader together) | (no ADR; just a `test/integration/` directory) | -| Golden snapshot | Phase 7+ | ADR-0026 (future) | -| Property-based | Phase 8+ | (no ADR; `test/prop/` directory) | -| Fuzz | Phase 6+ | (no ADR; `fuzz/` directory; cadence weekly) | -| Memory leak | Phase 5+ (GC activation) | (no ADR; ReleaseSafe + `--gc-stress`) | -| Concurrency | Phase 15+ (STM / atom / lock activation) | (no ADR; runs in Layer 1 with `--test-iter`) | -| Bench full | Phase 8 (lock baseline; landed at row 8.2) | ADR-0044 (was 0027 pre-audit; renumbered) | -| Wasm component | Phase 14+ | (no ADR; runs in Layer 2 with Wasm Pod fixture) | - -### Layer 3 (Differential) wiring - -ADR-0022 specifies the implementation contract for the -differential layer. This ADR (0021) only places it in the taxonomy; -the runner / cases.yaml shape lives in 0022. - -### Layer 5 (Conformance) deferral - -`test/clj/skip_taxonomy.yaml` (the YAML that maps every upstream -test file to a tier classification) is **not** drafted at Phase 4 -entry. ADR-0025 (future, Phase 11 entry) finalises that schema. - -## Alternatives considered - -### Alternative A — 13 layers from day 1 - -- **Sketch**: open `test/{integration,golden,prop,fuzz,memleak,concurrency,bench-full,wasm}/` at Phase 4 entry with placeholder READMEs. -- **Why rejected**: 8 empty directories add maintenance friction - without value before Phase 5-11. - -### Alternative B — 3 layers (unit / e2e / conformance) - -- **Sketch**: merge differential into e2e, drop bench-quick. -- **Why rejected**: ADR-0005 has differential as a separate - surface with `Evaluator.compare()` — collapsing into e2e loses - the per-backend visibility. bench-quick is Phase 4 task 4.0, - also a separate surface. - -## Consequences - -- **Positive**: the "which layer is this test" question has 5 - answers at Phase 4, and the Phase 11+ growth is mapped. Reviewers - can flag a misplaced test against the table. -- **Negative**: future layer additions must respect the deferred - layer naming so subsequent ADRs can refer to them. -- **Neutral / follow-ups**: `test/README.md` lists the 5 layers - with one-line guidance for "which to use when". A new - `.claude/rules/test_taxonomy.md` enforces the choice when an - agent writes a test. - -## Affected files - -- `.dev/ROADMAP.md` §11 — rewrite per 5-layer taxonomy + Phase - growth table -- `.dev/ROADMAP.md` §11.3 — fix Phase 8 → Phase 4 (ADR-0005 has - this from Phase 4) -- `.dev/ROADMAP.md` §11.6 row 6 — same fix -- `test/README.md` (new) — 5 layers with guidance -- `.claude/rules/test_taxonomy.md` (new, in follow-up commit) — - enforcement of placement - -## References - -- ADR-0005 (Dual-backend differential — Layer 3 owner) -- ADR-0022 (Differential test wiring — Layer 3 implementation) -- ADR-0025, ADR-0026, ADR-0044 (bench schema; was ADR-0027 pre-audit) -- `private/research-2026-05-23/TEST_STRATEGY.md` (working notes) -- `private/research-2026-05-23/REVIEW_TEST_STRATEGY.md` (review - that triggered the 13 → 5 reduction) - -## Revision history - -- 2026-05-23: Status: Proposed -> Accepted (initial landing). diff --git a/.dev/decisions/0022_differential_wiring.md b/.dev/decisions/0022_differential_wiring.md deleted file mode 100644 index a1e591e38..000000000 --- a/.dev/decisions/0022_differential_wiring.md +++ /dev/null @@ -1,175 +0,0 @@ -# 0022 — Differential test wiring (TreeWalk + VM, Phase 17+ JIT) - -- **Status**: Accepted -- **Date**: 2026-05-23 -- **Author**: Shota Kudo (drafted with Claude) -- **Tags**: phase-4-entry, test, differential, dual-backend, runner - -## Context - -ADR-0005 declares "dual-backend differential testing is the -oracle, CI mandatory from Phase 4". ADR-0005 sets the direction but -does not specify *how* the runner is shaped, where cases live, or -how a third backend (JIT) extends the layout in Phase 17. - -ADR-0021 places the differential layer at slot 3 of the test -taxonomy but defers the implementation contract to this ADR. - -## Decision - -`test/diff/` is the differential layer's home. - -### Files - -``` -test/diff/ - README.md ← V5-1 landing (this commit) - runner.zig ← Phase 4 task 4.10 landing - cases.yaml ← Phase 4 task 4.10 landing - fixtures/ ← Phase 4 task 4.10 landing (per-case Clojure sources) -``` - -### runner.zig shape - -```zig -//! test/diff/runner.zig — differential runner per ADR-0022. -//! -//! Pulls cases.yaml at comptime via @embedFile, dispatches each -//! case to TreeWalk and VM, compares Value bit-for-bit. - -const std = @import("std"); -const testing = std.testing; -const Evaluator = @import("../../src/eval/evaluator.zig").Evaluator; - -const cases_yaml = @embedFile("cases.yaml"); - -test "differential cases" { - var arena = std.heap.ArenaAllocator.init(testing.allocator); - defer arena.deinit(); - const alloc = arena.allocator(); - - const cases = try parseCases(alloc, cases_yaml); - - var rt = try Runtime.init(alloc); - defer rt.deinit(); - - for (cases) |case| { - if (case.skip_reason) |reason| { - std.debug.print("[skip] {s}: {s}\n", .{ case.name, reason }); - continue; - } - const result = try Evaluator.compare(&rt, rt.root_env, case.src); - if (!result.equal) { - std.debug.print( - "DIFF: {s}\n tree_walk = {f}\n vm = {f}\n", - .{ case.name, result.tree_walk, result.vm }, - ); - return error.DifferentialMismatch; - } - } -} -``` - -`parseCases` is a thin YAML reader (no full YAML spec; just the -subset needed for cases.yaml). - -### cases.yaml schema - -```yaml -cases: - - name: arith_add_basic - src: "(+ 1 2)" - expected: 3 - skip_reason: null - - - name: closure_capture_local - src: |- - ((let* [x 1] (fn* [y] (+ x y))) 2) - expected: 3 - skip_reason: null - - - name: try_catch_returns_handler_value - src: |- - (try* (throw (ex-info "boom" {})) (catch ExceptionInfo e :caught)) - expected: ":caught" - skip_reason: null - - - name: lazy_seq_realisation_order - src: "(take 3 (map inc (range)))" - expected: "(1 2 3)" - skip_reason: "Layer 1 covers this; differential not needed for Phase 4." -``` - -The `expected` field is informational (the runner does not assert -against it; the runner asserts TreeWalk == VM). It documents what -the case should evaluate to so a reviewer can sanity-check. - -### Phase 17 extension to 3-way - -When JIT lands: - -```zig -const result = try Evaluator.compareThree(&rt, rt.root_env, case.src); -if (!(result.tw_eq_vm and result.tw_eq_jit)) { - return error.DifferentialMismatch; -} -``` - -The case file does not change; only `runner.zig` adopts the 3-way -helper. ADR-0005 amendment at Phase 17 entry codifies this. - -### Wiring into `test/run_all.sh` - -`test/run_all.sh` calls `zig build test`, which discovers the -inline `test "differential cases"` block in `runner.zig` along with -the unit tests. No separate runner binary. - -## Alternatives considered - -### Alternative A — Cases written as Zig `test "..."` blocks - -- **Sketch**: each case is a `test "name"` block in `runner.zig`. -- **Why rejected**: cases.yaml is easier to edit by people who - don't write Zig, and a `skip_reason` per case is simpler to - manage than a Zig `if` guard. The case list grows to dozens in - Phase 4-5 and hundreds by Phase 11. - -### Alternative B — Bash script wrapper around `cljw` - -- **Sketch**: `test/diff/runner.sh` invokes `cljw -e '...'` twice - and diffs stdout. -- **Why rejected**: bytes-level stdout diff is noisy (Value - formatting differences, source-location lines). The - `Evaluator.compare()` API at the Zig level returns a structured - `result` and `.equal` boolean, which is the right granularity. - -## Consequences - -- **Positive**: differential cases live in one YAML, editable - without Zig knowledge. Phase 17 extension to 3-way is a - one-line change in `runner.zig`. The runner is `zig build test` - native, no extra wiring. -- **Negative**: a custom YAML reader in `parseCases`. Acceptable - scope (case file format is small), but adds ~50 lines to - `runner.zig`. -- **Neutral / follow-ups**: when fixtures get reused (Phase 5+), - `test/diff/fixtures/*.clj` files can be referenced by - `case.fixture: path/to/file.clj` instead of inline `src`. - -## Affected files - -- `test/diff/README.md` (new, this commit) -- `test/diff/runner.zig` (Phase 4 task 4.10) -- `test/diff/cases.yaml` (Phase 4 task 4.10) -- `test/diff/fixtures/` (Phase 4 task 4.10, populated as cases grow) -- `src/eval/evaluator.zig` (Phase 4 task 4.10, `compare()` API) - -## References - -- ADR-0005 (Dual-backend differential oracle, parent decision) -- ADR-0021 (Test taxonomy, Layer 3) -- ROADMAP §9.6 task 4.10 (`Evaluator.compare()`) - -## Revision history - -- 2026-05-23: Status: Proposed -> Accepted (initial landing). diff --git a/.dev/decisions/0023_comptime_stub_pattern.md b/.dev/decisions/0023_comptime_stub_pattern.md deleted file mode 100644 index ca103e600..000000000 --- a/.dev/decisions/0023_comptime_stub_pattern.md +++ /dev/null @@ -1,170 +0,0 @@ -# 0023 — Comptime conditional import + stub struct pattern - -- **Status**: Accepted -- **Date**: 2026-05-23 -- **Author**: Shota Kudo (drafted with Claude) -- **Tags**: phase-4-entry, zig, comptime, stub, phase-staging - -## Context - -cw v1 stages features across phases. At Phase 4 entry the mark-sweep -GC heap (ADR-0017) is reserved as a packed bit slot but not -allocated; the Object header lock (ADR-0009) is reserved similarly; -STM (ADR-0010) is declarations only; TypeDescriptor method tables -(ADR-0007 / 0008) are skeleton. - -When source needs to reference these features at compile time, two -forces collide: - -1. **No-op stub is forbidden** (`.claude/rules/no_op_stub_forbidden.md`). - A function that pretends to be the real implementation but does - nothing is the worst case. -2. **Compile errors slow the loop** when a feature is "obviously - not implemented yet". Catching it as a catalog error - (`feature_not_supported`) is the runtime-error path; the - compile-time path needs a parallel form. - -zwasm v1 uses `opaque {}` and `comptime` conditional imports to -maintain a build-time switch without inflating the symbol table. - -## Decision - -cw v1 adopts two complementary patterns for phase-staged code: - -### Pattern A — Comptime conditional import - -When a higher-zone import depends on a phase activation: - -```zig -const build_config = @import("build_options"); - -pub const GcHeap = if (build_config.phase_at_least_5) - @import("runtime/gc/mark_sweep.zig").GcHeap -else - @import("runtime/gc/stub.zig").GcHeap; -``` - -The compile decision is at `build.zig`, exposed as a comptime bool -in `build_options`. The phase boundary determines which import is -real and which is the stub. - -### Pattern B — Stub struct - -The stub module presents the same public API as the real one, but -every function returns `error.NotImplementedYet`: - -```zig -//! runtime/gc/stub.zig — Phase 4 stub for runtime/gc/mark_sweep.zig. -//! Activated at Phase 5. - -const std = @import("std"); - -pub const GcHeap = struct { - pub fn init(_: std.mem.Allocator) GcHeap { - return .{}; - } - - pub fn deinit(_: *GcHeap) void {} - - /// Allocation is the only function the rest of the runtime - /// reaches. Returning error.NotImplementedYet at this layer - /// surfaces through the top-level catch (ADR-0019) as a - /// catalog `internal_error` (since user input does not reach - /// this path directly at Phase 4 — only Phase 5 builtins - /// would). - pub fn alloc(_: *GcHeap, comptime _: type) error{NotImplementedYet}!*anyopaque { - return error.NotImplementedYet; - } -}; -``` - -Pattern B is **distinct from a no-op stub** because: - -1. The function returns an explicit `error.NotImplementedYet`, - which propagates as a `ClojureWasmError` to the catalog at the - call site. The user does not see this Code directly (`alloc` - is internal); the dispatching site translates it. -2. The module file carries `//! Phase N stub for ...` in the - doc-comment header. -3. The real module (`mark_sweep.zig`) has the same public API, - so swapping at Phase 5 changes only the comptime import. - -## Alternatives considered - -### Alternative A — `@panic("Phase 5 only")` in every stub function - -- **Sketch**: stub bodies are `@panic`. -- **Why rejected**: ADR-0019 forbids `@panic` in any path - reachable by user input. Even though stub `alloc` is internal, - Phase 5 builtins like `(vec ...)` would route there, making it - a user-reachable panic. `error.NotImplementedYet` keeps the - return as a catalog error. - -### Alternative B — `opaque {}` for everything - -- **Sketch**: every staged type is `opaque {}` in the stub module. -- **Why rejected**: `opaque` prevents the stub from carrying any - field at all, which forces the real-module API to be reshaped - to accommodate. A stub `struct { _: u8 = 0 }` keeps the API - parallel. - -### Alternative C — Build flag `-Dno_gc` and runtime checks - -- **Sketch**: leave `mark_sweep.zig` always imported, but the GC - is a no-op at Phase 4. -- **Why rejected**: this is the no-op stub pattern Shota - explicitly forbade. Even if the no-op only allocates from the - arena instead of the GC heap, the *signal* that the GC is not - running gets lost. - -## Consequences - -- **Positive**: feature staging is mechanical (one comptime - import, one stub module). The real module lands as a swap; - callers do not change. The pattern explicitly distinguishes - "not implemented yet" (Pattern B) from "no-op pretending to - work" (forbidden). -- **Negative**: each staged feature carries a parallel - `stub.zig` file. The cost is one ~30-line file per feature - (mark_sweep_stub, stm_stub, locking_stub, ...). Acceptable. -- **Neutral / follow-ups**: `build.zig` exposes `phase_at_least_N` - comptime bools for N in {4, 5, 7, 11, 14, 15, 17}. Each phase - boundary flips one bool from `false` to `true`. The list of - bools is the phase progression manifest. - -## Affected files - -- `build.zig` (Phase 4 task TBD) — add `phase_at_least_5` / - `phase_at_least_7` / ... comptime bools to `build_options` -- `src/runtime/gc/stub.zig` (Phase 4 task 4.17 area) — example - stub for ADR-0017 GcHeap -- `src/runtime/lock/stub.zig` (Phase 4 task 4.19 area) — stub for - ADR-0009 lock activation -- `src/runtime/stm/stub.zig` (Phase 4 task TBD) — stub for - ADR-0010 STM -- `.claude/rules/no_op_stub_forbidden.md` — cross-reference this - ADR as the canonical Pattern B example - -## References - -- ADR-0019 (Crash policy — Pattern B is Layer 2 in the crash - taxonomy) -- ADR-0009 / ADR-0010 / ADR-0017 (phase-staged features that - benefit from this pattern) -- `.claude/rules/no_op_stub_forbidden.md` -- zwasm v1 `src/wasm_validator_stub.zig` (precedent) - -## Revision history - -- 2026-05-23: Status: Proposed -> Accepted (initial landing). -- 2026-06-15 (ADR-0142 / D-440 R5): the `build_options.phase_at_least_N` - phase-activation manifest (Pattern A) is **RETIRED**. The flags guarded zero - live code paths in practice — every comptime-stub-staged feature shipped its - real impl directly rather than behind `if (build_options.phase_at_least_N)`, - so the flags were defined + unit-tested but never gated source. The §9 - completion-grade reframe (ADR-0142) replaced the phase-number model with gap - areas, making the per-phase flag meaningless. Removed from `build.zig` + - `src/main.zig`. The comptime-stub *pattern* itself (real-vs-stub module - selection without a runtime branch) remains valid; only the phase-numbered - flag manifest is gone. A future capability-gated build flag (if needed) would - be named by capability, not phase number (cf. D-443). diff --git a/.dev/decisions/0024_scan_and_runner.md b/.dev/decisions/0024_scan_and_runner.md deleted file mode 100644 index 2803f8324..000000000 --- a/.dev/decisions/0024_scan_and_runner.md +++ /dev/null @@ -1,152 +0,0 @@ -# 0024 — Source-scan framework + run_step runner pattern - -- **Status**: Accepted -- **Date**: 2026-05-23 -- **Author**: Shota Kudo (drafted with Claude) -- **Tags**: phase-4-entry, gate, scan, runner, bash - -## Context - -Two related concerns surface at Phase 4 entry: - -1. **Source-scan gates**. cw v1 has 4 scripts that grep `src/` for - forbidden patterns (`check_compat_tiers_sync.sh`, - `check_no_op_stub.sh`, `check_tier_d_error_msg.sh`, - `file_size_check.sh`). Each was written independently with - slightly different output formats and informational-vs-gate - handling. ADR-0018 and ADR-0019 will add two more - (`scan_catalog_only.sh`, `scan_panic_audit.sh`). Without a - shared library, the divergence compounds. - -2. **Test runner dispatch**. `test/run_all.sh` is the single - entry point but currently shells out to each gate in series - with no summary, no skip / only flags, and no per-step timing. - zwasm v1's `run_step` pattern shows the value of these - features at modest cost. - -The two concerns share `bash` as their substrate and benefit from -landing together (the scan scripts use the same logging helpers -that the runner uses). - -## Decision - -### Part 1 — `scripts/scan_lib.sh` - -A bash library that every source-scan script sources. Provides: - -- `scan_mode()` — returns `informational` (default) or `gate` - based on `$CLJW_SCAN_MODE` env var. Phase 5+ flips the default - to `gate`. -- `scan_count(pattern, dir)` — counts pattern hits. -- `scan_match_in_dir(pattern, dir)` — lists hits with file:line. -- `scan_report(gate_name, hits, threshold)` — prints - `[] PASS/FAIL: N hits (threshold T)` and returns 0 or 1. -- `scan_section(title)` — prints a section header for readability. - -Existing 4 scripts and new 2 scripts (`scan_catalog_only.sh` for -ADR-0018, `scan_panic_audit.sh` for ADR-0019) source the lib. - -### Part 2 — `test/run_all.sh` run_step pattern - -The runner exposes: - -- `run_step [optional]` — runs a step, captures - pass / fail / timing, prints `[pass] name (Ns)` or `[fail] - name (exit N)`. The `optional` flag marks the step as - non-blocking (recorded but does not fail the overall run). -- `--list` — lists step names without running. -- `--skip [,]` — skips named steps. -- `--only [,]` — runs only named steps. -- `print_summary` — final tally (passed / failed / failed - optional) and exits non-zero if any non-optional step failed. - -Step name convention: `_` (`zig_build_test`, -`zone_check`, `e2e_phase3_cli`, `scan_catalog_only`). The naming -is grep-friendly and matches the Layer taxonomy from ADR-0021 -(`unit` → Layer 1, `e2e_*` → Layer 2, `diff` → Layer 3, -`bench_quick` → Layer 4). - -### Naming alignment - -- `scripts/check_compat_tiers_sync.sh` → keeps existing name - (legacy, sourced from `scan_lib.sh` after refactor). -- `scripts/check_no_op_stub.sh` → same. -- `scripts/check_tier_d_error_msg.sh` → same. -- `scripts/file_size_check.sh` → same. -- `scripts/scan_catalog_only.sh` (new, ADR-0018) — uses - `scan_lib.sh`. -- `scripts/scan_panic_audit.sh` (new, ADR-0019) — uses - `scan_lib.sh`. - -The `check_` prefix is grandfathered for the four existing -scripts to avoid churn; new scans use `scan_` per Pollaroid -naming. - -## Alternatives considered - -### Alternative A — Keep scans independent - -- **Sketch**: each check_*.sh has its own helpers. -- **Why rejected**: divergence already started; the rename - `tier_d_form` → `tier_d_` in ADR-0018 amendment 2 had to - touch 4 scripts because the lookup logic was duplicated. - -### Alternative B — Make every check a Zig program - -- **Sketch**: `scripts/scan_lib.zig` and per-scan Zig binaries. -- **Why rejected**: bash + grep handles the surface in ~150 - lines. A Zig program adds build dependency, executable juggling, - and gives no expressive gain. - -### Alternative C — Use `pre-commit` framework - -- **Sketch**: adopt the `pre-commit` tool (Python-based). -- **Why rejected**: adds a runtime dependency; cw v1 uses - `.githooks/` + bash already. - -## Consequences - -- **Positive**: shared helpers across 6 scan scripts. Consistent - output format (`[] PASS/FAIL: N hits`). New scans land - with ~10 lines of bash on top of the lib. `test/run_all.sh` - gains skip / only / summary without a Zig rewrite. -- **Negative**: refactoring the 4 existing scripts has a small - cost (one-time grep replacement). Bounded. -- **Neutral / follow-ups**: Phase 5+ flips - `$CLJW_SCAN_MODE=gate` (default at that phase boundary) so - scan scripts begin to gate. Each script's - `informational → gate` transition is decided per scan - (ADR-0013 / 0016 / 0018 / 0019 etc. carry the phase). - -## Affected files - -Phase 4 entry (this commit batch): - -- `scripts/scan_lib.sh` (new) -- `scripts/scan_catalog_only.sh` (new, ADR-0018 enforcement) -- `scripts/scan_panic_audit.sh` (new, ADR-0019 enforcement) -- `test/run_all.sh` (refactor: run_step pattern + summary) - -Phase 5 entry (deferred — these scripts already work in standalone -informational mode, and their output format alignment with -`scan_lib.sh` is a low-priority polish that can ride with the -informational-to-gate mode flip at Phase 5): - -- `scripts/check_compat_tiers_sync.sh` (refactor: source lib) -- `scripts/check_no_op_stub.sh` (refactor: source lib) -- `scripts/check_tier_d_error_msg.sh` (refactor: source lib) -- `scripts/file_size_check.sh` (refactor: source lib) - -## References - -- ADR-0013 (Tier D — check_tier_d_error_msg consumer) -- ADR-0016 (File size — file_size_check consumer) -- ADR-0018 (Error catalog — scan_catalog_only consumer) -- ADR-0019 (Crash policy — scan_panic_audit consumer) -- zwasm v1 `run_step` pattern (precedent) -- Pollaroid ADR-0049 / 0051 / 0089 sister scanner pattern - (precedent) - -## Revision history - -- 2026-05-23: Status: Proposed -> Accepted (initial landing). diff --git a/.dev/decisions/0025_chapter_archive_boundary.md b/.dev/decisions/0025_chapter_archive_boundary.md deleted file mode 100644 index c557b2f2c..000000000 --- a/.dev/decisions/0025_chapter_archive_boundary.md +++ /dev/null @@ -1,169 +0,0 @@ -# 0025 — Big-bang chapter regeneration boundary at Phase-4 critical-path close - -- **Status**: Accepted -- **Date**: 2026-05-23 -- **Author**: Shota Kudo (drafted with Claude, autonomous-loop self-accept) -- **Tags**: phase-4-exit, documentation, learn_clojurewasm, learn_zig, archive, big-bang - -## Context - -The `code_learning_doc` skill (`.claude/skills/code_learning_doc/SKILL.md` -L177-197) already establishes a "big-bang regeneration policy" — when -the codebase undergoes a significant design transition, the existing -chapter sequence moves to `docs/ja/archive/learn_clojurewasm_v1_/` -and a new chapter sequence is generated in one batch covering the new -design. - -Two facts make Phase 4 critical-path close the natural boundary: - -1. **The Phase 1-3 chapter set (0001-0020) covers the pre-VM design.** - Phase 4 introduces a second backend (VM dispatch loop, compiled - bytecode, dual-backend driver, differential oracle) and rewrites - several Phase 1-3 surfaces (Function carries `bytecode: ?*const - BytecodeChunk`, vtable gains `evalChunk`, callFunction routes - through it, etc.). Writing Phase-4 chapters as continuations of - 0001-0020 would force readers to absorb two designs in series; a - regenerated set anchored on the post-VM shape is cleaner. - -2. **The Phase 4 cleanup wave (4.13-4.24) accumulated 11 source - commits without per-task notes.** The autonomous loop prioritised - forward progress over the chapter cadence. Hot context for those - commits is lost; retroactive chapter writing from `git log` would - produce low-quality material (per - `code_learning_doc/SKILL.md` L173-175: "Writing the chapter at - the *end* of the phase from `git log` only. By that point the - why-not's are forgotten."). - -The `learn_zig` companion (`docs/ja/learn_zig/`, 30 chapters / 2400 -lines) was scoped explicitly as the Phase-1-3 code-reading reader -("ROADMAP / phase progression からは独立した副読本" — see -`docs/ja/learn_zig/README.md` L8-10). Phase 4 introduces a substantial -amount of Zig material (vtable hooks with `*const anyopaque`, -comptime branching on build_options, `std.atomic.Mutex`, `packed -struct(u32)` for gc_and_lock, etc.) that the existing 30 chapters do -not cover. A regeneration boundary aligns both reader sets. - -## Decision - -Execute the big-bang boundary now (Phase 4 critical-path close at -commit 2868e21): - -1. **Archive existing chapters in place.** - - `docs/ja/learn_clojurewasm/0001-0020 + README.md` → - `docs/ja/archive/learn_clojurewasm_v1_phase1to3/` (rename - unchanged). - - `docs/ja/learn_zig/README.md` → - `docs/ja/archive/learn_zig_v1/README.md`. - -2. **Put the chapter cadence into dormant mode.** - - `code_learning_doc/SKILL.md` two-cadence rule's per-concept - chapter half is suspended. The per-task notes half continues - (`private/notes/-.md` from hot context — these - remain load-bearing as Step 7 input). - - `scripts/check_learning_doc.sh` becomes a no-op gate while - dormant (returns success without enforcing pairing). The Skill - and the gate together carry an explicit "dormant" marker so - re-activation is a one-line edit. - - CLAUDE.md `code_learning_doc` skill reference updated to note - dormancy + pointer to this ADR. - -3. **Re-resumption conditions.** - - A future ADR (`ADR-NNNN — Resume chapter sequence at Phase - entry`) explicitly switches the cadence back on. The - trigger is at the project owner's discretion; the most natural - candidates are Phase 4 closure (4.13-4.26.f complete) or Phase - 5 entry (mark-sweep GC + TypeDescriptor activation). - - At resumption, the new chapter sequence starts at `0001` of - the regenerated set under `docs/ja/learn_clojurewasm/`, using - the surviving per-task notes as Step-7 input. Archived - chapters stay read-only references for "how the design used - to look". - -4. **per-task notes during dormancy.** - - Continue writing per-task notes under `private/notes/` (Step 7 - of the per-task TDD loop). They are not gated; they are the - hot-context capture that the eventual regeneration consumes. - - Notes already written (4.6 / 4.7.a / 4.7.b / 4.7.c) are - preserved. Tasks 4.8-4.24 have no per-task notes; this is - accepted as a one-time gap and not retroactively filled. - -## Alternatives considered - -### Alternative A — Continue the chapter cadence as 0021 forward - -- **Sketch**: Write chapters 0021-0024 for tasks 4.6-4.12 right now - from `git log` + ADRs + source, mark 4.13-4.24 as "skeleton wave, - no chapter". -- **Why rejected**: the hot context for 4.6-4.12 has largely - decayed (the autonomous loop ran straight through them without - pausing). Retroactive chapter quality would breach the skill's - "do not write from `git log` alone" guidance. The result would - read like a session log, not a textbook. - -### Alternative B — Defer the boundary until Phase 5 entry - -- **Sketch**: keep accumulating chapters in dormant mode through - Phase 4 closure, then archive at Phase 5 entry. -- **Why rejected**: deferring the archive while the cadence is - effectively dormant accumulates more debt without offsetting - benefit. The trigger is "design transition completed", which - Phase 4 critical-path close has already crossed. - -### Alternative C — Archive only `learn_clojurewasm`, keep `learn_zig` live - -- **Sketch**: `learn_zig` is positioned as ROADMAP-independent, so - it could continue. -- **Why rejected**: the Phase-4 Zig material list above (vtable - with `*const anyopaque`, comptime build_options branching, - `std.atomic.Mutex`, packed struct gc_and_lock, ADR-0023 comptime - stub patterns) is substantial and would force `learn_zig` to - either grow uneven (Phase-1-3 chapters + Phase-4 chapters with - different editorial conventions) or stay stale. Archiving and - regenerating both keeps the editorial convention coherent. - -## Consequences - -- **Positive**: chapter cadence stops accumulating debt during - dormancy. The eventual regenerated set covers the post-VM design - coherently. Both reader sets archive at the same boundary. -- **Negative**: tasks 4.8-4.24 have no per-task notes and the - regeneration will lean on `git log` + ADRs + source for those. - Mitigated by the ADR + commit-message density (each Phase-4 - commit has a multi-paragraph rationale; ADR-0006 a1 / ADR-0012 a1 - cover the load-bearing decisions). -- **Neutral / follow-ups**: when the resumption ADR fires, it - references this ADR and lays out the new chapter scope. - -## Affected files - -- `docs/ja/learn_clojurewasm/0001-0020 + README.md` → archive -- `docs/ja/learn_zig/README.md` → archive -- `.claude/skills/code_learning_doc/SKILL.md` (dormancy marker) -- `scripts/check_learning_doc.sh` (no-op while dormant) -- `CLAUDE.md` (skill reference annotated with dormancy) -- `.dev/handover.md` (cleanup wave status) - -## References - -- `code_learning_doc/SKILL.md` L177-197 — the policy this ADR - invokes -- `private/notes/4.6.md` / `4.7a.md` / `4.7b.md` / `4.7c.md` — the - surviving per-task notes that survive into regeneration -- ADR-0006 amendment 1 + ADR-0012 amendment 1 (commit bbae2e0) — - example of the in-loop ADR amendment that the regenerated - chapters will need to cover - -## Revision history - -- 2026-05-23: Status: Proposed → Accepted (autonomous-loop - self-accept per CLAUDE.md § "ADR-level designs are handled - inline"). -- 2026-05-24 (clarification): **Resumption is user-triggered - only.** Per `project_facts.md` F-007, the user does not want - the chapter cadence to resume unprompted. The autonomous loop - must **not** propose, draft, or re-activate the cadence on its - own. The "Re-resumption conditions" listed in §3 remain - informational (signals the user may consider), but the trigger - itself is a user-only action. Until that signal arrives, the - cadence stays dormant and the existing 20-chapter set + - learn_zig companion stay read-only under `docs/ja/archive/`. diff --git a/.dev/decisions/0026_phase5_entry_scope.md b/.dev/decisions/0026_phase5_entry_scope.md deleted file mode 100644 index 148b27ff4..000000000 --- a/.dev/decisions/0026_phase5_entry_scope.md +++ /dev/null @@ -1,154 +0,0 @@ -# 0026 — Phase 5 entry scope decree (cleanup-wave audit outcome) - -- **Status**: Accepted -- **Date**: 2026-05-24 -- **Author**: Shota Kudo (drafted with Claude, autonomous-loop self-accept per CLAUDE.md § "ADR-level designs are handled inline" after Devil's-advocate subagent review) -- **Tags**: phase-5-entry, cleanup-audit, activation-ordering, D-028 - -## Context - -Phase 4 closed at commit 1f2406a with all §9.6 rows `[x]`. The Phase 5 entry inherits eight Phase-4 skeleton landings (rows 4.13 / 4.17 / 4.18 / 4.19 / 4.20 / 4.22 / 4.23 / 4.24 / 4.25 per debt D-028) that explicitly deferred behaviour to Phase 5. §9.7 task 5.0 (D-028 cleanup-wave audit) ran the survey at `private/notes/phase5-skeleton-audit.md` (676 lines, commit a891c56 task open). This ADR encodes the audit's two load-bearing conclusions — the activation classification table and the critical-path ordering — so §9.7 tasks 5.1-5.16 execute against a fixed surface map without re-deriving the classification. - -The audit ran under three constraints: F-002 (finished-form wins over diff size), F-003 (decision-deferral on structural plans), and `no_copy_from_v1` (cw v0 references describe shape, not a port target). - -## Decision - -### 1. Classification of the eight Phase-4 skeletons - -| Row | File | Audit verdict | Activation owner | -|------|-------------------------------------|--------------------------------|----------------------------------| -| 4.13 | `runtime/io_interface.zig` | matches FF | (deferred Phase 14, ADR-0015 a2) | -| 4.17 | `runtime/type_descriptor.zig` | needs restructure | §9.7 row 5.11 | -| 4.18 | `runtime/protocol.zig` | needs restructure | Phase 7 (D-040) | -| 4.19 | `runtime/value.zig::ObjectHeader` | matches FF | §9.7 row 5.3 | -| 4.20 | `runtime/host/_host_api.zig` | matches FF | Phase 6 (host class wave) | -| 4.22 | `runtime/binding_stack.zig` | reverted (6a48e90) — terminal | (none) | -| 4.23 | `runtime/numeric/big_int.zig` | needs restructure | §9.7 rows 5.2 + 5.9 | -| 4.24 | `runtime/lazy_seq.zig` | needs restructure | §9.7 row 5.7 | -| 4.25 | `runtime/dispatch/method_table.zig` | matches FF | Phase 7 (D-040) | - -"matches FF" = the current skeleton shape is what the activation phase will use; activation is body-only work. "needs restructure" = activation touches the struct shape (field add/remove/move), expected and accepted under F-002. - -### 2. Critical-path activations for Phase 5 exit - -The §9.7 row 5.16 exit smoke (`(get {:a 1} :a)` / `(reduce + (range 1e6))` / `(/ 1 3)` / `(* 9223372036854775807 2)` / `(deftype Point [x y])`) depends on these activations in order: 5.2 (`value.zig` split with F-004 layout), 5.3 (mark-sweep GC), 5.4-5.6 (persistent Vector / HashMap / HashSet), 5.7 (lazy_seq `force()`), 5.8 (List / Cons), 5.9-5.10 (numeric tower), 5.11 (TypeDescriptor), 5.12 (deftype / defrecord / reify), 5.15 (`build_options.phase_at_least_5 = true` flip — mechanical after 5.12, sequential per §9.7.x dependency graph). 5.13 / 5.14 are doc / split clean-ups that ride parallel. - -The 🔒 OrbStack gate lock applies per-row: every §9.7 task must pass Mac and Ubuntu x86_64 gates at its own Step 5, not only the exit-smoke row 5.16. The framework finalisation (per ROADMAP §A18) lands when the per-row check_*.sh gates flip from informational to blocking at 5.15. - -### 3. Forward-leaning inputs to 5.1 (NOT settled here) - -The audit surfaced eight constraints that ADR-0027 (NaN-box 第二世代) + ADR-0028 (mark-sweep GC) co-issue at §9.7 task 5.1 must respect. They live as the survey's "5.1 input bullets" section at `private/notes/phase5-skeleton-audit.md`. 5.1's Step 0/1 reading list already pulls the survey per CLAUDE.md § Step 1a "Phase reading list"; the bullets are quoted into 5.1's ADR Context section at the moment they bind, not pre-committed here. - -This split follows the Devil's-advocate Alt 1 recommendation (below): the *verdict table* in §1 commits Phase 5's surface map and needs tracked form; the *constraint bullets* are inputs and live more naturally in the consuming ADR's Context. F-003 (decision-deferral) also covers this: ADR-0026 should not pre-commit decisions the survey only surfaces. - -## Alternatives considered - -A fresh-context `general-purpose` subagent was forked with the F-001..F-008 envelope as Devil's-advocate per CLAUDE.md § "ADR-level designs are handled inline". The output is reflected here verbatim; the main loop applied Alt 1 (recommendation) with §3 reduced to a pointer instead of removed entirely (the inputs are load-bearing enough to warrant a tracking line; the bullets themselves stay in the survey). - -### Subagent summary judgement - -The ADR is **substantively correct** but **structurally awkward**: it carries 8 detailed constraint bullets that the §9.7 row 5.1 ADR cluster will *re-quote* (the ADR itself acknowledges this duplication under Consequences/Negative). The audit verdict table is the load-bearing artifact; the 8 bullets are forward-leaning preparation material that arguably belongs in 5.1's "Inputs from 5.0 audit" section, not in ADR-0026. This is the central tension the alternatives explore. - -A possibly-more-serious concern: **ADR-0026's "Decision §4 Constraints this ADR does NOT settle" overlaps F-003 (decision-deferral) to the point of restating principle as decision. That section reads like a meta-disclaimer rather than a load-bearing choice.** - -### Alt 1 — Smallest-diff: Strip §3 + §4 to a pointer; keep only the verdict table - -Sketch: Cut ADR-0026 down to the §1 classification table + the §2 critical-path ordering. Replace §3 (8 constraint bullets) with one line: *"Constraints surfaced by the audit are recorded in `private/notes/phase5-skeleton-audit.md` §'5.1 input bullets' and will be quoted into ADR-0027/0028's Inputs section."* Delete §4 entirely (F-003 already covers deferral as project law; restating it per-bullet adds no commitment). - -Better than current draft: - -- Removes the duplication Consequences/Negative already concedes. -- Honors ADR-0020's "ONE load-bearing decision" rule more strictly — the load-bearing decision is the verdict table; constraints are inputs. -- Smaller surface to keep synchronized when 5.1 ADRs land. - -Breaks / costs: - -- Forces 5.1 to depend on a gitignored survey for its 8-bullet Inputs section. This contradicts the Alternative-A rejection rationale in the current draft (which argues the constraints need a tracked record because `private/notes/` is gitignored). -- Loses the "this ADR explicitly says these 8 things bind 5.1" tracking property. -- Resolution: those 8 bullets could be promoted into 5.1's own ADR Context once 5.1 starts — but that timing risk is real (5.1 author might paraphrase, missing constraint #2's `std.atomic.Mutex` subtlety). - -### Alt 2 — Finished-form-clean: Split into ADR-0026 (verdict) + a tracked `.dev/phase5_entry_inputs.md` promotion - -Sketch: ADR-0026 keeps §1 (verdict table) + §2 (critical-path ordering) + §4 (non-decisions, kept as the F-003 deferral record per row). The 8 constraint bullets in §3 get promoted out of the gitignored survey into a new tracked file `.dev/phase5_entry_inputs.md` (or `.dev/decisions/0026_inputs.md` as an appendix). ADR-0026 §3 becomes a one-line pointer to that tracked file. - -Better than current draft: - -- Separates two ADR-level concerns cleanly: "what activates where" (ADR-0026) vs "constraints discovered during audit" (tracked inputs doc). -- Tracked inputs doc can be updated independently when 5.1 lands without rewriting ADR-0026. -- Removes the awkward "ADR carries bullets, 5.1 quotes them" double-track. -- F-002 finished-form bias: the cleanest Phase 5 entry record IS a tracked inputs file separate from the verdict ADR. - -Breaks / costs: - -- Introduces a new tracked-file genre (`.dev/_entry_inputs.md`) that no other Phase uses. Sets a precedent for Phase 6+ entries that may not generalize. -- Two commits at Phase 5 entry instead of one ADR commit. Mild ROADMAP §17 paperwork increase. -- The "inputs file" pattern blurs with `private/notes/` for surveys that *don't* surface ADR-binding constraints — when is an audit important enough to graduate? - -### Alt 3 — Wildcard: Re-frame as ADR-0026 "Phase entry audit pattern" + per-Phase audit appendix - -Sketch: ADR-0026 becomes a **pattern decision** (like ADR-0020 governance): "every Phase entry runs a skeleton-audit task and its outputs are promoted from `private/notes/-skeleton-audit.md` into a §-audit appendix at the bottom of ROADMAP §9.." Phase 5's specific verdict table + constraints live as **ROADMAP §9.7 appendix** rather than as ADR body. ADR-0026's body teaches the pattern; the Phase 5 instance is the worked example referenced from ROADMAP. - -Better than current draft: - -- One pattern ADR amortizes across Phase 5 / 6 / 7 / … entries. -- Verdict tables live where activations live (in ROADMAP §9. next to the task rows that cite them), keeping the navigation tight. -- F-002 finished-form: when Phase 12 happens, future agents read ADR-0026-pattern + ROADMAP §9.12 appendix, not "ADR-0026 + ADR-0048 + ADR-0091" (per-phase ADRs). -- Sidesteps the ADR-0020 "ONE decision" tension entirely — the one decision is the *pattern*, not a single phase's verdict. - -Breaks / costs: - -- Phase 5's specific decisions get less ADR-grade prominence — only listed in ROADMAP, which is mutable per §17 and harder to track historically. -- Pattern ADRs invite premature generalization; Phase 6/7 may not actually need a skeleton audit (depends on how many skeletons land at each Phase 5 entry — and only Phase 4 was the entry that *deliberately* deferred 8 skeletons). -- Bigger structural change at Phase 5 entry; runs against F-002's "diff size is second-tier" only if the pattern actually recurs. If it doesn't recur, this is pure overhead. - -### Load-bearing constraints the ADR omits (NEW) - -1. **OrbStack gate lock applies to every §9.7 row, not noted in critical path §2.** Each activation task must pass both Mac and Ubuntu x86_64. The exit-smoke row 5.16 carries the 🔒 OrbStack icon in ROADMAP, but 5.1-5.15 each have their own gate too. Not strictly an ADR-level decision, but a finished-form Phase-5-entry record should at minimum cite §A18's framework adoption (per ROADMAP A18, Phase 5 is when `scan_lib.sh` framework finalizes — relevant to 5.x's check_*.sh gate flips at 5.15). -2. **§9.7 row 5.15 build_options flip ordering vs §4 of the ADR.** ADR-0026 §2 lists critical-path activations 5.2 → 5.3 → 5.4-5.6 → 5.7 → 5.8 → 5.9-5.10 → 5.11 → 5.12 but omits 5.15 (build_options flip). Per ROADMAP §9.7.x, 5.15 is sequential-spine ("…→ 5.12 → 5.15 (flip) → 5.16 (smoke)"). The ADR should either list 5.15 in critical path or explicitly say it's omitted because the flip is mechanical-after-5.12. -3. **Constraint #5 + #6 conflate `gc_mark` bit allocation with free-pool link placement.** Bullet 5 says 29 spare bits in `gc_and_lock.gc_mark`; bullet 6 says a `next: ?*HeapHeader` free-pool link is needed somewhere (8 bytes on 64-bit). These are independent decisions — the free-pool link cannot fit in the 29 spare bits (pointer is 64 bits, not 29). The ADR should clarify they are two separate slot-placement questions, not one. - -### Redundancy check against existing ADRs - -- ADR-0007 / 0008 / 0009 / 0017: ADR-0026 references but does not duplicate; it *cites the activation owner row* for each skeleton these ADRs introduced. No redundancy. -- ADR-0020 (governance): ADR-0026 follows the template (has Affected files). The "Decision §4 Constraints this ADR does NOT settle" section is unusual under ADR-0020 — non-decisions belong implicitly to F-003 / handover queue rather than as a numbered Decision sub-item. Mild structural smell, not a blocker. -- ADR-0025 (chapter dormancy): orthogonal — no overlap. -- No outright duplication with any prior ADR. - -### Subagent recommendation (non-binding within F-NNN envelope) - -Alt 1 (smallest-diff) as the highest-value change: strip §3 + §4, keep only the verdict table + critical path. The 8 constraint bullets are already faithfully recorded in the survey which the ADR cites. When 5.1 opens, that task's Step 0/1 reading list already includes the survey per CLAUDE.md "Phase reading list"; 5.1's ADR drafts can quote the bullets at the moment they bind. - -The current draft's Alternative-A rejection ("private/notes/ is gitignored, need tracked record") is **partially valid but over-applied**: the *verdict table* (§1) genuinely needs tracking because it commits 5.1-5.12 to specific activation owners; the *constraint bullets* (§3) are forward-leaning inputs, not commitments, and live more naturally in the consuming ADR's Context section. - -If the main loop prefers tracked constraint bullets, Alt 2 is the cleaner shape. Alt 3 is best long-term if Phase 6+ entries also accumulate skeleton-deferred work, but premature without a second data point. - -No F-NNN violation in any alternative. - -### Main loop disposition - -Alt 1 applied with one carve-out: §3 stays as a one-paragraph pointer (not deleted) so future readers see the link to the survey constraint bullets without having to re-discover them. §4 removed entirely per the subagent's accurate F-003-overlap observation. Omitted-constraints #1 and #2 reflected into §2 (per-row OrbStack lock note + 5.15 in critical-path enumeration). Omitted-constraint #3 (gc_mark / free-pool conflation) is moot now that §3 is a pointer — the conflation lived in the bullets that are no longer in this ADR. - -## Consequences - -- **Positive**: §9.7 rows 5.1-5.16 execute against a tracked verdict table (which activations belong to which row). Phase 7's D-040 rename plan and Phase 14's ADR-0015 a2 deferral are explicitly preserved. The ADR body shrinks to the load-bearing decision (Alt 1 application). -- **Negative**: 5.1's ADR-0027 / ADR-0028 cluster must pull the 8 constraint bullets from the gitignored survey at the moment they bind. CLAUDE.md § Phase reading list already requires 5.1 to read the survey, so this is mechanically supported, but the risk that 5.1 author paraphrases-and-loses-subtlety (e.g., the `std.atomic.Mutex` Zig-0.16 hazard in bullet #2) is real. Mitigation: 5.1's ADR Inputs section must quote the survey bullets verbatim, not paraphrase. -- **Neutral / follow-ups**: the "needs restructure" verdicts for 4.17 / 4.18 / 4.23 / 4.24 set expectations — under F-002, restructure is information for §9.7 task ordering, not a reason to avoid the work. Alt 3 (pattern ADR for all Phase entries) is recorded for re-consideration at Phase 6 entry: if Phase 6 also accumulates audit-shaped deferred work, the pattern is worth lifting. - -## Affected files - -- `private/notes/phase5-skeleton-audit.md` (the survey deliverable — gitignored, cited) -- `.dev/ROADMAP.md` §9.7 task table (the activation owners cited above; row 5.0 will flip to `[x]` in the same commit as this ADR) -- `.dev/debt.md` D-028 (this ADR satisfies the audit row; D-028 stays open until the surveyed skeletons all activate) -- Future: `.dev/decisions/0027_*.md` + `.dev/decisions/0028_*.md` (the 5.1 ADR cluster) - -## References - -- `.dev/project_facts.md` F-002 (finished-form wins), F-003 (decision-deferral), F-004 (NaN-box 64 slot), F-006 (mark-sweep GC + 3-layer allocator) -- `.dev/principle.md` "Structural imagination phase" (the discipline the audit + this ADR jointly satisfy) -- `.dev/decisions/0007_type_descriptor_option_beta.md`, `0008_protocol_dispatch_unify.md`, `0009_object_header_heap_only_lock.md`, `0012_nan_box_valuetag_day1.md`, `0017_allocator_strategy.md`, `0020_adr_governance.md` (the entry ADR set) -- `.dev/debt.md` D-028 (this ADR's row of origin), D-040 (collision deferred to Phase 7) -- `private/notes/phase5-skeleton-audit.md` (the survey) - -## Revision history - -- 2026-05-24: Status: Proposed → Accepted (autonomous-loop self-accept after Devil's-advocate subagent review reflected verbatim into Alternatives considered above; Alt 1 applied with §3 reduced to a pointer rather than deleted entirely, §4 removed, omitted constraints #1 and #2 reflected into §2). diff --git a/.dev/decisions/0027_nan_box_second_generation.md b/.dev/decisions/0027_nan_box_second_generation.md deleted file mode 100644 index 935b755be..000000000 --- a/.dev/decisions/0027_nan_box_second_generation.md +++ /dev/null @@ -1,260 +0,0 @@ -# 0027 — NaN-box layout 第二世代 (64-slot, 44-bit shifted pointer) - -- **Status**: Accepted -- **Date**: 2026-05-24 -- **Author**: Shota Kudo (drafted with Claude, autonomous-loop self-accept per CLAUDE.md § "ADR-level designs are handled inline" after Devil's-advocate subagent review) -- **Tags**: phase-5-entry, value-tag, nan-box, second-generation, F-004 -- **Co-issued with**: ADR-0028 (mark-sweep GC + 3-layer allocator). The two ADRs land together as a paired Accepted set per §9.7 row 5.1 — the GC needs the slot map and per-tag dispatch table; the slot map needs the GC's per-tag finaliser hook. - -## Context - -cw v1's NaN-box `Value` is the single most load-bearing data structure (ADR-0012). The first generation (ADR-0012, ADR-0012 amendment 1) ships **4 group × 8 sub-type = 32 slot** with group bands `0xFFF8` … `0xFFFB` and a 45-bit pointer. By Phase 4 close that layout has cost two pieces of evidence: - -1. **Slot pressure**: 26 day-1 tags + the 4.23 / 4.24 amendments + the F-004 day-1 types (`range` / `map_entry` / `tagged_literal` / `string_seq` / `array_seq` / `sorted_map` / `sorted_set` / `persistent_queue` / `funcref` / `externref`) cross 32 with no breathing room. F-005 numeric tower needs three contiguous slots (`big_int` / `ratio` / `big_decimal`) and ADR-0012 amendment 1 had to scavenge the released `wasm_module` / `wasm_fn` slots. -2. **Slot-sharing pain (cw v0 evidence)**: cw v0's final layout crammed 5 reference types (`future` / `promise` / `delay` / `agent` / `ref`) into a single `delay` slot via a discriminant byte, and `ratio` + `big_decimal` shared one slot. Both cases produced debug pain that the post-mortem records as a design surrender. - -User invariant **F-004** (`.dev/project_facts.md`) fixes the second-generation shape: **4 group × 16 sub-type = 64 slot, 44-bit shifted pointer (128 TB user-space address)**, day-1 enumeration of the types listed above. F-004 also commits the 4 group / 64 slot indicative slot map (Group A hot data + persistent collections / Group B callables + reader extra / Group C mutable + concurrency / Group D numeric + wasm + extension). - -The 5.0 cleanup-wave audit (ADR-0026 + `private/notes/phase5-skeleton-audit.md`) bound 8 input constraints; they are quoted verbatim in §"Inputs from 5.0 audit" below. - -cw v0 NaN-box archaeology was surveyed at `private/notes/phase5-5.1-survey.md` (~580 lines). The relevant findings: - -- cw v0's `Tag` enum reached 45 entries (`runtime/value.zig`) with no inline back-pointer table — class-of dispatch is **string-based** through `class-of` (`predicates.zig:387-390`). cw v1 must instead build a comptime `Tag → ?*const TypeDescriptor` lookup so 5.11 TypeDescriptor activation has a constant-time dispatch root. -- cw v0 has no `TypeDescriptor` at all — classes are a comptime `ClassDef` const array (`lang/interop/class_registry.zig:54-121`). cw v1's TypeDescriptor / TypedInstance split (ADR-0007) is net-new; the Tag → descriptor table is genuinely cw-v1 territory. - -## Decision - -### 1. Layout: 4 group × 16 sub-type = 64 slot - -Bit allocation inside the quiet-NaN payload (51 payload bits): - -| Bits | Field | Width | Notes | -|--------|-------------------|-------|--------------------------------------------------------------------------------------------------------------------------------------------| -| 63..51 | quiet-NaN signal | 13 | sign (1) + exp_all_one (11) + quiet (1); band selector `0xFFF8` … `0xFFFB` for heap groups A / B / C / D | -| 50..48 | band selector low | 3 | low 3 bits of `0xFFFx` (in top16 = bits 63..48). Heap: `0b000`..`0b011` = A..D. Immediate: `0b100`..`0b111` = INT / CONST / CHAR / BUILTIN | -| 47..44 | sub-type | 4 | 16 sub-types per group → 64 heap slots total | -| 43..0 | pointer payload | 44 | shifted right by 3 (align-8 invariant) → 47-bit byte address = **128 TB user space** per F-004 decree | - -The 44-bit shifted pointer matches F-004 verbatim: the 47-bit byte address covers the canonical 48-bit user-mode VA's user portion (128 TB) on Linux x86_64 / aarch64, macOS aarch64, Windows x86_64. F-004 is decree on this field width. 4 sub-type + 44 pointer = 48 bits exactly fills the payload below top16; **no residual reservation bit exists in the g2 layout** — the bit count is exact per F-004 + the existing top16 partition. - -The g2 layout differs from g1 by widening sub-type 3 → 4 bits and shrinking pointer 45 → 44 bits. NB_HEAP_SUBTYPE_SHIFT moves from 45 (g1) → 44 (g2); NB_ADDR_SHIFTED_MASK from `0x1FFF_FFFF_FFFF` (45-bit) → `0x0FFF_FFFF_FFFF` (44-bit). The encode/decode contract (§4) preserves the align-8 invariant — `encodePointer` asserts `addr & 0b111 == 0`, which keeps the low 3 bits of the byte address zero so the shifted address fits the 44-bit field. - -`int_53` (immediate 53-bit signed integer) keeps its dedicated tag band (`NB_INT_TAG = 0xFFFC`) reserved by g1 — it is not in the 64-slot group/sub-type space because it consumes the full 53-bit payload as a value, not a pointer. - -### 2. Slot map (Tag enum, 64 entries day-1) - -The Tag enum is decreed by F-004 — owner implements, does not re-decide. Each group occupies 16 contiguous slots; reserved slots stay declared but no-behaviour. The current best placement (subject to fine-grained shuffle within a group only): - -**Group A (slots 0..15) — Hot data + persistent collections** -``` -A0 string A4 vector A8 lazy_seq A12 range -A1 symbol A5 array_map A9 cons A13 string_seq -A2 keyword A6 hash_map A10 chunked_cons A14 array_seq -A3 list A7 hash_set A11 chunk_buffer A15 map_entry -``` -(Matches F-004 indicative slot map verbatim. The g2 draft erroneously inserted `nil_` at A3 + `boolean_true` / `boolean_false` at D14 / D15 — see amendment 1 below for the correction. `nil` / `boolean_true` / `boolean_false` are **immediates** encoded via the `NB_CONST_TAG` band (payload 0 / 1 / 2), orthogonal to the 64-slot heap-pointer space — they do not consume a Tag slot. `hash_set` lives at A7 alongside `hash_map` (Group A's "persistent collection" cluster); the F-004 indicative map and ADR-0026 audit verdict agreed on this placement.) - -**Group B (slots 16..31) — Callables + reader extra** -``` -B0 fn_val B4 var_ref B8 tagged_literal B12 type_descriptor -B1 multi_fn B5 ns B9 reader_conditional B13 host_instance -B2 protocol B6 delay B10 class B14 typed_instance -B3 protocol_fn B7 regex B11 reified_instance B15 reserved -``` - -**Group C (slots 32..47) — Mutable + concurrency + transient + sorted/queue** -``` -C0 atom C4 future C8 transient_vector C12 array_chunk -C1 agent C5 promise C9 transient_map C13 persistent_queue -C2 ref C6 reduced C10 transient_set C14 sorted_map -C3 volatile C7 ex_info C11 reserved C15 sorted_set -``` - -**Group D (slots 48..63) — Numeric + Clojure collection internals + wasm tail** -``` -D0 big_int D4 hamt_node D8 tval D12 wasm_module -D1 ratio D5 tail_node D9 matcher D13 wasm_fn -D2 big_decimal D6 hamt_map_node D10 tuple D14 wasm_funcref -D3 array D7 hash_collision_map_node D11 box D15 wasm_externref -``` -(Layout per amendment 6, the D-248 reorg 2026-06-15 — Clojure persistent-collection -internals up to D4..D8, wasm surfaces at the D12..D15 tail. See the revision history.) - -`wasm_funcref` / `wasm_externref` at D14 / D15 are **inline-tagged**, committed by F-004 (decree). zwasm v2 §4.1 encodes `*const FuncEntity` with `align(8)` (F-008) — fits the 45-bit shifted pointer field with margin. The D-036 Phase 16 decision is **only about the marshalling wrapper** (whether `host_func.zig` exposes an additional Pod-shaped wrapper around the inline body for FFI hand-off); the slot encoding itself is decreed inline by F-004. This resolves the internal contradiction Devil's-advocate flagged on the §2 draft (cross-coupling §"wasm_funcref / wasm_externref inline reservation" row). - -Group D is now fully named (16/16, no anonymous reserves); D-043's slot-reservation concern is closed for Group D (the remaining D-043 scope is B-band). Per F-002 reservations are memos — naming every slot when a consumer exists is the finished form. - -`array` (D3) is Java-array compat (`Object[]` boundary for host interop). `matcher` (D9) is `java.util.regex.Matcher` analogue. `tuple` / `box` (D10 / D11) are reserved for transducer / mutable-box scaffolding (Phase 7). - -**Big-int / ratio rotation**: ADR-0012 amendment 1 parked `big_int` at HeapTag slot 29 and reserved 30 for `ratio` (smallest-diff under the 32-slot layout). g2 moves both into Group D at canonical positions (`big_int = D0`, `ratio = D1`, `big_decimal = D2`). The rotation is the §9.7 row 5.2 `value.zig` split's mechanical landing; every `HeapTag.big_int` reference rotates in that commit (4 sites grep-located: `numeric/big_int.zig::init`, encode/decode tests). - -### 3. Per-Tag dispatch infrastructure — owned by ADR-0028 - -The g2 draft of this ADR initially declared a `tag_descriptor_table: [64]?*const TypeDescriptor` here. Per Devil's-advocate Alt 1 (ADR-0020 governance: "one ADR, one load-bearing decision") + the cross-coupling §"Triple Tag-indexed table" concern, the per-Tag dispatch infrastructure consolidates into ADR-0028 alongside the per-Tag GC-trace and finaliser tables. ADR-0028 §4 hosts the three Tag-indexed `[64]?*const X` tables together (or a single `TagOps` struct over them). - -This ADR commits only the slot map + bit layout + encode/decode contract. 5.11 TypeDescriptor activation reads the descriptor table from ADR-0028 §4; 5.3 GC implementation reads the trace + finaliser tables from the same. - -### 4. Encode / decode contract - -`runtime/value/nan_box.zig` exports two pairs: - -```zig -pub fn encodePointer(tag: Tag, ptr: *anyopaque) Value; -pub fn decodePointer(v: Value) *anyopaque; -pub fn encodeImmediate(tag: Tag, payload: u44) Value; // for inline funcref / externref (44-bit per F-004) -pub fn decodeImmediate(v: Value) u44; -``` - -The `align(8)` invariant on pointer payloads is asserted in `encodePointer` via `std.debug.assert(@intFromPtr(ptr) & 0b111 == 0)`. Heap allocator (`gc_alloc`) guarantees the alignment per ADR-0028 §3. - -### 5. Migration scope (cw v1 Phase 1-4 source rewrite) - -The g1 → g2 surgery rewrites: - -- `runtime/value.zig` — split per ADR-0029 alias (D-029) into `runtime/value/{value.zig, nan_box.zig, heap_tag.zig, heap_header.zig}` (§9.7 row 5.2 owns the split mechanics + decreed file boundary per `.dev/structure_plan.md`). -- `Tag` enum widens from 32 entries → 64. Every existing `@enumFromInt(u5, …)` / `@intFromEnum(Tag)` site widens to `u6`. **Pre-migration grep count** (mechanical surface estimate so 5.2 owner does not under-budget the row): `rg -nE '@enumFromInt\(.*Tag|@intFromEnum\(.*Tag|@intFromEnum\(.*HeapTag' src/ test/` returns the site list — 5.2 owner runs this immediately on row open. The exhaustive `switch` arms in `analyzer.zig` / `tree_walk.zig` / encode / decode acquire new arms for the day-1 additions (defaulted to `Code.feature_not_supported` until the owning §9.7 row activates the body, per `no_op_stub_forbidden.md`'s "explicit-error stub" pattern). -- `HeapTag.big_int` rotates from slot 29 → Group D slot 0 (4 grep sites; mechanical). -- The per-Tag dispatch tables (descriptor / trace / finaliser) are new comptime artefacts landed by 5.3 alongside the GC body per ADR-0028 §4 — not by 5.2 (split commit stays minimal). - -The rewrite is expected per ROADMAP §A25 + F-002. Depth 3 (cross-file refactor) is the typical depth; depth 4 only if a follow-up ADR replaces the 64-slot envelope itself. - -### 6. What this ADR does NOT decide - -- Per-tag finaliser dispatch — owned by ADR-0028 §4 (paired ADR). -- LazySeq `force()` mutex shape — deferred to §9.7 row 5.7 per audit bullet #2 (Zig-0.16 `std.Io.Mutex` requires threaded `Io`; row 5.7 owner picks blocking-via-`io_default` vs spin-via-`std.atomic.Value`). -- D-040 `MethodEntry` rename — Phase 7 entry per debt row; this ADR's `tag_descriptor_table` does not touch the name. -- Phase 16 inline-vs-Pod choice for `wasm_funcref` / `wasm_externref` — D-036. - -## Inputs from 5.0 audit - -The 8 input bullets at `private/notes/phase5-skeleton-audit.md` §"5.1 input bullets" bind this ADR. Quoted verbatim (paraphrase-loss protection per ADR-0026 §3): - -> 1. GC root-set must enumerate `LazySeq.thunk` + `LazySeq.ctx` + `LazySeq.seq_cache`. … The ADR-0028 root-source list must name `LazySeq` explicitly. -> 2. `LazySeq.lock: std.atomic.Mutex` is a Zig-0.16 hazard. Per `zig_tips.md`, `std.atomic.Mutex` survives as the lock-free `tryLock` / `unlock` shape only; the concurrent semantics ADR-0009-amendment-2 specifies for `force()` (double-checked locking) may need `std.Io.Mutex` (blocking, requires `io` threading). 5.1 should not pre-commit to the shape; 5.7 owns the re-evaluation when activation lands. -> 3. TypeDescriptor's `method_table: []const MethodEntry` slice must land in a GC-aware allocator, because the namespace owning it is itself process-lifetime (`infra_alloc` per F-006). … The GC ADR must distinguish "TypeDescriptor live forever" vs "TypedInstance is a GC root entry-point through its descriptor pointer". -> 4. F-004 64-slot Tag enum expansion changes `TypeDescriptor.lookupMethod`'s receiver-classification path. … The ADR-0028 + ADR-0027 pair must show how the 45-tag table is built — comptime `StaticStringMap` over Tag? per-tag `?*const TypeDescriptor` array? — because that table is itself GC-rooted via its descriptor pointers. -> 5. `HeapHeader.gc_and_lock.gc_mark: u30` is 30 bits today. … The ADR-0028 mark phase needs at most 1 bit (or 2 for tri-colour); the remaining bits are a free-pool size-class hint candidate per cw v0's free-pool optimisation. 5.1 should record the choice of "what the other 29 bits carry". -> 6. `HeapHeader` is 8 bytes today (tag + flags + pad + gc_and_lock). F-006 free-pool needs a `next: ?*HeapHeader` link somewhere when an object sits on the free list — either inside the freed object's payload (cw v0 path) or a separate per-size-class chain. 5.1 should at least note which the ADR-0028 commits to. -> 7. D-040 `MethodEntry` collision is unresolved (deferred to Phase 7). 5.1's protocol-related text must not re-name either of the two MethodEntry types … -> 8. `big_int` HeapTag = 29 today (per ADR-0012 amendment 1). F-004 Group D moves it to Group D slot 0 (Group D base = 24, sub-type 0). … - -ADR-0027 §1-5 satisfies bullets #4 (Tag table) + #7 (no rename) + #8 (rotation). Bullets #1 / #3 / #5 / #6 are ADR-0028 territory. Bullet #2 is deferred to §9.7 row 5.7 owner per the audit's explicit guidance. - -## Alternatives considered - -A fresh-context `general-purpose` subagent was forked as Devil's advocate per CLAUDE.md § "ADR-level designs are handled inline" (depth ≥ 2 mandate). Brief: F-001..F-008 envelope; produce 3 alternative shapes (smallest-diff / finished-form-clean / wildcard) per ADR; do not propose F-NNN-violating alternatives. Full output at `private/notes/phase5-5.1-devils-advocate.md` (~440 lines). The ADR-0027 portion is reflected here verbatim. - -### Would-violate-F-NNN findings - -**None.** All three ADR-0027 alternatives stay within F-001 / F-004 / F-005 / F-006 / F-008. - -### Subagent summary judgement (ADR-0027 portion, verbatim) - -> ADR-0027 (NaN-box 第二世代) — substantively correct. F-004 hands the loop the decree (4 × 16 = 64 slot, 44-bit shifted pointer); the ADR's load-bearing additions on top of F-004 are (a) the `tag_descriptor_table` and (b) the `nil/bool` intra-slot multiplex at A3 and (c) the slot map's *concrete* placement. Of those, only (c) is genuinely re-decidable (F-004 named it indicative); (a) is forced by ADR-0007's "primitives back-pointer to native TypeDescriptor"; (b) is forced by F-004's only-three-immediate slot map. The smells that *do* land: -> -> - **Reservation-as-bias** at D11–D15 (five reserved slots in Group D) and B15 (one reserved in Group B) — six reserves total are placed with no Phase-15 horizon argument written, only "for future use". F-002 says reservations are memos, not contracts; the ADR records them as if they were contracts. -> - **Over-scope** at §3 `tag_descriptor_table` — the table is a load-bearing decision in its own right that ADR-0020 governance would flag as "two load-bearing decisions in one ADR" (slot map + dispatch table). Splittable. -> - **Bit 0 reservation** at §1 (1-bit "future tag-bit (e.g. interned-flag)") — same smell, no use case sketched. - -### Alt 1 — Smallest-diff (verbatim) - -> **Sketch**: Land the 64-slot envelope and slot map; **defer the `tag_descriptor_table` to ADR-0028 §5 (per-tag GC-trace table)**. Both tables index by Tag; both are `[64]?*const X`. ADR-0028 already ships the trace table. Add a sibling field on the same per-Tag row, or co-locate the descriptor pointer with the trace fn in a single `TagOps = { trace, finaliser, descriptor }` struct. Result: ADR-0027 stops at "the slot map + bit layout + immediate encoding"; the per-Tag dispatch infrastructure (one shape) lives in ADR-0028. -> -> **Better than current draft**: ADR-0020 governance — "one ADR, one load-bearing decision". 0027 becomes purely about the *Value bit layout*; 0028 becomes purely about the *per-Tag dispatch infrastructure* (trace, finaliser, descriptor). Eliminates the §3 / ADR-0028 §4 / ADR-0028 §5 triple-table smell. 5.2 `value.zig` split commit becomes smaller. -> -> **Breaks**: Reading order: 5.11 TypeDescriptor activation now references ADR-0028 §5 instead of ADR-0027 §3. Minor. "Co-issued" framing softens. -> -> **Cost**: tiny. - -### Alt 2 — Finished-form-clean (verbatim) - -> **Sketch**: Same Alt 1 split, **plus**: -> -> 1. **Replace D11–D15 + B15 + bit 0 reservations with named placeholders.** F-002 says reservations are memos. Either name the type (e.g., D11 = `weak_ref`, D12 = `value_cell` for STM Phase 15, D13 = `transducer_state` for Phase 7's xform work) or **shrink the slot count to the actual day-1 + Phase 5-15 horizon**. The Tag enum doesn't need to *be* 64; it needs to *fit in* 64. If day-1 + horizon is 53 slots, declare 53 and leave the bit field at 6 bits for room — but stop pretending the 6 reserved entries are part of the design. -> 2. **Replace `nil/bool` multiplex at A3 with three separate Tag slots** (`nil`, `boolean_true`, `boolean_false`). F-004 gives 64 slots; the multiplex is *cw v0 cramming*, not a finished-form move. F-002 + F-004 say cleanliness wins; A3's "this is the only intra-slot multiplexing the layout permits, and it is bounded at 3 values forever" is the same shape of surrender cw v0 made (5 ref-types in `delay`). Don't carry forward the spirit of the surrender into the finished form. -> 3. **Decide the bit-0 reservation explicitly** — either commit to "interned-flag for keyword + symbol identity-check accelerator" (with a benchmark target), or remove the bit and widen the pointer field to 45 bits. Don't reserve "for future use". -> -> **Better than current draft**: F-002 finished-form-wins applied uniformly to the slot map (no cramming, no anonymous reservations). Bit 0 either earns its keep or is gone. A3 split lets 5.11 dispatch to the three primitive descriptors without the discriminant byte. -> -> **Breaks**: Tag count goes from 64 → ~50. ADR text changes shape. More text in the ADR; less in the source debt later. -> -> **Cost**: medium. - -### Alt 3 — Wildcard (verbatim, abridged) - -> **Sketch**: **Inline keyword and symbol via 32-bit interned ID + 4-bit length hint, dropping the pointer indirection for the common case.** Burn an immediate-tag band (like `int_53` at `NB_INT_TAG = 0xFFFC`) for **interned-id-15-bit keywords and symbols** + a *separate* pointer slot at A1/A2 for the overflow. -> -> **Why within F-NNN envelope**: F-004 §1.4 explicitly carved the immediate-band exception (`int_53`). F-002 finished-form-wins: keyword identity check today is `(decode_ptr(a) == decode_ptr(b))`; wildcard makes it `(a.raw == b.raw)`. -> -> **Breaks**: Big new encode/decode contract. Interned-id allocator becomes a Phase-5 dependency (Pull-forward cost). Phase 16 zwasm bridge has to know about the immediate band. -> -> **Cost**: large. If the smell sensor surfaces it twice in two phases, mint a future ADR with this body. - -### Load-bearing concerns the ADR omits (verbatim) - -> 1. `@enumFromInt(u5, …)` → `u6` widening at every call site is asserted but not enumerated. Grep surface count is not given. Add a count. -> 2. F-004 says `wasm_funcref` / `wasm_externref` are inline-tagged (D6, D7). §2's text says "inline-tagged per F-008 ... fits the 44-bit shifted pointer field exactly" but then §2 last paragraph contradicts: "the day-1 inline reservation is non-committal — D4 / D5 / D6 / D7 stay declared but Phase 16 entry decides whether the body wires inline or routes through Pod." So is it inline (per F-004) or undecided (per ADR text)? F-001 says F-004 wins. -> 3. Group A intra-slot density vs Group D sparseness — Group A is full (16/16, with hash_set displaced to C13 to keep A dense), Group D is 11/16 used (5 reserved). The displacement of hash_set into Group C is *itself* a smell. - -### Main loop disposition - -Devil's-advocate Alt 1 **applied**: §3 `tag_descriptor_table` removed from this ADR; per-Tag dispatch infrastructure consolidates into ADR-0028 §4. - -Alt 2 item 2 **applied**: A3 nil/bool multiplex split — A3 = `nil_`, D14 = `boolean_true`, D15 = `boolean_false`. The day-1 demonstration that cw v1 finished-form discipline differs from cw v0's surrender (per the principle test cited). - -Alt 2 item 3 **applied**: bit 0 reservation removed; pointer widens to 45 bits (256 TB shifted byte address, exceeds canonical 48-bit user VA). Interned-flag candidate recorded as `D-043` for Phase 7+ immediate-band ADR consideration. - -Alt 2 item 1 **partially applied**: D11 / D12 / D13 remain as anonymous reserves (down from 5 reserves to 3 after the A3 nil/bool split moved into D14 / D15). Per Devil's-advocate "shrink the slot count" recommendation: full shrink would require re-drawing F-004's 4 × 16 group boundaries (depth-4 surgery deferred per F-003 to a Phase 7+ owner with measured dispatch-frequency data). The 3 remaining reserves are captured by debt `D-043` so the next Phase entry inherits the foresight rather than a silent reservation. B15 reservation also moves into D-043 scope. - -Alt 3 (wildcard) **not applied** — recorded as a future-ADR candidate per Devil's-advocate's "not recommended for Phase 5 entry" verdict. Carried by `D-043` (immediate-band candidate analysis) for Phase 7+ owner. - -§2 wasm_funcref **contradiction resolved** per Devil's-advocate Load-bearing-concern #2: ADR text aligned to F-004 (slot encoding committed inline); D-036 Phase 16 entry decision narrowed to marshalling-wrapper choice only. - -Load-bearing concern #1 (grep count for `Tag` widening) **applied** as a procedural note in §5 (5.2 owner runs the grep immediately on row open to size the row). - -Load-bearing concern #3 (Group A density vs Group D sparseness; `hash_set` displacement smell) **acknowledged inline** in §2 Group A paragraph; redraw deferred per F-003 to a Phase 7+ owner. - -## Consequences - -- **Positive**: 64-slot envelope absorbs F-004 day-1 enumeration + Phase 5-15 additions with ~10 slots reserve. Constant-time `Tag → TypeDescriptor` dispatch replaces cw v0's string-based class-of. Inline `funcref` / `externref` reservation aligns with zwasm v2 §4.1 `@intFromPtr(*const FuncEntity)` + 0 sentinel encoding (F-008) so Phase 16 entry's integration is mechanical-not-design. -- **Negative**: Tag enum widens from 32 → 64 (5-bit → 6-bit). Every existing switch arm gains 32 new cases; the §9.7 row 5.2 split commit is large by line count (~1500 lines moved across 4 files). Acceptable per F-002 (finished form wins over diff size); the alternative is a third generation in Phase 7-10 when slot pressure recurs. -- **Neutral / follow-ups**: ADR-0028 (paired) consumes the slot map for per-tag finaliser table + per-tag GC-trace dispatch. ADR-0007 amendment 1's "primitives gain a back-pointer to a native TypeDescriptor" wires through `tag_descriptor_table` (filled by 5.11). ADR-0012 a1's transient slot placement is superseded for `big_int` (slot 29 → D0); the amendment 1 history stays in ADR-0012 as the smallest-diff record. - -## Affected files - -- `.dev/decisions/0028_*.md` (the paired ADR — see §0 Co-issued-with). -- `src/runtime/value.zig` (split per ADR-0029 alias / D-029 at §9.7 row 5.2 into `runtime/value/{value, nan_box, heap_tag, heap_header}.zig`). -- `src/runtime/numeric/big_int.zig` (HeapTag rotation 29 → D0; init helper unchanged). -- `src/runtime/native_descriptors.zig` (new file; 5.11 lands the descriptor globals). -- `src/eval/analyzer.zig`, `src/eval/backend/tree_walk.zig`, `src/eval/backend/vm/compiler.zig`, `src/runtime/dispatch.zig` (Tag switch arms widen; defaulted `Code.feature_not_supported` for slots whose body lands later per `no_op_stub_forbidden.md`). -- Test surface: every `expectEqual(., …)` referencing HeapTag rotates per §9.7 row 5.2; no behaviour change. - -## References - -- `.dev/project_facts.md` F-001 (zwasm v2 unavoidable), F-002 (finished form wins), F-003 (decision-deferral), F-004 (NaN-box 64 slot — decree), F-005 (numeric tower surface), F-006 (mark-sweep + 3-layer alloc), F-008 (zwasm v2 spec review — Group D inline reservation). -- `.dev/principle.md` § Bad Smell catalogue (Smallest-diff bias, Reservation-as-bias) + § Structural imagination phase (this ADR is the F-004 implementation, not a re-imagination). -- `.dev/decisions/0007_type_descriptor_option_beta.md` (TypeDescriptor — `tag_descriptor_table` consumer). -- `.dev/decisions/0009_object_header_heap_only_lock.md` (HeapHeader.gc_and_lock — ADR-0028 owns the 30 bits). -- `.dev/decisions/0011_host_extension_distributed.md` (host_instance Tag slot at B13). -- `.dev/decisions/0012_nan_box_valuetag_day1.md` (g1 — superseded for slot enumeration; preserved as history per F-002). -- `.dev/decisions/0017_allocator_strategy.md` (3-layer alloc — paired ADR-0028 lands the body). -- `.dev/decisions/0023_comptime_stub_pattern.md` (the deferred-body stub pattern that 5.2 uses for the 32 new Tag arms). -- `.dev/decisions/0026_phase5_entry_scope.md` (verdict table + critical-path ordering; this ADR is row 5.1's deliverable). -- `.dev/decisions/0020_adr_governance.md` (this ADR follows the ONE-decision rule; the load-bearing decision is the 64-slot envelope + slot map + Tag table — paired with ADR-0028 by §0). -- `.dev/structure_plan.md` `runtime/value/` decreed split (F-004). -- `.dev/debt.md` D-027 (this ADR satisfies the row), D-029 (paired structural surgery), D-040 (Phase 7 rename — untouched here). -- `private/notes/phase5-skeleton-audit.md` (5.1 input bullets — quoted verbatim above). -- `private/notes/phase5-5.1-survey.md` (cw v0 archaeology — DIVERGENCE per F-002). -- `.claude/rules/zig_tips.md` § Mutex (the Zig 0.16 constraint cited under bullet #2 / deferral to row 5.7). - -## Revision history - -- 2026-05-24: Status: Proposed → Accepted (autonomous-loop self-accept after Devil's-advocate subagent review reflected verbatim into Alternatives considered; Alt 1 applied — §3 dispatch table moved into ADR-0028 §4; Alt 2 item 2 applied — A3 nil/bool multiplex split into separate Tag slots; Alt 2 item 3 applied — bit-0 reservation reclaimed, pointer widened to 45 bits; §2 wasm_funcref contradiction resolved by aligning to F-004; remaining anonymous reserves captured by D-043). Co-issued with ADR-0028. -- 2026-05-24 (amendment 1): Reverted §2 slot map placement of `nil_` (was A3) + `boolean_true` / `boolean_false` (were D14 / D15) — these are **immediates** encoded via the `NB_CONST_TAG` band (payload 0 / 1 / 2) per the existing g1 invariant; F-004's 64-slot enumeration is the heap-pointer Tag space, orthogonal to immediate encoding. The amendment-as-shipped: A3 = list, D14 / D15 = reserved (per F-004's indicative map verbatim). Devil's-advocate Alt 2 item 2 ("split A3 nil/bool into three separate Tag slots") was responding to the misframe in the original draft and is **withdrawn**: the underlying smell (cw v0 cramming surrender pattern) does not apply because `nil` / `true` / `false` were never crammed — they live in the immediate band and have always been independent Tag classifications (`Tag.nil` / `Tag.boolean`). D-043 reservation count updates from 3 (D11–D13) to 6 (C11 / D11–D15) to reflect the reverts; the §"shrink the slot count" verdict still applies but the redraw stays deferred per F-003. Smell trigger: caught in-flight during §9.7 row 5.2 implementation prep when reconciling the existing NB_CONST_TAG encoding against the §2 draft. Smell category: Spec-drift (ADR text drifted from F-NNN decree + existing source invariant) + Reservation-as-bias (the D14/D15 boolean placement was a contrived "extension subset" framing to host non-heap values in a heap-slot map). -- 2026-05-24 (amendment 2): §1 bit layout corrected — pointer field is **44 bits** (not 45) per F-004 decree "44-bit shifted pointer (128 TB)". The original g2 draft + amendment 1 narrative claimed bit-0 reclamation to widen the pointer to 45 bits / 256 TB byte address; that exceeded F-004's 128 TB bound and was an F-NNN violation. Per priority chain (F-NNN > ADR), F-004 wins; the ADR aligns. encode/decode contract (§4) updated: `encodeImmediate` / `decodeImmediate` payload widths change from `u45` to `u44` to match. Smell trigger: caught during 5.2 implementation prep when reconciling the existing `NB_ADDR_SHIFTED_MASK` constant (45-bit mask for the old 32-slot 3-bit-sub-type layout) against F-004's 44-bit-pointer decree. Smell category: Spec-drift (ADR text exceeded F-NNN envelope) + Smallest-diff bias. - - **Amendment 2's narrative was itself partly wrong** (corrected in amendment 3): it claimed a "1 residual bit at position 0 reserved per F-004 implicit" arising from "51-bit payload − 2 group − 4 sub − 44 pointer = 1 residual". That arithmetic conflated the 13-bit NaN signal at bits 63..51 with the 3-bit band selector at bits 50..48 — the actual payload below top16 (bits 47..0) is 48 bits, of which g2 uses 4 sub-type + 44 pointer = 48 exactly, with no residual bit. The "Devil's-advocate Alt 2 item 3 → withdrawn because F-004 violation" disposition stays correct (widening to 45-bit pointer does violate F-004), but the "bit-0 is F-004-decreed reservation" framing in §1 was incorrect. -- 2026-05-24 (amendment 3): §1 bit layout table re-corrected — there is **no residual reservation bit**. g2 partition: bits 63..51 quiet-NaN signal (13) + bits 50..48 band selector low (3, part of `0xFFFx`) + bits 47..44 sub-type (4) + bits 43..0 pointer payload (44) = 64 exactly. The `NB_HEAP_SUBTYPE_SHIFT` constant moves from 45 (g1) → 44 (g2); the `NB_ADDR_SHIFTED_MASK` moves from `0x1FFF_FFFF_FFFF` (45-bit) → `0x0FFF_FFFF_FFFF` (44-bit). 5.2.b source landing uses the corrected constants (`src/runtime/value/nan_box.zig`). Devil's-advocate Alt 2 item 3 stays withdrawn — there is no bit to reclaim or name; the layout is exact. Smell trigger: caught when 5.2.b's test gate failed on `runtime.value.value.test.F-004 day-1 Tag additions encode + decode through Group A` ("expected .range, found .type_descriptor") — the SHIFT=45 spilled the sub-type's high bit into top16's bit 48, corrupting the band identifier. The principled fix matches F-004's decree exactly. Smell category: Spec-drift (the amendment-2 arithmetic was wrong). -- 2026-05-24 (amendment 4): §2 Group D slot map names **D11 = `hamt_node`** + **D12 = `tail_node`** for PersistentVector internals (§9.7 row 5.4.a). Both are vector backing types per the survey at `private/notes/phase5-5.4-survey.md` — HamtNode is the 32-slot HAMT interior/leaf node, TailNode is the 32-element tail array. Naming 2 of the 6 D-043 anonymous reserves at this row matches the "name when a use case lands" disposition recorded in the original D-043 row text. Per F-NNN envelope: F-004's "indicative slot map" + F-002 finished-form let the ADR name a reserve when a concrete consumer lands; this is not an F-NNN amendment, just a slot-naming refinement within the F-004 envelope. D-043 reservation count drops 6 → 4 (B15 / C13–D13 / D14 / D15 remain anonymous and stay on D-043's Phase 7 entry deadline). -- 2026-05-24 (amendment 5): §2 Group D slot map names **D13 = `hamt_map_node`** + **D14 = `hash_collision_map_node`** for PersistentHashMap internals (§9.7 row 5.5.a). Per the survey at `private/notes/phase5-5.5-survey.md`: CHAMP-style HAMT node with `data_map` + `node_map` bitmaps + `[64]Value slots` for K/V pairs and child pointers; hash-collision bucket type lands at 5.5.c but its Tag slot is declared upfront so the trace dispatch table reservation is stable. Same "name when use case lands" disposition as amendment 4. D-043 reservation count drops 4 → 2 (B15 / C11 / D15 remain anonymous — wait, C11 is now `persistent_queue` per F-004 amendment 1, so B15 + D15 = 2 remain). Per F-NNN envelope: F-004 indicative + F-002 finished-form let the ADR consume reserves as consumers land. -- 2026-06-15 (amendment 6): §2 Group D slot map **reordered** (debt D-248) — the Clojure persistent-collection internals move UP to D4..D8 (`hamt_node`=52 / `tail_node`=53 / `hamt_map_node`=54 / `hash_collision_map_node`=55 / `tval`=56), `matcher`/`tuple`/`box` shift to D9..D11 (57..59), and the four wasm surfaces (`wasm_module`/`wasm_fn`/`wasm_funcref`/`wasm_externref`) move to the D12..D15 **tail** (60..63). Finished-form cleanliness (the user's `layout-gc-decisions-2026-06-05.md` noted wasm was reserved early, pushing Clojure essentials to the end). CHEAP + NON-BREAKING + NOT an F-004 amendment (F-004 fixes 4×16=64 slots + the day-1 type set, not the intra-group ORDER; ADR-0027 §2 owns the slot-map order, loop-amendable). Confirmed safe: runtime dispatch is enum-NAME-based (no hardcoded numeric tag value), the per-Tag GC trace/finaliser tables register by `@intFromEnum(tag)` so they auto-follow the renumber, and `cljw build` serialized bytecode uses a DECOUPLED `ValueTag` (per-Value tag-byte, not raw u64 bits) — no serialized artifact is touched. HeapTag + Value.Tag renumbered in lockstep (the `value.zig` sync test asserts it); `D0 big_int = 48` unchanged; the D15 boundary check (`tag_ops.zig`) updated from `tval` to `wasm_externref`. All values stay in 48..63, so the 2-bit group band is preserved. 1109/1109 units green (-Dwasm). Group D is now fully named 16/16 (D-043's Group-D reserve concern closed; B15 remains). Smell category: none (mechanical finished-form refinement within the F-004 envelope, name-based dispatch verified). Source: `value/heap_tag.zig`, `value/value.zig`, `gc/tag_ops.zig`. diff --git a/.dev/decisions/0028_mark_sweep_gc_three_layer_allocator.md b/.dev/decisions/0028_mark_sweep_gc_three_layer_allocator.md deleted file mode 100644 index cd153f7fa..000000000 --- a/.dev/decisions/0028_mark_sweep_gc_three_layer_allocator.md +++ /dev/null @@ -1,482 +0,0 @@ -# 0028 — Mark-sweep GC + 3-layer allocator (Phase 5 entry) - -- **Status**: Accepted -- **Date**: 2026-05-24 -- **Author**: Shota Kudo (drafted with Claude, autonomous-loop self-accept per CLAUDE.md § "ADR-level designs are handled inline" after Devil's-advocate subagent review) -- **Tags**: phase-5-entry, gc, allocator, mark-sweep, free-pool, finaliser, F-006 -- **Co-issued with**: ADR-0027 (NaN-box 第二世代). Paired Accepted set per §9.7 row 5.1. ADR-0027 owns the slot map + per-tag dispatch table; this ADR owns the per-tag GC-trace + finaliser dispatch + the 3-layer allocator boundary. - -## Context - -ADR-0017 (Phase 4 entry) committed cw v1 to a **3-allocator** model — per-evaluation arena + general-purpose + mark-sweep GC heap — and reserved the `gc_mark: u30` bits on `HeapHeader` (ADR-0009 amendment 2). The skeleton landed at task 4.0a / 4.19 (Phase 4); no `collect()` body, no free-pool, no per-tag finaliser. - -User invariant **F-006** (`.dev/project_facts.md`) fixes Phase 5's body: - -- **mark-sweep, single generation** (cw v0 path inheritance). Generational deferred to ROADMAP §89.2. -- **3-layer allocator**: `infra_alloc` (GPA, process-lifetime) + `node_arena` (Arena, per-program lifetime — Reader Form / Analyzer Node) + `gc_alloc` (mark-sweep, Value-lifetime). -- **cw v0 free-pool optimisation inherited** — intrusive per-(size, alignment) free list, 3-7× speed-up on `gc_stress` / `nested_update` (cited verbatim under §3 Block C). -- **cw v0 D100 5 root-set gaps pre-enumerated** — (a) macro-expansion lazy-seq realisation paths, (b) ProtocolFn / MultiFn inline caches, (c) refer()-borrowed string pointers, (d) closure-captured Values in macro callFnVal, (e) valueToForm intermediate trees. cw v0 patched these late; cw v1 lists them in the root-source table from day 1. -- **zwasm v2 integration heap layout**: cw heap and Wasm linear memory are **separate spaces**; zwasm internal bookkeeping accepts a cw GC allocator at `Engine.init(allocator)` so the dual-GC lifecycle mismatch (cw v0 D110) does not recur. `wasm_module` / `wasm_fn` are cw-GC-managed Values; zwasm metadata lives underneath them. - -The 5.0 cleanup audit (ADR-0026 + `private/notes/phase5-skeleton-audit.md`) bound 8 input constraints; bullets #1 / #3 / #5 / #6 are ADR-0028's territory and are quoted verbatim in §"Inputs from 5.0 audit" below. - -cw v0 archaeology at `private/notes/phase5-5.1-survey.md` produced three load-bearing findings that change cw v1's shape vs. cw v0: - -1. cw v0 stores mark bits **externally** in `allocations: AutoArrayHashMapUnmanaged(usize, AllocInfo)` (`gc.zig:181-185, 296-301`). cw v1 stores marks **inline** per ADR-0009 + F-006; the spare 29 bits (1 mark used out of 30) are a free-pool size-class hint per F-006's inheritance language. -2. cw v0 overlays the free-pool `FreeNode { next }` at **offset 0** of the freed payload (`gc.zig:158-162`) — clobbering the header. cw v1's mandatory `HeapHeader` at offset 0 forces the overlay to **offset 8** (after the 8-byte header). -3. cw v0 calls **no per-tag finaliser** for BigInt (`gc.zig:1125-1129` literally just `markPtr(bi)`). `std.math.big.int.Managed`'s limb array leaks silently until the GC heap reclaims the backing — fragile when the limbs sit on a different allocator. cw v1 mandates a per-tag finaliser table from day 1. - -cw v0 also takes a `suppressCollection()` escape hatch (`analyzer/analyzer.zig:567-571`) for D100 root causes #5 and #6 (macro-expansion lazy-seq closure captures + half-built valueToForm trees) — turning GC off across the macro-expansion bracket. cw v1 refuses the escape hatch per F-002 (finished form wins): macro-expansion roots must be enumerable, not suppressed. - -## Decision - -### 1. GC strategy: single-generation tracing mark-sweep - -Stop-the-world tracing collector, two phases: - -- **Mark**: visit every root, recursively trace through GC-managed pointers, set `HeapHeader.gc_and_lock.gc_mark` bit 0 to 1 on every reached object. Per-tag dispatch (§5 trace table) decides which pointer fields to follow. -- **Sweep**: walk the GC heap's allocation list. For every object with `gc_mark == 0`: call the per-tag finaliser (§4), unlink from the live list, push onto the matching `(size, alignment)` free pool (§3). For every object with `gc_mark == 1`: clear the bit and keep. - -Trigger: `gc_alloc.alloc` checks `bytes_since_last_gc > threshold` with an **adaptive threshold** (per Devil's-advocate Load-bearing concern #2): `threshold = max(1 MiB, last_live_bytes * 2)`. The 1 MiB floor is for tests / cold startup; the doubling rule keeps growth amortised once a REPL or batch process has a live-set baseline (avoids the "first def triggers a full GC mid-load on a 4 MiB Clojure source" failure mode the concern flagged). Env-overridable for tests. Explicit `(System/gc)` Tier A in Phase 7 calls the same entry point. - -Concurrency: Phase 5 ships single-threaded (the `Runtime.io` defaults to the `io_default` single-thread accessor per Block A). The collector takes no mutex because there is no concurrent mutator. Phase 15 STM activation revisits this; the GC mutex hook lives behind `Runtime.gc_mutex: std.Io.Mutex = .init` (declared but not consulted at Phase 5). - -### 2. 3-layer allocator boundary - -Per F-006 + ADR-0017, three distinct allocators with non-overlapping lifetimes: - -| Layer | Allocator | Lifetime | What lives here | -|--------------|---------------------------|--------------------|---------------------------------------------------------------------------------------------------------------------------------------------------| -| 1 (longest) | `infra_alloc` = GPA | Process | `Runtime` itself, `Namespace`, `Var`, `TypeDescriptor` (incl. `method_table` slice), interned `keyword` table | -| 2 (middle) | `node_arena` = Arena | Per-program / eval | Reader `Form` tree, Analyzer `Node` tree, per-eval scratch buffers | -| 3 (shortest) | `gc_alloc` = GcHeap.alloc | Value-collected | All Heap Values (`string` payload, `vector` HAMT nodes, `hash_map` nodes, `lazy_seq` cells, `big_int` wrappers, `TypedInstance.field_values`, …) | - -Pointer ownership rule (per F-006): a Value field that **owns** another Value is `gc_alloc`. A Value field that **borrows** structurally-shared infrastructure is `infra_alloc`. Crossing the boundary requires an explicit copy-to-`node_arena` at the hand-off (cw v0 D100 fix (1) — `valueToForm` copies GC-allocated string data into `node_arena` before returning). - -`TypeDescriptor` itself is **`infra_alloc`** (namespace-lifetime — survives every GC); `TypedInstance.field_values: []Value` is **`gc_alloc`** (the slice's backing memory is GC-traced, and each element is a Value visited by §5 trace). This split satisfies audit bullet #3. - -**Block B reconciliation** (per Devil's-advocate Alt 2 item 4): Block B's cw v0 comment reads "GPA: infrastructure (never collected)". The "never collected" predicate applies to the **allocator state** (no mark-sweep over `infra_alloc` itself; cw v0's GC heap does not trace into GPA-owned memory). It does **not** forbid individual allocations on `infra_alloc` from being freed — explicit `infra_alloc.free(...)` from a per-tag finaliser (§4) is the normal lifetime-end path for the `std.math.big.int.Managed` limbs that F-005 places on `infra_alloc`. Without this clarification, a 5.3 GC implementation owner reading Block B in isolation may infer that finalisers cannot free. - -### 3. Free-pool recycling (inherited from cw v0; offset-8 overlay) - -After sweep, dead objects are not immediately returned to the GPA. They are pushed onto per-(size, alignment) intrusive linked lists kept on `GcHeap`: - -```zig -const FreeNode = struct { next: ?*FreeNode }; -const FreePoolKey = struct { size: usize, alignment: u8 }; -free_pools: std.AutoHashMapUnmanaged(FreePoolKey, ?*FreeNode) = .empty, -``` - -On the next `alloc(size, align)` call, the heap pops the matching pool's head in O(1) before falling back to `infra_alloc.rawAlloc`. cw v0's measured impact is the textbook (Block C, verbatim): - -``` -// Impact: gc_stress 324ms -> 46ms (7x), nested_update 124ms -> 41ms (3x). -``` - -**DIVERGENCE from cw v0** (per F-002 + ADR-0009): cw v0 overlays `FreeNode` at offset 0 of the freed payload (`gc.zig:158-162`), clobbering the `HeapHeader`. cw v1's `HeapHeader` is **invariantly at offset 0** (ADR-0009 mandate; the GC mark phase reads the header to decide tracing). The overlay slot therefore moves to **offset 8** (immediately after the 8-byte header): - -```zig -// Layout when an object is on a free pool: -// bytes [0..7] : HeapHeader (preserved; gc_mark cleared by sweep) -// bytes [8..15] : FreeNode { next: ?*FreeNode } -- overlaid in payload -// bytes [16..] : remainder of original payload (uninitialised garbage) -// -// Constraint: freed block payload >= @sizeOf(FreeNode) = 8 bytes. -// Heap allocations are min-16 bytes (header + 8) to satisfy this. -``` - -The 16-byte minimum is enforced in `gc_alloc.alloc`: requests below 16 bytes round up. Header preservation lets a future `(System/identityHashCode)` Tier B reuse the header even after free-and-realloc. - -### 4. Per-Tag dispatch infrastructure (descriptor + trace + finaliser) - -Per Devil's-advocate Alt 1 + cross-coupling §"Triple Tag-indexed table" consolidation: three comptime tables, each `[64]?*const X` indexed by `@intFromEnum(Tag)`, live together in `runtime/gc/tag_ops.zig` (alternative shape: a single `TagOps = struct { descriptor: ?*const TypeDescriptor, trace: ?*const TraceFn, finaliser: ?*const FinaliserFn }` — choice of "three parallel arrays" vs "single struct-of-arrays" deferred to §9.7 row 5.3 owner per F-003; both index by Tag, both are constant-time, both build at comptime). - -```zig -// Descriptor table — read by 5.11 TypeDescriptor activation + class-of dispatch. -pub const tag_descriptor_table: [64]?*const TypeDescriptor = blk: { - var t: [64]?*const TypeDescriptor = @splat(null); - t[@intFromEnum(Tag.string)] = &native_descriptors.string; - t[@intFromEnum(Tag.vector)] = &native_descriptors.vector; - // ... one entry per primitive-mapped Tag (5.11 fills) - break :blk t; -}; - -// Trace table — called from mark phase, walks outgoing GC-managed pointers. -pub const tag_trace_table: [64]*const fn (gc: *GcHeap, header: *HeapHeader) void = ...; - -// Finaliser table — called from sweep before unlink + free-pool push. -pub const tag_finaliser_table: [64]?*const fn (header: *HeapHeader) void = ...; -``` - -cw v0's class-of is a string switch (`predicates.zig:387-390`) — cw v1 replaces it with constant-time `tag_descriptor_table[@intFromEnum(value.tag())]` per F-004 + ADR-0007. - -**Tags that need a finaliser today (Phase 5 + foresight)**: - -- `big_int` / `ratio` / `big_decimal`: free `std.math.big.int.Managed` limbs back to `infra_alloc` (the limbs live on GPA per F-005's stdlib-affine internals; the Block B reconciliation in §2 documents that this `infra_alloc.free` is the normal path, not a violation of "never collected"). -- `wasm_module` / `wasm_fn`: drop zwasm v2 `Module` / `FuncEntity` references (Phase 16 entry wires the bodies; the slot stays declared and the table entry stays `null` until then — see "Sweep ordering invariant" below for the contract Phase 16 entry must honour before any allocation lands). -- `ex_info`: arena message-buffer release — only if the message string lives outside `gc_alloc`; current Phase 3 shipping path keeps it in arena so no finaliser entry. -- `host_instance`: dispatches through `TypeDescriptor.finaliser` (the descriptor knows the host class's deinit shape per ADR-0011). - -**Sweep ordering invariant**: sweep calls `tag_finaliser_table[tag](header)` **before** unlinking from the live list and **before** pushing onto the matching free pool. Finalisers are **no-alloc** — they may only `infra_alloc.free` / external deinit / no-op; allocation through `gc_alloc` from inside a finaliser is a Phase-5-enforced panic (`std.debug.panic("gc_alloc from finaliser")`). No inter-finaliser ordering guarantee (sweep walks the live list in allocation order; finaliser A's side-effects must not depend on whether finaliser B has run). Per Devil's-advocate Load-bearing concern #1 + Alt 2 item 3. - -**Phase 16 entry contract** (per Devil's-advocate Load-bearing concern #6): when Phase 16 lands the first `wasm_module` allocation, the `tag_finaliser_table[wasm_module]` entry **must be set first** — the order is (i) finaliser body lands, (ii) table entry flips from `null` to the deinit fn, (iii) first allocation. The `null`-during-Phase-5 stub is guarded by the access pattern (`if (tag_finaliser_table[tag]) |f| f(header);`) so a missed step at Phase 16 entry causes a silent leak, not a crash — the contract above prevents the leak. - -**DIVERGENCE from cw v0** (per F-002 + F-005): cw v0 has no finaliser dispatch (`gc.zig:1125-1129` just `markPtr(bi)`); BigInt limbs sit on the GC heap and get reclaimed only when the next sweep visits them, with no explicit `deinit` call. The fragility shows up when the limbs migrate to a different allocator (e.g. Ratio's two BigInts). cw v1's finaliser table is the day-1 plug for the gap. - -### 5. Per-tag GC-trace dispatch (root set + transitive trace) - -The trace table from §4 (`tag_trace_table: [64]*const fn (gc: *GcHeap, header: *HeapHeader) void`) is consulted by the mark phase. Each entry walks the type-specific outgoing pointers and calls `gc.mark(child)` on each. - -**Mark cycle invariant** (per Devil's-advocate Load-bearing concern #3): `gc.mark(child)` checks `child.header.gc_and_lock.mark == 1` before descending; the mark bit doubles as the visited-flag during the mark phase. Cycles (e.g. `LazySeq` whose `thunk` captures another `LazySeq` whose `seq_cache` points back) terminate at the second visit. - -**Root sources + tag-trace entries** (mark phase consumes both; entries are tagged with their **shape**: `E` = entry-point root walker, `T` = per-Tag trace entry registered into `tag_ops.tag_trace_table` from the owning module, `D` = documentary / closed-by-construction / no GC edge ever). See amendment 1 below for the demotion rationale. - -| # | Source | Shape | Reason | -|----|-----------------------------------------------------------------------------|-------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| 1 | `Runtime.envs.entries` — all Namespace `Var` roots | E | Vars are rooted **at `def` time**; `ns-unmap` (Phase 7+) removes the root atomically | -| 2 | `Env.current_frame` — dynamic binding stack (`env.zig` threadlocal) | E | `binding` form Values must survive across `*pushFrame*` boundaries | -| 3 | `Fn.closure_bindings` on every live `fn_val` | T | Not a root — Fn objects become reachable via Var (source #1) graph. `tag_trace_table[.fn_val]` registered from `tree_walk.zig` walks `closure_bindings`. cw v0 D100 #5 closed by transitive trace. | -| 4 | `LazySeq.thunk` + `LazySeq.ctx` + `LazySeq.seq_cache` | T | Not a root — LazySeq reachable via Var graph. `tag_trace_table[.lazy_seq]` registered at 5.7 alongside `force()`. cw v0 D100 #5 closed by transitive trace. | -| 5 | `ProtocolFn` / `MultiFn` inline caches (`cached_type_key`, `cached_method`) | D | **No live structs in cw v1** (zero grep hits). Phase 7 entry creates them; the cache state will live on `CallSite` (row 9), not embedded in the protocol fn (cw v1 splits per ADR-0008). | -| 6 | `refer()` borrowed symbol-name pointers → owned slice on `infra_alloc` | D | **Closed at construction**: `env.zig:229 referAll` already dupes the key onto `self.alloc` (= `rt.gpa`). cw v0's late-patch is unnecessary in cw v1. Var pointer rooted via source #1's `mappings`. | -| 7 | Macro-expansion scratch: `macro_root_slot` | E | cw v0 D100 #6 — `threadlocal var macro_root_slot: ?Value` declared in `runtime/gc/root_set.zig` (Layer 0); analyzer/macro_dispatch sets/clears via downward import. Refuses cw v0 `suppressCollection`. | -| 8 | `TypedInstance.field_values` (transitively from descriptor reachability) | T | Not a root — TypedInstance reachable via Var graph. `tag_trace_table[.typed_instance]` registered at 5.11 alongside `lookupMethod`. Audit bullet #3 transitive trace. | -| 9 | `CallSite.last_method` | D | Cache slots point at **namespace-owned** TypeDescriptor + MethodEntry (process-lifetime, not GC-managed). **No GC edge ever** — cw v1 splits the cache off the protocol fn so it carries no Value. | -| 10 | `GcHeap.permanent_roots: std.ArrayList(Value)` | E | Escape valve for embedder-pinned Values (FFI / test fixtures / future REPL prompt buffer). `GcHeap.pin`/`unpin` API. | - -**Live entry-point walkers (E)**: 4 — rows 1, 2, 7, 10. -**Tag-trace entries (T)**: 3 — rows 3, 4, 8 (registered into `tag_trace_table` from owning module). -**Documentary (D)**: 3 — rows 5, 6, 9 (no walker body, no trace entry; preserved as design invariants). - -> **Amendment 2 (2026-06-05, Phase B #3b — ADR-0091).** Rows 2 -> (`current_frame`) + 7 (`macro_root_slot`) are **subsumed into a single -> `thread_roots` E-source** that walks, per live thread (self + every -> registered worker `ThreadGcContext`), its binding-frame chain + macro slot -> + **VM operand-stack frame chain** (`vm.eval`'s `stack[0..sp]` + `locals`, -> newly rooted for Phase B real threads, #3b-step1) + an **in-flight -> fabrication self-guard** (`gc_self_guard`, #3b-step2b — the -> `op_vector/map/set_literal` accumulator window). Live E-walkers 4 → 3 (rows 1, the -> `thread_roots` union, 10); the `RootSource` Zig enum count 10 → 9. The -> per-thread union-addressing (`frameSourceAt`/`macroSourceAt` + the `src_idx` -> cursors from amendment-era #3a) is commonized into one `threadContextAt` -> helper + one thread-major cursor. Rationale + the Devil's-advocate fork -> (Option A fold rejected; Alt 1 clean-11th-source vs Alt 2 union vs Alt 3 -> thread-driven spine) live in **ADR-0091**. See the Revision history entry -> below. - -Per Devil's-advocate Alt 2 item 2: Var rooting is committed to "at `def` time" (cw v0's `permanent_roots` escape valve at row 10 was a late-answer workaround; cw v1 commits the timing in row 1). Per Load-bearing concern #4: `refer()` owned-slice lives on `infra_alloc` (per-namespace lifetime; explicit ownership stated). Per Load-bearing concern #5: row 9 demoted to D (no GC edge ever) so the "reserved as `null`" framing is dropped entirely — the cache holds no GC-managed pointers in any Phase. - -**Macro-expansion root slot** (refuses the cw v0 suppress escape hatch): `Analyzer.macro_root_slot: ?Value` is set before `callFnVal` for macro expansion, cleared after `valueToForm` completes. The mark phase walks it like any other root. This satisfies F-006's "5 root-set gaps must be pre-enumerated" requirement for cases (5) and (6) — the two cw v0 patched with `suppressCollection`. - -### 6. Header bit layout (`HeapHeader.gc_and_lock` 30 spare bits) - -`gc_and_lock: packed struct(u32) { lock_state: u2, gc_mark: u30 }` (ADR-0009). -Per Devil's-advocate Alt 1: this ADR commits **only** the bits the Phase-5 mark phase consumes; the partition of the remaining 28 bits is **deferred to §9.7 row 5.3 GC implementation owner** per F-003 decision-deferral (the size-class distribution that drives the split needs measurement on real Clojure workloads — 5.3 owner has the bench harness; 5.1 ADR draft does not). - -| Bit range | Field | Purpose | -|-----------|--------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| 0 | `mark` | Set during mark phase, cleared during sweep; doubles as cycle-visited flag (per §5 invariant) | -| 1 | tri-colour reserve | Declared, unused at Phase 5; reserved for Phase 7+ incremental-GC upgrade candidate | -| 2..29 | owned by 5.3 | **Partition deferred to §9.7 row 5.3 owner per F-003**; expected use = `size_class` field for free-pool key collapse (see §3) + future generational tag / weak-ref bit / allocation timestamp; the exact width is row 5.3's call based on measured size-class distribution | - -The free-pool key collapse from `(size, alignment)` to a single header-resident `size_class` index (which lets `free_pools` shrink from `HashMap` to `[N]?*FreeNode` flat array) is the **expected** consumer of bits 2..N — the row 5.3 owner picks N (Devil's-advocate Alt 3 wildcard sketch suggests 5 bits = 32 classes; the draft does not commit to it). The wildcard's quantisation-vs-exact tradeoff is row 5.3 owner's measurement. - -**DIVERGENCE from cw v0** (per F-006 inheritance language): cw v0 has no header bits at all — marks live in an external hash-map (`gc.zig:181-185`). cw v1's inline layout removes the hash-lookup per access. Audit bullet #5 is satisfied "in spirit" (the choice exists in writing — mark bit + cycle invariant + free-pool collapse target) but the exact partition for bits 2..29 lives with row 5.3. - -### 7. zwasm v2 allocator-injection seam (foresight, F-001 + F-008) - -`Runtime.gc.allocator()` exposes the cw GC as a `std.mem.Allocator` that satisfies the zwasm v2 §1 strict-pass: `zwasm.Engine.init(rt.gc.allocator(), .{})`. zwasm metadata (Module / FuncEntity / Instance state) is then allocated through the same heap as cw Values, eliminating cw v0's D110 "dual-GC lifecycle mismatch". `wasm_module` / `wasm_fn` Tag-D slots' finalisers (table from §4) drop the zwasm reference; the per-allocation GPA underneath is the same for both. - -Phase 5 wires the seam at declaration only (the `gc.allocator()` method exists; no consumer yet). Phase 16 entry consumes it per F-008 + D-036. - -### 8. Activation scope (Phase 5 source rewrite) - -Per ADR-0017 amendment 1 + ADR-0009 amendment 2 the Phase 5 activation rewrites every Phase 1-4 heap allocation site: - -- `runtime/value.zig` (and the post-5.2 `runtime/value/` split) — every `HeapHeader.init` call site routes through `gc_alloc` instead of `gpa.allocator()`. ~20 grep sites today. -- `runtime/collection/{string, list, ex_info}.zig` — `gpa.create(T)` → `gc.alloc(T)`; `gpa.dupe(u8, …)` → `gc.dupe(u8, …)`. ex_info message string stays in arena per ADR-0017 amendment 1 explicit carve-out. -- `runtime/lazy_seq.zig` (today's 4.24 skeleton) — `LazySeq` struct + thunk closure allocation routes through `gc.alloc` at 5.7. -- `runtime/numeric/big_int.zig` — wrapper allocation through `gc.alloc`; `std.math.big.int.Managed` continues to take `infra_alloc` per F-005 (the wrapper carries `infra_alloc` as the limbs' allocator; the finaliser releases through the same). -- `Catalog Codes removed by §9.7 row 5.15`: ADR-0017 amendment 1's `gc_*_not_supported` family + ADR-0009 amendment 2's `locking_*_not_supported` are removed at the `build_options.phase_at_least_5 = true` flip. The `locking_*` family stays — only the `gc_*` Codes go (lock_state activation is Phase 15 per ADR-0009). - -The rewrite is expected per ROADMAP §A25 + F-002. Depth 3 (cross-file refactor of alloc call sites) is typical; depth 4 only if the per-eval arena / GcHeap boundary itself needs an ADR revision. - -### 9. What this ADR does NOT decide - -- Tag enumeration / slot map — owned by ADR-0027 (paired). -- LazySeq `force()` mutex shape — deferred to §9.7 row 5.7 per audit bullet #2. -- Generational GC — F-006 explicit deferral; ROADMAP §89.2. -- Lock state activation (`gc_and_lock.lock_state`) — Phase 15 per ADR-0009. -- D-040 `MethodEntry` rename — Phase 7 per debt row. -- Phase 16 inline-vs-Pod choice for `wasm_funcref` / `wasm_externref` — D-036; the finaliser table reserves D4-D7 with `null` until then. - -## Inputs from 5.0 audit - -Bullets #1 / #3 / #5 / #6 from `private/notes/phase5-skeleton-audit.md` §"5.1 input bullets" bind this ADR. Quoted verbatim (paraphrase-loss protection per ADR-0026 §3): - -> 1. GC root-set must enumerate `LazySeq.thunk` + `LazySeq.ctx` + `LazySeq.seq_cache`. Today the skeleton uses `std.atomic.Value(?*SeqOpaque)` (anyopaque) for the cache — the GC marker cannot blindly trace an opaque pointer, so the mark path will need either (a) a tag-bit on the cache slot saying "this is a Seq*", or (b) a per-LazySeq sub-tag stored on the header. cw v0's F-006 "macro-expansion-time lazy-seq realisation paths" gap is exactly this. The ADR-0028 root-source list must name `LazySeq` explicitly. -> 3. TypeDescriptor's `method_table: []const MethodEntry` slice must land in a GC-aware allocator, because the namespace owning it is itself process-lifetime (`infra_alloc` per F-006). Slot placement decision: the descriptor itself is namespace-owned (not GC-managed), but a `TypedInstance.field_values: []Value` slice IS GC-managed (mark walks the slice). The GC ADR must distinguish "TypeDescriptor live forever" vs "TypedInstance is a GC root entry-point through its descriptor pointer". -> 5. `HeapHeader.gc_and_lock.gc_mark: u30` is 30 bits today. F-006 commits to single-generation mark-sweep; 30 bits is overkill for a one-bit mark (29 spare). The ADR-0028 mark phase needs at most 1 bit (or 2 for tri-colour); the remaining bits are a free-pool size-class hint candidate per cw v0's free-pool optimisation. 5.1 should record the choice of "what the other 29 bits carry" so 5.3 (GC impl) doesn't re-decide it under task pressure. -> 6. `HeapHeader` is 8 bytes today (tag + flags + pad + gc_and_lock). F-006 free-pool needs a `next: ?*HeapHeader` link somewhere when an object sits on the free list — either inside the freed object's payload (cw v0 path) or a separate per-size-class chain. 5.1 should at least note which the ADR-0028 commits to. - -Bullet #1 → §5 row 4 (LazySeq enumerated explicitly; the `seq_cache: std.atomic.Value(?*SeqOpaque)` tracing question resolves as "the cache slot's pointer is a `*Cons` post-realisation; per-LazySeq sub-tag stored on the cell at allocation lets the trace walk follow it"; 5.7 owner finalises the encoding). -Bullet #3 → §2 (`TypeDescriptor = infra_alloc`; `TypedInstance.field_values = gc_alloc`). -Bullet #5 → §6 (1 bit mark + 1 bit tri-colour reserve + 5 bit size_class + 23 bit reserved). -Bullet #6 → §3 (intrusive at offset 8, after header; 16-byte minimum allocation). - -## Verbatim cw v0 reference (paraphrase-loss protection) - -### Block A — `io_default.zig:9-21` rationale (the Zig-0.16 `std.Io.Mutex` constraint used in §1 + bullet #2 deferral) - -``` -//! Process-wide default `std.Io` accessor. -//! -//! Zig 0.16 removed `std.Thread.Mutex` and friends; the replacement -//! `std.Io.Mutex` requires an `io` argument for lock/unlock. CW carries -//! many module-level mutexes (interned keywords, hooks, namespaces, etc.) -//! that don't have access to an `init.io` value at the call site. -//! -//! This module exposes a single shared `std.Io` that defaults to a -//! single-threaded io suitable for tests and pre-init code paths. -//! Production entry points (main, cache_gen) call `set(init.io)` early -//! to upgrade the shared io to the real cancelable one used by -//! `thread_pool.zig`. After that, every mutex picks up the production io. -``` - -cw v1 adopts the same `io_default` shape when row 5.7 picks `std.Io.Mutex` for `LazySeq.force()`. Recorded here so §9.7 row 5.7 owner does not re-derive. - -### Block B — `gc.zig:153-156` 3-allocator comment (the textbook reference for §2) - -``` -/// Three-allocator architecture (D69-D70): -/// - GPA: infrastructure (never collected) -/// - node_arena: AST nodes (freed per-eval) -/// - GC allocator (this): runtime Values (mark-sweep collected) -``` - -cw v1 names the layers `infra_alloc` / `node_arena` / `gc_alloc` (§2) — semantics match cw v0 exactly. - -### Block C — `gc.zig:158-162, 208-219` FreeNode + measured impact (§3) - -``` -/// Free-list node — overlaid directly on freed allocation memory. -/// The freed block must be >= @sizeOf(FreeNode) to be recyclable. -const FreeNode = struct { - next: ?*FreeNode, -}; - -// --- Free-pool recycling --- -// Dead allocations from sweep are not immediately freed back to the OS. -// Instead, they are cached in per-(size, alignment) free pools. On the -// next allocation of the same size, the pool provides a recycled block -// in O(1) — a simple linked-list pop. This avoids the full GPA -// rawAlloc/rawFree overhead (which includes page-level bookkeeping). -// -// The FreeNode struct is overlaid directly on the freed memory (intrusive -// linked list), so no extra allocation is needed to track free blocks. -// -// Impact: gc_stress 324ms -> 46ms (7x), nested_update 124ms -> 41ms (3x). -``` - -cw v1 lifts the data structure and the perf rationale; the **DIVERGENCE** (§3) is the offset-8 placement so the header stays intact. - -## Alternatives considered - -A fresh-context `general-purpose` subagent was forked as Devil's advocate per CLAUDE.md § "ADR-level designs are handled inline" (depth ≥ 2 mandate). Brief: F-001..F-008 envelope; produce 3 alternative shapes (smallest-diff / finished-form-clean / wildcard) per ADR; do not propose F-NNN-violating alternatives. Full output at `private/notes/phase5-5.1-devils-advocate.md` (~440 lines). The ADR-0028 portion is reflected here verbatim. - -### Would-violate-F-NNN findings - -**None.** All three ADR-0028 alternatives stay within F-001 / F-005 / F-006 / F-008. - -### Subagent summary judgement (ADR-0028 portion, verbatim) - -> ADR-0028 (mark-sweep GC + 3-layer allocator) — substantively correct on the big shape (mark-sweep, three layers, offset-8 free node, per-tag finaliser, root-set enumeration). But it has the **most concentrated reservation-as-bias smell in the entire pair** at §6 (23 reserved bits in `gc_mark`), and one **load-bearing omission** at §5 row 1 (`Runtime.envs.entries` Var roots): Vars hold root Values, but the ADR does not name whether a freshly def'd Var that is *not yet referenced* is reachable from the namespace map — cw v0's `permanent_roots` workaround exists precisely because this question was answered late. - -### Alt 1 — Smallest-diff (verbatim) - -> **Sketch**: Land mark-sweep, 3-layer alloc, root-set table, finaliser table, offset-8 overlay — but **defer the §6 bit allocation past bit 0 (mark) to the 5.3 GC implementation owner**. Specifically, §6 in this ADR commits to: bit 0 = mark (used); bit 1 = tri-colour reserve (declared, unused at Phase 5); bits 2–29 = "owned by 5.3, decided when allocator + free-pool body lands". -> -> Same approach the ADR already takes for "LazySeq `force()` mutex shape — deferred to §9.7 row 5.7 per audit bullet #2". The bit-2..29 split is exactly the same kind of decision: it depends on the free-pool's measured size-class distribution, which is *5.3 owner's measurement to make*. Audit bullet #5 says "5.1 should record the choice of 'what the other 29 bits carry' so 5.3 doesn't re-decide it under task pressure" — but the *finished* free-pool may want a different split (e.g., 8-bit size_class + 22 reserved if Phase 7 transducers spawn many distinct sizes). Reserve the bits in the ADR text (don't burn them on a guess); 5.3 lands the split. -> -> **Better than current draft**: Removes the **23 reserved bits** smell at §6 (the largest Reservation-as-bias surface in either ADR). Aligns with F-003 (decision-deferral on structural plans). Removes the dual "finaliser table commits to N entries" / "size_class is 5-bit" double-binding. -> -> **Breaks**: Audit bullet #5 ("5.1 should record the choice") is partially un-satisfied — record only "1-bit mark + 1-bit tri-colour reserve; 28 bits available, partitioned at 5.3". The bullet is satisfied in spirit (the choice exists in writing) but not in letter. Acceptable per F-003. -> -> **Cost**: tiny. - -### Alt 2 — Finished-form-clean (verbatim) - -> **Sketch**: Alt 1 plus: -> -> 1. **Split ADR-0028 into ADR-0028a (mark-sweep GC body) and ADR-0028b (3-layer allocator boundary).** ADR-0020 governance says "one ADR, one load-bearing decision". 0028 has *two*: (a) mark-sweep + free-pool inherited and (b) 3-layer alloc boundary with TypeDescriptor on infra and TypedInstance on gc. These are independently amendable. -> 2. **Name the `Runtime.envs.entries` Var-root question explicitly.** §5 row 1 says "Var deref is the primary Value entry; ns survives forever" — but does *defining* a Var promote its initial Value to a root, or only does *deref* through the namespace map? cw v0's `permanent_roots` escape valve exists because cw v0 answered this late. cw v1 should commit: "Vars are rooted at `def` time; un-interning a Var (Phase 7+ `ns-unmap`) removes the root atomically." -> 3. **Add finaliser-ordering rule.** §5 sweep description says "call the per-tag finaliser, unlink from the live list, push onto the matching free pool". What if the finaliser *itself* allocates (e.g., logs the deinit somewhere)? The ADR should commit: "finalisers are no-alloc — they may only `infra_alloc.free` or no-op; allocation through `gc_alloc` from a finaliser is a Phase-5-enforced panic." -> 4. **Reconcile Block B contradiction.** Block B (cw v0 comment) says "GPA: infrastructure (never collected)" but cw v1 §4 finaliser releases `Managed` limbs back to `infra_alloc`. The ADR should write one sentence: "Block B's 'never collected' applies to the infra_alloc *as an allocator state* (no mark-sweep over it), not to individual allocations made on it; explicit `infra_alloc.free` from a finaliser is the normal path." -> -> **Better than current draft**: Each split ADR is amendable independently. The Var-root and finaliser-ordering rules close two of the cw v0 D100 gaps that F-006 names. Block B is no longer a contradiction the next session has to re-resolve. -> -> **Breaks**: More ADR numbers. The "paired ADR" framing with 0027 becomes a triplet. ADR-0029 was reserved for `runtime/value/` split aliasing. -> -> **Cost**: medium. - -### Alt 3 — Wildcard (verbatim, abridged) - -> **Sketch**: **Drop the per-(size, alignment) free-pool and use a single per-size-class freelist with the size_class stored on the header (already in §6).** Quantise allocation sizes to 32 size classes (powers-of-two from 16 to 4 KiB, then large), store the class on the header, and the free pools become `[32]?*FreeNode` — a flat array, no hash lookup at all. -> -> **Why within F-NNN envelope**: F-006 says "intrusive linked list per (size, alignment) free pool" — but the (size, alignment) wording was the cw v0 inheritance language. The *finished form* on cw v1 has fixed alignment, so the free-pool's (size, alignment) shape over-keys. -> -> **Better than current draft**: alloc fast-path becomes `pool[class].pop() orelse fallback()` — no hash compute, no key allocation. 23 reserved bits at §6 shrink to ~18. -> -> **Breaks**: Quantisation wastes memory for in-between sizes (estimate: 5-15 %). "32 size classes" is a magic number. Need a benchmark. -> -> **Cost**: medium. - -### Load-bearing concerns the ADR omits (verbatim) - -> 1. Sweep ordering vs finaliser side-effects. If finaliser A causes finaliser B to need re-running, sweep diverges. Commit "finalisers are independent; no inter-finaliser ordering guarantee". -> 2. Trigger threshold tuning. §1 says "default 1 MiB, env-overridable for tests". 1 MiB is fine for tests; for a REPL session loading a 4 MiB Clojure source, the first def triggers a full GC mid-load. Either name a higher default or commit to "adaptive: trigger = max(1 MiB, last_live_bytes * 2)". -> 3. Cycle handling for `LazySeq.seq_cache`. The ADR should commit: "mark recursion checks `header.gc_mark == 1` before descending; the bit doubles as visited-flag during mark phase." This is implied but not stated. -> 4. `refer()` borrowed string pointers — "move to owned-slice (no longer GC-traced) at refer time" says the data shape changes at refer time. Who owns the new slice? `infra_alloc` is the natural choice but not stated. -> 5. The §5 row 9 `CallSite.last_method` reservation. F-003 says reservations on structural plans defer to the owning Phase. Either move row 9 to a "Phase 7 adds these roots" note, or commit to "the slot is reserved as `null` until 5.x activates" — don't leave it ambiguous. -> 6. §4 finaliser for `wasm_module` / `wasm_fn` — text says "the slot is declared and the table entry stays `null` until then [Phase 16]". But F-001 says zwasm v2's `Engine` accepts the cw GC allocator at init; if zwasm v2 metadata is allocated through `gc_alloc`, the *finaliser* must release the zwasm Module *before* the underlying memory is recycled. Commit: "Phase 16 entry MUST land the finaliser before the first `wasm_module` allocation". - -### Main loop disposition - -Devil's-advocate Alt 1 **applied**: §6 bit allocation deferred — only mark bit + tri-colour reserve committed; bits 2..29 partition is row 5.3 owner's call per F-003. Removes the 23-reserved-bits Reservation-as-bias smell (the largest in the pair). - -Alt 2 item 1 (ADR split into 0028a / 0028b) **not applied**: the two decisions are tightly coupled at the per-tag dispatch level (§4 consolidates descriptor + trace + finaliser tables, all indexed by Tag; the dispatch tables live inside the GC body and the allocator boundary). Splitting would force `tag_ops.zig` to live in one ADR while its consumer lives in another. Recorded as a future-supersession candidate for `D-043`-style debt if a future amendment touches only one of the two halves; for now the coupling argues for the single ADR. - -Alt 2 item 2 **applied**: §5 row 1 commits "Vars are rooted at `def` time; `ns-unmap` (Phase 7+) removes the root atomically". - -Alt 2 item 3 **applied**: §4 commits the no-alloc finaliser invariant + the panic-on-violation rule + no-inter-finaliser-ordering guarantee. - -Alt 2 item 4 (Block B reconciliation) **applied**: §2 carries the one-sentence reconciliation paragraph distinguishing allocator-state from individual-allocation lifetimes. - -Alt 3 (wildcard size-class quantisation) **not applied at Phase 5 entry** — the wildcard becomes row 5.3 owner's measurement-driven choice within the F-NNN envelope (§6's "expected use = size_class field for free-pool key collapse" reserves the design space without committing the quantisation). Recorded for 5.3 owner as the strongest candidate. - -Load-bearing concerns #1 (sweep / finaliser ordering) **applied** at §4 Sweep ordering invariant; #2 (trigger threshold tuning) **applied** at §1 (adaptive threshold formula committed); #3 (cycle handling) **applied** at §5 Mark cycle invariant; #4 (`refer()` ownership) **applied** at §5 row 6 (owned slice on `infra_alloc`); #5 (row 9 ambiguity) **applied** at §5 row 9 (explicit `null` reserve); #6 (Phase 16 wasm_module finaliser ordering) **applied** at §4 Phase 16 entry contract. - -Per Devil's-advocate cross-coupling §"Triple Tag-indexed table" + ADR-0027 §3 consolidation: **§4 absorbs `tag_descriptor_table`** (moved out of ADR-0027 §3); three tables co-locate as `tag_descriptor_table` + `tag_trace_table` + `tag_finaliser_table`, with `TagOps` struct-of-arrays as an explicit row 5.3 owner's alternative. - -### Amendment 3 (2026-07-02) — explicit gray worklist for the mark phase - -A fresh-context Devil's-advocate subagent was forked per the depth ≥ 2 -mandate. Bug: the original recursive `mark()` (mark → trace_fn → mark per -child) overflowed the native stack on a ≥~400k-deep cons chain -(`(count (repeat 1000000 1))` SIGSEGV during a mid-walk collect; verified -by `CLJW_GC_THRESHOLD_MB=4096` making the same expr return 1000000). The -draft fix was: on-GcHeap `mark_worklist` gray stack, bit-set-then-push, -auto-drain guarded by a `mark_draining: bool`, degrade-to-recursion on -append OOM. The DA's findings (reflected near-verbatim): - -**Would-violate-F-NNN findings: none.** F-006 commits to mark-sweep / -single generation / 3-layer allocator and says nothing about mark-phase -traversal mechanics. No alternative touches F-004 / F-005 / the ADR-0009 -8-byte-header invariant. - -**Verification of the draft's concerns**: (1) traversal-order change has -no observers — sweep walks `allocations` in allocation order; -`persistent_marks` bits are cleared before marking and re-shaded on -reach; gc_torture asserts deterministic UAF surfacing, not mark order; -and no registered trace fn recurses internally (all ~35 route descent -through `mark`), so the conversion fixes deep cons chains, RB trees, and -HAMT spines uniformly with trace fns byte-for-byte unchanged. -(2) The draft's OOM-fallback was its one unsound joint: infra-append OOM -correlates with a large heap, which is where the deep chain lives, so -the fallback reintroduces unbounded recursion exactly under memory -pressure — and because the bit is set at push time each object is pushed -at most once, so `allocations + persistent_marks` bounds the worklist -and a single `ensureTotalCapacity` in `collect()` deletes the OOM branch -entirely. (3) The `mark_draining` bool is race-free under the -`gc_mutex`-serialized STW collect but is a hidden control-flow -inversion. (4) An intrusive gray-list through `HeapHeader` was -investigated and rejected: a link needs 8 bytes, the header is locked at -8 bytes by ADR-0009 (offset-8 FreeNode overlay + 16-byte minimum -allocation key on it), and growing every live object permanently to -optimize a transient mark structure contradicts the side-table -precedent recorded on `GcHeap`. Pointer-reversal (Deutsch-Schorr-Waite) -also rejected: it needs per-tag knowledge of which field is mid-traversal, -incompatible with the opaque `tag_trace_table` fn-pointer design. - -**Alt 1 — smallest-diff**: the draft + guaranteed-capacity worklist -(reserve in `collect()`, delete the OOM fallback, keep auto-drain). -Better: removes the only unsound path; mark itself never allocates. -Breaks: the reserve can OOM — but before any bit is set, where aborting -the collect is a safe degradation (heap intact; pressure resurfaces at -the next `gc.alloc`). - -**Alt 2 — finished-form-clean (ADOPTED)**: Alt 1's capacity guarantee + -drop `mark_draining` entirely; `mark()` is push-only (shade gray) and -`collect()` gains one explicit `drainGray()` (blacken) after all three -root-publication passes (root-set loop, `current_tx`, -`markRegisteredTxs`), before `sweep()`. `collect()` reads as the -textbook tri-color structure — publish roots → drain → sweep — which is -the shape the §6 bit-1 tri-colour reserve and a future incremental / -parallel marker inherit; the draft's auto-drain would have to be undone -by that owner. Cost: direct `mark()` callers relying on transitivity -(two unit tests: `stm/tval.zig` self-loop, the new deep-chain test) add -a `drainGray` call — per F-002 the test edits are not a valid tiebreaker. - -**Alt 3 — wildcard**: allocation-free fixed 4096-slot mark stack + -Boehm-style overflow rescan of `allocations` **and `persistent_marks`** -(a rescan that misses waypoints would strand their children — the D-251 -class). Zero allocation during collect; attractive for a -memory-budgeted Wasm-edge profile (gap area II). Rejected now: worst -case O(passes × live) on wide structures, more invariants, a magic -constant needing a bench. On record as the edge-profile candidate. - -**Applied**: Alt 2, plus the DA's four amendment obligations — -(a) bit-before-push recorded as a LOAD-BEARING invariant in `mark`'s -docstring (push-before-bit would let a failed push leave a -traced-looking-but-untraced object → live child swept → UAF); -(b) `collect()`'s "mark/sweep never allocate" comment rewritten to -"never allocate through `gc.alloc`; worklist capacity reserved up front -on `gc.infra` before any bit is set"; (c) `mark_worklist` freed in -`GcHeap.deinit`; (d) Consequences note added: the worklist is STW-only -state — a future parallel marker (gap area III) replaces it. - -## Consequences - -- **Positive**: cw v0's `suppressCollection` escape hatch is rejected by enumeration (§5 root row 7). cw v0's silent BigInt limb leak is closed by per-tag finaliser dispatch (§4). cw v0's external mark hash-map is replaced by inline bits (§6) — one less heap-write per object per mark, one less hash-lookup per access. cw v0's free-pool perf inheritance is preserved (Block C) under the offset-8 overlay (§3). -- **Negative**: 16-byte minimum allocation wastes up to 8 bytes per small object (currently only `boolean_true` / `boolean_false` could fit smaller — but those are NaN-box immediates, not heap allocations; the real impact is `cons` cell at ~16 bytes — already at the minimum). Per-tag dispatch tables (§4 finaliser + §5 trace) add 1 KiB ROM (.const data) for the 64 slots × 2 tables × 8-byte function pointers — negligible. -- **Neutral / follow-ups**: §7 zwasm seam is declaration-only at Phase 5; Phase 16 entry consumes per F-008 + D-036. Generational GC stays an open question (D-016) decided by Phase 5 bench results (ROADMAP §10.2 quick bench window). -- **Amendment 3**: mark depth is O(1) native stack regardless of object-graph depth (deep cons chains / RB trees / HAMT spines all mark iteratively). `gc.mark_worklist` retains its high-water capacity across collects (peak 8 bytes per live non-leaf object during a collect; no shrink policy — add one if a one-off huge collect ever matters). The worklist is STW-only state; a future parallel marker (gap area III) replaces it, per the same forward note `fabrication_depth` carries. - -## Affected files - -- `.dev/decisions/0027_*.md` (paired — see §0). -- `src/runtime/gc/` ★new directory per `.dev/structure_plan.md` (F-006 decree): - - `mark_sweep.zig` (mark / sweep bodies + `GcHeap` struct) - - `root_set.zig` (root enumeration table) - - `free_pool.zig` (per-size-class intrusive list + offset-8 overlay) - - `arena_node.zig` (Arena allocator factory) - - `gc_strategy.zig` (vtable for the future Arena ↔ MarkSweep switch per ADR-0023) -- `src/runtime/value.zig` (and post-5.2 `runtime/value/heap_header.zig`) — `HeapHeader.gc_and_lock.gc_mark` bit layout per §6. -- `src/runtime/native_descriptors.zig` (paired with ADR-0027 §3) — gains finaliser hook references for native-class wrappers (BigInt / Ratio / BigDecimal initially). -- `src/runtime/numeric/big_int.zig` (post-5.9) — gains `deinitGc` finaliser; allocator boundary commits to `infra_alloc` for limbs. -- `src/runtime/collection/*.zig` — `gc.alloc` routing in §8. -- `src/eval/analyzer/macro_expand.zig` (post-5.13 split) — `Analyzer.macro_root_slot` set / cleared per §5 row 7. -- `src/runtime/error_catalog.zig` — `gc_*_not_supported` family removed at §9.7 row 5.15 flip. -- Test surface: `test/gc/{mark_sweep_basic,free_pool,finaliser,root_set}.zig` (5.3 task lands), plus rewrites of existing tests that today expect `Code.gc_alloc_not_supported`. - -## References - -- `.dev/project_facts.md` F-001 (zwasm v2 unavoidable), F-002 (finished form wins), F-003 (decision-deferral), F-005 (numeric tower internal allocator), F-006 (GC strategy + 3-layer alloc + cw v0 D100 5 gaps — decree), F-008 (zwasm v2 allocator strict-pass). -- `.dev/principle.md` § Bad Smell catalogue (Workaround smell — the cw v0 `suppressCollection` rejection in §5). -- `.dev/decisions/0009_object_header_heap_only_lock.md` (HeapHeader.gc_and_lock — §6 owns the 30-bit split). -- `.dev/decisions/0017_allocator_strategy.md` (the 3-layer skeleton — this ADR is the body). -- `.dev/decisions/0023_comptime_stub_pattern.md` (gc_strategy.zig vtable for future Arena/MarkSweep switch). -- `.dev/decisions/0026_phase5_entry_scope.md` (verdict table; this ADR is row 5.1's deliverable). -- `.dev/decisions/0027_nan_box_second_generation.md` (paired — §1 §3 §4 reference its slot map + Tag widths). -- `.dev/structure_plan.md` `runtime/gc/` decreed split (F-006). -- `.dev/debt.md` D-011 (this ADR satisfies), D-020 (header bit helpers — 5.3 lands), D-016 (generational re-eval after bench), D-036 (zwasm Pod-vs-inline — Phase 16). -- `private/notes/phase5-skeleton-audit.md` (5.1 input bullets — quoted above). -- `private/notes/phase5-5.1-survey.md` (cw v0 archaeology — §1-3 DIVERGENCE call-outs). -- `.claude/rules/zig_tips.md` § Mutex (the constraint behind Block A's `io_default` pattern; deferral to row 5.7). - -## Revision history - -- 2026-07-02 (amendment 3): mark phase converted from recursive per-child descent to an explicit gray worklist (`GcHeap.mark_worklist` + push-only `mark()` + `drainGray()` as a named `collect()` phase between root publication and sweep). Trigger: `(count (repeat 1000000 1))` SIGSEGV — a ≥~400k-deep cons chain overflowed the native stack when a mid-walk collect marked it recursively (confirmed via `CLJW_GC_THRESHOLD_MB` deferral); found by the 2026-07-02 debt/code-truth audit as an unledgered bug. DA fork (Alt 1 capacity-guaranteed auto-drain / Alt 2 explicit-drain tri-color [adopted] / Alt 3 fixed-stack + Boehm overflow-rescan [edge-profile candidate, on record]) reflected into Alternatives considered § Amendment 3. Load-bearing: bit-set-BEFORE-push; `collect()` reserves `allocations + persistent_marks` capacity before any bit is set (failed reserve aborts the collect safely); trace fns unchanged. Smell category: the original recursion was a latent Reservation-as-bias on §5's "recursively traces" wording — depth-unbounded recursion was never a recorded decision, just inherited shape. -- 2026-05-24: Status: Proposed → Accepted (autonomous-loop self-accept after Devil's-advocate subagent review reflected verbatim into Alternatives considered; Alt 1 applied — §6 bit allocation deferred past mark/tri-colour to row 5.3 owner; Alt 2 items 2 / 3 / 4 applied — Var rooting timing + no-alloc finaliser invariant + Block B reconciliation; Load-bearing concerns #1-6 all applied; per-Tag dispatch tables absorbed from ADR-0027 §3 per Alt 1 cross-coupling resolution; Alt 2 item 1 ADR-split not applied — coupling argues for single ADR; Alt 3 wildcard recorded as row 5.3 owner candidate). Co-issued with ADR-0027. -- 2026-06-05 (amendment 2): §5 Root sources table re-tabled for Phase B #3b (D-244) per **ADR-0091**. Rows 2 (`current_frame`) + 7 (`macro_root_slot`) subsumed into a single `thread_roots` E-source (binding-frame chain + macro slot + **VM operand-stack frame chain**, per live thread — self TLS + every registered worker `ThreadGcContext`). The operand stack (`vm.eval` `stack[0..sp]` + `locals`) is newly rooted so a Phase-B worker mid-`eval` does not hold un-rooted Values a concurrent `collect()` would sweep (the ADR-0090 leading-finding UAF). Net: live E-walkers 4 → 3; `RootSource` Zig enum count 10 → 9; the triplicated per-thread union-addressing helpers (`frameSourceAt`/`macroSourceAt` + their `src_idx` cursors) commonized into one `threadContextAt` + one thread-major cursor (F-011). Runtime-inert at land (no live auto-collect; empty registry + null `eval_frame_head` at quiescent collect points = today's roots). The alloc-boundary safepoint that makes auto-collect / worker-collect fire is #3b-step2 (couples to #4). DA-fork (Option A fold rejected as a different-kind conflation; Alt 1 clean-11th-source vs Alt 2 union-cursor [adopted] vs Alt 3 thread-driven spine [Phase-15/JIT re-conception candidate]) reflected verbatim into ADR-0091's Alternatives considered. Smell category: structural-imagination (F-003) — the per-thread union became the dominant root structure once a third per-thread source [operand stack] arrived, so it is named once instead of triplicated. (#3b-step2b adds a 4th `thread_roots` sub-walk — the per-thread `gc_self_guard` in-flight-fabrication slot for the `op_vector/map/set_literal` accumulator window — via the same thread-major cursor, no enum change; the cursor topology Alt 2 chose accommodates it as another sub-phase per ADR-0091's "extends, not rewrites".) -- 2026-05-24 (amendment 1): §5 Root sources table re-shaped per the `private/notes/phase5-5.3.b.3-survey.md` finding that only 4 of the 10 enumerated sources are entry-point walkers in cw v1. Rows 3 / 4 / 8 demoted to **T** (tag-trace entries registered into `tag_ops.tag_trace_table` from the owning module — `tree_walk.zig` / `lazy_seq.zig` / `type_descriptor.zig`); rows 5 / 6 / 9 demoted to **D** (no live structs in cw v1 / closed-at-construction in `referAll` / cache holds namespace-owned pointers with no GC edge). Net: 4 live walkers + 3 tag-trace registrations + 3 documentary rows. Row 7 also moved from "Analyzer.macro_root_slot" to "macro_root_slot" because the slot lives in `runtime/gc/root_set.zig` (Layer 0) per the survey's zone-respecting access-path decision (Layer 1 reads/writes via downward import). Devil's-advocate Load-bearing concern #5's "reserved as `null` in 5.x" framing for row 9 withdrawn — the cache carries no GC-managed pointers in any Phase. Smell trigger: caught during 5.3.b.3 implementation prep when the survey traced each source against the actual cw v1 struct layouts; the original §5 table was inherited from ADR-0028's initial draft which mirrored cw v0's root-source enumeration without re-evaluating each source against cw v1's design (e.g. cw v1 splits ProtocolFn cache into per-CallSite per ADR-0008 — row 5 becomes empty by construction). Smell category: Spec-drift (ADR text inherited a cw v0 enumeration into a cw v1 design that doesn't have the same shape). diff --git a/.dev/decisions/0029_runtime_java_cljw_layout.md b/.dev/decisions/0029_runtime_java_cljw_layout.md deleted file mode 100644 index 45caa3d2d..000000000 --- a/.dev/decisions/0029_runtime_java_cljw_layout.md +++ /dev/null @@ -1,319 +0,0 @@ -# 0029 — Runtime java/ + cljw/ layout, feature-implementation neutrality, ADR-0011 supersede - -- **Status**: Accepted -- **Date**: 2026-05-24 -- **Author**: Shota Kudo (drafted with Claude during user-directed structural session) -- **Tags**: phase-5-late, structure, host, interop, java, cljw, supersedes-0011 -- **Supersedes**: ADR-0011 - -## Context - -ClojureWasm v1 is mid-Phase-5. Phase 6 entry will start landing Java -stdlib equivalents (`java.util.UUID`, `java.io.File`, `java.time.Instant`, -…), and the current ADR-0011 has reserved `src/runtime/host/{lang, io, -util, time, net, nio, math, security, sql, text, reflect, concurrent}/ -.zig` with 13 `_placeholder.zig` files. - -Before any of those host classes land, a user-directed structural -session (2026-05-24) surfaced six concerns: - -1. **Implementation sharing is required.** `(random-uuid)` (Clojure - core) and `(java.util.UUID/randomUUID)` (Java surface) must share - the same 16-byte random-bytes generator. The cw-v0 pattern (impl - written directly inside `src/lang/interop/classes/uuid.zig`) cannot - achieve this. -2. **File fan-out is structurally unavoidable.** One feature needs at - minimum three files (impl body / surface wrapper / Clojure-ns - registration). This is a structural lower bound; mitigation must - come from discoverability (feature-name consistency + index - integrity), not from collapsing files together. -3. **The `runtime/host/` name and layer are misleading.** `host` is - double-charged with both OS / syscall meaning and Java-compat-surface - meaning. A direct Java-namespace mirror `runtime/java//.zig` - reads more naturally. -4. **cljw-only features** (wasm component invoke, edge runtime, build - command, etc.) deserve the same thin-wrapper structure. A symmetric - `runtime/cljw//.zig` layout pays for itself once Phase - 12+ work begins. -5. **Smallest-diff bias of the "let's collapse 18 host classes into - their wrappers" shape** is almost certain to surface at Phase 12+ - when host-class count grows. A structural device that prevents the - AI loop from doing this autonomously is needed. -6. **Internal dependency direction inside `runtime/` is undocumented**, - but a 5-sub-zone scheme (an earlier draft) was judged excessive. - A single rule — "surface layer calls neutral impl, never the - reverse" — is sufficient. - -In addition, the zwasm-v2 Java-InterOp premise (recorded in user -context this session): - -> For what is supported, no error surfaces; the internal implementation -> need not mirror Java; equivalent inputs and outputs (with side -> effects where applicable) are achieved via Zig-idiomatic means. - -This premise means Java surfaces are *not obligated* to reproduce JVM -internal representations exactly, which is what justifies sharing -implementations between Clojure and Java entry paths. - -## Decision - -### D1. Directory layout - -Retire `runtime/host/`. Replace with: - -- **Neutral implementation layer**: flat under `src/runtime/`. - Examples: `runtime/uuid.zig`, `runtime/clock.zig`, `runtime/random.zig`, - `runtime/uri_parse.zig`, `runtime/path.zig`, `runtime/file_io.zig`, - `runtime/charset.zig`. Multi-file features land in sub-directories: - `runtime/regex/{compile, match}.zig`, - `runtime/crypto/{secure_random, message_digest}.zig`, - `runtime/time/{instant, local_date, …}.zig`. - These are **namespace-neutral** — they wrap OS / Zig std and depend - on neither the Clojure nor the Java namespace. - -- **Java-compat surface**: `src/runtime/java//.zig`. - Examples: `runtime/java/util/UUID.zig`, `runtime/java/io/File.zig`, - `runtime/java/time/Instant.zig`. Structure is 1:1 with the Java FQCN - (`java.util.UUID` → `java/util/UUID.zig`). Each file is a **thin - wrapper** that calls the neutral impl layer. - -- **cljw-native surface**: `src/runtime/cljw//.zig`. - Examples: `runtime/cljw/wasm/Engine.zig`, - `runtime/cljw/build/Compiler.zig`, `runtime/cljw/edge/Server.zig`. - cw-original value-add features (Zig extensions, not Clojure - libraries). Each file is a thin wrapper symmetric with the Java - surface. - -- **Disposition of the 13 existing placeholders**: - - `runtime/host/_host_api.zig` → moved to `runtime/java/_host_api.zig` - - `runtime/host/{lang, io, util, ...}/_placeholder.zig` → removed - - `runtime/host/` directory removed - -### D2. Dependency direction (single rule) - -> No file under `runtime/` other than those in `runtime/java/**` and -> `runtime/cljw/**` may import from `runtime/java/**` or -> `runtime/cljw/**`. - -That is: surface layers (java/, cljw/) call the neutral impl layer; -the reverse direction is forbidden. Cross-references inside the -neutral impl layer (`runtime/value/`, `runtime/collection/`, -`runtime/uuid.zig`, …) are left to natural import discipline and are -not codified in an ADR. If real friction emerges later, an additional -gate can be added at that point. - -### D3. Feature-implementation neutrality (promoted to F-009) - -A new user-declared invariant is added to `.dev/project_facts.md`: - -> Feature-implementation bodies (the place where OS / Zig std is -> invoked, or where cw-original compute logic lives) reside in -> **namespace-neutral locations** — `src/runtime/` (flat) or -> `src/runtime//` (when multiple files). Clojure-ns paths -> (`src/lang/primitive/.zig` or -> `src/lang/clj/clojure/<…>.clj`), Java-ns paths -> (`src/runtime/java//.zig`), and cljw-ns paths -> (`src/runtime/cljw//.zig`) all connect to the neutral -> impl **as thin wrappers from above**. Cross-surface calls -> (a Clojure-ns wrapper calling into a Java-ns wrapper, or vice versa) -> are forbidden — both share the neutral impl. - -F-009 status grants the Devil's-advocate subagent (CLAUDE.md § Smell -triggers are interrupts, not stops) the authority to automatically -reject "let's inline the impl into the surface file" alternatives as -out-of-envelope, with user-direction-only amendment. - -### D4. Guardrails (three scripts) - -| # | Script | Purpose | -|--------|------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| **G1** | `scripts/zone_check.sh` (extended) | Detects D2 violations: any non-`runtime/java/**` / non-`runtime/cljw/**` file importing from `runtime/java/**` or `runtime/cljw/**` | -| **G2** | `scripts/check_surface_marker.sh` | Requires each `runtime/java/**/*.zig` and `runtime/cljw/**/*.zig` to declare a Backend marker docstring header: `//! Backend: ` + `//! Impl deps: …` + `//! Clojure peer: …` | -| **G3** | `scripts/check_feature_keyword.sh` | Verifies that `compat_tiers.yaml` `keyword:` values appear in every file path listed under that entry's `files:` map (guarantees 100% grep-hit on the feature keyword) | - -All three are wired into `test/run_all.sh` and gate both Mac and -Linux (OrbStack x86_64) runs. - -### D5. `compat_tiers.yaml` schema extension - -The existing entry shape `{ fqn, native_ns, phase }` is extended to: - -```yaml -- fqn: java.util.UUID - cljw_ns: cljw.java.util.UUID # was: native_ns: cljw.host.* (host. prefix dropped) - keyword: uuid # validated by G3 - tier: A - phase: 5 - files: - surface: runtime/java/util/UUID.zig - impl: runtime/uuid.zig # neutral implementation body - impl_extras: [...] # additional sub-impls the impl calls - wrap: runtime/collection/string.zig - clojure_peer: lang/primitive/uuid.zig - methods: [randomUUID, fromString, toString, ...] - clojure_peer_vars: [clojure.core/random-uuid, clojure.core/parse-uuid] -``` - -A runtime command `cljw --list-host-classes` (planned by Phase 14) -reads this same YAML, giving development-time (static yaml grep) and -runtime (`cljw` command) two-axis coverage visibility. - -### D6. ADR-0011 disposition - -ADR-0011's `Status` field is changed from `Accepted` to `Superseded -by ADR-0029`. The `___HOST_EXTENSION` marker pattern itself -(distributed registration replacing a central registry) is carried -forward by this ADR; only the registration root changes from -`host/_host_api.zig` to `java/_host_api.zig`. - -## Alternatives considered - -Because this ADR was drafted during a user-directed structural session -with multiple rounds of alternative exploration in chat, the -Devil's-advocate fork pass has been replaced by recording each -considered shape here in full. - -### Alt 1: Five sub-zones inside `runtime/` (earlier draft, rejected) - -Split `runtime/` into core / clj-core / capability / extension / host, -fix a cross-reference table in the ADR, and gate the table. - -- **Pros**: structurally strict, every sub-zone responsibility named. -- **Cons**: five-layer naming (especially `capability`) feels abstract; - the relationship with the existing flat layout - (`runtime/io_interface.zig`, `runtime/hash.zig`, etc.) becomes - hand-wavy; seven guardrail scripts are excessive. -- **Rejected because**: user feedback was "we don't even need a `sys/` - partition; flat under `runtime/` reads better; the layer emphasis is - overdone." - -### Alt 2: `runtime/host/java/` + `runtime/host/cljw/` (mid-draft, rejected) - -Keep `host/` and place `java/` and `cljw/` beneath it. - -- **Pros**: aligns with the `host_classes` concept in - `compat_tiers.yaml`. -- **Cons**: double-meaning of `host` (OS / Java surface); one extra - directory level. -- **Rejected because**: user judged "`runtime/java/` is more - intuitive than `runtime/host/java/`." - -### Alt 3: `runtime/interop/java/` + `runtime/interop/cljw/` (exploratory, rejected) - -Rename `host` to `interop`, inheriting cw-v0's `src/lang/interop/` -naming. - -- **Pros**: continuity with cw v0. -- **Cons**: `interop` is vague (interop with what?); reads - particularly thin once Java-compat and cljw-original are listed - side by side. -- **Rejected because**: user judged "just inherit the Java namespace - directly." - -### Alt 4: Pull host out of `runtime/` and make `src/host/` a peer zone (late draft, rejected) - -Six-zone shape: `src/{runtime, capability, host, eval, lang, app}`. - -- **Pros**: purifies `runtime/`; host gains independence. -- **Cons**: directory tree grows from 4 to 6 top zones; - `___HOST_EXTENSION` registry discovery becomes "borrow types from - outside" rather than co-located. -- **Rejected because**: user judged "if we want runtime-internal - conveniences, host-ish and capability-ish things belong under the - runtime directory." - -### Alt 5: Keep neutrality as ADR-only, do not promote to F-009 (exploratory, rejected) - -Write the neutrality principle into ADR-0029 only; do not register -it as F-NNN. - -- **Pros**: lightweight; future flexibility for the AI loop to - amend. -- **Cons**: at Phase 12+ with many host classes the AI loop is likely - to propose "inline impl into the surface for fewer files," with the - Devil's-advocate subagent generating three plausible alternatives. - F-009 lets the Devil's-advocate subagent reject envelope-violating - alternatives automatically. -- **Rejected because**: user chose "promote to F-009." - -## Consequences - -### Positive - -- **Structural answer to fan-out.** F-009 + three guardrails + - keyword-consistent naming + `compat_tiers.yaml` index guarantee - that, even with fanned-out files, "grep by feature keyword - yields 100% hit" is maintained. -- **Symmetry.** Java surfaces and cljw-native surfaces use the same - shape (thin wrapper + Backend marker + cljw_ns registration). -- **No misreadings.** `runtime/java//.zig` is 1:1 with - Java FQCN; the double-meaning of `host/` disappears. -- **Smallest-diff bias prevention.** F-009 lets the Devil's-advocate - subagent automatically defend against future AI-driven design - erosion. -- **Dependency rule fits on one line.** The earlier draft's - five-layer cross-reference table compresses to "surfaces call - neutral, never the reverse." - -### Negative - -- **Large amendment fan-out among existing documents.** ADR-0011 - supersede / ROADMAP §5 §6.0 / structure_plan.md / - host_extension_layout.md (renamed) / compat_tiers.yaml (schema - extension) / project_facts.md (F-009 added) / handover.md. All - manageable in one user-directed sweep but non-trivial. -- **Existing 13 placeholders are deleted.** ADR-0011's reserved - directories are wiped. If any of them are needed later, they - re-emerge under `runtime/java/` or `runtime/cljw/`. The 40 - entries in `compat_tiers.yaml host_classes` all migrate cleanly - under `runtime/java/` via the schema extension. - -### Neutral / follow-ups - -- **`runtime/error/` and `runtime/io/` consolidation.** The old - `error.zig` / `error_catalog.zig` / `error_print.zig` files become - `runtime/error/{info, catalog, print}.zig`; the old - `io_interface.zig` becomes `runtime/io/interface.zig`, and Tier 2 - = `runtime/io/default.zig` lands later in Phase 5+ as a - continuation of the ADR-0015 setup. This is delivered as Commit 3 - alongside the other ADR-0029 mechanics and serves as the first - consolidation case study for follow-up moves (`hash/`, `keyword/`, - …). `runtime/io/` will briefly hold a single file; a `_README.md` - inside the directory explains the intent so the half-populated - state does not read as accidental. -- **`runtime/hash/` split** waits for a real need (e.g., when a - separate identity-hash variant becomes useful at Phase 6+) and - ships as a separate small ADR. -- **`cljw_ns` rename (`cljw.host.*` → `cljw.*`).** Docs that - reference the old prefix (`compat_tiers.yaml` body + related rule - files) get a single batch update. -- **`cljw --list-host-classes`** runtime command, planned by Phase 14 - to read `compat_tiers.yaml` and emit the index at startup. - -## References - -- **Supersedes**: ADR-0011 (host extension mechanism). -- **Related ADRs**: ADR-0013 (Tier classification), ADR-0007 - (TypeDescriptor / Option β), ADR-0017 (Allocator strategy), - ADR-0018 (error catalog SSOT), ADR-0023 (comptime stub). -- **New F-NNN**: F-009 (feature-implementation neutrality), landed - alongside this ADR. -- **Amended docs**: ROADMAP §5 / §6.0; `.dev/structure_plan.md`; - `.claude/rules/host_extension_layout.md` (renamed to - `.claude/rules/java_cljw_surface_layout.md`); `compat_tiers.yaml` - (schema extension). -- **New rule**: `.claude/rules/feature_name_consistency.md` - (keyword consistency + Backend marker contract). -- **Proposal draft trail**: `.dev/proposals/host_layer_design.md` - (v2 design memo) and `.dev/proposals/0029_runtime_java_cljw_layout_draft.md` - (Japanese ADR draft) — both lived under `.dev/proposals/` (a - scratch directory, never git-tracked) and were removed in - Commit 6 of the cluster (772e16c, 2026-05-24). -- **Frequency data**: `private/clojure_frequent_java_interop/00a_frequency_overview.md` - (host-class landing priority basis). - -## Revision history - -- 2026-05-24: Status: Proposed → Accepted (initial landing). Drafted - during a user-directed structural session; replaces ADR-0011. - F-009 added to `project_facts.md` in the same commit. diff --git a/.dev/decisions/0030_defer_defrecord_reify_to_phase7.md b/.dev/decisions/0030_defer_defrecord_reify_to_phase7.md deleted file mode 100644 index 96a921f12..000000000 --- a/.dev/decisions/0030_defer_defrecord_reify_to_phase7.md +++ /dev/null @@ -1,161 +0,0 @@ -# 0030 — Defer 5.12.b (defrecord) + 5.12.c (reify) to Phase 7 entry - -- **Status**: Accepted -- **Date**: 2026-05-24 -- **Author**: Shota Kudo (drafted by the autonomous loop with Devil's-advocate fork) -- **Tags**: phase-5, phase-7, scope, deftype, defrecord, reify, protocol-dispatch - -## Context - -ROADMAP §9.7 row 5.12 originally bundled `deftype` + `defrecord` + -`reify` (all three of ADR-0007 Option β's user-facing forms) into a -single Phase-5 deliverable. 5.12.a landed `deftype` + constructor + -field access against the TreeWalk backend at commit 7e7c6fd. - -The remaining halves of the row — `defrecord` (5.12.b) and `reify` -(5.12.c) — are **protocol-method-dispatch consumers**: - -- `defrecord` extends `deftype` with implicit `IPersistentMap` - semantics (`get` / `assoc` / `keys` / `vals` over field names). - Either it special-cases these four methods in eval/tree_walk - (skeleton; would be rewritten when Phase 7's dispatch lands), or - it uses `TypeDescriptor.lookupMethod` directly without the - CallSite-cache-driven dispatch. -- `reify` produces an anonymous TypeDescriptor whose method bodies - close over the surrounding lexical scope. Its only user-facing - purpose **is** protocol-method dispatch — without that, `reify` - has no semantics to honour. - -Phase 7 entry per ADR-0008 amendment 1 will land the canonical -`dispatch(rt, cs, receiver, protocol, method, args)` function, the -CallSite cache fill / hit signature, and the -`MethodEntry.fn_ptr` calling convention (Caller-context threading, -arg adapter, error path). All three are open design questions today. - -Landing `defrecord` and `reify` against today's pre-dispatch -machinery commits cw v1 to a Phase-5 shape that will almost -certainly be rewritten when Phase 7 ABI is decided — a textbook -Progress-pressure smell + Skeleton-with-no-shrink smell per -`.dev/principle.md`. - -## Decision - -ROADMAP §9.7 row 5.12 is narrowed to scope = **`deftype` only** -(already landed as 5.12.a). Two new sub-tasks are added to ROADMAP -§9.9 Phase 7 entry alongside 5.12.d: - -- **5.12.b** (defrecord) — deftype + implicit IPersistentMap - semantics, written against the Phase-7 dispatch ABI. -- **5.12.c** (reify) — anonymous TypeDescriptor + closure capture - + protocol-method bodies, written against the Phase-7 dispatch - ABI. - -`Code.feature_not_supported` continues to raise for `(defrecord -…)` / `(reify …)` at Phase 5 exit. The `phase_at_least_5` flip -(5.15) does **not** flip these. - -## Alternatives considered - -Per CLAUDE.md § ADR-level designs are handled inline, a -Devil's-advocate `general-purpose` subagent was forked with the -F-NNN constraints in context (`.dev/project_facts.md` F-001..F-009) -to produce alternatives within the F-NNN envelope. - -### Alt 1 — Smallest-diff: land defrecord + reify in Phase 5 with pre-dispatch shims - -`defrecord` lands as `deftype` + hard-coded IPersistentMap method -table via direct `TypeDescriptor.lookupMethod` (5.11 surface). -`reify` lands as anonymous TypeDescriptor + direct lookup. Phase 7 -rewires direct lookup through `dispatch + CallSite`; `reify`'s -protocol-method bodies are rewritten. - -- **Better**: ROADMAP §9.7 spine stays whole; Phase 5 exit smoke - can include `defrecord` and `reify` cases beyond the listed - `(deftype Point [x y])` example; Phase 7 entry stays slim. -- **Worse**: explicitly accepts a Phase-7 rewrite of `reify`'s - method-body codegen (F-002 says smallest-diff is secondary, this - alternative buys Phase-5 width by accepting that rewrite). - -**Rejected** — F-002 + F-003 prefer landing once against the real -ABI. - -### Alt 2 — Defer defrecord + reify, narrow 5.12 to 5.12.a (this ADR) - -ROADMAP §9.7 5.12 = deftype only. 5.12.b + 5.12.c move to Phase 7 -entry next to 5.12.d (protocol method dispatch). - -- **Better**: defrecord's IPersistentMap path and reify's protocol - bodies land once, against the real CallSite ABI. Aligns with F-003 - (defer to owning Phase) — both forms *are* protocol-dispatch - consumers. -- **Worse**: Phase 5 ships with `defrecord` / `reify` still raising - `Code.feature_not_supported`. Phase 7 entry grows by 2 tasks. - -**Accepted**. - -### Alt 3 — Wildcard: defer only reify, land defrecord in Phase 5 (no protocol bodies) - -`defrecord`'s required Phase-5 surface is the implicit map view -(get/assoc/keys/vals) — no user-defined protocol methods needed. -Land 5.12.b with that narrow scope; defer 5.12.c (reify). - -- **Better**: defrecord becomes usable at phase_at_least_5 (5.15) - flip, broader Tier A coverage. get/assoc reuse 5.4-5.6 map - machinery, no dispatch ABI dependency. -- **Worse**: defrecord's `(defrecord R [x] Proto (m [_] …))` - extended form still raises feature_not_supported until Phase 7, - creating a "half-supported" surface that the Skeleton-with-no- - shrink smell would flag. - -**Rejected** — F-002 finished-form purity prefers Alt 2. - -## Consequences - -### Positive - -- 5.12.b and 5.12.c land once, against the Phase-7 dispatch ABI. - No rewrite cost on the protocol-method-body codegen path. -- F-003 honoured: `defrecord` and `reify` are protocol-dispatch - consumers; their owner is the Phase that lands the dispatch ABI. -- Phase 5 exit smoke (5.16) keeps the listed - `(deftype Point [x y]) (.x (Point. 1 2))` example, which is the - only test the original ROADMAP §9.7 5.16 row spelled out for the - ADR-0007 family. - -### Negative - -- Phase 5 exits with `defrecord` and `reify` still raising - `Code.feature_not_supported`. Tier A coverage at the 5.15 flip - is narrower than the original §9.7 5.12 row implied. -- Phase 7 entry grows by 2 tasks (5.12.b + 5.12.c alongside the - already-planned 5.12.d). -- Tests using `defrecord` / `reify` (none in the current corpus - but the Phase 11 `test/clj/` port may surface some) stay on the - unsupported path until Phase 7. - -### Neutral / follow-ups - -- ROADMAP §9.7 5.12 description amended in-place to scope = - deftype only, with a one-line note pointing at this ADR. -- ROADMAP §9.9 Phase 7 entry placeholder gets 5.12.b + 5.12.c + - 5.12.d as three explicit task rows (the 4.21 / ADR-0007 Phase 5+ - migration note in §9.9 already foreshadows this). -- `.dev/debt.md` D-041 (catalog Code cleanup) is unaffected. - -## References - -- ROADMAP §9.7 row 5.12 (amended in same commit). -- ROADMAP §9.9 Phase 7 entry placeholder (amended in same commit). -- ADR-0007 (TypeDescriptor / Option β). -- ADR-0008 a1 (protocol method dispatch contract). -- F-003 (defer to owning Phase) + F-002 (finished-form wins). -- 5.12.a landing commit: 7e7c6fd. -- Devil's-advocate subagent output: see Alternatives considered - above (recorded inline per CLAUDE.md § ADR-level designs are - handled inline). - -## Revision history - -- 2026-05-24: Status: Proposed → Accepted (initial landing). - Devil's-advocate subagent forked with F-NNN context; output - reflected verbatim in Alternatives considered. diff --git a/.dev/decisions/0031_regex_engine_choice.md b/.dev/decisions/0031_regex_engine_choice.md deleted file mode 100644 index 71b521541..000000000 --- a/.dev/decisions/0031_regex_engine_choice.md +++ /dev/null @@ -1,217 +0,0 @@ -# 0031 — Regex engine choice for ClojureWasm - -- **Status**: Accepted -- **Date**: 2026-05-25 -- **Author**: Shota Kudo (drafted + accepted by the autonomous - loop with Devil's-advocate fork) -- **Tags**: phase-6, regex, dependency, java-pattern, exit-criterion - -## Context - -ROADMAP §9.8 row 6.6 calls for -`runtime/regex/{compile,match}.zig` + -`runtime/java/util/regex/Pattern.zig` + -`lang/primitive/regex.zig` so the Phase 6 exit criterion -`(re-find #"\d+" "abc123")` → `"123"` is met. The regex -literal reader (`#"..."`) already lands a `regex` Tag Value -(Phase 4); only the engine + matcher is missing. - -JVM Clojure delegates entirely to `java.util.regex.Pattern`. -That is one of the largest single dependencies in Clojure -core's IFn surface — `re-find` / `re-matches` / `re-seq` / -`re-groups` / `replace` / `split` all flow through it. The -cw v1 engine has to cover at minimum: - -- Character classes (`\d`, `\w`, `\s`, `[a-z]`, `[^abc]`). -- Quantifiers (`?`, `*`, `+`, `{n,m}`, lazy `*?` `+?`). -- Anchors (`^`, `$`, `\b`). -- Grouping (`(...)`, `(?:...)`, `(?...)`). -- Alternation (`|`). -- Java-compatible backslash escapes and `(?i)` flag. - -The decision is which engine to ship. - -## Decision - -**Adopt Alternative 2 — Two-tier (lazy-DFA cache over Pike-NFA) -with explicit `Program` IR boundary.** Pure-Zig in-tree -implementation (F-001 clean), with a `runtime/regex/compile.zig` -parser → AST → `Program` IR pipeline, a Thompson / Pike VM in -`runtime/regex/match.zig` as the correctness baseline, and a -lazy DFA cache in `runtime/regex/dfa.zig` for the no-capture / -anchored fast paths. `runtime/regex/exec.zig` dispatches per -call site. - -This is the finished-form-clean choice per F-002: the IR -boundary + two-tier matcher is what cw v1 will want by Phase -10+ regardless (regex appears in macro expansion, namespace -parsing, edn reader, spec.alpha-style validators). Paying once -now avoids a depth-4 Supersedes rewrite later. The 3-5 cycles -versus Alternative 1's 1 cycle is absorbed by the per-task -TDD loop without ceremony — CLAUDE.md's priority chain puts -F-002 above Phase 6 schedule pressure. - -**Post-acceptance follow-up** (debt row, not blocking): the -Devil's-advocate subagent surfaced Alternative 3 (reader-time -intern + AST walker + content-addressed `Program` cache) as a -genuinely Clojure-shaped framing — "patterns are immutable -interned values, like keywords" — that composes beautifully -with `comptime`-lifted core regex literals. Promoting the -Alternative-2 `Program` to live in an Alternative-3-style -intern cache is a single non-breaking refactor that can land -post-Phase 6.6 once measurement justifies. Recorded as a -recall-trigger debt row at Phase 10+ planning. - -### Candidate engines - -| Engine | Pros | Cons | -|--------------------------------------|------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------| -| **A. Custom-min (in-tree Zig)** | Zero dependency. Fits F-001 (zwasm v2 standalone). Matches deftype/numeric tower in-tree philosophy. | Re-implementing Java's Pattern surface is a large engineering bill. Subtle semantic gaps surface as conformance test failures later. | -| **B. zig-regex (3rd-party)** | Already exists; saves ~1500 LOC. Active maintenance. | New dependency, Zig 0.16 compat unknown; PCRE-ish syntax may diverge from Java's; would need a syntax-translation layer. | -| **C. PCRE C bind via build.zig.zon** | Most feature-complete; near-direct mapping to Java's Pattern. | C dependency violates F-001 zwasm v2 standalone goal. Wasm Component target (Phase 19) is complicated by it. | - -### Devil's-advocate brief (paste into next session's subagent fork) - -> Devil's advocate this ADR. The active F-NNN constraints from -> `.dev/project_facts.md` are: -> -> - F-001: zwasm v2 standalone (no JVM runtime dep). -> - F-002: finished-form-clean wins over smallest-diff. -> - F-009: feature-implementation neutral (impl in -> runtime/regex/, Java surface in runtime/java/util/regex/, -> Clojure peer in lang/primitive/regex.zig). -> -> Produce 3 alternative shapes **within those constraints** -> (one smallest-diff, one finished-form-clean, one wildcard); -> for each, name what it does better than candidates A / B / C -> and what it breaks. Do NOT propose alternatives that violate -> F-001 (no C deps, no JVM deps). If the only finished-form- -> clean option requires violating F-001, record that finding -> as the leading entry of Alternatives considered so the main -> loop sees it, but do not ask the loop to halt — F-NNN -> amendment is a user action. - -The subagent's output is reflected verbatim into the -"Alternatives considered" section below before the next -session flips `Status: Accepted`. - -## Alternatives considered - -*Devil's-advocate `general-purpose` subagent output, reflected -verbatim per CLAUDE.md § ADR-level designs are handled inline.* - -### Alternative 1 — Thompson-NFA with Java-syntax frontend, single-file core (smallest-diff) - -**Better than A/B/C**: Ships a working Phase 6.6 exit in one to two TDD cycles by adopting the classic Thompson NFA construction (Russ Cox's "Regular Expression Matching Can Be Simple And Fast"), which is the smallest correct engine that covers the full Phase 6.6 surface without backtracking pathologies. Unlike candidate A's loosely-scoped "custom-min" framing, this commits to a specific algorithm with a known LOC budget (~600-900 LOC, smaller than A's 800-1500 estimate) and a known correctness story. Unlike B, no third-party Zig 0.16 compatibility risk and no PCRE-vs-Java syntax translation layer. Unlike C, no libc / no build.zig.zon native dep — pure Zig, F-001 clean. The Thompson construction also gives `re-seq` linear-time scanning for free (no catastrophic backtracking), which a naive recursive-descent matcher under "custom-min" might botch. - -**Trades off**: No backreferences (`\1`), no lookaround (`(?=...)`, `(?...)`) work because they're just labelled capturing groups, not backrefs. Unicode property classes (`\p{L}`) are deferred to a later cycle (ASCII + `\d \w \s` only at Phase 6.6 exit). Performance is worst-case O(n·m) which is correct but not Hyperscan-tier; that's fine for a Clojure runtime where regex isn't the hot path. - -**File-by-file shape**: - -- `runtime/regex/compile.zig`: Parser produces an AST (`Node` tagged union: Lit / Class / Concat / Alt / Star / Plus / Quest / Group / Anchor), then a single-pass NFA builder emits a flat `[]Inst` instruction array (`Char`, `Range`, `Match`, `Jmp`, `Split`, `Save`). Java-syntax aware from day one: `(?i)`, `\d \w \s \b`, `[...]`, `{n,m}`, `(?:...)`, `(?...)`. ~400 LOC. -- `runtime/regex/match.zig`: Pike VM (Thompson with thread list) — two `Thread` lists (current/next), each thread carries a PC + capture-slot snapshot (small fixed-size array to avoid per-step allocation for the common case of ≤8 groups). `find` / `match` / `seq` / `groups` all reduce to one VM driver with different anchoring + iteration policies. ~300 LOC. -- `runtime/java/util/regex/Pattern.zig`: Thin wrapper exposing JVM-shaped `compile` / `matcher` / `Matcher.find` / `Matcher.group` over the in-tree NFA. Owns the compiled `Program` slice; `Matcher` is a borrow + VM state. -- `lang/primitive/regex.zig`: Clojure-side `re-find` / `re-matches` / `re-seq` / `re-groups` / `re-pattern` / `clojure.string/replace` + `split` dispatch into `java.util.regex.Pattern`. Reader-emitted regex Tag Value carries a pointer to the compiled `Program`. - -**Work split**: 1-cycle: parser + AST + NFA emit + Pike VM + `re-find` green on `(re-find #"\d+" "abc123")` → `"123"`. Follow-up cycles: `re-matches` / `re-seq` / `re-groups`, named groups, `(?i)` flag plumbing, `clojure.string/replace` + `split`, error messages aligned with JVM `PatternSyntaxException`. - -### Alternative 2 — Two-tier (DFA-cached over NFA) with explicit IR boundary (finished-form-clean) - -**Better than A/B/C**: Treats the engine as a small compiler with an explicit IR (`Program`) that is decoupled from both the parser and the runtime matcher. This is the shape JVM `java.util.regex.Pattern` evolves toward in practice (compiled `Node` graph) and is what cw v1 will *want* by Phase 10+ when regex shows up in macro expansion / spec.alpha-style validators / namespace parsing. The two-tier matcher — a fast on-demand DFA cache (subset construction memoized per state) backed by the NFA fallback for stateful features — gives near-Hyperscan throughput on the common `re-seq` / `re-find` cases while preserving correctness for `(?i)`, anchors, and (eventually) backreferences via the NFA path. This is what RE2 and Go's `regexp` ship, and it's the shape the codebase will not regret in 5 phases. F-001 is preserved (still pure Zig, no C dep). F-002 is honoured (finished-form wins): the eventual rewrite from "custom-min" to "real engine" is avoided by paying once now. - -**Trades off**: 2-4× the initial LOC of Alternative 1 (~1500-2000 LOC including DFA cache, state interning, and IR optimiser passes). Three-to-five cycles instead of one to ship the Phase 6.6 exit criterion. DFA state cache needs an eviction policy (LRU on a fixed-size table) to avoid pathological memory growth on adversarial inputs — that's a real design surface, not a footnote. Capture groups in DFA mode require a tagged-DFA variant (Laurikari's TDFA) or a fall-back-to-NFA-on-capture strategy; the latter is simpler and is what RE2 does, but it means the DFA fast path only helps `re-find`-without-groups and `clojure.string/split`. Higher initial cognitive load for whoever opens the file next. - -**File-by-file shape**: - -- `runtime/regex/compile.zig`: Parser → AST → IR (`Program`) with explicit `Inst` union, plus an IR optimiser (dead-state elimination, character-class merging, anchor hoisting). Exposed as `compile(pattern: []const u8, flags: Flags) !Program`. ~700 LOC. -- `runtime/regex/match.zig`: Thread-based NFA matcher (Pike VM) as the correctness baseline. ~350 LOC. -- `runtime/regex/dfa.zig`: Lazy DFA with on-the-fly subset construction, state-cache table keyed by sorted NFA-state-set hash, capped at e.g. 256 states with reset-on-overflow falling back to NFA. ~400 LOC. -- `runtime/regex/exec.zig`: Dispatcher choosing DFA vs NFA per call site (anchored + no-groups + no-backrefs → DFA; otherwise NFA). ~150 LOC. -- `runtime/java/util/regex/Pattern.zig`: JVM-shaped `Pattern` / `Matcher` over `exec.zig`. `Matcher.find` picks executor; `Matcher.group` always uses the NFA path (no DFA capture). -- `lang/primitive/regex.zig`: Same as Alternative 1, but the reader can stash a hint flag on the Tag Value indicating whether the pattern is DFA-eligible (computed at compile time) so `clojure.string/split` short-circuits to the fast path. - -**Work split**: 1-cycle: parser + AST + IR + NFA path only, `(re-find #"\d+" "abc123")` → `"123"` green via NFA. 2-cycle: DFA cache + dispatcher, validate `re-seq` throughput on a synthetic 1 MB log-line input. 3-cycle: capture-group plumbing, named groups, `re-groups`, `clojure.string/replace`. 4-cycle: `(?i)` (compile-time case-folding into character classes — no runtime cost), `\b`, `(?:...)`. 5-cycle: error messages aligned with `PatternSyntaxException`, debt row for backreferences (NFA-only follow-up). - -### Alternative 3 — Reader-time compile + content-addressed Program cache, AST-driven (wildcard) - -**Better than A/B/C**: Reframes "the regex engine" as "an interned compile result that the reader produces once per literal." Every `#"..."` literal in the source program goes through the parser **at read time**, lands an immutable `Program` in a process-global content-addressed cache (keyed by the SHA-256 or 64-bit FNV of the pattern source + flags), and the runtime Tag Value is just a pointer + length into that cache. Three structural wins fall out: - -1. The compile cost is paid once per literal across the entire program lifetime — `re-find` in a hot loop never re-parses. -2. `clojure.core` itself has a known finite set of regex literals (`clojure.string/replace`'s internal `\\` handling, ns parser, edn reader); these are *all* compilable at AOT time, lifted into the binary as `comptime`-built `Program` constants. No runtime compile cost for the standard library at all. -3. The matcher can be **AST-walking** instead of VM-driven for the small-pattern common case. JVM Clojure regex patterns are overwhelmingly short (≤ 30 chars, ≤ 5 nodes); a recursive `match(node, input, pos, captures)` over the AST is ~200 LOC, has no VM dispatch overhead, and inlines well. The Pike VM is only built for patterns that opt in via a complexity heuristic (alternation count + Kleene star count > threshold), so the engine has *two* matchers, picked at compile time, both small. - -The "AST not bytecode" choice is the genuinely unconventional bit — every textbook engine compiles to bytecode-ish IR because that's what RE2 / PCRE / Go does, but those engines amortise compile cost over throughput, which is the wrong target for a Clojure runtime where most patterns match short strings dozens to hundreds of times. - -**Trades off**: The compile-time cache adds a synchronisation surface (write-once map). On a single-threaded Phase 6 runtime this is trivial; once Phase 11+ adds threads it needs proper handling (one `Mutex`, or an immutable map swapped via CAS). The AST-walker doesn't handle catastrophic backtracking — `(a+)+b` against `aaaa...` blows up; the complexity heuristic must catch this at compile time and route to the Pike VM, which means the heuristic is now load-bearing for correctness. Memory: every distinct regex literal in the program's history lives forever in the cache (no eviction); for a typical Clojure program this is bounded (tens to low hundreds of patterns) but it's a real footprint. Reader-time compile means a malformed `#"..."` throws at read time, not match time — a *better* behaviour than JVM Clojure (which only fails on first use), but it's a visible semantic difference users may notice. - -**File-by-file shape**: - -- `runtime/regex/compile.zig`: Parser producing an immutable AST (`Node` tagged union, arena-allocated into the cache's arena). Computes complexity heuristic and stamps `Program.kind = .ast_walk | .pike_vm`. The Pike VM `Inst[]` is only emitted when needed. Exposes `intern(pattern, flags) !*const Program` against the global cache. ~500 LOC. -- `runtime/regex/match.zig`: Two matchers in one file. `matchAst(prog, input)` is a recursive walker (~250 LOC). `matchPike(prog, input)` is the Thompson VM for opt-in patterns (~250 LOC). Dispatcher is a one-line `switch (prog.kind)`. Both share a small `Captures` struct. -- `runtime/regex/cache.zig`: Process-global `PatternCache` — a `std.HashMap([]const u8, *const Program)` keyed by `pattern ++ flags` bytes, backed by a dedicated `ArenaAllocator` so `Program` lifetime equals process lifetime. ~80 LOC. -- `runtime/java/util/regex/Pattern.zig`: `Pattern.compile(s)` is `cache.intern(s, .{})`. `Matcher` is a value type holding `*const Program` + capture state — no heap allocation for matchers. JVM-shaped API on top. -- `lang/primitive/regex.zig`: Reader change (touches `lang/reader.zig` by reference): on `#"..."` token, call `cache.intern(...)` immediately and emit a Tag Value holding the `*const Program`. `re-find` / `re-matches` / `re-seq` / `re-groups` / `clojure.string/replace` + `split` all dereference the program pointer directly — no compile dispatch on the call path. - -**Work split**: 1-cycle: AST + recursive walker + cache + `intern` + reader integration + Phase 6.6 exit green. The Pike VM is **not** built in cycle 1; the complexity heuristic just rejects patterns it can't handle yet (debt row). 2-cycle: Pike VM for alternation-heavy / star-nested patterns, heuristic refinement. 3-cycle: capture groups in both matchers, `re-groups`, named groups. 4-cycle: `(?i)` via compile-time case-folding into the AST (the AST walker gets the folded form for free), `\b`, anchors. 5-cycle: `comptime`-lifted core patterns (a small build-time codegen step that scans `lang/core/*.clj` for `#"..."` literals and emits a `const _: Program = ...;` for each one — pure structural cleanup, optional). Backreferences and lookaround land as a separate Phase 8+ debt row. - -### Devil's-advocate recommendation - -In the main loop's shoes I'd take **Alternative 2 (two-tier with explicit IR)**. F-002 explicitly says finished-form-clean wins over smallest-diff, and the two-tier shape is what cw v1 will *want* by Phase 10+ regardless — building it now avoids the depth-4 rewrite when regex turns out to be on the macro expansion hot path. The IR boundary also pays back immediately by making the parser, optimiser, and matcher independently testable (one of the smells Alternative 1 risks is "parser and matcher fused into one tangle that's hard to refactor later"). The 3-5 cycles versus 1 cost is real, but the project priority order in CLAUDE.md (project_facts F-002 > ROADMAP > ADRs) puts the final shape ahead of Phase 6 schedule pressure, and the per-task TDD loop absorbs the extra cycles without ceremony. - -That said, Alternative 3's reader-time intern + AST-walker is the one I'd genuinely want to think harder about if I had another hour — the "patterns are immutable interned values, like keywords" framing is much more Clojure-shaped than "patterns are little compiled programs you re-derive at runtime", and it composes beautifully with `comptime`-lifted core patterns. If the main loop has bandwidth for one structural revision later, "promote the Alternative 2 `Program` to live in an Alternative-3-style intern cache" is a single non-breaking refactor — that's the path I'd plan for. The main loop is not bound by this recommendation; the choice is between Alternative 2 today (finished-form-clean immediately) versus Alternative 1 today + structural revision later (smallest-diff first, surgery follows when the smell sensor fires). Both are defensible; the F-002 reading favours 2. - -## Consequences (provisional) - -Depending on the engine chosen, Phase 6.6 implementation work -is: - -- Candidate A: 800-1500 LOC across `runtime/regex/`, - spread over 3-5 cycles. -- Candidate B: 200-400 LOC for the bind + syntax translation, - 1-2 cycles, plus a `build.zig.zon` edit and the Wasm - Component compat note. -- Candidate C: 100-200 LOC for the cgo-ish bind, 1 cycle — - but F-001 conflict. - -Once accepted, the regex Tag's `Value` extracts pattern bytes -and engine state through `runtime/regex/compile.zig` ; -`re-find` / `re-matches` / `re-seq` / `replace` / `split` -flow through `runtime/regex/match.zig` and surface in -`lang/primitive/regex.zig`. The Java surface -`runtime/java/util/regex/Pattern.zig` is a thin Backend -marker (`impl-only`) per ADR-0029 D4. - -## Affected files (provisional) - -- `runtime/regex/compile.zig` (new) -- `runtime/regex/match.zig` (new) -- `runtime/java/util/regex/Pattern.zig` (new) -- `lang/primitive/regex.zig` (new) -- `compat_tiers.yaml` (new fqn entry: `java.util.regex.Pattern`) -- `build.zig.zon` (if candidate B accepted) -- `lang/bootstrap.zig` (register the new primitives) - -## Revision history - -- 2026-05-25 (Proposed → Accepted, same session): Devil's-advocate - `general-purpose` subagent forked with the embedded brief. Output - reflected verbatim into "Alternatives considered". Main loop - selected Alternative 2 (two-tier IR + lazy-DFA over Pike-NFA) - per F-002 finished-form-clean wins. Alternative 3 logged as - post-acceptance follow-up (recall-trigger debt at Phase 10+). -- 2026-05-25 (cycle 1 progress, same session): cycle 1 first - cells landed (commits b5df7db..6a9eb52). All four skeleton - files wired: `runtime/regex/{compile,match}.zig`, - `runtime/java/util/regex/Pattern.zig`, - `lang/primitive/regex.zig`. Cycle-1 first-green: single-char - literal + multi-char literal (Node.concat) + `.` wildcard - (Node.class with all-set bitmap), all green via the - straight-line `tryMatchAt` driver. 12 Layer-1 unit tests. - Remaining for cycle 1: alternation `|`, quantifiers - (`*`/`+`/`?`/`{n,m}`), character classes `[abc]` / - `[^abc]` / `[a-z]`, escape sequences `\d \w \s \b`, anchors - `^` / `$`, group capture `(e)`, and the `re-pattern` / - `re-find` / `re-matches` clojure-peer registration once the - Value path through the reader lands. The proper Pike-VM - thread-list driver replaces `tryMatchAt` at the alternation / - Kleene-star landing. diff --git a/.dev/decisions/0032_bootstrap_multifile_inns.md b/.dev/decisions/0032_bootstrap_multifile_inns.md deleted file mode 100644 index dea10c072..000000000 --- a/.dev/decisions/0032_bootstrap_multifile_inns.md +++ /dev/null @@ -1,199 +0,0 @@ -# 0032 — Bootstrap multi-file loader + `(in-ns 'foo.bar)` primitive - -- **Status**: Accepted -- **Date**: 2026-05-25 -- **Author**: Shota Kudo (drafted + accepted by the autonomous - loop with Devil's-advocate fork) -- **Tags**: phase-6, bootstrap, namespace, in-ns, structural - -## Context - -ROADMAP §9.8 row 6.9 calls for `lang/clj/clojure/string.clj` -(Tier A, ~21 vars) — the first Clojure source file that lives -outside the `user` / `rt` default namespaces. The cw v1 -bootstrap pipeline today (`src/lang/bootstrap.zig::loadCore`) -embeds a single `@embedFile("clj/clojure/core.clj")` and runs -the read-analyse-eval loop with `env.current_ns` left at -`user` (its default per `Env.init`). - -To land `clojure.string/upper-case` etc., two structural pieces -are missing: - -1. **The loader is single-file**. There is no way to embed + - evaluate a second `.clj` file at boot. -2. **No namespace-switching mechanism exists**. `Env` carries - `current_ns` + `findOrCreateNs`, but no `(in-ns 'foo.bar)` - / `(ns ...)` / `(require ...)` primitive is registered. The - analyser does not know how to interpret `(ns ...)` forms. - -Both pieces also feed Phase 7's `require` landing and Phase 16's -WASM-component load (per F-001) — the shape chosen here propagates -forward. - -## Decision - -**Adopt Alternative 2 — `(in-ns 'foo.bar)` primitive + -flat file-iteration loader (Survey Option B, refined).** - -Concretely: - -1. **New primitive** `src/lang/primitive/ns.zig::inNs`, registered - in `rt/` by `lang/primitive.zig::registerAll`. Semantics: - `(in-ns 'sym)` mutates `env.current_ns` to the named namespace - (creating it via `findOrCreateNs` if absent) and returns the - namespace value (matches JVM `clojure.core/in-ns` contract). - Single-arity; sym arg only. `(in-ns 'rt)` / `(in-ns 'user)` - are valid (they target the auto-created namespaces). - -2. **Loader rewrite** `src/lang/bootstrap.zig::loadCore`. The - embed becomes a flat `[]const []const u8` table: - - ```zig - pub const FILES: []const FileEntry = &.{ - .{ .label = "", .source = @embedFile("clj/clojure/core.clj") }, - .{ .label = "", .source = @embedFile("clj/clojure/string.clj") }, - }; - ``` - - The loader iterates each entry's read/analyse/eval loop. The - loader carries **no namespace knowledge** — each `.clj` file - asserts its own namespace via a leading `(in-ns 'foo.bar)` - form. After the last file, the loader resets `env.current_ns` - to `user` so the REPL prompt starts in `user/`. - -3. **Source convention**: every `.clj` file outside `core.clj` - opens with `(in-ns 'name.space)`. `core.clj` does not need - one (it inherits the loader's initial `user` current_ns). - -4. **No analyser-cache invalidation work needed at Phase 6**: - the analyser does not currently cache resolved Vars across - forms, so `(in-ns)` mid-file just changes what `env.current_ns` - points to for the next form's resolution. When the analyser - adds resolution caching (Phase 7+), the `(in-ns)` primitive's - call site is the natural invalidation hook. - -This is the finished-form-clean choice per F-002: Phase 7's -`require` reuses `(in-ns)` verbatim, and the `.clj` files become -self-describing (a reader of `string.clj` sees `(in-ns -'clojure.string)` at the top and knows which namespace it -populates). Paying ~60 LOC for the primitive + ~30 LOC for the -loader rewrite now avoids a Phase 7 rewrite of the loader and -the bootstrap files. F-002 (finished-form wins) decides this -above the smallest-diff Alternative 1. - -**Out of scope** (deferred): - -- `(ns clojure.string (:refer-clojure :exclude [replace reverse]))` - macro — requires `:require` / `:refer` / `:as` semantics and a - namespace dependency graph. Phase 7+ when `require` lands. -- `Extension.init: ?*const fn () anyerror!void` signature widening - (D-048) — the `(in-ns)` primitive lands via `primitive.registerAll`, - not via the `Extension` host_api, so D-048 does not intersect this - ADR. D-048 stays scheduled at its current Status (Phase 6 entry — - already missed-boat but still actionable; carried as a separate - landing). - -## Alternatives considered - -The following 3 alternatives were produced by the Devil's-advocate -subagent (forked with fresh context to counter the main loop's -goal-drift / instruction centrifugation, per CLAUDE.md § Smell -triggers are interrupts, not stops). Verbatim output below. - -### Alternative 1 — Loader-side `current_ns` switch via `BootstrapFile` table (smallest-diff) - -**Shape**: `bootstrap.zig` defines `pub const FILES: []const BootstrapFile = &.{ .{ ns_name: "user", source: @embedFile("clj/clojure/core.clj") }, .{ ns_name: "clojure.string", source: @embedFile("clj/clojure/string.clj") }, ... };`. `loadCore` saves `env.current_ns`, sets it from `file.ns_name` (via `findOrCreateNs`), runs the read/analyse/eval loop, and on the final file resets to `user`. No `(in-ns)` primitive lands; no `.clj` header. ~30 LOC diff to `bootstrap.zig`. - -**What it does better than Option B**: Zero new analyser surface. No analyser-cache invalidation problem (D-048 stays untouched). The loader stays a single-purpose mechanism — no risk of an `(in-ns)` semantic drift between bootstrap-time and user-time. - -**What it breaks vs Option B**: The `.clj` file is not self-describing — a reader of `string.clj` sees no `(ns clojure.string)` clue and must know the loader assigns it. Cannot be re-evaluated through `cljw -e "$(cat string.clj)"` (it would land in `user/`). Pushes the `(in-ns)` decision to Phase 7 instead of now. - -**Phase 7 / Phase 16 horizon read**: Phase 7's `require` lands its own `(in-ns)` primitive — this alternative does **not** make that landing harder; it is orthogonal. Phase 16 WASM-component load (F-001) likely loads from a manifest with `(ns_name, source)` pairs anyway, so the `BootstrapFile` shape is actually closer to the WASM finished form than `(in-ns)` headers embedded in source. - -### Alternative 2 — `(in-ns 'foo.bar)` primitive + flat file iteration (finished-form-clean) - -**Shape**: New `lang/primitive/ns.zig::inNs` registered in `rt/`. `.clj` files start with `(in-ns 'clojure.string)`. `bootstrap.zig::FILES: []const []const u8` is a flat list; the loader has no namespace knowledge — the `(in-ns)` form mutates `env.current_ns` during evaluation. Loader resets `current_ns` to `user` after the last file. - -**What it does better than Alt 1**: `.clj` files are self-describing. The `(in-ns)` primitive is a real Phase 7 building block for `require`. Files are testable through `cljw -e` (modulo loader order). Matches JVM/babashka idiom. - -**What it breaks vs Alt 1**: New primitive surface lands now (small but real). Analyser caching across `(in-ns)` boundaries needs handling — mid-file ns-switch invalidates any per-ns cached resolutions (the survey notes this as Phase 6 scope). Couples D-048 (Extension.init signature) into this cycle. - -**Phase 7 / Phase 16 horizon read**: Phase 7's `require` reuses `(in-ns)` verbatim, saving the rewrite Alt 1 doesn't actually impose (since Alt 1 is non-blocking). Phase 16 WASM-component load could embed `(in-ns)` headers in component sources — works but is less natural than a manifest table; the headers-in-source approach assumes "all loadable units are .clj text" which a WASM component is not. - -### Alternative 3 — Wildcard: register Tier-A `clojure.string` as `rt/`-style primitive group, skip `.clj` entirely - -**Shape**: `lang/primitive/clojure_string.zig` registers each var (`upper-case`, `blank?`, ...) directly into a `clojure.string` namespace via `env.intern(clojure_string_ns, "upper-case", ...)`, identical to how `math.zig` registers `+` into `rt`. No `.clj` file. No `(in-ns)` primitive. `loadCore` stays single-file (just `core.clj`). - -**What it does better than Alts 1/2**: Maximum performance (no read/analyse/eval round-trip for 21 vars at boot). Vars carry direct Zig fn pointers, not closures. No bootstrap-loader extension needed at all. Aligns with how `rt/` already works — one consistent registration pattern for Tier-A namespaces. - -**What it breaks vs Alts 1/2**: **Recreates cw v0's mistake** that the survey §2.2 explicitly identifies — strings.zig at 1824 lines violating ROADMAP A6. Forces Clojure-level helpers (e.g., `split-lines` calling `split` calling `re-find-from`) to be expressed in Zig with manual Value-wrapping instead of native `.clj`. Removes the natural surface for the developer reading `clojure.string` to learn the implementation. **Crucially, this contradicts the survey's DIVERGENCE D2 rationale (ADR-0029 §3.1 places `clojure.string` as Tier-A `.clj` source).** - -**Phase 7 / Phase 16 horizon read**: Phase 7 `require` still needs `(in-ns)` (for user-loaded sources), so this alternative doesn't reduce Phase 7 work — it just delays it. Phase 16 unaffected. Tier-A clojure.* namespaces that arrive later (`clojure.set`, `clojure.walk`, `clojure.pprint`) would each have to choose between this Zig-primitive pattern and the `.clj` pattern, fragmenting the surface. - -### F-NNN constraint-violation check - -None of the three above violate F-001/F-002/F-003/F-009. No -`[VIOLATES F-NNN]` findings. - -### Devil's-advocate recommendation - -**Pick Alternative 2 (= survey's Option B)**, but with one -adjustment the survey under-emphasises: the loader iteration -should be a `[]const []const u8` flat list (refined to a `FileEntry -{label, source}` shape in this ADR), **not** a `(ns_name, source)` -tuple table. Letting `(in-ns)` carry the ns-switch — instead of -duplicating it in a Zig-side table — is what makes the `.clj` files -self-describing and re-evaluable, which is the whole point of -preferring B over A. F-002 (finished-form wins) decides between -Alt 1 and Alt 2: Phase 7 finished form ships `(in-ns)` regardless, -and the cost difference at Phase 6.9 is ~60 LOC + a cache- -invalidation hook — well below the threshold where F-002's -"shipping fast is second-tier" applies. Alt 3 is rejected on -DIVERGENCE D2 grounds (ADR-0029 §3.1 explicitly lists -`clojure.string` as a `.clj`-source namespace). Alt 1 stays a -viable fallback if D-048 resolution turns out to be more -disruptive than expected — but the survey already shows D-048 as -cleanly resolvable in the same cycle. - -## Consequences - -- **Phase 7+ `require` reuse**: the `(in-ns)` primitive lands its - finished-form contract today. `require` then layers - `:as` / `:refer` / dependency-graph computation on top of an - existing namespace-switch mechanism. -- **Bootstrap order is explicit**: the `FILES` table makes load - order grep-able in one place. New `.clj` modules add a row. -- **Single-file loader contract preserved for `core.clj`**: the - first entry's source label remains `` so the error - renderer's existing test cases (per `error_print.SourceContext`) - continue working. -- **`(in-ns)` is a real primitive, not a no-op**: per - `no_op_stub_forbidden.md`, the body mutates `env.current_ns` - and returns the namespace value. A stub that returned `nil` and - did not switch would be a permanent no-op and is forbidden. -- **D-048 stays separate**: `Extension.init` widening is unrelated - to this ADR and lands on its own schedule. -- **REPL prompt unchanged**: `current_ns` resets to `user` after - bootstrap; the REPL (Phase 10+ landing) sees its expected - starting namespace. - -## Affected files - -| File | Change | -|-------------------------------------------|------------------------------------------------------------------------------| -| `src/lang/primitive/ns.zig` | NEW — `inNs` primitive (~60 LOC + tests) | -| `src/lang/primitive.zig` | Add `ns` import + `ns.register(env, rt_ns)` call | -| `src/lang/bootstrap.zig` | Rewrite `loadCore` to iterate `FILES`; add `FILES` + `FileEntry` decls | -| `src/lang/clj/clojure/string.clj` | NEW — `(in-ns 'clojure.string)` header + cycle-1 vars | -| `src/main.zig` | If `bootstrap_ctx.text` / `file` reference assumes single-file source, adapt | -| `compat_tiers.yaml` | `clojure.string` entry (Tier A) + `in-ns` entry (Tier A, rt/ primitive) | -| `test/e2e/phase6_clojure_string_basic.sh` | NEW — cycle-1 e2e smoke | - -## Revision history - -- 2026-05-25 issued and accepted. Devil's-advocate subagent - consulted; output embedded verbatim above. Alternative 2 chosen - with the FileEntry shape refinement. Cycle 1 of Phase 6.9 will - land this ADR's surface; cycles 2-4 fill out the 21-var - `clojure.string` surface per the Step-0 survey at - `private/notes/phase6-6.9-survey.md`. diff --git a/.dev/decisions/0033_clojure_ns_placement_naming_polymorphism.md b/.dev/decisions/0033_clojure_ns_placement_naming_polymorphism.md deleted file mode 100644 index 49f03ec24..000000000 --- a/.dev/decisions/0033_clojure_ns_placement_naming_polymorphism.md +++ /dev/null @@ -1,401 +0,0 @@ -# 0033 — Clojure-ns var placement decision rule + `defn-`/`-name` naming + hybrid polymorphism + transducer 先取り + 10-cycle transient migration plan - -**Status**: Accepted (Devil's-advocate fork landed 2026-05-25) -**Date**: 2026-05-25 -**Author**: Shota Kudo (drafted with Claude autonomous loop) -**Tags**: phase-6-late, structure, clojure-source, self-hosting, -naming, transducer, polymorphism, F-009 - -## Context - -cw v1 が Phase 6.9-6.11 で着地した 32 vars (`clojure.string` 22 + -`clojure.set` 7 + `clojure.walk` 3) は全て **Zig 直書き**、 対応する -`.clj` ファイルは `(in-ns ...)` ヘッダーのみで defn 0 個。 これは -ADR-0032 (multi-file bootstrap loader) が用意した Layer 3 `.clj` 路線 -が未活用な状態を意味する。 - -問題: - -- (P-1) cw v0 で繰り返した「全部 Zig」 → user composition vacuum (= 「ユーザーが `(reduce conj #{} ...)` 書けない」)、 自己ホスティング時 1000+ 行翻訳作業、 JVM upstream diff visibility 喪失 -- (P-2) 「愚直に JVM `.clj` port」 = Java InterOp で詰む、 class hierarchy 全部要求、 「core 完成まで何も書けない」 で Zig に逃げる failure mode -- (P-3) Phase 7 (protocol + transducer fused)、 Phase 10 (Tier A polish official)、 Phase 12 (bytecode cache)、 Phase 14 (cljw build v0.1.0)、 Phase 16 (zwasm + ClojureScript)、 Phase 17 (JIT) の全 Phase で placement が stable な投資である必要 - -cw v0 真の構造 (`embedded_sources.zig` 6206 行 + `__zig-` prefix 規約) -は (P-1) を回避するための ad-hoc な「Pattern A `.clj` defn + Pattern B -Zig leaf」 折衷案だった (cw v0 evidence)。 cw v1 では ADR-0032 が clean -な multi-file loader を用意済、 ただし中身が空。 本 ADR がこれを埋める -formal placement + naming + 段階移行を確定する。 - -詳細な discussion は `private/notes/clj_vs_zig_split_proposal_v5.md` -(1593 行、 self-contained) を SSOT として参照。 - -## Decision - -### D1: 3-layer architecture (v5 §2.1) - -``` -Layer 3: Clojure source src/lang/clj/clojure/*.clj - (Pattern A defn + Pattern B2 1-line shim 同居) - ↓ calls -Layer 2: Zig primitive src/lang/primitive/ - (2a) core glue primitives ← Pattern A が依存 - (2b) `-name` leaf (B2) / 直 intern (B1) - ↓ calls -Layer 1: runtime/ pure Zig / OS / std binding (F-009 neutral) -``` - -### D2: Placement decision rule (B-Q1 → B-Q4) - -``` -B-Q1: impl は他の user-callable Clojure var の合成として書けるか? - YES → Pattern A (.clj defn)、 ingredient が Layer 2 既登録前提 - NO → B-Q2 -B-Q2: impl は neutral impl 1 個の薄いラッパーか? - YES → Tier A surface (clojure.X/v) → Pattern B2 (-name leaf + 1-line shim) - OS leaf / Java InterOp 純置換 → Pattern B1 (Zig 直 intern) - NO → B-Q3 -B-Q3: impl は polymorphic dispatch / core glue 自身か? - YES → Pattern B1 (Layer 2 2a) - NO → B-Q4 (ADR レベル判断、 Devil's-advocate fork) -``` - -完全 flowchart: v5 Appendix E。 - -### D3: 2 Patterns + sub-patterns - -| Pattern | 配置 | 適用ケース | -|---------|----------------------------------------------|-----------------------------------------| -| A | Layer 3 `.clj` (`defn`) | composition、 公開 var | -| B1 | Layer 2 `.zig` 直 intern (`.clj` 不在) | OS leaf、 JVM 対応 diff 不要、 公開 var | -| B2 | Layer 2 `-name` intern + Layer 3 1-line shim | Tier A surface、 JVM diff 維持 | -| C-thin | `.clj` で 1-2 条件分岐 + leaf 呼出 | `blank?` 等 | -| C-zig | Layer 2 全実装 (B1 同形) | multi-arity 複雑 / hot path | - -「C-fat」 「transient」 等の中間 label は廃止 ── transitional 状態は -`placement.yaml` の `status: transient_zig` field で表現。 - -### D4: Naming convention `defn-` + `-name` + metadata - -- 公開 var: `clojure.X/upper-case` (通常 `defn`、 prefix なし) -- 内部 leaf: `clojure.X/-upper-case` (single-dash prefix、 `defn-` = - `^:private` 同等、 `^:zig-leaf true` metadata 付き Zig intern) -- **両者同一 ns に共存** (sub-ns `impl/` 案は採用しない理由 = 完成後の - 取り残し risk + 分散コスト、 v5 §App E + 棄却根拠) -- JVM Clojure 慣習 `defn-` に完全準拠 -- **`__zig-` 等の別 namespace marker は採用しない** (= 機械生成感 + - Clojure 慣習からの drift) -- `private` だけで全ケース賄える: 公開 surface (`defn`) / 公開 Zig 直 - intern / 内部 B2 leaf (`-name` + `defn-`) / 内部 Zig leaf - (`^:private :zig-leaf` metadata) のいずれも prefix + metadata で - 区別可能 (v5 §3.1.1 確認表) - -### D5: Hybrid polymorphism (Zig 強さ + Protocol extension) - -Phase 6.16.a-1/a-2 で着地する polymorphic primitives (count / seq / -conj / reduce 等) は: - -- **Phase 6.16.a 段階**: Zig Tag switch hardcode (NaN-box Tag 0..15 を - comptime/runtime で switch、 inline 最適化 + 型推論利用) -- **Phase 7 段階** (D-069 row): Protocol extension point を **追加 path** - として開く (置換ではない)。 fast-path = Tag switch 維持、 slow-path = - `extend-protocol` されたものだけ通る。 user の `(extend-type MyColl - IPersistentColl ...)` を許す。 - -実装 shape (推奨): - -```zig -pub fn count(rt: *Runtime, coll: Value) !i64 { - switch (coll.tag()) { - .vector, .list, .cons, .lazy_seq, - .array_map, .hash_map, .hash_set, .string_seq => { - return countBuiltin(rt, coll); // fast path - }, - .protocol_extended => { - return rt.protocol.dispatch(.count, coll); // slow path - }, - else => return error.type_error, - } -} -``` - -**申し送り**: 「protocol 着地で全 dispatch を polymorphic 化」 ではなく -「fast-path 維持 + extension point を 追加」。 各 Phase のベスト形を -尽くす方針 (= 妥協ではない)。 D-069 で Phase 7 entry に formal -landing。 - -### D6: Core glue primitive set + var-level dependency order - -Phase 6.16.a-1/a-2/a-3 で着地する確定 set + 順序: - -``` -6.16.a-1 fundamentals (1 cycle): - count → seq → first → rest → cons → empty - -6.16.a-2 collection ops (1 cycle): - conj → disj → contains? → get → nth → assoc → dissoc → keys → vals - -6.16.a-3 higher-order + transducer (2-3 cycles): - apply → reduce (素朴版) → into → map → filter → take → drop → keep - → remove → every? → some → some? - + Layer 3 .clj defn: partial / comp / complement / constantly / juxt -``` - -順序は本 ADR で確定、 cycle 内で実装中に判明する依存は recall trigger -として permit (ADR amendment で記録)。 詳細 cycle deliverable は ROADMAP -§9.8 6.16.a-X rows。 - -### D6a: Transducer 先取り spec - -Phase 6.16.a-3 で `map` / `filter` / `take` / `drop` / `keep` / `remove` -を着地する際: - -- **multi-arity 完全実装**: 1-arg arity (transducer return) + multi-arg - arity (eager) を同時着地 -- **rf protocol を Layer 2 で正式登録**: `(rf)` (init) / `(rf result)` - (completion) / `(rf result input)` (step) -- **`into` も transducer-aware**: `(into [] xform coll)` 形 - -Phase 7 で着地する追加 (transduce / eduction / sequence transducer-aware -/ IReducingFn formal definition / partition-by 等 transducer-only HOF) -は **既存 defn 書き換え不要**。 - -### D7: 10 cycle 段階移行計画 - -``` -6.16.a-0: env.intern API metadata 拡張 (small prerequisite) -6.16.a-1: core glue fundamentals (6 vars) + Tier 0 metadata size 実測 -6.16.a-2: core glue collection ops (9 vars) -6.16.a-3: core glue higher-order + transducer (12 vars + 5 .clj HOF) - [2-3 cycles range] -6.16.b: clojure.set 12 vars .clj 化 (Group A+B+C 一括、 D-061 解消) -6.16.c: clojure.walk 10 vars 着地 (8 A + 1 B2 leaf + 1 declare-only - macroexpand-all、 Phase 7 で実装解除) -6.16.d: clojure.string Pattern B2 14 vars shim 化 -6.16.e: clojure.string Pattern A + 混合 8 vars .clj 化 - (D-062 cluster row 解消条件) -``` - -= 9-11 cycle 追加、 Phase 6 完了 3-4 month 後ろ倒し受容。 詳細は -ROADMAP §9.8 + v5 §9。 - -### D8: env.intern API metadata 拡張 - -`env.intern(ns, name, value)` を: - -```zig -pub fn intern( - self: *Env, - ns: []const u8, - name: []const u8, - value: Value, - metadata: ?MetadataMap, -) !*Var -``` - -`MetadataMap = { private: bool, zig_leaf: bool, unsupported: bool, -doc: ?[]const u8, arglists: ?[]const u8 }`。 analyzer special-case で -`^:private` 違反を compile-time `private_access_error` raise、 -`^:unsupported` declare-only var の呼出時に `feature_not_supported` -raise。 D-065 implementation row、 Phase 6.16.a-0 cycle で先行着地。 - -### D9: Babashka 機械解析 script (placement.yaml 初期生成) - -`scripts/analyze_clojure_upstream.bb` (Babashka script、 ADR-0033 起票 -**前** に着地、 v5 §10 + §15.4): - -- 入力: `~/Documents/OSS/clojure/src/clj/clojure/{string,set,walk,zip}.clj` -- 出力: `private/notes/upstream_var_analysis.edn` (per-var、 pattern - suggestion 付き) -- `scripts/gen_placement_yaml.bb` がこれを入力に `placement.yaml` - initial 版を生成、 手動補正で `recall_trigger` / `composition_deps` - refinement、 ADR-0033 起票時に commit - -事前測定済 (v5 §10.2): set/walk/zip は 100% pure Clojure、 string は -17% Java InterOp、 機械判定で十分。 - -### D10: JIT independence claim - -cljw build の embed 方式 (bytecode embed 確定、 ADR-0034) と将来の cw -独自 JIT 導入は **完全に直交**。 cw v0 Phase 32 (source bundling) の -後の Phase 37.4 で ARM64 JIT PoC が問題なく着地 (arith_loop 10.3x -speedup) した evidence。 - -- **narrow JIT** (hot loop only、 cw v0 同形): bytecode のみ入力で完結、 - 完全直交 -- **broad JIT** (tracing / method JIT、 zwasm v1 同形): optimization 用 - に source-level metadata が必要、 ADR-0034 amendment で bytecode に - optional source-metadata field 追加で対応 (ABI 維持しつつ拡張) - -Phase 17 go/no-go どちらでも本 ADR placement は影響なし。 - -## Alternatives considered - -Devil's-advocate fork (general-purpose subagent、 fresh context、 -2026-05-25 issuance、 F-001/F-004/F-005/F-006/F-009 envelope 内) -output verbatim: - -#### Alt 1: ad-hoc Zig-direct, no formal placement rule - -現状維持寄りの最小変更案。32 vars を Zig 直書きのまま保持し、`.clj` 化は「気付いた人がやる」式に任せる。ADR-0033 は起票せず、ROADMAP §9.6 cycle table に「opportunistic な .clj 化」一行を足す程度に留める。 - -- **Better**: 着地コストゼロ。Phase 6.16 entry を Phase 7 fast-path 設計に直結できる (placement 議論で時間を取らない)。env.intern metadata 拡張 / analyzer private check / babashka script / 10 cycle 移行表という新規 surface area を全て回避できる。`zig build test` の構造変更も不要。 -- **Breaks**: (i) Pattern A composition (例: `clojure.string/blank?` = `every? char-whitespace?`) を書く際に call boundary が Zig 側に閉じ、placement の SSOT が「source を読まないと分からない」状態が永続化する。(ii) cw v1 が Clojure 本流の「stdlib は .clj で書かれている」性質を再現できず、F-009 (feature-implementation neutrality) の "neutral impl 共有" 主張の説得力が下がる — 共有点が全部 Zig 関数になるため、Clojure-ns / Java-ns 双方から「同じ .clj source を呼んでいる」事実が成立しない。(iii) transducer の 1-arg arity と multi-arity を Zig 内で同時着地させる場合、rf protocol が Layer 2 に formal 登録されないまま hard-coded dispatch になり、Phase 7 protocol extension の merge コストが後ろ倒しになる。 -- **F-NNN 関係**: F-009 と弱衝突 (violate ではなく weaken)。F-001/F-004/F-005/F-006 は影響なし。 -- **F-NNN violation**: なし (envelope 内)。 - -#### Alt 2: ADR-0033 本提案 (3-layer + D1-D10) - -3-layer architecture (runtime / lang/primitive / lang/clj)、Pattern A/B1/B2/C-thin/C-zig 分岐、`defn-` + `-name` + `^:private :zig-leaf` metadata、Hybrid polymorphism (Tag fast-path + Protocol extension)、core glue primitive set + var-level dependency order、transducer 先取り (Phase 6.16.a-3)、10 cycle 段階移行、env.intern metadata 拡張、babashka 機械解析、JIT independence claim。 - -- **Better**: placement の SSOT が `.clj` source + `placement.yaml` に二重化され、人間にも機械にも explicit。Pattern A composition が Clojure source として直接読め、JVM upstream との diff が `bb run lint-placement` で機械検出可能。transducer rf protocol を Phase 6.16.a-3 で formal 登録するため Phase 7 protocol 拡張の追加コストが小さい (= 既存 path を置換しない、追加 path として開く)。F-009 neutral impl 共有が source level で観測可能になる (Clojure-ns / Java-ns が同じ .clj を呼ぶ)。 -- **Breaks**: (i) 10 cycle 移行中に Layer 2 thin wrapper と Layer 3 .clj source の責任境界が時々曖昧になる risk (特に Pattern B2 shim — Zig leaf + .clj shim の二段)。(ii) env.intern metadata 拡張 + analyzer private check は新規 surface area で、Phase 6.16.a-0 で 1 cycle 消費する。(iii) babashka 機械解析 script は JVM upstream の Clojure macro expansion に弱く、初期 placement.yaml 生成後の人間 review が事実上必須。(iv) `defn-` + `-name` の double-marker convention は読みやすさ減 (Clojure 慣習では `defn-` 単独で private を表す)。 -- **F-NNN 関係**: F-009 を strengthen (neutral impl 共有が source level で成立)。F-004 Tag fast-path は Hybrid polymorphism D5 で温存。F-001/F-005/F-006 は影響なし。 -- **F-NNN violation**: なし。 - -#### Alt 3: Protocol-only placement (Pattern A 全廃、全 var を polymorphic protocol method) - -wildcard 案。32 vars 全てを ADR-0008 Protocol dispatch に統合し、Pattern A composition / Pattern B1/B2 分岐を廃止する。clojure.string/blank? も clojure.set/union も clojure.walk/walk も全て protocol method として実装、Tag fast-path は protocol の default impl が Tag switch を内包する形で表現。`.clj` source は protocol 宣言 + extend-type 群のみで構成され、placement.yaml は不要 (protocol registry が SSOT)。 - -- **Better**: placement の表現単位が一つ (protocol method) に統一され、Pattern A/B1/B2 の使い分け判断が消える。Phase 7 protocol extension が「新しい何かを開く」のではなく「既存 mechanism の追加 implementer」となり、conceptual coherence が高い。Java-ns / Clojure-ns / cljw-ns の cross-surface 主張 (F-009) が protocol という単一 abstraction に立脚し、説明が単純化する。 -- **Breaks**: (i) NaN-box Tag 軸の direct dispatch (F-004 前提) と protocol dispatch の二重化を強要する — protocol method の default impl が Tag switch を内包する形になり、`count` / `first` 等 hot path に protocol resolution の overhead が乗る risk。Phase 7 narrow JIT で inlining し切れない場合 v0 baseline (arith_loop 10.3x) を下回る恐れ。(ii) transducer rf protocol が「protocol を protocol で実装する」入れ子になり、Layer 2 formal 登録の抽象度が一段上がる。(iii) Clojure 本流は protocol を「特定 abstraction の拡張点」として使い、stdlib 全体には用いない — JVM upstream との diff が大きくなり babashka 解析の前提が崩れる。 -- **F-NNN 関係**: F-004 (Tag 軸 polymorphic dispatch) と緊張 — protocol dispatch を主、Tag switch を従に位置付けるため、F-004 の "Tag 軸前提" を弱める方向に働く。F-009 は strengthen。F-001/F-005/F-006 は影響なし。 -- **F-NNN violation**: F-004 を strict reading すると violation 寄り (Tag dispatch を従属させるため)。loose reading (F-004 = NaN-box slot 数の確約のみ) なら envelope 内。本 alternative は loose reading 前提で envelope 内とする。strict reading を user が採る場合、本 alt は提案不可。 - -## Selection rationale - -Alt 2 (本提案) を選択。 F-002 (finished-form wins) + F-009 (feature- -implementation neutrality を strengthen) の合致が決定的。 Alt 1 は F-009 -weaken が致命的、 Alt 3 は F-004 strict reading 下で violation 寄り -(loose reading での envelope 内入りも hot path overhead risk が大きく -Phase 17 JIT go の場合の baseline 維持に不確実性を導入する)。 - -Alt 2 の Breaks (i)-(iv) は受容コストとして識別: -- (i) Pattern B2 shim 二段の責任境界曖昧化 → §15 placement.yaml SSOT で - per-var leaf_loc / target_loc 明示することで risk 低減 -- (ii) env.intern metadata 拡張の 1 cycle 消費 → Phase 6.16.a-0 - prerequisite として明示計画化、 サプライズなし -- (iii) babashka 解析の人間 review 必須 → ADR-0033 起票時の確認 - workflow に組込み済 (v5 §10) -- (iv) double-marker convention 可読性 → JVM Clojure `defn-` 慣習との - 互換性が user familiarity を担保、 Zig leaf marker (`^:zig-leaf`) は - metadata なので surface text は標準 Clojure - -## Consequences - -- 32 vars が 9-11 cycle で migrate、 Phase 6 完了 3-4 month 後ろ倒し -- `placement.yaml` (新規) が Clojure-ns var placement の SSOT、 - `compat_tiers.yaml` (既存) は Java/cljw surface SSOT として共存 -- Phase 7 protocol dispatch (ADR-0008) amendment が必須 (D-069)、 hybrid - polymorphism pattern の formal landing -- Phase 11 conformance Layer 5 で Pattern A `.clj` が JVM upstream test - と直接比較可能 (= conformance gap 検出が早期化) -- Phase 12 bytecode cache の主要対象 が Pattern A `.clj` defn、 build- - time bytecode 化により runtime cost ゼロ -- Phase 14 cljw build single mode + `cljw render-error` 着地時に - Pattern A `.clj` source が source map ref として有効活用 -- Phase 16 ClojureScript transpiler の入力として Pattern A `.clj` 自然 - (D-068 spec ADR)、 ただし `-name` leaf は JS interop 置換必要 -- Phase 17 JIT go/no-go どちらでも placement 無影響 (D10 claim) - -## Affected files - -- `placement.yaml` (新規、 initial scaffold 着地、 32 transitioning vars - + 6 core glue entries) -- `compat_tiers.yaml` (schema 変更なし、 既存維持) -- `.dev/ROADMAP.md` §9.8 (rows 6.16.a-0 から 6.16.e 追加) + - §9.14/§9.16/§9.18/§9.19 (deliverable 拡張) -- `.dev/debt.md` (D-062 cluster row + D-063 ADR-0035 起票 + D-065 - env.intern 拡張 + D-067 Wasm FFI spec + D-068 ClojureScript spec + - D-069 Phase 7 hybrid polymorphism + D-058 subsumed flip) -- `.dev/handover.md` (Resume contract = ADR-0033 起票 → Phase 6.16.a-0 - へ) -- `CLAUDE.md` (Data sources に placement.yaml 追加) -- `private/notes/clj_vs_zig_split_proposal_v5.md` (v5 plan、 SSOT、 - 1593 行、 self-contained) - -Phase 6.16.a-0 cycle 以降の Affected files: - -- `src/runtime/env.zig` (D-065 intern API metadata 拡張) -- `src/eval/analyzer/symbol.zig` (private violation check + unsupported - marker) -- `src/lang/primitive/core/{core,sequence}.zig` (Phase 6.16.a-1 から - fundamentals + collection ops) -- `src/lang/primitive/string.zig` / `set.zig` / `walk.zig` (Phase - 6.16.b/c/d/e で leaf rename + thin layer) -- `src/lang/clj/clojure/{string,set,walk}.clj` (Pattern A defn + Pattern - B2 1-line shim) -- `scripts/check_placement_status.sh` (新規、 audit) -- `scripts/analyze_clojure_upstream.bb` (新規、 Babashka generator) -- `scripts/gen_placement_yaml.bb` (新規、 Babashka yaml generator) - -## Revision history - -- 2026-05-25 issued + accepted with Devil's-advocate fork - (general-purpose subagent、 fresh context、 F-NNN envelope 内 3 - alternatives 取得 verbatim、 Alt 2 採択、 Alt 3 が F-004 strict - reading で violation 寄りの旨を明示)。 v5 plan (1593 行、 self- - contained) を SSOT として参照、 Phase 6.16.a-0 ~ .e の 10 cycle - segment が ROADMAP §9.8 に landing。 - -- 2026-05-26 D4 amendment (Phase 6.16.b-4 sub-cycle a) — D-071 Part 3 - landed. `LEAF_ENTRIES` (`-map-eager` 6 個) を `rt` ns から - `clojure.core` ns に再配置 + `.private = true` flip。 当初 D4 の - 「`-name` Pattern B2 leaves は rt 配置 + `(in-ns 'clojure.core)` - が core.clj に着地後に `.private = true` flip するだけで足りる」 - という framing は不正確 (= `higher_order.zig:391-394` の - deferral comment が描いた path)。 analyzer cross-ns private check - (`analyzer.zig:374-381`) は `env.current_ns` と `v_ptr.ns` を - 比較するため、 leaf が rt ns 滞在のままだと core.clj の wrapper - (`current_ns = clojure.core`, `v_ptr.ns = rt`) が異 ns 扱いで - bootstrap-time `private_access_error` を起こす。 leaf の所有 ns - 自体を `clojure.core` に移すことで same-ns wrapper resolution - + cross-ns user 拒否の両立が finished-form-clean に得られる - (F-002 reasoning)。 - - 実装: `Env.init` で `clojure.core` ns を boot-time 作成 - (`runtime/env.zig:223-237`)、 `primitive.zig::registerAll` が - `clojure_core_ns` を look up して `higher_order.register(env, - rt_ns, clojure_core_ns)` に渡す、 `higher_order.register` が - `ENTRIES` (`some?`) を rt_ns に、 `LEAF_ENTRIES` を - clojure_core_ns + `.private = true` で intern。 core.clj の - `(in-ns 'clojure.core)` は既存 ns への switch に変わる (idempotent - per `findOrCreateNs`)。 evalInNs / op_in_ns の rt-auto-refer は - core.clj が依然として `count`/`conj`/`reduce` 等の rt primitive を - unqualified で参照するため維持 (= 削除は core.clj の全 rt 参照を - qualify する大規模 surgery を伴うため別 cycle)。 - - Devil's-advocate fork (fresh context) で F-002 / F-007 / F-009 - envelope 内 3 alternatives 取得: Alt 1 (smallest-diff = "rt と - clojure.core を same-cluster 化する分析器拡張") は新 privacy 概念を - 発明する Smallest-diff bias smell、 Alt 2 (finished-form-clean = - proposal + evalInNs rt-auto-refer 削除) は user-facing access の - cleanup として推奨されたが core.clj が rt primitives を unqualified - で多数参照する事実を見落としており full Alt 2 採択は sub-cycle 範囲 - 外 (= 別 cycle で再評価)、 Alt 3 wildcard (a: D-071 Part 3 を Phase 7 - に defer、 b: `^:private` を convention-only に縮退、 c: macro inline) - はいずれも close-out predicate が満たされた状態を不必要に先送りする - Progress-pressure smell の逆方向。 採択: proposal そのまま (= leaf - re-home + private flip)。 violates-F-NNN finding: なし。 - - test 追加: `test/e2e/phase6_16_b_4_private_leaf.sh` 5 assertions - (public wrapper / user qualified denied / filter denied / same-ns - ok)。 D-071 row 更新は次 commit (sub-cycle b/c/d で ADR-0035 完了 - 時に併せて Part 3 完了に flip)。 - -- **2026-08-04 (ADR-0178)**: the **migration schedule** this ADR carried is - recorded **complete**; the **placement rule (D2-D5) is unchanged and still - governs new vars**. D-062, the cluster row tracking the Pattern A/B migration - of the Phase 6.9-6.11 Zig-written vars, was discharged 2026-05-27 with - `clojure.string/escape` as the sole residual (D-094) — but `data/placement.yaml` - never followed: 42 of its 49 rows still read `status: transient_zig` for vars - whose `.clj` bodies had already shipped, and the file had drifted further - (two `leaf_loc` paths pointing at nonexistent files, two of three cited helper - scripts never written, no gate reading it, 49 of ~683 publics covered). - ADR-0178 replaces the hand-maintained file with a **generated** index derived - from `(cljw.internal/__dump-placement)` over every bundled namespace, plus a - `placement_drift` gate. What the generated index deliberately does NOT record - is D7's aspirational half — "this var's current placement is not its intended - one" is a judgement with a barrier and belongs in `.dev/debt.yaml` (today one - row, D-094). No forward policy about where future `clojure.core` vars should - live is stamped: that would be F-003 seizure over an unbuilt residual, and D5 - already answers it for hot paths (fast-path = Tag switch). diff --git a/.dev/decisions/0034_cljw_build_single_mode_tier0_metadata_edn_decode.md b/.dev/decisions/0034_cljw_build_single_mode_tier0_metadata_edn_decode.md deleted file mode 100644 index 8162dc73e..000000000 --- a/.dev/decisions/0034_cljw_build_single_mode_tier0_metadata_edn_decode.md +++ /dev/null @@ -1,1049 +0,0 @@ -# 0034 — cljw build single mode + Tier 0 metadata + structured EDN error event + post-mortem decode - -**Status**: Accepted (Devil's-advocate fork landed 2026-05-25) -**Date**: 2026-05-25 -**Author**: Shota Kudo (drafted with Claude autonomous loop) -**Tags**: phase-12-prep, build-pipeline, error-observability, -edge-target, F-001, F-006, F-009 - -## Context - -cw v1 のメインターゲットは **エッジ実行** (Cloudflare Workers / -WASM Component / serverless cold-start)、 起動 < 5ms 要件。 `cljw -build` の output 形式 + error observability spec を本 ADR で確定する。 -v5 §11-§14 + §19.2 SSOT。 - -cw v0 履歴 (Phase 28 source bundle → Phase 31 bytecode embed → -Phase 32 source bundle default + 32.2-32.3 bootstrap cache → -Phase 37.4 narrow JIT 着地) の 4 段階を再評価: - -- Phase 32 で source bundle に統一したのは UI 整理 + multi-file - require chain の bytecode dependency order 解決コスト回避が動機、 - **技術的判断ではない** -- Phase 31.5 短期間の bytecode embed user app 体験 + Phase 32.2-32.3 - bootstrap cache (12ms → 3-4ms 起動高速化) が user の 「bytecode 方が - 優れていた気がする」 直感の合成記憶根拠 -- Phase 37.4 (ARM64 JIT PoC、 arith_loop 10.3x speedup) が source bundle - 形態の **後** に問題なく着地、 **JIT は embed 方式と完全直交** evidence - -エッジ用途で source bundle (cold start +20-50ms) は許容不可。 -**bytecode embed default** が技術的に正解。 ただし cw v0 が source -bundle を選んだ複雑さ (multi-file require chain の bytecode dependency -order) は本 ADR では ADR-0035 (require spec) に委譲し、 ADR-0034 は -build pipeline + error observability に focus する。 - -Phase 6.16.a-1 cycle (commit d35dc3b) terminus で Tier 0 metadata size -baseline 計測完了 (`bench/quick_baseline.txt` の `binary_size_bytes` -row、 release build = 820,568 bytes 近辺)、 これが本 ADR の Tier 0 -overhead measurement の reference baseline。 - -## Decision - -### D1: cljw build single mode、 flag ゼロ - -```sh -cljw build app.clj -o app - → app (single binary、 source 含めない) -``` - -`--debug` / `--source` / `--aot` / `--no-symbols` 等の flag を **全廃止**。 -default 動作のみ。 user は何も意識しない。 dev / REPL は別 command: - -```sh -cljw file.clj # direct source eval -cljw repl # interactive REPL -cljw render-error err.edn # post-mortem decode (D11) -``` - -build mode と dev mode を完全独立に運用、 dev 体験は source 直 eval で -snippet 描画含めて完全。 - -### D2: bytecode embed default、 source 含めない、 sidecar なし - -binary 内容: - -| 項目 | 必須? | 備考 | -|----------------------------|----------|----------------------------------------------------| -| cljw runtime | 必須 | Reader + Compiler + VM + GC 同梱 | -| Bootstrap cache (bytecode) | 必須 | core lib build-time bytecode 化 | -| User code bytecode | 必須 | source の 0.8-1.2x | -| Tier 0 metadata | 必須 | bytecode の 5-10% (D3) | -| build-id | 必須 | git SHA + cljw version (D5) | -| Source (full) | 含めない | dev 時の手元 git source + cljw render-error で復元 | - -**cljw runtime には Reader + Compiler が常時同梱** (= bytecode embed -default でも runtime の `(eval ...)` / `(load-string ...)` / `defmacro` -の macroexpand のため必須)、 bootstrap cache restore path で Reader + -Compiler を bypass。 - -### D3: Tier 0 metadata (var/file/line/col interned tables) 常時 ON - -各 bytecode op に対応する: -- var name (interned string table) -- file path (interned string table) -- line number (delta-encoded) -- col number (delta-encoded) - -= 最小限の symbolication 情報のみ、 source 全文は含めない。 stack -trace の line/col + var name を production 環境でも無条件に取れる。 - -size budget: 5-10% overhead (Phase 6.16.a-1 baseline = ~820 KB release -binary に対する measurement reference)。 超えた場合 ADR-0034 amendment -で削減策。 - -### D4: Deno-style binary trailer 形式 + bootstrap cache build.zig 統合 - -``` -[cljw runtime binary (Mach-O / ELF / Wasm)] -[bootstrap cache: serialized bytecode + var table + ns table] -[user app bytecode payload] -[Tier 0 metadata table] -[build-id block] -[u64 size of payload+metadata+build-id] -["CLJC" magic 4 bytes] -``` - -起動 flow: -1. cljw runtime が argv[0] から自身を open -2. 末尾 4 bytes を read、 magic check -3. u64 size を read、 payload + metadata + build-id 領域を mmap -4. bootstrap cache restore (~3-4ms、 cw v0 evidence) -5. user app bytecode → VM dispatch -6. (error 時) Tier 0 metadata から var/file/line/col を引いて trace 構築 - -bootstrap cache は build.zig 時に `cache_gen.zig` が全 -`src/lang/clj/clojure/*.clj` を eval して VM state を serialize、 -binary に `@embedFile` (cw v0 Phase 32.2-32.3 形態維持)。 - -### D5: build-id 形式 - -```edn -{:git "abc1234567890abcdef" ; commit SHA (40 chars or short) - :cljw "0.1.0" ; cljw version (semver) - :built #inst "2026-05-25T12:34:56Z"} -``` - -map 形式で binary 内に EDN として埋め込み。 error event 出力時に必ず -含める (D8)。 dev 側 post-mortem (`cljw render-error` D11) が `:git` SHA -で `git checkout` 同期を user に案内、 `:cljw` で decoder archive -dispatch (D11)。 - -### D6: format version policy = ABI commitment 不要 - -cljw build の output は **self-contained**: runtime + bytecode が同 -binary に接着、 mismatch 構造的に発生しない (= Deno / Bun / Go と同 -pattern)。 - -- bytecode format は **internal-only**、 user 視点では存在しない -- format version up は新 cljw build で再生成すれば済む、 既存 binary - は古い cljw runtime で永久動作 -- **format version policy 不要**、 ABI commitment 不要 - -唯一の互換性責任: **`cljw render-error` decoder の永久互換性** -(D11 archive policy)。 - -### D7: error output = stream-separated (TTY=human / pipe=structured EDN one-line) - -stderr は **常に 1 つの format**: - -| 環境 | format | 用途 | -|-------------------|------------------------------------------|----------------------------------| -| TTY (interactive) | human readable (色付き、 babashka-style) | 対話開発、 REPL、 dev binary | -| pipe (非 TTY) | structured EDN (1 event = 1 line) | production logs、 log aggregator | - -混合しない。 pipe で渡せば直接 jq/grep/awk で機械処理可能 (EDN→JSON -1 段変換)。 - -### D8: EDN event schema - -```edn -{:phase :runtime | :analysis | :read - :category :type_error | :name_error | :arity_error | :private_access_error | ... - :message "split: expected string, got nil" - :file "src/handlers/auth.clj" - :line 42 - :col 8 - :var clojure.string/split ; symbol - :ex-data {:input nil, :sep #"\s+"} ; user attached data - :trace [{:var handlers.auth/parse-token :file "..." :line 42 :col 8} - {:var handlers.auth/login-handler :file "..." :line 18 :col 3}] - :build-id {:git "abc1234" :cljw "0.1.0" :built #inst "2026-05-25T12:34:56Z"} - :timestamp #inst "2026-05-25T12:35:01.234Z" - :request-id "abc-123" ; optional、 user runtime が `cljw.error/with-context` で injectable - :trace-id "xyz-456"} ; 同上 -``` - -完全 spec は v5 Appendix C 参照。 1 event = 1 line (改行は `\\n` -escape)、 top-level key 順固定 (`:phase :category :message :file :line -:col :var :ex-data :trace :build-id :timestamp `)。 - -### D9: env var override - -``` -CLJW_ERROR_FORMAT=human # 強制 human -CLJW_ERROR_FORMAT=structured # 強制 structured EDN -CLJW_ERROR_FORMAT=both # 両方 (human stderr + structured CLJW_ERROR_LOG) -CLJW_ERROR_LOG=/var/log/cljw.edn # structured を別 file へ、 stderr は human -``` - -Phase 14 v0.1.0 release で stable spec として lock (D-066)。 - -### D10: human renderer = cw v1 既存 babashka-style 流用 - -cw v1 現状の renderer (`src/runtime/error/render.zig` 相当) を流用、 -trace 描画追加のみ。 D7 の TTY-aware default で structured EDN trailer -を混合しない (= human stderr / structured `CLJW_ERROR_LOG` は別 channel、 -混合 parse 問題なし)。 - -### D11: `cljw render-error` post-mortem tool + decoder archive - -CLI: -```sh -cljw render-error err.edn - → 1 event ごと human render 出力 (TTY なら色付き、 pipe なら plain) - → :file の source が手元 git repo にあれば snippet + caret 復元 - -cljw render-error --source-from-git $sha err.edn - → 指定 SHA で git checkout → source snippet 復元 - -cljw render-error --format json err.edn - → EDN を JSON に変換出力 (log aggregator 用) -``` - -decoder archive policy: -- `cljw-formats/.edn` archive を repo 内に永久保管 -- 例: `cljw-formats/0.1.0.edn` には 「op_call = 0x10、 op_const = 0x11、 - ...」 等の opcode table -- `cljw render-error` は build-id `:cljw` を読んで対応 archive を load - して decode -- 新 cljw release ごと archive 追加、 **削除しない** -- = decoder 単方向永久互換性、 encoder 互換性責任なし - -Phase 12 entry で internal API + decoder skeleton 着地 (D-064)、 Phase -14 v0.1.0 release で `cljw-formats/0.1.0.edn` archive lock (以降 add -only)。 - -## v0.1.0 build envelope + build-time-eval semantics (amendment 1) - -D4 fixes the *full* trailer layout (bootstrap cache + user bytecode -+ Tier 0 metadata + build-id + size + magic). At the v0.1.0 -`cljw build` landing (D-100(b), row 14.11(b)) the **minimal correct -subset** ships first; the remaining D4 blocks are deferred per D6. -This amendment fixes the two things D4 left unspecified — the -**payload envelope framing** and the **build-time evaluation -semantics** — and records the deferral of the additive blocks so -D3's "Tier 0 metadata 常時 ON" is not silently dropped. - -### A1-D1: payload = sequence of chunks (not a single `(do …)` chunk) - -The compiler produces **one `BytecodeChunk` per top-level form** -(`vm_compiler.compile(rt, arena, node)`). A file is NOT wrapped in -a synthetic `(do …)` — `analyzeDo` (`special_forms.zig`) analyzes -all subforms in a single pass with no intervening eval, so an -in-file `(defmacro m …)` / `(require [x :as a])` followed by `(m)` -/ `(a/f)` would be analyzed before the macro / alias is registered -(broken). The build instead mirrors the runner's per-form loop and -emits a **sequence** of chunks: - -``` -[u32 n_chunks, LE] -n_chunks × ( [u32 chunk_len, LE] [chunk_bytes (serializeChunk v2)] ) -``` - -Per-chunk length framing is required because the v2 chunk format -is not self-terminating for back-to-back concatenation. This -**payload envelope** is archived in `cljw-formats/0.1.0.edn` as a -`:payload-envelope` entry **distinct** from the `:chunk-format` v2 -entry, so framing and chunk semantics pin independently (matches -D6 / D11 decoder-permanence intent). - -### A1-D2: build-time eval of top-level forms (Clojure AOT semantics) - -The build loop is the runner's loop (`src/app/runner.zig` -L84-103) with a chunk sink: per top-level form -`read → analyzeForm → vm_compiler.compile (→ append chunk) → -driver.evalForm`. The **eval step is mandatory** — it registers -macros / requires / defs into `env` + `macro_table` so the next -form analyzes against an evolved environment. This matches Clojure -AOT (`compile` evaluates top-level forms as it compiles them). - -**Consequence (documented, not a defect):** a top-level -side-effecting form (e.g. `(println "hi")`) runs **during the -build**, not only at run time — identical to Clojure AOT. This is -surfaced in `cljw build` help text. The alternative (a classifier -that evals only "env-shaping" forms and skips side-effecting ones) -is undecidable in general and Clojure-divergent — see Alternatives -amendment-1 Alt C. A build-time form that *throws* aborts the -artifact with a non-zero exit + EDN error event (D11); it never -embeds a half-payload. - -**F-009:** the per-form compile-then-eval loop is the same code -`runner.zig` runs; it is factored into a neutral helper that both -`runner.zig` (run mode, discards chunks) and `src/app/builder.zig` -(build mode, collects chunks + aborts-on-throw) call, so run-mode -and build-mode cannot drift. - -### A1-D3: deferred D4 blocks at v0.1.0 (tracked, not dropped) - -The v0.1.0 artifact ships `[runtime binary][payload envelope][u64 -payload_len][magic]` only. Deferred per D6 (format is not -ABI-committed; a new `cljw build` regenerates, and a future -archive entry decodes the richer format): - -- **bootstrap cache block** — the built binary re-runs - `bootstrap.loadCore` at startup (same `@embedFile`'d core - sources); the cold-start <12 ms target (D-100(d)) was already met - without a serialized cache (ebf2979b). -- **Tier 0 metadata block** (D3 常時 ON) — deferred; stack-trace - symbolication for built binaries lands post-v0.1.0. -- **build-id block** (D5) — deferred. - -Tracked by **D-131** so D3's always-on contract has an explicit -recall owner rather than a silent gap. - -### A1-D4: trailer magic - -Trailer magic = **`"CLJC"`** (D4 literal), kept distinct from the -serializer's internal chunk magic `"CLJW"` (`serialize.zig` L54), -so the tail-detection key never aliases the payload head. - -## Alternatives considered - -Devil's-advocate fork (general-purpose subagent、 fresh context、 -2026-05-25 issuance、 F-001/F-004/F-005/F-006/F-009 envelope 内) -output verbatim: - -#### Alt 1: cw v0 Phase 32 形態踏襲 (smallest-diff) - -`cljw build` の出力を **source bundle default + `--bytecode` flag opt-in** とし、 Tier 0 metadata table・structured EDN error event・`cljw render-error` post-mortem tool のいずれも導入しない。 error 出力は既存の babashka-style human renderer のみ。 trailer 形式は cw v0 Phase 32.2-32.3 をそのまま継承し、 build-id は git SHA 単独 (map 化なし)。 - -- **利点**: 実装差分が最小 (v0 → v1 移行 cost が低い)、 source bundle により bundled artifact が `cljw -e` で即 inspectable、 schema 議論を Phase 後送りにできる。 -- **欠点**: production 配布で source 同梱が IP 漏洩 vector になる、 runtime error の machine-parseable 出力が存在しないため CI / log aggregation 統合に都度 ad-hoc parser が必要、 Tier 0 metadata 不在で stack trace の file/line/col 情報が空白、 post-mortem tool 不在で format version drift 発生時に過去 artifact を decode できない。 -- **F-NNN 関係**: F-001 / F-004 / F-005 / F-006 とは独立。 F-009 (feature-implementation neutrality) には準拠 — 既存形態の延命であって新規 feature 縛りなし。 -- **F-NNN violation**: なし。 - -#### Alt 2: ADR-0034 本提案 (D1-D11) (finished-form-clean) - -`cljw build` を flag ゼロの単一 mode に固定し、 bytecode embed default + Tier 0 metadata 常時 ON + Deno-style trailer + map 形式 build-id + stream-separated error output (TTY=human / pipe=EDN one-line) + `cljw render-error` post-mortem decoder archive を採用。 format version policy は ABI commitment 放棄 (self-contained binary により mismatch が構造的に発生しない) で、 互換性責任を decoder 側に集中。 - -- **利点**: distribution artifact が 1 binary に確定し flag 選択の cognitive cost ゼロ、 EDN event schema により CI / log pipeline が初日から structured parse 可能、 Tier 0 metadata の常時 ON 化で stack trace の file/line/col が空白にならず error UX が JVM Clojure 水準に到達、 post-mortem decoder archive により過去 build-id を持つ artifact を future cljw で reproducibly decode 可能、 build-id map (`:git :cljw :built`) で release 追跡が EDN 解析だけで完結。 -- **欠点**: Tier 0 metadata の size budget (5-10%) を恒久的に支払う、 EDN schema が `:phase :category :message :file :line :col :var :ex-data :trace :build-id :timestamp` の 11 key 固定で初期 design の拘束が強い (key 追加は decoder archive bump を要する)、 decoder archive (`cljw-formats/.edn`) を v0.1.0 release で lock する commitment が発生し、 archive の forward maintenance が future Phase の永続 cost、 stream-separated output は TTY 判定 (`isatty(2)`) に依存し redirect 経由の subtle pipeline で human 形式が EDN consumer に混入する事故 surface が残る。 -- **F-NNN 関係**: F-001 (zwasm v2 library import) — cljw user code path と zwasm wasm execution path の分離を破らない。 F-004 (NaN-box 64 slot) — metadata table は heap interned で NaN-box slot を消費しない。 F-005 (数値タワー JVM 表面互換) — error event `:ex-data` に numeric 値が乗っても tower semantics に影響なし。 F-006 (mark-sweep + 3-layer allocator) — Tier 0 table は permanent allocator tier に置くことで GC root に依存しない。 F-009 — feature 列挙ではなく単一 build mode 固定なので neutrality に準拠。 -- **F-NNN violation**: なし。 - -#### Alt 3: AOT native + JIT 完全廃止 (wildcard) - -`cljw build` を **WebAssembly AOT compile → ELF/Mach-O native binary 生成 ahead-of-time** に置き換え、 runtime の wasm interpreter / JIT path を artifact からは完全に剥がす (wasmtime-style AOT pre-compilation を build-time に強制)。 bytecode embed は不要となり、 Tier 0 metadata は DWARF debug section に流し込む。 EDN error event は標準入出力に streaming で出すが、 decoder archive は DWARF version dispatch に統合する。 - -- **利点**: artifact 起動 latency が interpreter loop ゼロで cold-start near-zero、 OS-native debugger (lldb / gdb) が DWARF 経由でそのまま stack trace を解釈し外部 tooling 統合が無料、 trailer format の独自 versioning を捨てて ELF/Mach-O の既存規格に乗れる、 distribution size が trailer-less で純粋 binary。 -- **欠点**: cljw user code の eval / REPL / dynamic var 再定義 が AOT 前提と root から衝突 (Clojure の dynamic recompilation surface を放棄することになる)、 DWARF への metadata mapping は platform-specific (Mac/Linux/Windows) で host abstraction の負担が爆発、 wasm 実行 path を build artifact から剥がすと cljw user code と zwasm wasm execution の path 分離が崩れる。 -- **F-NNN 関係**: **F-001 と直接衝突** — cljw が zwasm v2 を library として import し cljw user code path と wasm execution path を分離する前提が、 wasm AOT を cljw 側の build pipeline に組み込むことで融合してしまう。 F-004 / F-005 / F-006 とは独立に解決可能。 F-009 は build mode 単一固定の点で本提案と同等。 -- **F-NNN violation**: **あり (F-001)**。 zwasm v2 library import 前提を維持しながら AOT native binary を出すことは、 Clojure dynamic eval surface (F-009 の neutrality に含まれる REPL/eval 機能) を犠牲にせず実現する path が現時点で見えない。 finished-form-clean を志向する場合でも、 F-001 の amendment は user action であり、 主 loop は本 wildcard を採用すべきではない (= 記録のみ、 採用候補から除外)。 - -### Amendment-1 Devil's-advocate fork (2026-05-29, D-100(b) payload format + build semantics, F-001/F-002/F-009 envelope) — verbatim - -**F-NNN gate check (leading entry):** None of the three shapes below requires violating F-001, F-002, or F-009. F-009 note: the build loop's *semantics* (read→analyze→compile→eval per top-level form) are the runner's existing pipeline (`src/app/runner.zig` L84-103) already living in neutral `src/eval/` + `src/lang/`. The builder (`src/app/`) is a Layer-3 driver that *orchestrates* those neutral pieces and appends the trailer; no impl logic needs to move. One caveat surfaced (see Alt B (c)): the per-form-eval loop is the same loop `runner.zig` runs, so the finished form should **extract that loop into a neutral helper** rather than have `builder.zig` re-implement it — that is the only F-009-relevant decision here, and it is satisfiable. - -#### Alt A — Smallest-diff: single-chunk `(do …)` wrap, ungated landing - -(a) **Concrete shape.** `builder.zig` reads the file, wraps all top-level forms in one synthetic `(do f1 f2 … fn)` Form, calls `analyzeForm` once → `analyzeDo` produces one `DoNode`, `vm_compiler.compile` produces **one** `BytecodeChunk`. Payload = `serializeChunk(chunk)` bytes verbatim (existing v2 format, no new framing). Trailer `[u64 payload_len]["CLJC"]` appended to the cljw binary. Startup: read tail, `deserializeChunk`, VM-run the single chunk. Landing: ungated via ADR-0015 amendment 5. - -(b) **Better than the others.** Zero new payload framing — reuses the existing single-chunk serializer untouched, so the `cljw-formats/0.1.0.edn` archive (D-100(e)) records exactly the v2 chunk format already serialized, no count-header byte to archive. Smallest startup path (one deserialize, one run). - -(c) **Breaks / risks.** **This is the established-broken shape:** `analyzeDo` analyzes all subforms in a single pass with no intervening eval, so an in-file `(defmacro m …)` followed by `(m)`, or `(require [x :as a])` followed by `(a/f)`, fails at analyze time. Diverges from Clojure AOT. It is not a smaller path to the *same* finished form — it is a path to a **different, broken** finished form, so under F-002 the smallest-diff tie-breaker does not apply. Not recommendable except as the throwaway red-test baseline that motivates Alt B. - -#### Alt B — Finished-form-clean: sequence-of-chunks + per-form compile-then-eval, ungated landing - -(a) **Concrete shape.** Build loop mirrors `runner.zig` L84-103, extracted into a neutral helper both `builder.zig` and `runner.zig` share (F-009): per top-level form `reader.read → analyzeForm → vm_compiler.compile → append chunk → driver.evalForm` (env evolves: macros/requires/defs register for form N+1). Payload `[u32 n_chunks]` then `n_chunks × [u32 chunk_len][chunk_bytes]`. Per-chunk length framing required (v2 chunk has no self-terminating end marker). Startup: read trailer → slice payload → read `n_chunks` → loop deserialize+VM-run each in order. Matches Clojure AOT. Landing ungated, ADR-0015 amendment 5. - -(b) **Better than the others.** Only shape *correct* for in-file macros + require-then-use, the realistic `cljw build app.clj` corpus (D-121 just landed Java-static dispatch precisely so build-able corpora can reference statics). Sharing the compile-unit loop with `runner.zig` removes run-vs-build drift. Per-chunk framing also forward-useful for D-103 (per-chunk peephole-version concerns localize). - -(c) **Breaks / risks.** (1) **Build-time side effects:** a top-level `(println "hi")` runs during the build — exactly Clojure AOT (`*compile-files*` runs top-level forms incl. side effects). Acceptable + expected, but MUST be documented in ADR Consequences + `cljw build` help. (2) **Archive interaction:** the `n_chunks` + per-chunk-`len` framing is a new outer envelope around the v2 chunk format; archive it as a **separate** `:payload-envelope` entry distinct from the `:chunk-format` v2 entry, so envelope framing and chunk semantics pin independently per ADR-0034 decoder-permanence (option ii-1, recommended). (3) build-time eval can *throw*; the builder must surface a build error (non-zero exit, EDN event per D11), not embed a half-payload. - -#### Alt C — Wildcard: analyze-only + separate macro/require registration pass (no build-time side effects) - -(a) **Concrete shape.** Two-pass build with no eval of arbitrary top-level forms. Pass 1 *selectively* evaluates only env-shaping forms (`defmacro`, `require`/`use`/`import`, macro-referenced `def`), skipping side-effecting forms; Pass 2 re-analyzes + compiles every form against the populated env. Side effects run only at run time. Framing identical to Alt B. - -(b) **Better than the others.** The only shape that resolves in-file macros/requires **without** build-time side effects. Cleaner mental model for "build = compile, run = execute". - -(c) **Breaks / risks — this is the trap.** Classifying "env-shaping vs side-effecting" is **undecidable in general** and diverges from Clojure. `(def x (compute-config))` where a downstream `(defmacro m [] @x)` depends on the side-effecting def — Clojure runs *everything* at compile time top-to-bottom; macros can depend on prior side effects. Alt C must either approximate the classifier and silently mis-handle the dependent-def case (a permanent-no-op / silent-semantics-drop smell — successful build, macro expanded against stale `x`), or re-introduce full eval and collapse into Alt B. It reaches a **Clojure-divergent** finished form the owner would unwind. Under F-002 this disqualifies it. Build-time side effects (Alt B's "cost") are not a defect to engineer away — they are the *correct Clojure AOT semantics*. - -#### Recommendation (non-binding) - -**Alt B.** Only shape reaching the correct Clojure-AOT finished form for the realistic corpus. Alt A reaches a *broken* finished form; Alt C reaches a *Clojure-divergent* one by suppressing side effects that are actually correct. Per F-002 the smallest-diff tie-breaker (Alt A's only virtue) does not apply — candidates reach **different** finished forms, the correct one wins regardless of extra framing bytes + the shared-loop extraction. Specifics: (i) build-time eval is acceptable + *required*, document it; (ii) `[u32 n_chunks]` + per-chunk `[u32 chunk_len]` framing, archived as a separate `:payload-envelope` entry; (iii) ungated landing risk low — `build` has no `phase_at_least_14` dependency, mirror ADR-0015 amendment 3 with amendment 5. **Shared-loop extraction (F-009) is mandatory, not optional** — the per-form compile-then-eval loop must be the same code `runner.zig` runs, in a neutral helper both call; re-implementing it in `builder.zig` invites run-vs-build drift. - -## fn_val constant serialization (amendment 2) - -Amendment 1 chose the per-form compile-then-eval build loop + payload -envelope. Implementing `cljw build` (D-100(b) step 3b) surfaced a gap the -amendment-1 survey missed (Step 0.6 ~80% boundary): the bytecode -serializer (`serialize.zig`, "D-100(a) Discharged") **rejects `fn_val` -constants** with `UnsupportedValueTag`, but the VM compiler emits every -user function as an `fn_val` CONSTANT (`op_make_fn`'s operand indexes a -`fn_val` in the chunk constant pool, `vm/compiler.zig:236-242`). So as -landed, `cljw build` could not serialize ANY program containing -`(defn …)` / `(fn* …)` / `(defmacro …)` — i.e. every realistic program. -A v0.1.0 build command that fails on `(defn -main [] …)` is a hollow -deliverable. Per F-002 (finished form wins; cycle/diff size is not a -constraint) and the user's standing directive ("最終的にうつくしいものを -完成させる … 根本的に計画して対処"), this amendment closes the gap rather -than scoping around it. - -### A2-D1: v0.1.0 `cljw build` serializes function constants - -`writeValue` / `readValue` gain an `fn_val` arm. A function constant is -serialized by its CONTENTS (F-004 — never the heap pointer): `slot_base` -+ each method's `arity` / `has_rest` + the method's bytecode body, which -is itself a `BytecodeChunk` serialized **recursively** via the existing -`serializeChunk` (the recursion is the natural shape — a method body is a -chunk). A nested `fn*` inside a method body is a further `fn_val` constant -in that sub-chunk, handled by the same recursion to arbitrary depth. - -### A2-D2: closure capture is the only unserializable case (invariant guard) - -A `fn_val` that appears as a compile-time CONSTANT always has -`closure_bindings == null`: a runtime closure (bindings filled from live -locals) is created by `op_make_fn` at call time, never embedded as a -constant. A `slot_base > 0` constant is a *template* (`closure_bindings -== null`); it serializes fine and `op_make_fn` fills its bindings at -runtime exactly as for a freshly-compiled template. Therefore the serializer -raises an explicit error ONLY when `closure_bindings != null` — a true -invariant guard (a corrupt/impossible constant), not a feature gate. No -silent-drop (`no_op_stub_forbidden` satisfied). - -### A2-D3: deserialized-function lifetime + reconstruction - -- The reconstructed `Function` is `rt.gpa`-allocated + `trackHeap`'d + - `closure_bindings = null` — identical to a compiled top-level fn, so - `freeFunction` is **unchanged** (frees the methods slice + struct). -- The method bytecode sub-chunks are owned by the **same `allocator`** - that owns the parent chunk; `freeChunk` recurses into `fn_val` - constants and frees their sub-chunks. The recursion reads - `fn_val.methods[i].bytecode` (the gpa methods slice), which is valid as - long as `freeChunk`/`freeEnvelope` runs **before** `rt.deinit` - (freeFunction frees the methods slice). The run sequence + defer-LIFO - satisfy this ordering naturally; it is documented at the `freeChunk` - recursion site. -- `FunctionMethod.body` (a `*const Node`) is only read by the tree_walk - backend; the VM path uses `bytecode`. A deserialized fn has no source - Node, so its methods point at a shared **immortal sentinel** Node; a - pointer-equality guard at the tree_walk fn-body site raises a clear - internal error if it is ever reached (it is not — deserialized fns are - VM-only, `bytecode != null`). -- Method **param-name** strings are NOT serialized (reconstructed as an - empty slice). Param names are debug-only (error-frame labels); - dispatch uses the `arity` field. AOT-built functions therefore show no - param-name labels in error frames — a transparent minor fidelity gap, - not a semantics drop, tracked by **D-139** for a later cycle. - -### A2-D4: archive interaction (D-100(e)) - -`cljw-formats/0.1.0.edn`'s `:chunk-format` v2 entry gains the `fn_val` -(0x0F) ValueTag wire shape. Because a method body is a nested chunk, the -archive notes the recursion (a chunk's constant pool may contain a -`fn_val` whose methods are themselves chunks). No new top-level format -entry — `fn_val` is a constant kind within the existing v2 chunk format. - -### Amendment-2 Devil's-advocate fork (2026-05-29, fn_val serialization scope, F-002/F-004/F-009/F-010 envelope) — verbatim - -## Alternatives considered - -### Alt A — smallest-diff: keep rejecting `fn_val`, document the floor - -Leave `serialize.zig`'s `UnsupportedValueTag` rejection on `fn_val` exactly as-is; document `cljw build` in v0.1.0 as "programs whose top-level reachable constant pool contains no user `fn` constants" and file a debt row (D-NNN) carrying full fn serialization to the post-v0.1.0 quality/coverage loop (F-010's second half). What it does better: zero new serialization surface, zero deserialization-lifetime risk, smallest archive format, and the explicit error means no permanent-no-op violation (the user sees `UnsupportedValueTag`, not a silently broken artifact). What it breaks: `(defn …)` / `(fn* …)` / `(defmacro …)` are the first thing any non-toy `.clj` contains, so `cljw build` can only build arithmetic/`def`-of-literal scripts — a v0.1.0 deliverable that fails on `(defn -main [] …)` is a hollow feature in all but name. The explicit error keeps it honest, but "honest about not working" is not the same as "works"; D-100(b) was scoped as a real build command, and this reduces it to a demo. - -### Alt B — finished-form-clean: full `fn_val` serialization - -Serialize `fn_val` constants fully: write `header`, `slot_base`, the `methods[]` array (each method's `arity` / `has_rest` / `params[][]const u8` strings + recursive `serializeChunk(method.bytecode.?)`), and the `variadic` method if present; on deserialize, reconstruct a `Function` with `closure_bindings = null`, a sentinel/placeholder `body` Node (the VM path reads only `bytecode`, never `body` — fact (4)), arena-or-GC-owned `methods` slices, and recursively-deserialized sub-chunks tracked for `freeChunk` recursion + GC marking. Raise an explicit error ONLY on the genuinely-impossible `closure_bindings != null` case (fact (1): a compile-time constant can never carry runtime closure bindings, so this branch is a corruption/invariant guard, not a feature gate). What it does better: `cljw build` builds realistic programs (the whole point of D-100(b)); the recursion is the natural shape since `method.bytecode` is itself a `BytecodeChunk` already serializable by the existing `serializeChunk`; it stays in-zone (Function lives in `eval/backend/tree_walk.zig`, same Layer 1 — F-009 OK) and serializes contents not pointers (F-004 OK). What it breaks/risks: (1) lifetime — deserialized sub-chunks and `methods`/`params` slices need a clear owner (deserialize-arena vs GC heap) or they leak / dangle when the embedded payload's top chunk is freed; (2) the sentinel `body` Node must be a shared immortal singleton, never freed, never walked, or a future tree_walk-on-deserialized-fn path would segfault — needs a guard/assert; (3) `defmacro` carries macro metadata (the `:macro` flag lives on the Var, not the Function, so verify the build path captures it where defmacro is recorded — if macro-ness rides on Function it must serialize too); (4) `slot_base > 0` nested templates serialize fine as constants (bindings filled at runtime by `op_make_fn`), but round-trip tests must cover the nested-template case explicitly. - -### Alt C — wildcard: embed source text, re-analyze+compile at startup - -Don't serialize bytecode at all. Embed the original `.clj` source text (or the read forms) in the trailer and re-run the existing read → analyze → compile pipeline at process startup, producing fresh chunks (and thus fresh `fn_val` constants) in-process — sidestepping fn serialization entirely. Tradeoff: it does better on format simplicity (the trailer is just bytes of source, no recursive chunk schema, no Function reconstruction, no lifetime/sentinel-body problem, and it can never silently drop semantics because it runs the identical compile path) and it trivially handles `defmacro`/closures/everything the live runtime handles. But it breaks the stated deliverable shape ("serialized bytecode payload", D-100(a) Discharged) — startup now pays full analyze+compile cost on every run (defeating the precompile rationale), the binary must ship the entire compiler reachable at startup (already true, but now load-bearing for `cljw build`), and it makes the "compiled artifact" indistinguishable from `cljw run app.clj` plus a trailer, i.e. it quietly demotes D-100(a)'s serializer to dead code on the build path. A middle variant (a fn-proto table à la cw v0: hoist all `fn_val`s into a flat indexed table, store index references inline, deserialize the table first then patch references) avoids inline recursion and dedups shared protos, but adds a two-pass serializer + reference-fixup pass whose complexity exceeds Alt B's straightforward recursion for no finished-form benefit at v0.1.0 scale. - -### Recommendation - -Per F-002 (finished-form cleanliness wins; cycle/diff/LOC is not a constraint), **Alt B**. It is the only option that makes `cljw build` actually build the programs the deliverable promises, it reuses the already-recursive `serializeChunk` shape rather than inventing a parallel format, and the one explicit error it keeps (`closure_bindings != null`) is a true invariant guard rather than a feature gate — so it satisfies permanent-no-op-forbidden honestly. Alt A is the Cycle-budget-defer smell wearing a debt row; Alt C's source-re-compile path is genuinely clean but contradicts D-100(a)'s already-discharged bytecode-payload shape and demotes the serializer to dead code, so it would require re-opening D-100(a) rather than completing D-100(b). The main loop is not bound by this recommendation, but choosing Alt A or C on size grounds would itself be the forbidden defer smell. (Note: whichever is chosen, the lifetime/owner decision for deserialized sub-chunks and the immortal-sentinel-`body` guard are the two load-bearing details Alt B must nail in the same cycle.) - -### Amendment-2 decision - -**Alt B selected** (matches the DA recommendation). Divergence from the DA's -Alt B sketch: param-name strings are NOT serialized at v0.1.0 (A2-D3 — -reconstructed empty, tracked by D-139), because the param-string lifetime -would otherwise force a `freeFunction` ownership flag for no -dispatch-correctness benefit (param names are debug-only). The sub-chunk -lifetime is resolved via `freeChunk` recursion with documented -freeEnvelope-before-rt.deinit ordering (A2-D3), not an arena, keeping the -existing allocator-explicit deserialize contract. The DA's concern (3) -(defmacro macro metadata) is a non-issue: macro-ness is Var-level and only -consulted at analyze time, which has already happened at build time; a -deserialized macro fn is harmless dead weight at runtime. - -## require-closure embedding (amendment 3) - -Amendments 1-2 made `cljw build app.clj -o app` compile + embed the -**entry file's** top-level forms (sequence-of-chunks + per-form -compile-then-eval + fn_val serialization). Building the bookshelf -multi-file demo (D-356) surfaced two gaps the prior amendments left -open — both required for a multi-file app (`(require '[lib])` over a -classpath) to ship as a self-contained binary: - -1. **Classpath prerequisite (Part 1).** `buildFile` never set - `rt.load_paths` nor installed the filesystem require resolver - (`setupCore` installs the embedded-ONLY resolver), so a *build-time* - `(require '[lib])` raised `lib_not_found` — the build could not even - resolve a user lib. -2. **Require-closure embedding (Part 2, the actual feature).** Even - after Part 1 lets the build resolve the lib, the compiled - `(require '[lib])` chunk re-runs at run time under `tryRunEmbedded` - (`setupCoreAot` + embedded-only resolver, NO filesystem/classpath), - so `op_require` → `lib_not_found` (vm.zig): the required ns is not in - the payload. The binary is not self-contained. - -### A3-D1: the op_require-idempotency enabler - -`op_require` is idempotent (vm.zig:713-716: `already_loaded = -env.findNs(ns) exists AND mappings.count() > 0` → SKIPS the resolver). -So if every filesystem-resolved user ns the app depends on has its -defining chunks run BEFORE the entry's `(require …)` chunk at run time, -the entry require sees the ns loaded → skips the (absent) resolver → -the app runs with NO run-time filesystem/classpath. This is the run-time -mechanism that makes embedding sufficient; no resolver needs to exist in -the shipped binary. - -### A3-D2: capture mechanism = chunk-capture during the real load (Alt-2/Alt-B) - -The build collects the require-closure chunks **during the one real -build-time load**, not by a second recompile pass. A Layer-0 optional -callback `Runtime.build_chunk_sink` (type-erased — the `BytecodeChunk` -type lives in Layer 1, so the field cannot name it; the builder casts -back via `@ptrCast`) is installed by the builder (Layer 3) before the -entry eval. `loader.loadNamespace`'s per-form loop (Layer 1), when the -sink is set AND the source is filesystem-resolved, ALSO calls -`vm_compiler.compile` on each analyzed form (a same-Layer-1 import, no -zone inversion) and feeds the chunk to the sink. Because capture rides -the genuine load: - -- `current_ns` / alias / macro state is correct **by construction** - (the same eval the un-instrumented load performs). -- The closure is eval'd **exactly once** (no double-eval; build - semantics == run-once == Clojure AOT, even for build-time-effecting - libs — the disqualifier of the source-recompile alternative). -- Order is **post-order** = correct replay order: `loadNamespace` - recurses through nested requires during eval and the sink fires as - each ns's forms compile; a dep's forms complete (and fire) before its - dependent's. No toposort. Diamond deps load once (loadNamespace - early-returns on `loaded_libs.contains`), so the sink fires once per - ns — no explicit chunk-level dedup needed. - -The production load path is unaffected: `build_chunk_sink` is null at -run time, so the per-form `if (rt.build_chunk_sink) |sink|` guard skips -the compile. - -### A3-D3: bootstrap-ns exclusion via `ResolvedSource.from_filesystem` (F-013-clean) - -At run time `setupCoreAot` restores all bootstrap nses -(clojure.core/set/string/test/…), so a user lib's -`(:require [clojure.string])` chunk replays → `op_require` already_loaded -→ skip. Only **filesystem-resolved user nses** need embedding. The clean -discriminator (avoiding a hardcoded bootstrap-name allowlist — the -F-013 ad-hoc-allowlist smell): a `from_filesystem: bool` field on -`ResolvedSource`, set `true` by `filesystemResolver` and `false` by -`embeddedResolver` (chainedResolver passes the inner result through). -The sink captures only when `resolved.from_filesystem`. - -### A3-D4: classpath prerequisite (Part 1) — build path mirrors the run path - -`buildFile` gains a `load_paths: []const []const u8` param; it sets -`rt.load_paths = load_paths` + `require_resolver.installChained(&rt)` -**AFTER `setupCore`** (setupCore installs the embedded-only resolver at -bootstrap, so an earlier installChained would be overwritten). The -`cljw build` CLI dispatch branch parses ` -o [-cp ] -[-A:alias…]` and resolves `load_paths` **identically to the run path** -(`splitClasspath` + `loadDepsEdn`, the same neutral helpers -`dispatchArgsRest` uses), so build-mode and run-mode classpath -resolution cannot drift (F-009). - -### A3-D5: payload layout - -`buildEnvelope` returns `serializeEnvelope([closure chunks in post-order] -++ [entry chunks])`. The closure chunks accumulate (via the sink) during -the entry forms' eval; the combined list is serialized once. The -existing per-chunk framing (A1-D1) + fn_val serialization (A2) carry the -closure chunks unchanged (a lib's `(defn …)` is an fn_val constant). - -### Amendment-3 Devil's-advocate fork (2026-06-09, capture-mechanism scope, F-002/F-009/F-011/F-013 envelope) — verbatim - -All claims verified against source. `ResolvedSource` is `{ source, label }` (line 49-52), `require_resolver` is a single `?RequireResolverFn` field, `loaded_libs` is gpa-keyed. The post-order claim holds: `loadNamespace` marks `loaded_libs` at line 70 (END), recurses through nested requires during eval (line 64). Here is my Devil's-advocate analysis for the ADR. - -## Alternatives considered (Devil's-advocate fork, fresh context) - -Fork brief: 3 capture-mechanism shapes for collecting the require-closure chunks in correct replay order (bootstrap nses excluded), within the F-NNN envelope. F-002 (finished-form wins, diff size is NOT a constraint), F-009 (impl in neutral homes, builder is a thin driver), F-011 (shared mechanism over duplicated), F-013 (no per-library allowlist), zone deps (Layer-0 callback installed by higher layer = canonical vtable). - -### Leading note — no F-NNN block exists - -None of the three shapes below requires violating an F-NNN. The finished-form-clean option (Alt-2 / matches Alt-B) is fully F-compliant, so there is no "the only clean option breaks an invariant" finding to surface. The discriminator question (how to exclude bootstrap nses) has a clean F-013-compliant answer in all three (`from_filesystem: bool` on `ResolvedSource`, set by `filesystemResolver` vs `embeddedResolver`) — none needs a hardcoded bootstrap-name allowlist. - -### Alt-1 — smallest-diff: source-capture + recompile (== draft Alt-A) - -**(a) Shape.** Add a Layer-0 optional callback field `rt.require_sink: ?*const fn(*Runtime, ns_name, source, from_filesystem) void` on `Runtime`. The builder (Layer 3) installs it before the entry eval. At the tail of `loadNamespace` (loader.zig, after line 70, post-order), if the sink is set, invoke it with `(ns_name, resolved.source, resolved.from_filesystem)`. After the entry `buildEnvelope` finishes, the builder filters captured records to `from_filesystem == true`, runs `buildEnvelope`'s per-form loop again on each captured source (post-order = correct replay order, no toposort), and prepends the resulting chunks before the entry chunks. - -**(b) Better than the others.** The Layer-0 sink touches only `loadNamespace`'s tail — `eval/` stays ignorant of the vm compiler (the sink takes raw source bytes, not chunks), so the load path's zone surface does not widen toward Layer-1's compiler. The captured payload is just `[]const u8` source, trivially serializable/inspectable, and the recompile loop reuses the *existing* `buildEnvelope` verbatim (F-011: one compile-then-eval mechanism, no second copy). - -**(c) Breaks / risks.** Double-eval. Each captured lib's forms eval once during the real load AND once during the recompile pass. For pure-`def` libs this is idempotent; for a lib with a top-level side effect (`(println "loading")`, `(defonce …)` that isn't, an `(atom)` registered in a global registry) the effect fires twice — a *silently wrong* build for exactly the build-time-effecting programs the project already documents as a sharp edge. This is a permanent semantic gap, not a transient one: it cannot be tracked away with a PROVISIONAL marker because there is no upstream feature whose landing closes it; it is intrinsic to "re-run to recompile." Per F-002 this is the disqualifier — the finished form does not double-eval. - -### Alt-2 — finished-form-clean: chunk-capture during the real load (== draft Alt-B). RECOMMENDED. - -**(a) Shape.** Add a Layer-0 optional callback `rt.build_chunk_sink: ?*const fn(*Runtime, ns_name, *const BytecodeChunk, from_filesystem) anyerror!void` on `Runtime`. `loadNamespace`'s per-form loop (loader.zig line 61-65), when the sink is set, ALSO calls `vm_compiler.compile(rt, arena, node)` on each analyzed form and feeds the chunk to the sink — captured during the one real build-time load. The builder installs the sink, accumulates chunks keyed by post-order ns completion, filters to `from_filesystem`, and prepends them ahead of the entry chunks. `current_ns` is naturally correct because capture rides the genuine load where the lib's `(ns …)` has already switched it. Exactly-once eval, post-order, no toposort. - -**(b) Better.** No double-eval — the single disqualifier of Alt-1 is gone. Capture is a byproduct of the load that must happen anyway, so build semantics == run-once semantics == what Clojure AOT does. `current_ns`/alias/macro state is correct by construction (it is the *same* eval that the un-instrumented load performs), eliminating the "recompile-pass current_ns reconstruction" surface Alt-1 carries. This is the shape the finished-form owner would not unwind. - -**(c) Breaks / risks.** Couples the shared `eval/` load path to the Layer-1 vm compiler behind a build-only optional hook. This is the stated concern, but it does not violate zone deps: `loader.zig` is Layer 1 and `vm_compiler` is also Layer 1 (`eval/backend/vm/compiler.zig`) — a same-layer import, already permitted, no inversion needed. The sink itself is the canonical Layer-0-callback-installed-by-Layer-3 vtable pattern (explicitly allowed). The real cost is that the production load path now carries a compile call it skips at runtime (sink null) — a branch in the hot require path. That is a negligible, well-marked cost (a single `if (rt.build_chunk_sink) |sink|` guard), and F-009 is satisfied because the *compile mechanism* (`vm_compiler.compile`) stays in its neutral Layer-1 home; the builder only installs a sink and orders the output — it re-implements nothing. The minor sharp edge: the sink fires for EVERY loaded ns including bootstrap ones, so the `from_filesystem` filter must run on the builder side (cheap, already required). - -### Alt-3 — wildcard: post-load env-walk replay (no capture hook at all) - -**(a) Shape.** Capture nothing during load. After the entry eval finishes, the builder walks `rt.loaded_libs` (or an insertion-ordered sibling) and, for each ns marked `from_filesystem`, re-resolves its source via the resolver (the resolver is idempotent — re-reading a `.clj` is cheap) and runs `buildEnvelope` over each to produce chunks. Order is recovered by recording `loaded_libs` insertion order (which IS post-order, since `loadNamespace` marks at the tail). To carry `from_filesystem`, widen `loaded_libs`'s value from `void` to a small struct, or keep a parallel insertion-ordered list. No hook on the load path at all. - -**(b) Better.** Zero coupling of the load path to the compiler — `loadNamespace` is untouched; the production require path carries no build-only branch. The "ordering" problem is solved by data the runtime already has (`loaded_libs` is already maintained), so the mechanism leans entirely on existing state. Conceptually the cleanest *separation*: build is purely a post-pass over runtime state, the load path stays single-purpose. - -**(c) Breaks / risks.** Same double-eval as Alt-1 (the recompile re-evals each lib's forms) — disqualifying for the same F-002 reason. Worse, it adds a *re-resolve* (a second filesystem read per lib) and depends on `loaded_libs` being insertion-ordered, which `StringHashMapUnmanaged` is NOT — recovering post-order requires a new parallel ordered structure or a value-struct widening, i.e. a runtime data-shape change to serve a build-only need (a Layer-0 concession to Layer 3 that is heavier than Alt-2's optional callback). It trades Alt-2's "compile branch in the load path" for "re-eval + re-read + ordered-map retrofit," which is a larger and semantically-worse footprint. The "no hook" cleanliness is illusory once the ordering retrofit is counted. - -### Recommendation (non-binding) - -**Alt-2 (chunk-capture during the real load; == draft Alt-B).** It is the only one of the three that eval-once-s the closure, so it is the only one whose build semantics match run-once / Clojure-AOT semantics for build-time-effecting libs — Alt-1 and Alt-3 both double-eval, an intrinsic (non-closable) semantic gap that F-002's finished-form bar rejects. Its sole concern (load-path↔compiler coupling) is a *same-layer* Layer-1 import gated behind a null-checked build-only sink installed via the sanctioned vtable pattern, satisfying zone deps and F-009 (the compile mechanism stays neutral; the builder only installs + orders). Per F-002 I do not downgrade to Alt-1 on the grounds that "source capture is a smaller diff and keeps eval/ ignorant of the compiler" — that is the Cycle-budget / smallest-diff defer; the finished form captures chunks where they are correct-by-construction. The `from_filesystem: bool` discriminator on `ResolvedSource` (set by `filesystemResolver` returning `true`, `embeddedResolver` returning `false`) is the F-013-clean bootstrap exclusion for whichever shape is chosen, and should land regardless. - -One implementation note for whichever shape lands: the sink must capture at the ns granularity's post-order completion, and the builder must dedupe — a diamond require (A→B, A→C, B→D, C→D) loads D once (idempotent `loaded_libs`), so the sink fires for D once; the post-order tail-capture already gives D before B and C, so no toposort, confirmed against loader.zig line 36 (early-return on `loaded_libs.contains`) + line 70 (tail mark). - -### Amendment-3 decision - -**Alt-2 selected** (matches the DA recommendation). The double-eval gap is -the decisive disqualifier for Alt-1/Alt-3 under F-002 (a build-time-effecting -lib would fire its effects twice — a silent semantic divergence with no -close-out, not a transient stub). The load-path↔compiler coupling is a -same-Layer-1 import behind a null-checked build-only sink (the sanctioned -Layer-0-callback / Layer-3-install vtable pattern), so zone deps + F-009 hold. -The build-time `(-main …)` invocation hazard the bookshelf demo's -`build_main.clj` would hit (a top-level server-start runs at build per the -A1-D2 Clojure-AOT semantics, hanging the build) is **out of scope for this -amendment** — it is a serverless-v2 (demo) concern for D-362's runway, not the -cljw require-closure feature; the e2e cases use define-and-print libs that are -harmless to double-print (build stdout vs run stdout are distinct streams; the -e2e asserts the run output). Tracked as a demo-side note in D-356. - -## `-m` main-entry mode (amendment 4) - -Amendment 3 made `cljw build` embed the require closure so a multi-file -`(require '[lib])` app builds self-contained. But `cljw build` evals top-level -forms at build (A1-D2 Clojure-AOT), so an APP whose top-level starts a server -(`(my.app/-main 8080)`) HANGS the build. The finished-form answer is the -Clojure / GraalVM / cljw-run-path convention: put runtime logic in `-main` and -name it as the artifact's ENTRY POINT, invoked at RUN, not build. - -### A4-D1: two build modes, mirroring the run path - -- **Script mode** (`cljw build app.clj -o out`, unchanged): the entry file's - top-level forms ARE the program; build compiles+evals them (Clojure-AOT, - build-time side effects run — the documented wart, A1-D2); run VM-executes the - embedded chunks top-to-bottom. No `-main` required. Keeps today's behaviour. -- **Main mode** (`cljw build -m [args…] -o out`, new): build does - `(require '[])` → captures the closure (am3 build_chunk_sink) + registers - ``'s defns; the `-main` is DEFINED but NOT called at build (no hang / no - side effect). The produced binary, at RUN, invokes `(/-main args)` as the - entry point. This is the production-deploy path; it preserves fast cold-start - (bytecode closure + setupCoreAot, no re-parse of the program). - -Main mode is a thin layer over am3: the `-main` ns is reached via `(require)`, -so the whole program is embedded by the am3 closure mechanism; main mode only -adds (a) "don't call -main at build" and (b) "call -main at run". - -The undecidable env-shaping-vs-side-effecting classifier is NOT attempted (Alt C -of am1 stays rejected); cljw is Clojure-AOT-faithful (build = load). "Don't run -at build" is achieved the Clojure way — put logic in `-main`, use `-m`. The cw -v0 `http_server.build_mode=true` per-primitive side-effect suppression is -REJECTED (F-013 ad-hoc-allowlist smell). - -### A4-D2: entry point is artifact metadata (entry manifest), not a code chunk - -Every real executable format stores the entry point as HEADER metadata (ELF -`e_entry`, Mach-O `LC_MAIN`, jar `Main-Class`) — not as "code that calls main". -The payload envelope gains an optional **entry manifest** at its front: -`{ entry_ns: ?[]const u8, entry_args: []const []const u8 }`. Script mode → -`entry_ns = null` (run chunks, stop). Main mode → `entry_ns = `, -`entry_args` = the build-time `-m`/`:main-opts` args (possibly empty). One -payload schema covers both modes; the entry is inspectable artifact data and the -forward home for the deferred D4 metadata (build-id / Tier-0 — D-131). - -### A4-D3: run-side dispatch routes through the run-path `synthMainNs` (F-011) - -`tryRunEmbedded` runs the closure chunks (defines ``), then, if the manifest -names an entry: `all_args = entry_args ++ `; -`src = run_mode.synthMainNs(, all_args)` — the **same** helper -`cljw -M -m` uses (promoted to `pub`); read+analyze+eval it. So the built -binary's `-m` behaviour is byte-identical to `cljw -M -m` (F-011) — the -`requiring-resolve` guard (clean "no -main" error), the `*command-line-args*` -setter, the call — all shared, no parallel resolve+call to drift. `requiring- -resolve` at run is idempotent (the closure already loaded ``, op_require -skips), so no filesystem resolver is needed in the shipped binary. Runtime argv -gives `./out 8080` real-program semantics (8080 reaches `-main` at run); -build-time `entry_args` are baked defaults overridden by a non-empty runtime -argv. The startup parse of the one-line `synthMainNs` source is ~1-2% of cold -start (negligible; a `-m` binary is a long-running server). - -### A4-D4: deps.edn `:main-opts` drives the entry - -`cljw build -A:alias -o out` reads the alias's `:main-opts ["-m" args…]` -to derive the entry (mirrors `cljw -M:alias`), so the classpath (am3) AND the -entry both flow from deps.edn. An explicit `-m ` on the CLI wins over -`:main-opts`. - -### Amendment-4 Devil's-advocate fork (2026-06-09, -m entry-mode shape, F-002/F-009/F-011/F-013 envelope) — verbatim - -I have all I need. The build branch positionally consumes one bare `` (line 168 rejects a second bare token), takes `-o`/`-cp`/`-A`, and calls `buildFile(in, out, deps.load_paths)`. The serializer frames `serializeEnvelope(chunks)` — no manifest/header field for entry ns exists today. `tryRunEmbedded` just `runEnvelope`s the chunks with no post-run hook. - -## Alternatives considered - -I verified against `src/app/builder.zig` (`buildEnvelope`/`buildFile`/`tryRunEmbedded`), `src/app/deps/run_mode.zig` (`synthMainNs`/`writeClArgsSetter`), and `src/app/cli.zig` (build branch lines 133-198 + `dispatchArgsRest`). Confirmed: the payload is a flat `serializeEnvelope(chunks)` framing with **no header/manifest field**; `tryRunEmbedded` runs the chunks and has **no post-run resolve-and-call hook**; `synthMainNs` already emits the `*command-line-args*` setter + `requiring-resolve` guarded `-main` call; `op_require` idempotency + D-356 closure embedding are in place. The draft is mechanically buildable on what exists. - -### Leading finding: no alternative requires violating an F-NNN - -All three shapes below live inside the F-NNN envelope. The axis where they genuinely diverge is **where the "call -main at run, not at build" decision is encoded** — in a Clojure source chunk (Layer 2 surface, compiled like any form), in a bytecode opcode (Layer 1), or in payload metadata read by the Layer-3 driver. F-009/F-011 push hard toward the first; F-013 is the constraint that kills any "build-mode flag that suppresses -main" convenience. - -### Alt 1 — Smallest-diff: synthMainNs as a compile-only entry chunk (the draft) - -**(a) Concrete shape.** `cljw build -m [args…] -o out`. In `cli.zig`'s build branch, parse `-m ` + trailing main-args (mirror the `-A`/`-cp` parsing already there; `-m` consumes the rest as args, like the run path). `buildFile`/`buildEnvelope` gain an optional `entry: ?struct{ ns, args }`. When set: after the entry-file forms (or instead of an entry file — see the script-vs-main axis), eval `(require '[])` to trigger D-356 closure capture + register the defns, then **compile but do NOT eval** one extra chunk = `run_mode.synthMainNs(, args)`. Append that chunk last. At run, `runEnvelope` runs it like any other chunk → idempotent require resolves to already-loaded → `(-main args)`. `synthMainNs` is promoted from `fn` to `pub fn` in `run_mode.zig` (same Layer 3, same-layer reuse per F-009). Zero serializer change, zero `tryRunEmbedded` change. - -**(b) Better than the others.** Maximal reuse: the built binary's `-m` behaviour is **byte-identical** to `cljw -M -m` because it is literally the same synthesized source compiled through the same pipeline (F-011 behavioural equivalence is free, not engineered). No new opcode, no new payload field, no new run-path branch — the smallest possible new surface. The "don't call -main at build" property is expressed as *"this one chunk is compiled but not eval'd"*, which is a local, legible exception to `buildEnvelope`'s compile-then-eval loop. - -**(c) Risks / what it breaks.** The compile-then-eval loop in `buildEnvelope` (lines 89-95) is currently **uniform** — every form compiles then evals. Adding a single chunk that is compiled-but-not-eval'd introduces a special case into that loop's invariant. If done carelessly (an `if (is_entry_chunk) skip_eval`), it reads like the F-013 anti-pattern's cousin (a flag that changes per-form behaviour). The clean framing is to keep the entry-chunk synthesis *outside* the form loop entirely (synthesize + compile it after the loop, never feed it to `evalForm`), so the loop stays uniform and the entry chunk is just "an extra chunk we append, never evaluated at build by construction." Second risk: main-args are **build-baked** into the chunk's string literals (`synthMainNs(ns, args)` embeds them). That means `out 8080` bakes `8080`; the binary's own runtime argv is ignored. For an AOT artifact whose entry args come from deps.edn `:main-opts` this is arguably correct, but it diverges from `cljw -M -m my.app 8080` where args are runtime — see the args axis in the recommendation. - -### Alt 2 — Finished-form-clean: payload entry-manifest, driver resolves+calls at run - -**(a) Concrete shape.** Extend the payload framing so `serializeEnvelope` can carry an optional **entry manifest** in the trailer: `{ entry_ns: ?[]const u8, entry_args: []const []const u8 }` (a small typed header section ahead of the chunk stream, versioned). `buildEnvelope` at build does the `(require '[])` (closure capture + defn registration, no `-main` call) and records the manifest — it does **not** synthesize any entry chunk. `tryRunEmbedded` → `runEnvelope` runs all chunks, then, if the manifest names an entry ns, **calls `synthMainNs`-equivalent at run** (or directly `requiring-resolve` + invoke via the runtime's callFn vtable) using the manifest's ns + (manifest args ∪ the binary's own argv). Script mode = manifest with `entry_ns = null` (run chunks, stop). One payload schema covers both modes. - -**(b) Better than the others.** This is the **finished-form** answer to "how does main mode relate to script mode": they are one payload schema differing by one optional field, not two code paths. The entry intent is *data in the artifact* (inspectable: a future `cljw inspect out` can print "entry: my.app/-main"), not buried in a compiled string-literal chunk. Crucially it cleanly solves the **args axis**: the run-side resolve can splice the binary's *actual* argv into `*command-line-args*` and `-main`, so `./out 8080` works like a real program (Alt 1 can only bake build-time args). It keeps `buildEnvelope`'s compile-then-eval loop perfectly uniform (no special compiled-not-eval'd chunk). The manifest is also the natural home for future artifact metadata (build timestamp, cljw version, AOT cache id) — a forward-clean extension point. - -**(c) Risks / what it breaks.** Touches the serializer format (`serializeEnvelope`/`extractPayload`/`frameArtifact` + a version bump) and adds a run-path branch in `tryRunEmbedded` — the largest diff of the three. The behavioural-equivalence-to-`cljw -M -m` property (F-011) is no longer *free*: the run-side resolve+call is a **second implementation** of what `synthMainNs` does for the `-M` path, so the two can drift unless the manifest path *also* routes through `synthMainNs` (i.e. at run, build the synthMainNs source from the manifest and eval it — which recovers F-011 but means the manifest is just a deferred way of producing the same chunk Alt 1 produces at build). If you route through synthMainNs at run, ask what the manifest bought over Alt 1: the answer is *runtime args* + *uniform build loop* + *inspectability*, which are real, but the F-011 sharing must be deliberate, not assumed. - -### Alt 3 — Wildcard: unify script and main mode — every build is "main mode", script = synthetic -main - -**(a) Concrete shape.** Collapse the script/main distinction at the *model* level: `cljw build` always produces an artifact with one entry point. `-m ` sets entry = `/-main`. A bare `cljw build app.clj` (no `-m`) is sugar for "wrap the file's top-level forms as the body of a synthetic `(defn -main [& args] …)` in an anonymous ns, entry = that". Convention auto-detection: if `app.clj` itself defines `-main` and has no other top-level side-effecting forms, build treats it as main mode automatically (Clojure-CLI-ish convention). One mechanism: entry-invocation always goes through synthMainNs (or the manifest of Alt 2); script mode is the degenerate case where the synthetic -main *is* the top-level forms. - -**(b) Better than the others.** Conceptual unification — there is exactly one notion ("an artifact has an entry") and script mode stops being a separate thing. This is the most F-011-shaped *idea* (one mechanism, F-011 commonization outranks effort). Auto-detection gives the nicest UX (`cljw build app.clj` "just works" whether app.clj is a script or has a -main). - -**(c) Risks / what it breaks — and an F-013 hazard.** **Auto-detection is the F-013 trap in disguise.** "If the file defines -main and has no side-effecting top-levels, call it; otherwise run top-levels" is exactly the kind of *implicit, definition-derived-but-heuristic* behaviour switch that F-013 warns against — it is one short step from the rejected `build_mode=true` server-suppression: the build silently decides whether your `(start-server)` at top level runs or not based on a heuristic. It also breaks Clojure-AOT semantics (ADR-0034): top-level forms in a script *must* run at build (that is how form N+1 sees form N's macros/defs); you cannot retroactively decide they were "a -main body" that shouldn't have run at build. Wrapping top-levels in a synthetic `-main` would mean they DON'T run at build, defeating the compile-then-eval contract. So the unification is clean only if it is **explicit** (`-m` required to defer; bare build keeps today's run-at-build script semantics) — at which point the "unification" is just Alt 1 or Alt 2 with the script case relabelled, and the auto-detection (the only thing that made it a distinct wildcard) is dropped as an F-013 violation. Recorded here so the loop sees the auto-detect temptation and rejects it. - -### Non-binding recommendation - -**Alt 2 (entry-manifest), with the run-side resolve routed through `synthMainNs` to keep F-011 sharing explicit.** Citing F-002: it is the largest diff of the three, and that is not a reason to downgrade. The finished-form question the ADR is really answering is *"is the entry point an inspectable property of the artifact, or an opaque compiled chunk?"* — and the artifact owner (Phase 15-20 edge-native, where a deployed binary's metadata matters) wants it as data. Alt 2 also cleanly resolves the **args axis** in favour of runtime argv (`./out 8080` behaving like a real program), which Alt 1 structurally cannot, and keeps `buildEnvelope`'s compile-then-eval loop uniform (no compiled-not-eval'd special case). - -The one discipline Alt 2 must hold: the run-side entry invocation **must** build its source via the shared `synthMainNs` (not a parallel hand-rolled resolve+call), so the built binary's `-m` and `cljw -M -m` cannot drift (F-011). If the loop instead picks Alt 1 on the grounds that it is smaller and "F-011 is free," note that this is the **Cycle-budget defer smell**: Alt 1's smaller diff comes at the cost of build-baked-only args and a special-cased build loop, both of which the artifact owner would unwind later. The smell-free reasons to prefer Alt 1 would be (i) if runtime-argv entry is judged a non-goal for AOT artifacts, and (ii) if payload-format versioning is deemed premature before other trailer metadata is needed — both legitimate finished-form positions, but they are *design* arguments, not budget arguments, and should be stated as such in the ADR if Alt 1 is chosen. - -On the **script-vs-main-mode-share axis**: recommend they share one payload schema (Alt 2's optional-entry-field), explicitly **not** auto-detection (Alt 3's F-013 hazard) — `-m` is required to get deferred `-main`; bare `cljw build` keeps today's run-at-build script semantics per ADR-0034. - -### Amendment-4 decision - -**Alt 2 selected** (matches the DA recommendation), with the run-side routed -through the shared `run_mode.synthMainNs` (F-011 — the built `-m` binary and -`cljw -M -m` cannot drift; the DA's mandatory discipline). Rationale beyond the -DA: the runtime-argv property is achievable in Alt 2 *with* F-011 fidelity (run -reads the binary's argv and calls the SAME `synthMainNs(ns, argv)` the run path -calls), whereas Alt 1 can only get runtime argv via a divergent -`apply *command-line-args*` synthesis — so Alt 2 wins on the args axis without an -F-011 cost. The entry-as-artifact-metadata shape (manifest) mirrors every real -executable format (ELF/Mach-O/jar) and is the forward home for the deferred D4 -metadata (D-131). The startup parse of the one-line synth source is ~1-2% of -cold start (negligible for a long-running `-m` server). Alt 3's auto-detection is -rejected (F-013). The choice is on design grounds (finished-form entry encoding + -F-011-clean runtime args), NOT diff size. - -## type_descriptor class-value constant serialization (amendment 5) - -Amendments 2-3 closed the `fn_val` + side-table serialization gaps that blocked -`cljw build`. The cold-start AOT lever (D-452: AOT-cache the 23 non-core -bundled `.clj` libs into the startup envelope, ~2.9 ms across all benches, -ADR-0056 Cycle 3) surfaced the LAST `UnsupportedValueTag`: a `.type_descriptor` -constant. The analyzer's `resolveClassValue` mints a canonical boxed class-value -ref (ADR-0059) when a class symbol — a `deftype`/`defrecord`/`defprotocol` name -or a host class — is resolved at analyze time and baked into a chunk's constant -pool (it appears in `clojure.core/protocols` + `clojure.zip`). The v2 serializer -had no arm for it. - -### A5-D1: serialize by canonical name, re-resolve import-blind - -A `.type_descriptor` constant is serialized by NAME — the descriptor's `fqcn` -(its `rt.types` key: the SIMPLE name for a user deftype/record/protocol, the -dotted FQCN for a host surface), behind a new wire tag `type_descriptor = 0x10`. -No descriptor CONTENT (field layout / method table / `kind`) is serialized: -the descriptor's behaviour is reconstructed when its DEFINING chunk runs (the -`deftype`/`defrecord`/`defprotocol` form registers the type at runtime via the -`__deftype!`/`__defrecord!` primitives + `extend-type`), and `runEnvelope` -interleaves deserialize→run per chunk in form order, so the defining chunk -ALWAYS runs before any referencing chunk deserializes. Baking a second content -copy would fork the descriptor-of-record and break the ADR-0059 one-ref-per- -descriptor invariant. This mirrors the `var_ref` (write ns+name, re-resolve via -env) and static-call-descriptor (write fqcn, re-resolve via `resolveJavaSurface`) -precedents. - -On load the name is re-resolved through a NEW import-BLIND -`analyzer.resolveDescriptorByKey(rt, key)` — the shared dispatch-order SSOT -(native → class-value → `rt.types`) that `resolveClassValue` also calls for its -primary lookup — NOT `resolveClassValue` itself. `resolveClassValue`'s -ns-import + simple-name-fallback layers serve SOURCE symbols; the wire carries -an already-canonical key, so routing it back through import resolution would -mis-resolve a baked constant under a shadowing import at the deserialize-time -current ns. `makeTypeDescriptorRef` returns the SAME canonical ref the build-time -constant held (ADR-0059). A miss is a concrete `TypeDescriptorUnresolved`, never -a silent nil (`no_op_stub_forbidden`). An anonymous (reify) descriptor — null -`fqcn`, a runtime value never an analyze-time constant — is an invariant guard -(`TypeDescriptorUnnamed`). - -The wire `VERSION` bumps 2→3 (the format-version policy: adding a value tag is a -version bump). `cljw build` artifacts embed their own runtime binary -(Deno-style), so a v2 artifact runs with its v2 decoder; the new cljw only reads -v3 payloads it builds. The `cljw-formats/0.1.0.edn` archive gains the byte-16 -entry. - -### Amendment-5 Devil's-advocate fork (2026-06-16, type_descriptor wire encoding, F-002/F-004/F-009/F-011/ADR-0059 envelope) — verbatim - -> **Alt A — smallest-diff: serialize the raw `kind` byte alongside the name.** -> Keep the name-string write, additionally emit `kind` + field count; on read -> still call the resolver but assert the re-resolved descriptor matches. -> *Better:* a corruption tripwire for a name re-resolving to a DIFFERENT -> descriptor (host class vs user deftype colliding on a simple name — reachable -> via `resolveClassValue`'s simple-name fallback). *Breaks:* a half-measure — -> `kind` does not FIX the collision (two deftypes, same simple name + field -> count, still pass), and it bloats the wire with redundancy the -> var_ref/fn_val/static-descriptor precedents all refused. Smallest-diff bias: -> papers a detector over an ambiguous resolver instead of fixing it. -> -> **Alt B — finished-form-clean: resolve through a shadowing-proof -> `rt.types`/native lookup, not `resolveClassValue`.** Keep name-only (contents- -> not-pointers is F-004-correct; defining-chunk-runs-first is sound). Change the -> RESOLVER: `resolveClassValue` routes a bare name through -> `env.current_ns.imports` + a simple-name fallback, so the resolved descriptor -> depends on which ns is current at deserialize time — an import shadowing a bare -> class name could re-resolve a baked constant to the wrong descriptor. The wire -> already carries the canonical `rt.types` key, so re-resolve through a direct, -> import-blind lookup that cannot be shadowed. Should the wire also store -> `kind`/field-layout/method-table? NO — the descriptor's content is -> reconstructed by the defining chunk running (`registerType` + `extend-type`); -> serializing the method-table means serializing runtime-constructed `Value` fn -> pointers and forking the ADR-0059 descriptor-of-record. Name-only IS the -> finished form for identity; the only gap is WHICH name-resolution path -> re-hydrates, and Alt B closes it. *Better:* eliminates the ns-shadowing -> mis-resolution class structurally. *Risk:* needs a small new resolver entry -> point; mitigate dispatch-order drift by factoring the key→descriptor dispatch -> core into a shared helper both resolvers call (itself finished-form-positive). -> -> **Alt C — wildcard: don't bake `type_descriptor` constants at all; lower the -> class symbol to a name-load opcode.** Change the ANALYZER so a class symbol in -> value position emits an `op_load_class` (name as an already-serializable -> string/symbol constant), resolved at execution time — like `var_ref`. *Better:* -> removes a wire tag from the decoder-permanence ledger entirely; unifies -> class-value with var-value under "named runtime entity, resolved at execution". -> *Breaks:* a depth-3 analyzer+VM+compiler surgery touching the runtime HOT path -> for a cold-path benefit (perf-campaign regression risk); half-applied it is two -> mechanisms (strictly worse); an F-011 resolution-TIMING shift (analyze-time → -> execution-time for an undefined class) needs its own equivalence audit. Out of -> proportion to D-452. -> -> **Recommendation: Alt B**, with the shared-dispatch-core factoring. The chosen -> design is correct on the two hard questions (name-only, not content; defining- -> chunk-runs-first ordering — verified against `runEnvelope`). It does NOT need -> `kind`/layout/method-table (Alt A is actively wrong — forks the descriptor-of- -> record, breaks ADR-0059). Its one real weakness is reusing the import-aware -> `resolveClassValue` as the load-time resolver when the wire carries a canonical -> key deserving an import-blind resolver. Alt C is the deepest-clean conceptual -> form but disproportionate; record as forward debt. No F-NNN blocks Alt B -> (F-004-clean, F-009-neutral, F-011-stronger, ADR-0059-preserving). - -### Amendment-5 decision - -**Alt B selected** (matches the DA recommendation). The first-cut implementation -reused `resolveClassValue` (the smallest-diff convenience — it was already there -and mostly works); the DA fork rated that the Cycle-budget-defer trap and Alt B -finished-form-clean, so the loop re-picked Alt B per `principle.md`. Landed: -(a) the import-blind `resolveDescriptorByKey(rt, key)` shared helper (the -native → class-value → `rt.types` dispatch core, now the single SSOT -`resolveClassValue` also calls for its primary lookup — no dispatch-order drift); -(b) `serialize.readValue`'s `type_descriptor` arm calls it directly (no `env`, -no import resolution). Alt A's `kind`/content baking is rejected on ADR-0059 -grounds (the DA's leading correctness finding, not a size call). Alt C -(no-tag, `op_load_class`) is recorded as forward debt **D-453** — the deepest -conceptual finished form, deferred as a depth-3 hot-path surgery disproportionate -to the cold-start lever. The choice is on design grounds (shadow-proof canonical- -key re-resolution), NOT diff size. - -## Selection rationale - -Alt 2 (本提案) を選択。 F-002 (finished-form wins) + cljw メインターゲット -が エッジ実行であり cold-start < 5ms 要件下で source bundle 形態が -許容不可、 + Tier 0 metadata + structured EDN により ops observability -が初日から成立する利点が決定的。 Alt 1 は IP 漏洩 vector + machine- -parseable error 不在で production 適合性低い。 Alt 3 は F-001 violation -(zwasm 依存方向逆転 + dynamic eval surface 喪失) で envelope 外。 - -Alt 2 の Breaks 5 件は受容コストとして識別: -- (i) 5-10% size 恒久支払 → Phase 6.16.a-1 baseline (820KB) に対し - ~50KB overhead = エッジ 10MB platform 制約に対しても余裕 -- (ii) EDN schema 11 key 固定 → key 追加は ADR-0034 amendment + decoder - archive bump で対応、 cost は明示的 -- (iii) decoder archive forward maintenance → 削除 forbidden / add only、 - v0.1.0 release 時に initial archive 確定後は cycle ごとに 1 entry - 追加するだけ、 amortise 容易 -- (iv) human / EDN 混入事故 → env var `CLJW_ERROR_FORMAT` で強制可能、 - default は TTY 判定だが ops が pipeline 設計時に明示 override 推奨 -- (v) Phase 12 entry でこれら 11 D-items を一括 implement する必要 → - D-064 (post-mortem) + D-066 (env var spec) + ADR-0033 cluster - workflow で段階着地 path 設計済 - -## Consequences - -- **Phase 12 entry deliverable**: bytecode embed pipeline 完成 (`cljw - build app.clj -o app` internal API、 single mode、 flag ゼロ) + - Tier 0 metadata serializer/deserializer + build-id injection + - Deno-style binary trailer + bootstrap cache build.zig 統合 + - `cljw render-error` internal API + `cljw-formats/.edn` - archive 初版 + error output stream-separated (TTY=human / pipe=EDN) - + env var recognition -- **Phase 14 entry deliverable (v0.1.0 release)**: `cljw build` / - `cljw render-error` CLI surface 公開 + `cljw-formats/0.1.0.edn` - archive lock + `CLJW_ERROR_FORMAT` / `CLJW_ERROR_LOG` env var - 公式 spec 化 + `cljw.error/with-context` macro 公開 -- **Phase 16 entry**: ClojureScript transpiler が `defn-` + `-name` - leaf を JS interop に置換するレイヤで `^:zig-leaf` metadata を読む - (ADR-0033 D4 + D-068 と整合); Wasm Component output が同 binary - trailer 形式を踏襲可能 -- **Phase 17 entry**: JIT go の場合、 D6 (ABI commitment 不要) の下で - bytecode に source-level metadata 保持 option を broad JIT 用に追加 - 可能 (ABI 維持しつつ拡張)、 narrow JIT は完全直交 (ADR-0033 D10 と - 整合) -- placement.yaml + compat_tiers.yaml schema は本 ADR と独立 (= placement - は ADR-0033、 compat_tiers は ADR-0029) -- format version 概念を撤回することで `version_field` / `magic upgrade` - 系の implementation cost を回避、 decoder archive の add-only policy - で互換性責任を decoder 側に集中 -- EDN event schema の固定 11 key は user runtime injection (`:request- - id` 等) を許す形で拡張可能、 schema 自体の breaking change は ADR- - 0034 amendment - -## Affected files - -- `.dev/ROADMAP.md` §9.14 / §9.16 — 既に v5 wiring (757a0b5) で - expansion 済 (本 ADR は SSOT 化) -- `.dev/debt.md` D-064 (decoder archive) / D-066 (env var spec) — 既に - 起票済 (757a0b5) -- `private/notes/clj_vs_zig_split_proposal_v5.md` §11-§14 + §16.1 + §19.2 - + App C (EDN schema 完全 spec) — 本 ADR の SSOT - -Phase 12 entry 以降の Affected files (本 cycle 時点では未着地): - -- `src/app/builder.zig` (Phase 12 着地、 build-id 埋め込み + bytecode - serialize 完成) -- `src/runtime/error/render.zig` (TTY-aware + EDN serializer 拡張) -- `src/runtime/error/event.zig` (新規、 EDN event schema 実装) -- `src/app/render_error.zig` (新規、 post-mortem tool entry point) -- `cljw-formats/0.1.0.edn` (新規 archive、 v0.1.0 release 時に lock) -- `build.zig` (cache_gen + build-id injection 拡張) - -## Revision history - -- 2026-05-25 issued + accepted with Devil's-advocate fork - (general-purpose subagent、 fresh context、 F-NNN envelope 内 3 - alternatives 取得 verbatim、 Alt 2 採択、 Alt 3 が F-001 violation - 寄りの旨を明示)。 v5 plan §11-§14 + §19.2 (1593 行、 self- - contained) を SSOT として参照、 Tier 0 metadata size baseline - measurement (Phase 6.16.a-1 d35dc3b terminus) を起票 prerequisite - として参照。 -- 2026-05-29 (amendment 1): D-100(b) `cljw build app.clj -o app` - landing. Added "v0.1.0 build envelope + build-time-eval - semantics" section (A1-D1..A1-D4): payload = sequence of v2 - chunks with `[u32 n_chunks]` + per-chunk `[u32 len]` framing - (NOT a single `(do …)` chunk — `analyzeDo` single-pass analysis - breaks in-file macros/requires); build loop = runner's per-form - read→analyze→compile→eval (Clojure AOT, build-time side effects - documented); D4 bootstrap-cache / Tier-0-metadata / build-id - blocks deferred at v0.1.0 per D6 and tracked by **D-131** so the - D3 常時-ON contract is not silently dropped; trailer magic - `"CLJC"` kept distinct from chunk magic `"CLJW"`. Devil's-advocate - fork (general-purpose, fresh context, F-001/F-002/F-009 envelope, - 3 alternatives) output embedded verbatim in Alternatives - considered; Alt B (finished-form-clean) selected. Landing gate - narrated in ADR-0015 amendment 5 (ungated; `phase_at_least_14` - guards the io stub swap only). -- 2026-05-29 (amendment 2): fn_val constant serialization. Step 0.6 - surfaced that the v2 serializer rejected `fn_val` constants, so - `cljw build` could not serialize any program with `(defn …)` — - a hollow v0.1.0 deliverable. Added "fn_val constant serialization" - section (A2-D1..A2-D4): serialize fn constants by contents (slot_base - + per-method arity/has_rest + recursive `serializeChunk` of the method - body); `closure_bindings != null` is the only error (invariant guard, - never a constant); deserialized fns are gpa+trackHeap with sub-chunks - freed via `freeChunk` recursion (freeEnvelope-before-rt.deinit ordering - documented), sentinel `body` Node with a tree_walk guard, param-name - strings dropped (debug-only, D-139). Devil's-advocate fork - (general-purpose, fresh context, F-002/F-004/F-009/F-010 envelope, 3 - alternatives) embedded verbatim; Alt B (finished-form-clean) selected. - Closes the D-100(b) "serializer Discharged" overclaim — same - claimed-done-but-incomplete pattern as the cycle-4 lazy fns this - session. -- 2026-06-09 (amendment 3): require-closure embedding (D-356). Building the - bookshelf multi-file demo surfaced that `cljw build` only embedded the ENTRY - file's chunks — a `(require '[lib])` over a classpath (a) raised - `lib_not_found` at build (buildFile never set load_paths / installed the fs - resolver) and (b) even after that, raised `lib_not_found` at RUN (the lib ns - is not in the payload; tryRunEmbedded has no fs/classpath). Added the - "require-closure embedding" section (A3-D1..A3-D5): the op_require-idempotency - enabler (vm.zig:713-716) makes embedding the closure's chunks BEFORE the entry - require chunk sufficient (run-time require → already_loaded → skip, no - resolver needed); capture mechanism = chunk-capture during the real - build-time load via a Layer-0 type-erased `build_chunk_sink` callback - (post-order, exactly-once eval, no toposort, no double-eval); bootstrap-ns - exclusion via a new `ResolvedSource.from_filesystem` flag (F-013-clean, no - hardcoded allowlist); the classpath prerequisite (buildFile load_paths + - installChained-after-setupCore + cli.zig `-cp/-A` parse mirroring the run - path, F-009). Devil's-advocate fork (general-purpose, fresh context, - F-002/F-009/F-011/F-013 envelope, 3 alternatives) embedded verbatim; Alt-2 - (chunk-capture during load, finished-form-clean) selected — Alt-1/Alt-3 - double-eval, an F-002 disqualifier. The build-time `(-main)` server-start - hazard is scoped to the serverless-v2 demo (D-362), not this amendment. -- 2026-06-09 (amendment 4): `-m` main-entry mode (D-363) — resolves the am3 - build-time `(-main)` server-start hazard as a real cljw feature. Added the - "`-m` main-entry mode" section (A4-D1..A4-D4): two build modes (script = - unchanged top-level-as-program; main = `cljw build -m ` requires the - closure, embeds NO `-main` call, invokes `(/-main args)` at RUN only); - the entry point is artifact metadata (an optional entry manifest - `{entry_ns, entry_args}` at the front of the payload envelope), mirroring - ELF/Mach-O/jar — script mode = `entry_ns null`; run-side dispatch routes - through the shared `run_mode.synthMainNs` (promoted `pub`) so the built `-m` - binary is byte-identical to `cljw -M -m` (F-011), with runtime argv - (`./out 8080` → `-main`); deps.edn `:main-opts` drives the entry. The - undecidable env-vs-side-effect classifier stays rejected (Clojure-AOT-faithful - build=load; "-main is the escape hatch"); the cw v0 `build_mode` per-primitive - suppression stays rejected (F-013). Devil's-advocate fork (general-purpose, - fresh context, F-002/F-009/F-011/F-013 envelope, 3 alternatives) embedded - verbatim; Alt-2 (entry-manifest, finished-form) selected — entry-as-metadata - mirrors real executable formats + F-011-clean runtime argv; Alt-3 - auto-detection rejected (F-013 hazard). Choice on design grounds, not diff - size. -- 2026-06-16 (amendment 5): `type_descriptor` class-value constant serialization - (D-452) — the last `UnsupportedValueTag` blocking the cold-start AOT of the 23 - non-core bundled `.clj` libs. Added the "type_descriptor class-value constant - serialization" section (A5-D1): a class-value constant serializes by canonical - NAME (the descriptor `fqcn`, wire tag `0x10`); no descriptor content is baked - (the defining `deftype`/`defrecord`/`defprotocol` chunk reconstructs it at run, - and `runEnvelope` interleaves so it always runs first — content-baking would - fork the ADR-0059 descriptor-of-record). Re-resolved on load through a new - import-BLIND `analyzer.resolveDescriptorByKey` (the native → class-value → - `rt.types` dispatch core, now shared with `resolveClassValue`) — NOT - `resolveClassValue`, whose ns-import + simple-name-fallback layers would - mis-resolve a canonical wire key under a shadowing import at the deserialize ns. - Wire `VERSION` 2→3; `cljw-formats/0.1.0.edn` gains the byte-16 entry; a miss is - a concrete `TypeDescriptorUnresolved` (never silent nil). Devil's-advocate fork - (general-purpose, fresh context, F-002/F-004/F-009/F-011/ADR-0059 envelope, 3 - alternatives) embedded verbatim; **Alt B (finished-form-clean, import-blind - resolver) selected** — the first cut reused `resolveClassValue` (smallest-diff - convenience), the DA rated that the Cycle-budget-defer trap, the loop re-picked - Alt B per `principle.md`. Alt A (`kind`/content baking) rejected on ADR-0059 - grounds; Alt C (no-tag `op_load_class`, deepest conceptual form) recorded as - forward debt **D-453** (depth-3 hot-path surgery disproportionate to the - cold-start lever). Choice on design grounds, not diff size. diff --git a/.dev/decisions/0035_require_spec.md b/.dev/decisions/0035_require_spec.md deleted file mode 100644 index e95887fd2..000000000 --- a/.dev/decisions/0035_require_spec.md +++ /dev/null @@ -1,812 +0,0 @@ -# 0035 — `(ns ...)` analyzer special form + `require` runtime fn + alias storage + cross-ns `^:private` × `:refer` filter + circular-require detection + bootstrap loader topo-sort + per-file `SourceContext` registry + swappable resolver slot - -**Status**: Accepted (Devil's-advocate fork landed 2026-05-26) -**Date**: 2026-05-26 -**Author**: Shota Kudo (drafted with Claude autonomous loop) -**Tags**: phase-6-late, namespace, require, bootstrap, multi-file, -F-001, F-002, F-009 - -## Context - -cw v1 が Phase 6.16.b-3 で landed させた multi-file `.clj` -bootstrap (`core.clj` + `set.clj` + `string.clj` + `walk.clj`) は -provisional behaviour を 3 cluster (11 marker sites + 3 -`feature_deps.yaml` entries) 抱える: (i) `evalInNs` / `op_in_ns` -が `rt` と `clojure.core` を auto-refer する (ADR-0032 era 妥協)、 -(ii) `bootstrap.zig` 末尾と `primitive.zig::registerAll` と -`macro_transforms.zig::registerInto` の 3 ヶ所が hardcoded -`referAll` を fire する (= 妥当な fan-out 設計が確定するまでの placeholder)、 -(iii) 4 `.clj` 先頭が `(in-ns 'foo)` 裸書き (= `(ns ...)` macro -未着地)。 全 3 cluster は ADR-0035 (= D-063 起票) が `(ns ...)` -macro + `require` semantics + bootstrap loader 仕様確定で discharge -することが予定されていた。 - -加えて D-058 (bootstrap loader per-file `SourceContext`) -が同じ "loader が `.clj` file identity を意識する" 設計面に属するため、 -ADR-0035 で同時解消するよう v5 §19.3 D6 で予定されていた。 - -cw v1 の finished form は JVM Clojure の `(ns foo -(:refer-clojure :exclude [...]) (:require [x :as y :refer [z]]))` -macro 形を提供する必要があるが、 cw v1 distribution model -(F-001 = zwasm v2 統合不可避、 = embedded + AOT-style) は JVM -classpath-based file resolution と整合しない。 v5 §19.3 D3 -(`cljw_path` env var) と D5 (lazy vs eager load) は Phase 12+ -留保。 本 ADR は Phase 6.16.b-4 で着地する scope に限定し、 -Phase 12+ の swap-in 余地を残す形で `Runtime.requireResolver` -slot を導入する。 - -詳細な v5 plan は -`private/notes/clj_vs_zig_split_proposal_v5.md` §19.3 + §21.1 SSOT、 -本 cycle survey は `private/notes/phase6-6.16.b-4-survey.md` SSOT。 - -## Decision - -### D1: `(ns foo ...)` は analyzer special form (= macro でない) - -`src/eval/analyzer/special_forms.zig` に `analyzeNs` を追加。 -`(ns foo & references)` を IR 上で explicit op-sequence に -展開する: `in-ns 'foo` → (`:refer-clojure` 指定 unless explicit -override) `refer 'clojure.core` (with `:exclude` / `:only` filter) → -各 `(:require [x :as y :refer [...]])` 引数を `require` fn 呼出 -sequence 化。 - -実装は単一 `NsNode { name: []const u8, refer_clojure: ?ReferClojureSpec, requires: []RequireSpec }` を analyzer pass で -合成し、 backend (tree_walk + VM) が順次 in-ns → refer → require -を実行する。 `(:use ...)` は本 cycle で `feature_not_supported` -raise、 `(:import ...)` は no-op (cw v1 は Java Class import を -Var-ref qualified name 経由でカバー)、 `(:gen-class ...)` は -silent-ignore (JVM compile-only feature)。 - -**選定理由 (Devil's advocate Alt 2)**: analyzer special form -は (a) `(ns)` `(ns foo :bad-kw)` `(ns foo (:require))` 等の -malformed input に対し precise location-tagged error を analyze -時に出せる、 (b) user `(def ns ...)` で shadowing できない、 -(c) ADR-0032 の `in-ns` special form precedent -(`analyzer.zig:368-382`) と pattern 整合的。 macro path は -smallest-diff bias smell (F-002)。 - -### D2: `clojure.core/require` は runtime fn (varargs) - -`src/lang/primitive/require.zig` に `clojure.core/require` -builtin fn を実装。 varargs 受理: `(require 'sym ...)` または -`(require '[sym & opts] ...)`。 opts 認識: `:as `、 -`:as-alias ` (load なし alias 化)、 `:refer [a b c]`、 -`:refer :all`、 `:reload`、 `:reload-all`、 `:verbose` (no-op -本 cycle)。 - -per-libspec 処理: (a) resolver で source 取得、 (b) 未 load -なら load + intern (`:reload` 強制再 load)、 (c) `:as` / -`:as-alias` で alias 設定、 (d) `:refer` で refer 設定 -(D4 の private filter 適用)。 - -### D3: alias storage = env-table (Namespace.aliases) - -`Namespace.aliases: StringHashMapUnmanaged(*Namespace)` は -既存 (`src/runtime/env.zig:116, 128`)。 `Namespace.setAlias(env, -alias_name, target_ns)` public API を追加 (key を `env.alloc` -で duplicate、 既存 entry あれば silent overwrite)。 - -analyzer の `analyzeSymbol` で qualified `s/foo` を解析する -際: `env.current_ns.aliases.get("s")` を最初に試行、 hit したら -そこの ns で `foo` を resolve。 miss なら従来通り `env.findNs("s")`。 -REPL-time `(alias 's 'clojure.string)` で動的変更可能。 - -### D4: `:refer` × `^:private` = silent skip (implicit) + hard error (explicit) - -`env.referAll(from, to)` は private (= `Var.flags.private`) を -**silent skip** (JVM 互換、 `:refer :all` 経路)。 一方 -`env.referOne(from, to, name)` で explicit に名前指定された Var -が `^:private` の場合は `private_access_error` を raise -(= JVM より厳しい、 cw v1 は require call 時点で fail-fast に -する: ADR-0035 cycle の Bad Smell sensor "don't ship a lie" 適用)。 - -### D5: 循環 require 検出 = error (新 catalog Code `circular_require`) - -`Runtime.require_in_progress: std.StringHashMapUnmanaged(void)` 追加。 -`requireOne(env, ns_name)` は (a) `require_in_progress` に -`ns_name` がいたら `circular_require` を raise (message -template: `"Cyclic load dependency: {[chain]s}"`、 `chain` は -現在の require call stack を newline-join で render)、 (b) -そうでなければ set に add、 load 実行、 (c) errdefer で -return 時 remove。 - -`circular_require` Code を `src/runtime/error/catalog.zig` に -追加 (kind = `name_error`, phase = `analysis`)。 - -### D6: bootstrap topology = hybrid (boot list + 同じ require path) - -`bootstrap.loadCore` は `BOOT_NAMESPACES: []const []const u8 = -&.{ "clojure.core", "clojure.set", "clojure.string", "clojure.walk" }` -の固定 list を持つ (= 順序 explicit、 hidden dependency walking -なし)。 各 entry を `requireOne(env, ns_name)` 経由で load — -bootstrap と user code が同じ load path を共有する。 既存 -hardcoded `FILES` table は削除、 source 取得は D8 resolver 経由。 - -これにより (a) bootstrap fan-out の `referAll` 3 ヶ所は不要 -(`(ns ...)` macro 展開が `:refer-clojure` を発火するため)、 -(b) circular detection は bootstrap 内 require にも自動適用、 -(c) `evalInNs` の rt + clojure.core auto-refer も削除可能 (= -ns macro 展開が refer を担う)。 - -### D7: per-file `SourceContext` = Runtime registry (D-058 closure) - -`Runtime.source_registry: std.StringHashMapUnmanaged(SourceContext)` -追加 (key = file label like ``、 value = SourceContext -struct)。 `requireOne` が source load 時に entry を populate、 -`bootstrap.loadCore` が boot ns 列を iterate する際にも populate -する。 - -`src/runtime/error/print.zig` の renderer は -`info.location.file` を registry で lookup し、 hit したらその -`SourceContext.text` を source-line preview に使用、 miss なら -fallback として `bootstrap_ctx` を使用 (= synthetic location -safe default)。 D-058 closure。 - -### D8: resolver = single swappable fn slot (`Runtime.requireResolver`) - -`Runtime.requireResolver: *const fn(*Runtime, ns_name: -[]const u8) anyerror!?SourceText` (SourceText = `?[]const u8`、 -`null` = lib_not_found)。 boot 時に `embeddedResolver` を default -として install — 4 ns (`clojure.core` / `clojure.set` / -`clojure.string` / `clojure.walk`) 名 → `@embedFile` で取得した -source text を返す。 Phase 12+ で `cljw_path` / build artifact / -Wasm pod 用 resolver に swap される (= slot 1 個、 chain 抽象 -なし)。 - -`lib_not_found` Code を `src/runtime/error/catalog.zig` に追加 -(kind = `name_error`, template: `"Could not locate {[ns]s}.clj on -the require resolver"`)。 - -**選定理由 (Devil's advocate Alt 2)**: Phase 12+ classpath + -Phase 16+ Wasm pod の存在は F-001 で確定だが、 「2 つ以上の -resolver を同時に持つ chain」 は今この cycle に存在せず、 -chain 抽象を先取りするのは Reservation-as-bias smell (F-002)。 -slot swappable で F-001 future-proof を確保し、 chain は -2 ndaresolver が現れる cycle に判断を譲る。 - -### D9: 同 sub-cycle d で discharge する provisional cluster - -`feature_deps.yaml` 更新: - -- `runtime/eval/in_ns_auto_refer`: `provisional → landed`、 - 4 PROVISIONAL marker lines 削除 (tree_walk.zig × 2 + vm.zig × 2)。 - `evalInNs` / `op_in_ns` を pure ns-switch (auto-refer なし) に - 簡略化。 -- `runtime/bootstrap/refer_table`: `provisional → landed`、 - 3 PROVISIONAL marker lines 削除 (bootstrap.zig + primitive.zig - + macro_transforms.zig)。 該当 `referAll` 呼出を `(ns ...)` - macro 展開が代替するため削除。 -- `runtime/eval/bare_in_ns_decl`: `provisional → landed`、 - 4 `.clj` heads を `(in-ns 'foo)` → `(ns foo (:refer-clojure))` - に書換、 4 PROVISIONAL marker lines 削除。 -- `special_form/ns_macro`: `planned → landed`。 - -`.dev/debt.md`: -- D-058 → Discharged (sub-cycle c で per-file SourceContext registry - 着地)。 -- D-063 → Discharged (ADR-0035 全体着地)。 - -11 marker sites + 3 yaml entries (status flip) + 2 debt rows -(close) = atomic commit in sub-cycle d (`check_provisional_sync.sh` -hook が三者同期 enforce)。 - -## Alternatives considered - -Devil's-advocate fork (depth-3、 mandatory)、 fresh-context -`general-purpose` subagent、 F-001 / F-002 / F-007 / F-009 envelope -内で 3 alternatives 取得。 output verbatim: - -> ## Devil's-advocate review of ADR-0035 -> -> ### Alt 1 — smallest-diff: "runtime `require` fn only, defer the -> `(ns ...)` macro one cycle" -> -> **Shape (1 sentence)**: ship D2/D3/D5/D7/D8 this cycle as a -> runtime `clojure.core/require` fn + env-aliases + circular-detect -> + per-file SourceContext registry + embedded resolver; punt D1 -> (`(ns ...)` macro), D4 (`:refer × private`) and D6's full hybrid -> into Phase 6.16.b-5. -> -> **Per-D deltas**: -> - **D1**: defer. `.clj` heads stay `(in-ns 'foo)` + manual -> `(require ...)` (legal Clojure — the JVM REPL works this way -> too). No macro_transforms.zig change. -> - **D4**: defer. `:refer` only takes explicit name lists this -> cycle; `:refer :all` raises `not_implemented`; no `^:private` -> filtering path because no `:refer :all` consumers. -> - **D6**: keep the `FILES` table; bootstrap still hardcodes order, -> but each entry's body is `(in-ns 'foo) (require ...)` — i.e. -> bootstrap and user code share the runtime fn from D2, just -> bootstrap is allowed to also `in-ns` directly. No "hybrid -> loader" rewrite. -> - **D9**: only `runtime/eval/in_ns_auto_refer` flips (the 4 -> vm/tree_walk sites — once user code uses explicit `(require -> 'clojure.core)`, auto-refer can die). `refer_table` and -> `bare_in_ns_decl` stay PROVISIONAL → ADR-0035 part 2 next cycle. -> -> **Better than proposal**: -> - One-cycle blast radius is roughly half (no macro expansion table -> change, no JVM-shape `(ns ...)` edge cases like `:gen-class` -> silent-ignore semantics, no `(:refer-clojure :exclude ...)` -> filter logic). Lower regression surface in 6.16.b-4. -> - Keeps `(ns ...)` as a known-unknown for one more survey cycle. -> JVM `(ns ...)` is genuinely fiddly (`:refer-clojure :rename`, -> `:load`, AOT interactions); a one-cycle delay buys a Step-0 -> survey that focuses just on it. -> - Hardcoded FILES table is honest about what it is: a 4-line -> bootstrap manifest. Replacing it with "same `require` path as -> user code" creates a chicken-and-egg moment where loading -> `clojure.core` requires `clojure.core/require` to be defined — -> which means a hand-rolled `requireOne` Zig fn ends up underneath -> anyway. Alt 1 doesn't pretend that's gone. -> -> **Breaks / worsens**: -> - **F-002 violation risk**: leaves 7 of 11 PROVISIONAL markers -> in place after the cycle. The next cycle has to re-enter this -> whole design — that's the Smallest-diff bias smell from -> `.dev/principle.md`. The whole point of ADR-0035 per D-063 is -> that 3 provisional behaviours converge here. -> - Bare `(in-ns 'foo)` at file heads is not idiomatic Clojure; it -> ships JVM-divergent surface. Future readers (and any chapter- -> cadence resumption per F-007) have to explain why. -> - `bare_in_ns_decl` PROVISIONAL stays — meaning the `.clj` files -> are still self-documented as "not done". -> -> ### Alt 2 — finished-form-clean: "match proposal on -> D1/D2/D3/D5/D7/D9, but make `(ns ...)` an analyzer special form -> and split the resolver from D8" -> -> **Shape (1 sentence)**: D1 becomes an analyzer special form (not -> a macro); D6/D8 are conceptually split — the loader topology (D6) -> is hybrid as proposed, but the resolver (D8) is a tiny `*const -> fn` slot today with no "resolver chain" abstraction; everything -> else matches the proposal. -> -> **Per-D deltas vs proposal**: -> - **D1**: `(ns ...)` is an analyzer special form in -> `src/lang/analyzer.zig` alongside `in-ns`, not a macro in -> `macro_transforms.zig`. The analyzer reads the references list, -> emits an explicit op-sequence (in-ns + refer-clojure + -> require-per-arg) into the IR. No round-trip through -> reader/macro-expander. -> - **D8**: just `Runtime.requireResolver: *const fn(*Runtime, -> []const u8) anyerror!?SourceText` — no "stackable resolvers" -> framing. Phase 12+ swaps the slot when `cljw_path` lands; -> Phase 16+ swaps again for Wasm pods. F-001 (zwasm v2 -> unavoidable) is satisfied by "the slot is swappable", not by -> "we built a chain today". -> -> **Better than proposal**: -> - **Analyzer special form (D1) beats macro on error quality**: -> `(ns)` with no name, `(ns foo :bad-keyword [x])`, `(ns foo -> (:require))` all produce precise location-tagged errors at -> analyze time, not macro-expansion stage where positional info -> is harder to reconstruct. Compare ADR-0032's `in-ns` analyzer -> path at `analyzer.zig:368-382` — it already establishes the -> pattern. -> - Analyzer-form `(ns ...)` cannot be accidentally shadowed by a -> user `(def ns ...)` — Clojure JVM's macro can be, and people -> hit this. Cleaner finished form. -> - **Resolver as a single fn slot (vs resolver chain abstraction)**: -> F-002 says smallest-diff is tiebreaker when finished-forms -> match. A "chain of resolvers" is a Reservation-as-bias smell -> from `.dev/principle.md` — we don't have classpath, build -> artifact, OR pod loading today; one swappable slot is the -> finished form for "embedded source", and the next two are -> separate decisions for their respective ADRs (12+, 16+) without -> prejudging their composition. -> -> **Breaks / worsens**: -> - Analyzer special form means `(ns foo ...)` cannot be `apply`'d -> or programmatically constructed — though neither can JVM's -> `(ns ...)` macro be in practice (it must appear at top level). -> Mostly equivalent. -> - Loses the "any user can write a similar macro for their own -> DSL" angle. Probably irrelevant for cw v1 audience. -> - Adding a special form is a wider Zig diff than a -> macro_transforms entry; main-loop's "macro-table entry" was the -> smaller surgery. Within F-002, finished-form wins anyway. -> -> ### Alt 3 — wildcard: "defer D7 (per-file SourceContext) out of -> this ADR; keep D-058 open; ship D1–D6 + D8 + partial D9" -> -> **Shape (1 sentence)**: per-file SourceContext threading is a -> renderer-side concern that doesn't block the require/ns design -> surface — split it into a follow-up ADR-0036 and let ADR-0035 -> stay focused on `require`/`ns` semantics. -> -> **Per-D deltas**: -> - **D7**: dropped from ADR-0035. The renderer keeps using the -> single-`bootstrap_ctx` fallback. Multi-file bootstrap errors -> continue to point at the wrong source line (the current bug). -> D-058 stays open with `Last reviewed` reset. -> - **D8**: shipped as proposed — resolver is needed to make -> `require` work at all. -> - **D9**: discharge plan loses the D-058 closure but flips all -> the same yaml entries + 11 marker sites. -> -> **Better than proposal**: -> - **Tighter ADR scope = clearer Devil's-advocate framing for -> future amendments**. D7 is genuinely orthogonal: it's about -> which `SourceContext` the renderer pulls when an error fires, -> not about `require` semantics. Conflating them means future -> revisions of "how do we render multi-file errors" have to amend -> an ADR that also governs `(ns ...)` macro shape. -> - Lower commit-size risk in sub-cycle d. The proposed -> single-commit "11 marker sites + 3 yaml entries + 2 debt rows" -> is at the upper bound of what's reviewable. Dropping D7 trims -> to ~3 source files modified. -> - D-058's `Barrier: Phase 7 entry — require lands` was written -> before this cycle existed; the barrier is met by ADR-0035 D8 -> (the resolver) regardless of whether D7 ships in the same ADR. -> -> **Breaks / worsens**: -> - **Bad Smell — "I'll fix it later"**. D-058 has been open since -> 2026-05-25 with a known wrong-source-line bug. Once `require` -> lands and bootstrap loads 4+ files via a real loader path, the -> bug's blast radius widens (more files = more chances the wrong -> source line shows up). Deferring is exactly the -> Progress-pressure smell. -> - Two ADRs cycle through where one suffices. F-002 prefers -> consolidation when the finished-form is coherent — and -> `require` + per-file source context ARE one finished-form -> concern (both are "bootstrap loader knows about per-file -> identity"). -> - D-058's Barrier text in `.dev/debt.md:83` explicitly names -> Phase 7 entry as the fix moment; deferring requires editing -> that row to push out, which is a Reservation-as-bias adjacent -> move. -> -> ### Recommendation: **Alt 2** (finished-form-clean) with one -> carry-over from proposal -> -> Land Alt 2's analyzer-special-form `(ns ...)` (D1) and single-slot -> resolver (D8); take everything else from the proposal as-is. -> Specifically: -> -> - **D1 → Alt 2**: analyzer special form, not macro. F-002 wins -> on error quality + non-shadowability; ADR-0032's `in-ns` -> precedent at `analyzer.zig:368-382` makes the surgery shape -> known. The "macro table" path is smallest-diff bias. -> - **D2–D7, D9 → proposal**. -> - **D8 → Alt 2's single-slot framing**: `Runtime.requireResolver: -> *const fn(...)`, no chain abstraction. F-001 satisfied by -> swappability (Phase 12+ classpath swaps it, Phase 16+ Wasm pod -> swaps it). Building the chain today is a Reservation-as-bias -> smell — there is no second resolver in the room. -> -> **Reasoning**: F-002 (finished-form wins) dominates the D1 -> decision. F-001 + F-002 jointly dominate D8 — a swappable slot -> satisfies F-001 without the Reservation-as-bias of a pre-built -> chain. Alt 1 fails F-002 hard (7 of 11 PROVISIONALs stay → -> re-entry next cycle). Alt 3 fails the smell sensor on D-058 -> deferral. -> -> ### "Violates F-NNN" findings -> -> None of the three alternatives requires violating F-001 / F-002 -> / F-007 / F-009. The envelope is comfortably wide for this ADR. -> -> One adjacent observation (not a finding, just context for the -> main loop): Alt 2's "eliminate the `rt` namespace entirely" -> candidate from the brief's prompt would be a finished-form-clean -> shape but expands the cycle scope well past Phase 6.16.b-4; that -> is a depth-3 ADR of its own and is correctly excluded from this -> cycle's envelope, not a violation. - -## Selection rationale - -Devil's-advocate Alt 2 を採択 (D1 + D8 部分)、 残 D2-D7 + D9 は -proposal そのまま。 理由: - -- **D1 analyzer special form 採択**: F-002 finished-form 評価で - Alt 2 が proposal を上回る (error precision + shadow 不可)。 - smallest-diff (= macro table 1 entry) は本来の cleanness を - smallest-diff bias smell として却下。 -- **D8 single fn slot 採択**: F-001 future-proof (zwasm v2 + - Phase 12+ classpath) は slot swappable で十分成立。 chain - abstraction は今 cycle に複数 resolver が存在しない (= - Reservation-as-bias smell)。 -- **Alt 1 全体却下**: 7/11 PROVISIONAL markers が次 cycle に - 繰り越し = Smallest-diff bias の本丸。 D-063 が ADR-0035 を - "convergence point" と定めた前提が崩れる。 -- **Alt 3 部分却下**: D7 を分離する argument は理論的に正しいが、 - D-058 の barrier が「require lands」 で本 cycle が require - cycle そのもの、 deferral は Progress-pressure smell。 同 ADR - 内吸収を選択。 -- **Adjacent observation 留保**: rt ns elimination は本 cycle - scope 外で正しく除外。 sub-cycle a per-task note (D-NEW-A) に - evaluate 候補として残置済。 - -violates-F-NNN finding: なし。 - -## Consequences - -### Positive - -- 11 PROVISIONAL marker sites + 3 yaml entries + 2 debt rows - が単一 sub-cycle (d) で discharge、 ns-machinery cluster が - finished form に到達。 -- bootstrap と user code が同じ `requireOne` path を共有、 - load logic の 2-tier 分裂解消。 -- `Namespace.aliases` の REPL-time 変更 path が API として確立、 - Phase 12+ tooling 拡張に向けた seam が clean。 -- D-058 が closure、 multi-file bootstrap error の source-line - preview bug が解消。 - -### Negative / costs - -- analyzer special form 追加 (D1) は macro 化より surgery scope - が広い (analyzer.zig + node.zig + 2 backend 追加)。 cycle 内で - 完結するが review surface 増。 -- bootstrap.zig は hardcoded FILES table を失い `requireOne` 経由 - に統一されるため、 boot 初期化失敗時の trace が require chain - 経由になる (debug 経路が間接化、 ただし error message は - circular_require + lib_not_found で十分 explicit)。 -- `(:gen-class ...)` silent-ignore は user に「無視された」 - signal を出さない (compat tier 上 Tier D だが本 cycle で warn - channel 整備せず)。 - -### Deferred to Phase 12+ - -- `cljw_path` env var / `-cp` flag 経由の file resolution - (v5 §19.3 D3、 D-067 候補)。 -- lazy load on first-reference vs eager-on-require (v5 §19.3 D5)。 -- `:refer-clojure :rename` / `(:use ...)` / `(:load ...)` (JVM - feature parity の残り)。 - -### Deferred to Phase 16+ - -- Wasm pod / Component model 経由の require (F-001 + D-067)。 - -## Affected files - -### New - -- `.dev/decisions/0035_require_spec.md` (this file). -- `src/eval/analyzer/special_forms.zig` 内 `analyzeNs` (or 新 - `analyzer/ns_form.zig`、 D-030 split 状況次第)。 -- `src/eval/node.zig` 内 `NsNode` + `RequireNode` (or 個別 file)。 -- `src/lang/primitive/require.zig` (`clojure.core/require` builtin - fn)。 -- `test/e2e/phase6_16_b_4_ns_require.sh` (Layer 2 e2e)。 - -### Modified - -- `src/runtime/runtime.zig` — `requireResolver` slot + `require_in_progress` - set + `source_registry` registry を追加。 -- `src/runtime/env.zig` — `Namespace.setAlias` public API + 既存 - `referAll` の private filter 適用 + 新 `referOne`。 -- `src/runtime/error/catalog.zig` — `circular_require` + `lib_not_found` - Code 追加。 -- `src/runtime/error/print.zig` — `source_registry` lookup 追加、 - fallback path 維持。 -- `src/lang/bootstrap.zig` — `loadCore` を `BOOT_NAMESPACES` list - + `requireOne` 呼出に書換、 hardcoded `FILES` table 削除、 - `CORE_SOURCE` / `SOURCE_LABEL` compat exports 削除。 -- `src/lang/primitive.zig` — `referAll(rt_ns, user_ns)` 削除 - (PROVISIONAL discharge)。 -- `src/lang/macro_transforms.zig` — `referAll(rt_ns, user_ns)` - 削除 (PROVISIONAL discharge)。 -- `src/eval/backend/tree_walk.zig` — `evalInNs` の rt + clojure.core - auto-refer 削除 (2 PROVISIONAL marker 解消)、 pure ns-switch 化。 -- `src/eval/backend/vm.zig` — `op_in_ns` の rt + clojure.core - auto-refer 削除 (2 PROVISIONAL marker 解消)、 pure ns-switch 化。 -- `src/main.zig` — `bootstrap_ctx` 単一 SourceContext を撤廃、 - registry-backed lookup に切替。 -- `src/lang/clj/clojure/core.clj` — 先頭を `(ns clojure.core - (:refer-clojure))` (or 等価形) に書換。 -- `src/lang/clj/clojure/set.clj` — 同様、 `(ns clojure.set - (:require [clojure.core :as cc])` 等の正規形。 -- `src/lang/clj/clojure/string.clj` — 同様。 -- `src/lang/clj/clojure/walk.clj` — 同様。 -- `feature_deps.yaml` — 3 ns-machinery entries flip `provisional - → landed` + `special_form/ns_macro` flip `planned → landed` + - 11 PROVISIONAL marker line 削除。 -- `.dev/debt.md` — D-058 + D-063 を Discharged。 -- `test/run_all.sh` — 新 e2e 登録。 - -## Sub-cycle decomposition - -- **Sub-cycle b** (本 commit): ADR-0035 draft + Devil's-advocate - output verbatim 取込み + Accepted stamp。 source 変更なし。 -- **Sub-cycle c**: D1-D8 source 実装 (analyzer special form + - require fn + alias API + circular detect + per-file SourceContext - + resolver slot)。 TDD red→green→refactor で複数 commit に - 分割可。 既存 bootstrap path を一旦 backwards-compat に保ちつつ - 新 path を導入、 まだ `.clj` heads は `(in-ns ...)` のまま。 -- **Sub-cycle d**: D9 discharge atomic commit — `.clj` heads - 書換 (4) + PROVISIONAL marker 削除 (11) + yaml flip (3 → landed - + 1 planned → landed) + debt close (D-058 + D-063) + evalInNs - / op_in_ns simplification + bootstrap.zig finalization。 - `check_provisional_sync.sh` hook が三者同期 enforce。 - -## Revision history - -- 2026-05-26 issued + accepted with Devil's-advocate fork - (general-purpose subagent、 fresh context、 F-001 / F-002 / - F-007 / F-009 envelope 内 3 alternatives 取得 verbatim、 Alt 2 - D1 + D8 部分採択、 残 D2-D7 + D9 proposal そのまま)。 v5 plan - §19.3 + §21.1 SSOT、 survey - `private/notes/phase6-6.16.b-4-survey.md` SSOT、 sub-cycle a - per-task note `private/notes/phase6-6.16.b-4-sub-a.md` 参照。 - ADR-0032 (in-ns special form) + ADR-0033 D4 (D-071 Part 3 - closure) を依存先決定として参照。 D-063 + D-058 起票内、 - 実装は sub-cycle c/d で着地。 - -- 2026-05-26 D2 amendment (sub-cycle c.4) — `require` を - runtime fn ではなく analyzer special form として実装すること - に決定。理由: cw v1 は F-004 Group A slot 1 (symbol heap - value) を未実装、 quoted symbol を Value として表現できない - ため runtime fn の literal symbol arg 経路が立たない。 - analyzer special form は ADR-0032 の `in-ns` precedent - (`analyzer.zig:368-382`) と同じ Form-unwrap pattern を使う。 - 将来 symbol heap value が landing する Phase で runtime fn - surface への移行は可能 (impl は Zig 内部にあるため pattern - shift で済む)。 - -- 2026-05-26 D9 amendment (sub-cycle d) — 11 PROVISIONAL marker - sites の全削除 + 3 yaml entries (`in_ns_auto_refer` / - `refer_table` / `bare_in_ns_decl`) を `provisional → landed` - flip + `special_form/ns_macro` を `planned → landed` flip - + D-058 + D-063 を Discharged に。 ただし当初 D9 が描いた - 「evalInNs / op_in_ns の auto-refer 全削除 + bootstrap - refer_table 全削除」 は **部分採択**: (a) bootstrap fan-out - は `{user, set, string, walk}` → `{user}` のみに縮退 (= 4 - `.clj` heads は自前で `(ns foo (:refer-clojure))` を発火するため - set/string/walk 部分は冗長、 user/ は `.clj` でないため保持)、 - (b) evalInNs / op_in_ns の rt + clojure.core auto-refer は - 「in-ns convenience refer (REPL-style ns hopping 用)」 として - KEEP、 marker は PROVISIONAL から非-PROVISIONAL コメントに rename、 - (c) `primitive.zig::registerAll` の `referAll(rt, user)` と - `macro_transforms.zig::registerInto` の同様 site は「user/ ns - ergonomic refer」 として KEEP、 同じく非-PROVISIONAL に rename。 - Devil's-advocate (sub-cycle a) の Alt 2 full 採択 (= rt - auto-refer 全削除) が core.clj 全 rt 参照の mass-qualify を - 要する事実と整合しないため、 「auto-refer 自体は load-bearing で - finished form、 PROVISIONAL framing が間違っていた」 と再分類。 - これは Step 0.6 amendment A2 の延長線上にある分類 refresh で、 - smell sensor 上は finished form 修整 (Spec drift correction)。 - Devil's-advocate fork は本 D9 amendment では再起動せず、 - sub-cycle a fork の verdict (rt auto-refer 削除は scope 外) を - 根拠に inline 採択 — 再 fork は同じ verdict を返すと想定。 - Sub-cycle d commit が atomic discharge (= marker + yaml + - debt 三者同期は `check_provisional_sync.sh` hook が enforce)。 - -- 2026-05-26 sub-cycle 着地サマリ: - - sub-cycle a (9dc3a8e): D-071 Part 3 leaf re-home (ADR-0033 D4 - amend). - - sub-cycle b (7bac178): ADR-0035 draft + depth-3 fork. - - sub-cycle c.1 (f363342): Runtime infra (require_in_progress - + source_registry + require_resolver + 2 catalog Codes). - - sub-cycle c.2 (b9aa143): env.zig API (setAlias / referOne / - referAll private filter). - - sub-cycle c.3 (fb0f370): embeddedResolver in bootstrap.zig. - - sub-cycle c.4 (9727f44): `require` special form bare-symbol. - - sub-cycle c.5 (23d52d4): `:as` / `:refer` libspec + alias- - aware analyzeSymbol. - - sub-cycle c.6 (bacc2da): per-file SourceContext registry - hookup; D-058 closed. - - sub-cycle c.7 (58da6ff): `(ns ...)` analyzer special form. - - sub-cycle d (this commit): 11 PROVISIONAL markers removed + - 3 yaml flips + D-063 closed + Revision history amend (D2 + - D9 inline amendments). - -- 2026-05-26 D9 second amendment (Phase 7 entry T3) — widen - cw v1 `(:refer-clojure)` semantic to include rt namespace + - remove auto-refer blocks from evalInNs / op_in_ns + introduce - new opcode `op_ns_with_refer_clojure` per ADR-0036 dual- - backend parity contract. - - **Rationale**: D9 first amendment (sub-cycle d, commit a762ca9) - re-classified the rt + clojure.core auto-refer in evalInNs / - op_in_ns as "load-bearing convenience" — honest given that - core.clj uses rt primitives (`count` / `conj` / `reduce` / - `assoc`) unqualified throughout, so removing the auto-refer - would have required mass-qualifying ~50+ sites. But the - re-classification left an asymmetry: a reader of - `clojure.set/union`'s body using `reduce` could not grep - `referAll` to find where `reduce` becomes visible — it was - in `evalNs` Zig code, not the `.clj` source. The - 「規律がわからん」 shape the user flagged at Phase 7 entry - retrospective. - - **D9 second amendment** removes the asymmetry by: - - - **Widening the cw v1 `(:refer-clojure)` directive semantic** - to refer BOTH `clojure.core` AND `rt` namespaces into the - entering ns. `.clj` heads `(ns foo (:refer-clojure))` thus - install both refers explicitly via the analyzer special - form expansion. - - **Removing the rt + clojure.core auto-refer blocks** from - `evalInNs` (tree_walk) + `op_in_ns` (VM dispatch). User - `(in-ns 'foo)` becomes a naked ns switch; user must - explicitly `(refer 'rt)` or `(refer 'clojure.core)` or use - `(ns foo (:refer-clojure))`. - - **Fixing a pre-existing shape bug in `evalNs`**: at HEAD - pre-T3, evalNs's rt refer was unconditional while its - clojure.core refer was gated on `refer_clojure`. Post- - amendment both are gated on `refer_clojure`. - - **Adding a new VM opcode** `op_ns_with_refer_clojure` per - ADR-0036 parity contract. `compileNs` emits this when - `refer_clojure = true`; `compileInNs` continues to emit - bare `op_in_ns`. VM dispatch performs op_in_ns logic + - `referAll(rt, here)` + `referAll(clojure.core, here)`. - Same-commit retrofit lands tree_walk path + VM compile arm - + VM dispatch arm + ≥1 differential test case per - `.claude/rules/dual_backend_parity.md`. - - **Bootstrap fan-out (primitive.zig::registerAll + - macro_transforms.zig::registerInto + bootstrap.zig end-of- - loop refer of clojure.core into user/) stays** — user/ is - not a `.clj` file, so its initial refers must come from - boot-time setup. ADR-0033 D4 private leaf semantics in - clojure.core unaffected (private check is per-Var-ns, not - per-refer-source). - - **cw v1 divergence from JVM**: JVM has no `rt` ns; the - widened `(:refer-clojure)` semantic is cw-specific. JVM's - `(:refer-clojure)` only refers clojure.core (because there - is no rt to refer). Option (A) full rt elimination (move - all rt primitives to clojure.core; eliminate rt as user- - visible) is **deferred to Phase 10-11 entry** when Tier-A - polish provides budget for the ~50-site mass-qualify cost. - D9 second amendment is the F-002 finished form for Phase 7 - entry — the refer mechanism is fully regular without forcing - the rt-elimination prerequisite (per F-002 staged-finished- - form via ROADMAP §17). - - **VM-DEFER discharge**: T1 (ADR-0036) introduced the - `runtime/vm/ns_filter` VM-DEFER marker on - `_ = n.refer_clojure;` in `compileNs`. T3 consumes the - field by emitting `op_ns_with_refer_clojure`, discharging - that marker per the discharge discipline in - `.claude/rules/dual_backend_parity.md`. D-073 cluster sub- - site (e) closes; sites (a-d) remain (deftype/ctor_call/ - field_access pending row 7.6; require_libspec pending - ADR-0036 follow-up cycle). - - **Test impact** (per survey §6 corpus audit): all 5 - functional `(in-ns ...)` sites in the test corpus either - switch into a ns where the called Var is intern-local - (same-ns lookup, refer-independent), switch into user/ - (which has boot-installed refers), or use fully-qualified - names. No test relies on the in-ns auto-refer; zero - predicted breakage. - - **Devil's-advocate fork** (depth-2 mandatory, fresh context, - briefed against F-NNN envelope from - `.dev/project_facts.md`) verbatim: - - --- - - **F-NNN envelope verification**: F-002 (finished form wins), F-007 (chapter cadence dormant), F-009 (impl neutrality). No alternative below violates an F-NNN. F-002 is the active discriminator. - - ### Alt 1 — Smallest-diff (option C): keep behaviour, strengthen docstring only - - **Shape**: Leave `evalInNs` / `op_in_ns` auto-refer in place. Add a multi-line docstring at each site naming the asymmetry, citing ADR-0035 D9 first amendment, and explaining why the convenience is load-bearing. - - **Better than proposal**: Zero source surface change; zero new opcode; zero ADR-0036 parity surface; smell-audit depth 1 (docstring strengthening only). No test risk. Carries no `op_ns_with_refer_clojure` cost that has to be discharged later if option (A) lands at Phase 10-11. - - **Breaks**: This is the **Smallest-diff bias smell** by F-002's named definition — Alt 1 reaches a **different finished form** (asymmetry preserved as a permanent comment) than Alts 2 / 3 (asymmetry removed). F-002 §2 forbids picking smallest-diff *because it is smallest* when finished-form would diverge. The 「規律がわからん」 shape (triad §T3 L736) persists: a reader of `clojure.set/union` still cannot grep-trace the rt refer to its source — the docstring lives in Zig, not the `.clj` head. **Rejected** under F-002. - - ### Alt 2 — Finished-form-clean (option B as proposal, with refinements) - - **Shape**: Proposal as written. Three refinements to weigh: - - **Refinement 2a — opcode vs operand encoding**: The proposal adds `op_ns_with_refer_clojure` (distinct opcode). The alternative is `op_in_ns` with a flag-bit operand (`packed { name_idx: u24, do_refer: u1 }`). **Reject the flag-bit form**: ADR-0036 §D1 (T1's contract) treats each opcode as the unit of parity, and the exhaustive `switch (Opcode)` in `vm.zig` is the build-time presence enforcer. A flag-bit operand hides the second variant from the switch enforcement, regressing T1's compiler-enforced layer. Distinct opcode is correct. - - **Refinement 2b — migration warning on `(in-ns)`**: The proposal does not emit a deprecation notice. **Skip the warning**. cw v1 is pre-release; the test corpus (survey §6.1) shows zero breakage; emitting stderr noise for a non-issue is itself a **Silent default-shift smell** inversion (loud default-shift). Survey §6.1's evidence — every `(in-ns)` test site either same-ns-resolves, switches to user/, or uses qualified names — makes the warning gratuitous. - - **Refinement 2c — stage op_in_ns vs evalInNs removal**: Proposal removes both simultaneously. **Keep them coupled**. ADR-0036 §D1 makes TreeWalk + VM atomic for any backend-touching commit; staging would create a one-cycle window where TreeWalk `(in-ns 'foo)` is naked but VM `op_in_ns` still auto-refers, exposing a Layer-3 differential test divergence the new diff case would catch. Coupled removal honours the parity rule the same cycle exercises. - - **Better than proposal as written**: Documents the rejection of flag-bit / warning / staging so future readers do not re-relitigate. - - **Breaks**: Adds one opcode whose long-term home is Phase 10-11 option (A) elimination (where it gets retired alongside rt itself). That is acceptable — F-002 §3 endorses skeleton-then-rewrite when each skeleton shrinks the final rewrite. `op_ns_with_refer_clojure` becomes a no-op or a `referAll(clojure.core)` simplifier at Phase 10-11; cleanly retired. - - ### Alt 3 — Wildcard (option A): full rt elimination at Phase 7 entry - - **Shape**: Move ~30+ rt primitives into `clojure.core`; mass-qualify ~50 unqualified rt call sites in `core.clj` / `set.clj` / `string.clj` / `walk.clj`; delete the rt ns entirely from user-visible space; rewrite every `registerAll` site. - - **Better than proposal**: This is the **ultimate finished form** under F-002 §1 (the user explicitly named (A) as the cleanest end state, triad §T3 L739-743). After (A), `(:refer-clojure)` has its JVM meaning; no cw-specific divergence remains; no `op_ns_with_refer_clojure` is ever needed. Eliminates the asymmetry at the *namespace* layer, not the *refer-mechanism* layer. - - **Breaks**: Cost is 1-2 cycles plus high coordination risk (every `registerAll` + 4 `.clj` heads + analyzer ns-resolution + bootstrap fan-out + Phase 6.16.d shim audit). Surface-level: rt as a Zig-leaf-primitive home is structurally distinct from clojure.core (rt = Zig-implemented, cc = Clojure-or-Zig); collapsing them muddles the Layer-1 vs Layer-2 boundary that ADR-0033 D4 private-leaf semantics depend on. **F-002 §1 reading**: per ROADMAP §17, finished form may be reached in stages provided each stage lands a finished form within its own scope. Alt 2's `(:refer-clojure)` widening *is* a finished form for Phase 7 entry — the refer mechanism is fully regular — without forcing the rt-elimination prerequisite Tier-A polish provides budget for. **F-002-compliant** to defer (A); not F-002-compliant to never reach (A). Triad §T3 L760-762 user direction: defer. - - ### Recommendation - - **Alt 2 (refined)**. Distinct opcode (`op_ns_with_refer_clojure`); no migration warning; coupled `evalInNs` + `op_in_ns` removal; ADR-0035 in-place amendment with this Devil's-advocate verbatim. (A) deferred per user direction. The new opcode is a skeleton that shrinks at Phase 10-11 (F-002 §3 compliant). - - --- - - **Selected**: Alt 2 refined as recommended. T3 lands the - amendment + opcode + tree_walk evalNs gate fix + evalInNs/ - op_in_ns auto-refer removal + compileNs new emit path + - diff case + feature_deps.yaml updates + D-073 (e) - discharge in a single source commit, with this Revision - history amendment landing in a separate doc-first commit - per CLAUDE.md Step 6 depth-2 discipline. - - Affected files (T3 cycle): src/eval/backend/vm/opcode.zig - (new `op_ns_with_refer_clojure` = 0x17); src/eval/backend/ - vm.zig (new dispatch arm + remove auto-refer from - op_in_ns); src/eval/backend/vm/compiler.zig (compileNs - emits new opcode when refer_clojure=true; VM-DEFER on - `_ = n.refer_clojure;` removed); src/eval/backend/ - tree_walk.zig (evalNs gates rt refer on refer_clojure; - evalInNs removes auto-refer block); src/lang/diff_test.zig - (new diff case exercising widened semantic); feature_deps.yaml - (re-frame `runtime/eval/in_ns_auto_refer` notes; extend - `special_form/ns_macro` notes; discharge `runtime/vm/ns_filter` - status flip); `.dev/debt.md` (D-073 (e) discharge note). - -- 2026-06-13 D9 third amendment — **clojure.core overrides rt on the - auto-refer collision** (the refer-precedence revision). - - **Trigger**: core.clj's first redefinition of an rt-interned name — - `re-find` gained the matcher 1-arity wrapper (`([m] (when (.find m) - (re-groups m)))`, the java.util.regex.Matcher landing) — exposed a - latent structural bug: `referAll*` skips already-referred names - (first-wins) and every site refers `rt` BEFORE `clojure.core`, so the - public redefinition was unreachable unqualified in EVERY ns (user via - the boot fan-out, every library ns via `(ns … (:refer-clojure))`). - JVM semantics: clojure.core's def is what user code sees. - - **Decision**: add `Env.referAllOverriding` (same filter logic as - `referAllWithFilter`, but an existing refer of the same name is - REPLACED) and use it for the **clojure.core half only** of the - auto-refer pair at all 4 sites: `tree_walk.zig` evalNs, - `vm.zig` op_ns_with_refer_clojure + op_ns_with_filter, - `bootstrap.zig` finalizeUserNs. rt still refers first and fills - rt-only names; clojure.core overrides collisions. Explicit - `(refer …)` / `(require :refer …)` sites keep first-wins (JVM's - `Namespace.reference` replaces only core-originated mappings and - throws on other conflicts — uniform last-wins would mask genuine - user conflicts, an F-011 regression). - - **Discovery sweep** (framework_completion): `rg 'referAllWithFilter| - referAll\(' src/` enumerated every call site; the 4 clojure.core-half - sites flipped to overriding; the rt halves + explicit-refer sites - (primitive/namespace.zig ×4, tree_walk :require/:refer, vm :require/ - :refer, env.zig helpers) deliberately keep skip semantics. - - **Devil's-advocate fork** (fresh context, embedded per protocol): - - - **Alt 1 — order-swap (smallest-diff)**: refer clojure.core first, - rt second, keep skip-only. Better: no second refer mode. Breaks: - cannot cover the boot fan-out — `primitive.zig:152` + - `macro_transforms.zig:62` refer rt→user BEFORE clojure.core exists - (core.clj needs a working user ns to load), so finalizeUserNs still - skip-loses; the "smallest diff" grows into boot-order surgery or a - mixed-convention codebase. - - **Alt 2 — clojure.core as the SOLE auto-refer source - (finished-form-clean)**: boot-intern public rt vars into - clojure.core's mappings before core.clj loads (defs overwrite - in-place); rt demoted to qualified-only internal access; all sites - shrink to one `referAll(clojure.core)`. The collision class - vanishes; `:exclude` finally has a single source. This realises the - rt-elimination reservation this ADR already carries (deferred to - Phase 10-11 per the D9-second-amendment DA). Breaks: boot - restructure + per-var public-vs-internal curation + the - qualified-private question. **Recorded as the structural successor - → debt D-412** (anchored at the Phase 10-11 rt-elimination owner). - - **Alt 3 — merged "effective core" view (wildcard)**: one frozen - rt⊕core overlay map all auto-refer sites read. Breaks: a third - namespace-like entity; post-boot defs into clojure.core go stale — - a new bug class. Rejected. - - (b) de-interning colliding names from rt = a hand-maintained - collision list, a drift machine — rejected. (d) last-wins - everywhere = F-011 regression at explicit-refer sites — rejected. - - **DA verdict**: the chosen revision is sound — it encodes the - JVM-visible rule (public layer wins over internal) at all 4 sites - including the boot fan-out that defeats Alt 1, and is - behaviour-identical to what Alt 2 would produce, so it is a genuine - stepping stone, not smallest-diff bias. Alt 2 is the finished form; - it rides D-412 at the Phase 10-11 rt-elimination entry. - - **Affected files**: src/runtime/env.zig (referAllImpl + the - overriding variant), src/eval/backend/tree_walk.zig (evalNs cc - branch), src/eval/backend/vm.zig (both ns ops' cc branches), - src/lang/bootstrap.zig (finalizeUserNs), src/lang/diff_test.zig - (ns-override diff case locking both backends), .dev/debt.yaml - (D-412 structural successor row). diff --git a/.dev/decisions/0036_dual_backend_parity_contract.md b/.dev/decisions/0036_dual_backend_parity_contract.md deleted file mode 100644 index 95d9a77da..000000000 --- a/.dev/decisions/0036_dual_backend_parity_contract.md +++ /dev/null @@ -1,320 +0,0 @@ -# 0036 — Dual-backend parity contract (Phase 7 entry) - -- **Status**: Accepted (Devil's-advocate fork landed 2026-05-26) -- **Date**: 2026-05-26 -- **Author**: Shota Kudo (drafted with Claude autonomous loop) -- **Tags**: phase-7-entry, dual-backend, VM, tree-walk, parity, F-002, framework-completion - -## Context - -ADR-0005 declares dual-backend differential testing as the cw v1 -oracle for evaluator correctness: TreeWalk and VM must agree on -every source. ADR-0021 + ADR-0022 wire the differential test layer -(`src/lang/diff_test.zig`). F-002 makes "the finished form wins" -project law: dropping the dual-backend promise after the fact is -the Smallest-diff bias smell elevated to ADR scale. - -Reality at HEAD `0cd92fa` (Phase 7.1 dispatch ABI landing): - -- `vm/compiler.zig:94-130` switches exhaustively over the 24-variant - Node union — Zig surfaces a compile error if a new variant is - added without a corresponding arm. The *arm-presence* layer is - already enforced by the compiler. -- The *arm-body* layer is **not** enforced. 5 sites at HEAD ship - with arms that either raise `error.NotImplemented` (4 sites: - `deftype_node` / `ctor_call_node` / `field_access_node` triple - + `compileRequire` libspec branch) or silently drop a - significant field (1 site: `compileNs` discards - `n.refer_clojure`). The differential test layer covers only - 9 of 24 Node variants, so the gaps cannot trip a CI failure. -- cw v0 final state (v0.5.0) carried the same drift class — VM - arms shipped as "TODO Phase 7-10" and the catch-up consumed - weeks of late-cycle time. cw v1's discipline machinery - (PreToolUse hooks + Devil's-advocate fork mandate) was supposed - to prevent the same outcome, but no rule explicitly covered - "new analyzer Node → both backends in same commit," so the - drift recurred at Phase 6.16.b-4 sub-cycles c.4 + c.5 + c.7 - (RequireNode libspec + NsNode filter) and again at Phase 5.12.a - (deftype-family triple stub). - -D-073 at `.dev/debt.md:98` recorded one symptom of this drift -(`has_rest` VM gap) without naming the cluster. - -Source survey: `private/notes/phase7-T1-survey.md` (~440 lines, -HEAD `0cd92fa`). - -## Decision - -### D1 — Parity contract - -Every commit that adds a variant to the `Node` union in -`src/eval/node.zig` MUST also land, in the same commit: - -1. The TreeWalk eval arm in `src/eval/backend/tree_walk.zig`. -2. The VM compile arm in `src/eval/backend/vm/compiler.zig`. -3. The VM dispatch arm in `src/eval/backend/vm.zig` (when a new - opcode is introduced). -4. At least one differential test case in `src/lang/diff_test.zig` - exercising the new variant. - -The compiler-enforced exhaustive switch (`vm/compiler.zig`'s -switch over `Node` has no `else =>` arm) ensures **arm presence**. -This ADR adds the contract for **arm body discipline** + diff -coverage. - -### D2 — Allow-list marker `// VM-DEFER:` - -When a VM compile arm cannot land its real implementation in the -introducing cycle (typically because the bytecode shape depends on -a not-yet-landed design choice in a future Phase row), the arm may -ship as a transient stub (`return error.NotImplemented;`) or as a -documented silent-field-drop (`_ = n.some_field;`) provided the -line directly above carries: - -```zig -// VM-DEFER: [refs: D-NNN, feature_deps.yaml#] -``` - -The marker mirrors the `// PROVISIONAL:` shape in -`.claude/rules/provisional_marker.md`: - -- Single line. Multi-line rationale belongs in the `feature_deps.yaml` - entry body or the `.dev/debt.md` row. -- `[refs:` block mandatory; at least one `D-NNN` AND one - `feature_deps.yaml#` reference. -- Placed directly above the affected statement; no blank line - between marker and code. -- Removed (not commented-out) when the deferred implementation - lands. - -VM-DEFER is a sibling class to PROVISIONAL: same shape, different -concern. PROVISIONAL flags intermediate semantics that ride a -chicken-and-egg layer; VM-DEFER flags backend-asymmetry that rides -a Phase-N row's design choice. - -### D3 — Discovery + enforcement - -- Rule: `.claude/rules/dual_backend_parity.md` codifies the - contract + cross-references this ADR + the hook script. -- Hook: `scripts/check_dual_backend_parity.sh` (PreToolUse on - `git push`, registered in `.claude/settings.json`) blocks - pushes whose unpushed commit range contains a - `return error.NotImplemented` in `src/eval/backend/vm/*.zig` - or `src/eval/backend/tree_walk.zig` without an adjacent - VM-DEFER marker, OR a malformed VM-DEFER marker that lacks the - `[refs: D-NNN, feature_deps.yaml#]` block. - -### D4 — Retrofit in the same cycle - -Per `.claude/rules/framework_completion.md`, the introducing -cycle must run the discovery criterion against current HEAD and -retrofit each enumerated site. Sites at HEAD `0cd92fa`: - -| # | Site (`vm/compiler.zig`) | Body status | T1 disposition | -|---|--------------------------------------|--------------------------------------------------------|--------------------------| -| 1 | `deftype_node` arm (split from L115) | `return error.NotImplemented;` (triple-arm bundle) | Split + VM-DEFER (D-073) | -| 2 | `ctor_call_node` arm (split) | same | Split + VM-DEFER (D-073) | -| 3 | `field_access_node` arm (split) | same | Split + VM-DEFER (D-073) | -| 4 | `compileRequire` libspec branch L393 | `if (n.alias != null or n.refers.len > 0) return ...;` | VM-DEFER (D-073) | -| 5 | `compileNs` filter L380 | `_ = n.refer_clojure;` (silent field drop) | VM-DEFER (D-073) | - -All 5 sites mark with VM-DEFER referencing D-073 + a -`feature_deps.yaml#runtime/vm/` entry. D-073 is amended in -place from "`has_rest` VM gap" (single-site) to "VM backend -parity cluster" (5 sites + the original `has_rest` concern). - -Diff coverage gap: 15 of 24 Node variants are not exercised by -any `src/lang/diff_test.zig` case at HEAD. T1 lands 11 new diff -cases covering all non-deferred variants (`def_node`, `do_node`, -`quote_node`, `try_node`, `throw_node`, `in_ns_node`, -`require_node` bare form, `ns_node`, `vector_literal_node`, -`map_literal_node`, `set_literal_node`). The 5 VM-DEFER sites -do not require diff cases until their markers are discharged. - -### D5 — Scope of T1 vs follow-up - -T1 (this ADR's introducing cycle) lands the contract + 5 VM-DEFER -markers + 11 diff cases + rule + hook. T1 does **not** land any -real VM implementation for the 5 deferred sites. The first -real-feature exercise of this contract is the `require` libspec -impl (Alt 2-refined Devil's-advocate selection): a dedicated -follow-up cycle introduces `op_require_with_libspec` (or -equivalent encoding chosen via its own Devil's-advocate fork), -discharges the `runtime/vm/require_libspec` VM-DEFER, and adds -2 diff cases (bare + libspec). - -This split keeps T1 a pure discipline landing (smell-audit depth -2). Bundling the require-libspec opcode design would conflate -"land the contract" with "exercise the contract on a non-trivial -new opcode" and would push smell-audit to depth 3. - -## Alternatives considered - -The Devil's-advocate fork (fresh context, briefed against -F-NNN envelope from `.dev/project_facts.md`) produced 3 -alternatives. Embedded verbatim: - -### Alt 1 — Smallest-diff: retrofit-only, defer rule + hook + ADR - -**Shape**: Land only the 5-site retrofit (libspec impl + 4 VM-DEFER markers) + 11 diff cases. Defer ADR-0036 / rule / hook to "when row 7.6 adds MethodCallNode and forces the question." - -**Better than current draft**: ~450 LOC smaller, no hook-design risk surface (the hook may misfire on edge commits — e.g. cherry-pick chains, octopus merges; deferring lets a real second incident inform the matcher). Avoids Framework-incomplete risk on the *hook itself* (the hook is a new discipline whose own retrofit could be skipped in haste). - -**Breaks**: This is the Framework-incomplete smell (`principle.md`) verbatim. `framework_completion.md` §C explicitly demands rule + retrofit ride the same cycle. Without the hook, the next analyzer-Node-adding cycle (row 7.2 multimethod or row 7.6 MethodCallNode) re-enacts the exact drift T1 exists to prevent — the cw v0 catch-up trap (survey §8.6). **F-002 violation**: chooses smallest-diff *because it is smallest*, not because it lands finished-form. **Rejected.** - -### Alt 2 — Finished-form-clean (recommended, refined from proposal) - -**Shape**: Largely matches the proposal, with three refinements: - -1. **Defer `require` libspec impl to a follow-up cycle** alongside the rule + hook + ADR landing. Tag the existing libspec arm with `// VM-DEFER: require libspec pending op_require_with_libspec design [refs: D-073, feature_deps.yaml#runtime/vm/require_libspec]`. Rationale: bundling a new opcode design (operand encoding, heap Vector packaging, dispatch arm, 2 diff cases) into the T1 discipline-landing cycle conflates "land the contract" with "exercise the contract on a real payload." The proposal's reasoning ("require libspec is the only active user-visible gap, T2 needs it") is real but **T2's ADR-0035 D2 second amendment is itself a follow-up cycle**, not T2 proper — Symbol Value (T2) does not depend on require libspec VM parity, only T2's *optional* `require`-as-runtime-fn migration does. The libspec impl can land as the first exercise of ADR-0036 (cycle that proves the contract works on a real new feature). This keeps T1 as a pure discipline landing: rule + hook + markers + diff cases for **fully-working arms only**. - -2. **Single-commit cluster** (matches proposal): rule + hook + ADR + 5 marker retrofits + 11 diff cases in one Step 6 commit. `framework_completion.md` §C demands same-cycle; splitting into T1.a/T1.b violates the rule the cycle introduces. Smell-audit depth 2 is normal cw v1 practice (commits 8d4841c / a762ca9 set the precedent for multi-file structural cluster landings). - -3. **Hook stays at PreToolUse:Bash on `git push`** (matches proposal), not PreToolUse:Edit on node.zig. Rationale: the discovery criterion needs the *commit's full tree*, not a mid-edit view — Edit-time enforcement would block partial in-progress edits where the developer is mid-way through landing both arms. The push-time hook matches `check_provisional_sync.sh`'s shape and uses the proven `hook_iter_unpushed` walker. - -**Marker form**: `// VM-DEFER:` mirroring `// PROVISIONAL:` is correct — same `[refs: D-NNN, feature_deps.yaml#]` block, same single-line discipline. Differing surface name (`VM-DEFER` vs `PROVISIONAL`) preserves grep-distinguishability between "feature is intermediate semantics" (PROVISIONAL) vs "VM backend defers to tree_walk" (VM-DEFER) — these are different abstractions with different close-out predicates. - -**Diff coverage**: per-Node-variant requirement (proposal's choice) is correct. Per-commit-touching-backend would be looser and miss the case where a new Node variant lands without any new test (the precise gap that produced the 15 untested variants at HEAD). - -**Better than proposal**: cleaner separation between "contract lands" and "contract gets exercised on a non-trivial new opcode." The require libspec impl deserves its own Devil's-advocate (operand encoding choice between recommendation §3.1 options A/B/C is non-trivial — option A's `Vector` is the survey author's call but not subagent-verified). Bundling it forces a depth-3 decision into the depth-2 contract-landing. - -**Breaks**: leaves one user-visible gap (`(require '[clojure.set :as s])` still raises `NotImplemented` on `-Dbackend=vm`) for one additional cycle. Mitigated by VM-DEFER marker making the gap explicit + grep-discoverable + hook-tracked, plus the existing reality that VM backend is opt-in (`-Dbackend=vm`), so default users (tree_walk) see no change. - -### Alt 3 — Wildcard: kill VM backend - -**F-NNN violation finding (leading entry)**: **Alt 3 violates F-002.** ADR-0005 declares dual backend as the finished form; F-002 declares finished-form wins, and the Smallest-diff bias smell explicitly names "let me drop the hard-to-maintain thing post-hoc" as forbidden. Killing VM backend because parity discipline is expensive is the textbook Smallest-diff bias smell elevated to ADR scale. The main loop must not propose this alternative as a recommendation; it is recorded here per the Devil's-advocate brief's instruction to lead with F-NNN findings. - -**Shape (for the record)**: Delete `src/eval/backend/vm/`, delete `src/lang/diff_test.zig`, supersede ADR-0005 + ADR-0021 Layer 3 + ADR-0022, declare tree_walk as the only path until Phase 17 JIT lands. - -**Better than current draft**: eliminates the VM gap class permanently; saves ~3000 LOC of VM machinery; removes the entire dual-backend discipline cost from every future Node addition. - -**Breaks**: F-002 violation (above). Also: ADR-0005's "Phase 17 JIT joins the same comparison" extension premise dies; JIT activation later would need to re-build a comparison oracle from scratch. cw v0 retrospective evidence (survey §8.6) is that the VM backend itself is not the problem — *unenforced* parity discipline was. Alt 3 would solve a discipline gap by killing the discipline's target, which is structural surrender. - -**F-NNN compliance**: would require a user-direction F-NNN amendment to override F-002 + an explicit user direction to deprecate ADR-0005. Outside the loop's authority per `project_facts.md`. - -### Devil's-advocate recommendation - -**Alt 2 (refined)**: single-commit cluster landing rule + hook + ADR-0036 + 4 VM-DEFER markers (deftype / ctor_call / field_access / ns refer-clojure filter) + 11 diff cases + D-073 amendment in place. Defer **require libspec impl** to a dedicated follow-up cycle that becomes ADR-0036's first real-feature exercise (lands `op_require_with_libspec` + 2 diff cases + discharges that one VM-DEFER marker). - -F-NNN reasoning: F-002 forbids Alt 1 (smallest-diff bias) and Alt 3 (post-hoc surrender of dual backend finished form). F-009 keeps rule + hook outside `src/`. F-004 unaffected. F-007 unaffected. The refinement carves out require-libspec impl from T1 because the libspec opcode design is itself depth-3 territory (operand encoding choice) and deserves its own Devil's-advocate, not piggyback-acceptance under T1's contract-landing smell-audit. Net: T1 = pure discipline + 4 markers + 11 diff cases; require-libspec = one follow-up cycle that proves the contract works on its first new-feature exercise. - -## Selection rationale - -Selected: Alt 2 refined. - -- F-002: dual backend stays as the finished form; T1 mechanises - the protection. Alt 1 violates by deferring the discipline; - Alt 3 violates by abandoning the finished form. -- F-009: rule + hook live outside `src/`; VM compile arms remain - in `src/eval/backend/vm/`. No zone leak. -- F-004: T1 introduces no new heap value shape (the require - libspec opcode design that would touch encoding is carved out - to the follow-up cycle). -- F-007: chapter cadence dormant — no `docs/ja/learn_clojurewasm/` - touch. - -Devil's-advocate's carve-out of `require` libspec impl from T1 -is adopted: T1 = 5 VM-DEFER markers (not 4 markers + 1 impl). -This keeps T1's smell-audit at depth 2 and surfaces the opcode -encoding choice as its own depth-3 decision in the follow-up -cycle (where Devil's-advocate weighs Vector packaging -vs constant-pool sequencing vs op_invoke_builtin sub-chunk per -survey §3.1). - -## Consequences - -### Positive - -- Drift class eliminated mechanically. Future Node additions - trip the hook unless both backends + diff coverage land in the - same commit (modulo explicit VM-DEFER + tracked debt row). -- The 5 existing gaps become grep-discoverable + tracked. The - `_ = n.refer_clojure;` silent-drop pattern (the only true - silent-failure shape at HEAD) is now named, not buried. -- D-073 carries proper cluster scope; future Step 0.5 sweeps see - the full 5-site list rather than the single `has_rest` symptom. -- The 11 new diff cases close 11 of 15 untested-variant gaps in - `src/lang/diff_test.zig`, making the differential oracle a real - oracle for the bulk of the Node union. - -### Negative - -- Cycle cost ~1.5x vs a retrofit-only cycle (Alt 1). ~650 LOC - total change across ADR + rule + hook + markers + diff cases + - D-073 amendment + yaml entries. -- The hook adds one more PreToolUse:Bash gate to `git push` - latency. Self-test under `test/scripts/` keeps execution - bounded. -- `(require '[clojure.set :as s])` continues to raise - `error.NotImplemented` on `-Dbackend=vm` until the follow-up - cycle. Mitigated: VM backend is opt-in; default users see no - change. - -### Neutral / follow-ups - -- **Follow-up cycle A** (post-T1, when scheduled): `require` - libspec impl as ADR-0036's first real-feature exercise. - Introduces `op_require_with_libspec` (encoding TBD via its own - Devil's-advocate) + 2 diff cases; discharges - `runtime/vm/require_libspec` VM-DEFER. -- **Phase 7.6**: MethodCallNode landing discharges the 3 - deftype-family VM-DEFER markers (decides the dispatch-family - bytecode shape for deftype + ctor + field_access + method). -- **NsNode filter extension** (analyzer side, Phase 6.16.b-4 - follow-up or Phase 7+): when `:exclude` / `:only` directives - land, the VM `compileNs` arm picks up the filter encoding and - discharges the `runtime/vm/ns_filter` VM-DEFER. - -## Affected files - -- `.dev/decisions/0036_dual_backend_parity_contract.md` — this ADR (new). -- `.claude/rules/dual_backend_parity.md` — discipline rule (new). -- `.dev/principle.md` — adds "Dual-backend drift" Bad Smell entry. -- `.dev/debt.md` — amends D-073 in place from single-site - `has_rest` to 5-site cluster + `has_rest`. -- `feature_deps.yaml` — adds 3 entries: - `runtime/vm/dispatch_family`, `runtime/vm/ns_filter`, - `runtime/vm/require_libspec`. -- `src/eval/backend/vm/compiler.zig` — splits the deftype-family - triple arm into 3 separate arms; adds 5 VM-DEFER markers. -- `src/lang/diff_test.zig` — adds 11 new diff cases. -- `scripts/check_dual_backend_parity.sh` — PreToolUse hook (new). -- `scripts/hook_lib.sh` — used as-is, no edits required. -- `test/scripts/check_dual_backend_parity_test.sh` — hook - self-test (new). -- `.claude/settings.json` — registers the new hook in - PreToolUse:Bash chain. - -## References - -- ADR-0005 — dual-backend differential testing as oracle (this - ADR's foundational precedent). -- ADR-0021 — test taxonomy (Layer 3 differential). -- ADR-0022 — differential wiring. -- ADR-0008 amendment 1 — Phase 7.1 dispatch ABI (the row whose - landing surfaced the deftype-family triple stub as Phase 7 - territory). -- `.claude/rules/framework_completion.md` — discovery + sweep + - retrofit-in-same-cycle discipline this ADR operationalises. -- `.claude/rules/provisional_marker.md` — the PROVISIONAL marker - whose shape VM-DEFER mirrors. -- `.dev/project_facts.md` F-002 (finished-form wins), F-004 - (NaN-box layout), F-007 (chapter cadence dormant), F-009 (impl - neutrality). -- `private/notes/phase7-T1-survey.md` — Step 0 survey (~440 - lines) carrying the gap inventory + sweep result. -- `.dev/archive/phase7_entry_prereq_triad.md` §T1 — operational - driver (archived 2026-05-26 after triad completion per its - self-described lifecycle). - for this ADR's introducing cycle. - -## Revision history - -- 2026-05-26: Status: Proposed → Accepted (initial landing). - Devil's-advocate fork (depth-2 mandatory) executed against - F-NNN envelope; Alt 2 refined selected. T1 cycle lands this - ADR + rule + hook + 5 VM-DEFER markers + 11 diff cases as a - single-commit cluster per Devil's-advocate recommendation, - split into a doc-first commit (this file + rule + debt + yaml - + principle) followed by a source/hook commit per CLAUDE.md - Step 6 depth-2 discipline. diff --git a/.dev/decisions/0037_symbol_heap_value.md b/.dev/decisions/0037_symbol_heap_value.md deleted file mode 100644 index 63942b6c1..000000000 --- a/.dev/decisions/0037_symbol_heap_value.md +++ /dev/null @@ -1,313 +0,0 @@ -# 0037 — Symbol heap Value impl (F-004 Group A slot 1) - -- **Status**: Accepted (Devil's-advocate fork landed 2026-05-26) -- **Date**: 2026-05-26 -- **Author**: Shota Kudo (drafted with Claude autonomous loop) -- **Tags**: phase-7-entry, symbol, heap-value, F-004, interner - -## Context - -F-004 (in `.dev/project_facts.md`) reserved Group A slot 1 for -Symbol since the NaN-box second-generation landing (Phase 5, -ADR-0027). The slot has been declared at -`src/runtime/value/heap_tag.zig:56` day-1 (`symbol = 1`); the -module doc explicitly notes "behaviour-bearing call sites raise -`Code.feature_not_supported`" for declared-but-unwired tags. - -cw v1 read Symbol Forms from day 1 (`SymbolRef` at -`src/eval/form.zig:14-17`, `FormData.symbol`), but -`formToValue` at `src/eval/analyzer/analyzer.zig:631` raised -`feature_not_supported "Quoted symbol as Value"` on `.symbol` -Forms, forcing 5+ Phase 7 rows into special-form workarounds: - -- Phase 6.16.b-4 sub-cycle c.4 shipped `require` as an - **analyzer special form** rather than a runtime fn, - explicitly because `(require 'clojure.set)` would otherwise - fail at `formToValue` on the quoted symbol (ADR-0035 D2 - Revision history first amendment). -- Phase 6.16.c Group E shipped `macroexpand-all` as a - throw-stub for the same reason chain (Symbol Value gap → - reader-metadata gap → declare-only path unreachable). -- Phase 7 rows 7.3 defprotocol / 7.4 defrecord / 7.5 reify / - 7.7 extend-type / 7.8 multi-arity `fn*` would each need a - Symbol-Value workaround if T2 did not land. - -T2 (this ADR) lands the impl behind the existing reserved slot. -No F-004 amendment is needed; the slot was reserved precisely -so the impl could land at this natural cycle. - -T2 also unlocks the ADR-0035 D2 second amendment (`require` -migration from special form to runtime fn) — **out of T2 scope** -but tracked as a follow-up cycle. ADR-0036's dual-backend -parity contract (T1, just landed at HEAD `df79a38`) is satisfied -trivially: T2 adds no Node variant and no opcode; `analyzeQuote` -reaches `formToValue` identically on both TreeWalk and VM -backends. - -Source survey: `private/notes/phase7-T2-survey.md` (~808 lines, -HEAD `df79a38`). - -## Decision - -### D1 — Pointer-eq interning mirroring Keyword - -Symbol Values are interned by `(ns, name)` tuple in a registry -on `Runtime`. Two `'foo` literals from any source path produce -the same `*Symbol` pointer; equality via the existing -NaN-box-payload `u64` compare reduces to pointer-eq. This -matches Keyword's discipline (`src/runtime/keyword.zig`) and -gives every downstream caller (`identical?`, hash-map / set -key, multimethod dispatch cache, protocol satisfy cache) a -canonical-pointer fast path. - -### D2 — New file `src/runtime/symbol.zig` parallel to keyword.zig - -`Symbol` extern struct: `HeapHeader` + 6-byte pad + `ns: -?[]const u8` + `name: []const u8` + `hash_cache: u32`. -`SymbolInterner`: gpa-allocator + `std.array_hash_map.String(*Symbol)` -+ `std.Io.Mutex` (pre-wired for Phase 15 concurrency). Top-level -`intern(rt, ns, name)` + `find(rt, ns, name)` + `asSymbol(val)` -matching keyword's surface. No GC finaliser registration: -interner-owned + gpa-allocated; the `null` slot at -`tag_finaliser_table[symbol]` stays null (correct per keyword -precedent). - -Layer 0 (`src/runtime/`) placement per F-009 (impl neutrality). -A future `runtime/java/lang/Symbol.zig` surface or any Wasm -host-side symbol carrier would re-use the same `runtime/symbol.zig` -impl. - -### D3 — Runtime.symbols: SymbolInterner field - -Parallel to `Runtime.keywords: KeywordInterner`. Init/deinit -sibling. No NaN-box tag amendment (slot was day-1 reserved). - -### D4 — formToValue intern + printValue branch + nameFn extension - -- `analyzer.zig:631` `.symbol` arm: replace the raise with - `try symbol.intern(rt, sym.ns, sym.name)`. -- `print.zig` `printValue`: new `.symbol` arm between `.keyword` - and `.string`, rendering `(ns/)?name` with NO leading colon - (mirrors `form.zig:99-104` Form-side `pr-str`). -- `core.zig::nameFn`: add `.symbol` arm parallel to `.keyword` - returning `(name 'ns/x)` → `"x"`. Drop the "blocked on - F-004 symbol Value" docstring notes. - -### D5 — clojure.core/symbol constructor + symbol? predicate - -- `core.zig::symbolFn` (new): 1-arg - (string / keyword / symbol idempotent) + 2-arg - (string ns + string name). Mirror `keywordFn` shape exactly. - Register in `ENTRIES` table. -- `core.zig::symbolQ` (existing): already tag-dispatches - (`args[0].tag() == .symbol`), so no code change needed — it - silently returned `.false_val` for every Value before T2; - after T2 lights up automatically for interned symbols. - -### D6 — Defer per-Value metadata to D-075 - -Per-Value metadata (`^:dynamic`, `^:private`, user `with-meta`) -is **NOT** added to Symbol in T2. The cw v1 envelope frames -metadata as a cross-cutting concern via D-075 (Phase 7+ -metadata layer that lands meta on Symbol + Keyword + Var + -IObj-protocol Values uniformly), explicitly avoiding cw v0's -per-feature meta field which made the eventual migration -painful (`~/Documents/MyProducts/ClojureWasm/src/runtime/value.zig:295` -carried `meta: ?*const Value` on Symbol day 1). - -Within the current F-NNN envelope (metadata-less Symbol + -metadata-less Keyword), pointer-eq IS the finished form — -every interned Symbol is the canonical wrapper, no per-Value -metadata exists, so `=` and `identical?` agree by construction. -**No PROVISIONAL marker** required because Symbol Value runs -in its **finished form** today (the PROVISIONAL marker -discipline is for intermediate semantics; D6 is a deliberate -deferral of an orthogonal future concern, not intermediate -semantics). - -When D-075 lands, both Keyword and Symbol grow a -`meta_ptr: ?*Map` field in lockstep, plus the NaN-box decode -acquires a meta-flag bit (allocation TBD by D-075 itself). T2 -deliberately does **not** prepare for this — adding a day-1 -`meta_ptr: ?*Value = null` field is the Reservation-as-bias -smell (committing to D-075's eventual shape before D-075's own -Devil's-advocate has framed the decision; see Alt 2 refinement -rejection in Devil's-advocate output below). - -## Alternatives considered - -The Devil's-advocate fork (fresh context, briefed against -F-NNN envelope from `.dev/project_facts.md`) produced 3 -alternatives. Embedded verbatim: - -### Alt 1 — Smallest-diff: non-interned Symbol Value - -Allocate a fresh heap `Symbol` on every `formToValue(.symbol)`; `(= 'foo 'foo)` falls through to string-compare on `(ns, name)`; no `SymbolInterner`, no mutex, no `Runtime.symbols` field. - -- **Better than proposal**: ~80 LOC saving (no interner table + lock). Zero process-lifetime memory growth from repeated quote evaluation that the GC cannot reclaim (interner-owned Values pin until `Runtime.deinit`). -- **Breaks**: - - Pointer-eq is the load-bearing semantic everywhere downstream — `identical?`, hash-map / hash-set key comparison fast path, multimethod dispatch key cache, protocol satisfy cache. Each of these grows an `if (tag == .symbol) strcmp...` branch on the hot path. That is the **Cascade smell** + **Smallest-diff bias smell** in one move per principle.md. - - Diverges from Keyword's pointer-eq shape (`runtime/keyword.zig:101`), forcing every collection helper that today handles "interned identity Values uniformly" to split into a "keyword / symbol" two-case shape. F-002 finished-form lost. - - The "~80 LOC saving" is illusory: the interner code is verbatim mirror of keyword.zig, so the marginal author cost is ~10 minutes; the cross-cutting strcmp drift never goes away. - -**Reject** per F-002 (smallest-diff bias against a different finished form). - -### Alt 2 — Finished-form-clean (canonical pick, with two refinements) - -Pointer-eq interning mirror of Keyword exactly as the survey §3-§7 proposes. The refinements the brief asks about: - -- **Day-1 `meta_ptr: ?*Value = null` field on Symbol?** **Reject**. This is the Reservation-as-bias smell (committing to D-075's eventual shape before D-075's own Devil's-advocate fork has happened). Adding `meta_ptr` now forces (a) Keyword struct lockstep widening, (b) a meta-flag bit allocation in NaN-box encoding that D-075 has not yet framed, (c) equality semantic migration to per-Value before the metadata layer exists to justify it. Per survey §7.2 + §10 the deferral IS the finished form within the current envelope. -- **Share keyword's hash function?** Yes — use `hash.hashString` exactly as `keyword.zig:151-159` does. Diverging the hash function for symbol vs keyword has no measurable upside and breaks the "interned identity Value" symmetry. -- **Re-entrant-safe lock?** Stay with `lockUncancelable(rt.io)` per keyword precedent. Phase 15 concurrency rollout converts this uniformly; symbol diverging from keyword would be Framework-incomplete. -- **`.symbol` arm ordering in `printValue`?** Place between `.keyword` and `.string` per survey §5.1 — matches the heap_tag.zig declared order (string=0, symbol=1, keyword=2) which Zig's switch jump-table generation already optimises for. No re-ordering justification. - -The proposal-as-written matches finished form. The one mandatory addition beyond the brief: **open D-075 in the same commit** (or verify it exists; survey §10 + §12 step 10 already call this out). Without the row, the metadata deferral is the Defer-to-amnesia smell. - -### Alt 3 — Wildcard: defer Symbol Value past Phase 7 entry - -Push T2 to Phase 8 (transients), Phase 11 (self-hosting), or fold into D-075 (metadata layer, Phase 7+). - -- **Better than proposal**: ~1 cycle saved now; lets Phase 7 entry resume directly to row 7.2 (multimethod). -- **Breaks**: - - Forces 5 Phase 7 rows (7.3 defprotocol / 7.4 defrecord / 7.5 reify / 7.7 extend-type / 7.8 multi-arity fn*) plus the `require` runtime-fn migration into ADR-0035-D2-shaped analyzer-special-form workarounds. Each workaround needs its own ADR Revision history entry + its own un-do cycle when Symbol Value eventually lands. Cumulative cost is **substantially higher** than T2's ~150 LOC. - - F-004 reserved Group A slot 1 specifically so Symbol Value could land at the natural cycle (per F-004 declaration + heap_tag.zig:56 day-1 reservation). Deferring is not an F-004 violation per se (the slot stays declared, behaviour-bearing call sites raise `feature_not_supported` per the heap_tag.zig L17-18 contract), but it IS the Reservation-as-bias smell inverted — treating "we can defer because the slot will still be there" as a license to defer indefinitely. - - Folding into D-075 conflates two independent concerns: Symbol Value identity (a NaN-box / interner concern, scoped) and per-Value metadata (a cross-cutting equality-contract concern, large). D-075 becomes monolithic and harder to Devil's-advocate cleanly. - -**Reject** per F-002 (Progress-pressure smell: "let me skip T2 to move faster" produces 5x downstream catch-up cost) + cumulative-workaround analysis in survey §6. - -### Devil's-advocate recommendation - -**Alt 2 as proposed in the survey, with the explicit decision to defer per-Value metadata to D-075 (do NOT add `meta_ptr` to Symbol on day 1) and to open the D-075 debt row in the same commit if it does not already exist.** - -Rationale: F-002 picks Alt 2 over Alt 1 (smallest-diff bias against a different finished form) and over Alt 3 (Progress-pressure + Reservation-as-bias inverse). F-004's slot reservation is honored by implementing into the existing slot, not by inventing a new one (no Reservation-as-bias trigger). F-009 neutrality is preserved by landing impl in `src/runtime/symbol.zig` with `clojure.core/symbol` as a thin wrapper in `src/lang/primitive/core.zig`. ADR-0036 dual-backend parity contract is satisfied trivially (no Node variant, no opcode added; `analyzeQuote` reaches `formToValue` identically on both backends — survey §11.5). - -## Selection rationale - -Selected: Alt 2 as-is (verbatim Devil's-advocate recommendation). - -- F-002: pointer-eq + interner + hash_cache + mutex pre-wired - = finished form within the metadata-less envelope. Alt 1's - cross-cutting strcmp drift breaks finished form; Alt 3's - deferral compounds workaround cost. -- F-004: slot 1 already declared; T2 lands the impl behind - the existing reservation (= honors the reservation, not - biased by it). -- F-009: `runtime/symbol.zig` neutral; `clojure.core/symbol` - wrapper lives in `lang/primitive/core.zig`. No cross-surface - import. -- F-007: no chapter cadence touch. -- D-075 verified present in `.dev/debt.md` at HEAD `df79a38` - (row name "Phase 7+ (value metadata system)") — the - metadata deferral has its tracked debt row. - -The "day-1 `meta_ptr` field" Alt 2 refinement is explicitly -rejected per D6 (Reservation-as-bias smell — committing to -D-075's shape pre-empts its own Devil's-advocate). - -## Consequences - -### Positive - -- 5 Phase 7 rows (7.3 / 7.4 / 7.5 / 7.7 / 7.8) gain a working - prerequisite — symbol parsing in defprotocol / defrecord / - extend-type lands without ADR-0035-D2-shaped workarounds. -- ADR-0035 D2 second amendment (`require` runtime fn - migration) becomes possible (out of T2 scope; tracked as - follow-up). -- `(quote sym)` evaluates; `(= 'foo 'foo)` returns `true` - via pointer-eq; `(name 'ns/x)` returns `"x"`; - `(symbol "foo")` round-trips; `(symbol? 'foo)` returns - `true` (auto-lit via tag-dispatch). -- `macroexpand-all` Pattern A path becomes possible alongside - T3. -- `gensym` user-facing runtime fn becomes possible (returns - a fresh Symbol Value rather than an arena `[]const u8`). -- `runtime/symbol.zig` becomes the neutral home a future - `runtime/java/lang/Symbol.zig` host-surface or zwasm - symbol carrier would re-use (F-009). - -### Negative - -- ~150 LOC new code (`runtime/symbol.zig` mirror + wiring + - tests + ADR + ~100 LOC e2e+diff). Smell-audit depth 2 (new - ADR + new file + multi-file wiring). -- Interner-owned Symbol Values pin until `Runtime.deinit` - (process-lifetime). Acceptable per keyword precedent; - identity-bearing Values are not GC candidates. -- ADR-0035 D2 first amendment's `require`-as-special-form - remains active until the follow-up cycle migrates it. - -### Neutral / follow-ups - -- **Follow-up cycle A** (post-T3): `require` migration from - analyzer special form to runtime fn per ADR-0035 D2 second - amendment (now possible). -- **D-075 (Phase 7+)**: metadata layer lands `meta_ptr` on - Symbol + Keyword + Var + IObj Values uniformly. Symbol - struct widens in lockstep. -- **`(namespace 'ns/x)` primitive**: not in T2 scope; added - in a separate minor cycle when the broader Symbol / - Keyword surface lands. -- **Quoted vector / map / set Value lifting**: the other 3 - `formToValue` raise sites (D-080 clojure.zip block) — not - blocking Phase 7 rows, separate concern. - -## Affected files - -- `.dev/decisions/0037_symbol_heap_value.md` — this ADR (new). -- `src/runtime/symbol.zig` — new file (~150 LOC; verbatim - mirror of keyword.zig). -- `src/runtime/runtime.zig` — add `symbols: SymbolInterner` - field + init/deinit sibling to `keywords`. -- `src/runtime/print.zig` — add `.symbol` arm in `printValue` - (between `.keyword` and `.string` arms). -- `src/eval/analyzer/analyzer.zig` — replace `.symbol` raise - arm in `formToValue` with `try symbol.intern(rt, sym.ns, - sym.name)`. -- `src/lang/primitive/core.zig` — add `symbolFn` (1-arg + - 2-arg); extend `nameFn` with `.symbol` arm; register - `symbol` in `ENTRIES`; drop "blocked on F-004 symbol Value" - docstring notes. -- `src/main.zig` — add `_ = @import("runtime/symbol.zig");` - to the `test {}` aggregator (per `zig_tips.md` test - discovery trap). -- `test/e2e/phase7_symbol_value.sh` — new (smoke for - `(quote sym)` / `(symbol "foo")` / `(= 'foo 'foo)` / - `(name 'ns/x)` / `(symbol? 'foo)`). -- `src/lang/diff_test.zig` — new differential case - exercising `(quote sym)` round-trip (ADR-0036 active - exercise of parity contract; tree_walk + VM agree because - both reach `formToValue` via `analyzeQuote`). - -## References - -- F-004 (`.dev/project_facts.md`) — Group A slot 1 = symbol. -- ADR-0027 — NaN-box second generation (declared the slot). -- ADR-0035 D2 first amendment — shipped `require` as special - form because Symbol Value was missing. T2 unblocks the - second amendment. -- ADR-0036 — dual-backend parity contract (T1, just landed). - T2 satisfies trivially (no Node variant, no opcode). -- `.claude/rules/provisional_marker.md` — Skeleton vs - PROVISIONAL discipline; T2's pointer-eq is **finished - form** within current envelope, not PROVISIONAL. -- `.dev/debt.md` D-075 — Phase 7+ metadata layer (the - deferred concern D6 names). -- `private/notes/phase7-T2-survey.md` — Step 0 survey (~808 - lines). -- `.dev/archive/phase7_entry_prereq_triad.md` §T2 — operational - driver (archived 2026-05-26 post-triad completion). - driver for this cycle. -- `src/runtime/keyword.zig` — verbatim template (303 lines). - -## Revision history - -- 2026-05-26: Status: Proposed → Accepted (initial landing). - Devil's-advocate fork (depth-2 mandatory) executed against - F-NNN envelope (F-002 + F-004 + F-007 + F-009 + D-075 - deferral); Alt 2 selected as-is (refinement to add - `meta_ptr` rejected per D6 Reservation-as-bias). T2 cycle - lands this ADR alone in the doc-first commit (depth-2 - discipline); the source landing (`runtime/symbol.zig` + - wiring + tests) follows in the next commit per CLAUDE.md - Step 6. diff --git a/.dev/decisions/0038_analyzer_def_pre_register.md b/.dev/decisions/0038_analyzer_def_pre_register.md deleted file mode 100644 index 5a6bb5d28..000000000 --- a/.dev/decisions/0038_analyzer_def_pre_register.md +++ /dev/null @@ -1,241 +0,0 @@ -# 0038 — `analyzeDef` pre-registers Var at analyze time - -- **Status**: Accepted (Devil's-advocate fork landed 2026-05-26) -- **Date**: 2026-05-26 -- **Author**: Shota Kudo (drafted with Claude autonomous loop) -- **Tags**: analyzer, def, var-binding, jvm-parity, F-002, F-009, - D-084 (discharge) - -## Context - -`src/eval/analyzer/special_forms.zig::analyzeDef` historically built -a `DefNode` without interning the Var into the active namespace. -The actual `env.intern(ns, name, root, null)` call happens lazily at -`tree_walk.zig:466-475::evalDef` (and at VM runtime via the -`op_def` dispatch arm). - -This "lazy intern" shape — Var resolution happens only after the -preceding `def` form has *evaluated* — produces two user-visible -breakages versus Clojure JVM Compiler semantics: - -1. **Recursive `defn` fails at analyze time**: - `(defn f [n] (if (= n 0) 0 (f (- n 1))))` raises - `name_error [analysis]` on the inner `f` reference. The recursive - body's `f` is analyzed before the outer `def`'s value has been - reached at runtime; without an analyze-time pre-register, the - symbol is unbound. -2. **Forward references inside top-level `(do ...)` fail**: - `(do (def a 1) (def b a))` raises `name_error` on the second `a`. - `analyzeDo` walks its forms sequentially through `analyze(...)`; - the first sub-form's `DefNode` is built but no env binding has - been installed by then. - -Row 7.3 cycle 7 surfaced the issue through `defprotocol`'s macro -emission. The natural lowering of `(defprotocol P (m [x]))` is - -```clojure -(do - (def P (rt/__make-protocol! 'P ['m])) - (def m (rt/__make-protocol-fn! P "m"))) -``` - -which trips breakage #2. Cycle 7.1 (commit `99461f8`) shipped a -truncation workaround — emit only `(def P ...)` and defer the -per-method-Var binding to D-084 closure — and tracked the gap as -debt row D-084. - -This ADR discharges D-084 by changing `analyzeDef`'s contract to -match Clojure JVM Compiler's: the Var lands in `env` at analyze -time, with placeholder `nil` root; runtime evaluators (TreeWalk's -`evalDef`, VM's `op_def`) re-intern with the actual evaluated value. -`env.intern` at `env.zig:353-357` is documented idempotent — -update-in-place when the name is already bound — which lets the -re-intern at runtime overwrite the placeholder cleanly. - -## Decision - -In `analyzeDef`, after the existing symbol-shape validation and -**before** the recursive `analyze(...)` call on the value form, -intern the Var with `nil` root: - -```zig -const ns = env.current_ns orelse - return error_catalog.raiseInternal(form.location, "def: no current namespace"); -_ = try env.intern(ns, name_sym.name, .nil_val, null); -const value_node = if (items.len == 3) - try analyzer_mod.analyze(arena, rt, env, scope, items[2], macro_table) -else - try analyzer_mod.makeConstant(arena, .nil_val, items[1]); -// ... DefNode build as before ... -``` - -This unblocks all three cycle-7.1 dependents: - -- Recursive `defn`: `(defn f [n] ...)` body now resolves `f` at - analyze time because the placeholder Var was just interned. -- Forward refs inside `(do ...)`: `(do (def a 1) (def b a))` works - because `a` is interned before `(def b a)` is analyzed. -- `defprotocol` per-method-Var binding: `(do (def P ...) (def m P - ...))` reverts to the natural shape. Cycle 7.1's macro - truncation rolls back in the same source commit (= cycle 8.1 - re-emits the full method-Var-binding shape). - -### "Var exists but root is nil" window - -A consequence of the chosen finished form is an observable window: -between `analyzeDef` and `evalDef`, the Var is resolvable in `env` -but its `root` is `.nil_val`. The window is: - -- **Invisible today**: cw v1 evaluates analyze-then-eval back-to-back - inside the same `cljw -e` / file invocation; no introspection path - exists between the two passes. -- **JVM-parity**: Clojure JVM Compiler also interns the Var at parse - time. A failed value-expression analysis there too leaves the Var - interned at its previous root (or nil for a first-time def). -- **Documented for future**: live REPL streaming or lazy-compile - paths (Phase 10+) would need to handle this window. The same is - true on the JVM; cw v1 inherits the standard solution shape. - -### Error semantics - -If `analyze(value_expr)` raises after the pre-register, the -placeholder Var stays interned. On the next REPL form, the Var is -visible with root `.nil_val`. This matches JVM Clojure: a failed -`def` body leaves the namespace's Var binding at whatever value -the previous body set (or nil for a first-time def). cw v1 adopts -the same behaviour rather than introducing a rollback path. - -## Alternatives considered - -**Devil's-advocate fork (depth-2, fresh context) verbatim -embedding** — produced 3 alternatives within F-002 / F-009 + -ADR-0036 envelope (full text: -`private/notes/phase7-7.3-cycle8.1-devils-advocate.md`): - -> **Alt 1 — smallest-diff (status quo + workaround macros)** -> **Shape**: `analyzeDef` stays unchanged. Recursive `defn` is -> handled by macro expansion — `defn` rewrites to a `(letfn [(f -> [...] ...)] (def f f))`-style shape, where `letfn` introduces -> the self-binding at analyze time via a local Scope rather than -> at the Var layer. `defprotocol` keeps the cycle-7.1 truncation -> indefinitely. -> **Verdict within F-NNN**: F-002 disfavours — smallest-diff bias -> smell. The finished form on JVM is "the Var is interned before -> the body is analyzed"; status quo refuses this and pushes the -> chicken-and-egg to a macro layer that has no clean general -> solution for the cross-`def` reference case. F-009 neutral. -> -> **Alt 2 — finished-form-clean (env.intern at analyze)** -> **Shape**: as described above. `analyzeDef` calls -> `env.intern(ns, name_sym.name, .nil_val, null)` after symbol -> validation, before recursing into the value form. Idempotence -> at `env.zig:353-357` covers the runtime re-intern. -> **Verdict within F-NNN**: F-002 strongly favours — matches JVM -> Clojure Compiler's `DefExpr.parse` exactly. "Var exists but root -> is nil" window is a known consequence of the chosen finished -> form (JVM has it too) — not a cw v1 defect. ADR-0036 obligation -> is tractable (≥1 diff case in the same commit). F-009 neutral- -> to-favouring (analyzer's role as Var-binding-lifecycle owner is -> strengthened). -> -> **Alt 3 — wildcard (declare special form)** -> **Shape**: Introduce explicit `(declare name)` special form. -> Recursive `defn` requires `(declare f) (defn f ...)`. -> `defprotocol` macro emits `(do (declare P) (declare m1) ... -> (def P ...) (def m1 ...))`. -> **Verdict within F-NNN**: F-002 disfavours — user-visible -> divergence from JVM Clojure (recursive `defn` requires `declare` -> two-step). If `defn` macro masks via implicit `(declare)` -> emission, the `defn` macro becomes the analyzer-impurity surface -> — worst of both worlds. F-009 neutral but adds a new analyzer -> Node variant (more ADR-0036 parity work). -> -> **No hard F-NNN violation surfaced** across any alternative; -> findings are F-002-leaning. Recommendation: Alt 2. - -**Selected**: Alt 2. The Devil's-advocate's F-002 reading is the -load-bearing rationale: cw v1's lazy-intern is a smallest-diff -omission, not a design choice. The finished form (matching JVM -Compiler's `DefExpr.parse`) lands in this cycle. - -Alt 1 rejected per the smallest-diff bias finding: it calcifies the -cycle-7.1 workaround into the steady state and produces a *larger* -overall surgery (`letfn` shim in every `defn`) than touching -`analyzeDef`. Alt 3 rejected per the F-002 visibility finding: it -forces user-visible divergence from JVM Clojure or hides it via a -`defn` side-effect. - -## Consequences - -- **D-084 discharges in the same source commit as this ADR - amendment lands.** The cycle-7.1 `defprotocol` macro truncation - rolls back; per-method-Var binding works. -- **`(defn f [n] (... (f ...)))` works.** Recursive functions - become a first-class cw v1 surface — previously requiring - `loop` / `recur` exclusively, now supporting the natural shape. -- **`(do (def a 1) (def b a))` works.** Forward references inside - top-level `(do)` blocks unblock; the conformance porting path - (Phase 11+) inherits the JVM-equivalent semantics. -- **New diff cases land in the same commit per ADR-0036.** Two - cases minimum: `def_recursive_reference` and - `def_forward_in_do` exercising the analyze-time pre-register. -- **No dual-backend asymmetry.** Both TreeWalk and VM observe the - same runtime intern behaviour (analyzeDef's pre-register is - analyze-only; runtime intern paths are unchanged in both - backends). -- **No new Node variant, no new opcode.** ADR-0036 parity - obligations are minimal. - -## Affected files - -- `src/eval/analyzer/special_forms.zig` — `analyzeDef` body. -- `src/lang/macro_transforms.zig` — `expandDefprotocol` reverts - the cycle-7.1 truncation, emits full `(do (def P ...) (def m - P ...))` shape. -- `src/lang/diff_test.zig` — new differential test cases. -- `.dev/debt.md` — D-084 row moves to `## Discharged` (or status - flips to `landed`). -- `test/e2e/phase7_protocol.sh` — added cases exercising - per-method-Var binding (`(defprotocol IPing (ping [this])) (ping - ...)` — the call path still needs the `.protocol_fn` arm in - `vtable.callFn`, landing in cycle 8.2; this ADR's e2e covers - the macro-emission path only). - -## References - -- F-002 (`.dev/project_facts.md`) — finished-form cleanliness wins. -- F-009 (`.dev/project_facts.md`) — feature-implementation - neutrality. -- ADR-0036 — dual-backend parity contract. -- ADR-0008 amendments 1-3 — protocol dispatch evolution (cycle 7.1 - surfaced this issue). -- `private/notes/phase7-7.3-cycle8.1-devils-advocate.md` — full - Devil's-advocate output. -- D-084 (`.dev/debt.md`) — the debt row this ADR discharges. - -## Revision history - -- 2026-05-26: Status: Proposed -> Accepted (initial landing). -- 2026-06-01 (amendment, D-184): **pre-registration becomes _declare_, never - _reset_.** The original implementation pre-registered via `env.intern(ns, - name, .nil_val, null)`, which on an existing Var did `existing.root = - nil_val` — RESETTING a prior root at analyze time. That contradicted this - ADR's own "a failed `def` body leaves the Var at the previous value" intent - (a throwing re-`def` wiped the old root before the body ran) and blocked - `defmulti`'s defonce-style re-eval no-op (D-184). Fix: a new - `env.internDeclare(ns, name)` (register-if-absent, root untouched if - present) replaces the `intern(...nil...)` call in `analyzeDef` + - `analyzeDefmacro`. Resolvability (the pre-register's sole purpose — - recursive defn / forward refs) only needs the Var to EXIST, not to be nil; - `evalDef`/`op_def` set the value at eval time as before. Uses `ns.mappings` - (local only, NOT `resolve` — a refer'd name must not suppress a shadowing - local def). Consequence: `(def x 5) (def x (/ 1 0)) x` → 5 (JVM parity), and - `__make-multifn` can return the existing MultiFn so re-`defmulti` keeps its - `defmethod`s. A `general-purpose` Devil's-advocate fork (fresh context, - F-002/F-011) confirmed this is a **spec-drift fix, not a contract change** - (the ADR already claimed JVM error semantics the code violated), walked the - 5 def cases (recursive / forward-ref / refer-shadow / plain-redef / throwing - -redef) as all correct, and rejected the back-door-`__defmulti!` wildcard as - a Workaround smell. Tests: 4 e2e (phase14_redef) + phase7_multimethod case - 6; `--compare` OK on both. Blast radius = every def/defn re-eval, gated by - the 191-test suite + dual-backend diff oracle. diff --git a/.dev/decisions/0039_reified_instance_minimal_layout.md b/.dev/decisions/0039_reified_instance_minimal_layout.md deleted file mode 100644 index 0f8735b4b..000000000 --- a/.dev/decisions/0039_reified_instance_minimal_layout.md +++ /dev/null @@ -1,205 +0,0 @@ -# 0039 — `ReifiedInstance` minimal layout (drop reserved closure_* fields) - -- **Status**: Accepted (Devil's-advocate fork landed 2026-05-26) -- **Date**: 2026-05-26 -- **Author**: Shota Kudo (drafted with Claude autonomous loop) -- **Tags**: type-descriptor, reify, closure-capture, F-002, F-003, - F-006, layout-decision - -## Context - -`src/runtime/type_descriptor.zig::ReifiedInstance` (extern struct) was -reserved at row 7.5 entry by ADR-0007's TypeDescriptor reservation -table with three fields beyond `header` + `descriptor`: - -```zig -pub const ReifiedInstance = extern struct { - header: HeapHeader, - closure_count: u32, - _pad: [4]u8 = .{ 0, 0, 0, 0 }, - descriptor: *const TypeDescriptor, - closure_bindings_ptr: [*]Value, - ... -}; -``` - -The reservation anticipated that reified instances would carry -closed-over locals directly on the heap struct (mirroring JVM -Clojure's `Compiler.java::NewInstanceExpr` synthetic-class layout, -where final fields hold captured locals). - -Row 7.5 Step 0 survey (`private/notes/phase7-7.5-survey.md` §4) -discovered that **closure capture is already discharged by `fn*`'s -`allocFunction` snapshot** (`src/eval/backend/tree_walk.zig` -L177-L206). The reify macro lowers method bodies to `(fn* [this] -body)`, and each method-Value already carries its closure_bindings -when `__reify!` sees it at primitive-evaluation time. Storing the -same bindings a second time on `ReifiedInstance` would be either -**dead code** (one site shadows the other and the second site is -never read) or **silent-shadowing** (dispatch reads from the wrong -site). - -The reservation is now a memo against a need that does not -materialise. Per F-002 (finished-form cleanliness wins) and the -project's Reservation-as-bias smell, we drop the fields rather than -freeze them with a `PROVISIONAL:` marker that points at no upstream -gate (which would collapse into a permanent no-op per -`.claude/rules/provisional_marker.md` row 4). - -## Decision - -Shrink `ReifiedInstance` to the minimum that the dispatch ABI needs: - -```zig -pub const ReifiedInstance = extern struct { - header: HeapHeader, - descriptor: *const TypeDescriptor, - - comptime { - std.debug.assert(@alignOf(ReifiedInstance) >= 8); - std.debug.assert(@offsetOf(ReifiedInstance, "header") == 0); - std.debug.assert(@sizeOf(ReifiedInstance) == 16); - } -}; -``` - -Two cache-word size (16 bytes), naturally 8-aligned. Header at offset -0 (GC walker invariant). Descriptor at offset 8 — back-pointer to a -`*const TypeDescriptor` that itself lives on `rt.gpa` (process -lifetime, never traced). - -GC hooks for `.reified_instance`: - -- **Trace fn**: no-op. No Value fields on the struct; the descriptor - pointer is to `rt.gpa`-owned memory which is never GC-reachable. -- **Finalise fn**: no-op. No `gc.infra`-owned tail array (unlike - `TypedInstance` whose `field_values_ptr` slice needs releasing). - -The `allocReifiedInstance(rt, descriptor)` helper allocates the -struct on `rt.gc.alloc` and returns a `.reified_instance`-tagged -Value via `Value.encodeHeapPtr`. - -The dispatch ABI at `src/runtime/dispatch.zig` L139-L142 grows a -parallel `.reified_instance` arm reading `inst.descriptor` (same -offset-8 layout pattern as `TypedInstance`). - -## Consequences - -### Positive - -- F-002: finished form ships, not a reservation-frozen intermediate. -- F-003: per-instance metadata slot (D-075 IObj semantics — JVM - `reify` always implements IObj) is deferred to D-075's owner. D-075 - will add the slot to `TypedInstance` AND `ReifiedInstance` - symmetrically at that point, not asymmetrically pre-empt here. -- F-006: GC trace surface shrinks. `traceReifiedInstance` is a no-op - function; the mark phase pays zero cost for reified instances. - Cache line behaviour also benefits (16-byte struct fits 4-per-cache-line - on a 64-byte line). -- ADR-0007 reservation table is "honored by deletion" — the slot - existed, we considered it, the finished form does not need it. - An amendment paragraph (this ADR) records the deletion. - -### Negative - -- Future feature wanting per-instance metadata on reified instances - must add the slot then (D-075 owner). Trade-off: one focused - metadata cycle adds slots to both `TypedInstance` and - `ReifiedInstance` symmetrically vs. two scattered partial - pre-emptions. F-003's structural-deferral principle prefers the - former. - -### Neutral - -- F-004 (NaN-box slot economy): no Value tag change; `.reified_instance` - slot stays at 27. -- F-009 (feature-implementation neutrality): the alloc helper lives - in `runtime/type_descriptor.zig` (Layer 0), wrapped by the - `__reify!` primitive (Layer 2) — same shape as `TypedInstance` + - `allocInstance`. - -## Alternatives considered - -(Devil's-advocate fork output reflected verbatim, 2026-05-26) - -### Alt 1 — Smallest-diff: keep + mark PROVISIONAL - -Retain the current `closure_count` + `_pad` + `closure_bindings_ptr` -fields, add a `PROVISIONAL:` marker pointing at a new debt row. - -**Better than current**: zero edit to existing layout; ADR-0007 -reservation table needs no amendment. - -**Breaks**: This is the **Reservation-as-bias smell** verbatim -(F-002 ¶4). The `PROVISIONAL:` marker would point at no real -upstream feature gating discharge — `fn*` closure capture already -works; there is nothing to wait for. That collapses the third -PROVISIONAL row into the fourth (permanent no-op) per -`provisional_marker.md`. Also forces a GC trace decision (do we -walk `closure_bindings_ptr` or not?) for fields nothing writes to -— pure cognitive tax. - -**F-NNN impact**: violates F-002 (smallest-diff bias); F-004 neutral -(no slot change); F-006 negative (trace surface carries dead Value -slice); F-003 neutral. - -### Alt 2 — Finished-form-clean (Survey §4 Option A) — SELECTED - -See Decision section above. - -### Alt 3 — Wildcard: single `meta` Value slot now - -Pre-allocate one `meta: Value = nil_val` slot in anticipation of -D-075's IObj semantics. - -**Better than others**: Pre-allocates the slot D-075 is statistically -certain to want (JVM's `reify always implements IObj` semantics are -documented; survey §1 ¶4 names it). Saves one struct-layout -migration later. - -**Breaks**: Pre-empts D-075's decision asymmetrically -(`TypedInstance` does **not** get the same field in this cycle, so -the two diverge). That asymmetry is the bug D-075 would have to -undo on landing. PROVISIONAL marker has no upstream gate (`fn*` is -already done; D-075 is downstream — markers point at *upstream -blockers*, not *downstream consumers*), so the triad would be -misapplied per `provisional_marker.md` §"Boundary rules". - -**F-NNN impact**: weakly violates F-003 (pre-deciding what D-075's -owner should decide); F-002 ambiguous (cleaner than Option 1, less -clean than Option 2); F-006 slightly negative vs Option 2 (one -extra Value to trace); F-004 neutral. - -### DA recommendation - -Lead with Option 2. Survey §4 already converged on it; F-002 + F-003 -+ F-006 all point the same direction; the only argument against -Option 2 is "we'll re-add a slot in D-075", which is exactly the -kind of pre-emption F-003 forbids. - -## Affected files - -- `src/runtime/type_descriptor.zig` — shrink `ReifiedInstance` struct - (delete `closure_count` / `_pad` / `closure_bindings_ptr` + the - `closure()` accessor), add `allocReifiedInstance` helper, add - `traceReifiedInstance` / `finaliseReifiedInstance` no-op hooks, - extend `registerGcHooks` to register them. -- `src/runtime/dispatch.zig` — extend the descriptor-resolution arm - at L139-L142 to handle `.reified_instance` alongside - `.typed_instance`. -- `.dev/decisions/0007_type_descriptor.md` — amendment paragraph - recording the reservation deletion (cross-link to this ADR). - -## References - -- `.claude/rules/provisional_marker.md` — Skeleton vs transient stub - vs PROVISIONAL vs permanent no-op (the boundary table this ADR - honours). -- `.dev/principle.md` — Reservation-as-bias smell entry. -- `.dev/project_facts.md` — F-002, F-003, F-006. -- `private/notes/phase7-7.5-survey.md` §4 — full struct-layout - design space. -- ADR-0007 — TypeDescriptor reservation table this ADR amends. -- ADR-0009 — heap-value-only lock semantics (no interaction; reified - instances carry no per-instance lock per the same rule that - applies to `TypedInstance`). diff --git a/.dev/decisions/0040_vm_method_dispatch_opcodes.md b/.dev/decisions/0040_vm_method_dispatch_opcodes.md deleted file mode 100644 index 34b740382..000000000 --- a/.dev/decisions/0040_vm_method_dispatch_opcodes.md +++ /dev/null @@ -1,282 +0,0 @@ -# 0040 — VM bytecode shape for the deftype-family + method-dispatch cluster - -- **Status**: Accepted (Devil's-advocate fork landed 2026-05-26) -- **Date**: 2026-05-26 -- **Author**: Shota Kudo (drafted with Claude autonomous loop) -- **Tags**: vm, bytecode, opcode, dispatch, callsite-cache, F-002, F-003, - D-073, ADR-0036 - -## Context - -Phase 7 entry's ADR-0036 (dual_backend_parity contract) carved 5 -VM-DEFER sites in `src/eval/backend/vm/compiler.zig` for the -deftype-family analyzer Node variants — `deftype_node`, -`ctor_call_node`, `field_access_node`, `compileRequire` libspec -branch, `compileNs` filter — pending the bytecode shape decisions -each requires. Row 7.6 cycle 1 (commit `40268e2`) added a 6th site, -`method_call_node`, when it landed the `(.method instance args)` -TreeWalk surface + analyzer arm. - -Row 7.6 cycle 4 discharges 4 of the cluster's 5 sites: - -- `deftype_node` (a) -- `ctor_call_node` (b) -- `field_access_node` (c) -- `method_call_node` (f, new in cycle 1) - -Sub-sites (d) `require_libspec` and (e) `ns_filter` are out-of-scope -— they have their own follow-up cycles per ADR-0036 + D-073 row. - -## Decision - -Adopt **Shape 1 + Shape 1.b** from the row 7.6 survey -(`private/notes/phase7-7.6-survey.md` §5): **4 dedicated opcodes + -CallSite side-table on BytecodeChunk**. - -### New opcodes (4) - -- `op_deftype` (0x18) — operand = constants index of a heap - `TypeDescriptorRef` Value (pre-built at analyzer time by - `runtime/type_descriptor.zig::registerType`). Dispatch reads the - descriptor pointer and the VM equivalent of `evalDeftype` is a - no-op (descriptor already registered in `rt.types` by the - analyzer-time `registerType` call); the op pushes `nil` onto the - stack to match the TreeWalk return value. -- `op_ctor_call` (0x19) — operand = packed `(name_idx << 8) | - arg_count`. `name_idx` is the constants-index of a String holding - the type name. Pops `arg_count` values, looks up the descriptor - via `rt.types.get(name)`, and calls `allocInstance`. -- `op_field_access` (0x1A) — operand = constants-index of the field - name String. Pops the receiver, walks `descriptor.field_layout` - linearly, returns the matching field value (or raises - `symbol_unresolved`). -- `op_method_call` (0x1B) — operand = `call_site_idx` into the new - `BytecodeChunk.call_sites: []*CallSite` side-table. The CallSite - carries the method-name slice + the cached `MethodEntry` pointer - + `cached_generation`. Pops `arg_count` (encoded into the - call_site entry) + receiver, dispatches through - `cs.lookupWithCache(td, null, method_name, generation)` then - `vt.callFn(rt, env, method_val, args, loc)`. - -### BytecodeChunk extension - -`src/eval/backend/vm/opcode.zig::BytecodeChunk` grows one slice: - -```zig -pub const BytecodeChunk = struct { - instructions: []const Instruction, - constants: []const Value, - call_sites: []const CallSiteEntry = &.{}, -}; - -pub const CallSiteEntry = struct { - method_name: []const u8, - arg_count: u16, - cache: method_table.CallSite = .{}, -}; -``` - -Analyzer-arena-owned (chunk lifetime = analyzer arena lifetime). -The slice is empty for chunks that contain no method-call sites -(zero overhead for the common case). - -CallSite cache survives across calls into the same chunk (the -chunk is shared by all invocations of the function). When -`rt.protocol_generation` advances (new `extend-type` lands), the -cache's `cached_generation` mismatch triggers a re-lookup — -identical semantics to the TreeWalk path in row 7.3. - -### `lookupMethod` shape - -Cycle 1's optional-`protocol_name` extension (Path A2) was a -prerequisite: the `.method` form does not name a protocol, so VM -dispatch passes `null` for the protocol_name. The op_method_call -arm calls `cs.lookupWithCache(td, null, method_name, generation)`. -Today `CallSite.lookupWithCache` takes `protocol_name: []const u8` -non-optional; cycle 4 widens it to `?[]const u8` to mirror -`TypeDescriptor.lookupMethod`. - -## Consequences - -### Positive - -- F-002: finished form per the survey's Shape 1+1.b - recommendation. Opcode count (4) matches AST-variant count (4) - — each opcode names a single AST shape, simpler dispatch loop - branching. -- ADR-0036 dual_backend_parity contract honoured for the 4 cluster - sites. Diff_test cases land in `src/lang/diff_test.zig` - exercising each opcode. -- CallSite cache integration from day 1 on the VM side; matches - the row 7.3 TreeWalk path's monomorphic-per-site cache contract. -- Phase 17 JIT lowering surface: each opcode maps to one IR node - cleanly; the side-table CallSite layout mirrors HotSpot's inline - cache shape, making the eventual JIT inline-cache port a direct - data structure migration rather than an opcode rewrite. - -### Negative - -- 4 opcodes vs the DA-recommended 2 (Alt 2 below). The DA's argument - that "opcodes name evaluation strategies, not AST shapes" has - merit. Trade-off acknowledged: descriptor-side field-accessor - MethodEntry synthesis (Alt 2's prerequisite) requires extending - MethodEntry with a synthetic-accessor variant — non-trivial - shape change vs. the simple "one MethodEntry per declared - method" finished form. F-003 supports deferring the unification - to a future cycle when synthetic-accessor MethodEntries become - load-bearing for another reason. -- `BytecodeChunk` grows a slice field (~16 bytes per chunk via - `[]const CallSiteEntry` slice header; zero per chunk that - doesn't use `.method`). - -### Neutral - -- F-003: defers PIC (polymorphic inline cache) shape to Phase 17. -- F-004: CallSite is heap-side (no NaN-box slot). -- F-006: side-table slice is analyzer-arena-owned, no GC root - pressure (CallSite contents are pointers into descriptor table - which is GPA-rooted via Env per row 7.3). - -## Alternatives considered - -(Devil's-advocate fork output reflected verbatim, 2026-05-26) - -### Alt 1 — Smallest-diff: extend `op_call` + a `MethodHandle` builtin - -Zero new opcodes. Analyzer lowers `(.m inst args)` into -`(rt/__dispatch-method inst args...)` and emits -`op_invoke_builtin `. `deftype_node` lowers to -`(rt/__make-descriptor ...)` + `op_def`. `ctor_call_node` lowers -to `(rt/__construct args...)` + `op_invoke_builtin`. -CallSite caches live in the builtin's per-name interning table -(one CallSite per `(receiver-type, method-name)` pair, -runtime-global). - -**Better than others**: smallest patch in `compiler.zig` (4 arms -become 4 `lowerToBuiltin` calls; no `opcode.zig` extension). VM -dispatch loop is untouched. Diff under ~200 LOC. - -**Breaks**: (1) Loses **per-call-site monomorphism** — the -survey's central row-7.3 invariant. The runtime-global cache is -megamorphic by construction; F-002's "finished-form wins" -forbids regressing dispatch quality for diff size. -(2) Reintroduces the v0 `__java-method` shape the survey -explicitly DIVERGENCE-marked. (3) Phase 17 JIT cannot inline -through an `op_invoke_builtin` veneer without re-introducing the -opcode anyway → **net rewrite cost grows**. - -**F-NNN impact**: F-002 violated (smallest-diff bias smell). -F-003 neutral. F-004/F-006 neutral. - -### Alt 2 — DA-recommended: **2 opcodes** + Shape 1.b side-table - -Add **2** opcodes (not 4): `op_dispatch_method` (covers -`method_call_node` AND `field_access_node` — field read = zero-arg -method call against the descriptor's field-accessor entry, a -synthetic MethodEntry that reads `field_layout[i]`); `op_construct` -covers `ctor_call_node`. `deftype_node` lowers via `op_const -` + `op_def` (no new opcode). - -**Better than others**: (1) Opcode count = dispatch-shape count -(2), not site count (4) — matches the **finished-form rule that -opcodes name evaluation strategies, not AST shapes**. -(2) Field-access-as-zero-arg-method collapses the arity-2-vs-method-call -ambiguity (survey §4) into the descriptor. (3) Side-table -CallSite (Shape 1.b) is fixed-size opcodes, JIT-friendly, and -matches HotSpot's inline-cache layout — Phase 17 lowering is a -direct port. - -**Breaks**: `BytecodeChunk` grows a slice field. `TypeDescriptor` -must synthesize zero-arg field-accessor MethodEntries at -`deftype` time — small extension to row 7.3 `method_table.zig`. -The arity-2 ambiguity (survey §4) must be resolved as -**Option B** (descriptor lookup wins), which is a behaviour -change from 5.12.a for the FieldAccessNode arm — but ADR-0036 -already names row 7.6 as the discharge point so this is in-scope. - -**F-NNN impact**: F-002 ✓ (finished-form wins over Alt 1's diff). -F-003 ✓ (defers PIC to Phase 17). F-004 neutral. F-006 ✓. - -**Why NOT selected**: Synthesizing field-accessor MethodEntries -requires either (a) extending `MethodEntry` with a -synthetic-accessor variant (= shape-change to a row 7.3 -load-bearing struct), or (b) per-field BuiltinFn allocation at -registerType time (= every defrecord allocates N BuiltinFn -closures). Both routes pay a structural cost that the cycle 4 -scope did not budget for. Per F-003 the unification is a -finished-form goal but deferred; ADR-0040 lands the 4-opcode -shape today and a future cycle (when synthetic-accessor MethodEntries -become load-bearing for another reason) can collapse the two -opcodes via a clean shape migration. Recorded as the leading -runner-up alternative. - -### Alt 3 — Wildcard: PIC inline cache (Shape 1.a) from day 1 - -4 dedicated opcodes (Shape 1), but each `op_method_call` carries -a **variable-sized inline cache** appended in-line after the -opcode: `[opcode u8][method_idx u16][cache_entry_count u8][CacheEntry...]`. -Cache entries are `(type_id u32, method_ptr u64)` pairs grown -in-place at dispatch time (up to 4 = polymorphic, then megamorphic -fallback). Bytecode stream becomes variable-width. - -**Better than others**: Genuinely zero-indirection cache hit — no -side-table lookup, no chunk field access. Matches V8 / SpiderMonkey -inline-cache layout exactly. If Phase 17 JIT goes PIC-first, this -is the destination. - -**Breaks**: Variable-width opcodes break `Instruction` struct -(currently `{ opcode, u16 operand }` fixed) — -`BytecodeChunk.instructions` becomes `[]u8` flat stream -prematurely (the opcode.zig docstring explicitly defers this to -Phase 17+). The dispatch loop's PC-advance becomes opcode-dependent. -ADR-0036 diff_test machinery would need to compare bytecode -streams positionally rather than by-instruction. Heavy migration -cost paid before the JIT exists to justify it. - -**F-NNN impact**: F-002 ✗ (finished-form for Phase 7 is -monomorphic + side-table per row 7.3; PIC is Phase 17's call). -F-003 ✗ (decision-seizure — locks Phase 17's PIC shape before -its owner reviews). F-004 neutral. F-006 neutral. - -### DA recommendation - -Alt 2. The survey's lean (Shape 1+1.b) is right but under-justifies -relative to Alt 2's opcode-count collapse. Main loop overrode the -DA recommendation per the synthetic-accessor cost argument above -(F-003 deferral); Alt 2 stays as the leading runner-up for the -future synthesis cycle. - -## Affected files - -- `src/eval/backend/vm/opcode.zig` — add 4 opcodes (0x18..0x1B), - add `CallSiteEntry` struct, extend `BytecodeChunk` with - `call_sites` field. -- `src/eval/backend/vm/compiler.zig` — replace the 4 VM-DEFER - arms with real compile arms emitting the new opcodes; build - the `call_sites` side-table during compile. -- `src/eval/backend/vm.zig` — add 4 dispatch arms in the main - bytecode loop. -- `src/runtime/dispatch/method_table.zig` — - `CallSite.lookupWithCache` widens `protocol_name` to - `?[]const u8` (mirrors row 7.6 cycle 1's `lookupMethod` - widening). -- `src/lang/diff_test.zig` — add ≥1 diff_test case per opcode - exercising the dispatch parity contract per ADR-0036. -- `feature_deps.yaml` — `runtime/vm/dispatch_family` entry's - `status:` flips from `provisional` to `landed` for the - 4 cluster sites; the 2 remaining sites (`require_libspec`, - `ns_filter`) keep their open markers. -- `.dev/debt.md` D-073 row — annotate the 4-site discharge. - -## References - -- `.dev/decisions/0036_dual_backend_parity_contract.md` — the - contract this cycle 4 honours. -- `.dev/decisions/0008_protocol_dispatch.md` — protocol dispatch - + CallSite cache semantics (row 7.3 amendments incl. amendment - 3 binding here). -- `.dev/project_facts.md` — F-002, F-003. -- `.dev/principle.md` — Reservation-as-bias smell (Alt 1's failure - mode). -- `private/notes/phase7-7.6-survey.md` §3 + §5 + §6 cycle 4. -- ADR-0036 D5 — the cluster-discharge cycle this ADR realises. diff --git a/.dev/decisions/0041_multi_arity_fn_shape.md b/.dev/decisions/0041_multi_arity_fn_shape.md deleted file mode 100644 index ed82a9a99..000000000 --- a/.dev/decisions/0041_multi_arity_fn_shape.md +++ /dev/null @@ -1,310 +0,0 @@ -# 0041 — Multi-arity `fn*` shape: uniform `methods` slice on FnNode + Function - -- **Status**: Accepted (Devil's-advocate fork landed 2026-05-27) -- **Date**: 2026-05-27 -- **Author**: Shota Kudo (drafted with Claude autonomous loop) -- **Tags**: fn, analyzer, dispatch, multi-arity, recur, F-001, F-002, - F-009, D-070, ADR-0036 - -## Context - -Phase 7 row 7.8 (D-070) opens multi-arity `fn*` / `defn`: - -```clojure -(defn greet - ([] (greet "world")) - ([name] (str "hello, " name))) -``` - -JVM Clojure compiles each arity body to its own `invoke(N args)` -method on the generated class; the JIT dispatches by JVM method -overload resolution. cw v1 today accepts only single-arity -`(fn* [params] body)` (`src/eval/analyzer/bindings.zig:23-86`) -and emits a flat `FnNode { arity, has_rest, params, body }` -(`src/eval/node.zig:160-175`). `tree_walk.Function` -(`src/eval/backend/tree_walk.zig:100-124`) mirrors the flat shape. - -Several Phase 6 `.clj` defns are blocked on multi-arity: -transducer 1-arg arities (`(map f)`), multi-arg `comp` / `juxt` / -`partial` / `complement` / `every?`, the 3-arity -`clojure.set/join`. All carry `PROVISIONAL` markers pointing at -`feature_deps.yaml#special_form/fn_multi_arity` which discharges -at row 7.8 close. - -Row 7.8's Step 0 survey lives at -`private/notes/phase7-7.8-survey.md` (850 lines; reads JVM -`Compiler.java::FnExpr.parse` + `AFn.invoke`, cw v1 current shape, -cw v0 `extra_arities` prior art, dual-backend parity check per -ADR-0036). Survey §6 enumerates 3 options (A macro-desugar, B -uniform `methods` slice, C bytecode jump-table). Step 0.6 -amendments at survey §11 lock in REPLACE-not-additive for FnNode + -Function fields and MERGE per-method recur scope creation into -cycle 1 (eliminates the R1 silent-default-shift gap entirely). - -## Decision - -Adopt **Option B-extracted** — uniform `methods: []const FnMethod` -slice on both `FnNode` and `tree_walk.Function`, with `variadic: -?FnMethod` as a separate slot; single-arity ships as -`methods.len == 1`, `variadic == null`. Per-method recur scope -creation merges into cycle 1 (so the multi-arity surface opens -with correct `recur` semantics from day one — no window where -`recur` re-enters the wrong body). - -### `FnNode` shape (`src/eval/node.zig`) - -```zig -pub const FnMethod = struct { - arity: u16, - has_rest: bool, - params: []const []const u8, - body: *const Node, - bytecode: ?*const BytecodeChunk = null, -}; - -pub const FnNode = struct { - /// Fixed-arity methods, sorted by `arity` ascending. Single-arity - /// is a 1-element slice. JVM rule 2 (no two same-arity fixed) - /// enforced at analyzer-time. - methods: []const FnMethod, - /// At most one variadic per JVM rule 1. - variadic: ?FnMethod = null, - slot_base: u16 = 0, - loc: SourceLocation = .{}, -}; -``` - -### `tree_walk.Function` shape - -Same `methods` + `variadic` shape on the heap struct. The -existing flat fields (`arity` / `has_rest` / `params` / `body` / -`bytecode`) are dropped — single-arity reads `methods[0]`. - -### Analyzer dispatch (`analyzeFnStar`) - -JVM `FnExpr.parse` always normalises into multi-arity form before -parsing. cw v1's analyzer wraps single-arity (`(fn* [x] body)`) -into the multi-arity shape (`(fn* ([x] body))`) before walking the -body list; each `(params body...)` sublist becomes one FnMethod. - -JVM rules enforced at analyzer-time with three new error catalog -codes (per `.claude/rules/error_catalog_only.md` naming convention): - -- `fn_star_arity_duplicate` — two fixed methods share the same - required-arg count. -- `fn_star_variadic_duplicate` — more than one `[& rest]` body. -- `fn_star_fixed_exceeds_variadic` — a fixed method has more - params than the variadic's required count. - -Per-method scope: each FnMethod runs body analysis inside its -own `child_scope` with its own `recur_target`. `recur` inside -arity-N body re-enters arity-N body (JVM parity). - -### Runtime dispatch (`callFunction`) - -```zig -for (f.methods) |m| { - if (args.len == m.arity) { ... bind params + run body ... } -} -if (f.variadic) |v| { - if (args.len >= v.arity) { ... bind v.params + build rest list ... } -} -return error_catalog.raise(.arity_not_expected_multi, loc, .{ - .fn_name = ..., .got = args.len, .arities = ... }); -``` - -JVM rule "fixed-arity wins on exact match" naturally falls out: -the fixed loop scans first. - -New error code `arity_not_expected_multi` carries a `{arities}` -slot for the message "Wrong number of args (N) passed to: foo — -expected one of: 1, 2, or [3 & rest]". - -### VM compile dispatch - -Each FnMethod's body compiles to its own `BytecodeChunk`. The -`op_make_fn` payload extends to carry N chunks; the runtime -`Function.methods[i].bytecode` carries the per-method pointer. - -### Cycle plan (4 cycles, down from survey's 5) - -- **Cycle 1**: FnNode + Function uniform `methods` slice + analyzer - parse + per-method recur scopes + TreeWalk dispatch + VM compile - arm + VM call dispatch + 3 new error codes + ≥ 1 diff_test case - (this ADR is the cycle 1 ADR commit). -- **Cycle 2**: variadic-and-fixed coexistence dispatch + JVM rule 3 - analyzer-time enforcement + diff_test cases. -- **Cycle 3**: `defn` macro extension (multi-arity body desugar - surface); 2 e2e tests; docstring + meta-map deferred per survey - §11 Q6 to a new D-NNN follow-up row. -- **Cycle 4** (= survey's cycle 5): PROVISIONAL discharge for the - 5+ blocked `.clj` defns; flip - `feature_deps.yaml#special_form/fn_multi_arity` from `planned` - to `landed`; flip row 7.8 [x]. - -Survey's cycle 4 (optional perf fast-path inlined `arity0` slot) -is deferred to a debt row to be filed only if `bench/quick` -regresses on single-arity calls after cycle 1. - -## Devil's-advocate fork (depth-2, fresh context) — verbatim embedding - -The DA subagent ran with fresh context and the F-NNN envelope from -`.dev/project_facts.md`. Survey §6 already enumerates Options A -(macro-desugar), B (uniform slice), C (bytecode jump-table); the -DA produced **3 new alternatives** the main loop may be -attention-suppressed against (named D / E / F to follow the -survey's A/B/C): - -> **Alternative D — Analyzer-only multi-arity (Function stays -> single-arity; one Function value per arity; closure storage -> shared)**. -> Shape. Analyzer emits N FnNodes wrapped in a thin -> `MultiFnNode { arities, variadic, closure_bindings_shape }`. A -> new HeapTag `multi_arity_fn` (Group B reserved slot per F-004) -> wraps N per-arity Functions; closure-captured Values intern -> once behind a `*const ClosureCells` shared across arities. The -> existing single-arity Function and `callFunction` are -> **unchanged** — only a `callMultiArityFn` arm is added. -> Better than B-extracted. Zero churn in `Function` struct → -> zero risk of regressing the single-arity hot path (survey R4 -> evaporates). VM `op_make_fn` / `op_call` / `evalChunk` need no -> field-layout work; only the new tag arm. `(arglists f)` works -> identically. -> Cost. Burns a Group B reservation. Identity / meta semantics -> double the wrap layer. Most importantly: **two callable shapes** -> (`fn_val` vs `multi_arity_fn`) for what JVM calls "one fn" — -> every dispatch site (dispatch.zig, partial, apply, comp) has to -> learn both. Exactly the asymmetry F-002 finished-form penalises. -> F-NNN. F-001 ✓; F-002 ⚠ (asymmetric callable shapes contradict -> "symmetric methods slice with `len == 1` for single-arity" -> finished form). F-004 ✓. F-006 ✓. F-009 ✓. **Verdict: F-002 -> weak; smallest-diff bias smell against B-extracted.** -> -> **Alternative E — Reuse the existing `multi_fn` Tag as the -> dispatch backbone (row 7.2 multimethod infrastructure subsumes -> arity dispatch)**. -> Shape. The analyzer compiles `(fn* ([x] body1) ([x y] body2))` -> into a synthetic MultiFn (the existing `.multi_fn` Tag at -> `value.zig:63`) whose dispatch-fn is the built-in -> `arity-of-args`, and whose method table maps `1 → body1-as- -> Function`, `2 → body2-as-Function`, `:default-variadic → -> variadic-Function`. `callFunction` stays untouched. -> Better than B-extracted. One dispatch mechanism for "callable -> that branches on a property of its args" — multimethods AND -> arity dispatch share it. Row 7.2's hash-keyed table is O(1) -> dispatch. -> Cost. **`recur` semantics break catastrophically.** A MultiFn's -> per-method body is a separate Function value with its own -> scope frame; routing through `callMultiFn` adds a hash lookup -> + dispatch-fn evaluation + method-table read for every fn -> call (hot-path regression). The dispatch-fn must live outside -> lexical scope — opposite to JVM's "arity is the first thing -> decided, params bind second." `(arglists f)` on a MultiFn- -> shaped fn would have to special-case "is this really a fn?" -> F-NNN. F-001 ✓. F-002 **violation candidate** — subsumption -> produces a finished form where `(fn ...)` returns a -> *multimethod*; the Clojure surface says "fn returns a fn." F-005 -> ✓. F-006 ✓. F-009 ✓. **Verdict: rejected** — semantic-identity -> violation; the cleanness is on the wrong axis. -> -> **Alternative F — Discriminated-union FnNode + tagged Function -> (`Single` fast path inlined, `Multi` is the slice path)**. -> Shape. `FnNode` becomes a `union(enum) { single: SingleArityFn, -> multi: MultiArityFn }`. `SingleArityFn` keeps today's flat -> fields verbatim; `MultiArityFn` carries the `methods` slice + -> variadic. `Function` mirrors. `callFunction` does -> `switch (f.body) { .single => |s| ...today's fast path..., -> .multi => |m| ...linear scan + variadic... }`. -> Better than B-extracted. Single-arity Function never pays for -> the `methods` slice indirection (survey R4 evaporates without -> needing the deferred "inline arity0 fast slot" fix). The two -> shapes are explicit in the type — readers grep `.multi` to find -> all multi-arity code paths. **Minimum-risk** shape for cycle 1. -> Cost. Two code paths per Function operation forever — -> `(arglists f)`, GC root-set walk, bytecode chunk install at -> `op_make_fn`, etc. — each has a switch arm. Each future arity- -> touching feature must remember both arms. This is exactly the -> "primary + extras" asymmetry survey §3 rejected as v0 folklore. -> The `methods.len == 1` uniformity that B-extracted ships *is* -> the F-002 finished form precisely because it kills the "is this -> single or multi?" branch. -> F-NNN. F-001 ✓. F-002 ⚠ — duplicate code paths is the textbook -> smallest-diff bias shape (small per-cycle diff, larger finished -> form). F-006 ✓. F-009 ✓. **Verdict: B-extracted is the F-002- -> cleaner shape; F is the *risk-minimal* shape if cycle 1 cannot -> afford the unified-slice surgery, but the main loop's authority -> to take big surgery (F-002 says so explicitly) makes F a -> smallest-diff-bias trap.** -> -> **No hard F-NNN violation** across D / F (both flagged as F-002 -> weak, not breaches). Alt E is rejected for breaking JVM -> `(arglists f)` + recur ergonomics. -> -> **Recommendation**: B-extracted (the main agent's choice) is -> the strongest finished-form fit. F is held as the documented -> mitigation in `.dev/debt.md` if `bench/quick` later motivates -> it; D and E recorded as "considered and rejected" so future -> readers see why the symmetric-slice shape is load-bearing -> rather than a default. - -**Selected**: Option B-extracted (uniform `methods` slice). The -DA's recommendation aligns with the main agent's lean. Alt D -rejected because callable-shape asymmetry fan-outs across -dispatch / partial / apply / comp / meta — a permanent debt the -loop would carry. Alt E rejected per DA's "semantic-identity -violation" finding (a fn that is actually a multimethod). Alt F -preserved as a follow-up fallback debt row to be filed only if -cycle 1's bench shows the linear-scan-over-1-element-slice as a -real regression. - -## Affected files - -Cycle 1 source commit touches: - -- `src/eval/node.zig` — `FnNode` field rewrite (drop `arity` / - `has_rest` / `params` / `body`; add `methods` + `variadic`); new - `FnMethod` struct co-located with FnNode. -- `src/eval/analyzer/bindings.zig` — `analyzeFnStar` extension: detect - single-vs-multi body shape, parse each `(params body...)` into one - FnMethod, enforce JVM rules 1-2 (rule 3 deferred to cycle 2 along - with variadic-fixed coexistence). Per-method `child_scope` with own - `recur_target`. -- `src/eval/backend/tree_walk.zig` — `Function` field rewrite to mirror - FnNode shape; `callFunction` linear-scan dispatch. -- `src/eval/backend/vm/compiler.zig` — `op_make_fn` payload extension - to carry N chunks; emit one chunk per FnMethod. -- `src/eval/backend/vm.zig` — `op_call` arm consumes the FnMethod with - the matching arity from `Function.methods`. -- `src/runtime/error/catalog.zig` — three new Codes - (`fn_star_arity_duplicate`, `fn_star_variadic_duplicate`, - `arity_not_expected_multi`); `fn_star_fixed_exceeds_variadic` - deferred to cycle 2 with rule 3. -- `src/lang/diff_test.zig` — ≥ 1 diff_test case exercising 1-method - + N-method dispatch. - -Cycle 2 / 3 / 4 add `variadic` body parsing, `defn` macro extension, -and the PROVISIONAL discharge sweep. - -## Consequences - -**Required**: -- ADR-0036 dual-backend parity: cycle 1 lands TreeWalk + VM in the - same commit (`Function.methods` + `callFunction` + `op_make_fn` - payload extension all share the same `methods` slice shape). -- Step 6 cycle protocol: this ADR commit lands first; the source - commit lands separately. - -**Optional follow-ups**: -- If `bench/quick` shows single-arity regression after cycle 1, file - a `D-NNN` debt row promoting Alternative F (discriminated-union - Function) as the mitigation. Survey R4 predicts the regression is - negligible (single-element slice walk ≈ direct field read). -- `defn` docstring + meta-map support deferred per survey §11 Q6 to - a separate `D-NNN` follow-up row, filed at cycle 3 close. -- `feature_deps.yaml#special_form/fn_multi_arity` flips from - `planned` to `landed` at cycle 4 close. - -## Revision history - -- 2026-05-27 (Accepted): initial draft + DA fork + main-agent - selection of Option B-extracted. diff --git a/.dev/decisions/0042_apply_variadic_rest_bind.md b/.dev/decisions/0042_apply_variadic_rest_bind.md deleted file mode 100644 index 023db2cd4..000000000 --- a/.dev/decisions/0042_apply_variadic_rest_bind.md +++ /dev/null @@ -1,446 +0,0 @@ -# 0042 — `apply` variadic-callee fast-path: gated bind-direct in `callFunction` rest-pack - -- **Status**: Accepted (Devil's-advocate fork landed 2026-05-27) -- **Date**: 2026-05-27 -- **Author**: Shota Kudo (drafted with Claude autonomous loop) -- **Tags**: apply, variadic, dispatch, fn, lazy-seq, F-002, F-003, - D-072, ADR-0036, ADR-0041 - -## Context - -Phase 7 row 7.9 (D-072) refactors `apply` so the trailing seq is -not eagerly collected into a flat Zig slice when the callee is a -user variadic fn that can bind the seq directly to its `& rest` -parameter. - -Today `applyFn` -(`src/lang/primitive/higher_order.zig:56-90`) walks the trailing -seqable via `seq → first/rest` into a `std.ArrayList(Value)`, -then dispatches via `rt.vtable.callFn` with a flat slice. The -walk fully realises lazy_seq cells and produces O(N) intermediate -Zig storage even when the callee is a variadic fn whose body -would happily walk the seq incrementally. - -Row 7.9's Step 0 survey lives at -`private/notes/phase7-7.9-survey.md` (~640 lines incl. the -Step 0.6 main-agent amendment). The original survey framing -proposed an `IReduce` protocol fast-path on the trailing seq — -mirroring row 7.7 cycle 4's `reduceFn` IReduce bypass. The -Step 0.6 re-laying surfaced that this framing **does not match -JVM Clojure semantics**: JVM's `RestFn.applyTo` -(`OSS/clojure/src/jvm/clojure/lang/RestFn.java:132-160`) does -**variadic peel-and-pass-tail** using `RT.boundedLength` — IReduce -is consumed by `reduce`, not by `apply`. cw v0 also mirrors the -variadic shape (`collections.zig:961-1097` `__zig-apply` + -threadlocal `apply_rest_is_seq` flag at `dispatch.zig:64-67`). -The D-072 debt row body is amended in place per ROADMAP §17.4 to -reflect the corrected scope (variadic peel-and-pass-tail, not -IReduce on the trailing seq). - -The substrate is in place: - -- ADR-0041 (row 7.8) landed `FnNode.variadic: ?FnMethod` - (`src/eval/node.zig:179-189`) and - `tree_walk.selectMethod:790-797` already implements - "prefer fixed-arity over variadic" (JVM rule 3). -- `tree_walk.callFunction:743-788` is the single rest-pack - site: lines 765-777 cons-wrap `args[m.arity..]` into a fresh - heap List for the `& rest` slot. -- VM has **no separate rest-pack**. `vm.zig:573` installs - `tree_walk.treeWalkCall` as the VM's `vtable.callFn`; both - backends share `callFunction`. The dual-backend parity - contract (ADR-0036) is therefore satisfied by construction — - no new analyzer Node variant, no new opcode. - -The Devil's-advocate fork -(`private/notes/phase7-7.9-da-fork.md`, ~340 lines) enumerated -three alternatives within the F-001 / F-002 / F-003 / F-006 / -F-009 envelope plus one out-of-envelope pre-finding. The DA -recommendation is Alt 3 (gated bind-direct in `callFunction`). - -## Decision - -Adopt **Alt 3 — gated bind-direct rule inside -`tree_walk.callFunction`'s rest-pack step**. The dispatcher -contract (`rt.vtable.callFn` signature + entry count) is -unchanged. - -### The rule - -Inside `tree_walk.callFunction`, the rest-pack block at -lines 765-777 gains an early-bind branch: - -> If `m.has_rest` AND `args.len == m.arity + 1` AND the single -> trailing arg `args[m.arity]` has a **seq-shaped tag** (one of -> `.list`, `.cons`, `.chunked_cons`, `.lazy_seq`, `.nil`) — -> bind that arg directly to the rest slot. -> Otherwise — fall through to the existing cons-pack loop. - -The gate is generic — it does not look up whether the call -originated in `apply`. Any caller passing a single seq-shaped -arg into a variadic exact-arity slot benefits. The gate is -sound because cons-packing a single seq-shaped arg into a -one-element list would always be observably different from -direct binding only when the seq itself is the user-intended -single rest element — which the gate excludes by requiring a -seq-shaped tag (vectors are excluded; `(apply f x [y])` continues -to spread `[y]` element-by-element). - -### `applyFn` change - -`applyFn` is rewritten to its compact shape: - -1. Validate arity (≥ 2) — unchanged. -2. Build `call_args = [leading..., trailing]` — a single - contiguous slice of length `leading.len + 1`. The trailing - slot holds the trailing arg as-is (no walk, no realisation). -3. Dispatch via `vtable.callFn(rt, env, f, call_args, loc)`. - -When the callee is a variadic fn whose exact-arity matches the -shape (= `leading.len == m.arity` so `call_args.len == m.arity + -1`) AND `trailing` is seq-shaped, the gate fires and `trailing` -binds directly to `& rest`. Otherwise the callee's normal -arity-matching applies — for a fixed-arity callee, `call_args` of -length `leading + 1` won't match the fixed arity (unless -`leading + 1 == fixed_arity` in which case the trailing arg is -the user's single positional value, also correct), and the -arity error surfaces. For lazy / list / cons trailing args, the -direct bind happens. For vector trailing args, the gate skips, -the existing cons-wrap fires, the user's vector is wrapped into -a one-element list — wrong semantics for JVM compatibility. - -To preserve JVM `(apply f x [y])` semantics (the `[y]` vector -spreads to a one-element list), `applyFn` retains a **vector / -non-seq spread step** as a pre-pass: if `trailing` is a vector / -hash-set / array-map / etc., walk it to spread into individual -`call_args` slots (same as today's eager path); only seq-shaped -trailings flow through the direct-bind opportunity. This keeps -the spread semantics observable while opening the lazy-tail -fast-path for the cases that benefit (list, cons, lazy_seq). - -### Cycle structure - -- **Sub-cycle 1 (this commit)**: ADR-0042 lands + D-072 debt - row amended in place + Step 0.6 amendment recorded on - `private/notes/phase7-7.9-survey.md`. -- **Sub-cycle 2**: implement the gated bind-direct rule in - `callFunction:765-777` (≈12 LOC) + refactor `applyFn` to the - compact shape (≈25 LOC net reduction). Add ≥ 3 differential - test cases (positive: list / cons / lazy_seq tail with - variadic callee; negative: vector tail still spreads) + - e2e shell coverage. -- **Sub-cycle 3**: close row 7.9 — flip `[x]` in §9.9; per-task - note; smell sweep. - -## Implementation surface - -| File | Change | LOC | -|-------------------------------------------|--------------------------------------------------------------------------------------|---------| -| `src/eval/backend/tree_walk.zig` | `callFunction` rest-pack gains the gated bind-direct branch + 5-line comment block | ~12 LOC | -| `src/lang/primitive/higher_order.zig` | `applyFn` rewritten to compact shape + vector-spread pre-pass | -15 LOC | -| `src/lang/diff_test.zig` | 3 positive cases + 1 negative case for vector-spread; locks the gate rule on both BE | ~40 LOC | -| `test/e2e/phase7_apply_variadic.sh` (new) | e2e cases: lazy-tail apply on variadic user fn | ~30 LOC | -| `src/runtime/dispatch.zig` | **untouched** — vtable contract unchanged | 0 | - -Net: roughly +67 LOC across 4 files, -15 LOC in higher_order.zig -for a net ≈ +52 LOC. No new Node variant, no new opcode, no -vtable surgery — `dual_backend_parity.md`'s contract is -satisfied by construction. - -## Alternatives considered - -(Devil's-advocate subagent output verbatim from -`private/notes/phase7-7.9-da-fork.md`; the subagent's -recommendation is Alt 3 and the main loop adopts it.) - -### Pre-finding (out-of-envelope; recorded as leading entry per CLAUDE.md DA mandate) - -**Value-tag-marker variant violates F-004 (NaN-box 64-slot -budget).** A new `Value.Tag` (e.g. `apply_rest_seq`) expressing -"this is a prebound seq for `& rest`" would give the cleanest -in-band signal — no shared state, no vtable widening, no -caller / callee coupling. But F-004 fixes the second-generation -layout at 4 group × 16 sub-type = 64 slot with ~10 slots -reserved for future surprises; burning one on a transient -round-trip marker is the inverse of the Reservation-as-bias -smell. **Excluded as F-004 violation.** Main loop does not halt; -the in-envelope Alt 3 is still live. - -### Alt 1 — Smallest-diff: extend `callFn` signature with `prebound_rest: ?Value` - -Widen `vtable.callFn` to a 6-parameter form, passing `null` from -the 7 of 8 existing call sites that have no use for the new -slot. `applyFn` peels leading args + passes the trailing seq as -`prebound_rest` for the variadic case. - -- Surface: `dispatch.zig` (~3 LOC), `tree_walk.zig` rest-pack - branch (~20 LOC), 8 call sites pass `null` (~8 LOC), `applyFn` - (~40 LOC), diff cases. Total ~80 LOC. -- Better: single vtable entry remains canonical; - discoverability via `rg "vt\.callFn"` stays single-list. -- Worse: **Smallest-diff bias smell per F-002.** Mutates the - load-bearing dispatcher contract for a property meaningful - at exactly one call site. Adds a per-call cost (extra param - / null check) to every fn invocation. -- cw v0's threadlocal `apply_rest_is_seq` flag is rejected here - for the same reason at a finer grain: literal smallest diff - (zero signature change, zero call-site touches) but its - finished form is "shared mutable state with temporal - coupling" — cleaner finished form (no shared state) wins per - F-002 regardless of diff size. -- **F-002: violates** (Smallest-diff bias). F-001 / F-003 / - F-006 / F-009 neutral. - -### Alt 2 — Finished-form-clean: rename + sibling vtable entry - -Rename `vtable.callFn` → `callFnFlat`; add sibling -`callFnVariadic(rt, env, fn_val, leading, rest_seq, loc)`. -Names document the calling convention. Every existing site is -flat → mechanical 8-site rename. `applyFn` chooses between them -based on `f.tag() == .fn_val ∧ fn.variadic != null`. - -- Surface: ~100 LOC + 8-site rename. -- Better: names self-document; no naming wart - (`callFn` / `callFnWithPreboundRest` asymmetry); Phase 16 - zwasm v2 dispatch arm gets a clean `callFnFlat` entry. -- Worse: 8-site mechanical rename + every test fixture mocking - `vtable.callFn` (`dispatch.zig:298`, `multimethod.zig:790`) - needs updating; the rename ripple is non-trivially - front-loaded; the cost is paid once but timing matters at a - Phase mid-flight. -- **F-002: strongly positive** (finished form cleaner than - Option C); F-001 / F-003: positive; F-006 / F-009: neutral. - -### Alt 3 — Wildcard: move variadic-detect into the call-frame setup (ADOPTED) - -(Full shape in the Decision section above.) - -- Surface: ~12 LOC in `callFunction`, -15 LOC in `applyFn`, 0 - call-site touches, 0 vtable changes. **Net negative LOC.** -- Better than Option C / Alt 1 / Alt 2: zero vtable surgery; - reuses an existing decision point (rest-pack); no new public - entry; other callers benefit transparently; F-003 alignment - (no structural-plan widening to defer). -- Worse: action-at-a-distance between `applyFn` and - `callFunction` (mitigated by 5-line comment block + forward - links + diff_test lock-in); narrow JVM-conforming-but-v0- - divergent semantics on the multi-arity-with-empty-lazy-seq - case (v0 realises the first cell to prefer the fixed arm; cw - v1 binds the variadic arm with empty rest — arguably cleaner - than v0's lazy-realisation-as-dispatch-side-effect smell). -- **F-002: positive**; **F-003: strongly positive**; F-001 / - F-006 / F-009: neutral. - -### Why ADR-0042 adopts Alt 3 - -1. **F-002 finished-form**: Alt 3's finished form is "the - rest-binding step recognises an already-shaped seq and - accepts it" — a single local refinement of one existing - decision point. Option C / Alt 1 / Alt 2 all add structural - surface area to handle a property that is really about - call-frame binding. -2. **F-003 deferral**: Alt 3 is the only candidate that does - not invoke F-003's "imagine + defer" cycle for a structural - plan decision. Vtable widening would have to be re-walked at - Phase 16 entry (zwasm v2 dispatch arm); the rest-pack rule - lives inside `callFunction` and is invisible to Phase 16. -3. **Net negative LOC** — Alt 3 is the only candidate that - shrinks the codebase. cw v0's `__zig-apply` was 138 LOC of - apply + 70 LOC of helpers + threadlocal coordination; Alt 3 - collapses that to ~12 LOC in `callFunction` + ~15 LOC in - `applyFn`. -4. **No new dispatcher surface to maintain**. Two vtable entries - would set the precedent for a third (`callFnAsync`, - `callFnInWasm`, `callFnWithCallSite`) at later Phases. -5. **The semantics divergence from cw v0 is acceptable**. The - multi-arity-with-empty-lazy-seq case where v0 realises one - cell to prefer the fixed arm is itself a smell (lazy_seq - realisation as a side-effect of dispatch). cw v1 declares - per P3 + F-002 that v0 design pressure is not binding; this - is the canonical example. - -## Consequences - -### Positive - -- `applyFn` becomes a compact spread-or-passthrough body; - cw v0's threadlocal `apply_rest_is_seq` is not replicated; - the cross-Zone temporal coupling that v0 carried disappears. -- Any caller (not just `apply`) passing a single seq-shaped - arg to a variadic exact-arity slot benefits transparently. -- `vtable.callFn` shape stays uniform — Phase 16's wasm fn - dispatch arm lands without an apologetic `prebound_rest` - parameter. -- The dual-backend parity contract (ADR-0036) is satisfied by - construction — no new analyzer Node variant, no new opcode. - -### Negative - -- Action-at-a-distance between `applyFn` and `callFunction`. - Mitigated by a 5-line comment block in `callFunction` - explaining the gated bind-direct case + a forward link from - `applyFn`'s docstring + diff_test cases locking the rule. -- A narrow semantics divergence from cw v0 on the multi-arity- - with-empty-lazy-seq case (v0 realises one cell to prefer the - fixed arm; cw v1 binds the variadic arm with empty rest). - JVM-conforming, v0-divergent. Documented + locked via diff_test - case. -- The gate's tag set (`.list`, `.cons`, `.chunked_cons`, - `.lazy_seq`, `.nil`) must be kept in sync with future seq- - shaped tag additions (e.g. `.range`, `.string_seq`, - `.array_seq` when those land as user-visible seqs). The gate - becomes a forward-touched site for new seq tags. - -### Deferred - -- `IReduce` registration on `lazy_seq` TypeDescriptor (survey's - original cycle 1 candidate). Step 0.6 amendment showed this - has zero behavioural change vs the row 7.7 cycle 4 seq-walk - fallback (and would be slower if written in `.clj`). Defer - until a real consumer surfaces (e.g. chunked-lazy-seq with - native realisation arithmetic, Phase 8+). -- `boundedSeqLength` as a public helper (survey DIVERGENCE 3). - Alt 3 does not need length information; the gate keys on tag - alone. Defer until a real consumer surfaces (multi-arity - dispatch preference with lazy_seq trailing, spec validation - shortcuts). -- The forward-touched-site cost on the gate's tag set is a - future audit row candidate when `range` lands as a runtime - fn (currently the heap tag exists but no `range` Var; row - 7.10+ or Phase 8 lazy-seq builders cluster). - -## Cross-references - -- ROADMAP §9.9 row 7.9 (this row's task table entry). -- D-072 (`.dev/debt.md`) — body amended in place per ROADMAP - §17.4 to reflect the corrected "variadic peel-and-pass" - scope (was framed as "IReduce aware path" pre-survey). -- ADR-0036 — dual-backend parity contract (satisfied by - construction; no new Node / opcode). -- ADR-0041 — multi-arity `fn*` shape (`FnNode.variadic` - substrate this ADR consumes). -- ADR-0008 — protocol dispatch unify (the IReduce path on - `reduceFn` whose row 7.7 cycle 4 framing influenced - D-072's pre-survey body — re-oriented by the Step 0.6 - amendment). -- F-002 (`.dev/project_facts.md`) — finished-form cleanliness; - the rejection of Alt 1's smallest-diff path + cw v0 - threadlocal rides on this. -- F-003 — decision-deferral on structural plans; Alt 3 is the - only candidate that does not require a Phase-16 re-walk of - the vtable shape decision. -- F-001 — zwasm v2 integration (Phase 16); Alt 3 leaves - `vtable.callFn` shape pristine for Phase 16 to extend on its - own terms. -- `private/notes/phase7-7.9-survey.md` — Step 0 survey + Step 0.6 - main-agent amendment. -- `private/notes/phase7-7.9-da-fork.md` — Devil's-advocate - enumeration; this ADR's "Alternatives considered" section is - the DA output condensed. -- cw v0 `__zig-apply` at - `~/Documents/MyProducts/ClojureWasm/src/lang/builtins/collections.zig:961-1097` - + threadlocal at `src/runtime/dispatch.zig:64-67` — the - divergence anchor. -- JVM `RestFn.applyTo` at - `~/Documents/OSS/clojure/src/jvm/clojure/lang/RestFn.java:132-160` - — the semantics anchor. - -## Amendment 1 (2026-05-29) — the shape-only gate was a correctness bug - -**Status**: Accepted (Devil's-advocate fork landed 2026-05-29). - -The Alt-3 "gated bind-direct in `callFunction`" shipped a gate keyed on -the **shape** alone (`args.len == m.arity + 1 and isRestSeqShaped(...)`), -with the Decision claiming "the gate is generic — any caller benefits." -That claim is **false**: a NORMAL call passing a single seq-shaped -trailing arg has the identical shape to apply's spread but needs the -**opposite** binding. `((fn [a & xs] xs) 1 '(2 3))` must bind -`xs = ((2 3))` (cons-wrap the single list arg, per JVM Clojure), but the -shape-only gate bound `xs = (2 3)` — and `callFunction` cannot see its -caller, so it cannot distinguish "apply spread" from "literal single -list arg". Surfaced wiring `cljw.error/with-context` (row 14.13 / -ADR-0055): `(defmacro m [a & body] (cons 'do body))` with a single body -form produced `(do + 1 1)` instead of `(do (+ 1 1))`. - -**Fix.** The bind-vs-wrap decision becomes an explicit typed parameter, -not caller-invisible shape inference: - -- `callFunction` (the generic entry every backend reaches via - `vtable.callFn`) always cons-wraps — unconditionally correct. -- A dedicated `callFunctionBindingRest` is `apply`'s lazy-preserving - entry; `applyFn::canBindDirect` calls it directly (Layer 2 → Layer 1, - no vtable change). Both delegate to a private `callMethodImpl(…, - rest_mode: RestMode)` (`.wrap` / `.bind_direct`) — no frame-setup - duplication, no shared mutable state. - -This is **not** cw v0's `apply_rest_is_seq` threadlocal (which the -original Decision §"Why Alt 3" rejected on F-002 grounds as "shared -mutable state with temporal coupling"). The `RestMode` parameter is -in-band, type-checked, leak-free. The original Alt-1 (widen -`vtable.callFn` with `prebound_rest`) is also avoided — `applyFn` is in -`lang/` and calls `tree_walk` directly, so the vtable stays pristine for -F-001's Phase-16 integration. Verified: `((fn [a & xs] xs) 1 '(9 9))` → -`((9 9))`; `(apply (fn [a & xs] xs) 1 '(2 3))` → `(2 3)`; `(apply + -'(1 2 3 4))` → `10`. - -**Known residual (pre-existing, out of scope):** `canBindDirect` keys on -`v.arity == leading_count` and does not account for a fixed method whose -arity equals the *spread* length — `(apply (fn ([a b] :two) ([a & r] -:var)) x '(y))` binds-direct to the variadic where JVM spreads to -`(f x y)` → fixed `:two`. Single-method variadics (the common case) are -unaffected. Tracked as D-143. - -### Amendment 1 — Alternatives considered (Devil's-advocate, verbatim) - -> Constraints: F-002 (finished-form wins, diff not a constraint), F-009 -> (impl neutral), env.zig "threadlocals load-bearing not incidental". -> None of the three violate an F-NNN. -> -> **Alt 1 — Smallest-diff: the threadlocal flag (first fix attempt).** -> `pub threadlocal var apply_rest_prebound`; applyFn sets, callFunction -> reads+clears on entry. *Better*: zero signature churn, zero call-site -> touches; read+clear closes the leak window (verified). *Breaks*: the -> EXACT shape ADR-0042's own Decision (§"Why Alt 3", the cw v0 -> `apply_rest_is_seq` rejection) declined on F-002 grounds — re-adopting -> reverses that finding without superseding it. A transient one-bit -> round-trip = the textbook *incidental* threadlocal env.zig warns -> against. Latent: if a future `canBindDirect` edit lets the set-flag -> path reach a `builtin_fn`, the flag leaks to the next call — and that -> invariant lives in a different file from the clear. *F-NNN*: **F-002 -> negative.** -> -> **Alt 2 — Finished-form-clean: explicit sibling entry, generic path -> always wraps.** Delete flag + shape gate from the generic path (always -> cons-wraps). applyFn's direct-bind branch calls a -> `callFunctionBindingRest`; share a private frame-setup helper. -> *Better*: in-band typed signal, no thread-global, no read/clear -> window; generic path simpler + removes the `isRestSeqShaped` tag-set -> sync liability the original ADR flagged; does NOT widen -> `vtable.callFn` (applyFn Layer 2 imports tree_walk Layer 1 directly), -> so F-001 Phase-16 vtable property preserved. *Breaks*: a second -> frame-entry the VM must eventually mirror (ADR-0036) — visible/gated, -> diff-lockable, not a hidden thread-global. *F-NNN*: **F-002 strongly -> positive**, F-009 positive, F-001 neutral. -> -> **Alt 3 — Wildcard: apply owns spreading, callFunction owns wrapping.** -> Generic always wraps; apply calls a tiny `invokeVariadicWithRest`. -> *Better*: cleanest one-sentence invariant; no flag/gate/tag-sync. -> *Breaks*: duplicates the non-trivial frame epilogue (closure snapshot, -> AOT `evalChunk` hook, `deserialized_fn_body` guard) — a fresh F-002 -> liability unless the epilogue is shared, at which point it collapses -> into Alt 2. *F-NNN*: **F-002 mixed.** -> -> **Recommendation (non-binding): Alt 2.** The flag is correct/verified -> but finished-form-regressive — it re-adopts verbatim the shape this -> ADR rejected on F-002 grounds, and the invariant keeping its leak -> closed lives in a different file from the clear. Alt 2 makes the -> generic path unconditionally correct, deletes the gate + tag-set-sync, -> threads intent as in-band typed data, leaves the vtable pristine. Per -> F-002 the larger diff is not a reason to prefer the flag. - -The main loop adopted **Alt 2**, refined to a private -`callMethodImpl(…, rest_mode)` with two thin public wrappers — Alt 2's -in-band-typed signal with Alt 3's no-duplication (sharing the impl, not -a helper). The `canBindDirect ⇒ fn_val` invariant the DA flagged is -moot: applyFn calls `callFunctionBindingRest` directly, so no -`canBindDirect` edit can route a builtin into a bind-direct path. diff --git a/.dev/decisions/0043_zip_loc_record_shape.md b/.dev/decisions/0043_zip_loc_record_shape.md deleted file mode 100644 index 7bef9bd99..000000000 --- a/.dev/decisions/0043_zip_loc_record_shape.md +++ /dev/null @@ -1,292 +0,0 @@ -# 0043 — `clojure.zip` zipper representation: defrecord ZipLoc (Option B) - -- **Status**: Accepted (Devil's-advocate fork landed 2026-05-27) -- **Date**: 2026-05-27 -- **Author**: Shota Kudo (drafted with Claude autonomous loop) -- **Tags**: clojure-zip, defrecord, pattern-a, F-002, F-003, F-009, - D-080, D-075 - -## Context - -Phase 7 row 7.13 (D-080) opens `clojure.zip` Pattern A landing — 31 -vars (28 JVM-public + 3 cw v1-only predicates; DA fork amendment -expands to a 4-predicate set, see Decision below). A functional -zipper carries a "loc" pair of (current-node, navigation-context). -The navigation context records path / lefts / rights / end? so -`(up loc)` / `(left loc)` / `(right loc)` reconstruct ancestors and -siblings without re-walking the whole tree. - -JVM Clojure `clojure.zip` represents the loc as a **vector with -metadata**: `[node {:zip/path ... :zip/lefts ... :zip/rights ...}]`. -cw v0 (`~/Documents/MyProducts/ClojureWasm/src/lang/lib/clojure_zip.zig`, -608 lines) ports the JVM shape verbatim. The JVM-faithful shape -requires `with-meta` / `meta` / IObj / IMeta protocols + reader -`^{...}` form + per-Value metadata slot — all of which cw v1 tracks -as a Phase 7+ scope item (D-075, ≈500 LOC of its own; HeapHeader -layout review per F-004). - -The row 7.13 Step 0 survey -(`private/notes/phase7-7.13-survey.md`, 463 lines) enumerates the -representation choice and recommends Option B (defrecord ZipLoc). -The Devil's-advocate fork -(`private/notes/phase7-7.13-da-fork.md`, 492 lines, fresh context) -considers 3 alternatives within the F-NNN envelope and confirms -Option B with two amendments. This ADR records the decision. - -Substrate available at HEAD (rows 7.4 / 7.5 / 7.6 / 7.12 close): - -- ADR-0040 — `op_deftype` / `op_ctor_call` / `op_field_access` / - `op_method_call` opcodes (row 7.6 cycle 4). -- ADR-0041 — multi-arity `fn*` (row 7.8) + ADR-0042's apply - variadic peel-and-pass (row 7.9) unblock `(edit loc f & args)` - variadic. -- `defrecord` macro (row 7.4 cycle 1) + factory `(->Name ...)` - (cycle 4) + per-field accessors via `(.field instance)` - (row 7.6 cycle 1 + cycle 4 VM landing). -- `instance?` macro + `runtime/class_name.zig` registry that - includes user TypeDescriptor parent walk for typed_instance - (row 7.12 cycle 1) — `(instance? ZipLoc loc)` is the dispatch - predicate the public zipper API uses to validate inputs. -- `host_class.zig` Throwable hierarchy (row 7.11) — `(throw - (ex-info ...))` in `:else` arms is catchable as `ExceptionInfo`. - -D-085 (keyword-as-fn callable) is **not yet landed** (verified -2026-05-27: `(:a {:a 1})` raises "Cannot call value of type -'keyword'"). The xml-zip arm's `:content` lookup uses `(get node -:content)` workaround in cycle 1 — no PROVISIONAL marker needed -(`(get)` is fully working). - -## Decision - -Adopt **Option B — `(defrecord ZipLoc [node path lefts rights end?])`** -with the two Devil's-advocate amendments: - -1. **Forward-commitment statement**: the defrecord shape is the - **permanent finished form** of cw v1's zipper representation, - not a transient stopgap. Even after D-075 lands `with-meta` / - `meta` / IObj / IMeta, this ADR does NOT migrate to the - JVM-faithful vector-with-metadata shape. The migration would - churn every user-side `(instance? ZipLoc loc)` call + every - zipper-aware library + the public field-access surface - `(.node loc)` for marginal JVM-faithfulness gain. cw v1's - finished form keeps the defrecord shape; D-075 lands for - non-zipper consumers. -2. **Predicate-set rename + expansion**: the cycle 1 predicate - set becomes 4 instead of the survey's 3: - - `zip-loc?` (was the survey's `node-loc?` — renamed for - clarity: the predicate is "is this a ZipLoc?", not "is - this a loc-node?"). - - `seq-zip?` / `vector-zip?` (unchanged from survey). - - `xml-zip?` (new — symmetric with the other 3 source-shape - predicates). - -### `ZipLoc` shape (`src/lang/clj/clojure/zip.clj`) - -```clojure -(defrecord ZipLoc [node path lefts rights end?]) -``` - -- `node` — current node value (the user-visible "where" of the - zipper). -- `path` — parent `ZipLoc` or nil at root. Forms an upward - reverse-linked chain; `(up loc)` returns the parent loc with - the current node spliced back into the parent's child sequence - at position `(count (.lefts loc))`. -- `lefts` — vector of sibling nodes to the left of `node` in the - parent's child sequence. Empty vector at the leftmost sibling. -- `rights` — vector of sibling nodes to the right of `node`. - Empty vector at the rightmost sibling. -- `end?` — boolean marker. `(next loc)` sets this true when the - depth-first walk has exhausted; `(end? loc)` reads it. Allows - `(loop [loc z] (when-not (end? loc) ...))` idioms without an - out-of-band sentinel. - -The zipper-fn closures (`branch?`, `children`, `make-node`) are -NOT stored on the ZipLoc — instead `(zipper branch? children -make-node root)` returns a closure-bearing `ZipLoc` constructor -function, and the public API stores the 3 fns once per zipper- -type (e.g. `vector-zip` stores `(fn [x] (vector? x))` for -`branch?` etc.) in the cycle 1 `vector-zip` / `seq-zip` / -`xml-zip` constructor bodies. This avoids carrying 3 fn slots on -every ZipLoc instance — the slot cost would be wasted on the -vast majority of locs that share the same constructor's fns. - -### Public API surface (31 vars in 4 cycles) - -- **Cycle 1 (representation + ctors + leaves, ≈16 vars)**: - `zipper` / `vector-zip` / `seq-zip` / `xml-zip` / `node` / - `branch?` / `children` / `make-node` / `zip-loc?` / - `seq-zip?` / `vector-zip?` / `xml-zip?` / plus the underlying - defrecord + factory. -- **Cycle 2 (navigation, ≈7 vars)**: `down` / `up` / `right` / - `left` / `root` / `lefts` / `rights` / `path` / `leftmost` / - `rightmost`. -- **Cycle 3 (traversal, ≈3 vars)**: `next` / `prev` / `end?`. -- **Cycle 4 (mutation, ≈7 vars)**: `replace` / `edit` / - `insert-child` / `append-child` / `insert-right` / - `insert-left` / `remove`. -- **Cycle 5 (close)**: compat_tiers entries (31), placement.yaml - status flip, ROADMAP `[x]`, handover. - -## Alternatives considered - -(Devil's-advocate subagent output condensed from -`private/notes/phase7-7.13-da-fork.md`; the subagent's -recommendation is Alt 2 with the two amendments above, adopted -verbatim by the main loop.) - -### Alt 1 — Smallest-diff: vector + minimal-IObj shim - -JVM-faithful representation: `[node {:zip/path … :zip/lefts … -:zip/rights …}]`. cw v1 lacks `with-meta` / `meta`, so the path -forward is either to land the full D-075 system (~500 LOC, -out-of-scope for row 7.13) or to ship a minimal IObj-equivalent -JUST for zipper (e.g. per-vector tail-meta slot with only the -keys clojure.zip uses). - -- **Surface**: ~600 LOC (full minimal-IObj impl + zipper defns). -- **Better**: JVM-faithful — line-for-line port of `zip.clj`; - upstream zipper-aware libraries port without modification; - future Clojure conformance tests on zipper land without - shimming. -- **Worse**: F-003 violation — the minimal-IObj shim **seizes the - D-075 / F-004 owner's structural decision** (HeapHeader layout - review for the meta pointer). Survey + DA both flag this as - the headline reason to reject Alt 1; the per-nav `~3×` - hashmap-probe cost (Group A array_map / hash_map lookup vs - ZipLoc field access) and the "skeleton enlarges the future - rewrite" smell are secondary. -- **F-NNN: violates F-003** (premature structural decision in a - D-075 / F-004 owner's territory). Neutral on F-001 / F-002 / - F-006 / F-009. - -### Alt 2 — Finished-form-clean: defrecord ZipLoc (ADOPTED) - -(Decision section above.) The survey's Option B. - -### Alt 3 — Wildcard: dedicated `HeapTag.zip_loc` slot - -Promote ZipLoc to a first-class Tag rather than routing through -`.typed_instance`. Maximum cache density (no descriptor pointer -chase per field access; `node` reads a fixed offset). - -- **Surface**: ~250 LOC (zip_loc HeapTag + trace fn + finaliser - + per-field accessor + 31 defns). One F-004 NaN-box slot - consumed. -- **Better**: fastest per-op access; "zip is core enough to - warrant a slot" finished form for zipper-heavy workloads. -- **Worse**: **violates F-003 + F-004**. F-004 budget is finite - (12 slots remaining per the second-generation layout); spending - one on a single library function set sets a precedent that - exhausts the budget by Phase 8+. Also weakens F-009 — impl - moves into Zig (`runtime/collection/zip_loc.zig`) when the - whole point of Pattern A is .clj-side migration. -- **F-NNN: violates F-003 + F-004**. - -### Pre-finding (out-of-envelope, recorded leading per CLAUDE.md DA mandate) - -**`atom`-keyed sidecar (DA wildcard, sub-variant)**: a global -`atom`-keyed-by-node-identity hashmap could carry path state -hidden from user code entirely, giving a clean public API -without per-loc state. Survey's DA flagged: cw v1's `atom` -Layer-2 primitive is **NOT wired** (HeapTag slot 32 reserved but -no `atomFn` registration in `src/lang/primitive/`). Substrate -gap excludes this even as a wildcard. Recorded so the main loop -sees it but does not halt. - -### F-NNN comparison summary - -| Option | F-001 | F-002 | F-003 | F-004 | F-006 | F-009 | -|-----------------|-------|-------|-------|-------|-------|-------| -| Alt 1 | 0 | = | **-** | 0 | 0 | 0 | -| Alt 2 (ADOPTED) | 0 | + | + | 0 | 0 | + | -| Alt 3 | 0 | = | **-** | **-** | 0 | - | - -Alt 1 vs Alt 2 is a genuine F-002 ambiguity (JVM-faithful for -library compatibility vs defrecord for v1 internal consistency). -F-003 breaks the tie cleanly in Alt 2's favour. - -## Consequences - -### Positive - -- D-075 (with-meta / IObj / meta runtime) stays a separate Phase - 7+ entry decision; row 7.13 does not seize that owner's choice. -- defrecord substrate (rows 7.4 / 7.6) gets a meaningful Pattern A - consumer beyond the diff_test cases, proving the dispatch + - field-access path end-to-end on a real Clojure surface. -- `(instance? ZipLoc loc)` works out of the box via row 7.12 - cycle 1's class_name registry (user TypeDescriptor parent - walk). -- The 4-predicate set (`zip-loc?` / `seq-zip?` / `vector-zip?` / - `xml-zip?`) gives users symmetric query surface — JVM has - only the underlying instance check; cw v1's richer set is a - cw-original ergonomic improvement. -- `cycle 4`'s `(edit loc f & args)` variadic rides ADR-0041 - (multi-arity `fn*`) + ADR-0042 (apply variadic peel-and-pass) - — no follow-up debt. - -### Negative - -- Non-JVM-faithful representation: users porting zipper-aware - libraries from JVM Clojure must rewrite any code that calls - `(meta loc)` to inspect internal zipper state. Mitigation: the - public clojure.zip API (`up` / `down` / `node` / `path` etc.) - is identical to JVM — only internal-state inspection diverges. - Most user code uses the public API only. -- The forward-commitment statement above means future re-reads - of this ADR cannot re-open the JVM-faithful migration question - without a new ADR amendment chain. This is deliberate per the - DA's amendment (A): every future re-read re-opens the question - otherwise. - -### Deferred - -- D-075 `with-meta` / `meta` runtime is independent of this ADR. - Its landing does NOT trigger any ZipLoc migration. -- `xml-zip`'s `:content` keyword-as-fn lookup → use `(get node - :content)` workaround in cycle 1 (no PROVISIONAL marker - needed — `(get)` works fully). D-085 keyword-as-fn closure is - opportunistic; once landed, the cycle 1 `(get node :content)` - call can opportunistically flip to `(:content node)` for - ergonomic uniformity. - -## Cross-references - -- ROADMAP §9.9 row 7.13 (this row's task table entry). -- D-080 (`.dev/debt.md`) — `clojure.zip` 28 vars Pattern A - landing. ADR-0043 carries the representation decision + - expands the var count to 31 (4-predicate set per DA - amendment B). -- D-075 — `with-meta` / IObj / IMeta runtime. Permanently - unrelated to this ADR per the forward-commitment statement. -- D-085 — keyword-as-fn callable. Opportunistic discharge - trigger for the `(get node :content)` → `(:content node)` - ergonomic upgrade in `xml-zip`. -- ADR-0040 — `op_method_call` (the field-access path cycle 4's - `(edit loc f ...)` uses for `(.node ...)` reads). -- ADR-0041 — multi-arity `fn*` (cycle 4 `edit`'s variadic - substrate). -- ADR-0042 — apply variadic peel-and-pass (cycle 4 `(edit loc f - & args)` calls `(apply f node args)` under the hood). -- F-002 (`.dev/project_facts.md`) — finished-form cleanliness; - the rejection of Alt 1's smallest-diff JVM-faithful path - rides on this. -- F-003 — decision-deferral on structural plans; the headline - reason for Alt 1's rejection (would seize D-075 / F-004 - owner's territory). -- F-004 — NaN-box 64-slot layout; the headline reason for Alt 3's - rejection (would consume a precious slot for a single - library). -- F-009 — feature-implementation neutrality; Alt 2 keeps impl in - `.clj`, aligning with the F-009 surface-layering aesthetic. -- `private/notes/phase7-7.13-survey.md` — Step 0 survey. -- `private/notes/phase7-7.13-da-fork.md` — Devil's-advocate - enumeration; this ADR's "Alternatives considered" section is - the DA output condensed. -- `~/Documents/OSS/clojure/src/clj/clojure/zip.clj` — JVM - Clojure semantics anchor (=Alt 1 baseline). -- `~/Documents/MyProducts/ClojureWasm/src/lang/lib/clojure_zip.zig` - (608 LOC) — cw v0 implementation. NOT a clean template for - v1 because v0 had the with-meta system landed; ADR-0043 - explicitly diverges from v0's representation. diff --git a/.dev/decisions/0044_bench_history_schema.md b/.dev/decisions/0044_bench_history_schema.md deleted file mode 100644 index 9898c89af..000000000 --- a/.dev/decisions/0044_bench_history_schema.md +++ /dev/null @@ -1,245 +0,0 @@ -# 0044 — `bench/history.yaml` schema (per-commit aggregate with machine + distribution) - -> **ADR-number history note**: this ADR was originally drafted at -> slot `0027` (commit `8678052`, 2026-05-27 row 8.2) but collided -> with `0027_nan_box_second_generation.md` (issued 2026-05-24 row -> 5.2.b). The Phase 8 → 9 boundary audit caught the collision; ADR -> numbers are time-ordered `max+1` at issue per CLAUDE.md § -> Project spirit. Renumbered to `0044` at the Phase 8 → 9 audit -> absorption commit. Callers (`.dev/handover.md`, `.dev/ROADMAP.md`, -> `bench/history.yaml`, debt rows) were updated in the same commit. - -- **Status**: Accepted (Devil's-advocate fork landed 2026-05-27) -- **Date**: 2026-05-27 -- **Author**: Shota Kudo (drafted with Claude autonomous loop) -- **Tags**: bench, regression-gate, schema, F-001, F-002, F-003, D-005 - -## Context - -Phase 8 row 8.2 opens `bench/history.yaml` — the curated reference -layer the row 8.3 1.2x regression gate compares against. Today cw -v1 has: - -- `bench/quick.sh` (113 LOC) — CI-time bench runner; appends TSV - samples to `bench/quick_baseline.txt`. -- `bench/quick_baseline.txt` (5 789 lines at HEAD `d810322`) — flat - TSV log of ` ` rows. Mac - and OrbStack Linux samples interleave with no machine - differentiation. No commit SHA, no backend field, no σ-data. -- 5 `bench/fixtures/.clj` programs feeding `quick.sh`'s 5 - fixture benches + 4 derived metrics (binary size, cold start, - e-plus round-trip, read-100-forms). - -The 5 789-line append-log is fine as **high-volume telemetry** but -the row 8.3 gate needs a **curated lock layer** carrying: -- Per-(machine, backend) baseline numbers. -- σ-data (median + p99 + samples) so D-005 Phase-17 σ-gate - refinement consumes the same source without re-collection. -- Commit + reason annotation so the gate's diagnostic can say - "regression vs lock-point `8A.0` from `d810322`". - -Row 8.2's Step 0 survey -(`private/notes/phase8-8.2-survey.md`, 408 lines) recommends cw v0 -shape (c) "per-commit aggregate" with two cw v1-specific -amendments (machine bucket + distribution stats). The -Devil's-advocate fork -(`private/notes/phase8-8.2-da-fork.md`, 336 lines, fresh context) -considered three alternatives within the F-NNN envelope, confirms -shape (c) with no further amendment beyond noting one operational -invariant for the future `bench/record.sh` helper. - -cw v0 (`~/Documents/MyProducts/ClojureWasm/bench/history.yaml`, -1 149 LOC over 42 entries) ships shape (c) directly. cw v1 forks -the schema for the two amendments below. - -## Decision - -Adopt **shape (c) per-commit aggregate** for `bench/history.yaml` -with the cw v1-specific amendments noted below. Curated by an -explicit `bench/record.sh --id= --reason=` helper -(landed in cycle 2 alongside the row 8.3 gate); the CI -`quick.sh` append flow is unchanged. - -### Schema - -```yaml -# bench/history.yaml -schema_version: 1 # increments on breaking schema change -env: # file-global default machine (= row's omit case) - cpu_arch: aarch64 # narrow info; per-entry `machine:` overrides - os: Darwin - tool: bash + EPOCHREALTIME (Zig hyperfine equivalent at Phase 17+) -entries: - - id: "8A.0" # short curated label (PhaseLetter.cycle) - date: "2026-05-27" # YYYY-MM-DD - reason: "Phase 8 row 8.2 — initial lock-point for the new schema" - commit: "d810322" # short SHA at lock time - build: ReleaseFast # default for quick.sh - backend: tree_walk # one of: tree_walk, vm, (future: wasm) - machine: # per-entry machine bucket (cw v1 amendment) - id: mac-arm-m4pro # canonical short id for the gate to key on - cpu: Apple M4 Pro - cpu_arch: aarch64 - os: Darwin 25.5.0 - cores: 12 - lock: true # row 8.3 gate consumes only `lock: true` entries - results: # per-bench distribution (cw v1 amendment) - fib_recursive: - samples_us: [3200, 3180, 3220, ...] # raw samples for σ-gate - median_us: 3200 - p99_us: 3260 - n: 50 - arith_loop: { ... } - ... -``` - -### cw v1 amendments vs cw v0 shape (c) - -1. **`machine:` per-entry block** (cw v0 had file-global only). - Required: cw v1 runs the gate on both Mac (aarch64-darwin) and - OrbStack Ubuntu x86_64 per the dual-host discipline; bench - numbers cross the two hosts by >5×, so cross-machine - normalisation is infeasible. The gate keys on - `(machine.id, backend)` tuples. -2. **`results..{samples_us, median_us, p99_us, n}`** instead - of cw v0's mean-only. Preserves σ-data so D-005 Phase-17 - σ-gate refinement (an existing forward debt row) consumes - `history.yaml` directly without re-collection. Median (not - mean) per bench because cold-cache outliers in `cold_start_us` - skew the mean meaningfully. - -### Lock mechanism - -- **In-band `lock: true` flag** on the entry. The row 8.3 gate - iterates `entries:` for entries with `lock: true` matching the - current (machine.id, backend), compares the latest - `quick_baseline.txt` block's median against the lock's - `median_us` per bench, raises on >1.2× regression. -- Curated by `bench/record.sh --id= --reason=` (lands - cycle 2). Auto-CI never appends to `history.yaml`; only the - explicit helper does. This keeps `history.yaml` curated + - small (cw v0's 42 entries over 24 phases ≈ 1.7 entries/phase). - -### `reason:` field duplication invariant - -When an `id:` spans multiple `(machine, backend)` entries (e.g. -the same lock-point recorded for Mac tree_walk + Mac vm + -Linux tree_walk + Linux vm), the `reason:` field repeats. cw v1's -`record.sh` enforces: `--id=X --reason=Y` always writes the same -`(id, reason)` pair across all 4 entries; reviewer reads only the -first one. Future depth-3 concern (reshape to lock-point-level -hoisting) deferred to a new debt row if duplication actually -bites at Phase 12+. - -### Two-file role split - -- `bench/quick_baseline.txt` — CI telemetry. Cheap append per - bench run. The row 8.3 gate reads only the **latest block** for - median computation. -- `bench/history.yaml` — curated locks. Only `record.sh` writes. - Row 8.3 gate reads the matching lock-point per (machine, - backend) for comparison. - -Survey's "do not delete `quick_baseline.txt`" advice: the -historical samples stay valuable as forensic data even after the -gate switches to consuming `history.yaml`. The two layers are -complementary. - -## Alternatives considered - -(Devil's-advocate fork condensed from -`private/notes/phase8-8.2-da-fork.md`.) - -### Alt 1 — Flat-record list - -`entries: [{commit, date, bench_name, machine_id, backend, -sample_us}, ...]`. One row per (commit, bench, sample). Survey's -"shape (a)". - -- Surface: ~450 LOC across schema + record.sh + gate. -- Better: simplest yq pipeline; append-friendly; smallest-diff - from the current TSV log. -- Worse: **no raw samples means D-005 Phase-17 σ-gate refinement - cannot consume `history.yaml`** (would need to re-collect - samples after the schema lands). This conflicts with F-002 — - the finished form requires σ-data preservation. DA's load- - bearing objection. -- **F-002: violates** (forces a second migration at Phase 17). -- F-001 / F-003 / F-006 / F-009: neutral. - -### Alt 2 — Per-commit aggregate (ADOPTED, = survey shape c) - -(Decision section above.) - -- Surface: ~475 LOC across cycle 1 + 2. -- F-002: positive (finished form, no Phase-17 migration). - -### Alt 3 — Per-file split (`bench/history/.yaml`) - -One file per lock-point id. Git-native diff, no YAML merge -conflicts, O(1) atomic append. - -- Surface: ~395 LOC (slightly less than Alt 2 because no `yq -i` - edit dance). -- Better: file-per-lock means merge conflicts on parallel work - don't accumulate; reviewer reads one file per lock. -- Worse: file count grows linearly with locks (cw v0's 42 entries - → 42 files). Diverges from cw v0's reviewer muscle memory. - Phase 16+ wasm divergence is better solved by a second file - (cw v0 has a `wasm_history.yaml` precedent), not per-lock - split. -- F-NNN: neutral on all 5 dimensions; rejected for operational - continuity. - -## Consequences - -### Positive - -- Row 8.3's 1.2x gate has a deterministic per-(machine, backend) - comparison source. -- D-005 Phase-17 σ-gate refinement consumes `history.yaml` - directly (no second-pass migration). -- Curation via `record.sh` keeps the file small + reviewer- - friendly (vs auto-append from CI). -- cw v0's 42-entry track-record validates the schema's - long-horizon scaling (~1 MB by Phase 20 per the DA's cost - projection). - -### Negative - -- `reason:` field repeats across (machine, backend) variants of - the same `id:` — accepted as the `record.sh` invariant - (depth-1 mitigation per DA). -- The schema is structural; future divergence (e.g. JIT bench at - Phase 17) needs a `schema_version: 2` bump + migration. - -### Deferred - -- `bench/record.sh` helper — cycle 1 lands the schema + first - lock entry; the helper script lands at cycle 2 (= row 8.3's - cycle 1 alongside the gate). -- `scripts/check_bench_regression.sh` — row 8.3 cycle 1. -- D-005 σ-gate (Phase 17 σ-aware gate) — independent of this ADR. -- Wasm-bench schema variant (F-001 Phase 16+ entry) — assumed to - re-use this schema with a new `backend: wasm` value; - schema_version bump only if a new field is required. - -## Cross-references - -- ROADMAP §9.10 row 8.2 (this row's task table entry). -- ROADMAP §10 (Performance) — original bench-gate target. -- ADR-0005 — dual-backend differential (row 8.4 full-bench remit - rides on this schema). -- ADR-0023 — Phase-gated comptime bools (the row-8.4 `--compare` - may surface as a Phase-N flag; out of scope here). -- F-001 — zwasm v2 unavoidable (forward-fitness check passed: - schema absorbs `backend: wasm` without breaking shape). -- F-002 — finished-form cleanliness (decisive rejection of Alt 1). -- F-003 — decision-deferral on structural plans (row 8.2 is the - owning Phase entry for the schema decision). -- D-005 — Phase-17 σ-gate refinement (downstream consumer of - the `samples_us:` field this schema preserves). -- `private/notes/phase8-8.2-survey.md` — Step 0 survey. -- `private/notes/phase8-8.2-da-fork.md` — Devil's-advocate fork. -- cw v0 `~/Documents/MyProducts/ClojureWasm/bench/history.yaml` - (1 149 LOC over 42 entries) — schema lineage anchor. diff --git a/.dev/decisions/0045_phase9_scope_modules_externals.md b/.dev/decisions/0045_phase9_scope_modules_externals.md deleted file mode 100644 index 01976b137..000000000 --- a/.dev/decisions/0045_phase9_scope_modules_externals.md +++ /dev/null @@ -1,140 +0,0 @@ -# 0045 — Phase 9 scope reconciliation: external Clojure modules (not "protocols + multimethods + deep interop") - -- **Status**: Accepted -- **Date**: 2026-05-27 -- **Author**: Shota Kudo (drafted with Claude autonomous loop) -- **Tags**: roadmap, amendment, F-002, Phase-9, Phase-10 - -## Context - -ROADMAP §9 master table row 9 originally read: - -> | 9 | Protocols + Multimethods + Interop deep module | -> | defprotocol / defmulti work; single Interop module complete | - -This wording dates to the pre-Phase 7 ROADMAP draft, when the -Phase order placed the protocol + multimethod + host interop -work at Phase 9. The subsequent ADR cluster (ADR-0008 protocol -dispatch / ADR-0029 java + cljw surface layout) pulled that -scope **forward** into Phases 6 + 7: - -- Phase 6 (§9.8, closed 2026-05-26) landed `host_classes` first - wave: `java.util.UUID` / `java.io.File` / `java.time.Instant` - / `java.util.Random` / `java.lang.System` (= row 6.10+ of - §9.8) via ADR-0029's surface convention. -- Phase 7 (§9.9, closed 2026-05-27) landed defprotocol / - defmulti / deftype / defrecord / reify / `(.method obj args)` - protocol method dispatch (= rows 7.1-7.7 of §9.9). - -By the time Phase 9 (§9.11) opened on 2026-05-27, the original -Phase 9 mission was effectively discharged across Phases 6-7. -The Phase 9 entry owner (this autonomous-loop session) faced -the choice: - -1. Wait for protocol / multimethod / host interop work to - re-emerge → no scope candidates surfaced; the work was done. -2. Reuse the slot for the next-natural Tier-A surface that the - master table did NOT enumerate but the per-phase placeholder - notes flagged → "external Clojure modules" (json / csv / edn - / tools.cli) per `.dev/structure_plan.md:108-112` + D-034. - -The Phase 9 entry note (§9.11 placeholder, pre-amendment) said: - -> "the Deliverables line below currently reads 'protocol / host -> complete behaviours' which overlaps the Phase 7 (§9.9) -> protocol dispatch + Phase 6 (§9.8) host stdlib first-wave -> content. Reconcile the Phase 9 scope to match its actual -> focus (external Clojure modules — json / csv / edn) before -> opening the task table; the current text is a historical -> artefact from the pre-amendment ROADMAP." - -The Phase 9 entry owner picked option (2). Phase 9 landed: - -- row 9.1 — `modules/` top-level + zone rule (D-034 discharged) -- row 9.2 — `clojure.edn/read-string` + formToValue lift -- row 9.3 — `clojure.data.json/{read-str,write-str}` -- row 9.4 — `clojure.data.csv/{read-csv,write-csv}` RFC 4180 -- row 9.5 — `clojure.tools.cli/parse-opts` minimum -- row 9.6 — exit smoke + D-007 self-host viability discharged - -This ADR formalises the §9 master-table edit that the actually- -landed Phase 9 demands. - -## Decision - -Amend `.dev/ROADMAP.md` §9 master table row 9 in place per -§17.2 four-step amendment. New wording: - -> | 9 | Modules layer + external standard libraries (clojure.edn / .data.json / .data.csv / .tools.cli) | -> | All 4 external Tier-A namespaces ship; `modules/` top-level dependency rule enforced; self-host viability re-verified | - -The original protocol / multimethod / interop scope is **not -lost** — it landed across Phases 6 + 7 ahead of schedule. §9 -master table rows 6 + 7 already describe their realised content -correctly; only row 9 was mismatched. - -## Why now (Phase 10 row 10.1) - -The Phase 9 → 10 boundary audit flagged this as a medium -finding (not block-severity because the actually-landed Phase 9 -content is captured correctly in §9.11). Row 10.1 of §9.12 -schedules the §17 amendment as Phase 10's first task so the -master-table row stops contradicting the per-phase expansion. - -Per §17.1: this qualifies as "an exit form / scope row / -task description references a feature whose implementation is -scoped to a different phase" — exactly the canonical case for -the in-place amend. - -## Alternatives considered - -**Alt 1 — Leave row 9 as-is**, document the divergence in -handover only. Rejected: violates §17.3 "Editing ROADMAP -without an accompanying ADR for load-bearing changes (anything -in §1, §2, §4, §5, §9 phase rows...)". The §9 row IS a -load-bearing surface that downstream readers (= future cw v1 -maintainers, audit_scaffolding sweep) trust as the canonical -phase mission statement. - -**Alt 2 — Add a "Revision history" annotation to row 9**. -Rejected: §17.2 forbids inline change-bars / dated comments / -strikethrough. The §17 doctrine is "write as if it had always -been so"; the ADR is the changelog. - -**Alt 3 — Rename Phase 9 to a different number**. Forbidden by -§17.3 "Quiet renumbering of §N headings". Phase 9 stays -Phase 9; only its mission row text changes. - -## Consequences - -**Positive**: -- §9 master table becomes truthful at the granular row level. -- `audit_scaffolding` Phase-boundary sweeps no longer flag the - row 9 mismatch. -- Phase 10 entry owner can reference §9 row 9 directly when - discussing what's actually built. - -**Negative**: -- A non-load-bearing concern: external readers comparing this - ROADMAP to earlier git revisions (or to mirrors elsewhere) - see the row 9 wording change. The ADR's "Context" section - preserves the original wording for that audit trail. - -## Affected files - -- `.dev/ROADMAP.md` §9 master-table row 9 (this commit's diff) -- This ADR file (`0045_phase9_scope_modules_externals.md`) -- `git log -- .dev/ROADMAP.md` will show the amendment - referencing ADR-0045 - -## References - -- §17 Amendment policy (the procedure this ADR satisfies) -- §9.11 Phase 9 task list (the actually-landed content) -- §9.12 Phase 10 row 10.1 (the task that drives this ADR) -- ADR-0029 (Java + cljw surface layout — the carrier ADR for - the host interop work that landed across Phases 6 + 7) -- ADR-0008 (Protocol dispatch — the carrier ADR for protocol / - multimethod work that landed across Phase 7) -- Original Phase 9 mission text (preserved in this ADR's - "Context" section) diff --git a/.dev/decisions/0046_upstream_skip_taxonomy.md b/.dev/decisions/0046_upstream_skip_taxonomy.md deleted file mode 100644 index 6f3995755..000000000 --- a/.dev/decisions/0046_upstream_skip_taxonomy.md +++ /dev/null @@ -1,135 +0,0 @@ -# 0046 — Upstream skip taxonomy + Tier A 100% PASS gate semantics - -- **Status**: Accepted -- **Date**: 2026-05-27 -- **Author**: Shota Kudo (drafted with Claude autonomous loop) -- **Tags**: F-002, Phase-11, test-corpus, Tier-A, gate - -## Context - -Phase 11 ports the upstream Clojure test corpus -(`~/Documents/OSS/clojure/test/`) under `test/clj/` and wires -the Tier A 100% PASS gate into `test/run_all.sh`. The gate has -two failure modes the loop must distinguish: - -1. **A cw v1 bug** — the ported test fails because cw v1's - implementation diverges from JVM semantics. -2. **An intentional skip** — the ported test references a JVM - feature cw v1 has explicitly deferred or excluded (Tier B / - C / D per ADR-0013). - -Without a structured skip taxonomy, every Tier B/C/D-touching -test is a false-positive gate failure that the loop must -"explain away" in commit messages or skip comments. That -accumulates into the same opaque ad-hoc state the per-phase -expansions were designed to prevent. - -§9.13 placeholder originally named the deferred ADR -"ADR-0025"; that slot collides with ADR-0025 (chapter archive -boundary, 2026-05-25). Per CLAUDE.md § Project spirit ("ADR -numbers are time-ordered max+1 at issue time"), the upstream -skip taxonomy ADR mints as **ADR-0046** (next free slot at -issue time). - -## Decision - -Land `test/clj/skip_taxonomy.yaml` as the SSOT for which -upstream tests are skipped from the Tier A gate and why. -Schema: - -```yaml -version: 1 -last_updated: YYYY-MM-DD - -skipped: - - test_file: "clojure/test/test_helpers.clj" - test_name: "test_something" - tier: D # A | B | C | D (per ADR-0013) - reason: "Requires gen-class which is Tier D per ADR-0013." - recall_trigger: "(none — Tier D permanent)" - last_reviewed: "YYYY-MM-DD" -``` - -- `tier:` field maps to ADR-0013's classification — A (in scope, - bug), B (recommended), C (Tier C reflective), D (permanent - exclusion). -- `reason:` is a sentence the gate prints when the test is - skipped, sufficient for the user to triage without reading - this ADR. -- `recall_trigger:` is the testable predicate that, when true, - re-includes the test (typically "when D-NNN closes" or - "(none — Tier D permanent)"). -- `last_reviewed:` is the date a human re-confirmed the skip is - still appropriate. `audit_scaffolding` periodic sweeps flag - entries older than 90 days for re-evaluation. - -The Tier A gate (`test/run_all.sh::test_clj`) reads -`skip_taxonomy.yaml`, runs only tests NOT in the skipped set, -and asserts 100% PASS. A failing test that is not in the -skip set is a gate failure (loud red). A test in the skip set -that suddenly starts passing is a separate audit row (not a -gate failure but worth surfacing — "the recall_trigger may -have closed"). - -## Why now (Phase 11 row 11.1) - -Row 11.2 (clojure.test minimum surface) ships the -`deftest` / `is` / `run-tests` macros that the ported tests -will use. Row 11.3 (porting upstream tests) needs the skip -taxonomy active before the gate can usefully count "Tier A -100% PASS". The taxonomy SCHEMA mints here so rows 11.3 + 11.4 -can reference an existing SSOT shape from the moment they open. - -The actual `skip_taxonomy.yaml` FILE is created at row 11.3 -when the first ported test surfaces a skip-worthy case. - -## Alternatives considered - -**Alt 1 — Inline skip comments in each ported test**. Rejected: -matches what cw v0 did, doesn't scale; the loop has no -machine-readable way to count "Tier A pass rate" or to detect -stale skips. The yaml SSOT is the single point of truth. - -**Alt 2 — Use upstream's `:test/skip` metadata directly**. -Partial: JVM's test metadata is per-fixture annotation; cw v1's -ported tests would need to carry the same metadata. The skip -taxonomy yaml CAN coexist with `:test/skip` metadata — for -cycle 1, the yaml is the SSOT and `:test/skip` is not parsed. - -**Alt 3 — Reuse `feature_deps.yaml` for skip tracking**. -Rejected: `feature_deps.yaml` already tracks PROVISIONAL -behaviour with a different lifecycle. Mixing test-corpus skip -rows in there bloats its schema. A separate yaml is cleaner. - -## Consequences - -**Positive**: -- Gate failures are unambiguous (true bug vs intentional skip). -- `audit_scaffolding` can sweep `skip_taxonomy.yaml` for stale - entries the way it sweeps `debt.md`. -- Future cycles that close a `D-NNN` can grep - `skip_taxonomy.yaml` for matching `recall_trigger` rows and - re-include the affected tests automatically. - -**Negative**: -- One more SSOT file to maintain. Mitigation: only update when - porting a test that needs a skip; otherwise it lives - read-only. - -## Affected files - -- `.dev/decisions/0046_upstream_skip_taxonomy.md` (this file) -- `test/clj/skip_taxonomy.yaml` (created at row 11.3 first - skip-worthy port) -- `test/run_all.sh` `test_clj` step (added at row 11.4) -- `.dev/debt.md` D-NNN rows that the Tier A gate's - `recall_trigger` field references - -## References - -- ADR-0013 (Tier D permanent — the tier classification this - taxonomy uses) -- ADR-0021 (Test layer taxonomy — Layer 5 Conformance opens - at Phase 11) -- §9.13 Phase 11 task list (rows 11.2 / 11.3 / 11.4 use this - ADR's SSOT) diff --git a/.dev/decisions/0047_vm_peephole_optimizer.md b/.dev/decisions/0047_vm_peephole_optimizer.md deleted file mode 100644 index 030d074fa..000000000 --- a/.dev/decisions/0047_vm_peephole_optimizer.md +++ /dev/null @@ -1,186 +0,0 @@ -# 0047 — VM peephole optimizer architecture (Phase 13) - -- **Status**: Accepted -- **Date**: 2026-05-28 -- **Author**: Shota Kudo (drafted with Claude) -- **Tags**: phase-13-entry, vm, optimizer, peephole, differential-oracle - -## Context - -ROADMAP master-table row 13 (Phase 13) = `peephole.zig`: five -canonical benchmarks within 110% of cw v0 24C.10. cljw needs a -post-compile optimizer over the VM `Instruction` stream that -preserves the ADR-0005 differential oracle (TreeWalk ≡ VM Value). - -Phase 17's `super_instruction.zig` (100% target) is a **separate** -pass — instruction *fusion* (combining ops into new fused opcodes) -is Phase 17, NOT peephole. cw v0's so-called "peephole" -(`src/engine/compiler/compiler.zig:1387-1600`) was entirely fusion -(`add_locals` / compare-and-branch); only its jump-target-collection -+ IP-remap *machinery* is reusable here, not its fusion patterns. - -Step 0.6 (re-laying against finished-form) found that the Step 0 -survey's recommended first rule — `op_jump +0` elision — is a -**phantom**: cljw's codegen never emits a jump-to-next. `compileIf` -always emits an else branch (`op_const nil` when none) so its -forward jump skips ≥1 instruction; `compileTry` always has the -re-raise `op_throw` between the last catch clause's end-jump and -`end:`, so that jump is offset ≥1 too. An `op_jump +0` rule would -remove nothing from real programs. The actually-demonstrable -redundancy is **`compileDo`'s `; op_pop` after every -non-final form** (`compiler.zig:170-172`): `(do 1 2)` compiles to -`op_const c1; op_pop; op_const c2`, and the pure-push-then-discard -pair `op_const c1; op_pop` is removable. - -## Decision - -1. **File**: `src/eval/backend/vm/peephole.zig` — sibling to - `compiler.zig` / `opcode.zig`, Layer-1 (`eval/`), imports only - `opcode.zig`. No `optimize/` subdirectory yet (a one-file dir - is premature reservation; promote to `optimize/` when Phase 17's - `super_instruction.zig` joins it). - -2. **Runs inside `compiler.compile`** (last step, in/before - `finalize()`), and **recurses into fn sub-chunks** - (`compileFnMethodBody` produces a `*BytecodeChunk` per method — - hot loops live in fn bodies, e.g. `fib_recursive` / `arith_loop`). - Callers (`driver.zig`, `evaluator.compare`) get optimized - bytecode transparently — no caller knows the optimizer exists. - The Phase-12 serializer therefore caches the **optimized** chunk - (correct: the cache's contract is "skip recompilation"; caching - pre-optimization bytecode would defeat or invert the cache). - -3. **Core primitive `applyPlan(keep_mask)`** — the load-bearing, - reused IP-remap. Rules populate an arena-sized `[]bool keep` - mask (sized to `instructions.len`, never cw v0's fixed-64K stack - blob). `applyPlan` then: builds a prefix-sum `old→new` index map, - compacts the instruction slice, and re-resolves the three - position-referencing operands — `op_jump` / `op_jump_if_false` / - `op_push_handler`, all signed-i16-offset-relative-to-next - (`vm.zig:188-201,317`) — by mapping each old absolute target to - its new index and recomputing the offset against the op's new - position. Targets are resolved in old-index space first, then - mapped, then re-differenced (order-independent — no in-place - mutation while offsets are stale). `call_sites` / `libspecs` are - operand-indexed side-tables, untouched by instruction removal. - - **Physical removal is the only honest optimization for cljw's - current opcode set.** An index-stable `op_nop` null-out was - rejected: a nop still costs a dispatch turn (`vm.zig` `ip += 1` + - switch), so it optimizes nothing (permanent-no-op smell). No - index-stable *real* optimization exists for the current ops, so - the IP-remap is unavoidable; deferring it = deferring Phase 13. - -4. **First rule: pure-push + `op_pop` elision** — a *pure push* - immediately followed by `op_pop` is removable (net stack effect - 0, no side effect, no observable). Purity is an **exhaustive - `Opcode.isPurePush()` switch** in `opcode.zig`, NOT a - hand-maintained allowlist in `peephole.zig`: adding a new opcode - forces a purity decision at compile time. This closes the real - hazard — forgetting to add a pure op only loses optimization - (safe), but mis-classifying a *non-pure* op as pure (e.g. - `op_get_var`, which derefs a dynamic var and can throw) would - silently drop a side effect and break ADR-0005. Phase 13's pure - set is exactly `{ op_const, op_load_local }`. - -5. **No `Rule` registry yet.** The first rule is a plain function - (`markPurePushPop`). A one-entry data-driven rule registry is an - excessive skeleton / reservation-as-bias (F-002) — the registry - interface should emerge when rule 2 reveals what it actually - needs, not be speculated on one rule. - -6. **Correctness guards**: skip a removal if either removed index is - a branch target (do not orphan an incoming edge). The ADR-0005 - differential oracle is the end-to-end check; Layer-1 unit tests - assert the remapped instruction stream directly; Layer-3 `diff` - cases exercise a removal followed by a downstream jump so the - IP-remap is proven on a real `.clj` form. - -This is NOT a new analyzer Node variant, so the ADR-0036 dual- -backend Node-arm contract adds no arms; the spirit (VM change keeps -parity + carries a diff case) is honoured via the Layer-3 cases. - -## Alternatives considered (Devil's-advocate fork, fresh context, 2026-05-28) - -Ground truth re-verified by the subagent: jump operands are i16 -offsets relative to `ip`-after-jump; `call_sites`/`libspecs` are -operand-indexed (survive removal); `compileDo` emits `; -op_pop`; `op_jump +0` is never emitted. - -- **Alt 1 — smallest-diff**: sibling `peephole.zig`, one hardcoded - rule, ad-hoc inline `removed_before[i]` remap, top-level chunk - only. Better: minimal, one cycle, clears the demonstrable bar. - Breaks: the IP-remap is written as a one-rule concern; rule 2 - re-opens and generalises it under pressure (skeleton *enlarges* - the rewrite — F-002 risk); in-place mutation while offsets are - stale is where an off-by-one ships and passes the loop/arith - benches by luck. -- **Alt 2 — finished-form-clean (recommended envelope)**: build - `applyPlan(keep_mask)` once as a reusable, differential-tested - primitive; purity on the Opcode enum; recurse into fn sub-chunks; - sibling file. Better: the dangerous part (offset re-resolution) - is built and tested once and is exactly what Phase-17 fusion - reuses. Breaks (self-critique): erecting a data-driven `Rule` - registry on one rule is itself the excessive-skeleton smell — so - the narrowed form keeps `applyPlan` as the only abstraction and - writes the first rule as a plain function. -- **Alt 3 — wildcard**: land an identity-transform harness + the - differential wrapper first, defer the real rule, let the 110% - budget be met by the already-tight typed-struct VM. Better: zero - cycle-1 correctness hazard. Breaks: an identity-only pass is the - permanent-no-op smell unless paired with a real rule in the same - Phase; the "index-stable real optimization" half is vapour (no - such target in the current opcode set). -- Findings: **(a)** pure-push+op_pop is the right first rule (only - one that demonstrably fires; op_jump+0 is dead); the real hazard - is the purity classification → encode as `Opcode.isPurePush()` - exhaustive switch, not a list. **(b)** physical removal + remap is - the only honest option; op_nop null-out is fake (pays the dispatch - turn). **(c)** recurse into fn sub-chunks in Phase 13 — the benches - are fn bodies. **(d)** sibling `peephole.zig`, not a one-file - `optimize/` dir (reservation-as-bias). **(e)** optimize-then- - serialize is correct; the existing bytecode version field must - document that its scope includes the peephole rule set, else a - future rule change serves stale optimized caches (tracked D-103). -- DA recommendation (non-binding): Alt 2 narrowed by its own self- - critique — `applyPlan` as the sole load-bearing primitive, purity - on the enum, first rule as a plain function, recurse into fns, - sibling file, optimize-then-serialize + D-103. Main loop adopted - this. - -## Consequences - -- **Positive**: the IP-remap primitive is built and differential- - tested once, reused by every future removal rule and by Phase-17 - fusion. Compile-time purity safety. The Phase-12 cache stores the - true (optimized) compile output. -- **Negative**: the five current bench fixtures (arith_loop / - fib_recursive / let_chain / list_build / quote_chain) rarely - contain pure-push+op_pop pairs, so this rule shows little bench - movement. Row 13.4 measures the five-canonical parity and files a - debt row if a benchmark misses the 110% budget rather than - blocking — the 110% budget is expected to hold largely because - cljw's typed-struct VM is already tight. -- **Follow-up**: **D-103** — confirm/​document that the Phase-12 - bytecode-cache version field's scope includes the peephole rule - set (a rule-set change must bump the version so stale optimized - caches are rejected). - -## References - -- ROADMAP §9.15 (Phase 13 rows 13.3 / 13.4), §9 master table row 13 - (peephole, 110%) vs row 17 (`super_instruction.zig`, 100%) -- ADR-0005 (dual-backend differential oracle — the correctness gate) -- ADR-0036 (dual-backend parity contract) -- ADR-0034 (Phase-12 bytecode serializer — the cache that stores the - optimized chunk) -- `.dev/structure_plan.md` (`eval/backend/` subtree — `optimize/` dir - deferred to Phase 17) -- Step 0 survey: `private/notes/phase13-13.3-survey.md` (op_jump+0 - recommendation superseded by Step 0.6 per this ADR's Context) -- cw v0 prior art: `~/Documents/MyProducts/ClojureWasm/src/engine/compiler/compiler.zig:1387-1600` (fusion + 5-pass remap machinery) - -## Revision history - -- 2026-05-28: Status: Proposed -> Accepted (Phase 13 row 13.3 - entry; Devil's-advocate fork embedded in Alternatives considered). diff --git a/.dev/decisions/0048_state_machine_domains.md b/.dev/decisions/0048_state_machine_domains.md deleted file mode 100644 index 26d4f2507..000000000 --- a/.dev/decisions/0048_state_machine_domains.md +++ /dev/null @@ -1,164 +0,0 @@ -# ADR-0048 — State machine domain ADR for REPL / nREPL / build pipeline - -- **Status**: Proposed → Accepted (issued inline at row 14.9, 2026-05-28) -- **Affected**: `src/app/repl.zig` (REPL state machine, this row); - `src/app/nrepl.zig` (future, row 14.10); `src/app/builder.zig` - (future, row 14.11). All three are user-visible interactive - surfaces that benefit from an explicit state chart so error / - recovery paths are reviewable as a graph rather than a callgraph. -- **Supersedes**: none. The §9.17 placeholder's reservation - "ADR-0028 (state machine)" was discarded per F-002 Reservation- - as-bias smell at row 14.0; this ADR's id 0048 follows the - time-ordered allocation rule (`max(existing) + 1` at issue time). - -## Context - -Three Phase-14 surfaces — `cljw repl`, `cljw nrepl`, `cljw build` — -share a common pattern: an outer loop alternating between -read-from-input / process / produce-output, with per-iteration -recovery on transient errors. cw v1's principle.md flags -"untyped loops with implicit state" as a smell; documenting each -surface's state chart as a first-class diagram prevents the -implementation from accreting reachable-but-unaccounted-for -states (e.g. "we're in the middle of a multi-line read but the -prompt drew anyway"). - -The three surfaces also share **error-recovery discipline**: a -per-iteration error must not propagate out of the loop. This ADR -codifies the convention so future maintainers don't accidentally -`return err` in a place that should be `catch |e| renderAndContinue(e)`. - -## Decision - -Each of the three surfaces is described by an explicit state chart. -The Phase-14 landing only ships the REPL chart's implementation -(row 14.9); rows 14.10 (nREPL) and 14.11 (build) ship their charts' -implementations and reference this ADR. The chart is rendered as -ASCII in the implementation module's `//!` docstring; significant -refactors edit the chart in the same commit. - -### REPL chart (row 14.9 — landed) - -``` - ┌──────┐ prompt ┌──────────┐ line received ┌────────────┐ - │ idle │ ───────▶ │ reading │ ───────────────▶ │ evaluating │ - └──────┘ └──────────┘ └────────────┘ - ▲ │ │ - │ result printed │ EOF on stdin │ value | error - │ ▼ ▼ - ┌──────────┐ (exit) ┌──────────┐ - │ printing │ ◀──────────────────────────────── │ result │ - └──────────┘ └──────────┘ -``` - -States: -- `idle` — between prompts; no I/O in flight. -- `reading` — prompt drawn, waiting on stdin line. -- `evaluating` — parsing + analysing + evaluating one form. -- `result` — value (success) or error info (failure). -- `printing` — writing result + newline to stdout / stderr. - -Transitions: -- `idle → reading`: drawn by `stdout.print("{ns}=> ")`. -- `reading → evaluating`: line received. -- `reading → exit`: stdin EOF. -- `evaluating → result`: parse + analyse + eval finish (success - OR error). -- `result → printing`: format result Value via `print.printValue` - or render error via `error_render.renderError`. -- `printing → idle`: returns control to the prompt loop. - -**Recovery discipline**: every transition into `printing` is a -no-throw path. Any `error` in `evaluating` is caught by the loop -wrapper and converted to a `result(error)` transition. - -### nREPL chart (row 14.10 — landed at 2026-05-28) - -``` - ┌──────────┐ client connect ┌───────────────┐ clone op ┌─────────────┐ - │ accept │ ────────────────▶ │ session_init │ ──────────▶ │op_dispatch │ - └──────────┘ └───────────────┘ └─────────────┘ - ▲ │ - │ session_close │ each op - │ ▼ - ┌──────────┐ ┌─────────────┐ - │ closing │ ◀── close op / disconnect ────────────────────── │ response │ - └──────────┘ │ _send │ - └─────────────┘ -``` - -States: -- `accept` — blocking accept on TCP socket; one client at a time - per the Phase-14 single-thread runtime. -- `session_init` — session id minted on `clone`; the connection - state machine begins. -- `op_dispatch` — read one bencode dict, look at `op` key, route - to the per-op handler. Currently 4 ops: `clone` / `describe` / - `eval` / `close`. -- `response_send` — encode the response dict(s) via bencode + - write to the stream. `eval` sends one response per top-level - form + a final `status: ["done"]`; other ops send one response. -- `closing` — `close` op received OR client disconnected; clean - up session state; loop back to `accept`. - -Recovery discipline: any per-op error renders as a -`status: ["error" "" "done"]` response and the session -continues (unknown-op, parse error, eval error all recover). Only -TCP-level errors (client RST / system EBADF) break out of the -session loop; the accept loop then accepts the next client. - -Implementation deltas vs draft chart shape: -- `interrupt` op NOT yet supported — single-thread eval cannot - interrupt itself (Phase 15+ thread spawn unlocks this; D-117). -- `complete` / `load-file` / `info` ops NOT yet supported — - CIDER-specific extensions filed under D-117. -- stdout/stderr capture is empty (no `*out*` / `*err*` binding - yet; D-118). - -### Build pipeline chart (row 14.11 — implementation pending) - -[To be filled at row 14.11 entry.] States expected: `parse_argv`, -`load_source` (loop over files), `analyse_all`, `emit_bytecode`, -`write_trailer`, `done`. Linear with explicit per-file recovery -on `analyse_all` failure (continue parsing other files; aggregate -errors). - -## Alternatives considered - -Devil's-advocate fork was NOT performed: ADR-0048's "decision" is -**ADR-existence-vs-prose**, not an implementation choice. The -implementation choice (REPL line-buffered vs PTY line editor) IS -deferred to D-116 polish. Both shapes fit the chart unchanged — -the chart is the abstract contract, not the byte-level loop body. - -The alternative that WAS considered: -- (Alt A — recorded but rejected) **One state machine for all three - surfaces**: would force an over-general "operation" enum carrying - REPL prompts AND nREPL JSON-RPC frames AND build-pipeline file - paths. The shared substructure is too thin (just "loop with - per-iteration recovery"); merging the charts would add - variant-dispatch noise without removing real duplication. - -## Consequences - -- Each surface's `//!` docstring carries its chart. PR reviews can - diff the chart alongside the body. -- Future read-eval-loop additions (debugger? `:reload`?) update the - chart visibly. -- The chart is documentation, not enforced syntax. A future - hook could parse the docstring and lint added states without an - ADR amendment, but that mechanisation is deferred until the - charts have stabilised (Phase 15+). - -## Cross-references - -- ROADMAP §9.16 row 14.9 (this row); rows 14.10 + 14.11 (future). -- ADR-0015 amendment 2 (F140-F144 re-introduction table). -- `.dev/principle.md` Bad Smell catalogue — "untyped loop with - implicit state". -- `src/app/repl.zig` (the REPL implementation embedding the chart). - -## Revision history - -- 2026-05-28: ADR landed alongside `cljw repl` (row 14.9). nREPL - + build pipeline charts placeholder pending rows 14.10 / 14.11. diff --git a/.dev/decisions/0049_orbstack_linux_gate_retired.md b/.dev/decisions/0049_orbstack_linux_gate_retired.md deleted file mode 100644 index c2fede37b..000000000 --- a/.dev/decisions/0049_orbstack_linux_gate_retired.md +++ /dev/null @@ -1,190 +0,0 @@ -# ADR-0049 — Retire `orb run -m my-ubuntu-amd64` per-commit Linux gate; migrate to `ubuntunote` SSH host - -- **Status**: Proposed → Accepted (issued 2026-05-28, implementation - verified same day) -- **Affected**: `test/run_all.sh` (header comment); `CLAUDE.md` - § Working agreement + § Build & test; `.dev/ROADMAP.md` Phase - tracker + 🔒 OrbStack annotations; `.dev/orbstack_setup.md` - (deprecated header); `ARCHITECTURE.md` cross-arch test row; - new `scripts/run_remote_ubuntu.sh` + `.dev/ubuntunote_setup.md`. -- **Supersedes**: per-commit `orb run -m my-ubuntu-amd64` gate - language across CLAUDE.md / ROADMAP / ARCHITECTURE / orbstack_setup; - all retired in this commit. - -## Context - -`bash test/run_all.sh` has run twice per commit since Phase 1.12: -once on Mac (host, `aarch64-darwin`) and once on Linux x86_64 via -`orb run -m my-ubuntu-amd64 bash -c 'bash test/run_all.sh'`. The -OrbStack-driven Linux run validates cross-arch + cross-OS -portability of the cw v1 codebase. - -Two operational pains accumulated and prompted the migration: - -1. **Orphan-process hazard on session interruption.** cw's per-task - loop has invoked the Linux gate from `run_in_background:true` - bash agents to overlap with Mac measurement. When the parent - Claude session terminates abnormally (compaction crash / harness - kill / user interrupt), the `orb run` child inherits `PID 1`, - the OrbStack VM (`my-ubuntu-amd64`) holds the connection open, - and downstream pipeline parts (e.g. `grep` on a hung REPL - stdin) spin at 100 % CPU. A 2026-05-28 incident ran ~1h50m on - the user's Mac (fan + Helper CPU pinned) before a sibling - session reaped it. -2. **VM hang propagation.** When the OrbStack VM enters its own - hung state, subsequent `orb run` invocations queue rather than - error out, masking the issue and exhausting CPU budget on the - OrbStack Helper process. - -zwasm v2 (sibling project on the same host) retired the same -gate earlier under their ADR-0067; cw v1 mirrors the migration -here with the same `ubuntunote` SSH host already provisioned for -zwasm. - -## Decision - -Remove `orb run -m my-ubuntu-amd64` from the per-commit gate. -`bash test/run_all.sh` runs **only on the Mac host** for the loop's -TDD cycles. Cross-arch / cross-OS portability validation moves to: - -1. **Manual via `scripts/run_remote_ubuntu.sh`** — invoked by the - reviewer at Phase-boundary review chains, before the v0.1.0 - release tag, and on demand during a feature branch's late - stages. Drives the `ubuntunote` SSH host (native x86_64 Linux, - physical hardware) via `git fetch + reset --hard - origin/main` + `nix develop --command bash test/run_all.sh`. - Setup procedure at `.dev/ubuntunote_setup.md`. -2. **Future CI integration** — a GitHub Actions / equivalent - workflow that runs the gate on `ubuntu-latest` for every push - closes the gap for every commit. Filed as new debt D-120 - (opportunistic — first GitHub Actions push). - -Until CI lands, **cw v1 single-OS Mac coverage is the per-commit -contract**; cross-arch regressions surface at the next -`run_remote_ubuntu.sh` invocation + at CI activation. - -## Implementation verified (2026-05-28) - -The migration was implemented + verified within the same cycle: - -1. **`ubuntunote` SSH reachability** confirmed: `ssh ubuntunote - 'echo ok && nix --version'` returns `ok` + `Determinate Nix - 3.20.0`. -2. **cw repository clone** landed on ubuntunote: - `~/Documents/MyProducts/ClojureWasm` via - `git clone -b main - https://github.com/clojurewasm/ClojureWasm.git` (HTTPS is - sufficient for the read-only fetch flow; mirrors zwasm's - clone pattern). -3. **`nix develop --command zig version`** returns `0.16.0` from - the project's `flake.nix` pinned input. First invocation - bootstrap took 15 s (with cached store entries shared with - the zwasm bring-up on the same host). -4. **Full gate** `nix develop --command bash test/run_all.sh` - on ubuntunote: **84/84 PASS** at HEAD `a9045b10` (the +1 - diff vs Mac 85/85 is the zlinter step, Mac-only per - ADR-0003). -5. **Wrapper** `scripts/run_remote_ubuntu.sh` lands in this - commit (mirrors zwasm v2's `scripts/run_remote_ubuntu.sh` - shape: preflight + sync + gate with labelled failure - attribution). - -## Alternatives considered - -(Devil's-advocate fork not run — the decision is operational -governance, not a structural design choice. The three -alternatives below were enumerated inline by the issuing chat.) - -### A. Keep `orb run` per-commit + add `timeout 600` wrap - -- **Sketch**: every Linux gate invocation wraps `orb run …` in - `timeout 600 …` so a hung VM dies after 10 min instead of - spinning forever. -- **Why rejected**: addresses the symptom (CPU spin duration) - not the root (no orphan-reap when Claude session dies). 600 s - spin × repeated crashes still drains fan / battery. The - `timeout` wrap propagates SIGTERM to `orb run` but not into - the VM-side child process, so the VM can stay hung after - `orb run` exits. - -### B. Retire `orb run` AND adopt a containerised runner inline (Docker / Podman) - -- **Sketch**: replace `orb run` with `docker run --rm -v - $(pwd):/repo -w /repo zig-0.16:latest bash test/run_all.sh`. -- **Why rejected**: Docker / Podman is not currently a cw v1 - dev-env dependency; introducing one for a single gate pushes - the dev-env baseline. Defer to CI activation (D-120) where - the runner image is managed by GitHub Actions infrastructure. - -### C. (Adopted) Retire `orb run` + reuse the `ubuntunote` host already in zwasm - -- See § Decision. The shared host is already provisioned for - the sibling project; cw inherits the bring-up at no extra - cost beyond a one-time `git clone`. - -## Consequences - -- **Positive**: per-commit gate loses one OS-coverage surface - but gains stability; no orphan / fan / CPU regression risk; - cw's TDD loop is single-OS-fast. -- **Positive**: OrbStack VM lifecycle is no longer in the cw - per-commit critical path. `orb stop my-ubuntu-amd64` is - acceptable at any time without affecting cw work. -- **Positive**: ubuntunote is real hardware → no Rosetta - signal-delivery surprises (a JIT-era hazard cw v1 will hit - at Phase 17). -- **Negative**: cross-arch / cross-OS regressions can sneak in - between manual `run_remote_ubuntu.sh` invocations. Mitigation: - D-120 GitHub Actions activation. -- **Neutral**: `~/Documents/MyProducts/ClojureWasm` v1 reference - clone is unaffected — this ADR retires the **gate**, not the - **reference clone**. - -## Migration steps (landed in this commit) - -1. `scripts/run_remote_ubuntu.sh` (new): wrapper invoking the - ubuntunote gate via SSH + `nix develop`. Mirrors zwasm's - pattern (labelled failure attribution; `--branch` flag for - feature-branch verification). -2. `.dev/ubuntunote_setup.md` (new): cw-side setup notes; defers - the full Ubuntu bring-up procedure to - `~/Documents/MyProducts/zwasm_from_scratch/.dev/ubuntunote_setup.md` - §§ 1-3 (shared host); documents the cw-specific clone + - `flake.nix` chain. -3. `test/run_all.sh`: top-of-file comment block updated to point - at `scripts/run_remote_ubuntu.sh` for the Linux gate (the - script body did not invoke `orb run` itself). -4. `CLAUDE.md` § Working agreement + § Build & test: the - "OrbStack Ubuntu x86_64 before every commit" sentence is - removed; "Linux gate runs at Phase boundary + via - `scripts/run_remote_ubuntu.sh`" replaces it. -5. `.dev/ROADMAP.md`: 🔒 OrbStack annotations become 🔒 Mac-host - + a pointer to this ADR. -6. `.dev/orbstack_setup.md`: deprecation header added at the - top; body retained as historical reference + dev convenience. -7. `ARCHITECTURE.md`: cross-arch test row text amended. -8. New `D-120` debt row: "CI Linux gate activation" — blocker = - first GitHub Actions push enabling `ubuntu-latest` workflow. - -## References - -- ROADMAP §A1.12 (per-commit gate definition — amended). -- CLAUDE.md § Working agreement / § Build & test (amended). -- `.dev/ubuntunote_setup.md` (cw setup notes). -- `~/Documents/MyProducts/zwasm_from_scratch/.dev/ubuntunote_setup.md` - (shared host's full bring-up). -- `~/Documents/MyProducts/zwasm_from_scratch/scripts/run_remote_ubuntu.sh` - (reference shape for the wrapper). -- 2026-05-28 cross-session incident report (user chat): cw - Claude session orphan-leaked a REPL pipeline with grep - CPU-spinning ~1h50m; sibling zwasm session reaped it + - introduced `~/.claude/hooks/cleanup_orphans.sh` globally. -- ADR-0003 (zlinter on Mac only — explains the +1 PASS-count - diff between Mac and ubuntunote gates). - -## Revision history - -- 2026-05-28: Status: Proposed → Accepted (initial landing with - same-day verification — `ssh ubuntunote` + clone + `nix - develop` + full gate 84/84 PASS at HEAD `a9045b10` confirmed - before this ADR text was finalised). diff --git a/.dev/decisions/0050_unified_interop_call_node.md b/.dev/decisions/0050_unified_interop_call_node.md deleted file mode 100644 index eaed4ed2e..000000000 --- a/.dev/decisions/0050_unified_interop_call_node.md +++ /dev/null @@ -1,689 +0,0 @@ -# ADR-0050 — Unify Java/host interop dispatch under a single `InteropCallNode` - -Status: Accepted (2026-05-28, inline per CLAUDE.md § ADR-level designs -are handled inline) - -Supersedes: none. Amends — D-073 parity cluster row in `.dev/debt.md` -collapses 3 existing VM-DEFER markers (`method_call_node`, -`field_access_node`, `ctor_call_node`) into one consolidated -`interop_call_node` VM-DEFER. ADR-0007 (TypeDescriptor) + -ADR-0008 (CallSite cache) + ADR-0029 D5 (cljw-prefixed surface ns) -all stay in force; the dispatch primitive they describe is now -addressed by a single Node variant rather than four parallel ones. - -## Context - -D-121 in `.dev/debt.md` is the v0.1.0 release blocker for Java -static method dispatch (`(java.util.UUID/randomUUID)`, -`(java.lang.System/currentTimeMillis)`, etc.). The Step 0 survey -at `private/notes/phase14-d121-survey.md` proposed adding a new -sibling Node variant `StaticMethodCallNode` next to the existing -`MethodCallNode` for instance dispatch, plus a parallel one-line -fix to `evalCtorCall` for the cljw-prefix mismatch that breaks -`(java.io.File. "x")`. - -The Devil's-advocate fork -(`private/notes/phase14-d121-da-fork.md` — output captured -verbatim under "Alternatives considered" below) enumerated three -shapes within the F-NNN envelope. Its Alt 3 (a unified -`InteropCallNode { kind, ... }` that retires `MethodCallNode`, -`FieldAccessNode`, and `CtorCallNode` together) was rated -**finished-form-clean** by the DA itself, with the explicit -caveat that "the v0.5.0 maintainer would land this if given -infinite time" but that the cycle budget for D-121 made Alt 2 -(the survey path) the practical choice. - -The main loop initially picked Alt 2 on that "cycle budget" -reasoning. The user immediately surfaced the meta-smell: deferring -a finished-form-clean structural decision **because the current -cycle is too small** is the canonical **Progress-pressure smell** -+ **Smallest-diff bias smell** doubled. CLAUDE.md § Project -spirit makes "shipping fast and avoiding rework are second-tier" -explicit, and F-002 makes "finished-form wins" project law. The -correct response when a depth-3 surgery is exposed in mid-cycle -is to take the surgery, not to file it as deferred debt that -"rarely comes back" (the Progress-pressure smell's exact -warning). - -This ADR records the correction and the unified shape. - -## Decision - -cw v1 introduces `InteropCallNode` as the single Node variant -that drives every form of Java- or cljw-surface interop call: -static methods, instance methods, instance field reads, and -constructors. The previously-separate `MethodCallNode`, -`FieldAccessNode`, and `CtorCallNode` are retired in the same -commit. - -```zig -// src/eval/node.zig (new variant; replaces 3 existing variants) -pub const InteropCallNode = struct { - /// Dispatch shape selector. Determines whether the eval arm - /// expects a receiver, an analyze-time descriptor, or both. - kind: enum { - static_method, // (Class/method args...) — descriptor known at analyze - instance_method, // (.method recv args...) — descriptor resolved at eval - instance_field, // (.field obj) — descriptor resolved at eval - constructor, // (Class. args...) — descriptor known at analyze - }, - /// Analyze-time descriptor for static_method + constructor. - /// `null` for instance_method + instance_field (resolved from - /// the target's runtime tag at eval). - descriptor: ?*const TypeDescriptor = null, - /// Receiver expression for instance_method + instance_field. - /// `null` for static_method + constructor. - target: ?*const Node = null, - /// Method or field name. For `constructor`, the convention is - /// the literal string `""` — matches the JVM method - /// table key for class constructors. - name: []const u8, - /// Argument expressions. Empty slice for `instance_field`. - args: []const Node = &.{}, - loc: SourceLocation = .{}, -}; -``` - -### Dispatch logic in `evalInteropCall` (single TreeWalk arm) - -```zig -fn evalInteropCall(rt, env, locals, n) !Value { - return switch (n.kind) { - .static_method => evalStaticMethod(rt, env, locals, n), - .instance_method => evalInstanceMethod(rt, env, locals, n), - .instance_field => evalInstanceField(rt, env, locals, n), - .constructor => evalConstructor(rt, env, locals, n), - }; -} -``` - -- `.static_method`: `n.descriptor.lookupMethod(null, n.name)` → - `BuiltinFn`; call with `args` only (no receiver). -- `.instance_method`: eval `n.target`, resolve descriptor from - the receiver's runtime tag, `lookupMethod(null, n.name)`, call - with `[receiver, ...args]`. -- `.instance_field`: eval `n.target`, must be `typed_instance`, - look up via `field_layout`. -- `.constructor`: if `n.descriptor.kind == .deftype` or - `.defrecord`, use `type_descriptor.allocInstance(rt, td, - args)`; if `.native` (Java surface), look up - `""` in `method_table` and call as a `BuiltinFn`. - -### Class-name resolution - -A new `resolveJavaSurface(rt, env, ns_head) ?*const TypeDescriptor` -helper handles the cljw-prefix translation: descriptors are -keyed by `"cljw.java.util.UUID"` per ADR-0029 D5, but user -source writes `"java.util.UUID"`. The helper tries: -1. Literal `rt.types.get(ns_head)`. -2. `rt.types.get("cljw." ++ ns_head)`. -3. Short-name lookup via current-ns aliases (Phase 14+ ergonomic; - may return `null` at v0.1.0). - -The helper is used by both the analyzer's `(Class/method)` arm -and the constructor arm — closing the long-standing -`evalCtorCall` cljw-prefix gap that the survey called out. - -### VM compile arm - -```zig -.interop_call_node => { - // VM-DEFER: interop dispatch bytecode shape pending row 7.6.b family lowering [refs: D-073, feature_deps.yaml#runtime/vm/interop_call_node] - return error.NotImplemented; -}, -``` - -One VM-DEFER row covers all four kinds. The bytecode lowering -decision (single `op_interop_call` with a kind tag in the operand, -or four separate opcodes) lands at row 7.6.b under D-073's -existing VM parity cluster — the same row that was already going -to choose the shape for the previous 3 separate Nodes. Net -effect: the VM-DEFER marker count drops from 3 to 1. - -## Consequences - -### Positive - -- **Single dispatch abstraction for the entire Java/host interop - surface.** A future reader sees one Node variant carrying every - call shape; the kind tag picks the args layout. No "why does - instance_method look so similar to static_method" reading - burden. -- **VM-DEFER count drops 3 → 1.** D-073's parity cluster shrinks - on landing. The bytecode shape decision becomes one decision - instead of four parallel ones. -- **CallSite cache (ADR-0008) integration is unified.** Each call - site caches `(descriptor, name) → MethodEntry` once for both - instance and static; no parallel cache infrastructure. -- **The `evalCtorCall` cljw-prefix gap closes structurally.** The - same `resolveJavaSurface` helper that handles static dispatch - also fixes ctor lookup; `(java.io.File. "x")` works after this - ADR's commit without a separate one-line patch. -- **F-002 honored.** Finished-form wins over smallest-diff - convenience. The "defer fusion to Phase 7 mid" framing — which - the survey + DA-Alt-2 path implied — is removed. -- **F-009 honored.** The unified Node carries kind information - because cw's internal model needs it (args layout differs), not - because the JVM AST has parallel `StaticMethodExpr` / - `MethodExpr` nodes to mirror. The internal abstraction is "Java - surface dispatch with optional receiver", which is a cw v1 - shape choice. - -### Negative - -- **Migration scope expands.** This ADR's commit touches - `eval/node.zig` (replaces 3 variants), `eval/analyzer/analyzer.zig` - + `eval/analyzer/special_forms.zig` (4 analyzer arms now build - `InteropCallNode`), `eval/backend/tree_walk.zig` (one - unified eval arm), `eval/backend/vm/compiler.zig` (3 arms → - 1), `src/lang/diff_test.zig` (test descriptors renamed; cases - unchanged in semantics), and the Java surface files - (UUID/System/File method_table population). ~+450 LOC net - including ADR + e2e + per-task note. -- **Kind enum is a flat tag, not a tagged union.** A - `static_field` kind with non-empty `args` is meaningless but - the type system does not prevent it. Mitigated by a debug - assert in `evalInteropCall`'s entry; promoted to a tagged - union if a Phase-7-mid review surfaces real misuse. -- **Existing diff-case test names reference the retired Node - names** (`"diff: ctor_call_node second field"`). These get - renamed to `"diff: interop_call_node .constructor + field"`- - style in the same commit. Test bodies don't change. - -### Neutral - -- The `D-121` debt row closes on this commit's SHA. A new - `D-130` row opens to track the **VM lowering** of - `interop_call_node` (the remaining VM-DEFER) — this row inherits - D-073's cluster scope for the interop subset. -- ADR-0007 / ADR-0008 / ADR-0029 stay in force. The Node-level - unification is purely an analyzer-layer consolidation; the - TypeDescriptor + method_table + cljw-prefix conventions they - set continue to be the dispatch primitive. - -## Alternatives considered - -The Devil's-advocate subagent's output is reproduced verbatim -below (the alternatives the loop's accumulated momentum would not -have considered without an isolated context fork). The DA's -recommendation was Alt 2; the user's intervention surfaced that -this recommendation itself was Progress-pressure-smell-driven, -and that Alt 3 is the F-002-aligned choice. - -### Alt 1 — Reuse `MethodCallNode` with a sentinel receiver - -Synthesise a `type_ref_node` leaf and pass the class handle as -`args[0]`; each static `method_table` entry remembers "first arg -is the class". −110 LOC. - -**Rejected** on `silent-default-shift smell` grounds -(`.dev/principle.md`): every static `BuiltinFn` body has to -slice `args[1..]` and a forgotten slice silently passes the -class handle as user arg 0. The smell sensor catches it -specifically; not a candidate. - -### Alt 2 — Two-Node split: `StaticMethodCallNode` sibling to `MethodCallNode` - -The survey's recommendation. New variant for statics; existing -`MethodCallNode` / `FieldAccessNode` / `CtorCallNode` stay; +3 -VM-DEFER rows persist; +0 LOC vs survey estimate (290). - -**Rejected** on F-002 grounds. The DA itself flagged this as -"borderline F-009 friction (mirrors JVM AST shape)"; the user's -intervention crystallised that the "defer fusion to Phase 7 mid" -rationale is itself the Progress-pressure smell. Fusion is the -finished-form; landing it now closes 2 existing VM-DEFER rows -while solving D-121, and removes a debt row that would otherwise -sit deferred for indefinite time. - -### Alt 3 — Unified `InteropCallNode` with `kind` tag (this ADR) - -Single Node variant parameterised by a 4-way kind enum. Retires -`MethodCallNode` + `FieldAccessNode` + `CtorCallNode` in the same -commit; 3 VM-DEFER markers consolidate into 1; v0.5.0 maintainer's -shape per the DA's framing. +60 to +90 LOC vs Alt 2 (350-380 -total including the existing-Node migration). - -**Accepted.** The cycle-budget objection is itself the smell the -user surfaced; the structural benefit is exactly what F-002 -demands. - -## Implementation order - -1. Add `InteropCallNode` to `src/eval/node.zig` (alongside the - retiring variants for one transitional commit step — see - below). -2. Build `InteropCallNode` from each analyzer arm - (`analyzeStaticMethodCall` new; `analyzeCtorCall` / - `analyzeFieldAccess` / `analyzeMethodCall` retargeted). -3. Add `evalInteropCall` to `tree_walk.zig` with the 4-way - switch. -4. Replace 3 VM compile arms with 1 `.interop_call_node` arm + - VM-DEFER marker. -5. Remove the retired Node variants from `node.zig` (and the - matching arms from analyzer + tree_walk + vm/compiler) — the - compiler-enforced exhaustive switch verifies completeness. -6. Populate `method_table` for UUID/randomUUID, - System/currentTimeMillis, System/nanoTime, File/. -7. Add `resolveJavaSurface` helper; wire into static + ctor - analyzer arms. -8. Add Layer 2 e2e test `test/e2e/d121_java_static.sh` (4 cases). -9. Add Layer 3 diff_test case for static dispatch; rename the 3 - existing diff_test descriptors to reference `interop_call_node`. -10. Update `feature_deps.yaml`: consolidate the 3 existing - parity-defer entries into one `runtime/vm/interop_call_node` - entry; close D-121, open D-130. - -Steps 1-5 may land as one Node-level commit; step 6-9 as a second -commit if the diff stays focused. Step 10 lands with the source -commits (provisional-marker hook + dual-backend-parity hook -verify the bookkeeping is in sync). - -## Discharge condition - -This ADR's Decision is fully implemented when: - -- `grep -nE '\\.(method_call|field_access|ctor_call)_node\\b' src/` - returns 0 lines (all retired). -- `grep -nE '\\.interop_call_node\\b' src/eval/{node,backend/tree_walk,backend/vm/compiler}.zig` - returns exactly one analyzer + one TreeWalk + one VM-DEFER arm. -- `bash test/run_all.sh` passes on Mac with the new e2e + diff - cases; ubuntunote re-verified at Phase boundary. -- D-121 row in `.dev/debt.md` flips to `Discharged ()`. -- D-130 (VM lowering of `interop_call_node`) opens with the - expected barrier predicate (Phase 7 mid VM lowering row). - -## References - -- `.dev/debt.md` D-121, D-073, (new) D-130. -- `.claude/rules/dual_backend_parity.md` (the VM-DEFER discipline - this ADR consolidates 3 sites of). -- `.dev/principle.md` (Progress-pressure smell, Smallest-diff - bias smell — the user's intervention names both). -- `.dev/project_facts.md` F-002, F-009. -- `private/notes/phase14-d121-survey.md` (Step 0 output). -- `private/notes/phase14-d121-da-fork.md` (Devil's-advocate - output, verbatim above under Alternatives considered). -- ADR-0007, ADR-0008, ADR-0029 D5 (TypeDescriptor + method_table - + cljw-prefix; in force). -- ADR-0036 (dual-backend parity contract; the consolidation - honors this). - -## Amendment 1 (2026-05-29) — instance member-vs-field dispatch + native-type method coverage - -> Status: **Accepted** (design phase: survey + Step 0.6 + mandatory -> Devil's-advocate fork landed 2026-05-29). Source implementation is the -> next cycle's first commit (this amendment is the depth-2 doc commit that -> precedes it per CLAUDE.md Step 6). Supersedes the row-7.6 "Option A" -> arity-1-`.name`-is-a-field decision. - -### Context (amendment 1) - -Verified 2026-05-29: interop instance-method calls + native-type methods -are broken even in TreeWalk (the production-default backend): - -- **Q1 — `(Math/abs -5)` → "No namespace: 'Math'"**: the qualified-head - arm (`analyzer.zig` ~505) calls `resolveJavaSurface("Math")`, which only - tries `"Math"` / `"cljw.Math"`; `java.lang.Math` is unregistered - (`runtime/java/lang/` ships only `System.zig`) and there is no short-name - (`java.lang.` prefix) resolution path (deferred at ADR-0050 R3). Only the - FQCN form `(java.lang.System/...)` works today. -- **Q2 — `(.toUpperCase "hi")` → "field access: expected number, got - string"**: `analyzer.zig:484` routes arity-1 `.name` (`items.len == 2`) - UNCONDITIONALLY to the `.instance_field` kind (the row-7.6 "Option A" - decision, made when only deftype/record FIELD reads existed); - `.instance_method` needs `items.len >= 3`. So a no-arg instance method - becomes a field read, and `evalInstanceFieldRead`/`op_field_access` - hard-require a `.typed_instance` receiver — a String is not. Secondarily, - the `.string` native TypeDescriptor's `method_table` (`runtime.zig` - ~162-180) is EMPTY: no native String methods are wired anywhere. - -Two premises were corrected during Step 0.6 + the DA fork: (1) `(.-name)` -does NOT currently route to field — there is no dash-stripping arm, so -`(.-x p)` reads a field literally named `-x` and fails; `.-` support is -NEW work, not a preserved invariant. (2) The eval layer is ALREADY -receiver-polymorphic (`evalInstanceMethodCall` + VM `op_method_call` -resolve native receivers via `nativeDescriptor`); the bug is purely the -analyzer arity gate + the empty String `method_table`. - -### Decision (amendment 1) — Alt 2: single `.instance_member` kind - -Collapse `.instance_field` + `.instance_method` into ONE kind; resolve -member-vs-field at eval from the receiver's descriptor shape. -`InteropCallNode.Kind` → `{ static_method, instance_member, constructor }` -(3, down from 4). `.instance_member` carries `target` / `name` / `args` -(empty args legal) + a `field_only: bool = false` flag (set for `.-name`). - -- **Analyzer**: arity-≥1 `.name` → `.instance_member{field_only=false}` - (the arity-1-vs-≥3 split disappears); `.-name` → `field_only=true` (NEW - arm). `Name.` ctor + `Class/method` static arms unchanged. -- **Both backends — ONE shared receiver-keyed resolver**: - ``` - if recv is typed_instance/reified: // ONLY these have field_layout - if field_layout has name: field-read // FIELD-FIRST - if field_only: raise no-such-field - if not field_only: - if td.lookupMethod(name): call(recv, ...args) - raise (field_only ? no-field : no-member) - ``` - TreeWalk: one `evalInstanceMember` (converges the method + field arms). - VM: reuse `op_method_call`'s call-site machinery; **retire - `op_field_access`** (its typed-instance-only logic folds into the - resolver). ≥1 differential case each: native-method / native-field / - deftype-field (ADR-0036). -- **String surface**: `runtime/java/lang/String.zig` (the `System.zig` - pattern, F-009 thin over `runtime/collection/string.zig`) populating the - `.string` descriptor's `method_table`. - -**Three caveats (load-bearing — fold into the implementation):** - -1. **FIELD-FIRST, keyed on `field_layout` presence** (the typed_instance/ - reified branch), NOT method-then-field. `field_layout` is non-null ONLY - for deftype/record; native types have `field_layout == null` so go - straight to `method_table`. method-then-field would let a protocol - method named like a field silently shadow the field (Silent-default- - shift smell). This is the clean deftype `(.field rec)` regression - guarantee. JVM's method-wins precedence is moot here — cw has no JVM - reflection (ROADMAP §13); deftype fields are first-class named slots and - field-first is the cw-native precedence. -2. **`.-name` is NEW work** (no dash-stripping arm today). No corpus/test - uses `(.-` (confirmed). Alt 2 adds it via `field_only`; deferring is - defensible ONLY if the current `-x` mis-read becomes an explicit error. -3. **String `method_table` wiring is ORTHOGONAL + was under-scoped**: - `nativeDescriptor` LAZILY allocates per-Runtime with an empty - `method_table`; `System.zig` installs into a STATIC descriptor, but - native-tag descriptors are per-Runtime, so String needs an - `installNativeMethods(rt)` at runtime init populating - `rt.native_descriptors[@intFromEnum(.string)]`. - -`Math/abs` (Q1) — registering `java.lang.Math` + short-name resolution — -is the immediate follow-on, not in the minimal Alt 2 dispatch fix. D-130 -(the `.static_method` VM bytecode arm) rides this or stays its own cycle. - -### Alternatives considered (amendment 1) - -(Devil's-advocate subagent, fresh context, F-NNN envelope — verbatim. It -corrected the two premise errors above.) - -> **Alt 1 — Smallest-diff: flip the analyzer arity gate, keep two kinds.** -> Route arity-1 `.name` → `.instance_method` with method-then-field -> fallback at eval; add a `.-` arm → `.instance_field`; wire String -> method_table. *Better:* minimal blast radius, no Node-kind retirement, -> fastest to green. *Breaks:* method-then-field is the fragile ordering -> (field/method name-collision silently shadows the field — Silent- -> default-shift); fallback must be added to BOTH backends across two -> DIFFERENT opcodes → ADR-0036 drift; the kind tag no longer means what it -> says (decided at eval, not analyze); nativeDescriptor lazy-alloc wiring -> under-scoped. *F-NNN:* clears F-009/R4/Tier; FAILS F-002 (smallest-diff -> convenience the finished-form owner unwinds). -> -> **Alt 2 — Finished-form-clean: single `.instance_member` kind + one -> shared receiver-keyed resolver per backend.** [As decided above.] -> *Better:* kind tag stops lying; field-first keying eliminates the -> collision smell structurally; backends CONVERGE (one resolver + one -> opcode, down from two) → ADR-0036 parity is a SINGLE decision; retiring -> op_field_access removes the latent native-field rejection; mirrors -> ADR-0050's own original collapse-4-into-1 logic. *Breaks:* largest diff -> (retires a Node kind + an opcode; parity hook forces all arms in one -> commit); `field_only` is a 2nd discriminant (set in one analyzer arm); -> retiring op_field_access needs the field-read diff cases re-encoded onto -> the unified opcode (arg_count=1 receiver-only path is new); String -> method_table wiring still required + orthogonal. *F-NNN:* clears F-002/ -> F-009/ADR-0036/R4/Tier/§13 (precedence is a cw-native descriptor-shape -> choice; `.-` is the one observable-Clojure JVM nod, not a JVM-internal -> assumption). NO F-NNN amendment required. -> -> **Alt 3 — Wildcard: analyze-time resolution for statically-known native -> receivers.** Bake the resolved method/field into the node when the -> receiver is a literal of known native type; else Alt 2's runtime -> resolver. *Better:* zero runtime dispatch for literal receivers; -> analyze-time "no such method" diagnostic. *Breaks:* TWO code paths for -> the same form (re-introduces the reading burden ADR-0050 killed); a new -> static-type mini-analysis with edge cases; diverges from Clojure's -> runtime-reflective `.name` dynamism (`(def s "x")` then `(.toUpperCase -> s)` is not statically resolvable — inconsistent); baked nodes go stale if -> `extend-type` adds a method (re-implements the CallSite-cache -> invalidation at the wrong layer). *F-NNN:* clears F-002/F-009/Tier/R4 in -> principle but mismatches §13 / the ADR-0008 CallSite-cache finished form. -> -> **Recommendation (non-binding): Alt 2.** Only alternative that does not -> lie about the kind tag or split backends asymmetrically; it is the SAME -> structural move ADR-0050 itself made. Cycle/diff size is not a project -> constraint (F-002) — Alt 2's larger diff is the cost to pay. Caveats: -> (1) field-first keyed on field_layout, NOT method-then-field; (2) `.-` -> is new work (defer only with an explicit error for the `-x` mis-read); -> (3) String method_table needs `installNativeMethods(rt)` at init. - -### Affected files (amendment 1) - -- `src/eval/node.zig` (`InteropCallNode.Kind` → 3 variants + `field_only`) -- `src/eval/analyzer/analyzer.zig` (~478-491 dot arms: collapse field/ - method, add `.-` arm) -- `src/eval/backend/tree_walk.zig` (`evalInstanceMember` converges - `evalInstanceMethodCall` + `evalInstanceFieldRead`) -- `src/eval/backend/vm/compiler.zig` + `vm.zig` (converge onto - `op_method_call`; retire `op_field_access`) -- `src/runtime/runtime.zig` (`installNativeMethods` + `.string` - method_table population) + `src/runtime/java/lang/String.zig` (new) -- `src/lang/diff_test.zig` (native-method / native-field / deftype-field) -- `test/e2e/` (new: `(.toUpperCase "hi")` / `(.x deftype)` / `(.-x …)`) -- `.dev/debt.md` (D-130 rides-or-follows) - -### Consequences (amendment 1) - -- Instance methods on native types (String, …) work in BOTH backends; - `(.x rec)` deftype field reads preserved via field-first keying. -- One member-access kind + one opcode (down from two) — ADR-0036 parity is - a single decision; `op_field_access` retired. -- `Math/abs` short-name static resolution (Q1) is the immediate follow-on. -- D-130's `.static_method` VM arm is the remaining VM-DEFER; rides or - follows this cycle (its bytecode-shape decision is separate). - -### Implementation order (amendment 1) - -(1) This ADR commit (doc, depth-2) lands first. (2) node.zig Kind + analyzer -2 arms + TreeWalk `evalInstanceMember` + VM converge/retire + runtime -`installNativeMethods` + String surface + diff cases + e2e — one atomic -source commit (the parity hook + exhaustive switch force it). (3) Verify NO -deftype `(.field rec)` regression (gate + targeted deftype field e2e). - -### References (amendment 1) - -- `private/notes/interop-coverage-survey.md` (Step 0 output — root causes). -- `private/notes/interop-DA-alt2.md` (Step 0.6 + the DA output verbatim - above; locked design). -- `.dev/debt.md` D-130 (`.static_method` VM arm). -- `.dev/principle.md` (Silent-default-shift, Smallest-diff bias, Cycle- - budget defer — the smells this amendment's field-first + Alt-2 choices - answer). -- ADR-0007 / ADR-0008 (TypeDescriptor + method_table + CallSite cache); - ADR-0036 (dual-backend parity); ROADMAP §13 (no JVM class hierarchy). - -## Amendment 2 (2026-05-29) — `.static_method` VM bytecode lowering (D-130) - -> Status: **Accepted** (inline per CLAUDE.md § ADR-level designs are -> handled inline; Step 0 survey + Step 0.6 + mandatory Devil's-advocate -> fork landed 2026-05-29). Resolves the bytecode-shape question the base -> ADR's "VM compile arm" section + the D-130 debt row deferred to -> "row 7.6.b". The source implementation is the next commit (this is the -> depth-2 doc commit that precedes it per Step 6). - -### Context (amendment 2) - -After am1, `InteropCallNode.Kind` is `{ static_method, instance_member, -constructor }`. `.instance_member` (op_method_call, field-first) and -`.constructor` (op_ctor_call) lower to VM bytecode; `.static_method` -(`(Math/abs -5)`, `(java.util.UUID/randomUUID)`) is still a VM-DEFER stub -(`compiler.zig compileInteropCall .static_method => return -error.NotImplemented`, D-130), so static dispatch works ONLY in TreeWalk -and static interop has **no dual-backend differential-oracle coverage**. -am1 Q1 (commit c1ebe0c5) registered `java.lang.Math` static methods — -the first deterministic integer/bool-returning statics — which is the -hard prerequisite that makes the required ≥1 differential case possible -(UUID/randomUUID is heap-String + non-deterministic; System time is a -wall clock — neither survives the Phase-4 NaN-box-bit-pattern comparator). - -### Decision (amendment 2) — sibling `op_static_method_call`, reusing `CallSiteEntry`, no static cache - -1. **Bytecode shape: a sibling opcode `op_static_method_call` (0x1E)**, - NOT a unified `op_interop_call` that would collapse op_ctor_call + - op_method_call + op_static_method_call under a kind-tag operand. -2. **Descriptor carriage: REUSE the existing `CallSiteEntry`** side-table - (the one op_method_call already indexes), adding one optional field - `descriptor: ?*const TypeDescriptor = null`. `null` ⇒ instance - dispatch (descriptor derived from the receiver's runtime tag, existing - behaviour); non-null ⇒ static dispatch (use this analyze-time - descriptor; `arg_count` is the user-arg count with NO receiver). This - mirrors how am1 added `field_only` to the same struct — the in-repo - precedent is **a mode flag on the shared entry, not a new struct per - mode**. No new `StaticCallEntry` type. -3. **No CallSite cache for static** (DA Finding A): TreeWalk's - `evalStaticMethodCall` uses the raw `td.lookupMethod(null, name)` with - **no** cache (the Node carries no cache slot). The VM static arm does - the same — `entry.descriptor.?.lookupMethod(null, entry.method_name)` - — so the two backends share one lookup mechanism and cannot diverge - under `extend-type` generation churn. `CallSiteEntry.cache` stays - (instance dispatch uses it via `lookupWithCache`); the static arm - ignores it. (Java surfaces are not `extend-type`'d, so a cache would - be dead weight as well as a divergence risk.) -4. Remove the VM-DEFER marker; flip - `feature_deps.yaml#runtime/vm/interop_call_static_method` to `landed`; - close D-130. Add ≥1 differential case now that integer/bool statics - exist (`(integer? (Math/abs -5))` → true; `(Math/max 3 7)` → 7). - -**Confronting the D-130 row's recorded lean (DA Finding B).** The D-130 -row recorded "candidate (a) [unified `op_interop_call`] preferred per -F-002 (finished-form: 1 opcode for 1 Node variant)". This amendment -**reverses that lean**, and must rebut it rather than ignore it -(reservations are memos, not contracts — Reservation-as-bias). The -rebuttal: ADR-0050 unified the **Node** because the analyzer wants one -dispatch abstraction, but the three interop kinds have genuinely -different **stack disciplines** (ctor = allocate; instance = pop receiver -+ field-first; static = no receiver). A unified `op_interop_call` does -not erase that 3-way split — it relocates it from the (free) top-level -`switch (opcode)` jump table into a nested `switch (entry.kind)` in the -hot dispatch loop, re-introducing a discriminant the opcode already -encodes. "1 opcode for 1 Node variant" is an aesthetic mapping, not a -finished-form gain; the finished bytecode form is **one opcode per -dispatch discipline**. The unification's only concrete payoff (a cheap -4th interop kind) is speculative — no 4th kind is on the roadmap -(`Math/PI` static-field reads, D-148, are a *field read*, not a method -call, and their shape is undecided). Sibling + shared-`CallSiteEntry` -is the finished form here, and it happens to also be the smaller diff -(F-002 and smaller-diff coincide — no Cycle-budget-defer tension). - -### Alternatives considered (amendment 2) - -Devil's-advocate subagent, fresh context, F-NNN envelope — reflected -faithfully. The DA's ranking was **Alt 1 > Alt 2 > original draft > Alt -3**; this amendment adopts Alt 1, which the DA rated finished-form-clean -AND smaller-diff. The DA's two leading findings (A: the original draft's -"mirrors TreeWalk exactly" claim was false because TreeWalk's static -path has no cache; B: the draft silently overrode the D-130 row's -unification lean) are folded into the Decision above. - -> **Original draft — sibling `op_static_method_call` + a NEW -> `StaticCallEntry` side-table {descriptor, method_name, arg_count, -> cache}.** *Rejected by the DA as "the uncanny valley"*: it neither -> unifies (Alt 2) nor minimally reuses (Alt 1) — it adds a sibling -> opcode AND a near-clone struct (`CallSiteEntry` minus `field_only` -> plus `descriptor`), and carries a `cache` the TreeWalk static path -> lacks (Finding A). -> -> **Alt 1 — sibling `op_static_method_call`, REUSE `CallSiteEntry` via an -> optional `descriptor` field (ADOPTED).** *Better:* one fewer type / -> chunk slice / compiler ArrayList / finalize-dupe; the op_method_call -> and op_static_method_call arms read as obvious siblings differing only -> in descriptor source; consistent with the am1 `field_only` mode-flag -> precedent (the codebase already overloads `CallSiteEntry` with a mode -> flag). *Costs:* a `CallSiteEntry` now has two mutually-exclusive modes -> (instance call-sites carry an unused null `descriptor`; the static arm -> ignores `cache`) — a mild fat-struct, but the precedent makes it the -> consistent shape. *F-NNN:* clears F-002 (no near-duplicate type; the -> finished form is mode-flags on the shared entry), F-009 (N/A — VM -> internal), ADR-0036 (one opcode + one arm + one diff case), §13 -> (descriptor-keyed, no JVM assumption). -> -> **Alt 2 — unified `op_interop_call ` with a kind tag in one -> `InteropSiteEntry`, retiring op_ctor_call + op_method_call.** *Better:* -> bytecode topology mirrors the Node 1:1 and the entry's `kind` reuses -> the analyzer `InteropCallNode.Kind` enum (no second enum to drift); a -> future 4th interop kind is one enum arm + one nested-switch case -> instead of a whole new opcode + exhaustive-switch arms -> (`isPositionRelative` / `isPurePush`). The DA judged the survey's -> anti-B2 hot-path argument "thin" (a nested `switch (entry.kind)` and a -> top-level `switch (opcode)` compile to the same jump-table class; one -> extra L1-resident byte-compare is noise in a tree-walk-class runtime). -> *Costs:* the real objection is **migration risk, not cycle budget** — -> rewriting two landed, tested dispatch arms (vm.zig op_ctor_call + -> op_method_call) + re-encoding their diff cases in one parity-hook-forced -> commit, where a regression in ctor/method dispatch would be a -> v0.1.0-relevant blocker introduced by an additive feature; and the -> `cache`/`type_name` fields apply to only some kinds (fat union). -> *F-NNN:* clears F-002 on the "1 opcode for 1 Node, cheap 4th kind" -> axis — the strongest unification story — but the 4th-kind payoff is -> speculative (none planned). No F-NNN violation. The DA: "finished-form- -> plausible-but-not-clearly-cleaner"; take it only if the loop reads the -> unification as the true finished form — **but never the original draft -> over this.** -> -> **Alt 3 — wildcard: no new opcode; lower `.static_method` to -> `op_const … op_call`** (resolve the static method -> to its callable Value at analyze time, since statics have no receiver -> polymorphism). *Better:* zero new bytecode surface; rides the -> most-tested `op_call` path; the descriptor-carrying problem evaporates. -> This is essentially what cw v0 did (rewrite statics to a callable at -> analyze time). *Breaks:* baking `method_val` at analyze time means a -> later `extend-type` / re-`deftype` does NOT invalidate the baked -> constant (the constant pool has no generation check) — the **same -> defect am1 Alt 3 was rejected for**, at the bytecode layer; and it -> moves method resolution to compile-time in the VM while TreeWalk -> resolves at eval-time, breaking the "descriptor resolved at analyze, -> method looked up at dispatch" symmetry am1 committed both backends to -> (incl. error-timing for "no such static"). *F-NNN:* mismatches §13 / -> the ADR-0008 CallSite-cache finished form (live descriptor + generation, -> not a frozen callable). Does not cleanly violate an F-NNN today (Java -> statics are immutable), but the least finished-form-aligned. Rejected; -> its rejection rationale is *why* the descriptor is carried into -> dispatch at all rather than pre-resolved. - -### Affected files (amendment 2) - -- `src/eval/backend/vm/opcode.zig` (`op_static_method_call = 0x1E` + - `CallSiteEntry.descriptor: ?*const TypeDescriptor`; the two exhaustive - metadata switches `isPositionRelative`/`isPurePush`; opcode-value test) -- `src/eval/backend/vm/compiler.zig` (`compileInteropCall .static_method` - arm: emit a `CallSiteEntry { descriptor = n.descriptor, method_name, - arg_count = n.args.len }` + `op_static_method_call`; remove VM-DEFER) -- `src/eval/backend/vm.zig` (`op_static_method_call` dispatch arm: raw - `descriptor.lookupMethod(null, name)` + `callFn`, no receiver) -- `src/lang/diff_test.zig` (≥1 deterministic static diff case) -- `feature_deps.yaml` (`runtime/vm/interop_call_static_method` → - `landed`; notes corrected — am1 retired op_field_access / the - `.instance_field`+`.instance_method` kinds the stale notes still name) -- `.dev/debt.md` (D-130 → Discharged; D-150 opened — see below) - -### Consequences (amendment 2) - -- Static dispatch works in BOTH backends; static interop regains - differential-oracle coverage (the v0.1.0 VM-parity gap closes). -- One sibling opcode + one shared side-table mode flag; no new struct, - no kind-tag-in-operand. A future 4th interop kind pays the new-opcode - tax (accepted — none is planned; if one lands, revisit unification). -- D-150 opened: the VM `op_ctor_call` arm resolves via bare - `rt.types.get(type_name)` while TreeWalk's `evalConstructorCall` uses - `resolveJavaSurface` (literal + `cljw.` + `cljw.java.lang.` attempts). - So a VM-mode Java-class ctor like `(java.io.File. "x")` would miss the - cljw-prefix that TreeWalk resolves — a latent dual-backend parity gap - (ADR-0036), unexercised today (no diff/e2e case runs a Java-class ctor - on VM; deftype ctors use literal `rt.types` keys). Separate focused fix. - -### References (amendment 2) - -- `private/notes/phaseA26-d130-static-vm-survey.md` (Step 0 output). -- `.dev/debt.md` D-130 (discharged here) + D-150 (opened here). -- `feature_deps.yaml#runtime/vm/interop_call_static_method`. -- ADR-0008 (CallSite cache — why static skips it: Finding A) + ADR-0036 - (parity) + `.dev/principle.md` (Reservation-as-bias — the D-130 row - lean this amendment reverses; Cycle-budget defer — absent here since - finished-form and smaller-diff coincide). diff --git a/.dev/decisions/0051_interim_goal_recut_quality_loop.md b/.dev/decisions/0051_interim_goal_recut_quality_loop.md deleted file mode 100644 index 44eee8c41..000000000 --- a/.dev/decisions/0051_interim_goal_recut_quality_loop.md +++ /dev/null @@ -1,189 +0,0 @@ -# 0051 — Interim-goal re-cut: milestone M (Phase 15 + cw-v0 JIT) then a quality-elevation loop, wasm FFI re-sequenced - -**Status**: Accepted (Devil's-advocate fork landed 2026-05-29) -**Date**: 2026-05-29 -**Author**: Shota Kudo (drafted with Claude, user-directed re-cut session) -**Tags**: roadmap-resequence, F-010, F-001, F-002, F-003, quality-loop, jit - -## Context - -User intervention (2026-05-29): re-cut the interim provisional goal. -Captured as project-law invariant **F-010** (`.dev/project_facts.md`, -verbatim quote). The direction is user-fixed; this ADR records the -**engineering expression** of that direction in ROADMAP §9 + §14, -within the F-003 structural-deferral discipline. - -Three surveys grounded the re-cut (digested in -`private/notes/recut-goal-synthesis.md`): - -- cw v1 current coverage — biggest gaps: interop *syntax* - `.`/`new`/`set!`/`dosync` unimplemented; clojure.core daily-driver - cluster missing (`get-in`/`assoc-in`/`concat`/`mapcat` D-126, - `pr-str`/`prn` D-127); seqs eager not lazy; thin clj ns + spec/math - untouched; diff layer (TreeWalk vs VM) unwired. **The runtime - cannot yet run real Clojure code well.** -- cw v0 JIT — `src/engine/vm/jit.zig` 721 LOC, ARM64-only narrow - integer-loop JIT, counter trigger (64), one-shot deopt, leaf C-ABI, - zero GC interaction by avoidance; 10.3x on `arith_loop` only. cw v1 - prerequisite: a superinstruction/fusion pass (cw v1 opcodes are - generic). Zig 0.16 `std.posix.mmap`/`mprotect` available. -- cw v0 coverage baseline — 1130 vars / 90.9% / 651-706 clojure.core, - spec.alpha 100%. cw v0 dropped deftype/proxy/gen-class (Tier D); - **cw v1 native deftype gives a higher ceiling = parity-PLUS target.** - -## Decision - -### D1: Post-Phase-14 execution order (direction, per F-010) - -``` -Phase 14 (v0.1.0) → Phase 15 (concurrency, 7 buckets) - → JIT chain (superinstruction/fusion → go/no-go → - cw-v0-level narrow ARM64 JIT) ── completes M - → quality-elevation loop (repeatable, standing mode) - → wasm FFI breadth (ClojureScript, zwasm v2 import, - broad JIT if chosen) ── after the loop -``` - -This **re-sequences** the prior §9.2 milestone mapping (which placed -CLJS at Phase 16 / v0.2.0 before the JIT). The JIT (currently split -across §9.19 Phase 17 super_instruction + §9.22 Phase 20 narrow JIT) -is pulled into the M window, immediately after Phase 15. The wasm -phases (§9.18 Phase 16 CLJS, §9.19's wasm content, §9.22 zwasm import) -slide to **after** the quality loop. **F-001 (wasm FFI eventually -unavoidable) is NOT superseded** — it is re-scheduled; F-010 -cross-references it. - -### D2: ROADMAP expression = light re-label + debt-recorded foresight (NOT full renumber) - -Per **F-003** (decision-deferral on structural plans), F-010 fixes the -**direction** but not the **granularity** (exact phase renumbering, -quality-loop sub-phase structure, the JIT-subtree extraction shape, -the coverage-floor-vs-superinstruction ordering). Granularity is an -*open structural plan* → reserved for each owning Phase entry. This -ADR therefore: - -- Adds a **re-sequencing note** to ROADMAP §9 (after §9.16) recording - the D1 order + that detailed renumber/expansion defers to each Phase - entry owner. -- Records foresight as **debt rows**, not seized task tables: - - **D-132** — quality-elevation loop phase structure (sub-phase - count / numbering / ownership): open, decide at the M-exit entry. - - **D-133** — JIT coverage-floor prerequisite: the daily-driver - coverage floor (interop syntax; D-126/127 core cluster; true - lazy-seq) must be green so the JIT lands on a runtime that runs - real code; the *ordering* of this floor vs the superinstruction - pass is the JIT-phase entry owner's call (recorded, deferred). -- The zwasm/CLJS relocation is a **header/pointer move** (direction - fixed by F-010), NOT a renumber of the zwasm task body or its - D-036..D-039 entry-debt internals (that stays the owning entry's - call). - -### D3: The quality-elevation loop (standing post-M work mode) - -Repeatable cycle, each pass refactor-gated (F-002 applied to the -bug-fix stream): (1) coverage → cw-v0 parity-PLUS; (2) clojuredocs -posted-example differential vs JVM Clojure -(`clojuredocs-export-edn`, ~1528 vars with `:examples`); (3) -real-world library loading (`clojure-corpus`, 200+ libs); (4) -differential fuzzing vs JVM + generative properties; (5) `docs/works/` -code-主体 walkthrough ledger. **Refactor gate**: a fix that would -require a workaround triggers a depth-2+ surgery instead; periodic -`simplify` + smell-audit passes prevent codebase rot (the user's -explicit "ただし … workaround だらけ・設計破壊にしない" constraint). - -## Alternatives considered - -Devil's-advocate fork (general-purpose subagent, fresh context, -2026-05-29, F-010/F-002/F-003/F-001 envelope) output verbatim: - -**Framing.** F-010 fixes *what* (M = Phase 15 完遂 + cw-v0-level narrow ARM64 JIT; post-M quality loop with mandatory refactor gate; wasm-FFI breadth re-sequenced after the loop, F-001 not superseded; v0.1.0 first). F-002 fixes the *quality bar* (finished-form wins; cycle/diff/LOC is not a constraint). F-003 fixes the *who-decides* on structural plans (imagine the full horizon + record as debt scheduled at the owning Phase, **defer** the structural decision to that owner; do not seize). The coupled open question — ROADMAP expression × JIT-vs-coverage sequencing — must land *inside* this envelope. The three shapes below differ only on how aggressively the ROADMAP §9 phase plan is concretised now, and where a coverage-floor sits relative to the JIT. - -A constraint-collision finding belongs at the top per the brief: **the full re-number shape (Alt B candidate) is in direct tension with F-003.** Concretely renumbering/inserting Phase 16=JIT-chain, Phase 17+=quality sub-phases, Phase 20+=zwasm *with full task tables now* is decision-seizure on a reservation table (the §9 phase row queue) + dependency graph (JIT vtable seam D-035, `src/eval/backend/jit/` subtree, `runtime/jit/stub.zig`) + responsibility split (quality-loop sub-phase ownership). F-003's scope boundary (principle.md L243-265) says imagination defers *unless an F-NNN already fixes the direction*. F-010 fixes the **direction** (JIT earlier, quality loop standing, zwasm right) but **not the structural granularity** (how many quality sub-phases, their numbering, the JIT subtree extraction shape, the superinstruction-pass module boundary). So the granularity remains an *open structural plan* → F-003 governs → full task-table expansion now is the thing F-003 forbids. This does not halt the loop; it ranks the full re-number below the deferring shapes. - -### Alt A — smallest-diff: minimal re-label of §9.18+ headers, everything else deferred - -(a) **Concrete.** Edit only the §9.18-9.22 placeholder *headers* + Phase-tracker rows 16-20 names/exit-criteria to reflect the new direction (16 → superinstruction/fusion + JIT go/no-go + narrow ARM64 JIT; 17+ → quality-elevation loop; zwasm/cljs slide to a later row). Add one §14 decision-point reference cross-linking F-010. Leave all task-table expansion (the `[ ]` rows) to each phase's entry owner per the Phase-open procedure. No coverage-floor recorded as a JIT prerequisite — left implicit. Touches ~6 header lines + tracker table. - -(b) **Better.** Maximally honours F-003 (pure imagine-record-defer, zero seizure). Smallest audit surface; the §9 tracker stays a "now snapshot" per §17. Cheap to amend again when surveys sharpen. - -(c) **Breaks/risks.** Under-records the load-bearing finding from the cw v1 coverage survey: the runtime **cannot run real Clojure code** (interop `.`/`new`/`set!` unimplemented; `get-in`/`assoc-in`/`concat`/`mapcat` missing — D-126/127; seqs eager not lazy). A bare header re-label loses the *prerequisite relationship* between coverage and the JIT. A future Phase-16 owner could read "narrow ARM64 JIT" and land a JIT on a runtime that can't exercise it beyond `arith_loop` — exactly cw v0's 1.0x-on-`fib_recursive` trap, now with no daily-driver code to even reach the hot loop. That is a foresight gap F-003 *requires* the current loop to record (record ≠ seize). So Alt A is too thin: it defers the *decision* correctly but skips the *imagination-recording* duty. - -### Alt B — finished-form-clean: light re-label + **debt-recorded foresight** (recommended) - -(a) **Concrete.** As Alt A's header/tracker re-label, **plus** the F-003 imagination-recording the survey demands, expressed as debt rows + cross-refs, not as seized task tables: -- Re-aim §9.18 Phase 16 header: "superinstruction/fusion pass → JIT go/no-go → narrow ARM64 JIT (cw-v0-level, M-completing)". Move zwasm-v2/cljs content (current §9.18 body + D-036/037/038/039 entry debts) to a later placeholder (e.g. §9.20+), retaining F-001 "eventual" status — **no renumber of the structural body, just a header re-aim + a pointer that the zwasm entry-debt cluster relocates at its owning entry**. -- Re-aim §9.19+ as the post-M quality-elevation loop placeholder (coverage / clojuredocs-differential / corpus / fuzzing / `docs/works/` + refactor gate), **header + Deliverables prose only**; the repeatable sub-phase structure (how many passes, numbering, ownership) is logged as a **new debt row** "quality-loop phase structure — open, decide at M-exit entry" so the owning entry resolves it (F-003 defer). -- Record the coverage-floor↔JIT relationship as a **debt row + an Entry-debt line on the re-aimed Phase 16 placeholder**: "JIT prerequisite: daily-driver coverage floor (interop syntax `.`/`new`/`set!`; `get-in`/`assoc-in`/`concat`/`mapcat` D-126/127; true lazy-seq) must be green so the JIT lands on a runtime that runs real code — the *ordering* of this floor vs the superinstruction pass is the Phase-16 entry owner's call." This **records the prerequisite (imagination)** without **deciding the ordering (seizure)** — the exact F-003 split. -- Add the superinstruction/fusion-pass module-boundary question (generic opcodes → no arith superinstructions; raw-opcode-window matcher vs fusion pass; route via `runtime/jit/stub.zig` injected vtable per D-035) as a foresight note on the §9.19 (Phase 17) Entry-debt line, deferred to that owner. -- §14: re-aim 14.1 (JIT go/no-go now M-internal, not post-Phase-17 gated) + add a short F-010 cross-ref; leave 14.2/14.3/14.4 as-is (they are genuine future toggles). - -(b) **Better.** This is the only shape that satisfies **both** halves of F-003: defers every structural *decision* to the owning entry **and** discharges the imagine-record duty so the future owner inherits foresight, not a blank slate (principle.md L271-301). It directly answers the coverage-floor question — recorded as a prerequisite, ordering deferred — preventing a JIT-on-dead-runtime mistake without seizing the ordering call. It keeps F-001's audit trail clean (zwasm entry-debt cluster relocates intact, not deleted). Finished-form-clean per F-002: the §9 tracker ends up telling the truth about direction while staying a deferring placeholder. - -(c) **Breaks/risks.** More moving parts than Alt A (several debt rows + entry-debt lines + a placeholder relocation) — a reviewer could mistake the debt-row volume for seizure. Mitigation: every row is phrased "open — decide at ", which is recording, not deciding. Second risk: relocating the zwasm/cljs §9.18 body to a later number is itself a light touch on the row queue (a reservation table) — but it is a *direction* move fixed by F-010, so it falls inside F-003's "F-NNN fixes the direction → implement it" boundary, not the open-plan branch. Keep the relocation to a header/pointer move; do **not** renumber the zwasm task body or its D-036..039 internals (that stays the owning entry's call). - -### Alt C — wildcard: insert an explicit "Phase 15.5 — coverage floor / real-code gate" as a hard M-prerequisite, JIT strictly after it - -(a) **Concrete.** Beyond Alt B's re-label, **decide the ordering now**: insert a named gate phase between Phase 15 and the JIT — daily-driver coverage cluster + interop syntax + true lazy-seq must be green (a measurable "runs real clojuredocs `:examples` at ≥X%") *before* the superinstruction pass + JIT open. Renumber so JIT becomes Phase 16-after-the-gate. Expand the gate's task table now (pull D-126/127 + interop + lazy-seq rows in). - -(b) **Better.** Strongest protection against the JIT-on-dead-runtime failure: makes "real code runs" a literal blocking gate, not a deferred note. Arguably the most *finished-form-sensible* engineering sequence — you do not optimise a runtime that cannot run the programs the optimisation targets. Gives the JIT a real workload (clojuredocs corpus) to bench against beyond `arith_loop`. - -(c) **Breaks/risks.** **This decides the coverage-vs-JIT ordering on the owning entry's behalf — F-003 decision-seizure**, and expands a task table now (the same forbidden move as the full re-number). F-010 leaves the *ordering* explicitly with latitude ("M = Phase15+JIT" fixes membership, not internal order); seizing it here is precisely the structural decision F-003 reserves for the M-window entry owner. Also risks scope-creeping the quality loop *into* M (the coverage floor is a slice of the post-M quality loop pulled forward as a gate) — entangling the M definition with the standing loop F-010 keeps separate. The right shape is to **record** this as the leading option for the Phase-16 entry owner (Alt B's debt row), not to **enact** it now. - -### Non-binding recommendation (anchored to F-002 + F-003) - -**Take Alt B (light re-label + debt-recorded foresight).** It is the finished-form-clean shape *and* the F-003-compliant shape simultaneously — rare alignment, so the Cycle-budget defer smell does not even arise (no smaller-diff downgrade is being recommended; Alt A is *worse* on finished-form because it under-records foresight, not merely smaller). On the two explicit sub-questions: - -1. **Light re-label + defer per F-003 beats the full re-number** — yes, decisively. The full re-number (and Alt C) seize structural decisions (row queue, JIT subtree, quality sub-phase structure, coverage/JIT ordering) that F-010 leaves at *direction*, not *granularity*; F-003 reserves granularity for the owning entry. The "finished-form wins" instinct (F-002) might tempt enacting Alt C's clean ordering now — but F-002 governs *code/design quality*, while F-003 governs *who decides structural plans*; they do not conflict here because deferring the decision does **not** lower the finished-form quality (the owner decides with more context). Picking Alt C "because it's the cleaner sequence" would be reading F-002 as a license to override F-003 — it is not. - -2. **The ROADMAP should record the coverage-floor as a JIT prerequisite** — yes, as a **debt row + Entry-debt line on the re-aimed Phase 16 placeholder**, framed "prerequisite recorded; ordering deferred to the Phase-16 entry owner." This is the load-bearing finding (runtime can't run real code today) that F-003 *obligates* the current loop to surface as foresight. Recording it is imagination-discharge, not seizure; deciding the floor-vs-superinstruction ordering is seizure → leave it to the owner. Same for the superinstruction/fusion-pass module boundary and the D-035 JIT-vtable extraction — record as foresight on the Phase-17 entry-debt line, defer the shape. - -The one thing to guard against while landing Alt B: keep the zwasm/cljs relocation a header/pointer move (direction fixed by F-010 → implement) and resist letting the debt-row volume tip into pre-deciding the quality-loop sub-phase structure (granularity open → defer). - -## Selection rationale - -Alt B selected. It is simultaneously finished-form-clean (F-002) and -F-003-compliant — it discharges the imagine-record duty (the runtime -can't run real code today → coverage-floor recorded as a JIT -prerequisite) while deferring every structural *decision* (renumber, -quality-loop sub-phase structure, floor-vs-superinstruction ordering) -to the owning Phase entry. Alt A under-records foresight; Alt C seizes -the coverage-vs-JIT ordering that F-003 reserves for the owning entry. -The F-002-instinct to enact Alt C's clean ordering now is explicitly -rejected: F-002 governs design quality, F-003 governs who-decides; -deferring does not lower finished-form quality (the owner decides with -more context). - -## Consequences - -- ROADMAP §9 gains a re-sequencing note (after §9.16) recording the D1 - order; the §9.2 v0.x milestone mapping is annotated superseded-by-F-010. -- Debt rows D-131 (ADR-0034 deferred build blocks, from the D-100(b) - cycle) / D-132 (quality-loop phase structure) / D-133 (JIT - coverage-floor prerequisite) minted. -- `.dev/reference_clones.md` gains the Quality-elevation corpora - (`clojure-corpus` + `clojuredocs-export-edn`). -- No phase renumber lands now; the M-exit entry owner concretises the - quality-loop phases + the JIT/coverage ordering with full context. -- F-001 retains "eventual unavoidable" status; the zwasm/CLJS entry-debt - cluster (D-036..039) relocates intact at its owning entry, not deleted. -- v0.1.0 (Phase 14) is unaffected — it completes first; D-100(b) - `cljw build` remains its in-flight deliverable. - -## Affected files - -- `.dev/project_facts.md` — F-010 (the governing invariant; landed same - session). -- `.dev/ROADMAP.md` — §9 re-sequencing note + §9.2 annotation + light - pointers on the Phase 16/17/19/20 tracker context + §14.1 F-010 cross-ref. -- `.dev/debt.md` — D-131 / D-132 / D-133 rows. -- `.dev/reference_clones.md` — Quality-elevation corpora section. -- `private/notes/recut-goal-synthesis.md` + the 3 surveys — grounding. - -## Revision history - -- 2026-05-29 issued + accepted with Devil's-advocate fork - (general-purpose, fresh context, F-010/F-002/F-003/F-001 envelope, 3 - alternatives verbatim, Alt B selected). Records the user-directed - interim-goal re-cut (F-010) as a ROADMAP §9 re-sequencing expressed - per F-003 (light re-label + debt-recorded foresight, granularity - deferred to owning Phase entries). diff --git a/.dev/decisions/0052_universal_equality_eq_vs_numeric_equiv.md b/.dev/decisions/0052_universal_equality_eq_vs_numeric_equiv.md deleted file mode 100644 index 6c6002e71..000000000 --- a/.dev/decisions/0052_universal_equality_eq_vs_numeric_equiv.md +++ /dev/null @@ -1,170 +0,0 @@ -# 0052 — Universal `=` (value equality) vs `==` (numeric-tower equivalence) - -**Status**: Accepted (Devil's-advocate fork landed 2026-05-29) -**Date**: 2026-05-29 -**Author**: Shota Kudo (drafted with Claude autonomous loop) -**Tags**: correctness-bug, equality, F-005, F-009, D-136, numeric-category - -## Context - -`=` in cw v1 was a **correctness bug**: registered in -`lang/primitive/math.zig` as `equals` → `pairwise("=", … pEQ)` -(numeric pairwise via `ensureNumeric`), so `(= :a :a)` / `(= "a" "a")` -/ `(= [1 2] [1 2])` / `(= true true)` / `(= nil nil)` / `(= 1 nil)` -all raised `type_error` instead of returning a boolean. JVM `=` -(`clojure.lang.Util.equiv`) is universal value equality and never -raises. No `==` existed. Surfaced 2026-05-29 writing `dedupe` (D-134); -this bug blocks the whole coverage floor (D-133) — dedupe / distinct / -frequencies / group-by / set-membership on non-numbers all need it. - -Survey: `private/notes/phase14-D136-equality-survey.md`. Findings: -bit-pattern equality is correct for nil/bool/int/char/builtin_fn + -interned keyword/symbol; cw v1 has NO structural equality (greenfield); -`hash.zig` is Murmur3 only (the map-comment pointer to "deep = via -hash.zig" is misdirected). cw v0's `=` was **category-blind** -(`(= 1 1.0)`→true) = non-JVM; cw v1 diverges to JVM's category gate. - -## Decision - -### D1: `=` = universal value equality; `==` = numeric-tower equivalence - -- New neutral `src/runtime/equal.zig` (F-009) exposes - `valueEqual(rt, a, b) bool`. The `=` primitive folds it pairwise. -- `==` is added as the numeric primitive = the current `=` body - renamed (`pairwise("==", … pEQ)`); it keeps the numeric-only - `type_error` for non-numbers (JVM `==` is numeric-only too). -- `valueEqual` **never raises** — different/unhandled tags → false. - -### D2: `valueEqual` dispatch (Alt 2, finished-form-clean) - -1. **Identity fast path**: `@intFromEnum(a) == @intFromEnum(b)` → true. - Covers nil / bool / int / char / builtin_fn / interned keyword / - interned symbol / pointer-identical heap. -2. **Numeric arm** (both numeric), category-gated per **F-005**: - category(int, big_int) = `.integer`; float = `.floating`; - ratio = `.ratio`; big_decimal = `.decimal`. **Different category → - false** (`(= 1 1.0)`→false). Within category: int i48 eq, - `big_int` `Managed.eql`, float `==`, ratio reduced num/den eq, - big_decimal `compareTo == 0`. **All five constructible categories - are covered now** — leaving big_int/ratio/decimal on the - bit-pattern fallthrough would make `(= 1N 1N)` (distinct heap allocs) - → false, a silent wrong answer (permanent-no-op). -3. **Sequential arm** (both sequential): element-wise via a unified - tagged first/rest cursor, so `(= '(1 2) [1 2])`→true (JVM-correct). - Maps/sets are NOT sequential (`(= #{1} [1])`→false). - **Scope (implementation re-lay, Step 0.6)**: `valueEqual` lives at - Zone 0 (`runtime/equal.zig`) taking `rt` (signature - `valueEqual(rt, a, b) anyerror!bool` — real errors like OOM / - chunked-rest propagate; type mismatches return false). The general - seq-walk machinery (`sequence.zig`) is Layer 2 and `lazy_seq` - realization needs `env`, so the Zone-0 cursor covers the tags with - Zone-0 first/rest: **vector (nth/count), list, cons, chunked_cons**. - `range` / `array_seq` / `string_seq` / `lazy_seq` sequential - equality is **deferred** (env / lazy-realization dependency — these - are also barely constructible at Phase 14: range/repeat/iterate are - unimplemented per D-134). Tracked as a D-136 follow-up note. -4. **Same-tag content arms**: string (byte eq); array_map/hash_map - (count + per-key `get` + recurse, key lookup rides existing - bit-pattern `keyEq` — see D3); hash_set (count + `contains`); - keyword/symbol (name eq — backstop for any non-interned case). -5. **Different tag** (after the cross-type arms) → false. - -### D3: scope boundary — keyEq widening deferred to D-092 (a real dependency) - -The map ArrayMap `keyEq` stays bit-pattern this cycle. Widening it to -`valueEqual` requires a **validated structural `valueHash`** in -`hash.zig` (hash/eq consistency: equal values → equal hashes), which -does not exist. That is a genuine correctness dependency, **not** a -cycle-budget defer — tracked as **D-092**. `=` as a standalone -predicate is fully correct over structural collections without it -(only collection-keyed map/set *lookup* is affected, not `=`). - -Cross-category `==` (`(== 1N 1.0)`) is also deferred — it needs the -numeric-tower combine ladder (D-014a family), a real dependency. `=` -never needs it (the category gate makes cross-category `=`→false). - -### D4: recursion - -Persistent immutable structures cannot cycle, so `valueEqual` -recursion terminates. No depth guard for v0.1.0 (only pathological -deep nesting risks stack; not a daily-driver concern). Known limit, -revisit if a crash surfaces. - -## Alternatives considered - -Devil's-advocate fork (general-purpose, fresh context, 2026-05-29, -F-005/F-009/F-002 envelope) output verbatim: - -**Envelope reminder.** F-005 fixes the numeric-category *direction* (`(= 1 1.0)`→false, `==` widens) — not open. F-009 fixes the *location* (`runtime/equal.zig`, thin `=`/`==` wrappers) — not open. The open axes: (i) sequential cross-type now vs defer, (ii) numeric-category scope, (iii) `==` now vs later, (iv) keyEq widening in/out, (v) recursion depth guard. No alternative requires an F-NNN amendment. - -### Alt 1 — Smallest-diff - -(a) `valueEqual` with identity fast-path → numeric arm covering **int+float only** (big_int/ratio/big_decimal fall to same-tag fallthrough) → same-tag by-content arms (string byte eq; vector count+nth; list/cons same-tag-only walk, NO cross-type; map count+get; set contains) → different-tag→false. `==` = current `=` body renamed. Out: sequential cross-type, big/ratio/decimal same-category, keyEq (D-092), depth guard. - -(b) Better: least surface; lands the P0 fix touching only math.zig + new equal.zig; every deferred axis has a D-row. - -(c) Breaks: `(= '(1 2) [1 2])`→false diverges from JVM (F-005 surface mismatch; into/concat/distinct over mixed seq+vec mis-dedupe). `(= 1N 1N)` on distinct big_int heap allocs → false — a silent wrong answer (permanent-no-op row 4) with no marker. That last is the dangerous risk. - -### Alt 2 — Finished-form-clean (recommended) - -(a) Same file/signature. Numeric arm covers **all five categories** (integer{int,big_int}/floating/ratio/decimal) with category gate (F-005), within-category via int eq / `big_int Managed.eql` (big_int.zig:238) / float == / ratio num·den / big_decimal compareTo. **Sequential cross-type arm**: `isSequential` = list/cons/chunked_cons/vector/range/array_seq; both sequential → element-wise unified first/rest-or-nth cursor (`(= '(1 2) [1 2])`→true). Same-tag content arms (string/kw/sym/map/set) as Alt 1. `==` added now. keyEq widening OUT (D-092). No depth guard (bounded-nesting note). - -(b) Better: matches JVM `=` on every Phase-14-constructible value in one cycle; dedupe/distinct/set-membership/frequencies/group-by behave correctly over mixed collections immediately; no silent wrong answers (every constructible numeric category compared correctly). The sequential cursor is the natural shape — a later list==vector cycle would rewrite the list/vector arms anyway, so doing it now *shrinks* total work (permanent_noop_forbidden: skeleton must shrink the rewrite). - -(c) Breaks: larger cycle (unified cursor + 5-category ladder = more code + test surface); big_decimal/ratio same-category compare has scale/reduction subtleties (mitigated: ratios stored reduced, big_decimal via compareTo==0); the sequential cursor MUST treat maps/sets as non-sequential (`(= #{1} [1])`→false) — covered by a negative test. - -### Alt 3 — Wildcard - -(a) Land `valueEqual` PLUS structural `valueHash` in `equal.zig`, then widen map `keyEq` to `valueEqual` + wire structural hash into map key lookup — fold D-092 into this cycle. Numeric all-category + sequential as Alt 2. `==` now. - -(b) Better: complete Clojure equality story in one shot — `(get {[1] :x} [1])`→`:x`, collection-keyed maps/sets work; eliminates the D-092 follow-up. - -(c) Breaks: a genuine **dependency-bearing scope jump, not cycle-budget**: keyEq→valueEqual is only correct with a structural `valueHash` satisfying the hash/eq invariant for string/collection keys + a re-audit of the map's internal probing. `hash.zig` has no structural Value composition today; building+validating `valueHash` (collision behavior, ordered vs unordered for vector vs set/map) is its own correctness surface. Folding couples two independently-testable fixes, enlarging blast radius. **Distinct from F-002**: F-002 forbids deferring on *cycle size*, not respecting a real *correctness dependency boundary*; the hash/eq-consistency boundary is legitimate, so deferring keyEq (D-092) is NOT a Cycle-budget defer smell. - -### Recommendation (non-binding) — Alt 2, anchored to F-002 + F-005 - -1. **Sequential cross-type — INCLUDE now.** JVM-observable (F-005) + the D-134 dedupe/distinct unblock depends on it for mixed seq+vec; list+vector arms both need a first/rest-or-nth walk anyway, so unifying now is *smaller total* than same-tag-now + cross-type-rewrite-later. Deferring = Cycle-budget defer smell (no dependency block). Overrides survey §8's tentative defer. -2. **Numeric — ALL constructible categories now.** big_int/ratio/big_decimal are constructible at Phase 14 (`big_int.eql` exists); leaving them on bit-pattern → `(= 1N 1N)`→false silent lie (worse than a raise). Category gate is free (F-005). Cross-category `==` stays deferred (combine-ladder dependency, D-014a) — `=` never needs it. -3. **`==` — ADD now** (current `=` body renamed + one ENTRIES row; zero new logic). -4. **keyEq widening — OUT (D-092)**: the one *legitimate* defer — needs validated structural `valueHash` (hash/eq consistency), a separable correctness surface, not a budget punt. `=` is fully correct without it. -5. **Recursion depth — TRUST bounded** for v0.1.0 (immutable → terminates; one-line note, not a marker). - -Why Alt 2 over Alt 1: Alt 1's deferred arms are held back only by diff size (F-002 rejects that) and it ships the `(= 1N 1N)`→false lie. Why over Alt 3: Alt 3 crosses the real D-092 hash/eq dependency without a finished-form *correctness* gain to `=` itself. - -## Selection rationale - -Alt 2. It is the shape where every Phase-14-observable JVM `=` behavior -is correct (numeric category gate per F-005, sequential cross-type, -structural collections) and nothing is deferred except what a real -dependency forces (keyEq/hash → D-092; cross-category `==` → combine -ladder). Alt 1 ships a silent `(= 1N 1N)`→false lie + diverges on -sequential; Alt 3 couples the D-092 hash/eq surface without a -correctness gain to `=`. - -## Consequences - -- New `src/runtime/equal.zig` (`valueEqual` + numeric-category helper). -- `lang/primitive/math.zig`: `=` rewired to `valueEqual`; `==` added as - the numeric primitive (former `=` body). Both Tier A by clojure.core - aggregate (no per-var compat_tiers entry). -- Unblocks D-134 dedupe/distinct/frequencies/group-by (next cluster). -- D-092 (keyEq widening + structural valueHash) referenced as the - follow-up for collection-keyed map/set lookup; cross-category `==` - deferred to the numeric-combine ladder (D-014a family). -- Existing number-vs-number `=` tests keep passing. - -## Affected files - -- `src/runtime/equal.zig` (new) · `src/lang/primitive/math.zig` - (= rewire + == add) · `test/e2e/phase14_equality.sh` (new) · - `src/main.zig` test aggregator (if equal.zig needs reachability) · - `.dev/debt.md` (D-136 discharge, D-092 reference). - -## Revision history - -- 2026-05-29 issued + accepted with Devil's-advocate fork - (general-purpose, fresh context, F-005/F-009/F-002 envelope, 3 - alternatives verbatim, Alt 2 selected). Fixes the numeric-only `=` - bug (D-136); splits `=` (universal, Util.equiv) from `==` (numeric, - Numbers.equiv) per JVM; category-gated numeric `=` per F-005 - (diverging from cw v0's category-blind `=`). diff --git a/.dev/decisions/0053_general_compare.md b/.dev/decisions/0053_general_compare.md deleted file mode 100644 index df5321a17..000000000 --- a/.dev/decisions/0053_general_compare.md +++ /dev/null @@ -1,147 +0,0 @@ -# 0053 — General 3-way `compare` (clojure.lang.Util.compare) - -**Status**: Accepted (Devil's-advocate fork landed 2026-05-29) -**Date**: 2026-05-29 -**Author**: Shota Kudo (drafted with Claude autonomous loop) -**Tags**: correctness-bug, compare, ordering, F-005, F-009, D-137, sibling-of-ADR-0052 - -## Context - -`compare` in cw v1 is numeric-only (`math.zig` arity-2 + `ensureNumeric` -+ f64/i64 compare), so `(compare :a :b)` / `(compare "a" "b")` / -`(compare [1] [2])` raise type_error; only numbers compare. JVM -`clojure.lang.Util.compare` is a general 3-way comparator. This is the -direct sibling of D-136/ADR-0052 (universal `=`); it blocks -`sort`/`sort-by` (deferred behind it) + future sorted-map/sorted-set. - -Survey: `private/notes/phase14-D137-compare-survey.md`. Mirrors ADR-0052 -/ `runtime/equal.zig` structurally, with three crux differences: -3-way (`std.math.Order`); numeric **crosses the tower with NO category -gate** (`(compare 1 1.0)`→0 — opposite of `=`'s F-005 gate); mismatched -types **RAISE** (compare is not under the never-raise contract `=` has). - -## Decision - -### D1: `valueCompare` in a new neutral `runtime/compare.zig` - -`pub fn valueCompare(rt, a, b, loc) anyerror!std.math.Order`. A **new -sibling file** to `equal.zig` (NOT folded in): the two numeric arms are -structurally opposite (F-005 category gate vs no-gate), and a shared -file where "the numeric arm" means two contradictory things is a -maintainer hazard. ROADMAP A2 ("new features via new files") + F-009. -The `compare` primitive becomes a thin wrapper returning a -1/0/1 -integer Value. Takes `loc` because it raises on uncomparable pairs. - -### D2: dispatch (JVM Util.compare order) - -1. identity (`@intFromEnum` eq) → `.eq`. -2. **nil lowest**: nil vs nil → eq; nil vs x → lt; x vs nil → gt. -3. **numeric arm — HYBRID** (both numeric): same-category exact via the - existing Order fns (`int` i48 `std.math.order`; big_int↔big_int - `big_int.compareManaged`; ratio↔ratio `ratio.compareValue`; - decimal↔decimal `big_decimal.compareValue`); genuinely mixed-tower - (any float, or cross big-category) → `promote.toF64` collapse - (matches cw's existing `<`/`>`). **No category gate** (crosses - tower per F-005). Exact cross-category compare → **deferred** to the - numeric combine ladder (D-014a family — same dependency that gates - cross-category `==` in ADR-0052 D3; it also carries the NaN-ordering - subtlety: JVM `(compare ##NaN 1)`→1 while `(< ##NaN 1)`→false). -4. string → `std.mem.order` (lexicographic) → lt/eq/gt. -5. char → `asChar` u21 `std.math.order`. -6. bool → false < true. -7. keyword / symbol → ns-then-name: nil-ns sorts before non-nil-ns, - then `std.mem.order(ns)`, tiebreak `std.mem.order(name)`. Accessors - confirmed: `keyword.asKeyword(v).ns/.name`, `symbol.asSymbol(v).ns/.name`. -8. vector → **length-first** then element-wise recursion - (`APersistentVector.compareTo`: shorter < longer; equal length → - first differing element). -9. else (mismatched / uncomparable, incl. **lists** — not Comparable in - JVM) → raise `type_error` (same observable as JVM's CCE). - -### D3: sort/sort-by follow-up — stable-sort MANDATE (separate cycle) - -This unblocks `sort`/`sort-by` (D-134, deferred behind D-137). The sort -**algorithm** is a separate cycle (it needs eager materialisation + a -comparator closure). **The follow-up MUST use a stable sort** — Clojure -`sort` is stable; `std.sort.pdq` / `std.sort.block` are NOT guaranteed -stable. A hand-rolled stable mergesort over a materialised vector with -`valueCompare` as comparator is the target. Stability is a **contract, -not an optimisation**: the next cycle's smell sensor must treat "just -use `std.sort.pdq`" as a correctness regression. Recorded now so the -sort cycle cannot silently ship an unstable sort. - -## Alternatives considered - -Devil's-advocate fork (general-purpose, fresh context, 2026-05-29, -F-005/F-009/F-002 envelope) output verbatim: - -**Envelope reminder.** F-005 fixes the numeric *direction*: `compare` crosses the tower with **no category gate** (`(compare 1 1.0)`→0) — the opposite of `=`'s gate, and NOT open. F-009 fixes the *location*: neutral `runtime/` impl, thin `math.zig` `compare` wrapper — NOT open. F-002: cycle/diff/LOC is not a constraint; only a real dependency justifies deferral. The genuinely open axes are: (1) mixed-tower numeric **precision**, (2) **file placement** (new `compare.zig` vs fold into `equal.zig`), (3) **kw/sym** scope, (4) **list** compare, (5) **stable-sort mandate** for the sort follow-up. **No alternative below requires an F-NNN amendment** — the accessors for kw/sym ns·name exist (`keyword.asKeyword`/`symbol.asSymbol`, confirmed), the same-category Order fns exist (`big_int.compareManaged`/`ratio.compareValue`/`big_decimal.compareValue`), and `promote.toF64` exists. The only thing that does NOT exist is a unified cross-category numeric `Order` ladder (`comparePromoting`) — that is the one real dependency in play, and it is the SAME ladder D-136/ADR-0052 deferred cross-category `==` behind (D-014a family). - -### Alt 1 — Smallest-diff - -(a) **All in one file**: extend `equal.zig` (do NOT add `compare.zig`). Add `valueCompare(rt, a, b, loc) anyerror!std.math.Order` beside `valueEqual`, reusing the in-file `numCat`/`isSequential`/`Cursor` directly (no import, no duplication). **Numeric arm = pure f64-collapse** for everything via `promote.toF64(a)` vs `toF64(b)` — exactly mirrors the current `compare` body + `pairwise` + `<`/`>`. nil-lowest + identity fast-path + string (`std.mem.order`) + bool + kw/sym (ns-then-name) + vector (length-first). **List → raise** (uncomparable, mirrors JVM). **kw/sym INCLUDED** (accessors exist, trivial). Rewire `math.zig` compare to call it, drop `ensureNumeric`. Sort follow-up: **no stability mandate** — use `std.sort.pdq` (fast, unstable) and document the divergence. - -(b) Better: smallest surface — one file touched in `runtime/`, helpers shared with zero duplication, the numeric arm is ~4 lines (one `toF64` pair + `std.math.order`). Numeric behaviour is byte-identical to the existing `<`/`>`/`compare` surface, so there is zero new numeric-precision *inconsistency* introduced — `(compare big big)` is exactly as imprecise as `(< big big)` already is. Lands the whole D-137 fix + unblocks D-134 sort in minimal diff. - -(c) Breaks / risks: **(i) the shared-file numeric-semantics trap.** `equal.zig`'s numeric arm is **category-GATED** (F-005, `if (ca != cb) return false`); `compare`'s is **un-gated** (crosses tower). Two functions in one file where "numeric arm" means structurally-opposite things is exactly the confusion the survey §5 flagged — a future reader patching one gate risks touching the other. **(ii) f64-collapse for same-category big values is a needless precision regression** vs free precision: `(compare 1N (inc (bit-shift-left 1 60)))` collapses two distinct big_ints to equal f64 and returns `0` (wrong) even though `big_int.compareManaged` is sitting right there returning the exact Order for free — `valueEqual` already calls it. Using f64 where an exact same-category Order fn exists and is already imported is a *silent wrong answer* with no dependency excuse (worse than the `<`/`>` case, which has no such fn wired). **(iii) unstable sort** makes `(sort-by f coll)` reorder f-equal elements — observably non-Clojure; `sort` stability is part of the contract, not an optimisation. - -### Alt 2 — Finished-form-clean (recommended) - -(a) **New `src/runtime/compare.zig`** (Zone 0, sibling of `equal.zig`, F-009 + ROADMAP A2 "new features via new files"). It duplicates the *tiny* helper shapes it needs (`numCat`, length-first vector walk) rather than importing `equal.zig`, because the two files' numeric arms are deliberately opposite (gate vs no-gate) and must not share a "numeric arm" abstraction. Signature `valueCompare(rt, a, b, loc) anyerror!std.math.Order` (`loc` needed because compare RAISES on uncomparable, unlike `valueEqual`). Dispatch: identity→`.eq`; nil-lowest; **numeric HYBRID** (see below); string `std.mem.order`; bool false` precision behaviour). Cross-category-EXACT numeric compare → **DEFER** to the numeric-combine ladder (D-014a family, the same row that gates cross-category `==`). Sort follow-up: **mandate stable sort** in the ADR (the follow-up cycle uses a stable mergesort over a materialised vector with `valueCompare` as comparator), but the sort *algorithm* is a **separate cycle** — this ADR only records the *mandate*, not the impl. - -(b) Better: (i) same-category big/ratio/decimal compares are **exact for free** by routing to the Order fns that already exist and that `valueEqual` already calls — no silent `(compare 1N hugeN)`→0 lie. (ii) Separate file keeps the gate/no-gate semantics from colliding — a reader of `compare.zig` sees "numeric crosses tower" without the F-005 gate noise, and a reader of `equal.zig` sees the gate without the crossing noise. (iii) kw/sym/char/vector/bool all land now (every accessor exists; no dependency forces a defer — deferring any of them would be a pure Cycle-budget defer smell). (iv) The single deferral (exact cross-category) is anchored to a **real** missing primitive (`comparePromoting`), identical in kind to ADR-0052's cross-category `==` defer — provable as dependency-driven, not budget-driven. (v) Recording the stable-sort mandate now prevents the sort cycle from silently shipping `std.sort.pdq` (unstable) and diverging from Clojure's stable `sort`. - -(c) Breaks / risks: (i) a few helper shapes (`numCat`, length-first walk) are duplicated between `equal.zig` and `compare.zig` — mild DRY cost, justified by the opposite-semantics argument; if it grates later, a third `numeric_category.zig` could host `numCat` for both (out of scope now). (ii) the hybrid's "mixed big-category → f64" branch is a documented precision limit, not exact JVM parity — but it matches cw's own `<`/`>` and the exact path is a genuine dependency, so this is a DIVERGENCE-with-rationale, not a lie (and same-category, the common constructible case, stays exact). (iii) larger cycle than Alt 1 (the hybrid ladder + char arm + a real ADR) — but F-002 explicitly removes cycle size as a constraint. - -### Alt 3 — Wildcard - -(a) **Build the cross-category numeric `Order` ladder NOW** (`promote.comparePromoting(rt, a, b) !std.math.Order`) and make BOTH `compare`'s numeric arm AND the deferred cross-category `==` ride it — i.e. fold the D-014a-family numeric-combine dependency into this cycle. `comparePromoting` would mirror `addPromoting`'s contagion ladder (decimal > ratio > float > integer) but compute an Order instead of a Value: promote both operands to the widest category via `coerceToManaged`/ratio cross-multiply/big_decimal scale-align, then call the existing same-category Order fn. With it, `compare`'s numeric arm becomes a single `comparePromoting` call (exact across the whole tower, no f64 anywhere), `==`'s cross-category gap (ADR-0052 D3) closes for free, and `valueCompare` over numbers is fully JVM-exact. `compare.zig` as in Alt 2 for the non-numeric arms; stable-sort mandate as Alt 2. - -(b) Better: the **complete** Clojure numeric-comparison story in one shot — `(compare 1N (inc (bit-shift-left 1 60)))`, `(compare 1/3 0.333M)`, `(== 1N 1.0)` all exact; eliminates the D-014a-family follow-up for both `compare` and `==`; no f64 precision footnote anywhere; the numeric surface (`<`/`>`/`<=`/`>=` could later route through the same Order ladder too) becomes uniformly exact. - -(c) Breaks / risks: this is a **genuine dependency-bearing scope jump that belongs to its own cycle**, distinct from a cycle-budget defer. `comparePromoting` is the D-014a-family combine ladder — building it correctly means: float↔ratio comparison (a float is not exactly a ratio; needs a defined rounding/exactness rule matching `Numbers.java`'s `ratioToBigInteger`/`toBigDecimal` paths), big_decimal↔ratio scale-alignment, NaN ordering (JVM `Numbers.compare` has specific NaN behaviour: `Double.compare` sorts NaN as greatest, which `<` does NOT — so the ladder is NOT just "reuse `<`"), and a re-audit of how `<`/`>`/`==` should *also* migrate onto it for consistency. That is its own correctness surface with its own test matrix (the NaN-ordering subtlety alone is a trap: `(compare ##NaN 1)` returns `1` in JVM but `(< ##NaN 1)` is `false`). Folding it into the `compare`-3-way cycle couples two independently-testable fixes and enlarges blast radius across the entire numeric-comparison family. **This is the ADR-0052-Alt-3 shape exactly**: a real dependency boundary (there, hash/eq-consistency → D-092; here, the exact cross-tower combine ladder + NaN-order semantics → D-014a family), so deferring it is NOT a Cycle-budget defer smell — it is respecting a legitimate correctness boundary. - -### Recommendation (non-binding) — Alt 2 - -Answering the five axes: (1) numeric **HYBRID** (same-category exact via existing Order fns; mixed-tower f64-collapse matching `<`/`>`; cross-category-exact deferred to the D-014a combine ladder — pure-f64 Alt 1 ships the `(compare 1N hugeN)`→0 lie and is the Cycle-budget defer smell). (2) **New `compare.zig`** (opposite numeric semantics from `equal.zig` — shared file is a hazard; A2 + F-009). (3) kw/sym **INCLUDE now** (accessors confirmed; deferring = pure cycle-budget smell). (4) list **RAISE** via else-arm (mirrors JVM CCE; not a divergence). (5) stable-sort **MANDATE in ADR, IMPLEMENT in a separate cycle** (stability is a contract; the sort algorithm needs materialisation + comparator closure = own cycle). Alt 2 over Alt 1: Alt 1's wins (one file, pure-f64) are anti-finished-form, held back only by diff size. Alt 2 over Alt 3: Alt 3 crosses the real D-014a combine-ladder boundary (NaN-ordering + float↔ratio exactness) without a correctness gain to the 3-way story that same-category-exact + mixed-f64 doesn't already deliver for every Phase-14-constructible value. - -## Selection rationale - -Alt 2. Same-category numeric compares are exact for free (the Order fns -`valueEqual` already uses); mixed-tower f64 matches cw's own `<`/`>`; -the one deferral (exact cross-category) is the real D-014a combine-ladder -dependency (with its NaN-ordering trap), not a budget punt. New -`compare.zig` keeps the gate/no-gate numeric semantics from colliding -with `equal.zig`. kw/sym/char/vector/bool all land (accessors confirmed). -List-raises mirrors JVM. Stable-sort mandated for the follow-up cycle. - -## Consequences - -- New `src/runtime/compare.zig` (`valueCompare`); `math.zig` `compare` - rewired to it (drops `ensureNumeric`; returns -1/0/1). Tier A by - clojure.core aggregate. -- The `math.zig` "non-numeric compare → type_error" test narrows to a - genuinely-uncomparable pair (e.g. `(compare 1 "a")`), since most - former type_errors now succeed. -- **Unblocks D-134 sort/sort-by** — separate cycle, MUST use a stable - sort (D3). -- Deferred: exact cross-category numeric compare + NaN-ordering → - D-014a-family combine ladder (same dep as cross-category `==`); - sorted-map / sorted-set ordering. -- DIVERGENCE: mixed-tower big-category compare uses f64-collapse (matches - cw `<`/`>`), not exact — documented limit, not a silent lie - (same-category stays exact). - -## Affected files - -- `src/runtime/compare.zig` (new) · `src/lang/primitive/math.zig` - (compare rewire) · `test/e2e/phase14_compare.sh` (new) · - `.dev/debt.md` (D-137 discharge). - -## Revision history - -- 2026-05-29 issued + accepted with Devil's-advocate fork - (general-purpose, fresh context, F-005/F-009/F-002 envelope, 3 - alternatives verbatim, Alt 2 selected). Sibling of ADR-0052; fixes the - numeric-only `compare` bug (D-137). Numeric crosses the tower (no - category gate, per F-005 — opposite of `=`); mismatched types raise. diff --git a/.dev/decisions/0054_lazy_seq_layer2.md b/.dev/decisions/0054_lazy_seq_layer2.md deleted file mode 100644 index cc0300094..000000000 --- a/.dev/decisions/0054_lazy_seq_layer2.md +++ /dev/null @@ -1,180 +0,0 @@ -# 0054 — lazy-seq Layer-2: wire the producer + convert the eager seq surface to true laziness - -**Status**: Accepted (Devil's-advocate fork landed 2026-05-29) -**Date**: 2026-05-29 -**Author**: Shota Kudo (drafted with Claude autonomous loop) -**Tags**: structural, lazy-seq, seq-protocol, F-002, F-006, F-009, D-133, phase-6-gap - -## Context - -cw v1's clojure.core seq surface (`map`/`filter`/`take`/`drop`/`keep`/ -`remove` + `concat`/`mapcat`/`range`/`map-indexed`/`keep-indexed`) is -EAGER (returns vectors via `-*-eager` Zig leaves / Pattern-A `.clj`), -diverging from JVM lazy seqs. This is the **last unmet Phase-6 -exit criterion** (ROADMAP §9 row 6: `(take 5 (iterate inc 0))` → -`(0 1 2 3 4)`) and the D-133 coverage-floor item "seqs eager not lazy". - -Survey (`private/notes/lazy-seq-layer2-survey.md`): the lazy_seq -**CONSUMER** half is complete + unit-tested (`runtime/lazy_seq.zig` -force/first/rest/next/seq + GC trace + root category; the seq-protocol -`.lazy_seq` arms in `sequence.zig` are already wired), but the -**PRODUCER** half has **zero callers** — no `lazy-seq` macro, no -`__lazy-seq-create` primitive, and `iterate` does not exist. -`delay`/`future` are the proven producer triad (macro → -`(__X-create (fn* [] body))` → primitive → `mod.alloc(rt, thunk)`). - -DA fork surfaced a load-bearing constraint the survey under-weighted: -**`print.zig::printForm` takes `(w, v)` — NO `rt`/`env`** — so it cannot -`force` a lazy seq; `.lazy_seq` currently falls to `else => "#"`. -Rendering a lazy seq requires threading `rt + env` into the print path -(rippling to REPL / `pr-str` / error rendering). This is in-scope for -**cycle 1**, because cycle 1's exit criterion is a *print* of a lazy seq. - -## Decision (Alt B — finished-form-clean) - -### D1: producer triad (mirror delay/future) - -`lazy-seq` macro (reuse the `expandThunkWrapper("__lazy-seq-create", …)` -shape) → `__lazy-seq-create` primitive (`checkArity(1)` → -`lazy_seq.alloc(rt, thunk)`, env discarded like `delayCreateFn`) → -the existing `lazy_seq.alloc`. ~13 LOC total. - -### D2: convert the eager surface to lazy `.clj`; delete the `-*-eager` leaves - -`map`/`filter`/`take`/`drop`/`keep`/`remove`/`concat`/`mapcat`/ -`map-indexed`/`keep-indexed` become recursive lazy-cons `.clj` -(`(def map (fn* [f coll] (lazy-seq (when-let [s (seq coll)] (cons (f (first s)) (map f (rest s)))))))`). -Each `-*-eager` Zig leaf is **deleted in the same cycle that removes its -last caller** (no dead code across cycles; no `PROVISIONAL:` marker — -deletion is atomic with the rewrite). JVM-idiomatic, F-009-thin. - -### D3: print path gains `rt`/`env` (cycle 1, load-bearing) - -Thread `rt + env` into `printForm` (overload or signature change); add a -`.lazy_seq` arm that forces + walks like `printList`, **bounded** (honour -`*print-length*` / a finite default) so an infinite seq does not hang the -printer. Ripples to REPL / `pr-str` / error rendering — done once, here. - -### D4: lazy `=` (cycle 3) - -`equal.zig`'s sequential arm (`isSequential` = vector|list today) extends -to force-and-walk `.lazy_seq` (symmetric `rt/env` threading), landing -with the first `(= )` assertion. Genuine dependency at -first-use, not deferrable polish. - -### D5: chunking DEFERRED (separation of concern, not cycle-budget) - -Unchunked per-element lazy-cons is the correct **semantic** finished -form. JVM's 32-element chunking is an orthogonal **perf** axis (the -`chunked_cons` module exists, dormant). Chunking is a clean follow-up -perf cycle (Phase-15+/JIT era; D-row) once unchunked lazy is the proven -baseline. Chunking now would couple perf into a correctness ADR + change -the laziness contract (over-realization past a consumer's stop point). - -### D6: phasing — one ADR, ~4 cycles, inside the v0.1.0 window - -Per D-133/D-134 lazy seqs are the v0.1.0 coverage floor; eager map/filter -is a JVM-divergence lie (F-002). Phase 6 left a known structural gap; -honour it inside the v0.1.0 window (a §9.16 row). Cycles: - -1. producer triad + `iterate` + lazy `take` + **print `rt/env` threading** - → prove `(take 5 (iterate inc 0))` prints `(0 1 2 3 4)`. -2. lazy `map`/`filter`/`keep`/`remove` + delete those 4 leaves + laziness - oracle `(first (map inc (range))) → 1` (must not hang). -3. lazy `concat`/`mapcat`/`drop` + infinite `range` + **lazy `=`** + - `(= (take 3 (range)) '(0 1 2))`. -4. `repeat`/`repeatedly`/`cycle`/`take-while`/`drop-while`/`partition`. - -### D7: realization-timing de-risk - -(i) cycle-2 grep audit for `(map `-style call sites + -confirm/land `doall`/`dorun`; (ii) the laziness oracle as the eager→lazy -regression gate; (iii) an F-006 GC-pressure stress test (`force` mid -minor-GC) — verification only, the trace is complete (no GC redesign). - -## Alternatives considered - -Devil's-advocate fork (general-purpose, fresh context, 2026-05-29, -F-002/F-006/F-009/F-004 envelope) output verbatim: - -> Critical finding: `print.zig`'s switch falls to `else => |t| w.print("#<{s}>", .{@tagName(t)})` for `.lazy_seq` — so a lazy seq today would print `#`, not `(0 1 2 3 4)`. And crucially, **`print` takes only `(w, v)` — no `rt`, no `env`** — so it cannot call `force`/`seqFn` (which require `rt, env`). This is a load-bearing signature problem, not just a missing arm. - -### Leading finding: no F-NNN block, but one cross-cutting signature constraint - -There is **no** finished-form-clean option that requires violating an F-NNN. The producer triad + lazy `.clj` rewrite is fully expressible within F-002/F-006/F-009/F-004. **However**, code reading surfaced a constraint the survey under-weighted that reshapes the phasing of all three alternatives: - -- **`print.zig::printForm` takes `(w: *Writer, v)` — NO `rt`, NO `env`.** Its switch routes `.list => printList`; `.lazy_seq` falls into `else => w.print("#<{s}>", .{@tagName(t)})`. So the first lazy value ever produced prints `#`. Forcing a lazy seq for printing **requires threading `rt + env` into the print path** (to call `force`/`seqFn`, which are `(rt, env, v)`). This is not a one-arm addition — it is a print-API signature change rippling to every `printForm` caller (REPL, `pr-str`, error rendering). The survey called lazy-print "a probable hidden sub-task"; reading confirms it is **load-bearing and must land in the same cycle as the producer**, because `(take 5 (iterate inc 0))` at the REPL is exactly a print-of-lazy. This is a **genuine dependency**, not a cycle-budget defer. -- Symmetric but lower-severity: `equal.zig` explicitly defers `.lazy_seq` from `isSequential` (vector or list only). It likewise would need `rt + env` to force-and-walk. But `=` on a lazy result is not on cycle-1's critical path, so this is a real defer to the cycle that introduces the first `(= )` assertion. - -### Alternative A — Smallest-diff: lazy producers only, keep eager surface - -(a) Land only the producer triad + the inherently-infinite cluster (`iterate`/`repeat`/`repeatedly`/`cycle`/infinite `range`) as new lazy `.clj`. **Leave `map`/`filter`/`take`/`drop`/`keep`/`remove`/`concat`/`mapcat` eager.** Thread `rt/env` into print only enough to render the new lazy values. `take`/`take-while`/etc. get lazy `.clj` defns so they compose with infinite producers; eager `map`/`filter` stay. - -(b) Better. Smallest diff; zero `-*-eager` deletion; no project-wide side-effect-timing change. Honours the never-met Phase-6 exit criterion immediately. Lowest GC-pressure surface. - -(c) Breaks/risks. **Ships the divergence-lie F-002 names as the thing to fix.** `(map inc coll)` stays eager → `(first (map inc (range)))` **HANGS** (eager `-map-eager` walks the infinite range forever). A runtime that *looks* lazy but whose most-used seq fns silently aren't. Two-headed seq surface (`map` eager, `take-while` lazy) the finished-form owner unifies later anyway → eager leaves become dead code on a known schedule. Per F-002 the finished form is lazy; A is the smallest-diff-bias path. Reject unless an F-NNN blocked the rewrite — none does. - -### Alternative B — Finished-form-clean: producer triad + full eager-surface rewrite, unchunked, print/equal rt-env threaded (recommended) - -(a) Producer triad (reuse `expandThunkWrapper` ~5 LOC + `__lazy-seq-create` mirroring `delayCreateFn` ~8 LOC). Print path: thread `rt + env` into `printForm`, add `.lazy_seq` force+walk arm, **bound infinite print** via `*print-length*`. Eager→lazy `.clj` rewrite of map/filter/take/drop/keep/remove/concat/mapcat/map-indexed/keep-indexed; **delete the `-*-eager` leaves + LEAF_ENTRIES** with their last caller. Infinite/lazy cluster (`iterate`/`repeat`/`repeatedly`/`cycle`/infinite `range`/`take-while`/`drop-while`/`partition`) as lazy `.clj`; re-unify finite `range` lazy, retiring eager `-range-acc`. Equal path: extend `isSequential` + cursor to force-walk `.lazy_seq` when the first lazy-to-literal comparison lands. Chunking deferred (D-row). Side-effect-timing audit + `doall`/`dorun`. Phasing: one ADR ~4 cycles, print-`rt/env` folded into cycle 1. - -(b) Better. Eliminates the divergence-lie (F-002): every seq fn matches JVM laziness; `(first (map inc (range)))` → `1`. No two-headed surface, no scheduled dead code. Producer JVM-idiomatic `.clj`, Zig surface thin (F-009). Reuses the proven delay/future triad (low mechanism risk). Print/equal threading done once, correctly. Matches the survey depth-3 reading + D-134's "dedicated cycle". - -(c) Breaks/risks. Largest diff (print-API signature ripple to every `printForm` caller — but per F-002 diff size is not a constraint). Side-effect-timing flips project-wide: any `.clj`/test relying on eager `map` side-effects breaks (a *correctness fix*, mitigated by the cycle-2 grep audit + `doall`/`dorun`). GC-pressure under `force`-mid-minor-GC is the F-006 watch item — prudent stress test, **not** a redesign (trace complete). Recursive lazy `map` self-ref `fn*` confirmed working (D-070 discharged). - -### Alternative C — Wildcard: jump straight to chunked lazy seqs - -(a) Skip the unchunked intermediate; `range`/vector-seq/`map`/`filter` produce 32-element `.chunked_cons` batches from day one (the tags + module exist, self-tested only). chunk-first/chunk-rest/chunk-cons protocol arms land alongside. - -(b) Better. Matches JVM's *actual* finished form (chunked); avoids a later unchunked→chunked migration; exercises the dormant `.chunked_cons` machinery; best throughput. - -(c) Breaks/risks. **Chunking is a perf optimisation orthogonal to the correctness/laziness semantics this ADR is about.** Massively enlarges cycle-1 surface (chunk buffer protocol, partial-chunk realization, chunk-boundary first/rest); interaction with infinite seqs where a 32-chunk *over-realizes* past a consumer's stop — a subtle semantic hazard (`(first (map inc (range)))` realizes 32, fine for pure fns but **wrong for side-effecting** ones, changing the laziness contract). Couples a perf decision into a correctness ADR; "each skeleton must shrink the final-form rewrite" — building chunking before unchunked semantics are proven is premature. Survey §5d + D-134 defer chunking to a Phase-15+/perf cycle. Reject for this ADR; chunking is a clean follow-up once unchunked is the proven baseline. - -### Non-binding recommendation — Alt B - -F-002 decides A vs B: eager map/filter is the divergence-lie; "keep eager, rewrite later" is the Cycle-budget defer smell, and no F-NNN forces A. Take the surgery: B. B vs C is finished-form-vs-premature-optimization: C is more finished-form in perf but couples an orthogonal perf concern + changes the laziness contract (over-realization); the semantic finished form is unchunked lazy-cons, chunking a separable perf cycle → defer C (genuine separation-of-concern defer, not budget). Axes: (1) lazy `.clj` + delete leaves; (2) chunking deferred; (3) one ADR ~4 cycles, print-`rt/env` in cycle 1, Phase-14/v0.1.0 window; (3b) delete each leaf with its last caller, no `PROVISIONAL:`; (4) lazy print cycle-1 (signature change, bound via `*print-length*`), lazy `=` cycle-3; (5) de-risk via grep audit + `doall`/`dorun` + laziness oracle + F-006 GC stress (verification, not redesign). - -## Selection rationale - -Alt B. The producer half is the only missing piece (consumer + GC + -seq-protocol arms already complete from Phase 6), so wiring it + the -lazy `.clj` rewrite reaches the JVM-laziness finished form F-002 demands -without an F-NNN block. The print-`rt/env` threading is a genuine cycle-1 -dependency (cycle-1 exit is a print-of-lazy), not budget. Chunking (D5) -and lazy-`=` (D4, cycle 3) are separation-of-concern / first-use -dependencies, not cycle-budget defers. A ships the divergence-lie; C -prematurely couples perf chunking into a correctness ADR. - -## Consequences - -- New: `lazy-seq` macro + `__lazy-seq-create` primitive; `iterate` + - the lazy cluster as lazy `.clj`; lazy `map`/`filter`/… replace the - eager surface; `-*-eager` leaves deleted. -- `print.zig` gains `rt`/`env` (signature ripple to REPL/`pr-str`/error - render); `.lazy_seq` arm bounded by `*print-length*`. -- `equal.zig` sequential arm extends to `.lazy_seq` (cycle 3). -- Deferred: chunking (perf cycle, D-row); `doall`/`dorun` if absent - (cycle 2 prerequisite). -- Watch: F-006 GC-pressure under realization (stress test, not redesign); - side-effect-timing flip (cycle-2 audit). -- Unblocks D-134 lazy cluster (infinite range/iterate/repeat/cycle/ - take-while/drop-while/partition). -- v0.1.0-window §9.16 row opened for this (multi-cycle). - -## Affected files - -- `src/lang/macro_transforms.zig` (`lazy-seq`) · `src/lang/primitive/*` - (`__lazy-seq-create`; delete `-*-eager` leaves) · - `src/lang/clj/clojure/core.clj` (lazy rewrites + cluster) · - `src/runtime/print.zig` (rt/env + `.lazy_seq` arm) · - `src/runtime/equal.zig` (cycle 3) · `test/e2e/phase14_lazy_seq*.sh` · - `.dev/ROADMAP.md` §9.16 (new row) · `.dev/debt.md` (chunking D-row). - -## Revision history - -- 2026-05-29 issued + accepted with Devil's-advocate fork - (general-purpose, fresh context, F-002/F-006/F-009/F-004 envelope, 3 - alternatives verbatim, Alt B selected). Wires the Phase-6 lazy_seq - producer gap; converts the eager seq surface to true laziness across - ~4 cycles inside the v0.1.0 window. Print-`rt/env` threading identified - as a load-bearing cycle-1 dependency. diff --git a/.dev/decisions/0055_binding_special_form_and_error_context_dynamic_var.md b/.dev/decisions/0055_binding_special_form_and_error_context_dynamic_var.md deleted file mode 100644 index 6acbae73c..000000000 --- a/.dev/decisions/0055_binding_special_form_and_error_context_dynamic_var.md +++ /dev/null @@ -1,459 +0,0 @@ -# 0055 — `binding` special form + cw v1's first dynamic var (`cljw.error/*error-context*`) + error-context capture - -**Status**: Accepted (Devil's-advocate fork landed 2026-05-29) -**Date**: 2026-05-29 -**Author**: Shota Kudo (drafted with Claude autonomous loop) -**Tags**: dynamic-var, binding, special-form, error-context, with-context, F-002, F-004, F-009, D-075, D-141-sibling, row-14.13 - -## Context - -Row 14.13 deliverable (3) is `cljw.error/with-context`: `(with-context -{:request-id … :trace-id …} body)` stacks a context map on a dynamic -var; when an error event is emitted (EDN render in -`src/app/error_render.zig::formatErrorEdn`) the merged map's keys become -top-level event fields. - -A Step-0.6 re-lay (survey `private/notes/phase14-14.13-binding-survey.md` -§9) found the handover's "narrow prerequisite = just the `binding` -special form" **wrong on a load-bearing point**: - -- cw v1 has **ZERO production dynamic vars**. The dynamic-binding runtime - machinery is fully built (`env.zig`: `BindingFrame`, threadlocal - `current_frame`, `pushFrame`/`popFrame`/`findBinding`, `Var.deref` - consulting the chain when `Var.flags.dynamic`), and both backends honor - `DefNode.is_dynamic → Var.flags.dynamic` — but `analyzeDef` NEVER sets - `is_dynamic` from surface syntax, and `^:dynamic` reader metadata is - unimplemented (`(def ^:dynamic *x* 1)` → "def expects 1 or 2 args, got - 3"). Surface `^:keyword` metadata = **D-075** (a ~500-LOC Phase-7+ - feature) and is **F-004-entangled** (per-value metadata NaN-box slot is - "未確定, decided at Phase 5 entry"). -- So `with-context` needs cw v1's **first dynamic var**, and the natural - `(def ^:dynamic *error-context* {})` will not parse. -- `binding` is also unimplemented (absent from the analyzer special-form - set; `pushFrame`/`popFrame` have zero production call sites). -- Read-side constraint: the `binding` frame is popped (Zig `defer - popFrame`) as an error unwinds, so by render time the frame is GONE — - the context must be snapshotted **at raise time** while the frame is - live. And `catalog.zig::raise(comptime code, location, args)` has **no - `rt`/`env` access** (verified) — it only populates the threadlocal - `Info`. So an Env-scoped slot reachable from raise is infeasible - without rippling raise's signature across ~100 call sites. - -## Decision - -### D1: `binding` is a real special form (`BindingNode`), not a macro-transform - -Diverges from JVM (a `defmacro` over `push-thread-bindings`/try/finally) -AND cw v0 (`transformBinding` macro). cw v1's analyzer special-form set + -dual-backend Node contract make a dedicated `binding_node` the -finished-form-clean shape: no dependency on a `var`/`hash-map`/`try` -macro chain, no per-call hash-map Value allocation, dynamic-var -resolution at analysis time. (ROADMAP A2 "features via the analyzer -surface" + F-002.) - -- **Analyzer** (`bindings.zig::analyzeBinding`): even-position symbols - resolve to their existing `*Var` (same qualified/alias/current-ns path - as `analyzeSymbol`), NOT a lexical slot; odd-position init-exprs analyze - in the OUTER scope (JVM parallel-eval); body analyzes in the same scope. -- **TreeWalk** (`evalBinding`): stack `BindingFrame`, eval inits in outer - `locals`, validate each target `flags.dynamic` (else raise the new - catalog Code), `pushFrame` + `defer popFrame()` (Zig defer = JVM - finally), eval body. -- **VM** (real arm, no VM-DEFER — per F-002 + the handover's explicit - "analyzer + tree_walk + vm" scope): two opcodes - `op_push_binding_frame` / `op_pop_binding_frame`, compiled in the - `try { push; body } finally { pop }` shape (the binding's own cleanup - handler always pops the frame before an exception escapes its extent, - reusing the existing `op_push_handler`/`op_pop_handler` unwind — so no - threadlocal dangle on uncaught throw, and no new unwind path). The VM - shares `env.zig`'s threadlocal `current_frame` (single-threaded per - session), keeping `Var.deref` backend-agnostic. -- **Validation**: new catalog Code `binding_target_not_dynamic` ("Can't - dynamically bind non-dynamic var: ns/name", JVM-faithful text per - `error_catalog_only.md`), raised at eval/push time (JVM-faithful site). - -### D2: cw v1's first dynamic var = Zig-registered `cljw.error/*error-context*` - -`*error-context*` is interned at bootstrap into the `cljw.error` ns with -`flags.dynamic = true` set directly (no `MetadataMap` change, no reader -metadata, no F-004 value-metadata touch), root `{}`. System dynamic vars -being Zig-registered is the finished form (JVM's `*out*`/`*ns*` are -bootstrap-created clojure.core vars); this is NOT a stopgap. **User-facing -`^:dynamic`** (so users mark their OWN dynamic vars) stays deferred to -D-075 — with-context does not need it, and when D-075 lands NOTHING about -with-context changes (it was already `binding` over a real dynamic var). - -### D3: read-side = global `*Var` slot + raise-time snapshot into `Info.context` + render via `printValue` - -- A `?*const Var` slot in `runtime/error/` (next to the other error - threadlocals), set when `*error-context*` is registered. -- `setErrorFmt`/`raise` snapshots `slot.?.deref()` into a new - `Info.context: ?Value` field (default null) **at raise time** (frame - live). Non-map / empty-map → leave null. -- `formatErrorEdn` (Layer 3) iterates `info.context`'s map entries and - emits each `key value` pair as a top-level EDN field, rendering via the - **canonical `runtime/print.zig::printValue`** (exists; Layer 0) — NOT a - second hand-rolled printer. -- The slot is a **process-global**, not Env-scoped: raise has no env - access (an Env-scoped slot would ripple raise's signature across ~100 - sites). For the single-Env CLI path (the only path that renders EDN - error events — nREPL returns errors to the client, bypassing - `renderError`) this is correct. The multi-Env-nREPL-EDN-error case is - recorded as **D-142**. -- Field is **specific `Info.context`**, not a generic `Info.data` - ex-info payload: ex-info `Info`-integration is not scoped here and a - generic field would over-fit a not-yet-landed shape (Reservation-as-bias - per F-002). It can generalize when ex-info `Info`-payload lands. - -### D4: `with-context` = thin `.clj` macro over `binding` - -New ns `cljw.error` at `src/lang/clj/cljw/error.clj` (new -`src/lang/clj/cljw/` dir) + an `@embedFile` row in `bootstrap.zig` FILES -after `core.clj` (needs `merge`/`defmacro`/`binding`). The `*error-context*` -Var is Zig-registered into the same ns before/at load. - -```clojure -(in-ns 'cljw.error) -(defmacro with-context [ctx-map & body] - `(binding [*error-context* (merge *error-context* ~ctx-map)] ~@body)) -``` - -`merge` against the current `*error-context*` gives the stacking -(nested with-context accumulate). Reached as `(require [cljw.error :refer -[with-context]])` (F-009: `cljw.*` is the cljw-original ns family). - -## Alternatives considered - -(Devil's-advocate subagent, fresh context, F-NNN envelope — output -verbatim. The DA assumed raise could reach `rt.env`; the main loop -verified it CANNOT, which is why D3 takes the process-global slot rather -than the DA-recommended Env-scoped slot.) - -> **F-NNN clearance for all three:** none violates an F-NNN. All avoid -> touching NaN-box layout / per-value metadata / `with-meta`/IObj -> (F-004), keep impl namespace-neutral with `cljw.error` as a thin -> Clojure-ns surface (F-009), and ship a working read-side (no forbidden -> no-op). The `binding` special form lands in all three. The decision is -> purely the var-creation + snapshot wiring. -> -> **Alt 1 — Smallest-diff: Zig-registered var, raise snapshots into -> `Info.context: ?Value`.** Error subsystem stores the `*Var` in a -> global/threadlocal slot; `raise` reads `if (ctx_var) |v| info.context -> = v.deref()`. `info.zig` already imports `Value` (layering clean). -> `formatErrorEdn` walks the map → top-level EDN keys. *Better:* smallest -> moving surface; the Var-pointer-in-error-subsystem slot dissolves the -> Layer-3↔Var coupling. *Breaks:* (1) a global `*Var` slot is -> process-wide mutable state — a second Env (nREPL) races on it; should -> be threadlocal/Env-scoped. (2) a `deref()`+field-set on EVERY raise -> (incl. ~100 internal raises with no context) is an always-on cost. (3) -> the EDN map-walk still needs a Value→EDN printer; `error_render.zig` -> has none — real work the proposal hand-waves. -> -> **Alt 2 — Finished-form-clean: generic `Info.data: ?Value` + -> Env-scoped dynamic-var registry; render via the canonical Value EDN -> printer.** Same Zig-registered var, but (a) an `Env.system_vars` -> registry (home for every future Zig-owned dynamic var `*out*`/`*ns*`/ -> `*print-length*`) reached via `rt.env` so multi-session is correct by -> construction; (b) `Info.data` as a generic ex-info-style payload, not a -> context-specific field; render through the canonical Value→EDN printer -> cw needs anyway. *Better:* this is where the finished form lives — Alt -> 1's error-only slot gets unwound the moment `*out*` arrives -> (Reservation-as-bias); `Info.data` matches cw's ex_info direction; -> forcing the canonical printer kills the duplicate-printer smell; -> multi-session correctness falls out. *Breaks:* substantially larger; -> the system-var registry is a structural-plan touch that F-003 says to -> imagine-and-defer to the owning Phase (10/15 owns `*out*`); routing -> through the canonical printer may pull its scheduling forward; -> `Info.data` risks over-fitting a guessed ex-info shape (premature -> reservation) — UNLESS the registry is scoped to just `error_context` -> today (one field), barely larger than Alt 1 but multi-session-correct. -> -> **Alt 3 — Wildcard: no `Info` field — render-time deref through an -> error-subsystem callback.** Keep `Info` a pure scalar; install a -> `?*const fn() ?Value` provider; to beat frame-already-popped, capture -> the live frame pointer at raise and deref lazily at render. *Better:* -> most decoupled; `Info` stays Value-free; generalizes to "any subsystem -> contributes EDN fields". *Breaks:* capturing the stack-allocated -> frame pointer for later deref is a use-after-return; to be safe you -> must snapshot the VALUE at raise anyway — at which point the callback -> buys nothing over a field and adds a hop + the same global-slot race. -> Highest complexity, lowest payoff. -> -> **Recommendation (non-binding):** Alt 2 scoped down to one field — -> Zig-register `*error-context*`, reach it from raise via an Env-scoped -> slot (NOT process-global), snapshot `deref()` at raise into a generic -> `Info.data: ?Value`, render via the canonical Value→EDN printer. Do NOT -> build the full system-var registry now (F-003 defer — one -> `Env.system_vars.error_context: ?*Var` field is enough; debt-row the -> generalization). The two clean wins independent of the field's name: -> Env-scoped slot (kills the multi-session race) + canonical printer -> (kills the duplicate printer). - -## Selection rationale - -Took Alt 1's **var-creation mechanism** (Zig-registered system var) + the -DA's **canonical-printer** clean win (`printValue` exists — feasible), -and **rejected** the DA's Env-scoped-slot recommendation on a fact the DA -could not see: `catalog.zig::raise` has **no `rt`/`env` parameter**, so an -Env-scoped slot is unreachable from the snapshot site without a -~100-site signature ripple — disproportionate, and the EDN-error path is -single-Env (CLI) in practice (nREPL bypasses `renderError`). The -multi-Env race the DA flagged is therefore latent, not manifest; recorded -as **D-142** for the nREPL-EDN-error owner. - -Rejected the DA's generic `Info.data` for a **specific `Info.context`**: -ex-info `Info`-payload is not scoped here, and a generic field would be a -Reservation-as-bias bet on a not-yet-landed shape (F-002). Rejected the -full `Env.system_vars` registry per the DA's own F-003 caveat — but since -the slot can't be Env-scoped anyway (raise has no env), the registry -question is moot for this cycle. - -This shape is finished-form-clean for the with-context FEATURE: when -D-075 lands user `^:dynamic`, with-context is unchanged (it was already -`binding` over a real dynamic var; `*error-context*` stays Zig-registered -like `*out*`). A primitive push/pop with-context (no dynamic var) would -be unwound by that finished form = smallest-diff bias, so it was rejected. - -## Consequences - -- cw v1 gains its first dynamic var + the general `binding` special form; - `binding` is dogfooded by with-context. -- The VM gains a real `binding` arm (not VM-DEFER) — the first - control-construct beyond `try` to share the handler-unwind machinery. -- `Info` gains an optional `context` Value field; `formatErrorEdn` gains - a context-merge loop (only the `edn` format carries structured fields; - `text` is unaffected). -- Process-global `*error-context*` `*Var` slot: correct for CLI, latent - multi-Env race recorded as D-142. -- User-defined `^:dynamic` stays deferred to D-075; `binding` only works - on system (Zig-registered) dynamic vars until then. - -## Affected files - -- `src/eval/node.zig` (+`BindingNode`, union arm) -- `src/eval/analyzer/analyzer.zig` (special-form set + dispatch) -- `src/eval/analyzer/bindings.zig` (`analyzeBinding`) -- `src/eval/backend/tree_walk.zig` (`evalBinding` + node switch) -- `src/eval/backend/vm/opcode.zig` (+2 opcodes) -- `src/eval/backend/vm/compiler.zig` (`compileBinding`) -- `src/eval/backend/vm.zig` (dispatch arms + binding-frame stack) -- `src/runtime/error/catalog.zig` (+`binding_target_not_dynamic`) -- `src/runtime/error/info.zig` (+`Info.context` + `context_provider` hook + raise-time snapshot) -- `src/runtime/error/context.zig` (new — the `*Var` slot + `current`/`clear` + `register`) -- `src/runtime/env.zig` (+`on_deinit_hook` so the slot can't dangle past its Env) -- `src/app/error_render.zig` (`formatErrorEdn` context merge via `printValue`) -- `src/lang/clj/cljw/error.clj` (new — `with-context`) -- `src/lang/bootstrap.zig` (embed row + `error_context.register` in setupCore) -- `src/lang/diff_test.zig` (+`binding` case) -- `test/e2e/phase14_binding.sh` + `test/e2e/phase14_with_context.sh` (new) -- `.dev/debt.md` (D-142 multi-Env nREPL slot; D-144 user-throw EDN context) - -## Revision history - -- 2026-05-29 created: row 14.13 (3) re-lay after the Step-0.6 - prerequisite-gap finding (no production dynamic var; `^:dynamic` - reader metadata = D-075, F-004-gated). Devil's-advocate fork landed. -- 2026-05-29 amendment 1 (implementation): D3 originally claimed "Zig - unit tests ... never set the slot." **False** — `src/app/builder.zig` - test blocks call `setupCore` → `register`, so a test Env that - registered left the process-global slot pointing at a freed Var, - UAF-ing the next `setErrorFmt` (caught by the `info.zig` setErrorFmt - unit test). Fix: `register` arms `Env.on_deinit_hook = clear` - (env.zig) — every `Env.deinit` drops the slot before freeing its - Vars. The slot + provider live in a new `runtime/error/context.zig` - (not `info.zig`) so `info.zig` stays free of an `env.zig` import (the - provider is a plain fn-pointer hook). Read-side verified for the - catalog-error path (`(/ 1 0)` carries `:request-id`); user - `(throw ex-info)` does NOT (it bypasses `setErrorFmt`) — recorded as - D-144. -- 2026-05-29 amendment 2 (D-144 discharge): user-throw structured EDN - rendering. See § Amendment 2 below. - -## Amendment 2 (2026-05-29) — user-throw structured EDN rendering (D-144) - -### Context - -D3 left a read-side gap: a user `(throw v)` raises `error.ThrownValue` -(payload in `dispatch.last_thrown_exception`), NOT a catalog `raise`, so -`setErrorFmt` never runs and the threadlocal `Info` stays null. The -renderer's null-Info fallback emitted a degraded event -(`{:cljw/error true :file … :kind :unknown :message "ThrownValue"}`) with -**no `*error-context*` fields** — so `(with-context {…} (throw (ex-info …)))` -dropped its context while `(with-context {…} (/ 1 0))` (a catalog error) -carried it. D3 explicitly foresaw this: *"Field is specific `Info.context`, -not a generic `Info.data` ex-info payload … It can generalize when ex-info -`Info`-payload lands."* This amendment is that landing. - -### D5: model user-throw as `Info.origin == .thrown`, not a new `Kind` - -- **`Info.origin: Origin = .catalog`** (`enum { catalog, thrown }`) + - **`Info.data: ?Value = null`** (the thrown ex-info's ex-data, distinct - from the ambient `context`). A new `Info.kindLabel()` is the single - category-label source both renderers consult: `@tagName(kind)` for - `.catalog`, `"exception"` for `.thrown`. -- **Why not a `Kind.exception` variant** (the original draft): `info.zig` - documents a 1:1 `Kind` ⇄ `ClojureWasmError` mapping, but a user throw - raises `error.ThrownValue` (not a `ClojureWasmError`). A new `Kind` - would have no honest error tag — forcing a vestigial - `ClojureWasmError.Exception` or an `unreachable`/lie arm in the - exhaustive `kindToError`. Modelling user-throw as a distinct *origin* - keeps `Kind` the honest catalog-category enum it was designed as, and - matches reality: a user can `(throw 42)` — there is no catalog `Kind` - for "the integer 42". -- **Throw-time context snapshot**: `dispatch.last_thrown_context: ?Value`, - set in `evalThrow` (TreeWalk) + `op_throw` (VM) via - `info.snapshotContext()` while the `binding` frame is live, cleared with - `last_thrown_exception` on catch. Same constraint the catalog path - solves inside `setErrorFmt` (the frame is popped by `defer popFrame` - during unwind, before render). -- **Render-time synthesis**: `error_render.zig::buildThrownInfo` builds a - `{ origin = .thrown, message = ex-message | printValue, data = ex-data | - null, context = last_thrown_context }` Info from - `dispatch.last_thrown_exception`; both text + EDN route through it (one - printer, per D3). `formatErrorEdn` emits `:kind :exception`, a - `:data ` field (when present), and the merged context. -- **Exit code unchanged**: `renderAndExit` reads `peekLastError()` (null - for throws) → exit 1, the existing user-error code. The synthetic Info - is render-only and never round-trips through `kindToError`. -- **`class_name` deferred**: a per-exception class label - (`"ExceptionInfo"`) was considered and deferred — cw v1 has no class - hierarchy yet (ex_info.zig comment), so a class field would be a - Reservation-as-bias bet on a not-yet-landed multi-class-catch future - (F-002). When multi-class catch lands (Phase 5+), that cycle adds it. - -### Alternatives considered (amendment 2) - -(Devil's-advocate subagent, fresh context, F-NNN envelope — output -verbatim. The DA's leading finding — the `Kind` ⇄ `ClojureWasmError` 1:1 -break — drove the selection of the `origin` discriminator over the -draft's `Kind.exception`.) - -> **F-NNN clearance summary (all three):** None requires violating an -> F-NNN. All keep the snapshot-at-throw-time discipline (forced by the -> `defer popFrame()` constraint — `evalBinding` pops during unwind, -> `formatErrorEdn` runs at the CLI boundary after the frame is gone, so -> render-time deref is structurally impossible). All keep the impl -> namespace-neutral (the snapshot helper and the printer live in -> `runtime/`, never in a `java/`/`cljw/` surface — F-009 + R4 hold). -> None touches NaN-box layout or per-value metadata (F-004). The -> decision space is purely: how user-throw exceptions get represented in -> the `Info`/`Kind` model, where the throw-time context snapshot is -> owned, and whether `Info.data` generalizes now. -> -> **Leading finding — the `Kind` ⇄ `ClojureWasmError` 1:1 invariant is -> the real load-bearing decision, and the draft's `Kind.exception` breaks -> it.** `info.zig` documents a strict 1:1 mapping between every `Kind` -> and a `ClojureWasmError` tag; `kindToError` is exhaustive returning a -> non-optional `ClojureWasmError`. A user throw raises `error.ThrownValue` -> (not a `ClojureWasmError`). So a new `Kind.exception` has no honest -> counterpart — forcing a vestigial `ClojureWasmError.Exception` or an -> `unreachable`/lie arm. Both are smells the project would unwind. Not an -> F-NNN violation (F-NNN is silent on the `Kind` model) but the single -> most important thing to get right; resolvable internally, no halt. -> -> **Alt 1 — Smallest-diff: reuse an existing `Kind`, synthesize `Info` at -> render time, no new enum variant.** Add only `last_thrown_context`; -> build a synthetic Info reusing `Kind.value_error` ("a thrown Value is -> evaluation producing a raised value"); special-case `:data` off -> `thrown.tag() == .ex_info` directly in `formatErrorEdn`. *Better:* zero -> change to the `Kind` enum / `ClojureWasmError` / `kindToError` — the -> 1:1 invariant stays pristine; no premature `Info.data`. Smallest -> surface that fixes the observable bug; text path gets message + context -> free. *Breaks:* `:kind :value_error` for a user throw is a category lie -> to a log consumer (conflates user exceptions with analyzer value -> errors); the `:data` special-case scatters ex-info knowledge into the -> printer; `(throw 42)` gets arbitrary `:value_error`. *F-NNN:* clean; -> the `value_error` reuse is an F-002 cleanliness compromise, not a -> block. -> -> **Alt 2 — Finished-form-clean: model user-throw OUTSIDE the -> `Kind`⇄`ClojureWasmError` coupling via an `Info` origin discriminator.** -> Add `Origin = enum { catalog, thrown }` + `origin` field (+ optional -> `class_name` string + `data`); `Kind`/`ClojureWasmError` untouched. EDN -> `:kind` becomes a function of `origin`: `@tagName(kind)` for `.catalog`, -> the ex-info class / `:exception` for `.thrown`. Throw-time snapshot + -> Info-build via a neutral helper shared by both backends. *Better:* the -> finished form — `Kind` stays the honest catalog-category type; no -> vestigial error, no `unreachable`; `class_name` is truthful and -> forward-compatible with multi-class catch; the text path renders via an -> `originLabel` helper fixing text + EDN in one place (D3's "one -> printer", extended). *Breaks:* larger — touches `formatErrorWithContext` -> (hard-codes `@tagName(info.kind)`), `formatError`, `kindToExitCode`; -> `Info` grows fields. Unlike `Kind.exception`, none of it lies. *F-NNN:* -> clean; neutral helper honors F-009; zone-clean. -> -> **Alt 3 — Wildcard: populate a full `Info` AT THROW TIME inside -> `evalThrow`/`op_throw`, not at render time.** Have the throw site call a -> `setErrorFmt`-equivalent (message, data, context-while-live, -> `origin = .thrown`) into `last_error`, then return `error.ThrownValue`; -> the render path's null-Info fallback then rarely fires. *Better:* -> eliminates render-time reconstruction; `error_render.zig` stays a pure -> Info reader; snapshot-at-throw-time satisfied by construction; unifies -> the two error families at the Info layer. *Breaks:* the dangerous one — -> a `(try (throw …) (catch …))` that recovers leaves a populated -> `last_error` that MUST be cleared at every catch site (two backends, -> easy to miss → silent-wrong-output), and it makes throw-time a second -> writer of `last_error` (the `error_catalog_only.md` rule reserves that -> for the catalog — needs a carve-out). Re-raise/nested-try interaction -> is fiddly; always-on formatting cost on caught-in-loop throws. *F-NNN:* -> clean on F-004/F-009/zones, but collides with the `error_catalog_only` -> rule (a project rule, not F-NNN); stale-Info-after-catch is the real -> reason it is the wildcard. -> -> **Recommendation (non-binding): Alt 2**, with the throw-time context -> snapshot owned by `dispatch.last_thrown_context` and the Info-synthesis -> at render time (avoiding Alt 3's stale-Info hazard by keeping -> `last_error` catalog-owned). It is the only alternative that does not -> lie about the `Kind`⇄`ClojureWasmError` 1:1 invariant; the draft and -> Alt 1 are smallest-diff biases against this, and preferring them on -> "the amendment is only ~150-250 lines" grounds is the Cycle-budget -> defer smell. `Info.data` should land, but as the scoped ex-data sibling -> of `context` (the concrete field this feature needs), not a speculative -> generic payload. Snapshot lives in `dispatch` (sibling to -> `last_thrown_exception`); the label indirection (`kindLabel`) fixes text -> + EDN together. *Where it could be wrong:* if Phase 5+ folds user -> exceptions and catalog errors into one unified taxonomy, Alt 3's -> "everything is an Info by the boundary" might be truer — judged -> unlikely (`ClojureWasmError` is host-failure-bound, structurally -> distinct from user-thrown arbitrary Values). - -### Selection rationale (amendment 2) - -Took **Alt 2**, the DA's recommendation, per F-002 (finished-form wins — -the larger diff is correct because the draft's `Kind.exception` is the -shape a finished-form owner would unwind). Declined `class_name` for this -cycle (Reservation-as-bias on multi-class catch — deferred to the Phase -5+ owner), keeping the amendment to the `origin` + `data` + `kindLabel` -core. Kept the synthesis render-side (not Alt 3's throw-time `last_error` -write) to preserve the `error_catalog_only` invariant and dodge the -stale-Info-after-catch hazard. Snapshot owned by -`dispatch.last_thrown_context` (sibling to `last_thrown_exception`); set + -cleared together so it is valid iff a thrown exception is pending. - -### Affected files (amendment 2) - -- `src/runtime/error/info.zig` (+`Origin`, +`Info.origin`/`Info.data`, - +`Info.kindLabel`, +`snapshotContext`; `formatError` uses `kindLabel`) -- `src/runtime/dispatch.zig` (+`last_thrown_context` threadlocal) -- `src/eval/backend/tree_walk.zig` (`evalThrow` snapshots context; - `evalTry` clears it on catch) -- `src/eval/backend/vm.zig` (`op_throw` snapshots; handler clears) -- `src/runtime/error/print.zig` (`formatErrorWithContext` uses `kindLabel`) -- `src/app/error_render.zig` (`buildThrownInfo` + thrown branch in - `renderError`; `formatErrorEdn` uses `kindLabel` + emits `:data`) -- `docs/spec/error_format.md` (`:exception` kind + `:data` key) -- `test/e2e/phase14_user_throw.sh` (new) + `test/run_all.sh` (register) -- `.dev/debt.md` (D-144 discharged) - -### Consequences (amendment 2) - -- The EDN error schema gains `:kind :exception` (user throws) + an - optional `:data` key (ex-data). `:data` was already listed as an - anticipated forward-compatible key; `:exception` is a new `:kind` - value. v0.1.0 is HELD/un-tagged, so no released contract is broken; - the spec doc is amended to list both. -- `(with-context {…} (throw (ex-info …)))` now carries context into the - EDN event — the with-context read-side is complete for both catalog - errors and user throws. -- D-142 (multi-Env nREPL slot race) is unchanged — the throw path reuses - the same process-global provider; the latent race is the same. diff --git a/.dev/decisions/0056_aot_bootstrap.md b/.dev/decisions/0056_aot_bootstrap.md deleted file mode 100644 index 35351f29c..000000000 --- a/.dev/decisions/0056_aot_bootstrap.md +++ /dev/null @@ -1,304 +0,0 @@ -# 0056 — AOT-bootstrap: build-time bytecode envelope for the embedded `.clj` bootstrap (edge per-instance cold-start) - -**Status**: Accepted (Devil's-advocate fork landed 2026-05-30; adopted Alt-2 sharpened framing) -**Date**: 2026-05-30 -**Author**: Shota Kudo (drafted with Claude autonomous loop) -**Tags**: aot-bootstrap, edge, startup, bytecode-envelope, cache_gen, F-002, F-004, F-006, F-009, F-010, ADR-0034-D4, ADR-0036, D-131, D-103, D-139 - -## Context - -`src/lang/bootstrap.zig::loadCore` (`:127-165`) re-runs the full -read → analyze → macroexpand → eval pipeline over **12 `@embedFile`'d -`.clj` sources** (`:53-66`: core / string / set / walk / zip / edn / -data.json / data.csv / tools.cli / pprint / test / cljw.error) on -**every** process / Wasm-instance startup. `setupCore` (`:174-183`) is -shared by three call sites (F-009): the REPL/CLI runner, -`builder.buildFile` (build-time), and `builder.tryRunEmbedded` -(built-app startup, `builder.zig:148`). So even a `cljw build`-produced -binary — whose *user* payload is precompiled bytecode — still re-evals -the entire bootstrap at startup. ADR-0034 §A1-D3 + **D-131** record this -as a tracked deferred gap; ADR-0034 §D4 already planned a "bootstrap -cache (bytecode) … core lib build-time bytecode化, 12ms → 3-4ms" via a -`cache_gen.zig` build stage (cw v0 evidence). This ADR executes that -deferred block with the v1-fitted shape. - -**Why now, and why it is an edge task, not a native-CLI task.** The -mission (ROADMAP §1) is "cold start ≤ 10 ms, Edge execution (Cloudflare -Workers / Fastly / Fermyon Spin / Wasm Component Model)". Native CLI -cold-start is **already met** — `bench/quick_baseline.txt` -`cold_start_us = 6560` (budget 12000) WITH `loadCore` running every time -— because a native `cljw -e` is dominated by OS process spawn + dyld + -page-cache warm of the ~6.8 MB binary; the bootstrap eval is a few ms on -top (D-140 notes the native self-exe re-read isn't worth optimizing). -**On the edge/Wasm target the calculus inverts**: a Wasm Component -instantiated per request has no OS spawn and no dyld — the runtime is -already resident — so the per-instance bootstrap eval (parse+analyze+eval -of 12 `.clj` files into a fresh env) is *pure repeated overhead*. At the -per-request instance churn the edge mission targets, eliminating that -eval is the whole game. This is an **edge-readiness** task; the user -directed it now (2026-05-30) ahead of the default coverage sequence. - -**Two facts that shape the decision** (Step-0 survey -`private/notes/phaseA26-d-aot-bootstrap-survey.md`): - -1. v1's `serialize.zig` **already serializes `fn_val` by contents** - (ADR-0034 am2, `:252-289`): `slot_base` + per-method - `arity`/`has_rest` + recursive `serializeChunk` of the method's - bytecode. The on-disk form is **index/tag-based** (decoder-stable - `ValueTag`, ADR-0034 D11), never raw NaN-box bits — so restore - reconstructs Values via live constructors and **no pointer relocation - is needed** (F-004 / F-006 are untouched). The `cljw build` - user-payload path (`builder.buildEnvelope` + `serialize.serializeEnvelope` - + `EnvelopeIterator`) is already shipped and differentially verified. -2. The one real constraint: `loadCore` runs under the comptime-routed - **default backend = `tree_walk`** (`driver.zig:32-45`, `build.zig:35-37`, - ADR-0036), so bootstrap fns become TreeWalk **AST closures** - (`bytecode == null`) which the serializer cannot round-trip - (`serialize.zig:281-288` needs `m.bytecode`). This is the v0 - `vmRecompileAll` problem (`cache.zig:119-183`) — bootstrap must be - **VM-compiled** before it can be snapshotted. - -## Decision - -Implement **D2 (AOT bytecode bundle) composed with D3 (lazy bootstrap)**, -in the **Devil's-advocate Alt-2 sharpened framing**: route the eager -bootstrap, lazy-`require`, and `cljw build` through **one -`runEnvelope(rt, env, bytes)` primitive** over the single shipped -envelope format — no parallel "bootstrap cache" code path. F-009-cleanest -(the bytecode-def-interning impl lives once; `setupCore` and the -`require` resolver are callers). Reusing the shipped build path, not -porting v0's bespoke ~3000-line env-snapshot serializer. - -**Two finished-form obligations absorbed now (not deferred), per the DA:** - -- **VM-dispatch hybrid at the runner (the one real knot).** The - production runner installs the `tree_walk` vtable (ADR-0036), but - AOT-restored core fns are bytecode-only (`body == null`) and must - dispatch on the VM. The clean answer is the **per-method - `bytecode`/`body` invariant that already exists** on `FunctionMethod` - (`tree_walk.zig:107-123`): a fn with `bytecode != null` runs on the VM - dispatch path, one with only `body` runs TreeWalk — so a - tree_walk-default runner can host AOT core fns *and* TreeWalk-eval - fresh user REPL/file forms. Cycle 1 must wire this call path (and - confirm the VM dispatch is available — not comptime-excluded — in a - `tree_walk` build). -- **Discharge D-139 (param-name fidelity) in the cycle that ships AOT - core.** Deserialized fns currently drop param-name strings; once core - is AOT'd, that regresses the default REPL's stack-trace labels for - core fns. Serialize param names so core error frames keep their labels. - -1. **Build-time bootstrap compilation (`cache_gen` analog).** A build - stage VM-compiles the embedded bootstrap `.clj` sources to a bytecode - **envelope** (one `BytecodeChunk` per top-level form, exactly as - `builder.buildEnvelope` does for user programs) and embeds the blob. - The stage runs the compile under `-Dbackend=vm` so the produced - `fn_val`s carry `bytecode` (the `vmRecompileAll` equivalent), even - though the runtime's default backend stays `tree_walk` — sound because - the two backends are contract-equivalent (ADR-0036) and the runtime - already dispatches AOT bytecode fns regardless of its own default - (the `cljw build` user-payload path proves this). -2. **Startup restore.** A `setupCore` variant deserializes + VM-runs the - embedded envelope (each `op_def` interns its Var) **instead of** - `loadCore`'s parse+analyze+eval. Zig-side macros - (`macro_transforms.registerInto`) and primitives - (`primitive.registerAll`) register exactly as today — they are not in - the `.clj` files, so AOT'ing the `.clj` sources does not affect macro - availability for runtime REPL/eval. -3. **Lazy split (D3).** Only `clojure.core` (+ whatever it transitively - needs) is eager; the other libs deserialize + run on first `require` - (the embedded resolver path), mirroring v0's hybrid (§1.4). This - serves the edge small-handler case (don't pay for pprint/csv/json on a - one-liner) and is orthogonal to / composable with D2. -4. **Versioning.** The embedded blob carries the `serialize.zig` wire - version AND the peephole rule-set identity (D-103) so a stale cache is - rejected, not mis-run. - -`cljw build` artifacts gain the same benefit by construction: once the -runtime restores its bootstrap from the embedded bytecode, `tryRunEmbedded` -no longer needs the full `loadCore` (advancing D-131). - -### Incremental implementation plan (each a TDD cycle, gate-green) - -- **Cycle 0 — resolve the VM-dispatch knot (prerequisite).** Confirm - whether a bytecode-only `Function` can run in a `tree_walk`-default - build (is the VM dispatch comptime-excluded?). Wire the per-method - `bytecode`/`body` hybrid so the production runner dispatches a fn with - `bytecode != null` on the VM while TreeWalk-evaling user input. Smallest - proof: hand-build a bytecode fn (via `vm_compiler.compile`) and call it - from a tree_walk-default runner. Without this, AOT core cannot run. -- **Cycle 1 — mechanism proof on the biggest file + D-139.** AOT-compile - *only* `core.clj` to an embedded envelope at build; add the - `runEnvelope` primitive + a `setupCore` path that deserializes+VM-runs - it instead of parsing core.clj; **serialize param names (discharge - D-139)** so core error frames keep labels. Verify parity (every e2e + - diff green) + measure the parse/analyze skip. -- **Cycle 2 — the `cache_gen` build stage + all eager files.** Generalize - to the eager `FILES` subset; wire the 2-stage build (`build.zig`). -- **Cycle 3 — D3 lazy deferral** for the rare/heavy namespaces. -- **Cycle 4 — `tryRunEmbedded` uses the cache** (discharge D-131's - bootstrap-cache block) + edge/Wasm-instance cold-start measurement. - -Each cycle keeps `loadCore` as the fallback until its successor is -proven, per F-002 skeleton-then-rewrite (no silent semantic drop). - -## Alternatives considered - -Devil's-advocate fork (`private/notes/phaseA26-d-aot-bootstrap-da.md`, -fresh context, F-NNN envelope). **Leading finding: no F-NNN amendment is -required by any alternative** — the index/tag wire format (F-004) already -decouples on-disk from NaN-box; the only F-006 stressor (D4) is already -rejected. The DA also verified two facts that *strengthen* the decision: -(1) AOT'ing core does NOT break macros — `core.clj` has **zero** -`defmacro` (all of `let`/`when`/`->` are Zig-side `macro_transforms`); the -only bootstrap `defmacro` is `cljw.error/with-context` (+2 in test.clj), -and a `defmacro` lowers to `op_def` + `DEF_FLAG_MACRO` (`compiler.zig:569`) -stamped at run time (`vm.zig:172`), AOT-agnostic. (2) The build path -already runs dual-mode (`tryRunEmbedded` installs the VM vtable + runs -`setupCore` and deserialized chunks together). - -The three within-envelope alternatives (DA verbatim findings condensed; -full text in the DA note): - -- **Alt-1 — smallest-diff: D3-only lazy bootstrap, no AOT.** Split - `FILES` eager (`core` + `cljw.error`) vs lazy; stop force-loading the - lazy 10, let `require` pull them. *Better*: zero new serialization - surface, no VM-dispatch knot, no cache versioning, no stale-artifact - risk. *Breaks*: does NOT solve the stated problem — `clojure.core` (the - largest file) still re-evals on **every** edge instance (survey §3.D3). - Picking it for diff-size is "the Cycle-budget defer smell in its purest - form". **Verdict: reject as standalone; keep as the D3 layer D2+D3 - already includes.** -- **Alt-2 — finished-form-clean: unify bootstrap + lazy-require + - `cljw build` through one envelope-run primitive (this IS D2+D3, - sharpened).** *Better*: removes the last special case — one - `runEnvelope` helper, one place bytecode-defs intern (F-009-cleanest); - forces the VM-dispatch knot to be solved cleanly at the runner; single - artifact = single version field (D-103 solved once). *Breaks/cost*: the - VM-dispatch-in-tree_walk-runner knot is real engineering (the per-method - `bytecode`/`body` hybrid must be wired), and D-139 (param-name drop) now - hits **core** error frames so must be discharged in the same cycle. - **Verdict: the DA's pick — adopt D2+D3 in this framing.** (This ADR's - Decision.) -- **Alt-3 — wildcard: per-form analyzed-AST memo cache (not bytecode).** - Serialize the post-macroexpansion `Node` AST; restore + `evalForm` on - TreeWalk, skipping Reader+Analyzer. *Better*: no backend mismatch - (restored fns are TreeWalk closures, exactly what the default runner - expects — the VM-dispatch knot vanishes); param names survive (no - D-139). *Breaks*: the **wrong finished form** — needs a brand-new - bespoke serializer for the arena-lifetime, pointer-rich `Node` graph - (forks the serialization impl, violates F-009 + DIVERGENCE-1; harder to - make index/tag-stable than the already-flat bytecode), and becomes dead - weight the moment the VM is the hot-path backend (F-010's JIT). **Verdict: - reject — a clever local optimum the finished-form owner unwinds.** - -From the survey's design space (rejected before the DA pass): - -- **D1 — full env-snapshot (v0 style).** Serialize the whole - post-bootstrap env via a bespoke env-walker (mirrors v0 - `serializeEnvSnapshot`, ~3000 LOC) + deferred-var fixup lists + - static-var-cache re-pointing. **Rejected**: duplicates capability v1 - already has (`buildEnvelope` compiles a program to a chunk envelope); - porting v0's env serializer violates `no_copy_from_v1` and is a - larger, less-reused surface than treating bootstrap as "just another - program to AOT" (D2). Same `vmRecompileAll` prerequisite as D2 but more - new code. -- **D4 — heap image / unexec (GraalVM native-image style).** Dump the raw - GC heap at build, restore/mmap at startup. **Rejected**: the only - option that hits F-004/F-006 — v1's 44-bit *shifted* pointer - (`addr>>3`) is an absolute in-process address; a restored heap at a - different base (ASLR; Wasm fresh linear memory per instance) needs - every NaN-box payload + internal pointer relocated against a - non-moving mark-sweep heap. Disproportionate when D2 removes the - parse/analyze cost without touching the heap representation. - -## Consequences - -- **Edge/Wasm per-instance cold-start** drops to deserialize + VM-run - defs (no Reader / Analyzer / macroexpander on the hot path) — the - mission win. Native CLI is roughly unchanged (already OS-spawn-bound). -- **New build stage** (`cache_gen` analog) + a build-vs-runtime backend - split (build compiles under `-Dbackend=vm`; runtime default stays - `tree_walk`). The gate must build the cache stage. -- **`loadCore` remains** as the source-of-truth fallback during the - incremental rollout and for the non-AOT dev path; the AOT path is the - edge/release path. -- **D-139 fidelity gap** rides along: deserialized fns currently drop - param-name strings (AOT error frames lose param labels) — a fidelity - gap, not a semantics drop; tracked separately. -- **Cache invalidation** is a new failure surface — a bytecode blob - compiled by a different `serialize.zig` version / peephole rule set - (D-103) must be rejected at load, not mis-run. -- A `defmacro` *inside* a `.clj` file (if any exist) needs its macro - registered at runtime for user code to expand it — Cycle 1 must audit - the `.clj` files for `defmacro` and confirm the AOT path preserves - runtime macro availability (expected: macros are Zig-side, so none — - but verify, do not assume). - -## Affected files - -- `src/lang/bootstrap.zig` — `setupCore` variant (restore-from-envelope); - eager/lazy `FILES` split. -- `build.zig` — `cache_gen` build stage (compile bootstrap under - `-Dbackend=vm`, embed the blob). -- `src/app/builder.zig` — `buildEnvelope` reused for bootstrap; - `tryRunEmbedded` uses the cache (Cycle 4). -- `src/eval/bytecode/serialize.zig` — version/peephole identity in the - blob header (D-103); possibly an env-level envelope wrapper. -- `test/e2e/` — AOT-bootstrap parity + cold-start e2e. -- `.dev/debt.md` — advance/discharge D-131; note D-139. - -## Revision history - -- **2026-05-30 (initial issue + DA fork)**: Status Proposed → Accepted. - Devil's-advocate fork (`private/notes/phaseA26-d-aot-bootstrap-da.md`) - produced Alt-1 (D3-only, rejected — leaves core re-evaling), - Alt-2 (unify via one `runEnvelope` primitive, **adopted as the - sharpened framing**), Alt-3 (AST-memo cache, rejected — bespoke AST - serializer forks the impl, F-009 + DIVERGENCE-1). DA verified the - macro-availability worry is void (zero `defmacro` in core.clj) and that - the dual-mode build path already exists. Two finished-form obligations - pulled forward into the rollout (VM-dispatch hybrid → new Cycle 0; - D-139 param-name discharge → Cycle 1) per F-002 (surgery over the - smaller diff), not deferred. -- **2026-05-30 (Cycle 1 landed)**: `driver.runEnvelope` extracted (Alt-2 - one-primitive) + `bootstrap.setupCorePrefix` + an AOT round-trip test - (build core.clj → envelope → restore into a fresh env → run a - core.clj-only fn). **Implementation finding (not in the survey/ADR - plan):** `serialize.zig`'s var_ref deserialize resolved eagerly and - could not handle core.clj's **self-recursive / forward var_refs** (e.g. - `(def map (fn … (map …)))` — the constant pool is read before the - chunk's `op_def` runs). This is v0's `DeferredVarRef` problem - (survey §1.2), documented as out-of-v0.1.0-scope at the old - `serialize.zig:376`. Fixed by **forward-declaring** on a resolve-miss - (`env.intern` get-or-create with a nil placeholder root; the later - `op_def` binds the same var) — Clojure-correct, consistent with cljw's - no-unbound-sentinel nil-root model, and it also fixes a latent - recursive-fn gap in the `cljw build` embedded-run. D-139 (param-name - fidelity) deferred to Cycle 2, where AOT core becomes the runtime - default and error frames would otherwise regress (Cycle 1 is the - in-process proof; `loadCore` remains the production path). -- **2026-05-30 (Cycle 2 landed — AOT core LIVE everywhere)**: 2a — a - `cache_gen` host tool (build.zig) VM-compiles core.clj → a bytecode - envelope embedded as `bootstrap_cache.data` (23829 B). 2b — `runner. - runSource` restores core via `setupCoreAot` → `driver.runEnvelope` - instead of `loadCore`; all ~110 e2e flow through it, gate 125/125 ⇒ - AOT core is **production-faithful**. 2c — extracted `loadCoreAot` and - routed repl / nrepl / `tryRunEmbedded` (built apps) to it, so **all - four startup paths** now restore core from bytecode (advancing D-131). - `loadCore` (source, all 12) stays for the build tool + `setupCore`. - cold_start 6260µs (native is OS-spawn-bound; the edge/Wasm per-instance - win — no spawn — is the point). - **D-139 (param-name fidelity) DEFERRED, for memory-ownership care, not - cycle budget**: normal fns borrow `params` from the analyzer arena; - deserialized fns would need owned param strings; `freeFunction` cannot - distinguish them, so freeing deserialized params risks corrupting - arena-borrowed ones (and not-freeing leaks under the unit-test - allocator). A clean discharge needs a params-ownership marker (or an - always-arena-borrow + arena-based tests) — a focused-context design, - not a session-tail rush (F-002: don't introduce a memory bug). The - AOT-core error-frame regression is param LABELS only (gate green; no - test depends on them). Tracked: D-139. - **Cycle 3 (D3 lazy non-core files) remaining**: the 11 non-core `.clj` - files still load eagerly from source in `loadCoreAot`; lazy-`require` - needs a per-file eager-vs-lazy dependency analysis (survey §3.D3 - "hidden inter-namespace eager deps"). diff --git a/.dev/decisions/0057_sorted_collections_red_black.md b/.dev/decisions/0057_sorted_collections_red_black.md deleted file mode 100644 index 1dc823cc1..000000000 --- a/.dev/decisions/0057_sorted_collections_red_black.md +++ /dev/null @@ -1,130 +0,0 @@ -# 0057 — Sorted collections: persistent left-leaning red-black tree - -**Status**: Accepted (Devil's-advocate fork landed 2026-05-30; adopted the DA's finished-form Alt-2 over the survey's smallest-diff flat-array) -**Date**: 2026-05-30 -**Author**: Shota Kudo (drafted with Claude autonomous loop) -**Tags**: sorted-map, sorted-set, red-black, persistent, F-002, F-004, F-006, A6, ADR-0053, gap-sweep - -## Context - -`sorted-map` / `sorted-set` / `sorted-map-by` / `sorted-set-by` / `subseq` -/ `rsubseq` / `rseq` / `sorted?` are unimplemented (corpus gap sweep, -`.dev/core_coverage_gaps.md`). Tags `sorted_map = 46` / `sorted_set = 47` -are day-1 reserved; no interior-node struct was reserved. `compare` / -`runtime/compare.zig::valueCompare` (ADR-0053, D-137) already provides a -general 3-way comparator with a defined cross-type order, so there is no -prerequisite. The open question is the **representation**. - -The Step-0 survey (`private/notes/phaseA26-sorted-coll-survey.md`) -recommended a **flat sorted array** `[k0 v0 k1 v1 …]` + binary search, on -the grounds that a persistent red-black tree "buys speed, not a cleaner -shape." A mandatory Step-0.6 Devil's-advocate fork pressure-tested that -recommendation against F-002 and found it to be smallest-diff bias. - -## Decision - -Sorted collections use a **persistent left-leaning red-black (LLRB) tree** -(Sedgewick variant — one node struct + `color` field + 2-3 rotation -helpers, far less code than JVM `PersistentTreeMap`'s 8-subclass -hierarchy), in a new `runtime/collection/sorted.zig`: - -- `SortedMap { header, count: u32, comparator: Value, root: Value, meta: Value }`; - interior `RbNode { header, color: u8, _pad, key, val, left, right }` - under a new `rb_node` tag (claims `reserved_c11 = 43`). -- `SortedSet` **wraps a SortedMap** (element→element), mirroring the - existing `PersistentHashSet`→map precedent. -- Default comparator = `valueCompare` (ADR-0053); `-by` variants carry a - custom comparator Value invoked via `rt.vtable.callFn` / - `invokeCallable` (the path `sort` already uses). Duplicate keys collapse - by comparator-equality (compare → 0). -- `assoc`/`dissoc` are O(log n) **path-copy with structural sharing** — - the same persistent contract the HAMT, CHAMP, and vector trie honor. -- Dispatch: `sorted.zig` owns both tags; the primitive chokepoints - (`collection.zig`, `sequence.zig`, `lookup.zig` IFn, `print.zig`) gain - `.sorted_map`/`.sorted_set` arms. map.zig is NOT extended (already >1000 - LOC, A6). `rb_node` gets a `traceGc` (left/right/key/val) like - `traceHamtMapNode`. - -Cycle split: **A** structs + GC + `valueCompare`-ordered build / get / -contains / count / seq + `sorted-map`/`sorted-set` + `sorted?`; **B** -insert-rebalance assoc/dissoc-delete (the hard LLRB delete) + dup-collapse -+ `-by`; **C** in-order `seqFrom` leaf + `.clj` `subseq`/`rsubseq` + `rseq` -+ flip `reversible?` (currently hardcoded to `vector?`). - -## Alternatives considered - -The Devil's-advocate subagent (fresh context, briefed with the F-NNN -envelope) produced these, reflected verbatim: - -**Alt 1 — Smallest-diff: set-wraps-map + entries as a raw `[*]Value` heap -slice (one new `sorted_entries` tag).** Better: most cache-friendly -contiguous scan; reuses set-over-map. Breaks: adds a GC tag AND a *novel* -variable-length realloc allocation idiom the codebase has never used -(every existing struct is fixed-size inline); O(n) copy on every assoc. -F-NNN: respects all, but is "the textbook Smallest-diff-bias path — -minimizes feature-LOC while adding a GC tag + novel idiom, the trade F-002 -says is wrong." - -**Alt 2 — Finished-form-clean: persistent LLRB red-black tree (`rb_node`), -structural sharing on every assoc/dissoc.** Better: the ONLY alternative -honoring the persistent structural-sharing contract every other cljw -collection already honors; `(into (sorted-map) coll)` — the idiomatic -construction path — is O(n log n) vs the flat array's **O(n²)**; -`subseq`/`seqFrom` is a natural in-order tree walk matching the JVM -reference. Costs: LLRB delete is the hardest single algorithm in the -collection lib (no v0 precedent); claims the last free C tag; many small -node allocs (same *kind* as HAMT/vector, not a new idiom). F-NNN: respects -all (F-004 spend a reserved tag = memo per F-002; F-006 trace; A6 ~500-700 -LOC, split set out if needed). **The alternative F-002 points at.** - -**Alt 3 — Wildcard: inline-array small + promote-to-RB at a threshold -(ArrayMap→HAMT analogue).** Better: cache-friendly + no-new-idiom for the -common small case AND O(log n) for large; mirrors cljw's existing hash-map -promotion. Costs: highest total complexity (both paths + the -promote/demote boundary); cannot ship without first writing Alt 2's hard -delete; small-map win over plain RB is marginal (log 16 = 4). File as a -post-RB debt row if profiling later shows node-alloc overhead. - -**DA verdict (paraphrased):** the flat array does NOT hold under F-002. -"Finished form = simple form" conflates simplicity-of-algorithm with -cleanliness-of-shape; for a *persistent* collection the clean shape is the -structural-sharing one, and an O(n)-copy-on-assoc collection sitting next -to four O(log n) path-copy collections is the *inconsistent* shape. The -O(n²) idiomatic-construction path is the main path quadratically degraded, -not a tail case. "RB delete is hard / ~200 LOC" is the named -Cycle-budget-defer smell, which F-002 forbids as a basis for choosing. -Sub-choices: if flat were chosen, entries-as-PersistentVector beats a raw -slice (no new tag); sorted-set should wrap-a-sorted-map (the established -set-over-map shape), diverging from the survey's independent-array lean. -**Recommendation: Alt 2 (LLRB, set-wraps-map).** - -The main loop adopts Alt 2: no F-NNN blocks it, so per F-002 the -finished-form-clean shape must be taken; preferring the flat array on -"less hard code now" grounds is precisely the Cycle-budget-defer smell. - -## Consequences - -- O(log n) persistent assoc/dissoc/get; structural sharing consistent with - the other persistent collections. `(into (sorted-map) big-seq)` is - O(n log n). -- One new heap tag (`rb_node`) + trace fn; `reserved_c11` consumed. -- The LLRB delete (cycle B) is the highest-correctness-risk piece; cover it - with a build-N-delete-half unit test (like the HAMT canary) + the diff - oracle. Cycle A ships first (build/read) with delete raising an explicit - error until cycle B — no silent wrong answer. -- `rseq` + `reversible?` get fixed for vectors as a side effect (cycle C). - -## Affected files - -- New: `src/runtime/collection/sorted.zig`, `test/e2e/phase14_sorted*.sh`. -- Edit: `src/runtime/value/heap_tag.zig` (`reserved_c11` → `rb_node`), - `src/runtime/runtime.zig` (register trace), `lang/primitive/` - collection.zig / sequence.zig + `runtime/collection/lookup.zig` + - `runtime/print.zig` (dispatch arms), `src/lang/clj/clojure/core.clj` - (subseq/rsubseq/rseq/sorted?/reversible?), `src/lang/diff_test.zig`. - -## Revision history - -- 2026-05-30: drafted + accepted. DA fork overturned the survey's - flat-array recommendation on F-002 grounds (smallest-diff bias / - Cycle-budget-defer). LLRB chosen as the finished form. diff --git a/.dev/decisions/0058_eval_runtime_primitive_macro_table_reach.md b/.dev/decisions/0058_eval_runtime_primitive_macro_table_reach.md deleted file mode 100644 index 48964a698..000000000 --- a/.dev/decisions/0058_eval_runtime_primitive_macro_table_reach.md +++ /dev/null @@ -1,214 +0,0 @@ -# ADR-0058 — `eval` runtime primitive reaches the macro table via a typed Layer-1 driver verb - -- **Status**: Accepted -- **Date**: 2026-05-31 -- **Phase**: Phase 14 (post-v0.1.0 coverage) / cluster A26 -- **Supersedes**: — -- **Superseded by**: — - -## Context - -`clojure.core/eval` was absent (debt D-162). The wire-up is otherwise -ready — `analyzer.valueToForm` (Value→Form), `analyzer.analyze` -(Form→Node, needs a `*const macro_dispatch.Table`), and -`driver.evalForm` (Node→Value) — BUT `analyze` needs the -macro-expansion table, which until now was threaded only at -setup/bootstrap/analysis time (stack-owned per app entry point, -passed by pointer) and is **not reachable from a runtime primitive**. -A primitive runs at eval-time with `rt` + `env` in scope but no -`macro_table`. - -This is structural-defect class #3 in -`.dev/lessons/structural_defect_hunting.md` — "a runtime primitive -needs a resource that only exists at setup/analysis time" — and it -**will recur** (`load-string`, `eval` inside a macro, a future -`load`/`read+eval` REPL surface all need the same reach). So the fix -must set a clean, reusable precedent, not a one-off. - -Two confirmed facts shape the design: - -1. **The macro table is a process constant** — always exactly - `lang/macro_transforms.registerInto`'s output, no per-session - state. User `defmacro` macros do **not** live in it; they live on - the Var (`flags.macro_` + deref'd root, `macro_dispatch.zig:103/114`) - and are reachable via `env` alone. So a runtime `eval` re-entering - analysis needs the table **only** to expand built-in Zig macros - (`and`/`or`/`when`/`->`/…); user macros come free via `env`. -2. **The `eval` primitive is Layer 2** (`src/lang/primitive/`), which - imports `eval/` (Layer 1) freely. So it can call - `valueToForm`/`analyze`/`evalForm` directly — **no Layer-0 vtable - indirection is needed**. The single missing piece is the table. - -Zone constraint (`.claude/rules/zone_deps.md`): Layer 0 (`runtime/`) -must NOT import Layer 1 (`eval/`), so a typed -`macro_table: *const macro_dispatch.Table` field on `Runtime` is a -zone violation — it must be stored type-erased (`?*const anyopaque`), -the same concession `dispatch.VTable.evalChunk` already makes for a -Layer-1 `BytecodeChunk` pointer (`dispatch.zig:84-89`). - -Zig constraint: top-level fn pointers cannot close over locals, so -"install an `evalForm` fn on the vtable that closes over the table" -is impossible — any such fn would still have to read the table from -`rt` (anyopaque) or a module-level `var` (forbidden globals per -ROADMAP §13). The vtable option therefore does **not** escape the -anyopaque; it only relocates where it is read. - -## Decision - -Adopt **Alt 2 (finished-form-clean)** from the Devil's-advocate fork: - -1. **`Runtime.macro_table: ?*const anyopaque = null`** (Layer 0, - type-erased). Documented as a **borrow** of the Layer-1 - `macro_dispatch.Table`, set once in the universal funnel - `bootstrap.setupCorePrefix`, valid for the entry point's - setup-frame lifetime. (Adding a field is explicitly not an - ADR-level change per `runtime.zig` doc; the ADR is for the - structural-precedent choice.) - -2. **`driver.evalValue(rt, env, locals, arena, value, loc)`** — a new - **typed** Layer-1 verb that packages the whole runtime-eval - pipeline: cast `rt.macro_table` back to `*const macro_dispatch.Table` - once (raise an internal error if null = eval before bootstrap), - then `valueToForm → analyze(…, table) → evalForm`. The - `@ptrCast`/`@alignCast` lives at exactly **one** site, behind a - typed function. - -3. **`eval` primitive** (Layer 2, `lang/primitive/core.zig`): allocate - a per-call arena off `rt.gpa` (the analysed Node/Form live there; - the result Value is GC-allocated so it survives `arena.deinit()`), - a fresh `[MAX_LOCALS]Value` locals buffer, and call - `driver.evalValue` with **zero casts**. Registered as `eval` in - `clojure.core`. - -The reusable precedent for class #3 is therefore: *"a runtime -primitive that needs a setup-time analysis resource calls a typed -Layer-1 driver verb; the resource is threaded Layer-0→Layer-1 once, -type-erased only at the single Runtime field and cast back behind the -typed verb."* The next such primitive reuses `driver.evalValue` (or a -sibling verb) and never re-derives the cast. - -**Rejected**: Alt 1 (raw `?*const anyopaque` field cast in *each* -primitive) — it is the smallest diff but compounds: every recurrence -of class #3 adds another untyped field + bespoke cast (Smallest-diff -bias smell, and the diff-size advantage over Alt 2 is the -Cycle-budget-defer trap). **Rejected**: Alt 3 (process-singleton -table, anyopaque-free) — the only fully-anyopaque-free shape, but it -requires a forbidden module-level `var` (ROADMAP §13) + a new -process-global allocator lifetime, colliding with the deliberate -per-Runtime isolation `keywords`/`symbols`/`vtable` all enshrine. - -**Known fidelity limit (carried as follow-up):** `valueToForm` -(`analyzer.zig:797-820`) handles -{nil,bool,int,float,symbol,keyword,string,list,vector,array_map,hash_set} -and raises `macro_return_not_data` for everything else. So -`(eval \a)` / `(eval 1N)` / `(eval 1/2)` / `(eval 1.5M)` / -`(eval )` raise rather than self-evaluate. The D-162 -motivating case `(eval (read-string "(+ 1 2)"))` is all-handled-tags -and works. Extending `valueToForm` to cover char/bignum/hash_map is a -separate unit (it also governs macro returns) — recorded as a -follow-up, not blocking D-162. - -## Alternatives considered - -The following is the Devil's-advocate subagent's verbatim output -(fresh context, briefed with the F-NNN envelope + zone + Zig -constraints): - -> # Devil's-advocate review — Debt D-162: how `clojure.core/eval` reaches the macro-expansion table -> -> ## Verification of the brief's claims (cited) -> -> - **`evalChunk` anyopaque precedent**: confirmed (`dispatch.zig:84-89`). `chunk: *const anyopaque`, cast back at the VM end. This is the project's *one* existing Layer-0-carries-Layer-1-pointer concession, and its doc comment explicitly frames it as a zone-barrier workaround. -> - **`Runtime` "adding a field is OK"**: confirmed (`runtime.zig:50`). `require_resolver: RequireResolverFn` (runtime.zig:43) is the precedent for *a Layer-1 fn pointer typed in Layer 0, installed by bootstrap* — but note it is **typed** (`*const fn(*Runtime, []const u8) ...!?[]const u8`), NOT anyopaque, because its signature names only Layer-0 types. The macro_table cannot do this (its type lives in Layer 1). -> - **macro_table is a process constant**: confirmed (`macro_dispatch.zig:61-86`) — always `macro_transforms.registerInto`'s output; user `defmacro` lives on the Var (macro_dispatch.zig:103/114), reachable via `env`. -> - **eval primitive is Layer 2**: confirmed — it will live in `src/lang/primitive/`, which already imports `eval/` freely (bootstrap.zig itself, a Layer-2 file, imports analyzer + driver). So `valueToForm`/`analyzeForm`/`evalForm` need **no** vtable. The *only* gap is the table. -> - **The universal funnel exists**: confirmed and stronger than the brief stated. **Every** entry point (builder:107, repl:70, runner:66, nrepl:102, cache_gen:43) reaches the table through `setupCorePrefix` (bootstrap.zig:204) → `macro_transforms.registerInto`. There is exactly one install site to instrument. -> - **Lifetime caveat (important, the brief understated it)**: the table is `var macro_table = ...Table.init(gpa); defer macro_table.deinit();` *on the entry-point stack frame* (builder:105-106, repl:68-69, etc.). It outlives every eval in `setupCore`/the REPL loop because those run inside the same frame. **Any `?*const anyopaque` stored on `rt` is a borrow** — correct only while that stack frame is live. For the REPL/runner this is the whole session, so it is fine; but it is a borrowed pointer, not owned, and that constraint must be documented at the field. -> -> --- -> -> ## Alternative 1 — Smallest-diff: `Runtime.macro_table: ?*const anyopaque` (≈ the main loop's Option-2) -> -> Store the table pointer on `Runtime`, set once inside `setupCorePrefix` (the single funnel), cast back in the Layer-2 eval primitive. -> -> **Better than nothing / what it does well**: one field + one assignment line in the one funnel + one cast site. No new file, no vtable slot, no signature churn. Per-Runtime (not a global), so parallel tests are isolated. Mirrors `evalChunk`'s sanctioned concession exactly. -> -> - **(a) Avoids anyopaque?** No — it *is* the anyopaque. One new cast site (`@ptrCast`/`@alignCast` back to `*const macro_dispatch.Table` in the Layer-2 primitive). -> - **(b) Multi-Runtime safe?** Yes. Field on `Runtime`, not a `pub var`. Each test's Runtime carries its own pointer. -> - **(c) Lifetime correct?** Yes *but borrowed*. The field aliases the entry-point's stack-owned table. Correct as long as no code path stores a Runtime past its setup frame's lifetime. Today nothing does. The risk is future: if anyone ever returns a `Runtime` by value or heap-promotes it, the borrow dangles silently (anyopaque erases the type, so the compiler will not catch a stale pointer). -> - **(d) Reusable precedent for structural-class #3 ("primitive needs a setup-time resource")?** **This is its weakness.** It establishes "when a primitive needs resource X, bolt `?*const anyopaque` onto Runtime and cast in the primitive." That is a precedent that *scales badly* — class #3 WILL recur (the debt row itself flags it), and each recurrence adds another untyped anyopaque field + another bespoke cast site. Five recurrences = five anyopaque fields on Runtime with five separate "cast back, trust me" sites. That is the **Smallest-diff bias smell** compounding: each instance is cheap, the aggregate is a Runtime struct full of type-erased borrows. -> -> --- -> -> ## Alternative 2 — Finished-form-clean: a typed Layer-1 "analysis context" handle, vtable-thin, owned by the Runtime -> -> Promote the macro_table from a stack-local to a **Runtime-owned, Layer-1-typed resource reached through one narrow accessor**, so the anyopaque never appears. Concretely: introduce a small Layer-1 struct (call it `eval/analysis_env.zig`'s `AnalysisEnv`, or reuse the existing `Table` directly) that the Runtime *owns by value or owns the pointer to*, and expose it to Layer 2 via a **typed function on the Layer-1 driver** — `driver.evalValue(rt, env, locals, arena, value)` — that internally does `valueToForm → analyzeForm(…, rt's table) → evalForm`. The eval primitive (Layer 2) calls one typed Layer-1 entry point and never touches the table at all. -> -> The table reaches the driver because the driver is Layer 1 and *can* name `macro_dispatch.Table`. The Runtime stores the table as `?*const anyopaque` **only internally** OR — cleaner — the driver reads it from a Layer-1-side store. The decisive move: **the public contract Layer 2 sees is `driver.evalValue(...)`, fully typed, zero casts.** The anyopaque (if any remains) is confined to a single Layer-0↔Layer-1 plumbing line inside `runtime/` + its cast inside `driver`, both authored once, never copied by future primitives. -> -> **Better than Option-2**: -> - Layer 2 (where new primitives keep getting added) gets a **typed, reusable verb**: `driver.evalValue`. The next primitive that needs eval-at-runtime (`load-string`, `eval` inside a macro, `clojure.core/load`, `read+eval` REPL) calls the same typed function. No copy-paste cast. -> - The anyopaque, if it survives at all, lives at **one** Layer-0↔1 boundary with a typed cast immediately behind it (same shape as `evalChunk`, but consumed through a typed wrapper rather than re-cast at every call site). -> - Sets structural-class #3's precedent **correctly**: "a primitive that needs a setup-time analysis resource calls a typed Layer-1 driver verb; the resource is threaded Layer-0→Layer-1 once." That precedent *shrinks* future diffs instead of growing the anyopaque field count. -> -> - **(a) Avoids anyopaque?** Partially-to-fully. It does NOT eliminate the Layer-0↔1 type-erasure if the table is stored on `Runtime` (zone rule forces `?*const anyopaque` there). But it **relocates and confines** it: the cast happens once inside `driver`, and Layer 2 never sees it. Compared to Alt 1, the anyopaque stops being a per-primitive concern. If the table is instead held in a Layer-1 module-side store keyed by Runtime, the Runtime field disappears entirely — but a Layer-1 keyed store risks a `pub var` (forbidden) unless keyed off something the Runtime already carries, so the clean version still stores `?*const anyopaque` on Runtime and casts in driver. **Honest verdict: it relocates the anyopaque to one confined typed-wrapper site; it does not delete it.** -> - **(b) Multi-Runtime safe?** Yes — still a per-Runtime field/handle, no global. -> - **(c) Lifetime correct?** Same borrow caveat as Alt 1, but the typed wrapper is the natural place to document "borrowed for the Runtime's setup-frame lifetime," and a future heap-promotion of Runtime would force a decision at the single typed boundary rather than silently dangling N anyopaque fields. -> - **(d) Reusable precedent?** **Best of the three.** This is the finished-form answer to class #3: typed Layer-1 verbs, anyopaque confined behind them. F-002 (finished-form wins) points here. -> -> **What it breaks/risks**: larger diff (new typed `driver.evalValue` entry + threading the table into the driver's reach). If the table ends up stored on `Runtime` anyway, a critic could say "you added a typed wrapper around the same anyopaque field — Alt 1 with extra steps." The rebuttal: the wrapper is the reusable surface; the field's erasure is an implementation detail Layer 2 never re-derives. Also: `evalForm` already exists at driver.zig:54 and takes a `*const Node` (post-analysis) — `evalValue` must additionally hold the analyzer entry, so it couples driver to analyzer (already true — analyzer is Layer 1, driver is Layer 1, and bootstrap.zig:157 already calls `analyzeForm` then `evalForm` back-to-back; `evalValue` just packages that pair). -> -> --- -> -> ## Alternative 3 — Wildcard: make the macro_table a Layer-1 *process constant* reached by a pure comptime/once-init function, eliminating the field entirely -> -> The brief's own confirmed fact is the lever: **the macro_table is a process constant** — always exactly `macro_transforms.registerInto`'s output, no per-session state, no per-Runtime variation. If that is *truly* invariant, the table does not need to live on the Runtime at all. Instead: expose a Layer-1 (or Layer-2) function `macro_dispatch.canonicalTable()` that lazily builds the one canonical table **once per process** behind a `std.once`-style guard, returns `*const Table`, and is called by both bootstrap's `setupCorePrefix` AND the eval primitive. No field on Runtime, no anyopaque, no borrow, typed end-to-end. -> -> **Better than Option-2**: -> - **(a) Avoids anyopaque?** **Yes, completely.** The table is named with its real Layer-1 type at every site. eval (Layer 2) calls `macro_dispatch.canonicalTable()` and gets `*const Table`. Bootstrap does the same. Zero casts, zero erasure. The zone rule is satisfied because Layer 2 may import Layer 1, and Layer 0 (`runtime/`) is never involved — the table leaves the Runtime entirely. -> - **(c) Lifetime correct?** **Best possible** — process-lifetime constant, outlives every Runtime by construction. The borrow problem vanishes. -> - **(d) Reusable precedent?** Strong *for genuinely process-constant resources*: "a setup-time resource that is a true process constant is a lazily-initialized Layer-1 singleton, not a Runtime field." That is a clean, named category. -> -> **What it breaks/risks — and this is fatal-ish**: -> - **(b) Multi-Runtime safe? Only conditionally.** A `std.once`/lazy-singleton needs *somewhere* to store the built table + the once-guard. In Zig 0.16 that storage is a **module-level `var`** — i.e. a `pub var` (or file-private `var`) — which **ROADMAP §13 forbids** and which the brief explicitly rules out. A file-private (non-`pub`) `var` is a softer violation but is still the exact shape (mutable global state) §13 targets, and it breaks the project's "per-Runtime, no shared mutable global" invariant that `keywords`/`symbols`/`vtable` were all deliberately designed around (runtime.zig:73-88 comments repeatedly justify per-Runtime-ness "so parallel tests don't collide"). Even though the *table content* is constant, the *once-guard* and *storage slot* are process-global mutable state, and the table's `alloc` field (macro_dispatch.zig:63) binds it to one allocator — which allocator? A process singleton cannot belong to a per-Runtime `gpa`. It would need the page allocator or a process arena, a new lifetime category. -> - **Leading finding (per brief instruction)**: *The only fully-anyopaque-free option (Alt 3) requires either a forbidden module-level `var` (§13) or a new process-global allocator lifetime.* It does **not** require an F-NNN amendment — F-004/F-006 don't speak to this — but it **does** require either a §13 carve-out (an ADR amendment, loop-ownable) or accepting a file-private mutable singleton. Because §13 is a hard ROADMAP rule and the per-Runtime isolation is load-bearing for parallel tests, I flag this as the reason Alt 3 cannot be taken as-is despite being the only anyopaque-free shape. **Do not halt** — record and move on. -> - A weaker non-singleton variant (build the table fresh each `eval` call via `registerInto` into an arena) avoids the global but pays a full table rebuild per eval call — correctness-fine, performance-poor, and it duplicates the "table is built once" assumption the codebase relies on. Acceptable as a *stopgap* but not finished-form. -> -> --- -> -> ## Recommendation (advisory, not binding) -> -> Rank: **Alt 2 (finished-form-clean) > Alt 1 (smallest-diff) > Alt 3 (wildcard)**. Alt 3 is the only shape that fully deletes the anyopaque, and the brief's "macro_table is a process constant" fact makes it tempting — but it collides with ROADMAP §13 (no module-level `var`) and the deliberate per-Runtime isolation that `keywords`/`symbols`/`vtable` all enshrine, so it cannot be taken without a §13 carve-out and a new process-global allocator lifetime; that cost is not justified for one debt row. Between Alt 1 and Alt 2, the deciding question per the project's own Cycle-budget-defer smell is *not* diff size: Alt 1 is the Smallest-diff-bias trap because structural-class #3 ("a primitive needs a setup-time resource") **will recur** (the debt row says so), and Alt 1's precedent makes each recurrence add another untyped anyopaque field + bespoke cast, whereas Alt 2 gives Layer 2 a typed, reusable `driver.evalValue(...)` verb and confines the unavoidable Layer-0↔1 erasure to one wrapper authored once. Per F-002 (finished-form cleanliness wins, cycle size is not a constraint), take Alt 2: a typed Layer-1 `driver.evalValue` (`value → valueToForm → analyzeForm(rt's table) → evalForm`) consumed by the eval primitive with zero casts, with the table reaching the driver via a single documented `?*const anyopaque` Runtime field set in the `setupCorePrefix` funnel (mirroring `evalChunk`'s sanctioned concession, but consumed through a typed verb rather than re-cast per primitive). Document the field as a borrow of the entry-point-owned table with the setup-frame lifetime caveat, so a future Runtime heap-promotion surfaces at one typed boundary rather than dangling silently. - -The main loop adopts the DA's recommendation (Alt 2) unchanged. - -## Consequences - -- **Positive**: `eval` works for data forms. A typed, reusable - Layer-1 verb (`driver.evalValue`) now exists for every future - runtime-eval primitive (`load-string`, REPL `read+eval`, …); the - Layer-0↔1 type erasure is confined to one Runtime field + one cast - behind the typed verb. User macros expand via `env` for free; - built-in macros via the borrowed canonical table. Sets the clean - precedent for structural-defect class #3. -- **Negative / watch**: `Runtime.macro_table` is a **borrow** of the - entry-point's stack-owned table — valid only while that frame is - live (the whole session for repl/runner/builder). A future - heap-promotion of `Runtime` would dangle it; the type erasure means - the compiler will not catch that. Mitigated by documenting the - borrow at the field + confining the cast to `driver.evalValue`. -- **Fidelity limit**: eval of `char`/`BigInt`/`Ratio`/`BigDecimal`/ - large `hash_map` literals raises `macro_return_not_data` until - `valueToForm` is extended (separate unit; governs macro returns - too). - -## Affected files - -- `src/runtime/runtime.zig` — add `macro_table: ?*const anyopaque` - field (borrow doc). -- `src/lang/bootstrap.zig` — `setupCorePrefix` sets `rt.macro_table` - after `registerInto`. -- `src/eval/driver.zig` — new typed `evalValue` verb. -- `src/lang/primitive/core.zig` — `eval` primitive + registration. -- `test/e2e/` — `(eval …)` surface tests. -- `.dev/debt.md` — D-162 discharged. diff --git a/.dev/decisions/0059_class_type_return_interned_type_descriptor.md b/.dev/decisions/0059_class_type_return_interned_type_descriptor.md deleted file mode 100644 index 6add5d009..000000000 --- a/.dev/decisions/0059_class_type_return_interned_type_descriptor.md +++ /dev/null @@ -1,268 +0,0 @@ -# ADR-0059 — `class` / `type` return an interned `.type_descriptor` Value - -- **Status**: Accepted -- **Date**: 2026-05-31 -- **Phase**: Phase 14 (post-v0.1.0 coverage) / cluster A26 -- **Supersedes**: — -- **Superseded by**: — - -## Context - -`(class x)` and `(type x)` were absent (`name_error`, verified 3x). -They are extremely common clojure.core surface (`(group-by class -coll)`, `(condp = (type x) …)`, multimethod dispatch on `class`), so -the quality loop (F-010) needs them. The representation is a -**load-bearing, user-observable decision** — what value `class` -returns governs its print form, its `=` behaviour, and whether it is -a valid map key — so it rides an ADR with a Devil's-advocate fork. - -cljw has **no JVM Class**. The substrate it does have: - -1. A `.type_descriptor` Value tag (28, already in the F-004 NaN-box - layout) wrapping a `*const TypeDescriptor` via a `TypeDescriptorRef` - heap object (`type_descriptor.zig:171`). -2. Native types reach a descriptor through `rt.nativeDescriptor(tag)` - (`runtime.zig:190`), which caches one descriptor per Tag and - populates a clean `fqcn` via `nativeFqcnFor` (`runtime.zig:214`: - `.integer → "Long"`, `.float → "Double"`, `.string → "String"`, - `.vector → "PersistentVector"`, `.nil → "nil"`, …). -3. `(rt/__native-type :integer)` already maps a tag-keyword to that - `.type_descriptor` Value; `defrecord`/`deftype` already bind their - name Var to a `.type_descriptor` Value; `instance?` / `extends?` / - `satisfies?` already all speak `TypeDescriptor` / `.td_ptr`. - -So a type representation **already exists and is already wired into -the type-system machinery**. The cw-v0 precedent went the other way: -`class` and `type` were the *same* function returning a **keyword** -(`:integer`), Class was dropped entirely, and `(class nil)` → `:nil`. -v0 could afford the keyword shape because v0 had no protocol/instance -dispatch to compose with; cw v2 does. A keyword-as-class cannot carry -a method table, cannot be told apart from a user keyword `:Long`, and -does not compose with `instance?`/`extends?`/`satisfies?`. That is the -load-bearing reason v2 diverges from v0 on all three points — not -"changed our minds". - -**The load-bearing sub-problem — `=` / hash / print.** Today a -`.type_descriptor` Value: - -- prints as the generic placeholder `#` - (`print.zig:208` `else => "#<{s}>"`). -- has **no** equality arm: `valueEqual` (`equal.zig:271-276`) and the - HAMT key path fall to bit-identity / `else => false`, and - `valueHash` (`equal.zig:217`) falls to `else => - hash.hashLong(@bitCast(@intFromEnum(v)))` (hashes the wrapper bits). -- is minted **fresh per call** by `makeTypeDescriptorRef` (a new - `TypeDescriptorRef` heap object each time), even though the - underlying `*TypeDescriptor` is interned (native: cached per-Tag; - user: registered once). - -Consequence: today `(= (class 5) (class 6))` is **false** and `(class -5)` is broken as a map key, because two fresh wrappers carry different -NaN-box bits. JVM returns the interned `java.lang.Long` Class -(identity == equality), so `(= (class 5) (class 6))` is true and -`(group-by class coll)` works. Shipping `class` with broken equality -would be a **representation-divergence defect** (structural-defect -class #4 in `structural_defect_hunting.md`) — not acceptable per -F-002. So the finished form must fix identity/equality, not just add -a wrapper. - -## Decision - -Adopt **Alt 2 (finished-form-clean)** from the Devil's-advocate fork. - -1. **`(class x)` returns the existing `.type_descriptor` Value** - (Candidate A — one representation for native + user types, - composing with the already-wired `instance?`/`extends?`/`satisfies?`/ - `__native-type` machinery). `(class nil)` → `nil` (JVM semantics). - typed_instance / reified_instance carry their own descriptor; every - other value consults `rt.nativeDescriptor(v.tag())`. New primitive - `rt/__class` (Layer 1, in `protocol.zig` next to `nativeType`); - `.clj` wrapper `(def class (fn* [x] (rt/__class x)))`. - -2. **`(type x)` = `(or (:type (meta x)) (class x))`** — a pure `.clj` - wrapper. This **un-collapses** v0's `class`==`type` so `type` - honours `:type` metadata (JVM semantics), the one place `type` - legitimately differs from `class`. - -3. **Interning is a correctness invariant, not an opportunistic - cache** (the DA's sharpening of A1). `makeTypeDescriptorRef` - becomes the sole minting site and **always returns one canonical - boxed Value per descriptor**: it caches the wrapper in a new - `ref_cache: ?Value = null` field on `TypeDescriptor`, returning the - cached Value on every subsequent call. Two `(class 5)` calls then - return the **bit-identical** Value, so identity equality holds - automatically in `valueEqual` **and** the HAMT `keyEqValue` **and** - `valueHash` — `.type_descriptor` joins the interned-singleton family - (keyword / symbol / primitives), and `equal.zig` is **not touched**. - `(= (class p) Point)` works because `defrecord`'s `Point` and - `(class p)` resolve to the same cached wrapper. The cached ref is - `rt.gc.infra`-allocated (process-lifetime, never GC-collected) and - the descriptor is per-Runtime, so the cache never dangles and needs - no GC trace edge; a one-line invariant comment at - `makeTypeDescriptorRef` records that bit-equality ⇔ same descriptor, - so a future refactor cannot quietly add a non-interned minting path. - -4. **Print** adds a `.type_descriptor` arm to `print.zig` printing - `td.fqcn` (simple name `Long`, not JVM FQCN `java.lang.Long`, per - the no-JVM-assumption rule). Per the DA, the **simple-name-vs-FQCN - choice is scoped as separable**: the arm prints whatever - `nativeFqcnFor` policy yields; revisiting FQCN is an independent - print-policy decision, not re-litigating class identity. - -**Rejected — Alt 1 (A2 equality+hash arms, keep fresh wrappers):** -correct and shippable, and its explicitness (a visible -`.type_descriptor => a.td_ptr == b.td_ptr` arm) is a genuine virtue, -but it permanently complicates `equal.zig`'s equality/hash contract -with a **3-way sync liability** (valueEqual + keyEqValue + valueHash -must stay mutually consistent forever) and **invalidates** the -"bit-identity is contract-consistent for every non-string key" -invariant comment (`equal.zig:213`) — when interning is available and -makes all of that unnecessary. Its only edge over Alt 2 (no cache -field → no GC-reachability question) dissolves once Alt 2 documents -the infra-lifetime reasoning. Per F-002 the smaller blast-radius in -`makeTypeDescriptorRef` is a bits-today convenience, not a -finished-form gain. - -**Rejected — Alt 3 (dedicated `.class` NaN-box tag):** redundant -representation. `.type_descriptor` already models "a type"; a `.class` -tag forces either deleting `.type_descriptor`'s user role (a much -larger surgery) or a cross-tag equality arm (worse than Alt 1). It -re-breaks the native+user unification that is the decision's core -virtue and spends a finite F-004 reserve slot on a speculative -need (reservation-as-bias). F-004's own challenge text predicts this -verdict. - -## Alternatives considered - -The following is the Devil's-advocate subagent's verbatim output -(fresh context, briefed with the F-NNN envelope + the -`equal.zig`/`print.zig`/`type_descriptor.zig` file:line facts + -the A1-vs-A2 sub-decision): - -> ## Devil's-advocate: Alternatives considered (ADR-0059) -> -> **F-NNN gate result up front:** None of the three alternatives below requires violating an F-NNN. The finished-form-clean option (Alt 2) sits entirely inside the F-004 layout (reuses tag 28, no new tag) and inside F-009's carve-out (it edits `type_descriptor.zig` / `equal.zig` / `print.zig`, all language-core, explicitly out of F-009 scope). So there is **no "leading entry" forced disclosure** of an F-NNN block — the clean shape is reachable. The interesting tension is entirely *within* the envelope: A1 (intern) vs A2 (equality arms), and simple-name vs FQCN print, neither of which any F-NNN decides. -> -> --- -> -> ### Alternative 1 — Smallest-diff: A2 equality+hash arms, keep fresh wrappers (the draft's explicitly-rejected sub-option), simple-name print -> -> Keep `makeTypeDescriptorRef` minting a fresh ref per call. Add three `.type_descriptor` arms — one in `valueEqual` (equal.zig:271, the same-tag switch), one in `keyEqValue`, one in `valueHash` (equal.zig:217) — each dereferencing `.td_ptr` and comparing/hashing the *underlying* `*const TypeDescriptor` pointer (which IS interned: native via `rt.native_descriptors[idx]` runtime.zig:181/206, user via the registry). Print arm prints `td.fqcn`. -> -> **What it does better than the draft (A1):** -> - Does not introduce a cache field (`ref_cache: ?Value`) onto `TypeDescriptor`, so it does not add a GC-reachability question ("is the cached wrapper Value traced, or does it dangle if GC moves/frees?"). A1's cache field is a *new* live Value reference held by a runtime-core struct; Alt 1 holds zero new Values, sidestepping that analysis entirely. -> - The equality is *semantically explicit*: a reader of `valueEqual` sees a `.type_descriptor => a.td_ptr == b.td_ptr` arm and immediately knows "two type descriptors are equal iff same underlying descriptor". A1's equality is *implicit* — it works only because two calls happen to return bit-identical wrappers, a fact that lives in `makeTypeDescriptorRef`'s memoization, three files away from `valueEqual`. The invariant is "spooky action at a distance". -> - Robust against any *future* code path that mints a non-interned wrapper. If Phase-N code ever constructs a `TypeDescriptorRef` by a path that bypasses the cache (e.g. a deserializer, a reify fast-path, a `with-meta` on a class), A1 silently breaks `=` again (two valid wrappers for one descriptor → unequal); Alt 1 stays correct because it compares the payload, not the box. -> -> **What it breaks / risks:** -> - The draft's stated objection is real and load-bearing: `valueHash`'s own docstring (equal.zig:208-216) warns `keyEqValue(a,b) ⇒ valueHash(a) == valueHash(b)` must hold, and Alt 1 now has **three** arms (valueEqual, keyEqValue, valueHash) that must stay mutually consistent forever. The hash arm must hash `@intFromPtr(td_ptr)` while both equality arms compare `td_ptr` — a future edit to one without the others reintroduces a HAMT corruption (key present but unfindable). This is exactly the maintenance hazard the draft cites. -> - It does *not* shrink the surface — it grows `equal.zig`, a runtime-core hot file, by three arms in two functions whose comment block (equal.zig:208-216) explicitly enumerates "the ONLY branch needing care is `.string`". Alt 1 makes that comment false: now `.type_descriptor` also needs care. The comment must be rewritten, and the "bit-identity is contract-consistent for every non-string key" invariant (equal.zig:213) is now violated by a second tag. -> - It is a smallest-diff-against-current-bits choice, not a smallest-diff-against-finished-form choice: it optimizes "don't touch `makeTypeDescriptorRef`" at the cost of "permanently complicate the equality/hash contract". -> -> **Explicit answers:** -> - (a) unify native+user types? **Yes** — both go through `.type_descriptor` / `.td_ptr`, same as the draft. -> - (b) `(= (class 5) (class 6))` true? **Yes** — both wrappers' `.td_ptr` point at the single interned `native_descriptors[@intFromEnum(.integer)]` descriptor, so the new `valueEqual` arm returns true. -> - (c) class-as-map-key / `(group-by class coll)` work? **Yes** — the `valueHash` arm hashes `td_ptr` and the `keyEqValue` arm compares `td_ptr`, consistently, so the HAMT finds the key. (Conditional on the three arms staying in sync — the standing risk.) -> - (d) `(class 5)` prints as? **`Long`** (simple name, `td.fqcn` via nativeFqcnFor). -> - (e) composes with instance?/extends?/satisfies?? **Yes** — those already speak `TypeDescriptor` / `.td_ptr`, unchanged. -> - (f) reusable-precedent quality: **Medium-low.** The "add an equality+hash arm per heap-payload tag" pattern is a real precedent (`.vector` already did it, equal.zig:225/233 vectorHash + keyEqValue vector arm), so it is not unprecedented. But each such arm is a standing 3-way-sync liability, and the draft is right that the codebase should prefer *interning so bit-identity just works* over *adding payload-comparison arms* wherever interning is feasible. Vector keys can't be interned (infinitely many); type descriptors *can* (finite, already interned at the descriptor level). So the vector precedent does not transfer — type descriptors are the case where A1 is available and A2 is the inferior copy. -> -> --- -> -> ### Alternative 2 — Finished-form-clean: A1 intern-the-wrapper (the draft) BUT make interning a structural invariant of `makeTypeDescriptorRef`, not an opportunistic cache; FQCN-faithful print deferred to a print-policy decision -> -> This is the draft's A1, sharpened. The draft frames A1 as "memoize one `TypeDescriptorRef` Value per `TypeDescriptor` (e.g. a `ref_cache: ?Value` field, filled on first `makeTypeDescriptorRef`)". The finished-form-clean version makes the *non-existence of a non-interned path* the invariant: `makeTypeDescriptorRef` becomes the sole minting site and **always** returns the cached wrapper (allocating + caching on first call, returning the cache thereafter), and `asTypeDescriptorRef` / any other consumer never constructs a fresh ref. The cache field is documented as "the canonical boxed identity of this descriptor", and a `comptime`/debug assertion or a one-line invariant comment records that two `.type_descriptor` Values are bit-equal iff same descriptor — so the equality/hash contract is upheld *by construction at the box layer*, and `equal.zig` is genuinely never touched (the draft's central claim). -> -> **What it does better than the draft-as-written:** -> - The draft says "cache it, e.g. a `ref_cache` field". The word "e.g." and the framing "memoize on first call" leave it as an *optimization* ("we cache to avoid re-allocating"). The finished-form framing is stronger: interning is a *correctness invariant*, not a perf optimization. This matters because an optimization can be "optimized away" by a later well-meaning refactor (someone adds a `makeFreshTypeDescriptorRef` fast-path for a reify hot loop and silently breaks `=`), whereas an invariant with an assertion catches that at the door. The draft's A1 is correct but under-specified on *why the bits stay identical forever*; Alt 2 closes that gap. -> - It keeps `equal.zig`'s "bit-identity is contract-consistent for every non-string key" invariant (equal.zig:213) **literally true** — type descriptors join nil/bool/int/char/keyword/symbol as "interned, so bit-identity = value-identity". This is the cleanest possible finished form for the equality contract: the comment block at equal.zig:208-216 needs *zero* edits, because `.type_descriptor` is now genuinely in the "bit-identity-compared, hashing raw bits is contract-consistent" bucket. That is a strictly better finished state than Alt 1, which *invalidates* that comment. -> - It generalizes: interning the box is the same shape the codebase *already chose* for keyword and symbol (interned → identity equality → no equal.zig arm). A reader who understands why `(= :a :a)` needs no keyword arm in valueEqual immediately understands why `(class 5)` needs no type_descriptor arm. One mental model, reused. This is the highest-precedent-quality outcome. -> -> **What it breaks / risks:** -> - The cache field adds a Value held by a `TypeDescriptor` (which lives on `rt.gc.infra`, process-lifetime per runtime.zig:194/196 + the type_descriptor.zig:167 docstring "process-lifetime"). Since both the descriptor and the ref are infra-allocated and process-lifetime, the cached Value never dangles — but this must be *verified and stated*, not assumed. The draft does not currently note the GC-reachability reasoning; Alt 2 requires the ADR to record "cache holds an infra-allocated, process-lifetime, never-collected ref, so no trace edge is needed." If cw's GC ever moves infra objects, this invariant needs revisiting (low risk — infra is explicitly the non-moving arena). -> - The print question is *orthogonal and under-decided*. The draft picks simple-name (`Long`) via existing `nativeFqcnFor`. The no-JVM-assumption rule (`no_jvm_specific_assumption.md`) says cljw is not a JVM reimplementation, which the draft reads as license to diverge to simple names. But that rule cuts *both* ways: it argues against *assuming JVM class hierarchy*, not specifically for *simple names over FQCN*. JVM Clojure's `(pr-str (class 5))` → `java.lang.Long`; cw printing `Long` is a *visible round-trip divergence* that `(group-by class coll)` users will see in their output. Alt 2's position: the print *representation* is a separable decision from the *identity/equality* decision, and bundling them into one ADR risks locking simple-name print prematurely. Recommend the ADR explicitly scope print to "prints `td.fqcn`; whether fqcn is `Long` or `java.lang.Long` is the existing `nativeFqcnFor` policy, revisitable independently" rather than ratifying simple-name as a class-identity decision. -> - Bigger diff than Alt 1? No — it is essentially the same diff as the draft's A1 (cache field + print arm), plus an invariant comment/assertion and a tightened `makeTypeDescriptorRef`. The "expansion" is documentation and a guard, not code volume. Per F-002, even if it *were* larger, that is not a reason to prefer Alt 1. -> -> **Explicit answers:** -> - (a) unify native+user types? **Yes** — identical to the draft; one `.type_descriptor` representation for both. -> - (b) `(= (class 5) (class 6))` true? **Yes** — both calls return the bit-identical cached wrapper for the interned integer descriptor; `valueEqual`'s identity fast-path (equal.zig:247, `@intFromEnum(a) == @intFromEnum(b)`) returns true with zero new code. -> - (c) class-as-map-key / `(group-by class coll)` work? **Yes** — bit-identity means `keyEqValue` hits identity and `valueHash`'s `else => hash.hashLong(@bitCast(@intFromEnum(v)))` (equal.zig:226) hashes the *same* bits for both, so the HAMT contract holds automatically. No new hash arm, no sync liability. -> - (d) `(class 5)` prints as? **`Long`** with the draft's choice — but Alt 2 flags this as a separable print-policy decision, recommending the ADR not bind it as identity-load-bearing. -> - (e) composes with instance?/extends?/satisfies?? **Yes** — unchanged; they speak `.td_ptr`. -> - (f) reusable-precedent quality: **High — the highest of the three.** "Intern the box so bit-identity = value-identity, and add no equal.zig arm" is the exact precedent already set by keyword/symbol/interned-primitives. It reuses one mental model, leaves the equality contract's invariant comment literally true, and makes the next interned-singleton type (future: namespace objects? interned class hierarchy nodes?) follow the same recipe. -> -> --- -> -> ### Alternative 3 — Wildcard: a dedicated `class` tag (new NaN-box tag in F-004's Group B reserve), with print/equality/hash native to that tag, `.type_descriptor` retained only as the *internal* descriptor pointer -> -> Mint a fresh `.class` Value tag (consuming one of F-004's ~10 Group-B reserved slots) whose payload is the interned `*const TypeDescriptor` directly (no intermediate `TypeDescriptorRef` heap object — the tag *is* the boxed class). `(class x)` returns a `.class`-tagged Value; `.type_descriptor` / `TypeDescriptorRef` stays an internal-only thing the protocol/instance machinery uses, never user-visible. The new tag gets its own print arm (`Long` or `java.lang.Long`), and equality/hash via interned-pointer identity (same bit-identity story as Alt 2, because the interned descriptor pointer is the payload). -> -> **What it does better than the draft:** -> - *Separation of "the user-facing Class value" from "the internal descriptor reference."* Today `.type_descriptor` is overloaded: it is simultaneously what `__native-type` returns internally, what `defrecord`'s `Point` resolves to, what `instance?` consumes, *and* (post-draft) what `(class x)` returns to user code. A dedicated `.class` tag would let the user-facing surface (print as a class, equal as a class) evolve independently of the internal descriptor-reference plumbing. If cw ever wants `(class x)` to carry user-visible class metadata (a `getName` method, a `(supers c)` hierarchy walk) that the *internal* descriptor reference should not carry, the tag split is the seam for it. -> - Eliminates the `TypeDescriptorRef` heap indirection for the class value entirely — the tag NaN-boxes the `*const TypeDescriptor` pointer directly, one fewer allocation and one fewer pointer-chase than even Alt 2's cached ref. (Whether this matters perf-wise is unmeasured and almost certainly negligible.) -> -> **What it breaks / risks:** -> - **This is redundant representation, and F-002 weighs against it.** F-004 explicitly invites the challenge: "minting a fresh `class` tag when `.type_descriptor` already models 'a type' should be justified against F-002 (would it be a cleaner finished form, or redundant representation?)." The honest answer: it is **redundant**. `.type_descriptor` *already* models "a type"; `defrecord`'s `Point`, `__native-type`, `instance?`, `extends?`, `satisfies?` already all speak it. A `.class` tag would mean `(= Point (class p))` requires either (i) `Point` also becomes `.class`-tagged (then `.type_descriptor` has no remaining user-facing role and should be deleted, a much bigger surgery) or (ii) a cross-tag equality arm `.class == .type_descriptor when same td_ptr` (reintroducing the exact equal.zig arm-maintenance hazard Alt 2 avoids, now *worse* because it is cross-tag). Both outcomes are strictly less clean than Alt 2's single-representation finished form. -> - The hypothetical motivation ("class value evolves independently of internal descriptor reference") is **speculative** — there is no current requirement for `(class x)` to carry anything the descriptor reference doesn't. Building the seam now is a reservation-as-bias smell (`.dev/principle.md`): consuming an F-004 reserve slot for a need that does not materialize. F-004's reserve is finite (~10); spending one here on speculation is the kind of "memo, not contract" the project spirit warns against. -> - It consciously *re-breaks the unification* the draft's whole thesis rests on. The draft's load-bearing virtue is "UNIFIES native + user types under one representation and composes with existing instance?/extends?/satisfies?". Alt 3 *splits* the representation, then has to *re-bridge* it. That is motion away from the finished form, not toward it. -> -> **Explicit answers:** -> - (a) unify native+user types? **Partially / awkwardly** — both native and user classes would be `.class`-tagged, so *as classes* they unify; but the class representation is now *split from* the descriptor representation that `instance?`/`defrecord` use, requiring a bridge. Net: less unified than the draft. -> - (b) `(= (class 5) (class 6))` true? **Yes** — both `.class` values box the same interned integer descriptor pointer; bit-identity holds. -> - (c) class-as-map-key / `(group-by class coll)` work? **Yes** — interned-pointer payload → bit-identity → HAMT contract holds, same mechanism as Alt 2. -> - (d) `(class 5)` prints as? **Free choice** — a dedicated tag gets a dedicated print arm, so `Long` or `java.lang.Long` is equally easy. (This is the one genuine degree of freedom the tag buys, and it is also achievable in Alt 2 by choosing the print arm's format.) -> - (e) composes with instance?/extends?/satisfies?? **Worse** — those speak `.type_descriptor` / `.td_ptr` today; a `.class` tag forces either a conversion at every boundary or a dual-tag acceptance in each of those functions. The draft/Alt-2 single-tag approach composes for free; Alt 3 adds friction at exactly the composition points the draft cites as its strength. -> - (f) reusable-precedent quality: **Low.** "Spend an F-004 reserve slot to split a user-facing view from its already-existing internal representation, then re-bridge them" is a precedent the project should *not* want repeated — it is the redundant-representation anti-pattern F-004's own challenge text anticipates. -> -> --- -> -> ### Non-binding ranked recommendation -> -> 1. **Alt 2 (finished-form-clean A1: interning as a structural invariant, not an opportunistic cache; print-policy scoped as separable).** This is the draft's chosen direction, sharpened. It is the cleanest finished form: type descriptors join the interned-singleton family (keyword/symbol/primitives), `equal.zig`'s invariant comment stays literally true with zero edits, the equality/hash contract is upheld by construction at the box layer, and it reuses one mental model the codebase already teaches. The two things the draft should *add* (not change): (i) frame the cache as a correctness invariant with a one-line guard/assertion + a recorded GC-reachability note (infra-allocated, process-lifetime, never collected), so a future refactor cannot quietly add a non-interned minting path; (ii) explicitly scope the print decision as "prints `td.fqcn`; simple-name-vs-FQCN is the existing `nativeFqcnFor` policy, revisitable independently" so the ADR does not over-commit on print while deciding identity. -> -> 2. **Alt 1 (A2 equality+hash arms).** Correct and would ship, and its *explicitness* virtue is genuine — but it permanently complicates `equal.zig`'s equality/hash contract with a 3-way sync liability and *invalidates* the "bit-identity is contract-consistent for every non-string key" invariant comment, when interning is available and makes all of that unnecessary. The draft is right to reject it. Its only edge over Alt 2 (no cache field → no GC-reachability question) is dissolved by Alt 2 documenting that reasoning. Per F-002, the slightly-smaller-blast-radius-in-`makeTypeDescriptorRef` is a *bits-today* convenience, not a finished-form gain. -> -> 3. **Alt 3 (dedicated `class` tag).** Rejected. F-004's own challenge text predicts the verdict: it is redundant representation. It consumes a finite reserve slot for a speculative need, re-breaks the unification that is the draft's core virtue, and forces either a much larger surgery (delete `.type_descriptor`'s user role) or a worse cross-tag equality arm than Alt 1. The seam it builds (user-class-view independent of internal-descriptor-ref) has no current requirement — reservation-as-bias. -> -> **One cross-cutting flag for the ADR regardless of choice:** the cw-v0 precedent (`class`==`type`, returns a keyword `:integer`, `(class nil)`→`:nil`, no Class) that the draft consciously breaks on all three points is the *right* break — a keyword-as-class loses identity-as-a-type (you can't distinguish `(class 5)` the class from the keyword `:Long` someone might type), can't carry method tables, and doesn't compose with `instance?`. The ADR should record that the v0 keyword approach was rejected *because it cannot compose with the protocol/instance machinery that already exists in v2* (which v0 lacked), not merely "we changed our minds" — that is the load-bearing reason the divergence from v0 is finished-form-correct rather than churn. - -The main loop adopts the DA's recommendation (Alt 2) unchanged, -including both sharpening additions (interning-as-invariant + -print-policy-scoped-separable) and the cross-cutting v0-divergence -rationale. - -## Consequences - -- **Positive**: `class` / `type` work and compose with the existing - type system. `(= (class 5) (class 6))` is true; `(class 5)` is a - valid map key, so `(group-by class coll)` works. `.type_descriptor` - joins the interned-singleton family — `equal.zig`'s "bit-identity is - contract-consistent for every non-string key" invariant stays - literally true, no equality/hash arm added. Interning also removes - the per-call `TypeDescriptorRef` allocation (one ref per descriptor, - reused). Sets the reusable precedent "intern the box so bit-identity - = value-identity" for future singleton types. -- **DIVERGENCE from JVM**: `(class 5)` prints `Long`, not - `java.lang.Long` (no-JVM rule). `(class nil)` → `nil` (matches JVM). - Print form is a separable `nativeFqcnFor` policy, revisitable. -- **DIVERGENCE from cw-v0**: v0's keyword-returning `class`==`type` is - rejected because a keyword cannot compose with the v2 - protocol/instance machinery (carry a method table, be told apart - from a user keyword) — the load-bearing reason, not a mind-change. -- **Negative / watch**: `ref_cache` is a per-descriptor `?Value` - written through `@constCast` (logical-const memoization, mirroring - `extendType`'s existing `@constCast`). Valid because descriptors are - per-Runtime and infra-allocated (process-lifetime, never GC-moved); - a future GC that moves infra objects would need to revisit this. - -## Affected files - -- `src/runtime/type_descriptor.zig` — `ref_cache: ?Value = null` field - on `TypeDescriptor`; `makeTypeDescriptorRef` interns (cache-or-mint) - with the invariant comment. -- `src/runtime/print.zig` — `.type_descriptor` arm printing `td.fqcn`. -- `src/lang/primitive/protocol.zig` — `rt/__class` primitive + - registration + unit tests. -- `src/lang/clj/clojure/core.clj` — `class` / `type` wrappers. -- `test/e2e/` — `class` / `type` surface cases. -- `.dev/handover.md` — verified-gap list updated. diff --git a/.dev/decisions/0060_internal_errors_catchable_via_ex_info_class_name.md b/.dev/decisions/0060_internal_errors_catchable_via_ex_info_class_name.md deleted file mode 100644 index 081542f9e..000000000 --- a/.dev/decisions/0060_internal_errors_catchable_via_ex_info_class_name.md +++ /dev/null @@ -1,298 +0,0 @@ -# ADR-0060 — Internal runtime errors are catchable: synthesize a class-name-bearing ex_info at the try boundary - -- **Status**: Accepted -- **Date**: 2026-05-31 -- **Phase**: Phase 14 (post-v0.1.0 coverage) / cluster A26 -- **Supersedes**: — -- **Superseded by**: — - -## Context - -`(try (/ 1 0) (catch Exception e :caught))` returned the uncaught -message "Divide by zero" instead of `:caught` (confirmed; real Clojure -via `clj` throws `ArithmeticException`, caught by `(catch Exception …)`). -`(catch ArithmeticException …)` / `(catch Throwable …)` also failed. -But `(try (throw (ex-info "x" {})) (catch Exception e :caught))` WORKS. -So cljw's **internal** runtime errors (everything raised via -`error_catalog.raise` — divide-by-zero, type errors, index-out-of-range, -arity, …) are NOT catchable by try/catch; only user-`throw`-ed ex_info -Values are. This breaks the most common Clojure error-handling idiom and -is a behavioural-equivalence defect (F-011). - -Root cause (dual error channels): - -- `error_catalog.raise` (`catalog.zig:1009`) returns a typed - `ClojureWasmError` Zig error (e.g. `error.ArithmeticError`) and sets a - threadlocal `Info{kind, message, origin=.catalog}` via `setErrorFmt`. - It never sets `dispatch.last_thrown_exception` and never builds a - throwable Value. -- `evalThrow` (`tree_walk.zig:794`) sets `last_thrown_exception = v` + - returns `error.ThrownValue`. `evalTry` (`tree_walk.zig:804`) catches - `error.ThrownValue`, reads the Value, runs `catchMatches` → - `host_class.matches(thrown, class)` (walks the Throwable hierarchy). - Its `else =>` arm (`tree_walk.zig:840`) runs `finally` + `return err` — - internal errors propagate **uncaught**, never converted, never matched. - The VM mirror (`vm.zig:99`) is structurally the same; both backends - already converge on the single shared predicate `host_class.matches` - (`vm.zig:673-674`). -- `host_class.zig::thrownClassName` (L166) hardcodes every `.ex_info` - Value to class `"ExceptionInfo"`, so a synthesized throwable would - need to carry its real class to match `(catch ArithmeticException …)`. - -`Origin = enum { catalog, thrown }` (`info.zig:79`, ADR-0055) is a -deliberate two-origin model: an uncaught internal error renders with its -`[arithmetic_error]` Kind header (origin=.catalog); a `(throw v)` renders -as `:exception`. The fix must preserve this for the **uncaught** path. - -cw v0 DID make these catchable (`createRuntimeException`, -`~/Documents/MyProducts/ClojureWasm/.../tree_walk.zig:1450`): it switched -on `info.kind` to pick a `:__ex_type` string, packed a synthetic -exception map, and matched it via the same hierarchy walk cljw now has in -`host_class.zig`. Per `no_copy_from_v1.md`, cljw re-derives this in a -cljw-appropriate shape (struct field, not a map key; direct `host_class` -import, not a pub-var injection). - -## Decision - -Make internal **user-domain** errors catchable by **synthesizing a -class-name-bearing ex_info at the try boundary** and falling into the -existing shared catch-matching loop. cljw's `.ex_info` tag is the -**single unified throwable representation**; a `class_name` field -discriminates "which exception class this Value represents". This is the -F-011 commonised + behaviourally-equivalent finished form. - -1. **`ExInfo.class_name: ?[]const u8 = null`** (`ex_info.zig`). For a - real `(ex-info …)` Value it stays `null` (≡ `ExceptionInfo`); for a - synthesized internal error it carries the Kind-derived class - (`"ArithmeticException"`, …) — a comptime-static catalog string, so - no GC ownership / `mark` / `rawFree` change is needed. - -2. **`kindToHostClass(kind) []const u8`** (in `host_class.zig`) — the - Kind→class map (grounded against real `clj`): - - | Kind | catch class | - |--------------------|-----------------------------| - | `arithmetic_error` | `ArithmeticException` | - | `type_error` | `ClassCastException` | - | `index_error` | `IndexOutOfBoundsException` | - | `value_error` | `IllegalArgumentException` | - | `arity_error` | `ArityException` | - | `number_error` | `NumberFormatException` | - | `name_error` | `RuntimeException` | - | `syntax_error` | `RuntimeException` | - | `string_error` | `RuntimeException` | - | `io_error` | `IOException` | - - **Uncatchable** (return null → keep the current `return err` / - `[kind]` render): `internal_error`, `out_of_memory` (descend from - `Error`, matching JVM `(catch Exception …)` NOT catching them), and - `not_implemented` (a `(catch Exception …)` silently swallowing - "feature not yet built" is the Silent-default-shift smell; keep it a - loud uncaught signal — DA-endorsed). - -3. **`evalTry` `else =>` arm (TreeWalk) + the VM run-loop error path**: - if the propagating `ClojureWasmError`'s Kind maps to a class - (`kindToHostClass` non-null), read `info.peekLastError()`, build - `ex_info.alloc(rt, info.message, nil_data, nil)` with `class_name` - set, stash it in `last_thrown_exception`, clear the threadlocal Info, - and fall into the existing catch-matching loop. Otherwise keep the - current `return err` (uncatchable). Conversion is **local to the try - boundary**, so an uncaught internal error keeps its `[arithmetic_error]` - Kind header (origin stays `.catalog`). - -4. **`thrownClassName` (`host_class.zig:166`)**: for an `.ex_info` - carrying a non-null `class_name`, return it; else `"ExceptionInfo"`. - `matches()`/`isSubclassOf` then resolve `ArithmeticException` (and - `(instance? Throwable e)` etc.) via the existing hierarchy walk, with - **zero** change to the predicate itself. - -5. **`ex-data` / `ex-info?` gate on `class_name`** (`ex_info.zig` + - `lang/primitive/error.zig`): a synthesized exception (non-null, - non-`ExceptionInfo` class) reports `(ex-data e)` → `nil` and - `(instance? clojure.lang.ExceptionInfo e)` → `false` (the latter free - via `thrownClassName`), matching real Clojure where a bare - `ArithmeticException` is not an `IExceptionInfo`. `(ex-message e)` / - `(.getMessage e)` / `str` return the catalog message (already duped - through `rt.gc.infra` by `ex_info.alloc`). - -6. **Add `ArityException` (parent `IllegalArgumentException`)** to the - `host_class` table + FQCN map — the one class the table lacks, so - `(catch ArityException …)` parses. - -7. **Differential test** (`diff_test.zig`) so TreeWalk↔VM agree: - `(try (/ 1 0) (catch ArithmeticException e (ex-message e)))` + - `(catch Exception …)` + a no-match re-raise. - -**Why this over the DA's recommended Alt 2 (a first-class `.host_error` -Value tag).** The Devil's-advocate fork (below, verbatim) found the -naive Option A *masquerade* defect (instance?/ex-data wrong) and -recommended a dedicated `.host_error` tag as the "cleanest sibling -model". The main loop adopts the **correctness** of Alt 2 (gate -instance?/ex-data so a synthesized exception is observably NOT an -ExceptionInfo) **without** the new tag, because: - -- **F-011 (commonization) favours one unified throwable representation.** - `.ex_info` becoming "the throwable Value, with an optional class_name - saying which exception class it is" is the *shared mechanism*; a - parallel `.host_error` tag is a second representation of the same - concept (Cycle-budget defer's inverse — proliferation, not reuse). -- **F-011 (internals/representation may diverge; behaviour must match).** - Alt 2's "ArithmeticException is its own class, not ex_info wearing a - hat" is a JVM-faithfulness argument about *internal representation*. - F-011 explicitly frees internals: cljw may use one `.ex_info` rep - internally as long as `(instance? ExceptionInfo e)` / `(ex-data e)` / - `(catch …)` observably match — which the class_name gate achieves. -- **F-004 (the 64-slot NaN-box layout is a fixed user invariant).** Alt - 2 consumes a reserved slot + adds the tag to every switch - (`mark_sweep`, `rawFree`, `print`, ex-* builtins) — ~11 sites vs the - ~4 gate sites here. This is **not** a cycle-budget/diff-size rejection - (the DA's named smell) — it is an F-004 slot-economy + F-011 - commonization decision. The observable behaviour is identical to Alt 2. - -If a future need arises where a synthesized exception must carry -genuinely different *internal* structure from an ex_info (not just a -class name), the `.host_error` tag is the escalation; today the -class_name field on the unified `.ex_info` rep is the F-011 finished -form. - -## Alternatives considered - -The following is the Devil's-advocate subagent's verbatim output (fresh -context, briefed with the survey + the F-NNN envelope + verified -file:line). Its recommendation (Alt 2, a `.host_error` tag) is recorded -in full; the main loop adopts Alt 2's *correctness* via the unified -ex_info + class_name gate per F-011/F-004 as narrated above. - -> # Devil's-advocate: ADR-0060 (internal runtime errors catchable by try/catch) -> -> Verified against working tree. The draft's `catalog.zig:1009` (`raise`), `tree_walk.zig:804` (`evalTry`), `host_class.zig:166` (`thrownClassName`) line cites are all accurate. Two facts the draft under-weights, both load-bearing below: -> -> - **Both backends already converge on one predicate.** `vm.zig:673-674` literally comments "both backends share the predicate" and calls `host_class.matches(thrown, class_name)`; `tree_walk.catchMatches` (tree_walk.zig:855) does the same. The synthesis-then-match design therefore has a *single* match entry point already. This shrinks the dual-backend cost the draft assigns to Option B and changes the parity calculus across the board. -> - **`ExInfo` is an `extern struct`** (ex_info.zig:46) with a fixed C-ABI layout `{header, _pad, message_ptr, message_len, data, cause}` and a `mark`/`rawFree`/`asExInfo` trio that all assume that layout. The draft's "NEW field `class_name`" is not a free addition — it touches the GC trace fn, the rawFree path, and any code that hardcodes the struct's size. That is fine (it is language-core, F-009-exempt), but it makes the draft *more* invasive than the prose implies, which weakens "smallest-diff" as the draft's selling point and strengthens the finished-form alternative below. -> -> A third fact constrains every alternative: **`Origin = enum { catalog, thrown }`** (info.zig:79) is ADR-0055's deliberate two-origin model, kept precisely so a `(throw v)` has no fake `Kind`. Any alternative that synthesizes a `Kind`-bearing ex_info for caught internal errors must not collapse this distinction for the *uncaught* path. All three alternatives below preserve it. -> -> --- -> -> ## Alternative 1 — Smallest-diff: data-map key `:cljw.error/class`, no struct field -> -> **Shape.** Identical to the draft's Option A *except* sub-decision 2: instead of a new `ExInfo.class_name` field, the synthesized ex_info carries the class name as an entry in its `data` map under a reserved namespaced keyword `:cljw.error/class`. `thrownClassName` (host_class.zig:166) gains a branch that, for an `.ex_info`, reads `data`, looks up `:cljw.error/class`, and returns the string if present, else `"ExceptionInfo"`. Everything else — conversion at the try boundary, the Kind→class map, the uncatchable triad, the `ArityException` table addition — is unchanged from the draft. -> -> **What it does better than the draft.** -> - **Zero `extern struct` surgery.** `ExInfo`'s C-ABI layout, `mark` (ex_info.zig:94 walks `data`+`cause`), `rawFree`, and `asExInfo` are untouched. The class name rides inside the `data` Value, which `mark` already traces — so GC correctness is free, where the draft's new field needs a conscious "is `class_name` owned? does it need freeing?" decision (it is a `?[]const u8` of catalog-static bytes, so no — but that is one more invariant to get right and document). -> - **Survives a future serialization boundary.** If ex_info is ever printed/EDN-encoded/sent across a Wasm component boundary, the class lives in ordinary map data that the existing printer handles; a side-channel struct field would need bespoke plumbing at each such boundary. -> -> **What it breaks / risks.** -> - **Collision + leakage, exactly the draft's stated objection.** `(ex-data e)` on a caught `(/ 1 0)` now returns `{:cljw.error/class "ArithmeticException"}` instead of `{}` (or `nil`). Real Clojure's `(ex-data (try (/ 1 0) (catch Throwable e e)))` is `nil` (an `ArithmeticException` is not an `IExceptionInfo`). So this **leaks a synthetic key into user-visible `ex-data`** — a fidelity regression the draft explicitly flagged as "collision-prone." A user doing `(keys (ex-data e))` sees a key Clojure never puts there. -> - **Reserved-namespace is a weak guarantee.** Nothing prevents user data from already carrying `:cljw.error/class`; the synthesis would have to decide overwrite-vs-merge. For internal errors the `data` map is empty so it is moot *today*, but it is a latent footgun the field-based design does not have. -> -> **Explicit answers.** -> - (a) `(catch Exception e ...)` catches `(/ 1 0)`? **Yes** — synthesized ex_info maps via `isSubclassOf("ArithmeticException","Exception")` (host_class.zig:149 walk: ArithmeticException→RuntimeException→Exception). -> - (b) `(catch ArithmeticException e ...)` catches it specifically? **Yes** — `thrownClassName` returns "ArithmeticException" from the data key; `isSubclassOf("ArithmeticException","ArithmeticException")` is true (self, host_class.zig:235). -> - (c) Uncaught `(/ 1 0)` still renders `[arithmetic_error]`? **Yes** — synthesis happens only inside the try arm. With no enclosing try, the Zig error-union propagates to the CLI/REPL render path which reads the threadlocal `Info{kind=.arithmetic_error, origin=.catalog}`. Origin stays `.catalog`, Kind header intact. ADR-0055 preserved. -> - (d) Dual-backend parity cost? **Low.** Both backends already route through `host_class.matches` (vm.zig:674). The new conversion logic lives in two error-arms (tree_walk.zig:840 `else =>`, vm.zig:99 `return err`), but the *match* path is shared. One differential case covers both. Required per ADR-0036. -> - (e) `(ex-message e)`/`(ex-data e)` work? `ex-message` **yes** (real message from `info.message`). `ex-data` **yes but wrong** — returns `{:cljw.error/class ...}` where Clojure returns `nil`. **Fidelity regression.** -> - (f) Reusable-precedent quality? **Mediocre.** The next "internal error → catchable" need inherits a data-map-pollution pattern; each new synthetic field is another reserved key leaking into `ex-data`. The pattern does not scale cleanly. -> -> --- -> -> ## Alternative 2 — Finished-form-clean: a first-class `host_error` Value tag (not an ex_info masquerade) -> -> **Shape.** Stop pretending an internal error *is* an ExceptionInfo. Introduce a dedicated throwable Value variant — call it `.host_error` — that carries `{kind: Kind, message: []const u8, data: Value = nil}` and represents "a runtime-synthesized JVM-style exception" as its own thing, parallel to `.ex_info`. At the try boundary (tree_walk.zig:840 `else =>` and vm.zig:99), a user-domain `ClojureWasmError` is wrapped into a `.host_error` Value, stashed in `last_thrown_exception`, and matching falls into the existing shared loop. `thrownClassName` (host_class.zig:166) gains a `.host_error => kindToHostClass(v.kind)` arm — mapping is computed from the carried `Kind`, no string is stored redundantly. `ex-message`/`ex-data`/`ex-info?` are taught about `.host_error`: `ex-message` returns the message, `ex-data` returns `nil` (matching JVM: a bare `ArithmeticException` is not `IExceptionInfo`), `(instance? clojure.lang.ExceptionInfo e)` returns **false** for it (correct — `ArithmeticException` is not an `ExceptionInfo`), while `(instance? Throwable e)` / `(instance? Exception e)` / `(instance? ArithmeticException e)` return **true** via the hierarchy walk. -> -> This is what real Clojure's object model *is*: `ArithmeticException` and `ExceptionInfo` are siblings under `RuntimeException`, not one wearing the other's clothes. The draft's Option A makes every caught internal error an `ExceptionInfo` instance wearing a `class_name` hat — which means `(instance? clojure.lang.ExceptionInfo (try (/ 1 0) (catch Throwable e e)))` would wrongly return **true** under the draft, where real Clojure returns **false**. The draft did not list this defect; it is the strongest argument for Alt 2. -> -> **What it does better than the draft.** -> - **`(instance? ExceptionInfo e)` is correct.** The draft's masquerade makes a divide-by-zero look like an ExceptionInfo to `instance?` and to any `ex-data`-gated logic. Alt 2 gets the type identity right, which is the whole point of catch-by-class. -> - **No `extern struct` surgery, no data-map pollution.** `ExInfo` (ex_info.zig:46) is left exactly as designed for genuine `(ex-info …)` values; the new concern lives in its own tag. Clean separation: `.ex_info` = "user attached structured data", `.host_error` = "runtime raised a typed condition". -> - **`kind` is the single source; class name is derived, never stored.** No redundant string to keep in sync with the `Kind`. The Kind⇄class map is one comptime function consulted at match time — mirrors how `host_class.matches` already derives everything from the class-name string rather than caching identity. -> - **Cleanest precedent (f).** The next "internal error → catchable" need is *already handled*: it is just another `Kind` in the map. No new field, no new reserved key, no new struct variant per error family. -> -> **What it breaks / risks.** -> - **Largest diff of the three** — a new `Value.Tag` variant touches: the tag enum, `mark_sweep` trace, `rawFree`, the printer (`print` must render a `.host_error`), `ex-message`/`ex-data`/`ex-info?`/`instance?` builtins, the analyzer/VM where ex-* primitives dispatch on tag, plus the two try-arms and `thrownClassName`. **Per F-002, diff size is explicitly NOT a constraint, so this does not count against it.** It does mean more surface for a bug in the introducing cycle. -> - **New tag = new differential-parity obligation** (ADR-0036): every backend path that can produce or observe a `.host_error` needs a diff case. Mitigated by the shared `matches` predicate, but the *production* side (try-arm synthesis) exists in two places and both need coverage. -> - **`(throw (ArithmeticException.))`-style user construction** is not addressed (cljw has no Class constructors yet) — but neither is it in the draft; out of scope either way. -> -> **Explicit answers.** -> - (a) `(catch Exception e ...)` catches `(/ 1 0)`? **Yes** — `.host_error` with kind `arithmetic_error` → "ArithmeticException" → `isSubclassOf(...,"Exception")` true. -> - (b) `(catch ArithmeticException e ...)`? **Yes** — derived class equals target, self-subclass true. -> - (c) Uncaught still renders `[arithmetic_error]`? **Yes** — same reasoning as Alt 1: synthesis is local to the try arm; with no try, the threadlocal `Info{origin=.catalog, kind=.arithmetic_error}` drives the CLI render. ADR-0055 origin model untouched (and *reinforced*: the catalog origin is the uncaught path, the new `.host_error` Value is the caught path — clean duality). -> - (d) Dual-backend parity cost? **Moderate.** Match path shared (vm.zig:674). Production path duplicated across two try-arms + a new tag observed by ex-* builtins → more diff cases than Alt 1, but each is mechanical. Highest of the three on raw case count; lowest on *conceptual* risk because the model is correct. -> - (e) `(ex-message e)`/`(ex-data e)`? `ex-message` **yes** (carried message). `ex-data` **`nil`** — *correct*, matches JVM where a bare `ArithmeticException` carries no ex-data. This is the only alternative that gets `ex-data` right. -> - (f) Reusable-precedent quality? **Best.** Adding the next catchable internal-error family is one map row + (if a new class) one host_class table row. No struct/field/key growth. Models Clojure's actual sibling-class reality. -> -> --- -> -> ## Alternative 3 — Wildcard: lazy match, synthesize the bound Value only on catch (draft's "Option B", re-evaluated) -> -> **Shape.** Do not synthesize *anything* at the throw boundary. When a user-domain `ClojureWasmError` reaches the try arm, do not build an ex_info and do not set `last_thrown_exception`. Instead, stash the raw `(Kind, message)` in a lightweight threadlocal "pending host condition" slot (the `Info` is *already* there — peek it). Widen the match predicate so `catchMatches`/the VM mirror can answer "does class `C` match a pending host condition of `Kind K`?" directly from `kindToHostClass(K)` + `isSubclassOf`, **without** a Value. Only when a clause *matches* do we materialize the bound Value (an Alt-2 `.host_error` or an Alt-1 ex_info — the wildcard is agnostic) and bind it to the catch local. On no match, nothing was allocated; the Zig error re-propagates with the threadlocal Info still intact. -> -> **What it does better than the draft.** -> - **Zero allocation on the non-matching / re-throw path.** The draft (and Alt 1/2) allocate a heap Value at the throw boundary even when no clause matches and the error re-propagates to the CLI. Alt 3 allocates only on a successful catch. For deeply-nested try/catch where inner clauses do not match (common in `cond`-style error handling), this is strictly less GC pressure. -> - **Uncaught path is provably untouched.** Because nothing is synthesized until a match, the uncaught render path *cannot* be perturbed. -> -> **What it breaks / risks.** -> - **Two match entry points** — a genuine second predicate `matchesKind(kind, class_name)` both backends must call (≤5-line shims over `isSubclassOf`, but real). -> - **Late materialization complicates the catch-binding code.** `catchMatches` stops being a pure `bool`; the VM handler-unwind protocol (vm.zig:84-97) expects the Value already staged in `last_thrown_exception`, so Alt 3 is a more intrusive VM change than Alt 1/2. -> - **Keyword-target catch (`(catch :my/type e …)`, tree_walk.zig:860) still needs a real Value** (it inspects ex-data `:type`), forcing a materialization carve-out — the laziness does not generalize. -> -> **Explicit answers.** -> - (a)/(b) Yes / Yes. (c) Yes, most robustly. (d) Highest parity cost (two predicates × two backends + VM unwind protocol change). (e) Works once materialized; should pair with Alt 2's Value shape for correct `ex-data`. (f) Good for allocation-discipline, poor for simplicity. -> -> --- -> -> ## Non-binding ranked recommendation -> -> 1. **Alternative 2 (first-class `.host_error` Value tag) — recommended.** It is the only option that gets `(instance? clojure.lang.ExceptionInfo e)` and `(ex-data e)` *correct* — the draft's Option A silently makes every caught internal error masquerade as an ExceptionInfo (`instance? ExceptionInfo` → wrongly `true`, `ex-data` → wrongly non-nil), a fidelity defect the draft did not enumerate. Alt 2 models Clojure's actual sibling-class reality, derives the class name from the single `Kind` source with no redundant stored string, leaves the `extern struct ExInfo` untouched, and gives the best reusable precedent. It is the **largest diff** — but **per F-002 / F-009, diff size is not a project constraint, and the finished-form-clean shape wins.** Picking the draft's Option A *over* Alt 2 on "Option A is a smaller diff / Alt 2 touches the Value tag enum" grounds would be the **Cycle-budget defer smell** — re-pick Alt 2. -> -> 2. **Draft Option A (ExInfo `class_name` field) — acceptable fallback, with one correction.** If Alt 2's new-tag surface is judged too risky to land in one cycle, Option A works for catch-by-class. But it **must** additionally fix `instance?`/`ex-data` so a class-name-bearing ex_info reports `false` for `(instance? ExceptionInfo …)` and `nil` for `ex-data` consistent with the carried class — otherwise it ships the masquerade defect. With that correction Option A converges toward Alt 2 anyway, which is itself evidence Alt 2 is the finished form. -> -> 3. **Alternative 1 (`:cljw.error/class` data key) — not recommended.** Smallest diff, but leaks a synthetic key into user-visible `(ex-data e)` and still has the `instance? ExceptionInfo` masquerade. -> -> 4. **Alternative 3 (lazy match) — not recommended for this ADR.** Allocation-discipline win is real but premature; highest parity cost + perturbs the VM unwind protocol + the `:type_keyword` carve-out breaks laziness. Worth recording as a **future optimization** once a benchmark shows throw-boundary allocation matters. -> -> **Sub-decisions, resolved:** conversion at try-boundary (not raise-site) is **correct** — keeps the uncaught `[arithmetic_error]` header (ADR-0055 origin model). `not_implemented` stays **uncatchable** (Silent-default-shift smell otherwise). Uncaught Kind-header preservation **holds** for try-boundary conversion across all alternatives. - -The main loop adopts the DA's **correctness findings** (gate -instance?/ex-data; conversion at try boundary; not_implemented -uncatchable; add ArityException) but realises them on the **unified -`.ex_info` representation + `class_name` field** rather than a new -`.host_error` tag, per F-011 (commonization + internals-free) and F-004 -(slot economy) as narrated in § Decision. The override is F-NNN-grounded, -not a cycle-budget defer. - -## Consequences - -- **Positive**: `(try (/ 1 0) (catch Exception e …))` works; the most - common Clojure error idiom matches real Clojure (verified via `clj`). - `(catch ArithmeticException …)` / `(catch Throwable …)` resolve via - the existing hierarchy walk. `(instance? ExceptionInfo e)` → false and - `(ex-data e)` → nil for a synthesized exception (behavioural - equivalence). One unified throwable representation (`.ex_info` + - `class_name`); no new NaN-box slot; the next catchable-internal-error - family is one `kindToHostClass` row. -- **DIVERGENCE from JVM**: cljw matches by class-name STRING + a comptime - hierarchy table, not Class-object identity (no-JVM rule). The Kind→class - map is coarser than JVM's per-call class (cljw's one `type_error` Kind - maps to `ClassCastException`, where JVM might throw - `IllegalArgumentException` for some type errors) — `(catch Exception …)` - always catches; the precise subclass is approximate per Kind. -- **Uncatchable by design**: `internal_error`, `out_of_memory`, - `not_implemented` keep the `return err` path + `[kind]` header. -- **`phase14_float_div.sh`** asserts `(/ 1 0)` prints uncaught "Divide by - zero" — re-pointed: the bare form still renders the Kind header; a new - case covers the caught form. - -## Affected files - -- `src/runtime/collection/ex_info.zig` — `class_name` field + accessor; - `ex-data` gate. -- `src/runtime/error/host_class.zig` — `kindToHostClass` map; - `thrownClassName` reads `class_name`; `ArityException` table + FQCN. -- `src/eval/backend/tree_walk.zig` — `evalTry` `else =>` synthesises. -- `src/eval/backend/vm.zig` — run-loop error path synthesises (parity). -- `src/lang/primitive/error.zig` — `ex-data`/`ex-info?` gate. -- `src/lang/diff_test.zig` — differential cases. -- `test/e2e/` — catch-internal-error cases; `phase14_float_div.sh` - re-point. -- `.dev/debt.md` — D-048 host_instance arm note (related). diff --git a/.dev/decisions/0061_java_static_field_resolution.md b/.dev/decisions/0061_java_static_field_resolution.md deleted file mode 100644 index 1b512177f..000000000 --- a/.dev/decisions/0061_java_static_field_resolution.md +++ /dev/null @@ -1,291 +0,0 @@ -# ADR-0061 — Java static field resolution: `Class/FIELD` bare symbol → `TypeDescriptor.static_fields` - -- **Status**: Accepted -- **Date**: 2026-05-31 -- **Phase**: Phase 14 (post-v0.1.0 coverage) / cluster A26 (clj differential sweep) -- **Supersedes**: — -- **Superseded by**: — - -## Context - -`(Integer/parseInt "42")` (a call, WITH parens) works: `analyzeList` -(`analyzer.zig:498-515`) resolves the qualified head via -`special_forms.resolveJavaSurface(rt, env, ns_head)` + `td.lookupMethod` -→ an `InteropCallNode { .kind = .static_method }`. But a bare qualified -symbol `Integer/MAX_VALUE` (no parens, a static FIELD read) goes through -`analyzeSymbol` (`analyzer.zig:402-450`), whose qualified-symbol arm -resolves the namespace via `aliases.get(ns) orelse env.findNs(ns) orelse -raise(.namespace_unknown)` (L425-428). "Integer" is not a Clojure -namespace (the surface is registered in `rt.types` as -`cljw.java.lang.Integer`, not as an Env namespace), so `findNs` → null → -`namespace_unknown`. There is **no static-field resolution path**. - -clj-verified targets (cluster A26): `Integer/MAX_VALUE` 2147483647, -`Integer/MIN_VALUE` -2147483648, `Long/MAX_VALUE` 9223372036854775807, -`Long/MIN_VALUE` -9223372036854775808, `Double/MAX_VALUE` -1.7976931348623157E308, `Double/MIN_VALUE` 4.9E-324. - -The static-method path proves the resolution machinery exists; the field -path is the missing symmetric half. - -## Decision - -A Java surface's `TypeDescriptor` owns **both** its static methods -(`method_table`, consumed by `analyzeList`) and its static fields -(a new `static_fields` slot, consumed by `analyzeSymbol`). A bare -`Class/FIELD` and a `(Class/method …)` resolve the **identical** -`*const TypeDescriptor` via `resolveJavaSurface` and then ask that -descriptor — `lookupStaticField` vs `lookupMethod`. **Symmetry is the -invariant**: future fields follow the method recipe; one resolution -path, one registry, two parallel lookups (F-011 commonisation; mirrors -Java's `Class` owning both tables, without any JVM-internals -assumption per `no_jvm_specific_assumption.md`). - -1. **`TypeDescriptor.static_fields: []const StaticField = &.{}`** - (defaulted, so every existing descriptor literal — Math / System / - deftype / defrecord / reify — is untouched, exactly as the - `ref_cache: ?Value = null` precedent shows). `StaticField = - struct { name, value: StaticFieldValue }`, - `StaticFieldValue = union(enum) { int: i64, float: f64 }`. Plus - `lookupStaticField(name) ?*const StaticField` (linear, parallel to - `lookupMethod`). - -2. **`static_fields` is comptime-const, populated in the descriptor - literal — NOT in `init`.** This is the load-bearing asymmetry vs - `method_table`: a `MethodEntry` holds `Value.initBuiltinFn(&fn)`, - whose `@intFromPtr(fn)` is **not** comptime-known on Mac, forcing - `method_table` to be GPA-built in `init` and freed in `deinit` - (`_host_api.zig:46-61`). A `StaticField` holds only an `i64`/`f64` - scalar + a string-literal name — **all comptime-known** — so each - surface declares a module-scope `const` array and sets - `.static_fields = &integer_static_fields` directly in the - descriptor literal. `installAll`'s `td.* = ext.descriptor.*` - duplicates the slice *pointer* to the process-lifetime comptime - array; **no `init` alloc, no `deinit` free** (and a future - "tidy for consistency" pass must NOT move it into `init` or free it - in `deinit` — a one-line comment on the field records this). - -3. **`analyzeSymbol` field arm** (before the `namespace_unknown` - raise): when a qualified symbol's ns resolves via - `resolveJavaSurface(env.rt, env, ns)` AND `td.lookupStaticField - (sym.name)` hits → `makeConstant(arena, val, form)` where - `val` = `integerLiteralToValue(rt, i)` for `.int` (i48 → Long, - beyond → BigInt — the EXISTING literal path, `analyzer.zig:277`) - / `Value.initFloat(f)` for `.float`. Otherwise the existing - `namespace_unknown` raise stands. - -4. **Integer / Long / Double surfaces** declare comptime - `static_fields` arrays (MAX_VALUE / MIN_VALUE; Double adds the two - bound constants). - -**Dual-backend parity (ADR-0036)**: `Class/FIELD` resolves to a -`.constant` Node at analyze time, exactly as integer/float literals do. -Both backends consume the same Node tree → identical constant Value, no -per-backend code, parity-trivial by construction. - -**DIVERGENCE inherited, not introduced (D-165, F-005)**: `Long/MAX_VALUE` -/ `MIN_VALUE` exceed i48 (F-004), so `integerLiteralToValue` lifts them -to a heap BigInt — exact value, but prints `…N` where JVM prints a bare -Long. This is the same i48-NaN-box consequence already recorded as D-165 -for `parse-long` / `Long/parseLong`; it is **inherited** from the shared -literal path, not re-introduced here, and stays owned by the -numeric-tower owner (F-003). `Integer/*` (fit i48 → clean Long) and -`Double/*` (floats) match clj exactly. - -## Alternatives considered - -The following is the Devil's-advocate subagent's verbatim output (fresh -context, briefed with the verified file:line facts + the F-NNN -envelope). Its recommendation (Alt 2) is the shape adopted above. - ---- - -## Devil's-advocate: Alternatives considered - -**F-NNN gate result up front:** None of the three alternatives below requires violating an F-NNN. All three resolve `Class/FIELD` and store the field value somewhere reachable from `analyzeSymbol`, all produce a `.constant` Node (so ADR-0036 dual-backend parity is untouched — see the cross-cutting flag), and all three land the Long/MAX_VALUE BigInt the same clean way the draft does (via the existing `integerLiteralToValue`, no init-time heap alloc). The tensions are entirely *within* the envelope: where the field registry lives (a `TypeDescriptor` slot vs a per-surface fn vs a central map vs a `def`-injected magic var), and whether to reuse the existing method-table mechanism or add a parallel one. F-005 + D-165 (Long/MAX_VALUE prints `…N` as a BigInt) is a recorded divergence none of these alternatives can or should "fix" — it is owned by the numeric-tower owner. **No forced "leading entry" F-NNN-block disclosure is needed** — the clean shape is reachable. - -One verified-fact correction the alternatives lean on: the draft's claim that StaticField "needs NO `init`-time gpa allocation, NO `rt.deinit` free" because it is a comptime `const` array is **correct as stated for the value payload** (`i64`/`f64` scalars + string-literal names are comptime-known, unlike `Value.initBuiltinFn(&fn)` which calls `@intFromPtr` and is not comptime on Mac — that is precisely why `method_table` *must* be built in `init` and freed in `deinit`, per `_host_api.zig:46-61`). So the draft's StaticField genuinely rides as `.static_fields = &integer_static_fields` in the descriptor literal, and the `installAll` heap-copy (`_host_api.zig:127` `td.* = ext.descriptor.*`) just duplicates the slice *pointer* to the process-lifetime comptime array — no free needed, no `deinit` edit. This asymmetry (fields are comptime-const, methods are not) is real and is the draft's strongest structural argument; the alternatives are judged against it. - ---- - -### Alternative 1 — Smallest-diff: a per-surface `lookupStaticField` fn that returns the *raw scalar*, computed-to-Value at the analyzer call site (no `StaticFieldValue` union, no new struct on the hot path) - -Keep the registry in each surface, but instead of the draft's `static_fields: []const StaticField` slice + `StaticFieldValue` union + `lookupStaticField` on `TypeDescriptor`, give each surface module a free function `pub fn staticField(name: []const u8) ?ScalarField` where `ScalarField = union(enum){ int: i64, float: f64 }`, and have `analyzeSymbol` reach it through a tiny dispatch the same way `resolveJavaSurface` reaches the descriptor. The descriptor itself gains **nothing** — `TypeDescriptor` stays exactly as it is today. - -**What it does better than the draft:** -- Touches `type_descriptor.zig` (a runtime-core, hot, F-009-carve-out file that every dispatch path includes) **zero times**. The draft adds two new types (`StaticField`, `StaticFieldValue`) + a `static_fields` field + a `lookupStaticField` method to `TypeDescriptor`, permanently widening the struct that `instance?`/`extends?`/`satisfies?`/the CallSite cache all carry around, for a feature (bare static-field read) that only 3 surfaces in `java.lang` will ever use. -- Keeps the field registry physically next to the methods it sits beside in the same surface file, so `rg MAX_VALUE src/runtime/java/lang/Integer.zig` finds it with zero indirection (feature-name-consistency R1 grep-100%). - -**What it breaks / risks:** -- It **breaks the `analyzeSymbol`/`analyzeList` symmetry** the draft preserves (question (d)). `analyzeList`'s `(Class/method …)` arm resolves a `*const TypeDescriptor` via `resolveJavaSurface` and then asks the *descriptor* `td.lookupMethod(...)`. Alt 1 would have `analyzeSymbol` resolve the descriptor for the namespace head but then **bypass** it to call a free `staticField` fn keyed by some side-channel (the surface module, not the descriptor) — so field-read and method-call resolve through two different mechanisms for the same `Class/`. That is exactly the kind of "two ways to do one thing" the project's commonization invariant (F-011) forbids. -- The side-channel is the real problem: `resolveJavaSurface` hands back a `*const TypeDescriptor`, not the surface module. To reach a per-module free fn you need a *second* registry mapping fqcn → fn-pointer, which is strictly more machinery than the draft's one slot, defeating the "smaller" claim. The only way to avoid the second registry is to put a `static_field_fn: ?*const fn(...)` pointer on the descriptor — at which point you are back to editing `TypeDescriptor`, but with a non-comptime fn-pointer field (re-introducing the `init`-time-alloc + `deinit`-free burden the draft specifically *avoids* by using comptime scalars). -- So Alt 1's "don't touch `TypeDescriptor`" is a smallest-diff-against-current-bits choice that is *not* smallest-diff-against-finished-form: it optimizes "don't widen the struct" at the cost of a parallel resolution mechanism and a second registry. - -**Explicit answers:** -- (a) BigInt for Long/MAX_VALUE: handled identically to the draft — the analyzer call site does `.int => integerLiteralToValue(rt, i)`, so 9.2e18 lifts to a heap BigInt at analyze time. Clean. ✓ -- (b) GC/lifetime: the `.constant` Node holds the Value; the Node lives in the analyzer arena (per-program lifetime), the BigInt lives on `rt.gc` and is reachable from the constant Node, so it is GC-rooted exactly like an integer literal. ✓ (Identical to the draft.) -- (c) reflection: only value reads are needed — no path requires `(class Integer/MAX_VALUE)` to *find* the field; `(class Integer/MAX_VALUE)` resolves the field to a `Long`/`Double` Value first, then `class` runs on that Value. So no field-on-descriptor reflection requirement. ✓ (True of all three alts.) -- (d) `analyzeSymbol`/`analyzeList` symmetry: **No** — this is the alternative's central weakness. Field-read and method-call resolve through different mechanisms. - ---- - -### Alternative 2 — Finished-form-clean: the draft's `TypeDescriptor.static_fields` slot, sharpened so field-read and method-read are the *same* descriptor-keyed lookup shape (the draft, made symmetric and the comptime-vs-init asymmetry recorded as the invariant) - -This is the draft, kept, with three sharpenings rather than changes: - -1. **Symmetry is the invariant, not a coincidence.** The draft already lands `lookupStaticField` on `TypeDescriptor` parallel to `lookupMethod`, and `analyzeSymbol` reaches it via `resolveJavaSurface(env.rt, env, ns)` — the *same* helper `analyzeList` uses for `(Class/method …)`. The finished-form framing makes that explicit: **both a bare `Class/FIELD` and a `(Class/method …)` resolve the identical `*const TypeDescriptor` and then ask that descriptor** (`lookupStaticField` vs `lookupMethod`). One resolution path, one registry, two parallel lookups on the same object. This is the F-011 commonization win Alt 1 forfeits, and it mirrors the Java `Class` model (a class carries both its methods and its static fields) without assuming any JVM internals (`no_jvm_specific_assumption.md` — we are modelling Clojure's `Class/FIELD` surface, not JVM bytecode). - -2. **The comptime-scalar-vs-non-comptime-fn asymmetry is recorded as the reason the two slots are populated differently.** `method_table` is GPA-built in `init` and freed in `deinit` *because* `Value.initBuiltinFn(&fn)` is not comptime on Mac (`_host_api.zig:46-61`). `static_fields` is a module-scope comptime `const` array set directly in the descriptor literal *because* `i64`/`f64` scalars + string-literal names **are** comptime-known. A one-line comment on the `static_fields` field records this: "comptime-const (scalars + literal names), so unlike `method_table` it needs no `init`-time alloc and no `deinit` free; `installAll`'s `td.* = ext.descriptor.*` duplicates the slice pointer to the process-lifetime comptime array." Without that note a future maintainer "tidying for consistency" might move `static_fields` into `init` (pointless alloc) or, worse, add a `deinit` free of a comptime array (crash). The asymmetry is finished-form-correct; the comment defends it. - -3. **`StaticFieldValue` is scoped to exactly the two scalar shapes the numeric tower can lift cleanly at analyze time** (`int: i64`, `float: f64`) and `analyzeSymbol` converts via the existing `integerLiteralToValue(rt, i)` / `Value.initFloat(f)`. This is the same path the integer/float *literal* reader already uses (analyzer.zig:277, the D-014a discharge), so `Long/MAX_VALUE` becomes a heap BigInt by the *one* existing mechanism, and D-165 (BigInt-print divergence) is inherited from that mechanism rather than re-introduced — there is literally no new numeric code, which is why the divergence stays owned by the numeric-tower owner. - -**What it does better than the draft-as-written:** -- The draft says "parallel to lookupMethod" but does not *commit* to symmetry as the invariant; Alt 2 makes "field-read and method-read are the same descriptor-keyed shape" the load-bearing property, so a future `Class/FIELD2` or a future surface adds fields by the same recipe it adds methods — one mental model (the keyword/symbol-interning precedent ADR-0059 set, applied here to "the class carries both tables"). -- It records *why* the two slots are populated asymmetrically, closing the maintenance hazard the draft leaves implicit (the "tidy for consistency" trap). - -**What it breaks / risks:** -- It widens `TypeDescriptor` by one slice field (`static_fields: []const StaticField = &.{}`). The default `= &.{}` means Math/System/every deftype/defrecord/reify descriptor literal is **untouched** (the draft verified this against the `ref_cache: ?Value = null` precedent at type_descriptor.zig:69 — a defaulted field needs no edit at existing literal sites). The cost is ~16 bytes per descriptor on a process-lifetime struct (~25 of them); negligible. The honest risk is *conceptual*, not byte-count: `TypeDescriptor` now models "a class's static-field table", which is a small scope-creep on a struct whose docstring (type_descriptor.zig:43) frames it as the deftype/defrecord/reify type descriptor. Alt 2's answer: that creep is *correct* — Java's `java.lang.Integer` genuinely is a type that owns static fields, and the surface descriptors are `kind = .native` precisely to model host classes. The field is on the right object. -- Long/MAX_VALUE printing `9223372036854775807N` (BigInt suffix) where JVM prints a bare Long is a **visible divergence** (D-165). Alt 2 does not hide this — it inherits it from `integerLiteralToValue` and lets D-165's owner (numeric-tower) decide whether i48-vs-i64 inline-Long ever changes. Bundling a "fix" here would be the smallest-diff-bias trap of pre-empting an owner's decision (F-003). - -**Explicit answers:** -- (a) BigInt for Long/MAX_VALUE: **cleanest of the three** — reuses `integerLiteralToValue` verbatim, zero new numeric code, divergence (D-165) inherited not introduced. ✓ -- (b) GC/lifetime: `.constant` Node (arena, per-program) holds the Value; scalar ints/floats are inline NaN-box (no heap); the BigInt for Long/MAX_VALUE is on `rt.gc` reachable from the constant Node → GC-rooted like any literal. The `static_fields` array itself is process-lifetime comptime data (never freed, never traced — it holds raw scalars, not Values). ✓ -- (c) reflection: value-read only; no descriptor-side field reflection needed. ✓ -- (d) `analyzeSymbol`/`analyzeList` symmetry: **Yes — this is the alternative's defining virtue.** Same `resolveJavaSurface` → same `*const TypeDescriptor` → `lookupStaticField` (symbol path) vs `lookupMethod` (list path). One registry, one resolution, two parallel lookups. - ---- - -### Alternative 3 — Wildcard: a central comptime `StaticStringMap` of fully-qualified `"Class/FIELD"` → pre-computable scalar, resolved in `analyzeSymbol` *before* `resolveJavaSurface`, surfaces carry no field data at all - -Drop the per-surface registry entirely. Add one module-scope `std.StaticStringMap(ScalarField)` in the analyzer (or a small `runtime/java/static_fields.zig` leaf) keyed by the cljw-prefixed FQCN-plus-field string (`"cljw.java.lang.Integer/MAX_VALUE"` etc.), `.initComptime(.{ .{"…/MAX_VALUE", .{.int = 2147483647}}, … })`. `analyzeSymbol`, on a qualified symbol, first translates the ns head the way `resolveJavaSurface` does (literal / `cljw.` / `cljw.java.lang.` prefix), concatenates `"/FIELD"`, probes the map, and on a hit emits `makeConstant`. `TypeDescriptor` is **never** touched; the surfaces (Integer/Long/Double.zig) are **never** touched. - -**What it does better than the draft:** -- Zero edits to `TypeDescriptor` *and* zero edits to the three surface files — the entire feature lands in one new leaf + one `analyzeSymbol` arm. It is the literally-smallest *blast radius across files*. -- The static-field set for `java.lang.*` is genuinely fixed and tiny (the MAX/MIN trio × 3 classes + a handful like `Double/POSITIVE_INFINITY`), and a `StaticStringMap` is the project's blessed idiom for exactly "a small fixed lookup table" (`zig_tips.md` § `comptime StaticStringMap`, used for keyword/opcode tables). The values are pure constants — there is no per-class behaviour to attach. - -**What it breaks / risks:** -- **It severs the field data from the class it belongs to**, which is the same anti-pattern ADR-0059's Alt 3 was rejected for: a separate representation that then has to be re-bridged. `Integer/parseInt` lives on `Integer.zig`'s descriptor, but `Integer/MAX_VALUE` would live in a central map keyed by string — so the answer to "what does `Class/X` do" depends on whether `X` happens to be a method (descriptor) or a field (central map). That **fails grep-locality** (R1): `rg MAX_VALUE src/runtime/java/lang/Integer.zig` returns nothing; the reader must know to look in the central map. F-009's whole point is that a feature's data lives *with* the feature, discoverable by keyword. -- **It breaks `analyzeSymbol`/`analyzeList` symmetry worse than Alt 1** (question (d)): method-call resolves through `resolveJavaSurface` → descriptor → `lookupMethod`; field-read resolves through a string-concat → central map, never touching the descriptor at all. Two completely disjoint mechanisms for `Class/method` vs `Class/FIELD`. This is the strongest F-011 commonization violation of the three. -- **It duplicates `resolveJavaSurface`'s prefix-translation logic** (the literal / `cljw.` / `cljw.java.lang.` three-try cascade at special_forms.zig:54-66) inside the new arm to build the map key, or it has to call `resolveJavaSurface` to get the descriptor's `fqcn` and *then* concat `/FIELD` and probe the map — at which point it has already resolved the descriptor and is choosing *not* to ask it, which is strictly more steps than Alt 2's "ask the descriptor you already have." -- The "values are pure constants, no behaviour to attach" argument is true but proves too little: methods are *also* keyed by name on a per-class basis, and the project already chose to attach them to the descriptor. Fields are the same shape (per-class, name-keyed); the consistent finished form is "attach them the same place," which is Alt 2. - -**Explicit answers:** -- (a) BigInt for Long/MAX_VALUE: handled the same as the others (the central map stores `.int = 9223372036854775807` and the arm calls `integerLiteralToValue`). Clean. ✓ -- (b) GC/lifetime: identical to Alt 2 — `.constant` Node holds the Value, BigInt on `rt.gc` rooted from it, the `StaticStringMap` holds raw scalars (process-lifetime comptime, not Values). ✓ -- (c) reflection: value-read only. ✓ -- (d) `analyzeSymbol`/`analyzeList` symmetry: **No — the worst of the three.** Field and method resolution are fully disjoint; the descriptor is bypassed for fields entirely. - ---- - -### Non-binding ranked recommendation - -1. **Alt 2 (finished-form-clean: the draft's `TypeDescriptor.static_fields` slot, with symmetry made the invariant + the comptime-vs-init asymmetry recorded + numeric lift reusing `integerLiteralToValue`).** This is the draft's chosen direction, sharpened — not changed. It is the cleanest finished form: a `Class/` resolves to *one* `*const TypeDescriptor` and that descriptor owns *both* its methods (`lookupMethod`, used by `analyzeList`) and its static fields (`lookupStaticField`, used by `analyzeSymbol`), so field-read and method-call share one resolution path and one registry (F-011 commonization). It reuses the existing `integerLiteralToValue` for the Long/MAX_VALUE BigInt with zero new numeric code, so D-165's divergence is inherited (and stays owned by the numeric-tower owner per F-003), not re-introduced. The two things the ADR should *add* to the draft (not change): (i) state symmetry as the invariant — "field-read and method-read are the same descriptor-keyed lookup shape" — so future fields follow the method recipe; (ii) record the comptime-scalar-vs-non-comptime-fn asymmetry in a one-line comment on `static_fields`, so a future "tidy for consistency" pass cannot move it into `init` (pointless alloc) or add a `deinit` free of comptime data (crash). - -2. **Alt 1 (per-surface `staticField` free fn, raw scalar computed at the call site).** Keeps the field data grep-local to the surface file (a genuine virtue) and avoids widening `TypeDescriptor`, but it forfeits the `analyzeSymbol`/`analyzeList` symmetry: it needs either a *second* fqcn→fn registry (more machinery than the draft's one slot, defeating "smaller") or a non-comptime fn-pointer field on the descriptor (re-introducing the `init`-alloc/`deinit`-free burden the draft's comptime scalars specifically avoid). Per F-011 the parallel resolution mechanism is the disqualifier; per F-002 its "don't widen the struct" is a bits-today convenience, not a finished-form gain. - -3. **Alt 3 (central `StaticStringMap` keyed by `"Class/FIELD"`).** Smallest cross-file blast radius and uses a blessed idiom for a fixed tiny table, but it severs field data from the class it belongs to (fails F-009 grep-locality: `rg MAX_VALUE Integer.zig` finds nothing) and breaks `analyzeSymbol`/`analyzeList` symmetry the hardest (field and method resolution fully disjoint, the descriptor bypassed for fields). It is the same separate-representation-then-re-bridge anti-pattern ADR-0059's Alt 3 was rejected for, in miniature. Reservation-of-a-side-channel where the descriptor you already resolved would answer the question. - -**One cross-cutting flag for the ADR regardless of choice (ADR-0036 dual-backend parity):** all three alternatives resolve `Class/FIELD` to a `.constant` Node **at analyze time** (via `makeConstant`), exactly as integer/float literals already do. The TreeWalk and VM backends both consume the same analyzed Node tree, so both see the *identical* constant Value (an inline `Long`/`Double` for the in-range fields, a heap BigInt for `Long/MAX_VALUE`) with no per-backend code. The dual-backend diff oracle (`test/diff/cases.yaml`, Layer 3) therefore needs no special handling — a `Class/FIELD` case is parity-trivial by construction, because the divergence point (if any) is the numeric representation set at analyze time, which is upstream of the backend split. **Confirmed: no alternative touches the ADR-0036 parity contract.** - -A second cross-cutting note: the draft (and Alt 2) correctly does **not** "fix" the D-165 Long/MAX_VALUE BigInt-print divergence (`…N` vs JVM's bare Long). That divergence is the F-005 numeric-tower consequence of i48-inline-Long, owned by the numeric-tower owner per F-003 — pre-empting it here (e.g. by widening the inline integer payload) would require an F-004 amendment (user-owned) and is out of scope for a static-field-resolution ADR. The ADR should record D-165 as inherited-not-introduced, exactly as the draft frames it. - ---- - -The main loop adopts **Alt 2** (finished-form-clean, the DA's #1) verbatim, -including the two sharpenings: symmetry as the stated invariant + the -comptime-vs-init asymmetry comment on `static_fields`. - -## Consequences - -- **Positive**: `Integer/MAX_VALUE` / `MIN_VALUE`, `Long/MAX_VALUE` / - `MIN_VALUE`, `Double/MAX_VALUE` / `MIN_VALUE` resolve (clj parity for - the value). A surface adds a static field by the same recipe it adds a - method — one mental model. No new numeric code; the BigInt path is the - existing literal mechanism. No `init`/`deinit` machinery (comptime - const). Dual-backend parity is trivial (constant Node at analyze time). -- **DIVERGENCE (D-165, inherited)**: `Long/MAX_VALUE` / `MIN_VALUE` print - `…N` (BigInt) vs JVM's bare Long; value exact. Owned by the - numeric-tower owner (F-003). `Integer/*` + `Double/*` match clj exactly. -- **Scope**: only static FIELD READS (`Class/FIELD` as a value). Static - field WRITES do not exist in Clojure semantics; method references - without parens are out of scope (Clojure has no bare static-method - value form). Boolean/Character have no MAX/MIN constants in scope here. - -## Affected files - -- `src/runtime/type_descriptor.zig` — `StaticField` / `StaticFieldValue` - + `static_fields` slot (defaulted) + `lookupStaticField` + the - comptime-vs-init asymmetry comment. -- `src/eval/analyzer/analyzer.zig` — `analyzeSymbol` field arm before the - `namespace_unknown` raise; `staticFieldValue` helper. -- `src/runtime/java/lang/{Integer,Long,Double}.zig` — comptime - `static_fields` arrays + `.static_fields = &…` in the descriptor. -- `test/e2e/phase14_static_fields.sh` — clj-parity cases (incl. the - D-165 BigInt-N pin for `Long/MAX_VALUE`). -- `.dev/debt.md` — D-165 cross-reference (Long/MAX_VALUE field inherits it). - -## Revision history - -### 2026-05-31 — `StaticFieldValue` gains `.bool` (Boolean/TRUE/FALSE), cluster A26 cycle F - -`java.lang.Boolean` needs `Boolean/TRUE` → `true` / `Boolean/FALSE` → -`false` static fields, whose values are booleans — outside the original -`int`/`float`-only `StaticFieldValue`. Amendment: add `.bool: bool` to -`StaticFieldValue`; `analyzeSymbol`'s lift switch gains -`.bool => |b| Value.initBoolean(b)` (the existing `value.zig` helper, -matching the `.float => Value.initFloat(f)` arm). A cljw boolean is an -inline `NB_CONST_TAG` immediate (comptime-const), so the §2 comptime-const -property (no `init`/`deinit`) holds; the descriptor slot, resolution path, -and dual-backend parity are unchanged. **Forward note (do not -over-anticipate)**: `Double/POSITIVE_INFINITY` / `NEGATIVE_INFINITY` / -`NaN` need NO new variant — they are comptime f64 (`std.math.inf` / `-inf` -/ `nan`) carried by the existing `.float`; only a future -`Character/MAX_VALUE` (a char) would add a `.char: u21` variant. - -Devil's-advocate (mandatory at depth ≥ 2) ran fresh-context; its verbatim -output (3 alternatives + ranked recommendation) follows. The main loop -adopts its #1 (Alt 1: `.bool` variant + `Value.initBoolean`). - -> ## Devil's-advocate: Alternatives considered (StaticFieldValue grows a bool) -> -> **F-NNN gate result up front:** None of the three alternatives below requires violating an F-NNN. A cljw boolean is an inline `NB_CONST_TAG` immediate Value (`value.zig:34-35` `true_val`/`false_val`; F-004) — comptime-const exactly like the existing `int`/`float` scalars, so every alternative preserves the `static_fields` comptime-const property (no `init` alloc, no `deinit` free) that ADR-0061 §2 made load-bearing. All three emit a `.constant` Node at analyze time, so ADR-0036 dual-backend parity is untouched (confirmed per-alternative below). F-011 commonization (one descriptor-keyed lookup shape) is the axis the alternatives are judged on. The clean shape is reachable; no forced F-NNN-block disclosure needed. -> -> One verified-fact sharpening the alternatives lean on: `value.zig:192` already ships `Value.initBoolean(b: bool) Value` = `if (b) true_val else false_val`. The draft's inline `if (b) Value.true_val else Value.false_val` re-spells this helper at the call site; the alternatives prefer the named helper for symmetry with the existing `.float => Value.initFloat(f)` arm. -> -> ### Alternative 1 — Smallest-diff: add `.bool: bool` to `StaticFieldValue`, lift via `Value.initBoolean` (the draft, using the existing helper) -> -> The draft as written, with the one-line tightening: the lift arm is `.bool => |b| Value.initBoolean(b)` (reusing `value.zig:192`) rather than an inline `if`. `StaticFieldValue` becomes `union(enum) { int: i64, float: f64, bool: bool }`; `Boolean.zig` declares `boolean_static_fields = [_]StaticField{ .{ .name="TRUE", .value=.{.bool=true} }, .{ .name="FALSE", .value=.{.bool=false} } }`. -> -> **What it does better:** Minimal, and each variant maps 1:1 to the analyzer arm that lifts it — `int → integerLiteralToValue`, `float → initFloat`, `bool → initBoolean`. The union stays a flat list of the primitive scalar kinds Clojure static fields actually carry, each comptime-const. Grep-local (`rg TRUE src/runtime/java/lang/Boolean.zig` hits). The lift switch stays exhaustive (`zig_tips.md` narrow-enum rule), so a future variant added without a matching arm is a compile error — a real regression guard. -> -> **What it breaks / risks:** Each new scalar shape costs one variant + one arm. Today's question already names the next ones: `Double/POSITIVE_INFINITY`/`NEGATIVE_INFINITY`/`NaN` are **comptime f64 values — already covered by `.float`** (`std.math.inf(f64)` / `-inf` / `nan` are comptime-const, `Value.initFloat` lifts them, no new variant). `Long/SIZE = 64` fits `.int`. The *one* genuinely-new shape on the horizon is `Character/MAX_VALUE` (a char `￿`), which would need a `.char: u21` variant + a `.char => |c| Value.initChar(c)` arm (`value.zig:216`). So the union's realistic end state is 4 variants (`int`/`float`/`bool`/`char`) — small and closed; Clojure host static fields are scalars, never collections. The risk is purely "N small edits over time" — but each is a one-line variant + one-line arm, and the exhaustiveness check makes them safe. This is finished-form-clean, not merely smallest-diff: the union models "the primitive Value kinds a constant field carries", which is exactly what it is. -> -> **Explicit answers:** (a) comptime-const for all future kinds incl. `Double` inf (`.float`, already covered) and `Character/MAX_VALUE` (`.char`, comptime u21): **yes** — every kind is an inline immediate or a comptime scalar. (b) regress the BigInt path? **No** — `.int => integerLiteralToValue` is untouched; `Long/MAX_VALUE` still lifts to a heap BigInt at analyze time exactly as ADR-0061 specifies. (c) dual-backend parity: **unaffected** — `.bool` emits a `.constant` Node like every other arm; both backends see the identical inline `true_val`/`false_val`. -> -> ### Alternative 2 — `.value: Value` single variant holding any comptime-const Value -> -> Collapse the union into one variant `StaticFieldValue = struct { value: Value }` (or drop the union entirely: `StaticField = struct { name, value: Value }`). `Boolean.zig` writes `.value = Value.true_val`; `Math.zig`-style float fields write `.value = Value.initFloat(...)`; the analyzer arm becomes a trivial `makeConstant(arena, sf.value, form)` — **no lift switch at all**. -> -> **What it does better:** One variant covers bool, char, float, and any in-range int with **zero** future edits — `Character/MAX_VALUE` is just `.value = Value.initChar(0xFFFF)` in the surface, no `type_descriptor.zig` touch ever again. The analyzer arm shrinks to one line (no switch). -> -> **What it breaks / risks — this is the disqualifier:** It **regresses the `Long/MAX_VALUE` BigInt path**, which is the whole reason ADR-0061 stored a *raw scalar* rather than a `Value` in the first place (see the field docstring, `type_descriptor.zig:109-113`: *"the value is kept as a raw scalar (not a `Value`) because a heap `Value` (BigInt for `Long/MAX_VALUE`) cannot be comptime-const"*). `9223372036854775807` exceeds i48 (F-004), so its Value is a **heap BigInt** — which is **not comptime-const** (it needs a `rt.gc` allocation). A `static_fields` array of `Value` therefore cannot hold `Long/MAX_VALUE` as a comptime literal: you would either (i) lose `Long/MAX_VALUE` (a shipped ADR-0061 target — regression), or (ii) reintroduce an `init`-time heap-build + `deinit`-free for the array, destroying the comptime-const property ADR-0061 §2 made load-bearing and explicitly forbade. Either branch breaks a confirmed invariant. The raw-scalar union exists *precisely* to defer the i48→BigInt lift to analyze time (where `rt.gc` is available) via `integerLiteralToValue`; `.value: Value` cannot defer it. This is a Reservation-as-convenience trap: it looks like the cleanest "any Value" generalization, but it collides with F-004's i48 NaN-box ceiling. -> -> **Explicit answers:** (a) comptime-const for all future kinds: **no** — fails for any field whose Value is a heap type, and `Long/MAX_VALUE` is one **today**. (b) regress BigInt? **Yes, directly** — this is the central failure. (c) dual-backend parity: would be fine in isolation (constant Node), but moot given (a)/(b). -> -> ### Alternative 3 — Wildcard: store a `comptime`-evaluated thunk `*const fn() Value` per field -> -> Make `StaticFieldValue` hold a function pointer `produce: *const fn(rt: *Runtime) anyerror!Value`, so each field declares how to build its own Value (`Boolean/TRUE` → returns `true_val`; `Long/MAX_VALUE` → returns the BigInt build). The analyzer arm calls `sf.value.produce(env.rt)`. -> -> **What it does better:** Maximally general — any Value, including heap ones, with no union growth and no analyzer switch; the per-field producer encapsulates the lift. -> -> **What it breaks / risks:** It reintroduces the exact non-comptime-fn-pointer hazard ADR-0061 §2 identified as the reason `method_table` (which holds `Value.initBuiltinFn(&fn)`) must be `init`-built and `deinit`-freed: `@intFromPtr(fn)` is **not comptime on Mac**. A `produce: *const fn(...)` field makes `static_fields` non-comptime *in the same way method_table is*, forcing it back into `init`/`deinit` — re-merging the asymmetry ADR-0061 deliberately split (its strongest structural argument). It also over-engineers a constant read into a runtime call for zero benefit: static fields are pure constants with no per-field behaviour to encapsulate. Strictly more machinery, and it forfeits the comptime-const property that is F-004/§2's load-bearing win. -> -> **Explicit answers:** (a) comptime-const: **no** — fn pointers are non-comptime on Mac (the documented `_host_api.zig:46-61` constraint). (b) regress BigInt? It *handles* BigInt, but only by paying the `init`/`deinit` cost the whole design avoids. (c) dual-backend parity: still a constant Node, but the producer runs at analyze time — parity-fine yet irrelevant given (a). -> -> ### Non-binding ranked recommendation -> -> 1. **Alt 1 (`.bool: bool` variant + `.bool => Value.initBoolean(b)` arm).** Finished-form-clean: `StaticFieldValue` stays a flat closed list of the primitive scalar kinds a Clojure host static field carries (`int`/`float`/`bool`, +`char` when `Character/MAX_VALUE` lands), each comptime-const, each with a 1:1 exhaustive lift arm reusing the existing literal path. It preserves the `Long/MAX_VALUE` raw-scalar→analyze-time-BigInt deferral, keeps the §2 comptime-const invariant, and stays grep-local + descriptor-keyed (F-011 symmetry). Tightening over the draft: use `Value.initBoolean(b)` in the arm. Record that `Double` infinities need no variant (existing `.float`); only `Character/MAX_VALUE` later adds `.char`. -> 2. **Alt 2 (`.value: Value`).** Tempting as "the one generalization that never needs another variant", right in a world without the i48 ceiling — but it **regresses `Long/MAX_VALUE`** (non-comptime heap BigInt), and forcing it comptime needs the `init`/`deinit` machinery §2 forbids. F-004 disqualifies it. -> 3. **Alt 3 (per-field `*const fn() Value` producer).** Reintroduces the non-comptime-fn-pointer cost ADR-0061 §2 split away from, over-engineers a constant read into a runtime call. Most machinery, forfeits the comptime-const win. -> -> **Cross-cutting (ADR-0036 parity):** all three resolve `Boolean/TRUE`/`FALSE` to a `.constant` Node at analyze time holding an inline immediate; TreeWalk and VM see the identical constant. Parity-trivial by construction. Confirmed: no alternative touches the parity contract. diff --git a/.dev/decisions/0062_context_budget_reduction.md b/.dev/decisions/0062_context_budget_reduction.md deleted file mode 100644 index 99021b4a6..000000000 --- a/.dev/decisions/0062_context_budget_reduction.md +++ /dev/null @@ -1,140 +0,0 @@ -# ADR-0062 — Context-budget reduction: prune always-on injection, cap window at 200K, disable unused MCP, de-duplicate compaction hooks - -- **Status**: Accepted -- **Date**: 2026-05-31 -- **Phase**: Phase 14 (post-v0.1.0) — scaffolding maintenance (user-directed) -- **Supersedes**: — -- **Superseded by**: — - -## Context - -Auto-injected scaffolding had grown to dominate every turn's context. -Measured per-turn floor (approx, bytes/4): - -| Region | tokens/turn | -|------------------------------------------------------------|-------------| -| project CLAUDE.md | ~7,471 | -| global `~/.claude/CLAUDE.md` | ~2,288 | -| always-on rules (`paths: ["**"]`) | ~5,197 | -| handover.md + MEMORY.md (SessionStart) | ~2,004 | -| MCP server-instruction blocks (figma / datadog / slack …) | several K | - -A `.zig`-edit turn additionally fires ~18K of path-scoped rules -(`zig_tips` 6.8K, `provisional_marker` 2.8K, `feature_name_consistency` -2.1K, …), so a single source edit carried ~35K of scaffolding before any -file content. Three concrete defects amplified this: - -1. **`gate_cadence.md` had no `paths:` frontmatter** → it loaded on - *every* turn (~1.5K) instead of only on test-runner / settings edits. -2. **`orphan_prevention.md`** (always-on, `paths: ["**"]`) had grown to - 218 lines (~2.4K/turn) — mostly incident narratives + a one-time - "0-hits sweep" + caveat expansions that are reference material, not - per-turn instruction. -3. **Unused MCP servers** (datadog / figma / slack / Notion / - chrome-devtools / playwright) — irrelevant to a Zig project — injected - their server-instruction blocks + deferred tool lists on every turn. - -Separately, the autonomous loop rides one session toward the 1M Opus -window; auto-compact fires at ~95% (~950K), so the session bloats and -becomes expensive before any compaction relief. The user chose to cap -the window at the 200K standard so compaction is early, cheap, and -frequent — relying on the handover.md + post-compaction re-injection -hook as the durable cross-compaction bridge. - -**PostCompact-hook finding.** A prior belief held that Claude Code has -no `PostCompact` event and that the project's `PostCompact` block was -dead (= compaction resilience broken). Verified against the official -docs (code.claude.com/docs/en/hooks, 2026-05-31): **`PostCompact` and -`PreCompact` are both real events**, and `SessionStart` also fires after -compaction with `matcher: "compact"`. The block was *not* dead. The -actual defect was redundancy: `SessionStart` matched `"*"` (which -includes the `compact` source) **and** `PostCompact` matched `"*"`, so a -compaction re-injected the brief twice. - -## Decision - -A user-directed scaffolding diet. Concrete changes: - -1. **Window → 200K.** Project `.claude/settings.json` sets - `"model": "claude-opus-4-8"` (non-`[1m]`). Compaction becomes early / - cheap / frequent; handover.md + the PostCompact hook carry state - across. -2. **`gate_cadence.md`** gains a `paths:` frontmatter scoping it to the - test runner / gate scripts / settings — off the always-on floor. -3. **`orphan_prevention.md`** trimmed 218 → ~58 lines: the two load- - bearing rules (`timeout 600` for background long-runners; - `timeout 20` + `(take N …)` for `clj -M -e` oracle probes), the - `timeout`-non-propagation caveat, the gate launcher, and the counter- - examples stay. Incident narratives move to their existing home, - ADR-0049 § Context (a pointer remains). -4. **MCP disabled for this project only** (override, not global): - `enabledPlugins` sets figma / slack / notion / chrome-devtools-mcp / - playwright to `false`; `disabledMcpServers` lists `datadog-mcp` (a - global `~/.claude.json` `mcpServers` http entry, not a plugin). -5. **Compaction hooks de-duplicated** with explicit matchers: - `SessionStart` → `startup|resume|clear` (full handover inline), - `PostCompact` → `auto|manual` (concise `print_handover_brief.sh`). - No source overlaps, so compaction re-injects exactly once. -6. **CLAUDE.md** stale framing pruned: the "Phase 4 entry additions" - header + the ADR-0004–0024 enumeration (archaeology — the project is - past Phase 7 into the post-M / F-010 / F-011 quality loop); the - Linux-gate bullet and the Cycle-budget-defer paragraph condensed - against their canonical homes (ADR-0049, `principle.md`). -7. **`java_cljw_surface_layout.md` deleted** — a redundant pointer stub - whose `paths` (`src/runtime/java/**`, `src/runtime/cljw/**`) are - already subsumed by `feature_name_consistency.md`'s `src/runtime/**`, - so a java/cljw-surface edit loaded both. The canonical content lives - in `feature_name_consistency.md`. - -## Consequences - -- Always-on floor drops ~3.3K tokens/turn (gate_cadence frontmatter - ~0.9K + orphan_prevention trim ~1.8K + CLAUDE.md condensations - ~0.6K), plus the MCP server-instruction blocks (several K) once the - settings reload / restart takes effect. -- 200K window forces the loop to lean on handover.md as the durable - state — exactly the design the resume contract + `print_handover_brief.sh` - already serve. Frequent auto-compact is now the expected operating - mode, not an exception. -- **Verification required** (settings reload / restart): confirm the six - MCP servers no longer load (their tool lists + instruction blocks - vanish), and that `enabledPlugins: false` at project scope is honored - (the key is undocumented in the settings reference but is the same key - the global settings use; project overrides user). Fallback if not - honored: the `/plugin` menu. `disabledMcpServers` as a plain-string - array mirrors the per-project `~/.claude.json` practice; if ignored, - datadog stays and needs the `/mcp` UI toggle instead. -- No load-bearing rule was cut: the smell-audit / gate-cadence / - provisional-marker / dual-backend / orphan-`timeout` invariants and - the F-NNN priority chain are intact. - -## Alternatives considered - -This is a **user-directed** change; the user supplied the external check -(approving the 200K cap and the orphan_prevention trim depth via an -in-chat decision, and directing the MCP disable + PostCompact -investigation), so the autonomous-loop Devil's-advocate fork (CLAUDE.md -§ ADR-level designs) does not apply. - -- **Keep the 1M window** (rejected by the user): longer single sessions - but late, expensive, rare compaction and unbounded bloat. -- **1M + run-time `/compact` discipline** (rejected): relies on the loop - remembering to compact; the 200K cap makes it structural. -- **Trim orphan_prevention to ~25 lines / leave it untouched** - (rejected both): the ~58-line core keeps both rules + caveat + - counter-examples readable in one screen without re-deriving from - ADR-0049, while still shedding the narrative bulk. -- **Disable MCP globally** (rejected by the user): would affect every - other project; project-scoped override was required. - -## Affected files - -- `.claude/settings.json` — `model`, `enabledPlugins`, - `disabledMcpServers`, `SessionStart` / `PostCompact` matchers. -- `.claude/rules/gate_cadence.md` — added `paths:` frontmatter. -- `.claude/rules/orphan_prevention.md` — trimmed to core. -- `.claude/rules/dual_backend_parity.md` — dropped the one-time - `0cd92fa` sweep-result table. -- `.claude/rules/java_cljw_surface_layout.md` — deleted. -- `CLAUDE.md` — Data-sources header + ADR enumeration, Linux-gate - bullet, Cycle-budget-defer paragraph condensed. diff --git a/.dev/decisions/0063_compact_range_value.md b/.dev/decisions/0063_compact_range_value.md deleted file mode 100644 index 973fa827b..000000000 --- a/.dev/decisions/0063_compact_range_value.md +++ /dev/null @@ -1,283 +0,0 @@ -# ADR-0063 — Activate the reserved `.range` value (compact LongRange) for range performance - -**Status**: Proposed → Accepted (2026-05-31) - -**Context owner**: autonomous loop (user-directed perf pull-forward) - -## Context - -D-168 made `(range n)` / `(range a b)` lazy cons-seqs (correct, replacing -an eager vector that returned the wrong type). Correct, but each element -costs a cons + lazy_seq allocation, so `(count (range N))` ≈ 118 µs/element -and reduce/into/count over large ranges are pathologically slow (D-163). -The full Mac gate sits at ~244 s (borderline under the 300 s cap; load -variance tips it to timeout) largely because range-heavy e2e steps slowed. - -The user directed pulling the performance finished-form forward: -*"あからさまに遅いと数々のこれからのイテレーションのボトルネックになる -ので、 前倒しでもやっておく価値があります"* (2026-05-31), having earlier -authorised speed-ups *with clear markers* and asked that cw v0's approach -be investigated. - -cljw's value model **already reserves** `.range = 12` (Group A slot A12, -per F-004) — the tag round-trips through `encodeHeapPtr`/`decodePtr` and is -listed in `coll?`/`seq?`/`sequential?`/`conjOne`, but no struct, producer, -or handler exists (vestigial, like `.cons`). This ADR activates it. - -cw v0 (`ClojureWasm/src/lang/builtins/sequences.zig`) solves range perf with -(1) a compact range descriptor and (2) `fusedReduce` collapsing -map/filter/take chains to a 0-alloc pass (measured 1336x on lazy_chain). -This ADR takes only (1); the fusion (2) is increment 2 — see Non-goals. - -Survey: `private/notes/perf-range-value-survey.md`. - -## Decision - -**Activate `.range` (A12) as a compact value; its `seq` materialises a -chunked_cons (JVM `LongRange` → `LongChunk` shape). Increment 1 = the -compact value; map/filter reduce-fusion = increment 2 (deferred).** - -1. **Value.** `runtime/collection/range.zig` defines - `LongRange extern struct { header: HeapHeader; start: i64; end: i64; - step: i64; count: u32 }` (Cons/ChunkedCons template: HeapHeader at - offset 0, comptime align asserts). It owns no `Value` children, so its - GC trace is **null** (the `tag_trace_table` "no outgoing pointers" - case) — not a child-marking trace. - -2. **One iteration source (F-011).** `range.zig` is the single home for - range math: `count` (O(1), precomputed), `nth` (O(1), `start+i*step`), - `first` (O(1)), `reduceRange` (tight i64 loop honouring `reduced`), and - `seqChunk` (materialise ≤32 elements into a `chunked_cons` whose `next` - is the smaller `.range` or nil). count/nth/reduce/first shortcut the raw - `.range`; **generic traversal (`seq`/`rest`/`map`/`doseq`) goes through - `seqChunk` → the existing `chunked_cons` machinery**, paying one - allocation per 32 elements, not per element (the DA-A2 finding). - -3. **Producer.** A `-range` `:zig-leaf` (ADR-0033 D4 pattern) is the sole - producer. `core.clj`'s `range` 3-arg arm calls `-range` when start/end/ - step are all integers, step≠0, and the range is finite + non-empty; - otherwise it falls through to the existing lazy `.clj` cons body. - - `(range)` 0-arg stays `(iterate inc 0)`. - - Empty range → **nil** (cljw has no distinct empty-seq; D-164). The - producer never mints a count-0 `.range`. - - Float ranges, step=0 (infinite 0s), and bigint-bounded ranges → - lazy `.clj` fallback (the `.clj` body already handles any numeric via - `+`/`<`). These are F-005-owner territory (F-003 defer). - -4. **Call-site contract.** A `.range` arm is added to: `countFn` (O(1)), - `nthFn` (O(1), NOT the O(n) walk arm), `firstFn` (O(1)), `seqFn` - (→ `seqChunk`), `restFn`/`nextFn`/`seqNext`/`firstOfSeq`/`restOfSeq` - (→ `seqChunk` then the chunked_cons ops), `reduceFn` (tight loop), - `consFn` (range tail, via seq-view), `print` (deepRealize/printValue), - `equal` (isSequential + Cursor + isCountable). Predicates - (`coll?`/`seq?`/`sequential?`/`conjOne`) already list `.range`. *A - missed arm is a crash — the nth-on-lazy_seq class of bug just fixed in - D-168; the survey §4 checklist is the exhaustive list.* - -5. **Optimization governance (user-directed).** This is the first entry - in the new `.dev/optimizations.md` SSOT (O-001) and the first user of - the `// PERF:` marker convention - (`.claude/rules/perf_marker.md`): every speed-for-simplicity trade - carries a grep-discoverable marker + a ledger row recording the naive - contract it optimizes. - -## Non-goals (explicit) - -- **Map/filter/take reduce-fusion** (cw v0 `fusedReduce`). The compact - `.range` is the *substrate* this fusion operates over (DA-A2); fusion is - increment 2, a separate ADR in the D-163 perf window. Increment 1 must - NOT introduce meta-annotated lazy-seqs. -- **Bigint / float range** as a `.range` value (F-005-owner; lazy `.clj` - fallback handles them). -- **Fixing D-164** (empty≡nil), **D-165** (i48 element overflow → float), - or **D-178** (`.list`/`.cons` conflation). All three are inherited - unchanged — not fixed, not regressed. - -## Consequences - -- `count`/`nth`/`reduce`/`into`/`first` over a range become O(1) / tight - loop; generic `seq`-walk (map/doseq) pays 1 alloc / 32 elements (was 1 / - element under D-168's lazy-cons). The measured D-163 range pathology is - removed; the gate's range-heavy e2e get faster. -- The lazy `.clj` range body stays as the fallback + the source of truth - for float / infinite / bigint ranges. -- Range elements beyond ±2^47 silently become float (D-165, F-005-owner, - inherited). Range count beyond 2^47 likewise (rare; D-165 sibling). -- A new `runtime/collection/range.zig` + the §4 call-site arms; the - chunked-seq path reuses `chunked_cons` (no new chunk machinery). - -## Alternatives considered - -Sourced from a fresh-context Devil's-advocate subagent (mandatory at -depth ≥ 2), briefed with the F-002/F-004/F-005/F-011/F-003 envelope. Its -analysis is reproduced verbatim; the leading recommendation (Alt 2) is -the shape this ADR adopts. - -### DA verdict on the draft - -> The draft is F-NNN-compliant and is **not** the smallest-diff trap it -> could have been. Two things specifically clear it: it activates the -> *already-reserved* `.range` slot (F-004 fixes range as a standalone -> Group A slot, so a dedicated struct is the F-004-decreed shape, not a -> free design choice), and it routes count/nth/reduce/print/equal through -> one `range.zig` math module (F-011 commonisation). The Devil's -> advocate's job is therefore not "is the draft dirty" but "is the -> draft's *increment boundary* (unchunked start/end/step, lazy `.clj` -> fallback, per-step `.range` re-mint) the finished form, or a way-station -> the finished-form owner would unwind". On three of those sub-questions -> it is a way-station — and one (d, the per-step re-mint) is a latent -> performance cliff the draft's own perf goal does not survive. -> -> - **(d) is the sharp one.** `restNext` minting a fresh `.range` per -> element means a *generic walker* — `(map f (range n))`, -> `(doseq [x (range n)] …)`, `seqFn`-driven traversal — pays one -> `.range` allocation per step, exactly the cost profile D-168's -> lazy-cons range already has. The draft's O(1) win lands *only* on the -> call sites that special-case `.range` (count/nth/reduce). The ADR's -> own consequence ("range-using code gets fast") overclaims. -> - **(a) chunked_cons already pays for itself.** It already ships -> O(1)-ish `count`, `first`, and a `rest` that only re-allocates at -> chunk boundaries (every 32 elements). A range *is* the canonical -> chunked producer in JVM (`LongRange` → `LongChunk`). A second -> iteration vtable in `range.zig` when `chunked_cons` is the F-004 -> neighbour built for this is an F-011 tension. -> - **(e) no hard F-004/F-011 violation**, but the "ONE source" claim is -> weaker than it reads: the draft creates a *second* seq-iteration -> mechanism parallel to `chunked_cons`. - -### Alt 1 — Smallest-diff: keep lazy-cons range, add reduce/count/nth fast-paths only - -> Keep D-168's lazy-cons range; special-case the consumers (count/nth/ -> reduce) via a meta-annotation or an opaque marker. **Better:** smallest -> call-site surface (3 arms, no "missed arm = crash" class). **Breaks:** -> either reintroduces meta-annotated lazy-seqs (forbidden in increment 1) -> or smuggles a half-readable `.range` (the "5 types in one slot" -> anti-pattern F-004 forbids); the generic-walker cost (d) is unchanged. -> **F-NNN:** borderline F-004, fails F-002 (the smaller-diff path the -> finished-form owner unwinds). Cycle-budget-defer smell. **Not -> recommended.** - -### Alt 2 — Finished-form-clean: `.range` value whose `seq` produces a chunked_cons (RECOMMENDED) - -> Activate `.range` (compact start/end/step, O(1) count/nth, tight -> reduce) — but `seq`/`rest` of a `.range`, instead of re-minting a -> `.range`, materialises a 32-element `chunked_cons` lazily, JVM -> `LongRange`→`LongChunk` style. count/nth/reduce hit `.range` directly -> (O(1)); *generic* traversal goes `.range` → `chunked_cons` at one -> allocation per 32 elements (fixes d). The seq-family call sites mostly -> **already** route `.chunked_cons`, so the new-arm surface *shrinks* -> relative to the draft — many sites need a `.range` arm only for the -> O(1) fast path and otherwise fall through to the present `.chunked_cons` -> arm via a one-line `seq`-coercion. **Better:** makes the perf win real -> for *every* range consumer, not just the three that special-case -> `.range`; honours F-011 *across the seq family* (range is a producer -> feeding the one chunk mechanism, not a parallel mechanism); lays the -> runway for increment-2 fusion (the chunk is where fused reduce -> operates). **Breaks:** larger diff (chunk-materialisation path, GC -> interplay between `.range` and the `.chunk_buffer` it spawns); the -> empty-range/D-164 handling now has two exits that must agree. Per F-002, -> diff size is not a reason to prefer the draft — recommend anyway. -> **F-NNN:** fully compliant; the *stronger* F-011 reading. **This is the -> Devil's-advocate recommended shape**, on the strength of (d): the -> draft's per-step re-mint undermines its own perf goal for the most -> common traversal, and the chunked-seq path is the finished form JVM -> itself ships. - -### Alt 3 — Wildcard: all-Zig range + co-activate `.string_seq`/`.array_seq` behind a shared seq-producer vtable - -> Move all of range (incl. float/step-0/bigint) into Zig, and activate -> the three adjacent Group A seq-family slots together behind one -> `SeqProducer` shape. **Better:** kills the producer-leaf/`.clj`-fallback -> seam; amortises the ~14-arm call-site surgery once across three types -> instead of three times. **Breaks:** scope explosion against F-003 -> (`.string_seq`/`.array_seq` are owned by their own §9.x rows; no F-NNN -> fixes "activate the seq family together"); bigint-in-Zig range bounds -> are **F-005-owner territory** (the loop pre-empting the F-005 owner); a -> premature shared vtable risks the ROADMAP §13 `pub var` vtable ban. -> **F-NNN: partially violates F-003 and trespasses F-005's reserved -> decision** — recorded as a finding, not a recommendation. The defensible -> fragment (bigint/float in Zig) is left to the F-005 owner; the -> seq-family co-activation becomes a debt row at the `.string_seq` owning -> row, reusing Alt-2's chunked-seq substrate so the third surgery is a -> one-arm fall-through. - -### DA probe answers (verbatim) - -> - **(a)** Both — `.range` for the O(1) descriptor (F-004-decreed), -> `chunked_cons` as what its `seq` *produces*. Not "instead of"; "as a -> producer feeding". -> - **(b)** The *value* stays unchunked start/end/step (O(1) nth); its -> *seq* should be chunked from day one (Alt-2). The draft chunks -> neither, which is why (d) bites. -> - **(c)** Keep the producer-leaf + `.clj`-fallback split for this -> increment — the fallback cases (float/infinite/bigint) are genuinely -> F-005-owner territory; moving them to Zig now trespasses F-005. The -> seam is the F-005 deferral made concrete, not smallest-diff bias. -> - **(d) YES — strongest finding.** Per-step `.range` re-mint allocates -> per element for generic consumers; `seq` of a range should produce a -> `chunked_cons`. -> - **(e)** No hard violation; Alt-2 is the stronger F-011 reading (one -> chunk mechanism, range is a producer). - -### Main-loop decision - -Adopt **Alt 2**. Per CLAUDE.md's Cycle-budget-defer rule, the DA rating -Alt 2 finished-form-clean within the F-NNN envelope, with no F-NNN block, -means the larger-diff finished form is taken over the simpler draft. The -loop's earlier instinct toward the draft (generic-walk perf "belongs to -increment-2 fusion") was answered by the DA: the chunked-seq is the -*substrate* fusion operates over, not redundant with it. Alt 3's -seq-family co-activation insight is recorded as forward debt at the -`.string_seq`/`.array_seq` owning row (reuse Alt-2's chunked substrate). - -## Affected files - -- New: `src/runtime/collection/range.zig` (the value + iteration math + - `seqChunk`), `.dev/optimizations.md` (O-001), - `.claude/rules/perf_marker.md`. -- `src/runtime/value/value.zig` (register `.range` GC trace = null / - decode), `src/runtime/gc/tag_ops.zig` (no trace, document the null), - `src/runtime/runtime.zig` (registerGcHooks call if any). -- `src/lang/primitive/sequence.zig` (count/seq/first/rest/next/seqNext/ - firstOfSeq/restOfSeq + reduce path), `src/lang/primitive/collection.zig` - (nthFn / consFn), `src/lang/primitive/higher_order.zig` (`-range` leaf + - reduceFn `.range` arm), `src/lang/clj/clojure/core.clj` (range 3-arg - mints `.range`). -- `src/runtime/print.zig` (`.range` print), `src/runtime/equal.zig` - (`.range` sequential/cursor/countable). -- Tests: `test/e2e/phase14_range_indexed.sh` (extend) + a diff-oracle - case; `range.zig` unit tests. -- Debt: D-163 (perf blueprint already recorded), a new row noting - `.range` inherits D-165 for large elements + the Alt-3 seq-family - co-activation forward debt. - -## Cross-references - -F-002 / F-004 (A12 reservation) / F-005 (numeric tower) / F-011 -(commonisation) / F-003 (structural deferral scope boundary) · -D-163 (perf window + cw v0 fusedReduce blueprint) · D-168 (range→lazy -seq, the immediate predecessor) · D-164 / D-165 / D-178 (inherited) · -ADR-0033 D4 (`:zig-leaf` producer pattern) · survey -`private/notes/perf-range-value-survey.md`. - -## Revision history - -- **2026-05-31 — formalised into a ROADMAP perf campaign.** The user - extended the one-off range pull-forward into a sustained, ROI-ordered - speed-tuning campaign (*"ROI の高いものから自律的に判断して進め、 手戻りも - していい … 正式な計画に組み込みましょう … クリアセッションからまっさきに - 腰を据えてやってくれるよう配線"*). This ADR's governance (the `// PERF:` - marker + `.dev/optimizations.md` SSOT) now covers the whole campaign, - recorded as **ROADMAP §9.2.S Performance tuning campaign** (ACTIVE - resume target, ahead of the §9.2.R Phase-15/JIT sequence; F-003: no - renumber). Landed since this ADR: **O-002** (reduce index-walks a - vector, `0898ba2c`) and **O-003 / D-180** (bulk `vector.fromSlice` — - `toPersistent` builds the HAMT trie from the transient's flat buffer - in O(n) instead of N persistent conjs; `into`/`vec` route editable - targets through transients; `(count (vec (range 1e6)))` 121s → 2.4s). - NEXT: **D-163** (map/filter/take reduce-fusion — its own ADR) then - **D-140** (startup bootstrap cache). The campaign carries its own - focused units because D-180 touched the core Vector type (exhaustive - boundary tests at n ∈ {0,1,31,32,33,63,64,65,1023,1024,1025,1e5}) — - a deliberate "腰を据えて" treatment rather than a rushed mega-session - addition. diff --git a/.dev/decisions/0064_transient_hash_map.md b/.dev/decisions/0064_transient_hash_map.md deleted file mode 100644 index 885e41f26..000000000 --- a/.dev/decisions/0064_transient_hash_map.md +++ /dev/null @@ -1,307 +0,0 @@ -# ADR-0064 — Transient hash map handles > 8 entries via persistent-HAMT delegation (D-045 transient half) - -**Status**: Proposed → Accepted (2026-05-31) - -**Context owner**: autonomous loop (user-directed root-cause fix) - -## Context - -The §9.2.S perf campaign's O-003 work routed `into` / `vec` through -transients (`(persistent! (reduce conj! (transient to) from))`) for the -vector win (D-180, measured 50x). Routing `into {}` / `into #{}` through -transients surfaced a latent gap: cljw's **transient map caps at 8 -entries**. - -`src/runtime/collection/transient/transient_array_map.zig` was a flat -`[2*8]Value` buffer; `assoc!` / `conj!` past `ARRAY_MAP_THRESHOLD` (8) -raised `feature_not_supported` ("transient assoc! beyond ArrayMap -capacity (transient hash_map pending)"), and `(transient )` -raised `error.HashMapNotImplemented`. So `(into {} (map … (range 20)))` -ERRORED at the 9th entry — caught by `phase14_hamt_map` / -`phase14_ifn_callable` / `phase14_vector_keys` / `phase14_metadata`. - -The **persistent** side is complete: `map.assoc` promotes `.array_map` -→ `.hash_map` (CHAMP HAMT) at > 8 and dedups correctly; `(reduce conj -{} pairs)` builds a correct large map. Only the transient half (D-045's -transient component) was missing. The user directed fixing this at the -root (rather than narrowing `-editable?` to vectors), with upstream / -ClojureScript / cw v0 / web research (survey: -`private/notes/phase9.2.S-D045-transient-hashmap-survey.md`). - -## Decision - -**Adopt Alt 1 — persistent-HAMT delegation past the threshold.** The -`TransientArrayMap` gains an `overflow: Value` field: - -- **Flat mode** (`overflow == nil`, count ≤ 8): unchanged — flat - `entries` buffer, `assoc!` replaces-in-place on a key hit / appends on - a new key. -- **Hash mode** (`overflow` holds a persistent `.hash_map`): on the 9th - *distinct* key, the 8 flat pairs + the new key are replayed through - `map.assoc` into a persistent HAMT held in `overflow`; further - `assoc!` / `dissoc!` delegate to persistent `map.assoc` / `map.dissoc` - (which dedup + promote for free). `persistent!` returns `overflow` - directly (re-applying `meta`). `(transient )` seeds hash - mode by holding the source map directly (`map.assoc` is - copy-on-write, so the source is never mutated). - -This **reuses the one proven mechanism** (the persistent CHAMP HAMT), so -`persistent!` produces the *identical* map `(reduce conj {} …)` produces -— behavioural equivalence (F-011) is trivially guaranteed with zero new -hash structure and zero `extern struct` layout change. The transient -hash **set** (`transient_hash_set.zig`) wraps a `TransientArrayMap` as -its `inner_map`, so it is fixed transitively (`(into #{} big)` / -`(transient )` now work). - -**Hash-mode `assoc!` is persistent-path O(log32 n) copy-on-write, NOT an -in-place editable-CHAMP transient.** It is correct but delivers none of -the in-place allocation win a true transient gives. That speed gap is -tracked as **D-181** (the editable-CHAMP HAMT with a ClojureScript-style -edit token is the named finished-form candidate), deferred because maps -were never a measured §9.2.S bottleneck (the ROI there was vectors). The -delegation produces a correct HAMT — it is not a permanent-no-op / silent -lie; only the speed is provisional, and it is debt-tracked. - -## Non-goals (explicit) - -- **In-place editable-CHAMP transient** (Alt 3 — the JVM/cljs finished - form). Deferred to D-181; would touch the `HamtMapNode` `extern` - layout (F-004) for an unmeasured map win. -- **A parallel open-addressing side index + `map.fromSlice`** (Alt 2, - the survey's pick). Rejected — see Alternatives; its `map.fromSlice` - is not an O(n) bulk build (CHAMP slot placement is hash-driven, no - pre-grouping), so it pays for a second hash mechanism (F-011 tension) - to buy only a constant-factor allocation reduction. -- **HAMT → ArrayMap demotion** on `dissoc!` below 8 — the persistent - side has none (D-156), so the transient matches: once in hash mode it - stays hash. - -## Consequences - -- `(into {} big)` / `(into #{} big)` / `(transient )` / - `(persistent! (reduce conj! (transient {}) big))` now work (were - errors). The 4 e2e tests pass; diff oracle vs `clj` green at scale - (100-entry maps/sets, `=` vs `zipmap`/`set`, sorted maps/sets, meta - carry-through, `into nil` → list). -- Map/set print order vs `clj` stays the pre-existing accepted HAMT - iteration-order divergence (the transient path produces the *identical* - cljw order as the persistent path — verified — so no new divergence). -- `dissoc` gains an `rt` parameter (the hash-mode delegation needs it); - `transient_hash_set.disj` + the `dissoc!` / `disj!` primitives thread - it through. -- `meta` is now carried through `persistent!` (the old flat `toPersistent` - silently dropped it — a latent bug fixed here). -- Map-transient *speed* is unchanged (= persistent conj). D-181 owns the - in-place finished form. - -## Alternatives considered - -Sourced from a fresh-context Devil's-advocate subagent (mandatory at -depth ≥ 2), briefed with the F-002 / F-004 / F-006 / F-011 envelope. Its -analysis is reproduced verbatim; the leading recommendation (Alt 1) is -the shape this ADR adopts. - -> # Devil's-advocate analysis — Transient hash map (D-045 transient half) -> -> ## Grounding confirmed against actual source -> -> - `map.assoc` already does the full lifecycle: `assocArrayMap` -> replaces-in-place on key hit, copy-appends below 8, and calls -> `promoteArrayMap` on the 9th distinct key, producing a correct -> `.hash_map`. **The persistent side is complete and correct.** -> - `promoteArrayMap` is itself an O(n) "replay flat pairs into a fresh -> HAMT root via `hamtAssoc`" loop. A `map.fromSlice` bulk-builder would -> be this exact loop generalised to n pairs — there is no smarter -> bottom-up CHAMP build analogous to `vector.fromSlice` (vectors get a -> dense left-packed trie; a CHAMP node's slot placement is -> hash-bitmap-driven, so you cannot pre-group — you must `hamtAssoc` -> entry by entry regardless). This is a critical asymmetry the survey's -> "map twin of `vector.fromSlice`" framing **overstates**: -> `vector.fromSlice` is genuinely O(n) with cheap `@memcpy`-of-32 -> grouping; a map "fromSlice" is N × `hamtAssoc` = O(n·log32 n) of -> path-copying allocation, no cheaper than calling `map.assoc` N times. -> The map has no O(n) bulk-build shortcut today. -> - The `transient` primitive routes `.hash_map` source → -> `transient_array_map.fromMap` → `error.HashMapNotImplemented`. So -> `(transient some-big-map)` errors today regardless of `into`. -> - `HamtMapNode` is `extern struct { header, _pad[2], data_map:u32, -> node_map:u32, slots:[64]Value }` with `comptime` layout asserts and a -> dedicated GC trace. Adding an `edit` field is a real F-004 layout -> change touching the persistent node every map in the system allocates. -> -> ## Alt 1 — persistent-wrap past threshold -> -> - **Better:** Reuses the one proven, already-tested mechanism -> (`map.assoc`'s array→HAMT promotion + dedup). Zero new subsystem, zero -> layout change, zero F-004 exposure. Correctly dedups for free. The -> transient's `assoc!`-past-8 mirrors *exactly* what `(reduce conj {} -> pairs)` already does correctly. Smallest correctness surface to audit. -> - **Breaks / costs:** No speed win for maps. Per-`assoc!` past 8 is -> O(log32 n) path-copy with allocation of a fresh node chain per call. -> `into {}` of n keys = O(n·log32 n) time and allocations — same as the -> persistent fallback. Needs a discriminator field on the transient. -> - **Big-O:** `assoc!` O(log32 n) amortised (post-threshold); `into {} -> n` = O(n log n). `persistent!` O(1) (return `current`). -> - **F-NNN:** Fully compliant. No F-004 layout touch. F-011 **strongest -> of the three** — one mechanism (the CHAMP HAMT), no parallel hash -> structure, so the diff oracle contract is trivially met. -> -> ## Alt 2 — flat growable buffer + open-addressing side hash index + `map.fromSlice` (survey's pick) -> -> - **Better:** Genuinely O(1)-amortised `assoc!` (hash probe + append), -> deferring HAMT construction to one `persistent!` pass. Matches the -> transient_vector lifecycle shape superficially. No persistent-node -> layout change. -> - **Breaks / costs:** Introduces a **novel open-addressing hash table -> subsystem** that exists nowhere else in cljw — a second -> hashing/collision-resolution mechanism living *parallel to the CHAMP -> HAMT*. That is precisely the F-011 "one mechanism, not parallel -> copies" tension. The side index needs its own collision handling, -> resize policy, `valueHash`/`keyEqValue` integration, and its own -> correctness audit against the String value-eq rule (D-151). And the -> payoff is undercut: **`map.fromSlice` is NOT an O(n) bulk build** — it -> is N × `hamtAssoc`, so `persistent!` is O(n·log32 n) with n allocs -> anyway. Alt 2 moves the log-n cost from `assoc!` to `persistent!` but -> does not eliminate it; it only removes intermediate copy-on-write -> garbage (a constant-factor allocation win, not a complexity-class win). -> - **Big-O:** `assoc!` O(1) amortised; `into {} n` = O(n) index ops + O(n -> log n) at persistent!. Honest total O(n log n) like Alt 1, fewer -> intermediate allocations. -> - **F-NNN:** F-011 the weak point — a second hash mechanism parallel to -> CHAMP. -> -> **Answer to (a) — is the survey's "vector consistency" argument sound?** -> **No, it is the weakest argument in the survey.** The vector's -> flat-buffer-then-`fromSlice` shape is finished-form *because vectors -> need no dedup* — the flat buffer maps 1:1 to trie leaves and -> `fromSlice` is a true O(n) `@memcpy` grouping. The map breaks both -> halves: (1) it needs dedup, forcing the novel side index the vector -> never had; (2) its `fromSlice` is not an O(n) bulk build. So "vector -> consistency" is *surface* consistency hiding *mechanism* divergence (a -> whole open-addressing subsystem + a fake-O(n) builder). A finished-form -> owner would not want a second hash structure parallel to the CHAMP. -> -> ## Alt 3 — editable CHAMP HAMT with cljs-style edit token -> -> - **Better:** The only alternative with genuinely O(1) `persistent!` -> AND no per-`assoc!` full-path-copy garbage — true in-place mutation of -> nodes minted during this session. What JVM and ClojureScript both -> ship; cljs proves it needs no threads (edit token = a freshly-minted -> sentinel compared by identity, fitting cljw's single-threaded, -> no-`pub var` constraint). One hash mechanism — best F-011 story. -> - **Breaks / costs:** Touches the `HamtMapNode` `extern struct` layout -> (add `edit: ?*anyopaque` + adjust `_pad`, re-verify `comptime` -> asserts, extend GC trace) — a real F-004 change rippling to every -> persistent map allocation (an extra word per node, forever, for a -> transient-only feature). Requires a parallel editable-insert variant -> of `hamtAssoc`/`insertDataEntry`/`pushDownDataToNode` with an -> `ensureEditable(edit)` branch. Largest implementation surface. cljw -> also has no HAMT→ArrayMap demotion and an unfinished collision bucket -> (D-155) the editable HAMT inherits. -> - **Big-O:** `assoc!` O(log32 n) but in-place (no alloc when owned). -> `into {} n` = O(n log n), near-zero intermediate allocation. -> `persistent!` O(1). -> - **F-NNN:** F-004 — the one alternative that touches a load-bearing -> layout. F-011 strongest (single mechanism). -> -> ## The crux — is Alt 1 finished-form-clean, or the permanent-no-op / smallest-diff smell? -> -> **Alt 1 is finished-form-clean for the correctness layer, and is NOT a -> permanent no-op. But it is NOT the finished form for transient-map -> *performance* — and that distinction must be made explicit and tracked, -> or it silently becomes the smallest-diff smell.** -> -> 1. **Permanent-no-op test:** Alt 1 lands in the "function body that runs -> with intermediate semantics" row, NOT the forbidden "returns input -> unchanged / masks missing feature" row. It produces a *fully correct* -> `PersistentHashMap` — `(persistent! (into-via-transient {} big))` -> equals `(reduce conj {} big)` byte-for-byte because it literally -> calls the same `map.assoc`. The user does not see "success while -> semantics are dropped". **Clears the permanent-no-op bar cleanly.** -> 2. **Is "a transient not faster than persistent" itself a smell?** If -> shipped *and declared finished*, yes — "transient" would be hollow. -> The honest framing: Alt 1 is the finished form of *correctness* and -> an explicitly-provisional form of *performance*. With the speed gap -> tracked as debt (Alt 3 named), it is a tracked bootstrap layer, not -> Silent-default-shift. -> 3. **Deferring map-transient SPEED is finished-form-correct, not a -> dodge:** the perf campaign's measured ROI was vectors, not maps. Maps -> were never a measured bottleneck. Building Alt 2's subsystem or Alt -> 3's layout change *now*, for an unmeasured map-`into`, is -> smell-adjacent speculative mechanism. The finished-form-clean -> sequence: (a) unblock correctness with the one proven mechanism now -> (Alt 1), (b) file a debt row naming Alt 3 as the leading candidate, -> picked up when maps are measured to matter. -> -> **Answers:** (b) Alt 1 clears the permanent-no-op bar; finished-form for -> correctness, not for speed — mark it. (c) the perf campaign does NOT -> justify Alt 2/3 now; correctness (Alt 1) is the right scope. -> -> **(d) Alt 1 correctness traps (tested):** -> - Transition trigger = "new distinct key pushing count > 8", NOT "9th -> `assoc!` call". A replace of an existing key while full stays flat. -> - On promotion, seed `current` by replaying the 8 flat pairs through -> `map.assoc` then assoc the 9th. -> - `dissoc!` below 8 must NOT demote (persistent side has no -> HAMT→ArrayMap demotion); stay hash, `persistent!` yields `.hash_map`. -> - `conj!` of a `[k v]` pair routes through `assoc!`. -> - `meta` carry-through on `persistent!`. -> - `transient` of a `.hash_map` source: seed `overflow` with the source -> directly (removes the `HashMapNotImplemented` site). -> -> ## Recommendation -> -> **Recommend Alt 1 (persistent-wrap) for this cycle, with a mandatory -> debt row naming Alt 3 (editable CHAMP HAMT) as the finished-form speed -> target.** This is NOT a cycle-budget-defer downgrade: after grounding in -> the code, **Alt 2's central premise is unsound** — its `map.fromSlice` -> is not an O(n) bulk build, so it pays for a novel parallel -> open-addressing subsystem (F-011 liability) to buy only a constant-factor -> allocation reduction. Alt 2 is the worst trade. Between Alt 1 and Alt 3: -> Alt 3 is the true finished form, but maps are not a measured bottleneck, -> so paying an `HamtMapNode` layout change now is speculative. Unblock -> correctness with the single proven mechanism (Alt 1); let the editable -> HAMT be its own ROI unit (D-181). **Strongest single reason:** Alt 1 -> makes `persistent!` produce the identical HAMT `(reduce conj {} …)` -> produces, so F-011 equivalence is trivially guaranteed with zero new -> hash mechanism and zero layout risk — while the speed gap is honestly -> tracked. If map-`into` performance *is* a near-term target, F-002 flips -> the recommendation to Alt 3 outright (skip Alt 2 — its parallel hash -> index is what a finished-form owner would unwind). - -### Main-loop decision - -Adopt **Alt 1**. The user's directive was a *root-cause correctness* fix -("根本的に直して") for the transient map's 8-entry cap — not a map-perf -optimization. Alt 1 fixes that root (large maps/sets work through -transients, producing correct HAMTs identical to the persistent path) -with the one proven mechanism and zero layout risk. The DA's debunking of -Alt 2 (no O(n) map bulk-build; a parallel hash subsystem is an F-011 -liability) is decisive against the survey's pick. Alt 3 (the true -in-place finished form) is deferred to D-181 because maps were never a -measured §9.2.S bottleneck; building a per-node layout change now is -speculative. The speed gap is debt-tracked (D-181, Alt 3 named), so Alt 1 -is a tracked bootstrap layer, not a Silent-default-shift smell. - -## Affected files - -- `src/runtime/collection/transient/transient_array_map.zig` (the - `overflow` field + hash-mode `assoc`/`dissoc`/`toPersistent`/`fromMap` - + meta carry-through + GC trace; `dissoc` gains `rt`). -- `src/runtime/collection/transient/transient_hash_set.zig` (`disj` - gains `rt` — its `inner_map` is a `TransientArrayMap`, fixed - transitively). -- `src/lang/primitive/transient.zig` (`dissoc!` / `disj!` thread `rt`; - the `.hash_map`-source comment updated). -- Tests: new transient-map boundary tests (promotion, replace-in-place - stays flat, dissoc-below-8 stays hash, `.hash_map`-source seeding, meta - carry-through). -- Debt: D-181 (transient map in-place speed; Alt 3 = editable CHAMP). - -## Cross-references - -F-002 / F-004 / F-006 / F-011 · D-045 (HAMT body — this is the transient -half) · D-156 (no HAMT→ArrayMap demotion) · D-155 (collision bucket stub) -· D-181 (in-place editable-CHAMP speed, deferred) · ADR-0063 / O-003 / -D-180 (the §9.2.S into/vec transient routing that surfaced this) · survey -`private/notes/phase9.2.S-D045-transient-hashmap-survey.md`. diff --git a/.dev/decisions/0065_reduce_fusion_via_transducers_chunked.md b/.dev/decisions/0065_reduce_fusion_via_transducers_chunked.md deleted file mode 100644 index e511ae3e9..000000000 --- a/.dev/decisions/0065_reduce_fusion_via_transducers_chunked.md +++ /dev/null @@ -1,311 +0,0 @@ -# ADR-0065 — Lazy-chain reduce performance via chunked seqs + transducers (NOT cw-v0 meta-fuse); D-163 - -**Status**: Proposed → Accepted (2026-05-31) - -**Context owner**: autonomous loop (§9.2.S perf campaign, D-163) - -## Context - -`(count (map inc (range 1e5)))` ≈ 42s ≈ 420µs/element: each lazy_seq -element re-evaluates the `map` body through tree-walk (`(lazy-seq (cons -(f (first s)) (map f (rest s))))`) + allocates a Cons + lazy_seq node + -calls `f`, once per element. `reduceFn` has fast arms for `.range` -(O-001) and `.vector` (O-002) but the moment a `map`/`filter` wrapper -sits over the source, the chain drops to the generic per-element -`seqFn → firstFn → nextFn` walk. - -The D-163 debt row captured **cw v0's `fusedReduce`** as the blueprint: -meta-annotated lazy-seqs (`__zig-lazy-map`/`-filter`/`-take`) + a -meta-chain walk that fuses the chain into a 0-alloc pass (measured 1336× -on lazy_chain). That row predates cljw shipping its **complete, -JVM-shaped transducer machinery** (`map`/`filter`/`take`/`drop`/`keep`/ -`remove` 1-arg xform arities + `transduce`/`completing`/`cat`/`into`-with- -xform/`vec` — core.clj). Per CLAUDE.md, a debt row naming a future -approach is a *memo, not a contract*; this ADR re-lays it. - -Survey: `private/notes/phase9.2.S-D163-reduce-fusion-survey.md`. - -## Decision - -**Adopt the JVM finished form — chunked seqs for the implicit path + -transducers for the explicit 0-alloc opt-in — a single reducing-function -model. Reject cw-v0's meta-annotated `fusedReduce` (a second, parallel -fusion mechanism = F-011 violation). Land Alt C (chunk-aware map/filter + -a `chunked_cons` reduce arm) as the first increment of this finished -form.** - -- **Implicit path** `(reduce f (map g coll))` (user wrote no transducer): - `map`/`filter` PRESERVE the source's 32-element chunking (JVM - `chunk-cons` shape) — realise ≤32 elements per thunk into a - `ChunkBuffer`, emit a `chunked_cons` with a `lazy-seq` tail — and - `reduceFn` gains a `chunked_cons` arm that drains a whole chunk per - step. ~32× fewer thunk realisations. -- **Explicit path** `(transduce (map g) f coll)` / `(into [] (map g) - coll)`: the existing transducer arities already fuse with zero - intermediate seq allocation. No change needed; this is where the - 0-alloc win lives, by user opt-in (the JVM contract). - -There is **no cw-v0-style meta-fuse** — JVM has no such mechanism; cw v0 -invented it because it lacked the chunked-seq + transducer combination -cljw already has. - -### Realistic expectation (NOT overclaimed) - -Chunking collapses the per-thunk tree-walk + alloc term (~32× on that -term) but does **not** touch the per-element `f` vtable call. After -chunking, that per-element dispatch is the dominant residual — the same -`callFnVal` cost D-163 cross-refs to D-133 (JIT) / the superinstruction -pass, correctly **out of scope** for this fusion cycle. **Expected -overall: ~4-7× on the implicit path** (the timeout pathology becomes -~1-2s), not 32×. cw-v0 meta-fuse cannot fix the `f`-call term either, so -its marginal advantage over chunking is only ~10-15% (eliminating the -ChunkBuffer allocs) — not worth a permanent parallel mechanism. - -## First-cycle scope (one unit, gated on the headline benchmark) - -The `chunked_cons` reduce arm and chunk-aware `map`/`filter` **must land -together** — the reduce arm alone buys nothing because `map`'s current -`.clj` body re-wraps with plain `cons`, shredding chunks back to -per-element Cons cells (DA Trap 1). - -1. `reduceFn` `chunked_cons` arm: drain a whole `ChunkBuffer` per step - (between `.vector` and the generic arm), `reduced`-checked. `PERF:` - marker + O-NNN ledger row. -2. A `.clj`-callable chunk-builder surface backed by `chunked_cons.zig`: - `chunk-buffer` / `chunk-append` / `chunk` / `chunk-cons` / - `chunk-first` / `chunk-rest` / `chunked-seq?` (JVM-named). -3. Rewrite the 2-arg `map`/`filter`/`keep`/`remove` `.clj` bodies to - preserve chunking (realise ≤32 per thunk into a ChunkBuffer, emit a - `chunked_cons` with a `lazy-seq` tail) — JVM `chunk-cons` shape. -4. Gate on `(count (map inc (range 1e5)))` + `(reduce + (map inc (range - 1e5)))`; record the measured before/after in `optimizations.md`. - -Later increments (separate cycles, same mechanism — not throwaway): -route `into`/`sequence`/`eduction` so the explicit path is chunked -0-alloc too. No meta-fuse ever needed. - -## Non-goals (explicit) - -- **cw-v0 meta-annotated `fusedReduce`** — rejected (F-011; supersedes - the D-163 blueprint memo). -- **The per-element `f` vtable-dispatch cost** — that is D-133 / the - superinstruction window, not this cycle. -- **A `meta` union field on the LazySeq value** (Alt A would need it) — - not introduced. - -## Consequences - -- The implicit lazy path gets ~4-7× (timeout → ~1-2s); the explicit - transducer path stays the 0-alloc opt-in. Single mechanism (F-011). -- New `.clj`-callable chunk primitives (JVM-named) — real new surface, - but JVM-faithful and the substrate later increments reuse. -- `map`/`filter`/`keep`/`remove` 2-arg bodies become chunk-aware (their - observable lazy semantics unchanged — chunking is an internal - realisation detail, verified by the diff oracle). -- Composes with O-001 (`.range`→chunked) + O-002 (vector index-walk). - -## Alternatives considered - -Sourced from a fresh-context Devil's-advocate subagent (mandatory at -depth ≥ 2), briefed with the F-002 / F-011 / P3 / P4 / A6 envelope. Its -analysis is reproduced verbatim; the recommendation (Alt B finished form, -Alt C first increment, reject Alt A) is the shape this ADR adopts. - -> # Devil's-Advocate Analysis — D-163 lazy-chain reduce fusion -> -> ## Ground truth corrections to the framing -> -> 1. **`reduceFn` does NOT have a chunked_cons arm** — confirmed (L184 -> `.range`, L200 `.vector`, then the generic `seqFn→firstFn→nextFn` -> walk at L219). A `chunked_cons` falls to the generic per-element arm. -> 2. **`(rest )` already yields a chunked_cons** (sequence.zig -> L508/527 route `.range` through `seqChunk`). So the *source* side of -> a chain is already chunked. **But the `map`/`filter` clj bodies -> re-wrap with plain `cons`** (core.clj L53 `(cons (f (first s)) (map f -> (rest s)))`). `cons` builds a single Cons cell, not a chunk. So even -> though the source produces 32-wide chunks, the map output is a -> 1-element-Cons-per-thunk lazy chain. The chunk is shredded back to -> per-element at the map boundary. This means **Alt C's "chunk-aware -> map/filter" is NOT a small clj tweak** — it requires rewriting the -> map/filter bodies to allocate + fill a ChunkBuffer + emit a -> chunked_cons whose `next` is a `lazy-seq` thunk (JVM's -> `chunk-buffer`/`chunk-append`/`chunk-cons` shape), which today cljw -> has no `.clj`-callable surface for. -> -> ## Verdict per alternative -> -> ### Alt A — port cw-v0 meta-annotated `fusedReduce` -> - **Better:** Only design with **true 0-intermediate-allocation** for -> the implicit path (no per-element seq node AND no per-32 ChunkBuffer -> — a tight i64 loop over the `.range` base applying transforms inline). -> Proven 1336×. No clj call-site change. -> - **Breaks:** A **second fusion mechanism** parallel to the transducers -> cljw already ships, which must stay behaviourally identical to the -> transducer path forever (reduced, nil, chunk boundary, short-circuit) -> — two contracts for one behaviour. Reverses ADR-0054 (lazy-in-clj) by -> pulling map/filter 2-arg back into Zig `__zig-lazy-*` primitives. Adds -> a `meta` union field to the hot LazySeq value (F-004 surface). The -> 16-transform cap + filter-chain-collapse exist *only* for this -> mechanism. -> - **F-NNN:** **F-011 violation** (one mechanism). The decisive fact. The -> exact 89K-LOC duplication `no_copy_from_v1` prevents. -> - **Fixes implicit path?** Yes, best (0-alloc). -> -> ### Alt B — transducers + chunked seqs (JVM finished form) -> - **Better:** Single mechanism (F-011-clean). Matches JVM's actual -> answer. Explicit path routes through the existing transducer arities → -> 0-alloc by opt-in. Implicit path rides chunked seqs. Reuses O-001 -> `.range`→chunk + chunked_cons. -> - **Breaks:** Nothing structurally. The cost is chunk-preserving -> map/filter + a chunked_cons reduce arm (= Alt C, the first increment). -> - **F-NNN:** F-011/F-002/P3/P4-clean. -> - **Fixes implicit path?** Yes via chunking — **~32× constant factor, -> NOT 0-alloc** (still one ChunkBuffer + one ChunkedCons per 32 + calls -> `g` per element). -> -> ### Alt C — chunked_cons reduce arm + chunk-aware map/filter (first increment of B) -> - **Better:** Smallest finished-form-clean step. Lands the load-bearing -> `chunked_cons` reduce arm + chunk-preserving map/filter. Composes with -> O-001/O-002. No new value field, no parallel mechanism. -> - **Breaks:** Nothing — but "smallest diff" framing is **misleading**: -> chunk-aware map/filter requires a new chunk-builder surface callable -> from `.clj`. That is most of the work of B, not a tweak. Not a -> half-measure (a real increment of the same mechanism), but not cheap. -> - **F-NNN:** F-011/F-002-clean. -> - **Fixes implicit path?** Yes, ~32× on the amortised term. -> -> ## Explicit answers -> -> ### (a) Is Alt A (the debt-row blueprint) superseded? -> **Yes — superseded, and rejecting it is a sound F-011 call.** The D-163 -> row's blueprint predates cljw's complete transducer arities (the row is -> prior-art capture, and CLAUDE.md says debt rows naming future ADRs are -> memos, not contracts). Alt A would bolt a *second* fusion mechanism onto -> a problem the first already addresses — the textbook F-011 violation and -> the cw-v0 duplication `no_copy_from_v1` forbids. **Honest devil's point: -> Alt A buys true 0-allocation on the implicit path (B/C still allocate a -> ChunkBuffer + ChunkedCons per 32 — ~3,125 each for 1e5). But that win is -> recoverable inside the one mechanism via the *explicit* transducer -> surface (`(transduce (map inc) + (range n))`), which cljw already has -> and which IS 0-intermediate-alloc. Alt A's win is unique to fusion, and -> fusion is what transducers are. Verdict: superseded.** -> -> ### (b) Does B/C actually make the implicit path fast? Quantify. -> 420µs/element ≈ three additive costs: (1) per-thunk tree-walk of the -> map body (dominant), (2) per-element `f` vtable call, (3) per-element -> heap alloc. Chunked map/filter eliminates (1) + most of (3) by -> amortising over 32; it does **not** touch (2) — `f` is still called per -> element (Alt A also calls `f` per element). **Quantified residual:** if -> (1)≈300-350µs and (2)≈50-100µs, chunking collapses (1) ~32× to ~10µs and -> leaves (2). **Expected implicit-path: ~420µs → ~60-110µs/element, ~4-7× -> overall, NOT 32×.** Enough to convert the timeout to ~1-2s. The residual -> (2) is the `callFnVal`/dispatch cost D-163 cross-refs to D-133/ -> superinstruction — correctly out of scope. **Alt A would NOT fix (2) -> either**, so its true marginal advantage over C is ~10-15%, not worth an -> F-011 violation. -> -> ### (c) Is the finished form a HYBRID (chunked implicit + transducer explicit), no meta-fuse? -> **Yes — exactly the JVM-faithful finished form, no meta-fuse.** JVM: -> implicit `(reduce f (map g coll))` → `map` returns a chunked lazy-seq -> (`chunk-cons`), `reduce` drains via `IChunkedSeq`/`internal-reduce` -> 32-at-a-time (constant-factor, allocates chunks); explicit `(transduce -> (map g) f coll)` → no intermediate seq, 0-alloc. cljw has both -> substrates already. **There is no JVM mechanism corresponding to cw-v0's -> meta-fuse** — cw v0 invented it lacking the chunked+transducer combo. -> Confirmed: no meta-fuse needed. -> -> ### (d) Sequencing — "land Alt C first, measure" correct, or a trap? -> **Sequencing correct, but the "smallest diff" sizing is the trap.** -> Trap 1 — order within C: the `chunked_cons` reduce arm ALONE buys -> **nothing for the implicit `(map inc (range n))` path** because map -> shreds the chunk back to Cons cells. It only helps `(reduce f (range -> n))` over the seq-view + already-chunked sources. The load-bearing piece -> is **chunk-aware map/filter**. So "land the reduce arm first, measure" -> shows ~no improvement on the headline benchmark and misleads. Correct -> first cycle: **the reduce arm AND a `.clj`-callable chunk-builder -> surface AND chunk-aware map/filter together**, gated on the headline -> benchmark. Trap 2 — the chunk-builder surface is new public-ish API -> (`chunk-buffer`/`chunk-append`/`chunk`/`chunk-cons`/`chunk-first`/ -> `chunk-rest`); JVM-faithful but real new surface. Size the cycle -> honestly. Neither trap makes C wrong — both are increments of the one -> mechanism, nothing throwaway. -> -> ## Recommendation -> **Adopt Alt B as the finished form; land Alt C as its first increment; -> reject Alt A.** Strongest reason: cljw already ships the complete -> JVM-shaped transducer machinery AND the chunked `.range`/chunked_cons -> substrate, so the finished form is the JVM's own answer — chunked seqs -> (implicit) + transducers (explicit 0-alloc opt-in), a single -> reducing-function model. Alt A adds a second, permanently-co-maintained -> fusion mechanism for a problem this model already solves (F-011 -> violation, the cw-v0 89K-LOC pattern). Alt A's only genuine marginal win -> over C is ~10-15% (the residual chunk-alloc), and even Alt A cannot -> touch the dominant residual (the per-element vtable `f` call), which -> belongs to the later D-133/superinstruction window. **Alt A is -> unambiguously superseded by cljw having transducers.** First cycle = -> chunked_cons reduce arm + chunk-builder primitives + chunk-aware -> map/filter, gated on `(count (map inc (range 1e5)))`; sequence-1 without -> sequence-3 will not move the benchmark — they land together. - -### Main-loop decision - -Adopt **Alt B (finished form) / Alt C (first increment); reject Alt A.** -The DA confirms cljw's complete transducer machinery + chunked substrate -make cw-v0's meta-fuse a superseded F-011 violation. The realistic ~4-7× -(not the row's implied 1336×) is accepted honestly: the residual is the -per-element fn-dispatch cost owned by D-133/superinstruction, which no -fusion design in this cycle can remove. The first cycle lands the reduce -arm + chunk primitives + chunk-aware map/filter together (the reduce arm -alone is inert per DA Trap 1). - -## Implementation note (2026-05-31, measurement-validated — corrects the DA's residual estimate) - -Before implementing, the fill-loop design fork was resolved by measurement -(cljw is a **tree-walk interpreter**, so unlike JVM the `.clj` fill loop is -NOT compiled to bytecode — the question was whether a `.clj` `dotimes` fill -amortises the per-element lazy-seq machinery, or whether the fill itself -must be a Zig per-chunk primitive): - -- Pure tree-walk loop, 1e5 iters, multi-op body (`(loop [i 0 acc 0] (if (< - i 100000) (recur (inc i) (+ acc (inc i))) acc))`) = **0.71s** (~2.3µs/iter - after the 0.48s startup) → the per-element fn-call + loop overhead (the - DA's "term 2") is **~2µs, not 50-100µs**. -- `(count (map inc (range 1e5)))` = **41.3s** = ~408µs/element → **term 1 - (per-element lazy-seq machinery: lazy_seq + cons alloc + map-body - tree-walk + recursive map) is ~408µs and utterly dominates.** - -**Resolution:** the `.clj` `dotimes`/`loop` fill is fine — its ~2µs/iter is -dwarfed by the 408µs term-1 it amortises 32×. No Zig per-chunk fill -primitive is needed; the JVM `.clj` chunk-builder surface (thin Zig -`chunk-buffer`/`chunk-append`/`chunk-cons` + `-chunk-count`/`-chunk-nth`/ -`chunk-rest`/`chunked-seq?`) works for cljw. **Realistic win is ~25-30×** -(headline 42s → ~1.5s), NOT the DA's 4-7× — the DA overestimated term-2; -the residual after chunking is the amortised chunk overhead (~13µs/element) -+ the ~2µs fill step, not a 50-100µs fn-call. (The per-element `f` vtable -call genuinely is ~2µs and is NOT the bottleneck here; D-133/superinstruction -is for other hot paths, not this one.) High blast radius (map/filter are -used everywhere) → verify with the diff oracle across many seq scenarios -(empty / 1 / 31 / 32 / 33 / 1024 elements; nested map∘filter; lazy -infinite + take; nil; into/reduce/count/last/nth/seq= over the chunked -output) before gating. - -## Affected files (first cycle) - -- `src/lang/primitive/higher_order.zig` (`reduceFn` `chunked_cons` arm). -- `src/runtime/collection/chunked_cons.zig` + a primitive surface - (`chunk-buffer`/`chunk-append`/`chunk`/`chunk-cons`/`chunk-first`/ - `chunk-rest`/`chunked-seq?`) registered into `rt`. -- `src/lang/clj/clojure/core.clj` (`map`/`filter`/`keep`/`remove` 2-arg - bodies chunk-aware). -- `.dev/optimizations.md` (new O-NNN row), `.dev/debt.md` (D-163 blueprint - re-laid), tests (e2e + diff oracle on lazy chains; bench on the headline - benchmark). - -## Cross-references - -F-002 / F-011 / P3 / P4 / A6 · D-163 (re-laid: cw-v0 meta-fuse → JVM -transducer+chunked) · D-133 (JIT — owns the residual per-element dispatch) -· O-001 (`.range`→chunked, the substrate) · O-002 (vector index-walk) · -ADR-0054 (lazy-in-clj — Alt A would have reversed it) · ADR-0063 (§9.2.S -campaign governance) · survey -`private/notes/phase9.2.S-D163-reduce-fusion-survey.md`. diff --git a/.dev/decisions/0066_deftype_macro_conversion.md b/.dev/decisions/0066_deftype_macro_conversion.md deleted file mode 100644 index 2a55275a7..000000000 --- a/.dev/decisions/0066_deftype_macro_conversion.md +++ /dev/null @@ -1,171 +0,0 @@ -# ADR-0066 — `deftype` becomes a macro mirroring `defrecord` (retire the special form) - -- **Status**: Proposed → Accepted (2026-06-01) -- **Discharges**: D-087 (deftype Name/->Name unbound + protocol body silently dropped) -- **Related**: ADR-0007 (TypeDescriptor / Option β), ADR-0041 (fn multi-arity), - ADR-0050 (interop call node), `.claude/rules/dual_backend_parity.md` (ADR-0036), - F-002 / F-009 / F-011, D-048 (catch-by-deftype-class, orthogonal) - -## Context - -`deftype` was a **special form** (`analyzer/special_forms.zig::analyzeDeftype` -→ `deftype_node` → `tree_walk::evalDeftype` / VM `op_deftype`). `analyzeDeftype` -parsed only `(deftype Name [fields])` and **silently ignored any protocol-impl -body** after the field vector — `(deftype T [v] IFoo (foo [_] ...))` "succeeded" -while dropping the `IFoo` impl (a permanent-no-op, forbidden by -`provisional_marker.md`). It also never bound the positional constructor `->Name` -(`(->T 42)` → "Unable to resolve symbol"), unlike JVM Clojure. - -`defrecord` is **already a macro** (`macro_transforms.zig::expandDefrecord`) -lowering to `(do (def Name (rt/__defrecord! 'Name ['fields])) (def ->Name -(fn* [..] (Name. ..))) extend-type-sections...)`. deftype and defrecord differ -only in the registered `TypeDescriptor.kind` (`.deftype` vs `.defrecord`) and the -absence of map semantics for deftype — and the downstream map-protocol arms -already gate on `inst.descriptor.kind != .defrecord` (collection.zig:429/564/615), -so a `.deftype`-kind instance is correctly excluded from map behaviour. - -A Devil's-advocate review (general-purpose subagent, fresh context, F-NNN- -constrained) also surfaced a **latent VM bug (BUG-1)**: the VM `op_deftype` arm -pushed nil and relied on a (false) "analyzer-time registerType" comment — a -deftype run on the VM backend ALONE never registered the type. The diff harness -masked this by running TreeWalk first on the same `rt` (state leak). - -## Decision - -Convert `deftype` to a macro `expandDeftype` mirroring `expandDefrecord`, and -**retire the special form**. Per the DA's correction, do NOT duplicate -`defrecordPrim`: extract a shared kind-parameterised registration helper that -both `rt/__defrecord!` and the new `rt/__deftype!` thunk into (F-011 — the two -primitives differ only in the `.kind` passed to `registerType`). - -Removed (the deletion is fully enumerable; the VM's exhaustive `Opcode` / `Node` -switches make completeness compiler-enforced): - -- `deftype` from the analyzer special-forms table + `deftype_form` enum tag + - dispatch arm + `analyzeDeftype`. -- `deftype_node` from the `Node` union + `evalDeftype` (TreeWalk) + the VM - compile arm + `op_deftype` (opcode enum + emit + VM dispatch). -- The false "analyzer-time registerType" comments. - -The macro's `rt/__deftype!` call is a backend-neutral primitive (executed -identically by TreeWalk eval and VM `op_call`), so registration no longer splits -by backend — **BUG-1 is fixed as a side effect**. - -## Consequences - -- `(deftype Name [fields] Proto (m [..] ..))` now binds `Name` + `->Name` and - applies the protocol impls via the shared `extend-type` lowering. clj-grounded. -- deftype gets NO map semantics (kind-gated) — matches JVM `deftype` vs - `defrecord`. -- catch-by-deftype-class is unaffected: `host_class.isKnownException` scans a - static list, never `rt.types` (D-048 remains the orthogonal open row). -- `(Name. args)` constructor still resolves via eval-time `rt.types.get` inside - the macro's `(do ...)` — same ordering as before. -- A **VM-only** (fresh-`rt`) deftype test is added so the diff harness no longer - masks backend-specific registration (dual_backend_parity test point 4). -- The analyzer unit test asserting `n.* == .deftype_node` is rewritten to assert - macro expansion. - -## Alternatives considered (Devil's-advocate output, verbatim summary) - -- **Alt 1 — smallest-diff: keep the special form, parse the body + emit ->Name in - `analyzeDeftype`.** Rejected: re-implements `expandDefrecord`'s entire - extend-type lowering in analyzer/Node space — a direct F-011 violation - (commonization outranks effort); leaves BUG-1 unfixed. Its only honest sub- - variant (raise `feature_not_supported` for the body) is a strictly-worse - transient stub than implementing it. -- **Alt 2 — finished-form: the macro conversion (CHOSEN).** Maximal F-011 - commonization (`expandDeftype`/`expandDefrecord` differ only in primitive name + - `.kind`); F-009-clean; deletes a Node variant + opcode; incidentally fixes - BUG-1. DA corrections folded in: (1) share a kind-parameterised registration - primitive instead of a duplicated `__deftype!`; (2) verified the map-semantics - gate keys on `.kind` not `field_layout`. -- **Alt 3 — wildcard: one `expandDefType(kind)` body + one kind-param primitive - for BOTH deftype and defrecord.** Primitive-level unification adopted (the - registration genuinely never diverges); macro-level unification rejected — - defrecord grows map-only factory/assoc/`=`-by-fields that deftype must never - have, so a single macro body would accumulate a `kind`-branch thicket. Two thin - macro bodies sharing extracted helpers is cleaner than one branchy body. - -DA recommendation (non-binding): Alt 2 with the shared kind-param primitive — the -main loop adopts exactly this. No F-NNN blocks any alternative; cycle/diff size -did not factor into the ranking. - -## Amendment 1 (2026-06-25) — cross-section same-name-arity overload merge (D-530) - -clj lets a deftype / defrecord / reify implement the same method NAME at -different arities across DIFFERENT protocol sections — `clojure.lang.Seqable` -`seq[this]` + `clojure.lang.Sorted` `seq[this asc]` (`NewInstanceMethod` keys -methods by `[name, arity]`). data.priority-map's `subseq`/`rsubseq` need it. -cljw already merged same-name arities WITHIN one protocol section into a -multi-arity `fn*` (D-279, in `expandExtendType` + inline in `expandReify`), but -two sections never met that grouping, so each emitted its own single-arity -method-table entry and the dot-form `(. inst seq true)` resolved the first -(arity-1) row → "Wrong number of args (2)…expected 1". - -**Decision**: a cross-section merge in the lowering. `lowerDefType` (deftype + -defrecord) and `expandReify` each pre-scan all sections, grouping impls by method -name + a per-name section bitset; a name appearing in >1 section (popcount > 1) -emits its FULL arity set under EACH contributing protocol, so `expandExtendType` -/ the reify per-section builder produce the complete multi-arity `fn*` for each. -`MethodEntry` stays `{protocol_name, method_name, method_val}` (NO arity field — -F-004), `lookupMethod` stays first-match, the dot-form call path + BOTH backends -are untouched — `selectMethod` already dispatches a multi-arity fn by arg count, -and the fix is in macro lowering (upstream of the analyzer Node split), so -TreeWalk + VM get it identically (no VM-DEFER). One file pair, lowering-only, -reusing the proven D-279 multi-arity-fn path. - -**Risks (DA-surfaced, resolved):** -- *Same-arity duplicate* (two clauses, identical arity, which clj rejects - "Duplicate method name&signature"): already caught — the merged `(fn* …)` hits - the existing multi-arity validator "can't have two overloads with same arity", - so cljw errors too. **F-011 accept/reject parity holds** (verified both sides). -- *Per-protocol fn duplication*: the merged fn registered under two protocols is - two `MethodEntry` rows holding the SAME heap-fn handle — F-004's 8-byte Value - is non-owning, GC traces the fn once, no double-free. Benign by construction. -- *Arity-surface widening* (Sorted's arity-2 `seq` also reachable via the Seqable - entry): only the dot-form (null-protocol, arg-counted) reaches the union arity; - no cljw protocol-fn is multi-arity, so a `(seq inst)` protocol call still - selects arity-1 — the widening is unobservable. Protocol MEMBERSHIP is tracked - separately (`protocol_impls` / `declaresProtocol`), not inferred from the shared - method row, so sharing the fn does not make Sorted "satisfy" Seqable. -- *reify scope*: reify has the same gap and clj allows the overload on reify too, - so fixing only deftype/defrecord would be a finished-form asymmetry (F-002). The - same cross-section merge is applied to `expandReify` in the SAME cycle. - -### Amendment 1 — Alternatives considered (Devil's-advocate output) - -A fresh-context `general-purpose` DA was briefed with the active F-NNN envelope -(F-004 uniform Value slot / F-009 neutrality / F-002 finished-form / F-011 parity -/ ADR-0008 am3 method_val convergence) and produced three shapes: - -- **Alt 1 — smallest-diff: the chosen merge + a per-protocol arity filter + a - same-arity error.** Closes the widening by filtering each protocol's entry to - its own declared arities, but to keep the dot-form's union it must add a - synthetic fallback row OR a `selectMethod` arity-miss-fallthrough — i.e. it - reaches into the dispatch path anyway, while leaving "is Seqable's seq arity-1 - or 2?" answered by whichever workaround. -- **Alt 2 — finished-form (DA-recommended): one canonical multi-arity fn per - method NAME (interned once), with the protocol-fn surface routed through the - same shared `selectMethod` as the dot-form**, so F-009 ("reachable identically - from every surface") holds by construction rather than by current wiring. - Bigger diff (touches `dispatch.zig`); the DA recommended it citing F-002. -- **Alt 3 — wildcard: leave lowering untouched, make `lookupMethod`/`selectMethod` - arity-aware at the dispatch layer** (return all same-name rows, pick by argc). - Most faithful to clj's `[name,arity]` keying and never widens a protocol's - surface, but it pushes work into the dispatch hot path and creates a split-brain - (within-section → merged fn at lowering; cross-section → arity-lookup) unless - the within-section D-279 path is ALSO migrated — a much larger change. - -**Main-loop decision (within the F-NNN envelope; DA recommendation is -non-binding):** keep the lowering-merge (the chosen design ≈ the DA's "Option A"), -NOT Alt 2. The DA's case for Alt 2 rests on the arity-surface widening being an -F-009 weakness; but the widening is **unobservable** (no cljw protocol-fn is -multi-arity, so only the arg-counted dot-form reaches the union arity, where -by-argc dispatch is correct), and protocol membership is tracked independently of -the method table — so Option A satisfies F-009 in practice, not merely "by current -wiring". Adopting Alt 2's `dispatch.zig` changes for a non-manifesting concern is -over-reach, not finished-form. The two DA risks that ARE real were both addressed: -the same-arity duplicate already errors via the `fn*` validator (F-011), and reify -is fixed in the same cycle (F-002 symmetry). Alt 3 is rejected as a split-brain -absent migrating the within-section path too. diff --git a/.dev/decisions/0067_eduction_as_reiterable_deftype.md b/.dev/decisions/0067_eduction_as_reiterable_deftype.md deleted file mode 100644 index 72c50bc86..000000000 --- a/.dev/decisions/0067_eduction_as_reiterable_deftype.md +++ /dev/null @@ -1,79 +0,0 @@ -# ADR-0067 — `eduction` as a re-iterable `deftype`, not an alias for `sequence` - -- **Status**: Proposed → Accepted (2026-06-01) -- **Discharges**: D-160 (the `eduction` residual; `sequence` landed earlier the same day) -- **Related**: ADR-0066 (deftype macro), ADR-0036 (`.claude/rules/dual_backend_parity.md`), - F-002 (finished form wins), F-011 (behavioural equivalence to Clojure JVM), - D-189 (first/rest Seqable→seq coercion, surfaced here), the `sequence` push→pull - bridge (`-tx-seq-pump`, core.clj) - -## Context - -`(eduction xform* coll)` returns a value that is both **reducible** (each -`reduce` re-runs the transducer pipeline) and **seqable** (a lazy view). -cljw's `sequence` (the push→pull lazy bridge) already exists. The open -question: implement `eduction` as a thin alias over `sequence`, or as its -own type? - -Facts (verified in-repo): cljw's `reduce` (`higher_order.zig:177`) routes a -receiver carrying an `IReduce -reduce` MethodEntry through it -(receiver-first: `(-reduce coll f)` / `(-reduce coll f init)`); `seq` -(`sequence.zig:195`) routes through `Seqable -seq`. `deftype` protocol -bodies + variadic protocol-method params (`[this f & more]`) both work. -So a `deftype` extending `IReduce` + `Seqable` is dispatchable. - -## Decision - -Implement `eduction` as a `deftype Eduction [xform coll]` extending -`IReduce` (`-reduce`→`transduce`, variadic to serve both the 2-arg and -`into`'s 3-arg call) and `Seqable` (`-seq`→`sequence`). `(completing f)` -wraps the reducing fn so an arbitrary `f` (e.g. `conj`, which has no 1-arg -completion arity) works through the transducer protocol. - -This makes eduction **re-iterable** — the JVM contract — and **distinct -from `sequence`**: a counter xform reduced twice over the same eduction -fires 6 times (cljw + clj), where a cached-lazy-seq alias would fire 3. - -## Consequences - -- eduction is a genuine reducible (no seq allocation on the reduce path) - and a lazy seqable (via the `sequence` bridge) — F-011 faithful. -- First live consumer of the `IReduce -reduce` **and** `Seqable -seq` - deftype slow paths together; the `-reduce` variadic widens the - IReduce-declared `[c f]` to cover `(-reduce c f init)`. -- `first`/`rest` applied **directly** to an Eduction raise (cljw's - first/rest dispatch `ISeq -first`/`-rest` without a Seqable→seq - coercion fallback). `(seq e)` / `(into [] e)` / `(map f e)` all work. - The direct-coercion gap is **D-189** (broader than eduction). -- No new Zig primitive, no new analyzer Node — pure `.clj` over existing - machinery; both backends agree (e2e on the real binary; the diff - Fixture loads no core.clj so eduction is verified there via `--compare` - + the shared dispatch path, not a diff case). - -## Alternatives considered (Devil's-advocate output, verbatim summary) - -A `general-purpose` Devil's-advocate fork (fresh context, F-002/F-011/F-009 -constraints) produced three shapes: - -- **Alt 1 — smallest-diff (closure):** `(def eduction (fn* [& a] (sequence - (apply comp (butlast a)) (last a))))`. One line, rides the landed bridge. - **Breaks:** NOT re-iterable — `sequence` caches (the `-tx-seq-pump` - volatile realizes once), so a second `reduce` replays the cache, not the - xform; a side-effecting `(eduction (map prn) …)` prints once vs JVM's - every-reduce. Observationally **identical to `sequence`** → redundant-alias - smell (Smallest-diff bias, principle.md). Real F-011 divergence. -- **Alt 2 — finished-form-clean (deftype):** `deftype Eduction` + IReduce + - Seqable (the chosen shape). Re-iterable, distinct from `sequence`, no seq - alloc on reduce. Cost: first consumer of both slow paths; `-reduce` must - be variadic for the collapsed-IReduce 3-arg `init` call. -- **Alt 3 — wildcard (reducible-only deftype):** IReduce but no Seqable; - `(seq e)` raises. **Breaks:** JVM eduction IS seqable; withholding `-seq` - is a gratuitous F-011 gap when `sequence` gives the seq view for free. - -DA recommendation: **Alt 2** — the unique shape that is both re-iterable -(F-011) and not a `sequence` alias (F-002). Explicitly NOT downgraded to -Alt 1 on diff/LOC grounds (that is the Cycle-budget defer smell; F-002 says -size is not a constraint). The DA flagged a prerequisite — multi-arity -protocol-method bodies are unsupported — resolved by a **variadic** -`-reduce` `[this f & more]` (confirmed working), so no PROVISIONAL was -needed. The main loop adopts Alt 2. diff --git a/.dev/decisions/0068_sequential_marker_print_form.md b/.dev/decisions/0068_sequential_marker_print_form.md deleted file mode 100644 index 0f38fd674..000000000 --- a/.dev/decisions/0068_sequential_marker_print_form.md +++ /dev/null @@ -1,297 +0,0 @@ -# ADR-0068 — `Sequential` marker protocol drives seq-print + `sequential?`; record map-style print form (D-190) - -- **Status**: Proposed → Accepted (2026-06-01) -- **Discharges**: D-190 (eduction prints `#Eduction[..]` not `(2 3 4)`; record - prints `#Pt[1 2]` not `#user.Pt{:x 1, :y 2}`). Also closes the latent - `(sequential? eduction)` → `false` divergence (clj: `true`). -- **Related**: ADR-0067 (`eduction` as a `deftype`), ADR-0008 (protocol + - multimethod dispatch), F-002 (finished form wins), F-009 (impl neutrality / - zone layering), F-011 (behavioural equivalence to Clojure JVM, commonisation - over effort), D-189 (Seqable→seq coercion), D-058/D-079 (ns surface — the - record-fqcn ns-prefix residual is parked there) -- **Defers**: user-extensible `print-method` multimethod (open dispatch, - `(defmethod print-method MyType …)`) to a future ADR — see § Deferred. - -## Context - -`(eduction (map inc) [1 2 3])` prints `#Eduction[# [1 2 3]]` in cljw -but `(2 3 4)` in Clojure; `(->Pt 1 2)` for `(defrecord Pt [x y])` prints -`#Pt[1 2]` not `#user.Pt{:x 1, :y 2}`. The VALUE is correct in both cases -(`into`/`reduce`/`seq`/`first` all match — ADR-0067); only the `prn`/`pr` -PRINT FORM diverges. - -**Architecture (verified in-repo).** `prn`/`pr`/`pr-str` are Zig primitives -(`src/lang/primitive/core.zig`) that funnel through one sink: -`print_mod.printResult(rt, env, w, v)` (Layer-0 `src/runtime/print.zig`). -`printResult` already carries `rt`+`env`. It calls `deepRealize` (realizes -`.lazy_seq`/`.list`/`.vector`; `else => return v`, so a `.typed_instance` -passes through unrealized) then the pure `printValue`, whose `.typed_instance` -arm is `printTypedInstance` → `#[field0 field1 …]` (values only). That -single default is the source of BOTH divergences. - -**Two stages, two bug kinds (the crux).** The eduction bug is a -**realization miss** (deepRealize's `else` swallows the `.typed_instance`); -the record bug is a **rendering default** (`#fqcn[..]` instead of map-style). -cljw has materialised what Clojure expresses through one `print-method` -interface as **two distinct stages**: `deepRealize` (realization, has rt) then -`printValue` (rendering, pure). The two bugs live in those two stages. - -**The discriminator is `Sequential`, not `Seqable` (oracle-verified).** A -record is `[sequential? false, seqable? true]` yet prints map-style; an -eduction is `sequential? true` and prints as its seq. So "print as a seq" ⟺ -`Sequential`, NOT `Seqable` (realizing every Seqable typed_instance would wrongly -turn a record into its entry-seq). cljw has **no `Sequential`** (grep: 0 hits as -a protocol; `sequential?` is a hard-coded tag switch with `else => false`, so -`(sequential? eduction)` is wrongly `false`). Clojure's `Sequential` is a -**marker interface** (zero methods). cljw's `defprotocol` currently **rejects** -zero-method protocols (`args.len < 2` guard) — but JVM allows `(defprotocol -Marker)` with `(satisfies? Marker x)` → `true` (oracle-verified), so accepting -markers is behavioural equivalence, not a new divergence. - -cljw already has a full `MultiFn` infrastructure (`multimethod.zig`, JVM -`clojure.lang.MultiFn`-equivalent: dispatch fn / method table / prefer table / -hierarchy / cache; `defmulti`/`defmethod`/`prefer-method` work). The MultiFn -doc-comment even names `clojure.core/print-method` as its example. So the full -`print-method`-as-multimethod finished form is *reachable* — but it additionally -requires a print-recursion layer relayer (the recursive descent lives in Layer-0 -`printValue`; a Clojure multimethod is Layer-2) and user-extensibility that -D-190's bug does not need. See § Deferred + § Alternatives. - -## Decision - -Fix each bug in its architecturally-correct existing stage, with a real -`Sequential` marker as the realization discriminator. One marker, two -consumers (print-realize + `sequential?`) — F-011 commonisation. - -1. **Relax `defprotocol` to accept zero-method marker protocols.** - `expandDefprotocol` guard `args.len < 2` → `args.len < 1` (a name suffices; - empty `method_sigs` expands to `(rt/__make-protocol! 'name [])`). JVM-faithful. - Update the `defprotocol_form_incomplete` template to drop "and at least one - method signature". - -2. **Define `(defprotocol Sequential)` marker in core.clj**, and extend it on - `Eduction`. **Marker protocols must work end-to-end on a `deftype`** — and at - grounding time they did NOT (see § Grounding correction). The machinery: - - `lowerDefType` + `expandExtendType`: a zero-impl protocol section / a - 2-arg `(extend-type Name Marker)` is now allowed (was rejected) — it is a - marker extension, not a malformed form. - - `__extend-type!` (`extendType`): records `proto.fqcn()` into the - descriptor's `protocol_impls` via the new `protocol.addProtocolImpl` - (dedup, infra-realloc mirroring `extendTypeWithImpls`) — so a zero-method - marker (no `method_table` entry) is still detectable, and `protocol_impls` - becomes an honest "implements P" set. Teardown frees the slice (not the - borrowed fqcn entries) in `Runtime.deinit` (user + native) + `registerType` - re-register. - -3. **Commonise `sequential?`** (`core.zig:sequentialQ`): keep the native-tag - fast path; ADD — for a `.typed_instance` — a `descriptor.declaresProtocol( - "Sequential")` check (the shared `TypeDescriptor.declaresProtocol` helper that - walks `protocol_impls` + parent). `(sequential? eduction)` → `true`, matching - clj. The same marker + helper the print path consumes drives the predicate - (one SSOT, no second source of truth). - -4. **print.zig — realize (eduction):** `deepRealize` gains a `.typed_instance` - arm: if `descriptor.declaresProtocol("Sequential")`, coerce via the `Seqable - -seq` protocol (`dispatch.dispatchOrNull`, Layer-0 — `lazy_seq_mod` cannot - coerce a typed_instance itself; that is the seq primitive's job, Layer-2 / - D-189), then `deepRealize` the coerced lazy_seq/list → prints as `(2 3 4)`. - `dispatchOrNull` (not `dispatch`) so a Sequential-but-not-Seqable deftype - falls back to the default render instead of raising mid-print. Zone-legal: - print.zig (Layer-0) reads `protocol_impls` + calls `dispatch.zig` (Layer-0); - it does NOT import `lang/`. - -5. **print.zig — render (record):** `printTypedInstance` branches on - `inst.descriptor.kind == .defrecord` → emit `#{:name val, …}` from - `field_layout` (declared field names) + `fields()`; else the existing - `#[..]`. Pure (no rt). Declared fields only — assoc'd extra keys - (`__extmap`, D-086) are a separate residual. - -### Why this is the finished form for D-190's scope (not smallest-diff bias) - -The discriminator is the REAL one (`Sequential`), JVM-faithful, not a `kind` -heuristic. It is commonised: one marker drives both print and `sequential?` -(F-011). Each fix sits in its architecturally-correct stage. And it is a genuine -**shrinking** skeleton toward the deferred `print-method` multimethod: the -`Sequential` marker, the deepRealize-realize arm, and the record renderer all -survive verbatim as that multimethod's `:default` / `Sequential` / record method -bodies (DA-confirmed). The scope boundary is answered on merits, not diff size: -"is *user-extensible* `print-method` in D-190's finished form?" → No; overriding -how a type prints is a separate feature from cljw's own types printing correctly. - -## Grounding correction (Step 0.6 honesty) - -The pre-implementation grounding asserted "the deftype/reify descriptor builder -already records one entry per declared interface in `protocol_impls`, so the -marker registers." **This was only true for the `reify` path.** The -`deftype`/`defrecord` path (`registerType`) leaves `protocol_impls = &.{}` and -records protocols solely via `method_table` (`__extend-type!`), AND the macro -lowering *rejected* a zero-impl protocol section (`lowerDefType` / -`expandExtendType` raised `extend_*_invalid`). So a marker on a `deftype` was -unrepresentable. Decision step 2 above is the corrected shape — the marker -machinery (guard relaxations + `addProtocolImpl` + teardown) is part of this -ADR, surfaced during implementation per the Step 0.6 re-laying discipline. This -is a genuine capability gain (marker protocols now work on `deftype`), not just -a D-190 patch — consistent with F-002 (the cleaner finished form is worth the -extra surgery). - -## Deferred (future ADR, not this cycle) - -- **User-extensible `print-method` multimethod.** `(defmethod print-method - MyType …)` is unsupported until a future ADR routes the print sink through a - real `print-method` `defmulti` (reusing the existing MultiFn) and decides the - print-recursion layer (lift the recursive renderer to Layer-2, or keep the - Layer-0 default + a Layer-2 override seam). The three pieces above become its - method bodies. `print-dup` and exotic `prefer-method` nuances are out of scope - there too (recorded DIVERGENCE — avoids an excessive skeleton now). - -## Scope / recorded divergences - -- **Record fqcn ns-prefix.** This cycle renders `#Pt{:x 1, :y 2}`; clj renders - `#user.Pt{:x 1, :y 2}`. The `{:k v}` map shape is the D-190 fix; the `user.` - ns-prefix depends on the ns surface and is parked on D-058/D-079 (F-003 - structural-deferred). Residual, recorded. -- **Plain non-Seqable `deftype`.** clj prints `#object[user.Foo 0x… "…"]`; cljw - prints `#Foo[..]`. Untouched here — a separate acceptable divergence (cljw's - form is arguably more useful and is not D-190's subject). -- **`(satisfies? Marker x)` on a marker protocol.** cljw's `satisfies?` checks - `method_table` (`protocol.satisfies`), so a zero-method marker returns - `false` where JVM returns `true`. Out of D-190 scope (print + `sequential?` - both consult `protocol_impls` via `declaresProtocol`, which IS marker-aware). - Aligning `satisfies?` to also consult `protocol_impls` is a clean follow-up - if marker-`satisfies?` is ever needed; filed as the D-190 spinoff note. - -## Alternatives considered - -(Devil's-advocate fork, fresh context, F-002/F-009/F-011 envelope. Reflected -verbatim; recommendation is non-binding — the main loop chose Alt 3 sharpened -with a real `Sequential` marker, having answered the scope question against -near-term user-extensible `print-method`.) - -### Alt 1 — Smallest-diff: two arms in the Zig printer - -Touch only `print.zig`. (i) `deepRealize` `.typed_instance` arm: if the -descriptor is `Seqable`/`IReduce`-bearing (or `kind != .defrecord`), drive the -seq loop → eduction prints `(2 3 4)`. (ii) `printTypedInstance` branches on -`kind == .defrecord` → `#fqcn{:field val …}`; else `#fqcn[..]`. No core.clj -change, no new var. -**Better:** stays in the single sink → trivially F-011-correct (one site); zero -layering risk; no new dispatch dep, no Clojure↔Zig seam. -**Breaks:** the dispatch logic ("is this a seq? a record?") is hard-coded in the -Zig printer **by `kind` enum**, not by the value's own protocol participation — -the exact "ad-hoc per-site patch" F-011 names as the failure mode. The next -printable type (custom-print deftype, future `IPrintable`, `print-dup`) forces -ANOTHER arm. The print decision lives in the wrong layer (printer enumerates -types vs the value's type says how it prints). It accretes; does not generalise. -**F-NNN:** F-011 ✅ sink-commonisation but ❌ *mechanism* commonisation. F-009 ✅ -(Layer-0 only). F-002 ❌ — smallest-diff a finished-form owner unwinds at the -third printable type; *enlarges* the eventual rewrite, so not an endorsable -skeleton. - -### Alt 2 — Finished-form-clean: `print-method` as a real multimethod - -Define `print-method` as a genuine `defmulti` (dispatch on `(class x)`), reuse -the existing `MultiFn`. `:default` method = thin primitive calling the Zig -`printValue` (scalars + collection literals stay Zig). Add a record method -(map-style) and a `Sequential` method. **Seam to build:** `prn`/`pr-str`/`pr-on` -route through the `print-method` var (the Layer-2 primitive resolves+invokes the -var), AND `Sequential` must be created as a real marker, and Eduction re-tagged -to carry it. -**Better:** JVM-faithful — `print-method` *is* a multimethod; records/Sequential -*are* methods; user `(defmethod print-method MyType …)` for free. F-011's "single -shared mechanism" satisfied at the *mechanism* level (open dispatch). -**Breaks/costs:** two new structural commitments larger than the bug. (1) A -**Zig-primitive → Clojure-var call seam** at the sink — `print.zig` is Layer-0 -and may not import `lang/`; the recursive descent (nested values) must re-enter -`print-method` per element, so either a Layer-0/Layer-2 ping-pong or lift the -whole recursive renderer to Layer-2. (2) **`Sequential` marker must be invented** -anyway (separate structural feature) + Eduction re-tagged. **Perf:** every nested -print element pays a multimethod dispatch (cache helps, but vs a Zig switch it's -real). Genuinely finished-form-clean — but its finished form is *larger than -D-190*: it pulls in `Sequential` AND a print-recursion relayer AND user -extensibility. -**F-NNN:** F-011 ✅✅ (true open mechanism). F-009 ⚠️ solvable but load-bearing -(renderer's home layer must be decided). F-002 ✅ this is the finished form **for -the print SUBSYSTEM** — but D-190's scope is a strict subset; adopting it means -D-190 grows into "print-method subsystem", endorsable under F-002 (big surgery is -the default) **provided `Sequential` is built as a co-requisite, not stubbed.** - -### Alt 3 — Wildcard: the two-mechanism hybrid, made principled - -Accept that record-print and eduction-print are **two mechanisms** and route -each by its actual nature. **Eduction:** fix at `deepRealize` generically — -realize any value that participates as a seq (drive `lazy_seq_mod.seq`), one -general "realize seqable" arm, not an Eduction special-case; prints through the -existing list printer; no new marker. **Record:** a *pure* `#fqcn{:k v}` render in -`printTypedInstance` keyed on `kind == .defrecord` + `field_layout`; needs no rt, -no dispatch. -**Better:** honest about the real two-concern split — eduction-print is a -**realization** problem (shares the deepRealize seq-walk; F-011-common with -lazy_seq/list), record-print is a **rendering** problem (pure field formatting). -Forcing them into one multimethod (Alt 2) is itself a mild conflation (the -Sequential method body just re-seqs = realization wearing a rendering hat). Puts -each concern where it already belongs; no new marker, no Zig→Clojure seam, no -perf regression; smaller than Alt 2 yet not Alt 1's ad-hoc, because the eduction -arm is the *general* realize-seqable arm (covers future Seqable typed_instances). -**Breaks/costs:** no user-extensible `print-method` (recorded DIVERGENCE). If the -finished form wants open printing, Alt 3 is a way-station Alt 2 subsumes — -cleanly: Alt 3's two pieces survive as Alt 2's method bodies. Record-print stays -a Zig `kind` branch — defensible (records are a *core* type with fixed print -syntax; Clojure hardcodes record print in `core_print.clj` too). -**F-NNN:** F-011 ✅ (eduction arm is the shared seqable-realize mechanism; record -render is the inherent core formatter, not a comment-apology local fix). F-009 ✅ -(Layer-0, rt present, no seam). F-002 ⚠️ clean **iff** user-extensible -`print-method` is not in the near-term finished form; if it is, Alt 3 is a -skeleton-toward-Alt-2 (acceptable *because it shrinks* Alt 2's rewrite — its two -pieces become Alt 2's method bodies verbatim). - -> **DA's crux assessment:** the unifying "one mechanism" is real only as an -> *extensibility interface* (`print-method` open dispatch), NOT as a single code -> path. cljw already materialises the realize/render split as two stages -> (`deepRealize` then `printValue`); the eduction bug is a realization miss, the -> record bug a rendering default — two fixes in two stages. -> **DA ranking (non-binding):** Alt 3 (best F-002/F-009/F-011 balance for D-190's -> actual scope; its pieces survive into Alt 2 → never wasted) > Alt 2 (true -> subsystem finished form; choose only if user-extensible `print-method` is -> near-term, with `Sequential` as co-requisite) > Alt 1 (rejected: `kind`-switch -> is the F-011 ad-hoc failure mode; enlarges the rewrite). - -**Main-loop decision:** Alt 3 **sharpened** — the realization discriminator is a -real `Sequential` marker protocol (not Alt 3's `kind`/Seqable heuristic, which -the oracle shows mis-handles records). This keeps the eduction fix principled -(`Sequential` → seq, JVM-faithful), commonises the latent `sequential?` -divergence onto the same marker, and stays a strict shrinking skeleton toward -the deferred Alt 2. The scope question ("is user-extensible `print-method` -near-term?") is answered No on merits, per F-002's "downgrade only on the scope -question, never on diff size". - -## Consequences - -- `(prn (eduction (map inc) [1 2 3]))` → `(2 3 4)`; `(sequential? (eduction …))` - → `true`; `(prn (->Pt 1 2))` → `#Pt{:x 1, :y 2}` (ns-prefix residual deferred). -- `defprotocol` now accepts marker protocols (`(defprotocol Sequential)`), - matching JVM. Existing ≥1-method protocols unaffected. -- One `Sequential` marker is the SSOT for both seq-print and `sequential?` — no - second source of truth to drift. -- A future ADR can route the print sink through a real `print-method` multimethod - reusing these three pieces as method bodies; nothing here is throwaway. - -## Affected files - -- `src/lang/macro_transforms.zig` — `expandDefprotocol` guard `< 2`→`< 1`; - `lowerDefType` + `expandExtendType` zero-impl-section / 2-arg allowance. -- `src/runtime/error/catalog.zig` — `defprotocol_form_incomplete` template. -- `src/lang/clj/clojure/core.clj` — `(defprotocol Sequential)`; extend on - `Eduction`. -- `src/runtime/protocol.zig` — `addProtocolImpl` (record protocol in - `protocol_impls`). -- `src/lang/primitive/protocol.zig` — `extendType` calls `addProtocolImpl`. -- `src/runtime/type_descriptor.zig` — `declaresProtocol` helper; `registerType` - re-register frees `protocol_impls`. -- `src/runtime/runtime.zig` — `deinit` frees `protocol_impls` (user + native). -- `src/lang/primitive/core.zig` — `sequentialQ` `declaresProtocol` arm. -- `src/runtime/print.zig` — `deepRealize` `.typed_instance` Sequential coerce- - and-realize arm (`dispatchOrNull` `-seq`); `printTypedInstance` record - map-style branch; `realizeSeqWalk`/`typedInstanceIsSequential` helpers. -- `test/e2e/phase14_transducers.sh` + `test/e2e/phase7_defrecord.sh` — RED → - green print-form cases. -- `.dev/debt.md` — D-190 → Discharged. diff --git a/.dev/decisions/0069_contains_vector_index_semantics.md b/.dev/decisions/0069_contains_vector_index_semantics.md deleted file mode 100644 index 0409045e2..000000000 --- a/.dev/decisions/0069_contains_vector_index_semantics.md +++ /dev/null @@ -1,89 +0,0 @@ -# ADR-0069 — `contains?` on a vector matches JVM index semantics (reverse DIVERGENCE D1) - -- **Status**: Proposed → Accepted (2026-06-02) -- **Supersedes**: DIVERGENCE D1 in `lang/primitive/collection.zig::containsQFn` - (the Phase-6.16 "reject vector contains? as a cleaner set/map-only semantic" - note + its locking unit test). -- **Related**: F-011 (behavioural equivalence with `clj`), F-010 (run real - Clojure libraries), F-002 (finished-form wins), `no_jvm_specific_assumption.md`, - ADR-0059 (no-JVM-Class surface divergences). Surfaced by the `clojure.data/diff` - clj-diff sweep. - -## Context - -`(contains? coll k)` carried **DIVERGENCE D1**: cljw rejected -`(contains? [1 2 3] 1)` with `type_error`, on the rationale that JVM Clojure's -"vectors test index-membership" is a documented newcomer footgun and a -"cleaner set/map-only semantic" was preferable. A unit test locked this -(`expectError(error.TypeError, ...)`). - -D1 was written in Phase 6.16, **before F-011** (2026-05-31) made behavioural -equivalence with `clj` project law. JVM Clojure: `(contains? [1 2 3] 1)` → `true`, -`(contains? [1 2 3] 5)` → `false`, `(contains? [1 2 3] :x)` → `false` — a pure -**index-validity** predicate (`integer?(k) ∧ 0 ≤ k < count`), never throwing. - -The `clojure.data/diff` sweep made the conflict concrete: `diff`'s sequential -path calls `(contains? shorter-vector out-of-range-index)` and **requires it to -return `false`, not throw**. Under D1, real `clojure.core`-shape code cannot run -— directly blocking F-010's named engine (running real libraries through cljw). - -## Decision - -Reverse D1. `contains?` on a vector tests index validity exactly as JVM: -integer `k` in `[0, count)` → `true`; non-integer or out-of-range → `false`; -never throws. Set / map / sorted / Associative-protocol semantics are unchanged. - -The footgun is inherited deliberately: it is *documented Clojure behaviour*, and -F-011 prioritises observable equivalence (including the gotchas) over an -implementer's judgment that the semantic "should" be cleaner. - -## Why D1 was not a legitimate divergence under F-011 - -F-011 permits a deliberate surface divergence only when it is **forced by a -no-JVM necessity** (e.g. `(class 5)` → `Long` not `java.lang.Long`, ADR-0059 — -there is no `java.lang.Long` to name). `contains?` on a vector needs **zero JVM -internals**: it is a pure integer-range check. D1 was a *judgment-divergence* -("we disagree with the JVM semantic"), which is exactly the category F-011's -equivalence mandate overrules. Per the priority chain (F-NNN > ADR/rule), the -older D1 implementation decision is edited to align with F-011, never the reverse. - -## Alternatives considered - -(Devil's-advocate subagent, fresh context, briefed with F-002/F-010/F-011.) - -- **Alt 1 (smallest-diff) — VIOLATES F-011, recorded not selectable**: keep D1 - throwing publicly, special-case `clojure.data/diff` with a private non-throwing - index check. Better: zero change to `contains?`. Breaks: a direct F-011 - violation (observable `(contains? [1 2 3] 1)` still diverges with no no-JVM - forcing fact); a lie-by-construction (public vs internal predicate disagree); - every other library using `(contains? vec i)` still breaks → F-010 whack-a-mole - forever; two semantics for one op → rot (F-010 refactor gate). Listed only - because the brief requires a smallest-diff entry. -- **Alt 2 (finished-form-clean) — CHOSEN**: delete D1, full JVM-faithful index - semantics, record the reversal as this ADR, add the canonical exprs + - `clojure.data/diff` to the clj corpus. Better: satisfies F-011; unblocks F-010 - + `clojure.data/diff`; removes the two-semantics rot; closes a footgun-blocker - as a corpus-backed area. Breaks: reintroduces the genuine Clojure footgun — but - that is *fidelity*, not a defect; requires inverting the D1 test. No F-NNN - violated. -- **Alt 3 (wildcard)**: Alt 2 + an opt-in, non-semantic lint/warning flagging - `(contains? )` to protect newcomers. Better: full - Alt-2 win plus captures D1's footgun-protection *intent* in a layer that never - alters return values; points at cljw's tooling story. Breaks: invents a lint - subsystem with its own design surface (false positives on macro-expanded code, - JIT interaction) — scope creep. Deferred to a `debt.md` row at the Phase that - owns dev-tooling, not bundled here. - -DA recommendation: Alt 2. Main loop concurs — F-011 is dispositive and F-010 -sharpens "should" to "must"; the lint idea is filed for later, not built now. - -## Consequences - -- `containsQFn` gains a `.vector` arm (index-validity, no throw). -- The D1 unit test inverts from "asserts throw" to "asserts true/false/false". -- The D1 doc-comment is replaced by a reference to this ADR. -- `clojure.data/diff` (and any real library using `(contains? vec i)`) now runs. -- 3 differential test cases (shared by both backends) + the `data_diff` corpus - lock the equivalence; corpus regression re-checks it every gate. -- Follow-up (deferred, not this cycle): the Alt-3 footgun lint — file when a - dev-tooling/diagnostics Phase opens. diff --git a/.dev/decisions/0070_vm_default_intent_and_parity_blockers.md b/.dev/decisions/0070_vm_default_intent_and_parity_blockers.md deleted file mode 100644 index fa2183294..000000000 --- a/.dev/decisions/0070_vm_default_intent_and_parity_blockers.md +++ /dev/null @@ -1,88 +0,0 @@ -# ADR-0070 — VM is the intended production default (§349); flip gated on the parity blockers (D-196) - -- **Status**: Proposed → Accepted → **Realised 2026-06-02** (D-196 closed; - build.zig default flipped to `vm`; F-012 reality-aligned). See Decision - step 4's "LANDED" note. -- **Records**: ROADMAP §349 ("default is VM after Phase 4") + build.zig 4.8 - comment ("4.12 flips the default once differential parity is green") were the - ORIGINAL plan; the flip was **never executed** (drift). This ADR makes the - intent + the concrete blocker list explicit, and establishes F-012. -- **Corrects drift in**: ADR-0050/0056/0036, which narrated `tree_walk` as "the - production-default backend" — calcifying the un-flipped interim state as if - intended. No ADR ever *decided* to override §349. -- **Related**: ADR-0005 (dual-backend oracle), F-002, F-010, F-011, F-012 (new), - D-196 (the parity blocker cluster). Surfaced 2026-06-02 by the user. - -## Context - -§349 always intended VM as the default after Phase 4; tree_walk is the -differential oracle. 4.8 made tree_walk a **temporary** default "until 4.12 -confirms parity"; 4.12 [x] confirmed Phase-4 smoke parity — but `build.zig` -kept `orelse .tree_walk` and the flip never happened. - -A 2026-06-02 experiment flipped `build.zig` to `vm` and ran the full gate. The -clj-grounded corpus reproduced **161/161** on VM, but the **full e2e suite -failed 5 cases on VM-default** — proving the VM is NOT yet production-ready and, -more importantly, that **tree_walk-default was masking VM gaps** (the per-commit -gate only ran the e2e on tree_walk; VM was covered by unit+diff_test only). The -flip was reverted; the gaps are tracked as D-196. - -## Decision - -1. **VM is the intended production default** (F-012). tree_walk is retained as - the differential oracle / reference implementation, always selectable via - `-Dbackend=tree-walk`. -2. **The flip is GATED on D-196** — `build.zig` default stays `tree_walk` ONLY - until every D-196 parity blocker is closed and the full gate is green under - `-Dbackend=vm`. This is no longer silent drift: the interim default has an - explicit, enumerated exit condition. -3. **VM gaps must stop being masked.** A VM-parity probe (`scripts/check_vm_parity.sh`, - builds VM + runs e2e + corpus) tracks the failing-case count; it is - informational until D-196 closes, then promoted to a hard per-commit gate so - no new VM divergence is masked again. -4. On D-196 close: flip `build.zig` default to `vm`, mark F-012 reality-aligned, - promote the VM-parity probe to a gate. - -**Step 4 LANDED 2026-06-02.** All 5 D-196 blockers closed (check_vm_parity = -0 fails); `build.zig` default flipped `orelse .tree_walk` → `orelse .vm`. The -per-commit gate now exercises the e2e surface on `vm` (the production default, -via `build_cljw`'s bare `zig build`); `run_all.sh` keeps unit coverage on BOTH -backends (`zig_build_test_vm` = default + `zig_build_test_tree_walk` = -`-Dbackend=tree_walk`), and the diff_test oracle compares both. The 3 -backend-forcing `phase4_*` e2e restore the DEFAULT (now vm) via bare -`zig build` so the shared binary stays vm for guarded e2e. F-012 is now -reality-aligned: VM is the production backend, tree_walk the differential -oracle. Residual follow-up (tracked, not a blocker): repurpose -`check_vm_parity.sh` to run the e2e suite on the NON-default backend -(tree-walk oracle) so an oracle-only rendering regression can't hide behind -the vm-default gate (on-demand / Phase-boundary per ADR-0049 cost concern); -Linux (ubuntunote) verification of the vm-default gate per ADR-0049 before the -v0.1.0 tag. - -## The parity blockers (D-196, evidence = VM-default gate 2026-06-02) - -| Gap | e2e | Status | -|-----------------------------------------------------------------------|------------------------------------------|----------------------------------------------------------| -| `catch :keyword` type dispatch | phase14_catch_keyword | **CLOSED 2026-06-02** — op_match_type_keyword (D-014b) | -| `(ns …)` `:refer-clojure :exclude` + libspec | phase14_ns_directive | **CLOSED 2026-06-02** — op_ns_with_filter (D-098) | -| java-surface constructor (`(java.io.File. …)`) | phase14_java_static_dispatch | **CLOSED 2026-06-02** — shared constructInstance | -| dynamic error-context (`with-context`/`:request-id`, ex-info `:data`) | phase14_with_context, phase14_user_throw | **CLOSED 2026-06-02 — ADR-0071** (cleanup-handler kind) | - -**All 5 e2e blockers are CLOSED (2026-06-02)** — `check_vm_parity.sh` reports -0 failing groups (corpus 375/375 + every blocker e2e green on `-Dbackend=vm`): -catch-keyword via op_match_type_keyword; ns-directive via op_ns_with_filter; -the java-surface ctor via the shared `special_forms.constructInstance`; -error-context via [ADR-0071](0071_vm_cleanup_handler_kind.md)'s cleanup-handler -kind. Per Decision step 4, the remaining work is the **build.zig default flip** -(`orelse .tree_walk` → `orelse .vm`) + promoting `check_vm_parity.sh` to a hard -per-commit gate, gated on a full gate run under `-Dbackend=vm` (Mac + ubuntunote) -confirming 200 pass. That flip is the next cycle (F-012 reality-aligned). - -## Consequences - -- The §349 intent is now law (F-012) with a tracked path, not a drifted memory. -- D-196 enumerates the work; closing each blocker is a VM backend cycle (verify - on the VM build, not just tree_walk). -- The VM-parity probe prevents re-masking. -- No DA fork: the decision (VM is the default) is mandated by §349 + F-012 (user - law) — there is no contested design choice, only the blocker list to clear. diff --git a/.dev/decisions/0071_vm_cleanup_handler_kind.md b/.dev/decisions/0071_vm_cleanup_handler_kind.md deleted file mode 100644 index 0190fe52e..000000000 --- a/.dev/decisions/0071_vm_cleanup_handler_kind.md +++ /dev/null @@ -1,196 +0,0 @@ -# ADR-0071 — VM cleanup handlers vs catch handlers (error-context parity) - -- Status: Proposed → Accepted -- Date: 2026-06-02 -- Supersedes: none -- Refs: D-196 (VM-default flip blocker 4), ADR-0070 (F-012 VM-default - intent), ADR-0060 (catalog→exception conversion), ADR-0055 am2 / - D-144 (dynamic error-context), ADR-0036 (dual-backend parity). - -## Context - -D-196 blocker (4): the VM backend loses the dynamic -`cljw.error/*error-context*` (and collapses the catalog Kind) when an -error propagates out of a `binding` (`with-context`) form. With -`CLJW_ERROR_FORMAT=edn`: - -- `(with-context {:request-id "abc"} (/ 1 0))` → VM renders - `:kind :exception` (should be `:arithmetic_error`) and drops - `:request-id`. tree_walk renders `:kind :arithmetic_error - :request-id "abc"`. -- `(with-context {:request-id "abc"} (throw (ex-info "boom" {})))` → - VM renders `:kind :exception :message "boom"` but drops - `:request-id`. - -### Root cause - -`compileBinding` (`vm/compiler.zig`) lowers `binding`'s -cleanup-on-unwind with `op_push_handler` (a **catch** handler) + -`op_pop_binding_frame; op_throw`. When the body raises, the VM outer -loop (`vm.zig`) treats it as catchable: - -1. **Catalog path** — converts the catalog error (`arithmetic_error`) - into a synthetic exception via `allocException(rt, info.message, - class)`, reading only `info.message` and `clearLastError()`-ing → - drops `info.context` AND collapses the Kind to `:exception`. -2. **Throw path** — nulls `last_thrown_context`, and the cleanup - re-throw `op_throw` runs AFTER `op_pop_binding_frame`, so its - context re-snapshot reads the now-restored empty `{}`. - -tree_walk's `evalBinding` uses `defer popFrame`: cleanup unwinds -**without catching**, so an uncaught catalog error keeps its `Info` -(Kind + context, captured at `setErrorFmt` time) and a user throw's -context snapshot (taken at the original throw, frame still live) -survive intact. - -The VM overloads ONE mechanism (`op_push_handler` + `op_throw`) for -two semantics — `catch` (intercept + convert + match) and `defer` -(unwind + re-raise unchanged). That overload is the bug. - -## Decision - -Distinguish **cleanup** handlers from **catch** handlers in the VM, so -a `binding`/bare-`try` cleanup behaves as the `defer` it semantically -is — no conversion, no context clear, no re-snapshot. - -1. `Handler` gains `kind: enum { catch_clause, cleanup }` (`vm.zig`). -2. New opcode `op_push_cleanup` — identical to `op_push_handler` but - tags the handler `.cleanup`. -3. New opcode `op_reraise` — re-fires the in-flight error WITHOUT - conversion or context mutation, via a - `dispatch.vm_pending_reraise: ?anyerror` threadlocal the cleanup - branch stashes immediately before jumping to the cleanup tail. -4. VM outer loop: **check the nearest handler's kind first**. If - `.cleanup`, do NOT convert catalog→exception, do NOT clear - `last_thrown_context`; stash the original Zig error, restore `sp`, - jump to the cleanup ip. The cleanup bytecode runs (e.g. - `op_pop_binding_frame`) then `op_reraise` returns the stashed error - so the ORIGINAL error (catalog `Info` intact, or thrown value + - context snapshot intact) propagates unchanged. - Idempotency: because the cleanup branch is tested *before* the - conversion branch and stashes immediately before each jump, a - re-raised catalog error meeting an OUTER cleanup handler is stashed - again (never re-converted); meeting an outer CATCH handler converts - exactly once (correct — the catch needs a class to match). -5. compiler: `compileBinding` uses `op_push_cleanup` + `op_reraise`. - `compileTry` with **zero** catch clauses (bare `try` / - finally-only) uses `op_push_cleanup` + `op_reraise`. `compileTry` - WITH catch clauses keeps `op_push_handler` + `op_throw` (the catch - path needs the catalog→exception conversion so `op_match_class` - can match a class name). - -### Scope / out-of-scope - -In scope (D-196 blocker 4): `binding`, bare `try`, finally-only `try`. -Out of scope (separate, no e2e, tracked as a D-196 follow-up note): -the `try`-WITH-catch no-match re-raise edge still converts + can lose -context when an unmatched catalog error escapes a catch inside a -`with-context`. That edge needs the same "re-raise the original, not -the converted, error" treatment but interacts with the -duplicated-finally lowering; deferred to keep this cycle focused on -the binding blocker per the user's one-blocker-at-a-time directive. - -## Consequences - -- VM error-context parity with tree_walk for `binding`/bare-`try` - cleanup edges; D-196 blocker (4) discharged (phase14_with_context + - phase14_user_throw green on `-Dbackend=vm`). -- Two new opcodes (exhaustive `Opcode` switch enforces dispatch arms). -- `compileTry`'s no-catch unit test updates to the new shape - (`op_push_cleanup` / `op_reraise`). -- The cleanup edge now NAMES its `defer` semantics, so a future reader - does not re-derive this analysis (the silence that hid this bug). - -## Alternatives considered - -(Devil's-advocate, fresh-context `general-purpose` fork — verbatim.) - -**F-NNN envelope confirmed:** F-002 (finished-form cleanliness wins; -cycle/diff/LOC is *not* a constraint), F-012 (VM is production default, -tree_walk is the differential oracle, both backends MUST produce -identical observable behaviour). None of the three alternatives below -requires violating an F-NNN; the one design vector that *would* brush -against F-012's parity intent is flagged at the end. - -**Shared root-cause framing.** The bug is two distinct loss points: (a) -the catalog→exception conversion at the handler-unwind drops -`info.context` and collapses the Kind; (b) the throw path nulls -`last_thrown_context` and the cleanup re-throw re-snapshots AFTER the -frame pop (empty). tree_walk avoids both via `defer popFrame` (no -handler interception). The draft's insight — "a binding/finally-only -try cleanup is `defer`, not `catch`" — is the load-bearing observation; -the alternatives differ in where it is encoded. - -**Alt 1 — Smallest-diff: fix the two loss points in place, no new -opcodes.** Keep the existing lowering; (1) seed the synthetic exception -with `info.context` + preserve the original Kind (needs a new ex_info -field since `buildThrownInfo` hard-codes `origin = .thrown` → -`:exception`); (2) make `op_throw` keep the pre-pop snapshot if the -post-pop one is empty, and don't null context at the handler. -*Better:* no opcode growth, no `Handler.kind`, no threadlocal carrier. -*Breaks:* it patches symptoms at both leaks instead of separating the -semantics — the conversion still fires inside `binding`, forcing a new -ex_info field purely to undo a conversion that should never happen on a -cleanup edge (Smallest-diff-bias smell vs F-002). The -`op_throw orelse keep-old-snapshot` heuristic conflates "no context -bound" with "context popped" → a nested rebind-to-empty re-throw leaks -the outer context (silent wrong-context bug, and an *observable* -divergence the opposite direction from the fix = F-012 parity -regression risk). Two channels stay un-unified. - -**Alt 2 — Finished-form-clean (RECOMMENDED): the draft's structure, -cleanup edge as a true pass-through.** Distinguish `.cleanup` from -`.catch_clause`; add `op_push_cleanup` + `op_reraise`. On the cleanup -edge: do NOT convert, do NOT `clearLastError()`, do NOT null context — -`last_error` (Info, context captured at `setErrorFmt`) stays put and -`op_reraise` returns the original Zig error (e.g. `error.ArithmeticError`, -not `error.ThrownValue`); the catalog Info then reaches `renderError` -exactly as in the uncaught tree_walk path → `:kind :arithmetic_error -:request-id "abc"`. For a thrown value, `last_thrown_exception` + -`last_thrown_context` are preserved verbatim (never nulled). Carry the -in-flight error in the threadlocal carrier; `compileBinding` and -`compileTry` with zero catch clauses switch to the cleanup ops; a `try` -WITH catches keeps `op_push_handler` (the catch needs the conversion to -match). *Critical subtlety:* the `try`-with-catch no-match-with-finally -edge must re-raise the original (not converted) error to fully match -tree_walk — the draft under-specifies this; pin it with a differential -case per edge. *Better than draft/Alt 1:* makes the cleanup edge -*semantically* a `defer` (F-012 parity by construction, not symptom -patching); the catalog error is never mutilated, so Alt 1's -Kind-preservation problem evaporates; `op_push_cleanup`/`op_reraise` -NAME the `defer` semantics the current code hid. *Risks:* largest diff -(two opcodes, `Handler.kind`, four compiler arms) — per F-002 not a -reason to downgrade; the try+catch+finally no-match edge is intricate -(mitigate with ~4 diff cases); `op_reraise` returning a *catalog* error -means the outer-loop unwind must be idempotent w.r.t. conversion when -an outer handler exists (assert via a nested-binding-inside-try case). - -**Alt 3 — Wildcard: lower binding/finally cleanup as a dispatcher-level -scope guard, not bytecode.** Record env-frame depth at `eval()` entry; -on any error, a Zig `defer` restores `current_frame` to that depth -(mirrors tree_walk). The binding body compiles to just -`op_push_binding_frame` + body + `op_pop_binding_frame` with NO handler; -the error propagates raw with Info/context intact. *Better:* most -faithful to tree_walk, zero new opcodes, smallest `compileBinding`. -*Breaks:* a `finally` body has observable side effects that must run on -the operand stack BEFORE the error propagates — a dispatcher-level -`defer` cannot run arbitrary user cleanup code in order, so this handles -`binding` but NOT `try`/`finally`, splitting the two cases the draft -unifies (a smell). Restoring frame depth by count couples the VM unwind -to a global mutable invariant (`env_mod.current_frame` chain) ROADMAP -§13 is wary of. Does not address loss point (a) for the try-with-catch -case, so it must be COMBINED with part of Alt 1/2 → not actually -smaller net. - -**Recommendation: Alt 2** — the only option that makes the VM cleanup -edge behave as the `defer` it semantically is, which is what F-012 -parity demands. Per F-002 the larger diff is not a reason to prefer -Alt 1. - -### Decision vs recommendation - -Accept **Alt 2**, scoped to `binding` + zero-catch `try` (the D-196 -blocker-4 surface). The DA's flagged try-with-catch no-match-with-finally -edge is acknowledged as out-of-scope here and recorded as a D-196 -follow-up (no e2e/blocker today); the cleanup mechanism this ADR lands -is the reusable primitive that edge will adopt when it is taken up. diff --git a/.dev/decisions/0072_protocol_extension_over_native_classes.md b/.dev/decisions/0072_protocol_extension_over_native_classes.md deleted file mode 100644 index 5ed0f2de6..000000000 --- a/.dev/decisions/0072_protocol_extension_over_native_classes.md +++ /dev/null @@ -1,177 +0,0 @@ -# ADR-0072 — Protocol extension over native / java classes (native-descriptor identity) - -- Status: Accepted -- Date: 2026-06-02 -- Supersedes: none -- Refs: D-203 (this decision's debt entry), D-202 gap (2) (the - superseded plan), ADR-0036 (dual-backend parity), ADR-0059 - (one-canonical-descriptor-ref invariant), F-002 / F-005 / F-009. - -> Landed 2026-06-02 exactly as the Decision below specifies: -> `class_name.nativeTagFor` accessor + the `analyzeSymbol` -> `symbol_unresolved` fallback arm. 8 e2e cases -> (`test/e2e/phase7_extend_native_class.sh`, incl. `--compare` -> dual-backend) + 2 unit tests on `nativeTagFor`. The SSOT -> consolidation in the Deferred section remains its own cycle (tracked -> in D-203). - -## Context - -cljw cannot `(extend-type java.lang.Long P …)` / `(extend-type String P -…)` — extending a protocol to a built-in / java class. The macro -`expandExtendType` emits `(rt/__extend-type! P impls)`; -`__extend-type!` (`lang/primitive/protocol.zig:163`) requires `` -to evaluate to a `.type_descriptor` Value. A cljw `deftype`/`defrecord` -Name works (it is `def`'d to a TypeDescriptorRef); a class symbol like -`String` or `java.lang.Long` fails to resolve as a symbol. - -### Root cause (the D-202 gap (2) plan was wrong) - -D-202's barrier proposed "resolve the target via -`special_forms.resolveJavaSurface`". That is **incorrect**. Protocol -dispatch maps a primitive receiver (integer `5`, string `"x"`) to a -descriptor via `rt.nativeDescriptor(receiver.tag())` -(`runtime/runtime.zig`) — a per-Tag cached descriptor with fqcn -`"Long"`/`"String"` — which is a **different object** from any -`cljw.java.lang.Long` entry in `rt.types` that `resolveJavaSurface` -returns. So for `(q 5)` to dispatch to the impl, the impl MUST be -registered on `rt.nativeDescriptor(.integer)` — the descriptor the -dispatch path actually resolves — not on a `rt.types` surface -descriptor. `src/runtime/class_name.zig` already carries the needed -relation: `NATIVE_ENTRIES` (name→Tag) + `FQCN_MAP` (FQCN→simple name). -`type_descriptor.makeTypeDescriptorRef(rt, td)` mints the canonical -boxed Value (ADR-0059). - -## Decision - -Resolve a known **native** class symbol to its native TypeDescriptor in -the analyzer's `analyzeSymbol` (`eval/analyzer/analyzer.zig:477`), on -the `symbol_unresolved` fallback — i.e. **after** local + Var -resolution, so a user `(def String …)` / `(deftype String …)` shadows -the class name: - -``` -const v_ptr = ns.resolve(sym.name) orelse { - if (sym.ns == null) if (class_name.nativeTagFor(sym.name)) |tag| { - const td = try env.rt.nativeDescriptor(tag); - return makeConstant(arena, try type_descriptor.makeTypeDescriptorRef(env.rt, td), form); - } - return error_catalog.raise(.symbol_unresolved, ...); -}; -``` - -- Add `class_name.nativeTagFor(name) ?Tag` — an accessor over the - **existing** `NATIVE_ENTRIES` (normalising via `FQCN_MAP` first). NOT - a new table. -- Only `NATIVE_ENTRIES` exact-tag classes resolve - (String/Long/Double/Boolean/Character/Keyword/Symbol/Persistent\*/Pattern). - Interface-shaped names (Number/IFn/IPersistentMap) do **not** resolve - (no single tag) — document as a divergence; ideally raise a hint - error suggesting the concrete types. - -This makes bare `String`/`Long`/`java.lang.Long` evaluate to the same -descriptor `(class 5)` already returns (clj-faithful: a class symbol is -a value) — a coherence requirement, not just a feature: a narrow -target-position-only resolution would leave `(class 5)` printing `Long` -while `Long` alone throws. One locus covers extend-type / -extend-protocol / `lowerDefType` extend sections / the lower-level -`extend` fn / bare-class-as-value. - -### Deferred (NOT bundled) - -The DA recommended also consolidating the name↔Tag relation -(`runtime.zig::nativeFqcnFor` Tag→name vs `class_name.zig::NATIVE_ENTRIES` -name→Tag, which have already drifted: nativeFqcnFor knows -BigInt/Ratio/BigDecimal, NATIVE_ENTRIES does not) into one SSOT. This -is **declined for this cycle** — not on cost grounds (F-002), but -because consolidation would broaden `instance?`/`class` coverage to -BigInt/Ratio/BigDecimal, a separable finished-form decision that risks -an unrelated surface. Tracked in D-203 as its own cycle. - -## Consequences - -- `(extend-type Long P …)` / `(extend-type String P …)` work; the impl - lands where dispatch finds it. Verify via e2e + dual-backend - `--compare`. -- Bare native-class symbols become values (= `(class x)`); clj-faithful. -- `(deftype String …)` shadowing is correct by placement (after Var - resolution). -- Interface names still raise `symbol_unresolved` (acceptable; document). - -## Affected files - -- `src/runtime/class_name.zig` (+ `nativeTagFor`). -- `src/eval/analyzer/analyzer.zig` (`analyzeSymbol` fallback arm). -- e2e `test/e2e/phase7_extend_native_class.sh` (new) + a diff_test case. - -## Alternatives considered - -(Verbatim from the mandatory Devil's-advocate subagent, fresh context, -F-NNN envelope. Recommendation: the Decision above = the DA's "Alt 2" -locus, minus the bundled consolidation which D-203 defers.) - -### Alternative 1 — Smallest-diff: resolve in the macro via `__native-type` - -Leave `analyzeSymbol`/`__extend-type!`/class-as-value untouched. In -`expandExtendType` (+ extend-protocol / lowerDefType callers), if the -target is a bare symbol whose normalised name hits -`class_name.nativeTagFor`, rewrite the target to `(rt/__native-type -:)` (the existing primitive that already mints -`rt.nativeDescriptor(tag)`). Narrowest blast radius (cannot leak into -`(class …)`), zero new dispatch object, covers extend-protocol / -lowerDefType for free (they re-enter expandExtendType). BUT it **cannot -honor shadowing** (macro runs before analysis, no Var visibility, so it -must unconditionally steal `String` from a user `(deftype String …)`), -and bare `Long`-as-value still fails. Smallest-diff-bias trap: trades -away shadowing correctness + class-as-value. - -### Alternative 2 — Finished-form (RECOMMENDED): resolve in analyzeSymbol + consolidate name↔Tag SSOT - -The Decision's locus, plus consolidating the three faces of the -name↔Tag relation (nativeFqcnFor, NATIVE_ENTRIES, the new nativeTagFor) -into one bidirectional SSOT in class_name.zig (Layer 0), with -nativeFqcnFor deriving from it. Placing resolution after Var resolution -is the only locus that simultaneously honors `(deftype String …)` -shadowing without macro-time namespace visibility, covers all entry -surfaces through one path, and makes `Long` agree with the descriptor -`(class 5)` already produces (so the "broad" behavior is a coherence -requirement, not a risk — a narrow resolution would make `(class 5)` -print `Long` yet `Long` alone throw). Costs a bigger diff (touches -class_name.zig + runtime.zig + analyzer.zig); F-002 says recommend it -anyway. One under-specified point in the draft: the SSOT for the -relation, where three faces already drifted (nativeFqcnFor knows -BigInt/Ratio, NATIVE_ENTRIES does not) — fix the consolidation now even -though it enlarges the diff; F-009 puts the relation in neutral Layer-0 -class_name.zig with the analyzer as a pure consumer. Interface-shaped -names (Number/IFn) intentionally do not resolve — make that a -documented, tested divergence with a precise error, not silent. - -> **Main-loop deviation from the DA recommendation**: the Decision -> adopts Alt 2's *locus* but **defers the SSOT consolidation** to D-203. -> Not on cycle/diff grounds (that would be the Cycle-budget defer smell) -> — the consolidation would *broaden* `instance?`/`class` coverage to -> BigInt/Ratio/BigDecimal (which NATIVE_ENTRIES lacks), which is a -> separable finished-form decision with its own surface implications. -> Bundling it into a protocol-extension feature risks an unrelated -> surface. The feature's own finished form (shadow-correct, class-as- -> value, all-loci, native-descriptor identity) is fully delivered -> without the consolidation. - -### Alternative 3 — Wildcard: `__extend-type!` accepts any target shape (Path-A unification) - -Relax `extendType`'s `args[0].tag() == .type_descriptor` requirement; -accept a quoted symbol/string and resolve native-class names in the -primitive (Layer 2, widest legal reach), with the macro auto-quoting -the target like `instance?` already does. Unifies on the `instance?` -"Path A" idiom (one mental model: class symbols auto-quoted + resolved -primitive-side) and sidesteps the "bare-Long-as-value too broad?" -debate entirely. BUT loses class-as-value (same gap as Alt 1, relocated -to the primitive); re-opens a loud-rejection contract the codebase -deliberately closed (`extendType` rejecting non-descriptor args); and -the auto-quote decision for lowerDefType's *user-type* targets needs -namespace visibility the macro lacks (reintroducing Alt 1's -macro-can't-see-Vars discrimination problem) — so `(deftype String …)` -+ `(extend-type String …)` would resolve String as a native class, -ignoring the user deftype. Architecturally interesting (Path-A -unification) but forfeits two finished-form properties the Decision -holds. diff --git a/.dev/decisions/0073_reader_tagged_literal_infra.md b/.dev/decisions/0073_reader_tagged_literal_infra.md deleted file mode 100644 index d5669021d..000000000 --- a/.dev/decisions/0073_reader_tagged_literal_infra.md +++ /dev/null @@ -1,217 +0,0 @@ -# ADR-0073 — Reader infrastructure for EDN tagged literals + data readers - -- Status: Accepted -- Date: 2026-06-02 -- Supersedes: none -- Refs: D-200 (this decision's debt entry), D-075 (reader-metadata - `^:dynamic` — NOT a blocker here, see Decision), ADR-0036 (dual-backend - parity), F-002 / F-009. A separate later ADR owns the `#uuid`/`#inst` - VALUE-TYPE decision (out of scope here). - -## Context - -cljw's reader parses Clojure/EDN syntax but has **no tagged-literal -(`#tag form`) support**: `(read-string "#foo 5")` fails at the tokenizer -(`#`+letter → `.invalid`). `clojure.core/read-string` and -`clojure.edn/read-string` are the SAME 1-arity primitive -(`lang/primitive/edn.zig::readStringFn`) running `reader.readOne` → -`analyzer.formToValue`. The 2-arity `(read-string opts s)` with -`:readers`/`:default`/`:eof` is unimplemented. The data-lift SSOT is -`formToValue(rt, form)` (`analyzer.zig:798`), called from ~8 sites (quote, -meta, collection elements, read-string) with signature `(rt, form)`. - -This ADR is for the **reusable reader infra only**. The `#uuid`/`#inst` -value-type decision (cljw has UUID-as-36-char-STRING, no distinct UUID -type; `Value.Tag.tagged_literal = 24` reserved-but-unused) is a separate -later ADR. The seam holds **provided the infra raises on an unknown tag** -(not a silent placeholder value) — confirmed correct below. - -## Decision (DA Alt 2 — finished-form-clean over the existing dynamic-var stack) - -1. **`FormData.tagged = struct { tag: SymbolRef, form: *const Form }`** — - SymbolRef (not a bare string) so a qualified record-tag `#my.ns/Rec` - reads cleanly later (DIVERGENCE from cljw-v0's bare-string tag). -2. **Tokenizer**: a `.tagged` marker token in `readDispatch`'s else-branch - when `#` is followed by a symbol-start char; otherwise `.invalid`. -3. **Reader `readTagged`**: read the tag symbol token, then the next form; - build `.tagged` (mirrors `readVarQuote`'s marker-then-follow shape). -4. **`formToValue`'s `.tagged` arm is the dispatch home** — NOT analyzer - call-rewrite (cljw-v0), NOT dynamic-var-deref-mid-tokenize (JVM - LispReader). `read-string` is pure data, and `formToValue` is the - pure-data SSOT, so the data-reader application belongs there. The arm - resolves the tag against the **`*data-readers*` dynamic Var** - (`findBinding(var) orelse var.root`), the SAME threadlocal `BindingFrame` - chain `(binding …)` already drives. Found → recursively lift the inner - form to a Value, then `vtable.callFn` the reader fn with it. Not found → - consult `*default-data-reader-fn*`; still nothing → raise - `reader_tag_unknown` ("No reader function for tag {tag}") — clj parity, - NOT a silent `tagged-literal` fallback. -5. **Table home = real `^:dynamic` Vars, not a Runtime mutable field.** - `clojure.core/*data-readers*` (root `{}`) + `*default-data-reader-fn*` - (root `nil`) are interned at bootstrap with `flags.dynamic = true` - (mirroring `cljw.error/*error-context*`, `runtime/error/context.zig:54`); - their `*Var` pointers are cached as two immutable-after-bootstrap `?*Var` - fields on `Runtime` so the Layer-1 `formToValue` can reach them without - an Env. `clojure.edn/read-string` 2-arity `[opts s]` installs - `:readers`/`:default` by **pushing a `BindingFrame`** (`defer popFrame()`) - over those Vars — exactly `(binding [*data-readers* …] …)`'s mechanism. - - This corrects the initial draft, which proposed a mutable - `rt.data_readers` field + bespoke save/restore. That was the - **Smallest-diff bias smell**: cljw already ships the full dynamic-var - `BindingFrame` stack (`env.zig` `pushFrame`/`popFrame`/`findBinding`, - `Var.deref` consulting the chain). The cited blocker (D-075, - `^:dynamic` reader-metadata) is irrelevant — bootstrap interns dynamic - Vars directly (`env.zig:560`). The field+save/restore is also a - nested-read re-entrancy hazard (`#foo "#bar 5"`); a BindingFrame is - re-entrancy-safe by construction and makes user - `(binding [*data-readers* …] (read-string …))` work for free (clj - parity the field would silently drop). - -6. **Unknown tag raises during the infra cycle**: `#uuid`/`#inst` raise - "No reader function for tag uuid/inst" until the value-type ADR lands. - Honest transient state (a `reader_tag_unknown` raise is not a lie / not - a no-op), tracked by the still-open D-200 row. - -## Consequences - -- `(read-string "#foo 5")` → `reader_tag_unknown` (clj parity). -- `(clojure.edn/read-string {:readers {'foo …}} "#foo 5")` applies the user - reader; `:default` handles unmatched tags; `:eof` honoured. -- `(binding [*data-readers* {'foo …}] (read-string "#foo 5"))` works for - free — same Var, same frame chain. -- `Value.Tag.tagged_literal = 24` stays reserved-unused; the later value-type - ADR + `clojure.core/tagged-literal`/`tagged-literal?` activate it (the - point at which `*default-data-reader-fn*` can default to `tagged-literal`, - clj's richer behaviour). Reserving-without-using now is a forward pointer, - not Reservation-as-bias (we do not consult it). -- DIVERGENCE (Guard 2): tag dispatch at `formToValue` time over a dynamic - Var, vs cljw-v0's analyzer call-rewrite and JVM's mid-read deref. - -## Affected files - -- `src/eval/form.zig` (`FormData.tagged` + `typeName`/`format` arms). -- `src/eval/tokenizer.zig` (`.tagged` in `readDispatch` else-branch + `TokenKind`). -- `src/eval/reader.zig` (`readTagged` + `readForm` arm). -- `src/eval/analyzer/analyzer.zig` (`.tagged` arm in `formToValue` + the - `valueToForm` inverse if present). -- `src/runtime/runtime.zig` (two `?*Var` cache fields). -- `src/runtime/error/catalog.zig` (`reader_tag_unknown` Code). -- bootstrap (intern `*data-readers*` + `*default-data-reader-fn*` dynamic Vars). -- `src/lang/primitive/edn.zig` (2-arity `[opts s]` + BindingFrame install). -- e2e `test/e2e/phase14_tagged_literal.sh` (new) + a diff_test case. - -The `binding`/`findBinding`/`pushFrame` machinery is reused unchanged. - -## Alternatives considered - -(Verbatim from the mandatory Devil's-advocate subagent, fresh context, -within the F-NNN envelope. The Decision adopts the DA's Alt 2.) - -**Leading finding (no F-NNN violation):** None of the three alternatives -requires violating an F-NNN. The finished-form-clean shape is fully -buildable today (Alt 2). The draft's central premise ("a Runtime field + -save/restore models clj's dynamic `*data-readers*` without needing -`^:dynamic` infra") is **factually wrong about the codebase**: cljw already -ships a complete threadlocal dynamic-var stack (`env.zig` -`BindingFrame`/`pushFrame`/`popFrame`/`findBinding`, `Var.flags.dynamic`, -`Var.deref` consulting the chain, plus the `binding` special form with -`binding_target_not_dynamic` enforcement). The draft proposes hand-rolling a -parallel, weaker save/restore for a problem the project already solved -generally — the **Smallest-diff bias smell** dressed as pragmatism. - -### Alternative 1 — Smallest-diff: bare-string tag + explicit `formToValue` context param, no table threading - -`FormData.tagged = struct { tag: []const u8, form: *const Form }` (bare -string). `formToValue` gains a `LiftCtx` param (`readers`/`default_fn`) -threaded through ~8 call sites + recursive helpers. edn 2-arity builds the -ctx; 1-arity passes `.{}`. No Runtime field, no save/restore, no dynamic var. - -**Better:** No hidden re-entrancy hazard — an explicit context param is -re-entrancy-safe by construction (each frame owns its ctx), unlike the -draft's mutated shared field which a nested `read-string` (`#foo "#bar 5"`) -or an error long-jump can corrupt. Makes "lifting depends on a reader table" -visible in the signature. - -**Breaks:** Ripple across ~8 call sites + every recursive helper -(`vector/map/set/listFormToValue`, `^meta` recursion, `analyzeQuote` -callback). F-002 says ripple size is not a reason to reject — but it is when -a cleaner shape exists, and cljw already has the dynamic-var mechanism that -makes the ripple unnecessary. Bare-string tag forecloses `#my.ns/Rec` -record-tag readiness. Threading an eval-ish table through `formToValue` -blurs the "pure data lift" character. - -### Alternative 2 — Finished-form-clean (RECOMMENDED): SymbolRef tag + real `*data-readers*` / `*default-data-reader-fn*` dynamic Vars over the existing BindingFrame stack - -(1) `FormData.tagged = struct { tag: SymbolRef, form: *const Form }`. -(2) Tokenizer `.tagged` + reader `readTagged` — same as draft (clean). -(3) `formToValue`'s `.tagged` arm looks the tag up by `env.findBinding` on -the `*data-readers*` Var (the same threadlocal chain `binding` drives), -falling back to the Var root. `*data-readers*`/`*default-data-reader-fn*` -are interned at bootstrap as `^:dynamic` Vars (root `{}` / `nil`, mirroring -core.clj). Found → lift inner form, `vtable.callFn` the reader fn; not found -→ `*default-data-reader-fn*`; nothing → raise `reader_tag_unknown`. -(4) edn 2-arity installs `:readers`/`:default` by pushing a `BindingFrame` -binding those Vars for the read's dynamic extent (`defer popFrame()`). - -**Reaching `env` from `formToValue`:** `findBinding(v: *const Var)` is a free -function on the threadlocal `current_frame` (env.zig:202) — it needs the -*Var pointer*, not an Env. Cache the two Var pointers on the Runtime (two -`?*Var`, resolved once at bootstrap, immutable after) — a smaller, -immutable footprint than the draft's mutable map field. So `formToValue` -reads `findBinding(rt.data_readers_var.?) orelse rt.data_readers_var.?.root`. -No signature ripple, no mutable shared state. - -**Better:** Re-entrancy-safe for free (nested reads push/pop their own -frames; error-unwind still pops via `defer`). It IS clj's model, not a model -"of" it — `binding` over a `^:dynamic` Var is already wired; D-075 is about -reader-metadata `^:dynamic` (attaching `:dynamic` via `^`), irrelevant here -since bootstrap interns these two Vars with `flags.dynamic = true` directly -(env.zig:560/605 precedent). `(binding [*data-readers* …] (read-string …))` -works at the surface with zero extra code — falls out of using the real Var; -the draft's field would silently ignore a user `binding`, a latent parity -bug. SymbolRef gives record-tag readiness. - -**Breaks/costs:** Bootstrap interns two dynamic Vars + caches pointers (two -`?*Var` fields) — slightly more setup than one field, but immutable after -bootstrap vs mutated-per-read. `formToValue` (Layer 1) calling `findBinding` -(Layer 0 free fn on a Layer-0 Var) is clean (eval→runtime allowed; vtable -only needed for `callFn` on the reader fn, which the draft also needs). No -layering violation. - -### Alternative 3 — Wildcard: defer the table; `.tagged` arm constructs a `tagged-literal` (Tag-24) Value, apply readers in a `clojure.core/read-string` layer - -`formToValue`'s `.tagged` arm builds a `TaggedLiteral` Value (activating -reserved Tag-24) carrying `{tag-symbol, lifted-form-value}`, full stop. -Data-reader application moves up into a `clojure.core/read-string` peer that -walks the lifted structure applying `*data-readers*`. Unknown tag with no -reader → the `TaggedLiteral` survives as data (clj's `*default-data-reader-fn*` -= `tagged-literal` is literally this). - -**Better:** Resolves the (c)/(d) tension — clj's default -`*default-data-reader-fn*` + `tagged-literal`/`tagged-literal?` (core.clj -L7961) mean the TaggedLiteral value is a first-class part of the reader -contract, so splitting "infra" from "Tag-24 value type" may be a false seam. -Keeps `formToValue` a pure lift (no fn-calling during lift). - -**Breaks:** Diverges from clj on `read-string` unknown-tag (no opts, nil -default-fn) which *raises* — Alt 3 must replicate the raise in the apply -layer, so it doesn't escape `reader_tag_unknown`, only relocates it. Pulls -Tag-24 value-type design into scope (exactly what the seam defers); -activating a NaN-box slot is F-004-adjacent — doing it as a side effect of -reader infra is the **Reservation-as-bias inverse** (using the slot because -it's there). More layers (Zig reader → Clojure apply) for a feature whose -finished form (Alt 2) is a single dispatch arm. - -### Recommendation (DA) - -**Adopt Alternative 2.** Finished-form-clean *and* lands on infra the -project already built+tested (BindingFrame stack + `binding`), so the -"bigger diff" the draft feared is actually smaller and more correct than the -draft's bespoke save/restore. Per F-002, recommend it regardless of size; -here finished-form and diff size agree. Concretely: (1) drop the -`rt.data_readers` field + save/restore → two `^:dynamic` Vars + cached -`?*Var` pointers, edn installs via BindingFrame; (2) keep SymbolRef tag; -(3) keep `reader_tag_unknown` raise. On the seam (d): infra-ADR vs -value-type-ADR split is defensible **only if the infra ADR commits -unknown-tag = raise** (not a placeholder value) — which it does. diff --git a/.dev/decisions/0074_uuid_value_type.md b/.dev/decisions/0074_uuid_value_type.md deleted file mode 100644 index b33a3f486..000000000 --- a/.dev/decisions/0074_uuid_value_type.md +++ /dev/null @@ -1,222 +0,0 @@ -# ADR-0074 — `#uuid` reads to a real UUID value type (not a string) - -- Status: Accepted -- Date: 2026-06-02 -- Supersedes: none -- Refs: D-200 cycle 3 (this decision's debt), ADR-0073 (the reader infra - that dispatches `#uuid` to a data reader), F-002 / F-004 / F-009, - `no_jvm_specific_assumption`. `#inst`/Date is SPLIT to its own later ADR. - -## Context - -ADR-0073 landed the tagged-literal reader infra: `#tag form` dispatches -through `*data-readers*`, raising on an unregistered tag. To make -clj's built-in `#uuid "…"` reader work, cljw must decide what VALUE it -produces. Today cljw represents UUIDs as **canonical 36-char strings**: -`clojure.core/random-uuid` + `parse-uuid` (`lang/primitive/uuid.zig`) and -`java.util.UUID/randomUUID` (`runtime/java/util/UUID.zig`) all return a -string; the neutral impl `runtime/uuid.zig` has `generateV4`/`format`/ -`parse` (16-byte ⇄ canonical). No `uuid?` predicate exists. - -The decision is **string-parity** (consistent with the shipped string -surface, but lossy) vs a **real UUID value type** (clj-faithful, but -migrates the three string-returning primitives). Full DA analysis below. - -## Decision (DA Alt 2 — real `uuid` heap value type, `#uuid` only) - -Introduce a dedicated UUID heap value and migrate every UUID entry point -to it, because **EDN round-trip is the defining property of a reader -literal** and string-parity permanently breaks it -(`(pr-str #uuid "…")` → `"…"`, not `#uuid "…"`). - -1. **Value**: a `uuid` heap tag reusing the reserved NaN-box slot - `reserved_b15 = 31` (renamed `uuid = 31`). This keeps F-004's - **4×16 = 64-slot shape unchanged** — only a reserved slot is assigned - (a memo, per the project-spirit reservation rule), NOT a layout - amendment, so no F-004 user-amendment is needed. The carrier - (`runtime/uuid/value.zig`, modelled on `runtime/regex/value.zig`) holds - the 16 bytes **inline** (no `gc.infra` payload) + a `meta` slot. -2. **Surfaces** (clj-faithful, all over the neutral `runtime/uuid.zig` - per F-009): print → `#uuid ""` (round-trips); equal → - 128-bit compare; hash → hash the 16 bytes; `(class x)` → a `UUID` - TypeDescriptor (cljw-native, per `no_jvm_specific_assumption`, not a - real `java.lang.Class`); `(instance? java.util.UUID x)` + - `class_name.zig` entry; `clojure.core/uuid?` predicate. -3. **`#uuid` reader**: register a `uuid` entry in the **root** - `*data-readers*` table (so it works without a `binding`, unlike the - user-supplied readers of ADR-0073) → parse via `runtime/uuid.zig` → - the new value. A bad UUID string raises (clj throws on malformed - `#uuid`). -4. **Migrate the three string-returning primitives** to return the new - type — `random-uuid`, `parse-uuid` (nil on bad input, unchanged), and - `java.util.UUID/randomUUID` — so cljw has ONE coherent UUID - representation (`(= (random-uuid) #uuid "…")` is type-comparable, not a - string-vs-type mismatch). `(str )` still yields the canonical - string (UUID toString). - -**SemVer**: the `String → UUID` migration is an observable change, but -v0.1.0 has **not shipped** — paying it now is SemVer-free; deferring it -(string-parity now, promote later) would force a MAJOR break to undo and -is the Cycle-budget-defer smell the DA flagged. - -### Out of scope (split to own later ADRs) -- **`#inst` / `java.util.Date`** — heavier (timezone, `clojure.instant` - grammar, a separate `runtime/time/` impl, F-009-distinct concern - boundary). Its own ADR. -- **Generic `tagged_literal` (slot 24) carrier as the unknown-tag - FALLBACK** — clj's `tagged-literal`/`tagged-literal?` is the *fallback* - for tags with no reader, NOT how `#uuid`/`#inst` are represented - (`(class #uuid …)` is UUID, not TaggedLiteral). Changing ADR-0073's - unknown-tag-raises contract to a non-throwing carrier is a separate - decision; slot 24 stays reserved-unused until then. - -## Consequences - -- `#uuid "…"` round-trips through `pr-str`; `(uuid? #uuid "…")` → true; - `(class #uuid "…")` → UUID; `(= #uuid "a…" #uuid "a…")` → true. -- One UUID representation across reader literal + `random-uuid` + - `parse-uuid` + `java.util.UUID/randomUUID`. -- Existing tests/corpus asserting a STRING from `random-uuid`/`parse-uuid` - are updated in the migration commit (the change is intended). -- One reserved NaN-box slot (`reserved_b15`→`uuid`) is now assigned; - F-004's 64-slot shape is unchanged. - -## Affected files - -- `src/runtime/value/value.zig` (`reserved_b15` → `uuid` tag). -- `src/runtime/uuid/value.zig` (new — the heap carrier) + `runtime/uuid.zig` - (neutral impl already present; add any value-wrap helper). -- GC: `runtime/gc/tag_ops.zig` + mark-sweep registration (trace `meta`). -- `src/runtime/print.zig` (print arm), `src/runtime/equal.zig` (equal + - hash arms), `src/runtime/class_name.zig` (UUID class entry + `instance?`). -- `src/lang/primitive/uuid.zig` (migrate `random-uuid`/`parse-uuid`, add - `uuid?`), `src/runtime/java/util/UUID.zig` (migrate `randomUUID`). -- Default `*data-readers*` registration (`lang/bootstrap.zig` or the uuid - primitive) for the `uuid` reader. -- Tests: `test/e2e/phase14_uuid_literal.sh` + diff_test + unit tests. - -## Alternatives considered - -(Verbatim from the mandatory Devil's-advocate subagent, fresh context, -within the F-NNN envelope. The Decision adopts the DA's Alt 2.) - -### Leading finding: F-004 is NOT a blocker — all options fit the live 64-slot shape - -Verified against `value.zig`: the `Tag` enum is already the 64-slot second -generation. `tagged_literal = 24` exists today (Group B), and Group B has -`reserved_b15 = 31`. A dedicated UUID heap tag consumes one reserved slot; -a combined Date tag a second; activating `tagged_literal = 24` consumes -zero new slots. None requires amending F-004's 4×16=64 shape. F-004 is not -a hard block; the real decision is representation semantics, governed by -F-002 (finished-form) and F-009 (neutral impl). - -### Alternative 1 (smallest-diff): String-parity — `#uuid`/`#inst` read to canonical String / ISO-string - -`formToValue` registers `uuid` → canonical String (reusing -`runtime/uuid.zig`), `inst` → ISO-8601 String. No new heap tag, no -`uuid?`/`inst?` (or they heuristically test `string?` — ugly). - -**Better:** Internal consistency with the already-shipped surface — -`random-uuid`, `parse-uuid`, `java.util.UUID/randomUUID` all return the -36-char String today, so string-parity means `(= (random-uuid) #uuid "…")` -works and cljw has exactly ONE UUID representation; a runtime where `#uuid` -produced a distinct type while `random-uuid` produced a String would be -internally incoherent. Smallest diff: one data-reader registration per tag, -reuses `runtime/uuid.zig` verbatim, no print/equal/hash/class/GC. `#inst` -trivial (ISO string round-trips). - -**Breaks:** Diverges from clj on `(class #uuid "…")` (→ String descriptor); -`uuid?` cannot be `(instance? UUID x)` (no UUID type). **`pr-str` does not -round-trip** — clj prints `#uuid "a…"`, cljw prints `"a…"`; reading -`(pr-str x)` back gives a String, not the literal — breaks EDN round-trip, -the *entire point* of reader literals. A reader-literal feature whose -output does not re-read to itself is not implementing the reader literal; -it is "parse the string at read time and forget the tag." **SemVer trap**: -String-parity now → real type later is a `String→UUID` observable break = -MAJOR; F-002 forbids choosing it on cost grounds. **Smallest diff but -finished-form-dirty**: permanently sacrifices EDN round-trip. - -### Alternative 2 (finished-form-clean, RECOMMENDED): Real `uuid` heap type + migrate the 3 String-returning primitives - -Mint a dedicated UUID heap value (16-byte payload in `runtime/uuid/value.zig` -modelled on `runtime/regex/value.zig` — HeapHeader at offset 0, `gc.alloc`, -trivial finaliser since 16 bytes inline, trace only `meta`). Wire the five -touch-points regex shows: print (`#uuid "…"`), equal (128-bit), hash (16 -bytes), `instance?`/`class` via TypeDescriptor, GC registration. -`formToValue`'s `uuid` reader produces it. Then migrate `random-uuid`, -`parse-uuid`, `java.util.UUID/randomUUID` to return it (else two UUID -representations). - -**Better:** clj-faithful and EDN-round-tripping — `(class #uuid "…")` → -UUID, `(uuid? x)` → true, `(= … …)` → true by 128 bits, `(pr-str …)` → -`#uuid "…"`. Round-trip holds — the feature *is* the reader literal. -Coherent after migration: one representation, no `(= (random-uuid) #uuid …)` -surprise. No SemVer trap going forward — permanent shape. Fits F-004 (one -reserved slot). - -**Breaks:** Changes already-shipped behaviour — `random-uuid`/`parse-uuid` -return String today (Tier A). Migrating to UUID is a `String→UUID` MAJOR -break — BUT no v0.1.0 has shipped, so doing it NOW is SemVer-free; -deferring is the trap. Largest diff (5 arms + 3 migrations + tests), but -F-002 says diff size is not a constraint. `#inst` deferral still open. -**The finished-form-clean alternative; the only "break" is SemVer-timing, -which resolves in its favour because the migration is pre-release.** - -### Alternative 3 (wildcard): Generic `tagged_literal` (slot 24) carrier for ALL tags including `#uuid`/`#inst` - -Activate `Tag.tagged_literal = 24` as a generic `{tag-symbol, payload}` -heap value (matching clj's `tagged-literal`/`tagged-literal?`, -core.clj L7961). `formToValue` wraps every `#tag form` into it; -`uuid?` becomes `(= (:tag x) 'uuid)`. - -**Better:** One mechanism serves `#uuid`, `#inst`, user record-tags, AND -clj's `tagged-literal` API; activates a slot F-004 already reserved for -exactly this; EDN round-trip trivial (the carrier IS `{tag, form}`). - -**Breaks — and why it is wrong for `#uuid`/`#inst`:** clj does NOT -represent `#uuid`/`#inst` as TaggedLiterals — `tagged-literal` is the -*fallback* for tags with no reader; `#uuid`/`#inst` have registered readers -producing real UUID/Date. So `(class #uuid "…")` in clj is `java.util.UUID`, -not `TaggedLiteral`. A generic carrier makes `(class #uuid …)` → -TaggedLiteral and `(uuid? x)` → false — diverges on the exact predicates -the feature exists to satisfy. It conflates the *reader fallback* -(TaggedLiteral, which cljw SHOULD build for unknown tags) with the -*registered-reader result* (real value). **Correct synthesis:** Alt 3 is -right for the FALLBACK path and wrong for `#uuid`/`#inst`. Finished form = -Alt 2's real types for built-in readers PLUS Alt 3's carrier as the -unknown-tag fallback (a separate, later decision — it changes ADR-0073's -unknown-tag-raises contract). The two are different layers, not competitors. - -### Point-by-point - -(a) **Consistency vs fidelity** — internal consistency is a real -finished-form virtue, but Alt 1 buys it by sacrificing EDN round-trip; Alt 2 -buys BOTH consistency and fidelity by migrating the 3 primitives, and a -coherent runtime can be coherent around the *correct* representation just as -easily as the lossy one. Consistency favours Alt 2 + migration. -(b) **Generic carrier vs dedicated** — dedicated is finished-form for -`#uuid`/`#inst` (clj produces real UUID/Date, `class` must match); the -carrier is finished-form for the unknown-tag fallback. Coexist at different -layers. -(c) **`#inst` scope — SPLIT.** `#inst`→Date drags timezone parsing + -`clojure.instant` grammar (non-trivial, instant.clj L100-274) + a Date heap -type. `#uuid` is 16 bytes with a parser that ALREADY exists. Splitting on a -real concern boundary (different impl, `runtime/time/`, F-009-distinct) is -finished-form-clean, not Cycle-budget-defer (which is about diff size). -(d) **SemVer — decisive.** Alt 1 = ship String now, MAJOR break later -(trap). Alt 2 = pay `String→UUID` now, before v0.1.0, SemVer-free. The -pre-release window is exactly when the representation should be nailed down. - -### Recommendation (DA) - -**Alt 2 (real `uuid` heap type) + migrate the 3 String-returning primitives -in the same ADR, `#uuid` only, `#inst` split to its own later ADR.** -Finished-form-clean per F-002: clj fidelity (`class`/`uuid?`/`pr-str` -round-trip), restored coherence (one type across all four entry points), -fits the live F-004 64-slot envelope with no amendment (reserved slot), and -pays the only "break" (String→UUID) inside the SemVer-free pre-release -window. The regex carrier is the cost model (minus the gc.infra payload — -16 bytes sit inline). The generic `tagged_literal` carrier (Alt 3) is the -correct SEPARATE mechanism for the unknown-tag fallback, deferred. `#inst` -split on a real concern boundary. The instinct toward Alt 1 on "consistent + -tiny diff" is the Cycle-budget-defer smell — re-pick Alt 2. diff --git a/.dev/decisions/0075_tagged_literal_value.md b/.dev/decisions/0075_tagged_literal_value.md deleted file mode 100644 index 3c3c1e2e2..000000000 --- a/.dev/decisions/0075_tagged_literal_value.md +++ /dev/null @@ -1,189 +0,0 @@ -# ADR-0075 — Activate slot-24 `tagged_literal` as clojure.lang.TaggedLiteral - -- Status: Accepted -- Date: 2026-06-02 -- Supersedes: none -- Refs: D-200 (the reader-literal debt this completes a piece of), ADR-0073 - (reader infra + the `*default-data-reader-fn*` plumbing this reuses), - ADR-0074 (which deferred this generic carrier), F-002 / F-004 / F-009. - -## Context - -ADR-0073 landed the tagged-literal reader infra (`#tag form` → data-reader -dispatch over `*data-readers*` / `*default-data-reader-fn*`, else raise -`reader_tag_unknown`). ADR-0074 landed `#uuid` as a real value and noted the -generic `tagged_literal` carrier — clj's `clojure.lang.TaggedLiteral` — as -"the correct SEPARATE mechanism for the unknown-tag FALLBACK, deferred." -This ADR is that piece: the `tagged-literal` value type + `tagged-literal?`. - -clj's `TaggedLiteral` is a **fixed two-field class** (`tag: Symbol`, -`form: Object`) `implements ILookup` answering only `:tag`/`:form`: -`(tagged-literal 'foo 5)` → `(tagged-literal? t)`=true, `(:tag t)`=foo, -`(:form t)`=5, `(pr-str t)`=`#foo 5`, `=` by (tag, form), -`hashCode = 31*hash(tag)+hash(form)`. `*default-data-reader-fn*` defaults to -`nil` (so unknown tags throw) — this ADR does NOT change that default. - -## Decision (DA Alt 2 — a dedicated slot-24 heap struct) - -clj's TaggedLiteral is a fixed two-field class (NOT a deftype), so the -faithful finished form is a dedicated two-field Zig struct, not a `.clj` -deftype reusing `typed_instance` (which would drag the whole protocol stack -into the Layer-1 reader fallback and still hand-roll the print form). - -1. **Value**: `runtime/tagged_literal.zig` heap carrier - `extern struct { header, tag: Value, form: Value }` on the day-1-named - slot `tagged_literal = 24`, modelled on `runtime/uuid.zig`. GC **trace** - marks **both** `tag` and `form` (both are GC Values); **no finaliser** - (nothing on `gc.infra`, unlike regex's payload / uuid's inline bytes). -2. **Constructors / predicate** (`clojure.core`): `tagged-literal` - (2-arity, `[tag form]` — `tag` must be a symbol) + `tagged-literal?` - (O(1) tag compare, not an `instance?` walk). Both Zig builtins, so the - reader fallback (`liftTagged`'s `invokeReaderFn` builtin fast path) reaches - them without the protocol stack being hot. -3. **ILookup-only** (NOT map-like — no assoc/keys/seq/count, matching clj): - a dedicated `valAt(v, key, not_found)` in `tagged_literal.zig` answering - `:tag`/`:form`/not-found, wired into the `get` + keyword-invoke path - (`collection/lookup.zig`, the same single-source pattern `recordGet` uses - for `typed_instance`). -4. **print** arm: `#` + tag pr-str + " " + form pr-str (→ `#foo 5`, EDN - round-trips). **equal** arm: `=`(tag) ∧ `=`(form). **hash**: - `31*hash(tag)+hash(form)` (match clj where observable). -5. **No change to ADR-0073's default unknown-tag-raises contract.** The - carrier is reachable two clj-grounded ways: the `tagged-literal` - constructor, and `(binding [*default-data-reader-fn* tagged-literal] - (read-string "#unknown 5"))` — the existing plumbing already passes - `(tag-symbol, form-value)` to the default fn. - -### Slot allocation note (DA fact-correction) - -Slot 24 is NOT the last reserved slot (Group B/C/D carry unwired declared -entries — `reader_conditional`=25, `host_instance`=29, `reified_instance` — -plus D-043 anonymous reserves). So this is NOT scarcity-driven and NOT -Reservation-as-bias: the justification is "clj has a distinct fixed-shape -TaggedLiteral value and cljw wants the faithful analogue," not "the slot -exists, fill it." Activating a day-1-named slot is WITHIN F-004's 64-slot -shape (the loop's call); the shape is unchanged (no F-004 amendment). -`#inst`/Date (D-200's remaining piece) does NOT need a fresh slot — its -natural cljw home is `host_instance` (a `java.util` class). - -## Consequences - -- `(tagged-literal 'foo 5)` / `(tagged-literal? x)` / `(:tag x)` / `(:form x)` - / `(pr-str …)` → `#foo 5` / `=` by (tag, form) — clj-parity. -- `(binding [*default-data-reader-fn* tagged-literal] (read-string "#x 5"))` - carries unknown tags instead of raising (opt-in; default still raises). -- `reader_conditional` (slot 25) is the sibling reader-support carrier; a - `.cljc`-gated follow-up (tracked in D-200's remaining notes) co-designs it. - -## Affected files - -- `src/runtime/value/value.zig` / `heap_tag.zig` (slot 24 already named; wire - the value behaviour). -- `src/runtime/tagged_literal.zig` (new — carrier + valAt + trace + GC hooks). -- `src/runtime/runtime.zig` (registerGcHooks). -- `src/runtime/print.zig` (print arm), `src/runtime/equal.zig` (equal + hash), - `src/runtime/collection/lookup.zig` (`.tagged_literal` ILookup arm). -- `src/lang/primitive/core.zig` (or a peer): `tagged-literal` / `tagged-literal?`. -- Tests: `test/e2e/phase14_tagged_literal.sh` (+cases) + a unit test. - -## Alternatives considered - -(Verbatim from the mandatory Devil's-advocate subagent, fresh context, within -the F-NNN envelope. The Decision adopts Alt 2 with all fact-corrections.) - -### Fact corrections the draft absorbed (verified against source) - -1. **"Slot 24 is the ONLY reserved-but-unused slot left" is false.** - `heap_tag.zig` shows `tagged_literal = 24` is a *named day-1 entry* in - Group B (B8, between `regex` B7 and `reader_conditional` B9). The header - still lists `reader_conditional`, `reified_instance`, `host_instance` as - declared-but-unwired, and D-043 tracks anonymous reserves. Claiming slot 24 - does not exhaust the space; the "last slot" urgency evaporates — the - decision rests on feature value, not scarcity. -2. **`tagged-literal` is NOT a deftype in clj** — it is - `clojure.lang.TaggedLiteral`, a hand-written class `implements ILookup` - with two fields, `valAt` returning only `:tag`/`:form`/notFound, structural - `equals` (tag+form), `hashCode = 31*hash(tag)+hash(form)`. Print is `#` + - tag + space + form (NOT `#`; the draft's print arm was wrong). -3. **`*default-data-reader-fn*` defaults to `nil`** → unknown tag throws. The - draft's "no contract change to ADR-0073's raise" is correct — confirmed. - -### Alternative 1 — smallest-diff: reuse `typed_instance` (define `TaggedLiteral` as a deftype in `lang/clj/`) - -Don't touch slot 24. Define a `(deftype TaggedLiteral [tag form] ILookup …)` -in bootstrap `.clj` + `tagged-literal`/`tagged-literal?` over it; `(:tag t)`/ -`(:form t)` ride the already-wired `recordGet` declared-field path. - -**Better:** Near-zero Zig — no new heap struct, trace, print/equal/hash arms, -or ILookup wiring (the `typed_instance` paths exist + are tested). F-009-clean -by construction. - -**Breaks:** (1) The Layer-1 reader fallback (`liftTagged`) calls -`*default-data-reader-fn*` at analyze time; a deftype-backed `tagged-literal` -lives in Layer-2 `.clj`, reachable only via `rt.vtable.callFn` — works, but -makes the fallback depend on the whole protocol stack being live, a heavier -prerequisite than a Zig primitive. (2) deftype default print is -`#user.TaggedLiteral{…}` unless you wire a print-method — so you still -hand-roll the `#tag form` print, just in `.clj`. (3) `=`/`tagged-literal?` -correctness depends on deftype's structural-equality + `instance?` wiring. -**Anti-Reservation-as-bias** (its strongest argument): refuses to claim slot -24 just because the enum names it. - -### Alternative 2 — finished-form-clean (RECOMMENDED): dedicated slot-24 heap struct - -The draft, corrected: `runtime/tagged_literal.zig` `{header, tag: Value, -form: Value}` on slot 24, modelled on uuid.zig (trace marks **both** fields, -**no finaliser** since both are GC Values). `tagged-literal`/`tagged-literal?` -builtins. ILookup via a dedicated `valAt`. Print `#` + tag + " " + form; -equal (tag,form); hash `31*hash(tag)+hash(form)`. - -**Better:** (1) Matches clj's actual representation — a fixed two-field class, -not a deftype; the Zig struct is the faithful analogue. (2) The Layer-1 reader -fallback needs no Layer-2 dependency (`tagged-literal` as a builtin takes -`invokeReaderFn`'s `.builtin_fn` fast path), the cleaner bootstrap story. -(3) `tagged-literal?` is O(1), not an `instance?` walk. (4) Self-contained -(~40 lines of arms, grep-discoverable by keyword per `feature_name_consistency`). - -**Breaks:** (1) Permanently spends slot 24 — but per fact-correction #1 slots -are not scarce, so the cost is small. (2) ILookup wiring is 2 call-sites (the -keyword-invoke branch ends `else => unreachable`, only `typed_instance` is -special-cased) — route `.tagged_literal` through a dedicated `valAt`. (3) Mild -shape-overlap with `reader_conditional` (slot 25, also a small Value-field -carrier) — but clj keeps them distinct classes, so clj-parity argues for two. -**(b) Minimal ILookup surface:** answer ONLY `:tag`/`:form`/notFound (clj's -`valAt`); do NOT make it map-like (no assoc/keys/seq) — that would be -divergence-creating over-reach. - -### Alternative 3 — wildcard: defer entirely; transient stub, claim no slot now - -Leave `tagged-literal`/`tagged-literal?` raising `feature_not_supported` -(permitted transient stub); file a debt row scheduled to co-implement with -`reader_conditional` (slot 25) in one big-bang push (per `clj_diff_sweep.md` -Discipline 2) when `.cljc` lands. - -**Better:** Avoids spending a slot on a feature with (arguably) no current -consumer; co-designs both reader-support carriers coherently; honours the -Micro-coverage-grind warning. - -**Breaks:** (1) Leaves a `feature_not_supported` stub on core-since-1.7 fns -(`tagged-literal`/`tagged-literal?`) — a visible hole. (2) The "no consumer" -claim is weak — EDN round-trip of unknown tags via `*default-data-reader-fn* -tagged-literal` is a real defensive-config idiom; deferring keeps a -`clj_diff_sweep` DIFF open. (3) `.cljc`/reader-conditional may be many Phases -out; binding `tagged_literal`'s fate to it defers a cheap, useful, clj-grounded -type — a Progress-by-coupling risk. - -### Recommendation (DA) - -**Alt 2 (dedicated slot-24 struct)** with all fact-corrections. clj's -TaggedLiteral is a fixed two-field class, so the dedicated struct is the -faithful finished form; Alt 1's deftype reuse is the smaller-diff convenience -the finished-form owner would unwind (protocol-stack dependency in the -Layer-1 fallback + hand-rolled print). The slot cost is not the scarcity the -draft claimed. Alt 3's defer loses on the weak "no consumer" claim + the open -clj-parity DIFF; if honouring big-bang, the better move is Alt 2 now + note -the slot-25 `reader_conditional` sibling for a future co-design. Corrections -that MUST land: print is `#`+tag+" "+form (no angle brackets); ILookup-only -(`:tag`/`:form`); hash `31*hash(tag)+hash(form)`; trace marks both fields, no -finaliser. No F-NNN is violated (activating a day-1-named slot is within -F-004's shape). diff --git a/.dev/decisions/0076_clj_parity_campaign_and_accepted_divergences.md b/.dev/decisions/0076_clj_parity_campaign_and_accepted_divergences.md deleted file mode 100644 index 327099994..000000000 --- a/.dev/decisions/0076_clj_parity_campaign_and_accepted_divergences.md +++ /dev/null @@ -1,205 +0,0 @@ -# ADR-0076 — clj-parity root-cause campaign + accepted-divergence framework - -**Status**: Proposed → Accepted (2026-06-02, user-directed) - -## Context - -The post-M F-010 quality loop runs a differential sweep (cljw vs the real -`clj` oracle, F-011). A periodic audit surfaced the user's concern verbatim -(2026-06-02): - -> 「たまにしか監査しないけど、そういう細かい不一致が、利用されるときの -> 不信感につながりそうだと思ってます。…まず、ねじこみで、それらを根本的に -> 解決する調査と取り組みをつぶし、また、そういう妥協や先送りを、A系は解消。 -> Bは…妥当な差異なので、しっかり許容した、と記録…ようにルール化や自動防御 -> したい。次のクリアセッションからさっそく対処がねじ込まれるよう、配線や参照 -> チェーンを準備した…」 - -The divergences split two ways: - -- **A — real gaps** that visibly differ from clj where clj succeeds (a user - could hit them and lose trust). Several were *deferred* as "structural" or - "user-gated" — the user is now lifting that deferral and directing - root-cause resolution (F-002: finished-form wins; big surgery is the - default, not the exception). -- **B — intentional divergences** that are correct for cljw's design (no-JVM, - numeric-tower, unordered-collection semantics). These must be *recorded as - designed* so they neither erode trust nor get accidentally "fixed". - -## Decision - -Two faces of one discipline, in one ADR (DA Alt 2 + Alt 3's corpus-pin): - -### (1) Accepted-divergence framework (the B half) - -- **SSOT** `.dev/accepted_divergences.yaml` — AD-001…AD-007, each with - `summary` / `example` / `why` / **`derives_from`** (an F-NNN or ADR) / - **`pin`** (a corpus/e2e test locking cljw's divergent value). -- **Rule** `.claude/rules/accepted_divergences.md` — every clj DIFF is - classified **bug→fix** OR **accepted→AD-NNN with a `derives_from`**, never - left floating (the Defer-to-amnesia smell). Auto-loads on runtime/harness - edits. -- **Gate** `scripts/check_accepted_divergences.sh --gate` (in - `test/run_all.sh`) — enforces `derives_from` present + `pin` paths exist + - COVERAGE.md points at the SSOT. The **pin reuses the existing - `check_corpus_regression.sh`** (a corpus line carrying cljw's expected - value) rather than a second regression runner (F-011, anti-D-177): if a - future change flips an accepted divergence, the corpus regression fails → - a conscious decision is forced. - -### (2) clj-parity root-cause campaign §9.2.P (the A half) - -A ROADMAP campaign overlay (mirrors §9.2.S perf campaign; does not renumber -§9.2.R), anchored as a standing **`quality-loop floor: clj-parity`** debt -Barrier so it is repeatable and cannot rot half-done (F-010). The seven A -items, with the DA's loop-resolvable-vs-user-owned split made HONEST (no -false "root-cause" promise on items that need a user F-NNN amendment): - -| Unit | Debt | Gap | あるべき論 (finished form) | Owner | -|-----------|-------|-------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------|------------------------| -| C1 (lead) | D-164 | empty `()`/seq collapses to `nil`; `(seq? '())`→false | interned distinct empty value inside existing `.list`/seq tags (no new slot); uniform across every seq fn | **loop** | -| C2 | D-205 | BigDecimal map-key `(get {1.5M :v} 1.5M)`→nil | numeric arm in rt-free `keyEqValue` + scale-normalized hash, or canonical-scale at construction | **loop** | -| C3 | D-207 | `.toString`/`.equals`/`.hashCode`/`.getClass` unimplemented | dispatch-level Object fallback → `str`/`=`/`hash`/`class` (F-009 wrapper) | **loop** | -| C4 | D-209 | `map-entry?` + distinct MapEntry | activate the **reserved** Group-A `.map_entry` slot (consuming a reservation ≠ amending F-004) | **loop** | -| C5 | D-198 | `(Exception. "x")` host-class ctors | D-048 host-class machinery (dependency-ordered) | **loop** (after D-048) | -| C6 | D-200 | `#inst`/Date | ship the **no-slot** cljw-native `typed_instance` Date (loop-resolvable); a dedicated `.date` slot is a **user F-004** option | **loop** + user (slot) | -| C7 | D-165 | long ∈ (2^47, 2^63] prints as BigInt `…N` | full-Long needs a wider NaN-box payload (F-004) or heap-boxed-Long-without-`N` (F-005) — **both user-owned LAW** | **user F-NNN** | - -**Honest scope**: C1–C5 are loop-resolvable now. C6 ships the no-slot Date -autonomously; the dedicated-slot variant + C7 require a **user F-004/F-005 -amendment** and are surfaced as decision points, NOT auto-decided (treating -F-004/F-005 as informational would be the Smallest-diff-bias smell on -user-LAW). The campaign records C7 (and the C6 slot variant) as a debt row -flagged `user-owned F-NNN decision`, not as an accepted divergence and not -as a false root-cause promise. - -**Order**: C1 (highest leverage — one fix, every seq fn) → C2 → C3 → C4 → -C6(no-slot) → C5(after D-048). C7 + C6-slot await the user's F-NNN call. - -## Consequences - -- The next clean session resumes onto C1 (handover Resume contract + the - `quality-loop floor: clj-parity` Barrier wire it as the first work). -- Each unit is its own commit (revert-friendly), F-002/F-011-held, gated, - and leaves a corpus line behind (anti-D-177 mechanical re-check). -- B divergences are locked + justified; a future sweep sees AD-001…007 as - ACCEPT, not noise, and cannot silently flip them. -- D-164/D-165 move from the "discharged-structural-defer" limbo back to - active campaign units with honest ownership. - -## Affected files - -- new: `.dev/accepted_divergences.yaml`, `.claude/rules/accepted_divergences.md`, - `scripts/check_accepted_divergences.sh` -- edit: `test/run_all.sh` (gate), `test/diff/clj_corpus/COVERAGE.md` (point at - SSOT), `.claude/rules/clj_diff_sweep.md` (cross-ref), `.dev/ROADMAP.md` - (§9.2.P), `.dev/debt.yaml` (D-164/165/198/200/205/207/209 statuses + - `quality-loop floor: clj-parity` anchor), `.dev/handover.md` (Resume - contract). - -## Alternatives considered - -(Devil's-advocate subagent, fresh context, per CLAUDE.md depth-≥2 mandate. -Verbatim; full copy in `private/notes/da-adr-0076.md`.) - -### Alt 1 — Smallest-diff: two ADRs, ship only the framework now; campaign is a placeholder - -ADR-0076 records ONLY framework (a); a separate ADR-0077 opens §9.2.P as an -un-accepted placeholder, each A-item un-defer being its own later ADR. - -**Better:** clean separation of a permanent mechanism (framework) from a -transient schedule (campaign); lets `check_accepted_divergences.sh` go green -immediately with AD-001..007 without waiting on hard structural work; each -A-item gets its own fresh-context DA fork when it actually fires. - -**Breaks/risks:** the campaign loses its teeth — a bare placeholder with no -acceptance is exactly the decision-deferral the user is directing the loop to -stop, i.e. the **Cycle-budget defer smell** in process-hygiene costume, and -re-opens the drip-feed (Micro-coverage-grind) the clj_diff_sweep "big-bang" -Discipline 2 forbids. The classify-rule is half a loop without the campaign -saying which deferred DIFFs are bugs vs accepted, leaving the A-items in the -floating-DIFF state the rule bans. **Not recommended on F-002 grounds.** - -### Alt 2 — Finished-form-clean: one ADR; campaign as a standing F-010 `quality-loop floor: clj-parity`; route each A-item to its true owner - -One ADR (framework + campaign are two faces of one discipline). Campaign is a -debt-driven standing floor (repeatable per F-010, cannot rot half-done) -rather than a one-shot phase with seven pre-decided directions. The SSOT -carries both `accepted` and `bug-deferred` statuses (so "never floating" is -gate-checkable). The pin is generalised to a **corpus line carrying cljw's -expected value, enforced by the already-existing `check_corpus_regression.sh`** -(no second regression runner — F-011). Each A-item routes to its real -structural owner. - -**Better:** resolves the F-003 tension head-on instead of papering over it — -the user lifting the deferral un-defers the *work*, but the *representation -choice* for the two F-004/F-005-touching items still routes to the user as -F-NNN owner because the loop literally cannot pick a clean shape there without -amending user-LAW. Reuses the corpus regression mechanism as the pin (F-011, -anti-D-177). Keeps the campaign repeatable so it cannot rot. - -**Breaks/risks:** larger up-front mechanism (not an F-NNN cost — F-002 -accepts large finished-form diffs); surfaces user-owned amendment touchpoints -for D-165 and the D-200 dedicated-tag (the loop does NOT halt — it ships the -loop-resolvable shapes and records the rest as user-owned). **Recommended per -F-002.** Critical correction over the draft: D-165 and a dedicated #inst slot -cannot be loop-resolved without a user F-004/F-005 amendment, so the campaign -must classify them as user-owned rather than promise root-cause, or it ships -a false-positive-discharge lie. - -### Alt 3 — Wildcard: no separate SSOT; fold both halves onto the corpus + debt machinery, let the oracle be the gate - -No `accepted_divergences.yaml` / `check_accepted_divergences.sh`. An accepted -divergence is a corpus line with an inline `# AD: [derives: F-NNN]` -annotation recording cljw's value; `check_corpus_regression.sh` auto-pins it; -a grep generates the catalog. Campaign rides debt.yaml `quality_floor: -clj-parity` + the existing F-010 floor. - -**Better:** strongest F-011 reading — zero new SSOT, zero new gate; the pin is -intrinsically the regression test; no parallel scheduling structure. - -**Breaks/risks:** loses structured `derives_from`/`why` enforcement (a comment -convention is far weaker than a gate checking "every AD has a derives_from"); -conflates a regression *fixture* with a design *record* (burying AD rationale -in test comments, making COVERAGE.md point at scattered files); property-shaped -divergences (set print-order, error-Kind) have no natural single-expression -corpus line. **Best as a contributor to Alt 2 (corpus line as the pin), not -the whole shape.** - -### Decision vs the alternatives - -**Alt 2 chosen, folding in Alt 3's corpus-line-as-pin** (the `pin` field -points at the corpus/e2e that locks the value; the gate checks existence, the -existing `check_corpus_regression.sh` does the re-run — no second runner). -Alt 1 rejected: it re-introduces the deferral the user is directing the loop -to eliminate (F-002). The DA's leading flag is honoured: **C7 (D-165) and the -C6 dedicated-`.date`-slot have no finished-form-clean shape inside the current -F-004/F-005 envelope — each needs a user amendment; the campaign records them -as user-owned, ships the 5 loop-resolvable items + the no-slot Date, and does -not halt.** - -## Revision history - -- 2026-06-02 created (Accepted): user-directed. Framework + campaign landed - as wiring this session; C1… execution begins next clean session. -- 2026-06-02 amendment 1 (am1): the two "user-owned F-NNN" flags resolved by - user decision in chat, so **all 7 units are now loop-resolvable**: - - **C6 #inst** — user confirmed the **no-slot `typed_instance` Date** (β); - no dedicated `.date` slot. `(class …)`→`Date` is consistent with AD-003 - (simple class names). C6 is fully loop. - - **C7 D-165 (long range)** — user decided **heap-boxed Long, representation - B2** (a `Long-origin/fits-i64` flag on the heap-integer struct; **NO new - NaN-box slot, F-004 layout UNCHANGED**). This is NOT an F-NNN amendment: - F-005's stated surface already wants Long-to-full-i64, and the internal - representation is F-005's "Zig-native" domain (loop's call). NaN-box - i64-inline was confirmed fundamentally impossible (cw v0 also i48; an - 8-byte Value cannot hold tag + i64). Heap-box mirrors the JVM (`Numbers.num(long)`→`Long.valueOf`). - - **Long overflow past i64** — user ratified **auto-promote** ("A は Promote - 推奨どおり"): cljw `+`/`*` promote to BigInt on i64 overflow (per F-005's - existing promote-clause — **F-005 NOT amended**). clj's checked-`+`/`*` - THROW (`Math.addExact`) is recorded as the accepted divergence **AD-008**. - The promoting `+'`/`*'` family is unimplemented + F-005's `+'`-clause is - clj-inverted → deferred to **D-211**. - - Net: no user-LAW amendment was needed (F-004 + F-005 unchanged); the - campaign is now 7 loop-resolvable units. The C7-as-user-owned text in the - Decision table above is superseded by this am1. diff --git a/.dev/decisions/0077_bigdecimal_scale_independent_key.md b/.dev/decisions/0077_bigdecimal_scale_independent_key.md deleted file mode 100644 index 6e9915099..000000000 --- a/.dev/decisions/0077_bigdecimal_scale_independent_key.md +++ /dev/null @@ -1,181 +0,0 @@ -# ADR-0077 — BigDecimal scale-independent map key via a cached normalized form - -**Status**: Proposed → Accepted (2026-06-03, clj-parity campaign C2 / D-205) - -## Context - -ADR-0076 §9.2.P clj-parity campaign unit **C2 (D-205)**: BigDecimal cannot be a -map key / set element. `(get {1.5M :v} 1.5M)` → cljw nil (clj `:v`); -`(contains? {1.5M :v} 1.5M)` → cljw false (clj true). BigInt and Ratio were -fixed earlier (2026-06-02) by adding numeric arms to the **rt-free** -`equal.zig::keyEqValue`; BigDecimal was explicitly punted because value-equality -across different scales needs scale alignment (multiply an unscaled BigInt by -10^Δscale = allocation), which a rt-free function cannot do. - -Two functions block BigDecimal keys, BOTH rt-free (they run deep in the HAMT -lookup hot path — D-151 invariant: the key path must not require an allocator): - -- `keyEqValue` `.decimal` arm — falls through (rt-free scale alignment impossible). -- `valueHash` `.big_decimal` arm — hashes `(unscaled, scale)` directly, so - `1.5M`=(15,1) and `1.50M`=(150,2) land in different HAMT buckets. - -clj semantics (oracle-verified): the BigDecimal key/hash is **scale-independent** -(stripTrailingZeros — `1.5M`/`1.50M`/`12000M`/`12000.00M` collide), `=` is -NUMERIC (`compareTo==0`), and PRINT **preserves scale** (`(pr-str 1.50M)` → -"1.50M"). So the value has two projections: scale-dependent (print / arithmetic -result-scale / `.scale` / `.precision`) and scale-independent (key / hash / `=`). - -The sibling heap numerics work rt-free because they are **canonical at -construction**: Ratio is gcd-reduced, so equal ⇒ identical fields ⇒ a direct -rt-free field compare. BigDecimal lacks that property — and, unlike Ratio, it -*cannot* simply adopt it, because scale is JVM-observable (java.math.BigDecimal -itself stores scale non-canonically; `1.50M` keeps scale 2). - -## Decision - -Add a **cached normalized (stripped-trailing-zeros) projection** to the -BigDecimal heap struct, computed once at construction (where `rt` exists): - -``` -pub const BigDecimal = extern struct { - header: HeapHeader, - _pad: [6]u8, - unscaled: *BigInt, // AUTHORITATIVE value (JVM BigDecimal): print / arithmetic / .scale - scale: i32, - norm_scale: i32, // the stripped scale ┐ the Clojure scale-independent - norm_unscaled: *BigInt, // the stripped unscaled┘ key/hash projection (cached) -}; -``` - -- **Construction** (`allocFromManagedScale`) strips trailing zeros - (`while unscaled≠0 and unscaled%10==0: unscaled/=10, scale-=1`; zero → (0,0)) - and stores the result as `(norm_unscaled, norm_scale)`. The original - `(unscaled, scale)` is left untouched — it remains the authoritative value for - print, arithmetic result-scale, and any scale accessor. -- **`keyEqValue` `.decimal` arm**: `norm_scale_a == norm_scale_b && - norm_unscaled_a.eql(norm_unscaled_b)` — rt-free (BigInt const compare, no - allocation), like the Ratio arm. -- **`valueHash` `.big_decimal` arm**: hash `(norm_unscaled, norm_scale)`, so - `1.5M`/`1.50M` collide in the same bucket (scale-independent, matching clj). -- **`traceGc`** marks `norm_unscaled` too (it is GC-managed). -- **Print / arithmetic / `compareValue` unchanged** — they read the - authoritative `(unscaled, scale)`, so the verified scale-faithful behaviour - (`(pr-str 1.50M)`→"1.50M", `(+ 1.50M 0.5M)`→"2.00M") is untouched (F-011). - -This mirrors the real two-layer model: java.math.BigDecimal keeps scale -authoritative; Clojure's hasheq/`=` derives a stripped view. cljw caches that -derived view because it cannot recompute it on the rt-free hot path (the same -"cache an expensive-to-derive key projection" pattern as a cached hashCode). - -## Consequences - -- `(get {1.5M :v} 1.5M)`/`(get {1.5M :v} 1.50M)`/`(contains? #{1.5M} 1.50M)` → - match clj. `(hash 1.5M)` == `(hash 1.50M)` (scale-independent). -- +1 pointer (`norm_unscaled`) +1 i32 (`norm_scale`) per BigDecimal; a strip - loop + one extra BigInt alloc at construction; one extra GC mark edge. Paid - at construction (cold), never on the rt-free key lookup (hot). -- Closes D-205 (the BigDecimal remainder; BigInt/Ratio already done). Leaves a - corpus line. Next campaign unit: C3 = D-207. - -## Affected files - -- `src/runtime/numeric/big_decimal.zig` (struct + normalize-at-construction + - traceGc + accessors + layout test). -- `src/runtime/equal.zig` (keyEqValue `.decimal` arm + valueHash `.big_decimal` - arm). -- `test/diff/clj_corpus/bigdecimal_key.txt` (new corpus, clj-verified). - -## Alternatives considered - -(Devil's-advocate subagent, fresh context, per CLAUDE.md depth-≥2 mandate. The -subagent RECOMMENDED Alt 2; the main loop chose Option A — rationale below the -alternatives. Within the F-005 envelope; cycle/LOC is not a constraint.) - -### Alt 1 — Smallest-diff: precomputed scale-independent hash (u32) + rt-free streaming digit compare - -Add only `norm_hash: u32` (fits the existing `_pad2[4]` → zero net struct -growth); `valueHash` returns it; `keyEqValue` does an rt-free streaming -digit-aligned compare of the two unscaled values across the 10^Δscale shift. - -**Better:** smallest struct (no second GC pointer, `traceGc` untouched); hot -`valueHash` is a single field load. - -**Breaks/risks:** the streaming compare is the load-bearing novelty and is hard -to get right rt-free — base-10 scaling does not map to base-2^64 limb shifts, so -it degrades to schoolbook digit extraction (repeated divmod 10) which may itself -need scratch space; if it overflows a fixed buffer it reintroduces the allocator -dependency it was meant to avoid. A bug here is a *silent wrong-bucket / -wrong-equality* in the HAMT — the worst failure class. Concentrates all -difficulty in the one allocator-free corner. Viable fallback only if its -rt-free-ness is proven against a fixed scratch buffer with a documented -overflow-to-error path. - -### Alt 2 — Finished-form-clean (DA-RECOMMENDED): normalize the stored form + a separate `print_scale` - -Make the *stored* `(unscaled, scale)` the canonical stripped form (like Ratio); -add `print_scale: i32` recording the display scale; print re-pads. Then -`keyEqValue`/`valueHash` need **no** decimal-specific arm — the generic -canonical-numeric pattern works, matching the BigInt/Ratio sibling invariant -("equal ⇒ identical fields"). - -**Better:** the truest finished-form *if* BigDecimal were naturally canonical — -no shadow fields, no decimal-specific equal/hash arm, +1 i32 and no second GC -pointer, uniform with the other heap numerics (F-011). - -**Breaks/risks (the DA flagged these):** the **largest blast radius** and -F-005-observable. Arithmetic result-scale is JVM-observable: `(+ 1.50M 0.5M)` → -`2.00M` (scale = max of operand scales). `alignedCombine` computes -`result_scale = @max(ad.scale, bd.scale)`; if `scale` is canonical (stripped), -`1.50M` stores scale 1 and the max-rule breaks scale-preservation. So ALL -arithmetic (`allocAdd/Sub/Mul/allocQuotient`/`alignScales`), print, and any -scale/`precision` accessor must migrate to `print_scale`, and `.precision` -depends on the *unstripped* magnitude. A partial migration = a silent F-005 -violation (a canonical value printed without its trailing zeros). The DA's own -fallback note concedes this subtlety. - -### Alt 3 — Wildcard: intern/canonicalize BigDecimals via an rt-side dedup table - -Construction returns the same heap pointer for numerically-equal BigDecimals; -the existing `@intFromEnum(a)==@intFromEnum(b)` identity short-circuit then -handles keys for free. - -**Better:** cleanest hot path (zero new equal/hash logic). - -**Breaks/risks — FATAL under F-005:** interning by numeric value collapses -`1.5M` and `1.50M` to one pointer → only one scale survives → `(pr-str 1.50M)` -can no longer be "1.50M". Interning by `(unscaled, scale)` keeps print but makes -them distinct pointers again (identity short-circuit does not fire cross-scale) -→ back to needing a real arm. Also a global mutable GC-aware table on the hot -construction path. Dead end. Rejected. - -(Brief (e) "thread rt into the HAMT key path" — rejected against the D-151 -rt-free invariant: a depth-4 change to the central data-structure contract to -serve one numeric type.) - -### Decision vs the alternatives — why Option A over the DA's Alt 2 - -The DA's Alt 2 case rests on "BigDecimal should be canonical at construction -like Ratio." **That analogy is the flaw.** Ratio's reduced form *is* its value -— there is no display variant. BigDecimal's scale is **JVM-observable and -authoritative**: java.math.BigDecimal stores scale non-canonically *on purpose*, -because print, arithmetic result-scale, `.scale`, and `.precision` all depend on -the unstripped form. Alt 2 makes the stored scale non-authoritative and forces -every one of those F-005-observable behaviours to be re-derived on `print_scale` -— with the silent-divergence risk the DA itself enumerated. - -Option A keeps the authoritative `(unscaled, scale)` exactly as java.math.BigDecimal -holds it (so the verified arithmetic/print is untouched — F-011 behavioural -equivalence) and adds the stripped projection as the **cached Clojure-key view**. -The "shadow field" the DA names a smell is the faithful modelling of the genuine -two-layer reality (JVM scale-authoritative + Clojure strip-for-key); caching a -hot-path-too-expensive derived key is a standard pattern, not a smell. The -choice is on finished-form-fidelity grounds, NOT cycle/LOC (Option A is in fact -*more* localised, but that is incidental — the deciding factor is that Alt 2 -fights the type's JVM-observable nature). - -## Revision history - -- 2026-06-03 created (Accepted): clj-parity C2 / D-205. Option A (cached - normalized projection) chosen over the DA-recommended Alt 2 (canonical storage - + print_scale) on finished-form-fidelity grounds — BigDecimal scale is - JVM-authoritative, unlike Ratio. diff --git a/.dev/decisions/0078_mapentry_distinct_type.md b/.dev/decisions/0078_mapentry_distinct_type.md deleted file mode 100644 index 012ad1d9a..000000000 --- a/.dev/decisions/0078_mapentry_distinct_type.md +++ /dev/null @@ -1,156 +0,0 @@ -# ADR-0078 — Distinct MapEntry value type (activate the reserved `.map_entry` slot) - -**Status**: Proposed → Accepted (2026-06-03, clj-parity campaign C4 / D-209) - -## Context - -ADR-0076 §9.2.P clj-parity campaign unit **C4 (D-209)**: cljw represents a map -entry as a plain 2-element PersistentVector, so `(map-entry? (first {:a 1}))` → -`name_error` (no such fn) and there is no way to distinguish `(first {:a 1})` -(a MapEntry in clj) from a literal `[1 2]`. - -clj ground truth (oracle-verified): `clojure.lang.MapEntry extends AMapEntry -extends APersistentVector implements IMapEntry`. A MapEntry **IS-A vector** in -every observable way — `(vector? (first {:a 1}))`→true, `(= (first {:a 1}) [:a -1])`→true (both directions), `nth`/`count`/`seq`/`pr-str`/destructure all behave -as a 2-vector — EXCEPT two discriminators: `(map-entry? …)`→true and `(class …)` -→MapEntry. Critically `(conj (first {:a 1}) 99)` → `[:a 1 99]` a PLAIN -PersistentVector (`map-entry?`→false): conj/assoc route through `asVector()` and -DROP the map-entry nature. - -The F-004 Group-A `.map_entry` HeapTag slot (value 15) is DECLARED + NaN-box -encode/decode already unit-tested (value.zig:460), `coll?` already forward-wired -(core.zig:306), but otherwise unwired. - -## Decision - -Activate the reserved `.map_entry` slot as a distinct heap type — the finished -form (matches JVM's distinct class; F-004 reserved the slot for exactly this; -activating a reservation is *consuming a memo*, NOT amending F-004). - -``` -src/runtime/collection/map_entry.zig: MapEntry = extern struct { header, key: Value, val: Value } -``` - -- Substrate mirrors `range.zig` (ADR-0063 / D-178) but owns 2 Value children, so - it registers a `traceMapEntry` GC hook (mark key + val) — unlike range. -- Map-seq producers (`map.zig::seqArrayMap` + `hamtSeqInto`) and `find` - (core.clj) emit a `.map_entry` instead of a 2-vector. -- A `.map_entry` arm is added to every op a map-entry flows through, delegating - to the 2 fields so it behaves EXACTLY as a 2-vector — EXCEPT: - - `map-entry?` (new core fn) → `tag()==.map_entry` (the discriminator). - - `conj`/`assoc` on a `.map_entry` → a PLAIN `.vector` `[k v …]` (JVM - `asVector()` — the nature drops, `map-entry?`→false on the result). - - `class` → "MapEntry" simple name (AD-003, no-JVM). -- `key`/`val` (core.clj, `(nth e 0/1)`) work unchanged once `nth` handles - `.map_entry`. A single arm-set covers both backends (TreeWalk + VM share the - primitive table — no per-backend duplication). - -## Consequences - -- `(map-entry? (first {:a 1}))`→true, `(map-entry? [1 2])`→false, while - `(vector? (first {:a 1}))`→true and `(= (first {:a 1}) [:a 1])`→true (both - directions) — clj parity. `(conj (first {:a 1}) 99)`→`[:a 1 99]` plain vector. -- Memory: a MapEntry (header + 2 Values) is SMALLER than a 2-element vector (no - tail-node array), so this is perf neutral-to-win. -- Blast radius (DA correction — see Alternatives): the `.vector` arms span ~12 - files; each is a place the `else =>` value-dispatch idiom silently swallows a - `.map_entry` until wired. A **corpus pin per behaviour** (`map_entry.txt`) - converts every potentially-missed arm into a gate failure (anti-D-177). -- Closes D-209. `.getName`/`.getSimpleName` on the class object remain (D-207 - residual). Next campaign unit: C6 = D-200. - -## Affected files - -- new: `src/runtime/collection/map_entry.zig` (substrate + traceGc) + - `test/diff/clj_corpus/map_entry.txt` (corpus pins). -- edit (`.map_entry` arms): `runtime/equal.zig` (isSequential / isCountable / - seqEqual cursor / valueHash cursor), `runtime/compare.zig`, `runtime/print.zig`, - `lang/primitive/core.zig` (vector?/sequential?/associative? + new `map-entry?`), - `lang/primitive/sequence.zig` (count/first/rest/nth/seq), `lang/primitive/ - collection.zig` (nth/conj→vector/contains?), `runtime/collection/map.zig` - (producers), `lang/clj/clojure/core.clj` (`find` producer + `map-entry?`), - `runtime/runtime.zig` (registerGcHooks). Plus any of walk.zig / json.zig / - lookup.zig / transient the sweep surfaces. - -## Alternatives considered - -(Devil's-advocate subagent, fresh context, per CLAUDE.md depth-≥2 mandate. -Verbatim-reflected; the DA confirmed Option A = Alt 2 with two corrections. -Within the F-004 envelope — no new slot; `map-entry?` stays faithful.) - -### Alt 1 — Smallest-diff: an `is_map_entry` flag bit on the `.vector` HeapHeader - -Set a spare `HeapHeader.Flags` bit on the 2-vector map-seq produces; `map-entry?` -reads it; every other op ignores it (a flagged vector behaves as a plain vector). - -**Better:** near-zero blast radius — no substrate, no GC hook (the existing -`traceVector` already walks both slots), no per-op arms; `map-entry?` is one -header-bit read. - -**Breaks/risks (fatal, F-011):** conj/assoc/pop/subvec/with-meta must DROP the -nature (clj). The flag rides the source vector's header, but those ops clone the -header, so the flag must be *explicitly cleared* on every structural/meta path — -auditing MORE sites than Alt 2's arms, in the most error-prone direction (silent -wrong-`true`). And it makes `.vector` a **two-headed type** (vector OR map-entry -by a bit): every future `.vector` reader must know the bit exists — the -Silent-default-shift / two-headed smell, the textbook smallest-diff-bias the -finished-form owner unwinds (F-002). Rejected: smaller diff, larger finished-form -debt. - -### Alt 2 — Finished-form-clean (CHOSEN): distinct `.map_entry` tag, per-arm - -Option A as proposed. The map-entry nature is **isolated in the tag**, so -`.vector` stays single-headed; "drops nature on conj" is automatic-by- -construction (the conj arm just builds a `.vector` — no flag to forget). Mirrors -`.range` / `.string_seq` / `.array_seq` (each a distinct heap tag with scattered -per-op arms — the codebase's own established precedent; the DA grep confirmed -there is NO `asVectorView` funnel today, so `.map_entry` follows the same -per-arm pattern as every other non-`.vector` sequential heap type). - -**DA corrections folded in:** (1) the blast radius is **~12 files** (`equal`, -`compare`, `print`, `lookup`, `vector`, `macro_transforms`, `metadata`, -`transient`, `sequence`, `json`, `collection`, `walk`), not "14 arms in 4 files" -— `compare` / `clojure.walk` / `json` are first-class arms, the easiest to miss; -enumerate the full set, and let a broad clj-diff sweep surface any miss. (2) the -`else =>` idiom makes a missed arm a *silent* bug → a **corpus pin per behaviour -is mandatory in the same cycle** (the guard, not exhaustive-switch). **Risk:** a -missed arm degrades a MapEntry silently until the pin catches it — mitigated by -the sweep + corpus. - -### Alt 3 — Wildcard: `.map_entry` laid out as a 2-vector + introduce the `asVectorView` funnel (option d) - -Give `.map_entry` the same in-memory layout as a 2-element vector (so a -`*const Vector` reinterpret is valid) AND introduce the `asVectorView(Value) -?*const Vector` choke point that does NOT exist today, routing the uniform ops -(count/nth/seqEqual/valueHash/print) through one `.map_entry` arm; only the 3 -divergent behaviours (map-entry?/conj/class) keep bespoke arms. - -**Better:** the only shape that realizes option (d)'s arm-reduction; leaves a -reusable `asVectorView` funnel a future `.tuple`/small-vector opt could ride. - -**Breaks/risks:** layout-punning couples `.map_entry`'s ABI to the internal -persistent-vector tail layout (a future vector representation change silently -corrupts MapEntry — needs a comptime offset assertion + still couples two types -F-009 keeps independent). And it is **scope creep beyond D-209**: introducing -`asVectorView` + retrofitting `.range`/`.string_seq`/`.array_seq` through it is a -separate refactor (its own ADR); bundling it risks taking map-entry parity down -with it (Surgical-changes). The net arm saving is modest (N full arms → N -view-producing arms + 1 funnel). Deferred to its own ADR — do not gate D-209 on -it. - -### Decision - -**Alt 2 chosen** (per the DA, = the survey's Option A), with the blast-radius -correction (~12 files, enumerate fully) + the mandatory corpus-pin-per-behaviour -guard. Alt 1 rejected (two-headed `.vector` smell + leak-prone flag clearing). -Alt 3's `asVectorView` funnel is a genuine improvement but is new construction + -ABI coupling + cross-type refactor — deferred to its own ADR, not bundled into -the parity fix. No F-NNN amendment needed (A15 is the reserved map_entry slot). - -## Revision history - -- 2026-06-03 created (Accepted): clj-parity C4 / D-209. Distinct `.map_entry` - type (Option A) over the flag-bit (Alt 1) and the layout-pun+funnel (Alt 3), - on finished-form-cleanliness grounds; DA blast-radius correction (~12 files) + - corpus-pin mandate folded in; the `asVectorView` funnel deferred to its own ADR. diff --git a/.dev/decisions/0079_inst_date_typed_instance.md b/.dev/decisions/0079_inst_date_typed_instance.md deleted file mode 100644 index 85b034c0e..000000000 --- a/.dev/decisions/0079_inst_date_typed_instance.md +++ /dev/null @@ -1,145 +0,0 @@ -# ADR-0079 — `#inst` / java.util.Date as a no-slot typed_instance value - -**Status**: Proposed → Accepted (2026-06-03, clj-parity campaign C6 / D-200) - -## Context - -ADR-0076 §9.2.P clj-parity campaign unit **C6 (D-200)**: `#inst "2024-01-01"` -→ cljw `reader_tag_unknown` (the `inst` data-reader is not registered); clj → -a `java.util.Date` printing `#inst "2024-01-01T00:00:00.000-00:00"`, with -`(inst? …)`→true, `(class …)`→java.util.Date, `(= …)` by epoch-ms, -`(inst-ms …)`→ms-since-epoch. - -The tagged-literal reader infra is COMPLETE (D-200's reader cycles + ADR-0074 -`#uuid` + ADR-0075 generic `tagged_literal`): `#tag form` → tokenizer `.tagged` -→ reader → `*data-readers*` dispatch. So C6 is "add an `inst` data-reader + a -Date value type", not "build reader infra". - -**Representation is user-settled (F-004 LAW): β = a no-slot value.** All 64 -NaN-box slots are named (the `heap_tag.zig` "reserved" comment is stale — -D11–D15 are hamt/tail/map-node/tval). A dedicated `.date` slot would need a -user F-004 reshuffle; the user chose **β = a `.typed_instance` Date**. The -slot-vs-typed_instance axis is therefore NOT re-litigated here. - -## Decision - -Implement Date as a `.typed_instance` carrying ONE epoch-ms `Value` field + -the canonical Date `TypeDescriptor`. The Devil's-advocate fork (mandatory at -depth ≥2) corrected the survey's "1-line" framing and surfaced four latent gaps -the chosen shape (DA Alt 2 hybrid) closes: - -1. **Storage**: `allocInstance(rt, date_descriptor, &.{Value.initInteger(epoch_ms)})` - — a single epoch-ms field. i48 inline integer holds ms to year ~6429 (max - ±1.4e14), so no BigInt; keeps the typed_instance layout honest (a real - field array, not a pointer that lies about its shape). -2. **One canonical descriptor**: reuse the EXISTING `runtime/java/util/Date.zig` - surface descriptor (do NOT mint a second ad-hoc one — two descriptors would - break the print/`instance?`/`class` pointer identity). Its pointer is cached - on `Runtime` at `installAll`; the reader + `inst-ms` build/read instances - against it. `fqcn` set so `(class …)`→"Date" (AD-003 simple name); a - `class_name` FQCN_MAP row maps "java.util.Date"→"Date" so - `(instance? java.util.Date d)` resolves through the existing `matchUserType`. -3. **Print** via a new `TypeDescriptor.print_tag: ?[]const u8 = null`: - `printTypedInstance` (which has NO `rt`) emits `# ""` when - non-null, formatting the epoch-ms field. Date sets `print_tag = "inst"`. - Chosen over a `print.zig` descriptor-pointer compare because - `printTypedInstance` has no `rt` to reach a cached pointer — the descriptor - carrying its own print tag is the rt-free, data-driven shape, and it - generalises to any future reader-tag native value. -4. **`=` by epoch-ms is NOT free** (the DA's load-bearing finding): - `typedInstanceEqual` returns `false` for any `kind != .defrecord` (native = - identity), so two distinct Date allocations would be `≠`. Add an epoch-ms - equality arm gated on the Date descriptor / `print_tag != null`. Mandatory; - pinned by a corpus case. -5. **Parse/format**: pure-Zig closed-form civil↔epoch-ms in `runtime/time/ - instant.zig` (F-009 neutral home) — parse the `#inst` grammar (year-only … - full ISO + tz, offset subtracted to UTC), format epoch-ms → the canonical - `#inst "YYYY-MM-DDTHH:MM:SS.mmm-00:00"`. The civil↔days conversion is new - (instant.zig only had now→ms); use Hinnant's public-domain algorithms. Do - NOT reuse v1's regex+callCore plumbing. -6. **`inst?`/`inst-ms`** core fns: Date-only (cljw has no `Instant` value type). - Do NOT pre-build a forward-compatible Inst protocol (Reservation-as-bias); - widen when an `Instant` value actually lands. - -## Consequences - -- `#inst "2024-01-01"` round-trips as `#inst "2024-01-01T00:00:00.000-00:00"`; - `(inst? …)`→true, `(inst-ms #inst "1970-01-01T00:00:00.000-00:00")`→0, - `(= #inst "2024-01-01" #inst "2024-01-01T00:00:00Z")`→true (epoch-ms), - `(class …)`→"Date". -- A reusable `TypeDescriptor.print_tag` mechanism (one consumer today; the next - reader-tag native value reuses it). Dispatch is backend-shared (typed_instance). -- Closes D-200's last piece (reader infra + `#uuid` already done). Next campaign - unit: C5 = D-198 (after D-048). - -## Affected files - -- new: civil parse/format in `runtime/time/instant.zig`; `test/diff/clj_corpus/ - inst_date.txt`; e2e `test/e2e/phase14_inst_literal.sh`. -- edit: `runtime/type_descriptor.zig` (`print_tag` field), `runtime/java/util/ - Date.zig` (fqcn + print_tag + epoch field shape), `runtime/runtime.zig` - (cache the Date descriptor pointer at installAll), `lang/bootstrap.zig` - (register the `inst` data-reader), `lang/primitive/*` (`instReader` + - `inst?`/`inst-ms` builtins), `runtime/print.zig` (print_tag branch in - printTypedInstance), `runtime/equal.zig` (Date epoch-ms equality arm), - `runtime/class_name.zig` (FQCN_MAP "java.util.Date"→"Date"). - -## Alternatives considered - -(Devil's-advocate subagent, fresh context, depth-≥2 mandate. Within the β + -F-NNN envelope — slot-vs-typed_instance is user-settled, NOT re-litigated.) - -### Alt 1 — Smallest-diff: single epoch-ms field + ad-hoc cached native descriptor - -The survey's literal shape: `allocInstance` with one epoch-ms field; print via a -`print.zig` descriptor-pointer compare; a fresh `rt.date_descriptor` cache. - -**Better:** minimal new types. - -**Breaks/risks:** the **two-descriptor hazard** (an ad-hoc cache + the existing -`installAll` `cljw.java.util.Date` surface descriptor → print/`instance?`/`class` -key off whichever pointer the value carries); the print pointer-compare needs -`rt` in `printTypedInstance`, which has none (forces threading rt through the -print path); a `gc.infra` side-allocation for the 1-element field array; and it -STILL needs the equality arm + FQCN wiring it glosses as "confirm it works". -Rejected: it hides the load-bearing parts (equal arm, descriptor reconciliation). - -### Alt 2 — Finished-form-clean (CHOSEN, hybrid): canonical descriptor + `print_tag` + epoch-ms equality arm - -The decision above. Reuses the existing Date surface descriptor (one canonical), -adds the rt-free `print_tag` mechanism (no rt threading; reusable), the mandatory -epoch-ms equality arm, and the FQCN_MAP row. Storage stays a single epoch-ms -field-array (honest typed_instance layout) rather than a bespoke inline-i64 -`DateValue` struct — the inline-i64 micro-opt's only payoff is one avoided small -allocation, not worth a `.typed_instance` that lies about its layout across -print/equal/dispatch. - -**Better:** one canonical descriptor (no hazard); rt-free print; `=` correct; -`instance?`/`class` ride existing machinery; no layout footgun. - -**Risks:** adds a `TypeDescriptor` field (`print_tag`, `= null` default → zero -call-site churn); the civil-date parse/format is genuinely new code (the -correctness-critical part). Mitigated by corpus pins on round-trip print + -epoch-`=`. - -### Alt 3 — Wildcard: `#inst` as a `tagged_literal`, no Date value type - -Read `#inst "…"` into a `.tagged_literal{tag=inst, form}`; implement -`inst?`/`inst-ms`/print over it. - -**Better:** near-zero new machinery (tagged_literal already prints `#tag form`). - -**Breaks/risks — disqualifying (F-011):** clj's `#inst` IS a java.util.Date, not -a tagged literal: `(class …)` wrong; round-trip print would echo the raw input -string, NOT the normalised `…T00:00:00.000-00:00`; `(= #inst "2024-01-01" -#inst "2024-01-01T00:00:00Z")` would be false (structural) instead of true -(epoch-ms); `inst-ms` would re-parse each call. Wrong value identity → would -need an unwanted accepted-divergence. Rejected. - -## Revision history - -- 2026-06-03 created (Accepted): clj-parity C6 / D-200. No-slot typed_instance - Date (user β, F-004 unchanged). DA Alt 2 hybrid: canonical existing descriptor - + `TypeDescriptor.print_tag` (rt-free print) + a mandatory epoch-ms equality - arm (the DA's key finding — native typed_instances default to identity `=`) + - pure-Zig civil parse/format in `runtime/time/`. diff --git a/.dev/decisions/0080_heap_boxed_long.md b/.dev/decisions/0080_heap_boxed_long.md deleted file mode 100644 index 542cfa980..000000000 --- a/.dev/decisions/0080_heap_boxed_long.md +++ /dev/null @@ -1,174 +0,0 @@ -# ADR-0080 — Heap-boxed Long: an `IntOrigin` flag on the heap-integer struct - -**Status**: Proposed → Accepted (2026-06-03, clj-parity campaign C7 / D-165) - -## Context - -ADR-0076 §9.2.P clj-parity campaign unit **C7 (D-165)** (the LAST unit): -cljw's NaN-box inline integer is i48 (±2^47 ≈ ±1.4e14). A Long value beyond -±2^47 but within i64 currently promotes to a heap **BigInt** (D-014a), so it -prints with an `N` suffix and `(class …)` → BigInt. clj keeps it a primitive -**Long** to the full i64 range. - -Oracle-verified: `(parse-long "999999999999999")` → cljw `999999999999999N` -(BigInt) / clj `999999999999999` (Long); `(* 1000000000 1000000)` → cljw -`1000000000000000N` / clj `1000000000000000`; `9999999999999999` (16-digit -no-`N` literal, > i48, ≤ i64) → cljw "BigInt" / clj java.lang.Long. The VALUE -is exact; only the print form (`N`) + `(class)` diverge. - -**Representation is user-settled (F-005 LAW, ADR-0076 am1 — NOT re-litigated): -B2 = a flag on the heap-integer (BigInt) struct.** NO new NaN-box slot (F-004 -UNCHANGED — an 8-byte Value can't hold tag + i64; i64-inline is physically -impossible, cw v0 also i48). A heap-boxed Long mirrors the JVM (boxes Long on -the heap, stays class Long). - -The flag is **intent-based, not value-range-based** (oracle): -`(parse-long "999999999999999")` → Long despite exceeding i48; `(bigint 5)` / -`5N` → BigInt despite fitting a Long; `99999999999999999999` (no-`N`, past i64) -→ BigInt (`…N`). So the producing call sets the flag; it is never inferred from -the stored magnitude. - -## Decision - -A non-defaulted `IntOrigin` enum on the heap-integer struct, set by every -producing call (the Devil's-advocate fork's Alt 2, chosen over the survey's -defaulted bool — see Alternatives): - -```zig -pub const IntOrigin = enum { long, bigint }; -pub const BigInt = extern struct { - header: HeapHeader, - origin: IntOrigin, // 1 byte — replaces ONE _pad byte; m's offset preserved - _pad: [5]u8 = .{ 0, 0, 0, 0, 0 }, - m: *Managed, -}; -``` - -- **No default** on `origin` — the alloc constructors take it as a REQUIRED - arg, so a forgotten classification is a COMPILE ERROR, not a silent - mis-print on the common (Long) path. This is the core of the DA's correction: - the work is "classify every producing site", and a required enum makes - omission un-compilable + each site self-documenting (`.long` / `.bigint`). -- `wrapI64` / `wrapManaged` (the "exact i64 → Value" collapse path) hard-wire - `.long` — they are the Long-category overflow/collapse entry points. -- **Arithmetic needs NO flag propagation** (the DA's key simplification): the - dispatch ARM already encodes category. In `promote.zig`, the both-int arm - (Long ⊕ Long, e.g. `addPromoting` line 243's i64-overflow `allocAddManaged`) - passes `.long`; the both-Managed / contagion arm (any BigInt operand, e.g. - line 249, `quotPromoting`'s `isExactBig` branch) passes `.bigint`. The same - `allocAddManaged` fn is called from two arms with OPPOSITE origins — the - enum makes that visible. No operand `origin` is ever READ during arithmetic; - only written per-arm. -- **Literals** (analyzer `parseBigIntLiteral` / reader): no-`N` literal in - (2^47, i64] → `.long`; no-`N` literal > i64 → `.bigint`; `N`-literal → - `.bigint`. (Oracle: `9999999999999999`→Long, `99999999999999999999`→BigInt - `…N`, `5N`→BigInt.) -- **`bigint`/`biginteger` fns + bigint coerce** → `.bigint` (BigInt regardless - of magnitude); `long`/`int` coerce → `.long`; `parse-long`/Long·Math statics - / JSON int → `.long`. -- **print** (`printBigInt`) gates the `N` suffix on `origin == .bigint`. -- **`(class)` + `instance?`** (`class_name.zig`): the `.big_int` tag now yields - TWO class names by `origin` — `.long` → "Long", `.bigint` → "BigInt". The DA - flagged that this makes `nativeTagFor` non-injective: a heap-Long answers - `(instance? Long x)`→true / `(class)`→"Long" while tagged `.big_int`. Both - the class-display and the `isInstance`/`matchUserType` "Long" paths consult - `origin`. -- **`=` / hash UNCHANGED** — already value-based (D-205), so `(= 5N 5)`→true and - `(= (hash 5N) (hash 5))`→true hold for free; the flag is print/class only. - (Pinned by a corpus regression line regardless.) - -## Consequences - -- `(parse-long "999999999999999")`→Long (no `N`, class Long); `(* 1000000000 - 1000000)`→Long; `9999999999999999`→Long; `(bigint 5)`→5N; `5N`→BigInt; - `99999999999999999999`→BigInt. The i47-inline window (±1.4e14) covers - ms-timestamps (~4400 yr) / counters / sizes; only sub-ms timestamps / - Snowflake IDs / 64-bit hashes take the heap-Long path (still ≤ JVM, which - boxes everything > 127). -- Closes D-165 — the LAST clj-parity campaign unit. After C7, C1..C7 are - complete; D-210 anchor becomes a standing `quality-loop floor` (drain new - sweep DIFFs). Overflow-past-i64 promote-vs-throw stays the separate - user-ratified **AD-008**. - -## Affected files - -- `runtime/numeric/big_int.zig` (`IntOrigin` + struct field + `origin`-taking - `allocFromI64`/`allocFromManaged`), `runtime/numeric/promote.zig` (`wrapI64`/ - `wrapManaged` → `.long`; each arith arm passes its origin), - `runtime/numeric/ratio.zig` + `big_decimal.zig` (internal bigints → `.bigint`; - `@offsetOf` asserts unchanged), `eval/analyzer/analyzer.zig` + - `eval/reader.zig` (literal origin by range/`N`), `lang/primitive/math.zig` - (bigint/long/int coerce, parse-long), `lang/primitive/json.zig` (int read → - `.long`), `runtime/print.zig` (`N` gate), `runtime/class_name.zig` (class + - isInstance by `origin`). Corpus `heap_long.txt` + e2e. - -## Alternatives considered - -(Devil's-advocate subagent, fresh context, depth-≥2 mandate. Within B2 + the -F-NNN envelope — slot-vs-flag is user-settled, NOT re-litigated.) - -### Alt 1 — Smallest-diff: defaulted `long_origin: bool = false` + thin `…Long` wrapper variants - -The survey's shape: `long_origin: bool = false` (BigInt-default, preserves -unmigrated paths' `N`); `allocFromManagedLong`/`allocFromI64Long` wrappers set -true; contagion arms call the base (false) variants. - -**Better:** minimal struct churn; existing caller signatures unchanged. - -**Breaks/risks:** the `= false` default is a loaded gun on the COMMON path — -most heap integers in real code are overflowed Longs, not bigints, so every -miss (a forgotten arm, a future Phase-15 arith path) silently prints `N` on a -value that should be a plain Long = the C7 bug re-introduced by omission. No -compiler help for the 8 promote.zig arms; the `allocAddManaged`-called-from- -two-arms-with-opposite-flags reality is invisible. The **Smallest-diff bias -smell**: cheap struct edit, classification cost pushed onto unaided vigilance. - -### Alt 2 — Finished-form-clean (CHOSEN): non-defaulted `IntOrigin` enum on a unified constructor - -The decision above. **Omission is a compile error** (required enum arg); the -two-arms-one-fn trap becomes two visibly-different `.long` / `.bigint` call -sites; no default-direction debate (there is no default); self-documenting. - -**Better:** classification is STRUCTURALLY enforced, not vigilance-enforced -(F-002 finished form); future arith arms inherit the same compiler pressure. - -**Risks:** larger diff — every `allocFromManaged`/`allocFromI64` caller -(ratio/big_decimal/math/json/analyzer + the ~12 sites) adds an `origin` arg. -But that IS the C7 task (classify each), and cycle/diff size is not an F-NNN -constraint (anti-Cycle-budget-defer). Ratio numerator/denominator internal -bigints pass `.bigint` (a harmless fixed choice — they print via ratio's own -path, never standalone). - -### Alt 3 — Wildcard: a separate `HeapTag.long_box` tag (no new NaN-box slot) - -A distinct `HeapTag` value (not a new NaN-box slot) for heap-Long; genuine -BigInt keeps `.big_int`; both point at identical `BigInt` structs. `class_name` -becomes injective again; print gates on tag; `=`/hash tag-agnostic. - -**Better:** cleanest type-system answer — every `switch (v.tag())` is forced to -consider both; `nativeTagFor` stays injective. - -**Breaks/risks — REJECTED:** it relitigates the user-settled B2 ("a FLAG on the -struct", not a new tag) and carries a >20-site `.big_int or .long_box` sweep -across promote.zig (coerceToManaged / toF64 / isExactBig / every arith arm) — a -missed one silently treats a heap-Long as a non-integer (far larger regression -surface than Alt 2's write-side-only risk). Cleaner in theory, highest -regression in practice; recorded, not adopted (would need the user to reopen -B2). - -### Decision - -**Alt 2 chosen** (per the DA, over the survey's defaulted-bool Alt 1) — the -non-defaulted enum makes the per-site classification compile-enforced, killing -the common-path silent-mis-print risk. Two DA-surfaced sites the survey missed -are fixed in the same cycle: `isInstance`/`nativeTagFor` non-injectivity, and -the oracle-pinned past-i64-literal print. Alt 3 (new HeapTag) rejected -(relitigates B2 + >20-site sweep). No F-NNN amendment (B2 + F-004 + F-005 hold). - -## Revision history - -- 2026-06-03 created (Accepted): clj-parity C7 / D-165 (LAST unit). `IntOrigin` - enum (DA Alt 2) over the defaulted bool (survey); arithmetic classifies by - dispatch-arm category (no operand-flag propagation); `=`/hash untouched - (D-205); class/isInstance/print gate on `origin`. AD-008 (overflow-past-i64) - stays separate. diff --git a/.dev/decisions/0081_atom_watches.md b/.dev/decisions/0081_atom_watches.md deleted file mode 100644 index 52bfc72ca..000000000 --- a/.dev/decisions/0081_atom_watches.md +++ /dev/null @@ -1,99 +0,0 @@ -# ADR-0081 — Atom watches via an appended `watches` field (Phase 15 entry) - -**Status**: Proposed → Accepted (2026-06-03, Phase 15 concurrency entry / D-157) - -## Context - -`add-watch` / `remove-watch` (Clojure atom watchers) were unimplemented -(debt row D-157) — `(add-watch a :k f)` → cljw `name_error` / clj registers a -watcher fired on every `swap!`/`reset!`/`compare-and-set!` change with -`[key ref old new]`. This is the first task of Phase 15 (concurrency); the -runtime is single-threaded, so **synchronous** notification (fire in the -swapping call, after the in-place mutation) is the correct finished form — -clj also fires watches in the swapping thread. - -`src/runtime/atom.zig`'s `Atom` is an `extern struct { header, _pad[6], current }`. -Its header comment notes watches/validator are Phase-15 deferrals and that — -per F-003 / the Reservation-as-bias smell — **no watches field was reserved**. -So adding watches is a genuine structural decision, handled with the mandatory -Devil's-advocate fork (depth ≥ 2). - -clj semantics (verified): watches fire on EVERY change including `old == new` -(no equality short-circuit); `add-watch` with an existing key REPLACES; order -is unspecified (map-iteration order); `compare-and-set!` fires only on success. - -## Decision - -**Alt 2 (DA-recommended): append a `watches: Value` field to the `Atom` extern -struct, holding `nil` (zero-watch atoms — no allocation) or a persistent map -`{key → fn}`.** - -- `add-watch` = `assoc` into the watches map (or a fresh empty map); `remove-watch` - = `dissoc` — reusing the existing persistent-map machinery gives clj-faithful - key-replace + remove-absent-is-no-op for free. -- `traceGc` gains a one-line `mark(watches)` — the map traces its keys + fns - transitively. GC-correct by the same "reachable through the atom" invariant - the existing `current`-trace embodies; when the atom is swept, its watches map - becomes unreachable and is collected (no leak / dangling-key class that the - side-table alternative suffers). -- Notify (synchronous, in `swap!`/`reset!`/`compare-and-set!` after the in-place - set, success-only for CAS): snapshot the watches map, iterate via - `map.keys` + `map.get` (works for any map kind), call each fn `[key ref old new]` - through `rt.vtable.callFn`. Fires on every change incl. `old == new`. -- **`validator` is NOT added now** (F-003): set-validator! lands its own field by - the same one-line repetition when that task arrives. The extern-struct shape - (`header` at offset 0, `current` traceable) is preserved; appending after - `current` keeps the comptime asserts valid. - -F-004 untouched (atom is one `.atom` NaN-box slot; this changes the pointed-at -heap struct, not the 64-slot layout). F-006 honored (watches traced). F-011 -(behavioural equivalence vs clj). - -## Alternatives considered - -(Devil's-advocate fork, fresh context — verbatim summary.) - -**No F-NNN block.** Alt 2 lives within F-004/F-006/F-002/F-011/F-003. - -- **Alt 1 — SMALLEST-DIFF: Runtime side-table `AutoHashMap(*Atom, WatchList)`.** - No struct change. BREAKS: GC correctness — the side-table holds Values the GC - must trace but it is not a heap value with a trace hook, and (fatal) when an - atom is swept its entry is never cleaned → the table grows monotonically, - keeps watch fns alive forever, and the `*Atom` key dangles (re-alloc at the - same address aliases a dead atom's watches onto a new atom). Fixing it needs - GC finalization cw lacks. Validator would be a second side-table with the same - disease; threading makes the global table a contention point. Reject (smallest- - diff bias the finished-form owner would unwind). - -- **Alt 2 — FINISHED-FORM: append `watches: Value` (a persistent map).** GC-correct - by construction (reachable through the atom, collected with it); reuses map - `assoc`/`dissoc` (clj-faithful key-replace/remove, almost no new data-structure - code); extends to `validator` by literal one-field repetition; heap-local field - is threading-friendlier than a global table (future per-atom lock attaches to - the cell). Risks: struct size changes (re-confirm the `@offsetOf(header)==0` / - `@alignOf>=8` asserts — they hold; only `atom.zig` + `primitive/atom.zig` touch - the struct, both via accessors); notify site must thread `rt`+`env` to `callFn` - (available; `resetFn`/`compareAndSetFn` drop their `_ = rt; _ = env;`). Recommended. - -- **Alt 3 — WILDCARD: a dedicated `AtomWatches` heap block** (watches + validator - + future agent/meta in one lazily-created side-block pointed to by one atom - field). BEST for "lots of reference metadata" + cache-local, GC-correct. But it - reinvents map key-handling as bespoke array code (more surface, diverges from - clj's "watches IS a map" model) and is F-003 over-reservation — it builds a - metadata container for imagined agents/meta that are not this task. Reject on - F-003 grounds (not diff size). - -**DA recommendation (non-binding): Alt 2.** Implementation notes adopted: keep -`watches = nil` for zero-watch atoms (no empty-map alloc); remove the -`_ = rt; _ = env;` discards at the notify sites. The main loop accepts Alt 2. - -## Consequences - -- `add-watch`/`remove-watch` land for atoms (the IRef family present today); - `swap!`/`reset!`/`compare-and-set!` fire watches synchronously. -- A re-entrant `swap!` inside a watch fn reads the already-committed value - (in-place `setCurrent` happened before notify) — matches clj. -- A future `set-validator!` + threading attach to the same cell with no re-lay. -- Affected files: `src/runtime/atom.zig` (field + accessors + trace), - `src/lang/primitive/atom.zig` (notify + add/remove-watch + wire swap/reset/CAS), - corpus `test/diff/clj_corpus/atom_watch.txt`, e2e `phase14_atom_watch.sh`. diff --git a/.dev/decisions/0082_syntax_quote.md b/.dev/decisions/0082_syntax_quote.md deleted file mode 100644 index 01bab480d..000000000 --- a/.dev/decisions/0082_syntax_quote.md +++ /dev/null @@ -1,105 +0,0 @@ -# ADR-0082 — Syntax-quote: reader Form nodes + analyzer expansion (hybrid) - -**Status**: Proposed → Accepted (2026-06-03, D-226). STAGE 1 (templates/`~`/`~@`/ -`foo#` + the `valueToForm` lazy-seq-forcing fix) + STAGE 2 (symbol qualification: -var→home-ns, `rt`→`clojure.core`, current-ns fallback, special-forms/interop/ -class bare) both LANDED 2026-06-03 — D-226 DISCHARGED. Residuals: nested backtick -(D-228), `macroexpand` (D-229). - -## Context - -Syntax-quote (`` ` ``/`~`/`~@`/`foo#`) is unimplemented (D-226) — `` `(+ ~x 1) `` -→ "Unable to resolve symbol: '`'" (verified from a file). It is the single -highest-value clj-parity gap: every real-world Clojure library macro uses -backtick, so `(require some-lib)` fails on the lib's macros. It gates the -"run real Clojure libs" goal (D-158). - -cljw's reader is namespace-UNAWARE by deliberate design (read = pure text→Form; -ns-resolution lives in the analyzer — `::name` auto-keywords set an -`auto_resolve` flag the analyzer resolves via `resolveAutoKeyword` against -`env.current_ns`). clj's reader, by contrast, resolves syntax-quote symbols at -READ time (its reader consults `*ns*`). - -The three sub-problems split unevenly: `~`/`~@`/nesting/auto-gensym are pure -structural transforms (need no ns); **symbol qualification** (`` `foo `` → -`current-ns/foo`, `` `+ `` → `clojure.core/+`) is the only ns-dependent part and -the entire source of macro hygiene. - -## Decision - -**Alt 2 (DA-recommended): hybrid.** The reader adds Form nodes -`.syntax_quote`/`.unquote`/`.unquote_splicing` and does ZERO expansion (stays -pure). The **analyzer** owns expansion: on a `.syntax_quote` node it has -`env.current_ns` + the alias/refer table, so it builds the -`(seq (concat (list …) …))` tree — generalizing the `resolveAutoKeyword` -"reader marks / analyzer resolves" precedent instead of inventing a parallel -resolver, and keeping the reader's pure text→Form invariant. - -**Staged (DA-endorsed bisectable path):** -1. **This commit — structural transform, PROVISIONAL (non-qualifying)**: tokenizer - `` ` ``/`~`/`~@`; reader Form nodes; analyzer expander handling `~`/`~@`/ - nesting/`foo#` auto-gensym + list/vector/map/set, with symbols left BARE - (`` `foo `` → `(quote foo)`). This makes single-ns + core-symbol macros work - (the caller refers core / same ns), but **HARD-FAILS on a macro that backtick- - references its own ns's private helper** (`` `helper `` → bare `helper`, - unresolved at the call site) — so D-226 is **NOT discharged**; a `PROVISIONAL:` - marker + feature_deps + debt row track the close-out. -2. **Next cycle — qualification**: the expander resolves each non-special, non- - `~`, non-`foo#`, non-`.`-interop symbol to its ns-qualified form (var's ns if - resolvable, else current ns) — the hygiene that makes real libs work. Also - qualify the emitted `seq`/`concat`/`list`/`apply` as `clojure.core/…` so a - caller shadow can't break the expansion. THEN discharge D-226 against a corpus - of real-library-shaped macros (same-ns private helper + caller-shadowed core). - -F-002 (finished form = Alt 2 with qualification; the provisional step is marked, -not a terminal shape). F-011 (a backtick macro must expand+run like clj). - -## Alternatives considered - -(Devil's-advocate fork, fresh context — verbatim summary.) - -The design axis collapses to: **where does qualification live, and is it done at -all?** The structural transform (`~`/`~@`/gensym/nesting) is shared machinery. -Correction the DA flagged: clj resolves syntax-quote symbols to fully-qualified -`(quote ns/sym)` literals INSIDE the expansion, AND qualifies the emitted -`seq`/`concat`/`list` as `clojure.core/…` (a second hygiene surface). - -- **Alt 1 — SMALLEST-DIFF: reader builds the full expansion, NON-qualifying.** - Entirely in the reader (mirrors `readQuote`), bare symbols, reader-local - gensym map. BREAKS F-011 and worse than "hygiene nicety": a macro that - backtick-references its OWN ns private helper produces a bare symbol with NO - call-site referent → hard failure, pervasive in real libs. Shipping it as a - TERMINAL shape closing D-226 = a false-positive discharge (clj_diff_sweep - Discipline 1). Acceptable ONLY as an explicitly-PROVISIONAL first commit. - -- **Alt 2 — FINISHED-FORM: hybrid, reader `.syntax_quote` node + analyzer - expansion with full ns.** F-011-correct (true qualification → hygiene → real - libs work); resolution lives where `resolveAutoKeyword` already lives (no - second resolver); reader stays pure; nesting-depth handled in ONE auditable - analyzer fn; natural future home for `macroexpand-1`. One narrow divergence: - `read-string` of a backtick form returns the `.syntax_quote` Form (clj returns - the expanded data) — fixable by routing `read`/`read-string` through the same - expander (debt row). Must also expand under `quote` (`` '`foo `` expands in clj). - -- **Alt 3 — WILDCARD: thread current-ns INTO the reader (clj-faithful read-time).** - Maximal fidelity incl. the `read-string` edge. REJECTED on cleanliness (not - cost — F-002 forbids cost-downgrade): it REVERSES the deliberate pure-reader / - ns-aware-analyzer split and duplicates resolution (reader for backtick, - analyzer for `::`). clj does read-time resolution only because its reader and - evaluator are fused with no separate analyzer; cljw has the split already, so - the clean finished form puts all ns-resolution in the analyzer. - -**DA recommendation (non-binding): Alt 2**, landing the structural transform -first (optionally as the marked-provisional non-qualifying intermediate of Alt -1's machinery), then the analyzer qualification pass — discharging D-226 only -when real-library-shaped macros run equivalently to clj. The main loop accepts -Alt 2 with this staging. - -## Consequences - -- This commit: `~`/`~@`/`foo#`/nesting work; single-ns + core-symbol macros run; - D-226 stays open (qualification pending), PROVISIONAL-marked. -- Affected: `src/eval/tokenizer.zig` (3 tokens), `src/eval/form.zig` (3 Form - nodes), `src/eval/reader.zig` (read the nodes), `src/eval/analyzer/` (the - syntax-quote expander), corpus + e2e. Next cycle adds qualification + - closes D-226 + the `read-string`-of-backtick debt. diff --git a/.dev/decisions/0083_namespace_as_value.md b/.dev/decisions/0083_namespace_as_value.md deleted file mode 100644 index 6abb3f886..000000000 --- a/.dev/decisions/0083_namespace_as_value.md +++ /dev/null @@ -1,186 +0,0 @@ -# ADR-0083 — Namespace-as-value, `*ns*`, and the ns-reflection cluster - -- **Status**: Proposed → Accepted (2026-06-03) -- **Debt**: D-230 (this), unblocks D-227 (clojure.test), D-158 (real-lib load) -- **Depends on**: ADR-0027 (day-1 HeapTag reservation activation), ADR-0059 - (no-JVM), AD-002 (opaque-ref print) -- **Supersedes**: none - -## Context - -`clojure.test` (D-227) is the densest real-bug-finder and the enabler for -running real libraries' own test suites on cljw (D-158). Its clean finished -form (per the D-227 design probe / DA fork `a7b21f9`) is a **per-namespace -test registry keyed by the current ns symbol**, so `(run-tests 'foo.test)` — -the call every external runner makes — works. The smallest-diff alternative -(a global registry that ignores the ns argument) is the DA-flagged trap: it -passes a toy `(deftest)(is)` demo but fails the real-runner call that is the -whole point of D-158. - -That per-ns form needs **current-namespace identity at the Clojure level**. -cljw lacks it entirely: the current namespace lives only as an internal -`*Namespace` struct (`env.current_ns`), never wrapped as a `Value`; and -`*ns*`, `ns-name`, `the-ns`, `find-ns`, `all-ns`, `ns-interns`, -`ns-publics`, `ns-map`, `ns-resolve` are all unresolved. This ADR adds the -**ns-reflection cluster**: a first-class Namespace value, `*ns*`, and the -read-only reflection functions. - -**Out of scope (stays deferred):** the `*out*` / print-control half of the -system-var-registry. clojure.test's `report` writes through the existing -print path (Runtime.io → stdout); `*out*` is forced in only by -output-redirection (`(binding [*out* w] (run-tests))`), a nicety the per-ns -registry does not need. The cut is clean (DA-confirmed). `remove-ns` is also -deferred — it is a mutation+lifetime-distinct sub-area (a dangling `.ns` -Value pointing at a freed Env-lifetime `*Namespace` is a use-after-free that -needs a tombstone design), not drip-feed. - -## Decision - -Adopt **Alt 2** (DA-recommended): - -1. **Representation (A).** A Namespace becomes a `Value` via - `Value.encodeHeapPtr(.ns, ns_ptr)` wrapping the existing **Env-lifetime** - `*Namespace` directly. The HeapTag slot `ns = 21` is **already reserved** - (`heap_tag.zig:78` / `value.zig:67`, type-declared, unwired); this - activates the reservation per ADR-0027 — **no new F-004 slot, no user - amendment**. Identity is free: `findOrCreateNs` is idempotent, so same - name → same `*Namespace` → same Value bits → `=` (pointer identity). -2. **GC membrane.** `.ns` decodes to an Env-lifetime `*Namespace` and is - **GC-skipped**, exactly like its sibling `.var_ref` (slot 20). The GC must - NOT trace into `mappings`/`refers` (those are Env-owned - `StringHashMapUnmanaged`, not GC-heap — tracing them would invert - ownership; this is why a HeapHeader'd Namespace struct, option B, is - wrong). -3. **`*ns*` mechanism (a).** Introduce `env.setCurrentNs(ns)` that updates - **both** `current_ns` AND the root of a Runtime-cached `*ns*` dynamic Var - (`rt.ns_var`) to `encodeHeapPtr(.ns, ns)` (via the existing `Var.setRoot`). - All `current_ns =` assignment sites route through it. `Var.deref` is - **unchanged** (no signature ripple). `*ns*` is interned at bootstrap as a - `^:dynamic` Var (the `*data-readers*` precedent) and referred into `user`, - so `binding`/`alter-var-root` on `*ns*` work for free — mirroring clj, - whose `*ns*` is a real dynamic Var that `in-ns`/`load` set via `.set()`. -4. **Reflection surface** in a new F-009 peer `src/lang/primitive/namespace.zig` - (keyword `namespace`): `ns-name` (→symbol), `the-ns` (→ns-value or throw), - `find-ns` (→ns-value or nil), `all-ns` (→seq of ns-values), `create-ns` - (`findOrCreateNs` wrap), `ns-interns` / `ns-publics` / `ns-map` - (→`{sym → Var-value}`, Var already a `.var_ref` Value), `ns-resolve` - (→Var-value or nil). `in-ns` (today an analyzer form node) is also exposed - as a runtime callable routing through `setCurrentNs`. The whole read-only - cluster lands in one push (anti-Micro-coverage-grind). `remove-ns` - deferred. -5. **Print + equality.** `=` is pointer-identity (heap-ptr default). - `(str *ns*)` → `"user"`, `(name (ns-name *ns*))` → `"user"`. - `(pr-str *ns*)` → `#object[Namespace "user"]` (no `0xADDR` — cljw cannot - mirror a JVM identity hash). This print divergence is recorded as - **AD-010** (`derives_from`: ADR-0059 no-JVM + AD-002 opaque-ref). - -## Consequences - -- clojure.test (D-227) is unblocked: its registry keys on `(ns-name *ns*)` - at `deftest`-expansion time. -- Broadly reusable beyond clojure.test: `clojure.repl/dir`, REPL - introspection, tooling. -- The one real surface is the `current_ns =` reroute (6 sites: tree_walk ×2, - vm ×3, bootstrap ×1). A missed site → `*ns*` silently stale; mitigated by - routing every assignment through `setCurrentNs` (make direct field-assign - the exception, not the rule). -- AD-010 added to `.dev/accepted_divergences.yaml` with a pin test. -- `remove-ns` + `*out*`/`with-out-str` remain tracked debt (separate rows). - -## Affected files - -- `src/runtime/value/value.zig` / `heap_tag.zig` — `.ns` print/type-name/eq - activation (encode/decode/tag already generic). -- `src/runtime/print.zig` — `.ns` → `#object[Namespace ""]` (AD-010). -- GC membrane (wherever `.var_ref` is skipped) — add `.ns` sibling skip. -- `src/runtime/env.zig` — `setCurrentNs`; `Namespace`→Value helper. -- `src/runtime/runtime.zig` — `ns_var: ?*Var` cache. -- `src/lang/bootstrap.zig` — intern `*ns*` dynamic Var, refer into `user`, - cache on `rt`; initial `setCurrentNs`. -- the 6 `current_ns =` sites (tree_walk / vm / env init). -- `src/lang/primitive/namespace.zig` (NEW) + registration in `primitive.zig`. -- `compat_tiers.yaml` — keyword `namespace` entry. -- `.dev/accepted_divergences.yaml` — AD-010 + pin. -- corpus `test/diff/clj_corpus/namespace.txt`, e2e `phase15_namespace.sh`. - -## Alternatives considered - -(Verbatim from the mandatory Devil's-advocate fork, fresh context, agent -`aa513cf4030cd67cf`. Recommendation Alt 2; the main loop concurs.) - -> All claims verified. Critical findings: `setRoot` already exists on `Var` -> (line 108) — the `*ns*`-as-Var mutation primitive is already there. -> `findOrCreateNs` / `findNs` exist. The `*data-readers*` precedent confirms -> the dynamic-Var-cached-on-Runtime pattern works with `binding` for free. -> `current_ns =` is at 6 sites. None of the reflection fns exist; `in-ns` is -> only an analyzer form node, not a runtime callable. No F-NNN-violating -> findings — all three alternatives stay inside the reserved slot. -> -> ### Alt 1 — SMALLEST-DIFF ("symbol-handle `*ns*`, analyzer-resolved") -> (a) Representation (C): `*ns*` resolves to the symbol of the current ns -> name; reflection fns take/return symbols. Reserved `.ns` slot stays -> dormant. (b) analyzer special-cases `*ns*` → current-ns-name node. (c) -> `ns-name` identity-on-symbol; fns read `findNs(sym).mappings`. (d) `*ns*` -> prints as symbol `user`; `(= *ns* (the-ns 'user))` may not hold (symbol vs -> ns-value). Better: zero reroute risk, smallest LOC. Breaks: `(ns-name -> *ns*)` looks right but type is wrong (`instance?`/round-trip diverge); -> `(binding [*ns* …])` impossible (no Var). This is the v1 mistake the brief -> calls out — leaving a clean reservation unused to save diff = Smallest-diff -> bias smell in pure form. -> -> ### Alt 2 — FINISHED-FORM-CLEAN ("`.ns` wraps Env `*Namespace`; `*ns*` real -> dynamic Var via centralized `setCurrentNs`") ⭐ -> (a) Representation (A): `encodeHeapPtr(.ns, env_namespace_ptr)` wraps the -> existing Env-lifetime `*Namespace`. Identity free (findOrCreateNs -> idempotent). GC-skip like `.var_ref`; GC does NOT trace mappings/refers -> (Env-owned). (B) is wrong — a GC-traced Namespace struct forces GC to -> own/trace the VarMaps, a lifetime inversion. (b) `env.setCurrentNs(ns)` -> updates both `current_ns` and `rt.ns_var` root; route all 6 sites through -> it; `Var.deref` unchanged. `Var.setRoot` already exists, so setCurrentNs is -> `current_ns = ns; rt.ns_var.setRoot(encodeHeapPtr(.ns, ns));`. -> `binding`/`alter-var-root` work for free — mirrors clj most faithfully. (c) -> full cluster in `namespace.zig` (keyword `namespace`); `in-ns` promoted -> from analyzer-only to runtime callable routing through setCurrentNs -> (implementer note: reroute the 6 existing sites, don't write fresh). (d) -> AD for `#object[Namespace "user"]` (no 0xADDR; derives_from ADR-0059 + -> AD-002); `(str *ns*)`→"user"; `=` pointer-identity (correct, same-name→same -> pointer). Better: every clj idiom works (`(binding [*ns* …])`, `(= *ns* -> (the-ns 'user))`, `(resolve '*ns*)`, alter-var-root); uses reserved slot as -> designed; no GC tangle; no deref ripple. Breaks/risk: the 6-site reroute is -> the only surface; a missed site = `*ns*` silently stale (mitigate: -> debug-assert `rt.ns_var.root` decodes to `current_ns` at eval-loop top). -> Per F-002 the reroute is NOT a reason to downgrade to Alt 1 (that is the -> Cycle-budget-defer smell). -> -> ### Alt 3 — WILDCARD ("`*ns*` self-rooting Var; no central mutator") -> (a) representation (A) again, but invert (b): `rt.ns_var.root` becomes the -> SSOT; `env.currentNs()` = `decodePtr(*Namespace, rt.ns_var.deref())`. No -> separate `current_ns` field. (b) `*ns*` Var is SSOT; `(binding [*ns* x] …)` -> actually changes what `current_ns` resolves to during the binding — clj's -> eval model. Better: eliminates dual-write staleness; `(binding [*ns*])` is -> load-bearing not just tolerated. Breaks/risk: large blast radius — every -> reader of `current_ns` (resolve/intern/refer/analyzer/bootstrap) becomes -> `currentNs()` paying a `Var.deref` + threadlocal walk on EVERY name -> resolution (hot path) → perf regression needing a PERF-cached fast-path -> (reintroducing dual-state); plus bootstrap-binding-leak risk. High risk for -> a feature whose driver (clojure.test) does NOT need live rebinding (it keys -> the registry at registration time). -> -> ### Scope-boundary critique -> Excluding `*out*`/print-control is the right cut — clojure.test's `report` -> writes via the existing print path; `*out*` is forced only by -> output-redirection, independently deferrable. (Caveat: a test using -> `with-out-str` needs `*out*` — separate debt row, don't expand this ADR.) -> Do the WHOLE read-only reflection cluster in one push (small, bounded, -> anti-Micro-coverage-grind): include `ns-publics`/`ns-map`/`ns-resolve` now. -> Defer `remove-ns` (mutation+lifetime: dangling `.ns`→freed Env `*Namespace` -> = use-after-free, needs tombstone) — genuinely different sub-area, not -> drip-feed. -> -> ### One-line recommendation -> Alt 2 — representation (A) + mechanism (a) (centralized setCurrentNs, -> 6-site reroute, Var.deref untouched). Most clj-faithful, activates the -> reserved slot per ADR-0027, no new F-004 slot, no GC tangle, no deref -> ripple; the 6-site reroute is finished-form cost, not a reason to fall back -> to Alt 1. Land the print-divergence AD, full read-only cluster, defer -> remove-ns. diff --git a/.dev/decisions/0084_filesystem_require.md b/.dev/decisions/0084_filesystem_require.md deleted file mode 100644 index 29ba5f7b3..000000000 --- a/.dev/decisions/0084_filesystem_require.md +++ /dev/null @@ -1,161 +0,0 @@ -# ADR-0084 — `require` loads namespace source from the filesystem (D-158) - -- **Status**: Proposed → Accepted (2026-06-03) -- **Debt**: D-158 (this). Capstone of the real-lib-compat stack (D-226 - syntax-quote, D-230 `*ns*`/ns-reflection, D-227 clojure.test all met). -- **Depends on**: ADR-0035 (require/in-ns scaffolding, `require_in_progress` + - `circular_require` reservation), ADR-0036 (dual-backend parity), zone_deps - (Layer 1 `eval/` vs Layer 2 `lang/`). - -## Context - -`require` resolves only bootstrap-EMBEDDED namespaces today. Both backends -already (a) check `env.findNs` for already-loaded, (b) call -`rt.require_resolver` for source bytes, (c) get the bytes — then raise -`feature_not_supported` instead of evaluating them (`tree_walk.zig:~464`, -`vm.zig` `op_require:~557` + `op_require_with_libspec:~584`). `:as`/`:refer`/ -multi-`:require` in `(ns …)` already work (the `ns` macro expands to per-spec -single-libspec `require` calls); only the SOURCE-LOAD is missing. This ADR adds -filesystem loading so a real lib (a directory of `.clj` files) and its -`clojure.test` suite run on cljw. - -## Decision - -Adopt **Alt 2** (DA-recommended). Plain dirs of `.clj`/`.cljc` only — no JVM -classloader, no jar/zip resources (the prominent divergence). - -1. **Classpath = `rt.load_paths: []const []const u8`**, populated in the CLI - from `--classpath`/`-cp` (colon-separated) and the `CLJW_PATH` env, default - `["."]`. Two roots (`src:test`) is the real-lib minimum; a colon-path list is - exactly that. **deps.edn parsing deferred** (own ADR — it just populates the - same `load_paths` later; correct layering, not a smallest-diff dodge). -2. **Resolver returns `?ResolvedSource { source, label }`** (signature change to - the reserved `RequireResolverFn` slot — a reservation is a memo, not a - contract). The `label` is the SourceContext key: a real filesystem path for - disk libs (so errors render `src/foo/bar.clj:12`), the `` sentinel - for embedded. This is what makes error rendering correct for both source - kinds without the helper guessing. -3. **Resolver chaining = a 2-element `chainedResolver`, embedded-FIRST** then - filesystem. Embedded-first is mandatory: `clojure.core`/`clojure.test` must - never be shadowed by a stray on-disk file of the same name. A resolver LIST - is premature (only two sources exist; the Phase-16 pod resolver is a - reservation — build the list when a third consumer actually lands; YAGNI). -4. **Shared load helper in a NEW Layer-1 module `src/eval/loader.zig`** (NOT - bootstrap.zig — that is Layer 2; `eval/backend/*` importing it would violate - zone_deps). `loader.loadNamespace(rt, env, macro_table, arena, ns, source, - label)`: `registerSource(label, source)` → if `require_in_progress.contains(ns)` - → `circular_require` → `put(ns)` → save `env.current_ns` → Reader-loop - {read → analyzeForm → evalForm} (the loaded file's own `(ns …)` switches - current_ns) → `defer setCurrentNs(saved)` → `loaded_libs.put(ns)` → - `require_in_progress.remove(ns)`. Both backends call this one body (parity). -5. **Idempotency = `findNs` + NEW `rt.loaded_libs`**. `findNs(mappings>0)` is - necessary but NOT sufficient: during `a→b→a`, ns `a` exists with partial - mappings before its body finishes, so a re-entrant require could skip and - return a half-built ns. `require_in_progress` (in-flight → `circular_require`) - + `loaded_libs` (completed → idempotent skip) is the robust pair (F-011 - `*loaded-libs*` parity). `:reload` deferred (orthogonal). -6. **New error Code `lib_load_failed`** — a file that exists but fails to - open/read (permissions, I/O error). `lib_not_found` (null resolver) conflates - "no such lib" with "found but unreadable"; clj distinguishes them. (No new - F-004 slot — `load_paths`/`loaded_libs` are plain Runtime fields.) -7. **Install the chain in CLI run + REPL** (both embedded-first, so core stays - deterministic). **Test Fixtures + the embedded bootstrap path stay - embedded-only** so a stray `.clj` in a test's cwd can never perturb a unit / - diff test. - -Deferred to their own debt rows: deps.edn (D-NNN), `:reload`/`:reload-all`, -the standalone-`require` multi-libspec wall (`special_forms.zig:480` — real -`(ns …)` forms don't hit it). - -## Consequences - -- A real pure-Clojure lib (dir of `.clj`) loads via `(require '[lib.core])` and - its `clojure.test` suite runs on cljw — the D-158 goal + the payoff of the - whole macro/ns/test stack. -- `RequireResolverFn` signature change ripples to the 3 raise sites + the - resolver-direct tests (bootstrap.zig / runtime.zig) — mechanical. -- `loaded_libs` holds gpa-owned dup'd ns-name keys, freed in `Runtime.deinit` - (mirroring `require_in_progress` teardown). -- Errors inside a loaded lib render with the lib's path:line (registerSource). - -## Affected files - -- `src/runtime/runtime.zig` — `load_paths`, `loaded_libs` fields + deinit; - `RequireResolverFn` → `?ResolvedSource`. -- `src/eval/loader.zig` (NEW) — `loadNamespace` + `ResolvedSource`. -- `src/lang/require_resolver.zig` (NEW) — `filesystemResolver` (ns→path munge, - search `load_paths`, read via `std.Io.File`) + `chainedResolver`. -- `src/lang/bootstrap.zig` — `embeddedResolver` returns `{source, label}`. -- `src/eval/backend/tree_walk.zig` + `vm.zig` — replace the 3 raises with - `loader.loadNamespace`. -- `src/runtime/error/catalog.zig` — `lib_load_failed` Code. -- `src/app/cli.zig` + `repl.zig` — parse `-cp`/`CLJW_PATH` → `load_paths`, - install `chainedResolver`. -- `test/e2e/phase15_require_fs.sh` (NEW) + a fixture lib under - `test/e2e/fixtures/`. - -## Alternatives considered - -(Verbatim from the mandatory Devil's-advocate fork, fresh context, agent -`a006f0ffeb7733a02`. Recommendation Alt 2; the main loop concurs.) - -> ## Alt 1 — SMALLEST-DIFF (cwd-relative, chained resolver, helper in -> bootstrap.zig) -> (a) cwd-relative only; munge + open `./.clj`/`.cljc`. No load_paths, -> no flag. (b) chainedResolver embedded-first in main. (c) helper in -> bootstrap.zig. (d) reuse findNs + activate require_in_progress, no loaded_libs. -> Better: minimal surface. **Breaks**: real libs are `src:test` (two roots) — -> D-158's own goal; cwd-only cannot find `test/foo_test.clj` from repo root. -> bootstrap.zig placement violates F-009 AND zone_deps (Layer-2 import from -> Layer-1 backends) — an independent disqualifier. A false economy: fails the -> goal and still pays the helper+cycle+dual-backend cost. -> -> ## Alt 2 — FINISHED-FORM-CLEAN (load_paths + new loader module, embedded-first -> chain) ⭐ -> (a) `rt.load_paths` from `-cp`/`CLJW_PATH`, default `["."]`; two roots is the -> real-lib minimum; defer deps.edn (correct layering, sits on the same field -> later). (b) single slot + `chainedResolver` embedded-FIRST (core never -> shadowed); a resolver list is premature (two sources; pod is a reservation — -> YAGNI). (c) new `src/eval/loader.zig` (Layer 1 — bootstrap.zig is Layer 2 and -> must not be imported by eval/backend/*); loadNamespace = registerSource → cycle -> guard → save ns → Reader-loop analyze+eval → defer restore ns → loaded_libs → -> pop in-progress; macro_table threads in; resolver returns {source,label} so the -> label is the path (disk) or `` (embedded). (d) findNs is necessary but -> NOT sufficient (partial mid-load ns exists with mappings>0 before body -> finishes); need require_in_progress + loaded_libs both; :reload defer. Better: -> the only alt satisfying D-158's multi-root goal; clean zone placement; cycle + -> idempotency as finished form; chain immune to shadowing. Breaks/risks: -> RequireResolverFn signature change ripples to 3 sites + resolver-direct tests -> (mechanical); loaded_libs lifetime = gpa-dup'd keys freed in deinit. Larger -> diff — recommend anyway per F-002. -> -> ## Alt 3 — WILDCARD (resolver returns loader closures / NsProvider objects) -> load_paths modeled as `rt.providers: []NsProvider`, each a self-contained -> resolve(ns)→{source,label} with its own munge policy; slot becomes a -> provider-list iterator. Better: maximally extensible (Phase-16 pod + AOT-cache -> providers slot in with zero churn). Breaks/risks: introduces a vtable-ish -> provider abstraction BEFORE the second non-embedded consumer exists — textbook -> Reservation-as-bias + over-skeleton (enlarges the finished-form rewrite, dead -> weight until Phase 16; risks the pub-var/vtable patterns zone_deps warns -> against). -> -> ## Cross-cutting -> (f) Error rendering: registerSource(label, bytes) as the FIRST step, label = -> resolved path; embedded label = `` sentinel (matches `` -> convention). The resolver returning {source,label} is what makes this correct -> for both kinds. (g) Standalone-require multi-libspec wall: DEFER — the ns macro -> expands per-spec, so real `(ns …)` never hits it; lifting it is a separate REPL -> parity item (own debt row), bundling = scope-creep. (h) Install: main/CLI + -> REPL get the chain (embedded-first preserves core determinism); test Fixtures + -> embedded bootstrap stay embedded-only (deterministic). New F-004 slot: none. -> New error Code: `lib_load_failed` (found-but-unreadable, distinct from -> lib_not_found); circular_require + lib_not_found already exist. -> -> ## Recommendation -> Alt 2: load_paths + `-cp`/`CLJW_PATH`, new `src/eval/loader.zig` shared by both -> backends, embedded-first 2-element chain (no provider list), -> require_in_progress + loaded_libs both activated, RequireResolverFn→{source, -> label}, deps.edn + :reload + multi-libspec wall deferred. The only alt that -> satisfies the multi-root goal; Alt 1 fails it + breaks zone_deps; Alt 3 builds -> Phase-16 machinery with no present consumer. Larger diff is not a downgrade -> reason (F-002). diff --git a/.dev/decisions/0085_namespace_as_runtime_value.md b/.dev/decisions/0085_namespace_as_runtime_value.md deleted file mode 100644 index 94b0c9796..000000000 --- a/.dev/decisions/0085_namespace_as_runtime_value.md +++ /dev/null @@ -1,205 +0,0 @@ -# ADR-0085 — Namespace as a runtime value: `*ns*`/`current_ns` binding round-trip + runtime `in-ns`/`use`/`refer` - -Status: Proposed → Accepted (2026-06-04) - -## Context - -The D-232 validation campaign cannot load ~20 upstream `clojure.test` -suites because they depend on `clojure.test-helper`, whose `temp-ns` / -`eval-in-temp-ns` use: - -```clojure -(binding [*ns* *ns*] - (in-ns (gensym)) - (apply clojure.core/use 'clojure.core uses) - *ns*) -``` - -Two cljw gaps block this (both confirmed by probing the binary + -reading source): - -1. **`(binding [*ns* *ns*] (in-ns 'tmp) …)` does not round-trip.** After - the binding frame pops, `current_ns` is still `tmp` — the next - top-level form runs in the wrong namespace (`println` unresolved). - ROOT: `*ns*` the Var is fine — `Var.deref` (env.zig:98) already - consults `findBinding` before the root, so `*ns*`'s *value* respects - thread-bindings. But `env.current_ns` is a SEPARATE stored `?*Namespace` - field (env.zig:266) read by ~40 resolution sites, and the generic - `binding` push/pop machinery never touches it. `in-ns`→`setCurrentNs` - writes the field + the Var root directly, bypassing the thread-binding, - so on pop the field stays stale. The field and the Var desync exactly - when a `binding [*ns* …]` frame is live. -2. **`in-ns`/`require` are analyzer special forms requiring a literal/ - quoted symbol; `refer`/`use`/`refer-clojure` do not exist at all.** - `(in-ns (gensym))` / `(apply clojure.core/use …)` need a RUNTIME - FUNCTION surface. cljw evaluates files one form at a time - (read→analyze→eval→next, runner.zig / loader.zig), so a runtime - `in-ns` that mutates the ns during eval of form N IS visible to the - analysis of form N+1 — the special form was a compile-time - convenience, not a necessity, for the top-level case. - -This is a structural change to how the namespace is represented at -runtime + to the special-form set. Per CLAUDE.md it is handled inline -with a mandatory Devil's-advocate fork (below). - -## Decision - -Make `current_ns` a **derived view of `*ns*`** (single source of truth), -realised by **eager materialisation**: the `current_ns` field is refreshed -from `env.ns_var.deref()` at the small, generic set of binding boundaries -— `setCurrentNs` (ns switch), `binding`-frame push, `binding`-frame pop — -so it always equals `*ns*`'s effective (thread-binding-respecting) value. -Readers keep reading the field (free; no per-resolution deref). The -refresh is generic (push/pop re-derive unconditionally; they never inspect -*which* var changed), so the binding machinery carries no per-var special -case, and the field is no longer an authoritative second copy — it is a -materialised cache of the Var. - -`in-ns` becomes `set!`-semantics on `*ns*`: mutate the innermost thread -binding if one holds `*ns*`, else the root (reusing the existing `set!` -helper), then re-derive `current_ns`. `in-ns`/`use`/`refer`/`refer-clojure` -gain a runtime-function surface (`clojure.core` Vars over one shared -`ns_ops` impl, reachable as `clojure.core/use` for the `apply` site); the -existing `in-ns`/`require`/`ns` special forms stay as literal-head fast -paths and fall through to the var on a non-literal arg. `in-ns` returns the -ns Value (ADR-0083), not nil. - -This realises the Devil's-advocate's recommended **Option 2** (single -source of truth = `*ns*`) while avoiding its 40-site `field→currentNs()` -call-site churn and the `*Env`-mutability (const-ness) wrinkle a lazy -caching accessor would impose: eager materialisation keeps the field a -plain read, and the refresh sites are the same few the lazy-cache -generation-bump would touch. - -### Commit decomposition - -- **Commit 1 (prerequisite, representation fix):** `current_ns` becomes - an eagerly-materialised view of `*ns*` — refresh helper called at - setCurrentNs + binding push + binding pop; `in-ns` set!-semantics. Test: - `(binding [*ns* *ns*] (in-ns 'tmp.x)) (println "ok")` resolves in the - restored outer ns. Dual-backend diff case (ADR-0036) — VM has its own - binding + in_ns arms. -- **Commit 2 (surface, runtime fns):** `in-ns`/`use`/`refer` builtins + - `refer-clojure` macro over a shared `ns_ops` impl; special-form - non-literal fall-through. Test: the `temp-ns` / `eval-in-temp-ns` shapes. -- **Commit 3 (optional):** turn on the ~20 test-helper-dependent suites. - -Commit 1 must precede Commit 2: without the round-trip fix, a runtime -`in-ns` inside `(binding [*ns* *ns*] …)` leaves `current_ns` at the temp -ns after pop, masking whether the fns work. - -## Consequences - -- The `binding [*ns* …]` round-trip works → `temp-ns` / `eval-in-temp-ns` - → the test-helper cluster (~20 suites) becomes loadable. -- `current_ns` is no longer an authoritative second copy of the namespace; - it is a materialised view of `*ns*`. Any FUTURE binding entry point (a - Phase-16 thread that inherits bindings, a `with-bindings` primitive) must - call the single refresh helper — this residual is contained to one helper - invoked at the enumerable binding boundaries, not a per-var special case. -- Both backends change identically (both refresh via the same helper), - preserving TreeWalk↔VM parity (F-012); the ADR-0036 dual-backend gate - rides Commit 1. -- `in-ns`/`use`/`refer` reach clj's "these are functions" finished form - (F-002); the special forms are reframed as literal-head fast paths over - the canonical Vars (F-011 single mechanism). - -### Divergences from clj (intentional) - -- `in-ns` returns the ns Value (ADR-0083), matching clj. -- `(do (require …) (aliased/sym))` within a SINGLE form: the alias is - installed at require-eval, after the whole `do` was analysed, so the - in-form aliased symbol does not resolve. clj has the same top-level- - sequential constraint; only multi-form files are affected, which cljw - handles via per-form eval. Unchanged by this ADR. -- `:rename` / `:as-alias` on a runtime libspec stay an explicit - unsupported raise (separate deferred feature). - -## Alternatives considered - -The mandatory Devil's-advocate subagent (fresh context, briefed with the -F-002/F-009/F-011/F-012 envelope) produced the following, reflected -verbatim: - -> **Key correction:** `Var.deref` ALREADY consults `findBinding` for -> dynamic vars before root — so `*ns*` deref already respects -> thread-bindings. The bug is purely that `current_ns` (the FIELD read by -> ~40 resolution sites) is not kept in sync with the `*ns*` var's effective -> binding. Also: the `in_ns_node`/`require_node`/`ns_node` DO have VM arms -> (vm.zig:515-555) — the survey's "VM-free" claim is wrong; the -> dual-backend gate (ADR-0036) applies to Commit 1. -> -> **Option 1 — smallest-diff:** keep `current_ns` a field; make `binding` -> push/pop `*ns*`-aware (sync the field only when a pair binds `ns_var`); -> `in-ns` setBinding-if-frame-else-setRoot. Better: zero hot-path cost, -> smallest blast radius, reuses the existing `set!` pattern. Breaks/risks: -> the dual representation PERSISTS — every future binding entry point must -> remember to sync the field; a per-var special-case lives in the generic -> binding machinery forever; forgetting it silently re-arms the desync bug. -> This is the Smallest-diff bias smell. ~6 files. -> -> **Option 2 — finished-form-clean (RECOMMENDED):** `current_ns` becomes a -> derived view of `*ns*` (`currentNs() = ns_var.deref()` decoded), with a -> generation-counted cache (binding push/pop bump a generation; currentNs() -> recomputes only on generation change → one int-compare in the common -> case, same cost as today's field read). `in-ns` is set!-semantics on -> `*ns*` with no field to also update; the binding machinery needs NO -> `*ns*` special-case (the generic mechanism already makes the deref -> correct). Better: collapses the dual representation — exactly one answer -> to "what is the current ns", impossible to desync, every future binding -> path / backend / thread correct for free. Breaks/risks: the cache is -> load-bearing correctness (one generation-bump site to get right, vs -> Option 1's N field-sync sites — strictly fewer); touching `currentNs()` -> ripples to ~40 mechanical call-site renames (per F-002 NOT a reason to -> prefer Option 1 — that instinct is the Cycle-budget-defer smell); the -> bootstrap window (ns_var == null) needs a field fallback. ~15 files, -> mostly one-token edits. Largest diff; cleanest result. Does NOT violate -> any F-NNN. -> -> **Option 3 — wildcard:** eliminate the hot `current_ns` reads instead of -> caching them — bake alias/refer resolution fully into the Node at analyze -> time so backends never read ambient ns; `currentNs()` then an uncached -> deref read only by the (cold) analyzer + rare ns-mutating ops. Better: no -> cache AND no field. Breaks/risks: the premise (hot reads are eliminable) -> is UNVERIFIED — runtime reads survive at vm.zig def-target + macro_dispatch -> (warm); converting them is a deep change (def lowering, macro synthetic -> vars) of unbounded blast radius until an audit that is itself the risk; -> does not by itself fix the round-trip for sites that legitimately read -> ambient ns at runtime (`resolve`). Highest uncertainty. Revisit as a -> later optimisation on top of Option 2 if profiling demands. -> -> **Recommendation: Option 2, decomposed into Commit 1 (representation -> fix) + Commit 2 (runtime fns) + optional Commit 3 (turn on suites).** -> The "general binding machinery, no per-var special case" property is the -> decisive cleanliness win over Option 1; the larger call-site diff is not -> a valid reason to prefer Option 1 (Smallest-diff / Cycle-budget-defer -> smell). Option 1 is the fallback only if a call-site is found where the -> derived view is impossible (none found in the audit). - -**Main-loop decision within the F-NNN envelope:** adopt Option 2's single -source of truth, but realise `currentNs()` by **eager materialisation of -the `current_ns` field** (refresh from `*ns*` deref at the setCurrentNs + -binding-push + binding-pop boundaries) rather than a lazy caching accessor. -This keeps Option 2's "current_ns is a view of `*ns*`, not an authoritative -copy" semantics and its "no per-var special case" property (push/pop -re-derive unconditionally), while avoiding (a) the 40-site `field → -currentNs()` churn and (b) the `*Env`-mutability requirement a self-caching -accessor imposes on ~40 read sites, several of which would need const→mut -signature changes. The residual — a generic refresh helper invoked at the -binding boundaries — is the same small surface Option 2's generation-bump -would occupy; it is NOT the per-var push/pop special-case that made Option 1 -a smell (the refresh never inspects which var changed). Per F-002 this is -chosen on finished-form grounds (it is strictly cleaner than both -enumerated options for cljw's const-discipline), not on diff-size grounds. - -## Cross-references - -- D-232 (validation campaign) — the driver; the test-helper cluster is - the unblock target. -- ADR-0083 (`*ns*` as `.ns` Value + `ns_var` cache + `setCurrentNs` - contract) — this ADR makes `current_ns` a view of that Var. -- ADR-0036 (dual-backend diff gate) — Commit 1 rides it. -- F-002 (finished-form wins), F-011 (one shared mechanism + clj-oracle - equivalence), F-012 (VM is production default — both backends change - identically). -- private/notes/phase15-in-ns-runtime-fn-survey.md (Step-0 survey). diff --git a/.dev/decisions/0086_macro_form_env_args.md b/.dev/decisions/0086_macro_form_env_args.md deleted file mode 100644 index 61a0485ab..000000000 --- a/.dev/decisions/0086_macro_form_env_args.md +++ /dev/null @@ -1,188 +0,0 @@ -# ADR-0086 — Implicit `&form` / `&env` arguments for `defmacro` - -Status: Proposed → Accepted (2026-06-04) - -## Context - -cljw's `defmacro` does not thread the implicit `&form` / `&env` -arguments that every JVM Clojure macro receives. `analyzeDefmacro` -(special_forms.zig) lowers `(defmacro m [a] body)` to `(fn* [a] body)` -and `expandIfMacro` (macro_dispatch.zig) calls that fn with only the -user args. The original D-099 landing recorded this as a deliberate -omission ("Tier-A test corpora do not introspect them; threading both -is D-099-followup") — at the time no consumer needed them. - -The post-M real-world-library drain (F-010/F-011) changes that. `&env` -appears in **104 corpus files**, and two concrete consumers fail to -load today: - -- **slingshot** (`slingshot/support.clj:198`): `(contains? &env sym)` - — needs `&env` to be a real map whose KEYS are the in-scope local - symbols. -- **meander** (`meander/epsilon.clj`): `(meta &form)` — needs `&form` - to carry the call form's reader `:line`/`:column` metadata. - -Survey: `private/notes/phase15-form-env-survey.md`. - -### JVM semantics (the convention to match) - -`Compiler.java` macroexpand1 calls the macro fn as -`(form, LOCAL_ENV, ...user-args)`, so the lowered fn is -`[&form &env & user-args]`: - -- **`&form`** = the whole call form (head symbol included), a seq that - is `IObj` and carries the reader's `{:line :column}` meta. `(meta - &form)` → that map. -- **`&env`** = a `PersistentHashMap` keyed by the in-scope local - **Symbol**, valued by an opaque `LocalBinding`. **No `:ns` key** (the - `:ns`/`:locals` shape is ClojureScript, not JVM). `{}` (never nil) at - the top level. -- Reported user arity is `method-arity − 2`. - -## Decision - -Thread `&form` and `&env` through `defmacro`, matching the JVM -calling convention, with three cljw-specific representation choices -forced by cljw's design (no JVM classes, metadata-less symbols): - -1. **Lowering** (`analyzeDefmacro`): prepend `&form &env` to every - arity's param vector, so the macro fn is `(fn* [&form &env & params] - …)` (single- and multi-arity). This mirrors clj core.clj's - `add-implicit-args`. - -2. **Dispatch** (`expandIfMacro` + its analyzer call site): the - analyzer — which owns the lexical `Scope` and the call `Form` — - builds two Values and the dispatcher prepends them before the user - args: - - **`&form`**: the call form `(head a b …)` rebuilt as a list - Value, with `{:line :column}` metadata synthesized from - `Form.location` and attached via `list.withMeta`. (cljw Forms are - metadata-less as *symbols* per ADR-0037, but list Values carry - meta, so `(meta &form)` works.) - - **`&env`**: a map Value whose keys are the in-scope local symbols - (walked up the `Scope.parent` chain). Because cljw has no - `LocalBinding` class (ADR-0009 / no-JVM ADR-0059), the **value** - for each key is the local symbol itself (a stable, metadata-free - placeholder). **No `:ns` key** (mirror JVM, not cljs). Empty map - `{}` at the top level, never nil. - -3. **Arity**: user-visible arity errors subtract the 2 implicit params - (the macro author wrote `[a b]`, not `[&form &env a b]`). - -Lowering and dispatch **land in the same commit** — either alone yields -arity mismatches. - -### cljw divergences from JVM (recorded as accepted) - -- **`&env` values are the local symbol, not a `LocalBinding`.** No JVM - class exists; macros use `&env` KEYS (`contains?`, `keys`), not the - opaque values, so a symbol placeholder is behaviour-preserving for - the real consumer set. (AD candidate, derives_from ADR-0059/ADR-0009.) -- **No `:ns` key in `&env`.** Matches JVM; cljw is not ClojureScript. -- **`&form` metadata is synthesized from `Form.location`** at expansion - time rather than carried on a metadata-bearing reader Value. - -## Consequences - -- `defmacro` macros can introspect their call form (`&form`, incl. - `(meta &form)` for source location) and the lexical environment - (`&env`, incl. `(contains? &env 'x)`). -- slingshot + meander advance past their `&env`/`&form` walls. -- The macro calling convention changes: any code that defined a macro - and relied on the old no-implicit-args lowering would break — but - that lowering was non-conforming, so this is a fix, not a break. -- Closes D-099-followup. - -## Affected files - -- `src/eval/analyzer/special_forms.zig` — `analyzeDefmacro` lowering - (prepend `&form &env` to each arity's params). -- `src/eval/macro_dispatch.zig` — `expandIfMacro` signature (+ the - call form + the env map) + prepend before user args. -- `src/eval/analyzer/analyzer.zig` — call site builds the `&form` - (with synthesized meta) + `&env` (from `scope`) Values. -- Tests: `test/e2e/phase14_defmacro_user.sh` (+`&form`/`&env` cases), - `src/lang/diff_test.zig` (dual-backend), corpus where clj-expressible. - -## Devil's-advocate resolution (the key finding, verified) - -The DA fork's central objection was that cljw's `Scope` contains -destructuring **gensym temps** (`vec__N` / `map__N`) in addition to -user locals, so an `&env` built from all Scope keys would "leak" them -and diverge from JVM (which the survey claimed is user-locals-only). - -**This premise is false — verified against the clj oracle:** - -```clojure -(defmacro envkeys [] (prn (sort (map str (keys &env)))) nil) -(let [[a b] [1 2] {c :c} {:c 3} x 5] (envkeys)) -;; clj → ("a" "b" "c" "map__142" "vec__139" "x") -``` - -JVM `&env` **includes the destructuring gensym temps**. So building -cljw's `&env` from the full `Scope` binding chain (user locals + temps) -is the F-011-**correct** behaviour, not a leak. The only divergence is -the temp *names* (cljw names its gensyms differently), which is an -accepted opaque divergence — gensym names are never stable or compared, -exactly like set/hash print order (AD-001). The DA's Alt-2 "key-set -fidelity" work (filter temps, thread user-only symbols) would actually -make cljw *diverge* from JVM, so it is rejected. - -### `&env` value representation (refinement vs the DA's Alt 2) - -The DA recommended `&env` values be a `{:name :tag}` map for forward -type-hint introspection. **Rejected as Reservation-as-bias** -(`.dev/principle.md`): cljw's `Scope` stores only `name → slot` (no -type hints), so `:tag` would be perpetually `nil` — speculative -structure for a feature (lexical type-hint capture) that does not -exist. The F-011-observable contract is `(contains? &env sym)` → true, -`(keys &env)` → the local symbols, `(get &env sym)` → truthy/opaque -(JVM returns an opaque `LocalBinding`; cljw has no such class per -ADR-0059). The **local symbol itself** satisfies all three (truthy, -opaque-enough) without speculative keys. **Upgrade path:** when cljw -gains lexical type-hint capture, the value becomes a binding-info map -at that point (when `:tag` is actually populable) — not before. - -`nil`-as-value (DA Alt 1) is rejected: it breaks the legal -`(if-let [b (get &env sym)] …)` idiom (present local reads as absent). - -The DA's two correct points are **adopted**: (1) `&form`'s synthesized -`{:line :column}` meta is meander-sufficient (kept as-is); (2) the -`&form`/`&env` Values are built on the **runtime GC heap** (like the -existing `value_args` in `expandIfMacro`), not the analyzer arena, so -the VM compile path (F-012) sees stable Values. - -## Alternatives considered - -(Devil's-advocate fork output, verbatim.) - -**Alt 1 — smallest-diff: `&env` value = nil, keys = raw Scope names.** -Better: honest that cljw has no LocalBinding; `nil` cannot be mistaken -for a binding. Breaks: same temp-key behaviour; `(get &env sym)` → nil -indistinguishable from absent (breaks `if-let` idiom). Verdict: worse -on the `get` idiom, not recommended. - -**Alt 2 — finished-form-clean: `{:name :tag}` value + key-set fidelity -(filter gensym temps).** Better (per DA): forward-compat `:tag`; "fixes" -the temp leak. Breaks: the temp-leak "fix" is wrong — JVM includes temps -(verified above), so filtering them *causes* a divergence; `:tag` is -speculative (no type hints in Scope). Verdict: rejected on the verified -oracle behaviour + Reservation-as-bias. - -**Alt 3 — wildcard: `&env` always `{}`, `&form` real, defer keys.** -Better: eliminates any temp-key question; `&form` lands uncoupled. -Breaks: `(contains? &env sym)` always false → **silently drops -slingshot's locals** (the named consumer), a permanent-no-op / -silent-semantics-drop (forbidden). Verdict: only viable as a labelled -PROVISIONAL, never finished form. - -**DA recommendation:** Alt 2. **Main-loop decision (within F-NNN):** -neither the DA nor the survey verified the JVM `&env` key-set; the -oracle shows JVM *includes* destructure temps, which inverts Alt 2's -core premise. The finished form is therefore: `&env` keys = the full -Scope local set (matches JVM incl. temps), `&env` value = the local -symbol (the type-hint map is the documented upgrade when hints land, -not a speculative reservation now), `&form` = the DA-endorsed -synthesized-meta list on the runtime heap. This is more finished-form -than Alt 2 because it matches the *actual* clj behaviour rather than an -unverified assumption about it. diff --git a/.dev/decisions/0087_persistent_queue.md b/.dev/decisions/0087_persistent_queue.md deleted file mode 100644 index 9b6571f8c..000000000 --- a/.dev/decisions/0087_persistent_queue.md +++ /dev/null @@ -1,184 +0,0 @@ -# ADR-0087 — PersistentQueue (FIFO persistent collection, slot 45) - -Status: Proposed → Accepted (2026-06-04) - -## Context - -`clojure.lang.PersistentQueue` (the immutable FIFO queue) is a real -Clojure collection that user code uses directly via -`clojure.lang.PersistentQueue/EMPTY` + `conj`/`peek`/`pop`/`seq`. cljw -reserved the NaN-box tag `persistent_queue = 45` (F-004 enumerated -types) but never implemented it: no collection module, no constructor, -no dispatch. The recurring real-world-library drain (medley, aero) and -the interface-membership sweep this session (which already wired -`persistent_queue` into every `instance? clojure.lang.X` arm, -`sequential?`, `coll?`, `seq?`) leave the actual value type as the only -missing piece. - -Survey: `private/notes/phase15-persistent-queue-survey.md`. - -### JVM semantics (verified) - -Okasaki batched queue: `cnt` + front `ISeq f` + rear `PersistentVector r` -(+ meta). `conj`/`cons`: the first element seeds `f`; subsequent -elements append to `r`. `peek`: `(first f)`. `pop`: `(rest f)`; when `f` -empties, `r` becomes the new front (via `seq`) and `r` resets — **pop of -empty returns the empty queue, no throw**. `seq`: front ++ rear-seq, or -`nil` when empty. `count` = `cnt`. `empty` = `EMPTY` with this queue's -meta. `=`/hash are ordered-sequential. `EMPTY` is a static singleton. - -**Print (verified on host clj):** JVM has NO `print-method` for queues — -it prints the opaque `#object[clojure.lang.PersistentQueue 0xADDR -"…@hash"]`, embedding an identity hash cljw cannot and should not -reproduce. The `<-(…)-<` form is ClojureScript, not JVM. - -## Decision - -Implement PersistentQueue as a real collection on the reserved tag 45, -mirroring the JVM batched-queue representation with cljw's existing -collections, and resolving the EMPTY-static + print questions cljw's way: - -1. **Representation** (`src/runtime/collection/persistent_queue.zig`, an - `extern struct` like `map_entry.zig`): `count: i64`, `front: Value` - (a cljw seq/list, or nil), `rear: Value` (a cljw vector, or nil), - `meta: Value`. `conj` seeds `front` on the first element then conjes - to `rear`; `peek` = first of `front`; `pop` = rest of `front`, - migrating `rear`→`front` (via `seq`) when `front` empties (pop of - empty returns EMPTY); `seq` = front ++ (seq rear) or nil; `count` - returns the field; `empty` = EMPTY (meta-preserving). **`=`/hash: a - queue is `Sequential`, so it routes through the SHARED - Sequential-equality + ordered-hash paths** (add `.persistent_queue` - to `isSeqLike` in equal.zig + the seq-hash dispatch — no bespoke - queue equality). **A queue is `=` to ANY other Sequential (vector / - list / seq / lazy-seq) with equal elements in order** - (oracle-verified: `(= (conj EMPTY 1 2 3) [1 2 3])` → **true**, - `(= EMPTY [])` → true), and **NOT** `=` to a set or `nil` - (`(= EMPTY #{})` / `(= EMPTY nil)` → false); hash = the ordered hash - of the equal vector/list. GC trace marks front/rear/meta (mirror - `atom.zig` traceGc). - -2. **EMPTY singleton**: a per-Runtime `rt.empty_queue`, allocated once on - the GC `infra` arena and never swept (the `rt.empty_list` pattern from - ADR-0076 / D-164), freed in `Runtime.deinit`. - -3. **`clojure.lang.PersistentQueue/EMPTY` resolution**: the ADR-0061 - bare-static-field path uses a scalar-only `StaticFieldValue` union — - extend it with a **`singleton: enum { empty_queue }`** variant - (NOT a raw `value: Value` — the analyzer may run before/independently - of the runtime that owns `rt.empty_queue`; the enum defers the pointer - to resolution time, and future `…/EMPTY` fields reuse the seam). The - resolver maps `.empty_queue` → `rt.empty_queue`. Register - `PersistentQueue` in **both** `class_name.NATIVE_ENTRIES`/FQCN_MAP - (so `(class q)` → `PersistentQueue`, AD-003) **and** `rt.types` (so - `PersistentQueue/EMPTY` resolves via the Java-surface path) — the two - use different tables (survey §4). - -4. **Print** (accepted divergence — JVM's opaque `#object[… 0xADDR - "…@hash"]` embeds a non-reproducible identity hash; `(str q)` and - `(pr-str q)` differ only in that wrapper, both non-reproducible): - cljw prints a **reader-round-trippable** `#queue (e1 e2 …)` form - **and ships a matching `queue` data-reader** (`#queue coll` → - `(into EMPTY coll)`) so the form is honest EDN, not a false - tagged-literal promise (the DA's key correction). `str` and `pr-str` - of a queue are intentionally identical (clj's differ only in the - non-reproducible wrapper). `(pr-str (conj EMPTY 1 2 3))` → - `#queue (1 2 3)`; empty → `#queue ()`. Recorded as a new `AD-NNN` - deriving from ADR-0059 (no-JVM identity hash). - -5. **No new Node / VM opcode** (ADR-0036): a queue has no reader literal; - it is reached purely via `conj`/EMPTY, so both backends produce it - identically (same as `map_entry`). A differential case confirms parity. - -## Consequences - -- `(conj clojure.lang.PersistentQueue/EMPTY 1 2 3)` builds a queue; - `peek`/`pop`/`seq`/`count`/`=`/`into`/`empty?` work; `(queue? x)` via - `instance?` already works (this session). -- medley/aero advance past their PersistentQueue walls (they have other - deep deps, so this is a standalone-collection win, not a full unblock). -- The queue print form diverges from JVM's opaque `#object` (AD-NNN) — - intentionally readable. - -## Affected files - -- `src/runtime/collection/persistent_queue.zig` (new) — struct + conj / - peek / pop / seq / count / empty / make / EMPTY / GC trace. -- `src/runtime/runtime.zig` — `empty_queue` singleton field + init/deinit. -- `src/main.zig` — `test { _ = @import("…persistent_queue.zig"); }` reach. -- `src/lang/primitive/collection.zig` — `conj` arm. -- `src/lang/primitive/sequence.zig` — seq / first / rest / next / count / - empty arms. -- `src/lang/clj/clojure/core.clj` — `peek`/`pop` queue arms. -- `src/runtime/equal.zig`, `src/runtime/hash.zig` — queue equality/hash. -- `src/runtime/print.zig` — `#queue (…)` print. -- `src/eval/analyzer/analyzer.zig` + the StaticFieldValue union — - `PersistentQueue/EMPTY` resolution. -- `src/runtime/class_name.zig` — `PersistentQueue` NATIVE_ENTRIES + - FQCN_MAP (already in the instance? interface arms). -- `.dev/accepted_divergences.yaml` — AD-NNN (queue print form). -- Tests: `test/e2e/`, `src/lang/diff_test.zig`, corpus. - -## Devil's-advocate resolution - -The DA fork ran the clj oracle and surfaced three corrections, all -adopted into the Decision above: - -1. **Equality (highest priority — the original draft was wrong).** The - draft said "ordered-sequential" and the framing assumed a queue is - NOT `=` to a vector. The oracle says the opposite: - `(= (conj PQ/EMPTY 1 2) [1 2])` → **true**. A queue is `=` to any - other `Sequential` with equal elements (vector / list / seq / - lazy-seq), `≠` set/`nil`, and `(hash q) == (hash [1 2 3])`. The - implementation reuses the shared Sequential-equality + ordered-hash - paths. The corpus pins `(= q [1 2 3]) → true`, `(= EMPTY []) → true`, - `(not= EMPTY nil)`. -2. **Print.** `#queue (…)` without a reader falsely implies EDN data. - Adopted the DA's finished form: ship the `#queue (…)` print **plus** a - `queue` data-reader so it round-trips in cljw (clj's own `#object` - form does not round-trip either, so cljw is strictly better). -3. **EMPTY resolution.** A raw `value: Value` baked at analyze time is - GC-unsafe (the singleton may not exist yet); use a `singleton: enum` - variant resolved to `rt.empty_queue` at resolution time. - -The representation (extern-struct list+rear-vector on reserved tag 45) -and the no-Node/no-opcode dual-backend conclusion were confirmed -finished-form and kept. Using slot 45 (not a `typed_instance` shim) is -correct — consuming the reservation like map_entry consumed slot 15; a -typed_instance would route every dispatch through user-protocol slow -paths. - -## Alternatives considered - -(Devil's-advocate fork output, verbatim.) - -**Alt 1 — smallest-diff: ADR representation + cljs `<-(…)-<` print + -D2a name-keyed EMPTY.** Better: smallest EMPTY plumbing (no -StaticFieldValue surgery). Breaks: D2a adds a `"EMPTY"` string-compare -in the hot symbol-analysis path that the finished-form owner unwinds -when PersistentList/Vector/HashSet `…/EMPTY` land (re-litigated 3×) — -Reservation-as-bias-in-reverse. Rejected on F-002. - -**Alt 2 — finished-form-clean (RECOMMENDED): ADR representation + D2b -generic singleton static-field + reuse seq-equality/hash + `#queue (…)` -*with* a matching data-reader (else `<-…-<`).** Better: D2b is the seam -every future `…/EMPTY` reuses (no hot-path string branch); equality is -correct + free + honestly documented; print either round-trips for real -or makes no false promise. Breaks/risks: D2b touches the -`StaticFieldValue` union (Zig exhaustive switch makes the fan-out -compiler-enforced — a feature); shipping the `queue` reader is a small -extra surface but the only way `#queue` is non-mendacious. The F-002 -pick. - -**Alt 3 — wildcard: single vector + head-index deque instead of -list+rear-vector.** Better: one GC child; O(log32) front random access; -no rear→front migration branch. Breaks: pop-without-compaction leaks -(drained elements pinned forever — unbounded growth under conj/pop); -adding compaction reintroduces complexity + breaks structural sharing; -diverges from JVM's amortized-O(1) Okasaki guarantee; throws away the -exact-mirror-of-JVM auditability. Rejected (better only on child count, -which is not a constraint; worse on correctness/finished-form). - -**DA recommendation:** Alt 2. **Main-loop decision:** adopt Alt 2 with -the three corrections folded into the Decision. The representation, slot -choice, and dual-backend conclusion stand; the equality prose, print -form, and EMPTY-resolution shape are corrected per the oracle + the DA. diff --git a/.dev/decisions/0088_print_length_level.md b/.dev/decisions/0088_print_length_level.md deleted file mode 100644 index 16127601f..000000000 --- a/.dev/decisions/0088_print_length_level.md +++ /dev/null @@ -1,276 +0,0 @@ -# ADR-0088 — `*print-length*` / `*print-level*` print-control vars - -Status: Proposed → Accepted -Date: 2026-06-04 -Phase: post-M quality loop (F-011 clj-parity) - -## Context - -The print-control dynamic var family is entirely absent in cljw: -`*print-length*`, `*print-level*`, `*print-dup*`, `*print-readably*`, -`*print-meta*`, `*print-namespace-maps*`, `*flush-on-newline*` all fail -to resolve as symbols. Binding them is a no-op — a real F-011 -behavioural gap (a user's `(binding [*print-length* 100] …)` silently -prints unbounded, which in a REPL on an infinite/large structure is the -difference between a readable result and a hang). - -This ADR lands the two highest-value members — `*print-length*` and -`*print-level*` — the truncation controls that make printing a large or -deeply-nested (or infinite, bounded by length) structure safe. The -remaining siblings are deferred (see Consequences): `*print-dup*` has no -cljw print-dup path, `*print-readably*` is already modelled by the -`print_readably` threadlocal, `*print-meta*` needs the value-meta print -surface, `*print-namespace-maps*` is a toggle over already-correct -behaviour (its absence only blocks *disabling* the compact form). - -### Exact JVM semantics (probed against `clj` 1.12, the corpus golden) - -- `*print-length*` (root nil = unlimited): each collection prints at most - N elements; if more remain, a `...` pseudo-element is emitted after the - normal separator. Exactly-N prints all N with no `...`. `0` → `(...)`. - Maps count **entries** (k/v pairs), not scalars. - - `(binding [*print-length* 3] (pr-str (range 10)))` → `"(0 1 2 ...)"` - - `(binding [*print-length* 3] (pr-str [1 2 3]))` → `"[1 2 3]"` - - `(binding [*print-length* 0] (pr-str [1 2 3]))` → `"[...]"` - - `(binding [*print-length* 2] (pr-str {:a 1 :b 2 :c 3}))` → `"{:a 1, :b 2, ...}"` -- `*print-level*` (root nil = unlimited): a **collection** at nesting - depth `d` (root collection = depth 0) prints as the single char `#` - iff `d >= level`. Scalars are never replaced (only collections route - through the check). - - `(binding [*print-level* 2] (pr-str [[[[1]]]]))` → `"[[#]]"` - - `(binding [*print-level* 1] (pr-str [1 [2 [3]]]))` → `"[1 #]"` - - `(binding [*print-level* 0] (pr-str [1 2]))` → `"#"` -- Composed: `(binding [*print-level* 2 *print-length* 2] (pr-str [1 2 3 [4 5 6 [7]]]))` - → `"[1 2 ...]"` (length cuts at 2 before the nested vector is reached). - -## Decision - -1. **Define** `(def ^:dynamic *print-length* nil)` and - `(def ^:dynamic *print-level* nil)` in `src/lang/clj/clojure/core.clj` - next to `*unchecked-math*` (line ~25). - -2. **Cache** a `*const Var` pointer to each from `bootstrap.zig` after - `core.clj` loads (mirroring `registerNsVar` → `env.ns_var`, - `bootstrap.zig:244`), via a new `print.zig` setter - `initPrintLimitVars(len_var, lvl_var)`. The Var *identity* is fixed - after intern, so the pointer is a process-global `?*const Var`; the - *binding* is read live. `Var.deref()` (`env.zig:103`) reads the - threadlocal `current_frame` binding stack and needs no `rt`/`env`, so - the pure `printValue(w, v)` renderer can read the user's current - binding through the cached pointer with no signature change. - -3. **Snapshot-at-surface, deref-once** (revised after DA fork). Each - top-level print surface (`pr`/`prn`/`print`/`println`/`pr-str`/ - `print-str` and `printResult`) snapshots the two limits ONCE at entry - into `threadlocal var print_length_limit: ?i64` / - `print_level_limit: ?i64` by dereffing the cached Var pointers, and - resets `threadlocal var print_depth: i64 = 0`. Because the surface - runs *inside* the user's `(binding [*print-length* …] …)` scope, the - single deref reads the user's current binding — responds to `binding` - exactly, while paying one binding-stack walk per top-level print - instead of one per element (the DA fork's per-element-deref objection). - Printing never evaluates user code, so no re-entrant top-level print - occurs; a hypothetical nested top-level print re-snapshots fresh - (depth-0), matching clj. This reuses the established `print_readably`/ - `print_namespace_maps` surface-threadlocal pattern rather than - introducing a new state model. - -4. **`*print-level*` threading** — centralised in `printValue`'s - dispatch for the collection tags **including `.map_entry` and the - `tagged-literal` form** (clj counts a MapEntry as a vector level): - before dispatching to a collection helper, `if (print_level_limit) - |lvl| if (print_depth >= lvl) { write "#"; return; }`, then - `print_depth += 1; defer print_depth -= 1;` around the helper call. - -5. **`*print-length*` threading** — each collection element loop gains a - guarded `if (print_length_limit) |n| { if (count >= n) { emit - separator ++ "..."; break; } }`. The two RECURSIVE walkers - (`printHamtEntries`, `printSortedEntries`) thread an extra - `remaining`/`count` pointer alongside the existing `*bool first` — the - DA fork correctly flagged that decision 5 is NOT a flat `i >= n` for - the unordered/sorted collections. All collections are covered (the - big-bang discipline), even though hash-set/hash-map > 8 keys stay out - of the parity corpus per the AD-001 order note below. - -6. The default (both nil) path is byte-identical to today → near-zero - regression surface. The snapshot is read as null (= unlimited) when - the Var is unbound, nil, or non-integer (clj's `(and *print-length* …)` - guard). - -### Known limitation — infinite lazy seq + `*print-length*` (deferred) - -`printResult` deep-realizes lazy seqs BEFORE printing, so -`(binding [*print-length* 3] (prn (iterate inc 0)))` still hangs in -realization (the print-loop guard never fires). This unit delivers -truncation for **finite collections and the compact `.range` type**; -infinite-seq termination needs print-while-forcing (an `rt`-carrying -printer, entangled with the D-164 seq-realization overhaul) and is filed -as a debt row with the clj-`(0 1 2 ...)`-terminates case as its pin. The -parity corpus uses finite / range / bounded cases only — no over-claim. - -## Alternatives considered - -**Chosen vs DA recommendation.** The DA fork (verbatim below) recommended -Alt 2 (a value-threaded `PrintCtx`) extended with Alt 3's lazy-seq fix. -This ADR adopts neither wholesale, on **merit grounds (not cycle budget)**: - -- The DA's two concrete objections to "threadlocals" both target the - *per-element-deref* shape it assumed. The **snapshot-at-surface** - refinement (decision 3) derefs ONCE per top-level print, eliminating - the per-element cost, and re-snapshots fresh on any (hypothetical) - nested top-level print — so both objections (perf + re-entrancy - unbalance) dissolve without `PrintCtx`. The DA did not evaluate this - hybrid. -- `PrintCtx` would also require migrating the two EXISTING, working - surface threadlocals (`print_readably`, `print_namespace_maps`) into - the struct for consistency — churning correct code for an aesthetic - unification. clj itself models these as dynamic vars (= thread-local - bindings), so the threadlocal shape is the **F-011-faithful** finished - form, not a compromise. `PrintCtx`'s remaining advantages - (concurrency / eval-during-print isolation) are **future** — cljw has - no eval-during-print path today; threads get independent threadlocals - regardless. Filed as a debt note: revisit `PrintCtx` if Phase 15 - introduces concurrent or eval-during printing. -- Alt 3's finding (the infinite-lazy-seq hang) is ACCEPTED as a real gap - and filed as debt (see Known limitation); its proposed mechanism - (couple `deepRealize` to the print Var) the DA itself judged the wrong - layer. -- Option C (merge the 7 loops into one `printSequential`) is REJECTED, - per the DA's grounded analysis: the backings are heterogeneous - (array / HAMT recursion / LLRB recursion / range arithmetic) and a - uniform iterator either heap-materialises (destroying deliberate - alloc-free properties) or relocates the heterogeneity into a tagged - iterator — a leakier finished form, not a cleaner one. - -The `.map_entry` depth-count fix the DA flagged is folded into decision 4. - -### Devil's-advocate subagent output (verbatim) - -I read the ADR, all eight collection helpers in `src/runtime/print.zig`, -and the F-NNN constraints. Three alternatives within the F-NNN envelope, -then a recommendation. - -**Grounding observations from the actual code** - -1. The "~7 loops" are genuinely heterogeneous backings. `printVector` - (array index), `printRange` (pure `start + i*step` arithmetic, zero - allocation, no `rt`), `printList`/`printQueue` (Cons walk), - `printMap`/`printSet` (HAMT recursion via `printHamtEntries`, threading - a `*bool first` through pre-order traversal), `printSortedMap`/ - `printSortedSet` (LLRB in-order recursion via `printSortedEntries`, - also `*bool first`-threaded). The HAMT and LLRB cases are not flat - loops — they are recursive node walks where "element index" is an - accumulated traversal position. Decisive for Option C. -2. `printValue` is `Writer.Error!void` — a pure renderer with no - `rt`/`env`. The cached-`*const Var` + `Var.deref()` mechanism is the - only way to read a user binding without a signature change. - `Var.deref()` reads `current_frame` threadlocal, so it works. -3. `printResult` already does a deep-realize pass (`deepRealize`) before - `printValue` runs, materialising lazy seqs. Correctness trap for - `*print-length*` over infinite seqs (Alt 3). -4. `print_readably`/`print_namespace_maps` are surface-set threadlocals, - not user-bindable. `*print-length*`/`*print-level*` MUST respond to - `(binding …)`, so the value must be read live through a Var, not - snapshotted by the surface to a fixed value. The ADR gets this right — - the single most important correctness call. - -**Alt 1 — Smallest-diff:** ADR-as-drafted but fold the length guard into a -tiny inline helper `emitEllipsisIfOver(w, i, sep) !bool`; leave level -centralised. BETTER: removes copy-paste across 7 sites (DRY; one place to -regress). Smallest behavioural surface. RISK: HAMT/LLRB have no `i` — you -must thread a `*count` through the two recursive walkers; the ADR's -"each loop gains a guarded `if (i >= n)`" is false for the 4 -unordered/sorted collections. Threading length through `printHamtEntries` -buys no parity-corpus value (hash collections are corpus-excluded for -order); only `printSortedEntries` is corpus-relevant. - -**Alt 2 — Finished-form-clean:** a single `PrintCtx` value struct -(`depth`, `length`, `level`, `readably`, `namespace_maps`) threaded -through every printer, replacing the free-standing threadlocals. -`printValue(w, v)` → `printValue(w, v, ctx)`; surfaces build one ctx -reading the Vars once; recursion passes `.{ .depth = ctx.depth + 1 }`. -BETTER: kills three module-global threadlocals; re-entrancy/unbalance -structurally impossible (nested `pr-str` gets fresh depth-0 ctx, matching -clj); reads the Var once not per-element; backend-agnostic (F-012). RISK: -signature change to `pub printValue` + every helper (large diff — but -F-002 says size is not a downgrade reason); changes the documented "pure -(w,v) renderer" contract; does not (correctly) merge the 7 loops. - -**Alt 3 — Wildcard:** push `*print-length*` UP into `deepRealize` so -realization stops at N+1 elements. BETTER: actually delivers the ADR's -headline ("readable result, not a hang") — ADR-as-drafted hangs on -`(iterate inc 0)` because deepRealize runs before the print guard; only -the compact `.range` works. Makes the corpus claim honest. RISK: wrong -layer (realization depending on a print Var); getting length+level+ -laziness jointly correct fights the realize-then-print split; touches the -lazy-seq engine (D-164 territory), higher regression risk. The finding is -the most valuable output; the mechanism is probably not the finished form -(that would be print-while-forcing with `rt` in a PrintCtx). - -**On Option C (merge loops): No.** `printRange` is alloc-free scalar -arithmetic; HAMT/LLRB are recursive `*bool`-threaded walks, not loops. A -uniform `printSequential(iter)` either heap-materialises an element list -(destroying alloc-free properties the comments call out as deliberate) or -is a tagged-union iterator whose `next()` is the same 5-arm switch plus a -new type. Option C is smallest-conceptual-surface bias dressed as -cleanliness; it reaches a leakier finished form. Reject — the ADR is -right. The win is commonising the guard expression (Alt 1) and the state -(Alt 2), not the iteration. - -**On the Var mechanism:** read-at-print-time via Var is mandatory and -correct (`print_readably`-style snapshot can't respond to `binding`). The -only refinement is to deref once at top-level entry rather than per -element. - -**On `>= level` vs `> level`:** the ADR states the rule correctly -(collection at depth `d` prints `#` iff `d >= level`; root = depth 0; -`*print-level* 0` → `#`). Trap: the check must read current depth before -incrementing, and the increment must wrap only the collection-helper -call, not scalar branches. Note `.map_entry` prints as `[k v]` inline in -`printValue` WITHOUT a collection helper — a map-entry IS a vector in clj -and counts as a level; the tag list must include `.map_entry` (and -tagged-literal `form`) or depth under-counts for `(seq {:a {:b {:c 1}}})`. - -**Recommendation:** Adopt Alt 2 (PrintCtx) extended to absorb the Alt 3 -finding (give PrintCtx `rt`/`env` to force-and-truncate lazy seqs -element-by-element). The ADR-as-drafted adds a fourth ambient threadlocal -with a re-entrancy hazard "defensive reset" only papers over, and reads -the Var per-element. If the `rt`-in-printer extension is too entangled -with D-164 to land in one cycle, land PrintCtx now and file the -lazy-seq-bounding as debt with the clj-hang pin — but do NOT ship the -per-element-threadlocal shape, and do NOT ship a `*print-length*` that -silently hangs on the exact infinite-seq case it was sold to fix. Fix -decision 3's tag list to include `.map_entry` regardless. - - - - -## Consequences - -- `*print-length*` / `*print-level*` become live; `binding` over them - truncates as clj does. Default (both nil) is byte-identical to today — - every existing print test is unaffected. -- The `print_depth` threadlocal + `printLength`/`printLevel` accessors - are the plumbing the deferred siblings (`*print-meta*` especially) can - reuse. -- `*print-length*` truncation order for unordered collections - (`hash-set`, `hash-map` > 8 keys) follows cljw's hash order, not clj's - — an extension of the existing AD-001. Such cases are kept OUT of the - parity corpus (deterministic-order cases only: vector / list / range / - array-map ≤ 8 keys / sorted). -- Deferred siblings tracked as a debt row (D-NNN) so the family is not - silently considered "done". - -## Affected files - -- `src/lang/clj/clojure/core.clj` — two `^:dynamic` defs. -- `src/runtime/print.zig` — cached Var pointers, `initPrintLimitVars`, - `printLength`/`printLevel`/`print_depth`, centralised level check in - `printValue`, length guard in each collection loop. -- `src/lang/bootstrap.zig` — resolve + cache the two vars post-load. -- `test/e2e/` + `test/diff/clj_corpus/print_control.txt` — coverage. -- `.dev/debt.yaml` — deferred-siblings row. diff --git a/.dev/decisions/0089_recut_concurrency_and_drift_methods.md b/.dev/decisions/0089_recut_concurrency_and_drift_methods.md deleted file mode 100644 index 5be4cab96..000000000 --- a/.dev/decisions/0089_recut_concurrency_and_drift_methods.md +++ /dev/null @@ -1,179 +0,0 @@ -# ADR-0089 — post-M re-cut (A→B→C), §7 concurrency drift + redesign-deferral, drift-detection method - -Status: Proposed → Accepted -Date: 2026-06-04 -Supersedes-in-part: §9.2.R sequencing wording (this ADR records the restore); -amends ROADMAP §7.1 / §7.2 / §9.2.S / §9.16-14.13.5. - -## Context - -User-directed re-cut (2026-06-04 chat). Two drivers: - -1. **The clean-bounded clj-parity frontier is drained.** This session ran the - F-010 post-M quality-elevation loop deep into `clojure.core` + print-control + - regex; a 40+ idiom broad probe came back all-green. The loop now lacks - bounded wins — every remaining `clojure.core` gap is deep / owner-gated / - long-tail. - -2. **Doc/debt drift accumulated where the user predicted** (audit - `private/audit-2026-06-04.md`): the ROADMAP re-cut sections (§9.2.R/§9.2.S), - the v0.1.0 row (14.13.5), and especially **§7 (concurrency)** describe a plan - the implementation has overtaken — AND that §7 predates Zig 0.16's stdlib - removals. - -The user's strategic correction: implementing the KNOWN-unimplemented CORE -(concurrency-led) must come **before** library-driven gap-hunting — "if those -aren't implemented, libraries fail en masse, and under progress-pressure they -get slapdash implementations." Do it plan-driven, finished-form, **rework-OK -with test guards** (F-002), aiming at the project identity: a lightweight, fast, -near-full-featured Clojure runtime reaching territory no non-JVM Clojure impl -except Babashka has — real STM / agents / threads in a Zig runtime. - -## Decision - -### 1. Post-M execution re-cut (restores §9.2.R Phase-15-first intent) - -``` -Phase A Consolidation — sweep doc/guard drift, align records to reality. - ↓ (doc/scaffold-only: no per-item test gate; batch-resolve) -Phase B KNOWN-unimplemented CORE, concurrency-led: §7 redesign + STM - ↓ transactions / agent / locking / real threading / Thread interop, - + Java arrays (F-004) / *out*·*in*·*err* IO vars (D-238) / - with-local-vars (D-237) / reflection. finished-form, rework-OK. -Phase C Library-driven gap-hunt (old "Phase B") — on the concurrency- - capable foundation; workaround remediation folds in here. -``` - -This is **not new** — §9.2.R already sequenced Phase 15 (concurrency) before the -quality loop. The session had drifted by running the quality loop pre-Phase-15; -this re-cut restores the intended order and names the consolidation pass (A) that -clears the deck first. - -### 2. §7 concurrency drift — corrected, redesign DEFERRED to Phase B entry - -Verified state (2026-06-04 probes): `future`/`promise`/`delay`/`pmap`/`pcalls` -all run **synchronously** today (`future.zig:10` "eager-inline"); the GC is -**single-threaded** (`arena.zig:14`). Concurrency primitives missing entirely: -`dosync`/`alter`/`commute`/`ensure`/`ref-set` (only the `ref` shell exists), -`agent`/`send`/`await`/`agent-error`, `locking`, `Thread` interop. - -Three §7 lies/staleness corrected in ROADMAP (this ADR is the rationale): - -- **§7.2 "STM is implemented"** — FALSE. `ref` shell only; the transaction - machinery is unbuilt (§7.2's own staging table assigns it to Phase 14/15). - Corrected to "designed (ADR-0010); transactions land in Phase B." -- **§7.1 Zig-mechanism column** references **removed Zig 0.16 APIs** - (`std.Thread.Pool` / `std.Thread.Mutex` / `Condition`) — per `zig_tips.md` - these are gone; the real impl must use `std.Io.*` / `std.atomic`. The §7 - mapping is a **pre-0.16, pre-current-architecture design** and must be - **redesigned at Phase B entry** (its own DA-fork + Structural-imagination per - §17/F-003), not patched here. The `runtime/binding_stack.zig` reference is dead - (the binding stack lives in `env.zig`). - -Phase B's concurrency design forces revisiting single-threaded assumptions -(GC thread-safety, atom/ref CAS atomicity, threadlocal-binding conveyance to -spawned threads — `bound-fn`/D-241, future/pmap real parallelism). This is the -F-002 rework territory; it is **deferred to Phase B entry with full ADR -discipline**, not decided here. - -### 3. Drift-detection method — FOLD into existing homes (no new mechanisms) - -The user added a method: exhaustively read **all code comments** to surface -"drift from finished-form" that doc/debt sweeps miss (comments encode -`stub` / `for now` / `single-threaded so fine` / `eager-inline` / `Phase N will` -/ stale-API-refs that were never promoted to `debt.yaml`). Per the mandatory -Devil's-advocate fork (reflected below), this and three sibling proposals are -**folded into existing mechanisms**, NOT added as new ones — adding a §G + a -skill + a rule + an F-013 would contradict the compress-guards goal that is -Phase A's whole point: - -- **comment-drift sweep** → widen `audit_scaffolding` **§E2.7** pattern list - (`stub` / `single-threaded so fine` / `eager-inline` / stale-API), and **run - it this cycle** (deferring to "a fresh session" is the Defer-to-amnesia smell; - grep precision is context-independent; `framework_completion.md` mandates the - sweep-introducing cycle run the sweep). The exhaustive per-comment - classification (the judgment-heavy half) runs as fan-out subagents (each - fresh-context by construction) as the first Phase-A work item → promotes - (b)/(c) findings to debt rows / Phase B inputs. -- **spike-before-structural** (throwaway Zig spikes, web-search + Zig-0.16-source - to validate a Clojure-equivalent design before the concurrency ADR) → one - sentence in `principle.md`'s Structural-imagination phase; NOT a new rule - (Step-0 survey + DA-fork + Structural-imagination + `exploration_vs_done.md` - freedom already cover it). -- **"user-observable parity, internals free"** → already **F-011 §2** verbatim + - `no_jvm_specific_assumption.md`. NO new F-013 (duplicate-law bloat; loop-minted - F-NNN also violates the append-only/user-declared rule). Cite F-011. - -## Alternatives considered (Devil's-advocate subagent, verbatim) - -Grounding read: project_facts.md (F-001..F-012, esp. F-002/F-009/F-011), -no_jvm_specific_assumption.md, principle.md Bad Smell catalogue -(Framework-incomplete, Defer-to-amnesia, Stale-phase-ref, Reservation-as-bias + -Structural-imagination), audit_scaffolding/CHECKS.md §E2.7, framework_completion.md, -ROADMAP §9.2.R. Confirmed the compress-guards tension is real: 31 rules / 40 -scripts / 89 ADRs already standing. - -**Choice 1 — comment-drift sweep as new §G: FOLD into E2.7.** E2.7 -("telltale-pattern provisional sweep") already greps `src/` + `test/e2e/` for -`for now|stands in for|until Phase N|temporarily|placeholder|TBD`; E2.6 owns the -`Phase N` half; no_jvm_specific_assumption owns the stale-API half. A new §G -duplicates E2.6+E2.7 (the B2 Duplicated-claims condition). The genuine delta -(`stub`/`single-threaded so fine`/`eager-inline`) is a one-line widening of -E2.7's `rg` alternation, not a new mechanism. (a) new skill = worst (a whole -skill for one grep line); (c) run-once = undersells (keep it periodic). Action: -widen E2.7 by ~4 tokens. - -**Choice 2 — build-now/run-later (fresh context): RUN NOW.** "fresh context = -more exhaustive" is the Defer-to-amnesia smell almost verbatim; a grep does not -get more exhaustive with empty context — precision is in the regex + classifier -discipline. Deferring couples the drift-finder to "next session," defeating the -Phase A whose point is to drain drift now. framework_completion's counter-example -is exactly "ship a matcher without running it against the current tree." Run it -in the Phase A cycle that widens E2.7. - -**Choice 3 — spike_before_structural.md rule: DROP rule, FOLD note.** Design -rigor for Phase B is already triple-covered (Step-0 survey incl. Zig stdlib, the -depth≥2 DA-fork, principle.md Structural-imagination). A rule asserting "spikes -are allowed" mostly grants a permission that already exists (exploration_vs_done -+ ~/.claude/CLAUDE.md "気軽に実験コードを作ってよい"). A new paths-scoped rule -would itself trigger framework_completion (discovery + sweep + retrofit) — a -disproportionate landing cost for near-zero new constraint. The one useful nugget -("validate the Zig-0.16 primitive in a spike before committing the ADR shape") -belongs in principle.md's Structural-imagination phase. Append a sentence; no -32nd rule. - -**Choice 4 — F-013 promotion: DROP, sharpen F-011 if anything.** Already F-011 §2 -nearly verbatim ("Behavioural equivalence is the correctness target; internals -are free … observable input→output incl. error cases must match real Clojure; -internal mechanics + representation may diverge freely"), with -no_jvm_specific_assumption as its operational arm. A new F-013 = the same -project-law in two F-NNN entries (B2 + anti-duplication), inviting drift + a -"which wins" decision. F-NNN entries are user-declared + append-only, amended -only by user direction; minting F-013 to restate an existing fact would be the -loop manufacturing project-law on its own — barred by the priority-order header. -Action: nothing, or a user-directed one-line sharpening of F-011 §2 — never a new -F-013. - -**Cross-cutting:** all four "add a mechanism" instincts fail the compress-guards -default for the same reason — each one's novel content is a 1-3 line widening of -an existing mechanism, and the existing audit/rule/F-NNN homes are already the -canonical owners. Phase A is better served by *tightening the four existing -homes* than by adding §G + a skill + a rule + an F-NNN. - -## Consequences - -- Phase A wiring is LEAN: ROADMAP alignment + E2.7 widen (+run) + a principle.md - sentence + handover + debt rows. No new rule, no new skill, no new F-NNN. -- Phase B is the project's largest + hardest unit (real concurrency in Zig); it - opens with a §7-redesign ADR + Structural-imagination + DA-fork at its entry. -- The §7.2 STM-lie and §7.1 stale-API are corrected in ROADMAP (drift removed). -- Per-item test gating is dropped for Phase A's doc/scaffold work (user-directed: - "こまめなテストはかえって遅くなる"). - -## Affected files - -- `.dev/ROADMAP.md` §9.2.R / §9.2.S / §7.1 / §7.2 / §9.16-14.13.5. -- `.claude/skills/audit_scaffolding/CHECKS.md` §E2.7 (pattern widen). -- `.dev/principle.md` (Structural-imagination: spike-the-primitive sentence). -- `.dev/handover.md` (resume = Phase A). -- `.dev/debt.yaml` (それ-tier rows; §7.2-lie tracking; discharged sweep). diff --git a/.dev/decisions/0090_phase_b_concurrency_redesign.md b/.dev/decisions/0090_phase_b_concurrency_redesign.md deleted file mode 100644 index 3babcad82..000000000 --- a/.dev/decisions/0090_phase_b_concurrency_redesign.md +++ /dev/null @@ -1,473 +0,0 @@ -# 0090 — Phase B concurrency architecture (real STM / agents / threads on pinned Zig 0.16) - -- **Status**: Proposed → Accepted -- **Date**: 2026-06-04 -- **Author**: autonomous loop (Phase B entry ceremony, D-242) -- **Tags**: concurrency, gc, stm, runtime, phase-b -- **Supersedes-in-part**: ROADMAP §7.1 (the pre-0.16 Clojure↔Zig primitive - mapping table) + §7.2 barge wording; corrects ADR-0089 §2's "spawn-based plan - is unbuildable" framing. - -## Context - -Phase B (ADR-0089 re-cut) implements the KNOWN-unimplemented concurrency CORE -that libraries depend on: STM transactions, `agent`, `locking`, real threading -(`future`/`promise`/`delay`/`pmap` run SYNCHRONOUSLY today — `future.zig` -eager-inline), `Thread` interop, GC thread-safety, and dynamic-binding -conveyance to spawned threads (D-241). The `ref`/`TVal` MVCC history ring is a -built SHELL with no transaction engine. This ADR is the Phase-B-entry -§7-redesign mandated by ADR-0089 + F-003 (Structural-imagination) with a -mandatory Devil's-advocate fork (principle.md depth ≥ 2). - -**Corrected premise (the load-bearing fact this session verified).** ADR-0089 §2 -and the ROADMAP §7.1 note state the `std.Thread.spawn` plan is "unbuildable" / -"APIs removed in Zig 0.16". This is **half-wrong**, and the correct half matters: - -- The **pinned Zig 0.16.0** (flake.nix → `/nix/store/…-zig-0.16.0`, what cljw - actually compiles against) **KEEPS** `std.Thread.spawn`/`join`/`getCpuCount` - and `std.atomic.Value`; it **REMOVED** `std.Thread.{Mutex,Condition,Semaphore, - Pool,WaitGroup,Futex}` and moved the sync primitives to **`std.Io`** - (`std.Io.Mutex.lockUncancelable(io)`, `Io.Condition`, `futexWaitTimeout`, - `Io.Threaded` thread pool, `io.async`/`concurrent`/`Future`). -- The add-dir clone `~/Documents/OSS/zig` is a **post-0.16 master** (Nov 2025) - that re-added `std.Thread.Mutex` — it is the WRONG tree for 0.16 API questions - (memory `zig016-pinned-vs-oss-checkout`). -- **cw v0 already ships** real STM (MVCC LockingTransaction), agents (action - queue + `thread_pool.zig`), futures (`FutureResult{Io.Mutex,Io.Condition}`), - and a global-`gc_mutex` GC against THIS exact pinned compiler — buildability is - proven, not speculative. -- A throwaway spike this session (`private/spike_concurrency_0.16.zig`, gitignored) - validated the three load-bearing assumptions on the current pinned compiler: - (1) a `std.mem.Allocator.VTable` callback CAN lock a `std.Io.Mutex` via a - process-wide `io_default` singleton (the answer to "vtable callbacks cannot - take an `io` arg" — `arena.zig:13-17`); (2) `std.Thread.spawn` works (4 mutator - threads × 1000 allocs serialized through one heap mutex); (3) `std.atomic.Value` - cmpxchg works. It also surfaced two API-currency facts: `GeneralPurposeAllocator` - was renamed (use `page_allocator`/`DebugAllocator`); `std.Io.Mutex` inits via - `.init` (not `.{}`). - -So the §7 redesign is NOT "the plan is impossible, start over" — it is "the sync -primitives moved to `std.Io`, reached via an `io_default` singleton; adopt cw -v0's proven pinned-0.16 approach, **re-derived cljw-clean** per `no_copy_from_v1` -(reference, not copy)." - -**The GC-safety hole the DA-fork caught (the most important finding).** The -naive "ONE global heap mutex, mark-sweep takes it stop-the-world" wording is a -*latent lie inherited from v0*: v0's `gc_mutex` serializes *allocation*, but v0 -does NOT actually suspend running mutators (its `ThreadRegistry` is a vestigial -counter that scans no thread's stack). A mutator between allocations — holding a -freshly-`msAlloc`'d untraced `[*]u8` on its native stack, or running -non-allocating code — keeps running while the sweep reclaims its allocation → -**use-after-free under real parallelism**. This is masked today only because -`future`/`pmap` run synchronously (D-242). Real `Thread.spawn` opens the hole. -Shipping it would violate F-011 (observable correctness). The redesign must -confront this, not paper over it. - -Grounding read: `.dev/project_facts.md` F-002 / F-004 / F-005 / **F-006** (GC = -single-generation mark-sweep + 3-layer allocator; generational/concurrent GC -explicitly deferred to ROADMAP §89.2) / **F-011 §2** (behavioural equivalence is -the target; internals free) / **F-012** (production backend = VM, tree_walk = -oracle; dual-backend parity); `private/notes/phaseB-entry-concurrency-survey.md` -(the Zig-0.16 availability matrix + v0's proven pattern + per-primitive -observable contracts); `.dev/debt.yaml` D-242 / D-241; `no_jvm_specific_assumption.md`; -`no_copy_from_v1.md`. - -## Decision - -cljw's Phase B concurrency is **OS-threads + a global-allocation-lock GC with a -root-publication handshake + MVCC STM**, re-derived cljw-clean from cw v0. The -north star is **user-observable parity, internals free** (F-011 §2): the -internals below need NOT mirror the JVM, only the observable input→output -(including error cases) must match real Clojure. - -1. **Threading substrate** — real OS threads via `std.Thread.spawn`. - `future`/`promise`/`pmap`/`pcalls` run on spawned threads; `delay` stays - lazy-once. A `runtime/concurrency/thread_pool.zig` (re-derived from v0) backs - `agent` send/send-off + bounded `pmap` fan-out. Sync primitives are - `std.Io.Mutex`/`Io.Condition` reached through a process-wide **`io_default` - singleton** (`runtime/io_default.zig`, re-derived) so call sites and - `Allocator.VTable` callbacks that have no `io` in hand can still lock. - `future`/`promise` block on deref via an `Io.Mutex`+`Io.Condition` result - cell; a thunk that throws is caught and re-raised at deref (the Value-carried - exception channel, D-115). - -2. **GC thread-safety — global allocation lock + root-publication handshake - (the mandatory DA-fork correction; Alternative 2 below).** Allocation - serializes on one global heap `Io.Mutex` via `io_default` (F-006-clean, - single-gen mark-sweep unchanged). Collection-safety is a first-class - **root-publication protocol**, NOT v0's vestigial counter: every thread that - can hold cw Values owns a `ThreadGcContext` (registered at `Thread.spawn`, - deregistered at join) publishing a pointer to its current `RootSet` (operand - stack + binding-frame chain head). Collection takes the global lock, then - walks the **union of all live `ThreadGcContext`s** to build the root set, - marks, sweeps. Correctness invariant (stated, not assumed): *a raw `[*]u8` - from the allocator is not a root until installed into a traced location; the - install window is collection-free because allocation and collection share the - lock.* Hence collection only happens at allocation points (lock held, roots - quiescent), and every other thread is either blocked in the allocator (its - published roots current + traced) or running non-allocating code (holding only - already-installed, hence-marked, Values). Mutation-under-sweep (transient - conj, atom CAS) is safe because the mutated structure is reachable from a - published root and was marked — **no write barrier, no generational machinery, - no per-step safepoint poll** (zero hot-loop tax; JIT-friendly since leaf - integer loops don't allocate and so never reach a collection point). This - stays strictly inside the F-006 envelope and answers the survey's open - question #4. - -3. **STM — MVCC `LockingTransaction`, retry-only.** `dosync`/`alter`/`commute`/ - `ensure`/`ref-set` over the existing per-ref `Io.Mutex` + `Ref`/`TVal` history - ring: read-point, in-transaction value cache, ordered lock acquisition by ref - identity (deadlock-free), commit-point stamping, bounded retry. **MVCC is - chosen for observable snapshot-semantics fidelity** under the F-010 - real-library quality loop (a Clojure programmer's "reads see a consistent - snapshot; writers retry" mental model, and library code that observes a - long-running read transaction proceeding against a snapshot), **NOT because - the ring already exists** (treating the built ring as the reason would be the - Reservation-as-bias smell, F-002 §4). The global-transaction-lock STM - (wildcard alternative below) is observably equivalent for the *guaranteed* - contract and is recorded as a rejected alternative. - -4. **Divergence — no barge (AD-013).** cljw STM is **retry-only**; it drops JVM's - "barge" (younger-transaction preemption for starvation control). Result - equivalence holds; only contention fairness/throughput differ. Recorded as - accepted divergence **AD-013** (not a bug). The global-lock STM would have - neither barge nor retry; MVCC-retry is the closer observable match. - -5. **agent / locking / atom / ref CAS / volatile** — `agent`: action queue + - the thread pool, send (fixed pool) vs send-off (unbounded), error-mode + - handler, `await` via a condition. `locking`: a heap-value lock via `Io.Mutex` - (ADR-0009 header `lock_state`), not a JVM monitor. `atom`/`ref` CAS: - `std.atomic.Value` cmpxchg with validators + watches (already half-landed, - D-157). `volatile!`: unsynchronized `@atomicLoad`/`@atomicStore`. - -6. **Dynamic-binding conveyance** — clone the threadlocal binding frame - (`env.zig current_frame`, the `binding-conveyor-fn` shape) at `Thread.spawn` - so `(future (binding [...] ...))` and `bound-fn` (D-241) convey to the worker. - `*out*`/`*err*`/`*in*` per-thread routing folds in here (D-238). - -7. **Dual-backend (F-012)** — every primitive works identically on tree_walk - (oracle) and VM (production). The `ThreadGcContext` publishes a backend-agnostic - root set (operand-stack-equivalent + binding-frame head), so the GC handshake - needs no per-backend hot-loop edit. STM/agent/future bodies just call `eval`, - which dispatches per backend. - -**File layout** (Structural-imagination, F-003): a new `src/runtime/concurrency/` -subtree — `io_default.zig`, `thread_pool.zig`, `lock_tx.zig` (the -LockingTransaction engine over the existing `stm/{ref,tval}.zig`), `gc_thread.zig` -(the `ThreadGcContext` registry + union root walk, co-owned with `gc/`). The -existing `runtime/{future,promise,delay,atom,volatile}.zig` gain real bodies; -`runtime/agent.zig` + `runtime/locking.zig` are new. This is rework-OK with test -guards (F-002): the synchronous shells are replaced, not patched. - -## Alternatives considered (Devil's-advocate fork, fresh context, 2026-06-04, verbatim) - -Briefed with F-002 / F-006 / F-011 §2 / F-012 / F-004 / F-005 as hard -constraints. Grounded in the Phase-B concurrency survey, `.dev/debt.yaml` -D-242/D-241, and a direct read of cw v0's `io_default.zig` / `gc.zig` -(msAlloc/gcCollect/collectIfNeeded/ThreadRegistry) / `stm.zig` / `thread_pool.zig`. - -### Leading finding: the proposed Candidate A contains a load-bearing mis-statement about the GC - -The proposed ADR §2 said "ONE global heap mutex … mark-sweep collection takes -the lock stop-the-world." This is **false against the very v0 design it claims to -re-derive**. v0's `gcCollect`/`collectIfNeeded` take `gc_mutex` for the duration -of `traceRoots + sweep`, which blocks *other threads trying to allocate* — but a -mutator thread NOT currently inside an allocator call keeps running concurrently -with the sweep. v0's `ThreadRegistry` (gc.zig:1228) is documented "for -stop-the-world GC" but only does `fetchAdd/Sub` on a counter; nothing suspends a -running mutator, nothing scans a running thread's native stack/registers. The -`RootSet` is supplied by whichever thread triggered collection — a *second* -mutator thread's in-flight Values (a freshly-`msAlloc`'d `[*]u8` in a Zig local, -an interpreter temporary on thread 2's stack) are invisible to `traceRoots`. The -sweep reclaims them → **use-after-free under real parallelism.** Masked today -only because `future`/`pmap` run synchronously (D-242). "Global heap mutex" makes -*allocation* thread-safe; it does NOT make *collection* safe with other running -mutators. The ADR MUST pick a real answer; inheriting v0's latent hole verbatim -is a Reservation-as-bias / Progress-pressure smell. - -### Alternative 1 — Smallest-diff: cooperative safepoint over v0's machinery - -Keep Candidate A verbatim but make the GC honest with a cooperative safepoint: -each worker polls a `gc_requested: atomic.Value(bool)` between eval steps and -parks on a condition; the collector sets the flag, waits until all others park -(v0's `ThreadRegistry` counter, now load-bearing), collects with each parked -thread's `RootSet` registered, clears + broadcasts. -- **Better**: closes the use-after-free hole with the least new code; reuses v0's - vestigial counter + condition idiom; fastest convergence to a correct green. -- **Breaks/costs**: requires a safepoint poll in EVERY unbounded non-allocating - loop (tree_walk loop AND VM dispatch AND long native primitives — F-012 - two-edit surface); a missed site = collection-hang (worse to debug than a - crash); a permanent per-step branch in the hottest loop (perf tax the future - JIT must fight). Smallest-diff answer to a structural question — F-002 says it - loses when finished forms differ. -- **F-NNN**: F-006 ✓ (still single-gen, safepoint is coordination not a new - algorithm); F-011 ✓ (correct); F-012 ⚠ (two in-sync hot-loop edits). - -### Alternative 2 — Finished-form-clean: root-publication handshake decoupled from the allocator lock (RECOMMENDED) - -Separate the two concerns v0 conflated. (1) Allocation stays behind the -`gc_mutex` global lock via `io_default` — fine, F-006-clean. (2) Collection -safety becomes a first-class root-publication handshake: each thread owns a -`ThreadGcContext` (registered at spawn, deregistered at join) carrying a pointer -to its current `RootSet` (operand-stack slice + binding-frame chain head + a -small pinned-across-collection guard list). Collection acquires the global lock, -walks the registry of ALL live contexts to build the union root set, marks, -sweeps. The airtight invariant: *a raw `[*]u8` from the allocator is not a root -until installed into a traced location; the window between `msAlloc` returning -and that install must not contain a collection point.* Because allocation and -collection share `gc_mutex`, that window is automatically collection-free for the -allocating thread — `msAlloc` returns under the lock, the caller installs, and -only a subsequent allocation (re-taking the lock) can trigger collection. So the -invariant holds with NO per-step safepoint poll: collection only happens at an -allocation point (lock held, roots quiescent), and other threads are either (i) -blocked in `msAlloc` (last-published root context current + traced) or (ii) -running pure non-allocating code (holding only already-installed/traced Values). -Subtle case: a thread in (ii) mutating a traced structure (transient conj, atom -CAS) while another sweeps — safe because the structure is reachable from the -published root and was marked (no write barrier needed; not generational — -F-006 ✓). The one unsafe pattern (fabricating a new Value on the native stack and -not installing it while another collects) cannot occur, because fabricating a -Value IS an allocation, which takes the lock, which excludes concurrent -collection. **That is the load-bearing argument; it is why this is correct -without safepoints.** -- **Better than A & Alt 1**: correct by construction with ZERO hot-loop tax (no - per-step poll, no missed-poll hang); states the GC-safety invariant explicitly - (survey open-question #4); cleanest base for the future JIT (leaf integer loops - don't allocate → no collection point → no safepoint instrumentation, which Alt - 1 would have to thread through JIT'd code). -- **Breaks/costs**: forces a `ThreadGcContext` registration discipline at every - `Thread.spawn`; `RootSet` becomes per-thread-published rather than - passed-by-the-triggering-thread (a real change to v0's `traceRoots(self, - roots)` → `traceRoots(self, registry)`); more upfront design than Alt 1. The - "mutation under sweep is safe because the object was marked" argument depends - on collection fully under the lock (no concurrent mark) — fine here, and it - closes the door on concurrent/incremental mark as a later optimization unless a - write barrier is added then. Acceptable: F-006 defers generational/concurrent - GC to §89.2, so the finished form FOR THE F-006 ENVELOPE is exactly this - stop-collection-at-the-lock design. Per F-002 the larger upfront design is the - right call — different (correct, tax-free) finished form than Alt 1. -- **F-NNN**: F-006 ✓✓ (single-gen, 3-layer alloc, no write barrier, no - generational machinery — handshake is coordination, fully in-envelope; cw-heap - separate from zwasm); F-011 ✓✓ (correct under real parallelism; invariant - stated); F-012 ✓✓ (root-publication context is backend-agnostic; no per-step - poll → no per-backend hot-loop edit, unlike Alt 1). - -This is the shape the DA picks: satisfies F-006 MORE cleanly than the proposed -"stop-the-world" wording (which is neither true nor needed), answers survey -open-question #4, and per F-002 its larger design cost is the right trade. The -per-ref MVCC STM, `io_default` singleton, and `FutureResult` machinery from -Candidate A are retained unchanged — Alt 2 differs from Candidate A ONLY in the -GC-collection-safety mechanism (§2), the one part of Candidate A that was a lie. - -### Alternative 3 — Wildcard: `std.Io` structured-concurrency executor + single global-transaction-lock STM - -(1) Concurrency substrate = `Io.Threaded` executor + `io.async`/`io.concurrent` -+ `Future.await` (survey Candidate B), not raw `Thread.spawn`; `future`/`pmap` -become `io.concurrent` tasks whose results land in a GC-heap `FutureResult` cell; -`future-cancel` maps to `Future.cancel` (which raw spawn can't offer cleanly). -(2) STM = drop per-ref MVCC; `dosync` takes one process-wide `Io.Mutex`, runs the -body, commits, unlocks — no read-point, no history ring, no retry. -- **Better**: dramatically less STM machinery (~30 lines vs the MVCC ring + - read/commit-point + retry + commute-replay); per F-011 §2 a global-lock STM is - observably equivalent for Clojure's *guaranteed* contract (atomic + isolated - bodies; commute reorderable; side effects may repeat) — a global lock trivially - gives atomicity+isolation, commute runs in-order, "side effects may repeat" is - vacuously satisfiable (never retries — even more predictable). Kills the - no-barge AD entirely (no barge AND no retry). `Future.cancel` gives real - `future-cancel`; the executor is swappable (`Io.Threaded`↔`Io.Evented`) which - matters for the F-001 zwasm-v2 async-I/O future. -- **Breaks/costs**: the global txn lock serializes ALL `dosync` program-wide — a - throughput cliff under heavy STM contention vs MVCC's per-ref concurrency (fine - for a runtime whose differentiator is not STM throughput, but a real - latency-under-contention difference — record AD if chosen). `Io.Future(R)` is - typed/scoped around a call while a Clojure `future` is an unbounded-lifetime GC - heap object deref'able arbitrarily later → bridging needs `io.concurrent` + a - manual GC-heap cell anyway, so part of the "use the blessed model" cleanliness - evaporates. And `Io.Threaded` workers still allocate cw Values → still need Alt - 2's handshake, so the wildcard does NOT dodge the leading GC finding. -- **F-NNN**: F-006 needs Alt 2's handshake regardless; F-011 ✓ (global-lock STM - observably equivalent — fewer divergences than MVCC's no-barge); F-012 ⚠ (the - async model must be backend-agnostic — holds since both call `eval`). -- **Verdict**: the global-transaction-lock STM is the strongest under-considered - idea (less code, more predictable, fewer divergences). The loop's attraction to - MVCC is *partly* Reservation-as-bias (the ring is already built). But the honest - tie-breaker is observable fidelity: MVCC reproduces JVM's read-point snapshot - model, and real library code under F-010 may observe MVCC-specific interleaving - (a long-running read transaction proceeding against a snapshot that a global - lock would serialize). So MVCC is the safer finished-form bet for observable - fidelity, and the built ring is a legitimate secondary bonus, not the reason. - The `Io.async` executor half is a reasonable later/scoped option for `pmap` - fan-out (survey Candidate C) but not for unbounded-lifetime `future`/`agent`. - -### DA synthesis / recommendation - -1. Adopt Candidate A's spine (proven on the pinned compiler, F-NNN-clean). -2. **Replace §2's "stop-the-world global heap mutex" with Alt 2's - root-publication handshake** — the one mandatory correction (the proposed - wording is false against v0 and would ship a use-after-free; F-011 violation). - Alt 2 stays fully in F-006, adds zero hot-loop tax, answers survey #4. -3. Keep MVCC STM (reject the global-lock wildcard) but for the right reason: - observable snapshot fidelity under F-010, NOT "the ring already exists" - (Reservation-as-bias). Record the global-lock STM as observably-equivalent for - the guaranteed contract but possibly divergent on interleaving. -4. Defer the `Io.async` executor to a later scoped use (pmap fan-out / Phase-16 - async I/O), not the Phase-B `future`/`agent` spine. - -The single highest-value finding: **the GC §2 "stop-the-world" claim is a latent -lie inherited from v0's vestigial `ThreadRegistry`; the ADR must not stamp it as -written.** The main loop adopted this correction (Decision §2). - -## Consequences - -- **ROADMAP §7.1** mapping table is superseded by Decision §1-7 (Mutex/Condition - → `std.Io` via `io_default`; `binding_stack.zig` → `env.zig`; phases → Phase B). - §7.1 + §7.2 edited to align (§17). -- **ADR-0089 §2** "spawn-based plan is unbuildable" is corrected: `std.Thread.spawn` - is KEPT in pinned 0.16; sync moved to `std.Io`; v0 + the spike prove buildability. -- **GC root-publication handshake** is the load-bearing new invariant; the GC - Phase-5 work gains a `ThreadGcContext` registry. Still single-gen mark-sweep - (F-006); concurrent/incremental mark stays deferred to §89.2. -- **AD-013** is RESERVED for the STM retry-only / no-barge divergence; the - `accepted_divergences.yaml` entry + its `pin` land **with the Phase B concurrent - STM test**, not now — `check_accepted_divergences.sh --gate` enforces that every - `pin` test path exists, so an AD cannot be pre-added before its locking test - does. (Reserving the number is a memo, not a contract — F-002.) The no-barge - decision itself is recorded here + in ROADMAP §7.2. -- **D-242** Phase-B scope is refined by this ADR's §1-7 sequencing; the GC-safety - handshake is a prerequisite for ANY real-threading primitive (it lands first). -- **Dual-backend (F-012)** preserved: backend-agnostic root publication, eval-based - bodies. -- The synchronous `future`/`promise`/`delay`/`pmap` shells are replaced (rework-OK - with test guards, F-002), not patched. - -## Affected files - -- `.dev/ROADMAP.md` §7.1 / §7.2 (align to this ADR). -- `.dev/accepted_divergences.yaml` (AD-013 STM no-barge — added WITH the Phase B - concurrent STM test that pins it, per the gate's pin-existence rule; not now). -- `.dev/debt.yaml` D-242 (reference this ADR as the Phase-B §7 design). -- Phase-B implementation (future commits, rework-OK + per-commit gate): - `src/runtime/concurrency/{io_default,thread_pool,lock_tx,gc_thread}.zig` (new), - `src/runtime/gc/*` (`ThreadGcContext` registry + union root walk), - `src/runtime/{future,promise,delay,atom,volatile}.zig` (real bodies), - `src/runtime/{agent,locking}.zig` (new), `src/runtime/stm/*` (transaction - engine), `src/runtime/env.zig` (binding-frame conveyance). - -## Revision history - -- **2026-06-04 — GC-safety gap surfaced at #3 implementation (D-244).** While - designing increment #3 (the `ThreadGcContext` root-publication handshake), the - root-set walker (`runtime/gc/root_set.zig`) was found to enumerate only - `ns_vars` / `current_frame` (dynamic-binding) / `macro_root_slot` / - `permanent_roots` — it does **NOT** root the **VM operand stack** - (`vm.zig` `stack: [OPERAND_STACK_MAX]Value`, a Zig local array in the run loop) - nor tree_walk's native-Zig-stack intermediates. This is safe TODAY only because - `collect()` runs exclusively at **quiescent explicit call points** (no - auto-collect; `alloc` never calls `collect`), where no operand stack is live. - **For Phase B real threads (#4), this breaks**: a worker thread mid-`eval` - holds live Values on its VM operand stack / native stack that are NOT a root - source, so a `collect()` triggered on another thread would sweep them → - use-after-free. Additionally, a non-allocating worker mutating its - dynamic-binding frame chain (`pushFrame`/`popFrame`) DURING another thread's - root walk is a read-during-write race on the chain. - **Consequence for the Decision**: Alternative 2's "no safepoint needed" - argument (install-window collection-free) covers *allocation* but is - **insufficient for a mid-`eval` worker** — its un-rooted operand/native-stack - Values are the gap. The GC-safety mechanism must be re-analysed: a **safepoint - (Alternative 1)** where collection runs only when every thread is quiescent - (operand stacks empty/saved), OR making the **VM operand stack a published - root** (per-`ThreadGcContext`) plus a rule that tree_walk workers (native-stack - intermediates, un-enumerable) do not run during collect. This re-analysis + - its own **DA-fork** is the FIRST step of increment #3, **before** the handshake - code lands (tracked: **D-244**). The §1-2 / §5-7 spine (threads, alloc-lock, - STM, agent, conveyance) is unaffected; only §2's collection-safety mechanism - (the Alt-1-vs-Alt-2 choice) is re-opened. Increments #1 (io_default) + #2 - (global alloc lock) are correct and land regardless (the alloc lock is needed - by every candidate mechanism). **DECIDED 2026-06-04 — see the D-244 decision - section below.** - -- **2026-06-04 — D-244 DECIDED: Alternative B (alloc-boundary safepoint).** A - Step-0 survey (`private/notes/D244-gc-safety-survey.md`) + a mandatory DA-fork - resolved the GC-safety mechanism. Survey verdict: **cw v0 has the same latent - bug** (its `ThreadRegistry` is a vestigial counter; "safepoint comes in 48.3" - never landed) — no working mechanism to re-derive; the global alloc lock - (increment #2) is the one correct, necessary-not-sufficient piece. **Decision = - Alternative B** (the DA's recommendation, correcting the originally-proposed - C2a on three points): - 1. **Worker eval runs on the VM backend only** (F-012; `build.zig` already - defaults to `.vm` since D-196's in-place discharge). `future`/`pmap`/`agent` - thunks are **force-compiled to a chunk and run via the VM `evalChunkErased` - vtable slot** regardless of the build's `-Dbackend` (so the tree_walk oracle - build can still exercise them single-threaded; tree_walk's un-enumerable - native intermediates never run on a worker). This decouples D-244 from any - backend-default question — Q2's "tree_walk-default fork" does NOT fire. - 2. **The safe point is the ALLOCATION BOUNDARY, not the bytecode back-edge.** - Q1 is real and non-empty: `op_vector_literal`/`op_map_literal`/`op_set_literal` - + `callMethodImpl`'s rest-list cons-wrap loop hold a fresh accumulator in a - Zig local *across the next allocation*, un-installed on any operand slot — a - back-edge poll (C2a) does NOT cover this intra-`stepOnce` window. A thread - parks at *its own* `alloc` entry (where its live Values are installed on - published frames); the collecting thread **self-guards its own in-flight - allocation partial** (a single, enumerable, self-scoped spot — NOT a - guard-list discipline at every fabrication site, which would be an F-011 - hazard). A **liveness-only** back-edge poll survives (re-scoped from - correctness to: park a non-allocating loop + quiesce the binding-frame chain - for Q3), costing the same single branch C2a would. - 3. **Publication is a per-thread CHAIN of operand-stack frames, not one stack.** - `vm.eval` recurses (`op_call → callFn → callMethodImpl → eval`), each - invocation a fresh `stack`/`locals` Zig-local array; a `ThreadGcContext` must - register each `eval` frame on entry / pop on return, plus the thread's - binding-frame head. `collect()` walks the union of all threads' frame chains - + binding-frame heads + ns_vars + macro_root_slot + permanent_roots. - Q3 (the `pushFrame`/`popFrame` race) is closed by the STW-at-safepoint: parked - workers don't mutate the chain during the walk; the collector reads each - thread's `current_frame` head with the park/broadcast as the acquire fence. - **No F-NNN amendment needed** — the F-006 envelope (coordination-point STW, - precise roots by owning the operand-stack representation, no write barrier) is - exactly expressive enough; B forecloses concurrent/incremental mark without a - write barrier, which F-006 already defers to §89.2. **Alternative C** - (arena-isolated data-parallel workers with copy-out) is REJECTED for the spine - (shared-mutable `atom`/`agent`/`ref` identity → F-011 divergence on - `(let [a (atom 0)] @(future (swap! a inc)) @a)`) but retained as a future - detected-pure `pmap` fan-out optimization (Phase 15+/zwasm async). The #3 - implementation (the safepoint machinery + per-frame `ThreadGcContext` chain + - force-VM worker thunks + the collecting-thread self-guard) now proceeds from - this decision; D-244's barrier is the implementation checklist. - -### Alternatives considered (D-244 decision, DA-fork fresh-context, verbatim) - -Briefed with F-006 (single-gen mark-sweep, no write barrier, generational/ -concurrent GC OUT → §89.2), F-011 §2, F-012 (VM = production since D-196 discharge; -tree_walk = single-threaded oracle). Grounded in `vm.zig` (`eval`/`stepOnce`), -`tree_walk.zig` `callMethodImpl`, `gc_heap.zig` (alloc lock landed), `root_set.zig`, -D-244/D-196. - -**Leading finding:** C2a's published-root model is structurally under-specified — -a worker holds a CHAIN of operand-stack frames (`vm.eval` recurses via -`op_call → callFn → callMethodImpl → eval`, each a fresh Zig-local `stack`/`locals`), -not one; every C2 variant must publish a per-thread *stack of frames*. No -finished-form-clean option requires violating an F-NNN. - -**Alternative A — smallest-diff: alloc-point-only collection, no poll (C2b), per-frame publication.** Collection fires only inside `alloc` (lock held); each `eval` frame registers on a per-thread list; no `gc_requested` flag/poll/parking. -- Better: zero hot-loop tax; reuses the landed lock; cleanest for non-allocating leaf loops; closest to ADR-0090 §2's original intent. -- Breaks: **Q1 is fatal.** `op_vector_literal`/`op_map_literal`/`op_set_literal` build via a `conj`/`assoc` loop holding the partial accumulator in a Zig local across the next alloc; `callMethodImpl`'s rest-list cons-wrap loop likewise. A collect triggered by the next iteration's alloc sweeps the un-published partial → UAF. Only fixable by a guard-list discipline at every fabrication site — uncheckable, an F-011 hazard. Q3's `op_pop_binding_frame` (non-allocating chain edit) also races a concurrent walk with no coordination point. -- F-NNN: F-006 ✓; F-011 ⚠ (correct only if guard discipline complete — silent UAF otherwise); F-012 ✓✓. - -**Alternative B — finished-form-clean: alloc-boundary safepoint + per-frame publication (RECOMMENDED).** The collection point is the allocation slow path made a true safe point: a thread crossing the threshold sets `gc_requested`, waits for every other thread to reach a safe point (alloc entry, or a liveness back-edge poll), walks the union of frame chains + binding heads, marks, sweeps, broadcasts. The safe point is *before* the fabricating allocation, so at it every live Value is installed on a published frame — Q1's window closed by construction; the collecting thread self-guards its own current partial. -- Better than C2a/A: closes Q1 correctly (C2a's back-edge poll misses the intra-`stepOnce` accumulator window — a worker 50 iterations into `(into [] (range 1e7))` holds an un-published partial between back-edges); no fragile per-site guard discipline (the obligation moves to "every unbounded non-allocating loop must poll" — a small enumerable set: VM dispatch + `recur`). -- Costs: **liveness** — a non-allocating loop never hits the alloc poll, so ONE back-edge liveness poll is still needed (rare-taken relaxed-atomic load + predicted-not-taken branch); a long native primitive bounds time-to-safepoint (HotSpot-like, acceptable). More upfront design (frame registry + handshake). -- F-NNN: F-006 ✓✓ (safepoint+publication is coordination; single-gen; no write barrier — mutation-under-sweep safe because fully STW-at-safepoint, never concurrent mark); F-011 ✓✓ (correct by construction); F-012 ✓✓ (one back-edge poll is VM-only; tree_walk never runs on a worker). - -**Alternative C — wildcard: single-threaded heap + arena-isolated data-parallel workers with copy-out.** Workers run against a per-worker arena (no shared-heap alloc, no lock, no collection); result copied/realized back on the owning thread. Shared GC heap stays literally single-threaded → `collect()` unchanged, zero GC changes. -- Better: GC byte-identical to single-threaded; real parallelism for pure computation; sidesteps tree_walk-native-stack entirely; friendly to zwasm-v2 linear-memory instances. -- Breaks (disqualifying for the spine): Clojure `future`/`agent` bodies share mutable references — `(let [a (atom 0)] @(future (swap! a inc)) @a)` must observe `1`, but an arena worker swaps its *copy*; deep-copy breaks `identical?`. Preserving these means reaching into the shared heap = back to needing B's safepoint. So C only works for a pure subset → F-011 violation for the general contract. Conflicts with §6 binding conveyance (conveyed frame references shared Vars). -- F-NNN: F-006 ✓✓✓ (GC unchanged); F-011 ✗ for the general contract (disqualifying); F-012 ✓. -- Verdict: rejected for the Phase-B spine; retained as a future detected-pure `pmap` fan-out optimization (Phase 15+/zwasm async). - -**Q1 (fresh-uninstalled-local window):** YES, real, non-empty — vector/map/set-literal accumulator loops + the rest-list cons-wrap loop hold a fresh partial in a Zig local across the next alloc, NOT on any `stack[0..sp]` slot. The back-edge poll does NOT close it (the window is between back-edges, inside `stepOnce`); the **allocation boundary** is the safe point; the collecting thread self-guards its own current partial (a single self-scoped spot, not a per-site discipline). - -**Q2 (enforce VM-only workers):** the survey's "tree_walk-default" premise is **STALE** — D-196 is discharged and `build.zig` already reads `orelse .vm` (line 37). Force-compile worker thunks to a chunk and run via the VM `evalChunkErased` slot (already wired) regardless of `-Dbackend`, so the invariant is runtime-enforced independent of the build flag; the oracle build can still exercise `future` single-threaded. The "tree_walk workers → C1/C3 fork" never fires. - -**Q3 (pushFrame/popFrame race):** closed under B's STW-at-safepoint (parked workers don't mutate the chain during the walk; `op_pop_binding_frame`'s unlink+destroy is the thread's own, post-resume, on a frame the collector already saw as unreachable). The collector reads each thread's `current_frame` head with the park/broadcast as the acquire fence. NOT closed under a pollless model (mid-`popFrame` torn read) — confirming D-244's "this alone may force a coordination point." - -**Recommendation: Alternative B**, force-VM workers (Q2), per-frame chain publication, alloc-boundary safe point + liveness-only back-edge poll, collecting-thread self-guard. C2a corrected on the three points above. No F-NNN amendment required. diff --git a/.dev/decisions/0091_operand_stack_root_thread_union.md b/.dev/decisions/0091_operand_stack_root_thread_union.md deleted file mode 100644 index bcb3715fd..000000000 --- a/.dev/decisions/0091_operand_stack_root_thread_union.md +++ /dev/null @@ -1,240 +0,0 @@ -# 0091 — Operand-stack GC root publication via a thread-roots union cursor (Phase B #3b-step1) - -- **Status**: Proposed → Accepted -- **Date**: 2026-06-05 -- **Author**: autonomous loop (Phase B #3b implementation, D-244) -- **Tags**: gc, concurrency, root-set, vm, phase-b -- **Supersedes-in-part**: ADR-0028 §5 root-source table rows 2 (`current_frame`) - + 7 (`macro_root_slot`) — subsumed into a single `thread_roots` union source - (ADR-0028 gains amendment 2 pointing here). -- **Implements**: ADR-0090 "D-244 decision" (Alt B) §3 — "Publication is a - per-thread CHAIN of operand-stack frames"; D-244 barrier #3b. - -## Context - -Phase B #3a landed the worker-thread GC-root registry (`ThreadGcContext` + -`thread_registry`) and a UNION walk: the `current_frame` and `macro_root_slot` -cursors each iterate a per-thread dimension (source index 0 = the collecting -thread's TLS, k>=1 = registered worker k-1). That dimension was FOLDED into the -two existing cursors because a worker's binding frames + macro slot are the -SAME KIND of root as the self thread's. - -#3b must root a genuinely NEW kind of state: the VM **operand stack**. `vm.eval` -(`src/eval/backend/vm.zig:79`) holds `stack: [256]Value` (only `stack[0..sp]` -valid — above `sp` is `undefined`) + `locals: []Value` (256, nil-initialized); -and it recurses (`op_call → treeWalkCall → callMethodImpl → evalChunkErased → -eval`), so each thread holds a CHAIN of operand-stack frames. Today these are -NOT rooted, which is safe ONLY because `collect()` is invoked exclusively from -tests at quiescent top-level points (grep-confirmed: zero live auto-collect; -`alloc` never calls `collect`). For #4 (real worker threads) a worker mid-`eval` -holds un-rooted operand Values → a concurrent collect sweeps them → UAF -(ADR-0090's leading finding). - -The structural question this ADR resolves: **how is the operand-stack root -wired into the source-ordered `RootIterator`?** A mandatory Devil's-advocate -fork (depth ≥ 2, touches the ADR-0028 §5 root-source reservation table) reviewed -three shapes; its output is reflected verbatim below. - -## Decision - -Adopt the DA's **Alternative 2**: extract the per-thread union as a first-class -`thread_roots` `RootSource`, with the operand stack as its third sub-walk. - -1. **`thread_roots` subsumes `current_frame` + `macro_root_slot`.** The - `RootSource` enum drops both per-thread E-sources and gains one - `thread_roots`; the enum count goes 10 → 9. One `threadContextAt(idx)` helper - (idx 0 = self TLS, k>=1 = `thread_registry[k-1]`, `.end` past the cap) - replaces the triplicated `frameSourceAt`/`macroSourceAt`/(would-be) - `evalFrameSourceAt` addressing helpers — the F-011 commonization that - shipping the operand stack as its own source (clean Option B / DA Alt 1) - would have triplicated. -2. **The `thread_roots` cursor walks thread-major**: for each thread index, it - walks that one thread's complete root contribution — binding-frame chain - (each frame's `BindingMap` values) → macro slot → eval-frame chain (each - `EvalFrame`'s `stack[0..sp.*]` + all `locals`) — then advances to the next - thread. This mirrors the #3b-step2 safepoint's mental model 1:1 ("for each - parked thread, mark its roots"), which a source-major scatter would not. -3. **Operand-stack publication**: `EvalFrame { stack, sp, locals, parent }` + - `threadlocal var eval_frame_head: ?*EvalFrame` are declared in `root_set.zig` - (Layer 0 — same home + downward-import pattern as `macro_root_slot`; the - fields are raw `Value` pointers, no VM-type leak into Layer 0). `vm.eval` - pushes its frame on entry + `defer`-pops on return. `ThreadGcContext` gains - `eval_frame_slot: *const ?*EvalFrame`. -4. **Runtime-inert today (like #3a)**: with no live auto-collect + single - thread, `collect()` runs only at quiescent points where `eval_frame_head == - null` (no `vm.eval` on the C stack) and the registry is empty → the - `thread_roots` walk yields exactly today's `current_frame`+`macro` roots. - The existing gate stays green by construction. -5. **Scope boundary**: this ADR is #3b-**step1** (publication infra, inert). The - alloc-boundary **safepoint** + liveness back-edge poll + collecting-thread - self-guard (#3b-step2) couple to #4 (force-VM worker thunks) and land later. - -**Alt 3 (full thread-driven iterator spine, retiring `RootSource` as the -iteration driver) is recorded as the Phase-15/JIT-era re-conception candidate**, -not adopted now — it spends a large extra diff to neutralize the "how many -sources" question that Alt 2 already neutralizes, and sacrifices the -iterator-as-checklist property (every source accounted-for in `next()`). - -This stays strictly inside F-006 (precise roots, STW-at-safepoint at #3b-step2, -no write barrier) and F-012 (backend-agnostic publication; tree_walk never runs -on a worker, so only VM workers publish an eval chain). - -## Alternatives considered (Devil's-advocate fork, fresh context, 2026-06-05, verbatim) - -Briefed with F-002 / F-003 / F-006 / F-011 / F-012 as hard constraints, grounded -in a direct read of `root_set.zig` (#3a registry + union walk), ADR-0028 §5, -ADR-0090's D-244 decision, `.dev/debt.yaml` D-244, and `vm.zig` `eval`. - -### Framing the actual question - -The narrow question is "Option A (fold into `current_frame` source #2) vs Option B (new 11th `RootSource`)". But the grounding makes a third axis visible that neither A nor B sit cleanly on: the just-landed `RootIterator` is **source-ordered with a per-thread sub-dimension grafted into two of its cursors** (`current_frame.src_idx`, `macro_root_slot.src_idx`). The operand-stack chain is a *third* per-thread source. So the real design tension is not "fold vs add an enum row" — it is **"how many places does the per-thread-union dimension want to live: 0, 2, or 3+ cursors, and is source-ordered iteration even the right spine once 3 of N sources are per-thread?"** I keep that axis in view throughout. - -One load-bearing structural fact, verified against `root_set.zig` and the D-244 #3a barrier: the `ThreadGcContext` today is `{ frame_slot, macro_slot }` — two pointers into a worker's TLS. #3b's plan adds `eval_frame_slot` (a third). The registry, the union-addressing helpers (`frameSourceAt`/`macroSourceAt`), and the `src_idx` cursor pattern are **already a three-fold-repeated mechanism after #3b** (frame, macro, eval-frame), each re-deriving "index 0 = self TLS, k>=1 = registry[k-1], .end terminates". That repetition is the F-011 (DRY/commonization) signal that dominates my analysis. - -### Alternative 1 — smallest-diff: new `.operand_stacks` source (Option B), no iterator re-shape - -The smallest *correct* diff is **not** Option A — it is Option B done minimally. This is the inversion the brief should hear first. - -**Shape.** Add `operand_stacks` as an 11th `RootSource` between `macro_root_slot` and `typed_instances` (or after `permanent_roots` — placement is free). Add an `OperandFrameCursor { src_idx, frame: ?*EvalFrame, slot_idx, primed }` to the `Cursor` union. Add `evalFrameSourceAt(idx)` mirroring `frameSourceAt`. Walk `stack[0..sp.*]` then all of `locals` per frame, then `frame.parent`, then advance `src_idx` to the next thread. Extend `ThreadGcContext` with `eval_frame_slot: *const ?*EvalFrame`. Declare `threadlocal var eval_frame_head` in `root_set.zig`. Amend the two tests (`"RootSource enum lists 10 sources"` → 11) and ADR-0028 §5 (add row 11, E-shape). - -**Why "smallest-diff" and not Option A:** Option A (fold into `current_frame`) looks smaller (no enum change, no ADR-0028 amendment) but it is smaller *in the wrong ledger*. It saves a doc edit and a test-constant bump while **conflating two semantically distinct root kinds inside one cursor's state machine** — `CurrentFrameCursor` would have to carry both `frame: ?*BindingFrame` (a Clojure dynamic-binding concept) and `eval_frame: ?*EvalFrame` (a VM-backend concept), interleaving their drains. That conflation is itself a future "split this back out" rework (see §future-horizon). The diff Option A saves is doc-shaped; the debt it creates is code-shaped. By F-002's own ordering (finished-form > avoiding-rework > diff-size), Option A trades a clean-shape loss for a doc-edit saving — the wrong direction. - -**Better than the others:** Each `RootSource` stays one root *kind*. Matches the existing precedent exactly — `current_frame=#2` and `macro_root_slot=#7` are *already* distinct sources for distinct per-thread execution roots; the operand stack is a third such root and gets the third number. A reader doing `rg operand_stacks src/` finds the whole walker. The `nextOperandStack` body is independently testable in isolation (mirrors the existing per-source test shape). It is the **shape most consistent with what #3a already built** for macro_root_slot: #3a did NOT fold macro into current_frame even though both are per-thread — it gave macro its own cursor + its own `macroSourceAt`. The operand stack is more distinct from binding-frames than macro is, so if macro earned its own source, the operand stack certainly does. - -**Breaks / costs:** -- Breaks the `test "RootSource enum lists 10 sources per ADR-0028 §5"` assertion (root_set.zig:388) — must bump to 11. -- Requires an ADR-0028 §5 amendment adding row 11 (E-shape, "VM operand-stack frame chain per live thread") + bumping "Live entry-point walkers (E): 4 → 5". This is exactly the amendment the #3a barrier's guidance (iv) said folding "avoids". -- Does **nothing** about the three-fold repetition of the union-addressing pattern. After this lands, `frameSourceAt` / `macroSourceAt` / `evalFrameSourceAt` are three near-identical functions, and three cursors each re-implement the `if (primed) src_idx += 1 else primed = true; switch (sourceAt(src_idx))` walk. F-011 left on the table. - -**F-NNN compliance:** -- **F-002** ✓ — finished-form-clean *for the source-ordered iterator as it stands*; the enum-count "10" is correctly treated as a memo, not a contract (the bump is right). -- **F-003** ⚠ — correct for #3b in isolation but does not discharge the structural-imagination horizon (the per-thread dimension is now in 3 cursors; Phase 15/JIT will add more thread-state roots). -- **F-006** ✓ — precise root, owns the operand-stack representation, no write barrier, STW-at-safepoint. In envelope. -- **F-011** ⚠ — leaves the 3-fold union-addressing duplication uncommonized; that is the exact "shared mechanism outranks effort" case F-011 names. -- **F-012** ✓ — `EvalFrame` is a VM concept but the published-root shape is backend-agnostic; tree_walk never runs on a worker so only VM publishes. Clean. - -### Alternative 2 — finished-form-clean: extract the per-thread union as a first-class `ThreadRootCursor`, operand stack is its third sub-walk (RECOMMENDED) - -This is the shape the F-011 + F-003 reading points at. It is bigger than Alt 1 and I recommend it anyway per F-002. - -**Diagnosis first.** After #3b, the iterator has **three** per-thread sources (current_frame, macro_root_slot, operand_stacks) and **two** non-per-thread sources (ns_vars, permanent_roots). The per-thread dimension is the dominant structure, but #3a expressed it as an ad-hoc `src_idx` field copy-pasted into each per-thread cursor, with a parallel `xSourceAt(idx)` helper per source. That is the duplication. The finished form names the per-thread union **once**. - -**Shape.** Introduce one cursor that, for a given thread-source-index `k` (0 = self, k>=1 = `thread_registry[k-1]`), walks that *one thread's complete root contribution*: its binding-frame chain, its macro slot, its eval-frame chain. The `RootSource` enum collapses its three per-thread rows into a single `.thread_roots` source whose cursor carries `{ thread_idx, sub: enum { binding_frames, macro_slot, eval_frames }, ...inner state... }`. `ns_vars` stays source #1, `thread_roots` is source #2 (subsuming old #2 and #7), `permanent_roots` stays last; the deferred tag-trace/documentary rows (#3-#9) stay as documentary enum members but the *iterator's* live-source count drops from "4 of 10" to "3 of N". One `threadContextAt(k)` helper returns a small struct `{ frame_head: ?*BindingFrame, macro: ?Value, eval_head: ?*EvalFrame }` for thread k (self reads TLS directly, k>=1 reads the registry) — replacing the three `xSourceAt` helpers with one. - -**Why this is the finished form (structural-imagination horizon).** The brief asks which shape ages best across #3b-step2 (safepoint), #4 (real workers), and the JIT. Walk it: -- **#3b-step2 (safepoint):** the collector walks "the union of all live threads' roots". With `.thread_roots`, that union is *literally one cursor iterating thread indices* — the safepoint's mental model ("for each parked thread, mark its roots") maps 1:1 onto the iterator. Under Alt 1, the same parked-thread's roots are smeared across three different sources visited at three different times in source-order; the collector still works, but the code no longer mirrors the safepoint's per-thread quiescence argument. -- **#4 (real workers):** every new per-thread root (the binding conveyance frame from ADR-0090 §6, `*out*`/`*err*` per-thread routing from D-238, a future thread-local transducer scratch) becomes a new `sub` arm on the *one* cursor + a new field on the *one* `threadContextAt` struct — not a new enum row + new cursor + new `xSourceAt` + new test-count bump each time. Alt 1's cost is paid once per future per-thread root; Alt 2's is paid once, total. -- **JIT (Phase 15+):** when JIT'd frames need to publish their live-Value spill slots, they publish *another sub-walk of the same thread context*. The thread-centric cursor is exactly where a JIT frame's root contribution slots in. A source-ordered iterator would need yet another top-level source. - -**Better than Alt 1 & Alt 3:** Commonizes the union-addressing pattern that #3b would otherwise triplicate (F-011 directly). Makes the "10 sources" → "N sources, 3 live, one of which is the per-thread union" the *honest* description of the post-Phase-B root set — which is what ADR-0028 §5 should say anyway. The per-thread walk becomes unit-testable as "given a thread context with {frames, macro, eval-chain}, the cursor yields all three" — one test covers the shape for self and workers identically. - -**Breaks / costs (be concrete):** -- **This is a rewrite of the just-landed #3a iterator**, four commits old (`6ca347ae` root-walker union, `cc6881b5` registry foundation). #3a folded the worker dimension into `current_frame`/`macro_root_slot` cursors; Alt 2 says that fold was the smallest-diff-correct move for #3a-in-isolation but the *wrong cursor topology* once a third per-thread source arrives. So Alt 2 unwinds `frameSourceAt`/`macroSourceAt` + the two `src_idx` cursors and re-expresses them under `.thread_roots`. Per F-002 this rework is a feature, not a cost — but it IS a rework of green, tested, pushed code, and the brief should see that plainly. -- Touches every per-source test (`current_frame walker`, `macro_root_slot walker`, `union walk`) — they move from asserting per-source to asserting per-thread-sub. The two #3a robustness tests (registry churn) are unaffected (registry API unchanged). -- ADR-0028 §5 amendment is *larger* than Alt 1's: it re-tables rows 2 and 7 into a single "thread-roots union (binding frames + macro slot + operand stack), per live thread" E-row, and re-states "Live entry-point walkers: 4 → 3 (one being a 3-part per-thread union)". This is a more invasive doc edit, but it makes §5 *more* truthful about the post-Phase-B shape. -- **Risk concentration:** the root walker is, per the D-244 barrier, "the most correctness-critical module." Collapsing three sources into one cursor concentrates more logic in one state machine. Mitigation: the sub-walk is a flat `switch (sub)` with the same drain-then-advance shape already proven in #3a; the per-thread test asserts all three sub-walks. But a reviewer should weigh that the #3a barrier explicitly counselled "proceed in the smallest tested steps" *because* this module is delicate — Alt 2 is the opposite of a small step. - -**F-NNN compliance:** -- **F-002** ✓✓ — cleanest finished form; treats "10 sources" and the #3a fold as memos (both reshaped). Larger diff accepted per F-002's explicit "diff size is not a constraint." -- **F-003** ✓✓ — directly discharges the structural-imagination horizon: the per-thread union is named once and every Phase 15-20 per-thread root extends it without new top-level structure. -- **F-006** ✓ — identical GC envelope to Alt 1 (precise, STW-at-safepoint, no write barrier); only the iterator topology differs. -- **F-011** ✓✓ — the headline win: one `threadContextAt` + one `.thread_roots` cursor replaces three `xSourceAt` + three cursors. This is the commonization F-011 ranks above effort. -- **F-012** ✓✓ — backend-agnostic by construction; the thread context carries the VM `eval_head` but the cursor shape is identical whether a thread has an eval chain or not (self/main thread at top-level has `eval_head` populated during eval, empty between). - -### Alternative 3 — wildcard: thread-centric outer iterator, source-ordered inner (full re-conception) - -Alt 2 collapses three sources into one cursor but keeps the source-ordered *spine* (ns_vars → thread_roots → permanent_roots). The wildcard goes further: **invert the iteration nesting entirely.** Make the outer loop "for each root-bearing entity" where entities are {the shared ns/permanent roots} ∪ {self thread} ∪ {each registered worker thread}, and each entity contributes its sub-roots. The iterator becomes `for thread in {self, workers}: yield thread.{binding_frames, macro, eval_frames}; then yield shared.{ns_vars, permanent_roots}`. - -**Shape.** A two-level iterator: outer `entity_idx` over `[shared, self, worker_0, worker_1, …]`; inner dispatch picks the sub-walk set for that entity (shared entity → ns_vars + permanent_roots; thread entity → the 3-part per-thread walk). The `RootSource` enum is *retired as the iteration driver* and survives only as documentary classification in ADR-0028 §5 (the E/T/D taxonomy is a *description* of root kinds, not the iterator's control variable). Effectively `RootSource`-as-enum-order stops being load-bearing. - -**Better:** Maximally mirrors the GC's actual mental model under Phase B — "mark the shared roots, then mark each live thread's roots" is *exactly* what STW-at-safepoint does. The safepoint collector's loop and the root iterator's loop become the same loop. Cleanest possible base for #4 and the JIT: a thread is the unit, full stop. Fully decouples the iterator from the "how many RootSources" question — the enum-count test stops existing because the count stops being meaningful (the iterator is thread-driven, not source-driven). - -**Breaks / costs (disqualifying-adjacent):** -- This discards the source-ordered `RootIterator` *and its contract* wholesale — every existing per-source test is rewritten, not just amended. The deferred-source early-return symmetry (the `.fn_closures, .lazy_seqs, … => advance()` arm that documents tag-trace/documentary sources *in the iterator*) loses its home; ADR-0028 §5's careful E/T/D shape taxonomy, which currently maps 1:1 onto enum members the iterator visits, becomes pure documentation with no code mirror. That mirror has review value (a reader sees all 10 sources accounted-for in `next()`); the wildcard trades it away. -- The `ns_vars` + `permanent_roots` "shared entity" is a slightly forced grouping — they are not per-thread and bundling them as "entity 0" is a touch artificial (they have nothing in common except "not per-thread"). Alt 2 keeps them as honest top-level sources, which reads better. -- Biggest single diff of the three; highest review surface on the most correctness-critical module; furthest from the #3a barrier's "smallest tested steps" counsel. -- **No additional F-006/F-011/F-012 win over Alt 2** that I can identify — it buys "the enum-count question disappears" but Alt 2 already neutralizes that question (the count becomes "3 live, one a union" which is fine). So the wildcard pays a large extra diff for a marginal conceptual-purity gain over Alt 2. - -**F-NNN compliance:** -- **F-002** ✓ — finished-form-clean, arguably the *most* conceptually pure; diff size correctly not a constraint. -- **F-003** ✓✓ — same horizon discharge as Alt 2, slightly more thorough (thread is the literal unit). -- **F-006** ✓ — same envelope. -- **F-011** ✓ — commonizes the union but at the cost of the source-enum mirror's documentary value; net DRY win is real but partially offset. -- **F-012** ✓✓ — backend-agnostic, thread-as-unit. -- **Verdict:** strongest conceptual match to the safepoint model, but it spends a large extra diff over Alt 2 to neutralize a question Alt 2 already neutralizes, and it sacrifices the iterator-as-checklist property (every RootSource visibly accounted for in `next()`) that has genuine review value on this module. Recommend *not* for #3b; **note it explicitly as the Phase-15/JIT-era re-conception candidate** — if/when JIT frames add a fourth and fifth per-thread root and the shared/per-thread split becomes the dominant axis, revisit the thread-driven spine then. - -### On the precedent-tension the brief flagged (is "fold, don't add a source" binding here?) - -This deserves a direct answer because it is the crux. The #3a barrier guidance (iv) said: *"FOLD it into the current_frame/macro_root_slot cursors rather than add an 11th RootSource … adding a source needs an ADR-0028 amendment — folding avoids it."* - -**That guidance does not bind the operand-stack question, and invoking it here would itself be the Reservation-as-bias smell.** Two independent reasons: - -1. **Different KIND.** #3a's fold was the *worker dimension of an existing root* — a worker's binding frames are the *same kind of root* as the self thread's binding frames, "just from another thread" (the brief's own words, and root_set.zig:155-163 confirms: source-index 0 = self, k>=1 = worker, *same FrameSource type*). Folding same-kind-different-thread into one cursor is correct: it is one root kind with a thread dimension. The operand stack is a *different root kind* (VM-backend execution state vs Clojure dynamic-binding state). #3a's precedent is "don't split one root kind across threads into N sources" — it says nothing about "don't give a genuinely new root kind its own source." Macro_root_slot proves the point: #3a gave macro its *own* source (#7, own cursor, own `macroSourceAt`) precisely because it is a different kind from binding-frames, *and then* folded the worker dimension into that source. The operand stack is the same situation as macro: new kind → own treatment, with the worker dimension folded into it. - -2. **The stated rationale was "avoid an ADR-0028 amendment."** That is a process-cost-avoidance rationale, not a finished-form rationale. F-002 §4 names obeying a reservation *because it is reserved / because amending costs effort* as the Reservation-as-bias smell, forbidden. "Folding avoids the amendment" is exactly "obey the 10-count because changing it is a hassle." The amendment is a memo edit; if the finished form wants 11 sources (Alt 1) or a re-tabled union (Alt 2/3), the amendment follows the code, per ROADMAP §17. The #3a barrier was right *for #3a* (the worker dimension genuinely is not a new source); carrying its "avoid the amendment" reflex into #3b is the bias F-002 forbids. - -So: the precedent is real but *scoped to same-kind-different-thread*. For the different-kind operand stack, it is silent at best and a bias-trap at worst. - -### Synthesis / recommendation - -**No finished-form-clean option requires violating any F-NNN.** All three alternatives sit inside F-006 (precise roots, STW-at-safepoint, no write barrier) and F-012 (backend-agnostic publication, VM-only workers). I state that explicitly as the brief requested. - -**Recommendation: Alternative 2** — extract the per-thread union as a first-class `.thread_roots` cursor with the operand stack as its third sub-walk, rewriting #3a's `frameSourceAt`/`macroSourceAt`/`src_idx`-cursor triplication into one `threadContextAt` + one cursor. - -The reasoning chain, F-002-first: -- **Option A (fold into current_frame) is rejected outright** — it conflates a Clojure dynamic-binding root with a VM operand-stack root inside one cursor, which is a different-kind conflation that a future reader/maintainer would have to split back out (the "fold creates future split-out rework" hazard the brief asked me to weigh — it lands on A, not on B/Alt2). Its only advantage is dodging a doc amendment, which is the Reservation-as-bias trap. -- **Between Alt 1 (clean Option B) and Alt 2 (the union extraction)**, F-011 is decisive. After #3b, the per-thread union-addressing pattern is *triplicated* (frame, macro, eval-frame, each with its own `xSourceAt` + `src_idx` cursor). Alt 1 ships that triplication; Alt 2 commonizes it. F-011 ranks shared-mechanism above effort, and F-002 forbids me from downgrading to Alt 1 on the grounds that Alt 2 reworks just-landed code. Per the Cycle-budget-defer smell, "Alt 1 is a smaller diff because it doesn't touch #3a" is *not* a valid reason to prefer it — and that is precisely the instinct I am required to override here. -- **Alt 3 (thread-driven spine) is the cleaner *conceptual* match** to the safepoint model, and I record it as the **Phase-15/JIT-era re-conception candidate**, but it is not the right call for #3b: it spends a large extra diff over Alt 2 to neutralize the "how many RootSources" question that Alt 2 already neutralizes, and it sacrifices the iterator-as-checklist property (every root source visibly accounted-for in `next()`'s switch, mirroring ADR-0028 §5's E/T/D taxonomy) that has real review value on the most correctness-critical module in the tree. - -**The one honest tension in recommending Alt 2:** the D-244 #3a barrier explicitly counsels "the GC root walker is the most correctness-critical module, so proceed in the smallest tested steps." Alt 2 is a rewrite of green pushed code, not a small step — it is in direct rhetorical tension with that counsel. I resolve it in Alt 2's favor because (a) F-002 outranks step-size (step-size, like diff-size, is not an F-NNN constraint), (b) the rewrite is mechanically guarded — the existing per-source tests + the #3a union-walk test + the churn-robustness tests pin the *behaviour*, so the topology change is refactor-under-green, exactly the F-002-endorsed move, and (c) #3a's own commit message called itself "Phase B #3a-step2" / "foundation" — it was always the bounded first step, and reshaping a foundation when the third pillar arrives is the structural-imagination call F-003 mandates, not a regression. The "smallest tested steps" counsel is satisfied by *how* Alt 2 lands (refactor under the existing green tests, then add the operand-stack sub-walk + its test), not by choosing the smaller-topology Alt 1. - -If the main loop's instinct after reading this is "Alt 1 is less risky on the critical module, take it" — that instinct is the Cycle-budget-defer / Smallest-diff bias the loop is required to override unless Alt 2 hits a real F-NNN block. It does not. Pick Alt 2. - -### Main-loop adoption note - -Adopted Alt 2 verbatim. The main loop's own instinct WAS the "Alt 1 is less -risky on the critical module" pull the DA names in its closing paragraph; the -DA correctly identifies the mitigation (refactor-under-green: the #3a per-source -+ union + churn tests pin behaviour, so the topology change is a guarded -refactor, not a leap). The decisive non-cycle-budget reason is the #3b-step2 -safepoint mapping (§Decision 2) + the F-011 addressing commonization (§Decision -1). To honour the barrier's "smallest tested steps" *within* Alt 2, the source -landing keeps the existing #3a tests green through the refactor and adds the -operand-stack sub-walk + its own tests in the same commit (one cohesive, -fully-tested unit — not an Alt-1 way-station, which would be an excessive -skeleton that step-1b rewrites). - -## Consequences - -- **ADR-0028 §5** gains amendment 2: rows 2 (`current_frame`) + 7 - (`macro_root_slot`) are subsumed into a single `thread_roots` E-row (binding - frames + macro slot + VM operand-stack chain, per live thread); live E - walkers 4 → 3; the `RootSource` enum count 10 → 9. -- **`root_set.zig`** gains `EvalFrame` + `threadlocal eval_frame_head` + - `ThreadGcContext.eval_frame_slot` + `threadContextAt` + the `thread_roots` - cursor; `frameSourceAt`/`macroSourceAt` + their two cursors are removed. -- **`vm.zig`** `eval` pushes/`defer`-pops its `EvalFrame` (a PERF-relevant - hot-path write — candidate for a `// PERF:` marker + O-NNN if it shows on a - Release `scripts/perf.sh` number; not marked speculatively). -- **Runtime-inert** today (no behaviour change; gate green by construction). - The safepoint that makes auto-collect / worker-collect actually fire is - #3b-step2 (couples to #4). -- **Alt 3** (thread-driven iterator spine) is the recorded Phase-15/JIT-era - re-conception candidate. - -## Affected files - -- `.dev/decisions/0028_mark_sweep_gc_three_layer_allocator.md` — amendment 2 - (§5 re-table, pointer here). -- `src/runtime/gc/root_set.zig` — the `thread_roots` union cursor + `EvalFrame` - publication infra + tests. -- `src/eval/backend/vm.zig` — `eval` EvalFrame push/pop. -- `.dev/debt.yaml` D-244 — #3b-step1 landed; #3b-step2 (safepoint) remains. - -## References - -- ADR-0090 ("D-244 decision" Alt B) — the parent decision this implements. -- ADR-0028 §5 — the root-source table this re-tables. -- `.dev/debt.yaml` D-244 — the #3 implementation checklist. -- `.dev/project_facts.md` F-002 / F-003 / F-006 / F-011 / F-012. -- `private/notes/phaseB-3b-operand-stack-publication.md` — the Step 0.6 re-lay. diff --git a/.dev/decisions/0092_heap_value_monitor_locking.md b/.dev/decisions/0092_heap_value_monitor_locking.md deleted file mode 100644 index 49f080bf5..000000000 --- a/.dev/decisions/0092_heap_value_monitor_locking.md +++ /dev/null @@ -1,177 +0,0 @@ -# ADR-0092 — Heap-value monitor for `locking` (header spinlock skeleton toward thin→fat inflation) - -- Status: Proposed → Accepted (2026-06-05) -- Phase: B (concurrency), task #6 -- Supersedes / Amends: realises the ADR-0009 `lock_state`-bit reservation -- Related: ADR-0090 (Phase B concurrency redesign), ADR-0091 (thread_roots - GC walk), F-004 (NaN-box), F-006 (mark-sweep, no write barrier), F-011 - (behavioural equivalence), F-012 (VM backend) - -## Context - -`(locking obj body...)` must give Clojure's monitor semantics: **mutual -exclusion** across threads, **reentrancy** (the same thread re-acquires -without deadlock), and release on normal-or-error exit. JVM clj maps it to -an object's intrinsic monitor (reentrant + blocking). cljw has no JVM -object monitor; ADR-0009 reserved the heap header's `lock_state` (2 bits of -the `gc_and_lock` u32, the other 30 being `gc_mark`) for exactly this. - -The design question: how to build a per-object monitor on an **arbitrary -heap value** under cljw's stop-the-world mark-sweep GC (F-006) where the -lock word shares a u32 with the GC mark. - -## Decision - -**Finished form = Option C (thin→fat inflation).** A `lock_state=1` thin -lock (a header CAS) for the uncontended fast path; on contention it -inflates to `lock_state=2` + an off-heap blocking monitor cell -(`Io.Mutex`/`Io.Condition`, owner-tid + recursion-count, allocated and -finalised like `FutureCell`, keyed in a side-table by the heap pointer — -stable because GC is non-moving, F-006). A contended waiter then BLOCKS -(OS-park) instead of spinning. This is clj's actual monitor shape and the -F-011-clean target (blocking-not-spinning is observable when a lock holder -itself blocks). - -**This ADR lands Option A — Option C's uncontended fast path — as a -genuine, non-throwaway skeleton** (`runtime/concurrency/object_monitor.zig`): - -1. **Header spinlock.** `enter` CASes `lock_state` 0→1 on the `gc_and_lock` - u32, **preserving the gc_mark bits** (CAS the whole u32, set only the - low 2 bits). `exit` CASes back to 0. -2. **Safepoint-polling spin (the #1 correctness rule).** The acquire spin - is non-allocating, so it **polls `safepoint.gc_requested` and `park()`s - every iteration**. Without this a concurrent stop-the-world collect's - `stopWorld` waits forever for a park that never comes — a guaranteed - deadlock against the GC, strictly worse than the contention spin. (The - gc_mark-write vs lock CAS cannot otherwise race: the collect is STW, so - no CAS is in flight during the mark window.) -3. **Reentrancy via a threadlocal held-set.** A fixed `[32]` per-thread - array of (obj, depth). A re-acquire of an already-held object bumps the - depth (no second CAS, no deadlock); the outermost `exit` clears the - header bit. The held-set is **not** a GC root source — the locked object - is already rooted via the `EvalFrame` operand-stack/locals walk - (ADR-0091) for the body's duration; the set only counts re-entries. -4. **Immediates error.** `(locking )` has no `HeapHeader` to - lock → a clean `locking requires an object with identity` error - (AD-014, derives_from F-004), where JVM clj locks the box. - -Surface: the `locking` macro expands to `(__locking obj (fn* [] body...))` -(mirrors `dosync`); the `__locking` primitive holds the monitor across the -body thunk with a `defer` release (try/finally semantics). - -Option A is a **clean subset** of Option C: the gc_mark-preserving header -CAS, the safepoint-poll discipline, and the reentrancy held-set carry -forward into Option C unchanged — C only ADDS the 1→2 inflation + the -off-heap cell + the side-table. So landing A shrinks the C rewrite (per the -project spirit's "each skeleton must shrink the final-form rewrite"), and A -itself ships a correct, reentrant, GC-safe, clj-result-equivalent -`locking`. The only gap vs C is blocking-vs-spinning under contention — a -resource divergence (identical committed results; the clj-diff oracle sees -no DIFF), tracked as **D-245**. - -## Alternatives considered (Devil's-advocate subagent, fresh context, verbatim) - -> ### Alt 1 (smallest-diff): Bare-header spinlock, NO reentrancy — better: -> minimal; the lock_state CAS + safepoint-poll spin is the smallest thing -> that gives mutual exclusion across threads, no threadlocal bookkeeping. -> breaks: NON-REENTRANT — `(locking x (locking x ...))` or any re-entrant -> code path (a method calling a `locking`-guarded helper on the same object) -> SELF-DEADLOCKS. clj guarantees reentrancy; this silently regresses it (a -> permanent-no-op-class lie: looks like it locks, deadlocks the same -> thread). Rejected — violates F-011 behavioural equivalence (reentrancy is -> user-observable). -> -> ### Alt 2 (finished-form-clean): Off-heap blocking monitor cell + thin -> header inflation (= Option C, JVM thin→fat) — better: this is the actual -> clj finished form. `lock_state=1` (thin) is Option A's fast uncontended -> CAS path; on contention it inflates to `lock_state=2` + an off-heap -> `Monitor{ Io.Mutex, Io.Condition, owner_tid, recursion_count }` cell -> (allocated/tracked exactly like `FutureCell` — off-heap infra alloc, tied -> to the heap object, freed by a finaliser, side-table `obj-ptr → *Monitor`). -> A contended waiter BLOCKS on the Io.Condition (OS-park) instead of -> spinning, so it does NOT burn a core and does NOT livelock when the holder -> itself blocks (the holder parked in `future`/`promise` `deref` still owns -> the monitor; the waiter sleeps and is woken on release). Reentrancy is the -> `owner_tid` + `recursion_count` in the cell — exactly clj's monitor. -> breaks: more machinery (a thread-safe side-table keyed by heap pointer + -> GC cleanup of dead monitor entries on sweep; `Io.Mutex.lock` needs `io` -> threaded through). The side-table must survive the locked object moving — -> cljw GC is non-moving mark-sweep (F-006), so the pointer key is stable; -> OK. This is the recommended finished form. -> -> ### Alt 3 (wildcard): Park-on-the-safepoint-condition spinlock (no separate -> monitor cell) — better: reuses the EXISTING `safepoint` park/condition -> machinery instead of allocating a per-object cell; a contended `locking` -> waiter registers interest and waits on a shared lock-wait condition (woken -> on any `monitor-exit`), so it blocks (not spins) with zero new heap -> allocation and zero side-table. breaks: thundering-herd (every release -> wakes ALL lock-waiters, who re-contend — O(waiters) wakeups per release); -> no per-object fairness/owner tracking without a side-table anyway, so -> reentrancy still needs a threadlocal held-set; and entangling the GC -> safepoint condition with user-level lock contention risks priority -> inversion against a pending collect (a lock-waiter parked on the GC -> condition could delay `stopWorld`). Interesting but the entanglement with -> the STW path is a correctness hazard. Rejected. -> -> ### Recommendation: Option C (= Alt 2), with Option A as a genuine -> non-throwaway skeleton — Option A's header `lock_state=1` thin-lock CAS -> (preserving gc_mark, polling the safepoint each spin) IS Option C's -> uncontended fast path verbatim; Option C only ADDS the `lock_state=1→2` -> inflation + off-heap blocking cell on contention. So Option A is a clean -> SUBSET of Option C, not a dead-end — landing A first shrinks the C rewrite. -> Per F-002 (finished-form wins, diff size is not a constraint) and F-011 -> (blocking, not spinning, is user-observable), the finished form is C; A is -> the endorsed first skeleton toward it. -> -> The single most important correctness risk in Option A the main loop MUST -> handle: the spin loop is non-allocating, so it MUST call `safepoint.park()` -> (or poll `safepoint.gc_requested` and park) on EVERY spin iteration — -> exactly like the VM back-edge poll. If it spins on the bare CAS without the -> safepoint poll, a concurrent collect's `stopWorld` blocks forever (the -> spinner never parks) → the whole runtime hangs. Secondary, also -> load-bearing: the CAS must `cmpxchg` the WHOLE u32 to preserve `gc_mark`, -> and `(locking )` must error since immediates have no header. -> Note the held object does NOT need pinning — it is already a GC root via -> the EvalFrame operand-stack/locals walk (ADR-0091) for the body's duration. -> -> ### F-NNN check: NONE violated. Alt 1 (non-reentrant) would VIOLATE F-011 -> (reentrancy is user-observable) — recorded as the leading rejected -> alternative. - -The main loop adopted the DA's exact recommendation: **Option A now** (with -the three load-bearing risks — safepoint-poll, whole-u32 CAS, immediate- -error — all handled), **Option C tracked as D-245.** Choosing A as the -first skeleton is the DA's prescribed A-then-C path, not a cycle-budget -defer: A is F-NNN-complete (correct results, reentrant, GC-safe), and the -only gap (spin-vs-block) is a resource divergence with no F-NNN requiring C -*now* (F-011 is result-parity; A's results match clj). - -## Consequences - -- `(locking obj body)` works: mutual exclusion (a concurrent non-atomic RMW - under the lock lands all 400 increments), reentrancy (no deadlock), body - closes over the lexical env, releases on error. -- A contended waiter spins (polling the safepoint) rather than parks → wastes - a core while waiting. Acceptable for short critical sections; the blocking - inflation is D-245. **Not** observable in committed results (no clj DIFF). -- `monitor-enter` / `monitor-exit` are not exposed as separate primitives - (clj special forms); `locking` is the whole surface. Add later if needed. -- The safepoint-poll in the spin is correct only once every cljw-running - thread is registered with the safepoint (so `park()` accounting is sound). - Today auto-collect is OFF, so `gc_requested` is never set during locking - and the poll is dormant; the main-thread/worker registration audit rides - the **#4a' hardening** (D-244) before auto-collect turns ON. - -## Affected files - -- `src/runtime/concurrency/object_monitor.zig` (new) — the impl. -- `src/lang/primitive/locking.zig` (new) — the `__locking` primitive. -- `src/lang/macro_transforms.zig` — `expandLocking` + the `locking` BOOTSTRAP - entry. -- `src/lang/primitive.zig` — register the locking primitive. -- `src/runtime/error/catalog.zig` — `locking_needs_object` / - `locking_nest_overflow` / `locking_form_incomplete`. -- `src/main.zig` — test-aggregator import for `object_monitor.zig`. -- `test/e2e/phase16_locking.sh` (new) + `test/run_all.sh` registration. -- `.dev/accepted_divergences.yaml` — AD-014 (immediate errors). -- `.dev/debt.yaml` — D-245 (Option C blocking-monitor inflation). diff --git a/.dev/decisions/0093_agent_serial_executor.md b/.dev/decisions/0093_agent_serial_executor.md deleted file mode 100644 index 7c35a56b8..000000000 --- a/.dev/decisions/0093_agent_serial_executor.md +++ /dev/null @@ -1,313 +0,0 @@ -# ADR-0093 — Agent: serial async executor (single-drainer handoff, leaf-lock queue) - -- Status: Proposed → Accepted (2026-06-05) -- Phase: B (concurrency), task #6 (first slice) -- Related: ADR-0090 (Phase B concurrency redesign), ADR-0091 (thread_roots GC - walk), ADR-0092 (heap-value monitor), F-004 (NaN-box), F-006 (mark-sweep STW, - no write barrier), F-011 (behavioural equivalence), F-012 (VM backend) - -## Context - -`(agent init)` holds state mutated only by ACTIONS dispatched with `(send a f & -args)` / `(send-off a f & args)`. Actions on one agent run ONE-AT-A-TIME in send -order; different agents run concurrently. `@agent` is a non-blocking state read; -`(await a)` blocks until the actions sent so far have run. JVM Clojure uses an -`AtomicReference` + two thread pools (send=fixed, send-off=cached); -the pool choice is a non-observable throughput property. - -The design question under cljw's stop-the-world mark-sweep GC (F-006): how to -build the serial-execution queue + worker so the queue is GC-rooted and no thread -can deadlock the collector. - -## Decision (first slice) - -**Agent heap type** (tag 33, reserved): `extern { header, state: Value, rt, env, -cell: *AgentCell }`. `state` is read (`deref`) / written (drainer) atomically -(acquire/release) — the drainer is the SOLE writer (one drainer per agent at a -time), `@agent` the reader, so a single-word atomic load/store suffices (no CAS; -reuses the atom fix pattern). - -**Off-heap `AgentCell` (gpa-allocated, finaliser-freed)**: `{ mutex: Io.Mutex, -draining: bool, actions: ArrayList(Value), head: usize }`. The pending-action -queue is this **off-heap gpa list**, NOT a Value PersistentQueue — see the -leaf-lock invariant below. - -**Single-drainer handoff (serial execution)**: the thread that transitions the -queue empty→non-empty sets `draining=true` under `cell.mutex` and spawns the sole -drainer; the drainer pops actions until empty, then clears `draining` and exits. -`draining` is checked/set under `cell.mutex`, so a `send` either sees -`draining==true` (the live drainer picks its action up) or `draining==false` (it -spawns a fresh drainer). **Invariant: the drainer's "queue-empty test → clear -`draining` → release mutex → return" is ONE critical section** — never split it, -or an action can be stranded with no live drainer. - -**`cell.mutex` is a LEAF lock (the load-bearing GC invariant)**: under the STW -collector, a thread BLOCKED on `cell.mutex` is not at a safepoint, so if a collect -armed while it blocked, `stopWorld` would wait for a park that never comes. So -`cell.mutex` is held ONLY across the gpa queue push/pop — which never allocates on -the GC heap and never parks — NEVER across `callFn`, a GC allocation, or a park. -The action vector `[f & args]` is built (GC-allocated) BEFORE the lock; the action -runs (callFn + setState) AFTER the unlock. **This is why the queue is an off-heap -gpa list and not a Value PersistentQueue: `conj` would GC-allocate under the -mutex, breaking the invariant** (a parked-mid-conj drainer holding `cell.mutex` -would hang `stopWorld` for any peer blocked on that mutex). - -**GC**: `traceGc` marks `state` + every queued action in `cell.actions[head..]`, so -the queue is a root source the collector walks (mutators parked during a collect → -the list is quiescent while traced). The drainer registers a `ThreadGcContext` -like `future.worker`, so its in-flight action mid-`callFn` is rooted by its -operand stack. The fabrication window (a fresh action vector held only as a Zig -local in `send` before it is queued) rides the #4a' `gc_self_guard` hardening — -dormant while auto-collect is OFF, as today. - -**`await`** (core.clj, no new primitive): sends a sentinel action that delivers a -`promise` to each agent (so agents drain concurrently), then blocks on each -promise. A promise IS cljw's cross-thread latch (clj uses a CountDownLatch), held -alive by the sentinel action in the agent's queue. The sentinel returns state -unchanged. This is clj's latch-action semantics — NOT a "wait until idle" poll, -which the DA showed can hang a program that terminates under clj. - -**Spawn-per-drain-episode, no pool**: send/send-off share one path; the two-pool -starvation avoidance is non-observable (F-011). A real bounded pool / `send-via` -is a later perf tier. - -**Deferred to later slices**: watches/validator on agents (+ the shared IRef -substrate extraction the DA recommends, decided WHEN that surface lands and a -second consumer is concrete — Alt 2); error modes (`:continue`/`:fail`, -`agent-error`, `restart-agent`, `set-error-handler!`, `set-error-mode!`); -`send-via`; STM `send`-deferral; `*agent*`; nested-send/`release-pending-sends`; -`shutdown-agents`; `await-for` timeout. `(agent init opts...)` and add-watch / -set-validator! on an agent raise a clear "not yet" error (transient stubs, not -silent drops). - -## Alternatives considered (Devil's-advocate subagent, fresh context, verbatim) - -> ### Alt 1 (smallest-diff): Lock-free MPSC action queue (AtomicReference head) -> instead of mutex-guarded queue Value — better: removes the cell.mutex from the -> enqueue/drain hot path entirely; the empty→non-empty handoff becomes a single -> CAS on the queue head (the `draining` flag is folded into a tagged head -> pointer), so send/send-off never block and there is no "drainer holds cell.mutex -> at a non-safepoint" window. The `await` cond is the only remaining mutex use. -> breaks: it abandons the draft's "queue is a traced Value field" simplicity — a -> lock-free linked list of actions is NOT a single persistent-collection Value the -> existing `traceGc` can mark in one call, so it forces a NEW GC root source (the -> off-heap node chain must be walked by the collector, or each node must itself be -> a heap Value with its own tag). That re-opens exactly the #4a' concern the draft -> closed "by construction," and an off-heap atomic node chain mutated by the -> drainer concurrently with the STW collector's read needs its own memory-ordering -> proof. Net: smaller hot-path diff, larger GC-rooting surface — a smallest-diff -> bias against F-002's finished form. -> -> ### Alt 2 (finished-form-clean): Generalized IRef/handoff substrate shared by -> atom + agent (+ future ref/STM) — better: clj implements agent, atom, ref, and -> var watches/validators over ONE `IRef`/`ARef` machinery; the draft re-implements -> watches+validator+meta as fresh Agent fields duplicating atom's. F-011 -> (commonization outranks effort) + F-009 (neutral impl home) point hard at -> extracting a shared `runtime/concurrency/iref.zig` (watch-notify, validator- -> check, meta) and a shared `runtime/concurrency/handoff.zig` (the single-drainer -> empty→non-empty episode spawner) that BOTH agent and any future serial-executor -> reuse. The single-drainer handoff is the agent's distinctive logic; isolating it -> as a tested substrate means `await`, error-mode (:continue/:fail), and -> restart-agent later slices bolt onto a proven core instead of re-deriving the -> race each time. This is the cleanest finished form and I recommend it per F-002 -> even though it is a bigger diff than the draft. breaks: nothing F-NNN. It -> enlarges THIS slice (extract + retrofit atom's watches/validator onto the shared -> iref), and the watch/validator extraction touches already-shipped atom code -> (ADR-0081) — a depth-2/3 surgery, not a green-field add. The risk is over- -> abstracting the handoff before the second consumer (STM `send` deferral, agent -> pools) exists to validate the shape; mitigate by extracting only what atom+agent -> demonstrably share NOW (watch/validator/meta) and leaving the handoff agent-local -> until a second serial-executor appears. -> -> ### Alt 3 (wildcard): Pool-less but drainer-reuse via a parked-worker handoff -> (no spawn-per-episode) — better: instead of `std.Thread.spawn` per drain-episode, -> keep ONE long-lived drainer thread per agent that parks on `cell.cond` when the -> queue empties (instead of exiting) and is signalled by `send`. Eliminates per- -> episode OS-thread spawn/teardown cost, and a persistently-registered -> `ThreadGcContext` means the collector's root walk shape is stable instead of -> churning register/unregister every episode (the very churn the D-244 stress test -> was added to harden). breaks: it converts the agent into a permanently-live -> thread + permanently-registered GC root context that never exits until -> `shutdown-agents` — so a program that creates many short-lived agents leaks one -> parked OS thread each (clj's pool bounds this; a per-agent permanent thread does -> not). It also makes `shutdown-agents` (a deferred slice) a CORRECTNESS -> prerequisite rather than a nicety, because without it the process cannot exit -> cleanly (parked threads block join). Higher infra cost for a throughput win that -> is non-observable (F-011) — wildcard, not recommended for the first slice. -> -> ### Handoff-race verdict: race-free, with one required ordering constraint the -> draft already states. [the stranding interleaving cannot occur because the -> drainer's `draining=false` write and a send's `draining` read are both under -> `cell.mutex`; if a send enqueues before the drainer's empty-check, the drainer -> sees non-empty and keeps draining; if after the drainer cleared draining+ -> unlocked, the send reads draining==false and spawns a fresh drainer. Required -> invariant: the empty-test→clear-flag→release-mutex→return is a single critical -> section.] -> -> ### GC-queue verdict: traced-Value-field is sufficient for MARKING (STW means no -> concurrent mutator during traceGc), but the draft has a latent LIVENESS bug that -> is decisive: a worker reaches a safepoint ONLY at the alloc-prologue park or the -> VM back-edge poll. A thread BLOCKED on `cell.mutex` is NOT at a safepoint, so if -> a second thread is blocked on cell.mutex when STW arms, stopWorld hangs forever. -> The fix the main loop MUST apply: never hold `cell.mutex` across a VM call or an -> allocation — pop under the lock, UNLOCK, then run callFn+setState, re-lock for -> the next. (This is why the queue is gpa-backed, not a Value PersistentQueue whose -> conj allocates under the mutex.) -> -> ### await verdict: wait-until-idle is an observable divergence (a program that -> terminates under clj can HANG under cljw if another thread keeps sending). Ship -> the latch action (a sentinel that fires a per-await condition / promise), not -> wait-until-idle. -> -> ### Recommendation: Alt 2 (shared IRef/handoff substrate) for the surface -> (watches/validator/meta), with the draft's mutex-guarded queue + single-drainer -> handoff as the agent-local core — and ship the latch-based `await`. The single -> most important correctness risk: `cell.mutex` must be a LEAF lock, never held -> across `callFn`, an allocation, or a park. -> -> ### F-NNN check: none violated. Alt 1's lock-free off-heap node chain brushes -> against F-006 (recorded as the reason it is the rejected smallest-diff option). -> Alt 2/3 are F-NNN-clean. The latch-`await` strengthens F-011. `@agent` reusing -> atom's single-word atomic load is torn-read-free. - -The main loop adopted: the draft core (mutex-guarded queue + single-drainer -handoff) **with the off-heap gpa queue** (the leaf-lock invariant the DA proved -load-bearing forces gpa-backed over a Value PersistentQueue, since conj would -alloc under the mutex — this is a refinement of the draft, not a rejection), the -**latch-based `await`** (DA's correction over wait-until-idle), and **defers -watches/validator + the Alt 2 IRef extraction to the watches/validator slice** -(per the DA's own caution against over-abstracting before the second consumer is -concrete). Deferring watches/validator (a separate surface) is feature slicing, -not a cycle-budget defer — the iref extraction lands WITH that surface. - -## Consequences - -- `(agent init)` + send/send-off + `@agent` + await work, serial-per-agent + - concurrent-across-agents, clj-result-equivalent: send 100 → 100; (+5 then *3) - → 15; send-off conj → (2 1); 4×100 concurrent sends to one agent → 400 (handoff - race-free, 30/30 ReleaseSafe); 8 agents each +50 → all 50. -- An action that throws leaves state unchanged and draining continues (clj - `:continue` mode). clj's default (no handler) is `:fail`; the configurable - error-mode + `agent-error`/`restart-agent` is a later slice — recorded so the - default-mode divergence is not a silent drop. -- A fire-and-forget agent is `gc.pin`ned while its drainer runs (unpinned on - drainer exit), so it is not swept mid-drain. -- Process exit with a still-running drainer is an orphan (same limitation as a - still-running future worker); `shutdown-agents` is a later slice. - -## Affected files - -- `src/runtime/agent.zig` (new) — the Agent type + cell + single-drainer engine. -- `src/lang/primitive/agent.zig` (new) — `agent` / `send` / `send-off`. -- `src/lang/clj/clojure/core.clj` — `await` (latch over send + promise). -- `src/lang/primitive/stm.zig` — `deref` dispatch for `.agent`. -- `src/runtime/runtime.zig` — register the agent GC hooks. -- `src/runtime/error/catalog.zig` — `agent_options_unsupported`. -- `src/main.zig` — test-aggregator import for `agent.zig`. -- `test/e2e/phase16_agent.sh` (new) + `test/run_all.sh` registration. - -## Revision history - -### am1 (2026-06-10) — `await` delivers AFTER `notifyWatches` (deliver-in-body race fix, D-368) - -**Symptom.** The Ubuntu x86_64 full gate failed `e2e_phase16_agent` -`agent_add_watch`: `got '[[0 1] [1 2]]', want '[[0 1] [1 2] [2 2]]'`. The clj -oracle (ground truth, F-011) reliably yields the 3 fires `[[0 1] [1 2] [2 2]]` -— the `[2 2]` is `await`'s own count-down action's no-op watch fire. Mac/JVM -usually win the race; Linux x86_64 lost it. - -**Root cause.** `await` was `(send a (fn* [s] (deliver p s) s))` — the sentinel -delivered its promise INSIDE the action body. `runAction` runs the body -(step "newstate = callFn"), THEN stores, THEN `notifyWatches`. So `(deliver p s)` -released the awaiting thread BEFORE the sentinel action's own watch fired; the -awaiter could read `@log` before the `[2 2]` `swap!` ran. The e2e comment -documented the intended CONTRACT ("await guarantees every fire ran, incl. its own -`[2 2]`, matching JVM ARef") — the implementation violated its own contract. A -real concurrency-correctness bug, not a flaky test (user-directed root-fix -2026-06-10). - -**Decision.** Move `await`'s promise delivery from the action BODY to a -drainer-side step AFTER `notifyWatches`. The agent queue element becomes an -`Action { body: Value, completion: Value }`: `body` is the `[f & args]` vector -(nil = a pure barrier with no state change), `completion` is an optional promise -the drainer delivers AFTER the action stores its state AND fires its watches. -`await` is now `(__agent-await a)` — a primitive that enqueues a nil-body barrier -(which still fires the clj-faithful `[s s]` no-op watch via `notifyWatches`) and -returns the completion promise; `await` maps it over its agents then -`(dorun (map deref …))`. Deterministic, watch-order-independent, clj-faithful. -The fix is the **minimal** form of the Devil's-advocate Alt 2 below — the typed -2-field `Action` (NOT the full tagged-union `.normal/.barrier` + 3-phase -`runAction` refactor + speculative generic continuation slot, which is -gold-plating for post-action hooks no current slice needs; F-003 / excessive- -skeleton smell). - -**Leaf-lock invariant (F-006) preserved**: the completion deliver runs in the -drainer AFTER the action's pop+unlock, outside `cell.mutex`; promise.zig's own -leaf lock does not nest with the agent's. **Behaviour preserved**: send_await, -serial_order (100), send_args, send_off, concurrent_sends (400/4-thread), -concurrent_agents (8), remove_watch (`[1 1]`), nested_send, multi-agent await — -all verified green; `agent_add_watch` now deterministically `[[0 1] [1 2] [2 2]]` -(30/30 on Mac). - -**Affected files (am1)**: `src/runtime/agent.zig` (`Action` struct; `send`→ -`enqueueAction`; new `sendAwait`; `runAction` body-or-barrier + post-notify -deliver; `traceGc` marks body+completion; `promise.zig` import) · -`src/lang/primitive/agent.zig` (`__agent-await` primitive + registration + -`promise.zig` import) · `src/lang/clj/clojure/core.clj` (`await` over -`__agent-await`). - -#### Alternatives considered (Devil's-advocate fork, fresh context, 2026-06-10) - -> The bug: `await`'s sentinel action delivers its promise from *inside* `callFn` -> (step 2 of `runAction`), so the awaiting thread can wake before step 4 -> (`notifyWatches`) fires the sentinel action's own `[s s]` watch. The fix must -> move the deliver-equivalent wakeup to *after* the sentinel action's -> `notifyWatches` completes, deterministically, matching clj's 3-fire output -> `[[0 1] [1 2] [2 2]]`. All three options preserve the `[2 2]` fire (F-011) and -> never move work under `cell.mutex` (F-006). None requires a vtable hop for the -> wakeup, since `promise_mod.deliver` is directly callable from the drainer. -> -> **Alt 1 — Smallest-diff: barrier action recognised by a sentinel marker, -> deliver in a post-`notifyWatches` hook.** The await action vector carries a -> recognisable barrier marker (distinguished first element); after `runAction` -> returns, the drainer checks "was this a barrier?" and delivers the promise -> riding as element 2. Better: minimal new surface, one post-action branch. Risks: -> a per-action head type-test in the hot serial loop; couples the queue element -> shape to "first elem is fn OR barrier-marker" (a mild representation smell). F-006 -> preserved (deliver post-unlock). Multi-agent await preserved. -> -> **Alt 2 — Finished-form-clean: typed `Action` queue with a post-notify -> continuation.** Refactor `runAction` into compute→store→notify and make the -> queue `ArrayList(Action)` with `.normal([f&args])` / `.barrier(promise,[f&args])` -> variants; the drainer runs the action's continuation after `notify` (for -> `.barrier`, `deliver(p,new)`). Better: the await ordering becomes a structural -> property, not a magic-head convention; no per-action head type-test; the natural -> home for future post-action coordination (validators, completion callbacks). The -> variant tag is the dispatch; `traceGc` marks both the body vector and the barrier -> promise by construction. Risks: larger diff (queue element type change ripples -> through `send`/`restart`/`traceGc`/`finaliseGc` + every `vector.nth(action,…)`); -> more GC care. Per F-002 the larger diff is NOT a downgrade reason — the typed -> queue is the cleaner contract. F-006 preserved (deliver is a post-notify, -> post-unlock continuation). Multi-agent await preserved. **DA recommendation.** -> -> **Alt 3 — Wildcard: per-agent monotonic completed-counter + condvar, no barrier -> action.** `await` snapshots the enqueue index and waits until `completed >= -> target`. **Violates F-011 (leading entry):** a pure read-side counter barrier -> produces only `[[0 1] [1 2]]` — it does NOT enqueue a real action, so the -> clj-faithful `[2 2]` count-down fire never happens. A hybrid (counter + a no-op -> action to fire `[2 2]`) is strictly worse than Alt 2 (two mechanisms for one -> behaviour). F-006 is technically satisfiable (the condvar wait parks at a -> safepoint), but F-011 disqualifies it. Recorded so the rejection is visible. -> -> **Recommendation (non-binding): Alt 2** — the finished-form-clean shape under -> F-002. Alt 1 is the fallback only on a real F-NNN block on the `Action` struct -> (none). Alt 3 is disqualified by F-011. - -**Main-loop choice**: Alt 2's principle (typed `Action` carrying the completion), -in its **minimal** form (a 2-field struct `{body, completion}` + a nil-body -barrier), NOT the full tagged-union + 3-phase-refactor + generic continuation -slot. Rationale: the structural cleanliness the DA recommends (completion bound to -the action, no magic-head type-test) is captured by the 2-field struct; the -generic post-action-coordination substrate (validators/callbacks) is speculative -for needs no current slice has (F-003 defer-to-owner + the excessive-skeleton -smell). This stays finished-form-clean for the await need without gold-plating. diff --git a/.dev/decisions/0094_primitive_reentrancy_rooting.md b/.dev/decisions/0094_primitive_reentrancy_rooting.md deleted file mode 100644 index 2e6b2f752..000000000 --- a/.dev/decisions/0094_primitive_reentrancy_rooting.md +++ /dev/null @@ -1,118 +0,0 @@ -# ADR-0094 — Reentrant Zig primitives root their accumulators via the EvalFrame chain - -- **Status**: Proposed → Accepted (2026-06-05) -- **Amends**: ADR-0091 (operand-stack root + thread union) — adds a new - *kind of producer* of `EvalFrame`s (Layer-2 primitives), not a new root - source. -- **Driven by**: D-251 (GC root walk vs a real runtime env), surfaced by the - D-250 torture mode. - -## Context - -cljw's precise mark-sweep GC (ADR-0028) walks per-thread roots via the -`EvalFrame` chain (ADR-0091): each `vm.eval` activation pushes a Zig-local -`EvalFrame { stack, sp, locals, parent }` onto a threadlocal head and -`defer`-pops it, so a collect at any safe point marks the live operand -Values `stack[0..sp]` + `locals`. The worker registry already exposes this -chain per thread via `eval_frame_slot`. - -The D-250 torture mode (force a collect at every VM back-edge poll) revealed -that **Zig primitives that hold GC Values in Zig locals while re-entering -`eval` are a use-after-free class**. `reduceFn` (higher_order.zig) is the -exemplar: it keeps the accumulator `acc` and seq cursor `cur` in Zig locals -while it (a) calls the reducing fn via `invokeCallable` → VM `eval` and (b) -forces lazy/chunked seq elements via `firstFn`/`nextFn`/`seqFn` → VM `eval`. -A collect at a nested back-edge poll sweeps `acc`/`cur` (on no operand stack, -no `locals` slot, no trace) → corruption (e.g. `(vec (map str (range 1 80)))` -crashes reading a garbage vector `shift`). This is the same class as the -single-slot `gc_self_guard` (one in-flight fabrication value), generalised to -primitives holding *multiple* live Values across reentrant eval. - -This affects the whole seq-realiser family: `reduce` (→ `vec`/`into`/ -`interpose`/`mapv`/`frequencies`/…), the `map`/`filter` realisers, the -chunk-builder, `apply`-over-lazy, `sort`, etc. — wherever a Layer-2 primitive -drives a loop that re-enters eval while holding an accumulator/source. - -## Decision - -A reentrant Zig primitive that holds GC Values across a call back into `eval` -**publishes those Values by pushing a short `root_set.EvalFrame` onto the -existing threadlocal chain** (the same mechanism `vm.eval` uses), writing the -live Values into a Zig-local `[N]Value` operand array and updating them before -each reentrant call. No new root source, no new `ThreadGcContext` slot — the -existing root walk + worker exposure cover it for free. - -`EvalFrame`'s meaning widens from "one `vm.eval` invocation's operand stack" -to "one operand-stack-resident root scope" (VM eval activation OR a reentrant -primitive's accumulator frame). - -First application: `reduceFn` pushes a 2-slot frame `[acc, cur]`, refreshed -before every `invokeCallable`/`firstFn`/`nextFn`/`seqFn`. The remaining -seq-realiser primitives adopt the same 4-line `EvalFrame`-open as a campaign -(D-251), driven to torture-green big-bang per the `clj_diff_sweep` Discipline 2 -("exhaustively-then-closed"). - -## Consequences - -- **Positive**: zero new rooting infrastructure; one mechanism (`EvalFrame`) - covers VM operands + primitive accumulators + workers uniformly (F-002 - finished-form, F-006 single non-moving heap walk). The pattern is a small, - copy-paste-safe 4-line open per reentrant site. `gc_self_guard` becomes a - candidate to fold into this shape later (one fewer special-case root source). -- **Negative / risk**: each reentrant site must remember to open a frame + - refresh slots before each reentrant call (a `defer`-pop lifetime contract on - a Zig-local array — get the refresh-before-call ordering wrong and the value - is briefly unrooted). The torture mode is the mechanical guard that catches a - missed site/refresh deterministically. Hot-path cost: a few threadlocal - writes per loop iteration (accepted; perf is Release-measured separately, - the algorithmic shape is unchanged). -- **Not chosen**: a parallel per-thread primitive-root stack (a second concept - doing what `EvalFrame` does); a conservative C-stack scan (contradicts the - precise mark-sweep ADR-0028 / F-006 and misses register-resident roots). - -## Alternatives considered (Devil's-advocate fork, fresh context, verbatim) - -> Recommendation: **Alternative B1** — reuse the existing `EvalFrame` chain; -> `reduceFn` opens a tiny operand-stack-resident frame holding `acc`/`cur` and -> `defer`-pops it, with a docstring/naming widening of `EvalFrame` from "one -> `vm.eval` invocation" to "one operand-stack-resident root scope." - -**Alternative A — smallest-diff: a per-thread primitive-root *chain* (not a -fixed array).** Replace the draft's fixed `[K]Value` with a caller-stack- -resident linked `PrimRootFrame { roots: []Value, parent }` mirroring -`EvalFrame`. Better than the draft: unbounded depth at zero heap alloc (a -fixed `K` is an arbitrary cap that *will* be wrong for nested reducers). -Breaks/risks: still a brand-new root source — a second concept doing what -`EvalFrame` already does. F-clean. - -**Alternative B — finished-form-clean: no new root source; publish onto the -EXISTING `EvalFrame`.** B1: `reduceFn` opens a 2-slot `EvalFrame` (`acc`, -`cur`) reusing `root_set.EvalFrame` verbatim — zero new types, walk code, or -worker slot (the `eval_frame_slot` already covers it). B2 (deeper): make -`reduce` VM-resident so `acc` lives in a `locals` slot — much larger surgery -that still must drive `firstFn`/`nextFn` reentrantly, so it doesn't escape the -shape. Better than draft: **zero new rooting infrastructure**; one mechanism -covers operands + accumulators + workers uniformly (F-002/F-006). Risk: B1 -slightly widens `EvalFrame`'s "one `vm.eval`" docstring (a doc/naming fix); -the 2-slot array has the same `defer`-pop lifetime discipline as A. F-clean, -and the *most* aligned with F-006/F-002. - -**Alternative C — wildcard: conservative C-stack scan for the reentrant -window.** Scan the native stack for bit-patterns that decode as in-heap -NaN-boxed pointers. Better: zero per-site wiring, zero per-iteration cost, -new primitives can't forget. Breaks/risks: **the leading would-violate -entry** — it grafts a *conservative* collector onto ADR-0028's *precise* -mark-sweep (which F-006 codifies): false-positive retention, no cheap heap -range-check on a GPA-backed non-moving heap (F-006), and register-resident -`acc` is *not on the stack* → missed roots → UAF worse than the bug. Cannot be -the primary mechanism; recorded as the wildcard, recommended against. - -**Recommendation rationale**: F-002 (finished-form) + F-006 (one non-moving -heap, one walk) favour reusing `EvalFrame` over a parallel root source. -Picking the smaller-diff draft/Alt A over the F-clean B1 on cycle/LOC grounds -would be the Cycle-budget defer smell — re-pick B1. Per-iteration cost is 2 -slot writes, identical across draft/A/B1; only the rejected Alt C avoids it. -Caveat carried into implementation: B1 must be applied as a **uniform -campaign** to every reentrant seq-realiser primitive (the `mapFn` collected -vector, filter/realiser loops, apply-over-lazy, sort) — a half-applied fix -leaves the same UAF class live in the unfixed primitives. diff --git a/.dev/decisions/0095_external_container_gc_rooting.md b/.dev/decisions/0095_external_container_gc_rooting.md deleted file mode 100644 index 5b379c059..000000000 --- a/.dev/decisions/0095_external_container_gc_rooting.md +++ /dev/null @@ -1,240 +0,0 @@ -# ADR-0095 — GC children of non-swept containers must be re-reached every collect - -- **Status**: Proposed → Accepted (2026-06-05); Alt D landed (2026-06-05, next commit) -- **Amends**: ADR-0028 (mark-sweep GC) + ADR-0091 (operand-stack root walk). - Sibling to ADR-0094 (reentrant-primitive accumulator rooting) — same - campaign (D-251), a *different* root-gap class. -- **Driven by**: D-251 (GC root walk vs a real runtime env), surfaced by the - D-250 torture mode. - -## Context - -cljw's precise mark-sweep GC (ADR-0028) sweeps objects on the `gc.allocations` -live list and clears the `HeapHeader.gc_and_lock.gc_mark` cycle bit on every -survivor at the end of each `sweep` — so each collect starts with every -GC-swept object un-marked, and `mark()`'s cycle-invariant short-circuit -(`if mark==1 return`) is per-collect. - -The D-250 torture mode (force a collect at every VM back-edge poll) exposed two -distinct ways a **GC-allocated child becomes reachable only through a container -that does NOT live on `gc.allocations`**, so the child is swept even though it -is logically live. ADR-0094 covers the third way (a Zig local held across -reentrant eval); this ADR covers the two container classes: - -### Class 1 — process-lifetime mark-waypoints (the stale-bit membrane) - -`Function` / `ProtocolDescriptor` / `ProtocolFn` / `TypeDescriptorRef` are -`gpa.create`'d and registered via `Runtime.trackHeap` (freed only at -`Runtime.deinit`); they are NOT on `gc.allocations`, so `sweep` never clears -their mark bit. The bit doubles as the mark-phase cycle/visited flag. After the -FIRST collect marks such an object, its bit stays 1 forever, so from the SECOND -collect on `mark()` short-circuits before calling the object's per-tag trace — -and any GC child reachable **only** through it is never re-marked → swept. - -`Function.closure_bindings` is the exemplar: `(vec (map inc (range 1 200)))` -under torture sweeps the source `range` (captured in the `map` lazy-seq thunk's -closure) on the collect AFTER the thunk fn was first marked — the thunk fn's -stale bit blocks `traceFunction` from re-marking its closure. Confirmed by -instrumenting: the same `lazy_seq` is marked each collect, its `thunk` is a -`fn_val` each collect, yet the closure-captured range is marked on collect *N* -and swept on collect *N+1* (the collect that first leaves the thunk fn's bit -already set). - -### Class 2 — bytecode constant pools (the un-rooted literal) - -A string / collection literal in source compiles to a `gc.alloc`'d Value stored -in a `BytecodeChunk.constants` slice. The slice itself lives in the run-lifetime -analyser arena (never swept), but the **literal it points at is on -`gc.allocations`** and is reachable ONLY through the pool until an `op_const` -loads it onto the operand stack. The constant pool was never a GC root, so a -collect firing BEFORE the load (the torture case) sweeps the still-unloaded -literal; `op_const` then pushes a dangling pointer. `"hello"` alone corrupts to -`"\xef\xbf\xbd..."` under torture; every string / collection literal is -affected (the dominant reason torture crashed on "every real program"). - -The prior `traceFunction` comment asserted constants "live in the arena, never -GC-swept" — false for the literal *values* (only the slice is arena-resident), -and the secondary "walking them reaches non-GC-shaped headers" risk is closed -because `Value.heapHeader()` already returns `null` for `var_ref`/`ns` (the only -heap-tagged Env-lifetime non-GC types), the same filter the operand-stack walk -uses. - -## Decision - -**Class 1** — `GcHeap` keeps a `persistent_marks` list of every `trackHeap`'d -object's header (registered in `Runtime.trackHeap`). `collect()` clears the mark -bit on each at mark-phase start, so a process-lifetime waypoint is re-traced -every cycle and its GC children are re-reached. The objects are still never -swept (not on `gc.allocations`); clearing is harmless — its sole purpose is to -let the per-tag trace run each collect. A truly-unreachable waypoint simply -isn't re-marked, and its children are correctly collectible (it can never be -invoked again). - -**Class 2** — root every live chunk's constant pool, two complementary paths: - -- **Executing chunks** — `root_set.EvalFrame` gains a `constants: []const Value` - field; every `vm.eval` activation publishes `chunk.constants` alongside its - `stack`/`locals`, so the whole pool is a root for the chunk's entire - execution (covers the non-`Function` top-level eval chunk too). The root walk - drains `stack[0..sp]` → `locals` → `constants`, all via `heapHeader()`. -- **Dormant `Function` chunks** — `traceFunction` marks each method / variadic - chunk's constants (via `heapHeader()`), so a reachable-but-not-executing fn's - literals are rooted through the fn itself (which Class 1 now re-traces each - cycle). - -Both classes share one principle: **a GC-allocated child reachable only through -a container that lives outside `gc.allocations` must be re-reached by the mark -phase on every collect** — by clearing the container's stale mark bit (Class 1) -or by making the container's pool a walked root (Class 2). Literals stay -GC-allocated (one allocation path, F-011 commonization) rather than forking a -parallel arena-allocation path per collection type. - -### Landed-now vs the finished form (Devil's-advocate re-framing) - -The DA fork (below, verbatim) re-framed this correctly and is **adopted**: the -"Class 1 / Class 2" labels are two *manifestations*, not two mechanisms. The -real axes are (i) **one total heap/non-GC membrane** — every container-walk -(executing pool, dormant pool, closure bindings, protocol caches) rides the -same "is this Value safe to `mark()`?" filter; and (ii) **one orthogonal -bit-hygiene clear** — never-swept waypoints get their visited-bit reset each -collect. (ii) is independent of (i) and does not fold into it. - -**Landed in this ADR's commit** (safe, torture-green on the executing-chunk + -closure surface): the bit-hygiene clear (`persistent_marks` + -`clearPersistentMarks`, axis ii), the executing-chunk constant root -(`EvalFrame.constants`), and the `closure_bindings` trace. - -**Deferred to the immediate next unit — the GC-rooting SSOT (DA Alt D, the -chosen finished form)**: the membrane today is an **allow-list of two known -offenders** (`Value.heapHeader()` skips only `.var_ref`/`.ns`). The constant -pool holds heap-tagged Values pointing at non-GC memory beyond those two — a raw -dormant-chunk-constant trace crashed reading `tag_trace_table[112]` (`0x70` = a -pointer low byte mis-read as a tag), so that trace is reverted. The finished -form is a single SSOT predicate `tag_ops.isGcManaged(tag)` (a comptime `[64]bool` -derived from the GC-allocator registration set, with a comptime assert that the -`tag_trace_table` / `tag_finaliser_table` / `isGcManaged` sets agree) that -`heapHeader()` consults instead of the hand-listed switch. That makes the -membrane **total** (the next non-GC heap tag is filtered by construction, not by -a `switch` arm someone forgets), un-blocks the dormant-chunk trace, and is the -shared root-cause handle for the remaining `filter`/`defprotocol` torture gaps. -It is scheduled as the next unit (also the seed of the user-requested -GC-rooting-site SSOT + `GC-ROOT:` marker discipline + future-GC-migration doc), -NOT deferred indefinitely — per F-002 its larger diff is not a reason to ship -the allow-list permanently. - -## Consequences - -- **Positive**: torture goes from crash-on-every-program to green on the whole - literal + closure-realiser surface (`"hello"`, `(apply str …)`, - `interpose`, `vec`/`into`/`mapv`/`frequencies`/`sort` over `map`-of-`range`, - keyword/string map access). One uniform principle, no per-collection - allocation fork, reuses the existing `EvalFrame` walk + per-tag trace - dispatch. Class 1's clear also makes `Function` closures correct under ANY - future auto-collect, not just torture. -- **Negative / cost**: Class 1 adds an O(num trackHeap'd objects) bit-clear per - collect (Functions accumulate over a process — a known separate leak, see - D-251 follow-up; the clear is cheap, a single AND per object). Class 2 adds - the constant pool to the per-frame walk + per-`Function`-trace marking (bounded - by live constants). Perf is Release-measured separately; the algorithmic shape - is unchanged. -- **Precondition**: `trackHeap`'d objects must have `HeapHeader` at offset 0 - (every production caller — `Function`/`ProtocolDescriptor`/`ProtocolFn`/ - `TypeDescriptorRef` — is an extern struct that satisfies it; the cast is - deferred to clear-time so a unit-test box that never collects is unaffected). -- **Not chosen**: arena-allocating bytecode literals (Class 2 alt) — would fork - a non-GC allocation path for every literal collection type (anti-DRY, F-011); - making `Function` GC-managed (Class 1 alt) — larger surgery (chunk/closure - lifetime + finaliser) that also fixes the Function leak, deferred as its own - D-row. - -## Alternatives considered (Devil's-advocate fork, fresh context, verbatim) - -> **Recommendation: Alternative D (the `isGcManaged(tag)` SSOT classifier) as the finished form, UNIFYING both classes' container-walk safety under one membrane — but it does NOT subsume the Class-1 stale-bit clear, which is a separate concern that survives unchanged.** The draft's 2-class *split* is the wrong framing: the two classes are not two mechanisms, they are one mechanism (re-reach-every-collect) applied through two container kinds, plus one *orthogonal* defect (the persistent stale-bit short-circuit) that the draft has correctly tangled into Class 1 but which is really its own thing. - -### Constraint check (no alternative below violates an F-NNN) - -F-004 (47-bit pointers span full VA → no cheap heap-range check), F-006 (precise, non-moving, single-gen, one root walk — generational/card-marking explicitly future-not-now), F-011 (one allocation path per type; no parallel arena fork), F-002 (finished-form wins; LOC is not a constraint) are all respected by A/B/D. The would-violate option (card-marking / generational, Alt C) is recorded as the **leading entry** below because it is the one a naive reading of "Class 1 is a write-barrier problem" reaches for. - -### Leading entry — Alt C (wildcard, WOULD-VIOLATE-F-006): card-marking / remembered-set for the non-swept containers - -Treat `Function`/`ProtocolDescriptor` and the constant pool as an old generation, card-mark writes into their child slots, and at collect only re-trace dirtied cards instead of clearing all `persistent_marks` + walking every pool. -- **Better than draft**: O(dirty cards) instead of O(all trackHeap'd objects) per collect; the only alternative that addresses the draft's own stated negative (Functions accumulate over a process → the bit-clear loop grows unboundedly). -- **Breaks / why rejected**: F-006 codifies single-generation, non-moving, *one* root walk and names generational/card-marking a FUTURE candidate, not now. A write barrier on every closure-slot / constant-pool store is exactly the generational machinery F-006 defers. **This cannot be the primary mechanism today.** Recorded as the wildcard; the draft's full-clear is the F-006-compliant shape and the unbounded-clear cost is a real but *separately-tracked* concern (the Function-leak D-row), not a reason to import a barrier now. - -### Alt A — smallest-diff: keep the draft exactly as written (clear-all `persistent_marks` + `EvalFrame.constants` + the reverted dormant-chunk trace) - -- **Better than draft**: nothing — it *is* the draft. Its merit is that it is already torture-green on the executing-chunk + closure surface and ships the least new surface. -- **Breaks / risks**: it leaves the `traceFunction` dormant-chunk-constant path **permanently reverted** with a hand-waved "needs a GC-managed-tag classifier someday" comment. That deferral is load-bearing: a reachable-but-not-executing `Function` (stored in a Var, captured in another closure, sitting in a collection) whose body holds a string/collection literal will sweep that literal under any future auto-collect — the exact Class-2 bug, just in the dormant half. The draft's own "Remaining D-251 gaps" §1/§2 (`filter`/`keep` chunk-builder; `defprotocol`/`defrecord` swept-to-nil) smell like the *same* missing classifier resurfacing, not three independent bugs. Shipping A means re-opening this file 2–3 more times. - -### Alt B — finished-form-partial: unify Class 1 + Class 2 under the existing membrane, no new classifier - -Observe that both classes are literally "a slice of `Value` reachable only through an off-heap container." Class 2's `EvalFrame.constants` already proves the pattern. So drop the per-container special-casing and make BOTH register the same way: the constant pool publishes through `EvalFrame.constants` (done), and `traceFunction` walks `closure_bindings` (done) — and the Class-1 *stale-bit clear* stays as the draft has it, because it is genuinely orthogonal (it is about the visited-bit, not about reachability). -- **Better than draft**: reframes the ADR honestly — there is ONE reachability mechanism (walk the container's `Value` slice as a root or via the owner's trace) plus ONE bit-hygiene fix (clear the persistent visited-bit). The draft's prose already says this ("Both classes share one principle") but then presents them as a 2-class taxonomy, which invites the reader to think there are two mechanisms to maintain. -- **Breaks / risks**: B still cannot trace **dormant** chunk constants, because the raw constant-pool walk crashes (`tag_trace_table[112]` OOB). So B is exactly the draft with better prose — it does not close the dormant gap. The crash is the tell that B is not the finished form: the membrane (`heapHeader()`) is *under-filtering*, and B leaves that defect in place. - -### Alt D — FINISHED-FORM-CLEAN (recommended): an `isGcManaged(tag)` SSOT classifier in `tag_ops`, making the membrane total - -Root-cause the crash. `Value.heapHeader()` filters only `.var_ref`/`.ns`, but a bytecode constant pool can hold a heap-*tagged* Value whose pointer targets **non-GcHeap memory** — an arena/interned symbol or keyword, an Env-lifetime pointer, anything `gpa.create`'d but not on `gc.allocations`. `heapHeader()` happily decodes it and hands `mark()` a `*HeapHeader` whose first byte is arbitrary → `tag_trace_table[112]` OOB. The `var_ref`/`ns` skip is an *ad-hoc enumeration of two known offenders of a general class*: "heap-tagged but not GC-managed." - -The finished form is a single SSOT predicate `tag_ops.isGcManaged(tag: HeapTag) bool` (a comptime `[64]bool` table, registered the same way `tag_trace_table` / `tag_finaliser_table` are — a tag is GcManaged iff it has a registered allocator on `gc.alloc`). `heapHeader()` consults it instead of the hand-listed `.var_ref, .ns` switch. Then: -- The membrane is **total**: any heap-tagged-but-non-GC Value (today `var_ref`/`ns`; tomorrow interned symbols/keywords, arena literals, Env pointers, Wasm externref) is filtered in ONE place, by construction, not by a growing `switch` arm that the next offender forgets to extend. -- `traceFunction` can now safely walk dormant-chunk constants (un-revert the deferred path) — the OOB is structurally impossible because non-GC constants are filtered before `mark()`. -- Class 2's executing/dormant split **collapses**: both go through the same total membrane; `EvalFrame.constants` and `traceFunction`'s constant walk become two callers of one safe walk, not two policies. -- The draft's "Remaining gaps" §1/§2 become directly attackable: a protocol-path Value swept to nil and a chunk-builder source swept are both "container holds a Value the membrane mis-decoded or the walk missed" — the same classifier is the tool. - -- **Better than draft**: turns the membrane from an allow-list-of-known-exceptions into a closed predicate; subsumes Class 2's two sub-paths; un-blocks the dormant-chunk trace the draft was forced to revert; gives §1/§2 a shared root-cause handle. This is the "GC-rooting SSOT" the draft's own comment promises — D builds it now instead of deferring it. Per F-002, the larger diff is not a reason to prefer A/B. -- **Breaks / risks**: - 1. **Correctness of the predicate is now load-bearing for memory safety.** If `isGcManaged` returns `true` for a tag that is sometimes non-GC (e.g. a tag reused for both an interned and a heap form), `mark()` crashes again. Mitigation: derive the table mechanically from the `registerGcHooks` set (a tag is GcManaged iff a `gc.alloc` allocator registered it), with a comptime/init-time assert that every tag with a `tag_trace_table` or `tag_finaliser_table` entry is also `isGcManaged` — the three tables must agree. - 2. **`heapHeader()` is hot** (every root-walk yield, every mark descent). A `[64]bool` indexed load is the same cost class as the existing two-arm switch; not a regression, but verify no extra branch in the GC-yield inner loop (Release-measured separately per the draft's perf note). - 3. **Class 1's stale-bit clear is still needed and still orthogonal** — D does NOT remove `clearPersistentMarks`. The classifier fixes *which Values are safe to mark*; the persistent-mark clear fixes *the visited-bit hygiene of never-swept objects*. Conflating them would be a mistake; D keeps them separate (so the ADR should stop presenting "Class 1 vs Class 2" and present "one total membrane + one bit-hygiene clear"). - 4. **Symbol/keyword interning status must be confirmed.** D's payoff assumes some constant-pool Values are heap-tagged-but-non-GC (the crash proves at least one such class exists). The implementer must enumerate which constant kinds those are (interned symbol? arena keyword? the `index 112` decode points at a `Value` whose low byte is `0x70`=112, i.e. a pointer's low byte, NOT a tag — confirming it is a non-GC pointer mis-read as a header). This enumeration is the first implementation step and feeds the predicate table. - -### Recommendation rationale - -F-002 (finished-form) + F-011 (one mechanism, no parallel paths) both point at **Alt D**: the crash is not an incidental bug to route around with a revert — it is the membrane telling you it is incomplete. The draft's `EvalFrame.constants` (Class 2a) is correct and should stay, but framing it as half of a 2-class taxonomy *hides* that the real finished form is "make the heap/non-heap membrane total, then every container-walk — executing pool, dormant pool, closure bindings, protocol caches — rides it safely." Choosing A/B (ship the draft, keep the revert) over D on the grounds that D is a larger diff would be the **Cycle-budget defer smell**: the dormant-chunk gap, §1, and §2 are the same under-filtered-membrane defect resurfacing, and they will cost more in repeated re-opens than building the classifier once. D is recommended even though it expands this cycle's diff (F-002). The one hard caveat carried into implementation: **D must NOT delete `clearPersistentMarks`** — the stale-bit hygiene is a genuinely separate axis and the classifier does not touch it. - -**One-line verdict**: The 2-class *split* is the wrong finished form — fold Class 2 into a single **total `isGcManaged(tag)` membrane** (Alt D) that subsumes executing + dormant constant pools + closure bindings under one safe container-walk (and incidentally un-blocks the reverted dormant trace + the §1/§2 gaps), while keeping Class 1's `persistent_marks` *stale-bit clear* as an orthogonal bit-hygiene fix that the classifier does not replace. - -### Main-loop disposition - -Alt D is **accepted as the finished-form direction** and scheduled as the immediate next unit (the GC-rooting SSOT). This commit lands the F-clean, torture-verified subset that is safe today — the orthogonal bit-hygiene clear (axis ii) + the executing-chunk constant root + the closure trace — because they are correct and complete on their own surface and gate-green now. Alt D's membrane-completion (the `isGcManaged` SSOT + un-reverting the dormant trace) is the next commit, not an indefinite deferral; sequencing two correct units is not the Cycle-budget-defer smell (which is picking a smaller *alternative* as the final answer). The DA's caveat is honored: `clearPersistentMarks` stays as a separate axis. - -## Revision — Alt D landed (2026-06-05) - -The DA's recommended finished form shipped in the commit after this ADR's -first. `heap_tag.zig` gained `isGcManaged(tag)` — the membrane SSOT — -classifying the heap-tagged-but-non-GC set as `{symbol, keyword, var_ref, ns}` -(`symbol`/`keyword` are `gpa`-interned via the keyword/symbol interners, never -swept; `var_ref`/`ns` are Env pointers). `Value.heapHeader()` now consults it -(`Tag`→`HeapTag` is 1:1 by integer for the heap slots) instead of the -hand-listed `var_ref`/`ns` switch — so EVERY root walk (operand stack, locals, -chunk constants, closure bindings) filters the same set in one place. The -`tag_trace_table[112]` crash is structurally gone (keyword/symbol constants are -filtered before decode), so `traceFunction`'s dormant-chunk-constant trace is -**un-reverted** — a reachable-but-not-executing fn's literals are now rooted -(`(defn g [x] (str "n=" x)) (mapv g …)` keeps `"n="` alive). A `Runtime.init` -consistency guard asserts every tag with a registered trace/finaliser is -`isGcManaged`, so a future `gc.alloc`'d type misclassified as non-GC fails fast -rather than being silently swept while live. phase16_gc_torture.sh +2 -(`dormant_lit`, `kw_const`). - -The DA caveats are honored: `clearPersistentMarks` (the orthogonal bit-hygiene -axis) is untouched; the membrane is now the single classifier, not an -allow-list. The broader user-requested GC-rooting-site SSOT + `GC-ROOT:` marker -discipline + future-GC-migration doc remains the next unit (this is its first -SSOT piece). The remaining gaps below are now isolated to a different class. - -## Remaining D-251 torture gaps (not this ADR) - -Two torture failures remain AFTER this ADR, root-caused as distinct classes for -follow-up cycles (recorded in D-251): - -1. **`filter`/`keep`/`remove` chunk-builder over a chunked source**, when the - result is consumed by another lazy op (`(map inc (filter even? (range …)))`) - or the CLI auto-printer — `even?` receives a swept garbage float; the source - range / chunk_buffer is swept. Eager VM-driven consumers - (`doall`/`count`/`reduce`/`vec`/`println`) are clean. The structural - difference from the now-clean `map` realiser is the conditional - `(when (pred v) (chunk-append b v))` re-entering eval mid-chunk-build. -2. **`defprotocol`/`defrecord` under torture** — `__make-protocol-fn!: expected - protocol, got nil` (a protocol-path Value swept to nil). diff --git a/.dev/decisions/0096_set_bang_thread_bound_gate.md b/.dev/decisions/0096_set_bang_thread_bound_gate.md deleted file mode 100644 index 2bb07f736..000000000 --- a/.dev/decisions/0096_set_bang_thread_bound_gate.md +++ /dev/null @@ -1,145 +0,0 @@ -# ADR-0096 — `set!` is a runtime thread-bound gate + a clojure.main-style baseline binding frame - -- **Status**: Proposed → Accepted (2026-06-05) -- **Amends**: the `set!` special-form analysis (ADR-0036 dual-backend - contract applies — the runtime gate lands in both backends). Partially - discharges **D-241** (no clojure.main baseline binding frame) for the - dynamic vars that already exist in cljw. -- **Driven by**: D-254 (`set!`-on-dynamic-var diverges from JVM in 3 ways), - surfaced resuming the add-watch IRef campaign. - -## Context - -Oracle-confirmed JVM `Var.set(val)` rule: `(set! v val)` succeeds **iff `v` -is thread-bound on the current thread**; otherwise it throws -`"Can't change/establish root binding of: with set"`. There is **no -compile-time dynamic check** — a non-dynamic var simply can never be -thread-bound, so it hits the same throw. `set!` NEVER mutates a var's root. -`clojure.main` establishes ~21 baseline thread bindings around eval (incl. -`*warn-on-reflection*`, `*unchecked-math*`, the print vars), so -`(set! *warn-on-reflection* true)` works at top level while -`(set! user-dynamic-var v)` at top level throws. `(count (get-thread-bindings))` -is 21 at the JVM top level vs **0 in cljw**. - -cljw diverged three ways (D-254): - -1. **analyze-time fragility (genuinely broken).** `analyzeSetBang` - (`special_forms.zig`) checked `flags.dynamic` at *analyze* time, but the - flag is only set at *eval* time (`tree_walk.zig` / `vm.zig` def arms). A - same-compilation-unit def+set! therefore falsely errored: - `(do (def ^:dynamic z 0) (binding [z 1] (set! z 9)))` → cljw "Can't set! - non-dynamic var: z" but JVM = 9. -2. **silent root-write.** `evalSet` / `op_set_var` did - `if (!setBinding) setRoot(val)` — a top-level set! on an *unbound* dynamic - var silently mutated the root (JVM throws). This same root-write is also - the *only* reason `(set! *warn-on-reflection* true)` "worked" at top level - despite cljw having no baseline frame. -3. **message text** (cljw-native phrasing; lower concern, retained). - -## Decision - -Adopt the JVM model in two coupled parts (Alt B of the Devil's-advocate -fork): - -**Part 1 — `set!` is a runtime thread-bound gate.** Remove the analyze-time -dynamic check entirely (the analyzer only resolves the target Var). Both -backends (`evalSet`, `op_set_var`) become: - -``` -if (!setBinding(var, val)) return raise(.var_set_not_bound, …); // never setRoot -``` - -`setBinding` walks the whole frame chain, so a bound var (at any depth) is -updated; an unbound var — dynamic-and-unbound OR non-dynamic — uniformly -raises, mirroring JVM's single predicate. The now-unused -`set_target_not_dynamic` catalog Code is removed; the pre-existing -`var_set_not_bound` ("Can't set! var that is not thread-bound: X") is the -single message, consistent with `var-set`. - -**Part 2 — a clojure.main-style baseline binding frame.** At bootstrap -completion (`setupCore` / `setupCoreAot`, after the `.clj` load), push one -process-lifetime `BindingFrame` (`user_pushed = false`, so -`pop-thread-bindings` cannot pop it and a stray pop correctly raises -unmatched) binding the cljw-existing members of clojure.main's baseline set, -each to its current root: `*warn-on-reflection*`, `*unchecked-math*`, -`*print-meta*`, `*print-length*`, `*print-level*`, `*print-namespace-maps*`, -`*data-readers*`, `*default-data-reader-fn*`. The binding is transparent -(deref returns the same root value) but makes `(set! *warn-on-reflection* -true)` at top level succeed for the *correct* reason (the var is genuinely -thread-bound), exactly as in JVM. The frame lives in the bootstrap arena -(freed wholesale at teardown — no per-entry free, no leak). - -`*ns*` (its own materialized-view machinery, ADR-0085) and `*out*`/`*in*`/ -`*err*` (D-238, not yet first-class) are deliberately excluded. Standard -vars that do not yet exist in cljw (`*assert*`, `*math-context*`, -`*command-line-args*`, …) are **not fabricated** — creating a var whose -behaviour is unimplemented would write a parity cheque the runtime cannot -cash (the DA-flagged risk). They land with their features; D-241 stays open -for them. - -## Alternatives considered - -(Devil's-advocate subagent, fresh context, F-002/F-009/F-011 envelope — -reflected verbatim in spirit.) - -- **Alt A — smallest-diff: drop the analyze check, runtime throws on - unbound.** One runtime arm covers {non-dynamic} ∪ {dynamic-unbound} by - reusing the existing thread-bound catalog entry. *Better*: least new - machinery, message already matches JVM. *Breaks*: without a baseline - frame it regresses `(set! *warn-on-reflection* true)` from success→throw - — itself an F-011 violation in the other direction. So Alt A is only - F-011-clean once it bundles at least a minimal frame, at which point it - has converged on Alt B. -- **Alt B — finished-form-clean: full runtime gate + baseline frame.** The - chosen shape. *Better*: the only option with zero residual divergence and - zero AD on the set! axis; `set!` becomes derivable from the same - thread-bound predicate JVM uses, and `get-thread-bindings`/`with-bindings` - become correct downstream. DA recommended it, citing F-002 (pull D-241 - forward rather than ship the smaller-diff Alt A/C — the set!-parity work is - exactly what makes the baseline frame load-bearing; "LOW value now" on - D-241 was a diff-budget judgement, not an F-NNN block). *Cost*: larger - diff; the DA flagged that fabricating the ~15 missing standard vars would - introduce behavioural divergences for vars whose semantics aren't wired — - mitigated here by binding only the existing subset and NOT fabricating. -- **Alt C — wildcard: keep a dynamic check but move it to eval-time + AD the - root-write.** *Better*: smallest correct fix for divergence #1 alone; - keeps the friendlier "non-dynamic" message. *Breaks*: the AD it needs - ("top-level set! succeeds where JVM throws") cannot be written honestly — - there is no F-NNN invariant that justifies set!-succeeds-where-JVM-throws, - and the accepted_divergences discipline forbids convenience-only ADs. It - also keeps two code paths where JVM has one (dual-backend drift hazard, - F-009). Collapses into Alt B once the bad AD is refused. - -DA recommendation: Alt B (non-binding). Main loop concurs — per F-002 the -finished form wins and the cycle-budget instinct toward Alt A/C is the -Cycle-budget-defer smell. - -## Affected files - -- `src/eval/analyzer/special_forms.zig` — drop the analyze-time dynamic check. -- `src/eval/backend/tree_walk.zig` (`evalSet`) — raise on unbound. -- `src/eval/backend/vm.zig` (`op_set_var`) — raise on unbound. -- `src/runtime/error/catalog.zig` — remove the unused `set_target_not_dynamic` - Code + entry. -- `src/lang/bootstrap.zig` — `installBaselineBindings` at `setupCore` / - `setupCoreAot` completion. -- `src/eval/node.zig` — `SetNode` docstring (no longer "sets root when - unbound"). -- `test/e2e/phase15_set_bang.sh` — `root-set`/`not-dyn` cases move to the - JVM-parity behaviour; the same-unit def+set! regression case added. -- `src/lang/diff_test.zig` — a dual-backend case for `binding`+`set!`. - -## Consequences - -- `(set! v val)` on an unbound var (dynamic or not) raises uniformly — - trust-positive (a user bug is no longer silently absorbed by a root - mutation). -- The same-compilation-unit def+set! false error is gone (the analyze-time - race is removed). -- `(set! *warn-on-reflection* true)` and the other 7 baseline vars work at - top level for the correct reason; `(get-thread-bindings)` at top level now - returns the 8-var baseline (vs the empty map before) — a step toward D-241 - parity (full 21-var parity waits on the missing vars landing). -- Residual: a top-level `(set! *some-missing-standard-var* v)` still differs - from JVM only because the var does not exist yet — a var-coverage gap - (D-241), not a set! semantics gap. diff --git a/.dev/decisions/0097_with_local_vars_anon_var_lifetime.md b/.dev/decisions/0097_with_local_vars_anon_var_lifetime.md deleted file mode 100644 index 61246ddee..000000000 --- a/.dev/decisions/0097_with_local_vars_anon_var_lifetime.md +++ /dev/null @@ -1,114 +0,0 @@ -# ADR-0097 — `with-local-vars`: anonymous local Vars are gpa-owned, never freed (escape-safe), reclamation deferred - -- **Status**: Proposed → Accepted (2026-06-06) -- **Driven by**: D-237 (`with-local-vars` needs anonymous-Var creation + a heap - binding-frame primitive). var-get/var-set + push/pop-thread-bindings already - landed; the missing pieces are the anonymous Var + the macro. -- **Relates to**: F-006 (Vars gpa-owned, GC-membrane-filtered), F-011 - (behavioural equivalence), AD-015 (escaped-local-var print/deref divergence). - -## Context - -JVM `with-local-vars` (core.clj) lowers to: create N anonymous dynamic Vars -(`Var.create()` — no ns/name, root = Unbound), `pushThreadBindings` a map binding -them to inits, run the body in `try`, `finally` `popThreadBindings`. `var-get`/ -`var-set`/`@` operate on them inside the body. - -cljw facts that shape the design (verified): -- A cljw `Var` is `gpa.create`'d and a `.var_ref` Value is **filtered out of the - GC membrane** — the GC never reclaims a Var; it is freed only by explicit - `gpa.destroy`. -- The value a local var is bound to is **already GC-rooted** by the existing - `thread_roots` walker (it drains `frame.bindings.valueIterator()` down the - parent chain). So there is **no GC-reachability problem** — the only open - question is the **heap lifetime of the gpa-owned Var struct**. -- `Var.ns` is a non-optional `*Namespace` (print/error read `v.ns.name`), so an - anonymous var needs a sentinel `__local` Namespace kept OUT of - `env.namespaces`. -- cljw has **no Unbound sentinel** (var-get on an unbound dynamic var returns its - root). JVM's `(var-get )` returns an **unreproducible - opaque** `#object[clojure.lang.Var$Unbound 0x…]` — so cljw cannot be F-011-clean - on the escaped-var path no matter what it returns. - -The lifetime question (where/when the anon Var is freed, and what the escaped-var -case does) was taken to a Devil's-advocate fork. - -## Decision - -**Alt C — anonymous local Vars are gpa-owned and NOT freed (tombstone); the -common path leaks the small Var struct; struct reclamation is deferred (D-255).** - -- New primitive **`-create-local-var`** (0-arity): `gpa.create(Var)` with `.ns` = - a process-global sentinel `__local` Namespace (lazily created, NOT registered in - `env.namespaces`), `flags.dynamic = true`, root nil, name `--unnamed--`. Returns - a `.var_ref`. **The struct is intentionally never `gpa.destroy`'d** — see below. -- **`with-local-vars`** is a `core.clj` `defmacro` (mirrors the with-redefs / - with-bindings / with-open family) lowering to: - `(let [x (-create-local-var) …] (push-thread-bindings (hash-map x init-x …)) - (try body… (finally (pop-thread-bindings))))`. -- `var-get`/`var-set`/`@` inside the body hit the active thread binding (works). - After the extent (`pop-thread-bindings`), the binding is gone; an **escaped** - var_ref deref returns the var's root (nil) — **safe, no UAF** (AD-015). - -**Why not free at pop (Alt B).** Freeing the Var at `pop-thread-bindings` gives a -leak-free common path, but an escaped var_ref then dereferences freed memory = -**use-after-free** (a crash-class memory-safety bug). For a runtime, shipping a -latent UAF — even on a rare anti-pattern path — is unacceptable; and since JVM's -escaped-var result is an unreproducible opaque, Alt B's "F-011-clean escape" is -illusory (cljw diverges on escape either way). Alt C trades a small, bounded -common-path leak for guaranteed memory safety. - -**Reclamation (D-255).** The common-path struct leak is the cost of escape-safety -without reference tracking. The finished-form reclamation is a generation-handle -slotmap (free the struct, detect stale escaped refs → Unbound) — deferred as a -proportionate optimization for a rarely-used feature, tracked in D-255 with a -`// D-255` marker at the `-create-local-var` site. Not shipped now (a slotmap for -a marginal feature is over-engineering until a workload makes the leak material). - -## Alternatives considered - -(Devil's-advocate subagent, fresh context, F-002/F-006/F-011 envelope.) - -- **Alt A — macro `finally` frees via a `-free-local-var` primitive.** Tiny; - common case leak-free. BUT an escaped var_ref → UAF, and that UAF "derives from - no invariant" (freeing eagerly for simplicity) so it cannot be a clean AD — it - ships a latent memory-safety bug. **Rejected** (DA: "not recommendable"). -- **Alt B — `pop-thread-bindings` frees the `__local`-sentinel vars it pops.** - Finished-form ownership (the frame owns its locals), leak-free common path. Still - escaped-var UAF → needs an F-006-derived AD. DA: "acceptable fallback." Rejected - here in favour of memory safety (the escaped-var UAF is the same crash class as - Alt A; the leak-free win does not justify a shippable UAF when escape can't be - cheaply made safe). -- **Alt C — gpa-owned, never freed (tombstone); reclamation = slotmap later.** - The chosen shape. Memory-safe (escaped deref → nil root, no UAF), F-011 as clean - as possible given JVM's unreproducible Unbound opaque. Cost: a small bounded - common-path leak, tracked as D-255 (slotmap upgrade) + a code marker. DA - **recommended C** (with B+AD as the fallback). Main loop concurs: memory safety - outranks the leak, and the leak is a tracked optimization-class debt, not a - correctness or safety defect. - -DA recommendation: Alt C (non-binding). Main loop: Alt C — per F-002 the -memory-safe finished form wins; the leak is deferred as D-255, not shipped as a -lie. (Choosing C over B is NOT a cycle-budget downgrade — it is a -memory-safety-over-leak judgement; the slotmap that would make B-without-UAF / -leak-free-C is the heavier C'' deferred to D-255.) - -## Affected files - -- `src/runtime/runtime.zig` — cache the sentinel `__local` Namespace pointer. -- `src/lang/primitive/core.zig` — `-create-local-var` primitive (lazy sentinel ns; - Var never freed + `// D-255` marker). -- `src/lang/clj/clojure/core.clj` — `with-local-vars` defmacro. -- `test/e2e/phase15_with_local_vars.sh` — JVM-parity cases. -- `.dev/accepted_divergences.yaml` — AD-015 (escaped-local-var deref/print). -- `.dev/debt.yaml` — D-237 discharged; D-255 (anon-var struct reclamation slotmap). - -## Consequences - -- `(with-local-vars [x 1 y 2] (var-set x 10) (+ (var-get x) (var-get y)))` => 12, - matching clj. -- Escaping a local var past its extent is memory-safe (deref → nil) but diverges - from JVM's unreproducible `Var$Unbound` opaque — AD-015. -- A small bounded leak of the anon Var struct per `with-local-vars` invocation; - immaterial for the feature's real usage, reclaimed by the D-255 slotmap upgrade - if a workload ever makes it matter. diff --git a/.dev/decisions/0098_cljw_http_server_surface.md b/.dev/decisions/0098_cljw_http_server_surface.md deleted file mode 100644 index 36757f495..000000000 --- a/.dev/decisions/0098_cljw_http_server_surface.md +++ /dev/null @@ -1,104 +0,0 @@ -# ADR-0098 — `cljw.http.server`: cljw's own HTTP server on std.Io.net (Ring-style), activating the `runtime/cljw/` original-surface tree - -- **Status**: Proposed → Accepted (2026-06-06) -- **Driven by**: the Clojure/conj 2026 CFP (D-256) — the Playground + edge-demo - need cljw to serve HTTP itself (the "all-cljw, the runtime hosts its own - playground + edge app" story). User-directed this session: **orthodox approach, - not a workaround** (no external Go runner). -- **Pulls forward**: the planned `runtime/cljw/edge/ Server` surface - (structure_plan.md / feature_name_consistency.md R3, Phase 14+) — activated now - by user direction. **Relates to**: F-009 (impl/surface split), F-011, D-256. - -## Context - -cljw had no HTTP server (cw v0 had one, but it is **disabled** in cw v0 itself: -its `run-server`/client raise "temporarily disabled while the std.net → std.Io.net -migration is in progress" — cw v0 was written against pre-0.16 `std.net`, removed -in Zig 0.16, and never migrated). So cw v0 supplies the **API design** (Ring-style -request/response maps, `run-server`/`set-handler!`, a hidden `__handler` var for -GC rooting) but **not** a working 0.16 implementation. - -Zig 0.16 networking feasibility was proven end-to-end this session by a standalone -prototype: `std.Io.net.IpAddress.listen(io,…) → Server.accept(io) → Stream → -stream.reader/writer(io,buf).interface → std.http.Server.init(&r,&w) → -receiveHead() → req.respond(body,…)` served real `curl` requests -(`HTTP/1.1 200 OK`). cljw already carries `std.Io.Threaded` io + a Thread base -(agent/future) on native targets, so the runtime substrate is in place. - -The Playground needs untrusted-code execution; the **edge-demo** needs a trusted -persistent Clojure web app. An HTTP server in cljw serves the edge-demo directly, -and serves the Playground's HTTP front while still spawning a fresh sandboxed -`cljw -e` subprocess per `/run` (isolation stays at the process/microVM boundary, -not in-process eval). - -## Decision - -**Activate the `src/runtime/cljw/` original-surface tree and implement -`cljw.http.server` on `std.Io.net` + `std.http.Server`, Ring-style.** - -1. **Surface**: `cljw.http.server` (implemented) + `cljw.http.client` (placeholder - stub — each fn raises `feature_not_supported`, reserving the name + the split; - a transient stub per provisional_marker.md, NOT a permanent no-op). Registered - as host namespaces (interned builtin fns), so `(require '[cljw.http.server :as - srv])` resolves them via the already-loaded-ns path (evalRequire: `findNs` with - `mappings.count() > 0` skips the `.clj` load — same as `clojure.set`). -2. **API (Ring)**: `(run-server handler {:port N})` where `handler` is - `(fn [req] resp)`. Request map: `{:request-method :get :uri "/path" - :query-string "..." :headers {"h" "v" …} :body "…"}`. Response map: - `{:status 200 :headers {…} :body "…"}`. `run-server` blocks the calling thread - serving a serial accept loop (the server IS the process for deploy; - per-connection threading + a non-blocking stop-fn handle are a follow-on). -3. **GC rooting**: the handler Value is held live for the server's lifetime (an - Env/Runtime-rooted slot), so a collect mid-serve cannot sweep it. -4. **Naming rationale** (ecosystem-grounded, user-decided): Clojure / Java / - Python / Babashka all **split server vs client** (`org.httpkit.server` vs - `org.httpkit.client` / `babashka.http-client`; `http.server` vs `http.client`; - `java.net.http` client vs `httpserver`); only Node/Go fold both into one `http`. - cljw is Clojure-flavoured → split + **Ring** maps + the `run-server` name - (http-kit/babashka familiarity). Bare `cljw.http` is rejected as ambiguous. - **`cljw.edge` is reserved** for an edge-DEPLOYMENT convenience layer, NOT the - HTTP primitive (don't conflate "HTTP server" with "edge deployment"). - -## Alternatives considered - -(The user actively steered each choice this session — the adversarial/fresh-context -check the DA-fork normally provides was supplied by the user's direction + the -ecosystem research, not the loop's momentum.) - -- **External Go runner** (Go-Playground model: a tiny Go HTTP server spawning - `cljw -e`). Fastest to a deploy, cljw unchanged. **Rejected by the user** as a - workaround — not orthodox; introduces a non-cljw dependency; doesn't serve the - edge-demo's "Clojure web app" story. -- **wasm port (cljw→wasm, browser-side Playground)**. The CFP_v2 "server-less = - safe" path, but needs the 3-blocker wasm32 port (threads / u64 atomics / - NaN-box 8-align). Deferred — heavier; this ADR's native server is orthogonal and - can coexist (a wasm Playground later reuses the same Ring handlers). -- **Bare `cljw.http`** (Node/Go style, one ns both ways) — ambiguous (the - "http = client or server?" problem); rejected for the split. -- **`cljw.edge` as the HTTP server name** — conflates HTTP with edge deployment; - reserved instead for the deployment layer. -- **Per-connection threading + stop-fn handle now** — deferred; serial blocking is - correct + simplest for the deploy shape (server = process). Concurrency is a - follow-on (cljw has the Thread base). - -## Affected files - -- `src/runtime/cljw/http/server.zig` — the std.Io.net + std.http.Server impl - (listen/accept/serve loop, request→Ring-map, response-map→HTTP). -- `src/runtime/cljw/http/client.zig` — placeholder stub (`feature_not_supported`). -- `src/lang/primitive/cljw_http.zig` — the `run-server` builtin (rt/env/args - bridge: build req Value, `invokeCallable` the handler, render resp) + the - `register(env)` that creates the `cljw.http.server` / `cljw.http.client` host - namespaces; wired into `primitive.registerAll`. -- `test/e2e/phase16_http_server.sh` — bg `cljw` server process + `curl` + assert. -- (`.dev/structure_plan.md` note: `runtime/cljw/` activated ahead of Phase 14.) - -## Consequences - -- cljw can host its own Playground front + edge web app — no external runner; the - "all-cljw runtime serves itself" CFP narrative. -- `cljw.http.client` name is reserved (stub) for a later client (Ring-style maps). -- Serial blocking server is correct for the deploy shape; concurrency + a - non-blocking stop handle are tracked follow-ons (a debt row at land time). -- The Playground still spawns a sandboxed `cljw -e` subprocess per request — - isolation stays at the OS/microVM boundary, not in the server process. diff --git a/.dev/decisions/0099_wasm_ffi_minimal_surface.md b/.dev/decisions/0099_wasm_ffi_minimal_surface.md deleted file mode 100644 index 996099921..000000000 --- a/.dev/decisions/0099_wasm_ffi_minimal_surface.md +++ /dev/null @@ -1,205 +0,0 @@ -# ADR-0099 — Minimal polyglot Wasm FFI surface (`wasm/load` + `wasm/call`) behind a `-Dwasm` build flag, embedding zwasm v2 - -- **Status**: Proposed → Accepted (2026-06-06) -- **Driven by**: the Clojure/conj 2026 CFP (D-256) **P1 — the heart of the - talk**: a Clojure REPL loading a sandboxed Rust/Zig-compiled `.wasm` and - calling it like a namespace ("WebAssembly as an FFI"). This is the one thing - no other Clojure runtime demonstrates. -- **Pulls forward**: the planned `runtime/cljw/wasm/` surface - (structure_plan.md §wasm, feature_name_consistency.md R3, Phase 16) as an - **F-001 forward spike** — NOT the ADR-0006 Phase-16 full-FFI reintroduction. -- **Relates to**: F-001 (zwasm v2 unavoidable, lazy+flag-guarded), F-002 - (finished-form wins), F-004 (NaN-box Group D wasm slots), F-006 (GC - dual-heap), F-009 (impl/surface split), ADR-0006, ADR-0098, D-037/D-038 - (the zwasm embedding spike), D-259 (the provisional handle close-out). - -## Context - -The CFP's differentiator is polyglot: WebAssembly as an FFI, so a Clojure REPL -can `(wasm/load "add.wasm")` a module compiled from another language and -`(wasm/call inst "add" 2 40)` → `42`. The Zig-level feasibility is already -proven: the D-037 spike (`spike/zwasm_embed.zig`, `zig build zwasm-spike` — -both removed 2026-07-01 once the Phase-16 FFI shipped) consumed zwasm v2's -embedding API via a `build.zig.zon` relative-path import and returned `42`. -What is missing is the **Clojure-level surface**. - -F-001 is the governing constraint. zwasm v2 is under active AI development on -its `zwasm-from-scratch` long-lived branch; the loop **must not** make cljw's -default build or `test/run_all.sh` gate depend on it (a churning zwasm tree -must never break cljw's gate). The spike satisfies this with a LAZY, -flag-guarded dep (`-Dzwasm-spike`). The full Phase-16 form (`runtime/wasm/` -with linker/host_func/wasi/funcref-slots/trap_map/pod_boundary, -structure_plan.md:191-203) is owned by the Phase-16 entry (D-036 inline-vs-Pod -is still open). This ADR is the **forward spike toward** that form, behind the -same isolation, not its closure. - -zwasm v2 embedding ownership (read from `~/Documents/MyProducts/zwasm_from_scratch` -on the `zwasm-from-scratch` branch): `Engine` owns the c-api `*Store`; -`Module`/`Instance` hold pointer-value copies of `c_store` (not self-referential -into each other's storage), so an `Engine`+`Module`+`Instance` triple is -heap-boxable and movable — only deinit-order (instance→module→engine) and -"engine outlives instance" matter. The untyped dynamic path -`Instance.invoke(name, []const Value, []Value)` + `exportFuncSig(name) ?FuncType` -is the marshal driver (a comptime `typedFunc` is unusable — the wasm signature -is only known at runtime). - -## Decision - -**Implement a minimal `wasm/load` + `wasm/call` surface under -`runtime/cljw/wasm/`, embedding zwasm v2, activated only by a `-Dwasm` build -flag. The default build + gate set `wasm=false` → zero zwasm symbols, `wasm` -namespace absent (F-001 preserved).** - -1. **Build flag (F-001 isolation)**: `-Dwasm` (default false). When set, and - only when the lazy `zwasm` dep actually resolves, `build.zig` adds the - `zwasm` import to the cljw `exe_mod` AND sets `build_options.wasm=true`. - `build_options.wasm` is set `true` *only inside* the resolved-dep block, so - the null-first-pass of `b.lazyDependency` leaves it false (no `@import("zwasm")` - is then analysed → no compile error). Default `zig build` / `run_all.sh` - never reach `b.lazyDependency` with the flag → no fetch, no symbols. -2. **Surface placement** (Alt-2, per the DA): three files under - `runtime/cljw/wasm/`, drawn on the Phase-16 file boundaries so Phase-16 is - *addition, not extraction*: - - `engine.zig` — the zwasm Engine/Module/Instance lifecycle wrapper. Its - public fn shapes (`load(alloc, bytes) -> *Loaded`, `Loaded.invoke`, - `Loaded.exportSig`) match the Phase-16 module/instance split, so the later - split does not change callers. **F-006 seam**: the allocator is a *named - parameter* of `load`, never an inline global grab — the cw-GC-allocator - inject point structure_plan.md:192 names is already here (P1 passes - `rt.gpa`, the F-006 layer-1 backing allocator, NOT the moving GC heap). - - `marshal.zig` — Clojure Value ↔ `zwasm.Value`, driven by `exportFuncSig` - ValTypes (int→i32/i64, float→f64-bits via `@bitCast`, and back). The - load-bearing finished-form piece. - - `surface.zig` — `register(env)` + the `wasmLoadFn`/`wasmCallFn` builtins, - thin, mirroring `cljw/http/server.zig`. Registers ns `wasm` with `load`, - `call`. - Plus `wasm_handle.zig` (the single provisional-handle wrap/unwrap site, §4). - Wired by one line in `runtime/cljw/_host_api.zig` under - `if (build_options.wasm)`. -3. **API (2-call, auto-sig)**: `(wasm/load "path.wasm")` → an opaque instance - handle Value; `(wasm/call handle "export" & args)` → result (single result = - scalar, multiple = vector). Arg/result types are **derived from - `exportFuncSig`** — no hand-written `:params`/`:results` (the v0 shape). v0's - `load`→module / `fn`→callable / call split is a DIVERGENCE data point, not - copied (no_copy_from_v1.md); v2's runtime sig probe makes the 2-call shape - sufficient. -4. **Provisional instance handle (D-259)**: the handle is a GC-allocated - `WasmHandle { header: HeapHeader, loaded: *Loaded }` tagged with the - **already-reserved** `HeapTag.wasm_module` (slot D4), wrap/unwrap isolated to - `wasm_handle.zig`, `PROVISIONAL:`-marked. The slot is reserved (F-004) but - its *semantics* (module vs instance vs fn split, `funcref`/`externref` - first-classing, GC finalisation of the external `*Loaded`, auto-collect - rooting) are owned by the Phase-16/F-004 entry, not this demo cycle — so P1 - populates the slot minimally (HeapHeader + external pointer + `null` tag_ops - trace, the correct leaf default per F-006) and defers the structural calls to - D-259. The external `*Loaded` is intentionally not finalised for P1 (the - `cljw add.clj` demo is short-lived; auto-collect is off); D-259 owns - finalisation/rooting. -5. **Verification (F-001-honest)**: the demo is verified by `zig build -Dwasm` - + `cljw docs/examples/wasm/add.clj` → `42`, run on demand (and at the Phase-16 - gate / pre-tag), **NOT** in the default per-commit gate — the default gate - must never resolve zwasm. `docs/examples/wasm/` carries the prebuilt `add.wasm`, - its WAT source, and the repro command (PRIORITY.md P1). - -## Alternatives considered - -The mandatory Devil's-advocate subagent (fresh context, F-001/002/004/006 -pasted) produced the following, reflected verbatim: - -> **F-NNN CHECK: No alternative below requires violating any F-NNN. The -> finished-form-clean option (Alt-2) stays fully inside F-001/F-004/F-006. The -> only F-004 pressure point — the wasm-instance handle representation — is -> resolved by *deferral to the Phase-16/F-004 Group D owner*, which is what -> F-004 itself mandates ("slot allocation is owned by that Phase entry, NOT a -> demo cycle"). So there is no halt-flag here.** -> -> **(Alt-1) Smallest-diff — single `surface.zig`, zero-marshal-abstraction, -> raw boxed pointer.** One file containing everything; the marshal loop and -> box/unbox written inline next to the only caller, no `engine.zig`/`marshal.zig`. -> (a) Better: smallest possible diff; one Backend marker not three; honest about -> being a spike. (b) Breaks: directly contradicts **F-009** -> (`feature_name_consistency.md` R4: "Body inline in surface file: forbidden; -> surface is a thin wrapper over the neutral impl"); the marshal + engine -> lifecycle are neutral-impl, so inlining means the Phase-16 owner must -> *extract* them — negative work; and it scatters the provisional box, losing -> the single-swap-site guarantee. (c) Envelope: inside F-001/F-004/F-006 but -> **violates F-009** → out of the envelope on F-009 grounds. -> -> **(Alt-2) Finished-form-clean — split exactly on the Phase-16 file -> boundaries, with marshal as a neutral seam.** `engine.zig` (lifecycle, public -> fn shapes matching the structure_plan:194-195 module/instance split so -> Phase-16 is addition-not-extraction), `marshal.zig` (the load-bearing -> Value↔zwasm.Value piece, structure_plan:199), `surface.zig` (thin -> register+builtins mirroring http/server.zig:115,142-150), `wasm_handle.zig` -> (single provisional wrap/unwrap, D-259 + feature_deps + marker triad). Key -> refinement over the draft: the **F-006 allocator inject point** — `engine`'s -> `init`/`load` takes the allocator as a *named parameter* (structure_plan:192 -> "cw GC allocator inject point per F-006"), never an inline `rt.gpa` grab. -> (a) Better: commits engine signatures to the Phase-16 split + makes F-006 a -> named seam from day one — both zero-extra-LOC, both what the owner would -> otherwise retrofit. (b) Breaks: marginally more up-front signature thought (not -> LOC); over-committing engine's signature to a split that may change — cheap to -> amend. No semantic risk (P1 behaviour identical). (c) Envelope: fully inside; -> F-006 seam is *more* faithful than the draft; F-004 handle deferred-to-owner. -> -> **(Alt-3) Wildcard — separate demo binary `cljw-wasm` instead of a `-Dwasm` -> flag on the main binary.** A second `b.addExecutable` sharing the cljw root -> module + the wasm surface; the default `cljw` source never references -> `build_options.wasm`. (a) Better: strongest F-001 isolation (the default -> binary's source set never mentions wasm); a named artifact demos better on -> stage. (b) Breaks: two boot-path wirings = parallel-surface drift risk (the -> single-aggregator `_host_api.zig` exists to prevent exactly this); the -> Phase-16 finished form (structure_plan:191-203) assumes `wasm` is a namespace -> in the *one* cljw runtime, so a separate binary is throwaway scaffolding the -> owner unwinds (Skeleton-that-enlarges-the-rewrite smell). The `dlopen` -> sub-variant strays furthest from F-001's "lazy build-time dep" letter. -> (c) Envelope: inside F-001 (more isolated) but off-trajectory for the -> single-runtime Phase-16 form — clean isolation bought at the cost of clean -> finished form (the F-002 concern). -> -> **RECOMMENDATION: Alt-2** (the draft, tightened on two seams). Per F-002 the -> question is not "smallest diff to `42`" (Alt-1, rejected — violates F-009) nor -> "tightest isolation" (Alt-3, off-trajectory for the single-runtime Phase-16 -> form), but "which shape would the Phase-16 owner be happiest to inherit": the -> three-file split, additionally (i) committing `engine.zig`'s public signatures -> to the module/instance split so Phase-16 is addition-not-extraction, and (ii) -> making the F-006 allocator a named parameter seam. Both are zero-extra-LOC — -> finished-form fidelity, not a cycle-budget trade. On F-004 all alternatives -> correctly *defer* the NaN-box slot semantics to the owning entry; Alt-2 keeps -> that deferral honest via the single `wasm_handle.zig` swap site + D-259 triad. - -**Adopted: Alt-2**, with both seam-tightenings folded in. The instinct toward -Alt-1's smaller diff is the kind F-002 forbids (F-009 surface-thinness is a -project fact, not a cycle-budget question); Alt-3's isolation is real but -off-trajectory for the single-runtime finished form. - -## Consequences - -- **Positive**: the CFP's central demo (`cljw add.clj → 42` calling a - Rust/Zig-compiled wasm module) becomes a real, reproducible artifact with a - recordable URL. The surface lands *toward* the Phase-16 finished form - (engine/marshal/surface keep their names + responsibilities; the deferred - Phase-16 files — linker/host_func/wasi/table/global/memory/funcref/externref/ - trap_map/pod_boundary — are pure additions). The F-006 allocator seam + the - engine module/instance signatures are committed correctly from day one. -- **Negative / deferred (tracked)**: D-259 owns the provisional handle - close-out (NaN-box slot semantics + GC finalisation/rooting + the - module/instance/fn split + funcref/externref first-classing). The demo leaks - the external `*Loaded` (acceptable: short-lived process, auto-collect off). - The `-Dwasm` demo is not in the default gate (F-001 mandate) — it is an - on-demand / Phase-16 / pre-tag check; `docs/examples/wasm/README.md` carries the - repro so it is not silently un-exercised. -- **F-001 finding-handling**: any zwasm-side bug/gap surfaced while wiring this - is RECORDED + FED BACK via `private/notes/zwasm_v2_feedback.md` (no cljw-side - workaround, never edit zwasm); a cljw-side bug gets a real fix + debt row. - -## Affected files - -- `build.zig` — `-Dwasm` option + lazy-dep-guarded `exe_mod.addImport("zwasm")` - + `build_options.wasm`. -- `build.zig.zon` — reuses the existing lazy `.zwasm` path-dep (no change). -- `src/runtime/cljw/wasm/{engine,marshal,surface,wasm_handle}.zig` — new - (flag-guarded). -- `src/runtime/cljw/_host_api.zig` — one `if (build_options.wasm)` register line. -- `docs/examples/wasm/{add.wasm,add.wat,add.clj,README.md}` — the demo + repro. -- `.dev/debt.yaml` (D-259) + `feature_deps.yaml` — the provisional triad. diff --git a/.dev/decisions/0100_prime_ops_auto_promote.md b/.dev/decisions/0100_prime_ops_auto_promote.md deleted file mode 100644 index 959b240f9..000000000 --- a/.dev/decisions/0100_prime_ops_auto_promote.md +++ /dev/null @@ -1,159 +0,0 @@ -# ADR-0100 — The `'` arithmetic operators (`+'` `-'` `*'` `inc'` `dec'`) auto-promote, matching JVM Clojure (D-260) - -- **Status**: Proposed → Accepted (2026-06-06) -- **Driven by**: D-260 — found during the CFP campaign that cljw's `'` operators - raise `integer_overflow` on Long overflow, the exact inverse of JVM Clojure. -- **Relates to**: F-005 (numeric tower = JVM-surface; non-prime `+`/`*` already - auto-promote), F-011 (behavioural equivalence on the observable surface), F-002. - -## Context - -In JVM Clojure the **`'`-suffixed** operators are the **auto-promoting** family: -`+'` `-'` `*'` `inc'` `dec'` promote `Long`→`BigInt` on overflow and **never -throw**. The non-prime `+` `*` `-` `inc` `dec` are the **throwing** ones -(`(* 1000000000000 1000000000000)` → `ArithmeticException: long overflow`). -Verified against the clj 1.12 oracle: - -``` -clj (*' 1000000000000 1000000000000) => 1000000000000000000000000N ; promotes -clj (+' 9223372036854775807 1) => 9223372036854775808N ; promotes -clj (* 1000000000000 1000000000000) => ArithmeticException "long overflow" -``` - -cljw has this **inverted**. Per F-005, cljw's **non-prime** `+`/`*`/`-` auto-promote -(an intentional, documented divergence from JVM's throwing non-prime ops). But -cljw's `'` ops were implemented as **strict** — `plusStrict`/`minusStrict`/ -`starStrict`/`incStrict`/`decStrict` in `src/lang/primitive/math.zig`, routed -through `promote.addStrict`/`subStrict`/`mulStrict`, which raise `integer_overflow` -— with a docstring falsely claiming "Mirrors JVM Clojure's `+'`". So -`(*' 1e12 1e12)` **throws** in cljw but promotes in JVM. (D-260's first read -"`+'` works" was a false positive: its input `9223372036854775807` exceeds cljw's -i48 fixnum range and is already a BigInt, so no fixnum overflow occurred to raise -on.) There is **no AD-NNN or ADR** recording an intentional strict-`'` divergence; -the inversion rests solely on the wrong docstring. - -## Decision - -**Make `+'` `-'` `*'` `inc'` `dec'` auto-promote — route them through the same -promoting path the non-prime ops use — and delete the strict family entirely.** - -1. Re-point the `'` registrations in `math.zig` to the promoting fns (`plus`/ - `minus`/`star`/`inc`/`dec`). -2. **Delete** `plusStrict`/`minusStrict`/`starStrict`/`incStrict`/`decStrict` - (math.zig) **and** `addStrict`/`subStrict`/`mulStrict` (`promote.zig`) **and - their unit tests** — the DA verified these have zero callers outside the - registration table and their own pin tests, so deletion is clean and avoids a - dead-code island the `no_unused` gate would flag. -3. Fix the docstrings to state the truth: in cljw the `'` ops are the - auto-promoting family, **behaviourally identical to the non-prime ops**, - because cljw's non-prime ops also promote (F-005). JVM splits throwing-`*` vs - promoting-`*'`; cljw collapses the throw side into promote, so both spellings - legitimately converge on one promoting semantics. -4. Land `test/diff/clj_corpus/numeric_prime_ops.txt` (per `clj_diff_sweep.md` - Discipline 1) asserting the `'` ops promote, so the parity claim is mechanically - re-checkable and a regression flips the gate red. - -This is a **bug→fix** (F-011), not an accepted divergence: no project invariant -licenses diverging on `'`-op promotion. - -## Alternatives considered - -The mandatory Devil's-advocate subagent (fresh context, F-005/011/002 pasted) -produced the following, reflected verbatim: - -> **No alternative below requires violating an F-NNN. The finished-form-clean -> option (Alt-2) is fully inside the envelope.** F-005 already mandates that -> cljw's integer ops auto-promote Long→BigInt; F-011 mandates the user-observable -> surface match clj 1.12, where `*'` promotes. So "make `'` ops promote" satisfies -> both invariants simultaneously — there is no tension to resolve. -> -> I verified the blast radius before drafting: the `plusStrict`/`minusStrict`/ -> `starStrict`/`incStrict`/`decStrict` registrations and the `addStrict`/ -> `subStrict`/`mulStrict` promote.zig helpers have **zero callers** outside -> `math.zig`'s registration table and promote.zig's own unit tests. The strict -> family is a closed, dead-end surface. -> -> **Alt-1 — Smallest-diff: re-point the 5 registrations to the promoting fns.** -> Change only the registration table; leave the strict fns physically in the tree -> (unreferenced by the registry, still referenced by their own tests). Fix the -> false docstrings. (a) Better: minimal diff; the 5 prime ops immediately stop -> throwing; promote.zig `*Strict` unit tests keep passing untouched; reversible. -> (b) Breaks: leaves five dead Zig functions + three dead helpers reachable only -> from their own pin tests — a self-justifying dead-code island. The `no_unused` -> lint gate (ADR-0003) will likely flag the unreferenced math.zig `plusStrict`/etc -> as unused, so this "smallest" diff probably **does not even compile-pass the -> gate** without deleting them. This is the **Smallest-diff bias smell**. (c) -> Envelope: in-envelope behaviourally, but conflicts with the no_unused gate and -> F-002 finished-form spirit — in-envelope-but-smelly. -> -> **Alt-2 — Finished-form-clean: delete the strict family entirely, route `'` -> ops through the promoting path, add a corpus pin (= the draft, sharpened).** -> Delete the strict fns from math.zig AND promote.zig AND their unit tests. -> Register the `'` ops against the promoting fns. Fix docstrings to state the -> convergence. Add `test/diff/clj_corpus/numeric_prime_ops.txt`. (a) Better: the -> draft only names the math.zig fns; this also deletes the promote.zig `*Strict` -> helpers + tests, otherwise the strict semantics survive as a second dead island -> kept alive only by its own pin tests. Removes the entire strict surface so no -> future reader wonders "why are there two families and which is the contract." -> The corpus pin makes the parity claim mechanically re-checkable (anti-D-177). -> (b) Breaks: largest diff (deletes ~8 functions + tests). Removes cljw's *only* -> overflow-detecting path — after this, no cljw operator throws on overflow. Per -> F-002, diff size is explicitly not a constraint. (c) Envelope: fully in-envelope -> and closest to finished form. -> -> **Alt-3 — Wildcard: keep `'` strict, re-found it as cljw's deliberate -> overflow-detecting family and record AD-NNN.** Since cljw's non-prime ops -> promote (F-005), cljw has no throwing operator; Alt-3 repurposes `'` as that -> missing strict family with an AD. (a) Better: gives users who want overflow -> detection an operator; zero deletion. (b) Breaks — decisively: it **breaks -> F-011 head-on.** `'` is a decade-old spelling meaning "auto-promote" to every -> Clojure programmer; portable `.clj` uses `*'`/`+'` precisely because they want -> BigInt promotion (e.g. `(reduce *' (range 1 30))`). Under Alt-3 that throws -> where JVM produces a BigInt — exactly the divergence F-011 forbids, and -> `accepted_divergences.md` bars accepting it: an AD needs a *project invariant* -> as `derives_from`, and "we'd like a strict op so let's hijack a JVM-meaningful -> name" is convenience, not invariant. An AD here would be a lie dressed as a -> contract. On the sub-question — should cljw offer a strict op at all? Possibly, -> but it must NOT steal the `'` spelling; mint a new non-colliding name in its own -> ADR. (c) Envelope: **out of envelope — violates F-011.** Recording it as AD-NNN -> does not rescue it (no invariant to cite). This is the leading finding. -> -> **RECOMMENDATION — Alt-2 (the draft, sharpened to also delete the promote.zig -> `*Strict` layer + its tests).** F-011 is decisive and points one way: clj -> `(*' …)` promotes, cljw must match — only Alt-1/Alt-2 deliver that, Alt-3 is -> rejected because it breaks F-011 and the AD escape hatch is unavailable. F-005 -> makes Alt-2 coherent: cljw collapsed the throw side into promote, so both -> spellings legitimately converge on one promoting semantics. F-002 breaks the -> Alt-1/Alt-2 tie toward Alt-2: Alt-1 leaves a dead island the no_unused gate -> rejects and a finished-form owner would delete — "Alt-2 is a bigger diff" is not -> a valid reason to downgrade. Sharpening: also delete the promote.zig -> `addStrict`/`subStrict`/`mulStrict` + their tests (verified no other callers). -> Note for Consequences: after Alt-2 cljw has no integer operator that throws on -> overflow; a future overflow-detecting op is a separate ADR minting a new non-`'` -> name. - -**Adopted: Alt-2**, with the sharpening (delete the promote.zig strict layer + -tests). Alt-3 is the salvageable-idea source (a future strict op under a new -name) but is rejected here as F-011-violating. - -## Consequences - -- **Positive**: `(*' …)`, `(+' …)`, `(-' …)`, `(inc' …)`, `(dec' …)` now match - clj 1.12 (auto-promote, never throw) — an F-011 fix in the CFP-showcased numeric - tower. The strict family is gone, so cljw has exactly one integer-arithmetic - semantics (promote) under both spellings. -- **Consequence to record**: after this, **no cljw integer operator throws on - overflow** — every integer op promotes (the correct joint consequence of F-005 - + ADR-0100). If the project later decides users need overflow detection, that is - a *separate* ADR minting a new, non-`'` name; it must not reopen ADR-0100 or - re-strict the `'` ops. -- **Discharges**: D-260. - -## Affected files - -- `src/lang/primitive/math.zig` — delete the 5 `*Strict` fns; re-point the `'` - registrations to the promoting fns; fix docstrings. -- `src/runtime/numeric/promote.zig` — delete `addStrict`/`subStrict`/`mulStrict` - + their unit tests. -- `test/diff/clj_corpus/numeric_prime_ops.txt` — new parity corpus. -- `.dev/debt.yaml` — D-260 discharged. diff --git a/.dev/decisions/0101_deps_git_fetch.md b/.dev/decisions/0101_deps_git_fetch.md deleted file mode 100644 index e94c97ff0..000000000 --- a/.dev/decisions/0101_deps_git_fetch.md +++ /dev/null @@ -1,137 +0,0 @@ -# ADR-0101 — deps.edn `:git/url` resolution: `git` subprocess into a `~/.cljw/gitlibs` cache (Stage 1.2 slice 5) - -- **Status**: Proposed → Accepted (2026-06-06) -- **Driven by**: Convergence Campaign Stage 1.2 slice 5 — `:git/url`+`:git/sha` - deps are the last unresolved deps.edn coordinate; the real-world lib ladder - (`docs/works/`) has `blocked: no deps.edn yet` rows that need a git source - (e.g. `clojure.data.priority-map`). -- **Relates to**: F-002 (finished-form wins), `no_copy_from_v1` (re-derive v0's - `resolveGitDep`, don't port), the source-only policy (ADR-less, established by - the Stage 1.2 Maven reject — no JAR resolution), `zone_deps` (git_fetch.zig is - app-layer, may spawn). - -## Context - -deps.edn `:paths`/`:local/root`/`:aliases` resolution landed (slices 1–4). The -remaining coordinate is `{:git/url "..." :git/sha "..."}`: fetch the repo at the -pinned sha and add its `:paths` (joined to `:deps/root` if present) to the -classpath. - -**This is the first process-spawning code in cw v1.** `rg 'std.process.Child|std.process.run' src/` is empty — every prior feature is in-process. Spawning `git` is a new capability surface (subprocess, network, a user-data-controlled cache directory), which is why it is gated behind this ADR with a mandatory Devil's-advocate fork (CLAUDE.md § ADR-level designs are handled inline). - -v0's `resolveGitDep` (cli.zig L736–940) is the prior art (read-only reference, NOT ported): cache at `~/.cljw/gitlibs//`, `git clone` + checkout, `allow_fetch` gating, silent `return` on every error (HOME unset, clone failure). The cw-clean version must isolate the subprocess in its own module and surface real errors. - -## Decision - -1. **Module**: a new `src/app/deps/git_fetch.zig` (app-layer; `zone_deps` permits app→anything). It is the *only* file in cw v1 that spawns a subprocess; the isolation makes **`rg 'std.process.run' src/`** return exactly one module, so the subprocess surface is auditable in one place. (The concrete spawn-API string is `std.process.run` — NOT bare `std.process` (`std.process.exit`/`Init`/`currentPath` already appear runtime-wide, the DA's leading catch), and NOT the literal `std.process.Child` either: the 0.16 convenience is the free function `std.process.run`, which wraps `Child` internally — so `std.process.run` is the string that actually appears in the code and the one the Final-Stage audit must grep.) -2. **Spawn**: `std.process.run(gpa, io, .{ .argv = &.{"git", ...} })` (Zig 0.16 — the free function; `std.process.Child.run` does not exist in 0.16, only `Child.wait`/`kill`). `git` is invoked by argv vector (no shell string), so a malicious `:git/url` cannot inject shell metacharacters — the URL is a single argv element passed to `git clone`. -3. **Cache (Shape B — content-addressed full sha)**: `$CLJW_HOME/gitlibs///`, where `CLJW_HOME` defaults to `~/.cljw` but is overridable; a hard `error_catalog` error (not a silent skip) when **both** `CLJW_HOME` and `HOME` are unset. Keying on the **full** sha (not a 12-char prefix) makes the cache content-addressed and lets multiple shas of one repo coexist without aliasing. This intentionally diverges from v0 / the JVM `~/.gitlibs/` layout (the "shared mental model" is weak — the caches are not interchangeable artifacts; see Consequences). -4. **Algorithm**: cache hit (the sha dir exists) → use it; miss → `git clone ` then `git -C checkout `, then **`git -C rev-parse HEAD` and assert it equals `:git/sha`** (Shape B — catches a moved tag / tampered mirror), then atomically rename `` → the cache dir (so a half-clone or SIGINT never poisons the cache). The cached dep's own `deps.edn` is then resolved transitively by reusing `resolve.zig` (the same `:paths`/`:local/root` recursion), with `:deps/root` appended as a subdir. -5. **Failure UX (cw-clean improvement over v0's silent return)**: HOME+CLJW_HOME unset, `git` not on PATH, clone failure, sha mismatch, or sha-not-found each raise a real `error_catalog` error rendered against the deps.edn source (the same startup-error path slice 3 added) — never a silent skip that leaves the user with a confusing downstream `lib_not_found`. -6. **Network boundary**: fetching is **on by default** when a `:git/url` dep is present (matching the `clojure` CLI). The trust boundary is the user's own deps.edn — the same boundary as `-cp` or running a `.clj` file. The post-checkout sha-verification (point 4) closes the one gap where "trusted like a classpath entry" does not hold (a network-resolved tag can move; a locally-authored path cannot). No further sandboxing of the fetched source. -7. **Testing (diverged from the DA — real subprocess, not an injected seam)**: the e2e drives a hermetic **local `file://` bare git repo** (created in the test via `git init --bare` + a commit), so it exercises the *actual* clone/checkout/rev-parse path offline with no GitHub dependency. A `command -v git` guard SKIPs (with a printed SKIP line, not a silent pass) where `git` is absent. No `GitFetcher` vtable: a single implementation does not warrant the indirection, and a fake fetcher would leave the new subprocess code untested (see § Main-loop disposition). - -## Alternatives considered - -Devil's-advocate subagent output (verbatim, fresh context, F-NNN envelope): - -> **Leading finding (constraint check):** None of the three alternatives below requires violating `no_copy_from_v1`, source-only, or `zone_deps`. The finished-form-clean option is fully reachable inside the envelope, so there is no "the clean shape is blocked by an F-NNN/rule" entry to lead with. One factual correction the draft should absorb regardless of which shape wins: the claim that `git_fetch.zig` makes `rg 'std.process' src/` "return exactly one module" is **already false** — `std.process.exit`, `std.process.Init`, `std.process.currentPath`, and `std.process.executablePathAlloc` appear across `cli.zig`, `runner.zig`, `runtime.zig`, `System.zig`, `builder.zig`, etc. The auditable invariant the ADR actually wants is `std.process.Child` (subprocess spawning), which today returns *zero* hits. The Consequences/Decision-1 text should be narrowed to `std.process.Child` or the audit assertion is born a lie (the false-positive-trigger class the audit_scaffolding skill hunts). -> -> **Shape A — Smallest-diff: argv `git` subprocess, but defer fetch behind `:git/sha` verification + keep cache exactly as v0.** *Shape:* Essentially the draft as written (argv `std.process.Child.run`, `~/.cljw/gitlibs//`), with one subtraction: do **not** add the atomic-rename tmp dance or transitive-resolve recursion in this slice — clone+checkout straight into the cache dir, resolve only the fetched repo's own `:paths` (no nested `:git/url`). *Better:* Smallest surface to land slice 5; the recursion and half-clone-poisoning are real but unexercised by the ladder's near-term rows (`data.priority-map` is leaf-like). It gets a git coordinate resolving today and lets the corpus prove it. *Breaks:* A clone interrupted by SIGINT leaves a partial cache dir that the next run treats as a hit (the exact footgun the draft's atomic-rename fixes) — so this is strictly *worse* on finished-form cleanliness, and re-opening the cache module later to retrofit atomicity is rework the draft already avoided. Dropping transitive git-recursion also means a dep-of-a-git-dep silently fails to load; that is a correctness gap, not a scope deferral. This shape only "wins" on diff size, which is not a project constraint — flagging it as the Cycle-budget-defer smell. -> -> **Shape B — Finished-form-clean: argv `git` subprocess + content-addressed cache keyed on the *resolved* sha, with the subprocess capability modeled as an injected `GitFetcher` vtable so the network boundary is testable without a real `git`.** *Shape:* Keep argv `git` (not libgit2, not pure-Zig — see Recommendation), keep clone+checkout, keep the atomic tmp→rename. Two upgrades over the draft: (1) **Cache key on the full resolved sha, not a 12-char prefix**, and lay it out `$CLJW_HOME/gitlibs////` with `CLJW_HOME` defaulting to `~/.cljw` but overridable, and a hard error (not silent skip) when HOME is unset *and* `CLJW_HOME` is unset. Full-sha keying makes the cache content-addressed and lets multiple shas of the same repo coexist without collision; a 12-char prefix is a gratuitous truncation that can alias and has no upside now that disk is cheap. (2) **Model the spawn as a `GitFetcher` interface** (a struct of fn pointers on the deps-resolve context, default impl = the argv-`git` one in `git_fetch.zig`), so the e2e and unit tests inject a fake fetcher that populates the cache from a fixture dir — no `git` binary, no `file://` repo, no network, fully deterministic. The fetch-on-by-default + "trusted like -cp" trust boundary stays (it matches `clojure` CLI and the user authored the deps.edn), **but** the resolved sha is verified: after checkout, assert `git rev-parse HEAD == :git/sha` and raise `error_catalog` on mismatch, so a moved tag or a tampered mirror is caught rather than silently classpath'd. *Better:* This is the shape a finished-form audit wants — content-addressed cache (no prefix aliasing, clean multi-version coexistence), the subprocess boundary is an injectable seam (so testing doesn't depend on `git` being on the CI PATH or on `file://` URL parsing quirks), and sha-verification closes the one place "trusted like a classpath entry" genuinely *doesn't* hold: a classpath entry is authored locally, whereas a `:git/url` resolves over the network where a tag can move under you. The vtable seam is also the natural place a future sandbox/lockfile hook lands without re-surgery. *Breaks:* Bigger diff than the draft (the vtable indirection + cache-key migration + rev-parse step). It also means the cache layout intentionally *diverges* from JVM `~/.gitlibs/` and from v0 — losing the "shared mental model" the draft cites as a benefit. That benefit is weak (cw and JVM caches are not interchangeable artifacts; a user clearing one still clears the other by path), so the divergence is acceptable, but it must be named in Consequences so it isn't a silent break of the draft's stated goal. -> -> **Shape C — Wildcard: defer `git` subprocess entirely; ship `:paths`+`:local/root` for slice 5 and resolve `:git/url` by fetching a source *tarball* over the existing-or-new HTTP surface into the same content-addressed cache — no `git` binary on PATH, no subprocess at all.** *Shape:* No `std.process.Child`. For `{:git/url :git/sha}`, derive a codeload/archive URL (GitHub/GitLab expose `…/archive/.tar.gz`), fetch via `std.http.Client`, unpack into `$CLJW_HOME/gitlibs////`, resolve `:paths`. *Better:* Eliminates the entire new capability surface the ADR is gated on — no subprocess, no shell, no `git`-on-PATH dependency (a real friction for minimal containers and for the eventual wasm/edge target where spawning `git` is meaningless, which is cw's actual differentiator per F-010/F-012). A pure-HTTP fetch is far closer to "runs anywhere cljw runs" than "requires a POSIX host with git installed". *Breaks:* **This is the decisive flaw.** cw today has `std.http.Server` (the cljw-original http surface) but **no `std.http.Client`** — so this shape *adds a new HTTP-client network capability* rather than reusing existing surface, trading one new capability (subprocess) for another (outbound TLS HTTP client + tar.gz inflate + per-forge URL-shape knowledge). It is **not** source-only-violating (a source tarball is git source, not a Maven JAR/artifact), so it stays inside the rule — but it bakes in forge-specific URL templates (`github.com` → codeload, `gitlab` → different path, arbitrary `:git/url` self-hosted → no archive endpoint at all), which `git clone` handles uniformly. `:git/url` to a bare/self-hosted repo with no HTTP archive endpoint simply cannot be served this way. So C is more portable for the 90% (GitHub) and strictly broken for the long tail that `git` handles for free. It also can't reuse `git`'s credential/auth handling for private deps. Worth recording as the forward-looking option for the wasm/edge horizon, but it under-serves the general `:git/url` contract today. -> -> **Recommendation.** Adopt **Shape B**, with one explicit carve-out: argv-`git`-subprocess is correct *now* (libgit2 = a C FFI dependency cw has avoided everywhere and a heavier supply-chain surface; a pure-Zig git protocol client is a multi-month yak-shave that buys nothing slice 5 needs; deferring git entirely loses the `:git/url` contract). B's three upgrades over the draft — **content-addressed full-sha cache layout under an overridable `CLJW_HOME`**, **a `GitFetcher` injectable seam so tests don't depend on a real `git`/`file://` repo**, and **post-checkout `rev-parse` sha-verification** — are the parts a finished-form audit would otherwise force as rework, so per F-002 they belong in this slice even though they enlarge the diff. Fold in Shape C's *insight* (not its mechanism) as a Consequences note: the subprocess path is a host-bound capability that the wasm/edge horizon (F-010/F-012) will eventually need an HTTP-client alternative for — record it as a debt row, don't build it now. Finally, correct the ADR's `std.process` audit claim to `std.process.Child` so the "one module" invariant is true rather than a false-positive trigger. - -### Main-loop disposition (within the F-NNN envelope; the DA is advisory, not binding) - -**Adopted from Shape B**: (1) full-sha content-addressed cache under an overridable `CLJW_HOME` (default `~/.cljw`), hard error when both HOME and CLJW_HOME are unset — replaces the draft's 12-char-prefix layout; (2) post-checkout `git rev-parse HEAD == :git/sha` verification, raising `error_catalog` on mismatch; (3) the audit-claim correction (the DA's leading factual catch): the grep is `std.process.run` — the concrete spawn-API string in the code (the DA said `std.process.Child`, but 0.16's convenience is the free function `std.process.run`, which is what actually appears). (4) Shape C's insight recorded as a debt row (HTTP-client tarball fetch for the host-less wasm/edge horizon), not built now. - -**Diverged from Shape B — the `GitFetcher` vtable seam is NOT adopted.** Two finished-form reasons (not cycle-budget — the cache + rev-parse upgrades above *enlarge* the diff and are adopted): (a) a vtable for a **single** implementation, with the only second impl (Shape C HTTP) genuinely deferred to a debt row, is speculative indirection — the Reservation-as-bias smell; the seam's right introduction moment is when the second impl actually lands. (b) A fake fetcher would test *everything except the actual `git` clone/checkout/rev-parse* — the riskiest, newest code in the slice — so it reduces coverage of exactly what is new. A hermetic `file://` bare-repo e2e exercises the **real** subprocess path and is the finished-form test; the DA's "git on CI PATH" concern is handled by a `command -v git` skip-guard (a clear SKIP, not a silent pass) rather than by abstracting the subprocess away. - -## Consequences - -- cw v1 gains its first subprocess + network capability, isolated to one - module. A future audit (the campaign Final Stage) can assert - **`std.process.run`** (the spawn API actually used — not bare `std.process`, not the nonexistent-in-0.16 `std.process.Child.run`) appears only in - `git_fetch.zig`. -- Real-world git-coordinate libs become loadable; the ladder's - `blocked: no deps.edn yet` git rows can be probed. -- A new operational surface (the `$CLJW_HOME/gitlibs` cache, default - `~/.cljw/gitlibs`) that users may need to clear; documented in `docs/works/`. - The layout intentionally diverges from JVM `~/.gitlibs/` (content-addressed - full-sha dirs) — the caches are not interchangeable, so the divergence costs - nothing and buys collision-free multi-version coexistence. -- The subprocess path is a **host-bound** capability (needs `git` + a POSIX - host). The wasm/edge horizon (F-010 / F-012) — cw's actual differentiator — - cannot spawn `git`, so it will eventually need an HTTP-client tarball-fetch - alternative (the DA's Shape C insight). Recorded as a debt row, NOT built now - (it would add a new outbound-HTTP-client capability + per-forge URL knowledge - that `git` handles uniformly). -- Tests that actually clone are network-dependent — the e2e uses a **local - bare git repo** as the `:git/url` (a `file://` URL) so it is hermetic and - offline (no GitHub dependency in the gate), guarded by `command -v git`. - -## Affected files - -- `src/app/deps/git_fetch.zig` (new) — the subprocess + cache. -- `src/app/deps/resolve.zig` — wire the `:git/url` dep arm (currently skipped) - to `git_fetch` + transitive resolve. -- `src/main.zig` — test aggregator entry. -- `test/e2e/phase14_deps_edn.sh` — a hermetic `file://` git-dep case. -- `.dev/debt.yaml` — D-273 note (git coordinate now resolvable). - -## Amendment 1 (2026-06-07) — `:mvn/version` is SKIPPED, not rejected; empty `:paths` defaults to `src/` - -**Context.** The original `:mvn/version` policy was a hard parse-time error -("Maven not supported; use :git/url"). Driving the real-world ladder via -deps.edn git coordinates (the user's "mini deps.edn project replaces corpus -copies" direction) exposed that this is too brittle: **nearly every real lib's -own deps.edn declares `org.clojure/clojure {:mvn/version …}`** — i.e. cw -*itself*. Transitive resolution reads each dep's deps.edn, so a single -`org.clojure/clojure` mvn coord aborted the whole resolution. medley -(`{:deps {org.clojure/clojure {:mvn/version "1.9.0"}}}`) — a pure lib that -loads fine on cljw — was unresolvable for this reason alone. - -**Decision.** A `:mvn/version` dep is **recorded (`Dep.mvn_version`) and skipped -at resolve**, not rejected. Whether the lib is actually satisfied is decided at -**`require` time by namespace availability** (cw's bundled namespaces ∪ -source-resolved `:paths`), not at parse time — the *implicit provided-set* is -cw's bundled namespace surface (clojure.core + string/set/walk/edn/zip/pprint/ -test/data/math + data.json/data.csv/tools.cli + future backfill), which grows -automatically. A skipped non-provided coord is named in a one-line **stderr -summary warning**; `org.clojure/clojure` is suppressed (cw itself — warning -about it is pure noise). No hand-maintained coord→provided table (that would be -the F-013 個別最適化 smell); the only suppressed name is the universal -`org.clojure/clojure`, decided structurally. - -Additionally: a dep deps.edn that declares only `:deps` (no `:paths`) now -defaults `:paths` to `["src"]` (tools.deps convention, applies per-dep) — medley -keeps its source under `src/` with no `:paths` key, so without this it would -contribute nothing and `require` would miss it. - -**Result.** `(require '[medley.core])` via a `:git/url`+`:git/sha` mini -deps.edn project now works end-to-end (clone → skip the clojure-self mvn → -src-default → require). priority-map (zero-mvn deps.edn) already worked; medley -(only-clojure-mvn) is the canonical case this amendment unblocks. The boundary -stays honest: a lib whose *namespace* is neither bundled nor source-laid still -fails at `require` with the existing "Could not locate" error (now naming the -missing ns rather than aborting at parse). - -**Alternatives considered (main-loop, no DA fork — depth-1 amendment within the -F-NNN envelope):** (a) keep hard-reject — rejected: makes deps.edn unusable for -real libs (only zero-mvn libs resolve). (b) skip + a coord→provided allowlist — -rejected: a hand-grown allowlist is the F-013 個別最適化 smell; require-time -namespace resolution already IS the provided-set, implicitly. (c) skip silently, -no warning — rejected: a missing non-clojure dep would surface only as a later -"Could not locate" with no hint it was a deliberately-skipped mvn coord; the -summary warning is cheap UX. - -**Affected files (amendment).** `src/app/deps/parse.zig` (record `mvn_version`, -drop the raise + the now-unused error_catalog import), `src/app/deps/resolve.zig` -(skip + collect skipped coords via an optional out-param; empty-`:paths`→`src` -default), `src/app/cli.zig` (one-line stderr summary warning), -`test/e2e/phase14_deps_edn.sh` (Case 4 reworked: skip+warn; Case 4b: clojure -provided + src default). diff --git a/.dev/decisions/0102_host_interface_ssot.md b/.dev/decisions/0102_host_interface_ssot.md deleted file mode 100644 index 32d322ecf..000000000 --- a/.dev/decisions/0102_host_interface_ssot.md +++ /dev/null @@ -1,350 +0,0 @@ -# ADR-0102 — Host-interface recognition is a closed-set SSOT (`host_interfaces.yaml`) + a mechanical gate, not a hand-grown allowlist - -- **Status**: Proposed → Accepted (2026-06-07) -- **Driven by**: F-013 (just declared) clause 3 — the "個別最適化 entry" must be - closed *structurally*. D-275 slice 1 landed `Object/toString` recognition by - hand-coding `std.mem.eql(name, "Object")` at ~5 sites; slices 3+ - (`clojure.lang.*`) would multiply that scatter library-by-library. The user - asked for the entry to be structurally prevented, not watched. -- **Relates to**: F-013 (definition-derived comprehensive coverage), F-009 - (impl neutrality + `compat_tiers.yaml` as the cross-reference index + G1/G2/G3 - gate precedent), F-002 (finished-form wins), F-011 (behavioural equivalence vs - `clj`), ADR-0059 / AD-003 (no-JVM — `Object`/`clojure.lang.*` are NOT real host - classes), ADR-0066 (deftype macro), ADR-0008 (protocol dispatch), D-275 - (the Object slice this generalises), D-276 (`extend-type Object` as-target). - -## Context - -In `clj`, `deftype`/`reify`/`extend-type` may name host supertypes/interfaces in -the impl-spec position — `Object` (for `toString`/`equals`/`hashCode`) and the -`clojure.lang.*` family (`Seqable`, `ISeq`, `ILookup`, `IDeref`, `Counted`, -`IFn`, `IObj`, `IPersistentMap`, …) that collection libraries implement. cljw has -**no JVM Class** (ADR-0059); these are not cljw protocol Vars and not real -classes — they are **marker names that select a dispatch family** (`Object/toString` -→ the str/print consult; `clojure.lang.Seqable` → cljw's existing `-seq` -protocol). - -D-275 slice 1 made `Object` recognised by hand-coding `std.mem.eql(name, -"Object")` at 5 sites (`macro_transforms.zig` `isHostMarker` + 2 quote-wrap call -sites; `protocol.zig` `hostMarkerCanonicalName` + 2 `eql`/guard sites). This is -the **un-structured recognition entry**: when a real library (the convergence -campaign's gap-discovery technique, F-013 clause 1) hits a new `clojure.lang.*` -interface, the loop is tempted to add `or eql("IDeref")…` at each site — the -recognised set grows **library-by-library** (the ad-hoc "make this lib pass" that -F-013 clause 2 forbids). - -`compat_tiers.yaml` already structurally defends the **Java class** surface (Tier -A/B/C/D + cw-native alternative + G1/G2/G3 gates) so a library hitting a Java -class triggers a *tier lookup*, never a per-library special-case. The -deftype/reify host-interface surface has no equivalent SSOT — only the scatter. - -## Decision - -Apply `compat_tiers.yaml`'s proven closed-set + tier-gate pattern to the -host-interface surface, as a **dedicated SSOT** with a **materialized closed -set** (not re-derived from a mutable clone) + a **two-clause gate**. - -1. **SSOT — `host_interfaces.yaml`** (a new top-level file, sibling to - `compat_tiers.yaml` / `placement.yaml` / `accepted_divergences.yaml`; NOT a - `host_classes` extension — see Alternatives/DA). Each row: - ```yaml - - name: Object # the bare marker name - aliases: [java.lang.Object] # qualified spellings clj source may write - routes_to: object-method-family # protocol name | method-family | feature_not_supported - methods: # per-method wiring status (so a partial family is honest) - toString: { wires_to: "str/print consult", status: wired } - equals: { status: feature_not_supported } - hashCode: { status: feature_not_supported } - derives_from: "java.lang.Object — clj universal supertype; Clojure 1.12 deftype/reify legal supertype" - tier: A - ``` - The **rows ARE the closed set.** Each carries a `derives_from:` note citing the - Clojure interface (+ version) it corresponds to — the "is this in the language" - judgement happens **once, at row-authoring time** (with `~/Documents/OSS/clojure/` - as evidence recorded in the note), NOT re-derived from the pinned clone on every - gate run. This keeps the gate **reproducible + fully in-repo** (the DA's decisive - refinement — see below). - -2. **Single read point** — `src/runtime/host_interface.zig` (Layer 0, neutral) - exposes the recognition + routing surface as a `StaticStringMap` generated/ - checked from `host_interfaces.yaml`. The macro (`isHostMarker`), the canonical- - name resolver, and the method-wiring guard all read from this one module. The 5 - scattered `std.mem.eql("Object")` sites are deleted in favour of it. - -3. **Gate — `scripts/check_host_interface.sh`** (sibling to G1/G2/G3, runs in - `test/run_all.sh`): - - **(i) Set bound**: every name recognised in code (the `host_interface.zig` - table) ⊆ the rows in `host_interfaces.yaml`. A name cannot be recognised - without a row whose `derives_from:` justifies it as language-defined — so - libraries cannot drive growth beyond the definition. - - **(ii) Route soundness** (the real anti-個別最適化 lever): every row whose - `routes_to`/`wires_to` names a cljw protocol must point to an **actually- - modeled** protocol. You cannot recognise a name without a *generic* dispatch - surface behind it — a per-library shim has no slot. Unwired methods carry - `status: feature_not_supported` (explicit transient per ADR-0018), never a - silent drop. - -4. **Two facets of `Object` coexist (refined during implementation, - 2026-06-07)**: the implementing cycle (D-278) found that `java.lang.Object` - has *two distinct facets* — (a) the **deftype/reify supertype marker** (this - ADR's concern → `host_interfaces.yaml`) and (b) the **class reservation** for - `(instance? Object x)` / `(class x)` (the `compat_tiers.yaml host_classes` - `java.lang.Object` row, served via `class_name.zig`). The DA's "migrate the - row out" advice conflated the two; deleting the class-facet reservation would - risk the instance?/class surface. The finished-form resolution: the two facets - live in two SSOTs (which *reinforces* the one-SSOT-per-concern argument) — the - `host_classes` `java.lang.Object` row STAYS (class facet), and - `host_interfaces.yaml` owns the marker facet, with a cross-note on each. - -5. **Sequencing**: this SSOT + gate lands **before** D-275 slices 3+ - (`clojure.lang.*`). After it, adding an interface is "author a row + wire its - protocol", gate-checked — never a new `eql` site. 網羅 the *recognition table* - up-front (cheap, closed); wire impl *incrementally* (each generic + gated) — - the F-013 clause-4 reconciliation of comprehensiveness vs binary/startup cost. - -## Alternatives considered - -Devil's-advocate subagent output (verbatim, fresh context, F-NNN envelope): - -> **F-NNN reachability note (leading entry, per the brief):** No finished-form-clean option here requires violating any F-NNN. All three alternatives below sit inside the envelope. The genuine tension is not F-NNN-vs-clean; it is *which table is the SSOT* and *what the gate's closed set is derived from*. I flag one place where the proposed shape risks a subtle F-009 / F-013 violation (a `routes_to` table that becomes a dispatch map living in `compat_tiers.yaml`) and route around it in the recommendation. -> -> ### Alternative 1 — smallest-diff: single shared `StaticStringMap` in one Zig file, no YAML, no gate -> -> Collapse the 5 scattered `std.mem.eql` sites into one `host_marker.zig` (Layer 0, `runtime/`) exposing `recognize(name) -> ?HostMarker` where `HostMarker` is an enum (`.object`, future `.seqable`, `.ilookup`, …) plus a `routesTo()` method. The macro asks "is this a marker" (non-null), the primitive asks "what does it route to". The alias set (bare + `clojure.lang.`-qualified) lives in the `initComptime` table. -> -> - **Better than proposed:** Zero new YAML schema, zero new gate script, zero cross-file SSOT drift risk. The recognition logic and the routing logic are co-located in the one file a reader greps for — arguably *more* discoverable than a YAML row that points at a Zig table that points at a protocol. It directly kills the "5 scattered `eql` sites" problem, which is the ADR's stated structural target. It is also the most F-009-honest: the marker→family map is implementation, and implementation belongs in neutral `runtime/`, not in the cross-reference index. -> - **What it breaks:** It does **not** structurally close the 個別最適化 entry — the thing F-013 and the user most want. Adding `IDeref` is still "append one enum arm + one table row in one file," which is exactly the library-by-library growth temptation, just centralized. Nothing *mechanically prevents* a future loop from adding `.some_lib_specific_iface` because a lib needed it. There is no gate asserting the set ⊆ the language-defined set, and no gate asserting every route points at a real modeled protocol. It solves the *scatter* smell but not the *unbounded-growth* smell. Against F-013's "raise the system one level" bar, this is a refactor, not a structural defense. -> -> ### Alternative 2 — finished-form-clean: dedicated `host_interfaces.yaml` SSOT + generated Zig table + the two gates, NOT folded into `compat_tiers.yaml` -> -> Same closed-set + dual-gate idea as the proposed shape, but the SSOT is a **new dedicated file** `host_interfaces.yaml` (sibling to `compat_tiers.yaml` / `placement.yaml`), not an extension of `host_classes`. Each row: `{name, aliases: [bare, clojure.lang.X], routes_to, tier, status}`. The Zig `StaticStringMap` is generated/checked from it; `scripts/check_host_interface.sh` enforces (i) set ⊆ language-defined supertypes and (ii) every `routes_to: ` resolves to a modeled cljw protocol. -> -> - **Better than proposed:** Respects the role-split that the project already codified. `compat_tiers.yaml`'s own header (and the `placement.yaml` cross-reference in CLAUDE.md) draws the line explicitly: **`compat_tiers.yaml` = Java/cljw *Class*-level surface; `placement.yaml` = Clojure-ns *var*-level**. Host-supertype *markers* are a third category — they are neither real classes (ADR-0059/AD-003 says `Object`/`clojure.lang.*` are NOT host classes) nor vars. Folding them into `host_classes` forces the schema to host rows that *contradict the schema's own premise* (`host_classes` rows model classes that `(instance? …)` and ctor-resolution treat as real; these markers are explicitly not-real). A dedicated SSOT keeps each index honest about what it indexes — which is the same discipline `accepted_divergences.yaml` and `feature_deps.yaml` already embody (one SSOT per concern). The two gates deliver the F-013 "网罗 from the definition, no per-lib slot" structural close that Alt 1 cannot. -> - **What it breaks:** A fourth top-level YAML SSOT raises the scaffolding surface `audit_scaffolding` must sweep, and adds a `yq`/parser recipe + a new gate to `test/run_all.sh` (G4, after G1/G2/G3). The framework_completion.md rule then obligates this cycle to also run a discovery sweep + retrofit (the `Object` row currently living in `host_classes` must migrate out, and any other latent marker references must be found). That is more work than extending an existing schema — but per F-002 that is not a reason to reject it. -> -> ### Alternative 3 — wildcard: no marker table at all; model the families as real zero-method MARKER protocols in bootstrap `.clj`, recognized through the existing protocol path -> -> The codebase already has the machinery: `addProtocolImpl` records zero-method MARKER protocols (`Sequential`, per the D-190/ADR-0068 comment at protocol.zig:258-262). Push that further — define `Object`, `Seqable`, `ISeq`, `ILookup`, etc. as **actual cljw protocol Vars** in a bootstrap namespace (e.g. `clojure.lang` ns shipped in `lang/clj/`), each with the method set it routes to. Then `deftype`/`reify`/`extend-type` need **no special marker recognition at all** — the names resolve through the ordinary protocol-Var path (the existing `.protocol` arm), and the `.symbol` host-marker arm + all 5 `eql` sites **delete entirely**. `Object/toString` becomes a real protocol method dispatching to the str/print consult. -> -> - **Better than proposed:** This is the deepest "raise the system one level" reading of F-013 — it doesn't *recognize* markers, it *eliminates the marker concept*. The 5 scattered sites don't get consolidated; they cease to exist. Recognition, routing, and the "is it modeled" question all collapse into "does this protocol Var exist and is it satisfied," which is machinery the runtime already has and tests. No new YAML, no new gate, no closed-set-derivation maintenance question at all — the closed set IS "what bootstrap defines," enforced by the fact that an undefined name is an ordinary unresolved-Var error. It is maximally F-009-clean (impl is protocols-over-primitives, the project's blessed Phase-7.2 pattern per ADR-0008 amendment 2). -> - **What it breaks:** Three real problems. (a) `Object` is *not* a protocol in any normal sense — making it a cljw protocol Var means every type "satisfies Object," which needs a universal-satisfy short-circuit that is itself a special case (you've moved the special-casing, not removed it). (b) JVM semantics: in real Clojure these names are interfaces the *runtime* implements on built-in types, not protocols user code extends; a bootstrap-protocol model can drift from `clj` behaviour (F-011 risk) for `(satisfies? clojure.lang.ISeq x)`-style probes that real libraries do issue. (c) It front-loads a large bootstrap-`.clj` + dispatch-wiring effort before the marker set is even enumerated, and offers no gate preventing a future loop from defining a *per-library* marker protocol — the 個別最適化 door is still open, just relabeled as "define a protocol." Without a closed-set gate, Alt 3 regresses on the exact structural property F-013 demands. -> -> ### Recommendation (within F-NNN) -> -> **Adopt Alternative 2 — the dedicated `host_interfaces.yaml` SSOT with the two gates — over the proposed `host_classes` extension.** The proposed shape's *mechanism* (closed-set derivation + dual gate) is correct and is the only candidate that mechanically closes the 個別最適化 door F-013 targets; Alt 1 leaves the door open and Alt 3 both leaves it open and risks F-011 drift on `Object`/`satisfies?`. The single change I'd make to the proposed shape is the **home**: not `compat_tiers.yaml host_classes`. That schema's own documented contract is "Class-level Java surface," and ADR-0059/AD-003 explicitly declares these marker names are *not* host classes — so a `host_classes` row asserting `Object` as a class-level entry is self-contradictory and will confuse `cljw --list-host-classes` (which enumerates real surfaces). A dedicated SSOT mirrors the project's established one-SSOT-per-concern discipline (`accepted_divergences.yaml`, `feature_deps.yaml`, `placement.yaml`) and keeps the `routes_to` dispatch knowledge out of the cross-reference index — important because a `routes_to: ` column folded into `compat_tiers.yaml` would make the index a *dispatch map*, brushing against F-009 (impl detail leaking into the authoritative cross-reference). The extra YAML + G4 gate + framework_completion retrofit is more diff than extending `host_classes`, and per F-002 that is explicitly not a reason to prefer the smaller shape. -> -> ### On the "closed set derived from pinned Clojure source" maintenance question (explicitly assessed) -> -> This is the proposed shape's weakest seam and the ADR should address it head-on. Deriving the gate's allowed set from `~/Documents/OSS/clojure/` couples the gate to a **pinned reference clone** that is read-only and not version-tracked inside this repo. Two concrete hazards: (1) When the pinned Clojure version bumps, the `clojure.lang.*` interface roster can shift (interfaces get added across Clojure versions — e.g. `IKVReduce`, `IReduceInit`, `IDrop` arrived in different releases), so a gate that re-derives "legal supertypes" from whatever clone is on disk is **non-reproducible** and can flip green→red on an unrelated clone update. (2) The clone is a *survey textbook* per `reference_clones.md`, explicitly "do not treat as load-bearing" — wiring a hard gate to it elevates a survey input to a build dependency, which the project's own rules discourage. **The clean fix:** the closed set should be **materialized into `host_interfaces.yaml` itself** (the rows ARE the closed set, each with a `derives_from:`-style note citing which Clojure interface/version it corresponds to — mirroring `accepted_divergences.yaml`'s `derives_from` discipline), and the gate enforces "recognized-in-code ⊆ rows-in-YAML" — a fully in-repo, reproducible check. The "is this in the language" judgement happens *once, by a human/loop at row-authoring time* (with the pinned clone as evidence, recorded in the row's note), not *every gate run against a mutable clone*. That converts the version-coupling maintenance burden into an explicit, reviewable, version-pinned row — and makes the gate deterministic. The second gate ("every `routes_to: ` points at a modeled cljw protocol") is sound as proposed and needs no change; it is the real anti-個別最適化 lever, since it forces "model a generic protocol first" before any name can be recognized. - -### Main-loop disposition (within the F-NNN envelope; the DA is advisory, not binding) - -**Adopted from Alternative 2 (the DA's recommendation):** (1) a **dedicated -`host_interfaces.yaml`** SSOT, NOT a `compat_tiers.yaml host_classes` extension — -the role-split (Class-level vs marker) and the ADR-0059 "not a class" premise make -folding self-contradictory; (2) the **two-clause gate** (set-bound + route- -soundness), with route-soundness as the primary anti-個別最適化 lever; (3) the -DA's decisive refinement — **the closed set is materialized in the YAML rows** -(each with a `derives_from:` note), and the gate checks "recognised-in-code ⊆ -YAML rows", so it is reproducible and in-repo, NOT re-derived from the mutable -pinned clone every run; (4) the legacy `host_classes` `java.lang.Object` row -STAYS as the class facet (refined during implementation — see Decision 4; the -DA's "migrate out" conflated the marker and class facets). - -**Diverged from / not adopted**: Alternative 1 (no gate — leaves the unbounded- -growth door open, fails F-013 clause 3) and Alternative 3 (model-as-real-protocols -— moves rather than removes the `Object` special case via a universal-satisfy -short-circuit, and risks F-011 drift on `(satisfies? clojure.lang.ISeq x)`; also -leaves the growth door open with no closed-set gate). Alt 3's *insight* (route a -marker to a generic protocol surface) is preserved inside Alt 2's route-soundness -gate, without making `Object` itself a protocol Var. - -## Consequences - -- **The 個別最適化 entry is structurally closed**, not watched: a new - `clojure.lang.*` interface requires a `host_interfaces.yaml` row (gate (i): - justified as language-defined) wired to a generic modeled protocol (gate (ii): - no per-library shim slot). Adding `or eql("X")` at a code site is impossible — - there is one read point, and an unrowed name fails the gate. -- **The 5 scattered `eql("Object")` sites collapse to one module.** D-275 slice 1's - hand-coded recognition (the smell noted in its commit) is retired by the - implementing cycle. -- **Coverage trends monotonically (F-013 clause 2):** each interface wired is a - *generic* protocol reusable by every type + user code, so one library's gap-fix - improves the next library's odds — not a one-library shim. -- **`derives_from` makes the closed set reviewable + reproducible.** The gate never - reads the pinned Clojure clone at run time; the clone is evidence cited in the - row, mirroring `accepted_divergences.yaml`'s discipline. -- **New scaffolding surface**: a 4th top-level YAML SSOT + a G4 gate - (`check_host_interface.sh`) + `audit_scaffolding` sweep coverage. The - framework_completion.md "new discipline ⇒ discovery + retrofit" obligation is - satisfied by the scatter consolidation in the implementing cycle (D-278): the - 5 `std.mem.eql("Object")` sites are the complete existing population, all - migrated to the single read point in the same commit (no 2-tier residue). -- **D-276** (`extend-type Object` as-target = default-for-all-types) becomes a - `host_interfaces.yaml` row routed to a default-method dispatch tier, not a - separate `eql` special-case. -- **Binary/startup (F-013 clause 4, D-277)**: 網羅 applies to the *recognition - rows* (cheap — no impl), not to eager impl loading; the eager-vs-lazy modeling - tension stays tracked in D-277, revisited only if the cold-start / size mission - target is threatened. -- **This ADR is the first mechanical instance of F-013 clause 3.** Future hand- - maintained capability allowlists (should any arise) cite this pattern: closed- - set SSOT (keys from the definition) + a gate bounding the set + a gate requiring - a generic route. - -## Revision history - -### Amendment 1 (2026-06-10) — editable / transient collection family (D-286) - -The closed set absorbs the **editable / transient collection interface family** -unchanged (no new ADR; the ADR-0102 mechanism is the right shape): 7 new -`protocol_remap` rows — `IEditableCollection`, `ITransientCollection`, -`ITransientAssociative`, `ITransientMap`, `ITransientSet`, `ITransientVector`, -and `IPersistentSet` (the last as the **D-286b** work-fix). 6 new cljw -`defprotocol`s back the transient targets (cljw had no transient protocol Var; -transients are native Value tags). Driven by `flatland.ordered` (F-010/F-013 -library-discovery): `OrderedSet` declares `clojure.lang.IEditableCollection` -(the prior LOAD blocker at set.clj:95) + `IPersistentSet` with clj-named -methods, and the `Transient*` types declare the `ITransient*` family. - -**Two mechanism changes** beyond the data rows: -1. **Bare aliases route through `protocol_remap`.** A deftype `:import`s - `(clojure.lang IEditableCollection …)` and declares the BARE simple name, so - both the bare and qualified spellings are MARKERS keys → the same row. -2. **Self-targeting recursion guard** (`sectionNeedsRemap`, macro_transforms.zig). - `IPersistentSet`/`ITransient*` remaps self-target (`disjoin`→IPersistentSet/ - `-disjoin`), and the D-283 dual-registration re-emits the original clj name, so - the emitted section carried BOTH `-disjoin` (cljw) and `disjoin` (clj) and - re-routing translated `disjoin`→`-disjoin` forever (stack-overflow segfault). - The guard: a section carrying any already-cljw method *under the interface's own - protocol* is the rewrite's second pass → register directly, don't re-route. A - same-name-but-different-protocol remap (`equiv`→Object, `hasheq`→Object) is NOT - identity → still routes. This generically resolves the D-286 barrier's "harder - part" (a bare name that is BOTH a cljw protocol Var AND needs clj→cljw method - translation). - -**Disposition: LOAD-LEVEL + the D-286b dispatch fix** (DA Alt 2). Recognition lets -the family LOAD; D-286b makes ordered's declared clj-named methods DISPATCH -(verified: `conj`/`seq`/`count` on a deftype `IPersistentSet`). The native -`conj!`/`assoc!`/`persistent!`/`disj!` + `into`/`-editable?` typed_instance consult -is deferred to **D-369**, an honestly off-critical-path follow-up: cljw's `into` -branches on `-editable?` (native-tag-only), so a user editable type takes the plain -`conj` path and NEVER reaches the transient surface — wiring it does nothing for -`(ordered-set …)` today. ordered.set now advances past ALL its editable/transient -interfaces to its next, SEPARATE blocker `print-method` (a clojure.core multimethod -gap, D-370) — NOT claimed "fully working". - -#### Alternatives considered (Devil's-advocate fork, verbatim-reflected) - -The DA fork found a load-bearing fact the survey missed and it reframed the -decision. Recorded faithfully: - -- **Critical finding:** ordered's public api `(ordered-set 1 2 3)` → `(into - empty xs)` → cljw `into` branches on `-editable?` (NOT `(instance? - IEditableCollection)` like JVM); `-editable?` is native-tag-only → returns - false for a `.typed_instance` → the transient path is NEVER entered. So the - transient-dispatch wiring (the proposal's deferred follow-up) is genuinely OFF - the critical path, and the REAL work-blocker is D-286b (clj-named methods - mis-registering), which IS on the critical path. -- **Alt 1 — smallest-diff (LOAD-LEVEL only, defer D-286b too):** lands the family - recognition big-bang; matches the IFn/IObj/Sorted "load-level" precedent. BREAKS: - leaves ordered LOADABLE-BUT-BROKEN and mis-attributes the residual blocker — the - Defer-to-amnesia / false-positive-discharge class (the exact over-claim the D-286 - row already committed once). -- **Alt 2 — finished-form-clean (LOAD-LEVEL + D-286b same cycle; transient consult - a genuine follow-up):** the disposition that makes the driver WORK, not merely - load (F-013/F-011 bar). The clj-name disambiguation is a COMMONISED mechanism - (fixes every bare-named-interface deftype, F-011 clause 1), not a per-lib shim. - Larger diff — but diff size is not a valid reason to prefer smaller (F-002/F-011 - clause 5). **DA recommendation.** -- **Alt 3 — wildcard (wire `-editable?`/`into` to consult IEditableCollection on - typed_instances now):** deepest F-011 reading (user editable types ride the - transient fast-path like JVM); closes the `-editable?` native-tag divergence at - the root. BREAKS: still needs Alt 2's D-286b; front-loads the most machinery + - a silently-wrong-`into` hazard; the `-editable?` divergence is a DISTINCT - finished-form concern deserving its own ADR/AD — bundling it over-scopes this - cycle. Recorded as the D-369 follow-up (transient consult + `-editable?` - typed_instance detection, OFF the ordered critical path). -- **No F-NNN-violating option** was required; all three sit in the envelope. - -### Amendment 2 (2026-06-10) — java.util methods grouped under a clojure.lang section (D-372) - -clj's `IPersistentMap`/`IPersistentSet`/`IPersistentVector` **extend** -`java.util.Map`/`Collection`/`java.lang.Iterable`, so a library may place java -methods (`iterator`/`entrySet`/`keySet`/`values`/`size`/…) UNDER its -`clojure.lang.*` deftype-supertype section instead of a separate `java.util.*` -section (flatland.ordered.map's OrderedMap declares `iterator`/`entrySet` under -its IPersistentMap section). `rewriteProtocolRemap` raised `feature_not_supported` -for any method with no remap target. - -**Decision (DA Alt A): ACCEPT-AND-DROP the java method, load-level.** A new closed -set `host_interface.isJavaUtilMethod(name)` (derived from java.lang.Iterable + -java.util.{Collection,Map,List} read/iteration definitions — F-013); when an -unmapped `protocol_remap`-section method is in it, `rewriteProtocolRemap` drops it -(no registration, no emitted section) instead of raising. This is the **ADR-0103 -`host_inert` accept-don't-dispatch rule at METHOD granularity**: a whole -`java.util.Map` section is already dropped wholesale, and the disposition must be -identical whether the author groups the same methods under a java section or a -clojure.lang section (a cosmetic placement choice must not change cljw behaviour). -A genuinely-unwired *clojure.lang* method (not in the java set) still raises — the -F-013 discovery mechanism survives. **Scope: read/iteration only**; a persistent -type's java mutators (`put`/`add`/`clear`/`remove`) throw on the JVM too, so an -unmapped mutator correctly still raises (closer parity than dropping). - -**Not a `permanent_noop_forbidden` violation**: the drop removes a *registration* -(a later `(.iterator x)` is DECLINED with method-not-found), it does NOT install a -success-masking no-op that *answers* the call. The dropped body -(`(SeqIterator. …)`) is JVM-only/unrunnable in cljw (ADR-0059) and no cljw caller -reaches it. Recorded as **AD-027** (accepted_divergences.yaml) with a pin -(`test/e2e/phase14_java_method_grouping.sh` case 2: a dropped method is declined, -never silently run). `isJavaUtilMethod` carries a `derives_from`-style doc block -citing the java interface definitions (the Alt-A "set lives in code" seam the DA -flagged — closed by the comment; a future gate clause may bound it like gate (i)). - -#### Alternatives considered (Devil's-advocate fork, verbatim-reflected) - -> Fresh-context DA-fork (agent a7ac0278570032884), F-NNN envelope. The spike -> (Opt B implemented + validated: ordered.map parses past the iterator/entrySet -> grouping, next blocker unrelated `Map$Entry`) preceded the fork per ADR-0089. - -**Leading F-NNN entry:** no option requires violating an F-NNN. The real tension -is whether ACCEPT-AND-DROP is sound parity with ADR-0103 or a silent semantic drop -`permanent_noop_forbidden` forbids — assessed head-on: the NG criterion is "user -sees success while semantics are silently dropped." ACCEPT-AND-DROP passes because -(1) the dropped body is JVM-only by construction (constructs `clojure.lang.SeqIterator`, -unrunnable in cljw, no cljw caller reaches it); (2) it is the ADR-0103 rule at finer -granularity, making section-placement irrelevant (a consistency fix, not a -loosening); (3) it is observably honest — a `(.iterator m)` call gets method-not-found, -NOT a plausible wrong value (it DECLINES, it does not no-op-answer). The one guardrail -to keep: the drop stays bounded to the java.util/lang closed set. - -- **Alt A — closed-set drop (implemented):** the minimal *coherent* fix (placement - irrelevant); definition-derived (F-013 — unlocks the 15+ data-structure libs that - group java methods under clojure.lang sections: finger-tree/core.cache/rrb-vector/ - avl/priority-map/int-map/gvec, not just ordered.map); reuses ADR-0103 semantics at - a finer node; unmapped-clojure.lang still raises (discovery survives). Risk: the set - is a code `StaticStringMap`, not gated YAML rows — mild ADR-0102-discipline seam; - bounded because small + definition-derived. **DA recommendation.** -- **Alt B — map java read methods semantically** (entrySet→seq, keySet→keys, - size→count…) so `.size`/`.keySet` dot-calls dispatch. Better: deepest F-011 for the - analogue-having subset. Loses decisively: (1) **re-introduces the `size`/`count` - collision ADR-0103 made impossible by construction** (ordered.map declares `count` - under IPersistentMap; mapping its `size`→`-count` = duplicate `(IPersistentCollection, - -count)` last-wins hazard); (2) `iterator`/`entrySet` have NO sound cljw target - (no Iterable protocol; entrySet returns JVM `Map$Entry` — the very next blocker), so - B cannot even replace A, only add surface on top; (3) "should a no-JVM runtime expose - `.size`/`.keySet`" is its own AD/ADR (AD-003), over-scoping this cycle. Diff size is - NOT why B loses (F-002) — the collision re-introduction + no-sound-target are. B's - insight is a separate follow-up D-NNN. -- **Alt C — wildcard: drop ANY unmapped method (no closed set).** Better: zero - maintenance. **The F-013-clause-3 forbidden open-set posture**: a genuinely-unwired - *cljw* protocol method (a real gap cljw should model) would be silently dropped instead - of surfacing — destroys library-driven discovery (the next lib re-buries it); also masks - clj-rejected typos (F-011 divergence that HIDES errors). The symmetric finding to - ADR-0103's own DA. **Reject.** - -**Recommendation: Alt A as implemented**, scoped read/iteration-only. Fold in: (1) -an AD-NNN with a pin (done — AD-027); (2) a `derives_from` comment on the closed set -(done) [+ a future gate clause is optional]. Reject B (ADR-0103 collision + no sound -target + over-scope) and C (open-set inverts F-013 clause 3). diff --git a/.dev/decisions/0103_no_jvm_host_inert_supertype.md b/.dev/decisions/0103_no_jvm_host_inert_supertype.md deleted file mode 100644 index d189e60a7..000000000 --- a/.dev/decisions/0103_no_jvm_host_inert_supertype.md +++ /dev/null @@ -1,151 +0,0 @@ -# ADR-0103 — no-JVM host-inert deftype/reify supertypes (java.util.Map, java.lang.Iterable): recognised-but-inert - -- **Status**: Proposed → Accepted (2026-06-07) -- **Driven by**: D-281 — clojure.data.priority-map (and collection libs broadly) - declare `java.util.Map` (bare `Map`) + `java.lang.Iterable` as deftype - supertypes for java-interop compat. The D-280e `(require)` advances past every - clojure.lang.* declaration (ADR-0102) and stops at `Map` (priority_map.clj:374) - with name_error. cljw has no JVM (ADR-0059), so these interfaces' methods are - inert here — but the deftype must still LOAD. -- **Relates to**: ADR-0059 (no-JVM), AD-003 (simple class name), ADR-0102 (the - host-interface SSOT + recognition kinds this extends), F-013 (closed-set + - generic-route discipline), F-009, F-002, D-281. - -## Context - -A no-JVM Clojure runtime still has to *load* libraries that implement java -host interfaces for java-interop compatibility — `java.util.Map`, -`java.lang.Iterable`, `java.util.List`, `java.lang.Comparable`, … A collection -deftype frequently declares `java.util.Map` so JVM code can treat it as a map; -priority-map's `Map` section impls `size/isEmpty/containsValue/get/put/remove/ -putAll/clear/keySet/values/entrySet`, and its `Iterable` section impls -`iterator`. - -In cljw these method impls are **inert**: there is no java-interop dispatch -surface — no cljw code ever calls `.size()`/`.get()` on a value as a -`java.util.Map`. They are dead weight that exists only for the JVM. Two facts -make naive handling wrong: - -1. **Collision.** cljw dispatch matches `(protocol, method)` strictly. - priority-map declares BOTH `count` (clojure.lang.IPersistentMap → cljw - IPersistentCollection/`-count`, ADR-0102) AND `size` (java.util.Map). Mapping - `size`→`-count` collides with `count`→`-count`. Same for `get` (java.util.Map) - vs `valAt` (clojure.lang.ILookup → `-lookup`). -2. **No generic surface.** Unlike the clojure.lang.* family (which routes to real - cljw collection protocols), java.util.Map's methods have no meaningful cljw - target — they are JVM-only. - -ADR-0102's three recognition kinds don't fit: `protocol_remap` needs a real -target protocol (none exists); `marker` (zero-method) raises -`feature_not_supported` on a declared method; `method_family` (Object) is for -methods cljw DOES consult. - -## Decision - -Add a fourth recognition kind **`host_inert`** to the ADR-0102 host-interface -SSOT: a recognised supertype whose declared methods are **accepted and recorded -but never dispatched** by cljw (the honest ADR-0059 reading — cljw recognises -java.util.* for load-compat; it does not implement java-interop dispatch). - -1. **Closed set at the INTERFACE level.** `java.util.Map` + `java.lang.Iterable` - are `host_interfaces.yaml` rows with `kind: host_inert`. A new java host - interface cannot be recognised without a row (F-013 closed-set — the G4 gate - still bounds the recognised interface set). This is NOT an open "tolerate any - unknown host supertype" policy (the DA's rejected Alt C — that masks typos and - defeats library-driven discovery, since clj itself rejects unknown interfaces). -2. **Methods accepted-any-inert.** Under a `host_inert` interface, ANY declared - method is accepted and registered under the interface's canonical name (e.g. - `(java.util.Map, size)`), never dispatched. No per-method enumeration: every - method is inert, so a typo'd inert method is harmless (still dead), making the - method-level guardrail valueless here. The macro quote-wraps the supertype - (isMarker→true) so the analyzer never Var-resolves it; the primitive registers - the methods inertly instead of raising. -3. **No collision by construction.** Inert methods register under the interface's - own canonical name (`java.util.Map` / `Iterable`), a namespace cljw never - dispatches — so `size`/`get` cannot shadow `-count`/`-lookup`. -4. **Distinct kind keeps `marker`'s guardrail.** `marker` (Serializable, - MapEquivalence) still raises `feature_not_supported` on a stray method — the - guardrail that catches a misplaced method is intact for zero-method markers. - Only `host_inert` accepts-any. -5. **Recorded as an accepted divergence** (AD-style): cljw recognises java.util.* - host interfaces for load-compat but does not dispatch them (ADR-0059). The G4 - gate's route-soundness skips `host_inert` rows (their methods are inert, not - `wired`). - -## Alternatives considered - -Devil's-advocate subagent output (verbatim, fresh context, F-NNN envelope): - -> Verified: `protocol_remap` rows carry `wires_to_protocol`/`wires_to_method`, the `marker` kind raises `feature_not_supported` on declared methods, and G4 gate (ii) checks every wired method targets a real protocol. The collision facts (count/size, valAt/get) are documented in-file. I have enough to write the analysis. -> -> --- -> -> The proposed shape adds a 4th recognition kind `host_inert`: java.util.Map / java.lang.Iterable become closed-set rows whose declared methods are accepted and registered under a dedicated inert protocol (gate-exempt), never dispatched. Three alternatives within the F-NNN envelope follow. None of the three below violates an F-NNN; the smallest-diff one comes closest to grazing F-013 and is flagged. -> -> ### Alt A — smallest-diff: reuse `marker`, relax it to swallow declared methods -> -> Make the existing `marker` kind tolerate (and discard) any methods declared under a marker interface, instead of raising `feature_not_supported`. Add java.util.Map + java.lang.Iterable as `marker` rows. The deftype's java-section methods are parsed, recognized-as-belonging-to-a-marker, and dropped — no method_table entry, no inert protocol, no new kind. -> -> - **Better than proposed**: zero new recognition kind, zero new gate carve-out, zero inert protocol Var. The G4 route-soundness gate stays exactly as-is for the three existing kinds; markers already have no `wires_to`, so nothing new to exempt. Smallest possible surface. -> - **What it breaks**: it silently overloads the *meaning* of `marker`. Today `marker` means "zero-method supertype" (MapEquivalence, Serializable) and the `feature_not_supported`-on-method behaviour is a real guardrail — it catches a typo'd/misplaced method under a marker. Relaxing it to "discard any methods" deletes that guardrail for *all* markers, not just java.util.Map, so a genuine mistake (a method declared under Serializable) now passes silently. This is the **F-013 ad-hoc-pass risk made worse**: the discard is invisible at the row level (a `marker` row gives no signal that it carries dropped methods), so the ledger can't tell "intentionally inert java.util.Map" from "accidentally swallowed nonsense". It also conflates two genuinely different finished-form facts (zero-method-marker vs has-methods-but-inert) into one kind, which F-009's "thin, honest surfaces" disfavours. -> -> ### Alt B — finished-form-clean: `host_inert` as a first-class kind, but methods register under their *own* method names on a single shared `HostInert` protocol, with an explicit per-row method enumeration -> -> Essentially the proposed shape, hardened: keep `host_inert` as a distinct 4th kind, but require each row to **enumerate every inert method by name** (size/isEmpty/get/put/... for Map; iterator for Iterable) with `status: inert`, register them under one canonical `HostInert` protocol so the method_table stays uniform, and teach G4 gate (ii) a *third* clause: an `inert` method must target the `HostInert` protocol and must NOT collide with any `wired` method's (protocol, method) — making the count/size and valAt/get collisions a **gate error**, not a silent last-wins. -> -> - **Better than proposed**: the proposed shape hand-waves "under a dedicated inert protocol OR under the canonical name with the gate exempting it" — that ambiguity is itself a smell. Alt B picks the inert-protocol arm decisively and makes inertness *enumerated and gate-checked*, so F-013's "definition-derived comprehensive coverage" is satisfied mechanically: the row lists exactly what java.util.Map declares, the gate proves none of it shadows a real dispatch. The collision worry (size vs count) is impossible-by-construction because inert methods live on `HostInert`, never on IPersistentCollection — and the new gate clause asserts that separation rather than trusting it. This is the honest ADR-0059 reading: "cljw recognizes java.util.Map for load-compat; its methods are inert because there is no java-interop dispatch." An AD-NNN row documents the divergence; the enumeration is the corpus. -> - **What it breaks**: it requires per-interface method enumeration, which looks like the "per-library shim" F-013 warns against — except it isn't, because the rows are definition-derived from the *interface* (java.util.Map has a fixed method set), not from priority-map. The real cost: every future host interface that a library declares-with-methods now needs its full method list enumerated in the SSOT before it loads, which is more upfront work than a blanket tolerance. It also bakes a permanent `HostInert` protocol into the runtime that exists purely to host dead entries — a small but real "dead weight" the moving-GC / startup audits will keep tripping over. -> -> ### Alt C — wildcard: no java.util.Map row at all; a general analyzer policy that tolerates unknown host supertypes inertly -> -> Drop java.util.Map/Iterable from the closed set entirely. Instead, give the deftype/reify analyzer a **general no-JVM tolerance**: any supertype head that is (a) not a resolvable cljw protocol Var, (b) not a `host_interfaces.yaml` row, and (c) syntactically a host-class name (qualified, or a known-host bare spelling), has its method section accepted-and-discarded inertly with a one-time warning. No per-interface allowlist; the policy is "cljw has no JVM, so unknown host supertypes and their methods are inert by definition." -> -> - **Better than proposed**: maximally general and zero-maintenance — priority-map loads, and so does the *next* library declaring java.util.List / java.util.Comparator / java.lang.Comparable, with no SSOT edit. It is arguably the most honest expression of ADR-0059: "we don't have a JVM, so we don't model java.* — we tolerate it." It eliminates the closed-set-enumeration burden Alt B carries. -> - **What it breaks**: this is where the tension bites hardest, and it **loses on F-013**. F-013's whole thrust is *closed-set SSOT + a gate requiring every recognized name to route somewhere* — a blanket "tolerate anything host-shaped" policy is the structural opposite: it recognizes an *open* set, so the gate has nothing to check and the ledger has no row to read. Worse, **clj itself rejects unknown interface names** (`ClassNotFoundException` at compile) — so general tolerance is a *behavioural divergence from clj that masks real errors*: a typo'd `clojure.lang.IPersistentMpa` or a genuinely-needed-but-unmodeled interface (one cljw *should* wire) both slide through as "inert", and the gap never surfaces. That directly defeats F-013's "library-driven discovery surfaces gaps" — the gaps stop surfacing. It also collides with F-011's parity intent for the cases cljw *does* support: an interface cljw genuinely models would, if mis-spelled, be silently inerted instead of dispatched. The "syntactically a host-class name" predicate is itself fuzzy (is bare `Map` a host class or a typo'd protocol?), reintroducing the heuristic the closed set was built to remove. -> -> ### Recommendation -> -> Take **Alt B** (the hardened `host_inert`), not the proposed shape as loosely drafted and not Alt C. The proposed shape's "inert protocol OR gate-exempt canonical name" optionality is the part to kill: pick the inert-protocol arm, enumerate the methods per row, and add the G4 gate clause that makes a `(protocol, method)` collision between an `inert` and a `wired` method a hard error. That converts the project's central worry — that `host_inert` is an "ad-hoc pass disguise" — into the opposite: an enumerated, gate-verified, definition-derived row that satisfies F-013's closed-set-plus-routing discipline (it routes to a real, if inert, surface, and the gate proves no shadowing of real dispatch). Alt C is the seductive one because it reads as the purest ADR-0059 statement, but it inverts F-013's open-vs-closed posture and *removes* the discovery mechanism that surfaced this very gap — it would let the next typo'd or genuinely-missing interface rot silently, which is exactly the failure F-013 was declared to prevent; I do not recommend it despite its smaller long-run maintenance. Alt A is rejected outright: relaxing `marker` to swallow methods deletes a live guardrail for unrelated markers and leaves inertness unrecorded at the row level. The collision between `count`/`size` and `valAt`/`get` argues *against* any "map read-only methods onto the real collection protocols and drop duplicates" variant (raised in the brief): last-wins/first-wins dropping is a silent correctness hazard — whichever impl loses is the one priority-map's author wrote intending it to run on a JVM, and dropping it without a gate signal is precisely the unreviewable behaviour F-009/F-013 push against. Inert-on-a-separate-protocol makes the collision structurally impossible and gate-asserted, which is why Alt B dominates. - -### Main-loop disposition (within the F-NNN envelope; the DA is advisory) - -**Adopted: `host_inert` as a distinct 4th kind (the DA's core), with the -INTERFACE set closed/gated but METHODS accepted-any-inert — NOT the DA's -per-method enumeration.** Rationale for the one divergence from Alt B: - -- The DA's distinct-kind decision is adopted (rejects Alt A — `marker` keeps its - stray-method guardrail; rejects Alt C — the interface set stays closed, gate- - bounded, so discovery still surfaces unknown interfaces). -- The DA's per-method ENUMERATION is NOT adopted. Alt B's guardrail (gate every - inert method's name + assert no collision) protects against a typo'd/misplaced - method — but under a `host_inert` interface EVERY method is inert (dead in a - no-JVM runtime), so a typo is harmless (still dead). Enumerating ~20 dead - java.util.Map methods is noise the moving-GC/startup audits trip over (the DA's - own "dead weight" caveat), not signal. The collision the DA gates against is - made impossible-by-construction here by registering inert methods under the - interface's OWN canonical name (`java.util.Map`), never a real protocol — so no - `HostInert` protocol Var and no new gate clause are needed. -- This is NOT a cycle-budget shortcut (F-002 / Cycle-budget-defer smell): the - interface-level closed set + the distinct kind is the finished-form that closes - F-013's actual concern (the recognised INTERFACE set can't grow per-library) - while avoiding the dead-method enumeration that adds no correctness for a fully- - inert interface. If a future host interface MIXES inert + functional methods, - it is NOT `host_inert` — it is `protocol_remap` (functional methods routed) and - the route-soundness gate applies; `host_inert` is reserved for fully-inert - java interfaces. - -## Consequences - -- priority-map's `Map` + `Iterable` declarations load (its java methods inert); - combined with D-282 (IKVReduce) the next `(require)` advances past line 374/391 - to whatever body-level blocker follows (MapEntry type / subseq — separate rows). -- `host_inert` is reserved for FULLY-inert java host interfaces. A host interface - with functional methods routes via `protocol_remap` (gate-enforced real target). -- The recognised INTERFACE set stays closed (G4 gate): a new java interface needs - a `host_interfaces.yaml` row — discovery still surfaces unknowns (F-013). -- An accepted-divergence note (java.util.* recognised-but-inert, derives_from - ADR-0059) records the stance so a user who expects `.size()` java-interop on a - cljw value sees it is by-design absent, not broken. -- Future-facing: the moving-GC / startup audits should treat `host_inert` method - entries as inert leaves (no dispatch reachability) — recorded here so they are - not mistaken for live dispatch surface. diff --git a/.dev/decisions/0104_mutable_deftype_fields.md b/.dev/decisions/0104_mutable_deftype_fields.md deleted file mode 100644 index c41448682..000000000 --- a/.dev/decisions/0104_mutable_deftype_fields.md +++ /dev/null @@ -1,210 +0,0 @@ -# ADR-0104 — Mutable deftype fields (`^:unsynchronized-mutable` / `^:volatile-mutable` + `set!`) - -- Status: Proposed → Accepted (2026-06-07) -- Drives: D-288 (deftype mutable fields), campaign Stage 1.3 ladder rung 6 - (`clojure.tools.reader`'s `StringReader`). -- Related: D-202 (field-in-scope read), ADR-0036 (dual-backend parity), - ADR-0096 (`set!` Var semantics), F-002 / F-004 / F-005 / F-009 / F-011 / - F-012 / F-013. - -## Context - -cljw `deftype` instances are value records whose fields live in a per-instance -`gc.infra`-owned slice (`TypedInstance.field_values_ptr`, -`runtime/type_descriptor.zig`). That slice is **already mutable in place** and -GC-safe under the non-moving mark-sweep collector (no write barrier; each slot -is already traced) — it is simply never written after construction. - -Two real gaps block `(set! field v)` inside a method (the D-288 driver, -`clojure.tools.reader`'s `StringReader`, whose `^:unsynchronized-mutable ^long -s-pos` is mutated via `(set! s-pos (inc s-pos))`): - -1. **Mutability metadata is dropped at field-parse.** `lowerDefType` - (`lang/macro_transforms.zig`) reads only the field symbol's name; the - reader's `^:unsynchronized-mutable` lives on `field_form.meta` and is never - inspected. `FieldEntry` has no mutable flag. -2. **`set!` is Var-only and field READS are stale snapshots.** - `analyzeSetBang` (`eval/analyzer/special_forms.zig`) only resolves dynamic - Vars; a bare field symbol is a method-body local → "Unable to resolve - symbol". Worse, D-202's field-in-scope wraps the body in - `(let* [field (.field this)] body)` — a **value copy**, so even if a write - landed, a later read in the same body would see the stale snapshot. - -Oracle (clj 1.12) fixes the target semantics: - -- In-method `(set! n (inc n))` then `(str n)` reads the **live** updated value - (`#"1" / 2 3` across three calls) — reads of a mutable field MUST hit the - live slot, not a snapshot. -- External `(set! (.n c) 5)` **fails** ("No matching field found: n") — a - deftype mutable field is assignable ONLY inside the type's own methods. -- `(defrecord R [^:unsynchronized-mutable n])` **fails at macro time** - (":volatile-mutable or :unsynchronized-mutable not supported for record - fields"). - -## Decision - -Implement mutable deftype fields with **in-method-only** assignment, matching -clj. The field slice's in-place mutability is reused (no value-rep change). - -1. **Mutability is carried macro→analyzer, not on the descriptor.** - `lowerDefType` inspects each `field_form.meta` for `:unsynchronized-mutable` - / `:volatile-mutable` (a shared `fieldIsMutable` helper); the two keywords - are **unified to one "assignable" flag** while single-threaded (AD-018). - Mutable fields are **rejected on `__defrecord!`** at macro time (clj parity) - via a new `error_catalog` Code `defrecord_mutable_field`. The mutable - field-name set flows into `wrapMethodBodyWithFields` → the `__mut-fields!` - transport form → the analyzer `Scope`. - - A descriptor-level `FieldEntry.mutable` flag is **deferred** (YAGNI / - finished-form = no dead field): the MVP has no runtime consumer — the macro - is the mutability SSOT, the analyzer produces `set_field_node` only for - in-context mutable fields, and `set!` on a non-mutable field falls through to - the existing Var path (its "unresolved" error is acceptable; the precise - clj "Cannot assign to non-mutable" message is a deferred nicety that would - need all-fields-with-flag context). When a runtime consumer arrives - (introspection, the reference-identity equality refinement, or the precise - non-mutable error), add `FieldEntry.mutable` then. - -2. **Writable slot.** `TypedInstance.setField(index, v)` writes - `field_values_ptr[index]`. GC-safe (mark-sweep, slot already traced; no - barrier). The receiver value being written is on the eval stack / rooted by - the caller's frame at the write point (no new root site — `GC-ROOT` audit - not triggered; the write replaces a traced slot in an already-rooted - instance). - -3. **Field-context carried on the analyzer `Scope`.** A new optional - `Scope.mutable_fields: ?*const MutFieldCtx` (the `this` local slot + a - field-name set), chain-inherited exactly like `recur_target`. It is - **established by a minimal internal transport form** the macro emits: - `(rt/__mut-fields! this [mfield…] )`. The transport is necessary - because only the macro layer knows which fields are mutable; the form is to - `mutable_fields` what `loop*` is to `recur_target` (a form whose analyzer - handler pushes an analysis-context frame), not redundant machinery. - -4. **`wrapMethodBodyWithFields`** splits fields: **immutable** fields keep the - existing `(let* [field (.field this)])` value-copy (correct + faster — they - never change); **mutable** fields are NOT bound in the `let*` and the body is - wrapped in `(rt/__mut-fields! this [mfield…] …)`. - -5. **Read** (`analyzeSymbol`): a bare symbol in the `Scope.mutable_fields` - context lowers to the **existing `instance_member` read node** - (`(.field this)`, field-only) — live each eval, reusing the dot-read on BOTH - backends. No new read node. - -6. **Write** (`analyzeSetBang`): a bare symbol in the `mutable_fields` context - builds a new **`set_field_node { target, field_name, value_expr }`** (target - = the `this` local-ref). The field index is resolved at eval from the - receiver's descriptor (reusing `evalInstanceMember`'s field-first resolver). - Non-symbol `set!` targets (the external `(set! (.field obj) v)` form) stay - `feature_not_supported` — clj rejects them too, so this is not a gap - (F-013: comprehensive per the DEFINITION, and the definition is in-method - assignment). - -7. **Dual-backend** (ADR-0036 / F-012): `set_field_node` lands TreeWalk arm + - VM compile arm + `op_set_field` opcode + VM dispatch arm + a `diff_test` - case INCLUDING a read-after-write-in-one-body (`(do (set! n 1) n)` shape) to - lock both backends re-reading the live slot. - -8. **Accepted divergences** (pinned, per `accepted_divergences.md`): - - **AD-017** — `^long` / primitive field type hints are parsed-and-ignored - (NaN-box: every slot is a uniform 8-byte `Value`; cljw does not enforce - clj's "must assign primitive to primitive mutable"). `derives_from` - F-004 / F-005. - - **AD-018** — `:unsynchronized-mutable` ≡ `:volatile-mutable` (both → one - assignable flag) while single-threaded; the cross-thread visibility - distinction is dormant until Phase 15+ concurrency. `derives_from` the - single-thread posture; the original keyword is recoverable from source for - a future differentiation. - -### Stance: a mutable deftype is a reference type - -cljw's value model is otherwise immutable. A mutable deftype is a genuine -**reference type**: equality / hash stay identity-or-user-method (clj excludes -mutable fields from structural equality), and mutating a field breaks no -sharing because deftype instances are never structurally shared / copied -(unlike a persistent collection). GC is unaffected (per-instance slice, already -traced). This is the "mutability in an otherwise-immutable value model" tension -D-288 flagged; the resolution is "reference identity, no structural sharing". - -## Alternatives considered (Devil's-advocate, fresh context) - -> Verbatim from a fresh-context `general-purpose` Devil's-advocate fork -> (CLAUDE.md § ADR-level designs are handled inline). Active F-NNN envelope: -> F-002 / F-004 / F-005 / F-009 / F-011 / F-012 / F-013. - -**Leading finding:** no alternative requires violating an F-NNN. The two -prose decisions (`^long` ignore, keyword unification) must be PINNED as -`AD-NNN` rather than buried — adopted (AD-017 / AD-018). - -**Alt 1 — Smallest-diff: macro-level body rewrite to dot-forms (survey Shape -B).** Keep the analyzer untouched; recursively walk the method body rewriting -bare `field` → `(.field this)` and `(set! field v)` → `(set! (.field this) v)`, -then implement the non-symbol `set!` target as the field write. *Better:* no -new special form / Scope surgery / symbol-read codepath; the general -`(set! (.field obj) v)` it forces into existence is independently clj-real. -*Breaks:* the recursive symbol-rewrite must hand-re-implement shadowing / -quoting / nested-rebinding — the exact work `let*`/the analyzer give for free; -this is the **Smallest-diff bias smell**, and a wrongly-rewritten quoted -`field` is a silent correctness bug. Recommended against on F-002 grounds. - -**Alt 2 — Finished-form-clean (RECOMMENDED): field context on `Scope` + -general field-set, bare-field as sugar.** (2a) Thread mutable-field context -through the `Scope` chain (parallel to `recur_target`) rather than a new -special form. (2b) Implement `(set! (.field obj) v)` generally for any -receiver; bare-field set! is sugar over the same node. *Better:* reuses the -analyzer's existing analysis-context mechanism; one `set_field_node` for both -in-method and external; pins the two AD decisions. *Breaks:* larger analyzer -change (not a cost under F-002). - -> **Main-loop adjustment to Alt 2 (recorded, not deferred):** the oracle shows -> clj REJECTS the external `(set! (.field obj) v)` form for deftype fields -> ("No matching field found"), so Alt 2b's general external receiver -> **over-reaches clj semantics** and is dropped — F-013 mandates comprehensive -> coverage of the DEFINITION (in-method assignment), not a superset clj does -> not have. The `set_field_node` carries a general `target` Node (clean, no -> this-hardcoding) but `analyzeSetBang` PRODUCES it only for in-method bare -> mutable fields. Alt 2a (Scope-carried context) is adopted; the DA's -> "`__mut-fields!` is redundant" critique is answered by the macro-only-knows- -> mutability transport argument (Decision §3) — the Scope CARRIES the context, -> the transport form ESTABLISHES it, mirroring loop*→recur_target. - -**Alt 3 — Wildcard: per-field mutable cell.** Store each mutable field as a -one-slot reference cell; `set!` writes the cell interior. *Better:* makes -mutability first-class in the value model; clean Phase-15 volatile story; no -mid-eval slice-write consideration. *Breaks:* a per-field cell allocation + -a transparent deref-on-every-read clj does not have, diverging from clj's -identity model (F-011); solves a GC/visibility problem that does not exist -(the slot is already traced, mark-sweep needs no barrier). Recommended -against — Phase 15 can branch the slot-write on the recorded keyword instead. - -## Consequences - -- `(deftype C [^:unsynchronized-mutable n] …)` methods can `(set! n v)` / - `(set! n (inc n))` with live reads — `clojure.tools.reader` unblocks (ladder - rung 6). -- `defrecord` with a mutable field now errors at macro time (clj parity) — - previously the hint was silently dropped. -- One new analyzer Node (`set_field_node`) + one new opcode (`op_set_field`); - the dual-backend gate enforces both arms + a diff case in the same commit. -- Two new pinned accepted divergences (AD-017 / AD-018). -- External `(set! (.field obj) v)` stays unsupported (clj-faithful). - -## Affected files - -- `src/runtime/type_descriptor.zig` — `TypedInstance.setField` (write a slot). - (`FieldEntry.mutable` deferred — no MVP consumer.) -- `src/lang/macro_transforms.zig` — `fieldIsMutable` helper; `lowerDefType` - defrecord reject; `wrapMethodBodyWithFields` mutable/immutable split + - `__mut-fields!` wrap. -- `src/eval/analyzer/analyzer.zig` — `Scope.mutable_fields` + - `analyzeSymbol` mutable-field read hook + `__mut-fields!` special form. -- `src/eval/analyzer/special_forms.zig` — `analyzeSetBang` field arm + - `__mut-fields!` handler. -- `src/eval/node.zig` — `set_field_node`. -- `src/eval/backend/tree_walk.zig` — `set_field_node` eval arm. -- `src/eval/backend/vm/opcode.zig` / `vm/compiler.zig` / `vm.zig` — - `op_set_field`. -- `src/lang/diff_test.zig` — parity case (incl. read-after-write-in-one-body). -- `src/runtime/error/catalog.zig` — `defrecord_mutable_field` Code. -- `.dev/accepted_divergences.yaml` — AD-017 / AD-018 + pin tests. -- `test/e2e/` — mutable deftype counter + read-after-write cases. diff --git a/.dev/decisions/0105_java_arrays.md b/.dev/decisions/0105_java_arrays.md deleted file mode 100644 index b4b5c538a..000000000 --- a/.dev/decisions/0105_java_arrays.md +++ /dev/null @@ -1,206 +0,0 @@ -# ADR-0105 — Java arrays (aget / aset / make-array / *-array / alength / aclone) - -- Status: Proposed → Accepted (2026-06-07) -- Drives: D-287 (Java arrays); campaign Stage 1.3 ladder rung 4b - (flatland.ordered `aset`) + rung 6 (clojure.tools.reader `aget`). -- Related: ADR-0104 (TypedInstance mutable slice precedent), ADR-0059 / AD-003 - (no-JVM simple class name), F-002 / F-003 / F-004 / F-005 / F-009 / F-011 / - F-012 / F-013. - -## Context - -cljw has a RESERVED-but-unwired `.array` Value tag (slot 51 D3, -`value/value.zig:100` + `heap_tag.zig:105`) — no struct, no trace fn, no -primitive. `aget` → "Unable to resolve symbol". Real pure-Clojure libraries use -Java arrays as a mutable backing store: - -- **flatland.ordered** (rung 4b): `aset` at set.clj:84 (object-array store). -- **clojure.tools.reader** (rung 6): reader_types.clj:62 `aget`; - `InputStreamReader` (`^"[B"` byte-array + `(aget buf 0)` / `(byte-array 1)` / - `(.read is buf)`), `PushbackReader` (`^"[Ljava.lang.Object;"` object-array). - -## Decision - -Wire the reserved `.array` tag as a **type-erased mutable `[]Value`** array, -implemented as **plain `BuiltinFn` primitives** (no analyzer Node / VM opcode — -both backends free, like every collection op). Representation mirrors the -just-landed ADR-0104 `TypedInstance`: - -```zig -pub const JavaArray = extern struct { - header: HeapHeader, - len: u32, - _pad: [4]u8 = .{0,0,0,0}, - items_ptr: [*]Value, // gc.infra-owned, mutable, traced -}; -``` - -`aset` writes `items_ptr[i] = v` in place (non-moving mark-sweep, slot already -traced — no write barrier, exactly `TypedInstance.setField`). Trace mirrors -`traceTypedInstance`; finaliser frees the `gc.infra` slice. - -### Behavioural fidelity (F-011 — oracle-verified, NOT divergences) - -The element TYPE is erased after construction, but observable clj behaviour is -preserved: - -1. **Per-constructor init defaults** (oracle): `int-array`/`long-array`/ - `make-array ` → `0`; `double-array`/`float-array` → `0.0`; - `boolean-array` → `false`; `char-array` → `\space`; - `object-array`/`make-array ` → `nil`. The constructor fills the - correct default Value, then forgets the type. (A uniform nil-fill would be - an F-011 bug, not licensed by any AD.) -2. **Byte/short/char wrap** (oracle: `(vec (byte-array [1 2 300]))` → - `[1 2 44]`): `byte-array` / `aset-byte` (and short/char) apply the clj - modular wrap before storing the (boxed-int) Value. `aset-int`/`-long`/ - `-double` need no coercion (the f64 tower already holds them); they are - aliases of `aset`. -3. **Identity equality**: `(= (object-array 1) (object-array 1))` → false. - Verified: `equal.zig` already returns `else => false` for distinct `.array` - pointers and the identity fast-path covers equal ones — **no new `.array` - equal arm is added** (adding one would be over-specification). -4. **`aclone` / `to-array`**: copy contents into a fresh array (not - `identical?`). -5. **Multi-dim** `(make-array T d1 d2)` / `(aget a i j)` / `(aset a i j v)`: - nested arrays, looped indices (clj parity). - -### Type erasure is the one accepted divergence (AD-019) - -Array element type HINTS (`^"[B"`, `^"[Ljava.lang.Object;"`, `^ints`, -`^bytes`) parse to advisory `{:tag …}` and do not constrain element values; -`aset` to a "typed" array never raises on element-type mismatch (e.g. a String -into an `int-array`). `derives_from` F-004 (uniform 8-byte Value) + F-005 (no -primitive specialization) + ADR-0059. The `(class (object-array 0))` / -`(class (aget (int-array 3) 0))` simple-name surface is **already AD-003** -(no-JVM simple class name) — array cases are added to AD-003's coverage, NOT -minted as a separate AD. - -### Packed `[]u8` byte-array is DEFERRED to Phase 16 (F-003) - -The Devil's-advocate fork recommended a SEPARATE packed `ByteArray` (`[]u8`) -for `byte-array` (Phase-16 Wasm-linear-memory / binary-I/O / digest can -`@memcpy` it; `bytes?` becomes a distinct tag). That is a real finished-form -candidate — but **whether `byte-array` is the Phase-16 binary vehicle, or a -dedicated `WasmMemory`/`ByteBuffer` host object is (JVM: `byte-array` ≠ -`java.nio.ByteBuffer`), is an unmade Phase-16 structural decision.** Per F-003 -(decision-deferral over decision-seizure on structural plans) + the -principle.md Structural-imagination phase, the packed representation is -**deferred to its owning Phase-16 design**, recorded as a debt row, NOT seized -now on speculation. This is NOT the Cycle-budget-defer smell: the uniform -`[]Value` array with byte-WRAP is **fully F-011-behaviourally-equivalent** -today (packing is a performance/representation choice, not a behaviour gap); -A is the finished form for Phase 13's scope, and the packed rep is Phase 16's -finished-form decision to make with its own context. - -### Surface (F-013 — definition-derived comprehensive, not lib-minimum) - -`aget`, `aset` (+ `aset-int`/`-long`/`-double`/`-float`/`-short`/`-char`/ -`-byte`/`-boolean` aliases, with byte/short/char wrap), `make-array`, -`object-array`, `int-array`, `long-array`, `double-array`, `float-array`, -`short-array`, `byte-array`, `char-array`, `boolean-array`, `to-array`, -`to-array-2d`, `into-array`, `aclone`, `alength`, the `ints`/`longs`/`bytes`/ -`shorts`/`chars`/`floats`/`doubles`/`booleans`/`objects` cast fns; `amap` / -`areduce` as `core.clj` macros over aget/aset/alength. `clojure.core/{array? -is implicit via class}` + a `bytes?`-style predicate returns true for any -array (single erased kind) — until the Phase-16 packed split gives `bytes?` a -distinct tag. - -### Code placement (F-009 / zone) - -New `runtime/collection/java_array.zig` (neutral impl, `keyword: array`) + -`lang/primitive/array.zig` (primitive surface, registered in -`lang/primitive.zig registerAll`); Value glue in `value.zig` (initArray / -asArray), `class_name.zig`, `print.zig` (`#object` / simple form), the trace -table, `gc` finaliser; `amap`/`areduce` in `core.clj`. cw v0's -`src/lang/builtins/array.zig` (617 lines) is read-only prior art — its -`element_type` enum is dropped (it bought nothing; type erasure is cleaner). - -## Alternatives considered (Devil's-advocate, fresh context) - -> Verbatim from a fresh-context `general-purpose` Devil's-advocate fork. Active -> F-NNN envelope: F-002 / F-004 / F-005 / F-009 / F-011 / F-012 / F-013. - -**Leading finding (oracle-forced F-011 corrections, applied to the Decision -above, not alternative-specific):** (1) make-array/*-array init value is NOT -"nil for all" — it is the clj type-default per constructor (0 / 0.0 / false / -\space / nil); a uniform nil-fill is an F-011 bug. (2) `aset-byte`/`byte-array` -DO coerce (`300`→`44` signed-byte wrap) — a bare alias-of-`aset` with no wrap -is a silent F-011 divergence (Silent-default-shift smell). Both corrected. - -**Alt A — Smallest-diff: type-erased `[]Value` JavaArray with correct -per-constructor init + byte wrap (no stored element_type).** Better: the draft -minus the two F-011 bugs; reuses the proven ADR-0104 GC pattern; F-004/F-005 -clean; identity equality free (no new equal arm). Breaks/leaves: defers the -packed-byte question to Phase 16 — legitimate under F-003 only if byte path is -not needed now (tools.reader's `(.read is buf)` writes byte-as-int Values into -`items_ptr`, so `[]Value` works today); risk is a future copy at a Phase-16 -contiguous-`[]u8` boundary. - -**Alt B — Finished-form-clean (DA recommended): type-erased `[]Value` for -object/int/long/double PLUS a separate packed `ByteArray` (`[]u8`) for -byte-array.** Better: Phase-16-correct (contiguous buffer for Wasm memory / -binary I/O / digest, zero box/unbox per byte); honest `bytes?`; byte-wrap -intrinsic. The DA argues F-005 does NOT block it (F-005 forbids numeric-TOWER -specialization; a `[]u8` byte-array is STORAGE, boxed-on-`aget`, analogous to a -UTF-8 String being `[]u8` — F-004 also holds, the Value handle is still 8 -bytes). Breaks/costs: 2 array kinds → 2-arm dispatch on every array op; -asymmetric (only byte packed — why not int/double?); a second tag slot. **DA's -own decisive caveat**: B hinges on a Phase-16 representation question the draft -does not resolve — if Phase 16 uses a dedicated `WasmMemory`/`ByteBuffer` -distinct from clojure `byte-array`, then byte-array has no finished-form binary -consumer and A is finished-form-clean. "The ADR should state which world it -assumes." - -> **Main-loop decision (A, recorded not deferred):** adopt **Alt A** + the two -> F-011 corrections + a Phase-16 deferral debt row for the packed `[]u8` -> byte-array. The A-over-B choice is **F-003 (decision-deferral on structural -> plans)**, NOT cycle-budget: B's packed-byte rep depends on the unmade -> Phase-16 Wasm-memory ↔ byte-array structural decision (the DA's own caveat), -> and the Structural-imagination phase says imagine the horizon, record the gap -> as a debt row at the owning Phase, and DEFER the structural decision to that -> owner. Seizing B now risks building byte-only packing that Phase 16 unwinds -> if it chooses a dedicated buffer type. Crucially A is **fully F-011-equivalent -> today** (byte WRAP, not packing, gives `[1 2 44]`), so this defers a -> performance/representation choice, not a behaviour — exactly what F-003 -> covers. (If this were a behaviour gap, F-011 would forbid the defer.) - -**Alt C — Wildcard: back arrays with the existing transient_vector store.** -Better: zero new GC machinery (reuse transient tracing/rooting). Breaks: -imports HAMT-trie + tail growable machinery to model a fixed flat buffer -(Smallest-diff-bias); transient `assoc!` may return a new transient vs -array `aset`'s in-place-return-val; identity bookkeeping re-introduced; blocks -the Phase-16 packed path. Strictly worse than A on finished-form; recorded for -completeness, not competitive. - -**Cross-cutting fixes adopted from the DA:** no new `equal.zig` `.array` arm -(identity is free); array `(class …)` folds into AD-003 (not a new AD-020); -AD-019 tightened to "type hints advisory + `aset` no type-check" (init-defaults -+ byte-wrap are F-011 obligations, not AD content); F-013 surface widened -(to-array-2d, into-array, the cast fns, multi-dim); plain-primitive decision -confirmed F-012-clean. - -## Consequences - -- flatland.ordered (rung 4b) advances past `aset`; tools.reader (rung 6) past - `aget` — both unblock the array layer (further blockers may remain, e.g. - java.io.InputStream/Closeable for tools.reader's InputStreamReader). -- A new wired Value type (`.array`) + ~25 primitives + amap/areduce macros. All - additive plain primitives — no analyzer Node / VM opcode / dual-backend node. -- AD-019 (type hints advisory) + AD-003 array cases. One Phase-16 deferral debt - row (packed `[]u8` byte-array). - -## Affected files - -- `src/runtime/collection/java_array.zig` (NEW — impl). -- `src/lang/primitive/array.zig` (NEW — primitive surface) + register in - `src/lang/primitive.zig`. -- `src/runtime/value/value.zig` — `initArray` / `asArray` / `JavaArray` glue on - the `.array` tag. -- `src/runtime/gc/…` trace table + finaliser for `.array`. -- `src/runtime/print.zig` + `src/runtime/class_name.zig` — array print + simple - class name (AD-003). -- `src/lang/clj/core.clj` — `amap` / `areduce` macros. -- `.dev/accepted_divergences.yaml` — AD-019 + AD-003 array cases. -- `.dev/debt.yaml` — D-287 discharge + a NEW Phase-16 packed-byte-array - deferral row. -- `test/e2e/` + `src/lang/diff_test.zig` — array cases (incl. AD pins). diff --git a/.dev/decisions/0106_host_instance_java_random.md b/.dev/decisions/0106_host_instance_java_random.md deleted file mode 100644 index 1e0f3c63e..000000000 --- a/.dev/decisions/0106_host_instance_java_random.md +++ /dev/null @@ -1,199 +0,0 @@ -# ADR-0106 — `host_instance` as a general stateful-host-object container; java.util.Random first user - -- Status: Proposed → Accepted (2026-06-07) -- Drives: D-289 (java.util.Random stateful native object); campaign Stage 1.3 - ladder rung 5 (clojure.data.generators → test.check). -- Related: ADR-0104 (TypedInstance mutable slice), ADR-0050 (InteropCall), - ADR-0059/AD-003 (no-JVM class), D-048 (host_instance catch-class markers), - F-002 / F-003 / F-004 / F-005 / F-009 / F-011 / F-012 / F-013. - -## Context - -`(java.util.Random. 42)` fails (`arity_error … expected 0`) — the surface -(`runtime/java/util/Random.zig`) is reservation-only. clojure.data.generators -(rung 5) needs it with **Java-LCG parity** (F-011: a seed must reproduce the -JVM sequence). Java's 48-bit LCG (`0x5DEECE66D`) is confirmed bit-for-bit vs -the clj oracle + a Zig prototype; seed-42 first values: `.nextInt` -`-1170105035`, `.nextLong` `-5025562857975149833`, `.nextDouble` -`0.047939305137387644`, `.nextFloat` `0.9420735`, `.nextBoolean` `false`, -`(.nextInt r 100)` `30`, `.nextGaussian` `1.1419053154730547`. - -Random is the first **stateful native object** (a Value carrying mutable -state), distinct from deftype instances + the (landed) JavaArray. - -**Load-bearing constraint:** there is NO free Value tag — all 64 NaN-box heap -slots are assigned (F-004 64-slot ceiling; `encodeHeapPtr` guards `else => -unreachable`). The only reusable slot is `host_instance` (tag 29), which is -DEAD: no value is ever constructed with it; it is referenced only by PROVISIONAL -D-048 catch-class markers (`class_name.zig:246`, `host_class.zig:222`) that -reserved it for exactly this "host object" purpose. - -## Decision - -Repurpose `host_instance` (tag 29) as a **general stateful-host-object -container** (NOT Random-specific) — because it is the LAST generic-host slot, so -consuming it Random-only would corner every future stateful host object -(SecureRandom, a stateful Matcher, …) with no tag and no shared shape (the -no-free-tags reality makes "solve it later" a guaranteed future layout-ADR). - -```zig -pub const HostInstance = extern struct { - header: HeapHeader, // .host_instance tag, offset 0 - descriptor: *const TypeDescriptor, // the rt.types SURFACE descriptor - state: [4]u64, // inline fixed payload (no alloc, no finaliser) -}; -``` - -1. **`descriptor` is the surface descriptor** registered by `installAll` into - `rt.types` (FQCN `cljw.java.util.Random`) — the SAME descriptor - `resolveJavaSurface` hands the constructor — so the `` path and the - instance-method path share one method_table (no desync). It carries the - instance method_table (`.nextInt`/`.nextLong`/…). - -2. **Dispatch reads the descriptor FROM THE INSTANCE** (mirroring TypedInstance, - NOT the tag-keyed `rt.nativeDescriptor`, which would force all host types to - share one method_table). Add a `.host_instance => decodePtr(*HostInstance) - .descriptor` arm to `receiverDescriptor` (tree_walk.zig) + the VM - `op_method_call` receiver resolution (vm.zig). - -3. **`state` is inline-fixed `[4]u64`** (Random uses seed:u64 + - gaussian-cache:f64 bit-cast + a have-gaussian flag; the 4th word is a free - spare). NOT a `gc.infra` pointer — that would tax every host object with a - finaliser + alloc + indirection for a rare unbounded-state need. A future - big-state host type stores a `gc.infra` pointer in `state[0]` and registers - its own finaliser. - -4. **`(java.util.Random. seed)` / `(java.util.Random.)`** flow through - `constructInstance`'s `` path (special_forms.zig:124-129): the surface - descriptor carries an `` BuiltinFn that allocs a HostInstance with the - descriptor + Java-LCG-seeded state (0-arg seeds from entropy via - `runtime/random.zig`'s existing kernel-seed). Instance methods are BuiltinFns - in the descriptor's method_table, installed at `installAll`. **Java's exact - LCG** for the java.util.Random surface; `clojure.core/rand` keeps DefaultPrng - (no parity need) — the two coexist in `runtime/random.zig`. - -5. **`nextLong` returns a full 64-bit value (~5e18, past i48)** — it MUST box - via `big_int.allocFromI64(rt, v, .long)` (origin `.long` so `integer?` / - `(instance? Long …)` hold), NOT `initInteger` (which lossily floats anything - past i48 — an F-005/F-011 bug the draft's first wiring had). `nextInt` / - `nextInt(bound)` (i32) + `nextFloat` / `nextDouble` (f64) + `nextBoolean` are - range-safe through `initInteger` / `initFloat` / `initBoolean`. - -6. **Dual-backend free** (F-012): the constructor + instance members are - existing InteropCallNode paths — no new analyzer Node. Only the two - receiver-descriptor arms (point 2) are new, and the VM compile path is - unchanged. - -### GC trace hook DEFERRED per F-003 - -Random is a **leaf** (`state` holds only a u64 seed + f64 — no Value heap -refs), so tag 29 ships with **no trace registered** (leaf default), which is -correct for Random. The DA fork recommended adding a per-descriptor -`trace`/`finalise` hook on TypeDescriptor NOW so a future Value-holding -host_instance type cannot silently miss its trace. That hook is a **structural -change for a host type that does not exist yet** — F-003 (decision-deferral on -structural plans) defers it to its owning need. The deferral is made LOUD: (a) a -prominent comment on `HostInstance.state` ("leaf-only; a host type storing a -Value here MUST add a per-descriptor trace hook + register a tag-29 trace -dispatcher — see D-294"), and (b) debt row D-294. This is the F-003-correct -shape: ship Random's correct leaf, defer the future-type structural hook with a -loud guard, not a silent leaf-default on a shared tag. - -### Accepted divergence - -- **AD-020**: a `java.util.Random` (and any host_instance) prints opaquely as - `#` — clj prints `#object[java.util.Random 0x… …]` with a - non-reproducible identity hash. `derives_from` AD-002 (opaque-ref print) + - ADR-0059. `(class (java.util.Random.))` → simple `java.util.Random` is the - existing AD-003 surface (array/class-name), not a new AD. -- `nextGaussian` ships full parity (polar-method cache in `state`), so it is NOT - a divergence. - -## Alternatives considered (Devil's-advocate, fresh context) - -> Verbatim from a fresh-context `general-purpose` DA fork. F-NNN envelope: -> F-002 / F-004 / F-005 / F-009 / F-011 / F-012 / F-013. - -**LEADING FINDING (F-005 bug, applied to the Decision, not alternative-specific):** -`nextLong`'s ~5e18 result must use `big_int.allocFromI64(rt, v, .long)` — -`initInteger` (value.zig:198) lossily floats anything past i48, failing F-005 + -F-011. Only `nextLong` needs the boxed-Long path; pin the oracle long in a corpus. - -**Alt 1 — Smallest-diff: Random-specific struct on tag 29, dispatch via the -tag-keyed `nativeDescriptor(.host_instance)`.** Better: smallest diff; no -per-instance-descriptor arm; 8 fewer bytes/instance. Breaks: consumes the LAST -generic host slot for Random alone — the next stateful host object has no tag -(F-004 forbids a 65th) and must retrofit this exact generic surgery with a live -tag-29 consumer to migrate. "Solve it later" is not viable with zero spare tags; -it guarantees the corner. Rejected on F-002. - -**Alt 2 — Finished-form-clean (RECOMMENDED): general `HostInstance{header, -descriptor, state}` on tag 29, per-instance-descriptor dispatch.** Three -corrections the draft under-stated: (i) `descriptor` MUST be the rt.types -surface descriptor (else `` FQCN-keyed and `.nextInt` instance-keyed read -two tables and desync); (ii) `state` inline-fixed `[4]u64`, NOT a gc.infra -pointer (a pointer forces a finaliser + alloc + indirection on every host -object); (iii) a per-descriptor trace/finalise hook to make leaf-ness explicit -per host type and close the silent-missing-trace hazard when a Value-holding -host type arrives. Better: only shape that leaves the next stateful host object -a home, keeps Random a finaliser-free in-place leaf, and (with iii) structurally -closes the trace hazard. Breaks: larger diff (not an F-002 constraint). - -> **Main-loop adjustments (recorded):** Adopt Alt 2 with corrections (i) + (ii). -> Correction (iii)'s per-descriptor trace/finalise hook is **deferred per F-003** -> (structural change for a not-yet-existing Value-holding host type) — Random is -> a correct leaf today; the hook + a tag-29 trace dispatcher land with the first -> Value-holding host type (D-294 + a loud `HostInstance.state` comment as the -> guard). The DA rated (iii) "mandatory now"; F-003 (a user-declared invariant, -> higher than the DA recommendation) governs: defer the future-type structural -> hook with a loud marker rather than seize it now. The LEADING FINDING -> (nextLong boxed-Long) is applied — a present correctness fix, not deferred. - -**Alt 3 — Wildcard: ride `typed_instance` (tag 30) with a mutable field, no new -tag.** Better: consumes no tag; reuses ADR-0104 setField's GC-safe in-place -mutation + the already-traced field array (solves the Value-holding case for -free). Breaks: the seed round-trips through a heap Value every draw (per-draw -alloc churn vs a bare `u64`); gaussian needs extra fields → a record -masquerading as a Random; semantically lies ("a Random IS a record", but records -are immutable). The finished form of a stateful native leaf is a bare struct, -not a secretly-mutated record. Rejected — but it is the genuine wildcard -(questions whether tag 29 should be spent at all). - -**Probe answers:** general-over-specific (no spare tags → general-now is the only -non-cornering move); state inline N=4 (pointer taxes every host object); D-048 no -collision (Random isn't throwable; markers stay dormant, NOT discharged); -nextLong boxed-Long (leading finding); nextInt(bound)/gaussian/setSeed need -wrapping signed arith (`*%`/`+%`, `bits:u6`) per the prototype; GC leaf hazard -real but deferred per F-003 with the loud guard above. - -## Consequences - -- `(java.util.Random. seed)` + the instance method surface work with JVM-LCG - parity → clojure.data.generators (rung 5) → test.check unblocked (LIVE). -- `host_instance` (tag 29) becomes the general stateful-host-object container; - the next such object reuses it (descriptor-discriminated) — no tag exhaustion. -- New: HostInstance struct + 2 receiver-descriptor arms (both backends) + the - Random surface (`` + 8 methods, Java LCG). No new analyzer Node/opcode. -- `nextLong` boxed as a `.long`-origin BigInt (F-005). AD-020 (opaque print). -- D-294 opened: per-descriptor trace/finalise hook for a future Value-holding - host_instance type (F-003 deferral; Random is leaf, guarded by a loud comment). -- D-048 markers stay dormant (a Random is not throwable) — NOT discharged. - -## Affected files - -- `src/runtime/value/value.zig` + `heap_tag.zig` — `host_instance` (tag 29) - HostInstance struct + initHostInstance/asHostInstance glue. -- `src/runtime/random.zig` — Java-LCG `Random` (seed/next(bits)/nextInt/Long/ - Double/Float/Boolean/Int(bound)/Gaussian/setSeed), alongside DefaultPrng. -- `src/runtime/java/util/Random.zig` — fill the reservation: `` + instance - method_table (BuiltinFns over the LCG impl), via installNativeMethods-style. -- `src/eval/backend/tree_walk.zig` (receiverDescriptor) + `vm.zig` - (op_method_call receiver) — `.host_instance` arm. -- `src/runtime/class_name.zig` / `print.zig` — host_instance simple class name - (AD-003) + opaque print (AD-020). -- `compat_tiers.yaml` — refresh the java.util.Random `methods:` (add nextFloat/ - nextBoolean/nextGaussian). -- `.dev/accepted_divergences.yaml` — AD-020 + pin. -- `.dev/debt.yaml` — D-289 discharge + D-294 (deferred trace hook). -- `test/e2e/` + `test/diff/clj_corpus/` — parity corpus (the oracle seed-42 - values) + e2e. diff --git a/.dev/decisions/0107_pipeline_gate_smoke_authorized.md b/.dev/decisions/0107_pipeline_gate_smoke_authorized.md deleted file mode 100644 index 54342e67e..000000000 --- a/.dev/decisions/0107_pipeline_gate_smoke_authorized.md +++ /dev/null @@ -1,215 +0,0 @@ -# ADR-0107 — Two-tier gate: smoke-authorized per-commit + batched run-alone full gate - -- Status: Proposed → Accepted (2026-06-07) -- Supersedes the per-commit cadence half of `.claude/rules/gate_cadence.md` - (which remains the SSOT once rewritten — see § Rule edit). -- Related: ADR-0005 / ADR-0036 (dual-backend differential oracle), ADR-0024 - (run_all dispatcher), ADR-0049 (orphan/fan hazard), F-002 / F-012; the - `gate_parallel_e2e_timeout` + `premature_gate_notification` memories. - -## Context - -The full Mac gate (`bash test/run_all.sh --serial-e2e`, ~5 min) became the -per-iteration wall-clock bottleneck. The user asked for a way to **develop while -gates queue** (true dev∥gate concurrency). - -A Devil's-advocate fork (fresh context) found the concurrency premise **unsafe**, -and that the gate's cost/risk axis is **unit-vs-e2e, not correctness-vs-perf**: - -- **`zig build test` (both backends) IS the entire F-012 correctness guarantee.** - `src/lang/diff_test.zig` (the dual-backend differential oracle) is `@import`ed - from `src/main.zig`, so every diff case runs inside `zig build test` — which is - timing-independent, contention-tolerant, and fast-ish (two test exes). -- **The 248 e2e shell steps are ~60% of wall-time** and the only part that flakes - under load (`gate_parallel_e2e_timeout`: the -P8 pool blew past 1200 s with NO - concurrent dev; concurrent `zig build` worsens it AND contends the shared - `zig-out/bin/cljw`, the reason `SERIAL_STEPS` exists). Running e2e concurrently - with dev is unsafe, not just slow. -- **Perf/bench steps** (cold_start_threshold, exit_smoke, bench) false-fail under - contention — a boundary/tag ritual, not a per-commit cost. - -So the honest win is **iteration decoupling**, NOT concurrency: shrink the -per-commit check to the fast contention-tolerant core, and move the slow+flaky -e2e shell suite to a batched, run-alone gate. - -## Decision (Alt B — the DA's recommendation) - -A **two-tier gate split on the unit-vs-e2e axis**: - -1. **`run_all.sh --smoke` = the per-commit check (~fast, target ≤ 60 s).** Runs - the contention-tolerant correctness core: - - `zig build test` on BOTH backends (vm + tree_walk) — the **full** F-012 diff - oracle + every unit test (NOT a subset; a subset would weaken the oracle for - no speed gain), - - `zlinter` (Mac, catches deprecated-API rot), - - `build_cljw` (a broken release build must never batch), - - `corpus_regression` (cljw-only clj-parity replay, no network, fast), - - PLUS the changed/new e2e step(s) for the unit at hand (passed to `--smoke`). - On full success it stamps `.dev/.smoke_pass` with the - `scripts/gate_state_hash.sh` content fingerprint. - -2. **`run_all.sh` (full, unchanged) = the batched gate.** Adds the 248 e2e shell - steps + perf/bench. Stamps `.dev/.gate_pass` as today. **Run ALONE** (no - concurrent dev), `--serial-e2e` for determinism (or default -P8 only on a - confirmed-quiet machine). - -3. **Per-commit authorization.** A source-bearing commit is authorized by EITHER - a fresh `.dev/.gate_pass` (full, today's rule) OR a fresh `.dev/.smoke_pass` - (smoke-green for THIS exact content). `scripts/check_gate_cadence.sh` accepts - both fingerprints. - -4. **One batch ceiling, no other counters.** A single `.dev/.gate_cadence` - counter tracks **smoke-only commits since the last full gate**; the hook - **hard-blocks at N=5** (mirrors the existing additive-batch number), forcing a - full run. The hard block IS the mechanism — the counter does no work beyond - its ceiling (the DA's anti-rot finding). NO `.perf_debt` counter: perf is a - Phase-boundary / pre-tag ritual the boundary checklist already enforces, not a - per-commit-counted debt. - -5. **Always commit BEFORE launching the full batched gate** so its reference - point is a real SHA (a red maps to a committed SHA, enabling fix-forward; - never a vanished working-tree state). Trust `SENTINEL-EXIT=0` + - `.gate_pass == gate_state_hash.sh`, NEVER the task-completion notification - (`premature_gate_notification`). - -6. **Red-gate handling records the LAYER.** A full-gate red in `zig build test` - (the oracle/unit layer) is an F-012 divergence: if any later commit built on - the broken module, **revert + re-validate** (a parity break downstream-depended - on cannot be fix-forwarded). A red in a batched e2e-SHELL step with no semantic - dependents → **fix-forward** (next commit is the fix). The full gate is run - ALONE (decision §2), so its reds are trustworthy (not load flakes). - -### NOT adopted (the DA was decisive) - -- **dev ∥ gate concurrency** — unsafe (the e2e -P8 pool false-times-out under - load even alone; concurrent `zig build` worsens it + corrupts the shared - binary). The win is the fast smoke, not parallelism. -- **diff-case subset in smoke** — strictly worse than free; the full oracle is - already in `zig build test`, which smoke runs in full. -- **`.perf_debt` / `.gate_red` async-flag machinery** — over-engineered rot - surface; one ceiling counter + boundary rituals suffice. - -## Miss-window (the honest cost) - -Smoke runs `zig build test ×2` (all unit + the full diff oracle) + the changed -e2e, but NOT all 248 e2e shell steps. So a shared-file change (analyzer/reader) -that breaks an **unrelated, un-unit-testable shell behaviour** (CLI rendering, -exit codes, REPL/nREPL, require-filesystem, http server, GC torture) is caught -only at the batch/boundary gate, ≤ 5 commits later, needing a bisect over that -small range. Mitigations: N ≤ 5; for a **shared-file** diff run the full e2e set -rather than only-changed when convenient; give new shell-only behaviours a -diff/unit case where feasible so the window keeps shrinking. Evaluation-semantic -regressions are NOT in the window — they are caught by the diff oracle / unit -tests inside `zig build test`, which smoke always runs in full. - -## Alternatives considered (Devil's-advocate, fresh context — verbatim) - -> Active constraints: F-012 dual-backend oracle, releasability at Phase/tag -> boundaries, F-002 finished-form, orphan-prevention. LOC/diff size is not one. - -**Headline finding:** the original draft split on the wrong axis -(correctness-vs-perf) and proposed a diff SUBSET in smoke — strictly worse than -free, since the full diff oracle is already inside `zig build test` (fast, -contention-tolerant). The correct smoke is the full `zig build test ×2` + changed -e2e; batch only the slow e2e SHELL suite. This also collapses the multi-counter / -async-flag machinery as over-engineered. - -**Alt A — smallest-diff: keep gate_cadence.md, only add `run_all.sh ---no-e2e-shell`.** The "full gate" the existing hook demands becomes -`zig build test ×2 + lint + build + corpus` (no e2e shell / perf); the 248 e2e + -perf batch via the existing `.gate_cadence` counter + Phase boundaries. *Better:* -near-zero new machinery (reuses hook/counter/fingerprint/`--only`). *Breaks:* the -risky-vs-additive classifier still over-gates some commits; the e2e-shell -miss-window remains. - -**Alt B — finished-form-clean (RECOMMENDED): two-tier split on unit-vs-e2e.** -`--smoke` (zig build test ×2 + lint + build + corpus + changed e2e, ~fast) stamps -`.smoke_pass`; full gate adds e2e shell + perf, stamps `.gate_pass`. Per-commit -authorised by `.smoke_pass`; one e2e-batch ceiling (N=5) + Phase/tag boundary -force full + perf run-alone. No async, no concurrency, no perf counter. *Better:* -keeps 100% of the F-012 oracle per commit; smallest miss-window; two fingerprint -files mirror the existing `.gate_pass` the hook understands. *Breaks:* needs a -changed-e2e→diff mapping (conservatively: run all e2e for shared-file diffs, -only-changed for additive); the small e2e-shell window is non-zero. - -**Alt C — wildcard: post-commit advisory async full gate, no ceiling.** -Per-commit = `zig build test ×2 + lint + build` only; fire a `timeout 1800 … ---serial-e2e` background gate whose red is advisory. *Better:* fastest iteration. -*Breaks:* this is the unsafe concurrency premise — async serial-e2e under -concurrent `zig build` false-reds, and an ignored advisory red is exactly the -2026-05-31 debt-rot. Recommend against (trades a real correctness signal for -speed; violates the F-012 spirit). - -**DA recommendation: Alt B** — finished-form-clean, preserves the dual-backend -oracle in full per-commit, keeps boundary releasability rituals, no orphan-prone -background machinery. Adopted. - -## Risks (from the DA) - -1. **E2e-shell miss-window** (above) — caught at batch/boundary, ≤5-commit - bisect. Mitigate with small N + full-e2e on shared-file diffs. -2. **Oracle-red downstream contamination** — a `zig build test` parity red that - later commits depended on needs revert, not fix-forward. The full gate runs - alone so its reds are real; record the layer (oracle vs e2e-shell). -3. **Counter rot** — only the hard ceiling block enforces a drain; keep one - counter, perf is a boundary ritual (no counter). -4. **Concurrency is unsafe** — any batched gate is `timeout`-wrapped, - `--serial-e2e`, run-alone; never auto-block on a possibly-flaky e2e red. -5. **Premature notification** — trust `SENTINEL-EXIT` + fingerprint, not the - completion notification. - -## Consequences - -- Per-commit iteration drops from ~5 min to the smoke cost (`zig build test ×2` - dominated, ~tens of seconds), keeping the full F-012 oracle every commit. -- The slow + flaky e2e shell suite + perf move to a batched, run-alone gate - (every ≤5 smoke commits + Phase/tag boundaries). -- New: `run_all.sh --smoke` + `.dev/.smoke_pass`; `check_gate_cadence.sh` accepts - smoke authorization + a smoke-batch ceiling. No async/concurrency machinery. - -## Affected files - -- `test/run_all.sh` — `--smoke [e2e-step,…]` mode (run the correctness core + - named e2e; stamp `.dev/.smoke_pass`). -- `scripts/check_gate_cadence.sh` — accept `.dev/.smoke_pass` as commit - authorization; smoke-batch ceiling (block at N forcing a full gate). -- `scripts/gate_state_hash.sh` — reused for the `.smoke_pass` fingerprint. -- `.claude/rules/gate_cadence.md` — **rewrite to this two-tier policy. Rule - edits are permission-blocked for the loop (see memory); this is USER-OWNED.** - Until rewritten, this ADR is the authoritative policy and the scripts enforce - it; the rule text is updated by the user to match. -- `private/notes/` — the broad-reprobe + this ADR's measured smoke timing. - -## Rule edit (user-owned) - -`.claude/rules/gate_cadence.md` is the auto-loaded SSOT summary; rewriting it to -the two-tier policy is required for consistency but is **permission-blocked for -the autonomous loop**. Surfaced to the user. The scripts + this ADR carry the -enforceable policy in the meantime. - -## Revision history - -### 2026-07-21 — CI runs the full gate on push-to-main (not just nightly) - -The batched-full-gate cadence above is a *local* discipline. CI had mirrored it -too literally: `.github/workflows/ci.yml` ran the fast CORE tier (`--smoke`, **no -e2e**) on both PR *and* push-to-main, and the full e2e suite only on the nightly -schedule. So a shared-code change that broke an e2e step (not the changed step's -own smoke) passed the push CI and sat on `main` until the next nightly failed — -observed as the D-316 `def`-meta change breaking `phase14_var_metadata.sh`, which -failed four consecutive nightlies (2026-07-18..21) before being caught. That -made CI the *late* detector and left "the local full gate" and "CI" inconsistent. - -Fix: **push-to-main now runs the FULL gate** (`CLJW_CI_FULL=1` for every event -except `pull_request`), so every landed commit is e2e-verified by exactly the -local full gate (`test/run_all.sh --serial-e2e`). PR stays fast CORE for -iteration (its merge fires a push event that runs the full gate before it -lands). The heavier non-default-backend (tree_walk) parity sweep — a second -ReleaseSafe rebuild, **not** part of the local full gate — moved to its own -`CLJW_CI_PARITY` flag and stays nightly/dispatch-only, so it is an extra backstop -rather than a source of push-vs-local drift. `concurrency: cancel-in-progress` -already collapses a rapid push burst to one completing full run, so the added -per-push cost is bounded. - -Affected: `.github/workflows/ci.yml` (event→flag mapping), `scripts/ci_gate.sh` -(the parity sweep split onto `CLJW_CI_PARITY`). diff --git a/.dev/decisions/0108_clojure_lang_static_surface.md b/.dev/decisions/0108_clojure_lang_static_surface.md deleted file mode 100644 index 3232e3bb7..000000000 --- a/.dev/decisions/0108_clojure_lang_static_surface.md +++ /dev/null @@ -1,246 +0,0 @@ -# 0108 — `clojure.lang.*` host-static surface tree (clojure.lang.Util) - -- **Status**: Accepted -- **Date**: 2026-06-07 -- **Phase**: Phase 14 (post-v0.1.0 coverage) — Convergence Campaign Stage 1.3 (library ladder) -- **Amends**: ADR-0029 (surface layout adds a third tree) -- **Tags**: phase-14, host, interop, clojure-lang, surface, ladder, F-013 - -## Context - -The real-world pure-Clojure library ladder (Stage 1.3) finds that -data-structure libraries drop to `clojure.lang.*` runtime internals for -static helpers: `(clojure.lang.Util/hash x)` (data.finger-tree:405), -`(clojure.lang.Util/equiv …)`, `(clojure.lang.RT/count …)`, etc. A grep of -the official corpus shows ~95 `Util`/`RT` static call sites (Util/equiv 24×, -Util/isInteger 21×, Util/hash 14×, RT/map 12×, Util/hasheq 10×, Util/equals -10×, …). cljw has NO `clojure.lang.*` surface, so these error -`No namespace: 'clojure.lang.Util'`. - -cljw already has the host-surface mechanism (ADR-0029): a `___HOST_EXTENSION` -with `cljw_ns` + a `TypeDescriptor` whose `method_table` maps method names to -`BuiltinFn`s. Static dispatch resolves `(Class/method …)` via -`resolveJavaSurface(head)` → `rt.types.get(head)` then -`rt.types.get("cljw." ++ head)`. A surface registered as `cljw.clojure.lang.Util` -therefore makes `(clojure.lang.Util/method …)` resolve with no resolver change. - -But ADR-0029 established only TWO surface trees — `runtime/java/**` -(`cljw.java.*`) and `runtime/cljw/**` (`cljw.*`). `clojure.lang.*` is the JVM -Clojure runtime's internal namespace: neither `java.*` nor cljw-original. This -ADR adds a third tree. - -## Decision - -1. **A new `src/runtime/clojure/lang/` surface tree** mirrors the JVM package, - registered under `cljw.clojure.lang.*`. First member: `Util.zig` - (`cljw_ns = "cljw.clojure.lang.Util"`). This is the finished-form placement - (DA Alt 2) — an honest package mirror, not the `runtime/java/lang/Util.zig` - category-lie (DA Alt 1) where the `cljw_ns` string and the directory would - contradict each other. - -2. **Scope = the `clojure.lang.Util` class's pure public statics only** - (F-013 definition-derived; the class is a closed, enumerable API). `RT`, - `Numbers`, `APersistentMap` are separate definition-derived big-bang units - (debt rows), NOT crammed in here. The Tier-D statics (`loadWithClass`, - classloader/reflection) are excluded. - -3. **Method mapping (oracle-verified 2026-06-07):** - - | Util static | cljw impl | oracle parity | - |------------------|--------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| - | `equiv(a,b)` | `=` (equal.zig) | ✓ `(equiv 1 1.0)`→false = cljw `=` (both category-strict, F-005) | - | `identical(a,b)` | `identical?` | ✓ `(identical 1 1)`→true (small-int immediates) | - | `compare(a,b)` | `compare` | ✓ nil-least: `(compare nil 1)`→-1, `(compare nil nil)`→0 | - | `isInteger(x)` | `integer?` | ✓ Long/BigInt; **F-005 divergence** on cljw-absent Short/Byte (`(short 5)` is a cljw Long → true; clj false) → AD | - | `hash(x)` | cljw hash | **AD-009** (cljw-native hash; intra-cljw consistency only) | - | `hasheq(x)` | cljw hash | **AD-009** (same) | - | `classOf(x)` | `class` | ✓ `(classOf 5)`→Long | - | `equals(a,b)` | **custom** same-type-and-value | clj `(equals 1 1N)`→false (Java `.equals`, type-sensitive); cljw `.equals`/`=` both →true, so a faithful `equals` needs an explicit same-descriptor-AND-value check, NOT a fn alias | - | `pcequiv(a,b)` | `=` | persistent-collection equiv = `=` | - | `hashCombine` | `hash.zig` Murmur3 combine | AD-009 (intra-cljw) | - | `isPrimitive(c)` | `false` | cljw has no primitive Classes (F-005/ADR-0059) | - - `equiv`'s long/double/char/boolean overloads collapse to the `Object,Object` - form (F-005: no primitive specialization). - -4. **Gate amendments (Alt 2 is incomplete without them — the zone/marker gates - hardcode `runtime/java/*` + `runtime/cljw/*`):** - - `scripts/zone_check.sh`: add `runtime/clojure/*` to the surface whitelist - AND a D2 arm forbidding `runtime/clojure/* → runtime/java/* | runtime/cljw/*` - (the three surface trees all reach the shared neutral impl, never each other). - - `_host_api`: a `runtime/clojure/_host_api.zig` aggregator (or extend the - single `installAll`) lists the new surface. - - `.claude/rules/feature_name_consistency.md`: extend the scan set to - `runtime/clojure/**` + the `Backend:` marker convention — **CARRY-OVER** - (the auto-mode classifier blocks `.claude/rules/*` edits; surface to the - user). Enforcement is the scripts (editable); the `.md` is doc-accuracy. - -## Alternatives considered - -_Devil's-advocate subagent output (fresh context), verbatim:_ - -> **Pre-note on a factual error in the draft brief:** the brief cited AD-006 for -> the cljw-native-hash divergence; the correct anchor is **AD-009** (AD-006 is -> `Double/parseDouble`). Corrected throughout this ADR. -> -> **Alt 1 — Smallest-diff: `Util.zig` under `runtime/java/lang/`.** Better: zero -> new wiring — drops into the existing `installAll` next to `Math.zig`; -> `cljw_ns = "cljw.clojure.lang.Util"` resolves via the existing -> `rt.types.get("cljw." ++ head)` fallback; the zone gate + feature_name gate -> already cover `runtime/java/*` for free. Breaks: category lie — `clojure.lang.*` -> is not `java.*`; a `clojure.lang.Util` file physically in `runtime/java/lang/` -> next to `java.lang.Math` is exactly the smallest-diff convenience F-002 forbids; -> the `cljw_ns` string says `clojure.lang.Util` while the directory says `java/lang` -> — self-contradictory metadata. -> -> **Alt 2 — Finished-form: new `runtime/clojure/lang/` tree + aggregator -> (RECOMMENDED).** Better: honest package mirroring; scales to the inevitable -> `clojure.lang.RT`/`Numbers`/`APersistentMap`; the cljw_ns/directory/FQCN triple -> is internally consistent. Breaks/hidden cost the brief understates: the zone gate -> and consistency gate HARDCODE `runtime/java/*` + `runtime/cljw/*` — a third tree -> is invisible to G1, so `runtime/clojure/lang/Util.zig` could import from -> `runtime/java/**` with no gate complaint, silently violating ADR-0029 D2. So Alt 2 -> is incomplete unless it ALSO amends both gates to admit a third surface root (and -> decides `runtime/clojure/**` may NOT import the other surface trees). This is an -> ADR-0029 amendment, mandatory not optional. Per F-002 the size is irrelevant. -> -> **Alt 3 — Wildcard: flat FQCN-keyed host-static registry, tree-independent.** -> Better: decouples "internal runtime helper class" surfaces from directory layout; -> one registry serves Util/RT/Numbers without three directories. Breaks: a second, -> parallel dispatch mechanism competing with ADR-0029's `TypeDescriptor`/`method_table` -> — a fork in the surface model; loses the per-file `Backend:` marker + the neutral-impl -> zone discipline the file-tree gates enforce. Trades a category-lie for an un-gated -> escape hatch. Rejected. -> -> **Recommendation: Alt 2**, citing F-002 (finished-form wins; the new tree + the two -> gate amendments are the honest shape, LOC is not a constraint). Alt 1's category lie -> is the smallest-diff bias the project forbids; Alt 3 forks the dispatch model. The ADR -> must scope-in the `zone_check.sh` + `feature_name_consistency.md` amendments to -> recognize `runtime/clojure/**` as a third surface root. -> -> **Method-mapping correctness (flag-before-ship):** cljw `=` is category-gated, which -> the brief's draft mappings handle correctly for `equiv` (both category-strict → match) -> but NOT for `equals`: `Util/equals` is Java `.equals` (type-sensitive), -> `(Util/equals 1 1N)`→false, must NOT alias to cljw `=`/`.equals` (both →true) — needs a -> same-type-AND-value check, oracle-pinned. `Util/identical`→`identical?` safe for small -> ints but probe boxed Longs. `Util/isInteger` differs on cljw-absent Short/Byte (F-005) → -> AD. `Util/compare` confirm nil-least. `hashCombine` only matters intra-cljw under AD-009. -> -> **`Util/hash`/`hasheq` under AD-009: acceptable** — AD-009's contract is intra-cljw -> consistency (equal values hash equal — the HAMT key contract); libs call `Util/hash` to -> BUILD their own runtime structures, which is all they need. The one corruption vector is a -> lib shipping a precomputed JVM hash constant and asserting against it — pre-existing AD-009 -> surface, does not block; add a corpus pin `(= (Util/hash :a) (hash :a))` and never claim -> cross-runtime hash stability. -> -> **Scope (F-013): one class (Util) is the correct definition-derived unit** — a real, -> enumerable JVM class with a closed public-static API. NOT "all of clojure.lang statics" -> (RT alone has ~200, mostly Tier-D) — forcing RT+APersistentMap in would be the opposite -> F-013 failure (ad-hoc "make finger-tree pass"). Land Util's complete pure set now, build -> the tree + gate amendments now (next class = pure file-add), record RT/Numbers/APersistentMap -> as separate big-bang debt rows. -> -> **Gate-amendment checklist:** `zone_check.sh` add `runtime/clojure/*` whitelist + a D2 arm -> forbidding `runtime/clojure/* → runtime/java/* | runtime/cljw/*`; -> `feature_name_consistency.md` extend the scan set + Backend-marker convention; `_host_api` -> aggregate the new tree (hand-maintained enumeration must list `Util`). - -## Consequences - -- `(clojure.lang.Util/method …)` resolves for the pure statics; data.finger-tree - advances past :405. Broadly unblocks corpus libs that drop to `Util` internals. -- A third surface tree exists; future `clojure.lang.*` classes are pure file-adds - + an aggregator line + (if a new sub-namespace) a zone whitelist row. -- `Util/hash`/`Util/hasheq` inherit AD-009 (cljw-native hash). A new AD records - the `Util/isInteger` Short/Byte F-005 divergence. `equals` ships faithful - (same-type-and-value), oracle-pinned in the corpus. -- The `feature_name_consistency.md` scan-set update is a tracked carry-over until - the user lands it; the new tree carries the `Backend:` marker regardless. - -## Amendment 1 — APersistentMap / APersistentSet / Murmur3 (D-375, 2026-06-10) - -The "next class = pure file-add" path promised above is exercised. Custom-collection -libraries (flatland.ordered, data.avl, gvec, tech.ml.dataset) call clojure.lang -abstract-collection STATIC hash/equality helpers from their deftype hashCode/hasheq/ -equals bodies. flatland.ordered.map:123 `(hashCode [this] (APersistentMap/mapHash this))` -errored `No namespace: 'APersistentMap'`. Definition-derived closed set (F-013) across -three real clojure.lang classes: - -- **`clojure.lang.APersistentMap`** statics: `mapHash`, `mapHasheq`, `mapEquals`. -- **`clojure.lang.APersistentSet`** static: `setEquals` (JVM has NO `setHash` static — - hashCode/hasheq are instance methods; inventing a `setHash` would be the 個別最適化 trap). -- **`clojure.lang.Murmur3`** statics: `hashOrdered`, `hashUnordered`, `mixCollHash` — - thin wrappers; `hashOrdered`/`hashUnordered` delegate to the existing - `clojure.core/hash-ordered-coll`/`hash-unordered-coll` (the same coll-fold), `mixCollHash` - to `runtime/hash.zig::mixCollHash` (byte-for-byte JVM Murmur3, already verified). - -Wiring per the gate-amendment checklist (all already in place from the base ADR): 3 new -`runtime/clojure/lang/*.zig` surface files mirroring `Util.zig` + 3 `@import` lines in -`_host_api.zig::clojure_surfaces` + a neutral rt+env-taking helper in `runtime/coll_hash.zig` -(iterates a map-like via the `rt.vtable.callFn`→`clojure.core/seq` callback, the confirmed -`java/util/Iterator.zig` pattern, so it works on a native map AND a deftype instance) + -`compat_tiers.yaml` entries. zone_check / resolver unchanged. - -### The one design choice: `mapHash` return value (DA-fork, depth-2) - -cljw collapsed the JVM's two-hash split (additive Object.hashCode vs Murmur hasheq) into a -SINGLE value-hash: `Util/hash` and `Util/hasheq` both route to `equal.valueHash`, and -`(.hashCode native-map)` = `(hash native-map)` = `map.contentHash` (mixCollHash-finalized). -So `mapHash` and `mapHasheq` return the SAME cljw content hash over the seq'd entries -(matching an `=`-equal native map's `.hashCode`), NOT JVM's additive `sum(keyHash ^ valHash)`. -This is the AD-009/F-011 choice: intra-cljw equal-maps-equal-hash beats JVM-bit-parity (which -is unobservable and impossible since cljw hashes strings UTF-8). The neutral helper REUSES -the `entryHash` + order-independent `+%` + single `mixCollHash` fold `map.contentHash` uses -(fast-path native maps to `contentHash` directly; seq-walk for deftype instances). - -### Alternatives considered (Devil's-advocate, fresh context — verbatim) - -> **Fresh-context grounding:** cljw has ONE value-hash — `Util/hash` (Java hashCode) and -> `Util/hasheq` (Clojure hasheq) BOTH route to `equal.valueHash`; `valueHash` of a map → -> `map.contentHash` = `mixCollHash(Σ entryHash(k,v), count)` (order-independent, then -> mix-finalized). The JVM's additive-vs-murmur duality has no cljw counterpart. -> -> **Alt 1 — JVM-shape-additive** (`Σ keyHash ^ valHash`, not finalized): closest to the -> JVM source text. **Fatal:** `(.hashCode native-map)` in cljw IS finalized contentHash, so -> an additive mapHash returns a different integer → `(= (.hashCode om) (.hashCode -> equal-native-map))` is FALSE — the exact invariant the lib wired the call to preserve. -> Ports the unobservable JVM internal shape (AD-009) while breaking the observable collision -> contract (F-011). VIOLATES F-011. -> -> **Alt 2 — cljw-consistent** (return cljw's `contentHash`-equivalent, mix-finalized, over -> the seq'd entries): makes both `(.hashCode om)` and `(hash om)` agree with an equal native -> map within cljw. Honours cljw's "one value-hash" architecture. Risk: the helper MUST reuse -> the identical `entryHash`+`mixCollHash` fold (don't re-derive) or OrderedMap silently -> mis-collides. FULLY COMPLIANT (F-011 observable contract; AD-009; F-002). -> -> **Alt 3 — route the instance through universal `(hash …)` dispatch**: maximal -> commonization. **Fatal circularity:** `valueHash` on a `.typed_instance` hits the -> defrecord-fields hash or the deftype identity bit-hash, NEITHER equal to the native map's -> content hash — and the lib overrides hashCode *because* the default instance hash is wrong -> for a collection, so routing back re-introduces the wrong hash. VIOLATES F-011. -> -> **RECOMMENDATION: Alt 2.** The libs wire `(APersistentMap/mapHash this)` into hashCode for -> ONE observable reason — a custom map collides with an `=`-equal native map inside the -> runtime. cljw collapsed the two-hash split into one finalized `valueHash`/`contentHash`, so -> mapHash must return that same finalized value over the seq'd entries, reusing the native -> fold. Alt 1's JVM-shape fidelity is unobservable (AD-009) and trades against the observable -> contract (Smallest-diff bias). **mapHash and mapHasheq return the SAME value in cljw** — -> forced, not stylistic: cljw has one hash notion (Util/hash and Util/hasheq both → valueHash), -> so a single OrderedMap reporting two different hashes from hashCode vs hasheq would break -> the collision contract on one path. Preserving the JVM additive/murmur distinction here -> would be Reservation-as-bias (obeying the JVM shape because it is "the named algorithm"). - -Main loop adopts **Alt 2** verbatim. - -### Out of scope (filed as debt, NOT this unit) - -- **D-376** — `Murmur3/hashUnencodedChars` (2 sites, data.xml): hashes UTF-16 code units; - cljw `hashString` hashes UTF-8 (WASM/edge choice). Defer. -- **D-377** — cross-impl map-hash consistency: `(hash map)` (= `contentHash`, per-entry - `vh(k)*31+vh(v)`) ≠ `(hash-unordered-coll map)` (= `collHash`, per-entry the map_entry-as- - vector hash), whereas clj keeps them equal. A deftype map whose `hasheq` uses - `hash-unordered-coll` therefore will not `(hash)`-collide with a native map. Plus the - deeper gap that `(hash deftype-inst)` does not consult the deftype's own `hasheq`/IHashEq - impl (it returns the identity bit-hash). Both are a hash-consistency campaign affecting all - map hashes (corpus regen), distinct from adding the static surfaces. mapHash here is - unaffected (it computes contentHash directly). diff --git a/.dev/decisions/0109_host_class_value_resolver.md b/.dev/decisions/0109_host_class_value_resolver.md deleted file mode 100644 index f4722a5e7..000000000 --- a/.dev/decisions/0109_host_class_value_resolver.md +++ /dev/null @@ -1,160 +0,0 @@ -# 0109 — Unified host-class-VALUE resolver (`classDescriptor` + `kind`) - -- **Status**: Accepted -- **Date**: 2026-06-07 -- **Phase**: Phase 14 — Convergence Campaign Stage 1.3 (library ladder) -- **Amends**: ADR-0059 (class values), generalises `exceptionDescriptor` -- **Discharges**: D-293 (host-class-value facet) -- **Tags**: phase-14, interop, host-class, analyzer, isa, F-013 - -## Context - -The ladder repeatedly hits a host CLASS used as a VALUE that cljw fails to -resolve (`Unable to resolve symbol: 'Object'`): `(derive Object root-type)` -(algo.generic), `clojure.lang.IFn` (core.contracts), `java.lang.AssertionError` -(tools.trace), `Integer`/`java.math.BigInteger` (numeric-tower :98/:127). cljw -already RECOGNISES these names in other roles — host_class.zig (Throwable -ENTRIES, for `(catch E e)`), host_interfaces.yaml (deftype/reify supertype -markers), NATIVE_ENTRIES (real cljw types), and the F-005 collapsed-numeric set -— but NOT as resolvable symbol VALUES in `analyzeSymbol`. - -A reverted probe (D-293) showed resolving the collapsed-numerics as OPAQUE -distinct class values makes `(= (type 5) Integer)`/`(instance? Integer 5)` -clj-faithful (false), but is COUPLED to extend-type: `(extend-type Integer P …)` -then CRASHED (`__extend-type!` `@constCast`s the descriptor and mutates its const -`&.{}` method_table). - -## Decision - -A single resolver `classDescriptor(name) → {descriptor, kind}` (generalising -`rt.exceptionDescriptor`), resolving in precedence order -NATIVE_ENTRIES → OPAQUE (F-005 collapsed-numerics) / INERT (recognised-but- -unmodelled host) → throwable (host_class) → marker (host_interface) → -`name_error`, tagged `kind ∈ {:native, :opaque, :inert, :throwable, :marker}`. -`analyzeSymbol` resolves a recognised host-class name to its descriptor VALUE. -Consumers branch on `kind`: - -- **instance? / =**: `:opaque`/`:inert` → no cljw value has the type → false. -- **extend-type / protocol-registration**: `:opaque`/`:inert`/`:throwable`/ - `:marker` (non-deftype descriptors) → **load-only no-op**, NOT a crash. This is - F-011-FAITHFUL, not a forbidden permanent-no-op: clj also never dispatches an - extension on a never-instantiated class, so cljw's no-op is behaviourally - identical (a corpus line pins that clj never dispatches). The `OPAQUE ∩ NATIVE - = {}` comptime assertion keeps the must-implement (`:native`, e.g. Long) case - from ever being confused with the faithful-no-op (`:opaque`) case. -- **isa?**: a `:throwable` descriptor answers via its existing host_class PARENT - links (`(isa? AssertionError Throwable)` → true, faithfully). `Object` is wired - as the UNIVERSAL ROOT: `(isa? Object)` → true (one special case, - NOT a fabricated general hierarchy — needed because algo.generic's `(derive - Object root-type)` is only HALF-unblocked by Object-as-value; its INTENT - requires Object answer as every type's supertype). Only the collapsed-numeric - hierarchy gap (`(isa? Integer Number)` — cljw has no Number node) is an AD; the - `isa?` AD is NARROW, not blanket (a blanket AD would over-diverge, discarding - the Throwable structure cljw already owns). - -Scope = ONE definition-derived unit (the union of the three closed sets), NOT -staged — a half-swept resolver whose `kind` is non-total for some recognised -names is worse than none (consumers can't assume totality; numeric-tower would -stay red while the ledger reads "host-class values resolve"). - -Full JVM class hierarchy (making `(isa? Integer Number)` true) would require -amending ADR-0059 / adding collapsed-numeric-tower nodes — user-owned; excluded -from this envelope (the narrow AD records the gap). - -## Alternatives considered - -_Devil's-advocate subagent output (fresh context), verbatim:_ - -> **Alt 1 — Smallest-diff: keep three recognizers, analyzeSymbol tries each in -> turn (NO unification).** Better: minimal new surface; reuses three already-gated -> SSOTs without a fourth tag space; F-013-derivation trivially preserved. Breaks: -> does NOT cover the collapsed-numeric case (Integer/java.math.BigInteger) — none -> of the three recognizers own those names, so numeric-tower stays red; to fix you -> add a fourth recognizer anyway, and now four recognizers each mint descriptors -> with ad-hoc instance?/=/extend-type semantics, with `kind` re-derived at every -> call site by "which table answered" — the incidental complexity the `kind` field -> deletes. Smallest-diff bias: looks smaller, relocates the kind-dispatch into N -> call sites. -> -> **Alt 2 — Finished-form: ONE classDescriptor(name)→{descriptor,kind} (the D-293 -> design-of-record).** Better: `kind` is the single place "what can this -> class-value do" is answered; instance?/=/extend-type/future cast/bases read one -> tag instead of asking which table minted it; the recognizers' role-overlap -> (Object in host_class AND host_interface) resolves into one precedence. F-013 -> honoured: the resolver's input sets ARE the three closed SSOTs unioned by fixed -> precedence. Breaks: the precedence order becomes load-bearing and must be tested -> (Object is throwable-rooted AND a marker — order says throwable wins; both map to -> load-only no-op so observable behaviour coincides today, but needs a test, not -> just the comptime intersection check). -> -> **Alt 3 — Wildcard: one interned HostClass with a capability BITSET -> (INSTANTIABLE/CATCHABLE/EXTENDABLE_AS_NOOP/IN_HIERARCHY), no categorical kind.** -> Better: directly answers the isa? question — cljw HAS Throwable parent links, so -> `(isa? AssertionError Throwable)` is faithfully answerable TODAY; a blanket isa? -> AD is an OVER-divergence discarding real structure. The IN_HIERARCHY bit lets -> isa? answer what cljw models and AD only what it cannot (`(isa? Integer Number)` -> — no Number node). Bitset is also honest about Object (catchable + extendable + -> hierarchy-root simultaneously; a single enum forces a lossy choice). Breaks: more -> machinery than the ladder demands; 2^4 capability matrix mostly unused → -> dead-bit / Reservation-as-bias risk; widens scope beyond the four blockers. -> -> **Recommendation (per F-002): Alt 2 (single classDescriptor + kind) with Alt 3's -> narrowed-isa? insight absorbed — NOT a blanket AD.** Alt 2 is the finished form -> (the collapsed-numeric set has no existing recognizer, so a fourth source is -> needed regardless; once four sources mint class-values the consumers MUST -> distinguish no-cljw-value / matchable / load-only — intrinsic to the problem; -> encode it as a `kind` field where the data is, not as "which recognizer -> answered"). Alt 1's smaller diff merely relocates kind-dispatch (Smallest-diff -> bias). Alt 3's bitset is premature, but its isa? insight (`:throwable` -> descriptors answer isa? via existing parent links) must be absorbed; blanket AD -> rejected. -> -> **Assessment 2 — `(derive Object x)` only HALF-unblocks algo.generic.** Resolving -> Object to a descriptor VALUE lets `derive` receive its arg without a resolve -> error, but algo.generic's `root-type` pattern then expects `(isa? Object)` -> → true (Object as universal root). Object-as-value unblocks derive's STORAGE but -> NOT its INTENT unless isa? special-cases Object as universal supertype. FLAG: a -> real second requirement on the critical path — the ADR must wire `(isa? -> Object)`→true (one-line universal-root special case) or algo.generic stays -> half-blocked. isa? host semantics are not optional dressing. -> -> **Assessment 3 — extend-type-on-host no-op is FAITHFUL, not a forbidden lie.** -> provisional_marker.md forbids a permanent no-op that pretends to implement -> observable-as-missing behaviour. `(extend-type Integer P …)` extends a class for -> which NO cljw value can exist (F-005); in clj the same extension on a -> never-instantiated class is ALSO dead (no instance → no dispatch). cljw's no-op -> is behaviourally identical → F-011-faithful, passes no-op-forbidden. The -> forbidden case would be no-op'ing `(extend-type Long P …)` (Long HAS values); the -> OPAQUE ∩ NATIVE = {} assertion keeps those apart. Pin with a corpus line showing -> clj also never dispatches (Discipline 1). -> -> **Assessment 4 — blanket isa? AD is an over-divergence.** Throwable parent links -> make `(isa? AssertionError Throwable)` faithfully answerable; ADR-0059's -> no-JVM-hierarchy only truly bites the numeric tower (Integer/Number, no nodes). -> Finished form: isa? answers what host_class parent links model, ADs ONLY the -> collapsed-numeric hierarchy gap. -> -> **Assessment 5 — one definition-derived unit, NOT staged.** The recognised-host- -> class surface is DEFINED by host_class ENTRIES ∪ host_interfaces.yaml ∪ F-005 -> collapsed set — closed and known now. Staging (exceptions→markers→numerics) -> leaves the resolver answering some names and erroring on others for cycles — a -> half-swept surface whose ledger reads "host-class values resolve" while -> numeric-tower stays red (Micro-coverage-grind / half-swept-area smell). Drive all -> three sets through classDescriptor in one push, land the corpus, close it. -> -> No alternative requires violating an F-NNN. The Object-universal-root isa? wiring -> does NOT violate ADR-0059 (one special-case answer, not a fabricated general -> hierarchy). Full JVM hierarchy (`(isa? Integer Number)`→true) is user-owned -> (amend ADR-0059) — excluded; the narrow AD records the numeric-tower isa? gap. - -## Consequences - -- Bare/dotted recognised host-class names resolve as class VALUES; `(derive - Object …)` + Object-root isa? unblock algo.generic; `Integer`/`BigInteger` - value-compares land numeric-tower's :98/:127 (it then chains to deeper - java.math interop — D-302 stays parked on the BigDecimal surface). -- `extend-type` on a host class is a faithful load-only no-op (corpus-pinned). -- A new AD records ONLY the collapsed-numeric `isa?`-hierarchy gap. -- `exceptionDescriptor` → `classDescriptor` rename ripples to its call sites; - `classPrim`/`Util/classOf` (via `class_of.zig`) are unaffected (they take a - value, not a name). diff --git a/.dev/decisions/0110_symbol_value_metadata.md b/.dev/decisions/0110_symbol_value_metadata.md deleted file mode 100644 index db5828dd9..000000000 --- a/.dev/decisions/0110_symbol_value_metadata.md +++ /dev/null @@ -1,181 +0,0 @@ -# ADR-0110 — Symbol value-metadata (`with-meta`/`meta` on symbols) - -- **Status**: Accepted -- **Date**: 2026-06-07 -- **Discharges**: D-304 (the D-075 symbol residual) -- **Amends**: ADR-0095 (the GC membrane rationale — corrects the - symbol/keyword conflation; flips `.symbol` to GcManaged) -- **Cross-refs**: F-002, F-004, F-006, F-011; ADR-0037 D6 (where symbol - meta was deferred); D-239 (var/atom/ns mutable-ref meta — out of scope) - -## Context - -`(with-meta sym m)` / `(meta sym)` errored (`with-meta: cannot attach -metadata to symbol`) because `Symbol{header, ns, name, hash_cache}` carried -no meta field (deferred at ADR-0037 D6 → D-075 → D-304). This blocks the -real-library ladder: `clojure.core.cache` and -`clojure.algo.generic.math-functions` both die attaching meta to a symbol. - -clj oracle (verified 2026-06-07): - -- `(meta 'a)` → `nil`; `(meta (with-meta 'a {:x 1}))` → `{:x 1}`. -- `(= 'a (with-meta 'a {:x 1}))` → **true** — meta is NOT part of symbol - identity; `(= (with-meta 'a {:x 1}) (with-meta 'a {:y 2}))` → true. -- `(hash 'a)` == `(hash (with-meta 'a {:x 1}))` — hash ignores meta. -- `(get {'a 1} (with-meta 'a {:x 1}))` → `1` — map-key consistency. -- `(identical? 'a (with-meta 'a {:x 1}))` → **false** — distinct objects. -- `(with-meta :a {})` → ClassCastException — keyword has no meta (cljw - keeps its type error). - -The load-bearing crux is GC. A with-meta'd symbol carries a GC-managed -`meta` map that must be traced, but `.symbol` is excluded from the GC -membrane (`isGcManaged`, ADR-0095 Alt D) — the single per-tag classifier -that `Value.heapHeader()` consults so every root walk filters -heap-tagged-but-non-GC pointers before `mark()`. - -## Decision - -1. **`Symbol` gains `meta: Value = nil_val`.** Interned symbols leave it - nil (the interner mints them unchanged); only `with-meta` sets it. -2. **`with-meta` on a symbol gc.allocs a FRESH non-interned `Symbol`** - sharing the interned base's `ns`/`name` slices (no dupe → no finaliser, - slices stay interner-owned + alive) + the new `meta`. This is the - collection-meta precedent (`vector.withMeta`) applied to symbols: the - value is collectable transient data, so it lives in the GC layer (F-006), - not pinned. `(meta sym)` reads the field; `vary-meta` rides - `meta`+`with-meta`. -3. **Flip `.symbol` to GcManaged = true** + register a `.symbol` trace that - marks `sym.meta` (the `vector.traceVector` shape — a no-op when meta is - nil). This **corrects** ADR-0095's membrane comment, which wrongly lumped - `symbol`/`keyword` with the genuinely-headerless `var_ref`/`ns`: a - `Symbol` (like `Keyword`) HAS a valid `HeapHeader` at offset 0, so the - "mis-decode to a non-header" OOB never applied to it. Symbols were - excluded only for a *liveness-not-needed* reason (the interner keeps them - alive). The membrane's true meaning is "header-safe + markable"; symbols - are header-safe, so `true` is the honest classification. -4. **Symbol equality + hash become ns+name-structural (meta-ignored).** - `valueEqual` / `keyEqValue` gain a `.symbol` arm (`symbolStructEq`); it - only fires when pointers differ (the rare with-meta case — interned pairs - still hit the identity fast-path). `valueHash` gains a `.symbol` arm using - `hash_cache` (computed from ns+name, identical for the interned base and - its with-meta'd twin) — this closes a latent bug: the prior - `else`-pointer-bits hash would hash `'a` and `(with-meta 'a m)` apart, - breaking `(get {'a 1} (with-meta 'a m))`. -5. **Keyword stays GcManaged = false** — keywords never carry meta (clj - rejects), so they never need marking; the membrane comment is split to - state symbol=markable, keyword=filtered-(liveness-not-needed). - -### Why interned symbols need no `persistent_marks` registration - -Flipping the membrane means live interned symbols (in constant pools) are -now handed to `mark()` each GC: bit set + trace = a nil-meta no-op. Their -mark bit is never cleared (not on `gc.allocations`, not a `persistent_marks` -waypoint). This is **provably inert**: an interned symbol has NO GC child -(meta is structurally always nil — `with-meta` always mints a *non-interned* -symbol), so a stale bit cannot strand anything. `persistent_marks` is -documented for never-swept waypoints *whose trace reaches GC children* -(`Function.closure_bindings`); registering childless interned symbols there -is off-purpose AND adds O(interned-symbols) `clearPersistentMarks` work to -EVERY collect — a common-path perf cost for a rarely-used feature, buying -nothing. The with-meta'd symbol (the only symbol WITH a GC child) is -gc.alloc'd, so sweep clears its bit normally; correctness holds without the -registration. - -## Alternatives considered (Devil's-advocate, fresh-context subagent, verbatim) - -> **Three load-bearing facts (verified against source):** -> **Fact 1** — the ADR-0095 "mis-decode" claim is NOT accurate for symbols. -> `Symbol` is `header: HeapHeader` at offset 0; the `tag_trace_table[112]` -> OOB came from a non-header pointer (`var_ref`/`ns`). The ADR's own Revision -> says symbol/keyword were filtered because they are gpa-interned, never -> swept, "never need marking" — a liveness-not-needed reason conflated with -> the headerless reason. **Fact 2** — the `Runtime.init` guard is -> one-directional (`trace ⇒ GcManaged`), so flipping symbol + registering a -> trace passes; but the guard does not catch the opposite hazard -> (GcManaged-but-never-swept-and-unregistered). **Fact 3** — `heapHeader()` -> is the single funnel; flipping `.symbol` changes every root-walk site at -> once. -> -> **Alt 1 — smallest-diff: gc.alloc the meta-symbol, DON'T flip the -> membrane; root the meta via a per-object `has_meta` header bit read inside -> `heapHeader()`.** Better: interned symbols stay membrane-false → never -> handed to `mark()`, so the per-GC cost + never-cleared-bit question both -> evaporate; only the rare meta-symbol is markable. Breaks: makes -> `heapHeader()` — the hottest funnel — tag-aware with a per-object branch, -> regressing the membrane from "total, closed, tag-indexed" back to -> "tag-indexed plus a hand-coded exception" — the exact allow-list smell -> ADR-0095 Alt D closed. Corrodes the SSOT the project just built. -> -> **Alt 2 — finished-form-clean (recommended): flip the membrane to true, -> register the trace, AND register interned symbols as `persistent_marks` -> waypoints so bit-hygiene is correct by construction (reuse the Class-1 -> `registerPersistentMark`/`clearPersistentMarks` mechanism, F-011).** Better: -> closes the never-cleared-bit concern by reusing the existing mechanism -> instead of a "harmless no-op" hand-wave; honours the project's stated rule -> "never-swept markable object ⇒ registered in persistent_marks" so the next -> person who adds a symbol→GC-child edge is safe. Breaks: every interned -> symbol costs one persistent_marks slot (8 B) + one bit-clear per collect → -> O(interned symbols) growth (ADR-0095 already flagged the analogous Function -> accumulation). Honest finished form is to accept it (symbols are bounded by -> program text, far more benign than per-closure Functions). -> -> **Alt 3 — wildcard: a dedicated `symbol_with_meta` heap tag (burn a Group A -> slot).** Better: total separation — the hot interned path is byte-for-byte -> unchanged; GC cost paid only by rare meta-symbols. Breaks: Group A slots -> 0-15 are all assigned (no free slot → cross-group placement or reshuffle), -> AND it is the severe Cascade smell F-004 warns of — every `v.tag() == -> .symbol` site (reader / analyzer / dispatch / equality / hashing / printing -> / `symbol?` / resolution / quoting) must also test `.symbol_with_meta` or -> silently mis-handle a meta-symbol, with the compiler's `else =>` arms -> swallowing omissions; the `(= 'a (with-meta 'a m))→true` invariant now needs -> a cross-tag equality arm. Trades a bounded localized GC cost for an -> unbounded scattered correctness surface — the inversion of F-002 + F-011. -> Reject. -> -> **Recommendation: Alt 2.** The equality/hash arms the draft adds are -> correct and necessary (the `else` pointer-bits hash in `valueHash` is a -> latent bug for meta-symbols). Strengthen the `Runtime.init` guard with a -> sibling assert ("every never-swept GcManaged object is registered in -> persistent_marks") so the next membrane-flip can't repeat the latent bug. - -**Main-loop disposition (not bound by the DA recommendation within the F-NNN -envelope).** The loop takes the membrane flip + trace (Alt 2's core) but -declines Alt 2's `persistent_marks` registration of interned symbols. The -DA's bit-hygiene concern is real for waypoints *with GC children*; interned -symbols have none (structural — `with-meta` always mints a non-interned -symbol), so their stale bit is provably inert, and registering O(symbols) -childless entries adds common-path `clearPersistentMarks` cost for zero -correctness benefit and mis-uses a trace-the-children mechanism. This is a -**correctness+purpose** argument, not a cycle-budget defer (the chosen path is -not the smaller diff for diff's sake — it is the one that keeps -`persistent_marks` honest to its documented purpose). Alt 1's per-object -`heapHeader` branch is rejected for the SSOT-purity reason the DA gives. The -DA's guard-strengthening suggestion is folded as a code comment on the -membrane (a sibling assert would need to enumerate "never-swept" objects, -which the runtime does not track distinctly — the comment records the -inertness invariant instead). - -## Consequences - -- **Positive**: symbols join the metadata-bearing IObj family; the ladder - advances (core.cache / algo.generic.math-functions). The membrane comment - becomes honest (symbol = header-safe + markable). A latent symbol-hash bug - is closed. No new heap tag, no cascade. -- **Negative**: live interned symbols now incur a trivial `mark()` (bit + - nil-meta trace) per GC where they were filtered before — bounded by live - program text, the irreducible cost of tracing meta-symbols through the - shared tag. `(hash 'sym)` value changes (pointer-bits → ns+name hash); - internal-only (cljw hash is not JVM-bit-identical), no corpus pins it. - -## Affected files - -- `src/runtime/symbol.zig` — `meta` field, `withMeta`, `metaOf`, - `traceSymbol`, `registerGcHooks`. -- `src/runtime/value/heap_tag.zig` — `.symbol` out of the GcManaged-false - arm; comment split. -- `src/runtime/value/value.zig` — `heapHeader` membrane comment. -- `src/runtime/runtime.zig` — `symbol.registerGcHooks()` call. -- `src/lang/primitive/metadata.zig` — `.symbol` arms in `metaFn`/`withMetaFn`. -- `src/runtime/equal.zig` — `.symbol` arms in `valueEqual` / `keyEqValue` / - `valueHash` (`symbolStructEq`). -- `test/e2e/phase14_symbol_metadata.sh` — 14-case smoke. diff --git a/.dev/decisions/0111_deps_run_modes.md b/.dev/decisions/0111_deps_run_modes.md deleted file mode 100644 index dfe89be23..000000000 --- a/.dev/decisions/0111_deps_run_modes.md +++ /dev/null @@ -1,215 +0,0 @@ -# ADR-0111 — deps.edn run modes `-M` / `-X` (in-process clojure.main grammar) - -- **Status**: Proposed → Accepted -- **Date**: 2026-06-07 -- **Discharges**: D-309 (deps.edn run-mode, user idea) -- **Opens**: D-310 (`*command-line-args*` binding — deferred sibling) -- **Cross-refs**: ADR-0101 (+ amendment 1, deps.edn git-fetch / `:mvn`-skip), - F-002 (finished-form), F-009 (feature-impl neutrality — thin Zig surface over - the one bootstrap+eval chain), F-013 (definition-derived capability); - `private/notes/stage14-deps-run-mode-survey.md` - -## Context - -cljw resolved a deps.edn **classpath** only (Stage 1.2 / ADR-0101): `-A:aliases` -merged `:extra-paths`/`:extra-deps`, but `:main-opts` / `:exec-fn` / `:exec-args` -were parsed-skipped and the entry points were `cljw ` / `cljw -e`. A real -user runs an app with `clj -M:alias -m my.ns` or a tool with `clj -X:build -fn`. The user directed cljw to implement these run modes (and to align the -`verified_projects/` convention onto them so future proofs are deps.edn-idiomatic -`-M:verify` rather than a bare `verify.clj` script). - -The survey (`stage14-deps-run-mode-survey.md`) mapped the two nesting grammars -(the launcher `-A`/`-M`/`-X` and the `clojure.main` `-i`/`-e`/`-m`/`file` -mini-grammar) and found three concrete cw v0 gaps to NOT repeat: - -1. v0's `-M` called `(ns/-main)` with **no args** and did not append the user's - trailing args to the alias `:main-opts`. -2. v0 never bound `*command-line-args*` for `-M`/`-X`. -3. v0's `-X` emitted every value as a **quoted string** (`:n 5` → `{:n "5"}`), - losing EDN types. - -## Decision - -1. **In-process combined grammar.** cljw is JVM-less and single-process, so the - launcher + `clojure.main` grammar run in one process (no `exec java - clojure.main` hand-off — the cw v0 shape, kept). New `src/app/deps/run_mode.zig` - owns it; `cli.zig` detects `-M`/`-X`/bare-`-m`, drains the remaining argv as - verbatim run-args, resolves the classpath (+ parsed `cfg`), and dispatches. - -2. **`-M[:aliases] [main-opts…]`** — effective opts = the last selected alias's - `:main-opts` with the user's trailing args **APPENDED** (clj append, fixing - v0 gap 1/3). First-token dispatch: - - `-m ns [args]` → synthesize `(let [m (requiring-resolve 'ns/-main)] (if m (m - "arg"…) (throw (ex-info "Namespace ns has no -main fn" {}))))` — args reach - `-main` as strings; a missing `-main` is a clean message (survey edge 2). - - `-e expr` → eval + print non-nil (the standalone `cljw -e` contract). - - bare `file.clj` → load the script (no result print). - - `-h` → the `-M` usage. - -3. **`-X[:aliases] [ns/fn] [:k v…]`** — `:exec-fn` = a trailing CLI symbol - (overrides) else the alias's; `:exec-args` = the alias map merged under CLI - `:key value` pairs (CLI wins). Synthesize `(let [f (requiring-resolve 'ns/fn)] - (if f (f (merge (quote ALIAS) (quote {CLI}))) (throw …)))`. **Both maps are - `quote`d**: `-X` args are EDN *data*, never evaluated — so a numeric/boolean - value keeps its type (fixing v0 gap 3) and a bare-symbol value stays a literal - instead of being resolved. The result is **not printed** (clj `-X` contract). - -4. **`print_results` on `runner.runSource`.** `-M -m` / file / `-X` pass `false` - (`clojure.main` never prints a `-main`/`:exec-fn` value); `-e` and the plain - `cljw -e`/file paths pass `true`. One bool gates the per-form `printResult`; - the GC pin stays unconditional (forcing the value can still re-enter the VM). - -5. **Bare top-level `-m`** (`cljw -m my.ns a b`, no `-M`) routes through the same - main grammar — clj's launcher resolves it via `-M`, but in-process cljw accepts - it directly (the `-m` token leads the run-args). - -6. **`verified_projects//` convention flips to `-M:verify`.** Each project's - `deps.edn` gains `:paths ["."]` + `:aliases {:verify {:main-opts ["-m" - "verify"]}}`, and `verify.clj` becomes `(ns verify (:require …))` + `(defn - -main [& _] …asserts… (println "OK "))`. `scripts/verify_projects.sh` - runs `cljw -M:verify`, so the sweep exercises the real deps.edn run-mode path a - user invokes, not a bespoke entry. - -### Divergences from clojure.main (cljw is JVM-less) - -- **No second-process hand-off** — the whole grammar is in `cli.zig` + - `run_mode.zig`; the only contract preserved is the alias-`:main-opts`-then-user - -args APPEND. -- **`*command-line-args*` binding (D-310 part-1, landed same session).** A - `(def ^:dynamic *command-line-args* nil)` in core.clj; `run_mode.zig` prepends - `(alter-var-root (var clojure.core/*command-line-args*) (constantly (list …)))` - to every `-M`/`-X` run — a **root-set, not a `binding`**, because cljw evals - form-by-form and a `binding` cannot span a multi-form script. Covers -m / - file-script / -X uniformly (the trailing args; nil for none). Migrates cleanly - to the Alt-3 Clojure grammar (it would set the same root). -- **`-i` / `-r` / `--report` main-opts and `@resource` script paths are - deferred** (D-310) — the survey ranks them medium-value; `-m`/`-e`/file/`-X` - cover the real usage. An unrecognised `-M` main-opt is a clean error, not a - silent script-path fallthrough. - -## Alternatives considered (Devil's-advocate, fresh-context subagent, verbatim) - -> **Verdict up front:** the source-string-synthesis crux is a genuine -> smallest-diff convenience, not the finished-form-clean marshalling. F-009 (one -> eval chain) does *not* mandate source-string assembly — it mandates one -> *bootstrap+eval setup*. Resolving the var and applying it via the existing Zig -> apply surface (`invokeCallable` / `vtable.callFn`) inside that same `rt`/`env` -> is equally F-009-compliant and removes a re-parse + a hand-rolled -> escaping/quoting layer that is the ADR's own stated "correctness crux." None of -> the three alternatives below violates an F-NNN. -> -> **Alt 1 — Smallest-diff: keep synthesis, extract a shared quoting helper.** -> Move `writeStringLiteral` + the `quote`-EDN-splice into a reusable -> `synth_quote.zig` + a round-trip test (argv → literal → reader → equal bytes) + -> a "value with `}`/newline/unicode" corpus. Better: closes the draft's biggest -> latent risk — the splice is `writeAll(kvs[i+1])` *verbatim*, so an argv value of -> `{:a 1` or an unbalanced-paren string is injected raw; only string *literal* -> args are escaped, EDN values are not validated. A reader round-trip test catches -> malformed-EDN injection before it becomes a confusing eval error. Breaks: -> nothing structural — but it entrenches the re-parse (argv read once, serialized -> to text, re-read by the Clojure reader), the very thing F-002's finished form -> would not keep. Not recommended (cycle-minimizing). -> -> **Alt 2 — Finished-form-clean (RECOMMENDED): resolve + apply in Zig, no -> re-parse.** Add `runner.runEntryFn(… ns_sym, fn_name, arg_values: []const Value, -> print)` sharing `runSource`'s exact setup (refactor it into a `prepare()` -> returning the bootstrapped `rt`+`env` — F-009's one chain becomes literally one -> function). `run_mode.zig` then: `-M -m` → `requiring-resolve` the `ns/-main` var -> as a Value, build args as `[]const Value` of cljw string Values (no escaping — -> they are Values), `invokeCallable`. `-X` → parse CLI `:k v` with the existing -> reader into Values, `merge` onto the alias `exec-args` (already a `Form` in -> `cfg`!), apply. No `quote`, no string assembly. Better: (a) argv EDN read once; -> (b) deletes the entire escaping/quoting surface the ADR flags as the -> "correctness crux"; (c) the `exec-args` alias map is already parsed — re- -> serializing via `formatPrStr` only to re-read is pure waste removed; (d) errors -> point at the real fn/args, not a `<-X>` synthetic label; (e) the `-main`-missing -> guard moves into Zig `error_catalog.raise` (the SSOT per `error_catalog_only.md`; -> the draft's synthesized `ex-info` bypasses it). Breaks: factor `runSource` setup -> into `prepare()` (low ripple); needs `requiring-resolve` Zig-callable (it is, a -> clojure.core var); larger diff than Alt 1 — recommended anyway per F-002. -> -> **Alt 3 — Wildcard: a real `*command-line-args*` + a `clojure.main`-shaped -> Clojure entry.** Bind `*command-line-args*` now (the ADR defers it to D-310) and -> express the whole `-M` grammar as a small `clojure.main`-shaped fn in -> `clojure.core` (or `cljw.run`), invoked from Zig with the argv vector as Values. -> The Zig side shrinks to "resolve `cljw.run/main` + apply argv"; the grammar -> (`-m`/`-e`/`-h`/file) lives in Clojure, reader-checkable + `.clj`-unit-testable. -> Better: the grammar becomes data-driven Clojure (definition-derived per F-013 — -> one general mechanism, not Zig per-flag branches); discharges D-310 in one -> stroke; matches how real clojure.main *is* Clojure — most finished-form-aligned -> with "cljw is a Clojure runtime." Breaks: bootstrap ordering — `cljw.run` must be -> loadable before classpath/user-ns resolution (a real sequencing question); -> biggest blast radius; risks pulling D-310 + a new ns into one cycle. A -> reasonable *next* step after Alt 2, not instead of it. -> -> **On the sub-questions.** `print_results` bool: minor smell, not Alt-level -> surgery; Alt 2 makes it a property of the entry (`runEntryFn` is inherently -> `print=false`) rather than a threaded param. `quote`d-map for `-X`: correct -> semantically, but exists *only because* the data was flattened to text and must -> be protected from re-evaluation — in Alt 2 the data never becomes text, so -> `quote` is unnecessary; the need for it is a symptom of the re-parse. -> -> **Recommendation: Alt 2.** Finished-form-clean within the full F-NNN envelope -> (F-009 one-chain preserved by `prepare()`, F-013 untouched, JVM-less untouched, -> error_catalog SSOT restored). Deletes the self-identified correctness crux rather -> than testing around it. Larger diff is not a reason to downgrade (F-002). Alt 3 -> is the further-horizon ideal but couples in D-310 + bootstrap-ordering risk; -> sequence it after Alt 2. - -**Main-loop disposition (not bound by the DA recommendation within the F-NNN -envelope).** The loop keeps the **source-synthesis** approach and declines Alt 2, -on a *substantive* (not cycle-budget) ground: **`clojure.main` is itself Clojure -code** — `-m` is `(apply (ns-resolve (doto (symbol ns) require) '-main) args)`, -`-X` is `(the-fn merged-map)`, both *evaluated as Clojure*. For a from-scratch -Clojure runtime, the finished form keeps that orchestration **in Clojure-eval**, -with a thin Zig wrapper handing a Clojure form to the one chain — which is exactly -**F-009** (surfaces are thin wrappers; the semantics live in the neutral/Clojure -layer). Alt 2 pulls the resolve/apply/merge orchestration *into Zig*, thickening -the surface against F-009. The DA itself ranks **Alt 3 (grammar in Clojure)** as -"most finished-form-aligned with 'cljw is a Clojure runtime'" — confirming the -true finished form is Clojure-level, not Zig-Value-marshalling. Alt 3 is -bootstrap-ordering-gated (the grammar ns must load before user libs) and is -**scoped to D-310** (which also owns `*command-line-args*`); the current synthesis -is the bootstrap-safe intermediate that lands the capability now and migrates -cleanly to Alt 3 later (the synthesized form *is* the Clojure body Alt 3 would -host). This is **not** a smaller-diff-for-its-own-sake pick — it is the F-009-thin -surface, and Alt 3 (not Alt 2) is the agreed end state. - -Two DA critiques are folded without adopting Alt 2: (1) **malformed-EDN `-X` -value** — a verbatim-spliced bad token (`{:a`) yields a reader error against the -`<-X>` label; this matches clojure's own "`-X` value is EDN-read, malformed → -error" behaviour, so it is acceptable (a parse error, not a silent -mis-eval), recorded here rather than guarded. (2) the synthesized `ex-info` -"has no -main fn" is a **user-level Clojure throw** (mirroring clojure.main's own -missing-`-main` error), not a Zig `setErrorFmt` bypass — so `error_catalog_only.md` -(which governs Zig-raised runtime messages) is not violated; when Alt 3 lands, the -guard moves into the Clojure grammar fn. - -## Consequences - -- **Positive**: cljw runs real apps (`-M -m`) and tools (`-X` build fns); the - `verified_projects/` proofs become deps.edn-idiomatic and exercise the run-mode - path. The three v0 gaps are fixed (args to `-main`, append semantics, EDN-typed - `-X` values). All synthesis reuses `runner.runSource` (F-009 one-chain - bootstrap) — no second eval engine. -- **Negative**: the `-M` mini-grammar is a first-token subset (no mixed - `-e`-init-before-`-m`, no `-i`/`-r`/`--report`/`@resource`) — tracked as D-310. - `*command-line-args*` is unbound, so library code reading it sees nil until - D-310. Run-mode synthesis is source-string assembly (not a structured arg - vector); the quoting discipline (`-X` data via `quote`, `-m` args via string - literals) is the correctness crux. - -## Affected files - -- `src/app/deps/parse.zig` — `Alias` gains `main_opts` / `exec_fn` / `exec_args`; - `parseAliases` captures them; `symbolName` helper. -- `src/app/deps/run_mode.zig` — **new**; `-M`/`-X` synthesis + dispatch. -- `src/app/cli.zig` — `-M`/`-X`/bare-`-m` detection + run-arg drain; `loadDepsEdn` - returns `{load_paths, cfg}`; run-mode branch; `-h` text. -- `src/app/runner.zig` — `runSource` gains `print_results`. -- `src/main.zig` — test aggregator entry for `run_mode.zig`. -- `test/e2e/phase14_deps_run_mode.sh` — **new**; 7-case smoke. `test/run_all.sh` - wires it. -- `verified_projects/*/` — deps.edn `:verify` alias + `(ns verify)` `-main`; - `scripts/verify_projects.sh` runs `cljw -M:verify`; `verified_projects/README.md`. diff --git a/.dev/decisions/0112_typed_instance_metadata.md b/.dev/decisions/0112_typed_instance_metadata.md deleted file mode 100644 index 50e5ba941..000000000 --- a/.dev/decisions/0112_typed_instance_metadata.md +++ /dev/null @@ -1,184 +0,0 @@ -# ADR-0112 — typed_instance metadata (`with-meta`/`meta` on records) - -- **Status**: Proposed → Accepted -- **Date**: 2026-06-07 -- **Discharges**: D-312 (with-meta on typed_instance) -- **Cross-refs**: ADR-0110 (symbol value-metadata — the precedent mirrored), - F-002 (finished-form), F-011 (clj behavioural equivalence), F-013 - (definition-derived capability); `private/notes/stage14-typed-instance-meta-survey.md` - -## Context - -`(with-meta m)` raised `with-meta: cannot attach -metadata to typed_instance`. In clj a **defrecord always implements IObj** -(records carry metadata for free, in a hidden `__meta` field); a plain -**deftype** does NOT unless it declares `clojure.lang.IObj`, and a `with-meta` on -a non-IObj deftype is a ClassCastException. cljw's `TypedInstance` had no instance -meta field; `metadata.zig`'s `.typed_instance` arms dispatched ONLY to a user -`IObj/-with-meta` impl, so a record with no hand-written IObj impl hit the error. - -Surfaced exercising **clojure.data.zip** via `cljw -M:verify` (its `descendants` -attaches metadata to a zipper instance). It blocks any record-using library that -carries metadata. - -clj semantics to match (verified): `(meta (->R 1))` → nil; `(meta (with-meta -(->R 1) {:x 1}))` → `{:x 1}`; `(= (->R 1) (with-meta (->R 1) {:x 1}))` → **true** -(meta NOT part of record value equality); `(identical? (->R 1) (with-meta (->R 1) -{:x 1}))` → false; a non-IObj deftype → throws. - -## Decision - -Mirror the ADR-0110 symbol-meta pattern, adapted for the record case -(**Shape α**, native field): - -1. **`TypedInstance` gains `meta: Value = nil_val`** (`type_descriptor.zig`, - appended to the extern struct — `header` stays at offset 0). `allocInstance` - inits it nil (a freshly-constructed record has no meta). - -2. **`instWithMeta` mints a FRESH instance** sharing the descriptor + the new - meta. **The field array is COPIED, not shared** — `field_values_ptr` is - `gc.infra`-owned and freed by `finaliseTypedInstance`, so two instances - sharing one pointer would double-free on sweep. This is the **one divergence - from the symbol `withMeta`** (symbols safely share interner-owned ns/name - slices; records must copy). `(identical? r (with-meta r m))` → false. - -3. **`metadata.zig` `.typed_instance` arms are kind-gated.** A user `IObj` - `-meta`/`-with-meta` impl is consulted first (honours an explicit - `(extend-type R IObj …)`); else: - - `meta` → the native field (`instMetaOf`; nil when unset — `meta` never - throws, even on a non-IObj deftype). - - `with-meta` → if `descriptor.kind == .defrecord`, mint via `instWithMeta`; - otherwise raise `.with_meta_target_not_iobj` (= clj ClassCastException for a - non-IObj deftype). Records always; plain deftypes only via an explicit IObj - impl (the dispatch path above). - -4. **`reify` (`.reified_instance`) stays IObj-dispatch-only.** `ReifiedInstance` - is a fixed 16-byte struct with no field tail and no place to store a re-minted - instance's fields; a reify gets meta only via an explicit `(reify IObj - (-with-meta …))`. A native field is deliberately NOT added to reify (scope line). - -5. **GC trace marks the meta map** (`traceTypedInstance` gains one line, mirroring - `traceSymbol`). **No membrane flip, no new registration** — `.typed_instance` - is already GcManaged and already has a registered trace (strictly less GC work - than the symbol case, which needed both). - -6. **Equality/hash unchanged.** `typedInstanceEqual`/`typedInstanceKeyEq`/the - `.typed_instance` `valueHash` arm walk descriptor + fields only, never a meta - field, so `(= r (with-meta r m))` → true falls out by construction. (The symbol - case needed `symbolStructEq` because interned identity was pointer-eq; records - were already field-structural.) - -7. **Record structural ops thread the source meta** (the DA's load-bearing - finding, adopted same cycle — see Alternatives Alt 2). `(meta (assoc (with-meta - r m) :k v))` → `m`, clj-faithful. The record `assoc` re-mint site - (`collection.zig:618`) calls `allocInstanceMeta(…, inst.meta)` instead of - `allocInstance` (meta-nil); `update` rides `assoc`. This was NOT deferred: - shipping `with-meta` while `assoc` silently drops meta is a "ships a lie" smell - (the user's divergence-suppression priority + F-011), and the cost is one - `Value` argument on an alloc call already in that function. The only re-mint- - from-existing site is the record `assoc` path; every other `allocInstance` - caller constructs a FRESH instance (meta correctly nil). Mutable-deftype - `set!`/`assoc` is in-place (same instance → meta preserved naturally). - -## Alternatives considered (Devil's-advocate, fresh-context subagent, verbatim) - -> I verified the load-bearing facts against source: **Residual (d) confirmed** — -> record `assoc` (`collection.zig:613-618`) ends in `allocInstance` which nil-inits -> `meta`, so `(meta (assoc r :k v))` → nil even after `with-meta`; the drop is real, -> unconditional, on the hot record-update path. **Copy reasoning confirmed** — -> `finaliseTypedInstance` frees `field_values_ptr`; sharing would double-free -> (forced by F-006, not convenience). **Reify** is a fixed extern struct, no field -> tail — point (4)'s scope line is structurally honest. -> -> **Alt 1 — smallest-diff: native field, but gate `with-meta` on -> `declaresProtocol("IObj")` for both kinds instead of `kind == .defrecord`.** -> Better: one uniform predicate; a `(deftype T … clojure.lang.IObj …)` that declares -> IObj but writes no `-with-meta` body gets native storage for free, matching clj's -> "declares IObj ⇒ carries __meta" more faithfully than "must hand-write -with-meta". -> Breaks: needs the defrecord macro to stamp an IObj declaration + `declaresProtocol` -> reachable from metadata.zig; if a deftype declares IObj AND writes `-with-meta`, -> dispatch-first must still win (both branches kept). Does NOT address (d). -> -> **Alt 2 — finished-form-clean (recommended): native field + thread `meta` through -> the record structural-op path in the SAME cycle (close residual (d)).** Change -> record `assoc`/`update`/`dissoc` to thread `old.meta` into the re-minted instance, -> as the persistent collections already do. Better: ships the WHOLE IObj contract -> for records — `(meta (assoc (with-meta r {:x 1}) :a 5))` → `{:x 1}` matches clj. -> This is the F-011 / F-013 definition-derived shape: "records are IObj" means meta -> survives structural ops, part of the definition, not a follow-up. The draft's -> split ("core with-meta now, threading = D-313") is the Ad-hoc-pass smell inverted -> onto a time axis — it ships the recognised-capability surface while silently -> dropping a definition-mandated behaviour on the most common record operation -> (update). Cost is tiny: the field-copy alloc already happens in assoc; threading -> meta is one extra `Value` to the alloc call. No F-NNN blocker, no new tag (F-004), -> no extra GC work beyond the meta-mark the draft already adds. Breaks: touches -> assoc/update/dissoc (a handful more sites) — per F-002 + Cycle-budget-defer smell, -> "more sites" is NOT a reason to defer. The extend-type `Associative` slow path -> returns the user impl's value and can't thread meta the same way — out of scope by -> construction (document, don't block). -> -> **Alt 3 — wildcard: side meta-table keyed by instance pointer (no struct field).** -> Better: zero struct-layout change; reify + record share one mechanism; no per- -> instance 8-byte cost for the majority of records that never carry meta. Breaks: a -> GC nightmare under F-006 — the map either roots every meta'd instance forever -> (leak) or needs a finaliser hook deleting entries + the map value traced as a -> SECOND root source. Reintroduces the pointer-keyed-side-table fragility the -> project avoids. The native field is strictly cleaner under non-moving mark-sweep: -> meta lives inside the traced object, no auxiliary root set. **Reject** — trades a -> bounded 8-byte field for an unbounded GC-rooting surface (F-002/F-006 inversion). -> -> **Recommendation: Alt 2.** The native-field choice is correct under F-006 (the -> draft gets that right; Alt 3 rejected). The discriminator (Alt 1) is a minor -> refinement, not load-bearing. The load-bearing finding is **(d): deferring -> assoc-meta-threading to D-313 IS a "ships a lie" smell** — `(assoc r :k v)` runs, -> returns a record, and `(meta …)` → nil with no error: the permanent-no-op / -> Ad-hoc-pass pattern `provisional_marker.md`'s 4th row forbids (no PROVISIONAL -> marker, user-observable dropped semantics). Worse than the symbol/collection -> precedents because record assoc/update is the PRIMARY way records are used. Cost -> to close: one extra `Value` arg on an alloc already in that function. Per F-002 + -> F-011 + Cycle-budget-defer smell, "it's a follow-up" is not a valid deferral. If -> the loop keeps the slice, the minimum honesty bar is a PROVISIONAL marker + AD/pin -> on the current nil behaviour — but the clean answer is Alt 2: thread the meta now -> and delete D-313. - -**Main-loop disposition: ACCEPT Alt 2 in full.** The DA's point (d) is correct and -aligns with the user's stated priority (divergence suppression over shipping a -slice). The loop threads `inst.meta` through the record `assoc` re-mint -(`collection.zig:618` → `allocInstanceMeta`), covering `assoc` + `update` (update -rides assoc); the e2e asserts `(meta (assoc/update (with-meta r m) …))` → `m`. **D-313 -is closed in this same commit, not opened.** Alt 3 (side table) is rejected on the -F-006 GC-rooting ground the DA gives. Alt 1's `declaresProtocol` discriminator is -NOT adopted: `kind == .defrecord` is the precise clj line (records always carry -meta; a deftype carries it only by *implementing* IObj, and an IObj-implementing -deftype already routes through the user `-with-meta` MethodEntry via -`dispatchOrNull` — so the native path for a non-record is unreachable-by-design and -the simpler `.defrecord` gate is exactly right, not a smallest-diff shortcut). The -`dissoc`-on-record case (demotes to a plain map in clj) is not a record re-mint and -is left to its existing path. This is a finished-form landing, not a sliced one. - -## Consequences - -- **Positive**: records join the metadata-bearing IObj family; **clojure.data.zip - loads + functionally verifies** (added as a `verified_projects/` proof). The - change is strictly less surgery than ADR-0110 (no membrane flip, no equality/ - hash arm, no registration) — the symbol precedent paid down the hard parts. - Definition-derived (F-013): records get meta by being records, not per-lib. -- **Negative**: `with-meta` (and meta-preserving `assoc`) copies the field array - (a fresh `gc.infra` alloc) — the irreducible cost of not sharing a finaliser- - owned buffer. `dissoc` on a record (which clj demotes to a plain map) is left to - its existing path, outside the record meta-threading. - -## Affected files - -- `src/runtime/type_descriptor.zig` — `TypedInstance.meta` field; `instMetaOf`; - `instWithMeta` + `allocInstanceMeta` (pub; copies the field array); - `allocInstance` nil-init; `traceTypedInstance` meta-mark line. -- `src/lang/primitive/metadata.zig` — kind-gated `.typed_instance` arms (split - from the combined `.reified_instance` arm); `td_mod` import. -- `src/lang/primitive/collection.zig` — record `assoc` re-mint threads `inst.meta` - via `allocInstanceMeta` (D-313 folded in, Alt 2). -- `test/e2e/phase14_typed_instance_metadata.sh` — record round-trip + deftype - throw. `test/run_all.sh` wires it. -- `verified_projects/data.zip/` — the now-passing proof. -- `equal.zig` / `heap_tag.zig` / `runtime.zig` — **no change** (already - field-structural / GcManaged / registered). diff --git a/.dev/decisions/0113_deferred_host_class_ref.md b/.dev/decisions/0113_deferred_host_class_ref.md deleted file mode 100644 index d0b5b81f8..000000000 --- a/.dev/decisions/0113_deferred_host_class_ref.md +++ /dev/null @@ -1,138 +0,0 @@ -# ADR-0113 — Deferred unsupported `clojure.lang.*` host-class references - -- **Status**: Proposed → Accepted -- **Date**: 2026-06-07 -- **Discharges**: unblocks integrant (Stage 1.3 verified_projects, 9th proof); - partially relieves the standing "clojure.lang.Compiler/Reflector/RT deferred" - blocker class (schema / clip / data.avl park there). -- **Cross-refs**: ADR-0059 (no-JVM class hierarchy), F-002 (finished-form), - F-011 (behavioural equivalence), F-013 (definition-derived); AD-022 (the - recorded divergence); the no-op-stub discipline (`provisional_marker.md` - — explicit runtime error OK, silent no-op forbidden). - -## Context - -Many real Clojure libraries fail to **load** on cljw not because their core -needs unsupported interop, but because a **peripheral** function's body names a -JVM-internal class cljw does not implement (`clojure.lang.RT`, -`clojure.lang.Compiler`, `clojure.lang.Reflector`). cljw's analyzer resolves -these eagerly at definition time, so - -```clojure -(defn- resources [path] - (let [cl (clojure.lang.RT/baseLoader)] ; <- analysis-time name_error - (enumeration-seq (.getResources cl path)))) -``` - -raises `namespace_unknown` while *defining* `resources`, blocking the WHOLE -namespace — even though `resources` is on an optional code path (classpath -resource loading) the library's core never touches. - -Concrete driver: **weavejester/integrant** (a popular DI/lifecycle lib) parks -ONLY on `(clojure.lang.RT/baseLoader)` inside its optional `load-hierarchy` -helper. Its core (`init`/`halt!`/multimethods/refs) is pure. (cljw already -tolerates `(catch java.io.FileNotFoundException _)` — unknown catch classes -never match, which is correct.) - -## Decision - -When the analyzer reaches a qualified reference `Ns/member` whose `Ns` is an -unresolved class with the unambiguous JVM-internal prefix **`clojure.lang.`** or -**`clojure.asm.`**, instead of raising `namespace_unknown` at analysis, rewrite -it to a loud runtime call `(rt/__unsupported-host-ref "Ns/member")` that raises -`feature_not_supported` when **evaluated**. The enclosing fn DEFINES, the -namespace LOADS, and the reference errors only if the unsupported path is -actually invoked. - -- **One interception point** (`analyzeSymbol`'s `namespace_unknown` branch) - covers BOTH call-head `(clojure.lang.RT/baseLoader …)` and value position - `clojure.lang.Compiler/CHAR_MAP` — both flow through `analyzeSymbol`. -- **Representation = rewrite to an existing raising builtin**, NOT a new - analyzer Node. A pure "raise at eval" needs no bespoke node; reusing the call - path is dual-backend-correct by construction (no parity arms to keep in sync) - and a builtin call is itself a clean future-Wasm lowering point. (This is the - one place the main loop diverged from the Devil's-advocate's "dedicated node" - preference — see Alternatives; the DA's substantive point, *coverage of both - positions*, is fully taken.) -- **Strict prefix allowlist** `clojure.lang.` / `clojure.asm.` is the - discriminator: these are provably non-values under ADR-0059 and never a user - alias typo, so the deferral cannot mask a typo (a bare `(myalias/foo …)` stays - a loud `namespace_unknown`). `java.*` is intentionally EXCLUDED — those may be - real host surfaces cljw should implement (e.g. `java.net.URI`), so they stay - loud and the gap stays visible. - -Sibling additions this cycle (same integrant driver, same loud-stub philosophy): -`enumeration-seq` / `iterator-seq` ship as explicit `feature_not_supported` -stubs (clojure.core fns over `java.util.Enumeration`/`Iterator` host types cljw -has no model for); `defmethod` now accepts an empty body (`(defmethod m dv -[params])` → nil, clj parity); `defmulti` now skips + **attaches** an optional -docstring/attr-map before the dispatch fn (was mistaken for the dispatch fn → -"Cannot call value of type 'string'"). - -The divergence (a `clojure.lang.*` ref that JVM would resolve now raises at -call time) is recorded as **AD-022**. - -## Consequences - -- A class of libraries whose core is pure but whose periphery touches - `clojure.lang.*` internals can now LOAD and run their core (integrant proven; - schema/clip relieved of their `clojure.lang.*` blocker, though they may have - further gaps). -- An unsupported `clojure.lang.*` reference fails at CALL time, not load time — - a louder, later, but still explicit error (never a silent no-op). Consistent - with the no-op-stub discipline (transient-stub category). -- Typos stay loud: only the strict JVM-internal prefixes defer. - -## Alternatives considered - -Devil's-advocate fork (fresh context, F-NNN-constrained), verbatim: - -> **1. Smallest-diff** — Keep the narrow rewrite but make the discriminator -> *purely* "unresolved class-shaped head" and skip even the prefix gate; rewrite -> ANY unresolved `Foo.Bar/method` call-head to the raising builtin. -> - **Better**: zero allowlist maintenance; integrant AND any future lib parking -> on an unimplemented host method loads immediately. -> - **Breaks**: exactly the F-011 / "no silent lies" hazard. A user typo -> `(my.ns/hlper x)` now silently DEFINES and only errors when called — masking -> a real load-time bug. The prefix gate exists precisely to keep typos loud. -> Rejecting. -> -> **2. Finished-form-clean** — A dedicated `UnsupportedHostRef` analyzer node -> carrying `(class, member, position)`, emitted for BOTH call-position and -> value-position (`clojure.lang.Compiler/CHAR_MAP`) unresolved -> `clojure.lang.*`/`clojure.asm.*` refs, with dual-backend arms that raise -> `feature_not_supported` on evaluation. -> - **Better**: call-position-only WILL re-park the same class of libs (schema on -> `CHAR_MAP`); a node covering value-position closes the category instead of -> opening a sequel ADR. A real node gives the future Wasm backend a clean -> lowering point and a single audit site; the payload makes the error precise. -> - **Breaks/risks**: bigger — new Node kind, two eval arms, value-position -> semantics. Risk of over-generalizing position handling cljw has no other use -> for yet. -> -> **3. Wildcard** — Don't touch the analyzer; add a bootstrap shim namespace -> providing `clojure.lang.RT`/`Reflector`/`Compiler` as cljw-level stub vars -> whose members raise. -> - **Better**: pure-Clojure, no Zig analyzer change, trivially extensible. -> - **Breaks**: `clojure.lang.*` are CLASSES not namespaces; faking them as -> resolvable namespaces lies about cljw's structure (violates ADR-0059), and a -> member cljw later DOES implement now collides with a stub. Worst masking -> surface. Rejecting. -> -> **Recommendation**: Take **Alt 2 (dedicated node, both positions)** despite the -> larger diff — F-002 says size is not a constraint and call-position-only is a -> false economy that guarantees a sequel ADR. Keep the strict `clojure.lang.`/ -> `clojure.asm.` allowlist — these are provably non-values under ADR-0059, never -> a user typo, so the masking concern that kills Alt 1 does NOT apply. Record the -> JVM-divergence as an AD-NNN per F-011. - -**Main-loop resolution**: Adopt Alt 2's **coverage** (both call- and -value-position) and its **strict prefix allowlist** + AD record — those are the -substantive, finished-form points. Decline Alt 2's **dedicated-node -representation** in favour of rewrite-to-raising-builtin: for a behaviour that is -purely "raise at eval", a new Node adds four dual-backend arms (TreeWalk + VM -compile + VM dispatch + diff case) to maintain for ZERO added capability over a -builtin call, and the parity surface is itself a risk -(`dual_backend_parity.md`). A builtin call IS a clean Wasm lowering point. This -is a representation choice on engineering-surface grounds, not a cycle-budget -downgrade (the coverage — the thing F-002 protects — is fully taken). diff --git a/.dev/decisions/0114_hiccup_protocol_host_interop.md b/.dev/decisions/0114_hiccup_protocol_host_interop.md deleted file mode 100644 index 956d2a835..000000000 --- a/.dev/decisions/0114_hiccup_protocol_host_interop.md +++ /dev/null @@ -1,258 +0,0 @@ -# ADR-0114 — hiccup enablement: protocol-dispatch + host-interop extensions - -- **Status**: Proposed → Accepted -- **Date**: 2026-06-07 -- **Discharges**: unblocks **weavejester/hiccup** (Stage 1.3 verified_projects, - 10th proof — `hiccup.core` / `hiccup2.core` / `compiler` / `util` / `page`). -- **Cross-refs**: ADR-0106 (host_instance container — amended here with - `host_finalise` / `host_trace`), ADR-0103 (host_inert), ADR-0102/host_interface - (host-supertype markers), ADR-0059 (no-JVM class hierarchy), F-002 - (finished-form), F-006 (GC strategy), F-011 (behavioural equivalence), F-013 - (definition-derived). AD-023 (java.util.Map extend-target inert). Debt D-317 - (ISeq tag table), D-318 (host_instance moving-GC), D-319 (Object-as-chain-root). - -## Context - -hiccup 2.0 (`hiccup.core` delegates to `hiccup2.core`) renders a Clojure data -tree to HTML through a `HtmlRenderer` protocol + Java-interop string building. A -naive "add java.net.URI" (the handover's expectation) was the first of a -**seven-blocker chain**, each a general F-013 coverage gap — not a hiccup -special-case. Grouped into three ADR-level decisions, plus routine host surfaces -and three bug fixes. - -## Decision A — Object extension is a universal protocol-dispatch fallback - -`(extend-protocol P Object …)` registers the impl on the Object class -descriptor. cljw protocol dispatch (`runtime/dispatch.zig`) previously raised -`protocol_no_satisfies` on a per-receiver-type miss without consulting Object. -**Now**: after a per-type miss, consult the Object descriptor's method_table and -call that impl if present. **nil is EXCLUDED** — clj nil is not an Object, so a -type extends nil separately (verified: `(extend-protocol P nil …)` resolves via -the per-type path on the nil descriptor, dispatched before this fallback). This -is the clj-faithful "Object is the universal default" rule; hiccup's HtmlRenderer -`Object` branch renders String / number content this way. Lookup is uncached -(correct — `lookupWithCache` never caches a null miss; the perf cliff is D-319, -within the deferred-optimization envelope). - -## Decision B — host_instance gains optional finalise + trace hooks (amends ADR-0106) - -Two optional fn-pointer fields on `TypeDescriptor`, dispatched by a single shared -`.host_instance` tag finaliser / tracer: - -- `host_finalise(infra, *[4]u64)` — URI / StringBuilder free their gc.infra heap - state (a duped string / an `ArrayList(u8)`) when the instance is swept. -- `host_trace(gc, *[4]u64)` — java.util.Iterator marks its cursor seq Value (the - **first** host type to store a live Value in `state`, closing the ADR-0106 - D-294 note). Decode goes through `heapHeader()` (the G1 membrane). - -Leaf host types (Random) leave both null → no cost. **Moving-GC caveat**: the -Value lives in a raw `u64` slot the typed field-walker can't see, so a future -moving GC must RELOCATE via `host_trace` (mark-only suffices for the current -non-moving GC, F-006). Documented in `gc_rooting.md` §H + a `GC-ROOT:` marker; -the closed-`host_state_shape`-enum finished form is D-318. - -## Decision C — clojure.lang / java interfaces as extend-protocol TARGETS - -Three target classes, resolved at the macro / analyzer layer: - -1. **host_inert java interface** (`java.util.Map`) as TARGET → **load-only - no-op** (`expandExtendType` emits nil). No cljw value is a java.util.Map, so - the impl could never dispatch — AD-023, matching the `host_interfaces.yaml` - inert contract (ADR-0103). As a bare VALUE (`(not-hint? x java.util.Map)`) it - resolves to a named class descriptor (analyzeSymbol). -2. **native-implemented clj interface** — a new closed table - `host_interface.NATIVE_EXTEND_TARGETS` maps `IPersistentVector`→`[vector]`, - `ISeq`→`[list lazy_seq cons chunked_cons range string_seq array_seq]`, - `Named`→`[keyword symbol]`; `expandExtendType` distributes the impl over - `(rt/__native-type :tag)` per tag, so a native value dispatches the protocol. - This is the core `(html [:p …])` path. (Hand-maintained table; D-317 tracks the - derive-from-markers finished form + the silent-omission risk for a future tag.) -3. **registered host-surface class** (`java.net.URI`) as VALUE → its `rt.types` - descriptor (analyzeSymbol; the surface analogue of the ADR-0109 opaque arm). - -## Routine surfaces (compat_tiers rows, not ADR-level) - -`java.net.URI` (keyword `uri`) + `java.net.URLEncoder` (`url_encode`) + -`java.lang.StringBuilder` (`string_builder`) + `java.util.Iterator` (`iterator`) -+ `java.lang.String/valueOf` (static surface alongside the installNativeMethods -instances). `.iterator` is wired into `object_method`'s universal fallback, so -every cljw seqable is a java.util.Iterable. - -## Bug fixes (commit notes, not ADR-level) - -- **syntax-quote alias resolution** (`qualifySym`): an `ns`-qualified symbol whose - `ns` is an alias resolves to the alias target's full name — clj hygiene - (`` `str/join `` → `clojure.string/join`). Without it a macro template calling - an aliased ns breaks wherever expanded (hiccup.core `html` → - `hiccup2.core/html`). -- **syntax-quote `%N` anon-fn param** stays bare (`#()` lowers to `(fn* [%1 %2] - …)` at read time; was over-qualified to an invalid `user/%1` fn* parameter). -- **exception_descriptor method_table leak** at `Runtime.deinit` (freed only the - fqcn; a `(extend-protocol P Object …)` adds a method_table that leaked — - hiccup's Object branch first surfaced it). - -## Alternatives considered - -The following is the verbatim output of a Devil's-advocate subagent forked with -fresh context against the active F-NNN constraints (the mandatory depth-≥2 -review). Its recommendations are advisory; the main loop's choices + rationale -follow each. - -> ### Decision A — Object extension as universal protocol-dispatch fallback -> **Threshold correctness findings:** (1) The cache does NOT skip the Object -> fallback — `lookupWithCache` does `td.lookupMethod(...) orelse return null;` -> before the refill, so a miss is never cached as present; no correctness hole, -> but a latent perf cliff (a defaults-only hot site re-walks lookupMethod every -> call, never caches the Object entry). (2) nil exclusion matches clj (a -> protocol with only an Object impl, called on nil, throws — nil does not fall to -> Object), but correctness depends on nil having a real resolvable descriptor in -> the per-type path; must be verified, not assumed. -> -> **A-smallest-diff** — cache the resolved-to-Object result too (sentinel -> MethodEntry). Better: removes the re-walk cliff, confined to method_table.zig. -> Breaks: a third cache state the global generation guard must invalidate (it -> already bumps on extend, low risk) + complicates the hit-site read. -> -> **A-finished-form-clean** — Object is the reserved ROOT of the descriptor -> lookup chain; lookupMethod walks receiver_td→…→Object_td; the dispatch-site -> special case disappears; the CallSite cache caches the resolved entry -> uniformly (cliff dissolves structurally). Better: zero extra fast-path code, -> the cljw-native expression of "Object is the universal target" (one reserved -> descriptor, not a JVM hierarchy — ADR-0059-safe), the seam future -> host-supertype precedence needs. Breaks: nil-exclusion must move into chain -> construction (Object NOT appended to nil's chain); ordering (most-specific -> first, Object last) encoded in chain order. More design, but the design the -> finished form needs. Cite F-002 — recommend anyway despite larger diff. -> -> **A-wildcard** — eager copy of Object impls into every descriptor's table at -> extend time. Better: pure per-type lookup, fastest hit path. Breaks: violates -> the finished form badly — override-ordering minefield, open descriptor set -> (re-sweep on every deftype birth), table bloat. Rejected on F-002, recorded as -> wildcard only. -> -> **Recommendation A: switch-to-A-finished-form-clean.** The dispatch-site -> special case is a smallest-diff convenience; Object-as-chain-root dissolves the -> cache cliff for free, gives nil-exclusion an honest home, and is the seam -> host-supertype precedence will need. F-002 says the larger diff is not a reason -> to prefer the draft. -> -> ### Decision C — clojure.lang / java interfaces as extend-protocol targets -> **Threshold findings:** (3) The 7-tag ISeq list is almost certainly incomplete -> and incompleteness is SILENT (map/set seqs, rseq, iterate/repeat/cycle, empty -> list, MapSeq — any distinct tag omitted → `(extend-protocol P ISeq …)` silently -> fails to dispatch on it, the F-013 failure mode but invisible). A hardcoded -> list is a denylist-by-omission. (4) F-013 tension: a hand-maintained table that -> drifts from where tags declare seq-ness is the per-tag special-casing F-013 -> forbids — the question is derived-vs-hand-maintained, not table-vs-no-table. -> -> **C2-smallest-diff** — keep the table but back it with an exhaustive -> `Value.Tag` switch the compiler forces total, so a new tag breaks the build -> until classified (the zig_tips.md idiom). Better: kills silent omission -> minimally. Breaks: still hand-classification, two places know "this tag is a -> seq". -> -> **C2-finished-form-clean** — derive the interface→tag mapping from the native -> types' declared interface markers (extend the existing `host_interfaces.yaml` -> closed-set SSOT to the native ISeq/IPersistentVector/Named set). Better: -> F-013-clean (cannot drift, a new seq tag that declares ISeq is auto-included), -> unifies with Decision B's iterator + any future native seq, converges with -> A-finished-form-clean on "descriptors are the SSOT". Breaks: requires native -> descriptors to carry a queryable native-interface set (real upfront work). -> Cite F-002 — recommend anyway; avoids a second parallel interface registry. -> -> **C2-wildcard** — one synthetic `__native_ISeq` supertype descriptor every seq -> tag's chain includes; extend writes once; dispatch finds it by chain walk. -> Better: O(1) extend, no enumeration, composes with A-finished-form-clean. -> Breaks: depends on A's chain walk; precedence + multi-interface (small DAG) -> subtleties. The endgame once A lands. -> -> **Recommendation C-2: switch-to-C2-finished-form-clean** (derive from markers, -> extend host_interfaces.yaml), with C2-wildcard as the endgame once A's chain -> walk lands; if A stays as-drafted, the minimum bar is the compiler-forced -> exhaustive switch — never a silent-omission denylist. **C-1 and C-3 -> keep-as-drafted** (C-1's load-only no-op is exactly ADR-0103's inert contract; -> C-3 is finished-form-clean). -> -> ### Decision B — host_instance optional finalise + trace hooks -> **Threshold findings:** (5) GC-safety: the container was declared a LEAF with a -> "no write barrier" comment; storing a live Value makes that invariant false — -> (a) the write-barrier/generational trap for a future GC (F-006), the leaf -> assertion must flip to conditional-non-leaf; (b) a moving GC must RELOCATE the -> Value (tracing marks liveness, not relocation) — this new root site must be in -> gc_rooting.md with a GC-ROOT marker or the moving-GC migration silently misses -> it. (6) "raw u64 = @intFromEnum of a Value" — a Value is NaN-boxed; the trace -> hook must decode through the canonical heapHeader/isGcManaged membrane, not -> treat state[0] as an opaque pointer (else mark garbage or miss pointers). -> -> **B-smallest-diff** — the fn-pointer hooks as drafted + the leaf→conditional -> doc correction + a GC-ROOT marker registered in gc_rooting.md + the hook -> decodes via the canonical membrane. Better: closes #5/#6 with no structural -> change; the per-descriptor fn-pointer is genuinely flexible. Breaks: nothing -> structural; this is the FLOOR the draft must reach. -> -> **B-finished-form-clean** — a closed `host_state_shape` enum on the descriptor -> (leaf / owns_heap_buffer@idx / holds_value@idx), dispatched by the shared -> finaliser/tracer; fn-pointer escape hatch for the irregular case. Better: -> moving-GC relocation centralised in ONE arm (not N hooks), the rooting membrane -> stays queryable (an enum value, not grep-the-fn-pointers), bounds ad-hoc state -> shapes, matches the project's closed-set-SSOT preference. The current three -> types (Random=leaf, URI/StringBuilder=owns_buffer, Iterator=holds_value) are -> 100% covered by three members. Cite F-002 — recommend anyway; the shape the -> moving-GC future demands. -> -> **B-wildcard** — promote Value-holding state out of `[4]u64` into a real -> GC-managed `?Value` field; `[4]u64` reverts to honest "non-Value words only". -> Better: eliminates the NaN-box-bits hazard at the root (typed Value, decoded by -> the type system), leaf assertion stays true, moving GC handles a real field -> uniformly. Breaks: changes ADR-0106's "[4]u64 state" container shape — a -> depth-3/4 revision (the project spirit welcomes it). Worth it if several host -> types will hold live Values; overkill if Iterator is a one-off. -> -> **Recommendation B: switch-to-B-finished-form-clean** (enum + escape hatch) and -> unconditionally adopt B-smallest-diff's three corrections regardless. The one -> thing that must NOT ship is the draft without the gc_rooting registration (an -> unrooted-for-moving-GC site, F-006 hazard). -> -> ### One-line recommendations -> - A — switch-to-A-finished-form-clean (Object as descriptor-chain root). -> - C-2 — switch-to-C2-finished-form-clean (derive from markers); C-1/C-3 keep. -> - B — switch-to-B-finished-form-clean (host_state_shape enum) + unconditionally -> add the leaf-fix + GC-ROOT/gc_rooting registration + canonical-membrane decode. - -### Main-loop disposition (within the F-NNN envelope) - -- **Decision A** — KEEP as-drafted (uncached dispatch-site fallback); the DA - confirms it is CORRECT. A-finished-form-clean's only win is the perf cliff + - future host-supertype precedence — both governed by the explicit - optimization-deferral invariant (memory `optimization-deferred-until-15-libs`, - at 10 libs) and the structural-imagination "defer to the owning point" rule. - Recorded as **D-319** (perf + chain-root finished form), scheduled at - optimization-resumption / host-supertype-precedence. This is NOT a - Cycle-budget-defer smell: the deferral is the optimization invariant, not diff - budget. nil-exclusion VERIFIED correct (`(m nil)` dispatches the nil impl). -- **Decision B** — KEEP the fn-pointer hooks (B-smallest-diff is the accepted - floor) AND adopt all three mandatory corrections NOW: leaf→conditional comment, - `GC-ROOT:` marker + `gc_rooting.md` §H, canonical `heapHeader()` decode (the - draft already used it). The `host_state_shape` enum (B-finished-form-clean) is - **D-318**, scheduled at the moving-GC migration / a 4th irregular host type — - the F-006 non-moving invariant makes the current mark-only trace correct, so - the enum is genuine moving-GC-future work, not a deferred cleanup. -- **Decision C** — C-1/C-3 keep (DA concurs). C-2 KEEP the hand table, VERIFIED - complete vs the live Value.Tag seq family (no current omission). The - derive-from-markers finished form + the future-omission risk are **D-317**, - scheduled at a new-seq-tag landing / host_interfaces.yaml-as-native-SSOT. - -All three deferrals record the finished form as a scheduled debt row (F-002 -honoured: the clean shape is captured, not lost), and none requires violating an -F-NNN. The DA found no F-NNN-blocking issue. - -## Consequences - -- Protocol dispatch honours Object + native-interface extension targets → a large - class of real libs (anything extending a protocol to Object / IPersistentVector - / ISeq / Named) now loads + dispatches correctly. -- `.host_instance` can carry heap-owning + Value-holding state, GC-correct under - the current non-moving GC; the moving-GC relocation path is documented + D-318. -- Divergence AD-023: extending a protocol to `java.util.Map` is inert in cljw. -- Perf (D-319) + tag-table drift (D-317) + host-state enum (D-318) are scheduled - finished-form work, not silent gaps. diff --git a/.dev/decisions/0115_regex_lookahead_honeysql.md b/.dev/decisions/0115_regex_lookahead_honeysql.md deleted file mode 100644 index 780f1dd18..000000000 --- a/.dev/decisions/0115_regex_lookahead_honeysql.md +++ /dev/null @@ -1,172 +0,0 @@ -# ADR-0115 — regex lookahead + honeysql host-interop - -- **Status**: Proposed → Accepted -- **Date**: 2026-06-07 -- **Discharges**: **seancorfield/honeysql** (Stage 1.3 verified_projects, 11th - proof); **D-315** (the honeysql drip-feed park row). -- **Cross-refs**: ADR-0114 (NATIVE_EXTEND_TARGETS — extended with - IPersistentMap), ADR-0087 (Singleton static fields), ADR-0050 am1 - (locale-independent casing), ADR-0059 (no-JVM class), F-002 / F-011 / F-013; - `no_op_stub_forbidden` (the silent-semantic-drop ban that shaped Decision B). - Debt D-320 (lookahead matcher fuse + perf). NO new AD (Decision B reaches full - parity — see below). - -## Context - -honeysql (honey.sql) formats a Clojure map into parameterized SQL. The survey -predicted 2 blockers (Locale, regex lookahead); the probe loop surfaced **5** -(the F-013 "a lib is an N-blocker chain" pattern, see -`.dev/library_incorporation_playbook.md`). They land together (D-315 was parked -precisely to avoid the drip-feed of landing them one-per-cycle). - -## Decision A — regex zero-width lookahead `(?=e)` / `(?!e)` - -cljw's regex engine is a Pike NFA (Thompson thread-list VM, `runtime/regex/ -match.zig`), NOT a backtracker. A lookahead is a **zero-width predicate** — the -same shape as the existing `.anchor` inst (which `addThread` already uses to drop -a thread on a failed zero-width test). Implementation: - -- `Node.look {child, negate}` — parsed from `(?=` / `(?!` in `parseAtom`. -- `Inst.look {sub, negate}` — `sub` is `child` compiled to its own sub-Program - (terminated by `.match`); `Program.deinit` recurses to free it. -- In `addThread` (the epsilon-closure), a `look` runs a nested - `tryMatchAt(sub, input, pos)` anchored at the current position; the thread - continues (consuming nothing) iff a match exists XOR negate. - -~70 LOC across `compile.zig` + `match.zig`. clj-oracle bit-for-bit on the -honeysql `dehyphen` pattern + positive/negative/no-consume/re-seq/alternation -cases. - -## Decision B — captures thread through a positive lookahead (FULL parity) - -The first draft discarded captures inside a lookahead (a "capture-free" cut with -a proposed AD). The Devil's-advocate review (below) correctly flagged that a -**silently-dropped capture is the forbidden silent-semantic-drop** -(`no_op_stub_forbidden`: the user sees a successful match with a wrong group -vector), and that it does NOT qualify as an accepted divergence (the -justification was convenience, not a project invariant). So Decision B is the -**finished form**: a POSITIVE lookahead threads its inner capture groups into the -continuing thread (group indices share the global numbering, so the sub-match's -populated slots merge into the parent thread's `caps`); a NEGATIVE lookahead -exports no captures (JVM parity — it succeeds only when the sub fails). The -per-thread slot array already exists and is already copied, so the merge adds no -allocation. Result: `(re-find #"(?=(\w+))\w+" "abc")` → `["abc" "abc"]`, -bit-for-bit with clj. **No AD-024** — the divergence does not exist. - -## Routine (not load-bearing — compat_tiers rows / table additions) - -- **java.util.Locale/US + /ROOT** — OBJECT-valued static fields via the ADR-0087 - Singleton mechanism (`Singleton` enum + `locale_us`/`locale_root` + analyzer - arm). gc.infra process-lifetime `.host_instance` singletons cached on `rt` - slots (leaf — no rooting subtlety, the lesson of the 2026-06-07 revert). Neutral - impl `runtime/locale.zig` (so `Runtime.deinit` + the analyzer reach the - singletons without importing the `runtime/java/` surface tree — zone rule); - surface `runtime/java/util/Locale.zig` owns the descriptor + static-field table. -- **String.toUpperCase/toLowerCase 2-arg Locale overload** — accept + ignore the - Locale (cljw casing is locale-independent, ADR-0050 am1 → US/ROOT = the existing - impl, F-011-faithful). -- **clojure.lang.IPersistentMap extend-TARGET** → `[array_map hash_map sorted_map]` - in `NATIVE_EXTEND_TARGETS` (ADR-0114). Keyword/Symbol/IPersistentVector already - resolve to a single native tag via `class_name`, so they need no row. -- **(.sym keyword)** → the underlying Symbol (`symbol (namespace k) (name k)`); - `runtime/keyword_methods.zig` installs it on the `.keyword` native descriptor - (the `namespace_methods.zig` pattern). - -## Alternatives considered - -Verbatim Devil's-advocate output (mandatory depth-≥2 review, fresh context): - -> ### DECISION A — zero-width lookahead via nested `tryMatchAt` -> **Alt A1 — smallest-diff (as drafted):** the `look` inst runs a fresh -> `tryMatchAt` from inside `addThread`. The nested call uses a FRESH ThreadList, -> so the outer `seen` bitmap is untouched — no recursion-into-shared-state -> hazard. Greedy-longest is IRRELEVANT (the `look` only checks `!= null`), so -> `(?=a*)` correctly succeeds zero-width. The framing is sound on every -> existence-only question. Breaks: latent O(n·m) per look-eval blowup (two fresh -> ThreadList allocs + a full sub-scan per position per outer thread) — invisible -> for honeysql's `(?=\w)`, a real super-linear cliff for a finished-form engine; -> acceptable NOW under the optimization-deferral policy, but a perf debt. -> -> **Alt A2 — finished-form-clean:** keep the zero-width-predicate framing but -> replace the ad-hoc `tryMatchAt` re-entry with a named `lookaheadHolds(sub, -> input, pos)` helper carrying an explicit reentrancy/`seen`-disjointness contract -> + a stack-`seen` fast path for small sub-programs (the dominant case), removing -> A1's hot-path churn with no semantic change. The fully-fused single-pass variant -> (inline the sub-program's threads with a "in a zero-width assertion" flag) would -> kill the O(n²·m) entirely — the textbook Pike-VM lookahead — but is substantially -> more complex with a semantic trap (a lookahead is anchored/position-local while -> the parent thread-list advances globally; fusing risks correctness-fragility). -> Net: A2-as-named-helper is strictly cleaner with no semantic risk; A2-as-full- -> fuse is a later O-NNN perf optimization, not this cycle. -> -> **Alt A3 — wildcard (DISQUALIFIED, F-013 violation):** don't implement -> lookahead; rewrite honeysql's `(?=\w)` at its call site to dodge it. This is -> exactly the lib-specific patch F-013 forbids (`(?=e)` is a general regex -> feature; the root-cause fix is "implement lookahead"). Out of bounds. -> -> **Recommendation A: switch to A2 (named-helper form, NOT full-fuse).** Keep the -> `look` inst + zero-width framing (correct); make the excursion a named helper -> with explicit contract + stack-`seen` fast path. The full fuse is a future O-NNN. -> -> ### DECISION B — capture-free lookahead -> **Alt B1 — smallest-diff (as drafted, FORBIDDEN):** silently discard inner -> captures + record AD-024. Breaks: `(?=(\w+))\w+` compiles, runs, returns a -> match, and silently returns the WRONG group vector — the canonical -> `no_op_stub_forbidden` "user builds on a lie" failure. Recording an AD does not -> launder a silent-semantic-drop: per `accepted_divergences.md`'s own bar, the -> justification ("captures are hard to thread") is CONVENIENCE, not a project -> invariant, so it does NOT qualify as an accepted divergence. Disqualified. -> -> **Alt B2 — finished-form-clean:** thread captures through the positive -> lookahead. The slot array is ALREADY copied per-thread; the `look` inst merges -> the sub-match's set slots into the continuing parent thread's `caps`. FULL -> F-011 parity, NO AD-024 (eliminating the loss beats recording it). Cost is -> small: no new allocation, no new data structure (B1 has to deliberately zero -> capture_count; B2 just doesn't). Group-numbering already works (the parser -> assigns indices globally across the whole pattern incl. inside `(?=...)`). -> Negative lookahead exports no captures (JVM parity — nothing to capture). -> -> **Alt B3 — wildcard (the same-cycle backstop):** raise a LOUD -> `CompileError.NotImplemented` on a capturing group inside a lookahead. Better -> than B1 (no silent drop — the transient-stub row of the boundary table). Still -> a coverage gap under F-011; its legitimate role is the compile-time guard for -> any capture construct B2 doesn't yet cover (backreferences — already -> unsupported globally — nested-lookahead capture interplay), so no silent drop -> can ever occur. -> -> **Recommendation B: switch to B2** (thread captures through positive lookahead; -> delete AD-024), with B3's loud error as the backstop for any residual -> unsupported capture construct. B1 is FORBIDDEN, not merely suboptimal. -> -> ### One-line recommendations -> - A — switch to A2 (named helper; full fuse is a future O-NNN). -> - B — switch to B2 (full capture parity, delete AD-024); B3 loud-error backstop. - -### Main-loop disposition (within the F-NNN envelope) - -- **Decision A** — KEEP A1 (nested `tryMatchAt`); the DA confirms it is CORRECT - (fresh ThreadList → `seen` disjoint, greedy irrelevant, recursion-safe). A2's - only wins are perf (the O(n²·m) cliff) + a clarity refactor — both governed by - the optimization-deferral invariant (memory `optimization-deferred-until-15-libs`) - and recorded as **D-320** (named-helper + stack-`seen` + the eventual single-pass - fuse), scheduled at optimization-resumption. Not a Cycle-budget-defer: the - deferral is the optimization invariant, and the current shape is correct. -- **Decision B** — ADOPT B2 (full capture parity). This was NOT a deferrable - preference: B1 (silent discard) is forbidden by `no_op_stub_forbidden`, so the - compliant options were B2 (parity) or B3 (loud error). B2 is both cleaner AND - small (captures already flow through `tryMatchAt`; the merge adds no - allocation), so it ships now — **AD-024 is deleted**, the divergence does not - exist. B3's loud guard is implicitly satisfied: backreferences are already a - global `NotImplemented`, and nested-lookahead captures merge recursively, so no - capture construct silently drops. - -## Consequences - -- The regex engine gains lookahead → honeysql + any lib using `(?=…)`/`(?!…)` - loads, with full capture parity (no divergence). -- Object-valued static fields are a reusable pattern (the Singleton enum extends - cleanly per feature). -- Perf: a quantified lookahead re-runs a nested sub-match per thread-per-pos - (latent O(n²·m)); honeysql's single-step `(?=\w)` is negligible. D-320, deferred. -- D-315 (honeysql park) discharged; the library-incorporation campaign reaches 11 - proofs and goes to STAY (user directive 2026-06-07). diff --git a/.dev/decisions/0116_instance_interface_membership_ssot.md b/.dev/decisions/0116_instance_interface_membership_ssot.md deleted file mode 100644 index 95cb4a90d..000000000 --- a/.dev/decisions/0116_instance_interface_membership_ssot.md +++ /dev/null @@ -1,243 +0,0 @@ -# ADR-0116 — `instance?` clojure.lang interface membership SSOT + deref family - -- **Status**: Proposed → Accepted -- **Date**: 2026-06-08 -- **Discharges**: **D-308** (`(instance? clojure.lang.IDeref x)` + the deref / - pending / ref interface family raised `class_name_unknown`). Partially - addresses **D-317** (the `class_name.matchInterface` ↔ `NATIVE_EXTEND_TARGETS` - duplication) — see Decision C for the part deferred and why. -- **Cross-refs**: ADR-0109 (`instance?`/`isa?` class surface), ADR-0114 / - ADR-0102 (`host_interface` / `host_interfaces.yaml` — the deftype-supertype - marker concern, DISTINCT from this membership concern), ADR-0059 / AD-003 - (no-JVM class), F-002 / F-011 / F-013 / F-009; `zig_tips.md` (the - exhaustive-`Value.Tag`-switch guidance that shaped the table representation); - `provisional_marker.md` (permanent-no-op ban). Debt: D-271 (the `range`/ - `chunked_cons` no-meta-slot gap that gates IObj/IMeta), D-317 (extend-target - reconciliation residual). - -## Context - -`(instance? clojure.lang.IDeref x)` raised `name_error: class -'clojure.lang.IDeref' is not a known class name`. core.memoize:67 needs it -(its `RetryingDelay` deftype implements IDeref + IPending). A 2026-06-07 naive -fix (`expandInstanceQ`: rewrite `(instance? marker x)` → `(satisfies? proto x)`) -was REVERTED — it broke `(instance? clojure.lang.IFn :kw)` → clj `true`: a -keyword IS natively an IFn (callable-as-lookup) but does NOT extend an IFn -*protocol*, so satisfies? is false. **Lesson: instance?-on-host-interface = -native tag membership ∪ user-deftype protocol satisfaction — never a satisfies? -substitution.** - -`class_name.zig` already holds a definition-derived tag-set membership table for -~22 interfaces (the `matchInterface` switch-arm chain, clj-oracle-grounded). The -gap is (a) the deref/pending/ref family is absent from all three of its -hand-maintained structures (`FQCN_MAP`, the `isInterfaceName` 24-deep `or`-chain, -`matchInterface`), so the name is unrecognised; (b) `matchUserType` matches a -user type only by ancestor-`fqcn` name-equality, never by protocol satisfaction, -so a deftype that *extends* IDeref is not an IDeref instance. - -## Decision A — one in-code SSOT for native interface membership - -New `src/runtime/interface_membership.zig`: a single comptime table -`Entry{ name: []const u8, tags: []const Value.Tag }[]` mapping each recognised -clojure.lang interface to the native `Value.Tag` set that implements it. -Co-membership interfaces (IPersistentCollection/Seqable, Associative/ILookup, -Indexed/IPersistentVector, …) point at **shared tag-set constants** so a set is -authored ONCE (the DRY trap the DA flagged in R2). `class_name.zig` derives ALL -THREE of its consumers from this one table — `matchInterface` (membership), -`isInterfaceName`/`isKnown` (the recognised-name set, **retiring the `or`-chain** -— F-013 structural close), so the three can no longer drift. - -**Representation = forward `{name, tags}` table, NOT an inverse exhaustive -`switch (Value.Tag)`.** The DA's finished-form-clean Alt-2 proposed the inverse -switch for compile-time tag-exhaustiveness, but `zig_tips.md` explicitly rules -that wide value-dispatch enums (`Value.Tag`, heading 54→64 slots per F-004) use -`else =>`, NOT exhaustive enumeration ("a new tag is *meant* to fall into the -default until its primitives are wired in"). The forward table honours that idiom -and is a single source (no yaml-vs-code split → no drift gate needed; the -existing instance? membership was itself UN-gated, and a single in-code SSOT is -the structural close, not a second SSOT-of-record + gate). - -The deref/pending/ref family added (oracle-verified vs `clj`, see corpus -`test/diff/clj_corpus/instance_interfaces.txt`): - -| interface | native tags | -|----------------|----------------------------------------------------------------------| -| IDeref | atom, agent, ref, volatile, future, promise, reduced, delay, var_ref | -| IRef | atom, agent, ref, var_ref | -| IReference | atom, agent, ref, var_ref, ns | -| IPending | delay, future, promise, lazy_seq | -| IBlockingDeref | future, promise | - -## Decision B — user-deftype protocol-satisfaction ∪ arm (name-based, in `matchUserType`) - -`matchUserType` (class_name.zig) gains a second pass: after the ancestor-`fqcn` -walk, it scans each descriptor's `method_table[].protocol_name` and -`protocol_impls[]` for the normalised interface name. A deftype declaring -`clojure.lang.IDeref` registers its method under cljw protocol name `"IDeref"` -(the `host_interface` remap target = the bare canonical), and the FQCN_MAP -normalises `clojure.lang.IDeref` → `"IDeref"`, so the names meet. - -**This is name-based and lives in the `runtime/` leaf (no `rt`), deliberately -NOT the DA's R3 shape** (resolve the bare name → a protocol Var → `Var.deref()` -→ `protocol.satisfies`). The DA's own R3 finding is the reason: a user -protocol's `fqcn()` is namespace-QUALIFIED (`/`) while the deftype's -stored `protocol_name` for a host-remapped interface is the BARE canonical, so a -Var-resolution path risks a bare-vs-qualified mismatch (and a silent never-match -= permanent-no-op, forbidden). Matching the stored `protocol_name` string -directly — the same field `protocol.satisfies` itself compares — is both simpler -and exactly correct for the host-marker remap case. Native membership is PRIMARY, -this arm is ADDITIVE: the two are OR-unioned, never substituted (so keyword∈IFn -survives). - -## Decision C — partial D-317 derivation (the safe 3/4), IPersistentVector deferred - -`host_interface.nativeExtendTags` (the extend-protocol-TARGET native-tag -distribution, macro_transforms.zig:2405) is derived from the new SSOT for the -interfaces where the two consumers AGREE: **ISeq, Named, IPersistentMap** (the -7-tag ISeq list etc. now lives in ONE place). **IPersistentVector is NOT -unified**: instance? membership = {vector, map_entry} (a MapEntry IS an -IPersistentVector in clj), but extend-target distribution = {vector} only. -Deriving extend-target from the membership set would start distributing -IPersistentVector-extended protocols to `map_entry` — arguably MORE clj-correct, -but a behaviour change requiring verification that `map_entry` carries an -extendable native descriptor (and oracle confirmation). That is a genuine design -question, not a mechanical refactor — deferred to D-317 (kept open) per F-003 -(decision-deferral over decision-seizure on a structural reconciliation), NOT -cycle-budget (the divergence is real, not a size dodge). - -## Decision D — IObj / IMeta enumerated but NOT activated (D-271 coordination) - -The oracle shows IObj / IMeta membership is large and intricate (IObj = -promise/future + most collections/seqs + fn + symbol, EXCLUDING -map_entry/keyword/delay; IMeta = IObj ∪ atom/agent/ref/var/ns). Activating them -now would break F-011 behavioural equivalence: clj guarantees `(instance? IObj -x)` ⟹ `(with-meta x m)` succeeds, but cljw's `.range`/`.chunked_cons` substrate -has **no meta slot** (D-271), so `(instance? IObj (range))`→true while -`(with-meta (range) m)`→cljw error. Per `accepted_divergences.md`, an AD must -cite a project INVARIANT; "D-271 hasn't landed" is a temporal incomplete-impl -state, not an invariant — so an AD would launder an unfinished feature (itself a -smell, per the DA's R1). Therefore: the full IObj/IMeta tag sets are -**enumerated in an inactive comment block** in interface_membership.zig (cheap; -prevents re-discovery, F-013 clause 4 "網羅 the recognition table up-front, wire -impl incrementally"), their MEMBERSHIP ACTIVATION gated on D-271, and D-271's row -records IObj/IMeta membership as one of its deliverables. Until then -`(instance? clojure.lang.IObj x)` stays a loud `class_name_unknown` (an explicit -transient signal, not a silent lie). - -## Alternatives considered (Devil's-advocate subagent, fresh context, verbatim) - -> ### Alternative 1 — Smallest-diff: extend the existing three tables in place, no new SSOT -> Add the deref/pending/ref family directly to `class_name.zig`'s three existing -> structures (`FQCN_MAP`, `isInterfaceName`'s `or`-chain, `matchInterface`'s -> `switch`-arm chain), plus the ∪ arm. Leave `NATIVE_EXTEND_TARGETS` untouched — -> D-317 stays open. -> - **Better:** Zero migration risk (the Long/BigInt origin special-case in -> `matchNativeExact` is never disturbed); keeps the per-interface provenance -> comments. Smaller blast radius. -> - **Breaks/costs:** Defeats D-317; `isInterfaceName` stays the 24-deep -> `or`-chain the survey identifies as "the exact 個別最適化 shape F-013 clause 3 -> warns about". Under F-013 + F-002 this is the Smallest-diff-bias / -> Cycle-budget-defer smell — cheaper, reaches a dirtier finished form. -> Recommend against unless an F-NNN blocks B (none found). -> -> ### Alternative 2 — Finished-form-clean: single SSOT as an INVERSE exhaustive `switch (Value.Tag)`, fold D-317 + `isKnown` in -> Represent the SSOT as a comptime exhaustive `switch (Value.Tag)` returning each -> tag's interface set; both consumers + `isKnown`/`isInterfaceName` derive. -> - **Better:** A new `Value.Tag` breaks the build until classified -> (compile-time structural close, no gate script); subsumes the `or`-chain. -> - **Breaks/costs:** Larger diff; provenance comments must be re-homed; inverse -> reads less naturally for "what implements IDeref?". *[Main-loop note: this -> Alt is DECLINED because `zig_tips.md` rules wide `Value.Tag` switches use -> `else =>`, not exhaustive enumeration — the inverse switch fights the project -> idiom. Decision A takes the forward table, which is the F-013 structural close -> compatible with that idiom.]* -> -> ### Alternative 3 — Wildcard: drive native membership from the defprotocol/extend-protocol registry -> For interfaces that ARE cljw defprotocols, read membership from the protocol -> registry instead of a static set. -> - **Better:** Maximally DRY; one source for instance?/satisfies?/distribution. -> - **Breaks/costs:** RE-INTRODUCES the reverted bug — `(instance? IFn :kw)` must -> be true (keyword is *natively* IFn, not extend-protocol-registered), so a -> registry read under-reports every native-but-not-extended tag; plus a -> load-order hazard + a `runtime/`-leaf-imports-registry zone breach. Fails -> F-011 on the IFn family. Salvage = registry feeds ONLY the additive ∪ arm -> (which Decision B does, name-based). -> -> ### R1 — scope boundary (deref now, IObj/IMeta deferred): LEADING F-NNN finding -> The maximally-網羅 move adds IObj/IMeta in the same big-bang (Discipline 2). -> BUT the D-271 entanglement is REAL: clj's IObj ⟹ withMeta, cljw's range/cons -> have no meta slot, so claiming IObj membership breaks F-011 — and an AD to -> cover it would cite "D-271 unlanded" which is NOT an invariant (the AD rule -> forbids this), so it belongs in a debt row, not the AD ledger. Recommendation: -> defer ACTIVATION, but ENUMERATE the IObj/IMeta/IReference tag rows now (cheap, -> prevents re-discovery) with membership gated on D-271; classify IDeref's JVM -> super-interfaces (Supplier/Runnable/Callable/Comparable) as no-JVM uniformly- -> false in the same cycle. *[Main-loop disposition: adopted — Decision D -> enumerates inactive + gates on D-271. The JVM super-interfaces are left to a -> follow-up note (a separate java.* functional-interface family, not the -> clojure.lang deref family — out of this cycle's definition-derived class).]* -> -> ### R2 — SSOT migration risk: SAFE -> Read every `matchInterface` arm (L305-426): all 24 are pure tag-set membership; -> the Long/BigInt `origin`-flag special case is in `matchNativeExact` (L276-280), -> NOT `matchInterface` — a flat `{name, tags}` table reproduces it exactly. ONE -> trap: co-membership interfaces share a tag set (IPersistentCollection/Seqable, -> Associative/ILookup, Indexed/IPersistentVector) — the SSOT must alias them to a -> shared set or it re-introduces the very drift D-317 kills. `Number` has TWO -> definitions (core.zig `isNumberClass` = the live broad path incl. -> big_int/ratio/big_decimal; class_name L317 = narrow integer/float, dead for the -> primitive path) — either unify or document core.zig owns Number. No PERF marker -> needed (instance? is a predicate, not a hot inner loop). -> -> ### R3 — ∪ arm placement + precedence -> "Native PRIMARY, protocol ADDITIVE" = OR-union, never substitution (correct). -> CRITICAL HOLE: a user protocol's `fqcn()` is namespace-QUALIFIED -> (`/`), not bare `"IDeref"`, so the ∪ arm cannot compare the bare -> instance? symbol to `proto.fqcn()` — it must resolve the name → protocol Var → -> `Var.deref()` → `protocol.satisfies`, OR match the stored `protocol_name` -> directly. Non-protocol interfaces (Number/Iterable) must no-op cleanly (not -> error). `instanceQPrim` has `rt`+`env` (env.zig:173 resolve / :110 deref); -> `class_name.isInstance` does not. *[Main-loop disposition: Decision B matches -> the stored `protocol_name` string directly in `matchUserType` — the bare -> canonical the host-remap deftype actually stored — which sidesteps the -> qualified-fqcn mismatch the DA flagged and needs no `rt`. Verified empirically -> during green.]* -> -> ### R4 — gate / framework-completion -> `class_name.zig` membership is currently UN-gated (no script syncs FQCN_MAP / -> isInterfaceName / matchInterface). A new SSOT without a gate is the half-built- -> framework smell; recommend folding membership into `host_interfaces.yaml` -> (`native_implementors:` field) so the existing G4 gate extends, AND retire the -> `isInterfaceName` `or`-chain in the same cycle. *[Main-loop disposition: the -> `or`-chain IS retired (Decision A). The yaml-fold is DECLINED: native-tag -> membership is a DISTINCT concern from host_interfaces.yaml's deftype-supertype -> markers (one-SSOT-per-concern, ADR-0102's own principle); and a single in-code -> table that ALL consumers derive from has no record-vs-code split to gate (drift -> is structurally impossible). The corpus regression -> (`check_corpus_regression.sh`) + unit tests are the behavioural lock.]* - -## Consequences - -- `(instance? clojure.lang.IDeref/IRef/IReference/IPending/IBlockingDeref x)` - resolves for native values and user deftypes (∪ arm); core.memoize's IDeref/ - IPending needs are met. The IFn-keyword regression guard is locked. -- `class_name.zig`'s instance? membership is one derived-from-SSOT concern; the - `isInterfaceName` `or`-chain is gone. -- D-317 narrows (ISeq/Named/IPersistentMap unified) but stays open for the - IPersistentVector extend-target reconciliation. -- D-271 gains IObj/IMeta membership as a coordinated deliverable. -- New `Value.Tag`s still fall to `else` (no instance? membership) until - classified — consistent with zig_tips; a future tag's interface membership is - added to the SSOT rows as it is wired. - -## Affected files - -- NEW `src/runtime/interface_membership.zig` (SSOT + tests) -- `src/runtime/class_name.zig` (matchInterface / isInterfaceName / isKnown - derive; FQCN_MAP deref-family rows; matchUserType ∪ arm) -- `src/runtime/host_interface.zig` (nativeExtendTags derives ISeq/Named/ - IPersistentMap from the SSOT; IPersistentVector kept explicit) -- `src/main.zig` (test aggregator `_ = @import` of the new file) -- `test/e2e/phase14_instance_deref_family.sh` + `test/run_all.sh` registration -- `test/diff/clj_corpus/instance_interfaces.txt` (deref-family corpus rows) -- `.dev/debt.yaml` (D-308 discharged; D-317 narrowed; D-271 IObj/IMeta note) diff --git a/.dev/decisions/0117_cli_align_clj_script_semantics.md b/.dev/decisions/0117_cli_align_clj_script_semantics.md deleted file mode 100644 index bd2921326..000000000 --- a/.dev/decisions/0117_cli_align_clj_script_semantics.md +++ /dev/null @@ -1,156 +0,0 @@ -# ADR-0117 — CLI alignment to clj 本家: script vs eval-print semantics, `--version`/`--help` - -- **Status**: Proposed → Accepted -- **Date**: 2026-06-08 -- **Opens**: **D-322** (stdin `-` no-echo migration + no-args → REPL — the - follow-on units staged out of this ADR's first commit; see Decision D). -- **Cross-refs**: ADR-0048 (`repl`/`nrepl` subcommands), ADR-0059 / AD-003 - (no-JVM class — sibling no-JVM surface decision), AD-007 (error Kind, the - separate user-driven error-format call deferred here), F-002 (finished-form - wins; cycle/diff/LOC is not a project constraint), F-009, F-011 (behavioural - equivalence with JVM Clojure). Reference: cljw v0 `build.zig` / - `src/app/cli.zig` (the version auto-derivation pattern). - -## Context - -The user directed: align cljw's CLI to JVM Clojure (`clj`) behaviour, -referencing cljw v0 for display clarity + version auto-derivation, resolving -the CLI divergences in one pass. Empirically measured (clj 1.12.4 / -2026-06-08): - -| invocation | clj 本家 | cljw v1 (before) | -|-------------------|---------------------------------------------------------------------------------------|-------------------------------------| -| `-e ` | eval + print each result (echo) | echo (match) | -| `` | run as script, **no echo** | **echoes every top-level value** | -| `-` (stdin, dash) | run stdin as script, **no echo** (clj deprecates `-`, points at `-M -`, also no-echo) | echoes every top-level value | -| no args | REPL | prints a `ClojureWasm` smoke banner | -| `--version` | `Clojure CLI version …` | `Unknown option` (exit 1) | -| `--help` | tool help | usage, no version banner | - -cljw's top-level echo on a **file** run is the load-bearing divergence: it -pollutes demo/bench output with `nil`/`#'user/x` noise and breaks the bench -oracle (`$CLJW file | head -1` is meant to equal the program's printed -result, but echo prepends every intermediate value). Removing file echo both -matches clj and unblocks the benchmark suite (STREAM 2) with no harness change. - -## Decision A — `` runs as a script (no result echo) - -`runSource` already takes a `print_results: bool`. `cli.zig` now sets it -`false` for a bare-file source (`print_results` local, flipped in the -file-open branch) and `true` for `-e`. A file thus prints only what the -program prints (`println`/`prn`); clj-faithful. The `-M`/`-X` run modes were -already `print_results=false`. - -## Decision B — `-` (stdin) also runs as a script (no echo) — **target** - -The finished form is **uniform**: `` and `-` are both "source from -somewhere, run it"; only `-e` (and the REPL) echo. clj agrees: `clj -` is a -no-echo script (and is itself deprecated in favour of `-M -`, which is *also* -no-echo — deprecation does not bless an echo reading). - -An earlier draft kept stdin echoing and recorded it as an accepted divergence -(AD-024), justified by "cljw's `-` is a pipe-friendly batch eval-print mode" -+ "no-echo would break ~150 e2e heredoc fixtures". The Devil's-advocate fork -(below) correctly identified that as **smallest-diff bias wearing a divergence -costume**: two of its three legs are cost/convenience arguments, which F-002 -forbids as grounds, and the third (deprecation) does not support echo. **AD-024 -is withdrawn.** stdin `-` → no-echo is the recorded target. - -## Decision C — `--version` + `--help` banner (auto-derived) - -`build.zig` reads `build.zig.zon .version` into a `build_options.version` -string (cljw-v0 pattern); `cli.zig` gains a `--version` arm -(`ClojureWasm v`, exit 0) and prepends the same banner to `--help`. -The version string is **never hand-written** — it tracks the release tag the -user owns (currently `build.zig.zon` = `0.0.0`, so `--version` reports -`ClojureWasm v0.0.0` until the user bumps the tag; the loop does not invent a -version). - -## Decision D — staging (what lands when) - -This ADR's **first commit** lands Decisions A + C (file no-echo, version, -help banner) — they are complete, clj-faithful, gate-green, and deliver the -bench unblock. Decision B (stdin no-echo) is a 56-file e2e heredoc migration -(each echo-asserting `cljw - <)` vs cljw -`: [phase]`) is **out of scope** here — it is AD-007-adjacent -(no-JVM Kind) and the user is driving it interactively. - -## Alternatives considered (Devil's-advocate, fresh-context fork — stdin `-` echo) - -**Verdict on the draft (AD-024 "stdin keeps echo"): it is smallest-diff bias -wearing a divergence costume.** The rationale chain is revealing — two of its -three legs ("breaks ~150 e2e files", "becomes a pipe-friendly mode") are -cost/convenience arguments, and under F-002 cost is explicitly *not* a valid -reason. The third leg ("clj's `-` is deprecated") actually cuts the other way: -clj deprecates `-` *in favor of `-M -`, which is still no-echo script mode* — -it does not bless echo. So the honest finished-form answer is below as Alt 2, -and AD-024 should not be the lead. - -**Alt 1 — Smallest-diff: keep echo, AD-024 (the draft).** -Better: zero churn; the heredoc harness (`cljw - <` and clj.** -Better: true clj parity; one mental model — "stdin and file are both scripts, -only `-e` and the REPL echo"; no AD needed; `cat f | cljw -` == `cljw f` holds. -This is what shipped-cljw `-` *should* do if tests were irrelevant. -Breaks: ~150 e2e heredoc fixtures that assert echoed values. Per F-002 this -rewrite cost is **not** grounds for rejection — migrate them to `-e`/`-M -`-style -echo harness or wrap expected output in explicit `(prn …)`. The migration is -mechanical and is the correct sink for the effort. - -**Alt 3 — Wildcard: split the surface — `-` = no-echo (clj-faithful), add -explicit `-M -` or a new `--eval-print -`/`-p` flag for the echoing batch mode.** -Better: gives both the clj-parity default *and* a first-class, *named* pipe-eval-print -mode (so the heredoc harness migrates to one explicit flag, not 150 rewrites — -far cheaper than Alt 2 while staying parity-clean); the echo behaviour becomes a -documented feature, not a silent divergence. -Breaks: adds CLI surface clj doesn't have (mild F-011 footprint, but additive, -not a divergence on a shared form); the new flag needs its own tests/help text; -risks scope-creep into the deferred REPL/arg-parsing cycle. - -**Recommendation:** lead with Alt 2 (finished-form-clean, clj-faithful); if a -named echo mode is independently wanted, Alt 3 captures it cheaply without -compromising `-` parity. Do not lead with Alt 1/AD-024 — naming the test-churn -avoidance "an accepted divergence" is exactly the dressed-up smallest-diff the -prompt warns against. - -**Main-loop disposition**: adopt **Alt 2** (the DA's lead). Alt 3's named echo -flag is declined — a batch eval-print mode is thin niche surface that a clean -cljw (and clj) would not carry just to ease test migration; the `(prn …)` -wrapping (Alt 2's path) is the honest cost. Alt 1/AD-024 is rejected as -rationalization. - -## Consequences - -- **Bench oracle unblocked**: file runs print only the program's output, so - `run_bench.sh`'s `head -1` equals the result (STREAM 2 proceeds with no - harness change). -- **Demos/bench output** lose the top-level echo noise. -- **D-322** owes: stdin `-` no-echo (56-file e2e migration via `(prn …)`) + - no-args → REPL. Until discharged, `cljw -` still echoes (the transient - inconsistency Decision D names). -- **`--version`/`--help`** report `build.zig.zon`-derived version; the user's - release tag is the SSOT for the value. - -## Affected files - -- `build.zig` (version option from `build.zig.zon`). -- `src/app/cli.zig` (`print_results` from source kind; `--version`; `--help` - banner). -- `test/e2e/phase3_cli.sh` (file no-echo contract + `--version`/`--help` cases). -- `test/e2e/phase14_with_context.sh` (file value cases wrapped in `(prn …)`). -- `.dev/debt.yaml` (D-322). diff --git a/.dev/decisions/0118_error_display_v0_level.md b/.dev/decisions/0118_error_display_v0_level.md deleted file mode 100644 index ae6edcadc..000000000 --- a/.dev/decisions/0118_error_display_v0_level.md +++ /dev/null @@ -1,431 +0,0 @@ -# ADR-0118 — v0-level error display: loc back-fill + frame trace + numbered-context renderer - -- **Status**: Proposed → Accepted -- **Date**: 2026-06-08 -- **Opens**: **D-323** (the multi-cycle implementation tracker) + folds the two - clear telemetry bugs (EDN `:file "unknown"`, eval `:line 0 :column 0`). -- **Cross-refs**: ADR-0055 (one `Info` → text + EDN in lockstep — the renderer - changes MUST land in both formats), ADR-0059 / AD-007 (no-JVM: error Kind is - the surface, not a JVM class — this ADR keeps cljw-native labels, does NOT - adopt clj's `Execution error (Class)`), `dual_backend_parity.md` (frame push - must land in TreeWalk + VM with a differential test), F-002 (finished-form - wins), F-011. Reference impl: cw v0 (`~/Documents/MyProducts/ClojureWasm/`, - v0.5.0) — survey at `private/notes/v0-error-span-trace-survey.md`. - -## Context - -The v1 redesign regressed the error DISPLAY relative to v0. A runtime/eval -error in v1 renders: - -``` -<-e>:0:0: arithmetic_error [eval] -Divide by zero -``` - -— no source location (`0:0`), no caret, no surrounding-source window, no stack -trace. Only parse/analysis errors (which the reader/analyzer locate) get a -single source line + a lone `^`. v0, by contrast, located eval errors at -runtime and rendered a numbered context window, an inline `^--- message` caret, -and a `Trace:` of fn frames: - -``` -Arithmetic error at /tmp/nested.clj:1:17 - Divide by zero -Trace: - user/f (/tmp/nested.clj:1) - user/g (/tmp/nested.clj:2) - - 1 | (defn f [x] (/ x 0)) - ^--- Divide by zero - 2 | (defn g [y] (f y)) - 3 | (g 10) -``` - -The user (2026-06-08) confirmed v0's display was materially better and wants -v1 to reach v0-level error display — **including eval-error location and stack -trace** — by the CFP submission. The error SYSTEM stays cljw-native (no JVM -class fiction, per AD-007); only the DISPLAY is being raised to v0 level. - -**The carrier chain already exists in v1.** Reader Forms carry `.location`, -every Node carries `loc`, and `loc` is threaded through `callFn` into hundreds -of `raise(.code, n.loc, …)` sites. The gaps are narrow and well-identified -(survey Q5). - -## Decision A — loc back-fill via the live frame stack (closes `0:0`) — **DA Alt 2** - -Eval errors surface as `0:0` because deep raise sites pass an empty loc -(`raise(.code, .{}, …)` — e.g. VM ops `vm.zig:237,245,…`, and primitives that -raise with a default loc). - -The draft's first shape (TreeWalk node back-fill + a **VM per-IP `lines`/ -`columns` Chunk side table**) was rejected by the Devil's-advocate as the -weakest load-bearing choice: two divergent loc-recovery mechanisms = two test -matrices + a Chunk-size regression on **every** compiled unit. Adopted instead -(DA Alt 2 — strictly cleaner, finished-form): - -- **TreeWalk**: back-fills from the node `n` in hand at each eval arm (free — - no table) for sub-form precision. `evalCall` already threads `n.loc` into - `callFn`, so the TreeWalk path largely works; audit any `raise(…, .{}, …)`. - -- **VM** (Revision 1, 2026-06-08 — corrects the DA Alt 2 premise): the DA's - "the frame stack carries the loc, no VM table needed" assumed the call-site - `loc` is **already threaded into `callFn` in the VM**. It is NOT — `vm.zig:386` - `op_call` calls `vt.callFn(rt, env, callee, args, .{})` with an **empty loc**, - so the divide-by-zero primitive (which DOES take a `loc`, `math.zig:142`) - receives `.{}` → `0:0`. The VM operand stack discards node identity, so the - call-site loc must come from the **bytecode** (compile-time known: `CallNode` - carries `loc`). The finished-form carrier, matching v1's established - **sparse-side-table-on-`BytecodeChunk`** pattern (`call_sites` / `libspecs` / - `ns_filters` / `ctor_sites` / `import_sites`, all indexed off an operand): a - parallel **`locs: []SourceLocation`** array on the chunk, indexed by - instruction index (IP). The compiler populates it from each node's `loc` at - `emit`; the VM passes `chunk.locs[ip]` into `callFn` at `op_call` (and the - direct-raising ops annotate from it). The frame push (Decision B) reads its - loc from the same `locs[call_ip]`. - - This is v0's per-IP `lines`/`columns` approach re-derived onto v1's `Chunk`. - The DA's bloat objection stands as a real cost (≈ one `SourceLocation` per - instruction) but is **necessary** — there is no free loc in the VM to reuse; - the cost is bounded by code size and lives only on compiled fns. A later O-NNN - may sparsify it (only call/raise IPs) if it measures; per-IP is the simplest - correct first form. - -Start at sub-form (node) precision; add arg-precise carets (v0's 8-slot -`arg_sources`) only if a sweep shows it matters. - -## Decision B — frame-stack for `Trace:`, **pop-on-both + snapshot-at-raise** (DA correctness fix) - -`info.zig:180-256` already defines `StackFrame` + a 64-frame `call_stack` + -`pushFrame`/`popFrame`/`getCallStack`/`clearCallStack` — **never called**. Revive -it, pushing an error-frame at the single shared `callMethodImpl`/`callFn` choke -point (better than v0's two separate TreeWalk+VM push sites; both backends get -it from one site). - -**Pop on BOTH success and unwind — NOT pop-on-success-only.** The DA caught a -correctness trap: pop-on-success-only leaks frames on every *non-error* non-local -exit (`recur` is signalled not errored; a `try`-caught exception recovers; a -`reduced` early return) — stale frames then poison the *next* unrelated error's -trace. Instead, **snapshot the live frame stack into `Info` at `setErrorFmt`/ -raise time** and pop on both paths via `defer` — this reuses v1's existing -dynamic-context discipline (`info.zig:212`: the binding frame is popped during -unwind, so its content is snapshotted into `Info.context` before the renderer -reads it). The renderer reads the per-`Info` trace snapshot, never the (by then -empty) live stack. A caught-and-recovered error's `Info` is discarded with its -snapshot, so no cross-error poisoning. - -Do NOT fold the error-frame onto the `env_mod` binding-frame stack as a *shared* -stack, but DO mirror its snapshot-on-raise + pop-on-both lifecycle. Per -`dual_backend_parity.md`, the push lands in TreeWalk AND the VM call op in the -same change with a differential test asserting equal traces. - -## Decision C — renderer upgrade, two formats in lockstep (ADR-0055) - -Extend `formatErrorWithContext` (`runtime/error/print.zig`): a numbered ±2-line -window (` N | ` gutter, `countDigits` width), the error line bright with -a `^--- ` caret tail (column-padded), surrounding lines dim, then a -`Trace:` walk (innermost first, ` / (:)`). The EDN emitter -(`error_render.zig`) gains a parallel `:trace [{:ns :fn :file :line} …]` vector -**in the same change** — the text and EDN formats never drift (ADR-0055). This -is v1's deliberate divergence from v0 (v0 had only a text renderer). - -Optionally add a natural-name Kind label (`arithmetic_error` → `Arithmetic -error`) — but only via `info.kindLabel()` so text + EDN stay consistent; the -EDN keeps the `:kind :arithmetic_error` keyword (machine surface), the text -gets the natural label. - -## Decision D — spans stay on Node/Chunk, never on Values - -NaN-boxed 64-bit Values have no spare bits for a span and a Value↔span side-map -would be a GC-rooting liability. Source info lives on the compiled tree (Node -`loc`, already present) + the VM per-IP `Chunk` table (Decision A). Same choice -as v0 (off Values), different reason (v0: avoid bloating the tagged union; v1: -NaN-box has no room + GC-root hazard) — a `no_copy_from_v1.md` re-derivation. - -## Decision E — fold the two clear telemetry bugs - -- **EDN `:file "unknown"`**: the `-e` source label `<-e>` is present in the text - path but the EDN emits `:file "unknown"` — the label is dropped. Thread the - real source label into the EDN `:file`. -- **eval `:line 0 :column 0`**: "unknown position" rendered as a real `0` - misleads telemetry consumers. Once Decision A back-fills most evals, a - genuinely-unknown position should emit `:line nil`/omitted, not `0`. - -These ride the relevant cycle (E.1 with the renderer/EDN work, E.2 with the -back-fill) per the user's "fix bugs the moment found" directive. - -## Decision F — staging (3 + bugfix cycles, D-323) - -Independent TDD cycles, highest-value first: - -1. **Cycle 1 — loc back-fill (A) + eval position bug (E.2)**: closes the visible - `0:0` regression. TreeWalk node back-fill, VM per-IP table + annotator. -2. **Cycle 2 — renderer window + caret tail (C, text) + EDN `:file` bug (E.1)**: - the numbered context + `^--- msg` for the cases now located. -3. **Cycle 3 — frame trace (B) + `Trace:` render (C) + `:trace` EDN (C)**: the - stack-trace capture + render, dual-backend parity test. - -run-mode (`-M`/`-m`/`-X`) errors inherit all three uniformly (they flow through -the same eval/raise path; the synthetic `<-M>`/`<-X>` labels and the generic -`exception` Kind on `-X` throw are tracked as follow-on polish in D-323, not -blockers). - -## Alternatives considered (Devil's-advocate, fresh-context fork) - -**Alt 1 — Smallest-diff: loc back-fill only, no frame trace, text-only renderer.** -Ship choice A (close `0:0`) + a renderer that prints the context window + caret, -defer B (trace) and the EDN `:trace` to a later cycle. *Better:* zero new -threadlocal state on the hot call path; no pop-on-success-only correctness -surface; lands the highest-value UX win (eval location + caret) immediately. -*Breaks:* it is **smallest-diff bias dressed up** — the user explicitly named -"incl. eval location AND stack trace" as the v0-parity bar, so deferring B -misses the stated goal (F-002: the finished form is the whole v0 display, not -its cheaper half). It also tempts an ADR-0055 violation by touching only the -text emitter "for now" and back-filling `:trace` later, which is exactly the -drift ADR-0055 forbids. Reject on F-002 + ADR-0055 grounds. - -**Alt 2 — Finished-form-clean: unify location and trace under one carrier, drop -the VM per-IP side table.** The draft's TreeWalk-node-vs-VM-IP-table **split is -the weakest load-bearing choice**. Two divergent loc-recovery mechanisms means -two test matrices and two drift surfaces forever. Cleaner: make the *frame push -itself* the single loc carrier. At the shared `callFn` choke point, the pushed -`StackFrame` already records `{fn_name, ns, file, line, column}` from the -call-site `loc` that is **already threaded into `callFn`** (confirmed: hundreds -of `raise(.code, n.loc, …)` sites + `BuiltinFn` takes `loc`). So the *innermost -live frame's* loc is the back-fill source for *both* backends — no per-IP Chunk -side table, no per-arm TreeWalk back-fill, no Chunk bloat across every compiled -unit. The VM's operand stack discarding node identity stops mattering because -the frame stack (not the IP) carries location. *Better:* one mechanism, one -test, no Chunk size regression, backend parity falls out for free instead of -needing a dedicated parity test. *Breaks:* a `raise` *between* calls (e.g. a bad -arg checked before any sub-call) has no fresher frame than the enclosing one, so -its loc is the call site, not the sub-expression — slightly coarser than -per-node back-fill. Mitigation: keep choice A's node back-fill *only* in -TreeWalk where the node is in hand for free (no table), and let the frame loc -cover the VM. This is strictly cleaner than a bespoke per-IP table and should be -the pick. - -**Correctness trap the draft underweights (pop-on-success-only):** -pop-on-success-only **leaks frames on every non-local exit that is not an -error** — `recur` (signalled, not errored, in both `tree_walk.zig` and -`vm.zig`), `try`-caught exceptions, and early returns from `reduced`. The -draft's "survives the unwind" reasoning is correct *only* for the uncaught-error -path; a caught-and-recovered error leaves stale frames that poison the *next* -unrelated error's trace. The `binding`-stack analogy the draft rejects is -actually the right shape: frames must pop on **both** success and unwind, with -the trace **snapshotted into `Info.context` at `setErrorFmt` time** — which v1 -already does for dynamic context (`info.zig:212`, the "binding frame is popped -during unwind, before the renderer reads this" comment). Reuse that exact -snapshot discipline for the trace and pop-on-both; do not invent -pop-on-success-only. - -**Alt 3 — Wildcard: no threadlocal stack; reconstruct the trace from the GC -root-set / EvalFrame chain at raise time.** v1 already maintains rooted -`EvalFrame`s (per `gc_rooting.md`). Walk that live chain at `setErrorFmt` to -materialise the trace, instead of a parallel 64-slot threadlocal. *Better:* zero -hot-path push/pop cost, zero leak surface (the chain *is* the truth, nothing to -pop), no `max_call_depth` truncation lie. *Breaks:* couples error display to -GC-root internals (a moving-GC migration now must preserve frame fn-name/ns, -widening the `gc_rooting.md` contract); EvalFrame may not carry `fn_name`/`ns` -today, so it needs enrichment that touches the hottest struct. Higher risk than -Alt 2 for equal UX. Hold as the perf fallback if the threadlocal push/pop ever -measures on the call path. - -**Recommendation (non-binding, F-002):** Alt 2. It removes the per-IP Chunk side -table (a real bloat-every-Chunk cost the draft accepts) and the dual-mechanism -test burden, and forces the pop-on-both + snapshot fix that the draft's -pop-on-success-only gets wrong. The 64-frame threadlocal cost is two pointer -writes per call — negligible versus the back-fill it replaces — and -`max_call_depth` truncation is already documented as best-effort. Keep choice C -and D as drafted (EDN `:trace` in lockstep per ADR-0055; spans never on -NaN-boxed Values). - -**Main-loop disposition**: adopt **Alt 2's two correct contributions** — (1) the -revived frame stack with **pop-on-both + snapshot-at-raise** replacing the draft's -leaky pop-on-success-only (the DA's most valuable catch), and (2) TreeWalk node -back-fill where the node is free. **BUT Decision A Revision 1 reverses Alt 2's -"no VM per-IP table"**: implementation (2026-06-08) found the VM threads NO -call-site loc into `callFn` (`vm.zig:386` passes `.{}`), so the frame's loc has -no free source in the VM — a per-IP `locs` array on `BytecodeChunk` is necessary -(re-deriving v0's per-IP lines/columns onto v1's Chunk, matching v1's existing -sparse-side-table precedent). The DA's bloat objection is acknowledged as a real -(bounded, sparsifiable-later) cost, not a blocker. Alt 3 (GC-EvalFrame -reconstruction) is held as the perf fallback. Decisions C + D stand as drafted. - -## Consequences - -- Eval errors (incl. run-mode) gain location + numbered context + caret + trace - — v0-level display, cljw-native (no JVM class). -- **VM `Chunk` grows a per-IP `locs: []SourceLocation` array** (Decision A - Revision 1 — the DA's "no table" premise did not hold for the VM, which - threads no call-site loc into `callFn`). Cost ≈ one `SourceLocation` per - instruction, bounded by code size, on compiled fns only; a later O-NNN may - sparsify to call/raise IPs. Plus the frame stack: a 64-frame threadlocal + - two pointer writes per call (negligible; `max_call_depth` best-effort). -- The text + EDN renderers stay in lockstep (ADR-0055); both gain `:trace`. -- D-323 tracks the 3-cycle implementation + the run-mode label/Kind polish. - -## Affected files (anticipated) - -- `src/eval/backend/tree_walk.zig` (node loc back-fill + error-frame push at the - shared `callFn`/`callMethodImpl` choke point). -- `src/eval/backend/vm.zig` (error-frame push at the VM call op — same frame - API; NO per-IP Chunk table per Alt 2). -- `src/runtime/error/info.zig` (revive frame-stack use; snapshot trace into - `Info` at raise; position nil when unknown). -- `src/runtime/error/print.zig` (numbered window + caret tail + trace). -- `src/app/error_render.zig` (EDN `:file` fix + `:trace` vector). -- `test/e2e/*`, `test/diff/*` (eval-loc + trace cases, dual-backend parity). - -## Revision 2 (2026-06-08) — arg-precise carets promoted from deferred to REQUIRED - -**Trigger.** Decision A's last line said: *"Start at sub-form (node) precision; -add arg-precise carets (v0's 8-slot `arg_sources`) only if a sweep shows it -matters."* The user gave a direct directive (2026-06-08): the caret MUST land on -the culprit — `(/ 2 0)` on the `0`, not the `(`; nested `(+ 1 (/ 2 0))` on the -innermost; UNIFORM discipline, not per-primitive ad-hoc. The user IS the sweep -signal. This Revision promotes arg-precise carets to **required**, landing as a -distinct **cycle 2.5** (after the cycle-2 renderer, before the cycle-3 trace). -Cycle 2 (the renderer) is committed + pushed (6d36acc4); it renders the caret at -whatever `info.location` says, so it needs no change — 2.5 changes *what loc is -attributed*, not the renderer. - -**Decision (main loop): Alt 1 — threadlocal `arg_sources[N]` + primitive names -the culprit index.** Before invoking a primitive, the dispatch choke point -records each evaluated argument's loc into a threadlocal `arg_sources` side -channel (same storage class as `info.zig`'s existing `last_error` / `call_stack`). -A primitive that fails on argument *i* raises with `getArgSource(i)`. Back-fill of -the enclosing form happens only when no fresher loc is set (`line == 0` guard) → -depth-first eval makes the innermost culprit win for free. The primitive's only -new job is to NAME the culprit index — it never carries/computes a loc, and the -`BuiltinFn` ABI (`info.zig:262`, `(args, loc)`) is UNCHANGED (F-009 untouched). - -**Premise correction (supersedes the Rev-1 Consequences "Chunk grows a `locs[]` -array" line).** The DA found that the VM ALREADY carries per-operand loc: each -`Instruction` has `line`/`column` (`opcode.zig:362`), `compileNode` stamps every -instruction from its node's `loc()` (`compiler.zig:143`), and `compileCall` -compiles each arg via `compileNode(a)` BEFORE re-stamping the call-form loc onto -`op_call` (`compiler.zig:305`) — so the arg-pushing instruction already bears the -arg's column. Cycle 1 correctly used this per-instruction carrier (NOT a separate -`Chunk.locs[]` array — confirmed via `git show 0c36cc8a`). So: **no new -`BytecodeChunk.locs[]` table** (it would be a second copy = bloat-for-symmetry -smell); the VM arg-loc fill reads the existing instruction metadata. - -**Why Alt 1 and NOT the DA-recommended Alt 2 (an F-002 correctness call, not a -cycle-budget defer).** Alt 2's headline advantage — "primitive never names an -index", the eval-loc-stack top IS the culprit — assumes the primitive raises -*while descending into* the culprit sub-expression. **v1 does not work that way**: -the backend evaluates ALL args eagerly (`tree_walk.zig:1019` loop), THEN invokes -the primitive with `args: []const Value`. At raise time the eval-loc-stack top is -the *last* arg, not an arbitrary culprit (`(+ :foo 1)` → top is `1`, culprit is -`:foo` at index 0). To select the right arg, Alt 2 must still consult the index → -its premise collapses and it degenerates to "Alt 1 with a heavier hot path". So -for eager-arg v1, Alt 1 ("primitive names the index") is the *actual* finished -form. (Alt 2 would be right only under a lazy/CPS evaluator that suspends inside -arg evaluation — not v1's model. If v1 ever moves there, revisit.) Both DA hard -constraints are adopted regardless: (1) the channel is **threadlocal, never a -`Runtime` field** (`Runtime` is process-shared, `runtime.zig:6`, → races under -Phase-15 concurrency; threadlocal inherits v1's per-thread error-state story); (2) -**no `BytecodeChunk.locs[]` table** (per above). - -**Span/underline (start+end) is DEFERRED, explicitly.** A single `^` on the -correct column meets the directive. start+end needs an end-position threaded -through reader/tokenizer (`Form`/`Node` carry only line+col today) + multi-byte -display-width work — its own reader-surface change. Follow-on O-NNN under D-323. -Decisions C (text+EDN lockstep) and D (never on Values) stand; the EDN position -field is populated from the same resolved loc the text caret uses. - -**Boundary (state, don't over-claim uniformity).** A primitive that raises -*between* sub-evaluations, or from a variadic fold / lazy-seq element where the -culprit is not a positional arg (`(apply / coll)`; a reduce failing on the 4th -realised element of a lazy seq that was arg 0), falls back to the call-form loc -via `argLoc`'s `orelse fallback` — no worse than today, but not arg-precise. (The -as-built slice channel removes v0's `N=8` slot truncation; see "Implementation -landed" below.) - -**Cycle 2.5 edit sites + the verbatim DA output** are mirrored in -`private/notes/phase14-error-cycle2.5-caret-precision-plan.md` (gitignored -working memory). The load-bearing decision, constraints, and rationale are this -section (tracked); the plan note adds the file:line implementation map. - -**Implementation landed (2026-06-08, cycle 2.5).** Two refinements over the -planned prose, both finished-form improvements: - -- **The side channel is a `[]const SourceLocation` slice, not a fixed `[N]` - array.** `info.zig` holds `threadlocal var arg_sources: []const SourceLocation` - plus `swapArgSources` / `getArgSource` / `argLoc(i, fallback)`. Each call - boundary publishes a slice into its own stack-resident loc array (alive for the - synchronous call) and swaps the parent's slice back on return (stack- - disciplined). This **removes the v0 `N=8` truncation boundary entirely** — any - arity is covered, so the only remaining fallback is the non-positional-culprit - case (variadic fold / lazy-seq element / between-subcall), which `argLoc`'s - `orelse fallback` handles. -- **Innermost-wins falls out of depth-first eval + the swap/restore**, not a - `line == 0` back-fill guard. The inner call's boundary publishes its args, - the inner primitive raises reading them, and the error short-circuits before - the outer boundary publishes — so the innermost culprit's loc is the one baked - into `last_error`, on both backends. - -TreeWalk publishes in `evalCall`'s arg loop (`a.loc()` in hand). The VM keeps a -parallel `loc_stack` mirroring the operand stack: a post-switch sweep stamps each -newly-pushed operand with its instruction's compiled loc (uniform, one site), and -`op_call` reads `loc_stack[sp+1 .. sp+1+arg_count]` as the per-arg locs. Primitives -name the culprit index via `argLoc(i, loc)` — applied uniformly across the whole -arithmetic divide/type family (`/`, `quot`, `rem`, `mod`, `unchecked-divide-int`, -`unchecked-remainder-int`, and `ensureNumeric` for `+`/`-`/`*`). Other primitives' -arg-positional raises adopt the same one-token idiom incrementally; until then they -fall back to the call-form loc (no worse than before). Dual-backend parity is -locked by a `diff_test.zig` case asserting the caret column is equal (and exact) -on TreeWalk + VM for `(/ 2 0)`, `(+ :foo 1)`, and nested `(+ 1 (/ 2 0))`. - -### Alternatives considered (Devil's-advocate, fresh-context fork, Rev 2) - -**Premise correction the DA found first.** (As above — `Instruction` already -carries `line/column` at `opcode.zig:362`; `compileNode` stamps at -`compiler.zig:143`; `compileCall` compiles args before re-stamping the call loc -onto `op_call` at `compiler.zig:305`; so arg-pushing instructions already bear -per-operand locs and no parallel `Chunk.locs[]` array is needed.) - -**Alt 1 — Smallest-diff: threadlocal `arg_sources[N]`, primitives name the -index.** Reuses `info.zig:173-189`'s threadlocal class. TreeWalk fills in -`evalCall`'s arg loop (each `a.loc()` free); the VM fills at `op_call` from the -arg-pushing instructions' loc. Primitives raise with `getArgSource(i)` -(`ensureNumeric` already has `i` in scope). *Better:* zero new hot-path struct, -zero ABI change to the ~200 `BuiltinFn`s (F-009 intact), nested correctness free -from depth-first + `line==0` back-fill, identical in both backends. *Costs:* (a) -thread-safety inherits v1's threadlocal error-state story (safer than a shared -`Runtime` field — reject that option); (b) a real seam — variadic / lazy-element -/ between-subcall culprits fall back to call-form loc (slot `N` truncates); state -that boundary. - -**Alt 2 — Finished-form-clean (DA's pick): an evaluation-position stack -co-located with the call-frame stack.** Push the loc of the expression being -reduced at every value-producing point; pop as consumed; primitive raises -empty-loc and the back-fill reads stack top. Unifies the arg-precise + variadic -cases and merges Decision A's caret carrier with Decision B's trace carrier (one -threadlocal push/pop, one pop-on-both lifecycle). *Better:* removes the index -convention entirely (strongest F-009); closes the variadic seam structurally. -*Costs:* larger diff (every value-producing site push/pops; pop-on-both for -`recur`/caught-`throw`/`reduced` is the known-hard surface Rev-1 flagged for -frames); granularity subtlety (call-form-vs-culprit depends on *when* the -primitive raises relative to consuming args); two writes per sub-expression vs -per call. **Main-loop rebuttal (why rejected):** under v1's eager-arg model the -stack top at raise time is the last arg, not the culprit, so Alt 2 still needs -the index — its core advantage evaporates; see "Why Alt 1" above. - -**Alt 3 — Wildcard: resolve the culprit loc lazily at render time** from the -error's frame + the originating Node/Chunk, with a Code→position table -(`divide_by_zero` → divisor; `type_arg_not_number` → first non-numeric operand). -*Better:* zero hot-path cost, zero ABI/threadlocal change, one resolution site. -*Costs:* fragile — the primitive's actual runtime decision (a fold that failed on -element 3; a cast that tried `a` before `b`) is gone by render time, so a -Code→position table encodes only the *typical* culprit (data-dependent culprits -become un-fixable); inverts the dependency (errors-display reads eval/compiler -internals at render time, widening the renderer contract — opposite of Decision -D); the VM backward scan must be re-plumbed into the `Info` snapshot anyway, so -"zero cost" partly evaporates. Hold only as a perf fallback if Alt 1/2's push -path ever measures. - -**DA recommendation (non-binding):** Alt 2 on F-002 grounds. **Main loop chose -Alt 1** for the eager-arg correctness reason above (Alt 2 is not actually clean -under v1's evaluation model), adopting both DA hard constraints (threadlocal; no -`locs[]` table) and the span-defer. diff --git a/.dev/decisions/0119_callable_naming_surface.md b/.dev/decisions/0119_callable_naming_surface.md deleted file mode 100644 index bbc28ae9a..000000000 --- a/.dev/decisions/0119_callable_naming_surface.md +++ /dev/null @@ -1,246 +0,0 @@ -# ADR-0119 — Callable naming surface (restore fn names dropped in the v1 redesign) - -- **Status**: Proposed → Accepted (2026-06-08) -- **Depends on / feeds**: ADR-0118 (error display) cycle 3 (`Trace:`) consumes - the names this ADR restores. This ADR is the foundation; the trace is one - consumer (`pr`/metadata are others). -- **Supersedes the cycle-3 framing in**: ADR-0118 Decision B + the cycle-3 plan - note's Alt-2 lean. The frame-NAME source is now "name on the value" (Alt 3), - not the two-call-site Var read (Alt 2) — see Decision + Alternatives. - -## Context — a redesign regression, verified against the textbooks - -cw v1's `Function` (tree_walk.zig:129) carries **no name and no defining_ns** -(only `header / slot_base / methods / variadic / closure_bindings`); the -analyzer `FnNode` (node.zig:212) is likewise nameless. So a running function -does not know its own name. This was found while scoping the cycle-3 stack -trace: a trace needs "which of MY fns was running", which the fn value cannot -answer. - -This is a **regression introduced by the v1 ground-up redesign**, not an -inherent design choice — verified in the reference clones: - -- **cw v0 HAD it**: `FnProto.name: ?[]const u8` (v0 chunk.zig:133, bytecode), - `Closure.fn_node.name` (treewalk), `Fn.defining_ns` (v0 value.zig:869). v0 - could render `/` trace frames (`"anonymous"` fallback). -- **Clojure JVM**: EVERY fn gets a compile-time name baked into its class name - (`Compiler.java:4558-4569`): named → `foo__`, **anonymous → `fn__`**; - recovered in traces via `Compiler/demunge`. -- **Babashka / SCI** (the closest analog — JVM-less interpreter): the fn carries - its name via metadata + the resolved Var symbol captured at the call boundary. - -The redesign focused on NaN-boxing / GC / value representation and silently -dropped the name field; nothing forced it to be carried forward. This ADR -restores it as a **foundational** capability (traces + `pr` + future -`(:name (meta #'f))`), matching all three textbooks. - -### The callable surface is wider than `.fn_val` — and most kinds already self-name - -cw v1 has many IFn-implementing callables (the `treeWalkCall` switch, -tree_walk.zig:1040) plus a separate interop-method family. Crucially, **only -plain `.fn_val` is missing its name** — the siblings already carry theirs: - -| callable | name source | trace disposition | -|------------------------------------------------------------------|---------------------------------------------------------------|-----------------------------| -| `.fn_val` (plain user fn) | **NONE today → restored here** | PUSH | -| `.builtin_fn` | immediate NaN-boxed ptr (value.zig:221) — cannot hold a name | ELIDE | -| `.multi_fn` | `MultiFn.name: Value` (multimethod.zig:63) | PUSH | -| `.protocol_fn` | `descriptor.fqcn()` + `methodName()` (protocol.zig:56,132) | PUSH | -| `.keyword`/`.symbol` as IFn | the value itself | ELIDE (data accessor) | -| `.array_map/.hash_map/.hash_set/.vector/.sorted_map/.sorted_set` | none (accessor) | ELIDE | -| `.var_ref` | `Var{ns,name}` (env.zig:80-83); re-dispatches | name the var / skip | -| `.typed_instance`/`.reified_instance` IFn `-invoke` | TypeDescriptor fqcn | PUSH (optional) | -| interop `(.m o)` / `(T/m x)` | InteropCallNode method + descriptor (area D) | secondary family (deferred) | - -That `.fn_val` is the **only** kind without a self-name is the consistency -argument that decides Alt 3 over Alt 2 (below): the siblings name from the -*value*, so plain fns should too. - -### Scope: eval-phase runtime trace only - -A trace is the **eval-phase** call stack. Macros are expanded away before -execution (macroexpand phase), so they never appear as runtime frames -(`(when x body)` shows `body`, not `when`). Special forms are control flow, not -callables — no frames. parse / analysis / macroexpand errors are located by -`Info.phase` + loc but are NOT framed by this mechanism (a macroexpansion trace -would be a separate, future concern). Record this so "why is there no `when` -frame" is not chased as a bug. - -## Decision - -Restore the name on the function **value** (Alt 3), give every callable a -uniform name resolver, and consume it in `pr` (Stage 1) and the `Trace:` -renderer (Stage 2). - -1. **`FnNode` + `Function` gain `name: ?[]const u8` and `defining_ns: - ?[]const u8`.** The name lives on the value, so the single shared choke - point `treeWalkCall` can read it — this is Alt 3's win over Alt 2's two - call-site reads. -2. **String ownership = borrow the analyzer arena** (`rt.load_arena`, - session-lifetime), exactly as `FunctionMethod.params`/`body` already do — NOT - `gpa.dupe`, NOT intern. Safe because cw v1 never resets the analyze/load - arena mid-session (loader.zig:58, repl.zig:44). **This unwritten invariant - (which `params`/`body` already silently ride) is hereby documented**: a - future per-form-reset arena would UAF the name AND `params`/`body` — a shared - invariant, not one this cycle introduces. -3. **Analyzer threading (3 cases now, 1 deferred)**: `analyzeDef` post-patches - the analyzed `.fn_node` with `name_sym.name` + `env.current_ns.?.name` (covers - `defn` macro→def→fn* AND raw `(def x (fn ..))`); `analyzeFnStar` mints a - gensym `fn__` (via existing `rt.gensym`) for anonymous fns + sets - `defining_ns`; `analyzeLetfn` post-patches each binding's fn with the binding - name. The `(fn name ..)` self-name case (lowered to `letfn*` today, so the fn - stays anonymous) needs a new `analyzeFnStar` self-name arm — **deferred** - (D-325), rare. -4. **`defining_ns` is display/metadata only.** v1 resolves every symbol to a - `*const Var` at analyze time (analyzer.zig:698), so — unlike v0 — there is no - per-call `current_ns` restore and none is needed. `defining_ns` feeds the - trace `ns` column + future Var-fn metadata, nothing else. (Recorded so no one - re-introduces a v0-style restore expecting it to matter.) -5. **Anonymous fns are gensym-named `fn__`** (JVM parity), not left null. -6. **`calleeName(callee) -> ?{ns, name}`** resolver (Stage 2) switches over the - callable tags per the table above; ELIDE arms (builtin / collection / bare - keyword) return null = no frame. Builtins are elided from traces (the cw - analog of clj's `RestFn`/`AFn` plumbing); their `pr`-name (reverse ptr→Var - map) is a separate deferred concern (D-327). -7. **Single-choke-point push** at `treeWalkCall` entry: `calleeName` → `pushFrame` - + `defer popFrame()`. `.var_ref` re-dispatches (tree_walk.zig:1061) — **skip - the push on `.var_ref`** and let the inner `.fn_val` arm push (simpler than - naming the var then deduping). Snapshot `getCallStack()` into a new - `Info.trace` at `setErrorFmt` time; pop-on-both via `defer` (the - recur/try/reduced-safe lifecycle from ADR-0118 Decision B). -8. **Interop method frames** (`(.m o)` / `(T/m x)`, a SECOND push family - distinct from IFn dispatch) are **deferred** (D-326). - -### Staging (each stage: lightweight local verify → full gate at the commit) - -- **Stage 1 — naming threaded onto the value, verified white-box.** Struct - fields + allocator literals (tree_walk.zig:263/301/399) + VM reconstruct - (vm.zig:499) + analyzer threading (cases 1/3/4). Observable = unit + diff - tests that read `Function.name`/`defining_ns` after analysis on BOTH backends - (the VM reconstruct is the only divergence risk). **`pr`/`str` of a fn is - deferred (D-328)**, not bundled here: a user-facing fn print form couples to - the `(class fn)` label (currently the raw tag `fn_val`) and the `#object[…]` - convention (AD-010), which is its own format + accepted-divergence design — - orthogonal to the name *threading* this stage lands. Keeping Stage 1 to the - threading + white-box verification avoids dragging a `(class fn)` redesign - into the naming foundation. (Refinement found at implementation start, - Step-0.6-style.) -- **Stage 2 — `Trace:` (ADR-0118 cycle 3).** `calleeName` resolver + revive the - frame stack + push at `treeWalkCall` + `Info.trace` snapshot + renderer - `Trace:` (text) + EDN `:trace` + decoder + `clearCallStack` per top-level - form. Dual-backend parity diff test: nested named-fn error → identical trace; - recur → a single frame (not N). -- **Stage 3 — deferred follow-ons** (debt rows D-325/326/327), not pushed here. - -## Alternatives considered (Devil's-advocate, fresh-context fork) - -A fresh-context DA was run on the frame-name source. Its three shapes (verbatim -intent) and the main-loop's choice: - -- **Alt 1 — smallest-diff**: push at the single choke point, name from the - callee *Value* (which has none for `.fn_val`) → render position-only / `"fn"`. - *Rejected*: guts the feature (a trace of unnamed `fn (…)` lines is useless); - Smallest-diff-bias smell — smaller precisely because it abandons the finished - form. -- **Alt 2 — finished-form for the CURRENT representation (DA's recommendation)**: - push at the TWO call sites (`evalCall` + `op_call`) where the Var name still - exists, via a shared `info.pushFrame` helper. Sources the name where it lives - (the Var, pre-deref). *Better*: no struct change; reuses the cycle-2.5 - parallel-array + `defer`-restore mechanism. *Costs*: two push sites = two - parity surfaces; cannot name HOF-passed fns (`(map h coll)` → `h` is - unrecoverable at the inner call); and crucially it makes `.fn_val` the **odd - one out** — named from the call site while its siblings (`.multi_fn`, - `.protocol_fn`) are named from the value. -- **Alt 3 — name on the value (DA's wildcard; CHOSEN)**: stamp `name` + - `defining_ns` onto `Function` at def/analyze time. *Better*: the name travels - with the callee everywhere (matching clj's class-name and v0's `proto.name`); - makes the single-choke-point push viable; names anonymous fns via gensym; and - — the deciding point the all-callables enumeration surfaced — makes `.fn_val` - **consistent with its siblings** (all callables self-name on the value). - *Costs*: touches the `Function` struct + FnNode + both backends' fn-allocation - + the analyzer (the heaviest diff), and the string-ownership question (resolved: - arena-borrow, area G). - -**Main-loop choice: Alt 3, NOT the DA-recommended Alt 2.** This is an F-002 -finished-form call, not a cycle-budget defer: (a) Alt 3 *restores a property v0 -+ clj + SCI all have* (the regression framing — Alt 2 would entrench a -half-named callable surface); (b) the sibling callables already name from the -value, so Alt 3 is the *consistent* shape and Alt 2 leaves `.fn_val` an -exception; (c) the name-on-value is foundational beyond traces (`pr`, metadata). -The DA itself rated Alt 3 the genuine finished form and flagged its only real -cost as analyzer threading (a clean, separable unit), not GC (a static -`[]const u8` is GC-inert). Alt 2's HOF-naming claim is *unrecoverable* anyway, -so its supposed parity advantage over Alt 3 does not exist. - -## Stage 2 implementation landed (2026-06-08) - -The `Trace:` consumer landed on the Stage-1 names. As-built refinements: - -- **`pushFrame` now returns `bool`** (recorded / dropped-at-cap). The push site - pops only when it actually pushed, so push/pop stays balanced past the - 64-frame cap (an unconditional `defer popFrame` would underflow). This means - the live `call_stack` self-zeroes between top-level forms (every `defer pop` - runs on success AND error return), so **no production `clearCallStack` is - needed** — a clean divergence from v0's defensive per-form clear. (Tests call - `clearCallStack` for isolation only.) -- **Single-choke-point push at `treeWalkCall`** via `calleeFrame(callee, loc)`: - `.fn_val`/`.multi_fn`/`.protocol_fn` push a value-sourced frame; everything - else (builtin, data-as-IFn, `.var_ref`) elides. Both backends share it (VM - op_call → vt.callFn). `Info.trace` is snapshotted into a threadlocal buffer at - `setErrorFmt` (mirrors the `.context` snapshot), read by the renderer after - the live stack has unwound. -- **Frame line = the CALL-SITE loc** (where the fn was invoked), not the - execution-point line clj/v0 show (v0 patched the innermost frame to the IP via - `updateTopFrame`). The precise error spot is already in the header + caret; - the trace gives the call chain. Execution-point precision is deferred (D-334). -- **Text `Trace:` + EDN `:trace` land in lockstep** (ADR-0055), innermost-first, - ` / (:)` with the same `file_label` fallback as the - header. The post-mortem `render-error` decoder's `:trace` parse is deferred - (D-333). Dual-backend parity + e2e (Case 11/12) cover both. -- **Post-happy-path verification backlog** (user-directed): trace under - multi-threading (D-329), async (D-330), multi-module require chains (D-331). - -### Trace-visibility discipline (D-332, user-directed — landed 2026-06-08) - -The user flagged that host/stdlib/user is inherently non-uniform (some core fns -are Zig builtins, some `.clj`, some AOT-loaded with names dropped, some nameless -internals) and asked for a **principled discipline, not ad-hoc per-fn choices**, -for good UX. Resolution: `tree_walk.isUserNs(ns)` — a frame is kept iff its -owning namespace is a USER ns; cljw reserves `clojure.*` / `cljw.*` for its -embedded stdlib, so a frame in those (or with a null ns — an unnamed internal / -host-built fn) is implementation and is elided, as are all host builtins. One -uniform ns rule makes the non-uniform implementation consistent: `(map userfn)` -(`.clj` `map` over the `-map-eager` builtin, possibly nameless internal frames) -and `(reduce userfn)` (builtin) now BOTH show only the user fn. The trace is -"your call chain"; the premise is cljw's impl is correct and the bug is in user -code. Diverges from clj (which shows clojure.core frames) — **AD-024**, pinned by -the e2e `error_trace_discipline`. A verbose/full-trace mode (clj `pst` style) -surfacing stdlib frames is a future option, not needed for the default UX. - -## Consequences - -- A function knows its own name → traces, `pr`, and future `(:name (meta #'f))` - all become possible. Stage 1 alone improves `pr` of fns. -- The string-ownership invariant ("analyze/load arena never resets mid-session") - is now documented; `params`/`body` already depended on it implicitly. -- `defining_ns` is display-only — no resolution-correctness coupling. -- Builtins, collection-as-IFn, bare keywords, special forms, and macros are - NOT framed (documented scope), so traces show only user-meaningful frames. -- Two backends touched → dual-backend parity diff tests are mandatory in the - Stage-1 and Stage-2 commits (ADR-0036). -- Deferred: `(fn name ..)` self-name (D-325), interop method frames (D-326), - builtin `pr`-name reverse map (D-327). - -## Affected files (the 8 change groups; full file:line map in -`private/notes/phase14-cycle3-naming-investigation.md`) - -1. Structs: node.zig `FnNode`, tree_walk.zig `Function`. -2. Allocators: tree_walk.zig:263/301/399 literals + vm.zig:499 reconstruct. -3. Analyzer: special_forms.zig `analyzeDef`, bindings.zig `analyzeFnStar` + - `analyzeLetfn`. -4. Resolver: new `calleeName` in tree_walk.zig (Stage 2). -5. Push: tree_walk.zig:1047 `treeWalkCall` entry (Stage 2). -6. Renderer + lifecycle: print.zig / error_render.zig / render_error.zig + - clearCallStack in the REPL/loader form loop (Stage 2). -7. Tests: diff_test.zig parity cases (both stages). -8. Print (Stage 1 consumer): the `.fn_val` render in print.zig. diff --git a/.dev/decisions/0120_cross_thread_error_fidelity.md b/.dev/decisions/0120_cross_thread_error_fidelity.md deleted file mode 100644 index 3141e8337..000000000 --- a/.dev/decisions/0120_cross_thread_error_fidelity.md +++ /dev/null @@ -1,204 +0,0 @@ -# ADR-0120 — Cross-thread error fidelity (uniform worker-error marshalling) - -- **Status**: Proposed → Accepted (2026-06-08) -- **Resolves**: D-335 (user-directed: uniform cross-thread error discipline, - not just the trace) + D-330 (future trace lost at the boundary). Composes with - ADR-0119 / AD-024 (the user-frame trace discipline). -- **Feeds**: D-115 (the Value-carried exception channel) — this is its resolution. - -## Context — a real, verified gap (user-flagged 2026-06-08) - -cljw spawns REAL OS threads: `future` (future.zig:103 `std.Thread.spawn`), -`agent` drainers (agent.zig:153/304), Phase-B `pmap`/workers. When a worker runs -user code that throws, the worker catches the Zig error and the FULL error — -kind / message / location / trace / ex-data — lives in THREADLOCAL state -(`info.zig` `last_error` + `msg_buf` + `trace_snapshot`) that DIES with the -thread. The consuming thread gets a degraded placeholder: - -- **future**: `stm.zig:58` raises a generic `future_thunk_failed` — loses - everything, even the message. `@(future (/ 1 0))` shows `future_thunk_failed`, - not `Arithmetic error: Divide by zero at :` + `Trace:`. -- **agent**: `agent.zig:247 captureThrown` converts to an `ex_info` Value — but - hardcodes the class `"clojure.lang.ExceptionInfo"` (a WRONG-CLASS bug: a - marshalled `(/ 1 0)` reports `ExceptionInfo`, not `ArithmeticException`, so - `(catch ArithmeticException …)` around the read fails to match) AND loses - location + trace. - -The user directive: full error FIDELITY across the boundary (not just the -trace), via ONE uniform discipline for future/agent/pmap — not ad-hoc per -construct. - -### The fidelity ceiling is in the RENDERER, not the marshal - -`error_render.zig:86 buildThrownInfo` is the only path that turns a thrown Value -back into an `Info`, and it recovers ONLY `message` + `data` — `location` stays -`.{}` (→ "unknown"), `trace` stays `null`. So ANY approach routed through the -thrown-Value channel inherits this ceiling unless `ExInfo` is widened. (This is -also a LATENT in-thread gap: `(throw (ex-info "x" {}))` already renders with -`location = "unknown"` today.) - -## Decision - -Adopt the Devil's-advocate's finished-form synthesis: **the cross-thread carrier -IS a location+trace-extended `ExInfo`, not a parallel type.** One exception -representation that is simultaneously user-catchable (ex-data intact), -GC-rooted/traced (rides the existing `Future.cached` / `Agent.error_val` / pmap -slot), and fully renderable (location + trace). - -1. **Widen `ExInfo`** (ex_info.zig) with `origin_loc: SourceLocation` (file bytes - GC-owned like `message`) and `trace: ?[]StackFrame` (frames deep-copied, - frame strings GC-owned). Populate at synthesis: `allocException` / - `op_throw` / `evalThrow` capture the live `info.location` + `getCallStack()`. -2. **Teach `buildThrownInfo`** (error_render.zig) to read `origin_loc` + `trace` - into the synthesised `Info` — closing the renderer ceiling for ALL thrown - values (cross-thread AND the latent in-thread `(throw (ex-info …))` gap). -3. **A neutral `runtime/concurrency/worker_error.zig`** with the uniform pair: - `capture(rt) Value` (worker-side, at the catch — produces the widened ex_info - on the worker thread, BEFORE it exits, while `msg_buf`/`trace_snapshot` are - still valid; the kind-derived class flows through `allocException`, fixing the - wrong-class bug) and `reraise(v) ClojureWasmError` (consumer-side — - republishes `dispatch.last_thrown_exception = v` + returns `error.ThrownValue`, - rendering byte-identically to an in-thread throw). -4. **Wire all three constructs** to the one helper: `future` (the missing one, - stm.zig deref + future.zig worker), `agent` (replace `captureThrown`, fixing - its class bug), and pmap when it lands. The carrier rides each construct's - existing GC-traced slot (no new GC tag). - -### Staging (each: lightweight local verify → full gate at the commit) - -- **Stage A — widen ExInfo with `origin_loc` (LOCATION) + in-thread fidelity.** - ex_info gains `origin_file_ptr/len` (GC-owned, duped like `message`) + - `origin_line`/`origin_column` + GC finalise frees the file; `allocException` - populates from the live loc; `buildThrownInfo` reads them. Observable in-thread: - `(throw (ex-info "x" {}))` renders with a real location (closes the latent - "unknown" gap). **Trace-on-ExInfo is split OUT to a follow-on (D-336):** a - single `origin_file` string is a bounded extern-struct widening, but a full - trace is an ARRAY of `StackFrame` each with 3 GC-owned strings on the - most-allocated heap type — meaningfully heavier than the DA's "same machinery" - framing implied. Location-first is the finished-form CORE (the user's primary - ask is kind/message/location fidelity, "not just the trace"); the trace is an - ADDITIVE enhancement that does not rework the location fields. (Step-0.6 - refinement found at implementation.) -- **Stage B — uniform cross-thread helper.** `worker_error.{capture,reraise}` + - wire future (deref re-raises the marshalled ex_info instead of - `future_thunk_failed`) + agent (replace captureThrown, fix the class) + pmap. - Observable: `@(future (/ 1 0))` and an agent error surface the REAL error + - trace on the consuming thread. Tests pin the kind (catchable) + message + - location + trace. - -### Landed (2026-06-08) - -- **Stage A** (a8b88c35): ExInfo `origin_loc` + `buildThrownInfo` reads it → - thrown exceptions render their location (closes the latent in-thread gap). -- **Stage B**: neutral `runtime/concurrency/worker_error.{capture,reraise}` + - wired into `future` (stm.zig deref re-raises the marshalled error; - future.zig worker captures) and `agent` (captureThrown delegated → the - hardcoded-ExceptionInfo class bug is fixed). `@(future (/ 1 0))` now renders - "Divide by zero" + `:1:14` + caret and is catchable as - `ArithmeticException`. `pmap` is sequential (core.clj:415) → no wiring. -- **§1 trace-on-ExInfo** (D-336): `ExInfo` gains `trace_ptr`/`trace_len`, - deep-copied (frame array + each frame's `fn_name`/`ns`/`file` strings - GC-owned, freed in `finaliseGc` — the same ownership as `message`/ - `origin_file`). `allocExceptionLoc` gains a `trace` param; the 3 synth/ - marshal sites (vm.zig, tree_walk.zig, worker_error.zig) pass `info.trace`; - `buildThrownInfo` reads `originTrace`. `@(future (boom))` renders - `Trace: user/boom` across the OS-thread boundary, and the latent in-thread - `(throw (ex-info …))` trace gap closes too. The perf note's lazy - frame-pointer scheme is deferred (Phase 15); the eager deep-copy is the - correct + simple form. Closes D-336 + D-330 + D-335. - -### Scope / deferrals - -- **Trace rides v1** (per the user "not just the trace"): the deep-copy machinery - for location is the same as for trace frames; splitting is the larger total - diff. The `max_call_depth=64` cap applies as in-thread (best-effort). -- **Perf**: GC-owned trace deep-copy lands on the throw path (hot). Acceptable - per `optimization_deferred_until_15_libs`; the Phase-15 perf pass may make the - trace deep-copy LAZY (capture frame pointers + a generation guard, materialise - on render) — a perf refinement of the SAME shape, not a redesign. PERF/O-NNN - marker at land. -- **D-329** (per-thread isolation) becomes testable once Stage B makes worker - errors observable on the consumer. - -## Alternatives considered (Devil's-advocate, fresh-context fork — verbatim) - -> Three facts the DA surfaced first: (1) the **agent path already does the -> smallest-diff** (`agent.zig:244 captureThrown` → ex_info in `Agent.error_val`) -> — so futures are strictly worse (no message even), and "choose a marshal" understates -> the asymmetry. (2) The **Value channel's fidelity ceiling is in the renderer** -> (`buildThrownInfo` recovers only message+data; location/trace structural) — not -> fixable by careful marshalling, only by widening ExInfo. (3) The **lifetime trap**: -> `setErrorFmt` already snapshots message→`msg_buf` + trace→`trace_snapshot` -> (threadlocal, dies with the worker); the throw path's Value lives on the GC heap -> (survives if rooted) — the two channels have different lifetime stories. - -**Alt 1 — smallest-diff: lift `captureThrown` into a shared helper, store an -ex_info in `Future.cached`, re-throw at deref.** Better: minimal surface, reuses -the agent precedent so future+agent become identical; `cached` already GC-traced -+ published under the cell mutex (no new GC/happens-before); the `(throw v)` path -is fully faithful. Breaks: BAKES IN the fidelity ceiling — `@(future (/ 1 0))` -gets the message but location stays "unknown", trace null; AND a WRONG-CLASS bug -(captureThrown hardcodes `"clojure.lang.ExceptionInfo"`, but `(/ 1 0)` synthesises -`"ArithmeticException"` → `(catch ArithmeticException …)` around a deref fails to -match — a real correctness regression, not cosmetic). - -**Alt 2 — finished-form: a marshalled-`Info` carrier re-installed EXACTLY on the -consumer.** A GC-managed `MarshalledError` deep-copies the whole `Info` -(message/location.file/every trace frame's strings) out of the threadlocals into -GC storage before the worker exits; consumer `setErrorInfo(info)` re-installs it + -returns `kindToError(info.kind)` → renders byte-identical to an in-thread error, -correct Kind (fixes Alt 1's class bug). Better: the ONLY option meeting the user -requirement (full location + trace + correct kind), genuinely uniform. Breaks: -largest diff (new GC tag + trace/finalise + `setErrorInfo`); and `agent-error` -returns the ex_info Value to USER code, so a `MarshalledError` carrier must ALSO -degrade to a catchable ex_info Value with ex-data — that dual role (faithful-render -carrier AND user-catchable Value) collapses Alt 2 toward "widen ex_info" (Alt 3). - -**Alt 3 — wildcard: widen `ExInfo` itself with optional location + trace, so the -ONE exception type carries full fidelity and the thrown-Value channel stops being -lossy.** Add `origin_loc` + `trace` (GC-owned) to ExInfo; populate in -`allocException`/`op_throw`; `buildThrownInfo` reads them → closes the renderer -hole for ALL thrown values. Cross-thread then needs NO special carrier (the -ex_info already carries everything). Better: eliminates the fork — one exception -representation, GC-managed/catchable/renderable; cross-thread becomes trivially -uniform; also fixes the latent in-thread `(throw (ex-info …))` location gap. -Breaks: puts GC-owned trace deep-copy on the HOT throw path (perf, deferred-opt -territory, lazy-materialisation fallback later); enlarges every ex_info (~48→80B -heap, not Value slots, F-004 unaffected); touches the most-allocated heap type -(blast radius beyond future/agent/pmap — a regression regresses all error -rendering); two trace-snapshot sites (allocException + setErrorFmt) to keep in -lockstep. - -**Recommendation (non-binding): Alt 2's `capture`/`reraise` helper + home, backed -by Alt 3's widened ExInfo as the carrier** — the carrier IS a location+trace- -extended ExInfo, not a parallel `MarshalledError`. Fixes Alt 1's wrong-class + -missing-location/trace; avoids Alt 2's dual-role awkwardness; one exception type -that is user-catchable + GC-rooted (rides existing slots, no new tag) + fully -renderable, with future/agent/pmap on one helper. NOT downgrading to Alt 1 -despite it being the agent precedent — that precedent is the low-fidelity floor -this ADR exists to raise (Cycle-budget-defer smell avoided). Leading F-NNN entry: -the recommendation adds GC-owned trace deep-copy on the hot throw path — no F-NNN -forbids it (F-006 accommodates; F-004 unaffected), but the Phase-15 perf pass must -measure it; lazy-materialisation is the clean fallback (same shape). **Trace rides -v1** (the deep-copy machinery for location is the same as for trace; splitting is -larger total diff + re-opens the lifetime trap twice). - -## Consequences - -- `@(future (boom))` / agent errors / pmap errors render the REAL error - (correct kind/message/location/trace) on the consuming thread — closing the - cross-thread fidelity gap uniformly. -- The latent in-thread `(throw (ex-info …))` location/trace gap closes too. -- `ExInfo` grows two fields + GC trace/finalise for the trace; the throw path - gains a trace deep-copy (perf-deferred, lazy fallback noted). -- The agent wrong-class bug is fixed (kind-derived class via `allocException`). -- D-330 (future trace) + D-335 (uniform cross-thread) resolved; D-329 (isolation) - becomes testable. - -## Affected files - -ex_info.zig (widen + trace/finalise), error_render.zig (buildThrownInfo reads -origin_loc/trace), catalog.zig/info.zig (allocException + the throw path populate), -vm.zig/tree_walk.zig (op_throw/evalThrow stamp), runtime/concurrency/worker_error.zig -(new — capture/reraise), future.zig + stm.zig (future wiring), agent.zig (replace -captureThrown), diff_test.zig + e2e (tests). diff --git a/.dev/decisions/0121_callable_print_naming.md b/.dev/decisions/0121_callable_print_naming.md deleted file mode 100644 index 3a6312eb9..000000000 --- a/.dev/decisions/0121_callable_print_naming.md +++ /dev/null @@ -1,307 +0,0 @@ -# ADR-0121 — Callable values print their name (`pr`/`str` of fn/multifn/protocol-fn) - -- **Status**: Proposed → Accepted (2026-06-08) -- **Resolves**: D-328 (`(pr fn)`/`(str fn)` show the fn's name, not `#`). -- **Composes with**: ADR-0119 Stage 1 (fns carry name+defining_ns), AD-002 - (opaque refs print `#<…>` not clj's `#object[…]`), AD-003 (simple names), - ADR-0059 (no JVM class). Adds AD-025. - -## Context - -`src/runtime/print.zig`'s value switch has no `.fn_val` / `.multi_fn` / -`.protocol_fn` arm, so all three fall through to `else => #<{tag}>` and leak the -internal Zig heap-tag name: `#`, `#`, `#`. ADR-0119 -Stage 1 already put `name`/`defining_ns` on the `Function` value (and `MultiFn` -carries a `name` Symbol, `ProtocolFn` a descriptor fqcn + method name), so the -information to print a meaningful name exists — it just is not wired to the -printer. clj prints `#object[user$boom__7 0xHASH "user$boom__7@…"]`; cljw drops -the JVM munged class + identity hash per AD-002/AD-003/ADR-0059. - -`str` and `pr` of a callable reach the SAME site (`writeStrValue`'s `else` routes -to `printValue`), and clj renders them identically too — so there is no str↔pr -split to design. - -### The zone constraint shapes the mechanism - -`print.zig` is Layer 0 (runtime); `Function` lives in -`src/eval/backend/tree_walk.zig` (Layer 1). The zone rule forbids runtime/ -importing eval/, and `printValue(w, v)` carries no `*Runtime`. BUT `MultiFn` -(`runtime/multimethod.zig`) and `ProtocolFn` (`runtime/protocol.zig`) are Layer-0 -`extern struct`s — `print.zig` reads them directly. **Only `.fn_val` crosses a -zone.** (The Devil's-advocate assumed all three callables needed a cross-zone -hook and recommended generalising into a `tag_ops` display table; that premise is -factually wrong — two of the three are Layer 0 — so a `tag_ops` table would be a -one-registrant speculative generality. The DA's big-bang FORM insight is adopted; -its mechanism is corrected to a single-consumer accessor.) - -## Decision - -1. **Form (big-bang the whole callable family, no partial sweep):** every named - callable prints `#`; an unnamed one prints `#` — AD-002's `#<…>` - envelope with the raw tag replaced by the qualified name. Landed in one push - for all three tags: - - `.fn_val` → `#` (via the accessor below) / `#`. - - `.multi_fn` → `#` from the `MultiFn.name` Symbol (Layer-0 read). - - `.protocol_fn` → `#` from the descriptor (Layer-0 read). - `.builtin_fn` keeps `#builtin` (no name slot until D-327; the envelope is - co-designed so D-327 only fills the name). Recorded as **AD-025** (the clj - `#object[…]` divergence), `derives_from: AD-002 + AD-003 + ADR-0059`, - pinned by a test exercising fn + defmulti + protocol-fn + anon. -2. **Mechanism (single cross-zone consumer):** a file-private - `var fn_name_accessor: ?*const fn(Value) FnIdentity` + `setFnNameAccessor` - in `print.zig`, installed from `tree_walk.registerGcHooks` (already called - unconditionally for both backends from `driver.installVTable`). This mirrors - the existing `runtime/error/info.zig` `context_provider` setter-injection (NOT - a `pub var` vtable — ROADMAP §13). `print.zig` keeps the `#<…>` format policy - in one place; the accessor returns only `{ ns, name }` data. `.multi_fn` / - `.protocol_fn` need no accessor (Layer-0 direct read). -3. **`(class fn)` deferred:** it returns the raw tag `fn_val` today; a callable's - class is a TypeDescriptor (type-system) concern, not a print concern. - Co-designing it would drag TypeDescriptor design into a print cycle. Out of - scope; AD-025 notes the print form implies no class decision. - -### Scope / deferrals - -- **Builtins (D-327):** `#builtin` stays; the `#` envelope is chosen now - so D-327 only supplies the reverse ptr→name lookup, not a new form. -- **Layer-0 header-resident name (the DA's Alt 3):** moving `name`/`defining_ns` - into a Layer-0-visible header would dissolve the accessor entirely, but it is - depth-4 layout surgery (the `Function` offset assert, the VM `op_make_fn` - reconstruct, and a NEW GC-rooting obligation on what is today a deliberately - GC-inert static name). Disproportionate for a print fix; revisit when a - non-print consumer (fn `class` / AOT name tables) first needs callable identity - in Layer 0. - -## Alternatives considered (Devil's-advocate, fresh-context fork — verbatim) - -> **Devil's-advocate review of ADR-0121 / D-328 (fn name in `pr`/`str`)** -> -> Setup verified against source. `printValue` (`print.zig:595-678`) has no -> `.fn_val`, `.multi_fn`, or `.protocol_fn` arm — all three fall through to -> `else => |t| w.print("#<{s}>", .{@tagName(t)})` at L677, printing the raw -> heap-tag name (`#` / `#` / `#`). `.builtin_fn` -> at L601 already prints `#builtin`. `writeStrValue` (L88-156) has no `.fn_val` -> arm either; its `else` (L154) routes to `printResult` → `printValue`, so -> **`str` and `pr` of a fn already share the same rendering site** — there is no -> str↔pr split to design here, unlike regex/uuid/ns/bigint which `writeStrValue` -> special-cases. clj also renders them identically (both the `#object[…]` form), -> so unified is the correct target. The `Function` struct (`tree_walk.zig:130-167`) -> carries `name`/`defining_ns` as `?[]const u8`, both static analyzer-arena slices -> (GC-inert). The `context_provider` precedent (`info.zig:122-141`) is a -> file-private `var ?*const fn() ?Value` + `setContextProvider`, installed from a -> higher layer — exactly the inversion shape the draft mirrors. `tag_ops` -> (`gc/tag_ops.zig`) already holds three `pub var [64]?fn` tables registered via -> `registerTrace`/`registerFinaliser` from `driver.installVTable` → -> `tree_walk.registerGcHooks` (`driver.zig:107-125`). These are the two -> zone-crossing vehicles in play. -> -> None of the three alternatives requires violating an F-NNN. The -> finished-form-clean option (Alt 2) is fully expressible within AD-002 + -> ADR-0059 + zone_deps. -> -> **Alt 1 — smallest-diff: file-private accessor + `#` (≈ the current -> draft, minus co-design)** -> -> A file-private `var fn_name_accessor: ?*const fn(Value) FnIdentity = null;` + -> `setFnNameAccessor` in print.zig (or a tiny `runtime/fn_identity.zig` helper), -> installed from `driver.installVTable` alongside `registerGcHooks`. Add one -> `.fn_val` arm to `printValue`: call the accessor, format `#` (named) -> / `#` (anonymous). `writeStrValue` inherits it via its `else`→`printValue` -> fall-through (no second edit). Record AD-025. Leave `multi_fn`/`protocol_fn`/ -> `builtin_fn` and `(class fn)` out. -> -> - **(a) Better than the draft:** essentially *is* the draft, so nothing — -> except it is the honest floor: one arm, one accessor, one AD row, smallest -> surface to get wrong. Reuses the exact, already-blessed `context_provider` -> injection idiom (not a new mechanism the reader must learn). `FnIdentity` (a -> `struct { ns: ?[]const u8, name: ?[]const u8 }`) is value-returning, so no -> lifetime/GC question crosses the zone — the slices are analyzer-arena-static -> and only read for immediate formatting. -> - **(b) Breaks/risks:** It is *one of N callable tags*. `multi_fn` (`defmulti`) -> and `protocol_fn` are sibling user-named callables that will keep printing -> `#` / `#` — leaking the internal tag name, the -> **exact** bug D-328 names, for two tags out of four. Shipping a -> `.fn_val`-only arm is a half-swept area (the Micro-coverage-grind / -> partial-sweep smell in `clj_diff_sweep.md` Discipline 2): the debt ledger -> reads "fn print: done" while `defmulti`/protocol fns still leak. A second -> risk: a *single file-private accessor for fn_val only* invites a third and -> fourth accessor when multi_fn/protocol_fn get the same treatment — three -> near-identical file-private `var`+setter triples, which is the -> Reservation-as-bias / smallest-diff accretion the project warns against. The -> print form `#` is defensible but under-specified vs clj: see the -> form discussion in the recommendation. -> -> **Alt 2 — finished-form-clean: a `tag_ops`-registered print descriptor for the -> whole callable family, name carried in a runtime-readable place** -> -> Two moves, both at the finished-form altitude: -> -> 1. **Mechanism = extend `tag_ops` with a per-tag *display* hook**, not a fourth -> bespoke accessor. `tag_ops` is *already* the Layer-0 registry the eval layer -> injects per-tag behaviour into (`registerTrace`/`registerFinaliser` from -> `driver.installVTable`). Add a `tag_display_table: [64]?DisplayFn` + -> `registerDisplay`, where `DisplayFn = *const fn(Value, *Writer) -> Writer.Error!void` (or returns a small `FnIdentity` the print site formats, -> to keep the `#<…>` envelope policy in *one* place — print.zig — rather than -> scattered into eval). `printValue`'s `else` arm consults -> `tag_display_table[tag]` before falling to `#`. `tree_walk.registerGcHooks` -> (already the install point) registers display fns for `.fn_val`, `.multi_fn`, -> `.protocol_fn` in one place. This generalises the zone-crossing to *any* heap -> tag whose pretty-name lives above Layer 0 — which is the same shape -> `.ns`/`.host_instance`/`.typed_instance` already needed and got via ad-hoc -> routes. The reader who knows `registerTrace` instantly knows `registerDisplay`; -> no new idiom. -> 2. **Form = decide the *callable* print envelope once, for all four callable -> tags coherently.** Named user callable (`fn`/`defmulti`/`defn`-protocol-method) -> → `#`; anonymous → `#` (or `#` if a gensym id is cheaply -> available); builtin (no name slot, D-327) → keep `#builtin` *or* upgrade to -> `#` once D-327 lands a name. The AD row (AD-025) covers the -> whole family with one `derives_from: AD-002 + AD-003 + ADR-0059`, one pin test -> exercising fn + defmulti + protocol-fn + anon + builtin. -> -> - **(a) Better than the draft:** Closes the *entire* callable-print leak in one -> push, satisfying `clj_diff_sweep.md` Discipline 2 (big-bang, don't drip-feed) -> — no "multi_fn still leaks" residue, no second/third cycle re-deriving the same -> accessor. Mechanism is the project's *own* established generalisation -> (`tag_ops`), so it removes the temptation to grow N file-private accessors; the -> next opaque-tag pretty-name (a future `wasm_fn`, etc.) drops into the same -> table for free. Puts the `#<…>` envelope policy in exactly one site -> (print.zig), honouring error_catalog-style "one message source" hygiene for -> print forms. One AD row instead of three. -> - **(b) Breaks/risks:** Larger blast radius — a new `pub var` table in -> `tag_ops.zig` (note: `tag_ops` already uses `pub var` tables, so this is -> *consistent with existing precedent*, not a §13 `pub var` *vtable* violation — -> the §13 ban is on `pub var` **VTable structs**, and these all-null-default -> per-tag arrays are the sanctioned shape per ADR-0028 §4). Risk: -> over-generalising before a second consumer exists could read as speculative -> (YAGNI) — but there are already *three* immediate consumers -> (fn/multi_fn/protocol_fn) plus the `.ns`/`.host_instance` ad-hoc arms that -> retroactively fit, so the second-consumer test is already met; this is not -> speculative. Touching `multi_fn`/`protocol_fn` formatting risks a -> differential-test ripple (their current `#` strings may be asserted -> somewhere) — a grep + diff-corpus pass is needed, which is correct work, not a -> reason to shrink. The `DisplayFn`-returns-Writer variant lets eval write -> directly into the print buffer, which slightly muddies "print policy lives in -> Layer 0"; the `FnIdentity`-returning variant is cleaner and recommended. -> -> **Alt 3 — wildcard: drop the zone-crossing entirely by giving the print form -> what it needs at the Value level — a name slot reachable from Layer 0** -> -> Instead of injecting a function pointer from eval, make the identity **readable -> from Layer 0 directly**. Two sub-shapes: -> -> - **3a (header-resident name):** move `name`/`defining_ns` out of the Layer-1 -> `Function` struct into a Layer-0-visible prefix — e.g. a small -> `runtime/callable_header.zig` struct that `Function` (and `MultiFn`/`ProtocolFn`) -> embed right after `HeapHeader`, at a fixed offset Layer 0 can decode without -> importing eval. print.zig reads `callable_header.nameOf(v)` with no accessor, -> no setter, no install step. The zone rule is satisfied because the *struct -> lives in Layer 0* and eval merely fills it. -> - **3b (name as interned symbol Value in the header):** stash the qualified name -> as an already-existing `.symbol` Value (a Layer-0 type) in a header slot; print -> just `printValue`s that symbol inside `#<…>`. Reuses symbol printing verbatim. -> -> - **(a) Better than the draft:** *No injection mechanism at all* — no -> file-private `var`, no setter, no startup-order dependency, no "is the accessor -> installed yet?" failure mode (the draft's accessor is null until `installVTable` -> runs; a print before that prints `#` again — a real ordering footgun -> 3a/3b eliminate). The name becomes a first-class property of the callable -> Value, which is arguably where it *should* live (clj's fn name is intrinsic to -> the object, not injected). 3b reuses symbol printing for free and round-trips -> naturally. -> - **(b) Breaks/risks:** Biggest structural surgery — moving `name`/`defining_ns` -> into a Layer-0 header touches the `Function` layout (which has a load-bearing -> `comptime` offset assert at `tree_walk.zig:164-166` and a GC-rooting note), the -> VM `op_make_fn` closure reconstruct (`tree_walk.zig:160` notes name is copied -> there too), and every alloc site. 3b additionally makes the name a GC-managed -> `.symbol` Value, which means `traceFunction` must mark it (currently name is -> GC-inert static `[]const u8` — a deliberate property the struct comment calls -> out); that *adds* a rooting obligation the current design specifically avoids. -> This is depth-4 surgery for a print-form fix — likely disproportionate *now*, -> but it is the genuinely different angle: it treats "Layer 0 can't see the name" -> as the root problem to dissolve rather than to bridge. Defensible as the -> finished-finished form if a future Phase already needs callable identity in -> Layer 0 (e.g. `class`/`type` of a fn, demunging, AOT name tables) — otherwise -> premature. -> -> **On the three sub-questions the brief flags:** -> -> - **(i) Print form:** `#` is the right envelope — it is AD-002's -> established `#` shape with the tag *replaced by the qualified name*, which -> is strictly more informative and still drops clj's non-reproducible -> `#object[user$boom__N 0xHASH …]` identity-hash+munged-class per -> AD-002/AD-003/ADR-0059. Do **not** mimic `#object[…]` (it implies a JVM class + -> would re-invent the munged `user$boom__7` name AD-003 exists to avoid). -> Anonymous → `#` is coherent. `builtin_fn` (no name, D-327) should *not* be -> forced into this cycle — `#builtin` stays until D-327 gives it a name slot; but -> the *form* should be chosen now so D-327 just fills the name -> (`#`-style), i.e. co-design the *envelope*, defer the *data*. -> `str` vs `pr` must **not** differ (verified: both reach the same site; clj -> agrees) — this is a feature, not a gap, and the AD row should state it -> explicitly so a future sweep doesn't "fix" them apart. -> - **(ii) Mechanism:** `tag_ops`-registered display (Alt 2) over a bespoke -> file-private accessor (Alt 1) — it is the *existing* Layer-0↔Layer-1 injection -> registry, already installed at exactly the right startup point, and it scales -> to multi_fn/protocol_fn without a second/third accessor. Threading `*Runtime` -> through `printValue` is the worst option (touches every print call site, and -> `printValue` is deliberately a free function — a large ripple for no benefit, -> smallest-diff-*negative*). Moving the name into a Layer-0 header (Alt 3) is the -> cleanest *eventual* shape but disproportionate surgery now. -> - **(iii) `(class fn)`:** correctly **deferred**. `(class x)` returns a -> TypeDescriptor simple name (AD-003); fn's class story is a *type-system* -> concern (what TypeDescriptor does a closure carry?), not a *print* concern, and -> co-designing it now would drag TypeDescriptor design into a print-form cycle. -> Defer with its own debt row + a one-line note in AD-025 that the print form -> does not imply a class decision. -> -> **Non-binding recommendation.** The finished-form-cleanest shape is **Alt 2 with -> Alt 1's `#` form decision** — i.e. *not* the bespoke file-private -> accessor, but a `tag_ops`-registered per-tag display hook covering the whole -> callable family (`fn_val`, `multi_fn`, `protocol_fn`) in one big-bang push, with -> the `#` / `#` envelope and a single AD-025 row + multi-tag pin test. -> The draft's accessor (Alt 1) is finished-form-*adjacent* but trips two project -> smells: it leaves `multi_fn`/`protocol_fn` leaking the raw tag name -> (partial-sweep, `clj_diff_sweep.md` Discipline 2) and it seeds a family of -> near-identical file-private accessors instead of using the `tag_ops` registry the -> project *already* built for exactly this Layer-0↔Layer-1 inversion. Per F-002, -> the slightly larger diff of generalising into `tag_ops` is not a reason to prefer -> the accessor — and crucially the generalisation has three immediate consumers, so -> it is not speculative. Alt 3 (Layer-0 header-resident name) is the genuinely-final -> form and worth a one-line forward-pointer in the ADR, but its depth-4 layout -> surgery (offset assert, VM reconstruct, and — for 3b — a *new* GC-rooting -> obligation on what is today a deliberately GC-inert name) is disproportionate for -> a print-form fix; defer it to whenever a non-print consumer (fn `class`/AOT name -> tables) first needs callable identity in Layer 0. `(class fn)` stays out of scope -> with its own debt row. - -### Main-loop response to the DA (why the mechanism diverges from the recommendation) - -The DA's big-bang FORM insight is **adopted** — all three callable tags land -`#` in this one push; no partial sweep. Its *mechanism* recommendation -(a `tag_ops` display table) rests on a premise the main loop verified false: -`MultiFn` (`runtime/multimethod.zig:55`) and `ProtocolFn` (`runtime/protocol.zig`) -are **Layer-0 `extern struct`s** that `print.zig` reads directly — only `.fn_val` -crosses a zone. So the table would have a **single registrant**, making it the -speculative-generality the DA's own Alt 2 risk (b) flags ("over-generalising -before a second consumer exists"). With one cross-zone consumer, the bespoke -`context_provider`-style accessor IS the proportionate finished form and seeds no -accessor family. This is not a cycle-budget downgrade (F-002) — it is the -finished-form choice once the consumer count is corrected from 3 to 1. - -## Consequences - -- `(pr fn)` / `(str (defmulti …))` / `(pr protocol-fn)` render `#` - instead of the internal tag name. The latent `#`/`#` - leaks close in the same push. -- `print.zig` gains a `fn_name_accessor` (one setter-injected fn pointer, mirroring - `context_provider`) + a `printCallable` helper; `tree_walk.registerGcHooks` - installs the accessor. -- AD-025 records the clj `#object[…]` divergence with a pin test. -- `(class fn)` unchanged (deferred). Builtins unchanged (`#builtin`, D-327). - -## Affected files - -print.zig (FnIdentity + setFnNameAccessor + printCallable + 3 switch arms + -fallback-test retarget + new tests), tree_walk.zig (fnIdentity accessor + -registerGcHooks install), .dev/accepted_divergences.yaml (AD-025), e2e + -diff_test (print forms, both backends), .dev/debt.yaml (D-328 discharge). diff --git a/.dev/decisions/0122_read_string_eval_free.md b/.dev/decisions/0122_read_string_eval_free.md deleted file mode 100644 index fa4638987..000000000 --- a/.dev/decisions/0122_read_string_eval_free.md +++ /dev/null @@ -1,82 +0,0 @@ -# ADR-0122 — `read-string` is eval-free by design (`#=` read-eval unsupported) - -- **Status**: Proposed → Accepted (2026-06-09) -- **Resolves**: SE-8 (security audit, `private/security_audit/50_sharp_edges.md`) — - the secure-by-default property that reading data never executes code. Adds AD-026. -- **Composes with**: F-011 (behavioural equivalence — security-improving - divergences are recorded, not silently matched), AD-002/AD-003/ADR-0059 (cljw is - not a JVM reimplementation). - -## Context - -JVM Clojure's `clojure.core/read-string` reads with `*read-eval*` **true** by -default, so `#=(form)` (the read-eval reader macro) is **executed at read time**: -`(read-string "#=(+ 1 2)")` returns `3` on the JVM. This is a well-known -remote-code-execution footgun — `read-string` on untrusted input is `eval` on -untrusted input. - -cljw's `read-string` is the same full-reader `readOne → formToValue` path as -`clojure.edn/read-string`: the reader has **no `#=` reader function**, so -`(read-string "#=(+ 1 2)")` raises `No reader function for tag =` — it reads data, -it never evaluates. This is the secure-by-default property SE-8 confirmed -(`read-string` is `#=`-free; only `read-string` + `eval` form the eval surface, -and they are separate calls). - -The question this ADR settles: is cljw's `#=`-free reader an **intentional, -permanent** design property (a deliberate divergence from clj, to be locked), or -merely an unimplemented reader macro that a future "clj parity" sweep might "fix" -back into an eval-on-read footgun? **It is intentional and permanent.** - -## Decision - -cljw's reader (`read-string` and the EDN reader it shares) is **eval-free by -design**. `#=` / read-eval is **not supported and will not be added**: reading a -string or EDN never executes code. Evaluation is reached only via an explicit, -separate `eval` call on already-read data. - -This is recorded as **AD-026** (an accepted clj-divergence) with -`derives_from: ADR-0122`, pinned by `test/e2e/phase14_read_string.sh` -(`rs_no_read_eval`: `(read-string "#=(+ 1 2)")` must NOT yield `3`). A future -clj-diff sweep that sees cljw diverge from clj's `3` here must treat it as the -locked decision, never a gap to close. - -## Consequences - -- `read-string` is safe on untrusted input w.r.t. read-time code execution - (the SE-8 secure-by-default property is now a mechanically-checked guarantee, - not an accident of a missing reader macro). -- Divergence from clj is explicit (AD-026), so it reads as "designed", not - "broken", when a user hits it. -- This does NOT confine the separate `eval` / nREPL surface — that is the - eval-free-deploy-build item (SE-8 second half), tracked as debt **D-341**. - -## Alternatives considered - -Sourced from a fresh-context devil's-advocate review of the whole SE-2/3/6/7/8 -"design-gap" cluster (the disposition question: how to record + schedule the -cluster in one unit). Its findings, verbatim-in-substance: - -- **Smallest-diff** — skip any ADR; just add debt rows + an AD entry. Better: no - ceremony, debt rows are the project's deferral SSOT (F-003). Risk: the `#=`-free - property needs a *named invariant* to be a legitimate AD (the - accepted_divergences discipline forbids an AD without `derives_from`) — a bare - debt row does not supply one. **This ADR exists precisely to be that invariant** - (the narrow, specific decision), which is why the DA's broader "skip the ADR" - applies to the *umbrella* posture ADR, not to this one. -- **Finished-form-clean** — build the two implementable-now items (SE-2 import - allowlist, SE-6/7 FS-jail) this unit; debt-row only SE-3/SE-8. The DA rated this - its recommendation. Adopted in part: SE-6/7 (FS-jail) is the next build unit - (D-340). SE-2 is NOT built now — there is **zero host-import-providing code path - today**, so an allowlist would gate nothing (the "excessive skeleton" smell the - DA itself flagged); it lands in the same commit as the first host import (D-338). -- **Wildcard** — an executable security-property corpus asserting today's true - secure-by-default facts (no `#=` eval, zero-import-only load). Adopted in spirit: - the `#=`-free fact is locked as the `rs_no_read_eval` pin; the others become - debt-row barriers rather than a corpus of not-yet-built behaviour (a corpus for - unbuilt behaviour is itself a skeleton). - -The DA explicitly rejected a single **umbrella "deploy-mode posture" ADR** as -over-ceremony / an excessive skeleton, since 3 of 4 cluster items gate nothing -today. This ADR is therefore deliberately **narrow** (one already-true property, -locked), and the remaining cluster lives as scheduled debt rows -(D-338 SE-2, D-339 SE-3, D-340 SE-6/7, D-341 SE-8), not as a posture skeleton. diff --git a/.dev/decisions/0123_fs_jail_deploy_mode.md b/.dev/decisions/0123_fs_jail_deploy_mode.md deleted file mode 100644 index b0506d571..000000000 --- a/.dev/decisions/0123_fs_jail_deploy_mode.md +++ /dev/null @@ -1,130 +0,0 @@ -# ADR-0123 — Opt-in filesystem jail for deploy mode (`CLJW_FS_ROOT`) - -- **Status**: Proposed → Accepted (2026-06-09) -- **Resolves**: SE-6 / SE-7 (security audit) — `slurp`/`spit`/`wasm/load` take any - string path with no confinement; at the edge, request data reaching a path arg - is arbitrary read/write + `../../etc/passwd` traversal. Discharges D-340 (v1). -- **Schedules**: D-342 (symlink-safe finished form — the Alt-2 below). -- **Composes with**: F-009 (one neutral enforcement point), F-011 (jail OFF by - default → local-CLI `slurp`/`spit` unchanged; only opt-in deploy mode diverges). - -## Context - -`slurp`/`spit` (`lang/primitive/file_io.zig`) and `wasm/load` (`wasm/surface.zig`) -— plus `java.io.File` — all route file reads/writes through -`runtime/file_io.zig` `readAll`/`writeAll`. None confine the path. A deployed edge -app that lets request data reach a path argument is an arbitrary-FS-read/write -+ path-traversal primitive. Local CLI use is fine by design (the user owns the -shell). The audit's secure-by-default target: an **opt-in preopen-style FS root** -for deployed mode, off for local CLI. - -### Why not the obvious kernel mechanism - -Zig 0.16 `std.Io.Dir.openFile` exposes `resolve_beneath`, but it maps to -`posix.O.RESOLVE_BENEATH` via `@hasField` — present **only on FreeBSD**. On macOS -and Linux (cljw's two gate hosts) the option is **silently ignored**. A jail built -on it would be **false security** on every platform cljw actually runs. `std.Io` -`realPath` is itself stdlib-flagged "limited platform support, advisable to avoid -entirely." So the portable mechanism must be **lexical**. - -## Decision - -A jail root is configured at startup via the `CLJW_FS_ROOT` environment variable -(matching cljw's existing `init.environ_map.get("CLJW_…")` convention). When set, -it is canonicalised ONCE to an absolute `root_abs` (a one-time `realPath` of an -existing directory — loud failure at startup on misconfiguration) and stored on -`Runtime.fs_jail_root` (`?[]const u8`, default `null` = unconfined). - -A neutral `file_io.enforceJail(alloc, root_abs, path) !void` is the single policy -point (F-009). Each FS surface (`slurp`/`spit`/`wasm/load`/`java.io.File`) calls it -before the I/O op and maps its error to the catchable catalog Code -`fs_jail_escape` (Kind `.value_error` → IllegalArgumentException). When -`root_abs == null` it is a no-op (local CLI unchanged). - -`enforceJail` resolves the user path under the root with -`std.fs.path.resolvePosix(alloc, &.{root_abs, path})` (lexical `.`/`..` -resolution) and requires the result to be `root_abs` itself or strictly beneath -it (`startsWith(resolved, root_abs) and resolved[root_abs.len] == '/'` — the -boundary guard that distinguishes `/jail/x` from a sibling `/jailX`). Otherwise → -`error.FsJailEscape`. A `..`-traversal escape and an absolute path **outside** the -root are both rejected by this containment (an absolute path resets resolvePosix, -then fails the prefix check); an absolute path that lands **inside** the root -resolves to within and is allowed — there is no silent escape in either case. A -non-absolute `root` is a misconfiguration and fails closed (deny all). - -### Guarantee + documented residual (this is v1, NOT terminal) - -This blocks `..` traversal and absolute-path escape **deterministically and -portably**. It does NOT resolve symlinks, so a symlink *planted inside the jail* -pointing outside is still followed. That residual is **explicit** (here, in the -error text's spirit, and in deploy guidance: mount the jail read-only / forbid -symlinks in it). A jail that does exactly what it documents is honest, not -false-security — the false-security case is the macOS-ignored `resolve_beneath`, -which this deliberately avoids. The symlink-safe finished form (per-component -`openat`-relative resolution / `openat2(RESOLVE_BENEATH|RESOLVE_NO_SYMLINKS)` on -Linux, `O_NOFOLLOW` walk on macOS) is scheduled as **D-342**, not deferred to -"later" — v1 is an honest documented step toward it. - -## Consequences - -- A deploy sets `CLJW_FS_ROOT=/srv/app/data`; all `slurp`/`spit`/`wasm/load` - confine to that subtree; traversal/absolute escapes raise a catchable error. -- Local CLI (no env var) is byte-for-byte unchanged (F-011-safe). -- The symlink residual is a known, documented, scheduled (D-342) limitation — - not a silent gap. - -## Scope: DATA surfaces, not code-loading - -The jail confines the **data** FS surfaces (`slurp`/`spit`/`wasm/load`) — the ones -a request-derived path reaches. It deliberately does NOT confine **code-loading**: -`(require …)` reads `.clj` libs off `rt.load_paths` (`require_resolver.zig`), and -those load paths are operator-supplied deploy config (`CLJW_PATH` / `deps.edn`), -not request data — and a lib tree legitimately lives OUTSIDE the data root, so -jailing `require` to `CLJW_FS_ROOT` would break normal library loading. The other -`std.Io.Dir` opens in `src/app/**` (the entry script, `deps.edn`, the AOT cache, -`.nrepl-port`, `CLJW_ERROR_LOG`) are all startup/operator-controlled config, not -request-reachable, so they are out of scope by the same reasoning. The one -request-reachable residual is `(require (symbol request-data))` (munged-ns -traversal) — narrow and unusual; tracked as **D-343** (confine code-loading / -sanitise the munged path if a deploy needs it). A future security note must not -imply the jail confines code-loading; it confines request-driven data I/O. - -## NUL-byte guard (review follow-up) - -A fresh-eyes review found that lexical `resolvePosix` treats an embedded NUL as an -ordinary byte (so `..\x00` is not seen as `..` and passes containment), while the -kernel's C-string `open` truncates at the NUL — a check-vs-open mismatch that, in -the shipping ReleaseFast build (where posix's NUL-absence assert is compiled out), -could open a path the jail did not check. `jailResolve` now rejects any path or -root containing a NUL up front (no legit path has one). Pinned by a unit test. - -## Alternatives considered - -From a fresh-context devil's-advocate fork (F-NNN-constrained), verbatim in -substance: - -- **Smallest-diff** — the standalone `enforceJail` lexical check (this ADR's v1). - Cleanest F-009 split; a ~30-line pure predicate, FS-free unit tests. Its only - question is the symlink residual (below). Risk if taken as *terminal*: the - smallest-diff-bias smell. -- **Finished-form-clean** — symlink-safe per-op `openat`/`fstatat`-relative - resolution (kernel-enforced: `openat2 RESOLVE_BENEATH|RESOLVE_NO_SYMLINKS` on - Linux, `O_NOFOLLOW` per-component walk on macOS) + a Clojure dynamic var - (`*fs-root*`) layered over the env default for per-request tightening. Closes - the symlink residual *honestly*. Cost: forces a root dir-handle into - `file_io.zig` (real surgery, not a thin surface), and the macOS per-component - walk is fiddly (TOCTOU unless the dirfd is kept). **This is the F-002 finished - form — adopted as the SCHEDULED target D-342**, with v1 (lexical) shipping now. -- **Wildcard** — a capability-handle model (opening returns an opaque file - capability; ops take the handle, no path crosses the jail). Eliminates the - traversal class by construction, but detonates F-011 (JVM `slurp`/`spit` take - strings); viable only as a *separate* future edge API, not a confinement of the - named fns. Rejected as the jail; noted as a future edge-API direction. - -The DA's binding-free recommendation: ship the lexical v1 **with** explicit -symlink-residual documentation **and** D-342 scheduling Alt 2 — Alt 1 as an honest -documented step, not as terminal. Adopted. The DA also corrected (b) -paths-relative-to-jail-root (preopen, absolute rejected) over cwd-then-contained, -and (e) the exhaustive boundary-test list (prefix boundary, `..` past root, -absolute, empty/`.`/trailing-slash, `root_abs` itself, the symlink-followed -residual asserted so it can't silently change, double-slash) — all adopted. diff --git a/.dev/decisions/0124_wasm_run_wasi_command.md b/.dev/decisions/0124_wasm_run_wasi_command.md deleted file mode 100644 index 6b657ebaa..000000000 --- a/.dev/decisions/0124_wasm_run_wasi_command.md +++ /dev/null @@ -1,138 +0,0 @@ -# ADR-0124 — `(wasm/run …)`: run a WASI command module from Clojure - -- **Status**: Proposed → Accepted (2026-06-09) -- **Adds**: a third var to the `wasm` ns — `(wasm/run "path.wasm" opts)` — that - runs a WASI command module (Rust / Go / … compiled to `wasm32-wasip1`), - capturing stdout / stderr / exit. Complements `wasm/load` + `wasm/call` - (scalar pure-compute, no WASI) from ADR-0099. -- **Extends zwasm** (F-001 co-dev, user-authorized 2026-06-08 direct edits): - `runWasmCapturedFull` (the old `runWasmCapturedOpts` became a back-compat - shim) + a `capture_alloc` field on the WASI `Host` so capture buffers grow - with the caller's allocator. -- **Schedules**: D-347 (budgets through the C-API WASI run path), D-348 (`:env`), - D-349 (unbounded capture buffer), D-350 (unified `wasm/load`+handle finished - form — DA alternative-b, deferred to Phase 16 per F-003). -- **Composes with**: F-009 (surface in `runtime/cljw/wasm/`, impl-thin), - F-013 (preopens surfaced as the full `:dir`/`:dirs` class, not one-mount), - ADR-0123 (FS-jail confines the module path AND every preopen `:dir`). - -## Context - -ADR-0099's `wasm/call` invokes an exported function with i32/i64/f32/f64 -scalars — perfect for pure-compute modules (`wasm32-unknown-unknown`, no -imports). But the compelling polyglot demo is "call a real Rust/Go program, -with its full stdlib, from Clojure": JSON transforms, hashing, a sqlite-backed -store. Those compile to `wasm32-wasip1` and import `wasi_snapshot_preview1.*` -(args, env, preopens, fd I/O, clock, random). zwasm implements all 46 WASI -preview1 functions and already runs off-the-shelf Rust/Go/TinyGo wasip1 -modules — but cljw's `wasm/load` path went through `Module.instantiate`, which -wires NO WASI host, so a wasip1 module failed to instantiate from Clojure. - -A WASI *command* (`_start`) is single-shot by spec: it runs `main`, calls -`proc_exit`, and the instance is spent. That lifecycle does not fit `wasm/call`'s -re-invokable handle, so it gets its own one-shot surface. - -## Decision - -Add `(wasm/run path)` / `(wasm/run path {:args [..] :stdin ".." :dir ".." :dirs [[h g]..]})` -returning an array-map `{:out :err :exit }`. - -- **One-shot**: compile → instantiate-with-WASI → run `_start`→`main`→first - export → capture → teardown, inside the builtin. No long-lived handle. -- **Exit is data, not an exception.** A non-zero exit — including a guest trap - (→ exit 1) — is returned in `:exit` (a process runner's exit code is data, - matching `clojure.java.shell/sh`'s `:exit`). Only *cljw-side* failures are - catchable exceptions: bad path/opts type (`ClassCastException`-kind), FS-jail - escape, unreadable file, compile/instantiate/preopen failure. Every failure - stays a catchable `ClojureWasmError`; none is an exit-70 crash. -- **argv[0] is the program name** by convention (verbatim — cljw does not inject - the path). -- **Preopens**: `:dir "d"` is sugar for one host dir mapped to guest `/`; `:dirs - [["host" "/guest"] …]` maps N dirs (a read input + a write output dir is the - common case). Every host path is FS-jail resolved (ADR-0123) — a preopen - cannot escape `CLJW_FS_ROOT`. -- **Implementation**: `surface.zig` parses opts (arena-scratch for argv/preopen - slices; string views into GC strings stay valid across the run since no cljw - allocation happens during it). `engine.zig::run` calls - `zwasm.cli.run.runWasmCapturedFull`, passing two caller-owned `ArrayList(u8)` - capture buffers + optional stdin. The captured bytes are copied into GC - strings for the result map. - -### The zwasm extension (capture_alloc) - -The WASI host's `fd_write` appends guest stdout/stderr to a caller-supplied -`ArrayList(u8)`, but grew it with the host's own allocator (`c_allocator` from -`zwasm_wasi_config_new`) while the embedder freed it with a different allocator -→ a cross-allocator invalid-free (caught immediately under the DebugAllocator). -Fix: a `capture_alloc: ?Allocator` on the `Host`; `fd.writeSlice` appends with -`host.capture_alloc orelse host.alloc`; `runWasmCapturedFull` sets it to the -allocator the caller will free with. The buffer's grow-allocator and the -caller's free-allocator now agree. This is the clean ownership model (vs. -inlining zwasm's C-API dance in cljw, which would duplicate the engine/store/ -preopen/diagnostic sequence and reach across the F-009 boundary). - -## Consequences - -- Clojure can run any `wasm32-wasip1` command and read its output — unlocking - Go (stdlib intact) and Rust wasip1 demos, and a preopened-dir file round-trip - (the substrate for a sqlite-via-wasm store). -- `wasm/call` is unchanged (and its no-leak guard still passes — the - capture_alloc change does not touch the FFI path). -- Known bounds, all tracked: wasm/run is currently **unmetered** (D-347 — - budgets are not threaded through the C-API run path; the playground bounds it - with an OS sandbox), buffers the whole output (D-349), and omits `:env` - (D-348). The unified-handle finished form is recorded as D-350 (Phase 16). - -## Alternatives considered - -(Verbatim from a fresh-context Devil's-advocate fork, per CLAUDE.md § ADR-level -designs. The fork read `project_facts.md` + both surface/engine layers + zwasm -`run.zig`/`host.zig`.) - -### Leading finding — F-013 tension (no F-NNN violated) - -No alternative violates an F-NNN, but F-013 (definition-derived comprehensive -coverage) is in genuine tension with deferring `:env` and capping at one `:dir`, -because the WASI config surface is a small closed set that zwasm already -supports underneath. Resolution: `:dirs` (the whole preopen class) was pulled -forward into this cycle; `:env` is deferred with a tracked debt row (D-348) — a -recorded gap, not a silent "make this demo pass". - -### (a) Smallest-diff — return just stdout (string), throw on non-zero exit - -Smaller zwasm footprint (stdout-only capture already exists; no stderr/stdin -extension). But it lands a *worse* finished form: stderr (compiler diagnostics, -panics) becomes uncapturable, and throw-on-nonzero is wrong for a process -runner (exit codes are data — `grep` exits 1 normally). Rejected on F-002 -(different/worse finished form), not on diff size. - -### (b) Finished-form-clean — unified `wasm/load` with `:wasi {…}`, return a handle - -One "embed a module" concept: `(wasm/load path {:wasi {:args :env :dirs :stdin}})` -→ handle; `wasm/run`/`wasm/call` are operations on it; reusable persistent WASI -instances; `:env`/`:dirs` fall out of the config map (F-013 structural). This is -the cleaner *eventual* finished form (F-002), and is **not** downgraded on -diff-size grounds. Its blocker is a zwasm API gap: the C-API WASI run path is -one-shot; a persistent WASI instance needs zwasm to expose instantiate-with-WASI -separately from run-entry on the public Engine/Instance surface. Per **F-003** -(structural-plan deferral) this cw↔zwasm instance-lifetime split is recorded as -**D-350** and deferred to the Phase-16 owner (D-036 territory) rather than -seized in a demo-hardening cycle — so the clean form is named and owned, not -lost, while the live demo is not blocked on a zwasm rewrite. - -### (c) Wildcard — streaming process handle (deref for exit, lazy out/err) - -Bounds unbounded output (real robustness gap → captured as **D-349**) and fits -the Phase-14 `cljw.edge` direction. But no streaming substrate exists in zwasm -(the run loop appends to completion); building it needs threads/async (Phase 15) -or a zwasm re-architecture — premature, and it widens FFI breadth past the -authorized demo-hardening scope (F-010 tension). Rejected as substrate-premature; -its one real insight (unbounded buffer) is tracked, not built. - -### Verdicts on the five decided points - -(ii) `{:out :err :exit}` map and (iv) extending zwasm with `capture_alloc` are -finished-form-clean — kept. (v) `:dir`→`:dirs` and (iii)-env were smallest-diff -seams: `:dirs` fixed in-cycle, `:env` tracked (D-348). (iii)-budgets is a real -`wasm/load`-bounded-vs-`wasm/run`-unbounded asymmetry → D-347 + a zwasm -feed-back. (i) unified-handle is the true finished form → D-350 (Phase 16). diff --git a/.dev/decisions/0125_eval_execution_budget.md b/.dev/decisions/0125_eval_execution_budget.md deleted file mode 100644 index 9ec2fdf1b..000000000 --- a/.dev/decisions/0125_eval_execution_budget.md +++ /dev/null @@ -1,336 +0,0 @@ -# ADR-0125 — In-process eval execution budget (deadline + step ceiling), uncatchable on expiry - -- Status: Proposed → Accepted -- Date: 2026-06-09 -- Discharges (in part): D-351 (native in-process eval deadline / step budget, - isolation dim (a)). Sibling: D-352 (heap cap, dim (b)), D-355 (playground - drops babashka), D-354 (crash isolation — out of scope here). - -## Context - -cljw has NO eval timeout: an infinite loop in evaluated code hangs the -process. The `$MY/playground-v2` demo therefore wraps every eval in a -babashka supervisor that spawns a sandboxed child `cljw -e` bounded by OS -`timeout` / `ulimit`. The mission goal (drop babashka, run the playground -on cljw itself) needs an **in-process** time + step bound so untrusted code -can be evaluated without an external supervisor. - -zwasm v2 is the cited template: `Module.Budget = union(enum){ unmetered, -limited: u64 }` charged in the dispatch hot loop (`src/interp/dispatch.zig`), -exhaustion surfacing as `Trap.Interrupted` / `Trap.OutOfFuel` that unwind -the interpreter past any guest handler — never a host panic. - -cljw already has the cheap-poll site to ride: the VM back-edge safe point -(`vm.zig:127`, one predicted-not-taken atomic load) and the TreeWalk loop -back-edges (`tree_walk.zig:910` eval-loop, `tree_walk.zig:1280` fn-tail -recur). It also already has an **uncatchable-error** mechanism: -`host_class.kindToHostClass(kind)` returns null for `internal_error` / -`out_of_memory` / `not_implemented`, so those propagate past every `catch` -to the top-level CLI handler (`tree_walk.zig:968`). - -## Decision - -1. **New file** `src/runtime/concurrency/eval_budget.zig` (sibling to - `safepoint.zig`; A2 new-feature-new-file, A6 ≤1000 lines) defining: - - ``` - EvalBudget = struct { - step_ceiling: ?u64, // max back-edge crossings; null = no step bound - deadline_ns: ?i128, // monotonic deadline (clock.nanoTime); null = no time bound - steps: u64 = 0, // running counter - // tick(io) -> ClojureWasmError!void : ++steps; if past ceiling -> raise; - // every 1024th step read clock, if past deadline -> raise (throttled). - } - ``` - -2. **`Runtime.eval_budget: ?EvalBudget = null`** — mirrors `fs_jail_root` - (null = unmetered; F-006 the budget reads `rt.io` for the clock, never a - global). One optional-unwrap at each poll site → ~zero cost when unmetered - (P3 core stable; F-013 single-binary cold-start untouched), matching the - existing single `gc_requested` load. - -3. **Two axes, both shipped** (P4 no ad-hoc patch): a wall-clock **deadline** - (the real playground bound) AND a **step ceiling** (the deterministic, - dual-backend-testable bound). The clock read is throttled (every 1024 - steps) so the per-step cost stays a counter increment + compare. - -4. **Checks ride all three existing back-edge poll sites** — `vm.zig:127`, - `tree_walk.zig:910`, `tree_walk.zig:1280` — so a non-allocating infinite - loop is caught under BOTH backends (F-011 behavioural equivalence; the - step unit is "one back-edge crossing", uniform across backends). The - introducing commit lands all three sites + the test together (ADR-0036). - -5. **Expiry is UNCATCHABLE.** A new `Kind` (`resource_exhausted`) is added - that `kindToHostClass` does NOT map (returns null), so the budget error - propagates past any `(try … (catch …))` to the top-level handler — exactly - like `out_of_memory` and exactly like zwasm's trap unwinding past a guest - handler. This is the correction to D-351's "catchable" wording: a catchable - budget error lets untrusted code `(try (catch Throwable _ :ok))` - swallow the timeout, so the playground could never reliably report it. - New Codes: `eval_deadline_exceeded`, `eval_steps_exceeded` (catalog - templates, no Phase/ADR/path leakage per error_catalog_only). - -6. **Arming surfaces** (complementary, both finished-form): - - **CLI env var** (this cycle): `CLJW_EVAL_MAX_STEPS` + `CLJW_EVAL_DEADLINE_MS`, - mirroring `CLJW_FS_ROOT`, armed in `runner.zig` next to `rt.fs_jail_root`. - A whole-run bound; e2e-testable. - - **Clojure programmatic** (D-355 cycle): a `cljw.eval/with-budget` scoped - form for the playground's per-eval in-process bound. Deferred to the - cycle that consumes it (F-003); not stubbed now. - -7. **Complementary, unchanged**: the existing `MAX_CALL_DEPTH = 512` stack - guard (`tree_walk.zig:584`) stays — it bounds non-tail deep recursion that - a back-edge budget does not see. - -## Consequences - -- An embedder / the playground can bound untrusted eval in-process by time - and steps, removing the OS-subprocess dependency for the time/step - dimensions (D-355 unblocks; crash isolation D-354 still needs a process / - wasm boundary). -- The hot loop gains one optional-unwrap per back-edge when unmetered - (negligible; same shape as the GC poll). -- A new `Kind` widens the error taxonomy by one; `kindToHostClass` gains one - null-mapped arm. - -## Affected files - -- `src/runtime/concurrency/eval_budget.zig` (new) -- `src/runtime/runtime.zig` (`eval_budget` field) -- `src/eval/backend/vm.zig` (poll-site check) -- `src/eval/backend/tree_walk.zig` (two poll-site checks) -- `src/runtime/error/info.zig` (`resource_exhausted` Kind) -- `src/runtime/error/catalog.zig` (two Codes + templates) -- `src/runtime/host_interface.zig` / host_class (null map for the new Kind) -- `src/app/cli.zig` + `src/app/runner.zig` (env-var arming) -- tests: e2e (env budget + infinite loop → uncatchable budget error), unit per backend. - -## Extension landed: heap axis (D-352, isolation dim (b)) - -The memory axis the DA anticipated (DP4 note D-4) landed alongside steps/deadline: - -- **`heap_ceiling: ?usize` on `GcHeap`** (not `EvalBudget`) — the byte accounting - lives on the heap, so the check rides `GcHeap.alloc` directly. Checked at the - **alloc boundary, NOT the back-edge poll**: a Zig primitive (e.g. a bulk seq - realization) can allocate megabytes without crossing an eval back-edge, so the - deadline/step polls would miss it. The cap compares **live bytes** - (`bytes_allocated - bytes_freed`), which stays correct if threshold-driven - auto-collect ever lands (today collection is explicit-only). -- **REFUSE, never trigger-a-collect**: past the cap `alloc` returns - `error.OutOfMemory` (uncatchable — a raw Zig error with no catalog Info is not - matched by any `(catch …)`; with the hook below, the Info's `resource_exhausted` - Kind keeps it uncatchable too). -- **Message via a vtable hook** (`heap_exceeded_hook: ?*const fn(usize) void`): - `gc_heap` may not import the error catalog (the `catalog → big_int → gc_heap` - cycle), so a higher layer (`eval_budget.heapExceededHook`, installed on - `rt.gc` by `runner`) SETS the `eval_heap_exceeded` Info; `alloc` then returns - `error.OutOfMemory`. The hook returns **void** (not `anyerror` — a returning- - error hook poisons `alloc`'s inferred error set to the global set, breaking - every explicit-error-set caller, e.g. the analyzer / vm compiler). -- **Armed via `CLJW_EVAL_MAX_HEAP_MB`** (mirrors the step/deadline env vars). -- New catalog Code `eval_heap_exceeded` (Kind `resource_exhausted`). - -## Main-loop decision after Devil's-advocate - -The DA (verbatim below) recommends **Alt 2 (per-eval `ExecBudget` ownership)** -over the draft's ambient-`?EvalBudget`-on-`Runtime`, on the DP4 reset concern: -a long-lived playground server evaluating many snippets in one process has no -clean reset for a single mutable counter on `Runtime`. - -The main loop **keeps ambient-on-`Runtime` for D-351**, for reasons the DA's -framing missed, NOT on cycle/diff grounds (that would be the Cycle-budget -defer smell): - -1. **The ownership model is owned by D-355 (F-003 deferral).** Whether the - playground evaluates in-process (many snippets, one `Runtime`) or keeps a - **fresh `cljw` process per eval** is undecided — and D-354 explicitly argues - the process boundary stays for *crash* isolation (a Zig panic in untrusted - code can only be contained by a process / wasm boundary). In the - process-per-eval shape (the likelier finished form given D-354), each eval is - a fresh `Runtime` with a whole-run env-var budget — the draft's ambient model - is then exactly right and per-eval threading is dead weight. Prejudging this - in D-351 violates F-003. -2. **The DA's DP4 "manual reset" critique assumes the wrong reset mechanism.** - The in-process case resets by **save/restore scoping**, not by per-call-site - mutation: the deferred `cljw.eval/with-budget` form saves `rt.eval_budget`, - installs a fresh one for its dynamic extent, and restores on exit (the same - shape as a dynamic-var binding). Reset is scoped, not manual — so ambient-on- - `Runtime` serves the in-process case too, on the same substrate - `fs_jail_root` already uses (a single shared isolation knob, single-threaded - eval). -3. **Re-pointing later is local, not a Supersedes chain.** If D-355 picks fully- - in-process-concurrent eval, the budget reads move from `rt.eval_budget` to a - frame field at exactly the 3 poll sites + signatures — bounded surgery, owned - by the cycle that needs it. - -The DA's **clearly-correct, architecture-independent fixes are adopted into -cycle 1**: - -- **Explicit latch (DP1b)**: `EvalBudget` carries `tripped: bool`; once any axis - trips, every subsequent `tick` re-raises immediately, so an uncatchable error - reaching a `(catch …)` body (it cannot, but defensively) gets at most one - back-edge-free straight-line burst, never a second loop. -- **Uncatchable from Clojure via `kindToHostClass → null`** (as drafted) is kept; - the **host-observable graceful-report path (DP1a / Alt 2 two-frame split)** is - recorded as a D-355 concern — when the playground's host frame exists, it can - catch `error.BudgetExceeded` and report "timed out" without tearing down the - server. Cycle 1's degenerate host frame is the CLI top-level. -- **Cross-backend step-count divergence (DP2) — documented here, NOT a clj-AD.** - On reflection the DA's "add an AD-NNN" is the wrong SSOT: `accepted_divergences.yaml` - classifies **cljw-vs-JVM-clj** divergences, not VM-vs-TreeWalk internal ones. The - dual-backend differential oracle asserts *equal Value*, and the budget-exceeded - **outcome is equal** under both backends (both raise `error.ResourceExhausted` / - Kind `resource_exhausted`) — only the never-asserted internal step count differs, - so nothing floats in either the clj sweep or the diff oracle. The intent is locked - by `evaluator.zig`'s `test "ADR-0125: a step budget kills an infinite loop under - BOTH backends"` (fresh budget per backend → each counts independently to the same - ceiling and trips), and the rule that budget-expiry is never diff-tested on the - exact count lives in this ADR. No `AD-NNN` row is minted. - -Deferred to follow-ups (noted on D-351's debt row, not stubbed): the **call- -boundary poll site (DP5)** that would close the time-overshoot hole in non- -allocating straight-line bursts, and the **shared-IR deterministic step unit -(Alt 3)** that would make budget-expiry itself diff-testable. Cycle 1 ships the -back-edge sites the debt row scoped; steps is the deterministic unit for unit -tests, the deadline is gate-asserted only via an e2e "killed within a generous -ceiling". - -## Alternatives considered - -(Devil's-advocate subagent, fresh context, verbatim.) - -### Alt 1 — Smallest-diff: steps-only, single uncatchable ceiling on `rt` - -Drop the deadline axis from cycle 1 entirely. Ship only `step_ceiling: ?u64` on -`Runtime`, checked at the three back-edges, raising the uncatchable -`resource_exhausted`. No clock read, no `i128`, no 1024-throttle, no `rt.io` -dependency in `tick`. The wall-clock deadline lands in the D-355 cycle that -actually consumes it (the playground), co-located with `with-budget`. - -- **Better than the draft:** removes the one axis that is *inherently untestable - in the gate* (wall-clock firing is load-dependent — see DP3). `tick` becomes a - pure counter compare with zero F-006 surface. Smallest possible widening of the - error taxonomy that still kills an infinite loop under both backends. -- **Breaks/costs:** the playground's *real* bound is wall-clock ("your code ran - > 2 s"), not step count. Steps-only means the operator must hand-pick a step - number that correlates poorly with time. Defers the feature D-355 was filed for. - Per F-002 this is the smaller-diff option and is recorded but not recommended. - -### Alt 2 — Finished-form-clean: per-eval `*ExecBudget` on the eval/call boundary, host-observable / Clojure-uncatchable trap - -Stop modelling the budget as a mutable field on the long-lived `Runtime`. The -playground is a long-lived server evaluating *many* snippets — a single `steps` -counter on `Runtime` is a category error (accumulates across snippets, or every -call site must reset it). Instead: `ExecBudget` is a value owned by ONE eval -invocation, passed as `?*ExecBudget` on `eval(rt, env, locals, expr)` and the VM -frame; `rt` holds only the ambient default (CLI env-var path). Reset = construction. -Keep expiry uncatchable *from Clojure* but make the trap a distinct unwind the -**Zig embedder boundary** catches as a normal `error.BudgetExceeded` return — the -playground's host frame reports "timed out" in-process without tearing down the -server, exactly as zwasm's host gets `Trap.Interrupted` while the guest cannot -catch it. The cljw analogue of zwasm's "host" is the Zig/`with-budget` frame, not -the Clojure `try`. - -- **Better than the draft:** solves the reset-semantics hole (DP4); gives the - embedder the graceful "timed out" path without making it catchable by untrusted - Clojure; matches the cited zwasm semantics precisely. -- **Breaks/costs:** threads a parameter through the eval/VM dispatch signature; - the VM carries it on its frame struct; the host-frame catch needs a boundary - primitive (outermost metered frame converts trap → normal return). Larger diff; - recommended anyway per F-002, not downgraded on size grounds. - -### Alt 3 — Wildcard: deterministic logical-step budget as the only tested bound; wall-clock demoted to explicit AD - -Define the step unit as something both backends increment identically — not -"back-edge crossing" (which diverges) but **evaluated analyzer nodes** (shared -IR). Then the step ceiling is contract-tested (a diff_test asserts equal expiry -because the unit is on the shared analyzer IR), and the wall-clock deadline is -shipped but documented as an AD accepted divergence (operational kill-switch, -e2e-only). Additionally ride the alloc prologue / call boundary so straight-line -blow-ups are bounded (DP5). - -- **Better than the draft:** repairs the cross-backend step divergence (DP2) and - the back-edge-only granularity gap (DP5); the only option where a budget - diff_test is meaningful. -- **Breaks/costs:** requires an op→node charge model the VM lacks today; per-op - increment is a heavier hot-path touch (F-013 inert-when-unmetered must be - re-verified across far more sites). Largest diff. - -### Per-decision-point analysis - -**DP1 — Catchable vs uncatchable.** Uncatchable-*from-Clojure* is correct (a -latched budget lets a `(catch)` body run at most one back-edge-free burst, not an -unbounded second loop — the draft must state the latch explicitly, it currently -doesn't). But "uncatchable" should not mean "propagate to the CLI top-level and -tear down the run". The right analogue of zwasm's host is the Zig/`with-budget` -embedder frame, not the Clojure `try` — zwasm's guest can't catch the trap *and -the host still gets it as a return value*. The draft collapses these, losing the -embedder's graceful-report path. So: both tiers, split by **frame**, not by -soft/hard severity (a soft/hard severity tier reintroduces the swallow-the-timeout -hole for the "soft" tier). - -**DP2 — Step non-determinism across backends.** This *does* dent F-011 and the -draft under-states it. A budget-exceeded eval under VM vs TreeWalk produces the -same Kind but at a different step count; the diff harness can cover only -non-expiry. Promote "exact count may differ" to an explicit AD-NNN (it currently -floats, which clj_diff_sweep.md forbids). "Back-edge crossing" is not the right -unit if you want expiry itself diff-tested; a shared-IR unit (Alt 3) is. If you -keep back-edge units, budget tests are necessarily per-backend + e2e only — state -that in the ADR. - -**DP3 — Deadline determinism.** A wall-clock deadline is non-reproducible by -design — a property, not a bug, but a gate liability if any test asserts *when* it -fires. Resolution: ship it but never gate on its timing (e2e asserts "infinite -loop dies within a generous ceiling"; unit tests cover only steps). Steps is the -testable proxy; deadline is the operational truth. - -**DP4 — Where the budget lives.** The draft's biggest unforced error. `?EvalBudget` -inline on the long-lived `Runtime` has no answer to "how does it reset between the -playground's many snippets". Either every snippet mutates `rt.eval_budget.?.steps -= 0` (reset-discipline bug, racy if concurrent) or the counter accumulates. The -finished-form answer is per-eval ownership (Alt 2): budget constructed per eval, -carried on the frame; `rt` holds only the ambient default. Reset = construction. - -**DP5 — Granularity gap.** Back-edge-only polling misses straight-line blow-ups -(giant `(str …)`, large literal build, non-looping realization with no recur). -Leaning entirely on D-352 is fine for *memory* runaway, false for *time* runaway -in non-allocating straight-line code: a burst with no back-edge can overshoot the -deadline arbitrarily. The step check should also ride the **call boundary** (every -fn invocation — cheap, frequent, backend-shared) to close most of the gap without -full alloc-prologue coupling. - -### DA recommendation - -Alt 2 (per-eval budget + host-observable/Clojure-uncatchable trap), amended with -Alt 3's two correctness fixes (call-boundary poll site; AD-NNN for the step -divergence). Keep both axes (steps deterministic-tested, deadline operational-only). -Do not adopt Alt 1 (dropping the deadline defeats the mission need). The draft as -written is shippable for the whole-run CLI env-var case but has a latent DP4 reset -bug the D-355 playground cycle hits on its second eval. - -*(Main-loop disposition of this recommendation is in the "Main-loop decision" -section above: ambient-on-`Runtime` is retained for D-351 because the ownership -model is D-355's F-003-owned structural choice and save/restore scoping serves the -in-process case; the latch + AD-NNN fixes are adopted now; call-boundary + shared- -IR unit are deferred follow-ups noted on D-351.)* - -## Revision history - -- **2026-06-09 — the deferred `with-budget` host surface landed (D-355 Path A).** - `cljw.eval/with-budget opts thunk` (`src/runtime/cljw/eval/with_budget.zig`) - realizes the host-observable path the DA's Alt 2 / DP1 described, on top of the - ambient-on-`Runtime` substrate this ADR kept: it SAVES the prior budget + heap - cap, installs the opts (`:max-steps` / `:deadline-ms` / `:max-heap-mb`), calls - the thunk via `rt.vtable.callFn`, and **catches the (Clojure-uncatchable) budget - breach as a raw Zig error** (`error.ResourceExhausted` / `error.OutOfMemory`), - returning `{:cljw.eval/exhausted :steps|:deadline|:heap}` and restoring the saved - budget over the dynamic extent (the save/restore scoping this ADR predicted, so a - long-lived server resets per call). This is the sanctioned embedder frame — the - Clojure `(try …)` still cannot swallow the budget, only this Zig host frame can — - exactly the two-frame split the DA recommended. Subtlety found in implementation: - the result map must be built AFTER restoring the cap, or a heap breach re-trips on - the map's own allocations. With this, the playground evaluates untrusted code - in-process (no babashka subprocess) — `eval` / `read-string` / `with-out-str` - already exist — and a runaway is reported, not fatal. Crash isolation (a Zig - panic, D-354) is still out of scope for in-process; fly.io instance isolation + - restart covers it at deploy time. diff --git a/.dev/decisions/0126_clojure_java_io.md b/.dev/decisions/0126_clojure_java_io.md deleted file mode 100644 index f08c3e589..000000000 --- a/.dev/decisions/0126_clojure_java_io.md +++ /dev/null @@ -1,368 +0,0 @@ -# ADR-0126 — clojure.java.io + java.io.File (full surface, cljw-style) - -- Status: Proposed → Accepted -- Date: 2026-06-09 -- Deciders: autonomous loop (user-directed: "clojure.io の機能を full に入れたい") -- Tier: A for `java.io.File` (per the `compat_tiers.yaml` SSOT, which is - authoritative for tier; the ROADMAP §6 table's "B" is superseded by the - SSOT). `clojure.java.io` ns: Tier B (ROADMAP §6: same names, Zig-native - I/O backing). - -## Context - -The babashka-independent playground (D-355) needs file existence checks, -path manipulation, and (eventually) binary serving. More broadly the user -directed a **full** introduction of `clojure.java.io`, settling in for the -long haul ("時間がかかってもよい / しっかり腰を据えて"), doing it in -ClojureWasm style where the JVM mechanism does not map, completing the -partial `java.io.File` skeleton along the existing flow, and enumerating -gaps for an ideal-form introduction. - -`clojure.java.io` (upstream `src/clj/clojure/java/io.clj`) is a polymorphic -I/O system built on two protocols + a multimethod: - -- **Coercions** — `as-file`, `as-url`. -- **IOFactory** — `make-reader`, `make-writer`, `make-input-stream`, - `make-output-stream`; with public wrappers `reader`/`writer`/ - `input-stream`/`output-stream`. -- **do-copy** — a `defmulti` keyed on `[(type input) (type output)]`, - surfaced as `copy`. -- Plus `file`, `as-file`, `as-relative-path`, `delete-file`, - `make-parents`, `resource`. - -### Prerequisite chain (verified 2026-06-09) - -- `extend-type` / `extend-protocol` work (macro_transforms.zig → `rt/__extend-type!`). - Upstream's `(extend T IOFactory (assoc default-streams-impl …))` fn-form is - re-expressed cljw-style with `extend-protocol` + explicit per-type impls. -- `defprotocol` / `defmulti` / `defmethod` / `with-open` / `reify` / - `deftype` all present; `type` fn present; defmulti vector-dispatch is - e2e-tested (phase7/phase15). -- Canonical host-type pattern = `runtime/java/util/Random.zig`: a - `.host_instance` (ADR-0106) carrying a per-instance `*const TypeDescriptor`, - `` + instance methods registered into `method_table` in the `init` - callback. Dispatch: analyzer → `InteropCallNode{.instance_member}` → - `tree_walk.evalInstanceMember` → `td.lookupMethod`. -- FS-jail (`file_io.zig`, ADR-0123): every FS-touching op routes the path - through the jail resolve and maps `error.FsJailEscape` → `.fs_jail_escape`. - -## Decision - -Introduce `java.io.File` (full method surface) + a `clojure.java.io` -namespace covering the **char/text + File + stream-object** surface that -cljw's current phase supports without unwinding a deferred structural -decision. Land it in coherent, individually-gated TDD cycles: - -1. **java.io.File host type** — rebuild the `runtime/java/io/File.zig` - skeleton from the (wrong) typed_instance `field_layout` shape to the - `host_instance` shape (`fqcn = "java.io.File"`, payload = path string). - Methods: `exists`, `isFile`, `isDirectory`, `getName`, `getPath`, - `getAbsolutePath`, `getCanonicalPath`, `getParent`, `getParentFile`, - `length`, `isAbsolute`, `canRead`, `canWrite`, `delete`, `mkdir`, - `mkdirs`, `list`, `listFiles`, `toString`; 1- and 2-arg ctor. FS-touching - methods reuse the jail; pure path ops (getName/getParent/…) skip it. - Neutral impl in `runtime/io/` per F-009. - -2. **Coercions + file family** — `clojure.java.io.clj` embedded via - bootstrap. `as-file` (String/File/nil), `file`, `as-relative-path`, - `delete-file`, `make-parents`. - -3. **Host stream types — ONE generic `host_stream` representation** - (adopting Devil's-advocate Alt 2, F-002). A single `host_instance` - carrying a Zig handle/buffer + a `kind` tag - (`{input,output,reader,writer} × {file,buffered,string,byte}`) + - direction/encoding. The JVM class *names* (`BufferedReader`, - `FileInputStream`, …) are **rows in a closed-set descriptor table** - (the F-013 / ADR-0102 `host_interfaces.yaml` pattern) that all route to - the one generic stream surface — NOT ~10 sibling TypeDescriptors. Under - the no-JVM rule (ADR-0059) there is no `extends` relationship to model - (`BufferedReader is-a Reader` is a JVM-class-hierarchy fact cljw does not - represent), so minting a class tree would import a hierarchy the project - rejected. **Binary stream transport lands NOW**: the payload lives in a - Zig `[]u8` inside the host instance and never surfaces as a cljw - byte-array Value, so D-051 does not block it (D-051 defers the byte-array - *Value*, not byte *transport*). FS-jail (`jailResolve`) is applied at - every stream-open site, not just `slurp`/`spit`. - -4. **IOFactory** + `reader`/`writer`/`input-stream`/`output-stream`. - -5. **copy** (`do-copy`) over File/Reader/Writer/InputStream/OutputStream/ - String — incl. binary stream-to-stream / stream-to-file (Zig `[]u8` - buffers). This is what the playground needs to serve binary wasm/png. - -6. **resource** — no classpath concept in cljw → explicit - `feature_not_supported` + debt. - -7. **cljw.json + cljw.fs wrappers** — require-able `cljw.*` aliases. Per - F-009 the JSON parse/emit + map↔value conversion bodies live in a - **namespace-neutral impl** (`runtime/json.zig` + a fs-helper impl); - `cljw.json`/`cljw.fs` (and the existing `clojure.data.json` surface) are - thin wrappers over it — never a fork. `read-str` 2-arity `:key-fn`/ - `:value-fn` coercion lives in the neutral impl. - -### Deferral boundary (cljw-style / phase-gated, each a tracked debt row) - -- **byte-array-Value-facing arms only** — `(.read stream ba off len)` into a - cljw byte-array, and the byte-array / char-array `IOFactory` + `do-copy` - arms — depend on the cljw byte-array Value representation, deferred to - Phase 16 (D-051, F-003). The generic `host_stream` reserves these method - slots as transient `feature_not_supported` stubs so the Phase-16 landing - is a wire-in, not a struct migration. **Binary byte transport is NOT - deferred** (see Decision item 3). -- **URL / URI / Socket coercions + IOFactory arms** — no host types yet - (ROADMAP `java.net.*` Phase 14+). Ship as `feature_not_supported`. -- **resource** — needs a classpath/classloader concept cljw lacks. - -These are not workarounds: the underlying subsystem genuinely does not -exist at this phase. Each is recorded so the surface is honestly partial, -not silently lying (per `provisional_marker.md` / no-op-stub rules). - -## Consequences - -- File + stream host types are the first sizeable host-object subsystem; - they establish the Zig-file-handle-backed stream pattern future I/O - (sockets, channels) reuses. -- `clojure.java.io` becoming real unblocks the D-355 playground port's file - ops without babashka.fs. -- The deferral rows keep the URL/Socket/byte-array gaps visible for the - phase that owns them. - -## Alternatives considered - -> Devil's-advocate subagent (fresh context) output, reflected verbatim. The -> main loop **adopted Alt 2** (its recommendation), revising Decision items -> 3, 5, 7 + the deferral boundary above to match. - -The 7-cycle draft (full `java.io.File` surface + `clojure.java.io` -Coercions/IOFactory/copy + ~10 stream host TypeDescriptors mirroring JVM -class names + `cljw.json`/`cljw.fs`, with byte-array/binary-I/O and -URL/URI/resource deferred) was stress-tested against the F-NNN envelope. -Three alternatives within those constraints: - -### Alt 1 — Smallest-diff: File + char/text streams only, drop the IOFactory/copy/json/fs layers this cycle - -**Shape.** Cycles 1+3 only: rebuild `java.io.File` to the `host_instance` -shape, plus the char/text stream host types -(`Reader`/`Writer`/`BufferedReader`/`StringReader`/`PushbackReader`) backed -by Zig `[]u8`. `clojure.java.io` ships just -`as-file`/`file`/`delete-file`/`make-parents` + a 1-arm `reader`/`writer` -over File. No `IOFactory` protocol, no `do-copy` defmulti, no -`cljw.json`/`cljw.fs`, binary streams deferred alongside byte-arrays. - -**Better than the draft.** Lands File + the highest-frequency -`slurp`/`with-open`/`line-seq` path with the least surface; defers every -multi-arm protocol/defmulti until the byte-array repr question is settled, -so nothing built this cycle has to be revisited when D-051 closes. Cleanest -*per-cycle* closure. - -**What it breaks.** It is a **drip-feed of one coherent surface** — -`clojure.java.io` is a single conceptual unit (Coercions + IOFactory + copy -together), and shipping `reader`/`writer` without the `IOFactory` protocol -means the dispatch backbone gets retrofitted later rather than designed -once. That is the Micro-coverage-grind / Smallest-diff-bias pattern F-002 + -`clj_diff_sweep.md` Discipline 2 forbid: a half-covered `clojure.java.io` -reads as "covered" in the ledger while the protocol that makes it -extensible is absent. Recommended against on F-002 grounds (cycle-count is -not a constraint). - -### Alt 2 — Finished-form-clean: ONE generic host-stream type + binary streams land NOW; mirror-class names become descriptor metadata, not distinct TypeDescriptors - -**Shape.** Two corrections to the draft, both F-013/no-JVM-driven: - -1. **One `host_stream` representation, not ~10 class-named - TypeDescriptors.** A single host-instance carrying a Zig handle/buffer + - a `kind` tag (`{input,output,reader,writer} × {file,buffered,string,byte, - pushback}`) + a `direction`/`encoding` field. The JVM class *names* - (`BufferedReader`, `FileInputStream`, …) become **rows in a closed-set - descriptor table** (the F-013 / ADR-0102 `host_interfaces.yaml` pattern) - that all route to the one generic stream surface — exactly clause 3 of - F-013 ("every recognised entry routes to a *generic* surface; a - per-library shim has no slot"). Under the no-JVM rule (ADR-0059: - TypeDescriptor, no Class hierarchy) there is **no `extends` relationship - to model** — `BufferedReader is-a Reader` is a JVM-class-hierarchy fact - cljw explicitly does not represent, so minting 10 sibling TypeDescriptors - imports a hierarchy the project rejected. `instanceof`/`isa?` questions - answer off the `kind`/`direction` fields, not a descriptor parent chain. - This also collapses the F-004 slot pressure (one `host_instance` slot vs. - a family). - -2. **Binary streams (`InputStream`/`OutputStream`/`FileInputStream`/ - `FileOutputStream`/`ByteArray*Stream`) land NOW**, because **D-051 is not - actually a blocker for them.** D-051 defers the *cljw byte-array Value* - (`[]Value`, one Value/byte). A binary stream's payload lives in a **Zig - `[]u8` inside the host instance** and never surfaces as a cljw Value — - identical to how the draft's char/text path already holds `[]u8` without - surfacing a cljw string-of-bytes. The only operations that *require* a - byte-array Value are `(.read stream byte-array off len)` and the - byte-array `do-copy`/`IOFactory` arms — those, and only those, defer to - Phase 16. `(io/copy in-stream out-stream)`, - `(io/input-stream "f.png")`, stream-to-stream - `(io/copy (io/input-stream src) (io/output-stream dst))` all work today - over `[]u8` buffers. **This is the load-bearing finding for question - (a):** the playground serving binary wasm/png is *stream-to-stream / - stream-to-file copy*, which needs zero byte-array Value. Deferring all - binary I/O conflates "byte-array Value" (correctly deferred per - D-051/F-003) with "binary byte transport" (not blocked). Deferring - transport is a smallest-diff convenience the finished-form owner would - unwind, and it leaves the playground's actual use case unserved. - - *Confinement (question c):* `jailResolve` is a stateless lexical resolve - called at open time and returns the confined path; it does not enforce - per-read. For long-lived open streams this is **correct as-is** — the - file handle is bound to the jail-resolved path at open, and the OS handle - cannot later escape the jail (the path was validated before `openFile`). - The residual is the documented D-342 symlink-inside-jail case, which is - orthogonal to stream lifetime. No new confinement model is needed; the - draft's "reuse the slurp/spit jail" is right, *provided* the jail resolve - happens at every stream-open site (not just `slurp`/`spit`) — make that - an explicit cycle-3 assertion. - -3. **json/fs map-conv handy fns live in a namespace-neutral impl (F-009), - not in `cljw.json`/`cljw.fs`.** Per F-009 the JSON parse/emit + - map-conversion bodies belong in `runtime/json.zig` (+ `runtime/fs.zig` - for path handy fns); `cljw.json`/`cljw.fs` are **thin Clojure/cljw-surface - wrappers** over them, and — critically — so is any future - `clojure.data.json` compatibility surface. Forking the impl into a - `cljw.json`-owned body (question d: alias-vs-fork) would re-create the - cw-v0 anti-pattern F-009 was written to kill (impl trapped inside one - surface, unshareable). So: **neutral impl + thin wrapper, never fork.** - `read-str :key-fn` keyword-coercion lives in the neutral impl. - -**Better than the draft.** Aligns with the project's two structural laws -that the draft half-followed: no-JVM (one descriptor, not a mirrored class -tree) and F-013 (closed-set table routing to a generic surface). Serves the -playground's binary use case *now* without violating D-051. Puts json/fs -impl where F-009 mandates so the eventual `clojure.data.json` surface is a -wrapper, not a rewrite. Collapses F-004 slot consumption. - -**What it breaks.** Larger diff this cycle (one richer host-stream module + -a descriptor table + binary open/copy paths). Per F-002/F-011 this is -**not** a reason to downgrade — cycle/diff/LOC is not a project constraint. -The one genuine cost: the generic-stream `kind` tag must be designed to -admit the deferred byte-array `read(ba,off,len)` arm without reshaping when -D-051 closes — i.e. the host-stream struct reserves the byte-array-facing -method slots as `feature_not_supported` stubs (transient-stub row of -`provisional_marker.md`), not absent fields. That keeps the Phase-16 -byte-array landing a wire-in, not a struct migration. - -### Alt 3 — Wildcard: define byte-array repr now as a host-instance over Zig `[]u8`, decoupled from D-051's `[]Value` collection-tower question - -**Shape.** Observe that D-051 is specifically about the *collection-tower* -byte-array (the `[]Value` representation that participates in -`seq`/`conj`/`aget` as a first-class persistent-ish value, deferred to -Phase 16 with the rest of the Java-array tower per F-003). But a -**transport-only byte buffer** — a `host_instance` wrapping a Zig `[]u8`, -supporting only `aget`/`alength`/`(.read …)`/`(.write …)` and round-tripping -through streams — is a *different object* that could land now and satisfy -`(byte-array n)`, `(.read in buf)`, the byte-array `IOFactory`/`do-copy` -arms, and binary `slurp` returning bytes. Phase 16's `[]Value` tower would -then either subsume or coexist with it. - -**Better than the draft (and Alt 2).** Unblocks the *complete* -`clojure.java.io` surface incl. every byte-array arm in one push (true -big-bang per Discipline 2), and gives the playground binary read-into-buffer, -not just stream-to-stream copy. - -**What it breaks — and the F-NNN finding (leading entry).** This is **the -alternative that requires violating an F-NNN**, recorded first per the -brief. D-051 + F-003 deliberately defer the byte-array *representation -decision* to the Phase 16 owner (the Java-array tower owner), because -byte-array repr is a **structural plan** (it interacts with the F-004 slot -map's `array` slot in Group D, the numeric-tower boxing of byte values, and -`aget`/`aset` polymorphism across all primitive array types). Minting a -*second* byte-array object now — even a "transport-only" one — is exactly -the decision-seizure F-003 forbids: it pre-commits the Phase-16 owner to -either subsume or reconcile a representation the current cycle invented, -which is the structural pre-emption F-003 exists to prevent. The "it's a -different object" framing is the rationalization that makes it feel safe; in -finished form there is **one** byte-array concept and its shape is the -Phase-16 owner's call. **Verdict: rejected — violates F-003 (structural -decision-deferral) and front-runs D-051.** The transport need it targets is -already met by Alt 2's Zig-`[]u8`-inside-the-stream (which surfaces *no* -byte-array object at all), so the F-003-violating path buys only the -byte-array-Value-facing arms, which are correctly Phase-16 work. - -**Recommendation (non-binding):** Take **Alt 2** — one generic -`host_stream` + closed-set descriptor table (no-JVM/F-013), binary stream -*transport* landed now over Zig `[]u8` (D-051 does not block it; the -playground needs it), json/fs impl in neutral files with thin `cljw.*` -wrappers (F-009), byte-array-Value-facing arms left as transient -`feature_not_supported` stubs for Phase 16. - -## Revision history - -### 2026-06-09 — Amendment: stream `class`/`instance?` made clj-faithful (concrete+chain) + lexical import resolution (D-358) - -Decision point 3 originally modelled the JVM stream class names as a FLAT -closed-set table all routing to the one generic stream, on the reasoning -"no-JVM (ADR-0059) means there is no `extends` relationship to model." The -first D-358 implementation followed that literally: any reader leaf was -`instance?`-true and `(class s)` returned the family (`java.io.Reader`). The -**`clj` oracle proved this DIVERGES** (F-011): clj's `io/reader` returns a -*concrete* `java.io.BufferedReader`, `(instance? java.io.FileReader (io/reader f))` -is **false** (a sibling), and `(class …)` is the concrete buffered type. The -"no extends to model" reasoning was right about cljw's INTERNALS but wrong to -let the OBSERVABLE answers diverge — F-011 requires `class`/`instance?` to -mirror clj even when the internal representation (one generic `host_stream`, -no real hierarchy) does not. - -**What changed:** -1. `runtime/io/stream_classes.zig` now models, per kind, the CONCRETE buffered - class clj's coercion returns + its java.io superclass CHAIN (the - `instance?`-true set): reader→{BufferedReader,Reader}, - writer→{BufferedWriter,Writer}, input→{BufferedInputStream,FilterInputStream, - InputStream}, output→{BufferedOutputStream,FilterOutputStream,OutputStream}. - host_stream stamps the concrete as the descriptor `fqcn` (→ `(class s)` - clj-faithful) and the chain as `protocol_impls` (the existing - `class_name.matchUserType` arm). cljw still keeps ONE generic `host_stream` - internally — only the observable answers mirror clj. -2. The recognised `isKnown` set is the COMPREHENSIVE java.io stream surface - (chain ∪ a sibling list covering every public Reader/Writer/InputStream/ - OutputStream subtype), so `(instance? java.io.LineNumberReader rdr)` is a - clj-faithful **false**, not `class_name_unknown` (F-013 clause 4: 網羅 the - recognition table — cheap rows, no impl, definition-derived from the java.io - package, NOT a per-library allowlist). A name outside that set still raises - `class_name_unknown`: cljw has no classpath to confirm an arbitrary FQCN is a - real class (an inherent no-classpath divergence; clj would resolve it or - `ClassNotFound`). -3. An imported simple class name resolves to its FQCN for `instance?` in TWO - layers: (a) **lexically at analyze time** via `special_forms.resolveInstanceClassArg` - (reads the lexical ns's `(:import …)` map — the same D-235 `ns.imports` that - `resolveJavaSurface` reads for `(Class. …)`), so a fn that closes over an - `(:import …)` resolves when called from another ns (clj resolves class symbols - at compile time); (b) **at runtime** in `__instance?` (`resolveClassImport`) - for a class imported AFTER analysis (REPL `(import …)` then use). Both - `(import …)` and `(ns … (:import …))` populate `ns.imports`, so both forms - work. No per-class allowlist — the same shared lexical resolver constructors - use (F-013-canonical / DA Alt b). - -**Devil's-advocate (fresh-context, F-NNN-bounded) — 3 alternatives + 2 gaps it -raised, both resolved before landing:** -- **Alt (a) smallest-diff** — `isKnown(java.io.*)` by a suffix *shape rule* - instead of a sibling enumeration. Better: no curated list. Worse: a shape - rule accepts a TYPO (`java.io.FooReader`) as known-false where clj raises - `ClassNotFound` — trades the F-013 allowlist worry for an F-011 edge - divergence. -- **Alt (b) finished-form-clean (ADOPTED for AXIS-1)** — a single shared - class-symbol resolver used by `instance?`/`class`/constructors + a - definition-derived supertype set. We adopted its AXIS-1 shape: lexical - resolution lives in `special_forms` next to `resolveJavaSurface` (the - analyzer's existing env-based class resolver), NOT threaded through all 55 - Zig macro expanders (wrong layer — expansion is syntactic) and NOT via an - `rt` side-channel field (hidden mutable state, duplicates `Env.current_ns`). -- **Alt (c) wildcard** — per-stream-instance concrete class + macro-expansion - (lexical) import resolution. Its lexical-correctness insight is real and is - what AXIS-1 (b) delivers via the analyzer; its macro-expansion placement was - rejected because it cannot see a runtime `(import …)` (a different F-011 - divergence) and cannot fix `(class …)` (a fn, not a macro). -- **Gap 1 (AXIS-1 lexical-vs-dynamic ns)** — the DA showed a fn closing over an - import, called under a different `*ns*`, broke with runtime-only resolution - (clj returns true; cljw threw). **Resolved this cycle** (user-directed) by the - analyze-time lexical resolver in (3a); e2e `import_cross_ns` locks it. -- **Gap 2 (F-013 sibling allowlist)** — the first sibling list was a curated - 12-name subset. **Resolved** by making it the comprehensive java.io stream set - (2); the no-classpath boundary for non-stream/unknown FQCNs is recorded as an - inherent divergence, not over-claimed (the corpus probes only recognised - names). - -Corpus: `test/diff/clj_corpus/io_stream_class.txt` (13 clj-verified lines). diff --git a/.dev/decisions/0127_print_method_multimethod.md b/.dev/decisions/0127_print_method_multimethod.md deleted file mode 100644 index 43a78c516..000000000 --- a/.dev/decisions/0127_print_method_multimethod.md +++ /dev/null @@ -1,196 +0,0 @@ -# ADR-0127 — `print-method` is a real multimethod with a native default, consulted by the pr path via a dirty-flag fast path - -- **Status**: Proposed → Accepted (2026-06-10) -- **Driven by**: D-370 — loading `flatland.ordered.set` fails at set.clj:153 - `(defmethod print-method OrderedSet [o w] (print-method (seq o) w))` → - `Unable to resolve symbol: 'print-method'`. cljw has no `print-method` var; - libraries routinely extend it to customise how their types print (F-010/F-013 - library-driven discovery; high leverage — unblocks many libs, not just ordered). -- **Relates to**: F-011 (behavioural equivalence vs `clj`, incl. nested - per-element recursion; commonization over effort), F-009 (impl-neutral writer - handle in `runtime/`, thin surfaces above — ADR-0029), F-002 (finished-form - wins; cycle/diff size is not a constraint), F-013 (definition-derived: cover the - whole print-method recursion contract, not just the direct-print case), - F-004 (2026-06-05 revision: writers are `host_instance` B13, no new NaN-box - slot), ADR-0059 / AD-003 (no-JVM: a "writer" is not a real `java.io.Writer` - class; `(class w)` is a simple name), ADR-0088 (cached-Var pattern, - `initPrintLimitVars`), D-238 (bindable `*out*` — the owning slice for a general - Writer/`*out*` redirect), D-369 (transient dispatch — sibling off-critical-path - follow-up). - -## Context - -`clj` routes ALL printing through `print-method`, a multimethod dispatching on -`(class x)`, with default methods per type. A user `(defmethod print-method T -[o w] …)` overrides T's printing, and the collection printers recurse by calling -`print-method` on every element — so an override fires even when its type appears -NESTED inside a native collection. - -cljw prints natively: `pr`/`prn`/`pr-str`/`print` are native primitives -(`src/lang/primitive/core.zig`) driving `src/runtime/print.zig` with a Zig -`w: *std.Io.Writer` (NOT a Clojure Value). There is no `print-method` var and no -`java.io.Writer` Value surface. cljw's multimethod stack -(`src/runtime/multimethod.zig`) is complete and already dispatches on deftype -types; `(class x)` returns interned `.type_descriptor` Values. `writeStrValue` -already carries the "consult an override, else native" idiom twice (`Object/ -toString` for typed_instance, `host_instance` via `rt.vtable.callFn`). - -Two load-bearing choices: (A) how the writer `w` passed to a user method is -represented (the user threads it back via `(print-method child w)`); (B) whether -the native pr path consults `print-method` per nested element (clj parity) and -how, given the recursive `printValue` is a pure `(w, v)` walk with no `rt`/`env`. - -## Decision - -**`print-method` is a real `defmulti` in `core.clj`** dispatching on `(class x)`, -whose `:default` delegates to a native `__print-method-default` primitive (the -existing native printer) — NOT ~40 re-derived clj default methods. The native pr -entry points consult it. - -**Choice A — writer representation = A2: a `host_instance` writer handle** -wrapping the active `*std.Io.Writer`. `state[0]` holds the writer pointer (a raw -`usize`, non-GC — no `host_trace` hook); its descriptor registers `write`/ -`append`/`flush` methods that unwrap `state[0]` and call straight through. The -consult mints this handle around the CURRENT sink (`out_capture` if set, else -`rt.stdout`), so `with-out-str`/`*out*` redirection works automatically. `(class -w)` is a clean simple name (no `java.io.Writer` FQCN — AD-003). This is the -F-004-declared finished form (writers = `host_instance` B13). **Invariant**: the -handle is single-print-scoped — `state[0]` is nulled after the print completes so -a stale `(.write w)` after the pr returns raises rather than reads a dangling -stack pointer. - -**Choice B — recursion gating = B2 with single-walk (b-ii)**: a dirty flag -`any_print_method_override` (set by `__add_method!` on the print-method MultiFn) -gates the consult. Flag false (99% + bootstrap) ⇒ pure native `printValue`, zero -clojure calls. Flag true ⇒ `printValue` carries a nullable `rt`/`env` (null on the -pure path) and the collection arms re-enter the per-element consult — ONE walk, -not two parallel collection-walk forms (F-011 commonization over the effort of -threading the signature). Matches clj's per-element recursion (a nested override -fires). **Recursion guard**: `__print-method-default` renders via `printValue` -directly, BYPASSING the consult on the SAME value; only child elements re-enter -the consult — terminating because each child lands on either an override or the -native default. The print-method Var is cached (ADR-0088 `initPrintLimitVars` -pattern), nullable-until-installed so early-bootstrap `pr` is safe. - -`print-dup` is DEFERRED (off the flatland.ordered critical path; `*print-dup*` -moot per D-222). The general bindable-`*out*`/Writer-protocol shape is D-238's -slice — the A2 handle is the down-payment value type that future `*out*` binds -to (so A2 shrinks, not enlarges, that rewrite; F-003 defer-to-owner). - -## Consequences - -- A `print-method` multimethod + `__print-method-default` primitive land; pr/prn/ - print/pr-str consult it behind the dirty flag. -- A writer `host_instance` type + descriptor (`write`/`append`/`flush`) lands in - `runtime/` (neutral); `(.write w s)` / `(.append w s)` work in a user method. -- `printValue` gains nullable `rt`/`env` params (the single-walk B2(b-ii) cost) — - every printer signature is touched; the pure fast path passes null. -- Lifetime invariant: the writer handle is single-print-scoped (nulled after); - documented + defended. -- Bootstrap: the print-method Var is cached nullable; pr before it installs takes - the native path. -- Perf: zero overhead when no override is registered (dirty flag); when set, one - cached identity-probe `getMethodNonDefault` per element (acceptable, deferred - per `optimization_deferred_until_15_libs`). -- e2e + corpus: a user `(defmethod print-method T …)` overrides direct + nested - printing, matching clj; `with-out-str` over an override. - -## Alternatives considered (Devil's-advocate fork, verbatim) - -> Fresh-context `general-purpose` DA-fork (agent abba199f8b683d98c), briefed with -> the F-NNN envelope. Reflected verbatim. - -Grounding facts shaping every ruling: `(class x)` on a `host_instance` returns its -descriptor's interned `.type_descriptor` — a `host_instance` writer is -dispatchable and `(class)`es as a simple name (`Writer`/`StringWriter`), NOT a -`java.io.Writer` FQCN (AD-003). `host_instance` is the proven home for a Value -wrapping a non-Value handle (File holds a path; the GC-ROOT §H raw-u64 pattern -exists). `writeStrValue` already has the consult-else-native idiom twice; -`with-out-str` is a threadlocal `out_capture` sink, not a Value. The native -printer splits cleanly: `printResult`/`writeStrValue` carry `rt`+`env`; the -recursive `printValue` switch is pure `(w, v)`. F-004's 2026-06-05 revision -already designates writers as `host_instance` B13 (no new slot). - -**CHOICE A — writer-value `w`:** -- **A1 — a `java.io.Writer` host-surface Value** (`runtime/java/io/Writer.zig`). - Better: matches the `^java.io.Writer` hint; reuses host_instance dispatch; - discoverable under ADR-0029. Breaks: exposing `java.io.Writer` as a CLASS is - AD-003-dishonest (cljw has no java.io; `(class w)` is a simple name anyway), and - the user almost never inspects `w`'s class — they thread it back / `.write` — - so the java naming is surface theater wrapping nothing cljw-original (F-009 has - no neutral impl to wrap). -- **A2 — a dedicated `host_instance` writer handle wrapping the active - `*std.Io.Writer`** (recommended). Better: the honest finished form; `state[0]` - = the writer pointer (raw usize, no GC trace), descriptor `write`/`append`/ - `flush` unwrap + call through (File.zig METHODS pattern 1:1); the consult mints - the handle around the current sink so `with-out-str`/`*out*` work; `(class w)` - is a clean simple name. Sits in F-004 (B13 host_instance), F-009 (neutral impl), - ADR-0059 (no fake class). Cost: LIFETIME — the `*Writer` is stack-scoped; a - user stashing `w` and reading after the pr reads a dangling pointer. Mitigation: - single-print-scoped handle; null `state[0]` after the print so a stale write - raises. Minor: a host_instance alloc per top-level pr (gated by the dirty flag). -- **A3 — `w` is the `*out*` sink / a cljw-native Writer protocol** (wildcard). - Better: most Clojure-spirit; user could pass their own Writer; subsumes - `with-out-str`. Breaks: over-reach (excessive-skeleton smell) — D-370's driver - is "let ordered customise set printing"; a bindable-`*out*`-as-writer-Var is - D-238's deferred slice; introduces a protocol nobody on the critical path - implements; pre-empts a structural decision F-003 assigns to its owner. -- **Ruling A → A2.** Already the F-004-declared finished form. A1's java-class - naming is AD-003 theater buying nothing on the thread-back/`.write` call - pattern; F-009 has no neutral impl for it to wrap. A3 is the right GENERAL - `*out*` shape but out of scope + collides with deferred D-238 (excessive - skeleton). Cycle size is NOT why A1 loses (F-002) — A1 loses on honesty + - wrapper-around-nothing. The lifetime invariant is a Consequence, not a downgrade - reason. Follow-on (debt, F-003): when D-238's bindable-`*out*` lands, the A2 - handle is exactly the value type `*out*` binds to — A2 is the down-payment on - A3, shrinking that rewrite. - -**CHOICE B — per-element recursion gating** (clj fact: clj's collection printers -call `print-method` on EVERY element, so an override on a nested element DOES fire -— the common case for a collection-like custom type): -- **B1 — consult only for typed_instance; natives always native.** Better: - cheapest; zero native consult. Breaks: parity hole that matters — a user can - override a NATIVE type's print, and even a custom-type override fails to fire - when nested inside a native collection (the pure `printValue` has no consult). - Fails the flatland.ordered nested-in-larger-structure case. Ad-hoc-pass smell / - F-013 violation (direct probe passes, definitional recursion uncovered). -- **B2 — consult every value when ANY override registered (dirty flag), else pure - native** (recommended). Better: matches clj's per-element recursion exactly when - an override exists, zero cost otherwise (flag gates it). One consult mechanism, - uniform, parity-correct (F-011). Cost/tension: `printValue` is pure `(w, v)` but - the consult needs `rt`/`env`. Two ways: (b-i) an rt/env-carrying collection- - printer variant on the flag-true path only (duplicates the collection walk — - F-011 frowns); (b-ii) `printValue` itself carries nullable `rt`/`env`, null on - the pure path (ONE walk, F-011-clean, touches every printer signature). F-011 - outranks effort ⇒ (b-ii). Perf when flag true: one cached identity probe per - element (bounded). -- **B3 — pr IS print-method always; native default per type** (wildcard). Better: - purest, closest to clj's model. Breaks: bootstrap (the MultiFn must exist before - the first load-time pr — the nullable cached var is removed), perf (every value - pays a multimethod resolve + clojure-fn call, no fast path — the shape the - survey rejects), infinite recursion (the default re-enters print-method on the - same value while consulting children — a fragile invariant the dirty-flag design - sidesteps because flag-false is purely native). -- **Ruling B → B2 (b-ii single-walk).** B1 fails the driver (nested override) + - is the Ad-hoc-pass smell. B3 is model-pure but reintroduces the bootstrap - hazard, kills print-path perf, makes the recursion invariant fragile. B2 is the - only option matching clj per-element recursion AND keeping zero overhead + safe - bootstrap via the dirty flag. Within B2, pick ONE walk ((b-ii) nullable rt/env), - NOT two ((b-i)) — cycle/diff size is not a valid reason for the duplicated form - (Cycle-budget-defer smell). Recursion guard (ADR invariant): the native default - renders via `printValue` DIRECTLY, bypassing the consult on the same value; only - children re-enter. - -**Leading entry — F-NNN:** No amendment required. A2 + B2 sit inside the envelope -and are in two places the DECLARED finished form: A2 is F-004-declared -(host_instance B13); A2 + B2 honor AD-003 (no java.io.Writer class), F-009 -(neutral impl), F-011 (one commonized per-element consult, parity-verified). The -one finding surfaced (not a halt): A1's `java.io.Writer`-as-class would lean -AGAINST AD-003 — a reason to reject A1 (done), not to amend AD-003. A future -`(instance? java.io.Writer w)` truth is a separate host-supertype-marker decision -(`host_interfaces.yaml`, ADR-0102) attachable to the A2 handle's descriptor -without changing A2. - -**Net:** A2 (host_instance writer handle) + B2(b-ii) (dirty-flag per-element -consult, single walk), native default bypasses the consult on the same value, the -writer handle single-print-scoped with a defended stale-write invariant. diff --git a/.dev/decisions/0128_instance_of_fn_class_value_surface.md b/.dev/decisions/0128_instance_of_fn_class_value_surface.md deleted file mode 100644 index 3c57f1700..000000000 --- a/.dev/decisions/0128_instance_of_fn_class_value_surface.md +++ /dev/null @@ -1,227 +0,0 @@ -# ADR-0128 — `instance?` is a fn over a class VALUE; complete the membership oracle - -- Status: Proposed → Accepted (2026-06-10) -- Related: ADR-0109 (host-class-value resolution — the partial this completes), - ADR-0116 (interface_membership SSOT), ADR-0059 / AD-003 (no-JVM simple class - name), ADR-0102 / F-013 (host-interface closed-set), F-002 / F-011 (finished-form - + commonization), D-373 (the driving debt), D-293 (the classDescriptor unify this - folds in). -- Drives the discharge of D-373; folds in the classDescriptor-rename half of D-293. - -## Context - -`instance?` was a cljw MACRO (`expandInstanceQ`, macro_transforms.zig): it -auto-quoted the class symbol, `(instance? String x)` → `(__instance? (quote -String) x)`, so an unresolvable interface marker (`clojure.lang.IPersistentMap`) -worked as the literal class arg without the symbol needing to evaluate. But a -macro cannot be passed as a higher-order fn value, so `(condp instance? obj -Map$Entry …)` (flatland.ordered.map:59) evaluated `instance?` to nil → -"Cannot call value of type nil". In real Clojure `instance?` is a FN taking a -class VALUE (a class symbol evaluates to a Class). This blocked loading -flatland.ordered and any library using higher-order `instance?` (condp / map / -partial / apply — common). Confirmed red on a ReleaseSafe binary 2026-06-10: - -- `(condp instance? "x" String :s :o)` → `Cannot call value of type 'nil'` -- `(map (partial instance? String) ["a" 1])` → same -- `(prn IPersistentVector)` → `Unable to resolve symbol: 'IPersistentVector'` -- `(instance? String "x")` → `true` (direct macro path worked) - -The finished form (user-directed 2026-06-10, **NOT** a workaround): make -`instance?` a real fn. For that, every class symbol a library passes must -evaluate to a class VALUE — which means completing the class-symbol-as-value -surface ADR-0109 left partial (native / exception / opaque / Object / Number / -IFn landed; the `clojure.lang.*` interface markers — IPersistentMap / ISeq / -ILookup / Indexed / … — did NOT, so `(prn IPersistentVector)` was a NameError). - -## Decision (Alt 2′ — complete the membership oracle, eliminate the special-cases) - -Five parts: - -1. **`instance?` becomes a fn.** `core.clj`: `(def instance? (fn* [c x] - (rt/-instance-of? c x)))`; drop `expandInstanceQ` + its macro registration. - A new primitive `-instance-of?` takes a class VALUE (`.type_descriptor`), - extracts its `fqcn`, and returns `class_name.isInstance(x, fqcn)`. Because the - class arg now evaluates, higher-order use (condp / map / partial / apply) works - automatically — same as `clj`. - -2. **Complete `class_name.isInstance` so it needs NO special-cases.** The macro's - runtime half (`instanceQPrim`) special-cased opaque→false / Object→non-nil / - Number→`number?` *before* calling `isInstance`. Those special-cases dissolve - into the membership SSOT instead of being re-stated at the new primitive: - - **opaque** (Integer / java.math.BigInteger): already returns false naturally - — an opaque name is in no match table, so `isInstance` falls through to false - (clj-faithful: a cljw int IS a Long, not an Integer). - - **Number**: widen `interface_membership.NUMBER_TAGS` from `{integer, float}` - to `{integer, float, big_int, ratio, big_decimal}` so it equals `number?`. - (Verified safe: `number?` already uses that set; nothing extend-protocol- - targets `Number`.) - - **Object**: one line in `isInstance` — `if (isUniversalClass(simple)) return - v.tag() != .nil;` (clj: every non-nil value is an Object; nil is not). - `-instance-of?` is then JUST `isInstance(x, fqcn)` — no taxonomy branch. This is - strictly cleaner than centralising the special-cases (the DA's Alt 2 `class_role` - enum, below): the deepest F-011 reading removes the taxonomy rather than - relocating it. - -3. **One `class_name`-recognised analyzer arm** replaces the scattered - Object / Number / IFn / opaque / exception / host_inert arms (analyzer.zig - ~618-678). A single `classValueKeyFor(name)` helper returns the descriptor key - for any recognised non-native class (exception / opaque / Object / interface - marker / host_inert / stream), routing them all through ONE - `classDescriptor(key)` call. `nativeTagFor → nativeDescriptor` (arm 1, keyed by - tag for `(class x)` identity) and `rt.types.get` (registered Java surfaces) - stay distinct descriptor sources, as before. The previously-missing interface - markers now resolve as class values (so `(prn IPersistentVector)` and - `(condp instance? x ISeq …)` work — definition-derived coverage per F-013, not - a per-library add). - -4. **`exceptionDescriptor` → `classDescriptor` (D-293 unify).** The function was - already generic (mints a name-keyed `kind=.native` descriptor; "exception" was a - historical misnomer used for opaque / Object / Number / IFn / host_inert too). - Rename it + its `exception_descriptors` cache field across the ~16 call sites - (runtime.zig / class_of.zig / analyzer.zig / dispatch.zig) so the name is honest - (F-011 commonization). Pure rename, behaviour-preserving. - -5. **`Map$Entry` resolves.** Add `java.util.Map$Entry` / `Map$Entry` → `MapEntry` - to `class_name.FQCN_MAP`. `MapEntry` is already a `NATIVE_ENTRY` (tag - `.map_entry`), so the symbol resolves to the native descriptor and - `(instance? Map$Entry (first {:a 1}))` is true — clj's MapEntry implements - java.util.Map$Entry. - -`prefer-method` (the trivial sibling — a needless macro that was a pure -pass-through) was already converted to a fn in 869fa3db. - -## Why Alt 2′ over the DA's Alt 2 - -The DA fork (verbatim below) recommended Alt 2: add a `class_role` enum field to -`TypeDescriptor`, computed once at mint, so `-instance-of?` / `classIsaPrim` / -the analyzer `switch` on role instead of re-deriving opaque/Object/Number from the -name string at three sites. The DA's F-011 diagnosis is correct — the -re-derivation IS a scatter. But its fix relocates the taxonomy onto the -descriptor; the cleaner fix **eliminates** it: once `isInstance` is a complete -membership oracle (part 2), `-instance-of?` has no taxonomy branch at all, so -there is nothing to centralise. A `class_role` enum would then be a second -classification axis to keep in sync with the name (the SSOT) — a new way to -introduce a bug, for no remaining caller. `classIsaPrim` keeps its -`isUniversalClass`/`isNumberClass`/`isIFnClass` reads, but that is the class-vs-class -`isa?` *hierarchy* question (legitimately distinct from value membership), and -those are shared predicate calls, not a duplicated taxonomy. Per F-002, the -choice is on finished-form cleanliness, not diff size; Alt 2′ is both cleaner and -smaller, so there is no cycle-budget tension to resolve. - -## Consequences - -- Higher-order `instance?` works (condp / map / partial / apply) — unblocks - flatland.ordered and the 15+ data-structure libs that follow it. -- `(prn IPersistentVector)` etc. resolve as class values (definition-derived). -- `(instance? Number 1N)` is now true via the membership table (was true only via - the special-case); `isInstance("Number")` == `number?` — one source. -- Unknown class in `instance?` now errors at ANALYSIS (`symbol_unresolved`) rather - than the macro's runtime `class_name_unknown` — closer to clj's compile-time - unresolved-class error. Existing e2e/corpus expectations updated to match. -- The classDescriptor rename touches ~16 sites; behaviour-preserving. - -## Verification (F-011 clj-oracle; D-177 anti-false-discharge) - -- Re-run ALL `instance?` corpus/e2e (instance_interfaces / instance_iterable / - isa_class_hierarchy / exception_class / class_as_value / opaque_host_class) - against the `clj` oracle. -- Add a clj-diff corpus line for higher-order `instance?` - (`(condp instance? obj …)`, `(map (partial instance? String) xs)`, - `(filter (partial instance? Number) xs)`) per clj_diff_sweep.md Discipline 1 — - the discharge text must not claim higher-order works without a corpus probe. -- `(class x)` / AD-003 simple-name unchanged; `(extend-type Opaque P …)` stays a - load-only no-op. -- Verify the FULL flatland.ordered.map chain past Map$Entry (further blockers may - follow — they are separate units). - -## Alternatives considered (Devil's-advocate, fresh context — verbatim) - -> Active F-NNN: F-002 (finished-form > diff), F-011 (commonization + clj -> equivalence), F-013 (definition-derived, no per-lib allowlist), ADR-0059/AD-003 -> (no-JVM simple name), F-005 (numeric collapse). LOC/diff size is NOT a constraint. - -> # Devil's-advocate review — D-373 (instance? macro → fn) -> -> A structural fact dominates the analysis: **`class_name.isInstance(v, name)` -> already does the entire membership decision from the name string** — Throwable -> hierarchy, native-exact, interface, user-type. The opaque/Object/Number -> special-cases in `instanceQPrim` are *also* keyed off the name string via -> `host_class.isKnownOpaqueClass` / `isUniversalClass` / `isNumberClass`. So the -> runtime already has a single name-keyed membership oracle; the analyzer's 8-arm -> cascade is a *second*, parallel name-keyed oracle that re-derives "which -> descriptor does this name get". The duplication Alt B consolidates is real and -> is an F-011 violation in the current code. -> -> ## Alternative 1 — smallest-diff -> Keep `expandInstanceQ` as the macro for the direct-call case, but additionally -> `(def instance? (fn* [c x] (rt/-instance-of? c x)))` as a fallback Var; add a -> class-value arm for *only* the interface markers flatland touches + Map$Entry. -> **(a) Better:** smaller blast radius; the proven direct-call path untouched. -> **(b) Breaks/risks:** a **macro-and-fn-with-the-same-name** shape — in clj -> `instance?` is *only* a fn. Two code paths (macro auto-quotes; fn evaluates) can -> diverge the instant a user writes `(let [c String] (instance? c x))`. This is -> the Smallest-diff-bias smell + a NEW duplication. The "only the markers flatland -> touches" arm is an **F-013 violation outright** — the library-by-library -> allowlist F-013 prohibits. **(c) F-NNN: violates F-011 and F-013. Rejected on -> invariant grounds, not diff size.** -> -> ## Alternative 2 — finished-form-clean (the `class_role` field) -> Alt B (drop the macro, fn over a class value, one `canonicalClassValueName` SSOT, -> rename exceptionDescriptor→classDescriptor, consult ns.imports, add Map$Entry) -> PLUS a `class_role` enum on TypeDescriptor. The same category taxonomy -> (opaque/universal/number/ifn/native/interface/host_inert/user) is re-derived from -> the name string at THREE sites: `instanceQPrim` (core.zig:77-84), `classIsaPrim` -> (core.zig:109-113), analyzer (analyzer.zig:630-658) — each with its own ordering -> and bug surface. The clean form: give `TypeDescriptor` a `class_role` field set -> ONCE at mint from the single `canonicalClassValueName` SSOT; `-instance-of?` -> becomes `switch (td.class_role)` → opaque⇒false / universal⇒non-nil / -> number⇒numeric? / else⇒`isInstance(x, fqcn)`; no name re-matching. `TypeKind` -> (type_descriptor.zig:33) already exists but is about *origin* (native/deftype/ -> reify), not *class-membership role* — `class_role` is orthogonal and additive. -> **(a) Better than Alt B:** Alt B consolidates the analyzer's 8 arms but leaves -> `instanceQPrim`/`classIsaPrim` still re-deriving opaque/Object/Number from the -> name — the half-swept-area smell (clj_diff_sweep.md Discipline 2). Alt 2 closes -> all three: the name reverts to identity/label, the role is computed once and -> consumed by an exhaustive `switch` (compiler-checked). **(b) Risks:** larger diff -> (every mint site sets the field); a wrong role at a mint site → wrong instance? -> (mitigated by exhaustive switch + clj-diff corpus). **(c) F-NNN:** fully -> compliant; the strongest F-011 reading. **Verdict on name-vs-scatter:** the name -> is a legitimate SSOT *for identity* (`(= (class x) Long)` must compare names) but -> an illegitimate scatter *for role decisions*. Alt 2 splits the two: name = -> identity, class_role = decision. -> -> ## Alternative 3 — wildcard (membership predicate fn-pointer per descriptor) -> Each class-value descriptor carries `membership: *const fn(Value) bool`; -> `-instance-of?` is `return td.membership(x)`; name-string matching disappears from -> the hot path. **(a) Better:** maximal devirtualization (a latent PERF win); the -> closest cljw analogue to a JVM Class's `isInstance`. **(b) Breaks/risks:** fn -> pointers in the descriptor literal are exactly the cost type_descriptor.zig:54-66 -> documents — `method_table`'s `Value.initBuiltinFn(&fn)` "is not comptime on Mac, -> forcing an init-time alloc + deinit free"; a `membership` fn pointer drags every -> `classDescriptor` out of the comptime-const path the named-descriptor cache -> enjoys. It also **scatters** the membership logic into N closures — the inverse of -> F-011. Adjacent to the `pub var` vtable prohibition (ROADMAP §13). **(c) F-NNN:** -> compliant in letter but weaker on F-011 than Alt 2 (N closures vs one switch) and -> fights the settled comptime-const descriptor discipline. -> -> ## RECOMMENDATION: Alternative 2 (the `class_role` field). -> Alt B is correct + clean for the analyzer but incomplete — it commonizes the 8 -> arms while leaving the identical taxonomy re-derived at `instanceQPrim` and -> `classIsaPrim` (the half-swept-area pattern). F-011 points past Alt B to Alt 2: -> compute `class_role` once, consume by exhaustive switch. Per F-002 the larger -> diff is NOT a reason to prefer Alt B — that downgrade would be the Cycle-budget- -> defer smell. Alt 1 rejected on F-011 + F-013; Alt 3 weaker-than-Alt-2 on F-011. -> **Caveat for the ADR regardless of choice:** add a clj-diff corpus line for -> higher-order `instance?` per clj_diff_sweep.md Discipline 1 — the discharge text -> must not claim higher-order works without a corpus probe (the D-177 lesson). - -### Main-loop disposition of the DA recommendation - -The main loop adopts **Alt 2′** — Alt 2's analyzer-arm consolidation + ns/SSOT -unification + rename, but it **completes the `isInstance` membership oracle so the -taxonomy is eliminated, not relocated onto a `class_role` enum**. The DA's -half-swept-area diagnosis of Alt B is the reason Alt 2′ exists; Alt 2′ honours it -more completely than the DA's own Alt 2 (zero remaining taxonomy branch in -`-instance-of?` vs a `switch` on a second axis). `classIsaPrim` keeps its -predicate reads (the `isa?` hierarchy is a distinct question). The DA's caveat -(higher-order corpus line) is adopted verbatim in § Verification. diff --git a/.dev/decisions/0129_deftype_hasheq_equiv_hamt_key.md b/.dev/decisions/0129_deftype_hasheq_equiv_hamt_key.md deleted file mode 100644 index ce786ab18..000000000 --- a/.dev/decisions/0129_deftype_hasheq_equiv_hamt_key.md +++ /dev/null @@ -1,320 +0,0 @@ -# ADR-0129 — deftype/reify `hasheq` + `equiv` participate in HAMT key bucketing & comparison (D-377 facet 2) - -- Status: Proposed → Accepted (2026-06-10) -- Related: D-377 (the driving debt — facets 1 + `=`/facet 3 landed; this is - facet 2), ADR-0127 (the `active_consult` thread-local precedent this mirrors), - ADR-0119 (`treeWalkCall` = the single both-backend call choke point this arms), - ADR-0110 (the `valueHash` interned-cache arms this completes), D-280d1/d5 (the - `(hash x)` primitive's inline hasheq dispatch this commonizes), D-151 (the - `keyEq` residual comment this closes), F-002 / F-011 / F-013 (finished-form + - commonization + definition-derived coverage), F-005. -- Drives D-377 facet 2 toward discharge; opens a deferred perf row (lazy - hasheq cache). - -## Context - -cljw maps/sets use a HAMT. Key **bucketing-hash** and within-bucket -**key-equality** are computed by two rt-FREE functions in `equal.zig`: -`valueHash(v) u32` (bucketing) and `keyEqValue(a, b) bool` (compare). "rt-free" -= no `*Runtime`/`*Env`, so neither can dispatch a user `deftype`/`reify` -instance's `Object/hasheq` or `Object/equiv` method (dispatch needs the -per-Runtime vtable + an Env). For a non-record deftype/reify `valueHash` -returns the identity bit-hash and `keyEqValue` compares identity. - -Measured on a ReleaseSafe binary (2026-06-10): - -- `(hash (->Box 1))` with custom `hasheq`→12345 ⟹ **12345** — the `(hash x)` - PRIMITIVE (`core.hashFn`) already dispatches `Object/hasheq`→`hashCode` for - typed/reified instances (D-280d1/d5). So the top-level `(hash deftype)` was - NOT the gap. -- `#{(->Box 1) (->Box 2)}` (hasheq const 7, equiv true) ⟹ **2** (clj 1 — no - dedup; the SET key path uses neither the deftype's hash nor its equiv). -- `(get {(->Box 1) :a} (->Box 9))` (hasheq 7, equiv true) ⟹ **nil** (clj `:a` — - the map-key path dispatches neither hash nor equiv). - -So the real gap: a `deftype`/`reify` with a custom `hasheq`+`equiv` used as a -**map key / set element** does not bucket or dedup against an `=`-equal value, -breaking the `(= a b) ⟹ (= (hash a) (hash b))` contract for mixed -deftype/native keys & elements. Real driver: flatland.ordered.map LOADS and -`(= ordered-map native)` is true (the `=`/facet-3 `instanceEquiv` landed last -session), but `#{om nm}` does not dedup and `(contains? #{om} nm)` is false. - -User directive (2026-06-10): **take it あるべき論 — wide blast radius is OK, -but NO ad-hoc; a proper fix of cljw's hash architecture.** F-013 forbids a -"just make flatland pass" patch; the fix must be definition-derived and raise -the system one level (any deftype with `hasheq`/`equiv` works as a key). - -### Architecture facts (verified this session) - -1. `dispatch.current_env: ?*Env` is a thread-local whose doc says "The Env - currently being evaluated. Set on call entry, cleared on exit … needed in - low-level callbacks where env isn't otherwise in scope" — but it is set - ONLY in a unit test, NEVER in production. It is dormant, half-built code for - exactly this purpose. `Env.rt` reaches the Runtime. -2. `driver.evalForm` is the single per-top-level-form eval entry (both - backends). `tree_walk.treeWalkCall` is the single value-producing call - choke point for BOTH backends (`evalCall`→`vt.callFn`; VM `op_call`→ - `vt.callFn`; ADR-0119). Arming `current_env` at these two sites makes it set - during ALL evaluation (the top-level-form window + every nested call). -3. ADR-0127 precedent: native-collection printing consults a deftype's - `print-method` via a dedicated armed thread-local (`print.active_consult`, - armed at the print boundary, gated on an override dirty flag). The hash/equiv - problem is the same shape (a Layer-0 rt-free routine must dispatch a user - method mid-evaluation) — but its arming locus is eval-time, not print-time - (print runs at the REPL boundary AFTER eval returns), so the two contexts - stay distinct. -4. `map.keyEq` (map.zig:134) is the single local wrapper every within-bucket / - array-map compare delegates to; its doc already names this residual - ("Collection / ratio / big_int keys stay identity-compared … the recursive - `=` … needs `rt`, which the ~68 map call sites lack"). The bucketing-hash - sites inline `equal.valueHash(k)`. - -## Decision - -**Revive `current_env` as the ambient eval-env, and add Layer-0 rt-aware -consult wrappers that the HAMT key sites use; commonize the `(hash x)` -primitive onto the shared dispatch core.** - -1. **Arm `dispatch.current_env`** (save→set→`defer` restore) at - `driver.evalForm` AND `tree_walk.treeWalkCall`. This completes the TL's - documented-but-unwired intent; `current_env` is now the Env currently being - evaluated, available to Layer-0 callbacks. Null outside evaluation - (bootstrap / host-init) ⇒ consult falls to the rt-free path (safe — those - paths never build custom-hash deftype-keyed maps). - -2. **Shared dispatch core in `equal.zig`**: `hashDispatch(rt, env, v) u32` = - non-record typed/reified instance with an `Object/hasheq` impl → dispatch it - (then `hashCode`), truncate to u32; else `valueHash(v)`. The `(hash x)` - primitive (`core.hashFn`) is refactored to call this (F-011 — it currently - inlines the identical dispatch). The equiv core already exists as - `instanceEquiv` (left-operand asymmetric, clj-exact — ADR/D-377 facet 3). - -3. **Layer-0 consult wrappers** (read `current_env` → its `.rt`): - `equal.hashConsult(v) u32` = armed + custom-hash deftype → `hashDispatch`; - else `valueHash(v)`. `equal.eqConsult(a, b) bool` = armed → `instanceEquiv` - consult ahead of `keyEqValue`; else `keyEqValue(a, b)`. - -4. **Wire the HAMT**: a new `map.keyHash(k)` wrapper → `equal.hashConsult`, - used at the bucketing-hash sites (hamtGet/Contains/Assoc/Dissoc + the - assoc-split / array-map→hamt rehash of EXISTING keys); `map.keyEq` → - `equal.eqConsult`. set.zig delegates to map, so it is covered. The map's own - recursive content-hash sites (`contentHash` / `entryHash`) stay on - `valueHash` (the nested-deftype-in-a-collection-key residual, shared with the - existing `(hash x)` primitive residual). - -5. **Defer the perf cache** (a lazy per-instance `hasheq` cache so a collection - deftype does not recompute its O(size) content-hash on every bucket op). It - is a PERFORMANCE optimization, not a correctness requirement — correctness - here comes from arming coverage (every deftype-key hash/compare runs under - `evalForm`/`treeWalkCall`). Tracked as a new debt/optimization row; landed - later under F-010's perf window. Eager-at-construction caching was rejected - precisely because it makes building an N-entry ordered-map O(N²). - -### Residuals (documented, not silent) - -- **Nested deftype inside a collection that is itself a key** hashes rt-free - (the recursive `valueHash`/`contentHash` walk has no `current_env` hook) — - shared with the existing `(hash x)` primitive residual; the top-level - deftype key is the common case. -- **equiv asymmetry**: `eqConsult` consults the left/stored operand's `equiv` - (matching cljw `=` / clj `Util.equiv`). Whether `#{om nm}` and `#{nm om}` - both dedup is decided against the live `clj` oracle during implementation - (F-011), and any accepted asymmetry is pinned as an AD-NNN. - -### Why this shape (vs. the alternatives below) - -It reaches behavioural equivalence with `clj` for the verification targets -(`#{om nm}`→1, `(contains? #{om} nm)`→true) by completing a mechanism the -codebase already half-built (`current_env`) rather than minting a third -thread-local; it commonizes the `(hash x)` dispatch (F-011); and it keeps the -correctness/perf concerns separate (arming for correctness, a deferred cache -for perf) so the landed diff is the minimum that is *correct and clean*, not a -perf-coupled rewrite. It does not touch the working print path (ADR-0127). - -## Consequences - -- `current_env` becomes load-bearing: a future backend/eval entry that produces - values must arm it (or the F-012 dual-backend oracle catches the asymmetry). - Anchored with a comment at both arm sites. -- A per-call thread-local pointer write at `treeWalkCall` (+ per-form at - `evalForm`). Negligible; F-010 defers optimization. The deferred lazy cache - removes the bigger O(size)-per-bucket-op cost when it lands. -- `equal.zig` gains `hashDispatch` / `hashConsult` / `eqConsult`; `map.zig` - gains `keyHash` and routes `keyEq` to `eqConsult`. `core.hashFn` shrinks - (shares `hashDispatch`). - -## Affected files - -- `src/runtime/dispatch.zig` — `current_env` doc updated to "live". -- `src/eval/driver.zig` (`evalForm`) + `src/eval/backend/tree_walk.zig` - (`treeWalkCall`) — arm/restore `current_env`. -- `src/runtime/equal.zig` — `hashDispatch` / `hashConsult` / `eqConsult`. -- `src/runtime/collection/map.zig` — `keyHash` wrapper; bucketing sites → - `keyHash`; `keyEq` → `eqConsult`. -- `src/lang/primitive/core.zig` — `hashFn` shares `hashDispatch`. -- `test/e2e/phase14_deftype_key_hash.sh` (new) + a unit/diff case. -- `.dev/debt.yaml` (D-377 facet 2 status; new deferred perf row), handover. - -## Alternatives considered - -The mandatory Devil's-advocate fork (general-purpose, fresh context, briefed -with the F-NNN envelope) returned the following verbatim: - -> ### Leading finding on F-NNN -> None of the three alternatives below requires violating an F-NNN. All reach -> behavioural equivalence with `clj` for the verification targets (`#{om nm}` → -> 1, `(contains? #{om} nm)` → true) within F-002/F-011/F-013. The proposed -> design and all three alternatives are F-compliant; the choice is purely about -> which reaches the cleanest finished form. -> -> ### Alternative 1 — SMALLEST-DIFF: dedicated thread-local armed at `treeWalkCall`, no `current_env` revival -> Mirror ADR-0127's `active_consult` exactly. Add `equal.hash_consult: ?struct -> { rt, env }` (a thread-local local to `equal.zig`), armed/restored at the -> single `treeWalkCall` entry — and only there, because vm.zig:418 confirms the -> VM's value-producing call already funnels through `vt.callFn = -> &tree_walk.treeWalkCall`. Gate the arming on a dirty flag ("any deftype with a -> non-default `Object/hasheq`-or-`equiv` impl is registered globally"), exactly -> as `print.zig` gates on a registered non-default `print-method`. -> `hashConsult`/`eqConsult` read this TL; `valueHash`/`keyEqValue` stay rt-free -> fallbacks. -> - Better than proposed: Does NOT revive the dormant `current_env` as a -> general-purpose mechanism — `current_env` is a tempting but unbounded -> surface ("any low-level callback can now reach rt/env"), and reviving it -> invites future code to depend on it implicitly. A purpose-named -> `equal.hash_consult` is the narrowest correct surface and directly -> precedented by ADR-0127. The dirty-flag gate means the common/bootstrap path -> pays zero per-call TL write — proposed design arms `current_env` -> unconditionally at every `treeWalkCall`, a per-call write on the hottest -> path in the interpreter, which F-010 tolerates but is gratuitous when a -> dirty flag makes it free. -> - Breaks / costs: It is a second consult thread-local with the same shape as -> `print.zig`'s — two near-identical armed-TL mechanisms is a DRY smell -> (F-011) the proposed design partly avoids by reusing one `current_env`. The -> honest finished form would extract a shared "armed rt/env consult" helper -> (see Alt 2). Also, arming only at `treeWalkCall` is correct ONLY while the -> VM has no independent value-producing call path; if a future Phase gives the -> VM its own inline `apply` that bypasses `vt.callFn`, the arming site silently -> goes stale (the F-012 dual-backend oracle catches the resulting asymmetry, -> but as a gate failure, not by construction). -> -> ### Alternative 2 — FINISHED-FORM-CLEAN: one unified "evaluation consult context" TL, shared by print + hash + equiv (F-011 commonization) -> The finished form recognises that print-method consult (ADR-0127), hash -> consult, and equiv consult are the same problem: a Layer-0 rt-free routine -> that, mid-evaluation, must dispatch a user method on a deftype and therefore -> needs the ambient `{rt, env}`. Introduce ONE canonical mechanism — -> `runtime/eval_consult.zig` (or fold into `dispatch.zig`) — exposing `armed: -> ?struct { rt, env }`, `arm(rt, env) → token` / `disarm(token)`, and a single -> per-feature dirty flag set. Arm it once at `treeWalkCall`. `print.zig`'s -> `active_consult`, `equal.hashConsult`, and `equal.eqConsult` all read this ONE -> context. Retire `dispatch.current_env` (the dormant half-built version of -> exactly this) and retire `print.active_consult` (its feature-specific -> predecessor), folding both into the unified surface. The `(hash x)` -> primitive's inline hasheq/hashCode dispatch also routes through `hashConsult`. -> - Better than proposed AND Alt 1: This is the F-011 finished form. Today the -> codebase will end up with `current_env` (dormant), `print.active_consult` -> (live), and a new hash/equiv TL — three thread-locals expressing one -> concept, drifting independently. Unifying them is the "raise the system one -> level" outcome F-013 demands: the next Layer-0-needs-rt consumer (a future -> user-defined `compare` for sorted-map keys, or `toString` consult) plugs -> into the existing armed context for free instead of minting TL #4. It also -> resolves the proposed design's ambiguity about where to arm by making "the -> single value-producing call entry" the canonical, documented, one-and-only -> arm site. -> - Breaks / costs: Widest blast radius — it rewrites `print.zig`'s consult -> wiring (a working, gated mechanism) and deletes `current_env`, touching the -> print path's gate-covered behaviour. Per F-002 this rework is a feature, not -> a cost, and per the user's 2026-06-10 directive it is squarely in scope. The -> genuine risk is correctness regression in printing while refactoring a path -> the proposed design leaves untouched — must be driven by the existing print -> e2e/diff corpus staying green. The arming-site staleness hazard from Alt 1 -> remains (single arm site assumes VM has no independent value-producing -> call), but now it is centralized and documented in one place with a -> `GC-ROOT`-style anchor comment. -> -> ### Alternative 3 — WILDCARD: cache the deftype instance's hasheq at construction (rt-free hash), keep equiv dispatch via the armed TL -> Attack the asymmetry the consistency question raises. Hash can be made rt-free -> by caching: when a deftype/reify instance with a custom `Object/hasheq` is -> constructed (construction already runs under evaluation, so rt/env are in -> scope), dispatch hasheq once and store the resulting `u32` in a `hasheq_cache` -> slot on `TypedInstance`/`ReifiedInstance`. Then `valueHash`'s `.typed_instance` -> arm reads `inst.hasheq_cache` with NO rt/env — staying genuinely rt-free, -> exactly as the existing `.symbol`/`.keyword`/`.big_int` arms cache or derive -> their hash without dispatch. Equiv cannot be cached (equality is binary, -> between two values), so the within-bucket key compare still needs the armed-TL -> `eqConsult` — but equiv is consulted far less often than hash (only on a hash -> collision within a bucket). -> - Better than proposed: Directly eliminates the consistency hazard — -> "inserted while armed (rt-aware hash) but looked up while unarmed (rt-free -> hash) → mismatch → key lost." With a construction-time cache, an instance's -> hash is the SAME value forever regardless of whether any TL is armed at -> insert vs lookup. In the proposed/Alt-1/Alt-2 designs, a deftype key whose -> hasheq is consulted at `assoc` time but NOT at a later `get` (because that -> `get` ran outside `treeWalkCall` — e.g. a host-internal `map.get` during GC -> trace, REPL result rendering, or a native reduce) buckets inconsistently and -> silently loses the key. Caching makes the hash a property of the value, -> which is what `clj`'s `IHashEq` contract semantically is (`hasheq()` is -> idempotent per instance). It also keeps `valueHash` honestly rt-free. -> - Breaks / costs: (1) Adds a field to every deftype/reify instance (memory + a -> construction-time dispatch even for instances never used as keys — though -> gated on "has custom hasheq impl"). (2) Does not solve equiv — the -> within-bucket compare STILL needs rt/env, so you still build the armed-TL -> mechanism for `eqConsult`, meaning this is additive to Alt 1/2, not a -> replacement. (3) The equiv side reintroduces a narrower version of the same -> consistency hazard. (4) Caching at construction assumes the fields a custom -> `hasheq` reads are immutable post-construction — true for deftype; a `reify` -> closing over a mutable atom could compute a time-varying hasheq; clj has the -> identical latent issue, so this is an accepted edge, worth an AD-NNN if -> pursued. -> -> ### Overall recommendation (non-binding) -> Alt 2 (unified evaluation-consult TL) is the finished-form-clean choice and -> the one I recommend on F-011/F-013/F-002 grounds, with Alt 3's -> construction-time hasheq cache folded in as the hash-side representation — -> they are complementary, not competing. The strongest single finding is that -> the codebase is heading toward three thread-locals for one concept; the -> proposed design reduces this to two by reusing `current_env`, but -> `current_env` is an unbounded surface whereas the print precedent is a -> purpose-named, dirty-flag-gated surface. On the consistency hazard (the most -> dangerous property): the pure-TL designs all share it for hash — a key can -> bucket differently depending on whether a TL happened to be armed at insert vs -> lookup, and lookups from host-internal paths run unarmed. Alt 3's -> construction-time hasheq cache is the only shape that eliminates it for the -> hash side. So the cleanest finished form is Alt 2's unified TL for equiv + -> Alt 3's cached hasheq for hash, accepting a residual equiv-only consistency -> edge that can be pinned as an AD-NNN. - -### Main-loop disposition of the DA output - -The DA's two strongest findings are **accepted and reshaped**, not rejected: - -1. **Consistency hazard (DA's central point).** The DA assumed the arming - locus is `treeWalkCall`-only, under which host-internal map ops (native - reduce/into/merge, a top-level map literal evaluated before the first call) - run unarmed and a deftype key buckets inconsistently. The proposed design - closes this by arming at **both** `evalForm` (the top-level-form window — - exactly the `(get {deftype-key …} …)` case) **and** `treeWalkCall` (every - nested call). With that coverage, every deftype-key hash/compare reachable - from user code is armed; the only unarmed map ops are bootstrap/host-init, - which never use custom-hash deftype keys. The hazard is closed by **arming - coverage** rather than by a per-instance cache — so the cache is demoted to - what it actually is (a perf optimization, deferred per F-010), avoiding its - O(N²) ordered-map-construction cost and the extra instance field. Caching is - the right *perf* answer later, not a *correctness* prerequisite now. - -2. **Three-TLs-for-one-concept (DA's DRY/F-011 point).** Accepted in - substance: the proposed design **removes the dormant dead `current_env`** by - giving it its real purpose (the DRY win the DA wanted) instead of minting a - new TL. Full Alt-2 unification (deleting `print.active_consult`, folding - print into the same context) is **rejected** because the two contexts have - genuinely different arming loci and lifetimes: `print.active_consult` is - armed at the print boundary and must be live during REPL result rendering, - which runs AFTER eval returns (when `current_env` is restored to null). Print - therefore cannot read `current_env`; forcing a shared surface would couple a - working, gate-covered path (ADR-0127) to this change for no behavioural gain - — the F-002 "excessive rework that does not improve the finished form" line. - The shared **`hashDispatch` core** captures the commonization that is real - (the `(hash x)` primitive + the key-hash site dispatch the same way). - -This disposition keeps the landed change the minimal *correct + clean* form, -defers the perf cache explicitly (no silent omission), and leaves the -print-path untouched. diff --git a/.dev/decisions/0130_arith_intrinsic_opcodes.md b/.dev/decisions/0130_arith_intrinsic_opcodes.md deleted file mode 100644 index 881633b57..000000000 --- a/.dev/decisions/0130_arith_intrinsic_opcodes.md +++ /dev/null @@ -1,216 +0,0 @@ -# ADR-0130: Arithmetic / comparison intrinsic opcodes - -Status: Proposed → Accepted (2026-06-11) -Deciders: autonomous loop (perf-parity campaign, user-directed) -Refs: F-002, F-004, F-005, F-006, F-011, F-013; ADR-0036 (dual-backend parity); -`.dev/perf_v0_baseline.md`; cw v0 `src/engine/compiler/compiler.zig` F95. - -## Context - -cljw compiles every call — including `(+ a b)` — as a generic `CallNode` → -`op_call ` (compiler.zig:297): load the callee Var, load args, `op_call`, -which at runtime resolves the Var and dispatches the `BuiltinFn`. For hot -arithmetic (`fib`, `tak`, `arith_loop`) this per-operation var-resolve + dispatch -+ arg-slice is the dominant overhead. cw v0's "arithmetic intrinsics" (direct -opcodes for `+ - * < …`) drove `fib_recursive` 502→41 ms (12×); it is the **#1 -lever** of the v0-parity perf campaign (`perf_v0_baseline.md`). v1 currently has -no arithmetic opcodes. - -Key facts that shaped the decision (from a v1 source read + the Devil's-advocate -fork below): - -- **Rebinding safety is already structural.** The analyzer resolves a call's - head through `analyzeSymbol`; a lexical binding (`(let [+ str] …)`) yields a - `.local_ref`, NOT a `.var_ref` to `clojure.core/+`. So the intrinsic gate is - just "callee is `.var_ref` AND its `var_ptr` is the canonical `clojure.core` - arith Var" — unfoolable by `let` shadowing, `:refer :rename`, or a user-ns - `(def + …)` (different Var pointer). -- **The numeric tower already lives in `promote.zig`** (`addPromoting`, - `subPromoting`, `mulPromoting`, each i64 `@addWithOverflow` → `wrapI64`, which - produces inline-i48 / heap-Long / BigInt per F-005). The intrinsic dispatch - calls these EXISTING functions — it does NOT reimplement arithmetic. So there - is no second numeric implementation to drift from the builtin (F-011). - `Value.initInteger` must NEVER be used to box a result (it overflows i48 → - float; the i48..i63 window must become a heap-Long via `wrapI64`). - -## Decision - -Add a small set of **binary intrinsic opcodes** whose VM dispatch calls the -existing `promote.*` / comparison primitive directly, gated by canonical-Var -pointer identity. (Devil's-advocate Alternative 2, the finished-form-clean shape.) - -1. **Opcodes (first cut):** `op_add`, `op_sub`, `op_mul` (→ `promote.addPromoting` - / `subPromoting` / `mulPromoting`) and comparisons `op_lt`, `op_le`, `op_gt`, - `op_ge` (→ the existing numeric comparison primitive). **Deferred:** `/` - (integer `/` → Ratio / divide-by-zero, no fixnum fast path), `=` / `not=` - (must match full `valueEqual` on the non-numeric tail — marginal win, real - risk). They keep using `op_call`. -2. **Shared mechanism (both backends).** The dispatch body is a single - `intrinsic.fastBinary(rt, op, a, b)` that delegates to the `promote.*` / - comparison primitive. The VM `op_add` arm AND the TreeWalk call path both - route an intrinsifiable 2-arg core-arith call through it, so parity is - STRUCTURAL (one code path), not merely diff-tested — the direct F-011 - "commonization outranks effort" answer and the ADR-0036 drift-prevention. -3. **Compile-time recognition + folding** (compiler.zig `compileCall`): when the - callee is a `.var_ref` whose `var_ptr` equals a Runtime-cached canonical core - arith Var AND it is not deopted (point 5), fold by arity — `(+)`→`op_const 0`, - `(+ x)`→`x`, `(+ a b)`→`op_add`, `(+ a b c …)`→left-folded chain of `op_add` - (matches the builtin's left fold, incl. float non-associativity). Every - intrinsic opcode is strictly binary (stack effect −1), so no runtime variadic - loop exists — which also removes the O-005-class UAF risk (no runtime - accumulator held across a promotion alloc). -4. **Var-pointer cache.** Runtime caches the canonical `clojure.core` Var - pointers for the intrinsifiable names at bootstrap (new; rt has none today). -5. **Deopt on redefinition (the one F-011 hole the analyzer does NOT close).** - `(alter-var-root #'clojure.core/+ …)` / a `(def +)` that re-roots the - canonical Var mutates the same pointer → the gate would still fire. A - `core_arith_pristine` flag (or per-Var `intrinsic_safe` bit) is cleared when a - core arith Var is redefed/alter-var-root'd; the compiler gate consults it and - falls through to `op_call`. (Already-compiled chunks that captured an - intrinsic are a non-issue in the common case — bootstrap defines `+` once - before user code; a stricter generation/epoch invalidation is a follow-up if - a test needs it.) -6. **GC (F-006).** The non-overflow integer path allocates nothing → no - safepoint, no rooting (the O-007 precedent). The overflow path allocs inside - `wrapI64`; operands are read into the i64 op before the alloc, and compile-time - variadic folding means no extra intermediate is held across it. - -### Layer - -Recognition + folding: `src/eval/backend/vm/compiler.zig` + a closed-set table -in a new `src/eval/backend/intrinsic.zig` (the F-013 structural defense: a gate -asserts the recognized Var set ⊆ the clojure.core numeric/comparison surface). -Dispatch: arms in `src/eval/backend/vm.zig`. Shared primitive: existing -`src/runtime/numeric/promote.zig` + the comparison primitive. TreeWalk consults -`intrinsic.fastBinary` in its call path. - -## Alternatives considered (Devil's-advocate fork, fresh context) - -**Alt 1 — smallest-diff:** pointer-identity Var gate + i64 fast path in each -opcode arm calling `wrapI64`; 2-arity only; non-fixnum defers to the builtin. -*Better:* most correctness-conservative (fallback re-invokes the same builtin). -*Breaks:* `(+ a b c)` un-intrinsified; the `alter-var-root` hole; TreeWalk gets -no speedup and parity is test-dependent, not structural. - -**Alt 2 — finished-form-clean (CHOSEN):** unified `fastBinary` shared by both -backends + compile-time variadic fold + closed-set Var table. *Better:* -eliminates the dual-backend drift class entirely (one fast path); variadic / -0-1-arity solved at compile time; F-013 closed-set defense; no runtime-variadic -UAF risk. *Breaks/risks:* largest diff (F-002: not a constraint); touches the -TreeWalk oracle path (mitigated — `fastBinary` returns to the builtin for any -non-(fixnum,fixnum) case, so oracle values are provably unchanged); float -left-fold associativity must be locked by a diff case. - -**Alt 3 — wildcard:** skip opcodes; do compare-branch (`op_lt_branch`) + -reduce-over-range fusion (v0 24A.3/24C.7/37.3) targeting `arith_loop`'s 34× hole -directly. *Better:* hits the measured worst regression; on-roadmap (Phase 17 -`super_instruction.zig`). *Breaks:* doesn't deliver the fib/tak recursion win -(recursion ≠ counted loop); bigger seq-realization semantic surface; inverts the -baseline's stated lever order. **Sequenced as the NEXT unit after this ADR, not a -substitute.** - -## Riskiest correctness traps (from the fork; each gets a diff/e2e lock) - -1. `alter-var-root #'clojure.core/+` — the redef hole (point 5 deopt). -2. `/` excluded — integer `/` → Ratio / divide-by-zero arg-precise raise. -3. `=`/`not=` deferred — must match `valueEqual` (`(= 1 1.0)`→false, NaN, AD-001). -4. i48→i64 boxing — use `wrapI64`, never `initInteger`; diff case at - `(+ 140737488355327 1)` (the i48 boundary → heap-Long, not float). -5. GC rooting only on the overflow/promotion path (non-overflow allocates none). -6. Float variadic left-fold associativity — diff case `(+ 0.1 0.2 0.3)`. - -## Consequences - -- fib/tak/arith hot paths skip var-resolve + BuiltinFn dispatch + arg-slice; - expected fib win in the v0 12× class (arith_loop needs the Alt-3 fusion next - for its full drop). -- One new module (`intrinsic.zig`) + a closed-set gate; new opcodes per - ADR-0036 (compile arm + dispatch arm + TreeWalk parity + diff cases, same - commit). Naive form (the builtin) remains the F-011 contract. -- Verified per cycle by a focused quick-bench + the differential oracle + the - six trap diff cases; GC-torture on the dispatch arms (O-005/O-013 discipline). - -## Amendment 1 (2026-06-11) — family generalized; `=` included fixnum-only - -After op_add validated the pipeline (O-014: arith_loop 170→134 ms, fib call-bound), -the family landed generalized to a **closed-set table** (`Runtime.arith_vars: -[8]?*anyopaque` indexed by `intrinsic.ArithOp`): op_sub / op_mul / op_lt / op_le / -op_gt / op_ge, plus **op_eq — reversing the original `=` deferral**. The reversal -is safe because the intrinsic is **fixnum-only**: two inline fixnums compare by -integer equality (unambiguous), and EVERY other operand pair (float, bigint, -string, …) returns `null` from `fastBinaryFixnum` → defers to the builtin `=`, -which keeps full value-equality semantics (`(= 1 1.0)`→false per F-005, NaN, -AD-001). So op_eq cannot diverge on the non-(fixnum,fixnum) tail — the exact -property the original deferral was protecting. `=` is worth including because it -is the loop condition in `arith_loop`/`tak`-shaped code (`(= i n)`), a hot site. -`/` and `not=` stay deferred (ratio/raise; `(not (= …))`). - -Mechanics: `intrinsic.recognize(rt, var_ptr)` scans the cached table (pointer -identity) → the opcode to emit; one grouped VM dispatch arm handles all 8 -(`intrinsic.fromOpcode(instr.opcode)` → ArithOp; fallback uses -`rt.arith_vars[@intFromEnum(op)]`). `alter-var-root` on ANY cached op clears -`core_arith_pristine`. Coverage: diff_test family block (sub/mul/comparisons/= -inline + `(= 1 1.0)` defer + shadow) + phase4_cli op_mul i48-heap e2e. - -## Implementation plan (file-by-file, derived from a v1 source read) - -Ground truth found while writing this ADR (so the implementing cycle is -mechanical): - -- `CallNode { callee: *const Node, args, loc }` (node.zig:266); `VarRef { - var_ptr: *const Var }` (node.zig:115). Let-shadowing already yields a - `.local_ref`, so the gate is `n.callee.* == .var_ref` + pointer identity. -- The numeric tower is `promote.addPromoting/subPromoting/mulPromoting(rt,a,b)` - (promote.zig:255/288/320) — each does i64 `@addWithOverflow` → `wrapI64` - (i48/heap-Long/BigInt per F-005). The builtin `plus` (math.zig:76) calls - `addPromoting` then `catch`-translates errors to a caret-precise - `type_arg_not_number`. **Zone rule (Layer 1 vm/ MUST NOT import Layer 2 - lang/):** the dispatch arm CANNOT call `math.plus`. So the non-fixnum / error - fallback re-dispatches through the cached `+` builtin Value via `vt.callFn` - (identical to what op_call did, minus var-resolution) — full error parity. -- Runtime caches Vars as type-erased `?*anyopaque` populated in - `bootstrap.setupCorePrefix` (runtime.zig pattern: `data_readers_var`). -- op_call dispatch arm shape: vm.zig:400 (pop callee+args, `vt.callFn`, push). - -Steps (one commit, ADR-0036 dual-backend contract): - -1. **`src/runtime/runtime.zig`** — add `arith_intrinsics: ?ArithIntrinsicCache` - holding, per op (+,-,*,<,<=,>,>=): the `*anyopaque` Var ptr + the builtin - `Value` (for the fallback) + a `core_arith_pristine: bool` (true at - bootstrap). Value is Layer-0 so no erasure; Var ptr type-erased like - `data_readers_var`. -2. **`src/lang/bootstrap.zig` setupCorePrefix** — after core is interned, - look up `clojure.core/{+,-,*,<,<=,>,>=}` Vars + their root Values, populate - the cache, set pristine = true. -3. **`src/lang/primitive/core.zig`** — `alterVarRootFn` (1402) + the `def` - root-set path: if the target Var is one of the cached arith Vars, set - `rt.…core_arith_pristine = false` (the one F-011 hole the analyzer doesn't - close). -4. **`src/eval/backend/vm/opcode.zig`** — add `op_add/op_sub/op_mul/op_lt/ - op_le/op_gt/op_ge`; NOT `isPurePush` (they have operands on the stack). -5. **`src/eval/backend/intrinsic.zig`** (new, Layer 1) — `ArithOp` enum + - `fastBinaryFixnum(rt, op, a, b) !?Value` (null unless both `.integer`): - add/sub/mul → `promote.*Promoting`; comparisons → `Value.bool(a.asInteger() - b.asInteger())`. + a compile-time recognizer - `opcodeFor(rt, callee_var_ptr) ?Opcode` (pointer-identity table). Closed-set - gate test (F-013): recognized set ⊆ the 7 names. -6. **`src/eval/backend/vm/compiler.zig` compileCall (297)** — before the - generic path: if `n.callee.* == .var_ref` and `n.args.len == 2` and - `intrinsic.opcodeFor(rt, vp)` hits, compile the 2 args (NOT the callee) + - emit that opcode. Variadic/0-1-arity fold (`(+)`→const 0, `(+ x)`→x, - `(+ a b c)`→left-folded chain) is a follow-up; first cut = 2-arg only, - everything else falls through to op_call. -7. **`src/eval/backend/vm.zig`** — dispatch arm per opcode: pop b,a; if - `rt.pristine and intrinsic.fastBinaryFixnum(...)` non-null → push it; else - `vt.callFn(rt, env, cached_builtin_val, &.{a,b}, instr_loc)`. -8. **`src/eval/backend/tree_walk.zig` treeWalkCall** — same gate: a 2-arg core - arith call routes through `intrinsic.fastBinaryFixnum` first (shared path → - structural parity), else the existing builtin dispatch. -9. **`src/lang/diff_test.zig`** — diff cases: `(+ 1 2)`, `(- 5 3)`, `(* 4 6)`, - `(< 1 2)`, the i48 boundary `(+ 140737488355327 1)` (→ heap-Long not float), - float `(+ 0.1 0.2)`, `(let [+ str] (+ "a" "b"))` (→ "ab", not op_add), - `(reduce + (range 5))`, and a redef-deopt e2e (`alter-var-root` on `+`). -10. **`src/main.zig`** test aggregator — `_ = @import("eval/backend/intrinsic.zig");`. - -First sub-cut may land just `op_add` (steps 1-2,4-9 for `+` only) to validate the -whole pipeline end-to-end, then add `-,*,<,<=,>,>=` in the next commit. diff --git a/.dev/decisions/0131_in_vm_call_frame_stack.md b/.dev/decisions/0131_in_vm_call_frame_stack.md deleted file mode 100644 index e70950910..000000000 --- a/.dev/decisions/0131_in_vm_call_frame_stack.md +++ /dev/null @@ -1,391 +0,0 @@ -# ADR-0131 — In-VM call-frame stack: flatten `op_call` host recursion - -- **Status**: Proposed → Accepted -- **Date**: 2026-06-11 -- **Phase**: §9.2.S perf-parity campaign (cljw-vs-Python) -- **Supersedes / relates**: builds on ADR-0130 (O-015 frame-rooting), - ADR-0091/0095 (GC EvalFrame A1), ADR-0119 (trace), ADR-0129 (consult-env), - ADR-0060/0071 (VM try/catch handler stack), ADR-0036 (dual-backend parity), - ADR-0005 (differential oracle). v0 reference: `ClojureWasm/src/engine/vm/vm.zig`. - -## Context - -The perf-parity campaign (handover; F-002 finished-form wins; not ROI-gated) -needs cljw to beat Python on `fib_recursive` (2.4×) and `nested_update` (3.1×) — -both **call-bound**. After O-014 (arith intrinsics) and O-015 (exact-count frame -rooting), the residue is the call path itself. - -**Two pieces of evidence settle the lever (this cycle):** - -1. **Design 2 measured null.** A monomorphic `op_call` fast path that bypassed - `vt.callFn` + the `treeWalkCall` switch + `callFunction` + `vt.evalChunk` - (the 2 vtable hops + the choke switch), keeping consult-env/trace/bind inline - and calling `eval` directly, moved fib_recursive 57→56 ms / tak 18 ms — - **no measurable win** (10-run ReleaseSafe). It was reverted as an - excessive-skeleton; only the `bindCallFrame` extraction was kept (`ab1959c2`). - -2. **`sample` profile of `(fib 35)`** (ReleaseSafe): 100% of samples are a single - **5-host-frame recursion cycle** that nests a real C frame per Clojure call - (indentation grows unbounded). **No GC / alloc / collect heavy leaf** (fib is - fixnum-only). The pass-through hops Design 2 removed were already - tail-call-collapsed by the optimizer; the irreducible cost is the **non-tail - `eval` re-entry** — op_call must resume after the callee returns, so it cannot - be a tail call, and is a genuine recursive host frame per call. Design 2 kept - that frame; only flattening the recursion removes it. - -The conclusion matches v0's proven design: v0 dispatches a call **in-place on a -flat in-VM frame stack**, staying inside one `run()` loop (fib 16 ms, beats -Python). cw v1's per-call `vm.eval` host re-entry is the structural tax. - -Full survey: `private/notes/9.2.S-flat-frame-survey.md` (+ its EMPIRICAL UPDATE). - -## Decision - -Replace `vm.eval`'s **one-host-frame-per-call** activation model with an -**explicit in-VM call-frame stack** iterated by a single `eval` loop. A -monomorphic bytecode-`.fn_val` `op_call` **pushes a frame and continues the -same loop**; `op_ret` **pops the frame and continues**; only the top-level -boundary returns a host value. Everything not in the fast set (builtins, -multi_fn, protocol_fn, coll/keyword/var-as-IFn, IFn-instance, **TreeWalk-kind -fns**) keeps routing through `vt.callFn → treeWalkCall` — a host call out, -preserving the F-012 dual-backend oracle seam (the mandated DIVERGENCE from v0, -which has no such seam). - -### Operand/locals model — DECIDED: Alt A (preserve v1's split, base-window both) - -**Key correction from the Devil's-advocate fork: v1 ≠ v0 on the activation -record.** v0 unifies locals INTO the operand stack (`locals = stack[base+slot]`, -one array, one root slice). **v1 keeps them SEPARATE** — `locals: []Value` is -caller-owned (a `[256]Value` allocated in `callMethodImpl`) and the operand -`stack` is a distinct `eval`-local (`vm.zig:85-99`). That split is **deliberate -v1 architecture, not incidental**: it is the seam that lets `bindCallFrame` be -the ONE source shared by TreeWalk (which has NO operand stack, only locals) and -the VM (F-011 one-source, just landed `ab1959c2`). - -Therefore the finished form is **Alt A, not v0's unification (Alt B)**: keep the -locals/operand split, base-window BOTH into shared arenas — -`stack[0..global_sp]` (operands) + `locals[0..global_loc_high]` (call-frame -slots), each frame addressing its window via an operand `base` and a `loc_base`. -`bindCallFrame` is reused **as-written** — point it at a sub-window of the shared -`locals` arena instead of a private `[256]Value`. Alt B (full v0 unification) -was REJECTED because it would FORK `bindCallFrame` (VM in-place splice vs -TreeWalk private array), regressing the F-011 one-source property; v1's split is -its finished form. The shared arenas are O(depth) base markers — the per-frame -memory worry the first draft raised is moot, and so is v0's "one slice" claim -(Alt A roots two slices, strictly simpler than today's N-chain, see GC below). - -### Per-frame state (the `CallFrame` struct) - -Each frame owns the state that is `eval`-local today: `ip`, `chunk`, operand -`base` + `loc_base` (windows into the shared arenas), `fs` (its locals window -width), the **handler-stack base** (try/catch), the **consult-env restore** -value (ADR-0129), and the **trace-pushed flag** (ADR-0119). `op_ret` restores -consult-env + pops the trace frame from the frame record, not from a Zig `defer`. - -**No `saved_ns` slot** (DA prerequisite a): unlike v0's D68, **v1 does NOT switch -`current_ns` on a fn call** — `Function.defining_ns` is read ONLY by `calleeFrame` -to build the trace frame, never to rebind ns at runtime (the only `current_ns` -writes are in `in-ns`/`ns`/binding-frame ops). Reserving a per-frame `saved_ns` -would be the Reservation-as-bias smell (rooting a save/restore for state that -never changes per call). It is deliberately absent. - -`loc_stack` (ADR-0118, the parallel `[256]SourceLocation`) is windowed with the -operand stack — same `base`, same `global_sp` — so `op_call`'s -`swapArgSources(loc_stack[…])` reads the right window (DA prerequisite c). - -### GC root reshape (A1 — F-006) - -Today: one `EvalFrame` per `vm.eval`, chained by the host recursion, each rooting -`stack[0..sp]` + `locals` + `constants`. The flat model (Alt A) publishes **ONE -`EvalFrame` per `eval` invocation** rooting the whole shared `stack[0..global_sp]` -+ the whole shared `locals[0..global_loc_high]` + **each active frame's -`chunk.constants`** (a small per-frame list). This is **two big Value slices + a -constants list — strictly simpler than today's N×2-slice chain**, not v0's single -slice (the split is preserved). Every below-`sp` operand slot is written only by -pushes; every locals slot is nil-init'd by `bindCallFrame` (O-015) on frame push; -a frame reads ONLY its own `[base..sp]` / `[loc_base..loc_base+fs]` window, never -a sibling's — so the shared arenas are safe under the existing nil-valid -invariant. `gc_rooting.md §A` row A1 + `root_set.EvalFrame` get a coordinated -edit. - -**The torture lock must ALLOCATE per frame (DA prerequisite d — the sharpest -F-006 point).** fib is fixnum-only (zero allocation — the profile confirms no GC -leaf), so fib under `CLJW_GC_TORTURE` proves NOTHING about rooting. The locking -e2e MUST be a **deep recursion that allocates per frame** (e.g. builds a -list/map per level) so a missed root surfaces as a deterministic UAF. This is -the gate before trust (the O-005 UAF class lives here). - -### Exception unwinding — a `while` loop around the existing catch (not a new subsystem) - -DA correction: this is **cheaper than the first draft billed**. The existing -`eval` catch block (`vm.zig:152-207`) already does handler-stack unwind + -cleanup-vs-catch (ADR-0071) + catalog→exception synth (ADR-0060). Flat: on a -throw with no live handler in the top in-VM frame, **pop the frame** (restore its -consult-env + trace; no ns — see above), decrement `frame_count`, retry the -handler check against the new top frame — a `while` loop around the existing -logic. The `handlers[]` stay one array per `eval` invocation with a per-frame -`handler_base`, so "pop frames until a live handler" is "scan down past each -frame's base." - -**Reentry-boundary handler scoping (DA prerequisite b — the highest parity -risk).** The ~30 reentrant `vt.callFn` callers (reduce / apply / lazy-seq force / -sort comparator / multimethod dispatch-fn / watchers / print-method …) re-enter -via `vt.callFn → treeWalkCall → callMethodImpl → evalChunk → eval`, each starting -a **fresh nested `eval` with its OWN frame array + OWN `handlers[]`**. Because -handler stacks stay **per-`eval`-invocation** (NOT promoted to a process-global -stack), a throw inside a reduce step-fn unwinds only the inner `eval`'s frames and -re-propagates the Zig error to the reduce caller — it CANNOT be caught by a -handler the outer `eval` established. This scopes handlers across reentry for -free (v0 needs `call_target_frame` only because it has nested `executeUntil` -loops; v1's per-invocation handler array avoids that). **Lock with a diff/e2e -case: "throw inside a reduce step-fn caught by an OUTER `try`."** - -### Deep recursion: SIGSEGV → catchable StackOverflow (behaviour improvement) - -Deep cljw recursion currently overflows the host C stack (SIGSEGV, uncatchable). -The flat model grows the in-VM frame array to a cap, then raises a **catchable -`StackOverflow`** — matching v0 (`vm.zig:1611`) and clj (`StackOverflowError`). -A new `error_catalog` Code + a frame cap. A diff/e2e case locks it. - -### The shared arena lives in a per-thread `VmArena` (as landed in 2a) - -The "shared base-windowed arena" cannot be a per-`eval` host-stack array: deep -recursion needs the arena to hold thousands of frames (a fixed host-stack array -big enough would blow the C stack, and nested reentrant `eval`s would multiply -it). v0 resolves this with ONE VM stack reused across calls. cw v1 adopts the -same as a **threadlocal `VmArena`** — but with **inline arrays (static BSS, -demand-paged), NOT a lazy heap alloc**: a never-freed heap arena leaks under the -test `DebugAllocator`, whereas static storage has nothing to free. -`ARENA_SLOTS = 16384` keeps the global watermark `op_top` within `u16`, so -`root_set.EvalFrame.sp` is unchanged (no blast radius to the other GC producers). -Each `eval` borrows from `op_top` (restored on return; nested reentrant evals -stack above); the A1 `EvalFrame` roots `stack[0..op_top]`. - -### Implementation is incremental (green, gated increments) - -1. **2a — operand arena (LANDED `a12fdb09`, O-016)**: the operand stack + its - parallel loc stack moved off the per-`eval` host C stack into the threadlocal - `VmArena`; locals stay caller-passed; one frame per eval (host recursion - retained). The A1 `EvalFrame` points into the arena. Planned behaviour-identical - — turned out a **real ~25% win** (fib 56→41 ms, tak 18→15) from the reused warm - arena vs cold fresh host arrays. The key invariant: during `op_call`'s - `vt.callFn` the deferred `sp` write-back leaves `op_top` at the PRE-step - high-water, so the callee's args stay rooted AND the nested eval borrows above - them — NO publish-before-nest (the initial 4-site plan was wrong, would UAF the - args). Verified under the allocating `frame_local_alloc` torture gate. -2. **2b — the flatten**: a `frames` array of `CallFrame`; `op_call` (callee - `.fn_val` + bytecode method + `!has_rest` + `!chunk.has_handlers` + frame cap) - binds the callee's locals into a NEW `VmArena.local_arena` window via - `bindCallFrame` + pushes a `CallFrame` + continues the loop; `op_ret` pops. - **GC: each `CallFrame` carries its OWN inline `root_set.EvalFrame`** pushed on - the existing chain at flatten / popped at ret — so the collector's existing - chain-walk roots every active frame's locals + constants with NO `root_set` - change (it cannot hold a vm-type `CallFrame` — zone rule). Bounded throw-unwind - (flattened frames are handler-free, so the handler stack is invariant across - them → pop flattened frames then run the base frame's catch). Deep recursion → - catchable `StackOverflow` at the frame cap. Slow cases keep `vt.callFn`. Gate + - allocating-torture + diff + the reduce-throw-outer-try case + deep-recursion + - cross-lang bench. (`op_call` must NOT pop the args before the eval loop binds - them — args stay rooted via `op_top` until `bindCallFrame` copies them.) - Full spec: `private/notes/9.2.S-flat-frame-survey.md § CONVERGED 2b FLATTEN`. - -Each increment is its own revert-friendly commit with an O-NNN row -(`.dev/optimizations.md`) + a clj corpus line (F-011). - -## Consequences - -- **AS-LANDED FINDING (2b, 2026-06-11): the flatten is perf-NEUTRAL.** The - hypothesis that the host-recursion was the fib/tak tax was WRONG. A `sample` - re-profile of `(fib 35)` after 2b shows the 5-host-frame cycle COLLAPSED (the - flatten fires; fib runs in one eval loop) yet fib stays 39-41 ms / tak 15 ms. - Both Design 2 (kept recursion) and 2b (removed it) measured neutral → the real - tax is **per-instruction DISPATCH** (`stepOnce` called per op + its sp/ip - read+write-back + the 3 per-op eval-loop polls; ~0.3 ns/op vs v0 ~0.1). The - next campaign lever is dispatch efficiency (inline `stepOnce`, batch the polls - v0-style every N ops, then computed-goto / superinstructions — D-NNN), NOT the - frame model. See `private/notes/9.2.S-flat-frame-survey.md § 2b LANDED`. -- **Positive (what 2b DOES deliver)**: fixes the latent deep-recursion **SIGSEGV** - — a deep non-tail recursion that crashed at ~1300 host frames now runs to the - 2048 frame cap then raises a bounded `stack_overflow` (deep1500 ✓, deep5000 → - clean error). It is v0's finished form (F-002) and keeps the dual-backend oracle - seam for all non-fast tags. (The `stack_overflow` is uncatchable for now — - `resource_exhausted` kind; a follow-up maps it to a catchable StackOverflowError - for full clj parity.) -- **Negative / risk**: the hottest, most-tested loop is rewritten for a - perf-neutral result. GC rooting (each flattened frame's own `EvalFrame` on the - chain) is UAF-critical — mitigated by the allocating `frame_local_alloc` - `CLJW_GC_TORTURE` gate, the full diff oracle, the reduce-throw-outer-try parity - case, and the deep-recursion repro. Kept (not reverted like Design 2) because it - is a real bug fix + the finished form, not dead weight. -- **Dual-backend (F-012)**: no new opcode (reuses `op_call`/`op_ret`); the - parity surface is "same Value out", covered by the diff corpus + targeted - recursion cases. TreeWalk keeps its host recursion (it has no VM). -- **Zone**: all in `eval/backend/vm.zig` + `runtime/gc/root_set.zig` (Layer 0 - EvalFrame shape). No upward import. - -## Affected files - -- `src/eval/backend/vm.zig` — `eval` loop, frame struct/array, `op_call`, - `op_ret`, handler unwind, error annotation reads top frame. -- `src/runtime/gc/root_set.zig` — `EvalFrame` shape (shared-stack + per-frame - constants list) if the A1 reshape needs it. -- `src/runtime/error/catalog.zig` — `StackOverflow` Code. -- `.dev/gc_rooting.md` §A row A1 — root-shape edit. -- `src/lang/diff_test.zig` + `test/diff/clj_corpus/` — recursion parity cases. -- `.dev/optimizations.md` (O-016/O-017 rows), `.dev/debt.yaml` (if a sub-risk - defers). - -## Alternatives considered - -Devil's-advocate subagent output (fresh context, mandatory at depth ≥ 2 per -CLAUDE.md § ADR-level designs are handled inline), reflected verbatim. Its -findings drove the Decision above: Alt A over Alt B (preserve v1's locals/operand -split = the F-011 `bindCallFrame` seam), drop the phantom `saved_ns`, per-`eval` -handler stacks for free reentry scoping, window `loc_stack`, and the -allocating-torture-not-fib F-006 lock. The recommendation (Alt B) is NOT binding; -the main loop chose Alt A on the explicit judgement the DA flagged — v1's split -is essential v1 architecture, not v0 inheritance, so v1's finished form preserves -it (and keeps `bindCallFrame` single-source, which Alt B would fork). - ---- - -### Reframing: v1 ≠ v0 on the locals/operand split - -v0 unifies locals into the operand stack (`stack[base+slot]`, one array, one root -slice, `shiftStackRight` splice). v1 keeps `locals: []Value` (caller-owned, passed -into `eval`) SEPARATE from the operand `stack` (`eval`-local). The first ADR draft -conflated two decisions: (1) do operand stacks share one base-windowed array? and -(2) do locals merge into it (v0) or stay separate (v1)? The memory argument + the -"one slice" A1 claim only hold for full v0 unification; the reuse of `bindCallFrame` -(targets a private `[256]Value`) points the other way. The ADR must pick one — and -the choice is load-bearing for F-006. - -### Alt A — SMALLEST-DIFF / arguably v1's finished form: flat frames, locals stay per-frame & separate ("two stacks, base-windowed each") - -Keep v1's split: shared operand `stack` (base-windowed) + shared `locals` arena -(frame i at `locals[loc_base_i .. +fs_i]`). `op_call` monomorphic bytecode -`.fn_val`: selectMethod → `bindCallFrame` writes the callee window into shared -`locals` at the new `loc_base` → push `CallFrame{ip, chunk, operand_base, loc_base, -fs, handler_base, consult_restore, trace_pushed}` → continue. `op_ret` pops, drops -operand `sp` to caller window, pushes result. A1: root `stack[0..global_sp]` + -`locals[0..global_loc_high]` + per-frame `constants`. **Better:** reuses -`bindCallFrame` exactly as written; minimal change; smallest F-012 surface (opcodes -keep operand/locals distinct); `op_make_fn`'s `locals[0..slot_base]` snapshot -keeps working. **Breaks:** two base markers/frame; two caps (operand + locals) -with two `StackOverflow`s; not v0's unified form — but the split is *v1's* -finished form (it anchors the dual-backend `bindCallFrame` seam), so this may be -cleaner than v0 for v1. The memory worry is moot (one shared `locals` arena = -O(depth) bases). - -### Alt B — FINISHED-FORM-CLEAN (v0 model): fully unify locals into the operand stack, single root slice - -Delete the separate `locals` param; locals become `stack[base+slot]`. -`bindCallFrame` rewritten to splice in-place on the operand stack (v0 -`shiftStackRight`). `op_load_local slot` → `stack[base+slot]`; `op_make_fn` -snapshots `stack[base..base+slot_base]`. A1 collapses to v0's one slice -`stack[0..global_sp]` + per-frame `constants`; `EvalFrame` loses `locals`. -**Better:** the genuinely clean finished form; the ONLY shape delivering "one root -slice"; F-006 maximally simple (one array, one nil-valid invariant, no separate -locals tail to under-root — the O-005 class). Matches v0's 16ms structure. -**Breaks:** large blast on the SHARED binder — `bindCallFrame` is shared with -TreeWalk (which wants a private locals array, no operand stack). Unifying forks -VM vs TreeWalk binding, breaking the F-011 one-source property just bought -(`ab1959c2`). You'd thread `dest:[]Value, dest_base` through both, but in-place -`shiftStackRight` for rest/closure differs from "memcpy into a fresh frame", so -unification is partial. F-002 says pay it — but the ADR must own that -`bindCallFrame`'s signature changes and re-verify TreeWalk parity. - -### Alt C — WILDCARD: keep Zig-error threading; flatten frame allocation, not control flow - -Keep `op_call` pushing a frame into the same loop (host recursion gone for the -normal path) but keep Zig `error.ThrownValue`/`RecurSignaled` for exceptional -paths. On a throw, the single loop's existing catch (`vm.zig:152-207`) walks the -IN-VM frame array popping frames until one has a live handler — a `while` around -the existing logic, no new mechanism. **Better:** shows "frame-aware unwind" is -mostly already there (the catch block already unwinds one frame's handler stack); -the single-loop design AVOIDS v0's `call_target_frame` cross-bridge handler -leak (`vm.zig:372-373`) because there is one catch site. **Breaks/risk:** the ~30 -reentrant `vt.callFn` callers (reduce/apply/lazy/sort/multimethod/watchers/print) -still re-enter `eval` → a nested host loop with its own frame array, so the -handler-scoping-at-reentry-boundary issue returns (v0's `call_target_frame` -problem). Contribution: the exception part is cheaper than billed; the subtle part -is the reentry boundary, which the ADR was silent on. - -### Verdict (a): shared base-windowed vs per-frame-array — F-006 - -The shared stack is cleaner **only in Alt B's full unification** (one slice, one -invariant). The first draft's rejection of per-frame arrays "on memory grounds" is -a size argument dressed as correctness (a smell); the correctness fact is that -per-frame arrays = today's chained `EvalFrame` model (N slices), which already -works under torture — not unsafe, just N roots. In the draft's AMBIGUOUS half-state -(operands shared, locals separate-but-also-shared) F-006 is actively MORE dangerous -than per-frame arrays: the nil-init invariant spans a shared mutable array where one -frame's stale operands sit below another's base (the O-005 tail-slot class becomes -globally-coupled instead of per-call-isolated). **Recommendation: commit to Alt B's -full unification (one array/slice/invariant) or you get the worst of both; do not -ship the half-state.** Lead the rationale with rooting-simplicity, not memory. - -### Verdict (b): is frame-aware exception unwind necessary? - -Necessary, but the draft OVER-rates difficulty ("on par with the GC reshape"). -It's a `while` loop around the existing catch (Alt C): pop frame, restore -consult-env + trace (NOT ns — v1 doesn't switch ns per call), retry handler check. -Handlers stay one shared array with per-frame `handler_base`. The genuinely-hard -part is the REENTRY BOUNDARY (a throw must not escape a `callFunction` re-entry to -be caught by the outer eval) — naturally solved IF handlers stay -per-`eval`-invocation (NOT a process-global stack). The ADR should (i) downgrade -the difficulty, (ii) reuse the existing catch, (iii) add the reentry-boundary -design + a "throw inside reduce caught by outer try" diff case (the single most -likely parity bug). - -### Prerequisites the ADR ASSUMED but the code does NOT support - -- **(a) MAJOR — no per-call ns switch.** v1 does NOT write `current_ns` on a fn - call (grep: only `in-ns`/`ns`/binding-frame ops do). `Function.defining_ns` is - read only by `calleeFrame` (`tree_walk.zig:1100-1101`) for the trace frame. The - per-frame `saved_ns` slot the draft reserved (copied from v0 D68) is a PHANTOM — - Reservation-as-bias. Drop it (or note the v1/v0 divergence explicitly). -- **(b) MAJOR — ~30 reentrant `vt.callFn` callers + handler scoping.** - reduce/apply/lazy-seq force (`lazy_seq.zig:113`)/sort (`sorted.zig:99`)/ - multimethod (`multimethod.zig:329`)/watchers (`iref.zig:44`,`agent.zig:296`)/ - print (`print.zig:152`). Each re-enters → a fresh nested `eval` + frame array. - State explicitly: handler stacks remain per-`eval`-invocation so reentry scopes - handlers for free; add the reduce-throw-outer-try diff case. -- **(c) MODERATE — `loc_stack` must be windowed too** (`vm.zig:99`, parallel to - operands; `op_call`'s `swapArgSources(loc_stack[sp+1..])` needs the right - window). Name it in increment-1. -- **(d) MODERATE→SHARPEST for F-006 — the torture e2e must ALLOCATE per frame.** - fib is fixnum-only (zero alloc; the profile confirms no GC leaf), so fib under - `CLJW_GC_TORTURE` proves nothing about rooting. The locking e2e MUST be an - allocating deep recursion (build a list/map per level) or torture is green for - the wrong reason. (Good: increment-1 `frame_count==1` already exercises the new - A1 shape under torture before flattening; multi-frame torture coverage arrives - at increment-2.) - -### Summary recommendation (non-binding) - -Direction right, evidence solid. (1) Pick Alt B (full unification, one root slice) -per F-002 — but own that `bindCallFrame`'s signature changes and re-verify F-011. -Do NOT ship the ambiguous half-state. (2) Drop the phantom `saved_ns`. (3) Add the -reentry-boundary handler-scoping design + the reduce-throw-outer-try diff case -(highest parity risk). (4) Downgrade the exception-unwind difficulty. (5) Make the -torture e2e an ALLOCATING deep recursion, not fib. Closing judgement the main loop -must make consciously: whether v1's locals/operand split is essential architecture -(→ Alt A is v1's finished form) or incidental (→ Alt B's v0 unification). F-002's -"finished form wins" does not automatically mean "v0's finished form wins" — v1's -dual-backend architecture is a deliberate divergence from v0. - ---- - -**Main-loop decision on the DA's closing judgement.** The split IS essential v1 -architecture: it is the single seam that keeps `bindCallFrame` one-source across -TreeWalk (no operand stack) and the VM. Alt B forks that binder — a direct -regression of `ab1959c2` and the F-011 property. Therefore **Alt A is v1's -finished form** (chosen above), with the DA's prerequisite corrections (a)–(d) all -folded into the Decision. The DA's F-006 warning about the "ambiguous half-state" -is answered by Alt A's explicit base-windowing discipline: a frame reads only its -own `[base..sp]` / `[loc_base..loc_base+fs]` window (never a sibling's), and -`bindCallFrame` nil-inits each locals window on push (O-015) — so the shared -arenas carry the SAME nil-valid invariant as today, walked as 2 big slices instead -of an N-chain (simpler, not more dangerous). Verdict (a)'s "worst of both" only -applies to an UN-disciplined half-state; Alt A is the disciplined one. diff --git a/.dev/decisions/0132_gate_releasesafe_binary_invariant.md b/.dev/decisions/0132_gate_releasesafe_binary_invariant.md deleted file mode 100644 index b49c3384e..000000000 --- a/.dev/decisions/0132_gate_releasesafe_binary_invariant.md +++ /dev/null @@ -1,100 +0,0 @@ -# ADR-0132: The gate's shared e2e binary MUST be ReleaseSafe (the Debug-binary perf cliff) - -- Status: Proposed → Accepted (2026-06-12) -- Supersedes / relates to: ADR-0107 (two-tier gate cadence), D-385 (slow - gate). Amends `.claude/rules/gate_cadence.md` (the ~5-min / ~30-min - framing was a symptom, not an inherent cost). - -## Context - -D-385 framed the full gate (`test/run_all.sh`) as an inherent ~30-min run -whose only relief was `--resume` + `--serial-e2e`. Investigating "why is the -gate so slow" (2026-06-12) found the premise was **false**: the gate was -running its 291-step e2e suite against a **Debug** `cljw` binary. - -Measured facts (Apple Silicon, host idle): - -| binary | `cljw -e '(+ 1 2)'` cold-start | `phase14_seq_helpers2.sh` | -|-------------|--------------------------------|---------------------------| -| Debug | **1.70 s** | **391 s** | -| ReleaseSafe | **~0.005 s** | **0.66 s** | - -The e2e suite spawns ~3200 short-lived `cljw` processes; at the Debug -cold-start that is ~90 min of pure process startup. On ReleaseSafe the whole -e2e suite is ~5 s serial / ~1.5 s parallel. The intended design was already -ReleaseSafe (`run_all.sh` exports `CLJW_OPT=ReleaseSafe`; `build_cljw` builds -it once; e2e scripts skip their own build via `CLJW_SKIP_BUILD`), but **three -compounding bugs silently reverted the shared `zig-out/bin/cljw` to Debug**: - -1. **`test/clj/run_tier_a.sh` ran a bare `zig build`** (no `-Doptimize`, no - `CLJW_SKIP_BUILD` guard) → installed a **Debug** binary right before the - 291-step parallel pool. Every subsequent step then ran Debug. *(primary)* -2. **`build_cljw` was in the `--resume` ledger** → a `--resume` pass skipped - it, so the ReleaseSafe binary was never (re)built and e2e ran whatever - stale Debug binary existed. -3. **The ReleaseSafe guard ran only once**, right after `build_cljw`, so a - *later* reversion (bug 1) was invisible. - -Amplifier: on a Debug binary the `-P8` parallel pool is *slower* than serial -(concurrent cold-starts contend on the kernel VM reservation / AMFI signature -check — measured -P8 13.7 s vs -P4 4.1 s for 40 Debug-run scripts). On -ReleaseSafe, parallelism is harmless (-P1 0.72 s … -P8 0.19 s for the same -40). So "parallel is bad" is true *only* on Debug; the real defect is the -Debug binary, and parallelism merely magnified it. - -This is a **silent perf cliff**: every test still PASSes (Debug and -ReleaseSafe compute identical Values), only wall-time differs ~100×, so no -failure ever pointed at it. - -## Decision - -Make "the shared e2e binary is optimised" a **mechanically enforced -invariant**, not an implicit assumption. - -1. **Every gate-invoked `zig build` either honours `CLJW_SKIP_BUILD` or - passes `-Doptimize="${CLJW_OPT:-ReleaseSafe}"`** — never a bare - `zig build` (= Debug). Fixed: `test/clj/run_tier_a.sh` (the live bug), - `scripts/clj_diff_sweep.sh`, `scripts/check_corpus_regression.sh` (latent - `[ -x "$BIN" ] || zig build` fallbacks). -2. **`build_cljw` + `assert_e2e_releasesafe` are `NO_RESUME_STEPS`** — they - run on every pass (including `--resume`), so the optimised binary is - always (re)established before e2e. A cache-hit rebuild is ~0.3 s. -3. **Semantic build-mode guard, not a size heuristic.** `cljw --version` now - bakes in `@import("builtin").mode` → - `ClojureWasm v (ReleaseSafe)`. The post-`build_cljw` guard AND a new - **pre-parallel-pool backstop** both assert the mode is not `Debug`, - aborting loudly if a step reverted the binary. (Replaces the interim - `wc -c < binary` < 8 MB size check — semantic, per user direction to use - a Zig-native build-mode signal.) - -## Consequences - -- **Full gate: ~113 s** (was multi-hour / timing out across 5 `--resume` - passes). FAIL 0, skipped 0, `.gate_pass` fingerprint matches. The "~30-min - gate" was the Debug cliff; D-385's resume/timing slices (already landed) - remain useful but the throughput crisis is resolved. -- A future bare `zig build` in any gate-path script can no longer silently - cost 100× — the backstop aborts before the pool with a build-mode message. -- `cljw --version` is now self-describing for users too (no more "guess the - build from the binary size"). -- **Not fixed here (tracked separately):** the `phase16_wasm_*` e2e are not - wired into `run_all.sh`, so the Wasm-FFI *execution* path is not gate- - tested (user-flagged 2026-06-12). New debt row — see D-385 sibling. - -## Alternatives considered - -- **Size-threshold guard (`wc -c` < 8 MB).** Landed first as the interim - guard; rejected as the final form — a heuristic that drifts as the binary - grows, and the user explicitly asked for a semantic Zig build-mode signal. - `@import("builtin").mode` is comptime-known and exact. -- **Dedicated `cljw --build-mode` flag.** Cleaner to parse, but a separate - flag the user would not see. Folded into `--version` instead so the signal - is both human-facing (the original "size is inconvenient" complaint) and - machine-checkable. -- **Remove / cap the parallel e2e pool.** Tempting given "-P8 is slower", - but the measurement shows parallelism is *fine* on ReleaseSafe; the pool is - not the defect. Keeping it preserves headroom; the real fix is the binary. -- **Isolate the e2e binary to a dedicated immutable path** so no `zig build` - can clobber it. The structurally cleanest option, but needs `build.zig` to - emit a second artifact name; deferred — the invariant guard + the - bare-build ban achieve the same guarantee with far less surgery. diff --git a/.dev/decisions/0133_default_gate_builds_wasm.md b/.dev/decisions/0133_default_gate_builds_wasm.md deleted file mode 100644 index 22be85052..000000000 --- a/.dev/decisions/0133_default_gate_builds_wasm.md +++ /dev/null @@ -1,111 +0,0 @@ -# ADR-0133: The default full gate builds `-Dwasm` throughout + runs the wasm e2e - -- Status: Proposed → Accepted (2026-06-12) -- Governing fact: **F-001** Revision history 2026-06-12 (user-directed - amendment — zwasm v2 is provisionally complete, so wasm joins the default - gate). Supersedes the D-387 / ADR-0132-era "wasm is a SEPARATE opt-in - runner" shape. Builds on ADR-0132 (the shared-binary-stays-optimised - invariant) — this is its `-Dwasm` sibling. - -## Context - -D-387 (2026-06-12, earlier same day) wired the wasm-FFI execution e2e into a -SEPARATE opt-in runner (`scripts/run_wasm_gate.sh`) specifically because F-001 -then forbade the default gate from resolving zwasm. Hours later the user -amended F-001: zwasm v2 is now complete, so the Wasm-FFI path (the headline -feature) MUST be in the default full gate (Mac AND ubuntunote). Verbatim -intent: 「デフォルトゲートフルゲート（Ubuntu含む）に含むべき … 冒頭から -Dwasm -でビルドする … 単品でビルドするものもすべて sh に -Dwasm 必要」. - -## Decision - -**Every `zig build` that actually executes in the gate carries `-Dwasm`**, and -the wasm e2e are ordinary gate steps: - -- `build_cljw`, the dual-backend diff oracle (`zig build test` ×2), and - `zlinter` → `-Dwasm`. -- `run_tier_a.sh` + the `[ -x "$BIN" ] || zig build` conditional builders - (`clj_diff_sweep`, `check_corpus_regression`) → `-Dwasm`. -- **`phase4_{cli,exit,exit_codes}.sh` → `-Dwasm`** on every `zig build` - (the `-Dbackend` rebuilds AND the default-restore). This was the live bug: - phase4 rebuilds the shared binary for backend-difference testing and ran - *non-wasm* `zig build -Doptimize=ReleaseSafe`, **reverting the binary to - non-wasm mid-gate**, so the wasm e2e (which run later in the parallel pool) - saw `(ReleaseSafe)` not `(ReleaseSafe, wasm)` and failed their wasm-enabled - assertion. This is the ADR-0132 shared-binary-reversion class, `-Dwasm` - variant — found by the wasm e2e failing in the default gate. -- `phase16_wasm_{ffi,run}.sh` are `run_step` e2e; they reuse the shared - `-Dwasm` binary (`CLJW_SKIP_BUILD`), drop the stale - `[ ! -d ../zwasm_from_scratch ]` skip (it false-skipped on tag-pin hosts - like ubuntunote), and assert `cljw --version | grep wasm`. - -**zwasm is resolved via the `build.zig.zon` tag-pin** (`v2.0.0-alpha.2` + -`.hash`, `lazy`), so no `../zwasm_from_scratch` sibling is needed — the gate -runs on ubuntunote (which has only the cljw clone). If `-Dwasm` cannot resolve -zwasm, `build_cljw` fails first: wasm is a **gate prerequisite** now, not a -graceful skip. - -**The "every gate `zig build`" scope is the EXECUTING ones only.** The ~291 -`test/e2e/*.sh` `[ -n "$CLJW_SKIP_BUILD" ] || zig build` lines are -`CLJW_SKIP_BUILD`-skipped in the gate (dead there) — they do NOT need `-Dwasm`. -An initial bulk edit added `-Dwasm` to all 291; it was reverted as the wrong -target (the user's "単品でビルドするもの" = the *unconditional* in-gate builders, -i.e. phase4, not the skipped e2e lines). - -## Consequences - -- The default full gate covers the Wasm-FFI execution path (add→42, no-leak, - catchability, wasm/run WASI). A `(wasm/call)` regression now fails the gate. -- `cljw --version` shows `(ReleaseSafe, wasm)` for the gate binary (the - `build_options.wasm` flag, ADR-0132 build-mode work) — the semantic signal - the wasm e2e + the gate guard read. -- Gate wall grows (~113s non-wasm → ~250s) from the `-Dwasm` test/build - compiles + the wasm e2e; acceptable for full coverage of the headline - feature. zwasm is cached after first fetch. -- `scripts/run_wasm_gate.sh` (D-387) becomes a convenience quick-runner - (wasm-only, no full gate) rather than the primary coverage. -- **ubuntunote verification is the remaining step** — `zig build -Dwasm` - must fetch the zwasm tag + build on Linux there (the user flagged the - Ubuntu build-differentiation as unchecked). - -## Alternatives considered - -- **Keep wasm as a separate opt-in runner** (D-387 / pre-amendment). Rejected: - F-001 was amended to require default-gate coverage; an opt-in runner that - nobody auto-invokes lets regressions ship green. -- **Only build_cljw `-Dwasm`, leave phase4 non-wasm.** Insufficient — phase4 - reverts the shared binary mid-gate (the live failure). Every executing - in-gate `zig build` must carry `-Dwasm`. -- **`-Dwasm` on all 291 e2e standalone build lines.** Wrong target — they skip - in-gate; pure churn (298-file diff). Reverted to the executing builders only. -- **Make the shared binary `-Dwasm` only via a dedicated wasm-binary path.** - Cleaner isolation but needs `build.zig` to emit a second artifact name; - deferred — the "all gate builds carry `-Dwasm`" rule keeps one binary. - -## Revision history - -- **2026-06-13 (user-directed)**: the "all 291 e2e standalone build lines" - alternative is UN-rejected and landed (310 files by now). The original - rejection reasoned only about in-gate behaviour (the lines are - `CLJW_SKIP_BUILD`-skipped there — still true). What it missed is the - **standalone path**: a bare `zig build` fallback (a) builds **Debug**, - overwriting the gate's ReleaseSafe `zig-out/bin/cljw` so subsequent manual - probes run the ~10-100× Debug cliff, and (b) uses a third build config - (non-wasm Debug) so alternating standalone e2e ↔ gate builds thrashes the - zig cache (the "zig build is slow" report, 2026-06-13). Every - `test/e2e/*.sh` fallback now reads - `zig build -Dwasm -Doptimize="${CLJW_OPT:-ReleaseSafe}"` — identical to - `build_cljw`, so a standalone run is a cache hit (~0.6s measured) and never - degrades the shared binary. In-gate behaviour is unchanged (lines stay - dead). Manual-probe guidance follows the same form; remaining non-unified - builders (`bench/*.sh`, `scripts/perf.sh` — measurement-continuity - question) are tracked as a debt row, not silently flipped. -- **2026-06-13 (user-directed, later same day)**: the bench/perf builders are - unified too — the user resolved D-411's continuity question by directive: - Debug (or any non-gate config) must have NO foothold in e2e/bench/perf - tooling; `zig build` bare stays acceptable ONLY for ad-hoc hand experiments. - All 9 remaining builders (`bench/{build_bench,release_metrics,compare_langs, - run_bench,wasm_bench,simd/run_simd_bench}.sh`, `scripts/{perf, - check_vm_parity,verify_projects}.sh`) now carry `-Dwasm` + Release mode, and - the bench history is RE-BASELINED under the unified config (same cycle), so - pre-2026-06-13 bench numbers are not directly comparable (non-wasm binary). diff --git a/.dev/decisions/0134_iobj_imeta_activation.md b/.dev/decisions/0134_iobj_imeta_activation.md deleted file mode 100644 index dc6f7b07b..000000000 --- a/.dev/decisions/0134_iobj_imeta_activation.md +++ /dev/null @@ -1,206 +0,0 @@ -# ADR-0134: IObj/IMeta activation — make every clj-IObj substrate metable (D-271) - -- Status: Proposed → Accepted (2026-06-12) -- Governing facts: **F-011** (behavioural equivalence on VALUE), **F-002** - (finished-form wins; cycle/LOC is not a constraint), **F-013** (closed-set - recognition up-front). F-004 is UNTOUCHED (a `meta: Value` field on a HEAP - extern-struct is not a NaN-box slot). F-003 does NOT govern: D-271 is a - clj-parity quality-loop FLOOR item (`tech_debt_consolidation.md` — the post-M - quality loop owns clj-parity floor items), so there is no future-Phase owner to - defer the layout change to. Closes the ADR-0116 Decision D deferral. -- Supersedes: the D-271 "SCOPE CORRECTION" reading that treated this as - F-003-deferrable. - -## Context - -`clojure.lang.IObj`/`IMeta` resolve to NEITHER a class value NOR `instance?` -membership in cljw (both name_error; contrast IFn/Object/Number, which ARE -landed via the analyzeSymbol interface-marker arm + an `interface_membership.zig` -table row). The membership sets are enumerated INACTIVE in -`interface_membership.zig` (ADR-0116 Decision D), gated on D-271 because clj -guarantees the invariant `(instance? clojure.lang.IObj x)` ⟹ `(with-meta x m)` -works, and cljw's `.range`/`.cons`/etc. substrates carry no meta slot — claiming -membership without with-meta support would be an internally inconsistent core -predicate. - -This blocks two real things: (a) `(with-meta (range 3) m)` (the original D-271 -bug); (b) **clojure.datafy** — its core guards `(instance? clojure.lang.IObj v)`, -which name_errors, so the whole ns cannot load clj-faithfully (the datafy P3 -ns-backfill is otherwise unblocked: Throwable->map landed D-389, core.protocols -present). - -clj-oracle-verified IObj membership (2026-06-12): **IObj** = vector, list, -hash_set, sorted_map, sorted_set, persistent_queue, cons, lazy_seq, range, -chunked_cons, array_map, hash_map, string_seq, array_seq, symbol, promise, -future, **and fns** (`(meta (with-meta (fn [] 1) {:a 1}))` → `{:a 1}`). -EXCLUDES delay, keyword, map_entry. **IMeta** = IObj ∪ {atom, agent, ref, -var_ref, ns}. Already-metable in cljw: vector, list, hash_set, array_map, -hash_map, lazy_seq, symbol, persistent_queue (+ var_ref/atom/record/reify). - -## Decision - -**Finished form** = every clj-IObj substrate metable + full IObj/IMeta membership -+ value-resolution matching clj's oracle-verified set (DA-fork Alt 2 "Full" — the -only shape where the clj invariant holds for every tag with zero exceptions, no AD -needed). This is the terminal target and does not change. - -**Execution = VALUE-DRIVEN, not speculative-complete-now (amended 2026-06-12 after -a user finished-form reconsideration that caught a Progress-pressure / -scope-escalation smell in the original "do all 13 substrates now" framing).** The -original D-271 bug (`(with-meta (range 3) m)`) is NICHE, but the IObj/IMeta -VALUE-RESOLUTION + membership half now has **TWO real consumers** (found via P4 -lib-load, 2026-06-12): **clojure.datafy** (its core guards -`(instance? clojure.lang.IObj v)`) and **instaparse** (instaparse.auto_flatten_seq -is a deftype EXTENDING `clojure.lang.IObj`/`IMeta` + `(instance? clojure.lang.IObj -obj)`; blocks at auto_flatten_seq.cljc:157). instaparse additionally needs the -clojure.lang.* deftype-supertype recognition/remap (D-280 family) for the -extend-type half. So the value-resolution + metable-tag-membership slice (the -first slice below) is now PULLED — no longer purely speculative; the remaining -cross-zone fn-meta (tree_walk closure ctors + multimethod + protocol, for the rare -`(with-meta a-fn m)`) is still high-effort / near-zero-value and stays last. So: - -- A substrate is made metable + admitted to IObj/IMeta membership **when a real - consumer pulls it** (or opportunistically when its module is already open), NOT - speculatively to chase clj-completeness. -- Membership stays **consistent at every step**: `(instance? IObj x)` is activated - for tag x ONLY once x is genuinely metable, so the invariant - `instance? IObj ⟹ with-meta` never breaks. The set being NARROWER than clj - (range/cons/fn report `(instance? IObj x)` → false until made metable) is an - HONEST, TRACKED gap (D-271), NOT a lie and NOT an AD — it is ordinary - incremental growth, and `(instance? IObj range)`→false is graceful (a ported - `(if (instance? IObj x) (with-meta x m) x)` returns x unchanged, not an error). - This is the DA's permitted sequencing variant (interim narrow membership + - completion debt row), elevated to the execution path. -- **The terminal Full membership flip is reached by accretion, not a single - speculative leap.** "Big-bang" (the original framing) is reinterpreted: never - ship a narrow membership as PERMANENT (that would be the gap-as-AD lie), and - never drip COVERAGE claims — but DO grow the metable set value-first. - -The first value-driven slice (whenever taken): resolve `clojure.lang.IObj`/`IMeta` -as class values + activate membership for the CURRENTLY-metable tags (vector/list/ -hash_set/array_map/hash_map/lazy_seq/symbol/persistent_queue/var_ref/atom/record/ -reify) — this clears the `name_error` and is the part datafy's guard exercises. -Each later substrate (range/cons/sorted/promise/future/fn) joins when pulled. - -Per-substrate work (template = `lazy_seq.zig`'s `meta: Value = nil` field): - -1. Add a `meta: Value` field to the extern structs of the currently-unmetable - IObj tags: **cons, range (LongRange), chunked_cons (ChunkedCons; ChunkBuffer - is an internal node, no meta), string_seq, array_seq, sorted_map, sorted_set, - promise, future**, and the **fn tags** (fn_val/builtin_fn/multi_fn/protocol_fn - — a callable carrying meta; mind the closure/dispatch structs). -2. Mark the new `meta` field in each tag's GC-trace arm (`gc/tag_ops.zig`) — the - one genuine hazard; a missed mark is a use-after-free. -3. Add `metaOf` + `withMeta` (shallow-copy sharing internals, new meta) per - struct; wire the `metadata.zig` `metaFn`/`withMetaFn` switch arms. -4. Activate the IObj/IMeta rows in `interface_membership.zig` (flip INACTIVE → - table rows pointing at the verified tag-set constants). -5. Resolve `clojure.lang.IObj`/`IMeta` as class values (analyzeSymbol - interface-marker arm, mirroring IFn/Object). - -**Execution sequencing (value-driven; amended 2026-06-12 per the Decision above).** -Steps 1-3 (adding a `meta` field + GC-trace + metaOf/withMeta to a substrate) are -an INDEPENDENT `(with-meta m)` bug-fix per tag, done WHEN PULLED. Steps -4-5 (membership rows + value-resolution) activate PER TAG as that tag becomes -metable — `(instance? IObj x)` is switched on for x ONLY once x is genuinely -metable, so the one-directional invariant (`instance? IObj` ⟹ `with-meta`) holds at -every step. The membership set therefore GROWS toward clj's full set by accretion; -it is narrower than clj in the interim (an honest, D-271-tracked gap, NOT a -permanent AD). "Big-bang" is reinterpreted as the two PROHIBITIONS it really -protects: (1) never ship a narrow membership as PERMANENT (gap-as-AD lie), (2) -never drip COVERAGE claims — NOT a mandate to build all 13 substrates before any -membership activates. The cross-zone fn tags (`eval/backend/tree_walk.zig` closure -ctors + `multimethod.zig` + `protocol.zig`) are LAST and only if a consumer needs -fn-meta (rare). Each substrate's GC trace is torture-verified independently before -the next. - -**Execution mandates (make Full safe):** -- **Per-tag with-meta round-trip corpus line under `CLJW_GC_TORTURE`** for each - newly-metable struct (`(meta (with-meta {:a 1}))` → `{:a 1}`). This is - what converts F-012 (diff oracle) + torture coverage from theoretical to actual - on the GC-trace risk — without a per-tag line that traverses the new field, the - torture coverage is only nominal. -- Match clj's set EXACTLY (exclude delay/keyword/map_entry from IObj) so F-013's - closed-set recognition stays honest; no over-claim. -- Big-bang per `clj_diff_sweep.md` Discipline 2 — all substrates in one focused - push, NOT one-struct-per-commit (the drip-feed smell). - -## Consequences - -- `(with-meta (range 3) m)` / cons / sorted-map / fn / etc. all work; `(meta …)` - round-trips; `(instance? clojure.lang.IObj x)` / `IMeta` match clj on every - tag. clojure.datafy unblocks. D-271 discharges; no residual AD. -- Extern-struct sizes grow by 8 bytes (one Value) on ~13 substrate structs. The - fn tags are the delicate part (meta on a callable — confirm it composes with - dispatch/closure capture and does not perturb the call path). -- sorted_map/sorted_set carry comparator state; the meta field is orthogonal to - ordering (verify withMeta preserves the comparator). -- GC-trace correctness is the load-bearing risk, covered by the per-tag torture - corpus mandate above. - -## Alternatives considered - -(Verbatim from the mandatory Devil's-advocate fork — `general-purpose`, fresh -context, F-NNN envelope.) - -**Leading finding:** No F-NNN is violated by any alternative. F-004 untouched -(heap field, not a NaN-box slot). F-003 does NOT forbid the loop doing the layout -change — D-271 is a clj-parity floor item the post-M quality loop OWNS; there is -no Phase owner to defer to. The choice is governed by F-011 + the clj invariant, -not an F-NNN block. Pivotal sub-ruling: value-resolution and membership are -SEPARABLE code points, and datafy is unblocked by partial-but-correct membership -(it only feeds maps/vectors to its own guard) — but a core predicate returning -**false-where-clj-true** is a SEMANTIC divergence (categorically worse than the -representational AD-001/AD-003), so a narrower-than-clj IObj is an acceptable -INTERIM but not a finished form. - -- **Alt 1 — Narrow (smallest-diff):** resolve values + activate membership ONLY - for currently-metable tags; AD for the clj-IObj-but-cljw-not-metable tags - (cons/range/sorted/etc. report `(instance? IObj x)` → false where clj true). - BETTER: unblocks datafy + literal/collection with-meta with zero substrate - risk, internally consistent. BREAKS: ships a core predicate that LIES vs clj on - ~9 tags — a semantic gap any IObj-guarding ported lib hits; sorted_map/set - exclusion especially sharp. Finished-form-clean ONLY if cljw's IObj is - permanently a subset, which contradicts F-011. Legitimate interim, NOT terminal. -- **Alt 2 — Full (finished-form-clean) — RECOMMENDED + CHOSEN:** add meta to all - unmetable IObj substrates THEN activate full membership. BETTER: clj invariant - holds for every tag, no AD, F-011 fully satisfied — the only option with no - core predicate lying (F-002 terminal shape). BREAKS/RISK: multi-struct layout + - per-struct GC-trace (the real hazard) — covered by F-012 + CLJW_GC_TORTURE - PROVIDED each new field gets a torture-exercised with-meta round-trip corpus - line (else torture coverage is theoretical). -- **Alt 3 — Wildcard (uniform side-table meta), REJECTED on correctness:** store - meta in a pointer-keyed side-table, one GC-trace site. BETTER: one mechanism, - no layout change. BREAKS: violates Clojure's meta-is-value-not-identity - contract (with-meta returns a NEW object; pointer-identity keys conflate - allocation with value, desync on copy/realize, leak without weak keys which a - moving GC makes unstable — collides with F-006 + `gc_rooting.md`). A - misuse-resistant-looking shortcut that is a correctness violation. The - sequencing sub-variant (Alt 1 now → Alt 2 next, with a completion debt row) is - the ONLY acceptable form of staging, never Alt 1 standing alone (drip-feed - smell). - -DA recommendation: **Alt 2 (Full)**, with the per-tag torture corpus mandate + -exact-clj-set matching. The main loop adopts this unchanged. - -## Revision history - -- **2026-06-20 — `.reified_instance` value-driven slice (puller: clojure.spec.alpha).** - spec's `with-name` attaches `::name` metadata to every spec object, and a spec - object is a `reify` instance — so spec's registry silently stored `nil` (with-name - fell through cljw's not-IObj reify arm) until reify became metable. This slice - executes the per-substrate template (steps 1-5) for the `reified_instance` tag: - a `meta: Value` slot on `ReifiedInstance` (16→24B), marked in - `traceReifiedInstance`; `reifiedInstWithMeta`/`reifiedInstMetaOf`; the - `metadata.zig` / `meta.zig` reify arms mint-fresh / read-slot instead of raising; - and `.reified_instance` added to `IOBJ_TAGS` + `IMETA_TAGS`. GC-trace verified - under `CLJW_GC_TORTURE=1` (round-trip corpus line in `metadata.txt` + a 50-instance - survival stress). **Correction to this ADR's own text:** the "first value-driven - slice" list (and the "already-metable … reify" note) named `reify` among - currently-metable tags — that was wrong for the auto path: a plain `reify` (no - explicit `clojure.lang.IObj` impl) was NOT metable; only a reify DECLARING IObj - answered true via `matchUserType`. clj's reify ALWAYS implements IObj, so the slot - is unconditional now. A plain `deftype` (`.typed_instance`) stays NOT-auto-IObj — - that clj asymmetry is preserved (e2e `deftype-not-iobj`). This is execution of the - already-accepted Alt-2 template for one more tag, not a new decision (no fresh - accept-gate / DA fork). e2e: `phase14_reify_metadata.sh`. diff --git a/.dev/decisions/0135_wasm_component_as_namespace.md b/.dev/decisions/0135_wasm_component_as_namespace.md deleted file mode 100644 index 48ca19a21..000000000 --- a/.dev/decisions/0135_wasm_component_as_namespace.md +++ /dev/null @@ -1,447 +0,0 @@ -# ADR-0135 — Wasm Component Model as a first-class Clojure namespace (`require` a component, call its exports with Clojure data) - -- **Status**: Proposed → Accepted (2026-06-13, user-directed) -- **Driven by**: the project's north-star differentiator (ROADMAP §1.2 axis 2; - ADR-0099 / the Clojure-conj CFP "WebAssembly as an FFI — calling a `.wasm` - *like a namespace*, the one thing no other Clojure runtime demonstrates"). - This ADR fixes the **finished form** that ADR-0099's minimal - `(wasm/load)`+`(wasm/call handle "export")` core-module spike points toward. -- **Relates to**: ADR-0099 (minimal core-module FFI spike — the v0.1 layer - below this), F-001 (zwasm v2 unavoidable, lazy + `-Dwasm`-flag-guarded), - F-009 (impl/surface split), ROADMAP §8 (Wasm/edge strategy), §6 (tier - system). External dependency: **zwasm's Component Model + WASI-P2 campaign - (zwasm ADR-0170, `-Dcomponent`)** — functional, embedding API **not yet - frozen**; this ADR is designed to land **paced by that API freeze**. - -## Context - -The four Clojure dialects each bind one host: clj↔Java (`:import`), -cljs↔JavaScript, cljd↔Dart. Each is **host-specific**. ClojureWasm's -analogue — clojure↔**WebAssembly Component** — is structurally *better* -because the Wasm Component Model contract is **language-neutral, -spec-defined, and self-describing**: - -- A WIT (`WebAssembly Interface Types`) interface describes a component's - exports with a rich, typed value language (records, variants, lists, - options, results, resources) that maps **almost 1:1 onto Clojure data**. -- A **component binary embeds its own interface types** (Component-Model - binary format) — so the export list + their WIT types are recoverable by - *decoding the `.wasm` alone*. **No `.wit` sidecar file is required.** (The - contract is the public spec: `CanonicalABI.md` / `WIT.md` / `Binary.md`.) -- cljw is Zig-native, so the lift/lower marshalling is a **hand-written fast - path**, not a reflection layer — and the optimisation ceiling is the metal - (see ROADMAP §1.2 the Zig-level thesis). - -Current state (2026-06-13): cw v1 ships only the minimal core-module FFI -(`wasm/load` + `wasm/call handle "export"` — string-keyed, i32/i64/f-only, -manual marshalling; ADR-0099). cw v0 had the *seed* of the higher form -(`:cljw/wasm-deps` named modules in `deps.edn` + a WIT signature parser). -zwasm is building the Component-Model runtime (WIT type system + Canonical -ABI lift/lower + component instantiate/invoke) to wasmtime-equivalent -conformance, behind `-Dcomponent`. - -## Decision - -**A WebAssembly *component* is loadable as a first-class Clojure namespace.** -`require`-ing it introspects the component's embedded WIT exports and interns -one Clojure Var per exported function; arguments and results are negotiated -by the Canonical ABI so the caller passes/receives **Clojure data**, never -raw `i32`s. - -### Surface (two shapes; deps+require is the finished form, load is the REPL hatch) - -```clojure -;; deps.edn — a component declared like any dependency (cw v0 :cljw/wasm-deps, -;; evolved from :path module to :component). -{:cljw/wasm-deps {markdown {:component "libs/markdown.wasm"}}} - -;; finished form — same hand-feel as importing Java -(ns my.app (:require [markdown :as md])) -(md/render "# hi") ;; md/render generated from the component's WIT export; - ;; args/return are Clojure data, docstring/arglists from WIT - -;; REPL / dynamic escape hatch -(def md (cljw.wasm/load-component "markdown.wasm")) ;; introspect → ns-like handle -;; or a def-ing macro: -(cljw.wasm/import-component "markdown.wasm" :as md) -``` - -The component's WIT exports are the **SSOT** for the generated surface — no -hand-written wrapper, no `.wit` sidecar. - -### The contract — WIT ↔ Clojure value mapping (Canonical ABI; spec-derived, stable) - -This table is the finished-form artifact. It is derivable from the public -Canonical ABI spec and changes only when the spec does. - -| WIT type | Clojure value | Notes | -|---------------------|-------------------------------------------|-------------------------------------------------------------------| -| `bool` | boolean | | -| `s8…s64`/`u8…u64` | long | `u64`/`s64` may promote to BigInt at the f64 edge (F-005) | -| `f32`/`f64` | double | | -| `char` | char | Unicode scalar | -| `string` | string | UTF-8 ↔ cljw string | -| `list` | vector | element-wise lift/lower | -| `tuple` | vector `[a b …]` | fixed arity | -| `record{a,b,…}` | map `{:a … :b …}` | field names → keyword keys (kebab preserved) | -| `option` | `nil` \| T | `none`→nil, `some(x)`→x | -| `result` | `[:ok v]` / `[:err e]` | payload slot omitted when the arm carries none (amendment 2) | -| `variant` | `[:case-name payload]` | same tagged-vector shape as `result` (amendment 2) | -| `enum` | keyword | | -| `flags` | set of keywords | | -| `resource` | opaque handle, deterministically released | ADR-0159: NO GC finaliser; `own` drop on scope exit (amendment 2) | - -Exports that are interfaces/worlds nest as sub-namespaces or qualified -names (TBD with zwasm's introspection shape). Imports the component *needs* -(e.g. WASI) are satisfied by cljw-provided host functions (the inverse -direction — exposing cljw fns as WIT exports — is ROADMAP §1.2 axis 2's -second half, a later ADR). - -### Dependency & sequencing (paced by zwasm) - -1. **Now (v0.1)**: keep ADR-0099's `wasm/load`+`wasm/call` core-module FFI. - This ADR does *not* deprecate it — core modules without a component - wrapper still need it. -2. **When zwasm's CM embedding API freezes** (zwasm ADR-0170): it must expose - (a) decode a component → list exports + their WIT types; (b) invoke an - export with Canonical-ABI lift/lower of host values. cljw then builds the - `require`/`import-component` introspection + the mapping table above. -3. **`deps.edn` `:cljw/wasm-deps`** is the resolution layer (cw v0's seed). -4. cljw-as-component-output (exporting cljw itself / Clojure fns as WIT) and - WasmGC are later (ROADMAP §8.3). - -## Alternatives considered - -- **Core-module-only, string-keyed call (status quo, ADR-0099)** — keeps - marshalling manual (no records/strings/lists ergonomically). Correct as the - low layer; insufficient as the finished form. *Kept as the layer below.* -- **Require a hand-written `.wit` sidecar** — rejected: the component binary - is self-describing, so a sidecar is redundant and drifts from the binary. - (`.wit` may still be *accepted* as an override/doc, but is not required.) -- **Static codegen (wit-bindgen-style) at build time** — heavier; loses the - dynamic `(require …)`/REPL story that is the differentiator's heart. The - introspect-at-require approach is more Clojure-native. (A build-time AOT - path can be added later for size/speed without changing this contract.) -- **A `variant` as a bare 2-vector `[case value]`** — considered vs the - tagged-map shape; deferred to the implementing cycle (pick the shape that - round-trips cleanly through `case`/`match` ergonomics). - -## Consequences - -- **Differentiator becomes concrete + provable**: "ClojureWasm `require`s a - Wasm component and calls it with Clojure data, types negotiated by the - Canonical ABI — no other Clojure runtime does this, and the marshalling is - Zig-native." This is the CFP heart in finished form. -- **The mapping table is a stable, spec-anchored contract** — a future AI / the - user can implement against it without re-deriving the design. -- **Two-repo coordination is clean**: zwasm owns the CM runtime + embedding - API; cljw owns the ns surface + the value mapping. The freeze of zwasm's CM - embedding API is the one gating event. -- **No core-VM risk**: like ADR-0099, all of this is `-Dwasm`/`-Dcomponent` - flag-guarded; the default gate never resolves zwasm (F-001). - -## Affected files (when implemented) - -- `runtime/cljw/wasm/` (surface + a new `component.zig` introspection/marshal - layer over zwasm's CM API), `lang/require_resolver.zig` (the `:cljw/wasm-deps` - + component-require path), `deps/parse.zig` (`:cljw/wasm-deps` schema). -- Tracked by debt rows D-404 (impl, blocked-by zwasm CM API freeze) and the - conformance/proof rows. - ---- - -## Amendment 1 (2026-06-21, user-directed) — the finished-form surface is settled; blocker dissolved; implementation begins - -**Status of the blocker**: zwasm's Component-Model embedding API is now **functional -and default-ON** (`-Dwasi=p2`, wasmtime-equivalent: real `wasm32-wasip2` components run -e2e, typed embedder introspection + `invokeTyped`, official corpus 158/0/0). cljw already -ships the lower layers: `wasm/load-component` / `wasm/component-exports` (surfaces the -typed WIT signature) / `wasm/component-call`, plus a `require-component` macro, and -`component.zig` already lifts/lowers record/tuple/variant/enum/option/result/flags/string/ -list per the §"contract" table. **D-404's "blocked-by zwasm CM API freeze" is DISSOLVED.** -This amendment fixes the remaining finished-form decisions and starts the build-out. - -### A1. `:require` is overloaded with a STRING libspec (CLJS/CLJD lineage) - -The finished `ns` surface follows the **modern Clojure-family convention**, NOT JVM -`:import`. JVM Clojure binds Java *classes* via `:import` (a flat, sub-functionless -namespace); **ClojureScript** (`["react" :as React]`) and **ClojureDart** -(`["package:flutter/material.dart" :as m]`) bind host *modules* via **`:require` with a -STRING lib name**. A Wasm component is a module (a namespace of typed functions + -resources), so it takes `:require`, not `:import`: - -```clojure -(ns my.app - (:require [clojure.string :as str] ; Clojure ns (unchanged) - ["greet.wasm" :as greeter] ; Wasm component → aliased ns - ["img.wasm" :refer [resize crop]])) ; refer specific exports -(greeter/greet "world") -(resize photo 800 600) -``` - -A **string-headed libspec is unambiguously a Wasm component** in cljw (cljw has no -JavaScript, so the CLJS "string = JS module" meaning cannot collide). `:as` aliases the -component's exports under a namespace; `:refer` interns named exports into the current ns -— identical mechanics to the existing `require-component` macro, which becomes the -**dynamic/REPL escape hatch** (the `:require` directive is the static form; same -`require` fn vs `:require` directive duality as Clojure). **`:import` stays Java-only** -(running existing JVM-Clojure assets); `:require`-string is the *new-code worldview*. - -### A2. Resolution order (CLI-handy first; deps.edn; registry later) - -cljw is lightweight + CLI-handy, so a component must resolve **without** a `deps.edn`. -The resolution kinds are distinguished by the **shape of the string** so precedence is -explicit, not an implicit shadowing chain (the DA-fork's risk #1 — a bare `greet.wasm` -next to the source must NOT silently shadow a classpath one): - -1. **Explicit relative** — a string starting `./` or `../` (`["./greet.wasm"]`) resolves - relative to the **executing source file** (the `.clj` doing the `:require`). Opt-in - only; a bare name never resolves source-relative. -2. **Absolute path** — a string starting `/` (`["/opt/libs/greet.wasm"]`). -3. **A bare logical name / classpath** (`["greet.wasm"]` or `["greet"]`) — resolves on - the existing `rt.load_paths` classpath + a `:cljw/wasm-deps` `deps.edn` alias - (`{:cljw/wasm-deps {greet {:component "libs/greet.wasm"}}}`, cw v0's seed). A bare name - is classpath-first (NOT source-relative-first) — this is the safe default that avoids - dependency-confusion. -4. **Registry coordinate (deferred — debt row)** — a WIT package coordinate - (`["wasi:http/proxy@0.2.0"]`, `namespace:package@version`) resolves through a registry: - **OCI artifacts** (`application/vnd.wasm.config.v0+json`) + the **`wkg` / wasm-pkg-tools** - resolver (`registry.json` at `/.well-known/wasm-pkg/registry.json`; Warg is being - superseded by OCI). **Scoped IN but deferred**: land explicit-relative + absolute + - classpath first, registry as a follow-on. - -The reuse target: `lang/require_resolver.zig` (already maps ns→.clj on `load_paths`) + -`app/deps/{parse,resolve}.zig` (already parse deps.edn + git deps). The component path is -a sibling resolution arm, not a new subsystem. - -### A3. Always-latest Wasm/WASI — a consumer-side invariant (→ F-016) - -zwasm serves Wasm-1.0-only runtime users (`-Dwasi=p1` / lean opt-outs). **ClojureWasm, -as a zwasm *consumer*, forces the full modern surface**: it always embeds zwasm with the -**Component Model + WASI ≥ p2 (the latest zwasm ships; p3/async as it lands)** — there is -no cljw build axis that downgrades to Wasm-1.0-only or drops the component model. Rationale: -cljw is a *language for new code*; its users want the modern, self-describing component -world, and removing the version-negotiation axis is a pit-of-success default ("reach for -cljw's wasm interop → you get Wasm 3.0 + WASI latest, period"). Recorded as **F-016** -(`project_facts.md`) — a user-declared invariant. - -### A4. Type information is RETAINED and LEVERAGED - -A component binary is self-describing, and `component-exports` already decodes the typed -WIT signature (`{:name :params [[name type]…] :result type}`). The finished form attaches -that to each interned Var and uses it: - -- **`:arglists` + `:doc` metadata** on each generated Var, derived from the WIT signature - (param names + types, result type) — so `(doc greeter/greet)` and editor arglist hints - work, identical hand-feel to a normal Clojure fn. -- **Compile-time arity checking — gated to statically-resolved components only** (the - DA-fork's risk #2). When the `.wasm` resolves at analysis time (explicit-relative / - absolute / classpath / embedded), the analyzer decodes its exports and can reject a - wrong-arity call before runtime — richer than Java reflection. But a genuinely-dynamic - `(wasm/load runtime-path)` has **no** signature at analysis time, so the check would - silently degrade to none — two reliability tiers under one syntax. So: the compile-time - check fires ONLY for the static `:require` form (where the component is present + will - be embedded); the dynamic `wasm/load` path is runtime-checked, and that tiering is - **documented + explicit**, never a silent "sometimes-checked". Build-time note: a static - check means the `.wasm` must be present + byte-identical at every analysis (CI included); - the build embeds it (ADR-0158), so "present at build" and "present at analysis" coincide. -- The WIT↔Clojure mapping table (§"contract") is the lift/lower SSOT; types are never - erased to "just call it". - -### A5. Resource ergonomics (prior-art-informed) - -WIT `resource` (a stateful, owned/borrowed handle) maps to an opaque GC-finalised handle -(finaliser calls `resource.drop`; own/borrow tracked). The *call ergonomics* follow the -CLJD precedent for host objects (`obj.method` → `(.method obj)` / `alias/Method`): a -component exporting `resource counter { constructor; increment: func; }` surfaces as -`counter/new` (constructor) + `(counter/increment c)` (method on the handle). Final -method-access shape (a `counter/increment` Var taking the handle as arg 1, vs a -`.increment` interop form) is settled in the implementing cycle; the handle + GC-drop -contract is fixed here. - -### A6. Single-binary embedding → ADR-0158 - -`cljw build` (the bytecode-envelope single-binary builder) must produce a **self-contained** -binary for a module that `:require`s a component: the component's `.wasm` bytes are embedded -into the envelope (cljw already `@embedFile`s the bundled `.clj` core), and the embedded-run -startup resolves component `:require`s from the embedded bytes, not the filesystem. The -mechanism is its own decision — see **ADR-0158**. - -### Alternatives considered (Amendment 1 — Devil's-advocate fork, 2026-06-21) - -A fresh-context DA fork challenged the design within the F-NNN envelope; its output: - -- **Alt A (smallest-diff) — keep `wasm/require-component` as the ONLY surface, no `ns` - integration.** Better: zero risk to the load-bearing `:require` analyzer path; never - mints a new string-libspec special case. Breaks: components aren't declared in `ns`, so - the single-binary static resolver (ADR-0158) must walk top-level forms — a *worse* - static-resolution story. **Rejected (F-002)**: a smaller-diff convenience; the finished - form wants deps in `ns`. -- **Alt B (finished-form-clean) — a dedicated `(:components ["greet.wasm" :as g])` - directive, NOT an overloaded `:require`.** Better: zero string-libspec ambiguity; one - meaning per directive; the DA's sharpest point — *CLJS overloaded strings because it had - JS modules; cljw has no JS, so inheriting the overload is cargo-culting the lineage, not - its rationale*; one well-known key for the static resolver. Breaks: diverges from - CLJS/CLJD muscle memory; a Clojure dev won't reach for `:components` reflexively. No - F-NNN violation. **The DA recommended this.** -- **Alt C (wildcard) — WIT-coordinate / logical-name only (`[greet :as g]` via - `:cljw/wasm-deps`), no path strings in committed `ns`.** Better: kills source-relative - resolution for committed code (relocatable sources); registry future drops in with no - surface change. Breaks: defeats the "CLI-handy WITHOUT deps.edn" goal (a logical name - needs a resolution map). No F-NNN violation but fights the stated goal. - -**Decision: keep the `:require`-string overload (Alt B *not* adopted).** This is the one -point the user decided explicitly ("`:require` 上書きでいいです"). The DA's Alt-B argument -is genuinely strong and recorded here, but the choice is a close, defensible-both-ways -call (purity vs. CLJS/CLJD family-consistency + single-directive cognitive load + genuine -zero-ambiguity in a no-JS runtime), which the user resolved by explicit preference; the -permission-flip covers *open* points + *clearly-better* finished-form choices, not the -override of a deliberate explicit decision on a toss-up. **The DA's three cross-cutting -risk fixes ARE adopted** (they are not about Alt-B): resolution-order safety (A2 reordered -— relative is `./`-opt-in, bare names are classpath-first), compile-time-check tiering (A4 -— static-only, explicit degraded tier), and F-016 kept as a *capability* invariant, not a -consumer-side *rejection* policy (an older component the embedded runtime can satisfy still -loads). - -### Implementation phasing (debt-tracked) - -A (ns `:require`-string wiring) → B (resolution order) → C (type leverage) → D (single- -binary embed, ADR-0158) → E (resource ergonomics + registry). D-404 reframed from -"blocked-by zwasm" to "the impl epic"; sub-rows per phase. - -### Sources (research, 2026-06-21) - -- Component distribution via OCI + `wkg`/wasm-pkg-tools; Warg → OCI transition: - [bytecodealliance/wasm-pkg-tools](https://github.com/bytecodealliance/wasm-pkg-tools), - [component-model: distributing](https://component-model.bytecodealliance.org/composing-and-distributing/distributing.html), - [MS OSS blog: components over OCI](https://opensource.microsoft.com/blog/2024/09/25/distributing-webassembly-components-using-oci-registries/). -- Single-binary embedding precedent (precompile + link): Wasmtime pre-compiling - (`.cwasm`), Wasmer `create-exe` (wasm→object→link). - -## Amendment 2 (2026-08-04) — `result` and `variant` become tagged vectors; the table is corrected against the code - -The 2026-08-04 cross-project audit compared this table against ClojureWit's -`doc/design/0012-wit-clojure-type-mapping.md` (the sibling repo's mapping, -implemented and falsified against a real `echo` component). Three rows differed. -A Devil's-advocate fork then read `src/runtime/cljw/wasm/component.zig` and found -that **this table also disagreed with cljw's own implementation on three rows** — -so the inter-repo divergence was the smaller problem. - -### What was wrong here - -- **`result` discarded the error value.** The table promised - `throw (ex-info … {:wit/err e})`. The code raised `.wasm_trap` with no - payload, whose template reads "WebAssembly module trapped (e.g. - divide-by-zero, out-of-bounds, or an unreachable instruction)". A component - returning `err("not found")` reached the user as a *trap* — a different kind - of event — with the error value gone. Under `provisional_marker.md`'s table - that is the forbidden row: the user sees a plausible outcome while the - intended semantics are silently dropped. -- **`result`-as-throw has no meaning off the return position.** `lift` is - recursive, so a `result` nested inside a `list` or `record` aborted the whole - lift and discarded the successfully-lifted prefix. `list>` - had no representation at all. `Explainer.md` is normative that `result` is the - *recovery* channel — a value, not a control transfer. -- **`resource` said "GC-finalised; finaliser calls `resource.drop`".** ADR-0159 - deliberately did the opposite: release is **deterministic**, and there is no - `host_finalise`. Both this project and ClojureWit independently landed on - explicit close, so this row recorded agreement as if it were a design. - -### The decision - -`result` lifts as **`[:ok v]` / `[:err e]`**, and `variant` as -**`[:case-name payload]`**, with the payload slot omitted when the arm carries -none. One tagged-vector idiom now covers every WIT sum type, and it destructures -directly: `(let [[tag v] (c/f …)] …)`. - -The reasoning is cljw's own, not deference to the sibling. The nested-position -hole and the discarded payload are defects in this implementation, and the -Canonical ABI settles the direction. That ClojureWit reached the same shape from -the same spec is corroboration — and where the spec constrains the *value* and -not the *representation*, the two runtimes are expected to differ. `char` is -exactly that case and is recorded below as a binding difference, not a bug. - -### `char` — a binding difference, not a divergence - -| | cljw | ClojureWit (JVM) | -|--------|----------------|----------------------| -| `char` | Clojure `char` | `Integer` code point | - -ClojureWit rejected `Character` because a JVM `Character` is 16-bit and -truncates the astral planes. That reason is host-specific and does not transfer: -cljw's char is a 21-bit code point. Verified 2026-08-04 — -`(char 0x10437)` returns `𐐷` on cljw and throws "Value out of range for char" on -JVM Clojure. Both bind the same semantic row (*a Unicode scalar value*); the -representation differs because the host char types differ. Shared Clojure glue -moving between the two must convert (`(int c)` / `(char n)`). - -The surrogate-range hazard is shared: a value in `[0xD800, 0xDFFF]` is -representable on both hosts and must trap on lowering. - -### Also corrected in the same cycle - -The integer rows were a *panic* rather than a mapping: lowering used a bare -`@intCast` (a ReleaseSafe safety panic on out-of-range caller data) and lifting -`u64` did the same on **guest-controlled** data. Fixed in `3662d5a9` by sharing -`wasm/call`'s existing range check; `u64` above `i64` max and `s64` outside -`i48` now lift exactly via BigInt, which this table already promised. - -### Not decided here - -`own` / `borrow` are one table row over a two-tier behaviour — `own` wraps only -on the cached `component-call` path and lifts as a bare integer (no drop) on the -one-shot `component-invoke` path. That is a real gap and belongs to D-404's -resource phase, not to a mapping amendment. - -The shared falsifier — one typed `echo` component driven by both repos — is the -mechanism that would keep the two tables from drifting again, and it discharges -D-404's stated blocker ("the full marshalling-table verification is BLOCKED on a -typed component FIXTURE") as a side effect. Recorded as the next step rather -than done here, because it is a cross-repo artifact with its own pinning -question. - -## Amendment 3 (2026-08-04) — `result` / `variant` lowering existed only on paper - -Amendment 2 opened by stating that "`component.zig` already lifts/lowers -record/tuple/variant/enum/option/result/flags/string/list per the §contract -table". The lift half was true. The lower half was not: `component.zig` raised -`feature_not_supported` for `result` **and** `variant` on the parameter side, -with the justification "rare on the input side". - -Two things follow from that, and neither was visible while the claim stood: - -- **A component taking a `result` or `variant` parameter could not be called at - all** — not degraded, not approximate; the invoke aborted. `variant` is the - Component Model's ordinary sum type, so this was not a corner. -- **The amendment's own round-trip was untestable.** The tagged-vector shapes - `[:ok v]` / `[:err e]` / `[:case-name payload]` were decided *because* one - destructuring idiom should cover every WIT sum type — but a shape the lift - emits and the lower side rejects is not a round-trip, and nothing could have - caught the disagreement. - -The `echo` fixture this ADR names as the shared falsifier carries -`echo-result: func(v: result) -> result` and -`echo-shape: func(v: shape) -> shape` precisely to exercise the parameter -direction. Those two rows had never run. - -**Decision**: both lower. `result` accepts `[:ok v]` / `[:err e]`; `variant` -accepts `[:case-name payload]`, with the payload slot present exactly when the -case's type says it is. The two share one shape-check helper, mirroring the way -the lift side shares one emitter — so the shapes cannot drift apart again -without the helper being edited. - -A tag that names no case is an error rather than case 0, and a payload arity -disagreeing with the case type is an error in both directions. Silently picking -a case, or silently dropping a supplied payload, sends the guest a value the -caller did not write; for a sum type that is a wrong-branch bug in the guest, -not a lost detail. - -Verified against the `echo` fixture (all five rows round-trip, including the -payload-less `[:point]`) and pinned by unit tests at the `lower()` boundary, so -the coverage does not depend on a component fixture the pin cannot yet open. - -**Generalisation.** This is the fifth instance this cycle of *documented intent -vs implementation mismatch* — a doc naming a capability the code does not have. -The other four were caught by turning the claim into a gate. That is why the -coverage here is a unit test rather than a note: a claim nothing executes is -indistinguishable from a claim that is false. diff --git a/.dev/decisions/0136_host_frontier_boundary.md b/.dev/decisions/0136_host_frontier_boundary.md deleted file mode 100644 index 79e79806f..000000000 --- a/.dev/decisions/0136_host_frontier_boundary.md +++ /dev/null @@ -1,198 +0,0 @@ -# ADR-0136 — The host-frontier boundary: a 4-rule decision procedure for "language feature" vs "Tier D JVM detail" - -- **Status**: Proposed → Accepted (2026-06-13, autonomous per CLAUDE.md - § ADR-level designs are handled inline; Devil's-advocate fork embedded - below) -- **Driven by**: D-406 (the boundary was implicit / reactive - marker-by-marker, so "Clojure assets run as-is" read as an unbounded - treadmill) + the 2026-06-12/13 conformance campaign that produced enough - concrete classifications to derive the rule (D-391..D-405, the - bouncer/clj-time finding, the D-400 marker-family close-out). -- **Relates to**: ADR-0013 (Tier D rationale), ADR-0059 (no-JVM), ADR-0102 - (`host_interfaces.yaml` closed-set SSOT — the R2 leaf), ADR-0134 - (value-driven membership), ADR-0135 (Wasm components — the R1 refusal's - forward pointer), F-009 (neutral impl + thin surfaces), F-011 (clj-oracle - behavioural equivalence), F-013 (definition-derived coverage, closed-set - SSOTs + gates), ROADMAP §6 (tier system; this ADR adds §6.5). - -## Context - -cljw's compatibility promise is "pure-Clojure assets run as-is" — but real -libraries constantly touch the JVM host surface (`clojure.lang.*`, -`java.*`). Until now each hit was classified reactively: a marker row here -(D-394/395/397/399/400), a Tier-D refusal there (gen-class), with the line -living only in precedent. D-406 named the consequence: without a stated -boundary the grind cannot *converge* — every new library may demand a new -judgement call, and "compatible" has no finite definition. - -The 2026-06-12/13 conformance campaign (15 lib corpora, 284 golden pairs) -supplied the missing evidence: which host names real Clojure code actually -reaches, and which of those cljw could satisfy canonically vs which it must -refuse. This ADR turns that precedent into a **decision procedure** so the -next session classifies by running rules, not by pattern-matching history. - -## Decision - -A `clojure.lang.*` / `java.*` name hit during the compat grind classifies -by running these rules **in order** (first match wins): - -- **R1 — Artifact test → Tier D.** Does satisfying the name require a Java - *artifact* (joda-time, Jackson, guava, …) or JVM *machinery* (bytecode - emission, classloaders, reflection metadata)? Then it is Tier D. The - refusal's catalogue template names the cljw-native path (a pure-Clojure - alternative, or a future Wasm component per ADR-0135 / F-001). The - canonical instance: bouncer hard-requires clj-time (joda) — cljw never - shims a Java library. R1 runs first because it is the cheapest and most - total test. -- **R2 — Polymorphism-seam test → language feature.** Is the name a - `clojure.lang.*` abstraction that clojure.core itself dispatches on — in - deftype/reify/extend SUPERTYPE position or at the CLASS FACET - (`instance?` / `isa?` / `extend-protocol` / `print-method` dispatch)? - Then it is a language feature; coverage is definition-wide (the whole - interface, F-013) and the closed set is **`host_interfaces.yaml`** - (ADR-0102, gate G4). -- **R3 — Value-semantics test → language feature.** Is the observable - behaviour a pure function of Clojure values (`Util/equiv`, `Murmur3`, - `String`/`Long`/`Double` statics, `StringBuilder`/`StringWriter`, - `Pattern`/`Matcher`, `UUID`, `Date`/`Instant`, the java.util - collection-VIEW methods `.size`/`.contains`/`.indexOf`/`.comparator`/…)? - Then it is a language feature, provided as a thin surface over a neutral - impl (F-009); the closed set is **`compat_tiers.yaml`** (gates G2/G3). - **Tiebreak (the Murmur3 case)**: a name that is both value-semantic and - arguably a JVM implementation detail goes to R3 (language feature) iff - its behaviour is clj-oracle-checkable per F-011; otherwise R4. -- **R4 — Implementation-leak default → Tier D.** Everything else — - `clojure.lang.Compiler/*`, reflection (`getConstructor`/`newInstance`/ - `clojure.reflect` over host classes), JVM-internal statics - (`PersistentArrayMap/createAsIfByAssoc`), raw `java.util.concurrent` - surface beyond cljw's own concurrency vocabulary. **Default-deny**: a - name no rule claims is Tier D until an ADR amendment claims it. - -Operational consequences: - -1. Each closed set declares its admitting rule at the FILE level — - `host_interfaces.yaml` IS the R2 set, `compat_tiers.yaml` host_classes - are R3-admitted (header declarations; a per-row constant tag would be - ceremony — the DA's own over-formalisation warning). The one - discriminating population — future `tier: D` rows, where R1 vs R4 - matters for the refusal text — carries `frontier_rule:` per row; the - schema note mandates it and the gate lands with the first such row - (avoiding an aspirational gate over a zero population). This is F-013 - clause 3 applied to the frontier itself. -2. ROADMAP gains **§6.5** carrying the procedure + the worked-example - table, so the grind's convergence claim is structural: side-1 coverage - is a finite, definition-derived surface; side-2 is refused with - catalogued, alternative-naming errors. -3. The upstream-corpus scan (Alternative 3 below) is admitted as - **non-normative evidence** for R2/R3 judgements (a - `~/Documents/OSS/clojure/src/clj/**` hit is strong evidence of a seam), - never as the rule. - -## Alternatives considered - -(Devil's-advocate fork output, fresh-context subagent, 2026-06-13 — -embedded verbatim:) - -> **Devil's-advocate review.** No F-NNN violation is required for any -> finished-form-clean shape — the draft and all three alternatives below -> fit inside F-002/F-009/F-011/F-013/ADR-0059/F-001. The draft's main -> exposure: its nine buckets (a)–(i) are **campaign-derived, not -> definition-derived** — they generalise what the 2026-06-12/13 grind -> happened to hit. Bucket (d) ("java.* classes Clojure programs traffic -> in") is the weak point: "traffic in" is an observational predicate, -> which is exactly the treadmill D-406 exists to stop, and the -> bucket-shaped rule gives the next grind session no decision *procedure* -> for a name outside the examples. -> -> **Alt 1 — smallest-diff: no new ADR; amend ADR-0013 + add a `frontier:` -> field to compat_tiers.yaml / host_interfaces.yaml rows.** Each -> classification lands as data on the existing SSOTs; ADR-0013 gains a -> short "frontier" amendment; no ROADMAP §6.5. -> *Better than the draft:* zero new documentation surface; keeps §6.0's -> "the YAML carries the classification, the narrative documents the -> framework" division intact; every classification is already mechanically -> checkable by the existing G3/G4 gates. -> *Breaks:* the **rules themselves have no normative home** — the line -> stays emergent from row-by-row precedent, which is the exact "implicit -> (reactive marker-by-marker)" state D-406 names as the debt. Convergence -> is asserted, not derivable. Also leaves Java-library deps homeless: -> joda-time has no compat_tiers row to annotate. Rejected on F-002: it is -> the smallest-diff convenience the finished-form owner would unwind. -> -> **Alt 2 — finished-form-clean (recommended): replace the nine buckets -> with a 4-step ordered DECISION PROCEDURE, each leaf bound to a -> closed-set SSOT + gate.** [R1 artifact / R2 polymorphism-seam / R3 -> value-semantics / R4 default-deny, as adopted above.] -> *Better than the draft:* definition-derived in the F-013 sense -> (predicates, not observed buckets); the next grind name classifies by -> running the procedure instead of pattern-matching nine examples; -> default-deny makes convergence structural rather than aspirational; -> bounds bucket (d) — a java.* class enters only by passing R3, killing -> the "traffic in" treadmill. -> *Breaks:* substantially larger landing (schema change on two YAMLs + -> gate script + ROADMAP §6.5 rewrite around the procedure); R3 has genuine -> hard cases (Murmur3 is value-semantic *and* a JVM hashing implementation -> detail — the procedure needs an explicit tiebreak: R2/R3 win over R4 -> when behaviour is clj-oracle-checkable per F-011); risk of -> over-formalising leaves that only ever hold one row. Recommended anyway -> per F-002 — the diff size is not a constraint, and this is the shape -> under which the draft's own buckets become regression examples instead -> of the law. -> -> **Alt 3 — wildcard: derive SIDE 1 mechanically from upstream Clojure's -> own .clj corpus.** A host name is a language feature iff it appears in -> `~/Documents/OSS/clojure/src/clj/**/*.clj`; ship the scan as a script -> regenerating a frozen `frontier_corpus.yaml`. -> *Better than the draft:* the most literally definition-derived option — -> zero human bucket-judgement, re-derivable by anyone, auto-answers future -> names. -> *Breaks:* upstream core.clj freely uses `clojure.lang.Compiler`, -> `RT/classForName`, gen-class — the raw scan over-includes precisely the -> Tier D side, so an exclusion list re-imports all the judgement it -> claimed to remove; it under-includes names third-party libs call that -> core.clj never does; and it couples cljw's *language* definition to JVM -> Clojure's implementation file layout — tension with ADR-0059's spirit. -> Worth keeping as a *cross-check input* to Alt 2's R2/R3, not as the -> rule. -> -> **Recommendation: Alt 2**, absorbing the draft's buckets as the -> worked-example table and Alt 3's corpus scan as a non-normative evidence -> column. - -The main loop adopted Alt 2 verbatim (finished-form first per F-002), with -Alt 3 admitted as evidence-only and the draft's buckets demoted to §6.5's -worked examples. - -## Worked examples (the campaign's classifications under the rules) - -| Name hit | Rule | Side | Landing | -|---------------------------------------------|---------------|------------------|------------------------------------------------------| -| deftype declares `clojure.lang.IKVReduce` | R2 | language feature | host_interfaces.yaml + reduce-kv consult (D-400) | -| `(instance? clojure.lang.IPersistentMap x)` | R2 | language feature | declared-interface class facet (2026-06-13) | -| `clojure.lang.Murmur3` static hash | R3 (tiebreak) | language feature | clj-oracle-checkable value fn (ADR-0108 am1) | -| `(.indexOf [1 2] 2)` (java.util.List view) | R3 | language feature | value-search trio on native colls (2026-06-13) | -| `java.io.StringWriter` | R3 | language feature | host_instance container surface (2026-06-13) | -| bouncer → clj-time → joda-time | R1 | Tier D | lib dropped from the conformance seed; doc'd example | -| `gen-class` / `definterface` bytecode | R1 | Tier D | per-form catalogue Codes (ADR-0018) | -| `clojure.lang.Compiler/eval` | R4 | Tier D | default-deny | -| `(.getConstructor c)` reflection | R4 | Tier D | default-deny | - -## Consequences - -- The compat grind has a finite goal: cover the R2/R3 closed sets - definition-wide; refuse R1/R4 with catalogued errors. "Compatible" - becomes checkable. -- New-name classification is procedural; ADR amendments are needed only - to *move* a name across the line (per §6.3's promotion rule), not to - classify one. -- The `frontier_rule:` schema + gate retrofit is the mechanisation half - (framework_completion discipline) and lands in the same cycle as this - ADR. - -## Affected files - -- `.dev/ROADMAP.md` — new §6.5 (the procedure + worked examples). -- `host_interfaces.yaml` — header declares the file as the R2 closed set. -- `compat_tiers.yaml` — header declares host_classes R3-admitted + the - Tier-D `frontier_rule:` schema mandate. -- `.dev/debt.yaml` — D-406 discharge. diff --git a/.dev/decisions/0137_scope_goal_line.md b/.dev/decisions/0137_scope_goal_line.md deleted file mode 100644 index 00ccb143a..000000000 --- a/.dev/decisions/0137_scope_goal_line.md +++ /dev/null @@ -1,206 +0,0 @@ -# ADR-0137 — Scope goal line: Java surface boundary, per-class completeness, pure-leaning library re-selection, cljw.* differentiator surface - -**Status**: Proposed → Accepted (2026-06-14). User-directed (chat 2026-06-14); -operationalises the new **F-014** (project_facts). Complements F-013 (how to -respond to a discovered gap: comprehensive + canonical) by fixing **where the -boundary is**. - -> **Renumbered 0136 → 0137 (2026-06-14)**: this ADR was issued the day after -> ADR-0136 (host-frontier boundary, 2026-06-13) but collided on the same number. -> Per the time-ordered max+1 rule (CLAUDE.md), the earlier host-frontier ADR -> keeps 0136 and this scope ADR becomes 0137. The two are siblings: 0136 is the -> per-method admit/deny decision procedure; 0137 (this) is the per-class -> completeness + library + cljw.* goal line built on top of it. - -## Context - -cljw is a Clojure runtime in Zig, **not a JVM/Java reimplementation** (ADR-0059). -Library-driven discovery (F-013) keeps surfacing Java/`clojure.lang` surface to -add. Without an explicit boundary the loop drifts two ways: (a) chasing one -library's deep internals (instaparse GLL, D-430) past the point of general value, -and (b) leaving **partial classes** — a class with some methods wired and a -sibling method missing, which is a worse trap than an absent class (the user hits -a missing method on a class they thought worked). The user asked to "draw the -goal line" across three axes and to make touched classes complete. - -## Decision - -### Axis 1 — Java surface: "linguistically general" in, JVM-platform artifacts out - -A Java class is **in scope** iff it is *linguistically general* — it models a -concept any language runtime needs, or that idiomatic Clojure code uses -pervasively — NOT a JVM-platform artifact. **Made decidable (DA sharpening c)**: -"linguistically general" is anchored to the same `clojure-corpus` frequency data -(`00a_frequency_overview.md`) Axis 2 uses — a class is in-scope iff it clears a -usage-frequency threshold in real Clojure AND is not a D-425 R1 (Java-artifact) / -R4 (impl-leak) disposition. Axis-1 in/out becomes a data lookup, not a per-class -debate, sharing one source with Axis-2's method corpora. The D-425 six-model tree -stays; the goal line per disposition: - -- **IN — full surface (host_instance / native-collapse / stateless wrapper)**: - String, the numeric tower (Long/Integer/Short/Byte/Double/Float/Boolean/ - Character/BigInteger/BigDecimal/Ratio — F-005), Math, System, StringBuilder, - Object; `java.util` containers as host_instance (ArrayList, HashMap, LinkedList - =ArrayList alias, HashSet); regex Pattern/Matcher; `java.time` common set - (Instant/Duration/LocalDate(Time)/ZonedDateTime) + Date (#inst); Random (exact - LCG); UUID; MessageDigest; **string/byte IO** (StringReader/PushbackReader/ - StringWriter/ByteArray streams + the `*in*` reader subsystem, D-414); the common - Throwable hierarchy (catch vocabulary); Thread (currentThread/sleep/getName) + - the atomics Clojure actually uses. -- **OPAQUE — recognition only (libs load, no instances)**: boxed-numeric - supertypes, `java.sql.*`, `TimeUnit`, `*-TYPE` markers, `java.nio.ByteBuffer` - (deferred), the `java.util.{Map,List,Set}` interfaces for `instance?`. -- **OUT — Tier D (explicit per-form error)**: gen-class / gen-interface / - proxy(deep) / compile / bean(deep), deep reflection, JDBC (`java.sql` impl), - Jackson, AWT/Swing, `java.nio` channels, `javax.xml` StAX impl, real OS threads - as a Java API. - -### Axis 2 — Per-class completeness (the user's central invariant → F-014) - -> **Any Java class that exposes ANY surface method MUST be complete for its -> *commonly-used surface* — no intentional partial omission within an in-scope -> class.** Partial classes are forbidden: finish the class's common surface in -> the same campaign that touches it. - -- "Commonly-used surface" = the methods idiomatic Clojure / real libraries call, - NOT every JVM overload. Sibling methods ship together (e.g. `.subSequence` - rides with `.substring`; never deferred to "a real consumer"). -- This sharpens F-013's 網羅 from "the whole capability class" to "the whole - *touched class's* common surface." - -**The completeness MECHANISM (chosen = DA Alt 2 — oracle-derived per-class corpus).** -The DA fork found that the SSOT this ADR first pointed at does NOT exist: -`compat_tiers.yaml` carries `methods:` for only 28/82 host_classes and has NONE -for `java.lang.String` (42 methods wired in `String.zig`) or `Object`, and where -`methods:` exists it is a flat *wired* list with no intended/complete semantics — -so "wired ⊆ intended" had nothing to diff against (a vibe, not a gate). Per F-013 -clause 3 (closed structurally, not by vigilance) + F-011 (clj equivalence), the -mechanism is therefore a **definition-derived per-class method corpus gated -against the `clj` oracle**: - - For each in-scope class, build `test/diff/class_corpus/.txt` — one - expr per common method (`(.substring "hello" 1 3)`, `(.subSequence "hello" 1 3)`, - …), enumerated from the JVM class's public-instance surface **filtered by the - `clojure-corpus` frequency data** (`00a_frequency_overview.md`) to the - methods real Clojure actually calls. - - Run through `scripts/clj_diff_sweep.sh`; `check_corpus_regression.sh` (already - in the gate) re-runs them and fails on any DIFF. A method clj answers and - cljw does not = a per-class completeness gap, caught **even when the author - under-scoped** (the oracle knows the methods a hand-list would forget — the - failure mode Alt 1's wired==declared check misses). - - A method deliberately outside the common surface is an explicit - OPAQUE/Tier-D line, never silent absence. - - `compat_tiers.yaml` `methods:` becomes a generated index derived from the - passing corpus, not a hand-maintained list. - -**Mechanism LANDED (2026-06-14, D-431 first cut).** The infra is wired and the -two highest-frequency bare classes are closed: - - `scripts/clj_diff_sweep.sh` gained `--class-corpus ` (writes - `test/diff/class_corpus/.txt`); `scripts/check_corpus_regression.sh` - now scans `clj_corpus/` AND `class_corpus/`, so the per-class corpus is gated - by the existing smoke step (`corpus_regression`) — no new gate wiring. - - `String.txt` (44 goldens) + `Object.txt` (15 goldens, universal methods) - landed. Building String's corpus surfaced real completeness gaps the JVM - surface had and cljw lacked — `.indexOf`/`.lastIndexOf` fromIndex+codepoint - overloads, `.stripLeading`/`.stripTrailing`, `.compareToIgnoreCase`, - `.intern` — all fixed in the same cycle (the oracle-catches-under-scoping - property in action). Accepted-divergence methods (`.hashCode` AD-009, - `.getClass` AD-003, nil-receiver) are correctly OUT of the corpus (locked by - their AD pins instead). - - REMAINING (D-431 continues): the ~18 other in-scope bare classes - (java.util containers / Pattern-Matcher / throwable family), the mechanical - stop-chasing rule (sharpening b), the frequency-anchored Axis-1 list - (sharpening c), and the generated `compat_tiers.yaml` `methods:` index. - -### Axis 3 — Clojure libraries: pure-leaning re-selection - -The verification engine is `verified_projects/` (committed `-M:verify` proofs). -Goal line for the candidate ladder (`docs/works/ladder.md`): - -- **Target pure-degree ≤ 3.** Re-select TOWARD pure famous libs (data.* family, - spec/malli, test.check, tools.* where pure, routing-core, utility cljc libs). -- **Re-select AWAY from Java-tight-coupled famous libs** — cheshire→Jackson, - next.jdbc→JDBC, clj-time→joda, core.async→threads/executors. Keep them as - boundary markers; do not chase. -- **Stop-chasing rule (made mechanical, DA sharpening b)**: a blocker is "general" - iff its failing surface has a home in a SHARED class corpus (a `class_corpus/` - line — advances all libs); it is "library-specific deep internal" iff the - failing surface is a library-private construct with no class-corpus home - (instaparse GLL, D-430). Corpus membership IS the general-vs-specific decision — - computed, not re-litigated per library. Pursue the former; file + skip the latter. - -### Axis 4 — cljw.* original surface = the differentiator, not JVM conveniences - -`cljw.*` (ADR-0029) carries **what makes cljw worth using over the JVM**, never -re-implements JVM conveniences (those live under the java/clojure compat surface): -fast cold-start CLI, Wasm components, edge/serverless runtime, single-binary -build. Current: `cljw.wasm` / `cljw.http` / `cljw.eval`. Roadmap ideas (not -commitments): `cljw.wasm.*` (require-as-namespace + dropResource — the real -differentiator), `cljw.edge.*` (request/response edge runtime), `cljw.build` -(AOT single binary), `cljw.repl`/`cljw.nrepl` (dev surface), `cljw.os`/`cljw.proc` -(sandboxed process/env). A `cljw.*` addition must answer "why not just the JVM?". - -## Alternatives considered - -(Devil's-advocate fork, fresh context, 2026-06-14 — reflected faithfully.) - -**Verification finding framing all three**: the draft's central completeness -claim ("compat_tiers.yaml lists each class's intended method set; a wired-but- -incomplete class is a finding") **does not hold** — 82 host_classes, only 28 carry -`methods:`; String (42 wired) and Object carry none; `methods:` is a flat *wired* -list with no intended/complete/partial semantics (grep = 0). So per-class -completeness was a vibe, not a gate; all alternatives must repair this or inherit -the lie (F-013 clause 3: "closed structurally, not by vigilance"). - -- **Alt 1 — smallest-diff**: redefine `methods:` as the intended set, backfill the - ~54 bare classes, gate `wired == declared` (`check_class_completeness.sh`, reuses - the G3 grep infra). *Better*: real PreToolUse gate in one cycle. *Risk*: still - author-judged — catches later *drift* but not initial *under-scoping* (lists 3 - of String's 10 → green but partial); internal consistency, not completeness. -- **Alt 2 — finished-form (RECOMMENDED)**: definition-derived per-class method - corpus enumerated from the JVM public-instance surface filtered by the - `clojure-corpus` frequency data, gated against the `clj` oracle - (`clj_diff_sweep.sh` + `check_corpus_regression.sh`, both already in the gate); - `methods:` becomes a generated index. *Better*: the ONLY option where - under-scoping is caught (the oracle knows the methods the author forgot); - instantiates F-013 (definition-derived 網羅) + F-011 (clj equivalence) + the - corpus-backed-discharge discipline (clj_diff_sweep.md Discipline 1 — same lie - class). Stop-chasing + linguistically-general both get teeth from the shared - corpus/frequency source. *Risk*: larger up-front corpus build (~20 classes); a - global frequency threshold is a one-time judgment (vs per-class re-litigation). -- **Alt 3 — wildcard**: codegen dispatch + `methods:` + corpus skeleton from a - single per-class spec; "partial relative to declaration" becomes a compile - error. *Better*: eliminates wired-vs-declared drift at the representation level. - *Risk*: largest surgery (codegen over all wired classes + build.zig); still - doesn't decide the *common* surface (under-scoping hole relocated); risks the - per-method error-message fidelity String.zig's hand-wired dispatch gives. Must - be paired with Alt 2's oracle to actually close completeness. - -**DA recommendation**: Alt 2, on F-002 (larger build is not a reason to downgrade) -+ F-013 clause 3 (structural, not vigilance, *requires* a definition-derived -check — only Alt 2 provides) + F-011 grounds. No alternative violates an F-NNN. - -**Main-loop decision**: ADOPT Alt 2 as Axis-2's mechanism + all three sharpenings -(a corrected text above; b mechanical stop-chasing; c frequency-anchored Axis-1). -The corpus build is a follow-up campaign (this ADR sets the decision; it does not -require the ~20 corpora to land in this commit). - -## Consequences - -- The loop has an explicit in/out boundary; "should I add class X?" is answered by - the linguistically-general test + the 6-model tree, not case-by-case drift. -- Per-class completeness becomes a *real* gate (not the draft's fiction): the - per-class clj-oracle method corpus (Axis-2 mechanism) fails the commit on any - missing common method — catching author under-scoping, not just drift. Follow-up - campaign: build the ~20 in-scope class corpora + the frequency-anchored in/out - list; until then, the rule is honoured by hand-discipline (sibling methods - ship together) with the corpus as the converging structural backstop. -- Library work has a stop-chasing rule, protecting the perf/Wasm differentiator - from breadth-grind (the Micro-coverage-grind smell, principle.md). -- `cljw.*` growth is purpose-gated. - -## Affected files - -- `.dev/project_facts.md` (F-014, the user-declared invariant). -- `compat_tiers.yaml` (per-class intended surface = the completeness SSOT). -- `docs/works/ladder.md` (re-curated per the pure-leaning criteria). -- `.claude/rules/feature_name_consistency.md` / D-425 taxonomy (the 6-model tree). diff --git a/.dev/decisions/0138_out_in_writer_reader_value.md b/.dev/decisions/0138_out_in_writer_reader_value.md deleted file mode 100644 index e81e88313..000000000 --- a/.dev/decisions/0138_out_in_writer_reader_value.md +++ /dev/null @@ -1,253 +0,0 @@ -# ADR-0138 — `*out*`/`*err*`/`*in*` as first-class cljw writer/reader values - -- Status: Proposed → Accepted (2026-06-14) -- Deciders: autonomous loop (Track C, sweep_plan.md §1) -- Supersedes: ADR-0036's `out_writer_method` sentinel fallback (D-434) -- Relates: F-004 (host_instance writer home), F-009 (impl neutrality), - ADR-0059 / AD-003 (no-JVM, simple class name), F-011 (behavioural - equivalence), F-002 (finished-form wins), ADR-0126 (host_stream), - ADR-0127 (print-method writer handle), D-414 (`*in*` reader subsystem), - D-435/D-436 (大整理 epic) - -## Context - -cljw's `*out*`/`*err*` roots are keyword **sentinels** -(`:clojure.core/stdout` / `:clojure.core/stderr`, core.clj:46-47), not -Writer objects. Three collaborating mechanisms make print + capture work -around that sentinel: - -1. **`out_capture` threadlocal** (core.zig:715, `setOutCapture` 723-727) - — `with-out-str` and nREPL swap a thread-local Allocating sink that - `emitToStdout` (729-771) consults *first*, bypassing the `*out*` Var - entirely. This is a Layer-2 (lang/primitive) threadlocal invisible to - the Layer-1 (eval) interop dispatch — a cross-zone hack. -2. **D-434 `out_writer_method.zig`** — because `(type *out*)` is a - Keyword with no `.write`, a Writer-interop dot-call `(.write *out* s)` - (used by `clojure.data.csv/write-csv`, `clojure.pprint`) has no method - to dispatch. A shared dispatch-fallback (consulted last in BOTH - backends, tree_walk.zig:744 + vm.zig:1302) special-cases the sentinel - and routes `.write`/`.append`/`.flush` through `clojure.core/print`. - Scope: stdout only; `*err*` interop is unrouted. -3. **`emitToStdout` precedence** — capture threadlocal → a non-keyword - `*out*` binding (745-757, the half-built writer-value path that already - calls `writeToWriterValue`) → `rt.stdout`. - -The `*in*` side (D-414, already discharged) is cleaner: `*in*` root is -`nil` (uses the Var, no threadlocal), `with-in-str` binds it to a -`host_stream` reader value, `read-line` derefs it. But the reader lacks -`read-char`/`peek-char`/`unread-char`, and the -`clojure.lang.LispReader$StringReader` shim is a free fn + a -`constructInstance` ctor special-case (special_forms.zig:166-175) rather -than an operation on the reader value. - -The user flagged (D-436 candidate b) that the sentinel-not-a-writer shape -forces every Writer-interop method to be special-cased forever. JVM -Clojure's `*out*` IS a `java.io.Writer`; `with-out-str` is just -`(binding [*out* (StringWriter.)] …)`. The finished form is the same: -make `*out*`/`*err*`/`*in*` bind to **first-class cljw writer/reader -values** so interop dispatch is the primary `lookupMethod` path and -capture is plain Var rebinding. - -## Decision - -Introduce a **durable, host_instance-backed Writer value and Reader -value** (F-004's declared writer home; no new NaN-box tag — slots 0-63 -are full) in a new neutral module **`src/runtime/io/text_io.zig`**, -distinct from `host_stream.zig` (which keeps owning the `clojure.java.io` -file streams `BufferedReader`/`BufferedWriter`/…). - -### Writer value - -A `host_instance` with descriptor `fqcn = "Writer"` (simple name per -AD-003; UTF-8 fixed, no Charset/PrintWriter/BufferedWriter hierarchy — -the mandatory DIVERGENCE from JVM). Backed by a `TextWriterState` -carrying a **mode**: - -- `.stdout` — writes route to `rt.stdout` (preserving the D-096 single - offset-tracking interleave with the runner's result-print); `.flush` - flushes it. Root of `*out*`. -- `.stderr` — writes route to a stderr sink. Root of `*err*` (finally - wiring `*err*`, which the sentinel never did). -- `.string` — owns a `gc.infra` `ArrayList(u8)` accumulator; backs - `with-out-str` and nREPL capture. A `(rt/__writer->str w)` accessor - returns the buffer. - -Methods on the descriptor's static `method_table`: `write` (string OR -int-codepoint→char, the latter folding out_writer_method.zig:62-66), -`append` (writes, returns receiver — chainable), `flush`, `close`. - -### Reader value - -A `host_instance` with descriptor `fqcn = "Reader"`, backed by an -in-memory byte buffer + a read cursor + a 1-slot char pushback. Methods: -`read` (next byte int / -1 — kept for stream compat), `read-char` -(next UTF-8 codepoint as char / -1), `peek-char` (no advance), -`unread-char` (1-slot pushback), `readLine`, `close`. The -`lispStringReader` D-414 shim folds in as a reader operation. `*in*` root -stays `nil` (no process-stdin reader yet); `with-in-str` mints a -`.string` reader. - -### Wiring changes - -- core.clj: `(def ^:dynamic *out* (rt/__stdout-writer))` / - `*err* (rt/__stderr-writer)`; `*in*` stays `nil`. -- `emitToStdout` collapses to: render args into a scratch buffer, then - `writeToWriterValue` on the bound `*out*`. The capture branch and the - sentinel-fallthrough both disappear. -- `with-out-str` becomes `(binding [*out* (rt/__string-writer)] body - (rt/__writer->str *out*))` — pure Var rebinding. -- nREPL capture (nrepl.zig:308-322): rebind `*out*` to a string writer - via a binding frame for the eval, read the buffer after, instead of - `setOutCapture`. -- **Delete**: `out_capture` threadlocal + `setOutCapture` (core.zig:715-727), - `out_writer_method.zig` (whole file) + its two consult sites, the - `:clojure.core/stdout`/`stderr` sentinel + `isStdoutSentinel`. - -### Coexistence with writer_value.zig (print-method handle) - -`runtime/writer_value.zig` (ADR-0127) wraps a **borrowed, single-print-scoped** -`*std.Io.Writer` for `(defmethod print-method T [o w] …)`. It models a -different lifetime (borrowed vs owned) and stays as-is; both carry -fqcn "Writer" (the class name is cosmetic, dispatch is per-value -`method_table`). Folding the two is a possible later cleanup, not part of -this ADR's finished form (the lifetimes are genuinely distinct). - -## Consequences - -- `(type *out*)` ⇒ `Writer` (was `Keyword`); `(.write *out* s)` / - `.append` / `.flush` dispatch on the value's own method_table (primary - path) — no fallback. `*err*` interop now works. -- D-436(b) discharged; D-434 superseded (file deleted); D-414 reader - shims folded. D-435 (diff-oracle bootstrap gap) is independent and - stays on the D-436 epic — but a writer VALUE is directly unit-testable. -- One behavioural divergence (no Charset / no PrintWriter auto-flush - nuance) derives from no-JVM (ADR-0059) → an accepted divergence, not a - gap. -- Depth-3/4 surgery: roots, print pipeline, both backend dispatch chains, - a deleted file. Landed across 3 TDD commits (writer foundation → root - flip + deletions → reader value). - -## Build order - -1. **Writer foundation** (additive, no root flip): `text_io.zig` Writer - value + `.stdout`/`.stderr`/`.string` modes + `__stdout-writer`/ - `__stderr-writer`/`__string-writer`/`__writer->str` prims + the - write/append/flush/close method_table. Unit tests. `writeToWriterValue` - already dispatches `.write`, so this rides the existing seed. -2. **Root flip + deletions**: flip `*out*`/`*err*` roots; rewrite - `emitToStdout` to uniform `writeToWriterValue`; rewrite `with-out-str` - over `binding`; rewire nREPL; delete `out_capture`/`setOutCapture`/ - `out_writer_method.zig` + consult sites + the sentinel. e2e: - print/println/with-out-str + `(.write *out* s)` + data.csv write-csv. -3. **Reader value**: `read-char`/`peek-char`/`unread-char` (UTF-8, 1-slot - pushback); fold `lispStringReader`. e2e: with-in-str + read-line + - instaparse safe-read-string corpus still green. - -## Alternatives considered - -(Devil's-advocate fork, fresh context, F-NNN envelope: F-002 finished-form-wins, -F-004 host_instance reuse / no new NaN-box tag, F-009 impl/surface/peer split, -F-011 behavioural equivalence, ADR-0059/AD-003 no-JVM simple class name.) - -### Alt 1 — Smallest-diff: extend `host_stream.zig`'s StreamState with `.stdout`/`.stderr`/`.string` writer modes; no new module - -Reuse the existing `StreamState` (host_stream.zig:52-65) as the single -writer/reader carrier; add the string sink mode + `append`/int-codepoint + -`read-char`/`peek-char`/`unread-char` to the existing method tables; root flip + -emitToStdout rewrite + nREPL rewire + deletions identical to the draft. - -- **Better**: F-009-honest with zero structural churn — host_stream already owns - the gc.infra accumulator, cursor, finaliser, `__string-reader`, and - `lispStringReader`; the reader half collapses to "add three methods." -- **Breaks**: `stdout`/`stderr` don't fit StreamState's flush-to-`dest`-file - contract (host_stream.zig:163-171 writes the whole accumulator to a file each - flush — wrong for a live stream; it must write-through `rt.stdout` for the - D-096 interleave). Co-locating a write-through live stream and a - buffer-to-disk file sink in one method_table means every method grows a mode - branch. Smallest-diff bias: overloads a type whose invariant is "I am a file - buffer." - -### Alt 2 — Finished-form-clean (DA-RECOMMENDED): the draft's `text_io.zig`, but fold host_stream's file streams onto the same carrier so the durable text reader/writer is defined exactly once - -Keep `text_io.zig` owning the durable Writer/Reader value; additionally -re-express host_stream's file streams as `text_io`'s reader+writer *plus a file -dest* (`mode ∈ {stdout, stderr, string, file}`), so `lispStringReader` + -`read-char` family live once; host_stream shrinks to the `clojure.java.io` -open/copy/slurp primitives over the one carrier. - -- **Better**: removes the draft's self-deferred two-carrier coexistence - (TextWriterState beside StreamState, overlapping `write`/`flush`/`close`/ - `read`/`readLine`) — claimed Reservation-as-bias / Smallest-diff smell under - F-002. Claims a latent F-011 gap: a file `reader` would lack - `read-char`/`unread-char` that `*in*` has. One finaliser, one - read-char definition. -- **Breaks**: rewrites the landed ADR-0126 host_stream subsystem (its - `stream_classes` SSOT, concrete-class `(class …)` faithfulness, protocol_impls - chains); per-mode fqcn nuance (one struct, four descriptors — simple - "Reader"/"Writer" for `*in*`/`*out*`, concrete "BufferedReader"/"BufferedWriter" - for files) must be preserved or `(class …)` regresses. - -### Alt 3 — Wildcard: keep the sentinel root, lazily realise + cache a text_io value on first deref - -Same text_io value type; `*out*` root stays the keyword sentinel, realised to a -process writer value on first `print`/`read` and cached into the Var root. - -- **Better**: sidesteps the bootstrap-ordering question (no host_instance minted - at core.clj:46); shrinks the root-flip blast radius. -- **Breaks**: permanent-no-op / silent-default-shift smell in a feature costume — - KEEPS the sentinel (the exact thing D-436(b)/D-434 delete), relocates - `isStdoutSentinel` to the hot deref path, makes `(type *out*)` only - *eventually* a Writer (F-011 violation before first print), and resurrects the - realise-or-fallback branch out_writer_method.zig embodies today. Rejected. - -### Three correctness cautions (load-bearing regardless of shape — adopted) - -1. **`.stdout` write-through, not buffer-to-disk.** `.stdout`/`.stderr` `write` - calls `rt.stdout.writeAll`, `flush` flushes `rt.stdout` (runtime.zig:101) — - NOT host_stream's accumulate-then-`file_io.writeAll`. This is the load-bearing - reason `.stdout` is a distinct mode; preserves the D-096 offset-tracking - interleave + per-call write semantics exactly. -2. **nREPL rewire needs no new runtime helper, per-thread isolation preserved.** - `current_frame` (env.zig:249) is already `threadlocal`, so a - `binding [*out* string-writer]` frame is per-connection-thread exactly as - `out_capture` was. nrepl.zig (Layer 3) may push a BindingFrame directly - (zone-legal); save/restore maps 1:1 to push/defer-pop. An optional - `rt`-level `withOutCaptured(thunk)` convenience reads cleaner but is not - required for correctness. -3. **`unread-char` is codepoint-aware**, not a byte decrement: a dedicated - `pushback: ?u21` field consulted first by `read-char`/`peek-char` sidesteps - multi-byte codepoint corruption. - -### Main-loop decision vs the DA recommendation (non-binding) - -**The main loop keeps the draft's two-value-type separation (text_io for -`*out*`/`*err*`/`*in*`; host_stream unchanged for file streams) and REJECTS -Alt 2's carrier fold — on finished-form + F-011 grounds, NOT diff budget.** - -Reasoning: - -- **Alt 2's headline F-011 argument is backwards.** JVM Clojure's `*in*` is a - `LineNumberingPushbackReader` (has `unread`/pushback); `(reader "f.txt")` is a - `BufferedReader` (NO pushback). They are genuinely different class families - with different method sets. Giving a file reader `unread-char` would make cljw - *less* JVM-faithful, not more — so there is no parity gap to close, and the - draft's separation mirrors the JVM class split correctly (simple "Reader" for - `*in*` with pushback, concrete "BufferedReader" for files without). -- **The invariants are genuinely different**, not duplicated: write-through live - stream (stdout/stderr) vs buffer-to-disk file vs in-memory string. Folding - forces a 4-way `mode` branch inside every `write`/`flush`/`close` — the exact - per-method mode-branch the DA used to reject Alt 1, made worse. Separate value - types with single-purpose methods is the cleaner finished form here. -- **The DA's *valid* residual concern is mechanism duplication** (byte buffer + - read cursor + `readLine` + UTF-8 `read-char`). The finished-form answer is a - **shared low-level helper** (a `ByteCursor`-style struct: buffer + pos + - `?u21` pushback + readLine/read-char), composed by both `text_io` and - (optionally) `host_stream` — sharing the *mechanism* without conflating the - *value types / descriptors / fqcn / flush semantics*. Whether host_stream - adopts the shared helper is decided at build-step 3 (where pushback logic - lands); it is a behaviour-preserving refactor, not a value-type merge. - -This is not the Cycle-budget defer smell: Alt 2 is declined because its value-type -fold is *less* finished-form-clean (less JVM-faithful, more per-method branching), -and the duplication it correctly identifies is addressed by a shared helper. All -three DA correctness cautions are adopted verbatim. diff --git a/.dev/decisions/0139_seq_as_map_key_content_hash.md b/.dev/decisions/0139_seq_as_map_key_content_hash.md deleted file mode 100644 index 07bf4a5b4..000000000 --- a/.dev/decisions/0139_seq_as_map_key_content_hash.md +++ /dev/null @@ -1,138 +0,0 @@ -# ADR-0139 — seq / lazy / Sequential-instance as a map/set KEY hashes by content - -- Status: Proposed → Accepted (2026-06-14) -- Deciders: autonomous loop (Track D D1, sweep_plan.md § Track D) -- Extends: ADR-0129 (ambient `dispatch.current_env` for the rt-free key - path) — broadens its arming + its rt-aware hash/eq to sequential keys -- Relates: ADR-0056 (`runEnvelope` AOT driver — gains the same arming), - D-427 (Sequential deftype `=` element-wise), D-092 (rt-free key - hash/eq), F-011 (behavioural equivalence), F-002 (finished-form wins), - F-004/F-006 (NaN-box / GC — the deferred Alt-2 owner) -- Discharges: D-432, D-408 -- Opens: D-437 (nested seq/lazy key + rt-free memoized seq-key hash — - the deferred finished form) - -## Context - -A content-equal `lazy_seq` / `cons`-over-lazy / `range` / Sequential- -declaring `deftype`/`reify` used as a **map/set KEY** hashed by IDENTITY, -not by element content. So `(get {(map inc [0 1 2]) :x} '(1 2 3))` → cljw -`nil` where clj returns `:x`, even though `(= '(1 2 3) (map inc [0 1 2]))` -is already `true` (D-427). The `=` side is rt-threaded (`Cursor` forces -thunks); the KEY side (`valueHash` → `seqHash` over the rt-free -`SeqKeyCursor`, + `keyEqValue`) had no Runtime, so a lazy/instance key -could not be realized and `SeqKeyCursor` documented it as an identity -residual (truncation fallback). A content-equal key thus landed in a -different HAMT bucket and was never found — a silent miss (the worst -F-011 failure class), not an accepted divergence (clj supports it). - -ADR-0129 (D-377) had ALREADY broken the "key path is rt-free" premise: it -added `hashConsult`/`eqConsult`, which read the ambient -`dispatch.current_env` threadlocal to dispatch a deftype's custom -`hasheq`/`equiv` on the key path. The lazy/range/instance arms simply were -never added to that ambient path — so temporising D-432 further (low -frequency, "rt-threading is broad") was the Reservation-as-bias the -2026-06-14 audit flagged as the #1 finished-form win. - -## Decision - -**Option A**: make the key hash/eq path rt-aware via the EXISTING ADR-0129 -ambient `current_env` — no public signature change. - -- `hashDispatch` (the shared rt-aware hash core, used by both the HAMT - key sites via `hashConsult` AND the `(hash x)` primitive) realizes a - `lazy_seq` / `range` / `chunked_cons` / cons-over-lazy `.list` / - Sequential-instance key to a native list, then hashes it with the - existing rt-free `seqHash`. So `(hash (map inc [0 1 2]))` == - `(hash [1 2 3])` == `(hash '(1 2 3))` — the formula is unchanged; only - the input is realized first (exactly what clj's `LazySeq.hashCode` does). -- `eqConsult` compares two sequential keys element-wise: the rt-free walk - first, then — on a lazy tail that truncates it — realize both and retry - via `seqKeyEq`. This is ahead of a custom `equiv`, mirroring - `valueEqual` routing `isSequential` before `instanceEquiv` (D-427). -- A Sequential deftype/reify hashes **element-wise** (ignoring any custom - `hasheq`), matching its element-wise `=` (D-427) — so the - `=`-implies-equal-hash invariant holds by construction for these types. -- `realizeSeqToList` (new) walks the shared `lazy_seq.seq`/`first`/`rest` - protocol, GC-rooting the accumulator + cursor; Sequential instances - reuse the existing `realizeSequentialInstance`. -- `seqHashChecked` / `seqKeyEqChecked` (new) return null on a lazy-tail - truncation so the rt-aware path can detect "needs realize" without a - separate pre-walk; the old `seqHash`/`seqKeyEq` are thin - `orelse identity` wrappers (the unarmed residual, now unreachable in - real evaluation). - -**Alt-1 (mandatory, part of the fix)**: arm `current_env` in -`driver.runEnvelope`. The DA fork (below) found `runEnvelope` — the second -top-level-form driver, used by `cljw build` AOT artifacts + the -AOT-bootstrap restore — calls `vm.eval` directly WITHOUT arming -`current_env` (only `evalForm` + `treeWalkCall` armed it). Without Alt-1, -an AOT-compiled top-level seq-keyed literal would hash its key by identity -at run and silently miss — and the existing ADR-0129 deftype-custom-hash -key feature had the SAME latent hole. Arming `runEnvelope` (save/set/ -restore, mirroring `evalForm`) closes both. e2e `aot_top_level_seq_key` -guards it. - -The unarmed fallback (identity) is now reachable only before the evaluator -is up (host-init / bootstrap), where no user `assoc` keys a map by a lazy -seq — acceptable. - -## Alternatives considered (Devil's-advocate fork, fresh context) - -The DA fork ran within the F-NNN envelope (F-002/F-009/F-011). Verbatim -summary: - -- **Alt-1 (smallest-diff, ADOPTED alongside A)** — Option A + arm - `runEnvelope`. Better: removes the AOT-envelope silent-miss that plain A - leaves (A *asserts* the unarmed case unreachable; this makes it true). - Breaks: nothing; also fixes ADR-0129's identical AOT hole. The DA - proved the unarmed case **reachable**, not unreachable, via - `runEnvelope` (driver.zig:48 → `vm.eval`, no arming; live for - `bootstrap.loadCoreAot` + `app/builder` `cljw build`). Verified - independently against the source. -- **Alt-2 (finished-form-clean, DEFERRED → D-437)** — give lazy/range/ - instance values a memoized realized-hash so the seq-key path becomes - rt-free permanently, RETIRING the dual ambient mechanism + the nested- - key residual. Better: completeness (no armed/unarmed duality, fixes the - nested map-as-key case uniformly). Breaks: needs a memo slot on the - value + a new GC root for the cached list (F-006 root surface) — - per F-003/F-004/F-006 the rooting/layout decision is the owning - surface's call, not a bug-fix cycle's. NOTE: the DA over-stated this as - needing a "NaN-box layout owner Phase" — re-verified, Alt-2 needs only a - heap-struct field + one `traceLazySeq` root, NOT a NaN-box change, so - D-437 is a standalone quality-loop item, NOT Phase-gated. -- **Alt-3 (wildcard) — REJECTED** — thread `rt` explicitly through - `keyHash`/`get`/`contains` (the survey's Option C). Re-litigates - ADR-0129's settled threadlocal decision for a ~68-site ripple with no - finished-form gain over Alt-2. Cycle-budget is not the reason (F-002); - it is dominated by Alt-2 on cleanliness. - -DA recommendation: "land Option A + Alt-1 now; record Alt-2 as the -finished-form target (D-437)." The main loop adopted it. - -## Consequences - -- `(get/contains?)` + `(hash)` are clj-faithful for lazy/range/cons-over- - lazy/Sequential-instance keys; the `=`-implies-equal-hash invariant - (F-011) holds for them. Corpus `test/diff/clj_corpus/seq_key_hash.txt` - (10 cases) + e2e `phase14_seq_key_hash.sh` (7, incl. the AOT path) lock it. -- A queue key now also content-compares in the armed path (it already - content-HASHED via `valueHash`'s `.persistent_queue => seqHash` arm) — - closes a pre-existing hash/eq inconsistency. -- Hot path preserved: scalar/keyword/vector keys are untouched (the - rt-aware switch only engages for seq/instance tags); a plain list key - takes one extra function indirection (`seqHashChecked`), same walk cost. -- Residual (→ D-437): a lazy/seq key NESTED inside a collection key - (`{{(map inc xs) 1} :outer}`) still hashes the inner lazy element by - identity — `valueHash`/`contentHash`/`seqHash`'s per-element recursion - stays rt-free. Rare; the finished form (Alt-2, rt-free memoized hash) - fixes it uniformly. - -## Affected files - -- `src/runtime/equal.zig` — `realizeSeqToList`, `seqHashChecked`/ - `seqKeyEqChecked`, `realizeKeyForCompare`, `isSeqKeyValue`; rewrote - `hashDispatch` / `hashConsult` / `eqConsult`. -- `src/eval/driver.zig` — `runEnvelope` arms `current_env` (Alt-1). -- `test/e2e/phase14_seq_key_hash.sh` + `test/run_all.sh` registration. -- `test/diff/clj_corpus/seq_key_hash.txt`. diff --git a/.dev/decisions/0140_caught_exception_frame_seq.md b/.dev/decisions/0140_caught_exception_frame_seq.md deleted file mode 100644 index 5923059a0..000000000 --- a/.dev/decisions/0140_caught_exception_frame_seq.md +++ /dev/null @@ -1,117 +0,0 @@ -# ADR-0140 — caught-exception cljw-shaped frame-seq accessor - -- Status: Proposed → Accepted (2026-06-14) -- Deciders: autonomous loop (Track D D2, sweep_plan.md § Track D) -- Relates: ADR-0120 (trace deep-copied onto the ExInfo value), ADR-0059 - (no JVM Throwable / StackTraceElement), AD-024 (user-frames-only), - AD-003 (simple class name), F-002/F-009/F-011, D-273 (clojure.stacktrace) -- Amends: AD-029 (`[no stack trace available]` marker → cljw-native frames; - per-frame FORMAT still diverges from JVM) -- Discharges: D-389 (Throwable->map `:trace`/`:at`) -- Adds: AD-033 (Throwable->map `:trace`/`:at` are cljw maps, not JVM 4-vectors) -- Fixes: the dangling **D-232** cross-reference (AD-029 / D-389 / sweep_plan - named the validation-campaign row D-232 as the frame-accessor owner; that - owner row never existed — this ADR + D-438 are it) - -## Context - -A caught exception in cljw is an ExInfo value (ADR-0059: no JVM Throwable). -The frames ARE on the value — `StackFrame{ns, fn, file, line, column}`, -deep-copied at throw time (ADR-0120), rendered as the live-stderr `Trace:` -section + EDN `:trace`. But there was **no Clojure-level accessor** to pull -frames off a caught `e`, so `clojure.stacktrace/print-stack-trace` degraded to -`[no stack trace available]` (AD-029) and `Throwable->map`'s `:trace`/`:at` -were omitted (D-389 PARTIAL). AD-029 / D-389 / sweep_plan all named **D-232** -as the owner of this surface, but D-232 is the validation campaign — a -dangling reservation (Reservation-as-bias), no real owner row existed. - -## Decision - -Add a cljw-original primitive **`(stack-trace e)`** (`lang/primitive/error.zig`, -auto-referred like `ex-message`/`ex-data`/`ex-cause`) returning a vector of -**`{:ns :fn :file :line :column}` maps**, **innermost-first** (clj's `:trace` -convention; `info.trace` is innermost-last push order, so reverse). `:fn` is the -**bare** fn name, `:ns` separate (the renderer combines `/`). A -non-ex_info OR a never-thrown ex-info (no captured trace) → an **empty vector**. - -Consumers: -- `clojure.stacktrace/print-stack-trace` iterates `(stack-trace tr)` and prints - `/ (:)` per frame; a frame-less exception keeps the AD-029 - marker. New `print-trace-element` formats one frame map. -- `clojure.core/Throwable->map` fills `:trace` (root cause's frames) + per-`:via` - `:at` (each link's top frame) from `(stack-trace …)`, as the SAME maps — closing - D-389. Omitted (absent, never empty) when an exception is frame-less. - -The frame shape is a map, not a JVM `StackTraceElement` 4-vector — class/method -are fictions cljw won't fake (ADR-0059/AD-003); the map keeps `column` (a -4-vector drops it) and matches cljw's native `StackFrame` + SCI's interpreter- -frame map precedent. AD-024 (user-frames-only) holds free (elision at push time). - -## Alternatives considered (Devil's-advocate fork, fresh context) - -The DA fork ran within the F-NNN envelope. Verbatim-digested: - -- **Alt 1 (smallest-diff) — no new var; populate `Throwable->map :trace`/`:at` - as 4-vectors, drive clojure.stacktrace off `(:trace (Throwable->map e))`** (cw - v0's choice). Better: one surface, no new symbol, matches clj's documented - vector shape. Breaks: carrying `ns/fn` in `[class method …]` slots is a - JVM-lie or a mislabel (AD-003 rejects); drops `column`; opaque per-frame - access. REJECTED. -- **Alt 2 (finished-form-clean, ADOPTED) — `(stack-trace e)` → maps, AND - `Throwable->map :trace`/`:at` as the SAME maps** (one shape everywhere). - Better: one shape, no map↔vector projection to sync (Option C's hedge is the - Smallest-diff-bias smell); honest at every surface; keeps `column`; matches - native `StackFrame` + SCI. Breaks: `Throwable->map :trace`/`:at` diverge from - clj's documented vector → ONE recorded AD (AD-033) — consistent with the - already-accepted AD-007 (cljw's whole `#error{…}` trace representation - diverges). No F-NNN violated. -- **Alt 3 (wildcard) — a printer primitive reusing `error/print.zig`'s `Trace:` - renderer; no Clojure data shape; `:trace` stays omitted (permanent AD).** - Better: single trace-format source (live stderr == clojure.stacktrace by - construction). Breaks (decisive): no DATA accessor — blocks datafy + any - programmatic frame consumer (a real pending need); optimizes the print path at - the cost of the data path (violates F-002 cleanest-complete-form). REJECTED. - -DA answers to the sub-questions: -- **Naming**: `(stack-trace e)` — verified clj has NO `stack-trace` core fn (no - collision); `ex-trace` rejected (invents a non-existent `ex-*` clj member); - `cljw.error/frames` rejected (needlessly buried — frames are a first-class - property like message/data/cause). A named accessor IS the finished form (vs - Alt 1's "frames only via Throwable->map") — minimize-surface points the wrong - way here (F-002). -- **Map keys**: `:fn` **bare** ("boom") with `:ns` separate (the native struct - carries them separately; combine at print time) — corrects the survey's - qualified-`:fn` example. `:fn` over `:name` (cljw-native vocabulary; `:name` - noted as the SCI-tooling-interop alternative). -- **AD-029**: **amend, not remove** — the marker is gone for caught exceptions, - but the per-frame FORMAT still diverges (cljw `ns/fn` vs JVM `class.method`, - no demunge; ADR-0059). Removing it would assert non-existent parity. -- **D-389 `:trace`/`:at`**: fill THIS cycle, as the same maps (not deferred, not - projected to vectors) — leaving them split from `(stack-trace e)` would be the - "two ways, different shapes" inconsistency. - -The main loop adopted the DA recommendation verbatim (Alt 2 + the four answers). - -## Consequences - -- `clojure.stacktrace/print-stack-trace`/`print-cause-trace` print real - cljw-native frames for caught exceptions; `Throwable->map` is complete - (`:cause`/`:via`/`:trace`/`:data`, per-via `:at`). datafy's Throwable path is - unblocked on the `:trace` axis (other datafy blockers remain — D-271). -- A frame-less (never-thrown) ex-info keeps the honest marker / omitted keys — - AD-029's honest-degradation invariant survives for that case. -- Divergence recorded: AD-033 (map-shaped `:trace`/`:at`). -- The D-232 dangling reference is fixed; D-438 is the real frame-accessor row - (discharged on landing this cycle). - -## Affected files - -- `src/lang/primitive/error.zig` — `stackTrace` primitive (+ `stack-trace` entry). -- `src/lang/clj/clojure/stacktrace.clj` — `print-trace-element` + frame-printing - `print-stack-trace`; header/ns-doc updated. -- `src/lang/clj/clojure/core.clj` — `Throwable->map` `:trace`/`:at` (PROVISIONAL - marker removed; triad discharged). -- `test/e2e/phase15_clojure_stacktrace.sh` (+frame cases), - `test/e2e/phase14_throwable_map.sh` (+caught `:trace`/`:at` cases). -- `.dev/accepted_divergences.yaml` (AD-029 amend + AD-033), `feature_deps.yaml`, - `.dev/debt.yaml` (D-389 discharge, D-438), `.dev/sweep_plan.md`. diff --git a/.dev/decisions/0141_completion_grade_reframe.md b/.dev/decisions/0141_completion_grade_reframe.md deleted file mode 100644 index 5b3d2d848..000000000 --- a/.dev/decisions/0141_completion_grade_reframe.md +++ /dev/null @@ -1,99 +0,0 @@ -# ADR-0141 — Completion-grade reframe: retire blind Phase-deferral; reorganize the roadmap + debt + AI-instruction system to the accurate current position - -- Status: Proposed → Accepted (2026-06-15) -- Deciders: user directive (chat 2026-06-15) + autonomous loop (recording + plan) -- Operationalises: **F-015** (completion-grade posture — user-declared law) -- Re-sequences: F-010 (milestone M; the post-M quality loop = this posture) -- Opens: **D-440** (the reorganization epic — the work items) -- Relates: F-003 (structural-deferral, now narrowed), F-011 (clj parity → concurrency), - ROADMAP §9 (phase tracker to be rewritten), `.dev/principle.md` (defer discipline) - -## Context - -The project has reached near-completion, but the roadmap + guardrails still encode -an **early-phase, defer-to-future-Phase posture** that no longer matches reality: - -- **Concurrency is BUILT** (2026-06-15 assessment): `atom` CAS (4×100 concurrent - swaps → 400), `future`/`promise`/`deliver`, `ref`+`dosync`+`alter`, `agent`+`send` - +`await`, `pmap`, `delay`, `locking` all run correctly. Yet it is officially - "Phase 15 (PENDING)" + "Phase B hardening deferred" — so its tests, official clj - parity, and load/stress cases are thin (the user's concern: "公式にそのPhaseに - 先送り…されると手薄になりがち"). -- **19+ active debt rows** carry a future-Phase gate ("Phase 15-19" / "Phase B" / - "blocked-by" / "GATED on Phase"). Many barriers have dissolved (the thing is - built); the gate is now framing, not a real blocker. -- ROADMAP §9 lists Phases 15-20 as `PENDING, expand at Phase N entry` — a - from-scratch framing for areas that are largely already present. - -The reflexive "defer to a future Phase" move (which this session itself kept -hitting — `seque`, `:volatile-mutable` visibility, concurrency hardening — all -waved off as "Phase 15 territory") is the symptom. Per F-015 it is retired. - -## Decision - -Adopt the **completion-grade posture** (F-015) and execute a **reorganization arc** -(D-440): - -1. **Phases 15-20 reframed** from "future work" to **gap areas to bring to - completion grade**. The default is to assess "is this already built?" and, if - so, strengthen it (tests / clj parity / load) NOW — not to defer. F-003 - (structural-decision deferral) is **narrowed** to genuinely-unbuilt - layout/representation decisions; it is not a licence to park built-but-unhardened - behaviour behind a Phase number. -2. **Concurrency completion-grade pass FIRST** (highest-value, the user's lead - example): a concurrency test layer, clj-parity verification of the concurrency - vars (atom/ref/STM/agent/future/promise/delay/locking/pmap), and load/stress - cases. D-242/D-244/D-245 ("Phase B" hardening / GC-safety / locking-parking) are - un-deferred; their barriers re-evaluated as "do now". -3. **Roadmap rewrite**: ROADMAP §9 phase tracker + the PENDING-Phase sections - rewritten to the accurate current position; the old numbering is an input, not a - constraint (F-015 cl.4). -4. **Debt整理**: re-evaluate every Phase-gated debt row (Step 0.5 sweep, but - exhaustive) — flip the ones whose barrier dissolved; reclassify the rest with an - honest, non-Phase-number barrier. -5. **AI-instruction 大整理**: principle.md (the structural-imagination / defer - discipline), CLAUDE.md (phase-entry rules, the loop's Phase-boundary chain), and - the gate/guardrails — audited + reorganized so they describe a near-complete - project, not an early-phase one. - -The work items + sequencing live in **D-440** (the epic). This ADR records the -reframe decision; D-440 is the execution ledger. - -## Alternatives considered - -This is a **user-directed F-015-level posture change** (chat 2026-06-15), not a -loop-autonomous design choice, so the mandatory Devil's-advocate fork (for -loop-self-decided ADRs) does not gate it. The decision space the user left open is -*how* to reorganize, not *whether*; the options weighed: - -- **(A, chosen) Reframe in place + a scheduled reorganization arc (D-440).** Keep - the F-NNN/ADR/debt machinery; rewrite the phase model + drain the Phase-gated - debt + audit the AI instructions as a tracked epic. Honest, incremental, - reversible per-step. Matches the user's "queue it into your near future". -- **(B, rejected) Abolish the phase model entirely now (flat backlog).** Cleaner - end-state but a big-bang rewrite that throws away the still-useful §9 structure - and the audit trail in one step; higher risk, and the user said "queue it", not - "do it all this instant". -- **(C, rejected) Do nothing structural; just un-defer concurrency.** Fixes the - lead symptom but leaves the stale phase model + 19 Phase-gated rows + the - early-phase guardrails — the user explicitly asked for the roadmap/debt/ - instruction reorganization, not only the concurrency fix. - -## Consequences - -- The loop stops reflexively deferring built behaviour to a future Phase; "is it - already built? then harden it now" becomes the default (F-015). -- Concurrency gets first-class tests + parity + load coverage (un-deferred). -- The roadmap, debt ledger, and AI-instruction system get reorganized to a - completion-grade project (D-440), freeing future work from stale early-phase - framing. -- F-010's milestone M "Phase 15 完遂" collapses into the concurrency completion - pass; the post-M quality loop and this posture are one standing mode. - -## Affected files - -- `.dev/project_facts.md` — F-015 (the user-declared posture). -- `.dev/decisions/0141_*.md` — this ADR. -- `.dev/debt.yaml` — D-440 (the reorganization epic). -- `.dev/sweep_plan.md` + `.dev/handover.md` — D-440 wired as the imminent arc. -- (D-440 execution) ROADMAP §9, principle.md, CLAUDE.md, the Phase-gated debt rows. diff --git a/.dev/decisions/0142_roadmap_completion_grade_reframe.md b/.dev/decisions/0142_roadmap_completion_grade_reframe.md deleted file mode 100644 index e32839d25..000000000 --- a/.dev/decisions/0142_roadmap_completion_grade_reframe.md +++ /dev/null @@ -1,162 +0,0 @@ -# ADR-0142 — ROADMAP §9 completion-grade reframe (Track R R3) - -- **Status**: Accepted (2026-06-15) -- **Context tier**: structural (ROADMAP §9 amendment per §17) -- **Supersedes / amends**: amends ROADMAP §9 in place; executes ADR-0141 / - F-015 / debt D-440 item 3. Does NOT supersede the §9.3-9.16 DONE archive - rows or the §9.2.R/S/P campaign overlays (those are preserved). - -## Context - -F-015 (project_facts) declares the project near-complete and retires the -early-phase "defer-to-future-Phase / expand-at-entry" posture. The R2 -accurate-position survey (`private/notes/p14-r2-accurate-position-survey.md`) -established the ground truth by reading source/tests, not prose: - -- Phases 9-13 are **DONE** (Phase 12 mislabeled "DONE-PARTIAL" — the bytecode - cache is complete); Phase 14 is **done modulo the v0.1.0 tag**. -- Phases 15-20 are **not "future work"** — most is BUILT: concurrency (Phase 15) - is built + race-hardened; Wasm component build/run/require is built (mislocated - across "Phase 16/19"); VM superinstruction/fusion (Phase 17) is partial (a real - slice landed); math + module/deps (Phase 18) are done. -- Genuinely unbuilt: ClojureScript→JS (Phase 16), C FFI (Phase 18), broad JIT - (Phase 20), `future-cancel`/`seque` (D-442), the java.time trio (D-105). - -The §9 tracker still frames the built work as "PENDING, expand at entry" with -stale stub-swap "Final activation step" / `build_options.phase_at_least_N` -sentences for impls that already shipped (STM/locking/agent), and a v0.1.0 -version ladder that no longer matches `build.zig.zon` (`1.0.0-alpha.1`). - -## Decision - -Reframe ROADMAP §9 to the **completion-grade gap-area model** (DA-fork -Alternative 2, executed fully): - -1. **Tracker table** — Phases 9-13 → DONE; Phase 12 DONE (not DONE-PARTIAL); - Phase 14 → "release mechanics" (the single open item is the tag + version - reconcile). Phases 15-20 rows reframed from "PENDING" to their **gap-area - role + maturity** (BUILT / hardening / future). -2. **Gap-area grouping** — the Phase 15-20 work is organized into **three - completion-grade gap areas** + a small **genuinely-future bucket**: - - **(I) Concurrency hardening** — BUILT + race-hardened; gaps = parity/load - (Track R R1, done this session) + D-442 (future-cancel/seque/legacy-agent - surface) + AD-018 (`:volatile-mutable` cross-thread re-eval) + D-105 - (java.time trio). - - **(II) Wasm / edge-native** — component build/run/require BUILT; gap = WIT - marshalling (D-404) + zwasm integration shape (D-036/D-350). - - **(III) VM perf (fusion → JIT)** — a fusion/superinstruction slice landed - (D-386 / O-018/019/021/023); gap = remaining fusion surface + the narrow - ARM64 JIT (F-010 milestone M) → broad JIT go/no-go (distal). - - **Genuinely-future bucket** — ClojureScript→JS, C FFI, broad JIT. -3. **Anchor stability (the hazard mitigation)** — the §9.17-9.22 section IDs are - **preserved** (retitled to their gap-area role, bodies fully rewritten — NOT - a banner-on-skeleton, which is the rejected Alt 1 smell). A **phase-number → - gap-area redirect table** is added at the top of §9. Together these keep every - "Phase N" / §9.N citation in the ~41 ADRs + ~30 debt rows + the §9.2.R/S/P - overlays + `phase_at_least_N` flags resolving, so the collapse does not create - a dead-citation transient window (the DA-fork's #1 hazard). -4. **Drop the stale framing** — the `phase_at_least_N` "Final activation step" - sentences for shipped impls (STM/locking/agent) are deleted; the §9.2.P - "ACTIVE — resume here next session" header (stale — D-164/C1-C7 largely - worked, D-441 discharged this session) is corrected. -5. **Version reconcile** — §9.1/§9.2 note that `1.0.0-alpha.1` is the actual - version string; the v0.1.0/v0.2.0/v0.3.0 ladder is kept-for-history. -6. The deeper citation rewrites (debt rows' "Phase N target" barriers → - gap-area barriers; the AI-instruction phase-entry machinery) land in the - **same D-440 arc** as R4 (debt整理) + R5 (instruction 大整理) — the DA-fork's - required sequencing so the redirect table is a bridge, not a permanent - tombstone. - -F-015 cl.4 frees the numbering ("an input, not a constraint"); F-002 makes -diff size a non-constraint (so Alt 2 over Alt 1 is not a budget call). - -## Alternatives considered - -Verbatim from the mandatory Devil's-advocate fork (fresh context, F-NNN-briefed): - -> **Leading F-NNN finding (no violation):** None of the three alternatives -> violates an F-NNN. F-015 cl.4 frees the numbering, so even the wildcard is -> in-envelope. F-002 makes diff-size a non-constraint. F-010's milestone-M -> references "Phase 15" by name — ADR-0141 already collapsed M's "Phase 15 完遂" -> into the concurrency pass, so a reframe must keep an M-anchor (a named -> milestone, not a number) or it orphans F-010's gate (a breakage to manage, -> not a block). -> -> **Alt 1 — Smallest-diff (status-flip + stale-sentence strike, numbering kept):** -> flip Phase 12/9-13 to DONE, mark 14 done-modulo-tag, prepend a STATUS banner to -> §9.17-9.22 instead of rewriting bodies, delete stale `phase_at_least_N` -> sentences, reconcile version in one place. BETTER: lowest cross-reference blast -> radius — every "Phase 15/16/19" citation still resolves; no ADR Supersedes -> edits; §9.2.R/S/P overlays untouched; reversible. BREAKS: it is exactly the -> posture F-015 retires — a "PENDING" section with an "actually built" banner is -> the literal Smallest-diff-bias smell; the reader still sees the skeleton; the -> banner-vs-body contradiction is worse than either pure state. This is ADR-0141's -> already-rejected option (C) wearing a roadmap hat. REJECTED. -> -> **Alt 2 — Finished-form-clean (3-gap-area reframe, executed fully) [RECOMMENDED]:** -> collapse §9.17-9.22 into three completion-grade gap-area sections (Concurrency -> hardening / Wasm-edge-native / VM-perf fusion→JIT) + a genuinely-future bucket -> (CLJS, C FFI, broad JIT, future-cancel/seque, java.time trio); Phases 9-13 DONE, -> 14 release-mechanics; drop every `phase_at_least_N` activation sentence for -> shipped impls; reconcile version; add a phase-number→gap-area redirect table. -> BETTER than Alt 1: it IS the finished form (F-015 cl.1 completion-grade, F-002 -> cl.2 cleaner-form-wins); the reader sees a near-complete project; it absorbs the -> survey's mislocations (wasm under "Phase 16/19") into capability areas where the -> feature lives, killing "can't trust the number to locate a feature". BREAKS -> (concrete, and the mitigation that makes it the recommendation): (1) ~41 ADRs -> cite "Phase 15/16/17/19/20" by number; (2) ~30 debt rows carry "Phase N -> target/entry"; (3) §9.2.R/S/P overlays say "drawing on §9.19 Phase 17 + §9.22 -> Phase 20"; (4) `build_options.phase_at_least_15/17` + ADR-0023 reference phase -> numbers. MITIGATION (must ship same arc, per F-002 "rework that yields a cleaner -> form is a feature"): the redirect table is necessary but not sufficient — D-440 -> item 4 (debt re-barrier) + item 5 (instruction 大整理) must run in the same arc -> so dangling citations are rewritten at source. If R3 ships the collapse but -> R4/R5 lag, 40+ docs cite dead phase numbers — that transient is the one genuine -> hazard, a sequencing discipline, NOT a reason to downgrade to Alt 1 -> (downgrading on that ground would be the Cycle-budget-defer smell). Also: F-010's -> milestone-M "Phase 15" anchor must be renamed to a milestone token in the same -> edit or F-010's gate orphans. -> -> **Alt 3 — Wildcard (abolish phase-numbering; capability-matrix + milestone-ledger):** -> drop the phase-number axis entirely; replace with (1) a capability matrix -> (capability area × maturity `shipped`/`hardening`/`future` + evidence + -> draining-debt) and (2) a milestone ledger (named dated milestones referencing -> capability rows). BETTER: the TRUE finished form for a near-complete runtime — -> O(1) "is X done?", never rots into PENDING-vs-built contradiction (maturity is a -> first-class column), cleanly separates what (capability) from when (milestone) — -> exactly the F-010 Phase-15-is-both-capability-and-milestone confusion. A -> capability matrix is not the flat-backlog ADR-0141 rejected (it preserves -> structure, drops only the number axis). BREAKS: largest blast radius — every -> phase-number citation becomes meaningless with no 1:1 redirect (one-to-many: -> Phase 16 = wasm + CLJS); the whole `phase_at_least_N` / ADR-0023 mechanism loses -> its referent; CLAUDE.md's phase-entry reading-list machinery assumes a phase -> axis. It is a depth-4 rewrite of the project's organizing metaphor — correct as -> a finished form, but it converts D-440 from "reorganization arc" into -> "re-architect the planning system", exceeding what R3 can land safely before -> R4/R5. -> -> **Recommendation: Alt 2, executed fully, redirect table in R3 + citation -> rewrites in the same D-440 arc (R4 + R5).** Cleanest form reachable within the -> D-440 arc as scoped. Alt 3 is cleaner still (and per F-002 I am obligated to say -> so, not reflexively pick the smaller change) — but it is a *successor epic*, not -> R3: it redefines the planning axis, best opened as its own ADR once Alt 2 has -> drained the phase-number citations to a small set. Record Alt 3 as a forward -> debt row so the finished-form pull is captured, not lost. Alt 1 rejected outright. - -The main loop adopts the recommendation. Alt 3 is recorded as forward debt -**D-443** (capability-matrix + milestone-ledger successor model). - -## Consequences - -- §9 reads as a near-complete project: Phases 9-14 done, three live hardening - gap areas + an honest future bucket, no skeleton residue. -- The phase-number→gap-area redirect table + preserved §9.N anchors mean - existing "Phase N" citations keep resolving through the R4/R5 cleanup. -- **Sequencing obligation**: R4 (debt re-barrier) + R5 (instruction 大整理) run - in the same arc to rewrite "Phase N target" barriers + the phase-entry - machinery to gap-area terms. Until they land, the redirect table is the bridge. -- F-010's milestone-M anchor is named ("M / concurrency-complete + narrow JIT"), - not phase-numbered. -- `build_options.phase_at_least_N` flags are now vestigial (every flag guards - shipped code); their retirement is folded into R5 (not this ADR). -- D-443 captures the Alt-3 capability-matrix model as the natural successor. diff --git a/.dev/decisions/0143_lazyseq_force_thread_safety.md b/.dev/decisions/0143_lazyseq_force_thread_safety.md deleted file mode 100644 index cdbd506f4..000000000 --- a/.dev/decisions/0143_lazyseq_force_thread_safety.md +++ /dev/null @@ -1,213 +0,0 @@ -# ADR-0143 — LazySeq.force thread-safety (inline atomic CAS-claim) - -- **Status**: Accepted (2026-06-15) -- **Context tier**: structural (concurrency primitive; gap area I) -- **Supersedes / amends**: discharges debt **D-046** (the "Phase 15 entry — - STM activation contention bench" barrier is MET — STM/threads are live at - HEAD). Amends `lazy_seq.zig`'s `## Mutex shape` module doc in place. Extends - the race-hardening ADR-0142 credits to Phase 15 concurrency with the - remaining unsynchronized once-only-memo path. - -## Context - -`lazy_seq.zig`'s `force()` realizes a LazySeq's thunk (a 0-arity `Fn`) on -first access, caches it in `realized: Value`, and sets `realized_flag: u8 = 1` -so later calls short-circuit. It shipped with **no synchronization** under a -single-thread invariant. That invariant no longer holds: `future.zig` and -`agent.zig` spawn real `std.Thread` OS threads. Two threads can now race in -`force()`: - -- both observe `realized_flag == 0` and both invoke the thunk — JVM Clojure - guarantees **at-most-once** (observable for a side-effecting - `(lazy-seq (do (effect) …))` body); -- the `realized` / `realized_flag` writes can tear or be observed out of order - (no fence), so a reader can see `flag == 1` with a stale `realized`. - -`seq`/`first`/`rest` all loop through `force`, so **every** lazy-seq element -walk crosses this path — and LazySeq is the highest-cardinality heap object in -the language (one per `map`/`filter`/`range`/`iterate`/`cons`-over-lazy step). -This is exactly the workload (a `future`/`pmap` over a shared lazy-seq) that -gap area I (concurrency-hardening) exists to harden, so "document the footgun" -is no longer acceptable (F-015). - -## Decision - -Make `force()` thread-safe with an **inline lock-free double-checked atomic -flag + CAS-claim**, matching the established cljw lock-free concurrency idiom -(atom/volatile/ref-doGet/safepoint, D-246) — NOT the off-heap `Io.Mutex` cell -that delay/promise/future use. - -The existing `realized_flag: u8` becomes a 3-state atomic word (accessed via -the `@atomicLoad`/`@cmpxchgStrong`/`@atomicStore` builtins on the plain field, -the atom.zig idiom — **no `std.atomic.Value` wrapper, no new field, no struct -growth, no off-heap allocation, no GC finaliser**): - -- `PENDING = 0`, `REALISED = 1` (unchanged — `isRealised`/`(realized? ls)` and - the pre-cache test semantics are preserved), `CLAIMING = 2`. -- **Fast-path read** (lock-free, the steady state): `if - (@atomicLoad(u8, &ls.realized_flag, .acquire) == REALISED) return - ls.realized;`. A realised-seq walk pays one acquire-load — clj's actual - shape (clj nulls a `volatile Lock` so steady-state reads are lock-free). -- **Claim**: `@cmpxchgStrong(u8, &ls.realized_flag, PENDING, CLAIMING, - .acq_rel, .acquire)`. The single winner runs the thunk, writes `realized`, - then publishes `@atomicStore(u8, &ls.realized_flag, REALISED, .release)`. - The release/acquire pair on the flag fences the plain `realized` write - (clj's volatile-lock trick). At-most-once is preserved (exactly one CAS - winner). -- **Loser**: spins `while (@atomicLoad(u8, &ls.realized_flag, .acquire) != - REALISED)`, and on each iteration honors the ADR-0092 §2 safepoint protocol - (`if (safepoint.gc_requested.load(.monotonic)) safepoint.park();` then - `std.atomic.spinLoopHint()`) — the thunk eval is exclusion-bearing, so a - non-parking busy-spin would delay/deadlock a stop-the-world collector. This - is the one piece of delay's `lockMutexAtSafepoint` discipline (D-244 #4) - that carries over even in the lock-free shape. - -The winner runs the thunk via the normal eval path (`vt.callFn`), which polls -safepoints at its own alloc/back-edges, so the winner is STW-cooperative -during the thunk; only the loser's spin needs the explicit poll added. - -### Why not block the loser (the crux) - -JVM clj blocks the loser on a `ReentrantLock` (the loser yields its core). A -blocking wait in Zig 0.16 requires `std.Io.Mutex`/`Io.Condition` — both are -non-`extern` so they must live in an **off-heap cell** (the delay/promise -shape), which on the language's highest-cardinality object means a per-element -heap allocation + a per-element GC finaliser. `std.Thread.Futex` (which would -allow a blocking wait on the inline flag word with no off-heap cell) **does -not exist in Zig 0.16.0** — all `std.Thread.*` sync primitives are gone -(verified against the pinned stdlib; ROADMAP §13 bans them regardless). So the -only blocking shapes force the off-heap cell. The spin's rare-contention core -cost is the right trade against a steady-state per-element allocation cost on -the hot object (runtime perf of the hottest object IS a finished-form -property; F-002 protects diff size, not runtime perf). - -## Alternatives considered - -> Verbatim output of a fresh-context Devil's-advocate `general-purpose` -> subagent fork (CLAUDE.md § ADR-level designs are handled inline). The -> deciding fact it flagged — `std.Thread.Futex` legality — was then resolved: -> the type does **not exist** in Zig 0.16.0, which removes Alt 3 and confirms -> the b1 ship. - -### Preface: F-NNN envelope check on the current b1 draft - -The b1 draft is F-NNN-clean: zero struct growth (reuses `realized_flag` + the existing `_pad: [5]u8`), no off-heap cell, no finaliser (F-006 friendly — no sweep-time `destroy`), lock-free steady-state read (F-011-faithful to JVM's volatile-null'd lock), at-most-once via CAS-claim (F-011 side-effect contract), and the safepoint-honoring spin (F-006/ADR-0092 STW-safe). It does NOT route through the header `lock_state` machinery (ADR-0009 respected). I found no F-NNN violation in the draft itself. My job is therefore to pressure the *design choice among legal shapes*, not to find an illegality. - -### Alt 1 — Smallest-diff: relaxed at-most-once, lock-free flag only (NO CAS-claim) - -**Shape.** Make `realized_flag` a 2-state `atomic.Value(u8)` (0=pending, 1=realised). Fast-path read `if (flag.load(.acquire) == 1) return realized;` exactly as b1. But on the cold path, DROP the CAS-claim entirely: every thread that sees 0 just runs the thunk, then does `realized = result; flag.store(1, .release);`. Last writer wins. No 3-state, no spin, no safepoint coupling on this path (the thunk eval still self-parks via the existing `alloc`/back-edge polls, so STW is unaffected). - -(a) **Better than b1:** Dramatically simpler — no spin loop, so the worst-case-core-waste critique (below) literally cannot occur; no `claiming` state to reason about; no safepoint coupling *added* to `force` (the thunk's own alloc-poll already covers STW); reentrant self-force just works (the thread sees 0, runs its own thunk, no self-deadlock — see the reentrancy section). It is the *closest* shape to the actual single-thread code on disk (lines 114/130/131): literally make the flag atomic and add `.acquire`/`.release`. - -(b) **Breaks / costs:** Violates clj's **at-most-once** guarantee. Two threads concurrently first-forcing the same LazySeq both run the thunk. For pure thunks (the overwhelming majority) this is invisible and wasteful-but-correct. For a side-effecting `(lazy-seq (do (println "x") (cons 1 ...)))` body — which clj's ReentrantLock guarantees prints once — this prints twice. There's also a `realized` tear window: thread A stores `realized=resultA, flag=1`; thread B (started later) overwrites `realized=resultB` AFTER A's reader already returned resultA — two callers see DIFFERENT seq head objects. They're `=`-equal in value but not identity-equal, which can surprise identity-sensitive code (rare in idiomatic Clojure, but `(identical? (first s) (first s))` across threads could flip). - -(c) **F-NNN:** **Violates F-011** (behavioural equivalence — clj's at-most-once is observable for side-effecting bodies). This is a real, named violation, so per the brief it is NOT the recommendation. I record it as a legal-but-inferior shape; it is the honest "smallest diff" but it ships a clj divergence that would need an AD-NNN to be defensible, and there is no project invariant that *mandates* dropping at-most-once — so it would be an *accepted divergence of convenience*, which `accepted_divergences.md` explicitly forbids ("If you cannot name the invariant, it is not accepted — it is a bug"). - -### Alt 2 — Finished-form-clean: off-heap `Io.Mutex` realise-cell, exactly like `delay.zig` - -**Shape.** Add a `cell: *LazySeqCell` field (an off-heap `struct { mutex: std.Io.Mutex }`), allocated in `alloc`, freed in a new `finaliseLazySeq`. `force` becomes a near-verbatim copy of `delay.zig force()`: lock-free `if (flag.load(.acquire) == 1) return realized;` pre-check, then `safepoint.lockMutexAtSafepoint(&ls.cell.mutex)`, re-check under lock, run thunk, store, unlock. The loser BLOCKS on the mutex (parked at a safepoint via `lockMutexAtSafepoint`) and yields its core — no spin. - -(a) **Better than b1:** (1) **The loser never burns a core** — it blocks and the OS deschedules it. This is the single biggest advantage and the crux of the whole review (see pressure-test). (2) It is **identical in shape to the existing `delay.zig` precedent** — one realise-memo primitive, one locking pattern, one mental model; a reader who understands `delay.zig` understands this for free. That uniformity is itself a finished-form virtue (the project prizes "one way"). (3) `lockMutexAtSafepoint` is already the audited STW-safe blocking primitive (D-244 #4); b1 re-derives a *second* STW-safe waiting mechanism (safepoint-polling spin) that has to be re-audited against the same hazard. Fewer distinct concurrency mechanisms = smaller correctness surface. - -(b) **Breaks / costs:** (1) **Per-LazySeq off-heap allocation + a GC finaliser.** LazySeq is the highest-cardinality heap object in the language (one per map/filter/range/iterate step). Every `(map inc (range 1e6))` would `gpa.create(LazySeqCell)` a million times and register a million finalisers. This is a *severe* allocation/GC-pressure regression versus b1's zero-alloc reuse of `_pad`. The `delay.zig` precedent is safe *because Delays are rare* (one per `(delay …)` form); LazySeqs are not. (2) `Io.Mutex` is non-extern, so it genuinely cannot live inline in the `extern struct` — the off-heap cell is forced, you can't shrink it away. (3) Finaliser cost at sweep time scales with live-LazySeq count. - -(c) **F-NNN:** F-NNN-clean (no violation). But the allocation explosion makes it a *finished-form-clean shape that the finished-form owner would unwind on perf grounds* — and crucially, **perf is NOT an F-NNN constraint** (F-002 says cycle/diff/LOC isn't a constraint; it does NOT say runtime perf isn't). Runtime performance of the hottest object in the language IS a finished-form property. So this is the rare case where "finished-form-clean" cuts *against* the heavyweight option: the cleanest *final shape* for a million-cardinality object is the zero-alloc one, not the one that mirrors the rare-object precedent. I rate Alt 2 **inferior to b1 on finished-form grounds**, not superior. - -### Alt 3 — Wildcard: futex-style block on the flag word itself (lock-free read, blocking cold wait, zero off-heap) - -**Shape.** Keep b1's 3-state atomic `realized_flag` inline (zero struct growth, zero off-heap). Keep b1's lock-free fast read and CAS-claim winner. The ONLY change is what the *loser* does: instead of `while (...) spinLoopHint()`, the loser **blocks on the flag word** via a futex-style wait — `std.Thread.Futex.wait(&flag_as_u32, claiming_value)` (or the Zig 0.16 `Io`-mediated equivalent if `std.Thread.Futex` is among the removed primitives — needs verification against the ROADMAP §13 ban; if futex is banned, this Alt collapses into Alt 2). The winner, after `flag.store(1, .release)`, does `Futex.wake(&flag, all)`. The loser still polls `safepoint.gc_requested` before each (re)wait and routes the block through `enterBlocked`/`exitBlocked` so a STW collector counts it as parked. - -(a) **Better than b1:** Combines b1's zero-alloc inline flag with Alt 2's core-yielding block — the loser sleeps instead of spinning, but there is NO per-LazySeq off-heap cell and NO finaliser. It is strictly the hybrid the brief asks for: lock-free read + real blocking wait *only* on the cold realise-contention path, with none of Alt 2's allocation cost. - -(b) **Breaks / costs:** (1) **Futex layering against the safepoint protocol is subtle.** A futex wait that is interrupted by a STW signal must wake, park, and re-wait — interleaving futex-wake with safepoint-park correctly is genuinely tricky and is a fresh concurrency mechanism the project has never audited (vs `lockMutexAtSafepoint` which is audited). (2) **`std.Thread.Futex` may be banned** — ROADMAP §13 / the zig_tips table bans all `std.Thread.*` *sync primitives* (`Mutex`/`RwLock`/`Condition`/`Semaphore`/`WaitGroup`). Futex is a borderline case (it's lower-level than those); if the ban is read broadly it's forbidden and this Alt is illegal-by-rule (not by F-NNN, by ROADMAP §13 — a level-3 amendable rule, but still a barrier). (3) Cross-platform futex semantics differ; the project would own that portability surface for its hottest object. (4) Contention on first-realise is genuinely rare, so the machinery may rarely fire — paying audit complexity for a cold path. - -(c) **F-NNN:** F-NNN-clean. The risk is ROADMAP §13 (level 3, amendable but real) on the `std.Thread.Futex` question. If §13's ban covers futex, prefer the `Io`-mediated wait, which lands back near Alt 2's off-heap requirement — so the wildcard's whole advantage (zero-alloc + blocking) hinges on futex being legal. **Needs a one-line ROADMAP §13 verification before adoption.** - -### Pressure-test of b1's weakest point: the loser busy-spins for the whole thunk-eval - -This is b1's real soft spot and the brief is right to flag it. The assumption "lazy-seq thunks are trivial cons-cells, concurrent-first-realise is rare, so a brief spin is fine" is **NOT uniformly safe**: - -- **Thunk does I/O / derefs a promise / blocks.** `(lazy-seq (cons (deref some-promise) ...))` or a thunk that reads a file. The winner's thunk eval can take *milliseconds to seconds*. Every losing thread spins a full core for that entire duration. With `pmap` over a shared lazy-seq (the brief's exact example), N-1 cores can spin-burn while one core blocks on I/O. This is a pathological core-waste: the machine is "busy" at 100%×(N-1) doing nothing. JVM clj here BLOCKS the losers on the ReentrantLock — they yield, the OS schedules other work, total throughput is far higher. -- **Thunk triggers GC.** Worse than waste: the winner's thunk allocates, hits `gc_requested`, and the winner parks at the alloc safepoint to let the collector run. The losing spinners, IF correctly safepoint-polling per the draft, also park — fine. But if the safepoint poll in the spin loop is *ever* mis-placed (polled once per outer iteration but the thunk eval is one long call that doesn't return to the spin loop), the losers spin THROUGH the STW window holding no lock but pinning cores, *delaying* the collection the winner is waiting to finish → a throughput cliff, not a deadlock. The draft's correctness here depends entirely on the spin's poll being airtight; Alt 2's `lockMutexAtSafepoint` gets this right by construction (the blocked thread is definitionally parked). -- **Is the "rare" assumption safe?** For sequential single-consumer code, first-realise contention is zero (no second thread). The contention ONLY appears when the SAME LazySeq head is shared across threads and forced concurrently — which is *exactly* what `pmap`, `future` over a shared seq, and `(let [s (map …)] (future (reduce + s)) (reduce * s))` do. These are not exotic; they are the marquee use cases of the concurrency gap-area (I) this ADR serves. So the spin's worst case lands precisely on the workloads gap-area I exists to harden. That is uncomfortable. - -**Verdict on the spin:** acceptable for trivial pure thunks under low contention; *bad* for I/O/blocking thunks under `pmap`-style sharing. Since b1 cannot tell the two apart at the spin site, it pays the worst case whenever contention happens to be high. **A hybrid that blocks (Alt 3 if futex is legal; else Alt 2's mutex on the cold path only) strictly dominates the spin on the bad workloads and is no worse on the good ones** (uncontended first-force never reaches the wait at all — the CAS winner just runs). The brief's instinct — "could the loser block on a condition instead of spinning?" — is correct, and the answer is yes, and it's better. - -### Reentrant self-force: b1 self-deadlocks; is the divergence acceptable? - -A thunk that forces its own LazySeq (`(def s (lazy-seq (cons 1 s)))` forced) is degenerate. clj's ReentrantLock permits re-entry → the body recurses → **StackOverflowError**. b1's non-reentrant CAS-claim: the same thread that won the claim (`flag==2`) re-enters `force`, sees `flag==2` (not 1), and spins forever waiting for *itself* to publish → **hang** (a livelock, since the spin also safepoint-polls, so it's a *parking* hang, not a frozen core — but a hang nonetheless). `delay.zig`'s `Io.Mutex` has the identical self-deadlock (non-reentrant mutex, same thread re-locks → deadlock). So: - -- b1 diverges from clj: **hang vs StackOverflowError**. Both are "the program is broken"; neither is a useful result. But clj's StackOverflow is *observable and recoverable* (it's a throwable the user can catch in principle), whereas b1's hang is **not observable** — the program just stops, and under STW it parks silently. That is a *worse* failure mode for debugging. -- **Is it an F-011 problem?** Marginally. The input `(lazy-seq (cons 1 s))` self-referencing-forced is genuinely degenerate user code; clj's own behaviour (StackOverflow) is itself a crash, so "behavioural equivalence" on a crash input is weak. BUT F-011 + `accepted_divergences.md` say a DIFF must be *classified*, not left floating. A silent hang vs a throwable is a real observable DIFF. The clean disposition: **classify it as an accepted divergence (AD-NNN) with `derives_from: + the non-reentrant-realise invariant`, and a pin test** asserting cljw hangs-bounded (or, better, *detect* the self-claim: if the CAS-claiming thread re-enters and sees `flag==2` set by its OWN thread-id, raise a `stack-overflow`-flavoured error instead of spinning). The last option — record the claiming thread-id and raise on self-re-entry — would make cljw STRICTLY BETTER than clj (a clean error vs a StackOverflow) at the cost of one thread-id field. That is worth considering as a b1 amendment regardless of which alt ships; it converts a silent hang into a diagnosable error and is finished-form-clean. Alt 1 (no claim) sidesteps this entirely (self-force just recurses → Zig stack overflow → same as clj, modulo message), which is a minor point in Alt 1's favour. - -### Recommendation - -**Ship a hybrid: b1's inline zero-alloc 3-state flag + lock-free read + CAS-claim, but replace the loser's busy-spin with a blocking wait — Alt 3 (futex on the flag word) IF ROADMAP §13 permits `std.Thread.Futex`; otherwise the b1 spin is the pragmatic ship with a mandatory self-re-entry guard.** - -Reasoning, finished-form-first per F-002 (and noting cycle/diff size is explicitly NOT a downgrade reason): - -1. **Alt 2 is out** despite being "precedent-uniform" — the per-LazySeq off-heap cell + finaliser is a real perf regression on the language's highest-cardinality object, and runtime perf of the hot object IS a finished-form property (F-002 does not protect it; it protects *diff size*, the opposite). Do not mirror the rare-`Delay` shape onto a million-cardinality object just for uniformity. -2. **Alt 1 is out** — it's the only option that violates a *named* F-NNN (F-011 at-most-once), and the divergence is one of convenience, not invariant, so `accepted_divergences.md` forbids accepting it. -3. **Between b1-spin and Alt 3-block**, Alt 3 strictly dominates on the `pmap`/I/O-thunk workloads that gap-area I exists to serve, with no off-heap cost. The *only* thing blocking Alt 3 is the §13 futex-legality question. **Resolve that first** (one grep of ROADMAP §13 + the zig_tips ban scope). If futex is legal, ship Alt 3. If §13 bans it, the `Io`-mediated block forces an off-heap cell (= Alt 2's cost), at which point **b1's spin becomes the right ship** — because then the choice is "spin on the cold contended path" vs "off-heap-allocate the hottest object in the language", and the latter's steady-state cost outweighs the former's rare-contention cost. -4. **Regardless of which ships, add the self-re-entry guard**: record the claiming thread-id; on self-re-entry while `flag==2`, raise a stack-overflow-flavoured error rather than hang/spin. This converts b1's silent-hang divergence into a diagnosable error that is *better* than clj's StackOverflow, and lets the reentrancy DIFF be classified as an AD-NNN with a pin test rather than left floating. - -Net: the draft's b1 is F-NNN-legal and a defensible ship, but its busy-spin is a measurable footgun on exactly the concurrency workloads this ADR targets, and its silent self-deadlock is an unclassified DIFF. The cleanest final shape is "b1's inline flag, but the loser blocks not spins, and self-re-entry errors instead of hangs." Verify the §13 futex question; that single fact decides spin-vs-futex. - -## Resolution of the DA's open questions - -1. **`std.Thread.Futex` legality (the deciding fact).** Verified against the - pinned Zig 0.16.0 stdlib: `std.Thread` exposes **no** `Futex` (and no - `Mutex`/`Condition`) — every `std.Thread.*` sync primitive is gone (ROADMAP - §13 bans them anyway). **Alt 3 is therefore not implementable in Zig - 0.16.** Per the DA's own decision tree (point 3), with futex unavailable the - only blocking shape is the `Io`-mediated wait, which forces Alt 2's off-heap - cell — so **b1's spin is the right ship**. The spin's rare-contention core - cost beats a per-element off-heap allocation on the hottest object. -2. **The spin's poll must be airtight.** The DA's GC-throughput-cliff concern - is real and is addressed: the loser polls `safepoint.gc_requested` + `park` - on *every* spin iteration (the canonical worker-loop poll from - `safepoint.zig`), and the winner runs the thunk through the normal eval path - which self-parks at its own alloc safepoints. No code path spins through an - STW window. -3. **Reentrant self-force guard — deliberately NOT added.** The DA's thread-id - guard would make cljw strictly better than clj, but: (a) `delay`/`promise`/ - `future` already carry the **identical** non-reentrant self-deadlock and the - survey's mandate is to *match that precedent*, not diverge from it on - LazySeq alone; (b) a thread-id field risks struct growth or a truncated-tid - false-positive (wrongly raising on a real cross-thread contention with a - colliding truncated id) on the language's highest-cardinality object. The - finished-form choice that matches the memo-once family is to **not** special- - case LazySeq. The self-forcing input (`(lazy-seq (seq s))` forced) is - degenerate (clj StackOverflows on it). It is recorded as a known limitation - of the once-only-memo family below, not opened as an AD-NNN with an - untestable hang-pin. - -## Consequences - -- `force()` becomes thread-safe: at-most-once thunk invocation + a - release/acquire-fenced result, with a lock-free steady-state read. The data - race that real `future`/`agent`/`pmap` sharing exposed is closed. -- **Zero cost on the realised steady state** beyond one acquire-load (was a - plain load); zero struct growth; zero off-heap allocation; zero finaliser. -- `isRealised` reads the flag with an acquire-load for visibility (was a plain - read — a benign stale-false race before, now correctly fenced). -- **Known limitation (shared with delay/promise/future):** a thunk that forces - its *own* LazySeq (degenerate self-reference) hangs (the claiming thread - spins for a publish that can only come from itself) rather than clj's - StackOverflowError. Not guarded, for consistency with the memo-once family - and to avoid per-element cost on the hot object. clj also crashes on this - input (StackOverflow); the divergence is hang-vs-throwable on degenerate - code. -- The loser's spin can burn a core while a *slow* thunk realises under high - contention (the DA's pressure-test). Accepted as the right trade vs a - per-element off-heap allocation, given Zig 0.16 offers no inline blocking - wait. If a future Zig brings back a futex (or a measured `pmap`-over- - slow-lazy-seq workload proves the spin pathological), revisit per Alt 3 — - recorded as a recall note, not a debt row (no present barrier). - -## Affected files - -- `src/runtime/lazy_seq.zig` — `force()` rewrite (CAS-claim + safepoint spin), - `isRealised` acquire-load, `## Mutex shape` module-doc update, the - concurrent-force unit test. -- `.dev/debt.yaml` — D-046 → discharged. diff --git a/.dev/decisions/0144_extend_via_metadata_dispatch.md b/.dev/decisions/0144_extend_via_metadata_dispatch.md deleted file mode 100644 index 601031219..000000000 --- a/.dev/decisions/0144_extend_via_metadata_dispatch.md +++ /dev/null @@ -1,143 +0,0 @@ -# ADR-0144 — `defprotocol :extend-via-metadata true` dispatch - -- **Status**: Proposed → Accepted (2026-06-15) -- **Amends**: ADR-0008 (protocol dispatch unify) — adds a metadata-fallback - precedence step to the protocol-fn dispatch path. ADR-0068 (protocol_impls - SSOT) is untouched: metadata extension is per-VALUE, not a declared interface. -- **Debt**: D-314 (discharged by this ADR's implementation). -- **Devil's-advocate**: `private/notes/D314-da-structural.md` (full output; - alternatives reflected below). - -## Context - -Clojure's `(defprotocol P :extend-via-metadata true (m [x]))` lets a value -supply a protocol-method impl through its **metadata**: a fn stored under the -protocol-defining-ns-qualified method **symbol** (e.g. `user/m`) dispatches when -the protocol method is called on that value. Used by honeysql (the InlineValue / -`sqlize` protocol) and clojure.datafy (Datafiable / Navigable). - -cljw PARSED the option (the protocol loads) but DROPPED the flag (D-314): it was -never threaded to the protocol descriptor, and dispatch never consulted receiver -metadata, so a value carrying `^{user/m fn}` hit `protocol_no_satisfies`. - -### clj contract (oracle-verified 2026-06-15) - -- **Key = namespaced method SYMBOL** (`user/sized`), NOT a keyword. A keyword key - does not match (clj raises "No implementation"). -- **Precedence**: on-interface-instance > **receiver-metadata** > registered - extend-type/Object impl > no-method. i.e. **metadata beats extend-type** (looser - than D-314's original "fall back when no impl" wording). -- **`satisfies?` / `extends?` IGNORE metadata** → `false` for a meta-only - extension (they read the descriptor's impl SSOTs, not the value's meta). - -## Decision - -**Alt 1 — metadata-dispatch as a Layer-0 helper invoked from the protocol-fn -dispatch entry.** - -1. Add `extend_via_metadata: bool` to `ProtocolDescriptor` (consumes one of the 6 - reserved `_pad` bytes — ABI-clean, F-004). Threaded: `expandDefprotocol` emits - a 3rd arg to `rt/__make-protocol!` when `:extend-via-metadata true` is parsed; - the primitive `makeProtocol` accepts 2-or-3 args (default false for the 2-arg - bootstrap forms); `runtime/protocol.zig::makeProtocol` sets the field. -2. Extract the unified meta read into a Layer-0 `runtime/meta.zig::metaOf` - (the switch currently in `lang/primitive/metadata.zig::metaFn`, all of whose - reads are Layer-0); `metaFn` delegates to it (F-011 commonization — one meta - read, not two). This gives the Layer-0 dispatch path a meta reader without the - Layer-2 `metadata.zig` (zone_deps: Layer 0/1 must not import Layer 2). -3. Add `dispatch.metaDispatch(rt, env, descriptor, receiver, method_name, args, - loc) anyerror!?Value` (Layer 0): when `descriptor.extend_via_metadata`, read - `meta.metaOf(receiver)`; if it is a map, build the qualified method symbol - (`symbol.intern(rt, defining_ns, method_name)`, `defining_ns` = the - descriptor fqcn's prefix before the last `/`); `map.get` it; if a callable - value, invoke via `rt.vtable.callFn` and return it. Else `null` (fall through). -4. `tree_walk.callProtocolFn` calls `metaDispatch` **before** `dispatch.dispatch`: - `if (try dispatch.metaDispatch(...)) |v| return v;`. Both backends route - protocol-fn calls through `callProtocolFn` (VM wires `.callFn = - tree_walk.treeWalkCall`, vm.zig:1391), so this single site serves both. - -### Precedence (matches clj) - -`callProtocolFn` runs `metaDispatch` first, so **metadata beats extend-type** -(metaDispatch returns before `dispatch.dispatch`'s type-impl / Object-default / -raise chain). Gated on the flag → non-metadata protocols pay nothing. Minor -divergence: clj's on-interface (a JVM-interface-implementing class) beats -metadata; cljw has no generated JVM interface, so a deftype that BOTH -inline-implements the protocol AND a specific instance carries an overriding -`^{ns/m fn}` would let metadata win where clj's on-interface would not. This is a -pathological corner (no realistic code) — documented, not chased. - -### Cache-bypass (load-bearing correctness) - -The per-`CallSite` cache keys on the receiver's **TypeDescriptor** (per-TYPE); -metadata is **per-VALUE** (two instances of one type carry different/no meta; a -`(with-meta {} {…})` map shares its TD with every map). `metaDispatch` runs as a -separate branch that returns on hit **without** calling `dispatch.dispatch`/ -`dispatchOrNull`, so a meta hit never writes the per-type cache slot (which would -poison every other value of that type). Pinned by a unit test: two instances of -one deftype — one with meta, one without — dispatch differently in either order. - -### `satisfies?` unchanged - -cljw `satisfies?`/`extends?` read the descriptor's impl SSOTs, not the value's -meta, so a meta-only extension reports `false` — matches clj for free. Pinned by -an e2e case so a future change can't make `satisfies?` peek at meta. - -## Alternatives considered (Devil's-advocate, fresh context) - -The DA's leading finding: `dispatch.dispatch`'s SIGNATURE carries only strings, -but `callProtocolFn` (its descriptor-holding caller) already holds -`pfn.descriptor` and throws it away as `.fqcn()`. So no registry is needed for -the `(m x)` path. The DA enumerated: - -- **Alt 1 (chosen) — flag on descriptor, metadata probe at the protocol-fn - entry.** Zero new Runtime state; the flag rides the descriptor; cache-bypass - natural (probe before `dispatch.dispatch`). -- **Alt 2 (DA-recommended) — thread the `descriptor` into Layer-0 - `dispatch.dispatch`** so the whole precedence lives in one function (DA cited - F-011/DRY). Cost the DA surfaced: callers without a descriptor need one. -- **Alt 3 (rejected by DA) — name→descriptor registry on `rt`.** Pays a - per-dispatch hash lookup to recover a pointer the caller already held; the - reusable index it builds is D-443 (reflection) forward-debt, not D-314's need. - -**Why the main loop chose Alt 1 over the DA's Alt 2 (NOT a cycle-budget -decision):** verification (not available to the DA) showed `dispatch.dispatch` -has **18+ callers** — the polymorphic primitives (`sorted.zig`, `sequence.zig`, -`collection.zig`) dispatch built-in protocols (`Associative`, `Sorted`, -`IPersistentCollection`, `Seqable`, …) by hardcoded string with **no -descriptor**. None of those built-in protocols are extend-via-metadata, and a -user-defined extend-via-metadata protocol dispatches ONLY via `callProtocolFn`. -So Alt 2 would force a `null`/descriptor argument across 18+ sites to thread a -flag meaningful to exactly **one** caller — noise, not unification, and it would -need a "well-known-descriptor set on Runtime" for the built-in-protocol callers -(a back-door registry, the very thing Alt 3 was rejected for). The metadata -step's semantic scope **is** the protocol-fn path; placing the logic in a Layer-0 -helper (`dispatch.metaDispatch`, co-located with dispatch policy in -`dispatch.zig`) invoked from `callProtocolFn` keeps the policy in Layer 0 (the -DA's F-011 spirit) without polluting the shared `dispatch.dispatch` hot path. -This choice survives an equal-diff-size test (Alt 1 scopes the metadata step -correctly even if Alt 2 were the same size), so it is finished-form-grounded. -Alt 3 rejected on F-002 (premature reflection registry = D-443 scope-bleed). - -## Consequences - -- `(defprotocol P :extend-via-metadata true …)` now dispatches via receiver - metadata under the namespaced method symbol key, matching clj, including - metadata-beats-extend-type precedence and `satisfies?` ignoring metadata. -- `runtime/meta.zig` is the new Layer-0 SSOT for "a value's metadata map"; - `metaFn` (the `(meta x)` primitive) delegates to it — one meta read. -- Existing protocols (flag unset) are unaffected: `metaDispatch` returns `null` - immediately, no meta read, no hot-path cost. -- The deftype-inline-vs-metadata pathological corner is a documented minor - divergence (no AD row — not realistically constructible; revisit only if hit). - -## Affected files - -- `src/runtime/protocol.zig` — `ProtocolDescriptor.extend_via_metadata` + `makeProtocol` param. -- `src/runtime/meta.zig` — NEW: Layer-0 `metaOf` (extracted from `metaFn`). -- `src/lang/primitive/metadata.zig` — `metaFn` delegates to `meta.metaOf`. -- `src/runtime/dispatch.zig` — `metaDispatch` helper. -- `src/eval/backend/tree_walk.zig` — `callProtocolFn` calls `metaDispatch` first. -- `src/lang/macro_transforms.zig` — `expandDefprotocol` emits the flag arg. -- `src/lang/primitive/protocol.zig` — `makeProtocol` reads `args[2]`. -- `test/e2e/phase16_extend_via_metadata.sh` — the behavioural pins. diff --git a/.dev/decisions/0145_jit_go_no_go_defer_for_loser_campaign.md b/.dev/decisions/0145_jit_go_no_go_defer_for_loser_campaign.md deleted file mode 100644 index 81cff060c..000000000 --- a/.dev/decisions/0145_jit_go_no_go_defer_for_loser_campaign.md +++ /dev/null @@ -1,127 +0,0 @@ -# ADR-0145 — Narrow ARM64 JIT (D-133): go/no-go = NO-GO now; re-sequence last; run the 3-loser runtime campaign first - -- **Status**: Proposed → Accepted (2026-06-15) -- **Relates to**: F-010 (interim milestone M = Phase 15 完遂 + cw-v0-程度 JIT; - its prerequisite chain has an explicit **"JIT go/no-go"** step), F-015 - (completion-grade posture), D-133 (the JIT-ordering debt row), D-386 (dispatch - perf — superinstructions + ip-hoist + alloc-free arith now landed). Does NOT - amend F-010 (the JIT remains a nominal M component; this exercises F-010's - go/no-go step + re-sequences). -- **Devil's-advocate**: `private/notes/jit-go-no-go-da.md` (full fresh-context - output; the three alternatives + verdict reflected below). -- **Survey**: `private/notes/9.2.S-jit-d133-survey.md` (v0 jit.zig map + cljw - integration points + the Step 0.6 re-measurement). - -## Context - -F-010 sets the interim milestone **M = Phase 15 完遂 + a cw-v0-程度 JIT landed**, -with the prerequisite chain **"superinstruction/fusion pass → JIT go/no-go → -narrow JIT."** The superinstruction/fusion pass has landed (O-018…O-022) plus the -dispatch-inline / ip-register-hoist (O-017/O-028) and the alloc-free fixnum arith -fast path (O-029). This ADR is the **JIT go/no-go** the chain calls for. - -**The go/no-go data (Step 0.6 re-measurement, ReleaseSafe, hyperfine -N -r12):** - -| metric | cljw | Python | note | -|------------------------------|---------|---------|-------------------------------| -| `04_arith_loop` (N=1e6) cold | 30.8 ms | 57.6 ms | **cljw 1.87× faster** | -| startup (`-e 1`) | 6.0 ms | 18.2 ms | → arith_loop *compute* also wins (~25 vs ~40 ms) | - -The JIT was the §9.2.S campaign's planned last lever because v0 measured -`arith_loop` 31→3 ms with it. But the superinstruction pass already pushed v1's -`arith_loop` to **beat Python by 1.87× on both cold AND compute**. So the JIT's -**beat-Python value (goal (a), primary) is zero** — already won. Its residual -value is **v0-parity (goal (b), secondary)** on an already-winning bench, at the -**highest cost/risk unit in the project** (executable memory + W^X + -codesign/AMFI entitlement + cross-platform codegen + the register-clobber bug -class v0 shipped three of — per the survey §5). - -Meanwhile the "beat Python on EVERY bench" primary goal (F-002/F-015 completion -grade) has exactly **three open failures the narrow arith JIT does not touch**: -sieve 1.5×, regex_count 1.99×, nested_update 1.44× — lazy-seq / regex-engine / -persistent-update bound, which v0 itself won with **runtime levers, not the JIT** -(filter-chain collapsing 24C.7 → sieve 103×; update-in Zig builtins 24C.9). - -## Decision - -1. **JIT go/no-go = NO-GO now.** Re-sequence the narrow ARM64 JIT (D-133) to - **last** in the perf campaign. It is NOT cancelled (F-010's M still nominally - includes it — only the user amends F-010); it is gated behind a concrete - **ROI predicate**: re-open D-133 when **(i)** a real hot loop surfaces where the - JIT moves a *Python-losing* or *v0-losing* bench the runtime levers cannot - reach, OR **(ii)** the 3 losers are closed and v0-parity-on-arith becomes the - active goal, OR **(iii)** the user re-affirms the JIT as a standalone priority. -2. **Run the 3-loser runtime campaign first** (DA Alt 2 — the only option that can - *close* the primary goal): **sieve first** (proven cross-platform filter-chain - collapsing / fused-reduce lever, no equivalence-audit gate), then - **nested_update** (update-in/assoc-in Zig builtins + vector COW), then - **regex_count** (after its cross-lang equivalence audit per - `perf_v0_baseline.md:108`). All cross-platform, no machine code, F-011 §4 - shared-mechanism. -3. **Optional cheap probe before sieve**: the computed-goto / tail-threaded - dispatch (D-386 lever (c), DA Alt 3) — but v0 measured tail-call dispatch - **0% on Apple M4**, so budget it as a single **measure-first probe, expect ~0**, - revert on a flat result. Not a committed unit. -4. **If/when the JIT is built** (ROI predicate fires): module shape = - **`src/eval/optimize/jit/` dedicated subtree** (portable `CodeBuffer` + - per-arch `arm64/`/`x86_64/` encoders + the lifted `zwasm` `jit_mem.zig` - MAP_JIT/W^X), build-flag-gated, cross-platform from day 1, ReleaseSafe-correct - with a callee-saved trampoline. NOT v0's single-file port (finished-form-dirty: - unscoped, unflagged, ARM64-only); NOT an SSA mini-IR (over-engineering for a - narrow fixnum loop, F-002 "excessive skeletons" smell). - -## F-010 / F-015 reconciliation (the mode question the DA flagged) - -F-010's M = Phase 15 + JIT; deferring the JIT means M is not *literally* reached, -so a strict reading puts the loop **pre-M**, contradicting the "post-M operating -mode" framing in the scaffolding (memory `tech_debt_consolidation` etc.). Resolved: - -- The deferral is **F-010 process-compliant** — F-010's own prerequisite chain - contains the "JIT go/no-go" step; a data-driven NO-GO exercises that step. The - JIT stays a nominal M component (owed, re-openable), so F-010 is **not amended** - (only the user amends F-NNN). -- F-015 (newer, 2026-06-15) reframes the project to **completion-grade posture**: - the quality-elevation loop runs on the built surface NOW, "not bound by the old - roadmap/guardrails." So the operative mode is **F-015 completion-grade**, with - the JIT as a **deferred-per-go/no-go** M component — not a literal "post-M". The - loop should describe its mode as "F-015 completion-grade; M's JIT half deferred - pending its ROI predicate," not "post-M". (Scaffolding that says "post-M" is - imprecise but not load-bearingly wrong — the quality loop is the active mode - either way; corrected opportunistically, not as a churn.) - -## Alternatives considered (DA fork, `private/notes/jit-go-no-go-da.md`) - -- **Alt 1 — port v0's `jit.zig` minimally**: only "wins" on the forbidden - diff-size axis; finished-form-dirty (violates A3 dedicated-subtree + - cross-platform-day-1). Goal (a) +0, goal (b) +1 bench at max cost. Rejected. -- **Alt 2 — 3-loser runtime campaign, defer JIT [RECOMMENDED, ADOPTED]**: advances - the PRIMARY goal directly (each loser closed is a beat-Python win the JIT cannot - deliver), fraction of the cost/risk, cross-platform, no machine code. Goal (a) - +3 → potentially COMPLETE; goal (b) +2-3 side effect. The only option that can - close goal (a). -- **Alt 3 — computed-goto dispatch**: v0 measured 0% on M4; expected ~0; a - measure-first probe, not a unit. -- **Verdict**: committing to the full JIT now is NOT right; the ROI questioning is - sound finished-form judgment (data-grounded, not effort-grounded → does NOT trip - the Cycle-budget-defer smell — Alt 2 is larger-or-comparable scope AND - finished-form-cleaner). Adopt Alt 2. - -## Consequences - -- The perf campaign's next unit is **sieve** (filter-chain collapsing / fused - reduce), validated by the diff oracle + `clj` corpus (F-011) + a bench - re-measurement; GC-rooting-sensitive lazy-seq changes are torture-gated - (`CLJW_GC_TORTURE` + the new `CLJW_GC_TORTURE_ALLOC`). -- D-133 moves to a **go/no-go = no-go-now; re-sequenced last; ROI predicate** - status (not discharged — the JIT is still owed per F-010). -- No executable-memory / codegen / codesign surface is added now (the project's - highest-risk surface stays unopened until the ROI predicate fires). -- `handover.md` first-task points at sieve. - -## Affected files - -- `.dev/debt.yaml` — D-133 status update (go/no-go = no-go-now, re-sequenced). -- `.dev/handover.md` — first-task = sieve (Alt 2). -- `private/notes/9.2.S-jit-d133-survey.md` (Step 0.6 section) + - `private/notes/jit-go-no-go-da.md` (DA output) — the grounding, preserved for - the day the ROI predicate fires. diff --git a/.dev/decisions/0146_filter_chain_collapse.md b/.dev/decisions/0146_filter_chain_collapse.md deleted file mode 100644 index 3e876325e..000000000 --- a/.dev/decisions/0146_filter_chain_collapse.md +++ /dev/null @@ -1,166 +0,0 @@ -# ADR-0146 — Filter-chain collapsing: go/no-go = NO-GO (the cljw sieve is fn-call-bound, not depth-bound); redirect the sieve to the fn-call/closure-invocation lever - -- **Status**: Proposed → **Rejected (NO-GO)** (2026-06-15) -- **Relates to**: ADR-0145 (JIT no-go → run the 3-loser runtime campaign, sieve - first — this ADR is that unit's investigation), F-002 (finished-form wins; - also: do not ship complexity for a rare-pattern win that misses the campaign - target), F-006 (published-root GC), F-011 (behavioural equivalence vs `clj`), - O-023 (the existing `fuse` reduce-fusion slug), D-386 (dispatch perf — the - fn-call lever this redirects to). -- **Devil's-advocate**: fresh-context fork (2026-06-15), reflected verbatim in - § Alternatives considered. It recommended the Zig producer (Alt 2) over the - `.clj` walk (Alt 1) — sound *given the survey's premise*, which the - implementation-time measurement then **falsified** (see § Decision). -- **Survey**: `private/notes/9.2.S-sieve-filter-chain-survey.md` (v0 mechanism + - cljw state + the Step-0.6 re-laying) + this ADR's measurement section. - -## Context - -The `sieve` benchmark (functional nested-filter Sieve of Eratosthenes) is one of -ADR-0145's three open Python-losers: sieve(1000) cljw 27.5 ms vs Python 18.8 ms -(py 1.46×); sieve(3000) cljw 133.5 ms vs Python 18.2 ms (py 7.33×). The survey + -DA both modelled the pathology as v0 did — a **k-deep nested `filter` lazy-seq** -whose depth multiplies the per-element evaluator re-entry — and v0 won this exact -benchmark 1645→16 ms (103×, label 24C.7) with a Zig `lazy_filter_chain` producer -that collapses the nesting to one flat-pred walk. - -A full Stage-1 implementation of that producer was built and measured (a dedicated -`chain` slot on `LazySeq` + a Zig flat-walk producer in `force` + a `-filter-chain` -primitive + the GC-root frame). It **works and collapses correctly** (instrumented: -the pred vector grows 1→2→3→… while depth stays 1). - -## Decision — NO-GO, with the measurement that overturned the premise - -**The cljw sieve is fn-call-bound, NOT depth-bound. The collapse does not move it, -so it is rejected; the sieve's real lever is fn-call/closure-invocation speed.** - -Measured (ReleaseSafe, hyperfine), collapse vs naive: - -| workload | naive (nested) | collapsed | effect | -|---------------------------------------|----------------|-----------|--------------| -| sieve(3000) | 139.1 ms | 133.0 ms | ~1.05× (noise) | -| deep 40-nest `(filter pos? …)` cheap | 23.7 ms | 9.0 ms | **2.6×** | - -The collapse gives a real **2.6×** on *deep* nests with *cheap* predicates — but -**~0 on the sieve**, because the per-element cost in the sieve is the **predicate -call**, not the lazy-seq nesting. Isolation (1M elements, single filter): - -| predicate | time | per element | -|---------------------------------------------|---------|-------------| -| user `fn*` `(fn [x] (not= 0 (mod x 7)))` | 766 ms | ~766 ns | -| builtin `odd?` | 138 ms | ~138 ns | - -A user-`fn*` predicate costs **~5.5× a builtin** (frame setup + arg bind + var -derefs + the nested `mod`/`not=` builtin calls). The sieve does on the order of a -million such user-`fn*` calls; that dominates. The depth multiplier that crippled -v0 (1645 ms) is **already largely absent in cljw** — cljw's lazy machinery (O-004 -chunking + the cheap `force`) starts at 27 ms where v0 started at 1645 ms, i.e. -cljw is already near v0's *post-collapse* 16 ms. **v0's pathology did not transfer; -the survey/DA premise was v0-shaped and wrong for cljw.** - -Two further nails (beyond "it doesn't help"): - -1. **Correctness gap requiring Layer-0 seqable replication.** The producer lives in - `force` (Layer 0, `lazy_seq.zig`) and used `lazy_seq.seq/first/rest`, which only - handle lazy/list/chunked/range — **not vector/map/string bases**. `(filter pos? - [1 2 3])` returned `nil` for `first` → `pos? nil` TypeError; `CLJW_GC_TORTURE=1` - surfaced it. (The diff oracle PASSED — both backends run the same producer, so - the bug is identical on both and produces no TreeWalk≠VM DIFF; the real oracle is - `clj`, and torture/direct probing caught it. A useful reminder that the diff - oracle is necessary, not sufficient.) Fixing it correctly means replicating the - Layer-2 seqable+chunking handling (`(seq coll)` + the chunked arm) inside Layer 0 - — significant added complexity for a rare-pattern win. - -2. **Cost vs value.** Even fixed, the win is a *rare* pattern (40-deep filter nests - are not written; 2-3 deep get a modest shave), paid for with a permanent slot on - the **highest-cardinality heap object** (`LazySeq`), a branch in the hot `force` - path, and the seqable-replication complexity. Per F-002, shipping that to bank a - narrow win that misses the campaign target is the wrong trade. The valuable output - of this cycle is the **finding**, not the code. - -**Redirect.** The sieve (and fib, and all user-fn-heavy code) is gated on -**`fn*`/closure invocation speed**. That is the next lever — the D-386 dispatch / -intrinsic territory (O-028/029 already shaved fib via ip-hoist + alloc-free arith; -the closure-call path — frame setup, arg binding, var deref of `mod`/`not=` — is the -next surface). It is **higher value** than collapse: it moves every Python-losing -*compute* bench, not one rare lazy pattern. The collapse is recoverable from this -ADR + git history if a real deep-nested-filter workload ever justifies it -(commits/history are the backup — user, 2026-06-15). - -**fused-reduce in its proper form stays a live, separate goal** (user direction, -2026-06-15: *"fused reduce はめちゃくちゃ効果が高かった…正しい姿で入れたい / 実験も -柔軟に…何段階かで諦めないでいきたい"*). It is the **reduce** path (O-023 already beats -Python on `map_filter_reduce`); its 正しい姿 is the unified transform descriptor / -extending fusion — pursued when a reduce-heavy bench needs it, NOT bundled with the -sieve (which never reduces). Tracked as **D-445** (open, not deferred-forever). - -## Alternatives considered (Devil's-advocate fork, fresh context, 2026-06-15) - -> Reflected verbatim. The DA grounded against the actual cljw code and v0. Its -> recommendation (Alt 2 / Zig producer) was correct **conditional on the premise -> that the sieve is depth-bound** — the premise the implementation-measurement -> falsified. Preserved as the rationale record + the reason the NO-GO is evidence-, -> not effort-, grounded. - -**Fact A — v0's winning fix WAS a2's shape, not a1's.** The 103× sieve win -(1645→16ms) came from the Zig `lazy_filter_chain` union arm producing the next -survivor directly in Zig and re-stamping the chain marker on `rest_source`. The -separate fused-reduce only got sieve to 40ms (2× Python). So the `.clj`-tree-walk -approach is what left sieve above Python in v0; the Zig producer is what crossed. - -**Fact B — sieve never reduces.** It forces via `first`/`rest`/`take`, never -`reduce`. cljw's O-023 fuse fires only in `reduceFn`'s 3-arg arm. So a chain marker -must be read on the lazy force path. - -### Alt 1 — smallest-diff: `chain` slot read on the lazy force path only (≈ a1) -`.clj`-level flattening read; flat short-circuit loop over N preds per survivor. -- Better: honest about what sieve exercises; clears the torture gate trivially. -- Breaks: keeps the per-element `.clj` `lazy-seq`/`cons`/`loop` tree-walk; v0's data - says ~40ms-class, plausibly still above Python — a correct but incomplete landing. - -### Alt 2 — finished-form-clean: Zig `lazy_filter_chain` producer (v0's shape) [DA pick] -Zig flat-walk; predicates via `callFn`; tail re-stamps the chain. -- Better: the only alternative with measured evidence of beating Python *in v0*; - removes both the depth re-entry and the per-element `.clj` tree-walk. -- Breaks / the real axis: a new Zig GC-root site (F-006) — bounded and - gate-verifiable (the `reduceFn` `gc_roots[]` idiom is copy-adaptable). [This was - built; the GC-root frame worked. What the DA could not foresee without measuring - was that the cljw sieve isn't depth-bound, so removing the depth cost banks ~0.] - -### Alt 3 — wildcard: fold `chain` + `fuse` into one "lazy transform descriptor" -One `xf_desc` slot consumed by `-fused-reduce` AND a flat producer. -- Better: the most finished-form-clean answer to "two slots is duplication"; the - user's fused-reduce 正しい姿. Future-proofs `map`-chain collapse. -- Breaks: touches the gate-green O-023 path; a transducer stack ≠ a pred vector — a - deliberate refactor ADR, and (post-measurement) one whose lazy-walk half is moot - for the sieve. The reduce half lives on as D-445. - -### Verdict (DA, then corrected by measurement) -DA: commit to Alt 2 (the Zig producer); a1-first was the Cycle-budget-defer smell. -**Measurement correction:** the whole a1-vs-a2 axis assumed the sieve is -depth-bound. It is fn-call-bound. So Alt 2's "removes the per-element `.clj` -tree-walk" advantage banks nothing on the sieve (the `.clj` scaffolding was never -the cost). The Cycle-budget-defer smell does not apply to a NO-GO grounded in -*measured no-effect on the target* — that is finished-form judgment (F-002), not an -effort dodge. Both a1 and a2 are rejected for the sieve; the lever is fn-call speed. - -## Consequences - -- No source lands this cycle (the Stage-1 implementation was reverted after the - measurement). The cycle's deliverable is the **finding** + this redirect. -- The §9.2.S sieve loser stays **OPEN**, retargeted at **`fn*`/closure invocation - speed** (the next stage; D-386 dispatch / intrinsic family). -- **D-445** (open, NOT deferred-forever): fused-reduce in its proper form (the - unified transform descriptor) — the user's live interest, on the **reduce** path, - pursued when a reduce-heavy bench needs it. -- A lesson worth keeping: **the diff oracle (TreeWalk≡VM) is necessary but not - sufficient** — a bug identical on both backends passes it; `clj` corpus + torture - + direct probing are the backstops (already project doctrine; reinforced here). -- No `chain` slot is added to `LazySeq`; the hot `force` path is unchanged. - -## Affected files - -- None land. (Reverted: `src/runtime/lazy_seq.zig`, `src/lang/primitive/higher_order.zig`, - `src/lang/clj/clojure/core.clj`.) -- `.dev/debt.yaml` — D-445 row (fused-reduce proper form, reduce path). -- This ADR + the survey note — the preserved investigation. diff --git a/.dev/decisions/0147_regex_perf_prefilter_lazy_dfa.md b/.dev/decisions/0147_regex_perf_prefilter_lazy_dfa.md deleted file mode 100644 index e27c25238..000000000 --- a/.dev/decisions/0147_regex_perf_prefilter_lazy_dfa.md +++ /dev/null @@ -1,234 +0,0 @@ -# ADR-0147 — Regex perf: borrow-and-adapt the literal-prefilter + lazy-DFA techniques into cljw's Pike-NFA, equivalence-locked (goal = properly incorporate the optimizations, not just beat Python) - -- **Status**: **Accepted** (2026-06-16). Stages 1 (O-034/O-035) + 2 (O-036 leading - first-byte prefilter) committed + WIRED (the default matcher win). Stage 3 (lazy DFA) - was built as decided = **Alternative 2 — forward + reverse lazy DFA** (`dfa.zig`, - equivalence-locked vs the Pike VM), **but NOT wired** — direct measurement (§ Stage 3 - measured outcome) showed the wired DFA *regresses* regex_count (17→23 ms) and sparse - `\d+` (0.05→0.47 s) vs the S2-prefiltered Pike VM, because the DFA's forward scan does - not use the S2 prefilter (visits every byte) and its reverse pass is uncached. The DFA - stays a committed, correct, RESERVED engine (D-449); the Pike VM + S2 prefilter remains - the default matcher. Alt 1 (anchored-restart-only) re-rejected as cycle-budget-defer; - Alt 3 (anchors/`\b` in the DFA state) is a follow-up ADR. - Authored 2026-06-15 as the next-session bridge (user-directed "腰を据えて … 工夫をしっかり入れ込む"). -- **Relates to**: ADR-0031 (regex engine choice — Pike NFA over backtracking, the - reserved lazy-DFA design `src/runtime/regex/dfa.zig`), ADR-0145 (regex after sieve + - nested_update, with the cross-lang equivalence audit as the gate — DONE), F-002 - (finished-form: do the optimization properly), F-011 (clj-equivalence — locked by - the 48-golden corpus), F-015 (completion-grade), the perf campaign §9.2.S / - memory `perf-beat-python-every-bench`. -- **Audit (the ADR-0145 gate, DONE)**: `private/notes/9.2.S-regex-equivalence-audit.md` - + `test/diff/clj_corpus/regex_equivalence.txt` (48 goldens, committed = the F-011 - optimization-safe surface). - -## Context - -`regex_count` (cljw 45ms vs py 26ms = 1.74×) is the last open Python-loser. cljw's -matcher is a **Pike NFA** (Thompson thread-list VM, non-backtracking, ReDoS-immune, -ADR-0031) in `src/runtime/regex/{compile,match,value}.zig`; re-seq is eager (`.clj`). -The cross-lang equivalence audit is done (GO): 48/48 vs clj; cljw's 5 gaps (reluctant -quantifiers, lookbehind, named groups, backrefs, `\A\z\Z`, POSIX `[[:class:]]`) all -**raise NotImplemented (never silently wrong)**. - -**User direction (2026-06-15): do this properly.** The goal is NOT "barely beat -Python" — it is to **incorporate the proven fast-regex techniques into cljw's -architecture**, adapting (not using) the Zig/C/Rust references. "どうせやるなら -しっかり." So this ADR sets the *full* approach, staged, rather than the audit's -minimal 3-lever path. - -## What the fast engines actually do (research synthesis, 2026-06-15) - -Engine families: **backtracking** (PCRE/Python `sre`/Java — feature-rich, ReDoS, -O(2ⁿ)) · **Thompson-NFA / Pike VM** (grep/RE2-fallback/**cljw** — linear, ReDoS-immune, -captures+`\b` are the cost) · **DFA** (RE2/grep/Rust — fastest table-walk, no captures, -O(2ᵐ) build blowup). The fast safe engines (RE2, Rust `regex`/burntsushi, the Zig -`ezi-gex`) layer these techniques, by ROI: - -1. **Literal prefilter (biggest practical win)** — extract the pattern's determinable - leading literal/class and SKIP via fast byte search to candidate positions; run the - engine only there. `\d+` → memchr/byte-class-scan to the next digit. memchr/Two-Way - (single literal, SIMD rare-byte), Teddy (multi-literal, `pshufb`), Aho-Corasick (huge - sets). **This is why Python/Java's `findall` is fast — the prefilter, not the VM.** -2. **Lazy DFA** (RE2/grep core) — build the DFA on-the-fly, cache visited states, fall - back to NFA on cache thrash. ~DFA speed, no O(2ᵐ) build. cljw's ADR-0031 reserved - `dfa.zig` for exactly this. -3. **Two-pass for captures** — find match bounds with the fast DFA, extract captures with - the NFA only inside the matched region. -4. **reverse-DFA two-pass** (ezi-gex) — "prone" patterns (`\w+@\w+`, start-everywhere) use - forward-end + reverse-leftmost-start; "non-prone consuming loops" (`\w+`, `\d+` — the - bench) use **anchored-restart on the eager-DFA frozen table** (one pass). -5. **NFA opts** — sparse states + literal tries + UTF-8 minimal DFAs (Daciuk) to kill the - ε-closure cost that dominates a naive Thompson VM. -6. **SIMD + zero-alloc** — prefilters are SIMD; caller-supplied Scratch (no per-match alloc). - -## Decision (Proposed — the approach) - -Keep cljw's **Pike NFA as the safe base** (ReDoS-immune, F-011, ADR-0031 — do NOT -reintroduce backtracking) and **incorporate the techniques in stages**, each -**equivalence-locked** by the 48-golden corpus (48/48 after every change) + the diff -oracle, each **measured** (`bench/benchmarks/35_regex_count`): - -- **Stage 1 — the supporting shaves** (equivalence-neutral, quick): (a) `seen` - generation-counter in `match.zig` `tryMatchAt` (O(1) clear vs per-position `@memset`); - (b) a `rt/re-find-all` Zig primitive backing `re-seq` (one pass, one ThreadList reuse, - removes the ~10ms `.clj` layer). Audit measured (a) at 45→41.5ms. -- **Stage 2 — the literal/first-byte/class PREFILTER** (the beat-Python lever): extract - the Program's determinable leading byte-set at compile time (a 256-bit bitmap on the - `Program`); `findFrom` skips to the next position whose byte is in that set instead of - restarting `tryMatchAt` everywhere. Disabled when no leading set is determinable - (`.`/`^`/wide alternation). Equivalence-neutral (only skips provably-non-startable - positions). Borrow the SIMD byte-scan shape from `ezi-gex/src/engine/memmem.zig` + - `backends/literal.zig` (rare-byte `@Vector` masks) — re-derive in cljw's Zig 0.16 - idiom; a scalar byte-class scan is the portable floor, SIMD the accelerator. -- **Stage 3 — the lazy DFA** (the structural ceiling, ADR-0031's reserved `dfa.zig`): an - on-the-fly DFA over the byte automaton with a bounded state cache + NFA fallback; - anchored-restart for `\w+`/`\d+`-shaped consuming loops (ezi-gex's `auto` routing). This - is the "しっかり" half — it makes cljw's regex fast across patterns, not just the bench. - Captures still ride the Pike VM (two-pass: DFA bounds → NFA captures-in-region). -- **NO-GO**: backtracking (ReDoS), a machine-code JIT (ADR-0145 / cross-platform), a - fully-compiled eager DFA for untrusted patterns (O(2ᵐ) build). Reluctant quantifiers + - the `compile.zig:22-23` stale lookaround-doc are an optional parity-gap follow-up - (D-051/D-447), pairing naturally with the engine work. - -## Borrow-and-adapt discipline (references, NOT dependencies) - -cljw does NOT depend on or vendor any Zig regex lib — it **reads the references for the -algorithm + re-derives** in cljw's own engine (`no_copy_from_v1` spirit extended to OSS -refs; the 48-golden corpus is the F-011 equivalence proof that the re-derivation is -correct). Cloned / available references (`.dev/reference_clones.md` § Perf-reference): - -- **`~/Documents/OSS/ezi-gex/`** (cloned 2026-06-15) — a Zig Thompson-NFA engine with - Literal Prefilter + Lazy/Eager DFA + Teddy + zero-alloc, WASM-compatible. **Targets Zig - 0.17.0-dev — will NOT compile on stable 0.16** (so it is a *source blueprint*, not a - dependency). The directly-applicable files: `src/engine/memmem.zig` (2-byte rare-byte - SIMD), `src/engine/teddy.zig`, `src/engine/simd.zig`, `src/engine/backends/{literal, - dfa,edfa,pikevm,auto}.zig`, `src/engine/redos.zig`. -- **burntsushi `regex-internals`** (https://burntsushi.net/regex-internals/) + the Rust - `regex-automata` architecture — the authoritative meta-engine design (PikeVM + lazy DFA - + bounded backtracker + one-pass + literal prefilters + Teddy). -- **RE2** (Google) — the safe-non-backtracking lazy-DFA design (refuses backrefs, like - cljw). **CPython `Modules/_sre`** (the `SRE_OP_LITERAL` prefix scan) + **OpenJDK - `java.util.regex`** (`BnM` Boyer-Moore prefilter) — clone on demand. - -## Process (the implementing session) - -Per the user's workflow preference (2026-06-15): **the main agent does the -design-exploration + measurement DIRECTLY** (read `ezi-gex`/RE2/burntsushi, reason, -measure-first), NOT via a survey/DA fork — premise errors hide in forked summaries (the -ADR-0146 filter-chain lesson: a forked survey propagated a wrong "depth-bound" premise -that direct measurement falsified). Reserve forks for the **well-specified mechanical -implementation** (the build-iterate-test loop, once the design + the prefilter byte-set -extraction are settled by direct reasoning). Each stage: design directly → implement -(self or fork) → diff oracle + corpus 48/48 + bench re-measure → commit. Stamp this ADR -Accepted (with the Devil's-advocate pass on the lazy-DFA structural choice) when Stage 2 -or 3 is committed; Stages can land as separate commits under this one ADR. - -## Alternatives considered (Stage 3 lazy-DFA structural choice — Devil's-advocate pass, 2026-06-16) - -Forked `general-purpose` subagent, fresh context, briefed with the F-002 / F-011 / -F-015 / ADR-0031 envelope. Output reflected verbatim: - -> ### Alternative 1 — smallest-diff: the proposed minimal slice, but with the leftmost-start bug made explicit -> -> **What it is (concrete).** A new `src/runtime/regex/dfa.zig` implementing an on-the-fly cached lazy DFA whose state is the sorted NFA PC-set in priority order with cut-on-match (leftmost-first). It handles only **anchor-free, look-free, capture-free consuming** patterns; the compiler tags each program with a `dfa_eligible: bool` (set false the moment an `anchor`/`look`/`save` Inst appears, or `$`/`\b` IR is emitted). `find` walks leftmost-start by **anchored-restart from each S2-prefilter candidate position**: at candidate `i`, run the DFA forward from `i`; on a match-state, record the end; the first candidate that produces any match wins the leftmost rule because S2 already enumerates candidates in increasing position order. Transitions are cached in caller-owned scratch keyed by `(state_id, byte_class)` using ezi-gex's byte-equivalence-class compression of the 256-alphabet. The Pike VM stays the base and the sole capture engine. -> -> **What it does better than the proposed minimal slice.** Essentially nothing in capability — it *is* the proposed slice — but it removes the brief's one buried hazard: "leftmost-start via anchored-restart from S2 candidates" is only correct when **every** match start coincides with an S2-prefilter member byte. That is true for a leading exact first-byte set (the only S2 form that currently exists), false the instant S2 grows a more permissive prefilter (e.g. a memchr-of-literal-substring anywhere, or a `.`-leading pattern that disables the prefilter entirely → candidate set = all positions → the restart is Θ(n²) again). This alternative pins that down: it asserts `dfa_eligible ⟹ S2 produced an exact-leading-byte-set candidate stream`, and DECLINEs to the Pike VM whenever S2 is in its degenerate all-positions mode. So it is the same constant-factor win as proposed but with the correctness precondition made a compile-time gate instead of an implicit assumption. -> -> **What it breaks / risks.** It cements the O(n²) worst-case class (brief acknowledges this — the current Pike `find` is already Θ(n²) on `\d+x` over a long digit run). It ships a *second* engine that must be kept byte-identical to the first forever, doubling the surface the diff oracle must cover, while delivering only a constant-factor improvement on the eligible subset. Under F-002 this is the textbook smallest-diff bias: it leaves the actual quadratic differentiator (the reverse DFA) unbuilt and creates a permanent two-engine maintenance tax for a constant. -> -> **Equivalence-lock story.** DFA state = priority-ordered PC-set with cut-on-match ⟹ leftmost-first identical to the Pike VM by construction (same thread-priority discipline, just memoised). The lock is mechanical: feed every eligible pattern in the 51-golden corpus through *both* dfa.zig and the Pike VM in the dual-backend oracle and assert identical spans; add a fuzz harness that generates random eligible patterns + inputs and diffs the two engines (cheap because both are in-process). The `dfa_eligible` gate guarantees ineligible patterns never reach the new path, so the corpus's anchor/look/capture goldens are untouched. -> -> ### Alternative 2 — finished-form-clean: forward + reverse lazy DFA, true O(input) `find` -> -> **What it is (concrete).** Build the full ezi-gex shape: dfa.zig hosts **two** lazy DFAs over the same byte-level `Inst` IR — a forward DFA and a **reverse DFA** compiled from the program run backwards (reverse each Inst's edges; swap match/start roles). `find` is the classic two-pass RE2 algorithm: (1) forward lazy DFA from the start of input (or from the first S2 candidate) until it reaches a match state → this yields the match **end** offset `e` in O(input) with no restart; (2) reverse lazy DFA from `e` walking backwards until *its* match state → yields the leftmost **start** offset `s`. Both DFAs are lazy/cached in caller-owned scratch with byte-equivalence-class alphabet compression and the priority-ordered cut-on-match state representation for leftmost-first. The eligibility gate is the same (anchor-free / look-free / capture-free consuming patterns); on a hit, the Pike VM is invoked **once** over the now-known `[s, e)` window for capture extraction (two-pass span-then-capture, exactly as the brief frames the Pike VM's residual role). Anchors that are *program-global* (`\A`, `\z`, fully-anchored patterns) are cheap to fold into the forward/reverse start conditions and can stay eligible; only *interior* position-dependent constructs (`\b` mid-pattern, `$` as interior multiline, lookarounds) force DECLINE. -> -> **What it does better than the proposed minimal slice.** It kills the quadratic. `\d+x` over a megabyte of digits is O(input) instead of Θ(n²) — this is the *entire* reason the brief cites the reverse DFA as "crucially" part of ezi-gex, and the reason the minimal slice is explicitly a constant-factor-only win. It also removes the dependency on S2's candidate stream being exact-leading-byte-only: the forward DFA scans from position 0 (or memchr-skips to the first plausible byte) and finds the end in one pass regardless of how permissive the prefilter is, so it composes cleanly with any future S2 enhancement instead of silently regressing when S2 grows. It makes `find` and `re-find-all` (Stage 1's one-pass driver) both linear, which is the throughput story Stage 3 was opened to deliver. -> -> **What it breaks / risks.** The reverse DFA is the genuinely hard part: compiling the reverse program correctly (especially around zero-width and greedy/lazy quantifier priority) is where leftmost-*longest* vs leftmost-*first* subtleties bite, and a reverse-pass priority bug produces a *wrong start offset* that the forward-only tests won't catch. Two cached DFAs double the scratch-memory footprint and the cache-eviction policy must be defined (lazy DFAs can blow their state cache on adversarial-but-non-backtracking inputs → need a "cache full ⟹ flush, or fall back to Pike VM" path, which is itself a correctness-neutral but must-be-tested branch). Risk: the reverse pass interacts with byte-equivalence classes (the reverse alphabet's classes are not the forward classes) — two class tables to derive and keep consistent. -> -> **Equivalence-lock story.** Strongest of the three. The forward DFA's end-offset is locked against the Pike VM's match-end (run both, diff). The reverse DFA's start-offset is locked against the Pike VM's match-start. Critically, because the Pike VM remains the capture engine and is invoked over `[s, e)` on every hit, the *final returned match including all groups* is always produced by the existing, already-corpus-locked Pike VM — the DFA only narrows the window the Pike VM runs over, so a DFA span bug manifests as "Pike VM given the wrong window → group offsets shift" and is caught by the existing 51-golden corpus the moment any golden is eligible. Add: (a) a property test asserting `dfa.find(p,s) == pike.find(p,s)` over fuzzed eligible inputs including the pathological `\d+x`-class quadratic triggers, and (b) an oracle assertion that the cache-full fallback path produces identical spans to the no-fallback path. -> -> ### Alternative 3 — wildcard: lazy DFA over the *full* IR via an in-DFA-state position/assertion lattice (anchors & word-boundaries eligible) -> -> **What it is (concrete).** Drop the eligibility gate's anchor/look exclusions by encoding position-dependent assertions **into the DFA state itself**, the way RE2/Rust-`regex` handle `^`, `$`, and `\b`. The DFA alphabet is extended from "byte class" to "byte class × entry-context", where entry-context is a small bitset capturing the facts an assertion can test at a position: `was-prev-byte-word`, `is-at-text-start`, `is-at-text-end`, `is-at-line-start`, `is-at-line-end`. The lazy state-construction step, when it crosses an `anchor`/`look` Inst, consults the current entry-context bits to decide whether the zero-width assertion passes, and the cached transition key becomes `(state_id, byte_class, context_bits)`. This makes `^`, `$`, `\A`, `\z`, `\b`, `\B`, and multiline anchors all **DFA-eligible** — only true *lookahead/lookbehind over arbitrary sub-patterns* (which cljw's `look` Inst may or may not generalise to) stays on the Pike VM. Combined with the Alt-2 forward+reverse pair, almost the entire non-capturing regex surface runs on the linear DFA. -> -> **What it does better.** Maximal coverage: the common real-world patterns the minimal slice DECLINEs — `^\d+$`, `\bword\b`, `foo$` — all get the linear DFA + reverse-start treatment, so anchored line-scans over big inputs go linear too, not just the bare `\d+` case. This is the closest to "a real production regex engine" and the strongest answer to the user's "腰を据えて工夫をしっかり入れ込む" directive. It also subsumes Alt 2 (Alt 2 is this with the context lattice pinned to the empty set). -> -> **What it breaks / risks.** Highest complexity and the sharpest equivalence cliff. `\b` semantics must match JVM Clojure's `java.util.regex` Unicode word-boundary definition exactly — if cljw's IR is byte-level (the brief says it is) but Java's `\b` is codepoint/Unicode-property-level, then a byte-granular `was-prev-byte-word` bit is **wrong for multibyte UTF-8** and would silently diverge on non-ASCII (an F-011 violation). This is the leading risk and it is *not* hypothetical: it is the exact place a byte-level DFA and a Unicode-aware oracle disagree. The context-bit lattice also multiplies the cache key cardinality (state × class × context), worsening the cache-blowup pressure from Alt 2. And the reverse DFA + context bits together is the most subtle correctness territory in the whole engine. -> -> **Equivalence-lock story.** Same forward/reverse-vs-Pike-VM diffing as Alt 2, **plus** a mandatory dedicated `\b`/anchor sub-corpus that includes non-ASCII / multibyte inputs (the failure surface), diffed against the JVM oracle directly — not just against the Pike VM, because if cljw's *own* Pike VM `\b` is also byte-granular then DFA-vs-Pike agreement would be a false green (both wrong, identically). The lock therefore must be DFA-vs-`clj`-oracle on the word-boundary sub-corpus, and the eligibility gate must DECLINE any `\b` pattern whose input is detected non-ASCII until the Unicode-word-boundary semantics are proven byte-identical to Java. -> -> ### Recommendation -> -> **Build Alternative 2 (forward + reverse lazy DFA).** Per F-002, the finished-form-clean shape wins and cycle/LOC is not a constraint, so the minimal slice (Alt 1) is rejected outright: it is a constant-factor win that leaves the project's actual quadratic differentiator — the reverse DFA the brief itself flags as "crucial" — unbuilt, while imposing a permanent two-engine maintenance tax; choosing it on effort grounds would be the forbidden cycle-budget-defer smell. Alt 2 delivers the true O(input) `find` that is the entire point of Stage 3, composes cleanly with future S2 work, and has the strongest equivalence-lock because the Pike VM stays the corpus-locked capture engine over a DFA-narrowed window. Alt 3 is the genuinely-finished production shape and the right *follow-up*, but its `\b`/anchor coverage rides on resolving the byte-level-IR-vs-Unicode-word-boundary semantics (a real F-011 exposure that must be proven against the `clj` oracle, not the Pike VM, before any `\b` pattern is admitted) — so it should be a separate ADR after Alt 2's forward/reverse machinery is locked, not folded into this slice. -> -> **Note (not a halt):** none of the three requires violating an F-NNN as proposed. Alt 3 *contains* a latent F-011 violation (byte-granular `\b` on multibyte input) that is avoided by its eligibility gate DECLINEing non-ASCII `\b` until proven — flagging it here as the leading finding so the loop scopes Alt 3 around it rather than discovering it in the oracle. - -**Main-loop decision**: Alt 2. The DA's recommendation aligns with F-002; Alt 1 was the -loop's initial (cycle-budget-driven) instinct and is re-rejected per CLAUDE.md's -cycle-budget-defer rule. Alt 3 → a follow-up ADR (the byte-vs-Unicode `\b` exposure is -its scoping constraint; cljw's `\b` is currently `isWordByte` = ASCII-only in -`match.zig`, so the Alt-3 sub-corpus must diff against `clj`, not the Pike VM). Stage 3 -builds the forward + reverse lazy DFA in `src/runtime/regex/dfa.zig`, span-only, with the -Pike VM as the two-pass capture engine + the DECLINE fallback. **Reverse-DFA scope note**: -cljw's current Pike-VM `find` is itself per-position anchored-restart (same Θ(n²) class), -so even a forward-only milestone is not a regression — but per the DA the *committed* S3 -form is the forward+reverse pair, not a forward-only intermediate. - -## Stage 3 measured outcome (2026-06-16 — measurement falsified the "DFA wins" premise) - -The Alt-2 forward+reverse lazy DFA was built and equivalence-locked (`dfa.zig`: -`find` == Pike VM span over a 120-pair fuzz matrix + leftmost-first + quadratic-shape -cases). It was then WIRED behind a Regex-Value-cached `LazyDfa` (eligible programs; -captures via two-pass Pike-VM `matchAnchored`) — clj-equivalence held (full corpus -3120/3120, captures correct, anchors→Pike fallback). **But the wired DFA REGRESSED -perf** (ReleaseSafe, measured directly): - -| workload | S2-prefiltered Pike VM | wired lazy DFA | result | -|------------------------------------|------------------------|----------------|-----------------| -| regex_count (dense `\d+`, 10000×) | 17 ms | 23 ms | **~35% slower** | -| sparse `\d+` (~4000-char, 20000×) | 0.05 s | 0.47 s | **~9× slower** | - -**Why** (the falsified premise): the DA's recommendation assumed the DFA's O(input) -`find` would win. It does not, for cljw's regime, because (1) the DFA's forward scan -does NOT use the S2 leading-byte prefilter — it visits *every* byte, where the -prefilter skips to candidate-start bytes (the whole sparse win); (2) the reverse pass -is uncached (recomputes the closure per backward step); (3) for cljw's typical SHORT -regex inputs the forward+reverse two-pass overhead exceeds a single prefiltered Pike -pass. The DFA's asymptotic edge only materialises on huge / pathological inputs cljw -rarely sees, and even there it would need the prefilter integrated into its forward -scan to compete. This mirrors the ADR-0146 lesson: a forked/textbook premise ("the -reverse DFA is the point") that **direct measurement falsified**. - -**Decision**: do NOT wire the DFA (a regression is not shippable; regex_count was -already CLOSED by S1+S2). The wire-in (Regex-Value cache + `findLeftmost`/`collectBounds` -dispatch) was reverted; `dfa.zig` stays committed as a correct, equivalence-locked, -RESERVED engine. Re-wiring requires (a) integrating the S2 prefilter into the DFA's -forward `findEnd` start-scan + (b) caching the reverse transitions, and even then likely -only wins above a large input-size gate — tracked as **D-449** with a review trigger (if -no huge-input workload + the optimization is not pursued, remove `dfa.zig` rather than -let it rot as dead code). The S2-prefiltered Pike VM is the finished-form default matcher -for cljw's workloads. - -## Consequences - -- The regex unit is a **multi-cycle engine unit** done properly (prefilter → lazy DFA), - not a one-cycle alloc shave. Stage 2 (prefilter) is the beat-Python milestone; Stage 3 - (lazy DFA) is the "incorporate the optimization" goal (fast across patterns). -- F-011 is held by the 48-golden corpus (48/48 after every change) + the diff oracle. - cljw stays ReDoS-immune (no backtracking) and cross-platform (no machine code). -- New tracked refs: `ezi-gex` cloned; ADR-0031's `dfa.zig` finally built in Stage 3. -- A follow-up debt **D-447** (regex parity gaps: reluctant quantifiers / lookbehind / - named groups / backrefs / `\A\z\Z` / POSIX classes — currently NotImplemented) is the - home for the equivalence gaps the perf work does not need but can opportunistically - close (reluctant quantifiers pair with the engine work). - -## Affected files (the implementing session) - -- `src/runtime/regex/match.zig` (seen-gen, the prefilter byte-scan in `findFrom`), - `compile.zig` (leading byte-set extraction onto `Program` + the stale doc fix), - `dfa.zig` (Stage 3 lazy DFA), `value.zig`. -- `src/lang/primitive/regex.zig` + `src/lang/clj/clojure/core.clj` (`rt/re-find-all` + - re-seq re-point). -- `test/diff/clj_corpus/regex_equivalence.txt` (extend as gaps close, anti-D-177) + - `.dev/optimizations.md` (O-NNN rows). `.dev/debt.yaml` D-447 (parity gaps). diff --git a/.dev/decisions/0148_fastest_script_interpreter_campaign.md b/.dev/decisions/0148_fastest_script_interpreter_campaign.md deleted file mode 100644 index 131f5ac90..000000000 --- a/.dev/decisions/0148_fastest_script_interpreter_campaign.md +++ /dev/null @@ -1,142 +0,0 @@ -# ADR-0148 — Perf campaign goal raised: fastest script interpreter on the 9 top-gap benches (ratio_sum…sieve), via beyond-v0 exploration + experiment-and-revert - -- **Status**: **Accepted** (2026-06-16, user-directed). Extends the §9.2.S perf - campaign: the goal rises from "beat Python on every bench" to **"be the FASTEST - script-language interpreter"** — fastest among `{cljw, Python, Ruby, Node.js, - Babashka}` cold-start — on the 9 benches where cljw is currently *not* fastest - AND the gap is above noise. This is a campaign-direction ADR (the per-optimization - design choices it surfaces each get their own ADR + Devil's-advocate fork per the - standing rule); user-directed, so the campaign *definition* itself needs no DA pass. -- **Relates to**: §9.2.S (the perf campaign), `.dev/perf_campaign_essence.md` (the - reminder SSOT, updated alongside), F-002 (finished-form), F-011 (equivalence vs - clj), F-015, memory `perf-beat-python-every-bench` (updated). Supersedes the - "beat Python" framing of the campaign goal. - -## Context (the 2026-06-16 cross-lang re-measurement) - -The refreshed `bench/cross-lang-latest.yaml` (all 8 toolchains, 2026-06-16, hyperfine -5 warmup + 10 runs) shows cljw is the **fastest script interpreter in 19/30** benches. -The user directed: make cljw fastest-script on the top-gap benches **down to `sieve`** -(the near-ties below — nqueens 1.04× and regex_count 1.04× — are excluded: they are -cold-start-floor / measurement noise, not a real compute gap; regex_count already beats -Python). cljw being slower than a *compiled* language (Java/Go/C) is expected and NOT a -target — the bar is the script-interpreter peer set only. - -## The 9 targets (cold-start µs, 2026-06-16; ratio = cljw ÷ fastest-script) - -| Bench | cljw | fastest-script | ratio | what it measures | optimization direction (hypothesis — the experiments decide) | -|------------------|--------|----------------|--------|----------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------| -| ratio_sum | 110380 | Babashka 35042 | 3.15× | exact-rational (Ratio) harmonic sum ×1000 | per-op BigInt alloc + GCD re-normalisation each step; a fused rational accumulator / lazy reduction; mine JVM `clojure.lang.Ratio` + `BigInteger` internals | -| gc_alloc_rate | 99354 | Babashka 35324 | 2.81× | 200K short-lived vectors — raw GC alloc throughput | mark-sweep alloc path vs a bump/generational allocator; nursery / free-list tuning (F-006 GC strategy) — biggest-impact, largest | -| gc_large_heap | 66075 | Babashka 29705 | 2.22× | GC cycle with 100K live maps — mark-sweep trace cost | full-live re-trace every cycle vs generational; incremental / generational marking (F-006) | -| destructure | 75656 | Babashka 43867 | 1.72× | map+vector destructuring ×100K — binding-form overhead | destructuring expands to intermediate get/nth; compile to direct slot access; cache the binding structure | -| json_parse | 54295 | Python 34109 | 1.59× | parse ~25KB JSON ×100 (read-str = Zig std.json) | std.json→persistent-coll conversion allocates heavily; direct construction, fewer intermediate allocs (Python's C json is the floor — 1.6× is closeable) | -| bigint_factorial | 26396 | Babashka 17715 | 1.49× | 100! via auto-promoting `*` ×1000 — numeric tower | inherent `std.math.big` mul + per-step alloc; a fused BigInt accumulator (reuse limbs) — the deferred lever, likely needs combination with the alloc work | -| nested_update | 20243 | Babashka 14786 | 1.37× | assoc-in/update-in/get-in 3-level ×10K | O-033 already made compute beat Python; residual = per-level HAMT node alloc + cold-start floor | -| string_ops | 28649 | Babashka 21259 | 1.35× | str-convert + count ×100K — string overhead | value→string allocates per call; fast int→string, str fast-path for common types | -| sieve | 20605 | Python 16784 | 1.23× | prime sieve via filter — HOF + list ops | O-031/O-032 made compute CLOSED (~0.96×); residual is the **cold-start floor** (AOT-core load, D-140), not compute | - -**Theme grouping** (campaign ordering follows ROI = gap × tractability): -1. **GC architecture** (gc_alloc_rate 2.81× · gc_large_heap 2.22×; also drags ratio/bigint/nested via alloc) — generational/nursery (F-006). Largest impact, largest effort. -2. **Numeric-tower alloc** (ratio_sum 3.15× · bigint_factorial 1.49×) — fused accumulators that kill per-op alloc; independent of GC arch. -3. **Conversion / binding alloc** (destructure 1.72× · json_parse 1.59× · string_ops 1.35×) — cut intermediate allocation in the prim→Value / value→string / binding-form paths. -4. **Cold-start-floor-bound** (nested_update 1.37× · sieve 1.23×) — compute is competitive/won; the residual is the ~AOT-core-load startup → the **D-140 startup bootstrap cache** lifts both at once (highest cross-target leverage for this group). - -## Methodology (beyond v0 — v0 references are mostly exhausted) - -The cw-v0 precedent (`.dev/perf_v0_baseline.md`, the superinstruction/fused-reduce/ -JIT levers) is **largely mined**. The campaign now sources levers from: - -1. **Other-language internal-implementation study** — read how the *winner* does it: - JVM `clojure.lang.{Ratio,PersistentHashMap,BigInteger}`, Babashka/GraalVM - native-image GC + SCI, CPython `Objects/` (the C alloc + json), Node/V8. The - winner's data structure / alloc strategy is the lever to re-derive cljw-clean (F-004). -2. **Web search** — published techniques (generational GC for managed runtimes, - rational-arithmetic fast paths, bignum limb reuse, persistent-structure alloc). -3. **cljw v1 deep-dive + measurement-driven experiments** — profile the hot path - directly (the actual alloc/trace cost), form a hypothesis, MEASURE before/after. - The hypotheses in the table above are starting points, not commitments. -4. **Recombining deferred items** — a single deferred lever (e.g. the BigInt-reduce - accumulator, the lazy-DFA, a startup cache) may be ineffective alone but pay off - **in combination**; revisit `.dev/debt.yaml` deferred rows for compounding wins. - -## Experiment-and-revert discipline (user-directed, 2026-06-16) - -The solid test suite (F-012 diff oracle ×2 + 3157-golden corpus + `CLJW_GC_TORTURE`) -+ the commit log make bold experimentation safe: - -- **Experiment aggressively; revert freely.** Try a lever; if it does not pay (or - regresses), revert the *code*. **A failed-experiment commit MAY stay in the log** — - the log is the record, and a reverted-but-recorded experiment is a learning, not - debt. (This relaxes the usual "no broken commits"; the gate still guards `main`'s - green-ness — never leave `main` red, only leave *recorded reverted* experiments.) -- **No regressions.** Every landed change keeps the diff oracle + corpus green; a perf - win that breaks equivalence is not a win. -- **Short benches carry noise.** A 1.0–1.2× "gap" can be cold-start-floor or - measurement noise (~3 ms ± 10% spawn floor). Confirm a real gap with enough runs - (hyperfine ≥10) + the compute-vs-cold distinction before chasing it; don't optimize - noise (this is why nqueens/regex_count are excluded). - -## Success criterion - -For each of the 9: cljw is the **minimum cold-start** among `{cljw, Python, Ruby, -Node.js, Babashka}` in a fresh `bench/compare_langs.sh` run (≥10 runs), robust to -re-measurement. Refresh `bench/README.md` (generated) when targets close. A target whose -residual is provably cold-start-floor (sieve, partly nested_update) closes via the D-140 -startup cache rather than per-bench tuning. - -## Flow (the autonomous chain — a clear `/continue` runs this) - -`/continue` → handover Resume contract → **this ADR** → the 9 targets, highest-ROI -first. When the 9 are fastest-script (or provably cold-start-floor + D-140 landed), the -campaign's original remaining front resumes: **VM-perf D-386** (per-instruction -dispatch → superinstructions → JIT, the "close on v0's numbers" goal) and the §9.0 -gap areas. The `.dev/.perf_campaign_active` flag stays set; the -`perf_campaign_essence.md` reminder reflects this goal. - -## Consequences - -- The campaign goal is now peer-relative (fastest-script), not Python-absolute — - Babashka (GraalVM native-image, JVM-grade GC/BigInteger/regex-JIT) is the bar on 8 - of the 9, which pushes cljw's GC + numeric-tower + alloc paths hardest. -- GC-architecture work (generational/nursery) is now on the critical path (it gates - gc_alloc_rate/gc_large_heap directly + drags ratio/bigint/nested) — an F-006-level - decision that will get its own ADR + DA fork when its design is chosen. -- D-140 (startup bootstrap cache) rises in priority: it lifts every cold-start-floor - target at once + is the highest dev-velocity ROI. - -## Measurement update (2026-06-16) — the GC-pair hypothesis is refuted; the lever is dispatch/construction - -First campaign cycle landed O-037..O-040 and `sample`-profiled the top targets. Two -table hypotheses were **empirically corrected** (measure-first, F-002): - -- **gc_alloc_rate / gc_large_heap are NOT malloc-bound.** Both show ~0.5% of leaf - self-time in malloc/free (18/3729 and 14/2806) — the free_pool already recycles - short-lived objects. The table's "mark-sweep alloc path vs bump/generational - allocator" hypothesis does not hold for these benches; a nursery/bump allocator - would not move them. **The GC-arch (F-006 generational) work is therefore - de-prioritised for these two targets** (it may still matter for genuinely - alloc-bound future workloads, but not the campaign's GC pair). gc_large_heap's - residual is ~200K closure invocations (map + reduce) → call dispatch. -- **The universal lever across ratio_sum-residual + the GC pair is dispatch / - construction, i.e. D-386** (sequenced "phase 2" originally). Re-sequence: D-386 is - the primary front once the localized construction/alloc levers are mined. - -Landed (all pushed, diff oracle + corpus 3157 green): -- **O-037** ratio operand zero-copy refs · **O-038** ratio stack-arena scratch → - ratio_sum 108.1→85.8 ms (3.15×→2.45×). -- **O-039** alias BigInt operands → bigint_factorial 26.4→22.8 ms (1.49×→1.29×). -- **O-040** `op_vector_literal` `fromSlice` (was empty+N×conj; mirrors O-026 map - fast path) → **gc_alloc_rate 108.4→48.1 ms (2.81×→1.36×)**, the cycle's biggest win - — construction-bound, not malloc-bound. - -None of the four are CLOSED to fastest-script yet; the residuals are dispatch (D-386). -Next: mine the conversion group (destructure / json_parse / string_ops) for localized -O-040-style levers, then D-386 dispatch. - -## Affected files - -- `.dev/perf_campaign_essence.md` (goal + exploration modes + experiment-revert - discipline), `.dev/handover.md` (resume contract), `.dev/debt.yaml` (D-450 campaign - row + the 9 targets), memory `perf-beat-python-every-bench`. `bench/README.md` is the - measured scoreboard (regenerated as targets close). diff --git a/.dev/decisions/0149_small_ratio_inline_i64.md b/.dev/decisions/0149_small_ratio_inline_i64.md deleted file mode 100644 index 69b72d524..000000000 --- a/.dev/decisions/0149_small_ratio_inline_i64.md +++ /dev/null @@ -1,205 +0,0 @@ -# ADR-0149 — Small-ratio inline-i64 representation (canonical two-tier Ratio) - -- **Status**: **Accepted** (2026-06-16, ADR-0148 campaign — ratio_sum, the lone - far target after cycle 1). Implements the small-ratio lever the ADR-0148 - numeric-tower theme named. Devil's-advocate fork run (mandatory at depth ≥2); - output verbatim in *Alternatives considered*. -- **Relates to**: F-004 (NaN-box slots — Ratio at Group D slot 49), F-005 (numeric - tower: JVM-surface observable, Zig-affine internals, runtime auto-promotion), - F-011 (behavioural equivalence vs clj), ADR-0027 (Ratio slot), ADR-0148 - (campaign), D-450. Builds on O-037/O-038 (ratio operand zero-copy + arena scratch). - -## Context (the measurement) - -`ratio_sum` `(reduce + (map #(/ 1 %) (range 1 51)))` ×1000 is cljw's worst -remaining perf gap (81 ms; 2.34× vs Babashka 35 ms). Decomposition: div-creation -33 ms, reduce-sum +48 ms. Each `(/ 1 x)` (x ≤ 50 — a tiny ratio `1/x`) costs -~600 ns and does **~6 heap allocations** (2 operand Managed coercions + 1 scratch -arena + 2 result BigInt allocs + 1 Ratio struct) — it is ALLOC-bound. The -harmonic accumulator's denominator only exceeds i64 after ~15 of 50 terms, so the -vast majority of intermediate ratios fit i64. Babashka pays JVM BigInteger (fast -gcd + TLAB alloc); cljw pays `std.math.big` + GPA per tiny ratio. - -## Decision - -Add a **canonical two-tier Ratio**: store numerator+denominator **inline as i64** -when both fit i64 in reduced form; otherwise the existing `*BigInt` pair. The -representation is **canonical** — a Ratio is small **iff** its reduced numer AND -denom both fit i64. No value has two representations. Auto-promote to big on any -i64 overflow during construction/arithmetic. Observable behaviour is unchanged -(F-005: JVM-surface, Zig-affine internals, runtime-owned promotion). - -```zig -pub const Ratio = extern struct { - header: HeapHeader, - is_small: u8, // invariant: 1 IFF reduced numer,denom both fit i64 - _pad: [5]u8, - a: u64, // small: @bitCast i64 numer ; big: *BigInt numer - b: u64, // small: @bitCast i64 denom ; big: *BigInt denom -}; -``` - -Fits the existing Group D slot 49 (discriminant in the former `_pad`) — **no new -F-004 slot** consumed. - -## The central design constraint (the DA's leading finding) - -`equal.zig::ratioKeyEq` and the `.ratio` arm of `hash.zig::valueHash` are -**rt-free** (map-key-callable) and compare/hash the BigInt **limbs directly**, -justified by "two equal ratios have bit-identical numer+denom". A small-ratio -representation would break this — UNLESS the **canonicalization invariant** holds: - -> small **iff** reduced value fits i64 ⟹ a small `1/2` and a big `1/2` **cannot -> coexist** ⟹ small-vs-big are **never equal** ⟹ `ratioKeyEq` stays rt-free: -> `if (a.is_small != b.is_small) return false;` then a same-kind compare -> (i64-vs-i64 for small, limb-vs-limb for big). - -Hash parity is **free**: `big_int.managedHash` already routes an i64-fitting -Managed through `hash.hashLong(i)`, so a small ratio hashing its i64s via -`hashLong` produces the identical hash a big ratio of the same value would — and -the "never coexist" invariant means the mixed case never arises. - -**This makes the invariant the linchpin**: EVERY Ratio constructor must funnel -through the canonicalizing allocator (post-reduce `toInt(i64)` collapse to small; -i64-overflow promote to big). A single constructor that forgets the collapse -silently breaks ratio map keys with no rt to repair them. Single-source-of-truth -+ a corpus pin (`(= 1/2 )` and `(hash …)` parity) guard it. - -## Implementation plan - -1. **ratio.zig**: the two-tier struct; `allocFromI64Pair` small fast path (i64 - gcd via `std.math.gcd` on `@abs`, sign-normalise denom>0, **MIN_I64 → - promote to big**); `allocFromManagedPair` post-reduce `toInt(i64)` collapse to - small; `parts(v) RatioParts` accessor (`union(enum){small:{n,d:i64}, big:{n,d:*const BigInt}}`); - `traceGc` = `if (is_small) return;` (small has no GC children); `compareValue` - small/big arms; remove/restate the `@offsetOf(Ratio,"numer")==@offsetOf(BigInt,"m")` - comptime assert (no longer always a pointer). -2. **promote.zig**: `ratioArith` small fast path — i64 cross-multiply with - `@mulWithOverflow`/`@addWithOverflow`; **any overflow bit → fall back to the - Managed path** (a naive `n1*d2+n2*d1` is a silent wrong-answer bug); `partsOf` - handles small (no clone, like O-037); the BigDecimal→Ratio arm funnels through - the canonical allocator. -3. **equal.zig / hash.zig**: `ratioKeyEq` + `.ratio` hash arm branch on - `is_small` (i64 compare/`hashLong` for small) — staying rt-free per the - invariant. -4. **print.zig**: small arm reads i64s directly (no BigInt round-trip). -5. **Tests**: unit (small/big/collapse/promote/MIN_I64/overflow-cross-mul) + - diff oracle + corpus pins for cross-rep `=`/`hash`/`compare` + `clj_diff_sweep`. - -## Edge cases (must-handle — naive impl bugs) - -- **i64 cross-multiply overflow** even when operands fit (`numer·denom` can - exceed i64) → `@mulWithOverflow` guard, promote on overflow. -- **MIN_I64 negation / `@abs`** overflow in sign-normalise + gcd → promote to big - if either limb is MIN_I64 (Zig `std.math.gcd` is unsigned; `@abs(MIN_I64)` - overflows). -- **Collapse on reduce**: a big-operand operation reducing to a small value MUST - emit small (else `=` breaks); a small operation overflowing MUST emit big. - -## Expected outcome - -The `(/ 1 x)` div path drops from ~6 allocs to ~1 (the Ratio struct) + i64 gcd -(~10× on the 33 ms div part); the early small-accumulator `+` terms go i64. Target: -ratio_sum 81 → ~45-55 ms (toward Babashka 35; the late bignum terms stay big). -Broad win for all small-rational arithmetic. - -## Consequences - -- The canonicalization invariant is load-bearing for correctness (not just perf): - it is what keeps the rt-free map-key path valid. A debt row + corpus pin guard it. -- Concurrency (Gap-area I): the small path is allocation-free and reads-only — no - new shared mutable state (unlike the DA's Alt 3 intern table, rejected). -- F-004 slot budget unaffected (reuses slot 49). - -## Alternatives considered (Devil's-advocate fork, verbatim) - -**LEADING FINDING — no alternative violates an F-NNN.** All three shapes stay -within F-004 (existing Group D slot 49 has room for a flag/union; no new slot -needed) and F-005 (internal small→BigInt auto-promotion is explicitly endorsed). -The hard constraints are not the F-NNNs — they are two already-shipped, rt-free, -field-bit-comparing invariants in `equal.zig` and `hash.zig` that any naive -implementation will silently break. - -**The under-stated risk (all 3 shapes).** `equal.zig::ratioKeyEq` and the -`.ratio` arm of `valueHash` are rt-free and compare/hash the `numer.m`/`denom.m` -limbs directly ("two equal ratios have bit-identical numer+denom"). A small-ratio -rep destroys that: a small `1/2` and a heap-BigInt `1/2` are value-equal but -physically different. Either (a) these gain an `rt` param (ripples through every -rt-free map-key call site) or (b) keep a guaranteed canonical form so the field -compare still holds. **(b) is mandatory** — *"a small ratio reduced from a big -operation MUST collapse to small, and a big ratio MUST never be storable when it -fits i64, or `=`/`hash` break for map keys with no rt to fix it up."* - -**ALT 1 — smallest-diff: tagged union, accessors lazily materialize a BigInt.** -`kind` byte in `_pad`; `allocFromI64Pair` writes i64s, zero alloc; -`asNumer`/`asDenom` keep `*const BigInt` but lazily allocate+cache on first call. -*Better*: smallest ripple (`ratioArith`/`print`/`compareValue` unchanged). *Risks -— a trap*: the `*const BigInt` accessor FORCES a BigInt to exist → the first -arithmetic op (`partsOf` per accumulate) re-allocates → **alloc win evaporates**. -Caching mutates behind `*const` + needs a conditionally-present GC child → -`traceGc` branches on `kind` AND cache-state (a data race for Gap-area I). -`traceGc` marking i64-as-pointer for small = heap corruption if the branch is -missed. `ratioKeyEq`/`valueHash` still read `.numer.m` → break; the union ADMITS -overlap (same value, two forms) — the worst form of the canonicalization problem. -Slot 49, no new slot. *Verdict*: deceptively small, most bug-prone. - -**ALT 2 — finished-form-clean: canonical two-tier, accessors return value-typed -`RatioParts`.** `is_small` discriminant; **invariant: small iff reduced numer+ -denom both fit i64**. `allocFromManagedPair` checks `toInt(i64)` after reduction -→ a big-operand reduction landing on `3/4` collapses to small; a small op -overflowing promotes. **No value has two representations** → `ratioKeyEq` = -`if a.is_small != b.is_small return false` then same-kind compare, still rt-free; -small-vs-big never equal (a big ratio whose value fit i64 would be small by -invariant). Hash: small via `hashLong` on the i64s, which **already matches** -`managedHash`'s i64 path → cross-rep hash parity for free, mixed case never -arises. `parts(v) RatioParts` union accessor; `ratioArith` i64 fast path with -`@mulWithOverflow`/`@addWithOverflow` → Managed fallback on overflow; `print` -reads i64s. *Better*: the invariant makes the rt-free key path correct BY -CONSTRUCTION; delivers the alloc win (no `*const BigInt` materialization); the -small path is complete (arith/print/compare/hash/eq native i64); `traceGc` is -`if (is_small) return;` with no cache ambiguity. *Risks (each test-pinnable)*: -i64 cross-multiply overflows even when operands fit (`@mulWithOverflow` mandatory); -MIN_I64 negation in sign-normalise + `std.math.gcd` unsigned `@abs(MIN_I64)` -overflow → promote; cross-rep equality depends entirely on the invariant holding -→ EVERY constructor must funnel through the canonicalizing allocator (incl. the -BigDecimal→Ratio path) — single-point-of-truth + corpus test; `extern struct` -@bitCast → the `@offsetOf(Ratio,"numer")==@offsetOf(BigInt,"m")` assert becomes -wrong, remove/restate; wider diff (ratio/promote/equal/hash/print) — NOT a reason -to prefer Alt 1 (Cycle-budget-defer smell). Slot 49, no new slot. *Verdict*: the -finished form; the only shape where the rt-free key invariant survives by -construction. - -**ALT 3 — wildcard: keep the representation; make the ALLOCATOR cheap (per-Runtime -small-ratio intern cache + thread-local Managed scratch pool).** Ratio struct -unchanged. (i) Intern table keyed by `(i64 numer, i64 denom)` → cached `Value` -(the harmonic sum produces the same `1/x` every one of the 1000 outer iters — -identical values, so `(/ 1 x)` becomes a lookup after first occurrence); (ii) -reusable per-Runtime Managed scratch arena for `ratioArith`'s reduce. -*Better*: ZERO representational change → `equal`/`hash`/`traceGc`/`print`/asserts/ -slot untouched, none of the rt-free-key landmines touched; exploits value -recurrence across the ×1000 loop (runs the i64 gcd 50× total vs Alt 2's 50×1000); -can stack on Alt 2 later. *Risks*: the intern table is a GC ROOT surface (cached -Values must be roots or dangle — a new root slot per `.dev/gc_rooting.md`; a moving -GC must update them); cache invalidation / unbounded growth (needs a cap or it -leaks); it's a BENCHMARK-shaped fix not a representation fix — a distinct-ratio -workload (continued-fraction expansion) gets only the scratch-pool win; per-Runtime -mutable table = a Gap-area-I data race. Slot: no struct change. *Verdict*: clever, -only shape with zero rt-free-key risk, but solves a narrower problem and trades -representation landmines for a GC-rooting landmine. - -**Recommendation: Adopt Alt 2 (canonical two-tier).** The only shape where the -already-shipped rt-free `=`/`hash` map-key invariant survives by construction (via -"small iff fits i64 in reduced form → small-vs-big never equal"); delivers the -alloc win without Alt 1's `*const BigInt` materialization; its risks are concrete -and test-pinnable, not architectural. Larger diff is not a reason to fall back -(Cycle-budget-defer smell). Fold in Alt 3's scratch-arena reuse as a follow-on -micro-opt only if profiling after Alt 2 still shows the reduce step hot. - -## Affected files - -- `src/runtime/numeric/ratio.zig` (the two-tier struct + canonical constructors + - `parts` + traceGc/compareValue), `src/runtime/numeric/promote.zig` (`ratioArith` - i64 fast path + `partsOf` small + BigDecimal arm), `src/runtime/equal.zig` - (`ratioKeyEq`), `src/runtime/hash.zig` (`.ratio` arm), `src/runtime/print.zig` - (small arm). `.dev/debt.yaml` (canonicalization-invariant pin row), - `.dev/optimizations.md` (the O-NNN), `test/diff/clj_corpus/` (cross-rep pin). diff --git a/.dev/decisions/0150_fabrication_no_collect_region.md b/.dev/decisions/0150_fabrication_no_collect_region.md deleted file mode 100644 index 4859a30ec..000000000 --- a/.dev/decisions/0150_fabrication_no_collect_region.md +++ /dev/null @@ -1,300 +0,0 @@ -# ADR-0150 — Fabrication no-collect region (D-244 #4 alloc-torture rooting) - -- **Status**: **ACCEPTED 2026-06-16.** (Briefly REJECTED earlier the same day on a - premise error — see Revision history.) A bounded no-collect **fabrication region** - brackets each multi-alloc collection BUILDER so a mid-builder stop-the-world - collect is DEFERRED — closing the D-244 #4 alloc-torture corruption. Correct under - F-006 (non-moving mark-sweep). Implemented as `GcHeap.enterFabrication` / - `exitFabrication` + a threadlocal `fabrication_depth` gating the alloc-torture (and - the future ADR-0028 alloc-driven auto-collect) in `gc_heap.zig`. -- **Deciders**: autonomous loop + two Devil's-advocate forks (general-purpose, fresh - context, F-006/F-002/F-011 envelope). -- **Supersedes / relates**: ADR-0028 (mark-sweep GC + the alloc-driven auto-collect - trigger), F-006 (non-moving GC strategy), ADR-0090 (D-244 = the multi-thread - safepoint mechanism), D-244 #4, D-386 sub-step 2 (the op_top register hoist whose - alloc-torture validation this unblocks). - -## Context - -`CLJW_GC_TORTURE_ALLOC=1` forces a stop-the-world mark-sweep collect inside EVERY -`gc.alloc` — the strictest rooting validation (catches an object unrooted across a -mid-alloc collect). It was BLOCKED: any multi-alloc collection builder corrupted -under it. Confirmed live (ReleaseSafe): - -- `[1 2 3 4 5]` → `[1 nil nil nil nil]` (silent corruption) -- `(conj [1 2 3] 4)` → `@memcpy arguments alias` panic — **a single conj**, so the - bug is at the BUILDER level, not just the op_*_literal fabrication ops -- `(conj #{1 2} 3)` / `(hash-map :a 1 :b 2)` / `(vector 1 2 3)` / `(list 1 2 3)` → - OOB / overflow / corruption - -Cause: a builder (`conj` → `alloc(TailNode)` then `alloc(Vector)`; HAMT `assoc` → -leaf + interior nodes; `fromSlice` → a whole `level_nodes: []*HamtNode` tower; -the `list` builtin's `acc = consHeap(rt, x, acc)` cons-fold) holds an intermediate -NODE — a `*TailNode`/`*HamtNode`/`*Cons`, NOT a `Value` — in an unrooted Zig local -across the next alloc. A per-alloc collect sweeps it. - -**Not reachable in normal operation today**: auto-collect is OFF (gc_heap.zig:280); -collects fire only at safepoints / eval-reentry, never mid-pure-Zig-builder. It is a -torture-tool artifact today + a latent hazard once ADR-0028's **alloc-driven** -auto-collect (a `bytes_since_last_gc > threshold` check inside `gc.alloc`) is wired. - -## Decision - -**A narrowest-span no-collect region per builder.** A threadlocal -`fabrication_depth: u32`; each multi-alloc collection builder brackets its body with -`enterFabrication()` / `defer exitFabrication()`. The per-alloc collect (the torture -block in `gc.alloc`, and ADR-0028's future auto-collect) skips when `depth > 0`. - -- **Whole-body bracket, not first-alloc-out.** Deferring the builder's *first* alloc - too is what makes a Zig-caller fold loop safe: in `op_set_literal` / the transient - HAMT-promote loop / the `consHeap` fold, the partial accumulator is held in a Zig - local with **no alloc between builder calls**, so a whole-body bracket removes every - synchronous collect point and the partial is never exposed. (The existing - `gc_self_guard` `?Value` slot — ADR-0090 #3b-step2b — stays as the inert - publication infra for the FUTURE worker/safepoint trigger, a different path not - exercised by the synchronous alloc-torture. It is kept, not retired.) -- **Covers all call sites + nests correctly** (a builder calling a wrapped builder - balances). The **alloc-torture is the completeness guard**: a builder that forgets - the wrap still trips it → self-enforcing regression test. -- **Cost**: 2 threadlocal increments per builder (no lock). Always-on (the region - must be live in production once ADR-0028 auto-collect lands), so the array-map / - small-map single-alloc fast paths pay it too — measured negligible vs the alloc. - -**Scope**: this fixes the SYNCHRONOUS builder window (no eval reentry). The -complementary class — a collect during an eval-REENTRANT lazy-seq realization / -reduce that sweeps the accumulator (e.g. `(into {} (map f (range N)))`, which under -alloc-torture yields a wrong count) — is the `gc_self_guard` / EvalFrame -reentrant-rooting scope (the D-251/ADR-0094 family). It is tracked as **D-244 #4b**, -a distinct and larger effort, NOT closed here. - -**Correct under F-006 (non-moving mark-sweep)**: a collect suppressed across a -BOUNDED builder runs harmlessly just after (nothing relocates; the result is rooted -on the operand stack by then). **No regression vs today** — today no collect fires -mid-builder anyway (folds already pile up until the next safepoint); the region -preserves that and extends it to the future alloc-driven auto-collect case. - -**Forward debt** (barriered on the nearer trigger, not the distal moving GC): when a -MOVING/concurrent GC — or any collect that must run mid-builder and RELOCATE — is -committed, the region must be replaced with published precise roots (a node-level -guard). A non-moving alloc-driven auto-collect (ADR-0028) does NOT require this: -deferring a bounded builder's collect is a legitimate brief critical section, not a -correctness mask. Row: D-244 #4 forward. - -## Why the "never suppress" stance does NOT forbid this (the reversal) - -This ADR was briefly rejected on the belief that a no-collect region contradicts an -already-decided "publish a precise root, do NOT disable the collector" stance. Read -directly, that belief was wrong: - -- **ADR-0090's D-244 decision text** (lines 389-435) decides the **multi-thread - safepoint** mechanism. Its "self-guard not a guard-list" language (410-413) rejects - a per-site guard-LIST for partial **VALUES** in favour of a single `gc_self_guard` - slot — it is about the `op_*_literal` accumulator (a *Value*), and says **nothing** - about the raw `*TailNode`/`*HamtNode`/`*Cons` allocated INSIDE one builder. ADR-0090 - also explicitly defers concurrent/incremental mark to ROADMAP §89.2 under F-006. -- The "Refuses cw v0's `suppressCollection` escape hatch" line lives in - `root_set.zig` (the `gc_self_guard` docstring) — a **code comment** on the - partial-Value slot, loop-amendable, not ADR-0090 decision text and not an F-NNN. -- A bounded no-collect region for a **pure-Zig builder** (no user code, no - eval-reentry) is **not** cw v0's un-scoped `suppressCollection` (which could wrap - arbitrary eval-reentry — the real F-006 hazard). The region never blinds a - safepoint / back-edge collect (`depth` is 0 there). Same verb, different hazard - class. - -The code comment is amended in the same change to record this distinction (still -refused: un-scoped suppression that could wrap eval-reentry; introduced: the bounded -pure-Zig-builder region). - -## Alternatives considered (Devil's-advocate forks, fresh context — verbatim) - -### Fork 1 (design) — recommended Alt A, the no-collect region - -> **Shared premise**: under F-006 (non-moving mark-sweep) a mid-fold collect changes -> nothing semantically — objects don't move; the only corruption is sweeping an -> unrooted partial. In normal operation `gc.alloc` never auto-collects (gc_heap.zig:280), -> so the corruption is unreachable in production today — a torture artifact + a latent -> hazard for a future mid-alloc-collecting GC (which F-006 does not commit to). -> -> **Alt A (no-collect fabrication region)** — a threadlocal depth counter; fabrication -> brackets its fold; the collect skips when >0. *Better*: structurally complete with the -> smallest correct surface — covers the partial AND every internal node regardless of how -> many allocs a builder does; no node-guard infra, no per-alloc overhead, no per-builder -> wiring that rots silently. *Risks*: makes alloc-torture skip mid-fold collects — but -> only the mid-pure-Zig-fold collect F-006 forbids; it does NOT blind any -> safepoint/eval-reentry collect (depth is 0 there). Residual risk is purely forward (a -> moving GC), mitigated by a forward-note + debt row. *Diagnostic value*: preserved for -> every hazard real under F-006; declines only the non-hazard — the rule's sanctioned case. -> -> **Alt B (root every intermediate: partial Value via gc_self_guard + a NEW node-level -> guard)** — *Better*: also correct under a future moving/concurrent GC. *Breaks/gold- -> plating*: pervasive — `fromSlice` holds a whole `level_nodes: []*HamtNode` slice across -> the level loop (a scalar guard cannot root it → a second mini-root-set); `conj` has up -> to 4 unrooted-node windows; every future builder must remember to wire it or silently -> reintroduce the bug. The ENTIRE payoff is correctness against a GC the project has NOT -> committed to (F-006) — Reservation-as-bias compounded by gold-plating. Under non-moving -> mark-sweep it buys nothing A does not, at many times the surface. -> -> **Alt C (single-alloc builder + publish the one internal node)** — *Fatal, verified in -> code*: `fromSlice` does NOT have one internal node; it holds the whole tower. Publishing -> "the one TailNode" leaves the tower unrooted → torture moves the corruption from the tail -> to the tree. To fix C you rebuild Alt B's tower guard → C is either incorrect or Alt B in -> disguise. -> -> **Recommendation: Alt A.** Finished-form for the COMMITTED architecture. Alt B is right -> ONLY if F-006 flips to a moving GC — at which point B's node-guard is subsumed by the -> moving GC's own barrier/handle machinery, so B is not even the right shape for that -> future. - -### Fork 2 (reversal-check) — steelmanned Alt B, then confirmed the reversal - -> **Steelman of Alt B**: a GC's value is *one* invariant — "every live object is reachable -> from a precise root, always." A `suppressCollection`-shaped hatch creates two contracts -> ("rooted" OR "inside a suppress window") that every future collect-trigger must check; -> cw v0 deliberately refused exactly this. Under that lens Alt A is functionally identical -> today but a *worse finished form*. -> -> **Why the steelman does NOT survive F-006**: (1) the precise-root surface is ALREADY a -> union of bespoke mechanisms (permanent_roots, eval_frame chain, binding heads, ns_vars, -> macro_root_slot, AND gc_self_guard — itself an exception to the operand-stack model); a -> bounded pure-Zig critical section is no more a "second contract" than macro_root_slot -> already is. (2) A bounded region over pure-Zig builders that cannot reenter eval is NOT -> v0's un-scoped hatch (which could wrap eval-reentry — the real hazard); same verb, -> different hazard class. (3) "Right shape for a moving GC" cuts AGAINST B: a moving GC -> supplies its own root-fixup machinery, so B's hand-rolled node-guard array would be -> ripped out — B is speculative infra mis-shaped for the future it targets -> (Reservation-as-bias). -> -> **Stance verdict**: ADR-0090's decision TEXT does not forbid Alt A for builder-internal -> nodes; the "never suppress" line is a code-comment preference (root_set.zig:150), -> loop-amendable. **Reversing the rejection is sound, not the opposite premise error**, -> provided: (a) justify on finished-form grounds, not diff size; (b) amend the code -> comment to distinguish the still-refused un-scoped suppression from the introduced -> bounded region; (c) keep the forward (moving-GC) debt row. -> -> **Recommendation: Alt C (hybrid)** — keep the landed `gc_self_guard` for the op-loop -> partial-Value window, add the bounded region for the builder-internal raw-node window; -> collapse to pure Alt A only if gc_self_guard proves subsumed (verify, don't assume). - -**Implementation finding (post-Fork-2)**: verified that under the SYNCHRONOUS -alloc-torture the whole-body region subsumes the op-loop partial-Value window (no -alloc between builder calls in the in-VM folds), so the synchronous fix is pure Alt A -and `gc_self_guard` stays inert future infra (kept). The reentrant partial-Value -window (lazy-realization/reduce) is the genuinely-different D-244 #4b scope. - -## Consequences - -- `CLJW_GC_TORTURE_ALLOC` validates all SYNCHRONOUS collection-building code again - (16 builder probes in `test/e2e/phase16_gc_torture.sh`) → unblocks D-386 sub-step 2 - (op_top register hoist) validation for synchronous builders. -- A real latent UAF/corruption (reachable once alloc-driven auto-collect lands) is - closed for the synchronous builder class. -- Per-builder cost: 2 threadlocal increments (no lock). Always-on; measured negligible. -- The eval-reentrant lazy-realization/reduce rooting under alloc-torture remains open - as **D-244 #4b** (a wrong-count, not a panic — see the `(into {} (map f …))` probe). - -## Affected files - -- `src/runtime/gc/gc_heap.zig` — `fabrication_depth` threadlocal + - `enterFabrication`/`exitFabrication` + the alloc-torture gate (+ the future - auto-collect note). -- `src/runtime/collection/{vector,map,set,list}.zig` (+ `transient/*`) — wrap each - multi-alloc builder (`vector.conj`/`fromSlice`/`assoc`/`pop`; `map.assoc`/`dissoc`; - `set.conj`/`disj`; `list.consHeap`; transient `toPersistent`/`fromSet`). `map.assoc`'s - array-map fast path + `map.fromLiteralPairs` are single-alloc → not bracketed. -- `src/runtime/gc/root_set.zig` — amend the `gc_self_guard` docstring's "never - suppress" line to scope it to un-scoped eval-reentry suppression. -- `test/e2e/phase16_gc_torture.sh` — the 16 alloc-torture builder probes. -- `.dev/debt.yaml` — D-244 #4 discharged (synchronous builders) + the D-244 #4b - (reentrant realization) follow-on + the forward (moving-GC) row. - -## Amendment 1 (2026-07-13, D-559) — the region also defers the safepoint PARK - -**Problem.** This ADR's contract says "a mid-builder stop-the-world collect is -DEFERRED", but the implementation deferred only the SAME-thread torture / future -auto-collect. The alloc-prologue safepoint park (`gc_heap.alloc`, ADR-0090) ran -BEFORE the `fabrication_depth` gate, so a PEER's `stopWorld` parked a worker -mid-bracket — its in-progress builder nodes (raw `*TailNode`/`*HamtNode`/`*Cons` -in unrooted Zig locals) were invisible to the collect and got swept; the resumed -builder then read recycled memory. Surfaced as D-559: a nested-cross-agent send -(`add-watch` on agent A sending to agent B) crashing `@memcpy arguments alias` -(exit 134, ~3-7% under `CLJW_GC_TORTURE_ALLOC=1` with the D-418 window-collect -injection; the same signature this ADR's Context recorded for the original -unbracketed `conj`). `mark_sweep.zig`'s "workers are parked at their -alloc-prologue, where their intermediates are bracket-covered" comment was -false — D-559 is its disproof. - -**Decision.** The alloc-prologue park HONORS the region: skip the park while -`fabrication_depth > 0`; the thread parks at its next safepoint OUTSIDE any -bracket (VM back-edge poll, non-bracket alloc, blocking-lock safepoint, or -unregister). This is the JVM-GCLocker shape (JNI critical regions delay GC; -threads in a region run to completion; the collector waits). No deadlock: a -waiting `stopWorld` holds no lock a bracket needs (brackets are bounded pure -builders — no eval, no locks; their allocs take only `gc_mutex`, which the -collector takes only AFTER the rendezvous). - -- **No park at bracket ENTRY or EXIT** — both are unsound: at exit the just-built - result is an unpublished Zig local; at entry a fold's partial accumulator - (`acc = consHeap(rt, x, acc)`) is an unpublished raw node between brackets. - There is NO safe bracket-boundary park without new rooting machinery; do not - "optimize" one in. -- **Enforcement**: `fabrication_depth` moved to `root_set.zig` (the shared lower - layer); `safepoint.park` AND `safepoint.enterBlocked` now - `assert(fabrication_depth == 0)` — the chokepoint guarding every other park - site (back-edge, lazy_seq, lock_tx, monitor) and the enterBlocked side door - against a future bracket that reaches one. -- **Accepted cost (observable)**: the rendezvous delay is **unbounded in - fold-chain length** — back-to-back brackets with no poll between (a large - `into`'s per-element builders) defer the park until the fold ends. Tracked as - **D-560** with the publish-roots redesign (DA Alt 3 below) as the named - discharge shape; `safepoint.max_stopworld_wait_ns` records the worst wait so - the cliff is observable, not anecdotal. -- **Scope**: this does NOT close the D-556 residual (tree_walk eval ON A WORKER - holds C-stack Value intermediates; the conservative scan remains - collecting-thread-only). At the sanctioned park sites a VM worker's live - values are published (operand stack / eval frames / bindings); the same claim - for tree_walk-on-worker is NOT made. - -**Alternatives considered (Devil's-advocate fork, fresh context — summary; -full text `private/notes/D559-da-critique.md`):** - -- **Alt 1 (smallest-diff, taken)**: the park-deferral + asserts + honest wording - + wait stat. DA verdict: sound (a registered in-bracket worker never counts as - parked, so `collect` can never overlap a bracket); inherits the structural - smell that the fabrication region now couples into the safepoint protocol. -- **Alt 2 (Boehm per-parked-worker conservative stack scan)**: closes D-559 AND - the D-556 residual in one mechanism, no latency cliff. Rejected as - finished-form-HOSTILE on this project's horizon: wasm value stacks are - unscannable, so gap area II (Wasm-edge-native) cannot port it, and it deepens - platform coupling. -- **Alt 3 (wildcard, DA-recommended per F-002)**: flip brackets from - suppress-collect to PUBLISH-ROOTS — `gc.alloc` auto-appends in-bracket - allocations to a threadlocal scratch root array the collector marks; parks - become safe anywhere, no rendezvous delay, no fabrication knowledge in the - safepoint layer, torture gates deletable, wasm-portable. Costs: per-in-bracket- - alloc append, a new clear-point invariant, an ADR-0150 supersede (depth 3). - **Main-loop ruling**: this ADR's accepted premise (F-006 non-moving → the - no-collect region IS the finished form; publish-roots was Fork 2's rejected - gold-plating) still stands — D-559 was an implementation gap in the region's - contract, not evidence the regime is wrong. Alt 3 becomes the named discharge - shape of D-560, triggered by MEASURED pause-time harm (or an F-006 amendment, - user-owned), not pre-emptively. - -## Revision history - -- 2026-07-13 **Amendment 1** (D-559): the region also defers the alloc-prologue - safepoint park (a peer-STW mid-bracket park corrupted worker builders); - `fabrication_depth` moved to `root_set.zig`; park/enterBlocked asserts; the - fold-chain rendezvous-delay cost recorded as D-560 with the publish-roots - redesign (DA Alt 3) as its discharge shape. -- 2026-06-16 issued + **REJECTED same day**, then **REVERSED to ACCEPTED same day**. - The design (Alt A no-collect region) was DA-forked (Fork 1, recommended Alt A). It - was then rejected on the belief that Alt A contradicts ADR-0090's "never suppress, - publish precise root" stance. A reversal-check (Fork 2, steelmanning Alt B) + - reading ADR-0090's decision text directly found that belief rested on **over-reading - root_set.zig:150's code comment as binding ADR text** — ADR-0090 decides the - multi-thread safepoint and rejects a per-site guard-LIST for partial VALUES; it says - nothing about builder-internal raw nodes and defers concurrent GC to §89.2. Under - F-006 the no-collect region is the finished form; Alt B is gold-plating for an - uncommitted moving GC. Reversed and implemented. Design notes: - `private/notes/9.2.S-d244-4-design.md` (rejected-Alt-A record) + - `private/notes/9.2.S-d244-4-decision.md` (the reversal). diff --git a/.dev/decisions/0151_narrow_arm64_integer_loop_jit.md b/.dev/decisions/0151_narrow_arm64_integer_loop_jit.md deleted file mode 100644 index 810e3d755..000000000 --- a/.dev/decisions/0151_narrow_arm64_integer_loop_jit.md +++ /dev/null @@ -1,241 +0,0 @@ -# ADR-0151 — Narrow ARM64 integer-loop JIT (D-133), as a third backend - -**Status**: Accepted (Proposed → Accepted same cycle, 2026-06-16; ADR-level -design handled inline per CLAUDE.md § ADR-level designs are handled inline). - -**Supersedes / relates**: D-133 (the debt row this discharges the design for); -F-010 (interim goal M = Phase 15 + a "cw-v0-程度 JIT"); ADR-0145 (sequenced the -JIT last, after dispatch micro-opts); ADR-0148 (the fastest-script campaign); -ADR-0125 (eval-budget / back-edge poll the JIT must cooperate with); ADR-0090 -(GC safe-point); F-004 / F-005 / F-006 / F-011 / F-012 (the envelope). - -## Context - -The 2026-06-16 measurement arc (private/notes/9.2.S-d386-flatten-path-orientation.md) -empirically **exhausted the non-JIT lever space** for the 5 open ADR-0148 -fastest-script targets (destructure 1.05× / gc_large_heap 1.08× / gc_alloc_rate -1.15× / sieve 1.23× / json_parse 1.14×): - -- Redundancy wins landed (O-048 single-scan `fastGet`, O-049 simple-key - `eqConsult`): destructure 55→45.9 ms (−16.5%), gc_large_heap 33.5→32.0 ms. -- Micro-levers all **inert** (A/B, reverted): TLV-address caching, the per-call - error-trace push (D-386 "lever (a)" — REFUTED, 0.2–0.8 ms), TailNode/ArrayMap - memset-skip, the `gc.alloc` mutex. -- Auto-collect recycling **net-negative** (`CLJW_GC_TORTURE=N`: 42→59→131→510 ms - as collect frequency rises — the STW mark-sweep cost exceeds the malloc - savings). -- The **call-ABI fast-path ceiling measured ~3–4.5%** (A/B stripping both - `treeWalkCall` current_env + trace-push), and that ceiling is unreachable - (current_env is needed for ADR-0129 correctness; trace-push needs a risky - lazy-rebuild). REFUTED as worthwhile. - -So the remaining targets are dispatch / allocation bound, and the decisive lever -is to **stop interpreting the hot integer loop** — a JIT. ADR-0145's gate -("dispatch / alloc / call micro-opts exhausted") is now met. F-010 sanctions -exactly this: a narrow, cw-v0-程度 JIT (~700–1000 LOC, counter trigger, leaf -C-ABI, deopt-on-non-int), NOT a broad/optimizing JIT. - -Step 0 survey (private/notes/9.2.S-d133-jit-survey.md) found the key enabler: -**cljw already ships the superinstruction opcodes cw v0's JIT consumed** -(O-017..O-022: `op_add/sub_{locals,local_const}`, `op_branch_{ne,ge,gt}_*`, -`op_recur_loop`). So D-133 is a pure codegen + trigger addition with **zero new -IR**. - -## Decision - -Add a **narrow ARM64 integer-loop JIT as a true third execution backend** -(the DA-fork's recommended Alt 2, chosen over the lighter draft per F-002), at a -new `src/eval/backend/jit/` subtree. - -1. **Backend seam, not a VM escape hatch.** The JIT exposes a small backend - surface symmetric with `tree_walk.zig` / `vm.zig` — `compileLoop` / `run` / - `deoptToVM` only; the VM holds an opaque `JitCache` and never reaches into - codegen internals. This keeps F-011 (no second mechanism bolted into VM - dispatch) and F-012 (the JIT is a backend the oracle validates, like - VM-vs-TreeWalk) structural rather than prose. -2. **Trigger.** A **named** back-edge counter co-located with the existing - `op_recur_loop` arm (vm.zig) — NOT overloaded onto the GC-poll branch - (Alt 1's coupling hazard). At threshold (~64) compile the loop body once into - a per-VM single-slot `JitCache`. -3. **First milestone (smallest coherent slice).** Compile exactly a **pure-fixnum - `loop`/`recur` accumulator loop** — fixnum add/sub + a fixnum comparison - branch + recur, no `mul`, no calls, no allocation/collection mutation — so the - JITted leaf allocates nothing. (fib = call-recursion, sieve = in-loop - collection mutation; both are explicitly NOT first-milestone.) -4. **Codegen.** Raw u32 ARM64 emission into an mmap'd page; on Apple Silicon use - `MAP_JIT` + `pthread_jit_write_protect_np` toggling + `sys_icache_invalidate` - (via `@extern`; Zig 0.16 `std.posix.mmap` + `std.c.mprotect`). Leaf C-ABI - `(stack_ptr, base, constants_ptr) -> {value, status}`, x0–x17 only, no frame. -5. **Typed deopt** (not a bare `status != 0`): an enum - `{ ok, deopt_non_fixnum, deopt_overflow, deopt_safepoint }`. On a non-fixnum - value (top16 != cljw's **0xFFFC** inline-fixnum tag, F-004) OR on i48 - arithmetic overflow (F-005 forbids cw-v0's silent wrap), the leaf bails - allocation-free without mutating interpreter state and the VM runs the loop - normally (and auto-promotes to BigInt on the overflow path). Cooperates with - the ADR-0125 / ADR-0090 back-edge safe-point by capping native iterations or - polling `gc_requested`. -6. **Oracle-wired day one (F-012).** Every JITted loop runs under the dual-backend - diff harness against its TreeWalk form, asserting bit-identical NaN-boxed - results AND identical deopt→VM→same-result, with a `--backend=jit` selector - mirroring the existing VM/TreeWalk selection. - -### Mandatory carry-overs (DA-fork, all shapes) - -1. **Tag/payload constants come from cljw `value.zig` (F-004 `0xFFFC`), NEVER - ported from cw v0's `0xFFF9`** — the F-011 trap hiding in "modeled on v0". -2. **Overflow-deopt MUST be proven against the VM's BigInt auto-promotion under - the diff oracle + a corpus line** (anti-D-177 over-claim) — never listed as - landed without the probe. -3. **The leaf's allocation-free property MUST be asserted, not assumed** — pair - it with `CLJW_GC_TORTURE` so a stray alloc inside JITted code surfaces as a - deterministic UAF, not silent rooting debt (F-006). - -### Forward debt - -Alt 3 (lower the hot leaf to a **zwasm-JITed Wasm function** — portable to the -edge-native deployment target, gap area II) is **recorded as forward debt**, not -adopted: it strains F-010's letter (not a v0-shaped native leaf), has a per-loop -build/instantiate latency floor a 64-iteration leaf may never amortize, and -mid-loop deopt across the Wasm boundary loses the allocation-free resumable -deopt. Re-open it when broad-JIT / edge-native work opens (a new debt row at that -time; the capability-matrix successor is D-443). - -## Alternatives considered (Devil's-advocate fork, verbatim) - -> Forked `general-purpose` subagent, fresh context, briefed with the proposed -> shape + the active F-NNN envelope. Output embedded verbatim per -> `.dev/principle.md` (Devil's-advocate mandatory at depth ≥ 2). - -### Preface — leading F-NNN finding - -There is no F-NNN that the finished-form-clean direction must violate. F-010 -explicitly *blesses* a "cw-v0-程度 JIT (~700-1000 LOC, counter trigger, leaf -C-ABI, deopt-on-non-int)" as part of interim goal M. The draft sits squarely -inside that envelope. The real tension is **F-011** (no second mechanism -duplicating an existing one) and **F-012** (VM is production default; TreeWalk is -the differential oracle). The draft introduces a *third* execution path, which is -the sharpest F-NNN-adjacent risk to scrutinize — not a violation, but the thing -most likely to drift into one. All three alternatives below stay inside every -F-NNN. - -A grounding correction the draft should absorb: the deopt mask `top16 != 0xFFFC` -is the **cljw F-004 tag** (correct) — but NOT cw v0's tag (`0xFFF9...` INT_TAG). -"Modeled on cw v0's jit.zig" must not copy v0's boxing constants; source them from -cljw's `value.zig`. This is an F-011 behavioural-equivalence trap hiding in the -word "modeled." - -### (1) Smallest-diff — inline trigger, reuse the back-edge poll, no new subtree - -Single `jit.zig` under `src/eval/backend/vm/`; hook at `op_recur_loop` (vm.zig:895) -+ reuse the existing back-edge poll (vm.zig:270) as the warmup counter; per-VM -single-slot `JitState` as cw v0. -- **Better**: reuses the already-paid back-edge branch (zero new cold-path cost), - co-locates the trigger with the GC poll it must cooperate with, smallest oracle - surface, defers the subtree decision. -- **Breaks**: couples the JIT trigger to the GC-poll branch semantics (a future - GC-poll refactor silently moves the trigger — ADR-0125 coupling hazard); a lone - `jit.zig` next to `vm.zig` reads as a VM helper not a peer backend, postponing - the structural decision (Structural-imagination-deferral). -- **F-NNN**: compliant; tag-mask still cljw-native; GC-poll coupling is a risk not - a violation. - -### (2) Finished-form-clean — JIT as a true third backend behind a dispatch seam - -`src/eval/backend/jit/` with a backend interface symmetric with tree_walk/vm -(`compileLoop` / `run` / `deopt-to-VM` only; VM holds an opaque `JitCache`). -Diff-oracle wired from the first commit (every JITted loop + its VM form run under -the dual-backend harness, bit-identical NaN-boxed results + deopt→VM→same-result), -`--backend=jit` selector. Typed deopt enum `{ ok, deopt_non_fixnum, -deopt_overflow, deopt_safepoint }` instead of bare `status != 0`. -- **Better**: serves F-002 + F-012 — the JIT is a backend the oracle validates the - same way it validates VM-vs-TreeWalk, not a special-cased side path; the seam - pre-exists for milestone 2-3 growth (no re-architecture). The typed-deopt result - turns F-005's "deopt, don't wrap" from an honor-system comment into a testable - invariant (overflow → `deopt_overflow` → VM auto-promotes → corpus line locks - it, anti-D-177). Kills the F-011 "second mechanism" risk at the root. -- **Breaks**: substantially larger diff (seam + typed enum + oracle wiring + - selector) vs "pure codegen+trigger, zero new IR"; some seam is speculative until - backend #4 exists; risk of over-abstracting for two-and-a-half backends - (Premature-generalization); the JIT is not *fully* a peer (it deopts to the VM - mid-loop by construction — really a VM accelerator), so forced interface - symmetry slightly misrepresents the dependency. -- **F-NNN**: fully compliant, most F-011/F-012-aligned. Per F-002 the larger diff - is not a reason to downgrade. - -### (3) Wildcard — lower the hot leaf loop to a zwasm-JITed function - -At threshold, lower the detected fixnum leaf loop to a tiny Wasm function (i48 -add/sub + compare + br_if + loop) and hand it to zwasm's existing JIT (F-001: -zwasm unavoidable; cw v0's tree even ships a zwasm with its own jit.zig). Deopt -becomes a guard inside the generated Wasm. -- **Better**: zero new arch-specific codegen / mmap / W^X / icache management; - **portable to every target zwasm runs on, including the Wasm-edge-native - deployment target** (gap area II — a native ARM64 JIT does nothing for a cljw - running inside a Wasm sandbox at the edge); strongest F-011 story (reuses an - existing mechanism — the JIT *is* zwasm). -- **Breaks**: **contradicts F-010's letter** ("a cw-v0-程度 JIT … native leaf - modeled on v0") — F-010-spirit-compliant (it is *some* JIT for hot integer - loops) but letter-divergent; flagged explicitly per the brief. It does NOT - violate F-010 outright (F-010 describes the M milestone's JIT *shape*; a finished - form could amend implementation-shape while keeping the M capability) but - adopting it requires reading F-010 as capability-not-implementation — a - deliberate stretch. Plus: build→instantiate→call latency floor a 64-iteration - leaf may never amortize (v0's whole point was a *cheap* leaf compile); mid-loop - deopt back into the cljw interpreter across the Wasm boundary is genuinely hard - (cannot resume the interpreter at an arbitrary IP with live unboxed values) — you - likely lose the allocation-free, state-non-mutating deopt. -- **F-NNN**: F-001/F-006/F-011/F-012 compliant, most portable; **F-010 - letter-divergent** (capability-compliant, implementation-shape-divergent). Not - recommended as the milestone-M shape. - -### Recommendation - -**Adopt Alternative (2).** The draft is already F-010/F-006/F-005-compliant; the -gap is that its F-012 oracle integration and F-005 deopt guarantee are stated as -prose/comments, where the finished form makes them mechanical (typed deopt + -day-one diff-oracle wiring + a clean backend seam). That is the F-002 -"finished-form cleanliness wins" call, and the larger diff is not a reason to -prefer the draft (cycle/LOC is not a constraint). NOT recommending the -smaller-diff Alt (1) on "one file, mirrors v0, smallest surface" grounds — that is -the Cycle-budget defer smell, and its GC-poll coupling postpones the structural -decision the ADR should settle. Take from Alt (1): the trigger co-locates with the -back-edge site but as an explicit *named* counter, not by overloading the GC-poll -branch. The wildcard (3) is the right long-horizon question for gap-area II -(edge-native) — record as forward debt, do not adopt now. - -Mandatory carry-overs (any shape): (1) tag/payload constants from cljw `value.zig` -(F-004 `0xFFFC`), never ported from v0's `0xFFF9`; (2) overflow-deopt proven -against the VM's BigInt auto-promotion under the diff oracle + a corpus line; (3) -the leaf's allocation-free claim asserted via `CLJW_GC_TORTURE`, not assumed. - -## Consequences - -- A new `src/eval/backend/jit/` backend subtree + a `JitCache` slot on the VM + a - named back-edge counter + a `--backend=jit` test selector. The compiler / IR is - unchanged (zero new opcodes — the superinstruction family already exists). -- ARM64-only initially (the leaf codegen is arch-specific); on a non-aarch64 host - the JIT is inert (the VM runs the loop) — no behaviour change, the diff oracle - still validates VM≡TreeWalk there. (Portability is the wildcard's forward-debt - domain.) -- The diff oracle gains a third participant; a JIT/VM divergence becomes a gate - failure, so the JIT cannot silently diverge (F-012). -- Implementation is the next TDD unit (the arith_loop milestone), not this commit. - This ADR locks the design + the DA analysis; the source lands against it. - -## Affected files (at implementation time) - -- `src/eval/backend/jit/` (new) — codegen + trigger + `JitCache` + deopt. -- `src/eval/backend/vm.zig` — named back-edge counter at the `op_recur_loop` arm; - opaque `JitCache` hook; deopt return path. -- `src/runtime/value/value.zig` — (read-only) the F-004 `0xFFFC` tag/payload - constants the codegen sources. -- the diff harness / `--backend=jit` selector + a corpus line for the - overflow-deopt proof. -- `.dev/debt.yaml` D-133 (status), `.dev/structure_plan.md` (`jit/` subtree), - `.dev/optimizations.md` (the O-NNN at landing). - -## Revision history - -- 2026-06-16 — Accepted. Design + Devil's-advocate fork (Alt 2 chosen per F-002) - captured ahead of the arith_loop milestone implementation. Non-JIT lever space - empirically exhausted (ADR-0148 arc); ADR-0145 JIT gate met. diff --git a/.dev/decisions/0152_volatile_mutable_happens_before.md b/.dev/decisions/0152_volatile_mutable_happens_before.md deleted file mode 100644 index bda22af6f..000000000 --- a/.dev/decisions/0152_volatile_mutable_happens_before.md +++ /dev/null @@ -1,127 +0,0 @@ -# ADR-0152 — `:volatile-mutable` deftype fields get cross-thread happens-before - -- **Status**: Proposed → Accepted (2026-06-16) -- **Context drivers**: D-444 (gap area I — Concurrency, F-015), un-accepts AD-018. -- **Supersedes**: AD-018 (the "unified assignable flag while single-threaded" accept). - -## Context - -On the JVM, `deftype`'s two mutable-field keywords differ only in cross-thread -memory visibility: `^:volatile-mutable` compiles to a `volatile` field (JMM -happens-before ordering); `^:unsynchronized-mutable` compiles to a plain field -(no ordering — sharing it across threads without external synchronisation is a -data race = undefined under the JMM). - -cljw collapses both into one "assignable" flag: a plain in-place slot write -(`TypedInstance.setField`, ADR-0104) + a plain read. AD-018 accepted this on the -rationale "cljw is single-threaded." **That rationale is now false** — future / -agent drainers run user code on real OS threads (F-015 / ADR-0142). A deftype -instance with a `:volatile-mutable` field can be shared (in an atom / ref / -agent) and mutated via a method on one thread while read on another, so the -missing happens-before is a real correctness bug, not a dormant divergence. - -## Decision - -Un-accept AD-018 and make the keyword distinction **observably real**: - -- `:volatile-mutable` field READ → `@atomicLoad(.acquire)`; WRITE → `@atomicStore(.release)`. -- `:unsynchronized-mutable` field → unchanged plain slot access (this is the - JVM-faithful behaviour: a plain field shared across threads is a data race - on the JVM too, so leaving it plain is parity, **not** a hole). - -Mechanism: thread the field KIND (volatile vs unsynchronized) from the deftype -macro's keyword parse onto a per-field flag on `TypeDescriptor`'s field-entry, -and branch the read + `set!` accessor on it. Both backends (tree_walk + vm) -route through one accessor so the branch lives in a single place. - -### Memory ordering: acquire/release, uniform with cljw's volatile-grade substrate - -cljw's entire lock-free substrate uses `.acquire`/`.release` (CAS: `.acq_rel`/ -`.acquire`), never `.seq_cst`: `atom.zig`, `volatile.zig` (whose docstring -already calls a volatile "JMM-volatile"), `iref.zig`, `agent.zig`. -`:volatile-mutable` **joins that convention** rather than becoming a `.seq_cst` -outlier stronger than its own siblings. - -The DA fork (below) correctly notes that literal JMM `volatile` is sequentially -consistent *across independent volatile accesses* (the IRIW / two-independent- -volatiles total-order property), which `.acquire`/`.release` does **not** give — -acquire/release synchronises a release-store with the acquire-load that *reads -that value*, but imposes no total order over independent locations. So -acquire/release is exactly sufficient for the dominant single-field -publish/consume pattern (≥99.9% of real code) and weaker than the JVM only for -the pathological multi-volatile cross-thread-relative-ordering case. - -This residual is **consciously elected** (not a cycle-budget defer): cljw's -finished form is a *uniform* acquire/release volatile-grade ordering across all -its concurrency cells, with the IRIW cross-cell total-order gap recorded once as -an accepted sub-divergence (AD-034) that covers atom / volatile / iref / agent / -volatile-mutable alike. A lone `.seq_cst` for `:volatile-mutable` (or a -project-wide `.seq_cst` sweep for a guarantee no real Clojure code depends on) -would be the *less* clean finished form — internal inconsistency, or a fence -tax everywhere for an unused property. - -Note: a naturally-aligned 8-byte `Value` (NaN-boxed u64) load/store is already -hardware-atomic on aarch64/x86_64 — the atomic builtin is needed only for the -ordering fence + to stop the compiler reordering/coalescing the access. - -## Testability - -Happens-before is non-deterministic and has no sound deterministic test (a -missing fence manifests only probabilistically under contention, and never on -strongly-ordered x86_64). The honest coverage ceiling — the same epistemic -status the existing atom/agent code lives with — is: - -1. **By-construction wiring**: a unit test that a `:volatile-mutable` field - routes through the atomic accessor and `:unsynchronized-mutable` through the - plain one. -2. **Single-threaded preservation**: the existing deftype-mutable e2e/pin stays - green (atomic ops on an uncontended word = ordered plain ops). - -A probabilistic N-thread stress test is **NOT** gate-added (flaky, false-green -on x86_64; a future Layer-7 Concurrency on-demand harness per test_taxonomy.md). - -## Alternatives considered (Devil's-advocate fork, fresh context — verbatim digest) - -- **Alt 1 — smallest-diff (keep AD-018, doc-only warning)**: zero cost / zero - blast radius / honest about the testability ceiling. REJECTED — F-011-non- - compliant: a silently-stale cross-thread read is a correctness bug a warning - does not fix (Defer-to-amnesia / Smallest-diff-bias smell). Leading rejected - alternative. -- **Alt 2 — finished-form (per-field flag + atomic accessor, RECOMMENDED by DA - and CHOSEN)**: correct JMM parity for the publish/consume case; makes the - keyword distinction observably real; keeps `:unsynchronized-mutable` plain - (= JVM parity, not a hole); single-threaded behaviour bit-for-bit preserved. - Costs a `FieldEntry` flag + a predictable per-access branch. -- **Alt 3 — wildcard (route ALL mutable fields through acquire/release, no - per-field flag)**: simplest runtime surgery, never under-orders. REJECTED — - over-orders `:unsynchronized-mutable` (a guarantee the JVM does not give + - re-erases the keyword distinction = a flavour of AD-018 reopened) and taxes - the exact tight-loop use-case the keyword exists to make cheap. -- **Ordering — seq_cst vs acquire/release**: DA's literal-JMM recommendation was - `.seq_cst` (IRIW total order) with a follow-on to upgrade the sibling cells. - This ADR instead elects uniform acquire/release + AD-034 for the IRIW gap, on - internal-consistency + real-code-irrelevance grounds (see Decision). The DA's - caveat — "do not pick acquire/release silently on cycle-budget grounds" — is - honoured: the election is on finished-form/consistency grounds, recorded here. - -## Consequences - -- `:volatile-mutable` fields now have cross-thread happens-before (publish/ - consume); `:unsynchronized-mutable` unchanged (JVM-faithful plain access). -- New **AD-034**: cljw volatile-grade cells use acquire/release; the JMM - cross-independent-volatile (IRIW) total order is not guaranteed (derives from - this ADR's uniform-acquire/release election; real Clojure code does not depend - on it). `pin`: the by-construction wiring test. -- AD-018 removed from the ledger (un-accepted → fixed). - -## Affected files (implementation plan) - -- deftype macro (`src/lang/.../macro_transforms.zig`) — parse `^:volatile-mutable` - vs `^:unsynchronized-mutable`, carry the kind. -- `src/runtime/type_descriptor.zig` — per-field volatility flag on the field - entry + `setFieldVolatile`/`getFieldVolatile` (`@atomicStore(.release)` / - `@atomicLoad(.acquire)`). -- `src/eval/backend/tree_walk.zig` + `src/eval/backend/vm.zig` — field read + - `set!` route a volatile field through the atomic accessor. -- `.dev/accepted_divergences.yaml` — remove AD-018, add AD-034. -- tests — by-construction wiring unit test + keep the deftype-mutable e2e pin. diff --git a/.dev/decisions/0153_future_cancel_cooperative.md b/.dev/decisions/0153_future_cancel_cooperative.md deleted file mode 100644 index 106df75d9..000000000 --- a/.dev/decisions/0153_future_cancel_cooperative.md +++ /dev/null @@ -1,145 +0,0 @@ -# ADR-0153 — `future-cancel` via cooperative cancellation at the worker safepoint - -- **Status**: Proposed → Accepted (2026-06-16); sub-steps 1 + 2 landed - (2026-06-16) — see Revision history. Deref-wait abort residual tracked in D-442. -- **Context drivers**: D-442 (gap area I — Concurrency, F-015) — `future-cancel` / - `future-cancelled?` are NOT-YET-IMPLEMENTED. - -## Context - -cljw futures run a thunk on a DETACHED `std.Thread` (future.zig `worker`), with -no interrupt/cancel mechanism. clj's `(future-cancel f)` = `Future.cancel(true)`: -returns true if cancelled (task pending/running — interrupts the thread), false -if already done; `(future-cancelled? f)` → cancelled?; `(deref )` → -`CancellationException`. cljw cannot OS-interrupt a detached thread. - -Key fact (from the DA fork): **JVM interruption is itself best-effort** — a tight -CPU-loop thunk (`(future (loop [] (recur)))`) is NOT interrupted on the JVM -either (interrupt fires only at blocking points — `sleep`/`wait`/`park`/ -interruptible I/O — or explicit `Thread.interrupted()` checks). So faithful -parity does NOT require aborting arbitrary running code — only cooperatively -aborting at blocking points, exactly as the JVM does. - -## Decision - -Implement cooperative cancellation reusing the existing worker substrate (the -`ThreadGcContext` + GC safepoint each worker already registers/parks at): - -1. **Per-Future `cancel_requested: atomic bool`.** `future-cancel`: under - `cell.mutex`, if state is `.pending` → set `cancel_requested` + (for a not-yet- - started/blocked worker) the cooperative check aborts it; return **true**; else - (`.realised_*`/`.cancelled`) return **false**. (`.pending` covers "running" - since cljw stays `.pending` until the worker stores — so a running future - correctly returns true, matching clj.) -2. **Cooperative check at BLOCKING PRIMITIVES ONLY** (`Thread/sleep`, a nested - future `deref`, promise wait) — **NOT** the per-op GC safepoint (vm.zig:277). - *(Amended 2026-06-16, pre-implementation: the original draft said "at the GC - safepoint + blocking primitives, not every back-edge" — but the GC safepoint - IS the per-op back-edge poll, so checking the cancel flag there WOULD be every - back-edge and would over-interrupt a tight CPU loop. To match the JVM's - best-effort semantics — a blocking thunk aborts; a tight CPU loop runs on, as - on the JVM where interrupt only fires at blocking points — the check lives at - the blocking primitives, leaving the per-op eval loop untouched. This also - keeps the change OFF the load-bearing eval loop.)* A worker in a tight CPU loop - is marked `.cancelled` (future-cancel returns true, future-cancelled? true) but - runs to completion — exactly as a JVM `cancel(true)` on a non-interruptible loop - (interrupt flag set, loop runs on). -3. The cooperative check raises a **distinct cancellation signal** up the - worker's eval stack; the worker's existing catch (future.zig:131-141) marshals - it via `worker_error.capture`. deref re-raises it through the existing - `worker_error.reraise` channel as a **`CancellationException`-classed** value - (add the class to host_class.zig under RuntimeException/Exception) — NOT the - stale `future_thunk_failed` placeholder. The signal must be distinctly typed / - uncatchable so a `(try … (catch Throwable …))` in the thunk does not swallow it. -4. **`FutureState += .cancelled`** for the terminal state + `future-cancelled?`. - The worker's store (future.zig:144-148) is **guarded `if (state == .pending)`** - so a canceller that won the mutex first is not clobbered (the mark-cancelled- - wins ordering; both serialize on `cell.mutex`). - -No AD is needed: cljw's cancellation now matches clj's best-effort semantics -(blocking thunks abort, CPU loops do not), and the thread + GC pin release -promptly on a blocking-point cancel. - -## Defects in the naive "mark-only" design this ADR fixes (DA fork) - -- **Worker-store race**: the current unconditional worker store would clobber a - `.cancelled` set by a canceller — must guard on `state == .pending` (point 4). -- **Pin/thread persistence**: a naive mark-only (no cooperative abort) leaves the - worker thread + GC pin alive until the thunk completes (a 10s `sleep` stays - pinned 10s). Cooperative abort at blocking points releases promptly. - -## Alternatives considered (Devil's-advocate fork, fresh context — digest) - -- **Alt 1 — smallest-diff: mark-only (no cooperative abort) + a single AD.** - `future-cancel` marks `.cancelled`; the thunk runs to completion, result - discarded; deref raises. REQUIRES the worker-store guard (defect 1) + an AD for - the residual (thread+pin persist; blocking thunks not aborted = a real - divergence since clj WOULD abort a sleeping thunk). Acceptable as a fallback, - but per F-002 "B is more work" is not a reason to prefer it. -- **Alt 2 — finished-form (CHOSEN): cooperative cancel at the worker's existing - safepoint + blocking points only.** Real cancellation of blocking thunks, - prompt pin/thread release, best-effort semantics matching the JVM (tight loop - not interrupted) → **no AD**. Reuses the existing safepoint/ThreadGcContext/ - worker_error substrate. Risk: touches the worker eval path + an atomic flag the - safepoint reads; the cancellation signal must be un-swallowable. -- **Alt 3 — wildcard: model cancellation as a delivered CancellationException via - the existing `realised_error` path** (+ a `cancelled` bool for - `future-cancelled?`), reusing the ADR-0120 marshalled-exception channel with - zero new deref branch. A representation optimization on Alt 1 (still no abort → - still needs the AD); its plus is a real `CancellationException` class for free - and `future-done?` → true for a cancelled task (matches clj `isDone`). Folded - into Alt 2's error-class choice (point 3). - -The DA's recommendation was **Alt 2** (finished-form, no AD); cycle size is not a -project constraint (F-002). Alt 1 is the fallback only on a real F-NNN block -(none found). - -## Consequences - -- `future-cancel`/`future-cancelled?` land; deref of a cancelled future throws a - `CancellationException`-classed error (catchable, distinct from - `future_thunk_failed`). -- The stale `future_thunk_failed` placeholder Kind is NOT reused for cancellation. -- Testability: semi-non-deterministic — a `Thread/sleep`-bearing future gives a - deterministic cancel window (spawn slow future → future-cancel → assert - future-cancelled? + deref throws + the thread/pin released). No tight-race - (no-sleep) test in the gate. - -## Revision history - -- **2026-06-16 — sub-step 2 landed (Thread/sleep + precise class).** - - **2b (class)**: a new catchable `Kind .cancellation_error` maps to - `java.util.concurrent.CancellationException` (a RuntimeException via - IllegalStateException; `host_class.zig` ENTRIES + FQCN map). `.future_cancelled` - remapped off `.value_error` → `.cancellation_error`, so `(deref )` - now throws the precise CancellationException, catchable by its name + supertypes - and NOT by a sibling IllegalArgumentException. - - **2a (cooperative abort, `Thread/sleep`)**: a threadlocal `future.current_future` - (set by the worker) lets `Thread/sleep` poll `future.cancelRequested` in 20 ms - slices and raise the UNCATCHABLE `future_cancel_abort` (new `Kind - .cancellation_abort` → null in `kindToHostClass`), which unwinds the thunk past - its own `(catch Throwable …)` so a cancelled sleeping worker releases its thread - + GC pin promptly. Off a worker (`current_future == null`) sleep is unchanged - (one syscall). Deterministic e2e: cancel a `(Thread/sleep 500)` future, wait - 900 ms — `@a` stays `:init` (neither `:slept` nor `:caught`). - - **Residual (tracked in D-442, NOT an AD)**: the cooperative abort covers - `Thread/sleep` only. A worker blocked on a *nested* `(deref future/promise)` is - NOT yet aborted promptly (needs a `future/promise.deref` `?Value`→`anyerror!?Value` - signature ripple + degrading the condWait park to a poll). Low value - (resource-release only; un-observable via state, currently inert with - auto-collect off) → deferred as a D-442 follow-on with a concrete barrier, not a - blanket Alt-1 AD (this is Alt 2 for the primary blocking primitive, not a - wholesale mark-only fallback). - -## Affected files (implementation plan) - -- `src/runtime/future.zig` — `cancel_requested` flag + `.cancelled` state + - guarded worker store (`if state == .pending`) + `cancel` fn. -- the BLOCKING primitives (`Thread/sleep`'s sleep loop; nested-future `deref` / - promise wait cond loops) — read the worker's `cancel_requested` + raise the - cancellation signal. (NOT the per-op GC safepoint — see Decision point 2 amend.) -- `src/lang/primitive/stm.zig` — `future-cancel`/`future-cancelled?` primitives + - deref's cancellation re-raise. -- `src/runtime/error/host_class.zig` + `error/catalog.zig` — `CancellationException` - class + the cancellation Code/Kind. -- tests — a `Thread/sleep`-based e2e (cancel a slow future). diff --git a/.dev/decisions/0154_record_extmap.md b/.dev/decisions/0154_record_extmap.md deleted file mode 100644 index 8d39ed088..000000000 --- a/.dev/decisions/0154_record_extmap.md +++ /dev/null @@ -1,288 +0,0 @@ -# ADR-0154 — defrecord `__extmap` (non-declared keys on a record) - -- **Status**: Proposed → Accepted -- **Date**: 2026-06-16 -- **Discharges**: D-086 (assoc a non-declared key onto a defrecord) -- **Cross-refs**: ADR-0112 (the `meta` trailing-field twin this mirrors), - ADR-0068 (record map-style surface), F-002 (finished-form), F-004 - (NaN-box 64-slot — no new tag), F-006 (non-moving mark-sweep GC), - F-011 (clj behavioural equivalence), F-013 (definition-derived - capability); `private/notes/9.0-D086-record-extmap-plan.md` (the - wiring/reference-chain audit). - -## Context - -A JVM `defrecord` keeps non-declared keys in a hidden `__extmap` map, so a -record's IRecord/IPersistentMap identity survives `(assoc rec :extra v)`: - -- `(assoc (->R 1 2) :z 9)` → `#user.R{:x 1, :y 2, :z 9}` (clj). cljw raised - `defrecord_assoc_undeclared_key`. -- `(map->R {:x 1 :y 2 :z 3})` → keeps `:z` (clj). cljw dropped it. -- `(count (assoc (->R 1 2) :z 9))` → 3; `(keys …)`/`(vals …)`/`(seq …)` - include the extra; `(get … :z)`/`(contains? … :z)` see it; `(dissoc … :z)` - removes it; `(= (assoc r :z 9) (assoc r :z 9))` → true and they hash equal. - -cljw's `TypedInstance` (extern struct: header + `field_count` + descriptor -pointer + flat `[]Value` field tail + `meta`) had no place to hold extras. -D-086 (written 2026-05-26) framed this as a heavy structural seizure -deferred to a layout owner (F-003). The **`meta` field (ADR-0112, landed -2026-06-07, after the row)** since proved the exact pattern — a trailing -optional `Value` on the same extern struct + a one-line GC trace + assoc -re-mint threading — so the struct/GC half is precedented and low-risk, and -the F-003 "layout owner" deferral resolves to the loop itself under the -gap-area model (ADR-0142). - -## Decision - -Mirror the ADR-0112 `meta` shape, with the differences extmap demands -(extmap is part of record **value identity**; meta is not): - -1. **`TypedInstance` gains `extmap: Value = .nil_val`** (`type_descriptor.zig`, - appended after `meta` — `header` stays at offset 0). Invariant: extmap is - either `nil` (no extras — the common case) or a **non-empty** persistent - map. `allocInstance` inits it nil; `allocInstanceFull` threads an explicit - extmap (and meta). `instWithMeta` / the record `assoc` re-mint thread the - source instance's extmap so it survives `with-meta` / declared-field - updates. - -2. **One descriptor-owned partition chokepoint** (the DA's Alt 2). The - "is this key a declared field, and at what slot?" rule lives once as - `TypeDescriptor.fieldSlotByName(name) ?u16`; `assoc`/`dissoc`/`get`/ - `contains?` all call it instead of each re-walking `field_layout`. A key - that resolves to a slot writes/reads the field; everything else is an - extmap key. This makes the declared/extmap split definition-derived - (F-013) and structurally identical across `assoc` and `map->R`, so two - `=` records can never diverge on which keys are "declared" (the DA's - hazard 2). - -3. **assoc** (`collection.zig` `.typed_instance` arm): a non-declared key (or - empty/absent layout) re-mints via `allocInstanceFull` with the key - assoc'd into extmap; multi-pair folds over single-pair (clj reduces over - pairs), so `(assoc r k1 v1 k2 v2 …)` works. The two PROVISIONAL raise - sites and the multi-pair raise are removed. - -4. **dissoc** (`collection.zig`): a non-declared key in extmap → re-mint with - extmap dissoc'd, **normalized back to nil when it empties** (so the result - `=` a record that never had extras — the DA's leading finding). A - **declared** key still demotes the record to a plain map (clj parity), and - that demoted map now folds in the extmap entries too. - -5. **Read paths route declared-then-extmap**: `recordGet` (declared field → - extmap → ILookup → default); `contains?` (declared OR extmap key); `keys` - / `vals` (declared in order, then extmap); `count` (`field_count` + extmap - size); `seq` rides `recordToMap`, which now appends extmap entries (so - `first`/`rest` follow free). - -6. **print** (`printTypedInstance`): emit extmap entries after the declared - fields (`#R{:x 1, :y 2, :z 9}`). The `#R` vs clj's ns-qualified `#user.R` - name remains a **separate** tracked divergence (D-058/D-079, the ns - surface) — out of scope here. - -7. **Equality + hash include extmap** (unlike `meta`). `typedInstanceEqual` / - `typedInstanceKeyEq` compare extmap with the existing value `=` after the - positional field zip (map `=` is order-independent; nil-normalization - makes nil and empty agree). `typedInstanceHash` adds the extmap as **one - separate term** `h *% 31 +% valueHash(extmap)` only when extmap is - non-nil — `valueHash` on a map is already the order-independent HAMT - content hash, so `(assoc r :a 1 :b 2)` and `(assoc r :b 2 :a 1)` hash - equal (the DA's leading-finding hazard 1), and a record with no extras - hashes exactly as before (no churn for the common case). - -8. **`map->R` factory** (`macro_transforms.zig`): the generated body changes - from `(Name. (get m :f1) …)` (extras dropped) to - `(reduce-kv assoc (Name. (get m :f1) …) (dissoc m :f1 …))` — the positional - ctor for declared fields, then the now-extmap-aware `assoc` folds the - leftover keys. When `m` carries no extras the `dissoc` empties and - `reduce-kv` no-ops. - -9. **reify (`.reified_instance`) is untouched** — like meta, a fixed struct - with no field tail; extmap is a defrecord capability only. - -## Alternatives considered (Devil's-advocate, fresh-context subagent, verbatim) - -**Verification of the draft's structural premise (confirmed against source).** -The `meta` field (ADR-0112/D-312) is a trailing `meta: Value = Value.nil_val` -on the `TypedInstance` extern struct, set to nil in `allocInstance`, threaded -explicitly in `allocInstanceMeta`, marked by a single line in -`traceTypedInstance`, and ignored by `typedInstanceEqual`/`typedInstanceHash`. -Adding a trailing `extmap: Value = .nil_val` is the structurally identical -move; the only deltas vs. `meta` are (a) extmap participates in equality/hash, -and (b) extmap is mutated through structural ops, not just `with-meta`. The -draft is sound at the storage layer. The alternatives below differ on *where -the declared/extmap split is resolved* and *how the equality invariant is -guaranteed*. - -**Leading finding — the equality/hash normalization hazard is not fully -covered by "empty extmap → nil".** The draft's stated invariant ("an empty -extmap after dissoc must equal a freshly-minted record with nil extmap") is -necessary but **not sufficient** for F-011. Two deeper hazards: - -1. **Hash must fold extmap order-independently.** `typedInstanceHash` - (equal.zig L617-624) currently folds fields through a *positional* - `h = h *% 31 +% valueHash(fv)` loop. Extmap is an unordered map: - `(assoc r :a 1 :b 2)` and `(assoc r :b 2 :a 1)` are `=` and MUST hash - equal, but the persistent map's internal iteration order is not guaranteed - stable across construction paths. So extmap's hash contribution cannot - reuse the positional fold — it must use the **order-independent map hash** - (the same `hashUnordered`/commutative combiner clj uses for - `IPersistentMap`), folded in *separately* from the positional field fold. A - naive "append extmap to the field loop" silently produces order-dependent - hashes that break the equal-implies-equal-hash contract. This is the - genuine risk the draft underweights. - -2. **`map->R` and `assoc` must agree on the declared/extmap partition.** If - `(map->R {:x 1 :z 3})` puts `:z` in extmap but `(assoc (->R 1 nil) :z 3)` - reaches a different code path that partitions differently (e.g. one treats - a declared key written as nil as "present", the other as "absent"), then - two records that are `=` under clj diverge. The partition rule (declared - key → slot; everything else → extmap) must be applied at exactly **one** - chokepoint, or equality drifts. This argues against the draft's "every op - routes over declared-fields-then-extmap" if each op re-derives the - partition independently. - -Neither hazard requires violating an F-NNN; both are containable. They shape -which alternative is cleanest. - -**Alternative 1 — Smallest-diff: extmap field + per-op routing as drafted, -hash via a separate unordered fold.** This is the draft, with the one -mandatory correction from the leading finding: `typedInstanceHash` gains a -separate `hashUnordered(extmap)` term (combined into `h` with the commutative -combiner clj uses), not a positional append; `typedInstanceEqual` compares -extmap via the existing map `=` after the positional field zip; both -short-circuit `extmap == nil` for the common no-extras record. Each structural -op (`assoc`/`dissoc`/`get`/`keys`/...) routes "declared? → slot : extmap" -inline. -- *Better than the draft as written*: nothing structurally — it IS the draft, - plus the hash fix that the draft must adopt regardless. -- *Breaks*: the partition logic is duplicated across ~8 ops. Each duplication - is an independent chance to diverge on the edge cases (a declared key - explicitly assoc'd to nil; `dissoc` of a declared key — clj actually - *cannot* dissoc a declared key and returns a plain map, a behaviour each op - must independently remember). This is the **Smallest-diff bias smell**: it - minimizes the field-layer diff while scattering the invariant. Per F-013, - "is this key declared?" is a property of the *descriptor*, so deriving it - inline in every op is a per-feature bolt-on rather than a definition-derived - capability. - -**Alternative 2 — Finished-form-clean: extmap field + a single -descriptor-owned partition chokepoint.** Same storage as the draft (trailing -`extmap: Value`, mirroring `meta`), but the declared/extmap split is resolved -in **one** place: a descriptor-level helper (e.g. -`TypeDescriptor.partitionKey(key) -> .declared(index) | .extra` and a paired -`assocKey`/`dissocKey`/`getKey` on the instance) that every op — `assoc`, -`dissoc`, `map->R`, `get`, `keys`, `vals`, `count`, `seq`, `print`, equality, -hash — calls. The descriptor already knows the declared field names (it must, -to construct `->R`), so the partition is *definition-derived* (F-013): a -record carries extmap and routes keys because it is a record over a known -field set, not because each op was taught the rule. The hash uses the separate -unordered fold (leading finding) inside the chokepoint, so it cannot be gotten -wrong per-op. `map->R` and `assoc` share the same `assocKey`, structurally -guaranteeing the Alternative-1-hazard-2 agreement. -- *Better than the draft*: the F-011 equality invariant and the F-013 - definition-derived property are both enforced by construction at one - chokepoint, not re-asserted in 8 ops. The "declared key cannot be dissoc'd, - returns a plain map" clj edge lives once. This is the finished-form per - F-002 — the partition is the record's contract, expressed once. -- *Breaks*: nothing semantically; it is a larger diff than Alternative 1 (a - new descriptor-level key-routing surface). Per F-002, diff size is - explicitly not a constraint and must not downgrade this. The only real cost - is that `print`/`seq` ordering (declared-fields-then-extmap) becomes a - property of the chokepoint's iteration contract, which must be specified — - but that is a clarification, not a breakage. - -**Alternative 3 — Wildcard: no extmap field; an extended record is a distinct -value that delegates to a plain map.** Mirror clj's actual *observable* -contract differently: when a non-declared key is assoc'd, do **not** re-mint a -`TypedInstance` with extras — instead the declared fields plus extras are -projected and the result is represented as the record's declared slots -*backed by* a persistent array-map for the extras, reusing the existing map -machinery wholesale for everything extmap-related (iteration, `=`, hash, -`dissoc`, `get` on extra keys). Concretely: keep `TypedInstance` exactly as-is -(no new field), and store extras in a side representation reached through the -descriptor — but **not** a pointer-keyed side-table (that would violate F-006: -a pointer-keyed table that roots instances forever / needs finaliser hooks is -exactly the anti-pattern F-006 names). The only F-006-clean way to do this -without the extmap field is to make the "extended record" a *different -descriptor variant* or a wrapper Value that holds both the base instance and -the extras map as traced children. -- *Better than the draft*: maximal reuse of the existing persistent-map - `=`/hash/iteration — the order-independent-hash hazard (leading finding) is - solved for free because the extras genuinely *are* a map value. No new - equality/hash code in `typedInstanceEqual` beyond "compare the extras map". -- *Breaks*: it needs either a new wrapper Value (an extended-record tag) — - which **violates F-004** (a new value tag on the 64-slot NaN-box is - expensive/constrained and explicitly not wanted) — or a second descriptor - variant, which fragments "what is a record" and makes `(record? x)` / - `instance?` / `=` between a plain and an extended instance of the same type - harder to keep clj-faithful (they must still be `=` when fields+extras - match). **This is the F-NNN-blocked option**: the cleanest form of the - wildcard (a dedicated extended-record value) requires a new tag and is ruled - out by F-004. The descriptor-variant fallback avoids the tag but - re-introduces the partition-fragmentation that Alternative 2 exists to - eliminate, landing strictly worse than Alternative 2 on F-013. Recorded here - per the brief: the tag-based wildcard would be F-004-violating; do not - pursue it. - -**Recommendation (non-binding): Alternative 2 — the descriptor-owned -single-chokepoint partition, with extmap stored as the drafted trailing -`Value` field and hash folded via a separate unordered combiner.** It is the -finished-form-clean shape (F-002), makes the extmap capability -definition-derived (F-013), and structurally guarantees the F-011 -equality/hash invariant that Alternative 1 scatters and that the -leading-finding hash hazard makes too easy to break per-op. The draft is -correct at the storage layer and should keep its `meta`-mirroring field; it -needs only to consolidate the per-op routing into one descriptor-level -chokepoint and adopt the separate unordered hash fold. - -## Main-loop disposition: ACCEPT Alternative 2 - -The draft's storage layer (trailing `extmap: Value`, `meta`-mirroring) is -kept. The DA's **leading finding is adopted in full**: extmap hashes as one -separate `valueHash(extmap)` term (order-independent, since `valueHash` on a -map is the HAMT content hash), and the empty→nil normalization on dissoc is -the equality invariant. The DA's **Alternative 2 chokepoint is adopted** at -the granularity the existing code supports: the declared-field-slot lookup -duplicated across `recordFieldGet`/`assoc`/`contains?` is consolidated into -`TypeDescriptor.fieldSlotByName`, so the partition rule lives once; -`map->R` and `assoc` reach extmap through the same `assoc` path, so they -cannot disagree (hazard 2 closed structurally). Per-op *read shaping* (keys -returns a key list, count returns a size, print emits entries) necessarily -differs by op and is not forced into one function — what is unified is the -**partition predicate** + the shared `recordToMap` full-view that `seq` -already rides. Alternative 3 is rejected on the F-004 ground the DA gives (a -dedicated extended-record tag) and the F-013 ground (descriptor-variant -fragmentation). Cycle/diff size did not enter the choice (F-002). - -## Consequences - -- **Positive**: records become fully IPersistentMap-faithful (F-011/F-013): - `assoc`/`dissoc`/`get`/`keys`/`vals`/`count`/`seq`/`print`/`=`/`hash` and - `map->R` all honour extra keys. The two PROVISIONAL markers + D-086 close. - The partition rule is defined once (`fieldSlotByName`). -- **Negative**: a non-declared `assoc`/`dissoc` re-mints (copies the field - array, like `with-meta`) and now also carries an extmap map alloc — the - irreducible cost of immutability, paid only when extras exist. A record with - no extras is unchanged in layout cost (one extra nil `Value` slot, - `meta`-equivalent) and hashes/compares exactly as before. - -## Affected files - -- `src/runtime/type_descriptor.zig` — `TypedInstance.extmap` field; - `TypeDescriptor.fieldSlotByName`; `allocInstanceFull` (meta + extmap); - `allocInstance`/`allocInstanceMeta` rewired through it; `traceTypedInstance` - extmap-mark line; `instExtmapOf`; `instWithMeta` threads extmap. -- `src/lang/primitive/collection.zig` — `assocFn` (.typed_instance: extmap - re-mint + multi-pair fold, PROVISIONAL raises removed); `dissocFn` - (extmap dissoc + normalize-to-nil; declared-key demote folds extmap); - `containsQFn`; `keysFn`; `valsFn`. -- `src/runtime/collection/lookup.zig` — `recordGet` consults extmap. -- `src/lang/primitive/sequence.zig` — `recordToMap` appends extmap; - `countFn` record arm adds extmap size. -- `src/runtime/print.zig` — `printTypedInstance` emits extmap entries. -- `src/runtime/equal.zig` — `typedInstanceEqual` / `typedInstanceKeyEq` / - `typedInstanceHash` include extmap. -- `src/lang/macro_transforms.zig` — `map->Name` threads leftover keys. -- `test/e2e/phase9_record_extmap.sh` (new) — clj-parity round-trips; - `test/run_all.sh` wires it. -- `feature_deps.yaml` (`runtime/record_extmap` → landed) + `.dev/debt.yaml` - (D-086 → discharged). diff --git a/.dev/decisions/0155_legacy_agent_executor_surface.md b/.dev/decisions/0155_legacy_agent_executor_surface.md deleted file mode 100644 index 7fe281c35..000000000 --- a/.dev/decisions/0155_legacy_agent_executor_surface.md +++ /dev/null @@ -1,200 +0,0 @@ -# ADR-0155 — Legacy / executor agent surface: per-function disposition (D-442) - -Status: Proposed → Accepted (2026-06-20) - -## Context - -D-442 (the Track-R agent-parity micro-survey) left 8 agent-system functions -unimplemented after the `future-cancel` arc (ADR-0153). They are the -pre-1.0-deprecated + the 1.5 executor-configuration surface: -`agent-errors`, `clear-agent-errors`, `send-via`, `set-agent-send-executor!`, -`set-agent-send-off-executor!`, `shutdown-agents`, `*agent*`, -`release-pending-sends`. Each currently raises `unresolved-symbol`. - -cljw's agent model (the classification baseline, from `runtime/agent.zig`): -**one DETACHED drainer thread per agent** (spawned on the empty→non-empty queue -transition, drains the FIFO serially, exits), **no shared executor pool**, a -real **`nested_pending` ThreadLocal** mirroring clj's `Agent.nested` (D-388), -and **no `*agent*` dynamic var**. This diverges from clj/babashka (both -JVM-pool-backed) AND from cljw v0 (its own worker pool + `*agent*` binding). - -The classification must obey `provisional_marker.md` § "permanent no-op -forbidden": a function may not ship returning a default that masks a dropped -semantic. So a thin no-op is allowed ONLY where cljw genuinely satisfies the -observable semantic; otherwise the honest disposition is an explicit error -(accept-divergence) recorded as an AD. Survey: -`private/notes/D442-legacy-agent-surface-survey.md`. - -## Decision - -Three dispositions, by whether cljw's model makes the semantic meaningful: - -| Function | Disposition | Rationale | -|--------------------------------------------------------------------------|-------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `agent-errors` | **implement** | clj-deprecated 1.2; pure sugar over the existing `agent-error` (`(when-let [e …] (list e))`). | -| `clear-agent-errors` | **implement** | clj-deprecated 1.2; `(restart-agent a @a)` — both halves exist. | -| `*agent*` | **implement** | the drainer knows `a`; bind `*agent*` around each action body (clj's `binding [*agent* a]`). | -| `release-pending-sends` | **implement** | the `nested_pending` buffer EXISTS — flush it + return the count (v0's `return 0` would now lie). | -| `shutdown-agents` | **implement (real)** | flip a process-global `agents_shut_down` flag; the send path then DROPS new dispatches — send returns the agent, no throw, action never runs (clj-faithful: clj's executor rejects + `Action.execute` swallows the exception). Detached drainers already don't block exit. Faithful in the common case; AD-046 scopes the `:error-handler` edge (see § Alternatives). | -| `send-via` / `set-agent-send-executor!` / `set-agent-send-off-executor!` | **accept-divergence (raise)** | cljw exposes NO executor object (no `java.util.concurrent.ExecutorService` to construct or pass), so there is nothing real to thread/configure. Silently ignoring the user's explicit executor is a masked-semantic lie (DA finding); raise an explicit "cljw has no configurable executor" error → AD-045. | - -The `implement` group is real behaviour. The three executor-shaped functions all -raise: cljw genuinely has no executor concept, so neither a no-op (would mask the -dropped semantic) nor a real impl (would require a `java.util.concurrent` pool -subsystem cljw deliberately lacks) is honest — the explicit error is. - -New ADs: **AD-045** (the three executor-shaped fns raise — cljw exposes no -configurable executor) + **AD-046** (post-shutdown send disposition; see below). - -> **Premise correction (2026-06-20, part 2 landing).** This ADR originally claimed -> "`shutdown-agents` needs no AD: post-shutdown sends *reject*; only the error -> *Kind* differs, which AD-007 covers." The clj oracle proved that premise FALSE: -> after `(shutdown-agents)`, clj's `send` **returns the agent and does NOT throw** -> — `Action.execute` catches the executor's `RejectedExecutionException` and either -> routes it to the agent's `:error-handler` (if set) or swallows it; the action is -> dropped. So cljw must DROP, not throw. The common case (no handler) is -> byte-identical to clj; the only residual divergence is that cljw does not -> synthesize a fake rejection exception to fire a handler — recorded as **AD-046** -> (AD-007 is inapplicable: it governs error *Kind* when BOTH runtimes throw, and -> here clj does not throw at all). - -## Alternatives considered (Devil's-advocate fork, fresh context — digest) - -The DA fork (fresh context, F-NNN-constrained) produced three overall shapes and -stress-tested the two borderline calls; its findings REVISED this ADR: - -- **Alt A (smallest-diff: make the executor setters no-ops too)** — rejected: a - bare no-op on `set-agent-send-executor!` is the canonical masked-semantic lie - (the user expects subsequent dispatch on their executor; cljw silently keeps its - drainer). Violates `permanent_no_op_forbidden`. -- **Alt B (finished-form: a real `DispatchFn` seam so the setters + `send-via` - become real)** — rejected on an F-NNN-adjacent ground the DA did not fully - price: cljw has no `ExecutorService` TYPE and no way to construct one, so the - user has no executor VALUE to pass; making the setters "real" would require the - whole `java.util.concurrent` executor subsystem, which is a distinct large - feature out of D-442's scope (and arguably the point — cljw exposes no - executors). So within cljw's model the setters genuinely cannot be made real; - raise is the honest disposition. -- **Alt C (raise on everything cljw can't satisfy, incl. `shutdown-agents`)** — - rejected: `shutdown-agents` raising breaks any ported program's graceful-exit - coda even though the *intent* (let the process exit) IS satisfied. Over-corrects. - -Borderline corrections the DA forced (both adopted): - -1. **`send-via`** — the draft's "accept-and-ignore the executor" was WRONG. The - ADR-0093 D1 "pool choice is non-observable" precedent covers only the internal - send/send-off split the user never named; `send-via`'s executor is passed - EXPLICITLY, so ignoring it is a lie ("the user thinks work runs on their bounded - pool"). Reclassified from silent-no-op to **raise** (joins the executor setters). -2. **`shutdown-agents`** — the draft's pure nil no-op did not just drop clj's "no - new actions accepted" clause, it did the OPPOSITE (ran work clj guarantees won't - run). Reclassified from thin-no-op to **real**: a cheap process-global - `agents_shut_down` flag the send path checks. No F-006 interaction (a flag, not a - thread/lock). **(Part-2 correction:** the DA + draft both assumed the flag should - make the send *throw*; the oracle later showed clj DROPS the send without - throwing. The landed behaviour is the clj-faithful drop, scoped by AD-046 — a - second DA fork validated drop-vs-throw and chose drop on F-011 priority over the - *letter* of permanent-no-op-forbidden.) - -The four `implement (real)` calls (`agent-errors`/`clear-agent-errors`/`*agent*`/ -`release-pending-sends`) were uncontested by the DA. - -## Alternatives considered (part 2 — `shutdown-agents` send disposition, 2026-06-20 DA fork) - -When part 2 landed, the clj oracle contradicted this ADR's part-1 premise (clj -drops, does not throw). A second Devil's-advocate fork (fresh context) stress- -tested three dispositions; its output, verbatim: - -> **Alt A — match clj (send returns agent, action silently dropped, no throw).** -> Does better: pure F-011 parity, zero AD entropy, identical to clj for a ported -> cleanup-path send. The `permanent_no_op_forbidden` objection is weaker than it -> looks — that principle bans cljw-*invented* silent drops where the user expects -> an effect; here the user issued the terminal teardown and clj itself documents -> the drop, so "the drop IS the contract." Breaks: collides with the *letter* of -> permanent_no_op_forbidden, so it needs an explicit carve-out (an AD/ADR note) -> or a future audit re-flags it. -> -> **Alt B — keep the explicit throw, record AD-046.** Does better: honors the -> letter of permanent_no_op_forbidden with no interpretation; loud failure. -> Breaks: manufactures a *behavioral* F-011 divergence where clj has a clean -> observable behavior cljw could simply match — the worst kind of AD (elective, -> not forced by a missing subsystem). A ported program that sends post-shutdown -> runs clean on clj and CRASHES on cljw — exactly the portability hazard F-011 -> exists to prevent. derives_from = permanent_no_op_forbidden is a misuse: that -> invariant protects against *cljw* dropping effects, not against *clj* defining -> a terminal no-op. -> -> **Alt C — route the dropped send to the error-handler if set, else drop.** Does -> better: most mechanically faithful to clj's `execute()` swallow-then-route. -> Breaks: highest-cost, lowest-value — clj's routing is an implementation artifact -> of catching `RejectedExecutionException`, not a contract anyone ports against; -> cljw would synthesize a fake exception of a type it lacks, CREATING a fresh DIFF -> (the fabricated `ex`'s class/message) in the path it's trying to make faithful. -> Net negative. -> -> **Recommendation: Alt A, with an AD recording the principle exception (not a -> behavioral divergence).** F-011 + clj-parity are the binding invariants, and -> only Alt A satisfies them with zero behavioral DIFF. The permanent_no_op -> collision resolves on inspection: that principle guards against cljw hiding its -> OWN unimplemented semantics behind a success return — it does not force cljw to -> *diverge from clj* to be louder than the reference. Alt B inverts the priority -> order (elevates the letter of a depth-4 heuristic above an F-NNN/F-011 -> invariant). The AD's derives_from is F-011; its body records the audited -> permanent-no-op exception; ADR-0155's "no AD / AD-007 covers it" is corrected. - -The main loop adopted Alt A. One refinement the loop added that the DA could not -see from outside: under Alt A `(await a)` after shutdown drops its barrier and so -**hangs forever** (clj-faithful — clj's `CountDownLatch` never trips either), so -no e2e exercises await-after-shutdown. The realized AD-046 also pins the -handler-routing edge the DA flagged in Alt C: the oracle confirmed clj fires a -set `:error-handler` with a `RejectedExecutionException` on the dropped send; -cljw drops without firing it (no exception type to synthesize). - -## Consequences - -- The deprecated-but-real surface (`agent-errors`/`clear-agent-errors`) + - `*agent*` + `release-pending-sends` + a faithful `shutdown-agents` round out - agent parity for ported code. -- One AD (AD-045) documents the no-executor divergence so it reads as designed. -- `*agent*` adds a dynamic-var binding in the drainer (small, real capability). -- `shutdown-agents` adds a process-global flag checked in the send path; faithful - to clj's process-wide, irreversible pool shutdown. - -## Revision history - -- 2026-06-20: created; classifies the D-442 legacy/executor surface. DA fork - revised two borderline calls: `send-via` silent-ignore → raise (a passed - executor is observable); `shutdown-agents` no-op → real (flag + send-path - reject). Alt B (DispatchFn seam) rejected — cljw has no executor type to make - the setters real. Part 1 landed: the three executor raises (AD-045) + - `agent-errors`/`clear-agent-errors` sugar. -- 2026-06-20 (part 2): landed `*agent*` (drainer binds it around each action via a - cached `rt.agent_var`), `release-pending-sends` (flush `nested_pending` + re-arm - to empty, return count), and `shutdown-agents`. **Premise correction**: the - part-1 "`shutdown-agents` rejects/throws, no AD" claim was factually wrong (clj - DROPS the post-shutdown send without throwing). A second DA fork validated - drop-vs-throw → DROP (Alt A, F-011 priority); the landed behaviour is the - clj-faithful silent drop, scoped by the new **AD-046** (handler-routing edge). - No `agents_shut_down` catalog Code was added (the drop needs no error). - -## Affected files (implementation plan) - -- `src/lang/clj/clojure/core.clj` — `agent-errors` (`(when-let [e (agent-error a)] - (list e))`) / `clear-agent-errors` (`(restart-agent a @a)`) defns. -- `src/lang/bootstrap.zig` — intern `clojure.core/*agent*` `^:dynamic` (root nil) - + cache `rt.agent_var` (mirrors `*ns*` / `*data-readers*`, so the Layer-0 drainer - can reach the Var pointer; NOT a core.clj `def`, which the drainer can't cheaply - resolve). `src/runtime/runtime.zig` — the `agent_var` cached slot. -- `src/lang/primitive/agent.zig` — `release-pending-sends` primitive (flush - `nested_pending`, return count), `shutdown-agents` (set the flag, nil), the three - executor-shaped fns (raise the no-executor error). -- `src/runtime/agent.zig` — a process-global `agents_shut_down` flag set by - `shutdown-agents` + checked in `enqueueDirect` (DROP new dispatches after - shutdown, no throw); `releasePending` (re-arm-to-empty flush); bind `*agent*` - around each action body in the drainer. -- `src/runtime/error/catalog.zig` — the `agent_executor_unsupported` Code (the - no-configurable-executor message). No `agents_shut_down` Code — the post-shutdown - drop is a clj-faithful no-op (returns the agent), so there is no error to raise. -- `.dev/accepted_divergences.yaml` — AD-045 (executor raises) + AD-046 - (post-shutdown drop / handler-routing edge), each pinned. -- e2e (`test/e2e/phase16_agent.sh`) for the implemented surface. diff --git a/.dev/decisions/0156_bundle_official_stdlib_eager.md b/.dev/decisions/0156_bundle_official_stdlib_eager.md deleted file mode 100644 index 29961f607..000000000 --- a/.dev/decisions/0156_bundle_official_stdlib_eager.md +++ /dev/null @@ -1,132 +0,0 @@ -# ADR-0156: Bundle official Clojure stdlib eagerly; contrib for completeness - -- **Status**: Accepted (2026-06-20) -- **Deciders**: autonomous loop + user directive (2026-06-20 chat) -- **Supersedes / relates**: ADR-0035 (embedded resolver), ADR-0056 (AOT - bootstrap cache), D-452 (eager non-core libs in the blob), D-475 (the - spec.alpha port), D-477 (the latent baseline-binding gap this surfaced). - -## Context - -The clojure.spec.alpha port (D-475) reached functional clj-parity on a -`-cp` classpath dev harness. Promoting it for real means bundling it. This -forced a standing decision the project had not made explicitly: **which -Clojure namespaces ship inside the `cljw` binary, and are they loaded -eagerly at startup or lazily on require?** - -Today all 22 non-core bundled `clojure.*` libs are **eager** — AOT-compiled -into the startup bytecode blob (cache_gen → `loadCoreAot` replay). Cold-start -is ~0ms because the blob is replayed bytecode, not re-parsed. The concern: a -stdlib + contrib *sweep campaign* (the 2026-06-20 user directive — port every -official stdlib ns + verify against every contrib lib for completeness / bug -audit) would add many libs; if all go eager, the blob + binary grow -cumulatively. - -Two facts frame the choice: - -- **Official stdlib** (everything in `clojure/src/clj/clojure/` — ships in - `clojure.jar`, no external deps) is part of "the language". A user expects - `clojure.set` / `clojure.spec.alpha` present without a dependency step. -- **contrib** (`org.clojure/core.async`, `data.json`, `math.combinatorics`, - … — separate Maven artifacts, `deps.edn`-required upstream) is NOT part of - `clojure.jar`. cljw already bundles a few (data.json/csv, tools.cli) for - convenience, but contrib need not be in the startup binary. - -## Decision - -1. **Official Clojure standard-library namespaces are eager-bundled** into - the `cljw` binary (the existing `FILES` AOT mechanism). `clojure.spec.alpha` - + `clojure.spec.gen.alpha` are added now (FILES[24]/[25], gen before alpha). -2. **contrib libraries are used for completeness/bug-audit verification** - (clj-diff sweep against the cloned `org.clojure/*` repos under - `~/Documents/OSS/`); they need NOT be eager-bundled, but the require path - should stay easy (the embedded resolver + a filesystem resolver over the - clones). Bundling a high-demand contrib lib (as with data.json) remains - acceptable, not required. -3. **Accept the startup-time + binary-size increase for simplicity now** - (measured: cold-start stayed ~0ms after spec — per-lib AOT replay is - negligible). Keep the bundled-lib list **data-driven** (`FILES` + - `lookupEmbeddedFile`) so the future eager→lazy switch is a local change. -4. The spec source is upstream spec.alpha with **5 no-JVM adaptations** - (Spec-class→protocol-var; fn-sym→nil; RT/checkSpecAsserts→atom; - bytes?-gen-builtin dropped; the top-level `(set! *warn-on-reflection* true)` - dropped — a cljw no-op that would also trip the eager-load baseline-binding - gap, D-477). EPL variant-① attribution per `clj_attribution.md`. -5. A build-time diagnostic landed in `buildBootstrapEnvelope`: an AOT eval - error now prints `[AOT-FAIL] form #N` (was a bare `ValueError`), - so the campaign's future bootstrap traps are locatable. - -This is the user's directed choice (chat, 2026-06-20): "起動時間/バイナリ -サイズの増加は許容してシンプルに進める、ただし将来の切り替えやすさは意識". - -## Alternatives considered (Devil's-advocate fork, verbatim) - -**Leading caveat:** The cleanest finished form (lazy/on-demand stdlib AOT -blobs) is *partially* in tension with the user's directive ("stdlib SHOULD be -in the binary; proceed simply"). That directive is project law, so a -pure-lazy stdlib violates it. But "in the binary" ≠ "eagerly *loaded* at -startup" — a blob can ship in the binary yet replay on first reference. Alts -2 and 3 exploit that gap without violating the directive. - -**1. SMALLEST-DIFF — eager-bundle spec, but split policy from mechanism.** -Bundle spec as proposed, but do NOT inscribe "all clojure.jar namespaces are -eager" as a standing policy ADR yet; record it as an observation. *Better:* -avoids a Reservation-as-bias smell — the policy is asserted from one data -point (spec, ~0ms); gen.alpha pulls test.check-style generators; future eager -stdlib members may not be ~0ms (F-003: defer structural policy to its owner). -*Costs:* the "easy mental model" the user wanted is left less crisp. - -**2. FINISHED-FORM-CLEAN — ship-in-binary, replay-on-first-require (lazy-AOT) -for the whole bundled set.** Every bundled lib's AOT blob lives in the binary -(satisfies "in the binary"), but `loadCoreAot` replays only true-core eagerly; -spec and peers replay on first `require`/reference. *Better:* satisfies F-002 -directly — the data-driven eager→lazy switch the ADR *promises as future work* -IS the finished form, so do it now rather than bank debt; keeps cold-start -structurally flat regardless of how many libs accrete; the ~0ms claim stops -being per-lib luck. *Costs/breaks:* needs a require-time blob-replay path that -doesn't exist yet; first-require latency moves off cold-start onto first-use -(must verify spec's macro-heavy load stays sub-perceptible); larger diff (not -a valid objection per F-002). - -**3. WILDCARD — vendor spec verbatim; move the no-JVM adaptations into the -runtime, not the .clj.** Bundle unmodified upstream spec.alpha; implement -Spec-class→protocol, fn-sym→nil, checkSpecAsserts as host shims so the .clj is -byte-identical to clojure.jar. *Better:* maximal F-011 parity + trivial future -upstream resync (re-vendor, no re-patch); the adaptations become reusable host -primitives other stdlib libs will also need. *Costs:* upfront runtime shim -work; risk the shims under-cover spec's reflection assumptions; over-engineered -if no second consumer materializes. - -**DA recommendation:** Alt 2 (finished-form-clean). - -**Loop's decision within the F-NNN + user-directive envelope:** the user -explicitly chose the eager default for simplicity *now*, with the eager→lazy -switch kept easy for later. The user directive is project law above the -cycle-budget concern, so Alt 2 is recorded as **the finished form the future -switch (D-477 / a later lazy-AOT ADR) will realize**, not adopted this cycle. -Alt 1's "don't over-inscribe policy" point is mooted because the user -*declared* the stdlib-eager policy (not a one-data-point inference). Alt 3's -runtime-shim resync benefit is noted for the campaign but deferred (the 5 -in-.clj adaptations are small and documented in the file header). - -## Consequences - -- spec.alpha/gen.alpha load from the bundle with no `-cp`; clj-verified - surface (corpus `test/diff/clj_corpus/spec.txt`, 16 cases). Residual diffs - are the accepted `rt/`-qualification family (`rt/int?` vs `clojure.core/int?` - in printed pred forms) — derive from ADR-0033/AD-038. -- Binary grows by spec's bytecode; cold-start unchanged (~0ms). -- The stdlib-sweep campaign now has a clear rule: stdlib → eager FILES entry; - contrib → verify + optional bundle. -- D-477 records the latent eager-load baseline-binding gap (a future bundled - lib with a meaningful top-level config-var `set!`). -- When the eager blob grows enough to matter, Alt 2 (lazy-AOT) is the - pre-decided finished form. - -## Affected files - -- `src/lang/bootstrap.zig` — FILES[24]/[25] + lookupEmbeddedFile entries. -- `src/lang/clj/clojure/spec/alpha.clj`, `…/spec/gen/alpha.clj` — bundled. -- `src/app/builder.zig` — AOT-fail diagnostic. -- `test/diff/clj_corpus/spec.txt` — corpus lock. -- `.dev/debt.yaml` — D-477. diff --git a/.dev/decisions/0157_eval_reentry_depth_guard.md b/.dev/decisions/0157_eval_reentry_depth_guard.md deleted file mode 100644 index 6b2e3f710..000000000 --- a/.dev/decisions/0157_eval_reentry_depth_guard.md +++ /dev/null @@ -1,165 +0,0 @@ -# ADR-0157: Stack-overflow robustness — catchable Kind + self-calibrating native-stack guard - -- **Status**: Accepted (2026-06-21) — Devil's-advocate reflected; decision revised from - the proposed eval-entry COUNTER to a catchable Kind (2b) + self-calibrating - stack-pointer guard (2a). **2b LANDED** (commit aaf391e3: `stack_overflow` own - catchable Kind → StackOverflowError; re-entry + nested-eval overflows now catch). - **2a LANDED** (commit 48be8e91: self-calibrating native-stack guard — threadlocal - stack-base anchor + 6 MiB byte-budget at `vm.eval` entry; validator/reducer/watch - re-entry SIGSEGV → graceful catchable error; watch-256 partial subsumed + removed). - **D-485 + D-486 DISCHARGED — ADR fully implemented.** D-486 (the same-eval direct- - recursion catch edge) was fixed by unifying the flatten path's overflow into the - shared `else |err|` arm in `vm.eval`. Every overflow path — re-entry AND direct - recursion — now raises a graceful, catchable StackOverflowError (clj parity). -- **Deciders**: autonomous loop (differential bug-sweep finding) -- **Supersedes / relates**: D-485 (the tracking row + full mechanism trace), - the watch-nesting partial fix (commit b69d97a9, `iref.enterWatchNotify` cap 256 - — this ADR's general fix subsumes it), ADR-0131 (the flattened in-VM call - frame / `vm_arena`), ADR-0059 (no-JVM error model), AD-007 (cljw error Kind). - -## Context - -A higher-order primitive that re-enters the evaluator through a user callback -overflows the **native** Zig stack → SIGSEGV (exit 134), where JVM Clojure -raises a catchable `StackOverflowError`. Confirmed repros (all were SIGSEGV): - -- `(let [a (atom 0)] (add-watch a :w (fn [k r o n] (swap! r inc))) (swap! a inc))` -- `(let [a (atom 0)] (set-validator! a (fn [n] (swap! a inc) true)) (reset! a 5))` -- `(reduce (fn f [acc x] (reduce f acc [x])) 0 [1 2 3])` - -A comparator that recurses on *itself* (`(sort (fn cmp [x y] (cmp x y)) …)`) is -already graceful — it is direct VM self-recursion bounded by `FRAMES_MAX=2048`. -So only **primitive→callFn re-entry** is unguarded, not VM self-recursion. - -**Mechanism** (traced in D-485): `vm.eval` (vm.zig:194) saves -`frame_base = ar.frame_top` on entry and *continues* from it — the VM frame -budget is **already global** across re-entry, and `FRAMES_MAX` (vm.zig:227 + -`flattenPush`:149) checks the global depth. Plain VM recursion is safe at 2048 -because `op_call` flattens in-VM (iterative — ~1 native frame per VM frame). But -each primitive→callFn re-entry (callBuiltin→notify/validate→`invokeCallable`→ -`treeWalkCall`→`callMethodImpl`→`eval`) adds ~6-10 **native** frames per ~1-2 -**VM** frames, so `frame_top` **under-counts native depth** and the native stack -overflows before `frame_top` reaches 2048. - -F-011 forbids cljw crashing on an input clj merely throws on; this is a -robustness gap that must close. - -## Decision - -Adopt the Devil's-advocate's **Alternative 2** (finished-form-clean), staged: - -- **2b (land first, independent)** — give `.stack_overflow` its **own catchable - Kind** mapping to `"StackOverflowError"` in `kindToHostClass`, NOT the current - `.resource_exhausted` (which maps to `null` → uncatchable, and is *deliberately* - uncatchable for the eval-budget sandbox bound — the opposite requirement). clj's - `StackOverflowError` is catchable (`Throwable`); cljw must match - (`(try (deep-recursion) (catch StackOverflowError _ :ok))` → `:ok`). The existing - plain-recursion path already raises `.stack_overflow` gracefully at 2048, so 2b - is independently testable. -- **2a (the SIGSEGV fix, the remaining D-485 work)** — a **self-calibrating - native-stack guard** instead of a fixed counter/cap: capture the thread's stack - base (main thread at startup, workers at spawn), and at the `vm.eval` entry - (± `invokeCallable`) compare a stack-local address against base − a generous - safety margin (the SpiderMonkey `CheckRecursionLimit` / CPython - `_Py_CheckRecursiveCall` / Go `morestack` pattern). It measures the actual - resource (native bytes remaining), so it is immune to per-primitive frame - variance, optimization level (Debug vs ReleaseSafe), and platform (arm64 vs - x86_64) — the cross-host SIGSEGV risk that sinks a fixed cap. It matches clj's - "error when the stack is actually nearly full" model and **subsumes the - watch-256 partial** (removed when 2a lands). - -The originally-proposed fixed-cap eval-entry counter (Alternative 1) is REJECTED: -a single constant cannot be both ≤ native-stack/worst-frame and ≥ deepest-legit-HOF, -and it drifts across optimization level / platform — a latent cross-host crash. -Alternative 3 (trampoline primitive callbacks through the VM frame stack so -`frame_top` becomes an accurate native bound) is the eventual finished form but a -depth-4 GC-rooting surgery across every HOF primitive — recorded as forward debt -(gap-area-III primitive-calling-convention rework), not opened now. - -## Alternatives considered - -The following is the fresh-context Devil's-advocate critique, reflected verbatim -(per CLAUDE.md § ADR-level designs). Its central code-level finding — -`.stack_overflow` is currently uncatchable (Kind `.resource_exhausted` → -`kindToHostClass` `null`) while clj's is catchable — is what revised the decision -from Alternative 1 to Alternative 2. - -> **Recommendation: Alternative 2** (self-calibrating stack guard + catchable -> `stack_overflow` with its own Kind). Removes the cap-calibration fragility, -> matches clj's stack-bound depth model far more closely than a frame-count proxy, -> and discharges the catchability F-011 divergence in the same unit rather than -> calcifying it. -> -> **Alt 1 (smallest-diff, counter at `vm.eval` entry, keep uncatchable)** — stops -> the segfault but (i) locks in the uncatchable divergence (clj programs that -> `(catch StackOverflowError _ :recovered)` would abort under cljw); (ii) counts at -> the wrong granularity (per VM activation, not per native re-entry — a -> high-variance proxy for native bytes); (iii) fixed-cap fragility: the safe value -> drifts with optimization level, platform, and any new deeper-native primitive — a -> cap calibrated on Mac ReleaseSafe can SIGSEGV on a Debug Linux gate. -> -> **Alt 2 (finished-form-clean, RECOMMENDED)** — (2a) self-calibrating stack guard: -> measure remaining native stack via a stack-local address vs the captured thread -> base, raise below a 64 KB-ish margin (the SpiderMonkey/CPython/Go pattern); -> self-calibrating, immune to frame-size/opt-level/platform variance, matches clj's -> depth model. (2b) make `stack_overflow` catchable with its OWN Kind → -> `StackOverflowError` (NOT reusing `.resource_exhausted`, which must stay -> uncatchable for the eval-budget sandbox — the two limits look alike but have -> opposite catchability requirements; the current conflation is a latent bug). Cost: -> per-thread stack-base capture (workers capture at spawn) + a portable -> remaining-stack read (`@frameAddress()`); the safety margin is one *one-sided -> over-estimable* number, far less fragile than a two-sided cap. -> -> **Alt 3 (wildcard, trampoline)** — re-express every HOF primitive's callback as -> "push a VM frame, yield to the iterative `eval` loop, resume with the result" so -> primitive re-entry costs ~1 native frame per VM frame and the existing -> `FRAMES_MAX=2048` becomes an accurate native bound (no new limit). Most -> finished-form-pure and likely a perf win (D-450), but a depth-4 CPS transform of -> the primitive layer touching the GC-rooting surface (the reduce accumulator moves -> from a native local to a collector-walked VM slot). Right *eventual* shape when -> the JIT/fusion work re-touches the calling convention; over-reach now. -> -> **(a) chokepoint**: `vm.eval` entry is a correct *counter* floor but wrong -> *granularity* (counts all activations equally); `invokeCallable` is the narrower -> primitive-re-entry-specific site. The stack-pointer guard dissolves the question — -> it measures the resource directly; site it at `vm.eval` entry for coverage ± -> `invokeCallable` for earlier detection. -> **(b) self-calibration**: a stack-remaining guard relocates the one constant from -> a two-sided tightrope cap to a one-sided over-estimable margin. (A guard page is -> the OS-level version but fragile across the GC's own signal use + the Wasm-edge -> target; the explicit SP check is the portable no-signal form.) -> **(c) catchability**: in-scope, non-optional. `.stack_overflow` Kind -> `.resource_exhausted` → `kindToHostClass` `null` → uncatchable; clj's is -> catchable. Must give stack-overflow its OWN Kind → `StackOverflowError`, distinct -> from the (correctly-uncatchable) eval-budget bound. Land 2b first (independently -> testable), then 2a. -> **(d) depth vs clj**: a frame *count* cap is a coarse proxy for clj's -> JVM-stack-bytes model and is pessimistic (counts non-primitive nested evals too); -> the stack-pointer guard errors "when the real stack is nearly full" = clj's model -> off only by the margin. Alt 3 matches exactly (same 2048 limit as plain -> recursion) at depth-4 cost. - -## Consequences - -- **Positive**: (2b) `stack_overflow` becomes catchable, matching clj — closes the - F-011 catchability divergence the DA surfaced. (2a) no input crashes the process; - all unbounded callback recursion errors gracefully, self-calibrated to the real - native stack (no cross-host cap fragility); subsumes the watch-specific partial. -- **Negative**: (2a) per-thread stack-base capture is more plumbing than a - threadlocal counter (workers capture at spawn); a portable remaining-stack read - (`@frameAddress()`) + one over-estimable safety margin remain. -- **Follow-up**: Alternative 3 (trampoline primitive callbacks → `frame_top` - accurate) recorded as forward debt for the gap-area-III primitive-calling- - convention rework. - -## Affected files - -- **2b**: `src/runtime/error/catalog.zig` (stack_overflow → own catchable Kind) + - `src/runtime/host_class.zig` (Kind → `"StackOverflowError"`) + an e2e asserting - `(catch StackOverflowError …)` works. -- **2a**: `src/eval/backend/vm.zig` (capture stack base + remaining-stack check at - `eval` entry ± `invokeCallable`) + `src/runtime/iref.zig` + - `src/lang/primitive/atom.zig` (remove the subsumed watch-nesting partial) + - `test/e2e/phase15_watch_recursion.sh` (extended: validator + reducer repros). -- `.dev/debt.yaml` — D-485 discharged when 2a lands; a new forward-debt row for - Alternative 3. diff --git a/.dev/decisions/0158_single_binary_component_embedding.md b/.dev/decisions/0158_single_binary_component_embedding.md deleted file mode 100644 index 25c674676..000000000 --- a/.dev/decisions/0158_single_binary_component_embedding.md +++ /dev/null @@ -1,107 +0,0 @@ -# ADR-0158 — `cljw build`: embed `:require`d Wasm components into the single binary - -- **Status**: Proposed → Accepted → Implemented (2026-06-21, user-directed; D-404 Impl D) -- **Driven by**: the user directive that a module which `(:require ["x.wasm" :as x])`s a - Wasm component must, after `cljw build`, produce a **self-contained single binary** (no - external `.wasm` at runtime). Single-binary deployability is a cljw selling point - (lightweight + CLI-handy) and must not be forfeited when the new-code worldview reaches - for Wasm components (ADR-0135 Amendment 1). -- **Relates to**: ADR-0135 (component-as-namespace, the surface), ADR-0034 (`cljw build` - bytecode envelope), F-001 (zwasm embedding), F-016 (always-latest Wasm/WASI consumer - invariant). External: zwasm's Component-Model embedding API (functional, default-ON). - -## Context - -`cljw build` compiles a program + the bundled `.clj` core to one **bytecode envelope** -embedded in the binary (`app/builder.zig`: `buildEnvelope` / `buildBootstrapEnvelope`); -the embedded-run startup installs an **embedded-only resolver** that serves the bundled -namespaces from the envelope instead of the filesystem (so a shipped binary needs no -`.clj` on disk). ADR-0135 Amendment 1 adds `(:require ["x.wasm" :as x])` — a component -load. Naively, the built binary would still read `x.wasm` from the filesystem at runtime, -breaking the single-binary contract. - -## Decision - -**`cljw build` embeds the bytes of every statically-resolvable `:require`d component into -the envelope, and the embedded-run startup resolves component `:require`s from those bytes -— a parallel arm to the existing embedded `.clj` resolver.** - -1. **Collect (build time)**: while compiling, the builder records each string-libspec - component `:require` it encounters, resolves it (ADR-0135 A2 order: relative / absolute - / classpath), and reads the `.wasm` bytes. -2. **Embed**: the bytes go into a **component table** in the envelope (keyed by the - resolved logical id — the alias or normalized path), beside the bytecode. Raw component - bytes are embedded **as-is** (not native-precompiled — see Alternatives); the embedded - zwasm loads/JITs them at runtime exactly as a filesystem load would. -3. **Resolve (embedded run)**: startup installs an **embedded component resolver** that, - given a `:require`d component id, returns the embedded bytes (and only falls through to - the filesystem for a genuinely dynamic `(wasm/load "runtime.wasm")` whose target was not - known at build time). So `(:require ["x.wasm" :as x])` in a built binary loads from - memory; the binary is self-contained + portable. - -The mechanism mirrors the `.clj` envelope arm exactly (embedded table + embedded-only -resolver), so it is *addition, not new subsystem*. - -## Alternatives considered - -- **Native-precompile each component (Cranelift `.cwasm` / Wasmer `create-exe` - object+link)** — rejected for v1: it ties the binary to one CPU arch (the cljw binary is - cross-arch via `-Dcpu=baseline`), is far heavier (a linker step per component), and - loses parity with the filesystem-load path (which loads raw bytes). Embedding raw bytes - keeps one load path (the embedded zwasm JITs at runtime, same as a normal load) and stays - arch-neutral. A precompile fast-path can be added later for cold-start without changing - this contract. -- **External `.wasm` sidecar next to the binary** — rejected: defeats the single-binary - contract the user explicitly requires. -- **Embed every component, including dynamic `(wasm/load runtime-path)`** — impossible: a - path only known at runtime cannot be embedded at build time. So the embeddable set is - the **statically-resolvable `:require`-string** components; truly-dynamic `wasm/load` - remains a filesystem read in the built binary (documented; the embedded resolver falls - through). This is the correct boundary, not a gap. - -## Consequences - -- A `cljw build`'d program that `:require`s components is **fully self-contained** — one - portable binary, no `.wasm` sidecars. Startup loads components from memory (no FS). -- **Binary size** grows by the embedded component sizes (a markdown component ~hundreds of - KB). Acceptable + expected; the builder should `log` the embedded component count + total - bytes so the size is visible, not silent. -- The boundary "static `:require` = embeddable / dynamic `wasm/load` = filesystem" is a - clear, documented contract; it mirrors how AOT vs runtime-`require` already differ. -- No core-VM risk: all `-Dwasm`-flag-guarded; the default gate never resolves zwasm (F-001). - -## Affected files (when implemented) - -- `app/builder.zig` (the component-collection + embed into the envelope; the embedded - component resolver at startup), `runtime/cljw/wasm/component.zig` + the resolution arm - from ADR-0135 A2 (a load-from-embedded-bytes path beside load-from-file), the envelope - serialize/deserialize (`eval/bytecode/serialize.zig`) for the component table. -- Tracked as ADR-0135's implementation phase **D** (sub-row of the D-404 epic). - -### As implemented (2026-06-21, D-404 Impl D) - -- **Envelope format**: the component table is the OUTERMOST envelope section - (`[u32 n_components]` then per component `[u32 path_len][path][u32 byte_len][wasm bytes]`), - written ahead of the entry manifest + chunk list by `serialize.serializeEnvelope` (now - takes a `components: []const EmbeddedComponent` arg). `readEnvelopeEntry` / - `deserializeEnvelope` / `EnvelopeIterator.init` each `skipComponentTable` first; - `readComponentTable` materialises it (slices into the payload) for the startup install. - No version bump — the envelope is internal to one binary (build + run share serialize.zig); - an empty table (`n=0`) is written for every component-free build. -- **Type**: `Runtime.EmbeddedComponent { path, bytes }` lives in Layer 0 so the Layer-1 - serializer and the Layer-0 component surface share one type (zone-clean downward import). -- **Collect**: `analyzeNs` appends each resolved component path to `rt.component_sink` - (set ONLY during a build; null on REPL/run) — the key matches what the desugar bakes as - the run-side `wasm/load-component` arg, so the embedded-table lookup hits. -- **Impl-B build-side gap fixed in the same cycle**: `buildEnvelope` did not thread the - source label into the reader, so source-relative `./x.wasm` resolved against cwd (not the - source dir) at BUILD — the build-time component load raised IoError. `buildEnvelope` now - takes `source_label` and sets `reader.file_name` (mirror of `runner.zig`), so build and - run resolve identically. - -### Sources (research, 2026-06-21) - -- Single-binary Wasm embedding precedents: Wasmtime pre-compiling (`.cwasm`, - `engine.precompile_module`), Wasmer `create-exe` (wasm → static object → C-linker). Both - inform the *rejected* native-precompile alternative; cljw embeds raw bytes for arch - neutrality + load-path parity. \ No newline at end of file diff --git a/.dev/decisions/0159_wasm_resource_lifecycle.md b/.dev/decisions/0159_wasm_resource_lifecycle.md deleted file mode 100644 index 1b0d18c29..000000000 --- a/.dev/decisions/0159_wasm_resource_lifecycle.md +++ /dev/null @@ -1,318 +0,0 @@ -# ADR-0159 — Wasm Component `resource` lifecycle: `own`-handle ownership + drop (D-404 Impl E) - -- **Status**: Proposed → Accepted (2026-06-21, user-directed; D-404 Impl E resource ergonomics; DA-fork incorporated) -- **Driven by**: ADR-0135 Amendment 1's promise that a `:require`d Wasm Component's - exports feel native — including WIT `resource`s. Today an `own`/`borrow` handle lifts to - a bare integer; methods work (the `resource_counter` demo returns 6) but nothing ever - runs the guest `resource.drop` destructor for an `own` handle → a **resource leak** for - the lifetime of the component. -- **Relates to**: ADR-0135 (component-as-namespace), ADR-0158 (single-binary embed), - F-001 (zwasm embedding), F-006 (mark-sweep GC; cljw heap ⟂ Wasm linear memory — separate - spaces), F-016 (always-latest). External: zwasm `Opened.dropResource` (REQ-5). - -## Context - -A WIT `resource` (e.g. `resource counter { constructor(start:u32); increment:func()->u32; -get:func()->u32 }`) exports a constructor returning `own` + methods taking `borrow`. -cljw's `invokeTyped` lift currently maps `.own`/`.borrow` → `Value.initInteger(handle)`, so -a handle is an opaque integer the Clojure code threads back into method calls. This works -but: - -- **Leaks**: nothing calls `resource.drop`, so the guest destructor never runs until the - whole component is torn down (the `ComponentLoaded` box's GC finaliser runs - `Opened.deinit()`, which frees the entire resource table — so there is no leak *across* - components, only *within* a long-lived component holding many short-lived resources). -- **Untyped**: a bare integer carries no own/borrow distinction and no tie to its owning - component — a handle from component A could be passed to component B's method. - -zwasm already exposes the mechanism: **`Opened.dropResource(handle: u32)` (REQ-5)** runs the -declared destructor for an `own` handle (graph path only; a single-module component has no -resource table → `NoResourceTable`). Verified zwasm semantics (`resource_table.zig`): - -- **Double-drop traps** (stale handle) — drop is NOT idempotent; the cljw wrapper must guard. -- **Handle indices are reused** (free-list) — a handle is opaque, single-use across drop, - scoped to one `Opened`; cljw must never cache/compare a handle past its drop. -- **`own` vs `borrow`**: dropping an `own` runs the destructor; dropping a `borrow` - decrements the lender's `num_lends`; an `own` can't be dropped while lent - (`HandleStillBorrowed`). cljw reads own-vs-borrow straight from the WIT result type - (`resolveFuncSig` → `.own`/`.borrow`). - -(A mailbox note `from_cljw_05` asks zwasm to confirm these four points; all are read from -zwasm source, so the design proceeds on them — confirmation is a safety check, not a gate.) - -## The crux — finaliser ordering hazard (cljw-side, NOT a zwasm concern) - -cljw's GC is **mark-sweep, non-moving** (F-006), with NO guaranteed finaliser ordering -between two host objects collected in the same cycle. An `own`-handle wrapper whose GC -finaliser calls `componentBox.opened.dropResource(handle)` risks a **use-after-free**: if -the wrapper AND the `ComponentLoaded` box are both unreachable in one GC cycle and -`componentFinalise` (which `Opened.deinit()`s) runs first, the resource finaliser touches a -freed `Opened`. Rooting the component from the wrapper (hold the component handle Value in -`host_trace`-marked state) keeps the component alive while a resource is *reachable*, but -does not order the two finalisers when both are simultaneously garbage. The Decision below -picks the rooting + ordering mechanism. - -## Decision - -**Deterministic release is the contract; the resource wrapper's finaliser NEVER touches -zwasm state.** An `own` result becomes a typed wrapper; release is via a deterministic -`(wasm/resource-drop h)` / a `with-resource` scope (the Clojure-idiomatic `with-open` -pattern), and `componentFinalise`'s existing `Opened.deinit()` drains the whole resource -table at component teardown as the backstop. The wrapper roots its owning component (so a -held resource cannot outlive its `Opened`), but carries **no finaliser that calls -`dropResource`**. - -Concretely: - -1. **Wrapper**: an `own` result → a `.host_instance` (`resource_descriptor`) with - `state = { component-handle Value (state[0]), raw handle u32 (state[1]), dropped-flag - (state[2]) }`. `descriptor.host_trace` marks state[0] (the `ComponentLoaded` box Value), - rooting the component while any resource is reachable — the java.util.Iterator - cursor-rooting precedent (mark-only, non-moving GC; `gc_rooting.md §H`). **No - `host_finalise`** on the resource wrapper. -2. **lower**: a wrapper passed to a `borrow`/`own` param yields state[1] (the raw handle); - a wrapper whose dropped-flag is set raises a use-after-drop error before the call. -3. **release** (`(wasm/resource-drop h)` / `with-resource` exit): if not already dropped, - reach the component via state[0] → `ComponentLoaded.opened.dropResource(state[1])` + set - the one-shot dropped-flag (double-drop guarded cljw-side; `dropResource` would otherwise - trap on the stale handle). Safe because the caller holds the wrapper, which roots the - component → `Opened` is live at the call. -4. **backstop**: `componentFinalise` already runs `Opened.deinit()`, freeing the entire - resource table — every un-dropped resource is released at component teardown. - -### Divergence from the Devil's-advocate recommendation (the safety correction) - -The DA recommended **C+** = `with-resource` + a **B′ registry where the resource wrapper's -finaliser deregisters-and-drops** ("valid because the rooting edge proves `Opened` is alive -at that instant"). **This per-resource-finaliser drop is NOT safe in cljw's GC** and is -therefore rejected: rooting via `host_trace` keeps the box alive only while the resource is -**reachable**; once the resource is garbage it no longer marks the box, so when a resource -and its component become garbage in the **same mark-sweep cycle**, both are swept together -with no ordering guarantee (the DA's own fact 4). At the resource finaliser's instant the -box may already be freed — so *any* box/`Opened` touch from the resource finaliser (drop OR -even "deregister", which dereferences the box's intrusive set) is a potential -use-after-free. The structurally safe form removes the resource finaliser entirely: the -wrapper holds the component Value by `host_trace` (a passive mark, no finaliser), release is -deterministic (held wrapper ⇒ live component), and the only GC-time drop is -`componentFinalise` draining its OWN table — exactly one finaliser, touching only state it -owns. The DA's deterministic-`with-resource`-is-the-finished-form thesis is adopted; only -its unsafe per-resource-finaliser path is dropped (this is a correctness divergence, not a -cycle/LOC downgrade — F-002 is upheld: the chosen form is *more* correct, not smaller). - -### Scope / known limitation (documented, not a leak) - -An individual resource that is GC'd **without** an explicit `resource-drop` / `with-resource` -is released at **component** teardown, not at its own death. For a long-lived component -minting many short-lived resources, use `with-resource` for timely release. This is the safe -trade vs. an unsafe finaliser; "GC drop" is honoured as *component*-GC drops all its -resources. The one-shot `(wasm/component-invoke …)` path (no cached component) keeps the -bare-integer lift for `own` results — there is no persistent component to own them; the -typed wrapper applies to the `require-component` / `component-call` cached-handle path. - -## Alternatives considered - -*(Devil's-advocate fork, fresh context, verbatim — the per-resource-finaliser-drop in its -B′/C+ is corrected in the Decision's "Divergence" note above.)* - -> **Leading finding — no F-NNN is violated by any candidate.** A GC-finalizer drop touches -> only zwasm's `Opened` (zwasm's separate space, F-006), reached through a cljw-side pointer -> the resource wrapper already holds; it never traces Wasm linear memory into cljw's heap, -> never unifies the two spaces. F-001/F-006/F-016/F-002 are all satisfiable by every shape -> below. The real axis is **finalizer-ordering safety + idiomatic determinism**, not law. -> -> **(1) Smallest-diff — A, explicit-drop-only.** `own` becomes a thin wrapper host_instance -> holding `{component-handle Value, handle:u32, dropped-flag}`; `host_trace` marks the -> component Value; `(wasm/resource-drop h)` calls `opened.dropResource`, flips the one-shot -> flag; double-drop guarded cljw-side; NO finalizer drop. Leak bounded by `componentFinalise`'s -> full-table teardown. *Better*: zero finalizer-ordering surface — the hazard cannot arise. -> *Breaks*: contradicts ADR-0135's "GC drop"; a resource never explicitly dropped lives until -> *component* teardown — a long-lived component minting many short-lived resources accumulates -> dead handles. -> -> **(2) Finished-form-clean — B′, finalizer-via-registry (the box drops, not the resource).** -> GC-finalizer auto-drop, but the resource finalizer never calls `dropResource` directly; -> the `ComponentLoaded` box owns an intrusive set of live handles; the wrapper roots the box; -> the wrapper's finalizer only deregisters; `componentFinalise` drains the set before -> `opened.deinit()`. *Better*: true GC-lifecycle drop with the ordering hazard structurally -> dissolved — exactly one site touches `Opened`. *Breaks*: the per-resource immediate-drop-on- -> deregister path must prove the box is alive at that instant; double-drop guard still needed; -> the intrusive set must be freed; `HandleStillBorrowed` means a still-lent `own` can't drop. -> -> **(3) Wildcard — C+, `with-open`/`with-resource` deterministic scope + B′ backstop.** -> Primary path is `(with-resource [c (counter/new 5)] …)` dropping at scope exit -> (idiomatic, `with-open`-shaped); the GC-finalizer (B′) is best-effort for escapees. -> *Better*: deterministic release is the Clojure-idiomatic answer AND has a real backstop. -> *Breaks*: largest surface; scope-exit drop and the backstop must share the one-shot flag -> or double-drop; the macro needs a `try/finally` lowering. -> -> **Recommendation: adopt C+ layered on B′** (deterministic `with-resource` as the contract, -> registry-backed finalizer where the box drops). Per F-002 do NOT downgrade to A on -> diff/LOC grounds — A leaves "drop-when-the-value-dies" unsolved. -> -> **What the implementer must verify first:** that the fixture's only drop-observable signal — -> a method on a dropped handle *traps* — is reachable through cljw's catch surface as a -> deterministic assertable error, AND that the rooting edge keeps the box alive across a -> forced GC while a resource is reachable. - -## Consequences - -- Resources are released **deterministically** (`with-resource` / `resource-drop`) or at - **component teardown** (the `Opened.deinit()` backstop) — no leak across components, and - **no use-after-free** (no resource-finaliser → `Opened` path). -- An `own` handle is a **typed wrapper tied to its component** (can't be passed to another - component's method); double-drop is guarded; use-after-drop raises a catchable error. -- **Limitation** (documented above): an un-scoped individual resource releases at component - teardown, not at its own GC — `with-resource` is the timely path. One-shot - `component-invoke` keeps the bare-int `own` lift. -- Implementation is incremental: **cycle 1** = the wrapper + `host_trace` rooting + lift/lower - + `(wasm/resource-drop)` + the use-after-drop guard + the "method-on-dropped-handle traps" - e2e; **cycle 2** = the `with-resource` macro sugar. Both deliver the same finished form; - each is independently green. - -## Affected files (when implemented) - -- `runtime/cljw/wasm/component.zig` — the `own`-handle wrapper (`.host_instance` + - `resource_descriptor` with `host_finalise`/`host_trace`); `lift`/`lower` changes to wrap - an `own` result + extract the raw handle from a wrapper passed to a `borrow`/`own` param. -- `lang/clj/cljw/wasm.clj` — the surface (`require-component*` interns method Vars; a drop - form / `with-open` per the Decision). -- e2e `phase16_wasm_require_component.sh` + a fixture — drop observed via "method on a - dropped handle traps" (the `resource_counter` fixture has no drop-observable export). - -## Amendment 1 (2026-08-04) — the one-shot's `own` was a dangling number, and the "tied to its component" Consequence was never enforced - -This ADR's Scope section reads: "The one-shot `(wasm/component-invoke …)` path -(no cached component) keeps the bare-integer lift for `own` results — there is -no persistent component to own them." That framed the one-shot lift as -*untyped*. It is worse than untyped: `componentInvokeFn`'s `defer opened.deinit()` -destroys the instance and its resource table **before the Value reaches the -caller**, so the integer is already invalid when it is returned. - -Measured on `resource_counter`: the one-shot constructor returned `1`, and -passing that `1` to a method gave an opaque "WebAssembly module trapped". It -trapped only because the fresh instance's table happened to be empty. zwasm's -`resource_table.zig` is a dense per-instance array with index 0 reserved, so -**the first resource any instance mints is handle `1`** — one `resource.new` -away, that same call silently succeeds against a *different* object. - -Reviewing the fix surfaced two more holes on the path this ADR recommends as -the cure, both of which contradict its own Consequences: - -- `rawResourceHandle` accepted a bare integer via `@intCast(v.asInteger())`. - `asInteger` does not check the tag, so any Value became a table index, and a - negative integer was a **ReleaseSafe `@intCast` panic on caller data** — the - exact host-crash class the numeric marshalling rows in the same file had - already closed. -- Nothing checked that a resource wrapper belonged to the component being - called. This ADR's Consequences claim an `own` handle "can't be passed to - another component's method"; that was never implemented. Two live components - each holding a resource at index 1 meant `(wasm/component-call c method a)` - operated on `c`'s resource while reading as `a`'s. - -### Decision - -**The component's lifetime is its reachability whenever the result names -something inside it.** `componentInvokeFn` heap-allocates the same -`ComponentLoaded` triple `wasm/load-component` builds, and decides from the -resolved signature — before invoking — which lifetime applies: - -- result holds no resource → the triple is freed at return, exactly as before; - the common case is unchanged and still deterministic. -- result holds a resource at any depth → the triple goes into the GC-owned box, - the `own` lifts through the existing `makeResourceHandle`, and `resourceTrace` - roots the component for as long as the resource is reachable. - -`own` therefore has ONE meaning on both entry points, which discharges the gap -ADR-0135 amendment 2 recorded as "a real gap … `own`/`borrow` are one table row -over a two-tier behaviour". `component-invoke` becomes `component-call` where -the component is anonymous — a difference in naming, not in what values mean. - -Two consequences of that principle, landed in the same cycle: - -- **A resource handle names its own component**, so `wasm/component-call` - accepts one as its first argument. Without this a resource returned by a - one-shot invoke would be live but unusable: its component has no other name, - and the caller could only drop it, never call a method. -- **`rawResourceHandle` takes only a wrapper, and only one whose owner is the - component being called.** There is no legitimate source of a raw handle - number — every one a caller can hold came out of a component call already - wrapped. Two new Codes (`wasm_resource_expected`, `wasm_resource_foreign`) - rather than reusing `wasm_opts_invalid`, whose template hard-codes - `wasm/load:` and would mislabel the raise site. - -`witTypeHoldsResource` recurses through list / option / tuple / record / -variant / result: a top-level-only check would send -`record { text: string, handle: own }` down the teardown branch and -reproduce the defect one level down. - -`openComponentBoxed` / `freeComponentBox` now hold every partial-failure unwind -for the triple, and both entry points use them. That is not tidying: the -one-shot decides ownership *after* opening, so caller-side `errdefer`s for the -same resources fire alongside its own teardown. The first version of this fix -did exactly that and segfaulted on `0xaa…` — freed memory — on the error path. - -### Verification - -`test/e2e/fixtures/wasm_component_probe.clj` gains two cases, both of which fail -against the pre-fix binary (checked by rebuilding without the source change): -the one-shot resource surviving a forced `System/gc` and a full -construct → method → drop → use-after-drop chain, and the cross-component -rejection in both directions plus the raw-integer and negative-integer cases. -`witTypeHoldsResource`'s recursion is unit-tested at every arm, including -`list }>`, because no fixture in the tree has an -aggregate carrying a resource. - -### What this does not change - -The GC-drop limitation stands: an un-scoped resource still releases at component -teardown rather than at its own death, and `with-resource` remains the timely -path. That is now uniform across both entry points instead of applying only to -one. - -### Alternatives considered (amendment 1) - -*(Devil's-advocate fork, fresh context, verbatim. Its "facts verified in code" -preamble is preserved because the three code findings in it — the pre-call -placement, the dense-table handle numbering, and the two unenforced holes on the -cached path — are what changed the shape of this amendment.)* - -> **Leading finding — no F-NNN blocks any candidate below.** Every shape keeps the `Opened` in zwasm's separate space (F-006) and reaches it only through a cljw-side pointer the wrapper already holds; none traces Wasm linear memory into cljw's heap; none needs a zwasm API that the v2.4.0 pin lacks (F-001). The axis is **where the owner identity lives**, not law. Per F-002 the recommendation below is the largest-diff shape. -> -> **Facts verified in code, not assumed.** -> -> 1. *The result type is known before the call.* `invokeOnOpened` resolves `sig` and does not call `invokeTyped` until later. `sig.result` and every `sig.params[i].ty` are in hand, fully expanded (zwasm's `resolveFuncSig` chases named/nested provenance and preserves `own`/`borrow` at every depth). **Any rejection must be pre-call.** The draft as stated ("lift the result, notice it contains `own`, raise") runs the guest constructor first: the resource is minted, the guest allocates, any WASI side effect the export performs has happened, and only then does the caller get an error. Pre-call refusal is both cheaper and semantically different — it is the only form where "this call is not expressible" is true rather than "this call already happened and I am discarding it". -> 2. *The integer has no cross-instance meaning — provably, not probably.* zwasm's `resource_table.zig` is a per-`WasiP2Ctx` dense `slots` array plus a `free` list, with index 0 reserved as the `None` sentinel and freed slots tombstoned and **reused**. `add` returns `slots.items.len` on an empty table, i.e. **the first resource any fresh instance mints is handle `1`**. The measured trap is therefore the benign corner of the defect; the malign corner ("index 1 names a different object and the call silently succeeds") is exactly one `resource.new` away. There is no cross-instance semantics to preserve, so no alternative needs to keep the integer for compatibility. The only surviving "legitimate" use is a handle the caller intends to *ignore* — which matters for the compound-result case below. -> 3. *The same defect is live on the parameter side, and it is worse.* `rawResourceHandle` falls through to `@intCast(v.asInteger())` for any non-wrapper. `Value.asInteger` is an unchecked `@truncate` + `@bitCast` — it does not verify the tag. So `(wasm/component-call c method "oops")` indexes a table with garbage, and a negative integer (`-1`) is a **ReleaseSafe `@intCast` panic on caller data** — precisely the host-crash class this file's own comments congratulate themselves on having closed for the numeric rows. Furthermore `lower` never receives the callee's component identity, so a wrapper minted by component A is accepted verbatim by component B's method. **ADR-0159's Consequences claim "an `own` handle is a typed wrapper tied to its component (can't be passed to another component's method)" is not enforced anywhere in the code.** A change that fixes only the one-shot lift ratifies a same-shaped bug on the path it recommends as the cure. -> 4. *A one-shot invoke whose PARAMS contain `own`/`borrow` is unusable for the identical reason.* The instance is opened inside `componentInvokeFn` with an empty table; no handle the caller could possess names anything in it. The draft does not cover this. -> 5. *`component-invoke` has no user-facing contract to break.* It is interned with `null` meta (no docstring); `docs/` never mentions it; its only description is a Zig doc comment calling it "the experiment's roundtrip probe". ADR-0135 Amendment 2 already records this exact split as "**a real gap**". Latitude to redefine it is total. -> 6. *Precedent.* wasmtime does not answer this with "use the other API". `ResourceAny` carries its store's identity and a cross-store use is a **detected error at the use site**; the `&mut Store` borrow makes it structurally impossible for a handle to outlive the store as a usable thing. The industry answer is *make the handle carry its owner and validate at use*, not *refuse the call*. JVM Clojure's nearest idiom is `with-open`: it does not forbid a function from returning the stream it closed; the failure is `IOException: Stream closed` at use. -> -> **(1) Smallest-diff — pre-call refusal (the draft, corrected).** In `invokeOnOpened`, immediately after `sig` resolves and **before** `invokeTyped`, when `component_handle == null`, walk `sig.result` and every `sig.params[i].ty` with a small recursive `containsResource(WitType) bool` and raise. Add a dedicated Code rather than reusing `.wasm_opts_invalid`, whose template is hard-coded `"wasm/load: {detail}"` and would mislabel the raise site. -> *Better than the others*: nothing invalid is ever constructed, and no guest code runs before the refusal — the diagnosis is deterministic instead of depending on whether the fresh instance's table happens to be empty. Smallest surface, no lifetime change, no GC interaction. -> *Breaks*: it rejects the **whole call** when the resource is an ignorable leaf — `record { text: string, handle: own }` now returns nothing, and the perfectly-liftable `text` is lost. It makes `component-invoke` non-total over the export surface `component-exports` advertises, so any code generator driven by that listing acquires holes. And it leaves fact 3 untouched: the bare-integer acceptance and the missing owner check on the *cached* path are the same bug wearing a different hat, and this alternative sends users toward them. -> -> **(2) Finished-form-clean — delete the one-shot's private lifetime; the component is owned by whoever can still reach it.** `componentInvokeFn` stops being a second implementation of instantiation. It allocates the `Engine`/`WasiHost`/`ComponentLoaded` triple on the heap exactly as `loadComponentFn` does (today they are **stack locals** — that is the whole mechanism of the bug), wraps it in the `component_descriptor` host_instance, and passes that Value into `invokeOnOpened` as `component_handle`. An `own` result then lifts through the *existing* `makeResourceHandle`, whose `resourceTrace` already roots the component; `wasm/resource-drop` and `with-resource` work identically on both paths; `componentFinalise`'s `Opened.deinit()` remains the backstop. Deterministic teardown is preserved for the overwhelmingly common case by deciding pre-call from `sig.result`: **no resource in the result ⇒ tear down at return exactly as today; resource in the result ⇒ the component's lifetime is its reachability.** One sentence describes it: *the component lives as long as anything can still name a thing inside it.* -> The same cycle closes fact 3, which this shape exposes rather than creates: `rawResourceHandle` stops accepting bare integers (a resource param must be a wrapper — a raw index is not a value a user can legitimately construct), and `lower` gains the callee's component identity so a wrapper whose `state[0]` is a different component is rejected with a named error instead of indexing a foreign table. That check also subsumes alternative (1)'s parameter case for free: the only wrapper you could pass to a one-shot invoke belongs to some other component, so the identity check produces the right diagnostic without a separate rule. -> *Better than the others*: it deletes the defect class instead of diagnosing it. `own` acquires **one** meaning across every entry point, which discharges the gap ADR-0135 Amendment 2 explicitly deferred, and reduces `component-invoke` to "`component-call` where the component is anonymous" — a difference in *naming*, not in *what values mean*. The compound-result case returns everything, usable. It is the shape wasmtime arrived at (owner-carrying handle, validated at use). -> *Breaks*: the largest diff — `componentInvokeFn` inherits `loadComponentFn`'s `errdefer` LIFO discipline, and the conditional-teardown branch is a second lifetime rule inside one function, which must be documented at the function, not only in the ADR. The real residual cost is F-006-adjacent: Wasm linear memory is invisible to cljw's GC pressure heuristics, so a loop of one-shot invokes each returning a resource can hold many instances alive until a cljw-heap-triggered collection. That exposure already exists for `load-component`; this makes it reachable from a function whose name reads "one-shot". The honest mitigation is documentation plus `with-resource`, not a smaller design. -> -> **(3) Wildcard — the born-dropped tombstone.** Lift `own`/`borrow` on a component that is about to be torn down into the same `resource_descriptor` host_instance, but constructed **already dropped**: `state[2] = 1`, `state[0]` empty. The call succeeds, the enclosing record/tuple/list lifts intact, `wasm/resource-drop` is a no-op (it is already idempotent), and passing it anywhere raises the **existing** `.wasm_resource_dropped` — "wasm component resource handle used after it was dropped", which is *literally true*: the one-shot teardown dropped it. No new error Code, no lifetime change, ADR-0159's guard reused verbatim. -> *Better than the others*: it preserves partial results and does not punish a call made for its side effects; the failure lands at the point of misuse rather than at a call the user may have had every right to make; and it is honest about the mechanism (the instance died at return) instead of asserting the operation is unsupported. -> *Breaks*: it manufactures a value whose only purpose is to fail later — a subtler contract than an error, and a caller who ignores the result learns nothing, which is how the current bug got shipped. `resourceTrace` unconditionally does `@enumFromInt(state[0])` and marks it, so an empty `state[0]` needs an explicit guard — the tombstone is not free in the GC surface. It perpetuates the two-tier `own` mapping ADR-0135 already flagged. And it leaves the deferred-error debugging distance essentially where the bug is today; only the message improves. There is also no `wasm/resource-dropped?` predicate today, so the tombstone is not even inspectable without adding one. -> -> **Recommendation: (2), explicitly citing F-002.** (1) is the smaller diff and (3) is the cleverer one, but both accept as permanent the thing that produced this bug: **one WIT type having two meanings depending on which cljw function you entered through.** (2) removes the second meaning. Do not downgrade to (1) on diff size — the diff is larger precisely because it also closes the unenforced ADR-0159 Consequence (bare-int acceptance, missing owner check, `asInteger` panic on caller data), which is the same defect on the path the draft would recommend as the cure. If (2) is rejected, then (1) must still be taken **with** the parameter arm, the pre-call placement, and a separate follow-up row for the cached-path holes — the draft as written covers none of the three. -> -> **What the implementer must verify first.** (a) That a one-shot `own` result actually reaches the wrapper path — `Opened.dropResource` is `NoResourceTable` for the `.single` variant, so a resource-bearing single-module component would surface as the catch-all `.wasm_trap` ("WebAssembly module trapped"), a misleading message that deserves its own Code under **any** of the three shapes. (b) That the rooting edge survives a forced GC between the invoke's return and the resource's first use. (c) `test/e2e/fixtures/wasm_component_probe.clj` currently has **no** one-shot resource case at all; the winning shape needs a probe there driving `resource_counter`'s constructor through `component-invoke`, plus its expectation line in `test/e2e/phase16_wasm_component.sh`, or the same false-positive-discharge class that `clj_diff_sweep.md` Discipline 1 was written against reappears. - -**Adopted: (2)**, per F-002 and the DA's own reasoning. Its three -implementer-verification items were all discharged in this cycle: (b) the forced -`System/gc` assertion and (c) both e2e probes are in -`wasm_component_probe.clj`. (a) remains open as a message-quality gap — a -`.single`-variant component carrying a resource surfaces `NoResourceTable` as -the catch-all `wasm_trap`; tracked as **D-568**, not fixed here because no -fixture in the tree reaches it. - -## Revision history - -- 2026-08-04: Amendment 1 — one-shot `own` lifetime + the two unenforced - cached-path holes (`wasm_resource_expected`, `wasm_resource_foreign`). diff --git a/.dev/decisions/0160_rounding_mode_host_enum_singleton.md b/.dev/decisions/0160_rounding_mode_host_enum_singleton.md deleted file mode 100644 index cdef57654..000000000 --- a/.dev/decisions/0160_rounding_mode_host_enum_singleton.md +++ /dev/null @@ -1,132 +0,0 @@ -# ADR-0160 — `java.math.RoundingMode` enum constants as host-enum object singletons (not int ordinals) - -- **Status**: Proposed → Accepted (2026-06-24; clj-parity sweep; DA-fork incorporated) -- **Driven by**: the stdlib/contrib clj-parity differential sweep. cljw did not - register `java.math.RoundingMode`, so `java.math.RoundingMode/HALF_UP` raised - `No namespace: 'java.math.RoundingMode'` and the clj-modern rounding call - `(.setScale (bigdec "3.14159") 2 java.math.RoundingMode/HALF_UP)` errored where - clj returns `3.14M`. -- **Relates to**: ADR-0061 (`Class/FIELD` static-field read), ADR-0087 (heap-singleton - static fields resolved at analyze time), ADR-0115 (Locale host-object singletons — the - pattern this mirrors), ADR-0106 (`.host_instance` container). F-005 (numeric tower = - JVM surface), F-011 (behavioural equivalence), F-013 (closed-set host SSOT), F-002 - (finished-form wins). AD-002 (opaque-ref `#` print). - -## Context - -In JVM Clojure the modern rounding API is the `java.math.RoundingMode` **enum** -(`RoundingMode/HALF_UP` etc.); the `BigDecimal.ROUND_*` **int** constants are the -deprecated legacy form. The enum's 8 ordinals match the `ROUND_*` ints exactly -(UP=0 … UNNECESSARY=7). cljw already carried the deprecated int constants -(`BigDecimal/ROUND_HALF_UP` → 4) and a `setScale(int,int)` that accepts an int mode -0-7, but had no `RoundingMode` class at all. - -clj's observable surface for an enum constant: - -- `(str RoundingMode/HALF_UP)` → `"HALF_UP"` (the enum's toString = its name) -- `(class RoundingMode/HALF_UP)` → `java.math.RoundingMode` -- `(= RoundingMode/HALF_UP RoundingMode/HALF_UP)` → `true`; `(= RoundingMode/HALF_UP 4)` → **false** (enum ≠ Long) -- `(pr-str RoundingMode/HALF_UP)` → `#object[java.math.RoundingMode 0x… "HALF_UP"]` (non-reproducible address) -- `(.setScale bd n RoundingMode/HALF_UP)` → rescaled value - -## Decision - -Register `java.math.RoundingMode` as a host class whose 8 enum constants are -**host-enum object singletons** (the Locale `singleton` pattern, ADR-0115), NOT -plain int ordinals: - -- Each constant is a per-Runtime cached `.host_instance` carrying its ordinal in - `state[0]`, with a descriptor `fqcn = "java.math.RoundingMode"` and a `.toString` - method returning the bare enum name. Caching (8 `rt.rounding_modes` slots) gives - `=` / `identical?` parity. -- A new `StaticFieldValue.rounding_mode: u8` arm carries the ordinal; the analyzer's - `staticFieldValue` resolves it to the cached singleton (same analyze-time lift as - Locale's `singleton`). -- `BigDecimal.setScale`'s mode arg accepts **both** a `RoundingMode` host_instance - (read `state[0]`) and the deprecated `ROUND_*` int — both JVM-faithful, both decode - to 0-7. -- The canonical name↔ordinal mapping lives once in the neutral - `runtime/rounding_mode.zig`; **both** the `RoundingMode/` enum table AND - BigDecimal's `ROUND_` int table are comptime-generated from it (no dual - source of the ordinal numbering). - -This reaches clj parity on `str` / `class` / `=` / `setScale`. The only divergence is -the opaque print form `#` vs clj's `#object[… 0x… "HALF_UP"]` -— the non-reproducible identity-hash class **already covered by AD-002** (same as -Locale / atom / every host opaque ref). No new accepted-divergence entry is required. - -## Alternatives considered - -A `general-purpose` Devil's-advocate subagent (fresh context) was forked with the -active F-NNN constraints and produced the following (reflected verbatim): - -> **No alternative requires violating an F-NNN.** All three sit inside the envelope -> (F-005, F-013, AD-003, F-009). The int-ordinal draft is itself F-NNN-compliant; the -> disagreement is which finished form a future maintainer respects. -> -> **The fact that reframes the decision.** clj's `with-precision` (core.clj L5153-5157) -> expands `:rounding HALF_UP` to `(. java.math.RoundingMode HALF_UP)` and passes the -> result to `(java.math.MathContext. precision rm)`. So the dominant consumer of -> `RoundingMode` constants is a **`MathContext` constructor argument**, not `setScale`. -> cljw has no `MathContext` host class yet. An int passed to a future `MathContext.` -> constructor would be ambiguous with the `(int precision)` first arg — a typed enum -> disambiguates where a bare int cannot. -> -> **Alt 1 — Smallest-diff: int-ordinal but on the SHARED neutral surface.** Keep the -> int representation but generate both tables from one `[8]{name,ord}` source (no -> duplicated literal). Better: eliminates dual-source ordinal drift. Breaks everything -> the draft breaks: `(class …)` → Long; prints `4`; future `MathContext.` int/precision -> ambiguity. Reject on F-002 — but **fold the commonization into whatever wins**. -> -> **Alt 2 — Finished-form-clean: host-enum singleton (Locale pattern), ordinal in -> state[0].** `(class …)` correct; future `MathContext.` typed arg; `(str …)` = "HALF_UP" -> (draft gives "4"). `(= RoundingMode/HALF_UP 4)` is **false** here and in clj — whereas -> the draft makes it **true**, a divergence clj does NOT have. **AD-051 as drafted would -> lock in a non-clj equality** — heavier than the draft's consequence section admits. -> Would a maintainer unwind "HALF_UP = int 4"? Yes, high-confidence: the moment -> `MathContext`/`with-precision :rounding` lands, the int/precision ambiguity + `(class)` -> = Long breakage force a conversion. The int draft is a skeleton that *enlarges* the -> eventual rewrite (ships an AD that must be un-accepted + a setScale that must learn a -> second arg shape) — the disqualifying signal per F-002. -> -> **Alt 3 — Wildcard: a general host-enum mechanism.** `RoundingMode` is one instance of -> a recurring need (`TimeUnit`, `DayOfWeek`/`Month` — already ad-hoc in `TemporalPrint`, -> `ChronoUnit`). Grow a first-class `StaticFieldValue.host_enum_const {fqcn,name,ordinal}` -> + interned singleton + uniform bare-name print, subsuming the bespoke temporal arms. -> Correct everywhere + reusable. Cost: largest diff (not an F-002 constraint), but it is a -> **structural-plan** change — per F-003 the loop imagines + records + defers to the owning -> unit, it does not seize it mid-sweep. Record as a debt row; land `RoundingMode` today in -> the shape the mechanism absorbs cleanly = Alt 2's singleton. -> -> **Recommendation: Land Alt 2 (host-enum singleton), with Alt 1's commonized constant -> source folded in, and open a debt row for Alt 3.** Reject the int-ordinal draft: it -> locks a non-clj equality and a Long class identity a maintainer will unwind the instant -> `MathContext` lands. Alt 2 reaches the correct finished form now, is the exact shape -> Alt 3 generalizes, and its only true divergence (opaque print address) is the -> already-accepted AD-002 class. - -The main loop adopted the recommendation in full: **Alt 2 + Alt 1's commonized constant -source**, with **Alt 3 deferred to debt D-510**. - -## Consequences - -- `java.math.RoundingMode/` resolves (FQCN always; bare `RoundingMode/` after - `(import java.math.RoundingMode)`, matching clj which also requires the import). -- `setScale` accepts the enum constant; the deprecated `ROUND_*` int path is unchanged. -- `str` / `class` / `=` match clj; `pr-str` diverges under the pre-existing AD-002 (no new AD). -- The ordinal numbering is single-sourced — a future ordinal audit reconciles one table. -- `divide` with a `(scale, RoundingMode)` overload and a `MathContext` host class remain - gaps; both are the natural next consumers of this shape (and of D-510's mechanism). - -## Affected files - -- `src/runtime/rounding_mode.zig` (new — neutral: Mode/name/singleton/deinit) -- `src/runtime/java/math/RoundingMode.zig` (new — surface: descriptor + static fields + toString) -- `src/runtime/type_descriptor.zig` (`StaticFieldValue.rounding_mode` arm) -- `src/eval/analyzer/analyzer.zig` (`staticFieldValue` resolves the new arm) -- `src/runtime/runtime.zig` (`rounding_modes` slots + deinit call) -- `src/runtime/java/math/BigDecimal.zig` (setScale dual-accept; ROUND_* table commonized) -- `src/runtime/java/_host_api.zig` (register the surface) -- `compat_tiers.yaml` (java.math.RoundingMode entry, keyword `rounding_mode`) -- `test/e2e/phase14_bigdecimal_setscale.sh` (9 RoundingMode cases) -- `.dev/debt.yaml` (D-510 — general host-enum mechanism, deferred) diff --git a/.dev/decisions/0161_general_host_enum_mechanism.md b/.dev/decisions/0161_general_host_enum_mechanism.md deleted file mode 100644 index 79f539683..000000000 --- a/.dev/decisions/0161_general_host_enum_mechanism.md +++ /dev/null @@ -1,325 +0,0 @@ -# ADR-0161 — General host-enum mechanism: one `.host_instance` registry for the four JVM host enums - -- **Status**: Proposed → Accepted (2026-06-24; D-510 discharge; DA-fork incorporated) -- **Driven by**: D-510. ADR-0160 (RoundingMode) was the second one-off host-enum - singleton after ChronoUnit; the DA-fork there explicitly recorded that the - finished form is a first-class mechanism and deferred it to D-510. The barrier - (a 2nd/3rd host enum actually needed) has dissolved — there are now FOUR host - enums across TWO representations, which is the duplication D-510 exists to retire. -- **Relates to**: ADR-0160 (RoundingMode host-enum singleton — the immediate - predecessor this generalises), ADR-0106 (`.host_instance` container), ADR-0115 - (Locale singleton pattern), ADR-0061/0087 (static-field resolve at analyze time), - ADR-0059 / AD-003 (no-JVM simple class name), AD-002 (opaque-ref `#` print), - D-462 (DayOfWeek/Month typed_instance values). F-002 (finished-form wins), F-004 - (NaN-box 64-slot — no new tag), F-009 (impl/surface/peer split), F-011 - (behavioural equivalence), F-013 (closed-set host SSOT). - -## Context - -cljw modelled four JVM host enums with **two different representations**: - -**Group A — `.host_instance` static-field singletons** (`java.math.RoundingMode`, -`java.time.temporal.ChronoUnit`): each constant a per-Runtime cached `.host_instance` -(`state[0]` = ordinal), reached via a `StaticFieldValue.rounding_mode: u8` / -`.chrono_unit: u8` union arm. `RoundingMode/HALF_UP` works today. - -**Group B — `.typed_instance` getter-minted values** (`java.time.DayOfWeek`, -`java.time.Month`): each a `.typed_instance` (`fields()[0]` = ISO value) with ONE -per-Runtime descriptor, `temporal_print = .day_of_week`/`.month`, minted **fresh** -on every getter call (`(.getDayOfWeek ld)`). NO static-field constants — -`DayOfWeek/MONDAY` raises a Name error. - -The clj oracle (the finished form to match), verified 2026-06-24: - -- All four enums **have** static-field constants (`DayOfWeek/MONDAY`, `Month/JANUARY`, - `RoundingMode/HALF_UP`, `ChronoUnit/DAYS`). -- `(str X)`: DayOfWeek/Month/RoundingMode → enum NAME; ChronoUnit → DISPLAY name - ("Days", not "DAYS"). `.name` → always enum name. `.ordinal` → 0-based. - `.getValue` (DayOfWeek/Month only) → 1-based ISO value. -- **Getters return the SAME singleton as the constant**: - `(identical? (.getDayOfWeek some-monday) DayOfWeek/MONDAY)` → **true**. cljw's - fresh-mint Group B makes this impossible today (and there is no constant to compare). -- clj's print form `#object[java.time.DayOfWeek 0x "MONDAY"]` embeds an - unmirrorable identity hash — the AD-002 opaque-ref class. - -So cljw's two groups even diverge from **each other**: Group A prints `#` -(opaque, no enum name); Group B prints bare `MONDAY` (name, no wrapper). Neither -matches clj. And Group B lacks the static-field constants AND the getter-singleton -identity that clj has. The two-representation split is a maintenance trap: a fifth -host enum has no single mechanism to copy, and `getValue` / `ordinal` / display-name -are scattered across four files. - -## Decision - -Introduce ONE host-enum mechanism, all four enums on `.host_instance`, with a -**comptime registry** + a **single flat registry-indexed cache + single interning -entry point** (the draft + DA Alt-3, adopted in full): - -1. **`src/runtime/host_enum.zig`** (neutral) — a comptime registry. Each row = - `{ fqcn, count, cache_base, name table, to_string table, value table (optional) }`. - The four rows: RoundingMode (8), ChronoUnit (16), DayOfWeek (7), Month (12). - Folds `rounding_mode.zig` + `chrono_unit.zig` + `time/day_of_week_value.zig` + - `time/month_value.zig` into it. Exposes `singleton(rt, enum_idx, ordinal)`, - `name`, `toString`, `value`, `ordinalCount`. -2. **One flat cache** `rt.host_enum_consts[43]` (8+16+7+12) indexed by - `registry[enum_idx].cache_base + ordinal`, replacing `rt.rounding_modes[8]`, - `rt.chrono_units[16]`, `rt.day_of_week_descriptor`, `rt.month_descriptor`. - `singleton` is the **sole** minting point — every path (static-field read, - getter return, method dispatch) routes through it, so `identical?` parity is - structural: no getter can re-mint a fresh value. -3. **One `StaticFieldValue.host_enum { enum_idx: u8, ordinal: u8 }`** arm replacing - `.rounding_mode` + `.chrono_unit`. The analyzer's `staticFieldValue` resolves it - via `host_enum.singleton`. -4. **`TypeDescriptor.host_enum_idx: ?u8`** identifies a descriptor as a host enum - (which one), replacing the `TemporalPrint.day_of_week`/`.month` arms. The - print / equal / compare paths branch on it. -5. **DayOfWeek/Month GAIN static-field constants** via new surface files - (`runtime/java/time/DayOfWeek.zig`, `Month.zig`) with `static_fields` tables - registered in `rt.types` — so `DayOfWeek/MONDAY` resolves (clj-parity win). - Their getters return the cached singleton (closes the `identical?` gap). -6. **One print path**: the `.host_instance` printer emits the enum's `toString` - (name, or display name for ChronoUnit) keyed by `host_enum_idx`, dropping both - the Group-A `#` opaque form and the Group-B `temporal_print` arms. - `equal` (by descriptor + ordinal — interned, so pointer-identical) and `compare` - (by ordinal — all four are Comparable in JVM) migrate to `.host_instance`. - -`java.math.MathContext` is **not** a member: it is not a Java enum (value class with -a public constructor + standard constants, `state = {precision, mode}`). It stays a -`.host_instance` value class and remains a *consumer* of the mechanism (its -`:rounding` arg is now a typed RoundingMode enum). Its `.math_context: u8` arm is -unchanged. - -Print divergence from clj (`#` form vs `#object[… 0x… "MONDAY"]`) is the -pre-existing AD-002 opaque-ref class (unmirrorable identity hash) — no new AD. - -## Alternatives considered - -A `general-purpose` Devil's-advocate subagent (fresh context) was forked with the -active F-NNN constraints + the draft, and produced the following (reflected verbatim): - -> ## Grounding (verified against source, 2026-06-24) -> -> - **Group A** (`rounding_mode.zig`, `chrono_unit.zig`): `.host_instance`, -> `state[0]=ordinal`, reached via `StaticFieldValue.rounding_mode: u8` / -> `.chrono_unit: u8` arms in `type_descriptor.zig` (L216/L219). Per-Runtime -> fixed-array caches give `=`/`identical?` parity. `MathContext` is a -> `.math_context: u8` arm consumer. -> - **Group B** (`time/day_of_week_value.zig`, `time/month_value.zig`): -> `.typed_instance`, `fields()[0]=ISO value`, ONE per-Runtime descriptor each, -> `temporal_print = .day_of_week`/`.month`. Getters mint **fresh** via -> `day_of_week_value.make(rt, …)` (`local_date_value.zig:66`, -> `local_date_time_value.zig:72`). Participate in `equal.zig:1119-1129` -> (`isDayOfWeek`/`valueOf` by value field) and `compare.zig:203` (`temporalOrder` -> typed_instance arm). NO static-field constants — `DayOfWeek/MONDAY` raises. -> - Both representations are F-004-legal (reuse existing tags 29 `.host_instance` -> and the `.typed_instance` tag). The draft picks `.host_instance` as the single -> home. -> -> The draft is sound and finished-form-clean. Below are three alternatives inside -> the F-NNN envelope, then a recommendation. -> -> ## Alternative 1 — Smallest-diff: leave representation split, add the four missing static-field constants only -> -> **Shape.** Do NOT unify representations. Keep Group A on `.host_instance` and -> Group B on `.typed_instance` exactly as today. Close only the *observable* parity -> gaps: add `DayOfWeek/MONDAY` … and `Month/JANUARY` … static-field constants by -> giving `type_descriptor.zig` two new `StaticFieldValue` arms -> (`.day_of_week_const: u8`, `.month_const: u8`) that resolve to a per-Runtime -> cached singleton (a `rt.days_of_week[7]` / `rt.months[12]` fixed-array cache -> mirroring `rt.rounding_modes`). Repoint `getDayOfWeekFn`/`getMonthFn` from -> `day_of_week_value.make` (fresh mint) to that cache so -> `(identical? (.getDayOfWeek monday) DayOfWeek/MONDAY)` holds. Add `name`/`ordinal`/ -> `getValue` methods to each of the four descriptors individually. -> -> **Better than the draft.** Surgically minimal — touches no -> `equal.zig`/`compare.zig`/`print.zig` arms (they keep working as-is on -> typed_instance). Zero risk of regressing the temporal `=`/compare/print paths -> that are already green and corpus-locked. The `identical?` parity (the one -> genuinely impossible-today behaviour) is closed with the least moving parts. No -> new neutral module, no fold, no cache-shape migration. -> -> **What it breaks / costs.** It is the **Smallest-diff bias smell** crystallised, -> and it explicitly **violates the project's own forward reservation**: -> `chrono_unit.zig:11` already records "the eventual general host-enum mechanism -> (D-510) folds both", and the draft's whole reason for existing is that four -> near-identical enums with two representations, two cache shapes, two print paths, -> two equality paths, and (now) **four** static-field-arm pairs is exactly the -> duplication D-510 was opened to retire. This alternative *adds* a fifth and sixth -> `StaticFieldValue` arm and a third and fourth fixed-array cache, growing the very -> surface the registry was meant to collapse. Per F-002 (finished-form wins; -> cycle/diff size is not a constraint), shipping this because it is smaller is the -> disposition CLAUDE.md names forbidden. It also leaves the two-representation split -> as a permanent trap: the next person adding a host enum must guess which of two -> mechanisms to copy, and `getValue` (DayOfWeek/Month only) vs `ordinal` (all four) -> vs display-name (ChronoUnit only) stays scattered across four files instead of one -> registry table. -> -> ## Alternative 2 — Finished-form-clean (rival to the draft): unify on `.typed_instance` with a registry, NOT `.host_instance` -> -> **Shape.** One host-enum mechanism, but consolidate the **other** direction — fold -> all four onto `.typed_instance` (one integer field = the enum's distinguishing -> value), with a comptime registry `host_enum.zig` keyed by a per-Runtime descriptor -> pointer: Registry row = `{ fqcn, count, value_kind: .ordinal | .iso_value, name_fn, -> to_string_fn }`. RoundingMode/ChronoUnit migrate from `.host_instance` *to* -> `.typed_instance` (field[0]=ordinal); DayOfWeek/Month stay `.typed_instance` -> (field[0]=ISO value). Static-field constants resolve via ONE -> `StaticFieldValue.host_enum { enum_idx, ordinal }` arm to a per-Runtime cached -> singleton typed_instance. Print stays on the existing `temporal_print`-style -> descriptor flag, generalised to a `host_enum_print` enum that all four share -> (`name` vs `display` selectable per registry row) — extends the path -> `print.zig:1143` already has rather than introducing a `host_instance`-keyed print -> path. `equal.zig`/`compare.zig` keep their existing typed_instance arms (already -> present for DayOfWeek/Month at `equal.zig:1119` / `compare.zig:203`); -> RoundingMode/ChronoUnit *gain* a typed_instance equality arm (by descriptor + -> field), which is the same shape already proven for the temporal types. -> -> **Better than the draft.** Two concrete advantages. (1) **DayOfWeek/Month/ChronoUnit -> already have live, corpus-locked `equal`/`compare`/`print` machinery on -> `.typed_instance`** — this alternative *extends* proven paths and migrates only the -> two `.host_instance` enums *into* that fold, whereas the draft does the reverse: it -> rips DayOfWeek/Month *out* of their working typed_instance arms (the -> `equal.zig:1119-1129` `isDayOfWeek`/`valueOf` arms, the `compare.zig` `temporalOrder` -> arm, the `print.zig` `.day_of_week`/`.month` arms) and re-implements all of them -> host_instance-keyed — a larger, riskier touch on the more heavily-exercised half. -> (2) `.typed_instance` carries a real `field_layout` and participates in the standard -> field-walker, so a future moving GC sees the value field natively; -> `.host_instance.state[]` is a raw `[4]u64` that `host_instance.zig:50-56` explicitly -> flags as **invisible to the field-walker** (needs a `host_trace` hook for moving GC, -> debt D-318). Enum singletons are immutable scalars so this is academic today, but the -> typed_instance home is the one that does not owe the moving-GC migration a hook. -> -> **What it breaks / costs.** RoundingMode/ChronoUnit currently *justify* -> `.host_instance` in their own docstrings ("same discipline as the Locale -> singletons"), and `.host_instance` is the canonical home for FQCN-keyed host objects -> with a shared method_table — moving them to `.typed_instance` means each enum needs -> its own per-Runtime descriptor (four descriptors instead of leaning on the shared -> host_instance tag dispatch), and method dispatch (`name`/`ordinal`/`toString`) moves -> from the host_instance descriptor's method_table to typed_instance method tables. It -> also makes the four enums structurally indistinguishable from user `deftype` values -> at the tag level (both `.typed_instance`), so any code that branches "is this a host -> value vs a user record" must go through the descriptor, not the tag — a subtle -> invariant the `.host_instance` tag currently makes free. Net: it is finished-form-clean -> and arguably *cleaner on the GC and print axes*, but it pays with four descriptors and -> a tag that no longer self-identifies "host". -> -> ## Alternative 3 — Wildcard: enum constants as **interned keyword-like scalars**, not heap instances at all -> -> **Shape.** Observe that an enum constant is a pure `(enum_idx, ordinal)` pair with no -> per-instance state — it is closer to a keyword than to a `java.util.Random`. Instead -> of a heap `.host_instance`/`.typed_instance`, give each constant a single canonical -> heap value **interned once per (enum, ordinal)** and handed out by identity, the way -> `type_descriptor.zig:111-119`'s `ref_cache` already interns a descriptor's boxed -> identity. Concretely: keep them as `.host_instance` (F-004: no new tag) but make -> `host_enum.singleton` the *sole* minting point for every path — static-field read, -> getter return, and `name`/`ordinal`/`getValue` all route through one interner so the -> per-(enum,ordinal) value is bit-identical everywhere. The wildcard twist is to drop -> the per-enum *fixed array* caches (`rt.rounding_modes[8]`, `rt.chrono_units[16]`) in -> favour of ONE flat `rt.host_enum_consts` array sized at the comptime sum of all -> registry counts (8+16+7+12 = 43), indexed by `registry[enum_idx].cache_base + ordinal`. -> One cache, one interner, one deinit loop. -> -> **Better than the draft.** Collapses *four* per-Runtime cache slots -> (`rt.rounding_modes`, `rt.chrono_units`, + the two the draft/Alt-1 would add for -> DayOfWeek/Month) into **one** flat array with a comptime base-offset table — the -> registry literally owns the cache layout, so adding a fifth host enum is one registry -> row and the cache auto-resizes at comptime with zero new `rt` fields and zero new -> deinit loop. This is *more* finished-form-clean than the draft on the Runtime-struct -> axis (the draft still implies a per-enum cache or a single cache; making it explicitly -> one flat registry-indexed array is the cleaner end state). It also makes `identical?` -> parity structural rather than per-call-site: there is exactly one interning entry -> point, so no getter can accidentally mint a fresh value (the bug Group B has today at -> `make(rt, …)`). -> -> **What it breaks / costs.** The base-offset indexing (`cache_base + ordinal`) is a -> small piece of comptime cleverness that the per-enum fixed arrays avoid — a future -> enum with a *sparse* or *non-zero-based* ordinal set would need care (none of the four -> have this; JVM enum ordinals are always dense 0-based, so the risk is theoretical). It -> does not, on its own, resolve the `.host_instance` vs `.typed_instance` print/equal/ -> compare question — it still has to pick one tag (it picks `.host_instance` like the -> draft), so it inherits the draft's work of migrating DayOfWeek/Month's typed_instance -> `equal`/`compare`/`print` arms to host_instance. So Alt-3 is best read as **a refinement -> layered on top of the draft (or Alt-2)**, not a standalone rival: "whichever tag wins, -> make the cache one flat registry-indexed array with a single interning entry point" — -> that is the genuinely novel, finished-form-improving idea here. -> -> ## No F-NNN forces a violation -> -> All three alternatives and the draft stay inside F-004 (reuse `.host_instance` or -> `.typed_instance`; no new tag), F-009 (impl `host_enum.zig` neutral, surfaces under -> `runtime/java/**` import it, never the reverse — the zone rule holds for every option), -> and F-002 (none is recommended on size grounds). There is **no** clean option that would -> require violating an F-NNN, so there is no leading "would-violate-F-NNN" finding to record. -> -> ## RECOMMENDATION -> -> **Adopt the draft (unify on `.host_instance` via a comptime registry), but fold in -> Alternative 3's single flat registry-indexed cache + single interning entry point.** The -> draft is correctly finished-form-clean and the registry is the right answer to the -> two-representation duplication that `chrono_unit.zig` itself flags as D-510's target; -> Alternative 1 is the Smallest-diff smell (it adds two more static-field arms and two more -> caches to the very surface D-510 exists to collapse) and should be rejected on F-002 -> grounds even though it is the smallest diff. Between the two finished-form rivals, the -> draft's choice of `.host_instance` is the better *tag* choice because `.host_instance` -> self-identifies "host value" at the tag level (Alt-2's all-typed_instance world makes host -> enums indistinguishable from user `deftype` records without a descriptor probe) and because -> `.host_instance` is the established home for FQCN-keyed host objects with a shared -> method_table — exactly what `name`/`ordinal`/`toString`/`getValue` dispatch wants. -> Alternative 2's one real edge — that it *extends* the already-green DayOfWeek/Month -> typed_instance `equal`/`compare`/`print` arms instead of re-homing them — is a -> transition-cost argument, not a finished-form argument, so under F-002 it does not outrank -> the draft; but the migration of those arms is the draft's highest-risk step and must land -> with the DayOfWeek/Month corpus cases re-run to lock parity. Alternative 3 is the one idea -> worth absorbing wholesale: make the registry own a single flat cache (`cache_base + ordinal`, -> comptime-sized to 43) with one interning entry point, so `identical?` parity is structural -> (no getter can re-mint) and adding the fifth host enum is a one-row registry edit with zero -> new `rt` fields. Advisory only — the main loop chooses. - -The main loop adopted the recommendation **in full**: the draft's `.host_instance` -registry + Alt-3's single flat `rt.host_enum_consts[43]` cache and sole interning -entry point. Alt-1 rejected (Smallest-diff smell, F-002). Alt-2's flagged risk -(migrating DayOfWeek/Month's `equal`/`compare`/`print` arms) is honoured as a hard -gate: the DayOfWeek/Month corpus + e2e cases are re-run to lock parity in the same -cycle the arms move. - -## Consequences - -- `DayOfWeek/` and `Month/` now resolve (clj-parity gap closed); their - getters return the cached singleton, so `(identical? (.getDayOfWeek monday) - DayOfWeek/MONDAY)` is **true** (was impossible). -- `RoundingMode` / `ChronoUnit` behaviour is preserved (regression-locked by the - existing corpus); they move onto the shared mechanism with no observable change. -- One print path for all four host enums: the enum `toString` (name, or display name - for ChronoUnit). Opaque print divergence stays under AD-002 (no new AD). -- Adding a fifth host enum (e.g. `TimeUnit`) is a one-row registry edit + one surface - file — zero new `rt` fields, zero new `StaticFieldValue` arms, zero new cache. -- The two-representation split + four scattered name/ordinal/value definitions - collapse into one `host_enum.zig` table. `MathContext` stays a consumer (typed - RoundingMode arg), not a member. -- Staged across two TDD cycles: cycle 1 builds the mechanism + migrates Group A - (behaviour-preserving); cycle 2 folds Group B (the parity wins + the arm migration). - -## Affected files - -- `src/runtime/host_enum.zig` (new — neutral: registry + flat cache + singleton/name/ - toString/value/deinit; folds rounding_mode.zig + chrono_unit.zig + day_of_week_value.zig - + month_value.zig) -- `src/runtime/rounding_mode.zig`, `src/runtime/chrono_unit.zig`, - `src/runtime/time/day_of_week_value.zig`, `src/runtime/time/month_value.zig` (deleted — folded) -- `src/runtime/type_descriptor.zig` (`StaticFieldValue.host_enum` arm replaces - `.rounding_mode`/`.chrono_unit`; `host_enum_idx` field; TemporalPrint loses - `.day_of_week`/`.month`) -- `src/eval/analyzer/analyzer.zig` (`staticFieldValue` resolves `.host_enum`) -- `src/runtime/runtime.zig` (`host_enum_consts[43]` replaces `rounding_modes`/ - `chrono_units`/`day_of_week_descriptor`/`month_descriptor`; deinit) -- `src/runtime/print.zig`, `src/runtime/equal.zig`, `src/runtime/compare.zig` - (host-enum arms keyed by `host_enum_idx`; remove the Group-B typed_instance arms) -- `src/runtime/java/math/RoundingMode.zig`, `src/runtime/java/time/ChronoUnit.zig` - (regenerate static_fields via the registry; set `host_enum_idx`) -- `src/runtime/java/time/DayOfWeek.zig`, `Month.zig` (new — surface with static_fields - + getValue), registered in `runtime/java/_host_api.zig` -- `src/runtime/time/local_date_value.zig`, `local_date_time_value.zig` (getters return - the cached singleton) -- `src/main.zig` (test aggregator imports), `compat_tiers.yaml` -- `test/e2e/phase14_*.sh` + `test/diff/clj_corpus/` (DayOfWeek/Month static-field + - identical? cases; RoundingMode/ChronoUnit regression cases) -- `.dev/debt.yaml` (D-510 discharged) diff --git a/.dev/decisions/0162_cold_start_bootstrap_architecture.md b/.dev/decisions/0162_cold_start_bootstrap_architecture.md deleted file mode 100644 index 8beda59ab..000000000 --- a/.dev/decisions/0162_cold_start_bootstrap_architecture.md +++ /dev/null @@ -1,232 +0,0 @@ -# ADR-0162 — Cold-start bootstrap architecture: lazy-namespace bytecode + zero-copy deserialize now; heap-snapshot deferred to the GC unit - -- **Status**: Proposed → Accepted (2026-06-24; D-450 startup-floor lever; DA-fork incorporated verbatim below) -- **Driven by**: D-450 (gap-III fastest-script campaign) reconstructed lever order - named the **startup floor** as the highest cross-cutting lever (it is added to - EVERY cold bench, and the top gap `sieve` is floor-sensitive). Measure-first - (2026-06-24) attributed the floor; this ADR fixes the architecture the floor work - follows so the optimisation does not drift into the cljw-v0 "Zig-ify the bootstrap" - rut or into an impressive-but-wrong heap-snapshot. -- **Relates to**: ADR-0056 (AOT bytecode envelope — the mechanism this evolves), - ADR-0034 am1 (interleaved chunk envelope), ADR-0158 (single-binary embedding), - D-140 (self-exe footer-seek), D-452 (the eager-whole-bootstrap blob this re-laces), - D-515 (binary-size axis), the "Eager comprehensiveness vs single-binary size/startup" - standing tension (debt.yaml, F-013 clause 4). F-002 (finished-form wins), F-004 - (NaN-box 64-slot, absolute pointer in payload), F-006 (mark-sweep GC, non-moving), - F-011 (behavioural equivalence), F-013 (single-binary, no side artifacts). -- **User-declared invariant honoured**: the optimisation NEVER moves a `.clj`-defined - var into hand-written Zig (the cljw-v0 89K-LOC rut); it touches only the RESTORE - MECHANISM and the EAGER SET. .clj stays the definition language. - -## Context - -`cljw -e 1` cold floor ≈ 9.4 ms (quiet Mac M4 Pro, ReleaseSafe). Env-gated profiler -(`CLJW_PROFILE_STARTUP=1`, ADR-introduced) attribution: - -| phase | time | note | -|-------------------------------------------|---------|-----------------------------------------------------------------| -| self-exe read (`tryRunEmbedded`, cli.zig) | ~1.0 ms | reads the whole 8.8 MB binary to check a 12-byte footer (D-140) | -| `runEnvelope` deserialize | ~2.1 ms | parse 891 bytecode chunks into heap structures | -| `runEnvelope` vm.eval | ~2.5 ms | intern vars, build collections, def fns | -| `registerAll` (primitives) | ~0.2 ms | — | -| exec/linker/page-fault residual | ~3.4 ms | un-addressable process overhead (in-process-unmeasured) | - -The eager bootstrap is **891 chunks**: clojure.core = 294 top-level forms (~21%); the -~28 non-core stdlib namespaces (string/set/walk/zip/edn/data.json/data.csv/tools.cli/ -pprint/test/data/math/spec.alpha/…) = 1148 forms (**~79%**). A `cljw -e 1` one-liner — -and most edge handlers — touch essentially only clojure.core. D-452 Part B made all 29 -namespaces one eagerly-replayed blob "so non-core libs no longer re-parse from source"; -that fixed a re-parse cost by paying a full eager-replay cost. Neither "lazy + bytecode" -(no re-parse AND no eager replay) nor "no replay at all" exists yet. - -An industry survey (SBCL `save-lisp-and-die`, Emacs pdumper, V8 snapshots, GraalVM -image-heap, Ruby bootsnap, PEP 690) found: runtimes that beat ~5 ms do not replay a -bootstrap — they **map a build-time heap snapshot**. cljw is at the "bootsnap tier" -(replay precompiled bytecode); `@embedFile` already removes the disk-IO bootsnap targets, -so that tier offers nothing new. The headline industry answer is a heap snapshot -("Candidate B"). - -A devil's-advocate red-team (folded verbatim below, checked against `nan_box.zig`, -`cache_gen.zig`, `mark_sweep.zig`, `root_set.zig`) inverted the front-runner: **B buys -only core's ~1 ms replay share, behind a ~3.4 ms un-addressable wall, at the cost of the -worst bug class this codebase can ship** (a single un-relocated absolute / `builtin_fn` -pointer = silent heap corruption with no fault), a second per-tag relocate-slot -reflection surface the GC does not provide today, and a per-target cache_gen↔runtime -layout-skew build-matrix lock. B is also GC-coupled: a future moving/generational GC -already needs the identical "rewrite every pointer slot" machinery and may adopt -base-relative encoding that makes snapshot relocation free — so building B now against -the non-moving GC is a twice-built / soon-unwound shape (Smallest-diff-bias-against- -finished-form smell). - -## Decision - -Pursue the cold-start floor in this sequence; each step is GC-independent and never -touches the F-004/F-006 pointer surface: - -1. **D-140 — self-exe footer-seek.** `stat` the binary size, seek, read only the last - 12 bytes (the `[u64 len]["CLJC"]` footer); positioned-read the payload region only on - a magic match. Removes the ~1 ms whole-binary read on every normal startup. - Independent, trivial, zero risk. **Land first.** - -2. **Lazy-namespace bytecode loading (Candidate A).** The eager set shrinks to the true - minimal "needed for the default runtime to behave correctly with no `require`" = - clojure.core + its load-time dependencies (e.g. core.protocols / the error-render - peer / core macros' helpers — determined by audit, not guessed). Every other namespace - becomes its **own embedded bytecode envelope** (still `@embedFile`d — F-013 held, no - external files) that `require` / first-unqualified-var-reference replays on demand via - the existing `driver.runEnvelope`. Expected `-e 1` floor → **~4.8 ms** (with D-140). - - **F-011 correctness gate (mandatory, same arc):** (a) a build-time check that EVERY - embedded namespace replays clean (cache_gen already builds each envelope, so this is - a small extension) — neutralises "load errors move from boot to first-use"; (b) an - audit of load-time global-table side effects (`defmethod` / `defmulti` / `derive` / - `extend-type` / `print-method` installs) so no default-path behaviour silently - depends on a now-lazy namespace. This matches JVM Clojure (core auto-loads; the rest - you `require`) — the current eager-all gives *accidental* eager semantics that must - not be silently load-bearing. - -3. **Zero-copy in-place bytecode deserialize (Candidate Alt-3).** Redesign the envelope/ - chunk format so chunks are position-independent and the VM reads bytecode arrays + - constant pools straight from the `@embedFile`'d `.rodata` with no per-chunk - allocation/copy. Attacks the ~2.1 ms deserialize with **zero pointer-relocation risk** - (rodata is read-only; the VM builds live heap objects the normal way, so every pointer - is born absolute-correct through the existing codec). Applied to core's eager chunks - AND reused by the lazy non-core chunks. Expected floor → **~3.5–3.8 ms** (sub-4 ms), - no F-004/F-006 surface touched. - -4. **Defer Candidate B (heap snapshot) to the generational/nursery-GC unit.** Recorded as - forward debt. When the moving-GC unit decides the pointer encoding, B's relocation - rides that unit's already-built slot-rewrite machinery (possibly *free* under - base-relative encoding), turning B from a premature twice-built ~1 ms-for-max-risk - gamble into a clean increment. B is NOT pursued now. - -Also landed up front (this commit): the env-gated startup profiler -(`src/runtime/startup_profile.zig` + coarse marks in cli/runner/bootstrap), the campaign's -cold-start measurement instrument — off by default, a single null-check on the hot path -when unset (a <10 ms process is invisible to `sample`, so in-binary marks are the only way -to attribute the floor). - -## Consequences - -- **Sub-4 ms cold floor reachable without opening the heap-relocation correctness surface.** - The ~3.4 ms exec/linker/page-fault residual becomes the ceiling; further floor work is - the binary-size axis (D-515: smaller binary → fewer first-touch faults), not bootstrap. -- **Lazy-ns discharges the eager-comprehensiveness-vs-size/startup standing tension** and - caps the growth the spec.alpha/contrib sweep would otherwise add to every startup. -- **A new correctness contract**: the eager set is a deliberate, audited boundary, and a - build-time per-ns replay-clean gate guarantees no lazy namespace ships broken. The - F-011 side-effect-visibility audit is a one-time cost paid in step 2. -- **Alt-3 makes the envelope format load-bearing/rigid** — a chunk-layout change becomes an - on-rodata-format change. Accepted: the format is already AOT-embedded; the future JIT - (ADR-0200) operates on a different layer. -- **B's deferral is recorded forward debt**, explicitly co-owned with the moving-GC unit, - so it is not lost and not built prematurely. - -## Alternatives considered - -The devil's-advocate red-team output (fresh-context subagent, facts checked against the -named sources) is reproduced verbatim. Its verdict drove the Decision above — in -particular the inversion of the industry-survey front-runner (Candidate B) into a -deferred, GC-coupled increment, and the elevation of Alt-3 (zero-copy deserialize) as the -correct partner for lazy-ns. - -> **Leading finding.** Candidate B (build-time heap snapshot) is not a clean fit for this -> codebase as currently shaped. The GC reaches live objects through TWO mechanisms — -> `root_set.enumerate` (small fixed root set) and per-tag *trace* functions -> (`tag_ops.tag_trace_table`). A trace fn answers "what does this object point TO" for the -> mark phase; it does not hand you the ADDRESS of the slot holding the pointer. -> Relocation needs the latter. So B requires building a SECOND, parallel per-tag -> *relocate-slot* table that does not exist today, ~doubling the per-tag maintenance the -> trace table imposes; every one of the 64 possible heap tags must implement BOTH or -> silently corrupt the snapshot. -> -> **Alt 1 (smallest-diff): lazy-namespace (A) + D-140, core stays eager-bytecode.** No -> value-representation coupling, no relocation table, zero NaN-box correctness surface; -> the only option needing no pointer-layout understanding. Largest single win (3.6 ms) for -> the dominant real case. Risk: deferred load-time side effects (`defmethod`/`derive`/ -> `extend-type`/`print-method` into global tables invisible until the ns is required) — -> an F-011 hazard (JVM Clojure is equally lazy, but cljw's single-blob gave accidental -> eager semantics users may depend on); and load errors move boot→first-use, mitigated by -> a build-time "every embedded ns replays clean" gate. Respects all F-NNN. -> -> **Alt 2 (finished-form-clean): snapshot CORE's heap only (B scoped to core) + lazy -> bytecode the rest, base-relative pointer encoding internal to the image.** Snapshotting -> all 29 ns is wasted — you relocate 1148 forms a one-liner never dereferences. Scope the -> snapshot to always-needed core (relocation table ~5× smaller, correctness surface ~5× -> smaller); the un-snapshotted 79% costs ZERO at startup (lazy). A and B-scoped-to-core -> are COMPLEMENTARY (disjoint cost: core replay vs non-core replay). Risk: inherits B's -> relocation surface (smaller, fully build-time-controlled = best case) + `builtin_fn` -> (0xFFFF) payloads are linker-set native addresses needing a V8-style external-reference- -> by-index table (the most error-prone sub-part). Respects F-NNN *if* the external-ref -> table is correct; F-004 respected because base-relative is image-internal (relocated to -> absolute before any runtime read). -> -> **Alt 3 (wildcard): zero-copy in-place bytecode deserialize.** Attacks the 2.1 ms -> deserialize by making chunks directly executable from rodata (no per-chunk alloc/copy); -> vm.eval's 2.5 ms still runs (it legitimately must). Captures most of B's deserialize win -> with NO pointer relocation (rodata read-only, pointers born absolute-correct), no -> external-ref table, position-independent instruction stream (version-lock + interior- -> pointer hazards evaporate). Composes with A's laziness for free. Caps at ~6.3 ms alone / -> ~3.5–3.8 ms combined with A. Most F-NNN-safe option (no F-004/F-006 surface touched). -> Risk: serialize format becomes load-bearing/rigid; possible over-invest if the JIT -> replaces it. -> -> **Sequencing verdict.** A and B are complementary, NOT a smallest-diff convenience B -> unwinds — PROVIDED B is scoped to core. B-the-full-29-snapshot subsumes A but is -> wasteful; B-scoped-to-core is complementary and is the finished form. Start with A (it -> is independently the largest win, lowest risk, and a PREREQUISITE for clean B since B -> only makes sense once non-core is lazy). A-before-B is the correct dependency order, not -> smallest-diff bias. The smell to avoid: shipping A then declaring victory at ~4.8 ms -> without ever doing core's snapshot (Progress-pressure) — OR doing full B for a ~1 ms win -> against a 3.4 ms wall (prestige over measurement). -> -> **NaN-box relocation correctness.** "Derive the relocation table from the GC's precise -> pointer map" is NOT sound as stated: (1) the trace table yields referent VALUES, not -> mutable SLOT addresses — a separate per-tag relocate table is needed; (2) roots in -> Zig-side structs off the GC heap (`Var.root`/`.meta`/`.watches`, namespace metadata in -> `infra_alloc`) need relocation but are reached by the root walker, not the heap trace — -> the relocation set is strictly LARGER than the GC's heap trace; (3) `builtin_fn` (0xFFFF) -> payloads are 48-bit linker-set native pointers, not heap pointers — poison unless -> resolved via an external-reference-by-index table; (4) payload is `addr>>3` (8-byte -> aligned) — delta must be a multiple of 8 and mmap base ≥ 8-aligned (page alignment gives -> this, but it is an invariant the loader must ASSERT). A single missed pointer = silent -> heap corruption with no fault. Relocation is achievable but NOT by reusing the GC trace -> map. -> -> **Version-lock.** Largely a non-issue (snapshot embedded in the same binary that consumes -> it — a rebuild regenerates both halves). The real hazard is `cache_gen` (host build tool) -> vs runtime REPRESENTATION SKEW: a heap snapshot encodes in-memory layout, so if cache_gen -> is built for the host but the runtime ships for a different target (Wasm/x86_64/arm64) -> with different padding/pointer-width/endianness, the snapshot is wrong. Bytecode replay -> (A/Alt-3) is immune (target-neutral indices); heap snapshot (B/Alt-2) makes -> cache_gen-target == runtime-target a hard, multiplicative build-matrix invariant. -> -> **Is ~3.4 ms exec/page-fault the real ceiling?** Mostly yes — and it materially weakens -> full B. After A+D-140 ≈ 4.8 ms; after Alt-2 ≈ 3.5–3.8 ms. The residual is `fork/exec` + -> page faults on a multi-MB static binary, which no bootstrap-architecture change -> addresses. B's marginal win over A is ~1 ms (core's replay share), below `fork/exec` -> noise, bought with the entire relocation correctness surface + build-matrix skew. Get -> core's last millisecond via Alt-3's zero-copy deserialize, not heap relocation. -> -> **Co-design B with the moving GC?** Yes — and that is a strong argument to defer B now. -> A generational/nursery GC that moves objects must already solve "rewrite every pointer -> slot when an object moves" = the identical machinery B needs, and may adopt base-relative -> encoding that makes snapshot relocation FREE. Building B now against the non-moving GC = -> building it twice or a throwaway that base-relative encoding unwinds. A and Alt-3 are -> GC-independent and ship now. -> -> **Recommended sequencing:** (1) D-140 footer-seek (−1 ms, zero risk). (2) Lazy-namespace -> bytecode (−3.6 ms common case) + the build-time replay-clean gate + the F-011 side-effect -> audit → ~4.8 ms (meets sub-5 ms). (3) Zero-copy in-place deserialize (−2.1 ms, no pointer -> risk) → ~3.5–3.8 ms (sub-4 ms). (4) Defer heap-snapshot to the GC unit (forward debt). -> -> **The one thing that makes this a mistake:** chasing the ~1 ms B targets while the -> ~3.4 ms wall makes it invisible, paying with a silent-heap-corruption surface + a -> per-target build-matrix lock — driven by the survey's prestige ("SBCL/V8/GraalVM all -> snapshot, so we should") instead of the measurement, which says the impressive answer -> buys an imperceptible millisecond at the worst-testable risk. - -The main loop's choice WITHIN the F-NNN envelope (the subagent's recommendation is not -binding): adopted the recommended sequencing in full. The Decision diverges from the -subagent only in emphasis — it treats Alt-2 (core-scoped snapshot) as the same deferred-B -increment co-owned with the GC unit (step 4), rather than a near-term step, because its -relocation surface is the GC unit's to build. diff --git a/.dev/decisions/0163_lazy_namespace_bytecode_loading.md b/.dev/decisions/0163_lazy_namespace_bytecode_loading.md deleted file mode 100644 index 87a4e6102..000000000 --- a/.dev/decisions/0163_lazy_namespace_bytecode_loading.md +++ /dev/null @@ -1,256 +0,0 @@ -# ADR-0163 — Lazy-namespace bytecode loading: `loaded_libs`-keyed loader, eager set = JVM Clojure's no-`require` set (SSOT `bootstrap.EAGER_NS`), one position-independent blob - -> The Decision below first set "eager = clojure.core only"; **Amendment 1 (bottom) -> revised it to the measured clj no-`require` set** after an F-011 regression. The -> live SSOT is `bootstrap.EAGER_NS` + `bootstrap.isEagerNs`. Read Amendment 1 first. - -- **Status**: Proposed → Accepted (2026-06-24; D-516; ADR-0162 step 2 realization; DA-fork folded verbatim below) -- **Driven by**: ADR-0162 step 2 (lazy-namespace bytecode) + a Step-0 survey + Step-0.6 - + a DA red-team on the implementation design. The eager bootstrap replays - clojure.core + ~28 non-core stdlib namespaces every startup (~5ms; non-core = 79% of - 891 chunks); a `cljw -e 1` one-liner uses ~only core. This ADR pins HOW the non-core - set becomes lazy. -- **Relates to**: ADR-0162 (cold-start architecture — this is its step 2), ADR-0056 - (AOT bytecode envelope), ADR-0034 am4 (envelope manifest), D-516 (this unit), D-517 - (zero-copy in-place deserialize — step 3, which this format must serve). F-002 - (finished-form wins), F-011 (behavioural equivalence — clj requires explicit require), - F-013 (single binary, no side artifacts). User-declared invariant: no .clj→Zig - bootstrap rewrite (cljw-v0 rut) — this touches only the restore mechanism + eager set. - -## Context - -Today `buildBootstrapEnvelope` (builder.zig:204, the `for (files)` loop) compiles ALL -of `bootstrap.FILES` into ONE bytecode envelope eval'd in FILES order, replayed whole at -startup. The non-core 79% is replayed even when unused. Three facts from the survey + -Step 0.6 shape the design: - -1. **Minimal eager need = clojure.core only.** core.clj `(require ...)`s no non-core ns; - the default `-e`/REPL/print/error path touches only core. The one apparent exception — - `error_context.register` needs the `cljw.error` ns — interns `*error-context*` from - Zig via `findOrCreateNs` (it does NOT need cljw.error.clj loaded; the `with-context` - macro is user-facing). -2. **The loader's existing-ns guard is already subtly wrong.** `loadOrFindNs` - (loader.zig:110) short-circuits on `if (mappings.count() > 0) return existing`. A ns - that has interned vars but whose `.clj` body has not run (exactly the cljw.error case - after `register`, and the general `(intern 'foo 'x 1)` then `(require 'foo)` case) - is wrongly treated as "loaded". `loadNamespace` already maintains the correct signal, - `rt.loaded_libs` (loader.zig:38 contains / :70 put). -3. **Several bundled libs call non-core vars WITHOUT declaring `:require`** (verified: - clojure.data→clojure.set; clojure.test→clojure.string/clojure.walk; cljw.json→ - clojure.data.json/clojure.walk; cljw.fs→clojure.java.io; clojure.pprint→clojure.string). - The eager monolith masks this via FILES load order (the dep is always eval'd earlier). - Lazy loading exposes it as a runtime NameError when the lib is required without its dep. - Upstream Clojure declares these requires; cljw dropped them under eager-all. - -## Decision - -Adopt the DA-recommended shape (Alt B loader on Alt C format), in THREE sequenced commits. - -**1. Loader guard = `rt.loaded_libs`, not `mappings.count()`.** `loadOrFindNs`'s -existing-ns short-circuit keys off `rt.loaded_libs.contains(name)`. This fixes the latent -intern-then-require bug for ALL namespaces (not just the bootstrap instance), and makes -"ns exists (has Zig-interned vars) but .clj not yet replayed" an expressible state — so -**cljw.error becomes lazy too** and the **eager set shrinks to clojure.core ONLY**. The -bootstrap (setupCoreAot) must register clojure.core in `loaded_libs` so a user -`(require 'clojure.core)` is the correct no-op. The 4 `loadOrFindNs` callers -(namespace.zig require/use) are unchanged — only the guard inside flips. - -**2. Registry format = ONE contiguous position-independent blob + tail offset manifest.** -`[eager core region][lib₁ region]…[lib₂₇ region][offset manifest: name→(byte_offset,len)]`. -Each lib region is a self-contained mini-envelope (`EnvelopeIterator.init(blob[off..off+len])` -works verbatim). The manifest is a flat name→offset table at the tail. NOT 27 separate -`serializeEnvelope` outputs (which would hand D-517 twenty-seven separately-framed regions -to skip) and NOT 27 `@embedFile` sites — one embed, one position-independent rodata span, -exactly what D-517 (zero-copy in-place deserialize, the next step) wants. The regions carry -no absolute pointers (`EnvelopeIterator.next` already returns slices INTO the bytes), so the -blob is relocation-free. - -**3. Trigger = explicit `require` only** (NOT first-unqualified-var-reference auto-load). -The lazy path slots into `loadOrFindNs`: on a registry hit, `runEnvelope` the lib's region -+ `loaded_libs.put`; on miss, fall through to the source resolver (filesystem user libs). -This is a **clj-parity correction** — JVM Clojure requires explicit `require`; cljw's -eager-all was the divergence. - -**4. Missing-`:require` fix lands FIRST (commit 1), independently.** Add the missing -`:require`s to the lib heads (from each lib's ACTUAL non-core usage, matching upstream's -declare-your-deps). Verified safe for the eager monolith TODAY (every dep loads earlier in -FILES order, so the added require is an idempotent no-op) and clj-oracle-provable. It must -precede the split so a post-split failure is not ambiguous (require-fix bug vs split bug), -and because a require can in principle reorder the build-time chunk stream. - -**5. Isolated-replay build gate (the completeness proof).** cache_gen building each region -proves it replays clean against the FULL eager set (all files in FILES order) — which is -NOT what the lazy runtime does. So a hard build gate must, for EACH lazy ns, spin a fresh -runtime, replay the eager set (core only), `(require 'that-ns)`, and assert clean. This -catches a lib that builds (its dep was eval'd earlier at build time) but fails at lazy -runtime when required without its dep — i.e. it is the mechanical proof the missing-require -set is COMPLETE. Without it, an un-fixed missing-require regresses only for users who -require in the "wrong" order — the worst, hardest-to-catch failure class. - -**Sequencing (three commits):** -1. Missing-`:require` fix alone — verified against eager monolith + clj oracle + corpus. -2. Loader guard rewrite (`loaded_libs`-keyed) + the isolated-replay build gate — still - eager-all, a pure refactor verifiable against current behaviour. -3. The lazy split (format + trim eager set to core) + **blast-radius triage**: run the full - corpus/e2e against a lazy build, enumerate every newly-failing bare-non-core-var case, - and disposition each (test relied on the divergence → add the require; or genuine). - -## Consequences - -- **Eager set = clojure.core only** → the full non-core replay (~3.6ms) leaves the common - cold path; floor → ~4.8ms (with D-140). Discharges the eager-vs-size standing tension. -- **A real loader bug is fixed** (intern-then-require), not just D-516's symptom — F-011 net - positive beyond startup. -- **cljw moves TOWARD clj parity** (explicit require). The blast-radius triage (commit 3) is - a one-time correction cost; each newly-failing case was relying on the eager-all divergence. -- **The blob is D-517-ready** — one position-independent rodata span, so step 3 (zero-copy) - drops on without re-cutting the format. -- **A new hard build gate** (isolated per-lib replay) becomes the standing guarantee that the - lazy set stays self-contained as libs evolve. -- **Risk owned**: the #1 subtle-regression vector is an incompletely-fixed missing-`:require` - set masked by FILES order; the isolated-replay gate is its sole defense and is therefore - mandatory, not optional. - -## Alternatives considered - -The DA red-team output (fresh-context subagent, grounded in `loader.zig` / `builder.zig` / -`serialize.zig` + the debt rows) is reproduced verbatim; its recommendation drove the -Decision (it inverted the proposed Option-C: eager-cljw.error → a smallest-diff dodge of a -real loader bug; 27-separate-envelopes → a cost pushed into D-517). - -> **Alt A — smallest-diff: ONE envelope, per-ns chunk-range index (loader seeks).** Keep the -> single monolith; record per-ns `[start,end)` chunk ranges in a manifest; replay only -> core's range at startup; `loadOrFindNs` seeks the iterator to the ns's range. Better: -> smallest serialize change, one contiguous blob (D-517-friendly), staged migration possible. -> Breaks: `EnvelopeIterator.next` is sequential (skip is O(N), needs a `skip(n)`); a single -> envelope weakens isolation (a mis-ordered range still rides whatever ran); does NOT force -> the missing-`:require` fix (chunks for clojure.set still physically precede clojure.data, so -> range-replay MIGHT find it interned if the user required it first — load-order-dependent -> non-determinism). F-011 weaker (does not reliably expose the missing-require bug). F-002 -> disfavours. -> -> **Alt B — finished-form-clean: N self-contained envelopes + explicit per-ns `loaded` flag -> (fixes the guard bug).** Option C's split PLUS replace `mappings.count()>0` with -> `rt.loaded_libs` membership (which `loadNamespace` already maintains). With a real -> loaded-flag, cljw.error can be lazy: `register` interns `*error-context*` (mappings>0) but -> does NOT add cljw.error to `loaded_libs`, so the flag correctly says "ns exists, .clj not -> replayed" → `with-context` loads on first require. Better: fixes a REAL loader bug (the -> guard is already wrong for intern-then-require), shrinks the eager set to pure clojure.core, -> deletes the 27-vs-28 special case. Risk: higher blast radius (every loadOrFindNs caller now -> keys off loaded_libs); must verify loadCore populates loaded_libs for core; confirm the -> Zig-interned cljw.error ns's initial flag is "not loaded". F-011 strongest; F-002 prefers it -> despite the larger diff (Cycle-budget-defer smell says diff size is not a reason to pick A). -> -> **Alt C — wildcard: single concatenated blob + offset manifest, designed for D-517 from day -> one.** One `@embedFile`'d blob `[core region][lib regions…][tail offset manifest -> name→(offset,len)]`; each region a self-contained mini-envelope; laid out -> position-independent (chunk-relative offsets, no absolute pointers — serialize already does -> this) so D-517's zero-copy reader drops on with no relocation. Better: co-designs the format -> with D-517 (which ADR-0162 sequences next and requires composing) instead of letting D-517 -> re-cut 27 separately-framed regions; fewer @embedFile sites than Option C. Risk: new -> serialize format (`writeOffsetManifest`/`readOffsetManifest`) — the one thing Option C -> avoided; speculative if D-517 is descoped (Reservation-as-bias). F-013 cleaner (one embed); -> F-002 defensible as finished-form since D-517 IS committed. -> -> **Registry format pressure-test.** Option C (27 separate `serializeEnvelope` outputs) makes -> D-517 HARDER: D-517 wants one position-independent rodata span with no per-chunk alloc/copy, -> but 27 separately-serialized envelopes each carry their own component-table + manifest -> framing that D-517's zero-copy reader must locate and skip 27 times, plus a separate name -> index. Finished-form-clean = ONE contiguous blob + offset manifest (Alt C), or at minimum -> Alt A's single-envelope-with-ranges. Option C's "reuse serializeEnvelope verbatim 27 times" -> is a build-time convenience that pushes cost into the D-517 cycle — the -> smallest-diff-bias-against-finished-form pattern. **Recommendation: one blob (regions) + -> offset manifest, not 27 separate envelopes.** -> -> **The `mappings.count()>0` guard.** Keeping cljw.error eager is a smallest-diff DODGE of a -> real loader bug. The guard is already incorrect for "user interns a var then requires the -> ns" — no current test hits it, but lazy loading makes the guard load-bearing for -> correctness. The clean fix is the explicit flag, and `loadNamespace` ALREADY maintains -> `rt.loaded_libs` (contains-check + put). Keying `loadOrFindNs` off `loaded_libs` (i) lets -> cljw.error be lazy (eager set = pure clojure.core), (ii) fixes the latent intern-then-require -> bug, (iii) removes the 27-vs-28 special-case. "Not worth untangling for a 27-line file" -> inverts the cost: the untangle IS the cheap correct fix; the dodge leaves a bug behind. -> -> **Missing-`:require` fix — land FIRST, unambiguously.** It is the finished form (matches -> upstream; F-011 IS clj-parity which requires explicit require) AND a correctness change -> provable against the clj oracle independently of laziness. The load-bearing risk: adding a -> require CAN change load order in the eager monolith TODAY — a newly-added head `:require` -> triggers a nested require during build-time eval that pushes the dep's chunks earlier in the -> post-order chunk stream; idempotent if the dep was already eval'd, but a reorder if FILES had -> the user before the dep. So validate it BEFORE the lazy split confounds it. Commit 1 = add -> all missing requires, green against eager monolith + oracle + corpus; commit 2+ = the split. -> Bundling makes a post-split failure ambiguous. -> -> **Trigger = explicit require only — clj-parity-correct, yes.** JVM Clojure requires explicit -> require for every non-clojure.core ns; eager-all is the CURRENT cljw divergence. Blast radius -> (MUST quantify before shipping): every existing e2e/corpus/test using a non-core var without -> requiring it newly fails (suspects: bare clojure.string/set/walk/test). Mechanical sweep: -> grep e2e/corpus for non-core qualified vars + unqualified-via-eager-refer uses; run full -> corpus+e2e against a lazy build and COUNT failures; each is a test bug (add the require) or -> genuine. Acceptable as a parity correction ONLY with the failure list triaged, not a blind -> flip. -> -> **Build-time replay-clean gate — NOT redundant.** cache_gen building each envelope proves it -> replays clean against the FULL eager set (all files in FILES order). The lazy runtime replays -> against the TRIMMED eager set. A lib that builds because clojure.set was eval'd before -> clojure.data in FILES order FAILS at lazy runtime if clojure.data is required without -> clojure.set — and the build gate won't catch it (clojure.set WAS there at build). So the gate -> must replay each lazy lib IN ISOLATION (fresh runtime, eager-only, require the one lib, assert -> clean). It is the mechanical proof the require-fix is complete. -> -> **Recommended shape + sequencing.** Adopt Alt B's loader (`loaded_libs`-keyed guard, eager = -> clojure.core only, cljw.error lazy) on Alt C's format (one contiguous position-independent -> blob + tail offset manifest, D-517-ready). Reject Option C's 27-separate-envelopes + -> eager-cljw.error (both smallest-diff dodges). Commits: (1) missing-`:require` fix alone, -> oracle-verified; (2) loader guard rewrite + isolated-replay build gate, eager-all pure -> refactor; (3) lazy split + blast-radius triage. -> -> **The ONE thing most likely to ship a subtle regression:** an incompletely-fixed -> missing-`:require` set masked by FILES eval order — a lazy ns that builds clean (dep eager at -> build) but is required at runtime before its dep — works for the common order, fails for an -> uncommon one, invisible in build or a require-order-lucky test. Sole defense: the isolated -> per-lib replay gate for ALL 27 libs as a hard build gate. - -Main-loop choice WITHIN the F-NNN envelope (the subagent's recommendation is not binding): -adopted in full — the DA's reasoning is F-002/F-011-aligned and the larger diff (guard -rewrite + new format) is the finished form, so the Cycle-budget-defer smell forbids picking -the smaller Option C. - -## Amendment 1 (2026-06-24) — the eager set = JVM Clojure's no-`require` set (F-011) - -The Decision above said "eager set = clojure.core ONLY". Implementation surfaced -an F-011 regression (user-flagged): JVM Clojure makes several non-core namespaces -usable WITHOUT an explicit `require`, and core-only-eager broke them -(`(clojure.string/upper-case "hi")` → Name error in cljw, "HI" in clj). The user -directive: *match clj — what clj bundles without require is by usage frequency, not -logic; align cljw to it*, and keep the eager/lazy split a **single SSOT**. - -**Measured (2026-06-24, fresh `clj`, `(some? (find-ns 'X))` with no require):** - -| auto-loaded in clj (TRUE) → cljw EAGER | NOT auto-loaded (FALSE) → cljw LAZY (require needed) | -|-----------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| clojure.core, clojure.string, clojure.walk, clojure.edn, clojure.java.io, clojure.core.protocols, clojure.uuid, clojure.instant, clojure.spec.alpha | clojure.set, clojure.zip, clojure.data.json, clojure.data.csv, clojure.tools.cli, clojure.pprint, clojure.test, clojure.data, clojure.math, clojure.template, clojure.stacktrace, clojure.datafy, clojure.repl (+ cljw.* originals) | - -**SSOT** = `bootstrap.EAGER_NS` (a `StaticStringMap`) + the single predicate -`bootstrap.isEagerNs`. `loadCoreAot` runs ONLY `isEagerNs` regions (in FILES order); -every other ns is a lazy region replayed on first `require`. spec.alpha's own deps -(spec.gen.alpha / core.specs.alpha) load transitively when its eager region runs. -This is STRICT clj-parity: clj's `clojure.set` is NOT auto-loaded either, so cljw -requiring it matches (verified: `set` no-require → Name error in BOTH). - -**Cold-start cost of parity**: floor 8.6ms (all-29 eager) → 4.6ms (eager = the 9-ns -auto-set) vs 3.1ms (core-only, which violated parity). The +1.5ms over core-only -buys exact clj-parity — accepted per F-011/F-002 over a parity-violating 3.1ms. - -**Unwind caution (user-flagged)**: eager-all was partly a D-452 optimization -(avoid re-parse). The lazy regions stay BYTECODE (no re-parse on lazy load either), -so that optimization is preserved; only the *eager replay* of unused libs is removed. -A second require path bug surfaced + was fixed in the same arc: `evalRequire` -(tree_walk) + `op_require` / `op_require_with_libspec` (vm) each carried an inline -copy of the load logic with the `mappings.count() > 0` short-circuit (which mis-fires -when `registerAll` pre-interned a ns's private leaves — clojure.string's `-upper-case`); -all three now route through the single `loader.loadOrFindNs`. And `loadOrFindNs` keeps -the `mappings.count() > 0` check as a LAST resort (after loaded_libs + region + source -all miss) so an INLINE-defined ns — `(ns mylib …)` then `(require '[mylib])`, no region -or file — is still found-as-is rather than raising lib_not_found (a real regression the -core-only flip introduced, caught by phase14_macro_shadow_builtin). diff --git a/.dev/decisions/0164_eval_auto_collect.md b/.dev/decisions/0164_eval_auto_collect.md deleted file mode 100644 index 84f34041f..000000000 --- a/.dev/decisions/0164_eval_auto_collect.md +++ /dev/null @@ -1,99 +0,0 @@ -# ADR-0164 — Threshold-driven auto-collect during eval (BOTH alloc-boundary + VM back-edge), within mark-sweep - -- **Status**: Proposed → Accepted → **IMPLEMENTED + GO-PASSED** (2026-06-24; D-450 gc_alloc_rate root cause; DA-fork folded verbatim below). The BOTH-sites auto-collect landed at 4MB default + the `CLJW_GC_THRESHOLD_MB` knob; the load-robust OFF-vs-ON A/B (interleaved hyperfine, 20 runs) passed decisively — see **§ GO result** below. The DA's predicted "partial win" was too conservative: ON is FASTER than OFF on all 8 measured benches (1.12–1.65×), the `string_ops` canary INCLUDED (1.28× faster, not slower). -- **Driven by**: D-450's surviving fastest-script gap `gc_alloc_rate` (cljw 45.3ms / bb 39.9ms = 1.14×). Root-caused load-independently via the new `CLJW_GC_STATS=1` counter (committed 07176327): a tight allocating loop shows **reuse=0%, collects=0** — cljw runs **NO threshold-driven auto-collect during eval**, so it mallocs unboundedly (1.28GB for 4M short-lived vectors) and never reuses the free pool, while Babashka wins via JVM TLAB-bump + young-gen collect-and-reuse. This is BOTH the gap AND a latent **memory bug** (default `heap_ceiling=null` → no cap + no auto-collect → a real allocating loop OS-OOMs). -- **Relates to**: F-006 (GC = mark-sweep + 3-layer allocator; a moving/generational/nursery GC is a SEPARATE deferred unit — D-518 / the moving-GC unit), F-004 (NaN-box absolute pointers — non-moving collect), F-011 (GC timing not observable), F-002 (finished-form wins). ADR-0028 (mark-sweep + free-pool + adaptive threshold), ADR-0090 (STW safe-points), D-352/D-361 (the `heap_ceiling`/`checkInfraCap` alloc-boundary memory check this complements), the `CLJW_GC_TORTURE_ALLOC` validation path (the proven template). - -## Context - -`gc_heap.zig::alloc` already carries, at the alloc boundary (roots quiescent — the new object does not exist yet): a worker safe-point park, the `CLJW_GC_TORTURE_ALLOC` collect path (gated on a torture period), the `heap_ceiling` REFUSE check (D-352/D-361 — placed here precisely because a Zig primitive can allocate megabytes without crossing a VM back-edge), and `bytes_since_last_gc += size`. The adaptive threshold (`threshold_bytes = max(1MB, last_live*2)`, reset to 0 by each collect, mark_sweep.zig:171-175) exists but **nothing checks `bytes_since_last_gc > threshold_bytes` during normal eval** — only an explicit collect, torture, or a peer worker's STW request runs the GC. The VM back-edge poll (`vm.zig:305`, run once per bytecode instruction via `stepOnce`) checks only `safepoint.gc_requested` (a peer's request), not the heap threshold. - -## Decision - -Add threshold-driven auto-collect at **BOTH** sites (the DA's Alt 2 — finished-form), within F-006 mark-sweep (non-moving, no nursery): - -1. **Alloc-boundary** (`gc_heap.zig::alloc`, beside the existing `heap_ceiling` check): when `bytes_since_last_gc > threshold_bytes` AND `fabrication_depth == 0` AND not a registered worker / in-torture AND a live `active_env`, run `collectStopTheWorld` — the `CLJW_GC_TORTURE_ALLOC` path VERBATIM, threshold-gated instead of period-gated. This is the **completeness floor**: it catches bulk-primitive allocation (`(vec (range 1e6))`) that never crosses a VM back-edge — the SAME rationale (and SAME boundary) as the `heap_ceiling`/`checkInfraCap` memory check it complements (its *collecting* twin). - -2. **VM back-edge poll** (`vm.zig:305`, beside the existing `gc_requested` poll): the same threshold check. This is the **cheap tight-loop path**: a `recur` loop back-edges every iteration but allocs once, so the common case pays the threshold compare on the already-paid poll branch (cleaner safe-point: `sp`/`op_top` settled, `fabrication_depth==0` by construction). The shared `bytes_since_last_gc` reset (collect sets it to 0) keeps the two sites idempotent — no double-collect. - -3. **Raise `default_gc_threshold_bytes` 1MB → 4MB** + add a `CLJW_GC_THRESHOLD_MB` env knob. For a churn workload `last_live≈0` so the adaptive `max(1MB, last_live*2)` arm is inert → the initial threshold IS the operative number; 4MB gives ~16 collects vs ~64 (same total sweep work, fewer STW fences) at negligible memory cost. The adaptive arm is unchanged (correct for growth workloads). - -4. **Keep `CLJW_GC_TORTURE_ALLOC` in the gate.** Threshold-driven is a WEAKER completeness probe (the 1MB/4MB cadence can step over a 2-alloc-unrooted window for years; torture fires frequently enough to catch it). Threshold ships ALONGSIDE torture, never instead. - -5. **Implementation + GO is a SEPARATE next unit needing a QUIET Mac.** The change alters WHEN the GC fires for EVERY allocating program. Correctness is load-independently validatable (diff oracle + `CLJW_GC_TORTURE` + `CLJW_GC_STATS`: collects>0 / reuse>0 / bytes bounded). But the KEEP/REVERT decision (experiment-and-revert) is **wall-clock-gated**: a now-firing mid-bench collect could REGRESS one of the 7 fastest-script benches the cold-start arc just won — **`string_ops` is the canary** (continuous string alloc → crosses the threshold → new STW cost). A count proxy (collects>0) is INSUFFICIENT (it hides the wall-cost). GO requires a before/after wall-clock re-run of all 7 won benches on a ReleaseSafe binary (a quiet Mac; the won/lost margin is ms); raise the threshold until every won bench holds. **Do not commit the implementation until that wall-clock GO passes.** - -## Consequences - -- Fixes the latent **unbounded-alloc memory bug** decisively (any allocating loop now collects at the threshold instead of growing until OS-OOM) — the more important half. -- **Partial** `gc_alloc_rate` perf win, NOT full closure. Honest framing (DA PT5): collect+reuse trades malloc-storm for pool-reuse + per-threshold mark+sweep; it narrows but likely does not fully close 1.14× — bb's edge is bump-allocation (no malloc, just pointer-increment), which is the DEFERRED nursery unit (Alt 3, out of F-006 scope). The last slice of the gap awaits that unit. **(Measured — see § GO result: the wall-clock win was LARGER than this prediction; the OFF-vs-ON A/B shows ON beat OFF 1.39× on gc_alloc_rate itself, 49→35ms, because collect+reuse also wins cache/page-fault locality. Whether cljw fully beats bb's bump-allocator still awaits a quiet-Mac peer re-rank + possibly the nursery.)** -- A global GC-timing change: re-validated by the wall-clock all-bench gate (§5) before it ships. -- `gc_large_heap` (already a cljw win) and any >threshold-retention workload may shift; the wall-clock gate covers them. - -## GO result (2026-06-24) — KEEP, decisively - -Implemented as a shared `GcHeap.maybeAutoCollect()` (the torture-alloc collect verbatim, -gated on `bytes_since_last_gc > threshold_bytes`) called from BOTH sites: the `alloc` -boundary (after the torture block, before `lockMutex`) and the VM back-edge poll -(`vm.zig`, after the torture poll). Default floor 1MB→4MB via a new per-heap -`threshold_floor_bytes` field (so `CLJW_GC_THRESHOLD_MB` survives each collect's recompute); -torture kept in the gate. - -**Load-independent validation (conclusive):** -- Diff oracle (`zig build test -Dwasm`, ReleaseSafe) green, no failures. -- `CLJW_GC_STATS`: 200K vectors → collects 0→**15**, reuse 0%→**93%**, mallocs 401K→**27,335**; - 4M vectors → collects **305**, reuse **99%**, mallocs **27,335 (FLAT)** = memory bounded - regardless of loop size (was 1.28 GB unbounded — the latent OS-OOM bug, fixed decisively). -- Rooting: the 15/305 auto-collects fire MID-EVAL yet the sums are mathematically exact - (`20000300000` / `8000006000000`); torture-alloc on the bench + `(+ 1 1)` clean. - -**Wall-clock GO — the load-robust OFF-vs-ON knob A/B** (same ReleaseSafe binary, -`CLJW_GC_THRESHOLD_MB=1000000` = auto-collect off ≈ pre-D-519, vs default 4MB = on; -interleaved hyperfine 20 runs/5 warmup, so transient load hits both equally): - -| bench | OFF (no auto-collect) | ON (4MB) | ON faster | -|---------------------|-----------------------|-------------|------------| -| gc_alloc_rate | 49.2 ± 4.3 ms | 35.4 ± 0.6 | **1.39×** | -| string_ops (canary) | 27.3 ± 5.5 | 21.3 ± 0.3 | **1.28×** | -| sieve | 21.9 ± 4.7 | 17.4 ± 0.6 | 1.26× | -| map_filter_reduce | 16.2 ± 6.6 | 9.8 ± 0.3 | 1.65× | -| gc_large_heap | 35.9 ± 1.2 | 31.9 ± 0.5 | 1.12× | -| destructure | 57.2 ± 7.1 | 50.5 ± 1.4 | 1.13× | -| nested_update | 14.7 ± 1.2 | 10.9 ± 0.2 | 1.35× | -| bigint_factorial | 21.9 ± 1.1 | 17.8 ± 0.4 | 1.23× | - -ON is faster on EVERY bench AND ~10× lower variance (σ 0.3–1.4 vs 4–7). The mechanism the -DA under-weighted (PT5): collect-and-reuse keeps the working set small → far fewer page -faults / malloc syscalls / cache misses, which OUTWEIGHS the mid-bench STW cost — so the -"the single most likely regression" (string_ops, PT2) instead IMPROVED. No threshold -tuning was needed; 4MB holds across the board. The separate **absolute peer re-rank** (does -cljw beat bb on the 9) is a campaign-standing question needing a genuinely quiet Mac (a -cross-run peer measure at load ~4 was contamination-noisy) — NOT a D-519 blocker, since the -A/B proves D-519 strictly improves cljw's own numbers. - -## Alternatives considered - -The DA red-team output (fresh-context subagent, grounded in `gc_heap.zig` / `vm.zig` / `mark_sweep.zig` / `vector.zig` / `root_set.zig`) is reproduced verbatim. Its verdict drove the Decision (BOTH sites over the proposed alloc-only; the wall-clock GO gate; the honest partial-win framing; keep-torture). - -> **Code grounding.** `alloc` (gc_heap.zig:345-423): park → `torture:` block (the template) → gc_mutex → heap_ceiling check → free-pool pop / rawAlloc → record + `bytes_since_last_gc +=`. `fabrication_depth>0`, `is_registered_worker || in_alloc_torture`, `active_env orelse break` guards all already present. `vm.zig:297-324`: `while(true)` runs ONE `stepOnce` per iteration (line 326); the back-edge poll (`gc_requested`→park, line 305) + the torture safe-point collect (322-324) are at the loop TOP → once per bytecode op, including right after `op_recur_loop` jumps. `op_vector_literal` (1298-1311) → `fromSlice`, elements rooted on the operand stack (op_top), bracketed by `enterFabrication`/`exitFabrication` (18 fabrication sites). Adaptive threshold (mark_sweep.zig:171-175): `max(1MB, last_live*2)`, `bytes_since_last_gc=0`. -> -> **Alt 1 — smallest-diff: back-edge-poll-only.** Don't touch `alloc`; at vm.zig:305 add `if (bytes_since_last_gc > threshold_bytes) collectStopTheWorld(...)` beside the `gc_requested` poll. Better: maximally clean safe-point (sp/op_top settled, fabrication_depth 0 by construction); zero `alloc` hot-path change → no won-bench alloc regression; rides infra the torture campaign already validated at vm.zig:322. **Fatal gap**: a Zig-primitive bulk alloc that never crosses a back-edge (`(vec (range 1e6))` = one op, `fromSlice` does N×alloc internally) is never collected → re-opens the D-361 OS-OOM hole `checkInfraCap` exists to close; also does nothing for TreeWalk (no stepOnce loop). F-compliant but under-delivers the memory-bug half. -> -> **Alt 2 — finished-form: BOTH (alloc-boundary + back-edge).** The proposal PLUS Alt 1, clean division: alloc-boundary = the completeness floor for bulk primitives (same boundary + rationale as the existing heap_ceiling at gc_heap.zig:387-391); back-edge = the cheap tight-loop path (recur back-edges every iter, allocs once → triggers here first, cheaper). Shared `bytes_since_last_gc` reset → idempotent. Better: the ONLY shape closing BOTH halves — the tight-loop gap AND the bulk-primitive OOM — mirroring the existing heap_ceiling/checkInfraCap two-site pattern (finished-form-consistent); robust to backend (TreeWalk has no back-edge, alloc-boundary still fires). Risk: largest diff (two sites); double-collect risk mitigated by the shared reset (needs a test: one recur iter that both back-edges + allocs doesn't collect twice); the alloc-boundary mid-op state audit is the SAME the proposal needs, +no new surface (the back-edge site is strictly safer). F-002: do NOT pick the smaller diff (Alt 1) when the larger is the complete form — Cycle-budget-defer smell. -> -> **Alt 3 — wildcard: free-pool-prewarm + bump arena (attack reuse=0%, not collect cadence).** A size-segregated bump arena for the dominant small-vector size class, auto-collect only as a high-threshold backstop. Better: directly attacks the measured reuse=0% + is the structurally-faithful answer to "why is bb faster" (TLAB bump, not collect-frequency). **Borderline-VIOLATES F-006**: introduces a nursery/bump tier F-006 reserves for the separate deferred moving-GC unit; a bump arena that never individually frees needs moving compaction (F-004 violation) or a mark-sweep over the arena (square one). Leading would-violate-F-NNN finding: the genuinely-fastest path (bump allocation) needs the deferred GC work — OUT OF SCOPE for an in-mark-sweep fix; the loop should NOT take Alt 3 now. -> -> **PT1 — alloc vs back-edge vs BOTH.** The back-edge poll EXISTS in the hottest path (vm.zig:305, once per op) but checks only `gc_requested` → a threshold check there is 1 line. BOTH is forced by the existing heap_ceiling/checkInfraCap split: the codebase ALREADY decided memory-bounding lives at the alloc boundary because bulk primitives skip back-edges; auto-collect is its collecting twin → same boundary. But the tight-recur bench back-edges every iter → the back-edge poll catches it first + cheaper. Alloc-only (proposal) pays a per-alloc compare + fires mid-op; back-edge-only re-opens bulk-primitive OOM. **Verdict: proposal (alloc-only) is 2nd-best; Alt 2 (BOTH) is finished-form. Do not ship alloc-only citing diff budget.** -> -> **PT2 — global-effect risk + proving no regression.** `alloc` is NOT entered in a non-allocating loop (integers are NaN-box immediates) → zero cost there; the per-alloc branch (where alloc happens) is the same shape as the existing heap_ceiling null-check (negligible). The REAL risk is the now-firing COLLECTS: `string_ops` allocates strings continuously → crosses 1MB → auto-collect fires mid-bench → STW mark+sweep wall-time the won measurement never paid = **the single most likely regression**. `nested_update`/`map_filter_reduce` similar; `gc_large_heap` may improve (bounded) or regress (collect overhead). **A count proxy (collects>0) is NOT sufficient** — it shows GC fired, not the wall-cost; the margin is ms (45.3 vs 39.9). **Wall-clock re-run of all 7 benches via `bench/compare_langs.sh`, before/after, ReleaseSafe (the Debug cliff masks the signal), is MANDATORY.** Count is a secondary localizer (a bench newly showing collects>0 is the one to scrutinize). Acceptance: no won bench regresses past the loss line; if string_ops regresses, raise the threshold (PT4). -> -> **PT3 — correctness of mid-`alloc` collect at threshold cadence.** Threshold fires rarely (~1MB) vs torture's every-N → exercises FEWER distinct mid-op states but the SAME KIND (collect at the alloc boundary, before the new object exists) — a strict subset of what torture drives. The proposal's "structure is proven" is correct (same guard set: fabrication_depth, gc_self_guard, active_env, operand-stack/op_top root walk). Residual risk: a NEW Zig primitive that calls `gc.alloc` twice with an unrooted intermediate between, forgetting BOTH brackets — torture catches it (frequent), threshold MIGHT NOT (1MB cadence steps over the 2-alloc window for years). **So threshold is a WEAKER completeness probe → must NOT replace the torture gate; keep `CLJW_GC_TORTURE_ALLOC`.** The bench itself is SAFE (4 immediate-int elements, fromSlice brackets fabrication, rooted on operand stack). -> -> **PT4 — threshold tuning.** 1MB / 64MB-bench = ~64 collects, each O(allocs-since) sweep over the grown `gc.allocations` (~64K records/collect → ~4M sweep iters total) — real but bounded, replaces unbounded malloc. 4MB = 16 collects, same total sweep work, fewer mark-phase fixed costs + STW fences, +3MB peak (trivial) → strictly better on collect-overhead. The adaptive `last_live*2` is INERT for churn (last_live≈0 → stays 1MB); it only engages on growing-retention. **Raise the floor to 4MB (or a `CLJW_GC_THRESHOLD_MB` knob); leave the adaptive arm.** -> -> **PT5 — is auto-collect even the right lever? (honest verdict).** Post-collect, `alloc` pops the free pool (cheaper than malloc but not free; keyed on exact size class — holds for the 1-size-class bench). Net = trade N mallocs for N pool-pops + (N/threshold) collects; positive only if (malloc−poolpop)×N > collect×(N/threshold), tilted positive by a larger threshold. **Honest verdict: likely bounds memory + MODESTLY improves alloc rate, but will NOT reliably close the full 1.14× (need ~12% off).** bb's edge is bump-allocation (Alt 3 / deferred nursery). **Do not over-promise: frame as "bounds memory (the real bug) + partial perf win; full closure awaits the nursery."** -> -> **Recommended shape + sequencing.** (1) Ship Alt 2 (BOTH), not alloc-only, not back-edge-only. (2) Raise default threshold to 4MB + `CLJW_GC_THRESHOLD_MB` knob; keep adaptive arm. (3) Keep `CLJW_GC_TORTURE_ALLOC` in the gate. (4) Gate on wall-clock re-run of all 7 won benches (ReleaseSafe, before/after); `string_ops` is the canary; `CLJW_GC_STATS` reuse%/collects as secondary signals. (5) Frame the perf claim honestly (memory bug fixed + partial win; full closure = deferred nursery). -> -> **The ONE thing most likely to make this a mistake.** A won bench regresses on WALL-CLOCK because auto-collect now fires mid-bench and the STW mark+sweep exceeds the malloc savings — HIDDEN by a count proxy (collects=2-3 looks innocuous). `string_ops` is the danger. If validated by collect-count instead of wall-clock, this ships green and silently regresses a bench the cold-start arc just won. **Mitigation non-negotiable: wall-clock re-run, ReleaseSafe, raise the threshold until every won bench holds.** The secondary danger (a future 2-alloc primitive forgetting its fabrication bracket, stepped over by the 1MB cadence — PT3) is real but deferred-detectable; the bench regression is immediate + more probable. - -Main-loop choice WITHIN the F-NNN envelope (the subagent's recommendation is not binding): adopted Alt 2 (BOTH) + the 4MB threshold + keep-torture + the honest framing + the mandatory wall-clock GO gate in full. Alt 3 (bump nursery) is recorded as the would-violate-F-006 finding and correctly deferred to the moving-GC unit. The implementation + the wall-clock GO is a separate unit requiring a quiet Mac (Decision §5) — NOT committed under host load on a count proxy (the DA's #1 mistake). diff --git a/.dev/decisions/0165_collection_perf_strategy.md b/.dev/decisions/0165_collection_perf_strategy.md deleted file mode 100644 index 33bb68fcc..000000000 --- a/.dev/decisions/0165_collection_perf_strategy.md +++ /dev/null @@ -1,234 +0,0 @@ -# ADR-0165 — Collection-perf strategy: keep best-of-breed algorithms, win on Zig-native layout (measured-loss-led; see Amendment 1) - -- **Status**: Accepted as the **direction** for §9.2.S collection work (2026-06-24). The - per-lever activation is the TDD-loop's job; this ADR is the durable research synthesis + - ROI ordering + the experimentation/regression protocol. Driven by the 2026-06-24 clean - peer re-rank (ADR-0148 follow-up): cljw LOSES bb 1.02–1.16× on the collection-heavy - benches (sieve/destructure/gc_*/bigint) but WINS pure-compute 2× (fib_loop/tak) + ratio_sum - — so the gap is the COLLECTION/LIBRARY layer, not the VM engine. -- **Amendment 1 (2026-06-24, measurement-driven lever re-order)**: implementation Step 0.6 - re-laying refuted the "L1 transients FIRST / transients are a STUB" premise. See - § "Amendment 1" below — the lever ordering is now MEASURED-loss-led, the L1 stub claim is - corrected, and the keyword-map-get fast path (O-051) landed as the first PROVEN lever. -- **Relates to**: ADR-0148 (fastest-script campaign — this is its collection sub-strategy), - F-002 (finished-form), F-005 (numeric tower), F-006 (GC; GC-backed sharing avoids per-node - refcount), F-011 (behavioural equivalence — the diff oracle is the safety net). D-520 (the - draining debt row). The JIT (D-386 / ADR-0200 zwasm JIT) is the SEPARATE, larger compute - frontier (JVM-Clojure territory; babashka structurally cannot JIT — SCI is not a Truffle - interpreter, GraalVM native-image is AOT-only). - -## Context — the measured truth - -Clean **interleaved cljw-vs-bb** A/B (load-robust; the cross-run compare_langs table at -load ~9 was contaminated and over-claimed "7/9 won"): - -| class | benches | verdict | -|---------------------------------|----------------------------------------------------------------------------|-----------------------------| -| pure compute (interpreter loop) | fib_loop 2.0× · tak 2.2× · arith_loop 1.33× | **cljw WINS** | -| exact rational | ratio_sum 1.42× | **cljw WINS** | -| collection / library | sieve · destructure · gc_alloc_rate · gc_large_heap · bigint_factorial | **cljw loses 1.02–1.16×** | - -bb = SCI (tree-walk interpreter) + clojure.lang + java.math, all GraalVM-native-AOT, **no -JIT**. So bb and cljw are non-JIT-interpreter PEERS; cljw's bytecode VM actually BEATS SCI on -pure compute. cljw loses only where clojure.lang's 15-year-tuned data structures + JDK -BigInteger beat cljw's younger implementations. → the lever is collection implementation. - -## Decision — the unifying strategy (the 3 routes converge as layers) - -1. **Algorithm layer — keep best-of-breed; cljw ALREADY chose right.** Direct code read - confirms: real **CHAMP** map/set (`data_map`+`node_map` two bitmaps + `@popCount`), - radix-vector+32-tail, array-map≤8 linear, `std.math.big` (Karatsuba) + i48-immediate - fast-path. No data-structure rewrite. **RRB-trees REJECTED** (size-table taxes the common - indexed path; Clojure keeps it out of core — opt-in only for concat-heavy code). -2. **Implementation/layout layer — diverge from Java HERE; this is where Zig structurally - wins.** Every measured Immer/im-rs win over Clojure/Scala (1.3–6.7× iteration, 3–25× - equality) comes from **no object headers + tight cache layout**, which Zig gives by - construction; and cljw's **GC-backed** trie avoids im-rs's per-node atomic `Arc`. So the - "different path from Java" is at LAYOUT/memory, not algorithm. - -## ROI-ordered levers (all implementation/constant-factor; map to the losing benches) - -**Tier 1 (do first — biggest, lowest risk, helps real code beyond benches):** -- **L1 — Transients (single-threaded owner-token) + bulk build paths.** cljw's transients - are a STUB today (vector.zig / map.zig). #1 in both clojure.lang and the industry survey. - `into`/`vec`/`mapv`/`reduce conj` dominate real scripts. Zig win: owner-token = POINTER - compare (cheaper than JVM `AtomicReference`); MemoryPool per node size-class. Benches: - gc_alloc_rate/gc_large_heap/nested_update. **Highest ROI; the recommended first move.** -- **L2 — Chunked seqs on the WALK path.** cljw chunks + fuses only the `reduce` path; - `map`/`filter` realize one Cons per element via a thunk Fn call on first/rest/seq walks. - clojure.lang chunks 32-wide. Zig win: chunk = inline 32-slot array, not 32 heap thunks. - Bench: sieve, map_filter_reduce. - -**Tier 2 (medium):** -- **L3 — Keyword-map get fast path (destructure, worst 1.16×):** (a) monomorphic `bits==bits` - before the dispatch-aware `eqConsult` (900K calls/bench); (b) `@Vector firstIndexOfValue` - over the ≤8 array-map keys (cljw uses NO SIMD scan today); (c) optional keyword-keyed-map - threshold 8→64 (clojure's `KW_HASHTABLE_THRESHOLD`). Also audit the analyzer destructure lowering. -- **L4 — Variable-size vector tail + array-indexed size-class free-list.** TailNode is fixed - 32-slot (256B) → a 4-elem vector over-allocates; free_pool is a hashmap (a small-size-class - array free-list is cheaper). Bench: gc_alloc_rate. - -**Tier 3 (smaller / later):** -- **L5 — `big.int.Mutable` + reused limbs_buffer** for in-place factorial multiply (gap already - 1.02× — low priority). -- **L6 — CHAMP canonicalization-on-delete** for fast map `=` (if not already complete). - -## Experimentation + regression protocol (how to do this safely) - -- **Branch per lever**: `develop/collection-` off latest main. Experiment-and-revert / - commit-without-push until a lever PROVES a win; only then merge. A reverted experiment's - commit may stay in the branch log; never leave a merged branch red. -- **Correctness net (load-independent; collection changes are GLOBAL → run FULL, not smoke)**: - the F-012 dual-backend diff oracle (`zig build test -Dwasm`) + `corpus_regression` + - `CLJW_GC_TORTURE`. A semantics-preserving collection change passes these mechanically — this - is the "tests guard it" safety the change relies on. -- **Perf net (load-robust)**: per lever, the **interleaved cljw-vs-bb A/B + OFF/ON-knob** - pattern (validated in the D-519 GO) on a quiet-ish Mac; record a baseline first. Keep iff - diff-oracle-green AND bench win AND no regression on the other 8 ADR-0148 benches. -- **Smell/refactor gate (F-002/F-010)**: each lever held to finished-form; a workaround - triggers surgery, not a patch. - -## Consequences - -- Lands cljw as clearly fastest-script (flips the 6 collection losses to wins) AND improves - real-world code (transients/chunked-seqs are general, not bench-specific). -- Touches collection internals broadly; the diff oracle + corpus are the guard (per-lever full - run, not smoke). No algorithm rewrite, so behavioural risk is bounded. -- The compute frontier beyond bb (JVM-Clojure territory) remains the SEPARATE JIT unit — not - in this ADR's scope. - -## Alternatives considered (from the 3 surveys, full notes in the proposal record) - -- **RRB-tree vector** — rejected (common-path tax; Clojure keeps it contrib-only). Opt-in - `flex_vector` only if a concat-heavy workload ever justifies it. -- **Full data-structure rewrite** — rejected; cljw's algorithm choices are already best-of-breed - (CHAMP/radix+tail/array-map). The gap is constant-factor implementation. -- **Touch the VM engine for these benches** — rejected; the engine WINS pure compute 2×. The - lever is collections; the engine's frontier is the JIT (separate unit). - -Survey sources + the full proposal: `private/notes/collection-perf-proposal-20260624.md`, -`private/notes/9.2.S-clean-peer-rerank-20260624.md`, `private/notes/cljw_collection_codetruth.md`. - -## Amendment 1 (2026-06-24) — measurement-driven lever re-order - -The implementation Step 0.6 re-laying (main-agent, before the first lever's code) MEASURED -the current peer standing on a ReleaseSafe binary (hyperfine -N, 20-30 runs, load ~3, -interleaved cljw-vs-bb AND clean old-vs-new-binary A/B) and read the code-truth of each -losing bench. Two premises in the body above did not survive contact: - -1. **"cljw's transients are a STUB today" is FALSE.** `transient_vector.zig` ships a working - flat-buffer transient (`conj!`/`assoc!`/`pop!`/`persistent!`, the latter an O(n) bulk - `vector.fromSlice`), and `into`/`vec` in `core.clj` ALREADY route through it. The flat - buffer is arguably faster than a HAMT-editable transient for build-from-empty (contiguous, - no trie nav); its only weakness is `(into big-existing few)` copying `big-existing` up - front — which none of the benches hit. - -2. **No measured-LOSING bench is gated on a slow transient-build path**, so "L1 first, - highest ROI" was a Reservation-as-bias (the lever was planned, not evidence-led). Mapping: - - `gc_alloc_rate` (LOSES 1.06×) = 200K 4-elem vector LITERALS → L4 (small-tail over-alloc: - a 4-elem vector takes a fixed 256B `[32]Value` TailNode). Already partly addressed by - O-040 (`op_vector_literal` one-shot `fromSlice`). - - `gc_large_heap` (LOSES 1.19×) = map literals + `(into [] (map …))` — `into` is ALREADY - transient; the cost is small-map allocation + the lazy `map` walk. - - `destructure` (LOSES 1.18×) = keyword-map GET → L3. - - `sieve` (LOSES 1.14×) = lazy `filter` element-at-a-time walk → L2. - - `nested_update` now **WINS 1.39×** (the handover/peer-rerank that listed it as a loss is - STALE — D-519 auto-collect closed it). `bigint` is a tie. - - Corrected peer-standing table (replaces the body's "cljw loses 1.02–1.16×" row): - - | bench | cljw | bb | verdict (bb/cljw) | - |---------------|------|------|-------------------| - | gc_large_heap | 33.3 | 28.1 | LOSES 1.19× | - | destructure | 49.5 | 41.9 | LOSES 1.18× | - | sieve | 17.4 | 15.3 | LOSES 1.14× | - | gc_alloc_rate | 35.1 | 33.2 | LOSES 1.06× | - | nested_update | 11.3 | 15.7 | **WINS 1.39×** | - | bigint | 17.5 | 17.7 | tie | - -**Re-ordered levers (evidence-led; replaces the Tier-1/2/3 ordering for activation -purposes):** - -- **L3 (keyword-map get fast path) — LANDED first as O-051.** The clearest hot path - (decomposition micro-benches isolated map-destructure's get cost + a ~10ms binding-lowering - residual; `getonly` showed cljw keyword gets ~1.8× bb). Fix: `array_map` `get`/`contains` - compare keyword keys by raw NaN-box payload bits (keywords are interned ⟹ `=` is - bit-identity), skipping the per-entry `keyEq`→`eqConsult`→`keyEqValue` error-union chain. - Clean old-vs-new A/B (30 runs): destructure −6.6%, gc_large_heap −4.5%, 300k-get −11.0%. -- **L2 (chunked lazy-seq walk)** — sieve's `(filter … (rest s))` realizes one Cons per - element via a thunk Fn call; clojure.lang chunks 32-wide on the WALK path, not just reduce. -- **L4 (right-size small-collection allocation)** — the unifying theme of gc_alloc_rate + - gc_large_heap: fixed `[32]Value` TailNode / `[16]Value` ArrayMap over-allocate for small - collections. More invasive (touches core layout + GC trace) → after L2/L3. -- **L1 (transients)** is NOT retired — it remains available for the `(into big few)` / - HAMT-editable-transient improvement — but it is **demoted from "first"**: it does not move - any current losing bench, so it is not evidence-led right now. -- L5/L6 unchanged (low priority). - -This amendment does not change the body's core thesis (keep best-of-breed algorithms, win on -Zig-native layout); it corrects which lever is evidence-led FIRST and fixes two factual -premises. Per F-002, the lever order follows the finished form (measured loss + diagnostic -clarity), not the originally-written sequence. - -### Alternatives considered (Devil's-advocate) - -> **Provenance**: the mandatory fresh-context DA fork was attempted TWICE -> (2026-06-24) and both launches died on an external `API Error: 529 -> Overloaded` (0 subagent tokens — never ran). Per CLAUDE.md § The only stop, an -> external API block does not halt the loop; the DA analysis below is the -> main-agent fallback. It is genuinely adversarial — the correctness surface was -> stress-tested during implementation — and a fresh-context re-run can be added -> later if the bias-check is wanted, but the change is already PROVEN (measured -> A/B) and CORRECTNESS-VERIFIED (dual-backend diff oracle green). - -**(A) Three alternative shapes (within the F-NNN envelope):** - -1. **Smallest-diff** — keep ADR-0165's L1-first ordering as written; land the - keyword-get only as an unscheduled bonus, leaving the "transients are a stub" - text. _Better_: zero doc churn. _Breaks_: perpetuates a factually false - premise + the Reservation-as-bias; the next session re-derives the same - measurement to re-discover L1 is inert (wasted cycle). Fails F-002 (finished - form = evidence-led order), so rejected. -2. **Finished-form-clean** — the proposed measured-loss re-order, AND generalise - the keyword fast path into a typed `isIdentityKey` predicate (keyword + - boolean + nil + char — the immediate/interned bit-identity types) applied - uniformly to `get`/`contains` AND the `hash_map` HAMT path. _Better_: one - coherent symmetric abstraction. _Risk_: boolean/nil/char map keys are rare - (no measured win), and the hash_map extension still computes `keyHash` so the - payoff there is marginal — more surface for little gain. Adopt the re-order - now; record the `isIdentityKey`/hash_map generalisation as a LOW-priority - follow-up, not this cycle. -3. **Wildcard** — skip per-key fast paths; instead attack the destructure - LOWERING with a monomorphic inline-cache so a `{:keys [a b c]}` in a hot loop - resolves keyword→entry-index ONCE and reuses it (the ~10 ms binding-lowering - residual the decomposition micro-benches found). _Better_: targets a cost the - get fast path does not. _Risk_: needs a map-shape guard + invalidation; it is - hidden-class/shape-cache territory = the D-386 JIT frontier, out of the - collection-perf unit's scope. Defer to D-386. - -**(B) Correctness attack on `@intFromEnum(entry_key) == @intFromEnum(k)` (k a keyword):** - -- _Un-interned keyword_ — keyword.zig interns by (ns, name) (identical pairs share - one heap object), so two `:a` are bit-identical. SAFE; moreover `keyEqValue`'s - existing identity-first check ALREADY relies on this invariant, so nothing new - is assumed. -- _False hit vs a non-keyword entry_ — the compare is on the FULL `@intFromEnum` - (NaN-box tag + payload), and distinct `Value.Tag`s occupy distinct tag bits, so - a keyword `k` can only bit-match a keyword entry. A keyword is never `=` to a - non-keyword in clj. IMPOSSIBLE. -- _with-meta'd keyword_ — cljw keywords carry no per-instance meta (interning - ignores it; only SYMBOLS have the meta-bearing variant, which is why symbols are - kept on the general path via `symbolStructEq`). SAFE. -- _Namespaced keyword_ `:ns/name` — interned by the (ns, name) pair → distinct - object, bit-equal iff same ns+name. SAFE. -- _hash_map (>8 entries) path_ — UNCHANGED (still `hamtGet`/`keyEq`); correct as - before, just no speedup. Keyword maps that large are rare. Not a correctness - issue. -- _Float/NaN_ — unreachable (k is a keyword). SAFE. -- Conclusion: no wrong-answer path exists given keyword interning; the green - dual-backend diff oracle empirically confirms across the full corpus. - -**(C) Recommendation**: accept the amendment (measured-loss re-order + O-051 as the -first landed lever). The keyword-only restriction is the finished form for the get -path now; Alt 2's generalisation is a low-priority follow-up and Alt 3's -inline-cache belongs to D-386. None require violating an F-NNN. diff --git a/.dev/decisions/0166_publicization_polish_sweep.md b/.dev/decisions/0166_publicization_polish_sweep.md deleted file mode 100644 index e2935a98c..000000000 --- a/.dev/decisions/0166_publicization_polish_sweep.md +++ /dev/null @@ -1,104 +0,0 @@ -# ADR-0166 — Public-ization polish sweep (the post-perf operating mode) - -- **Status**: Accepted as the operating mode after the §9.2.S perf campaign's - accessible levers were exhausted (2026-06-24, user-directed). Repeatable, like - the F-010 quality-loop floor — NOT a one-shot phase. Each `/continue` - self-selects ONE polish-sweep category and drains it highest-value-first. -- **Relates to**: ADR-0165 + Amendment 1 (perf accessible levers exhausted — - O-051 was the last cheap one; the rest is GC-arch / JIT-fenced), F-002 - (finished-form), F-010 (quality-loop operating mode this mirrors), F-013 - (single-binary public artifact). Drains via `quality_floor: "public-ization: …"` - debt rows (D-522…D-529). ROADMAP §9.2.T. - -## Context — why this mode now - -The perf campaign (§9.2.S / ADR-0148) reached the boundary of its cheap levers -this session (O-051 keyword-map-get was the last clean constant-factor; the -remaining 9-bench gaps need a GC-arch change (L4 variable-length objects) or the -fenced/risky VM-dispatch/JIT frontier — ADR-0165 Amendment 1 + D-520/D-386). The -project is otherwise **near-complete** and is a **public artifact** (public repo, -single static binary, EPL-2.0). Its scaffolding, however, still reads as a -**private dev environment**: per-session `private/notes/` memos baked into the -loop, comments that are terse **pointers** to ADR/debt numbers rather than -self-contained explanations, docs written against an older code state, and a -large auto-loaded rules corpus. The next high-value work is **polishing the -project for an outside reader/contributor who clones it into their own -environment** — not new runtime capability. - -The user's framing (2026-06-24, verbatim intent): work `(b)` — the finite quality -sweeps that were deferred as "low ROI" — now becomes the standing `/continue` -mode. Source the sweep from the categories below; many are "あと少しが欠落" -(almost-done, small gap) or "古い" (stale-vs-code-truth) items. - -## Grounding (measured 2026-06-24 — the scale of each category) - -| Category | Measured scale | -|------------------------------------|---------------------------------------------------------------------| -| ADR/debt pointer comments in `src` | 1409 `ADR-NNN` + 1528 `D-NNN` lines across `src/**/*.zig` | -| Marker comments | PROVISIONAL 4 · PERF 65 · GC-ROOT 41 · VM-DEFER 0 | -| Auto-loaded rules | 31 files / 3736 lines (`.claude/rules/`) | -| Docs | 29 `*.md` (`docs/ja`, `docs/spec`, `docs/works`, `docs/ja/archive`) | -| Per-session notes | 853 files under `private/notes/` | -| Java-interop surface | 49 `runtime/java/**/*.zig` files / 92 `compat_tiers.yaml` fqn rows | - -## Decision — the sweep categories (each a standing debt row, drained by Step 0.5) - -The loop self-selects highest-value-first; the rows are the menu, ADR-0166 is the -intent. A correctness/clj-parity floor still outranks pure polish (CLAUDE.md). - -1. **D-522 — Comment de-pointering + condensation.** Replace pointer comments - (`[refs: D-NNN]`, "per ADR-NNN", "see D-NNN") with **self-contained - explanation** — a reader without the ADR/debt ledger should understand the - code. ADR documents themselves STAY (they are the rationale record); only the - in-code *pointers to them* become prose. Condense over-verbose comments. This - is the largest category (≈3000 lines) — explicitly **gradual, a little each - `/continue`**. Keep the `Why`-non-obvious comments (CLAUDE.md comment rule). -2. **D-523 — Doc audit vs code-truth.** The docs were written against an older - code state. Audit each against what the code actually does; fix, **simplify**, - delete what is no longer meaningful, **archive** (move to `docs/ja/archive/` - or similar) what is historically meaningful but dead. -3. **D-524 — `private/` decoupling + per-session-note abolition.** `private/` is - gitignored and assumes the user's machine; the loop's Step 7 writes a per-task - note every cycle (853 accumulated). For a public artifact this should not be - load-bearing. Decouple CLAUDE.md / the `continue` + `code_learning_doc` skills - from `private/notes/`; retire the mandatory per-task note (or make it opt-in). -4. **D-525 — Rules + skills public-ization review.** 31 rules / 3736 lines auto- - load into every turn. Review for an outside contributor: prune/merge stale or - redundant rules, ensure each earns its context cost, de-private-ize wording. - Same for the skill set. -5. **D-526 — Java-interop static-member gap catalog + fill.** Some static members - are missing. If they cannot be enumerated mechanically, the FIRST step is to - **catalogue** them (extend `compat_tiers.yaml` / a coverage doc), then fill the - "あと少し欠落" tail. -6. **D-527 — clj-parity alignment with upstream progress.** Clojure itself has - advanced; align with new upstream behaviour **where meaningful**. Folds into - the standing D-175 `quality-loop floor: clj-parity` (sweep surfaces DIFFs). -7. **D-528 — Real `deps.edn` famous-library usage.** Load real, popular Clojure - libraries via their actual `deps.edn` and run them — surface bugs / unimplemented - surfaces from real-world use, not synthetic tests. -8. **D-529 — Marker-comment inventory + validity.** Walk the 110 markers - (PROVISIONAL / PERF / GC-ROOT); validate each is still accurate + earning its - keep; retire stale ones; keep the meaningful. - -## Consequences - -- The codebase + scaffolding converge on a **clean public artifact** an outside - reader can navigate without the private ledger. -- Pure polish; **no behaviour change** is the default (doc/comment/scaffolding - edits). Where a sweep DOES touch code (D-526 interop fill, D-527 parity, D-528 - bug fixes), the normal gate (diff oracle + corpus) applies. -- The §9.2.S perf campaign is **paused** (`.dev/.perf_campaign_active` removed) — - its remaining levers (D-520 collection / D-386 dispatch / D-005 JIT) stay in - standing debt for a future deliberate decision, not the active mode. - -## Alternatives considered - -- **A one-shot "cleanup phase" with a fixed task list** — rejected: the categories - are open-ended (de-pointering ≈3000 lines, doc audit across 29 files) and better - drained incrementally per `/continue` (the F-010 repeatable-mode shape), so a - fixed list would either under-scope or stall. -- **A dedicated big-bang "大整理" cycle (D-436)** — D-436 remains the epic for - finished-form *code* deviations; this ADR is the broader **public-ization** mode - (docs / comments / scaffolding / interop / parity), drained alongside it. -- **Leave the scaffolding as-is** — rejected: the project is public; private-env - scaffolding + pointer-only comments are a real barrier to outside contribution. diff --git a/.dev/decisions/0167_release_mechanics_1_0_0_rc_1.md b/.dev/decisions/0167_release_mechanics_1_0_0_rc_1.md deleted file mode 100644 index dd05a2723..000000000 --- a/.dev/decisions/0167_release_mechanics_1_0_0_rc_1.md +++ /dev/null @@ -1,176 +0,0 @@ -# ADR-0167 — Release mechanics for 1.0.0-rc.1 (the public-artifact readiness track) - -- **Status**: Accepted (2026-07-01, user-directed: "1.0.0-rc.1 を目指す綿密な - 作業計画"; full-scope A+B, fully autonomous execution). Shape = DA-fork - Alternative 2 (explicit finite readiness gate; B fully drained but - non-blocking for the rc.1 *signal*). -- **Relates to**: ADR-0166 (public-ization polish sweep — the *internal-quality* - drain: D-522…D-529), F-002 (finished-form), F-013 (single-binary public - artifact). Sibling to ADR-0166: **0166 = internal polish (comments / docs / - scaffolding / interop / parity); 0167 = the release *mechanics* an external - clone needs (CI / community-health files / CHANGELOG / attribution / - environment decoupling / version staging).** Reference template: the zwasm - v2.0.0-rc.1 release series (S0…S7, `~/Documents/MyProducts/zwasm_from_scratch`). - -## Context — why this ADR now - -ADR-0166 framed the public-ization *quality* sweep but omits the concrete -**release machinery** a first external clone needs. An independent gap-analysis -against the zwasm v2 release series (S0…S7) found that CWFS is missing the entire -release-mechanics layer that zwasm shipped: - -- **No CI at all** — `.github/` does not exist; an external contributor gets zero - automatic verification on push/PR (zwasm called this "the headline - public-readiness gap", S3). -- **Community-health files** — only `CONTRIBUTING.md` exists (and it is - inconsistent with README's "Issues/PRs paused → Discussions" posture); missing - `SECURITY.md`, `CODE_OF_CONDUCT.md`, `.github/ISSUE_TEMPLATE/config.yml`, - `PULL_REQUEST_TEMPLATE.md`, `FUNDING.yml`. -- **No `CHANGELOG.md`**, no `THIRD_PARTY.md` (deps attribution beyond the EPL - `NOTICE`), no `.gitattributes` (LF) / `.editorconfig`. -- **Personal-environment coupling** — `scripts/run_remote_ubuntu.sh` hard-codes - the `ubuntunote` SSH host; `.claude/settings.json` carries personal - `additionalDirectories`; two `src` comments leak author-local paths. - -CWFS is otherwise in good public shape: README (logo, Discussions posture, no -internal-ID leaks), `--version` threaded from `build.zig.zon .version` as SSOT, -EPL-2.0 `LICENSE` + `NOTICE`, only 2 benign path leaks in `src`. - -## Decision — the release-mechanics tracks (each a debt row, drained by Step 0.5) - -**Track A — rc.1-blocking release mechanics** (finite, high-value; mirrors zwasm -S0-S3 + attribution + CHANGELOG): - -- **D-536 (=S0) — debt-ledger code-truth reconciliation.** Reconcile the 84 - `active:` rows against code truth ahead of the public cutover; delete/narrow - stale + false-positive rows (the zwasm S0 pattern). No source touched. -- **D-537 (=S1) — community-health files.** `SECURITY.md` (scope: a runtime - executing untrusted Wasm bytecode; private reporting) / `CODE_OF_CONDUCT.md` - (Contributor Covenant 2.1) / `.github/ISSUE_TEMPLATE/config.yml` - (`blank_issues_enabled: false` → Discussions) / `.github/PULL_REQUEST_TEMPLATE.md` - (PRs-paused notice) / `.github/FUNDING.yml`. **+ reconcile `CONTRIBUTING.md`** - with README's posture (Issues/PRs paused → Discussions; build instructions → - `-Dwasm -Doptimize=ReleaseSafe`; the `--serial-e2e` full-gate caveat). -- **D-538 (=S2) — personal-environment decoupling.** SSH host → - `${CLJW_UBUNTU_HOST:-ubuntunote}` (behavior-preserving default); the 2 `src` - path-leak comments generalized; personal `additionalDirectories` moved to the - gitignored `settings.local.json`. -- **D-539 (=S3) — CI wiring ★headline.** `.github/workflows/ci.yml` - (push `main` + PR; macOS + Ubuntu matrix) driven by a single - `scripts/ci_gate.sh` SSOT so **CI can never verify less than the per-host - gate**; `.github/dependabot.yml` (weekly actions); a gitleaks config that scans - the working tree (the lone historical false-positive `.gitleaksignore` was - removed 2026-07-01 in the top-level tidy — current tree is clean, so no - allowlist is needed); Zig 0.16.0 pin mirrored for CI. - **CWFS-specific**: the full gate MUST run `--serial-e2e` (D-418 load-race) and - probe a ReleaseSafe binary (ADR-0132). -- **D-540 (=CHANGELOG + attribution + formatting).** `CHANGELOG.md` - (Keep-a-Changelog) added to `build.zig.zon .paths`; `THIRD_PARTY.md` naming - **exact license + pinned version** per dependency (embedded zwasm = Apache-2.0 - @ `fc7ff0b3b` / `v2.0.0-alpha.3`; zlinter dev-dep = MIT; the EPL-1.0/2.0 clj - lineage `NOTICE` records) — an EPL-2.0 project embedding an Apache-2.0 dep; - `.gitattributes` (LF) + `.editorconfig`. Also ship `NOTICE` (currently absent - from `.paths` despite the EPL `src/lang/clj/**` shipping inside `src`). -- **D-542 (=release-artifact workflow, PREPARED-not-fired).** A tag-triggered - `.github/workflows/release.yml` that builds the static `cljw` binary for - macOS + Linux (`-Dwasm -Doptimize=ReleaseSafe -Dcpu=baseline` for the deploy - artifact, ADR-0132) and attaches it to the GitHub release. For a - "single static binary" product (F-013) a pre-built `cljw` is most of what a - non-Zig user needs. The workflow is authored and merged but **only fires on a - tag the user pushes** — it does not tag (mirrors zwasm's `release.yml`). -- **D-543 (=dependency-pin coherence / reproducibility).** (a) Resolve the - incoherent stability story of a `1.0.0-rc.1` embedding `zwasm v2.0.0-alpha.3`: - either bump the pin to a coherent zwasm release once one exists (user-owned, - co-dev CODEV), or document the version relationship explicitly in - `THIRD_PARTY.md` + the zwasm-capabilities ledger as a known pre-1.0 coupling. - (b) The `zlinter` dev-dep is fetched **eagerly** (a doc-reader's first - `zig build` pulls an external repo — an offline/reproducibility footgun); make - it lazy/dev-only if the build.zig restructure is tractable, else record it as a - **named pre-1.0 wart with a tracking row** (not a bare "deferred"). - -**The rc.1 readiness gate is FINITE and explicit** = Track A (D-536…D-540) + -D-542 + D-543(a-doc/b-decision) + version staging (D-541). This finite set is -the SSOT for "can the user cut the `1.0.0-rc.1` tag?" — recorded as a checklist -in `handover.md` and kept code-true by the D-175 audit. A *release candidate* is -gated on a finite, honest readiness set, **not** on an open-ended discovery loop. - -**Track B — the ADR-0166 internal-quality drain (D-522…D-529)** is, per the -user's full-scope decision (2026-07-01), **fully drained** (comment -de-pointering + condensation, doc audit vs code-truth, `private/` decoupling, -rules/skills review, java-interop static-member gap fill, clj-parity upstream -alignment, real-`deps.edn` library usage, marker-comment inventory) — but as a -**parallel quality track that does NOT gate the rc.1 readiness signal**. This is -F-compliant: F-002 governs per-task quality, and since 2026-06-25 no longer -governs inter-task *order*; nothing requires a release candidate to wait until -all debt is zero (that would contradict rc semantics, and D-529 real-lib usage -is an F-013 *discovery* mechanism that surfaces open-ended new gaps by design). -Track B is drained EASIEST-FIRST alongside Track A; the two converge, but the -tag can be cut when the finite Track-A gate is green. - -**Version staging (D-541, user-owned boundary).** All rc.1 preparation lands on -`main` **while `build.zig.zon .version` stays `1.0.0-alpha.1`** until the user's -bump. To avoid a transient SSOT contradiction (zwasm hit exactly this — a -dedicated "fix CHANGELOG version line" commit): the CHANGELOG rc.1 entry sits -under an **`## [Unreleased]` heading**, and every `1.0.0-rc.1` string in -CHANGELOG / THIRD_PARTY / docs is **staged text that the user's single -`.version` bump + tag activates**. The final `1.0.0-rc.1` bump **and the -`git tag`/publish are the user's action** — the loop prepares everything but -NEVER tags or publishes (build.zig.zon is the SSOT; "the user owns the value via -the release tag"). This ADR does not authorize the loop to cut the tag. - -## Consequences - -- An external clone gets: automatic CI verification on push/PR, the standard - community-health files, a CHANGELOG, complete third-party attribution, and no - author-home coupling in shipped files. -- Track A is pure additive/mechanical (no runtime behaviour change); Track B's - code-touching rows (interop fill / parity / lib bugs) take the normal diff-oracle - + corpus gate. -- The rc.1 tag becomes cuttable the moment the **finite** readiness gate (Track A - + D-541/542/543) is green; the bulk `D-522` comment de-pointering (~3000 lines, - gradual) and the open-ended `D-529` real-lib discovery continue draining in - parallel **past** the tag without blocking it — this is the intended rc shape, - not a wart. -- Named pre-1.0 warts (tracked, not silently deferred): the eager `zlinter` - dev-dep fetch (D-543b) and the zwasm-pin/version-line coherence (D-543a). - -## Alternatives considered - -The following is the Devil's-advocate subagent's output (fresh context, briefed -with the active F-NNN constraints), reflected verbatim per CLAUDE.md § ADR-level -designs. Its recommendation (Alternative 2) was adopted. - -> ## Leading finding: no F-NNN block -> None of the three alternatives requires violating F-002 / F-013 / the user-owned-tag invariant. In particular, **decoupling the rc.1-readiness gate from the Track-B quality drain does NOT violate F-002** — F-002 governs per-task quality and explicitly stopped governing inter-task order (2026-06-25); it does not say "release only when all debt is zero" (that would contradict the very notion of a release *candidate*). So the finished-form-clean option below is F-compliant and is recommended despite a larger diff. -> -> ## Alternative 1 — smallest-diff (label + gap-fill in place) -> Keep the draft's two-track structure verbatim; add only (a) crisp **blocking / non-blocking** tags to each debt row, and (b) three missing Track-A rows: release-artifact workflow, zwasm-pin coherence, README build-command accuracy. -> - **Better than draft**: removes the blocking-set ambiguity and closes the biggest contributor-facing gaps at near-zero restructuring cost. -> - **Breaks / risks**: leaves the "Track B folded into the same rc.1 goal" wording intact, so the scope-creep / indefinite-delay pressure persists culturally even if individual rows are tagged non-blocking. Half-measure. -> -> ## Alternative 2 — finished-form-clean (explicit rc.1 readiness gate as an SSOT) — RECOMMENDED -> Introduce a first-class, code-true **`release readiness gate`** enumerating the *finite* blocking set, and reclassify Track B out of the blocking set: -> - Blocking = Track A (D-536…D-540) + version-staging prep (D-541) + **new D-542 release-artifact workflow** (tag-triggered static-`cljw` build, *prepared-not-fired*, per the user-owned-tag invariant — mirrors zwasm's `release.yml`) + **new D-543 dependency-pin coherence** (align embedded `zwasm` to a released/pinnable tag consistent with the "v2" attribution claim — shipping a `1.0.0-rc.1` that embeds `zwasm 2.0.0-alpha.3` is an incoherent stability story; and resolve the eager `zlinter` fetch to lazy/dev-only OR document it as a named pre-1.0 wart with a tracking issue rather than a bare "deferred"). -> - Track B (D-522…D-529) becomes a **standing parallel quality drain** — fully drained per the user's full-scope directive, but explicitly **NOT gating the rc.1 readiness signal**. -> - **Better than draft**: crisp and honest; matches what zwasm *actually* shipped (release.yml + versions.lock + contract canary, none of which the draft's D-539 mentions); decouples the unbounded F-013 discovery (D-529) from the release gate, so rc.1 can be cut when the finite set is done. -> - **Breaks / risks**: larger diff — recommend anyway per F-002. The readiness-gate doc is one more thing to keep code-true (fold into the D-175 5-lens audit). Reclassifying B as non-blocking must be worded so it does not read as softening the user's "full-scope A+B" call (mitigation: B is still fully drained; only the *gate coupling* is removed). -> -> ## Alternative 3 — wildcard (rc as the discovery vehicle; invert the sequence) -> Treat rc.1's explicit purpose as *inviting external friction*. Front-load only the honesty/safety-critical minimum — LICENSE/THIRD_PARTY completeness, SECURITY.md, README build accuracy, CI-verifies-what-the-gate-verifies, zero author-home leaks, and a `release.yml` producing binary artifacts — as a tight blocking set. Defer community-health niceties (FUNDING/CoC), comment de-pointering, doc-audit depth, dependabot to rc.2/final. -> - **Better than draft**: fastest *honest* exposure; the F-013 discovery loop gets real external input instead of synthetic `deps.edn` probing. -> - **Breaks / risks**: "rc" connotes feature-complete/stable, so a deliberately-rough rc risks a reputational-label mismatch — an alpha/beta posture fits this intent better, mildly tensioning the user's stated "1.0.0-rc.1" target. A first public impression missing CoC/templates reads as unfinished, and under-delivers on the "綿密な作業計画" the user asked for. -> -> ## Direct critique of the draft -> 1. **The rc.1 blocking set is internally contradictory.** Track B says D-522…D-529 are "folded into the same rc.1 goal," but Consequences says the ~3000-line D-522 de-pointering "may extend past the rc.1 tag." So is B blocking or not? Undefined. Folding the *entire* D-522…D-529 drain into the rc.1 goal is the wrong call — D-529 (real-lib usage) is an F-013 discovery mechanism that surfaces open-ended new gaps, and the ~3000-line sweep is explicitly gradual. Gating a release candidate on an unbounded discovery + open-ended sweep is a direct indefinite-delay risk. Fix: finite blocking set (Track A + version prep), B as parallel non-blocking drain. -> 2. **Missing Track-A items:** binary-release artifacts (CI verifies but produces no artifact; zwasm shipped `release.yml`) → D-542; dependency-pin coherence + eager zlinter fetch → D-543; README build-command accuracy (`-Dwasm -Doptimize=ReleaseSafe`) as an explicit row; THIRD_PARTY must name license + exact version. -> 3. **Version-string coordination hazard (zwasm already hit it).** `.version` stays `1.0.0-alpha.1` until the user's bump, yet D-540 references `1.0.0-rc.1`. Specify the staging convention up front (CHANGELOG rc.1 under `Unreleased`; all rc.1 strings staged text the bump activates) so a version-consistency check does not flag the prepared state. -> 4. **Consider a public-API/consumer contract canary** (zwasm shipped one). CWFS ships both a binary and an embeddable library; a pinned-consumer smoke would catch accidental API breakage before the tag. Non-blocking, cheap insurance. -> -> ## Non-binding recommendation -> Adopt **Alternative 2.** Make the rc.1 readiness gate an explicit finite SSOT (Track A + D-541 + D-542 + D-543); reclassify Track B as fully-drained-but-non-blocking; add the version-string staging convention. This keeps the user's full-scope A+B intent (B is still drained) while making "when can the user cut the tag?" answerable and finite. - -**Adopted**: Alternative 2, with the version-staging convention (critique §3) -folded into D-541 and the consumer-contract canary (critique §4) recorded as a -non-blocking follow-up under D-539. Critique §1's contradiction is resolved in -the Decision section (finite gate + non-blocking B). The README build-command -accuracy (critique §2) is folded into the D-537 CONTRIBUTING/README reconcile. diff --git a/.dev/decisions/0168_reconcile_operating_model_p1_rescope.md b/.dev/decisions/0168_reconcile_operating_model_p1_rescope.md deleted file mode 100644 index 46a6645b6..000000000 --- a/.dev/decisions/0168_reconcile_operating_model_p1_rescope.md +++ /dev/null @@ -1,182 +0,0 @@ -# 0168 — Reconcile ROADMAP's operating-model text with the autonomous loop; rescope inviolable principle P1 - -- **Status**: Proposed → Accepted -- **Date**: 2026-07-01 -- **Author**: the ClojureWasm authors -- **Tags**: governance, roadmap, autonomous-loop - -## Context - -The round-2 multi-agent quality audit (find → adversarially verify → synthesize) -surfaced a coherent **stale-operating-model cluster** in `.dev/ROADMAP.md`: the -file still describes an *older* operating model that later decisions superseded, -while the loop already follows the newer model documented in CLAUDE.md -(`.claude/CLAUDE.md`). Concretely: - -- **Inviolable principle P1** (§2, table row) reads *"Move forward only with - understanding — Interactive Claude Code use. No overnight batch commits."* The - project's **default is continuous autonomous execution** (CLAUDE.md - § Autonomous Workflow / § The only stop), and the user has repeatedly directed - overnight / batch autonomous work. -- **§0 / §12.4** state the loop has **two** stop conditions ("explicit user - request, or a physical block = unrecoverable build/test failure"); CLAUDE.md - § The only stop declares exactly **one** (explicit user request) and states - "Test failure / gate red — does not stop." -- **§0 / §16** present the retired **phase-queue / "expand §9. - placeholder"** model as active (retired by ADR-0142 → gap-area I/II/III + - easiest-first debt drain). -- **§12.4** labels the per-commit gate **"Mac+Linux"** (Linux is boundary-only - per ADR-0049; the gate is two-tier smoke/full per ADR-0107). -- Minor drift: §0 says "25 ADRs" (now 168), and "ARCHITECTURE.md" (moved to - `docs/architecture.md`). - -The user directed both remedies: (1) fix the §0/§12.4/§16 live-procedure -contradictions via a ROADMAP §17 in-place amendment; (2) rescope P1 via this ADR -so continuous autonomous execution is canonical. - -Per the project priority chain (`project_facts.md`), the only allowed -reconciliation direction is to align the lower-priority doc (ROADMAP) **up** to -law (F-NNN) and to CLAUDE.md — never the reverse. - -## Decision - -1. **Rescope P1 (keep the name/intent, rewrite the Effect).** P1 stays *"Move - forward only with understanding."* Its Effect becomes: *"Continuous autonomous - execution is the default (CLAUDE.md § Autonomous Workflow); overnight / batch - autonomous commits are expected. 'Understanding' is enforced per-commit by the - two-tier gate (ADR-0107) + the mandatory Step-6 `Smell-audited:` audit + P9 - (never commit red) — not by human interactivity."* The genuine safety intent - of the old clause (don't ship code the loop cannot explain) is **relocated** - to the gate + smell-audit, not deleted. - -2. **Collapse the stop model to one condition.** The loop stops only on - CLAUDE.md § The only stop's single condition (explicit user request). The - former second condition ("physical block") is not removed behaviourally — its - intent already lives elsewhere: gate-red → **diagnose and fix in-flight** - (does not stop); a genuinely underivable external fact (credential / product - preference) → **in-task `AskUserQuestion`**, not a session stop. - -3. **Declare CLAUDE.md the SSOT for the loop procedure + stop model.** ROADMAP - §12.4's 7-step table is retained as a **cold-start quick-ref explicitly marked - derivative** ("change the procedure in CLAUDE.md, not here"), and its stale - cells are fixed (gate label → "Mac smoke per-commit; full + Linux batched"; - two → one stop condition). This resolves the authority-drift (the defect that - caused this ADR) while keeping the readable inline table and minimising - cross-reference risk. **Depth-vs-authority split (deliberate):** the SSOT for - the *procedure* is intentionally the lower-priority CLAUDE.md, while the - *authority to change the model* stays with ROADMAP / ADR / F-NNN — a future - reader must not "fix" this back into ROADMAP. - -4. **Retire the phase-queue prose.** §0 and the §16 glossary "Phase open - procedure" row are updated to the gap-area self-selection model (ADR-0142); - the dead glossary row is struck, not merely re-pointed. Stale §0 counts/paths - are fixed as a byproduct (Step 0-8 → 0→7; Next Phase Queue → First task on - resume; 25 ADRs → "the ADRs"; ARCHITECTURE.md → docs/architecture.md). - -## Alternatives considered - -The mandatory Devil's-advocate subagent (fresh context, F-002/F-009 + the -priority chain grounded) produced the following, reflected verbatim: - -> **Leading finding — F-NNN block: NONE.** No active F-NNN forbids any shape -> below. F-002 (finished-form wins over smallest-diff) *disfavours* the -> smallest-diff option: choosing the minimal line-patch on cycle-budget grounds -> would itself be the Smallest-diff-bias smell F-002 names. F-009 is irrelevant -> (no feature-neutrality surface). The autonomous model is already the default -> per CLAUDE.md, so making it canonical aligns a lower-priority doc up to law + -> CLAUDE.md — the only allowed direction. One structural nuance to record (not a -> block): CLAUDE.md says "if anything conflicts with the roadmap, the roadmap -> wins," yet §0/§12.4 *already* delegate the step-by-step spec to CLAUDE.md — so -> a pointer-to-CLAUDE.md shape is consistent with existing practice, but the ADR -> should state that the SSOT for the *procedure* is intentionally the -> lower-priority doc while the *authority to change the model* stays with -> ROADMAP/ADR/F-NNN (a deliberate depth-vs-authority split). -> -> **Shape A — Smallest-diff: line-by-line in-place patch.** Rewrite only the -> wrong cells. Better: minimal blast radius, locally auditable, honours §17 -> "edit in place." Breaks: preserves the **root defect** — §12.4 keeps -> *duplicating* the loop procedure CLAUDE.md owns; two copies of a living -> procedure drift again (this ADR exists because they drifted). Under F-002 this -> lands a dirtier finished form (two copies), so cycle-size cannot justify it. -> -> **Shape B — Finished-form-clean: SSOT consolidation (RECOMMENDED).** Make -> CLAUDE.md the sole SSOT; reduce §12.4 to a pointer; rewrite P1's Effect (keep -> the name), collapse to one stop condition, delete the retired phase-queue -> prose in §0/§16, fix stale counts/paths. Better: eliminates the duplication -> class permanently (the redundancy finding is *resolved*, not patched); one -> place to change the model; priority chain reads cleanly. Recommended even -> though its diff is larger — cycle/LOC is not a constraint. Risks: (1) a reader -> on §12.4 no longer sees the procedure inline (mitigate with an explicit -> pointer; §0 already routes to CLAUDE.md first); (2) the priority-inversion -> nuance becomes load-bearing (must be stated); (3) larger edit = enumerate every -> §16/§15 dependent or a stale glossary row survives. -> -> **Shape C — Wildcard: preserve P1 by *narrowing*, not overwriting.** Keep P1 -> with its Effect narrowed to the surviving safety kernel; add a new principle -> "Continuous autonomous execution" carrying the overnight-batch mandate + a -> "supersedes P1's former interactive clause" note. Optionally promote the -> autonomous model to a fresh F-NNN so it stops being re-litigated at ROADMAP -> level. Better: makes the intent surgery legible (which part of P1 died vs -> survived). Breaks: two principles cover one topic (principle-count inflation); -> the F-NNN-promotion half is **not loop-decidable** (needs user direction + -> Revision-history entry per project_facts.md); cuts slightly against §17's -> edit-in-place house style. -> -> **Recommendation (non-binding): Shape B, with Shape C's intent-preservation -> discipline.** (1) Rewrite P1's Effect but do NOT merely delete "No overnight -> batch commits" — relocate its genuine intent (don't ship code you don't -> understand) explicitly onto the gate (ADR-0107) + Step-6 smell-audit + P9. (2) -> Collapse 2→1 stop conditions but say where the old second went (gate-red → -> diagnose-and-fix; underivable external fact → in-task AskUserQuestion). (3) -> Consolidate §12.4 to a pointer and strike the dead §16 glossary row. - -**Chosen:** Shape B's SSOT-consolidation intent + Shape C's intent-preservation -(findings 1-2), with one adjustment: §12.4's 7-step table is **kept as a -derivative quick-ref carrying an SSOT pointer** rather than fully gutted to a -bare pointer. Rationale (not cycle-budget — F-002-clean): a derivative table -with a declared SSOT + "change it in CLAUDE.md" note is itself a clean finished -form (SSOT declared, drift-authority removed), keeps the cold-start quick-ref -readable, and minimises the §15/§16 cross-reference risk the DA flagged for full -gutting. The DA's full-B (gut to pointer) and the Shape-C F-NNN promotion remain -available as user-owned follow-ups. - -## Consequences - -- **Positive**: ROADMAP's operating-model text matches reality; the autonomous - model is canonical at principle level; the loop-procedure SSOT is single - (CLAUDE.md) with the authority-to-change staying at ROADMAP/ADR/F-NNN; the - redundancy finding is addressed via the SSOT pointer. -- **Negative**: §12.4 retains a derivative table (a small, marked duplication) - rather than a bare pointer — accepted for readability; the SSOT pointer guards - against re-drift. -- **Neutral / follow-ups**: (a) promoting the autonomous-execution model to a - fresh **F-NNN** is a user-owned option (per DA Shape C) the loop surfaces but - does not self-execute. (b) The broader "make CLAUDE.md the *sole* inline - procedure and gut §12.4" (full Shape B) can be revisited on user request. - -## Affected files - -- `.dev/ROADMAP.md` §0 (reading-order + step-count + stop-framing + ADR-count + - paths), §2 (P1 Effect), §12.4 (gate label + single stop condition + SSOT - pointer), §16 (glossary "Phase open procedure" struck) — amended in place per §17. -- `.dev/decisions/0168_reconcile_operating_model_p1_rescope.md` — this ADR. -- `.dev/handover.md` — synced (§17 step 3). - -## References - -- ROADMAP §0 / §2 / §12.4 / §16 (amended); §17 amendment policy. -- Related ADRs: ADR-0142 (phase-queue retired → gap-area model), ADR-0107 - (two-tier smoke/full gate), ADR-0049 (Linux gate boundary-only), ADR-0025 - (chapter cadence dormant). -- CLAUDE.md § Autonomous Workflow / § The only stop (the loop-procedure SSOT). -- `project_facts.md` F-002 (finished-form wins) + the priority chain. - -## Revision history - -- 2026-07-01: Status: Proposed → Accepted (initial landing). Surfaced by the - round-2 quality audit; Devil's-advocate output reflected above. -- 2026-07-01: Amended (user-authorized) — folded in the **P3** truth-fix (§2 Effect - "Extensions go to `modules/`" → "`src/lang/clj/` or pods"; `modules/` retired) and the - D-547 ROADMAP relabels (`check_adr_history` → on-demand audit; the 4 scripts' retired - "Phase 5 promotion" → on-demand). Same operating-model-reconciliation theme; P3 is a - truth-fix (intent unchanged), not a rescope, so no separate DA fork. diff --git a/.dev/decisions/0169_analysis_roots_frame.md b/.dev/decisions/0169_analysis_roots_frame.md deleted file mode 100644 index 6d28af31a..000000000 --- a/.dev/decisions/0169_analysis_roots_frame.md +++ /dev/null @@ -1,179 +0,0 @@ -# ADR-0169: Analysis-roots frame — rooting analyze/compile/deserialize-time Values - -- Status: Accepted (2026-07-02) -- Deciders: autonomous loop (per CLAUDE.md § ADR-level designs are handled inline) -- Related: ADR-0028 (§5 root sources; amendment 3 gray worklist), ADR-0091 - (thread_roots union walk), ADR-0164/D-519 (alloc-boundary auto-collect), - D-430, D-531, `.dev/gc_rooting.md` - -## Context - -GC Values produced BEFORE any frame executes had no root: - -- **Analyzer literals** — `makeConstant` stores a freshly-allocated Value - (string / bigint / regex / tagged / empty-list) in an arena `Node`; - `analyzeQuote` stores a fully-built quoted structure in a `quote_node`. -- **Compiler constants** — `Compiler.addConstant` appends to an arena - ArrayList; 7 more `string_mod.alloc` sites feed it. -- **Deserialized AOT constants** — `serialize.deserializeChunk` fills an - arena slice one `readValue` at a time. -- **Macro-expansion intermediates** — `expandIfMacro`'s `&form` / `&env` / - arg Values and the expansion result cross between runtime and analysis in - Zig locals. - -`EvalFrame.constants` (D-251) roots a chunk's pool only WHILE `vm.eval` runs. -So any collect in the window between a Value's creation and its execution -swept it: (a) a user-macro expansion mid-analysis runs arbitrary eval (the -minimized repro: `(defmacro m [x] x) (deftype T [] Object (toString [self] -(m 1)))` under `CLJW_GC_TORTURE=1` — the `"toString"` string cell was swept -and recycled as the `"->T"` factory-name string, so extend-type raised -"host-marker method not yet wired" on a fully-wired method); (b) the D-519 -alloc-boundary auto-collect fires mid-analysis of any large in-eval load once -the 4MB threshold crosses — instaparse's per-run garbage-bytes symbol -NameError (D-430) and the D-531 "index out of bounds: index 16, len 16" -panic family. - -## Decision - -A per-thread **analysis-frame chain** (`root_set.AnalysisFrame` + -threadlocal `analysis_frame_head`), the EvalFrame's analysis-side sibling: - -1. **Brackets**: every analyze/compile/deserialize→eval seam opens a frame on - its C stack (`beginAnalysis` / `defer endAnalysis`) and keeps it open - THROUGH the form's evaluation (also covers tree_walk Node constants, - which have no EvalFrame pool). Seams bracketed: `driver.evalTopLevelForm`, - `driver.runEnvelope` (per chunk — keeps the high-water mark at one form's - constants), `app/repl.zig`, `app/nrepl.zig`, `app/builder.zig` (5 sites), - `eval/evaluator.zig` (`--compare` oracle twin). -2. **Producers push**: `analyzer.makeConstant`, `special_forms.analyzeQuote`, - `Compiler.addConstant`, `serialize.deserializeChunk`'s constant loop, and - `expandIfMacro` (amp_form / amp_env / value_args / expansion result). The - frame stores the allocator (`gc.infra`) so producer signatures are - unchanged. Immediates are filtered at push. -3. **Fail-loud tripwire**: `pushAnalysisRoot` asserts a frame is open (safe - builds). A producer running outside any bracket is a caught bug at the - source, not a silent unrooted window — future drivers self-announce. -4. **Walker**: a new `.analysis` sub-phase in `ThreadRootsCursor` (after - `.self_guard`) drains the chain; `ThreadGcContext.analysis_frame_slot` - (default static null) publishes each worker's chain — the ADR-0091 - union-walk pattern, one new section in `gc_rooting.md`, not one new - concept. -5. **`macro_root_slot` RETIRED**: it never gained a production writer - (gc_rooting.md §B), and its declared purpose (macro-expansion - intermediates) is exactly what the frame now carries — `expandIfMacro` - pushes them. The walker `.macro` phase, `ThreadRoots.macro`, - `ThreadGcContext.macro_slot`, and the threadlocal are removed; worker - registration sites now publish `analysis_frame_head` instead. Net concept - count in the rooting SSOT: +1 −1. - -## Alternatives considered - -A fresh-context Devil's-advocate subagent was forked (depth ≥ 2 mandate). -Summary of its findings (near-verbatim; the full text is in the fork's -output, reflected here in condensed form): - -- **Premise verified**: ~14 analyzer literal arms + 7 compiler string sites + - the deserialize loop are unrooted until execution; `maybeAutoCollect` fires - at the alloc boundary whenever analysis happens inside an outer eval - (`require`/`load`/`eval` mid-program) — both repro legs structurally - confirmed. The macro-expansion result Value was flagged as a sibling - unrooted window to fold into the same change (done). -- **The draft's original `evalTopLevelForm`-only watermark boundary was - WRONG**: six seams escape it (REPL, nREPL, builder ×N, the `--compare` - oracle twin, `runEnvelope`, bootstrap AOT), and `deserializeChunk` was a - missed third producer ON THE DEFAULT STARTUP PATH. This pushed the design - from an implicit watermark registry to the explicit self-enforcing frame. -- **Alt A (smallest-diff)** — watermark over the existing `permanent_roots` - pin list: zero new walker code, but the pin list is process-global, so two - concurrently-analyzing threads' truncates free each other's live pins — a - knowingly-shipped race under landed Phase-15 concurrency, plus a - meaning-membrane violation (embedder-pin vs transient). Rejected. -- **Alt B (finished-form-clean, ADOPTED)** — the AnalysisFrame chain as - described, with the assert-on-unbracketed-producer property converting the - seam-enumeration failure mode (the same enumeration failure that left - `macro_root_slot` unwired for its whole life) into a mechanical invariant. -- **Alt C (wildcard)** — no GC allocation at analysis time at all: keep - literals in the arena and materialize on first `op_const` under the - already-published EvalFrame root. Eliminates the bug class and pre-pays - moving-GC checklist item 4 ("literal-copy-to-heap"), but makes the constant - pool dual-representation, moves data-reader/`#inst` errors from load-time - to first-eval-time (an F-011 behavioural-equivalence risk needing its own - audit), and still needs half of Alt B for the macro window. Recorded on - `gc_rooting.md`'s moving-GC checklist as the noted option, not adopted. -- **Simpler alternatives rejected**: deferring auto-collect during analysis - (an `analysis_depth` gate) is the cw v0 `suppressCollection` escape hatch - F-006's history refuses — macro expansion legitimately needs collects and - torture-mode stays red; extending ADR-0150 fabrication regions is correct - only for pure-Zig no-reentry code, which analysis is not. -- **Costs verified**: double-rooting is O(1) per already-marked object at - mark time; retention is bounded by the per-form bracket (same class as - `EvalFrame.constants` during execution); the bootstrap loads core one form - at a time so the registry peaks at one form's literal count. - -## Consequences - -- The D-430 corruption family (garbage-symbol NameError / recycled-cell - mis-dispatch / D-531 OOB-16 panic class) is closed at the root: no - analysis/compile/deserialize-time Value is ever collect-exposed. -- A new driver seam that forgets its bracket fails loudly (assert) on the - first literal it analyzes in a safe build, instead of corrupting under - load. -- Worker threads that analyze (eval on a future/agent) publish their chain - via `analysis_frame_slot`; workers that never analyze pay one static null. -- STW-only state like the gray worklist: collects cannot run concurrently - with a mutating chain (safepoint parks), and the walker reads `items.len` - at walk time. -- `macro_root_slot` is gone; ADR-0028 §5 row 7's slot is subsumed (recorded - there via this ADR's cross-reference). - -## Affected files - -`src/runtime/gc/root_set.zig` (frame + walker + ThreadGcContext; macro slot -removed), `src/eval/analyzer/analyzer.zig` (`makeConstant`), -`src/eval/analyzer/special_forms.zig` (`analyzeQuote`), -`src/eval/backend/vm/compiler.zig` (`addConstant`), -`src/eval/bytecode/serialize.zig` (`deserializeChunk`), -`src/eval/macro_dispatch.zig` (`expandIfMacro`), `src/eval/driver.zig`, -`src/app/repl.zig`, `src/app/nrepl.zig`, `src/app/builder.zig`, -`src/eval/evaluator.zig`, worker registration sites -(`future.zig` / `agent.zig` / `safepoint.zig` / `vm.zig`), -`.dev/gc_rooting.md`, `test/e2e/phase16_gc_torture.sh` -(`analysis_const_root`). - -## Verification residuals (landed in the same arc) - -Driving the instaparse repro to a deterministic state after the frame landed -exposed two SIBLING unrooted classes (not analysis-window bugs, but the same -"GC Value reachable only through a non-GC container" shape): - -1. **formToValue builder accumulators** (`vectorFormToValue` / - `mapFormToValue` / `setFormToValue` / `listFormToValue` / the `form.meta` - branch): `out`/`acc` + in-flight elements lived in Zig locals across - per-element allocs — alloc-torture panicked "@memcpy arguments alias" - (the swept vector tail was recycled AS the new tail). The Value→Form - direction (D-253) had frames; the Form→Value direction did not. Fixed - with the same §C EvalFrame idiom (bracket-independent — formToValue also - runs at runtime under `read-string`, outside any analysis frame). -2. **`TypeDescriptor` values had no GC trace** (the gc_rooting §C **C8** - KNOWN-OPEN): the descriptor struct is gpa-owned and never swept, but its - `method_table[].method_val` (deftype/reify method Functions — often - reachable ONLY through the descriptor) and `meta` are ordinary GC - objects. Past the 4MB collect threshold they were swept — instaparse's - `(set! cached-seq …)` then read its field-name chunk-constant as garbage. - Fixed: `.type_descriptor` tag trace (the trackHeap'd ref is a persistent - mark-waypoint, so the trace re-runs every collect) + `markDescriptorValues` - from both instance traces (typed + reified, parent chain included). - -After all three land, the instaparse load/parse is byte-deterministic across -runs (the remaining failure is a plain parity gap at instaparse -`core.cljc:361`, tracked in D-430). - -## Revision history - -- 2026-07-02: Status: Proposed → Accepted (autonomous-loop self-accept; DA - fork reflected into Alternatives considered; Alt B adopted over the - draft's watermark registry after the DA surfaced the six escaped seams + - the deserializeChunk producer). Same-day: § Verification residuals added — - the two sibling unrooted classes (formToValue builders; TypeDescriptor - method-table trace = C8's root cause) found while verifying, fixed in the - same source commit. diff --git a/.dev/decisions/0170_nrepl_rearchitecture_cider_fidelity.md b/.dev/decisions/0170_nrepl_rearchitecture_cider_fidelity.md deleted file mode 100644 index 734a8ca9e..000000000 --- a/.dev/decisions/0170_nrepl_rearchitecture_cider_fidelity.md +++ /dev/null @@ -1,320 +0,0 @@ -# ADR-0170: nREPL server re-architecture — base-protocol fidelity (CIDER end-to-end) - -- **Status**: Accepted -- **Date**: 2026-07-13 -- **Related**: ADR-0015 am2 (F142 re-introduction), ADR-0048 (nREPL state chart), - ADR-0138 (`*out*` capture via binding frame), ADR-0019 (exit/error policy), - D-117 (nREPL polish), D-118 (out/err capture), F-002 / F-009 / F-011 / F-014 - -## Context - -The row-14.10 minimal nREPL server (`src/app/nrepl.zig`, one file, 7 ops) -predates most of the runtime's current capabilities. A CIDER field report -(REPL buffer RET dead, no completion, bare `NameError` errors) triggered an -empirical probe of the live server (two independent bencode replay drivers, -2026-07-13). Confirmed defects: - -1. **Read-loop stranding**: the session loop does one `fillMore()` → - one `decode` → one op per iteration, so it *blocks for new socket - bytes even when complete bencode dicts already sit buffered*. - Pipelined requests strand off-by-one (probe: of 2 requests written - in one TCP segment, the second is answered only after a *third* - write arrives). CIDER overlaps requests during connect (init eval + - capability probes), so the init eval's `done` strands → the REPL - prompt marker never renders → RET inserts a newline forever. - `C-c C-e` works because it is a lone in-flight request. -2. **4 KiB frame ceiling**: `decode` sees at most the fixed 4096-byte - stream buffer; a larger message can never decode → `catch break` - closes the connection (probe: 10.6 KiB eval → connection reset). - `C-c C-k` (load-file) on any real file kills the connection. -3. **Session identity broken**: every `clone` returns the *same* - connection-constant id, and every reply stamps that constant - instead of echoing the request's `session`. CIDER clones two - sessions (main + tooling) over one socket and relies on distinct - ids (nrepl-client.el:619-639; the CIDER mock server echoes the - request session and mints a fresh `new-session` per clone). -4. **Error replies malformed × 4**: (a) `err` text is the raw Zig - `@errorName` (`"NameError"`) instead of the rich caret rendering - the CLI already produces via `error_render.zig`; (b) `err` and - `status ["error" "eval-error" "done"]` ride ONE dict, which - CIDER's mutually-exclusive response `cond` mis-routes - (nrepl-client.el:744-770); (c) a second bare `done` follows - (double-done per request); (d) evaluation *continues* past a - failing form (JVM nREPL stops). -5. **Missing ops**: no `completions`/`complete` (CIDER gates - completion on these being advertised — cider-completion.el:136-142), - no `lookup`/`info`/`eldoc` (eldoc + doc popups dead). -6. **describe lies**: `versions` hardcodes `"0.1.0-pre"` (real version - = `build_options.version` = the `build.zig.zon` value, currently - 1.1.0); ops list is hand-maintained separately from the dispatch - chain (drift by construction). - -The runtime meanwhile grew the exact capabilities the server lacks: -`error_render.zig` (rich caret rendering, EDN mode "for CIDER / -editors" per its own doc comment), `line_editor.zig::handleTab` -(Env-wide completion-candidate enumeration), `Var.meta` -(`:arglists`/`:doc` for `clojure.repl/doc`). The single-file server -simply never re-consumed them — the textbook early-asset inertia -F-002 exists to overrule. - -**Reference precedent**: babashka.nrepl (`~/Documents/OSS/babashka.nrepl`) -is the JVM-less nREPL that drives CIDER fully. Its load-bearing idioms: -a single `response-for` choke point that stamps *every* reply with the -request's `session`+`id`; sessions as plain fresh-UUID tags; op -multimethod with `:default` → `unknown-op`; the 3-message error -protocol (`err` → `ex`/`root-ex`/`eval-error` → `done`); `completions` -+ `lookup`/`info`/`eldoc` built from interpreter introspection. Its -compromises (no stdin op, no mid-eval interrupt, no cider-nrepl -extension middleware) are CIDER-graceful and adopted here. - -## Decision - -Rewrite the server as a package under `src/app/nrepl/` (precedent: -`src/app/repl/` for the line editor), with one shared introspection -module extracted to the runtime layer: - -| File | Responsibility | -|----------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `src/app/nrepl.zig` | Thin entry: bootstrap + `.nrepl-port` + sequential accept loop (concurrency model unchanged; CIDER uses one socket). | -| `src/app/nrepl/transport.zig` | Framing + reply choke point. Growable receive buffer; decode **all** buffered messages before blocking (`bencode.DecodeError.UnexpectedEof` = truncated → wait for bytes; other decode errors = protocol error → close); message size cap. Every reply passes through `responseFor`, which echoes the request's `session` + `id` — ops cannot mis-stamp. | -| `src/app/nrepl/session.zig` | Registry: `clone` → fresh unique id; `close`; `ls-sessions`. Per-session state: current-ns name + `*1`/`*2`/`*3`/`*e` Values, registered as permanent GC roots (`.dev/gc_rooting.md` discipline) and released on close. | -| `src/app/nrepl/ops.zig` | `comptime` StaticStringMap op → handler. `describe` **derives** its ops list from the same table (drift impossible) and reports `versions` from `build_options.version`. Unknown op → `status ["error" "unknown-op" "done"]`. | -| `src/app/nrepl/eval.zig` | Eval engine: per-form read/analyze/eval; honors the request `ns` key (binds the session ns; `status ["namespace-not-found" "done" "error"]` on miss); per-form `*out*` capture (ADR-0138 shape); updates `*1`/`*2`/`*3`/`*e`; **stops on first error** (JVM parity); errors render via `error_render` into `err`, then `{ex, root-ex, status [eval-error]}`, then `done` — three separate dicts, exactly one `done` per request. | -| `src/app/eval_session.zig` | **Shared REPL eval engine** (DA deepening (a)): per-form read/analyze/eval with the D-430 AnalysisFrame bracket, `*out*` **and `*err*`** capture, `*1`/`*2`/`*3`/`*e` rotation over the interned dynamic Vars, stop-on-first-error, error rendering through `error_render` — driven through caller-supplied sinks. Consumers: `nrepl/eval.zig` (bencode sinks) and `repl.zig` (terminal sinks — the CLI REPL gains `*1`..`*e` in the same cycle; duplicating the rotation would be the F-011 violation the DA flagged). | -| `src/runtime/introspect.zig` | Shared introspection: completion-candidate enumeration (extracted from `line_editor.zig::handleTab`'s collect* fns — same-layer move, Env is layer 0), returning **typed** candidates `{name, ns?, kind}` (fn/macro/var/namespace — the completions op needs the type; the line editor ignores it) + var-meta lookup (`:arglists`/`:doc` + macro flag from `Var`). Consumers: line editor (candidate cap 64), nREPL `completions`/`lookup`/`info`/`eldoc`, future `--list-vars`. | - -**Op surface** (all advertised by the derived `describe`): `clone`, -`close`, `describe`, `eval`, `load-file`, `interrupt`, `ls-sessions`, -`completions`, `complete`, `lookup`, `info`, `eldoc`. -Response shapes mirror babashka.nrepl (CIDER-proven): -- `completions`/`complete` → `{completions: [{candidate, type, ns?}…], status [done]}` - (`type` from Var flags: macro / function / var; namespace candidates → namespace). -- `info` → flat `{ns, name, arglists-str, doc?, file?, line?}` + done; - `lookup` → the same map nested under `info`; `eldoc` → - `{ns, name, eldoc [[arg…]…], type, docstring?}` + done, `no-eldoc` - status on miss. - -**Core surface addition**: `*1` / `*2` / `*3` / `*e` are interned in -`clojure.core` as dynamic vars in the upstream core.clj shape (JVM -parity — today they don't resolve at all, which is itself a gap; -interning is also what makes `(inc *1)` *analyzable*, so a -session-struct-only shape would silently not resolve). Both REPLs -bind and rotate them via `eval_session.zig` in this cycle. - -**Session state rooting**: each session's held Values (`*1`..`*e`) -are rooted via `GcHeap.pin`/`unpin` (the existing `permanent_roots` -embedder surface — no new root class), swapped on rotation and -released on session close; the `GC-ROOT:` marker + `.dev/gc_rooting.md` -row land in the same commit per that SSOT's discipline. - -**Transport memory contract** (DA finding #5): the pre-ADR server fed -every decode/encode/print into the process-lifetime arena — unbounded -growth over a long editor session. `transport.zig` owns a per-message -scratch arena (bencode decode + reply encode + value-print buffers, -reset after each reply flushes); reader forms / analysis nodes stay on -the persistent node arena as today (they can be referenced by -persisted defs — F-006's allocator layering). The bencode codec -needs NO change: `DecodeError.UnexpectedEof` fires only on truncation -(→ buffer more, up to the size cap), other decode errors are protocol -errors (→ close). The size cap is load-bearing: a huge declared -string length (`999999999:…`) reads as truncation forever without it. - -**describe versions**: `{cljw: , nrepl: -{version-string: "1.3.1"}}` — deliberately **no** `clojure` key. -This is the exact babashka shape (its describe payload is -`{"versions" {"babashka" , "babashka.nrepl" }}`, no -`clojure` key). CIDER keys its runtime detection on -`versions.clojure`; claiming it flips CIDER to 'clojure runtime and -triggers cider-nrepl middleware-missing warnings on every connect. -'generic runtime (like let-go before its upstream CIDER patch) -degrades gracefully while all op-gated features (completion, eldoc) -work. Upstreaming a `cljw` runtime key to CIDER is a -separately-tracked idea, not a blocker. - -## Principled compromises (recorded, babashka-matching) - -| Compromise | Rationale | Tracking | -|---|---|---| -| No `stdin` op | single-threaded `read-line` semantics need the concurrency area | D-117 residual | -| No mid-eval `interrupt` (acks `done`+`session-idle`) | a hung eval owns the only thread; true interrupt = thread-per-session (gap area I) | D-117 (a) | -| No cider-nrepl extension middleware (debugger/inspector/test-runner/apropos ops) | separate ~15 kLOC JVM-introspection project; CIDER degrades gracefully | out of scope (F-014 goal line) | -| `out`/`err` streamed per top-level form, not per write | **A divergence FROM babashka** (bb's proxy Writer pushes an `out` response per write, mid-eval — the DA corrected the draft's "bb-proven" claim). ADR-0138 capture shape; user-visible symptom: a long-running form's prints arrive only when the form returns | D-118 re-narrowed to this | - -## Consequences - -- CIDER works end-to-end against the base protocol: REPL buffer - evaluation, streaming out, completion, eldoc, doc lookup, load-file - of real-size files, rich error text identical to the CLI. -- `test/e2e/phase14_nrepl.sh` grows CIDER-replay coverage: distinct - clone ids, session echo, pipelined burst, >4 KiB frames, the error - triple, completions/lookup shapes, describe derivation. The old - single-session happy-path e2e was green while CIDER was broken — - the proxy-test lesson (memory `verify_actual_pattern_not_proxy`). -- The CLI REPL gains `*1`/`*2`/`*3`/`*e` in the same cycle via the - shared `eval_session.zig` engine (the F-011 dividend that proves the - extraction earned its keep). -- D-117 is re-narrowed by code-truth (its "CIDER ops LANDED" claim was - false — only load-file had landed); D-118's remaining gap narrows to - per-write streaming. -- `structure_plan.md`'s single-file `nrepl_server.zig` note and the - `runtime/cljw/repl/NReplServer.zig` reservation are superseded by - this package shape (reservations are memos, F-002); if a - `(cljw.repl/start-server!)` Clojure surface ever lands, the - app-layer engine is reachable via the `driver.installVTable` - precedent — no code now. -- `line_editor.zig` sheds its private candidate enumeration for the - shared `runtime/introspect.zig` (behaviour-preserving extraction, - cap 64 kept at the caller). -- The stale `"0.1.0-pre"` version lie is structurally impossible - (single source: `build_options.version`). - -## Alternatives considered - -(Devil's-advocate subagent output, fresh-context fork 2026-07-13, -reflected per CLAUDE.md § ADR-level designs. Its leading finding: no -F-NNN blocks any shape. Its fact-checks — the draft's "per-form out is -bb-proven" claim was FALSE (bb streams per write), `*err*` capture was -missing, the "runs before bootstrap" argument for Zig-side introspection -was factually wrong (the server is always post-bootstrap; the real -argument is the keystroke path + F-011), the arena-growth gap, and the -bencode-already-distinguishes-truncation observation — are folded into -Context/Decision above.) - -**Alternative 1 — smallest-diff (fix-in-place, one file).** Keep -`nrepl.zig` single-file (~900 LOC): growable buffer, per-clone counter, -3-message errors, stop-on-error, `build_options.version`, completions -via `pub`-ing `line_editor`'s collect fns. Better: zero structural -churn, each fix independently verifiable. Breaks: sharing completion -logic by exposing private methods of a raw-terminal line editor is the -ad-hoc coupling F-011 forbids; an app-layer home forecloses any future -Layer-2 surface; the if-else op chain keeps the describe-drift class. -DA verdict: "reaches a different, worse finished form — disqualified -under F-002 since the probe evidence shows five orthogonal concern -clusters (framing, sessions, ops, eval, introspection)". - -**Alternative 2 — finished-form-clean (draft + four deepenings) — -RECOMMENDED and ADOPTED.** (a) Extract the shared eval-session core -NOW: once `*1`..`*e` are interned dynamic Vars, the rotation logic is -identical for both REPLs; deferring it is "the Cycle-budget-defer smell -wearing a 'correct sizing' costume" — nREPL-only `*1` (works in CIDER, -absent in `cljw repl`) is indefensible under F-011. (b) Session state -rooted via the existing pin surface + gc_rooting.md row, not a new root -class. (c) `runtime/introspect.zig` with typed candidates; cite F-011 -commonization + the zone graph (NOT F-009, whose out-of-scope clause -covers env.zig). (d) Transport scratch-arena split + `*err*` capture + -honest per-form-out divergence row. Cost: substantially larger diff -(repl.zig + bootstrap + gc_rooting.md ride along) — recommended anyway -per F-002; the risk (eval-extraction disturbing the AnalysisFrame -bracket placement, D-556 class) is covered by the existing smoke + -phase14 e2e. - -**Alternative 3 — wildcard: Clojure-side ops (the literal bb -transplant).** Implement completions/lookup/info/eldoc in `.clj` -(bb's `impl/sci.clj` shape) — maximum Clojure-level commonization, -dogfoods cljw, no new Zig surface. Breaks: the line editor's TAB path -would either eval per keystroke inside a raw-mode terminal (GC + -output-capture + error isolation on the keystroke path) or keep its -Zig walk — recreating the duplication across a language boundary where -drift is harder to see; needs a generic Value→bencode walker; a throw -inside a completions eval must not rotate the session's `*e`. DA -rating: "the right *eventual* direction for lookup-class ops -(data-shaped, latency-tolerant), wrong today for keystroke-shared -completion enumeration" — recorded as a forward note: lookup-class ops -MAY migrate to `.clj` when a `cljw.repl` ns lands (F-014 clause 4 -would accept a `(cljw.repl/start-server!)` surface; the app-layer -engine would then be reached via the `driver.installVTable` precedent). - -Judgment-call stress-tests adopted from the DA: transport owns the -session/id echo with the clone edge (echo the parent session, emit the -fresh `new-session`); the op table + describe derive from ONE comptime -tuple (StaticStringMap does not iterate cleanly); aliases are separate -entries pointing at one handler so describe advertises them all; -`interrupt` acks `["done" "session-idle"]` when idle (spec) instead of -bare done; per-session state is kept (JVM nREPL sessions are per-session -binding maps — the F-011 oracle; bb's global `set!` across id-tags is -bb's simplification and would let tooling-session evals rotate the -user's `*1`). - -## Amendment 1 — `cider/analyze-last-stacktrace` (the `*cider-error*` buffer) - -**Context** (user field report, 2026-07-14): CIDER's `*cider-error*` -buffer renders "at [no stack trace available]" against cljw. CIDER's -error handler is a 3-way decision (cider-compilation.el -`cider-default-err-handler`): the op `cider/analyze-last-stacktrace` -when advertised → the rich numbered-causes buffer with Show/Hide frame -filters; else a `clojure.stacktrace` eval; else the bare print -fallback cljw currently hits. The op's wire contract (cider-nrepl -middleware/stacktrace.clj + orchard/stacktrace.clj + the -cider-stacktrace.el renderer's `nrepl-dbind-response` bindings): - -- One response dict per CAUSE: `class` (string), `message`, optional - `phase` (clojure.error phase name), optional `data` (printed - ex-data), `stacktrace` = list of frame dicts; then `{status [done]}`. - No last error → `{status [no-error]}` then `{status [done]}`. -- Frame keys the renderer binds: `name` `ns` `fn` `file` `file-url` - `line` `class` `method` `var` `flags`; `flags` (strings) drive the - Show/Hide filters (`clj` / `java` / `repl` / `tooling` / `project` / - `dup`). - -**Decision (DA Alt 2 — `*e` is the single channel).** cljw implements -the op natively (no cider-nrepl), and the op **analyzes the session's -`*e` on demand** — cider-nrepl's own model (`(@session #'*e)`) — rather -than keeping a parallel capture channel: - -- **`*e` is set for EVERY caught REPL error, catalog errors included** - — closing a real F-011 gap: JVM `clojure.main/repl` sets `*e` for - compiler errors too, while cljw only did for thrown Values. The - engine materializes the peeked catalog Info (the existing - `peekLastError()` — the VM catch path's `allocExceptionLoc` shape) - into a GC-owned exception Value stamped with the error `phase` - (a new byte on `ExInfo`), and `stars.setE`s it. Covers the reader- - error branch too. Alloc failure leaves `*e` unset (honest - no-error degradation). `(ex-message *e)` after `hoge` now works in - BOTH REPLs (e2e-pinned). -- **`throw` stamps the live call stack** onto a trace-less exception - (`ex_info.stampTraceIfAbsent`, both backends symmetric) so a user - `(throw (ex-info …))` carries frames exactly like a catalog raise. -- The op walks `*e`'s `ex-cause` chain: one response dict per cause — - `class` (the stored host-class name, e.g. "RuntimeException" for a - name error — exactly what the JVM shows for unresolved symbols), - `message`, `phase` (mapped: `parse`→`read-source`, - `analysis`→`compile-syntax-check`, `macroexpand`→`macroexpansion`, - `eval`→`execution` — drives CIDER's buffer-vs-overlay routing), - `data` (printed at op time), `stacktrace` frames innermost-first. -- Flags, cljw's take: every frame `clj` (there is no Java); `repl` - for `<…>`-labelled sources; `project` + `file-url` for on-disk - paths (navigable); `dup` per orchard's rule (same fn, or same - REAL file+line). `java` / `tooling` never fire — the filter - buttons degrade gracefully in CIDER. -- The bare `analyze-last-stacktrace` alias is advertised too. -- `inspect-last-exception` (the inspector) stays out of scope — and - the `*e`-Value model (vs string snapshots) deliberately keeps that - door open. -- **Fixed en route**: the re-architecture's `evalImpl` fed the - SCRATCH-owned request `code` into evals whose products (defn'd - fns, forms) borrow source slices and outlive the message — a - use-after-free surfacing as garbage strings under scratch-reuse - pressure. The source is now duped onto the persistent arena - (mirroring the CLI REPL's per-line dupe). - -**Amendment alternatives considered** (fresh-context DA fork, -2026-07-14): *Alt 1 smallest-diff* — sink-side gpa string-snapshot -capture into `session.last_error`; rejected: fossilizes the `*e` -F-011 divergence behind a second truth channel, forecloses the -inspector, adds a free-on-close lifecycle and a two-channels-drift -class. *Alt 2 (adopted)* — as above; the DA's factual corrections -(peekLastError already existed; the reader branch bypassed -handleEvalError; frame `name` is display-inert but `file-url`/`var` -drive navigation; `phase` is routing, not decoration; the no-error -shape is CIDER-safe byte-parity) are folded in. *Alt 3 wildcard* — -implement the analysis in bootstrap `.clj` over `*e` + small -primitives; right eventual direction (the ADR's existing lookup-op -forward note), premature while the Value→bencode walker doesn't -exist. - -## Revision history - -- 2026-07-13: Proposed. -- 2026-07-14: Amendment 1 — `cider/analyze-last-stacktrace` op + - per-session structured last-error capture (the `*cider-error*` - buffer); DA-fork alternatives reflected below. diff --git a/.dev/decisions/0171_rt_ns_merge_into_clojure_core.md b/.dev/decisions/0171_rt_ns_merge_into_clojure_core.md deleted file mode 100644 index 3329eeec4..000000000 --- a/.dev/decisions/0171_rt_ns_merge_into_clojure_core.md +++ /dev/null @@ -1,163 +0,0 @@ -# ADR-0171: Merge the `rt` kernel namespace into `clojure.core`; internals move to `cljw.internal` - -- **Status**: Accepted (2026-07-15) -- **Driver**: CIDER completion-parity campaign (user-directed 2026-07-15); - survey `private/notes/gapT-completion-parity-survey.md`; oracle baseline - 5/18 probes matching (`scripts/completion_oracle.py --diff`). - -## Context - -Zig-implemented builtin vars intern into an internal `rt` namespace and are -referred (Var-sharing) into `user` / every entered ns; bootstrap Zig macros -home there too. The home ns stays `rt`, which leaks into every observable -introspection surface, diverging from mainline Clojure (F-011 violation): - -- `(pr-str (resolve '+))` → `#'rt/+` (mainline: `#'clojure.core/+`); same - for `doc`/`meta` `:ns`, eldoc/lookup/completion `ns` fields, callable - printing, and error messages. -- nREPL completion candidates annotate `(rt)` where mainline shows - `(clojure.core)` — the user-visible trigger for this campaign. -- `(ns-publics 'clojure.core)` is missing every Zig builtin (they are - refers, not mappings). -- `(find-ns 'rt)` / `(all-ns)` expose an undocumented ns; `(in-ns 'rt)` - can enter and clobber the kernel. -- Structural scar tissue exists only because of the split: - `Namespace.resolveQualified`'s hard-coded clojure.core refers-exception - (env.zig), `referAllOverriding`, the per-ns rt refer in both backends, - and the per-site display hack `syntax_quote.zig` carries (rt → - clojure.core rename during syntax-quote resolution). - -## Decision - -**Full merge** (the Devil's-advocate fork's Alt 2 — chosen over the -lighter "re-home publics but keep `rt` alive" shape): - -1. Public Zig builtins and bootstrap Zig macros intern with **home ns = - `clojure.core`** at registration. -2. `__`-prefixed (and `-`-prefixed) internal vars move to a new, documented - **`cljw.internal`** namespace (consistent with the owned `cljw.*` - surface; its presence in `(all-ns)` is a recorded, documented fact - rather than an undocumented leak). -3. The `rt` namespace is **deleted**: `Env.init` no longer creates it, the - boot referAll and the per-ns rt refer in both backends go away, and the - `resolveQualified` clojure.core exception + `referAllOverriding`'s - rt-collision purpose dissolve. -4. The bundled `.clj` `rt/…` qualified references (~48 code sites) migrate - to `cljw.internal/…` (internals) or bare/`clojure.core/…` (publics); - the 12 Zig-side `rt/`-qualified symbol emissions (macro hygiene, - `@deref` lowering, analyzer fallback) emit `clojure.core/…` — exactly - what mainline's expansions emit. -5. Rebind-recursion hazards (a core.clj wrapper that redefines a name - while calling the rt builtin qualified, e.g. `re-find`) are enumerated - mechanically (rt mappings ∩ core.clj re-defs) and those builtins are - renamed to `cljw.internal/__` so the wrapper calls the internal, - not itself. -6. The embedded AOT bootstrap cache regenerates at build (self-consistent). - Externally persisted `cljw build` archives from before this change bake - `"rt"` into var_ref constants and cannot resolve it once `rt` is gone: - the serializer format version is bumped so old archives fail with the - explicit version message, not a resolution error. - -## Consequences - -- `#'clojure.core/+`, `(:ns (meta #'+))`, `ns-publics`, completion/lookup/ - eldoc ns fields, callable printing all match mainline **at the data - layer** — no per-surface normalization, no whack-a-mole (F-013). -- syntax_quote's per-site rt→clojure.core rename is deleted. -- Future introspection surfaces (`cljw --list-vars`, new nREPL ops) are - correct by construction. -- `cljw.internal` is user-visible in `(all-ns)`; completion hides its - `__` vars (the campaign's hidden-filter); an AD row records the - divergence "mainline has no cljw.internal". -- One-time migration cost: ~60 sites + 7 test files + serializer version - bump. Accepted per F-002 (finished-form wins; diff size is not a - constraint). - -## Alternatives considered (Devil's-advocate fork output, verbatim) - -### Alternative 1 — smallest-diff: shared display normalization - -One `introspect.publicVarNs(v)` (rt→"clojure.core") consulted by -completion/eldoc/lookup (introspect.zig:68; ops.zig:347,363), `printVarRef` -(print.zig:1318), the callable printer (print.zig:609/1039 via -`indexBuiltinNames`, primitive.zig:175), and syntax-quote (deleting the -syntax_quote.zig:95 per-site hack); plus a `__`-prefix filter in -`emitVarMap`. - -- Better than the full merge: zero bootstrap-order risk; zero `.clj` - churn; no AOT interaction; no rebind hazards; lands in one cycle. -- Breaks / fails to fix: it is display-layer lying — the data still says - `rt`. `(ns-name (:ns (meta (resolve '+))))` → `rt` unless var-meta is - faked too (another site); `(ns-publics 'clojure.core)` still missing - builtins; `(find-ns 'rt)`, `(all-ns)`, `(ns-resolve 'rt '+)`, - `(in-ns 'rt)` clobber hazard all remain. Every future introspection - surface must remember to call the normalizer — the exact per-site - whack-a-mole F-013 forbids, and it does not reach the same finished - form, so F-002's smallest-diff tiebreaker does not apply. Rejected as - terminal shape. - -### Alternative 2 — finished-form-clean: FULL MERGE, `rt` eliminated (ADOPTED) - -Intern public Zig builtins + bootstrap macros with home = `clojure.core` -at registration (primitive.zig:71-111 target swap; macro_transforms.zig:48); -move `__`/`-`-prefixed internals to a real, documented `cljw.internal` ns. -Delete `rt` entirely: Env.init (env.zig:390), the boot referAll -(primitive.zig:169), the per-ns rt refer in both backends (vm.zig:1217,1247; -tree_walk.zig:578), and the `resolveQualified` clojure.core exception -(env.zig:222-228) — it dissolves, as does `referAllOverriding`'s -rt-collision raison d'être and `cacheArithIntrinsics`' rt fallback -(bootstrap.zig:445-451). Migrate the ~48 `.clj` sites and 12 Zig emission -sites (`rt/or` → `clojure.core/or`, `rt/deref` → `clojure.core/deref` — -exactly what clj's own expansions emit). - -- Better than keep-rt-alive (b): (b) keeps `(find-ns 'rt)`, `(all-ns)`, - `(ns-resolve 'rt '+)`, `(in-ns 'rt)` as observable divergences — the - F-011 violation is only ~90% closed, and the env.zig:222 special case - must stay. Full merge closes the class, matches clj's actual shape, - fixes `ns-publics` of clojure.core as a free side effect, and deletes - ~4 pieces of structural scar tissue instead of preserving them. -- Breaks (concrete): (i) rebind-recursion hazard — core.clj wrappers that - redefine a name while calling the rt builtin qualified (`re-find`, - core.clj:2145) would rebind the same Var and recurse; enumerate - mechanically (diff rt mappings vs post-bootstrap clojure.core mappings) - and rename those to `cljw.internal/__…`. (ii) Tests: env.zig:919-968, - primitive.zig:183-200, protocol.zig:957, core.zig:2033, math.zig:1391, - macro_transforms.zig:4116, root_set.zig:891 — mechanical updates. - (iii) Bootstrap order unchanged (registerAll already receives - `clojure_core_ns`; clojure.core exists from Env.init). (iv) AOT: - serialize.zig:295-300 bakes `var_ptr.ns.name` into var_ref constants; - the embedded cache regenerates at build; external pre-change archives - need a format-version bump (currently 3, serialize.zig:10). (v) - placement.yaml Pattern B2 already interns private leaves directly into - clojure.core — the merge generalizes the existing pattern. Diff is - large (~60 sites + ledger/docs) — per F-002 not a counter-argument. - -### Alternative 3 — wildcard: Env-level double-key ns aliasing - -Register `"rt"` as a second `env.namespaces` key pointing at the same -`*Namespace` as `"clojure.core"`, and swap the registration target. - -- Better: cheapest route to home-ns = "clojure.core" everywhere; all - 48+12 qualified refs keep resolving. -- Breaks: `Namespace.deinit` double-frees under two owning keys (needs an - alias-aware ownership flag); `(find-ns 'rt)` returns clojure.core (a - new divergence); `__` internals become clojure.core-homed → pollute - `(ns-publics 'clojure.core)` and completion worse than today; the - `re-find` recursion hazard fires immediately. A trap dressed as a - shortcut — also the reason (b)'s "alias ns" sub-option is distrusted. - -## Affected files (planned) - -`src/runtime/env.zig`, `src/lang/primitive.zig`, -`src/lang/macro_transforms.zig`, `src/lang/bootstrap.zig`, -`src/eval/backend/vm.zig`, `src/eval/backend/tree_walk.zig`, -`src/eval/analyzer/{analyzer,syntax_quote}.zig`, `src/eval/reader.zig`, -`src/app/builder/serialize.zig` (version bump), -`src/lang/clj/clojure/core.clj` + `java/io.clj` + `instant.clj` -(rt/ → cljw.internal/ migration), affected unit tests, -`data/placement.yaml` (Pattern B wording), `.dev/accepted_divergences.yaml` -(cljw.internal AD row). - -## Revision history - -- 2026-07-15: Accepted (DA fork output embedded above). diff --git a/.dev/decisions/0172_binary_size_budget_and_ledger.md b/.dev/decisions/0172_binary_size_budget_and_ledger.md deleted file mode 100644 index 0dd934e0a..000000000 --- a/.dev/decisions/0172_binary_size_budget_and_ledger.md +++ /dev/null @@ -1,472 +0,0 @@ -# ADR-0172 — Binary size: measured composition, per-component budget, lever ledger, and a size-claims gate (the binary-size SSOT) - -- **Status**: Proposed → Accepted (2026-07-16; user-directed investigation; DA - fork folded verbatim below) -- **Driven by**: user report (2026-07-16): the brew-installed cljw shows ~9.5-9.8 - MB while README still claimed "about 3.8 MB" — "バイナリサイズが小さい、と名乗れ - なくなってきた". The user asked for a root-level investigation (build flags, - Clojure/JVM/Zig angles), self-critiqued options, a concrete ADR, and a - consolidation of every scattered binary-size mention (debt rows + ADR - fragments) into one place. -- **Relates to**: D-515 (binary-size debt axis — re-narrowed to point here), - D-277 (eager-comprehensiveness-vs-size standing tension), D-517 / ADR-0162 / - ADR-0163 (envelope format + lazy-ns — the embedded-data lever folds into - D-517), ADR-0158 (single-binary component embedding), ADR-0132 (ReleaseSafe = - gated/shipped config), O-008 (release strip, `.dev/optimizations.md`), - F-001 (zwasm unavoidable), F-002 (finished-form wins), F-011 (behavioural - equivalence), F-013 clause 4 (comprehensiveness bounded by single-binary - cost), F-014 (scope goal line). - -## Context - -### The incident, and why the old targets died - -The shipped macOS arm64 binary (v1.3.1, ReleaseSafe, `-Dwasm`, stripped) is -**9,469,816 bytes (~9.5 MB decimal)**. Four tracked documents disagreed with -that reality and with each other: - -| Location | Stale claim | -|----------------------------------|-----------------------------------------| -| `README.md` (Features) | "about 3.8 MB" | -| `ROADMAP.md` §"No JVM" top-line | "target binary ≤ 5 MB" | -| `ROADMAP.md` §10.3 targets | "< 3.5 MB target / < 2 MB stretch" | -| `.dev/debt.yaml` D-515 | "8.8 MB (incl. a ~1.4 MB … blob)" | -| `test/run_all.sh` comment | "ReleaseSafe ≈ 3.5 MB … exceeds 8 MB" | - -The pre-engine targets were not merely missed — they became **arithmetically -unreachable the day F-001's engine landed in the text segment**: the embedded -zwasm engine + component-model API alone is ~3.0 MB of code, i.e. F-001's -mandatory payload exceeds the old §10.3 target by itself. The binary grew -because the differentiator (Wasm FFI with an embedded JIT) and the -comprehensiveness laws (F-013/F-014: full-Unicode UCD tables, spec.alpha -bundled, CIDER-fidelity nREPL, TLS-capable http client) landed. The growth is -a consequence of the mission; the rot was that no mechanism compared any prose -claim against the measured artifact. Both problems are fixed here: honest -numbers derived from measurement, and a gate that keeps them honest. - -### Measured composition (2026-07-16, macOS arm64, v1.3.1 config) - -Total 9,469,816 B. Method: `scripts/binary_size_report.sh` (this ADR's tool) -+ symbol attribution on a `-Dprofile=true` build. - -| Component | Size | Notes | -|----------------------------------------|----------|----------------------------------------------------------------------------------| -| `__text` (code) | 6.90 MB | breakdown below | -| — zwasm (engine/api/validate/ir/…) | 2.99 MB | 44% of code; `engine.codegen.arm64.emit.compile` alone 691 KB | -| — cljw proper (runtime/lang/eval/app) | 2.57 MB | 38% of code | -| — Zig std | 1.17 MB | incl. std.crypto TLS 0.43 MB (https client), std.sort monomorphizations 0.22 MB | -| `__const` + `__cstring` (data) | 1.59 MB | AOT bytecode blob 698 KB + raw `.clj` sources 451 KB + UCD tables ~207 KB + misc | -| `__eh_frame` + `__unwind_info` | 0.75 MB | unwind tables | -| headers / `__LINKEDIT` / misc | ~0.23 MB | | - -Supporting facts: the whole binary gzips to 2.8 MB (the release tarball) — -high code redundancy, typical of monomorphized Zig; symbols are already -stripped (O-008); `zig build` release config already omits debug info. - -### Optimize-mode measurements (same source, same flags otherwise) - -| Config | Bytes | Δ vs shipped | Trade | -|-------------------------------------|-----------|---------------|---------------------------------------------| -| ReleaseSafe (shipped) | 9,469,816 | — | — | -| ReleaseSafe + `unwind_tables=.none` | 8,731,096 | −739 KB | see L1 (measured cost ≈ zero) | -| ReleaseFast | 8,172,744 | −1.30 MB | drops all safety checks | -| ReleaseSmall | 4,841,624 | −4.63 MB | drops safety checks + speed (unbenchmarked) | - -(`error_tracing=false` was measured a no-op: `strip=true` already disables -error-return tracing — the probe binary prints "stack tracing is disabled" -on panic with and without unwind tables.) - -## Decision - -### 1. This ADR is the binary-size SSOT - -Composition, budget, and lever dispositions live **here**. D-515 is re-narrowed -to the drain anchor pointing at this ledger; the four stale claims above are -corrected in the same commit; `scripts/binary_size_report.sh` is the -measurement recipe (report mode) and the enforcement (check mode). -`bench/history.yaml`'s `binary_size_bytes` remains the bench-side time series; -the user-facing per-release record is CHANGELOG (each release entry states the -measured size per platform from this release on). - -**Reference platform** for every number in this ledger: macOS arm64, -ReleaseSafe, `-Dwasm`, stripped (= the brew artifact). The Linux x86_64 -`-Dcpu=baseline` deploy artifact is recorded at each release alongside it -(expected within ~±15%; the v1.3.1 Linux tarball implies ~10 MB uncompressed). - -### 2. Per-component budget (derived ceiling, not a reverse-engineered line) - -Size growth is feature-coupled (F-013/F-014 guarantee it), so the budget is -per-component with explicit headroom; the total ceiling **derives** from the -components, and a breach localizes itself via the report tool: - -| Component | Measured (2026-07-16) | Budget | Headroom rationale | -|------------------------|-----------------------|------------------------------------|-----------------------------------------------------------------| -| zwasm (engine + api) | 1.94 MB | 2.5 MB | thunk collapse landed (v2.2.1); x86_64 emitter is comptime-gated (0 B on arm64) | -| cljw text | 2.47 MB | 3.5 MB | F-013/F-014 comprehensiveness growth | -| Zig std text | 1.17 MB | 1.5 MB | | -| embedded data | 0.76 MB | 1.0 MB | L2 LANDED 2026-07-16 (ADR-0173 C2'-C5': pool + flate regions + flate .clj) | -| unwind + linkedit etc. | 0.23 MB | 0.3 MB | L1 LANDED 2026-07-16 (O-052): tables dropped, budget re-set | -| **Derived ceiling** | **8.73 MB** | **≈ 11 MB → ≈ 10.3 MB post-L2** | | - -A component crossing its budget line triggers: attribute (report tool) → -either a lever lands or the budget line is consciously amended **in this ADR** -(Revision history entry). Never silently. - -### 3. Size-claims gate (the recurrence-prevention mechanism) - -`test/run_all.sh` gains the `size_claims` step (full gate, after -`build_cljw`): `scripts/binary_size_report.sh --check` fails when README's -headline " MB" claim drifts >10% from the freshly built binary. This is the -house pattern (check_clj_attribution / check_smell_audit): closed structurally, -not by vigilance (F-013 clause 3). A prose size claim can no longer rot 2.5×. - -### 4. Lever ledger (each with a measured/estimated Δ and a disposition) - -- **L1 — `unwind_tables = .none` on release builds: LANDED 2026-07-16 (O-052; - shipped size 9,469,816 → 8,731,096 B).** Original disposition: - Measured −739 KB. Measured cost ≈ zero: the stripped shipped binary - *already* prints no native stack trace on a Zig-level panic (probe - 2026-07-16, "stack tracing is disabled" both ways); cljw renders Clojure - errors from its own StackFrame stack (O-008). arm64-macOS keeps frame - pointers, so an attached debugger can still walk frames post-mortem. - Keep unwind tables on Debug (and under `-Dprofile`). -- **L2 — embedded-data shrink, folded into D-517's envelope-format redesign: - ADOPT-with-D-517** (one format decision, not two). Shape: eager core region - stays uncompressed for D-517 zero-copy in-place reads; **lazy lib regions + - raw `.clj` sources are flate-compressed** (measured: blob 698→112 KB flate, - `.clj` 451→125 KB; decompressing the *whole* blob costs 0.46 ms, and lazy - regions decompress only on `require`). **Format compaction is evaluated in - the same pass** (varint operands, cross-region constant-pool dedup) — it - attacks the same bytes with no decompression copy and no RSS cost. - Estimated −0.9 to −1.1 MB. Co-decide in D-517; do not build the format twice. -- **L3 — ship ReleaseSmall as the default artifact: REJECT.** Halves the - binary (4.84 MB measured) but drops every Zig safety check from the shipped - runtime; ADR-0132's shipped==gated ReleaseSafe identity stands. Recorded for - perspective; a user who wants a minimal binary can build one. -- **L4 — hybrid: zwasm module compiled ReleaseSmall under a ReleaseSafe cljw: - REJECT for the untrusted-input path; PENDING CODEV split otherwise.** The - wasm parser/validator/JIT-emitter processes attacker-controlled `.wasm` - bytes — exactly where ReleaseSafe's checks earn their keep; stripping them - there is a safety regression, not a size win. If zwasm ever splits - compute-only internals into a separate module, the hybrid can be re-scored - for that module alone (bundle with the L5 CODEV request). -- **L5 — zwasm-side size campaign: FILE via CODEV** (co-developed, so - tractable). Targets: the 691 KB monolithic `engine.codegen.arm64.emit.compile` - (table-driven encoding), comptime-gating component-model surfaces cljw never - calls. The zwasm budget line (4.0 MB) is the contract this campaign defends - while x86_64 JIT lands. -- **L6 — monomorphization dedup: LANDED 2026-07-16 (O-053; 7,073,080 → 6,974,584 B).** Original disposition: - std.sort instantiations alone are 0.22 MB (many ~20-28 KB `sort.block` - clones); type-erased comparator shims and `std.fmt` surface narrowing are - classic Zig size work with no safety/startup trade. Driven by the report - tool's symbol output; batch with L1's cycle or a quality-loop unit. -- **L7 — UCD table re-encoding: LOW.** Two-stage packed tables could halve the - ~207 KB rodata (generator: `scripts/gen_unicode_case.py`). Take only when - touching the generator anyway. -- **L8 — drop the raw `.clj` sources: REJECT.** They are load-bearing three - ways: bootstrap-error `SourceContext` line rendering, `cljw build`'s - re-envelope of user programs, and the non-AOT fallback path - (`clojure.repl/source` notably does NOT read them — it throws honest - not-available, D-513). Their fate is compression under L2. -- **L9 — feature-gate the TLS client: REJECT.** 0.43 MB buys an https-capable - `slurp`/http client; a runtime whose http can't do https is a partial-class - trap (F-014). Optional micro-probe: verify unused cipher-suite/cert-path - code is actually dead-stripped. -- **L10 — UPX / binary packers: REJECT.** macOS AMFI kills modified/packed - binaries (memory: `macos_cp_binary_breaks_codesign`); also opaque to the - report tool and to users. - -**Landing order**: L1 (single build.zig line, next cycle) → L2 inside the -D-517 unit → L6/L7 opportunistic. L5 filed via the CODEV channel -(`.dev/zwasm_capabilities.md` / `from_cljw`). Projected post-L1+L2 shipped -size: **~7.6-7.8 MB** on the reference platform. - -### 5. Narrative posture (README) - -README states the **measured** size (kept honest by the gate) and frames it -against verified peers: babashka 1.12.217's native binary is 70,933,648 B -(~71 MB, measured locally 2026-07-16) — cljw is ~7.5× smaller *with* an -embedded Wasm JIT engine. "Small" remains earned in the runtime's class; -the 3.8 MB era is not the claim anymore. - -### 6. Ledger reconciliations in this commit - -- **D-515**: re-narrowed — points here for composition/budget/levers; keeps - the standing drain-anchor role; stale 8.8 MB / 1.4 MB figures corrected. -- **D-277**: status updated — the size half of the tension now has a - mechanism (budget + gate); the startup half was discharged by ADR-0163 - lazy-ns. The tension itself remains standing (comprehensiveness keeps - consuming the cljw-text/data budgets) but is observable, not vigilance-based. -- **D-517**: barrier gains the L2 fold-in (compression + compaction co-decided - with zero-copy in ONE format redesign). -- **ROADMAP**: top-line "≤ 5 MB" and §10.3 "< 3.5 MB" amended per §17 (this - ADR is the amendment record). -- **`test/run_all.sh`**: stale size-heuristic comment rewritten; `size_claims` - step added. - -## Alternatives considered (Devil's-advocate fork, 2026-07-16, verbatim) - -> ## Leading entry: F-NNN violation check -> -> **No finished-form-clean alternative requires violating an F-NNN.** All -> three alternatives below are within the envelope. One important adjacent -> finding: the *old* size targets (ROADMAP line 73 "target binary ≤ 5 MB" and -> §10.3 "< 3.5 MB target / < 2 MB stretch") are **arithmetically unreachable -> while F-001 holds** — the zwasm engine+component-api alone is ~3.0MB, i.e. -> F-001's mandatory payload exceeds the §10.3 target by itself. Those targets -> predate the engine embedding and their retirement is a legitimate ROADMAP -> §17 amendment (ROADMAP-level, not an F-NNN), but the ADR must *say* this -> arithmetic out loud — "the old target died the day F-001's engine landed in -> the text segment" is the honest story, and it is a much stronger -> justification for a new budget than "we measured 9.5 and drew a line above -> it." -> -> ## Alternative 1 — smallest-diff: no new ADR; re-narrow D-515 + fix the -> stale numbers + land L1 -> -> D-515 already exists as the standing binary-size axis row and already -> sketches most of the draft's levers. The smallest-diff shape: update -> D-515's stale "8.8MB" to the measured 9,469,816 + fold the measured -> composition and lever dispositions into its barrier text; fix README 3.8MB; -> amend both ROADMAP claims; land L1 as an O-NNN row; no ADR-0172 at all. -> -> - **Better than the draft**: honours debt_dedup ("many additions are -> actually re-tagging an existing entry") — the draft creates a second SSOT -> over an existing SSOT row and then has to re-point the row at the ADR, a -> two-hop indirection for one axis. Zero governance machinery to maintain. -> - **Breaks**: the measured composition table (a genuinely valuable dataset -> — the first real attribution of the 9.5MB) has no durable, formatted -> home; debt-row prose is where tables go to die. The budget amendment -> (retiring ≤5MB / <3.5MB) *is* an ADR-level decision under ROADMAP §17 — -> quiet edits are forbidden — so this alternative still needs at least an -> amendment ADR, at which point you have most of ADR-0172 anyway. And it -> leaves governance at exactly the vigilance level that let 3.8MB rot for -> months. Not recommended. -> -> ## Alternative 2 — finished-form-clean: budget-and-gate ADR, not -> posture-and-ledger memo (RECOMMENDED) -> -> Keep ADR-0172, but change what kind of document it is. The draft is a -> *posture* doc: measured facts + a lever list + a soft convention. The -> finished form is a *budget mechanism*: numbers derived from something, -> enforced by something. -> -> 1. **Retire the old targets with the F-001 arithmetic**, amending **both** -> stale ROADMAP claims — the draft names only the line-73 top-line and -> misses the §10.3 table row. An ADR whose stated purpose is killing -> number-rot that itself leaves a stale number in §10.3 is self-refuting. -> 2. **Per-component budgets, not one total.** Size growth is feature-coupled -> (F-013 網羅 / F-014 per-class completeness guarantee it grows), so a -> flat byte ceiling structurally fights the comprehensiveness laws and -> will be re-drawn every time it's hit — the same rot loop. Budget the -> axes separately … A total ceiling then *derives* from the components -> plus stated headroom, and a breach localizes itself. -> 3. **Mechanical enforcement, not convention.** This repo's own doctrine — -> F-013 clause 3, verbatim: "closed structurally, not by vigilance" — -> condemns the draft's governance. Extend the pre-tag/release path with a -> small check: measure the binary, stamp binary_size_bytes + platform into -> CHANGELOG mechanically, **fail** if README's stated size drifts >10% -> from measured, warn on >5% inter-release growth. ~30 lines of script. -> Note bench/history.yaml already records binary_size_bytes — reuse or -> explicitly supersede it; a *third* uncoordinated size record is more rot -> surface. -> 4. **Fold L2 into D-517's format redesign as ONE envelope-format decision.** -> D-517 (zero-copy) is UNBLOCKED and actionable. Landing compression now -> and reshaping the format for zero-copy later is the twice-built shape -> ADR-0162's own DA red-team warned about. The finished form decides the -> rodata format once: zero-copy eager core region (uncompressed), flate -> lazy lib regions + .clj sources (ADR-0163's region blob is already cut -> for exactly this). Also consider **format compaction** (varint operands, -> cross-region constant-pool dedup) *inside* that same redesign — it -> shrinks the blob with no decompression copy and no RSS cost; -> compression-of-a-bloated-format vs designing-a-compact-format is -> precisely a smallest-diff-vs-finished-form fork. -> 5. **Re-score L4 on safety, not perf.** -> 6. **Also touch D-277**, the standing eager-comprehensiveness-vs-size -> tension row that F-013 clause 4 names — else the ledger contradicts the -> SSOT. -> -> This expands the cycle. **Recommend it anyway, citing F-002** — every added -> piece is the difference between "we wrote down that size matters" and "size -> cannot silently rot again," and the draft's own motivating incident is a -> pure vigilance failure. -> -> - **Breaks**: heavier cycle; the per-component budget needs a repeatable -> measurement recipe that must be maintained as composition shifts; genuine -> over-engineering risk for a solo project if the gate is made blocking -> rather than warn-mostly — keep only the README-drift check hard-fail. -> -> ## Alternative 3 — wildcard: attack the mass, and build per-symbol -> observability instead of a total-bytes ratchet -> -> The draft's ADOPT-now levers total ~1.6MB and mostly nibble the 17% that is -> data + overhead, while **44% of the binary is zwasm** and gets only "file a -> CODEV request". The wildcard inverts the emphasis: (a) make the durable -> artifact of this cycle a **per-symbol size report tool** (nm/segment diff, -> checked-in per release) so any future +5% is *localized* the day it lands, -> not just detected; (b) open a real CODEV size campaign against zwasm with -> measured targets — the 691KB monolithic emit.compile (single biggest symbol -> in the whole product), plus comptime-gating component-model surfaces cljw -> never calls — since the user co-develops zwasm, this is far more tractable -> than for a normal dependency; (c) run a cljw-side monomorphization dedup -> sweep off the report (std.sort instantiations 0.22MB, std.fmt, container -> instantiations). -> -> - **Better than the draft**: targets where the bytes actually are; the -> report tool is recurrence-prevention that *names the culprit*; the dedup -> sweep is durable cljw-side work no compression lever ever reaches. -> - **Breaks**: cross-repo — zwasm's timeline isn't cljw's to schedule, and -> the SHA pin bump is user-gated, so the headline −MB has no landing date -> cljw controls; an emitter rewrite risks JIT perf regressions; the -> nm-based tooling is platform-fiddly (the 9.5MB is a macOS arm64 number, -> the shipped-artifact Linux numbers are unmeasured and may differ — the -> ADR should say which platform its numbers and ratchet bind to). -> -> Alt 2 and Alt 3's observability/CODEV pieces compose; if forced to one, -> Alt 2. -> -> ## (a) Is the two-tier budget honest or rationalization? -> -> **As drafted, rationalization — fixable.** Three tells: (1) the ceiling is -> reverse-engineered from the measurement — a budget that never constrains -> anything at birth isn't a budget, it's a description; what makes the new -> line rot-proof where the old one wasn't? (2) the ~7MB "target" is -> current-minus-known-levers — approximately the *floor*, leaving no headroom -> for F-013/F-014-mandated growth, so it will be quietly missed the same way -> 3.8MB was; honest forms: per-component budgets with explicit headroom, or a -> ceiling anchored to a user-facing rationale. (3) the retreat is unexplained -> — 5MB → 10MB doubles the top-line without stating why the old number was -> wrong; the F-001 arithmetic is a complete, honest, even flattering -> explanation. Also verify the peer number against a current bb release -> before publishing — shipping an unverified peer number in the same commit -> that fixes your own unverified number would be poetic. -> -> ## (b) Missing / mis-scored levers -> -> L4 is mis-scored — the blocker is safety, not perf: ReleaseSmall on the -> zwasm module strips bounds/overflow checks from **the code that parses -> attacker-controlled .wasm bytes** — the one component where ReleaseSafe's -> checks earn their keep most. L1's cost is asserted, not measured — probe a -> panic in a stripped unwind-less build first (measure-first is the standing -> directive). Bytecode/format compaction is absent — its absence manufactures -> the L2-vs-zero-copy conflict. Drop-the-raw-.clj is unexamined — the ledger -> must state what reads it at runtime and give it a disposition. TLS -> feature-gating: absent — correct disposition is almost certainly REJECT -> (F-013/F-014), but a comprehensive ledger should say so. Monomorphization -> dedup: absent despite being named in the measurement. Linker-level levers: -> state dead-strip status rather than silently assume. Platform binding: -> declare the reference platform. -> -> ## (c) Does the governance prevent README-rot recurrence? **No** (as -> drafted). -> -> (1) It's the same mechanism class that already failed — bench/history.yaml -> recorded binary_size_bytes all along; nothing *compared* the prose claim to -> the measurement. The fix is the comparison: a pre-tag check that greps -> README's size claim and fails on >10% drift — this repo's exact house -> pattern, and F-013 clause 3 is the in-project law for it. (2) "Investigate" -> has no actor — unless the ratchet is a script, it fires zero times. (3) +5% -> per release at monthly cadence compounds to ~1.8x/year — sanctioned creep; -> state the ratchet-vs-ceiling interplay. -> -> ## Verdict -> -> The draft's measurement work is excellent and its L3/L7 rejections are -> right. Its three soft spots: the budget is drawn around the tape measure -> instead of derived (fix with the F-001 arithmetic + per-component budgets), -> the governance is vigilance re-labeled as mechanism (fix with a ~30-line -> pre-tag check), and the ledger under-weights safety (L4) and misses the -> cljw-actionable heavy levers (format compaction, monomorphization dedup) -> while ADOPTing a compression lever that risks rebuilding the D-517 format -> twice. Alternative 2 keeps everything the draft got right and closes those -> three; recommend it despite the larger cycle, per F-002. - -The accepted Decision above **is** Alt 2 with Alt 3's report tool and CODEV -campaign composed in; every (a)/(b)/(c) critique item was folded (F-001 -arithmetic stated, per-component budgets with headroom, `size_claims` gate, -L4 re-scored on safety, L1 probed before ADOPT, compaction added to L2, L8/L9 -dispositions grounded, platform binding declared, peer number measured). - -## Consequences - -- Binary-size claims can no longer rot silently: the full gate fails on >10% - README drift, and each release's CHANGELOG entry records the measured bytes. -- The old "small binary" marketing number is retired honestly; the story - becomes "9.5 MB single static binary including a Wasm JIT engine, ~7.5× - smaller than babashka; ~7.6-7.8 MB after the queued levers". -- D-517's format redesign becomes the single owner of every embedded-data - byte decision (zero-copy + compression + compaction), preventing a - twice-built format. -- The zwasm budget line creates the first cross-repo size contract; the CODEV - request (L5) carries measured targets instead of a vague "make it smaller". -- Risk: the per-component budget table requires re-measurement discipline at - budget-relevant moments (release, big feature). Mitigation: the report tool - makes a measurement a one-command act; the gate keeps at least the headline - honest even if a component drifts between audits. - -## Revision history - -- **2026-07-17 (D-561 Unicode name table, conscious)**: `Character/getName` - + `codePointOf` land the full UCD 16.0.0 name table as pre-compressed - word-indexed blobs (40,013 names; 545 KB raw → ~280 KB stored after the - delta-varint index rework — the first fixed-width index cost +394 KB and - was reworked before landing). macOS 7,073,240 → 7,352,376 B (+279 KB); - Linux est. ~8.29 MB (~94% of ceiling — the NEXT size-relevant feature - must measure Linux first and likely needs a lever or a ceiling - amendment). README claim moved 7.5 → "about 7.8 MB" (covers both - platforms within the ±10% gate). - -- **2026-07-17 (ADR-0174 feature growth, conscious)**: the host-class - identity & member-surface campaign grew the binary macOS - 6,974,584 → 7,073,240 B (+98.7 KB — Thread lifecycle worker + stdio - streams + Duration ISO parser + member tables) and Linux x86_64 - ≈ 7.5 → 8.01 MB (+~0.5 MB; Linux codegen amplifies the same additions). - Legitimate F-013/F-014 growth, within the 8.8 MB ceiling on both. - README's claim moved 7.1 → "about 7.5 MB" so the ±10% `size_claims` - gate covers BOTH platforms (the Linux gate run compares the same - README number against its own binary — the 2026-07-17 ubuntunote - failure class). Linux at 8.01 MB is ~91% of ceiling — the next - size-relevant feature should re-measure Linux first. -- **2026-07-16 (L2 complete + governance hardening)**: ADR-0173 C5′-b landed - compressed `.clj` sources (7,384,984 → **7,073,080 B**; the ~450 KB - startup gpa copy of raw text is also gone — `rt.source_resolver` - decompresses on demand and memoizes). The embedded-data budget line - re-sets to 1.0 MB per this ADR's own clause; derived ceiling ≈ 8.8 MB - (measured 7.07). **Campaign total: 9,469,816 → 7,073,080 (−25.3%).** - Governance mechanized further (user directive): `binary_size_report.sh - --check` now also FAILS when the built binary exceeds the derived ceiling - (`BUDGET_CEILING_BYTES`, kept in sync with this table), and the new - auto-loaded `.claude/rules/binary_size.md` carries the campaign's - operational lessons (instantiation-count metric, comptime cross-products, - embed-compression pattern, median-of-20 protocol) for every future - src/build edit. - -- **2026-07-16 (L5 outcome — same day)**: zwasm accepted the CODEV request - (their ADR-0204) and shipped **v2.2.1** same-day: the JIT host-callback - thunk collapse (`api.jit_host_bridge` 1,311 → 232 KB, −82%) — measured - cljw effect on re-pin: **8,583,352 → 7,499,896 B (−1,083 KB)**. The - table-driven-emitter half of L5 is **REFUTED by zwasm's reversible - experiment**: `emit.compile`'s 707 KB is once-called-handler AGGREGATION, - not duplication — out-lining was size-neutral (+28.8 KB, reverted). - Lesson folded into the ledger method: symbol-size attribution overstates - recoverable size when the code under a symbol has a single call site; the - predictive metric is instantiation/call-site count. zwasm budget line - re-set 4.0 → 2.5 MB (measured 1.94); derived ceiling ≈ 11 → ≈ 9.5 MB - (≈ 8.8 post-L2). Corrections also adopted: the x86_64 emitter already - exists (comptime-gated, 0 B on arm64). - -- **2026-07-16 (same day, campaign start)**: L1 landed (O-052) — shipped - ReleaseSafe 9,469,816 → 8,731,096 B; unwind/overhead budget line re-set to - 0.3 MB per the Decision's own re-set clause; derived ceiling ≈ 12 → ≈ 11 MB - (≈ 10.3 MB once L2 lands). Public comparison page - `docs/works/binary_size.md` (measured 2026-07-16 industry survey) landed as - the narrative-posture realization of Decision §5. - -## Affected files - -- `scripts/binary_size_report.sh` (new — report + check modes) -- `test/run_all.sh` (`size_claims` step; stale comment rewritten) -- `README.md` (honest size + verified peer comparison) -- `.dev/ROADMAP.md` (top-line + §10.3 amendments, §17 procedure) -- `.dev/debt.yaml` (D-515 re-narrow, D-277 update, D-517 L2 fold-in) -- `.dev/handover.md` (resume pointer) diff --git a/.dev/decisions/0173_envelope_v7_zero_copy_pool_flate.md b/.dev/decisions/0173_envelope_v7_zero_copy_pool_flate.md deleted file mode 100644 index 3fbd1c4c0..000000000 --- a/.dev/decisions/0173_envelope_v7_zero_copy_pool_flate.md +++ /dev/null @@ -1,437 +0,0 @@ -# ADR-0173 — Envelope format v7: executable wire instructions (zero-copy), lazily-materialized constant pool, flate-compressed lazy regions - -- **Status**: Proposed → Accepted (2026-07-16; DA fork folded verbatim below — - its four required changes are all adopted in this Decision) -- **Driven by**: D-517 (zero-copy in-place deserialize, ADR-0162 step 3 — the - ~2.1 ms deserialize half of the cold floor) **co-decided with** ADR-0172 L2 - (embedded-data size lever: −0.9〜1.1 MB) per ADR-0172's "one format decision, - not two". Step-0 survey: `private/notes/D517-envelope-survey.md` (+ Step 0.6 - amendments). -- **Relates to**: ADR-0034 (format-version policy: decoder-only permanent - compatibility + `docs/spec/formats/.edn` archive), ADR-0118 (VM - error locations — per-instr line:col on compiled chunks; amended by this ADR: - AOT chunks gain real `source_file` labels), ADR-0162/0163 (cold-start - architecture / region blob), ADR-0172 (size SSOT), D-515, F-004 (NaN-box law - — no raw Value archiving), F-006 (heap Values built at runtime), F-011/F-012 - (dual-backend oracle green per commit). - -## Context (measured, 2026-07-16) - -- Blob 698,159 B / 31 regions; eager 434 KB (62%), lazy 264 KB. 28,637 eager - instructions decode element-wise 3 B wire → 12 B memory (alloc + copy). - Constants dominate chunk bytes (977 KB of 1.17 MB decoded); instrs are - 135 KB — so the pool, not the instr copy, is likely the bigger half of the - 2.1 ms. Per-step attribution is mandatory during the arc - (`CLJW_PROFILE_STARTUP=1`). -- **42% of the blob (~294 KB) is duplicate interned-name encoding**: 15,907 - interned-name constant occurrences, only 3,342 unique (`var - clojure.core/list` ×1,141). Net size win after pool+refs bookkeeping - ≈ −230 KB (u32 refs = 63.6 KB + unique pool ≈ 60 KB replace ~294 KB+uniques - of inline encoding). Dedup also cuts intern/hash calls ~5×. -- 97.7% of operands < 128, **but** varint decoding forces a copy — it - structurally conflicts with zero-copy, and flate covers the wire-size win - for the lazy half. -- `@embedFile` guarantees alignment 1; in-memory `Instruction` is auto-layout - 12 B (opcode u8 / operand u16 / line u32 / column u16); AOT chunks always - carry line=0 col=0 (ADR-0118). -- **`readValue` precondition (DA-verified, load-bearing)**: the var_ref arm - (serialize.zig:455) requires the namespace to ALREADY exist at decode time; - type_descriptor likewise requires the defining chunk to have RUN. Today this - is guaranteed only by interleaved chunk-by-chunk decode+run. Any design that - decodes all constants up front breaks it (or, "fixed" by forward-creating - namespaces, becomes an F-011 regression: `(find-var 'clojure.set/union)` - must fail before `(require 'clojure.set)` as in clj). -- flate: whole-blob decompress measured 0.46 ms (~0.3 ms for the eager 434 KB); - lazy regions decompress only on `require`; `.clj` sources 451 KB → ~125 KB. - Zig 0.16 `std.compress.flate` direct-mode recipe verified against the pinned - std (fixed reader → `.raw` container → `readSliceAll` into an exact - `uncompressed_len` buffer; no window buffer, no hidden alloc). - -## Decision - -1. **WireInstr — the only instruction representation the VM executes.** - 4-byte `extern struct { opcode: u8, reserved: u8, operand: u16 LE }`. - `reserved` MUST be 0 and is validated at load (corruption tripwire + - forward space; a u24 `operand_ext` was considered — operand-space pressure - exists (`DEF_NAME_IDX_MAX` 13-bit, fused-op 8-bit packing) but has no - measured need yet). `BytecodeChunk.instructions: []const WireInstr` with a - parallel `locs: ?[]const InstrLoc` sidecar. - **Split-at-finalize (DA recommendation adopted)**: the compiler's - build-time representation KEEPS the 12 B loc-carrying `Instruction`, so - emit/patchJump/peephole/fusion rewrites carry loc atomically with no - lockstep hazard; `finalize` performs the one split into - `[]WireInstr + []InstrLoc`. Guards: (i) - `assert(locs == null or locs.?.len == instructions.len)` at finalize; (ii) - a NEW e2e asserting exact line:col of a VM error raised AFTER a fused - compare-branch and after a peephole-modified region (the diff oracle - compares values, not error locations — it does NOT cover this); (iii) the - C1 representation change is A/B'd in isolation (protocol below) with an - explicit revert trigger. -2. **Zero-copy over aligned bytes; ONE decoder.** Aligned embed wrapper - (comptime copy into `align(8)` rodata) + the serializer pads each chunk's - instr section to 4-byte offsets from blob start; - `deserializeChunkInPlace` slices instrs via `bytesAsSlice` after a linear - **opcode-validation scan** (`WireInstr.opcode` stays raw u8; the scan - preserves the `DeserializeError` fail-closed contract — critical for CLJC - user files, where reinterpret-without-validation would turn a bad byte into - dispatch-time UB/panic). Endianness: `comptime assert(little)` — the dead - non-LE runtime fallback is NOT kept (all shipped targets are LE; - an unreachable fallback is the permanent-dead-path shape the project - forbids). The copying decoder survives the arc only until C3′ replaces its - callers, then is deleted. Byte sources are three, one code path: rodata - (eager), `rt.load_arena` (lazy decompressed), and the **CLJC artifact - payload read into a kept-alive 4-aligned session buffer** (fn constants - capture their chunks — the buffer must outlive them; audited in C3′). - Ownership: in-place chunks are borrowed views + per-chunk mutable side - arrays (`call_sites` cache); the tree_walk.zig:401 freeChunk recursion is - audited in C3′ and deleted where the borrowed shape makes it moot. -3. **Constant pool — lazily-materialized name-slots (DA-corrected shape).** - The pool section stores **tag + name bytes** (deduped); the runtime pool is - a slot array (`[]?Value`-shaped) where a slot materializes through the - EXISTING `readValue` arms on first reference by a decoding chunk — this - preserves resolution timing exactly (the first referencing chunk resolves - precisely when today's inline copy would have; 2nd+ duplicates memoize to - the same interned Value, observationally identical), keeps the ~5× - intern/hash dedup and the size win, and avoids eagerly compiling pooled - regexes for never-required lazy namespaces. Pool refs are **u32** - (boring-safe; no overflow class for `cljw build` user payloads — a u16 - variant would need a fail-closed `PoolOverflow`; the 32 KB delta is not - worth the failure mode). GC rooting: interned symbols/keywords need no - root; pooled gc-managed Values (strings, regexes) make the slot array a - published root — one root slot + `GC-ROOT:` marker + `.dev/gc_rooting.md` - row, tracing filled slots only. Artifact payloads carry their own - self-contained pool. -4. **Flate lazy regions + `.clj` sources** (`.raw` container + explicit - `u32 uncompressed_len` per compressed region — raw has no framing; MUST in - the spec). Each lazy region compressed individually (per-region beats a - super-block: first `require` dirties only its own region, sub-0.1 ms each; - the flate-window ratio loss across small regions is a few KB — noise). - Decompressed into `rt.load_arena` on `require`; the same in-place reader - runs over the arena buffer. The eager core region stays uncompressed for - zero-copy (the compress-eager-too alternative is Rejected below WITH its - arithmetic, per the DA). Raw `.clj` sources compressed with - decompress-on-demand handles (consumers: error `SourceContext` render, - `cljw build` re-envelope, `embeddedResolver`; `cache_gen` host tool keeps - raw inputs). CLJC artifacts do NOT gain a compression flag in v7 (no - measured need; user-local files). -5. **v7 rides three latent fixes**: `has_handlers` serialized (today it - defaults true on AOT chunks → no bootstrap fn is ever frame-flattened; the - *consumer flip* is its own commit, C6′, with oracle+bench attribution), - `source_file` preserved (AOT chunks stop reporting "unknown" — a one-line - ADR-0118 amendment: labels improve, line stays 0), inner `fn_val` chunks - stop repeating magic+version. -6. **Rejected** (each with its arithmetic): - - **Varint instruction stream**: 97.7% of operands < 128, but decode = copy; - structurally anti-zero-copy; flate covers the lazy wire size anyway. - - **Compress the eager region too** (DA Alt 3): kills the whole alignment - apparatus and saves ~360 KB more binary, but costs ~+0.3 ms cold start - every boot and converts 434 KB of clean, evictable file-backed rodata - into dirty anon arena pages. D-450/D-517's floor axis outranks the - residual size delta here — and the alignment apparatus is a one-time - format cost, not a standing tax. Revisit only if ADR-0172's embedded-data - budget line is breached after C5′. - - **rkyv-style object-graph archiving / mmap-ELF container**: F-004/F-006 — - heap Values are never archived as bits (the whole D-517-vs-heap-snapshot - risk story, ADR-0162). - - **Keeping pre-v7 decoders**: ADR-0034 — archive the spec - (`docs/spec/formats/7.edn`), not the decoder. - - **Eagerly-decoded `[]Value` pool** (the survey's original §5(ii) shape): - breaks `readValue`'s findNs/already-ran preconditions; the - forward-creating-ns "fix" is an F-011 regression. Replaced by the - lazy-slot shape in Decision 3. - - **Non-LE runtime fallback decoder**: unreachable on every shipped target - = permanent dead path; `comptime` assert instead. - -Projected: embedded data 1.59 MB → ~0.9 MB in-binary (ADR-0172 budget line -re-sets to 1.0 MB on C5′); eager deserialize cost mostly eliminated (instr -copy gone + ~5× fewer interns); cold floor toward sub-4 ms. Both claims are -measured per-commit (protocol below), not asserted. - -## Revision history - -- **2026-07-16 (C1 landing)**: Decision 1's "opcode stays a raw u8, dispatch - converts via `op()`" was **measured at +5-6% on fib/arith** (the - per-dispatch `@enumFromInt` ReleaseSafe validity check). Amended: - `WireInstr.opcode` is the **typed `Opcode` field** (enum(u8), defined - layout, extern-legal) so dispatch reads it directly; untrusted input is - validated by a **raw-byte scan BEFORE any `[]WireInstr` view is formed** - (C3′), so an invalid enum value never materializes in a typed slice — - the fail-closed `DeserializeError` contract is unchanged. With the typed - field the C1 A/B is within noise (fib 1.02±0.11, arith 1.01±0.07 and - 1.03±0.11 order-swapped, 30 runs hyperfine -N ReleaseSafe; sieve / - map_filter_reduce neutral) — the revert trigger does not fire. Guard e2e - `phase16_vm_error_loc_sidecar` (exact 5:6 after a peephole-elided region + - fused compare-branch) landed with C1. - -- **2026-07-16 (C2′ landing — the single v7 wire cut)**: serializer+decoder - emit/read final v7 (4B WireInstr wire, 4B-aligned instr sections via - chunk-body padding + 8B-aligned region starts, `source_file` + - `has_handlers` serialized, headerless nested fn-method chunks, - `pool_ref` 0x11 + blob-level shared pool / payload-inline pool). - Spec archived at `docs/spec/formats/1.4.0.edn` INCLUDING the v4-v6 - backfill (three earlier bumps had violated the ADD-ONLY archive policy — - caught by this arc). **Measured**: blob 698,159 → 544,909 B (−22%, pool - dedup alone); shipped binary 8,731,096 → 8,583,352 B (−148 KB). Consumer - flip (`has_handlers` flatten) remains C6′. -- **2026-07-16 (C3′ landing + measurement CORRECTION)**: in-place instr - views landed (raw-byte validation scan → `bytesAsSlice`; - `borrowed_instrs` ownership flag; aligned embed wrapper; copy path - retained for misaligned/owned callers). **Correction of the C2′ note's - startup claim**: the "6,123 → 4,190 µs (−32%)" figures were SINGLE-SHOT - probes under load — a clean median-of-20 comparison (quiet machine) - shows eager `runEnvelope` **neutral within noise** (baseline 3,234 µs vs - post-C3′ 3,261 µs median; hyperfine total-startup 1.01 ± 0.12). The v7 - arc's verified wins so far are SIZE (−22% blob / −148 KB binary), not - startup; the remaining startup candidate is C4′'s memoized pool slots - (15.9K → 3.3K intern/resolve calls), to be judged by the same - median-of-20 protocol. D-517's "~2.1 ms deserialize half" premise is - itself load-suspect and is re-anchored by these medians. - -- **2026-07-16 (C5′-a landing — flate lazy regions)**: region-blob index v2 - (stored-len + flags + uncompressed-len); lazy regions flate-compressed - (`.raw`, level best) at cache_gen time, decompressed into `rt.load_arena` - (8-aligned — the in-place instr views run over the buffer) on first - `require`; the eager set stays raw for rodata zero-copy (compressed-eager - = build bug, fail-closed). **Measured**: blob 544,909 → 425,125 B; shipped - binary 7,499,896 → **7,384,984 B**. Two probe-caught bugs recorded: - `Compress.init` asserts a non-empty output buffer (`initCapacity`, the - cache_gen ABRT), and the zero-buffer "direct mode" Decompress recipe - usize-underflows via the indirect rebase path under `readSliceAll` — a - REAL 32 KB history window is required (the survey's §4 recipe is hereby - corrected). C5′-b (compressed `.clj` sources, est −326 KB) remains. - -- **2026-07-16 (C5′-b landing — flate `.clj` sources)**: the raw sources - left the binary (cache_gen emits a second artifact, the compressed - sources blob; `bootstrap.sourceText` decompresses on demand into - `rt.load_arena`; `rt.source_resolver` serves error renders registry-miss - → memoized; cache_gen keeps raw @embedFile via the dual - `embed_raw_clj_sources` options module — its own stub `bootstrap_sources` - import avoids the WriteFiles dependency loop probed during landing). - **Measured**: shipped binary 7,384,984 → **7,073,080 B**. The eager - ~450 KB registerSource gpa copy at startup is gone (resolver is lazy). - Remaining in the arc: C6′ (has_handlers consumer flip). - -## Measurement protocol (mandatory, per commit) - -- VM-hot A/B for C1 (representation change, isolated): - `bash bench/run_bench.sh --quick --bench=` (hyperfine, ReleaseSafe - only — ADR-0132) across fib_recursive / arith-recur loops / sieve; bar: - no regression beyond noise (σ < 5%). **A C1 regression REVERSES the split** - (loc returns inline; the wire can still be 4 B with a load-time widen — - the wire win decouples from the in-memory gamble). -- Startup attribution for C2′/C3′/C4′/C5′: `CLJW_PROFILE_STARTUP=1` before/ - after each commit; the deserialize line must show the claimed step-wise - drops (and the flate term appears honestly in C5′). -- Size: `scripts/binary_size_report.sh` before/after C5′; ADR-0172 budget - table + `size_claims` README figure updated in the same commit. - -## Sequencing (single wire cut; every commit oracle + smoke green) - -- **C1** — representation split at finalize: build-time `Instruction` stays; - `BytecodeChunk` becomes `[]const WireInstr` + `locs` sidecar; VM dispatch - reads the 4 B form. No wire change. A/B bench + new error-loc e2e. -- **C2′** — **the one v7 wire cut**: serializer emits final v7 (padded - WireInstr section + pool section + `has_handlers` + `source_file` + inner - header drop + per-region `uncompressed_len` slots); deserializer reads all - of it (may still copy instrs and materialize pool slots per-chunk at this - commit). `docs/spec/formats/7.edn` archived HERE, once. CLJC artifacts are - v7-coherent from this commit on — no mid-arc cross-binary window. -- **C3′** — in-place instr reading (rodata + kept-alive CLJC buffer + arena; - opcode-validation scan; freeChunk recursion audit/deletion). No wire change. -- **C4′** — lazy pool slots wired as the constants path; intern-dedup - measured. No wire change. -- **C5′** — flate lazy regions + `.clj` handles (ADR-0172 L2 lands; size - numbers + budget update). Container-level change only. -- **C6′** — `has_handlers` consumer flip (frame-flattening for handler-free - AOT fns) — VM behavior change, own oracle + bench attribution. - -## Alternatives considered (DA fork, 2026-07-16 — verbatim) - -> ## Leading entry (mandated): F-NNN compliance of the finished-form-clean option -> -> **No alternative below requires violating any F-NNN.** All three shapes stay -> inside F-002/F-004/F-006/F-011/F-012/F-013 and ADR-0034's decoder-policy. -> However, one part of the **draft itself is an F-011 hazard as currently -> worded** — the constant pool "decoded ONCE at startup into `[]Value`" -> (survey §5(ii), draft item 3). See critique (b): eager pool materialization -> of `var_ref`/`type_descriptor` entries either **hard-fails at load** -> (checked against `readValue` :455 — `env.findNs(ns_bytes) orelse return -> DeserializeError` requires the ns to already exist, which today is -> guaranteed only by interleaved chunk-by-chunk decode) or, if "fixed" by -> forward-creating namespaces, **changes observable semantics** -> (`(find-var 'clojure.set/union)` before `(require 'clojure.set)` must fail -> as in clj). The fix is inside the envelope (lazily-materialized pool slots), -> not an F-NNN amendment — but the ADR text must change before acceptance. -> -> ## Alternative 1 — smallest-diff: "pool + flate + latent fixes, NO WireInstr -> / NO zero-copy" -> -> Land C3 (constant pool, as name-bytes with per-chunk resolution), C4 (flate -> lazy + .clj), C5 (has_handlers/source_file/inner-header-drop) — and skip the -> instruction-representation change entirely. -> -> - Better: zero touch on the VM's hot representation; zero ADR-0118 risk; -> captures the measured majority of both wins (constants = 977KB of 1.17MB -> decoded vs instrs 135KB; the instr copy is plausibly 0.3-0.5ms of the -> 2.1ms, not half). -> - Breaks: D-517's named deliverable stays open; 12B→4B icache + the last -> ~0.3-0.5ms forfeited. Not recommended on diff-size grounds (the -> smallest-diff pole); its one legitimate use: if the C2 A/B shows the instr -> half < ~0.3ms, Alt 1's scope IS the measured shape. -> -> ## Alternative 2 — finished-form-clean: "one representation, one decoder, -> no fallback" -> -> The compiler emits WireInstr directly (or split-at-finalize); DELETE the -> copying decoder (comptime LE assert, no runtime fallback — a fallback with -> no reachable caller is the permanent-dead-path shape the project forbids); -> the CLJC artifact path becomes a lifetime rule (kept-alive 4-aligned -> session buffer, same in-place reader), not a second decoder; ownership -> unifies (chunks always borrowed views + arena side arrays; the -> tree_walk.zig:401 freeChunk recursion is deleted, not worked around). -> -> - Better: no dual-decoder bitrot; no C1-then-C2 half-supersede; one codec -> for blob + artifact; one answer to "what is a chunk in memory". -> - Costs: compiler emit/patchJump/peephole surgery in the same arc; -> `CallSiteEntry` mutable cache split into a side array; loses hypothetical -> big-endian support (no F-NNN requires it). **Recommended over the draft's -> shape** — the "retain the copying decoder as fallback" is the -> Smallest-diff bias wearing a portability costume. -> -> ## Alternative 3 — wildcard: "compress EVERYTHING; one bulk decompress into -> an aligned arena; in-place over the arena" -> -> Flate the eager region too (~72KB embedded vs 434KB), decompress at boot -> (~0.3ms) into an 8-aligned session arena, run the same in-place readers. -> -> - Better: kills the ENTIRE alignment problem (no aligned-embed trick, no -> serializer padding, no bytesAsSlice asserts); one lifetime story; ~360KB -> more binary shrink (embedded data → ~0.5-0.6MB); still satisfies D-517's -> letter (no per-chunk alloc/copy). -> - Breaks: +~0.3ms cold start every boot (floor ~3.8-4.1 instead of -> ~3.5-3.8); +~280KB dirty anon RSS at boot vs clean evictable rodata; the -> profiler gains a fixed decompress term. A genuine trade the ADR must -> decide WITH numbers, not dismiss. -> -> ## (a) WireInstr 4B + parallel locs -> -> The `_pad` byte: spec it as reserved-MUST-be-0 validated at load; note the -> u24 operand-ext consideration (operand-space pressure exists: -> DEF_NAME_IDX_MAX 13-bit, fused-op 8-bit packing) without adopting it. -> **The real ADR-0118 risk is peephole/fusion lockstep**: locs as a second -> array that every emit/fusion/peephole must mutate in lockstep; the silent -> failure is locs off-by-N after a stream-shrinking pass — errors report a -> WRONG line, which no existing test distinguishes. Guards: len-assert at -> finalize + every pass exit; a NEW e2e asserting exact line:col AFTER a fused -> compare-branch and a peephole-modified region; say explicitly the diff -> oracle does NOT cover error locations. **Split-at-finalize removes the -> lockstep hazard entirely** (build-time keeps 12B; only the finalized form -> splits) — may be the strictly better shape the draft never considers. -> The icache claim is a premise, not a measurement — C1 must be A/B'd in -> isolation (`run_bench.sh --quick`, hyperfine, ReleaseSafe, σ<5% bar) with -> an explicit revert trigger (loc back inline; wire stays 4B with load-time -> widen — decouples the wire win from the in-memory gamble). -> -> ## (b) Constant pool — the draft's weakest section -> -> Resolution timing is NOT preserved by the survey's design (verified against -> readValue :455/:486 — findNs precondition + "defining chunk has already -> run"). Interleaved decode+run supplies today's invariant; a boot-time pool -> decode breaks it; forward-creating namespaces is an F-011 regression. -> **Finished-form repair: pool = tag + name bytes; runtime pool = -> lazily-materialized slot array** filled through the EXISTING readValue arms -> on first reference — preserves timing exactly, keeps the 5× dedup (each -> unique name read+hashed once), keeps the size win, avoids eagerly compiling -> pooled regexes for lazy-only namespaces. GC rooting: interned syms/kws need -> no root; pooled STRINGS are gc-managed → slot array = published root (nil- -> init, trace filled only, GC-ROOT marker + gc_rooting.md row). -> Arithmetic: u32 refs = 15,907×4 = 63.6KB + unique pool ≈ 60KB replace -> ~294KB+uniques → **net ≈ −230KB, not −294KB**. u16 halves ref cost but -> needs fail-closed PoolOverflow for cljw-build user payloads; u32 is the -> boring-safe pick. Show the arithmetic in the ADR either way. -> -> ## (c) flate -> -> The 0.46ms local measurement on the pinned 0.16 std outranks regression -> folklore; pin the direct-mode recipe (fixed reader → .raw → readSliceAll -> into exact uncompressed_len; no window buffer) and store u32 -> uncompressed_len per region (raw has no framing) as a spec MUST. -> Per-region beats super-block (first require dirties only its region; -> window-ratio loss across small regions = a few KB = noise). The -> eager-compression question IS Alternative 3 — carry its arithmetic in -> Rejected. **RSS clean-vs-dirty**: uncompressed rodata is file-backed clean -> evictable; arena-decompressed bytes are dirty anon forever (load_arena is -> session-lifetime by design; fns capture their bytecode). Compressing raw -> .clj is near-pure win (dirty only on cold error-render); worst-case -> all-lazy-required ≈ +264KB dirty = 1% of the <25MB idle target — write the -> number; today's copying decoder already builds >wire-size dirty structures, -> so in-place-over-arena is RSS-neutral-to-better vs today. -> -> ## (d) Sequencing — one thing genuinely backwards; the version question has -> a sharp answer -> -> C2→C3 as two wire-changing commits under one "v7" breaks the ADR-0034 -> archive discipline (spec written twice or describing a one-commit layout) -> and CLJC cross-binary integrity (two artifacts both saying "7" disagreeing -> on the constants section). For the @embedFile'd blob the version genuinely -> does not matter (regenerated every build; self-consistency assert only); -> for CLJC it does. **Fix: format cut ONCE (C2′ emits+reads final v7), -> migrations after (C3′ in-place, C4′ lazy pool, C5′ flate container, C6′ -> flatten flip)** — every post-C2′ commit is wire-stable and independently -> A/B-attributable. Which v7 parts reach CLJC: WireInstr layout+padding, -> per-payload pool, has_handlers/source_file, inner-header drop — ALL; -> region-blob flate + .clj compression — NOT (container level). Recommend no -> artifact compression flag in v7. Pin the artifact in-place story (4-aligned -> kept-alive buffer; unaudited by the survey — audit in C3′). -> **Latent hazard for C3′: opcode validation** — today's copy loop validates -> via std.enums.fromInt; bytesAsSlice + @enumFromInt dispatch would turn an -> invalid byte into safety-checked UB/panic instead of DeserializeError; for -> CLJC user files it must fail closed. Keep opcode raw u8 + one linear -> validation scan at load. Riders: flipping the has_handlers CONSUMER is its -> own commit (VM behavior change); ADR-0118 gets a one-line amendment -> (source_file labels improve). -> -> ## Verdict -> -> The draft's core skeleton is sound and F-NNN-clean, but four changes are -> required before Accepted: (1) lazily-materialized pool slots (the hardest -> finding, verified in code); (2) single wire cut before consumer migrations; -> (3) adopt Alt 2's decoder story at least partially (kill the dead non-LE -> fallback; kept-alive in-place artifact path; consider split-at-finalize to -> erase the locs lockstep hazard); (4) numbers where premises are (C1 A/B -> with revert trigger; Alt-3 arithmetic + clean/dirty RSS analysis in -> Rejected). None of these blocks the unit; all fit the F-NNN envelope. - -All four required changes are adopted: Decision 3 is the lazy-slot pool, -the Sequencing is the single-wire-cut order, Decision 2 kills the non-LE -fallback + pins the kept-alive artifact path + adopts split-at-finalize, and -the Measurement protocol + Rejected arithmetic sections carry the numbers. - -## Consequences - -- The envelope format becomes load-bearing/rigid (accepted by D-517's - barrier); every future chunk-layout change is an on-rodata-format change - with a mandatory spec archive. -- The VM's executed instruction representation changes (4 B + loc sidecar) — - A/B-benched at C1 with an explicit revert trigger; the compiler's build-time - representation is unchanged (split-at-finalize), so the peephole/fusion - layer carries no lockstep risk. -- `cljw build` artifacts are v7 from C2′ on; a v6 artifact is rejected by a - v7 cljw with the standard version error (acceptable per ADR-0034 policy; - pre-2.0). -- RSS: worst-case all-lazy-required ≈ +264 KB dirty anon (1% of the <25 MB - idle target); compressed `.clj` render paths dirty at most ~451 KB on cold - error-render; today's copying decoder already builds larger dirty - structures per region, so the net is neutral-to-better. -- ADR-0118 amendment (one line): AOT chunks now carry real `source_file` - labels; per-instr line stays 0 for AOT. - -## Affected files - -- `src/eval/bytecode/serialize.zig`, `src/eval/backend/vm/opcode.zig` (+ VM - dispatch), `src/eval/backend/vm/compiler.zig` (finalize split), - `src/eval/backend/tree_walk.zig` (freeChunk recursion audit/deletion), - `src/app/builder.zig`, `src/cache_gen.zig`, `build.zig` (aligned embed - wrapper), `src/lang/bootstrap.zig` (compressed `.clj` handles), - `docs/spec/formats/7.edn` (new), `.dev/gc_rooting.md` (pool root), - `.dev/debt.yaml` (D-517), ADR-0172 (revision on C5′), ADR-0118 (amendment). diff --git a/.dev/decisions/0174_host_class_identity_and_member_completion.md b/.dev/decisions/0174_host_class_identity_and_member_completion.md deleted file mode 100644 index cdfcba336..000000000 --- a/.dev/decisions/0174_host_class_identity_and_member_completion.md +++ /dev/null @@ -1,278 +0,0 @@ -# ADR-0174: Host-class identity unification + member-surface completion campaign - -- **Status**: Accepted (2026-07-17) -- **Driver**: user chat 2026-07-16 — `System` unresolvable in value position, - `(System/getProperties)` → misleading "No namespace: 'System'", - `(Thread. f)` ctor missing; user directed a campaign: fix the symmetry - problems, add the members worth having, survey the other installed - classes, release when done. Push + release explicitly authorized. -- **Inputs**: probe survey `private/notes/host-class-symmetry-campaign-survey.md` - (639 static members probed: 381 present / 257 absent); - mandatory DA fork `private/notes/host-class-symmetry-DA-critique.md` - (Alt B adopted; its findings are folded in below). - -## Context - -Five user-visible defects share one root — class-name/identity truth is -dispersed and inconsistent across the host-class subsystem: - -1. **Value-position asymmetry**: `Long`/`String`/`UUID`/`Object` resolve as - class values (native/opaque/marker paths) but `System`/`Math`/`Thread`/ - `StringBuilder`/`java.util.Date` do not — `resolveClassValue` - (analyzer.zig) never consults `host_class_resolve.resolve`, the only - place the `cljw.` key translation + `java.lang` auto-import live. -2. **Misleading member-miss diagnostic**: a resolved class + missing member - falls through to `namespace_unknown` ("No namespace: 'System'"); the - analyzeList comment narrating a `symbol_unresolved` fall-through is stale. -3. **fqcn spelling zoo (user-visible leak)**: natives/user/exceptions = - simple ("Long"); typed_instance impls = simple ("Date", "Instant"); - host-enums = JVM FQCN ("java.math.RoundingMode"); host_stream = JVM FQCN - ("java.io.BufferedReader", corpus-pinned); host_instance surfaces MIXED — - File = "java.io.File" but StringBuilder = "cljw.java.lang.StringBuilder", - visible via `(class x)`, `.getName`, and print (`#`). -4. **typed_instance two-descriptor split**: Date/Instant/LocalDate/… carry a - per-Runtime lazy impl descriptor (instances; fqcn "Date") + a static - surface descriptor ("cljw.java.util.Date", `` only). Consequences: - `(instance? java.util.Date d)` → Name error, `(resolve 'java.util.Date)` - → nil, and any fix that resolves the class symbol to the surface - descriptor breaks `=` with `(class d)` — including across the AOT wire - (serialize writes `descriptor.fqcn`; the import-blind - `resolveDescriptorByKey` (ADR-0034 am5) would return the surface). -5. **`(class Long)` → "type_descriptor"** raw tag leak (clj: `java.lang.Class`). - -Plus a process failure: `data/compat_tiers.yaml` member lists are rotted in -BOTH directions (Long/MAX_VALUE works but unlisted; MessageDigest/getInstance -listed but absent) — a one-time refresh without a mechanical gate re-rots -(F-013 clause 3). - -## Decision - -### D1+D2 — one canonical descriptor per class, JVM-FQCN identity (one unit) - -**Identity rule (one line)**: a class backed by the Java-compat surface tree -(host_instance / typed_instance / host-enum / host_stream) has -`fqcn = its JVM FQCN`; cljw-native tags, user deftypes/records, the -Throwable family, interface markers, and `Object` keep SIMPLE names -(AD-003, amended to state this scope explicitly). - -- All `cljw.`-prefixed descriptor fqcns become prefix-less JVM FQCNs - (~25 surface files). rt.types keys stay `= fqcn` (mechanism unchanged). -- `host_class_resolve`: auto-import formats flip to `java.lang.{s}` / - `java.math.{s}`; the now-dead `"cljw." ++ head` translation step and - `displayName` are DELETED (fully-remove, no compat shim); `bareName`'s - prefix tests flip to `"java.lang."` / `"java.math.Big"` (completion's - bare-name source dies silently otherwise — nrepl completion e2e is the - canary). -- Hardcoded `"cljw.java.*"` registry-key strings across the tree (Thread.zig, - lang/Runtime.zig, _host_api tests, …) ride the same commit (grep-sweep; - ~53 files carry the string). -- **Descriptor merge (typed_instance family)**: the rt.types-registered - surface descriptor becomes THE descriptor — statics + `` + instance - methods in one `method_table`; instances carry it; the per-Runtime lazy - impl descriptors (`rt.date_descriptor` … `rt.local_time_descriptor`) and - their deinit paths are deleted; instance-method init moves into the - surface `init` callbacks. This is "make the outlier match the norm" — - host_instance classes (Thread/File/URI/…) already use one descriptor for - both. -- **Equality stays identity** (clj oracle: JVM Class objects are - singletons). The `ref_cache` interning invariant in type_descriptor.zig - ("do NOT mint a fresh ref for an already-wrapped descriptor") is the - guarantee. The DA-considered fqcn-string-`=` arm is REJECTED as unsound - (no hash arm; breaks classes-as-map-keys; re-implements interning badly). -- **Collision guard**: `method_table` is flat (no static/instance bit). - The merged family has no same-name static+instance member today - (Integer/Long static `toString` is dodged because native instance dispatch - uses the tag-keyed native descriptor). A registration-time assert fails - loudly if a same-name pair ever lands on one descriptor; an `is_static` - bit is the named escalation if a real collision is ever wired. -- `resolveClassValue` gains the `host_class_resolve.resolve` fallback - (import-aware, analyzer side). `resolveDescriptorByKey` (shared with the - AOT deserializer) stays import-blind and is CORRECT unchanged under the - merge — the wire fqcn is the registry key. -- **Wire compat**: baked class-value constants change spelling - ("Date" → "java.util.Date"), so `cljw build` artifacts from ≤1.4.0 are - version-rejected — envelope bump v7 → v8, CHANGELOG breaking note (the - in-tree bootstrap cache regenerates at build; no in-tree issue). -- Print/`.getName`/`str` of class values need no per-egress translation — - the descriptor fqcn IS the user-facing name (the StringBuilder leak class - dies structurally; Alt A's translation-at-every-egress was rejected as - the Smallest-diff bias smell in canonical form). - -### D3 — member-miss diagnostics (position-split, clj-shaped) - -Two new error codes replacing the misleading fall-through: - -- value position (analyzeSymbol, bare `Class/member` symbol): - `static_field_unknown` — "Unable to find static field: {member} in class - {class}" (clj's wording). -- call position (analyzeList, `(Class/member …)` head with resolved class + - member miss): `static_method_unknown` — "No static method '{member}' on - class {class} (not defined, or not implemented in ClojureWasm)". - -`isDeferredHostNs` (clojure.lang.* / clojure.asm.*) keeps the AD-008 -deferred-rewrite path — member-miss there must NOT hard-fail analysis. -The stale analyzeList comment is fixed in the same commit. - -### D4 — `Class` becomes a first-class marker - -- `fqcnForTag(.type_descriptor)` → "Class": `(class Long)` → `Class`. -- Bare `Class` resolves as a class value (classDescriptor marker, like - `Object`); `(instance? Class (class 5))` → true (tag arm). -- `Class/forName` → explicit OPAQUE row (no classpath; F-014 clause 2 - forbids silent absence next to a freshly-minted name). - -### D5 — System close-out - -- Add: `getProperties` (cljw persistent map of the static table + the - existing `rt.system_properties` overlay — `setProperty` already exists; - map-not-Properties recorded as an AD), 0-arg `getenv` (env map), - `clearProperty` (overlay remove, returns previous), `identityHashCode` - (cljw identity hash — heap pointer / immediate bits; AD note: not JVM - object headers), `gc` (real collect trigger). -- Static fields `in`/`out`/`err`: real stdio-backed stream singletons - (ADR-0087 heap-singleton static-field precedent). `out`/`err` are a new - `print_stream` stream kind — chain {PrintStream, FilterOutputStream, - OutputStream}, PrintStream REMOVED from `SIBLING_NAMES` (the "never - produced" invariant edit is owned here); methods write/print/println/ - flush; println autoflushes (JVM stdout oracle) so cljw's own buffered - print path and System/out writes do not interleave nondeterministically. - `in` reuses the input kind over stdin (clj `(class System/in)` = - BufferedInputStream = our input concrete). -- Skipped members (console, getLogger, load, loadLibrary, mapLibraryName, - inheritedChannel, setIn/setOut/setErr, setProperties, SecurityManager - family, runFinalization) = explicit OPAQUE rows + D3's clean error. - -### D6 — Thread lifecycle surface (user-authorized F-014 exception) - -F-014 lists "OS threads-as-Java" OUT; the user's 2026-07-16 chat directive -authorizes the minimal lifecycle surface (F-014 Revision history entry rides -this ADR's commit). New `runtime/thread.zig` neutral impl on the future.zig -worker pattern (std.Thread + ThreadGcContext + gc.pin + Io.Mutex/Condition): - -- `(Thread. f)` / `(Thread. f name)` ctor; `.start` (second start → - IllegalThreadState-shaped error), `.join` (0-arg + ms arg), `.isAlive`, - `.getName` / `.setName`, `.setDaemon` / `.isDaemon`, auto-names - "Thread-N". -- **Non-daemon default is JVM-faithful**: a live non-daemon thread registry - is joined at main exit (the DA-flagged silent daemon-true divergence is - thereby dissolved; `setDaemon(true)` = removed from the join-set — - both values real, no accept/error split). -- `Thread/currentThread` returns a threadlocal per-OS-thread object (fixes - the existing `.getName`-hardcodes-"main" lie; pinned by test); statics - `yield`, `onSpinWait`, fields MIN/NORM/MAX_PRIORITY (1/5/10). -- interrupt family (interrupt/isInterrupted/interrupted) → unsupported - error + debt row (a flag-only interrupt that cannot wake a sleeping - thread would be a semantic lie); get/setPriority → OPAQUE rows. - -### D7 — constants + enum statics + Pattern flags - -- Constants: BigDecimal ZERO/ONE/TWO/TEN; Math/TAU; File separator/ - separatorChar/pathSeparator/pathSeparatorChar + createTempFile/listRoots; - Long/Integer/Double BYTES/SIZE (+ Double MIN_NORMAL); Instant - EPOCH/MIN/MAX; Duration/ZERO; LocalTime MIDNIGHT/NOON/MIN/MAX; - LocalDate/LocalDateTime MIN/MAX (+ LocalDate/EPOCH). Java-19+ additions - (TAU, TWO) verified against the local clj oracle before corpus-pinning. -- Host-enum uniform statics via the ADR-0161 registry: `values` (returns a - cljw Java array — `(vec (Month/values))` oracle-checked), `valueOf`, and - `of` where JVM has it (Month, DayOfWeek) — one stroke for RoundingMode/ - ChronoUnit/Month/DayOfWeek. -- Pattern: the 9 flag int constants + 2-arg `Pattern/compile` mapping - {CASE_INSENSITIVE→(?i), MULTILINE→(?m), DOTALL→(?s), COMMENTS→(?x)}; - other flags → unsupported error. The regex value stores DISPLAY source - separately from compiled source so `(str p)` returns the original - pattern (clj fidelity). - -### D8 — java.time fill (bounded) - -`Duration/parse` (ISO-8601), `Duration/of(amount, ChronoUnit)`, -`LocalDate/ofEpochDay`, `LocalTime/ofSecondOfDay` + `ofNanoOfDay`. -TemporalAccessor machinery (from/query/range/adjustInto/with) stays OUT — -OPAQUE rows (JVM temporal-framework depth, not linguistically general). -ZonedDateTime remains D-105 (unbuilt; out of campaign). - -### D9 — data truth + the mechanical member gate (F-013 clause 3) - -- New hidden CLI flag `cljw --dump-host-classes` prints machine-readable - EDN of every registered surface class (fqcn, static fields, methods) — - reusing the completion introspection walk (a future `--list-vars` - building block). -- New gate `scripts/check_compat_members.sh` (wired into the full gate): - (a) every compat_tiers.yaml listed member exists on the registered - descriptor; (b) every descriptor member is listed OR has an explicit - OPAQUE row. compat_tiers.yaml schema gains `opaque_members:` per class. -- compat_tiers.yaml refreshed from dump truth for all 41 surface classes - (fixes both rot directions; MessageDigest's aspirational `getInstance` - row corrected to match D-106 reality). -- Debt: new rows for Thread-interrupt, instance-method fills judged out of - this campaign (StringBuilder set, File instance set, Locale, Matcher, - Socket); Alt C (single class-registry naming SSOT, `class_name.zig` + - `host_class_resolve.zig` + rt.types keys + compat rows derived from one - table) recorded as the named follow-on debt row with this ADR as - prerequisite. - -## Implementation order - -D1+D2 (one unit; smoke diff-oracle early — descriptor dispatch is hot) → -D3 → D4 → D5 → D6 → D7 → D8 → D9 → docs/CHANGELOG → full gate -(--serial-e2e, ALONE) + ubuntunote → release v1.5.0 → tap bump. - -## Alternatives considered (DA fork, verbatim conclusions) - -- **Alt A (smallest-diff)** — keep all keys/fqcns; fix leaks point-wise by - routing every egress through `displayName`; add only the resolveClassValue - fallback + features. Rejected: institutionalises the spelling zoo behind a - translation layer every FUTURE egress must remember (the StringBuilder - leak exists precisely because one egress forgot) — Smallest-diff bias in - canonical form; leaves the two-descriptor `=` problem unsolved (F-002). -- **Alt B (adopted)** — D1 + descriptor merge as one unit; identity - equality becomes structural (one canonical descriptor per class), fixing - `=`, instance?, resolve, completion, and the AOT wire with zero routing - arms; deletes ~8 lazy descriptor fields. Cost: substantially larger diff - (time/*.zig lifecycle reconciled to the registerExtension gpa contract) — - recommended anyway citing F-002. -- **Alt C (wildcard)** — one `class_registry.zig` owning ALL class-name - truth (native tags / surfaces / markers / siblings / OPAQUE as kind-tagged - rows), from which class_name.zig, resolution, completion, and the yaml - recognition rows derive. Attacks the root cause of the zoo; largest - surgery; recorded as the follow-on debt row (prerequisite: this ADR), - not this cycle's scope. -- **fqcn-string-based `=`** (draft's open D2 branch) — struck as unsound: - no `.type_descriptor` equality/hash arm exists; string-`=` without a hash - arm breaks classes-as-map-keys; the ref_cache interning invariant already - guarantees identity correctness for a canonical descriptor. - -## Consequences - -- `(class x)` name universe becomes two-regime and stable: simple for - cljw-native/user/exception/marker types (AD-003), JVM FQCN for - Java-surface-backed classes (clj-faithful; matches the already-pinned - stream/enum behaviour). -- `(= java.util.Date (class d))`, `(instance? java.time.Instant i)`, - `(resolve 'java.lang.System)`, bare `System`/`Math`/`Thread`/ - `StringBuilder` as values, and `(group-by class …)` with host classes all - work — including across AOT round-trips. -- Every unimplemented-member hit becomes a precise, clj-shaped diagnostic - instead of "No namespace". -- compat_tiers.yaml member truth is machine-guaranteed from this ADR on. -- Breaking: `cljw build` artifacts from ≤1.4.0 are version-rejected (v8 - envelope); `(class (java.util.Date. 0))` prints `java.util.Date` (was - `Date`) — corpus pins updated, AD-003 amended. - -## Affected files (representative, not exhaustive) - -`src/eval/analyzer/analyzer.zig`, `src/eval/analyzer/special_forms.zig`, -`src/runtime/host_class_resolve.zig`, `src/runtime/class_name.zig`, -`src/runtime/type_descriptor.zig`, `src/runtime/runtime.zig`, -`src/runtime/java/**` (~25 fqcn flips + System/Thread/Pattern/File/Math/ -BigDecimal/time surfaces), `src/runtime/time/*_value.zig` + `date.zig` -(descriptor merge), `src/runtime/io/stream_classes.zig` + -`host_stream.zig` (print_stream kind), new `src/runtime/thread.zig`, -`src/runtime/error/catalog.zig`, `src/app/cli.zig` (--dump-host-classes), -`scripts/check_compat_members.sh` (new gate), `data/compat_tiers.yaml` -(refresh + opaque_members schema), `.dev/accepted_divergences.yaml` -(AD-003 amendment + new ADs), `.dev/project_facts.md` (F-014 revision), -`.dev/debt.yaml`, envelope version (v8), corpus/e2e updates. - -## Revision history - -- 2026-07-17: Accepted. DA fork output folded in (Alt B); campaign started. diff --git a/.dev/decisions/0175_worker_registration_safepoint.md b/.dev/decisions/0175_worker_registration_safepoint.md deleted file mode 100644 index f95519ec3..000000000 --- a/.dev/decisions/0175_worker_registration_safepoint.md +++ /dev/null @@ -1,178 +0,0 @@ -# ADR-0175 — Worker registration is a GC safepoint (spawn-to-register window) - -- **Status**: **ACCEPTED 2026-07-29.** -- **Deciders**: autonomous loop + one Devil's-advocate fork (general-purpose, - fresh context, F-006/F-002/F-011/F-009 envelope; output reflected verbatim - in Alternatives considered). -- **Supersedes / relates**: ADR-0090 (the STW safepoint rendezvous this - extends), ADR-0150 + am1 (fabrication region / park semantics — sibling - park-site rule), D-244 #4, D-548 (the load-flake ledger this explains a leg - of), D-418/D-559 (the two previously-fixed agent-path GC races). - -## Context - -The scheduled full-gate CI on x86_64-linux (run 30333558512, 2026-07-28) failed -`e2e_phase16_gc_torture` with SIGABRT (exit 134) in the alloc-torture agent -block (`agent_alloc 'drain'`), on a commit whose previous six nightly runs were -green — an intermittent race, not a regression. Reproduced on a 4-vCPU-pinned -x86_64 host at ~1.5%/run; a `-Dprofile` build captured the crash four times -with one identical signature: - -> General protection exception in `root_set.nextThreadRoots` -> (the thread-roots walk), on the MAIN thread, inside the STW collect injected -> by `agent.tortureCollectInWindow` during `send`. - -Root cause: **`safepoint.stopWorld` computes its park-rendezvous target from -`registered_count`, but a worker thread that has been SPAWNED and not yet -REGISTERED is invisible to it.** The agent drain case churns drainers (each -queue-empty exit + next send spawns a fresh one) while the alloc-torture -injects a collect at every enqueue, so this interleaving fires readily on a -slow/loaded runner: - -1. `send` #k: queue empty → spawn drainer D. `std.Thread.spawn` returns; main - continues while D is still starting up (OS scheduling delay — the reason - this is a 4-vCPU-Linux flake and a 10-core-Mac never). -2. `send` #k+1: the injected collect runs `stopWorld` → `registered_count` is - 0 (D not yet registered) → target 0 → returns immediately → the collector - begins the root walk + sweep. -3. D comes to life, `registerThread` succeeds (it never looked at - `gc_requested`), D pops an action and runs it — pushing binding/eval frames - that the collector's `nextThreadRoots` is concurrently reading through the - published TLS slot pointers (torn frame chain → GP fault, the observed - crash), while the popped action itself has left the queue's traceGc range - (swept-in-use → the drainer-side UAF variant, also observed as a heap-address - SIGSEGV). - -The walk's stated safety justification — "the safepoint guarantees no -concurrent register/unregister during collect" — was false for exactly this -window. Every prior fix in this family (D-418 enqueue-window rooting, D-559 -park-honors-fabrication) assumed the rendezvous itself was sound; this is the -rendezvous's own hole. - -## Decision - -**Registration is itself a GC safepoint, and the registration handshake is -seq_cst.** Concretely (all landed in this arc): - -1. **`registerThread` parks the newborn** (root_set.zig): after publishing its - slot + count (under `registry_mutex`) and releasing the mutex, the worker - loads `gc_requested` (seq_cst) and calls `safepoint.park()` if set — so a - worker born while a collect is arming/running quiesces before its first - mutator step. A freshly registered ctx publishes only empty TLS roots, so a - collector already mid-walk reading its slot sees an empty contribution. -2. **Seq_cst Dekker pairing**: `stopWorld`'s `gc_requested.store(true)` + its - `registeredCountLockFree()` read, and `registerThread`'s - `registered_count.fetchAdd` + its `gc_requested` load, are all seq_cst. - The count add sits under `registry_mutex` and the collector's count read - under `sp_mutex` — no common lock, so without seq_cst the both-sides-miss - interleaving (store-buffer/Dekker) is real. Under seq_cst either the - collector waits for the newborn's park, or the newborn parks at - registration — never both-miss. **Do not relax these four orderings** — - both sides carry comments naming the pair. -3. **Atomic registry-slot access**: `threadContextAt` / `markRegisteredTxs` - read slots with `@atomicLoad(.acquire)`, register/unregister write with - `@atomicStore(.release)`. A newborn may legally publish its slot mid-walk - (it then parks); mid-collect slots only transition null→ctx (unregistration - is impossible mid-collect — every already-registered worker is parked or - blocked-counted for the rendezvous to have completed). -4. **The `TooManyThreads` fallthrough is closed** (the DA fork's finding — the - same corruption class, deterministically reachable at the 64-worker cap, - e.g. `(dotimes [i 100] (future …))`): `future.worker` and `thread.worker` - previously ran their thunk UNREGISTERED when registration failed. Now: a - future realises as `realised_error` with nil cached (no allocation — an - unregistered thread may not touch the GC heap; `errorValue`'s new nil-guard - routes `deref` to the generic `future_thunk_failed` raise), and a Thread - reports one non-allocating stderr line + falls through to its done-epilogue. - The agent drainer already bailed correctly (releases the drainer slot; the - queued actions wait for the next send-spawned drainer). -5. **Shared worker-context builder**: `root_set.workerContext(tx_slot)` builds - the standard 5-slot ctx from the calling thread's own TLS; the three worker - families (agent drainer / future / Thread) use it, so a future worker family - cannot silently drop a slot. - -### Invariants this ADR makes explicit (previously unwritten) - -- **Pre-registration code touches only spawner-pinned objects + pure bit-ops.** - Every spawn site `gc.pin`s the object the worker will address before - `Thread.spawn` (agent/future/thread all do); under F-006 non-moving GC that - makes the newborn's pre-register reads safe. A new spawn site must keep this. -- **No lock may be held at a `registerThread` call site** (the registration - park would strand it). True at all sites today; documented as a contract on - `registerThread`. -- **On `TooManyThreads` the worker must NOT run its mutator body**, and its - failure report must not allocate on the GC heap. - -## Verification - -- New deterministic unit test (safepoint.zig, "registerThread during an active - STW collect parks the newborn until resumeWorld"): red pre-fix (the newborn - observably runs between stopWorld-return and resumeWorld), green post-fix. -- The seq_cst half is not deterministically testable; the statistical guard is - the e2e repro: pre-fix `agent_alloc 'drain'` crashed 3/221 and 4/600 (two - campaigns) on a 4-vCPU-pinned x86_64 host; post-fix the same harness must run - clean (recorded in D-548 on landing). -- The existing `phase16_gc_torture` agent block remains the standing e2e guard - (it is what caught this on CI). - -## Alternatives considered (DA fork output, condensed faithfully) - -The fork's leading finding: **no alternative avoids the core mechanism** — every -shape ends up needing the newborn to check `gc_requested` at its first -GC-visible act; the alternatives only vary the fence spelling and walk -protection. No F-NNN conflict exists in any shape. - -1. **Smallest-diff — collector-held `registry_mutex` + rendezvous - re-validation**: collector re-reads the count under `registry_mutex` after - the rendezvous, re-waits if it grew, and holds the mutex across walk+sweep; - registration simply blocks. Better: one synchronization vocabulary (no - seq_cst reasoning), the old quiescence comments become true again. Breaks: - the re-validation loop bounces between `sp_mutex` and `registry_mutex` - (choreography arguably worse), creates a latent hold-order hazard against - `noteWorkerLeft`, and does nothing for the `TooManyThreads` hole. The naive - no-re-validation variant deadlocks (a tiny-action worker blocks in - `unregisterThread` on the held lock) — rejected. -2. **Finished-form-clean (ADOPTED)**: the mechanism above + fallthrough - closure + shared ctx builder + documented invariants. The fork also - recommended a safe-build assert in `gc_heap.alloc` ("allocating thread is - main or registered") — deferred to **D-566**: the survey found in-tree unit - tests (e.g. lazy_seq's force-race workers) that alloc from unregistered - test threads, so the assert needs a test-audit sweep first; nrepl/http are - serial on the calling thread, so production is assert-clean today. -3. **Wildcard — spawner-side accounting**: (a) a pending-registration counter - `stopWorld` waits on self-deadlocks when the collector IS the spawner - inside the window (exactly the reproduced topology: the injected collect - fires between counter-increment and `Thread.spawn`); (b) register-before- - spawn is structurally impossible (the ctx holds pointers to the newborn's - threadlocals) and its placeholder-adopt repair re-derives the registration - park plus extra machinery. Both rejected; the salvageable boilerplate- - collapse instinct became the shared ctx builder. - -## Consequences - -- STW pause acquires no new waits: the collector never waits for unborn - threads (they cannot touch the heap); it only refuses to let them START - mid-collect. -- `registeredCountRelaxed` renamed `registeredCountLockFree` (it is seq_cst - now; the old name lied). -- D-548 exposure (a) — the future/promise SIGABRT residual — was a plausible - beneficiary, but the post-fix protocol run (4 rounds of 8× parallel - `phase14_future_promise_delay` on 8-core linux) still reproduced 6/32 at - the pre-fix baseline rate: **(a) is a distinct race and stays open**, - signature unchanged (recorded in the row). -- Deferred hardening: **D-566** (alloc-side unregistered-mutator assert after - a test-audit sweep). The dormant worker-initiated-collect gap (main thread - unregistered — root_set.zig `is_registered_worker` doc) is pre-existing and - out of scope here. - -## Affected files - -- `src/runtime/gc/root_set.zig` — registration safepoint + contract docs + - `workerContext` + atomic slots + walk-safety comment rewrite + rename. -- `src/runtime/concurrency/safepoint.zig` — seq_cst arm + doc + the new - deterministic unit test. -- `src/runtime/future.zig` — fallthrough closure + `errorValue` nil-guard + - ctx builder adoption. -- `src/runtime/thread.zig` — fallthrough closure + ctx builder adoption. -- `src/runtime/agent.zig` — ctx builder adoption. -- `.dev/gc_rooting.md` — E3/E4 rows note the registration safepoint. -- `.dev/debt.yaml` — D-548 recurrence note; new D-566. diff --git a/.dev/decisions/0176_live_worker_teardown_guard.md b/.dev/decisions/0176_live_worker_teardown_guard.md deleted file mode 100644 index 0b02280ea..000000000 --- a/.dev/decisions/0176_live_worker_teardown_guard.md +++ /dev/null @@ -1,184 +0,0 @@ -# ADR-0176 — Live-worker teardown guard (D-548(a) exit-race hard-exit) - -- **Status**: **ACCEPTED 2026-07-29** (DA-fork review incorporated; both - verification campaigns recorded in Revision history). -- **Deciders**: autonomous loop + one Devil's-advocate fork (general-purpose, - fresh context, F-006/F-002/F-011/F-009 envelope). -- **Supersedes / relates**: ADR-0174 D6 (the daemon-Thread hard-exit idiom this - generalises; the 2026-07-17 ubuntunote teardown-race fix), ADR-0175 (the - sibling registration-safepoint fix; its verification campaign surfaced this - as a distinct race), D-548 (exposure (a) — this ADR's discharge), D-244. - -## Context - -D-548(a): `e2e_phase14_future_promise_delay` intermittently SIGABRTs (exit -134) under parallel load — 6/32 runs on an 8-core x86_64 Linux host, the -failing case ROAMING per run, recorded since 2026-06-25 and surviving every -prior GC-race fix (D-418, D-559, ADR-0175). A stderr-capture harness (the -per-invocation wrapper on ubuntunote) finally caught the abort text: - -> `malloc_consolidate(): unaligned fastbin chunk detected` -> `corrupted double-linked list` - -— glibc allocator-metadata corruption, on expressions that leave a -**fire-and-forget worker** behind (`(future 1)` never deref'd; a -`(future (deref d))` whose value nobody waits for). - -Root cause: **process-teardown vs detached-worker race.** After the program -body, `runSource` runs `thread.joinAllNonDaemon` — which joins only the -`(Thread. f)` registry and hard-exits only for live *daemon* Threads — then -its defers run `env.deinit()` + `rt.deinit()`, freeing the GC heap and every -gpa-backed structure. Detached future workers and agent drainers are invisible -to that barrier, so a worker still running its thunk — or, subtler, still in -its exit EPILOGUE (`join`/deref wake at the result broadcast, which happens -BEFORE the worker's `gc.unpin` + `unregisterThread`) — mutates freed memory -and corrupts the allocator. Roaming case selection (whichever invocation's -worker lags), load sensitivity (worker epilogues delayed on a busy host), and -the Linux bias (glibc's metadata checks abort; macOS malloc is quieter) all -follow. - -## Decision - -Generalise the established daemon hard-exit idiom: **graceful Runtime teardown -is legal only when no worker thread is live; otherwise flush and -`std.process.exit(0)`** (abandon the heap; the OS reclaims — observably -identical to today's immediate exit, minus the corruption). - -1. **`root_set.live_workers`** (seq_cst atomic): the spawner increments BEFORE - `Thread.spawn` (rolling back on spawn failure) at all four spawn sites - (future, agent send-spawn, agent restart-spawn, Thread.start); the worker - decrements as its very LAST action (its first-declared defer — after the - result store, `gc.unpin`, and `unregisterThread`). Distinct from - `registered_count`: it covers the pre-register and post-unregister spans. - The worker's seq_cst decrement pairs with the boundary's seq_cst read, so - count==0 proves every epilogue completed. -2. **`thread.joinAllNonDaemon` half 3**: after joining non-daemon Threads and - the daemon check, `if (live_daemon or liveWorkerCount() > 0)` → flush + - hard exit. The count can only drop after the program body (only live - workers spawn workers), so a 0 read proves teardown is race-free. -3. **Every app success-exit boundary calls the barrier** (renamed - `thread.exitBarrier` — it now owns three duties: join non-daemon Threads, - daemon hard-exit, worker-quiescence-or-hard-exit): `runSource`, - `runSourceCompare` OK-branch, `repl.run`, `nrepl.run`'s exit return - (which deletes `.nrepl-port` first — a hard exit skips the defer), - `builder.buildArtifact`, `builder.tryRunEmbedded`. `noreturn` error paths - (`renderAndExit`/`std.process.exit`) skip teardown and are safe as-is. -4. **The teardown chokepoint is guarded too** (the DA fork's decisive - finding): error-UNWIND paths — a broken stdout pipe mid-print, any failed - post-eval `try` — reach `defer rt.deinit()`/`env.deinit()` WITHOUT - passing an app barrier, reproducing the race. Both deinits now run - `awaitQuiescentWorkers(50ms)` first: epilogue stragglers quiesce in µs → - real teardown; a timeout (= a genuinely running thunk) → **skip all - freeing and return** (abandon the heap; the imminent process exit - reclaims; exit codes and the remaining unwind proceed normally). No - `std.process.exit` inside library code, so `zig build test` can never be - truncated. -5. **Barrier prefers graceful teardown** (DA Alt-2 shape, one counter): the - barrier waits the same bounded 50ms — the common case (a completed - thunk's µs epilogue) proceeds to real teardown so Debug leak reports stay - meaningful and deterministic; only a genuinely running thunk hard-exits. -6. **Deterministic regression canaries** via `CLJW_TORTURE_TEARDOWN_DELAY_MS` - (test-only, parsed at CLI startup): `Runtime.deinit` sleeps AFTER its - guard and before freeing, widening the window so a guard regression - aborts deterministically. E2e cases `exit_pending_future` (the AD-056 - pin), `exit_teardown_injected_running`, `exit_teardown_fireforget_epilogue` - in `phase14_future_promise_delay.sh`. - -**Deliberately preserved divergence — now LEDGERED as AD-056** (per the DA -fork's process finding: a header comment is not the ledger): on the JVM a -fire-and-forget future keeps the process alive (non-daemon pool threads; -`shutdown-agents`); cljw exits immediately. This ADR only makes that exit -memory-safe. The JVM-exact wait-at-exit flip is *more* F-011-compliant -(including its hang on an undelivered promise — parity, not a defect) but -would hang every fire-and-forget script and needs shutdown-agents plumbing; -rejected here as a product choice, recorded in AD-056 with the -`exit_pending_future` pin. - -## Verification - -Two layers (the DA fork corrected the draft's "statistical only" claim: a -WORKER-side sleep hides the race, but a TEARDOWN-side delay amplifies it): - -- **Statistical, the primary measure** — the capture harness, same host, - same 8×-parallel protocol (the pre-fix rate is load-dependent, so the - harness must be replicated exactly): pre-fix 6/32 aborts (historically - 3/8, 1/8, 2/8 rounds); post-fix bar 0 over ≥ 64 runs. Results in Revision - history + D-548. -- **Deterministic canaries** — the `CLJW_TORTURE_TEARDOWN_DELAY_MS` - injection (Decision item 6) turns a guard regression into a reliable - abort; the three e2e cases run in every gate on every platform. - -## Alternatives considered (DA fork output, condensed faithfully) - -The fork reviewed the six-barrier draft against the actual tree and found it -**narrowed the race rather than closing the class** — its Finding 1 (error- -unwind paths reach `defer rt.deinit()` past every barrier: broken stdout pipe -in `printResult`, failed `try`s in repl/nrepl/builder post-eval code) drove -Decision items 4-5. Its other adopted findings: the `.nrepl-port` stale-file -divergence on hard exit (fixed — cleanup before the barrier); "deterministic -red is unattainable" was overstated (the teardown-side delay injection is a -legitimate amplifier — adopted as the canary knob); the immediate-exit -divergence must be an AD row with a pin, not a header comment (AD-056); and -the `joinAllNonDaemon` name lied about its three duties (renamed -`exitBarrier`). - -1. **Smallest-diff — six app barriers with `std.process.exit` only** (the - original draft): leaves every post-eval `try` an open re-entry point for - the bug and makes teardown-path selection racy (nondeterministic leak - reports). Rejected as the Smallest-diff-bias shape. -2. **Finished-form (ADOPTED, combined Alt 1+2)**: the counter stays; the - guard moves to the teardown chokepoint (`Runtime.deinit`/`Env.deinit` - skip-freeing-and-return — no exit() in library code, no test truncation), - with a bounded epilogue-wait so the dominant case gets REAL graceful - teardown; the barrier keeps the hard exit for genuinely running thunks. - The fork's full two-counter variant (`thunks_running` separate from - `live_workers`) was simplified to one counter + bounded wait: epilogues - cannot block, so a 50ms timeout separates the two populations with the - same outcomes and one fewer ordering proof. -3. **Wildcard — shutdown as permanent stop-the-world** (reuse - `safepoint.stopWorld`, never resume, then free with mutators provably - parked): the only shape that is graceful ALWAYS, but it still needs the - spawn-side counter (STW sees only registered workers), requires blocked- - in-condWait workers to count as parked or it deadlocks, and freeing - `FutureCell` condvars with live waiters is exactly the glibc-metadata - territory being escaped. Enumerated and rejected. - -Not adopted (recorded): the fork's suggestion of a Debug -`assert(liveWorkerCount()==0)` in the freeing path is subsumed by the -skip-guard itself (the guard handles the case the assert would trip on). Its -(d) note — nREPL mid-session hazards (per-message `scratch_arena.reset` vs a -still-running worker's thunk references; session-close frees) are OUT of this -ADR's scope and are NOT claimed fixed; if symptoms appear they are a separate -D-row, not a D-548 reopen. - -## Consequences - -- The fpd suite's `ncpu < 4` SKIP (which cites exactly this SIGABRT class) - is removed once the campaign passes — the 3-vCPU hosted runner runs the - suite again; the nightly is the watchdog. Exposure (b) (pmap wall-clock - assert, `phase14_parallel_seq`) is a separate timing-envelope issue and - keeps its gate; D-548 re-scopes to (b)-only. -- A worker that outlives the program body is now ALWAYS killed by hard exit - (previously: killed by teardown+exit, sometimes corrupting on the way). - Its side effects were never guaranteed; that contract is unchanged. -- `Runtime.deinit` remains a true teardown for tests; the guard lives only at - app exit boundaries. - -## Affected files - -- `src/runtime/gc/root_set.zig` — `live_workers` counter + notes API. -- `src/runtime/future.zig`, `src/runtime/agent.zig`, `src/runtime/thread.zig` - — spawner-increment + worker final-decrement. -- `src/runtime/thread.zig` — `joinAllNonDaemon` half 3. -- `src/app/{runner,repl,nrepl,builder}.zig` — barrier at success exits. -- `test/e2e/phase14_future_promise_delay.sh` — ncpu gate removal (post-camp). -- `.dev/debt.yaml` D-548 — (a) discharge + (b) re-scope. - -## Revision history - -- 2026-07-29: drafted alongside the fix; six-barrier mechanism verified on - the capture harness — fpd 0/64 (pre-fix 6/32) + gc_torture drain 0/100 - regression-clean on the 4-vCPU-pinned x86_64 host. -- 2026-07-29 (same cycle): DA-fork review incorporated (chokepoint guards, - graceful-preferred barrier, rename, AD-056, canaries) → ACCEPTED; final - confirmation campaign on the complete shape recorded in D-548. diff --git a/.dev/decisions/0177_capability_claims_are_gated.md b/.dev/decisions/0177_capability_claims_are_gated.md deleted file mode 100644 index edd1cb686..000000000 --- a/.dev/decisions/0177_capability_claims_are_gated.md +++ /dev/null @@ -1,288 +0,0 @@ -# ADR-0177 — Capability claims carry a status marker, and the marker is gated - -- **Status**: Proposed → **Accepted** (2026-08-04) -- **Supersedes**: nothing. Amends ROADMAP §1.1, §1.2, §1.3, §8.1, §8.3, §10.3 - per ROADMAP §17. -- **Related**: ADR-0172 (binary-size budget + `size_claims`), ADR-0135 - (component-as-namespace), D-552 (cljw-self-as-Wasm, research-first), - F-002, F-003, F-010, F-015, zwasm lesson - `2026-08-03-ungated-negative-doc-claim-rotted-into-a-lie`. - -## Context - -The 2026-08-04 cross-project audit measured the shipped artifact and compared -it against what the project says about itself. Two classes of claim had rotted, -and the boundary between rotted and correct was exact: **every document a gate -read was true, and every document no gate read was false.** - -The numeric half was fixed in `77f690a3` (the `size_claims` gate now reads -`README.md`, `docs/landscape.md`, and `bench/RELEASE_METRICS.md`, and each -` MB` figure either matches the built binary within 10 % or carries an -explicit `` exemption). This ADR is the capability half. - -What was measured, at HEAD `e897cbfc`: - -- `build.zig` contains no `wasm32`, `wasi`, or `freestanding` target. The - `-Dwasm` flag *embeds zwasm* — it is about consuming Wasm, and is easy to - misread as producing it. -- `cljw build` (`src/app/builder.zig`) emits a native binary with an embedded - bytecode payload (Deno-style trailer), not a `.wasm`. -- Against that, the ROADMAP asserted, in the present tense: - - §1.1 — "**Edge execution**: runs on Cloudflare Workers / Fastly / Fermyon - Spin and other Wasm Component Model hosts". - - §1.2 axis 1 — "v2 makes Wasm Component a first-class output". - - §1.3 — users "who want to call a Clojure runtime as a component". - - §8.1 — "**Two artifacts from one source tree**", artifact 2 being - `cljw.wasm`. - - §8.3 — a Phase-14 row reading "Component build begins". - - §10.3 — a "Wasm cold start < 50 ms" performance target for that artifact. -- Two further stale figures: "measured 9.5 MB at v1.3.1" appeared **twice** - (§1.1 and §10.3) while the enforced ceiling is 8,800,000 bytes and the built - binary is 7,368,808; and §1.1's "shrink code volume to 30–40 % of v1 (89K - LOC)" against a measured 124,712 LOC of Zig (~119 % of v1 even after - excluding the 18,534 generated Unicode lines). - -The failure is not that someone wrote something false. It is that the ROADMAP -is the project's **single source of truth** (CLAUDE.md § References) and no -mechanism connected any sentence in it to the code. - -## Decision - -**Retire the claim; keep the goal.** Three parts. - -### 1. A status marker on capability claims - -Two inline markers, mirroring `SIZE_EXEMPT_MARKER` exactly so the convention is -one idea rather than two: - -- `` — this exists and is exercised by a test. -- `` — this does not exist; `D-NNN` owns it. - -### 2. `scripts/check_capability_claims.sh`, wired into the full gate - -Two fail-closed invariants: - -1. Every `claim:planned:D-NNN` names a debt row that **exists and is still - open**. A planned claim whose owner has been discharged is a claim with - nobody left to build it — which is precisely how "planned" becomes - "abandoned, still advertised". -2. A curated list of phrases that were found asserting unbuilt capabilities - may not appear **unmarked** in a claim-bearing document. - -It is deliberately **not** a general prose checker. A heuristic -"is this sentence a capability claim?" scanner over a 1,600-line ROADMAP would -false-positive constantly and be disabled, which is worse than no gate. The -curated list grows when an audit finds a new rotted phrase — the list is a -record of actual failures, not a guess at future ones. - -### 3. The ROADMAP amendments - -§1.1, §1.2 axis 1, §1.3, §8.1, §8.3 and §10.3 now state the shipped capability -(load a Wasm core module or component in-process and call it like a namespace) -and mark cljw-as-a-Wasm-guest as an **aim with an owner** (D-552), not a -capability. §10.3's "Wasm cold start" row is removed — it targeted an artifact -that does not exist, and returns with D-552. The LOC target is retired in -favour of the property it proxied (§1.2 axis 3, readable end-to-end); the -binary-size row is marked superseded by ADR-0172's live ledger. `.dev/ROADMAP.md` -joins `SIZE_CLAIM_FILES`, so its numeric figures are gated like README's. - -**This ADR changes no scope.** F-010 (2026-05-29, user-directed) already -de-prioritised widening the Wasm FFI surface, and D-552 (2026-07-02, -user-directed) already made cljw-self-as-Wasm research-first with -implementation user-nod-gated. The sequencing was the user's; this is -documentation catching up to it. - -## Alternatives considered - -*(Devil's-advocate fork, fresh context, 2026-08-04. Reproduced as returned.)* - -### Leading finding — no F-NNN blocks this amendment, but one framing does - -No alternative below violates an F-NNN, and neither does the draft. F-001 -(embed zwasm) and F-016 (component model always on as a *consumer*) both -concern cljw **consuming** Wasm; neither asserts anything about cljw **being** -Wasm. F-010 explicitly de-prioritises wasm-FFI breadth. So the edge claim is -not F-protected. - -The one framing that *would* trip a constraint is **deciding** the question. -F-003 reserves structural decisions for the owning unit, and D-552 names -exactly the structural content at issue ("efficient memory management under -Wasm, the GC/allocator story in linear memory, sandbox-first assumptions"). An -amendment that reads as *"cljw will not run as a Wasm guest / that is -ClojureWit's job"* is decision-seizure on an unbuilt structural plan and -brushes F-003; an amendment that retires the false **present-tense claim** -while preserving the **aim** does not. The distinction is load-bearing and -should appear in the Decision text, not just be implied: **retire the claim, -keep the goal.** Corollary: the §1.1 opening sentence ("with first-class edge -and Wasm support") is the project's identity statement and is user-owned -territory — leave it (the Wasm half is true in the interop sense that ships). - -Two evidence corrections to the draft's Context before it is stamped: - -- **The Cloudflare size argument is weak and should be dropped or softened.** - 3 MB gzipped vs "the shipped native binary is 7,368,808 bytes" compares two - different artifacts. A hypothetical `wasm32-wasi` cljw would not carry the - native-codegen paths or (necessarily) the embedded JIT engine, and would be - gzipped. Citing it as justification over-claims and hands a future reader a - refuted premise. -- **The LOC figure needs its denominator stated.** 124,712 total Zig; minus - the 18,534 generated Unicode lines that is ~106K = ~119 % of v1's 89K. The - target fails on either denominator, so the conclusion holds — but restating a - *new* number in an ungated doc simply re-arms the same rot. Prefer retiring - the numeric LOC target in favour of the property it proxies (readable - end-to-end, §1.2 axis 3), or gating it. - -### The four interrogated questions, answered - -**Is retiring right, or is "keep the goal, mark it planned" the honest fix?** -Both, in different registers. F-003 protects deferral of *decisions*; it says -nothing in favour of keeping a false *assertion of fact* in the present tense. -"Runs on Cloudflare Workers / Fastly / Fermyon Spin" is not a plan, it is a -claim, and `build.zig` has no `wasm32` target (`b.standardTargetOptions` + a -`-Dwasm` flag that *embeds zwasm*, line 79 — the flag is about consuming Wasm, -and is easy to misread as producing it). But F-015 equally rules out the lazy -repair: parking it as "PLANNED, Phase N" with no owner is precisely the -blind-Phase-deferral posture F-015 retires. The correct shape is *aim + named -owner*: unbuilt, tracked as D-552, user-nod-gated. - -**Does F-010 make this documentation catch-up rather than a scope change?** -Yes, and the ADR should say so in one sentence. F-010 § What this changes for -the loop ¶1: "wasm FFI breadth is de-prioritised, NOT cancelled… the loop must -not widen wasm FFI surface… before M + a quality-loop pass, unless the user -re-directs." D-552 (2026-07-02, user-directed) then made cljw-self-as-Wasm -research-first and implementation user-nod-gated. The sequencing was therefore -decided by the user in 2026-05-29 and 2026-07-02; this ADR changes no scope and -should disclaim doing so. Framing it as a scope decision would overstate the -loop's authority over a user-owned mission line — the F-003 hazard above. - -**Is the north star being destroyed?** No — and the draft's scope proves it by -accident. §1.2 names **axis 2** (component-as-namespace) as "the north star", -and §9.18 records it **BUILT** (`cljw.wasm/*` over -`src/runtime/cljw/wasm/{engine,component,marshal,surface}.zig`, four real e2e -steps). The unbuilt thing is **axis 1**, and axis 1's cell still reads "v2 -makes Wasm Component a first-class output" — the *same false claim* as §8.1, in -the differentiation table, untouched by the draft. That is the single strongest -objection to the draft as scoped: it would land §8.1 marked NOT IMPLEMENTED -while §1.2 row 1 continues to assert the artifact exists. - -**Convention over three sentences?** Yes, with one-commit-old precedent. -`77f690a3`'s own message: *"Fixing the three claims without widening the gate -would have re-created the condition, so the gate moved with them."* The draft -as scoped does the thing that commit refused to do. Concretely: -`.dev/ROADMAP.md` is **not** in `SIZE_CLAIM_FILES` (`README.md -docs/landscape.md bench/RELEASE_METRICS.md`), and the stale "9.5 MB at v1.3.1" -appears **twice** — line 74 and line 1586. Hand-fixing §1.1 leaves 1586 -rotting, re-creating the failure inside one document. - -### Alternative A — smallest-diff: status-token the claims in place, change nothing else - -Keep §1.1's bullet and §8.1's artifact 2 verbatim, prefixed with `PLANNED -(unbuilt — D-552)`; correct the three figures; touch nothing else. - -*Better than the draft*: zero movement on mission scope, so the F-003 surface -is nil; smallest re-activation cost if the user later nods D-552; no new -machinery to maintain. -*Breaks*: nothing checks the token, so this is the ungated-claim pattern again; -a "PLANNED" with no owner is the F-015-retired posture unless D-552 is named; -and it leaves the identical claim standing in §1.2 axis 1, §1.3 bullet 2 -("Wasm-ecosystem users who want to call a Clojure runtime as a component"), -§8.3 rows *WASI 0.2 → "Component build begins"* and *WasmGC*, §9 tracker line -1014, and §10.3's "Wasm cold start < 50 ms" target for a nonexistent artifact. - -### Alternative B — finished-form-clean (recommended, F-002): one sweep plus a claim gate - -(i) Sweep **every** site carrying the claim, not three: §1.1 bullet 2, §1.2 -axis 1 cell, §1.3 bullet 2, §3's in-scope "Wasm Component pod loading", §8.1 -artifact 2, §8.3's Phase-14 "Component build begins" row, §9 tracker line 1014, -§10.3's "Wasm cold start" row. (ii) Add `.dev/ROADMAP.md` to -`SIZE_CLAIM_FILES` so lines 74 and 1586 are gated like README's, with -`` on the deliberate historical rows. (iii) Introduce an -inline `` / `` marker plus -`scripts/check_capability_claims.sh`, wired as a `run_all.sh` step — mirroring -`SIZE_EXEMPT_MARKER` exactly, so the exemption is greppable and never silent. - -*Better*: closes the class rather than the instance (F-002); makes the ROADMAP -agree with `docs/landscape.md`, which received precisely this treatment 24 -hours earlier and currently **contradicts** the ROADMAP; satisfies F-015 by -giving every unbuilt item a debt owner instead of a phase number; and the -numeric rot cannot recur. -*Breaks*: substantially larger diff (new script, new gate step, marker sweep, -likely README/docs follow-on) — recommended anyway per F-002, cycle budget is -not a project constraint. The present-tense-capability grep is heuristic and -will need scoping (restrict to §1/§8/§9-tracker/§10 lines carrying a bolded -capability name) or it will false-positive across a 1600-line document. Every -future capability line then needs a marker — a real maintenance tax, and the -honest counter-argument to (iii). - -### Alternative C — wildcard: the ROADMAP stops carrying capability status at all - -Generate a `docs/CAPABILITIES.md` from sources that already exist and are -already gated — `data/compat_tiers.yaml`, `data/feature_deps.yaml`, -`.dev/debt.yaml`, the `test/e2e/phase16_wasm_*.sh` step list, and `build.zig`'s -flags. §1/§8 retain only aims and rationale (which cannot rot, being -non-factual) and link out for status. - -*Better*: rot becomes structurally impossible rather than merely gated — the -doc cannot disagree with the code because it is derived from it; permanently -kills the "N documents disagreeing with the binary" class that the 2026-08-04 -audit found; and it feeds the already-planned `cljw --list-vars` capability -introspection from one source. -*Breaks*: it must first define "shipped", and §9.18 shows that is genuinely -subtle — `-Dwasm` is "gated `-Dwasm` opt-in **by design** (lazy dep), NOT -unimplemented", which no mechanical rule infers. A generator is itself code -that can lie. Restructuring the ROADMAP's authority over §1/§8 is a structural -plan whose owner is not this cycle (F-003). And it risks over-rotation: a -ROADMAP is a *plan*, and plans legitimately state intentions; mechanising -status pressures the document toward only-what-exists, which is how a north -star quietly disappears. - -**Recommendation (non-binding): Alternative B**, with the leading finding's -"retire the claim, keep the goal" wording, the two evidence corrections, and an -explicit "no scope change — F-010 and D-552 already sequenced this" sentence in -the Decision. - -## Resolution - -**Alternative B is adopted**, with all three of the fork's qualifications: - -- The wording is "retire the claim, keep the goal". §1.1's opening identity - sentence is left alone as user-owned territory. -- The Cloudflare size argument is **dropped** from the reasoning. It compared - two different artifacts and would have handed a future reader a refuted - premise. -- The LOC target is **retired** rather than restated with a corrected number, - because a corrected number in an ungated sentence re-arms the same rot. The - measured figure is quoted once, here, where it is dated and attributed. - -The fork's own counter-argument to (iii) — that every future capability line -now needs a marker — is accepted as a real cost and answered by scope: the gate -checks a *curated* phrase list plus marker integrity, not every sentence. A -line only needs a marker if it uses a phrase an audit has already caught being -false. - -## Consequences - -- The ROADMAP now distinguishes, mechanically, what cljw does from what it - intends. A reader can trust §1 and §8 again. -- A discharged debt row can no longer silently orphan a "planned" claim. -- Cost: two gate steps instead of one, and a curated list that must grow when a - new class of false claim is found. That growth is a record of real failures. -- `docs/landscape.md` and the ROADMAP now agree with each other and with the - binary. Before this ADR they disagreed with both. -- The v0.1.0-era `10.3` targets table is now partly historical; a future perf - campaign should re-cut it against measured v1.x figures rather than patch it. - -## Affected files - -- `.dev/ROADMAP.md` — §1.1, §1.2 axis 1, §1.3, §8.1, §8.3, §10.3 -- `scripts/check_capability_claims.sh` — new -- `scripts/binary_size_report.sh` — `.dev/ROADMAP.md` added to `SIZE_CLAIM_FILES` -- `test/run_all.sh` — `capability_claims` step -- `docs/landscape.md`, `bench/RELEASE_METRICS.md`, `README.md` — landed in `77f690a3` - -## Revision history - -- 2026-08-04: Status: Proposed → Accepted. Alternative B adopted with the - Devil's-advocate fork's three qualifications (retire-the-claim-keep-the-goal - wording, Cloudflare size argument dropped, LOC target retired rather than - restated). diff --git a/.dev/decisions/0178_placement_index_is_generated.md b/.dev/decisions/0178_placement_index_is_generated.md deleted file mode 100644 index 65f89dbb3..000000000 --- a/.dev/decisions/0178_placement_index_is_generated.md +++ /dev/null @@ -1,415 +0,0 @@ -# ADR-0178 — The placement index is generated from the runtime, not maintained by hand - -- **Status**: Proposed → **Accepted** (2026-08-04) -- **Amends**: ADR-0033 (Clojure-ns placement / naming / polymorphism) — the - *migration schedule* is recorded complete; the *placement rule* (D2-D5) stands - unchanged. ROADMAP §5 layout line + §15.x `status/vars.yaml` note per §17. -- **Related**: ADR-0174 D9 (`check_compat_members.sh`, the same fix applied to - compat_tiers member lists), ADR-0163 (lazy namespace loading), ADR-0177 - (capability claims), D-062 (discharged 2026-05-27), D-094 (`clojure.string/escape`), - F-002, F-003, F-009, F-013. - -## Context - -`data/placement.yaml` was introduced by ADR-0033 as the per-var record of the -Pattern A / B1 / B2 / C-thin placement decision for Clojure-namespace vars, and -`.claude/CLAUDE.md` listed it as an authoritative data source. Measured -2026-08-04 at HEAD `c601aa75`: - -- It covered **49 vars**. `cljw` exposes ~683 publics in `clojure.core` alone, - and 1,326 across all bundled namespaces. Coverage ~4 %. -- Its header cited three helper scripts. **Two never existed** - (`check_placement_schema.sh`, `analyze_clojure_upstream.bb`, - `gen_placement_yaml.bb` — only `check_placement_status.sh` was real). -- **No gate read it.** `test/run_all.sh` had no `run_step` for it. -- Two `leaf_loc` paths pointed at files that do not exist - (`src/lang/primitive/core/core.zig`, `.../core/sequence.zig`). -- It was written in Japanese, against the CLAUDE.md language policy for - configuration. -- Its content had not changed substantively since the 2026-05-25 scaffold. - -And most importantly: **42 rows read `status: transient_zig` — "Zig for now, -`.clj` migration pending" — for vars whose migration had already shipped.** -D-062, the cluster row tracking that migration, was **discharged 2026-05-27** -with `clojure.string/escape` named as the sole residual (now D-094). The code -agrees with the discharge: `clojure/set.clj` carries 12 `(def …)` bodies, -`string.clj` 21, `walk.clj` 11. The plan executed; the ledger did not follow. - -## Decision - -**Generate the index; gate the generation; keep judgement out of it.** - -1. **`(cljw.internal/__dump-placement)`** (new, `src/lang/primitive/core.zig`, - alongside `__dump-host-classes`) emits one line per Var in every registered - namespace's own `mappings`, deriving the implementation home from the root - value's tag: `builtin_fn` → `zig`, `fn_val` / `multi_fn` → `clj`, otherwise - `value`, and `unbound` for a rootless declare. Flags (`private`, `macro`, - `dynamic`, `zig-leaf`, `unsupported`) ride along. - -2. **`scripts/gen_placement.sh`** requires every bundled namespace first — - namespaces load lazily (ADR-0163), so a bare dump sees only the eager set — - then renders `data/placement.yaml`: a sorted, English, generated file with a - do-not-hand-edit banner. Measured output: **1,326 vars across 31 namespaces - — 394 `zig`, 725 `clj`, 201 `value`, 6 `unbound`.** - -3. **`scripts/check_placement_status.sh` becomes the `placement_drift` gate** - (wired into `test/run_all.sh`): regenerate, diff, fail on any difference. Its - previous body — an audit that proposed `status:` flips for the ADR-0033 - migration schedule — is retired with the schedule. - -4. **Intent stays out of the generated file.** "This var's current placement is - not its intended one" is a judgement with a barrier, so it belongs in - `.dev/debt.yaml`. Today that is exactly one row: D-094 - (`clojure.string/escape`). ADR-0033 D2-D5 remains the placement rule the - index reports against. - -### What is deliberately NOT decided here - -**No forward policy about where future `clojure.core` vars should live.** The -draft that opened this cycle wanted to declare "Zig implementation is the -finished form for `clojure.core` hot paths". That is a structural decision over -an unbuilt residual, which F-003 reserves for the owning unit — and it is -redundant anyway: ADR-0033 D5 already says fast-path = Tag switch, slow-path = -`extend-protocol`. - -## Alternatives considered - -*(Devil's-advocate fork, fresh context, 2026-08-04. Reproduced as returned.)* - -### Leading finding — the draft's crux premise is factually wrong (and no F-NNN blocks the clean option) - -No finished-form-clean alternative here requires violating any F-NNN. But before -the alternatives: **two of the draft's load-bearing claims do not survive contact -with the files.** - -**(a) The `ready_for_migration` shortlist is not a plan to move hot primitives -into `.clj`.** All six rows (`count`/`seq`/`first`/`rest`/`cons`/`empty`, -`data/placement.yaml:435-482`) carry `pattern: B1`, and **none has a -`target_loc`**. ADR-0033 D3 defines B1 as "Layer 2 `.zig` 直 intern (`.clj` -不在)" — i.e. **B1's finished form *is* Zig, with no `.clj` file at all**. The -`status: ready_for_migration` on those rows is a *status-vocabulary misuse*: the -inline comment on `count` reads `# = Phase 6.16.a-1 cycle で着地予定 (新規)` — -"scheduled to *land* in cycle 6.16.a-1", i.e. not-yet-implemented, not "awaiting -`.clj` migration". Per the schema's own enum those rows should read `stable` -("最終形 (Pattern B1 等で Zig が最終形)"). ADR-0033 D5 already states the -hot-path rule the draft thinks it is introducing: "fast-path = Tag switch 維持, -slow-path = extend-protocol". **So "retire the plan before it moves `count` into -`.clj` and costs perf" attacks a straw man, and "declare Zig implementation the -finished form for `clojure.core` hot paths" is a restatement of an -already-Accepted ADR clause, not a new decision.** - -**(b) The plan is not unexecuted — it is executed with an un-updated ledger.** -`debt.yaml:3160` D-062 is **Discharged 2026-05-27**, and the discharge text -enumerates set 12 + walk 10 + string 12 B2 shims + GREEN trio + YELLOW pair + -`replace`/`replace-first` Pattern A, with **`clojure.string/escape` as the sole -residual** (tracked as D-094). Code confirms: `src/lang/clj/clojure/set.clj` has -12 `(def …)` bodies, `string.clj` 21, `walk.clj` 11 — real Pattern A composition -and real B2 one-line shims over private `-name` leaves. Yet all 12 -`clojure.set` rows and 42 rows overall still read `status: transient_zig`. The -file is therefore **a stale mirror of a completed migration**, not a live plan -with 42 unexecuted items. That reframes the whole decision: what rotted is the -*maintenance mechanism*, not the *plan*. - -Two further defects the draft did not catch, which strengthen the "it is broken" -case but point at repair rather than deletion: - -- **Dangling `leaf_loc` pointers**: of four distinct `leaf_loc` paths, - `src/lang/primitive/core/core.zig` and `src/lang/primitive/core/sequence.zig` - **do not exist** (the tree has `src/lang/primitive/core.zig`). All three - `target_loc` paths do exist. -- **The named future consumer is already stale**: - `.dev/cw_v0_parity_and_gap_plan.md:143` (G11) says `cljw --list-vars` will read - `compat_tiers.yaml` + `placement.yaml`. There is **no `--list-vars` in - `src/app/`**; `src/runtime/introspect.zig` is the building block and it **walks - `Env`/`Namespace`/`Var`**, not a yaml. So the consumer argument for keeping the - file as-is is weaker than it reads — but it is also an argument that the - *right* placement index is derived from `Env`, not hand-written. - -And one fact that argues against plain deletion: **ROADMAP §15.x records -`.dev/status/vars.yaml` + `generate_vars_yaml.clj` as SUPERSEDED *by* -`placement.yaml`** ("Per-var tracking is covered by `placement.yaml` … The -originally-planned `status/vars.yaml` + generator were not built", -`.dev/ROADMAP.md:2049-2052`). Deleting `placement.yaml` without a successor -leaves per-var placement tracking with **no owner at all** — a hole the roadmap -explicitly closed once already. - -### Alternative 1 — smallest-diff: truthify in place, gate it, and stop calling it an SSOT - -Keep the file and ADR-0033 intact. In one cycle: (i) delete the three phantom -script citations from the header (`check_placement_schema.sh`, -`analyze_clojure_upstream.bb`, `gen_placement_yaml.bb` — replace with a note that -D9's generator route was never built); (ii) flip statuses to code truth (42 -`transient_zig` → `migrated`/`stable` per D-062's discharge text; the six B1 rows -→ `stable`; keep `clojure.string/escape` as the one honest open row pointing at -D-094); (iii) fix the two dangling `leaf_loc` paths; (iv) wire -`scripts/check_placement_status.sh --check` as a `run_step` in -`test/run_all.sh` (it already has a `--check` mode, unused); (v) translate the -header to English per the language policy; (vi) rewrite the -`.claude/CLAUDE.md:591` and `.dev/README.md:37` wording from "SSOT / -authoritative" to "index of the ADR-0033 6.16-arc migration (49 vars), not a -complete `clojure.core` index"; (vii) fix the stale G11 sentence in -`cw_v0_parity_and_gap_plan.md`. **Sub-variant 1b**: since D-062 is discharged, -move the file to `.dev/archive/` as a *closed migration record* and drop the SSOT -claim entirely, keeping only D-094 live. - -- **Better than the draft**: makes zero structural decision — no policy is - declared about where future `clojure.core` vars live, so no F-003 exposure on - the unbuilt residual. Preserves the Pattern A/B1/B2 vocabulary that ADR-0171 - (`:131` "placement.yaml Pattern B2 already interns private leaves directly into - clojure.core"), ADR-0043 (`:128`), ADR-0102 (`:122`, which cites the - compat_tiers/placement role split as *precedent* for its own design) and source - comments (`src/lang/primitive/string.zig:269, 617`) all depend on. Fixes every - defect the audit actually measured. Leaves `--list-vars`'s future free. -- **Breaks**: it re-instates a **hand-maintained parallel index**, which is - precisely the mechanism that produced the rot — nothing derives status from - code, so it will drift again the first cycle nobody remembers to update it (the - exact failure the 2026-05-31 audit found for debt rows). And it does not answer - F-013: an index that covers 49 of 683 publics is not 網羅 from the definition; - it is the set of vars that happened to be in scope in the 6.16 arc. Wiring - `--check` gates *schema validity only* (its `check_mode` merely validates the - status enum) — it cannot catch a `transient_zig` row whose var is already - `.clj`, which is the actual lie class here. -- **F-NNN**: no violation. Weak on F-013 (partial coverage retained as "index"), - neutral on F-009 (ledger survives, still hand-fed). - -### Alternative 2 — finished-form-clean (RECOMMENDED): stop maintaining placement, start *deriving* it, over all 683 publics - -Replace the hand-written 49-row file with a **generated, total placement index -plus a drift gate**, and shrink the hand-written residue to only the -non-derivable field. - -The observation that makes this clean: **placement is a property of the code, and -the code already knows it.** For every var in every `clojure.*` / `cljw.*` ns, -`Env` + the loaded `.clj` sources determine the pattern mechanically — B1 = -interned from Zig with no `.clj` `def`; B2 = a `.clj` `def` whose body is a -single call to a same-ns `^:private`/`-name` leaf; A = a `.clj` `def` composed of -other Clojure vars; C-thin = a `.clj` body with 1-2 branches over a leaf. That is -exactly the classification ADR-0033 D2's B-Q1→B-Q4 flowchart asks a human to -make, and it is decidable from the AST plus the leaf metadata `env.intern` -already carries (ADR-0033 D8: `zig_leaf`, `private`). - -Concretely: extend `src/runtime/introspect.zig` (already documented as the -`--list-vars` building block) with a placement classifier; land `cljw ---list-vars --placement` as the CLI that `cw_v0_parity_and_gap_plan.md` G11 has -been promising; check in the emitted snapshot as `data/placement.yaml` -(regenerated, **all** publics, English, machine-owned with a "generated — do not -hand-edit" banner); add a `run_step` to `test/run_all.sh` that regenerates and -diffs, failing on drift. Keep a small hand-written companion section (or, better, -leave it entirely in `.dev/debt.yaml`) for the only genuinely non-derivable fact: -**"current placement is not the intended finished form"** — today exactly one -var, `clojure.string/escape` (D-094). - -- **Better than the draft**: (1) It answers **F-013** as written — - "definition-derived comprehensive coverage in canonical form, never an ad-hoc - *just make this one pass*": 683/683 instead of 49/683, derived from the - definition (the pattern rules) rather than from what one arc touched. The - draft's own strongest datum (7.2% coverage) is an argument for *completing* the - index, not for deleting it; deleting it is the "ad-hoc" move F-013 forbids in - its own domain. (2) It preserves **F-009**'s only live ledger. F-009 is the - invariant "impl bodies namespace-neutral, Clojure/Java/cljw surfaces thin - above", and it explicitly anticipates this cycle: *"At Phase 12+ the AI loop is - statistically very likely to propose 'inline impl into the surface for fewer - files.' F-009 lets the Devil's-advocate subagent automatically reject these - envelope-violating alternatives."* The placement index is the per-var evidence - that the Clojure surface is thin; killing it removes the observability for a - *confirmed, live* invariant while the invariant stands. (3) It kills the rot - **mechanism** rather than the artifact — no human can make it lie again, which - is the only durable fix (Discipline 1 of `clj_diff_sweep.md`: leave the probe - behind, mechanically re-checkable). (4) It delivers G11 and closes the ROADMAP - §15.x `status/vars.yaml` hole in the same move. (5) It matches the project's own - established shape: ADR-0102 chose "dedicated SSOT + generated/checked Zig table - + gate" over hand-extension, citing the compat_tiers/placement role split as the - precedent — this alternative is that same design applied back to placement - itself. -- **Breaks**: the classifier must handle real shapes — `(def x (fn* …))` rather - than `defn` (which is what `set.clj`/`string.clj` actually use — note `grep - '^(defn'` returns **zero** in all three files), multi-arity `fn*`, macros, the - `rt` → `cljw.internal` rename from ADR-0171, and `def`-with-computed-value. - Misclassification is possible and a wrong generated row is a *new* lie class - (though a gated, diffable one). It costs 1-2 cycles plus a `run_all.sh` step. - Most importantly it **changes what the artifact is**: a generated mirror can - record *what is*, never *what should be* — so the aspirational half of ADR-0033 - (D7's cycle plan, `recall_trigger`) must be explicitly retired into debt rows, - and ADR-0033 needs a Revision-history amendment saying so. That amendment is the - honest version of the draft's "retire the plan": the *migration schedule* is - done (D-062 discharged), the *placement rule* (D2-D5) stays law. -- **F-NNN**: violates none. Strengthens F-013 (網羅 from the definition), F-009 - (ledger complete and machine-true), F-011 (one mechanism, behavioural claim - mechanically re-checkable), F-002 (finished form over the smaller repair). - Larger diff than Alt 1 — **per F-002 that is explicitly not a reason to prefer - Alt 1**. - -### Alternative 3 — wildcard: delete the ledger by making the invariant executable - -Delete `data/placement.yaml` *and* the tracking concept, and re-express the only -load-bearing thing it guarded — F-009's "surfaces are thin, cross-surface calls -forbidden" — as an **executable check for the Clojure-ns surface**, extending the -existing G1-G4 guardrail family (`zone_check.sh`, `check_surface_marker.sh`, -`check_feature_keyword.sh`, `check_host_interface.sh`). The check asserts: every -public Clojure-ns var either (a) resolves to a Zig leaf whose impl lives in a -namespace-neutral `runtime/` home, or (b) is composed solely of other Clojure -vars — and never reaches a `runtime/java/**` or `runtime/cljw/**` surface. -Pattern A/B1/B2 stop being *recorded* because they become -*unrepresentable-if-wrong*. Human-readable output moves to a generated -contribution map in `docs/` (the `--list-vars` surface), aimed at the -Issue-template audience opened in `e897cbfc`. - -- **Better than the draft**: the draft deletes the ledger and leaves the invariant - unobserved; this deletes the ledger *and* raises the invariant's enforcement - from documentary to mechanical, which is strictly stronger than what exists - today. Highest information-per-artifact of the three; nothing can rot because - nothing is hand-written. Directly serves the contribution-surface concern with a - generated artifact contributors can read. -- **Breaks**: (1) the "no cross-surface call" property is far harder to decide for - Clojure-ns vars than for Zig imports — a `.clj` body calling another `.clj` var - is legal and the graph is dynamic, so the check is not the simple import-graph - test G1 runs; realistically it degenerates into the same AST classifier Alt 2 - needs, minus the artifact, i.e. the same cost with less output. (2) It - **discards a shared vocabulary that four ADRs and live source comments depend - on** (ADR-0171:131, ADR-0043:128, ADR-0102:122, `string.zig:269/617`, - `data/feature_deps.yaml:17` "placement.yaml — Pattern A vs Pattern B - classification per…"), forcing edits across all of them or leaving dangling - terms — the audit's own complaint about dead references, reproduced. (3) It has - nowhere to put per-var *intent* (`escape` is Zig-for-now-`.clj`-later) except - debt rows, which is acceptable at n=1 today but re-opens the §15.x hole the - moment n>5. -- **F-NNN**: no violation. Strengthens F-009's enforcement, weakens F-013's - coverage-visibility (the index disappears), neutral on F-003 (declares no future - policy). - -### Direct answers to the five interrogations - -**1. Is "kill it" decision-seizure?** *Split verdict.* F-015 narrowed F-003 -explicitly: "F-003 … still holds for genuinely-unbuilt layout/representation -decisions, but it is NOT a licence to park built-but-unhardened behaviour behind -a Phase number" (`project_facts.md:1141-1143, 1168`). The -`clojure.set`/`string`/`walk` split is **built** (D-062 discharged), so F-003 -does *not* shield the ledger from repair — F-015 says fix it now. But the draft's -second clause — *"declare Zig implementation the finished form for -`clojure.core` hot paths"* — is a forward policy over the **unbuilt residual** -(where do the next N `clojure.core` vars live?), and that is precisely a -responsibility/dependency-graph decision F-003 reserves for its owning unit, -which is not this audit-remediation cycle. Correct move: **repair or replace the -artifact (permitted); do not stamp the policy (seizure)**. Note also that for the -hot primitives the policy is redundant — ADR-0033 D5 already says fast-path = Tag -switch — so dropping the clause costs nothing. - -**2. Is the shortlist wrong?** No — and the draft has the pattern backwards. -Those six rows are **B1**, whose ADR-0033 D3 finished form is Zig-with-no-`.clj`. -Nothing in the plan proposes moving `count`/`seq`/`first` into `.clj`. -Separately, on the general question: B2 is **not** free at runtime — a shim is a -real extra Var deref + fn invoke per call (`(def upper-case (fn* [s] (-upper-case -s)))`, `string.clj:41`) — which is exactly why ADR-0033 routes hot/multi-arity -vars to B1/C-zig ("multi-arity 複雑 / hot path") and reserves B2 for "Tier A -surface, JVM diff 維持". The classification rule already encodes the perf concern -the draft thinks it is defending. The rows are **mislabelled, not misplaced**; -the fix is a status flip to `stable`, not a retirement. - -**3. What does F-009 say?** F-009 is `confirmed` and live, and placement.yaml is -its only per-var observability for the Clojure-ns row of F-009's own surface -table. F-009's rationale section pre-registers this exact cycle as a predicted -envelope violation ("At Phase 12+ the AI loop is statistically very likely to -propose 'inline impl into the surface'"). Killing the ledger does not violate -F-009 by itself — the invariant lives in the ADRs and guardrails — but it removes -the artifact that makes F-009 auditable per-var while offering no successor. Alt -2 and Alt 3 both supply a successor; the draft does not. - -**4. What is the contribution-surface argument worth?** Less than the draft -implies, and it **cuts the other way**. The 1:14 ratio is not comparable to -upstream: the 124.7K Zig includes the VM, GC, JIT, wasm engine, and Java/cljw -host surfaces, which in the JVM implementation are Java (~tens of KLOC), not -`core.clj`; the honest comparison is cljw's `lang/` layer against `src/clj/`, not -the whole tree. More importantly, if outside contribution by Clojure programmers -is the goal, the response is *more* `.clj` surface and *better* visibility into -which vars are `.clj`-able — i.e. Alt 2's total index or Alt 3's generated -contribution map. Deleting the `.clj`-ability ledger three commits after opening -Issue templates (`e897cbfc`) is backwards. It should not drive this decision -regardless: contribution surface is a product/community question whose owner is -the release unit, and F-003's residual applies to it. - -**5. Is there a third state?** Yes, three, and the draft considered none: **(a) -generated mirror** (Alt 2) — machine-owned, total, gated; **(b) demoted index** -(Alt 1) — keep it, strip "SSOT/authoritative" from `.claude/CLAUDE.md:591` and -`.dev/README.md:37`, state the 49-var scope honestly, gate it; **(c) -archive-as-closed-record** (Alt 1b) — D-062 is discharged, so move it to -`.dev/archive/` as the migration's historical record with the one live residual -(`escape`/D-094) promoted to debt. Any of the three is strictly better than -deletion, because deletion also silently re-opens the ROADMAP §15.x -per-var-tracking hole that `status/vars.yaml` was superseded *into*. - -### Recommendation - -**Adopt Alternative 2.** The measured facts justify *repairing the mechanism*, -not retiring the plan: the plan largely executed, the ledger did not follow, and -the fix that cannot rot is derivation plus a drift gate. It is the only option -that satisfies F-013 (683/683 from the definition rather than 49/683 from one -arc), keeps F-009 observable, closes G11 and the §15.x hole, and follows the -project's own ADR-0102 precedent of "dedicated SSOT + generated table + gate". It -is the largest diff of the three, and per **F-002** that is not a reason to prefer -Alternative 1. - -If the loop nonetheless lands Alternative 1 for sequencing reasons, the two -clauses that must **not** ship in any variant are (i) the phrase "SSOT / -authoritative" over a 49-var file, and (ii) the forward policy "Zig is the -finished form for `clojure.core` hot paths" — the first is the measured lie, the -second is F-003 seizure of an unbuilt-residual decision and is already covered by -ADR-0033 D5 anyway. - -## Resolution - -**Alternative 2 is adopted**, and both forbidden clauses are excluded: the -"SSOT / authoritative" wording is gone from `.claude/CLAUDE.md` and -`.dev/README.md`, and no forward placement policy is stamped. - -Every factual claim in the fork's leading finding was re-verified against the -tree before acting on it: the six rows are `pattern: B1` with no `target_loc`; -D-062 is discharged with `escape` as the residual; `set.clj`/`string.clj`/ -`walk.clj` carry 12/21/11 `(def …)` bodies and zero `(defn`; both flagged -`leaf_loc` paths are absent. - -**One deliberate narrowing of the fork's proposal.** It asked for the full -A / B1 / B2 / C-thin classification, decided by an AST classifier over the `.clj` -bodies. What shipped derives the **`zig` / `clj` / `value` / `unbound`** axis -from the Var's root tag instead. The reason is the fork's own "Breaks" note: an -AST classifier must handle `(def x (fn* …))`, multi-arity `fn*`, macros, and -`def`-with-computed-value, and a misclassified generated row is a new lie class. -The root-tag axis is decided by the runtime rather than inferred, so it cannot be -wrong — and it already answers the question the rotted rows were lying about -("is this var Zig or `.clj` today?"). The finer split is available on top -whenever a consumer needs it: `zig-leaf` already separates B1 from a B2 leaf, and -the flags ride in the same index. A classifier that guesses is worse than an -index that reports. - -## Consequences - -- The index went from 49 hand-written rows to **1,326 generated ones across 31 - namespaces**, and can no longer disagree with the code. -- **Measured, previously invisible**: 394 vars are Zig-implemented and **725 are - Clojure-implemented**. The audit that opened this cycle had argued from the - 124.7K:8.8K *LOC* ratio that the Clojure surface was structurally thin. At the - var level it is the majority. LOC was the wrong denominator, and nothing in the - tree could have corrected it before this index existed. -- `clojure.spec.alpha` shows up with 123 `clj` vars + 17 values — it is bundled - and working, which the same audit had wrongly recorded as unimplemented because - `docs/works/ladder.md` and `COVERAGE.md` carry no row for it. **A working - feature absent from the ledgers misleads exactly like a broken one.** -- ADR-0033's migration *schedule* is recorded complete; its placement *rule* - (D2-D5) is untouched and still governs new vars. -- Cost: one more full-gate step, and the index must be regenerated in the same - commit as any var that moves between Zig and `.clj`. The gate says so on - failure. - -## Affected files - -- `src/lang/primitive/core.zig` — `__dump-placement` -- `scripts/gen_placement.sh` — new -- `scripts/check_placement_status.sh` — rewritten as the drift gate -- `test/run_all.sh` — `placement_drift` step -- `data/placement.yaml` — regenerated (was hand-written) -- `.claude/CLAUDE.md`, `.dev/README.md`, `.dev/ROADMAP.md` (§5 layout, §15.x), - `.dev/cw_v0_parity_and_gap_plan.md` (G11) - -## Revision history - -- 2026-08-04: Status: Proposed → Accepted. Alternative 2 adopted, narrowed to - the root-tag-derived axis per the fork's own misclassification warning. diff --git a/.dev/decisions/0179_value_tag_space_is_measured_not_counted.md b/.dev/decisions/0179_value_tag_space_is_measured_not_counted.md deleted file mode 100644 index 1d089cb41..000000000 --- a/.dev/decisions/0179_value_tag_space_is_measured_not_counted.md +++ /dev/null @@ -1,523 +0,0 @@ -# ADR-0179 — The NaN-box tag space is measured by use, not counted by name - -- **Status**: Proposed → **Accepted** (2026-08-04) -- **Amends**: ADR-0027 (NaN-box slot map) — adds the boxable/heap-only - distinction and its enforcement. Does **not** amend F-004. -- **Related**: F-004 (NaN-box layout, user-owned), F-002, F-003, ADR-0012, - ADR-0178 (derive rather than maintain), D-043 (the Group-D renumber - precedent), D-259 (the deferred wasm-value decision). - -## Context - -The 2026-08-04 cross-project audit counted the 64 named entries in -`src/runtime/value/heap_tag.zig` and concluded: *"the tag space is 100 % full; -core.async's channel, java.time, WeakRef and WasmGC refs have nowhere to go; -F-004 must be amended, and that is a user decision."* It was recorded as the -single structural problem the loop could not solve alone. - -**That conclusion was reached by counting names.** Names are not uses. Measured -by use, the space is not full. - -### The corrected census (mechanical, and it closes) - -| Class | Count | Test | -|---|---|---| -| NaN-boxed as a `Value` in production | **53** | reachable `Value.encodeHeapPtr` | -| Heap-only — `HeapHeader.tag` only, never a `Value` | **2** | `tail_node`, `tval` | -| Named but with zero production reference | **9** | `reader_conditional`, `class`, `array_chunk`, `matcher`, `tuple`, `box`, `wasm_fn`, `wasm_funcref`, `wasm_externref` | -| **Total** | **64** | | - -So **11 of 64 addresses are not doing NaN-box work**, and the "nowhere to go" -premise is false. - -Two of the nine are subsumed by a different representation, verified by running -the binary rather than by reading the enum: - -- `class` — a `Class` value is a `.type_descriptor` - (`src/lang/primitive/core.zig:43`); `(class (class 5))` → `Class`. -- `matcher` — `Matcher` is an ADR-0106 `host_instance` container - (`src/runtime/java/util/regex/Matcher.zig:3`); - `(type (re-matcher #"a" "a"))` → `java.util.regex.Matcher`. - -### Getting to that census took three tries, and that is the finding - -The first classification, and the draft ADR built on it, were wrong three -separate ways — all of them counting errors of the same family as the audit's: - -1. **A computed-tag encode site was missed.** The test was the *syntactic* - `grep 'encodeHeapPtr(\.'`. `src/runtime/collection/map.zig:695` reads - `Value.encodeHeapPtr(@enumFromInt(n.header.tag), n)`, and its consumers - decode the tag straight back out (`map.zig:893, 909, 945, 1126`). So - `hamt_map_node` and `hash_collision_map_node` are live boxed Values, not - heap-only. -2. **A test-only encode was counted as production.** `wasm_funcref` / - `wasm_externref` are encoded exactly once each, at `value.zig:556-557`, - inside a unit test. They are as unallocated as the other seven. -3. **The census summed to 63, not 64.** `@"volatile"` uses Zig's `@"…"` - escape and the regex did not match it. *A three-way partition of a - 64-entry enum that silently drops an entry is the same defect as the - audit's, one layer down.* - -The lesson is not "grep more carefully". It is that **a claim about which tags -are used is an execution fact, and it must be produced by execution**, the same -conclusion ADR-0178 reached about placement one commit earlier. - -## Decision - -### 1. Nothing is deleted or renamed - -The nine unallocated names stay. Two reasons, both from the project's own -record rather than from caution: - -- **F-004 fixes the day-1 type set, not just the layout.** F-004's body carries - a "Types day-1 in the 64-slot plan" section that names `reader_conditional` - ("`#?(:clj …)` for `.cljc`") and the Wasm family (`funcref` / `externref`, - justified by F-001). D-043's discharge note states the reading verbatim: - *"NOT an F-004 amendment (F-004 fixes 64 slots + the day-1 type set, not - intra-group ORDER)"*. Removing or repurposing a named day-1 type is - therefore user territory, and the loop does not take it. -- **`wasm_fn` / `wasm_funcref` / `wasm_externref` carry an open deferred - decision.** D-259's barrier says Phase 16 decides how they become - first-class cljw Values. Deleting the reservations would pre-empt an - unbuilt structural decision with a named owner — F-003. - -CLAUDE.md's Reservation-as-bias smell cuts the other way here, and it is worth -being exact about why. The smell is *obeying a reservation because it is -reserved*. This ADR does not obey the reservations — it **measures** them, -publishes that nine are unallocated, and hands the allocation question to the -owner who holds it. Refusing to spend a slot the loop does not own is not bias; -inventing a blocker out of a name count was. - -### 2. Boxability becomes a compile-time property - -`Value.encodeHeapPtr` takes `comptime ht: HeapTag` and comptime-rejects a tag -marked heap-only. 182 of its 184 call sites already pass an enum literal, so -they are unchanged; the one computed-tag site (`map.zig:695`) routes through an -explicit `encodeHeapPtrRuntime` that asserts the same property at runtime. A -future attempt to box `tval` is a **compile error at the call site**, not a -convention. - -### 3. The census closure is asserted at comptime - -`{boxable} ∪ {heap-only} ∪ {unallocated}` must partition `HeapTag`: pairwise -disjoint, union exhaustive, cardinality equal to the field count. This is the -check that would have caught both the audit's error and all three of mine — a -partition that does not close fails the build with the arithmetic in the -message. - -### 4. The record is corrected where it is read - -`heap_tag.zig`'s header states the measured census and that 11 addresses are -free, so the next reader counting names has the correct number in front of -them. - -### What is left for the user, stated precisely - -The audit said a user decision was needed to get **any** headroom. The accurate -statement is narrower: **11 addresses are free; 2 are loop-recoverable; using -any of the 9 named reservations for a different type is an F-004 question.** -When `core.async` needs a channel tag, the question to ask is not "may we amend -the 64-slot layout" (no amendment is needed) but "may `box` become `chan`" — -a much smaller question with a much cheaper answer. - -## Alternatives considered - -*(Devil's-advocate fork, fresh context, 2026-08-04. Reproduced as returned. -Every factual claim in its leading entry was re-verified against the tree -before this ADR was written; all held.)* - -### LEADING ENTRY — the draft's own premise fails the same test it was written to fix, in two places, and one deletion class is F-004-amendment territory - -**(i) `hamt_map_node` is NOT heap-only. It is NaN-boxed as a Value.** - -The draft's mechanical test — "`Value.encodeHeapPtr(.tag, ptr)` literally -present ⇒ the tag is a Value" — is a *syntactic* test, and it misses -computed-tag encode sites. `src/runtime/collection/map.zig:694`: - -```zig -fn nodeValue(n: *HamtMapNode) Value { - return Value.encodeHeapPtr(@enumFromInt(n.header.tag), n); -} -``` - -`nodeValue` is called at six sites (`map.zig:750, 804, 865, 950, 1003, 1133`), -each writing the result into a `slots[…]` array whose element type is `Value`; -and the consumers *decode* the tag back out of the Value — `map.zig:893, 909, -945, 1126`: `if (child_slot.tag() == .hash_collision_map_node)`. So -`.hamt_map_node` and `.hash_collision_map_node` are both live NaN-boxed Values -whose sub-type field is load-bearing. The heap-only set is **two** tags -(`tail_node`, `tval`), not three. - -`tail_node` and `tval` do check out: `PersistentVector.tail` is a raw -`?*TailNode` field (`vector.zig:65`), never encoded; `tval` is reached only -through `Ref.tvals: *TVal` (`stm/ref.zig:51`), and `stm/tval.zig:29` states the -invariant in its own header. - -**(ii) The unused-reservation count is 9, not 7 — `wasm_funcref` and -`wasm_externref` are equally dead.** - -Their only `encodeHeapPtr` call sites are `value.zig:556-557`, inside the unit -test *"F-004 day-1 Tag additions encode + decode"*. There is no production -producer, no `HeapHeader.init`, no `tag_ops` registration, and zero references -outside `heap_tag.zig` / `value.zig` (verified: `grep -rn -'wasm_funcref\|wasm_externref' --include='*.zig' src/` returns only enum -declarations, the sync test, and that one test). The draft's classifier counted -a test-only encode as "live", so it under-reported. `wasm_module` alone is real -(`runtime/cljw/wasm/wasm_handle.zig:47-48`). - -**(iii) The draft's census does not sum to 64.** 53 + 3 + 7 = **63**. A -three-way classification of a 64-entry enum that drops an entry is precisely -the counting failure mode this ADR exists to correct. The corrected partition -does close: - -| Class | Count | Members | -|---|---|---| -| NaN-boxed in production | 53 | (the remainder) | -| Heap-only (`HeapHeader.tag` only) | 2 | `tail_node`, `tval` | -| Zero production references | 9 | `reader_conditional`, `class`, `array_chunk`, `matcher`, `tuple`, `box`, `wasm_fn`, `wasm_funcref`, `wasm_externref` | -| **Total** | **64** | | - -Reclaimable headroom is therefore **11**, not "~10" — but see (iv) before -spending it. - -**(iv) Deleting the reservations is not obviously F-004-neutral, and the -project has already written down the opposite reading.** - -F-004's *layout* clause ("4 group × 16 sub-type = 64 slot, 44-bit shifted -pointer") is untouched by any of this — the draft is right about that. But -F-004 also carries a body section headed **"Types day-1 in the 64-slot plan"** -which names `reader_conditional` with its rationale (`#?(:clj …)` for `.cljc`) -and names the Wasm family `funcref` / `externref` with F-001 as the -justification (`project_facts.md:356-360`). And the loop's own D-043 discharge -note (`.dev/debt.yaml:2869`, written 2026-06-15) reads: - -> "NON-BREAKING + NOT an F-004 amendment (**F-004 fixes 64 slots + the day-1 -> type set**, not intra-group ORDER…)" - -That is this project's written precedent that the day-1 *type set* is F-004-fixed -and only the *order* is loop-owned. Under that precedent, deleting -`reader_conditional`, `wasm_funcref`, `wasm_externref` — and arguably `class` / -`matcher` / `tuple` / `box` / `wasm_fn` / `array_chunk`, which appear in -F-004's "Indicative slot map" — is an F-004 amendment, which the loop may not -make. D-200's status note (`debt.yaml:5237`) reinforces this: it framed reusing -`reader_conditional` for `#inst` as *"stealing `reader_conditional`'s -reservation"* and chose the no-slot `typed_instance` path instead, explicitly -on F-004 grounds. - -There is a defensible counter-reading: F-004 labels the map "**Indicative** -slot map … the final placement lands when the Phase 5 ADR draft is reviewed", -which subordinates the map to ADR-0027. I do not think that reading extends to -*removing members*, only to *placing* them. **Recommendation: do not delete -anything.** The entire capacity benefit is available without touching a single -name (Alt B below), so the loop never has to test how far "indicative" -stretches. If the ADR nonetheless deletes, it must say plainly that it is -overriding the D-043 reading of F-004's day-1 type set, and it should still not -delete `wasm_fn` / `wasm_funcref` / `wasm_externref` — see Alt A on D-259. - -### Alternative 1 — smallest-diff: allocation status without namespace surgery - -Keep `HeapTag` as one 64-entry enum and keep every name. Add (a) a single -derived `pub const unallocated: []const HeapTag` (or an `allocation_status` -comptime table) declaring which of the 64 slots carry no behaviour; (b) the -census-closure gate from §Q5 below; (c) an ADR-0027 amendment recording that an -unallocated slot is **freely renameable in place** by any later cycle — a -rename changes no layout bit, no wire byte, and no dispatch index. When -core.async needs a channel, `box → chan` is a one-line enum edit plus the sites -that add behaviour. - -**Better than the draft:** zero deletion, therefore zero F-004 exposure on the -day-1 type set and no need to litigate (iv). Zero renumbering, so -`tag_ops.zig`'s three `[64]` tables, the `native_descriptors[70]` array, -`isGcManaged`, and the 81 `switch (tag)` sites are all untouched. It ships the -*actual* correction the handover needs — "the space is not full, 11 slots are -free" — in the smallest verifiable form, and it makes the reservations' intent -explicit rather than deleting it. - -**What it breaks:** it does not raise the ceiling above 64, so the next 12th new -type re-opens the same conversation. It leaves `tail_node` and `tval` burning -two NaN-box addresses they provably cannot use. It leaves the real defect — one -enum serving two namespaces of different widths (`HeapHeader.tag` is `u8`/256, -the NaN-box sub-type is 4 bits within a 2-bit band/64) — unfixed, so the -conflation that produced the "100% full" audit error can produce it again. And -the rename-in-place trick is arguably the same F-004 type-set mutation as -deletion, just less legible; if (iv) is a real constraint, renaming inherits it. - -### Alternative 2 — finished-form-clean (RECOMMENDED): two types, one **derived** from the other, nothing deleted - -Split the namespace as the draft proposes, but reject the draft's -hand-maintained framing. Concretely: - -- **`HeapTag: u8`** stays the heap-object discriminant. It is the superset. It - indexes `tag_ops.zig`'s three tables (which become - `[@typeInfo(HeapTag).@"enum".fields.len]`, not `[64]`). It has room for 256 - and will not be full in this project's lifetime. `tail_node` and `tval` live - here and only here. -- **`ValueTag`** is the NaN-box-encodable subset, and its integer values *are* - the 0..63 slot indices. `encodeHeapPtr` takes a `ValueTag`; `HeapHeader.init` - takes a `HeapTag`. Passing a non-boxable tag to `encodeHeapPtr` becomes a - **compile error**, not a gate. -- **The mapping is generated, not written.** Per ADR-0178's ruling ("generate - the index; gate the generation; keep judgement out of it") the slot index - must be *derived* from one declaration, not maintained in a second list. One - `comptime` table in `heap_tag.zig` declares, per entry, `{ name, group, - boxable }`; `HeapTag`, `ValueTag`, and `valueTagOf(HeapTag) ?ValueTag` are - all `@Type`-generated from it. Nothing is kept in sync by hand. -- **`Value.Tag` (the 70-entry mirror in `value.zig:44-124`) is deleted as a - hand-written enum** and generated as `ValueTag` ∪ the six immediates. Today - that mirror is guarded by `value.zig:468-478`, a test that spot-checks - **seven** hand-picked pairs — it would not catch a dropped or shifted entry - in the middle of a group. Generation removes the failure mode instead of - testing for it. -- Unallocated slots keep their names and get an explicit `boxable = true, - allocated = false` marking. - -**Better than the draft:** it is the only shape that makes the invariants the -draft wants to *assert at runtime* into things the compiler *cannot express -wrongly*. It kills the two hand-maintained parallel structures -(HeapTag↔Value.Tag, and the draft's proposed new heap-tag↔Value-tag split) -rather than adding a third. It answers the reviewer's question 4 in the -affirmative: yes, the split creates a sync burden — **unless** it is derived, -and ADR-0178 is the in-house precedent that derivation is this project's answer -to exactly that. It gives genuinely unbounded heap-tag headroom (WeakRef, WasmGC -internals, java.time internals, STM/channel internals that never need to be a -Value) while leaving F-004's 64 NaN-box addresses *exactly* as decreed. And it -deletes no name, so (iv) never arises. - -**What it breaks:** it is by far the largest diff — 1,666 -`HeapTag`/`Value.Tag`/`.tag()` references and 81 `switch (tag)` sites must each -be re-typed to whichever enum they actually mean, and the compiler will surface -every one as an error rather than silently accepting the wrong one (that is the -point, but it is the cost). `native_descriptors: [70]` (`runtime.zig:412`) and -the three `[64]` tables become derived lengths. `isGcManaged` -(`heap_tag.zig:157`) and the `Runtime.init` assert at `runtime.zig:609` need -re-typing. The D-043-era invariant "HeapTag and Value.Tag renumber in lockstep" -is replaced by a derivation and needs its own gate. -`(cljw.internal/__native-type :kw)` (`lang/primitive/protocol.zig:310`, an -`inline for` over every `Value.Tag` field by name) changes which keywords it -accepts — an observable, if obscure, surface change. - -Per **F-002** I recommend this one anyway. Diff size is not a project -constraint; only F-NNN is, and this alternative is the *most* F-004-conservative -of the three because it is the only one that adds capacity without renaming or -removing anything F-004 names. - -### Alternative 3 — wildcard: one slot as an escape hatch (`extended`) - -Allocate exactly one of the 64 sub-types as `HeapTag.extended`. A Value tagged -`.extended` carries a pointer whose `HeapHeader.tag` (already a `u8` on every -GC-managed object) holds the *real* discriminant from a second 256-entry space. -`tag()` gains one branch: if the decoded sub-type is `extended`, load -`header.tag` and map it. Hot tags (string/vector/map/fn/…) stay pure -shift-and-mask; cold, rare, or internal tags (WeakRef, java.time internals, -channel internals, WasmGC refs) live behind the hatch and pay one dependent -load. - -**Better than the draft:** it converts "how do we ration 64 addresses forever" -into a closed question — the ceiling becomes ~319 and never binds again — while -the F-004 bit layout is *bit-for-bit unchanged* (still 4 groups × 16 sub-types, -still a 44-bit shifted pointer). It requires no renumbering, no deletion, no -second hand-maintained enum, and it composes with either Alt 1 or Alt 2. It also -gives the honest answer to the question the original audit was really asking -("where does core.async's channel go?") without spending a scarce slot on it. - -**What it breaks:** `tag()` stops being branch-uniform, which is a direct cost -to gap area III (VM-perf); the extra branch is predictable but the dependent -load on the extended path is not free, and `tag()` is one of the hottest -functions in the runtime. The escape hatch is only available to tags whose -pointer target has a real `HeapHeader` at offset 0 — `heap_tag.zig:140-164` -documents that `var_ref`, `ns`, and `keyword` do **not** satisfy this, so the -hatch is structurally closed to that class. It also introduces a two-tier tag -space whose tier assignment is a judgement call made per-type forever, which is -a slow-drip version of the ADR-0178 problem. - -**The variant we cannot take:** the maximal version of this — delete the 4-bit -sub-type field entirely, widen the pointer payload back to 48 bits, and read -*every* heap tag from `HeapHeader.tag` — is the cleanest possible finished form -and is a **direct F-004 violation**: F-004's confirmed direction is verbatim -"4 group × 16 sub-type = 64 slot, 44-bit shifted pointer". Recording it here per -the brief; not proposing it. It would additionally break the three header-less -heap tags above. - -### Answers to the five interrogation points - -**Q1 — is the central claim right?** Partly. The *conclusion* ("the space is -not full; ~10 slots are recoverable without amending F-004's layout") survives, -and is if anything understated (11). The *classification* has two errors in -opposite directions, both caused by using a syntactic test where an execution -test was needed: a computed-tag `encodeHeapPtr` was missed (`hamt_map_node`), -and a test-only `encodeHeapPtr` was counted as production (`wasm_funcref`, -`wasm_externref`). The other Value-construction paths I checked are clean: -`initInteger` / `initFloat` / `initChar` / `initBuiltinFn` / `initBoolean` use -the immediate bands and never touch the sub-type field; there is no -`@bitCast`-into-Value or inline bit-pattern construction anywhere in `src/`; the -only `@enumFromInt`-into-a-tag sites are `map.zig:695`/`829` (the computed-tag -path above), `value.zig:159` (`heapTagToTag`, a decode), `runtime.zig:609` (an -assert), `protocol.zig:312` (`__native-type`, name-keyed), and -`serialize.zig:1794` (a different enum). The two independently-verified -subsumptions hold: `class` is subsumed by `.type_descriptor` -(`lang/primitive/core.zig:43`) and `matcher` by ADR-0106 `host_instance` -(`runtime/java/util/regex/Matcher.zig`), and the runtime probes cited confirm -both. - -**Q2 — does deleting violate F-004, and is renumbering a wire break?** Deleting -does not change the *layout*; see (iv) for why it may nonetheless be -F-004-amendment territory via the day-1 type set. **Renumbering is not a wire -break.** The AOT envelope uses a decoupled `ValueTag` -(`src/eval/bytecode/serialize.zig:128`), documented as *"Stable enum — the -`docs/spec/formats/.edn` archive records this byte for each constant. -Adding a tag is a version bump; removing one is forbidden"*, with hand-assigned -values `0x00`–`0x11` bearing no relation to `HeapTag`. No -`@intFromEnum(HeapTag)` or `header.tag` byte is ever written to disk; heap -values are serialized structurally through the `ValueTag` arms. D-043 -(`debt.yaml:2869`) already renumbered all of Group D in lockstep on 2026-06-15 -and verified non-breaking, which is direct precedent. `tag_ops.zig`'s tables are -indexed by `@intFromEnum` and populated by *name* at startup -(`registerTrace(.tail_node, …)`), so they follow renumbering automatically. The -two things that do **not** follow automatically are the hand-written magic -numbers `[64]` (`tag_ops.zig:53, 74, 96`) and `[70]` (`runtime.zig:412`), plus -the boundary assertion `tag_ops.zig:198` — all three should become derived -lengths in whichever alternative lands. - -**Q3 — delete, or keep-and-mark?** **Keep and mark.** Three of the nine have -live, documented intent that deletion would silently discard: D-259's barrier -(`debt.yaml:2390-2393`) states verbatim that "F-004 reserves `wasm_module` / -`wasm_fn` / `wasm_funcref` / `wasm_externref` separately; P1 collapses to a -single `wasm_module`-tagged handle… Phase-16 decides… how `wasm_fn` / -`funcref` / `externref` become first-class cljw Values." That is an *open -deferred structural decision* with a named owner — deleting its reservations -pre-empts it, which is an F-003 problem on top of the F-004 one. -`reader_conditional` likewise is unimplemented intent rather than dead intent: -`#?` is handled at read time (`eval/reader.zig:151, 861, 893`) so no Value is -needed *today*, but `read-string` with `:read-cond :preserve` returns a -`clojure.lang.ReaderConditional` object in Clojure, and cljw has no `:preserve` -support yet — the reservation is exactly the slot that feature will want. Only -`class`, `matcher`, `array_chunk`, `tuple`, `box` are genuinely dead intent, and -deleting five names buys nothing that marking them unallocated does not. What is -lost by deleting: the co-location intent (`wasm_fn` beside `wasm_module` in the -D12..D15 tail is a deliberate D-248 grouping), the pointer back to the deferring -debt row, and the ability to answer "why is there a gap here?" in two years. - -**Q4 — is the split finished-form, or a second thing to keep in sync?** As -drafted — two hand-written enums plus a runtime assertion — it is a second thing -to keep in sync, and it is the *third* such structure in this file family -(HeapTag, `Value.Tag`, and now the split), guarded by a seven-assertion -spot-check test rather than a proof. That is the same class of defect ADR-0178 -just fixed in `placement.yaml`, and the smell sensor should fire on it. It -becomes finished-form only if the mapping is **derived** from one declaration — -Alt 2. The reviewer's instinct here is correct and should be the ADR's decision. - -**Q5 — what should the gate actually assert?** Note first that the failure mode -is *counting*, and that grep is not a counting instrument: my own scripted -census produced a false negative on `@"volatile"` (the Zig `@"…"` escape -defeats a `\.volatile\b` pattern; `volatile.zig:36` encodes it) and false -positives on `matcher` (prose and Java method names spell `.matcher`). A -grep-based "no tag is unreferenced" check would reproduce the original error -with new decoration. The gate must be: - -1. **Type-level where possible.** In Alt 2, "every tag passed to - `encodeHeapPtr` is in the Value-tag set" is not an assertion — it is the - function signature, checked by the compiler at every one of the ~150 call - sites. Prefer this over any runtime check. -2. **Observed-set, derived from execution — the ADR-0178 shape.** Under - `builtin.is_test` (or a Debug-only build flag), have `encodeHeapPtr` and - `HeapHeader.init` each set a bit in a static 64-/256-wide bitset. Run - `zig build test` ×2 plus the full corpus, dump the two observed sets, and - diff against a checked-in expectation. This converts "is this tag used" from - a textual guess into an execution fact, catches computed-tag sites (which is - exactly what the draft missed on `hamt_map_node`), and distinguishes - test-only from production coverage if the two runs are kept separate (which - is what the draft missed on `wasm_funcref`). -3. **Census closure — the check that would have caught both errors.** - `comptime` assert that {boxed} ∪ {heap-only} ∪ {unallocated} **partitions** - the enum: pairwise disjoint, union exhaustive, `|union| == fields.len`. The - 2026-08-04 audit failed by counting names and not uses; the draft failed by - producing three counts summing to 63. A partition assertion fails loudly on - both. -4. **Full-field sync, not spot-checks.** Replace `value.zig:468-478`'s seven - hand-picked `expectEqual`s with an `inline for` over *every* field asserting - name-and-value equality between the two enums plus equal field counts — or - delete the mirror entirely by generating it (Alt 2). -5. **Cheap exact statics:** `fields.len <= 64` for the boxable set; every - boxable value `< 64`; table lengths derived from `@typeInfo` rather than - literal `64`/`70`; and the existing `Runtime.init` invariant ("every tag with - a registered trace or finaliser is GcManaged", `runtime.zig:609`) extended to - "…and is in the heap-tag set". -6. **Make the expected set the SSOT and fail in both directions** — an unlisted - tag that gets exercised fails, and a listed tag that never gets exercised - fails. One-directional gates are how a stale claim survives, which is the - D-177 lesson `.claude/rules/clj_diff_sweep.md` already records. - -## Resolution - -**Alternative 2's principle is adopted — boxability becomes a property the -compiler checks — in the form its own cost analysis shows is actually -required.** - -The fork priced Alt 2 at "1,666 references and 81 switch sites must be -re-typed". Measured, that is not what the change costs: **182 of -`encodeHeapPtr`'s 184 call sites pass an enum literal**, and a Zig enum literal -coerces by name to whatever enum the parameter declares, so they need no edit at -all. Exactly one site passes a computed tag (`map.zig:695`). The 1,666 figure -counts every `HeapTag` reference, and the great majority of those legitimately -stay on `HeapTag` — they are heap-header and dispatch uses, which is the whole -point of the distinction. So the enforcement is available at a fraction of the -quoted cost, and the cost was the only argument for preferring Alt 1. - -What is adopted: - -- **Compile-time enforcement** (Alt 2's prize) via a `comptime` boxable - predicate on `encodeHeapPtr`, with the single computed site routed through an - explicit runtime-checked entry point. A non-boxable tag becomes a compile - error at the call site. -- **The comptime census-closure partition** (fork Q5.3), which is the check that - fails on both the audit's error and all three of mine. -- **Nothing deleted or renamed** (fork's leading recommendation), for the F-004 - day-1-type-set and F-003 D-259 reasons it sets out. - -What is deferred, with reasons: - -- **Generating `ValueTag` / `Value.Tag` via `@Type`** (Alt 2's derivation half). - The fork is right that a second hand-maintained enum would be the ADR-0178 - defect repeated. This ADR therefore does not create one — it adds a comptime - *predicate over the existing enum*, not a parallel type. Generating the - existing `Value.Tag` mirror away is a real improvement and a separate unit; - it is not needed for the capacity correction and would change - `__native-type`'s observable keyword surface. -- **Alt 3's `extended` escape hatch.** Correctly identified as a direct cost to - `tag()`, one of the hottest functions in the runtime, and gap area III owns - that trade. It also is not needed: 11 free addresses is not a ceiling anyone - is pressed against. -- **Relocating `tail_node` / `tval` above 63** to free their two addresses. - Loop-owned per D-043 (order, not type set) and non-breaking per the same - precedent, but it buys 2 of 11 addresses at the cost of touching the group-band - invariant — worth doing in the cycle that actually needs an address, not - speculatively. - -## Consequences - -- The handover no longer carries "the tag space is full and only the user can - unblock it". It carries a measured census and a precisely-scoped user - question. -- A future attempt to box a heap-only tag fails at compile time. -- A future census that does not close fails at build time, with the arithmetic - in the message. -- Cost: `encodeHeapPtr` gains a `comptime` parameter, which forbids a - runtime-tag call — deliberately, since the one legitimate such call is now - explicit and named. -- The three counting errors are recorded rather than quietly fixed, because the - pattern (a syntactic test standing in for an execution fact) is the reusable - part. - -## Affected files - -- `src/runtime/value/heap_tag.zig` — census, boxable predicate, partition assert -- `src/runtime/value/value.zig` — `encodeHeapPtr` comptime tag, - `encodeHeapPtrRuntime` -- `src/runtime/collection/map.zig` — the one computed-tag site -- `.dev/handover.md` — the corrected statement - -## Revision history - -- 2026-08-04: Status: Proposed → Accepted. Alternative 2's enforcement adopted - at its measured cost; its derivation half deferred as a separate unit; - nothing deleted per the fork's leading recommendation. diff --git a/.dev/decisions/0180_bundled_ns_lookup_is_derived.md b/.dev/decisions/0180_bundled_ns_lookup_is_derived.md deleted file mode 100644 index 7a7f5964c..000000000 --- a/.dev/decisions/0180_bundled_ns_lookup_is_derived.md +++ /dev/null @@ -1,220 +0,0 @@ -# ADR-0180 — a bundled file's namespace is a field, not a second list - -- Status: Proposed → Accepted -- Date: 2026-08-04 -- Supersedes: none -- Related: ADR-0163 (lazy bootstrap regions), ADR-0178 (placement.yaml is - generated), D-569 - -## Context - -`src/lang/bootstrap.zig` had two ways to answer "which bundled file defines -namespace N", maintained differently. - -`FILES` rows carried a display **label** (``), and -`nsNameFromLabel` recovered the namespace from it by stripping the angle -brackets — with `` → `clojure.core` special-cased. Four call sites -used that (`markFilesLoaded`, the eager-region walk, the source registry) plus -`builder.zig`'s region keying. Alongside it sat `lookupEmbeddedFile`, a -**hand-written** chain of 30 `if (mem.eql(u8, ns_name, "…")) return FILES[N];` -over 32 rows. - -The first draft of this ADR argued that the chain was dangerous because it had -drifted: `clojure.repl` has a `FILES` row but no chain row, so a build without a -region blob could not resolve it. **A devil's-advocate review falsified that, -and the truth is a stronger argument, so it replaces it here.** - -`loadCore` calls `markFilesLoaded(rt, ACTIVE_FILES)` (`bootstrap.zig:351`) -before evaluating anything, and `builder.zig:230` does the same for -`cache_gen`. Every bundled namespace is therefore in `rt.loaded_libs` from the -start, and `loadOrFindNs` returns at `loader.zig:121` before consulting any -resolver. The shipped binary additionally carries a region for every -`ACTIVE_FILES` row, and `loadOrFindNs` checks the region blob before the -resolver. Measured on the built binary: `(require 'clojure.repl)` and -`(require 'clojure.core-meta)` both already succeed. - -So the chain has **no production consumer for any bundled name**. Its only live -callers were its own unit tests. A hand-maintained table whose correctness -nobody can observe is worse than a drifting one — drift at least eventually -shows up. - -That also settles the question this ADR had to answer before deleting anything: -*do the two lists encode genuinely different things — "is bundled" versus "is -publicly requireable" — and would collapsing them lose an intentional -distinction?* **No.** The one exclusion the code claimed to enforce -(`clojure.core-meta`, with a comment saying so) is not enforced: requiring it -works today via `loaded_libs`. The distinction has never held. A partial second -list is worse than none, and D-569's own proposed shape — move the exclusions -onto `FileEntry` as a field — would have made a field out of a distinction that -does not exist. - -Ten comments in `FILES` exist only to defend the index keying — *"appended last -so earlier `FILES[N]` indices stay stable"* — so the ordering of the -bundled-source table has been shaped ten times by a constraint belonging to a -function nothing calls. - -## Decision - -**`FileEntry` carries `ns` as a first-class field. The label is derived from -it, not the other way round.** - -```zig -fn f(comptime ns: []const u8, comptime path: []const u8) FileEntry { - return .{ .ns = ns, .label = defaultLabel(ns), .source = embedSrc(path) }; -} -``` - -`clojure.core` is the single row that overrides its label (``), which -is exactly where that special case belongs — on the row it describes, rather -than inside a shared helper four callers had to route through. - -`nsNameFromLabel` is deleted. All five call sites wanted the namespace and none -wanted the label; that they went through a label-un-decorator is the evidence -that the derivation ran backwards. - -The lookup iterates `ACTIVE_FILES`: - -```zig -fn lookupEmbeddedFile(ns_name: []const u8) ?FileEntry { - for (ACTIVE_FILES) |file| { - if (std.mem.eql(u8, file.ns, ns_name)) return file; - } - return null; -} -``` - -`ACTIVE_FILES` — not `FILES` filtered by `isActiveFile` at run time. It *is* -`FILES` filtered by `isActiveFile`, at comptime. Iterating it makes the -resolver/walk coupling structural instead of re-derived, and the resolver -finally joins the set of walkers `ACTIVE_FILES`'s own doc comment describes. - -A **comptime uniqueness assertion** over `ns` is added. First match wins in -every namespace lookup here — the resolver, the region key, `markFilesLoaded` — -so a duplicate would shadow silently. Neither the chain nor a naive loop can -detect that; deriving is the moment to fix it rather than inherit it. Verified -by renaming one row to collide: `error: two bundled files claim the namespace -clojure.walk`, at compile time. - -The `FILES` ordering constraint is lifted with the ten comments defending it. -Load ORDER still matters — `core.clj` first, `clojure.data` after -`clojure.set`, and so on — because that is a real constraint about what has -interned what. What is gone is the unrelated one about indices not shifting. - -### The coupling test asserts both directions now - -It previously asserted only that an **inactive** file is absent from both -`ACTIVE_FILES` and the lookup, with a comment stating that an active file "need -not be name-resolvable". That absence is what let a 30-`if` chain sit beside -`FILES` and disagree with it. - -It now also asserts that every active file resolves to itself. Under the -derived lookup that is close to a tautology, and the test says so in a comment: -it is a **tripwire against reintroducing a hand-written second table**, not a -correctness check. A future reader must not delete it as vacuous. - -## Consequences - -- One list. A new bundled namespace is one row, placed where it reads best. -- `(require 'clojure.repl)` / `(require 'clojure.core-meta)` now also resolve - through the source path. Nothing depended on them being declined: every - consumer of `embeddedResolver` / `chainedResolver` was traced, and the - filesystem resolver never gets a shot at a bundled name in any configuration - because `loadOrFindNs` short-circuits first. Verified empirically that a - user's own `clojure/repl.clj` on `-cp` does not shadow today and does not - after this change. -- A duplicate namespace is a compile error rather than a silent shadow. -- Performance is not a question worth hedging about: the call site fires once - per `require`, only when both `loaded_libs` and the region blob miss — which - is never, in the shipped binary. `sourceText` already runs the identical - linear scan over `ACTIVE_FILES` on the strictly hotter error-render path. -- D-569 is discharged, and its status text — which asserted the same - unreproducible failure this ADR's first draft did — is corrected rather than - left as a record of something that never happened. - -## Alternatives considered - -*(Devil's-advocate fork, fresh context, verbatim. Its "facts checked first" -preamble is preserved in full: three of its findings falsified the draft's -Context, and one of them — that `clojure.core-meta` is not actually withheld — -is the decisive evidence for the Decision.)* - -### Facts checked first (three of them change the draft) - -**(a) The Context's failure story does not reproduce.** The draft says that in a build without a region blob — "the unit-test runtime, the `cache_gen` host tool" — `(require 'clojure.repl)` cannot resolve. Neither named configuration can reach `lookupEmbeddedFile` for a bundled name: - -- The source path: `loadCore` (`src/lang/bootstrap.zig:351-352`) calls `markFilesLoaded(rt, ACTIVE_FILES)` and then `loadCoreFiles(…, ACTIVE_FILES)`. Every bundled namespace is put in `rt.loaded_libs` *and* evaluated into the env, so `loadOrFindNs` returns at `src/eval/loader.zig:121-123` before any resolver is consulted. There is no laziness in the source path at all. -- `cache_gen` goes through `builder.buildBootstrapEnvelope`, which calls `bootstrap.markFilesLoaded(rt, files)` at `src/app/builder.zig:230` before the compile loop — the same early return. -- The shipped binary carries a region for *every* `ACTIVE_FILES` row (`src/app/builder.zig:272-276`), including `clojure.repl` and `clojure.core-meta`, and `loadOrFindNs` consults the region blob first (`src/eval/loader.zig:126-136`). - -Empirically, on the current `zig-out/bin/cljw`: `(do (require 'clojure.repl) (clojure.repl/demunge "a_QMARK_"))` → `"a?"`; `(do (require 'clojure.core-meta) :okmeta)` → `:okmeta`. Both already work. - -**(b) Therefore `lookupEmbeddedFile` has no live production caller for any bundled name.** Its only live callers today are its own unit tests (`src/lang/bootstrap.zig:853-876`, `:917`). The single production path that could still reach it is a *forward-reference* `(:require later-bundled-ns)` from inside a bundled file during `cache_gen`, where `loaded_libs` hits but `env.findNs` misses (`src/eval/loader.zig:121-123`) — and no bundled file does that today, because load order is arranged so dependencies come first. - -This weakens the ADR's motivation and strengthens its conclusion. The second list is not merely drifting; it is a hand-maintained table with **zero production consumers**, whose correctness nobody can observe. That is a stronger argument for deriving it than the one the draft makes, and it should replace the Context's "dangerous silent gap" narrative, which as written is not true. - -**(c) The `clojure.core-meta` "deliberate withholding" is already unenforced.** The comment at `src/lang/bootstrap.zig:303-307` presents core-meta's absence as intentional ("it is in `EAGER_NS`, loaded at startup, never required"). But `(require 'clojure.core-meta)` succeeds today via `loaded_libs`. The asymmetry the second list claims to encode has never held. This is the decisive answer to "do the two lists encode genuinely different things — 'is bundled' vs 'is publicly requireable'?" **They do not.** One list claims a distinction the runtime does not implement. Under F-013's own reasoning ("a partial class is a worse trap than an absent one"), a partial second list is worse than none. - -Two smaller corrections: the chain is **30** `if` statements (`grep -c 'if (std.mem.eql(u8, ns_name'`) over **32** `FILES` rows, not 31; and there are **10** index-stability comments (lines 131, 141, 148, 151, 154, 157, 169, 175, 184, 188), not eight — the debt row D-569 says three. Use the measured numbers. - -### Alt 1 — smallest-diff: add the two missing rows, keep index keying - -Append `if (std.mem.eql(u8, ns_name, "clojure.core-meta")) return FILES[29];` and the `clojure.repl` → `FILES[30]` row, and add the positive-direction assertion to the coupling test. - -*Better:* one commit, no structural claim to defend; preserves whatever a reader thought the `FILES[N]` indices meant; keeps `embeddedResolver` able to decline a name in the future, should a policy of "bundled but not requireable" ever become real. - -*Breaks:* nothing mechanically — and that is the objection. It re-affirms a table with no production consumer, keeps the ordering constraint that has already deformed the `FILES` table ten times, and makes the positive-direction test a genuine (not tautological) assertion that will fail on the *next* forgotten row rather than preventing it. Under F-002 this is the Smallest-diff bias: it optimises for the size of this commit against the shape of the file. Reject. - -### Alt 2 — finished-form-clean: the namespace is the fact; the label is presentation - -Recommended. The draft derives the namespace *from* the display label; the finished form runs the derivation the other way. - -`FileEntry` gains `ns: []const u8` as a first-class field. `label` stays (it is the error-renderer's `SourceContext` key and the sources-blob key, `src/lang/bootstrap.zig:216-246`) but becomes derived-or-overridden: `` is `"<" ++ ns ++ ">"`; `clojure.core`'s `` is the one explicit override. `nsNameFromLabel` (`src/lang/bootstrap.zig:769-774`) is deleted; its five production call sites (`src/lang/bootstrap.zig:706, 714, 738, 755`; `src/app/builder.zig:272`) become `file.ns`. Every one of those five wants the namespace — none wants the label — which is the evidence that the current derivation direction is backwards: a `` special case exists in a shared helper solely so four callers can un-decorate a string that was decorated for the renderer's benefit. - -Then: - -```zig -fn lookupEmbeddedFile(ns_name: []const u8) ?FileEntry { - for (ACTIVE_FILES) |file| { - if (std.mem.eql(u8, file.ns, ns_name)) return file; - } - return null; -} -``` - -Note `ACTIVE_FILES`, not `FILES` + `isActiveFile`. `ACTIVE_FILES` (`src/lang/bootstrap.zig:199-209`) *is* `FILES` filtered by `isActiveFile` at comptime. Iterating it makes the resolver/walk coupling **structural** rather than re-derived, and the resolver finally joins the set of walkers that `ACTIVE_FILES`'s own doc comment describes. - -Add a comptime uniqueness assertion over `ns` (a `comptime { }` block or a `StaticStringMap` construction that would fail on a duplicate key). Neither the chain nor the draft's loop detects a duplicate — both silently take the first match. Today the 32 labels are unique and `nsNameFromLabel` is injective over `FILES` (verified), and the plausible near-term collisions the review asked about (`cljw.json` vs `clojure.data.json`) are distinct names. But "unique today" is exactly the property a derived design should assert rather than assume, and it is the one real hazard first-match ordering introduces. - -*Better:* one fact per row, stated where a reader is already looking; the `FILES` ordering is governed only by load order (a real constraint); the ten "appended last" comments go; the `` special case shrinks from a shared helper to one field on one row; a duplicate namespace becomes a compile error instead of a silent shadow. - -*Breaks:* the diff is larger than the draft's — a new field on 32 rows, a deleted public helper, five updated call sites, and `ACTIVE_FILES`'s doc comment (`src/lang/bootstrap.zig:196-198`) rewritten, since it currently asserts "`FILES` keeps its full shape (and its stable indices) so `lookupEmbeddedFile` stays index-keyed", which this ADR falsifies. Per F-002 and the Cycle-budget-defer smell, size is not a reason to prefer the draft's shape; recommend Alt 2 as written. - -### Alt 3 — wildcard: delete the embedded resolver entirely - -Given finding (b), the honest wildcard is to remove `lookupEmbeddedFile`, `embeddedResolver`, and `installEmbeddedResolver` (`src/lang/bootstrap.zig:265-336`), have `setupCorePrefix` install nothing, and let bundled-namespace resolution rest on the two mechanisms that actually serve it: `loaded_libs` (source path, `cache_gen`) and the region blob (shipped binary). `chainedResolver` (`src/lang/require_resolver.zig:90-93`) collapses to `filesystemResolver`, and the "bundled namespaces can never be shadowed" guarantee moves to the region blob — where it already lives in practice (verified: a user `clojure/repl.clj` and a user `clojure/set.clj` on `-cp` both fail to shadow today). - -*Better:* the largest reduction — it deletes a mechanism instead of maintaining one, and removes the only place where a "bundled" namespace could be served from two different sources. - -*Breaks:* it forecloses the eager→lazy switch that `src/lang/bootstrap.zig:169-170` explicitly reserves ("the list stays data-driven so a future eager→lazy switch (lazy-AOT, deferred) is local"). A source-path bootstrap that stops eagerly loading all 32 files would need exactly the resolver this alternative deletes. It also removes the honest "namespace not found" for `cljw.wasm` in a non-`-Dwasm` build by removing the code path rather than by keeping the gate, and it deletes four unit tests that are currently the only executed proof that the bundled sources decompress and parse. Not recommended, but recorded because it names the real question the draft only half-asks: *why does a second resolution mechanism exist at all?* - -### Attacks on the draft that survive - -1. **The Context is factually wrong and must be rewritten** (finding (a)). Left as-is, the ADR records a failure mode that cannot occur, which is the exact defect class ADR-0178 was written against. -2. **The draft's point 2 rests on a premise that is already false** (finding (c)). Its argument — "core-meta is withheld, but withholding is harmless, so no exception mechanism is needed" — reaches the right conclusion from a wrong premise. Core-meta is not withheld. Say so; it is the strongest evidence in the ADR that the two lists do not encode different things, and it also disposes of D-569's own SHAPE proposal ("move the two deliberate exclusions onto the `FileEntry` as a field"), which would have made a field out of a distinction that does not exist. -3. **`FILES` + `isActiveFile` should be `ACTIVE_FILES`.** The draft's point 3 claims uniform gating as a win, but re-applies at run time a predicate already applied at comptime. This is a smallest-diff residue. (The gate itself *is* a no-op as claimed — `isActiveFile` is `build_options.wasm or !entry.wasm_only` (`src/lang/bootstrap.zig:71-73`) and only `FILES[23]` `` sets `wasm_only`. Verified.) -4. **Nothing depends on the resolver declining `clojure.repl` or `clojure.core-meta`.** Traced every consumer: `chainedResolver` (`src/lang/require_resolver.zig:91`), `installChained` from `builder.zig:387` / `runner.zig:113` / `repl.zig:131`, `installEmbeddedResolver` from `setupCorePrefix` (`bootstrap.zig:525`), and the single read at `loader.zig:137`. The filesystem resolver never gets a shot at a bundled name in any configuration, because `loadOrFindNs` short-circuits on `loaded_libs` or the region blob first. Verified empirically with a user-supplied `clojure/repl.clj` on `-cp`: it does not shadow today and will not after the change. -5. **First-match ordering is unasserted.** Add the comptime uniqueness check (Alt 2). The chain had the same defect; deriving is the moment to fix it, not to inherit it. -6. **Performance is a non-question, and the ADR should say why rather than hedge.** The call site is `loadOrFindNs`, once per namespace `require`, only when both `loaded_libs` and the region blob miss — which is never, in the shipped binary. For scale: `sourceText` (`src/lang/bootstrap.zig:216-246`) already runs the identical linear `mem.eql` scan over `ACTIVE_FILES` *and then* a full walk of the compressed sources blob, on the error-render path, which is strictly hotter. The draft's "the chain of 31 `if`s it replaces was the same scan written out" is correct but understates the case. -7. **The coupling test.** Under Alt 2 both directions become tautological, and the draft's stated rationale ("close to a tautology, which is the point") is the right one — but the ADR should say plainly that the test is a *tripwire against reintroducing a hand-written table*, not a correctness check, so a future reader does not delete it as vacuous. - -**Adopted: Alt 2**, with every one of the seven attacks applied — the Context -rewritten to the measured facts, `ACTIVE_FILES` instead of a re-derived gate, -the comptime uniqueness assertion (falsified by inducing a collision), the -tripwire rationale written into the test, and the corrected counts (30 `if`s, -32 rows, 10 comments) used throughout. - -## Revision history - -- 2026-08-04: Proposed → Accepted. First draft's Context was falsified by the - DA fork and replaced; the DA's own finding (c) became the Decision's central - argument. diff --git a/.dev/decisions/0181_doc_coverage_is_a_ledger.md b/.dev/decisions/0181_doc_coverage_is_a_ledger.md deleted file mode 100644 index 26051c40e..000000000 --- a/.dev/decisions/0181_doc_coverage_is_a_ledger.md +++ /dev/null @@ -1,157 +0,0 @@ -# ADR-0181 — a generated file with no gate is not maintained, it is abandoned - -- Status: Proposed → Accepted -- Date: 2026-08-04 -- Related: ADR-0178 (placement.yaml is generated + gated), AD-057 - (core_surface_extras as a ledger of decisions), D-305, D-513 - -## Context - -`(doc reduce)` printed a bare header. So did `(doc assoc)`, `(doc conj)`, -`(doc first)`, `(doc apply)`, `(doc =)`. - -`clojure.core`'s `:doc` and `:arglists` come from -`src/lang/clj/clojure/core_meta.clj`, a file generated by -`scripts/extract_core_meta.sh` from the JVM oracle. It held **291** rows. -Re-running the same pipeline, unchanged, produced **628**. Crossing its symbols -against `data/placement.yaml` explains the shortfall exactly: - -| `clojure.core` impl | count | in `core_meta.clj` | -|---------------------|-------|--------------------| -| `clj` | 246 | 220 | -| `value` | 156 | 21 | -| `zig` | 290 | **0** | -| `unbound` | 4 | 0 | - -**Zero of the ~290 Zig-primitive core vars were covered** — and a Zig primitive -has no `defn`, so the generated table is the only thing that *can* document it. -The mechanism was deployed where it was least needed and absent where it was -the only option. - -Measured coverage before: 246 of 685 publics had `:doc`, 221 had `:arglists`. - -The consequence is worse than a blank. `clojure.repl/find-doc` and `apropos` -are advertised features; they were searching 36% of `clojure.core` and -returning confident "no match" answers for the rest. A missing docstring is -visible; a search that silently covers a third of its corpus is not. - -It rotted because **nothing re-ran it and nothing checked it**. -`test/run_all.sh` gates `placement_drift` (ADR-0178, generated + gated), but -`core_meta` had no equivalent. It was regenerated once, on 2026-07-07, and -drifted from that moment. - -Found by a devil's-advocate fork sent to review an unrelated question — the -contrib-namespace docstring gap. That gap is real but small; this was -underneath it. - -## Decision - -### 1. Regenerate, and record the cost honestly - -291 → 628 rows. Coverage: 246 → **572** of 684 publics with `:doc`, 221 → 551 -with `:arglists`. Binary 7.37 MB → **7.55 MB** (+180,704 B). Every gated size -claim — README, `docs/landscape.md`, `bench/RELEASE_METRICS.md`, ROADMAP — is -updated to the measured figure rather than left inside the 10% drift tolerance, -because a claim that is merely *within tolerance* is how the last one rotted. - -### 2. `data/doc_coverage.txt` — undocumented is a decision, not an accident - -The exact set of public vars across the bundled namespaces carrying no `:doc`, -asserted by `scripts/check_doc_coverage.sh` as the `doc_coverage` gate step. -312 today. - -The gate runs **cljw alone**. The oracle needs a JVM and CI has none — the same -constraint `data/core_surface_upstream.txt` already records. So this cannot -notice that upstream documents something cljw failed to import. What it *can* -do is make that set visible and reviewable, and visibility is exactly what the -accident depended on being absent. A var losing its docstring, or a new var -arriving without one, fails here. - -Same shape as `data/core_surface_extras.txt` (AD-057): a name appearing in the -ledger is a decision, not a formality. - -The ledger's header names the three clusters so it is read rather than skimmed: - -- **`clojure.pprint`** — ~50 of its 56 publics are cljw-internal `cl-*` - formatter helpers that upstream keeps private. That is a **leaked API - surface**, not a documentation gap; the fix is `defn-`, not prose. Recorded - here so it stops reading as 50 missing docstrings. -- **`clojure.math`** — all 45 upstream docstrings end in a - `java.lang.Math` Javadoc URL. Importing them verbatim would ship 45 Java-8 - links as the canonical reference for a runtime that has no `java.lang.Math`, - and they assert an `ArithmeticException` cljw does not throw. These need - authored prose. -- **`clojure.core`** — the residue is mostly `-`-prefixed protocol methods and - earmuffed dynamics that upstream does not document either. - -### 3. What is deliberately NOT decided here - -Whether the ~110 contrib vars with an upstream counterpart get **imported** or -**authored** text. The devil's advocate found strong evidence against verbatim -import (the Javadoc URLs above; upstream's `clojure.set/intersection` arglists -document a 1-arity minimum where cljw returns `nil` for the 0-arity call — a -real divergence that importing would have concealed). That is its own decision -with its own licensing consequences, and the ledger makes the choice -reviewable instead of urgent. - -## Consequences - -- `reduce`, `assoc`, `conj`, `first`, `apply`, `=` are documented. - `find-doc` searches 84% of `clojure.core` rather than 36%. -- +180,704 B, against an 8.8 MB ceiling — the binary-size ledger records what - the bytes bought rather than only that they arrived. -- A generated artifact in this repo now has a gate in every case where one - exists: `placement.yaml` (`placement_drift`), `core_surface_extras.txt` - (`core_surface`), the clj corpora (`corpus_regression`), and now docstrings - (`doc_coverage`). -- `check_extract_core_meta` is deliberately NOT added: regeneration needs a - JVM, so a drift gate would fail on every CI run. The coverage ledger is the - JVM-free proxy, and its limits are stated above rather than implied. - -## Alternatives considered - -*(Devil's-advocate fork, fresh context. It was briefed on a narrower question — -"docstrings in the defns versus generated per-namespace meta files" for six -contrib namespaces — and its first finding was that the premise understated the -problem by 337 vars. Reproduced in full because the measurements are the -evidence for everything above.)* - -> **1. The premise "clojure.core is covered by a different mechanism" is false at the 58% level.** `core_meta.clj` holds 291 `when-let` rows. Replaying the existing `scripts/extract_core_meta.sh` pipeline today yields **628 rows** — 337 `clojure.core` vars that the JVM oracle documents and cljw interns are simply absent. Measured coverage of `clojure.core`: **685 publics, 246 with `:doc`, 221 with `:arglists`.** … **Zero of 290 Zig-primitive core vars are covered.** The generated-table mechanism is deployed precisely where it is least needed (`impl: clj` vars have a `defn` that could carry the docstring) and absent precisely where it is the *only* possible carrier (a Zig primitive has no `defn`). It is stale, not broken. The file is generated on demand and **nothing regenerates or checks it**. `test/run_all.sh:394` gates `placement_drift`; there is no `core_meta` equivalent. It rotted silently since 2026-07-07. Consequence: `clojure.repl/find-doc` searches 246 of 685 core vars and returns *nothing* for the other 439 — a silently wrong answer, not a visibly missing one. -> -> **2. The gap inventory is both wider and narrower than stated.** Wider: nine bundled namespaces sit at 0% `:doc`, not six — add `clojure.data.json`, `clojure.data.csv`, `clojure.tools.cli`. Narrower: of the 185 publics in the six named namespaces, only **110 have an upstream documented counterpart at all**; **75 do not** (49 pprint, 11 zip, 9 test, 5 data). Neither shape can document those; they need original prose regardless. -> -> **3. 49 of pprint's 56 publics are not a documentation gap — they are a leaked API.** Upstream `clojure.pprint` interns 252 vars and publishes 26, of which exactly one is `cl-*` (`cl-format`). cljw publishes 56, of which 49 are cljw-invented reimplementation internals. Marking them `defn-` removes 49 from the denominator at zero documentation cost. Meanwhile 22 upstream-documented pprint vars cljw does *not* implement — that is the real pprint gap, and it is an API gap, not a doc gap. -> -> **4. The licensing posture already changed, and `legal/NOTICE` is wrong today.** Nine files carry the variant-① banner; NOTICE says three and asserts the rest reproduce none. `scripts/check_clj_attribution.sh:43` greps only for the SPDX line — **nothing gates NOTICE against the variant-① file set**, which is why it rotted. -> -> **5. Binary size does not discriminate between the shapes and does not block either.** All bundled `.clj` = 462,368 B raw → **127,920 B deflate**. Regenerating core_meta: +98,119 raw → **+19,892 deflate**. Full upstream doc+arglists payload for all six namespaces: 34,724 raw → **9,698 deflate**. Both shapes put identical text into identical places. **Size cannot choose between A and B.** -> -> **6. `(doc …)` on an unloaded lazy namespace is parity, not a gap.** `(clojure.repl/doc clojure.set/union)` without the require prints nothing in cljw. **`clj` does exactly the same.** There is nothing to fix. -> -> **7. The `arglists` half splits the shapes, and one blocker has already dissolved.** `clojure.set`'s 0/12 is structural: `set.clj:29` is `(def union (fn* [& sets] …))`, and `def` + `^{:doc}` attaches `:doc` but not `:arglists`. But **D-070 (multi-arity `fn*`/`defn`) is discharged** and multi-arity `defn` works today — the `[& sets]` workaround `set.clj:18-23` still explains as a D-070 sidestep is an obsolete fossil. -> -> **8. Upstream text is measurably wrong for cljw — the strongest argument against verbatim import under *either* shape.** `clojure.math`: **all 45** oracle docstrings terminate in a `java.lang.Math` Javadoc URL, and assert an `ArithmeticException` on overflow where cljw throws `"integer overflow"`. `clojure.set`: upstream `:arglists` for `intersection`/`difference` have no 0-arity and clj throws `ArityException`; **cljw returns `nil`**. Importing upstream arglists would document a contract cljw does not implement and conceal a divergence that is in neither `debt.yaml` nor `accepted_divergences.yaml`. -> -> **Attack on shape A (docstrings in the defns).** A is at most half a shape: it cannot reach the 290 Zig primitives — the largest part of the real gap has no `defn` to hold a docstring. On `clojure.set` it produces a worse artifact than today (prose under a blank arglists line). The licensing objection is aimed at the wrong target: flipping ② → ① is a consequence of *transcribing*, not of *inlining*. -> -> **Attack on shape B (generated per-namespace meta files).** "Regenerable, so it cannot rot" is disproved by its own precedent — `core_meta.clj` is regenerable and it rotted, because nothing regenerates it. Regenerability without a gate is a hope, not a property. "Cannot rot against upstream" is also the wrong invariant: cljw is not upstream, and pinning to upstream text is exactly what produces Oracle Javadoc URLs in a no-JVM runtime. Wrong factoring, too: nine namespace-keyed files for a six-namespace problem, leaving the 290-primitive problem to a tenth. -> -> **Alt 1 — smallest diff: regenerate, gate, and state the policy.** Re-run the extractor (291 → 628). Add a drift step beside `placement_drift`. Correct NOTICE and gate the variant-① set. Record the contrib namespaces as carrying no `:doc`. *Better:* fixes the largest, highest-traffic gap — the one structurally unfixable any other way — with no licensing change and no new mechanism. *Breaks:* `(doc clojure.set/union)` still prints a bare header. **Gate design constraint:** the extractor needs a JVM and CI has none — `data/core_surface_upstream.txt`'s header records exactly this. The drift gate must follow that precedent: check the oracle capture in as data, gate the cljw side only. -> -> **Alt 2 — finished-form-clean (recommended).** Docstrings are authored, not imported; one generated table scoped to vars that cannot carry one; a coverage gate. Narrow `core_meta.clj` to the ~290 primitives and move the 220 `impl: clj` rows into their `defn`s. The six contrib namespaces get authored prose describing cljw. `clojure.set` is rewritten to multi-arity `defn`, which fixes 0/12 arglists *and* forces the `(intersection)` 0-arity divergence to be decided. The 49 leaked pprint `cl-*` publics become `defn-`. NOTICE corrected and gated. A doc-coverage gate in the AD-057 shape. *Breaks:* by far the largest diff — ~136 authored docstrings, a `clojure.set` rewrite, a core_meta split. Per F-002, recommend it anyway. -> -> **Alt 3 — wildcard: docs are data, and the generator's output is not `.clj`.** Put var metadata in `data/var_meta.yaml` beside `placement.yaml`, generated and gated identically, attached at intern time or lazily on first `meta` read instead of replaying 628 `alter-meta!` forms through the evaluator at boot. *Better:* kills the load-order problem class; `:doc` becomes queryable without booting a runtime (what `cljw --list-vars` wants); can decompress lazily so the payload never enters startup RSS. *Breaks:* a new runtime mechanism in the var/meta path; does not solve the *content* problem — a table full of Oracle URLs is wrong in YAML exactly as in `.clj`. Best composed with Alt 2. -> -> **Is this worth doing at all?** Partly — and the split is not where the prompt puts it. **The `clojure.core` half is a correctness floor item and is nearly free.** Fixing it requires re-running one existing script and adding one gate. **The contrib half is genuinely low value**, and the honest alternative is a gated ledger recording those namespaces as deliberately doc-less — which converts a silent blank into a truthful statement, the actual user-facing defect. **The NOTICE drift must not ride along as a footnote**; it is a legal-accuracy defect in a public EPL-2.0 project, live regardless of which shape wins, and should land on its own. - -**Adopted: Alt 1 plus item 5 of Alt 2** — regenerate, gate coverage rather than -oracle drift (its JVM constraint applies here too), and record the contrib -namespaces as a ledger of decisions. The NOTICE correction landed separately -and first, as its closing paragraph asks. Alt 2's authoring work and Alt 3's -data-not-`.clj` shape are the ordered remainder, kept in D-513 rather than -folded in here — the ledger makes them reviewable instead of urgent, which is -the point of writing the set down. - -## Revision history - -- 2026-08-04: Proposed → Accepted. diff --git a/.dev/decisions/0182_the_eval_budget_bounds_blocking_calls.md b/.dev/decisions/0182_the_eval_budget_bounds_blocking_calls.md deleted file mode 100644 index 7314cc439..000000000 --- a/.dev/decisions/0182_the_eval_budget_bounds_blocking_calls.md +++ /dev/null @@ -1,270 +0,0 @@ -# ADR-0182 — the eval budget's wall clock must bound blocking calls, not just back-edges - -- Status: Proposed → Accepted -- Date: 2026-08-04 -- Related: ADR-0125 (in-process eval budget), ADR-0153 / D-442 (future-cancel - slicing), D-571 - -## Context - -`cljw.eval/with-budget` advertises three axes: steps, wall-clock deadline, heap. -The deadline did not bound wall clock. It bounded *back-edges that happen to -cross a deadline check*. - -`EvalBudget.tick` runs at the back-edge safe points of both backends. A blocking -primitive makes no back-edges, so it consumes time without consuming steps, and -the deadline never fires. Measured: - -```clojure -(cljw.eval/with-budget {:deadline-ms 2000 :max-steps 50000000} - (fn [] (Thread/sleep 20000) :finished)) -;; before: returned :finished after 20.0 s -``` - -That is not a slow check — it is no check. The value came back *normally*, so a -caller cannot even tell the budget was exceeded. - -**Found on the ClojureWasm playground**, which is a public demo evaluating -untrusted submissions in-process and relies on this budget for isolation. Two -measurements, on a locally-run instance with `PG_EVAL_DEADLINE_MS=3000`: - -- `POST /api/eval` with `(Thread/sleep 25000)` returned `:done` after - **25,004 ms** — the advertised 3,000 ms bound did nothing. -- `GET /api/health` issued while that eval was parked took **22.0 s** - (baseline 0.01 s), because `cljw.http.server` accepts serially (D-117(a)). - -Together those are a denial of service reachable by one unauthenticated request, -with the sleep argument — and therefore the outage length — chosen by the -caller. The budget is the only thing standing between that endpoint and the -process, and on this axis it was not standing. - -## Decision - -**Every blocking primitive waits on a LATCH, bounded by an instant.** - -Every blocking call cljw has is the same sentence: *wait until a fact becomes -true, no later than some instant*. A future is realised; a promise is delivered; -a thread is done; a cancel is requested. Each of those facts is **monotonic** — -false, then true once, never false again. That is a latch, and `std.Io.Event` is -a latch. These were all built on `std.Io.Condition`, which is the wrong shape for -a monotonic fact and, more to the point, has no timed wait. - -`runtime/concurrency/latch.zig` is the primitive: - -```zig -pub fn wait(self: *Latch, deadline_ns: ?i64) bool -``` - -built on `std.Io.Event.waitTimeout`, with the bound carried as an **absolute -instant** rather than a duration — `waitTimeout` also returns `error.Timeout` on -a spurious wakeup, so a duration would silently restart on every one. - -`FutureCell` and `PromiseCell` keep their mutex — it guards the *payload* — and -gain a latch for the *edge*. Publish under the mutex, then signal; `Event.set` -releases prior writes, so a waiter that observes the latch observes the payload. -`deref` then waits outside the mutex and takes it only to read. - -`EvalBudget` contributes the other instant. `deadlineOf(rt)` is the budget's -deadline; every blocking call passes `min(its own bound, that)` to `Latch.wait` -and calls `checkDeadlineNow` after, which converts "I waited out the deadline" -into the budget error. `Thread/sleep` is the degenerate case — a latch nobody -sets — and its cancel latch is the future's own `settled`, because cancellation -is one of the terminal states a deref'er waits for anyway. - -### Why not the smaller shape (and why the first draft of this ADR was wrong) - -The first version of this change added `budgetedSleep`, a helper that sliced a -sleep against the deadline, and deferred the rest of the family to D-571 on the -grounds that: - -> `future`/`promise` deref and `Thread.join` block on -> `std.Io.Condition.waitUncancelable`, and Zig 0.16 has no timed condition wait -> — which is why `future.waitRealised` already polls `isRealised` in 1 ms sleeps -> rather than waiting with a timeout. A sleep helper cannot bound a condvar wait; -> they need a *bounded wait* primitive of their own. - -**That premise is false.** A devil's-advocate fork checked it against the pinned -toolchain: `std.Io.Event.waitTimeout` (`Io.zig` L1827) is a timed wait, and -`Io.Timeout` (L1132) carries an absolute `.deadline` — the very concept this ADR -describes and the first version implemented by hand as a loop. Verified by -execution: a `.deadline` bound returns `error.Timeout` at 201 ms for a 200 ms -deadline, and a 30 s bound wakes at 105 ms when another thread sets the latch. -The 1 ms poll was a cljw choice, not a platform constraint, and the ADR had -rationalised it into one. - -So the deferral was not a scoping decision, it was a mistake resting on an -unchecked claim — in a cycle whose entire justification was that the previous -behaviour had never been checked. - -## Consequences - -Measured on the built ReleaseSafe binary. - -- **`@(promise)` under a deadline no longer hangs.** It returns - `#:cljw.eval{:exhausted :deadline}`. The first draft's Consequences section - listed this, `(await agent)` and `(read-line)` as *not fixed* and pointed at - D-571; the whole family now goes through the bounded wait. - -- **A live clj-parity break is closed, and it was never about budgets.** With no - budget armed at all, `@(future (Thread/sleep 2000))` measured **2321-2358 ms** - — 17% long, on the most idiomatic metered-blocking expression in Clojure — - because `future-cancel` had to notice a cancel promptly and the only way to do - that with a condition variable was to wake every 20 ms and look. Each wakeup - overshoots by a few ms. - - | | before | after | - |---------------------------------|--------------|------------------| - | `@(future (Thread/sleep 2000))` | 2321-2358 ms | **2000-2001 ms** | - | `(Thread/sleep 2000)` on main | 2002-2005 ms | 2004-2005 ms | - | `future-cancel` latency | ≤ 20 ms | **0 ms** | - - A latch is woken *by* the cancel, so there is nothing to look for. This is the - argument for the larger diff: the smaller shape left this untouched while - stating the principle that condemns it. - -- **`(Thread/sleep 25000)` under a 3000 ms deadline returns the exhausted - marker**, which is the bug this cycle started from. It is the UNCATCHABLE - budget error, so evaluated code cannot swallow its own timeout — verified that - `(try (Thread/sleep 3000) (catch Throwable t :SWALLOWED))` under a 500 ms - deadline still yields the marker. - -- **`@(future (Thread/sleep 20000))` under a deadline returns the marker rather - than escaping.** Under the first version it exited 1 with an uncaught - `Exception: evaluation exceeded its time budget` unless the user happened to - wrap it in a `try`, so the playground would have rendered a crash rather than - "execution exceeded its deadline budget". - -- **`(Thread/sleep 100000000000000)` no longer aborts the process.** A legal - `long` that JVM Clojure simply sleeps on used to panic on integer overflow - (`ms * ns_per_ms` overflowing `u64`); the conversion saturates and the value - is clamped again before the instant arithmetic. - -- Semantics are unchanged where they should be: timed deref - (`[:timed-out 42 :p-to]`), cross-thread delivery (`[:hi true]`), cancel - (`[true true :cancelled-on-deref]`), a failed future re-raising the worker's - real error (`"boom"`), and `(await agent)`. - -- **What is still NOT closed**, named so this ADR cannot be read as closing it: - a `Thread.` started INSIDE a `with-budget` extent outlives it unmetered, and - cljw's join-at-exit barrier keeps the process alive with it. More broadly, - `rt.eval_budget` is one non-atomic slot shared by every thread — a worker's - back-edges charge the *main* thread's step counter — so the budget needs a - thread-ownership story now that more threads read it. D-571. - -- The playground's *serial accept* half is untouched: a legitimately long eval - still blocks the next request for up to the deadline (D-117(a)). An in-process - budget is a resource governor, not an isolation boundary — it cannot bound what - the metered thread spawned or an abort. The demo-layer answer is a subprocess - under an OS timeout; D-572. - -## Alternatives considered - -*(Devil's-advocate fork, fresh context. Its verification of this ADR's two -load-bearing claims is what caused the rewrite above; its Alternative B is what -landed. Reproduced with the measurement tables condensed — the numbers it -produced are in Consequences.)* - -**No alternative required violating any F-NNN.** F-005 / F-006 / F-009 are not -engaged. F-002 and the clj-parity floor both point away from the first version -and toward Alternative B. Diff size was the only thing separating them, and diff -size is not a project constraint. - -**Verification of claim 1 — "metered sleeps keep their duration."** The claim -holds; the numbers in the draft did not. Ten runs each, sorted, idle machine: -unmetered 2000/2001.5/2005, steps-only 2000/2004.5/2005, deadline-armed -2000/2004/2006 (min/median/max). Indistinguishable — which is the honest form of -the point. The draft's single-shot table reported the *metered* rows as faster -than the unmetered one, a physically impossible ordering and a tell that the -numbers were not repeated. The claim was also incomplete: `budgetedSleep` clamped -to 20 ms whenever a cancellable worker was present regardless of any budget, so -`@(future (Thread/sleep 2000))` ran +17% — the exact parity break the ADR names -as unacceptable, live, on the most idiomatic shape. Pre-existing (D-442 / -ADR-0153), but the ADR named two *other* pre-existing holes so it "cannot be read -as closing them" and was silent on this one. - -**Verification of claim 2 — "Zig 0.16 has no timed condition wait."** True of -`std.Io.Condition` specifically; the conclusion drawn from it is wrong. -`std.Io.Event` (L1766) exposes `waitTimeout` (L1827); `Io.Timeout` (L1132) is -`union(enum){none, duration, deadline: Clock.Timestamp}`; `Io.futexWaitTimeout` -(L1558) is the general mechanism, and `Condition.waitInner` is thirty lines over -`io.futexWait`. Verified by execution under the pinned toolchain: `.duration` -150 ms → `error.Timeout` at 151 ms; `.deadline` t0+200 → `error.Timeout` at -201 ms; a 30 s bound with `set()` from another thread at 100 ms → returned at -105 ms, no polling. `waitRealised`'s 1 ms poll *is* the existing shape, but it is -a cljw choice, not a platform constraint, and it was the load-bearing premise for -deferring the whole family. - -**Alternative A — smallest-diff: no helper; compute the cut once, inside -`Thread/sleep`.** Cap the requested duration to `deadline − now`, sleep once. -~8 lines, no loop, no callback parameter; ADR-0153's cancel loop stays where it -is. *Better*: it cannot regress sleep duration anywhere, and the landed loop's -degenerate edges disappear — with a deadline armed and a cancel poll present, -`wait` can be computed as 0 when `now == deadline_ns` exactly and the loop spins -until the clock advances; and `remaining` was decremented by the *requested* -slice rather than the elapsed one, an accounting drift that exists only because -there is a loop. *Breaks*: it is exactly the shape the draft's own "why the -helper" section argues against, and that argument is correct — `Thread/sleep` -ends up with two hand-composed bounding mechanisms three lines apart, which is -the precise configuration that let this bug survive. It fixes one primitive and -leaves the family, and leaves the 17% overshoot and the 20 ms cancel latency -untouched. **Do not take this one.** Listed because the brief asks for it, and -because its diagnosis of the loop's edges is worth folding in. - -**Alternative B — finished-form-clean: one bounded wait over -`std.Io.Event.waitTimeout`, and the whole family expressed through it.** Every -blocking primitive is the same sentence — wait for a latch, no later than -`min(caller timeout, budget deadline)`, waking immediately on cancel — and -`Thread/sleep` is its degenerate case. *Better*: it fixes the family the draft -names and defers, so D-571 shrinks from "design a bounded wait" to "convert the -remaining call sites"; **all polling disappears**, so `future-cancel` latency -goes from ≤20 ms to ~0 and the future-wrapped sleep goes back to ~2005 ms, i.e. -it *actually* delivers the parity property the ADR claims on the path where cljw -did not have it; one wait lands on the deadline instead of a loop of sleeps, so -"the deadline is not polled for" becomes structurally true rather than -true-by-arithmetic; the callback parameter disappears, since with a wakeable -cancel latch there is nothing to poll. *Breaks*: substantially larger diff — -future, promise, thread, plus the budget. `Io.Event` carries no payload, so the -mutex stays for the value handoff and the two must be kept consistent (set the -Event *after* publishing under the mutex, which `Event.set`'s release ordering -supports). `waitTimeout` returns `error.Timeout` on spurious wakeups, so callers -re-check against the same absolute deadline. `Io.Event.reset` asserts no pending -waiter, so any reusable latch needs care. **This is the recommendation, per -F-002.** The diff is larger by a wide margin; that is not a reason to prefer A. -The draft's whole justification is that "the next blocking primitive added would -re-open the hole silently" — and it closed that by adding a helper only one -primitive can use, whose doc comment tells the next author that the family needs -something else that does not exist. It does exist. - -**Alternative C — wildcard: stop making the budget the callee's business.** -Zig 0.16 has first-class cooperative cancellation (`Io.Cancelable`, `io.async` / -`Io.Future.cancel`, `io.checkCancel`), and every blocking `std.Io` call that is a -cancellation point already unwinds with `error.Canceled`. Run the metered thunk -under `io.async`, cancel it at the deadline, map `error.Canceled` to the budget -raise at exactly one boundary. *Better*: the only shape where "the next blocking -primitive re-opens the hole" is structurally impossible, because nothing opts in; -it makes the budget a property of the evaluation *context*, which also dissolves -the shared-slot race by construction. *Breaks*: cljw calls `waitUncancelable` -everywhere and hands out a process-global `std.Io`; C means moving the tree onto -cancelable variants and threading real `io` through. Cancellation is cooperative, -so a compute loop still needs `tick` — C sits beside ADR-0125, not in place of -it. Most seriously it interacts with F-006: an unwind can now originate inside a -blocking wait on a worker, and every published GC root and pin on that path -(`.dev/gc_rooting.md`) has to survive it. *Disposition*: record C as the -direction, not this cycle's shape. B is a strict subset of the work C needs, so -none of B is wasted if C later lands. - -## Revision history - -- 2026-08-04: Proposed. -- 2026-08-05: Proposed → Accepted, as Alternative B rather than the drafted - helper. The devil's-advocate fork falsified the draft's load-bearing premise - (`std.Io.Event.waitTimeout` exists and was verified by execution), so the - deferral of the blocking family to D-571 was a mistake, not a scope decision. - Decision and Consequences rewritten against the landed diff and re-measured. -- 2026-08-05 (b): the "still NOT closed" Consequences bullet is now closed — - D-571 discharged. The budget is a refcounted per-evaluation object - (threadlocal `current`, spawner-captured references for `Thread.` / `future` - / each agent action), the step counter is a shared atomic total, and a worker - that outlives the extent trips on the same budget. This also lands the - precondition Alternative C names (a budget object owned by the evaluation, - not the Runtime). diff --git a/.dev/decisions/0183_gc_trigger_is_steered_by_measured_time.md b/.dev/decisions/0183_gc_trigger_is_steered_by_measured_time.md deleted file mode 100644 index 87eb09854..000000000 --- a/.dev/decisions/0183_gc_trigger_is_steered_by_measured_time.md +++ /dev/null @@ -1,151 +0,0 @@ -# ADR-0183 — the GC trigger is steered by measured time, not a byte constant - -- Status: Proposed → Accepted -- Date: 2026-08-04 -- Amends: ADR-0164 (threshold-driven auto-collect; the 4 MB floor it set) -- Related: ADR-0028 (mark-sweep + adaptive threshold), F-006 (non-generational), - D-573 - -## Context - -cljw collects when `bytes_since_last_gc > max(threshold_floor_bytes, -last_live_bytes * 2)`, with the floor a 4 MB constant. - -ADR-0164 chose that constant and, in the same breath, named why it would fail: -*"For a churn workload `last_live ≈ 0` so the adaptive `max(1MB, last_live*2)` -arm is inert → the initial threshold IS the operative number."* It then tuned -the constant against the benchmark suite — short scripts, 1 to 15 collections -each — and gated the decision on "does any of the 7 won benches regress". - -Nothing measured a program that runs for minutes. On one, the constant costs a -factor of two. - -**Measured** on a Rush Hour BFS (cw-arcade `apps/rush-hour`, a Clojure program -that allocates a persistent map per move and keeps a set of visited states), -identical 3.17 GB / 26.2 M allocations in every row: - -| floor | collections | wall | -|----------------|-------------|--------| -| 4 MB (default) | 755 | 17.7 s | -| 16 MB | 65 | 9.6 s | -| 64 MB | 47 | 7.6 s | -| 256 MB | ~40 | 7.1 s | - -A sampling profile put `collectStopTheWorld` at ~40 % of all samples. The -marginal cost of a collection, measured directly on the real workload by -forcing three different collection counts with `CLJW_GC_TORTURE_ALLOC` and -taking Δwall/Δcollects, is **13.6 ms** — and `last_live_bytes` reports -**747 KB**. So the run was spending ten of its eighteen seconds re-marking the -same graph, 755 times, because a constant said to. - -## Decision - -**The floor is steered by the measured GC time share.** - -`collectStopTheWorld` times itself; `steerFloor` compares -`gc_ns_total / (now − first_collect_ns)` against a target and moves -`threshold_floor_bytes` by doubling or halving, bounded below by whatever the -program started with (default, or `CLJW_GC_THRESHOLD_MB`) and above by 64 MB. - -The share is the right control variable on the theory as well as the -measurement: for a mark-sweep collector the GC share is -`mark_cost(live) / bytes_between_collections`, so the trigger *is* the share, -and steering on it is independent of how big the live set happens to be — which -matters here more than usual, for the reason in Consequences. - -**This is the shape the established runtimes use**, which a cross-runtime survey -(`private/notes/2026-08-04-gc-heap-sizing-survey.md`) confirmed after the fact: - -| runtime | mechanism | -|-----------------|----------------------------------------------------------------------------------------------------------------------------------| -| HotSpot | `-XX:GCTimeRatio` — goal is `1/(1+ratio)` of time in GC; the adaptive size policy grows the generations when the goal is missed | -| **SubstrateVM** | `GC_TIME_RATIO = 19` → a **5 %** budget, with the young gen *derived* from it | -| G1 | `-XX:MaxGCPauseMillis` — a time target, young-gen size the derived quantity | -| V8 | growing factor computed from measured `gc_speed` and `mutator_speed` | - -The SubstrateVM row is the one that matters: **babashka is compiled with GraalVM -`native-image` and therefore runs SubstrateVM's Serial GC with its adaptive -policy** — so the runtime cljw is benchmarked against already sizes its heap -from a measured GC time fraction. Same deployment shape (AOT single binary, -startup-sensitive, no JIT warmup), same answer. - -10 % rather than SubstrateVM's 5 %: cljw is non-generational (F-006), so it -re-marks the entire live set every cycle and cannot reach 5 % without holding -more heap. Measured, the two are within noise on the BFS (5 % → 7.55 s / -56 collections, 10 % → 7.54 s / 69), so the laxer target is taken for the -smaller footprint. - -### What was tried first, and why it is not what landed - -**A byte floor that ramps after N inert collections.** It works — but the survey -found no established collector shaped that way, and its three properties are -each wrong: one-way (never comes back down, so a long-lived REPL or nREPL -server keeps a floor its phase no longer needs), counter-triggered rather than -cost-triggered, and keyed on a quantity that under-reports (below). Go -considered and **rejected** an adaptive minimum heap for this same problem -(golang/go#22743). - -An earlier version ramped on the *first* inert collection and cost -`gc_alloc_rate` **20 %** — because collecting is also what refills the free -pool, so a program short enough to finish inside a few collections wants them. -That is ADR-0164's finding, and it is why the target is a *share* rather than a -threshold: a program that collects rarely has a low share and is left alone. - -## Consequences - -- The BFS goes 17.9 s → 7.5 s (755 → 69 collections). `medium`-difficulty - puzzle generation, the thing a user actually waits for, goes 57.8 s → 17.5 s. -- **No benchmark moves.** `gc_alloc_rate` 43→43 ms, `gc_large_heap` 39→39, - `string_ops` 29→29, `gc_stress` 29→28, `map_filter_reduce` 17→16, - `map_ops` 11→11 (best of 3 each). Their collection counts are 15, 2, 1, 4 — - far too few to accumulate a share worth steering on. -- Peak RSS on the BFS is unchanged (2.21 → 2.23 GB); it is dominated by the - free pool, not the trigger. Startup is unchanged at 10 ms. -- Two clock reads per collection. At 69 collections that is not measurable; at - 755 it was not either. -- **The deeper defect is NOT fixed, and this ADR is not a substitute for it.** - `last_live_bytes` counts only the GC record block — it omits finaliser-owned - side buffers (String bytes, BigInt limbs) and everything in - `persistent_marks`, which is traced on every collection but never swept and - never counted. Measured 78 MB reported against 359 MB RSS, a factor of 4.6. - So `last_live * 2` is not a proportional-to-cost heap target, and the - adaptive arm can read as *inert* while the collector is tracing a large - graph. The BFS is exactly that false negative: 747 KB reported, 13.6 ms to - collect. - - Steering on measured time works **regardless** — that is its virtue over both - the byte floor and the ratio arm, and why it could land before the accounting - is fixed. But with honest accounting the ratio arm may govern on its own and - this steering may become redundant. Go's answer to precisely this problem - (1.18) was to *complete the marginal cost model*, not to add a heuristic on - top of an incomplete one. Tracked as D-573. - -## Alternatives considered - -The cross-runtime survey in -`private/notes/2026-08-04-gc-heap-sizing-survey.md` is the alternatives -analysis: it enumerates Go's pacer (same `max(min_heap, live × ratio)` shape, -same problem, documented and unfixed), HotSpot's and SubstrateVM's time -ratios, G1's pause target, V8's measured growing factor, OCaml's -`space_overhead`, Ruby's ramping `RUBY_GC_MALLOC_LIMIT`, and Boehm's -`GC_free_space_divisor` — and recommends, in priority order: (1) run the -torture-slope experiment on the real workload before anything lands; (2) fix -the live accounting; (3) replace the ramp with a GC-time-fraction target; -(4) failing that, a larger static floor; (5) fix `conservativeStackScan`. - -(1) was run — 13.6 ms, which is what makes the rest apply. (3) is what landed. -(2) and (5) are D-573. - -## Revision history - -- 2026-08-04: Proposed → Accepted. -- 2026-08-05: the Consequences bullet "the deeper defect is NOT fixed" is now - half-resolved: `last_live_bytes` counts finaliser-owned side buffers via a - per-tag `ownedBytes` hook table (D-573 narrowed). The `persistent_marks` - half turned out to be a GROWING set (every closure allocation adds a - re-traced waypoint) and is measured but deliberately EXCLUDED from the byte - arm — folding it in regressed gc_alloc_rate ~11% by starving the free pool. - The division of labour stands: the byte arm sizes against the swept heap, - this ADR's time-share arm owns total traced cost (which is why the BFS still - needs it: its 13.6 ms/collect was waypoint tracing, invisible to any byte - count of the swept heap). CLJW_GC_STATS=2 prints the per-tag breakdown. diff --git a/.dev/decisions/0184_functions_are_collectable.md b/.dev/decisions/0184_functions_are_collectable.md deleted file mode 100644 index 68c5a75b0..000000000 --- a/.dev/decisions/0184_functions_are_collectable.md +++ /dev/null @@ -1,295 +0,0 @@ -# ADR-0184: Runtime-created Functions are ordinary collectable heap objects - -- Status: Accepted -- Date: 2026-08-06 -- Refs: D-450 (perf campaign, real-workload gap), ADR-0028 (mark-sweep, - per-tag trace/finaliser), ADR-0130 (frame rooting), ADR-0183 (trigger - steering), F-006 (GC strategy), D-556 (conservative scan residual) - -## Context - -Every fn value — including every closure created at runtime — is allocated -with `rt.gpa.create(Function)` + `rt.trackHeap(...)`, which registers it in -`gc.persistent_marks` (a mark-waypoint list) and `rt.heap_objects` (the -teardown free list). Nothing ever unregisters one, so **every closure a -program creates is immortal**. - -Measured on the D-450 headline workload (cw-arcade rush-hour `(g/generate 7 -:medium)`, M4 Pro, v1.9.0+O-055): - -- 15,502,063 persistent marks at exit, **1.98 GB retained** — the BFS - creates a closure per iteration and none is ever freed. -- `CLJW_GC_STATS=3` phase timing: gc_ms=3858 of a 16.3 s run, of which - **prep = 2830 ms** — `clearPersistentMarks` bit-clearing all 15.5M - waypoints on every one of the 122 collects (the walk grows with every - closure ever created, so the cost is quadratic-ish over the run). The - mark-worklist reserve is also inflated by `persistent_marks.len`. -- This is simultaneously the campaign's largest measured lever (~17% of - wall) and an unbounded-memory defect: any server-shaped cljw process that - creates closures per request grows without bound. - -The `persistent_marks` design is correct for its other occupants (protocol -descriptors, type-descriptor refs — one per definition, genuinely -process-lifetime). Function is the only churn-class occupant (plus reify's -per-evaluation anonymous TypeDescriptor, recorded as a residual below). - -Prior art (survey: `private/notes/D450-fn-gc-survey.md`): cw v0 treated -`.fn_val` as an ordinary collectable object; JVM Clojure closures are plain -GC'd instances of a compiled class. "Per-closure instance is light and -collectable; the shared code object is long-lived" is the established -finished form. cw v1's immortal closures are the deviation. - -## Decision - -(Amended after the DA fork below — the fork refuted the draft's rejection -of the inline cell and found the two preconditions incomplete.) - -1. **Function allocation moves to the GC heap as ONE variable-length - cell.** All four alloc paths (`allocFunctionWithBytecodes`, - `allocFunctionTemplate`, `allocFunctionFromSerialized`, and the - compile-time slot_base==0 path through the first) allocate a single - `gc.alloc`-class `.fn_val` cell whose trailing bytes hold the - `methods` array and `closure_bindings`; the struct keeps its **slice - fields**, initialized to point into the cell's own tail, so every - existing `f.methods` / `f.closure_bindings` access compiles unchanged. - Nothing resizes those slices post-alloc (`patchLetfnClosures` mutates - contents, not length), so the aliasing is safe. No finaliser is - needed — sweep frees one cell; `freeFunction` and the Function entries - in `rt.heap_objects` / `gc.persistent_marks` disappear. Free-pool - size-class dispersion is measured in the landing commit. -2. **Chunks stay immortal and unowned by Function.** Verified: chunks are - compiled once per `fn*` form into the run-lifetime analyzer arena - (`compiler.zig`), and `op_make_fn` copies chunk *pointers* into each - closure — 15.5M closures share one chunk. The serialize path's chunks - are serialize-allocator-owned. No chunk lifetime change. -3. **Precondition A — executing-callee roots, installed BEFORE the - binder.** A collectable callee must be reachable while it runs: - - tree_walk: `callMethodImpl` holds raw `*Function` + `*FunctionMethod` - (a pointer INTO `f.methods`) across the body/recur loop. Its - per-call EvalFrame gains an explicit `callee: Value` slot — and the - root must be live **before `bindCallFrame` runs**, because the - binder's rest-pack `consHeap` can trigger a collect while the callee - is held only as a raw pointer (DA gap A-1). - - VM: `flattenPush` sets `ar.op_top = fr.result_slot`, dropping the - callee below the rooted prefix; the flattened frame's `gc_frame` - gains the same `callee` slot (belt-and-braces on the VM — after - flatten nothing re-reads `f`/`m`; the load-bearing pin is - tree_walk's `m`). The `op_call` slow path is safe only because - `ar.op_top` is synced in stepOnce's defer and therefore stays - stale-HIGH across `vt.callFn` — that staleness is now a documented, - test-locked invariant (DA gap A-2). -4. **Precondition B — ALL THREE descriptor registries become roots, with - extend-type replace-in-place.** `rt.types`, `rt.native_descriptors` - (where `(extend-type Long/nil …)` impls actually land), and - `rt.class_descriptors` gain a root-enumeration cursor over their - method-table values (reify's anonymous TDs stay instance-mediated). - Because `extendTypeWithImpls` today APPENDS (`old ++ new`, nothing - dropped), walking tables as roots would immortalize every superseded - impl — so extend-type becomes **replace-in-place** (dedupe on - protocol+method) in the same cycle. This also fixes the pre-existing - parity defect the DA surfaced: `lookupMethod` scans front-first while - re-extends append at the end, so a re-extend never won (verify against - clj and pin). -5. **Compile-time Functions stay effectively immortal** via - `persisted_analysis_roots` — unchanged. Only runtime-created closures - become collectable, which is exactly the churn population. -6. **Named finished form to converge on (not obstructed by this shape): - the template/closure split** (DA Alt 3) — runtime closures borrow the - immortal template's `methods` and own only their bindings. Dissolves - the tree_walk `m` pin and removes the per-closure methods copy. - Recorded as the follow-up lever on D-450 together with the epoch-mark - idea (DA Alt 1) for the remaining waypoint/allocations clears. - -## Alternatives considered - -(Devil's-advocate fork output, reflected verbatim per the working -agreement. The fork was briefed with F-002/F-004/F-006/F-011/F-012 and -read the draft + survey + source read-only.) - -> ## Leading finding (F-NNN envelope) -> -> No finished-form-clean option requires violating any F-NNN. All three -> alternatives below fit F-004 (pointer identity — no object ever moves), -> F-006 (non-moving mark-sweep, per-tag trace/finaliser), F-011/F-012 -> (both backends change together through the shared binder). The draft's -> direction (fns collectable) is correct; the findings below are about -> **where the draft's two preconditions are incomplete** and about one -> rejected alternative whose rejection rationale does not hold up. -> -> **Alt 1 (smallest-diff): epoch-stamped marks — kill the prep walk -> without making fns collectable.** Replace the per-collect O(n) -> bit-clear with a global mark epoch ("marked" ≡ `header.mark_epoch == -> gc.current_epoch`; collect increments the epoch, O(1) prep). Removes -> the ~2.8 s prep cost with zero new UAF surface, and is a good idea -> independently (the allocations-list clear pays the same cost shape). -> Breaks nothing — but fixes only the perf half; the 1.98 GB unbounded -> retention remains. On F-002 grounds it loses as the whole answer; -> record as a compatible follow-up lever. -> -> **Alt 2 (finished-form-clean): single variable-length `gc.alloc` cell, -> landed NOW — the draft's rejection rationale is partly wrong.** Keep -> `methods` / `closure_bindings` as slice fields initialized to point -> into the cell's trailing bytes: every existing access compiles -> unchanged; only the four alloc paths and the free path change, and the -> finaliser disappears (sweep frees one cell). Nothing resizes those -> slices post-alloc (`patchLetfnClosures` mutates contents, not length). -> Size-class dispersion is the one genuinely unmeasured item, but the -> non-moving heap already handles variable-length cells (String bytes). -> Per F-002, an unmeasured guess plus a refutable access-shape claim must -> not demote the finished form to a follow-up: land the inline layout in -> this cycle, measuring dispersion in the landing commit. -> -> **Alt 3 (wildcard): two-tier template/closure split — per-closure cell -> = header + bindings only; methods borrowed from the immortal -> template.** `op_make_fn` today copies the whole `methods` array per -> closure even though all 15.5M closures share one chunk and -> byte-identical FunctionMethod values. Borrowing the template's methods -> dissolves most of Precondition A (`m` then points into immortal -> storage), and is the survey's own JVM/cw-v0 finished form taken -> literally. Breaks: template backreference + no-template paths must -> decide methods ownership. Substantially larger diff — recommended -> anyway per F-002 as the shape to converge on, with Alt 2's inline cell -> as its natural companion; if not taken this cycle, name it as the -> finished form the current shape must not obstruct (it does not). -> -> ## Precondition A stress-test — call shapes the draft misses -> -> Verified safe (state in the ADR): agent/future actions are already -> pinned (future.zig:15-17,154,229; agent.zig:232-245 — the queue is -> off-heap gpa, so `action.body`/`completion` are pinned); worker -> EvalFrames are collector-visible (root_set.zig:314-327, 413-421), so -> the callee slot works cross-thread; keyword-as-fn involves no -> Function; partial/comp inner fns are reached via the outer callee's -> closure_bindings trace; Var redefinition during execution is covered -> by the callee slot. -> -> The gaps: -> 1. **The binder runs BEFORE the frame is installed — with an -> allocation inside.** In `callMethodImpl`, `bindCallFrame` (which -> consHeap-allocates the `& rest` pack, tree_walk.zig:1436-1441) -> executes before the `gc_frame` push (1466-1472). A rest-arity -> call's cons alloc can collect while the callee is held only as a -> raw `*Function` — no root; on a worker thread there is no -> conservative scan (D-556), and main-thread stack survival is an -> optimizer accident, not a contract. Fix: install the callee root -> before `bindCallFrame`. The two drafted torture cases will NOT -> catch this — add a variadic-callee case that collects inside the -> rest-pack alloc (CLJW_GC_TORTURE_ALLOC through -> `(apply (fn [& xs] xs) …)` with the callee the only reference). -> 2. **The VM `op_call` slow path is safe only by a deferred-sync -> accident.** During `vt.callFn` (vm.zig:711-726) the callee+args sit -> below a stale-HIGH `ar.op_top` (synced only in stepOnce's defer, -> vm.zig:484-489) and stay rooted — nothing documents that the -> staleness is load-bearing. Any refactor that syncs earlier silently -> unroots the executing callee. Record the invariant next to the -> O-028 hoist comment and lock it with a test. -> 3. **"The callee root removes the fn-value instance of D-556" is -> overbroad.** It roots a fn only while its frame is live. Fn values -> fetched from heap objects and held in native locals across an -> alloc/safepoint on a worker (lazy_seq.force's `ls.thunk` exposure -> via the ls intermediate, delay, watch/validator loops, value_sort -> comparators, reduce/map step-fn drivers) join the D-556 residual -> class the moment fns become collectable — a UAF surface that -> immortality currently masks. Narrow the claim to the -> executing-callee instance and price the rest into the D-556 row. -> 4. **Newborn window**: a fresh gc.alloc'd fn is unrooted until it -> reaches a stack slot/local. Current paths look clean (follow-on -> allocs are gpa-side; op_make_fn pushes before any further -> gc.alloc), but require a per-path audit line; the remedy vocabulary -> is `gc_self_guard` / fabrication brackets (ADR-0150/D-559). -> 5. **Multimethod/protocol fetch-to-call window**: a method fn fetched -> from a table (`CallSite.last_method` is a raw `*MethodEntry`, "no -> GC edge", method_table.zig:37-46) is protected between fetch and -> frame-install only by the table staying reachable and the entry not -> being replaced. Mostly closed by Precondition B; combined with gap -> 1 (a variadic protocol method) it is real. One audit line. -> -> ## Precondition B stress-test -> -> 1. Cost is a non-issue: dozens of descriptors × <32 entries vs the -> 15.5M-entry clear being removed. -> 2. **The draft walks the WRONG (incomplete) registry set**: -> `rt.native_descriptors` ([70]?*TypeDescriptor, runtime.zig:417) is -> NOT in `rt.types` — and it is where `(extend-type Long/nil …)` -> impls land; `rt.class_descriptors` can also receive method entries; -> reify's anonymous TDs are deliberately not in `rt.types` and their -> instance-mediated trace + callee slot must be verified sufficient. -> Enumerate all three registries or the fix is porous. -> 3. **Root-vs-waypoint changes replacement semantics — because -> replacement never removes.** `extendTypeWithImpls` appends -> (`old ++ new`, runtime/protocol.zig:237-241); walked as a root, -> every superseded impl closure is immortal and an extend-type in a -> loop grows without bound — a miniature of the disease this ADR -> kills. Land replace-in-place (dedupe on protocol+method) in the -> same cycle. -> 4. **Adjacent latent bug**: `lookupMethod` scans front-first -> (type_descriptor.zig:291-297) while re-extends append at the END — -> a re-extend's new impl appears to never win (clj: later extend -> replaces). If confirmed, a pre-existing parity bug hidden by the -> append-only growth; the replace-in-place fix discharges both. Also: -> runtime/protocol.zig:250 frees the old slice while the test comment -> (392-396) claims never-free — the generation check keeps the code -> safe; clean the contradiction in passing. -> -> ## Mid-execution death / recur `m` pinning -> -> tree_walk: the callee slot pins `f.methods` from frame-install onward; -> the hole is entirely the pre-install window (gap 1). VM: pinned but -> not needed mid-execution — after flattenPush nothing re-reads `f`/`m` -> (recur is an intra-chunk jump; chunk arena-owned; constants rooted; -> captures memcpy'd into rooted locals). The only load-bearing -> methods-pin in the design is tree_walk's `m` — which is precisely what -> Alt 3 dissolves, strengthening it as the finished form. -> -> ## Recommendation -> -> Keep Decision 1/2/5 core. Amend Precondition A per gaps 1-2, narrow -> the D-556 claim per gap 3, rewrite Precondition B to cover all three -> registries with replace-in-place landed alongside. Adopt Alt 2 this -> cycle. Record Alt 3 as the named finished form (or take it now) and -> Alt 1 as an independent follow-up lever. - -## Consequences - -- A closure-churning program's fn cells are reclaimed like any other - value; `persistent_marks` returns to its intended dozens-scale - population, killing the per-collect prep walk (~2.8 s on the headline - workload) and the 1.98 GB retention. -- `Runtime.deinit`'s `heap_objects` drain no longer frees Functions; the - gc-side allocations drain does (one rawFree per cell), keeping the leak - gate green. -- The registry roots close the pre-existing KNOWN-OPEN strand on - extend-type captures. The callee slot covers the EXECUTING fn only; - fn values fetched from heap objects and held in native locals across a - safepoint on a worker thread join the D-556 residual class once fns are - collectable — that widening is priced into the D-556 row, not absorbed - silently (DA gap 3). -- `.dev/gc_rooting.md` §A/§D/§F + site census + migration checklist, the - `heap_tag.zig` membrane comment, and the `traceFunction` docstring are - updated in the landing commits. -- Locked by: full diff oracle ×2 backends, corpus, `CLJW_GC_TORTURE` e2e - suite + two NEW torture cases ("loop-created closures die; live ones - survive a collect", "self-recursive closure survives a mid-execution - collect"), DebugAllocator leak gate, and a rush-hour re-measure - (CLJW_GC_STATS=1: persistent_marks back to O(100), prep ≈ 0). - -## Measured outcome (landing commit, 2026-08-06, M4 Pro ReleaseSafe) - -Rush-hour `(g/generate 7 :medium)` headline: **16.5 s → 12.3 s (−26%, -median of 5)**. `CLJW_GC_STATS`: persistent_marks 15,502,063 → **110** -(retained 1.98 GB → 5 KB — the leak is gone), gc_ms 3858 → 840, prep -2830 ms → 0 ms. Collects 122 → 2421 at the 4 MB floor because the GC -share now sits at ~7% — INSIDE ADR-0183's 10% steering target, so the -floor no longer needs to ramp. Free-pool dispersion non-issue: mallocs -560k → 45k, reuse 99% (fn cells recycle through the pools). fib 32 -unchanged (459 ms) — no regression on closure-light workloads. The -cumulative campaign arc: 17.9 s (v1.9.0) → 12.3 s (−31%), clj gap 17x -→ 11.5x. - -## Residuals (recorded, not in this cycle) - -- reify's anonymous TypeDescriptor is trackHeap'd per **evaluation** - (protocol.zig) — the same immortality shape in miniature; new debt row. -- Worker-thread tree_walk conservative-scan residual (D-556) remains for - non-fn intermediates; the callee root removes the fn-value instance of - it. diff --git a/.dev/decisions/0185_one_gate_configuration.md b/.dev/decisions/0185_one_gate_configuration.md deleted file mode 100644 index 30113553f..000000000 --- a/.dev/decisions/0185_one_gate_configuration.md +++ /dev/null @@ -1,120 +0,0 @@ -# ADR-0185 — One gate configuration, everywhere - -- **Status**: Proposed → **Accepted** (2026-08-12) -- **Supersedes**: the tiering half of [ADR-0107](0107_pipeline_gate_smoke_authorized.md) - (revision 2026-07-21) and the nightly leg of - [ADR-0049](0049_orbstack_linux_gate_retired.md)'s batched-Linux arrangement. - ADR-0107's *local* smoke tier is untouched. -- **Affected files**: `.github/workflows/ci.yml`, `scripts/ci_gate.sh`, - `scripts/check_gate_parity.sh`, `scripts/check_vm_parity.sh`, - `scripts/run_remote_ubuntu.sh`, `.claude/rules/gate_cadence.md`, - `test/e2e/phase14_nrepl*.sh`, `test/e2e/entrypoint_eval_parity.sh` - -## Context - -CI ran three configurations and called all three "the gate": - -| trigger | what ran | -|--------------|-------------------------------------------------------------------------------------------------| -| pull_request | `test/run_all.sh --smoke` (no e2e shell suite) | -| push to main | `test/run_all.sh --serial-e2e` (== the local full gate) | -| nightly cron | the above **plus** `check_vm_parity.sh` (corpus + every e2e on a `-Dbackend=tree_walk` rebuild) | - -`scripts/check_gate_parity.sh` already existed because this class had bitten -before: for months `ci_gate.sh` passed `--serial-e2e` while `run_gate.sh` passed -nothing, so "the local gate is green" and "CI is green" were statements about -two different runs, and an e2e that died on SIGPIPE under serial passed under -parallel — green locally, red on every push. That check fixed the flag; the -tiers stayed. - -The nightly then produced the argument against itself. Its last run -(2026-08-12, `376f81c8`) failed on **`phase14_nrepl`: `.nrepl-port` not created -within 5s**, on the tree_walk build, on a loaded shared runner. The six nightlies -before it passed. Measured locally on that same tree_walk ReleaseSafe build, the -bind takes **0.08 / 0.14 / 0.08 s** — 35-60x under the bound. So the failure was -a fact about the runner, reported a day late, in a configuration no ordinary gate -run reproduces. `.claude/rules/test_taxonomy.md` had already written down why -("a RATIO upper bound is a flake generator … it eventually fails for the one -reason it was not written to detect"); the nightly was the only place positioned -to collect on it. - -The project is also no longer maintained (README, 2026-08-12), so a schedule that -wakes up daily to re-run a config nobody will triage is pure liability. - -## Decision - -**One configuration. PR, push, dispatch and `scripts/run_gate.sh` all run -`zig fmt --check src/` + `test/run_all.sh --serial-e2e`.** - -1. `ci.yml` loses the `schedule:` trigger and both `CLJW_CI_*` env switches. -2. `ci_gate.sh` loses its tier branches — one unconditional `run_all.sh` call. -3. `check_gate_parity.sh` is extended from "both pass `--serial-e2e`" to - "there is exactly ONE configuration": exactly one `run_all.sh` invocation in - `ci_gate.sh`, no `CLJW_CI_FULL`/`CLJW_CI_PARITY` in the script or the - workflow, no `check_vm_parity.sh` wired into a launcher. The greps read code, - not comments, so the headers can still name the retired switches to explain - them. -4. `check_vm_parity.sh` survives as an **on-demand** tool, its header saying so. -5. The nREPL port-file bounds go 5 s / 10 s / 15 s → **30 s**, with the reason - in the comment: an order-of-magnitude bound still separates "never came up" - from "busy runner", which is all the assertion is for. - -## Consequences - -- "Green" means one run, wherever it is said. A tier cannot return without - failing `gate_parity`. -- PR CI gets slower (smoke → full). Irrelevant here: PRs are disabled on this - repo, and the project is not taking new work. -- **The e2e shell suite no longer runs on the non-default backend in CI.** This - is the one real loss, and it is smaller than the sweep's name suggests: the - F-012 dual-backend differential oracle — the thing that actually pins - tree_walk ≡ vm semantics — runs inside `zig build test` on **both** backends - in every gate (`zig_build_test_vm` + `zig_build_test_tree_walk`). What is gone - is e2e *shell* coverage of the non-default backend, recoverable on demand with - `bash scripts/check_vm_parity.sh` (or `run_remote_ubuntu.sh --parity` for - Linux × tree_walk, the slowest combination and where timing bounds break - first). -- A genuine tree_walk-only e2e regression would now be found by a person running - that script, not by a schedule. Accepted: for an unmaintained project, a - finding nobody triages is not coverage. - -## What the unification turned up on its way in - -Removing the tier did not make the first full gate green — `entrypoint_eval_parity` -died with exit 137 and no explanation. Two real defects were behind it, both -found because the oracle was made to survive a dying entry point instead of -inheriting its status: - -1. **The oracle aborted instead of reporting.** `got=$(run_$ep …)` under - `set -e` propagates a killed child's 137 and takes the step with it — in the - one test whose entire job is to notice that an entry point behaved - differently. It now captures status and raw output per entry point, so a - death is a named divergence with the last 20 lines attached. That change is - what produced the diagnosis in one run. -2. **Every nREPL e2e leaked its server.** `( cd "$WORK" && "$BIN" nrepl … ) &` - makes `$!` the subshell, not the runtime, so `kill $!` orphaned one `cljw` - per start: 3 left behind by `phase14_nrepl_classpath`, 4 by - `phase14_nrepl_toplevel_do`, plus one per case in the new oracle. Enough - accumulated memory that macOS SIGKILLed an unrelated freshly-built binary — - a leak presenting as a crash. All three now reap by scoped port pattern, and - all four nREPL e2e end with zero surviving processes. - `.claude/rules/orphan_prevention.md` carries the pattern as rule 4. - -The CI runner had been reporting this leak all along, as a "Terminate orphan -process: pid (…) (cljw)" flood at job cleanup. Nobody read it as a finding. - -## Alternatives considered - -1. **Keep the nightly, raise the bound only** (smallest diff). Fixes this - symptom, keeps the class: CI still verifies something the local gate does not, - so the next divergence is again found a day late by a config nobody runs. It - also leaves a daily job on a repo that is winding down. -2. **Keep the nightly, run it in the local gate too** (parity by addition). - Genuinely closes the divergence and keeps the coverage — at a second - ReleaseSafe rebuild plus a full serial e2e pass (~5 min) on *every* gate, - local included. Rejected on the coverage/cost ratio given the diff oracle - already covers dual-backend semantics, and doubly so for a project that has - stopped. -3. **Delete `check_vm_parity.sh` outright.** Simplest final state, but it throws - away a diagnostic that costs nothing to keep and that a forker may want. The - divergence was the schedule, not the script. diff --git a/.dev/decisions/README.md b/.dev/decisions/README.md deleted file mode 100644 index d6506991e..000000000 --- a/.dev/decisions/README.md +++ /dev/null @@ -1,45 +0,0 @@ -# Architecture Decision Records - -> Load-bearing decisions only. ADRs document *why* a decision was made so -> that future readers (including future Claude sessions) do not re-litigate -> it. Skip ADRs for ephemeral choices ("not worth it right now") or for -> facts that are obvious from the code. - -## Filename convention - -`NNNN_.md` - -- `NNNN` — 4-digit sequential index, zero-padded -- `` — short English identifier in snake_case -- `0000_template.md` — template (do not delete or renumber) - -## Required structure - -Use [`0000_template.md`](./0000_template.md) as the starting point. Every -ADR has: - -- **Status**: Proposed / Accepted / Superseded by NNNN / Deprecated -- **Context**: what motivated the decision (constraints, prior art) -- **Decision**: what was chosen -- **Alternatives considered**: what was rejected and why -- **Consequences**: positive, negative, neutral -- **References**: ROADMAP §, related ADRs, external docs - -## Lifecycle - -- **Add**: when a load-bearing decision surfaces in the autonomous loop. - Number = max(existing) + 1. The AI drafts the ADR with - `Status: Proposed → Accepted` in the same cycle, lands the doc commit, - and proceeds with the source change. No external review gate. -- **Supersede**: do not edit a historical ADR. Add a new one and mark the - old one `Status: Superseded by NNNN`. -- **Reject after consideration**: also add an ADR with - `Status: Proposed → Rejected`. Records why the path was not taken. - -## Commit gate trigger - -Adding `.dev/decisions/NNNN_.md` (real ADRs; not `README.md` and -not `0000_template.md`) makes the commit "source-bearing" per -`scripts/check_learning_doc.sh`. The next doc commit must include the -ADR's commit SHA in its `commits:` front-matter list — see skill -`code_learning_doc`. diff --git a/.dev/gc_rooting.md b/.dev/gc_rooting.md index 817b16b2b..7e4f311cc 100644 --- a/.dev/gc_rooting.md +++ b/.dev/gc_rooting.md @@ -63,13 +63,23 @@ BEFORE `bindCallFrame` (the binder's rest-pack consHeap can collect). | A5 | `runtime/concurrency/lock_tx.zig` (`fireWatches`) | post-commit ref-watch firing | flat `[ref, old, new, ...]` notifies list published whole: an already-fired `old` recycled from the ring would otherwise be swept before a later notification fires | | A6 | `lang/primitive/agent.zig` (`sendFn`) | agent action ENQUEUE window (D-418) | 1 slot `[action]` — the `[f & args]` vector rooted across `agent.send` until `enqueueDirect` appends it to the off-heap queue (its traceGc root); a collect in the window otherwise sweeps the freshly-built vector → drainer reads recycled memory | | A7 | `lang/primitive/agent.zig` (`awaitFn`) | await barrier promise ENQUEUE window (D-418) | 1 slot `[p]` — the completion promise rooted across `sendAwait` until the barrier action carrying it is queued (traceGc walks `action.completion`); else `await` blocks on a swept promise (hang) | +| A8 | `runtime/lazy_seq.zig` (`realize`) | complete lazy realization (ADR-0197) | 1 slot `[raw]` roots the returned body across nested thunk calls and allocating Seqable coercion; immutable published inner links retain subsequent nodes | +| A9 | `runtime/seqable.zig` (`rest`, `instanceRest`, `instanceNext`) | accessor after Seqable coercion (ADR-0197) | 1 slot `[cursor]` retains a freshly returned ISeq and backing through allocating tail accessors or user callbacks | +| A10 | `runtime/java/util/Iterator.zig` (`fromSeqable`, `next`) | host iterator boundaries (ADR-0197) | 1 slot roots the fresh cursor until host allocation publishes it, or the returned head while advancing the cursor invokes user code | +| A11 | `lang/primitive/csv.zig` (`writeCsvFn`) | CSV traversal (ADR-0197) | 3 slots `[rows, row/cells, cell]` retain fresh sequence views and values across lazy, custom ISeq, and printing callbacks | +| A12 | `lang/primitive/sorted.zig` (`subseqSorted`, `takeWhileBound`) | custom Sorted bounds (ADR-0197) | 3 slots `[comparator, cursor, head]` retain callback results; the bounded walk separately roots its cursor, fresh entry and accumulated items while comparator callbacks can collect | +| A13 | `runtime/equal.zig` (`Cursor.next`) | equality cursor advance (ADR-0197) | 2 slots `[cursor, head]` retain a fresh head across custom rest callbacks until the outer equality frame receives it | **Migration-impact:** `stack`/`locals`/`constants` are **const** views; a moving GC must rewrite relocated operand/local/constant Values, so they become mutable and the walk forwards (not just marks). `chunk.constants` lives in the analyser arena — under relocation the literals it holds still move, so the pool needs writability or a literal-copy-to-heap. A2's `gc_roots` Zig array is already a -mutable local handed to the collector by address — migration-clean. +mutable local handed to the collector by address — migration-clean. A8–A13 +also publish mutable arrays; a moving collector must additionally reload the +decoded LazySeq pointer and accessor cursor locals from those forwarded roots +after callbacks/allocations, refresh copied comparator Values in bound structs, +and preserve the incoming receiver root contract. ## B. Threadlocal root slots @@ -101,7 +111,11 @@ collect) while holding a `Value` accumulator in a Zig local on no published stack. ROOTED → opened an `EvalFrame`. UNROOTED-CANDIDATE → latent UAF the next torture round can hit. -**ROOTED (safe):** `reduceFn` (A2) — the canonical exemplar. Also +**ROOTED (safe):** `reduceFn` (A2) — the canonical exemplar. Lazy realization +(A8) and post-coercion tail access (A9) retain their intermediate values too. +Allocation-torture verification must run on the CLI thread: registered nREPL +workers bypass the current alloc-torture trigger, although explicit `System/gc` +in their callbacks still exercises root enumeration. Also `equal.zig` `seqEqualInstance` (realizes a Sequential-instance operand via the ISeq `-next` protocol) and `seqEqualWalk` (walks two native seqs via lazy cursors — its frame roots the operands, the advancing cursor heads, AND the @@ -117,11 +131,15 @@ intermittently; minimal repro `(= (map inc (range 50)) (filter pos? (map inc (range 50))))` → false under alloc-torture). Now frame-rooted; e2e guard `phase16_gc_torture/lazy_eq_interleave`. -**UNROOTED-CANDIDATE (tracked D-252):** +**UNROOTED-CANDIDATE (tracked D-252; discharged rows marked):** + +C1 now roots both eager and bounded apply accumulators and cursors through +EvalFrame, refreshing the off-heap slice after each append. Native sequence +boundary tests force GC inside lazy callbacks and the invoked function. | # | Site | Held accumulator | Reentrant call | Severity | |----|------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------|------------------------------------------------------------------------------------------------| -| C1 | `higher_order.zig` `applyFn` | `collected: ArrayList(Value)` on `rt.gpa` (off-heap, off-stack) | `seqFn`/`firstFn`/`nextFn` per spread element | HIGH (apply-over-lazy; N-wide off-heap list) | +| C1 | **DISCHARGED 2026-09-06** `higher_order.zig` `applyFn` / `applyVariadic` | `collected: ArrayList(Value)` on `rt.gpa` (off-heap, off-stack) | `seqFn`/`firstFn`/`nextFn` per spread element | HIGH (apply-over-lazy; N-wide off-heap list) | | C2 | `higher_order.zig` `everyQFn` | `cur` seq cursor | `seqFn`/`firstFn`/`invokeCallable`/`nextFn` | med | | C3 | `higher_order.zig` `someFn` | `cur`, `r` | same as C2 | med | | C4 | `collection/sorted.zig` `insert` (recursion) | half-built RB-subtrees | `compareKeys → vt.callFn` (custom `sorted-map-by` comparator only) | HIGH on `-by` path | @@ -152,6 +170,17 @@ Fixed by bracketing the alloc in the ADR-0150 fabrication no-collect region `range.seqChunk`). e2e guards: `phase16_gc_torture` `rest_range` / `rest_rest_range` / `lazywalk_range` / `for_range` + `phase16_bfs_queue_gc`. +Host-call result builders, the same alloc-boundary window (2026-09-25): a host +fn that assocs several entries into a Zig-local map swept the half-built map +at the second alloc. Found as `cljw.http.client/get` answering `{:body "ok"}` +with `:status` gone. Now bracketed in fabrication regions: +`http/client.zig` (response), `http/server.zig` `requestValue` (request map, +built after `collectRequest` has read the body into arena bytes, so no I/O +runs inside the region), `wasm/component.zig` `componentExportsFn` and the +top-level `lift`, alongside the existing `process/run.zig` `resultValue` and +`wasm/surface.zig`. e2e guards: `phase16_gc_torture` +`alloc/http_client_response` / `alloc/http_server_request`. + ## D. Permanent / pinned roots | # | Mechanism | Roots | diff --git a/.dev/handover.md b/.dev/handover.md deleted file mode 100644 index 562042169..000000000 --- a/.dev/handover.md +++ /dev/null @@ -1,100 +0,0 @@ -# Session handover - -> ≤ 100 lines. Driving doc; framing per -> [`.claude/rules/handover_framing.md`](../.claude/rules/handover_framing.md). - -## Resume contract - -- **HEAD**: `main` (`git log` = SSOT). Per-commit = smoke locally; - **CI runs ONE configuration everywhere** — PR, push and dispatch all run the - identical `test/run_all.sh --serial-e2e` that `scripts/run_gate.sh` runs, so - "green" means the same run wherever it is said. The nightly schedule and its - tree_walk sweep are RETIRED (2026-08-12); `check_gate_parity.sh` fails if a - tier comes back. Commit **and** push (atomic - Step 6). `build.zig.zon` `.zwasm` = tag pin **v2.5.0** - (`278587f6`, 2026-08-12, FINAL — full WASI 0.3 + the `libzwasm.a` C-API - export fix; neither reaches cljw's Zig embedding surface). **zwasm now lives - at `github.com/zwasm/zwasm`** — its own org since 2026-08-12, separate - maintainership; the dep URL names it directly rather than riding the transfer - redirect, and the content `.hash` was unchanged by the move. Co-development - (the `dogfooding_handover` mailbox, `.dev/zwasm_capabilities.md`'s per-unit - refresh duty) is RETIRED — see F-001's 2026-08-12 entry. Latest release: - **v1.10.1** (2026-08-12) — **the FINAL release**. CHANGELOG is the - release-history SSOT. -- **First task on resume**: nothing is pending — the wind-down is COMPLETE. - README declares the project unmaintained; v1.10.1 is released (4 assets, - verified by download + checksum + running the macOS binary incl. the Wasm FFI - → 42); the homebrew tap is bumped (`brew fetch` resolves 1.10.1 after `brew - update`); the announcement is Discussion #15, edited to v1.10.1 with a comment - recording what changed (the user pins it — the API has no pinDiscussion). The - PERF CAMPAIGN (D-450) is **stopped, not paused** — findings stay in the row. -- **The demo shutdown is COMPLETE (2026-08-12)**, so the docs' and the - announcement's claims are now true: cw-playground / cw-serverless-demo / - cw-arcade archived (public, forkable), both Fly apps **destroyed** — - irreversible, and the bookshelf's 1 GB `bookshelf_data` volume went with them. - `fly apps list` is empty; both URLs refuse connections. Stopping the machines - would NOT have sufficed — both configs carry `auto_start_machines = true`, so - any request would have woken them. -- **Unreleased on main**: nothing. CHANGELOG `[1.10.1]` is the SSOT. -- **Release-process hazard** (`.github/workflows/release.yml`): cutting the - GitHub release BY HAND before the tag workflow runs breaks the artifact - upload (zwasm #160). The `gh release view || gh release create` TOCTOU across - the matrix legs is **FIXED** as of v1.10.1 — a failed create is accepted iff - the release exists afterwards; all four paths were verified against real exit - codes. Both workflows also clear the Zig unpack target before `tar -x`, so a - re-run cannot die on "Cannot open: File exists". -- **The v1.10.0 tag names `clojurewasm/zwasm`; v1.10.1 does not.** Anything - building from the older tag (both demo Dockerfiles pin `CLJW_REF=v1.10.0`) - rides GitHub's transfer redirect — verified working 2026-08-12. It holds only - while nothing claims the vacated name: **never create a repo called `zwasm` - under the `clojurewasm` org.** -- **Forbidden**: bare `zig build test` without `-Dwasm`; bare `zig build` for a - probe (use ReleaseSafe). The FULL gate runs `--serial-e2e`, ALONE; never a - concurrent build during a gate. `.claude/**` edits + cross-repo publishes may - hit the auto-mode block — surface to the user. **D-549 (Docker/ghcr/ - notarization) is user-LOCKED**, **D-560 trigger-gated** — neither - self-selects. External-publish payloads need `test -s` + read-back guards. - -## Current state (details = CHANGELOG + git log) - -- **Issues are DISABLED and always were** (deliberate — the maintainer cannot - service them). **PRs are NOT disabled and cannot be**: GitHub has no such - setting for an unarchived repo, so anyone can open one; none ever has, and - none will be reviewed. Say it that way — "PRs are disabled" was the wording - used until 2026-08-12 and it was false. **Discussions is the one channel**, - and outside contributors send branches to cherry-pick; such a commit is taken - as-is with authorship preserved. Debug tooling: `scripts/check_*.sh` headers - are each their own SSOT — read the script, not a list here. -- **The repo stays PUBLIC and UNARCHIVED** (user, 2026-08-12: support ends - gradually, the repo stays open, Issues/PRs stay unaccepted). Do not archive - ClojureWasm. The three demo repos (cw-playground / cw-serverless-demo / - cw-arcade) ARE archived as of 2026-08-12, and stay public. -- **2026-08-12: the README declares ClojureWasm no longer maintained** - (`1c8a9cbf`); the Sponsors badge, README footer line, and `.github/FUNDING.yml` - are gone with it. Wind-down context + the surveyed wording/transfer research: - memory `clojurewasm-wind-down-plan`, `private/notes/2026-08-11-{sunset-wording, - transfer-risk}-survey.md`. -- 2026-08-11/12: Discussions #12/#13/#14 (BuddhiLW) answered in code — three - fix branches cherry-picked with authorship preserved, each one's CLASS then - swept and fenced. #12 repr-misread → 3 more live bugs (multimethod isa? at 9 - defmethods, http client/server 9-entry headers, error_render context) + - `check_repr_decode.sh`; #13/#14 entry-point divergence → `cljw repl`'s - `[-cp][-A]` surface, `cljw build`'s missing top-level-`do` unroll, - `check_entrypoint_surface.sh` + `test/e2e/entrypoint_eval_parity.sh` (a - 6-entry-point differential oracle). D-374's discharge was a listed-not-probed - claim; corrected in place. Endgame row = D-576. - -## What was left unfinished (`.dev/debt.yaml` is the SSOT) - -The ledger is the honest list for a fork; nothing in it is scheduled. The -notable open ends: the perf campaign (D-520 / D-386 / D-513) stopped with its -baseline recorded; D-565 residuals (7)(8); D-548(b) pmap timing envelope; and -the north star — Wasm interop × VM-perf fusion→JIT — reached JIT-by-default but -not components-through-the-JIT, which was always upstream's call (D-500) and is -now another project's entirely. - -## Reading order (resume) - -handover → `yq` the live `active:` list → ADR-0166 → ROADMAP §9.0. Memories: -`verify_against_releasesafe_binary` / `smoke_first_batch_full_gate` / -`external-publish-payload-guard`. diff --git a/.dev/lessons/INDEX.md b/.dev/lessons/INDEX.md deleted file mode 100644 index 1fd294b60..000000000 --- a/.dev/lessons/INDEX.md +++ /dev/null @@ -1,59 +0,0 @@ -# Lessons index - -> Lessons are re-derivable observations distinct from load-bearing -> ADRs. Promote a lesson to ADR when any of the following holds: -> - Same observation cited 3+ times -> - One path adopted, alternatives explicitly rejected -> - ROADMAP / Phase / scope decision rests on it - -## Categories (populated as lessons emerge) - -### Cluster A: Implementation patterns - -- [`phase_deferred_scaffolds.md`](phase_deferred_scaffolds.md) — - Phase-deferred scaffolds lose their homing path; complementary - detection layers (check_test_reach / lazy decl ride-along / - debt review); the aspirational-rule sibling (cross-cluster, see - also Cluster E). -- [`io_isolation_findings.md`](io_isolation_findings.md) — ADR-0126/ - D-355/D-361: per-eval `with-budget` vs the process-wide `CLJW_EVAL_*` - budget (the long-running-server footgun); the heap ceiling is - gc.alloc-only so infra bulk allocs need `checkInfraCap` (D-361); - Zig 0.16 cwd path = `std.process.currentPathAlloc` (not getCwd); - a background, independent Linux gate as look-ahead surfaces - cross-platform bugs cheaply (cross-ref Cluster E). - -### Cluster B: Clojure semantics - -(empty — see `.claude/rules/clojure_spec_citation.md` for the discipline) - -### Cluster C: File size and refactor discipline - -(empty — Phase 5+ expected when collections split happens) - -### Cluster D: Process and documentation - -(empty) - -### Cluster E: Debugging and tooling - -- [`structural_defect_hunting.md`](structural_defect_hunting.md) — - corpus-driven large-input/edge probing surfaces STRUCTURAL defects - (eager non-TCO recursion class; designed-but-unconnected scaffolding; - missing eval-time reachability; representation divergence; hidden - O(n²)) that gap-filling misses. The hunting method + the patterns found - 2026-05-30 + the known structural work-queue (D-160/D-161/D-162). Fix - per F-002 (finished form), not ad-hoc. -- See cross-cluster reference: `phase_deferred_scaffolds.md` - (Cluster A) covers the test-orphan + compile-error-orphan - diagnostic + the `check_test_reach.sh` gate that catches them. - -## Lesson file format - -Each lesson is `.dev/lessons/.md`: -- 100-300 lines -- Title + Date + Cluster -- Observation (3-5 sentences) -- Concrete event (commit SHA, test case if available) -- Lesson (what to do next time) -- Related ADRs (link by ID, not path) diff --git a/.dev/lessons/io_isolation_findings.md b/.dev/lessons/io_isolation_findings.md deleted file mode 100644 index be085c26b..000000000 --- a/.dev/lessons/io_isolation_findings.md +++ /dev/null @@ -1,62 +0,0 @@ -# Lessons — io subsystem + in-process isolation (ADR-0126 / D-355 / D-361) - -Observational learnings from the clojure.java.io subsystem + the babashka-free -playground port. Re-derivable; not load-bearing ADRs (those are ADR-0125/0126). - -## 1. Per-eval budget ≠ process-wide budget (the embedder footgun) - -`CLJW_EVAL_MAX_STEPS` / `_DEADLINE_MS` / `_MAX_HEAP_MB` arm the **process-wide** -eval budget at startup (`eval_budget.installFromEnv`, runner.zig). For a -long-running embedder (an HTTP server that evaluates user code per request), -setting these as process env is a footgun: the **deadline trips a few seconds -into server uptime** and every back-edge afterwards (serving static files, -dispatching handlers) raises `resource_exhausted` → the server starts 500-ing -for no apparent reason. - -The correct shape: the server runs **unmetered**; only the per-eval call is -bounded, via `cljw.eval/with-budget` (which saves+restores the ambient budget -over its dynamic extent). The playground learned this the hard way — it now -reads playground-specific `PG_EVAL_*` env into the with-budget opts and never -sets `CLJW_EVAL_*` process-wide. (commit: playground-v2 "fully drop babashka".) - -**Takeaway for embedder docs:** `CLJW_EVAL_*` = whole-process kill-switch (good -for a one-shot `cljw -e`); `with-budget` = per-eval recoverable bound (good for a -server). Don't conflate them with one env name. - -## 2. The live-heap ceiling is gc.alloc-only — infra bulk allocs bypass it (D-361) - -`GcHeap.heap_ceiling` is checked inside `GcHeap.alloc` (GC-managed records). A -bulk allocation that goes through `gc.infra` directly bypasses it — notably the -**transient vector's element buffer** (`transient_vector.zig` grows via -`infra.realloc`). `(vec (range 1e8))` under a 16 MB cap grew that buffer to -~800 MB: a roomy host (macOS overcommit) tolerated it and tripped the cap later -at `persistent!`, but a memory-tight host (CI Linux) **OS-OOM-killed the process -first** — same code, opposite outcome, empty output + non-zero exit. - -Fix: `GcHeap.checkInfraCap(bytes)` called at the bulk-infra site before the -realloc. **Lesson:** "the cap is at the alloc boundary" has a hole — `infra` -bulk paths need an explicit `checkInfraCap`. When adding a new per-N infra -buffer, wire the cap check. - -## 3. Zig 0.16 cwd path: `std.process.currentPathAlloc(io, alloc)` (NOT getCwd) - -0.16 removed `std.fs.cwd`-path / `realpath` / `posix.getcwd` (ziglang/zig#19353: -"realpath is unportable, a bug magnet"). The live accessor in the new io model -is **`std.process.currentPath(io, buf)` / `currentPathAlloc(io, alloc)`** (takes -`io` = `rt.io`). A grep for `getCwd` finds nothing and misleads you into -thinking the cwd path is unavailable (it cost D-357 a wrong "deferred" premise). -`std.os.linux.getcwd(buf, size)` is the raw syscall fallback. `std.Uri` (parse) -+ `std.http.Client` + `std.Io.net` are all present, so URL/URI/Socket host types -are buildable now (the deferral is phase-ordering, not a hard block). - -## 4. A look-ahead, independent Linux gate surfaces cross-platform bugs cheaply - -The ubuntunote gate runs on a **separate machine**, so launching it in the -background against a just-pushed HEAD costs ~nothing locally and runs in parallel -with continued local work. This is how D-361 (a macOS-passes / Linux-fails -heap-cap bug, latent for a whole session) was caught — the Mac per-commit gate -never would have. **Takeaway:** at a milestone, fire `run_remote_ubuntu.sh` in -the background as look-ahead; don't treat the Linux gate as a serial -phase-boundary-only step. (The empty-output + non-zero exit signature also -instantly ruled out the "error renderer re-breaches" hypothesis — a process -killed before printing anything is OS/timeout, not a render-path issue.) diff --git a/.dev/lessons/phase_deferred_scaffolds.md b/.dev/lessons/phase_deferred_scaffolds.md deleted file mode 100644 index 6d8237d21..000000000 --- a/.dev/lessons/phase_deferred_scaffolds.md +++ /dev/null @@ -1,89 +0,0 @@ -# Lesson — Phase-deferred scaffolds lose their homing path - -- **Date**: 2026-05-25 (surfaced via zwasm v2 reciprocal observation - at `private/notes/zwasm_v2_reciprocal_observations.md`) -- **Cluster**: A (Implementation patterns) + E (Debugging and - tooling) -- **Promoted to lesson** at Wave 16 W16-7 (2026-05-26); previously - lived inline in `.claude/rules/zig_tips.md` L313-358. - -## Observation - -A file landed for a future phase, expecting some upstream wiring, -but the wiring never arrives. The file sits in the source tree -without any production-path caller. Two symptoms emerge: - -1. **Test-orphan**: the file has `test {...}` blocks that never - run, because Zig's lazy declaration analysis only pulls a file - into the test graph when something else references it. -2. **Compile-error-orphan**: the file references stale stdlib APIs - that never get checked, because the file is never compiled. - -Both root in: "we landed code expecting future wiring; future -wiring never came; code sits without a caller." - -## Concrete events - -- 2026-05-25 surgery: `runtime/regex/{compile, match}.zig` - skeletons shipped with 13 unit tests that never ran. Discovered - during the cycle-1 commit; fix landed via `src/main.zig`'s - `test {}` aggregator block + the new - `scripts/check_test_reach.sh` gate. -- D-053: `runtime/clock.zig` + `runtime/time/instant.zig` - referenced `std.time.nanoTimestamp`, which was removed in Zig - 0.16. The orphan-status meant no one type-checked these files - for multiple commits. -- 2026-05-26 review-fix on commit ef4f683: silent-failure-hunter - surfaced 5 missed retrofit sites in the Wave-15 provisional- - marker spike (same shape — code landed expecting later wiring - + the discovery criterion was implicit). Wave 16 W16-1 - introduced `framework_completion.md` to codify the discovery- - criterion + sweep + retrofit triad. - -## Lesson - -When a Phase-deferred scaffold lands, the author should: - -1. Put a `_ = @import("path/to/scaffold.zig");` line in - `src/main.zig`'s `test {}` aggregator block in the same - commit, so the file is in the test graph from day one - (covers both orphan symptoms above). -2. If the scaffold is genuinely too disruptive to compile until - the deferred phase lands (rare), file an explicit - `D-NNN blocked-by: Phase N landing` row in `.dev/debt.yaml` - so the deferral has a tracked owner — not a silent orphan. - -## Complementary detection layers cw v1 carries - -- **Test-presence orphan check** (`scripts/check_test_reach.sh`) - — high signal, allow-list-free, hard-fail gate. -- **Lazy decl analysis ride-along** — once a file enters the - test graph, latent compile errors against removed stdlib APIs - surface immediately. The check_test_reach gate is therefore - also a covert "is the code still buildable" gate for files - with no production-path caller. -- **Periodic debt review** — `Phase N target` / - `blocked-by: ` rows in `.dev/debt.yaml` that have been - blocked for ≥ 3 cycles get escalated at Step 0.5 sweep. - -## The aspirational-rule sibling - -A related shape: a `.claude/rules/.md` that declares an -enforcement (e.g., "Phase 5+ becomes a hard gate") without a -matching script — the rule is aspirational but no mechanical layer -enforces it. zwasm v2's `audit_table_sync.sh` is the canonical -"aspirational rule lifted to a mechanical gate" pattern; Wave 16 -W16-3 did the same for `handover_framing.md` forbidden phrases. - -## Related ADRs - -- ADR-0021 (test taxonomy) — the 5-layer test framework that - `check_test_reach.sh` operationalises. -- ADR-0024 (scan framework + run_step) — the gate-runner shape - this lesson's detection layers fit into. - -## Related Wave-16 follow-up - -The Wave-16 framework_completion.md rule + watch_findings.md SSOT -+ Bad Smell "Framework-incomplete" / "Defer-to-amnesia" entries -formalise the discipline this lesson surfaced. diff --git a/.dev/lessons/structural_defect_hunting.md b/.dev/lessons/structural_defect_hunting.md deleted file mode 100644 index ff300e8ac..000000000 --- a/.dev/lessons/structural_defect_hunting.md +++ /dev/null @@ -1,133 +0,0 @@ -# Lesson: structural-defect hunting (corpus-driven), fix per finished-form - -- **Date**: 2026-05-30 -- **Cluster**: E (Debugging and tooling) + cross-ref Cluster A - -## Observation - -The corpus-driven sweep (running common `clojure.core` ops on **large / -edge inputs** via `cljw -e`, not AI-guessed unit probes) surfaces a class -of bugs that self-probing misses: not "function X is absent" (an ad-hoc -gap), but **structural defects** — a design that is wired wrong, a -subsystem scaffolded-but-unconnected, a representation that diverges, or a -hidden super-linear cost. These are the high-value finds. When one is -structural, the response is **F-002**: fix the finished form (loop/recur, -proper wiring, rt plumbing), do the rework, do **not** ad-hoc patch the -symptom. (User directive 2026-05-30: "ad-hoc ではなく構造的欠陥を見つけ、 -完成形がきれいの原則に従い手戻りを厭わず取り組む".) - -This lesson is the resume contract for that mode — a fresh `continue` -session should treat structural-defect hunting as the operating mode, not -gap-filling. - -## The structural-defect PATTERNS found this session (the reusable catalog) - -1. **Eager non-TCO recursion class** (SEVERE — segfault). `(range 100000)`, - `(sort (range 5000))`, `(interleave …)`, `(zipmap …)` all segfaulted: - a `.clj` fn recursing fn-deep, or non-tail as `(into [x] (SELF (rest …)))`, - over collection size → stack overflow. **Fix**: `loop`/`recur` - (constant stack). **Detection**: large-input probe; grep - `(into (conj` + a self-call not wrapped in `lazy-seq`. Safe siblings: - lazy-seq-wrapped (map/filter/take-while/…) and log-depth (msort). - Commits 1f6ca79e / fcd712fb / 485e9142 / d4acdb14. - -2. **Designed-but-unconnected scaffolding**. The multimethod has a full - isa?-walk in `getMethod` (`dominates`/prefer-table/cache-snapshot), but - `makeMultiFn` sets `hierarchy_ref = nil`, `derefHierarchy` is a no-op, - and `isaCheck` receives the ref (not `:ancestors`) — so `defmulti` - never consults the hierarchy (`derive` doesn't affect dispatch). The - feature "works" in isolation; its **integration path is a silent - skeleton**. **Detection**: a feature works standalone but its - cross-subsystem wire is a no-op/nil-default. = **D-161**. - -3. **Missing eval-time reachability**. `eval` is blocked because - `analyzeForm` needs the `macro_table`, which is a setup-time param NOT - stored on `rt`/`env` — a runtime primitive can't reach it. The - defect is architectural (a needed resource isn't threaded to the - eval-time context), not a missing fn. **Detection**: a runtime - primitive needs a resource that only exists at setup/analysis time. - = **D-162**. - -4. **Representation divergence**. cljw UUIDs are plain strings (no `.uuid` - tag), so `uuid?` has nothing to check — the predicate is ambiguous - because the representation collapsed the type. **Detection**: a - `type?`-style predicate has no distinct tag/representation to test. - -5. **Hidden O(n²)**. `dedupe` (`(last acc)` per step) and `distinct` - (linear `some` per element) coll arities timed out at ~5000 — correct - but unusable at scale. **Fix**: delegate to the O(n) transducer. - **Detection**: large-input timeout (not a crash). Commit d4acdb14. - -## The HUNTING METHOD (how to keep finding these) - -0. **Use the real Clojure (`clj`) differential oracle** (F-011 + - `.dev/reference_clones.md`). Instead of guessing the expected output, - run the same form through `clj -M -e ''` and diff against - `cljw -e ''` — **including error cases** (the message format - differs, but the exception CLASS and the value must match; extract - the class with `grep -oE '\(([A-Za-z]+Exception|[A-Za-z]+Error)\)'`). - `clj` startup is ~1–2 s, so batch probes. A mismatch that is a - recorded ADR divergence (e.g. `(class 5)` → `Long` not - `java.lang.Long`) is not a defect; everything else is a candidate. - Use the harness `scripts/clj_diff_sweep.sh` (rule - `.claude/rules/clj_diff_sweep.md`); the tracked sweep state is - `test/diff/clj_corpus/COVERAGE.md`. Gitignored scratch log (optional): - `private/notes/phaseA26-clj-differential-oracle.md`. -1. Run **systematic large-input + edge `cljw -e` probes** over a category - (every coll-fn, then string/regex/arith, then nil-punning/destructuring/ - threading, …). Tag each result OK / crash / timeout / name_error / - **diverges-from-clj**. -2. A **crash or timeout** is almost always structural (recursion class, - O(n²)) — fix the finished form (loop/recur, transducer delegation). -3. A **wrong-but-no-crash** result or a **feature that works standalone - but no-ops in integration** is the subtler structural class (#2/#4) — - trace the wiring; if a scaffold is unconnected, wire it per finished - form (may need a Step-0 survey of the subsystem first). -4. A **name_error** is usually just an ad-hoc missing fn — fill it if - clean (`.clj` composition / reuse), else record as a debt with the - architectural blocker named (D-162-style). -5. When the fix requires **rework / rt plumbing / a subsystem rewrite**, - that is the expected case — F-002 says cycle size is not a constraint. - Do NOT defer on diff-size grounds (Cycle-budget defer smell). Defer - ONLY when it needs a focused Step-0 survey to avoid breaking intricate - scaffolding (e.g. D-161 touches the dispatch path) — record a precise - debt row with the wiring plan so the focused session is ~30 min. - -## Known structural-defect work-queue (for the resuming session) - -- **D-162** — `eval`: store `macro_table` on `rt` at `setupCore`, then - `eval` = valueToForm→analyzeForm(rt.macro_table)→evalForm. Architectural. -- **D-161** — `defmulti` dispatch: wire `hierarchy_ref` to the global - `-global-hierarchy` atom; `derefHierarchy` deref it; `getMethod` pass - `:ancestors`. Layer-0 dispatch path; survey the prefer/dominates/cache - scaffolding first. -- **D-160** — `sequence`/`eduction`: need a push→pull transducer bridge - (TransformerIterator-equivalent); cljw transducers are reduce-driven. -- **Representation**: UUID-as-string (no tag) — decide a UUID - representation before `uuid?`; same question latent for `type`/`class` - (no JVM Class → what does cljw return?). -- Continue the probe sweep on unswept surface (interop, dynamic vars, - IO, deftype/defrecord field access, protocol edge) for more of the above. - -## Lesson (what to do next time) - -The gap-map's "fill missing fns" is the floor; the **ceiling is structural -defects**, and they hide behind functions that *appear* to work. Hunt them -with large-input/edge probes + wiring audits, and fix the finished form -even when it means undoing earlier shape. A crash/timeout on common input -is never acceptable to ship; an unconnected integration scaffold is a lie -(silent no-op) by the `permanent_noop_forbidden` standard. - -## Related - -- `.dev/principle.md` — Bad Smell catalogue (Smallest-diff bias, - Cycle-budget defer, Silent default-shift) + four depths of revision + - Structural imagination phase. This lesson is the *applied* form. -- F-002 (`project_facts.md`) — finished-form wins; cycle size not a - constraint. -- `.dev/debt.yaml` D-160 / D-161 / D-162 — the named structural queue. -- `.dev/core_coverage_gaps.md` — CRASH FIXES + sweep sections (the - per-defect detail). -- `phase_deferred_scaffolds.md` (Cluster A) — the sibling "scaffold loses - its homing path" pattern (test-orphan / compile-error-orphan); #2 above - (unconnected integration) is its dispatch-path cousin. diff --git a/.dev/library_incorporation_playbook.md b/.dev/library_incorporation_playbook.md deleted file mode 100644 index 1dcdbbd73..000000000 --- a/.dev/library_incorporation_playbook.md +++ /dev/null @@ -1,144 +0,0 @@ -# Library-incorporation playbook — how cljw absorbs real-world Clojure libs - -> A standing reference for the `test/conformance/verified_projects/` coverage engine (F-010 / -> F-013, convergence_campaign Stage 1.3). Written 2026-06-07 after landing 11 -> libraries (medley → … → hiccup → honeysql). The campaign is on **STAY** (user -> directive 2026-06-07); this doc exists so a future re-expansion resumes with -> the accumulated know-how instead of re-deriving it. -> -> SSOT for the *method* lives in `test/conformance/verified_projects/README.md`; this doc is the -> SSOT for the *patterns, taxonomy, and coverage strategy* behind it. - -## 1. The core mental model - -**One real library is an N-blocker chain, not a single feature.** Every lib -landed so far revealed *more* blockers than the handover predicted: - -- **hiccup**: handover said "java.net.URI only" → actually **7 blockers** - (URI/URLEncoder/StringBuilder/Iterator surfaces, Object-extension dispatch - fallback, extend-protocol native-tag distribution, syntax-quote alias + `%N` - fix, exception_descriptor leak). ADR-0114. -- **honeysql**: survey said "Locale + lookahead" (2) → actually **5 blockers** - (Locale static + String 2-arg + regex lookahead + IPersistentMap extend-target - + `.sym` keyword method). ADR-0115. - -The corollary: **a probe that fails is the START of the work, not the end.** The -first `name_error` / `not_implemented` / `type_error` is blocker #1 of many; fix -it, re-probe, get blocker #2, repeat until green. Budget for the chain, not the -first error. - -## 2. The probe loop (the engine) - -``` -1. mkdir test/conformance/verified_projects/; write deps.edn (:git/url + :git/sha) + verify.clj - (require the lib's CORE ns + assert REAL outputs, not just `require`). -2. bash scripts/verify_projects.sh → read the FIRST blocker (file:line + Kind). -3. Classify the blocker (§4 taxonomy) → find its fix site (§4 "fixed in"). -4. Fix the ROOT CAUSE (F-013: definition-derived, never a per-lib patch). Probe a - minimal repro with `cljw -e` to confirm the fix in isolation. -5. Re-probe (step 2). New blocker → repeat. Green → run the FULL sweep - (scripts/verify_projects.sh, no filter) to confirm no regression, then the gate, then commit. -``` - -**Why `verify.clj` must assert real outputs**: a bare `(require …)` passes when a -namespace merely *loads*; the functional bar (`(sql/format {…})` → -`["SELECT …" …]`) catches the runtime gaps (`.sym` undefined, Object-fallback -missing) that load-only misses. Every blocker after the analysis-time ones is a -runtime dispatch/interop gap that only a real exercise surfaces. - -## 3. F-013 discipline (the thing that makes coverage compound) - -**Fix the root cause, neutrally — never special-case the lib.** Every fix landed -so far is a *general capability* that helps the next lib too: - -- hiccup's Object-extension fallback → every lib extending a protocol to Object. -- honeysql's regex lookahead → every lib using `(?=…)`. -- hiccup's syntax-quote alias resolution → every macro whose template calls an - aliased ns (extremely common). - -This is why the per-lib cost *decreases* over time: the 11th lib reused -mechanisms the 1st–10th built. A per-lib patch (a special case keyed on the lib's -name / a stub that fakes one call) would break this compounding and is forbidden -(the no-op-stub / permanent-no-op rule). - -## 4. Gap taxonomy — where each class of blocker lives (the high-value map) - -When a probe fails, classify the blocker and go straight to its fix site: - -| Blocker shape (what the error looks like) | Class | Fixed in | Examples (ADR) | -|-----------------------------------------------------------------------------|------------------------------|--------------------------------------------------------------------------------------------------------|-------------------------------------| -| `No namespace: 'java.x.Y'` on `(Y. …)` / `(Y/method …)` | **Host class surface** | new `runtime/java//.zig` + `_host_api.zig` `java_surfaces` + compat_tiers row | URI/URLEncoder/StringBuilder (0114) | -| stateful host object (mutable / holds state) | **host_instance** | `host_instance.alloc`; heap state → `host_finalise` hook; live Value in state → `host_trace` (D-318) | StringBuilder/Iterator (0114) | -| `No namespace: 'Class'` on `Class/STATIC` (scalar) | **Static field (scalar)** | descriptor `static_fields` (int/float/bool) | Integer/MAX_VALUE, Math/PI | -| `No namespace: 'Class'` on `Class/STATIC` (object value) | **Static field (object)** | `Singleton` enum (type_descriptor.zig) + analyzer arm + gc.infra singleton | Locale/US (0115, ADR-0087) | -| `No namespace: 'Class'` on `Class/staticMethod` | **Static method surface** | `___HOST_EXTENSION` static descriptor (fqcn `cljw.java..` for bare resolve) | String/valueOf (0114) | -| `No implementation of method 'm' on protocol '<.member>' for type 'T'` | **Native instance method** | `runtime/_methods.zig` installNativeMethods on `nativeDescriptor(.)` | .sym keyword (0115), .name ns | -| `.method: index/arity` — wrong arity on an existing method | **Method overload** | arity-range check in the method fn (ignore unused args if semantically irrelevant) | toUpperCase(Locale) (0115) | -| `Unable to resolve symbol: 'Interface'` as an `extend-protocol` TARGET | **extend-protocol target** | host_interface: `NATIVE_EXTEND_TARGETS` (native tags) / host_inert no-op / host-surface-class value | IPersistentVector/Map (0114/0115) | -| `No implementation … for type 'String'` when a protocol has an Object impl | **Object dispatch fallback** | dispatch.zig (Object-extension universal fallback) | hiccup HtmlRenderer (0114, D-319) | -| macro template references an aliased ns / `#()` param errors | **syntax-quote hygiene** | `qualifySym` (alias resolution, `%N` bare) | hiccup html→hiccup2 (0114) | -| `regex literal (unsupported syntax …)` | **regex engine feature** | `runtime/regex/{compile,match}.zig` | lookahead `(?=…)` (0115) | -| `No namespace: 'clojure.lang.X'` (call/value position, JVM-internal) | **deferred host ref** | ADR-0113 (defers to a CALL-time feature_not_supported; the ns LOADS) | integrant RT/baseLoader (0113) | -| a memory leak / crash surfaced by the above | **latent bug** | wherever — these are real bugs the new exercise first triggers; fix + regression-test | exception_descriptor leak (0114) | - -This table IS the methodology compressed: most future blockers fall into one of -these ~12 classes, and each has a known fix site. A genuinely novel class is rare -and warrants an ADR. - -## 5. Raising the coverage rate effectively - -- **Batch by shared blocker class, big-bang the mechanism.** Libs cluster on the - same gaps (many use `(extend-protocol P Object)`; many use aliased-ns macro - templates). Landing the *general* mechanism once clears a whole cluster. Pick - the next lib partly by which mechanism it would unlock for others. -- **Anti-drip-feed (D-315 lesson).** When a lib reveals blockers one-at-a-time, - do NOT land them one-per-cycle forever — the Micro-coverage-grind smell. honeysql - was parked precisely because Locale-alone landed no verified lib (the next - blocker, regex lookahead, was still there). Land the *whole* chain to green in - one focused push, or park the lib until you can. A half-fixed lib that still - fails verify is worse than an unstarted one (the ladder reads "covered"). -- **Probe-derived, not guess-derived.** The chain is discovered empirically by - re-probing, not by reading the lib's source up front and guessing. The probe is - cheaper and never wrong about what actually blocks. -- **Reuse `test/conformance/verified_projects/` as a regression net.** Each landed lib stays - as a committed proof; `scripts/verify_projects.sh` re-runs all of them, so a - later change that breaks an earlier lib is caught (run the full sweep before - every honeysql/hiccup-class commit). - -## 6. Discipline reminders (so the coverage stays honest) - -- **Divergence → AD-NNN** (`accepted_divergences.yaml`), never a silent gap - (java.util.Map extend-target inert = AD-023). BUT a divergence only qualifies - when its justification is a *project invariant*, not convenience — a silent - semantic drop dressed as an AD is forbidden (`no_op_stub_forbidden`). The - ADR-0115 DA fork caught exactly this: a "capture-free lookahead" AD was rejected - because discarding captures is a silent drop, so the finished form (full capture - parity) shipped instead. *Eliminating the loss beats recording it.* -- **Finished-form deferral → debt row** with the clean shape recorded + scheduled - at its owning point (D-317/318/319 for ADR-0114; D-320 lookahead perf for - ADR-0115). F-002: the clean shape is captured, not lost. -- **ADR-level decision → inline ADR + mandatory DA fork** (depth ≥ 2). The DA - fork challenges the design from fresh context; its output goes verbatim into - "Alternatives considered" (see ADR-0114/0115). -- **Never trust a "covered" claim without a corpus/verify proof** (anti-D-177): - the `test/conformance/verified_projects/` dir + its assertions ARE the proof. - -## 7. Where to look next (when re-expanding) - -- `docs/works/ladder.md` — the ranked candidate ladder + the first-blocking gap - per lib (the NEEDS-ROW list). The next libs + their predicted blockers live there. -- Parked libs (deeper blockers, handover): schema (`clojure.lang.Compiler/CHAR_MAP` - value-position), clip (`clojure.lang.Reflector`), data.avl (`clojure.lang.RT`/ - APersistentMap), data.xml (StAX), instaparse (java.io), data.json (PrintWriter/ - PushbackReader). Each names its blocker class (§4) — start from the fix site. -- The deferred-host-ref class (ADR-0113) is the biggest remaining lever: many libs - park on a single `clojure.lang.*` peripheral ref; the per-interface NATIVE- - implementer table (D-308) would unlock a cluster. - -## Cross-references - -- `test/conformance/verified_projects/README.md` — the mechanical how-to-add (the method). -- `.dev/convergence_campaign.md` Stage 1.3 — the campaign driver this doc serves. -- `docs/works/ladder.md` — the candidate ladder + per-lib first blocker. -- ADR-0113 (deferred host refs) / ADR-0114 (hiccup) / ADR-0115 (honeysql) — the - worked examples this playbook generalises. diff --git a/.dev/mutation/mutants-39677a2f-seed0-unit-golden.jsonl b/.dev/mutation/mutants-39677a2f-seed0-unit-golden.jsonl new file mode 100644 index 000000000..1093fbaf2 --- /dev/null +++ b/.dev/mutation/mutants-39677a2f-seed0-unit-golden.jsonl @@ -0,0 +1 @@ +{"id": "d317ae6644b1", "file": "src/runtime/print.zig", "line": 1480, "col": 42, "op": "branch_force_true", "before": "if (!first.*) try w.writeAll(if (kv) \", \" else \" \");", "after": "if (!first.*) try w.writeAll(if (true) \", \" else \" \");", "verdict": "killed"} diff --git a/.dev/mutation/mutants-ac3a7ff0-seed1-unit-golden.jsonl b/.dev/mutation/mutants-ac3a7ff0-seed1-unit-golden.jsonl new file mode 100644 index 000000000..5736a7761 --- /dev/null +++ b/.dev/mutation/mutants-ac3a7ff0-seed1-unit-golden.jsonl @@ -0,0 +1,12 @@ +{"id": "58155ae4f898", "file": "src/runtime/print.zig", "line": 293, "col": 69, "op": "cmp_gt_to_ge", "before": "while (ks.tag() == .list and list_collection.countOf(ks) > 0) : (ks = list_collection.rest(ks)) {", "after": "while (ks.tag() == .list and list_collection.countOf(ks) >= 0) : (ks = list_collection.rest(ks)) {", "verdict": "survived"} +{"id": "9141cbd613b4", "file": "src/runtime/print.zig", "line": 386, "col": 32, "op": "cmp_gt_to_ge", "before": "if (items.items.len > lim) break;", "after": "if (items.items.len >= lim) break;", "verdict": "survived"} +{"id": "b70c218a714b", "file": "src/runtime/print.zig", "line": 441, "col": 23, "op": "cmp_eq_to_ne", "before": "while (t.tag() == .list and list_collection.countOf(t) > 0) : (t = list_collection.rest(t)) {", "after": "while (t.tag() != .list and list_collection.countOf(t) > 0) : (t = list_collection.rest(t)) {", "verdict": "survived"} +{"id": "dd2dad043b8d", "file": "src/runtime/print.zig", "line": 502, "col": 30, "op": "cmp_le_to_lt", "before": "if (exp10 >= -3 and exp10 <= 6) {", "after": "if (exp10 >= -3 and exp10 < 6) {", "verdict": "killed"} +{"id": "84aafd32c0a5", "file": "src/runtime/print.zig", "line": 505, "col": 51, "op": "const_bump_in_arith", "before": "const point: usize = @intCast(exp10 + 1); // digits before the point", "after": "const point: usize = @intCast(exp10 + 2); // digits before the point", "verdict": "killed"} +{"id": "678ff77553f6", "file": "src/runtime/print.zig", "line": 847, "col": 5, "op": "branch_force_true", "before": "if (i > 0) try w.writeAll(sep);", "after": "if (true) try w.writeAll(sep);", "verdict": "survived"} +{"id": "39b153db36f7", "file": "src/runtime/print.zig", "line": 967, "col": 20, "op": "branch_force_true", "before": ".string => if (print_readably) try printString(w, string_collection.asString(v)) else try w.writeAll(string_collection.a", "after": ".string => if (true) try printString(w, string_collection.asString(v)) else try w.writeAll(string_collection.asString(v)", "verdict": "killed"} +{"id": "6d3d09bbd4e5", "file": "src/runtime/print.zig", "line": 1261, "col": 65, "op": "cmp_eq_to_ne", "before": "if (inst.field_count >= 1 and inst.fields()[0].tag() == .integer) {", "after": "if (inst.field_count >= 1 and inst.fields()[0].tag() != .integer) {", "verdict": "survived"} +{"id": "08f81fe36ca3", "file": "src/runtime/print.zig", "line": 1443, "col": 13, "op": "cmp_lt_to_le", "before": "while (i < n) : (i += 1) {", "after": "while (i <= n) : (i += 1) {", "verdict": "killed"} +{"id": "d317ae6644b1", "file": "src/runtime/print.zig", "line": 1480, "col": 42, "op": "branch_force_true", "before": "if (!first.*) try w.writeAll(if (kv) \", \" else \" \");", "after": "if (!first.*) try w.writeAll(if (true) \", \" else \" \");", "verdict": "survived"} +{"id": "32bf59779bf4", "file": "src/runtime/print.zig", "line": 1496, "col": 13, "op": "cmp_lt_to_le", "before": "while (j < child_count) : (j += 1) {", "after": "while (j <= child_count) : (j += 1) {", "verdict": "survived"} +{"id": "dff1e5c6d5d6", "file": "src/runtime/print.zig", "line": 1506, "col": 5, "op": "branch_force_true", "before": "if (compact_ns) |ns| {", "after": "if (true) |ns| {", "verdict": "unviable"} diff --git a/.dev/mutation/report-4b79a6d3-seed2.md b/.dev/mutation/report-4b79a6d3-seed2.md new file mode 100644 index 000000000..f7ec0c6bc --- /dev/null +++ b/.dev/mutation/report-4b79a6d3-seed2.md @@ -0,0 +1,17 @@ +# Mutation sweep — 4b79a6d3 (seed 2) + +- Candidates enumerated: 133 +- Mutants run: 3 +- Killed: 1 +- Survived: 0 +- Equivalent (registered, not scored): 2 +- Unviable (did not compile): 0 +- **Mutation score: 100.0%** (killed / (killed + survived)) + +No unexplained survivors in this sample. + +## Equivalent — registered as unobservable + +- `src/runtime/collection/vector.zig:284` const_bump_in_arith — Reached only when the tail holds exactly one element, i.e. count = 32m+1. arrayFor selects a 32-element leaf by i >> SHIFT_BITS, and (32m-1) >> 5 == (32m-2) >> 5, so `count - 2` and `count - 3` always name the same leaf. No input distinguishes them. +- `src/runtime/collection/vector.zig:529` const_bump_in_arith — popTail is entered only with new_count = 32m (a full-leaf boundary) and level >= SHIFT_BITS. new_count-1 and new_count-2 differ only in bit 0, which every level's >> level discards, so sub_index is identical at every level of the descent. + diff --git a/.dev/mutation/report-ac3a7ff0-seed1-unit-golden.md b/.dev/mutation/report-ac3a7ff0-seed1-unit-golden.md new file mode 100644 index 000000000..dcf134a49 --- /dev/null +++ b/.dev/mutation/report-ac3a7ff0-seed1-unit-golden.md @@ -0,0 +1,26 @@ +# Mutation sweep — ac3a7ff0 (seed 1) + +- Candidates enumerated: 280 +- Mutants run: 12 +- Killed: 4 +- Survived: 7 +- Equivalent (registered, not scored): 0 +- Unviable (did not compile): 1 +- **Mutation score: 36.4%** (killed / (killed + survived)) + +## Survivors — each is a line no test constrains + +| file | line | operator | change | +|---|---:|---|---| +| `src/runtime/print.zig` | 293 | cmp_gt_to_ge | `while (ks.tag() == .list and list_collection.countOf(ks) > 0) : (ks = list_collection.rest(ks)) {` -> `while (ks.tag() == .list and list_collection.countOf(ks) >= 0) : (ks = list_collection.rest(ks)) {` | +| `src/runtime/print.zig` | 386 | cmp_gt_to_ge | `if (items.items.len > lim) break;` -> `if (items.items.len >= lim) break;` | +| `src/runtime/print.zig` | 441 | cmp_eq_to_ne | `while (t.tag() == .list and list_collection.countOf(t) > 0) : (t = list_collection.rest(t)) {` -> `while (t.tag() != .list and list_collection.countOf(t) > 0) : (t = list_collection.rest(t)) {` | +| `src/runtime/print.zig` | 847 | branch_force_true | `if (i > 0) try w.writeAll(sep);` -> `if (true) try w.writeAll(sep);` | +| `src/runtime/print.zig` | 1261 | cmp_eq_to_ne | `if (inst.field_count >= 1 and inst.fields()[0].tag() == .integer) {` -> `if (inst.field_count >= 1 and inst.fields()[0].tag() != .integer) {` | +| `src/runtime/print.zig` | 1480 | branch_force_true | `if (!first.*) try w.writeAll(if (kv) ", " else " ");` -> `if (!first.*) try w.writeAll(if (true) ", " else " ");` | +| `src/runtime/print.zig` | 1496 | cmp_lt_to_le | `while (j < child_count) : (j += 1) {` -> `while (j <= child_count) : (j += 1) {` | + +### Survivors per file + +- `src/runtime/print.zig`: 7 + diff --git a/.dev/mutation/report-ce5cb6c8-seed1.md b/.dev/mutation/report-ce5cb6c8-seed1.md new file mode 100644 index 000000000..252a5ca75 --- /dev/null +++ b/.dev/mutation/report-ce5cb6c8-seed1.md @@ -0,0 +1,20 @@ +# Mutation sweep — ce5cb6c8 (seed 1) + +- Candidates enumerated: 141 +- Mutants run: 8 +- Killed: 3 +- Survived: 3 +- Unviable (did not compile): 2 +- **Mutation score: 50.0%** (killed / (killed + survived)) + +## Survivors — each is a line no test constrains + +| file | line | operator | change | +|---|---:|---|---| +| `src/runtime/collection/vector.zig` | 284 | const_bump_in_arith | `const new_tail_leaf = arrayFor(old, old.count - 2);` -> `const new_tail_leaf = arrayFor(old, old.count - 3);` | +| `src/runtime/collection/vector.zig` | 529 | const_bump_in_arith | `const sub_index = ((new_count - 1) >> @as(u5, @intCast(level))) & MASK;` -> `const sub_index = ((new_count - 2) >> @as(u5, @intCast(level))) & MASK;` | +| `src/runtime/collection/vector.zig` | 534 | bool_and_to_or | `if (new_child == null and sub_index == 0) {` -> `if (new_child == null or sub_index == 0) {` | + +### Survivors per file + +- `src/runtime/collection/vector.zig`: 3 diff --git a/.dev/mutation_equivalent.jsonl b/.dev/mutation_equivalent.jsonl new file mode 100644 index 000000000..785b93983 --- /dev/null +++ b/.dev/mutation_equivalent.jsonl @@ -0,0 +1,6 @@ +# Mutants proven to have no observable effect. Format: one JSON object per line, +# matched on file+op+before. A row here is a PROOF obligation: state why no input +# can distinguish the mutant. run.sh reports an entry that matches nothing as STALE. +{"file": "src/runtime/collection/vector.zig", "op": "const_bump_in_arith", "before": "const new_tail_leaf = arrayFor(old, old.count - 2);", "reason": "Reached only when the tail holds exactly one element, i.e. count = 32m+1. arrayFor selects a 32-element leaf by i >> SHIFT_BITS, and (32m-1) >> 5 == (32m-2) >> 5, so `count - 2` and `count - 3` always name the same leaf. No input distinguishes them."} +{"file": "src/runtime/collection/vector.zig", "op": "const_bump_in_arith", "before": "const sub_index = ((new_count - 1) >> @as(u5, @intCast(level))) & MASK;", "reason": "popTail is entered only with new_count = 32m (a full-leaf boundary) and level >= SHIFT_BITS. new_count-1 and new_count-2 differ only in bit 0, which every level's >> level discards, so sub_index is identical at every level of the descent."} +{"file": "src/runtime/print.zig", "op": "cmp_gt_to_ge", "before": "while (ks.tag() == .list and list_collection.countOf(ks) > 0) : (ks = list_collection.rest(ks)) {", "reason": "Ranges of every callee in the predicate, which is what decides this. map_collection.keys returns Value.nil_val for an empty map (the .array_map arm via keysArrayMap's `am.count == 0` guard, the .hash_map arm via `phm.root orelse`, and the .nil arm directly), and otherwise a chain built only by list_mod.consHeap seeded from Value.nil_val; sorted_collection.keys is the same shape, seeding keysInto with Value.nil_val. consHeap sets .count = countPrepending(tail), whose whole range is {1, n+1, COUNT_UNKNOWN} - every value >= 1. list_collection.rest of such a cell returns cell.rest, which in that chain is either another consHeap cell or the nil_val seed. So for every ks this loop can hold, ks.tag() == .list implies countOf(ks) >= 1: the conjunct is never the reason the loop stops, and >= 0 cannot change the iteration count. NOT the general claim that no empty .list exists - one does (rt.empty_list, list.zig, .count = 0, which is what '() evaluates to; verified: (list? '()) is true and (count '()) is 0). The equivalence is specifically that keys never returns it. If keys is ever changed to return the empty-list singleton for an empty map, re-derive this. One entry covers both loops (.array_map/.hash_map and .sorted_map) - the line text is identical and matching is on file+op+before."} diff --git a/.dev/mutation_id_aliases.jsonl b/.dev/mutation_id_aliases.jsonl new file mode 100644 index 000000000..92407233c --- /dev/null +++ b/.dev/mutation_id_aliases.jsonl @@ -0,0 +1,4 @@ +{"id":"dd2dad043b8d","scope":"printFloat","op":"cmp_le_to_lt","before":"if (exp10 >= -3 and exp10 <= 6) {","after":"if (exp10 >= -3 and exp10 < 6) {","reason":"Legacy line-based ID pinned by CLJW-PRINT-VERIFY before relocation-stable IDs."} +{"id":"84aafd32c0a5","scope":"printFloat","op":"const_bump_in_arith","before":"const point: usize = @intCast(exp10 + 1); // digits before the point","after":"const point: usize = @intCast(exp10 + 2); // digits before the point","reason":"Legacy line-based ID pinned by CLJW-PRINT-VERIFY before relocation-stable IDs."} +{"id":"39b153db36f7","scope":"printValueNative","op":"branch_force_true","before":".string => if (print_readably) try printString(w, string_collection.asString(v)) else try w.writeAll(string_collection.a","after":".string => if (true) try printString(w, string_collection.asString(v)) else try w.writeAll(string_collection.asString(v)","reason":"Legacy line-based ID pinned by CLJW-PRINT-VERIFY before relocation-stable IDs."} +{"id":"08f81fe36ca3","scope":"printRange","op":"cmp_lt_to_le","before":"while (i < n) : (i += 1) {","after":"while (i <= n) : (i += 1) {","reason":"Legacy line-based ID pinned by CLJW-PRINT-VERIFY before relocation-stable IDs."} diff --git a/.dev/mutation_targets.txt b/.dev/mutation_targets.txt new file mode 100644 index 000000000..dba3b310c --- /dev/null +++ b/.dev/mutation_targets.txt @@ -0,0 +1,25 @@ +# Layer 8 (ADR-0186) sweep targets — `scripts/mutation/run.sh --targets-from .dev/mutation_targets.txt` +# +# Ordering principle: a survivor is only worth the rebuild it costs, so sweep the +# code where an unconstrained line would be BOTH easy to get wrong and expensive +# to get wrong. That is the persistent collections (representation boundaries, +# hand-written HAMT arithmetic, and the site of Discussion #12) before anything +# that is mostly plumbing. +# +# Add a file when a sweep of the current list stops finding survivors, not +# before — the list is a queue, not an inventory. + +# --- the collections: boundaries, bit arithmetic, structural sharing --- +src/runtime/collection/map.zig +src/runtime/collection/set.zig +src/runtime/collection/vector.zig +src/runtime/collection/list.zig + +# --- value representation: the layer everything above it trusts --- +src/runtime/compare.zig +src/runtime/print.zig + +# --- next, once the above are clean --- +# src/eval/analyzer/analyzer.zig +# src/runtime/collection/sorted.zig +# src/runtime/collection/range.zig diff --git a/.dev/optimizations.md b/.dev/optimizations.md index 7ddbef666..8020e25d9 100644 --- a/.dev/optimizations.md +++ b/.dev/optimizations.md @@ -46,7 +46,7 @@ | O-001 | `runtime/collection/range.zig` + call sites | `(range a b s)` as a lazy cons-seq (one cons + lazy_seq per element) | Compact `.range` value `{start,end,step,count}`: O(1) count/nth, tight-loop reduce, chunked-cons `seq` | No per-element alloc on count/nth/reduce; 1 alloc/32 on walk | `phase14_range_indexed.sh` + diff oracle vs `clj` | D-163 / D-168 | | O-002 | `higher_order.zig::reduceFn` (`.vector` arm) | `reduce` over a vector via `seqFn` → `vectorToList` (N-element eager cons list), then walk via first/next | Index-walk: `vector.nth(coll, i)` in a tight `i` loop, honouring `reduced` | No N-element intermediate cons list; `(reduce f bigvec)` / `(into to bigvec)` went O(n) alloc → O(1). Measured `(reduce + (vec (range 1e6)))` 182s → fast | `phase14_*` reduce e2e + diff oracle vs `clj` | D-163 | | O-003 | `vector.zig::fromSlice` + `transient/transient_vector.zig::toPersistent` + `core.clj` `into`/`vec` | `persistent!` rebuilds the persistent vector via N persistent `conj`s (O(n log n)); `into`/`vec` = `(reduce conj …)`, also N persistent conjs | Bulk `fromSlice` builds the HAMT trie bottom-up from the transient's flat buffer in O(n) (32-element leaves → interiors grouped 32-at-a-time → root; last ≤32 = tail); `into`/`vec` route editable targets (vector/hash-map/hash-set, NOT sorted/nil/list) through `transient`/`conj!`/`persistent!` | `persistent!` O(n log n) → O(n); `into`/`vec` build O(n) over a flat buffer + one O(n) trie conversion, vs N persistent conjs. Measured `(count (vec (range 1e6)))` 121s → 2.4s; `(reduce + (vec (range 1e6)))` 123s → 2.5s | `vector.zig` boundary unit test (n ∈ {0,1,31,32,33,63,64,65,1023,1024,1025,1e5}: `fromSlice` == conj-built, same shift/tail/root) + diff oracle vs `clj` (into/vec over vector/map/set/sorted/nil/list + meta) | D-180 | -| O-004 | `core.clj` `map`/`filter`/`keep` 2-arg + `higher_order.zig::reduceFn` chunked arm + `sequence.zig::countFn` chunk drain + `chunked_cons.zig` chunk-builder | `(map f coll)` / `(filter pred coll)` build a meta-less lazy-seq walked one element per `nextFn` — each step allocs a cons + lazy_seq node and tree-walks the `.clj` body (~408µs/element measured) | Chunk-preserving: when the source is chunked (range seq / chunked map), transform a whole 32-chunk per thunk into a fresh chunk (JVM `chunk-cons` shape); `reduce`/`count` drain a chunk per step. Fill loop stays in `.clj` (a tree-walk loop is ~2µs/iter, negligible vs the 408µs amortised) | The ~408µs/element lazy-seq machinery is paid once per 32 elements, not per element. Measured `(count (map inc (range 1e5)))` 41.3s → 2.8s (~15x); `(reduce + (map inc (range 1e5)))` → 2.4s. (Residual is the per-element `f` vtable call ≈ 2µs — D-133's, not this cycle's.) | `phase14_chunked_seq.sh` (chunk-boundary count 1/32/33/65/1000 + reduce/nth/last/=/lazy-take) + diff oracle vs `clj` | D-163 / ADR-0065 | +| O-004 | `core.clj` `map`/`filter`/`keep` 2-arg + `higher_order.zig::reduceFn` chunked arm + `sequence.zig::countFn` chunk drain + `chunked_cons.zig` chunk-builder | `(map f coll)` / `(filter pred coll)` build a meta-less lazy-seq walked one element per `nextFn` — each step allocs a cons + lazy_seq node and tree-walks the `.clj` body (~408µs/element measured) | Chunk-preserving: when the source is chunked (range seq / chunked map), transform a whole 32-chunk per thunk into a fresh chunk (JVM `chunk-cons` shape); `reduce`/`count` drain a chunk per step. Fill loop stays in `.clj` (a tree-walk loop is ~2µs/iter, negligible vs the 408µs amortised) | The ~408µs/element lazy-seq machinery is paid once per 32 elements, not per element. Measured `(count (map inc (range 1e5)))` 41.3s → 2.8s (~15x); `(reduce + (map inc (range 1e5)))` → 2.4s. (Residual is the per-element `f` vtable call ≈ 2µs — D-133's, not this cycle's.) | the chunk-boundary pin set, landed as `phase14_chunked_seq.sh` and since migrated to `test/clj/suites/chunked_seq_test.clj` (count 1/32/33/65/1000 + reduce/nth/last/=/lazy-take) + diff oracle vs `clj` | D-163 / ADR-0065 | | O-005 **RETIRED 2026-06-11** | `eval/analyzer/{analyzer,bindings}.zig` + `eval/backend/tree_walk.zig::callMethodImpl` | Every TreeWalk fn call nil-inits the full `[MAX_LOCALS=256]Value` (~2 KB) call frame | (reverted) leave `[frame_size..256)` `undefined`, nil-init only the analyser frame high-water | **RETIRED — introduced a GC UAF.** `callMethodImpl`'s `locals` is ALSO the VM's frame: on the default VM backend it is handed to `vm.eval`, which publishes the WHOLE slice as a GC root (`gc_frame.locals`). The `undefined` tail was traced as Values under a `CLJW_GC_TORTURE` collect → SIGSEGV (deterministic repro: a deep call fills high stack slots, then a shallow fn runs under torture). The bound-the-slice fix then surfaced that `frame_high_water` *under-counts* the VM's slot needs for a fn containing a nested `fn*` (`slot_out_of_range`). 12 % on one microbench was not worth the correctness risk + the slot-accounting subtlety. The full-256 nil-init is restored. | repro `deep`+`shallow` under `CLJW_GC_TORTURE=1` (139→0 after revert) + full gate `e2e_phase16_gc_torture` | D-163 | | O-007 | `lang/primitive/higher_order.zig::sortNaturalFn` (`-sort-natural` leaf) + `core.clj` `sort` | `(sort coll)` ran the `.clj` `-msort` merge sort: per recursion level `(vec (take mid v))` + `(vec (drop mid v))`, and `-merge-sorted` does `(first a)`/`(rest a)`/`(conj acc …)`/`(empty? …)` + a `compare` call through the eval machinery per element | Default order copies the vector into a flat `[]Value` buffer, runs `std.mem.sort` (stable block sort) calling `valueCompare` directly (no eval reentry → no GC safepoint mid-sort → no frame rooting needed), and rebuilds via `vector.fromSlice` (O-003). Custom-comparator `(sort comp coll)` / `sort-by` stay on `.clj` `-msort` (a user comparator re-enters eval) | Eliminates the O(n log n) `.clj` take/drop/vec/rest/conj churn + per-comparison eval reentry. Measured `36_sort` bench (5×`(reduce + (take 100 (sort (vec (range 5000 0 -1)))))`) 0.39s → ~0.00s (startup-only) ReleaseFast | `test/diff/clj_corpus/sort.txt` (17 cases vs `clj`: empty/single/dups/mixed int·float ties/strings/keywords/nested vectors/custom comp/sort-by stability) + `zig build test` + `zig build lint` | D-163 | | O-008 | `build.zig` (`exe_mod.strip`) — **binary-size axis** | `b.installArtifact(exe)` shipped an UNSTRIPPED binary (the ~5.7K-symbol table rode in every release `cljw`); there was no packaging step to strip it, so the `bench/RELEASE_METRICS.md` "stripped" headline was aspirational, not the actual install | `.strip = optimize != .Debug` — every non-Debug build (ReleaseSafe / ReleaseFast / ReleaseSmall) strips at link time; Debug stays unstripped for `lldb`. cljw renders error traces from its own runtime `StackFrame` stack, not native symbols, so stripping costs no user diagnostics | Installed ReleaseSafe `cljw` 3.93 MB → **3.39 MB** (~400 KB / ~10% off the shipped artifact); ReleaseSmall also stripped (a further CLI `strip` floors it at 1.63 MB — Zig link-strip is less aggressive than CLI `strip` only for the ReleaseSmall layout) | `bash bench/release_metrics.sh` (3.24 MB on-disk ReleaseSafe) + smoke (stripped ReleaseSafe e2e: corpus 2289/2289, build_cljw pass) + `cljw -e '(sort [3 1 2])'` on the stripped binary | — | @@ -102,6 +102,11 @@ | O-040 | `eval/backend/vm.zig` (`op_vector_literal`) (ADR-0148; gc_alloc_rate) | `op_vector_literal` built an N-element vector via `vector.empty()` + N×`vector.conj` — each conj path-copies + allocates a fresh intermediate vector, so `[i (+ i 1) (+ i 2) (+ i 3)]` (gc_alloc_rate's loop body, ×200K) allocated 4 throwaway vectors per literal. `sample` showed the bench is NOT malloc-bound (18/3729 leaf, 0.5%) — the cost was the per-conj construction work (the map literal already had this fixed as O-026; vectors were left on the slow path). | One-shot `vector.fromSlice(rt, stack[sp-n..sp])`: n≤32 → 1 TailNode + 1 Vector (memcpy the elements); n>32 → bulk HAMT build. The elements stay rooted on the operand stack (op_top watermark) across the build. VM-only, mirroring O-026's map fast path (TreeWalk keeps empty+conj; diff oracle proves equality). | **`26_gc_alloc_rate` ReleaseSafe 108.4→45.8 ms (2.37×; 2.81×→1.30× vs Babashka 35.3 ms); system time 31→6.5 ms.** Broad win for every vector literal (esp. small literals in hot loops). | `zig build test -Dwasm` ×2 (diff oracle, TreeWalk slow ≡ VM fromSlice) + corpus + smoke; spot-checked n=0/1/40/nested vs clj (count 40, sum 780) | O-003, O-026, ADR-0148, D-450 | | O-039 | `runtime/numeric/promote.zig` (`operandManaged`/`OwnedManaged` + add/sub/mul BigInt else-branches) (ADR-0148; bigint_factorial) | The integer/BigInt arithmetic else-branch (reached when an operand is a heap BigInt) called `coerceToManaged` on BOTH operands, which **clones** a `.big_int` operand via `cloneWithDifferentAllocator` (alloc + limb memcpy) — yet `r.add`/`sub`/`mul` only READ them. `bigint_factorial` (`(reduce *' …)`, acc grows to ~9 limbs) clones the growing accumulator on every `*`. Same waste O-037 fixed for ratios, single-Managed flavour. | `operandManaged(rt, v, *OwnedManaged)` returns `*const Managed`: a BigInt aliases its stored Managed (zero alloc); an immediate Long materialises into a caller-stable `OwnedManaged` local (`deinit` only when active). The 3 else-branches use it; the `@addWithOverflow` both-int sub-paths keep `coerceToManaged` (both small, nothing to alias). | Removes the per-op accumulator clone. **`32_bigint_factorial` ReleaseSafe 26.4→22.4 ms (12 runs; 1.49×→~1.27× vs Babashka 17.7 ms).** ratio_sum unchanged (separate path). Broad win for all BigInt `+ - *`. | `zig build test -Dwasm` ×2 (diff oracle) + corpus + smoke; output unchanged (158 = digit count of 100!) | O-037, ADR-0148, D-450 | | O-038 | `runtime/numeric/promote.zig` (`ratioArith` scratch) + `runtime/numeric/ratio.zig` (`allocFromManagedPair` reduce-scratch) (ADR-0148; ratio_sum) | Each ratio op allocated its transient cross-multiply + gcd/divTrunc scratch as individual `Managed.init(rt.gc.infra)` calls — `ratioArith` 2-4 (rn/rd/lhs/rhs) + `allocFromManagedPair` 4 (gcd/r_num/r_den/rem) — so ~8 GPA malloc/free per `+`. `sample` profiling showed `libsystem_malloc` (`_xzm_*malloc`/`_xzm_free`/`memset`/`bzero`) ≈ 20% of leaf self-time. | Each function opens ONE `std.heap.ArenaAllocator` over `rt.gc.infra` and allocates all its scratch Managed from it; `defer arena.deinit()` bulk-frees on return. The two arenas don't alias (rn/rd live on `ratioArith`'s arena, read inside `allocFromManagedPair` which has its own). Result BigInts (numer/denom) still clone onto `gc.infra` (GC-managed, outlive the call). | Collapses ~8 individual malloc/free into ~2 arena chunk alloc/free per op. **`33_ratio_sum` 103.3→82.9 ms (12 runs); combined with O-037 108.1→82.9 (23%).** Re-profile: mem-leaf 227/1950 ≈ 11.6% (alloc no longer dominant; residual is interpreter dispatch + bignum compute → D-386). Broad win for all ratio arithmetic. | `zig build test -Dwasm` ×2 (diff oracle) + corpus 3157/3157 + smoke gate; output unchanged | O-037, ADR-0148, D-450 | +| O-056 | `lang/clj/clojure/core.clj` `pop` (vector arm) + `lang/primitive/collection.zig` `__vector-pop` (new) | `(pop v)` on a vector was `(into [] (take (dec (count coll)) coll))` — an O(n) REBUILD of a whole fresh trie. Meanwhile `runtime/collection/vector.zig::pop` already implements the JVM `PersistentVector.pop()` algorithm correctly (tail fast path; else pull the last leaf out of the root and collapse a level), and was reachable ONLY from `java.util.ArrayDeque`. A stratified-design ABSTRACTION-BARRIER violation (Grokking Simplicity ch.8): the layer above re-implemented a barrier operation with high-level seq ops instead of calling through it | Expose the existing barrier fn as the `__vector-pop` primitive and have `pop` call it. No new lower-layer code — the "minimal interface" move: the barrier already had the op | O(n) -> O(log32 n), and it now SHARES structure with the parent instead of allocating a fresh trie. Measured 2000 pops, ReleaseSafe: n=1000 539.5 -> 1.14 ms; n=4000 2151.5 -> 1.30 ms; **n=16000 8278.3 -> 2.47 ms (~3352x)**. Reference JVM clj 1.12.4 on the same box/probe: 1.93 ms — cljw was ~4288x SLOWER than clj here and is now within ~1.3x | `scripts/check_corpus_regression.sh` **4347/4347** golden clj outputs reproduced + e2e `phase14_peek_pop`, `phase14_deftype_collection_base`, `phase14_deftype_object`, `phase14_catch_internal` + spot checks (meta preserved, list arm -> rest, queue arm -> `__queue-pop`, empty still raises IllegalStateException) | D-044, ADR-0148, `.dev/bench/` | +| O-060 | `runtime/collection/string_seq.zig` (NEW, `.string_seq` A13 — a day-1 reservation with no producer) + `lang/primitive/sequence.zig` (`seqFn`/`countFn`/`firstFn`/`restFn`/`nextFn` + rev-walk `.string` arm → view) + `.string_seq` arms across `equal.zig` (isSequential via the interface_membership SSOT / isCountable / seqLen / valueHash / isSeqKeyValue) / `print.zig` (`printStringSeq`, ports-threaded) / `higher_order.zig` (fuse-base + chunkClassify) / `clojure_lang_method.zig` (native-collection arms) / `lazy_seq.zig` / `metadata.zig` / `meta.zig` / `runtime.zig` (GC trace registration) | `(seq s)` built an EAGER n-cell codepoint cons chain by asking `charset.codepointAt(s, i)` per element, and that restarts a `Utf8Iterator` from byte 0 every call — n calls at O(i) each = **O(n²)** overall. Measured cljw eager path: 2,000 chars 27 ms → 146,670 chars **89,910 ms**; downstream, bb-mcp paid ~79 s/boot asking `(seq value)` as a truthiness test on a 146,673-char nREPL response | A byte-offset VIEW satisfying the seq protocol: `StringSeq{backing, offset}`, two words. The cursor is a BYTE offset, not a codepoint index (an index would reintroduce the byte-0 walk): `first` decodes in place at `offset`, `rest` advances by the width it just read, `count` is a single forward byte scan (UTF-8 gives no length arithmetic — unlike the vector view's subtraction — but the scan allocates nothing and is no worse than `(count s)`). Consumers (`printStringSeq`, both `equal.zig` cursors) take the byte walk too rather than `nth(i)` (`nth` on UTF-8 is O(i) however the view is built, so the index-walk shape copied from array_seq would rebuild the quadratic inside the fix). `decodeAt` is total (malformed byte → width-1 raw), so a walk always advances — a view can never wedge or loop. Slot from the F-004 A13 day-1 reservation (already declared across interface_membership + class_name displayClassName), NO `heap_tag.unallocated` slot spent — the same position `.array_seq` (O-058) was in | **O(n²) → O(n).** NEW cljw ReleaseSafe end-to-end (incl. process startup + O(n) string build): `(count (seq (str-of-n "x")))` n=100k **0.13 s**, n=500k **0.60 s** (linear); the eager path's 89,910 ms at 146k chars is now sub-second (**>10,000× at that size**). clj JVM isolates the same seq+count to ~0 ms (StringSeq view + JIT); cljw's view is likewise O(1)/step, its residual wall time being the O(n) string construction, not the seq walk. Broad win: every hand-written `(seq s)` / `doseq`/`for` over a string and every `(seq value)` truthiness probe on a large string | `zig build -Dwasm -Doptimize=ReleaseSafe` (string-seq binary compiles) + corpus `string_seq.txt` (33 exprs vs clj, `33/33 reproduced`) + the 47-assertion observable-surface pin set, landed as e2e `string_seq_view.sh` and since migrated verbatim to `test/clj/suites/string_seq_view_test.clj` (vs clj: prints-as-seq, seq?/sequential?/coll?/not-string/not-vector, class=StringSeq, UTF-8 count/first/rest/nth/last, second/fnext/nthrest, `=`/hash with list AND vector + map-key + set-dedup + length-mismatch loses, vec/into/apply-str/reverse/take/drop/cons/concat, map/filter/reduce, empty→nil, scale count 20k + utf8 10k) | D-179, O-058, F-004, `feat/string-seq-view` | +| O-059 | `runtime/collection/sub_vector.zig` (NEW, `.sub_vector` D11) + `lang/clj/clojure/core.clj` `subvec` (fast-path) + `lang/primitive/collection.zig` (`__subvec` prim + `conjOne`/`getFn`/`nthFn`/`assocFn`/`containsQFn`/`__vector-pop`/`isEntrySeqTag`) + `.sub_vector` arms across `interface_membership.zig` (16 arrays) / `sequence.zig` / `core.zig` / `equal.zig` / `print.zig` / `compare.zig` / `lookup.zig` + `tree_walk.zig` / `meta.zig` / `metadata.zig` / `class_name.zig` / `lazy_seq.zig` / `walk.zig` / `json.zig` / `higher_order.zig` / `intrinsic.zig` / `serialize.zig` / `clojure_lang_method.zig` / `multimethod.zig` / `transient_array_map.zig` / `array_seq.zig` (backing arm). `vector.subvec`→`copyRange` (materializing; LBQ compaction) | `clojure.core/subvec` on a vector was `(into [] (take (- end start) (drop start v)))` — an eager O(n) rebuild of a whole fresh trie per call, the LAST catastrophic collection gap (measured 7480x slower than clj). JVM never copies: `subvec` returns an `APersistentVector$SubVector` sharing the parent's root+tail (D-044/D-583). The Zig `vector.subvec` (eager) was reachable only from LBQ compaction, which genuinely wants a copy — so the view had to be a NEW representation, not a rewrite of it | A new REPRESENTATION satisfying the existing IPersistentVector protocol: `SubVector{start,end,parent,meta}`, 32 bytes fixed whatever the span. `count = end-start` (O(1)), `nth(i) = parent.nth(start+i)`. JVM `SubVector` semantics: FLATTEN on nest (parent is always a plain `.vector`), never-empty (empty range → EMPTY plain vector), write ops stay SubVectors via `parent.assocN` (`conj`/`assoc`/`pop`, structural sharing — never materialize). Slot came from the F-004 `box`→`sub_vector` amendment (user-selected, ADR-0179), 64-slot layout unchanged. Interface membership matches JVM ground truth exactly: `.sub_vector` joins every array `.vector` is in EXCEPT `EDITABLE_TAGS` (no transient) and `IREDUCEINIT_TAGS` (SubVector is not IReduceInit) — verified live against clj 1.11.1. seq/rest/next route through the O-058 `.array_seq` view (its backing switch gained a `.sub_vector` arm), so a subvec's seq is O(1) too. `=`/hash reuse the equal.zig `.vec` cursor by init'ing it with the parent + absolute [start,end) indices — zero new cursor code | O(n) alloc + trie rebuild -> O(1). Measured 2000 subvec ops on a 16000-element vector, ReleaseSafe: **9215 -> 0.77 ms (~11967x)**. Reference JVM clj 1.11.1, same box/probe: 1.06 ms — cljw was ~8693x SLOWER than clj and is now at PARITY (slightly faster). Broad win: `subvec` is the idiomatic vector slice, and it now shares structure instead of rebuilding | `zig build test -Dwasm` ×2 (VM + tree_walk diff oracle, all Values equal) + new `sub_vector.zig` units (layout, flatten-on-nest, never-empty, conj/assoc/pop parent-untouched, with-meta) + the 52-assertion observable-surface pin set (vector?/hash-eq/map-key/set-dedup/compare/into-{}/walk/nested/structural-sharing/meta/class=SubVector/empty+full-range/OOB-throws), landed as e2e `subvec_view.sh` and since migrated verbatim to `test/clj/suites/subvec_view_test.clj` + corpus `subvec.txt` (31 exprs) + `phase14_subvec.sh` (bounds) still green | D-583, D-044, O-058, ADR-0179, F-004, `private/notes/subvec-seqview-design.md` | +| O-058 | `runtime/collection/array_seq.zig` (NEW, `.array_seq` A14) + `lang/primitive/sequence.zig` (`seqFn`/`restFn`/`nextFn`/`countFn`/`firstFn` + the three seq-walk helpers) + `.array_seq` arms in `equal.zig` / `print.zig` / `higher_order.zig` / `lazy_seq.zig` / `collection.zig` (`nthFn`) / `clojure_lang_method.zig` | `(seq v)` / `(rest v)` / `(next v)` on a vector built an EAGER `PersistentList` copy (`vectorToList` / `vectorTailAsList`): n `consHeap` allocations AND n trie descents per call. Clojure never copies — `PersistentVector$ChunkedSeq` is a VIEW holding `(vector, index)`. `.dev/bench/README.md` ranked this #2 and called SubVector (#1) its prerequisite, on the reading that a chunked seq's `next` must be "the vector minus its first 32 elements". It is not: `next` of a seq is a SEQ, and clj's own answer is a ChunkedSeq over the SAME vector at a higher index. So #2 never depended on #1, and the `heap_tag` census question #1 raises does not gate it | A new REPRESENTATION satisfying the existing seq protocol (OCP — no consumer changes, no `if` at call sites): `ArraySeq{backing, index, meta}`, 32 bytes, fixed size whatever it spans. `first` = one indexed read, `rest` = the same view at `index+1`, `count` = `backingCount - index`, `nth` = one indexed read (clj WALKS here — ChunkedSeq is Counted, not Indexed). Invariant: `make` answers nil when the view would be empty, mirroring `.range`, which is what makes `first` total. Costs NO slot from `heap_tag.unallocated`: A14 `.array_seq` was a day-1 reservation already declared across every `interface_membership.zig` protocol table and already named in `class_name.zig` but had no producer — `equal.zig`'s header literally anticipated "when a producer mints those tags". It also REPAIRS a latent census inaccuracy: A14 was classified boxed-in-production while nothing produced it | O(n) alloc + O(n log32 n) -> O(1) for all three. Measured 2000 ops on a 16000-element vector, ReleaseSafe: **`seq` 2123 -> 0.36 ms (~5897x), `rest` 2253 -> 0.39 ms (~5777x), `next` 2164 -> 0.38 ms (~5695x)**. Reference JVM clj 1.12.4, same box and probe: 0.60 / 0.87 / 0.99 ms — cljw was 2184-3549x SLOWER than clj on these and is now 1.7-2.6x FASTER. Broad win: this is what every hand-written `loop`/`recur` seq walk and every library seq walk pays (the idiomatic `reduce`/`doseq`/`into` paths already bypassed `seq` via O-002/O-004/O-032, which is why no single benchmark regressed on it) | `zig build test -Dwasm -Doptimize=ReleaseSafe` x2 (VM + tree_walk diff oracle) + new array_seq.zig units (layout, never-empty invariant, count-at-offset, rest/drop termination, saturating drop) + new the 76-assertion observable-surface pin set, landed as e2e `vector_seq_view.sh` and since split — 75 expression pins migrated to `test/clj/suites/vector_seq_view_test.clj`, the GC-torture rooting case kept in the e2e because CLJW_GC_TORTURE is read at process start. Pins: print-as-seq, `=`/`hash`/map-key interop with list AND vector, length-mismatch must lose, count exact at every offset, nth + default + OOB + negative, empty/one-element boundaries, next-walk visits each exactly once, 5000-element multi-level trie, nested collections) + `CLJW_GC_TORTURE=1` on the view (proves the trace fn reaches `backing`) + `phase16_gc_torture` | D-044, O-056, O-057, ADR-0179, F-004, `.dev/bench/README.md` | +| O-057 | `runtime/collection/list.zig` (`COUNT_UNKNOWN` + `countPrepending` + `exactCountOf`) + `lang/primitive/sequence.zig` `countFn` (`.list`/`.cons`/`.lazy_seq` arm) | `(count lst)` WALKED every `.list`, even though `Cons` already stores an O(1) `count`. The stored field was distrusted for a real reason: a `.list` cell may head a MIXED chain whose `rest` leaves `.list` — `(cons x (map …))`, `(conj (range 3) 99)` — and there `1 + countOf(tail)` records the `.list` PREFIX length as if it were the total. The prior comment deferred the fast path to the F-004 `.list`/`.cons` tag split (D-178) | Make the stored count SELF-VALIDATING rather than splitting the tag: `countPrepending` writes `COUNT_UNKNOWN` (`maxInt(u32)`) when the tail is a non-`.list` seq or itself unknown, so unknown-ness propagates head-ward and the field is never a prefix presented as a total. `exactCountOf` returns `?u32`; `countFn` takes the O(1) path when it is non-null and keeps the existing walk otherwise. Sentinel chosen so every existing `countOf(v) > 0` / `== 0` emptiness test keeps its meaning (a cons cell always holds ≥1 element) | O(n) -> O(1) for a pure list. Measured 2000 `count` ops on a 16000-element list, ReleaseSafe: **380.15 -> 0.19 ms (~2006x)**. Reference JVM clj 1.12.4, same box and probe: 0.18 ms — cljw was ~2711x slower than clj on this op and is now at PARITY. Mixed chains keep the walk, so they keep their exact answer | Mixed-chain probes all still exact (`(cons 1 (map inc [1 2 3]))` 4, `(conj (range 3) 99)` 4, `(cons 0 (seq [1 2 3]))` 4, nested `(cons 1 (cons 2 (map inc [1 2 3])))` 5, pure `(list 1 2 3)` 3, `(list)` 0) + `(= (cons 1 (lazy-seq [2 3])) (list 1 2 3))` still true (the equal.zig `isCountable` exclusion of `.list` is what keeps `COUNT_UNKNOWN` out of the length short-circuit) + list.zig units + the 22 exactness pins (landed as e2e `list_count_exact`, since migrated to `test/clj/suites/list_count_test.clj`) + corpus | D-178, D-044, `.dev/bench/` | | O-055 | `eval/backend/vm.zig` (eval loop) + `eval/backend/tree_walk.zig` (`evalLoop` + fn-tail-recur loop) — budget-poll TLV hoist (D-450 lever 1) | The per-op / per-back-edge budget poll read the `threadlocal eval_budget_mod.current` each iteration. On macOS every threadlocal access is a dyld `_tlv_get_addr` CALL (D-571 moved the budget off the near-free Runtime field into the threadlocal, silently voiding D-386's "polls are near-free" verdict); `sample` on the rush-hour BFS showed `_tlv_get_addr` ≈ 7% top-of-stack, part of it this poll | Read `current` ONCE into a local at eval/loop entry; the per-op poll unwraps the local. Semantics preserved because every writer of `current` is stack-disciplined around a host call that re-enters eval (`with-budget` installs → invokes via callFn = a NEW eval invocation that re-reads at its entry → restores before returning to the hoisted loop); worker adopt/release bracket the worker's own eval | Same-session A/B on the rush-hour BFS headline (`(g/generate 7 :medium)`, M4 Pro, ReleaseSafe): median A 16.86 s (8 runs) → B 16.48 s (10 runs), **−2.3%**. The remaining TLV cost sits under inlined `stepOnce` (error-info per-call trace push/pop candidates — D-450 lever 2) | `zig build test -Dwasm` ×2 (diff oracle + the ADR-0125 dual-backend budget kill test) + `budget_thread_ownership` + `phase16_eval_budget` e2e green on the hoisted binary + smoke | D-450, D-571, D-386, ADR-0125 | | O-053 | `runtime/collection/value_sort.zig` (new) + Arrays/Collections/higher_order sort sites + 3 tiny-N insertion swaps — **binary-size axis** (ADR-0172 L6) | Every `std.mem.sort(T, items, ctx, less)` call monomorphizes ~25KB of block sort per (element, context, comparator) triple — 9 call sites had accreted ~224KB / 93 symbols; three of them sort `[]Value` with site-local comparators, and three more sort tiny-N arrays (txn ref set / fn arities / bencode dict keys) where a full block sort buys nothing | ONE shared instantiation: `value_sort.sort(items, ctx: *anyopaque, less: fn-pointer)` — the three Value sites pass their context opaquely (cast back inside the comparator); the three tiny-N cold sites drop to `std.sort.insertion` (~1KB each). Hot index-permutation sort (O-010 sortByKeysFn) keeps its dedicated instantiation | Shipped ReleaseSafe **7,073,080 → 6,974,584 B (−98.5KB)**; sort perf NEUTRAL (50k-element `(sort xs)` ×10: 1.00±0.03 hyperfine A/B — the indirect call is invisible next to valueCompare) | `zig build test -Dwasm` ×2 + new value_sort unit + probes: `(sort [3 1 2])`, string sort, `java.util.Collections/sort` + smoke | ADR-0172 (L6), O-010, D-515 | | O-052 | `build.zig` (`exe_mod.unwind_tables`) — **binary-size axis** (ADR-0172 L1) | Release builds carried `__eh_frame` + `__unwind_info` (749 KB, ~8% of the 9,469,816 B shipped binary) for native stack unwinding — but the shipped binary is STRIPPED (O-008), and a stripped Zig binary already prints "stack tracing is disabled" on panic (probed 2026-07-16 with a forced OOB panic, identical output with/without tables), so the tables bought nothing user-visible | `.unwind_tables = if (optimize != .Debug and !profile) .none else null` — mirrors the O-008 strip predicate; Debug + `-Dprofile` keep tables for lldb/Instruments. cljw renders Clojure errors from its own StackFrame stack; arm64 keeps frame pointers for debugger frame-walks | Shipped ReleaseSafe `cljw` **9,469,816 → 8,731,096 B (−738,720 B / −7.8%)**; `__eh_frame`/`__unwind_info` gone from `size -m` | smoke (units + diff oracle + e2e core on the table-less ReleaseSafe binary) + `size_claims` gate + panic-output probe pre/post identical | ADR-0172 (L1), O-008, D-515 | diff --git a/.dev/orbstack_setup.md b/.dev/orbstack_setup.md index e8bf09201..de245d341 100644 --- a/.dev/orbstack_setup.md +++ b/.dev/orbstack_setup.md @@ -5,7 +5,7 @@ > The cross-arch verification host is the `ubuntunote` SSH box > (native x86_64 hardware); see > [`.dev/ubuntunote_setup.md`](ubuntunote_setup.md) and -> [`.dev/decisions/0049_orbstack_linux_gate_retired.md`](decisions/0049_orbstack_linux_gate_retired.md). +> ADR-0049. > > This file is retained because OrbStack remains a useful > dev-convenience host for interactive scratch (REPL probes, diff --git a/.dev/principle.md b/.dev/principle.md deleted file mode 100644 index 78936607c..000000000 --- a/.dev/principle.md +++ /dev/null @@ -1,371 +0,0 @@ -# ClojureWasm v1 — Principles - -> The project's working principles in one short file. It does not -> duplicate ROADMAP / ADR / CLAUDE.md. When plan and implementation -> conflict, this file wins; the others should be edited to match. -> Detail lives elsewhere; this file holds only fuzzy premises so it -> does not become a checklist. - -## Premises - -- **Reason backwards from the finished shape**. Judge today's choice - by how it will look in the finished form. Ask "how will this part - read when the project is done?" and pick the choice that fits - that view. -- **The plan is a means; the finished form is the end**. ROADMAP / - ADR / rule are tools for reaching the finished form. When the - tool distorts the finished form, fix the plan side. Ad-hoc - workarounds that bend the finished form are forbidden. -- **Follow the Bad Smell**. Mid-implementation "wait, this feels - off" is a signal. Do not suppress it; you have the latitude - (and the obligation) to interrupt the current activity and - investigate without compromise. The interrupt is not a stop — - the loop keeps running, the surgery just lands at the right - depth before the next step. -- **Latitude and discipline coexist**. Without latitude you can't - be creative; without discipline you don't reach the finished - form. Hold both. - -## Bad Smell catalogue - -Smells that surface during implementation. The list is a memory -aid, not a checklist. Interrupt the current activity on any of -these — and on others that fit the same shape: - -- **TODO smell** — wanting to leave a "fix this later" comment in - the code. -- **Magic constant smell** — "100 for now", "1024 should be fine". -- **Branch sprawl smell** — stacking `if` / `else` three or more - levels deep. -- **Stub smell** — a do-nothing implementation that lets you feel - "it works". -- **Spec drift smell** — ROADMAP / ADR text and the implementation - seem to disagree. -- **Cascade smell** — fixing one file forces touching five others. -- **Resignation smell** — settling for "it can't be helped" or - "the JVM does it this way too". -- **Premature commitment smell** — the implementation path is - already settled before the test is written. -- **Test skip smell** — leaning on "I'll write this test in a - later Phase". -- **Doc drift smell** — implementation changed but ROADMAP / - handover has not. -- **Shortcut smell** — the proper path looks costly, so a side - route looks attractive. -- **Workaround smell** — "if I can dodge this point, I can keep - moving". -- **Smallest-diff bias smell** — picking the smallest-diff option - *because it is smallest*, rather than because it lands at the - finished form. The project's premise is "the finished form - wins"; ROADMAP P5 ("smallest diff first") is a tie-breaker, not - a veto. If the smallest-diff choice would be re-done later - anyway, take the bigger surgery now and keep the finished form - in place. -- **Reservation-as-bias smell** — treating a "reserved" slot / - number / name as a hard constraint when it was only a memo. ADR - numbers, NaN-box slots, debt rows that say "future X is - reserved" — none of these are promises to obey. If the reservation - blocks the cleanest landing, edit the reservation and take the - number. -- **Progress-pressure smell** — the autonomous loop's forward - momentum encouraging "I'll fix this later" / "let me push and - move on" / "the next task is waiting". Forward momentum is a - feature, but it must not bypass depth-3 / depth-4 surgery when - the smell calls for it. "Let me finish 4.X first and come back" - rarely comes back. Specific surface forms that count as this - smell triggering: "defer to follow-up cycle", "defer to Phase - N mid", "open a debt row and pick it up later", "this would - exceed the LOC budget", "the fusion is the right shape but - too big for this cycle". When the deferral rationale is - *cycle-size-based* (not F-NNN-blocked), the smell has fired — - see also Cycle-budget defer smell. -- **Cycle-budget defer smell** — declining a finished-form-clean - structural decision **because the current cycle would grow** - (LOC / file count / migration scope). cw v1 cycles are not - bounded by an external time budget; the only constraints are - F-NNN (`project_facts.md`) and the Bad Smell catalogue itself. - Cycle / diff size is **not** a project constraint and not a - sufficient reason to file a structural surgery as deferred - debt. Concrete trigger shape: a Devil's-advocate fork rates - Alt N as finished-form-clean within the F-NNN envelope, and - the main loop picks Alt M < N citing "cycle budget" / "this - expansion is too large now" / "Phase N mid is the right place - for the rest". That ranking IS the smell. Take the finished- - form-clean surgery in the same cycle, even when it doubles the - diff. The exception (= NOT this smell): when the deferral is - driven by a real F-NNN block (e.g. "the cleaner shape requires - F-006 to amend; F-006 is user-owned, so we cannot land it - unilaterally"). In that case the main loop picks the best - F-NNN-compliant alternative and continues; that is not a - cycle-budget defer — it is an F-NNN-respecting pick. The - smell sensor's first question: "is this defer driven by F-NNN - or by cycle size?" If cycle size, land the surgery. -- **Direction-ask smell** — when the clean / obvious work thins, - reaching for `AskUserQuestion` (or any "which should I do next / - A vs B vs wind-down?" prompt) to have the user choose the next - task or direction. The autonomous loop SELF-SELECTS the next unit - (finished-form first) and keeps raising precision — coverage while - it lasts, then quality work (tests, robustness, error-path - fidelity, refactors, moderate features, audits). "Clean-trivial is - exhausted" / "the next items differ in character" / "the user has - been hands-on" are NOT reasons to ask — they are the moment to pick - the best next unit yourself. (`AskUserQuestion` is still fine for a - genuine in-task fact you cannot derive — an external credential, a - product preference with no default — never for "what next".) See - CLAUDE.md § The only stop. -- **Silent default-shift smell** — a previously established - default behaviour (decreed in docs OR established by past - commit pattern) is being silently changed to a different - default, with no committed text recording the new policy. The - loop tells itself "it's just my judgement; nobody set this in - stone" and shifts. Examples: through commit 025dea9 the loop - staged `bench/quick_baseline.txt` in source-bearing commits; - from commit 4.6 onward it silently stopped, with no - conventions / handover / ROADMAP note backing the change (the - user surfaced this on 2026-05-24 and the policy was made - explicit in commit bda8b4d as a result). Detection: when you - notice "what I'm about to do is different from what past - commits did", **stop and ask**: "is this a policy change? - Where is it recorded?" If nowhere, either (a) follow the past - pattern, or (b) record the new policy explicitly (CLAUDE.md / - principle.md / debt.yaml / a relevant `.claude/rules/*.md` rule) - before applying the new default. Same shape as Reservation-as-bias - and Smallest-diff bias — invisible drift away from committed - intent. -- **Stale-phase-ref smell** — code / docs / scripts cite a Phase - number ("Phase 4 entry: informational", "until Phase-5 GC - arrives", "Phase 7+ target") whose status has since shifted in - ROADMAP §9 (e.g., Phase 4 has closed; Phase 5 GC has landed). - The cite ages into a lie. Detection: when editing a file, scan - for `Phase \d+` references and verify against the ROADMAP - tracker before committing. Example: the 6 placeholder hook - scripts that said "Phase 4 entry: informational only" while - Phase 6 was the active phase (surfaced at Wave-16 inventory - 2026-05-26). -- **Framework-incomplete smell** — introducing a new discipline - (rule / SSOT / hook / marker convention) without, in the same - cycle, defining the discovery criterion + running the sweep + - retrofitting existing sites. The result is a 2-tier codebase: - new sites comply, old sites do not, and nothing tracks the - asymmetry. See `.claude/rules/framework_completion.md`. Example: - the Wave-15 spike landed `provisional_marker.md` + yaml + hook - but missed 5 retrofit sites; silent-failure-hunter caught them - at review (commit ef4f683 fixed). -- **Defer-to-amnesia smell** — saying "watch" / "defer" / "fix - later" without recording the deferral in - `.dev/watch_findings.md` with a testable revisit trigger. The - decision becomes unrecoverable — no future audit walks the - deferred set. Example: F4 / F7 / F9 review findings were - classified "watch" without rows; user prompt later surfaced - them as the Wave-15-follow-up cycle. The fix is to write the - `watch_findings.md` row in the same edit as the defer - decision. -- **Dual-backend drift smell** — landing a new analyzer Node - variant with a real TreeWalk arm + a VM compile arm that - silently raises `error.NotImplemented` (or silently drops a - significant field via `_ = n.some_field;`) without a - `// VM-DEFER:` marker + tracked debt row. ADR-0005 + F-002 - make dual backend the finished form per ADR-0036; an unmarked - VM gap is a deferred-amnesia variant specific to the backend- - parity contract. cw v0's late-Phase catch-up of accumulated VM - gaps consumed weeks of cycle time; cw v1's - `scripts/check_dual_backend_parity.sh` PreToolUse hook (per - ADR-0036 + `.claude/rules/dual_backend_parity.md`) mechanises - the gate, but the smell sensor remains the first line of - detection — if you're about to leave a VM arm as - `error.NotImplemented` without a marker, that is this smell. -- **Ad-hoc-pass smell (個別最適化 entry)** — when a real library - fails to load (the gap-discovery technique, F-013) and the fix - that comes to mind is "make *this* library pass": add the one - `clojure.lang.*` interface it used, the one missing var, the one - Java method — driven by *what this library happened to need* - rather than *what can legitimately appear, derived from the - definition*. The tell is a hand-maintained allowlist growing - library-by-library (`std.mem.eql(name, "Object") or eql("IDeref") - or …` at N sites). F-013 forbids this: the response must 網羅 - (cover the whole definition-derived class) in canonical - (Zig-equivalent, non-JVM) form, so the system rises one level and - *other* libraries' coverage improves too. The structural fix is a - closed-set SSOT whose keys come from the definition + a gate that - bounds the set and requires every entry to route to a *generic* - surface (no per-library slot) — ADR-0102 (`host_interfaces.yaml`) - is the canonical instance; `compat_tiers.yaml` is the prior art. - If you're about to add a name to an allowlist because one library - needed it, that is this smell — model the definition-derived - family + gate it instead. - -The catalogue is not exhaustive. Any felt smell counts. - -## When the smell triggers - -A trigger is an **interrupt**, not a stop. The loop suspends the -current activity, investigates without compromise, takes the -surgery at the right depth, then resumes. The procedure is fuzzy; -the writer chooses: - -1. Suspend the current edit. Re-read the related ADR / ROADMAP / - surrounding implementation as long as it takes. -2. Fork a subagent if a deeper look is warranted. -3. Imagine how this will look in the finished form. -4. Pick a depth and act on the spot. -5. Land the surgery (commit / ADR / Supersedes chain), then return - to the per-task TDD flow. - -Multiple triggers per cycle just mean multiple interrupts. They do -not accumulate into anything; each is handled and the loop -continues. "Let me finish this task first and come back" is the -Progress-pressure smell — interrupt now, not later. - -## Four depths of revision - -| Depth | Situation | Where the conclusion lives | -|----------------|---------------------------------------------|-----------------------------------------------------------------------| -| 1. Local fix | One file, no ADR cross-reference | commit message | -| 2. ADR amend | Wording of an existing ADR needs adjustment | ADR Revision history | -| 3. ADR cascade | Several ADRs need amending together | New ADR amendment + Affected files entries | -| 4. Big rewrite | ROADMAP-scale redesign | New ADR (`Supersedes NNNN`) + old surface moved to `private/archive/` | - -Choosing the depth is the writer's call. The four levels are not a -flow chart. - -**All four depths proceed within the autonomous loop.** Depth 2-4 -land their conclusion (ADR amendment / new ADR / archive move) in -a separate commit before the source commit, then the loop -continues. The AI drafts and accepts the ADR itself — there is no -external review gate. See CLAUDE.md § Autonomous Workflow -"ADR-level designs are handled inline". - -**Devil's-advocate subagent is mandatory at depth ≥ 2.** Before -the ADR is accepted, a `general-purpose` subagent is forked with -**fresh context** to produce 3 alternative shapes (smallest-diff / -finished-form-clean / wildcard) **within the active F-NNN -envelope from `project_facts.md`**. The brief explicitly -instructs the subagent to *not* propose alternatives that -violate any F-NNN; if the only finished-form-clean option would -require violating an F-NNN, the subagent records that finding as -the leading "Alternatives considered" entry — the main loop sees -it, picks the best F-NNN-compliant shape, and continues. The -finding never halts the loop (F-NNN amendment is a user action, -not a loop action). The full output (3 alternatives + any -"violates F-NNN" finding) is embedded verbatim into the ADR's -"Alternatives considered" section. This is the antidote to the -loop's accumulated goal-drift / instruction centrifugation — -alternatives sourced from a context without the main loop's -momentum surface options the main loop is attention-suppressed -against. - -## Three questions to picture the finished form - -When the sensor interrupts, ask: - -1. How will this read when the project is done? -2. Which later Phase actually feels the effect of this choice? -3. If I patch it lightly now, what does the redesign cost look like - later? - -## Structural imagination (before touching structural plans) - -> **Origin**: this phase exists because of user invariant **F-003** -> in `.dev/project_facts.md` (decision-deferral over -> decision-seizure on structural plans). The output of past -> imagination cycles lives in `.dev/structure_plan.md` -> (anticipated directory tree Phase 5-20). New imagination -> cycles amend that file in place. -> -> **Scope boundary** (added 2026-05-24 to close a sabotage -> path): Structural imagination applies only to **open -> structural plans** — plans where the direction is not yet -> determined by an `F-NNN` in `project_facts.md`. When an F-NNN -> already fixes the direction, that direction **is the decision**; -> the loop's job is to implement it cleanly, not to re-imagine -> it. Examples at 2026-05-24: -> -> - F-004 fixes NaN-box second generation = 4 × 16 = 64 slot, -> 44-bit pointer. The Phase 5 entry's loop does **not** imagine -> other layouts; it implements F-004. -> - F-005 fixes the numeric tower shape. The Phase 5 entry's -> loop does **not** imagine alternative numeric semantics; it -> implements F-005. -> - F-006 fixes the GC strategy. The Phase 5 entry's loop does -> **not** consider generational at Phase 5; it implements F-006. -> - F-007 fixes chapter cadence = dormant. The loop does **not** -> propose resumption on its own. -> -> Where F-NNN leaves room (= directory split timing, file naming -> within the laid-out tree, internal helper organisation, etc.), -> the Structural imagination phase **does** apply, and decisions -> defer to the owning Phase entry's owner as before. -> -> **F-015 narrowing (added 2026-06-15, ADR-0142)**: the project is -> near-complete and the phase-queue model is retired — there is no -> future "Phase N entry owner" to defer to for the **gap areas** -> (Concurrency / Wasm-edge-native / VM-perf, ROADMAP §9.0; all -> BUILT). For gap-area work, the draining unit **is** the owner: -> make the structural decision now (within the F-NNN envelope + -> the DA-fork discipline), do not defer it to a non-existent phase. -> Defer-to-a-future-owner survives **only** for the genuinely-future -> bucket (ClojureScript→JS / C-FFI / broad-JIT, §9.0) — structure -> there can still be imagined-and-deferred since it is distal. Read -> "the owning Phase entry" below as "the draining gap-area unit" -> for everything except that future bucket. F-003 (decision-deferral -> on genuinely-open future structure) is unchanged in spirit; F-015 -> only removes the blanket "defer to Phase N" for already-built areas. - -ROADMAP extends through the gap areas + the genuinely-future bucket -(§9.0; "Phase 20" = broad JIT, the distal tail). When a task touches a -**structural plan** that future phases will live with, the -autonomous loop is **not** allowed to decide on the structure's -behalf for the future. The decision belongs to the owning -Phase. What the loop **must** do is **imagine the full -ROADMAP range** so the future owner inherits foresight, not a -blank slate. - -Structural plans include (at minimum): - -- **Reservation tables**: NaN-box slots, enum slots, ROADMAP - row queue, debt row family. -- **Directory & file structure**: src/ subdirectory layout, - file-size soft cap (ROADMAP A6 ≤ 1000 lines), candidate splits - (`value.zig`, `analyzer.zig`, `main.zig` etc.). -- **Responsibility separation**: which file owns which concept, - whether mixed-concern files need to fan out. -- **Dependency graph**: zone layering (`zone_deps.md`), vtable - hooks, cross-module references that survive future phases. - -What the loop **must** do at any task that touches one of these: - -1. Spend a real moment imagining the full ROADMAP range - (Phase 5–20) and walking through what the structure will need - to absorb across that horizon. **Do not skip this step**; do - not shortcut to "let me just decide now". -2. If the imagination reveals a structural gap (table close to - exhaustion / file headed past the soft cap / responsibility - leaking / dependency about to cycle / planned ADR will - collide), record it as a **debt row** scheduled at the owning - Phase's entry — do not resolve it here. -3. The decision (delete / re-scope / split / keep / extend) - lands when the owning Phase's task is opened. The current - loop's job is to set the imagination output up so the future - owner can resolve cleanly. - -This is the antidote to the Progress-pressure smell on -structural work. The smell says "let me clean this up since -I'm here"; the structural imagination phase says "the owner -has the context, I have the foresight — give the owner the -foresight, don't seize the decision". - -**Spike the primitive before the structural ADR (ADR-0089).** When the -structural decision rides an unfamiliar host capability — chiefly Phase B -concurrency on Zig 0.16 (atomics, `std.Io.*` threading, GC thread-safety) — -validate the *Clojure-equivalent, internals-free* shape (F-011 §2 + -`no_jvm_specific_assumption`) in a throwaway spike first (web search + the Zig -0.16 source are fair game; spike code lives in `private/`, no gate). The spike -de-risks the ADR; it does not replace the DA-fork or the imagination pass. - -## How this file is maintained - -- Keep it short. Detail belongs in ROADMAP / ADR / rules. -- If another file (ROADMAP / ADR / CLAUDE.md / rule) contradicts - this one, edit the **other** file. This file is the meta layer. -- Edits to this file ride a normal commit, but they must pass the - same Bad Smell self-audit that the principle prescribes. diff --git a/.dev/probes/testcheck/README.md b/.dev/probes/testcheck/README.md new file mode 100644 index 000000000..b5c30f6b3 --- /dev/null +++ b/.dev/probes/testcheck/README.md @@ -0,0 +1,62 @@ +# probes/testcheck — landing clojure.test.check on cljw (D-298) + +Measurement scripts for the test.check rung. test.check is the generator engine +`malli.generator` sits on, so it gates every schema-driven property/mutation +suite (hive-schemas.test, hive-test.properties). + +Run each with the freshly built binary: + + B=zig-out/bin/cljw + $B .dev/probes/testcheck/