Skip to content

conformance: stripe the polyengine-deno leg across Deno Web Worker shards - #399

Merged
lannbot merged 1 commit into
mainfrom
polyengine-shard-fanout
Aug 23, 2026
Merged

conformance: stripe the polyengine-deno leg across Deno Web Worker shards#399
lannbot merged 1 commit into
mainfrom
polyengine-shard-fanout

Conversation

@lannbot

@lannbot lannbot commented Aug 23, 2026

Copy link
Copy Markdown
Contributor

Takes the shard item from #372 (the last unchecked leg-level lever): the polyengine-deno conformance leg — the matrix's only unstriped target and the conformance job's measured tail (263s shared-suite on the 4-vCPU runner, per run 32652990970) — now fans its case loop across Deno Web Worker shards using ct-runner 0.5.0's shard option (polymorph-components/polyengine#110, already pinned by the frozen import map; deno.lock untouched).

The topology is the one RunSuiteOptions.shard's doc comment documents: one runSuite call per worker with stripe i % count over census order; each shard emits its own envelope and terminator; the parent (run.ts) keeps one envelope — asserting every shard's is byte-identical, which also catches divergent artifact reads since the envelope embeds artifact-sha256 — reorders rows by the per-row census index emit now passes, writes the single terminator, and sums the per-shard counts. Execution moves to shard-worker.ts (translator + host-import wiring per worker); run.ts keeps CLI, suite specs, merge, and reporting. --shards N overrides the default (core count capped at 8); --shards 1 runs unsharded in a single worker. Containment is unchanged in kind: still one component instance per runSuite call (freshCases: false, the header's recorded trade), now bounded per shard stripe rather than per suite, and --fresh-cases still restores per-case containment.

The two suites keep running as concurrent processes (the #373 interim step); the stale "ct-runner has no shard option yet" justfile paragraph is rewritten to describe the shipped mechanism.

Verification:

  • Sharded (8) and unsharded outputs are row-identical including order for both suites after stripping the volatile duration-ms field (counts identical: 18756/33/0/302 shared, 686/84/0/2 signing); --shards 1 likewise.
  • just conformance-ct::all green end to end: 8 targets / 76,364 shared results and 7 targets / 2,316 signing results, 0 failing beyond declared expected-fail, 0 validation errors — the merged JSONL satisfies the aggregate's lockfile identity.
  • deno task check (run.ts, shard-worker.ts, browser entries) and just conformance-ct::polyengine-pin-check pass.

Local wall (18 cores, shards=8): shared 22.2s → 19.0s–10.7s depending on phase load, signing 3.0s → 1.0–2.6s; the local win is bounded by per-worker boot (translate + census enumeration). On the 4-vCPU runner the case work dominates (263s), so the expected tail is ~90–120s at 4 shards, per #372's estimate.

Part of #372 (the wizer adoption and cache-key items remain).

@lannbot
lannbot enabled auto-merge August 23, 2026 18:53
@lannbot
lannbot merged commit d756ee8 into main Aug 23, 2026
10 checks passed
@lannbot
lannbot deleted the polyengine-shard-fanout branch August 23, 2026 18:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants