Skip to content

Add execution receipt schema v1 for published results - #571

Draft
MaxGhenis wants to merge 6 commits into
mainfrom
w4/receipt-schema-v1
Draft

MaxGhenis wants to merge 6 commits into
mainfrom
w4/receipt-schema-v1

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #570

What this adds

policyengine.provenance.receipt.ExecutionReceipt (schema v1) is a strict, immutable, JSON round-trippable record of exactly what produced a UK result. It is the contract the UK Budget dashboard (W5) uses for per-measure receipts, and the scorecard (W6) binds to it. Both need it by 14 October.

Field Contents
wrapper, model, core Versions of policyengine, policyengine-uk and policyengine-core, read from the installed distributions. The schema fixes the names as consts. A wheel SHA-256 appears only with evidence: either wheel_paths= (the builder checks the wheel's members against its RECORD one to one, then the installed files against the wheel), or the installer's PEP 610 archive hash.
data Release, build, the build-time model version, dataset name, URI and revision, and the SHA-256 hashed from the bytes simulated.
certification_status, certificate certified only when the bundle certificate covers the installed model and core, those dataset bytes and name, and the running wrapper. Otherwise the receipt is uncertified. The certificate digest is RFC 8785 over data_releases.uk.
reform, baseline Both required. Each is current_law or a digest of {representation, payload}, never normalized across representations. A Scenario with a module-level simulation_modifier is bound by code_revision (see below).
fiscal_year uk_fiscal_year: year Y is the fiscal year starting 6 April Y (2027-28). It is a const in v1.
outputs_sha256, runtime An optional output digest, and the runtime context, which is part of no digest.

Also added:

  • provenance.canonical (RFC 8785);
  • provenance.wheels;
  • the builders build_execution_receipt and build_execution_receipt_for_microsimulation;
  • verify_certificate;
  • the JSON Schema shipped in the wheel, with x-policyengine-rules and x-policyengine-digests;
  • docs/receipts.md.

Invariants (each tested)

  1. No false certification. A certificate's model, core, dataset digest and wrapper equal what ran. The builder certifies only when the bundle covers all of them.
  2. Verification is exact. verify_certificate(receipt, bundle) rebuilds the certificate and the data identity from the bundle record, using the builder's own derivation. Any consistent forgery of a certified member is reported, even one that keeps the genuine record digest. Tests cover each of 16 single-field forgeries, Hypothesis-generated combinations, and a builder/verifier differential over bundle edits. certificate_source is the producer's statement, not evidence.
  3. Digest separation. The execution identity depends only on wrapper, model, core, data and certificate content. The receipt digest adds reform, baseline, years and outputs, never runtime context or labels (Hypothesis).
  4. Strict, canonical digests.
    • Nothing is stringified: sets, Decimal, arrays, NaN, datetimes, longdouble and deep nesting are refused with ReceiptError.
    • Member order and null-versus-omitted don't change a digest.
    • Output matches the rfc8785 package (differential) and the W6 golden fixtures byte for byte.
  5. A modifier digest names only code that ran.
    • The digest covers the function's source, its module's source and its defaults, plus code_revision with a full commit sha.
    • It is identical across checkout paths and on Python 3.11–3.14.
    • The module's source as read must compile to the loaded code for every module-scope function, and module-scope literal constants must still hold their values. Edits after import and constants reassigned at run time are refused.
    • Closures, lambdas, methods, partials and non-Scenario objects are refused.
  6. Wheel evidence is real. A junk zip carrying a copied RECORD fails. Only launchers named in entry_points.txt are exempt, and stray .pyc or INSTALLER files fail. The real policyengine-core 3.33.0 install verifies: 199 members and 199 files.
  7. Model and schema agree (differential): both accept every generated valid receipt and reject the same malformed ones.

Review history

Independent adversarial review by an Opus 5.5 agent; the Codex lanes were unavailable. Each round executed reproductions and mutation runs.

  • Round 1: 13 findings, fixed in 2d4cb9a.
  • Round 2: 9 findings, fixed in 7df93cf. These covered certificate forgery, path-dependent modifier digests, wheel launchers and contents, caller-supplied metadata, non-Scenario inputs, unbound defaults, longdouble and recursion, and schema strictness.
  • Round 3: all nine round-2 findings were confirmed fixed. Its four new items are fixed in 5ec9bbf: module edits after import, seven mutation gaps, refusal messages and exception types.
  • Round 4: in progress at 402900a.

Breaking against earlier heads (v1 is unreleased):

  • metadata_module= is removed;
  • release_manifest_sha256 is removed from data;
  • the fiscal convention is UK-only;
  • the package-role models are ModelIdentity and CoreIdentity.

Re-vendor from this head.

Checks run locally

  • 202 receipt, wheel and canonical tests pass; the full suite passes (1,450 passed at 7df93cf).
  • ruff (CI version) and mypy on the new modules are clean.
  • generate_receipt_schemas.py --check passes, and the schema output is identical on pydantic 2.10.6 and 2.14.0.

axiom: n/a: provenance/infrastructure contract, no policy change.

🤖 Generated with Claude Code

A receipt names exactly what produced a published result. It records the
installed wrapper, model and core versions; the data release and the
SHA-256 of the simulated dataset bytes; the certificate digest, attached
only when it covers that model and those bytes; reform and baseline digests
bound to their representation; and the fiscal-year convention. Digests use
RFC 8785 canonical JSON, so the dashboard and scorecard can recompute them.
The JSON Schema ships in the wheel.

Fixes #570

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis and others added 4 commits October 11, 2026 15:44
…ificate

Independent review of #571 found ways a receipt could claim more than
what ran. Reform digests no longer collide for Scenario.from_reform
closures, lambdas, partials or bound methods: those are refused, and a
module-level modifier is bound by source, bytecode and revision. Digests
are over strict JSON, so numpy scalars, sets, Decimals and colliding keys
are converted exactly or refused instead of stringified. canonical_json
writes numpy floats and int subclasses as their values. Certification now
requires the installed core version, records whether the certificate was
packaged or caller-supplied, and verify_certificate checks it against a
published manifest. Wheel hashes are recorded only when the builder
verified the installed files against the wheel RECORD. The execution
identity no longer depends on how the builder was called, reform and
baseline are required, models parse strictly, overlay datasets get
uncertified receipts, and v1 covers UK results.

Refs #570

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- verify_certificate rebuilds the certificate from the bundle record with
  the builder's own derivation and compares every identity member plus the
  whole data identity; a forged receipt that keeps the genuine record digest
  is now reported. The "or require certificate_source packaged" rule is gone.
- Modifier identity no longer digests marshalled bytecode or the Python
  version, so the digest is the same in any checkout. The source is
  recompiled and compared with the loaded code (ignoring file and line
  numbers), defaults and the module source are bound, and code_revision
  needs a full commit sha.
- Wheel verification hashes every archive member against the wheel's own
  RECORD (one to one) and exempts only the launchers an installer generates
  for the wheel's declared console/GUI scripts; stray .pyc and INSTALLER
  files outside their installer locations fail.
- The public metadata_module hook is removed; versions always come from
  importlib.metadata.
- scenario_reform_identity refuses anything that is not a Scenario.
- strict_json_value refuses numpy longdouble and deep nesting with
  ReceiptError; content_sha256 raises ReceiptError for unsafe integers.
- The schema fixes package names and the UK fiscal-year convention as
  consts, and drops the never-populated release_manifest_sha256.

Refs #570

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Before binding a modifier, the module's source as read now must match
  what was imported: every module-scope function in that file compiles to
  its loaded code, and every module-scope name bound once to a literal
  still holds that value. Edited helpers, edited constants and constants
  reassigned at run time are refused instead of named by the digest.
- Functions defined in module-level blocks bind (source is dedented before
  compiling); methods are refused with a clear reason.
- Unreadable wheels (not a zip, duplicate RECORD entries) raise
  ReceiptError from installed_package_identity.
- Tests now kill the round-3 surviving mutants: each forgery alone, the
  record-version and basis checks, operator-only and nested-code edits,
  forged function globals and duplicated wheel members.

Refs #570

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Wheel verification now checks bytecode: every __pycache__ file this
  interpreter would load for a verified source (any optimisation level,
  listed in RECORD or not, under sys.pycache_prefix) must compile from
  that source. Caches the interpreter would ignore are reported as inert.
  Launchers are exempt only when they exist in bin/Scripts outside the
  site directory.
- The microsimulation adapter requires the file to still hash to the
  SHA-256 verified at load time, and the receipt to name the dataset, URI,
  revision and package recorded then; unmanaged loads cannot be receipted.
- Data identity is resolved with the materializer's own resolver
  (overlays, per-dataset package and revision, release manifest revision),
  and a receipt never gives an artifact's name to other bytes.
- Modifier binding: helper defaults, block-scoped literals and
  comprehension scope are checked; every module function's defaults are
  bound with their types; foreign helper globals and conditionally
  defined functions with different code are refused.
- Only .whl direct-URL archive hashes are recorded as wheel_sha256.
- Masked, 0-d, complex, datetime and structured NumPy values are refused.
- Years must be integers and schema_version may not be a boolean.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Publish an execution receipt schema (v1) for published UK Budget results

1 participant