Repository navigation
Add the UK Budget reform contract runner - #577
Merged
Merged
Conversation
run_budget_reform_contract() runs dated Budget reforms through the policyengine-uk routes (reform dict, Scenario parameter_changes, Reform.from_dict) and the policyengine.py routes (calculate_household, Simulation(policy=...)). It checks each against expected values derived independently of the model: statute arithmetic for the indexed thresholds, and a day-weighted fiscal-year annualisation written here. It also checks that every reformed simulation's baseline equals an unreformed one, runs bounded Hypothesis sampling (annual values against the oracle, route agreement plus baseline isolation), and, given a certified dataset, runs population checks on a deterministic household subsample. The report is JSON for release qualification to gate on and receipt. Contract 1.0.0 expects prior law's projected path to resume after a bounded change. The statutory-resumption expectations for an extended freeze are recorded as pending under cos d1344. Fixes #576. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pters Caching values lets each simulation be released after its case (the previous run peaked at 5.8 GB). The household route's cache key could not be hashed, the policy route's in-memory dataset lacked a region, and a pending check whose oracle cannot be computed now stays pending instead of gating as an error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…al year The sampled-property oracle built a blended year from effective-date pieces, so a change that began before 6 April was overridden by prior law for the whole year. It now takes each day's value directly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nsion A reform can reach a parameter without reaching the award, as when an award is fixed as an input before the reform applies (UC standard allowance +10% scored £0 across several policyengine-uk releases for that reason). Two household cases check the award itself on every route: a claimant with no income is awarded the reformed standard allowance (£500 x 12), and a full-rate pensioner gets the reformed new State Pension (£280 x 52). Two population checks raise every UC standard allowance, and the basic and new State Pension rates, by 10% for 2027-28 and require the allowances and pensions to scale by exactly 10%, no award to fall and the totals to rise. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The household route summed a variable across every entity that carried it, so a benefit-unit award also reported under the household counted twice (universal_credit read £12,000 for a £6,000 award). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ences, modifier cases Review (GPT-6.1 Sol, review-runner-out.md): - The population April-rate check tests total 2027 income tax, not each person: in Scotland a higher UK basic rate can lower one person's tax through Gift Aid and the allowance taper (executed: -£3.60). - The mid-year CGT case checks capital gains tax (2027: 159/366 of a year at 20%), and the VAT case checks the reformed VAT, on every route. - Expected values read a copy of current law taken before any reform; every route builds its unreformed reference before the reform and a fresh one after it; a run-level row checks the shared parameters are unchanged. - Unknown or empty case and route selections raise; a run in which nothing passed has an error row. - Both property engines turn any exception into an error row, and every counterexample names the sampled change and income. - The routes-agree property compares capital gains tax and Universal Credit as well as income tax. Also adds W5's shape: one scenario or policy with a parameter-setting modifier and dated parameter changes, in either order. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…laimant The responsiveness checks need an award to move; with earners only they fail, as they should, and a test now says so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The VAT case reads vat for two checks, which doubled its isolation rows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…er cache Review round 2 (GPT-6.1 Sol, review-runner-r2-out.md): - The run-level parameters row passes whenever nothing leaked, so it masked the 'nothing passed' guard on any in-process selection; passes are now counted before it is added. - module_parameters_unchanged also compares policyengine-uk's cached processed tree (which every new simulation copies), cached statutory tree and cached projected timelines, where the installed version has them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review round 3 (GPT-6.1 Sol, review-runner-r3-out.md, APPROVE with P3s): - module_parameters_unchanged compares each parameter's whole timeline and metadata, not 1 January samples, so a leak on 6 April or after 2036, or into uprating metadata, shows. - The rebuild memo, which the run fills, is documented as unwatched. - A test covers the projected-timelines branch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #576
Draft: GPT-6.1 Sol requested eight changes in round 1. All are fixed (below), and round 2 is running. Merge #575 first, since the wrapper routes need its interval keys. The interface is stable for W4 (qualification) to import; their gate is #574.
What this adds
policyengine.tax_benefit_models.uk.budget_reform_contract.run_budget_reform_contract(*, routes=None, dataset_path=None, seed=0, max_examples=20, cases=None) -> dict. Contractpolicyengine-uk/budget-reform-contract, version 1.0.0.Routes
country.reform_dictpolicyengine_uk.Simulation(reform=dict)withstart.stopkeyscountry.scenario_parameter_changesSimulation(scenario=Scenario(parameter_changes=dict))country.reform_from_dictSimulation(reform=Reform.from_dict(dict))wrapper.householdpe.uk.calculate_household(reform=dict): lone date for a change held from a date,start.stopfor a bounded onewrapper.policySimulation(policy=compile_reform_to_policy(...))over an in-memory dataset of the case householdapi.sim_apiunsupportedin-process, with a reason. The sim API's own gate importsCASES.Cases
Each case's expected values come from statute or an annualisation written here, never from model output. Where an expectation needs current law (for example, the allowance a freeze holds back), it reads only current-law parameters.
april_rate_change: basic rate 21% from 6 April 2027. FY2027 gets the full rate; the income tax change for £30k is (30,000 − PA) × 1%.finite_rate_change: the same, for 2027-28 only. 2028-29 reverts.extended_freeze: PA and BRL frozen to 2033-34.pending(cos d1344).april_benefit_uprating: UC single 25+ £500 from April 2027 and £520 from April 2028, giving £6,000 / £6,240 / £6,240.bounded_change_on_uprated_benefit: £500 for 2027-28 only. Later years equal the baseline.mid_year_blended_rate: CGT basic rate 20% from 30 October 2027. FY2027 = (0.18×207 + 0.20×159)/366. For £10,000 of gains within the basic band, capital gains tax rises by (gains − exempt amount) × 2% × 159/366 in 2027 (£60.82) and by the full year's 2% in 2028 (£140).september_cpi_moves_indexed_thresholds: a 5% September 2030 CPI gives PA £13,200 and BRL £39,600 (s57(3), s21(3)). Country routes only.uc_rebalancing_reform_after_data_load: protected LCWRA £500/month gives £6,000.structural_switch: marriage allowance without its income condition gives £1,260 (s55B).baseline_vat_is_current_law(#2188):baseline_vatunder reform equals unreformedvat, and VAT rises by £10,000 × 2% / current law's microdata VAT coverage (£526.32), on every route.modifier_then_parameter_changesandparameter_changes_then_modifier: one scenario (country routes) or onePolicysum (wrapper.policy) carries both a modifier that sets the dividend basic rate to 8.75% on the annualised parameters and a basic rate of 21% from 6 April 2027. Income tax changes by (dividends − allowance) × (8.75% − current rate), plus the basic-rate uplift from 2027. On policyengine-uk 2.125.1 the modifier's rate reverted to 10.75%. On wrapper main, a policy carrying both lost its values; Route UK dated reforms through policyengine-uk's statutory pipeline #575 fixes that.unknown_path: every route rejects it.Reform.from_dictreads a bare year as onwards; a single-day dict key overrides one day; effective dates given latest first.unreformed_after_reformrows). Country routes also check the reformed simulation's ownbaseline. A run-levelmodule_parameters_unchangedrow compares every parameter, 2024–2036, with the copy.Properties (bounded Hypothesis sampling, no shrinking)
annual_values_match_the_oracle: random single changes on rate, uprated, indexed and blended parameters. The annual values match the independent annualisation on every country route.routes_agree_and_baselines_hold: random changes give identical household income tax, capital gains tax, UC standard allowance and UC award on every in-process route, and each route's baseline and later unreformed runs hold.failand a crash iserror, with the sampled change and income in the counterexample.Population checks (
dataset_path)These run on a deterministic subsample of 1,000 households:
Report
contract_id,contract_version,prior_law_option, installedversions,dataset(path, sha256),cases(one row per case, route and check, withstatusin pass / fail / error / unsupported (withreason) / pending (withdecision_id)),properties,population,summaryandholds(no fail and no error). A run in which nothing passed gets anerrorrow, and unknown or empty case or route selections raiseValueError.The contract lives here, not in policyengine-uk, so a candidate model release never grades itself (agreed with W4).
Measured runs (this machine under heavy load)
After the review fixes (12 Oct, 71fba3c + #575 + policyengine-uk#2305 at 603fecbb2): the full qualification run, with every route, every case, seed 0, 20 examples and population checks on the certified enhanced_frs_2024_25 (sha256 5ba399ea…). Holds: 682 pass, 0 fail, 0 error, 6 pending (d1344), 79 unsupported with reasons. Both properties passed on 20 examples, and 19 of 19 population checks passed. 24.7 minutes, peak RSS 3.0 GB. 8718f37 then removed a few duplicate VAT isolation rows.
Earlier runs, before the review fixes (11 Oct):
On the pinned model the dated cases fail, as they should; those failures are the defects the companion PRs fix. The 6 pending rows quantify the open d1344 choice: in 2034, prior law's projected path gives PA £13,630 and BRL £41,000; resuming indexation from the frozen amounts (ITA 2007 s57/s21) would give £12,830 and £38,500. A full qualification run (all routes, 20 examples, certified dataset) takes about 20 minutes here. Reports are in
~/reviews/w1-budget-reform-contract-2026-10-11/contract-report-*.json.Tests
tests/test_uk_budget_reform_contract.pycovers:Since the review, the tests also cover selection validation, fresh-after isolation, the reference copy, both property engines' fail and error rows, the parameter modifier, the modifier cases, and the population checks with and without award recipients.
Locally: 28 passed, 3 skipped on the pinned model. All 31 pass with policyengine-uk#2305 and #575 installed.
🤖 Generated with Claude Code