Use ACS-local data for US regional simulations

[anth-volk]

Fixes #717

Blocking upstream dependencies

Do not merge this PR or proceed with a deployment push until all three prerequisites are complete:

• PolicyEngine/policyengine-us#9462 is merged and released: the US model must apply source-declared SPM measurement scope, retain outside-population records for ordinary tax/benefit calculations, and avoid requesting SPM thresholds for those records.
• PolicyEngine/policyengine.py#519 is merged and released: poverty summaries and JSON must preserve missing SPM outcomes rather than treating excluded records as nonpoor.
• A new published, certified PolicyEngine.py bundle binds the compatible model/wrapper versions and a certified ACS-local dataset that supplies SPM scope declarations. This includes connecting Microcosm's merged #977 producer to dataset construction/export, verifying declarations for every requested simulation year, updating this PR's package pins and frozen lockfile, and passing the real ACS image smoke calculation.

The current 6.2.7 bundle is installable but does not satisfy these prerequisites. The real Utah ACS image smoke run fails with SPM_COMPOSITION_REQUIRED because group-quarters minors reach the SPM threshold calculator without a classified adult. This is a separate downstream issue from the previously fixed WIC input compatibility problem. Do not bypass the failure by inventing adult/head roles, excluding records from all tax/benefit calculations, or disabling the real ACS check.

Summary

• Read certified regional dataset mappings from the installed PolicyEngine.py bundle and resolve them to dataset identities, revisions and SHA-256 hashes.
• Select the certified ACS-local dataset for US state/DC and congressional-district requests, keeping the sparse national dataset for national and multi-region requests.
• Apply the selection consistently to gateway response metadata, direct calculations, budget windows and Stage 12 report inputs.
• Require the runtime dataset and Stage 12 input provenance to agree.
• Prepare regional years on demand. Keep deployment artifact precompute limited to the existing national dataset for 2025, 2026 and 2027; do not prebuild ACS-local files.
• Allocate 64 GiB to the US workers that can load ACS-local data; existing national segmented worker allocations are unchanged.
• Add real Utah ACS preparation and WIC/net-income calculation to the existing PR image check.

Published dependency and merge order

Rebased onto main at 789ae283c4bb753e4504ee99a6b41ed28e795690. Both executor dependency groups and the frozen lockfile retain main's published PolicyEngine 6.2.7, supplied by #746. There is no remaining dependency downgrade or pending-publication lockfile mismatch.

6.2.7 retains PolicyEngine/policyengine.py#552's certified regional dataset mappings and #566's temporary compatibility fix for missing WIC participation decisions on explicitly identified ACS people.

The certified regional source remains populace-us-2024-buildo-acs-local-767312d60-20260923T074941Z, with SHA-256 769756c31f3ca646d12c272511744dec04c0e68870c6946dd945fbba65b6a7ec.

Merge and successfully deploy this PR before PolicyEngine/policyengine-api#3868. Preserve the API's existing simulation-bundle compatibility requirement. #746's package upgrade alone does not supply this PR's regional metadata and Stage 12 integration.

Validation after rebase

• Installed the executor's frozen lockfile and confirmed PolicyEngine 6.2.7, Core 3.33.0, US 2.2.1 and UK 2.125.1.
• Focused release-bundle, Stage 12 bundle/runtime, US validation and UK weight-matrix tests: 95 passed.
• Gateway endpoint tests: 74 passed.
• Shared Stage 12 manifest contract tests: 15 passed.
• Simulation entry Stage 12 adapter tests: 18 passed.
• uv lock --check, source formatting checks and git diff --check: passed.
• The complete executor suite, real ACS Modal calculation, container builds and live deployments were not run manually for this update. Fresh PR CI must pass, including the real ACS check.

Deployment scope

No new environment variables, secrets, service accounts, databases or deployment targets are introduced. This update makes no manual infrastructure or traffic changes. The PR image check uses one ephemeral 8-CPU, 64-GiB Modal function with a hard 30-minute execution timeout; it does not write to production SQL or GCS. See docs/us-regional-dataset-pr-validation.md.

[anth-volk]

The WIC failures in the deployment after #733 are expected to be fixed by PolicyEngine.py 6.2.5, provided that release includes PolicyEngine/policyengine.py#566.

Failed run: https://github.com/PolicyEngine/policyengine-sim-api/actions/runs/37944413271

Failed integration job: https://github.com/PolicyEngine/policyengine-sim-api/actions/runs/37944413271/job/113873278852

The staging worker was running 6.2.3. The budget-window, US state-region, and specific-model integration tests all failed on Utah requests with:

ValueError: Cannot map stored 'would_claim_wic' onto 'takes_up_wic_if_eligible' for 2024: the stored column has missing values.

6.2.3 selects the certified ACS-local dataset for these US regional requests, but lacks #566's temporary handling of missing ACS WIC participation decisions. #566 fills only missing decisions on explicitly identified ACS people with True, preserves existing decisions, and leaves model eligibility unchanged.

This PR already requests 6.2.5 in both executor dependency groups, but its frozen lockfile still resolves 6.2.3. Once #566 is released as 6.2.5, regenerate and validate the lockfile, install the frozen dependencies, and rerun the real ACS validation and complete staging integration suite before production deployment. The current dependency mismatch is expected while that release is pending; a frozen install does not yet validate 6.2.5.