Skip to content

Count households once: sum the weight column unweighted - #767

Draft
PavelMakarchuk wants to merge 1 commit into
mainfrom
fix/households-weighted-sum
Draft

PavelMakarchuk wants to merge 1 commit into
mainfrom
fix/households-weighted-sum

Conversation

@PavelMakarchuk

Copy link
Copy Markdown
Collaborator

Fixes #766

Summary

budget.households was the sum of squared household weights.

build_budgetary_impact summed household_weight through _sum_output_variable, which calls data[variable].sum(). Here data is the household MicroDataFrame, which is weighted by household_weight, so .sum() on the weight column is Σ w × w. On the sparse US data that is about 10⁴ times the household count, on every run. For example, state/ny 2026 reports 135,597,233,840 households.

The fix:

  • _count_output_entities(simulation, entity, weight) counts entities with a plain sum of the weight column.
  • build_budgetary_impact uses it for households.
  • There is no aggregate fallback, because a weighted frame always holds its weight column, and the aggregate path would weight it again.

The other budget fields are not changed. They are meant to be weighted sums (Σ w × value).

Tests

  • New tests/test_simulation_output_budget.py builds the budget from real MicroDataFrames, as policyengine does. The existing builder test uses plain pd.DataFrames, which is why it never caught this.
  • The new test fails on main with assert 5.0 == 3.0, and it passes with the fix.
  • The same test checks that baseline_net_income, tax_revenue_impact and state_tax_revenue_impact stay weighted sums.
  • uv run pytest tests/ --ignore=tests/integration: 733 passed, 23 skipped.
  • ruff format --check and ruff check are clean on the changed files.
  • I did not run Pyright or the integration tests.

The test file is separate from test_simulation_output_builder.py so that it does not conflict with #765, which reorders that file's imports.

Downstream

policyengine-app-v2 shows the average change per household as -budget.budgetary_impact / budget.households, so that headline is about 10⁴ times too small today. Cached results keep the wrong count until they are recomputed.

🤖 Generated with Claude Code

budget.households summed household_weight through the household
MicroDataFrame, whose columns are already weighted by household_weight,
so it returned the sum of squared weights: about 10^4 times the count on
every run (state/ny 135.6bn). Count entities with a plain sum of the
weight column, and test the budget on a real MicroDataFrame.

Fixes #766

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

budget.households is the sum of squared household weights

1 participant