Skip to content

Build whole models from mathspec (whole-model-only) - #958

Merged
FabianHofmann merged 17 commits into
masterfrom
spec-whole-model
Oct 28, 2026
Merged

FabianHofmann merged 17 commits into
masterfrom
spec-whole-model

Conversation

@FabianHofmann

@FabianHofmann FabianHofmann commented Sep 18, 2026 •

Copy link
Copy Markdown
Collaborator

Note

The following content was generated by AI.

Changes proposed in this Pull Request

A mathspec spec (a YAML declaration of a linear model) builds a whole linopy model from linopy/spec/. mathspec (>=0.4,<0.5, from PyPI, Python >= 3.12) is the optional extra linopy[spec] and imported lazily, so import linopy never pulls it in.

This branch was re-ported onto master (after #1041, #1042, #1051–#1059), migrated to mathspec 0.4 and leaned down. The older 67-commit history is kept under the tag spec-whole-model-v1. The simplification from #1068 is merged into this branch.

API

  • One constructor. Model.from_spec(spec, sources, **model_kwargs) builds variables, SOS, constraints, the objective and named expressions with a variable term. spec is a Path, YAML text, dict or mathspec.Spec. A one-line str ending in .yaml/.yml raises a ValueError that asks for a Path.
  • Data lives in model.parameters. Every parameter (unfilled, NaN at holes) and every relation is stored under its declared name. assign_coords, copy and the netcdf round trip therefore carry the spec data. There is no model.spec accessor and no stamp on variables or constraints.
  • Results are functions of (spec, model). linopy.spec.evaluate(spec, model, *names) folds named expressions over the solution and the parameters. linopy.spec.parameter(spec, model, name) returns filled values with the declared dtype. linopy.spec.__all__ is SpecDataError, evaluate, parameter.
  • Typesetting is mathspec's own: mathspec.typeset(spec, fmt) and typeset_line.
  • Rebuild from a saved model. Model.from_spec(path, read_netcdf(f).parameters) rebuilds without the original sources.
  • linopy.testing.synthetic_sources takes every spec form from_spec takes.

Sources

  • NaN in a DataArray, Dataset or wide DataFrame is a missing row under the missing: policy. Long tables still refuse NaN; a long table may carry its values in a column named after the parameter.
  • Sets, extension arrays, Categorical and masked indexes work as dimension sources. Each source key is read once, so a lazy Mapping runs one query per parameter.
  • Labels that differ only by type get a "differ in type, not spelling" error with a datetime hint.

mathspec 0.4 semantics

  • Relations are joins: sum(x, over=d, by=rel[c]), at(x, by=rel[c]), sums over several axes. Grouped sums on terms use master's Relation group-by.
  • missing: per parameter and relation (refused default, neutral, a literal, absent). Absence travels as a mask, never as NaN.
  • missing: refused only raises for holes a live row demands. A hole is excused under a where: mask, a masked variable, an absent row of another parameter, an unfilled shift edge or a failed lookup. Build and evaluate share one check that names the parameter, the declaration and the missing labels.
  • ordered:, conditions: and the dual(c) sign convention are supported.

Changes for all models

  • Object-dtype parameters are stored in netcdf as codes plus categories, so NaN holes round-trip instead of coming back as ''.
  • Mixed-type object parameters (for example ['a', 1]) raise on write instead of being written as strings.
  • read_netcdf keeps parameter coordinates that no data variable uses.

Limits

Docs, CI, benchmarks

  • New page doc/spec-data.rst (source contract, SQLite example, holes, rebuild, variants via mathspec). Notebook examples/building-models-from-specs.ipynb, doc/api.rst, user guide and release notes.
  • CI clones the mathspec examples at the installed version and sweeps operators/, ports/ and the top level. On Python 3.11, where mathspec is not installable, the spec tests skip.
  • Benchmark benchmarks/models/spec_commitment.py (commitment.yaml plus a sum_by-shaped relation case). codspeed and benchmark-smoke install no spec extra, so these benchmarks skip in CI.
API review findings deferred to issues

Updating data and re-solving (editing model.parameters changes evaluate but not the model); applying a spec to an existing model and composing programs; bare relations and sets derived from data; sparse data densified; data-only named expressions not in model.expressions; lazy per-parameter reads.

Local verification
uv run ruff check . && uv run ruff format --check .     # pass
uv run mypy linopy test                                 # pass
uv run pytest -q --doctest-modules test -k spec         # 405 passed, 15 skipped
# Python 3.11 without mathspec:
uv run --python 3.11 --extra dev pytest --collect-only --doctest-modules linopy test   # 5159 collected

Checklist

  • AI-generated content is marked (see AGENTS.md).
  • Code changes are sufficiently documented; i.e. new functions contain docstrings and further explanations may be given in doc.
  • Unit tests for new features were added (if applicable).
  • A note for the release notes doc/release_notes.rst of the upcoming release is included.
  • I consent to the release of this PR's code under the MIT license.

@codspeed

codspeed Bot commented Sep 18, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 181 untouched benchmarks
⏩ 181 skipped benchmarks1


Comparing spec-whole-model (de6530f) with master (9ad435d)

Open in CodSpeed

Footnotes

  1. 181 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@github-actions

github-actions Bot commented Sep 18, 2026 •

Copy link
Copy Markdown

Build cost — v1 vs legacy

v1 build peak & time relative to legacy, on this commit — not a comparison against master (that is CodSpeed).

peak — v1 / legacy time — v1 / legacy
peak v1/legacy time v1/legacy
Full table (time + peak, mean)
benchmarks/drivers/test_build.py::test_build[basic-n=10]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.09022 (1.10)   │   15.03 (1.00) 
 (v1)        0.08229 (1.0)   │    15.00 (1.0) 

benchmarks/drivers/test_build.py::test_build[basic-n=250]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.09851 (1.12)   │   12.04 (1.00) 
 (v1)        0.08763 (1.0)   │    12.04 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=0]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.03844 (1.07)   │   15.20 (1.0) 
 (v1)        0.03598 (1.0)   │   15.20 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=100]
                  time (s)        peak (MiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.05552 (1.06)   │   44.93 (1.0) 
 (v1)        0.05256 (1.0)   │   44.93 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=50]
                  time (s)        peak (MiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.04195 (1.04)   │   11.51 (1.0) 
 (v1)        0.04034 (1.0)   │   11.51 (1.0) 

benchmarks/drivers/test_build.py::test_build[expression_arithmetic-n=10]
                 time (s)         peak (KiB) 
 name                mean   │           mean 
─────────────────────────────────────────────
 (legacy)   0.1001 (1.06)   │   24.34 (1.06) 
 (v1)       0.09443 (1.0)   │    23.04 (1.0) 

benchmarks/drivers/test_build.py::test_build[expression_arithmetic-n=250]
                 time (s)         peak (MiB) 
 name                mean   │           mean 
─────────────────────────────────────────────
 (legacy)   0.1211 (1.17)   │   16.12 (1.00) 
 (v1)        0.1033 (1.0)   │    16.12 (1.0) 

benchmarks/drivers/test_build.py::test_build[knapsack-n=10000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.02413 (1.07)   │   752.18 (1.10) 
 (v1)         0.0225 (1.0)   │    685.15 (1.0) 

benchmarks/drivers/test_build.py::test_build[knapsack-n=100]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.02414 (1.08)   │   3.12 (1.33) 
 (v1)        0.02234 (1.0)   │    2.34 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=0]
                  time (s)          peak (MiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.06437 (1.21)   │   126.16 (1.44) 
 (v1)        0.05337 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=100]
                  time (s)          peak (MiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.06332 (1.21)   │   126.16 (1.44) 
 (v1)        0.05235 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=50]
                  time (s)          peak (MiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.06326 (1.20)   │   126.16 (1.44) 
 (v1)        0.05269 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[masked-n=100]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.05572 (1.02)   │    715.12 (1.0) 
 (v1)        0.05456 (1.0)   │   787.73 (1.10) 

benchmarks/drivers/test_build.py::test_build[masked-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.05359 (1.08)   │   4.54 (1.27) 
 (v1)        0.04942 (1.0)   │    3.57 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=0]
                 time (s)          peak (KiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.3762 (1.05)   │   704.12 (1.09) 
 (v1)        0.3572 (1.0)   │    643.85 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=100]
                 time (s)        peak (MiB) 
 name                mean   │          mean 
────────────────────────────────────────────
 (legacy)   0.3941 (1.04)   │   18.34 (1.0) 
 (v1)        0.3798 (1.0)   │   18.34 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=50]
               time (s)       peak (MiB) 
 name              mean   │         mean 
─────────────────────────────────────────
 (legacy)   0.39 (1.04)   │   9.54 (1.0) 
 (v1)       0.374 (1.0)   │   9.54 (1.0) 

benchmarks/drivers/test_build.py::test_build[milp-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.07643 (1.07)   │   3.77 (1.12) 
 (v1)        0.07111 (1.0)   │    3.37 (1.0) 

benchmarks/drivers/test_build.py::test_build[milp-n=50]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.07651 (1.09)   │   216.59 (1.10) 
 (v1)        0.07003 (1.0)   │    196.23 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=0]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.03872 (1.07)   │   938.49 (1.0) 
 (v1)        0.03603 (1.0)   │   938.49 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=100]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.03975 (1.09)   │   9.66 (1.0) 
 (v1)        0.03659 (1.0)   │   9.66 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=50]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.03942 (1.09)   │   5.32 (1.0) 
 (v1)         0.0362 (1.0)   │   5.32 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=0]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.01952 (1.00)   │   1.28 (1.0) 
 (v1)        0.01944 (1.0)   │   1.28 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=100]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.01947 (1.01)   │   1.28 (1.0) 
 (v1)         0.0193 (1.0)   │   1.28 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=50]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.01948 (1.02)   │   1.28 (1.0) 
 (v1)        0.01919 (1.0)   │   1.28 (1.0) 

benchmarks/drivers/test_build.py::test_build[piecewise-n=1000]
                 time (s)          peak (KiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.1884 (1.02)   │   949.04 (1.06) 
 (v1)        0.1851 (1.0)   │    893.73 (1.0) 

benchmarks/drivers/test_build.py::test_build[piecewise-n=10]
                 time (s)         peak (KiB) 
 name                mean   │           mean 
─────────────────────────────────────────────
 (legacy)   0.1863 (1.05)   │   10.03 (1.07) 
 (v1)        0.1782 (1.0)   │     9.38 (1.0) 

benchmarks/drivers/test_build.py::test_build[qp-n=1000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.04827 (1.08)   │   147.70 (1.06) 
 (v1)        0.04489 (1.0)   │    139.87 (1.0) 

benchmarks/drivers/test_build.py::test_build[qp-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.04751 (1.07)   │   2.60 (1.09) 
 (v1)        0.04431 (1.0)   │    2.38 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=0]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.03976 (1.09)   │   696.75 (1.03) 
 (v1)        0.03637 (1.0)   │    673.70 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=100]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)    0.08343 (1.0)   │   137.97 (1.0) 
 (v1)       0.08367 (1.00)   │   137.97 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=50]
                  time (s)        peak (MiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.06422 (1.12)   │   69.22 (1.0) 
 (v1)        0.05726 (1.0)   │   69.22 (1.0) 

benchmarks/drivers/test_build.py::test_build[sos-n=1000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.04559 (1.10)   │   402.33 (1.00) 
 (v1)        0.04161 (1.0)   │    402.30 (1.0) 

benchmarks/drivers/test_build.py::test_build[sos-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.04495 (1.10)   │   3.19 (1.19) 
 (v1)        0.04083 (1.0)   │    2.69 (1.0) 

benchmarks/drivers/test_build.py::test_build[sparse_network-n=10]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.05029 (1.07)   │   29.00 (1.54) 
 (v1)        0.04702 (1.0)   │    18.84 (1.0) 

benchmarks/drivers/test_build.py::test_build[sparse_network-n=250]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.06051 (1.10)   │   37.95 (1.43) 
 (v1)        0.05504 (1.0)   │    26.51 (1.0) 

benchmarks/drivers/test_build.py::test_build[storage-n=10]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)    0.09397 (1.0)   │    410.93 (1.0) 
 (v1)       0.09719 (1.03)   │   427.84 (1.04) 

benchmarks/drivers/test_build.py::test_build[storage-n=250]
                 time (s)         peak (MiB) 
 name                mean   │           mean 
─────────────────────────────────────────────
 (legacy)    0.1014 (1.0)   │     9.94 (1.0) 
 (v1)       0.1038 (1.02)   │   10.22 (1.03) 

📊 Interactive plots + CSV: download the semantics-report-v1-vs-legacy artifact from this run.

Report-only · not a gate · refreshed on every push · obsolete once legacy is dropped.

@FabianHofmann FabianHofmann added enhancement New feature or request math-spec YAML/text math spec: schema, from_spec/add_spec, operators, expr container labels Sep 18, 2026
@FabianHofmann
FabianHofmann marked this pull request as ready for review September 18, 2026 07:45
@FabianHofmann FabianHofmann changed the title Build whole models from math-spec (whole-model-only) Build whole models from mathspec (whole-model-only) Sep 28, 2026
…pec 0.4

Core hooks (spec stamps, add_spec/from_spec, netcdf spec block, expression
stamps) in master's shape; linopy.spec rewritten for mathspec 0.4 nodes,
joins, missing:, ordered:, conditions:. Phase 4 fixes 28 spec tests.
Fold duplicated helpers (prefixing, netcdf filters, lookups, zeros, gathers,
as_expression), drop dead checks and single-use wrappers; align only dense
input. Test string parameters given as rows.
…y handles, KeyError for unknown names

Fold the binary-operator branches of evaluate into one table, drop the
redundant source-key cache and guards in attach, and build the row
presence mask with Index.isin.
Keep every parameter unfilled in ModelSpec.parameters and fill on read;
drop the .spec stamp, presence dataset, add_spec, lower_text and the
file-name sniffing of str specs. Public surface is ModelSpec,
NamedExpression, SpecDataError, Declaration and attach. Notebook updated.
from_spec writes raw parameters and relations into model.parameters;
linopy.spec.evaluate and linopy.spec.parameter replace ModelSpec.
Object parameters round-trip netcdf with NaN holes; grid sources accept
NaN as a missing row; sets, extension arrays and named value columns are
accepted; str file names are refused with a Path hint. Docs: spec-data page.
A hole in a refused parameter is excused where where: masks, a masked
variable, an absent row, an unfilled shift edge or a failed lookup make
the term absent anyway. build and evaluate share one check that names
the parameter, the declaration and the missing labels.
@FabianHofmann
FabianHofmann merged commit d2592b5 into master Oct 28, 2026
22 checks passed
@FabianHofmann
FabianHofmann deleted the spec-whole-model branch October 28, 2026 07:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request math-spec YAML/text math spec: schema, from_spec/add_spec, operators, expr container

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant