diff --git a/README.md b/README.md
index b7aada0..3ea4bbf 100644
--- a/README.md
+++ b/README.md
@@ -17,13 +17,13 @@ tag. They depend on each other by version, so install the ones you use
**and the packages they depend on** from the same tag:
```bash
-pip install "lambdafeedback-slr-parsing @ git+https://github.com/lambda-feedback/lambda-feedback-packages.git@v0.3.0-rc.2#subdirectory=slr_parsing"
-pip install "lambdafeedback-criteria @ git+https://github.com/lambda-feedback/lambda-feedback-packages.git@v0.3.0-rc.2#subdirectory=criteria"
-pip install "lambdafeedback-expression-parsing @ git+https://github.com/lambda-feedback/lambda-feedback-packages.git@v0.3.0-rc.2#subdirectory=expression_parsing"
-pip install "lambdafeedback-units @ git+https://github.com/lambda-feedback/lambda-feedback-packages.git@v0.3.0-rc.2#subdirectory=units"
+pip install "lambdafeedback-slr-parsing @ git+https://github.com/lambda-feedback/lambda-feedback-packages.git@v0.3.0-rc.3#subdirectory=slr_parsing"
+pip install "lambdafeedback-criteria @ git+https://github.com/lambda-feedback/lambda-feedback-packages.git@v0.3.0-rc.3#subdirectory=criteria"
+pip install "lambdafeedback-expression-parsing @ git+https://github.com/lambda-feedback/lambda-feedback-packages.git@v0.3.0-rc.3#subdirectory=expression_parsing"
+pip install "lambdafeedback-units @ git+https://github.com/lambda-feedback/lambda-feedback-packages.git@v0.3.0-rc.3#subdirectory=units"
```
-With Poetry: `lambdafeedback-units = { git = "https://github.com/lambda-feedback/lambda-feedback-packages.git", tag = "v0.3.0-rc.2", subdirectory = "units" }`, and likewise for each package.
+With Poetry: `lambdafeedback-units = { git = "https://github.com/lambda-feedback/lambda-feedback-packages.git", tag = "v0.3.0-rc.3", subdirectory = "units" }`, and likewise for each package.
## Packages
diff --git a/criteria/CHANGELOG.md b/criteria/CHANGELOG.md
index 08c20fb..4cdd184 100644
--- a/criteria/CHANGELOG.md
+++ b/criteria/CHANGELOG.md
@@ -1,6 +1,6 @@
# Changelog: lambdafeedback-criteria
-## 0.3.0 (unreleased; pre-release 0.3.0rc2, tag v0.3.0-rc.2)
+## 0.3.0 (unreleased; pre-release 0.3.0rc3, tag v0.3.0-rc.3)
The first public release, after the extraction refactor. The import path is now `lambdafeedback.criteria` (was `criteria`). There was no 0.2.0 release: all four `lambdafeedback-*` packages share one version, and the first release is 0.3.0. Supports Python 3.12 to 3.14 (3.11 is not supported); `__version__` gives the installed version.
@@ -63,7 +63,7 @@ Behaviour differences to be aware of when adopting:
### Added
- `build_quantity_criteria_parser(reserved_expressions, token_list=, productions=)` parses criteria on physical quantities (`response matches answer`, `dimension(response) matches dimension(answer)`, and `=`, `<=`, `>=`, `<`, `>` between quantities). Its defaults are `quantity_grammar.QUANTITY_TOKENS` / `QUANTITY_PRODUCTIONS`. It was `generate_criteria_parser` in compareExpressions' physical-quantity context, and its parse trees are unchanged. It takes the same arguments as `build_criteria_parser`, so either can fill the same slot. `reserved_expressions` isn't used yet: only `response` and `answer` scan as quantities.
-- `split_criteria(criteria)` splits a string of criteria on the commas outside `()`, `[]` and `{}`, and strips each criterion. It was `split_criteria` in compareExpressions' `evaluation.py`. Unlike that version, it raises `CriteriaSyntaxError` (new, a `CriteriaError` and `ValueError`) when a bracket is closed before it is opened, or left open. The old version stopped splitting at the first such bracket instead.
+- `split_criteria(criteria)` splits a string of criteria on the commas outside `()`, `[]` and `{}`, and strips each criterion. It was `split_criteria` in compareExpressions' `evaluation.py`. Any closing bracket closes any opening one, so a half-open interval keeps its comma: `response in [0, 1), response > 0` is two criteria. The old version counted each kind of bracket on its own and stopped splitting after `[0, 1)`, running the rest into one criterion. Unlike that version, it raises `CriteriaSyntaxError` (new, a `CriteriaError` and `ValueError`) when a bracket is closed when none is open, or left open. The old version stopped splitting at the first such bracket instead. It also raises `CriteriaSyntaxError` for an empty criterion (`response = answer,`, `a = b,, c = d` or an empty string), which the old version returned as `""` for the criteria parser to fail on.
- `where_substitutions(substitutions)` gives `[(name, expression), ...]` for the substitution part of a `where` criterion (the second child of a `WHERE` node), with the expressions as text. It replaces the node-reading half of compareExpressions' `parse_where_substitutions`; the caller now parses the expressions, so this package doesn't need `expression_parsing`. The grammar also accepts other relations after `where` (`where x > 2`, `where x proportional to y`). compareExpressions' `where_substitution_nodes` returned the node itself for these, instead of a list, and this function raises `CriteriaSyntaxError` for them. A `SEPARATOR` that mixes in such a relation (`where a = 2; x > 3`) used to be read as substitutions, and also raises now.
### Changed
diff --git a/criteria/README.md b/criteria/README.md
index 85f29a2..a809222 100644
--- a/criteria/README.md
+++ b/criteria/README.md
@@ -14,7 +14,7 @@ evaluation function.
From git (not on PyPI yet):
```bash
-pip install "lambdafeedback-criteria @ git+https://github.com/lambda-feedback/lambda-feedback-packages.git@v0.3.0-rc.2#subdirectory=criteria"
+pip install "lambdafeedback-criteria @ git+https://github.com/lambda-feedback/lambda-feedback-packages.git@v0.3.0-rc.3#subdirectory=criteria"
```
Install the packages it depends on (`slr_parsing`) from the same tag; see the [repository README](../README.md#install).
diff --git a/criteria/pyproject.toml b/criteria/pyproject.toml
index b248e3c..a4f4e2c 100644
--- a/criteria/pyproject.toml
+++ b/criteria/pyproject.toml
@@ -4,7 +4,7 @@ build-backend = "poetry.core.masonry.api"
[project]
name = "lambdafeedback-criteria"
-version = "0.3.0rc2"
+version = "0.3.0rc3"
description = "Criteria parser and criteria graphs that turn evaluations into feedback, for Lambda Feedback evaluation functions."
readme = "README.md"
license = "MIT"
@@ -25,7 +25,7 @@ classifiers = [
]
requires-python = ">=3.12"
dependencies = [
- "lambdafeedback-slr-parsing (>=0.3.0rc2,<0.4.0)",
+ "lambdafeedback-slr-parsing (>=0.3.0rc3,<0.4.0)",
]
[project.urls]
diff --git a/criteria/src/lambdafeedback/criteria/nodes.py b/criteria/src/lambdafeedback/criteria/nodes.py
index e947556..08f045d 100644
--- a/criteria/src/lambdafeedback/criteria/nodes.py
+++ b/criteria/src/lambdafeedback/criteria/nodes.py
@@ -27,11 +27,6 @@
NodeKind: TypeAlias = Literal["evaluation", "criterion", "output"]
-# Mermaid node shapes (opening, closing).
-EVALUATION_STYLE = ("([", "])")
-CRITERION_STYLE = ("[", "]")
-OUTPUT_STYLE = ("{{", "}}")
-
class ResultLike(Protocol):
"""The part of a result object ``CriteriaGraph.export_feedback`` needs (``lf_toolkit.evaluation.Result`` fits)."""
@@ -91,7 +86,7 @@ class EvaluationNode(Node):
"""
KIND: ClassVar[NodeKind | None] = "evaluation"
- STYLE: ClassVar[tuple[str, str] | None] = EVALUATION_STYLE
+ STYLE: ClassVar[tuple[str, str] | None] = ("([", "])")
evaluate: Evaluate | None = None
replacement: str | None = None
@@ -119,7 +114,7 @@ class CriterionNode(Node):
"""An outcome of an evaluation, with the generator for its feedback text."""
KIND: ClassVar[NodeKind | None] = "criterion"
- STYLE: ClassVar[tuple[str, str] | None] = CRITERION_STYLE
+ STYLE: ClassVar[tuple[str, str] | None] = ("[", "]")
feedback_string_generator: FeedbackStringGenerator = no_feedback
@@ -139,7 +134,7 @@ class OutputNode(Node):
"""A terminal node; output nodes have no outgoing edges."""
KIND: ClassVar[NodeKind | None] = "output"
- STYLE: ClassVar[tuple[str, str] | None] = OUTPUT_STYLE
+ STYLE: ClassVar[tuple[str, str] | None] = ("{{", "}}")
def copy_into(self, graph: CriteriaGraph) -> OutputNode:
return graph.add_output_node(self.label, self.summary, self.details)
diff --git a/criteria/src/lambdafeedback/criteria/render.py b/criteria/src/lambdafeedback/criteria/render.py
index 200eae7..3dccf6c 100644
--- a/criteria/src/lambdafeedback/criteria/render.py
+++ b/criteria/src/lambdafeedback/criteria/render.py
@@ -6,7 +6,8 @@
from collections.abc import Collection, Mapping
from typing import TYPE_CHECKING, Any
-from .nodes import CRITERION_STYLE, EVALUATION_STYLE, OUTPUT_STYLE, EvaluationNode, Node
+from .errors import CriteriaGraphError
+from .nodes import EvaluationNode, Node
if TYPE_CHECKING:
from .graph import CriteriaGraph
@@ -52,20 +53,24 @@ def entry(node: Node, outgoing: bool = True) -> dict[str, Any]:
)
+def _style(node: Node) -> tuple[str, str]:
+ """``node``'s mermaid shape (opening, closing): its kind's ``STYLE``."""
+ if node.STYLE is None:
+ raise CriteriaGraphError(f"Cannot display node {node.label} of type {type(node).__name__}.")
+ return node.STYLE
+
+
def graph_to_mermaid(graph: CriteriaGraph) -> str:
"""Mermaid flowchart source for ``graph``; sufficiencies are drawn as dotted edges."""
- node_groups: list[tuple[Mapping[str, Node], tuple[str, str]]] = [
- (graph.evaluations, EVALUATION_STYLE),
- (graph.criteria, CRITERION_STYLE),
- (graph.outputs, OUTPUT_STYLE),
- ]
- keys = {label: f"N_{g}_{i}" for g, (nodes, _) in enumerate(node_groups) for i, label in enumerate(nodes)}
+ node_groups: list[Mapping[str, Node]] = [graph.evaluations, graph.criteria, graph.outputs]
+ keys = {label: f"N_{g}_{i}" for g, nodes in enumerate(node_groups) for i, label in enumerate(nodes)}
lines = ["flowchart TD"]
# Ordered de-duplication (dict keys), so the output doesn't depend on the hash seed.
edges: dict[tuple[str, str], None] = {}
sufficiencies: dict[tuple[str, str], None] = {}
- for nodes, (opening, closing) in node_groups:
+ for nodes in node_groups:
for label, node in nodes.items():
+ opening, closing = _style(node)
lines.append(f'{keys[label]}{opening}"{_text(label)}
---
{_text(node.details)}"{closing}')
edges.update(
dict.fromkeys((keys[e.source.label], keys[e.target.label]) for e in node.outgoing + node.incoming)
diff --git a/criteria/src/lambdafeedback/criteria/split.py b/criteria/src/lambdafeedback/criteria/split.py
index 5025230..2cde173 100644
--- a/criteria/src/lambdafeedback/criteria/split.py
+++ b/criteria/src/lambdafeedback/criteria/split.py
@@ -4,32 +4,40 @@
from .errors import CriteriaSyntaxError
-_OPENING = {")": "(", "]": "[", "}": "{"}
-"""The opening bracket for each closing bracket."""
+_OPENING = frozenset("([{")
+_CLOSING = frozenset(")]}")
def split_criteria(criteria: str) -> list[str]:
"""``criteria`` split on the commas outside brackets, with each criterion stripped.
- Each kind of bracket (``()``, ``[]``, ``{}``) is counted on its own. Raises
- :class:`CriteriaSyntaxError` if a bracket is closed before it is opened,
- or left open.
+ Any closing bracket (``)``, ``]``, ``}``) closes any opening one, so
+ half-open intervals such as ``x in [0, 1)`` keep their comma. Raises
+ :class:`CriteriaSyntaxError` if a bracket is closed when none is open,
+ or left open, or if a criterion is empty (``a = b,`` or ``a = b,, c = d``).
"""
- depth = dict.fromkeys(_OPENING.values(), 0)
+ depth = 0
parts = []
start = 0
for index, char in enumerate(criteria):
- if char in depth:
- depth[char] += 1
- elif char in _OPENING:
- opening = _OPENING[char]
- if depth[opening] == 0:
- raise CriteriaSyntaxError(f"`{char}` at position {index} closes no `{opening}` in: {criteria}")
- depth[opening] -= 1
- elif char == "," and not any(depth.values()):
- parts.append(criteria[start:index].strip())
+ if char in _OPENING:
+ depth += 1
+ elif char in _CLOSING:
+ if depth == 0:
+ raise CriteriaSyntaxError(f"`{char}` at position {index} closes no bracket in: {criteria}")
+ depth -= 1
+ elif char == "," and depth == 0:
+ parts.append(_criterion(criteria, start, index))
start = index + 1
- if any(depth.values()):
+ if depth:
raise CriteriaSyntaxError(f"Unclosed bracket in: {criteria}")
- parts.append(criteria[start:].strip())
+ parts.append(_criterion(criteria, start, len(criteria)))
return parts
+
+
+def _criterion(criteria: str, start: int, end: int) -> str:
+ """``criteria[start:end]``, stripped; raises :class:`CriteriaSyntaxError` if that leaves nothing."""
+ criterion = criteria[start:end].strip()
+ if not criterion:
+ raise CriteriaSyntaxError(f"Empty criterion at position {start} in: {criteria}")
+ return criterion
diff --git a/criteria/tests/test_criteria.py b/criteria/tests/test_criteria.py
index e6daf20..cd5eed1 100644
--- a/criteria/tests/test_criteria.py
+++ b/criteria/tests/test_criteria.py
@@ -92,15 +92,23 @@ class TestSplitCriteria:
("response = f(x, y), response > 0", ["response = f(x, y)", "response > 0"]),
("response = [1, 2], response = {a, b}", ["response = [1, 2]", "response = {a, b}"]),
("response = f([x, {y, z}]), answer > 0", ["response = f([x, {y, z}])", "answer > 0"]),
- ("response = answer,", ["response = answer", ""]),
- ("", [""]),
+ ("response in [0, 1), response > 0", ["response in [0, 1)", "response > 0"]),
+ ("response in (0, 1], answer in (a, b]", ["response in (0, 1]", "answer in (a, b]"]),
],
)
def test_splits_on_commas_outside_brackets(self, criteria, expected):
assert split_criteria(criteria) == expected
@pytest.mark.parametrize(
- "criteria", ["response = x), answer > 0", "response = (x, answer > 0", "response = (x], y"]
+ "criteria",
+ ["", " ", "response = answer,", "response = answer, ", ", response > 0", "response = answer,, x > 0"],
+ )
+ def test_empty_criterion_raises(self, criteria):
+ with pytest.raises(CriteriaSyntaxError, match="Empty criterion"):
+ split_criteria(criteria)
+
+ @pytest.mark.parametrize(
+ "criteria", ["response = x), answer > 0", "response = (x, answer > 0", "response = [0, 1)), y", "f(x"]
)
def test_unbalanced_brackets_raise(self, criteria):
with pytest.raises(CriteriaSyntaxError):
@@ -149,6 +157,16 @@ def test_mermaid_returns_string(self):
g.add_evaluation_node("EVAL", "eval", "eval details")
assert isinstance(g.to_mermaid(), str)
+ def test_mermaid_draws_each_kind_of_node_in_its_shape(self):
+ g = CriteriaGraph("root")
+ g.add_evaluation_node("EVAL", "eval", "eval details")
+ g.add_criterion_node("CRIT", "crit", "crit details")
+ g.add_output_node("OUT", "out", "out details")
+ lines = g.to_mermaid().split("\n\t")
+ assert 'N_0_0(["EVAL
---
eval details"])' in lines
+ assert 'N_1_0["CRIT
---
crit details"]' in lines
+ assert 'N_2_0{{"OUT
---
out details"}}' in lines
+
def test_duplicate_evaluation_node_rejected(self):
g = CriteriaGraph("root")
g.add_evaluation_node("EVAL", "eval", "eval details")
diff --git a/expression_parsing/CHANGELOG.md b/expression_parsing/CHANGELOG.md
index 7e963f6..e0515a9 100644
--- a/expression_parsing/CHANGELOG.md
+++ b/expression_parsing/CHANGELOG.md
@@ -1,6 +1,6 @@
# Changelog: lambdafeedback-expression-parsing
-## 0.3.0 (unreleased; pre-release 0.3.0rc2, tag v0.3.0-rc.2)
+## 0.3.0 (unreleased; pre-release 0.3.0rc3, tag v0.3.0-rc.3)
The first public release. Further tidy-up after the 0.2.0 extraction: previewing a response and mapping failures to learner-facing feedback text are app-layer concerns, so they move out of this package. `__version__` gives the installed version.
@@ -48,13 +48,14 @@ The first public release. Further tidy-up after the 0.2.0 extraction: previewing
### Security
- `MAX_INPUT_LENGTH` (10,000 characters): `parse_expression(s)` and `parse_latex` reject longer input. An expression nested beyond Python's recursion limit (a sum of about 800 terms) raises `ExpressionParsingError` saying it is too long or too deeply nested, instead of "maximum recursion depth exceeded".
-- **`simplify` is bounded.** With `simplify`, parsing evaluates, so `9**9**9`, `factorial(10**8)`, `primepi(10**13)` or `(x+1)**100000` never finished. Before evaluating, the unevaluated expression is checked: an exact number evaluation would build may have at most `MAX_NUMBER_DIGITS` (1000) digits, and a sum containing symbols may be raised to at most `MAX_SIMPLIFY_POWER` (500, exported); otherwise `ExpressionParsingError`. SymPy's symbolic algorithms aren't bounded, and the calculus helpers (`summation`, `product`, `integrate`, `limit`, `diff`) are evaluated in every mode, since compareExpressions relies on that: `summation(1/k**k, (k, 1, 10**6))` still never finishes. Run evaluation of untrusted input under a time limit.
+- **`simplify` is bounded.** With `simplify`, parsing evaluates, so `9**9**9`, `factorial(10**8)`, `primepi(10**13)` or `(x+1)**100000` never finished. Before evaluating, the unevaluated expression is checked: an exact number evaluation would build may have at most `MAX_NUMBER_DIGITS` (1000) digits, and a sum containing symbols may be raised to at most `MAX_SIMPLIFY_POWER` (500, exported); otherwise `ExpressionParsingError`. A `Product` with numeric limits may have at most `MAX_PRODUCT_TERMS` (1000, exported) factors, and its index counts as large as its largest bound (`product(k, (k, 1, 500))` is rejected, as `factorial(500)` is). SymPy's symbolic algorithms aren't bounded: a limit, a derivative or a product with symbolic or infinite limits can still be slow to simplify. Run evaluation of untrusted input under a time limit.
+- **`integrate`, `summation`, `product` and `limit` are unevaluated without `simplify`.** They were worked out while parsing in every mode (they are plain functions, so parsing without evaluation didn't reach them), and `product(k!, (k, 1, 3000))` or `summation(1/k**k, (k, 1, 10**6))` never finished. They now give the unevaluated `Integral`, `Sum`, `Product` and `Limit`, like any other function without `simplify`. `simplify` works out a product or limit (after the checks above) and leaves an integral or a finite sum unevaluated, as SymPy's `simplify` does. `diff` is still worked out in every mode.
- **Number literals are limited in size.** `Float('1e999999')` takes minutes, so a response such as `1e999999` hung `parse_expression` without `rationalise` (with it, the result was an integer too long for `str()`), and a mantissa of 100000 digits was as slow. A number may now have at most `MAX_NUMBER_DIGITS` (1000) digits and an exponent of at most ±`MAX_NUMBER_EXPONENT` (1000); larger ones raise `ExpressionParsingError` before anything is evaluated. An integer over Python's 4300-digit limit raised "Invalid syntax"; it now gets the same message. Powers such as `10^999999` are unaffected: they stay unevaluated.
- **`parse_expression` no longer evaluates arbitrary Python.** SymPy's `parse_expr` `eval`s the transformed input, and its default namespace includes Python's builtin functions: a learner response such as `open('/etc/hosts')` was executed (it only failed afterwards, as not a SymPy expression), and `__import__(...)` and attribute access were reachable. Parsing now goes through `_safe_parse.safe_parse_expr`: the generated code is checked against an allowlist of syntax (arithmetic, comparisons, calls of plain names, numbers; a string only as the name in `Symbol('x')`/`Function('f')` or the digits in `Float('1.5')`/`Rational`/`Integer`/`Number`, since SymPy `sympify`s string arguments with its unrestricted namespace), then evaluated with SymPy's mathematical names only and no builtins. `parse_latex` parses symbol aliases the same way (they were evaluated with builtins too). compareExpressions, where this code came from, has the same exposure.
### Fixed
-- With `elementary_functions`, `sum(expr)` without limits is an unevaluated sum, `UNEVALUATED_SUM(expr)` (exported; an undefined function `sum`), which `sympy_to_latex` prints as `\sum{\left(expr \right)}`. It became `summation(expr)`, which SymPy rejects without limits, so every response to such a question failed to parse. Two such sums are equal when their summands are; nothing is factored out, as the index is unknown. `sum(f, (k, a, b))` is still summed. An applied `sum(...)` is a sum even where the task defines `sum` as a symbol code or alias (`ALWAYS_FUNCTIONS` in `symbol_tables`); on its own, `sum` is still the task's symbol. compareExpressions had read it as that symbol, by accident of its substitution order.
+- With `elementary_functions`, `sum(expr)` without limits is an unevaluated sum, `UNEVALUATED_SUM(expr)` (exported; an undefined function `sum`), which `sympy_to_latex` prints as `\sum{\left(expr \right)}`. It became `summation(expr)`, which SymPy rejects without limits, so every response to such a question failed to parse. Two such sums are equal when their summands are; nothing is factored out, as the index is unknown. `sum(f, (k, a, b))` with finite limits is an unevaluated `Sum`, even with `simplify` (it was summed while parsing, so `sum(k!, (k, 1, 3000))` took seconds and then failed on a 6,000-digit number; `sum(k, (k, 1, n))` gave `n**2/2 + n/2`): call `.doit()` to sum it, and comparing with `simplify(a - b)` still finds `sum(k, (k, 1, n))` equal to `n*(n + 1)/2`. A sum with an infinite limit is still summed (`sum(1/k^2, (k, 1, oo))` gives `pi**2/6`, a divergent one `oo`), since SymPy can't approximate a divergent series numerically, and comparing one needs its value. An applied `sum(...)` is a sum even where the task defines `sum` as a symbol code or alias (`ALWAYS_FUNCTIONS` in `symbol_tables`); on its own, `sum` is still the task's symbol. compareExpressions had read it as that symbol, by accident of its substitution order.
- (since 0.3.0rc1) `parse_latex` leaves division between numbers unevaluated, as the lambda-feedback fork did: `\frac{10}{2}`, `10/2` and `10 \div 2` give `10/2`, and `\frac{3}{6}` gives `3/6`. In rc1 latex2sympy2_extended divided two integers as a `Rational` (`5` and `1/2`), so a "written as" criterion couldn't tell `\frac{3}{6}` from `\frac{1}{2}`. `2\frac{1}{2}` is the product `2*1/2` again (rc1 read it as the mixed number `5/2`).
- `parse_expression` leaves every SymPy function unevaluated, not only the ones SymPy's own `evaluate=False` covers (trigonometric, `exp`, `log`, ...). `factorial`, `binomial` and the number-theory functions computed while parsing: `10000000!` took minutes, and `100000!` gave an integer too long to print. As a result `5!` now parses as `factorial(5)`, not `120`, like `sin(0)` stays `sin(0)`; with `simplify` they are still computed. `parse_latex` likewise leaves `\binom` unevaluated (`\binom{5}{2}` gives `binomial(5, 2)`, not `10`): latex2sympy2_extended computed it, and `\binom{100000000}{50000000}` didn't finish. Operator classes called by name (`Pow`, `Add`, `Eq`, ...) are left unevaluated too: `Pow(9,Pow(9,9))` never finished, in every mode.
- **One LaTeX differential broke every later LaTeX parse in the process.** latex2sympy2 (and the lambda-feedback fork) overwrote a module-level variable when parsing `\mathrm{d}x`, after which every `parse_latex` call failed with "argument of type 'Symbol' is not iterable"; in a warm evaluation-function container, one learner's answer broke LaTeX for every later request. latex2sympy2_extended keeps no state between calls. (Upstream issue OrangeX4/latex2sympy#4, closed unfixed.)
diff --git a/expression_parsing/README.md b/expression_parsing/README.md
index 0428b9b..6d4b656 100644
--- a/expression_parsing/README.md
+++ b/expression_parsing/README.md
@@ -11,7 +11,7 @@ functions such as compareExpressions.
From git (not on PyPI yet):
```bash
-pip install "lambdafeedback-expression-parsing @ git+https://github.com/lambda-feedback/lambda-feedback-packages.git@v0.3.0-rc.2#subdirectory=expression_parsing"
+pip install "lambdafeedback-expression-parsing @ git+https://github.com/lambda-feedback/lambda-feedback-packages.git@v0.3.0-rc.3#subdirectory=expression_parsing"
```
Install the packages it depends on (`slr_parsing`) from the same tag; see the [repository README](../README.md#install).
diff --git a/expression_parsing/pyproject.toml b/expression_parsing/pyproject.toml
index 70ab602..d0a99ef 100644
--- a/expression_parsing/pyproject.toml
+++ b/expression_parsing/pyproject.toml
@@ -5,7 +5,7 @@ build-backend = "poetry.core.masonry.api"
[project]
name = "lambdafeedback-expression-parsing"
-version = "0.3.0rc2"
+version = "0.3.0rc3"
description = "Safe SymPy parsing and preprocessing of learner-written mathematical expressions, for Lambda Feedback evaluation functions."
readme = "README.md"
license = "MIT"
@@ -27,7 +27,7 @@ classifiers = [
]
requires-python = ">=3.12"
dependencies = [
- "lambdafeedback-slr-parsing (>=0.3.0rc2,<0.4.0)",
+ "lambdafeedback-slr-parsing (>=0.3.0rc3,<0.4.0)",
"sympy (>=1.14,<2.0)",
"latex2sympy2-extended[antlr4-13-2] (>=1.11.0,<2.0)",
]
diff --git a/expression_parsing/src/lambdafeedback/expression_parsing/__init__.py b/expression_parsing/src/lambdafeedback/expression_parsing/__init__.py
index 32b06ed..ca69a56 100644
--- a/expression_parsing/src/lambdafeedback/expression_parsing/__init__.py
+++ b/expression_parsing/src/lambdafeedback/expression_parsing/__init__.py
@@ -18,7 +18,13 @@
import logging
from importlib.metadata import PackageNotFoundError, version
-from ._safe_parse import MAX_INPUT_LENGTH, MAX_NUMBER_DIGITS, MAX_NUMBER_EXPONENT, MAX_SIMPLIFY_POWER
+from ._safe_parse import (
+ MAX_INPUT_LENGTH,
+ MAX_NUMBER_DIGITS,
+ MAX_NUMBER_EXPONENT,
+ MAX_PRODUCT_TERMS,
+ MAX_SIMPLIFY_POWER,
+)
from ._scanning import find_matching_parenthesis
from .conventions import apply_convention, convention_parser
from .errors import (
@@ -72,6 +78,7 @@
"MAX_LATEX_NESTING",
"MAX_NUMBER_DIGITS",
"MAX_NUMBER_EXPONENT",
+ "MAX_PRODUCT_TERMS",
"MAX_SIMPLIFY_POWER",
"UNEVALUATED_SUM",
"UNICODE_OPERATORS",
diff --git a/expression_parsing/src/lambdafeedback/expression_parsing/_safe_parse.py b/expression_parsing/src/lambdafeedback/expression_parsing/_safe_parse.py
index d2ee96a..ef9e8cd 100644
--- a/expression_parsing/src/lambdafeedback/expression_parsing/_safe_parse.py
+++ b/expression_parsing/src/lambdafeedback/expression_parsing/_safe_parse.py
@@ -24,6 +24,7 @@
import ast
import math
from collections.abc import Mapping
+from types import MappingProxyType
from typing import Any
import sympy
@@ -32,12 +33,16 @@
Add,
Basic,
Function,
+ Integral,
+ Limit,
Max,
Min,
Mod,
Mul,
Pow,
+ Product,
Rational,
+ Sum,
arg,
ceiling,
conjugate,
@@ -46,8 +51,10 @@
frac,
im,
log,
+ oo,
re,
sign,
+ zoo,
)
from sympy.core.function import AppliedUndef, FunctionClass
from sympy.core.operations import AssocOp
@@ -59,7 +66,10 @@
from .errors import ExpressionParsingError
# Plain Python functions (not Basic subclasses) that are part of the maths.
-_MATH_FUNCTIONS = ("sqrt", "root", "cbrt", "real_root", "diff", "integrate", "summation", "product", "limit")
+_MATH_FUNCTIONS = ("sqrt", "root", "cbrt", "real_root", "diff")
+# Functions that work out an integral, sum, product or limit, read as the unevaluated
+# object instead: working one out can take practically forever (sum(k!, (k, 1, 10**6))).
+_UNEVALUATED_FUNCTIONS = {"integrate": Integral, "summation": Sum, "product": Product, "limit": Limit}
def _math_namespace() -> dict[str, Any]:
@@ -69,6 +79,7 @@ def _math_namespace() -> dict[str, Any]:
if isinstance(obj, Basic) or (isinstance(obj, type) and issubclass(obj, Basic)):
namespace[name] = obj
namespace.update({name: getattr(sympy, name) for name in _MATH_FUNCTIONS})
+ namespace.update(_UNEVALUATED_FUNCTIONS)
# The spellings parse_expr's default namespace provides via builtins.
namespace.update({"abs": Abs, "max": Max, "min": Min})
namespace["__builtins__"] = {}
@@ -268,6 +279,9 @@ def check_input_length(text: str, error: type[ExpressionParsingError] = Expressi
MAX_SIMPLIFY_POWER = 500
"""Largest power a sum containing symbols may be raised to when simplifying (``(x+1)**500``)."""
+MAX_PRODUCT_TERMS = 1000
+"""Most factors a product with numeric limits may have when simplifying (``product(k, (k, 1, 1000))``)."""
+
# Functions whose value is no larger than their arguments' (or not an exact number at all).
_BOUNDED_FUNCTIONS = (Abs, sign, re, im, conjugate, floor, ceiling, frac, Mod)
_INEXACT_FUNCTIONS = (
@@ -290,10 +304,15 @@ def check_evaluation_cost(expr: Basic, text: str) -> None:
from their argument. Any exact number evaluation could build is bounded
by :data:`MAX_NUMBER_DIGITS` digits, like a number the learner types. A
sum containing symbols may be raised to at most :data:`MAX_SIMPLIFY_POWER`:
- simplifying ``(x+1)**1000`` takes seconds.
+ simplifying ``(x+1)**1000`` takes seconds. A ``Product`` with numeric
+ limits may have at most :data:`MAX_PRODUCT_TERMS` factors, and its index
+ counts as large as its largest bound (``product(k, (k, 1, 500))`` builds
+ a 1,135-digit number). A ``Sum`` isn't checked: simplifying leaves it
+ unevaluated.
This bounds evaluation, not SymPy's symbolic algorithms: simplifying can
- still be slow on large or contrived expressions.
+ still be slow on large or contrived expressions, and a limit, or a
+ product with symbolic or infinite limits, isn't bounded.
"""
try:
_log_size(expr)
@@ -314,16 +333,21 @@ def _magnitude(log_size: float) -> float:
return math.inf if log_size > 300 else float(10**log_size)
-def _log_size(node: Basic) -> float:
+def _log_size(node: Basic, index_sizes: Mapping[Basic, float] = MappingProxyType({})) -> float:
"""At least ``log10`` of the largest exact number (numerator or denominator) evaluating ``node`` builds.
Symbols and inexact values (floats, ``pi``, ``sin(x)``) count as 1: their
- powers don't build exact numbers.
+ powers don't build exact numbers. The index of an enclosing product
+ counts as its size in ``index_sizes``.
"""
if isinstance(node, Rational):
# Exact for small numbers: an overestimate here compounds through nested powers.
return math.log10(max(abs(node.p), node.q))
- sizes = [_log_size(argument) for argument in node.args]
+ if node in index_sizes:
+ return index_sizes[node]
+ if isinstance(node, Product):
+ return _checked(_product_log_size(node, index_sizes))
+ sizes = [_log_size(argument, index_sizes) for argument in node.args]
largest = max(sizes, default=0.0)
if isinstance(node, Add):
size = largest + math.log10(len(sizes) or 1)
@@ -339,7 +363,7 @@ def _log_size(node: Basic) -> float:
)
size = base * power if base else 0.0
elif isinstance(node, _BOUNDED_FUNCTIONS) or not isinstance(node, Function):
- size = largest # also symbols, relations, tuples, Max/Min, unevaluated sums and integrals
+ size = largest # also symbols, relations, tuples, Max/Min, sums, integrals and limits
elif isinstance(node, _INEXACT_FUNCTIONS):
size = 0.0
else:
@@ -347,9 +371,37 @@ def _log_size(node: Basic) -> float:
# argument, with values of up to about n*log10(n) digits (n! has fewer).
n = _magnitude(largest)
size = n * math.log10(2 * n)
+ return _checked(size)
+
+
+def _checked(size: float) -> float:
+ """``size``, unless it is more than :data:`MAX_NUMBER_DIGITS` digits."""
if size > MAX_NUMBER_DIGITS:
raise _TooCostlyError(
f"Evaluating {{text}} builds a number of more than {MAX_NUMBER_DIGITS} digits; "
f"at most {MAX_NUMBER_DIGITS} are allowed."
)
return size
+
+
+def _product_log_size(node: Product, index_sizes: Mapping[Basic, float]) -> float:
+ """:func:`_log_size` of a product: at most :data:`MAX_PRODUCT_TERMS` factors, each index as large as its bounds."""
+ sizes = dict(index_sizes)
+ terms = 1.0
+ for index, lower, upper in node.limits:
+ sizes[index] = max(_log_size(lower, sizes), _log_size(upper, sizes))
+ terms *= _term_count(lower, upper, sizes[index])
+ if terms > MAX_PRODUCT_TERMS:
+ raise _TooCostlyError(f"{{text}} has too many factors to work out; at most {MAX_PRODUCT_TERMS} are allowed.")
+ return _log_size(node.function, sizes) * terms
+
+
+def _term_count(lower: Basic, upper: Basic, bound_size: float) -> float:
+ """How many values an index from ``lower`` to ``upper`` takes; 1 for symbolic or infinite limits (not bounded)."""
+ if lower.free_symbols or upper.free_symbols:
+ return 1.0 # worked out in closed form, if at all
+ if lower.has(oo, -oo, zoo) or upper.has(oo, -oo, zoo):
+ return 1.0
+ if isinstance(lower, Rational) and isinstance(upper, Rational):
+ return float(abs(upper - lower) + 1)
+ return 2 * _magnitude(bound_size) # an unevaluated bound such as 10**6: as many as its size allows
diff --git a/expression_parsing/src/lambdafeedback/expression_parsing/substitution.py b/expression_parsing/src/lambdafeedback/expression_parsing/substitution.py
index c15058c..4f4ab26 100644
--- a/expression_parsing/src/lambdafeedback/expression_parsing/substitution.py
+++ b/expression_parsing/src/lambdafeedback/expression_parsing/substitution.py
@@ -135,6 +135,13 @@ def _function_original(text: str) -> str | FunctionName:
return FunctionName(text) if text.casefold() in EMBEDDABLE_FUNCTION_NAMES else text
+def _always_function_original(text: str, symbol_aliases: set[str]) -> str | FunctionName:
+ """An :data:`ALWAYS_FUNCTIONS` spelling; one the task also uses as an alias only matches where applied."""
+ if text in symbol_aliases:
+ return FunctionName(text)
+ return _function_original(text)
+
+
def _named_substitutions(table: Sequence[tuple[str, Sequence[str]]], expr: str, padding: str) -> list[Substitution]:
"""``(spelling, padded name)`` for each name and alias in ``table`` found in ``expr``."""
replacement = padding.format
@@ -196,11 +203,16 @@ def substitute_input_symbols(expr: str, params: ExpressionParams) -> str:
if params.elementary_functions:
for table, original in ((ELEMENTARY_FUNCTIONS, _function_original), (GREEK_SYMBOLS, str)):
for name, aliases in table:
- taken = set() if name in ALWAYS_FUNCTIONS else symbol_aliases
- if name in taken:
+ spellings = [spelling for spelling in (name, *aliases) if spelling in expr]
+ if name in ALWAYS_FUNCTIONS:
+ # Applied, still the function; on its own, the task's symbol.
+ substitutions += [
+ (_always_function_original(spelling, symbol_aliases), " " + name) for spelling in spellings
+ ]
+ continue
+ if name in symbol_aliases:
continue # the task uses this name for one of its own symbols
- spellings = [name] if name in expr else []
- spellings += [alias for alias in aliases if alias in expr and alias not in taken]
+ spellings = [spelling for spelling in spellings if spelling not in symbol_aliases]
substitutions += [(original(spelling), " " + name) for spelling in spellings]
for code, spec in params.symbols.items():
diff --git a/expression_parsing/src/lambdafeedback/expression_parsing/sympy_parsing.py b/expression_parsing/src/lambdafeedback/expression_parsing/sympy_parsing.py
index 94f6031..c99e033 100644
--- a/expression_parsing/src/lambdafeedback/expression_parsing/sympy_parsing.py
+++ b/expression_parsing/src/lambdafeedback/expression_parsing/sympy_parsing.py
@@ -14,7 +14,7 @@
from typing import Any
import sympy
-from sympy import Basic, Chi, E, Equality, Function, I, Lambda, Set, Symbol, Tuple, beta, gamma, summation, zeta
+from sympy import Basic, Chi, E, Equality, Function, I, Lambda, Set, Sum, Symbol, Tuple, beta, gamma, summation, zeta
from sympy.parsing.sympy_parser import (
_token_splittable,
auto_number,
@@ -104,8 +104,24 @@ def sympy_symbols(symbols: Iterable[str]) -> dict[str, Symbol]:
def _summation(*args: Any) -> Any:
- """``summation``, or :data:`UNEVALUATED_SUM` when called with just a summand."""
- return UNEVALUATED_SUM(*args) if len(args) == 1 else summation(*args)
+ """An unevaluated ``Sum``, or :data:`UNEVALUATED_SUM` when called with just a summand.
+
+ A sum with finite limits stays unevaluated, even with ``simplify``
+ (which leaves sums alone): ``sum(k!, (k, 1, 10**6))`` would not finish.
+ A sum with an infinite limit is worked out straight away, since
+ comparing one needs its value (``oo`` for a divergent series), which
+ SymPy can't approximate numerically.
+ """
+ if len(args) == 1:
+ return UNEVALUATED_SUM(*args)
+ if _has_infinite_limit(args[1:]):
+ return summation(*args)
+ return Sum(*args)
+
+
+def _has_infinite_limit(limits: Sequence[Any]) -> bool:
+ """True if any ``(index, lower, upper)`` in ``limits`` has an infinite bound."""
+ return any(bound.is_infinite for limit in limits for bound in Tuple(*limit)[1:])
def _base_symbol_dict(params: ExpressionParams) -> dict[str, Any]:
diff --git a/expression_parsing/tests/test_regressions.py b/expression_parsing/tests/test_regressions.py
index 8a059a3..c8fd9af 100644
--- a/expression_parsing/tests/test_regressions.py
+++ b/expression_parsing/tests/test_regressions.py
@@ -7,7 +7,22 @@
import warnings
import pytest
-from sympy import Abs, Eq, Function, Integral, Rational, Symbol, sqrt, srepr
+from sympy import (
+ Abs,
+ Eq,
+ Function,
+ Integral,
+ Limit,
+ Product,
+ Rational,
+ Sum,
+ Symbol,
+ factorial,
+ oo,
+ pi,
+ sqrt,
+ srepr,
+)
from lambdafeedback.expression_parsing import (
UNEVALUATED_SUM,
@@ -771,16 +786,43 @@ def test_a_task_alias_still_names_the_symbol_where_not_applied(self):
x, n = Symbol("x"), Symbol("n")
assert self.parse("sum(x^n) + sum*n", simplify=True, **params) == UNEVALUATED_SUM(x**n) + Symbol("S") * n
+ def test_a_task_alias_of_the_function_name_still_names_the_symbol_where_not_applied(self):
+ params = {"symbols": {"S": {"latex": r"\sum", "aliases": ["summation"]}}}
+ x, n = Symbol("x"), Symbol("n")
+ assert self.parse("2*summation", **params) == 2 * Symbol("S")
+ assert self.parse("summation(x^n) + summation*n", simplify=True, **params) == (
+ UNEVALUATED_SUM(x**n) + Symbol("S") * n
+ )
+
def test_equal_when_the_summands_are(self):
assert self.parse("sum(x+x)", simplify=True) == self.parse("sum(2*x)", simplify=True)
assert self.parse("2*sum(x)", simplify=True) != self.parse("sum(2*x)", simplify=True)
+ @pytest.mark.parametrize("simplify", [False, True])
@pytest.mark.parametrize(
("expr", "expected"),
- [("sum(2*k - 1, (k, 1, n))", Symbol("n") ** 2), ("summation(x, (x, 1, 3))", 6)],
+ [
+ ("sum(k, (k, 1, n))", Sum(Symbol("k"), (Symbol("k"), 1, Symbol("n")))),
+ ("summation(x, (x, 1, 3))", Sum(Symbol("x"), (Symbol("x"), 1, 3))),
+ # Summing this would take seconds, then fail on a 6,000-digit number.
+ ("Sum(k!, (k, 1, 3000))", Sum(factorial(Symbol("k")), (Symbol("k"), 1, 3000))),
+ ],
)
- def test_with_limits_is_still_summed(self, expr, expected):
- assert self.parse(expr) == expected
+ def test_with_finite_limits_is_left_unevaluated(self, expr, expected, simplify):
+ assert self.parse(expr, simplify=simplify) == expected
+
+ @pytest.mark.parametrize("simplify", [False, True])
+ @pytest.mark.parametrize(
+ ("expr", "expected"),
+ [
+ # A divergent series is oo: SymPy can't approximate it, so comparing it unevaluated fails.
+ ("sum(2*k - 1, (k, 1, oo))", oo),
+ ("sum(1/k^2, (k, 1, oo))", pi**2 / 6),
+ ("summation(1/k^2, (k, -oo, -1))", pi**2 / 6),
+ ],
+ )
+ def test_with_an_infinite_limit_is_summed(self, expr, expected, simplify):
+ assert self.parse(expr, simplify=simplify) == expected
@pytest.mark.parametrize(
("expr", "latex"),
@@ -800,6 +842,54 @@ def test_reported_answer_parses(self):
assert self.parse(expr, simplify=True, **params).has(UNEVALUATED_SUM)
+class TestCalculusFunctions:
+ """``integrate``, ``product`` and ``limit`` were worked out while parsing, even without ``simplify``."""
+
+ x, k, n = Symbol("x"), Symbol("k"), Symbol("n")
+
+ @staticmethod
+ def parse(expr, **params):
+ params = ExpressionParams.from_dict(params)
+ return parse_expression(preprocess_expression(expr, params), SympyParsingConfig.from_params(params))
+
+ @pytest.mark.parametrize(
+ ("expr", "expected"),
+ [
+ ("integrate(x, (x, 0, 1))", Integral(x, (x, 0, 1))),
+ ("product(k, (k, 1, n))", Product(k, (k, 1, n))),
+ # Working this out would build a number of millions of digits.
+ ("product(k!, (k, 1, 3000))", Product(factorial(k), (k, 1, 3000))),
+ ("limit(x^2, x, 0)", Limit(x**2, x, 0)),
+ ],
+ )
+ def test_are_left_unevaluated_without_simplify(self, expr, expected):
+ assert self.parse(expr) == expected
+
+ @pytest.mark.parametrize(
+ ("expr", "expected"),
+ [
+ ("product(k, (k, 1, 5))", 120),
+ ("product(k, (k, 1, n))", factorial(n)),
+ ("limit(x^2, x, 0)", 0),
+ ("Derivative(x^2, x)", 2 * x),
+ ],
+ )
+ def test_simplify_works_them_out(self, expr, expected):
+ assert self.parse(expr, simplify=True) == expected
+
+ @pytest.mark.parametrize(
+ ("expr", "message"),
+ [
+ ("product(k, (k, 1, 500))", "more than 1000 digits"), # 500! has 1,135 digits
+ ("product(k!, (k, 1, 3000))", "too many factors"),
+ ("product(2, (k, 1, 10^6))", "too many factors"),
+ ],
+ )
+ def test_simplify_rejects_a_product_too_costly_to_work_out(self, expr, message):
+ with pytest.raises(ExpressionParsingError, match=message):
+ self.parse(expr, simplify=True)
+
+
def test_parsing_logs_its_decisions_at_debug_only(caplog):
import logging
diff --git a/poetry.lock b/poetry.lock
index ed35c2b..70f2fc4 100644
--- a/poetry.lock
+++ b/poetry.lock
@@ -1278,7 +1278,7 @@ type = ["pygobject-stubs", "pytest-mypy (>=1.0.1)", "shtab", "types-pywin32"]
[[package]]
name = "lambdafeedback-criteria"
-version = "0.3.0rc2"
+version = "0.3.0rc3"
description = "Criteria parser and criteria graphs that turn evaluations into feedback, for Lambda Feedback evaluation functions."
optional = false
python-versions = ">=3.12"
@@ -1287,7 +1287,7 @@ files = []
develop = true
[package.dependencies]
-lambdafeedback-slr-parsing = ">=0.3.0rc2,<0.4.0"
+lambdafeedback-slr-parsing = ">=0.3.0rc3,<0.4.0"
[package.source]
type = "directory"
@@ -1295,7 +1295,7 @@ url = "criteria"
[[package]]
name = "lambdafeedback-expression-parsing"
-version = "0.3.0rc2"
+version = "0.3.0rc3"
description = "Safe SymPy parsing and preprocessing of learner-written mathematical expressions, for Lambda Feedback evaluation functions."
optional = false
python-versions = ">=3.12"
@@ -1304,7 +1304,7 @@ files = []
develop = true
[package.dependencies]
-lambdafeedback-slr-parsing = ">=0.3.0rc2,<0.4.0"
+lambdafeedback-slr-parsing = ">=0.3.0rc3,<0.4.0"
latex2sympy2-extended = {version = ">=1.11.0,<2.0", extras = ["antlr4-13-2"]}
sympy = ">=1.14,<2.0"
@@ -1314,7 +1314,7 @@ url = "expression_parsing"
[[package]]
name = "lambdafeedback-slr-parsing"
-version = "0.3.0rc2"
+version = "0.3.0rc3"
description = "Dependency-free SLR(1) parser generator for Lambda Feedback evaluation functions."
optional = false
python-versions = ">=3.12"
@@ -1328,7 +1328,7 @@ url = "slr_parsing"
[[package]]
name = "lambdafeedback-units"
-version = "0.3.0rc2"
+version = "0.3.0rc3"
description = "Parse physical quantities such as 9.81 m/s^2 into value and unit, with SI conversion and dimensional analysis."
optional = false
python-versions = ">=3.12"
@@ -1337,8 +1337,8 @@ files = []
develop = true
[package.dependencies]
-lambdafeedback-expression-parsing = ">=0.3.0rc2,<0.4.0"
-lambdafeedback-slr-parsing = ">=0.3.0rc2,<0.4.0"
+lambdafeedback-expression-parsing = ">=0.3.0rc3,<0.4.0"
+lambdafeedback-slr-parsing = ">=0.3.0rc3,<0.4.0"
sympy = ">=1.14,<2.0"
[package.source]
diff --git a/slr_parsing/CHANGELOG.md b/slr_parsing/CHANGELOG.md
index 368aa5f..a60f512 100644
--- a/slr_parsing/CHANGELOG.md
+++ b/slr_parsing/CHANGELOG.md
@@ -1,6 +1,6 @@
# Changelog: lambdafeedback-slr-parsing
-## 0.3.0 (unreleased; pre-release 0.3.0rc2, tag v0.3.0-rc.2)
+## 0.3.0 (unreleased; pre-release 0.3.0rc3, tag v0.3.0-rc.3)
The first public release, after the extraction refactor. The import path is now `lambdafeedback.slr_parsing` (was `slr_parsing`). There was no 0.2.0 release: all four `lambdafeedback-*` packages share one version, and the first release is 0.3.0. Supports Python 3.12 to 3.14 (3.11 is not supported); `__version__` gives the installed version.
@@ -46,7 +46,8 @@ The first public release, after the extraction refactor. The import path is now
- a head that isn't exactly one symbol (only the first symbol was used);
- an undeclared start, end or null symbol (was `ScanError`);
- a production that isn't a `(head, body, action-or-None)` triple (a 2-tuple raised `IndexError`; a non-callable action raised `TypeError` on the first parse);
- - an error handler that isn't a `(condition, action)` pair of callables (was `TypeError`).
+ - an error handler that isn't a `(condition, action)` pair of callables (was `TypeError`);
+ - a reduction with no goto from the state below it, which only an error action that rewrites the stack can cause (was a bare `KeyError`).
- **Token patterns are literals, not regular expressions.** The pattern of a `(pattern, label)` token specification is a plain string, matched exactly, or the new `Lexeme(forms, space_before=False, space_after=False)`: the first of several literal forms, optionally with spaces around it, where a space inside a form matches one or more spaces. `(r" *\+ *", "PLUS")` becomes `(Lexeme(("+",), True, True), "PLUS")`, and `(r"\(", "LPAR")` becomes `("(", "LPAR")`. Anything else (a run of digits, say) is scanned with a matcher, the `(symbol, label, matcher)` shape, which was already supported. Ties between equally long matches go to the specification with the larger `Lexeme.weight` (the longest form, plus 2 per side allowing spaces) or the longer literal, mirroring the old order by pattern length. `build_expression_parser` builds `Lexeme`s for its operators and delimiters, so its callers are unaffected.
- `repr(ExprNode)` matches `str(ExprNode)`.
- Modules: `tokens`, `actions`, `tags`, `grammar` (with the grammar format documented), `errors`, `scanner`, `table`, `parser`, `builder`. Fully type-annotated (`py.typed`). The type aliases used in public signatures (`ErrorCondition`, `ErrorAction`, `TraverseStep`, `TagRule`, `TagCondition`) are exported from the package root.
diff --git a/slr_parsing/README.md b/slr_parsing/README.md
index 5756a8c..142c150 100644
--- a/slr_parsing/README.md
+++ b/slr_parsing/README.md
@@ -17,7 +17,7 @@ evaluation function.
From git (not on PyPI yet):
```bash
-pip install "lambdafeedback-slr-parsing @ git+https://github.com/lambda-feedback/lambda-feedback-packages.git@v0.3.0-rc.2#subdirectory=slr_parsing"
+pip install "lambdafeedback-slr-parsing @ git+https://github.com/lambda-feedback/lambda-feedback-packages.git@v0.3.0-rc.3#subdirectory=slr_parsing"
```
Requires Python 3.12+. The import root is `lambdafeedback.slr_parsing`.
diff --git a/slr_parsing/pyproject.toml b/slr_parsing/pyproject.toml
index 38ffbbd..424023c 100644
--- a/slr_parsing/pyproject.toml
+++ b/slr_parsing/pyproject.toml
@@ -4,7 +4,7 @@ build-backend = "poetry.core.masonry.api"
[project]
name = "lambdafeedback-slr-parsing"
-version = "0.3.0rc2"
+version = "0.3.0rc3"
description = "Dependency-free SLR(1) parser generator for Lambda Feedback evaluation functions."
readme = "README.md"
license = "MIT"
diff --git a/slr_parsing/src/lambdafeedback/slr_parsing/parser.py b/slr_parsing/src/lambdafeedback/slr_parsing/parser.py
index ee0a06a..76db404 100644
--- a/slr_parsing/src/lambdafeedback/slr_parsing/parser.py
+++ b/slr_parsing/src/lambdafeedback/slr_parsing/parser.py
@@ -62,9 +62,6 @@ def __init__(
)
self.states = self.table.states
self.parsing_table = self.table.rows
- # Looked up on every parse step, so bound here rather than reached through the table's methods.
- self._symbol_ids = self.table.symbol_ids
- self._gotos = self.table.gotos
self._report_unreachable()
# ------------------------------------------------------------------
@@ -122,7 +119,7 @@ def _report_unreachable(self) -> None:
def parsing_action(self, state: int, symbol: Token) -> Entry:
"""The parse-table entry for ``state`` with ``symbol`` next."""
- return self.parsing_table[state][self._symbol_ids[symbol]]
+ return self.table.entry(state, symbol)
def _next_action(self, state: ParseState) -> Shift | Reduce | Accept:
"""The table entry for the current state and lookahead, after any error-recovery handlers, or raise."""
@@ -148,7 +145,7 @@ def _reduce(self, p: int, state: ParseState, trace: bool) -> None:
raise GrammarError(f"Production {tuple(self.productions[p][:2])} has no reduction action.")
state.output = reduction(production, state.output, self.tag_handler)
del state.stack[-len(production[1]) :]
- state.stack.append(self._gotos[state.stack[-1]][production[0]])
+ state.stack.append(self.table.goto(state.stack[-1], production[0]))
if trace:
body = "".join(x.content for x in production[1])
logger.debug("reduce by %s --> %s %s", production[0].content, body, state.output)
diff --git a/slr_parsing/tests/test_engine.py b/slr_parsing/tests/test_engine.py
index d8abf8a..215bd26 100644
--- a/slr_parsing/tests/test_engine.py
+++ b/slr_parsing/tests/test_engine.py
@@ -225,6 +225,17 @@ def give_up(parser, state):
parse("a+", make_parser(error_handlers=[(lambda items, next_symbol: True, give_up)]))
+def test_a_missing_goto_is_a_grammar_error():
+ # An error action that leaves the stack in a state the grammar can't reach used to raise a bare KeyError.
+ def corrupt_stack(parser, state):
+ after_e = parser.table.goto(0, parser.scanner.grammar_symbol("E", "Test symbol"))
+ after_id = parser.table.entry(0, parser.scan("a")[0]).state
+ state.stack[:] = [after_e, after_id] # reducing F --> id then needs a goto on F from after_e
+
+ with pytest.raises(GrammarError, match="No goto from state"):
+ parse("a)", make_parser(error_handlers=[(lambda items, next_symbol: True, corrupt_stack)]))
+
+
def test_parsing_table_string_tells_accept_from_reductions():
# The accept entry was printed as "r0", like a reduction by production 0.
parser = make_parser()
diff --git a/units/CHANGELOG.md b/units/CHANGELOG.md
index fd09cec..364f076 100644
--- a/units/CHANGELOG.md
+++ b/units/CHANGELOG.md
@@ -1,6 +1,6 @@
# Changelog: lambdafeedback-units
-## 0.3.0 (unreleased; pre-release 0.3.0rc2, tag v0.3.0-rc.2)
+## 0.3.0 (unreleased; pre-release 0.3.0rc3, tag v0.3.0-rc.3)
The first public release. This is a further tidy-up after the 0.2.0 extraction: the package now only does unit conversion. Package metadata (MIT licence, README, URLs) is in place for PyPI. Python 3.13 and later is supported, now that `lambdafeedback-expression-parsing` no longer depends on latex2sympy2. SymPy 1.14 or later is required (was 1.12): 1.12 predates Python 3.12, and a configuration with a declared function (`symbol_assumptions` `function`) can only be pickled from 1.14.
diff --git a/units/README.md b/units/README.md
index 14f32f6..d6baa8d 100644
--- a/units/README.md
+++ b/units/README.md
@@ -17,7 +17,7 @@ evaluation function.
From git (not on PyPI yet):
```bash
-pip install "lambdafeedback-units @ git+https://github.com/lambda-feedback/lambda-feedback-packages.git@v0.3.0-rc.2#subdirectory=units"
+pip install "lambdafeedback-units @ git+https://github.com/lambda-feedback/lambda-feedback-packages.git@v0.3.0-rc.3#subdirectory=units"
```
Install the packages it depends on (`slr_parsing`, `expression_parsing`) from the same tag; see the [repository README](../README.md#install).
diff --git a/units/pyproject.toml b/units/pyproject.toml
index e057171..e03aff1 100644
--- a/units/pyproject.toml
+++ b/units/pyproject.toml
@@ -4,7 +4,7 @@ build-backend = "poetry.core.masonry.api"
[project]
name = "lambdafeedback-units"
-version = "0.3.0rc2"
+version = "0.3.0rc3"
description = "Parse physical quantities such as 9.81 m/s^2 into value and unit, with SI conversion and dimensional analysis."
readme = "README.md"
license = "MIT"
@@ -25,8 +25,8 @@ classifiers = [
]
requires-python = ">=3.12"
dependencies = [
- "lambdafeedback-slr-parsing (>=0.3.0rc2,<0.4.0)",
- "lambdafeedback-expression-parsing (>=0.3.0rc2,<0.4.0)",
+ "lambdafeedback-slr-parsing (>=0.3.0rc3,<0.4.0)",
+ "lambdafeedback-expression-parsing (>=0.3.0rc3,<0.4.0)",
"sympy (>=1.14,<2.0)",
]
diff --git a/units/src/lambdafeedback/units/data.py b/units/src/lambdafeedback/units/data.py
index d3b561e..57c65f0 100644
--- a/units/src/lambdafeedback/units/data.py
+++ b/units/src/lambdafeedback/units/data.py
@@ -16,6 +16,7 @@
from __future__ import annotations
from collections.abc import Mapping
+from functools import cache
from types import MappingProxyType
from typing import NamedTuple
@@ -190,18 +191,19 @@ def units_in(unit_sets: frozenset[str] | set[str]) -> tuple[AnyUnit, ...]:
return tuple(dict.fromkeys(unit for name, units in UNIT_SETS.items() if name in unit_sets for unit in units))
-def unsplittable_names(unit_sets: frozenset[str] | set[str], all_forms: bool) -> list[str]:
+@cache
+def unsplittable_names(unit_sets: frozenset[str], all_forms: bool) -> tuple[str, ...]:
"""Names that must not be split into letters: prefixes, units, and dimension names.
For the units in ``unit_sets``. ``all_forms`` adds unit symbols,
alternative spellings and plurals (used for the value's preview) to the
- unit names (used for parsing).
+ unit names (used for parsing). Cached: every quantity parse asks for them.
"""
names = {prefix.name for prefix in SI_PREFIXES} | {unit.name for unit in SI_BASE_UNITS}
names |= {unit.dimension for unit in SI_BASE_UNITS}
for unit in units_in(unit_sets):
names |= {unit.name, unit.symbol, *unit.alternatives, *unit.plurals} if all_forms else {unit.name}
- return sorted(names)
+ return tuple(sorted(names))
def _conversion_to_base_si() -> dict[str, str]: