diff --git a/README.md b/README.md index b7aada0..3ea4bbf 100644 --- a/README.md +++ b/README.md @@ -17,13 +17,13 @@ tag. They depend on each other by version, so install the ones you use **and the packages they depend on** from the same tag: ```bash -pip install "lambdafeedback-slr-parsing @ git+https://github.com/lambda-feedback/lambda-feedback-packages.git@v0.3.0-rc.2#subdirectory=slr_parsing" -pip install "lambdafeedback-criteria @ git+https://github.com/lambda-feedback/lambda-feedback-packages.git@v0.3.0-rc.2#subdirectory=criteria" -pip install "lambdafeedback-expression-parsing @ git+https://github.com/lambda-feedback/lambda-feedback-packages.git@v0.3.0-rc.2#subdirectory=expression_parsing" -pip install "lambdafeedback-units @ git+https://github.com/lambda-feedback/lambda-feedback-packages.git@v0.3.0-rc.2#subdirectory=units" +pip install "lambdafeedback-slr-parsing @ git+https://github.com/lambda-feedback/lambda-feedback-packages.git@v0.3.0-rc.3#subdirectory=slr_parsing" +pip install "lambdafeedback-criteria @ git+https://github.com/lambda-feedback/lambda-feedback-packages.git@v0.3.0-rc.3#subdirectory=criteria" +pip install "lambdafeedback-expression-parsing @ git+https://github.com/lambda-feedback/lambda-feedback-packages.git@v0.3.0-rc.3#subdirectory=expression_parsing" +pip install "lambdafeedback-units @ git+https://github.com/lambda-feedback/lambda-feedback-packages.git@v0.3.0-rc.3#subdirectory=units" ``` -With Poetry: `lambdafeedback-units = { git = "https://github.com/lambda-feedback/lambda-feedback-packages.git", tag = "v0.3.0-rc.2", subdirectory = "units" }`, and likewise for each package. +With Poetry: `lambdafeedback-units = { git = "https://github.com/lambda-feedback/lambda-feedback-packages.git", tag = "v0.3.0-rc.3", subdirectory = "units" }`, and likewise for each package. ## Packages diff --git a/criteria/CHANGELOG.md b/criteria/CHANGELOG.md index 08c20fb..4cdd184 100644 --- a/criteria/CHANGELOG.md +++ b/criteria/CHANGELOG.md @@ -1,6 +1,6 @@ # Changelog: lambdafeedback-criteria -## 0.3.0 (unreleased; pre-release 0.3.0rc2, tag v0.3.0-rc.2) +## 0.3.0 (unreleased; pre-release 0.3.0rc3, tag v0.3.0-rc.3) The first public release, after the extraction refactor. The import path is now `lambdafeedback.criteria` (was `criteria`). There was no 0.2.0 release: all four `lambdafeedback-*` packages share one version, and the first release is 0.3.0. Supports Python 3.12 to 3.14 (3.11 is not supported); `__version__` gives the installed version. @@ -63,7 +63,7 @@ Behaviour differences to be aware of when adopting: ### Added - `build_quantity_criteria_parser(reserved_expressions, token_list=, productions=)` parses criteria on physical quantities (`response matches answer`, `dimension(response) matches dimension(answer)`, and `=`, `<=`, `>=`, `<`, `>` between quantities). Its defaults are `quantity_grammar.QUANTITY_TOKENS` / `QUANTITY_PRODUCTIONS`. It was `generate_criteria_parser` in compareExpressions' physical-quantity context, and its parse trees are unchanged. It takes the same arguments as `build_criteria_parser`, so either can fill the same slot. `reserved_expressions` isn't used yet: only `response` and `answer` scan as quantities. -- `split_criteria(criteria)` splits a string of criteria on the commas outside `()`, `[]` and `{}`, and strips each criterion. It was `split_criteria` in compareExpressions' `evaluation.py`. Unlike that version, it raises `CriteriaSyntaxError` (new, a `CriteriaError` and `ValueError`) when a bracket is closed before it is opened, or left open. The old version stopped splitting at the first such bracket instead. +- `split_criteria(criteria)` splits a string of criteria on the commas outside `()`, `[]` and `{}`, and strips each criterion. It was `split_criteria` in compareExpressions' `evaluation.py`. Any closing bracket closes any opening one, so a half-open interval keeps its comma: `response in [0, 1), response > 0` is two criteria. The old version counted each kind of bracket on its own and stopped splitting after `[0, 1)`, running the rest into one criterion. Unlike that version, it raises `CriteriaSyntaxError` (new, a `CriteriaError` and `ValueError`) when a bracket is closed when none is open, or left open. The old version stopped splitting at the first such bracket instead. It also raises `CriteriaSyntaxError` for an empty criterion (`response = answer,`, `a = b,, c = d` or an empty string), which the old version returned as `""` for the criteria parser to fail on. - `where_substitutions(substitutions)` gives `[(name, expression), ...]` for the substitution part of a `where` criterion (the second child of a `WHERE` node), with the expressions as text. It replaces the node-reading half of compareExpressions' `parse_where_substitutions`; the caller now parses the expressions, so this package doesn't need `expression_parsing`. The grammar also accepts other relations after `where` (`where x > 2`, `where x proportional to y`). compareExpressions' `where_substitution_nodes` returned the node itself for these, instead of a list, and this function raises `CriteriaSyntaxError` for them. A `SEPARATOR` that mixes in such a relation (`where a = 2; x > 3`) used to be read as substitutions, and also raises now. ### Changed diff --git a/criteria/README.md b/criteria/README.md index 85f29a2..a809222 100644 --- a/criteria/README.md +++ b/criteria/README.md @@ -14,7 +14,7 @@ evaluation function. From git (not on PyPI yet): ```bash -pip install "lambdafeedback-criteria @ git+https://github.com/lambda-feedback/lambda-feedback-packages.git@v0.3.0-rc.2#subdirectory=criteria" +pip install "lambdafeedback-criteria @ git+https://github.com/lambda-feedback/lambda-feedback-packages.git@v0.3.0-rc.3#subdirectory=criteria" ``` Install the packages it depends on (`slr_parsing`) from the same tag; see the [repository README](../README.md#install). diff --git a/criteria/pyproject.toml b/criteria/pyproject.toml index b248e3c..a4f4e2c 100644 --- a/criteria/pyproject.toml +++ b/criteria/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "poetry.core.masonry.api" [project] name = "lambdafeedback-criteria" -version = "0.3.0rc2" +version = "0.3.0rc3" description = "Criteria parser and criteria graphs that turn evaluations into feedback, for Lambda Feedback evaluation functions." readme = "README.md" license = "MIT" @@ -25,7 +25,7 @@ classifiers = [ ] requires-python = ">=3.12" dependencies = [ - "lambdafeedback-slr-parsing (>=0.3.0rc2,<0.4.0)", + "lambdafeedback-slr-parsing (>=0.3.0rc3,<0.4.0)", ] [project.urls] diff --git a/criteria/src/lambdafeedback/criteria/nodes.py b/criteria/src/lambdafeedback/criteria/nodes.py index e947556..08f045d 100644 --- a/criteria/src/lambdafeedback/criteria/nodes.py +++ b/criteria/src/lambdafeedback/criteria/nodes.py @@ -27,11 +27,6 @@ NodeKind: TypeAlias = Literal["evaluation", "criterion", "output"] -# Mermaid node shapes (opening, closing). -EVALUATION_STYLE = ("([", "])") -CRITERION_STYLE = ("[", "]") -OUTPUT_STYLE = ("{{", "}}") - class ResultLike(Protocol): """The part of a result object ``CriteriaGraph.export_feedback`` needs (``lf_toolkit.evaluation.Result`` fits).""" @@ -91,7 +86,7 @@ class EvaluationNode(Node): """ KIND: ClassVar[NodeKind | None] = "evaluation" - STYLE: ClassVar[tuple[str, str] | None] = EVALUATION_STYLE + STYLE: ClassVar[tuple[str, str] | None] = ("([", "])") evaluate: Evaluate | None = None replacement: str | None = None @@ -119,7 +114,7 @@ class CriterionNode(Node): """An outcome of an evaluation, with the generator for its feedback text.""" KIND: ClassVar[NodeKind | None] = "criterion" - STYLE: ClassVar[tuple[str, str] | None] = CRITERION_STYLE + STYLE: ClassVar[tuple[str, str] | None] = ("[", "]") feedback_string_generator: FeedbackStringGenerator = no_feedback @@ -139,7 +134,7 @@ class OutputNode(Node): """A terminal node; output nodes have no outgoing edges.""" KIND: ClassVar[NodeKind | None] = "output" - STYLE: ClassVar[tuple[str, str] | None] = OUTPUT_STYLE + STYLE: ClassVar[tuple[str, str] | None] = ("{{", "}}") def copy_into(self, graph: CriteriaGraph) -> OutputNode: return graph.add_output_node(self.label, self.summary, self.details) diff --git a/criteria/src/lambdafeedback/criteria/render.py b/criteria/src/lambdafeedback/criteria/render.py index 200eae7..3dccf6c 100644 --- a/criteria/src/lambdafeedback/criteria/render.py +++ b/criteria/src/lambdafeedback/criteria/render.py @@ -6,7 +6,8 @@ from collections.abc import Collection, Mapping from typing import TYPE_CHECKING, Any -from .nodes import CRITERION_STYLE, EVALUATION_STYLE, OUTPUT_STYLE, EvaluationNode, Node +from .errors import CriteriaGraphError +from .nodes import EvaluationNode, Node if TYPE_CHECKING: from .graph import CriteriaGraph @@ -52,20 +53,24 @@ def entry(node: Node, outgoing: bool = True) -> dict[str, Any]: ) +def _style(node: Node) -> tuple[str, str]: + """``node``'s mermaid shape (opening, closing): its kind's ``STYLE``.""" + if node.STYLE is None: + raise CriteriaGraphError(f"Cannot display node {node.label} of type {type(node).__name__}.") + return node.STYLE + + def graph_to_mermaid(graph: CriteriaGraph) -> str: """Mermaid flowchart source for ``graph``; sufficiencies are drawn as dotted edges.""" - node_groups: list[tuple[Mapping[str, Node], tuple[str, str]]] = [ - (graph.evaluations, EVALUATION_STYLE), - (graph.criteria, CRITERION_STYLE), - (graph.outputs, OUTPUT_STYLE), - ] - keys = {label: f"N_{g}_{i}" for g, (nodes, _) in enumerate(node_groups) for i, label in enumerate(nodes)} + node_groups: list[Mapping[str, Node]] = [graph.evaluations, graph.criteria, graph.outputs] + keys = {label: f"N_{g}_{i}" for g, nodes in enumerate(node_groups) for i, label in enumerate(nodes)} lines = ["flowchart TD"] # Ordered de-duplication (dict keys), so the output doesn't depend on the hash seed. edges: dict[tuple[str, str], None] = {} sufficiencies: dict[tuple[str, str], None] = {} - for nodes, (opening, closing) in node_groups: + for nodes in node_groups: for label, node in nodes.items(): + opening, closing = _style(node) lines.append(f'{keys[label]}{opening}"{_text(label)}
---
{_text(node.details)}"{closing}') edges.update( dict.fromkeys((keys[e.source.label], keys[e.target.label]) for e in node.outgoing + node.incoming) diff --git a/criteria/src/lambdafeedback/criteria/split.py b/criteria/src/lambdafeedback/criteria/split.py index 5025230..2cde173 100644 --- a/criteria/src/lambdafeedback/criteria/split.py +++ b/criteria/src/lambdafeedback/criteria/split.py @@ -4,32 +4,40 @@ from .errors import CriteriaSyntaxError -_OPENING = {")": "(", "]": "[", "}": "{"} -"""The opening bracket for each closing bracket.""" +_OPENING = frozenset("([{") +_CLOSING = frozenset(")]}") def split_criteria(criteria: str) -> list[str]: """``criteria`` split on the commas outside brackets, with each criterion stripped. - Each kind of bracket (``()``, ``[]``, ``{}``) is counted on its own. Raises - :class:`CriteriaSyntaxError` if a bracket is closed before it is opened, - or left open. + Any closing bracket (``)``, ``]``, ``}``) closes any opening one, so + half-open intervals such as ``x in [0, 1)`` keep their comma. Raises + :class:`CriteriaSyntaxError` if a bracket is closed when none is open, + or left open, or if a criterion is empty (``a = b,`` or ``a = b,, c = d``). """ - depth = dict.fromkeys(_OPENING.values(), 0) + depth = 0 parts = [] start = 0 for index, char in enumerate(criteria): - if char in depth: - depth[char] += 1 - elif char in _OPENING: - opening = _OPENING[char] - if depth[opening] == 0: - raise CriteriaSyntaxError(f"`{char}` at position {index} closes no `{opening}` in: {criteria}") - depth[opening] -= 1 - elif char == "," and not any(depth.values()): - parts.append(criteria[start:index].strip()) + if char in _OPENING: + depth += 1 + elif char in _CLOSING: + if depth == 0: + raise CriteriaSyntaxError(f"`{char}` at position {index} closes no bracket in: {criteria}") + depth -= 1 + elif char == "," and depth == 0: + parts.append(_criterion(criteria, start, index)) start = index + 1 - if any(depth.values()): + if depth: raise CriteriaSyntaxError(f"Unclosed bracket in: {criteria}") - parts.append(criteria[start:].strip()) + parts.append(_criterion(criteria, start, len(criteria))) return parts + + +def _criterion(criteria: str, start: int, end: int) -> str: + """``criteria[start:end]``, stripped; raises :class:`CriteriaSyntaxError` if that leaves nothing.""" + criterion = criteria[start:end].strip() + if not criterion: + raise CriteriaSyntaxError(f"Empty criterion at position {start} in: {criteria}") + return criterion diff --git a/criteria/tests/test_criteria.py b/criteria/tests/test_criteria.py index e6daf20..cd5eed1 100644 --- a/criteria/tests/test_criteria.py +++ b/criteria/tests/test_criteria.py @@ -92,15 +92,23 @@ class TestSplitCriteria: ("response = f(x, y), response > 0", ["response = f(x, y)", "response > 0"]), ("response = [1, 2], response = {a, b}", ["response = [1, 2]", "response = {a, b}"]), ("response = f([x, {y, z}]), answer > 0", ["response = f([x, {y, z}])", "answer > 0"]), - ("response = answer,", ["response = answer", ""]), - ("", [""]), + ("response in [0, 1), response > 0", ["response in [0, 1)", "response > 0"]), + ("response in (0, 1], answer in (a, b]", ["response in (0, 1]", "answer in (a, b]"]), ], ) def test_splits_on_commas_outside_brackets(self, criteria, expected): assert split_criteria(criteria) == expected @pytest.mark.parametrize( - "criteria", ["response = x), answer > 0", "response = (x, answer > 0", "response = (x], y"] + "criteria", + ["", " ", "response = answer,", "response = answer, ", ", response > 0", "response = answer,, x > 0"], + ) + def test_empty_criterion_raises(self, criteria): + with pytest.raises(CriteriaSyntaxError, match="Empty criterion"): + split_criteria(criteria) + + @pytest.mark.parametrize( + "criteria", ["response = x), answer > 0", "response = (x, answer > 0", "response = [0, 1)), y", "f(x"] ) def test_unbalanced_brackets_raise(self, criteria): with pytest.raises(CriteriaSyntaxError): @@ -149,6 +157,16 @@ def test_mermaid_returns_string(self): g.add_evaluation_node("EVAL", "eval", "eval details") assert isinstance(g.to_mermaid(), str) + def test_mermaid_draws_each_kind_of_node_in_its_shape(self): + g = CriteriaGraph("root") + g.add_evaluation_node("EVAL", "eval", "eval details") + g.add_criterion_node("CRIT", "crit", "crit details") + g.add_output_node("OUT", "out", "out details") + lines = g.to_mermaid().split("\n\t") + assert 'N_0_0(["EVAL
---
eval details"])' in lines + assert 'N_1_0["CRIT
---
crit details"]' in lines + assert 'N_2_0{{"OUT
---
out details"}}' in lines + def test_duplicate_evaluation_node_rejected(self): g = CriteriaGraph("root") g.add_evaluation_node("EVAL", "eval", "eval details") diff --git a/expression_parsing/CHANGELOG.md b/expression_parsing/CHANGELOG.md index 7e963f6..e0515a9 100644 --- a/expression_parsing/CHANGELOG.md +++ b/expression_parsing/CHANGELOG.md @@ -1,6 +1,6 @@ # Changelog: lambdafeedback-expression-parsing -## 0.3.0 (unreleased; pre-release 0.3.0rc2, tag v0.3.0-rc.2) +## 0.3.0 (unreleased; pre-release 0.3.0rc3, tag v0.3.0-rc.3) The first public release. Further tidy-up after the 0.2.0 extraction: previewing a response and mapping failures to learner-facing feedback text are app-layer concerns, so they move out of this package. `__version__` gives the installed version. @@ -48,13 +48,14 @@ The first public release. Further tidy-up after the 0.2.0 extraction: previewing ### Security - `MAX_INPUT_LENGTH` (10,000 characters): `parse_expression(s)` and `parse_latex` reject longer input. An expression nested beyond Python's recursion limit (a sum of about 800 terms) raises `ExpressionParsingError` saying it is too long or too deeply nested, instead of "maximum recursion depth exceeded". -- **`simplify` is bounded.** With `simplify`, parsing evaluates, so `9**9**9`, `factorial(10**8)`, `primepi(10**13)` or `(x+1)**100000` never finished. Before evaluating, the unevaluated expression is checked: an exact number evaluation would build may have at most `MAX_NUMBER_DIGITS` (1000) digits, and a sum containing symbols may be raised to at most `MAX_SIMPLIFY_POWER` (500, exported); otherwise `ExpressionParsingError`. SymPy's symbolic algorithms aren't bounded, and the calculus helpers (`summation`, `product`, `integrate`, `limit`, `diff`) are evaluated in every mode, since compareExpressions relies on that: `summation(1/k**k, (k, 1, 10**6))` still never finishes. Run evaluation of untrusted input under a time limit. +- **`simplify` is bounded.** With `simplify`, parsing evaluates, so `9**9**9`, `factorial(10**8)`, `primepi(10**13)` or `(x+1)**100000` never finished. Before evaluating, the unevaluated expression is checked: an exact number evaluation would build may have at most `MAX_NUMBER_DIGITS` (1000) digits, and a sum containing symbols may be raised to at most `MAX_SIMPLIFY_POWER` (500, exported); otherwise `ExpressionParsingError`. A `Product` with numeric limits may have at most `MAX_PRODUCT_TERMS` (1000, exported) factors, and its index counts as large as its largest bound (`product(k, (k, 1, 500))` is rejected, as `factorial(500)` is). SymPy's symbolic algorithms aren't bounded: a limit, a derivative or a product with symbolic or infinite limits can still be slow to simplify. Run evaluation of untrusted input under a time limit. +- **`integrate`, `summation`, `product` and `limit` are unevaluated without `simplify`.** They were worked out while parsing in every mode (they are plain functions, so parsing without evaluation didn't reach them), and `product(k!, (k, 1, 3000))` or `summation(1/k**k, (k, 1, 10**6))` never finished. They now give the unevaluated `Integral`, `Sum`, `Product` and `Limit`, like any other function without `simplify`. `simplify` works out a product or limit (after the checks above) and leaves an integral or a finite sum unevaluated, as SymPy's `simplify` does. `diff` is still worked out in every mode. - **Number literals are limited in size.** `Float('1e999999')` takes minutes, so a response such as `1e999999` hung `parse_expression` without `rationalise` (with it, the result was an integer too long for `str()`), and a mantissa of 100000 digits was as slow. A number may now have at most `MAX_NUMBER_DIGITS` (1000) digits and an exponent of at most ±`MAX_NUMBER_EXPONENT` (1000); larger ones raise `ExpressionParsingError` before anything is evaluated. An integer over Python's 4300-digit limit raised "Invalid syntax"; it now gets the same message. Powers such as `10^999999` are unaffected: they stay unevaluated. - **`parse_expression` no longer evaluates arbitrary Python.** SymPy's `parse_expr` `eval`s the transformed input, and its default namespace includes Python's builtin functions: a learner response such as `open('/etc/hosts')` was executed (it only failed afterwards, as not a SymPy expression), and `__import__(...)` and attribute access were reachable. Parsing now goes through `_safe_parse.safe_parse_expr`: the generated code is checked against an allowlist of syntax (arithmetic, comparisons, calls of plain names, numbers; a string only as the name in `Symbol('x')`/`Function('f')` or the digits in `Float('1.5')`/`Rational`/`Integer`/`Number`, since SymPy `sympify`s string arguments with its unrestricted namespace), then evaluated with SymPy's mathematical names only and no builtins. `parse_latex` parses symbol aliases the same way (they were evaluated with builtins too). compareExpressions, where this code came from, has the same exposure. ### Fixed -- With `elementary_functions`, `sum(expr)` without limits is an unevaluated sum, `UNEVALUATED_SUM(expr)` (exported; an undefined function `sum`), which `sympy_to_latex` prints as `\sum{\left(expr \right)}`. It became `summation(expr)`, which SymPy rejects without limits, so every response to such a question failed to parse. Two such sums are equal when their summands are; nothing is factored out, as the index is unknown. `sum(f, (k, a, b))` is still summed. An applied `sum(...)` is a sum even where the task defines `sum` as a symbol code or alias (`ALWAYS_FUNCTIONS` in `symbol_tables`); on its own, `sum` is still the task's symbol. compareExpressions had read it as that symbol, by accident of its substitution order. +- With `elementary_functions`, `sum(expr)` without limits is an unevaluated sum, `UNEVALUATED_SUM(expr)` (exported; an undefined function `sum`), which `sympy_to_latex` prints as `\sum{\left(expr \right)}`. It became `summation(expr)`, which SymPy rejects without limits, so every response to such a question failed to parse. Two such sums are equal when their summands are; nothing is factored out, as the index is unknown. `sum(f, (k, a, b))` with finite limits is an unevaluated `Sum`, even with `simplify` (it was summed while parsing, so `sum(k!, (k, 1, 3000))` took seconds and then failed on a 6,000-digit number; `sum(k, (k, 1, n))` gave `n**2/2 + n/2`): call `.doit()` to sum it, and comparing with `simplify(a - b)` still finds `sum(k, (k, 1, n))` equal to `n*(n + 1)/2`. A sum with an infinite limit is still summed (`sum(1/k^2, (k, 1, oo))` gives `pi**2/6`, a divergent one `oo`), since SymPy can't approximate a divergent series numerically, and comparing one needs its value. An applied `sum(...)` is a sum even where the task defines `sum` as a symbol code or alias (`ALWAYS_FUNCTIONS` in `symbol_tables`); on its own, `sum` is still the task's symbol. compareExpressions had read it as that symbol, by accident of its substitution order. - (since 0.3.0rc1) `parse_latex` leaves division between numbers unevaluated, as the lambda-feedback fork did: `\frac{10}{2}`, `10/2` and `10 \div 2` give `10/2`, and `\frac{3}{6}` gives `3/6`. In rc1 latex2sympy2_extended divided two integers as a `Rational` (`5` and `1/2`), so a "written as" criterion couldn't tell `\frac{3}{6}` from `\frac{1}{2}`. `2\frac{1}{2}` is the product `2*1/2` again (rc1 read it as the mixed number `5/2`). - `parse_expression` leaves every SymPy function unevaluated, not only the ones SymPy's own `evaluate=False` covers (trigonometric, `exp`, `log`, ...). `factorial`, `binomial` and the number-theory functions computed while parsing: `10000000!` took minutes, and `100000!` gave an integer too long to print. As a result `5!` now parses as `factorial(5)`, not `120`, like `sin(0)` stays `sin(0)`; with `simplify` they are still computed. `parse_latex` likewise leaves `\binom` unevaluated (`\binom{5}{2}` gives `binomial(5, 2)`, not `10`): latex2sympy2_extended computed it, and `\binom{100000000}{50000000}` didn't finish. Operator classes called by name (`Pow`, `Add`, `Eq`, ...) are left unevaluated too: `Pow(9,Pow(9,9))` never finished, in every mode. - **One LaTeX differential broke every later LaTeX parse in the process.** latex2sympy2 (and the lambda-feedback fork) overwrote a module-level variable when parsing `\mathrm{d}x`, after which every `parse_latex` call failed with "argument of type 'Symbol' is not iterable"; in a warm evaluation-function container, one learner's answer broke LaTeX for every later request. latex2sympy2_extended keeps no state between calls. (Upstream issue OrangeX4/latex2sympy#4, closed unfixed.) diff --git a/expression_parsing/README.md b/expression_parsing/README.md index 0428b9b..6d4b656 100644 --- a/expression_parsing/README.md +++ b/expression_parsing/README.md @@ -11,7 +11,7 @@ functions such as compareExpressions. From git (not on PyPI yet): ```bash -pip install "lambdafeedback-expression-parsing @ git+https://github.com/lambda-feedback/lambda-feedback-packages.git@v0.3.0-rc.2#subdirectory=expression_parsing" +pip install "lambdafeedback-expression-parsing @ git+https://github.com/lambda-feedback/lambda-feedback-packages.git@v0.3.0-rc.3#subdirectory=expression_parsing" ``` Install the packages it depends on (`slr_parsing`) from the same tag; see the [repository README](../README.md#install). diff --git a/expression_parsing/pyproject.toml b/expression_parsing/pyproject.toml index 70ab602..d0a99ef 100644 --- a/expression_parsing/pyproject.toml +++ b/expression_parsing/pyproject.toml @@ -5,7 +5,7 @@ build-backend = "poetry.core.masonry.api" [project] name = "lambdafeedback-expression-parsing" -version = "0.3.0rc2" +version = "0.3.0rc3" description = "Safe SymPy parsing and preprocessing of learner-written mathematical expressions, for Lambda Feedback evaluation functions." readme = "README.md" license = "MIT" @@ -27,7 +27,7 @@ classifiers = [ ] requires-python = ">=3.12" dependencies = [ - "lambdafeedback-slr-parsing (>=0.3.0rc2,<0.4.0)", + "lambdafeedback-slr-parsing (>=0.3.0rc3,<0.4.0)", "sympy (>=1.14,<2.0)", "latex2sympy2-extended[antlr4-13-2] (>=1.11.0,<2.0)", ] diff --git a/expression_parsing/src/lambdafeedback/expression_parsing/__init__.py b/expression_parsing/src/lambdafeedback/expression_parsing/__init__.py index 32b06ed..ca69a56 100644 --- a/expression_parsing/src/lambdafeedback/expression_parsing/__init__.py +++ b/expression_parsing/src/lambdafeedback/expression_parsing/__init__.py @@ -18,7 +18,13 @@ import logging from importlib.metadata import PackageNotFoundError, version -from ._safe_parse import MAX_INPUT_LENGTH, MAX_NUMBER_DIGITS, MAX_NUMBER_EXPONENT, MAX_SIMPLIFY_POWER +from ._safe_parse import ( + MAX_INPUT_LENGTH, + MAX_NUMBER_DIGITS, + MAX_NUMBER_EXPONENT, + MAX_PRODUCT_TERMS, + MAX_SIMPLIFY_POWER, +) from ._scanning import find_matching_parenthesis from .conventions import apply_convention, convention_parser from .errors import ( @@ -72,6 +78,7 @@ "MAX_LATEX_NESTING", "MAX_NUMBER_DIGITS", "MAX_NUMBER_EXPONENT", + "MAX_PRODUCT_TERMS", "MAX_SIMPLIFY_POWER", "UNEVALUATED_SUM", "UNICODE_OPERATORS", diff --git a/expression_parsing/src/lambdafeedback/expression_parsing/_safe_parse.py b/expression_parsing/src/lambdafeedback/expression_parsing/_safe_parse.py index d2ee96a..ef9e8cd 100644 --- a/expression_parsing/src/lambdafeedback/expression_parsing/_safe_parse.py +++ b/expression_parsing/src/lambdafeedback/expression_parsing/_safe_parse.py @@ -24,6 +24,7 @@ import ast import math from collections.abc import Mapping +from types import MappingProxyType from typing import Any import sympy @@ -32,12 +33,16 @@ Add, Basic, Function, + Integral, + Limit, Max, Min, Mod, Mul, Pow, + Product, Rational, + Sum, arg, ceiling, conjugate, @@ -46,8 +51,10 @@ frac, im, log, + oo, re, sign, + zoo, ) from sympy.core.function import AppliedUndef, FunctionClass from sympy.core.operations import AssocOp @@ -59,7 +66,10 @@ from .errors import ExpressionParsingError # Plain Python functions (not Basic subclasses) that are part of the maths. -_MATH_FUNCTIONS = ("sqrt", "root", "cbrt", "real_root", "diff", "integrate", "summation", "product", "limit") +_MATH_FUNCTIONS = ("sqrt", "root", "cbrt", "real_root", "diff") +# Functions that work out an integral, sum, product or limit, read as the unevaluated +# object instead: working one out can take practically forever (sum(k!, (k, 1, 10**6))). +_UNEVALUATED_FUNCTIONS = {"integrate": Integral, "summation": Sum, "product": Product, "limit": Limit} def _math_namespace() -> dict[str, Any]: @@ -69,6 +79,7 @@ def _math_namespace() -> dict[str, Any]: if isinstance(obj, Basic) or (isinstance(obj, type) and issubclass(obj, Basic)): namespace[name] = obj namespace.update({name: getattr(sympy, name) for name in _MATH_FUNCTIONS}) + namespace.update(_UNEVALUATED_FUNCTIONS) # The spellings parse_expr's default namespace provides via builtins. namespace.update({"abs": Abs, "max": Max, "min": Min}) namespace["__builtins__"] = {} @@ -268,6 +279,9 @@ def check_input_length(text: str, error: type[ExpressionParsingError] = Expressi MAX_SIMPLIFY_POWER = 500 """Largest power a sum containing symbols may be raised to when simplifying (``(x+1)**500``).""" +MAX_PRODUCT_TERMS = 1000 +"""Most factors a product with numeric limits may have when simplifying (``product(k, (k, 1, 1000))``).""" + # Functions whose value is no larger than their arguments' (or not an exact number at all). _BOUNDED_FUNCTIONS = (Abs, sign, re, im, conjugate, floor, ceiling, frac, Mod) _INEXACT_FUNCTIONS = ( @@ -290,10 +304,15 @@ def check_evaluation_cost(expr: Basic, text: str) -> None: from their argument. Any exact number evaluation could build is bounded by :data:`MAX_NUMBER_DIGITS` digits, like a number the learner types. A sum containing symbols may be raised to at most :data:`MAX_SIMPLIFY_POWER`: - simplifying ``(x+1)**1000`` takes seconds. + simplifying ``(x+1)**1000`` takes seconds. A ``Product`` with numeric + limits may have at most :data:`MAX_PRODUCT_TERMS` factors, and its index + counts as large as its largest bound (``product(k, (k, 1, 500))`` builds + a 1,135-digit number). A ``Sum`` isn't checked: simplifying leaves it + unevaluated. This bounds evaluation, not SymPy's symbolic algorithms: simplifying can - still be slow on large or contrived expressions. + still be slow on large or contrived expressions, and a limit, or a + product with symbolic or infinite limits, isn't bounded. """ try: _log_size(expr) @@ -314,16 +333,21 @@ def _magnitude(log_size: float) -> float: return math.inf if log_size > 300 else float(10**log_size) -def _log_size(node: Basic) -> float: +def _log_size(node: Basic, index_sizes: Mapping[Basic, float] = MappingProxyType({})) -> float: """At least ``log10`` of the largest exact number (numerator or denominator) evaluating ``node`` builds. Symbols and inexact values (floats, ``pi``, ``sin(x)``) count as 1: their - powers don't build exact numbers. + powers don't build exact numbers. The index of an enclosing product + counts as its size in ``index_sizes``. """ if isinstance(node, Rational): # Exact for small numbers: an overestimate here compounds through nested powers. return math.log10(max(abs(node.p), node.q)) - sizes = [_log_size(argument) for argument in node.args] + if node in index_sizes: + return index_sizes[node] + if isinstance(node, Product): + return _checked(_product_log_size(node, index_sizes)) + sizes = [_log_size(argument, index_sizes) for argument in node.args] largest = max(sizes, default=0.0) if isinstance(node, Add): size = largest + math.log10(len(sizes) or 1) @@ -339,7 +363,7 @@ def _log_size(node: Basic) -> float: ) size = base * power if base else 0.0 elif isinstance(node, _BOUNDED_FUNCTIONS) or not isinstance(node, Function): - size = largest # also symbols, relations, tuples, Max/Min, unevaluated sums and integrals + size = largest # also symbols, relations, tuples, Max/Min, sums, integrals and limits elif isinstance(node, _INEXACT_FUNCTIONS): size = 0.0 else: @@ -347,9 +371,37 @@ def _log_size(node: Basic) -> float: # argument, with values of up to about n*log10(n) digits (n! has fewer). n = _magnitude(largest) size = n * math.log10(2 * n) + return _checked(size) + + +def _checked(size: float) -> float: + """``size``, unless it is more than :data:`MAX_NUMBER_DIGITS` digits.""" if size > MAX_NUMBER_DIGITS: raise _TooCostlyError( f"Evaluating {{text}} builds a number of more than {MAX_NUMBER_DIGITS} digits; " f"at most {MAX_NUMBER_DIGITS} are allowed." ) return size + + +def _product_log_size(node: Product, index_sizes: Mapping[Basic, float]) -> float: + """:func:`_log_size` of a product: at most :data:`MAX_PRODUCT_TERMS` factors, each index as large as its bounds.""" + sizes = dict(index_sizes) + terms = 1.0 + for index, lower, upper in node.limits: + sizes[index] = max(_log_size(lower, sizes), _log_size(upper, sizes)) + terms *= _term_count(lower, upper, sizes[index]) + if terms > MAX_PRODUCT_TERMS: + raise _TooCostlyError(f"{{text}} has too many factors to work out; at most {MAX_PRODUCT_TERMS} are allowed.") + return _log_size(node.function, sizes) * terms + + +def _term_count(lower: Basic, upper: Basic, bound_size: float) -> float: + """How many values an index from ``lower`` to ``upper`` takes; 1 for symbolic or infinite limits (not bounded).""" + if lower.free_symbols or upper.free_symbols: + return 1.0 # worked out in closed form, if at all + if lower.has(oo, -oo, zoo) or upper.has(oo, -oo, zoo): + return 1.0 + if isinstance(lower, Rational) and isinstance(upper, Rational): + return float(abs(upper - lower) + 1) + return 2 * _magnitude(bound_size) # an unevaluated bound such as 10**6: as many as its size allows diff --git a/expression_parsing/src/lambdafeedback/expression_parsing/substitution.py b/expression_parsing/src/lambdafeedback/expression_parsing/substitution.py index c15058c..4f4ab26 100644 --- a/expression_parsing/src/lambdafeedback/expression_parsing/substitution.py +++ b/expression_parsing/src/lambdafeedback/expression_parsing/substitution.py @@ -135,6 +135,13 @@ def _function_original(text: str) -> str | FunctionName: return FunctionName(text) if text.casefold() in EMBEDDABLE_FUNCTION_NAMES else text +def _always_function_original(text: str, symbol_aliases: set[str]) -> str | FunctionName: + """An :data:`ALWAYS_FUNCTIONS` spelling; one the task also uses as an alias only matches where applied.""" + if text in symbol_aliases: + return FunctionName(text) + return _function_original(text) + + def _named_substitutions(table: Sequence[tuple[str, Sequence[str]]], expr: str, padding: str) -> list[Substitution]: """``(spelling, padded name)`` for each name and alias in ``table`` found in ``expr``.""" replacement = padding.format @@ -196,11 +203,16 @@ def substitute_input_symbols(expr: str, params: ExpressionParams) -> str: if params.elementary_functions: for table, original in ((ELEMENTARY_FUNCTIONS, _function_original), (GREEK_SYMBOLS, str)): for name, aliases in table: - taken = set() if name in ALWAYS_FUNCTIONS else symbol_aliases - if name in taken: + spellings = [spelling for spelling in (name, *aliases) if spelling in expr] + if name in ALWAYS_FUNCTIONS: + # Applied, still the function; on its own, the task's symbol. + substitutions += [ + (_always_function_original(spelling, symbol_aliases), " " + name) for spelling in spellings + ] + continue + if name in symbol_aliases: continue # the task uses this name for one of its own symbols - spellings = [name] if name in expr else [] - spellings += [alias for alias in aliases if alias in expr and alias not in taken] + spellings = [spelling for spelling in spellings if spelling not in symbol_aliases] substitutions += [(original(spelling), " " + name) for spelling in spellings] for code, spec in params.symbols.items(): diff --git a/expression_parsing/src/lambdafeedback/expression_parsing/sympy_parsing.py b/expression_parsing/src/lambdafeedback/expression_parsing/sympy_parsing.py index 94f6031..c99e033 100644 --- a/expression_parsing/src/lambdafeedback/expression_parsing/sympy_parsing.py +++ b/expression_parsing/src/lambdafeedback/expression_parsing/sympy_parsing.py @@ -14,7 +14,7 @@ from typing import Any import sympy -from sympy import Basic, Chi, E, Equality, Function, I, Lambda, Set, Symbol, Tuple, beta, gamma, summation, zeta +from sympy import Basic, Chi, E, Equality, Function, I, Lambda, Set, Sum, Symbol, Tuple, beta, gamma, summation, zeta from sympy.parsing.sympy_parser import ( _token_splittable, auto_number, @@ -104,8 +104,24 @@ def sympy_symbols(symbols: Iterable[str]) -> dict[str, Symbol]: def _summation(*args: Any) -> Any: - """``summation``, or :data:`UNEVALUATED_SUM` when called with just a summand.""" - return UNEVALUATED_SUM(*args) if len(args) == 1 else summation(*args) + """An unevaluated ``Sum``, or :data:`UNEVALUATED_SUM` when called with just a summand. + + A sum with finite limits stays unevaluated, even with ``simplify`` + (which leaves sums alone): ``sum(k!, (k, 1, 10**6))`` would not finish. + A sum with an infinite limit is worked out straight away, since + comparing one needs its value (``oo`` for a divergent series), which + SymPy can't approximate numerically. + """ + if len(args) == 1: + return UNEVALUATED_SUM(*args) + if _has_infinite_limit(args[1:]): + return summation(*args) + return Sum(*args) + + +def _has_infinite_limit(limits: Sequence[Any]) -> bool: + """True if any ``(index, lower, upper)`` in ``limits`` has an infinite bound.""" + return any(bound.is_infinite for limit in limits for bound in Tuple(*limit)[1:]) def _base_symbol_dict(params: ExpressionParams) -> dict[str, Any]: diff --git a/expression_parsing/tests/test_regressions.py b/expression_parsing/tests/test_regressions.py index 8a059a3..c8fd9af 100644 --- a/expression_parsing/tests/test_regressions.py +++ b/expression_parsing/tests/test_regressions.py @@ -7,7 +7,22 @@ import warnings import pytest -from sympy import Abs, Eq, Function, Integral, Rational, Symbol, sqrt, srepr +from sympy import ( + Abs, + Eq, + Function, + Integral, + Limit, + Product, + Rational, + Sum, + Symbol, + factorial, + oo, + pi, + sqrt, + srepr, +) from lambdafeedback.expression_parsing import ( UNEVALUATED_SUM, @@ -771,16 +786,43 @@ def test_a_task_alias_still_names_the_symbol_where_not_applied(self): x, n = Symbol("x"), Symbol("n") assert self.parse("sum(x^n) + sum*n", simplify=True, **params) == UNEVALUATED_SUM(x**n) + Symbol("S") * n + def test_a_task_alias_of_the_function_name_still_names_the_symbol_where_not_applied(self): + params = {"symbols": {"S": {"latex": r"\sum", "aliases": ["summation"]}}} + x, n = Symbol("x"), Symbol("n") + assert self.parse("2*summation", **params) == 2 * Symbol("S") + assert self.parse("summation(x^n) + summation*n", simplify=True, **params) == ( + UNEVALUATED_SUM(x**n) + Symbol("S") * n + ) + def test_equal_when_the_summands_are(self): assert self.parse("sum(x+x)", simplify=True) == self.parse("sum(2*x)", simplify=True) assert self.parse("2*sum(x)", simplify=True) != self.parse("sum(2*x)", simplify=True) + @pytest.mark.parametrize("simplify", [False, True]) @pytest.mark.parametrize( ("expr", "expected"), - [("sum(2*k - 1, (k, 1, n))", Symbol("n") ** 2), ("summation(x, (x, 1, 3))", 6)], + [ + ("sum(k, (k, 1, n))", Sum(Symbol("k"), (Symbol("k"), 1, Symbol("n")))), + ("summation(x, (x, 1, 3))", Sum(Symbol("x"), (Symbol("x"), 1, 3))), + # Summing this would take seconds, then fail on a 6,000-digit number. + ("Sum(k!, (k, 1, 3000))", Sum(factorial(Symbol("k")), (Symbol("k"), 1, 3000))), + ], ) - def test_with_limits_is_still_summed(self, expr, expected): - assert self.parse(expr) == expected + def test_with_finite_limits_is_left_unevaluated(self, expr, expected, simplify): + assert self.parse(expr, simplify=simplify) == expected + + @pytest.mark.parametrize("simplify", [False, True]) + @pytest.mark.parametrize( + ("expr", "expected"), + [ + # A divergent series is oo: SymPy can't approximate it, so comparing it unevaluated fails. + ("sum(2*k - 1, (k, 1, oo))", oo), + ("sum(1/k^2, (k, 1, oo))", pi**2 / 6), + ("summation(1/k^2, (k, -oo, -1))", pi**2 / 6), + ], + ) + def test_with_an_infinite_limit_is_summed(self, expr, expected, simplify): + assert self.parse(expr, simplify=simplify) == expected @pytest.mark.parametrize( ("expr", "latex"), @@ -800,6 +842,54 @@ def test_reported_answer_parses(self): assert self.parse(expr, simplify=True, **params).has(UNEVALUATED_SUM) +class TestCalculusFunctions: + """``integrate``, ``product`` and ``limit`` were worked out while parsing, even without ``simplify``.""" + + x, k, n = Symbol("x"), Symbol("k"), Symbol("n") + + @staticmethod + def parse(expr, **params): + params = ExpressionParams.from_dict(params) + return parse_expression(preprocess_expression(expr, params), SympyParsingConfig.from_params(params)) + + @pytest.mark.parametrize( + ("expr", "expected"), + [ + ("integrate(x, (x, 0, 1))", Integral(x, (x, 0, 1))), + ("product(k, (k, 1, n))", Product(k, (k, 1, n))), + # Working this out would build a number of millions of digits. + ("product(k!, (k, 1, 3000))", Product(factorial(k), (k, 1, 3000))), + ("limit(x^2, x, 0)", Limit(x**2, x, 0)), + ], + ) + def test_are_left_unevaluated_without_simplify(self, expr, expected): + assert self.parse(expr) == expected + + @pytest.mark.parametrize( + ("expr", "expected"), + [ + ("product(k, (k, 1, 5))", 120), + ("product(k, (k, 1, n))", factorial(n)), + ("limit(x^2, x, 0)", 0), + ("Derivative(x^2, x)", 2 * x), + ], + ) + def test_simplify_works_them_out(self, expr, expected): + assert self.parse(expr, simplify=True) == expected + + @pytest.mark.parametrize( + ("expr", "message"), + [ + ("product(k, (k, 1, 500))", "more than 1000 digits"), # 500! has 1,135 digits + ("product(k!, (k, 1, 3000))", "too many factors"), + ("product(2, (k, 1, 10^6))", "too many factors"), + ], + ) + def test_simplify_rejects_a_product_too_costly_to_work_out(self, expr, message): + with pytest.raises(ExpressionParsingError, match=message): + self.parse(expr, simplify=True) + + def test_parsing_logs_its_decisions_at_debug_only(caplog): import logging diff --git a/poetry.lock b/poetry.lock index ed35c2b..70f2fc4 100644 --- a/poetry.lock +++ b/poetry.lock @@ -1278,7 +1278,7 @@ type = ["pygobject-stubs", "pytest-mypy (>=1.0.1)", "shtab", "types-pywin32"] [[package]] name = "lambdafeedback-criteria" -version = "0.3.0rc2" +version = "0.3.0rc3" description = "Criteria parser and criteria graphs that turn evaluations into feedback, for Lambda Feedback evaluation functions." optional = false python-versions = ">=3.12" @@ -1287,7 +1287,7 @@ files = [] develop = true [package.dependencies] -lambdafeedback-slr-parsing = ">=0.3.0rc2,<0.4.0" +lambdafeedback-slr-parsing = ">=0.3.0rc3,<0.4.0" [package.source] type = "directory" @@ -1295,7 +1295,7 @@ url = "criteria" [[package]] name = "lambdafeedback-expression-parsing" -version = "0.3.0rc2" +version = "0.3.0rc3" description = "Safe SymPy parsing and preprocessing of learner-written mathematical expressions, for Lambda Feedback evaluation functions." optional = false python-versions = ">=3.12" @@ -1304,7 +1304,7 @@ files = [] develop = true [package.dependencies] -lambdafeedback-slr-parsing = ">=0.3.0rc2,<0.4.0" +lambdafeedback-slr-parsing = ">=0.3.0rc3,<0.4.0" latex2sympy2-extended = {version = ">=1.11.0,<2.0", extras = ["antlr4-13-2"]} sympy = ">=1.14,<2.0" @@ -1314,7 +1314,7 @@ url = "expression_parsing" [[package]] name = "lambdafeedback-slr-parsing" -version = "0.3.0rc2" +version = "0.3.0rc3" description = "Dependency-free SLR(1) parser generator for Lambda Feedback evaluation functions." optional = false python-versions = ">=3.12" @@ -1328,7 +1328,7 @@ url = "slr_parsing" [[package]] name = "lambdafeedback-units" -version = "0.3.0rc2" +version = "0.3.0rc3" description = "Parse physical quantities such as 9.81 m/s^2 into value and unit, with SI conversion and dimensional analysis." optional = false python-versions = ">=3.12" @@ -1337,8 +1337,8 @@ files = [] develop = true [package.dependencies] -lambdafeedback-expression-parsing = ">=0.3.0rc2,<0.4.0" -lambdafeedback-slr-parsing = ">=0.3.0rc2,<0.4.0" +lambdafeedback-expression-parsing = ">=0.3.0rc3,<0.4.0" +lambdafeedback-slr-parsing = ">=0.3.0rc3,<0.4.0" sympy = ">=1.14,<2.0" [package.source] diff --git a/slr_parsing/CHANGELOG.md b/slr_parsing/CHANGELOG.md index 368aa5f..a60f512 100644 --- a/slr_parsing/CHANGELOG.md +++ b/slr_parsing/CHANGELOG.md @@ -1,6 +1,6 @@ # Changelog: lambdafeedback-slr-parsing -## 0.3.0 (unreleased; pre-release 0.3.0rc2, tag v0.3.0-rc.2) +## 0.3.0 (unreleased; pre-release 0.3.0rc3, tag v0.3.0-rc.3) The first public release, after the extraction refactor. The import path is now `lambdafeedback.slr_parsing` (was `slr_parsing`). There was no 0.2.0 release: all four `lambdafeedback-*` packages share one version, and the first release is 0.3.0. Supports Python 3.12 to 3.14 (3.11 is not supported); `__version__` gives the installed version. @@ -46,7 +46,8 @@ The first public release, after the extraction refactor. The import path is now - a head that isn't exactly one symbol (only the first symbol was used); - an undeclared start, end or null symbol (was `ScanError`); - a production that isn't a `(head, body, action-or-None)` triple (a 2-tuple raised `IndexError`; a non-callable action raised `TypeError` on the first parse); - - an error handler that isn't a `(condition, action)` pair of callables (was `TypeError`). + - an error handler that isn't a `(condition, action)` pair of callables (was `TypeError`); + - a reduction with no goto from the state below it, which only an error action that rewrites the stack can cause (was a bare `KeyError`). - **Token patterns are literals, not regular expressions.** The pattern of a `(pattern, label)` token specification is a plain string, matched exactly, or the new `Lexeme(forms, space_before=False, space_after=False)`: the first of several literal forms, optionally with spaces around it, where a space inside a form matches one or more spaces. `(r" *\+ *", "PLUS")` becomes `(Lexeme(("+",), True, True), "PLUS")`, and `(r"\(", "LPAR")` becomes `("(", "LPAR")`. Anything else (a run of digits, say) is scanned with a matcher, the `(symbol, label, matcher)` shape, which was already supported. Ties between equally long matches go to the specification with the larger `Lexeme.weight` (the longest form, plus 2 per side allowing spaces) or the longer literal, mirroring the old order by pattern length. `build_expression_parser` builds `Lexeme`s for its operators and delimiters, so its callers are unaffected. - `repr(ExprNode)` matches `str(ExprNode)`. - Modules: `tokens`, `actions`, `tags`, `grammar` (with the grammar format documented), `errors`, `scanner`, `table`, `parser`, `builder`. Fully type-annotated (`py.typed`). The type aliases used in public signatures (`ErrorCondition`, `ErrorAction`, `TraverseStep`, `TagRule`, `TagCondition`) are exported from the package root. diff --git a/slr_parsing/README.md b/slr_parsing/README.md index 5756a8c..142c150 100644 --- a/slr_parsing/README.md +++ b/slr_parsing/README.md @@ -17,7 +17,7 @@ evaluation function. From git (not on PyPI yet): ```bash -pip install "lambdafeedback-slr-parsing @ git+https://github.com/lambda-feedback/lambda-feedback-packages.git@v0.3.0-rc.2#subdirectory=slr_parsing" +pip install "lambdafeedback-slr-parsing @ git+https://github.com/lambda-feedback/lambda-feedback-packages.git@v0.3.0-rc.3#subdirectory=slr_parsing" ``` Requires Python 3.12+. The import root is `lambdafeedback.slr_parsing`. diff --git a/slr_parsing/pyproject.toml b/slr_parsing/pyproject.toml index 38ffbbd..424023c 100644 --- a/slr_parsing/pyproject.toml +++ b/slr_parsing/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "poetry.core.masonry.api" [project] name = "lambdafeedback-slr-parsing" -version = "0.3.0rc2" +version = "0.3.0rc3" description = "Dependency-free SLR(1) parser generator for Lambda Feedback evaluation functions." readme = "README.md" license = "MIT" diff --git a/slr_parsing/src/lambdafeedback/slr_parsing/parser.py b/slr_parsing/src/lambdafeedback/slr_parsing/parser.py index ee0a06a..76db404 100644 --- a/slr_parsing/src/lambdafeedback/slr_parsing/parser.py +++ b/slr_parsing/src/lambdafeedback/slr_parsing/parser.py @@ -62,9 +62,6 @@ def __init__( ) self.states = self.table.states self.parsing_table = self.table.rows - # Looked up on every parse step, so bound here rather than reached through the table's methods. - self._symbol_ids = self.table.symbol_ids - self._gotos = self.table.gotos self._report_unreachable() # ------------------------------------------------------------------ @@ -122,7 +119,7 @@ def _report_unreachable(self) -> None: def parsing_action(self, state: int, symbol: Token) -> Entry: """The parse-table entry for ``state`` with ``symbol`` next.""" - return self.parsing_table[state][self._symbol_ids[symbol]] + return self.table.entry(state, symbol) def _next_action(self, state: ParseState) -> Shift | Reduce | Accept: """The table entry for the current state and lookahead, after any error-recovery handlers, or raise.""" @@ -148,7 +145,7 @@ def _reduce(self, p: int, state: ParseState, trace: bool) -> None: raise GrammarError(f"Production {tuple(self.productions[p][:2])} has no reduction action.") state.output = reduction(production, state.output, self.tag_handler) del state.stack[-len(production[1]) :] - state.stack.append(self._gotos[state.stack[-1]][production[0]]) + state.stack.append(self.table.goto(state.stack[-1], production[0])) if trace: body = "".join(x.content for x in production[1]) logger.debug("reduce by %s --> %s %s", production[0].content, body, state.output) diff --git a/slr_parsing/tests/test_engine.py b/slr_parsing/tests/test_engine.py index d8abf8a..215bd26 100644 --- a/slr_parsing/tests/test_engine.py +++ b/slr_parsing/tests/test_engine.py @@ -225,6 +225,17 @@ def give_up(parser, state): parse("a+", make_parser(error_handlers=[(lambda items, next_symbol: True, give_up)])) +def test_a_missing_goto_is_a_grammar_error(): + # An error action that leaves the stack in a state the grammar can't reach used to raise a bare KeyError. + def corrupt_stack(parser, state): + after_e = parser.table.goto(0, parser.scanner.grammar_symbol("E", "Test symbol")) + after_id = parser.table.entry(0, parser.scan("a")[0]).state + state.stack[:] = [after_e, after_id] # reducing F --> id then needs a goto on F from after_e + + with pytest.raises(GrammarError, match="No goto from state"): + parse("a)", make_parser(error_handlers=[(lambda items, next_symbol: True, corrupt_stack)])) + + def test_parsing_table_string_tells_accept_from_reductions(): # The accept entry was printed as "r0", like a reduction by production 0. parser = make_parser() diff --git a/units/CHANGELOG.md b/units/CHANGELOG.md index fd09cec..364f076 100644 --- a/units/CHANGELOG.md +++ b/units/CHANGELOG.md @@ -1,6 +1,6 @@ # Changelog: lambdafeedback-units -## 0.3.0 (unreleased; pre-release 0.3.0rc2, tag v0.3.0-rc.2) +## 0.3.0 (unreleased; pre-release 0.3.0rc3, tag v0.3.0-rc.3) The first public release. This is a further tidy-up after the 0.2.0 extraction: the package now only does unit conversion. Package metadata (MIT licence, README, URLs) is in place for PyPI. Python 3.13 and later is supported, now that `lambdafeedback-expression-parsing` no longer depends on latex2sympy2. SymPy 1.14 or later is required (was 1.12): 1.12 predates Python 3.12, and a configuration with a declared function (`symbol_assumptions` `function`) can only be pickled from 1.14. diff --git a/units/README.md b/units/README.md index 14f32f6..d6baa8d 100644 --- a/units/README.md +++ b/units/README.md @@ -17,7 +17,7 @@ evaluation function. From git (not on PyPI yet): ```bash -pip install "lambdafeedback-units @ git+https://github.com/lambda-feedback/lambda-feedback-packages.git@v0.3.0-rc.2#subdirectory=units" +pip install "lambdafeedback-units @ git+https://github.com/lambda-feedback/lambda-feedback-packages.git@v0.3.0-rc.3#subdirectory=units" ``` Install the packages it depends on (`slr_parsing`, `expression_parsing`) from the same tag; see the [repository README](../README.md#install). diff --git a/units/pyproject.toml b/units/pyproject.toml index e057171..e03aff1 100644 --- a/units/pyproject.toml +++ b/units/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "poetry.core.masonry.api" [project] name = "lambdafeedback-units" -version = "0.3.0rc2" +version = "0.3.0rc3" description = "Parse physical quantities such as 9.81 m/s^2 into value and unit, with SI conversion and dimensional analysis." readme = "README.md" license = "MIT" @@ -25,8 +25,8 @@ classifiers = [ ] requires-python = ">=3.12" dependencies = [ - "lambdafeedback-slr-parsing (>=0.3.0rc2,<0.4.0)", - "lambdafeedback-expression-parsing (>=0.3.0rc2,<0.4.0)", + "lambdafeedback-slr-parsing (>=0.3.0rc3,<0.4.0)", + "lambdafeedback-expression-parsing (>=0.3.0rc3,<0.4.0)", "sympy (>=1.14,<2.0)", ] diff --git a/units/src/lambdafeedback/units/data.py b/units/src/lambdafeedback/units/data.py index d3b561e..57c65f0 100644 --- a/units/src/lambdafeedback/units/data.py +++ b/units/src/lambdafeedback/units/data.py @@ -16,6 +16,7 @@ from __future__ import annotations from collections.abc import Mapping +from functools import cache from types import MappingProxyType from typing import NamedTuple @@ -190,18 +191,19 @@ def units_in(unit_sets: frozenset[str] | set[str]) -> tuple[AnyUnit, ...]: return tuple(dict.fromkeys(unit for name, units in UNIT_SETS.items() if name in unit_sets for unit in units)) -def unsplittable_names(unit_sets: frozenset[str] | set[str], all_forms: bool) -> list[str]: +@cache +def unsplittable_names(unit_sets: frozenset[str], all_forms: bool) -> tuple[str, ...]: """Names that must not be split into letters: prefixes, units, and dimension names. For the units in ``unit_sets``. ``all_forms`` adds unit symbols, alternative spellings and plurals (used for the value's preview) to the - unit names (used for parsing). + unit names (used for parsing). Cached: every quantity parse asks for them. """ names = {prefix.name for prefix in SI_PREFIXES} | {unit.name for unit in SI_BASE_UNITS} names |= {unit.dimension for unit in SI_BASE_UNITS} for unit in units_in(unit_sets): names |= {unit.name, unit.symbol, *unit.alternatives, *unit.plurals} if all_forms else {unit.name} - return sorted(names) + return tuple(sorted(names)) def _conversion_to_base_si() -> dict[str, str]: