Repository navigation
fix(examples): stop trusting candidate-reported values in example evaluators - #518
Merged
codelion merged 1 commit intoOct 19, 2026
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #514: the
matmulandfunction_minimizationexample evaluators scored values the candidate program reported about itself. Those values are produced inside the EVOLVE-BLOCK — i.e. under the LLM's control during evolution — so a candidate could score above the state of the art without solving the task.The exploits (verified at HEAD
a1e9d91before the fix)matmul (
examples/alphaevolve_math_problems/matmul/evaluator.py):verify_tensor_decompositionprinted a warning and returned early when any factor array was empty, so verification was silently skipped.BENCHMARK / rankusing the candidate-reportedrank.(np.zeros(0), np.zeros(0), np.zeros(0)), 2, 4, 5, 0.0, 2scored 16.0 (32.0 with a claimed rank of 1); a correct rank-32 decomposition scores 1.0.function_minimization (
examples/function_minimization/evaluator.py):run_search()returned three values, the evaluator used the returnedvaluedirectly.(0.5, 0.5, -1.519)scored 0.793 (value_score1.0) at a point whose true objective value is 0.6931 (honest(0.5, 0.5)return scores 0.449).Changes
(n, m, p)are checked against the fixed<2, 4, 5>target; the rank is derived fromfactor_matrix_1.shape[1]rather than the reported value.(x, y)via a new_objective_functionhelper, in bothevaluate()andevaluate_stage1().tests/test_example_evaluator_integrity.pywith 5 regression tests covering both exploits plus the dimension check.Verification
function_minimization/initial_program.pystill evaluates end-to-end with sensible scores (value_score0.974); a valid rank-40 decomposition with a lied-about rank now correctly scores 32/40.black-clean (line-length 100, per repo config);function_minimization/evaluator.pyhad pre-existing black drift outside the touched hunks, left alone.Closes #514