Skip to content

fix(examples): stop trusting candidate-reported values in example evaluators - #518

Merged
codelion merged 1 commit into
algorithmicsuperintelligence:mainfrom
zichen0116:fix/example-evaluator-integrity
Oct 19, 2026
Merged

codelion merged 1 commit into
algorithmicsuperintelligence:mainfrom
zichen0116:fix/example-evaluator-integrity

Conversation

@zichen0116

Copy link
Copy Markdown
Contributor

Summary

Fixes #514: the matmul and function_minimization example evaluators scored values the candidate program reported about itself. Those values are produced inside the EVOLVE-BLOCK — i.e. under the LLM's control during evolution — so a candidate could score above the state of the art without solving the task.

The exploits (verified at HEAD a1e9d91 before the fix)

matmul (examples/alphaevolve_math_problems/matmul/evaluator.py):

  • verify_tensor_decomposition printed a warning and returned early when any factor array was empty, so verification was silently skipped.
  • The score was BENCHMARK / rank using the candidate-reported rank.
  • A candidate returning (np.zeros(0), np.zeros(0), np.zeros(0)), 2, 4, 5, 0.0, 2 scored 16.0 (32.0 with a claimed rank of 1); a correct rank-32 decomposition scores 1.0.

function_minimization (examples/function_minimization/evaluator.py):

  • When run_search() returned three values, the evaluator used the returned value directly.
  • Returning (0.5, 0.5, -1.519) scored 0.793 (value_score 1.0) at a point whose true objective value is 0.6931 (honest (0.5, 0.5) return scores 0.449).

Changes

  • matmul: empty factor arrays now fail verification (score 0) instead of skipping it; the reported (n, m, p) are checked against the fixed <2, 4, 5> target; the rank is derived from factor_matrix_1.shape[1] rather than the reported value.
  • function_minimization: the objective is always recomputed from the returned (x, y) via a new _objective_function helper, in both evaluate() and evaluate_stage1().
  • Added tests/test_example_evaluator_integrity.py with 5 regression tests covering both exploits plus the dimension check.

Verification

  • Reproduced both exploits on the pre-fix code (scores matched the issue: 16.0 and 0.793); after the fix the cheats score 0.0 / equal-to-honest.
  • New tests: 5/5 pass on fixed code; 4/5 fail on pre-fix code (the 5th locks in the dimension-check behavior, which the old shape validation also rejected).
  • Honest path unchanged: the bundled function_minimization/initial_program.py still evaluates end-to-end with sensible scores (value_score 0.974); a valid rank-40 decomposition with a lied-about rank now correctly scores 32/40.
  • New test file is black-clean (line-length 100, per repo config); function_minimization/evaluator.py had pre-existing black drift outside the touched hunks, left alone.

Closes #514

@CLAassistant

CLAassistant commented Oct 18, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@codelion
codelion merged commit bc36680 into algorithmicsuperintelligence:main Oct 19, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Example evaluators accept candidate-reported values without verification (matmul, function_minimization)

3 participants