openevolve does not correctly clean up when there are failures that lead to retries

Recently was running an experiment with openevolve and our compute cluster was, unbenownst to me, down. This lead to some interesting openevolve behavior -- it would fail, retry a bunch of times based on the config, and then instead of actually exiting when all the retries failed, it would...keep going?

Here's an example of the output logs:

2025-10-11 21:36:42,862 - ERROR - All 1 attempts failed with error: Connection error.                                                                                                                                                                                                                                                                                                     
2025-10-11 21:36:42,862 - DEBUG - Raising connection error                                                                                                                                                                                                                                                                                                                                
2025-10-11 21:36:42,862 - ERROR - LLM generation failed: Connection error.                                                                                                                                                                                                                                                                                                                
2025-10-11 21:36:42,862 - ERROR - LLM generation failed: Connection error.                                                                                                                                                                                                                                                                                                                
2025-10-11 21:36:42,862 - ERROR - All 1 attempts failed with error: Connection error.                                                                                                                                                                                                                                                                                                     
2025-10-11 21:36:42,862 - ERROR - LLM generation failed: Connection error.                                                                                                                                                                                                                                                                                                                
2025-10-11 21:36:42,863 - ERROR - LLM generation failed: Connection error.                                                                                                                                                                                                                                                                                                                
2025-10-11 21:36:42,864 - WARNING - Iteration 147 error: LLM generation failed: Connection error.                                                                                                                                                                                                                                                                                         
2025-10-11 21:36:42,864 - DEBUG - Sampled parent 3b530fc4-8c86-4afe-809a-2773e5d8d917 and 0 
inspirations from island 0                                                                                                                                                                                                                                                                    
2025-10-11 21:36:42,864 - WARNING - Iteration 148 error: LLM generation failed: Connection error.                                                                                                                                                                                                                                                                                         
2025-10-11 21:36:42,865 - DEBUG - Sampled parent 3b530fc4-8c86-4afe-809a-2773e5d8d917 and 0 
inspirations from island 0                                                                                                                                                                                                                                                                    
2025-10-11 21:36:42,865 - WARNING - Iteration 149 error: LLM generation failed: Connection error.                                                                                                                                                                                                                                                                                         
2025-10-11 21:36:42,865 - DEBUG - Using selector: EpollSelector                                                                                                                                                                                                                                                                                                                           
2025-10-11 21:36:42,865 - DEBUG - Sampled parent 3b530fc4-8c86-4afe-809a-2773e5d8d917 and 0 
inspirations from island 0                                                                                                                                                                                                                                                                    
2025-10-11 21:36:42,865 - INFO - Sampled model: gpt-oss-120b                                                                                                                                                                                                                                                                                                                              
2025-10-11 21:36:42,865 - WARNING - Iteration 150 error: LLM generation failed: Connection error.                                                                                                                                                                                                                                                                                         
2025-10-11 21:36:42,865 - DEBUG - Using selector: EpollSelector                                                                                                                                                                                                                                                                                                                           
2025-10-11 21:36:42,865 - DEBUG - Sampled parent 3b530fc4-8c86-4afe-809a-2773e5d8d917 and 0 
inspirations from island 0                                                                                                                                                                                                                                                                    
2025-10-11 21:36:42,866 - INFO - Sampled model: gpt-oss-120b                                                                                                                                                                                                                                                                                                                              
2025-10-11 21:36:42,866 - WARNING - Iteration 151 error: LLM generation failed: Connection error.                                                                                                                                                                                                                                                                                         
2025-10-11 21:36:42,866 - DEBUG - Using selector: EpollSelector                                                                                                                                                                                                                                                                                                                           
2025-10-11 21:36:42,866 - DEBUG - Sampled parent 3b530fc4-8c86-4afe-809a-2773e5d8d917 and 0 
inspirations from island 0                                                                                                                                                                                                                                                                    
2025-10-11 21:36:42,866 - INFO - Sampled model: gpt-oss-120b                                                                                                                                                                                                                                                                                                                              
2025-10-11 21:36:42,866 - WARNING - Iteration 152 error: LLM generation failed: Connection error.                                                                                                                                                                                                                                                                                         
2025-10-11 21:36:42,866 - DEBUG - Using selector: EpollSelector                                                                                                                                                                                                                                                                                                                           
2025-10-11 21:36:42,866 - DEBUG - Sampled parent 3b530fc4-8c86-4afe-809a-2773e5d8d917 and 0 
inspirations from island 0                                                                                                                                                                                                                                                                    
2025-10-11 21:36:42,867 - INFO - Sampled model: gpt-oss-120b                                                                                                                                                                                                                                                                                                                              
2025-10-11 21:36:42,867 - WARNING - Iteration 153 error: LLM generation failed: Connection error.                                                                                                                                                                                                                                                                                         
2025-10-11 21:36:42,867 - DEBUG - Using selector: EpollSelector                                                                                                                                                                                                                                                                                                                           
2025-10-11 21:36:42,867 - DEBUG - Sampled parent 3b530fc4-8c86-4afe-809a-2773e5d8d917 and 0 
inspirations from island 0                                                                                                                                                                                                                                                                    
2025-10-11 21:36:42,867 - INFO - Sampled model: gpt-oss-120b                                                                                                                                                                                                                                                                                                                              
2025-10-11 21:36:42,867 - WARNING - Iteration 154 error: LLM generation failed: Connection error.

Is there any kind of error handling that I am missing here? It seems like this is all happening inside the openevolve run function, so I can't figure out how to stop this behavior without monkeypatching the library.

More generally, I can imagine all kinds of issues with our server that I'd want openevolve to robustly fail for, e.g. if openevolve hits a max token limit or times out our server

Here is how I am currently patching openevolve locally:

 72                 if not hasattr(self, '_consecutive_failures'):
 73                     self._consecutive_failures = 0
 74                     self._max_consecutive_failures = 3
 75
 76                 def _is_connection_error(msg):
 77                     msg_lower = str(msg).lower()
 78                     patterns = ["connection", "network", "unreachable", "timed out",
 79                                 "getaddrinfo failed", "failed to connect", "llm generation failed"]
 80                     return any(p in msg_lower for p in patterns)
 81
 82                 if result.error and _is_connection_error(result.error):
 83                     self._consecutive_failures += 1
 84                     logger.error(
 85                         f"🚨 Connection error in iteration {completed_iteration} "
 86                         f"({self._consecutive_failures}/{self._max_consecutive_failures}): {result.error}"
 87                     )
 88                     if self._consecutive_failures >= self._max_consecutive_failures:
 89                         logger.error(
 90                             f"💥 FATAL: {self._consecutive_failures} consecutive connection errors. "
 91                             "This usually means the LLM server URL is incorrect or unreachable. "
 92                             "Halting execution."
 93                         )
 94                         self.shutdown_event.set()
 95                         break
 96                 elif not result.error or not _is_connection_error(result.error):
 97                     # Reset counter on successful iteration or non-connection error
 98                     if hasattr(self, '_consecutive_failures'):
 99                         self._consecutive_failures p 0
100

added to process_paralle.py

This essentially kills the entire process if it gets 3 consecutive failures, which, with retries, is 3 * num-retries. Could add more patterns in that pattern list too, thats obviously a bit of a kludge. You could make that part of the config, maybe some kind of 'fast-fail-list'?

There may be intermittent errors in long running processes so the design choice here was to retry and let it continue with next iteration. If we are consistently getting failures from the LLM inference layer then there isn't much that can be done. Currently it will proceed to completion but there won't be any useful outputs.

You can imagine how this is problematic though. One issue that we run into is sometimes our openevolve jobs get long enough where they start running into max token errors or backend timeouts. In these cases, we eat tons of resources only for the job to not make any progress at all. On a shared cluster, that usage of resources is costly because we are running compute alloc that could go to someone else.

In the code snippet above, it checks for intermittent failures N times, and if there is ever a success, it rewrites the count. So in our case, we have our openai config retry count set to 5, and this set to 3. Which means it will only fail if we get 15 consecutive LLM connection failures. That's a lot!

Part of why I think this could be configurable is bc it does not seem all that hard to implement a check where you do something like 'if this config value is set, add smarter fast fail options, otherwise just do the default thing'. We can have the cake you're describing and eat it too

Adding a minimal, deterministic repro for this at HEAD a1e9d91 (identical at 80945ed, 2026-03-18), plus one consequence the original report doesn't mention: the run doesn't only keep going — it finishes with exit code 0, prints Evolution complete!, and saves the untouched seed as the best program.

Repro (no API key needed). A local endpoint that answers every request with HTTP 404:

# dead_endpoint.py
import http.server, json
class H(http.server.BaseHTTPRequestHandler):
    def do_POST(self):
        self.rfile.read(int(self.headers.get("Content-Length", 0)))
        body = json.dumps({"error": {"code": "DeploymentNotFound", "message": "deployment does not exist"}}).encode()
        self.send_response(404); self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body))); self.end_headers(); self.wfile.write(body)
    def log_message(self, *a): pass
http.server.HTTPServer(("127.0.0.1", 18404), H).serve_forever()

Then the function_minimization example with api_base: "http://127.0.0.1:18404/v1" in its config:

python openevolve-run.py examples/function_minimization/initial_program.py \
  examples/function_minimization/evaluator.py --config cfg.yaml --output out ; echo "exit=$?"

Observed: every model call returns 404 (logged as WARNING ... Error on attempt 1/4 ... Retrying...), 10 iterations are counted, and the run ends with

Evolution complete!
Best program metrics:
  combined_score: 1.2148
exit=0

out/best/best_program.py is byte-identical to initial_program.py (best_program_info.json: generation 0, iteration 0, parent_id null). Same outcome with HTTP 500 and with HTTP 200 + an empty completion.

Why it matters: scripted pipelines check the exit code and the final banner, so a run with zero successful generations looks like a successful run that simply found nothing better.

Suggested fix: count successful LLM generations; if it is zero at the end, log an error and exit non-zero (e.g. Evolution FAILED: 0/N generations succeeded), and don't present the seed as an evolved best program. Happy to open a PR.

_{Found during an evaluation-integrity audit of LLM-driven program search; logs available. Prepared with Claude Code.}

Navigation Menu

Repository navigation

openevolve does not correctly clean up when there are failures that lead to retries #288

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

openevolve does not correctly clean up when there are failures that lead to retries #288

Description

Activity

theahura commented on Oct 12, 2025

codelion commented on Oct 12, 2025

theahura commented on Oct 12, 2025

iamgahr commented on Oct 14, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions