[x15sr71]

In raising this pull request, I confirm the following (please check boxes):

• I have read and understood the contributors guide.
• I have checked that another pull request for this purpose does not exist.
• I have considered, and confirmed that this submission will be valuable to others.
• I accept that this submission may not be used, and the pull request closed at the will of the maintainer.
• I give this submission freely, and claim no ownership to its content.

My familiarity with the project is as follows (check one):

• I have never used the project.
• I have used the project briefly.
• I have used the project extensively, but have not contributed previously.
• I am an active contributor to the project.

---

Problem

Since 2026-10-25 06:07 UTC, no test can start: /repository returns Transport endpoint is not connected. The VM ran out of memory, and the kernel's OOM killer killed gcsfuse:

06:07:33 systemd[1]: repository.mount: A process of this unit has been killed by the OOM killer.
06:07:34 kernel: Out of memory: Killed process 673 (gcsfuse) ...

Before that, from 00:48 UTC, it had killed mysqld 17 times.

Root Cause

Cron starts cron.py every 10 minutes, and nothing stops two runs from overlapping.

• While the queue was stuck: each run ended within seconds, so runs didn't overlap.
• Since fix(ci): keep the cron queue moving when one test fails to start #1244: a run works through the backlog and takes 4–35 minutes, so runs pile up.
• At every OOM kill on Oct 25: 2–4 cron processes were running, using 425–634 MB together, on a VM with 2 GB of RAM and no swap.

Fix

cron() takes a non-blocking fcntl.flock on a lock file in the temp directory. If a previous run still holds it, the new run logs a warning and exits. The kernel releases the lock when the process exits, even if it's killed.

The change is in cron.py, so the normal deploy picks it up and the crontab doesn't change.

Test

• Unit test: test_cron_job_skips_while_previous_run_holds_lock. With the lock held, cron() doesn't call start_platforms and logs the warning. Fork CI run: 808 tests OK on Python 3.12–3.15, and all code checks pass.
• Real processes on Ubuntu 24.04, running this cron.py with the platform modules stubbed and start_platforms sleeping:
  • A second run started during the first is skipped.
  • After the first run is killed with SIGKILL, the next run takes the lock.
  • After a normal finish, the next run proceeds.

Note

The bucket still needs to be remounted on the VM. It's best done after this is deployed, so the backlog doesn't fill the memory again.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[knowprasad]

Took a look through mod_ci/cron.py and the test additions. Preventing overlapping cron runs to avoid OOM killer spikes on low-memory VMs makes complete sense.

A couple of small notes from looking at the implementation:

1. import fcntl is imported at module top-level in mod_ci/cron.py. Since fcntl is Unix-only, this raises ModuleNotFoundError for contributors running or testing on Windows. Wrapping it with a fallback guard (e.g. try: import fcntl except ImportError: fcntl = None) ensures it won't break local development environments.

2. In cron(), open(LOCK_FILE, w) truncates the lockfile on each invocation. Opening in append or read/write mode (a or os.O_CREAT | os.O_RDWR) avoids modifying/truncating the file while another process holds an active lock.

3. When cron() completes normally, lock.close() is never called, so the file descriptor relies on GC or process exit to release. Wrapping the core cron body in a try...finally block to call lock.close() keeps cleanup explicit.

Overall a very clean and needed fix for the queue backlog.