[x15sr71]

In raising this pull request, I confirm the following (please check boxes):

• I have read and understood the contributors guide.
• I have checked that another pull request for this purpose does not exist.
• I have considered, and confirmed that this submission will be valuable to others.
• I accept that this submission may not be used, and the pull request closed at the will of the maintainer.
• I give this submission freely, and claim no ownership to its content.

My familiarity with the project is as follows (check one):

• I have never used the project.
• I have used the project briefly.
• I have used the project extensively, but have not contributed previously.
• I am an active contributor to the project.

---

Problem

No test has started on the platform since 2026-10-18 02:02 UTC. As of 2026-10-24, 92 queued tests (ids 9669–9809) have never started. They include:

• current ccextractor master d7eb132e: 9750 (Linux) and 9751 (Windows)
• a current PR, ccextractor#2424: 9808 and 9809

Every 10-minute cron run stops at the same test, 9700 (Linux, master aed3c8ae). The platform logs show "Starting test 9700" more than 900 times since Oct 18, and no other test after the early hours of that day. The latest cron.log:

  File "mod_ci/controllers.py", line 565, in start_platforms
  File "mod_ci/controllers.py", line 722, in gcp_instance
  File "mod_ci/controllers.py", line 1166, in start_test
  File "mod_ci/controllers.py", line 1754, in save_xml_to_file
OSError: [Errno 5] Input/output error

The file it fails on is the first one start_test() writes for that test, vm_data/linux-9700/ci-tests/Hardsubx.xml on the bucket mount. Even stat on it returns Input/output error, while the other 14 files in that folder read fine. The night it started, the platform logs show repeated QUOTA_EXCEEDED and ZONE_RESOURCE_POOL_EXHAUSTED retries, which rewrote these files on every cron run.

Root Cause

In gcp_instance(), the PR-status check already catches errors and moves on (log.error(...), then continue), but the start_test() call right below it isn't covered. The exception ends the whole cron run, before any later Linux test is tried and before the Windows half runs at all. Nothing is recorded for 9700, so it's pending again, and first in line, on the next run.

Fix

Wrap the start_test() call the same way: log the exception with its traceback (log.exception) and continue with the next test. The test that failed stays pending and is retried on the next run, which is what already happens for transient GCP and artifact errors, so a temporary bucket problem can't fail someone's PR. start_test() only writes to the database at its very end, through safe_db_commit(), so a test that fails part-way leaves nothing half-written for the next one.

Test

test_start_platforms_continues_after_a_test_fails_to_start: three queued tests (two Linux, one Windows), where the first start_test() call raises OSError(5, 'Input/output error'). It checks that the second Linux test and the Windows test are still started, and that the error is logged once.

• With the fix: fork CI run on Python 3.12–3.15. 807 tests OK (master's 806 plus this one), and all code checks pass.
• Without the fix (same test, controllers.py change reverted): this run fails on every Python version. This test is the only error, raised along the same start_platforms → gcp_instance path as the traceback above.

Notes

• Every cron run picks up all waiting tests by itself. Once 9700 stops blocking, the rest of the 92 waiting tests are started in order. Tests for PRs that have since been closed are descheduled automatically, as they always are. Any test that hits the GCP quota stays queued and starts on a later run, because quota and zone errors are already retried, so the backlog clears over a few runs.
• Test 9700 itself is still retried, and its error logged, on every run, but it no longer blocks anything. It can only run once its broken file is fixed, by removing vm_data/linux-9700/ci-tests/Hardsubx.xml or remounting the bucket. Nothing cancels it automatically. If it shouldn't run, an admin can stop it with the "Stop Test" button in the test list.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[knowprasad]

Tested this locally with Python 3.12:

• Successfully ran the new test test_start_platforms_continues_after_a_test_fails_to_start via TESTING=True nose2 — passed cleanly without regressions.
• Verified that wrapping start_test in try...except prevents unhandled I/O or filesystem exceptions from aborting the entire gcp_instance loop, allowing subsequent queued tests and platforms to proceed.
• Since start_test handles DB commits via safe_db_commit towards the end, early filesystem exceptions leave the DB state clean.

One minor thought for future consideration:
Since a test with persistent corruption remains pending and logs on every cron tick, do we want a retry counter or timeout in the future, or is leaving it pending for admin action preferred as noted?

Overall, very clean and well-scoped fix to unblock the cron queue! LGTM 👍