Improve bigmem tests #75876
Description
Activity
I think that the process of running bigmem tests should be improved.
-
Every bigmem test should run in separate process. It's effect on memory fragmentation shouldn't affect other tests.
-
Only one bigmem test should be run at the same time. Otherwise several parallel bigmem tests can exhaust RAM and lead to swapping. In multiprocessing mode the process running a bigmem test should send a signal to other processes, wait until they finished their current tests and paused, run a bigmem test, and send a signal to other processes that they can continue.
-
The hard limit of addressed memory should be set for every bigmem test, according to the declared requirements. It is better to crash one test and report the failure, than hang on swapping.
-
All other tests should be run with the limit set to 2 GiB (or 1 GiB?). This will help to find memory consuming tests which should be made bigmem tests.
-
The maxrss of the process running a bigmem test should be reported in verbose mode after finishing a test.
Reacted by Zachary WareReacted by Zachary Ware-
+1 to all this :-)
Currently
bigmemtestin verbose mode only reports memory usage on Linux by way of reading/proc. I do a lot of work on macOS and would like to be able to see quickly the memory usage there. Hoping to add Windows and macOS support to help improvebigmemtest. With that also fix 5 above, printing the maxrss after finishing each test.To support that I'd like to add a private C extension function that uses platform-specific APIs to get process memory usage and update
_MemoryWatchdog/memory_watchdog.pyto use that information in its print. That makes it so the platform-specific code lives entirely inside C conditional compilation.The code currently pre-opens the
/procfile and passes that through asstdinto reduce PID recycling issues if the parent exits. Unfortunately there is no cross-platformpidfdtype mechanism to make this really simple. Instead the new code relies on a pipe to the parent to determine "has the parent been lost" which lets us detect parent process termination and prevent issues if the PID is recycled. It also means we should exit more quickly when the parent intentionally exits and gives a good point to add the end of test print.Reacted by Serhiy StorchakaThis issue has not been touched for 9 years, I suggest you opening a new issue for your idea. We can add a private function in a test module such as
_testcapi. I didn't understand the pidfd issue, I should see the code to understand :-)Reacted by Cody Maloney and Serhiy StorchakaCurrently on our buildbot the test takes over one and a half hours, while some of that is unavoidable, it would be good to decrease it a little if possible.
Currently bigmemtest in verbose mode only reports memory usage on Linux by way of reading /proc. I do a lot of work on macOS and would like to be able to see quickly the memory usage there. Hoping to add Windows and macOS support to help improve bigmemtest.
I modified
test.libregrtestto log the memory usage with a newget_process_memory_usage()function intest.libregrtest.utils. The function works on Linux, Windows, macOS and FreeBSD.I also modified
@support.bigmemtestto reuse thisget_process_memory_usage()function.Example:
Using random seed: 2542641886 0:00:00 load avg: 2.46 mem: 23.8 MiB Run 505 tests in parallel using 8 worker processes (timeout: 1 hour 40 min, worker timeout: 1 hour 45 min) (...) 0:22:04 load avg: 5.86 mem: 14.8 GiB [485/505/1] test_descr passed -- running (3): test_bigmem (22 min 3 sec), test_lzma (22 min 3 sec), test_hashlib (4 min 8 sec) 0:22:05 load avg: 5.86 mem: 14.8 GiB [486/505/1] test.test_asyncio.test_streams passed -- running (3): test_bigmem (22 min 4 sec), test_lzma (22 min 4 sec), test_hashlib (4 min 8 sec) 0:22:06 load avg: 5.86 mem: 17.2 GiB [487/505/1] test_mimetypes passed -- running (3): test_bigmem (22 min 5 sec), test_lzma (22 min 5 sec), test_hashlib (4 min 9 sec) (...) 0:41:07 load avg: 2.82 mem: 30.7 GiB [504/505/1] test_lzma passed (41 min 6 sec) -- running (1): test_bigmem (41 min 6 sec) 0:41:38 load avg: 2.29 mem: 30.1 GiB running (1): test_bigmem (41 min 37 sec) (...) 1:31:12 load avg: 1.52 mem: 30.1 GiB running (1): test_bigmem (1 hour 31 min) 1:31:44 load avg: 1.62 mem: 30.1 GiB running (1): test_bigmem (1 hour 31 min) 1:32:14 load avg: 1.72 mem: 16.1 GiB running (1): test_bigmem (1 hour 32 min) 1:32:45 load avg: 1.43 mem: 4.4 GiB running (1): test_bigmem (1 hour 32 min) 1:32:51 load avg: 1.48 mem: 74.4 MiB [505/505/1] test_bigmem passed (1 hour 32 min)The
mem: 14.8 GiBpart is new!Reacted by Cody Maloney- added 5 commits that reference this issue
on Aug 6, 2026 #155302 makes the bigmem tests running in a separate subprocess -- point 1 in my plan. Thanks to reusing the runInSubprocess machinery it is transparent to the user -- you can use all TestCase methods. The follow-ups will implement other points (I already have 3 and 5).
#155307 corrects underspecified memory usage. It was needed to be fixed before imposing the hard limit (point 3). There were only 6 incorrect values in 233 bigmem tests -- for most tests the estimate was very accurate, as result of my past work.
- added a commit that references this issue
on Aug 7, 2026 #156024 implements point 5 -- reports the maxrss and majflt of the process running a bigmem test.
#156023 implements point 2 -- limits the address space of a bigmem test. There are a few subtleties. Glibc reserves some amount of address space per thread per CPU core (it is not really used), so there is an option to skip the address space limitation for tests with large amount of threads. ASan build and macOS also reserve huge address space -- it is skipped there too. For these cases (and for Windows which does not have such API), the watchdog kills the test when it's real use exceeds the limit.
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
Linked PRs