Repository navigation
Test erros on PyTorch Release 25.08 on Series 50 #8587
Description
Describe the bug
Running ./runtests.sh -u -q --net --coverage withing the (b5bc69d, dev, release 1.5.1) version of the repo.
I am sharing the results of b5bc69d, I will be sending the results of the dev branch soon, if there are any differences.
To Reproduce
Steps to reproduce the behavior:
- Clone monai:
git clone https://github.com/Project-MONAI/MONAI.git - Check out the relevant branch:
git checkout b5bc69dorgit checkout dev - Run pytorch:25.08 image:
docker run -v ./:/workspace/ --name torch2508 --gpus=all -it nvcr.io/nvidia/pytorch:25.08-py3 bash - Run the tests via
./runtests.sh -u -q --net --coverage
Expected behavior
Returns no errors.
The following errors are returned from b5bc69d:
======================================================================
ERROR: test_dataloading_img_label_0 (tests.data.test_arraydataset.TestArrayDataset.test_dataloading_img_label_0)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/torch/utils/data/dataloader.py", line 1286, in _try_get_data
data = self._data_queue.get(timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/multiprocessing/queues.py", line 113, in get
if not self._poll(timeout):
^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/multiprocessing/connection.py", line 440, in _poll
r = wait([self], timeout)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/multiprocessing/connection.py", line 1136, in wait
ready = selector.select(timeout)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/utils/data/_utils/signal_handling.py", line 73, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 3473) is killed by signal: Bus error. It is possible that dataloader's workers are out of shared memory. Please try to raise your shared memory limit.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
return func(*(a + p.args), **p.kwargs, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/tests/data/test_arraydataset.py", line 184, in test_dataloading_img_label
new_data = next(iter(loader)) # test batching
^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/utils/data/dataloader.py", line 735, in __next__
data = self._next_data()
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/utils/data/dataloader.py", line 1493, in _next_data
idx, data = self._get_data()
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/utils/data/dataloader.py", line 1455, in _get_data
success, data = self._try_get_data()
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/utils/data/dataloader.py", line 1299, in _try_get_data
raise RuntimeError(
RuntimeError: DataLoader worker (pid(s) 3473) exited unexpectedly
======================================================================
ERROR: test_multiprocess_mapping_file (tests.data.test_mapping_file.TestWriteFileMapping.test_multiprocess_mapping_file)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/torch/utils/data/dataloader.py", line 1286, in _try_get_data
data = self._data_queue.get(timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/multiprocessing/queues.py", line 113, in get
if not self._poll(timeout):
^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/multiprocessing/connection.py", line 440, in _poll
r = wait([self], timeout)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/multiprocessing/connection.py", line 1136, in wait
ready = selector.select(timeout)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/utils/data/_utils/signal_handling.py", line 73, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 4575) is killed by signal: Bus error. It is possible that dataloader's workers are out of shared memory. Please try to raise your shared memory limit.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/workspace/tests/data/test_mapping_file.py", line 102, in test_multiprocess_mapping_file
for _ in multi_loader:
File "/usr/local/lib/python3.12/dist-packages/torch/utils/data/dataloader.py", line 735, in __next__
data = self._next_data()
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/utils/data/dataloader.py", line 1493, in _next_data
idx, data = self._get_data()
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/utils/data/dataloader.py", line 1455, in _get_data
success, data = self._try_get_data()
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/utils/data/dataloader.py", line 1299, in _try_get_data
raise RuntimeError(
RuntimeError: DataLoader worker (pid(s) 4575) exited unexpectedly
======================================================================
ERROR: test_get_history (tests.integration.test_auto3dseg_hpo.TestHPO.test_get_history)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/workspace/tests/integration/test_auto3dseg_hpo.py", line 174, in test_get_history
NNIGen().run_algo(obj_filename, self.work_dir)
File "/workspace/monai/apps/auto3dseg/hpo_gen.py", line 233, in run_algo
self.algo.train(self.params)
File "/workspace/monai/apps/auto3dseg/bundle_gen.py", line 300, in train
return self._run_cmd(cmd)
^^^^^^^^^^^^^^^^^^
File "/workspace/monai/apps/auto3dseg/bundle_gen.py", line 277, in _run_cmd
return run_cmd(cmd.split(), run_cmd_verbose=True, env=ps_environ, check=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/monai/utils/misc.py", line 889, in run_cmd
return subprocess.run(cmd_list, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/subprocess.py", line 571, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['python', '/tmp/tmpflaic_ks/workdir/swinunetr_0_override_None/scripts/train.py', 'run', "--config_file='/tmp/tmpflaic_ks/workdir/swinunetr_0_override_None/configs/hyper_parameters.yaml,/tmp/tmpflaic_ks/workdir/swinunetr_0_override_None/configs/network.yaml,/tmp/tmpflaic_ks/workdir/swinunetr_0_override_None/configs/transforms_infer.yaml,/tmp/tmpflaic_ks/workdir/swinunetr_0_override_None/configs/transforms_train.yaml,/tmp/tmpflaic_ks/workdir/swinunetr_0_override_None/configs/transforms_validate.yaml'"]' returned non-zero exit status 1.
======================================================================
ERROR: test_run_algo (tests.integration.test_auto3dseg_hpo.TestHPO.test_run_algo)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/workspace/tests/integration/test_auto3dseg_hpo.py", line 142, in test_run_algo
NNIGen().run_algo(obj_filename, self.work_dir)
File "/workspace/monai/apps/auto3dseg/hpo_gen.py", line 233, in run_algo
self.algo.train(self.params)
File "/workspace/monai/apps/auto3dseg/bundle_gen.py", line 300, in train
return self._run_cmd(cmd)
^^^^^^^^^^^^^^^^^^
File "/workspace/monai/apps/auto3dseg/bundle_gen.py", line 277, in _run_cmd
return run_cmd(cmd.split(), run_cmd_verbose=True, env=ps_environ, check=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/monai/utils/misc.py", line 889, in run_cmd
return subprocess.run(cmd_list, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/subprocess.py", line 571, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['python', '/tmp/tmpqaxb1d91/workdir/swinunetr_0_override_None/scripts/train.py', 'run', "--config_file='/tmp/tmpqaxb1d91/workdir/swinunetr_0_override_None/configs/hyper_parameters.yaml,/tmp/tmpqaxb1d91/workdir/swinunetr_0_override_None/configs/network.yaml,/tmp/tmpqaxb1d91/workdir/swinunetr_0_override_None/configs/transforms_infer.yaml,/tmp/tmpqaxb1d91/workdir/swinunetr_0_override_None/configs/transforms_train.yaml,/tmp/tmpqaxb1d91/workdir/swinunetr_0_override_None/configs/transforms_validate.yaml'"]' returned non-zero exit status 1.
======================================================================
ERROR: test_run_optuna (tests.integration.test_auto3dseg_hpo.TestHPO.test_run_optuna)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/workspace/tests/integration/test_auto3dseg_hpo.py", line 157, in test_run_optuna
study.optimize(
File "/usr/local/lib/python3.12/dist-packages/optuna/study/study.py", line 490, in optimize
_optimize(
File "/usr/local/lib/python3.12/dist-packages/optuna/study/_optimize.py", line 63, in _optimize
_optimize_sequential(
File "/usr/local/lib/python3.12/dist-packages/optuna/study/_optimize.py", line 160, in _optimize_sequential
frozen_trial_id = _run_trial(study, func, catch)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/optuna/study/_optimize.py", line 258, in _run_trial
raise func_err
File "/usr/local/lib/python3.12/dist-packages/optuna/study/_optimize.py", line 201, in _run_trial
value_or_values = func(trial)
^^^^^^^^^^^
File "/workspace/monai/apps/auto3dseg/hpo_gen.py", line 338, in __call__
self.run_algo(obj_filename, output_folder, template_path)
File "/workspace/monai/apps/auto3dseg/hpo_gen.py", line 396, in run_algo
self.algo.train(self.params)
File "/workspace/monai/apps/auto3dseg/bundle_gen.py", line 300, in train
return self._run_cmd(cmd)
^^^^^^^^^^^^^^^^^^
File "/workspace/monai/apps/auto3dseg/bundle_gen.py", line 277, in _run_cmd
return run_cmd(cmd.split(), run_cmd_verbose=True, env=ps_environ, check=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/monai/utils/misc.py", line 889, in run_cmd
return subprocess.run(cmd_list, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/subprocess.py", line 571, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['python', '/tmp/tmpez132vik/optuna_test/swinunetr_0_override_learning_rate_0.01/scripts/train.py', 'run', "--config_file='/tmp/tmpez132vik/optuna_test/swinunetr_0_override_learning_rate_0.01/configs/hyper_parameters.yaml,/tmp/tmpez132vik/optuna_test/swinunetr_0_override_learning_rate_0.01/configs/network.yaml,/tmp/tmpez132vik/optuna_test/swinunetr_0_override_learning_rate_0.01/configs/transforms_infer.yaml,/tmp/tmpez132vik/optuna_test/swinunetr_0_override_learning_rate_0.01/configs/transforms_train.yaml,/tmp/tmpez132vik/optuna_test/swinunetr_0_override_learning_rate_0.01/configs/transforms_validate.yaml'", '--learning_rate=0.01']' returned non-zero exit status 1.
======================================================================
FAIL: test_affine_028 (tests.transforms.test_affine.TestAffine.test_affine_028)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
return func(*(a + p.args), **p.kwargs, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/tests/transforms/test_affine.py", line 197, in test_affine
test_resampler_lazy(resampler, non_lazy_result, lazy_input_param, input_data, output_idx=output_idx)
File "/workspace/tests/lazy_transforms_utils.py", line 77, in test_resampler_lazy
assert_allclose(lazy_out, non_lazy_out, rtol=rtol, atol=atol)
File "/workspace/tests/test_utils.py", line 140, in assert_allclose
np.testing.assert_allclose(actual, desired, *args, **kwargs)
File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 1504, in assert_allclose
assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
File "/usr/lib/python3.12/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 797, in assert_array_compare
raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=1e-05, atol=1e-07
Mismatched elements: 63 / 64 (98.4%)
Max absolute difference: 0.002
Max relative difference: 0.001
x: array([[[[0. , 0.041667, 0.208333, 0.125 ],
[0.083333, 0.25 , 0.583333, 0.333333],
[0.416667, 0.916667, 1.25 , 0.666667],...
y: array([[[[0. , 0.041728, 0.208516, 0.125183],
[0.083455, 0.250244, 0.583659, 0.3337 ],
[0.417033, 0.917074, 1.250488, 0.667277],...
======================================================================
FAIL: test_affine_029 (tests.transforms.test_affine.TestAffine.test_affine_029)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
return func(*(a + p.args), **p.kwargs, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/tests/transforms/test_affine.py", line 197, in test_affine
test_resampler_lazy(resampler, non_lazy_result, lazy_input_param, input_data, output_idx=output_idx)
File "/workspace/tests/lazy_transforms_utils.py", line 77, in test_resampler_lazy
assert_allclose(lazy_out, non_lazy_out, rtol=rtol, atol=atol)
File "/workspace/tests/test_utils.py", line 140, in assert_allclose
np.testing.assert_allclose(actual, desired, *args, **kwargs)
File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 1504, in assert_allclose
assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
File "/usr/lib/python3.12/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 797, in assert_array_compare
raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=1e-05, atol=1e-07
Mismatched elements: 63 / 64 (98.4%)
Max absolute difference: 0.002
Max relative difference: 0.001
x: array([[[[0.25 , 0.416667, 0.083333, 0. ],
[0.541667, 0.916667, 0.25 , 0.041667],
[0.708333, 1.25 , 0.583333, 0.208333],...
y: array([[[[0.250366, 0.417033, 0.083455, 0. ],
[0.542216, 0.917074, 0.250244, 0.041728],
[0.709005, 1.250488, 0.583659, 0.208516],...
======================================================================
FAIL: test_affine_058 (tests.transforms.test_affine.TestAffine.test_affine_058)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
return func(*(a + p.args), **p.kwargs, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/tests/transforms/test_affine.py", line 197, in test_affine
test_resampler_lazy(resampler, non_lazy_result, lazy_input_param, input_data, output_idx=output_idx)
File "/workspace/tests/lazy_transforms_utils.py", line 77, in test_resampler_lazy
assert_allclose(lazy_out, non_lazy_out, rtol=rtol, atol=atol)
File "/workspace/tests/test_utils.py", line 140, in assert_allclose
np.testing.assert_allclose(actual, desired, *args, **kwargs)
File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 1504, in assert_allclose
assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
File "/usr/lib/python3.12/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 797, in assert_array_compare
raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=1e-05, atol=1e-07
Mismatched elements: 63 / 64 (98.4%)
Max absolute difference: 0.002
Max relative difference: 0.001
x: array([[[[0. , 0.041667, 0.208333, 0.125 ],
[0.083333, 0.25 , 0.583333, 0.333333],
[0.416667, 0.916667, 1.25 , 0.666667],...
y: array([[[[0. , 0.041728, 0.208516, 0.125183],
[0.083455, 0.250244, 0.583659, 0.3337 ],
[0.417033, 0.917074, 1.250488, 0.667277],...
======================================================================
FAIL: test_affine_059 (tests.transforms.test_affine.TestAffine.test_affine_059)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
return func(*(a + p.args), **p.kwargs, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/tests/transforms/test_affine.py", line 197, in test_affine
test_resampler_lazy(resampler, non_lazy_result, lazy_input_param, input_data, output_idx=output_idx)
File "/workspace/tests/lazy_transforms_utils.py", line 77, in test_resampler_lazy
assert_allclose(lazy_out, non_lazy_out, rtol=rtol, atol=atol)
File "/workspace/tests/test_utils.py", line 140, in assert_allclose
np.testing.assert_allclose(actual, desired, *args, **kwargs)
File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 1504, in assert_allclose
assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
File "/usr/lib/python3.12/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 797, in assert_array_compare
raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=1e-05, atol=1e-07
Mismatched elements: 63 / 64 (98.4%)
Max absolute difference: 0.002
Max relative difference: 0.001
x: array([[[[0.25 , 0.416667, 0.083333, 0. ],
[0.541667, 0.916667, 0.25 , 0.041667],
[0.708333, 1.25 , 0.583333, 0.208333],...
y: array([[[[0.250366, 0.417033, 0.083455, 0. ],
[0.542216, 0.917074, 0.250244, 0.041728],
[0.709005, 1.250488, 0.583659, 0.208516],...
======================================================================
FAIL: test_affine_088 (tests.transforms.test_affine.TestAffine.test_affine_088)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
return func(*(a + p.args), **p.kwargs, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/tests/transforms/test_affine.py", line 197, in test_affine
test_resampler_lazy(resampler, non_lazy_result, lazy_input_param, input_data, output_idx=output_idx)
File "/workspace/tests/lazy_transforms_utils.py", line 77, in test_resampler_lazy
assert_allclose(lazy_out, non_lazy_out, rtol=rtol, atol=atol)
File "/workspace/tests/test_utils.py", line 140, in assert_allclose
np.testing.assert_allclose(actual, desired, *args, **kwargs)
File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 1504, in assert_allclose
assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
File "/usr/lib/python3.12/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 797, in assert_array_compare
raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=1e-05, atol=1e-07
Mismatched elements: 63 / 64 (98.4%)
Max absolute difference: 0.002
Max relative difference: 0.001
x: array([[[[0. , 0.041667, 0.208333, 0.125 ],
[0.083333, 0.25 , 0.583333, 0.333333],
[0.416667, 0.916667, 1.25 , 0.666667],...
y: array([[[[0. , 0.041728, 0.208516, 0.125183],
[0.083455, 0.250244, 0.583659, 0.3337 ],
[0.417033, 0.917074, 1.250488, 0.667277],...
======================================================================
FAIL: test_affine_089 (tests.transforms.test_affine.TestAffine.test_affine_089)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
return func(*(a + p.args), **p.kwargs, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/tests/transforms/test_affine.py", line 197, in test_affine
test_resampler_lazy(resampler, non_lazy_result, lazy_input_param, input_data, output_idx=output_idx)
File "/workspace/tests/lazy_transforms_utils.py", line 77, in test_resampler_lazy
assert_allclose(lazy_out, non_lazy_out, rtol=rtol, atol=atol)
File "/workspace/tests/test_utils.py", line 140, in assert_allclose
np.testing.assert_allclose(actual, desired, *args, **kwargs)
File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 1504, in assert_allclose
assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
File "/usr/lib/python3.12/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 797, in assert_array_compare
raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=1e-05, atol=1e-07
Mismatched elements: 63 / 64 (98.4%)
Max absolute difference: 0.002
Max relative difference: 0.001
x: array([[[[0.25 , 0.416667, 0.083333, 0. ],
[0.541667, 0.916667, 0.25 , 0.041667],
[0.708333, 1.25 , 0.583333, 0.208333],...
y: array([[[[0.250366, 0.417033, 0.083455, 0. ],
[0.542216, 0.917074, 0.250244, 0.041728],
[0.709005, 1.250488, 0.583659, 0.208516],...
======================================================================
FAIL: test_affine_118 (tests.transforms.test_affine.TestAffine.test_affine_118)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
return func(*(a + p.args), **p.kwargs, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/tests/transforms/test_affine.py", line 197, in test_affine
test_resampler_lazy(resampler, non_lazy_result, lazy_input_param, input_data, output_idx=output_idx)
File "/workspace/tests/lazy_transforms_utils.py", line 77, in test_resampler_lazy
assert_allclose(lazy_out, non_lazy_out, rtol=rtol, atol=atol)
File "/workspace/tests/test_utils.py", line 140, in assert_allclose
np.testing.assert_allclose(actual, desired, *args, **kwargs)
File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 1504, in assert_allclose
assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
File "/usr/lib/python3.12/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 797, in assert_array_compare
raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=1e-05, atol=1e-07
Mismatched elements: 63 / 64 (98.4%)
Max absolute difference: 0.002
Max relative difference: 0.001
x: array([[[[0. , 0.041667, 0.208333, 0.125 ],
[0.083333, 0.25 , 0.583333, 0.333333],
[0.416667, 0.916667, 1.25 , 0.666667],...
y: array([[[[0. , 0.041728, 0.208516, 0.125183],
[0.083455, 0.250244, 0.583659, 0.3337 ],
[0.417033, 0.917074, 1.250488, 0.667277],...
======================================================================
FAIL: test_affine_119 (tests.transforms.test_affine.TestAffine.test_affine_119)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
return func(*(a + p.args), **p.kwargs, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/tests/transforms/test_affine.py", line 197, in test_affine
test_resampler_lazy(resampler, non_lazy_result, lazy_input_param, input_data, output_idx=output_idx)
File "/workspace/tests/lazy_transforms_utils.py", line 77, in test_resampler_lazy
assert_allclose(lazy_out, non_lazy_out, rtol=rtol, atol=atol)
File "/workspace/tests/test_utils.py", line 140, in assert_allclose
np.testing.assert_allclose(actual, desired, *args, **kwargs)
File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 1504, in assert_allclose
assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
File "/usr/lib/python3.12/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 797, in assert_array_compare
raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=1e-05, atol=1e-07
Mismatched elements: 63 / 64 (98.4%)
Max absolute difference: 0.002
Max relative difference: 0.001
x: array([[[[0.25 , 0.416667, 0.083333, 0. ],
[0.541667, 0.916667, 0.25 , 0.041667],
[0.708333, 1.25 , 0.583333, 0.208333],...
y: array([[[[0.250366, 0.417033, 0.083455, 0. ],
[0.542216, 0.917074, 0.250244, 0.041728],
[0.709005, 1.250488, 0.583659, 0.208516],...
======================================================================
FAIL: test_affine_028 (tests.transforms.test_affined.TestAffined.test_affine_028)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
return func(*(a + p.args), **p.kwargs, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/tests/transforms/test_affined.py", line 186, in test_affine
test_resampler_lazy(resampler, non_lazy_result, lazy_input_param, call_param, output_key="img")
File "/workspace/tests/lazy_transforms_utils.py", line 77, in test_resampler_lazy
assert_allclose(lazy_out, non_lazy_out, rtol=rtol, atol=atol)
File "/workspace/tests/test_utils.py", line 140, in assert_allclose
np.testing.assert_allclose(actual, desired, *args, **kwargs)
File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 1504, in assert_allclose
assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
File "/usr/lib/python3.12/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 797, in assert_array_compare
raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=1e-05, atol=1e-07
Mismatched elements: 63 / 64 (98.4%)
Max absolute difference: 0.002
Max relative difference: 0.001
x: array([[[[0. , 0.041667, 0.208333, 0.125 ],
[0.083333, 0.25 , 0.583333, 0.333333],
[0.416667, 0.916667, 1.25 , 0.666667],...
y: array([[[[0. , 0.041728, 0.208516, 0.125183],
[0.083455, 0.250244, 0.583659, 0.3337 ],
[0.417033, 0.917074, 1.250488, 0.667277],...
======================================================================
FAIL: test_affine_029 (tests.transforms.test_affined.TestAffined.test_affine_029)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
return func(*(a + p.args), **p.kwargs, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/tests/transforms/test_affined.py", line 186, in test_affine
test_resampler_lazy(resampler, non_lazy_result, lazy_input_param, call_param, output_key="img")
File "/workspace/tests/lazy_transforms_utils.py", line 77, in test_resampler_lazy
assert_allclose(lazy_out, non_lazy_out, rtol=rtol, atol=atol)
File "/workspace/tests/test_utils.py", line 140, in assert_allclose
np.testing.assert_allclose(actual, desired, *args, **kwargs)
File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 1504, in assert_allclose
assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
File "/usr/lib/python3.12/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 797, in assert_array_compare
raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=1e-05, atol=1e-07
Mismatched elements: 63 / 64 (98.4%)
Max absolute difference: 0.002
Max relative difference: 0.001
x: array([[[[0.25 , 0.416667, 0.083333, 0. ],
[0.541667, 0.916667, 0.25 , 0.041667],
[0.708333, 1.25 , 0.583333, 0.208333],...
y: array([[[[0.250366, 0.417033, 0.083455, 0. ],
[0.542216, 0.917074, 0.250244, 0.041728],
[0.709005, 1.250488, 0.583659, 0.208516],...
======================================================================
FAIL: test_affine_058 (tests.transforms.test_affined.TestAffined.test_affine_058)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
return func(*(a + p.args), **p.kwargs, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/tests/transforms/test_affined.py", line 186, in test_affine
test_resampler_lazy(resampler, non_lazy_result, lazy_input_param, call_param, output_key="img")
File "/workspace/tests/lazy_transforms_utils.py", line 77, in test_resampler_lazy
assert_allclose(lazy_out, non_lazy_out, rtol=rtol, atol=atol)
File "/workspace/tests/test_utils.py", line 140, in assert_allclose
np.testing.assert_allclose(actual, desired, *args, **kwargs)
File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 1504, in assert_allclose
assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
File "/usr/lib/python3.12/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 797, in assert_array_compare
raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=1e-05, atol=1e-07
Mismatched elements: 63 / 64 (98.4%)
Max absolute difference: 0.002
Max relative difference: 0.001
x: array([[[[0. , 0.041667, 0.208333, 0.125 ],
[0.083333, 0.25 , 0.583333, 0.333333],
[0.416667, 0.916667, 1.25 , 0.666667],...
y: array([[[[0. , 0.041728, 0.208516, 0.125183],
[0.083455, 0.250244, 0.583659, 0.3337 ],
[0.417033, 0.917074, 1.250488, 0.667277],...
======================================================================
FAIL: test_affine_059 (tests.transforms.test_affined.TestAffined.test_affine_059)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
return func(*(a + p.args), **p.kwargs, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/tests/transforms/test_affined.py", line 186, in test_affine
test_resampler_lazy(resampler, non_lazy_result, lazy_input_param, call_param, output_key="img")
File "/workspace/tests/lazy_transforms_utils.py", line 77, in test_resampler_lazy
assert_allclose(lazy_out, non_lazy_out, rtol=rtol, atol=atol)
File "/workspace/tests/test_utils.py", line 140, in assert_allclose
np.testing.assert_allclose(actual, desired, *args, **kwargs)
File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 1504, in assert_allclose
assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
File "/usr/lib/python3.12/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 797, in assert_array_compare
raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=1e-05, atol=1e-07
Mismatched elements: 63 / 64 (98.4%)
Max absolute difference: 0.002
Max relative difference: 0.001
x: array([[[[0.25 , 0.416667, 0.083333, 0. ],
[0.541667, 0.916667, 0.25 , 0.041667],
[0.708333, 1.25 , 0.583333, 0.208333],...
y: array([[[[0.250366, 0.417033, 0.083455, 0. ],
[0.542216, 0.917074, 0.250244, 0.041728],
[0.709005, 1.250488, 0.583659, 0.208516],...
======================================================================
FAIL: test_affine_088 (tests.transforms.test_affined.TestAffined.test_affine_088)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
return func(*(a + p.args), **p.kwargs, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/tests/transforms/test_affined.py", line 186, in test_affine
test_resampler_lazy(resampler, non_lazy_result, lazy_input_param, call_param, output_key="img")
File "/workspace/tests/lazy_transforms_utils.py", line 77, in test_resampler_lazy
assert_allclose(lazy_out, non_lazy_out, rtol=rtol, atol=atol)
File "/workspace/tests/test_utils.py", line 140, in assert_allclose
np.testing.assert_allclose(actual, desired, *args, **kwargs)
File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 1504, in assert_allclose
assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
File "/usr/lib/python3.12/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 797, in assert_array_compare
raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=1e-05, atol=1e-07
Mismatched elements: 63 / 64 (98.4%)
Max absolute difference: 0.002
Max relative difference: 0.001
x: array([[[[0. , 0.041667, 0.208333, 0.125 ],
[0.083333, 0.25 , 0.583333, 0.333333],
[0.416667, 0.916667, 1.25 , 0.666667],...
y: array([[[[0. , 0.041728, 0.208516, 0.125183],
[0.083455, 0.250244, 0.583659, 0.3337 ],
[0.417033, 0.917074, 1.250488, 0.667277],...
======================================================================
FAIL: test_affine_089 (tests.transforms.test_affined.TestAffined.test_affine_089)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
return func(*(a + p.args), **p.kwargs, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/tests/transforms/test_affined.py", line 186, in test_affine
test_resampler_lazy(resampler, non_lazy_result, lazy_input_param, call_param, output_key="img")
File "/workspace/tests/lazy_transforms_utils.py", line 77, in test_resampler_lazy
assert_allclose(lazy_out, non_lazy_out, rtol=rtol, atol=atol)
File "/workspace/tests/test_utils.py", line 140, in assert_allclose
np.testing.assert_allclose(actual, desired, *args, **kwargs)
File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 1504, in assert_allclose
assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
File "/usr/lib/python3.12/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 797, in assert_array_compare
raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=1e-05, atol=1e-07
Mismatched elements: 63 / 64 (98.4%)
Max absolute difference: 0.002
Max relative difference: 0.001
x: array([[[[0.25 , 0.416667, 0.083333, 0. ],
[0.541667, 0.916667, 0.25 , 0.041667],
[0.708333, 1.25 , 0.583333, 0.208333],...
y: array([[[[0.250366, 0.417033, 0.083455, 0. ],
[0.542216, 0.917074, 0.250244, 0.041728],
[0.709005, 1.250488, 0.583659, 0.208516],...
======================================================================
FAIL: test_affine_118 (tests.transforms.test_affined.TestAffined.test_affine_118)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
return func(*(a + p.args), **p.kwargs, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/tests/transforms/test_affined.py", line 186, in test_affine
test_resampler_lazy(resampler, non_lazy_result, lazy_input_param, call_param, output_key="img")
File "/workspace/tests/lazy_transforms_utils.py", line 77, in test_resampler_lazy
assert_allclose(lazy_out, non_lazy_out, rtol=rtol, atol=atol)
File "/workspace/tests/test_utils.py", line 140, in assert_allclose
np.testing.assert_allclose(actual, desired, *args, **kwargs)
File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 1504, in assert_allclose
assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
File "/usr/lib/python3.12/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 797, in assert_array_compare
raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=1e-05, atol=1e-07
Mismatched elements: 63 / 64 (98.4%)
Max absolute difference: 0.002
Max relative difference: 0.001
x: array([[[[0. , 0.041667, 0.208333, 0.125 ],
[0.083333, 0.25 , 0.583333, 0.333333],
[0.416667, 0.916667, 1.25 , 0.666667],...
y: array([[[[0. , 0.041728, 0.208516, 0.125183],
[0.083455, 0.250244, 0.583659, 0.3337 ],
[0.417033, 0.917074, 1.250488, 0.667277],...
======================================================================
FAIL: test_affine_119 (tests.transforms.test_affined.TestAffined.test_affine_119)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
return func(*(a + p.args), **p.kwargs, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/tests/transforms/test_affined.py", line 186, in test_affine
test_resampler_lazy(resampler, non_lazy_result, lazy_input_param, call_param, output_key="img")
File "/workspace/tests/lazy_transforms_utils.py", line 77, in test_resampler_lazy
assert_allclose(lazy_out, non_lazy_out, rtol=rtol, atol=atol)
File "/workspace/tests/test_utils.py", line 140, in assert_allclose
np.testing.assert_allclose(actual, desired, *args, **kwargs)
File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 1504, in assert_allclose
assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
File "/usr/lib/python3.12/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 797, in assert_array_compare
raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=1e-05, atol=1e-07
Mismatched elements: 63 / 64 (98.4%)
Max absolute difference: 0.002
Max relative difference: 0.001
x: array([[[[0.25 , 0.416667, 0.083333, 0. ],
[0.541667, 0.916667, 0.25 , 0.041667],
[0.708333, 1.25 , 0.583333, 0.208333],...
y: array([[[[0.250366, 0.417033, 0.083455, 0. ],
[0.542216, 0.917074, 0.250244, 0.041728],
[0.709005, 1.250488, 0.583659, 0.208516],...
======================================================================
FAIL: test_flash_attention_0 (tests.networks.blocks.test_crossattention.TestResBlock.test_flash_attention_0)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
return func(*(a + p.args), **p.kwargs, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/tests/networks/blocks/test_crossattention.py", line 174, in test_flash_attention
assert_allclose(out_1, out_2, atol=1e-4)
File "/workspace/tests/test_utils.py", line 140, in assert_allclose
np.testing.assert_allclose(actual, desired, *args, **kwargs)
File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 1504, in assert_allclose
assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
File "/usr/lib/python3.12/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 797, in assert_array_compare
raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=1e-07, atol=0.0001
Mismatched elements: 219 / 2048 (10.7%)
Max absolute difference: 0.
Max relative difference: 0.835
x: array([[[ 0.0093 , -0.26159 , -0.350302, ..., 0.028707, 0.324385,
0.086146],
[-0.059419, -0.61913 , 0.034504, ..., -0.658143, -0.110547,...
y: array([[[ 0.0093 , -0.26159 , -0.350302, ..., 0.028707, 0.324385,
0.086146],
[-0.059226, -0.61921 , 0.034631, ..., -0.658143, -0.110504,...
======================================================================
FAIL: test_flash_attention_1 (tests.networks.blocks.test_crossattention.TestResBlock.test_flash_attention_1)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
return func(*(a + p.args), **p.kwargs, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/tests/networks/blocks/test_crossattention.py", line 174, in test_flash_attention
assert_allclose(out_1, out_2, atol=1e-4)
File "/workspace/tests/test_utils.py", line 140, in assert_allclose
np.testing.assert_allclose(actual, desired, *args, **kwargs)
File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 1504, in assert_allclose
assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
File "/usr/lib/python3.12/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 797, in assert_array_compare
raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=1e-07, atol=0.0001
Mismatched elements: 16 / 2048 (0.781%)
Max absolute difference: 0.
Max relative difference: 0.511
x: array([[[-0.029829, 0.030035, -0.015213, ..., 0.130288, 0.059629,
0.06318 ],
[-0.062274, -0.03078 , -0.053948, ..., 0.15801 , 0.027424,...
y: array([[[-0.029799, 0.030049, -0.015215, ..., 0.130326, 0.059602,
0.063214],
[-0.06229 , -0.030772, -0.053921, ..., 0.15801 , 0.027432,...
----------------------------------------------------------------------
Ran 17268 tests in 1100.158s
FAILED (failures=18, errors=5, skipped=991)
tests finished, printing completed times >10.0s in ascending order...
test_gen_patches_0 (tests.data.test_masked_patch_wsi_dataset.TestSlidingPatchWSIDatasetOpenSlide.test_gen_patches_0) (10.1s)
test_content_1 (tests.handlers.test_handler_garbage_collector.TestHandlerGarbageCollector.test_content_1) (13.5s)
test_script (tests.networks.nets.test_highresnet.TestHighResNet.test_script) (14.9s)
test_against_official_code_using_cg (tests.transforms.test_ultrasound_confidence_map_transform.TestUltrasoundConfidenceMapTransform.test_against_official_code_using_cg) (16.1s)
test_start (tests.integration.test_loader_semaphore.TestImportLock.test_start) (16.1s)
test_pdb (tests.bundle.test_config_parser.TestConfigParser.test_pdb) (16.3s)
test_verify_0__workspace_tests_testing_data_metadata_json (tests.bundle.test_bundle_verify_metadata.TestVerifyMetaData.test_verify_0__workspace_tests_testing_data_metadata_json) (21.5s)
test_verify_0__workspace_tests_testing_data_metadata_json (tests.bundle.test_bundle_verify_net.TestVerifyNetwork.test_verify_0__workspace_tests_testing_data_metadata_json) (21.6s)
test_bundle (tests.networks.nets.test_bundle_init_bundle.TestBundleInit.test_bundle) (21.7s)
test_default_value_1_model (tests.bundle.test_bundle_ckpt_export.TestCKPTExport.test_default_value_1_model) (21.8s)
test_export_1_model (tests.bundle.test_bundle_ckpt_export.TestCKPTExport.test_export_1_model) (21.9s)
test_default_value_0_ (tests.bundle.test_bundle_ckpt_export.TestCKPTExport.test_default_value_0_) (21.9s)
test_verify_fp16_0__workspace_tests_testing_data_metadata_json (tests.bundle.test_bundle_verify_net.TestVerifyNetwork.test_verify_fp16_0__workspace_tests_testing_data_metadata_json) (22.1s)
test_export_0_ (tests.bundle.test_bundle_ckpt_export.TestCKPTExport.test_export_0_) (22.2s)
test_export_2_model (tests.bundle.test_bundle_ckpt_export.TestCKPTExport.test_export_2_model) (22.5s)
test_default_value_2_model (tests.bundle.test_bundle_ckpt_export.TestCKPTExport.test_default_value_2_model) (22.5s)
test_multiprocess_mapping_file (tests.data.test_mapping_file.TestWriteFileMapping.test_multiprocess_mapping_file) (35.7s)
test_run_algo (tests.integration.test_auto3dseg_hpo.TestHPO.test_run_algo) (41.5s)
test_run_optuna (tests.integration.test_auto3dseg_hpo.TestHPO.test_run_optuna) (41.7s)
test_get_history (tests.integration.test_auto3dseg_hpo.TestHPO.test_get_history) (42.1s)
test discovery time: 5.56s
total testing time: 4.68e+02s
Remember to check above times for any errors!Environment
Ensuring you use the relevant python executable, please paste the output of:
python -c "import monai; monai.config.print_debug_info()"
TensorRT-LLM is not installed. Please install TensorRT-LLM or set TRTLLM_PLUGINS_PATH to the directory containing libnvinfer_plugin_tensorrt_llm.so to use converters for torch.distributed ops
================================
Printing MONAI config...
================================
MONAI version: 0+unknown
Numpy version: 1.26.4
Pytorch version: 2.8.0a0+34c6371d24.nv25.08
MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
MONAI rev id: None
MONAI __file__: /workspace/monai/__init__.py
Optional dependencies:
Pytorch Ignite version: 0.4.11
ITK version: 5.4.4
Nibabel version: 5.3.2
scikit-image version: 0.25.2
scipy version: 1.15.3
Pillow version: 11.3.0
Tensorboard version: 2.16.2
gdown version: 5.2.0
TorchVision version: 0.23.0a0+428a54c9
tqdm version: 4.67.1
lmdb version: 1.7.3
psutil version: 7.0.0
pandas version: 2.2.3
einops version: 0.8.1
transformers version: NOT INSTALLED or UNKNOWN VERSION.
mlflow version: 3.4.0
pynrrd version: 1.1.3
clearml version: 2.0.3rc0
For details about installing the optional dependencies, please visit:
https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies
================================
Printing system config...
================================
System: Linux
Linux version: Ubuntu 24.04.2 LTS
Platform: Linux-6.8.0-83-generic-x86_64-with-glibc2.39
Processor: x86_64
Machine: x86_64
Python version: 3.12.3
Process name: python
Command: ['python', '-c', 'import monai; monai.config.print_debug_info()']
Open files: []
Num physical CPUs: 16
Num logical CPUs: 32
Num usable CPUs: 32
CPU usage (%): [2.1, 41.4, 1.6, 2.1, 1.6, 1.6, 1.6, 1.6, 1.7, 1.9, 1.6, 1.6, 1.6, 1.6, 1.6, 28.0, 1.6, 15.8, 1.9, 1.6, 1.6, 1.6, 1.6, 2.1, 3.7, 1.6, 1.6, 1.4, 1.6, 1.6, 1.6, 17.5]
CPU freq. (MHz): 2373
Load avg. in last 1, 5, 15 mins (%): [3.0, 9.8, 8.8]
Disk usage (%): 13.7
Avg. sensor temp. (Celsius): UNKNOWN for given OS
Total physical memory (GB): 125.7
Available memory (GB): 115.2
Used memory (GB): 9.2
================================
Printing GPU config...
================================
Num GPUs: 1
Has CUDA: True
CUDA version: 13.0
cuDNN enabled: True
NVIDIA_TF32_OVERRIDE: None
TORCH_ALLOW_TF32_CUBLAS_OVERRIDE: 1
cuDNN version: 91200
Current device: 0
Library compiled for CUDA architectures: ['sm_75', 'sm_80', 'sm_86', 'sm_90', 'sm_100', 'sm_120', 'compute_120']
GPU 0 Name: NVIDIA GeForce RTX 5090
GPU 0 Is integrated: False
GPU 0 Is multi GPU board: False
GPU 0 Multi processor count: 170
GPU 0 Total memory (GB): 31.4
GPU 0 CUDA capability (maj.min): 12.0Additional context
Activity
garciadias commentedon Sep 26, 2025
From dev, there are two more errors, and 38 more tests are skipped.
Environment:
python -c "import monai; monai.config.print_debug_info()"garciadias commentedon Nov 28, 2025
Results with 20.10
Running
./runtests.sh -u -qpresents the same 18 failed tolerance tests.Running
./runtests.sh -u -q --netgets stuck attests/integration/test_auto3dseg_hpo.py:test_run_optuna.It hangs there indefinetly (I waited over 24h).
Commenting this test out, only the 18 failed tolerance tests remain failing.
ericspod commentedon Dec 1, 2025
I've been working on a few issues as well in #8640 which I think intersect with some of the errors. Some of the fails may be addressed by the changes I've done there, some others are from using Docker without enough shared memory so I've added
--shm-size=10gbto thedocker runcommand. The image I'm working with there also uses Python 3.11 so the different versions might produce different results. If we have only tolerance fails we may just have to adjust the tolerance levels for specific hardware for now, but really it shouldn't be so far out like this without some underlying cause we can look into.garciadias commentedon Dec 18, 2025
Hi Eric, sorry for the late reply.
I have run the tests on your branch and using the
--shm-size=10gbtag, as you suggested.The same 18 tolerance fails are still showing. And this time, I got an extra error with ONNX. Is there something else you would like me to try?
garciadias commentedon Jan 23, 2026
Hi @ericspod , I have applied the latest changes to #8640, and keep getting the same 18 tolerance errors. Did those resolved to you?
ericspod commentedon Jan 23, 2026
That PR is getting a bit behind, I was going to wait until the other recent ones are merged after the patch release before updating it. I didn't see those tolerance issues myself at the time though.
fix: address PR Project-MONAI#8770 review feedback (Project-MONAI#8587)
fix: update slim image and runtime check for Blackwell (sm_120) suppo…
fix: address PR Project-MONAI#8770 review feedback (Project-MONAI#8587)
8587 test erros on pytorch release 2508 on series 50 (#8770)