You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
microvm-worker: a failed timer start logs under the listener lock #832
The MicroVM worker's listener holds self._lock while it starts its idle timer (_start_idle_time) and its grace timer (_start_grace). Every caller of both methods holds the lock.
When threading.Timer.start() raises, each method logs an error through the caller's logger, with describe(error), while it still holds the lock.
The caller passes that logger. safe_logger catches its exceptions, but it does not move the call outside the lock.
The lock is not reentrant.
So a logger that is slow blocks every hook and job request. A logger that calls back into the listener, for example a handler that calls close(), deadlocks it.
The error itself comes from threading.Timer.start(), not from a caller's client. So the case needs a failed thread start and a logger that blocks or calls the listener. #831 fixed the same pattern for the failed-SuspendMicrovm warning.
Where
packages/aws-durable-execution-sdk-python-microvm-worker/src/aws_durable_execution_sdk_python_microvm_worker/worker.py, in _start_idle_time ("could not start the idle time") and _start_grace ("could not start the end of the refusal").
Proposed fix
Record the failure under the lock, and log it after the caller releases the lock. Add a test whose logger calls listener.close() when a timer cannot start, bounded by a timeout.
Activity
fix(microvm-worker): log timer failures after lock (#838)