[ParidelPooya]

Summary

This PR fixes two defects in the MicroVM worker's auto-suspend. A retried SuspendMicrovm could suspend the MicroVM during the next job. A second delivery of a known job got HTTP 503 while the worker refused jobs.

Why

1. The worker's default lambda-microvms client made up to 3 attempts.
2. An attempt can suspend the MicroVM and lose its answer in the freeze.
3. The durable function then resumes the MicroVM. The resume hook ends the worker's refusal, and the next job starts.
4. botocore then retried. The retry suspended the MicroVM with the job inside. Nothing resumed it, so the job failed at its heartbeat timeout or its timeout.
5. A live test showed that SuspendMicrovm succeeds on a MicroVM that is already suspending or suspended. So a retry gains nothing, and one attempt is enough.

The second defect:

1. The durable function sends a job again when it lost the first 202, or when its request step runs again on replay.
2. The worker answered such a delivery with HTTP 503 while it refused jobs for a suspend or a terminate hook.
3. After the 503, the durable function resumed the MicroVM, sent the job again, and got 202. So the MicroVM ran again for nothing.

Both defects come before the planned suspend_after option on vm.invoke, which uses the same suspend path. The JavaScript SDK fixes the same defects in aws/aws-durable-execution-sdk-js#1100.

Changes

• create_microvms_client uses total_max_attempts of 1.
• suspend_microvm takes a resumed callable. The before-send handler raises SuspendResumedError when it returns True. So a client from create_microvms_client that retries starts no attempt after a resume hook. botocore cannot cancel an attempt in flight, so an attempt that started before the hook still runs to its end.
• The worker counts resume hooks, and passes the count at the call's start to the handler. SuspendResumedError counts as an unknown outcome. The resume hook has already ended the refusal, so the worker starts no new one.
• A resume hook during the call logs stopping the suspend call: the MicroVM resumed during it, as the JavaScript worker does.
• With one attempt, a throttle from the default client now counts as refused for now. So the worker accepts jobs again at once, and tries the call again after at least 30 seconds.
• The job handler checks for a known callback ID before the suspend and terminate refusals. A second delivery of a finished job gets HTTP 202 in every state. A second delivery of a running job gets HTTP 202, except during a suspend refusal. There it gets HTTP 503, because the running job may freeze, and the 503 makes the durable function check the state and resume the MicroVM. A new job still gets HTTP 503 during the refusal. The run hook already checked for a known job first.
• README "Auto-suspend" and "Suspend and terminate hooks", and the docstrings, are updated.

Testing

• New tests: the default client makes one attempt, for an error answer and after a connection error; a client that retries a throttle gets an unknown outcome; the handler starts no attempt after a resume hook, also before the first attempt, and starts the attempt while no hook arrived; the worker sends no attempt after the resume hook; a second delivery of a running or finished job gets 202 during a suspend refusal and after the terminate hook; a resume hook while the client is created stops the first attempt; a suspend hook after a stopped call keeps its refusal; a second resume hook logs the stop line once; a resend of a running job gets 503 during a suspend refusal, and a composed test with the extras request step resumes the MicroVM in that case.
• hatch run dev-microvm-worker:cov: 473 tests pass, with 100% coverage of worker.py and self_suspend.py. typecheck, hatch fmt --check, and lintcommit.py pass.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[yaythomas]

[github-actions]

Codex AI review

No actionable findings in the PR changes. Tests were inspected but not executed under the review constraints; live MicroVM freeze/resume timing remains unverified by this review.

Reviewed commit 1488985f26e0fc2809ebc025807bec20870fe43b. Workflow run