Skip to content

Proposal: handle SIGTERM by default to initiate graceful worker shutdown #1948

Open

Description

@seanbollin

Summary

A Temporal Python worker installs no OS signal handler by default, so when the process receives SIGTERM (Kubernetes pod stop, Cloud Run scale-down/redeploy, docker stop, systemd), the OS default disposition applies and the process is terminated immediately — before the worker can stop polling or drain in-flight work. In-flight activities are abandoned and retried server-side after their timeouts. (Python installs a handler for SIGINT but deliberately leaves SIGTERM at the OS default.)

Proposal: consider initiating the worker's existing graceful-shutdown path on SIGTERM by default (opt-out-able), so the common containerized case — stop polling → drain for graceful_shutdown_timeout → cancel — works without extra wiring.

Current workaround

loop.add_signal_handler(signal.SIGTERM, ...) to trigger await worker.shutdown() (or to exit an async with worker: block).

Precedent

The TypeScript SDK already does this: its Runtime installs handlers for SIGINT/SIGTERM/SIGQUIT/SIGUSR2 by default and calls shutdown() on every worker.

Trade-offs / open questions (why it is not already the default)

  • A worker usually runs inside a larger process (HTTP server, DI host, other lifecycle). A library installing a process-global signal handler can conflict with the host application's own signal handling, so any default must be clearly opt-out-able and should no-op when a host framework already owns the signal.
  • Default-on vs. an explicit opt-in flag?
  • If added, match the TypeScript semantics and honor the existing grace/drain config.

Why this comes up

SIGTERM-before-SIGKILL is the standard stop protocol for Kubernetes, Cloud Run, ECS, docker stop, and systemd. Users commonly expect a worker to drain on SIGTERM without extra wiring, and it is a recurring support question (e.g. https://community.temporal.io/t/proper-way-to-shutdown-on-sigterm-sigint/5485).

One of a set of sibling issues on default SIGTERM handling across the SDKs that do not auto-handle it today (Go, Python, Java, .NET, Ruby, Rust). TypeScript already does; PHP delegates process lifecycle to RoadRunner.

Activity

  1. self-assigned this
    on Oct 8, 2026
  2. AlexeyShalaev commented on Oct 27, 2026

    @AlexeyShalaev

    On the "should no-op when a host framework already owns the signal" question, a concrete case: a worker started from a FastAPI lifespan under uvicorn, which is a common setup.

    uvicorn installs its SIGTERM handler before it runs the lifespan startup, so signal.getsignal(signal.SIGTERM) there returns Server.handle_exit. If the SDK then calls loop.add_signal_handler(SIGTERM, ...), it replaces uvicorn's handler: the worker shuts down, but uvicorn never sees the signal and keeps serving until SIGKILL. Minimal check with uvicorn 0.54, a FastAPI app whose lifespan installs a handler that way:

    SIGTERM handler at lifespan startup: Server.handle_exit
    ---- SIGTERM sent
    worker: SIGTERM -> worker.shutdown()
    ---- still running 10 s after SIGTERM, HTTP still answers 200
    

    So if this becomes default-on, installing only when signal.getsignal(signal.SIGTERM) is signal.SIG_DFL covers that case, and an explicit opt-in (Worker(..., handle_signals=True) or a run_until_signal() helper) would avoid the question entirely.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions