Skip to content

Timeouts that stop running steps, on_timeout: :fail fix, versioning phase 1 - #19

Merged
kasvith merged 6 commits into
mainfrom
feat/v1.2-correctness
Oct 24, 2026
Merged

kasvith merged 6 commits into
mainfrom
feat/v1.2-correctness

Conversation

@kasvith

@kasvith kasvith commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

Roadmap items 1, 2 (phase 1 and design) and 5. Each commit is one changelog entry and passes the suite on its own.

Changes

Timeouts

  • A step's timeout: is now enforced. It was documented but never applied. A slower attempt is stopped and fails with a "timeout" error, which is retried like any other error.
  • The workflow timeout: stops a step that's still running. Before, it was only checked between steps. The run fails without a retry and runs its compensations. Parallel branches inherit the run's deadline.
  • Steps with a time limit run in a task. That's a step with its own timeout:, or any step in a workflow with one. The task gets a copy of the worker's process dictionary and returns its own, so context, put_context and log capture work as before. Only self() differs. Steps without a limit still run in the worker.
  • wait_for_input(..., on_timeout: :fail) now ends the run failed, not cancelled. The timeout worker used to write cancelled straight to the row. That skipped compensations, never notified a parent run, and could overwrite a run that had already moved on. Now it resumes the run with the error, and the executor fails it like any other failure.

Step options

  • Time helpers in step options now work. retry: [base: seconds(1)] used to reach the backoff as quoted code. Step options are now evaluated the way workflow options already were.

Versioning, phase 1

  • A run no longer completes silently after a deploy removes its step. Before, a run resumed at a step that no longer existed was marked completed and every remaining step was skipped. It now fails with definition_changed and doesn't compensate, so restoring the step and calling Durable.retry/2 finishes it.
  • Design doc for phase 2: agents/milestones/M08-workflow-versioning.md covers version pinning. It has three open questions:
    • API shape: several version: blocks in one module, or one module per version.
    • Whether to ship the required migration in 1.2 or wait for 2.0.
    • What happens to a run whose version was deleted.

Formatter

Behaviour changes

These need a note in the 1.2.0 release:

  • on_timeout: :fail runs end failed (or compensated), not cancelled.
  • A workflow timeout: now interrupts a running step.
  • A step timeout: is now enforced.
  • Steps with a time limit run in a separate process, so code that relies on self() being the worker behaves differently.

Testing

  • Core: 492 tests pass. Credo strict and compile --warnings-as-errors are clean.
  • Dashboard: 237 tests pass against the local durable.
  • New tests:
    • The deadline stops a running step and the run compensates.
    • A step timeout stops the attempt and it's retried, with put_context and logs kept.
    • on_timeout: :fail fails the run, and a retry waits for the input again.
    • Step options are evaluated.
    • A renamed current step fails the run, and a retry recovers it.

Step options were stored as quoted code: `retry: [base: seconds(1)]` reached
the backoff as a tuple, and `timeout: seconds(5)` as the call's AST. They're
now evaluated in the workflow module, the way workflow options already are.
… :fail

The timeout worker wrote :cancelled straight to the run, so compensations
never ran, a parent run was never told, and a run that had already moved on
could be overwritten. It now resumes the run with the error, and the executor
fails it at the waiting step like any other failure: the run ends :failed (or
:compensated), not :cancelled. A retried run waits for the input again.
A step's `timeout:` was documented but never enforced, and a workflow
`timeout:` could only fail a run between steps. A step with a time limit now
runs in a task that is killed when the limit passes. Past its own timeout the
attempt fails and is retried like any error; past the run's deadline the run
fails without a retry and runs its compensations. Parallel branches get the
run's deadline too.

The task starts with a copy of the worker's process dictionary and hands its
own back, so context, put_context and log capture behave as before; only
self() differs. Steps without a time limit still run in the worker.
A run resumed at a step the deployed workflow no longer has was marked
completed, with every remaining step skipped. It now fails with a
`definition_changed` error, without compensating, so restoring the step and
calling Durable.retry/2 finishes it. This is phase 1 of workflow versioning.
Follow-up to #10. durable's formatter now applies the locals_without_parens it
exports, and the dashboard and the Phoenix demo import them.
@kasvith
kasvith merged commit 146d350 into main Oct 24, 2026
5 checks passed
@kasvith
kasvith deleted the feat/v1.2-correctness branch October 24, 2026 01:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant