Skip to content

[Bug]: dstack attach --logs reports failure after a successful task #4381

Open

Description

@peterschmidt85

Steps to reproduce

  1. Submit a short task on an existing fleet:

    dstack apply -f task.dstack.yml -y -d
  2. Attach while it is running and let it finish:

    dstack attach <run-name> --logs
    echo $?
  3. If the CLI reports a timeout, check the run again:

    dstack run get <run-name> --json

This is timing-dependent. In the observed run, the task performed GPU/environment checks, slept for 30 seconds, and exited successfully.

Actual behaviour

After the logs end, the CLI exits 1 and prints:

Lost run connection. Timed out waiting for run final status. Check `dstack ps` to see if it's done or failed.

The run subsequently reaches done, with job exit code 0. In one observed run, the server took 46.2 seconds after the runner finished to mark the run done. No SSH disconnect was observed while that task was running.

Expected behaviour

The CLI reports the successful result and exits 0. Waiting for the server to finish updating the run should not be reported as a lost connection.

dstack version

Development branch based on master fe6fe22e4, with unchanged shared pipeline and CLI code; runner build 8254.

Notes

The CLI waits for only 30 status refreshes after logs end. One confirmed contributor is that run inspection resets an unchanged job's processing timestamp, postponing the next runner poll. This defect reproduces without provider API calls.

That scheduling defect was introduced by #3686 in 0.20.14 (opt-in pipelines), enabled by default in 0.20.16 through #3728. This was verified against the preceding code and release tags. The CLI timeout itself is older.

Preserving the timestamp fixes that polling defect, but did not eliminate the CLI timeout in live testing. The remaining finalization delay and the CLI's wait need to be addressed together.

Activity

  1. changed the title [-][Bug]: CLI can time out while successful runs are finalizing[/-] [+][Bug]: Attached CLI exits 1 after a successful task while the server is still finalizing[/+] on Oct 17, 2026
  2. changed the title [-][Bug]: Attached CLI exits 1 after a successful task while the server is still finalizing[/-] [+][Bug]: dstack attach --logs reports failure after a successful task[/+] on Oct 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions