Add Lium backend

[peterschmidt85]

This PR adds experimental Lium backend support for GPU runs and S3-backed volumes. It uses the online provider in dstackai/gpuhunt#259.

• Execution: Starts the stock dstack runner through Lium's root SSH, preserving the image's environment and working directory. Jobs use dstack's existing image-user detection and privilege drop, so non-root images work without changing the runner.
• Private registries: Creates a temporary credential when registry_auth is set. Each job gets a private template; the backend removes the template and credential after the pod is deleted.
• Volumes: Supports one S3-backed volume per pod, without size or region. Writable mounts require root; non-root jobs without volumes work normally. Local storage is unencrypted and deleted with the pod.

Validation: 211 targeted tests passed (36 skipped), Ruff, Pyright, strict docs build, and package build. Live E2E covered private-image authentication, non-root execution, volume persistence across rentals, cancellation, and cleanup. Temporary capacity failures recovered through normal no-capacity retries. Long-running reliability and RDMA remain unverified.

Before merging:

• Correct RAM and disk reporting in gpuhunt: advertised RAM must not exceed the allocation, and separate filesystem quotas must not be added together as one usable disk.
• Merge dstackai/gpuhunt#259 and publish a gpuhunt release.
• Replace the temporary Datura-ai/gpuhunt@lium-provider dependency with the released version, remove allow-direct-references, and recheck against that release.

[surcyf123]

Thanks @peterschmidt85. We tried this branch (eafa49b) with a local server, a lium backend and a nodes: 0..2 fleet. gpuhunt came from lium-provider at a405412, which includes both resource fixes. Everything below ran on real 1-GPU pods.

What worked

• Task on python:3.11-slim. On an RTX 3090 at $0.24/h it was RUNNING about 70 s after submit and exited 0. Inside the pod the limits matched the offer:
  • memory.max = 20 GiB (offer mem=20GB)
  • cpu.max = 10 CPUs (offer cpu=10)
  • / had 37 GB free (offer disk=30GB)
• PyTorch task on pytorch/pytorch:2.4.0-cuda12.1-cudnn9-runtime. It ran an fp16 matmul on the GPU and exited 0. Provisioning to BUSY took about 3m20s, mostly the image pull.
• Dev environment. It reached RUNNING, dstack attach worked, and ssh <run> reached the GPU. The working dir was /root. dstack stop deleted the pod and its template.
• Cancel during provisioning. We stopped a task about 30 s in, while the pod was still being created. Our account log shows the delete requested, the in-flight create then failing, and the pod reaching DELETED. No paid pod was left behind.
• Cleanup. After every run, Lium's API showed no leftover pods or private templates.

Observations

1. Default disk filters out cheap offers. dstack requests disk=100GB.. by default. Now that disk_size is only the container storage quota, that default cut 1-GPU offers from 16 to 10 at the time of the test, and dropped the cheapest one, the 30 GB 3090. The backend mounts the other 2/3 of the share as the /root volume. For dev environments that is also the working dir, so they had more room than advertised (69 GB on /root here). Lower bounds are the right default. We could also look on the Lium side at giving container storage a larger part of the share for templates that don't need /root persistence, if that would help dstack.
2. "Lost run connection". Two attached dstack apply runs ended with "Lost run connection. Timed out waiting for run final status" even though both exited 0. In the torch run the job hit TERMINATING (DONE_BY_RUNNER) at 03:17:53 and the run reached DONE at 03:18:15, about 4 s after the CLI's 30 s wait ran out. This looks like local server pacing rather than the backend. We mention it in case run status should not wait on the instance going BUSY → TERMINATING.
3. Not covered: we did not exercise private images, non-root images or volumes, since your E2E covered those.

We didn't find anything blocking on our side. Happy to try a specific scenario if you'd like.

[apedintensor]

We also tried this branch (eafa49b) with a local server, a lium backend only, a nodes: 0..1 fleet and disk: 20GB... gpuhunt came from the pinned direct reference.

What worked

• dstack offer returned 45 Lium offers. The cheapest was an RTX 3090 at $0.24/h in ru-mow.
• The first run failed fast with NoCapacityError('Lium node is unavailable (node_unavailable)') on both $0.24 3090 offers, and the run ended as no capacity, as expected.
• With retry: {on_events: [no-capacity], duration: 5m}, the retry got a 3090 in ru-mow ($0.24/h). It ran nvidia-smi and exited 0. Inside the pod, memory.max was 20 GiB, matching the offer.
• No pods were left afterwards.

One observation (possibly our fault)
After the run, one private template remained in the account: dstack-main-lium-smoke-0-0-<suffix>, image dstackai/base:0.15-base-ubuntu24.04. It was created at the time of the successful attempt, not the failed ones. We stopped the local server about 20 s after the run reached exited (0), so template cleanup may simply not have run yet. If template deletion is meant to happen synchronously with pod deletion, this may be worth a look. We removed the template manually. The account's other templates and pods were untouched.

[peterschmidt85]

Thanks for testing! Template cleanup runs on a later pass, at least 30 seconds after requesting pod deletion. Could you repeat the test and leave the server running until the instance is fully terminated, then check whether the template disappears?

[surcyf123]

@peterschmidt85, we repeated the test on eafa49b and left the server running until the instance was fully terminated. The template is deleted. There were no leftovers this time.

Setup: a nodes: 0..2 lium fleet, and a task with gpu: 1, disk: 20GB.. and a no-capacity retry. It ran on an RTX 5070 Ti ($0.40/h) and exited 0 on the first attempt.

Timeline from our Lium account event log (UTC):

Time	Event
14:08:08	template created
14:08:10	pod rented (dstack-main-lium-template-cleanup-0-0-…)
14:10:52	pod delete requested
14:11:28	pod deleted
14:12:03	template deleted (71 s after the delete request, 35 s after the pod was gone)
14:12:04	instance TERMINATING -> TERMINATED in the dstack server log

Afterwards, the API listed no pods and no dstack-* templates. The leftover template in the earlier report was therefore from stopping the server about 20 s after the run finished, before the cleanup pass.

[surcyf123]

Code review of src/dstack/_internal/core/backends/lium/ at eafa49b, checked against the Lium API server code. Only real findings are listed.

1. One template per provisioning attempt can hit Lium's template rate limit (most important)

compute.py:148-168 creates a private template on every run_job, and compute.py:334 deletes it when the rent is refused, for example with node_unavailable. The next offer then creates another one.

On the Lium side, POST /templates is limited per account to 1 per 5 s, 5 per 60 s and 20 per hour. Today that counter is also shared with the other rate-limited calls, rent included. That sharing is a bug on our side, and a fix is in review. Even after the fix, a few quick offer fallbacks, parallel replicas or several runs on one account will get a 429. _is_definitive_error (compute.py:476-482) treats 429 as final, so the offer fails, and the next offer runs into the same limit.

Suggested fix: reuse one private template per (image, registry credential), for example keyed by a hash in the template name. The template carries only the image, since the start command goes over SSH, so it is safe to share across instances. Treating 429 as retryable would also help.

2. Template and registry credential can be orphaned if pod deletion outlasts the termination window

Cleanup runs only in terminate_instance once the pod is gone (compute.py:268-280). While the pod is DELETING it raises NotYetTerminated. After TERMINATION_RETRY_MAX_DURATION (15 min, server/background/pipeline_tasks/instances/common.py:32), dstack marks the instance terminated and never comes back.

On Lium, teardown goes through the node's validator with retries, so a slow or offline node can keep a pod in DELETING for longer than 15 minutes. The template is then left behind, and so is the docker credential holding the user's registry password. The template cannot be deleted earlier: Lium returns 400 Template is used by a pod while a pod still references it, so waiting for 404 is right. Two things would limit the leftovers: reusing the template from #1, which leaves at most one shared template per image, and a later sweep of dstack-* templates and credentials that no pod uses.

3. The account-events fallback needs the read API-key scope

_find_pending_pod (compute.py:362) calls GET /users/me/events. That route requires an API key with the read scope. Default keys have it, but a key created with narrower scopes gets 403. Then update_provisioning_data and terminate_instance keep raising on the uncertain-rent path until the 15-minute deadline, ending in the same orphaning as #2. A live pod is still found first from GET /pods, so no paid pod is lost. A line in the backend docs ("the API key needs the read scope"), or treating 403 there as "no event found", would cover it.

Checked and correct

• Endpoints, methods and auth: every path the client calls exists with the same trailing slashes (/docker-credentials/, /templates, /executors/{id}/rent, /pods, /users/me/events with since and limit up to 1000, /version), and all accept X-API-Key.
• Payloads: the rent, template, credential and volume payloads match the server's request models.
• Responses: the fields read from them exist (pod_id, status, ssh_connect_cmd, ports_mapping keyed by the internal port as a string).
• Idempotency: Idempotency-Key and Idempotent-Replayed match the server. A refused rent replays as a refusal, and a different body under the same key gets 422 idempotency_key_reused. We found no double-rent path.
• Error codes: every code in NO_CAPACITY_CODES, plus pod_state_conflict, exists on the server with the status the client expects.
• Pricing: rent charges price_per_gpu × gpu_count, which matches the offer price.
• Units: memory and disk units match. Lium reports GiB.
• Minor, harmless: the server has no DELETED pod status. A deleted pod returns 404, which the client already handles, so the status == "DELETED" checks never match.

[peterschmidt85]

@surcyf123 thanks for catching this.

Template creation limits — merge blocker

The limits (1 per 5s, 5 per minute, 20 per hour) look too restrictive for normal dstack use. Caching helps with repeated images, but runs using new images still need new templates. Could you raise the default limits to support automated workloads?

Pod-start API limits — clarification

Separately, once the shared-counter bug is fixed, how many requests to start pods will one account be allowed to send per minute or hour? This is about API throttling, not the number of GPUs we can run at once. We haven't identified a problem with this limit; we'd just like to understand it.