Repository navigation
Sync upstream snapshot storage and sandbox lifecycle updates - #91
Merged
Merged
Conversation
atelet can upload a paused actor's snapshot and detach the actor before ateapi records the upload, since it reports the upload best-effort. ateapi then restored the actor without snapshot.object, atelet failed with NotFound, and the revert that followed deleted the uploaded snapshot as PENDING. ateapi now names the upload location on every local Restore, whether or not it has recorded the upload, and atelet requires it. atelet reads it only when it holds no usable local copy; the manifest, uploaded last, tells a complete upload from a partial one. Part of agent-substrate#2069 and agent-substrate#2571 #### Breaking change atelet requires snapshot.object on Restore - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…-substrate#2817) CreateTag on a paused actor whose snapshot is still only on its node now uploads that snapshot first, then tags it, instead of failing with FAILED_PRECONDITION. The call can take as long as the upload. - CreateTag asks the atelet on the snapshot's node to upload it with PromoteSnapshot. atelet keeps its local copy, so the actor still resumes on that node. - CreateTag checks that atelet reports the requested snapshot as durable, then records the upload on the actor. - A failed upload fails the tag with no pending tag left behind, and leaves the actor as it was. - The actor lease is held for the whole tag process, so the actor cannot be resumed or suspended during the upload. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…g its default runtime (agent-substrate#2756) Step 5 of agent-substrate#2675: the sandbox asset cache gets an eviction pass with a budget, and each worker's default runtime is kept in the cache by links under the worker's own directory. **Commit 1, filecache.** A pass never evicts an entry a consumer still hard-links. A link is how a consumer declares an entry in use; retiring such an entry frees nothing while the link lasts and makes the next caller download a second copy beside it. Linked entries are skipped and counted as `SkippedLinked`, and every retired entry frees its full size, so the stats need no `PendingBytes`. The listing runs lock-free, so the retire step re-checks the links under the hit lock as it re-checks the last-use clock, and a link gained since the listing vetoes the entry even when the hit's best-effort touch did not land. The contract is that eviction never takes what a caller holds and its only cost is a re-download. **Commit 2, atelet.** *Design.* The cache has one protection rule, the link rule above, and both things that must survive a pass use it the same way. An actor protects its runtime through the links in its own directory for as long as it lives on the node. A worker protects its default runtime through links in its own directory for as long as the pod exists. There is no root set the pass consults, no reference count across pods, no prune, and nothing to reconcile after a restart: the directory of a pod still on the node keeps protecting, and the startup sweep that already removes the directories of pods that are gone releases their links with them. Several pods of one pool each hold their own links to the same entries; the filesystem counts them, and the entry becomes evictable when the last pod's directory goes. *Directory layout.* Under each worker pod's directory, `<base path>/workers/<pod UID>/`, which the syncer already owns for the pod's sandbox runtimes file: ``` workers/<pod UID>/ sandbox-runtimes.json default-runtime/ <hash>/ one per default runtime the file advertises, by bundle hash gvisor/runsc … hard links into the cache, made by the staging call Run uses ``` *Lifecycle.* The syncer owns the directories and the prewarmer only fills them. Once a pod's file is written and registered, the syncer makes `default-runtime/` match what the file names as defaults: the directory of a default the file no longer names is removed, which unlinks that runtime, and an empty directory is created for each default that lacks one. Then it hands the prewarmer one item per default, the hash and its directory. The hand-over comes after the build on purpose: a build that fails, as a permanent runsc probe failure does, leaves the file and so the directories as they were, with the previous default still linked and still advertised. The prewarm starts late by only a little: for a gVisor default the build's probe has already fetched the release, so the prewarm is links and the pause image pull, and for a micro-VM default the build takes milliseconds. The prewarm job checks that its directory exists, stages into it with a variant of the staging call that creates no directories, pulls the pause image beside it, and on an error checks again: a directory gone, after a later sync or the pod's removal, means the item was superseded, and it ends without a retry and without recreating anything. A directory the syncer already filled costs a stat per asset, so a re-sync with an unchanged default downloads nothing and leaves the links alone. A failed download retries with the existing backoff and gives up after eight attempts; the directory stays, empty or partial, until the next sync, and a Run placed on that default meanwhile fetches on demand and shares the download. One queue and one worker drain the items in order, so defaults download one at a time across the node and never compete for its bandwidth. *Why the old default is not kept during the download.* By the time the directories change, the file names the new default and the worker has registered it, so the scheduler places nothing new on the old one. Actors already running on it hold their own links, and restores of snapshots taken on it go through the on-demand path like every non-default version. Keeping its links would only hold cache space. *GC loop.* `sandbox_asset_gc.go` adds `--sandbox-asset-cache-gc-period` (5m, 0 disables), `--sandbox-asset-cache-max-bytes` (0 means unbounded: no pass runs, warned at startup) and `--sandbox-asset-cache-gc-dry-run`. A pass that finds the cache over budget asks the store to free down to 90% of it, oldest first. Same shape as the image GC: immediate first pass, serialized passes, panic recovery, shortfall warnings that back off to a periodic reminder. *Metrics.* `ate.sandboxassetcache.size` (observable up/down counter, bytes), `ate.sandboxassetcache.evictions` and `ate.sandboxassetcache.gc.passes` by `ate.sandboxassetcache.gc.outcome` (`complete`, `shortfall`, `skipped`); a pass within budget records nothing, and a dry run records its pass but no evictions. Registry group and entries added, `hack/verify/metrics.sh` passes, rows and a paragraph in `docs/observability.md`. **What a shortfall means.** Everything left is linked by an actor, linked by a worker as its default, younger than the min age, or busy. The size then measures what the node's actors and defaults need, not waste; the budget is too small or the node hosts too many runtimes. **Not in this PR.** A shared GC driver for the image and asset caches, which would also give the image GC the metrics it lacks; filed as a follow-up once this lands. Step 6, the summary doc, which agent-substrate#2792 carries and which will be rewritten for this layout. **Tests.** filecache: a linked file, a partly linked tree and a fully linked tree all survive a pass; unlinked trees still go oldest first; an entry linked between the listing and the retire, without a touch, is vetoed at retire time. atelet: a prewarm links the runtime into its directory and an eviction pass keeps the entry; a second prewarm of the same directory fetches nothing; removing the pod directory makes the entry evictable; a job whose directory is gone before it runs, or whose pod goes while it stages, fetches nothing, recreates nothing, and is not retried; the sync hands the default over only after the build, removes a retired default's directory, keeps the current one and its contents across a re-sync, hands nothing over and touches no directory when the build fails, and the pod's removal takes the directories; the end-to-end informer test asserts the link under the worker's directory; retry, pause-image and attempt-timeout cases as before. GC: target table, flag validation, quiet, skipped, complete and shortfall passes with the metrics checked through a manual reader, the size callback, immediate first pass, ticking, panic recovery. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
…trate#2819) Late Checkpoint and PromoteSnapshot requests can conflict with a newer snapshot on atelet. Return `Aborted` for stale generations and same-generation UUID conflicts, and have ateapi preserve the actor and its worker assignment while reporting the conflict to the caller. Other `FailedPrecondition` errors retain their existing crash policy. Checkpoint conflicts are handled at the suspend call site. CreateTag maps PromoteSnapshot conflicts in its own upload error handler. Both ateapi and atelet must include this change to avoid crashing actors on snapshot generation conflicts; mixed versions retain the previous crash behavior. Regression tests cover the atelet error-conversion boundary, in-flight and background-operation conflicts, caller cancellation, and synchronous/asynchronous SuspendActor recovery. The API test keeps the last completed snapshot and its objects through repeated conflicts, then retries the same pending snapshot after the conflict clears. The API guide describes the retry behavior and its limits. Fixes agent-substrate#2808. Validated `13e53825`, including main `ddb46651`, on Linux/amd64 with Go 1.27.2 as a non-root user with `CI=true`. - `go test -race -p=4 -count=1 ./...` passed: 167 packages, 10,695 passing test and subtest results, 43 skips, and no failures. - Builds of `ateapi` and `atelet` passed. - `hack/verify-all.sh` passed all 16 checks. Regeneration and verification left the source tree clean. - Focused PostgreSQL/envtest regressions passed. Removing the conflict mappings made the synchronous and asynchronous Checkpoint cases and the tag-upload conflict case fail as expected. Root-gated tests and real runtime cluster E2E were not run. AI assistance: Codex helped implement and test this change.
…r it (agent-substrate#2854) In `demos/counter/README.md`, each command block folded under a `<details><summary>` opened on the line right after the summary tag. A summary tag opens an HTML block, and Markdown inside an HTML block renders only after a blank line, so GitHub showed those twelve blocks as literal backticks instead of code. This inserts the blank line after each of them. No other Markdown file in the repo has a fence glued to an HTML line. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
…oint, and takes its own exporter settings (agent-substrate#2189) Follow-up to agent-substrate#1878 (agent-substrate#1877). ateworker exports through atelet's relay, or directly to its own OTLP endpoint when there is no relay. With neither, it used to export to `localhost:4317`, where no collector runs. It now skips the trace, metric and log exporters and logs a warning per signal with `reason="no OTLP relay or endpoint"`. ate-controller now always sets the worker pod identity (`OTEL_RESOURCE_ATTRIBUTES`), metric export tuning and sampler, so telemetry sent through the relay carries pod identity even without an endpoint. The worker exporters come from new worker-only `ATE_WORKER_OTEL_{LOGS,TRACES,METRICS}_EXPORTER` keys in `ate-otel-config` (flags `--otel-{logs,traces,metrics}-exporter`). The controller's own values don't fit the workers: `prometheus` means scraped, and ateworker serves no `/metrics`. #### Breaking change Please add the `breaking-change` label. - Worker pods no longer take `OTEL_LOGS_EXPORTER` from ate-controller. If you set it to send worker usage records over OTLP, also set `ATE_WORKER_OTEL_LOGS_EXPORTER` to the same value in `ate-otel-config`. The kind overlay already does. - On installs where ate-controller has no OTLP endpoint, WorkerPool Deployments roll once on upgrade because the worker env changes. Installs that set the endpoint in `ate-otel-config` (the default) don't roll. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…ate#2727) #### Breaking change `DeleteActor` now deletes an actor in any state, and the `any_state` field and the `kubectl ate delete actor --any-state` flag are gone. - Callers that relied on `DeleteActor` returning `FAILED_PRECONDITION` for a running actor no longer get that guard. A running actor is now terminated, its volumes detached, and its worker released. To avoid deleting an actor that changed since you read it, check its state yourself before deleting, or pass `options.uid` and `options.version`. - `DeleteActorRequest.options` moved from field 3 to field 2. Regenerate clients from the new `ateapi.proto`. Old stubs that send `options` will have it silently dropped, and old stubs that set `any_state` will fail to parse. - Remove `any_state` / `AnyState` from `DeleteActorRequest` in your code. - Drop `--any-state` from any scripts that call `kubectl ate delete actor`. The flag is now rejected as unknown. Fixes agent-substrate#1161 > It's a good idea to open an issue first for discussion. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
The store's pgx pools carried no tracer, so no span was ever emitted for database work: a traced GetActor arrived as a single gRPC server span with the PostgreSQL time indistinguishable from handler overhead, and even the multi-span suspend/resume traces showed step, atelet and snapshot spans but nothing for the many store round-trips between them. The store blind spot in docs/metrics/substrate.yaml describes the metrics half of this; the trace half is what this change closes. Attach a QueryTracer to the pool config (the watch pool inherits it via Copy). Each statement becomes a client span named after its leading keyword (db.SELECT, db.UPDATE, ...) with the parameterized statement text as db.query.text — arguments never appear in it. Spans only join an existing trace: statements from background work (outbox polling, lease maintenance) carry no surrounding span, and opening a root span for each would flood the backend with single-span traces. pgx.ErrNoRows leaves the span unmarked, since the store maps it to NotFound as an expected lookup outcome. Fixes agent-substrate#1455 > It's a good idea to open an issue first for discussion. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
atelet's `DetachActor` returned UNIMPLEMENTED. Without it, ate-api-server can't make a suspended actor's latest snapshot durable and free it from the node on demand. Resuming the actor on another node needs that. The handler validates the request and runs the snapshot tracker's `Detach`. That's the same detach atelet runs on its own after `--snapshot-detach-delay`. The handler returns the actor's new snapshot state. #### API changes - `Atelet.DetachActor` (atelet internal) is now implemented. The proto comment drops its "returns UNIMPLEMENTED" TODO. No fields change. #### Not in this PR - Detach deletes the actor's local snapshots but not the rest of the actor's directory. The detach atelet runs on its own does the same. Removing the whole directory overlaps agent-substrate#2742 and agent-substrate#2772, and belongs in the tracker so both paths get it. - Nothing in ate-api-server calls `DetachActor` yet. Resuming elsewhere will be a follow-up.
…te#2761) **What** A benchmark that runs atunnel in its own process and drives load through it at rising rates, reporting atunnel's CPU, memory and latency per rate. Includes a first full run: `runs/20261016-123913/results.txt`, and `report.md`, which explains the results and two findings (one filed as agent-substrate#2759). **How** - atunnel is started and hosts actors with the worker's own code: `tunnel.Start`, then per actor `PrepareEgress`, `actornet.ServeSandbox` and `Activate`. Only atelet, the actors' programs, the router and the egress gateway are faked. - Everything is one program, `atunnel-bench`, with a subcommand per process: `tunnel` (atunnel), `worker`, `sink`, `load`, `pki`. - `run-docker.sh` runs three containers: - **worker** (privileged): atunnel in its own cgroup with CPU and memory limits, plus the fake atelet, an ingress actor in every actor's sandbox, and an egress actor in the first; - **sink**: the fake egress gateway; - **load**: the load generator, which sends ingress and reads atunnel's stats. - Seven scenarios cover ingress (HTTPS and CONNECT) and egress (reused, new, and idle connections, and throughput). **For reviewers** - `.golangci.yaml`: adds a depguard exception so the benchmark can import `internal/ateworker`. - `internal/workerhost` has a root-gated test that runs traffic through a real sandbox. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…nt-substrate#2880) Now that gVisor supports `SNAPSHOT_FIDELITY_ROOTFS` alongside micro-VM (agent-substrate#2673), a snapshot's rootfs upper layer depends only on the container image (`ActorTemplate`) and not on the worker's sandbox runtime version or host CPU features. - **`requiredSnapshotRuntime`**: Only impose a `RequiredRuntime` placement constraint when restoring `SNAPSHOT_FIDELITY_MEMORY` snapshots; `SNAPSHOT_FIDELITY_ROOTFS` and `SNAPSHOT_FIDELITY_VOLUMES` snapshots can schedule onto any eligible worker of the same sandbox class without runtime matching. - **`ensureAteletRestored`**: When a durable `MEMORY` snapshot falls back onto a runtime-incompatible worker, downgrade the restore fidelity to `SNAPSHOT_FIDELITY_ROOTFS` (preserving both rootfs writes and durable volumes) instead of `SNAPSHOT_FIDELITY_VOLUMES`. Snapshots whose `ActorTemplate` was replaced continue to downgrade to `SNAPSHOT_FIDELITY_VOLUMES`. - **Tests & Docs**: Update `workflow_resume_test.go`, `functionaltest/actor_test.go`, and `docs/api-guide.md` to reflect `ROOTFS` fallback semantics. Follow-up to agent-substrate#2673 and agent-substrate#2660. <img width="1364" height="1688" alt="image" src="https://github.com/user-attachments/assets/7a105b64-2c30-440f-8679-5afbeb591f7d" /> - [ ] Tests pass - [ ] Appropriate changes to documentation are included in the PR
…bstrate#2801) MicroVM actors with a one character container name never started: We used the name as the kata container ID, and kata rejects IDs shorter than two characters. gVisor and Kubernetes allow these names. https://github.com/kata-containers/kata-containers/blob/4.1.0/src/libs/kata-sys-util/src/validate.rs#L17-L22 The kata ID is now `c_<name>`. It only depends on the name, since a restored guest keeps the IDs it booted with. This won't hit any limits, and it can't conflict with container names. It also fixes a collision risk for container named `durable` + durable dir volumes (doesn't exist for gvisor). I had an e2e test, but it seems unlikely we'll regress this and not worth the compute on every PR. AI-assisted. Tested on kind (micro-VM and gVisor) and I reviewed the changes. Breaking change: Like other pre-GA changes, you'll lose old snapshots. This should be small enough to land pre-GA. Fixes agent-substrate#2800 > It's a good idea to open an issue first for discussion. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
Fixes agent-substrate#2407, agent-substrate#2436. > It's a good idea to open an issue first for discussion. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR ## What changes One rename in the volume API, split out of agent-substrate#2567 so the breaking part lands before GA and the shared-volume feature can follow as a non-breaking addition. **`Volume.external_volume_template` becomes `Volume.external`**, typed `ExternalVolumeSource`, with the existing `ExternalVolumeTemplate` under its `template` member. The field name now matches its siblings (`durable_dir`, `system_info`, `image`) and the message follows the `*VolumeSource` convention (agent-substrate#2407, agent-substrate#2436). The `ExternalVolumes` preview gate moves onto the new member, so gating behaviour is unchanged. `ExternalVolumeSource` is a union with one member today; the shared-volume reference is added as a second member in agent-substrate#2567. ```yaml volumes: - name: data external: template: capacity: 10Gi storageClassName: standard ``` The per-actor record in ActorStatus gets wrapped in a new ExternalVolumes message. This will make changes for shared volume support easier post GA. Callers, tests, demos, benchmark manifests and docs are updated; generated code is regenerated. ## Testing Unit, store (Postgres), controlapi and functional suites pass; `hack/verify/gofmt.sh`, `hack/verify/boilerplate.sh`, `apitool validate` and its exemption test pass. #### Breaking change `ActorTemplate.volumes[].externalVolumeTemplate` is gone; templates must use `volumes[].external.template`. Field number 3 is retired, not reused, so templates stored under the old field decode with no volume source and must be recreated (they fail validation on their next update). Go consumers of `pkg/proto/ateapipb` see the new `ExternalVolumeSource` type and the moved getter. Requesting the `breaking-change` label. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
…ials on GCS (agent-substrate#2762) Part of agent-substrate#777 The control plugin sidecar in the ate-api-server pod now serves a credential exchange: the node plugin will present the worker-for-actor JWT atelet gives it with a transfer and receive a GCS access token scoped to that actor's snapshot prefixes, read-write on its own and read-only on the tag it was created from. Nothing calls it yet, the node plugin and atelet side come in the next PR, so transfers keep running on the pod's own credentials until then. Four commits: - `actorlookup` fetches the actor a verified JWT names over the loopback, `127.0.0.1:443`, with the pod identity certificate, and refuses one whose UID changed. - `exchange` and its private proto, `internal/proto/objectstoreexchangepb`: verify the JWT with `FileKeySource` and `WithSubstrateClaims`, accept only the worker-for-actor kind so an actor cannot trade its own token for its snapshots, derive the grants with `actorscope`, mint through `gcsissuer`. Errors follow the table in the proto comment. - Wiring in control mode: with `--exchange-listen` set, a second gRPC server over TLS with the pod's service DNS certificate. On S3 the exchange answers `UNIMPLEMENTED`. The README gains the package map and which pieces are shared between backends. - Manifests: the sidecar's port, the issuer read from the same ConfigMap key ate-api-server uses, three read-only mounts including only the `jwks` item of the pool Secret, the Service port, and a network policy rule admitting only atelet pods. On an install that enforces authorization the lookup needs agent-substrate#2757, the `api_server_sidecar` role. Without it the exchange answers `UNAVAILABLE`, which nothing observes until the node plugin calls it. Verified on a GKE dev cluster from inside the ate-api-server pod with a JWT signed by the cluster's pool: exchange in about 100ms, cache hit on the second call, forced mint, the four refusals with their codes, and the token reading and writing exactly within the grants, with 403 on a sibling prefix, another actor's prefix, and a list. A repoint followed by a suspend moved the grant to the new location and the cache minted on its own since the old token no longer covered it. The end-of-hour window of the source credential was observed as documented in agent-substrate#2722. Key rotation keeps `jwks` in step with `pool` through the current `kubectl-ate` or an ate-setup rerun after rotating with an older one. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…t in the metrics suite (agent-substrate#2792) > agent-substrate#2756 has merged; this is one commit on `main`. The page describes agent-substrate#2756's final design: the default runtime is linked under the worker's own directory, in a directory per default the worker's file names, made and removed by the syncer and gone with the pod, so the on-disk table, syncer section, eviction rule, and runbook follow that layout and a subsection gives its lifecycle; and with no budget set atelet starts no eviction pass. Step 6 of agent-substrate#2675, in its small form: the e2e under eviction pressure was dropped as too much machinery for what it proves, and this is what remained worth doing. **Metrics suite.** `ate_sandboxassetcache_size` joins `PlatformMetricPrefixes`, so `TestPlatformMetricsEmitted` proves on every kind lane that atelet registers the instrument, its callback walks a real cache directory without error, and the OTLP path to the collector carries it. Only size qualifies: it is an observable instrument read on every collection, so it has a series from the first export on any install. The eviction and pass counters emit only once a pass had work to do, which never happens with the default unbounded budget. **Doc.** `docs/sandbox-asset-cache.md`, for the operator or engineer who meets the cache for the first time: what lives under `/var/lib/ate` and why it must share a mount; how an actor gets its assets, from the runtime the request names through the index, the cache and the hard links, and what Run, Restore and Checkpoint each do; how the worker sandbox syncer decides which runtimes a node needs, prewarms and pins the defaults, and prunes; the eviction rules and the six flags; the three metrics and what a shortfall means; a runbook for a node over budget; and what is deliberately left out. Linked from the README's doc list, the node supervisor section of the architecture page, and the metrics paragraph of the observability guide. The page describes the request as naming the runtime the worker registered, which is agent-substrate#2788's behavior rather than this stack's. It documents the state once both land rather than the interim. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
…ubstrate#2719) ## Problem A node's suspend/resume turnover plateaus with most of the wall time spent waiting on GCS, even though the node's network has headroom. `pkg/objectstorage` already keeps a pool of eight `storage.Clients` because one client holds a single HTTP/2 connection and multiplexes every request onto it. But the pool only served the parts of a composite upload (objects above 64 MiB) and the ranges of a parallel download (objects above 16 MiB). Everything smaller went through `putSingle` and the first range of `GetObject`, both hard-wired to the wrapped client. Compressed snapshots are usually below those thresholds (about 10 MiB for gVisor, about 50 MiB for micro-VM). The object-store plugin is one process per node with one shared client, so when a node suspends and resumes many actors at once, every one of their snapshot transfers queues on the same TCP connection. ## Change - Add `nextClient`, which hands out pooled clients round-robin for a transfer that is one request and so has no part or range index of its own. - `putSingle` and the first range of `GetObject` use it. Parts and ranges keep their index-based pooled client. - Build the pool in `NewGCSClient` alongside the wrapped client instead of lazily on first use, so a burst of transfers on a fresh node does not wait behind eight sequential client constructions. The `sync.Once` and the stored options go away with it. - Reword the fallback warning, which still spoke of part uploads, and have it report how many pooled clients were built out of how many wanted. - Comments updated to say why small objects go through the pool. No change to how objects are split, named or stored. Small objects still go up and come down as one request, which the package's existing measurements show is fastest for them. The change is which connection that request uses. ## Testing - New `TestSingleRequestTransfersSpreadOverThePool`: six alternating small gets and puts land two on each of three pooled clients and none on the wrapped client. - Tests that build a `gcsClient` by hand no longer need to pin a lazy pool. With no pool set they fall back to their test client, which is what they always wanted. - `go test ./pkg/objectstorage/ ./internal/objectstoreplugin/... ./cmd/objectstore-plugin/... ./cmd/atelet/` pass, with and without `-race`. `vet`, `gofmt`, boilerplate and `golangci-lint` are clean.
…torageLocation override in preview mode (agent-substrate#2544) > The PR title becomes the release note. Write it for users: what changed for them, not how the code changed. #### Breaking change ate-api-server now stores snapshots under a new, operator-set `--snapshot-location` (for example `gs://BUCKET/` or `gs://BUCKET/PREFIX/`). `ActorTemplate.snapshotConfig.storageLocation` is now a preview field, honored only with `--preview=PerTemplateSnapshotLocation`. `snapshotConfig` itself is now optional; an omitted one takes every default. What you must do when upgrading: - **ate-setup installs:** set `SNAPSHOT_LOCATION` (`--snapshot-location` / `storage.snapshotLocation`). ate-setup passes it to ate-api-server and, unless the `PerTemplateSnapshotLocation` preview gate is enabled, fails before touching the cluster if it is unset. `BUCKET_NAME` alone is not enough. The kind profile defaults it to `gs://$BUCKET_NAME/` (`gs://ate-snapshots/`). ate-setup does not create the bucket or grant access to it on GKE (`go run ./tools/setup-gcp bootstrap` does). - **Custom deployments of ate-api-server:** add `--snapshot-location` (or `ATE_API_SNAPSHOT_LOCATION` with `--snapshot-location=@env`). Without it, the server refuses to start unless the `PerTemplateSnapshotLocation` preview gate is enabled. - **Templates that set `storageLocation`:** without the gate, that value is ignored and new snapshots and tags go under `--snapshot-location`. If you rely on per-template buckets or paths (IAM, retention, billing), enable `--preview=PerTemplateSnapshotLocation` to keep the old behavior. With the gate, the field stays optional: a template that leaves it empty uses `--snapshot-location`, and `CreateActorTemplate` fails with `FailedPrecondition` if neither is set. - **Clients that set `storageLocation` with the gate enabled:** they must opt in to preview features (`kubectl ate --preview`, or the `Ate-Preview-Allowed: true` header). Otherwise `CreateActorTemplate` fails with `InvalidArgument`. - **Existing actors:** snapshots already taken keep working for resume. If an actor's snapshot location changes (by upgrading with the gate off, changing the flag, or toggling the gate), its next suspend writes under the new location and releases the previous snapshot from the old one. Until that suspend, changing the actor's `actor_template` is rejected, and deleting the actor collects its snapshots under the old location. If releasing the previous snapshot fails, its objects stay under the old location and a later delete does not collect them; ate-api-server logs a warning naming the snapshot URI. To keep the old locations, enable the gate before upgrading. ActorTemplate is not the right place to specify snapshot location. Bucket is an implementation detail of snapshots, workloads shouldn’t need to care about it. It is the responsibility of substrate admins to provision the buckets and give access to Substrate. This PR adds a cluster wide storage location as a building step towards the final goal of adding these details in a separate SnapshotStorageConfig. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…te#2494) The RPC log line carries the whole response; a List page of a thousand actors is tens of kilobytes and rotates the lifecycle lines out of the container log. - `List*` responses are logged as `<ListActors response elided, N bytes>`; the response to the client is untouched. - A failed `List` call logs the handler's `null`, not an elided placeholder. - Tests cover both. `go test -race ./internal/ateinterceptors/`, lint clean.
The AteomHerder listener required a client certificate chaining to the pod-identity CA, but any pod in the cluster can obtain one, and the caller's identity was never checked. Any pod reaching atelet could drive actor lifecycles (Run/Checkpoint/Restore/Terminate) on the node. - Authorize the verified client cert's SPIFFE ID against `--herder-allowed-spiffe-id` (default `spiffe://cluster.local/ns/<atelet ns>/sa/ate-api-server`) during the TLS handshake. - Install client verification on the per-handshake config returned by GetConfigForClient. crypto/tls replaces the whole config with that one, so the same-node check previously set on the outer config of the ateom-facing credential broker never ran; it now does. - Drop hostPort 8085 from the atelet DaemonSet: ate-api-server dials the atelet Pod IP, so the node-IP exposure only widened reachability. Fixes agent-substrate#2073 #### Breaking change Only ate-api-server can call AteomHerder now. Anything else that called atelet on `:8085` (including via `<node-ip>:8085`) is rejected at the TLS handshake. Installs that rename the ate-api-server ServiceAccount or run it in another namespace must pass `--herder-allowed-spiffe-id` to atelet. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR (none needed)
…lag (agent-substrate#2884) `ate-setup` passes a `--preview` argument to every substrate binary it installs. Neither fake-atelet nor fake-workersync defined that flag, so both exited at startup with `unknown flag: --preview`, and a `--fake-data-plane` install never came up. This defines `--preview` in both binaries and initializes the preview gates from it, the same way the other control-plane binaries do. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR (none needed)
…substrate#2485) Bootstrap no longer turns off the Filestore add-on on a cluster it did not create. Clusters it creates still start with it off. Fixes agent-substrate#2357 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…rate#2178) Fixes agent-substrate#2154 Large GCS uploads left their last compose intermediates behind. Now removed on return, and the duplicated cleanup is shared. Test runs against fake-gcs-server. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR --------- Signed-off-by: Eric Curtin <eric.curtin@docker.com>
…eaves that node (agent-substrate#2893) Part of agent-substrate#777 ate-api-server checks that the caller's node hosts the actor when it mints a worker-for-actor JWT, but the token then lives for an hour and the credential exchange trusted any valid one. A node that kept a JWT after the actor resumed elsewhere could still exchange it for a read-write token on the actor's prefix, and the exchange also answered for crashed, deleting or reverting actors. The JWT now carries a `node` claim, the node it was minted for, set by `MintWorkerActorJwt` from the identity it already verified against `HostingNode`. The exchange refuses the token unless that node is still the actor's `status.assignedNode` and the actor is in a state with a transfer: RESUMING, RUNNING, SUSPENDING or SUSPENDED. `assignedNode` is set by a resume, stays set while the actor is SUSPENDED with its local snapshot on that node, and is cleared when the node detaches the actor, by a revert and by a delete, or replaced by a resume onto another node, so a stale JWT stops working at that status write rather than at expiry. An actor's own token never carries a node. The transfer states are listed once, in `resources.TransferStates`, read by both the host check and the exchange, with a test that fails when a new actor state is added without being classified. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…t-substrate#2723) Update `.github/SECURITY.md`, `docs/threat-model.md`, `docs/security.md`, and `AGENTS.md`: - **`.github/SECURITY.md`**: Transition to disclosure via GitHub Private Vulnerability Reporting; remove preview caveats, disclaimers, and unverified CI claims; clarify platform boundaries and supported versioning. - **`docs/threat-model.md`**: Remove preview notices and author headers; condense architecture component definitions to align with recent runtime naming (`ateworker`); clarify that threat mitigation quality is dynamically evaluated against code in CI rather than tracked statically in the reference catalog. - **`docs/security.md`**: Retain high-signal architectural principles while trimming transient implementation minutiae; mark `Delete Guarantee` and `Atespace Separation` as in-progress; update workload identity with on-demand JWTs; add cross-references to Authentication and Authorization guides. - **`AGENTS.md`**: Replace early-stage disclaimers with core zero-trust security invariants. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…ts (agent-substrate#2794) Finalize applies the snapshot state atelet returns from Checkpoint. When that state conflicts with the actor's record, finalize returned an error, and the comment said a retry could call atelet again. A retry can't recover. Atelet has already destroyed the workload, so the retried Checkpoint fails and the actor crashes with a reason that hides the conflict. This PR crashes the actor at the conflict instead. The crash reason names the conflict, matching finalize's other checks for a corrupt record, and atelet's error is logged. No proto or API changes. Earlier versions of this PR also recorded the async upload URI and named it on resume, released dropped durable snapshots, and tested the snapshot's locality in the template-repoint guard. Main now has all of these from other PRs, so this PR keeps only the crash.
…te#2695) Follow-up to agent-substrate#2626 The control plane only found out a sandbox died when the next operation on the actor failed, which for an idle actor could be never. - New WorkerService.ReportActorCrashed (and the atelet AteomSupport mirror), authorized like RequestActorSuspend. A RUNNING actor is crashed under its lease, anything else is refused since the operation in flight owns it. - ateworker-microvm reaps cloud-hypervisor / virtiofsd as soon as they exit, and ateworker-gvisor watches the runsc sandbox. If one dies while ateworker isn't stopping it, ateworker tears the actor down itself (the crash clears the assignment, so nothing else would), writes its final usage record, and reports it - Exits are watched through pidfds on the netpoller, so watching doesn't cost a thread per process - The gVisor sandbox is pinned right after runsc create, which closes the pid reuse gap @sfunkenhauser pointed out on agent-substrate#2626 - Not covered yet: a workload exiting while its sandbox stays up, and a micro-VM guest panic, which reboots the guest without cloud-hypervisor exiting. Both need their own design ... Tested with a new e2e test, TestSandboxCrashIsReported (kills the sandbox from an ephemeral container, expects CRASHED, then revert + resume), plus both full lanes on GKE with no false crash reports. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR (none needed) This PR was AI assisted. I have tested everything on GKE and reviewed the changes.
…agent-substrate#2888) agent-substrate#2433 ## Breaking change The binary wire format changes for some messages. Field names are unchanged, so protojson and `kubectl ate` output in JSON or YAML look the same. - **Stored data:** the ateapi store keeps Actors, ActorTemplates, Workers, ActorAssignments, Tags, Atespaces and EgressPolicies as binary protos. Rows written before this change decode with fields in the wrong places, and CRASHED, DELETING and REVERTING Actors read back with the wrong state. Existing installs need a fresh database. There is no migration. - **Control API clients:** `kubectl ate`, and anything else built against the old `ateapipb`, must be rebuilt. An old client talking to a new server misreads fields without any error.
…#2891) `deploy apiserver` now deploys ate-idp-server too. In the past only `deploy ate-system` used to deploy it, so an upgrade has to put it in step 4e, which a rollback does not undo, and a broken new `ate-idp-server` had no way back. The two share only the actor JWT pool and need no particular version of each other, so they can move together. Rolling back step 4a, which runs the old `deploy apiserver`, now rolls ate-idp-server back as well, without a new upgrade step. Part of agent-substrate#1686 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
Atelet now runs at most one lifecycle RPC per actor at a time. Run, Restore, Checkpoint, PromoteSnapshot, DetachActor, and Terminate wait their turn for the actor. Before, Terminate could forget an actor and delete its local snapshots and directories while a Checkpoint or Restore of the actor was still using them, and a Checkpoint that committed afterward recreated snapshot state for the terminated actor. Terminate waits for any in-flight RPC for that actor to finish before forgetting its snapshot state and removing its local snapshots and actor directories. With RPCs serialized at the atelet boundary, snapshottracker only coordinates background uploads, detaches, and reports against the active RPC. Part of agent-substrate#2069, agent-substrate#2571, agent-substrate#2836 - [ ] Tests pass - [ ] Appropriate changes to documentation are included in the PR
…thz in the architecture (agent-substrate#2868) Part of agent-substrate#2617. - The authz security regression suite now checks that all five WorkerService RPCs reject non-atelet callers. It adds `MintWorkerActorJwt` and `ReportActorSnapshotState` (served since agent-substrate#2570), each called with the bootstrap owner's token and with atelet's own ServiceAccount token. - Drops the stale "not served yet" notes for `ReportActorSnapshotState` in the authz registry and `docs/authorization.md`, and describes the check its handler makes. - Updates `docs/architecture.md` now that control-plane authorization is enforced (agent-substrate#2591): removes it from the unaddressed problems, describes how `ate-api-server` authenticates and authorizes RPCs, and states the remaining ingress gap (any allowlisted client can reach any actor; the default `deprecated-insecure` mode accepts any client). - [x] Unit tests, vet and build pass locally - [ ] E2E (authz) runs in CI (needs a cluster) - [x] Appropriate changes to documentation are included in the PR
…entity docs (agent-substrate#2597) Part of agent-substrate#1756. **Why:** every install still has to set `actorIdentityJWTProvider`, and `docs/authentication.md` says it decides who may mint actor JWTs. It no longer decides anything: - agent-substrate#757 added it when minting lived in a separate `ActorIdentity` service. `MintJWT` refused any caller whose JWT came from a different provider's issuer, which in practice limited minting to atelet's Kubernetes service account tokens. - agent-substrate#1315 moved minting into `ateapi.Control` and dropped that check, leaving caller limits to the authz framework. - agent-substrate#1834 removed the last code that read the provider's issuer. So ate-api-server requires a field and fails to start if it names a missing provider, yet nothing uses it, and the docs say it decides who may mint, which authorization now decides. Section 7 of `docs/api-guide.md` still describes the old service and its atelet-only rules. **What changes:** - `actorIdentityJWTProvider` is gone from ate-api-server's authentication config, and ate-setup stops writing it. - `docs/authentication.md` drops it and keeps the pointer to `authorization.md` for what a caller may do. - Section 7 of `docs/api-guide.md` now describes `MintActorJwt` and `MintActorCertificate` on `ateapi.Control`, who may call them, their fields, and the JWT's claims, instead of the old service, its caller checks, and a wrong SPIFFE path. The ADK paragraph, which pointed at the same service, now describes the egress gateway attaching actor JWTs. For reviewers: the config is parsed strictly and ate-setup only creates `ate-api-authentication` when it's missing, so an existing install must delete the `actorIdentityJWTProvider` line from that ConfigMap before ate-api-server restarts on this version. Testing: unit tests for the config loader and ate-setup's rendered config. I didn't run e2e; CI's e2e installs a fresh kind cluster, so it doesn't exercise the upgrade case. - [ ] Tests pass - [x] Appropriate changes to documentation are included in the PR
…#2256) Part of agent-substrate#1660. - The `egresscredinject` suite's echo-origin rules, http and https, now also replace `X-Actor-Token` with an actor JWT. The test decodes the token the in-cluster echo server sends back and checks `sub`, `aud`, the lifetime (`exp - iat`) and the `actor.ate.dev` claim, including the actor's UID. It also fails if the token is the `worker-for-actor` kind. - A fetch without the header must reach the echo server without `X-Actor-Token`, matching the existing check for `Authorization`. - `assertEchoedAuthorization` is now `assertEchoedHeaders`, so one fetch checks both headers. The test doesn't check the JWT's signature. That needs the published key set, which the OIDC discovery e2e (agent-substrate#2606) covers. Testing: lint and vet pass. The previous head passed e2e in CI on both sandbox classes; this rebase only changes an import path for the module rename. - [ ] Tests pass - [ ] Appropriate changes to documentation are included in the PR (the user guide is agent-substrate#2596)
…etric family (agent-substrate#2889) Follow-up to agent-substrate#2756, so that agent-substrate#2796 can give the snapshot cache a budget without a third copy of the eviction loop. **What moves.** The sandbox asset cache's eviction loop, flags, instruments, target formula and shortfall backoff move from `cmd/atelet/sandbox_asset_gc.go` into a package, `cmd/atelet/internal/filecachegc`, written against the filecache store. The loop's behavior is unchanged: immediate first pass, serialized passes, panic recovery, dry run, budget with a 90% low water, shortfall warnings that back off to a periodic reminder. **How a cache uses it.** - `filecachegc.NewFlags("sandbox-asset-cache", "sandbox asset cache")` registers `--sandbox-asset-cache-gc-period`, `--sandbox-asset-cache-max-bytes` and `--sandbox-asset-cache-gc-dry-run`, so the asset cache's flags are the same as before. The snapshot cache will call it once with its own prefix. - `filecachegc.NewInstruments(meter)` creates the instruments once per process. - `filecachegc.Start(ctx, name, store, flags, instruments)` reports the cache's size, warns once when there is no budget, and runs the loop when there is a budget and a period. **One metric family.** `ate.filecache.size`, `ate.filecache.evictions` and `ate.filecache.gc.passes`, labeled by `ate.filecache.name` (today `sandbox_asset`) and, for passes, `ate.filecache.gc.outcome`, replace `ate.sandboxassetcache.size`, `.evictions` and `.gc.passes`. The caches run on one store and one loop, so one set of instruments describes them, and "which cache is over budget on this node" is one query grouped by name. The registry, `ateattr` and the observability guide are updated. The name is `filecache` rather than something broader because the family is about the filecache store; the image cache, which measures layers against volume watermarks, is not a member and keeps its loop. **Tests.** The asset GC's tests move into the package unchanged in what they prove, plus one that puts two stores on the same instruments and asserts two size series under their names. **Docs and e2e.** The sandbox asset cache page from agent-substrate#2792 and the e2e metrics suite follow the rename in the last commit: the suite asserts `ate_filecache_size`, and the page's metrics table and runbook use the new names and say where the label comes from. **Follow-up.** agent-substrate#2796 adopts the driver for the snapshot cache and adds the `snapshot` label value. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
…t-substrate#2848) > The PR title becomes the release note. Write it for users: what changed for them, not how the code changed. Fixes agent-substrate#2847 `Uploader.plan` reserved `max(streaming, ranged)` for every file. A file that the GCS client splits into range parts never takes the streaming path, yet the streaming term grows with the budget and `GOMAXPROCS`. On a 192-vCPU node with a 16Gi budget, a 292 MiB snapshot reserved 2.6 GiB while its four parts hold 192 MiB, so only 5–6 uploads fit at once. Synchronous suspends spent 82% of their checkpoint time waiting for the budget; measurements are in agent-substrate#2847. This change reserves for the path an upload takes: - A file `PutSparseFile` will split (two or more parts) reserves `parts × sparseUploadMemory`. - A streamed object whose sparse stream is bounded below `uploadCompositeMin` reserves that single request's buffers (the `uploadCompositeMin` lookahead plus the client's `uploadChunkSize` chunk buffer) plus its workers, instead of the composite part buffers. - `Send` sizes a small manifest's reservation the same way. S3 is unchanged: it has no range path and always buffers parts. The minimum budget (`384Mi` GCS, `224Mi` S3) is unchanged too, because a streamed object larger than one request still needs the composite buffers. | GCS, 192 vCPUs, 16Gi budget | Reserved before | Reserved after | | --- | --- | --- | | 292 MiB memory image (4 parts) | 2,688 MiB | 192 MiB | | 611 MiB memory image (9 parts) | 5,248 MiB | 432 MiB | | Metadata file (bytes to KiB) | 384 MiB | 192 MiB | | Manifest | 320 MiB | 128 MiB | On the cluster described in agent-substrate#2847 (one run each, 64 SWE-bench actors next to synchronous probe suspends), probe suspend p95 went from 18.2 s to 3.7 s at 16Gi, and from 112–136 s to 42 s at the 512Mi default. Those runs used an earlier revision that didn't count the client's 64 MiB chunk buffer for single-request objects. This version reserves 64 MiB more per metadata file and manifest. Validation: - `go test -race ./pkg/objectstorage ./internal/objectstoreplugin/... ./cmd/objectstore-plugin -count=2` passes. - `golangci-lint run ./pkg/objectstorage/...` reports 0 issues; `hack/verify/gofmt.sh` and `hack/verify/boilerplate.sh` pass. - `BenchmarkConcurrentSnapshotUploads` is within run-to-run noise of `main` (3 runs each: 362–369 vs 371–373 files/s at the shared 512 MiB budget). Its in-memory destination doesn't take the GCS paths this changes. - New `TestUploaderReservesOnlyTheChosenPath`; `TestUploaderPlansFromPopulatedBytes` and `TestUploaderReservesBothPipelines` are updated for the GCS reservations. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…ll (agent-substrate#1509) Fixes agent-substrate#1076. ## The bug `ensureLayer` and `retireLayer` shared one singleflight key per layer, and that shared flight was the retire/reuse interlock. A caller that joins a flight rather than leading it runs no code and learns nothing about what the flight did. When the flight was a retirement, `ensureLayer` returned the layer path anyway, just after the rename had moved it aside, and the pull failed its final check with `layer dir vanished during pull`. The window is microseconds wide, so it shows up as a rare CI flake. ## The fix Count a pull's in-flight use of each layer instead of using the flight as a lock. - A pull pins each of its layers before any layer work and unpins when it returns. - `retireLayer` never retires a pinned layer. The pin check, a re-stat and the rename run under one mutex, so either the pin lands first and retirement is vetoed, or the rename lands first and the pull unpacks the layer again. A missing or fresh layer is ruled out before the mutex is taken. - Retirement no longer uses the layer singleflight, so a joined flight is always another pull's, and its result can be trusted. ## Tradeoff A wedged pull holds its layers until it returns, bounded by the pull timeout. Under disk pressure that holds space longer than before. In exchange, a slow pull can no longer lose layers it already finished.
…ers (agent-substrate#2885) ate-api-server grants the controller role (list, register and delete Workers) only to `--controller-spiffe-id`, which by default is ate-controller's identity. On a `--fake-data-plane` install, fake-workersync manages Workers in place of ate-controller, using its own identity. With authorization enforced, it was denied listing Workers and exited, so the install never became ready. The fake-data-plane kustomize component now sets fake-workersync as the controller identity. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR (none needed)
…rate#2735) ## What is this PR ? This PR adds: - **`ate_env` boomer user class (`AteEnvUser`)** in `internal/benchmarking/boomer/ateenv`: mirrors `glutton`'s per-VU actor lifecycle (`CreateAtespace`, `CreateActor`, `ResumeActorFirstResume` / `ResumeActor`, live window with `AteEnvExec`, `SuspendActor` / `PauseActor`, and `DeleteActor`), replacing HTTP `/ping` with `ProcessService.StartProcess(["echo", "hello world"])` + `ProcessService.StreamProcessOutput(follow: true)` over plaintext HTTP/2 gRPC directly through `atenet-router` (`ate-target-actor: <atespace>/<actor>`). - **`ate-env` workload manifest template** (`benchmarking/workloads/manifests/ate-env-template.yaml.tmpl`) and opt-in `WORKLOAD_TEMPLATES="ate-env"` support in `benchmarking/workloads/deploy.sh` (with `ATE_ENV_IMAGE` check matching `glutton-storage`). - **Locust harness stub** (`benchmarking/locust/tests/ate_env.py`) and documentation in `benchmarking/README.md`. --- ## Live Cluster Smoke Validation Validated on a GKE cluster (`4` `c3-standard-4` worker nodes, `4` `benchmark-worker` replicas, `256Mi` actor memory limit) across `suspend` and `pause` modes for both `gvisor` and `microvm`. ### Snapshot Size Comparison (`gvisor` vs `microvm`, `256Mi` Actor Memory Limit) | Metric | `gvisor` (`ate-env`) | `microvm` (`ate-env`) | |---|---|---| | **Golden Snapshot — Uncompressed Total (`atelet_snapshot_size_bytes`)** | **2.46 MiB** (`pages.img`: 2.13 MiB, `checkpoint.img`: 328 KiB, `pages_meta.img`: 4 KiB) | **62.43 MiB** (`memory-ranges`: 62.35 MiB, `state.json`: 64 KiB, `rootfs-upper-guest.tar`: 8 KiB, `base-id`: 4 KiB, `config.json`: 4 KiB) | | **Golden Snapshot — Compressed in GCS** | **430.3 KiB** (4 objects, ~5.8× compression) | **16.37 MiB** (6 objects, ~3.8× compression) | | **Per-Checkpoint after `AteEnvExec` (`suspend`) — Uncompressed Mean** | **4.30 MiB** (`pages.img` mean: 3.94 MiB, p50: 3.17 MiB, p90: 6.12 MiB, p99: 9.20 MiB; `checkpoint.img` mean: 365 KiB) | **66.65 MiB** (`memory-ranges` mean: 66.54 MiB, p50: 71.53 MiB, p90: 90.60 MiB; `state.json` mean: 69 KiB; `rootfs-upper-guest.tar` mean: 36 KiB) | | **Per-Checkpoint after `AteEnvExec` (`pause`) — Uncompressed Mean** | **4.61 MiB** (`pages.img` mean: 4.24 MiB, p50: 3.60 MiB, p90: 7.82 MiB, p99: 9.37 MiB; `checkpoint.img` mean: 365 KiB) | **67.32 MiB** (`memory-ranges` mean: 67.20 MiB, p50: 71.53 MiB, p90: 90.60 MiB; `state.json` mean: 70 KiB; `rootfs-upper-guest.tar` mean: 46 KiB) | - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…t-substrate#2879) Completes the removal of `sandbox_assets` from the `ate-api-server` -> `atelet` wire protocol following agent-substrate#2780 and agent-substrate#2781. Since `atelet` now resolves the pause image and sandbox binaries for `Run` and `Restore` directly from `sandbox_runtime` via its advertised-runtime index, `ate-api-server` no longer needs to resolve `SandboxConfig` or send `SandboxAssets` on `RunRequest` and `RestoreRequest`. - **`ateletpb` & validation**: Remove `sandbox_assets` from `RunRequest` and `RestoreRequest`, delete `SandboxAssets`, `ArchAssets`, and `AssetFile`, and make `sandbox_runtime` required on both RPCs. - **`cmd/ateapi` & RBAC**: Delete `sandbox_assets.go`, remove `sandboxConfigLister` and `workerPoolLister` from `ActorWorkflow` and `NewRPCService`, drop `sandboxconfigs` from `ate-api-server` RBAC, and require a non-nil `DefaultRuntime` in `scheduler.Applies`. - **`cmd/atelet`**: Read `op.sandboxClass` from `sandbox_runtime` in `Restore`, delete dead `recordFromRequest`, and replace `poolConfigName` with `WorkerPool.ResolvedSandboxConfig()`. - [ ] Tests pass - [ ] Appropriate changes to documentation are included in the PR
EItanya
marked this pull request as ready for review
October 19, 2026 23:55
EItanya
marked this pull request as draft
October 20, 2026 02:20
Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
EItanya
force-pushed
the
sync/upstream-main-20261019
branch
from
October 20, 2026 02:47
7ab9c87 to
24a6ef4
Compare
EItanya
marked this pull request as ready for review
October 20, 2026 10:45
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merge upstream main through
34bde5827(38 upstream commits) in one merge commit, preserving both histories and the fork's agentgateway configuration.This brings in cluster snapshot storage defaults, per-actor credential exchange, sandbox crash reporting, serialized lifecycle operations, image-cache fixes, and the updated worker protocol. The chart follows the new snapshot, identity, telemetry, and network settings. Conflict resolutions retain the fork's workflow deadline and local TLS options.
The previous E2E failures came from agentgateway decoding the old protobuf field numbers: the API returned a worker assignment, but the gateway discarded the actor status. The updated image includes matching control API bindings, including actor JWT requests and actor states.
Upgrade clients and components together; the protobuf changes are not wire compatible. Remove
DeleteActorRequest.any_state/--any-stateand the unusedactorIdentityJWTProvidersetting. External-storage Helm installations must setateApi.snapshotLocation; bundled RustFS derives it from its bucket. Per-template overrides requirePerTemplateSnapshotLocationpreview. See the chart upgrade notes.Validation: all 244 root Go packages with race detection, API-tool race tests, 36 Helm tests, chart lint, manifest rendering, CRD parity, and repository verification passed locally. The gateway fix passes 3 wire-format regression tests, 30 Substrate unit tests, and 92 integration tests. The regression tests fail against the old bindings. All five CI checks pass on
24a6ef49d: unit tests and verification, default and preview gVisor/microVM E2E, Helm E2E, and Helm verification. Local Kind E2E was not run.E2E totals (tests/skips): default gVisor 160/8 and microVM 160/7; preview 161/2 in each runtime. Helm passed gVisor 156/1, microVM 32/1, nine credential-injection tests, and one trust-rotation test.
Merge with a merge commit to retain upstream ancestry.