diff --git a/.github/SECURITY.md b/.github/SECURITY.md
index c94264a56e..cd1640ace0 100644
--- a/.github/SECURITY.md
+++ b/.github/SECURITY.md
@@ -1,44 +1,28 @@
# Security Policy
-Agent Substrate is in a pre-release (preview) state. This policy is
-intentionally minimal for that stage and will be revisited as part of the
-General Availability (GA) milestone, see [Road to GA](#road-to-ga) below.
-
-In particular, all of the following currently apply:
-
-- Reports can be made exclusively via GitHub advisories.
-- Maintainers triage and fix advisories on a best-effort basis.
-- There is no bug bounty.
-- There is no security mailing list for announcements.
-- There are no SLOs on fix timeline.
-- There are no embargos.
-- Though we do wait to publish advisories until fixes have been merged,
- details may appear in PRs prior to publishing.
-
## Reporting a Vulnerability
**Do not open a public GitHub issue or pull request for security
vulnerabilities.**
Use [GitHub Private Vulnerability Reporting](https://github.com/agent-substrate/substrate/security/advisories/new)
-to report privately. This is the only supported reporting channel; there is
-no security mailing list at this stage. There is no bug bounty, and this
-project is not eligible for the
+to report security vulnerabilities privately. Note that this project is not
+eligible for the
[Google Open Source Software Vulnerability Rewards Program](https://bughunters.google.com/open-source-security).
Please include:
-- The affected component and version (commit SHA).
+- The affected component and version or commit SHA.
- Steps to reproduce, or a proof of concept.
- The potential impact, and mitigations if known.
### When to report
-- You think you found a potential security vulnerability in Agent Substrate.
-- You are unsure whether an issue affects Agent Substrate.
+- You found a potential security vulnerability in Agent Substrate.
+- You are unsure whether a security issue affects Agent Substrate.
- You found a vulnerability in a project Agent Substrate depends on and it
is exploitable through Agent Substrate. If that project has its own
- disclosure process, please report it there first.
+ disclosure process, please report it there as well.
### When not to report
@@ -48,16 +32,20 @@ Please include:
## What Counts as a Vulnerability
-See the [threat model](../docs/threat-model.md) for the trust boundaries and
-assumptions this section is based on.
+See the [Security Guide](../docs/security.md) and
+[Threat Model](../docs/threat-model.md) for the security architecture, trust
+boundaries, and assumptions this section is based on.
-In scope: the Agent Substrate control plane (`ateapi`), node supervisor
-(`atelet`, `ateworker`), networking stack (`atenet`), and CLI (`kubectl-ate`).
+In scope: the Agent Substrate control plane (`ateapi`, `atecontroller`,
+`podcertcontroller`), node and sandbox supervisors (`atelet`, `ateworker-gvisor`,
+`ateworker-microvm`, `objectstore-plugin`), networking and credential plane
+(`atenet`, `cmd/credential-provider/kubernetes-secrets`), and CLI
+(`kubectl-ate`, `ate-setup`).
-Agent Substrate multiplexes many actors onto a smaller pool of shared
-workers, so its most important security boundaries are between actors and
-between actors and the platform. Reports in these classes are the most
-valuable, listed roughly from most to least severe:
+Agent Substrate multiplexes untrusted actors onto a shared pool of workers, so
+its primary security boundaries are between actors and between actors and the
+platform. Reports in these classes are the highest priority, listed roughly
+from most to least severe:
- **Cross-actor access**: an actor reading or modifying another actor's
state (memory, filesystem, working memory) on a shared worker, or acting
@@ -65,67 +53,79 @@ valuable, listed roughly from most to least severe:
- **Snapshot integrity and confidentiality**: reading another actor's
suspend/resume snapshot, or resuming an actor from a tampered snapshot.
- **Routing**: traffic intended for one actor delivered to another.
-- **Identity**: forging actor identity (for example actor JWTs).
+- **Identity**: forging actor identity (such as actor SPIFFE X.509
+ certificates or JWTs) or component PodCertificates.
- **Control-plane authentication and authorization bypass**: reaching
`ateapi` RPCs without valid credentials (mTLS or JWT auth modes),
- misusing workerpool client certificates, or acting beyond what valid
- credentials are authorized to do.
+ bypassing OpenFGA authorization checks, misusing worker or node client
+ certificates, or acting beyond what valid credentials are authorized to do.
- **Actor-to-platform escalation**: an actor compromising `atelet`, `ateworker`,
- or the control plane, whether while the actor is still actively running
- or beyond its own lifecycle.
+ `atenet`, or the control plane, whether while the actor is actively running
+ or across lifecycle transitions.
- **Node lateral movement**: using a compromised worker node to reach other
- nodes, atespaces, or the control plane.
-- **Network isolation bypass**: an actor sending or receiving traffic that
- `atenet` policy was meant to prevent.
+ nodes, unscheduled actors, other atespaces, or the control plane.
+- **Network or credential isolation bypass**: an actor sending or receiving
+ traffic that `atenet` policy blocks, or accessing credentials outside its
+ authorized `EgressPolicy`.
### Out of scope
- The underlying Kubernetes cluster or cloud infrastructure.
-- The sandbox runtimes (gVisor, Kata Containers); report those to their
- respective projects.
-- The demo applications under `demos/`; they are examples, not production
- code.
-- Known limitations listed in [docs/roadmap.md](../docs/roadmap.md) and
- [AGENTS.md](../AGENTS.md), at the time of your report. These are living
- docs and can lag reality; if you're unsure whether a listed limitation is
- still accurate, report it anyway and let a maintainer make the call.
+- Upstream sandbox runtimes (gVisor, Cloud Hypervisor); report vulnerabilities
+ contained within those runtimes to their respective projects.
+- Sample applications under `demos/` and load-testing harnesses under
+ `benchmarking/`.
+- Documented scope boundaries and future capabilities listed in
+ [docs/security.md](../docs/security.md) and
+ [docs/roadmap.md](../docs/roadmap.md) at the time of your report. If you are
+ unsure whether a finding is covered by a documented limitation, report it
+ privately anyway and let a maintainer make the call.
### What usually does not qualify
- Denial of service against workloads deployed without resource limits.
- Issues that require cluster-admin permissions or root on the node; that
level of access already controls the platform.
+- Issues that require attacker control of the ActorTemplate.
- Vulnerability scanner or CVE reports about dependencies without an
exploitable path through Agent Substrate (see
[Dependencies](#dependencies)).
-- Sandbox escapes contained entirely within gVisor or Kata Containers;
- report those upstream (see [Out of scope](#out-of-scope)).
If you are unsure whether something qualifies, report it privately anyway.
-## Triage and Response
+## Triage, Coordinated Disclosure, and Advisories
-While the project is in preview there is no dedicated security team or
-response committee. Reports are triaged and fixed by the project maintainers,
-who own the repository.
+Reports are promptly triaged by the Agent Substrate maintainers. When a valid
+vulnerability is confirmed:
-What to expect at this stage:
+- We work with the reporter on the private advisory thread to validate the
+ issue and develop a remediation.
+- Fixes are prepared and patched on supported releases.
+- We coordinate disclosure timing with the reporter and publish a
+ [GitHub Security Advisory](https://github.com/agent-substrate/substrate/security/advisories)
+ describing the vulnerability, affected versions, and remediation, requesting
+ a CVE through GitHub when warranted.
-- We aim to acknowledge reports within a few business days, but there are no
- formal response or fix time SLOs yet.
-- Fixes are best effort and land on `main`. All valid reports will be
- patched by GA.
-- There is no embargo process and no pre-disclosure notification to
- downstream users or vendors. Though we do wait to publish advisories until
- fixes have been merged, some details may appear in PRs prior to publishing.
+We ask reporters to keep vulnerability details private until the coordinated
+advisory is published. Reporters are credited in the advisory unless they
+prefer to remain anonymous.
-We ask reporters to keep vulnerability details private until an advisory is
-published; we prefer to disclose as soon as a fix is available.
+## Supported Versions
+
+After release 1.0.0, security fixes will be applied to `main` and to supported
+stable releases.
+
+## Dependencies
+
+Container images are built on distroless bases. If a dependency vulnerability
+is exploitable through Agent Substrate, report it to us privately as well as
+upstream; otherwise a regular issue or pull request bumping the dependency is
+sufficient.
## Good-Faith Research
-We expect security research against Agent Substrate to be done in good
-faith, meaning:
+We expect security research against Agent Substrate to be conducted in good
+faith:
- Test only against clusters and deployments you own or are authorized to
test.
@@ -134,46 +134,5 @@ faith, meaning:
- Do not degrade service for others while testing.
This policy covers the Agent Substrate open source project only. It does not
-authorize testing of anyone else's deployment of Agent Substrate; those are
-governed by the respective operator's own policies.
-
-## Dependencies
-
-Dependency vulnerabilities are tracked with `govulncheck` in CI, and fixes
-are applied by updating the dependency on `main`. If a dependency
-vulnerability is exploitable through Agent Substrate, report it to us
-privately as well as upstream; otherwise a regular issue or pull request
-bumping the dependency is enough.
-
-## Supported Versions
-
-There are no stable releases yet. Security fixes are applied to `main` only;
-there are no release branches and no backports.
-
-## Disclosure
-
-After a fix is merged, we publish a
-[GitHub Security Advisory](https://github.com/agent-substrate/substrate/security/advisories)
-describing the vulnerability and the fix, and request a CVE through GitHub
-when warranted. We publish as soon as the fix is verified on `main`; there
-is no coordinated embargo or pre-disclosure period at this stage.
-
-Reporters are credited in the advisory unless they prefer to remain
-anonymous.
-
-## Road to GA
-
-As the project matures we expect to define, at or before GA:
-
-- What triggers the switch from this preview policy to the full process.
-- A defined triage process and ownership (for example a security response
- committee or rotation).
-- Response and fix SLOs by severity.
-- A security announcements channel (for example a mailing list).
-- An embargo and coordinated disclosure process, including pre-disclosure
- notification for downstream distributors.
-- A patch and release process for supported release branches, including
- backports.
-- Whether to incentivize reports with a bug bounty program.
-
-Until then, the preview-state process above applies.
+authorize testing of third-party deployments of Agent Substrate, which are
+governed by their respective operators' policies.
diff --git a/.golangci.yaml b/.golangci.yaml
index 3d70dd8b22..086c31d1cc 100644
--- a/.golangci.yaml
+++ b/.golangci.yaml
@@ -36,12 +36,14 @@ linters:
depguard:
rules:
# internal/ateworker is the ateworker binaries' shared code. Anything
- # else should import a flat package under internal/ instead.
+ # else should import a flat package under internal/ instead. The atunnel
+ # benchmark hosts atunnel as a worker does, so it may import it too.
ateworker:
files:
- "$all"
- "!**/cmd/ateworker-*/**"
- "!**/internal/ateworker/**"
+ - "!**/benchmarking/atunnel/**"
deny:
- pkg: go.ate.dev/substrate/internal/ateworker
desc: only the ateworker binaries may import internal/ateworker
diff --git a/AGENTS.md b/AGENTS.md
index e9ce36462f..6254f37216 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -96,14 +96,9 @@ See the [metric registry](docs/observability.md#the-metric-registry) section of
## Security Considerations
-The security story for Substrate is very early and many features are missing.
-However! Take care to respect security best practices when writing code in order to improve Substrate's security over time.
-The following is what Substrate currently offers.
-Keep this up to date when updating AGENTS.md.
+Agent Substrate is built for zero-trust, multi-tenant execution of untrusted actors. Preserve and extend these invariants when modifying the codebase (see `docs/security.md` and `docs/threat-model.md` for the full architecture):
-- **Workload Isolation**: The project uses `gVisor` (`runsc`) for sandboxing and security isolation of workloads on pods.
-- **JWTs and JWKs**: Signing and JWK set encoding (`internal/localjwtauthority`) and verification (`internal/oidcjwt`) are built on the standard library's crypto packages. Extend those packages rather than adding go-jose or another JOSE library, so the code that decides whether a token is valid stays small and supports only the algorithms Substrate uses.
-- **Ingress Authentication**: `atenet-router --ingress-auth-mode=static-mtls` requires clients to present a SPIFFE certificate signed by `--ingress-client-ca-file` whose ID is in `--ingress-allowed-spiffe-ids`; that list is the whole authorization policy, so any listed client may reach any Actor. The default, `deprecated-insecure`, checks a certificate only when a client presents one, and also serves plaintext listeners that accept any client. Install with it via `ate-setup --ingress-auth-mode=static-mtls` (`manifests/ate-install/components/router-static-mtls`); the allowlist lives in `manifests/ate-install/atenet-router.yaml`.
-- **Control API Authorization**: `ate-api-server` checks every Control RPC against an embedded OpenFGA model (`cmd/ateapi/internal/authz/model.fga`), on by default. Every served unary RPC needs an entry in `defaultRPCPermissions` (`registry.go`): with enforcement on, an RPC without one is denied, and a test fails the build. Access is granted per global scope and atespace through AccessPolicy (`kubectl ate access-policy`); the install's bootstrap owner is the shared `ate-client` ServiceAccount that `kubectl ate` uses. See `docs/authorization.md` for roles and known gaps.
-
-For future plans for security, reference `docs/roadmap.md`.
+- **Workload & Storage Isolation**: Actors run strictly inside `gVisor` (`cmd/ateworker-gvisor`) or `Cloud Hypervisor` microVM (`cmd/ateworker-microvm`) sandboxes; plain Linux containers are not supported. `atelet` scrubs actor directories across lifecycle transitions (`resetActorDirs`), and snapshots are stamped with `SandboxRuntime` metadata and transferred via `cmd/objectstore-plugin`.
+- **Component & Actor Identity (mTLS, JWTs, and JWKs)**: Inter-component communication uses mTLS backed by `cmd/podcertcontroller` (`podid`, `servicedns`, and `postgres` signers). Actor JWT signing (`internal/localjwtauthority`), verification (`internal/oidcjwt`), and key discovery (served via `cmd/ate-idp-server`) are built on the standard library's crypto packages. Extend those packages rather than adding `go-jose` or another JOSE library, so the code that decides whether a token is valid stays small and supports only the algorithms Substrate uses.
+- **Control-Plane Authentication & Authorization**: `ate-api-server` checks every Control RPC against an embedded OpenFGA model (`cmd/ateapi/internal/authz/model.fga`), on by default. Every served unary RPC needs an entry in `defaultRPCPermissions` (`registry.go`): with enforcement on, an RPC without one is denied, and a test fails the build. Access is granted per global scope and atespace through AccessPolicy (`kubectl ate access-policy`); the install's bootstrap owner is the shared `ate-client` ServiceAccount that `kubectl ate` uses. See `docs/authorization.md` for roles and known gaps. Callers are authenticated via `SPIFFE` mTLS or configured OIDC `JWTProvider`s (`cmd/ateapi/internal/apiauthn`).
+- **Network Isolation, Ingress, & Egress**: `manifests/ate-install/ate-system-networkpolicies.yaml` enforces default-deny ingress across `ate-system`, and `cmd/atecontroller` reconciles per-`WorkerPool` `NetworkPolicy` objects. Actors run in private `/30` point-to-point veth network namespaces (`internal/ateworker/actornet`) with link-local blocking, policy-enforced DNS filtering (`internal/atunnel/dns`), and zero ambient credentials. `atenet-router --ingress-auth-mode=static-mtls` requires clients to present a `SPIFFE` certificate signed by `--ingress-client-ca-file` whose ID is in `--ingress-allowed-spiffe-ids` (`manifests/ate-install/atenet-router.yaml`). Outbound traffic traverses `atunnel` to `atenet-egress`, which enforces per-actor `EgressPolicy` and injects credentials out-of-band via `cmd/credential-provider/kubernetes-secrets` (`k8s-credential-provider`) or actor JWT minting.
diff --git a/README.md b/README.md
index 6d8469001e..53823c2104 100644
--- a/README.md
+++ b/README.md
@@ -256,6 +256,7 @@ We provide several sample applications demonstrating Agent Substrate's capabilit
* [Egress Traffic](docs/egress-traffic.md): Which protocols an Actor may reach the outside world with and which are blocked.
* [Enabling man-in-the-middle (MITM) interception for Actor Egress policy](docs/egress-trust-bundle.md): Egress policies such as header injection depend on MITM interception of Actor traffic. This guide explains how an Actor should be configured to enable interception.
* [Request Parking](docs/request-parking.md): How the router parks requests through transient worker-pool saturation.
+* [Sandbox Asset Cache](docs/sandbox-asset-cache.md): How a node caches the sandbox binaries actors run on, decides which runtimes it needs, and evicts to a budget.
* [Upgrade Runbook](docs/upgrade.md): Upgrade a running install to a newer supported release, and roll back if needed.
* [Security Guide](docs/security.md): Security overview, security principles, and threat modeling.
* [Roadmap](docs/roadmap.md): Current limitations and what is planned next.
diff --git a/benchmarking/README.md b/benchmarking/README.md
index ae94e73bc4..51475da90e 100644
--- a/benchmarking/README.md
+++ b/benchmarking/README.md
@@ -270,6 +270,37 @@ nothing. Durations are in milliseconds.
are canceled: interrupted tasks log no task record, and startup failures logged after it
belong to the teardown, not the run.
+### ate-env Benchmark
+
+The `ate-env` benchmark (`--user-class ate_env`) exercises the
+[`ate-env-guest`](https://github.com/agent-substrate/env/tree/main/cmd/ate-env-guest)
+sandbox image from `agent-substrate/env`. `workloads/deploy.sh` deploys and
+deletes the `ate-env` `ActorTemplate` by default using the pinned release guest
+image (see
+[`workloads/manifests/ate-env-template.yaml.tmpl`](workloads/manifests/ate-env-template.yaml.tmpl)
+to override the guest image with a custom build). It follows `glutton`'s
+wake/hold/hibernate lifecycle, but replaces synthetic HTTP `/ping` calls with a
+real `"hello world"` process execution inside the guest (`StartProcess(["echo",
+"hello world"])` + `StreamProcessOutput(follow: true)`) over gRPC directly
+through `atenet-router` (`1 dns:///` gRPC channel per VU).
+
+#### ate-env Configuration Knobs
+
+Reuses `glutton`'s runtime and startup knobs:
+`--min-wait-time`, `--max-wait-time`, `--min-live-time`, `--max-live-time`,
+`--max-pings-per-wake`, `--lifecycle-mode` (`suspend` | `pause` | `none`),
+`--trace-probability`, and `--actors-per-user`.
+
+#### ate-env Reported Metrics
+
+* `CreateAtespace`, `CreateActor`, `ResumeActorFirstResume`, `ResumeActor`,
+ `SuspendActor`, `SuspendActorAsync`, `DeleteActor`: Control-plane lifecycle
+ RPC latencies.
+* `AteEnvExec`: Combined end-to-end latency of `"hello world"` (`StartProcess` +
+ `StreamProcessOutput` to exit with `exit_code == 0` and
+ `stdout == "hello world\n"`).
+* `CrashCount`: Actors that entered `ACTOR_STATE_CRASHED` during resume.
+
### Agent-Session Benchmark
The agent-session benchmark (`--user-class agentsession`) emulates a fleet of
diff --git a/benchmarking/atunnel/.gitignore b/benchmarking/atunnel/.gitignore
new file mode 100644
index 0000000000..64614b1d4c
--- /dev/null
+++ b/benchmarking/atunnel/.gitignore
@@ -0,0 +1,5 @@
+# Keep each run's results and report; the rest is large or regenerated.
+runs/*/*
+!runs/*/results.txt
+!runs/*/report.md
+runs/.work-*
diff --git a/benchmarking/atunnel/Dockerfile b/benchmarking/atunnel/Dockerfile
new file mode 100644
index 0000000000..334e3f17ea
--- /dev/null
+++ b/benchmarking/atunnel/Dockerfile
@@ -0,0 +1,23 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# One image holding the benchmark's one program; each container picks its
+# subcommand. The program is built outside, statically for linux, so the build
+# context is just it (run-docker.sh does this):
+#
+# CGO_ENABLED=0 GOOS=linux go build -o ctx/ ./benchmarking/atunnel/cmd/...
+# docker build -f benchmarking/atunnel/Dockerfile ctx/
+FROM gcr.io/distroless/static-debian12:nonroot
+
+COPY atunnel-bench /usr/local/bin/
diff --git a/benchmarking/atunnel/README.md b/benchmarking/atunnel/README.md
new file mode 100644
index 0000000000..2bc097d1f4
--- /dev/null
+++ b/benchmarking/atunnel/README.md
@@ -0,0 +1,141 @@
+# atunnel benchmark
+
+Measures what atunnel, the per-worker ingress and egress tunnel, costs in CPU
+and memory as load rises. atunnel is started and hosts its actors with the
+worker's own code, in a process of its own, so the numbers are atunnel's alone.
+
+```mermaid
+flowchart LR
+ A["A: load generator"] -- traffic --> B["B: atunnel"]
+ B -- traffic --> C["C: ingress actors
and fake gateway"]
+ A -. "reads B's CPU and memory" .-> B
+```
+
+The generator (A) sends traffic through atunnel (B) to fakes of what atunnel
+talks to in a cluster (C). A reads B's CPU and memory as it goes and prints a
+table per scenario.
+
+## Running
+
+Linux containers only: atunnel creates real network namespaces and nftables
+rules, so its container is privileged. On macOS, Docker Desktop provides the
+Linux.
+
+```sh
+benchmarking/atunnel/run-docker.sh # atunnel at 1 CPU, 1 GiB
+benchmarking/atunnel/run-docker.sh --tunnel-cpus 2 --tunnel-memory 2g -- --rungs 2000,4000,8000
+benchmarking/atunnel/run-docker.sh --cpuset-worker 0-3 --cpuset-rig 4-9 -- --scenarios Connect
+```
+
+Options before `--` shape the containers (`--tunnel-cpus`, `--tunnel-memory`,
+`--actors`, `--cpuset-worker`, `--cpuset-rig`); everything after goes to
+`atunnel-bench load` (`--scenarios`, `--rungs`, `--step`, `--warmup`, `--workers`).
+
+Each run writes to `benchmarking/atunnel/runs//`:
+`results.txt`, `results.json`, `profiles/` (a CPU and a heap profile of atunnel
+per rung), and `logs/`. Only `results.txt`, and a `report.md` if you write
+one, are kept in git. Open a profile with
+`go tool pprof -http=: runs/.../profiles/IngressH1-4000.cpu.pb.gz`.
+
+## How it works
+
+The benchmark is one program, `atunnel-bench`, with a subcommand per process.
+`run-docker.sh` runs it in three containers on one Docker network:
+
+```mermaid
+flowchart LR
+ subgraph L["atunnel-load"]
+ A["A: generator"]
+ end
+ subgraph W["atunnel-worker (privileged)"]
+ B["B: atunnel
own cgroup"]
+ S["actor sandboxes:
ingress actors,
egress actor"]
+ end
+ subgraph K["atunnel-sink"]
+ G["fake egress gateway"]
+ end
+ A -- "ingress: HTTPS :443, CONNECT :8443" --> B
+ B -- "into each sandbox" --> S
+ S -- "egress, redirected by nftables" --> B
+ B -- "egress CONNECT" --> G
+```
+
+Which subcommand runs where:
+
+| Subcommand | Container | Runs |
+|---|---|---|
+| `worker` | `atunnel-worker` | the rest of the worker pod: the fake atelet, `tunnel` in its own cgroup, and the actors in the sandboxes |
+| `tunnel` | `atunnel-worker`, started by `worker` | atunnel, hosting actors with the worker's own code: the process measured |
+| `ingress-actor`, `egress-actor` | `atunnel-worker`, inside the sandboxes | the actors' stand-ins, started by `worker` |
+| `sink` | `atunnel-sink` | the fake egress gateway |
+| `load` | `atunnel-load` | the load generator |
+| `pki` | run once, before the containers | the throwaway certificates the others share |
+
+### What is real
+
+`atunnel-bench tunnel` makes the calls `cmd/ateworker-gvisor` makes, in the
+same order:
+- at startup: `tunnel.Start` with the worker's own flags, through
+ `tunnel.RegisterFlags`;
+- per actor: `tunnel.PrepareEgress`, `actornet.ServeSandbox`, and
+ `tunnel.Activate`;
+- at teardown: `tunnel.Deactivate`, then closing the sandbox network.
+
+Each actor gets real network namespaces, the nftables redirect, an egress
+listener and DNS relay in its sandbox, and a certificate minted through
+atunnel's own atelet client.
+
+### What is faked
+
+| In a worker | Stand-in |
+|---|---|
+| atelet | `worker` serves the two atelet calls atunnel makes, certificate minting and egress policy |
+| the actor's program receiving requests | an ingress actor: a small HTTP server in each sandbox |
+| the actor's program making outgoing connections | the egress actor, in actor-0's sandbox |
+| the router | the generator, presenting the router's identity |
+| the egress gateway | `sink`, with the real gateway's Envoy HTTP/2 settings; it never dials out |
+| Pod certificates | files from a throwaway CA |
+
+### How atunnel is measured
+
+- atunnel runs in its own cgroup with the CPU and memory limits and no swap,
+ so going over the limit is an OOM kill, as in a pod. The actors run outside
+ it. If atunnel dies, `worker` logs it and starts a fresh one.
+- The generator is open loop: work is scheduled at a fixed rate whatever
+ earlier requests take, and latency is measured from when each unit was due.
+- For each rung, the generator reads atunnel's `/stats` (CPU, memory,
+ goroutines) when the warmup ends, every 500 ms after, and when the rung ends.
+- atunnel's process also holds the other subcommands' code, which it never
+ runs. That costs about 3 MB: atunnel idles at about 29 MB hosting 10 actors,
+ against 26 MB built on its own.
+
+## Scenarios
+
+Each scenario runs at every rate in `--rungs` (default `1000,2000,4000,8000`
+per second), for `--step` (10s) each, leaving the first `--warmup` (2s) out.
+
+The router reaches atunnel in one of two ways, set by its `--upstream-connect`
+flag: through CONNECT tunnels (the default), or over HTTPS (`false`).
+
+| Scenario | Router mode | What is sent | Tells you |
+|---|---|---|---|
+| IngressH1 | `--upstream-connect=false` | HTTPS GETs over HTTP/1.1 keep-alive to `:443`, the `ate-target-actor` header cycling over the actors | CPU per request through atunnel's reverse proxy |
+| IngressH2 | `--upstream-connect=false` | the same over HTTP/2 | the same, multiplexed |
+| Connect | default | per unit: new TLS connection to `:8443`, `CONNECT actor:80`, one GET inside | cost of setting up a tunnel, mostly the TLS handshake |
+| EgressHTTP | either | plain HTTP keep-alive from inside a sandbox | CPU per egress request on reused connections |
+| EgressNewConnection | either | the same with a new connection per request | cost of a new egress connection |
+| EgressIdleConns | either | rungs are connection counts: open at 200/s, one request each, hold | memory per open tunnel, read after a GC |
+| EgressTCP | either | 4 connections write 64 KiB chunks as fast as they can; first rung only | throughput and CPU per byte |
+
+## Caveats
+
+- **Not covered:** gVisor itself, the real router and gateway, and actors being
+ activated and deactivated during load.
+- **Shared machine.** The containers share the Docker host's CPUs. Pin them
+ apart with `--cpuset-worker` and `--cpuset-rig` for clean capacity numbers.
+- **Ephemeral ports.** Connect and EgressNewConnection leave tens of thousands
+ of sockets in TIME_WAIT. `run-docker.sh` widens the port range and enables
+ `tcp_tw_reuse` in its containers, and the egress actor does the same in its
+ sandbox.
+- **Heap figures** are Go's `HeapInuse`; for what is live, see the
+ EgressIdleConns line, which collects garbage first.
diff --git a/benchmarking/atunnel/cmd/atunnel-bench/commands_linux.go b/benchmarking/atunnel/cmd/atunnel-bench/commands_linux.go
new file mode 100644
index 0000000000..152e6d6788
--- /dev/null
+++ b/benchmarking/atunnel/cmd/atunnel-bench/commands_linux.go
@@ -0,0 +1,31 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//go:build linux
+
+package main
+
+import "github.com/spf13/cobra"
+
+// linuxCommands are the subcommands that host atunnel with the worker's code,
+// which uses Linux network namespaces, nftables, and cgroups.
+func linuxCommands() []*cobra.Command {
+ return []*cobra.Command{
+ tunnelCommand(),
+ workerCommand(),
+ ingressActorCommand(),
+ egressActorCommand(),
+ netnsExecCommand(),
+ }
+}
diff --git a/benchmarking/atunnel/cmd/atunnel-bench/commands_other.go b/benchmarking/atunnel/cmd/atunnel-bench/commands_other.go
new file mode 100644
index 0000000000..ab924d5a75
--- /dev/null
+++ b/benchmarking/atunnel/cmd/atunnel-bench/commands_other.go
@@ -0,0 +1,23 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//go:build !linux
+
+package main
+
+import "github.com/spf13/cobra"
+
+// linuxCommands host atunnel with the worker's code, which needs Linux; see
+// commands_linux.go.
+func linuxCommands() []*cobra.Command { return nil }
diff --git a/benchmarking/atunnel/cmd/atunnel-bench/load.go b/benchmarking/atunnel/cmd/atunnel-bench/load.go
new file mode 100644
index 0000000000..6f8a2ca961
--- /dev/null
+++ b/benchmarking/atunnel/cmd/atunnel-bench/load.go
@@ -0,0 +1,117 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package main
+
+import (
+ "fmt"
+ "net"
+ "os"
+ "path/filepath"
+ "strconv"
+ "time"
+
+ "github.com/spf13/cobra"
+
+ "go.ate.dev/substrate/benchmarking/atunnel/internal/load"
+ "go.ate.dev/substrate/benchmarking/atunnel/internal/pki"
+ "go.ate.dev/substrate/benchmarking/atunnel/internal/report"
+)
+
+func loadCommand() *cobra.Command {
+ var r load.Runner
+ var pkiDir, tunnelHost, scenarios, resultsPath string
+ var ingressPort, connectPort int
+ var rungs []float64
+ cmd := &cobra.Command{
+ Use: "load",
+ Short: "Drive the scenarios through atunnel and report its CPU and memory per rate",
+ Long: `load drives traffic through atunnel at rising rates, ingress from here and
+egress through the egress actor in an actor's sandbox, and reports, per rate,
+atunnel's own CPU and memory as read from its admin endpoint.`,
+ Args: cobra.NoArgs,
+ RunE: func(cmd *cobra.Command, _ []string) error {
+ selected, err := load.Select(scenarios)
+ if err != nil {
+ return err
+ }
+ for _, rate := range rungs {
+ if rate <= 0 {
+ return fmt.Errorf("--rungs: %v is not a positive number", rate)
+ }
+ }
+ if len(rungs) == 0 {
+ return fmt.Errorf("--rungs is empty")
+ }
+ if r.Warmup >= r.Step {
+ return fmt.Errorf("--warmup must be shorter than --step")
+ }
+ if r.ProfileDir != "" {
+ if err := os.MkdirAll(r.ProfileDir, 0o755); err != nil {
+ return err
+ }
+ }
+ at := func(port int) string { return net.JoinHostPort(tunnelHost, strconv.Itoa(port)) }
+ r.Targets = load.Targets{
+ Ingress: at(ingressPort),
+ Connect: at(connectPort),
+ ServerName: tunnelHost,
+ RouterBundle: filepath.Join(pkiDir, pki.RouterBundle),
+ CA: filepath.Join(pkiDir, pki.CAFile),
+ }
+
+ ctx := cmd.Context()
+ if err := r.Prepare(ctx); err != nil {
+ return err
+ }
+ var reports []*report.Report
+ for _, s := range selected {
+ fmt.Fprintf(os.Stderr, "running %s...\n", s.Name)
+ rep, err := r.Run(ctx, s, rungs)
+ if rep != nil {
+ fmt.Println(rep)
+ reports = append(reports, rep)
+ }
+ if resultsPath != "" {
+ // Rewritten after every scenario, so an interrupted
+ // run keeps what it measured.
+ if err := report.Write(resultsPath, reports); err != nil {
+ return err
+ }
+ }
+ if err != nil {
+ return err
+ }
+ }
+ return nil
+ },
+ }
+ f := cmd.Flags()
+ f.StringVar(&pkiDir, "pki-dir", "", "directory written by atunnel-bench pki")
+ f.StringVar(&tunnelHost, "tunnel-host", "atunnel-worker", "host atunnel runs on")
+ f.IntVar(&ingressPort, "ingress-port", 443, "atunnel's ingress HTTPS port (--atunnel-listen-address)")
+ f.IntVar(&connectPort, "connect-port", 8443, "atunnel's ingress CONNECT port (--atunnel-connect-listen-address)")
+ f.StringVar(&r.TunnelAdmin, "tunnel-admin", "http://atunnel-worker:9090", "atunnel's admin base URL")
+ f.StringVar(&r.SinkAdmin, "sink-admin", "http://atunnel-sink:9091", "the sink's admin base URL")
+ f.StringVar(&r.SharedDir, "shared-dir", "/shared", "directory shared with the worker container, holding the egress actor's and ingress actors' sockets")
+ f.StringVar(&scenarios, "scenarios", "all", "comma-separated scenarios to run, or all: "+load.Names())
+ f.Float64SliceVar(&rungs, "rungs", []float64{1000, 2000, 4000, 8000}, "offered rates per second, in order; connection counts for EgressIdleConns")
+ f.DurationVar(&r.Step, "step", 10*time.Second, "length of each rung")
+ f.DurationVar(&r.Warmup, "warmup", 2*time.Second, "leading part of each rung left out of the measurement")
+ f.IntVar(&r.Workers, "workers", 256, "units in flight at most")
+ f.StringVar(&r.ProfileDir, "profile-dir", "", "save a CPU and a heap profile of atunnel per rung into this directory")
+ f.StringVar(&resultsPath, "results-json", "", "write every scenario's results to this path as JSON")
+ _ = cmd.MarkFlagRequired("pki-dir")
+ return cmd
+}
diff --git a/benchmarking/atunnel/cmd/atunnel-bench/main.go b/benchmarking/atunnel/cmd/atunnel-bench/main.go
new file mode 100644
index 0000000000..8cd4fd52ec
--- /dev/null
+++ b/benchmarking/atunnel/cmd/atunnel-bench/main.go
@@ -0,0 +1,53 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// atunnel-bench measures what atunnel costs in CPU and memory as load rises.
+// Every process of the benchmark is a subcommand of this one program; see
+// run-docker.sh for how they fit together.
+package main
+
+import (
+ "context"
+ "fmt"
+ "os"
+ "os/signal"
+ "syscall"
+
+ "github.com/spf13/cobra"
+)
+
+func main() {
+ root := &cobra.Command{
+ Use: "atunnel-bench",
+ Short: "Measure atunnel's CPU and memory on its own",
+ Long: `atunnel-bench measures what atunnel costs in CPU and memory as load rises.
+Each process of the benchmark is a subcommand; run-docker.sh shows how they
+fit together.`,
+ // Usage is for mistakes in the command line, not for a run that fails.
+ SilenceUsage: true,
+ // main prints the error itself, once.
+ SilenceErrors: true,
+ // Nobody types these commands often enough to want shell completion.
+ CompletionOptions: cobra.CompletionOptions{DisableDefaultCmd: true},
+ }
+ root.AddCommand(pkiCommand(), sinkCommand(), loadCommand())
+ root.AddCommand(linuxCommands()...)
+
+ ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
+ defer stop()
+ if err := root.ExecuteContext(ctx); err != nil {
+ fmt.Fprintln(os.Stderr, "atunnel-bench:", err)
+ os.Exit(1)
+ }
+}
diff --git a/benchmarking/atunnel/cmd/atunnel-bench/pki.go b/benchmarking/atunnel/cmd/atunnel-bench/pki.go
new file mode 100644
index 0000000000..95b33cb270
--- /dev/null
+++ b/benchmarking/atunnel/cmd/atunnel-bench/pki.go
@@ -0,0 +1,42 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package main
+
+import (
+ "time"
+
+ "github.com/spf13/cobra"
+
+ "go.ate.dev/substrate/benchmarking/atunnel/internal/pki"
+)
+
+func pkiCommand() *cobra.Command {
+ var dir string
+ var opts pki.Options
+ cmd := &cobra.Command{
+ Use: "pki",
+ Short: "Write the throwaway certificates the other subcommands share",
+ Args: cobra.NoArgs,
+ RunE: func(*cobra.Command, []string) error {
+ return pki.Generate(dir, opts)
+ },
+ }
+ cmd.Flags().StringVar(&dir, "dir", "", "directory to write the certificates to")
+ cmd.Flags().StringSliceVar(&opts.WorkerHosts, "worker-hosts", []string{"atunnel-worker"}, "names and IPs atunnel's serving certificate is valid for")
+ cmd.Flags().StringSliceVar(&opts.GatewayHosts, "gateway-hosts", []string{"atunnel-sink"}, "names and IPs the egress gateway's certificate is valid for")
+ cmd.Flags().DurationVar(&opts.Validity, "validity", 30*24*time.Hour, "certificate lifetime")
+ _ = cmd.MarkFlagRequired("dir")
+ return cmd
+}
diff --git a/benchmarking/atunnel/cmd/atunnel-bench/sandbox.go b/benchmarking/atunnel/cmd/atunnel-bench/sandbox.go
new file mode 100644
index 0000000000..5c8685642b
--- /dev/null
+++ b/benchmarking/atunnel/cmd/atunnel-bench/sandbox.go
@@ -0,0 +1,152 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//go:build linux
+
+package main
+
+import (
+ "context"
+ "fmt"
+ "net"
+ "net/http"
+ "os"
+ "runtime"
+ "syscall"
+
+ "github.com/spf13/cobra"
+ "golang.org/x/sys/unix"
+
+ "go.ate.dev/substrate/benchmarking/atunnel/internal/load"
+ "go.ate.dev/substrate/benchmarking/atunnel/internal/procstats"
+ "go.ate.dev/substrate/benchmarking/atunnel/internal/sink"
+ "go.ate.dev/substrate/benchmarking/atunnel/internal/workerhost"
+)
+
+// The worker starts these inside the actors' sandboxes; they are not meant to
+// be run by hand.
+
+func netnsExecCommand() *cobra.Command {
+ return &cobra.Command{
+ Use: "netns-exec NETNS COMMAND [ARGS]",
+ Short: "Run COMMAND in the network namespace at path NETNS",
+ Hidden: true,
+ // Everything after NETNS belongs to COMMAND.
+ DisableFlagParsing: true,
+ Args: cobra.MinimumNArgs(2),
+ RunE: func(_ *cobra.Command, args []string) error {
+ return netnsExec(args[0], args[1:])
+ },
+ }
+}
+
+// netnsExec enters a network namespace and replaces this process with the
+// command. setns applies to the calling thread only, and execve keeps just
+// that thread, so the whole new program runs in the namespace.
+func netnsExec(netns string, command []string) error {
+ runtime.LockOSThread()
+ fd, err := unix.Open(netns, unix.O_RDONLY|unix.O_CLOEXEC, 0)
+ if err != nil {
+ return fmt.Errorf("opening %s: %w", netns, err)
+ }
+ if err := unix.Setns(fd, unix.CLONE_NEWNET); err != nil {
+ return fmt.Errorf("entering %s: %w", netns, err)
+ }
+ return syscall.Exec(command[0], command, os.Environ())
+}
+
+func ingressActorCommand() *cobra.Command {
+ var port, bodyBytes int
+ var statsSocket string
+ cmd := &cobra.Command{
+ Use: "ingress-actor",
+ Short: "Stand in for an actor's program receiving ingress: an HTTP server, with its counters on a Unix socket",
+ Args: cobra.NoArgs,
+ RunE: func(cmd *cobra.Command, _ []string) error {
+ return runIngressActor(cmd.Context(), port, bodyBytes, statsSocket)
+ },
+ }
+ cmd.Flags().IntVar(&port, "port", workerhost.ActorPort, "port to serve")
+ cmd.Flags().IntVar(&bodyBytes, "body-bytes", 16, "response body size")
+ cmd.Flags().StringVar(&statsSocket, "stats-socket", "", "Unix socket to serve the counters on")
+ _ = cmd.MarkFlagRequired("stats-socket")
+ return cmd
+}
+
+func runIngressActor(ctx context.Context, port, bodyBytes int, statsSocket string) error {
+ lis, err := net.Listen("tcp", fmt.Sprintf(":%d", port))
+ if err != nil {
+ return err
+ }
+ ingressActor := sink.NewIngressActor(bodyBytes)
+ statsLis, err := listenUnix(statsSocket)
+ if err != nil {
+ return err
+ }
+ statsServer := &http.Server{Handler: procstats.Handler(func() map[string]any {
+ return map[string]any{"ingress_actor": ingressActor.Stats()}
+ })}
+ go func() { _ = statsServer.Serve(statsLis) }()
+ context.AfterFunc(ctx, func() { _ = statsServer.Close() })
+ return ingressActor.Serve(ctx, lis)
+}
+
+func egressActorCommand() *cobra.Command {
+ var socket string
+ cmd := &cobra.Command{
+ Use: "egress-actor",
+ Short: "Stand in for an actor's program making outgoing connections, running egress loads for load",
+ Args: cobra.NoArgs,
+ RunE: func(cmd *cobra.Command, _ []string) error {
+ return runEgressActor(cmd.Context(), socket)
+ },
+ }
+ cmd.Flags().StringVar(&socket, "socket", "", "Unix socket the load generator drives it on")
+ _ = cmd.MarkFlagRequired("socket")
+ return cmd
+}
+
+func runEgressActor(ctx context.Context, socket string) error {
+ // The sandbox's kernel network stack stands in for gVisor's, with
+ // Linux's defaults; widen its ports as the containers' are, or a long run
+ // of short connections exhausts them on sockets in TIME_WAIT.
+ for path, value := range map[string]string{
+ "/proc/sys/net/ipv4/ip_local_port_range": "1024 65535",
+ "/proc/sys/net/ipv4/tcp_tw_reuse": "1",
+ } {
+ if err := os.WriteFile(path, []byte(value), 0o644); err != nil {
+ return fmt.Errorf("tuning the sandbox network: %w", err)
+ }
+ }
+ lis, err := listenUnix(socket)
+ if err != nil {
+ return err
+ }
+ return load.ServeEgressActor(ctx, lis)
+}
+
+// listenUnix listens on a fresh Unix socket that the load generator, running
+// as another user in another container, may connect to.
+func listenUnix(path string) (net.Listener, error) {
+ _ = os.Remove(path)
+ lis, err := net.Listen("unix", path)
+ if err != nil {
+ return nil, err
+ }
+ if err := os.Chmod(path, 0o777); err != nil {
+ _ = lis.Close()
+ return nil, err
+ }
+ return lis, nil
+}
diff --git a/benchmarking/atunnel/cmd/atunnel-bench/sink.go b/benchmarking/atunnel/cmd/atunnel-bench/sink.go
new file mode 100644
index 0000000000..c13d201d36
--- /dev/null
+++ b/benchmarking/atunnel/cmd/atunnel-bench/sink.go
@@ -0,0 +1,81 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package main
+
+import (
+ "context"
+ "log/slog"
+ "net"
+ "net/http"
+ "path/filepath"
+
+ "github.com/spf13/cobra"
+
+ "go.ate.dev/substrate/benchmarking/atunnel/internal/pki"
+ "go.ate.dev/substrate/benchmarking/atunnel/internal/sink"
+)
+
+func sinkCommand() *cobra.Command {
+ var pkiDir, gatewayListen, adminListen string
+ var bodyBytes int
+ cmd := &cobra.Command{
+ Use: "sink",
+ Short: "Run the fake egress gateway that atunnel's egress tunnels to",
+ Args: cobra.NoArgs,
+ RunE: func(cmd *cobra.Command, _ []string) error {
+ return runSink(cmd.Context(), pkiDir, gatewayListen, adminListen, bodyBytes)
+ },
+ }
+ cmd.Flags().StringVar(&pkiDir, "pki-dir", "", "directory written by atunnel-bench pki")
+ cmd.Flags().StringVar(&gatewayListen, "gateway-listen", ":9443", "address for the fake egress gateway")
+ cmd.Flags().StringVar(&adminListen, "admin", ":9091", "address for /stats and /healthz")
+ cmd.Flags().IntVar(&bodyBytes, "body-bytes", 16, "response body size for requests inside answered tunnels")
+ _ = cmd.MarkFlagRequired("pki-dir")
+ return cmd
+}
+
+func runSink(ctx context.Context, pkiDir, gatewayListen, adminListen string, bodyBytes int) error {
+ gateway, err := sink.NewGateway(sink.GatewayConfig{
+ CredentialBundle: filepath.Join(pkiDir, pki.GatewayBundle),
+ ClientCA: filepath.Join(pkiDir, pki.CAFile),
+ BodyBytes: bodyBytes,
+ })
+ if err != nil {
+ return err
+ }
+ gatewayLis, err := net.Listen("tcp", gatewayListen)
+ if err != nil {
+ return err
+ }
+ errc := make(chan error, 2)
+ go func() { errc <- gateway.Serve(ctx, gatewayLis) }()
+
+ adminServer := &http.Server{Addr: adminListen, Handler: sink.AdminHandler(gateway)}
+ go func() { errc <- adminServer.ListenAndServe() }()
+ context.AfterFunc(ctx, func() { _ = adminServer.Close() })
+
+ slog.Info("sink serving",
+ slog.String("gateway", gatewayLis.Addr().String()),
+ slog.String("admin", adminListen))
+ select {
+ case <-ctx.Done():
+ return nil
+ case err := <-errc:
+ if ctx.Err() != nil {
+ return nil
+ }
+ return err
+ }
+}
diff --git a/benchmarking/atunnel/cmd/atunnel-bench/tunnel.go b/benchmarking/atunnel/cmd/atunnel-bench/tunnel.go
new file mode 100644
index 0000000000..4dd997446a
--- /dev/null
+++ b/benchmarking/atunnel/cmd/atunnel-bench/tunnel.go
@@ -0,0 +1,106 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//go:build linux
+
+package main
+
+import (
+ "context"
+ "fmt"
+ "log/slog"
+ "net/http"
+ "net/http/pprof"
+ "os"
+ "time"
+
+ "github.com/spf13/cobra"
+
+ "go.ate.dev/substrate/benchmarking/atunnel/internal/procstats"
+ "go.ate.dev/substrate/benchmarking/atunnel/internal/workerhost"
+ "go.ate.dev/substrate/internal/ateworker/tunnel"
+)
+
+// tunnelCommand is process B of the benchmark: atunnel started and hosting
+// actors with the worker's own code (see workerhost), and nothing else, so
+// this process's CPU and memory are atunnel's. It takes the worker's atunnel
+// flags, serves its counters on /stats and Go's profiles on /debug/pprof, and
+// needs root, for the actors' network namespaces. The worker starts it.
+func tunnelCommand() *cobra.Command {
+ var actors int
+ var gateway, adminListen, logLevel string
+ var cfg *tunnel.Config
+ cmd := &cobra.Command{
+ Use: "tunnel",
+ Short: "Run atunnel, hosting actors with the worker's own code",
+ Args: cobra.NoArgs,
+ RunE: func(cmd *cobra.Command, _ []string) error {
+ var level slog.Level
+ if err := level.UnmarshalText([]byte(logLevel)); err != nil {
+ return fmt.Errorf("--log-level: %w", err)
+ }
+ // atunnel logs a warning per failed dial, which floods the
+ // output under a deliberately failing load.
+ slog.SetDefault(slog.New(slog.NewTextHandler(os.Stderr, &slog.HandlerOptions{Level: level})))
+ return runTunnel(cmd.Context(), *cfg, actors, gateway, adminListen)
+ },
+ }
+ cfg = tunnel.RegisterFlags(cmd.Flags())
+ cmd.Flags().IntVar(&actors, "actors", 10, "actors to host")
+ cmd.Flags().StringVar(&gateway, "gateway", "atunnel-sink:9443", "egress gateway every actor tunnels to, host:port")
+ cmd.Flags().StringVar(&adminListen, "admin", ":9090", "address for /stats, /healthz, and /debug/pprof")
+ cmd.Flags().StringVar(&logLevel, "log-level", "error", "log level: debug, info, warn, or error")
+ return cmd
+}
+
+func runTunnel(ctx context.Context, cfg tunnel.Config, actors int, gateway, adminListen string) error {
+ host, err := workerhost.Start(ctx, cfg, actors, gateway)
+ if err != nil {
+ return err
+ }
+ defer func() {
+ closeCtx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
+ defer cancel()
+ _ = host.Close(closeCtx)
+ }()
+
+ mux := http.NewServeMux()
+ mux.Handle("/stats", procstats.Handler(func() map[string]any {
+ return map[string]any{"tunnel": host.Info()}
+ }))
+ mux.HandleFunc("/healthz", func(w http.ResponseWriter, _ *http.Request) { _, _ = w.Write([]byte("ok")) })
+ mux.HandleFunc("/debug/pprof/", pprof.Index)
+ mux.HandleFunc("/debug/pprof/cmdline", pprof.Cmdline)
+ mux.HandleFunc("/debug/pprof/profile", pprof.Profile)
+ mux.HandleFunc("/debug/pprof/symbol", pprof.Symbol)
+ mux.HandleFunc("/debug/pprof/trace", pprof.Trace)
+ adminServer := &http.Server{Addr: adminListen, Handler: mux}
+ errc := make(chan error, 1)
+ go func() { errc <- adminServer.ListenAndServe() }()
+ context.AfterFunc(ctx, func() { _ = adminServer.Close() })
+
+ // Not through slog, which is quiet by default: the supervisor waits for
+ // /healthz, a person reading the log for this.
+ fmt.Fprintf(os.Stderr, "tunnel serving: ingress %s, connect %s, %d actors, admin %s\n",
+ cfg.ListenAddress, cfg.ConnectListenAddress, actors, adminListen)
+ select {
+ case <-ctx.Done():
+ return nil
+ case err := <-errc:
+ if ctx.Err() != nil {
+ return nil
+ }
+ return err
+ }
+}
diff --git a/benchmarking/atunnel/cmd/atunnel-bench/worker.go b/benchmarking/atunnel/cmd/atunnel-bench/worker.go
new file mode 100644
index 0000000000..36f391f579
--- /dev/null
+++ b/benchmarking/atunnel/cmd/atunnel-bench/worker.go
@@ -0,0 +1,323 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//go:build linux
+
+package main
+
+import (
+ "context"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "log/slog"
+ "net/http"
+ "os"
+ "os/exec"
+ "path/filepath"
+ "strconv"
+ "strings"
+ "syscall"
+ "time"
+
+ "github.com/spf13/cobra"
+
+ "go.ate.dev/substrate/benchmarking/atunnel/internal/fakeatelet"
+ "go.ate.dev/substrate/benchmarking/atunnel/internal/workerhost"
+ "go.ate.dev/substrate/internal/nodepath"
+)
+
+const cgroupRoot = "/sys/fs/cgroup"
+
+// workerCommand plays the rest of a worker pod around atunnel, in a
+// privileged container: it serves the fake atelet on the node-local socket,
+// starts atunnel (the tunnel subcommand) in a cgroup of its own with the CPU
+// and memory limits, places an ingress actor in every actor's sandbox and the
+// egress actor in the first, and replaces atunnel if it dies, logging why.
+// Arguments after -- go to tunnel.
+func workerCommand() *cobra.Command {
+ var pkiDir, memory, shared, admin string
+ var cpus float64
+ var bodyBytes int
+ cmd := &cobra.Command{
+ Use: "worker [flags] -- [tunnel flags]",
+ Short: "Run the worker pod around atunnel: fake atelet, atunnel in its own cgroup, and the actors",
+ RunE: func(cmd *cobra.Command, tunnelArgs []string) error {
+ memBytes, err := parseBytes(memory)
+ if err != nil {
+ return fmt.Errorf("--tunnel-memory: %w", err)
+ }
+ self, err := os.Executable()
+ if err != nil {
+ return err
+ }
+ s := &supervisor{self: self, tunnelArgs: tunnelArgs, admin: admin, shared: shared, bodyBytes: bodyBytes, cpus: cpus, memBytes: memBytes}
+ return s.run(cmd.Context(), pkiDir)
+ },
+ }
+ f := cmd.Flags()
+ f.StringVar(&pkiDir, "pki-dir", "/pki", "directory written by atunnel-bench pki")
+ f.Float64Var(&cpus, "tunnel-cpus", 1, "CPU limit for atunnel, in cores; 0 for none")
+ f.StringVar(&memory, "tunnel-memory", "1g", "memory limit for atunnel, with no swap, e.g. 512m or 1g; empty for none")
+ f.StringVar(&shared, "shared-dir", "/shared", "directory shared with the load generator: the actors' sockets and the death log")
+ f.StringVar(&admin, "tunnel-admin", "http://127.0.0.1:9090", "atunnel's admin base URL")
+ f.IntVar(&bodyBytes, "body-bytes", 16, "response body size of the ingress actors")
+ return cmd
+}
+
+// run serves the fake atelet and keeps an atunnel running until ctx ends.
+func (s *supervisor) run(ctx context.Context, pkiDir string) error {
+ atelet, err := fakeatelet.New(pkiDir)
+ if err != nil {
+ return err
+ }
+ go func() {
+ if err := atelet.Serve(ctx, pkiDir, nodepath.WorkerSupportSocket); err != nil {
+ slog.Error("fake atelet stopped", slog.Any("err", err))
+ }
+ }()
+ if err := delegateCgroups(); err != nil {
+ return fmt.Errorf("setting up cgroups: %w", err)
+ }
+ for incarnation := 1; ctx.Err() == nil; incarnation++ {
+ if err := s.runOnce(ctx, incarnation); err != nil {
+ return err
+ }
+ }
+ return nil
+}
+
+type supervisor struct {
+ self string
+ tunnelArgs []string
+ admin string
+ shared string
+ bodyBytes int
+ cpus float64
+ memBytes int64
+}
+
+// runOnce runs one atunnel, with its sandbox processes, until it exits. It
+// returns nil when atunnel died and should be replaced.
+func (s *supervisor) runOnce(ctx context.Context, incarnation int) error {
+ dir := filepath.Join(cgroupRoot, "tunnel-"+strconv.Itoa(incarnation))
+ fd, err := s.createCgroup(dir)
+ if err != nil {
+ return err
+ }
+ cmd := exec.Command(s.self, append([]string{"tunnel"}, s.tunnelArgs...)...)
+ cmd.Stdout, cmd.Stderr = os.Stdout, os.Stderr
+ // Started inside its cgroup, so nothing it does is counted elsewhere.
+ cmd.SysProcAttr = &syscall.SysProcAttr{UseCgroupFD: true, CgroupFD: fd}
+ err = cmd.Start()
+ _ = syscall.Close(fd)
+ if err != nil {
+ return fmt.Errorf("starting atunnel: %w", err)
+ }
+ exited := make(chan error, 1)
+ go func() { exited <- cmd.Wait() }()
+ stopBench := context.AfterFunc(ctx, func() { _ = cmd.Process.Signal(syscall.SIGTERM) })
+ defer stopBench()
+
+ var helpers []*exec.Cmd
+ defer func() {
+ for _, h := range helpers {
+ _ = h.Process.Kill()
+ _ = h.Wait()
+ }
+ }()
+ info, err := s.waitForSandboxes(ctx, exited)
+ if err == nil {
+ helpers, err = s.startSandboxProcesses(info)
+ }
+ var exitErr error
+ if err != nil {
+ // atunnel may have failed to start; its exit says why.
+ _ = cmd.Process.Kill()
+ exitErr = <-exited
+ if ctx.Err() != nil {
+ return nil
+ }
+ if errors.Is(err, errExited) {
+ s.logDeath(dir, exitErr)
+ return removeCgroup(dir)
+ }
+ return err
+ }
+ exitErr = <-exited
+ if ctx.Err() != nil {
+ return nil
+ }
+ s.logDeath(dir, exitErr)
+ return removeCgroup(dir)
+}
+
+var errExited = errors.New("atunnel exited")
+
+// waitForSandboxes polls atunnel's /stats until it hosts its actors.
+func (s *supervisor) waitForSandboxes(ctx context.Context, exited <-chan error) (workerhost.Info, error) {
+ client := &http.Client{Timeout: 2 * time.Second}
+ for {
+ var body struct {
+ Tunnel workerhost.Info `json:"tunnel"`
+ }
+ resp, err := client.Get(s.admin + "/stats")
+ if err == nil {
+ err = json.NewDecoder(resp.Body).Decode(&body)
+ _ = resp.Body.Close()
+ if err == nil && len(body.Tunnel.Sandboxes) > 0 {
+ return body.Tunnel, nil
+ }
+ }
+ select {
+ case <-ctx.Done():
+ return workerhost.Info{}, ctx.Err()
+ case err := <-exited:
+ return workerhost.Info{}, fmt.Errorf("%w during startup: %v", errExited, err)
+ case <-time.After(200 * time.Millisecond):
+ }
+ }
+}
+
+// startSandboxProcesses places an ingress actor in every sandbox and the egress
+// actor in the first.
+func (s *supervisor) startSandboxProcesses(info workerhost.Info) ([]*exec.Cmd, error) {
+ var cmds []*exec.Cmd
+ start := func(netns string, args ...string) error {
+ cmd := exec.Command(s.self, append([]string{"netns-exec", netns, s.self}, args...)...)
+ cmd.Stdout, cmd.Stderr = os.Stdout, os.Stderr
+ if err := cmd.Start(); err != nil {
+ return err
+ }
+ cmds = append(cmds, cmd)
+ return nil
+ }
+ for i, sb := range info.Sandboxes {
+ socket := filepath.Join(s.shared, "ingress-actor-"+strconv.Itoa(i)+".sock")
+ if err := start(sb.NetNS, "ingress-actor", "--port", strconv.Itoa(info.ActorPort), "--body-bytes", strconv.Itoa(s.bodyBytes), "--stats-socket", socket); err != nil {
+ return cmds, err
+ }
+ }
+ if err := start(info.Sandboxes[0].NetNS, "egress-actor", "--socket", filepath.Join(s.shared, "egress-actor.sock")); err != nil {
+ return cmds, err
+ }
+ fmt.Fprintf(os.Stderr, "worker: %d sandboxes with ingress actors, egress actor in %s\n", len(info.Sandboxes), info.Sandboxes[0].Actor)
+ return cmds, nil
+}
+
+// logDeath records why atunnel stopped, with the OOM kills its cgroup
+// counted, in the shared death log.
+func (s *supervisor) logDeath(dir string, exitErr error) {
+ ooms := oomKills(dir)
+ line := fmt.Sprintf("%s atunnel exited (%v); OOM kills: %d\n", time.Now().Format(time.TimeOnly), exitErr, ooms)
+ fmt.Fprint(os.Stderr, "worker: "+line)
+ f, err := os.OpenFile(filepath.Join(s.shared, "tunnel-deaths.log"), os.O_CREATE|os.O_APPEND|os.O_WRONLY, 0o644)
+ if err != nil {
+ return
+ }
+ defer f.Close()
+ _, _ = f.WriteString(line)
+}
+
+// delegateCgroups moves every process out of the container's root cgroup, so
+// its controllers can be handed to child cgroups (cgroup v2 allows processes
+// only in leaves), and enables the CPU and memory controllers for them.
+func delegateCgroups() error {
+ leaf := filepath.Join(cgroupRoot, "supervisor")
+ if err := os.MkdirAll(leaf, 0o755); err != nil {
+ return err
+ }
+ procs, err := os.ReadFile(filepath.Join(cgroupRoot, "cgroup.procs"))
+ if err != nil {
+ return err
+ }
+ for _, pid := range strings.Fields(string(procs)) {
+ _ = os.WriteFile(filepath.Join(leaf, "cgroup.procs"), []byte(pid), 0o644)
+ }
+ return os.WriteFile(filepath.Join(cgroupRoot, "cgroup.subtree_control"), []byte("+cpu +memory"), 0o644)
+}
+
+// createCgroup makes a cgroup with atunnel's limits and returns a descriptor to
+// start a process in it.
+func (s *supervisor) createCgroup(dir string) (int, error) {
+ if err := os.Mkdir(dir, 0o755); err != nil && !os.IsExist(err) {
+ return -1, err
+ }
+ if s.cpus > 0 {
+ const period = 100000
+ if err := os.WriteFile(filepath.Join(dir, "cpu.max"), []byte(fmt.Sprintf("%d %d", int(s.cpus*period), period)), 0o644); err != nil {
+ return -1, err
+ }
+ }
+ if s.memBytes > 0 {
+ if err := os.WriteFile(filepath.Join(dir, "memory.max"), []byte(strconv.FormatInt(s.memBytes, 10)), 0o644); err != nil {
+ return -1, err
+ }
+ // No swap, as in a Kubernetes pod: over the limit is an OOM kill.
+ if err := os.WriteFile(filepath.Join(dir, "memory.swap.max"), []byte("0"), 0o644); err != nil && !os.IsNotExist(err) {
+ return -1, err
+ }
+ }
+ return syscall.Open(dir, syscall.O_RDONLY|syscall.O_DIRECTORY|syscall.O_CLOEXEC, 0)
+}
+
+func oomKills(dir string) int {
+ b, err := os.ReadFile(filepath.Join(dir, "memory.events"))
+ if err != nil {
+ return 0
+ }
+ for _, line := range strings.Split(string(b), "\n") {
+ if v, ok := strings.CutPrefix(line, "oom_kill "); ok {
+ n, _ := strconv.Atoi(v)
+ return n
+ }
+ }
+ return 0
+}
+
+// removeCgroup removes an emptied cgroup, waiting briefly for the kernel to
+// finish with the dead process.
+func removeCgroup(dir string) error {
+ var err error
+ for range 50 {
+ if err = os.Remove(dir); err == nil || os.IsNotExist(err) {
+ return nil
+ }
+ time.Sleep(100 * time.Millisecond)
+ }
+ return fmt.Errorf("removing %s: %w", dir, err)
+}
+
+func parseBytes(s string) (int64, error) {
+ if s == "" {
+ return 0, nil
+ }
+ mult := int64(1)
+ switch strings.ToLower(s[len(s)-1:]) {
+ case "k":
+ mult = 1 << 10
+ case "m":
+ mult = 1 << 20
+ case "g":
+ mult = 1 << 30
+ }
+ if mult > 1 {
+ s = s[:len(s)-1]
+ }
+ n, err := strconv.ParseInt(s, 10, 64)
+ if err != nil {
+ return 0, err
+ }
+ return n * mult, nil
+}
diff --git a/benchmarking/atunnel/cmd/atunnel-bench/worker_test.go b/benchmarking/atunnel/cmd/atunnel-bench/worker_test.go
new file mode 100644
index 0000000000..e04d736269
--- /dev/null
+++ b/benchmarking/atunnel/cmd/atunnel-bench/worker_test.go
@@ -0,0 +1,78 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//go:build linux
+
+package main
+
+import (
+ "os"
+ "path/filepath"
+ "syscall"
+ "testing"
+)
+
+func TestParseBytes(t *testing.T) {
+ for in, want := range map[string]int64{"": 0, "512": 512, "4k": 4 << 10, "200m": 200 << 20, "1G": 1 << 30} {
+ got, err := parseBytes(in)
+ if err != nil || got != want {
+ t.Errorf("parseBytes(%q) = %d, %v; want %d", in, got, err, want)
+ }
+ }
+ if _, err := parseBytes("lots"); err == nil {
+ t.Error("parseBytes accepted a non-number")
+ }
+}
+
+// TestCreateCgroup writes the limits into a plain directory standing in for a
+// cgroup.
+func TestCreateCgroup(t *testing.T) {
+ dir := filepath.Join(t.TempDir(), "tunnel-1")
+ s := &supervisor{cpus: 1.5, memBytes: 1 << 30}
+ fd, err := s.createCgroup(dir)
+ if err != nil {
+ t.Fatal(err)
+ }
+ _ = syscall.Close(fd)
+ for file, want := range map[string]string{"cpu.max": "150000 100000", "memory.max": "1073741824", "memory.swap.max": "0"} {
+ got, err := os.ReadFile(filepath.Join(dir, file))
+ if err != nil || string(got) != want {
+ t.Errorf("%s = %q, %v; want %q", file, got, err, want)
+ }
+ }
+
+ unlimited := filepath.Join(t.TempDir(), "tunnel-2")
+ fd, err = (&supervisor{}).createCgroup(unlimited)
+ if err != nil {
+ t.Fatal(err)
+ }
+ _ = syscall.Close(fd)
+ if _, err := os.Stat(filepath.Join(unlimited, "cpu.max")); !os.IsNotExist(err) {
+ t.Errorf("cpu.max written with no CPU limit: %v", err)
+ }
+}
+
+func TestOOMKills(t *testing.T) {
+ dir := t.TempDir()
+ if got := oomKills(dir); got != 0 {
+ t.Errorf("oomKills without memory.events = %d, want 0", got)
+ }
+ events := "low 0\nhigh 0\nmax 12\noom 3\noom_kill 2\noom_group_kill 0\n"
+ if err := os.WriteFile(filepath.Join(dir, "memory.events"), []byte(events), 0o644); err != nil {
+ t.Fatal(err)
+ }
+ if got := oomKills(dir); got != 2 {
+ t.Errorf("oomKills = %d, want 2", got)
+ }
+}
diff --git a/benchmarking/atunnel/internal/fakeatelet/fakeatelet.go b/benchmarking/atunnel/internal/fakeatelet/fakeatelet.go
new file mode 100644
index 0000000000..4797dc86d8
--- /dev/null
+++ b/benchmarking/atunnel/internal/fakeatelet/fakeatelet.go
@@ -0,0 +1,129 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package fakeatelet serves the part of atelet's WorkerSupport API that atunnel
+// calls: minting actor certificates for egress, and actor egress policies for
+// the DNS relay. It listens where atelet does, on the node-local socket, with
+// atelet's identity, so atunnel's own client code talks to it unchanged.
+package fakeatelet
+
+import (
+ "context"
+ "crypto"
+ "crypto/rand"
+ "crypto/tls"
+ "crypto/x509"
+ "crypto/x509/pkix"
+ "fmt"
+ "net"
+ "net/url"
+ "os"
+ "path/filepath"
+ "time"
+
+ "google.golang.org/grpc"
+ "google.golang.org/grpc/credentials"
+
+ "go.ate.dev/substrate/benchmarking/atunnel/internal/pki"
+ "go.ate.dev/substrate/internal/credbundle"
+ "go.ate.dev/substrate/internal/proto/ateletpb"
+ "go.ate.dev/substrate/internal/substratex509"
+)
+
+// actorCertificateLifetime is how long minted actor certificates last. atunnel
+// renews at 90% of it, which no benchmark run reaches.
+const actorCertificateLifetime = 24 * time.Hour
+
+// Server is the fake atelet.
+type Server struct {
+ ateletpb.UnimplementedWorkerSupportServer
+ ca *x509.Certificate
+ caKey crypto.Signer
+}
+
+// New loads the CA from pkiDir, written by pki.Generate.
+func New(pkiDir string) (*Server, error) {
+ ca, caKey, err := pki.LoadCA(pkiDir)
+ if err != nil {
+ return nil, err
+ }
+ return &Server{ca: ca, caKey: caKey}, nil
+}
+
+// Serve listens on socketPath with atelet's certificate, requiring a worker
+// certificate signed by the CA, until ctx ends.
+func (s *Server) Serve(ctx context.Context, pkiDir, socketPath string) error {
+ cert, err := credbundle.Parse(filepath.Join(pkiDir, pki.AteletBundle))
+ if err != nil {
+ return err
+ }
+ clientCAs, err := credbundle.ParsePool(filepath.Join(pkiDir, pki.CAFile))
+ if err != nil {
+ return err
+ }
+ if err := os.MkdirAll(filepath.Dir(socketPath), 0o755); err != nil {
+ return err
+ }
+ _ = os.Remove(socketPath)
+ lis, err := net.Listen("unix", socketPath)
+ if err != nil {
+ return err
+ }
+ server := grpc.NewServer(grpc.Creds(credentials.NewTLS(&tls.Config{
+ MinVersion: tls.VersionTLS13,
+ Certificates: []tls.Certificate{*cert},
+ ClientAuth: tls.RequireAndVerifyClientCert,
+ ClientCAs: clientCAs,
+ })))
+ ateletpb.RegisterWorkerSupportServer(server, s)
+ context.AfterFunc(ctx, server.Stop)
+ return server.Serve(lis)
+}
+
+// MintActorCertificate signs the CSR atunnel sends as the actor's identity.
+func (s *Server) MintActorCertificate(_ context.Context, req *ateletpb.MintActorCertificateRequest) (*ateletpb.MintActorCertificateResponse, error) {
+ csr, err := x509.ParseCertificateRequest(req.GetCertificateSigningRequest())
+ if err != nil {
+ return nil, fmt.Errorf("parsing CSR: %w", err)
+ }
+ if err := csr.CheckSignature(); err != nil {
+ return nil, fmt.Errorf("CSR signature: %w", err)
+ }
+ id := &url.URL{Scheme: "spiffe", Host: "substrate-actor.local", Path: "/worker/actor/" + req.GetActorAtespace() + "/" + req.GetActorName()}
+ now := time.Now()
+ template := &x509.Certificate{
+ SerialNumber: pki.Serial(),
+ Subject: pkix.Name{CommonName: req.GetActorName()},
+ NotBefore: now.Add(-time.Minute),
+ NotAfter: now.Add(actorCertificateLifetime),
+ KeyUsage: x509.KeyUsageDigitalSignature,
+ ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth},
+ URIs: []*url.URL{id},
+ }
+ actor := &substratex509.ActorIdentity{Atespace: req.GetActorAtespace(), ActorName: req.GetActorName(), ActorUid: req.GetActorUid()}
+ if err := substratex509.AddActorIdentityToCertificate(actor, template); err != nil {
+ return nil, err
+ }
+ der, err := x509.CreateCertificate(rand.Reader, template, s.ca, csr.PublicKey, s.caKey)
+ if err != nil {
+ return nil, err
+ }
+ return &ateletpb.MintActorCertificateResponse{ActorCertificates: [][]byte{der}}, nil
+}
+
+// GetActorEgressPolicy reports no policy. Only the DNS relay asks, and the
+// benchmark's egress goes to addresses, not names.
+func (s *Server) GetActorEgressPolicy(context.Context, *ateletpb.GetActorEgressPolicyRequest) (*ateletpb.GetActorEgressPolicyResponse, error) {
+ return &ateletpb.GetActorEgressPolicyResponse{}, nil
+}
diff --git a/benchmarking/atunnel/internal/fakeatelet/fakeatelet_test.go b/benchmarking/atunnel/internal/fakeatelet/fakeatelet_test.go
new file mode 100644
index 0000000000..342b197d0b
--- /dev/null
+++ b/benchmarking/atunnel/internal/fakeatelet/fakeatelet_test.go
@@ -0,0 +1,90 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package fakeatelet
+
+import (
+ "context"
+ "crypto/tls"
+ "crypto/x509"
+ "path/filepath"
+ "testing"
+ "time"
+
+ "go.ate.dev/substrate/benchmarking/atunnel/internal/pki"
+ "go.ate.dev/substrate/internal/atunnel/egress"
+ "go.ate.dev/substrate/internal/credbundle"
+ "go.ate.dev/substrate/internal/installdefaults"
+ "go.ate.dev/substrate/internal/substratex509"
+)
+
+// TestMintThroughAtunnelsClient mints an actor certificate the way atunnel
+// does, with its own broker client, over the fake atelet's socket.
+func TestMintThroughAtunnelsClient(t *testing.T) {
+ ctx, cancel := context.WithCancel(context.Background())
+ t.Cleanup(cancel)
+ dir := t.TempDir()
+ if err := pki.Generate(dir, pki.Options{WorkerHosts: []string{"atunnel-worker"}, GatewayHosts: []string{"atunnel-sink"}}); err != nil {
+ t.Fatal(err)
+ }
+ server, err := New(dir)
+ if err != nil {
+ t.Fatal(err)
+ }
+ // Unix socket paths are limited to about 100 bytes; TempDir can be long.
+ socket := filepath.Join("/tmp", "fakeatelet-"+time.Now().Format("150405.000000")+".sock")
+ errc := make(chan error, 1)
+ go func() { errc <- server.Serve(ctx, dir, socket) }()
+
+ source, err := egress.NewBrokerCertificateSource(egress.BrokerConfig{
+ SocketPath: socket,
+ CredentialBundlePath: filepath.Join(dir, pki.WorkerBundle),
+ TrustBundlePath: filepath.Join(dir, pki.CAFile),
+ ActorAtespace: "bench",
+ ActorName: "actor-0",
+ ActorUID: "00000000-0000-4000-8000-000000000000",
+ AteletSPIFFEID: installdefaults.AteletSPIFFEID(installdefaults.SystemNamespace),
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+ var expiry time.Time
+ for range 50 {
+ if expiry, err = source.MintWorkerCertificate(ctx); err == nil {
+ break
+ }
+ time.Sleep(20 * time.Millisecond)
+ }
+ if err != nil {
+ t.Fatalf("minting: %v", err)
+ }
+ if time.Until(expiry) < time.Hour {
+ t.Errorf("certificate expires at %v, too soon", expiry)
+ }
+ cert, err := source.GetClientCertificate(&tls.CertificateRequestInfo{})
+ if err != nil {
+ t.Fatal(err)
+ }
+ roots, err := credbundle.ParsePool(filepath.Join(dir, pki.CAFile))
+ if err != nil {
+ t.Fatal(err)
+ }
+ if _, err := cert.Leaf.Verify(x509.VerifyOptions{Roots: roots, KeyUsages: []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth}}); err != nil {
+ t.Errorf("minted certificate does not verify: %v", err)
+ }
+ id, err := substratex509.ActorIdentityFromCertificate(cert.Leaf)
+ if err != nil || id == nil || id.ActorName != "actor-0" {
+ t.Errorf("minted certificate actor identity = %+v, %v", id, err)
+ }
+}
diff --git a/benchmarking/atunnel/internal/load/egress_actor.go b/benchmarking/atunnel/internal/load/egress_actor.go
new file mode 100644
index 0000000000..7b75d9445a
--- /dev/null
+++ b/benchmarking/atunnel/internal/load/egress_actor.go
@@ -0,0 +1,161 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package load
+
+import (
+ "bufio"
+ "bytes"
+ "context"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "net"
+ "net/http"
+ "time"
+
+ "go.ate.dev/substrate/benchmarking/atunnel/internal/procstats"
+ "go.ate.dev/substrate/benchmarking/atunnel/internal/report"
+)
+
+// Egress has to start inside an actor's sandbox, where the sandbox's nftables
+// redirect hands every outbound connection to atunnel. The egress actor runs
+// there and executes egress modes on the load generator's behalf; the two
+// talk over a Unix socket, which reaches across network namespaces.
+
+// The destinations the egress actor dials. Nothing answers them: atunnel takes
+// every connection, and the fake gateway answers or discards by port.
+const (
+ egressAnswerDestination = "192.0.2.10:80"
+ egressDiscardDestination = "192.0.2.10:9"
+)
+
+// egressActorHolding is the line the egress actor writes when EgressIdle has
+// opened its connections and holds them, so the generator can read atunnel
+// then.
+const egressActorHolding = "holding"
+
+// egressActorRequest is options on the wire.
+type egressActorRequest struct {
+ Mode Mode `json:"mode"`
+ Rate float64 `json:"rate"`
+ Duration time.Duration `json:"duration"`
+ Warmup time.Duration `json:"warmup"`
+ Workers int `json:"workers"`
+ Connections int `json:"connections"`
+ NewConnectionPerRequest bool `json:"new_connection_per_request"`
+ PayloadBytes int `json:"payload_bytes"`
+}
+
+type egressActorResult struct {
+ Load report.Load `json:"load"`
+ Error string `json:"error,omitempty"`
+}
+
+// isEgress reports whether a mode's traffic starts inside the sandbox.
+func isEgress(m Mode) bool {
+ return m == EgressHTTP || m == EgressIdle || m == EgressStream
+}
+
+// ServeEgressActor serves egress runs on lis until ctx ends.
+func ServeEgressActor(ctx context.Context, lis net.Listener) error {
+ return serveEgressActor(ctx, lis, Targets{Egress: egressAnswerDestination, EgressDiscard: egressDiscardDestination})
+}
+
+func serveEgressActor(ctx context.Context, lis net.Listener, targets Targets) error {
+ mux := http.NewServeMux()
+ // The egress actor generates the egress load, so its CPU is the generator's.
+ mux.Handle("GET /stats", procstats.Handler(nil))
+ mux.HandleFunc("POST /run", func(w http.ResponseWriter, r *http.Request) {
+ var req egressActorRequest
+ if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
+ http.Error(w, err.Error(), http.StatusBadRequest)
+ return
+ }
+ rc := http.NewResponseController(w)
+ opts := options{
+ mode: req.Mode, rate: req.Rate, duration: req.Duration, warmup: req.Warmup,
+ workers: req.Workers, connections: req.Connections,
+ newConnectionPerRequest: req.NewConnectionPerRequest, payloadBytes: req.PayloadBytes,
+ whileHolding: func(context.Context) {
+ _, _ = fmt.Fprintln(w, egressActorHolding)
+ _ = rc.Flush()
+ },
+ }
+ var res egressActorResult
+ l, err := execute(r.Context(), targets, opts)
+ if err != nil {
+ res.Error = err.Error()
+ }
+ res.Load = l
+ _ = json.NewEncoder(w).Encode(res)
+ })
+ server := &http.Server{Handler: mux}
+ context.AfterFunc(ctx, func() { _ = server.Close() })
+ if err := server.Serve(lis); !errors.Is(err, http.ErrServerClosed) {
+ return err
+ }
+ return nil
+}
+
+// executeOnEgressActor runs opts on the egress actor at socket.
+func executeOnEgressActor(ctx context.Context, socket string, opts options) (report.Load, error) {
+ body, err := json.Marshal(egressActorRequest{
+ Mode: opts.mode, Rate: opts.rate, Duration: opts.duration, Warmup: opts.warmup,
+ Workers: opts.workers, Connections: opts.connections,
+ NewConnectionPerRequest: opts.newConnectionPerRequest, PayloadBytes: opts.payloadBytes,
+ })
+ if err != nil {
+ return report.Load{}, err
+ }
+ client := &http.Client{Transport: &http.Transport{
+ DialContext: func(ctx context.Context, _, _ string) (net.Conn, error) {
+ return (&net.Dialer{}).DialContext(ctx, "unix", socket)
+ },
+ }}
+ req, err := http.NewRequestWithContext(ctx, http.MethodPost, "http://egress-actor/run", bytes.NewReader(body))
+ if err != nil {
+ return report.Load{}, err
+ }
+ resp, err := client.Do(req)
+ if err != nil {
+ return report.Load{}, fmt.Errorf("egress actor: %w", err)
+ }
+ defer resp.Body.Close()
+ if resp.StatusCode != http.StatusOK {
+ return report.Load{}, fmt.Errorf("egress actor: HTTP %d", resp.StatusCode)
+ }
+ lines := bufio.NewScanner(resp.Body)
+ lines.Buffer(make([]byte, 64<<10), 1<<20)
+ for lines.Scan() {
+ if lines.Text() == egressActorHolding {
+ if opts.whileHolding != nil {
+ opts.whileHolding(ctx)
+ }
+ continue
+ }
+ var res egressActorResult
+ if err := json.Unmarshal(lines.Bytes(), &res); err != nil {
+ return report.Load{}, fmt.Errorf("egress actor: %w", err)
+ }
+ if res.Error != "" {
+ return report.Load{}, fmt.Errorf("egress actor: %s", res.Error)
+ }
+ return res.Load, nil
+ }
+ if err := lines.Err(); err != nil {
+ return report.Load{}, fmt.Errorf("egress actor: %w", err)
+ }
+ return report.Load{}, fmt.Errorf("egress actor: no result")
+}
diff --git a/benchmarking/atunnel/internal/load/ladder.go b/benchmarking/atunnel/internal/load/ladder.go
new file mode 100644
index 0000000000..9e492353a3
--- /dev/null
+++ b/benchmarking/atunnel/internal/load/ladder.go
@@ -0,0 +1,445 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package load
+
+import (
+ "context"
+ "fmt"
+ "net"
+ "path/filepath"
+ "strconv"
+ "strings"
+ "time"
+
+ "go.ate.dev/substrate/benchmarking/atunnel/internal/procstats"
+ "go.ate.dev/substrate/benchmarking/atunnel/internal/report"
+)
+
+// sampleInterval is how often atunnel is polled for peaks during a rung.
+const sampleInterval = 500 * time.Millisecond
+
+// Runner runs scenarios against a running atunnel and sink.
+type Runner struct {
+ Targets Targets
+ // TunnelAdmin and SinkAdmin are the processes' admin base URLs.
+ TunnelAdmin string
+ SinkAdmin string
+ Step time.Duration
+ Warmup time.Duration
+ // Workers caps units in flight.
+ Workers int
+ // ProfileDir, when set, receives a CPU and a heap profile of atunnel per
+ // rung.
+ ProfileDir string
+ // SharedDir is shared with the worker container. It holds the Unix
+ // sockets of the egress actor, which runs egress scenarios inside an
+ // actor's sandbox, and of the ingress actors, which serve their counters.
+ SharedDir string
+
+ tunnel, sink *admin
+}
+
+// Prepare waits for both processes, learns the hosted actors and the actor
+// port from atunnel, and sends a little traffic down every path so that a
+// misconfiguration fails here rather than as a rung full of errors.
+func (r *Runner) Prepare(ctx context.Context) error {
+ r.tunnel, r.sink = newAdmin(r.TunnelAdmin), newAdmin(r.SinkAdmin)
+ readyCtx, cancel := context.WithTimeout(ctx, time.Minute)
+ defer cancel()
+ for _, a := range []*admin{r.tunnel, r.sink} {
+ if err := a.waitReady(readyCtx); err != nil {
+ return err
+ }
+ }
+ st, err := r.tunnel.stats(ctx)
+ if err != nil {
+ return err
+ }
+ if st.Tunnel == nil || len(st.Tunnel.Sandboxes) == 0 {
+ return fmt.Errorf("%s/stats does not describe a tunnel with hosted actors; is it atunnel-bench tunnel?", r.TunnelAdmin)
+ }
+ r.Targets.Actors = nil
+ for _, sb := range st.Tunnel.Sandboxes {
+ r.Targets.Actors = append(r.Targets.Actors, sb.Actor)
+ }
+ r.Targets.ActorPort = st.Tunnel.ActorPort
+ // The worker places the egress actor and the ingress actors in the sandboxes
+ // once atunnel hosts them, so they come up a moment after it.
+ if err := r.waitForSandboxSockets(readyCtx, len(st.Tunnel.Sandboxes)); err != nil {
+ return err
+ }
+
+ // The stream check goes first so its bytes have drained through the
+ // gateway before the first scenario reads the sink's counters.
+ for _, check := range []options{
+ {mode: EgressStream, connections: 1, payloadBytes: 1024},
+ {mode: IngressH1, rate: 20},
+ {mode: IngressH2, rate: 20},
+ {mode: Connect, rate: 20},
+ {mode: EgressHTTP, rate: 20},
+ } {
+ check.duration, check.workers = 300*time.Millisecond, 4
+ l, err := r.execute(ctx, check)
+ if err != nil {
+ return err
+ }
+ if l.Errors > 0 || l.Completed == 0 {
+ return fmt.Errorf("check of %s failed: %d completed, %d errors: %s",
+ check.mode, l.Completed, l.Errors, strings.Join(l.ErrorSamples, "; "))
+ }
+ }
+ return nil
+}
+
+// waitForSandboxSockets waits until the egress actor and every ingress actor
+// answer on their sockets in the shared directory.
+func (r *Runner) waitForSandboxSockets(ctx context.Context, ingressActors int) error {
+ if r.SharedDir == "" {
+ return nil
+ }
+ sockets := []string{filepath.Join(r.SharedDir, "egress-actor.sock")}
+ for i := range ingressActors {
+ sockets = append(sockets, filepath.Join(r.SharedDir, "ingress-actor-"+strconv.Itoa(i)+".sock"))
+ }
+ for _, socket := range sockets {
+ for {
+ conn, err := (&net.Dialer{}).DialContext(ctx, "unix", socket)
+ if err == nil {
+ _ = conn.Close()
+ break
+ }
+ select {
+ case <-ctx.Done():
+ return fmt.Errorf("waiting for %s: %w", socket, err)
+ case <-time.After(200 * time.Millisecond):
+ }
+ }
+ }
+ return nil
+}
+
+// Run runs s at each rung and reports it. A rung where atunnel stops
+// answering ends the scenario; Run then waits for atunnel to come back, as a
+// replaced atunnel does, and fails, with the report so far, if it does
+// not.
+func (r *Runner) Run(ctx context.Context, s Scenario, rungs []float64) (*report.Report, error) {
+ rep := &report.Report{Scenario: s.Name, Description: s.Description, Unit: s.Unit, Step: r.Step, Warmup: r.Warmup}
+ if s.holdsConnections {
+ rep.Warmup = 0
+ }
+ if s.streams > 0 {
+ // Bandwidth has no rate to offer; one rung is the measurement.
+ rungs = rungs[:1]
+ }
+ before, err := r.sinkStats(ctx)
+ if err != nil {
+ return nil, err
+ }
+ for _, target := range rungs {
+ rung, err := r.rung(ctx, s, target)
+ if err != nil {
+ return nil, fmt.Errorf("%s at %.0f: %w", s.Name, target, err)
+ }
+ rep.Rungs = append(rep.Rungs, rung)
+ if rung.Verdict == report.VerdictDown {
+ waitCtx, cancel := context.WithTimeout(ctx, 2*time.Minute)
+ err := r.tunnel.waitReady(waitCtx)
+ if err == nil {
+ // The replacement's sandboxes get their own egress actor and
+ // ingress actors a moment after it is ready.
+ err = r.waitForSandboxSockets(waitCtx, len(r.Targets.Actors))
+ }
+ cancel()
+ if err != nil {
+ return rep, fmt.Errorf("atunnel did not come back after %s at %.0f: %w", s.Name, target, err)
+ }
+ break
+ }
+ }
+ after, err := r.settle(ctx)
+ if err != nil {
+ return nil, err
+ }
+ rep.SinkCounters = sinkDelta(before, after)
+ return rep, nil
+}
+
+// settle waits, for up to settleTimeout, until every tunnel the sink saw has
+// closed, so bytes still draining through atunnel are counted in this
+// scenario rather than the next, and returns the sink's counters.
+func (r *Runner) settle(ctx context.Context) (Stats, error) {
+ const settleTimeout = 10 * time.Second
+ deadline := time.Now().Add(settleTimeout)
+ for {
+ st, err := r.sinkStats(ctx)
+ if err != nil || st.Gateway == nil || st.Gateway.TunnelsOpen == 0 || time.Now().After(deadline) {
+ return st, err
+ }
+ select {
+ case <-ctx.Done():
+ return st, ctx.Err()
+ case <-time.After(100 * time.Millisecond):
+ }
+ }
+}
+
+func (r *Runner) rung(ctx context.Context, s Scenario, target float64) (report.Rung, error) {
+ opts := options{
+ mode: s.mode,
+ rate: target,
+ duration: r.Step,
+ warmup: r.Warmup,
+ workers: r.Workers,
+ newConnectionPerRequest: s.newConnectionPerRequest,
+ }
+ var live Stats
+ var liveErr error
+ switch {
+ case s.holdsConnections:
+ // Open at a steady rate, stretching the rung if the count needs
+ // it, then hold for at least half a step. Every open is measured.
+ opts.connections = int(target)
+ opts.rate = idleOpenRate
+ opts.warmup = 0
+ opts.duration = max(r.Step, time.Duration(target/idleOpenRate*float64(time.Second))+r.Step/2)
+ opts.whileHolding = func(ctx context.Context) { live, liveErr = r.tunnel.liveStats(ctx) }
+ case s.streams > 0:
+ opts.connections = s.streams
+ opts.payloadBytes = 64 << 10
+ opts.rate = 0
+ }
+
+ type result struct {
+ load report.Load
+ err error
+ }
+ // The load stops early if atunnel dies: against a replacement it would
+ // only measure, or kill, that one.
+ loadCtx, stopLoad := context.WithCancel(ctx)
+ defer stopLoad()
+ done := make(chan result, 1)
+ go func() {
+ l, err := r.execute(loadCtx, opts)
+ done <- result{l, err}
+ }()
+
+ // The window opens when warmup ends.
+ select {
+ case <-ctx.Done():
+ return report.Rung{}, ctx.Err()
+ case <-time.After(opts.warmup):
+ }
+ tunnelWatch := startWatch(ctx, r.processStats(r.tunnel), sampleInterval, stopLoad)
+ sinkWatch := startWatch(ctx, r.sinkProcessStats, sampleInterval, nil)
+ generator := func(context.Context) (procstats.Snapshot, error) { return procstats.Read(), nil }
+ if isEgress(opts.mode) {
+ // Egress is generated by the egress actor in the sandbox.
+ generator = r.processStats(unixAdmin(filepath.Join(r.SharedDir, "egress-actor.sock")))
+ }
+ selfWatch := startWatch(ctx, generator, sampleInterval, nil)
+
+ var profiles []string
+ profileDone := make(chan error, 1)
+ if r.ProfileDir != "" {
+ seconds := max(1, int((opts.duration - opts.warmup).Seconds()))
+ path := filepath.Join(r.ProfileDir, profileName(s, target, "cpu"))
+ profiles = append(profiles, path)
+ go func() {
+ profileDone <- r.tunnel.fetch(ctx, "/debug/pprof/profile?seconds="+strconv.Itoa(seconds), path)
+ }()
+ } else {
+ profileDone <- nil
+ }
+
+ res := <-done
+ tunnel, tunnelErr := tunnelWatch.finish(ctx, res.load.Completed)
+ if res.err != nil {
+ if ctx.Err() != nil || tunnelErr == nil {
+ return report.Rung{}, res.err
+ }
+ // The load failed because atunnel died: stopped by the watch, or cut
+ // off when the worker replaced the egress actor with atunnel.
+ res.load = report.Load{Offered: opts.rate}
+ }
+ rung := report.Rung{Load: res.load}
+ if s.holdsConnections {
+ // The rung offers connections, and every open is a unit.
+ rung.Load.Offered = target
+ rung.Load.Achieved = float64(res.load.Completed)
+ }
+ sinkRes, sinkErr := sinkWatch.finish(ctx, res.load.Completed)
+ generatorRes, generatorErr := selfWatch.finish(ctx, res.load.Completed)
+ // When atunnel dies, the worker replaces the egress actor and ingress
+ // actors with it, so their readings fail too; only atunnel's failure is
+ // the result.
+ if tunnelErr == nil {
+ if sinkErr != nil {
+ return report.Rung{}, fmt.Errorf("reading sink stats: %w", sinkErr)
+ }
+ if generatorErr != nil {
+ return report.Rung{}, fmt.Errorf("reading generator stats: %w", generatorErr)
+ }
+ }
+ rung.Sink, rung.Generator = sinkRes, generatorRes
+ profileErr := <-profileDone
+ if s.holdsConnections && tunnelErr == nil {
+ tunnelErr = liveErr
+ }
+ if tunnelErr == nil {
+ tunnelErr = profileErr
+ }
+ if tunnelErr != nil {
+ // atunnel died or hung during the rung, as when it is killed for
+ // going over its memory limit. That is a result. The ingress actors and
+ // egress actor were replaced with it, so their readings are dropped too.
+ rung.Sink, rung.Generator = report.Resources{}, report.Resources{}
+ rung.Verdict = report.VerdictDown
+ rung.Reason = "atunnel stopped answering: " + tunnelErr.Error()
+ return rung, nil
+ }
+ rung.Tunnel = tunnel
+ if s.holdsConnections {
+ rung.Tunnel.HeapLiveBytes = live.Process.HeapInUseBytes
+ rung.Tunnel.GoroutinesLive = live.Process.Goroutines
+ }
+ if r.ProfileDir != "" {
+ path := filepath.Join(r.ProfileDir, profileName(s, target, "heap"))
+ if err := r.tunnel.fetch(ctx, "/debug/pprof/heap", path); err != nil {
+ return report.Rung{}, fmt.Errorf("fetching heap profile: %w", err)
+ }
+ profiles = append(profiles, path)
+ }
+ rung.Profiles = profiles
+ rung.Verdict, rung.Reason = verdict(s, rung)
+ return rung, nil
+}
+
+// execute runs opts here, or on the egress actor for egress modes.
+func (r *Runner) execute(ctx context.Context, opts options) (report.Load, error) {
+ if isEgress(opts.mode) {
+ if r.SharedDir == "" {
+ return report.Load{}, fmt.Errorf("%s needs the egress actor in the shared directory", opts.mode)
+ }
+ return executeOnEgressActor(ctx, filepath.Join(r.SharedDir, "egress-actor.sock"), opts)
+ }
+ return execute(ctx, r.Targets, opts)
+}
+
+func (r *Runner) processStats(a *admin) func(context.Context) (procstats.Snapshot, error) {
+ return func(ctx context.Context) (procstats.Snapshot, error) {
+ st, err := a.stats(ctx)
+ return st.Process, err
+ }
+}
+
+// sinkProcessStats is the CPU and memory of everything at the far end of
+// atunnel: the fake gateway and the ingress actors, summed.
+func (r *Runner) sinkProcessStats(ctx context.Context) (procstats.Snapshot, error) {
+ st, err := r.sink.stats(ctx)
+ if err != nil || r.SharedDir == "" {
+ return st.Process, err
+ }
+ sum := st.Process
+ sockets, err := filepath.Glob(filepath.Join(r.SharedDir, "ingress-actor-*.sock"))
+ if err != nil {
+ return sum, err
+ }
+ for _, socket := range sockets {
+ w, err := unixAdmin(socket).stats(ctx)
+ if err != nil {
+ continue
+ }
+ sum.CPUSeconds += w.Process.CPUSeconds
+ sum.RSSBytes += w.Process.RSSBytes
+ sum.HeapInUseBytes += w.Process.HeapInUseBytes
+ sum.Goroutines += w.Process.Goroutines
+ }
+ return sum, nil
+}
+
+func profileName(s Scenario, target float64, kind string) string {
+ return fmt.Sprintf("%s-%.0f.%s.pb.gz", s.Name, target, kind)
+}
+
+// Shares of a process's CPU past which it counts as saturated: atunnel, and
+// the generator or sink, which then set the pace instead of atunnel.
+const (
+ tunnelSaturated = 0.95
+ rigSaturated = 0.85
+)
+
+// verdict classifies a rung.
+func verdict(s Scenario, g report.Rung) (string, string) {
+ l := g.Load
+ if l.Errors > 0 {
+ return report.VerdictErrors, fmt.Sprintf("%d errors: %s", l.Errors, strings.Join(l.ErrorSamples, "; "))
+ }
+ if limit := cpuLimit(g.Tunnel); g.Tunnel.Cores >= tunnelSaturated*limit {
+ return report.VerdictSaturated, fmt.Sprintf("atunnel at %.2f of %g cores", g.Tunnel.Cores, limit)
+ }
+ for _, p := range []struct {
+ name string
+ res report.Resources
+ }{{"generator", g.Generator}, {"sink", g.Sink}} {
+ if limit := cpuLimit(p.res); p.res.Cores > rigSaturated*limit {
+ return report.VerdictRigLimited, fmt.Sprintf("%s at %.1f of %g cores", p.name, p.res.Cores, limit)
+ }
+ }
+ // With the generator and sink idle enough, a backed-up queue means the
+ // generator's workers were all waiting on atunnel.
+ if l.Skipped > 0 {
+ return report.VerdictShortfall, fmt.Sprintf("%d units found every worker busy", l.Skipped)
+ }
+ if l.SendLag.P99 > 10 {
+ return report.VerdictShortfall, fmt.Sprintf("units waited for a worker, p99 %.1fms", l.SendLag.P99)
+ }
+ if !s.holdsConnections && s.streams == 0 && l.Achieved < 0.97*l.Offered {
+ return report.VerdictShortfall, fmt.Sprintf("achieved %.0f of %.0f", l.Achieved, l.Offered)
+ }
+ return report.VerdictOK, ""
+}
+
+// cpuLimit is the cores a process may use: its cgroup's limit when it has
+// one, else its GOMAXPROCS.
+func cpuLimit(r report.Resources) float64 {
+ if r.CgroupCPULimitCore > 0 {
+ return r.CgroupCPULimitCore
+ }
+ return float64(max(1, r.GOMAXPROCS))
+}
+
+// sinkDelta is how much the sink's counters moved over a scenario. A counter
+// that went down was reset, by a replaced ingress actor, and is left out.
+func sinkDelta(before, after Stats) map[string]any {
+ out := map[string]any{}
+ delta := func(name string, b, a uint64) {
+ if a >= b {
+ out[name] = a - b
+ }
+ }
+ if before.IngressActor != nil && after.IngressActor != nil {
+ delta("ingress_actor_requests", before.IngressActor.Requests, after.IngressActor.Requests)
+ delta("ingress_actor_connections", before.IngressActor.ConnectionsSeen, after.IngressActor.ConnectionsSeen)
+ }
+ if before.Gateway != nil && after.Gateway != nil {
+ delta("gateway_connections", before.Gateway.Connections, after.Gateway.Connections)
+ delta("gateway_tunnels", before.Gateway.Tunnels, after.Gateway.Tunnels)
+ delta("gateway_tunnels_h2", before.Gateway.TunnelsH2, after.Gateway.TunnelsH2)
+ delta("gateway_requests", before.Gateway.Requests, after.Gateway.Requests)
+ delta("gateway_bytes_in", before.Gateway.BytesIn, after.Gateway.BytesIn)
+ delta("gateway_rejected", before.Gateway.Rejected, after.Gateway.Rejected)
+ }
+ return out
+}
diff --git a/benchmarking/atunnel/internal/load/load_test.go b/benchmarking/atunnel/internal/load/load_test.go
new file mode 100644
index 0000000000..9a5ecdf337
--- /dev/null
+++ b/benchmarking/atunnel/internal/load/load_test.go
@@ -0,0 +1,134 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package load
+
+import (
+ "context"
+ "net"
+ "net/http"
+ "path/filepath"
+ "sync/atomic"
+ "testing"
+ "time"
+
+ "go.ate.dev/substrate/benchmarking/atunnel/internal/report"
+)
+
+func TestPacerOffersTheRateWhateverTheWork(t *testing.T) {
+ p := &pacer{rate: 200, duration: time.Second, warmup: 200 * time.Millisecond, workers: 4, maxPending: 1000}
+ var measured atomic.Int64
+ p.run(context.Background(), func(_ context.Context, u unit) {
+ if u.measured {
+ measured.Add(1)
+ }
+ // Slower than the interval for one worker; four keep up.
+ time.Sleep(10 * time.Millisecond)
+ })
+ if got := p.scheduled.Load(); got != 200 {
+ t.Errorf("scheduled %d units, want 200", got)
+ }
+ if got := measured.Load(); got != 160 {
+ t.Errorf("measured %d units, want 160 after a 20%% warmup", got)
+ }
+ if got := p.skipped.Load(); got != 0 {
+ t.Errorf("skipped %d units with room to queue", got)
+ }
+}
+
+func TestPacerSkipsWhatCannotQueue(t *testing.T) {
+ p := &pacer{rate: 1000, duration: 200 * time.Millisecond, workers: 1, maxPending: 1}
+ p.run(context.Background(), func(context.Context, unit) { time.Sleep(50 * time.Millisecond) })
+ if p.skipped.Load() == 0 {
+ t.Error("no units skipped though one worker cannot keep up and the queue holds one")
+ }
+}
+
+func TestPercentiles(t *testing.T) {
+ var s samples
+ for i := 1; i <= 100; i++ {
+ s.record(time.Duration(i) * time.Millisecond)
+ }
+ got := s.percentiles()
+ want := report.Percentiles{P50: 51, P90: 91, P99: 100, Max: 100, Mean: 50.5}
+ if got != want {
+ t.Errorf("percentiles = %+v, want %+v", got, want)
+ }
+ if empty := s.percentiles(); empty != (report.Percentiles{}) {
+ t.Errorf("percentiles after draining = %+v, want zero", empty)
+ }
+}
+
+func TestSelect(t *testing.T) {
+ all, err := Select("all")
+ if err != nil || len(all) != 7 {
+ t.Fatalf("Select(all) = %d scenarios, %v; want 7", len(all), err)
+ }
+ some, err := Select("ingressh1, EgressTCP")
+ if err != nil || len(some) != 2 || some[0].Name != "IngressH1" || some[1].Name != "EgressTCP" {
+ t.Fatalf("Select = %v, %v", some, err)
+ }
+ if _, err := Select("Nope"); err == nil {
+ t.Fatal("Select accepted an unknown scenario")
+ }
+}
+
+// TestEgressActor runs egress modes through the egress actor, against a plain
+// HTTP server standing in for the far end of the sandbox's redirect.
+func TestEgressActor(t *testing.T) {
+ ctx, cancel := context.WithCancel(context.Background())
+ t.Cleanup(cancel)
+ target := listen(t)
+ go func() {
+ _ = http.Serve(target, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { _, _ = w.Write([]byte("ok")) }))
+ }()
+
+ socket := filepath.Join(t.TempDir(), "egress-actor.sock")
+ lis, err := net.Listen("unix", socket)
+ if err != nil {
+ t.Fatal(err)
+ }
+ egressTargets := Targets{Egress: target.Addr().String(), EgressDiscard: target.Addr().String()}
+ go func() { _ = serveEgressActor(ctx, lis, egressTargets) }()
+
+ l, err := executeOnEgressActor(ctx, socket, options{mode: EgressHTTP, rate: 100, duration: 300 * time.Millisecond, workers: 4})
+ if err != nil {
+ t.Fatal(err)
+ }
+ if l.Completed == 0 || l.Errors > 0 {
+ t.Fatalf("EgressHTTP through the egress actor: completed %d, errors %d: %v", l.Completed, l.Errors, l.ErrorSamples)
+ }
+
+ held := false
+ l, err = executeOnEgressActor(ctx, socket, options{
+ mode: EgressIdle, rate: 100, duration: 500 * time.Millisecond, workers: 4, connections: 5,
+ whileHolding: func(context.Context) { held = true },
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+ if !held || l.Completed != 5 {
+ t.Fatalf("EgressIdle through the egress actor: held %v, completed %d, want true and 5", held, l.Completed)
+ }
+}
+
+func listen(t *testing.T) net.Listener {
+ t.Helper()
+ lis, err := net.Listen("tcp", "127.0.0.1:0")
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { _ = lis.Close() })
+ return lis
+}
diff --git a/benchmarking/atunnel/internal/load/modes.go b/benchmarking/atunnel/internal/load/modes.go
new file mode 100644
index 0000000000..3385c0bb16
--- /dev/null
+++ b/benchmarking/atunnel/internal/load/modes.go
@@ -0,0 +1,420 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package load
+
+import (
+ "bufio"
+ "context"
+ "crypto/tls"
+ "fmt"
+ "io"
+ "net"
+ "net/http"
+ "strconv"
+ "sync"
+ "sync/atomic"
+ "time"
+
+ "go.ate.dev/substrate/benchmarking/atunnel/internal/report"
+ "go.ate.dev/substrate/internal/atenet"
+ "go.ate.dev/substrate/internal/credbundle"
+)
+
+// Mode is a kind of traffic the generator sends.
+type Mode string
+
+const (
+ // IngressH1 sends HTTPS requests over HTTP/1.1 to atunnel's ingress
+ // listener, as the router does.
+ IngressH1 Mode = "ingress-h1"
+ // IngressH2 is IngressH1 over HTTP/2.
+ IngressH2 Mode = "ingress-h2"
+ // Connect opens a TLS connection and a CONNECT tunnel to atunnel's CONNECT
+ // listener per unit and sends one request through it.
+ Connect Mode = "connect"
+ // EgressHTTP sends plain HTTP to atunnel's egress listener, as an actor's
+ // redirected connections arrive there.
+ EgressHTTP Mode = "egress-http"
+ // EgressIdle opens connections to the egress listener, completes one
+ // request on each, and holds them open.
+ EgressIdle Mode = "egress-idle"
+ // EgressStream writes bytes as fast as it can over a few connections to
+ // the egress discard listener.
+ EgressStream Mode = "egress-stream"
+)
+
+// Targets is where the generator sends traffic and with what credentials.
+type Targets struct {
+ Ingress string
+ Connect string
+ Egress string
+ EgressDiscard string
+ // ServerName is the name atunnel's ingress certificate is checked for.
+ ServerName string
+ // RouterBundle is the router client certificate; CA verifies atunnel.
+ RouterBundle string
+ CA string
+ // Actors are atespace/name values, cycled through the routing header.
+ Actors []string
+ // ActorPort is the port a CONNECT names.
+ ActorPort int
+}
+
+// options configure one run of a mode.
+type options struct {
+ mode Mode
+ rate float64
+ duration time.Duration
+ warmup time.Duration
+ workers int
+ // connections caps pooled connections for the HTTP modes, and is the
+ // count to hold for EgressIdle and to stream over for EgressStream.
+ connections int
+ // newConnectionPerRequest turns keep-alive off for the HTTP modes.
+ newConnectionPerRequest bool
+ payloadBytes int
+ // whileHolding is called by EgressIdle with every connection still open,
+ // once the opens are done and before they are closed.
+ whileHolding func(context.Context)
+}
+
+// run is one execution of a mode.
+type run struct {
+ opts options
+ targets Targets
+ latency samples
+ completed atomic.Uint64
+ errors atomic.Uint64
+ bytes atomic.Uint64
+ nextActor atomic.Uint64
+
+ samplesMu sync.Mutex
+ errorSamples []string
+}
+
+const maxErrorSamples = 5
+
+func (r *run) fail(err error) {
+ r.errors.Add(1)
+ r.samplesMu.Lock()
+ defer r.samplesMu.Unlock()
+ if len(r.errorSamples) < maxErrorSamples {
+ r.errorSamples = append(r.errorSamples, err.Error())
+ }
+}
+
+// done records a completed unit if it falls in the measured window.
+func (r *run) done(u unit, n int64) {
+ if !u.measured {
+ return
+ }
+ r.latency.record(time.Since(u.due))
+ r.completed.Add(1)
+ r.bytes.Add(uint64(n))
+}
+
+func (r *run) actor() string {
+ return r.targets.Actors[(r.nextActor.Add(1)-1)%uint64(len(r.targets.Actors))]
+}
+
+// execute runs opts against targets and summarizes what it saw. It returns an
+// error only when the run could not start.
+func execute(ctx context.Context, targets Targets, opts options) (report.Load, error) {
+ r := &run{opts: opts, targets: targets}
+ p := &pacer{rate: opts.rate, duration: opts.duration, warmup: opts.warmup, workers: opts.workers, maxPending: 10000}
+ var err error
+ switch opts.mode {
+ case IngressH1, IngressH2:
+ err = r.ingress(ctx, p)
+ case Connect:
+ err = r.connect(ctx, p)
+ case EgressHTTP:
+ err = r.egressHTTP(ctx, p)
+ case EgressIdle:
+ err = r.egressIdle(ctx, p)
+ case EgressStream:
+ err = r.egressStream(ctx)
+ default:
+ err = fmt.Errorf("unknown mode %q", opts.mode)
+ }
+ if err != nil {
+ return report.Load{}, err
+ }
+ measured := (opts.duration - opts.warmup).Seconds()
+ l := report.Load{
+ Offered: opts.rate,
+ MeasuredSeconds: measured,
+ Scheduled: p.scheduled.Load(),
+ Skipped: p.skipped.Load(),
+ Completed: r.completed.Load(),
+ Errors: r.errors.Load(),
+ Latency: r.latency.percentiles(),
+ SendLag: p.lag.percentiles(),
+ Bytes: r.bytes.Load(),
+ ErrorSamples: r.errorSamples,
+ }
+ if measured > 0 {
+ l.Achieved = float64(l.Completed) / measured
+ l.MBytesPerSec = float64(l.Bytes) / measured / 1e6
+ }
+ return l, nil
+}
+
+// routerTLS is the client TLS configuration the router presents to atunnel.
+func (r *run) routerTLS(nextProtos ...string) (*tls.Config, error) {
+ cert, err := credbundle.Parse(r.targets.RouterBundle)
+ if err != nil {
+ return nil, err
+ }
+ roots, err := credbundle.ParsePool(r.targets.CA)
+ if err != nil {
+ return nil, err
+ }
+ return &tls.Config{
+ MinVersion: tls.VersionTLS12,
+ Certificates: []tls.Certificate{*cert},
+ RootCAs: roots,
+ ServerName: r.targets.ServerName,
+ NextProtos: nextProtos,
+ }, nil
+}
+
+func (r *run) transport(tlsConfig *tls.Config) *http.Transport {
+ return &http.Transport{
+ TLSClientConfig: tlsConfig,
+ DialContext: (&net.Dialer{Timeout: 10 * time.Second}).DialContext,
+ MaxIdleConns: r.opts.connections,
+ MaxIdleConnsPerHost: r.opts.connections,
+ MaxConnsPerHost: r.opts.connections,
+ DisableKeepAlives: r.opts.newConnectionPerRequest,
+ IdleConnTimeout: 90 * time.Second,
+ }
+}
+
+// get sends one GET and drains the response.
+func (r *run) get(ctx context.Context, client *http.Client, u unit, url, actor string) {
+ req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
+ if err != nil {
+ r.fail(err)
+ return
+ }
+ if actor != "" {
+ req.Header.Set(atenet.TargetActorHeader, actor)
+ }
+ resp, err := client.Do(req)
+ if err != nil {
+ r.fail(err)
+ return
+ }
+ n, _ := io.Copy(io.Discard, resp.Body)
+ _ = resp.Body.Close()
+ if resp.StatusCode != http.StatusOK {
+ r.fail(fmt.Errorf("HTTP %d", resp.StatusCode))
+ return
+ }
+ r.done(u, n)
+}
+
+func (r *run) ingress(ctx context.Context, p *pacer) error {
+ proto := "http/1.1"
+ if r.opts.mode == IngressH2 {
+ proto = "h2"
+ }
+ tlsConfig, err := r.routerTLS(proto)
+ if err != nil {
+ return err
+ }
+ transport := r.transport(tlsConfig)
+ transport.ForceAttemptHTTP2 = r.opts.mode == IngressH2
+ defer transport.CloseIdleConnections()
+ client := &http.Client{Transport: transport, Timeout: 30 * time.Second}
+ url := "https://" + r.targets.Ingress + "/"
+ p.run(ctx, func(ctx context.Context, u unit) {
+ r.get(ctx, client, u, url, r.actor())
+ })
+ return nil
+}
+
+// connect costs a TLS handshake, a CONNECT, and atunnel's dial to the actor per
+// unit, as a new router tunnel would.
+func (r *run) connect(ctx context.Context, p *pacer) error {
+ tlsConfig, err := r.routerTLS("http/1.1")
+ if err != nil {
+ return err
+ }
+ dialer := &tls.Dialer{NetDialer: &net.Dialer{Timeout: 10 * time.Second}, Config: tlsConfig}
+ authority := net.JoinHostPort("actor", strconv.Itoa(r.targets.ActorPort))
+ p.run(ctx, func(ctx context.Context, u unit) {
+ n, err := r.connectOnce(ctx, dialer, authority)
+ if err != nil {
+ r.fail(err)
+ return
+ }
+ r.done(u, n)
+ })
+ return nil
+}
+
+func (r *run) connectOnce(ctx context.Context, dialer *tls.Dialer, authority string) (int64, error) {
+ conn, err := dialer.DialContext(ctx, "tcp", r.targets.Connect)
+ if err != nil {
+ return 0, err
+ }
+ defer conn.Close()
+ _ = conn.SetDeadline(time.Now().Add(30 * time.Second))
+ if _, err := fmt.Fprintf(conn, "CONNECT %s HTTP/1.1\r\nHost: %s\r\n%s: %s\r\n\r\n",
+ authority, authority, atenet.TargetActorHeader, r.actor()); err != nil {
+ return 0, err
+ }
+ reader := bufio.NewReader(conn)
+ resp, err := http.ReadResponse(reader, &http.Request{Method: http.MethodConnect})
+ if err != nil {
+ return 0, err
+ }
+ if resp.StatusCode != http.StatusOK {
+ return 0, fmt.Errorf("CONNECT %d", resp.StatusCode)
+ }
+ if _, err := io.WriteString(conn, "GET / HTTP/1.1\r\nHost: actor\r\nConnection: close\r\n\r\n"); err != nil {
+ return 0, err
+ }
+ inner, err := http.ReadResponse(reader, nil)
+ if err != nil {
+ return 0, err
+ }
+ defer inner.Body.Close()
+ n, _ := io.Copy(io.Discard, inner.Body)
+ if inner.StatusCode != http.StatusOK {
+ return 0, fmt.Errorf("HTTP %d inside tunnel", inner.StatusCode)
+ }
+ return n, nil
+}
+
+func (r *run) egressHTTP(ctx context.Context, p *pacer) error {
+ transport := r.transport(nil)
+ defer transport.CloseIdleConnections()
+ client := &http.Client{Transport: transport, Timeout: 30 * time.Second}
+ url := "http://" + r.targets.Egress + "/"
+ p.run(ctx, func(ctx context.Context, u unit) {
+ r.get(ctx, client, u, url, "")
+ })
+ return nil
+}
+
+// egressIdle opens up to connections connections at the pacer's rate,
+// completes one request on each so its tunnel exists end to end, and holds
+// them until the duration is over. whileHolding runs as soon as every open
+// has finished, while the connections are still held.
+func (r *run) egressIdle(ctx context.Context, p *pacer) error {
+ var mu sync.Mutex
+ var held []net.Conn
+ defer func() {
+ mu.Lock()
+ defer mu.Unlock()
+ for _, c := range held {
+ _ = c.Close()
+ }
+ }()
+ end := time.Now().Add(r.opts.duration)
+ var opened, finished atomic.Int64
+ allOpened := make(chan struct{})
+ finish := func() {
+ if finished.Add(1) == int64(r.opts.connections) {
+ close(allOpened)
+ }
+ }
+ hookDone := make(chan struct{})
+ go func() {
+ defer close(hookDone)
+ select {
+ case <-allOpened:
+ case <-ctx.Done():
+ return
+ case <-time.After(time.Until(end)):
+ }
+ if r.opts.whileHolding != nil {
+ r.opts.whileHolding(ctx)
+ }
+ }()
+ request := "GET / HTTP/1.1\r\nHost: " + r.targets.Egress + "\r\n\r\n"
+ dialer := &net.Dialer{Timeout: 10 * time.Second}
+ p.run(ctx, func(ctx context.Context, u unit) {
+ if opened.Add(1) > int64(r.opts.connections) {
+ return
+ }
+ defer finish()
+ conn, err := dialer.DialContext(ctx, "tcp", r.targets.Egress)
+ if err != nil {
+ r.fail(err)
+ return
+ }
+ _ = conn.SetDeadline(time.Now().Add(30 * time.Second))
+ resp, err := func() (*http.Response, error) {
+ if _, err := io.WriteString(conn, request); err != nil {
+ return nil, err
+ }
+ return http.ReadResponse(bufio.NewReader(conn), nil)
+ }()
+ if err != nil {
+ r.fail(err)
+ _ = conn.Close()
+ return
+ }
+ n, _ := io.Copy(io.Discard, resp.Body)
+ _ = resp.Body.Close()
+ _ = conn.SetDeadline(time.Time{})
+ r.done(u, n)
+ mu.Lock()
+ held = append(held, conn)
+ mu.Unlock()
+ })
+ // Hold for whatever is left of the duration, and until the hook is done.
+ select {
+ case <-ctx.Done():
+ case <-time.After(time.Until(end)):
+ }
+ <-hookDone
+ return nil
+}
+
+// egressStream writes payloads over connections connections for the duration.
+// Each write is a unit, so latency is per write.
+func (r *run) egressStream(ctx context.Context) error {
+ payload := make([]byte, r.opts.payloadBytes)
+ start := time.Now()
+ end, measureFrom := start.Add(r.opts.duration), start.Add(r.opts.warmup)
+ dialer := &net.Dialer{Timeout: 10 * time.Second}
+ var wg sync.WaitGroup
+ for range r.opts.connections {
+ wg.Go(func() {
+ conn, err := dialer.DialContext(ctx, "tcp", r.targets.EgressDiscard)
+ if err != nil {
+ r.fail(err)
+ return
+ }
+ defer conn.Close()
+ for time.Now().Before(end) && ctx.Err() == nil {
+ due := time.Now()
+ _ = conn.SetWriteDeadline(due.Add(10 * time.Second))
+ if _, err := conn.Write(payload); err != nil {
+ r.fail(err)
+ return
+ }
+ r.done(unit{due: due, measured: due.After(measureFrom)}, int64(len(payload)))
+ }
+ })
+ }
+ wg.Wait()
+ return nil
+}
diff --git a/benchmarking/atunnel/internal/load/pacer.go b/benchmarking/atunnel/internal/load/pacer.go
new file mode 100644
index 0000000000..26ce38785d
--- /dev/null
+++ b/benchmarking/atunnel/internal/load/pacer.go
@@ -0,0 +1,136 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package load
+
+import (
+ "context"
+ "sort"
+ "sync"
+ "sync/atomic"
+ "time"
+
+ "go.ate.dev/substrate/benchmarking/atunnel/internal/report"
+)
+
+// pacer issues units of work at a fixed rate whatever each one takes: open
+// loop. Each unit carries the instant it was due, and latency is measured from
+// then, so a generator that falls behind shows it as latency and lag instead
+// of quietly offering less load.
+type pacer struct {
+ rate float64
+ duration time.Duration
+ warmup time.Duration
+ workers int
+ // maxPending bounds units queued for a free worker; beyond it they are
+ // skipped and counted.
+ maxPending int
+
+ scheduled atomic.Uint64
+ skipped atomic.Uint64
+ lag samples
+}
+
+// unit is one scheduled piece of work.
+type unit struct {
+ due time.Time
+ // measured is false during warmup.
+ measured bool
+}
+
+// run calls work for every unit until the duration is over or ctx ends, with
+// at most workers in flight, and waits for them to finish.
+func (p *pacer) run(ctx context.Context, work func(context.Context, unit)) {
+ queue := make(chan unit, p.maxPending)
+ var wg sync.WaitGroup
+ for range p.workers {
+ wg.Go(func() {
+ for u := range queue {
+ if u.measured {
+ p.lag.record(time.Since(u.due))
+ }
+ work(ctx, u)
+ }
+ })
+ }
+ defer wg.Wait()
+ defer close(queue)
+
+ start := time.Now()
+ interval := time.Duration(float64(time.Second) / p.rate)
+ timer := time.NewTimer(0)
+ defer timer.Stop()
+ for i := 0; ; i++ {
+ due := start.Add(time.Duration(i) * interval)
+ if due.Sub(start) >= p.duration {
+ return
+ }
+ if wait := time.Until(due); wait > 0 {
+ timer.Reset(wait)
+ select {
+ case <-ctx.Done():
+ return
+ case <-timer.C:
+ }
+ } else if ctx.Err() != nil {
+ return
+ }
+ p.scheduled.Add(1)
+ select {
+ case queue <- unit{due: due, measured: due.Sub(start) >= p.warmup}:
+ default:
+ p.skipped.Add(1)
+ }
+ }
+}
+
+// samples collects durations. Runs are short enough to keep every sample, so
+// percentiles are exact.
+type samples struct {
+ mu sync.Mutex
+ d []time.Duration
+}
+
+func (s *samples) record(d time.Duration) {
+ s.mu.Lock()
+ s.d = append(s.d, d)
+ s.mu.Unlock()
+}
+
+func (s *samples) percentiles() report.Percentiles {
+ s.mu.Lock()
+ d := s.d
+ s.d = nil
+ s.mu.Unlock()
+ if len(d) == 0 {
+ return report.Percentiles{}
+ }
+ sort.Slice(d, func(i, j int) bool { return d[i] < d[j] })
+ var sum time.Duration
+ for _, v := range d {
+ sum += v
+ }
+ at := func(q float64) float64 {
+ return ms(d[min(len(d)-1, int(q*float64(len(d))))])
+ }
+ return report.Percentiles{
+ P50: at(0.50),
+ P90: at(0.90),
+ P99: at(0.99),
+ Max: ms(d[len(d)-1]),
+ Mean: ms(sum / time.Duration(len(d))),
+ }
+}
+
+func ms(d time.Duration) float64 { return float64(d) / float64(time.Millisecond) }
diff --git a/benchmarking/atunnel/internal/load/scenarios.go b/benchmarking/atunnel/internal/load/scenarios.go
new file mode 100644
index 0000000000..f1b9406844
--- /dev/null
+++ b/benchmarking/atunnel/internal/load/scenarios.go
@@ -0,0 +1,126 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package load is the benchmark's load generator: an open-loop pacer, the
+// traffic modes that exercise each atunnel path, the scenarios built from
+// them, and the ladder that runs a scenario at rising rates while reading
+// atunnel's own CPU and memory from its admin endpoint.
+package load
+
+import (
+ "fmt"
+ "strings"
+)
+
+// Scenario is a mode and how to drive it.
+type Scenario struct {
+ Name string
+ Description string
+ // Unit names one unit of work, for reading the report.
+ Unit string
+ mode Mode
+ // newConnectionPerRequest turns keep-alive off.
+ newConnectionPerRequest bool
+ // holdsConnections makes each rung a connection count to open, at
+ // idleOpenRate per second, and hold.
+ holdsConnections bool
+ // streams makes the scenario a bandwidth measurement over this many
+ // connections, run once at whatever the first rung is.
+ streams int
+}
+
+// idleOpenRate is how many held connections per second EgressIdleConns opens.
+// Each costs atunnel a TCP accept and a CONNECT stream to the gateway.
+const idleOpenRate = 200
+
+// Scenarios are every scenario, in the order they run.
+var Scenarios = []Scenario{
+ {
+ Name: "IngressH1",
+ Description: "router-style HTTPS requests over HTTP/1.1 keep-alive into ingress, proxied to the actor",
+ Unit: "request",
+ mode: IngressH1,
+ },
+ {
+ Name: "IngressH2",
+ Description: "IngressH1 over HTTP/2, many requests multiplexed on few connections",
+ Unit: "request",
+ mode: IngressH2,
+ },
+ {
+ Name: "Connect",
+ Description: "per request: new TLS connection, CONNECT to the actor's port, one request inside",
+ Unit: "tunnel",
+ mode: Connect,
+ },
+ {
+ Name: "EgressHTTP",
+ Description: "actor-style plain HTTP keep-alive out through egress; one CONNECT stream to the gateway per connection",
+ Unit: "request",
+ mode: EgressHTTP,
+ },
+ {
+ Name: "EgressNewConnection",
+ Description: "EgressHTTP with a new connection per request, so a new CONNECT stream per request",
+ Unit: "request",
+ mode: EgressHTTP,
+ newConnectionPerRequest: true,
+ },
+ {
+ Name: "EgressIdleConns",
+ Description: fmt.Sprintf("rungs are connection counts: open at %d/s through egress, one request each, then hold", idleOpenRate),
+ Unit: "held connection",
+ mode: EgressIdle,
+ holdsConnections: true,
+ },
+ {
+ Name: "EgressTCP",
+ Description: "4 connections write 64 KiB chunks as fast as they can through egress to a discarding gateway; one rung",
+ Unit: "write",
+ mode: EgressStream,
+ streams: 4,
+ },
+}
+
+// Select returns the scenarios named in a comma-separated list, or all of
+// them for "all".
+func Select(list string) ([]Scenario, error) {
+ if list == "" || list == "all" {
+ return Scenarios, nil
+ }
+ var out []Scenario
+ for _, name := range strings.Split(list, ",") {
+ name = strings.TrimSpace(name)
+ found := false
+ for _, s := range Scenarios {
+ if strings.EqualFold(s.Name, name) {
+ out = append(out, s)
+ found = true
+ }
+ }
+ if !found {
+ return nil, fmt.Errorf("unknown scenario %q; known: %s", name, Names())
+ }
+ }
+ return out, nil
+}
+
+// Names lists the scenario names.
+func Names() string {
+ var names []string
+ for _, s := range Scenarios {
+ names = append(names, s.Name)
+ }
+ return strings.Join(names, ",")
+}
diff --git a/benchmarking/atunnel/internal/load/stats.go b/benchmarking/atunnel/internal/load/stats.go
new file mode 100644
index 0000000000..14ab71ac1f
--- /dev/null
+++ b/benchmarking/atunnel/internal/load/stats.go
@@ -0,0 +1,286 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package load
+
+import (
+ "context"
+ "encoding/json"
+ "fmt"
+ "io"
+ "net"
+ "net/http"
+ "os"
+ "path/filepath"
+ "sync"
+ "time"
+
+ "go.ate.dev/substrate/benchmarking/atunnel/internal/procstats"
+ "go.ate.dev/substrate/benchmarking/atunnel/internal/report"
+ "go.ate.dev/substrate/benchmarking/atunnel/internal/sink"
+)
+
+// Stats is what a benchmark process's /stats serves. Each process fills the
+// fields that are its own.
+type Stats struct {
+ Process procstats.Snapshot `json:"process"`
+ // Tunnel is set by atunnel (atunnel-bench tunnel).
+ Tunnel *TunnelInfo `json:"tunnel,omitempty"`
+ // IngressActor is set by the ingress actors, Gateway by the sink.
+ IngressActor *sink.IngressActorStats `json:"ingress_actor,omitempty"`
+ Gateway *sink.GatewayStats `json:"gateway,omitempty"`
+}
+
+// sinkStats reads the gateway's counters and the ingress actors', summed.
+func (r *Runner) sinkStats(ctx context.Context) (Stats, error) {
+ st, err := r.sink.stats(ctx)
+ if err != nil || r.SharedDir == "" {
+ return st, err
+ }
+ sockets, err := filepath.Glob(filepath.Join(r.SharedDir, "ingress-actor-*.sock"))
+ if err != nil {
+ return st, err
+ }
+ total := sink.IngressActorStats{}
+ for _, socket := range sockets {
+ w, err := unixAdmin(socket).stats(ctx)
+ if err != nil || w.IngressActor == nil {
+ // An ingress actor of a replaced atunnel; its successor's are
+ // read under the same names.
+ continue
+ }
+ total.Requests += w.IngressActor.Requests
+ total.ConnectionsSeen += w.IngressActor.ConnectionsSeen
+ total.ConnectionsOpen += w.IngressActor.ConnectionsOpen
+ }
+ st.IngressActor = &total
+ return st, nil
+}
+
+// unixAdmin reads /stats served on a Unix socket.
+func unixAdmin(socket string) *admin {
+ return &admin{base: "http://unix", client: &http.Client{
+ Timeout: 5 * time.Second,
+ Transport: &http.Transport{DialContext: func(ctx context.Context, _, _ string) (net.Conn, error) {
+ return (&net.Dialer{}).DialContext(ctx, "unix", socket)
+ }},
+ }}
+}
+
+// TunnelInfo is the part of atunnel's hosted actors the generator uses.
+type TunnelInfo struct {
+ Sandboxes []struct {
+ Actor string `json:"actor"`
+ } `json:"sandboxes"`
+ ActorPort int `json:"actor_port"`
+}
+
+// admin reads a benchmark process's admin endpoint.
+type admin struct {
+ base string
+ client *http.Client
+}
+
+func newAdmin(base string) *admin {
+ return &admin{base: base, client: &http.Client{Timeout: 5 * time.Second}}
+}
+
+func (a *admin) stats(ctx context.Context) (Stats, error) {
+ return a.get(ctx, "/stats")
+}
+
+// liveStats is stats after a garbage collection in the process.
+func (a *admin) liveStats(ctx context.Context) (Stats, error) {
+ return a.get(ctx, "/stats?gc=1")
+}
+
+func (a *admin) get(ctx context.Context, path string) (Stats, error) {
+ var s Stats
+ req, err := http.NewRequestWithContext(ctx, http.MethodGet, a.base+path, nil)
+ if err != nil {
+ return s, err
+ }
+ resp, err := a.client.Do(req)
+ if err != nil {
+ return s, err
+ }
+ defer resp.Body.Close()
+ if resp.StatusCode != http.StatusOK {
+ return s, fmt.Errorf("%s/stats: HTTP %d", a.base, resp.StatusCode)
+ }
+ return s, json.NewDecoder(resp.Body).Decode(&s)
+}
+
+// waitReady polls /healthz until it answers or ctx ends.
+func (a *admin) waitReady(ctx context.Context) error {
+ for {
+ req, err := http.NewRequestWithContext(ctx, http.MethodGet, a.base+"/healthz", nil)
+ if err != nil {
+ return err
+ }
+ resp, err := a.client.Do(req)
+ if err == nil {
+ _ = resp.Body.Close()
+ if resp.StatusCode == http.StatusOK {
+ return nil
+ }
+ }
+ select {
+ case <-ctx.Done():
+ return fmt.Errorf("%s not ready: %w", a.base, ctx.Err())
+ case <-time.After(200 * time.Millisecond):
+ }
+ }
+}
+
+// fetch saves a pprof endpoint's response to path.
+func (a *admin) fetch(ctx context.Context, endpoint, path string) error {
+ client := &http.Client{}
+ req, err := http.NewRequestWithContext(ctx, http.MethodGet, a.base+endpoint, nil)
+ if err != nil {
+ return err
+ }
+ resp, err := client.Do(req)
+ if err != nil {
+ return err
+ }
+ defer resp.Body.Close()
+ if resp.StatusCode != http.StatusOK {
+ return fmt.Errorf("%s%s: HTTP %d", a.base, endpoint, resp.StatusCode)
+ }
+ f, err := os.Create(path)
+ if err != nil {
+ return err
+ }
+ if _, err := io.Copy(f, resp.Body); err != nil {
+ _ = f.Close()
+ return err
+ }
+ return f.Close()
+}
+
+// watch samples a process over a measured window: a snapshot at the start, at
+// the end, and every interval between, for peaks.
+type watch struct {
+ read func(context.Context) (procstats.Snapshot, error)
+ start procstats.Snapshot
+
+ mu sync.Mutex
+ peak procstats.Snapshot
+ err error
+ stop chan struct{}
+ done chan struct{}
+}
+
+// failuresToGiveUp is how many reads in a row must fail before a watched
+// process counts as gone.
+const failuresToGiveUp = 2
+
+// startWatch takes the first snapshot and samples every interval after. If
+// the process stops answering, it calls onGone, when set, once.
+func startWatch(ctx context.Context, read func(context.Context) (procstats.Snapshot, error), interval time.Duration, onGone func()) *watch {
+ w := &watch{read: read, stop: make(chan struct{}), done: make(chan struct{})}
+ w.start, w.err = read(ctx)
+ w.peak = w.start
+ go func() {
+ defer close(w.done)
+ ticker := time.NewTicker(interval)
+ defer ticker.Stop()
+ failures := 0
+ for {
+ select {
+ case <-w.stop:
+ return
+ case <-ticker.C:
+ s, err := read(ctx)
+ if err == nil && !s.StartedAt.Equal(w.start.StartedAt) {
+ err = fmt.Errorf("process restarted at %s", s.StartedAt.Format(time.TimeOnly))
+ failures = failuresToGiveUp - 1
+ }
+ w.mu.Lock()
+ if err != nil {
+ w.err = err
+ failures++
+ } else {
+ w.observe(s)
+ failures = 0
+ }
+ w.mu.Unlock()
+ if failures == failuresToGiveUp && onGone != nil {
+ onGone()
+ }
+ }
+ }
+ }()
+ return w
+}
+
+func (w *watch) observe(s procstats.Snapshot) {
+ w.peak.RSSBytes = max(w.peak.RSSBytes, s.RSSBytes)
+ w.peak.HeapInUseBytes = max(w.peak.HeapInUseBytes, s.HeapInUseBytes)
+ w.peak.Goroutines = max(w.peak.Goroutines, s.Goroutines)
+ if s.Cgroup != nil {
+ if w.peak.Cgroup == nil {
+ c := *s.Cgroup
+ w.peak.Cgroup = &c
+ }
+ w.peak.Cgroup.MemoryCurrentBytes = max(w.peak.Cgroup.MemoryCurrentBytes, s.Cgroup.MemoryCurrentBytes)
+ }
+}
+
+// finish ends the window and returns the process's cost over it, with CPU
+// spread over units.
+func (w *watch) finish(ctx context.Context, units uint64) (report.Resources, error) {
+ close(w.stop)
+ <-w.done
+ end, err := w.read(ctx)
+ if err == nil && !end.StartedAt.Equal(w.start.StartedAt) {
+ err = fmt.Errorf("process restarted at %s", end.StartedAt.Format(time.TimeOnly))
+ }
+ w.mu.Lock()
+ defer w.mu.Unlock()
+ if err != nil {
+ return report.Resources{}, err
+ }
+ if w.err != nil {
+ return report.Resources{}, w.err
+ }
+ w.observe(end)
+ elapsed := end.Time.Sub(w.start.Time).Seconds()
+ res := report.Resources{
+ CPUSeconds: end.CPUSeconds - w.start.CPUSeconds,
+ GOMAXPROCS: end.GOMAXPROCS,
+ RSSEndBytes: end.RSSBytes,
+ RSSMaxBytes: w.peak.RSSBytes,
+ RSSIsPeak: end.RSSIsPeak,
+ HeapEndBytes: end.HeapInUseBytes,
+ HeapMaxBytes: w.peak.HeapInUseBytes,
+ GoroutinesMax: w.peak.Goroutines,
+ }
+ if elapsed > 0 {
+ res.Cores = res.CPUSeconds / elapsed
+ }
+ if units > 0 {
+ res.MicrosPerUnit = res.CPUSeconds / float64(units) * 1e6
+ }
+ if end.Cgroup != nil && w.start.Cgroup != nil {
+ if elapsed > 0 {
+ res.CgroupCores = (end.Cgroup.CPUSeconds - w.start.Cgroup.CPUSeconds) / elapsed
+ }
+ res.CgroupMemEndBytes = end.Cgroup.MemoryCurrentBytes
+ res.CgroupMemMaxBytes = w.peak.Cgroup.MemoryCurrentBytes
+ res.CgroupCPULimitCore = end.Cgroup.CPULimitCores
+ }
+ return res, nil
+}
diff --git a/benchmarking/atunnel/internal/pki/pki.go b/benchmarking/atunnel/internal/pki/pki.go
new file mode 100644
index 0000000000..c3a23e2506
--- /dev/null
+++ b/benchmarking/atunnel/internal/pki/pki.go
@@ -0,0 +1,235 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package pki writes the throwaway certificates the benchmark processes share,
+// standing in for the pod-identity and service certificates a cluster
+// projects. One CA signs everything; the fake atelet keeps its key to sign
+// actor certificates.
+package pki
+
+import (
+ "crypto"
+ "crypto/ecdsa"
+ "crypto/elliptic"
+ "crypto/rand"
+ "crypto/x509"
+ "crypto/x509/pkix"
+ "encoding/pem"
+ "fmt"
+ "math/big"
+ "net"
+ "net/url"
+ "os"
+ "path/filepath"
+ "time"
+
+ "go.ate.dev/substrate/internal/installdefaults"
+ "go.ate.dev/substrate/internal/substratex509"
+)
+
+// File names written by Generate, relative to its directory.
+const (
+ // CAFile is the CA certificate. Every peer verifies against it.
+ CAFile = "ca.pem"
+ // CAKeyFile is the CA's private key, for the fake atelet.
+ CAKeyFile = "ca-key.pem"
+ // WorkerBundle is the worker Pod's credential bundle: atunnel serves
+ // ingress with it and authenticates to atelet with it.
+ WorkerBundle = "worker.pem"
+ // AteletBundle is the fake atelet's serving certificate.
+ AteletBundle = "atelet.pem"
+ // RouterBundle is the client certificate the load generator presents to
+ // atunnel's ingress, carrying the router's SPIFFE ID.
+ RouterBundle = "router.pem"
+ // GatewayBundle is the fake egress gateway's serving certificate.
+ GatewayBundle = "gateway.pem"
+)
+
+// The node the worker and atelet share. atunnel checks that atelet's
+// certificate names the same node as its own.
+var node = substratex509.PodIdentity{NodeName: "bench-node", NodeUID: "00000000-0000-4000-8000-000000000001"}
+
+// Options shape the certificates.
+type Options struct {
+ // WorkerHosts are the names and IPs atunnel's serving certificate is
+ // valid for.
+ WorkerHosts []string
+ // GatewayHosts are the names and IPs the gateway's certificate is valid for.
+ GatewayHosts []string
+ // Validity is how long every certificate lasts.
+ Validity time.Duration
+}
+
+// Generate writes the CA and the bundles into dir, creating it if needed.
+func Generate(dir string, opts Options) error {
+ if len(opts.WorkerHosts) == 0 || len(opts.GatewayHosts) == 0 {
+ return fmt.Errorf("pki: worker and gateway hosts are required")
+ }
+ if opts.Validity <= 0 {
+ opts.Validity = 30 * 24 * time.Hour
+ }
+ if err := os.MkdirAll(dir, 0o755); err != nil {
+ return err
+ }
+ notBefore := time.Now().Add(-time.Minute)
+ notAfter := notBefore.Add(opts.Validity)
+
+ caKey, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
+ if err != nil {
+ return err
+ }
+ caTemplate := &x509.Certificate{
+ SerialNumber: serial(),
+ Subject: pkix.Name{CommonName: "atunnel-bench CA"},
+ NotBefore: notBefore,
+ NotAfter: notAfter,
+ KeyUsage: x509.KeyUsageCertSign | x509.KeyUsageDigitalSignature,
+ BasicConstraintsValid: true,
+ IsCA: true,
+ }
+ caDER, err := x509.CreateCertificate(rand.Reader, caTemplate, caTemplate, &caKey.PublicKey, caKey)
+ if err != nil {
+ return err
+ }
+ ca, err := x509.ParseCertificate(caDER)
+ if err != nil {
+ return err
+ }
+ if err := os.WriteFile(filepath.Join(dir, CAFile), PEMCertificate(caDER), 0o644); err != nil {
+ return err
+ }
+ caKeyDER, err := x509.MarshalPKCS8PrivateKey(caKey)
+ if err != nil {
+ return err
+ }
+ if err := os.WriteFile(filepath.Join(dir, CAKeyFile), pem.EncodeToMemory(&pem.Block{Type: "PRIVATE KEY", Bytes: caKeyDER}), 0o600); err != nil {
+ return err
+ }
+
+ worker := node
+ worker.Namespace, worker.ServiceAccountName, worker.ServiceAccountUID = "bench", "worker", "00000000-0000-4000-8000-000000000002"
+ worker.PodName, worker.PodUID = "worker-0", "00000000-0000-4000-8000-000000000003"
+ atelet := node
+ atelet.Namespace, atelet.ServiceAccountName, atelet.ServiceAccountUID = installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, "00000000-0000-4000-8000-000000000004"
+ atelet.PodName, atelet.PodUID = "atelet-0", "00000000-0000-4000-8000-000000000005"
+
+ leaves := []struct {
+ file string
+ name string
+ usage []x509.ExtKeyUsage
+ hosts []string
+ uri string
+ pod *substratex509.PodIdentity
+ }{
+ {WorkerBundle, "worker", []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth, x509.ExtKeyUsageClientAuth}, opts.WorkerHosts, "", &worker},
+ {AteletBundle, "atelet", []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth}, nil, installdefaults.AteletSPIFFEID(installdefaults.SystemNamespace), &atelet},
+ {RouterBundle, "router", []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth}, nil, installdefaults.RouterSPIFFEID(installdefaults.SystemNamespace), nil},
+ {GatewayBundle, "egress-gateway", []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth}, opts.GatewayHosts, "", nil},
+ }
+ for _, l := range leaves {
+ key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
+ if err != nil {
+ return err
+ }
+ template := &x509.Certificate{
+ SerialNumber: serial(),
+ Subject: pkix.Name{CommonName: l.name},
+ NotBefore: notBefore,
+ NotAfter: notAfter,
+ KeyUsage: x509.KeyUsageDigitalSignature,
+ ExtKeyUsage: l.usage,
+ }
+ for _, h := range l.hosts {
+ if ip := net.ParseIP(h); ip != nil {
+ template.IPAddresses = append(template.IPAddresses, ip)
+ } else {
+ template.DNSNames = append(template.DNSNames, h)
+ }
+ }
+ if l.uri != "" {
+ u, err := url.Parse(l.uri)
+ if err != nil {
+ return err
+ }
+ template.URIs = []*url.URL{u}
+ }
+ if l.pod != nil {
+ if err := substratex509.AddPodIdentityToCertificate(l.pod, template); err != nil {
+ return err
+ }
+ }
+ der, err := x509.CreateCertificate(rand.Reader, template, ca, &key.PublicKey, caKey)
+ if err != nil {
+ return err
+ }
+ keyDER, err := x509.MarshalPKCS8PrivateKey(key)
+ if err != nil {
+ return err
+ }
+ // Leaf, then CA, then key: the layout of a Kubernetes Pod Certificate
+ // credential bundle.
+ bundle := append(PEMCertificate(der), PEMCertificate(caDER)...)
+ bundle = append(bundle, pem.EncodeToMemory(&pem.Block{Type: "PRIVATE KEY", Bytes: keyDER})...)
+ if err := os.WriteFile(filepath.Join(dir, l.file), bundle, 0o644); err != nil {
+ return err
+ }
+ }
+ return nil
+}
+
+// LoadCA reads the CA certificate and key Generate wrote.
+func LoadCA(dir string) (*x509.Certificate, crypto.Signer, error) {
+ certPEM, err := os.ReadFile(filepath.Join(dir, CAFile))
+ if err != nil {
+ return nil, nil, err
+ }
+ block, _ := pem.Decode(certPEM)
+ if block == nil {
+ return nil, nil, fmt.Errorf("pki: %s holds no certificate", CAFile)
+ }
+ cert, err := x509.ParseCertificate(block.Bytes)
+ if err != nil {
+ return nil, nil, err
+ }
+ keyPEM, err := os.ReadFile(filepath.Join(dir, CAKeyFile))
+ if err != nil {
+ return nil, nil, err
+ }
+ block, _ = pem.Decode(keyPEM)
+ if block == nil {
+ return nil, nil, fmt.Errorf("pki: %s holds no key", CAKeyFile)
+ }
+ key, err := x509.ParsePKCS8PrivateKey(block.Bytes)
+ if err != nil {
+ return nil, nil, err
+ }
+ signer, ok := key.(crypto.Signer)
+ if !ok {
+ return nil, nil, fmt.Errorf("pki: CA key cannot sign")
+ }
+ return cert, signer, nil
+}
+
+// PEMCertificate encodes a DER certificate as PEM.
+func PEMCertificate(der []byte) []byte {
+ return pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der})
+}
+
+// Serial returns a random certificate serial number.
+func Serial() *big.Int { return serial() }
+
+func serial() *big.Int {
+ n, _ := rand.Int(rand.Reader, new(big.Int).Lsh(big.NewInt(1), 62))
+ return n
+}
diff --git a/benchmarking/atunnel/internal/pki/pki_test.go b/benchmarking/atunnel/internal/pki/pki_test.go
new file mode 100644
index 0000000000..71a483fbe9
--- /dev/null
+++ b/benchmarking/atunnel/internal/pki/pki_test.go
@@ -0,0 +1,77 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package pki
+
+import (
+ "crypto/x509"
+ "path/filepath"
+ "testing"
+
+ "go.ate.dev/substrate/internal/ateletdial"
+ "go.ate.dev/substrate/internal/credbundle"
+ "go.ate.dev/substrate/internal/installdefaults"
+ "go.ate.dev/substrate/internal/substratex509"
+)
+
+func TestGenerate(t *testing.T) {
+ dir := t.TempDir()
+ if err := Generate(dir, Options{WorkerHosts: []string{"atunnel-worker"}, GatewayHosts: []string{"atunnel-sink"}}); err != nil {
+ t.Fatal(err)
+ }
+ roots, err := credbundle.ParsePool(filepath.Join(dir, CAFile))
+ if err != nil {
+ t.Fatal(err)
+ }
+ for _, tc := range []struct {
+ file string
+ usage x509.ExtKeyUsage
+ name string
+ uri string
+ }{
+ {WorkerBundle, x509.ExtKeyUsageServerAuth, "atunnel-worker", ""},
+ {WorkerBundle, x509.ExtKeyUsageClientAuth, "", ""},
+ {AteletBundle, x509.ExtKeyUsageServerAuth, "", installdefaults.AteletSPIFFEID(installdefaults.SystemNamespace)},
+ {RouterBundle, x509.ExtKeyUsageClientAuth, "", installdefaults.RouterSPIFFEID(installdefaults.SystemNamespace)},
+ {GatewayBundle, x509.ExtKeyUsageServerAuth, "atunnel-sink", ""},
+ } {
+ cert, err := credbundle.Parse(filepath.Join(dir, tc.file))
+ if err != nil {
+ t.Fatalf("%s: %v", tc.file, err)
+ }
+ if _, err := cert.Leaf.Verify(x509.VerifyOptions{Roots: roots, DNSName: tc.name, KeyUsages: []x509.ExtKeyUsage{tc.usage}}); err != nil {
+ t.Errorf("%s does not verify for %v: %v", tc.file, tc.usage, err)
+ }
+ if tc.uri != "" && (len(cert.Leaf.URIs) != 1 || cert.Leaf.URIs[0].String() != tc.uri) {
+ t.Errorf("%s URIs = %v, want %s", tc.file, cert.Leaf.URIs, tc.uri)
+ }
+ }
+
+ // atunnel's own atelet client accepts the pair: the worker certificate
+ // carries a Pod identity, and atelet's names the same node.
+ if _, err := ateletdial.TLSConfig(filepath.Join(dir, WorkerBundle), filepath.Join(dir, CAFile), installdefaults.AteletSPIFFEID(installdefaults.SystemNamespace)); err != nil {
+ t.Fatalf("ateletdial.TLSConfig rejects the worker certificate: %v", err)
+ }
+ worker, _ := credbundle.Parse(filepath.Join(dir, WorkerBundle))
+ atelet, _ := credbundle.Parse(filepath.Join(dir, AteletBundle))
+ w, _ := substratex509.PodIdentityFromCertificate(worker.Leaf)
+ a, _ := substratex509.PodIdentityFromCertificate(atelet.Leaf)
+ if w == nil || a == nil || w.NodeName != a.NodeName || w.NodeUID != a.NodeUID {
+ t.Fatalf("worker and atelet Pod identities do not share a node: %+v, %+v", w, a)
+ }
+
+ if _, _, err := LoadCA(dir); err != nil {
+ t.Fatal(err)
+ }
+}
diff --git a/benchmarking/atunnel/internal/procstats/procstats.go b/benchmarking/atunnel/internal/procstats/procstats.go
new file mode 100644
index 0000000000..e4674bc963
--- /dev/null
+++ b/benchmarking/atunnel/internal/procstats/procstats.go
@@ -0,0 +1,174 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package procstats reads what a benchmark process costs: CPU time, resident
+// memory, Go heap, goroutines, and, inside a cgroup v2 container, the
+// container's own CPU and memory accounting. Each benchmark process serves its
+// snapshot over HTTP so the load generator can read it per rung.
+package procstats
+
+import (
+ "encoding/json"
+ "net/http"
+ "os"
+ "runtime"
+ "runtime/metrics"
+ "strconv"
+ "strings"
+ "time"
+)
+
+// Snapshot is a process's resource counters at one instant.
+type Snapshot struct {
+ Time time.Time `json:"time"`
+ // StartedAt identifies the process: a reader seeing it change knows the
+ // process was replaced, as a restarted container is.
+ StartedAt time.Time `json:"started_at"`
+ // CPUSeconds is user plus system CPU time since the process started.
+ CPUSeconds float64 `json:"cpu_seconds"`
+ // RSSBytes is the resident set size. On Linux it is current; elsewhere it
+ // is the peak so far, and RSSIsPeak says so.
+ RSSBytes uint64 `json:"rss_bytes"`
+ RSSIsPeak bool `json:"rss_is_peak,omitempty"`
+ // HeapInUseBytes is Go heap spans in use, live objects plus garbage not yet
+ // swept.
+ HeapInUseBytes uint64 `json:"heap_in_use_bytes"`
+ // GoTotalBytes is all memory the Go runtime has mapped and not returned.
+ GoTotalBytes uint64 `json:"go_total_bytes"`
+ Goroutines int `json:"goroutines"`
+ GOMAXPROCS int `json:"gomaxprocs"`
+ // Cgroup is the container's accounting, when the process runs in a cgroup
+ // v2 hierarchy other than the root.
+ Cgroup *Cgroup `json:"cgroup,omitempty"`
+}
+
+// Cgroup is a cgroup v2 container's accounting.
+type Cgroup struct {
+ CPUSeconds float64 `json:"cpu_seconds"`
+ MemoryCurrentBytes uint64 `json:"memory_current_bytes"`
+ MemoryPeakBytes uint64 `json:"memory_peak_bytes,omitempty"`
+ // CPULimitCores is cpu.max as cores; zero means unlimited.
+ CPULimitCores float64 `json:"cpu_limit_cores,omitempty"`
+}
+
+var startedAt = time.Now()
+
+var samples = []metrics.Sample{
+ {Name: "/memory/classes/heap/objects:bytes"},
+ {Name: "/memory/classes/heap/unused:bytes"},
+ {Name: "/memory/classes/total:bytes"},
+ {Name: "/memory/classes/heap/released:bytes"},
+}
+
+// Read takes a snapshot of this process.
+func Read() Snapshot {
+ s := Snapshot{
+ Time: time.Now(),
+ StartedAt: startedAt,
+ CPUSeconds: cpuSeconds(),
+ Goroutines: runtime.NumGoroutine(),
+ GOMAXPROCS: runtime.GOMAXPROCS(0),
+ }
+ s.RSSBytes, s.RSSIsPeak = rss()
+ ms := make([]metrics.Sample, len(samples))
+ copy(ms, samples)
+ metrics.Read(ms)
+ s.HeapInUseBytes = value(ms[0]) + value(ms[1])
+ s.GoTotalBytes = value(ms[2]) - value(ms[3])
+ s.Cgroup = readCgroup(ownCgroup())
+ return s
+}
+
+// ownCgroup is this process's cgroup v2 directory, from /proc/self/cgroup, so
+// a process moved into a cgroup of its own reports that one rather than its
+// container's.
+func ownCgroup() string {
+ b, err := os.ReadFile("/proc/self/cgroup")
+ if err != nil {
+ return "/sys/fs/cgroup"
+ }
+ for _, line := range strings.Split(strings.TrimSpace(string(b)), "\n") {
+ if path, ok := strings.CutPrefix(line, "0::"); ok {
+ return "/sys/fs/cgroup" + strings.TrimSuffix(path, "/")
+ }
+ }
+ return "/sys/fs/cgroup"
+}
+
+func value(s metrics.Sample) uint64 {
+ if s.Value.Kind() != metrics.KindUint64 {
+ return 0
+ }
+ return s.Value.Uint64()
+}
+
+// Handler serves Read as JSON, merged with whatever extra returns. With ?gc=1
+// it collects garbage first, so the heap figures are what is live.
+func Handler(extra func() map[string]any) http.Handler {
+ return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ if r.URL.Query().Get("gc") == "1" {
+ runtime.GC()
+ }
+ body := map[string]any{"process": Read()}
+ if extra != nil {
+ for k, v := range extra() {
+ body[k] = v
+ }
+ }
+ w.Header().Set("Content-Type", "application/json")
+ _ = json.NewEncoder(w).Encode(body)
+ })
+}
+
+// readCgroup reads cgroup v2 accounting in dir. The root of a hierarchy has no
+// memory.current, so a process in a host's root cgroup reports nil.
+func readCgroup(root string) *Cgroup {
+ stat, err := os.ReadFile(root + "/cpu.stat")
+ if err != nil {
+ return nil
+ }
+ current, err := readUint(root + "/memory.current")
+ if err != nil {
+ return nil
+ }
+ c := &Cgroup{MemoryCurrentBytes: current}
+ for _, line := range strings.Split(string(stat), "\n") {
+ if v, ok := strings.CutPrefix(line, "usage_usec "); ok {
+ usec, _ := strconv.ParseUint(v, 10, 64)
+ c.CPUSeconds = float64(usec) / 1e6
+ }
+ }
+ if peak, err := readUint(root + "/memory.peak"); err == nil {
+ c.MemoryPeakBytes = peak
+ }
+ if b, err := os.ReadFile(root + "/cpu.max"); err == nil {
+ fields := strings.Fields(string(b))
+ if len(fields) == 2 && fields[0] != "max" {
+ quota, _ := strconv.ParseFloat(fields[0], 64)
+ period, _ := strconv.ParseFloat(fields[1], 64)
+ if period > 0 {
+ c.CPULimitCores = quota / period
+ }
+ }
+ }
+ return c
+}
+
+func readUint(path string) (uint64, error) {
+ b, err := os.ReadFile(path)
+ if err != nil {
+ return 0, err
+ }
+ return strconv.ParseUint(strings.TrimSpace(string(b)), 10, 64)
+}
diff --git a/benchmarking/atunnel/internal/procstats/procstats_test.go b/benchmarking/atunnel/internal/procstats/procstats_test.go
new file mode 100644
index 0000000000..bcbe9c730a
--- /dev/null
+++ b/benchmarking/atunnel/internal/procstats/procstats_test.go
@@ -0,0 +1,64 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package procstats
+
+import (
+ "os"
+ "path/filepath"
+ "runtime"
+ "testing"
+)
+
+func TestRead(t *testing.T) {
+ s := Read()
+ if s.GOMAXPROCS == 0 || s.Goroutines == 0 || s.HeapInUseBytes == 0 || s.GoTotalBytes == 0 {
+ t.Errorf("snapshot missing runtime figures: %+v", s)
+ }
+ if runtime.GOOS == "linux" || runtime.GOOS == "darwin" {
+ if s.CPUSeconds <= 0 || s.RSSBytes == 0 {
+ t.Errorf("snapshot missing process figures: %+v", s)
+ }
+ }
+}
+
+func TestReadCgroup(t *testing.T) {
+ dir := t.TempDir()
+ write := func(name, content string) {
+ t.Helper()
+ if err := os.WriteFile(filepath.Join(dir, name), []byte(content), 0o644); err != nil {
+ t.Fatal(err)
+ }
+ }
+ if c := readCgroup(dir); c != nil {
+ t.Fatalf("readCgroup of an empty directory = %+v, want nil", c)
+ }
+ write("cpu.stat", "usage_usec 2500000\nuser_usec 2000000\nsystem_usec 500000\n")
+ // The root cgroup has cpu.stat but no memory.current: not a container.
+ if c := readCgroup(dir); c != nil {
+ t.Fatalf("readCgroup without memory.current = %+v, want nil", c)
+ }
+ write("memory.current", "104857600\n")
+ write("memory.peak", "209715200\n")
+ write("cpu.max", "150000 100000\n")
+ got := readCgroup(dir)
+ want := Cgroup{CPUSeconds: 2.5, MemoryCurrentBytes: 100 << 20, MemoryPeakBytes: 200 << 20, CPULimitCores: 1.5}
+ if got == nil || *got != want {
+ t.Fatalf("readCgroup = %+v, want %+v", got, want)
+ }
+ write("cpu.max", "max 100000\n")
+ if got := readCgroup(dir); got.CPULimitCores != 0 {
+ t.Errorf("unlimited cpu.max read as %v cores", got.CPULimitCores)
+ }
+}
diff --git a/benchmarking/atunnel/internal/procstats/rusage_other.go b/benchmarking/atunnel/internal/procstats/rusage_other.go
new file mode 100644
index 0000000000..1f79fd004e
--- /dev/null
+++ b/benchmarking/atunnel/internal/procstats/rusage_other.go
@@ -0,0 +1,21 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//go:build !linux && !darwin
+
+package procstats
+
+func cpuSeconds() float64 { return 0 }
+
+func rss() (uint64, bool) { return 0, false }
diff --git a/benchmarking/atunnel/internal/procstats/rusage_unix.go b/benchmarking/atunnel/internal/procstats/rusage_unix.go
new file mode 100644
index 0000000000..53cd48eccb
--- /dev/null
+++ b/benchmarking/atunnel/internal/procstats/rusage_unix.go
@@ -0,0 +1,62 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//go:build linux || darwin
+
+package procstats
+
+import (
+ "os"
+ "runtime"
+ "strconv"
+ "strings"
+ "syscall"
+)
+
+func cpuSeconds() float64 {
+ var ru syscall.Rusage
+ if err := syscall.Getrusage(syscall.RUSAGE_SELF, &ru); err != nil {
+ return 0
+ }
+ return tv(ru.Utime) + tv(ru.Stime)
+}
+
+func tv(t syscall.Timeval) float64 {
+ return float64(t.Sec) + float64(t.Usec)/1e6
+}
+
+// rss returns current RSS from /proc on Linux, and peak RSS from getrusage
+// elsewhere, where current RSS needs a Mach call.
+func rss() (uint64, bool) {
+ if runtime.GOOS == "linux" {
+ if b, err := os.ReadFile("/proc/self/status"); err == nil {
+ for _, line := range strings.Split(string(b), "\n") {
+ if v, ok := strings.CutPrefix(line, "VmRSS:"); ok {
+ kb, _ := strconv.ParseUint(strings.TrimSuffix(strings.TrimSpace(v), " kB"), 10, 64)
+ return kb * 1024, false
+ }
+ }
+ }
+ }
+ var ru syscall.Rusage
+ if err := syscall.Getrusage(syscall.RUSAGE_SELF, &ru); err != nil {
+ return 0, true
+ }
+ maxrss := uint64(ru.Maxrss)
+ if runtime.GOOS == "linux" {
+ // Linux reports kilobytes; macOS reports bytes.
+ maxrss *= 1024
+ }
+ return maxrss, true
+}
diff --git a/benchmarking/atunnel/internal/report/report.go b/benchmarking/atunnel/internal/report/report.go
new file mode 100644
index 0000000000..39ec39c1d3
--- /dev/null
+++ b/benchmarking/atunnel/internal/report/report.go
@@ -0,0 +1,216 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package report holds what a benchmark run measured and prints it.
+package report
+
+import (
+ "encoding/json"
+ "fmt"
+ "os"
+ "strings"
+ "time"
+)
+
+// Percentiles summarize a latency distribution, in milliseconds.
+type Percentiles struct {
+ P50 float64 `json:"p50_ms"`
+ P90 float64 `json:"p90_ms"`
+ P99 float64 `json:"p99_ms"`
+ Max float64 `json:"max_ms"`
+ Mean float64 `json:"mean_ms"`
+}
+
+// Load is what the generator saw during one rung.
+type Load struct {
+ // Offered is the target rate per second, or for a connection scenario the
+ // number of connections to hold.
+ Offered float64 `json:"offered"`
+ // MeasuredSeconds is the window after warmup that everything covers.
+ MeasuredSeconds float64 `json:"measured_seconds"`
+ Scheduled uint64 `json:"scheduled"`
+ // Skipped units never left the generator: its queue was full.
+ Skipped uint64 `json:"skipped"`
+ Completed uint64 `json:"completed"`
+ Errors uint64 `json:"errors"`
+ // Achieved is completed units per measured second.
+ Achieved float64 `json:"achieved"`
+ Latency Percentiles `json:"latency"`
+ // SendLag is how late units left the generator.
+ SendLag Percentiles `json:"send_lag"`
+ Bytes uint64 `json:"bytes"`
+ MBytesPerSec float64 `json:"mbytes_per_sec"`
+ ErrorSamples []string `json:"error_samples,omitempty"`
+}
+
+// Resources is one process's cost over the measured window.
+type Resources struct {
+ CPUSeconds float64 `json:"cpu_seconds"`
+ Cores float64 `json:"cores"`
+ // MicrosPerUnit is CPU spread over the completed units.
+ MicrosPerUnit float64 `json:"cpu_us_per_unit"`
+ GOMAXPROCS int `json:"gomaxprocs"`
+
+ RSSEndBytes uint64 `json:"rss_end_bytes,omitempty"`
+ RSSMaxBytes uint64 `json:"rss_max_bytes,omitempty"`
+ // RSSIsPeak means the platform only reports the process's lifetime peak.
+ RSSIsPeak bool `json:"rss_is_peak,omitempty"`
+ HeapEndBytes uint64 `json:"heap_end_bytes,omitempty"`
+ HeapMaxBytes uint64 `json:"heap_max_bytes,omitempty"`
+ GoroutinesMax int `json:"goroutines_max,omitempty"`
+ // HeapLiveBytes and GoroutinesLive are read after a garbage collection
+ // while a connection scenario holds its connections.
+ HeapLiveBytes uint64 `json:"heap_live_bytes,omitempty"`
+ GoroutinesLive int `json:"goroutines_live,omitempty"`
+
+ // Cgroup fields are set when the process runs in a cgroup v2 container.
+ CgroupCores float64 `json:"cgroup_cores,omitempty"`
+ CgroupMemEndBytes uint64 `json:"cgroup_mem_end_bytes,omitempty"`
+ CgroupMemMaxBytes uint64 `json:"cgroup_mem_max_bytes,omitempty"`
+ CgroupCPULimitCore float64 `json:"cgroup_cpu_limit_cores,omitempty"`
+}
+
+// Rung is one offered rate of a scenario.
+type Rung struct {
+ Load Load `json:"load"`
+ // Tunnel is atunnel's own cost: process B.
+ Tunnel Resources `json:"tunnel"`
+ // Generator and Sink are processes A and C, reported so a saturated one
+ // is visible.
+ Generator Resources `json:"generator"`
+ Sink Resources `json:"sink"`
+ Verdict string `json:"verdict"`
+ Reason string `json:"reason,omitempty"`
+ // Profiles are files written for this rung.
+ Profiles []string `json:"profiles,omitempty"`
+}
+
+// Report is one scenario across its rungs.
+type Report struct {
+ Scenario string `json:"scenario"`
+ Description string `json:"description"`
+ Unit string `json:"unit"`
+ Step time.Duration `json:"step_ns"`
+ Warmup time.Duration `json:"warmup_ns"`
+ Rungs []Rung `json:"rungs"`
+ // SinkCounters are the fake actor's and gateway's counters after the
+ // scenario, for checking against the generator.
+ SinkCounters map[string]any `json:"sink_counters,omitempty"`
+}
+
+// Verdicts.
+const (
+ VerdictOK = "OK"
+ // VerdictRigLimited means the generator or sink, not atunnel, set the
+ // pace; atunnel's numbers at that rung are a floor.
+ VerdictRigLimited = "RIG-LIMITED"
+ VerdictErrors = "ERRORS"
+ // VerdictSaturated means atunnel used all the CPU it may: the rung is past
+ // its capacity.
+ VerdictSaturated = "SATURATED"
+ // VerdictDown means atunnel stopped answering during the rung, as when
+ // it is killed for going over its memory limit.
+ VerdictDown = "DOWN"
+ // VerdictShortfall means atunnel fell behind the offered load, by
+ // completing fewer units or by backing up the generator's queue, without
+ // errors and without being at its CPU limit.
+ VerdictShortfall = "SHORTFALL"
+)
+
+// String renders the report as a table.
+func (r *Report) String() string {
+ var b strings.Builder
+ fmt.Fprintf(&b, "=== %s: %s\n", r.Scenario, r.Description)
+ fmt.Fprintf(&b, " unit: %s; %s per rung, first %s not measured\n", r.Unit, r.Step, r.Warmup)
+ cgroup := false
+ for _, g := range r.Rungs {
+ if g.Tunnel.CgroupMemEndBytes > 0 {
+ cgroup = true
+ }
+ }
+ header := fmt.Sprintf("%9s %9s %8s %8s %8s %6s | %6s %7s %9s %9s %6s",
+ "offered", "achieved", "p50", "p99", "max", "errs", "cores", "µs/unit", "rss-max", "heap-max", "gor")
+ if cgroup {
+ header += fmt.Sprintf(" %8s %9s", "cg-cores", "cg-mem")
+ }
+ header += fmt.Sprintf(" | %7s %7s %s", "A cores", "C cores", "verdict")
+ fmt.Fprintf(&b, " %s\n", header)
+ for _, g := range r.Rungs {
+ l, t := g.Load, g.Tunnel
+ offered := "-"
+ if l.Offered > 0 {
+ offered = fmt.Sprintf("%.0f", l.Offered)
+ }
+ line := fmt.Sprintf("%9s %9.0f %8s %8s %8s %6d | %6.2f %7.0f %9s %9s %6d",
+ offered, l.Achieved, msString(l.Latency.P50), msString(l.Latency.P99), msString(l.Latency.Max), l.Errors,
+ t.Cores, t.MicrosPerUnit, rssString(t), MiB(t.HeapMaxBytes), t.GoroutinesMax)
+ if cgroup {
+ line += fmt.Sprintf(" %8.2f %9s", t.CgroupCores, MiB(t.CgroupMemMaxBytes))
+ }
+ line += fmt.Sprintf(" | %7.2f %7.2f %s", g.Generator.Cores, g.Sink.Cores, g.Verdict)
+ if g.Reason != "" {
+ line += " (" + g.Reason + ")"
+ }
+ fmt.Fprintf(&b, " %s\n", line)
+ if l.MBytesPerSec > 0 && r.Unit == "write" {
+ fmt.Fprintf(&b, " %9s throughput %.0f MB/s\n", "", l.MBytesPerSec)
+ }
+ if t.HeapLiveBytes > 0 && l.Completed > 0 {
+ fmt.Fprintf(&b, " %9s holding %d: live heap %s after GC, %.1f KiB and %.1f goroutines per connection\n", "",
+ l.Completed, MiB(t.HeapLiveBytes), float64(t.HeapLiveBytes)/float64(l.Completed)/1024, float64(t.GoroutinesLive)/float64(l.Completed))
+ }
+ }
+ tunnelColumns := "cores..gor"
+ if cgroup {
+ tunnelColumns = "cores..cg-mem"
+ }
+ fmt.Fprintf(&b, " %s are atunnel alone (process B); A and C are the traffic generator and the far end\n", tunnelColumns)
+ if len(r.Rungs) > 0 && r.Rungs[0].Tunnel.RSSIsPeak {
+ fmt.Fprintf(&b, " rss-max*: this platform reports only atunnel's lifetime peak RSS\n")
+ }
+ if len(r.SinkCounters) > 0 {
+ counters, _ := json.Marshal(r.SinkCounters)
+ fmt.Fprintf(&b, " sink counters: %s\n", counters)
+ }
+ return b.String()
+}
+
+func msString(v float64) string {
+ if v >= 100 {
+ return fmt.Sprintf("%.0fms", v)
+ }
+ return fmt.Sprintf("%.2fms", v)
+}
+
+func rssString(r Resources) string {
+ s := MiB(r.RSSMaxBytes)
+ if r.RSSIsPeak {
+ s += "*"
+ }
+ return s
+}
+
+// MiB formats bytes as mebibytes.
+func MiB(b uint64) string {
+ return fmt.Sprintf("%.1fMiB", float64(b)/(1<<20))
+}
+
+// Write saves reports to path as JSON.
+func Write(path string, reports []*Report) error {
+ data, err := json.MarshalIndent(reports, "", " ")
+ if err != nil {
+ return err
+ }
+ return os.WriteFile(path, data, 0o644)
+}
diff --git a/benchmarking/atunnel/internal/report/report_test.go b/benchmarking/atunnel/internal/report/report_test.go
new file mode 100644
index 0000000000..d58aa390a1
--- /dev/null
+++ b/benchmarking/atunnel/internal/report/report_test.go
@@ -0,0 +1,73 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package report
+
+import (
+ "encoding/json"
+ "os"
+ "path/filepath"
+ "strings"
+ "testing"
+ "time"
+)
+
+func sample() *Report {
+ return &Report{
+ Scenario: "EgressIdleConns", Description: "idle", Unit: "held connection", Step: 10 * time.Second,
+ Rungs: []Rung{
+ {
+ Load: Load{Offered: 1000, Achieved: 1000, Completed: 1000, Latency: Percentiles{P50: 2.1, P99: 6.5, Max: 150}},
+ Tunnel: Resources{Cores: 0.12, MicrosPerUnit: 1238, HeapMaxBytes: 156 << 20, HeapLiveBytes: 155 << 20, GoroutinesLive: 5100, CgroupMemEndBytes: 223 << 20, CgroupMemMaxBytes: 223 << 20},
+ Verdict: VerdictOK,
+ },
+ {Load: Load{Offered: 8000}, Verdict: VerdictDown, Reason: "atunnel stopped answering"},
+ },
+ SinkCounters: map[string]any{"gateway_tunnels": 1000},
+ }
+}
+
+func TestString(t *testing.T) {
+ out := sample().String()
+ for _, want := range []string{
+ "=== EgressIdleConns: idle",
+ "cg-cores", // shown because a rung has cgroup figures
+ "2.10ms", "150ms",
+ "holding 1000: live heap 155.0MiB after GC, 158.7 KiB and 5.1 goroutines per connection",
+ "DOWN (atunnel stopped answering)",
+ `sink counters: {"gateway_tunnels":1000}`,
+ } {
+ if !strings.Contains(out, want) {
+ t.Errorf("report lacks %q:\n%s", want, out)
+ }
+ }
+}
+
+func TestWrite(t *testing.T) {
+ path := filepath.Join(t.TempDir(), "results.json")
+ if err := Write(path, []*Report{sample()}); err != nil {
+ t.Fatal(err)
+ }
+ b, err := os.ReadFile(path)
+ if err != nil {
+ t.Fatal(err)
+ }
+ var got []*Report
+ if err := json.Unmarshal(b, &got); err != nil {
+ t.Fatal(err)
+ }
+ if len(got) != 1 || len(got[0].Rungs) != 2 || got[0].Rungs[1].Verdict != VerdictDown {
+ t.Errorf("round trip = %+v", got)
+ }
+}
diff --git a/benchmarking/atunnel/internal/sink/gateway.go b/benchmarking/atunnel/internal/sink/gateway.go
new file mode 100644
index 0000000000..cc3349c466
--- /dev/null
+++ b/benchmarking/atunnel/internal/sink/gateway.go
@@ -0,0 +1,252 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package sink
+
+import (
+ "bufio"
+ "context"
+ "crypto/tls"
+ "io"
+ "net"
+ "net/http"
+ "strconv"
+ "sync/atomic"
+ "time"
+
+ "go.ate.dev/substrate/benchmarking/atunnel/internal/procstats"
+ "go.ate.dev/substrate/internal/credbundle"
+)
+
+// Destination ports the gateway gives meaning to. atunnel names the original
+// destination in each CONNECT; the gateway never dials it.
+const (
+ // AnswerPort tunnels carry HTTP/1.1 requests, which the gateway answers
+ // with its fixed body, as a web server behind a real gateway would.
+ AnswerPort = "80"
+ // DiscardPort tunnels carry raw bytes, which the gateway reads and drops,
+ // for throughput.
+ DiscardPort = "9"
+)
+
+// GatewayStats are the fake gateway's counters.
+type GatewayStats struct {
+ // Tunnels counts CONNECTs accepted; TunnelsH2 those that arrived as
+ // HTTP/2 streams, which is how atunnel prefers to send them.
+ Tunnels uint64 `json:"tunnels"`
+ TunnelsH2 uint64 `json:"tunnels_h2"`
+ TunnelsOpen int64 `json:"tunnels_open"`
+ // Connections counts TLS connections from atunnel. Under HTTP/2 many
+ // tunnels share one.
+ Connections uint64 `json:"connections"`
+ Rejected uint64 `json:"rejected"`
+ // Requests are HTTP requests answered inside AnswerPort tunnels.
+ Requests uint64 `json:"requests"`
+ BytesIn uint64 `json:"bytes_in"`
+}
+
+// GatewayConfig configures a Gateway.
+type GatewayConfig struct {
+ // CredentialBundle is the gateway's serving certificate and key.
+ CredentialBundle string
+ // ClientCA verifies the actor certificates atunnel presents.
+ ClientCA string
+ BodyBytes int
+}
+
+// Gateway stands in for the remote egress gateway: an mTLS server accepting
+// CONNECT over HTTP/2 or HTTP/1.1 and handling the bytes inside each tunnel by
+// destination port.
+type Gateway struct {
+ tlsConfig *tls.Config
+ response []byte
+
+ tunnels atomic.Uint64
+ tunnelsH2 atomic.Uint64
+ tunnelsOpen atomic.Int64
+ connections atomic.Uint64
+ rejected atomic.Uint64
+ requests atomic.Uint64
+ bytesIn atomic.Uint64
+}
+
+// NewGateway loads the gateway's TLS material.
+func NewGateway(cfg GatewayConfig) (*Gateway, error) {
+ cert, err := credbundle.Parse(cfg.CredentialBundle)
+ if err != nil {
+ return nil, err
+ }
+ clientCAs, err := credbundle.ParsePool(cfg.ClientCA)
+ if err != nil {
+ return nil, err
+ }
+ body := fixedBody(cfg.BodyBytes)
+ response := "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nContent-Length: " + strconv.Itoa(len(body)) + "\r\n\r\n"
+ return &Gateway{
+ tlsConfig: &tls.Config{
+ MinVersion: tls.VersionTLS12,
+ Certificates: []tls.Certificate{*cert},
+ ClientAuth: tls.RequireAndVerifyClientCert,
+ ClientCAs: clientCAs,
+ NextProtos: []string{"h2", "http/1.1"},
+ },
+ response: append([]byte(response), body...),
+ }, nil
+}
+
+// Stats snapshots the counters.
+func (g *Gateway) Stats() GatewayStats {
+ return GatewayStats{
+ Tunnels: g.tunnels.Load(),
+ TunnelsH2: g.tunnelsH2.Load(),
+ TunnelsOpen: g.tunnelsOpen.Load(),
+ Connections: g.connections.Load(),
+ Rejected: g.rejected.Load(),
+ Requests: g.requests.Load(),
+ BytesIn: g.bytesIn.Load(),
+ }
+}
+
+// Serve serves TLS on lis until ctx ends.
+func (g *Gateway) Serve(ctx context.Context, lis net.Listener) error {
+ server := &http.Server{
+ Handler: http.HandlerFunc(g.serveHTTP),
+ TLSConfig: g.tlsConfig,
+ ReadHeaderTimeout: 10 * time.Second,
+ // The HTTP/2 settings of the real gateway's Envoy listener
+ // (manifests/ate-install/atenet-egress.yaml). They shape atunnel's
+ // side: it opens a new connection per 100 tunnels, and sizes each
+ // tunnel's send buffer from the advertised frame size, Envoy's
+ // default 16 KiB where Go's server would advertise 1 MiB.
+ HTTP2: &http.HTTP2Config{
+ MaxConcurrentStreams: 100,
+ MaxReadFrameSize: 16 << 10,
+ MaxReceiveBufferPerStream: 1 << 20,
+ MaxReceiveBufferPerConnection: 1 << 20,
+ },
+ ConnState: func(_ net.Conn, state http.ConnState) {
+ if state == http.StateNew {
+ g.connections.Add(1)
+ }
+ },
+ }
+ return serveUntil(ctx, server, func() error { return server.ServeTLS(lis, "", "") })
+}
+
+func (g *Gateway) serveHTTP(w http.ResponseWriter, r *http.Request) {
+ if r.Method != http.MethodConnect {
+ g.rejected.Add(1)
+ http.Error(w, "CONNECT required", http.StatusMethodNotAllowed)
+ return
+ }
+ _, port, err := net.SplitHostPort(r.Host)
+ if err != nil || (port != AnswerPort && port != DiscardPort) {
+ g.rejected.Add(1)
+ http.Error(w, "unknown destination port", http.StatusBadGateway)
+ return
+ }
+ g.tunnels.Add(1)
+ g.tunnelsOpen.Add(1)
+ defer g.tunnelsOpen.Add(-1)
+
+ var in io.Reader
+ var out io.Writer
+ if r.ProtoMajor == 2 {
+ // An HTTP/2 tunnel is the stream itself: the request body carries
+ // atunnel's bytes and the response body carries ours.
+ g.tunnelsH2.Add(1)
+ w.WriteHeader(http.StatusOK)
+ rc := http.NewResponseController(w)
+ if err := rc.Flush(); err != nil {
+ return
+ }
+ in, out = r.Body, flushWriter{w: w, rc: rc}
+ } else {
+ // An HTTP/1.1 tunnel takes over the connection.
+ conn, rw, err := http.NewResponseController(w).Hijack()
+ if err != nil {
+ return
+ }
+ defer conn.Close()
+ if _, err := rw.WriteString("HTTP/1.1 200 Connection Established\r\n\r\n"); err != nil {
+ return
+ }
+ if err := rw.Flush(); err != nil {
+ return
+ }
+ in, out = rw.Reader, conn
+ }
+
+ counted := countingReader{r: in, n: &g.bytesIn}
+ if port == DiscardPort {
+ _, _ = io.Copy(io.Discard, counted)
+ return
+ }
+ g.answer(counted, out)
+}
+
+// answer reads HTTP/1.1 requests from the tunnel and writes the fixed response
+// to each, until the tunnel closes.
+func (g *Gateway) answer(in io.Reader, out io.Writer) {
+ reader := bufio.NewReader(in)
+ for {
+ req, err := http.ReadRequest(reader)
+ if err != nil {
+ return
+ }
+ _, _ = io.Copy(io.Discard, req.Body)
+ _ = req.Body.Close()
+ g.requests.Add(1)
+ if _, err := out.Write(g.response); err != nil {
+ return
+ }
+ }
+}
+
+type countingReader struct {
+ r io.Reader
+ n *atomic.Uint64
+}
+
+func (c countingReader) Read(p []byte) (int, error) {
+ n, err := c.r.Read(p)
+ c.n.Add(uint64(n))
+ return n, err
+}
+
+// flushWriter pushes every write onto the HTTP/2 stream at once, as a tunnel
+// must.
+type flushWriter struct {
+ w io.Writer
+ rc *http.ResponseController
+}
+
+func (f flushWriter) Write(p []byte) (int, error) {
+ n, err := f.w.Write(p)
+ if err != nil {
+ return n, err
+ }
+ return n, f.rc.Flush()
+}
+
+// AdminHandler serves the gateway's counters on /stats, with the process's,
+// and /healthz.
+func AdminHandler(gateway *Gateway) http.Handler {
+ mux := http.NewServeMux()
+ mux.Handle("/stats", procstats.Handler(func() map[string]any {
+ return map[string]any{"gateway": gateway.Stats()}
+ }))
+ mux.HandleFunc("/healthz", func(w http.ResponseWriter, _ *http.Request) { _, _ = w.Write([]byte("ok")) })
+ return mux
+}
diff --git a/benchmarking/atunnel/internal/sink/ingress_actor.go b/benchmarking/atunnel/internal/sink/ingress_actor.go
new file mode 100644
index 0000000000..7a1a09913d
--- /dev/null
+++ b/benchmarking/atunnel/internal/sink/ingress_actor.go
@@ -0,0 +1,106 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package sink holds the peers atunnel talks to in the benchmark: the ingress
+// actor in each actor's sandbox, behind ingress, and the fake egress gateway,
+// behind egress. Both answer with a fixed body and count what they see.
+package sink
+
+import (
+ "context"
+ "errors"
+ "net"
+ "net/http"
+ "strconv"
+ "sync/atomic"
+ "time"
+)
+
+// IngressActorStats are the fake actor's counters.
+type IngressActorStats struct {
+ Requests uint64 `json:"requests"`
+ ConnectionsSeen uint64 `json:"connections_seen"`
+ ConnectionsOpen int64 `json:"connections_open"`
+}
+
+// IngressActor stands in for an actor's program receiving ingress: an HTTP
+// server answering every request with the same body. atunnel's ingress reaches it over HTTP/1.1,
+// or h2c for gRPC, and through CONNECT tunnels as raw HTTP/1.1.
+type IngressActor struct {
+ body []byte
+ requests atomic.Uint64
+ connectionsSeen atomic.Uint64
+ connectionsOpen atomic.Int64
+}
+
+// NewIngressActor returns an actor answering with bodyBytes of body.
+func NewIngressActor(bodyBytes int) *IngressActor {
+ return &IngressActor{body: fixedBody(bodyBytes)}
+}
+
+// Stats snapshots the counters.
+func (a *IngressActor) Stats() IngressActorStats {
+ return IngressActorStats{
+ Requests: a.requests.Load(),
+ ConnectionsSeen: a.connectionsSeen.Load(),
+ ConnectionsOpen: a.connectionsOpen.Load(),
+ }
+}
+
+func (a *IngressActor) ServeHTTP(w http.ResponseWriter, _ *http.Request) {
+ a.requests.Add(1)
+ w.Header().Set("Content-Type", "text/plain")
+ w.Header().Set("Content-Length", strconv.Itoa(len(a.body)))
+ _, _ = w.Write(a.body)
+}
+
+// Serve serves HTTP/1.1 and h2c on lis until ctx ends.
+func (a *IngressActor) Serve(ctx context.Context, lis net.Listener) error {
+ protocols := new(http.Protocols)
+ protocols.SetHTTP1(true)
+ protocols.SetUnencryptedHTTP2(true)
+ server := &http.Server{
+ Handler: a,
+ Protocols: protocols,
+ ReadHeaderTimeout: 10 * time.Second,
+ ConnState: func(_ net.Conn, state http.ConnState) {
+ switch state {
+ case http.StateNew:
+ a.connectionsSeen.Add(1)
+ a.connectionsOpen.Add(1)
+ case http.StateClosed, http.StateHijacked:
+ a.connectionsOpen.Add(-1)
+ }
+ },
+ }
+ return serveUntil(ctx, server, func() error { return server.Serve(lis) })
+}
+
+// serveUntil runs serve until ctx ends, then closes server.
+func serveUntil(ctx context.Context, server *http.Server, serve func() error) error {
+ stop := context.AfterFunc(ctx, func() { _ = server.Close() })
+ defer stop()
+ if err := serve(); !errors.Is(err, http.ErrServerClosed) {
+ return err
+ }
+ return nil
+}
+
+func fixedBody(n int) []byte {
+ body := make([]byte, n)
+ for i := range body {
+ body[i] = 'a' + byte(i%26)
+ }
+ return body
+}
diff --git a/benchmarking/atunnel/internal/sink/sink_test.go b/benchmarking/atunnel/internal/sink/sink_test.go
new file mode 100644
index 0000000000..3cbe5946d5
--- /dev/null
+++ b/benchmarking/atunnel/internal/sink/sink_test.go
@@ -0,0 +1,214 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package sink
+
+import (
+ "bufio"
+ "context"
+ "crypto/tls"
+ "fmt"
+ "io"
+ "net"
+ "net/http"
+ "path/filepath"
+ "testing"
+ "time"
+
+ "go.ate.dev/substrate/benchmarking/atunnel/internal/pki"
+ "go.ate.dev/substrate/internal/atunnel/egress"
+ "go.ate.dev/substrate/internal/credbundle"
+)
+
+func TestIngressActor(t *testing.T) {
+ ctx, cancel := context.WithCancel(context.Background())
+ t.Cleanup(cancel)
+ actor := NewIngressActor(5)
+ lis := listen(t)
+ go func() { _ = actor.Serve(ctx, lis) }()
+
+ resp, err := http.Get("http://" + lis.Addr().String() + "/")
+ if err != nil {
+ t.Fatal(err)
+ }
+ body, _ := io.ReadAll(resp.Body)
+ _ = resp.Body.Close()
+ if string(body) != "abcde" {
+ t.Errorf("body = %q, want abcde", body)
+ }
+ if st := actor.Stats(); st.Requests != 1 || st.ConnectionsSeen != 1 {
+ t.Errorf("Stats = %+v, want 1 request on 1 connection", st)
+ }
+}
+
+// TestGatewayThroughAtunnelsClient opens tunnels the way atunnel does, with
+// its own egress client, which prefers HTTP/2.
+func TestGatewayThroughAtunnelsClient(t *testing.T) {
+ gateway, dir := startGateway(t)
+ client := egressClient(t, gateway, dir)
+
+ conn, err := client.DialContext(context.Background(), net.JoinHostPort("192.0.2.10", AnswerPort))
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer conn.Close()
+ for range 2 {
+ if _, err := io.WriteString(conn, "GET / HTTP/1.1\r\nHost: x\r\n\r\n"); err != nil {
+ t.Fatal(err)
+ }
+ reader := bufio.NewReader(conn)
+ resp, err := http.ReadResponse(reader, nil)
+ if err != nil {
+ t.Fatal(err)
+ }
+ body, _ := io.ReadAll(resp.Body)
+ if resp.StatusCode != http.StatusOK || len(body) != 16 {
+ t.Fatalf("answer = %d %q, want 200 and a 16-byte body", resp.StatusCode, body)
+ }
+ }
+
+ discard, err := client.DialContext(context.Background(), net.JoinHostPort("192.0.2.10", DiscardPort))
+ if err != nil {
+ t.Fatal(err)
+ }
+ if _, err := discard.Write(make([]byte, 1000)); err != nil {
+ t.Fatal(err)
+ }
+ _ = discard.Close()
+
+ waitFor(t, func() bool {
+ st := gateway.g.Stats()
+ return st.Requests == 2 && st.BytesIn >= 1000 && st.TunnelsOpen <= 1
+ })
+ if st := gateway.g.Stats(); st.Tunnels != 2 || st.TunnelsH2 != 2 {
+ t.Errorf("Stats = %+v, want 2 tunnels, both HTTP/2", st)
+ }
+}
+
+func TestGatewayOverHTTP1(t *testing.T) {
+ gateway, dir := startGateway(t)
+ conn, err := tls.Dial("tcp", gateway.addr, clientTLS(t, dir, "http/1.1"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer conn.Close()
+ fmt.Fprintf(conn, "CONNECT 192.0.2.10:%s HTTP/1.1\r\nHost: 192.0.2.10:%s\r\n\r\n", AnswerPort, AnswerPort)
+ reader := bufio.NewReader(conn)
+ resp, err := http.ReadResponse(reader, &http.Request{Method: http.MethodConnect})
+ if err != nil || resp.StatusCode != http.StatusOK {
+ t.Fatalf("CONNECT = %v, %v; want 200", resp, err)
+ }
+ if _, err := io.WriteString(conn, "GET / HTTP/1.1\r\nHost: x\r\n\r\n"); err != nil {
+ t.Fatal(err)
+ }
+ inner, err := http.ReadResponse(reader, nil)
+ if err != nil || inner.StatusCode != http.StatusOK {
+ t.Fatalf("request in tunnel = %v, %v; want 200", inner, err)
+ }
+ if st := gateway.g.Stats(); st.Tunnels != 1 || st.TunnelsH2 != 0 {
+ t.Errorf("Stats = %+v, want 1 HTTP/1.1 tunnel", st)
+ }
+}
+
+func TestGatewayRejectsAnUnknownPort(t *testing.T) {
+ gateway, dir := startGateway(t)
+ client := egressClient(t, gateway, dir)
+ if _, err := client.DialContext(context.Background(), "192.0.2.10:443"); err == nil {
+ t.Fatal("tunnel to an unknown port opened")
+ }
+ if st := gateway.g.Stats(); st.Rejected != 1 {
+ t.Errorf("Rejected = %d, want 1", st.Rejected)
+ }
+}
+
+type runningGateway struct {
+ g *Gateway
+ addr string
+}
+
+func startGateway(t *testing.T) (runningGateway, string) {
+ t.Helper()
+ ctx, cancel := context.WithCancel(context.Background())
+ t.Cleanup(cancel)
+ dir := t.TempDir()
+ if err := pki.Generate(dir, pki.Options{WorkerHosts: []string{"127.0.0.1"}, GatewayHosts: []string{"127.0.0.1"}}); err != nil {
+ t.Fatal(err)
+ }
+ g, err := NewGateway(GatewayConfig{
+ CredentialBundle: filepath.Join(dir, pki.GatewayBundle),
+ ClientCA: filepath.Join(dir, pki.CAFile),
+ BodyBytes: 16,
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+ lis := listen(t)
+ go func() { _ = g.Serve(ctx, lis) }()
+ return runningGateway{g: g, addr: lis.Addr().String()}, dir
+}
+
+// egressClient is atunnel's egress client, presenting the router certificate,
+// which the gateway, checking only the chain, accepts as an actor's.
+func egressClient(t *testing.T, gateway runningGateway, dir string) *egress.Client {
+ t.Helper()
+ cert, err := credbundle.Parse(filepath.Join(dir, pki.RouterBundle))
+ if err != nil {
+ t.Fatal(err)
+ }
+ client, err := egress.NewClient(egress.ClientConfig{
+ GatewayAddress: gateway.addr,
+ ServerName: "127.0.0.1",
+ GetClientCertificate: func(*tls.CertificateRequestInfo) (*tls.Certificate, error) { return cert, nil },
+ TrustBundlePath: filepath.Join(dir, pki.CAFile),
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { _ = client.Close() })
+ return client
+}
+
+func clientTLS(t *testing.T, dir string, proto string) *tls.Config {
+ t.Helper()
+ cert, err := credbundle.Parse(filepath.Join(dir, pki.RouterBundle))
+ if err != nil {
+ t.Fatal(err)
+ }
+ roots, err := credbundle.ParsePool(filepath.Join(dir, pki.CAFile))
+ if err != nil {
+ t.Fatal(err)
+ }
+ return &tls.Config{Certificates: []tls.Certificate{*cert}, RootCAs: roots, ServerName: "127.0.0.1", NextProtos: []string{proto}}
+}
+
+func listen(t *testing.T) net.Listener {
+ t.Helper()
+ lis, err := net.Listen("tcp", "127.0.0.1:0")
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { _ = lis.Close() })
+ return lis
+}
+
+func waitFor(t *testing.T, cond func() bool) {
+ t.Helper()
+ deadline := time.Now().Add(5 * time.Second)
+ for !cond() {
+ if time.Now().After(deadline) {
+ t.Fatal("condition not met within 5s")
+ }
+ time.Sleep(10 * time.Millisecond)
+ }
+}
diff --git a/benchmarking/atunnel/internal/workerhost/workerhost.go b/benchmarking/atunnel/internal/workerhost/workerhost.go
new file mode 100644
index 0000000000..fc82d9974d
--- /dev/null
+++ b/benchmarking/atunnel/internal/workerhost/workerhost.go
@@ -0,0 +1,159 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//go:build linux
+
+// Package workerhost hosts atunnel the way an ateworker does, with the
+// worker's own code: tunnel.Start once, then per actor the calls
+// cmd/ateworker-gvisor makes to run one, in its order. Only the sandbox itself
+// is missing: each actor's network namespaces, veth, nftables redirect, egress
+// listener, and DNS relay are real, but no gVisor runs in them.
+package workerhost
+
+import (
+ "context"
+ "errors"
+ "fmt"
+ "log/slog"
+ "strconv"
+ "sync"
+
+ "go.ate.dev/substrate/internal/ateworker/actornet"
+ "go.ate.dev/substrate/internal/ateworker/tunnel"
+ "go.ate.dev/substrate/internal/nodepath"
+ "go.ate.dev/substrate/internal/proto/ateworkerpb"
+ "go.ate.dev/substrate/internal/resources"
+)
+
+// Atespace holds every benchmark actor.
+const Atespace = "bench"
+
+// ActorPort is where the ingress actor listens inside each sandbox: the port
+// of actornet.ActorHTTPUpstream, which ingress proxies to.
+const ActorPort = 80
+
+// Sandbox is one hosted actor, as a process placing an ingress actor in it needs it.
+type Sandbox struct {
+ // Actor is atespace/name, the value of the routing header.
+ Actor string `json:"actor"`
+ UID string `json:"uid"`
+ // NetNS is the namespace the actor's runtime would run in, where its
+ // ingress actor listens and its egress starts.
+ NetNS string `json:"netns"`
+}
+
+// Info describes the hosted actors to the load generator and the supervisor.
+type Info struct {
+ Sandboxes []Sandbox `json:"sandboxes"`
+ // ActorPort is the port a CONNECT should name to reach an ingress actor.
+ ActorPort int `json:"actor_port"`
+}
+
+// Host is a running atunnel with its actors.
+type Host struct {
+ tunnel *tunnel.Tunnel
+
+ mu sync.Mutex
+ actors []*actor
+}
+
+type actor struct {
+ attribution resources.ActorAttribution
+ session *actornet.SandboxSession
+}
+
+// Start starts atunnel from cfg, as a worker's main does, and hosts actors
+// actors whose egress tunnels to gatewayAddress.
+func Start(ctx context.Context, cfg tunnel.Config, actors int, gatewayAddress string) (*Host, error) {
+ t, err := tunnel.Start(ctx, cfg, actornet.ActorHTTPUpstream)
+ if err != nil {
+ return nil, err
+ }
+ h := &Host{tunnel: t}
+ for i := range actors {
+ name := "actor-" + strconv.Itoa(i)
+ attribution := resources.ActorAttribution{
+ Ref: resources.ActorRef{Atespace: Atespace, Name: name},
+ UID: fmt.Sprintf("00000000-0000-4000-8000-%012d", i),
+ }
+ if err := h.host(ctx, attribution, gatewayAddress); err != nil {
+ return nil, errors.Join(fmt.Errorf("hosting %s: %w", name, err), h.Close(context.Background()))
+ }
+ }
+ return h, nil
+}
+
+// host runs an actor's network as cmd/ateworker-gvisor's RunWorkload does:
+// clear any previous activation, prepare egress (minting the actor
+// certificate through atelet), set up the sandbox network, then activate.
+func (h *Host) host(ctx context.Context, attribution resources.ActorAttribution, gatewayAddress string) error {
+ if err := h.tunnel.Deactivate(ctx, attribution); err != nil {
+ return err
+ }
+ egress, err := h.tunnel.PrepareEgress(ctx, attribution, &ateworkerpb.EgressGateway{Address: gatewayAddress})
+ if err != nil {
+ return err
+ }
+ session, err := actornet.ServeSandbox(ctx, actornet.SandboxNetworkConfig{
+ ActorUID: attribution.UID,
+ Veth: true,
+ EgressPort: h.tunnel.EgressPort,
+ DNSPolicy: h.tunnel.ActorEgressPolicy(attribution),
+ }, h.tunnel.Egress, h.tunnel.DNSRelay)
+ if err != nil {
+ return err
+ }
+ h.mu.Lock()
+ h.actors = append(h.actors, &actor{attribution: attribution, session: session})
+ h.mu.Unlock()
+ return h.tunnel.Activate(attribution, session.Dialer(), egress)
+}
+
+// Info returns the hosted actors.
+func (h *Host) Info() Info {
+ h.mu.Lock()
+ defer h.mu.Unlock()
+ info := Info{ActorPort: ActorPort}
+ for _, a := range h.actors {
+ info.Sandboxes = append(info.Sandboxes, Sandbox{
+ Actor: a.attribution.Ref.Atespace + "/" + a.attribution.Ref.Name,
+ UID: a.attribution.UID,
+ NetNS: nodepath.ActorNetNSPath(a.attribution.UID),
+ })
+ }
+ return info
+}
+
+// Close tears every actor down as a worker does when an actor stops:
+// deactivate its traffic, then remove its network.
+func (h *Host) Close(ctx context.Context) error {
+ h.mu.Lock()
+ actors := h.actors
+ h.actors = nil
+ h.mu.Unlock()
+ var errs []error
+ for _, a := range actors {
+ if err := h.tunnel.Deactivate(ctx, a.attribution); err != nil {
+ errs = append(errs, err)
+ }
+ if err := a.session.Close(ctx); err != nil {
+ errs = append(errs, err)
+ }
+ }
+ if err := errors.Join(errs...); err != nil {
+ slog.ErrorContext(ctx, "tearing down actors", slog.Any("err", err))
+ return err
+ }
+ return nil
+}
diff --git a/benchmarking/atunnel/internal/workerhost/workerhost_linux_test.go b/benchmarking/atunnel/internal/workerhost/workerhost_linux_test.go
new file mode 100644
index 0000000000..5cfb9caa43
--- /dev/null
+++ b/benchmarking/atunnel/internal/workerhost/workerhost_linux_test.go
@@ -0,0 +1,193 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//go:build linux
+
+package workerhost
+
+import (
+ "context"
+ "crypto/tls"
+ "io"
+ "net"
+ "net/http"
+ "path/filepath"
+ "testing"
+ "time"
+
+ "go.ate.dev/substrate/benchmarking/atunnel/internal/fakeatelet"
+ "go.ate.dev/substrate/benchmarking/atunnel/internal/pki"
+ "go.ate.dev/substrate/benchmarking/atunnel/internal/sink"
+ "go.ate.dev/substrate/internal/atenet"
+ "go.ate.dev/substrate/internal/ateworker/tunnel"
+ "go.ate.dev/substrate/internal/credbundle"
+ "go.ate.dev/substrate/internal/egresspolicy"
+ "go.ate.dev/substrate/internal/installdefaults"
+ "go.ate.dev/substrate/internal/netns"
+ "go.ate.dev/substrate/internal/nodepath"
+ "go.ate.dev/substrate/internal/roottest"
+)
+
+// TestHostsActorsWithTheWorkersCode starts atunnel with two actors, then sends
+// ingress to an ingress actor in a sandbox and egress out of it, through the
+// sandbox's real network and redirect.
+func TestHostsActorsWithTheWorkersCode(t *testing.T) {
+ roottest.Require(t, "creates network namespaces and nftables rules")
+ ctx, cancel := context.WithCancel(context.Background())
+ t.Cleanup(cancel)
+ dir := t.TempDir()
+ if err := pki.Generate(dir, pki.Options{WorkerHosts: []string{"127.0.0.1"}, GatewayHosts: []string{"127.0.0.1"}}); err != nil {
+ t.Fatal(err)
+ }
+
+ socket := filepath.Join(dir, "worker-support.sock")
+ saved := nodepath.WorkerSupportSocket
+ nodepath.WorkerSupportSocket = socket
+ t.Cleanup(func() { nodepath.WorkerSupportSocket = saved })
+ atelet, err := fakeatelet.New(dir)
+ if err != nil {
+ t.Fatal(err)
+ }
+ go func() { _ = atelet.Serve(ctx, dir, socket) }()
+
+ gateway, err := sink.NewGateway(sink.GatewayConfig{
+ CredentialBundle: filepath.Join(dir, pki.GatewayBundle),
+ ClientCA: filepath.Join(dir, pki.CAFile),
+ BodyBytes: 16,
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+ gatewayLis := listen(t)
+ go func() { _ = gateway.Serve(ctx, gatewayLis) }()
+
+ ingressAddr := freeAddress(t)
+ host, err := Start(ctx, tunnel.Config{
+ ListenAddress: ingressAddr,
+ ConnectListenAddress: freeAddress(t),
+ CredentialBundle: filepath.Join(dir, pki.WorkerBundle),
+ TrustBundle: filepath.Join(dir, pki.CAFile),
+ ClientIdentity: installdefaults.RouterSPIFFEID(installdefaults.SystemNamespace),
+ BrokerIdentity: installdefaults.AteletSPIFFEID(installdefaults.SystemNamespace),
+ EgressListenAddress: "0.0.0.0:15001",
+ EgressTrustBundle: filepath.Join(dir, pki.CAFile),
+ DNSEgressPolicy: egresspolicy.DNSEgressPolicyEnforceInAtunnel,
+ }, 2, gatewayLis.Addr().String())
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() {
+ closeCtx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
+ defer cancel()
+ if err := host.Close(closeCtx); err != nil {
+ t.Errorf("closing: %v", err)
+ }
+ })
+ info := host.Info()
+ if len(info.Sandboxes) != 2 || info.ActorPort != ActorPort {
+ t.Fatalf("Info = %+v, want 2 sandboxes on port %d", info, ActorPort)
+ }
+
+ // The ingress actor: a server inside the first actor's sandbox.
+ sandbox, err := netns.GetFromName(nodepath.ActorNetNSName(info.Sandboxes[0].UID))
+ if err != nil {
+ t.Fatal(err)
+ }
+ ingressActorLis, err := netns.Listen(ctx, sandbox, []uint16{ActorPort})
+ if err != nil {
+ t.Fatal(err)
+ }
+ go func() {
+ _ = http.Serve(ingressActorLis[0], http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+ _, _ = w.Write([]byte("ingress-actor"))
+ }))
+ }()
+
+ // Ingress, as the router sends it.
+ router, err := credbundle.Parse(filepath.Join(dir, pki.RouterBundle))
+ if err != nil {
+ t.Fatal(err)
+ }
+ roots, err := credbundle.ParsePool(filepath.Join(dir, pki.CAFile))
+ if err != nil {
+ t.Fatal(err)
+ }
+ ingress := &http.Client{Timeout: 10 * time.Second, Transport: &http.Transport{TLSClientConfig: &tls.Config{
+ Certificates: []tls.Certificate{*router},
+ RootCAs: roots,
+ ServerName: "127.0.0.1",
+ }}}
+ req, err := http.NewRequestWithContext(ctx, http.MethodGet, "https://"+ingressAddr+"/", nil)
+ if err != nil {
+ t.Fatal(err)
+ }
+ req.Header.Set(atenet.TargetActorHeader, info.Sandboxes[0].Actor)
+ if got := get(t, ingress, req); got != "ingress-actor" {
+ t.Errorf("ingress reached %q, want the ingress actor", got)
+ }
+
+ // Egress from inside the sandbox, redirected to atunnel and tunneled to
+ // the gateway, which answers on port 80.
+ egress := &http.Client{Timeout: 10 * time.Second, Transport: &http.Transport{DialContext: netns.Dialer(sandbox)}}
+ req, err = http.NewRequestWithContext(ctx, http.MethodGet, "http://192.0.2.10:80/", nil)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if got := get(t, egress, req); len(got) != 16 {
+ t.Errorf("egress got %q, want the gateway's 16-byte body", got)
+ }
+ if st := gateway.Stats(); st.Tunnels == 0 {
+ t.Errorf("gateway saw no tunnels: %+v", st)
+ }
+}
+
+func get(t *testing.T, client *http.Client, req *http.Request) string {
+ t.Helper()
+ resp, err := client.Do(req)
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer resp.Body.Close()
+ body, err := io.ReadAll(resp.Body)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if resp.StatusCode != http.StatusOK {
+ t.Fatalf("%s: HTTP %d: %s", req.URL, resp.StatusCode, body)
+ }
+ return string(body)
+}
+
+func listen(t *testing.T) net.Listener {
+ t.Helper()
+ lis, err := net.Listen("tcp", "127.0.0.1:0")
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { _ = lis.Close() })
+ return lis
+}
+
+// freeAddress returns a loopback address with a port free a moment ago, for
+// listeners tunnel.Start binds itself.
+func freeAddress(t *testing.T) string {
+ t.Helper()
+ lis, err := net.Listen("tcp", "127.0.0.1:0")
+ if err != nil {
+ t.Fatal(err)
+ }
+ addr := lis.Addr().String()
+ _ = lis.Close()
+ return addr
+}
diff --git a/benchmarking/atunnel/run-docker.sh b/benchmarking/atunnel/run-docker.sh
new file mode 100755
index 0000000000..cb68d3828d
--- /dev/null
+++ b/benchmarking/atunnel/run-docker.sh
@@ -0,0 +1,150 @@
+#!/usr/bin/env bash
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Runs the atunnel benchmark as three containers on one docker network:
+#
+# atunnel-worker privileged; atunnel in a cgroup of its own with the
+# CPU and memory limits, hosting actors with the worker's own
+# code, plus the fake atelet and the ingress and egress actors
+# atunnel-sink the fake egress gateway
+# atunnel-load the load generator, in the foreground
+#
+# Arguments after the options go to `atunnel-bench load`:
+#
+# benchmarking/atunnel/run-docker.sh --tunnel-cpus 1 --tunnel-memory 1g -- --rungs 1000,2000,4000
+#
+# Options:
+# --tunnel-cpus N CPU limit for atunnel (default 1)
+# --tunnel-memory M memory limit for atunnel, without swap (default 1g)
+# --actors N actors atunnel hosts (default 10)
+# --cpuset-worker S pin the worker container to these CPUs, e.g. 0-3
+# --cpuset-rig S pin the generator and sink to these CPUs, e.g. 4-9
+#
+# Results land in benchmarking/atunnel/runs//: results.txt,
+# results.json, profiles/, and logs/.
+
+set -o errexit -o nounset -o pipefail
+
+tunnel_cpus=1
+tunnel_memory=1g
+actors=10
+cpuset_worker=""
+cpuset_rig=""
+while [[ $# -gt 0 ]]; do
+ case "$1" in
+ --tunnel-cpus) tunnel_cpus="$2"; shift 2 ;;
+ --tunnel-memory) tunnel_memory="$2"; shift 2 ;;
+ --actors) actors="$2"; shift 2 ;;
+ --cpuset-worker) cpuset_worker="$2"; shift 2 ;;
+ --cpuset-rig) cpuset_rig="$2"; shift 2 ;;
+ --) shift; break ;;
+ *) break ;;
+ esac
+done
+
+here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+repo="$(cd "${here}/../.." && pwd)"
+out="${here}/runs/$(date +%Y%m%d-%H%M%S)"
+# Under runs/ rather than /tmp so Docker Desktop can always mount it.
+work="${here}/runs/.work-$$"
+ctx="${work}/image"
+pki="${work}/pki"
+mkdir -p "${ctx}" "${pki}" "${out}/profiles" "${out}/logs"
+image=atunnel-bench:local
+network=atunnel-bench
+# The egress actor and the ingress actors serve on Unix sockets here. A named
+# volume, because sockets do not work on Docker Desktop's host mounts.
+shared=atunnel-bench-shared
+arch="$(docker info --format '{{.Architecture}}')"
+case "${arch}" in
+ x86_64) goarch=amd64 ;;
+ aarch64 | arm64) goarch=arm64 ;;
+ *) echo "unsupported docker architecture ${arch}" >&2; exit 1 ;;
+esac
+
+echo "building linux/${goarch} binaries and ${image}"
+(cd "${repo}" && CGO_ENABLED=0 GOOS=linux GOARCH="${goarch}" go build -o "${ctx}/" ./benchmarking/atunnel/cmd/...)
+docker build --quiet -t "${image}" -f "${here}/Dockerfile" "${ctx}" >/dev/null
+
+containers=(atunnel-load atunnel-worker atunnel-sink)
+remove_all() {
+ for c in "${containers[@]}"; do
+ docker rm -f "${c}" >/dev/null 2>&1 || true
+ done
+ docker network rm "${network}" >/dev/null 2>&1 || true
+ docker volume rm "${shared}" >/dev/null 2>&1 || true
+}
+cleanup() {
+ local code=$?
+ # The worker logs every atunnel death, with the OOM kills its cgroup counted.
+ if docker cp atunnel-worker:/shared/tunnel-deaths.log "${out}/logs/" >/dev/null 2>&1; then
+ local deaths ooms
+ deaths="$(wc -l <"${out}/logs/tunnel-deaths.log" | tr -d ' ')"
+ ooms="$(grep -c 'OOM kills: [1-9]' "${out}/logs/tunnel-deaths.log" || true)"
+ echo "atunnel died ${deaths} time(s), ${ooms} of them OOM-killed at its ${tunnel_memory} limit" | tee -a "${out}/results.txt"
+ fi
+ echo "run in ${out}"
+ for c in "${containers[@]}"; do
+ docker logs "${c}" >"${out}/logs/${c}.log" 2>&1 || true
+ done
+ remove_all
+ rm -rf "${work}"
+ exit "${code}"
+}
+trap cleanup EXIT
+remove_all
+docker network create "${network}" >/dev/null
+docker volume create "${shared}" >/dev/null
+
+# The nonroot image runs as uid 65532; it writes the PKI and the results.
+chmod 0777 "${out}" "${pki}" "${out}/profiles"
+docker run --rm --user 65532 -v "${pki}:/pki" "${image}" \
+ atunnel-bench pki --dir /pki --worker-hosts atunnel-worker --gateway-hosts atunnel-sink
+
+# The generator and atunnel open many short connections; without these a run
+# runs out of ephemeral ports to sockets in TIME_WAIT.
+port_opts=(--sysctl "net.ipv4.ip_local_port_range=1024 65535" --sysctl net.ipv4.tcp_tw_reuse=1)
+rig_opts=()
+if [[ -n "${cpuset_rig}" ]]; then
+ rig_opts+=(--cpuset-cpus "${cpuset_rig}")
+fi
+worker_opts=()
+if [[ -n "${cpuset_worker}" ]]; then
+ worker_opts+=(--cpuset-cpus "${cpuset_worker}")
+fi
+
+docker run --detach --network "${network}" ${rig_opts[@]+"${rig_opts[@]}"} --name atunnel-sink \
+ -v "${pki}:/pki:ro" "${image}" \
+ atunnel-bench sink --pki-dir /pki >/dev/null
+
+# Privileged for the actors' network namespaces, nftables, and cgroups. The
+# worker's credentials are mounted where a worker Pod's are projected, so
+# atunnel runs on its default flags.
+docker run --detach --network "${network}" --privileged --user 0 --cgroupns private \
+ "${port_opts[@]}" ${worker_opts[@]+"${worker_opts[@]}"} --name atunnel-worker \
+ -v "${pki}:/pki:ro" -v "${shared}:/shared" \
+ -v "${pki}/worker.pem:/run/podidentity.podcert.ate.dev/credential-bundle.pem:ro" \
+ -v "${pki}/ca.pem:/run/podidentity.podcert.ate.dev/trust-bundle.pem:ro" \
+ -v "${pki}/ca.pem:/run/servicedns.podcert.ate.dev/trust-bundle.pem:ro" \
+ "${image}" \
+ atunnel-bench worker --pki-dir /pki --shared-dir /shared \
+ --tunnel-cpus "${tunnel_cpus}" --tunnel-memory "${tunnel_memory}" -- \
+ --actors "${actors}" --gateway atunnel-sink:9443 --dns-egress-policy enforce-in-atunnel >/dev/null
+
+# The generator runs in the foreground; Ctrl-C stops it and the trap cleans up.
+docker run --network "${network}" "${port_opts[@]}" ${rig_opts[@]+"${rig_opts[@]}"} --name atunnel-load \
+ -v "${pki}:/pki:ro" -v "${shared}:/shared" -v "${out}:/out" "${image}" \
+ atunnel-bench load --pki-dir /pki --shared-dir /shared \
+ --results-json /out/results.json --profile-dir /out/profiles "$@" | tee "${out}/results.txt"
diff --git a/benchmarking/atunnel/runs/20261016-123913/report.md b/benchmarking/atunnel/runs/20261016-123913/report.md
new file mode 100644
index 0000000000..a533f7e20f
--- /dev/null
+++ b/benchmarking/atunnel/runs/20261016-123913/report.md
@@ -0,0 +1,193 @@
+# atunnel benchmark: what the results mean
+
+Run: `20261016-123913`. atunnel was given 1 CPU core and 1 GiB of memory
+without swap, roughly what a worker pod gets. Raw numbers are in `results.txt`.
+
+## What atunnel does
+
+atunnel runs in every worker and carries the network traffic of the actors on
+that worker:
+
+- **Ingress (incoming):** requests from the router to an actor. atunnel checks
+ which actor a request is for and forwards it into that actor's sandbox.
+- **Egress (outgoing):** connections an actor opens to the internet. The
+ sandbox redirects them to atunnel, which forwards them to the egress gateway.
+
+The benchmark measures how much CPU and memory atunnel uses as traffic grows.
+
+## How the test works
+
+atunnel is started and hosts 10 actors with the worker's own code: each actor
+gets a real sandbox network, with its own network namespaces, traffic
+redirection, and certificate. Small programs stand in for the actors' own: an
+ingress actor in every sandbox receives requests, and an egress actor in
+actor-0's sandbox makes outgoing connections. A load generator sends traffic
+at a fixed rate, 1000, 2000, 4000, then 8000 per second: ingress to the
+ingress actors, and egress through the egress actor to a fake egress gateway.
+At each rate, only atunnel's own CPU and memory are measured. The appendix
+explains every column of `results.txt`.
+
+The router reaches atunnel in one of two ways, set by its `--upstream-connect`
+flag: through CONNECT tunnels (the default), or over HTTPS (`false`). The
+scenarios cover both.
+
+## Results by scenario
+
+| Scenario | Router mode | What the generator sends | Last rate with verdict OK | Rate where atunnel used its whole core | CPU per unit while OK |
+|---|---|---|---|---|---|
+| IngressH1 | `false` | incoming HTTPS requests, HTTP/1.1, connections reused | 1,000/s (17% of the core) | 2,000/s | 170 µs per request |
+| IngressH2 | `false` | incoming HTTPS requests, HTTP/2, many requests share a connection | 4,000/s (79%) | 8,000/s | 160–220 µs per request |
+| Connect | default | a new secure connection per request, one request inside it | none: 1,000/s already used 98% | 1,000/s | 980 µs per connection |
+| EgressHTTP | either | outgoing HTTP requests, connections reused | 4,000/s (69%) | 8,000/s | 170–200 µs per request |
+| EgressNewConnection | either | outgoing HTTP requests, a new connection per request | 2,000/s (81%) | 4,000/s | 400–510 µs per request |
+| EgressIdleConns | either | outgoing connections opened and left idle | 4,000 connections held | not CPU-bound; ran out of memory at 8,000 | — |
+| EgressTCP | either | a continuous stream of data on 4 connections | — | used its whole core at 304 MB/s | — |
+
+Memory in EgressIdleConns, where each connection carries one request and then
+stays open with nothing sent:
+
+| Idle connections | atunnel memory in use (after garbage collection) | atunnel container memory |
+|---|---|---|
+| 1,000 | 155 MB | 192 MB |
+| 2,000 | 307 MB | 424 MB |
+| 4,000 | 610 MB | 786 MB |
+| 8,000 | killed for going over the 1 GiB limit, after about 2,400 connections | — |
+
+## Findings
+
+Code links are to upstream commit `3a2ca623`, the version benchmarked.
+
+### Finding 1: each idle outgoing connection holds about 157 KB of memory
+
+Filed as [#2759](https://github.com/agent-substrate/substrate/issues/2759).
+
+- **Observed.** In EgressIdleConns, memory grew by about 157 KB and 5
+ goroutines per open connection at every count tested, with no data moving.
+ At 8,000 connections atunnel went over its 1 GiB limit and was killed.
+- **Cause.** About 140 KB of it is data buffers allocated when a connection
+ opens and kept until it closes: two 32 KB copy buffers in
+ [`copyBothWays`](https://github.com/agent-substrate/substrate/blob/3a2ca62357485c4228fae466614ec98cf2caef69/internal/atunnel/egress/egress.go#L328),
+ a 32 KB read buffer held by each HTTP/2 stream's background reader while it
+ waits
+ ([`client.go#L733`](https://github.com/agent-substrate/substrate/blob/3a2ca62357485c4228fae466614ec98cf2caef69/internal/atunnel/egress/client.go#L733)),
+ and Go's 16 KB HTTP/2 send buffer.
+- **Impact.** About 6,000 idle connections fit in 1 GiB, and far fewer right
+ after a busy period: the kill at 8,000 came after only about 2,400 new
+ connections, because memory from the 4,000 held just before was not yet
+ returned. Idle connections are common: connection pools, keep-alive, long
+ polling.
+- **Fix direction.** Hold buffers only while data moves: read straight into
+ the caller's buffer in the stream reader, and pool or shrink the copy
+ buffers.
+- **Verify.** Rerun EgressIdleConns for memory per connection, and EgressTCP
+ to check throughput stays at about 304 MB/s.
+
+### Finding 2: atunnel keeps only 2 spare connections per actor, so under load it opens a new connection for many requests
+
+Applies when the router runs with `--upstream-connect=false`.
+
+- **Observed.** In IngressH1, the ingress actors received 40,746 new
+ connections for 96,807 requests. From 1,000 to 2,000 requests per second,
+ atunnel's CPU per request rose from 170 µs to 500 µs, and atunnel used its
+ whole core. IngressH2 saw 18,940 new connections for 150,000 requests.
+- **Cause.** atunnel forwards to the actor with Go's default HTTP client
+ settings
+ ([`newProtocolMirrorTransport`](https://github.com/agent-substrate/substrate/blob/3a2ca62357485c4228fae466614ec98cf2caef69/internal/atunnel/ingress/ingress.go#L240-L242)),
+ which keep at most 2 unused connections per destination. When more than 2
+ requests to one actor overlap, the extra connections are opened and then
+ closed. Under load more requests overlap, so more connections are opened,
+ which costs more CPU. Each one is also a new connection into the actor's
+ sandbox.
+- **Impact.** Near capacity, a small rise in load can tip atunnel from healthy
+ to saturated.
+- **Fix direction.** Raise `MaxIdleConnsPerHost` on the two transports to about
+ the number of requests the router sends one actor at once.
+- **Verify.** Rerun IngressH1 and IngressH2 and compare the sink's
+ `ingress_actor_connections` and CPU per request at 2,000 and 4,000 per second.
+
+## Not measured here
+
+gVisor itself (a plain process stands in for the actor's program), the real
+router and egress gateway, actors being started and stopped during load, and
+the Connect scenario's opposite: long-lived CONNECT tunnels carrying many
+requests each, which is how the router uses them by default.
+
+## Appendix
+
+
+How to read results.txt
+
+Each scenario is a table with one row per rate. Example row, EgressIdleConns
+holding 4,000 connections:
+
+```
+offered achieved p50 p99 max errs | cores µs/unit rss-max heap-max gor cg-cores cg-mem | A cores C cores verdict
+ 4000 4000 1.86ms 7.72ms 20.64ms 0 | 0.17 1083 734.8MiB 611.9MiB 20135 0.17 786.3MiB | 0.10 0.10 OK
+```
+
+Columns before the first `|` describe the traffic. Columns between the two
+`|` describe atunnel only. The last columns describe the test setup.
+
+**Traffic**
+
+| Column | Meaning |
+|---|---|
+| offered | units of work sent per second; for EgressIdleConns, connections to hold |
+| achieved | units completed per second; for EgressIdleConns, connections opened |
+| p50 | typical time per unit: half took less |
+| p99 | slow-end time per unit: 99 in 100 took less |
+| max | slowest unit |
+| errs | units that failed |
+
+A unit is one request, except in Connect (one new connection with one request
+inside), EgressIdleConns (one connection opened), and EgressTCP (one 64 KB
+write).
+
+**atunnel's CPU**
+
+| Column | Meaning |
+|---|---|
+| cores | CPU atunnel used, in cores: 1.00 means one core busy the whole time. Reported by the operating system for the atunnel process. |
+| µs/unit | CPU time per unit of work, in microseconds: `cores` divided by the work done |
+| cg-cores | the same, measured by the Linux kernel for atunnel's cgroup, which holds only atunnel. This is the counter its CPU limit is enforced on. |
+
+`cores` and `cg-cores` measure the same CPU from two places, the process and
+its cgroup, so they match; `cg-cores` is the one to compare with a pod's CPU
+limit.
+
+**atunnel's memory**, from smallest to largest:
+
+| Column | Meaning |
+|---|---|
+| heap-max | memory atunnel's code allocated for its data, such as connection buffers, at its peak during the rung. May include data no longer used but not yet cleaned up. |
+| rss-max | all of the atunnel process's memory in RAM at its peak: the heap, the program, thread stacks, and memory freed but not yet returned to the system |
+| cg-mem | memory the kernel charges to atunnel's cgroup: everything in rss-max plus kernel memory used for atunnel, mainly network buffers. **This is checked against the memory limit and decides when atunnel is killed.** |
+
+**Other**
+
+| Column | Meaning |
+|---|---|
+| gor | goroutines, the lightweight threads of Go, the language atunnel is written in. atunnel runs about 5 per open outgoing connection, so this tracks work in progress. |
+| A cores | CPU used by whatever generates the traffic: the load generator for ingress, the egress actor inside actor-0's sandbox for egress |
+| C cores | CPU used by the far end: the ingress actors and the fake gateway |
+
+`A cores` and `C cores` check the test setup itself. If either is near its
+limit, the setup rather than atunnel is the bottleneck.
+
+**Verdicts**
+
+| Verdict | Meaning |
+|---|---|
+| OK | atunnel kept up with the offered rate |
+| SATURATED | atunnel used its whole CPU limit, so work queued and times grew |
+| SHORTFALL | atunnel fell behind without being at its CPU limit |
+| RIG-LIMITED | the traffic generator or the far end was the bottleneck, so atunnel's numbers are a lower bound |
+| ERRORS | some units failed |
+| DOWN | atunnel was killed during the rung, here for going over its memory limit; the scenario's higher rates were skipped |
+
+Under each table, "sink counters" are what the ingress actors and gateway
+received during the scenario: requests, new connections, and bytes. They
+should agree with what the generator sent. The last line of the report counts
+atunnel's deaths and OOM kills.
+
+
diff --git a/benchmarking/atunnel/runs/20261016-123913/results.txt b/benchmarking/atunnel/runs/20261016-123913/results.txt
new file mode 100644
index 0000000000..c7757a597a
--- /dev/null
+++ b/benchmarking/atunnel/runs/20261016-123913/results.txt
@@ -0,0 +1,72 @@
+=== IngressH1: router-style HTTPS requests over HTTP/1.1 keep-alive into ingress, proxied to the actor
+ unit: request; 10s per rung, first 2s not measured
+ offered achieved p50 p99 max errs | cores µs/unit rss-max heap-max gor cg-cores cg-mem | A cores C cores verdict
+ 1000 1000 0.79ms 4.18ms 43.45ms 0 | 0.17 173 35.4MiB 8.7MiB 139 0.17 31.6MiB | 0.18 0.10 OK
+ 2000 2000 124ms 362ms 476ms 0 | 1.00 503 75.2MiB 36.6MiB 1252 1.00 108.2MiB | 1.15 0.35 SATURATED (atunnel at 1.00 of 1 cores)
+ 4000 3237 3722ms 5241ms 5317ms 0 | 0.99 502 130.0MiB 71.8MiB 2370 0.99 195.7MiB | 1.08 0.38 SATURATED (atunnel at 0.99 of 1 cores)
+ 8000 2204 4335ms 5481ms 5702ms 0 | 0.99 691 166.1MiB 98.2MiB 2897 0.99 241.3MiB | 1.06 0.38 SATURATED (atunnel at 0.99 of 1 cores)
+ cores..cg-mem are atunnel alone (process B); A and C are the traffic generator and the far end
+ sink counters: {"gateway_bytes_in":0,"gateway_connections":0,"gateway_rejected":0,"gateway_requests":0,"gateway_tunnels":0,"gateway_tunnels_h2":0,"ingress_actor_connections":40746,"ingress_actor_requests":96807}
+
+=== IngressH2: IngressH1 over HTTP/2, many requests multiplexed on few connections
+ unit: request; 10s per rung, first 2s not measured
+ offered achieved p50 p99 max errs | cores µs/unit rss-max heap-max gor cg-cores cg-mem | A cores C cores verdict
+ 1000 1000 0.83ms 1.45ms 12.64ms 0 | 0.22 218 45.2MiB 12.9MiB 139 0.22 105.1MiB | 0.15 0.11 OK
+ 2000 2000 0.87ms 1.60ms 3.92ms 0 | 0.32 160 44.9MiB 12.9MiB 140 0.32 101.0MiB | 0.23 0.22 OK
+ 4000 4000 0.40ms 1.28ms 3.55ms 0 | 0.79 199 44.3MiB 12.4MiB 141 0.79 94.2MiB | 0.47 0.37 OK
+ 8000 8000 10.97ms 66.53ms 108ms 0 | 1.00 126 57.0MiB 21.0MiB 498 1.00 98.6MiB | 0.56 0.98 SATURATED (atunnel at 1.00 of 1 cores)
+ cores..cg-mem are atunnel alone (process B); A and C are the traffic generator and the far end
+ sink counters: {"gateway_bytes_in":0,"gateway_connections":0,"gateway_rejected":0,"gateway_requests":0,"gateway_tunnels":0,"gateway_tunnels_h2":0,"ingress_actor_connections":18940,"ingress_actor_requests":150000}
+
+=== Connect: per request: new TLS connection, CONNECT to the actor's port, one request inside
+ unit: tunnel; 10s per rung, first 2s not measured
+ offered achieved p50 p99 max errs | cores µs/unit rss-max heap-max gor cg-cores cg-mem | A cores C cores verdict
+ 1000 1000 2.97ms 15.45ms 38.71ms 0 | 0.98 982 44.7MiB 12.2MiB 141 0.98 87.1MiB | 1.12 0.21 SATURATED (atunnel at 0.98 of 1 cores)
+ 2000 2000 3885ms 6382ms 6484ms 0 | 1.00 895 72.1MiB 33.4MiB 764 1.00 112.5MiB | 1.08 0.25 SATURATED (atunnel at 1.00 of 1 cores)
+ 4000 1784 8420ms 8694ms 8861ms 0 | 1.00 1150 73.3MiB 36.0MiB 764 1.00 113.0MiB | 1.07 0.25 SATURATED (atunnel at 1.00 of 1 cores)
+ 8000 1238 8490ms 8702ms 8834ms 0 | 1.00 1657 72.7MiB 34.2MiB 770 1.00 110.2MiB | 1.07 0.25 SATURATED (atunnel at 1.00 of 1 cores)
+ cores..cg-mem are atunnel alone (process B); A and C are the traffic generator and the far end
+ sink counters: {"gateway_bytes_in":0,"gateway_connections":0,"gateway_rejected":0,"gateway_requests":0,"gateway_tunnels":0,"gateway_tunnels_h2":0,"ingress_actor_connections":74633,"ingress_actor_requests":74633}
+
+=== EgressHTTP: actor-style plain HTTP keep-alive out through egress; one CONNECT stream to the gateway per connection
+ unit: request; 10s per rung, first 2s not measured
+ offered achieved p50 p99 max errs | cores µs/unit rss-max heap-max gor cg-cores cg-mem | A cores C cores verdict
+ 1000 1000 0.71ms 1.52ms 2.29ms 0 | 0.17 166 45.5MiB 13.5MiB 142 0.17 78.1MiB | 0.18 0.11 OK
+ 2000 2000 0.70ms 2.03ms 38.67ms 0 | 0.40 199 45.8MiB 12.9MiB 147 0.40 78.6MiB | 0.35 0.25 OK
+ 4000 4000 0.37ms 14.29ms 41.22ms 0 | 0.69 174 45.5MiB 13.8MiB 164 0.69 78.2MiB | 0.63 0.43 OK
+ 8000 8000 2.97ms 170ms 240ms 0 | 1.00 125 106.6MiB 65.5MiB 857 1.00 141.0MiB | 1.01 0.69 SATURATED (atunnel at 1.00 of 1 cores)
+ cores..cg-mem are atunnel alone (process B); A and C are the traffic generator and the far end
+ sink counters: {"gateway_bytes_in":14100000,"gateway_connections":3,"gateway_rejected":0,"gateway_requests":150000,"gateway_tunnels":47104,"gateway_tunnels_h2":47104,"ingress_actor_connections":0,"ingress_actor_requests":0}
+
+=== EgressNewConnection: EgressHTTP with a new connection per request, so a new CONNECT stream per request
+ unit: request; 10s per rung, first 2s not measured
+ offered achieved p50 p99 max errs | cores µs/unit rss-max heap-max gor cg-cores cg-mem | A cores C cores verdict
+ 1000 1000 1.00ms 12.35ms 40.94ms 0 | 0.51 509 43.3MiB 12.0MiB 105 0.51 76.4MiB | 0.34 0.29 OK
+ 2000 2000 0.69ms 8.26ms 34.09ms 0 | 0.81 403 43.2MiB 12.0MiB 121 0.81 76.4MiB | 0.57 0.48 OK
+ 4000 4000 1.19ms 71.31ms 117ms 0 | 1.00 249 66.8MiB 52.2MiB 1360 1.00 101.6MiB | 0.82 0.66 SATURATED (atunnel at 1.00 of 1 cores)
+ 8000 8000 429ms 659ms 684ms 0 | 1.00 135 107.7MiB 63.5MiB 1403 1.00 142.7MiB | 0.95 0.75 SATURATED (atunnel at 1.00 of 1 cores)
+ cores..cg-mem are atunnel alone (process B); A and C are the traffic generator and the far end
+ sink counters: {"gateway_bytes_in":16950000,"gateway_connections":1,"gateway_rejected":0,"gateway_requests":150000,"gateway_tunnels":150000,"gateway_tunnels_h2":150000,"ingress_actor_connections":0,"ingress_actor_requests":0}
+
+=== EgressIdleConns: rungs are connection counts: open at 200/s through egress, one request each, then hold
+ unit: held connection; 10s per rung, first 0s not measured
+ offered achieved p50 p99 max errs | cores µs/unit rss-max heap-max gor cg-cores cg-mem | A cores C cores verdict
+ 1000 1000 1.84ms 5.68ms 8.43ms 0 | 0.12 1197 153.8MiB 156.6MiB 5105 0.12 191.9MiB | 0.07 0.07 OK
+ holding 1000: live heap 155.2MiB after GC, 158.9 KiB and 5.1 goroutines per connection
+ 2000 2000 1.87ms 9.74ms 25.38ms 0 | 0.15 1107 380.7MiB 308.4MiB 10115 0.15 424.1MiB | 0.09 0.08 OK
+ holding 2000: live heap 306.9MiB after GC, 157.1 KiB and 5.1 goroutines per connection
+ 4000 4000 1.86ms 7.72ms 20.64ms 0 | 0.17 1083 734.8MiB 611.9MiB 20135 0.17 786.3MiB | 0.10 0.10 OK
+ holding 4000: live heap 610.3MiB after GC, 156.2 KiB and 5.0 goroutines per connection
+ 8000 0 0.00ms 0.00ms 0.00ms 0 | 0.00 0 0.0MiB 0.0MiB 0 0.00 0.0MiB | 0.00 0.00 DOWN (atunnel stopped answering: Get "http://atunnel-worker:9090/stats": dial tcp 172.18.0.3:9090: connect: connection refused)
+ cores..cg-mem are atunnel alone (process B); A and C are the traffic generator and the far end
+ sink counters: {"gateway_bytes_in":366912,"gateway_connections":35,"gateway_rejected":0,"gateway_requests":9408,"gateway_tunnels":9408,"gateway_tunnels_h2":9408}
+
+=== EgressTCP: 4 connections write 64 KiB chunks as fast as they can through egress to a discarding gateway; one rung
+ unit: write; 10s per rung, first 2s not measured
+ offered achieved p50 p99 max errs | cores µs/unit rss-max heap-max gor cg-cores cg-mem | A cores C cores verdict
+ - 4642 0.00ms 11.50ms 41.06ms 0 | 1.00 216 33.3MiB 8.4MiB 115 1.00 27.6MiB | 0.05 1.21 SATURATED (atunnel at 1.00 of 1 cores)
+ throughput 304 MB/s
+ cores..cg-mem are atunnel alone (process B); A and C are the traffic generator and the far end
+ sink counters: {"gateway_bytes_in":3050766336,"gateway_connections":1,"gateway_rejected":0,"gateway_requests":0,"gateway_tunnels":4,"gateway_tunnels_h2":4,"ingress_actor_connections":0,"ingress_actor_requests":0}
+
+atunnel died 1 time(s), 1 of them OOM-killed at its 1g limit
diff --git a/benchmarking/locust/tests/ate_env.py b/benchmarking/locust/tests/ate_env.py
new file mode 100644
index 0000000000..a1b5a5ade0
--- /dev/null
+++ b/benchmarking/locust/tests/ate_env.py
@@ -0,0 +1,37 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Stub AteEnvUser declaration.
+
+The real load implementation lives in the boomer-Go worker at
+cmd/benchmarking/boomer-worker/; this Python class is declared only so the master's
+--class-picker enumerates the name. The Python worker container sets
+LOCUST_NO_ATE_ENV_USER=1 to skip loading this file, leaving boomer as the
+sole owner of AteEnvUser load.
+"""
+
+import os
+
+if os.environ.get("LOCUST_NO_ATE_ENV_USER") != "1":
+ from locust import User, task
+ from common.boomer_config import init_boomer_config
+
+ init_boomer_config()
+
+ class AteEnvUser(User):
+ host = "api.ate-system.svc.cluster.local:443"
+
+ @task
+ def noop(self) -> None:
+ pass
diff --git a/benchmarking/workloads/deploy.sh b/benchmarking/workloads/deploy.sh
index e994a68fd4..f7f73aae2f 100755
--- a/benchmarking/workloads/deploy.sh
+++ b/benchmarking/workloads/deploy.sh
@@ -40,7 +40,7 @@ SWEPERF_TEMPLATE_DIR="${MANIFEST_DIR}/sweperf/generated"
# matching locust tests) are not deployed by default. The glutton-storage
# template is added when a StorageClass is set, because it needs a CSI driver
# that not every cluster has.
-read -r -a TEMPLATES <<<"${WORKLOAD_TEMPLATES:-sleep glutton glutton-durdir-data glutton-durdir-full}"
+read -r -a TEMPLATES <<<"${WORKLOAD_TEMPLATES:-sleep glutton glutton-durdir-data glutton-durdir-full ate-env}"
# template_manifest prints the manifest path for ActorTemplate NAME.
template_manifest() {
@@ -73,6 +73,9 @@ WORKER_MEMORY=""
# The StorageClass for the glutton-storage external volume. Empty means the
# template is not deployed.
STORAGE_CLASS_NAME="${STORAGE_CLASS_NAME:-}"
+# The image for the ate-env benchmark ActorTemplate. Defaults to the pinned
+# release guest image; set ATE_ENV_IMAGE to override with a custom build.
+ATE_ENV_IMAGE="${ATE_ENV_IMAGE:-us-docker.pkg.dev/gke-substrate-release/benchmark/ate-env-guest@sha256:1d38934ef1dac71fcb1f1286de734e06ab2e19747279bccbd37bfe69e0d5bc72}"
# The address to which an instrumented actor container sends its telemetry.
# --otlp-endpoint sets it. Without the flag, resolve_otlp_endpoint reads the
# address that the control plane uses.
@@ -164,6 +167,7 @@ substitute() {
-e "s|\${ACTOR_MEMORY}|${ACTOR_MEMORY}|g" \
-e "s|\${ACTOR_CPU}|${ACTOR_CPU}|g" \
-e "s|\${WORKER_RESOURCES}|${worker_resources}|g" \
+ -e "s|\${ATE_ENV_IMAGE}|${ATE_ENV_IMAGE:-}|g" \
-e "s|\${STORAGE_CLASS_NAME}|${STORAGE_CLASS_NAME}|g" \
"${manifest}"
}
@@ -441,6 +445,11 @@ if [[ "${action}" == "deploy" && -z "${STORAGE_CLASS_NAME}" && " ${TEMPLATES[*]}
exit 1
fi
+if [[ "${action}" == "deploy" && -z "${ATE_ENV_IMAGE:-}" && " ${TEMPLATES[*]} " == *" ate-env "* ]]; then
+ echo "Error: the ate-env template needs ATE_ENV_IMAGE" >&2
+ exit 1
+fi
+
for entry in "${FAKE_ATELET_ENV[@]}"; do
if ! [[ "${entry}" =~ ^FAKE_ATELET_[A-Z_]+=.+$ ]]; then
echo "Error: --fake-atelet-env must be FAKE_ATELET_=VALUE, such as FAKE_ATELET_DELAY=500ms, got '${entry}'" >&2
diff --git a/benchmarking/workloads/manifests/ate-env-template.yaml.tmpl b/benchmarking/workloads/manifests/ate-env-template.yaml.tmpl
new file mode 100644
index 0000000000..e25388ef55
--- /dev/null
+++ b/benchmarking/workloads/manifests/ate-env-template.yaml.tmpl
@@ -0,0 +1,51 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# ActorTemplate (protojson-shaped ateapipb.ActorTemplate) for the ate-env
+# benchmark workload, created through the ate API by workloads/deploy.sh.
+metadata:
+ atespace: benchmark-workloads
+ name: ate-env
+workerSelector:
+ matchLabels:
+ workload: benchmark-worker
+containers:
+- name: guest
+ # The guest image is built from the agent-substrate/env repo
+ # https://github.com/agent-substrate/env/tree/1c08a7a578afd896373511a83beb242aec1425c0/cmd/ate-env-guest
+ # (2026-10-14, which pins base bash:5.3@sha256:61962062d969cb46dfc2bad061d36342406fa485f64f246aa7e95693ca07df1f
+ # in .ko.yaml):
+ #
+ # KO_DOCKER_REPO="gcr.io//ate-env-guest" ko build --bare --platform=linux/amd64 ./cmd/ate-env-guest
+ #
+ # Substituted by benchmarking/workloads/deploy.sh (which defaults ATE_ENV_IMAGE
+ # to us-docker.pkg.dev/gke-substrate-release/benchmark/ate-env-guest@sha256:1d38934ef1dac71fcb1f1286de734e06ab2e19747279bccbd37bfe69e0d5bc72;
+ # set ATE_ENV_IMAGE to override with a custom digest).
+ image: "${ATE_ENV_IMAGE}"
+ command:
+ - "/ko-app/ate-env-guest"
+ - "--listen=:80"
+ wakeupProbe:
+ httpGet:
+ path: /readyz
+ port: 80
+resources:
+ limits:
+ - name: cpu
+ quantity: "${ACTOR_CPU}"
+ - name: memory
+ quantity: "${ACTOR_MEMORY}"
+snapshotConfig:
+ preferredFidelity: SNAPSHOT_FIDELITY_MEMORY
+ storageLocation: gs://${BUCKET_NAME}/benchmark-workloads/ate-env/
diff --git a/benchmarking/workloads/manifests/glutton-storage-template.yaml.tmpl b/benchmarking/workloads/manifests/glutton-storage-template.yaml.tmpl
index 24237aa064..7145925134 100644
--- a/benchmarking/workloads/manifests/glutton-storage-template.yaml.tmpl
+++ b/benchmarking/workloads/manifests/glutton-storage-template.yaml.tmpl
@@ -24,9 +24,10 @@ workerSelector:
workload: benchmark-worker
volumes:
- name: external-storage
- externalVolumeTemplate:
- capacity: 1Gi
- storageClassName: ${STORAGE_CLASS_NAME}
+ external:
+ template:
+ capacity: 1Gi
+ storageClassName: ${STORAGE_CLASS_NAME}
containers:
- name: glutton-storage
image: ko://go.ate.dev/substrate/cmd/benchmarking/glutton
diff --git a/charts/substrate/README.md b/charts/substrate/README.md
index 4f04e61153..25d09ca30b 100644
--- a/charts/substrate/README.md
+++ b/charts/substrate/README.md
@@ -61,6 +61,7 @@ See `values.yaml` for the full set; the important keys:
| `postgres.clientCertificates.enabled` | `false` | Project separate owner and runtime login certificates into the API server |
| `.resources` | see `values.yaml` | Requests and limits for each container, e.g. `ateApi.resources` and `atenetRouter.agentgateway.resources` |
| `rustfs.enabled` | `true` | Deploy an in-cluster S3-compatible RustFS bucket for snapshots |
+| `ateApi.snapshotLocation` | `""` | Cluster-wide snapshot URI; defaults to `gs:///` with bundled RustFS |
| `atelet.storageBackend` | `s3` | Default snapshot backend, wired to RustFS when `rustfs.enabled=true` |
| `atelet.imageCredentialProviderConfig` | `""` | Host path to the kubelet credential provider config; set together with the bin directory |
| `atelet.imageCredentialProviderConfigYAML` | `""` | Credential provider config as text, rendered into a ConfigMap, for nodes where that file cannot be authored or must differ from the kubelet's; mutually exclusive with the host path |
@@ -184,16 +185,30 @@ to your operators' own principals and grant tenant access with `kubectl ate
create access-policy`; anyone who can mint the shared client's token otherwise has
global owner access. See [authorization](../../docs/authorization.md).
-Upgrade clients and components together: upstream now uses the
+Upstream renumbered protobuf fields, so older clients and binaries are not
+wire compatible. Upgrade clients and components together: upstream now uses the
`go.ate.dev/substrate` Go module, and the worker protocol includes dynamic
runtime registration and the unified suspend/pause workflow. Custom worker
images and integrations must use the matching API version.
-Every checkpoint now requires an object-storage URI, including a pause that
-keeps its snapshot local. Ensure actor templates have a usable
-`snapshotConfig.storageLocation` before pausing them. Upgrade the API server,
-atelet, and workers together for the checkpoint validation and per-actor
-sandbox asset paths.
+The pinned agentgateway image includes the updated `Actor.status`, actor-state,
+and `MintActorJwtRequest` bindings. If you override `images.agentgateway`, use
+these bindings too; older images cannot resolve actor routes or mint actor JWTs
+against this API.
+
+Every checkpoint requires an object-storage URI, including a pause that
+keeps its snapshot local. The chart sets the cluster-wide snapshot location
+from `rustfs.bucket` for bundled storage; external storage installs must set
+`ateApi.snapshotLocation`. Actor templates inherit this location. Existing
+`snapshotConfig.storageLocation` overrides require the
+`PerTemplateSnapshotLocation` preview gate; remove those overrides when using
+the cluster default without the gate. Upgrade the API server, atelet, and
+workers together for checkpoint validation and per-actor sandbox asset paths.
+
+`DeleteActorRequest.any_state` and the CLI's `--any-state` flag are removed;
+actors can now be deleted in any state. Atelet's TLS listener now accepts only
+the API server's SPIFFE identity and no longer exposes a host port. The chart
+configures that identity for the release namespace and service account.
The `ate-idp-server` binary serves actor JWT discovery and public signing keys
over HTTPS. The chart deploys it behind the `idp` Service (prefixed with the
diff --git a/charts/substrate/templates/ate-api-server-envvars.yaml b/charts/substrate/templates/ate-api-server-envvars.yaml
index 8b04a2015e..841005e810 100644
--- a/charts/substrate/templates/ate-api-server-envvars.yaml
+++ b/charts/substrate/templates/ate-api-server-envvars.yaml
@@ -20,11 +20,19 @@ limitations under the License.
{{- if not .Values.postgres.ownerConnectionStringSecretRef.name }}
{{- fail "postgres.ownerConnectionStringSecretRef.name is required" }}
{{- end }}
+{{- $snapshotLocation := .Values.ateApi.snapshotLocation }}
+{{- if and (not $snapshotLocation) .Values.rustfs.enabled }}
+{{- $snapshotLocation = printf "gs://%s/" .Values.rustfs.bucket }}
+{{- end }}
+{{- if and (not $snapshotLocation) (not (has "PerTemplateSnapshotLocation" .Values.preview)) (not (has "ALL" .Values.preview)) }}
+{{- fail "ateApi.snapshotLocation is required when rustfs.enabled=false unless PerTemplateSnapshotLocation preview is enabled" }}
+{{- end }}
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ .Values.ateApiServerEnvVarsConfigMap }}
namespace: {{ .Release.Namespace }}
data:
+ ATE_API_SNAPSHOT_LOCATION: {{ $snapshotLocation | quote }}
ATE_API_POSTGRES_SCHEMA: {{ .Values.postgres.schema | quote }}
ATE_API_ACTOR_JWT_ISSUER: {{ include "substrate.actorJWTIssuer" . | quote }}
diff --git a/charts/substrate/templates/ate-api-server.yaml b/charts/substrate/templates/ate-api-server.yaml
index c6a9409149..50b782710f 100644
--- a/charts/substrate/templates/ate-api-server.yaml
+++ b/charts/substrate/templates/ate-api-server.yaml
@@ -26,7 +26,7 @@ rules:
resources: ["pods"]
verbs: ["get", "watch", "list"]
- apiGroups: ["ate.dev"]
- resources: ["workerpools", "sandboxconfigs", "csidriverconfigs"]
+ resources: ["workerpools", "csidriverconfigs"]
verbs: ["get", "watch", "list"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
@@ -89,6 +89,12 @@ spec:
args:
- control
- --socket=/run/objectstore-plugin/control.sock
+ - --exchange-listen=:8445
+ - --actor-jwt-issuer=$(ATE_API_ACTOR_JWT_ISSUER)
+ - '--ateapi-server-name={{ include "substrate.fullname" (list "api" .) }}.{{ .Release.Namespace }}.svc'
+ ports:
+ - name: exchange
+ containerPort: 8445
startupProbe:
exec:
command:
@@ -103,6 +109,15 @@ spec:
resources:
{{ toYaml .Values.objectStorePlugin.control.resources | indent 10 }}
env:
+ - name: ATE_API_ACTOR_JWT_ISSUER
+ valueFrom:
+ configMapKeyRef:
+ name: {{ .Values.ateApiServerEnvVarsConfigMap }}
+ key: ATE_API_ACTOR_JWT_ISSUER
+ - name: POD_NAMESPACE
+ valueFrom:
+ fieldRef:
+ fieldPath: metadata.namespace
{{- include "substrate.storage.env" . | nindent 8 }}
{{- with include "substrate.otel.env" . }}
{{- . | trim | nindent 8 }}
@@ -110,6 +125,9 @@ spec:
volumeMounts:
- name: objectstore-plugin
mountPath: /run/objectstore-plugin
+ - { name: servicedns, mountPath: /run/servicedns.podcert.ate.dev, readOnly: true }
+ - { name: podidentity, mountPath: /run/podidentity.podcert.ate.dev, readOnly: true }
+ - { name: actor-id-jwt-jwks, mountPath: /run/actor-id-jwt-jwks, readOnly: true }
containers:
- name: ate-api-server
image: {{ include "substrate.componentImage" (list "ateapi" .) }}
@@ -118,6 +136,7 @@ spec:
args:
- {{ printf "--preview=%s" (join "," .Values.preview) | quote }}
- --objectstore-plugin-socket=/run/objectstore-plugin/control.sock
+ - --snapshot-location=@env
- "--grpc-listen-addr=0.0.0.0:443"
- "--grpc-server-cred-bundle=/run/servicedns.podcert.ate.dev/credential-bundle.pem"
- --config-file=/run/apiconfig.ate.dev/apiconfig.textproto
@@ -135,6 +154,7 @@ spec:
- "--default-egress-gateway-address={{ include "substrate.fullname" (list "atenet-egress" .) }}.{{ .Release.Namespace }}.svc:443"
- '--atelet-service-account={{ include "substrate.fullname" (list "atelet" .) }}'
- '--controller-spiffe-id=spiffe://cluster.local/ns/{{ .Release.Namespace }}/sa/{{ include "substrate.fullname" (list "ate-controller" .) }}'
+ - '--api-server-sidecar-spiffe-id=spiffe://cluster.local/ns/{{ .Release.Namespace }}/sa/{{ include "substrate.fullname" (list "ate-api-server" .) }}'
- '--egress-gateway-spiffe-id=spiffe://cluster.local/ns/{{ .Release.Namespace }}/sa/{{ include "substrate.fullname" (list "atenet-egress" .) }}'
- '--ingress-router-spiffe-id=spiffe://cluster.local/ns/{{ .Release.Namespace }}/sa/{{ include "substrate.fullname" (list "atenet-router" .) }}'
- "--atelet-client-cred-bundle=/run/podidentity.podcert.ate.dev/credential-bundle.pem"
@@ -236,6 +256,13 @@ spec:
name: actor-id-jwt-pool
items:
- { key: pool, path: pool.json }
+ - name: actor-id-jwt-jwks
+ projected:
+ sources:
+ - secret:
+ name: actor-id-jwt-pool
+ items:
+ - { key: jwks, path: jwks.json }
- name: actor-id-ca-pool
projected:
sources:
@@ -305,3 +332,7 @@ spec:
protocol: TCP
port: 443
targetPort: 443
+ - name: exchange
+ protocol: TCP
+ port: 8445
+ targetPort: exchange
diff --git a/charts/substrate/templates/ate-controller.yaml b/charts/substrate/templates/ate-controller.yaml
index 69789defd8..40a6414374 100644
--- a/charts/substrate/templates/ate-controller.yaml
+++ b/charts/substrate/templates/ate-controller.yaml
@@ -91,6 +91,10 @@ spec:
valueFrom:
fieldRef:
fieldPath: metadata.namespace
+{{- range $signal := list "logs" "traces" "metrics" }}
+ - name: ATE_WORKER_OTEL_{{ upper $signal }}_EXPORTER
+ value: {{ ternary "otlp" "none" (not (empty (include "substrate.otel.signalEndpoint" (list $signal $)))) | quote }}
+{{- end }}
{{- with include "substrate.otel.env" . }}
{{- . | trim | nindent 8 }}
{{- end }}
diff --git a/charts/substrate/templates/ate-system-networkpolicies.yaml b/charts/substrate/templates/ate-system-networkpolicies.yaml
index 72430b546c..b176195162 100644
--- a/charts/substrate/templates/ate-system-networkpolicies.yaml
+++ b/charts/substrate/templates/ate-system-networkpolicies.yaml
@@ -79,6 +79,13 @@ spec:
ports:
- protocol: TCP
port: 443
+ - from:
+ - podSelector:
+ matchLabels:
+ app: atelet
+ ports:
+ - protocol: TCP
+ port: exchange
---
# The bundled postgres is reached only by ate-api-server. The installer runs
# psql through `kubectl exec`, and the probes are exec probes, so neither needs
@@ -105,7 +112,7 @@ spec:
---
# atelet's gRPC port is called only by ate-api-server, which dials the atelet
# pod IP. Worker pods talk to atelet over its node-local Unix socket, not this
-# port. The hostPort mapping lands on the pod, so this policy covers it too.
+# port.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
diff --git a/charts/substrate/templates/atelet.yaml b/charts/substrate/templates/atelet.yaml
index 278150d72f..3c5d15bd20 100644
--- a/charts/substrate/templates/atelet.yaml
+++ b/charts/substrate/templates/atelet.yaml
@@ -187,6 +187,7 @@ spec:
{{- end }}
- --grpc-server-cred-bundle=/run/podidentity.podcert.ate.dev/credential-bundle.pem
- --client-ca-certs=/run/podidentity.podcert.ate.dev/trust-bundle.pem
+ - '--herder-allowed-spiffe-id=spiffe://cluster.local/ns/{{ .Release.Namespace }}/sa/{{ include "substrate.fullname" (list "ate-api-server" .) }}'
- --ateapi-ca-file=/run/servicedns.podcert.ate.dev/trust-bundle.pem
{{- with .Values.atelet.extraArgs }}
{{ toYaml . | indent 8 }}
@@ -229,7 +230,6 @@ spec:
ports:
- name: grpc
containerPort: 21998
- hostPort: 21998
- name: prometheus
containerPort: 9090
protocol: TCP
diff --git a/charts/substrate/tests/control_plane_test.yaml b/charts/substrate/tests/control_plane_test.yaml
index 593aad8908..b984949e8a 100644
--- a/charts/substrate/tests/control_plane_test.yaml
+++ b/charts/substrate/tests/control_plane_test.yaml
@@ -66,7 +66,7 @@ tests:
apiGroups: [ate.dev]
resources: [sandboxconfigs, workerpools]
verbs: [get, list, watch]
-- it: probes atelet health without exposing metrics on the host
+- it: probes atelet health without exposing host ports
template: atelet.yaml
documentSelector:
path: kind
@@ -77,7 +77,6 @@ tests:
content:
name: grpc
containerPort: 21998
- hostPort: 21998
- contains:
path: spec.template.spec.containers[0].ports
content:
@@ -96,6 +95,11 @@ tests:
- equal:
path: spec.template.spec.containers[0].livenessProbe.httpGet.port
value: health
+ - notExists:
+ path: spec.template.spec.containers[0].ports[0].hostPort
+ - contains:
+ path: spec.template.spec.containers[0].args
+ content: --herder-allowed-spiffe-id=spiffe://cluster.local/ns/custom-system/sa/custom-ate-api-server
- contains:
path: spec.template.spec.tolerations
content:
@@ -250,3 +254,28 @@ tests:
apiGroups: [""]
resources: [serviceaccounts]
verbs: [create, patch]
+
+- it: retains the controller sandbox configuration permissions
+ template: role.yaml
+ documentSelector:
+ path: kind
+ value: ClusterRole
+ asserts:
+ - contains:
+ path: rules
+ content:
+ apiGroups: [ate.dev]
+ resources: [sandboxconfigs]
+ verbs: [get, list, patch, watch]
+- it: limits API configuration reads to pools and CSI drivers
+ template: ate-api-server.yaml
+ documentSelector:
+ path: kind
+ value: ClusterRole
+ asserts:
+ - equal:
+ path: rules[1]
+ value:
+ apiGroups: [ate.dev]
+ resources: [workerpools, csidriverconfigs]
+ verbs: [get, watch, list]
diff --git a/charts/substrate/tests/snapshot_exchange_test.yaml b/charts/substrate/tests/snapshot_exchange_test.yaml
new file mode 100644
index 0000000000..b9d5cf0a20
--- /dev/null
+++ b/charts/substrate/tests/snapshot_exchange_test.yaml
@@ -0,0 +1,172 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+suite: Snapshot storage and credential exchange
+templates:
+- ate-api-server.yaml
+- ate-api-server-envvars.yaml
+- ate-system-networkpolicies.yaml
+- ate-controller.yaml
+release:
+ name: custom
+ namespace: custom-system
+tests:
+- it: defaults snapshots to the configured bundled bucket
+ template: ate-api-server-envvars.yaml
+ set:
+ rustfs.bucket: custom-snapshots
+ asserts:
+ - equal:
+ path: data.ATE_API_SNAPSHOT_LOCATION
+ value: gs://custom-snapshots/
+- it: uses an explicit external snapshot location
+ template: ate-api-server-envvars.yaml
+ set:
+ rustfs.enabled: false
+ ateApi.snapshotLocation: gs://external/prefix/
+ asserts:
+ - equal:
+ path: data.ATE_API_SNAPSHOT_LOCATION
+ value: gs://external/prefix/
+- it: requires a snapshot location for external storage
+ template: ate-api-server-envvars.yaml
+ set:
+ rustfs.enabled: false
+ asserts:
+ - failedTemplate:
+ errorMessage: ateApi.snapshotLocation is required when rustfs.enabled=false unless PerTemplateSnapshotLocation preview is enabled
+- it: permits template-only locations with the preview gate
+ template: ate-api-server-envvars.yaml
+ set:
+ rustfs.enabled: false
+ preview: [PerTemplateSnapshotLocation]
+ asserts:
+ - equal:
+ path: data.ATE_API_SNAPSHOT_LOCATION
+ value: ""
+- it: supplies the exchange identity and only public JWT keys
+ template: ate-api-server.yaml
+ documentSelector:
+ path: kind
+ value: Deployment
+ asserts:
+ - contains:
+ path: spec.template.spec.containers[0].args
+ content: --snapshot-location=@env
+ - contains:
+ path: spec.template.spec.containers[0].args
+ content: --api-server-sidecar-spiffe-id=spiffe://cluster.local/ns/custom-system/sa/custom-ate-api-server
+ - contains:
+ path: spec.template.spec.initContainers[0].args
+ content: --exchange-listen=:8445
+ - contains:
+ path: spec.template.spec.initContainers[0].args
+ content: --actor-jwt-issuer=$(ATE_API_ACTOR_JWT_ISSUER)
+ - contains:
+ path: spec.template.spec.initContainers[0].args
+ content: --ateapi-server-name=custom-api.custom-system.svc
+ - contains:
+ path: spec.template.spec.initContainers[0].env
+ content:
+ name: ATE_API_ACTOR_JWT_ISSUER
+ valueFrom:
+ configMapKeyRef:
+ name: ate-api-server-envvars
+ key: ATE_API_ACTOR_JWT_ISSUER
+ - contains:
+ path: spec.template.spec.initContainers[0].volumeMounts
+ content:
+ name: actor-id-jwt-jwks
+ mountPath: /run/actor-id-jwt-jwks
+ readOnly: true
+ - notContains:
+ path: spec.template.spec.initContainers[0].volumeMounts
+ content:
+ name: actor-id-jwt-pool
+ any: true
+ - contains:
+ path: spec.template.spec.volumes
+ content:
+ name: actor-id-jwt-jwks
+ projected:
+ sources:
+ - secret:
+ name: actor-id-jwt-pool
+ items:
+ - key: jwks
+ path: jwks.json
+- it: exposes the exchange through the API service
+ template: ate-api-server.yaml
+ documentSelector:
+ path: kind
+ value: Service
+ asserts:
+ - contains:
+ path: spec.ports
+ content:
+ name: exchange
+ protocol: TCP
+ port: 8445
+ targetPort: exchange
+- it: admits only atelet to the credential exchange
+ template: ate-system-networkpolicies.yaml
+ documentSelector:
+ path: metadata.name
+ value: custom-ate-api-server
+ asserts:
+ - equal:
+ path: spec.ingress[1]
+ value:
+ from:
+ - podSelector:
+ matchLabels:
+ app: atelet
+ ports:
+ - protocol: TCP
+ port: exchange
+- it: enables worker logs independently of control plane exporter settings
+ template: ate-controller.yaml
+ documentSelector:
+ path: kind
+ value: Deployment
+ set:
+ otel.endpoint: http://collector:4317
+ otel.metrics.enabled: false
+ asserts:
+ - contains:
+ path: spec.template.spec.containers[0].env
+ content:
+ name: ATE_WORKER_OTEL_LOGS_EXPORTER
+ value: otlp
+ - contains:
+ path: spec.template.spec.containers[0].env
+ content:
+ name: ATE_WORKER_OTEL_TRACES_EXPORTER
+ value: otlp
+ - contains:
+ path: spec.template.spec.containers[0].env
+ content:
+ name: ATE_WORKER_OTEL_METRICS_EXPORTER
+ value: none
+- it: leaves worker telemetry off without a configured endpoint
+ template: ate-controller.yaml
+ documentSelector:
+ path: kind
+ value: Deployment
+ asserts:
+ - contains:
+ path: spec.template.spec.containers[0].env
+ content:
+ name: ATE_WORKER_OTEL_LOGS_EXPORTER
+ value: none
diff --git a/charts/substrate/values.yaml b/charts/substrate/values.yaml
index 9dcd7f6473..1e9295454f 100644
--- a/charts/substrate/values.yaml
+++ b/charts/substrate/values.yaml
@@ -107,6 +107,10 @@ objectStorePlugin:
# Additional arguments appended to the ateapi defaults.
ateApi:
+ # Cluster-wide snapshot URI. Empty uses gs:/// when RustFS is
+ # enabled (the S3 backend also uses gs:// URIs). Required for external storage
+ # unless the PerTemplateSnapshotLocation preview gate is enabled.
+ snapshotLocation: ""
logLevel: info
authorization:
enabled: true
@@ -229,5 +233,5 @@ image:
images:
rustfs: rustfs/rustfs:1.0.0-beta.3@sha256:378642b05b7dcb4849fb77ebe6aca4ced1c3f66e7e504247df95a5c9018d3358
awsCli: amazon/aws-cli:2.17.0@sha256:643507c10ada7964ca6157b3d799f030b90577643da9955d319a77399ed80d73
- agentgateway: ghcr.io/kagent-dev/substrate/agentgateway:3554db38a0b2@sha256:309f6811560635471958db67b0f68dd083d84048dffc681967599d159ebfc83d
+ agentgateway: ghcr.io/kagent-dev/substrate/agentgateway:b736d06f88eb@sha256:39898817ed3e94b78570d20862fec06f32142fe8dee8691070128e74b5290407
busybox: busybox:1.36
diff --git a/cmd/ate-setup/internal/cmd/deploy.go b/cmd/ate-setup/internal/cmd/deploy.go
index bbee1390db..45b35a3014 100644
--- a/cmd/ate-setup/internal/cmd/deploy.go
+++ b/cmd/ate-setup/internal/cmd/deploy.go
@@ -100,7 +100,7 @@ var deployAteletCmd = &cobra.Command{
var deployAPIServerCmd = &cobra.Command{
Use: "apiserver",
Aliases: []string{"ate-apiserver"},
- Short: "Deploy ate-api-server only",
+ Short: "Deploy ate-api-server and ate-idp-server",
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error {
return env.DeployAteAPIServer(cmd.Context())
diff --git a/cmd/ate-setup/internal/config/config.go b/cmd/ate-setup/internal/config/config.go
index 126ff83c2e..a5040e7228 100644
--- a/cmd/ate-setup/internal/config/config.go
+++ b/cmd/ate-setup/internal/config/config.go
@@ -140,6 +140,13 @@ type Config struct {
// BucketName is the snapshot bucket demos are templated with.
BucketName string
+ // SnapshotLocation is ate-api-server's --snapshot-location
+ // (SNAPSHOT_LOCATION): the cluster-wide object-storage URI actor snapshots
+ // are stored under. Required unless the PerTemplateSnapshotLocation
+ // preview gate is enabled, which also lets a
+ // template's snapshotConfig.storageLocation override it.
+ SnapshotLocation string
+
// KODockerRepo is where ko pushes images (KO_DOCKER_REPO).
KODockerRepo string
// KODefaultPlatforms constrains ko's build platforms.
@@ -428,6 +435,7 @@ func (c *Config) ScriptEnv() []string {
"KUBECTL_CONTEXT": c.Context,
"KUBECONFIG": c.kubeconfigEnv,
"BUCKET_NAME": c.BucketName,
+ "SNAPSHOT_LOCATION": c.SnapshotLocation,
"KO_DOCKER_REPO": c.KODockerRepo,
"KO_DEFAULTPLATFORMS": c.KODefaultPlatforms,
"PROJECT_ID": c.ProjectID,
diff --git a/cmd/ate-setup/internal/config/config_test.go b/cmd/ate-setup/internal/config/config_test.go
index d229737dbf..19996c0b2e 100644
--- a/cmd/ate-setup/internal/config/config_test.go
+++ b/cmd/ate-setup/internal/config/config_test.go
@@ -86,6 +86,7 @@ func loadEnv(t *testing.T) {
"KUBECTL_CONTEXT",
"MEMORYSTORE_INSTANCE",
"PROJECT_ID",
+ "SNAPSHOT_LOCATION",
"ATE_API_POSTGRES_CLOUDSQL_INSTANCE",
} {
// Unset, not blanked. An exported empty variable is a value a channel
diff --git a/cmd/ate-setup/internal/config/kinddefaults_test.go b/cmd/ate-setup/internal/config/kinddefaults_test.go
index e34665bac5..1414372268 100644
--- a/cmd/ate-setup/internal/config/kinddefaults_test.go
+++ b/cmd/ate-setup/internal/config/kinddefaults_test.go
@@ -23,14 +23,15 @@ import (
// kindDefaults are the values a Kind install fills in for settings nothing
// else supplied. applyKindDefaultsResolved writes them onto Config.
var kindDefaults = map[string]string{
- "context": "kind-kind",
- "ko.dockerRepo": "localhost:5001",
- "storage.bucketName": "ate-snapshots",
- "ko.defaultPlatforms": "linux/" + runtimeGOARCH(),
+ "context": "kind-kind",
+ "ko.dockerRepo": "localhost:5001",
+ "storage.bucketName": "ate-snapshots",
+ "storage.snapshotLocation": "gs://ate-snapshots/",
+ "ko.defaultPlatforms": "linux/" + runtimeGOARCH(),
}
// The report has to show the values the install uses. A Kind install fills in
-// four settings that nothing else supplied, and they reach Config without
+// settings that nothing else supplied, and they reach Config without
// reaching Resolved -- so the report says "" (from default) for a value the
// install is about to use.
func TestKindDefaultsReachTheReport(t *testing.T) {
@@ -96,7 +97,7 @@ func TestKindInstallRecordsWhatItUsed(t *testing.T) {
// A replayed Kind record has to resolve to the install it recorded. It does,
// but not because the record carries the values: the record holds only
// kindCluster.enabled, and replaying it re-enters the same default block,
-// which derives the four values again.
+// which derives the values again.
//
// That makes the round-trip hold only while the defaults stay put. A record
// is meant to pin an install against a later release that ships different
@@ -129,6 +130,7 @@ func TestKindRecordReplaysToTheSameInstall(t *testing.T) {
{"KODockerRepo", replayed.KODockerRepo, cfg.KODockerRepo},
{"KODefaultPlatforms", replayed.KODefaultPlatforms, cfg.KODefaultPlatforms},
{"BucketName", replayed.BucketName, cfg.BucketName},
+ {"SnapshotLocation", replayed.SnapshotLocation, cfg.SnapshotLocation},
} {
if tc.got != tc.want {
t.Errorf("replaying the record gives %s = %q, the run it recorded used %q",
diff --git a/cmd/ate-setup/internal/config/load.go b/cmd/ate-setup/internal/config/load.go
index da6c4d3c8a..f819645b50 100644
--- a/cmd/ate-setup/internal/config/load.go
+++ b/cmd/ate-setup/internal/config/load.go
@@ -233,6 +233,7 @@ func buildConfig(root string, env map[string]string, r *Resolved) (*Config, erro
APIAuthzBootstrapOwners: bootstrapOwners,
BucketName: r.String("storage.bucketName"),
+ SnapshotLocation: r.String("storage.snapshotLocation"),
KODockerRepo: r.String("ko.dockerRepo"),
KODefaultPlatforms: r.String("ko.defaultPlatforms"),
ActorJWTAlgorithm: r.String("actorJWT.algorithm"),
@@ -358,6 +359,11 @@ func applyKindDefaults(r *Resolved) {
if !namedExplicitly(r, "storage.bucketName") {
r.set("storage.bucketName", "ate-snapshots", OriginKindProfile)
}
+ // Follows the bucket, so the location names the one the local rustfs
+ // creates rather than a GCS bucket from the developer's environment.
+ if !namedExplicitly(r, "storage.snapshotLocation") {
+ r.set("storage.snapshotLocation", "gs://"+r.String("storage.bucketName")+"/", OriginKindProfile)
+ }
}
// namedExplicitly reports whether a flag or configuration file supplied the
diff --git a/cmd/ate-setup/internal/config/load_test.go b/cmd/ate-setup/internal/config/load_test.go
index 905f062657..b48f8b3ebe 100644
--- a/cmd/ate-setup/internal/config/load_test.go
+++ b/cmd/ate-setup/internal/config/load_test.go
@@ -29,6 +29,7 @@ func TestKindKeepsExplicitValuesButNotAmbientOnes(t *testing.T) {
loadEnv(t)
t.Setenv("BUCKET_NAME", "ambient-cloud-bucket")
t.Setenv("KO_DEFAULTPLATFORMS", "linux/ambient")
+ t.Setenv("SNAPSHOT_LOCATION", "gs://ambient-cloud-bucket/")
t.Run("ambient environment is discarded", func(t *testing.T) {
cfg, err := Load(Options{Kind: true})
@@ -38,6 +39,9 @@ func TestKindKeepsExplicitValuesButNotAmbientOnes(t *testing.T) {
if cfg.BucketName != "ate-snapshots" {
t.Errorf("BucketName = %q, want ate-snapshots", cfg.BucketName)
}
+ if cfg.SnapshotLocation != "gs://ate-snapshots/" {
+ t.Errorf("SnapshotLocation = %q, want gs://ate-snapshots/", cfg.SnapshotLocation)
+ }
if want := "linux/" + runtime.GOARCH; cfg.KODefaultPlatforms != want {
t.Errorf("KODefaultPlatforms = %q, want %q", cfg.KODefaultPlatforms, want)
}
@@ -46,7 +50,7 @@ func TestKindKeepsExplicitValuesButNotAmbientOnes(t *testing.T) {
t.Run("a configuration file is kept", func(t *testing.T) {
path := filepath.Join(t.TempDir(), "install.yaml")
body := "apiVersion: " + FileAPIVersion + "\nkind: " + FileKind + "\n" +
- "storage:\n bucketName: from-file\n" +
+ "storage:\n bucketName: from-file\n snapshotLocation: gs://from-file/prefix/\n" +
"ko:\n defaultPlatforms: linux/from-file\n"
if err := os.WriteFile(path, []byte(body), 0o644); err != nil {
t.Fatal(err)
@@ -62,6 +66,9 @@ func TestKindKeepsExplicitValuesButNotAmbientOnes(t *testing.T) {
if cfg.BucketName != "from-file" {
t.Errorf("BucketName = %q, want from-file", cfg.BucketName)
}
+ if cfg.SnapshotLocation != "gs://from-file/prefix/" {
+ t.Errorf("SnapshotLocation = %q, want gs://from-file/prefix/", cfg.SnapshotLocation)
+ }
if cfg.KODefaultPlatforms != "linux/from-file" {
t.Errorf("KODefaultPlatforms = %q, want linux/from-file", cfg.KODefaultPlatforms)
}
diff --git a/cmd/ate-setup/internal/config/setting.go b/cmd/ate-setup/internal/config/setting.go
index 216425f2c6..2a5fd61496 100644
--- a/cmd/ate-setup/internal/config/setting.go
+++ b/cmd/ate-setup/internal/config/setting.go
@@ -149,6 +149,10 @@ var Registry = []Setting{
Key: "storage.bucketName", Env: "BUCKET_NAME", Flag: "storage-bucket-name", Kind: KindString,
Usage: "Snapshot bucket the demos are templated with",
},
+ {
+ Key: "storage.snapshotLocation", Env: "SNAPSHOT_LOCATION", Flag: "snapshot-location", Kind: KindString,
+ Usage: "Cluster-wide object-storage URI ate-api-server stores snapshots under, for example gs://BUCKET/PREFIX/ (required if the PerTemplateSnapshotLocation preview gate is not enabled)",
+ },
{
Key: "atenet.dataplane", Env: "ATE_ATENET_DATAPLANE", Flag: "atenet-dataplane", Kind: KindString,
diff --git a/cmd/ate-setup/internal/demos/counter/counter.go b/cmd/ate-setup/internal/demos/counter/counter.go
index a424529a2b..ec00f4fcdf 100644
--- a/cmd/ate-setup/internal/demos/counter/counter.go
+++ b/cmd/ate-setup/internal/demos/counter/counter.go
@@ -40,9 +40,10 @@ func (d *demo) externalVolumeValues(e *steps.Env) map[string]string {
"EXTERNAL_VOLUME_MOUNTS": " - name: external-data\n" +
" mountPath: /external-data",
"EXTERNAL_VOLUMES": "- name: external-data\n" +
- " externalVolumeTemplate:\n" +
- " capacity: 1Gi\n" +
- " storageClassName: " + sc,
+ " external:\n" +
+ " template:\n" +
+ " capacity: 1Gi\n" +
+ " storageClassName: " + sc,
}
}
diff --git a/cmd/ate-setup/internal/demos/counter/counter_test.go b/cmd/ate-setup/internal/demos/counter/counter_test.go
index 05630230c2..2cfdd393b2 100644
--- a/cmd/ate-setup/internal/demos/counter/counter_test.go
+++ b/cmd/ate-setup/internal/demos/counter/counter_test.go
@@ -69,7 +69,8 @@ func TestExternalVolumeRenders(t *testing.T) {
for _, want := range []string{
"--validate-existing-file-path=/external-data/test.txt",
"mountPath: /external-data",
- "externalVolumeTemplate:",
+ "external:",
+ "template:",
tc.wantSC,
} {
if !strings.Contains(string(manifest), want) {
diff --git a/cmd/ate-setup/internal/steps/apienv.go b/cmd/ate-setup/internal/steps/apienv.go
index f59b560d1b..2675bdc954 100644
--- a/cmd/ate-setup/internal/steps/apienv.go
+++ b/cmd/ate-setup/internal/steps/apienv.go
@@ -29,6 +29,8 @@ import (
"go.ate.dev/substrate/cmd/ate-setup/internal/kube"
"go.ate.dev/substrate/cmd/ate-setup/internal/log"
"go.ate.dev/substrate/internal/installdefaults"
+ "go.ate.dev/substrate/internal/preview"
+ "go.ate.dev/substrate/internal/resources"
)
// envHashAnnotation carries a digest of the apiserver's environment on the pod
@@ -40,7 +42,7 @@ const envHashAnnotation = "ate.dev/env-hash"
// store: both DSNs and the schema into the ate-api-server-secret-envvars Secret,
// stable roles and Cloud SQL settings into the ConfigMap,
// and an external server CA into postgres-server-ca. The ConfigMap also
-// carries the actor JWT issuer.
+// carries the actor JWT issuer and the snapshot location.
//
// ate-api-server.yaml pulls both in through optional envFrom sources and
// resolves the PostgreSQL connection, role, and schema flags from
@@ -52,6 +54,10 @@ const envHashAnnotation = "ate.dev/env-hash"
// applies no Deployment.
func (e *Env) CreateAPIServerEnvVars(ctx context.Context) error {
log.Step("create_api_server_env_vars")
+ snapshotLocation, err := clusterSnapshotLocation(e.Cfg.SnapshotLocation, e.Cfg.PreviewGates)
+ if err != nil {
+ return err
+ }
if err := e.Kube.EnsureNamespace(ctx, e.Namespace()); err != nil {
return err
}
@@ -129,6 +135,9 @@ func (e *Env) CreateAPIServerEnvVars(ctx context.Context) error {
configVars["ATE_API_POSTGRES_POOL_MAX_CONNS"] = poolMaxConns
}
configVars["ATE_API_ACTOR_JWT_ISSUER"] = e.Cfg.ActorJWTIssuer
+ if snapshotLocation != "" {
+ configVars["ATE_API_SNAPSHOT_LOCATION"] = snapshotLocation
+ }
if err := e.Kube.ApplyConfigMap(ctx, e.Namespace(), ConfigMapAPIEnvVars, configVars); err != nil {
return err
}
@@ -170,6 +179,31 @@ func sharedBootstrapOwnerWarning(enabled bool, bootstrapOwners, namespace string
return ""
}
+// clusterSnapshotLocation validates location and returns it as ate-api-server's
+// --snapshot-location, the cluster-wide default storage location for actor
+// snapshots. It is used whether or not the PerTemplateSnapshotLocation preview
+// gate is enabled; with the gate, an actor template's
+// snapshotConfig.storageLocation overrides it. It may be empty only when the
+// gate is enabled, in which case every template must set its own.
+func clusterSnapshotLocation(location string, gates []string) (string, error) {
+ if location != "" {
+ if err := resources.ValidateSnapshotLocation(location); err != nil {
+ return "", fmt.Errorf("invalid --snapshot-location: %w", err)
+ }
+ return location, nil
+ }
+ // Snapshot location should always be set by --snapshot-location flag when not using preview mode.
+ if !previewGateEnabled(gates, preview.GatePerTemplateSnapshotLocation) {
+ return "", fmt.Errorf("--snapshot-location (SNAPSHOT_LOCATION) must be set unless --preview=%s is enabled", preview.GatePerTemplateSnapshotLocation)
+ }
+ return "", nil
+}
+
+// previewGateEnabled reports whether gates, a --preview value, enables gate.
+func previewGateEnabled(gates []string, gate preview.Gate) bool {
+ return slices.Contains(gates, preview.All) || slices.Contains(gates, string(gate))
+}
+
// buildAPIServerEnvVars is the Secret payload. ate-api-server takes both
// connection strings and the schema from it, and exits on an empty schema; an
// unrecognized key here reaches the container as a stray environment variable.
diff --git a/cmd/ate-setup/internal/steps/apienv_test.go b/cmd/ate-setup/internal/steps/apienv_test.go
index 2371e48dc5..a78d3f9378 100644
--- a/cmd/ate-setup/internal/steps/apienv_test.go
+++ b/cmd/ate-setup/internal/steps/apienv_test.go
@@ -108,6 +108,7 @@ func TestCreateAPIServerEnvVarsPostgresIdentities(t *testing.T) {
},
} {
t.Run(tc.name, func(t *testing.T) {
+ tc.cfg.SnapshotLocation = "gs://ate-snapshots/"
e := &Env{Cfg: &tc.cfg, Kube: fakeKube(t,
&corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: NamespaceAteSystem}},
&corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: ConfigMapAPIEnvVars, Namespace: NamespaceAteSystem}},
@@ -142,6 +143,7 @@ func TestCreateAPIServerEnvVarsPostgresIdentities(t *testing.T) {
func TestCreateAPIServerEnvVarsRejectsCustomBundledIdentity(t *testing.T) {
cfg := config.Config{PostgresReadWriteRole: "tenant_readwrite", PostgresOwnerRole: "tenant_owner"}
+ cfg.SnapshotLocation = "gs://ate-snapshots/"
e := &Env{Cfg: &cfg, Kube: fakeKube(t,
&corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: NamespaceAteSystem}},
&corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: ConfigMapAPIEnvVars, Namespace: NamespaceAteSystem}},
@@ -175,7 +177,8 @@ func TestCreateAPIServerEnvVarsKeepsExternalPostgres(t *testing.T) {
t.Run(tc.name, func(t *testing.T) {
cfg := config.Config{
PostgresReadWriteRole: config.DefaultPostgresReadWriteRole, PostgresOwnerRole: config.DefaultPostgresOwnerRole,
- CloudSQL: tc.cloudSQL,
+ CloudSQL: tc.cloudSQL,
+ SnapshotLocation: "gs://ate-snapshots/",
}
e := &Env{Cfg: &cfg, Kube: fakeKube(t,
&corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: NamespaceAteSystem}},
@@ -227,7 +230,7 @@ func TestCreateAPIServerEnvVarsKeepsOwnerLogin(t *testing.T) {
},
} {
t.Run(tc.name, func(t *testing.T) {
- cfg := config.Config{PostgresReadWriteConnectionString: readWrite, PostgresOwnerConnectionString: tc.owner}
+ cfg := config.Config{PostgresReadWriteConnectionString: readWrite, PostgresOwnerConnectionString: tc.owner, SnapshotLocation: "gs://ate-snapshots/"}
e := &Env{Cfg: &cfg, Kube: fakeKube(t,
&corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: NamespaceAteSystem}},
&corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: ConfigMapAPIEnvVars, Namespace: NamespaceAteSystem}},
@@ -265,6 +268,7 @@ func TestCreateAPIServerEnvVarsPoolSize(t *testing.T) {
PostgresReadWriteConnectionString: "postgres://runtime@postgres/atepg",
PostgresPoolMaxConns: "20",
}
+ cfg.SnapshotLocation = "gs://ate-snapshots/"
e := &Env{Cfg: &cfg, Kube: fakeKube(t,
&corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: NamespaceAteSystem}},
&corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: ConfigMapAPIEnvVars, Namespace: NamespaceAteSystem}},
@@ -293,6 +297,7 @@ func TestCreateAPIServerEnvVarsActorJWTIssuer(t *testing.T) {
PostgresReadWriteConnectionString: "postgres://runtime@postgres/atepg",
ActorJWTIssuer: "https://idp.example.com/prod",
}
+ cfg.SnapshotLocation = "gs://ate-snapshots/"
e := &Env{Cfg: &cfg, Kube: fakeKube(t,
&corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: NamespaceAteSystem}},
&corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: ConfigMapAPIEnvVars, Namespace: NamespaceAteSystem}},
@@ -353,6 +358,7 @@ func TestCreateAPIServerEnvVarsAdoptsPostgresIdentity(t *testing.T) {
},
} {
t.Run(tc.name, func(t *testing.T) {
+ tc.cfg.SnapshotLocation = "gs://ate-snapshots/"
e := &Env{Cfg: &tc.cfg, Kube: fakeKube(t,
&corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: NamespaceAteSystem}},
&corev1.ConfigMap{
@@ -530,7 +536,7 @@ func TestAnnotateAPIServerEnvHash(t *testing.T) {
// Deployment, patches the running one.
func TestOnlyAStandaloneEnvUpdateRollsTheAPIServer(t *testing.T) {
newEnv := func(t *testing.T) *Env {
- return &Env{Cfg: &config.Config{PostgresReadWriteConnectionString: "postgres://runtime@postgres/atepg"},
+ return &Env{Cfg: &config.Config{PostgresReadWriteConnectionString: "postgres://runtime@postgres/atepg", SnapshotLocation: "gs://ate-snapshots/"},
Kube: fakeKube(t,
&corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: NamespaceAteSystem}},
&corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: ConfigMapAPIEnvVars, Namespace: NamespaceAteSystem}},
@@ -699,6 +705,7 @@ func TestCreateAPIServerEnvVarsOmitsAuthz(t *testing.T) {
PostgresOwnerRole: config.DefaultPostgresOwnerRole,
APIExperimentalEnableAuthz: true,
APIAuthzBootstrapOwners: "system:serviceaccount:ate-system:ate-client",
+ SnapshotLocation: "gs://ate-snapshots/",
}, Kube: fakeKube(t,
&corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: NamespaceAteSystem}},
apiServerEnvVarsConfigMap(map[string]string{}),
@@ -750,3 +757,70 @@ func TestSharedBootstrapOwnerWarning(t *testing.T) {
})
}
}
+
+func TestCreateAPIServerEnvVarsSnapshotLocation(t *testing.T) {
+ tests := []struct {
+ name string
+ location string
+ gates []string
+ want string
+ wantErr string
+ }{{
+ name: "set",
+ location: "gs://ate-snapshots/prefix/",
+ want: "gs://ate-snapshots/prefix/",
+ }, {
+ name: "set with the gate on",
+ location: "gs://ate-snapshots/prefix/",
+ gates: []string{"PerTemplateSnapshotLocation"},
+ want: "gs://ate-snapshots/prefix/",
+ }, {
+ name: "unset when the gate is on",
+ gates: []string{"PerTemplateSnapshotLocation"},
+ }, {
+ name: "unset when every gate is on",
+ gates: []string{"ALL"},
+ }, {
+ name: "an error when unset and the gate is off",
+ gates: []string{"ExternalVolumes"},
+ wantErr: "--snapshot-location (SNAPSHOT_LOCATION) must be set unless --preview=PerTemplateSnapshotLocation is enabled",
+ }, {
+ name: "an error when invalid",
+ location: "gs:///no-bucket",
+ wantErr: "invalid --snapshot-location",
+ }}
+ for _, tc := range tests {
+ t.Run(tc.name, func(t *testing.T) {
+ cfg := config.Config{
+ PostgresReadWriteConnectionString: "postgres://runtime@postgres/atepg",
+ SnapshotLocation: tc.location,
+ PreviewGates: tc.gates,
+ }
+ e := &Env{Cfg: &cfg, Kube: fakeKube(t,
+ &corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: NamespaceAteSystem}},
+ &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: ConfigMapAPIEnvVars, Namespace: NamespaceAteSystem}},
+ &corev1.Secret{ObjectMeta: metav1.ObjectMeta{Name: SecretAPIEnvVars, Namespace: NamespaceAteSystem}},
+ )}
+ err := e.CreateAPIServerEnvVars(t.Context())
+ if tc.wantErr != "" {
+ if err == nil || !strings.Contains(err.Error(), tc.wantErr) {
+ t.Fatalf("CreateAPIServerEnvVars() error = %v, want %q", err, tc.wantErr)
+ }
+ return
+ }
+ if err != nil {
+ t.Fatal(err)
+ }
+ cm, err := e.Kube.GetConfigMap(t.Context(), NamespaceAteSystem, ConfigMapAPIEnvVars)
+ if err != nil {
+ t.Fatal(err)
+ }
+ got, ok := cm.Data["ATE_API_SNAPSHOT_LOCATION"]
+ if tc.want == "" && ok {
+ t.Errorf("ATE_API_SNAPSHOT_LOCATION = %q, want it unset", got)
+ } else if got != tc.want {
+ t.Errorf("ATE_API_SNAPSHOT_LOCATION = %q, want %q", got, tc.want)
+ }
+ })
+ }
+}
diff --git a/cmd/ate-setup/internal/steps/create.go b/cmd/ate-setup/internal/steps/create.go
index 7c1c9127c2..705365b25e 100644
--- a/cmd/ate-setup/internal/steps/create.go
+++ b/cmd/ate-setup/internal/steps/create.go
@@ -220,7 +220,7 @@ const inClusterIssuer = "https://kubernetes.default.svc"
// OIDC discovery handshake. A GKE or otherwise external issuer needs neither.
func buildAuthenticationConfig(issuer string) string {
config := fmt.Sprintf(
- "actorIdentityJWTProvider: kubernetes\njwtProviders:\n- name: kubernetes\n issuer: %s\n audiences: [api.ate-system.svc]\n",
+ "jwtProviders:\n- name: kubernetes\n issuer: %s\n audiences: [api.ate-system.svc]\n",
issuer)
switch issuer {
case inClusterIssuer, inClusterIssuer + ".cluster.local":
diff --git a/cmd/ate-setup/internal/steps/create_test.go b/cmd/ate-setup/internal/steps/create_test.go
index c818bfcd58..0cb3ece7e8 100644
--- a/cmd/ate-setup/internal/steps/create_test.go
+++ b/cmd/ate-setup/internal/steps/create_test.go
@@ -55,9 +55,8 @@ func TestBuildAPIServerEnvVars(t *testing.T) {
}
}
-// The expected strings here are what the shell installer's
-// create_api_authentication_config produced, so a regression shows up as a
-// diff rather than as an apiserver that silently trusts the wrong issuer.
+// The expected strings are the whole rendered file, so a regression shows up
+// as a diff rather than as an apiserver that silently trusts the wrong issuer.
func TestBuildAuthenticationConfig(t *testing.T) {
for _, tc := range []struct {
name string
@@ -67,8 +66,7 @@ func TestBuildAuthenticationConfig(t *testing.T) {
{
name: "external issuer needs no discovery credentials",
issuer: "https://container.googleapis.com/v1/projects/p/locations/l/clusters/c",
- want: "actorIdentityJWTProvider: kubernetes\n" +
- "jwtProviders:\n" +
+ want: "jwtProviders:\n" +
"- name: kubernetes\n" +
" issuer: https://container.googleapis.com/v1/projects/p/locations/l/clusters/c\n" +
" audiences: [api.ate-system.svc]",
@@ -76,8 +74,7 @@ func TestBuildAuthenticationConfig(t *testing.T) {
{
name: "in-cluster issuer projects the service account CA and token",
issuer: inClusterIssuer,
- want: "actorIdentityJWTProvider: kubernetes\n" +
- "jwtProviders:\n" +
+ want: "jwtProviders:\n" +
"- name: kubernetes\n" +
" issuer: https://kubernetes.default.svc\n" +
" audiences: [api.ate-system.svc]\n" +
@@ -87,8 +84,7 @@ func TestBuildAuthenticationConfig(t *testing.T) {
{
name: "cluster.local form is treated as in-cluster too",
issuer: inClusterIssuer + ".cluster.local",
- want: "actorIdentityJWTProvider: kubernetes\n" +
- "jwtProviders:\n" +
+ want: "jwtProviders:\n" +
"- name: kubernetes\n" +
" issuer: https://kubernetes.default.svc.cluster.local\n" +
" audiences: [api.ate-system.svc]\n" +
diff --git a/cmd/ate-setup/internal/steps/deploy.go b/cmd/ate-setup/internal/steps/deploy.go
index c8defae1af..c116307137 100644
--- a/cmd/ate-setup/internal/steps/deploy.go
+++ b/cmd/ate-setup/internal/steps/deploy.go
@@ -74,6 +74,11 @@ func (e *Env) DeployAteSystem(ctx context.Context, opts DeployOptions) error {
if err := e.CheckCSIDriver(opts.SetupCSI); err != nil {
return err
}
+ // The snapshot location is only written into ate-api-server's ConfigMap
+ // partway through, so check it before the cluster is touched.
+ if _, err := clusterSnapshotLocation(e.Cfg.SnapshotLocation, e.Cfg.PreviewGates); err != nil {
+ return err
+ }
if err := e.requireNoWorkerPods(ctx); err != nil {
return err
}
@@ -279,10 +284,15 @@ func setPodcertWorkersPerSigner(objs []*unstructured.Unstructured, workers int)
return unstructured.SetNestedSlice(dep.Object, containers, "spec", "template", "spec", "containers")
}
-// DeployAteAPIServer redeploys only ate-api-server.
+// DeployAteAPIServer redeploys ate-api-server and ate-idp-server.
func (e *Env) DeployAteAPIServer(ctx context.Context) error {
log.Step("deploy_ate_apiserver")
+ // Checked before the CRDs, namespace and Secrets go in, rather than when
+ // the ConfigMap is written.
+ if _, err := clusterSnapshotLocation(e.Cfg.SnapshotLocation, e.Cfg.PreviewGates); err != nil {
+ return err
+ }
if err := e.EnsureCRDs(ctx); err != nil {
return err
}
@@ -318,7 +328,17 @@ func (e *Env) DeployAteAPIServer(ctx context.Context) error {
if err := e.reconcileCloudSQLProxySidecar(ctx); err != nil {
return err
}
- return e.Kube.RolloutStatus(ctx, kube.KindDeployment, e.Namespace(), "ate-api-server", e.Cfg.RolloutTimeout)
+ // ate-idp-server shares only the actor JWT pool with ate-api-server. It
+ // comes along so that redeploying an older release rolls it back too.
+ if err := e.deployIDPServer(ctx); err != nil {
+ return err
+ }
+ for _, name := range []string{"ate-api-server", "ate-idp-server"} {
+ if err := e.Kube.RolloutStatus(ctx, kube.KindDeployment, e.Namespace(), name, e.Cfg.RolloutTimeout); err != nil {
+ return err
+ }
+ }
+ return nil
}
// apiServerManifestPath is the ate-api-server manifest for the environment.
diff --git a/cmd/ate-setup/internal/steps/deploy_test.go b/cmd/ate-setup/internal/steps/deploy_test.go
index 3777fceb9b..12c504fdb8 100644
--- a/cmd/ate-setup/internal/steps/deploy_test.go
+++ b/cmd/ate-setup/internal/steps/deploy_test.go
@@ -215,6 +215,7 @@ func TestComponentManifestsMatchSystemBundle(t *testing.T) {
kind, name, path string
}{
{"Deployment", "ate-api-server", e.apiServerManifestPath()},
+ {"Deployment", "ate-idp-server", e.idpServerManifestPath()},
{"DaemonSet", "atelet", e.ateletManifestPath()},
} {
want := findObject(bundle, component.kind, component.name)
diff --git a/cmd/ate-setup/internal/steps/idpserver.go b/cmd/ate-setup/internal/steps/idpserver.go
index ba74c4d21b..c27d6e5270 100644
--- a/cmd/ate-setup/internal/steps/idpserver.go
+++ b/cmd/ate-setup/internal/steps/idpserver.go
@@ -16,6 +16,7 @@ package steps
import (
"bytes"
+ "context"
"errors"
"fmt"
"strings"
@@ -26,6 +27,30 @@ import (
const idpIssuerFlag = "--issuer"
+// idpServerManifestPath is the ate-idp-server manifest.
+func (e *Env) idpServerManifestPath() string {
+ return e.Cfg.Manifest("ate-idp-server.yaml")
+}
+
+// deployIDPServer applies ate-idp-server on its own.
+func (e *Env) deployIDPServer(ctx context.Context) error {
+ manifest, err := e.renderIDPServer()
+ if err != nil {
+ return err
+ }
+ return e.ResolveAndApplyBytes(ctx, manifest)
+}
+
+// renderIDPServer renders the ate-idp-server manifest with the install's
+// issuer, as the system bundle carries it.
+func (e *Env) renderIDPServer() ([]byte, error) {
+ manifest, err := e.render(e.idpServerManifestPath())
+ if err != nil {
+ return nil, err
+ }
+ return setIDPServerIssuer(manifest, e.Cfg.ActorJWTIssuer)
+}
+
// setIDPServerIssuer sets --issuer on the ate-idp-server container in manifest,
// replacing any --issuer it already carries.
func setIDPServerIssuer(manifest []byte, issuer string) ([]byte, error) {
diff --git a/cmd/ate-setup/internal/steps/idpserver_test.go b/cmd/ate-setup/internal/steps/idpserver_test.go
index fadaaee2b7..23dafdec43 100644
--- a/cmd/ate-setup/internal/steps/idpserver_test.go
+++ b/cmd/ate-setup/internal/steps/idpserver_test.go
@@ -96,6 +96,32 @@ func TestIDPServerGetsTheIssuerAndSharesTheAPIServerPool(t *testing.T) {
}
}
+// deploy apiserver applies ate-idp-server on its own, so it has to set the
+// issuer as deploy ate-system does.
+func TestRenderIDPServerSetsTheIssuer(t *testing.T) {
+ const issuer = "https://idp.example.com"
+ e := &Env{Cfg: &config.Config{Root: repoRoot(t), ActorJWTIssuer: issuer}}
+ rendered, err := e.renderIDPServer()
+ if err != nil {
+ t.Fatalf("renderIDPServer: %v", err)
+ }
+ objs, err := kube.DecodeManifestBytes(rendered)
+ if err != nil {
+ t.Fatalf("DecodeManifestBytes: %v", err)
+ }
+ idp := findObject(objs, "Deployment", "ate-idp-server")
+ if idp == nil {
+ t.Fatal("renderIDPServer has no deployment/ate-idp-server")
+ }
+ var dep appsv1.Deployment
+ if err := runtime.DefaultUnstructuredConverter.FromUnstructured(idp.Object, &dep); err != nil {
+ t.Fatal(err)
+ }
+ if args := dep.Spec.Template.Spec.Containers[0].Args; !slices.Contains(args, "--issuer="+issuer) {
+ t.Errorf("ate-idp-server args = %v, want --issuer=%s", args, issuer)
+ }
+}
+
func TestSetIDPServerIssuerWithoutTheDeployment(t *testing.T) {
manifest := []byte("apiVersion: apps/v1\nkind: Deployment\nmetadata:\n name: ate-api-server\n")
if _, err := setIDPServerIssuer(manifest, "https://idp.example.com"); err == nil {
diff --git a/cmd/ate-setup/internal/steps/overlay_test.go b/cmd/ate-setup/internal/steps/overlay_test.go
index 9f4b270918..be583dbef0 100644
--- a/cmd/ate-setup/internal/steps/overlay_test.go
+++ b/cmd/ate-setup/internal/steps/overlay_test.go
@@ -446,6 +446,11 @@ func TestRenderCordonControlPlane(t *testing.T) {
path: func(e *Env) string { return e.apiServerManifestPath() },
want: []string{"ate-api-server"},
},
+ {
+ name: "idp server file",
+ path: func(e *Env) string { return e.idpServerManifestPath() },
+ want: []string{"ate-idp-server"},
+ },
{
name: "podcert file",
path: func(e *Env) string { return e.Cfg.Manifest("pod-certificate-controller.yaml") },
diff --git a/cmd/ate-setup/internal/steps/substrate.go b/cmd/ate-setup/internal/steps/substrate.go
index 94b70b04aa..1b09386138 100644
--- a/cmd/ate-setup/internal/steps/substrate.go
+++ b/cmd/ate-setup/internal/steps/substrate.go
@@ -167,11 +167,8 @@ func (e *Env) DeleteSubstrateDemo(ctx context.Context, refs []resources.ActorTem
}
actorRef := resources.ActorRefFromActor(actor)
log.Stepf(" deleting actor %s/%s", actorRef.Atespace, actorRef.Name)
- // AnyState skips the SUSPENDED precondition: teardown discards the
- // actor, so there is nothing to be gained by suspending it first.
if _, err := client.DeleteActor(ctx, &ateapipb.DeleteActorRequest{
- Actor: actorRef.ToObjectRef(),
- AnyState: true,
+ Actor: actorRef.ToObjectRef(),
}); err != nil {
log.Warnf("while deleting actor %s/%s: %v", actorRef.Atespace, actorRef.Name, err)
}
diff --git a/cmd/ate-setup/internal/steps/substrate_test.go b/cmd/ate-setup/internal/steps/substrate_test.go
index 9ff7498ef9..5728c2eb90 100644
--- a/cmd/ate-setup/internal/steps/substrate_test.go
+++ b/cmd/ate-setup/internal/steps/substrate_test.go
@@ -45,6 +45,7 @@ func TestActorTemplateFromManifestDemos(t *testing.T) {
benchmarkValues := map[string]string{
"${ACTOR_CPU}": "1",
"${ACTOR_MEMORY}": "256Mi",
+ "${ATE_ENV_IMAGE}": "gcr.io/example/ate-env-guest@sha256:1d38934ef1dac71fcb1f1286de734e06ab2e19747279bccbd37bfe69e0d5bc72",
"${OTLP_ENDPOINT}": "http://otel-collector.otel-system:4317",
}
@@ -65,6 +66,11 @@ func TestActorTemplateFromManifestDemos(t *testing.T) {
relPath: "demos/sandbox/manual-test-multi-template.yaml",
want: resources.ActorTemplateRef{Atespace: "ate-manual-test-multi", Name: "sandbox-template"},
},
+ {
+ relPath: "benchmarking/workloads/manifests/ate-env-template.yaml.tmpl",
+ values: benchmarkValues,
+ want: resources.ActorTemplateRef{Atespace: "benchmark-workloads", Name: "ate-env"},
+ },
{
relPath: "benchmarking/workloads/manifests/sleep-template.yaml.tmpl",
values: benchmarkValues,
diff --git a/cmd/ate-setup/internal/steps/systemnetworkpolicy_test.go b/cmd/ate-setup/internal/steps/systemnetworkpolicy_test.go
index 31a2ba629b..807a6e3f2c 100644
--- a/cmd/ate-setup/internal/steps/systemnetworkpolicy_test.go
+++ b/cmd/ate-setup/internal/steps/systemnetworkpolicy_test.go
@@ -453,6 +453,8 @@ func TestSystemNetworkPoliciesLeastPrivilege(t *testing.T) {
// The API admits the control plane and opted-in namespaces only.
api := in.pod(t, "ate-api-server")
+ // The credential exchange beside it is for the node plugins only.
+ requireOnlyFromApps(t, in, api, numberedPort(t, api, 8445), "atelet")
for _, rule := range in.admittingRules(t, api, numberedPort(t, api, 443)) {
if len(rule.From) == 0 {
t.Error("ate-api-server port 443 is admitted from any source")
diff --git a/cmd/ateapi/helm_test.go b/cmd/ateapi/helm_test.go
index 64bc7d96bf..a3855c5f04 100644
--- a/cmd/ateapi/helm_test.go
+++ b/cmd/ateapi/helm_test.go
@@ -243,6 +243,7 @@ func TestHelmObjectStorePlugins(t *testing.T) {
if external {
issuer = "https://actors.example.com"
args = append(args, "--set", "rustfs.enabled=false", "--set", "atelet.storageBackend=gcs",
+ "--set", "ateApi.snapshotLocation=gs://external-snapshots/",
"--set", "ateApi.actorJWTIssuer="+issuer,
"--set", "objectStorePlugin.extraEnv[0].name=GOOGLE_APPLICATION_CREDENTIALS",
"--set", "objectStorePlugin.extraEnv[0].value=/credentials/key.json")
diff --git a/cmd/ateapi/internal/apiauthn/config.go b/cmd/ateapi/internal/apiauthn/config.go
index 4a781631d1..294b98a929 100644
--- a/cmd/ateapi/internal/apiauthn/config.go
+++ b/cmd/ateapi/internal/apiauthn/config.go
@@ -24,8 +24,7 @@ import (
// AuthenticationConfig configures JWT authentication for ateapi.
type AuthenticationConfig struct {
- ActorIdentityJWTProvider string `json:"actorIdentityJWTProvider"`
- JWTProviders []JWTProviderConfig `json:"jwtProviders"`
+ JWTProviders []JWTProviderConfig `json:"jwtProviders"`
}
// JWTProviderConfig configures one trusted OIDC issuer.
@@ -90,11 +89,5 @@ func ValidateAuthenticationConfig(cfg *AuthenticationConfig) error {
}
}
}
- if cfg.ActorIdentityJWTProvider == "" {
- return fmt.Errorf("actorIdentityJWTProvider is required")
- }
- if !names[cfg.ActorIdentityJWTProvider] {
- return fmt.Errorf("actorIdentityJWTProvider %q does not name a JWT provider", cfg.ActorIdentityJWTProvider)
- }
return nil
}
diff --git a/cmd/ateapi/internal/apiauthn/config_test.go b/cmd/ateapi/internal/apiauthn/config_test.go
index 8592fa6bcd..eb65d00269 100644
--- a/cmd/ateapi/internal/apiauthn/config_test.go
+++ b/cmd/ateapi/internal/apiauthn/config_test.go
@@ -24,7 +24,6 @@ import (
func TestLoadAuthenticationConfig(t *testing.T) {
path := filepath.Join(t.TempDir(), "authentication.yaml")
if err := os.WriteFile(path, []byte(`
-actorIdentityJWTProvider: kubernetes
jwtProviders:
- name: kubernetes
issuer: https://kubernetes.default.svc
@@ -47,7 +46,6 @@ jwtProviders:
func TestLoadAuthenticationConfigRejectsUnknownField(t *testing.T) {
path := filepath.Join(t.TempDir(), "authentication.yaml")
if err := os.WriteFile(path, []byte(`
-actorIdentityJWTProvider: kubernetes
jwtProviders:
- name: kubernetes
issuer: https://kubernetes.default.svc
@@ -65,7 +63,6 @@ jwtProviders:
func TestValidateAuthenticationConfig(t *testing.T) {
valid := func() *AuthenticationConfig {
return &AuthenticationConfig{
- ActorIdentityJWTProvider: "kubernetes",
JWTProviders: []JWTProviderConfig{{
Name: "kubernetes",
Issuer: "https://kubernetes.default.svc",
@@ -79,7 +76,6 @@ func TestValidateAuthenticationConfig(t *testing.T) {
mutate func(*AuthenticationConfig)
}{
{name: "no providers", mutate: func(c *AuthenticationConfig) { c.JWTProviders = nil }},
- {name: "unknown actor identity provider", mutate: func(c *AuthenticationConfig) { c.ActorIdentityJWTProvider = "missing" }},
{name: "insecure issuer", mutate: func(c *AuthenticationConfig) { c.JWTProviders[0].Issuer = "http://issuer.example" }},
{name: "no audiences", mutate: func(c *AuthenticationConfig) { c.JWTProviders[0].Audiences = nil }},
{name: "duplicate provider", mutate: func(c *AuthenticationConfig) { c.JWTProviders = append(c.JWTProviders, c.JWTProviders[0]) }},
diff --git a/cmd/ateapi/internal/apivalidation/actor.go b/cmd/ateapi/internal/apivalidation/actor.go
index 564d10840d..b690c6c0c4 100644
--- a/cmd/ateapi/internal/apivalidation/actor.go
+++ b/cmd/ateapi/internal/apivalidation/actor.go
@@ -137,14 +137,14 @@ func ValidateCustom_WorkerAssignment_WorkerSpiffeId(_ context.Context, _ operati
return validateSPIFFEID(fldPath, *value)
}
-func ValidateCustom_ExternalVolume_VolumeType(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *string) field.ErrorList {
+func ValidateCustom_ExternalVolumeStatus_VolumeType(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *string) field.ErrorList {
return resources.ValidateVolumeType(fldPath, *value)
}
-func ValidateCustom_ExternalVolume_StorageVolumeId(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *string) field.ErrorList {
+func ValidateCustom_ExternalVolumeStatus_StorageVolumeId(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *string) field.ErrorList {
return resources.ValidateStorageVolumeID(fldPath, *value)
}
-func ValidateCustom_ExternalVolume_VolumeContext(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ map[string]string) field.ErrorList {
+func ValidateCustom_ExternalVolumeStatus_VolumeContext(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ map[string]string) field.ErrorList {
return resources.ValidateCSIMap(fldPath, value)
}
diff --git a/cmd/ateapi/internal/apivalidation/actor_template.go b/cmd/ateapi/internal/apivalidation/actor_template.go
index 2b558e74b2..7f20e9322b 100644
--- a/cmd/ateapi/internal/apivalidation/actor_template.go
+++ b/cmd/ateapi/internal/apivalidation/actor_template.go
@@ -18,6 +18,7 @@ import (
"context"
"fmt"
+ "go.ate.dev/substrate/internal/preview"
"go.ate.dev/substrate/internal/resources"
"go.ate.dev/substrate/pkg/proto/ateapipb"
"k8s.io/apimachinery/pkg/api/operation"
@@ -176,7 +177,17 @@ func ValidateCustom_ExternalVolumeTemplate_Capacity(_ context.Context, _ operati
// ValidateCustom_SnapshotConfig_StorageLocation ensures an
// ActorTemplate's snapshotConfig.location is a well-formed
// URI with a bucket, so a bad location fails fast.
-func ValidateCustom_SnapshotConfig_StorageLocation(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *string) field.ErrorList {
+//
+// The field is ignored while the PerTemplateSnapshotLocation gate is off, so a
+// declarative ifDisabled=forbidden would reject the templates that set it
+// on servers without the gate. Instead, with the gate on, a request that
+// did not opt in to preview features is forbidden from setting it.
+func ValidateCustom_SnapshotConfig_StorageLocation(ctx context.Context, _ operation.Operation, fldPath *field.Path, value, _ *string) field.ErrorList {
+ if preview.IsEnabled(preview.GatePerTemplateSnapshotLocation) && preview.IsOptedOut(ctx) {
+ return field.ErrorList{field.Forbidden(fldPath, fmt.Sprintf(
+ "requires a client that opts in to preview features (kubectl ate --preview, or the %s header) while the %s preview gate is enabled",
+ preview.Header, preview.GatePerTemplateSnapshotLocation))}
+ }
if err := resources.ValidateSnapshotLocation(*value); err != nil {
return field.ErrorList{field.Invalid(fldPath, *value, err.Error())}
}
diff --git a/cmd/ateapi/internal/apivalidation/actor_template_test.go b/cmd/ateapi/internal/apivalidation/actor_template_test.go
index 7b4d963cee..52bf65d114 100644
--- a/cmd/ateapi/internal/apivalidation/actor_template_test.go
+++ b/cmd/ateapi/internal/apivalidation/actor_template_test.go
@@ -119,17 +119,17 @@ func TestValidateCreateActorTemplateRequest(t *testing.T) {
})},
field.ErrorList{field.Invalid(field.NewPath("actor_template", "containers").Index(0).Child("volume_mounts").Index(0).Child("name"), "ghost-vol", "")},
}, {
- "missing snapshot_config",
+ "missing snapshot_config is allowed",
&ateapipb.CreateActorTemplateRequest{ActorTemplate: validActorTemplate(func(tmpl *ateapipb.ActorTemplate) {
tmpl.SnapshotConfig = nil
})},
- field.ErrorList{field.Required(field.NewPath("actor_template", "snapshot_config"), "")},
+ nil,
}, {
- "missing snapshot_config.storage_location",
+ "snapshot_config.storage_location is not needed when the preview gate is off",
&ateapipb.CreateActorTemplateRequest{ActorTemplate: validActorTemplate(func(tmpl *ateapipb.ActorTemplate) {
tmpl.SnapshotConfig.StorageLocation = ""
})},
- field.ErrorList{field.Required(field.NewPath("actor_template", "snapshot_config", "storage_location"), "")},
+ nil,
}, {
"storage_location without a bucket",
&ateapipb.CreateActorTemplateRequest{ActorTemplate: validActorTemplate(func(tmpl *ateapipb.ActorTemplate) {
@@ -190,7 +190,7 @@ func TestValidateCreateActorTemplateRequest(t *testing.T) {
func TestValidateCreateActorTemplateRequest_PreviewRestricted(t *testing.T) {
setPreviewForTest(t, "ALL")
req := &ateapipb.CreateActorTemplateRequest{ActorTemplate: validActorTemplate(func(tmpl *ateapipb.ActorTemplate) {
- tmpl.Volumes = []*ateapipb.Volume{{Name: "data", ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{Capacity: "10Gi", StorageClassName: "fast-ssd"}}}
+ tmpl.Volumes = []*ateapipb.Volume{{Name: "data", External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{Capacity: "10Gi", StorageClassName: "fast-ssd"}}}}
})}
t.Run("allowed", func(t *testing.T) {
@@ -198,11 +198,65 @@ func TestValidateCreateActorTemplateRequest_PreviewRestricted(t *testing.T) {
})
t.Run("restricted", func(t *testing.T) {
ctx := preview.ContextWithOptOut(context.Background())
- want := field.ErrorList{field.Forbidden(field.NewPath("actor_template", "volumes").Index(0).Child("external_volume_template"), "")}
+ want := field.ErrorList{
+ field.Forbidden(field.NewPath("actor_template", "volumes").Index(0).Child("external"), ""),
+ field.Forbidden(field.NewPath("actor_template", "snapshot_config", "storage_location"), ""),
+ }
assertValidateErr(t, ValidateCreateActorTemplateRequest(ctx, req), want)
})
}
+func TestValidateCreateActorTemplateRequest_PerTemplateSnapshotLocationOptIn(t *testing.T) {
+ storageLocation := field.NewPath("actor_template", "snapshot_config", "storage_location")
+ withoutLocation := func(tmpl *ateapipb.ActorTemplate) { tmpl.SnapshotConfig.StorageLocation = "" }
+ tests := []struct {
+ name string
+ gates string
+ optOut bool
+ mutate func(*ateapipb.ActorTemplate)
+ want field.ErrorList
+ }{{
+ name: "gate off, opted out, location set",
+ optOut: true,
+ }, {
+ name: "gate off, opted out, no location",
+ optOut: true,
+ mutate: withoutLocation,
+ }, {
+ name: "gate on, opted out, location set",
+ gates: "PerTemplateSnapshotLocation",
+ optOut: true,
+ want: field.ErrorList{field.Forbidden(storageLocation, "")},
+ }, {
+ name: "gate on, opted out, no location",
+ gates: "PerTemplateSnapshotLocation",
+ optOut: true,
+ mutate: withoutLocation,
+ }, {
+ name: "gate on, opted in, location set",
+ gates: "PerTemplateSnapshotLocation",
+ }, {
+ name: "gate on, opted in, no location",
+ gates: "PerTemplateSnapshotLocation",
+ mutate: withoutLocation,
+ }}
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ setPreviewForTest(t, tt.gates)
+ ctx := context.Background()
+ if tt.optOut {
+ ctx = preview.ContextWithOptOut(ctx)
+ }
+ var mutations []func(*ateapipb.ActorTemplate)
+ if tt.mutate != nil {
+ mutations = append(mutations, tt.mutate)
+ }
+ req := &ateapipb.CreateActorTemplateRequest{ActorTemplate: validActorTemplate(mutations...)}
+ assertValidateErr(t, ValidateCreateActorTemplateRequest(ctx, req), tt.want)
+ })
+ }
+}
+
func TestValidateGetActorTemplateRequest(t *testing.T) {
tests := []struct {
name string
@@ -359,9 +413,9 @@ func TestValidateActorTemplate(t *testing.T) {
},
want: field.ErrorList{field.Invalid(field.NewPath("worker_selector", "match_labels").Key("tier"), nil, "").WithOrigin("format=k8s-label-value")},
}, {
- name: "missing snapshot_config",
+ name: "missing snapshot_config is allowed",
mutate: func(tmpl *ateapipb.ActorTemplate) { tmpl.SnapshotConfig = nil },
- want: field.ErrorList{field.Required(field.NewPath("snapshot_config"), "")},
+ want: nil,
}, {
name: "storage_location too long",
mutate: func(tmpl *ateapipb.ActorTemplate) {
@@ -371,7 +425,9 @@ func TestValidateActorTemplate(t *testing.T) {
}, {
name: "missing storage_location",
mutate: func(tmpl *ateapipb.ActorTemplate) { tmpl.SnapshotConfig.StorageLocation = "" },
- want: field.ErrorList{field.Required(field.NewPath("snapshot_config", "storage_location"), "")},
+ // Optional with or without the gate: an empty field falls back to
+ // --snapshot-location.
+ want: nil,
}, {
name: "unspecified snapshot scope",
mutate: func(tmpl *ateapipb.ActorTemplate) {
@@ -771,52 +827,52 @@ func TestValidateActorTemplate(t *testing.T) {
}, {
name: "valid external volume template",
mutate: func(tmpl *ateapipb.ActorTemplate) {
- tmpl.Volumes = []*ateapipb.Volume{{Name: "data", ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{Capacity: "10Gi", StorageClassName: "fast-ssd"}}}
+ tmpl.Volumes = []*ateapipb.Volume{{Name: "data", External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{Capacity: "10Gi", StorageClassName: "fast-ssd"}}}}
},
- want: field.ErrorList{field.Forbidden(field.NewPath("volumes").Index(0).Child("external_volume_template"), "")},
+ want: field.ErrorList{field.Forbidden(field.NewPath("volumes").Index(0).Child("external"), "")},
wantWithPreview: map[string]field.ErrorList{"ALL": nil},
}, {
name: "external volume template missing capacity",
mutate: func(tmpl *ateapipb.ActorTemplate) {
- tmpl.Volumes = []*ateapipb.Volume{{Name: "data", ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{StorageClassName: "fast-ssd"}}}
+ tmpl.Volumes = []*ateapipb.Volume{{Name: "data", External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{StorageClassName: "fast-ssd"}}}}
},
- want: field.ErrorList{field.Forbidden(field.NewPath("volumes").Index(0).Child("external_volume_template"), "")},
- wantWithPreview: map[string]field.ErrorList{"ALL": {field.Required(field.NewPath("volumes").Index(0).Child("external_volume_template", "capacity"), "")}},
+ want: field.ErrorList{field.Forbidden(field.NewPath("volumes").Index(0).Child("external"), "")},
+ wantWithPreview: map[string]field.ErrorList{"ALL": {field.Required(field.NewPath("volumes").Index(0).Child("external", "template", "capacity"), "")}},
}, {
name: "external volume template malformed capacity",
mutate: func(tmpl *ateapipb.ActorTemplate) {
- tmpl.Volumes = []*ateapipb.Volume{{Name: "data", ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{Capacity: "ten gigs", StorageClassName: "fast-ssd"}}}
+ tmpl.Volumes = []*ateapipb.Volume{{Name: "data", External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{Capacity: "ten gigs", StorageClassName: "fast-ssd"}}}}
},
- want: field.ErrorList{field.Forbidden(field.NewPath("volumes").Index(0).Child("external_volume_template"), "")},
- wantWithPreview: map[string]field.ErrorList{"ALL": {field.Invalid(field.NewPath("volumes").Index(0).Child("external_volume_template", "capacity"), nil, "")}},
+ want: field.ErrorList{field.Forbidden(field.NewPath("volumes").Index(0).Child("external"), "")},
+ wantWithPreview: map[string]field.ErrorList{"ALL": {field.Invalid(field.NewPath("volumes").Index(0).Child("external", "template", "capacity"), nil, "")}},
}, {
name: "external volume template capacity at the length bound",
mutate: func(tmpl *ateapipb.ActorTemplate) {
- tmpl.Volumes = []*ateapipb.Volume{{Name: "data", ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{Capacity: strings.Repeat("1", 30) + "Gi", StorageClassName: "fast-ssd"}}}
+ tmpl.Volumes = []*ateapipb.Volume{{Name: "data", External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{Capacity: strings.Repeat("1", 30) + "Gi", StorageClassName: "fast-ssd"}}}}
},
- want: field.ErrorList{field.Forbidden(field.NewPath("volumes").Index(0).Child("external_volume_template"), "")},
+ want: field.ErrorList{field.Forbidden(field.NewPath("volumes").Index(0).Child("external"), "")},
wantWithPreview: map[string]field.ErrorList{"ALL": nil},
}, {
name: "external volume template capacity too long",
mutate: func(tmpl *ateapipb.ActorTemplate) {
- tmpl.Volumes = []*ateapipb.Volume{{Name: "data", ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{Capacity: strings.Repeat("1", 31) + "Gi", StorageClassName: "fast-ssd"}}}
+ tmpl.Volumes = []*ateapipb.Volume{{Name: "data", External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{Capacity: strings.Repeat("1", 31) + "Gi", StorageClassName: "fast-ssd"}}}}
},
- want: field.ErrorList{field.Forbidden(field.NewPath("volumes").Index(0).Child("external_volume_template"), "")},
- wantWithPreview: map[string]field.ErrorList{"ALL": {field.TooLong(field.NewPath("volumes").Index(0).Child("external_volume_template", "capacity"), nil, 32).WithOrigin("maxLength")}},
+ want: field.ErrorList{field.Forbidden(field.NewPath("volumes").Index(0).Child("external"), "")},
+ wantWithPreview: map[string]field.ErrorList{"ALL": {field.TooLong(field.NewPath("volumes").Index(0).Child("external", "template", "capacity"), nil, 32).WithOrigin("maxLength")}},
}, {
name: "external volume template missing storage_class_name",
mutate: func(tmpl *ateapipb.ActorTemplate) {
- tmpl.Volumes = []*ateapipb.Volume{{Name: "data", ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{Capacity: "10Gi"}}}
+ tmpl.Volumes = []*ateapipb.Volume{{Name: "data", External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{Capacity: "10Gi"}}}}
},
- want: field.ErrorList{field.Forbidden(field.NewPath("volumes").Index(0).Child("external_volume_template"), "")},
- wantWithPreview: map[string]field.ErrorList{"ALL": {field.Required(field.NewPath("volumes").Index(0).Child("external_volume_template", "storage_class_name"), "")}},
+ want: field.ErrorList{field.Forbidden(field.NewPath("volumes").Index(0).Child("external"), "")},
+ wantWithPreview: map[string]field.ErrorList{"ALL": {field.Required(field.NewPath("volumes").Index(0).Child("external", "template", "storage_class_name"), "")}},
}, {
name: "external volume template invalid storage_class_name",
mutate: func(tmpl *ateapipb.ActorTemplate) {
- tmpl.Volumes = []*ateapipb.Volume{{Name: "data", ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{Capacity: "10Gi", StorageClassName: "Fast SSD"}}}
+ tmpl.Volumes = []*ateapipb.Volume{{Name: "data", External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{Capacity: "10Gi", StorageClassName: "Fast SSD"}}}}
},
- want: field.ErrorList{field.Forbidden(field.NewPath("volumes").Index(0).Child("external_volume_template"), "")},
- wantWithPreview: map[string]field.ErrorList{"ALL": {field.Invalid(field.NewPath("volumes").Index(0).Child("external_volume_template", "storage_class_name"), nil, "").WithOrigin("format=k8s-long-name")}},
+ want: field.ErrorList{field.Forbidden(field.NewPath("volumes").Index(0).Child("external"), "")},
+ wantWithPreview: map[string]field.ErrorList{"ALL": {field.Invalid(field.NewPath("volumes").Index(0).Child("external", "template", "storage_class_name"), nil, "").WithOrigin("format=k8s-long-name")}},
}, {
name: "valid resources",
mutate: func(tmpl *ateapipb.ActorTemplate) {
diff --git a/cmd/ateapi/internal/apivalidation/actor_test.go b/cmd/ateapi/internal/apivalidation/actor_test.go
index 0cc84108b4..6223585c5a 100644
--- a/cmd/ateapi/internal/apivalidation/actor_test.go
+++ b/cmd/ateapi/internal/apivalidation/actor_test.go
@@ -620,7 +620,7 @@ func TestValidateActorUpdate(t *testing.T) {
name: "adding a volume",
oldVal: validActor(withActorStatus()),
newVal: validActor(withActorStatus(func(s *ateapipb.ActorStatus) {
- s.ExternalVolumes = []*ateapipb.ExternalVolume{{Name: "vol-a", VolumeType: "substrate.io/mock"}}
+ s.ExternalVolumes = &ateapipb.ExternalVolumes{Volumes: []*ateapipb.ExternalVolumeStatus{{Name: "vol-a", VolumeType: "substrate.io/mock"}}}
})),
want: nil,
wantWithPreview: map[string]field.ErrorList{"ALL": nil},
@@ -628,22 +628,22 @@ func TestValidateActorUpdate(t *testing.T) {
name: "adding a volume to an actor with no status",
oldVal: validActor(),
newVal: validActor(withActorStatus(func(s *ateapipb.ActorStatus) {
- s.ExternalVolumes = []*ateapipb.ExternalVolume{{Name: "vol-a", VolumeType: "substrate.io/mock"}}
+ s.ExternalVolumes = &ateapipb.ExternalVolumes{Volumes: []*ateapipb.ExternalVolumeStatus{{Name: "vol-a", VolumeType: "substrate.io/mock"}}}
})),
want: nil,
wantWithPreview: map[string]field.ErrorList{"ALL": nil},
}, {
name: "provisioning transition on an existing volume",
oldVal: validInput(withStatus(func(s *ateapipb.ActorStatus) {
- s.ExternalVolumes = []*ateapipb.ExternalVolume{{Name: "vol-a", VolumeType: "substrate.io/mock"}}
+ s.ExternalVolumes = &ateapipb.ExternalVolumes{Volumes: []*ateapipb.ExternalVolumeStatus{{Name: "vol-a", VolumeType: "substrate.io/mock"}}}
})),
newVal: validActor(withActorStatus(func(s *ateapipb.ActorStatus) {
- s.ExternalVolumes = []*ateapipb.ExternalVolume{{
+ s.ExternalVolumes = &ateapipb.ExternalVolumes{Volumes: []*ateapipb.ExternalVolumeStatus{{
Name: "vol-a",
VolumeType: "substrate.io/mock",
StorageVolumeId: "csi-426d29b7",
VolumeContext: map[string]string{"attachment": "iqn.2026-08.io.ate:vol-a"},
- }}
+ }}}
})),
want: nil,
wantWithPreview: map[string]field.ErrorList{"ALL": nil},
@@ -651,28 +651,28 @@ func TestValidateActorUpdate(t *testing.T) {
name: "duplicate external_volumes volume_name",
oldVal: validActor(withActorStatus()),
newVal: validActor(withActorStatus(func(s *ateapipb.ActorStatus) {
- s.ExternalVolumes = []*ateapipb.ExternalVolume{
+ s.ExternalVolumes = &ateapipb.ExternalVolumes{Volumes: []*ateapipb.ExternalVolumeStatus{
{Name: "vol-a", VolumeType: "substrate.io/mock"},
{Name: "vol-a", VolumeType: "substrate.io/mock"},
- }
+ }}
})),
- want: field.ErrorList{field.Duplicate(field.NewPath("status", "external_volumes").Index(1), nil)},
+ want: field.ErrorList{field.Duplicate(field.NewPath("status", "external_volumes", "volumes").Index(1), nil)},
wantWithPreview: map[string]field.ErrorList{
- "ALL": {field.Duplicate(field.NewPath("status", "external_volumes").Index(1), nil)},
+ "ALL": {field.Duplicate(field.NewPath("status", "external_volumes", "volumes").Index(1), nil)},
},
}, {
name: "too many external_volumes",
oldVal: validActor(withActorStatus()),
newVal: validActor(withActorStatus(func(s *ateapipb.ActorStatus) {
- vols := make([]*ateapipb.ExternalVolume, 33)
+ vols := make([]*ateapipb.ExternalVolumeStatus, 33)
for i := range vols {
- vols[i] = &ateapipb.ExternalVolume{Name: fmt.Sprintf("vol-%d", i), VolumeType: "substrate.io/mock"}
+ vols[i] = &ateapipb.ExternalVolumeStatus{Name: fmt.Sprintf("vol-%d", i), VolumeType: "substrate.io/mock"}
}
- s.ExternalVolumes = vols
+ s.ExternalVolumes = &ateapipb.ExternalVolumes{Volumes: vols}
})),
- want: field.ErrorList{field.TooMany(field.NewPath("status", "external_volumes"), 33, 32).WithOrigin("maxItems")},
+ want: field.ErrorList{field.TooMany(field.NewPath("status", "external_volumes", "volumes"), 33, 32).WithOrigin("maxItems")},
wantWithPreview: map[string]field.ErrorList{
- "ALL": {field.TooMany(field.NewPath("status", "external_volumes"), 33, 32).WithOrigin("maxItems")},
+ "ALL": {field.TooMany(field.NewPath("status", "external_volumes", "volumes"), 33, 32).WithOrigin("maxItems")},
},
}}
for _, tt := range tests {
diff --git a/cmd/ateapi/internal/apivalidation/validation_test.go b/cmd/ateapi/internal/apivalidation/validation_test.go
index 0b1d5e252c..af45ffb11e 100644
--- a/cmd/ateapi/internal/apivalidation/validation_test.go
+++ b/cmd/ateapi/internal/apivalidation/validation_test.go
@@ -871,8 +871,8 @@ func TestValidateTrustBundleDataSource(t *testing.T) {
}
}
-func validExternalVolume(mutate ...func(*ateapipb.ExternalVolume)) *ateapipb.ExternalVolume {
- v := &ateapipb.ExternalVolume{
+func validExternalVolume(mutate ...func(*ateapipb.ExternalVolumeStatus)) *ateapipb.ExternalVolumeStatus {
+ v := &ateapipb.ExternalVolumeStatus{
Name: "my-vol",
StorageVolumeId: "valid-storage-id",
VolumeType: "mock",
@@ -888,57 +888,57 @@ func TestValidateExternalVolume(t *testing.T) {
tests := []struct {
name string
- obj *ateapipb.ExternalVolume
+ obj *ateapipb.ExternalVolumeStatus
want field.ErrorList
}{{
name: "valid external volume",
obj: valid(),
}, {
name: "missing volume name",
- obj: valid(func(v *ateapipb.ExternalVolume) { v.Name = "" }),
+ obj: valid(func(v *ateapipb.ExternalVolumeStatus) { v.Name = "" }),
want: field.ErrorList{field.Required(field.NewPath("name"), "")},
}, {
name: "invalid volume name",
- obj: valid(func(v *ateapipb.ExternalVolume) { v.Name = "NOT A VOLUME" }),
+ obj: valid(func(v *ateapipb.ExternalVolumeStatus) { v.Name = "NOT A VOLUME" }),
want: field.ErrorList{field.Invalid(field.NewPath("name"), nil, "").WithOrigin("format=k8s-short-name")},
}, {
name: "valid external volume with empty storage volume id",
- obj: valid(func(v *ateapipb.ExternalVolume) { v.StorageVolumeId = "" }),
+ obj: valid(func(v *ateapipb.ExternalVolumeStatus) { v.StorageVolumeId = "" }),
}, {
name: "invalid storage volume id with null U+0000",
- obj: valid(func(v *ateapipb.ExternalVolume) { v.StorageVolumeId = "vol\x00id" }),
+ obj: valid(func(v *ateapipb.ExternalVolumeStatus) { v.StorageVolumeId = "vol\x00id" }),
want: field.ErrorList{field.Invalid(field.NewPath("storage_volume_id"), nil, "")},
}, {
name: "storage volume id too long",
- obj: valid(func(v *ateapipb.ExternalVolume) { v.StorageVolumeId = strings.Repeat("x", 257) }),
+ obj: valid(func(v *ateapipb.ExternalVolumeStatus) { v.StorageVolumeId = strings.Repeat("x", 257) }),
want: field.ErrorList{field.TooLong(field.NewPath("storage_volume_id"), nil, 256).WithOrigin("maxLength")},
}, {
name: "missing volume type",
- obj: valid(func(v *ateapipb.ExternalVolume) { v.VolumeType = "" }),
+ obj: valid(func(v *ateapipb.ExternalVolumeStatus) { v.VolumeType = "" }),
want: field.ErrorList{field.Required(field.NewPath("volume_type"), "")},
}, {
name: "invalid volume type with uppercase",
- obj: valid(func(v *ateapipb.ExternalVolume) { v.VolumeType = "MockPlugin" }),
+ obj: valid(func(v *ateapipb.ExternalVolumeStatus) { v.VolumeType = "MockPlugin" }),
want: field.ErrorList{field.Invalid(field.NewPath("volume_type"), nil, "")},
}, {
name: "valid volume type with 253 characters",
- obj: valid(func(v *ateapipb.ExternalVolume) { v.VolumeType = strings.Repeat("a", 253) }),
+ obj: valid(func(v *ateapipb.ExternalVolumeStatus) { v.VolumeType = strings.Repeat("a", 253) }),
}, {
name: "invalid volume type exceeding 253 characters",
- obj: valid(func(v *ateapipb.ExternalVolume) { v.VolumeType = strings.Repeat("a", 254) }),
+ obj: valid(func(v *ateapipb.ExternalVolumeStatus) { v.VolumeType = strings.Repeat("a", 254) }),
want: field.ErrorList{
field.Invalid(field.NewPath("volume_type"), nil, ""),
field.TooLong(field.NewPath("volume_type"), nil, 253).WithOrigin("maxLength"),
},
}, {
name: "valid volume with substrate.io prefixed volume type",
- obj: valid(func(v *ateapipb.ExternalVolume) { v.VolumeType = "substrate.io/mock" }),
+ obj: valid(func(v *ateapipb.ExternalVolumeStatus) { v.VolumeType = "substrate.io/mock" }),
}, {
name: "storage volume id at the bound",
- obj: valid(func(v *ateapipb.ExternalVolume) { v.StorageVolumeId = strings.Repeat("x", 256) }),
+ obj: valid(func(v *ateapipb.ExternalVolumeStatus) { v.StorageVolumeId = strings.Repeat("x", 256) }),
}, {
name: "volume_context with many entries, as a CSI driver may return",
- obj: valid(func(v *ateapipb.ExternalVolume) {
+ obj: valid(func(v *ateapipb.ExternalVolumeStatus) {
ctxMap := make(map[string]string, 100)
for i := range 100 {
ctxMap[fmt.Sprintf("key-%03d", i)] = "v"
@@ -947,12 +947,12 @@ func TestValidateExternalVolume(t *testing.T) {
}),
}, {
name: "volume_context with one long value, as a CSI driver may return",
- obj: valid(func(v *ateapipb.ExternalVolume) {
+ obj: valid(func(v *ateapipb.ExternalVolumeStatus) {
v.VolumeContext = map[string]string{"attachment": strings.Repeat("v", 4000)}
}),
}, {
name: "volume_context over the size bound",
- obj: valid(func(v *ateapipb.ExternalVolume) {
+ obj: valid(func(v *ateapipb.ExternalVolumeStatus) {
v.VolumeContext = map[string]string{"attachment": strings.Repeat("v", 40*1024)}
}),
want: field.ErrorList{field.TooLong(field.NewPath("volume_context"), nil, 40*1024)},
@@ -960,7 +960,7 @@ func TestValidateExternalVolume(t *testing.T) {
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
op := MakeCreateOp(context.Background())
- assertValidateErr(t, Validate_ExternalVolume(context.Background(), op, nil, tt.obj, nil), tt.want)
+ assertValidateErr(t, Validate_ExternalVolumeStatus(context.Background(), op, nil, tt.obj, nil), tt.want)
})
}
}
@@ -970,8 +970,8 @@ func TestValidateExternalVolume_Update(t *testing.T) {
tests := []struct {
name string
- oldObj *ateapipb.ExternalVolume
- newObj *ateapipb.ExternalVolume
+ oldObj *ateapipb.ExternalVolumeStatus
+ newObj *ateapipb.ExternalVolumeStatus
want field.ErrorList
}{{
name: "unchanged volume is valid",
@@ -979,37 +979,37 @@ func TestValidateExternalVolume_Update(t *testing.T) {
newObj: valid(),
}, {
name: "name changed is invalid",
- oldObj: valid(func(v *ateapipb.ExternalVolume) { v.Name = "vol1" }),
- newObj: valid(func(v *ateapipb.ExternalVolume) { v.Name = "vol2" }),
+ oldObj: valid(func(v *ateapipb.ExternalVolumeStatus) { v.Name = "vol1" }),
+ newObj: valid(func(v *ateapipb.ExternalVolumeStatus) { v.Name = "vol2" }),
want: field.ErrorList{field.Invalid(field.NewPath("name"), nil, "").WithOrigin("update")},
}, {
name: "storage_volume_id transition from empty to non-empty is valid",
- oldObj: valid(func(v *ateapipb.ExternalVolume) { v.StorageVolumeId = "" }),
- newObj: valid(func(v *ateapipb.ExternalVolume) { v.StorageVolumeId = "vol-id-1" }),
+ oldObj: valid(func(v *ateapipb.ExternalVolumeStatus) { v.StorageVolumeId = "" }),
+ newObj: valid(func(v *ateapipb.ExternalVolumeStatus) { v.StorageVolumeId = "vol-id-1" }),
}, {
name: "storage_volume_id changed once set is invalid",
- oldObj: valid(func(v *ateapipb.ExternalVolume) { v.StorageVolumeId = "vol-id-1" }),
- newObj: valid(func(v *ateapipb.ExternalVolume) { v.StorageVolumeId = "vol-id-2" }),
+ oldObj: valid(func(v *ateapipb.ExternalVolumeStatus) { v.StorageVolumeId = "vol-id-1" }),
+ newObj: valid(func(v *ateapipb.ExternalVolumeStatus) { v.StorageVolumeId = "vol-id-2" }),
want: field.ErrorList{field.Invalid(field.NewPath("storage_volume_id"), nil, "").WithOrigin("update")},
}, {
name: "storage_volume_id unset once set is invalid",
- oldObj: valid(func(v *ateapipb.ExternalVolume) { v.StorageVolumeId = "vol-id-1" }),
- newObj: valid(func(v *ateapipb.ExternalVolume) { v.StorageVolumeId = "" }),
+ oldObj: valid(func(v *ateapipb.ExternalVolumeStatus) { v.StorageVolumeId = "vol-id-1" }),
+ newObj: valid(func(v *ateapipb.ExternalVolumeStatus) { v.StorageVolumeId = "" }),
want: field.ErrorList{field.Invalid(field.NewPath("storage_volume_id"), nil, "").WithOrigin("update")},
}, {
name: "volume_type changed is invalid",
- oldObj: valid(func(v *ateapipb.ExternalVolume) { v.VolumeType = "mock" }),
- newObj: valid(func(v *ateapipb.ExternalVolume) { v.VolumeType = "pd.csi.storage.gke.io" }),
+ oldObj: valid(func(v *ateapipb.ExternalVolumeStatus) { v.VolumeType = "mock" }),
+ newObj: valid(func(v *ateapipb.ExternalVolumeStatus) { v.VolumeType = "pd.csi.storage.gke.io" }),
want: field.ErrorList{field.Invalid(field.NewPath("volume_type"), nil, "").WithOrigin("update")},
}, {
name: "volume_context changed is valid",
- oldObj: valid(func(v *ateapipb.ExternalVolume) { v.VolumeContext = nil }),
- newObj: valid(func(v *ateapipb.ExternalVolume) { v.VolumeContext = map[string]string{"foo": "bar"} }),
+ oldObj: valid(func(v *ateapipb.ExternalVolumeStatus) { v.VolumeContext = nil }),
+ newObj: valid(func(v *ateapipb.ExternalVolumeStatus) { v.VolumeContext = map[string]string{"foo": "bar"} }),
}}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
op := MakeUpdateOp(context.Background())
- assertValidateErr(t, Validate_ExternalVolume(context.Background(), op, nil, tt.newObj, tt.oldObj), tt.want)
+ assertValidateErr(t, Validate_ExternalVolumeStatus(context.Background(), op, nil, tt.newObj, tt.oldObj), tt.want)
})
}
}
diff --git a/cmd/ateapi/internal/apivalidation/worker.go b/cmd/ateapi/internal/apivalidation/worker.go
index 73ec35e07b..6f53f82eb5 100644
--- a/cmd/ateapi/internal/apivalidation/worker.go
+++ b/cmd/ateapi/internal/apivalidation/worker.go
@@ -82,6 +82,11 @@ func ValidateRequestActorSuspendRequest(ctx context.Context, req *ateapipb.Reque
return Validate_RequestActorSuspendRequest(ctx, op, nil, req, nil)
}
+func ValidateReportActorCrashedRequest(ctx context.Context, req *ateapipb.ReportActorCrashedRequest) field.ErrorList {
+ op := operation.Operation{Type: operation.Create}
+ return Validate_ReportActorCrashedRequest(ctx, op, nil, req, nil)
+}
+
func ValidateMarkWorkerAvailableRequest(ctx context.Context, req *ateapipb.MarkWorkerAvailableRequest) field.ErrorList {
// Call the generated validation.
op := operation.Operation{Type: operation.Create}
diff --git a/cmd/ateapi/internal/apivalidation/zz_generated.validation.go b/cmd/ateapi/internal/apivalidation/zz_generated.validation.go
index 18e09da61c..9067a434e4 100644
--- a/cmd/ateapi/internal/apivalidation/zz_generated.validation.go
+++ b/cmd/ateapi/internal/apivalidation/zz_generated.validation.go
@@ -784,7 +784,7 @@ func Validate_ActorStatus(
if earlyReturn {
return // do not proceed
}
- if e := validate.Maximum(ctx, op, fldPath, obj, oldObj, 9); len(e) != 0 {
+ if e := validate.Maximum(ctx, op, fldPath, obj, oldObj, 7); len(e) != 0 {
errs = append(errs, e...)
}
if e := validate.Minimum(ctx, op, fldPath, obj, oldObj, 1); len(e) != 0 {
@@ -836,7 +836,7 @@ func Validate_ActorStatus(
{ // field ateapipb.ActorStatus.ExternalVolumes
fn := func(
fldPath *field.Path,
- obj, oldObj []*ateapipb.ExternalVolume,
+ obj, oldObj *ateapipb.ExternalVolumes,
oldValueCorrelated bool) (errs field.ErrorList) {
// don't revalidate unchanged data
if oldValueCorrelated && op.Type == operation.Update {
@@ -846,34 +846,18 @@ func Validate_ActorStatus(
}
// call field-attached validations
earlyReturn := false
- if e := validate.PtrSliceNoNils[ateapipb.ExternalVolume](ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 {
- errs = append(errs, e...)
- earlyReturn = true
- }
- if e := validate.MaxItems(ctx, op, fldPath, obj, oldObj, 32).MarkShortCircuit(); len(e) != 0 {
- errs = append(errs, e...)
- earlyReturn = true
- }
- if e := validate.OptionalSlice(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 {
+ if e := validate.OptionalPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 {
earlyReturn = true
}
if earlyReturn {
return // do not proceed
}
- // lists with map semantics require unique keys
- if e := validate.PtrSliceUnique(ctx, op, fldPath, obj, oldObj,
- func(a *ateapipb.ExternalVolume, b *ateapipb.ExternalVolume) bool { return a.Name == b.Name }); len(e) != 0 {
- errs = append(errs, e...)
- }
- // iterate the list and call the type's validation function
- if e := validate.EachPtrSliceVal(ctx, op, fldPath, obj, oldObj,
- func(a *ateapipb.ExternalVolume, b *ateapipb.ExternalVolume) bool { return a.Name == b.Name }, ateDeepEqual, Validate_ExternalVolume); len(e) != 0 {
- errs = append(errs, e...)
- }
+ // call the type's validation function
+ errs = append(errs, Validate_ExternalVolumes(ctx, op, fldPath, obj, oldObj)...)
return
}
oldVal := safe.Field(oldObj,
- func(oldObj *ateapipb.ActorStatus) []*ateapipb.ExternalVolume {
+ func(oldObj *ateapipb.ActorStatus) *ateapipb.ExternalVolumes {
return oldObj.ExternalVolumes
})
errs = append(errs, fn(fldPath.Child("external_volumes"), obj.ExternalVolumes, oldVal, oldObj != nil)...)
@@ -1220,8 +1204,7 @@ func Validate_ActorTemplate(
}
// call field-attached validations
earlyReturn := false
- if e := validate.RequiredPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 {
- errs = append(errs, e...)
+ if e := validate.OptionalPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 {
earlyReturn = true
}
if earlyReturn {
@@ -2751,34 +2734,6 @@ func Validate_DeleteActorRequest(
errs = append(errs, fn(fldPath.Child("actor"), obj.Actor, oldVal, oldObj != nil)...)
}
- { // field ateapipb.DeleteActorRequest.AnyState
- fn := func(
- fldPath *field.Path,
- obj, oldObj *bool,
- oldValueCorrelated bool) (errs field.ErrorList) {
- // don't revalidate unchanged data
- if oldValueCorrelated && op.Type == operation.Update {
- if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) {
- return nil
- }
- }
- // call field-attached validations
- earlyReturn := false
- if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 {
- earlyReturn = true
- }
- if earlyReturn {
- return // do not proceed
- }
- return
- }
- oldVal := safe.Field(oldObj,
- func(oldObj *ateapipb.DeleteActorRequest) *bool {
- return &oldObj.AnyState
- })
- errs = append(errs, fn(fldPath.Child("any_state"), &obj.AnyState, oldVal, oldObj != nil)...)
- }
-
{ // field ateapipb.DeleteActorRequest.Options
fn := func(
fldPath *field.Path,
@@ -3721,13 +3676,64 @@ func Validate_EnvVar(
return errs
}
-// Validate_ExternalVolume validates an instance of ExternalVolume according
+var unionMembershipFor_go_ate_dev_substrate_pkg_proto_ateapipb_ExternalVolumeSource_ = validate.NewUnionMembership(validate.NewUnionMember("template"))
+
+// Validate_ExternalVolumeSource validates an instance of ExternalVolumeSource according
// to declarative validation rules in the API schema.
-func Validate_ExternalVolume(
+func Validate_ExternalVolumeSource(
ctx context.Context, op operation.Operation, fldPath *field.Path,
- obj, oldObj *ateapipb.ExternalVolume) (errs field.ErrorList) {
+ obj, oldObj *ateapipb.ExternalVolumeSource) (errs field.ErrorList) {
- { // field ateapipb.ExternalVolume.Name
+ if e := validate.Union(ctx, op, fldPath, obj, oldObj, unionMembershipFor_go_ate_dev_substrate_pkg_proto_ateapipb_ExternalVolumeSource_,
+ func(obj *ateapipb.ExternalVolumeSource) bool {
+ if obj == nil {
+ return false
+ }
+ return obj.Template != nil
+ }); len(e) != 0 {
+ errs = append(errs, e...)
+ }
+
+ { // field ateapipb.ExternalVolumeSource.Template
+ fn := func(
+ fldPath *field.Path,
+ obj, oldObj *ateapipb.ExternalVolumeTemplate,
+ oldValueCorrelated bool) (errs field.ErrorList) {
+ // don't revalidate unchanged data
+ if oldValueCorrelated && op.Type == operation.Update {
+ if ateDeepEqual(obj, oldObj) {
+ return nil
+ }
+ }
+ // call field-attached validations
+ earlyReturn := false
+ if e := validate.OptionalPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 {
+ earlyReturn = true
+ }
+ if earlyReturn {
+ return // do not proceed
+ }
+ // call the type's validation function
+ errs = append(errs, Validate_ExternalVolumeTemplate(ctx, op, fldPath, obj, oldObj)...)
+ return
+ }
+ oldVal := safe.Field(oldObj,
+ func(oldObj *ateapipb.ExternalVolumeSource) *ateapipb.ExternalVolumeTemplate {
+ return oldObj.Template
+ })
+ errs = append(errs, fn(fldPath.Child("template"), obj.Template, oldVal, oldObj != nil)...)
+ }
+
+ return errs
+}
+
+// Validate_ExternalVolumeStatus validates an instance of ExternalVolumeStatus according
+// to declarative validation rules in the API schema.
+func Validate_ExternalVolumeStatus(
+ ctx context.Context, op operation.Operation, fldPath *field.Path,
+ obj, oldObj *ateapipb.ExternalVolumeStatus) (errs field.ErrorList) {
+
+ { // field ateapipb.ExternalVolumeStatus.Name
fn := func(
fldPath *field.Path,
obj, oldObj *string,
@@ -3758,13 +3764,13 @@ func Validate_ExternalVolume(
return
}
oldVal := safe.Field(oldObj,
- func(oldObj *ateapipb.ExternalVolume) *string {
+ func(oldObj *ateapipb.ExternalVolumeStatus) *string {
return &oldObj.Name
})
errs = append(errs, fn(fldPath.Child("name"), &obj.Name, oldVal, oldObj != nil)...)
}
- { // field ateapipb.ExternalVolume.StorageVolumeId
+ { // field ateapipb.ExternalVolumeStatus.StorageVolumeId
fn := func(
fldPath *field.Path,
obj, oldObj *string,
@@ -3789,7 +3795,7 @@ func Validate_ExternalVolume(
return // do not proceed
}
// custom validation
- if e := ValidateCustom_ExternalVolume_StorageVolumeId(ctx, op, fldPath, obj, oldObj); len(e) != 0 {
+ if e := ValidateCustom_ExternalVolumeStatus_StorageVolumeId(ctx, op, fldPath, obj, oldObj); len(e) != 0 {
errs = append(errs, e...)
}
if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 256); len(e) != 0 {
@@ -3798,13 +3804,13 @@ func Validate_ExternalVolume(
return
}
oldVal := safe.Field(oldObj,
- func(oldObj *ateapipb.ExternalVolume) *string {
+ func(oldObj *ateapipb.ExternalVolumeStatus) *string {
return &oldObj.StorageVolumeId
})
errs = append(errs, fn(fldPath.Child("storage_volume_id"), &obj.StorageVolumeId, oldVal, oldObj != nil)...)
}
- { // field ateapipb.ExternalVolume.VolumeType
+ { // field ateapipb.ExternalVolumeStatus.VolumeType
fn := func(
fldPath *field.Path,
obj, oldObj *string,
@@ -3830,7 +3836,7 @@ func Validate_ExternalVolume(
return // do not proceed
}
// custom validation
- if e := ValidateCustom_ExternalVolume_VolumeType(ctx, op, fldPath, obj, oldObj); len(e) != 0 {
+ if e := ValidateCustom_ExternalVolumeStatus_VolumeType(ctx, op, fldPath, obj, oldObj); len(e) != 0 {
errs = append(errs, e...)
}
if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 253); len(e) != 0 {
@@ -3839,13 +3845,13 @@ func Validate_ExternalVolume(
return
}
oldVal := safe.Field(oldObj,
- func(oldObj *ateapipb.ExternalVolume) *string {
+ func(oldObj *ateapipb.ExternalVolumeStatus) *string {
return &oldObj.VolumeType
})
errs = append(errs, fn(fldPath.Child("volume_type"), &obj.VolumeType, oldVal, oldObj != nil)...)
}
- { // field ateapipb.ExternalVolume.VolumeContext
+ { // field ateapipb.ExternalVolumeStatus.VolumeContext
fn := func(
fldPath *field.Path,
obj, oldObj map[string]string,
@@ -3865,13 +3871,13 @@ func Validate_ExternalVolume(
return // do not proceed
}
// custom validation
- if e := ValidateCustom_ExternalVolume_VolumeContext(ctx, op, fldPath, obj, oldObj); len(e) != 0 {
+ if e := ValidateCustom_ExternalVolumeStatus_VolumeContext(ctx, op, fldPath, obj, oldObj); len(e) != 0 {
errs = append(errs, e...)
}
return
}
oldVal := safe.Field(oldObj,
- func(oldObj *ateapipb.ExternalVolume) map[string]string {
+ func(oldObj *ateapipb.ExternalVolumeStatus) map[string]string {
return oldObj.VolumeContext
})
errs = append(errs, fn(fldPath.Child("volume_context"), obj.VolumeContext, oldVal, oldObj != nil)...)
@@ -3957,6 +3963,61 @@ func Validate_ExternalVolumeTemplate(
return errs
}
+// Validate_ExternalVolumes validates an instance of ExternalVolumes according
+// to declarative validation rules in the API schema.
+func Validate_ExternalVolumes(
+ ctx context.Context, op operation.Operation, fldPath *field.Path,
+ obj, oldObj *ateapipb.ExternalVolumes) (errs field.ErrorList) {
+
+ { // field ateapipb.ExternalVolumes.Volumes
+ fn := func(
+ fldPath *field.Path,
+ obj, oldObj []*ateapipb.ExternalVolumeStatus,
+ oldValueCorrelated bool) (errs field.ErrorList) {
+ // don't revalidate unchanged data
+ if oldValueCorrelated && op.Type == operation.Update {
+ if ateDeepEqual(obj, oldObj) {
+ return nil
+ }
+ }
+ // call field-attached validations
+ earlyReturn := false
+ if e := validate.PtrSliceNoNils[ateapipb.ExternalVolumeStatus](ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 {
+ errs = append(errs, e...)
+ earlyReturn = true
+ }
+ if e := validate.MaxItems(ctx, op, fldPath, obj, oldObj, 32).MarkShortCircuit(); len(e) != 0 {
+ errs = append(errs, e...)
+ earlyReturn = true
+ }
+ if e := validate.OptionalSlice(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 {
+ earlyReturn = true
+ }
+ if earlyReturn {
+ return // do not proceed
+ }
+ // lists with map semantics require unique keys
+ if e := validate.PtrSliceUnique(ctx, op, fldPath, obj, oldObj,
+ func(a *ateapipb.ExternalVolumeStatus, b *ateapipb.ExternalVolumeStatus) bool { return a.Name == b.Name }); len(e) != 0 {
+ errs = append(errs, e...)
+ }
+ // iterate the list and call the type's validation function
+ if e := validate.EachPtrSliceVal(ctx, op, fldPath, obj, oldObj,
+ func(a *ateapipb.ExternalVolumeStatus, b *ateapipb.ExternalVolumeStatus) bool { return a.Name == b.Name }, ateDeepEqual, Validate_ExternalVolumeStatus); len(e) != 0 {
+ errs = append(errs, e...)
+ }
+ return
+ }
+ oldVal := safe.Field(oldObj,
+ func(oldObj *ateapipb.ExternalVolumes) []*ateapipb.ExternalVolumeStatus {
+ return oldObj.Volumes
+ })
+ errs = append(errs, fn(fldPath.Child("volumes"), obj.Volumes, oldVal, oldObj != nil)...)
+ }
+
+ return errs
+}
+
// Validate_GetActorEgressPolicyRequest validates an instance of GetActorEgressPolicyRequest according
// to declarative validation rules in the API schema.
func Validate_GetActorEgressPolicyRequest(
@@ -6581,6 +6642,175 @@ func Validate_RegisterWorkerRequest(
return errs
}
+// Validate_ReportActorCrashedRequest validates an instance of ReportActorCrashedRequest according
+// to declarative validation rules in the API schema.
+func Validate_ReportActorCrashedRequest(
+ ctx context.Context, op operation.Operation, fldPath *field.Path,
+ obj, oldObj *ateapipb.ReportActorCrashedRequest) (errs field.ErrorList) {
+
+ { // field ateapipb.ReportActorCrashedRequest.Worker
+ fn := func(
+ fldPath *field.Path,
+ obj, oldObj *ateapipb.ObjectRef,
+ oldValueCorrelated bool) (errs field.ErrorList) {
+ // don't revalidate unchanged data
+ if oldValueCorrelated && op.Type == operation.Update {
+ if ateDeepEqual(obj, oldObj) {
+ return nil
+ }
+ }
+ // call field-attached validations
+ earlyReturn := false
+ if e := validate.RequiredPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 {
+ errs = append(errs, e...)
+ earlyReturn = true
+ }
+ if earlyReturn {
+ return // do not proceed
+ }
+ func() { // cohort = "atespace"
+ earlyReturn := false
+ if e := validate.Subfield(ctx, op, fldPath, obj, oldObj, "atespace",
+ func(o *ateapipb.ObjectRef) *string { return &o.Atespace }, validate.DirectEqual, validate.ForbiddenValue).MarkBeta().MarkShortCircuit(); len(e) != 0 {
+ errs = append(errs, e...)
+ earlyReturn = true
+ }
+ if e := validate.Subfield(ctx, op, fldPath, obj, oldObj, "atespace",
+ func(o *ateapipb.ObjectRef) *string { return &o.Atespace }, validate.DirectEqual, validate.OptionalValue).MarkBeta().MarkShortCircuit(); len(e) != 0 {
+ earlyReturn = true
+ }
+ if e := validate.Subfield(ctx, op, fldPath, obj, oldObj, "atespace",
+ func(o *ateapipb.ObjectRef) *string { return &o.Atespace }, validate.DirectEqual, validate.OptionalValue).MarkBeta().MarkShortCircuit(); len(e) != 0 {
+ earlyReturn = true
+ }
+ if earlyReturn {
+ return // do not proceed
+ }
+ }()
+ // call the type's validation function
+ errs = append(errs, Validate_ObjectRef(ctx, op, fldPath, obj, oldObj)...)
+ return
+ }
+ oldVal := safe.Field(oldObj,
+ func(oldObj *ateapipb.ReportActorCrashedRequest) *ateapipb.ObjectRef {
+ return oldObj.Worker
+ })
+ errs = append(errs, fn(fldPath.Child("worker"), obj.Worker, oldVal, oldObj != nil)...)
+ }
+
+ { // field ateapipb.ReportActorCrashedRequest.Actor
+ fn := func(
+ fldPath *field.Path,
+ obj, oldObj *ateapipb.ObjectRef,
+ oldValueCorrelated bool) (errs field.ErrorList) {
+ // don't revalidate unchanged data
+ if oldValueCorrelated && op.Type == operation.Update {
+ if ateDeepEqual(obj, oldObj) {
+ return nil
+ }
+ }
+ // call field-attached validations
+ earlyReturn := false
+ if e := validate.RequiredPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 {
+ errs = append(errs, e...)
+ earlyReturn = true
+ }
+ if earlyReturn {
+ return // do not proceed
+ }
+ func() { // cohort = "atespace"
+ earlyReturn := false
+ if e := validate.Subfield(ctx, op, fldPath, obj, oldObj, "atespace",
+ func(o *ateapipb.ObjectRef) *string { return &o.Atespace }, validate.DirectEqual, validate.RequiredValue).MarkShortCircuit(); len(e) != 0 {
+ errs = append(errs, e...)
+ earlyReturn = true
+ }
+ if e := validate.Subfield(ctx, op, fldPath, obj, oldObj, "atespace",
+ func(o *ateapipb.ObjectRef) *string { return &o.Atespace }, validate.DirectEqual, validate.OptionalValue).MarkShortCircuit(); len(e) != 0 {
+ earlyReturn = true
+ }
+ if earlyReturn {
+ return // do not proceed
+ }
+ }()
+ // call the type's validation function
+ errs = append(errs, Validate_ObjectRef(ctx, op, fldPath, obj, oldObj)...)
+ return
+ }
+ oldVal := safe.Field(oldObj,
+ func(oldObj *ateapipb.ReportActorCrashedRequest) *ateapipb.ObjectRef {
+ return oldObj.Actor
+ })
+ errs = append(errs, fn(fldPath.Child("actor"), obj.Actor, oldVal, oldObj != nil)...)
+ }
+
+ { // field ateapipb.ReportActorCrashedRequest.ActorUid
+ fn := func(
+ fldPath *field.Path,
+ obj, oldObj *string,
+ oldValueCorrelated bool) (errs field.ErrorList) {
+ // don't revalidate unchanged data
+ if oldValueCorrelated && op.Type == operation.Update {
+ if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) {
+ return nil
+ }
+ }
+ // call field-attached validations
+ earlyReturn := false
+ if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 {
+ errs = append(errs, e...)
+ earlyReturn = true
+ }
+ if earlyReturn {
+ return // do not proceed
+ }
+ if e := validate.UUID(ctx, op, fldPath, obj, oldObj); len(e) != 0 {
+ errs = append(errs, e...)
+ }
+ return
+ }
+ oldVal := safe.Field(oldObj,
+ func(oldObj *ateapipb.ReportActorCrashedRequest) *string {
+ return &oldObj.ActorUid
+ })
+ errs = append(errs, fn(fldPath.Child("actor_uid"), &obj.ActorUid, oldVal, oldObj != nil)...)
+ }
+
+ { // field ateapipb.ReportActorCrashedRequest.Message
+ fn := func(
+ fldPath *field.Path,
+ obj, oldObj *string,
+ oldValueCorrelated bool) (errs field.ErrorList) {
+ // don't revalidate unchanged data
+ if oldValueCorrelated && op.Type == operation.Update {
+ if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) {
+ return nil
+ }
+ }
+ // call field-attached validations
+ earlyReturn := false
+ if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 {
+ errs = append(errs, e...)
+ earlyReturn = true
+ }
+ if earlyReturn {
+ return // do not proceed
+ }
+ if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 4096); len(e) != 0 {
+ errs = append(errs, e...)
+ }
+ return
+ }
+ oldVal := safe.Field(oldObj,
+ func(oldObj *ateapipb.ReportActorCrashedRequest) *string {
+ return &oldObj.Message
+ })
+ errs = append(errs, fn(fldPath.Child("message"), &obj.Message, oldVal, oldObj != nil)...)
+ }
+
+ return errs
+}
+
// Validate_ReportActorSnapshotStateRequest validates an instance of ReportActorSnapshotStateRequest according
// to declarative validation rules in the API schema.
func Validate_ReportActorSnapshotStateRequest(
@@ -7685,8 +7915,7 @@ func Validate_SnapshotConfig(
}
// call field-attached validations
earlyReturn := false
- if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 {
- errs = append(errs, e...)
+ if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 {
earlyReturn = true
}
if earlyReturn {
@@ -9223,7 +9452,7 @@ func Validate_VersionedSandboxCompat(
return errs
}
-var unionMembershipFor_go_ate_dev_substrate_pkg_proto_ateapipb_Volume_ = validate.NewUnionMembership(validate.NewUnionMember("durable_dir"), validate.NewUnionMember("external_volume_template"), validate.NewUnionMember("system_info"), validate.NewUnionMember("image"))
+var unionMembershipFor_go_ate_dev_substrate_pkg_proto_ateapipb_Volume_ = validate.NewUnionMembership(validate.NewUnionMember("durable_dir"), validate.NewUnionMember("external"), validate.NewUnionMember("system_info"), validate.NewUnionMember("image"))
// Validate_Volume validates an instance of Volume according
// to declarative validation rules in the API schema.
@@ -9242,7 +9471,7 @@ func Validate_Volume(
if obj == nil {
return false
}
- return obj.ExternalVolumeTemplate != nil
+ return obj.External != nil
},
func(obj *ateapipb.Volume) bool {
if obj == nil {
@@ -9319,10 +9548,10 @@ func Validate_Volume(
errs = append(errs, fn(fldPath.Child("durable_dir"), obj.DurableDir, oldVal, oldObj != nil)...)
}
- { // field ateapipb.Volume.ExternalVolumeTemplate
+ { // field ateapipb.Volume.External
fn := func(
fldPath *field.Path,
- obj, oldObj *ateapipb.ExternalVolumeTemplate,
+ obj, oldObj *ateapipb.ExternalVolumeSource,
oldValueCorrelated bool) (errs field.ErrorList) {
// don't revalidate unchanged data
if oldValueCorrelated && op.Type == operation.Update {
@@ -9346,14 +9575,14 @@ func Validate_Volume(
return // do not proceed
}
// call the type's validation function
- errs = append(errs, Validate_ExternalVolumeTemplate(ctx, op, fldPath, obj, oldObj)...)
+ errs = append(errs, Validate_ExternalVolumeSource(ctx, op, fldPath, obj, oldObj)...)
return
}
oldVal := safe.Field(oldObj,
- func(oldObj *ateapipb.Volume) *ateapipb.ExternalVolumeTemplate {
- return oldObj.ExternalVolumeTemplate
+ func(oldObj *ateapipb.Volume) *ateapipb.ExternalVolumeSource {
+ return oldObj.External
})
- errs = append(errs, fn(fldPath.Child("external_volume_template"), obj.ExternalVolumeTemplate, oldVal, oldObj != nil)...)
+ errs = append(errs, fn(fldPath.Child("external"), obj.External, oldVal, oldObj != nil)...)
}
{ // field ateapipb.Volume.SystemInfo
diff --git a/cmd/ateapi/internal/authz/registry.go b/cmd/ateapi/internal/authz/registry.go
index 4acacf82cf..fb25590eb9 100644
--- a/cmd/ateapi/internal/authz/registry.go
+++ b/cmd/ateapi/internal/authz/registry.go
@@ -335,10 +335,9 @@ var defaultRPCPermissions = map[string]rpcRule{
// WorkerService is served only to atelet, which each handler
// authenticates by its mTLS SPIFFE ID.
ateapipb.WorkerService_RegisterWorker_FullMethodName: inHandler("ateletauth: caller is atelet on the Worker's node"),
+ ateapipb.WorkerService_ReportActorCrashed_FullMethodName: inHandler("ateletauth: caller is atelet on the node of the Worker hosting the Actor"),
ateapipb.WorkerService_RequestActorSuspend_FullMethodName: inHandler("ateletauth: caller is atelet on the node of the Worker hosting the Actor"),
ateapipb.WorkerService_MintWorkerActorCertificate_FullMethodName: inHandler("ateletauth: caller is atelet on the node of the Worker the Actor is assigned to"),
ateapipb.WorkerService_MintWorkerActorJwt_FullMethodName: inHandler("ateletauth: caller is atelet on the node hosting the Actor"),
- // TODO(#2570): not served yet; the handler must authenticate atelet on the
- // node of the Worker hosting the Actor when it lands.
- ateapipb.WorkerService_ReportActorSnapshotState_FullMethodName: inHandler("ateletauth: caller is atelet on the node of the Worker hosting the Actor"),
+ ateapipb.WorkerService_ReportActorSnapshotState_FullMethodName: inHandler("ateletauth: caller is atelet on the node the Actor is placed on or holds a local snapshot of it"),
}
diff --git a/cmd/ateapi/internal/controlapi/access_policy_test.go b/cmd/ateapi/internal/controlapi/access_policy_test.go
index b0421c4fae..f3e39d81aa 100644
--- a/cmd/ateapi/internal/controlapi/access_policy_test.go
+++ b/cmd/ateapi/internal/controlapi/access_policy_test.go
@@ -48,7 +48,7 @@ func TestAccessPolicy_GlobalAndAtespaceGovernance(t *testing.T) {
}
persistence.SetPolicyManager(policyManager)
- svc := NewRPCService(persistence, nil, nil, nil, nil, nil, nil, nil, "", 0, nil, nil, "", nil, nil)
+ svc := NewRPCService(persistence, nil, nil, nil, nil, nil, "", 0, nil, nil, "", nil, nil, "")
interceptor := authz.UnaryServerInterceptor(authorizer, true)
userCtx := func(id string) context.Context {
diff --git a/cmd/ateapi/internal/controlapi/actor.go b/cmd/ateapi/internal/controlapi/actor.go
index 42ed43295e..3bef57aeb6 100644
--- a/cmd/ateapi/internal/controlapi/actor.go
+++ b/cmd/ateapi/internal/controlapi/actor.go
@@ -96,7 +96,7 @@ func (s *ServiceImpl) CreateActor(ctx context.Context, inActor *ateapipb.Actor)
// NOTE: whether the ExternalVolumes gate is enabled or not, we do not
// support cloning from a tag that has external volumes.
for _, volume := range template.GetVolumes() {
- if volume.GetExternalVolumeTemplate() != nil {
+ if volume.GetExternal().GetTemplate() != nil {
// TODO: Permit cloning after CSI volume snapshots are supported.
return nil, apierror.FailedPrecondition("Tag cloning does not support ActorTemplates with external volumes")
}
@@ -114,7 +114,7 @@ func (s *ServiceImpl) CreateActor(ctx context.Context, inActor *ateapipb.Actor)
name := inActor.GetMetadata().GetName()
// Volume creation is completed asynchronously after the actor is recorded.
- var initVols []*ateapipb.ExternalVolume
+ var initVols []*ateapipb.ExternalVolumeStatus
if iv, err := initialExternalVolumes(ctx, s.storageClassLister, template); err != nil {
return nil, err
} else if preview.IsEnabledFor(ctx, preview.GateExternalVolumes) {
@@ -131,7 +131,7 @@ func (s *ServiceImpl) CreateActor(ctx context.Context, inActor *ateapipb.Actor)
outActor := proto.CloneOf(inActor)
outActor.Status = &ateapipb.ActorStatus{
State: ateapipb.ActorState_ACTOR_STATE_SUSPENDED,
- ExternalVolumes: initVols,
+ ExternalVolumes: externalVolumesStatus(initVols),
}
if sourceTag != nil {
// The Actor starts out borrowing the tag's snapshot rather than
@@ -316,7 +316,7 @@ func (s *ServiceImpl) UpdateActor(ctx context.Context, actorRef resources.ActorR
if err != nil {
return err
}
- if err := validateSnapshotLocationUnchanged(oldVal, newTemplate); err != nil {
+ if err := s.validateSnapshotLocationUnchanged(oldVal, newTemplate); err != nil {
return err
}
oldTemplate, err := resolveActorTemplate(ctx, s.store, oldVal)
@@ -397,7 +397,7 @@ func validateTemplateVolumesUnchanged(oldTemplate, newTemplate *ateapipb.ActorTe
// already owns. This is needed to not leak snapshots when the actor is deleted:
// Deleting an actor collects everything under its external snapshot prefix. If
// the location prefix ever changes, we risk leaking the snapshots under the old prefix.
-func validateSnapshotLocationUnchanged(actor *ateapipb.Actor, newTemplate *ateapipb.ActorTemplate) error {
+func (s *ServiceImpl) validateSnapshotLocationUnchanged(actor *ateapipb.Actor, newTemplate *ateapipb.ActorTemplate) error {
snap := findLatestDurableSnapshot(actor.GetStatus(), ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED)
currentSnapshotURI := snap.GetDurableSnapshot().GetObject().GetSnapshotUri()
if currentSnapshotURI == "" {
@@ -413,7 +413,10 @@ func validateSnapshotLocationUnchanged(actor *ateapipb.Actor, newTemplate *ateap
}
// Compared as prefixes, so a newLocation spelled with and without a
// trailing slash counts as the same.
- newLocation := newTemplate.GetSnapshotConfig().GetStorageLocation()
+ newLocation, err := resolveSnapshotLocation(s.clusterSnapshotLocation, newTemplate)
+ if err != nil {
+ return err
+ }
// Generate what the new snapshot prefix would look like for this actor.
nextSnapshotLocationPrefix, err := currentURI.Owner().Prefix(newLocation)
if err != nil {
@@ -434,7 +437,7 @@ func (s *RPCService) DeleteActor(ctx context.Context, req *ateapipb.DeleteActorR
start := time.Now()
// Template dims only once the record resolved: the request names only the
// actor, so failures before the load carry none. No pool pair, even when
- // any_state deletes an actor that still holds a worker.
+ // the deleted actor still held a worker.
defer func() {
var attrs []attribute.KeyValue
if deleted != nil {
@@ -448,7 +451,7 @@ func (s *RPCService) DeleteActor(ctx context.Context, req *ateapipb.DeleteActorR
actorRef := resources.ActorRefFromObjectRef(req.GetActor())
setSpanActorRefAttributes(ctx, actorRef)
- deleted, err = s.actorWorkflow.DeleteActor(ctx, actorRef, req.GetAnyState(), toDeletePreconditions(req.GetOptions()))
+ deleted, err = s.actorWorkflow.DeleteActor(ctx, actorRef, toDeletePreconditions(req.GetOptions()))
if err != nil {
return nil, err
}
@@ -503,6 +506,21 @@ func (s *RPCService) SuspendActor(ctx context.Context, req *ateapipb.SuspendActo
return &ateapipb.SuspendActorResponse{Actor: actor}, nil
}
+// CrashActorReportedByWorker crashes a RUNNING actor whose sandbox died, on
+// behalf of the worker hosting it. The caller authorizes the worker.
+func (s *RPCService) CrashActorReportedByWorker(ctx context.Context, actorRef resources.ActorRef, actorUID, workerName, message string) (*ateapipb.Actor, error) {
+ setSpanActorRefAttributes(ctx, actorRef)
+ actor, err := s.actorWorkflow.CrashActorReportedByWorker(ctx, actorRef, actorUID, workerName, message)
+ if err != nil {
+ if errors.Is(err, store.ErrVersionConflict) {
+ return nil, apierror.Aborted("concurrent update conflict, please retry")
+ }
+ return nil, err
+ }
+ setSpanActorAttributes(ctx, actor)
+ return actor, nil
+}
+
func (s *RPCService) RevertActor(ctx context.Context, req *ateapipb.RevertActorRequest) (*ateapipb.RevertActorResponse, error) {
if errs := apivalidation.ValidateRevertActorRequest(ctx, req); len(errs) > 0 {
return nil, resources.ToAPIError(errs)
diff --git a/cmd/ateapi/internal/controlapi/actor_template.go b/cmd/ateapi/internal/controlapi/actor_template.go
index b9556f92f4..45df2feb5a 100644
--- a/cmd/ateapi/internal/controlapi/actor_template.go
+++ b/cmd/ateapi/internal/controlapi/actor_template.go
@@ -70,6 +70,12 @@ func (s *ServiceImpl) CreateActorTemplate(ctx context.Context, inTemplate *ateap
return nil, toGRPCInternalError(errs)
}
+ // Reject a template whose actors could never be snapshotted, rather than
+ // failing their first suspend or tag.
+ if _, err := resolveSnapshotLocation(s.clusterSnapshotLocation, outTemplate); err != nil {
+ return nil, err
+ }
+
return s.store.CreateActorTemplate(ctx, outTemplate)
}
diff --git a/cmd/ateapi/internal/controlapi/actor_template_test.go b/cmd/ateapi/internal/controlapi/actor_template_test.go
index b848388504..857915f7c4 100644
--- a/cmd/ateapi/internal/controlapi/actor_template_test.go
+++ b/cmd/ateapi/internal/controlapi/actor_template_test.go
@@ -17,6 +17,7 @@ package controlapi
import (
"context"
"errors"
+ "fmt"
"strings"
"testing"
@@ -25,6 +26,7 @@ import (
"go.ate.dev/substrate/cmd/ateapi/internal/store/storetest"
"go.ate.dev/substrate/internal/apierror"
"go.ate.dev/substrate/internal/objectstoreplugin/objectstoreplugintest"
+ "go.ate.dev/substrate/internal/preview"
"go.ate.dev/substrate/internal/resources"
"go.ate.dev/substrate/pkg/proto/ateapipb"
"google.golang.org/grpc/codes"
@@ -79,6 +81,72 @@ func TestCreateActorTemplate(t *testing.T) {
}
}
+// TestCreateActorTemplate_RequiresSnapshotLocation pins that create rejects,
+// with FailedPrecondition, a template whose actors would have no snapshot
+// location: neither an honored storageLocation nor --snapshot-location.
+func TestCreateActorTemplate_RequiresSnapshotLocation(t *testing.T) {
+ tests := []struct {
+ name string
+ gates []string
+ optOut bool
+ clusterSnapshotLocation string
+ location string
+ wantCode codes.Code
+ }{{
+ name: "gate off, cluster location set",
+ clusterSnapshotLocation: "gs://cluster-bucket/",
+ location: "gs://my-bucket/snapshots",
+ wantCode: codes.OK,
+ }, {
+ name: "gate off, no cluster location",
+ location: "gs://my-bucket/snapshots",
+ wantCode: codes.FailedPrecondition,
+ }, {
+ name: "gate on, location set, no cluster location",
+ gates: []string{"PerTemplateSnapshotLocation"},
+ location: "gs://my-bucket/snapshots",
+ wantCode: codes.OK,
+ }, {
+ name: "gate on, no location, cluster location set",
+ gates: []string{"PerTemplateSnapshotLocation"},
+ clusterSnapshotLocation: "gs://cluster-bucket/",
+ wantCode: codes.OK,
+ }, {
+ name: "gate on, opted out, no location, cluster location set",
+ gates: []string{"PerTemplateSnapshotLocation"},
+ optOut: true,
+ clusterSnapshotLocation: "gs://cluster-bucket/",
+ wantCode: codes.OK,
+ }, {
+ name: "gate on, no location, no cluster location",
+ gates: []string{"PerTemplateSnapshotLocation"},
+ wantCode: codes.FailedPrecondition,
+ }}
+ for i, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ preview.InitForTest(t, tt.gates...)
+ persistence := newTestPersistence(t)
+ impl := newServiceImpl(persistence, nil)
+ impl.clusterSnapshotLocation = tt.clusterSnapshotLocation
+ s := &RPCService{impl: impl}
+ ctx := context.Background()
+ if _, err := persistence.CreateAtespace(ctx, &ateapipb.Atespace{Metadata: &ateapipb.ResourceMetadata{Name: "ns1"}}); err != nil {
+ t.Fatalf("CreateAtespace failed: %v", err)
+ }
+ if tt.optOut {
+ ctx = preview.ContextWithOptOut(ctx)
+ }
+ req := &ateapipb.CreateActorTemplateRequest{ActorTemplate: validActorTemplate(func(tmpl *ateapipb.ActorTemplate) {
+ tmpl.Metadata = &ateapipb.ResourceMetadata{Atespace: "ns1", Name: fmt.Sprintf("tmpl-%d", i)}
+ tmpl.SnapshotConfig.StorageLocation = tt.location
+ })}
+ if _, err := s.CreateActorTemplate(ctx, req); apierror.Code(err) != tt.wantCode {
+ t.Errorf("CreateActorTemplate error = %v, want code %v", err, tt.wantCode)
+ }
+ })
+ }
+}
+
// TestCreateActorTemplateIgnoresServerOwnedFields pins the create contract:
// status on the request is dropped and new templates start with an empty
// status. The store persists whatever the handler hands it, so the handler is
@@ -126,6 +194,33 @@ func TestCreateActorTemplateIgnoresServerOwnedFields(t *testing.T) {
}
}
+// TestCreateActorTemplateDefaultsOmittedSnapshotConfig pins that
+// snapshot_config is optional: create fills an omitted one with every default.
+func TestCreateActorTemplateDefaultsOmittedSnapshotConfig(t *testing.T) {
+ persistence := newTestPersistence(t)
+ impl := newServiceImpl(persistence, nil)
+ impl.clusterSnapshotLocation = "gs://cluster-bucket/"
+ s := &RPCService{impl: impl}
+ ctx := context.Background()
+ if _, err := persistence.CreateAtespace(ctx, &ateapipb.Atespace{Metadata: &ateapipb.ResourceMetadata{Name: "ns1"}}); err != nil {
+ t.Fatalf("CreateAtespace failed: %v", err)
+ }
+
+ created, err := s.CreateActorTemplate(ctx, &ateapipb.CreateActorTemplateRequest{ActorTemplate: validActorTemplate(func(tmpl *ateapipb.ActorTemplate) {
+ tmpl.SnapshotConfig = nil
+ })})
+ if err != nil {
+ t.Fatalf("CreateActorTemplate failed: %v", err)
+ }
+ want := &ateapipb.SnapshotConfig{
+ PreferredFidelity: ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY,
+ GoldenSnapshotConfig: &ateapipb.GoldenSnapshotConfig{Mode: ateapipb.GoldenSnapshotMode_GOLDEN_SNAPSHOT_MODE_ENABLED},
+ }
+ if diff := cmp.Diff(want, created.GetSnapshotConfig(), protocmp.Transform()); diff != "" {
+ t.Errorf("CreateActorTemplate snapshot_config mismatch (-want +got):\n%s", diff)
+ }
+}
+
func TestDeleteActorTemplate(t *testing.T) {
tests := []struct {
name string
@@ -181,12 +276,12 @@ func TestDeleteActorTemplate(t *testing.T) {
tagURI := mustReservedTagSnapshotURI(t, tag)
objects.PutSnapshot(t, tagURI, "manifest.json")
svc := &RPCService{impl: newServiceImpl(persistence, nil), actorWorkflow: workflow, objectStorePlugin: objectstoreplugintest.ControlClient(objects)}
- // The handler must request AnyState to clean up an active golden actor.
+ // The handler must clean up an active golden actor.
mustUpdateActorStatus(t, ctx, persistence, actor, func(s *ateapipb.ActorStatus) {
s.State = ateapipb.ActorState_ACTOR_STATE_RUNNING
})
if tt.actorDeleted {
- if _, err := workflow.DeleteActor(ctx, goldenRef, true, store.DeletePreconditions{}); err != nil {
+ if _, err := workflow.DeleteActor(ctx, goldenRef, store.DeletePreconditions{}); err != nil {
t.Fatal(err)
}
}
diff --git a/cmd/ateapi/internal/controlapi/actor_test.go b/cmd/ateapi/internal/controlapi/actor_test.go
index 328f6430a4..d0cca17b95 100644
--- a/cmd/ateapi/internal/controlapi/actor_test.go
+++ b/cmd/ateapi/internal/controlapi/actor_test.go
@@ -33,6 +33,7 @@ import (
"go.ate.dev/substrate/internal/apierror"
"go.ate.dev/substrate/internal/ateattr"
"go.ate.dev/substrate/internal/localca"
+ "go.ate.dev/substrate/internal/preview"
"go.ate.dev/substrate/internal/resources"
"go.ate.dev/substrate/internal/substratex509"
"go.ate.dev/substrate/pkg/proto/ateapipb"
@@ -411,7 +412,9 @@ func TestUpdateActor_RepointTemplate(t *testing.T) {
// a template storing snapshots under the same location, so deleting the actor
// still collects everything it wrote. Stored data the check cannot parse is a
// server fault: the error carries no status, which ServerUnaryInterceptor
-// answers with INTERNAL.
+// answers with INTERNAL. With the PerTemplateSnapshotLocation preview gate off,
+// every template resolves to --snapshot-location, so template locations
+// cannot block a repoint.
func TestUpdateActor_RepointTemplateStorageLocation(t *testing.T) {
ctx := context.Background()
persistence, cleanup := storetest.SetupTestStore(t)
@@ -464,7 +467,11 @@ func TestUpdateActor_RepointTemplateStorageLocation(t *testing.T) {
template string
snapshot func(*testing.T, *ateapipb.Actor) string
repointTo string
- wantCode codes.Code
+ // gateOff disables the PerTemplateSnapshotLocation preview gate, so
+ // every template resolves to clusterSnapshotLocation.
+ gateOff bool
+ clusterSnapshotLocation string
+ wantCode codes.Code
}{
{
name: "no external snapshot moves freely",
@@ -514,10 +521,41 @@ func TestUpdateActor_RepointTemplateStorageLocation(t *testing.T) {
repointTo: "corrupt",
wantCode: codes.Internal,
},
+ {
+ name: "gate off: owned snapshot moves between template locations, which are ignored",
+ template: "same-location",
+ snapshot: snapshotOwnedByActor,
+ repointTo: "different-location",
+ gateOff: true,
+ clusterSnapshotLocation: sameLocation,
+ wantCode: codes.OK,
+ },
+ {
+ name: "gate off: an invalid template location is ignored",
+ template: "same-location",
+ snapshot: snapshotOwnedByActor,
+ repointTo: "corrupt",
+ gateOff: true,
+ clusterSnapshotLocation: sameLocation,
+ wantCode: codes.OK,
+ },
+ {
+ name: "gate off without --snapshot-location has no location to check",
+ template: "same-location",
+ snapshot: snapshotOwnedByActor,
+ repointTo: "different-location",
+ gateOff: true,
+ wantCode: codes.FailedPrecondition,
+ },
}
- svc := &RPCService{impl: newServiceImpl(persistence, nil)}
for i, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
+ if tt.gateOff {
+ preview.InitForTest(t)
+ }
+ impl := newServiceImpl(persistence, nil)
+ impl.clusterSnapshotLocation = tt.clusterSnapshotLocation
+ svc := &RPCService{impl: impl}
actor := storetest.MustCreateActor(t, ctx, persistence, &ateapipb.Actor{
Metadata: &ateapipb.ResourceMetadata{Atespace: testAtespace, Name: fmt.Sprintf("actor-%d", i)},
ActorTemplate: &ateapipb.ObjectRef{Atespace: testAtespace, Name: tt.template},
diff --git a/cmd/ateapi/internal/controlapi/crash.go b/cmd/ateapi/internal/controlapi/crash.go
index 5b505d0355..9aeee6aea0 100644
--- a/cmd/ateapi/internal/controlapi/crash.go
+++ b/cmd/ateapi/internal/controlapi/crash.go
@@ -39,6 +39,7 @@ const (
crashMessageLocalSnapshotNodeUnknown = "node holding the actor's local snapshot is unknown"
crashMessageLocalSnapshotMissing = "actor has no local snapshot"
crashMessageDurableSnapshotMissing = "actor has no durable snapshot"
+ crashMessageSnapshotStateConflict = "atelet's snapshot state conflicts with the actor's record"
crashMessageWorkerGone = "assigned worker no longer exists"
crashMessageWorkerDraining = "assigned worker is draining"
crashMessageWorkerReassigned = "assigned worker no longer hosts the actor"
diff --git a/cmd/ateapi/internal/controlapi/functionaltest/actor_template_test.go b/cmd/ateapi/internal/controlapi/functionaltest/actor_template_test.go
index 68a27b5dba..d86759c0b0 100644
--- a/cmd/ateapi/internal/controlapi/functionaltest/actor_template_test.go
+++ b/cmd/ateapi/internal/controlapi/functionaltest/actor_template_test.go
@@ -208,7 +208,7 @@ func TestGoldenTagLifecycle(t *testing.T) {
}
assertSnapshotPresent(t, tc, uri)
for _, actor := range []*ateapipb.Actor{early, late} {
- if _, err := tc.client.DeleteActor(ctx, &ateapipb.DeleteActorRequest{Actor: resources.ActorRefFromActor(actor).ToObjectRef(), AnyState: true}); err != nil {
+ if _, err := tc.client.DeleteActor(ctx, &ateapipb.DeleteActorRequest{Actor: resources.ActorRefFromActor(actor).ToObjectRef()}); err != nil {
t.Fatal(err)
}
}
@@ -314,7 +314,7 @@ func TestActorLifecycleWithoutGoldenSnapshot(t *testing.T) {
if err := assertNoGolden(ctx); err != nil {
t.Fatal(err)
}
- if _, err := tc.client.DeleteActor(ctx, &ateapipb.DeleteActorRequest{Actor: ref, AnyState: true}); err != nil {
+ if _, err := tc.client.DeleteActor(ctx, &ateapipb.DeleteActorRequest{Actor: ref}); err != nil {
t.Fatal(err)
}
if _, err := tc.client.DeleteActorTemplate(ctx, &ateapipb.DeleteActorTemplateRequest{ActorTemplate: resources.ActorTemplateRefFromActorTemplate(tmpl).ToObjectRef()}); err != nil {
@@ -406,8 +406,8 @@ func TestCreateActorTemplate_ExternalVolumesPreview(t *testing.T) {
VolumeMounts: []*ateapipb.VolumeMount{{Name: "data", MountPath: "/data"}},
}},
Volumes: []*ateapipb.Volume{{
- Name: "data",
- ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{StorageClassName: "standard", Capacity: "1Gi"},
+ Name: "data",
+ External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{StorageClassName: "standard", Capacity: "1Gi"}},
}},
Resources: &ateapipb.Resources{Limits: []*ateapipb.Limits{{Name: "cpu", Quantity: "1"}, {Name: "memory", Quantity: "1Gi"}}},
SnapshotConfig: &ateapipb.SnapshotConfig{StorageLocation: "gs://my-bucket/snapshots"},
@@ -419,3 +419,52 @@ func TestCreateActorTemplate_ExternalVolumesPreview(t *testing.T) {
})
}
}
+
+// TestCreateActorTemplate_PerTemplateSnapshotLocationPreview pins how
+// snapshotConfig.storageLocation combines the PerTemplateSnapshotLocation gate
+// with the client's preview opt-in. With the gate off the field is ignored,
+// so any client may set it. With the gate on, only a client that opts in may
+// set it; a template that leaves it empty uses the server's
+// --snapshot-location.
+func TestCreateActorTemplate_PerTemplateSnapshotLocationPreview(t *testing.T) {
+ ns := namespaceForTest("ns-template-storage-location-preview")
+ tc := setupTest(t, ns)
+ defer tc.cleanup()
+
+ for i, tt := range []struct {
+ name string
+ gates []string
+ optIn bool
+ location string
+ wantCode codes.Code
+ }{
+ {name: "gate off, no opt-in, location set", optIn: false, location: "gs://my-bucket/snapshots", wantCode: codes.OK},
+ {name: "gate off, opt-in, location set", optIn: true, location: "gs://my-bucket/snapshots", wantCode: codes.OK},
+ {name: "gate on, no opt-in, location set", gates: []string{"PerTemplateSnapshotLocation"}, optIn: false, location: "gs://my-bucket/snapshots", wantCode: codes.InvalidArgument},
+ {name: "gate on, no opt-in, no location", gates: []string{"PerTemplateSnapshotLocation"}, optIn: false, wantCode: codes.OK},
+ {name: "gate on, opt-in, location set", gates: []string{"PerTemplateSnapshotLocation"}, optIn: true, location: "gs://my-bucket/snapshots", wantCode: codes.OK},
+ {name: "gate on, opt-in, no location", gates: []string{"PerTemplateSnapshotLocation"}, optIn: true, wantCode: codes.OK},
+ } {
+ t.Run(tt.name, func(t *testing.T) {
+ preview.InitForTest(t, tt.gates...)
+ ctx := context.Background()
+ if tt.optIn {
+ ctx = metadata.AppendToOutgoingContext(ctx, preview.Header, "true")
+ }
+ _, err := tc.client.CreateActorTemplate(ctx, &ateapipb.CreateActorTemplateRequest{
+ ActorTemplate: &ateapipb.ActorTemplate{
+ Metadata: &ateapipb.ResourceMetadata{Atespace: testAtespace, Name: fmt.Sprintf("tmpl-loc-%d", i)},
+ Containers: []*ateapipb.Container{{
+ Name: "main",
+ Image: "example.com/app:v1@sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
+ }},
+ Resources: &ateapipb.Resources{Limits: []*ateapipb.Limits{{Name: "cpu", Quantity: "1"}, {Name: "memory", Quantity: "1Gi"}}},
+ SnapshotConfig: &ateapipb.SnapshotConfig{StorageLocation: tt.location},
+ },
+ })
+ if status.Code(err) != tt.wantCode {
+ t.Fatalf("CreateActorTemplate error = %v, want %v", err, tt.wantCode)
+ }
+ })
+ }
+}
diff --git a/cmd/ateapi/internal/controlapi/functionaltest/actor_test.go b/cmd/ateapi/internal/controlapi/functionaltest/actor_test.go
index 4a6cefcf38..bae2613d37 100644
--- a/cmd/ateapi/internal/controlapi/functionaltest/actor_test.go
+++ b/cmd/ateapi/internal/controlapi/functionaltest/actor_test.go
@@ -121,10 +121,10 @@ func TestCreateActor_WithExternalVolumes(t *testing.T) {
volumes := []*ateapipb.Volume{
{
Name: "ext-vol-1",
- ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{
+ External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{
StorageClassName: "standard",
Capacity: "10Gi",
- },
+ }},
},
}
mounts := []*ateapipb.VolumeMount{
@@ -145,10 +145,10 @@ func TestCreateActor_WithExternalVolumes(t *testing.T) {
t.Fatalf("CreateActor failed: %v", err)
}
- if len(createResp.GetStatus().GetExternalVolumes()) != 1 {
- t.Fatalf("expected 1 volume in CreateActor response, got %d", len(createResp.GetStatus().GetExternalVolumes()))
+ if len(createResp.GetStatus().GetExternalVolumes().GetVolumes()) != 1 {
+ t.Fatalf("expected 1 volume in CreateActor response, got %d", len(createResp.GetStatus().GetExternalVolumes().GetVolumes()))
}
- vol := createResp.GetStatus().GetExternalVolumes()[0]
+ vol := createResp.GetStatus().GetExternalVolumes().GetVolumes()[0]
if vol.GetName() != "ext-vol-1" {
t.Errorf("volume name = %q, want %q", vol.GetName(), "ext-vol-1")
}
@@ -163,10 +163,10 @@ func TestCreateActor_WithExternalVolumes(t *testing.T) {
if err != nil {
t.Fatalf("GetActor failed: %v", err)
}
- if len(getResp.GetStatus().GetExternalVolumes()) != 1 {
- t.Fatalf("expected 1 volume in GetActor response, got %d", len(getResp.GetStatus().GetExternalVolumes()))
+ if len(getResp.GetStatus().GetExternalVolumes().GetVolumes()) != 1 {
+ t.Fatalf("expected 1 volume in GetActor response, got %d", len(getResp.GetStatus().GetExternalVolumes().GetVolumes()))
}
- if got := getResp.GetStatus().GetExternalVolumes()[0].GetStorageVolumeId(); got != "" {
+ if got := getResp.GetStatus().GetExternalVolumes().GetVolumes()[0].GetStorageVolumeId(); got != "" {
t.Errorf("GetActor storageVolumeId = %q, want empty before resume", got)
}
}
@@ -183,8 +183,8 @@ func TestCreateActor_ExternalVolumesWithoutOptIn(t *testing.T) {
createTemplateWithVolumes(t, tc, ns,
[]*ateapipb.Volume{{
- Name: "ext-vol-1",
- ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{StorageClassName: "standard", Capacity: "10Gi"},
+ Name: "ext-vol-1",
+ External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{StorageClassName: "standard", Capacity: "10Gi"}},
}},
[]*ateapipb.VolumeMount{{Name: "ext-vol-1", MountPath: "/data"}})
@@ -215,8 +215,8 @@ func TestActorLifecycle_ExternalVolumesWithoutOptIn(t *testing.T) {
createTemplateWithVolumes(t, tc, ns,
[]*ateapipb.Volume{{
- Name: "data-vol",
- ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{StorageClassName: "fast", Capacity: "20Gi"},
+ Name: "data-vol",
+ External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{StorageClassName: "fast", Capacity: "20Gi"}},
}},
[]*ateapipb.VolumeMount{{Name: "data-vol", MountPath: "/mnt/data"}})
createWorkerPod(t, tc, ns, "worker-1", "node1", "pool1")
@@ -406,9 +406,9 @@ func TestCreateActor_RejectsSnapshotWithExternalVolumes(t *testing.T) {
}},
Volumes: []*ateapipb.Volume{{
Name: "data",
- ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{
+ External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{
Capacity: "1Gi", StorageClassName: "standard",
- },
+ }},
}},
Resources: resources.CPUMemory(1000, 512<<20),
},
@@ -1228,33 +1228,92 @@ func TestDeleteActor_MissingSnapshotBucket(t *testing.T) {
}
}
-func TestDeleteActor_NotSuspended(t *testing.T) {
- ns := namespaceForTest("ns-delete-notsuspended")
- tc := setupTest(t, ns)
+// TestDeleteActor_Running deletes a running actor that owns an external
+// snapshot and an external volume. The delete terminates the workload, detaches
+// and deletes the volume, collects the snapshot, and releases the worker.
+func TestDeleteActor_Running(t *testing.T) {
+ preview.InitForTest(t, "ExternalVolumes")
+
+ ns := namespaceForTest("ns-delete-running")
+ plugin := &attachFailVolumePlugin{}
+ tc := setupTestWithVolumePlugins(t, ns, map[string]volume.VolumePluginControlPlane{
+ "substrate.io/mock": plugin,
+ })
defer tc.cleanup()
- createTemplate(t, tc, ns)
+ createTemplateWithVolumes(t, tc, ns, []*ateapipb.Volume{
+ {
+ Name: "vol1",
+ External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{
+ StorageClassName: "standard",
+ Capacity: "10Gi",
+ }},
+ },
+ }, []*ateapipb.VolumeMount{
+ {Name: "vol1", MountPath: "/mnt/vol1"},
+ })
createWorkerPod(t, tc, ns, "worker-1", "node1", "pool1")
- _, err := tc.client.CreateActor(context.Background(), &ateapipb.CreateActorRequest{Actor: &ateapipb.Actor{
+ ctx := context.Background()
+ actorRef := &ateapipb.ObjectRef{Atespace: testAtespace, Name: "id1"}
+ if _, err := tc.client.CreateActor(ctx, &ateapipb.CreateActorRequest{Actor: &ateapipb.Actor{
Metadata: &ateapipb.ResourceMetadata{Atespace: testAtespace, Name: "id1"},
ActorTemplate: &ateapipb.ObjectRef{Atespace: testAtespace, Name: "tmpl1"},
- }})
- if err != nil {
+ }}); err != nil {
t.Fatalf("CreateActor failed: %v", err)
}
- _, err = tc.client.ResumeActor(context.Background(), &ateapipb.ResumeActorRequest{
- Actor: &ateapipb.ObjectRef{Atespace: testAtespace, Name: "id1"},
- })
- if err != nil {
+ // Suspend once so the actor owns an external snapshot, then resume it.
+ if _, err := tc.client.ResumeActor(ctx, &ateapipb.ResumeActorRequest{Actor: actorRef}); err != nil {
t.Fatalf("ResumeActor failed: %v", err)
}
+ if _, err := tc.client.SuspendActor(ctx, &ateapipb.SuspendActorRequest{
+ Actor: actorRef,
+ Options: &ateapipb.SuspendOptions{Mode: ateapipb.DurabilityMode_DURABILITY_MODE_SYNCHRONOUS},
+ }); err != nil {
+ t.Fatalf("SuspendActor failed: %v", err)
+ }
+ if _, err := tc.client.ResumeActor(ctx, &ateapipb.ResumeActorRequest{Actor: actorRef}); err != nil {
+ t.Fatalf("second ResumeActor failed: %v", err)
+ }
+ running, err := tc.client.GetActor(ctx, &ateapipb.GetActorRequest{Actor: actorRef})
+ if err != nil {
+ t.Fatalf("GetActor failed: %v", err)
+ }
+ if got := running.GetStatus().GetState(); got != ateapipb.ActorState_ACTOR_STATE_RUNNING {
+ t.Fatalf("actor state = %v, want ACTOR_STATE_RUNNING", got)
+ }
+ actorUID := running.GetMetadata().GetUid()
+ snapshotURI := durableSnapshotURI(running.GetStatus())
+ if snapshotURI == "" {
+ t.Fatal("running actor has no external snapshot")
+ }
+ assertSnapshotPresent(t, tc, snapshotURI)
- _, err = tc.client.DeleteActor(context.Background(), &ateapipb.DeleteActorRequest{
- Actor: &ateapipb.ObjectRef{Atespace: testAtespace, Name: "id1"},
- })
- assertGrpcError(t, err, codes.FailedPrecondition, "Actor test-atespace/id1 is not in a deletable state (state: ACTOR_STATE_RUNNING)")
+ tc.fakeAtelet.Reset()
+ plugin.mu.Lock()
+ plugin.detachedNodes = nil
+ plugin.deleted = nil
+ plugin.mu.Unlock()
+
+ if _, err := tc.client.DeleteActor(ctx, &ateapipb.DeleteActorRequest{Actor: actorRef}); err != nil {
+ t.Fatalf("DeleteActor of a running actor failed: %v", err)
+ }
+ if !tc.fakeAtelet.TerminateCalled {
+ t.Error("expected atelet Terminate on the running actor's node")
+ }
+ if diff := cmp.Diff([]string{"node1"}, plugin.detachedNodes); diff != "" {
+ t.Errorf("detached nodes mismatch (-want +got):\n%s", diff)
+ }
+ if diff := cmp.Diff([]string{"storage-substrate-" + actorUID + "-vol1"}, plugin.deleted); diff != "" {
+ t.Errorf("deleted volume IDs mismatch (-want +got):\n%s", diff)
+ }
+ assertSnapshotCollected(t, tc, snapshotURI)
+ if worker, err := tc.persistence.FindWorkerHostingActor(ctx, actorUID); !errors.Is(err, store.ErrNotFound) {
+ t.Errorf("FindWorkerHostingActor after delete = (%q, %v), want ErrNotFound", worker, err)
+ }
+ _, err = tc.client.GetActor(ctx, &ateapipb.GetActorRequest{Actor: actorRef})
+ assertGrpcError(t, err, codes.NotFound, "Actor test-atespace/id1 not found")
}
func TestDeleteActor_Crashed(t *testing.T) {
@@ -1388,10 +1447,9 @@ func TestDeleteActor_StateDeleting(t *testing.T) {
}
}
-// TestDeleteActor_Paused deletes a paused actor without any_state: a paused
-// actor is SUSPENDED (holding a node-local snapshot), so it is deletable by
-// default, and the delete terminates it on its assigned node so the local
-// snapshot is removed.
+// TestDeleteActor_Paused deletes a paused actor: a paused actor is SUSPENDED
+// (holding a node-local snapshot), and the delete terminates it on its
+// assigned node so the local snapshot is removed.
func TestDeleteActor_Paused(t *testing.T) {
ns := namespaceForTest("ns-delete-paused")
tc := setupTest(t, ns)
@@ -1426,7 +1484,7 @@ func TestDeleteActor_Paused(t *testing.T) {
tc.fakeAtelet.Reset()
if _, err := tc.client.DeleteActor(context.Background(), &ateapipb.DeleteActorRequest{Actor: actorRef}); err != nil {
- t.Fatalf("DeleteActor of a paused actor without any_state failed: %v", err)
+ t.Fatalf("DeleteActor of a paused actor failed: %v", err)
}
if !tc.fakeAtelet.TerminateCalled {
t.Error("expected atelet Terminate on the paused actor's node to remove its local snapshot")
@@ -1438,32 +1496,6 @@ func TestDeleteActor_Paused(t *testing.T) {
}
}
-func TestDeleteActor_WrongState(t *testing.T) {
- ns := namespaceForTest("ns-delete-wrong-status")
- tc := setupTest(t, ns)
- defer tc.cleanup()
- createTemplate(t, tc, ns)
-
- runningActor := &ateapipb.Actor{
- Metadata: &ateapipb.ResourceMetadata{
- Atespace: testAtespace,
- Name: "running-actor",
- },
- Status: &ateapipb.ActorStatus{State: ateapipb.ActorState_ACTOR_STATE_RUNNING},
- ActorTemplate: &ateapipb.ObjectRef{Atespace: testAtespace, Name: "tmpl1"},
- }
- if _, err := tc.persistence.CreateActor(context.Background(), runningActor); err != nil {
- t.Fatalf("CreateActor: %v", err)
- }
-
- _, err := tc.service.DeleteActor(context.Background(), &ateapipb.DeleteActorRequest{
- Actor: &ateapipb.ObjectRef{Atespace: testAtespace, Name: "running-actor"},
- })
- if err == nil {
- t.Fatalf("expected DeleteActor on ACTOR_STATE_RUNNING actor to fail, but it succeeded")
- }
-}
-
type failingVolumePlugin struct {
volume.VolumePluginControlPlane
deletedIDs []string
@@ -1493,10 +1525,10 @@ func TestDeleteActor_MultipleVolumeDeletionFailures(t *testing.T) {
ActorTemplate: &ateapipb.ObjectRef{Atespace: testAtespace, Name: "tmpl1"},
Status: &ateapipb.ActorStatus{
State: ateapipb.ActorState_ACTOR_STATE_SUSPENDED,
- ExternalVolumes: []*ateapipb.ExternalVolume{
+ ExternalVolumes: &ateapipb.ExternalVolumes{Volumes: []*ateapipb.ExternalVolumeStatus{
{Name: "vol1", StorageVolumeId: "storage-vol-1", VolumeType: "substrate.io/mock"},
{Name: "vol2", StorageVolumeId: "storage-vol-2", VolumeType: "substrate.io/mock"},
- },
+ }},
},
}
if _, err := tc.persistence.CreateActor(context.Background(), actor); err != nil {
@@ -1571,9 +1603,9 @@ func TestDeleteActor_VolumeDeletionFailure_RetrySuccess(t *testing.T) {
ActorTemplate: &ateapipb.ObjectRef{Atespace: testAtespace, Name: "tmpl1"},
Status: &ateapipb.ActorStatus{
State: ateapipb.ActorState_ACTOR_STATE_SUSPENDED,
- ExternalVolumes: []*ateapipb.ExternalVolume{
+ ExternalVolumes: &ateapipb.ExternalVolumes{Volumes: []*ateapipb.ExternalVolumeStatus{
{Name: "vol1", StorageVolumeId: "storage-vol-1", VolumeType: "substrate.io/mock"},
- },
+ }},
},
}
if _, err := tc.persistence.CreateActor(context.Background(), actor); err != nil {
@@ -1598,8 +1630,8 @@ func TestDeleteActor_VolumeDeletionFailure_RetrySuccess(t *testing.T) {
if getResp.GetStatus().GetState() != ateapipb.ActorState_ACTOR_STATE_DELETING {
t.Errorf("actor state = %v, want ACTOR_STATE_DELETING", getResp.GetStatus().GetState())
}
- if len(getResp.GetStatus().GetExternalVolumes()) != 1 {
- t.Errorf("actor volumes = %v, want the volume still recorded", getResp.GetStatus().GetExternalVolumes())
+ if len(getResp.GetStatus().GetExternalVolumes().GetVolumes()) != 1 {
+ t.Errorf("actor volumes = %v, want the volume still recorded", getResp.GetStatus().GetExternalVolumes().GetVolumes())
}
// 4. Recover the CSI plugin to simulate backend recovery.
@@ -1700,10 +1732,10 @@ func TestActorLifecycle_WithExternalVolumes(t *testing.T) {
volumes := []*ateapipb.Volume{
{
Name: "data-vol",
- ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{
+ External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{
StorageClassName: "fast",
Capacity: "20Gi",
- },
+ }},
},
}
mounts := []*ateapipb.VolumeMount{
@@ -1728,8 +1760,8 @@ func TestActorLifecycle_WithExternalVolumes(t *testing.T) {
if createResp.GetStatus().GetState() != ateapipb.ActorState_ACTOR_STATE_SUSPENDED {
t.Fatalf("expected initial state ACTOR_STATE_SUSPENDED, got %v", createResp.GetStatus().GetState())
}
- if len(createResp.GetStatus().GetExternalVolumes()) != 1 || createResp.GetStatus().GetExternalVolumes()[0].GetStorageVolumeId() != "" {
- t.Fatalf("expected 1 unprovisioned volume after CreateActor, got %v", createResp.GetStatus().GetExternalVolumes())
+ if len(createResp.GetStatus().GetExternalVolumes().GetVolumes()) != 1 || createResp.GetStatus().GetExternalVolumes().GetVolumes()[0].GetStorageVolumeId() != "" {
+ t.Fatalf("expected 1 unprovisioned volume after CreateActor, got %v", createResp.GetStatus().GetExternalVolumes().GetVolumes())
}
// 2. ResumeActor
@@ -1742,10 +1774,10 @@ func TestActorLifecycle_WithExternalVolumes(t *testing.T) {
if resumeResp.GetActor().GetStatus().GetState() != ateapipb.ActorState_ACTOR_STATE_RUNNING {
t.Fatalf("expected state ACTOR_STATE_RUNNING after resume, got %v", resumeResp.GetActor().GetStatus().GetState())
}
- if len(resumeResp.GetActor().GetStatus().GetExternalVolumes()) != 1 {
- t.Fatalf("expected 1 volume after ResumeActor, got %v", resumeResp.GetActor().GetStatus().GetExternalVolumes())
+ if len(resumeResp.GetActor().GetStatus().GetExternalVolumes().GetVolumes()) != 1 {
+ t.Fatalf("expected 1 volume after ResumeActor, got %v", resumeResp.GetActor().GetStatus().GetExternalVolumes().GetVolumes())
}
- if resumeResp.GetActor().GetStatus().GetExternalVolumes()[0].GetStorageVolumeId() == "" {
+ if resumeResp.GetActor().GetStatus().GetExternalVolumes().GetVolumes()[0].GetStorageVolumeId() == "" {
t.Fatalf("expected non-empty storageVolumeId after ResumeActor")
}
@@ -1850,17 +1882,17 @@ func TestResumeActor_VolumeCreationFailure(t *testing.T) {
volumes := []*ateapipb.Volume{
{
Name: "succ-vol1",
- ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{
+ External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{
StorageClassName: "standard",
Capacity: "10Gi",
- },
+ }},
},
{
Name: "fail-vol2",
- ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{
+ External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{
StorageClassName: "standard",
Capacity: "10Gi",
- },
+ }},
},
}
mounts := []*ateapipb.VolumeMount{
@@ -1905,11 +1937,11 @@ func TestResumeActor_VolumeCreationFailure(t *testing.T) {
}
// Verify that succ-vol1 was given a storageVolumeId, and fail-vol2 still has none
- if len(getResp.GetStatus().GetExternalVolumes()) != 2 {
- t.Fatalf("expected 2 volumes on actor, got %d", len(getResp.GetStatus().GetExternalVolumes()))
+ if len(getResp.GetStatus().GetExternalVolumes().GetVolumes()) != 2 {
+ t.Fatalf("expected 2 volumes on actor, got %d", len(getResp.GetStatus().GetExternalVolumes().GetVolumes()))
}
- volsByName := make(map[string]*ateapipb.ExternalVolume)
- for _, v := range getResp.GetStatus().GetExternalVolumes() {
+ volsByName := make(map[string]*ateapipb.ExternalVolumeStatus)
+ for _, v := range getResp.GetStatus().GetExternalVolumes().GetVolumes() {
volsByName[v.GetName()] = v
}
if v1, ok := volsByName["succ-vol1"]; !ok || v1.GetStorageVolumeId() == "" {
@@ -1994,17 +2026,17 @@ func TestResumeActor_VolumeCreationRetrySuccess(t *testing.T) {
volumes := []*ateapipb.Volume{
{
Name: "succ-vol1",
- ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{
+ External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{
StorageClassName: "standard",
Capacity: "10Gi",
- },
+ }},
},
{
Name: "retry-vol2",
- ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{
+ External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{
StorageClassName: "standard",
Capacity: "10Gi",
- },
+ }},
},
}
retryMounts := []*ateapipb.VolumeMount{
@@ -2044,8 +2076,8 @@ func TestResumeActor_VolumeCreationRetrySuccess(t *testing.T) {
t.Errorf("actor state after first resume = %v, want %v", getResp.GetStatus().GetState(), ateapipb.ActorState_ACTOR_STATE_SUSPENDED)
}
- volsByName := make(map[string]*ateapipb.ExternalVolume)
- for _, v := range getResp.GetStatus().GetExternalVolumes() {
+ volsByName := make(map[string]*ateapipb.ExternalVolumeStatus)
+ for _, v := range getResp.GetStatus().GetExternalVolumes().GetVolumes() {
volsByName[v.GetName()] = v
}
if v1, ok := volsByName["succ-vol1"]; !ok || v1.GetStorageVolumeId() == "" {
@@ -2073,7 +2105,7 @@ func TestResumeActor_VolumeCreationRetrySuccess(t *testing.T) {
if getResp.GetStatus().GetState() != ateapipb.ActorState_ACTOR_STATE_RUNNING {
t.Errorf("actor state after second resume = %v, want %v", getResp.GetStatus().GetState(), ateapipb.ActorState_ACTOR_STATE_RUNNING)
}
- for _, v := range getResp.GetStatus().GetExternalVolumes() {
+ for _, v := range getResp.GetStatus().GetExternalVolumes().GetVolumes() {
if v.GetStorageVolumeId() == "" {
t.Errorf("volume %s unexpected state after second resume: %v", v.GetName(), v)
}
@@ -2155,10 +2187,10 @@ func TestResumeActor_VolumeAttachFailureAndRetry(t *testing.T) {
volumes := []*ateapipb.Volume{
{
Name: "vol1",
- ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{
+ External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{
StorageClassName: "standard",
Capacity: "10Gi",
- },
+ }},
},
}
mounts := []*ateapipb.VolumeMount{
@@ -2202,10 +2234,10 @@ func TestResumeActor_VolumeAttachFailureAndRetry(t *testing.T) {
if getResp.GetStatus().GetWorkerAssignment() == nil || getResp.GetStatus().GetWorkerAssignment().GetWorkerPod() != "worker-1" {
t.Errorf("worker assignment = %v, want worker-1", getResp.GetStatus().GetWorkerAssignment())
}
- if len(getResp.GetStatus().GetExternalVolumes()) != 1 {
- t.Fatalf("expected 1 volume on actor, got %d", len(getResp.GetStatus().GetExternalVolumes()))
+ if len(getResp.GetStatus().GetExternalVolumes().GetVolumes()) != 1 {
+ t.Fatalf("expected 1 volume on actor, got %d", len(getResp.GetStatus().GetExternalVolumes().GetVolumes()))
}
- vol := getResp.GetStatus().GetExternalVolumes()[0]
+ vol := getResp.GetStatus().GetExternalVolumes().GetVolumes()[0]
if vol.GetStorageVolumeId() == "" {
t.Errorf("vol1 unexpected state after failed attach: %v", vol)
}
@@ -2244,8 +2276,7 @@ func TestResumeActor_VolumeAttachFailureAndRetry(t *testing.T) {
}
// TestResumeActor_VolumeAttachFailure_DeleteActor tests that when volume attachment fails during ResumeActor,
-// calling DeleteActor without any_state is rejected because the actor is in ACTOR_STATE_RESUMING,
-// but calling DeleteActor with any_state=true succeeds, cleaning up worker assignment and volumes.
+// leaving the actor in ACTOR_STATE_RESUMING, DeleteActor succeeds, cleaning up worker assignment and volumes.
func TestResumeActor_VolumeAttachFailure_DeleteActor(t *testing.T) {
preview.InitForTest(t, "ExternalVolumes")
@@ -2259,10 +2290,10 @@ func TestResumeActor_VolumeAttachFailure_DeleteActor(t *testing.T) {
volumes := []*ateapipb.Volume{
{
Name: "vol1",
- ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{
+ External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{
StorageClassName: "standard",
Capacity: "10Gi",
- },
+ }},
},
}
mounts := []*ateapipb.VolumeMount{
@@ -2302,19 +2333,12 @@ func TestResumeActor_VolumeAttachFailure_DeleteActor(t *testing.T) {
}
actorUID := getResp.GetMetadata().GetUid()
- // Calling DeleteActor without any_state should fail because RESUMING is not a deletable state
+ // DeleteActor on the RESUMING actor should succeed and cleanly tear down
_, err = tc.client.DeleteActor(context.Background(), &ateapipb.DeleteActorRequest{
Actor: &ateapipb.ObjectRef{Atespace: testAtespace, Name: "attach-fail-actor"},
})
- assertGrpcErrorRegex(t, err, codes.FailedPrecondition, "is not in a deletable state")
-
- // Calling DeleteActor with any_state=true should succeed and cleanly tear down
- _, err = tc.client.DeleteActor(context.Background(), &ateapipb.DeleteActorRequest{
- Actor: &ateapipb.ObjectRef{Atespace: testAtespace, Name: "attach-fail-actor"},
- AnyState: true,
- })
if err != nil {
- t.Fatalf("DeleteActor with AnyState failed: %v", err)
+ t.Fatalf("DeleteActor failed: %v", err)
}
// Verify volume was detached and deleted
@@ -2416,17 +2440,17 @@ func TestResumeActor_MultiVolumePartialAttachFailure_Retry(t *testing.T) {
volumes := []*ateapipb.Volume{
{
Name: "vol1",
- ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{
+ External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{
StorageClassName: "standard",
Capacity: "10Gi",
- },
+ }},
},
{
Name: "vol2",
- ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{
+ External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{
StorageClassName: "standard",
Capacity: "10Gi",
- },
+ }},
},
}
mounts := []*ateapipb.VolumeMount{
@@ -2588,10 +2612,10 @@ func TestSuspendActor_VolumeDetachFailure_RetrySuccess(t *testing.T) {
volumes := []*ateapipb.Volume{
{
Name: "vol1",
- ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{
+ External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{
StorageClassName: "standard",
Capacity: "10Gi",
- },
+ }},
},
}
mounts := []*ateapipb.VolumeMount{
@@ -2674,19 +2698,17 @@ func TestSuspendActor_VolumeDetachFailure_RetrySuccess(t *testing.T) {
}
}
-// TestSuspendActor_VolumeDetachFailure_DeleteActorAnyState tests scenario C:
-// when volume detachment fails during SuspendActor and leaves the actor in ACTOR_STATE_SUSPENDING:
-// 1. Standard DeleteActor (without AnyState) fails with FailedPrecondition because SUSPENDING is not deletable.
-// 2. DeleteActor with AnyState=true succeeds, tearing down the actor and its volumes.
+// TestSuspendActor_VolumeDetachFailure_DeleteActor tests scenario C:
+// when volume detachment fails during SuspendActor and leaves the actor in ACTOR_STATE_SUSPENDING,
+// DeleteActor succeeds, tearing down the actor and its volumes.
//
// Workflow:
// 1. Creates a template with an external volume and creates a mock worker pod on node1.
// 2. Creates and resumes an actor to ACTOR_STATE_RUNNING.
// 3. Calls SuspendActor, which fails due to simulated volume detachment failure, leaving the actor in ACTOR_STATE_SUSPENDING.
-// 4. Calls DeleteActor without AnyState and verifies it is rejected with FailedPrecondition ("is not in a deletable state").
-// 5. Resets the plugin to allow detachment, then calls DeleteActor with AnyState=true.
-// 6. Verifies the actor is deleted and GetActor returns NotFound.
-func TestSuspendActor_VolumeDetachFailure_DeleteActorAnyState(t *testing.T) {
+// 4. Resets the plugin to allow detachment, then calls DeleteActor.
+// 5. Verifies the actor is deleted and GetActor returns NotFound.
+func TestSuspendActor_VolumeDetachFailure_DeleteActor(t *testing.T) {
preview.InitForTest(t, "ExternalVolumes")
ns := namespaceForTest("ns-suspend-vol-detach-del")
@@ -2700,10 +2722,10 @@ func TestSuspendActor_VolumeDetachFailure_DeleteActorAnyState(t *testing.T) {
volumes := []*ateapipb.Volume{
{
Name: "vol1",
- ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{
+ External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{
StorageClassName: "standard",
Capacity: "10Gi",
- },
+ }},
},
}
mounts := []*ateapipb.VolumeMount{
@@ -2747,26 +2769,19 @@ func TestSuspendActor_VolumeDetachFailure_DeleteActorAnyState(t *testing.T) {
t.Fatalf("actor state = %v, want ACTOR_STATE_SUSPENDING", getResp.GetStatus().GetState())
}
- // 4. DeleteActor without AnyState fails with FailedPrecondition.
- _, err = tc.client.DeleteActor(context.Background(), &ateapipb.DeleteActorRequest{
- Actor: &ateapipb.ObjectRef{Atespace: testAtespace, Name: "suspend-del-actor"},
- })
- assertGrpcErrorRegex(t, err, codes.FailedPrecondition, "is not in a deletable state")
-
- // 5. Allow detach to succeed and call DeleteActor with AnyState=true.
+ // 4. Allow detach to succeed and call DeleteActor.
plugin.mu.Lock()
plugin.failUntil = 0
plugin.mu.Unlock()
_, err = tc.client.DeleteActor(context.Background(), &ateapipb.DeleteActorRequest{
- Actor: &ateapipb.ObjectRef{Atespace: testAtespace, Name: "suspend-del-actor"},
- AnyState: true,
+ Actor: &ateapipb.ObjectRef{Atespace: testAtespace, Name: "suspend-del-actor"},
})
if err != nil {
- t.Fatalf("DeleteActor with AnyState failed: %v", err)
+ t.Fatalf("DeleteActor failed: %v", err)
}
- // 6. Verify actor is NotFound.
+ // 5. Verify actor is NotFound.
_, err = tc.client.GetActor(context.Background(), &ateapipb.GetActorRequest{
Actor: &ateapipb.ObjectRef{Atespace: testAtespace, Name: "suspend-del-actor"},
})
@@ -2798,10 +2813,10 @@ func TestSuspendActorAsync_VolumeLifecycle_DetachResumeAndMigrate(t *testing.T)
volumes := []*ateapipb.Volume{
{
Name: "vol1",
- ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{
+ External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{
StorageClassName: "standard",
Capacity: "10Gi",
- },
+ }},
},
}
mounts := []*ateapipb.VolumeMount{
@@ -2973,7 +2988,7 @@ func TestSuspendActorAsync_VolumeLifecycle_DetachResumeAndMigrate(t *testing.T)
// 4. Verifies async SuspendActor returns an error and actor state is left in ACTOR_STATE_SUSPENDING.
// 5. Calls async SuspendActor a second time (retry); detachment succeeds.
// 6. Verifies actor state transitions to ACTOR_STATE_SUSPENDED with a local snapshot.
-// 7. Cleans up by deleting the actor with AnyState=true.
+// 7. Cleans up by deleting the actor.
func TestSuspendActorAsync_VolumeDetachFailure_RetrySuccess(t *testing.T) {
preview.InitForTest(t, "ExternalVolumes")
@@ -2988,10 +3003,10 @@ func TestSuspendActorAsync_VolumeDetachFailure_RetrySuccess(t *testing.T) {
volumes := []*ateapipb.Volume{
{
Name: "vol1",
- ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{
+ External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{
StorageClassName: "standard",
Capacity: "10Gi",
- },
+ }},
},
}
mounts := []*ateapipb.VolumeMount{
@@ -3063,10 +3078,9 @@ func TestSuspendActorAsync_VolumeDetachFailure_RetrySuccess(t *testing.T) {
t.Error("expected a local snapshot on the paused actor")
}
- // 7. Clean up by deleting the actor with AnyState=true.
+ // 7. Clean up by deleting the actor.
_, err = tc.client.DeleteActor(context.Background(), &ateapipb.DeleteActorRequest{
- Actor: &ateapipb.ObjectRef{Atespace: testAtespace, Name: "pause-detach-retry-actor"},
- AnyState: true,
+ Actor: &ateapipb.ObjectRef{Atespace: testAtespace, Name: "pause-detach-retry-actor"},
})
if err != nil {
t.Fatalf("DeleteActor failed: %v", err)
@@ -3136,6 +3150,7 @@ func TestResumeActor(t *testing.T) {
WorkerPodUid: podUID,
WorkerPodIps: []string{"127.0.0.1"},
NodeName: "node1",
+ SandboxRuntime: defaultWorkerRuntime("gvisor"),
},
},
}
@@ -3175,7 +3190,8 @@ func TestResumeActor(t *testing.T) {
Capacity: &ateapipb.WorkerResources{Actors: 1, Resources: resources.CPUMemory(1000, 512<<20)},
// All a listing reports of the assignments: one actor and the
// compute its template declares.
- Allocated: &ateapipb.WorkerResources{Actors: 1, Resources: resources.CPUMemory(1000, 512<<20)},
+ Allocated: &ateapipb.WorkerResources{Actors: 1, Resources: resources.CPUMemory(1000, 512<<20)},
+ DefaultRuntime: defaultWorkerRuntime("gvisor"),
},
}
@@ -3715,17 +3731,22 @@ func TestResumeActor_LocalRestoreFailureCrashesActor(t *testing.T) {
if _, err := tc.client.ResumeActor(context.Background(), &ateapipb.ResumeActorRequest{Actor: ref}); err != nil {
t.Fatalf("ResumeActor failed: %v", err)
}
- if _, err := tc.client.SuspendActor(context.Background(), &ateapipb.SuspendActorRequest{Actor: ref, Options: &ateapipb.SuspendOptions{Mode: ateapipb.DurabilityMode_DURABILITY_MODE_ASYNCHRONOUS}}); err != nil {
+ pauseResp, err := tc.client.SuspendActor(context.Background(), &ateapipb.SuspendActorRequest{Actor: ref, Options: &ateapipb.SuspendOptions{Mode: ateapipb.DurabilityMode_DURABILITY_MODE_ASYNCHRONOUS}})
+ if err != nil {
t.Fatalf("SuspendActor failed: %v", err)
}
+ paused := localSnapshot(pauseResp.GetActor().GetStatus())
+ if paused.GetDurableSnapshot().GetObject().GetSnapshotUri() == "" {
+ t.Fatalf("pause left no local snapshot naming its upload: %v", pauseResp.GetActor().GetStatus())
+ }
tc.fakeAtelet.Reset()
tc.fakeAtelet.FailRestore = status.Error(codes.Internal, "injected restore failure")
if _, err := tc.client.ResumeActor(context.Background(), &ateapipb.ResumeActorRequest{Actor: ref}); status.Code(err) != codes.Internal {
t.Fatalf("ResumeActor status code = %v, want Internal (err: %v)", status.Code(err), err)
}
- if got := tc.fakeAtelet.RestoreRequest.GetSnapshot(); got.GetSnapshotUuid() == "" || got.GetObject() != nil {
- t.Fatalf("restore snapshot = %v, want local snapshot (uuid set, no object)", got)
+ if got := tc.fakeAtelet.RestoreRequest.GetSnapshot(); got.GetSnapshotUuid() != paused.GetUuid() || got.GetObject().GetSnapshotUri() != paused.GetDurableSnapshot().GetObject().GetSnapshotUri() {
+ t.Fatalf("restore snapshot = %v, want the local snapshot %v, naming its upload", got, paused)
}
actor, err := tc.client.GetActor(context.Background(), &ateapipb.GetActorRequest{Actor: ref})
@@ -3851,6 +3872,7 @@ func TestSuspendActor(t *testing.T) {
assertSnapshotPresent(t, tc, taggedSnapshotURI)
wantTagSnap := newDurableSnapshot(1, ateapipb.SnapshotOwner_SNAPSHOT_OWNER_TAG, durableSnapshot(sourceActor.GetStatus()).GetFidelity(), "", snapshotName(t, taggedSnapshotURI), taggedSnapshotURI)
+ wantTagSnap.SandboxRuntime = defaultWorkerRuntime("gvisor")
wantTag := &ateapipb.Tag{
Metadata: &ateapipb.ResourceMetadata{Atespace: testAtespace, Name: tagName},
Scope: ateapipb.TagScope_TAG_SCOPE_ATESPACE,
@@ -3960,6 +3982,7 @@ func TestSuspendActor(t *testing.T) {
t.Fatalf("GetActor failed: %v", err)
}
wantActorSnap := newDurableSnapshot(2, ateapipb.SnapshotOwner_SNAPSHOT_OWNER_ACTOR, durableSnapshot(sourceActor.GetStatus()).GetFidelity(), tmpl.GetMetadata().GetUid(), snapshotName(t, snapshotURI), snapshotURI)
+ wantActorSnap.SandboxRuntime = defaultWorkerRuntime("gvisor")
want := &ateapipb.Actor{
Metadata: &ateapipb.ResourceMetadata{Name: name, Atespace: testAtespace},
ActorTemplate: &ateapipb.ObjectRef{Atespace: testAtespace, Name: "tmpl1"},
@@ -4178,9 +4201,14 @@ func TestResumeActor_PausedAfterRepointUsesLocalProvenance(t *testing.T) {
if got := tc.fakeAtelet.lastRestoreRequest().GetSnapshot().GetFidelity(); got != ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES {
t.Fatalf("first resume on v2 had scope = %v, want DATA", got)
}
- if _, err := tc.client.SuspendActor(ctx, &ateapipb.SuspendActorRequest{Actor: actorRef, Options: &ateapipb.SuspendOptions{Mode: ateapipb.DurabilityMode_DURABILITY_MODE_ASYNCHRONOUS}}); err != nil {
+ pauseResp, err := tc.client.SuspendActor(ctx, &ateapipb.SuspendActorRequest{Actor: actorRef, Options: &ateapipb.SuspendOptions{Mode: ateapipb.DurabilityMode_DURABILITY_MODE_ASYNCHRONOUS}})
+ if err != nil {
t.Fatalf("SuspendActor failed: %v", err)
}
+ paused := localSnapshot(pauseResp.GetActor().GetStatus())
+ if paused.GetDurableSnapshot().GetObject().GetSnapshotUri() == "" {
+ t.Fatalf("pause left no local snapshot naming its upload: %v", pauseResp.GetActor().GetStatus())
+ }
waitForWorkerAvailable(t, tc, worker)
// Resume from pause: the local checkpoint was captured on v2 and the
@@ -4189,8 +4217,8 @@ func TestResumeActor_PausedAfterRepointUsesLocalProvenance(t *testing.T) {
t.Fatalf("ResumeActor(from v2 pause) failed: %v", err)
}
restoreReq := tc.fakeAtelet.lastRestoreRequest()
- if got := restoreReq.GetSnapshot(); got.GetSnapshotUuid() == "" || got.GetObject() != nil {
- t.Errorf("restore request snapshot = %v, want local snapshot (uuid set, no object)", got)
+ if got := restoreReq.GetSnapshot(); got.GetSnapshotUuid() != paused.GetUuid() || got.GetObject().GetSnapshotUri() != paused.GetDurableSnapshot().GetObject().GetSnapshotUri() {
+ t.Errorf("restore request snapshot = %v, want the local snapshot %v, naming its upload", got, paused)
}
if got := restoreReq.GetSnapshot().GetFidelity(); got != ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY {
t.Errorf("restore request scope = %v, want FULL (local checkpoint was captured on v2)", got)
@@ -4270,6 +4298,8 @@ func TestSuspendActorAsynchronous(t *testing.T) {
if ls.GetUuid() == "" {
t.Error("Snapshot.Uuid is empty, want the name the pause checkpointed under")
}
+ wantPausedSnap := newPausedSnapshot(t, 2, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, tmpl.GetMetadata().GetUid(), getResp.GetMetadata().GetUid(), ls.GetUuid(), "node1")
+ wantPausedSnap.SandboxRuntime = defaultWorkerRuntime("gvisor")
want := &ateapipb.Actor{
Metadata: &ateapipb.ResourceMetadata{Name: name, Atespace: testAtespace},
ActorTemplate: &ateapipb.ObjectRef{Atespace: testAtespace, Name: "tmpl1"},
@@ -4279,7 +4309,7 @@ func TestSuspendActorAsynchronous(t *testing.T) {
LastAssignedGeneration: 2,
Snapshots: []*ateapipb.Snapshot{
newDurableSnapshot(1, ateapipb.SnapshotOwner_SNAPSHOT_OWNER_TAG, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, tmpl.GetMetadata().GetUid(), goldenSnapshotName, goldenSnapshotURI(t)),
- newPausedSnapshot(t, 2, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, tmpl.GetMetadata().GetUid(), getResp.GetMetadata().GetUid(), ls.GetUuid(), "node1"),
+ wantPausedSnap,
},
},
}
@@ -5039,6 +5069,140 @@ func TestSuspendActor_Mode(t *testing.T) {
}
}
+// TestSuspendActor_SnapshotConflictPreservesState verifies that checkpoint
+// conflicts preserve the actor's worker and last completed snapshot, and that
+// retries send the same pending snapshot until atelet accepts it.
+func TestSuspendActor_SnapshotConflictPreservesState(t *testing.T) {
+ for _, mode := range []ateapipb.DurabilityMode{
+ ateapipb.DurabilityMode_DURABILITY_MODE_SYNCHRONOUS,
+ ateapipb.DurabilityMode_DURABILITY_MODE_ASYNCHRONOUS,
+ } {
+ t.Run(mode.String(), func(t *testing.T) {
+ ctx := context.Background()
+ ns := namespaceForTest("ns-suspend-snapshot-conflict")
+ tc := setupTest(t, ns)
+ defer tc.cleanup()
+ createTemplate(t, tc, ns)
+ workerUID := createWorkerPod(t, tc, ns, "worker-1", "node1", "pool1")
+ ref := &ateapipb.ObjectRef{Atespace: testAtespace, Name: "id1"}
+ if _, err := tc.client.CreateActor(ctx, &ateapipb.CreateActorRequest{Actor: &ateapipb.Actor{
+ Metadata: &ateapipb.ResourceMetadata{Atespace: ref.GetAtespace(), Name: ref.GetName()},
+ ActorTemplate: &ateapipb.ObjectRef{Atespace: testAtespace, Name: "tmpl1"},
+ }}); err != nil {
+ t.Fatalf("CreateActor: %v", err)
+ }
+ if _, err := tc.client.ResumeActor(ctx, &ateapipb.ResumeActorRequest{Actor: ref}); err != nil {
+ t.Fatalf("first ResumeActor: %v", err)
+ }
+ saved, err := tc.client.SuspendActor(ctx, &ateapipb.SuspendActorRequest{
+ Actor: ref, Options: &ateapipb.SuspendOptions{Mode: ateapipb.DurabilityMode_DURABILITY_MODE_SYNCHRONOUS},
+ })
+ if err != nil {
+ t.Fatalf("first SuspendActor: %v", err)
+ }
+ waitForWorkerAvailable(t, tc, workerUID)
+ savedURI := durableSnapshotURI(saved.GetActor().GetStatus())
+ assertSnapshotPresent(t, tc, savedURI)
+ running, err := tc.client.ResumeActor(ctx, &ateapipb.ResumeActorRequest{Actor: ref})
+ if err != nil {
+ t.Fatalf("second ResumeActor: %v", err)
+ }
+
+ tc.fakeAtelet.Reset()
+ tc.fakeAtelet.SetObjectStore(tc.objectStore)
+ tc.fakeAtelet.FailCheckpoint = status.Error(codes.Aborted, "stale snapshot generation")
+ req := &ateapipb.SuspendActorRequest{Actor: ref, Options: &ateapipb.SuspendOptions{Mode: mode}}
+ _, err = tc.client.SuspendActor(ctx, req)
+ if got := status.Code(err); got != codes.Aborted {
+ t.Fatalf("SuspendActor error = %v, want Aborted", err)
+ }
+ stored, err := tc.client.GetActor(ctx, &ateapipb.GetActorRequest{Actor: ref})
+ if err != nil {
+ t.Fatalf("GetActor: %v", err)
+ }
+ st := stored.GetStatus()
+ if st.GetState() != ateapipb.ActorState_ACTOR_STATE_SUSPENDING || st.GetCrash() != nil {
+ t.Fatalf("status after conflict = %v, want SUSPENDING without a crash", st)
+ }
+ if !proto.Equal(st.GetWorkerAssignment(), running.GetActor().GetStatus().GetWorkerAssignment()) {
+ t.Error("snapshot conflict changed the worker assignment")
+ }
+ if got := durableSnapshotURI(st); got != savedURI {
+ t.Errorf("completed snapshot = %q, want %q", got, savedURI)
+ }
+ assertSnapshotPresent(t, tc, savedURI)
+ worker, err := tc.persistence.GetWorker(ctx, workerUID)
+ if err != nil {
+ t.Fatalf("GetWorker: %v", err)
+ }
+ if got := worker.GetStatus().GetAllocated().GetActors(); got != 1 {
+ t.Errorf("allocated actors = %d, want 1", got)
+ }
+ var pending *ateapipb.Snapshot
+ for _, snap := range st.GetSnapshots() {
+ if snap.GetGeneration() == st.GetLastAssignedGeneration() {
+ pending = snap
+ }
+ }
+ if pending.GetDurableSnapshot().GetStatus() != ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_PENDING || pending.GetLocality() != "" {
+ t.Fatalf("in-progress snapshot = %v, want PENDING without locality", pending)
+ }
+ assertSnapshotCollected(t, tc, pending.GetDurableSnapshot().GetObject().GetSnapshotUri())
+
+ // A repeated conflict must not finalize or allocate another generation.
+ tc.fakeAtelet.Reset()
+ tc.fakeAtelet.SetObjectStore(tc.objectStore)
+ tc.fakeAtelet.FailCheckpoint = status.Error(codes.Aborted, "stale snapshot generation")
+ _, err = tc.client.SuspendActor(ctx, req)
+ if status.Code(err) != codes.Aborted {
+ t.Fatalf("repeated SuspendActor error = %v, want Aborted", err)
+ }
+ if !tc.fakeAtelet.CheckpointCalled {
+ t.Fatal("repeated conflict did not call atelet Checkpoint")
+ }
+ conflicted := tc.fakeAtelet.CheckpointRequest.GetSnapshot()
+ if conflicted.GetSnapshotUuid() != pending.GetUuid() || conflicted.GetGeneration() != pending.GetGeneration() {
+ t.Errorf("repeated Checkpoint snapshot = %v, want pending snapshot %v", conflicted, pending)
+ }
+ repeated, err := tc.client.GetActor(ctx, &ateapipb.GetActorRequest{Actor: ref})
+ if err != nil {
+ t.Fatalf("GetActor after repeated conflict: %v", err)
+ }
+ if !proto.Equal(repeated.GetStatus(), st) {
+ t.Error("repeated conflict changed the actor's status")
+ }
+ assertSnapshotPresent(t, tc, savedURI)
+
+ // Once atelet accepts the request, retry the same persisted snapshot.
+ tc.fakeAtelet.Reset()
+ tc.fakeAtelet.SetObjectStore(tc.objectStore)
+ finished, err := tc.client.SuspendActor(ctx, req)
+ if err != nil {
+ t.Fatalf("SuspendActor after conflict cleared: %v", err)
+ }
+ if !tc.fakeAtelet.CheckpointCalled {
+ t.Fatal("retry did not call atelet Checkpoint")
+ }
+ retried := tc.fakeAtelet.CheckpointRequest.GetSnapshot()
+ if got := retried.GetSnapshotUuid(); got != pending.GetUuid() {
+ t.Errorf("retry snapshot UUID = %q, want %q", got, pending.GetUuid())
+ }
+ if got := retried.GetGeneration(); got != pending.GetGeneration() {
+ t.Errorf("retry snapshot generation = %d, want %d", got, pending.GetGeneration())
+ }
+ if got := finished.GetActor().GetStatus().GetState(); got != ateapipb.ActorState_ACTOR_STATE_SUSPENDED {
+ t.Errorf("state after retry = %v, want SUSPENDED", got)
+ }
+ if got := finished.GetActor().GetStatus().GetLastAssignedGeneration(); got != pending.GetGeneration() {
+ t.Errorf("generation after retry = %d, want %d", got, pending.GetGeneration())
+ }
+ if _, err := tc.client.ResumeActor(ctx, &ateapipb.ResumeActorRequest{Actor: ref}); err != nil {
+ t.Fatalf("ResumeActor after retry: %v", err)
+ }
+ })
+ }
+}
+
// TestCheckpointFailureCrashes: an atelet Checkpoint error while pausing or
// suspending a running actor crashes it, records atelet's error text, and
// releases its worker.
@@ -5826,7 +5990,7 @@ func TestDeleteActor_TerminateFailureLeavesActorDeleting(t *testing.T) {
tc.fakeAtelet.Lock.Lock()
tc.fakeAtelet.FailTerminate = status.Error(codes.Internal, "injected terminate failure")
tc.fakeAtelet.Lock.Unlock()
- if _, err := tc.client.DeleteActor(ctx, &ateapipb.DeleteActorRequest{Actor: ref, AnyState: true}); err == nil {
+ if _, err := tc.client.DeleteActor(ctx, &ateapipb.DeleteActorRequest{Actor: ref}); err == nil {
t.Fatal("DeleteActor succeeded despite failing terminate")
}
@@ -5848,7 +6012,7 @@ func TestDeleteActor_TerminateFailureLeavesActorDeleting(t *testing.T) {
tc.fakeAtelet.FailTerminate = nil
tc.fakeAtelet.TerminateCalled = false
tc.fakeAtelet.Lock.Unlock()
- if _, err := tc.client.DeleteActor(ctx, &ateapipb.DeleteActorRequest{Actor: ref, AnyState: true}); err != nil {
+ if _, err := tc.client.DeleteActor(ctx, &ateapipb.DeleteActorRequest{Actor: ref}); err != nil {
t.Fatalf("second DeleteActor failed: %v", err)
}
if _, err := tc.client.GetActor(ctx, &ateapipb.GetActorRequest{Actor: ref}); status.Code(err) != codes.NotFound {
@@ -6013,7 +6177,7 @@ func reportWorkerRuntime(t *testing.T, tc *testContext, workerName string, defau
// 2. A subsequent ResumeActor places the actor onto the runtime-compatible
// worker and restores at full MEMORY fidelity when that worker has room.
// 3. When the compatible worker is full, ResumeActor falls back to the other
-// worker, downgrades the restore to VOLUMES whether the snapshot is the
+// worker, downgrades the restore to ROOTFS whether the snapshot is the
// actor's own or a borrowed golden, and re-stamps the new worker's runtime
// on the next SuspendActor.
// 4. When the other worker advertises the snapshot's runtime in
@@ -6136,7 +6300,7 @@ func TestResumeActor_SandboxRuntimePlacementAndFallback(t *testing.T) {
}
// With worker-a full, resuming actor-1 must fall back to worker-b, downgrade
- // the restore to VOLUMES fidelity, and re-stamp rtB on the next suspend.
+ // the restore to ROOTFS fidelity, and re-stamp rtB on the next suspend.
resumed3, err := tc.client.ResumeActor(ctx, &ateapipb.ResumeActorRequest{Actor: actorRef})
if err != nil {
t.Fatalf("fallback ResumeActor failed: %v", err)
@@ -6147,8 +6311,8 @@ func TestResumeActor_SandboxRuntimePlacementAndFallback(t *testing.T) {
if got := resumed3.GetActor().GetStatus().GetWorkerAssignment().GetSandboxRuntime(); !proto.Equal(got, rtB) {
t.Fatalf("fallback WorkerAssignment.SandboxRuntime = %v, want %v", got, rtB)
}
- if got := tc.fakeAtelet.lastRestoreRequest().GetSnapshot().GetFidelity(); got != ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES {
- t.Fatalf("fallback RestoreRequest.Fidelity = %v, want VOLUMES", got)
+ if got := tc.fakeAtelet.lastRestoreRequest().GetSnapshot().GetFidelity(); got != ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS {
+ t.Fatalf("fallback RestoreRequest.Fidelity = %v, want ROOTFS", got)
}
suspended3, err := tc.client.SuspendActor(ctx, &ateapipb.SuspendActorRequest{Actor: actorRef, Options: &ateapipb.SuspendOptions{Mode: ateapipb.DurabilityMode_DURABILITY_MODE_SYNCHRONOUS}})
@@ -6163,7 +6327,7 @@ func TestResumeActor_SandboxRuntimePlacementAndFallback(t *testing.T) {
// Point the template's golden tag at a golden snapshot stamped with rtA and
// verify that a newly created actor borrowing the golden snapshot, when
// worker-a is full and it falls back onto worker-b, restores the golden at
- // VOLUMES fidelity like any other durable snapshot rather than cold-booting.
+ // ROOTFS fidelity like any other durable snapshot rather than cold-booting.
goldenSnap := newDurableSnapshot(1, ateapipb.SnapshotOwner_SNAPSHOT_OWNER_TAG, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, tmpl.GetMetadata().GetUid(), goldenSnapshotName, goldenSnapshotURI(t))
goldenSnap.SandboxRuntime = proto.Clone(rtA).(*ateapipb.SandboxRuntime)
stampedGoldenTag, err := tc.persistence.CreateTag(ctx, &ateapipb.Tag{
@@ -6224,8 +6388,8 @@ func TestResumeActor_SandboxRuntimePlacementAndFallback(t *testing.T) {
t.Fatalf("golden-actor fallback: RunCalled=%v RestoreCalled=%v, want RunCalled=false RestoreCalled=true", runCalled, restoreCalled)
}
goldenRestore := tc.fakeAtelet.lastRestoreRequest()
- if got := goldenRestore.GetSnapshot().GetFidelity(); got != ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES {
- t.Fatalf("golden-actor RestoreRequest.Fidelity = %v, want VOLUMES", got)
+ if got := goldenRestore.GetSnapshot().GetFidelity(); got != ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS {
+ t.Fatalf("golden-actor RestoreRequest.Fidelity = %v, want ROOTFS", got)
}
if got, want := goldenRestore.GetSnapshot().GetObject().GetSnapshotUri(), goldenSnapshotURI(t); got != want {
t.Fatalf("golden-actor RestoreRequest.SnapshotUri = %q, want golden %q", got, want)
diff --git a/cmd/ateapi/internal/controlapi/functionaltest/common_test.go b/cmd/ateapi/internal/controlapi/functionaltest/common_test.go
index a7978c0db3..b48290c3b0 100644
--- a/cmd/ateapi/internal/controlapi/functionaltest/common_test.go
+++ b/cmd/ateapi/internal/controlapi/functionaltest/common_test.go
@@ -38,7 +38,6 @@ import (
atev1alpha1 "go.ate.dev/substrate/pkg/api/v1alpha1"
"go.ate.dev/substrate/pkg/client/clientset/versioned"
"go.ate.dev/substrate/pkg/client/informers/externalversions"
- listersv1alpha1 "go.ate.dev/substrate/pkg/client/listers/api/v1alpha1"
"go.ate.dev/substrate/pkg/proto/ateapipb"
"go.opentelemetry.io/otel/attribute"
sdkmetric "go.opentelemetry.io/otel/sdk/metric"
@@ -83,17 +82,15 @@ var (
)
type testContext struct {
- service *controlapi.RPCService
- client ateapipb.ControlClient
- noOptInClient ateapipb.ControlClient // never sends preview.Header
- k8sClient kubernetes.Interface
- substrateClient versioned.Interface
- persistence store.Interface
- workerCache *workercache.Cache
- fakeAtelet *FakeAteletServer
- cleanup func()
- workerPoolLister listersv1alpha1.WorkerPoolLister
- sandboxConfigLister listersv1alpha1.SandboxConfigLister
+ service *controlapi.RPCService
+ client ateapipb.ControlClient
+ noOptInClient ateapipb.ControlClient // never sends preview.Header
+ k8sClient kubernetes.Interface
+ substrateClient versioned.Interface
+ persistence store.Interface
+ workerCache *workercache.Cache
+ fakeAtelet *FakeAteletServer
+ cleanup func()
// ateletIndexer is the index DialForAteletOnNode looks up atelets in.
// setupAteletOnNode waits on it so a test never dials a node whose atelet the
// informer has not seen yet.
@@ -162,8 +159,6 @@ func setupTestWithOptions(t *testing.T, ns string, options testOptions) *testCon
scLister := scFactory.Storage().V1().StorageClasses().Lister()
substrateInformerFactory := externalversions.NewSharedInformerFactory(substrateClient, 0)
- workerPoolLister := substrateInformerFactory.Api().V1alpha1().WorkerPools().Lister()
- sandboxConfigLister := substrateInformerFactory.Api().V1alpha1().SandboxConfigs().Lister()
csiDriverConfigLister := substrateInformerFactory.Api().V1alpha1().CSIDriverConfigs().Lister()
ctx, cancel := context.WithCancel(context.Background())
@@ -239,8 +234,6 @@ func setupTestWithOptions(t *testing.T, ns string, options testOptions) *testCon
service := controlapi.NewRPCService(
persistence,
wc,
- workerPoolLister,
- sandboxConfigLister,
csiDriverConfigLister,
scLister,
dialer,
@@ -252,6 +245,7 @@ func setupTestWithOptions(t *testing.T, ns string, options testOptions) *testCon
testActorJWTIssuer,
actorJWTAuthorityPool,
actorCAPool,
+ testStorageLocation,
)
// 5. Start REAL gRPC Server for ATE API
@@ -330,20 +324,18 @@ func setupTestWithOptions(t *testing.T, ns string, options testOptions) *testCon
}
return &testContext{
- service: service,
- client: client,
- noOptInClient: ateapipb.NewControlClient(noOptInConn),
- k8sClient: k8sClient,
- substrateClient: substrateClient,
- persistence: persistence,
- workerCache: wc,
- fakeAtelet: fakeAtelet,
- cleanup: cleanup,
- workerPoolLister: workerPoolLister,
- sandboxConfigLister: sandboxConfigLister,
- ateletIndexer: ateletInformer.GetIndexer(),
- metricReader: metricReader,
- objectStore: objectStore,
+ service: service,
+ client: client,
+ noOptInClient: ateapipb.NewControlClient(noOptInConn),
+ k8sClient: k8sClient,
+ substrateClient: substrateClient,
+ persistence: persistence,
+ workerCache: wc,
+ fakeAtelet: fakeAtelet,
+ cleanup: cleanup,
+ ateletIndexer: ateletInformer.GetIndexer(),
+ metricReader: metricReader,
+ objectStore: objectStore,
}
}
@@ -471,10 +463,6 @@ func createTemplateWithVolumes(t *testing.T, tc *testContext, ns string, volumes
func createTemplateWithContainersAndVolumes(t *testing.T, tc *testContext, ns string, containers []*ateapipb.Container, volumes []*ateapipb.Volume) *ateapipb.ActorTemplate {
t.Helper()
- // Sandbox binaries live on the (cluster-scoped) SandboxConfig the
- // actor's WorkerPool resolves. Create the gvisor-default SandboxConfig so
- // a boot-from-spec Run can resolve its assets.
- ensureDefaultGvisorSandboxConfig(t, tc)
createWorkerPool(t, tc, ns, "pool1", map[string]string{poolLabelKey: ns})
created, err := tc.client.CreateActorTemplate(context.Background(), &ateapipb.CreateActorTemplateRequest{
@@ -604,53 +592,6 @@ func tagSnapshotURI(tag *ateapipb.Tag) string {
return tag.GetStatus().GetSnapshot().GetDurableSnapshot().GetObject().GetSnapshotUri()
}
-// testPauseImage is the pause image the default test SandboxConfig carries;
-// it is what a resolved WorkloadSpec's sandbox assets should name.
-const testPauseImage = "pause@sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
-
-// ensureDefaultGvisorSandboxConfig creates the cluster-scoped "gvisor-default"
-// SandboxConfig (idempotently) and waits for it to appear in the lister.
-func ensureDefaultGvisorSandboxConfig(t *testing.T, tc *testContext) {
- t.Helper()
- ensureGvisorSandboxConfig(t, tc, "gvisor-default")
-}
-
-// ensureGvisorSandboxConfig creates a cluster-scoped gvisor SandboxConfig
-// (idempotently) and waits for it to appear in the lister.
-func ensureGvisorSandboxConfig(t *testing.T, tc *testContext, name string) {
- t.Helper()
- sc := &atev1alpha1.SandboxConfig{
- ObjectMeta: metav1.ObjectMeta{Name: name},
- Spec: atev1alpha1.SandboxConfigSpec{
- SandboxClass: atev1alpha1.SandboxClassGvisor,
- DefaultVersion: "v1",
- Versions: []atev1alpha1.SandboxVersionConfig{{
- Name: "v1",
- PauseImage: testPauseImage,
- Assets: map[string]map[string]atev1alpha1.AssetFile{
- "amd64": {"gvisor": {
- URL: "gs://gvisor/releases/nightly/2026-10-10/x86_64/gvisor.tar.zstd",
- SHA256: "4ae7fab988c3647d2a835d6c3c70c6a99021c562db97827592562b6c51c3e88c",
- }},
- "arm64": {"gvisor": {
- URL: "gs://gvisor/releases/nightly/2026-10-10/aarch64/gvisor.tar.zstd",
- SHA256: "aaf414f06eea7960238e43165c691f0a36cefc0593fb141e011e96211c7b6f57",
- }},
- },
- }},
- },
- }
- if _, err := tc.substrateClient.ApiV1alpha1().SandboxConfigs().Create(context.Background(), sc, metav1.CreateOptions{}); err != nil && !apierrors.IsAlreadyExists(err) {
- t.Fatalf("failed to create SandboxConfig %s: %v", name, err)
- }
- if err := wait.PollUntilContextTimeout(context.Background(), 100*time.Millisecond, 5*time.Second, true, func(ctx context.Context) (bool, error) {
- _, err := tc.sandboxConfigLister.Get(name)
- return err == nil, nil
- }); err != nil {
- t.Fatalf("SandboxConfig %s not synced into lister: %v", name, err)
- }
-}
-
func createWorkerPool(t *testing.T, tc *testContext, ns string, name string, labels map[string]string) {
t.Helper()
wp := &atev1alpha1.WorkerPool{
@@ -665,43 +606,15 @@ func createWorkerPool(t *testing.T, tc *testContext, ns string, name string, lab
SandboxClasses: []atev1alpha1.WorkerPoolSandboxClass{{Name: atev1alpha1.SandboxClassGvisor}},
},
}
- created, err := tc.substrateClient.ApiV1alpha1().WorkerPools(ns).Create(context.Background(), wp, metav1.CreateOptions{})
- if err != nil {
+ if _, err := tc.substrateClient.ApiV1alpha1().WorkerPools(ns).Create(context.Background(), wp, metav1.CreateOptions{}); err != nil {
t.Fatalf("failed to create WorkerPool: %v", err)
}
- // The status the controller would write once it confirmed the pool's
- // SandboxConfig: resume resolves the sandbox assets through it. Status is
- // a subresource, so it takes its own write.
- created.Status.SandboxClasses = []atev1alpha1.WorkerPoolSandboxClassStatus{{
- Name: atev1alpha1.SandboxClassGvisor,
- ConfigRef: &atev1alpha1.SandboxConfigReference{Name: "gvisor-default"},
- }}
- if _, err := tc.substrateClient.ApiV1alpha1().WorkerPools(ns).UpdateStatus(context.Background(), created, metav1.UpdateOptions{}); err != nil {
- t.Fatalf("failed to update WorkerPool status: %v", err)
- }
-
- waitForWorkerPoolInInformer(t, tc, ns, name)
-}
-
-// waitForWorkerPoolInInformer blocks until the pool the test just created,
-// with its resolved SandboxConfig, is visible to the scheduler and the resume
-// workflow, which read it through an informer rather than the API.
-func waitForWorkerPoolInInformer(t *testing.T, tc *testContext, ns, name string) {
- t.Helper()
- err := wait.PollUntilContextTimeout(context.Background(), 100*time.Millisecond, 5*time.Second, true, func(ctx context.Context) (bool, error) {
- pool, err := tc.workerPoolLister.WorkerPools(ns).Get(name)
- return err == nil && pool.ResolvedSandboxConfig() != "", nil
- })
- if err != nil {
- t.Fatalf("failed to wait for WorkerPool %s/%s in informer: %v", ns, name, err)
- }
}
// createTemplateWithSelector creates a substrate ActorTemplate in
// testAtespace with the given worker selector and no golden snapshot.
func createTemplateWithSelector(t *testing.T, tc *testContext, name string, selector *ateapipb.Selector) *ateapipb.ActorTemplate {
t.Helper()
- ensureDefaultGvisorSandboxConfig(t, tc)
created, err := tc.client.CreateActorTemplate(context.Background(), &ateapipb.CreateActorTemplateRequest{
ActorTemplate: &ateapipb.ActorTemplate{
Metadata: &ateapipb.ResourceMetadata{
@@ -764,11 +677,8 @@ func createWorkerPod(t *testing.T, tc *testContext, ns string, name string, node
t.Fatalf("failed to update worker pod status: %v", err)
}
- // The WorkerPool supplies the fields the syncer copies off it. Read it from
- // the lister the service itself uses, so a pool the informer has not caught
- // up on fails here rather than producing a Worker the scheduler will not
- // match.
- pool, err := tc.workerPoolLister.WorkerPools(ns).Get(poolName)
+ // The WorkerPool supplies the fields the syncer copies off it.
+ pool, err := tc.substrateClient.ApiV1alpha1().WorkerPools(ns).Get(context.Background(), poolName, metav1.GetOptions{})
if err != nil {
t.Fatalf("failed to get WorkerPool %s/%s: %v", ns, poolName, err)
}
diff --git a/cmd/ateapi/internal/controlapi/functionaltest/main_test.go b/cmd/ateapi/internal/controlapi/functionaltest/main_test.go
index 0444467da0..7fb90c6a46 100644
--- a/cmd/ateapi/internal/controlapi/functionaltest/main_test.go
+++ b/cmd/ateapi/internal/controlapi/functionaltest/main_test.go
@@ -206,7 +206,12 @@ func (f *FakeAteletServer) PromoteSnapshot(ctx context.Context, req *ateletpb.Pr
if err := f.writeSnapshot(req.GetSnapshot().GetObject().GetSnapshotUri()); err != nil {
return nil, err
}
- return &ateletpb.PromoteSnapshotResponse{}, nil
+ // Like atelet, report the snapshot durable and still kept on the node.
+ report := &ateletpb.SnapshotReport{Snapshot: proto.CloneOf(req.GetSnapshot())}
+ return &ateletpb.PromoteSnapshotResponse{State: &ateletpb.ActorSnapshotState{
+ LatestLocal: report,
+ LatestDurable: proto.CloneOf(report),
+ }}, nil
}
func (f *FakeAteletServer) Run(ctx context.Context, req *ateletpb.RunRequest) (*ateletpb.RunResponse, error) {
diff --git a/cmd/ateapi/internal/controlapi/functionaltest/tag_test.go b/cmd/ateapi/internal/controlapi/functionaltest/tag_test.go
index 8f2ca73aa4..9249c50888 100644
--- a/cmd/ateapi/internal/controlapi/functionaltest/tag_test.go
+++ b/cmd/ateapi/internal/controlapi/functionaltest/tag_test.go
@@ -151,6 +151,73 @@ func TestCreateTag_ReusedTagName(t *testing.T) {
}
}
+// TestCreateTag_FromPausedActor verifies that tagging a paused actor has the
+// atelet on its node upload the node-local snapshot first, and tags that
+// snapshot. The actor keeps the snapshot on its node.
+func TestCreateTag_FromPausedActor(t *testing.T) {
+ ns := namespaceForTest("ns-tag-paused")
+ tc := setupTest(t, ns)
+ defer tc.cleanup()
+
+ createTemplate(t, tc, ns)
+ createWorkerPod(t, tc, ns, "worker-1", "node1", "pool1")
+
+ ctx := context.Background()
+ ref := &ateapipb.ObjectRef{Atespace: testAtespace, Name: "actor-paused"}
+ if _, err := tc.client.CreateActor(ctx, &ateapipb.CreateActorRequest{Actor: &ateapipb.Actor{
+ Metadata: &ateapipb.ResourceMetadata{Atespace: ref.GetAtespace(), Name: ref.GetName()},
+ ActorTemplate: &ateapipb.ObjectRef{Atespace: testAtespace, Name: "tmpl1"},
+ }}); err != nil {
+ t.Fatalf("CreateActor failed: %v", err)
+ }
+ if _, err := tc.client.ResumeActor(ctx, &ateapipb.ResumeActorRequest{Actor: ref}); err != nil {
+ t.Fatalf("ResumeActor failed: %v", err)
+ }
+ if _, err := tc.client.SuspendActor(ctx, &ateapipb.SuspendActorRequest{
+ Actor: ref,
+ Options: &ateapipb.SuspendOptions{Mode: ateapipb.DurabilityMode_DURABILITY_MODE_ASYNCHRONOUS},
+ }); err != nil {
+ t.Fatalf("pause failed: %v", err)
+ }
+ paused, err := tc.client.GetActor(ctx, &ateapipb.GetActorRequest{Actor: ref})
+ if err != nil {
+ t.Fatalf("GetActor failed: %v", err)
+ }
+ pausedSnap := localSnapshot(paused.GetStatus())
+ if pausedSnap == nil {
+ t.Fatal("paused actor has no local snapshot")
+ }
+
+ tag, err := tc.client.CreateTag(ctx, &ateapipb.CreateTagRequest{
+ Tag: &ateapipb.Tag{
+ Metadata: &ateapipb.ResourceMetadata{Atespace: testAtespace, Name: "from-paused"},
+ Scope: ateapipb.TagScope_TAG_SCOPE_ATESPACE,
+ SourceActor: ref,
+ },
+ })
+ if err != nil {
+ t.Fatalf("CreateTag failed: %v", err)
+ }
+
+ if got := tc.fakeAtelet.PromoteRequest.GetSnapshot().GetSnapshotUuid(); got != pausedSnap.GetUuid() {
+ t.Errorf("PromoteSnapshot snapshot uuid = %q, want the paused snapshot %q", got, pausedSnap.GetUuid())
+ }
+ if len(snapshotObjectNames(t, tc, tagSnapshotURI(tag))) == 0 {
+ t.Errorf("the tag's external snapshot %s is empty", tagSnapshotURI(tag))
+ }
+ stored, err := tc.client.GetActor(ctx, &ateapipb.GetActorRequest{Actor: ref})
+ if err != nil {
+ t.Fatalf("GetActor failed: %v", err)
+ }
+ // The paused snapshot is now durable and stays on its node.
+ if got := durableSnapshot(stored.GetStatus()); got.GetUuid() != pausedSnap.GetUuid() {
+ t.Errorf("latest durable snapshot = %v, want the paused snapshot %q", got, pausedSnap.GetUuid())
+ }
+ if got := localSnapshot(stored.GetStatus()); got.GetUuid() != pausedSnap.GetUuid() {
+ t.Errorf("local snapshot = %v, want the paused snapshot %q kept on its node", got, pausedSnap.GetUuid())
+ }
+}
+
// suspendActorForTest creates, resumes and suspends an actor on workerName,
// and returns the URI of the external snapshot the suspend wrote.
func suspendActorForTest(t *testing.T, tc *testContext, workerName, name string) string {
diff --git a/cmd/ateapi/internal/controlapi/functionaltest/worker_contention_test.go b/cmd/ateapi/internal/controlapi/functionaltest/worker_contention_test.go
index 63863ffd06..29abc66a13 100644
--- a/cmd/ateapi/internal/controlapi/functionaltest/worker_contention_test.go
+++ b/cmd/ateapi/internal/controlapi/functionaltest/worker_contention_test.go
@@ -24,6 +24,7 @@ import (
"k8s.io/apimachinery/pkg/util/wait"
"go.ate.dev/substrate/cmd/ateapi/internal/store"
+ "go.ate.dev/substrate/internal/hardware"
"go.ate.dev/substrate/internal/resources"
"go.ate.dev/substrate/pkg/proto/ateapipb"
)
@@ -153,6 +154,24 @@ func setWorkerActorCapacity(t *testing.T, tc *testContext, pool string, actors i
}
}
+// defaultWorkerRuntime returns the default SandboxRuntime that
+// reportWorkerCapacity records on a test worker of the given sandbox class.
+func defaultWorkerRuntime(class string) *ateapipb.SandboxRuntime {
+ if class == "" {
+ class = "gvisor"
+ }
+ return &ateapipb.SandboxRuntime{
+ SandboxClass: class,
+ Version: &ateapipb.VersionedSandboxCompat{
+ SchemaVersion: hardware.SchemaVersionV1,
+ Attributes: []*ateapipb.AttributeEntry{
+ {Key: hardware.AttrArchitecture, Value: "amd64"},
+ {Key: hardware.AttrSandboxBundleHash, Value: "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"},
+ },
+ },
+ }
+}
+
// reportWorkerCapacity stands in for the worker's capacity report, which atelet
// forwards to WorkerService in a real cluster, with compute for actors of the
// size test templates declare. It waits for the worker cache, which placement
@@ -169,6 +188,9 @@ func reportWorkerCapacity(t *testing.T, tc *testContext, name string, actors int
Actors: actors,
Resources: resources.CPUMemory(int64(actors)*1000, int64(actors)*512<<20),
}
+ if toUpdate.Status.DefaultRuntime == nil {
+ toUpdate.Status.DefaultRuntime = defaultWorkerRuntime(toUpdate.GetSandboxClass())
+ }
return nil
}); err != nil {
t.Fatalf("setting capacity on worker %s: %v", name, err)
diff --git a/cmd/ateapi/internal/controlapi/functionaltest/worker_test.go b/cmd/ateapi/internal/controlapi/functionaltest/worker_test.go
index 01cc0e7306..d5fbebaf0d 100644
--- a/cmd/ateapi/internal/controlapi/functionaltest/worker_test.go
+++ b/cmd/ateapi/internal/controlapi/functionaltest/worker_test.go
@@ -168,8 +168,9 @@ func TestListWorkers(t *testing.T) {
SandboxClass: "gvisor",
Labels: map[string]string{"foo": "bar"},
Status: &ateapipb.WorkerStatus{
- State: ateapipb.WorkerState_WORKER_STATE_ACTIVE,
- Capacity: &ateapipb.WorkerResources{Actors: 1, Resources: resources.CPUMemory(1000, 512<<20)},
+ State: ateapipb.WorkerState_WORKER_STATE_ACTIVE,
+ Capacity: &ateapipb.WorkerResources{Actors: 1, Resources: resources.CPUMemory(1000, 512<<20)},
+ DefaultRuntime: defaultWorkerRuntime("gvisor"),
},
},
}
diff --git a/cmd/ateapi/internal/controlapi/hosting.go b/cmd/ateapi/internal/controlapi/hosting.go
index a0c93b1f6f..b155c9a7f7 100644
--- a/cmd/ateapi/internal/controlapi/hosting.go
+++ b/cmd/ateapi/internal/controlapi/hosting.go
@@ -18,8 +18,10 @@ import (
"context"
"errors"
"fmt"
+ "slices"
"go.ate.dev/substrate/cmd/ateapi/internal/store"
+ "go.ate.dev/substrate/internal/resources"
"go.ate.dev/substrate/pkg/proto/ateapipb"
)
@@ -42,6 +44,11 @@ import (
// actor it does not hold.
func HostingNode(ctx context.Context, st store.Interface, actor *ateapipb.Actor) (string, error) {
status := actor.GetStatus()
+ // The branches below pick the field to trust within the states a node
+ // transfers in.
+ if !slices.Contains(resources.TransferStates, status.GetState()) {
+ return "", nil
+ }
assignment := status.GetWorkerAssignment()
if assignment == nil {
if status.GetState() != ateapipb.ActorState_ACTOR_STATE_SUSPENDED {
diff --git a/cmd/ateapi/internal/controlapi/hosting_test.go b/cmd/ateapi/internal/controlapi/hosting_test.go
index 2e6556bf1c..54449ad009 100644
--- a/cmd/ateapi/internal/controlapi/hosting_test.go
+++ b/cmd/ateapi/internal/controlapi/hosting_test.go
@@ -166,7 +166,11 @@ func TestHostingNodeAfterResumeOntoNewWorker(t *testing.T) {
WorkerPodUid: testWorkerUID("stale-pod"),
NodeName: "node1",
SandboxClass: "microvm",
- Status: &ateapipb.WorkerStatus{State: ateapipb.WorkerState_WORKER_STATE_ACTIVE, Capacity: &ateapipb.WorkerResources{Actors: 1}},
+ Status: &ateapipb.WorkerStatus{
+ State: ateapipb.WorkerState_WORKER_STATE_ACTIVE,
+ Capacity: &ateapipb.WorkerResources{Actors: 1},
+ DefaultRuntime: &ateapipb.SandboxRuntime{SandboxClass: "microvm", Version: testSandboxRuntime().GetVersion()},
+ },
}
free := &ateapipb.Worker{
Metadata: &ateapipb.ResourceMetadata{Name: testWorkerUID("free-pod")},
@@ -177,7 +181,11 @@ func TestHostingNodeAfterResumeOntoNewWorker(t *testing.T) {
NodeName: "node2",
SandboxClass: "gvisor",
Labels: map[string]string{"pool": "b"},
- Status: &ateapipb.WorkerStatus{State: ateapipb.WorkerState_WORKER_STATE_ACTIVE, Capacity: &ateapipb.WorkerResources{Actors: 1}},
+ Status: &ateapipb.WorkerStatus{
+ State: ateapipb.WorkerState_WORKER_STATE_ACTIVE,
+ Capacity: &ateapipb.WorkerResources{Actors: 1},
+ DefaultRuntime: testSandboxRuntime(),
+ },
}
for _, w := range []*ateapipb.Worker{stale, free} {
if _, err := persistence.CreateWorker(ctx, w); err != nil {
diff --git a/cmd/ateapi/internal/controlapi/main_test.go b/cmd/ateapi/internal/controlapi/main_test.go
index d60fd4bc2e..9c30d006d5 100644
--- a/cmd/ateapi/internal/controlapi/main_test.go
+++ b/cmd/ateapi/internal/controlapi/main_test.go
@@ -15,11 +15,21 @@
package controlapi
import (
+ "fmt"
+ "os"
"testing"
"go.ate.dev/substrate/cmd/ateapi/internal/store/storetest"
+ "go.ate.dev/substrate/internal/preview"
)
func TestMain(m *testing.M) {
+ // Most tests store snapshots under their template's storageLocation, which
+ // is honored only with this gate. Tests that need it off call
+ // preview.InitForTest.
+ if err := preview.Init(string(preview.GatePerTemplateSnapshotLocation)); err != nil {
+ fmt.Fprintln(os.Stderr, err)
+ os.Exit(1)
+ }
storetest.RunTests(m)
}
diff --git a/cmd/ateapi/internal/controlapi/sandbox_assets.go b/cmd/ateapi/internal/controlapi/sandbox_assets.go
deleted file mode 100644
index ea8fdb83c6..0000000000
--- a/cmd/ateapi/internal/controlapi/sandbox_assets.go
+++ /dev/null
@@ -1,106 +0,0 @@
-// Copyright 2026 Google LLC
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package controlapi
-
-import (
- "fmt"
-
- "go.ate.dev/substrate/internal/apierror"
- "go.ate.dev/substrate/internal/proto/ateletpb"
- atev1alpha1 "go.ate.dev/substrate/pkg/api/v1alpha1"
- listersv1alpha1 "go.ate.dev/substrate/pkg/client/listers/api/v1alpha1"
- "go.ate.dev/substrate/pkg/proto/ateapipb"
- k8serrors "k8s.io/apimachinery/pkg/api/errors"
-)
-
-// resolvePoolSandboxConfig resolves the SandboxConfig the controller confirmed
-// for the WorkerPool of the actor's assigned worker.
-func resolvePoolSandboxConfig(
- workerPoolLister listersv1alpha1.WorkerPoolLister,
- sandboxConfigLister listersv1alpha1.SandboxConfigLister,
- assignment *ateapipb.WorkerAssignment,
-) (*atev1alpha1.SandboxConfig, error) {
- namespace, poolName := assignment.GetWorkerNamespace(), assignment.GetWorkerPool()
- if namespace == "" || poolName == "" {
- return nil, fmt.Errorf("worker assignment names no worker pool")
- }
- pool, err := workerPoolLister.WorkerPools(namespace).Get(poolName)
- if k8serrors.IsNotFound(err) {
- return nil, apierror.FailedPrecondition("WorkerPool %s/%s not found", namespace, poolName)
- }
- if err != nil {
- return nil, fmt.Errorf("while getting WorkerPool %s/%s: %w", namespace, poolName, err)
- }
- configName := pool.ResolvedSandboxConfig()
- if configName == "" {
- return nil, apierror.FailedPrecondition(
- "WorkerPool %s/%s has not resolved a SandboxConfig for sandbox class %q", namespace, poolName, pool.Spec.DefaultSandboxClass())
- }
- sc, err := sandboxConfigLister.Get(configName)
- if k8serrors.IsNotFound(err) {
- return nil, apierror.FailedPrecondition("SandboxConfig %q of WorkerPool %s/%s not found", configName, namespace, poolName)
- }
- if err != nil {
- return nil, fmt.Errorf("while getting SandboxConfig %q: %w", configName, err)
- }
- return sc, nil
-}
-
-// resolveSandboxAssets determines the sandbox binaries and pause image an actor
-// boots with on its assigned worker and projects them onto the
-// ateletpb.SandboxAssets atelet fetches: the default version of the
-// SandboxConfig the worker's pool resolved for its sandbox class, which is the
-// runtime the worker reports as its default.
-//
-// TODO: Remove with sandbox_assets. atelet already resolves the pool's
-// SandboxConfig and its versions for every worker on the node (its sandbox
-// runtimes syncer), and Run and Restore carry sandbox_runtime; once atelet
-// resolves the version from that field instead of taking the assets from the
-// request, this lookup, the WorkerPool and SandboxConfig listers on
-// ActorWorkflow, and ateapi's sandboxconfigs RBAC go.
-func resolveSandboxAssets(
- workerPoolLister listersv1alpha1.WorkerPoolLister,
- sandboxConfigLister listersv1alpha1.SandboxConfigLister,
- assignment *ateapipb.WorkerAssignment,
-) (*ateletpb.SandboxAssets, error) {
- sc, err := resolvePoolSandboxConfig(workerPoolLister, sandboxConfigLister, assignment)
- if err != nil {
- return nil, err
- }
- ver, ok := sc.Spec.DefaultVersionConfig()
- if !ok {
- return nil, apierror.FailedPrecondition(
- "SandboxConfig %q has no enabled version %q named by its defaultVersion", sc.Name, sc.Spec.DefaultVersion)
- }
- return sandboxAssetsProto(sc, ver), nil
-}
-
-// sandboxAssetsProto converts a version of a resolved SandboxConfig into the
-// proto atelet consumes.
-func sandboxAssetsProto(sc *atev1alpha1.SandboxConfig, ver *atev1alpha1.SandboxVersionConfig) *ateletpb.SandboxAssets {
- out := &ateletpb.SandboxAssets{
- SandboxClass: string(sc.Spec.SandboxClass),
- PauseImage: ver.PauseImage,
- Assets: make(map[string]*ateletpb.ArchAssets, len(ver.Assets)),
- }
- for arch, files := range ver.Assets {
- archAssets := &ateletpb.ArchAssets{Files: make(map[string]*ateletpb.AssetFile, len(files))}
- for name, f := range files {
- archAssets.Files[name] = &ateletpb.AssetFile{Url: f.URL, Sha256: f.SHA256}
- }
- out.Assets[arch] = archAssets
- }
- return out
-}
diff --git a/cmd/ateapi/internal/controlapi/sandbox_assets_test.go b/cmd/ateapi/internal/controlapi/sandbox_assets_test.go
deleted file mode 100644
index ae915b2eef..0000000000
--- a/cmd/ateapi/internal/controlapi/sandbox_assets_test.go
+++ /dev/null
@@ -1,186 +0,0 @@
-// Copyright 2026 Google LLC
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package controlapi
-
-import (
- "strings"
- "testing"
-
- atev1alpha1 "go.ate.dev/substrate/pkg/api/v1alpha1"
- listersv1alpha1 "go.ate.dev/substrate/pkg/client/listers/api/v1alpha1"
- "go.ate.dev/substrate/pkg/proto/ateapipb"
- metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
- "k8s.io/client-go/tools/cache"
- "k8s.io/utils/ptr"
-)
-
-// sandboxConfigListerFor builds a lister over the given SandboxConfigs, using
-// the same key function the informers use.
-func sandboxConfigListerFor(t *testing.T, configs []*atev1alpha1.SandboxConfig) listersv1alpha1.SandboxConfigLister {
- t.Helper()
- configIdx := cache.NewIndexer(cache.MetaNamespaceKeyFunc, cache.Indexers{})
- for _, c := range configs {
- if err := configIdx.Add(c); err != nil {
- t.Fatalf("adding SandboxConfig: %v", err)
- }
- }
- return listersv1alpha1.NewSandboxConfigLister(configIdx)
-}
-
-// workerPoolListerFor builds a lister over the given WorkerPools, using the
-// same key function the informers use.
-func workerPoolListerFor(t *testing.T, pools []*atev1alpha1.WorkerPool) listersv1alpha1.WorkerPoolLister {
- t.Helper()
- poolIdx := cache.NewIndexer(cache.MetaNamespaceKeyFunc, cache.Indexers{})
- for _, p := range pools {
- if err := poolIdx.Add(p); err != nil {
- t.Fatalf("adding WorkerPool: %v", err)
- }
- }
- return listersv1alpha1.NewWorkerPoolLister(poolIdx)
-}
-
-// resolvedWorkerPool is a gVisor WorkerPool whose controller-confirmed
-// SandboxConfig is configName; "" is a pool that has not resolved one yet.
-func resolvedWorkerPool(namespace, name, configName string) *atev1alpha1.WorkerPool {
- pool := &atev1alpha1.WorkerPool{
- ObjectMeta: metav1.ObjectMeta{Namespace: namespace, Name: name},
- Spec: atev1alpha1.WorkerPoolSpec{
- SandboxClasses: []atev1alpha1.WorkerPoolSandboxClass{{Name: atev1alpha1.SandboxClassGvisor}},
- },
- }
- status := atev1alpha1.WorkerPoolSandboxClassStatus{Name: atev1alpha1.SandboxClassGvisor}
- if configName != "" {
- status.ConfigRef = &atev1alpha1.SandboxConfigReference{Name: configName}
- }
- pool.Status.SandboxClasses = []atev1alpha1.WorkerPoolSandboxClassStatus{status}
- return pool
-}
-
-func testAssets() map[string]map[string]atev1alpha1.AssetFile {
- return map[string]map[string]atev1alpha1.AssetFile{
- "amd64": {"gvisor": {URL: "gs://bucket/gvisor.tar.bz2", SHA256: "abc"}},
- }
-}
-
-// TestResolveSandboxAssets pins the pool-side resolution: the SandboxConfig
-// the assigned worker's pool resolved is taken at its default version, a
-// missing default version, an unresolved pool, a missing pool or config is an
-// error, and the pause image travels with the sandbox binaries.
-func TestResolveSandboxAssets(t *testing.T) {
- const namedPause = "gcr.io/gke-release/pause@sha256:named"
- namedConfig := &atev1alpha1.SandboxConfig{
- ObjectMeta: metav1.ObjectMeta{Name: "gvisor-custom"},
- Spec: atev1alpha1.SandboxConfigSpec{
- SandboxClass: atev1alpha1.SandboxClassGvisor,
- DefaultVersion: "v2",
- Versions: []atev1alpha1.SandboxVersionConfig{{
- Name: "v1",
- PauseImage: "gcr.io/gke-release/pause@sha256:old",
- Assets: testAssets(),
- }, {
- Name: "v2",
- PauseImage: namedPause,
- Assets: testAssets(),
- }},
- },
- }
- // A config whose defaultVersion names no entry; the CRD rejects this, but
- // an object stored before the versions schema reads back this way.
- noDefaultConfig := &atev1alpha1.SandboxConfig{
- ObjectMeta: metav1.ObjectMeta{Name: "gvisor-no-default"},
- Spec: atev1alpha1.SandboxConfigSpec{SandboxClass: atev1alpha1.SandboxClassGvisor},
- }
- // A config whose defaultVersion names a Disabled entry; the CRD rejects
- // this too, and it must not resolve to the disabled version.
- disabledDefaultConfig := &atev1alpha1.SandboxConfig{
- ObjectMeta: metav1.ObjectMeta{Name: "gvisor-disabled-default"},
- Spec: atev1alpha1.SandboxConfigSpec{
- SandboxClass: atev1alpha1.SandboxClassGvisor,
- DefaultVersion: "v1",
- Versions: []atev1alpha1.SandboxVersionConfig{{
- Name: "v1",
- State: ptr.To(atev1alpha1.SandboxVersionStateDisabled),
- PauseImage: namedPause,
- Assets: testAssets(),
- }},
- },
- }
- pools := []*atev1alpha1.WorkerPool{
- resolvedWorkerPool("worker-ns", "custom", "gvisor-custom"),
- resolvedWorkerPool("worker-ns", "no-default", "gvisor-no-default"),
- resolvedWorkerPool("worker-ns", "disabled-default", "gvisor-disabled-default"),
- resolvedWorkerPool("worker-ns", "missing-config", "does-not-exist"),
- resolvedWorkerPool("worker-ns", "unresolved", ""),
- }
- assignment := func(pool string) *ateapipb.WorkerAssignment {
- return &ateapipb.WorkerAssignment{WorkerNamespace: "worker-ns", WorkerPool: pool}
- }
-
- tests := []struct {
- name string
- assignment *ateapipb.WorkerAssignment
- wantPauseImage string
- wantErr string
- }{{
- name: "pool with a resolved config",
- assignment: assignment("custom"),
- wantPauseImage: namedPause,
- }, {
- name: "config without its default version",
- assignment: assignment("no-default"),
- wantErr: `SandboxConfig "gvisor-no-default" has no enabled version ""`,
- }, {
- name: "config whose default version is disabled",
- assignment: assignment("disabled-default"),
- wantErr: `SandboxConfig "gvisor-disabled-default" has no enabled version "v1"`,
- }, {
- name: "pool whose config is missing",
- assignment: assignment("missing-config"),
- wantErr: `SandboxConfig "does-not-exist" of WorkerPool worker-ns/missing-config not found`,
- }, {
- name: "pool that has not resolved a config",
- assignment: assignment("unresolved"),
- wantErr: `WorkerPool worker-ns/unresolved has not resolved a SandboxConfig`,
- }, {
- name: "missing pool",
- assignment: assignment("does-not-exist"),
- wantErr: `WorkerPool worker-ns/does-not-exist not found`,
- }, {
- name: "assignment without a pool",
- assignment: &ateapipb.WorkerAssignment{},
- wantErr: "names no worker pool",
- }}
- for _, tt := range tests {
- t.Run(tt.name, func(t *testing.T) {
- poolLister := workerPoolListerFor(t, pools)
- configLister := sandboxConfigListerFor(t, []*atev1alpha1.SandboxConfig{namedConfig, noDefaultConfig, disabledDefaultConfig})
-
- got, err := resolveSandboxAssets(poolLister, configLister, tt.assignment)
- if tt.wantErr != "" {
- if err == nil || !strings.Contains(err.Error(), tt.wantErr) {
- t.Fatalf("resolveSandboxAssets() error = %v, want it to contain %q", err, tt.wantErr)
- }
- return
- }
- if err != nil {
- t.Fatalf("resolveSandboxAssets() error: %v", err)
- }
- if got.GetPauseImage() != tt.wantPauseImage {
- t.Errorf("pause image = %q, want %q", got.GetPauseImage(), tt.wantPauseImage)
- }
- })
- }
-}
diff --git a/cmd/ateapi/internal/controlapi/service.go b/cmd/ateapi/internal/controlapi/service.go
index 713dc3a6ec..511f554928 100644
--- a/cmd/ateapi/internal/controlapi/service.go
+++ b/cmd/ateapi/internal/controlapi/service.go
@@ -74,11 +74,12 @@ type VolumePluginRegistry interface {
//
// actorJWTIssuer is copied verbatim into the iss claim of every actor JWT.
// actorWorkflowDeadline bounds each Resume/Suspend workflow end-to-end.
+//
+// snapshotLocation is ate-api-server's cluster-wide --snapshot-location; see
+// resolveSnapshotLocation.
func NewRPCService(
persistence store.Interface,
workerCache *workercache.Cache,
- workerPoolLister listersv1alpha1.WorkerPoolLister,
- sandboxConfigLister listersv1alpha1.SandboxConfigLister,
csiDriverConfigLister listersv1alpha1.CSIDriverConfigLister,
storageClassLister storagev1listers.StorageClassLister,
dialer *AteletDialer,
@@ -90,8 +91,10 @@ func NewRPCService(
actorJWTIssuer string,
actorIDJWTPool localjwtauthority.Pool,
actorIDCAPool localca.Pool,
+ snapshotLocation string,
) *RPCService {
impl := newServiceImpl(persistence, storageClassLister)
+ impl.clusterSnapshotLocation = snapshotLocation
s := &RPCService{
impl: impl,
persistence: persistence,
@@ -105,7 +108,8 @@ func NewRPCService(
actorIDJWTPool: actorIDJWTPool,
actorIDCAPool: actorIDCAPool,
}
- s.actorWorkflow = NewActorWorkflow(impl, workerCache, dialer, workerPoolLister, sandboxConfigLister, storageClassLister, instruments, egressGatewayAddress, s, actorWorkflowDeadline, objectStorePlugin)
+ s.actorWorkflow = NewActorWorkflow(impl, workerCache, dialer, storageClassLister, instruments, egressGatewayAddress, s, actorWorkflowDeadline, objectStorePlugin)
+ s.actorWorkflow.clusterSnapshotLocation = snapshotLocation
s.workerWorkflow = NewWorkerWorkflow(impl)
return s
}
@@ -179,6 +183,10 @@ type ServiceImpl struct {
store store.Interface
storageClassLister storagev1listers.StorageClassLister
+
+ // clusterSnapshotLocation is ate-api-server's --snapshot-location; see
+ // resolveSnapshotLocation.
+ clusterSnapshotLocation string
}
var _ store.Interface = (*ServiceImpl)(nil)
diff --git a/cmd/ateapi/internal/controlapi/snapshot_helpers.go b/cmd/ateapi/internal/controlapi/snapshot_helpers.go
index f8ee69e529..6aa9977b98 100644
--- a/cmd/ateapi/internal/controlapi/snapshot_helpers.go
+++ b/cmd/ateapi/internal/controlapi/snapshot_helpers.go
@@ -15,6 +15,8 @@
package controlapi
import (
+ "go.ate.dev/substrate/internal/apierror"
+ "go.ate.dev/substrate/internal/preview"
"go.ate.dev/substrate/pkg/proto/ateapipb"
)
@@ -77,15 +79,6 @@ func latestSnapshotIsLocal(status *ateapipb.ActorStatus) bool {
return snapshotAtLatestGeneration(status).GetLocality() != ""
}
-// latestSnapshotIsLocalOnly reports whether the snapshot at the actor's latest
-// generation is node-local with no COMPLETED durable copy: a pause's snapshot
-// whose upload is not recorded yet.
-func latestSnapshotIsLocalOnly(status *ateapipb.ActorStatus) bool {
- snap := snapshotAtLatestGeneration(status)
- return snap.GetLocality() != "" &&
- snap.GetDurableSnapshot().GetStatus() != ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED
-}
-
// clearLocalSnapshots clears Locality across all Snapshots in status.Snapshots
// without dropping PENDING durable snapshots (so DeleteActor can still collect
// an in-flight durable snapshot after releasing the worker).
@@ -158,3 +151,25 @@ func tagDurableSnapshotURI(tag *ateapipb.Tag, storageStatus ateapipb.SnapshotSto
}
return st.GetObject().GetSnapshotUri()
}
+
+// resolveSnapshotLocation returns the base object-storage URI actorTemplate's
+// snapshots are stored under: its snapshotConfig.storage_location when the
+// PerTemplateSnapshotLocation preview gate is enabled and the field is set,
+// otherwise snapshotLocation, ate-api-server's cluster-wide
+// --snapshot-location. It fails with FailedPrecondition when neither is set.
+//
+// It reads the process-wide gate rather than preview.IsEnabledFor, so a
+// request's preview opt-out cannot move where an actor's snapshots live.
+func resolveSnapshotLocation(snapshotLocation string, actorTemplate *ateapipb.ActorTemplate) (string, error) {
+ if preview.IsEnabled(preview.GatePerTemplateSnapshotLocation) {
+ if location := actorTemplate.GetSnapshotConfig().GetStorageLocation(); location != "" {
+ return location, nil
+ }
+ }
+ if snapshotLocation != "" {
+ return snapshotLocation, nil
+ }
+ return "", apierror.FailedPrecondition(
+ "actor template %s/%s has no snapshot storage location: set snapshotConfig.storageLocation, or set ate-api-server's --snapshot-location",
+ actorTemplate.GetMetadata().GetAtespace(), actorTemplate.GetMetadata().GetName())
+}
diff --git a/cmd/ateapi/internal/controlapi/snapshot_helpers_test.go b/cmd/ateapi/internal/controlapi/snapshot_helpers_test.go
index b535d70072..207559c4d0 100644
--- a/cmd/ateapi/internal/controlapi/snapshot_helpers_test.go
+++ b/cmd/ateapi/internal/controlapi/snapshot_helpers_test.go
@@ -18,7 +18,10 @@ import (
"testing"
"github.com/google/go-cmp/cmp"
+ "go.ate.dev/substrate/internal/apierror"
+ "go.ate.dev/substrate/internal/preview"
"go.ate.dev/substrate/pkg/proto/ateapipb"
+ "google.golang.org/grpc/codes"
"google.golang.org/protobuf/testing/protocmp"
)
@@ -204,45 +207,6 @@ func TestSnapshotPruningHelpers(t *testing.T) {
})
}
-// TestLatestSnapshotIsLocalOnly pins a pause's snapshot whose upload is not
-// recorded yet: the latest snapshot is node-local with no COMPLETED durable
-// copy, whether it has no copy or a PENDING one. A durable latest snapshot, or
-// a local snapshot that is not captured yet, do not count; a RUNNING actor
-// that kept its local snapshot across a resume does.
-func TestLatestSnapshotIsLocalOnly(t *testing.T) {
- local := func(gen int32) *ateapipb.Snapshot {
- return newPausedSnapshot(t, gen, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", "local", "node-1")
- }
- durable := func(gen int32, st ateapipb.SnapshotStorageStatus) *ateapipb.Snapshot {
- return newDurableSnapshot(gen, ateapipb.SnapshotOwner_SNAPSHOT_OWNER_ACTOR, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", "durable", "gs://b/durable", st)
- }
- uploading := local(2)
- uploading.DurableSnapshot = &ateapipb.SnapshotStorage{Status: ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_PENDING, Object: &ateapipb.ObjectSnapshot{SnapshotUri: "gs://b/up"}}
-
- tests := []struct {
- name string
- status *ateapipb.ActorStatus
- want bool
- }{
- {"nil status", nil, false},
- {"no snapshots", &ateapipb.ActorStatus{}, false},
- {"local only above a durable one", &ateapipb.ActorStatus{LastAssignedGeneration: 2, Snapshots: []*ateapipb.Snapshot{durable(1, ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED), local(2)}}, true},
- {"local only alone", &ateapipb.ActorStatus{LastAssignedGeneration: 1, Snapshots: []*ateapipb.Snapshot{local(1)}}, true},
- {"durable only", &ateapipb.ActorStatus{LastAssignedGeneration: 2, Snapshots: []*ateapipb.Snapshot{durable(2, ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED)}}, false},
- {"local with an upload pending", &ateapipb.ActorStatus{LastAssignedGeneration: 2, Snapshots: []*ateapipb.Snapshot{uploading}}, true},
- {"local not yet captured", &ateapipb.ActorStatus{LastAssignedGeneration: 1, Snapshots: []*ateapipb.Snapshot{newPausedSnapshot(t, 1, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", "local", "")}}, false},
- {"older local only under a pause in progress", &ateapipb.ActorStatus{LastAssignedGeneration: 2, Snapshots: []*ateapipb.Snapshot{local(1), newPausedSnapshot(t, 2, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", "local", "")}}, false},
- {"latest generation dropped by a revert", &ateapipb.ActorStatus{LastAssignedGeneration: 3, Snapshots: []*ateapipb.Snapshot{local(2)}}, false},
- }
- for _, tc := range tests {
- t.Run(tc.name, func(t *testing.T) {
- if got := latestSnapshotIsLocalOnly(tc.status); got != tc.want {
- t.Errorf("latestSnapshotIsLocalOnly = %t, want %t", got, tc.want)
- }
- })
- }
-}
-
// TestLatestSnapshotIsLocal pins the shape a paused actor has: the latest
// snapshot is node-local, whether or not atelet has uploaded it. A latest
// snapshot with only a durable copy, or a local one not captured yet, does not
@@ -301,3 +265,85 @@ func TestTagDurableSnapshotURI(t *testing.T) {
t.Errorf("tagDurableSnapshotURI(pendingTag, UNSPECIFIED) = %q, want gs://b/tag-1", got)
}
}
+
+func TestResolveSnapshotLocation(t *testing.T) {
+ const (
+ templateLocation = "gs://template-bucket/root"
+ bucketPrefix = "gs://flag-bucket/prefix/"
+ )
+ tests := []struct {
+ name string
+ gateEnabled bool
+ bucketPrefix string
+ templateLocation string
+ want string
+ wantErr bool
+ }{
+ {
+ name: "gate disabled uses the flag and ignores the template location",
+ bucketPrefix: bucketPrefix,
+ templateLocation: templateLocation,
+ want: bucketPrefix,
+ },
+ {
+ name: "gate disabled uses the flag when the template sets none",
+ bucketPrefix: bucketPrefix,
+ want: bucketPrefix,
+ },
+ {
+ name: "gate disabled fails without the flag even with a template location",
+ templateLocation: templateLocation,
+ wantErr: true,
+ },
+ {
+ name: "gate enabled prefers the template location",
+ gateEnabled: true,
+ bucketPrefix: bucketPrefix,
+ templateLocation: templateLocation,
+ want: templateLocation,
+ },
+ {
+ name: "gate enabled falls back to the flag",
+ gateEnabled: true,
+ bucketPrefix: bucketPrefix,
+ want: bucketPrefix,
+ },
+ {
+ name: "gate enabled without the flag uses the template location",
+ gateEnabled: true,
+ templateLocation: templateLocation,
+ want: templateLocation,
+ },
+ {
+ name: "gate enabled with neither fails",
+ gateEnabled: true,
+ wantErr: true,
+ },
+ }
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ var gates []string
+ if tt.gateEnabled {
+ gates = []string{string(preview.GatePerTemplateSnapshotLocation)}
+ }
+ preview.InitForTest(t, gates...)
+ tmpl := &ateapipb.ActorTemplate{
+ Metadata: &ateapipb.ResourceMetadata{Atespace: "team-a", Name: "tmpl"},
+ SnapshotConfig: &ateapipb.SnapshotConfig{StorageLocation: tt.templateLocation},
+ }
+ got, err := resolveSnapshotLocation(tt.bucketPrefix, tmpl)
+ if tt.wantErr {
+ if apierror.Code(err) != codes.FailedPrecondition {
+ t.Fatalf("resolveSnapshotLocation() err = %v, want FailedPrecondition", err)
+ }
+ return
+ }
+ if err != nil {
+ t.Fatalf("resolveSnapshotLocation() err = %v", err)
+ }
+ if got != tt.want {
+ t.Errorf("resolveSnapshotLocation() = %q, want %q", got, tt.want)
+ }
+ })
+ }
+}
diff --git a/cmd/ateapi/internal/controlapi/snapshot_report.go b/cmd/ateapi/internal/controlapi/snapshot_report.go
index ca7355d7b5..5f57ff26f4 100644
--- a/cmd/ateapi/internal/controlapi/snapshot_report.go
+++ b/cmd/ateapi/internal/controlapi/snapshot_report.go
@@ -153,12 +153,12 @@ func applyActorSnapshotState(ctx context.Context, status *ateapipb.ActorStatus,
return nil
}
-// applyCheckpointState records an upload that atelet's state, as returned by
-// Checkpoint, says finished, with the same rules as a report. It records the
-// fidelity atelet reports, which for Checkpoint is the one requested. Unlike a
-// report, a state with no local snapshot is not a detach. The workflow that
-// made the call records where its snapshot lives.
-func applyCheckpointState(ctx context.Context, status *ateapipb.ActorStatus, state *ateletpb.ActorSnapshotState) error {
+// applyAteletResponseState records an upload that atelet's state, as returned
+// by Checkpoint or PromoteSnapshot, says finished, with the same rules as a
+// report. It records the fidelity atelet reports, which for both RPCs is the
+// one requested. Unlike a report, a state with no local snapshot is not a
+// detach. The workflow that made the call records where its snapshot lives.
+func applyAteletResponseState(ctx context.Context, status *ateapipb.ActorStatus, state *ateletpb.ActorSnapshotState) error {
return applyLatestDurable(ctx, status, snapshotReportFromAtelet(state.GetLatestDurable()))
}
diff --git a/cmd/ateapi/internal/controlapi/snapshot_report_test.go b/cmd/ateapi/internal/controlapi/snapshot_report_test.go
index 19228f3cf4..6921978722 100644
--- a/cmd/ateapi/internal/controlapi/snapshot_report_test.go
+++ b/cmd/ateapi/internal/controlapi/snapshot_report_test.go
@@ -555,7 +555,7 @@ func ateletReported(t *testing.T, gen int32) *ateletpb.SnapshotReport {
}}
}
-func TestApplyCheckpointState(t *testing.T) {
+func TestApplyAteletResponseState(t *testing.T) {
const (
pending = ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_PENDING
completed = ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED
@@ -674,15 +674,15 @@ func TestApplyCheckpointState(t *testing.T) {
}}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
- err := applyCheckpointState(context.Background(), tt.status, tt.state)
+ err := applyAteletResponseState(context.Background(), tt.status, tt.state)
if tt.wantCode != codes.OK {
if got := apierror.Code(err); got != tt.wantCode {
- t.Fatalf("applyCheckpointState() error = %v, want code %v", err, tt.wantCode)
+ t.Fatalf("applyAteletResponseState() error = %v, want code %v", err, tt.wantCode)
}
return
}
if err != nil {
- t.Fatalf("applyCheckpointState() error = %v", err)
+ t.Fatalf("applyAteletResponseState() error = %v", err)
}
if diff := cmp.Diff(tt.want, tt.status, protocmp.Transform()); diff != "" {
t.Errorf("status mismatch (-want +got):\n%s", diff)
diff --git a/cmd/ateapi/internal/controlapi/volumes.go b/cmd/ateapi/internal/controlapi/volumes.go
index fa9e25194e..dba16763d5 100644
--- a/cmd/ateapi/internal/controlapi/volumes.go
+++ b/cmd/ateapi/internal/controlapi/volumes.go
@@ -29,16 +29,25 @@ import (
storagev1listers "k8s.io/client-go/listers/storage/v1"
)
+// externalVolumesStatus wraps vols for ActorStatus, leaving the field unset
+// when the actor has no external volumes.
+func externalVolumesStatus(vols []*ateapipb.ExternalVolumeStatus) *ateapipb.ExternalVolumes {
+ if len(vols) == 0 {
+ return nil
+ }
+ return &ateapipb.ExternalVolumes{Volumes: vols}
+}
+
// initialExternalVolumes constructs the actor's volume objects before volume
// creation. A volume has no storage volume ID until it is provisioned.
-func initialExternalVolumes(ctx context.Context, scLister storagev1listers.StorageClassLister, template *ateapipb.ActorTemplate) ([]*ateapipb.ExternalVolume, error) {
+func initialExternalVolumes(ctx context.Context, scLister storagev1listers.StorageClassLister, template *ateapipb.ActorTemplate) ([]*ateapipb.ExternalVolumeStatus, error) {
if template == nil {
return nil, apierror.InvalidArgument("template is required")
}
- var volumes []*ateapipb.ExternalVolume
+ var volumes []*ateapipb.ExternalVolumeStatus
for _, vol := range template.GetVolumes() {
- if vol.GetExternalVolumeTemplate() != nil {
- scName := vol.GetExternalVolumeTemplate().GetStorageClassName()
+ if vol.GetExternal().GetTemplate() != nil {
+ scName := vol.GetExternal().GetTemplate().GetStorageClassName()
sc, err := scLister.Get(scName)
if err != nil {
if k8serrors.IsNotFound(err) {
@@ -47,7 +56,7 @@ func initialExternalVolumes(ctx context.Context, scLister storagev1listers.Stora
return nil, apierror.Internal("failed to get StorageClass %q: %v", scName, err)
}
- volumes = append(volumes, &ateapipb.ExternalVolume{
+ volumes = append(volumes, &ateapipb.ExternalVolumeStatus{
Name: vol.GetName(),
VolumeType: sc.Provisioner,
})
@@ -59,8 +68,8 @@ func initialExternalVolumes(ctx context.Context, scLister storagev1listers.Stora
// createActorVolumes provisions the external volumes in volumesToCreate that have no storage volume ID, using the provided volume plugin.
// It returns the list of external volumes (with storage IDs), or an error if any creation fails.
// Any volumes processed before or during a failure are returned alongside the error so they can be persisted on the actor.
-func createActorVolumes(ctx context.Context, registry VolumePluginRegistry, scLister storagev1listers.StorageClassLister, actorUID string, template *ateapipb.ActorTemplate, volumesToCreate []*ateapipb.ExternalVolume) (resultVolumes []*ateapipb.ExternalVolume, err error) {
- resultVolumes = make([]*ateapipb.ExternalVolume, 0, len(volumesToCreate))
+func createActorVolumes(ctx context.Context, registry VolumePluginRegistry, scLister storagev1listers.StorageClassLister, actorUID string, template *ateapipb.ActorTemplate, volumesToCreate []*ateapipb.ExternalVolumeStatus) (resultVolumes []*ateapipb.ExternalVolumeStatus, err error) {
+ resultVolumes = make([]*ateapipb.ExternalVolumeStatus, 0, len(volumesToCreate))
var currentIdx int
defer func() {
@@ -82,7 +91,7 @@ func createActorVolumes(ctx context.Context, registry VolumePluginRegistry, scLi
break
}
}
- if specVol == nil || specVol.GetExternalVolumeTemplate() == nil {
+ if specVol == nil || specVol.GetExternal().GetTemplate() == nil {
return resultVolumes, apierror.NotFound("volume %q not found in template", volName)
}
@@ -93,7 +102,7 @@ func createActorVolumes(ctx context.Context, registry VolumePluginRegistry, scLi
actVolID := actorVolumeID(actorUID, volName)
- scName := specVol.GetExternalVolumeTemplate().GetStorageClassName()
+ scName := specVol.GetExternal().GetTemplate().GetStorageClassName()
sc, err := scLister.Get(scName)
if err != nil {
if k8serrors.IsNotFound(err) {
@@ -113,7 +122,7 @@ func createActorVolumes(ctx context.Context, registry VolumePluginRegistry, scLi
resp, volErr := plugin.CreateVolume(ctx, volume.CreateVolumeRequest{
Name: actVolID,
- Capacity: specVol.GetExternalVolumeTemplate().GetCapacity(),
+ Capacity: specVol.GetExternal().GetTemplate().GetCapacity(),
Parameters: sc.Parameters,
})
if volErr != nil {
@@ -125,7 +134,7 @@ func createActorVolumes(ctx context.Context, registry VolumePluginRegistry, scLi
return resultVolumes, apierror.Internal("volume plugin returned no ID for volume %q", specVol.GetName())
}
- resultVolumes = append(resultVolumes, &ateapipb.ExternalVolume{
+ resultVolumes = append(resultVolumes, &ateapipb.ExternalVolumeStatus{
Name: volName,
StorageVolumeId: resp.VolumeID,
VolumeType: sc.Provisioner,
@@ -136,7 +145,7 @@ func createActorVolumes(ctx context.Context, registry VolumePluginRegistry, scLi
}
// deleteActorVolumes deletes all external volumes in the list.
-func deleteActorVolumes(ctx context.Context, registry VolumePluginRegistry, actorUID string, volumes []*ateapipb.ExternalVolume) error {
+func deleteActorVolumes(ctx context.Context, registry VolumePluginRegistry, actorUID string, volumes []*ateapipb.ExternalVolumeStatus) error {
if actorUID == "" {
return errors.New("actorUID is required")
}
@@ -168,8 +177,8 @@ func deleteActorVolumes(ctx context.Context, registry VolumePluginRegistry, acto
}
// getMountedActorVolumes filters the actor's volumes and returns only those that are declared and mounted in the ActorTemplate.
-func getMountedActorVolumes(ctx context.Context, ref *ateapipb.ObjectRef, volumes []*ateapipb.ExternalVolume, template *ateapipb.ActorTemplate) []*ateapipb.ExternalVolume {
- var mounted []*ateapipb.ExternalVolume
+func getMountedActorVolumes(ctx context.Context, ref *ateapipb.ObjectRef, volumes []*ateapipb.ExternalVolumeStatus, template *ateapipb.ActorTemplate) []*ateapipb.ExternalVolumeStatus {
+ var mounted []*ateapipb.ExternalVolumeStatus
for _, vol := range volumes {
// Find the corresponding volume in the ActorTemplate to check if it's mounted
var matchedTemplateVol *ateapipb.Volume
@@ -211,9 +220,9 @@ func detachActorVolumes(ctx context.Context, registry VolumePluginRegistry, acto
// in the template's containers. If the template is missing/deleted, fall back to
// attempting detachment for all external volumes recorded on the actor so we do
// not orphan attached disks on the worker node.
- volumesToDetach := actor.GetStatus().GetExternalVolumes()
+ volumesToDetach := actor.GetStatus().GetExternalVolumes().GetVolumes()
if template != nil {
- volumesToDetach = getMountedActorVolumes(ctx, ref, actor.GetStatus().GetExternalVolumes(), template)
+ volumesToDetach = getMountedActorVolumes(ctx, ref, actor.GetStatus().GetExternalVolumes().GetVolumes(), template)
}
// Collect errors for all volumes to detach, but continue processing so we attempt to detach all volumes.
var errs []error
diff --git a/cmd/ateapi/internal/controlapi/volumes_test.go b/cmd/ateapi/internal/controlapi/volumes_test.go
index 129cee7193..496ba31ca2 100644
--- a/cmd/ateapi/internal/controlapi/volumes_test.go
+++ b/cmd/ateapi/internal/controlapi/volumes_test.go
@@ -63,12 +63,12 @@ func TestActorVolumesStorageClassErrors(t *testing.T) {
tmpl := &ateapipb.ActorTemplate{
Volumes: []*ateapipb.Volume{{
Name: "data-vol",
- ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{
+ External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{
StorageClassName: "standard",
- },
+ }},
}},
}
- volumes := []*ateapipb.ExternalVolume{{
+ volumes := []*ateapipb.ExternalVolumeStatus{{
Name: "data-vol",
VolumeType: "mock-standard",
}}
@@ -120,9 +120,9 @@ func TestInitialActorVolumes(t *testing.T) {
Volumes: []*ateapipb.Volume{
{
Name: "data-vol-1",
- ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{
+ External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{
StorageClassName: "standard",
- },
+ }},
},
{
Name: "scratch-vol",
@@ -133,14 +133,14 @@ func TestInitialActorVolumes(t *testing.T) {
},
{
Name: "data-vol-2",
- ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{
+ External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{
StorageClassName: "fast",
- },
+ }},
},
},
}
- want := []*ateapipb.ExternalVolume{
+ want := []*ateapipb.ExternalVolumeStatus{
{
Name: "data-vol-1",
VolumeType: "mock-standard",
@@ -179,9 +179,9 @@ func TestCreateActorVolumes(t *testing.T) {
Volumes: []*ateapipb.Volume{
{
Name: "data-vol",
- ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{
+ External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{
StorageClassName: "standard",
- },
+ }},
},
},
}
@@ -190,21 +190,21 @@ func TestCreateActorVolumes(t *testing.T) {
Volumes: []*ateapipb.Volume{
{
Name: "vol1",
- ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{
+ External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{
StorageClassName: "standard",
- },
+ }},
},
{
Name: "vol2",
- ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{
+ External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{
StorageClassName: "standard",
- },
+ }},
},
{
Name: "vol3",
- ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{
+ External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{
StorageClassName: "standard",
- },
+ }},
},
},
}
@@ -212,22 +212,22 @@ func TestCreateActorVolumes(t *testing.T) {
tests := []struct {
name string
tmpl *ateapipb.ActorTemplate
- inputVolumes []*ateapipb.ExternalVolume
+ inputVolumes []*ateapipb.ExternalVolumeStatus
storageClasses map[string]*storagev1.StorageClass
wantErr bool
- wantRes []*ateapipb.ExternalVolume
+ wantRes []*ateapipb.ExternalVolumeStatus
}{
{
name: "partial failure returns error and preserves succeeded, failed, and remaining volumes",
tmpl: multiVolTmpl,
- inputVolumes: []*ateapipb.ExternalVolume{
+ inputVolumes: []*ateapipb.ExternalVolumeStatus{
{Name: "vol1", VolumeType: "mock-standard"},
// Its type differs from its StorageClass's provisioner, so creating it fails.
{Name: "vol2", VolumeType: "mock-fast"},
{Name: "vol3", VolumeType: "mock-standard"},
},
wantErr: true,
- wantRes: []*ateapipb.ExternalVolume{
+ wantRes: []*ateapipb.ExternalVolumeStatus{
{
Name: "vol1",
StorageVolumeId: "mock-vol-substrate-actor-uid-123-vol1",
@@ -240,14 +240,14 @@ func TestCreateActorVolumes(t *testing.T) {
{
name: "volume with a storage volume id is not created again",
tmpl: standardTmpl,
- inputVolumes: []*ateapipb.ExternalVolume{
+ inputVolumes: []*ateapipb.ExternalVolumeStatus{
{
Name: "data-vol",
StorageVolumeId: "existing-vol-id",
},
},
wantErr: false,
- wantRes: []*ateapipb.ExternalVolume{
+ wantRes: []*ateapipb.ExternalVolumeStatus{
{
Name: "data-vol",
StorageVolumeId: "existing-vol-id",
@@ -257,18 +257,18 @@ func TestCreateActorVolumes(t *testing.T) {
{
name: "volume not found in template returns error",
tmpl: &ateapipb.ActorTemplate{},
- inputVolumes: []*ateapipb.ExternalVolume{
+ inputVolumes: []*ateapipb.ExternalVolumeStatus{
{Name: "missing-vol"},
},
wantErr: true,
- wantRes: []*ateapipb.ExternalVolume{
+ wantRes: []*ateapipb.ExternalVolumeStatus{
{Name: "missing-vol"},
},
},
{
name: "storage class parameters are propagated to volume context",
tmpl: standardTmpl,
- inputVolumes: []*ateapipb.ExternalVolume{
+ inputVolumes: []*ateapipb.ExternalVolumeStatus{
{
Name: "data-vol",
VolumeType: "mock-standard",
@@ -285,7 +285,7 @@ func TestCreateActorVolumes(t *testing.T) {
},
},
wantErr: false,
- wantRes: []*ateapipb.ExternalVolume{
+ wantRes: []*ateapipb.ExternalVolumeStatus{
{
Name: "data-vol",
StorageVolumeId: "mock-vol-substrate-actor-uid-123-data-vol",
@@ -345,8 +345,8 @@ func (emptyIDVolumePlugin) CreateVolume(context.Context, volume.CreateVolumeRequ
func TestCreateActorVolumesRejectsEmptyPluginID(t *testing.T) {
tmpl := &ateapipb.ActorTemplate{
Volumes: []*ateapipb.Volume{{
- Name: "data-vol",
- ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{StorageClassName: "standard"},
+ Name: "data-vol",
+ External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{StorageClassName: "standard"}},
}},
}
registry := &mockPluginRegistry{
@@ -355,7 +355,7 @@ func TestCreateActorVolumesRejectsEmptyPluginID(t *testing.T) {
scLister := &fakeStorageClassLister{storageClasses: map[string]*storagev1.StorageClass{
"standard": {ObjectMeta: metav1.ObjectMeta{Name: "standard"}, Provisioner: "mock-standard"},
}}
- input := []*ateapipb.ExternalVolume{{Name: "data-vol", VolumeType: "mock-standard"}}
+ input := []*ateapipb.ExternalVolumeStatus{{Name: "data-vol", VolumeType: "mock-standard"}}
res, err := createActorVolumes(context.Background(), registry, scLister, "actor-uid-123", tmpl, input)
if got := apierror.Code(err); got != codes.Internal {
@@ -383,14 +383,14 @@ func TestDeleteActorVolumes(t *testing.T) {
tests := []struct {
name string
actorUID string
- volumes []*ateapipb.ExternalVolume
+ volumes []*ateapipb.ExternalVolumeStatus
wantDeleted []string
wantErr bool
}{
{
name: "uses storage volume ID when present",
actorUID: "uid-abc",
- volumes: []*ateapipb.ExternalVolume{
+ volumes: []*ateapipb.ExternalVolumeStatus{
{Name: "vol1", StorageVolumeId: "storage-vol-123", VolumeType: "mock"},
},
wantDeleted: []string{"storage-vol-123"},
@@ -399,7 +399,7 @@ func TestDeleteActorVolumes(t *testing.T) {
{
name: "falls back to actorVolumeID when storage volume ID is empty",
actorUID: "uid-abc",
- volumes: []*ateapipb.ExternalVolume{
+ volumes: []*ateapipb.ExternalVolumeStatus{
{Name: "vol1", StorageVolumeId: "", VolumeType: "mock"},
},
wantDeleted: []string{"substrate-uid-abc-vol1"},
@@ -479,10 +479,10 @@ func TestDetachActorVolumes(t *testing.T) {
Metadata: &ateapipb.ResourceMetadata{Name: "actor-1", Atespace: "default"},
Status: &ateapipb.ActorStatus{
AssignedNode: "node-1",
- ExternalVolumes: []*ateapipb.ExternalVolume{
+ ExternalVolumes: &ateapipb.ExternalVolumes{Volumes: []*ateapipb.ExternalVolumeStatus{
{Name: "vol1", StorageVolumeId: "storage-vol-1", VolumeType: "mock"},
{Name: "vol2", StorageVolumeId: "storage-vol-2", VolumeType: "mock"},
- },
+ }},
},
},
template: &ateapipb.ActorTemplate{
@@ -510,10 +510,10 @@ func TestDetachActorVolumes(t *testing.T) {
Metadata: &ateapipb.ResourceMetadata{Name: "actor-1", Atespace: "default"},
Status: &ateapipb.ActorStatus{
AssignedNode: "node-1",
- ExternalVolumes: []*ateapipb.ExternalVolume{
+ ExternalVolumes: &ateapipb.ExternalVolumes{Volumes: []*ateapipb.ExternalVolumeStatus{
{Name: "mounted-vol", StorageVolumeId: "storage-vol-mounted", VolumeType: "mock"},
{Name: "unmounted-vol", StorageVolumeId: "storage-vol-unmounted", VolumeType: "mock"},
- },
+ }},
},
},
template: &ateapipb.ActorTemplate{
@@ -539,10 +539,10 @@ func TestDetachActorVolumes(t *testing.T) {
Metadata: &ateapipb.ResourceMetadata{Name: "actor-1", Atespace: "default"},
Status: &ateapipb.ActorStatus{
AssignedNode: "node-1",
- ExternalVolumes: []*ateapipb.ExternalVolume{
+ ExternalVolumes: &ateapipb.ExternalVolumes{Volumes: []*ateapipb.ExternalVolumeStatus{
{Name: "vol1", StorageVolumeId: "", VolumeType: "mock"},
{Name: "vol2", StorageVolumeId: "storage-vol-2", VolumeType: "mock"},
- },
+ }},
},
},
template: &ateapipb.ActorTemplate{
@@ -569,10 +569,10 @@ func TestDetachActorVolumes(t *testing.T) {
Metadata: &ateapipb.ResourceMetadata{Name: "actor-1", Atespace: "default"},
Status: &ateapipb.ActorStatus{
AssignedNode: "node-1",
- ExternalVolumes: []*ateapipb.ExternalVolume{
+ ExternalVolumes: &ateapipb.ExternalVolumes{Volumes: []*ateapipb.ExternalVolumeStatus{
{Name: "vol1", StorageVolumeId: "storage-vol-1", VolumeType: "mock"},
{Name: "vol2", StorageVolumeId: "storage-vol-2", VolumeType: "mock"},
- },
+ }},
},
},
template: nil,
@@ -587,9 +587,9 @@ func TestDetachActorVolumes(t *testing.T) {
Metadata: &ateapipb.ResourceMetadata{Name: "actor-1", Atespace: "default"},
Status: &ateapipb.ActorStatus{
AssignedNode: "node-1",
- ExternalVolumes: []*ateapipb.ExternalVolume{
+ ExternalVolumes: &ateapipb.ExternalVolumes{Volumes: []*ateapipb.ExternalVolumeStatus{
{Name: "vol1", StorageVolumeId: "storage-vol-1", VolumeType: "mock"},
- },
+ }},
},
},
plugin: &mockDetachVolumePlugin{
@@ -608,10 +608,10 @@ func TestDetachActorVolumes(t *testing.T) {
Metadata: &ateapipb.ResourceMetadata{Name: "actor-1", Atespace: "default"},
Status: &ateapipb.ActorStatus{
AssignedNode: "node-1",
- ExternalVolumes: []*ateapipb.ExternalVolume{
+ ExternalVolumes: &ateapipb.ExternalVolumes{Volumes: []*ateapipb.ExternalVolumeStatus{
{Name: "vol1", StorageVolumeId: "storage-vol-1", VolumeType: "mock"},
{Name: "vol2", StorageVolumeId: "storage-vol-2", VolumeType: "mock"},
- },
+ }},
},
},
plugin: &mockDetachVolumePlugin{
@@ -632,9 +632,9 @@ func TestDetachActorVolumes(t *testing.T) {
Metadata: &ateapipb.ResourceMetadata{Name: "actor-1", Atespace: "default"},
Status: &ateapipb.ActorStatus{
AssignedNode: "node-1",
- ExternalVolumes: []*ateapipb.ExternalVolume{
+ ExternalVolumes: &ateapipb.ExternalVolumes{Volumes: []*ateapipb.ExternalVolumeStatus{
{Name: "vol1", StorageVolumeId: "storage-vol-1", VolumeType: "unknown-plugin"},
- },
+ }},
},
},
wantErr: true,
@@ -646,9 +646,9 @@ func TestDetachActorVolumes(t *testing.T) {
Metadata: &ateapipb.ResourceMetadata{Name: "actor-1", Atespace: "default"},
Status: &ateapipb.ActorStatus{
AssignedNode: "",
- ExternalVolumes: []*ateapipb.ExternalVolume{
+ ExternalVolumes: &ateapipb.ExternalVolumes{Volumes: []*ateapipb.ExternalVolumeStatus{
{Name: "vol1", StorageVolumeId: "storage-vol-1", VolumeType: "mock"},
- },
+ }},
},
},
wantDetachCalls: nil,
diff --git a/cmd/ateapi/internal/controlapi/workflow.go b/cmd/ateapi/internal/controlapi/workflow.go
index 832d616bc3..c7b8cbb3c1 100644
--- a/cmd/ateapi/internal/controlapi/workflow.go
+++ b/cmd/ateapi/internal/controlapi/workflow.go
@@ -29,7 +29,6 @@ import (
"go.ate.dev/substrate/internal/ateattr"
"go.ate.dev/substrate/internal/pluginsocket"
"go.ate.dev/substrate/internal/resources"
- listersv1alpha1 "go.ate.dev/substrate/pkg/client/listers/api/v1alpha1"
"go.ate.dev/substrate/pkg/proto/ateapipb"
objectstorev1 "go.ate.dev/substrate/pkg/proto/objectstorepb/v1"
"go.opentelemetry.io/otel"
@@ -106,18 +105,17 @@ func logActorState(ctx context.Context, actor *ateapipb.Actor, opName, state str
// each exported method begins with preview.ContextWithOptIn: the request was
// already validated, and what the workflow writes is the system's own state.
type ActorWorkflow struct {
- store actorWorkflowStore
- workerCache *workercache.Cache
- scheduler scheduling.Scheduler
- dialer *AteletDialer
- workerPoolLister listersv1alpha1.WorkerPoolLister
- sandboxConfigLister listersv1alpha1.SandboxConfigLister
- storageClassLister storagev1listers.StorageClassLister
- instruments *Instruments
- egressGatewayAddress string
- pluginRegistry VolumePluginRegistry
- workflowDeadline time.Duration
- objectStorePlugin objectstorev1.ControlProviderClient
+ store actorWorkflowStore
+ workerCache *workercache.Cache
+ scheduler scheduling.Scheduler
+ dialer *AteletDialer
+ storageClassLister storagev1listers.StorageClassLister
+ instruments *Instruments
+ egressGatewayAddress string
+ pluginRegistry VolumePluginRegistry
+ workflowDeadline time.Duration
+ objectStorePlugin objectstorev1.ControlProviderClient
+ clusterSnapshotLocation string
}
// NewActorWorkflow creates a new ActorWorkflow. workflowDeadline bounds
@@ -130,8 +128,6 @@ func NewActorWorkflow(
store actorWorkflowStore,
workerCache *workercache.Cache,
dialer *AteletDialer,
- workerPoolLister listersv1alpha1.WorkerPoolLister,
- sandboxConfigLister listersv1alpha1.SandboxConfigLister,
storageClassLister storagev1listers.StorageClassLister,
instruments *Instruments,
egressGatewayAddress string,
@@ -144,8 +140,6 @@ func NewActorWorkflow(
workerCache: workerCache,
scheduler: scheduling.New(workerCache),
dialer: dialer,
- workerPoolLister: workerPoolLister,
- sandboxConfigLister: sandboxConfigLister,
storageClassLister: storageClassLister,
instruments: instruments,
egressGatewayAddress: egressGatewayAddress,
@@ -155,6 +149,12 @@ func NewActorWorkflow(
}
}
+// snapshotLocation returns the base object-storage URI actorTemplate's
+// snapshots are stored under; see resolveSnapshotLocation.
+func (w *ActorWorkflow) snapshotLocation(actorTemplate *ateapipb.ActorTemplate) (string, error) {
+ return resolveSnapshotLocation(w.clusterSnapshotLocation, actorTemplate)
+}
+
// cleanupSnapshot deletes every object under prefix through the control
// object-store plugin.
func (w *ActorWorkflow) cleanupSnapshot(ctx context.Context, prefix resources.StoragePrefix) error {
diff --git a/cmd/ateapi/internal/controlapi/workflow_delete.go b/cmd/ateapi/internal/controlapi/workflow_delete.go
index de65c19a54..8317791a82 100644
--- a/cmd/ateapi/internal/controlapi/workflow_delete.go
+++ b/cmd/ateapi/internal/controlapi/workflow_delete.go
@@ -32,7 +32,7 @@ import (
)
// DeleteActor executes the workflow to delete an actor. Idempotent.
-func (w *ActorWorkflow) DeleteActor(ctx context.Context, actorRef resources.ActorRef, anyState bool, precondition store.DeletePreconditions) (*ateapipb.Actor, error) {
+func (w *ActorWorkflow) DeleteActor(ctx context.Context, actorRef resources.ActorRef, precondition store.DeletePreconditions) (*ateapipb.Actor, error) {
ctx = preview.ContextWithOptIn(ctx)
ctx, lease, err := w.acquireActorLease(ctx, actorRef)
if err != nil {
@@ -52,7 +52,7 @@ func (w *ActorWorkflow) DeleteActor(ctx context.Context, actorRef resources.Acto
return nil, apierror.Aborted("concurrent update conflict, please retry")
}
- if actor, err = w.ensureMarkedDeleting(ctx, actorRef, actor, anyState); err != nil {
+ if actor, err = w.ensureMarkedDeleting(ctx, actorRef, actor); err != nil {
return nil, err
}
@@ -181,7 +181,7 @@ func (w *ActorWorkflow) ensureAteletTerminated(ctx context.Context, actorRef res
// delete/suspend path.
workloadSpec = &ateletpb.WorkloadSpec{}
if preview.IsEnabled(preview.GateExternalVolumes) {
- for _, vol := range actor.GetStatus().GetExternalVolumes() {
+ for _, vol := range actor.GetStatus().GetExternalVolumes().GetVolumes() {
// StorageVolumeId is only populated once the volume is provisioned.
// Skip volumes that were never created (e.g. failed during PENDING state).
if vol.GetStorageVolumeId() != "" {
@@ -321,27 +321,14 @@ func (w *ActorWorkflow) ensureWorkerReleased(ctx context.Context, actorRef resou
// ensureMarkedDeleting transitions the actor to DELETING and persists the
// change, returning the stored copy. Skips when a previous attempt already
// marked the actor.
-func (w *ActorWorkflow) ensureMarkedDeleting(ctx context.Context, actorRef resources.ActorRef, actor *ateapipb.Actor, anyState bool) (updated *ateapipb.Actor, err error) {
+func (w *ActorWorkflow) ensureMarkedDeleting(ctx context.Context, actorRef resources.ActorRef, actor *ateapipb.Actor) (updated *ateapipb.Actor, err error) {
ctx, done := stepSpan(ctx, "MarkDeleting")
defer func() { err = done(err) }()
- st := actor.GetStatus().GetState()
- if st == ateapipb.ActorState_ACTOR_STATE_DELETING {
+ if actor.GetStatus().GetState() == ateapipb.ActorState_ACTOR_STATE_DELETING {
markSkipped(ctx, "actor already DELETING")
return actor, nil
}
- shouldDelete := false
- switch st {
- case ateapipb.ActorState_ACTOR_STATE_SUSPENDED,
- ateapipb.ActorState_ACTOR_STATE_CRASHED:
- shouldDelete = true
- default:
- // This allows deletion for any state
- shouldDelete = anyState
- }
- if !shouldDelete {
- return nil, apierror.FailedPrecondition("Actor %s is not in a deletable state (state: %v)", actorRef, st)
- }
storedActor, err := w.store.UpdateActor(ctx, actorRef, store.PreconditionFrom(actor), func(toUpdate *ateapipb.Actor) error {
toUpdate.Status.State = ateapipb.ActorState_ACTOR_STATE_DELETING
@@ -373,7 +360,7 @@ func (w *ActorWorkflow) ensureVolumesDeleted(ctx context.Context, actor *ateapip
// disabling the gate while in use would be an orphaned volume
// attachment, we are NOT checking the ExternalVolumes gate in the
// delete/suspend path.
- if err := deleteActorVolumes(ctx, w.pluginRegistry, actor.GetMetadata().GetUid(), actor.GetStatus().GetExternalVolumes()); err != nil {
+ if err := deleteActorVolumes(ctx, w.pluginRegistry, actor.GetMetadata().GetUid(), actor.GetStatus().GetExternalVolumes().GetVolumes()); err != nil {
return apierror.Internal("while deleting actor volumes: %v", err)
}
return nil
diff --git a/cmd/ateapi/internal/controlapi/workflow_delete_test.go b/cmd/ateapi/internal/controlapi/workflow_delete_test.go
index dedabf7635..863049b616 100644
--- a/cmd/ateapi/internal/controlapi/workflow_delete_test.go
+++ b/cmd/ateapi/internal/controlapi/workflow_delete_test.go
@@ -29,74 +29,39 @@ import (
"go.ate.dev/substrate/internal/resources"
"go.ate.dev/substrate/pkg/proto/ateapipb"
"google.golang.org/grpc/codes"
- "google.golang.org/grpc/status"
)
func TestDeleteActorWorkflow_ExecutionPaths(t *testing.T) {
tests := []struct {
name string
seedState ateapipb.ActorState
- anyState bool
missingTmpl bool
- wantErr bool
- wantCode codes.Code
}{
{
name: "delete suspended actor succeeds",
seedState: ateapipb.ActorState_ACTOR_STATE_SUSPENDED,
- anyState: false,
- wantErr: false,
},
{
name: "delete crashed actor succeeds",
seedState: ateapipb.ActorState_ACTOR_STATE_CRASHED,
- anyState: false,
- wantErr: false,
},
{
name: "delete deleting actor succeeds",
seedState: ateapipb.ActorState_ACTOR_STATE_DELETING,
- anyState: false,
- wantErr: false,
},
{
- name: "delete running actor rejected when not any_state",
+ name: "delete running actor succeeds",
seedState: ateapipb.ActorState_ACTOR_STATE_RUNNING,
- anyState: false,
- wantErr: true,
- wantCode: codes.FailedPrecondition,
- },
- {
- name: "any_state delete suspended actor succeeds",
- seedState: ateapipb.ActorState_ACTOR_STATE_SUSPENDED,
- anyState: true,
- wantErr: false,
- },
- {
- name: "any_state delete running actor succeeds",
- seedState: ateapipb.ActorState_ACTOR_STATE_RUNNING,
- anyState: true,
- wantErr: false,
- },
- {
- name: "any_state delete crashed actor succeeds",
- seedState: ateapipb.ActorState_ACTOR_STATE_CRASHED,
- anyState: true,
- wantErr: false,
},
{
name: "delete suspended actor with missing template succeeds",
seedState: ateapipb.ActorState_ACTOR_STATE_SUSPENDED,
- anyState: false,
missingTmpl: true,
- wantErr: false,
},
{
- name: "any_state delete suspended actor with missing template succeeds",
- seedState: ateapipb.ActorState_ACTOR_STATE_SUSPENDED,
- anyState: true,
+ name: "delete running actor with missing template succeeds",
+ seedState: ateapipb.ActorState_ACTOR_STATE_RUNNING,
missingTmpl: true,
- wantErr: false,
},
}
@@ -114,79 +79,43 @@ func TestDeleteActorWorkflow_ExecutionPaths(t *testing.T) {
}
seedWorkflowActor(t, ctx, st, actorRef, "ns", tmplName, tc.seedState)
- deleted, err := w.DeleteActor(ctx, actorRef, tc.anyState, store.DeletePreconditions{})
- if tc.wantErr {
- if got := apierror.Code(err); got != tc.wantCode {
- t.Fatalf("apierror.Code(err) = %v, want %v (err: %v)", got, tc.wantCode, err)
- }
- } else {
- if err != nil {
- t.Fatalf("DeleteActor failed: %v", err)
- }
- if deleted == nil {
- t.Fatalf("expected non-nil deleted actor")
- }
- if _, err := st.GetActor(ctx, actorRef); err == nil {
- t.Errorf("expected actor to be deleted from store, but it still exists")
- }
+ deleted, err := w.DeleteActor(ctx, actorRef, store.DeletePreconditions{})
+ if err != nil {
+ t.Fatalf("DeleteActor failed: %v", err)
+ }
+ if deleted == nil {
+ t.Fatalf("expected non-nil deleted actor")
+ }
+ if _, err := st.GetActor(ctx, actorRef); err == nil {
+ t.Errorf("expected actor to be deleted from store, but it still exists")
}
})
}
}
-func TestEnsureMarkedDeleting_StateMatrix(t *testing.T) {
- tests := []struct {
- name string
- anyState bool
- allowed map[ateapipb.ActorState]bool
- }{
- {
- name: "standard delete",
- anyState: false,
- allowed: map[ateapipb.ActorState]bool{
- ateapipb.ActorState_ACTOR_STATE_SUSPENDED: true,
- ateapipb.ActorState_ACTOR_STATE_CRASHED: true,
- ateapipb.ActorState_ACTOR_STATE_DELETING: true, // skipped
- },
- },
- {
- name: "any_state delete",
- anyState: true,
- allowed: map[ateapipb.ActorState]bool{
- ateapipb.ActorState_ACTOR_STATE_UNSPECIFIED: true,
- ateapipb.ActorState_ACTOR_STATE_RUNNING: true,
- ateapipb.ActorState_ACTOR_STATE_RESUMING: true,
- ateapipb.ActorState_ACTOR_STATE_SUSPENDING: true,
- ateapipb.ActorState_ACTOR_STATE_SUSPENDED: true,
- ateapipb.ActorState_ACTOR_STATE_CRASHED: true,
- ateapipb.ActorState_ACTOR_STATE_REVERTING: true,
- ateapipb.ActorState_ACTOR_STATE_DELETING: true, // skipped
- },
- },
- }
+// TestEnsureMarkedDeleting_AllStates checks that every actor state can be
+// marked DELETING.
+func TestEnsureMarkedDeleting_AllStates(t *testing.T) {
+ for _, seedState := range allActorStates {
+ t.Run(seedState.String(), func(t *testing.T) {
+ ctx := context.Background()
+ st, cleanup := storetest.SetupTestStore(t)
+ defer cleanup()
+ w := newTestActorWorkflow(t, st, "ns", "tmpl1")
- for _, tc := range tests {
- t.Run(tc.name, func(t *testing.T) {
- for _, seedState := range allActorStates {
- ctx := context.Background()
- st, cleanup := storetest.SetupTestStore(t)
- w := newTestActorWorkflow(t, st, "ns", "tmpl1")
-
- actorRef := resources.ActorRef{Atespace: "team-a", Name: "id1"}
- seedWorkflowActor(t, ctx, st, actorRef, "ns", "tmpl1", seedState)
- actor, err := st.GetActor(ctx, actorRef)
- if err != nil {
- t.Fatalf("state %v: get seeded actor: %v", seedState, err)
- }
+ actorRef := resources.ActorRef{Atespace: "team-a", Name: "id1"}
+ seedWorkflowActor(t, ctx, st, actorRef, "ns", "tmpl1", seedState)
+ actor, err := st.GetActor(ctx, actorRef)
+ if err != nil {
+ t.Fatalf("get seeded actor: %v", err)
+ }
- updated, err := w.ensureMarkedDeleting(ctx, actorRef, actor, tc.anyState)
- assertPrerequisiteResult(t, seedState, err, tc.allowed[seedState])
- if err == nil && seedState != ateapipb.ActorState_ACTOR_STATE_DELETING {
- if updated.GetStatus().GetState() != ateapipb.ActorState_ACTOR_STATE_DELETING {
- t.Errorf("state %v: ensureMarkedDeleting returned actor in %v, want DELETING", seedState, updated.GetStatus().GetState())
- }
- }
- cleanup()
+ updated, err := w.ensureMarkedDeleting(ctx, actorRef, actor)
+ if err != nil {
+ t.Fatalf("ensureMarkedDeleting failed: %v", err)
+ }
+ if got := updated.GetStatus().GetState(); got != ateapipb.ActorState_ACTOR_STATE_DELETING {
+ t.Errorf("ensureMarkedDeleting returned actor in %v, want DELETING", got)
}
})
}
@@ -334,7 +263,7 @@ func TestDeleteActor_CollectsInFlightSnapshotWithoutTemplate(t *testing.T) {
ctx := context.Background()
persistence := newTestPersistence(t)
objects := objectstoretest.New()
- w := NewActorWorkflow(persistence, nil, nil, nil, nil, nil, nil, "", nil, time.Minute, objectstoreplugintest.ControlClient(objects))
+ w := NewActorWorkflow(persistence, nil, nil, nil, nil, "", nil, time.Minute, objectstoreplugintest.ControlClient(objects))
actorRef := resources.ActorRef{Atespace: "team-a", Name: "actor-1"}
actor := storetest.MustCreateActor(t, ctx, persistence, &ateapipb.Actor{
@@ -356,7 +285,7 @@ func TestDeleteActor_CollectsInFlightSnapshotWithoutTemplate(t *testing.T) {
}
})
- if _, err := w.DeleteActor(ctx, actorRef, true, store.DeletePreconditions{}); err != nil {
+ if _, err := w.DeleteActor(ctx, actorRef, store.DeletePreconditions{}); err != nil {
t.Fatalf("DeleteActor: %v", err)
}
if left := objects.Prefix(t, inFlight.OwnerPrefix()); len(left) != 0 {
@@ -488,7 +417,7 @@ func TestDeleteActor_CollectsSnapshotsAfterWorkerDelete(t *testing.T) {
t.Errorf("DeleteWorker: %v", err)
return err
}
- return status.Error(codes.Unavailable, "worker pod is gone")
+ return apierror.Unavailable("worker pod is gone")
}
if _, err := actorWorkflow.SuspendActor(ctx, actorRef, suspendSync); apierror.Code(err) != codes.Unavailable {
t.Fatalf("SuspendActor err = %v, want Unavailable from the failed checkpoint", err)
@@ -503,7 +432,7 @@ func TestDeleteActor_CollectsSnapshotsAfterWorkerDelete(t *testing.T) {
if got := stored.GetStatus().GetState(); got != ateapipb.ActorState_ACTOR_STATE_CRASHED {
t.Fatalf("state = %v, want CRASHED", got)
}
- if _, err := actorWorkflow.DeleteActor(ctx, actorRef, true, store.DeletePreconditions{}); err != nil {
+ if _, err := actorWorkflow.DeleteActor(ctx, actorRef, store.DeletePreconditions{}); err != nil {
t.Fatalf("DeleteActor: %v", err)
}
if left := objects.Prefix(t, fresh.OwnerPrefix()); len(left) != 0 {
diff --git a/cmd/ateapi/internal/controlapi/workflow_reconcile_assignments_test.go b/cmd/ateapi/internal/controlapi/workflow_reconcile_assignments_test.go
index 3fcf4e5f7f..f30691fc57 100644
--- a/cmd/ateapi/internal/controlapi/workflow_reconcile_assignments_test.go
+++ b/cmd/ateapi/internal/controlapi/workflow_reconcile_assignments_test.go
@@ -575,7 +575,11 @@ func TestAssignWorkerAttempt_StampsWorkerEpoch(t *testing.T) {
WorkerPodUid: name,
SandboxClass: "gvisor",
Epoch: 1,
- Status: &ateapipb.WorkerStatus{State: ateapipb.WorkerState_WORKER_STATE_ACTIVE, Capacity: &ateapipb.WorkerResources{Actors: 1}},
+ Status: &ateapipb.WorkerStatus{
+ State: ateapipb.WorkerState_WORKER_STATE_ACTIVE,
+ Capacity: &ateapipb.WorkerResources{Actors: 1},
+ DefaultRuntime: testSandboxRuntime(),
+ },
})
if err != nil {
t.Fatalf("CreateWorker: %v", err)
diff --git a/cmd/ateapi/internal/controlapi/workflow_resume.go b/cmd/ateapi/internal/controlapi/workflow_resume.go
index ad8e18048b..bd15d0b7ef 100644
--- a/cmd/ateapi/internal/controlapi/workflow_resume.go
+++ b/cmd/ateapi/internal/controlapi/workflow_resume.go
@@ -52,8 +52,8 @@ type resumeSnapshotSource struct {
// TemplateReplaced is true when the external snapshot's recorded template
// UID differs from the actor's current template.
TemplateReplaced bool
- // RequiredRuntime is the runtime resolved from the snapshot's guest state
- // (via requiredSnapshotRuntime), or nil when guest state is not restored.
+ // RequiredRuntime is the runtime resolved from the snapshot's memory state
+ // (via requiredSnapshotRuntime), or nil when memory state is not restored.
RequiredRuntime *ateapipb.SandboxRuntime
}
@@ -179,6 +179,13 @@ func (w *ActorWorkflow) loadActorForResume(ctx context.Context, actorRef resourc
capturedUnder := snap.GetActorTemplateUid()
src.TemplateReplaced = capturedUnder != "" && capturedUnder != actorTemplate.GetMetadata().GetUid()
}
+ // A local restore also names the local snapshot's upload, so a URI that
+ // does not parse fails here too, before a worker is claimed.
+ if local := findLatestLocalSnapshot(actor.GetStatus()); local != nil {
+ if _, err = resources.ParseSnapshotURI(local.GetDurableSnapshot().GetObject().GetSnapshotUri()); err != nil {
+ return nil, nil, src, apierror.DataLoss("Actor %s local snapshot: %v", actorRef, err)
+ }
+ }
return actor, actorTemplate, src, nil
}
@@ -191,7 +198,7 @@ func (w *ActorWorkflow) ensureVolumesCreated(ctx context.Context, actorRef resou
ctx, done := stepSpan(ctx, "CreateVolumes")
defer func() { err = done(err) }()
- pending := slices.ContainsFunc(actor.GetStatus().GetExternalVolumes(), func(vol *ateapipb.ExternalVolume) bool {
+ pending := slices.ContainsFunc(actor.GetStatus().GetExternalVolumes().GetVolumes(), func(vol *ateapipb.ExternalVolumeStatus) bool {
return vol.GetStorageVolumeId() == ""
})
if !pending {
@@ -202,12 +209,12 @@ func (w *ActorWorkflow) ensureVolumesCreated(ctx context.Context, actorRef resou
return nil, apierror.FailedPrecondition("cannot create volumes for Actor %s in state %s", actorRef, st)
}
- volumes, createErr := createActorVolumes(ctx, w.pluginRegistry, w.storageClassLister, actor.GetMetadata().GetUid(), actorTemplate, actor.GetStatus().GetExternalVolumes())
+ volumes, createErr := createActorVolumes(ctx, w.pluginRegistry, w.storageClassLister, actor.GetMetadata().GetUid(), actorTemplate, actor.GetStatus().GetExternalVolumes().GetVolumes())
// createActorVolumes reports the state it got to even when it fails, so both
// paths persist the same field.
updatePrecondition := store.PreconditionFrom(actor)
persistVolumes := func(toUpdate *ateapipb.Actor) error {
- toUpdate.Status.ExternalVolumes = volumes
+ toUpdate.Status.ExternalVolumes = externalVolumesStatus(volumes)
return nil
}
if createErr != nil {
@@ -595,16 +602,6 @@ func workerAssignmentFrom(w *ateapipb.Worker, snapRuntime hardware.Requirement)
// The runtime the actor is started or restored under. Snapshots taken
// during this assignment are stamped with it, so a worker that later
// changes its default does not relabel state captured on the old one.
- //
- // TODO: Workers report restorable_runtimes, so a match on one records
- // that older runtime here and ensureAteletRestored sends it on Run and
- // Restore, but atelet still restores with the assets in the request
- // (the worker pool's SandboxConfig default version): the restore runs on
- // the newer version and fails, or its next checkpoint is stamped with a
- // runtime that did not produce it. A follow-up resolves the matching
- // version of the worker pool's SandboxConfig from sandbox_runtime in
- // atelet and removes sandbox_assets; until it lands, keep a config's
- // default version the only enabled one.
SandboxRuntime: proto.CloneOf(snapRuntime.RuntimeFor(w.GetStatus())),
}
}
@@ -690,21 +687,18 @@ func snapshotSandboxClass(actor *ateapipb.Actor) string {
}
// requiredSnapshotRuntime returns the SandboxRuntime stamped on the snapshot
-// that ResumeActor will restore guest state from, or nil when the resume does
-// not restore guest state (cold boot, VOLUMES fidelity, or replaced template).
-//
-// TODO: Once gVisor can restore ROOTFS snapshots, return nil for ROOTFS as
-// well: a rootfs upper is files tied to the image, not to the runtime, so only
-// MEMORY needs a runtime match. Until then ROOTFS is pinned like MEMORY.
+// that ResumeActor will restore memory state from, or nil when the resume does
+// not restore memory state (cold boot, ROOTFS or VOLUMES fidelity, or replaced
+// template).
func requiredSnapshotRuntime(actor *ateapipb.Actor, tmpl *ateapipb.ActorTemplate) *ateapipb.SandboxRuntime {
if localSnap := findLatestLocalSnapshot(actor.GetStatus()); localSnap != nil {
- if localSnap.GetFidelity() == ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES {
+ if localSnap.GetFidelity() != ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY {
return nil
}
return localSnap.GetSandboxRuntime()
}
if snap := findLatestDurableSnapshot(actor.GetStatus(), ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED); snap != nil {
- if snap.GetFidelity() == ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES {
+ if snap.GetFidelity() != ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY {
return nil
}
capturedUnder := snap.GetActorTemplateUid()
@@ -717,7 +711,7 @@ func requiredSnapshotRuntime(actor *ateapipb.Actor, tmpl *ateapipb.ActorTemplate
}
// canRelaxRuntime reports whether a resume may drop its RequiredRuntime
-// constraint and fall back to a volumes-only restore when no
+// constraint and fall back to restoring without memory state when no
// runtime-compatible worker has capacity. A local pause checkpoint is not
// relaxed: the resume stays pinned to its node, the local restore has no
// fidelity downgrade, and nothing invalidates the checkpoint files if the actor
@@ -750,7 +744,7 @@ func (w *ActorWorkflow) ensureVolumesAttached(ctx context.Context, actor *ateapi
ctx, done := stepSpan(ctx, "AttachVolumes")
defer func() { err = done(err) }()
- if len(actor.GetStatus().GetExternalVolumes()) > 0 && !preview.IsEnabled(preview.GateExternalVolumes) {
+ if len(actor.GetStatus().GetExternalVolumes().GetVolumes()) > 0 && !preview.IsEnabled(preview.GateExternalVolumes) {
return nil, fmt.Errorf("external volumes are not supported unless the ExternalVolumes preview gate is enabled")
}
@@ -761,7 +755,7 @@ func (w *ActorWorkflow) ensureVolumesAttached(ctx context.Context, actor *ateapi
ref := &ateapipb.ObjectRef{Atespace: actor.GetMetadata().GetAtespace(), Name: actor.GetMetadata().GetName()}
volumePublishContexts := make(map[string]map[string]string)
- for _, vol := range getMountedActorVolumes(ctx, ref, actor.GetStatus().GetExternalVolumes(), actorTemplate) {
+ for _, vol := range getMountedActorVolumes(ctx, ref, actor.GetStatus().GetExternalVolumes().GetVolumes(), actorTemplate) {
slog.InfoContext(ctx, "Attaching volume to node", slog.String("volume_id", vol.GetStorageVolumeId()), slog.String("node", node))
plugin, err := w.pluginRegistry.GetPlugin(ctx, vol.GetVolumeType())
if err != nil {
@@ -809,13 +803,6 @@ func (w *ActorWorkflow) ensureAteletRestored(ctx context.Context, actorRef resou
return tele, err
}
- // The sandbox binaries and pause image come from the assigned worker's
- // pool SandboxConfig on every path, restores included.
- sandboxAssets, err := resolveSandboxAssets(w.workerPoolLister, w.sandboxConfigLister, assignment)
- if err != nil {
- return tele, fmt.Errorf("while resolving sandbox assets: %w", err)
- }
-
// The assignment is what the next checkpoint is stamped from, so it is also
// what decides how this restore runs.
sandboxRuntime := toAteletSandboxRuntime(assignment.GetSandboxRuntime())
@@ -836,19 +823,19 @@ func (w *ActorWorkflow) ensureAteletRestored(ctx context.Context, actorRef resou
SnapshotUuid: localSnap.GetUuid(),
Generation: localSnap.GetGeneration(),
Fidelity: fidelity,
+ // Named even while the upload is PENDING: atelet may have
+ // uploaded the snapshot and detached the actor before its
+ // report of the upload lands. atelet reads it only when it
+ // holds no usable local copy, and the manifest, uploaded
+ // last, tells a complete upload from a partial one.
+ Object: &ateletpb.ObjectStorage{SnapshotUri: localSnap.GetDurableSnapshot().GetObject().GetSnapshotUri()},
},
- SandboxAssets: sandboxAssets,
SandboxRuntime: sandboxRuntime,
ActorUid: actor.GetMetadata().Uid,
EgressGateway: egressGateway,
CpuMilli: cpuMilli,
MemoryBytes: memBytes,
}
- // atelet uploads a pause's snapshot in the background. Once that upload
- // is recorded, atelet can download it if its local copy is gone.
- if durable := localSnap.GetDurableSnapshot(); durable.GetStatus() == ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED {
- req.Snapshot.Object = &ateletpb.ObjectStorage{SnapshotUri: durable.GetObject().GetSnapshotUri()}
- }
tele.WireFidelity = ateattr.SnapshotFidelityValue(fidelity)
if _, err = client.Restore(ctx, req); err != nil {
@@ -859,15 +846,15 @@ func (w *ActorWorkflow) ensureAteletRestored(ctx context.Context, actorRef resou
slog.InfoContext(ctx, "Actor has durable snapshot; Restoring from snapshot")
tele.SnapshotKind = ateattr.SnapshotKindLatest
scope := fidelityToAtelet(src.Fidelity)
- if src.TemplateReplaced || runtimeIncompatible {
- if runtimeIncompatible {
- logRuntimeFallback(ctx, actorRef, assignment, src.RequiredRuntime)
- }
- // TODO: Once gVisor can restore ROOTFS snapshots, drop an incompatible
- // runtime to ROOTFS, not VOLUMES: only the memory layer is invalid. A
- // replaced template still drops to VOLUMES, as the rootfs upper is
- // only valid over the image it was written on.
+ // Exclusive by construction: requiredSnapshotRuntime returns nil when
+ // the template changed, so runtimeIncompatible is false when
+ // src.TemplateReplaced is true.
+ switch {
+ case src.TemplateReplaced:
scope = ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES
+ case runtimeIncompatible:
+ logRuntimeFallback(ctx, actorRef, assignment, src.RequiredRuntime)
+ scope = ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS
}
tele.WireFidelity = ateattr.SnapshotFidelityValue(scope)
req := &ateletpb.RestoreRequest{
@@ -885,7 +872,6 @@ func (w *ActorWorkflow) ensureAteletRestored(ctx context.Context, actorRef resou
SnapshotUri: src.SnapshotURI.String(),
},
},
- SandboxAssets: sandboxAssets,
SandboxRuntime: sandboxRuntime,
ActorUid: actor.GetMetadata().Uid,
EgressGateway: egressGateway,
@@ -906,7 +892,6 @@ func (w *ActorWorkflow) ensureAteletRestored(ctx context.Context, actorRef resou
ActorName: actor.GetMetadata().GetName(),
ActorTemplateAtespace: actor.GetActorTemplate().GetAtespace(),
ActorTemplateName: actor.GetActorTemplate().GetName(),
- SandboxAssets: sandboxAssets,
SandboxRuntime: sandboxRuntime,
Spec: workloadSpec,
ActorUid: actor.GetMetadata().Uid,
@@ -921,11 +906,11 @@ func (w *ActorWorkflow) ensureAteletRestored(ctx context.Context, actorRef resou
}
}
-// logRuntimeFallback records a resume that restores the durable volumes alone
-// because the assigned worker does not run the runtime the snapshot's guest
+// logRuntimeFallback records a resume that restores without memory state
+// because the assigned worker does not run the runtime the snapshot's memory
// state was captured under.
func logRuntimeFallback(ctx context.Context, actorRef resources.ActorRef, assignment *ateapipb.WorkerAssignment, required *ateapipb.SandboxRuntime) {
- slog.WarnContext(ctx, "Assigned worker cannot restore the snapshot's runtime; restoring volumes only",
+ slog.WarnContext(ctx, "Assigned worker cannot restore the snapshot's runtime; restoring without memory state",
slog.String("actor", actorRef.String()),
slog.String("worker", assignment.GetWorkerPod()),
slog.Any("required_runtime", required),
diff --git a/cmd/ateapi/internal/controlapi/workflow_resume_test.go b/cmd/ateapi/internal/controlapi/workflow_resume_test.go
index fde73568e0..6d12b6b0e4 100644
--- a/cmd/ateapi/internal/controlapi/workflow_resume_test.go
+++ b/cmd/ateapi/internal/controlapi/workflow_resume_test.go
@@ -30,13 +30,13 @@ import (
"go.ate.dev/substrate/cmd/ateapi/internal/workercache"
"go.ate.dev/substrate/internal/apierror"
"go.ate.dev/substrate/internal/ateattr"
+ "go.ate.dev/substrate/internal/ateinterceptors"
"go.ate.dev/substrate/internal/hardware"
"go.ate.dev/substrate/internal/installdefaults"
"go.ate.dev/substrate/internal/preview"
"go.ate.dev/substrate/internal/proto/ateletpb"
"go.ate.dev/substrate/internal/resources"
"go.ate.dev/substrate/internal/volume"
- atev1alpha1 "go.ate.dev/substrate/pkg/api/v1alpha1"
"go.ate.dev/substrate/pkg/proto/ateapipb"
sdkmetric "go.opentelemetry.io/otel/sdk/metric"
"go.opentelemetry.io/otel/sdk/metric/metricdata"
@@ -206,7 +206,11 @@ func TestAssignWorkerAttempt_SkipsWorkerAssignedInOtherAtespace(t *testing.T) {
WorkerPod: "pod-1",
WorkerPodUid: testWorkerUID("pod-1"),
SandboxClass: "gvisor",
- Status: &ateapipb.WorkerStatus{State: ateapipb.WorkerState_WORKER_STATE_ACTIVE, Capacity: &ateapipb.WorkerResources{Actors: 1}},
+ Status: &ateapipb.WorkerStatus{
+ State: ateapipb.WorkerState_WORKER_STATE_ACTIVE,
+ Capacity: &ateapipb.WorkerResources{Actors: 1},
+ DefaultRuntime: testSandboxRuntime(),
+ },
}
if _, err := persistence.CreateWorker(ctx, worker); err != nil {
t.Fatalf("CreateWorker: %v", err)
@@ -264,7 +268,11 @@ func TestAssignWorkerAttempt_ReleasesIneligibleStaleWorker(t *testing.T) {
WorkerPod: "stale-pod",
WorkerPodUid: testWorkerUID("stale-pod"),
SandboxClass: "gvisor",
- Status: &ateapipb.WorkerStatus{State: ateapipb.WorkerState_WORKER_STATE_ACTIVE, Capacity: &ateapipb.WorkerResources{Actors: 1}},
+ Status: &ateapipb.WorkerStatus{
+ State: ateapipb.WorkerState_WORKER_STATE_ACTIVE,
+ Capacity: &ateapipb.WorkerResources{Actors: 1},
+ DefaultRuntime: testSandboxRuntime(),
+ },
}
free := &ateapipb.Worker{
Metadata: &ateapipb.ResourceMetadata{Name: testWorkerUID("free-pod")},
@@ -274,7 +282,11 @@ func TestAssignWorkerAttempt_ReleasesIneligibleStaleWorker(t *testing.T) {
WorkerPodUid: testWorkerUID("free-pod"),
SandboxClass: "gvisor",
Labels: map[string]string{"pool": "b"},
- Status: &ateapipb.WorkerStatus{State: ateapipb.WorkerState_WORKER_STATE_ACTIVE, Capacity: &ateapipb.WorkerResources{Actors: 1}},
+ Status: &ateapipb.WorkerStatus{
+ State: ateapipb.WorkerState_WORKER_STATE_ACTIVE,
+ Capacity: &ateapipb.WorkerResources{Actors: 1},
+ DefaultRuntime: testSandboxRuntime(),
+ },
}
for _, w := range []*ateapipb.Worker{stale, free} {
if _, err := persistence.CreateWorker(ctx, w); err != nil {
@@ -322,6 +334,7 @@ func TestAssignWorkerAttempt_RetryAfterConflictPicksFreshWorker(t *testing.T) {
ctx := context.Background()
persistence := newTestPersistence(t)
+ wantRuntime := testSandboxRuntime()
contested := &ateapipb.Worker{
Metadata: &ateapipb.ResourceMetadata{Name: testWorkerUID("contested-pod")},
WorkerNamespace: "worker-ns",
@@ -329,9 +342,12 @@ func TestAssignWorkerAttempt_RetryAfterConflictPicksFreshWorker(t *testing.T) {
WorkerPod: "contested-pod",
WorkerPodUid: testWorkerUID("contested-pod"),
SandboxClass: "gvisor",
- Status: &ateapipb.WorkerStatus{State: ateapipb.WorkerState_WORKER_STATE_ACTIVE, Capacity: &ateapipb.WorkerResources{Actors: 1}},
+ Status: &ateapipb.WorkerStatus{
+ State: ateapipb.WorkerState_WORKER_STATE_ACTIVE,
+ Capacity: &ateapipb.WorkerResources{Actors: 1},
+ DefaultRuntime: wantRuntime,
+ },
}
- wantRuntime := testSandboxRuntime()
fallback := &ateapipb.Worker{
Metadata: &ateapipb.ResourceMetadata{Name: testWorkerUID("fallback-pod")},
WorkerNamespace: "worker-ns",
@@ -434,7 +450,11 @@ func seedAssignFixture(t *testing.T, ctx context.Context, persistence store.Inte
WorkerPod: "pod-1",
WorkerPodUid: testWorkerUID("pod-1"),
SandboxClass: "gvisor",
- Status: &ateapipb.WorkerStatus{State: ateapipb.WorkerState_WORKER_STATE_ACTIVE, Capacity: &ateapipb.WorkerResources{Actors: 1}},
+ Status: &ateapipb.WorkerStatus{
+ State: ateapipb.WorkerState_WORKER_STATE_ACTIVE,
+ Capacity: &ateapipb.WorkerResources{Actors: 1},
+ DefaultRuntime: testSandboxRuntime(),
+ },
}); err != nil {
t.Fatalf("CreateWorker: %v", err)
}
@@ -739,7 +759,11 @@ func TestValidateAssignedWorker(t *testing.T) {
Actor: &ateapipb.ObjectRef{Atespace: "team-a", Name: "shared"},
ActorUid: "stale-incarnation-uid",
}
- activeStatus := &ateapipb.WorkerStatus{State: ateapipb.WorkerState_WORKER_STATE_ACTIVE, Capacity: &ateapipb.WorkerResources{Actors: 1}}
+ activeStatus := &ateapipb.WorkerStatus{
+ State: ateapipb.WorkerState_WORKER_STATE_ACTIVE,
+ Capacity: &ateapipb.WorkerResources{Actors: 1},
+ DefaultRuntime: testSandboxRuntime(),
+ }
drainingStatus := &ateapipb.WorkerStatus{State: ateapipb.WorkerState_WORKER_STATE_DRAINING, Capacity: &ateapipb.WorkerResources{Actors: 1}}
unavailableStatus := &ateapipb.WorkerStatus{State: ateapipb.WorkerState_WORKER_STATE_UNAVAILABLE, Capacity: &ateapipb.WorkerResources{Actors: 1}}
@@ -1070,6 +1094,8 @@ type capturingAtelet struct {
checkpointState *ateletpb.ActorSnapshotState
// promoteState is the snapshot state PromoteSnapshot returns.
promoteState *ateletpb.ActorSnapshotState
+ // promoteErr, when set, fails PromoteSnapshot after it is recorded.
+ promoteErr error
}
func (f *capturingAtelet) Restore(ctx context.Context, req *ateletpb.RestoreRequest) (*ateletpb.RestoreResponse, error) {
@@ -1107,6 +1133,9 @@ func (f *capturingAtelet) PromoteSnapshot(ctx context.Context, req *ateletpb.Pro
defer f.mu.Unlock()
f.promote = proto.Clone(req).(*ateletpb.PromoteSnapshotRequest)
f.promotes++
+ if f.promoteErr != nil {
+ return nil, f.promoteErr
+ }
return &ateletpb.PromoteSnapshotResponse{State: proto.CloneOf(f.promoteState)}, nil
}
@@ -1167,12 +1196,14 @@ func wireTestAssignment() *ateapipb.WorkerAssignment {
// newWireCaptureWorkflow builds an ActorWorkflow whose atelet dialer resolves
// to an in-process capturing fake. The dialer's conn cache is pre-warmed with
// a bufconn-backed connection for the atelet pod's UID and IP, so
-// DialForAteletOnNode returns it without dialing the pod IP.
+// DialForAteletOnNode returns it without dialing the pod IP. The fake's server
+// runs atelet's interceptor, so its errors reach the workflow with the code
+// atelet would send.
func newWireCaptureWorkflow(t *testing.T, persistence store.Interface) (*ActorWorkflow, *capturingAtelet) {
t.Helper()
fake := &capturingAtelet{}
- srv := grpc.NewServer()
+ srv := grpc.NewServer(grpc.UnaryInterceptor(ateinterceptors.InternalServerUnaryInterceptor))
ateletpb.RegisterAteletServer(srv, fake)
lis := bufconn.Listen(1 << 20)
go func() {
@@ -1201,22 +1232,7 @@ func newWireCaptureWorkflow(t *testing.T, persistence store.Interface) (*ActorWo
dialer := NewAteletDialer(newTestAteletIndexer(t, ateletPod), installdefaults.SystemNamespace, "", "")
dialer.ateletConns.Add("atelet-uid", &ateletConn{ip: "10.0.0.1", conn: conn})
- lister := sandboxConfigListerFor(t, []*atev1alpha1.SandboxConfig{{
- ObjectMeta: metav1.ObjectMeta{Name: "gvisor"},
- Spec: atev1alpha1.SandboxConfigSpec{
- SandboxClass: atev1alpha1.SandboxClassGvisor,
- DefaultVersion: "v1",
- Versions: []atev1alpha1.SandboxVersionConfig{{
- Name: "v1",
- PauseImage: "pause@sha256:abc",
- Assets: testAssets(),
- }},
- },
- }})
-
- pools := workerPoolListerFor(t, []*atev1alpha1.WorkerPool{resolvedWorkerPool("worker-ns", "pool", "gvisor")})
-
- return &ActorWorkflow{store: persistence, dialer: dialer, workerPoolLister: pools, sandboxConfigLister: lister, workflowDeadline: time.Minute}, fake
+ return &ActorWorkflow{store: persistence, dialer: dialer, workflowDeadline: time.Minute}, fake
}
// TestResumeActor_AteletWireRequest is the characteristic test for the
@@ -1257,6 +1273,13 @@ func TestResumeActor_AteletWireRequest(t *testing.T) {
}
return snap
}
+ // localSnapWithUpload is a local snapshot whose record names upload as
+ // its durable copy, or none when upload is nil.
+ localSnapWithUpload := func(name string, fidelity ateapipb.SnapshotFidelity, upload *ateapipb.SnapshotStorage) *ateapipb.Snapshot {
+ snap := localSnap(name, fidelity)
+ snap.DurableSnapshot = upload
+ return snap
+ }
// actorSeed is the actor status a row persists before resuming.
type actorSeed struct {
@@ -1274,10 +1297,6 @@ func TestResumeActor_AteletWireRequest(t *testing.T) {
sourceTag *ateapipb.ObjectRef
// assignedRuntime seeds WorkerAssignment.SandboxRuntime.
assignedRuntime *ateapipb.SandboxRuntime
- // poolConfigName, when set, is the SandboxConfig the assigned
- // worker's pool resolves instead of the "gvisor" config the
- // workflow's lister serves.
- poolConfigName string
}
// templateSeed is the ActorTemplate configuration a row persists.
type templateSeed struct {
@@ -1459,6 +1478,9 @@ func TestResumeActor_AteletWireRequest(t *testing.T) {
want: restoreWant{code: codes.DataLoss},
},
{
+ // A pause snapshot's upload is PENDING until atelet reports it,
+ // but a local restore names it anyway, so atelet can download the
+ // snapshot if it detached the actor before the report landed.
name: "13 Full pause snapshot restores locally as Full",
actor: actorSeed{
localSnapshot: localSnap(localSnapshotName, fullScope),
@@ -1466,6 +1488,7 @@ func TestResumeActor_AteletWireRequest(t *testing.T) {
},
want: restoreWant{
snapshotName: localSnapshotName,
+ snapshotURI: pauseUploadURI,
scope: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY,
},
},
@@ -1480,6 +1503,7 @@ func TestResumeActor_AteletWireRequest(t *testing.T) {
},
want: restoreWant{
snapshotName: localSnapshotName,
+ snapshotURI: pauseUploadURI,
scope: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY,
},
},
@@ -1490,6 +1514,7 @@ func TestResumeActor_AteletWireRequest(t *testing.T) {
},
want: restoreWant{
snapshotName: localSnapshotName,
+ snapshotURI: pauseUploadURI,
scope: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES,
},
},
@@ -1503,12 +1528,13 @@ func TestResumeActor_AteletWireRequest(t *testing.T) {
},
want: restoreWant{
snapshotName: localSnapshotName,
+ snapshotURI: pauseUploadURI,
scope: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES,
},
},
{
// The local snapshot takes precedence at restore, carrying its own
- // scope rather than the durable snapshot's.
+ // scope and upload rather than the durable snapshot's.
name: "17 local snapshot wins over a Full durable snapshot",
actor: actorSeed{
localSnapshot: localSnap(localSnapshotName, dataScope),
@@ -1516,6 +1542,7 @@ func TestResumeActor_AteletWireRequest(t *testing.T) {
},
want: restoreWant{
snapshotName: localSnapshotName,
+ snapshotURI: pauseUploadURI,
scope: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES,
},
},
@@ -1532,24 +1559,12 @@ func TestResumeActor_AteletWireRequest(t *testing.T) {
},
want: restoreWant{
snapshotName: localSnapshotName,
+ snapshotURI: pauseUploadURI,
scope: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY,
},
},
{
- // Restores take their sandbox from the assigned worker's pool,
- // like cold boots, so an unresolvable SandboxConfig stops them
- // before atelet.
- name: "19 durable snapshot restore with a missing pool SandboxConfig is rejected",
- actor: actorSeed{externalSnapshot: extSnap(actorURI, fullScope), poolConfigName: "missing"},
- want: restoreWant{code: codes.FailedPrecondition},
- },
- {
- name: "20 local snapshot restore with a missing pool SandboxConfig is rejected",
- actor: actorSeed{localSnapshot: localSnap(localSnapshotName, fullScope), poolConfigName: "missing"},
- want: restoreWant{code: codes.FailedPrecondition},
- },
- {
- name: "21 stamped Full durable snapshot on matching worker restores in Full",
+ name: "19 stamped Full durable snapshot on matching worker restores in Full",
actor: actorSeed{
externalSnapshot: stampedExtSnap(ateapipb.SnapshotOwner_SNAPSHOT_OWNER_ACTOR, actorURI, fullScope, amd64RT),
tmplUID: "current",
@@ -1562,7 +1577,7 @@ func TestResumeActor_AteletWireRequest(t *testing.T) {
},
},
{
- name: "22 stamped actor-owned Full durable snapshot on incompatible worker drops to Volumes",
+ name: "20 stamped actor-owned Full durable snapshot on incompatible worker drops to Rootfs",
actor: actorSeed{
externalSnapshot: stampedExtSnap(ateapipb.SnapshotOwner_SNAPSHOT_OWNER_ACTOR, actorURI, fullScope, amd64RT),
tmplUID: "current",
@@ -1571,11 +1586,11 @@ func TestResumeActor_AteletWireRequest(t *testing.T) {
want: restoreWant{
snapshotName: durableSnapshotName,
snapshotURI: actorURI,
- scope: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES,
+ scope: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS,
},
},
{
- name: "23 stamped borrowed golden snapshot on incompatible worker drops to Volumes",
+ name: "21 stamped borrowed golden snapshot on incompatible worker drops to Rootfs",
actor: actorSeed{
externalSnapshot: stampedExtSnap(ateapipb.SnapshotOwner_SNAPSHOT_OWNER_TAG, goldenURI, fullScope, amd64RT),
tmplUID: "current",
@@ -1585,17 +1600,31 @@ func TestResumeActor_AteletWireRequest(t *testing.T) {
want: restoreWant{
snapshotName: durableSnapshotName,
snapshotURI: goldenURI,
- scope: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES,
+ scope: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS,
},
},
{
- name: "24 stamped explicit source_tag clone on incompatible worker drops to Volumes",
+ name: "22 stamped explicit source_tag clone on incompatible worker drops to Rootfs",
actor: actorSeed{
externalSnapshot: stampedExtSnap(ateapipb.SnapshotOwner_SNAPSHOT_OWNER_TAG, actorURI, fullScope, amd64RT),
tmplUID: "current",
sourceTag: &ateapipb.ObjectRef{Atespace: "team-a", Name: "user-tag"},
assignedRuntime: arm64RT,
},
+ want: restoreWant{
+ snapshotName: durableSnapshotName,
+ snapshotURI: actorURI,
+ scope: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS,
+ },
+ },
+ {
+ // A repointed actor drops to Volumes whatever the worker's runtime.
+ name: "23 stamped Full durable snapshot with replaced template on incompatible worker drops to Volumes",
+ actor: actorSeed{
+ externalSnapshot: stampedExtSnap(ateapipb.SnapshotOwner_SNAPSHOT_OWNER_ACTOR, actorURI, fullScope, amd64RT),
+ tmplUID: "mismatch",
+ assignedRuntime: arm64RT,
+ },
want: restoreWant{
snapshotName: durableSnapshotName,
snapshotURI: actorURI,
@@ -1603,9 +1632,21 @@ func TestResumeActor_AteletWireRequest(t *testing.T) {
},
},
{
- // atelet uploads a pause's snapshot in the background. Once that
- // upload is recorded, a local restore sends it too, so atelet can
- // download the snapshot if its local copy is gone.
+ name: "24 stamped Rootfs durable snapshot on worker with different runtime stays Rootfs",
+ actor: actorSeed{
+ externalSnapshot: stampedExtSnap(ateapipb.SnapshotOwner_SNAPSHOT_OWNER_ACTOR, actorURI, rootfsScope, amd64RT),
+ tmplUID: "current",
+ assignedRuntime: arm64RT,
+ },
+ want: restoreWant{
+ snapshotName: durableSnapshotName,
+ snapshotURI: actorURI,
+ scope: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS,
+ },
+ },
+ {
+ // A recorded upload changes nothing: a local restore names the
+ // pause snapshot's upload whether or not it is COMPLETED.
name: "25 uploaded pause snapshot restores locally and sends its upload",
actor: actorSeed{
localSnapshot: uploadedLocalSnap(localSnapshotName, fullScope),
@@ -1616,15 +1657,24 @@ func TestResumeActor_AteletWireRequest(t *testing.T) {
scope: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY,
},
},
+ {
+ // A local restore names the pause snapshot's upload, so a record
+ // that cannot name it fails before a worker is claimed, as a
+ // malformed durable snapshot URI does.
+ name: "26 local snapshot with no upload recorded fails with DataLoss",
+ actor: actorSeed{localSnapshot: localSnapWithUpload(localSnapshotName, fullScope, nil)},
+ want: restoreWant{code: codes.DataLoss},
+ },
+ {
+ name: "27 local snapshot with a malformed upload URI fails with DataLoss",
+ actor: actorSeed{localSnapshot: localSnapWithUpload(localSnapshotName, fullScope, &ateapipb.SnapshotStorage{
+ Status: ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_PENDING,
+ Object: &ateapipb.ObjectSnapshot{SnapshotUri: malformedURI},
+ })},
+ want: restoreWant{code: codes.DataLoss},
+ },
}
- // Every request atelet receives, Run or Restore, carries the sandbox the
- // assigned worker's pool SandboxConfig resolves to.
- wantSandboxAssets := sandboxAssetsProto(
- &atev1alpha1.SandboxConfig{Spec: atev1alpha1.SandboxConfigSpec{SandboxClass: atev1alpha1.SandboxClassGvisor}},
- &atev1alpha1.SandboxVersionConfig{Name: "v1", PauseImage: "pause@sha256:abc", Assets: testAssets()},
- )
-
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
ctx := context.Background()
@@ -1632,9 +1682,6 @@ func TestResumeActor_AteletWireRequest(t *testing.T) {
w, atelet := newWireCaptureWorkflow(t, persistence)
storetest.MustCreateAtespace(t, ctx, persistence, "ns")
- if tt.actor.poolConfigName != "" {
- w.workerPoolLister = workerPoolListerFor(t, []*atev1alpha1.WorkerPool{resolvedWorkerPool("worker-ns", "pool", tt.actor.poolConfigName)})
- }
tmpl := &ateapipb.ActorTemplate{
Metadata: &ateapipb.ResourceMetadata{Atespace: "ns", Name: "tmpl1"},
SnapshotConfig: &ateapipb.SnapshotConfig{
@@ -1722,9 +1769,6 @@ func TestResumeActor_AteletWireRequest(t *testing.T) {
if run == nil || restore != nil {
t.Fatalf("atelet requests = (restore=%v, run=%v), want exactly one Run", restore, run)
}
- if !proto.Equal(run.GetSandboxAssets(), wantSandboxAssets) {
- t.Errorf("run SandboxAssets = %v, want %v", run.GetSandboxAssets(), wantSandboxAssets)
- }
if !proto.Equal(run.GetSandboxRuntime(), wantSandboxRuntime) {
t.Errorf("run SandboxRuntime = %v, want %v", run.GetSandboxRuntime(), wantSandboxRuntime)
}
@@ -1733,9 +1777,6 @@ func TestResumeActor_AteletWireRequest(t *testing.T) {
if restore == nil || run != nil {
t.Fatalf("atelet requests = (restore=%v, run=%v), want exactly one Restore", restore, run)
}
- if !proto.Equal(restore.GetSandboxAssets(), wantSandboxAssets) {
- t.Errorf("restore SandboxAssets = %v, want %v", restore.GetSandboxAssets(), wantSandboxAssets)
- }
if !proto.Equal(restore.GetSandboxRuntime(), wantSandboxRuntime) {
t.Errorf("restore SandboxRuntime = %v, want %v", restore.GetSandboxRuntime(), wantSandboxRuntime)
}
@@ -1784,10 +1825,10 @@ func TestEnsureVolumesAttached_ReturnsPublishContext(t *testing.T) {
Metadata: &ateapipb.ResourceMetadata{Atespace: "team-a", Name: "id1"},
Status: &ateapipb.ActorStatus{
State: ateapipb.ActorState_ACTOR_STATE_RESUMING,
- ExternalVolumes: []*ateapipb.ExternalVolume{
+ ExternalVolumes: &ateapipb.ExternalVolumes{Volumes: []*ateapipb.ExternalVolumeStatus{
{Name: "mounted", StorageVolumeId: "storage-mounted", VolumeType: "mock"},
{Name: "unmounted", StorageVolumeId: "storage-unmounted", VolumeType: "mock"},
- },
+ }},
},
})
actor, err := persistence.GetActor(ctx, actorRef)
@@ -1805,8 +1846,8 @@ func TestEnsureVolumesAttached_ReturnsPublishContext(t *testing.T) {
worker := &ateapipb.Worker{NodeName: "node-1"}
tmpl := &ateapipb.ActorTemplate{
Volumes: []*ateapipb.Volume{
- {Name: "mounted", ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{StorageClassName: "sc"}},
- {Name: "unmounted", ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{StorageClassName: "sc"}},
+ {Name: "mounted", External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{StorageClassName: "sc"}}},
+ {Name: "unmounted", External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{StorageClassName: "sc"}}},
},
Containers: []*ateapipb.Container{
{Name: "main", Image: "img", VolumeMounts: []*ateapipb.VolumeMount{{Name: "mounted", MountPath: "/data"}}},
@@ -1850,8 +1891,8 @@ func (p *createCountingVolumePlugin) CreateVolume(ctx context.Context, req volum
func TestEnsureVolumesCreated(t *testing.T) {
tmpl := &ateapipb.ActorTemplate{
Volumes: []*ateapipb.Volume{{
- Name: "data",
- ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{StorageClassName: "sc"},
+ Name: "data",
+ External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{StorageClassName: "sc"}},
}},
}
tests := []struct {
@@ -1872,7 +1913,7 @@ func TestEnsureVolumesCreated(t *testing.T) {
Metadata: &ateapipb.ResourceMetadata{Atespace: "team-a", Name: "id1"},
Status: &ateapipb.ActorStatus{
State: tt.state,
- ExternalVolumes: []*ateapipb.ExternalVolume{{Name: "data", VolumeType: "mock"}},
+ ExternalVolumes: &ateapipb.ExternalVolumes{Volumes: []*ateapipb.ExternalVolumeStatus{{Name: "data", VolumeType: "mock"}}},
},
})
actor, err := persistence.GetActor(ctx, actorRef)
@@ -1900,7 +1941,7 @@ func TestEnsureVolumesCreated(t *testing.T) {
if err != nil {
t.Fatalf("GetActor: %v", err)
}
- if provisioned := stored.GetStatus().GetExternalVolumes()[0].GetStorageVolumeId() != ""; provisioned != (tt.wantCreates > 0) {
+ if provisioned := stored.GetStatus().GetExternalVolumes().GetVolumes()[0].GetStorageVolumeId() != ""; provisioned != (tt.wantCreates > 0) {
t.Errorf("stored volume provisioned = %t, want %t", provisioned, tt.wantCreates > 0)
}
})
@@ -2047,8 +2088,9 @@ func contendedWorker(podName string) *ateapipb.Worker {
WorkerPodUid: testWorkerUID(podName),
SandboxClass: "gvisor",
Status: &ateapipb.WorkerStatus{
- State: ateapipb.WorkerState_WORKER_STATE_ACTIVE,
- Capacity: &ateapipb.WorkerResources{Actors: 1},
+ State: ateapipb.WorkerState_WORKER_STATE_ACTIVE,
+ Capacity: &ateapipb.WorkerResources{Actors: 1},
+ DefaultRuntime: testSandboxRuntime(),
},
}
}
@@ -2513,7 +2555,7 @@ func TestAssignWorkerAttempt_PrefersRuntimeMatchingWorkerAndFallsBack(t *testing
if got := worker.GetWorkerPod(); got != "arm-pod" {
t.Errorf("assigned worker = %q, want %q", got, "arm-pod")
}
- // The volumes-only restore runs under the fallback worker's default.
+ // The restore without memory state runs under the fallback worker's default.
assertAssignedRuntime(t, stored, arm64RT)
})
@@ -2523,6 +2565,7 @@ func TestAssignWorkerAttempt_PrefersRuntimeMatchingWorkerAndFallsBack(t *testing
// even for a volumes-only restore that needs no runtime at all.
for _, fidelity := range []ateapipb.SnapshotFidelity{
ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY,
+ ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS,
ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES,
} {
t.Run(fidelity.String(), func(t *testing.T) {
@@ -2690,12 +2733,11 @@ func TestRequiredSnapshotRuntime(t *testing.T) {
want: durableRT,
},
{
- name: "completed ROOTFS durable snapshot requires its runtime",
+ name: "completed ROOTFS durable snapshot restores no memory state",
snapshots: []*ateapipb.Snapshot{durable(1, rootfs, tmplUID, completed, durableRT)},
- want: durableRT,
},
{
- name: "VOLUMES durable snapshot restores no guest state",
+ name: "VOLUMES durable snapshot restores no memory state",
snapshots: []*ateapipb.Snapshot{durable(1, volumes, tmplUID, completed, durableRT)},
},
{
@@ -2731,6 +2773,13 @@ func TestRequiredSnapshotRuntime(t *testing.T) {
},
want: localRT,
},
+ {
+ name: "ROOTFS local checkpoint imposes no constraint beside a stamped durable snapshot",
+ snapshots: []*ateapipb.Snapshot{
+ local(2, rootfs, localRT),
+ durable(1, memory, tmplUID, completed, durableRT),
+ },
+ },
{
name: "VOLUMES local checkpoint imposes no constraint beside a stamped durable snapshot",
snapshots: []*ateapipb.Snapshot{
diff --git a/cmd/ateapi/internal/controlapi/workflow_suspend.go b/cmd/ateapi/internal/controlapi/workflow_suspend.go
index 0b4a65cf98..e567291a13 100644
--- a/cmd/ateapi/internal/controlapi/workflow_suspend.go
+++ b/cmd/ateapi/internal/controlapi/workflow_suspend.go
@@ -29,6 +29,8 @@ import (
"go.ate.dev/substrate/internal/resources"
"go.ate.dev/substrate/pkg/proto/ateapipb"
"go.opentelemetry.io/otel/attribute"
+ "google.golang.org/grpc/codes"
+ "google.golang.org/grpc/status"
"google.golang.org/protobuf/proto"
)
@@ -172,7 +174,7 @@ func (w *ActorWorkflow) ensureMarkedSuspending(ctx context.Context, actorRef res
// Fail here rather than at checkpoint time if the template's location
// cannot produce a usable URI: nothing has been written yet.
- uri, err := newInProgressSnapshotURI(actorTemplate, actor, snapshotName)
+ uri, err := w.newInProgressSnapshotURI(actorTemplate, actor, snapshotName)
if err != nil {
return nil, err
}
@@ -267,6 +269,10 @@ func (w *ActorWorkflow) ensureAteletSuspended(ctx context.Context, actorRef reso
resp, err := client.Checkpoint(ctx, req)
if err != nil {
+ // A snapshot conflict leaves the actor and its assignment intact for a retry.
+ if ctx.Err() == nil && status.Code(err) == codes.Aborted {
+ return wireFidelity, nil, apierror.Aborted("while calling atelet Checkpoint: %w", err)
+ }
return wireFidelity, nil, handleAteletError(ctx, w.store, actorRef, ateattr.OperationSuspend, "Checkpoint", false, err)
}
return wireFidelity, resp.GetState(), nil
@@ -274,9 +280,13 @@ func (w *ActorWorkflow) ensureAteletSuspended(ctx context.Context, actorRef reso
// newInProgressSnapshotURI is where the snapshot an actor is currently taking is
// written: under the actor's own prefix, so the objects name their owner.
-func newInProgressSnapshotURI(actorTemplate *ateapipb.ActorTemplate, actor *ateapipb.Actor, snapshotName string) (resources.SnapshotURI, error) {
+func (w *ActorWorkflow) newInProgressSnapshotURI(actorTemplate *ateapipb.ActorTemplate, actor *ateapipb.Actor, snapshotName string) (resources.SnapshotURI, error) {
+ location, err := w.snapshotLocation(actorTemplate)
+ if err != nil {
+ return resources.SnapshotURI{}, err
+ }
atespace := actor.GetMetadata().GetAtespace()
- uri, err := resources.NewActorSnapshotURI(actorTemplate.GetSnapshotConfig().GetStorageLocation(), atespace, actor.GetMetadata().GetUid(), snapshotName)
+ uri, err := resources.NewActorSnapshotURI(location, atespace, actor.GetMetadata().GetUid(), snapshotName)
if err != nil {
return resources.SnapshotURI{}, fmt.Errorf("while building the snapshot URI for actor %s/%s: %w", atespace, actor.GetMetadata().GetName(), err)
}
@@ -312,8 +322,7 @@ func (w *ActorWorkflow) ensureVolumesDetached(ctx context.Context, actor *ateapi
// is no longer SUSPENDING is left as it is and the step fails with
// FailedPrecondition.
// ateletState is the state atelet returned from Checkpoint, applied in the
-// same update. A state that conflicts with the record fails the step before it
-// frees the worker.
+// same update. A state that conflicts with the record crashes the actor.
func (w *ActorWorkflow) ensureSuspendedFinalized(ctx context.Context, actorRef resources.ActorRef, mode suspendMode, ateletState *ateletpb.ActorSnapshotState) (_ *ateapipb.Actor, err error) {
ctx, done := stepSpan(ctx, "FinalizeSuspended")
defer func() { err = done(err) }()
@@ -359,12 +368,14 @@ func (w *ActorWorkflow) ensureSuspendedFinalized(ctx context.Context, actorRef r
return nil, crashActorAndFail(ctx, w.store, actorRef, ateattr.OperationSuspend, crashMessageDurableSnapshotMissing)
}
- // Apply atelet's state before freeing the worker, so a state that
- // conflicts with the record fails the step while a retry can still call
- // atelet again.
+ // A state that conflicts with the record crashes the actor. Atelet has
+ // already destroyed the workload, so a retried Checkpoint can't succeed.
applied := proto.CloneOf(latestActor.GetStatus())
- if err := applyCheckpointState(ctx, applied, ateletState); err != nil {
- return nil, err
+ if err := applyAteletResponseState(ctx, applied, ateletState); err != nil {
+ slog.WarnContext(ctx, "Snapshot state atelet returned conflicts with the record",
+ slog.Any("actor", actorRef),
+ slog.Any("err", err))
+ return nil, crashActorAndFail(ctx, w.store, actorRef, ateattr.OperationSuspend, crashMessageSnapshotStateConflict)
}
// 1. Free the worker (if it hasn't been freed yet)
diff --git a/cmd/ateapi/internal/controlapi/workflow_suspend_async_test.go b/cmd/ateapi/internal/controlapi/workflow_suspend_async_test.go
index f6f57a3a2a..e99bb961c1 100644
--- a/cmd/ateapi/internal/controlapi/workflow_suspend_async_test.go
+++ b/cmd/ateapi/internal/controlapi/workflow_suspend_async_test.go
@@ -233,8 +233,8 @@ func TestSuspendActorAsync_Finalized_RecordsFidelity(t *testing.T) {
if got.GetStatus().GetState() != ateapipb.ActorState_ACTOR_STATE_SUSPENDED {
t.Fatalf("state = %v, want SUSPENDED", got.GetStatus().GetState())
}
- if !latestSnapshotIsLocalOnly(got.GetStatus()) {
- t.Errorf("snapshots = %v, want the latest one local only", got.GetStatus().GetSnapshots())
+ if !latestSnapshotIsLocal(got.GetStatus()) {
+ t.Errorf("snapshots = %v, want the latest one local", got.GetStatus().GetSnapshots())
}
localSnap := findLatestLocalSnapshot(got.GetStatus())
if localSnap.GetFidelity() != tc.want {
diff --git a/cmd/ateapi/internal/controlapi/workflow_suspend_test.go b/cmd/ateapi/internal/controlapi/workflow_suspend_test.go
index 721be71052..af9789d3f7 100644
--- a/cmd/ateapi/internal/controlapi/workflow_suspend_test.go
+++ b/cmd/ateapi/internal/controlapi/workflow_suspend_test.go
@@ -37,6 +37,110 @@ import (
"k8s.io/client-go/tools/cache"
)
+// TestSuspendActor_CheckpointError exercises checkpoint error policy through
+// the suspend workflow, including retrying the persisted snapshot.
+func TestSuspendActor_CheckpointError(t *testing.T) {
+ tests := []struct {
+ name string
+ err error
+ cancel bool
+ wantCode codes.Code
+ wantState ateapipb.ActorState
+ }{
+ {"snapshot conflict", apierror.Aborted("checkpoint rejected"), false, codes.Aborted, ateapipb.ActorState_ACTOR_STATE_SUSPENDING},
+ {"failed precondition", apierror.FailedPrecondition("checkpoint rejected"), false, codes.Internal, ateapipb.ActorState_ACTOR_STATE_CRASHED},
+ {"caller cancellation", apierror.Aborted("checkpoint rejected"), true, codes.Canceled, ateapipb.ActorState_ACTOR_STATE_SUSPENDING},
+ }
+ for _, tt := range tests {
+ for _, mode := range suspendModes {
+ t.Run(tt.name+"/"+mode.name, func(t *testing.T) {
+ ctx := context.Background()
+ persistence := newTestPersistence(t)
+ seedSuspendTemplate(t, ctx, persistence, &ateapipb.SnapshotConfig{PreferredFidelity: ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY})
+ w, atelet := newWireCaptureWorkflow(t, persistence)
+ actorRef := resources.ActorRef{Atespace: "team-a", Name: "actor-1"}
+ actor := storetest.MustCreateActor(t, ctx, persistence, &ateapipb.Actor{
+ Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name},
+ ActorTemplate: suspendTestTemplateRef(),
+ Status: &ateapipb.ActorStatus{
+ State: ateapipb.ActorState_ACTOR_STATE_RUNNING,
+ AssignedNode: wireTestAssignment().GetNodeName(),
+ WorkerAssignment: wireTestAssignment(),
+ },
+ })
+ seedWireWorker(t, ctx, persistence, assignmentOf(actor))
+ callCtx, cancel := context.WithCancel(ctx)
+ defer cancel()
+ atelet.onCheckpoint = func(context.Context) error {
+ if tt.cancel {
+ cancel()
+ }
+ return tt.err
+ }
+ _, err := w.SuspendActor(callCtx, actorRef, mode.suspendMode)
+ if got := apierror.Code(err); got != tt.wantCode {
+ t.Fatalf("SuspendActor error = %v (code %v), want %v", err, got, tt.wantCode)
+ }
+ if tt.cancel && !errors.Is(err, context.Canceled) {
+ t.Errorf("SuspendActor error does not wrap context.Canceled: %v", err)
+ }
+ if checkpoints, _ := atelet.calls(); checkpoints != 1 {
+ t.Fatalf("Checkpoint calls = %d, want 1", checkpoints)
+ }
+ stored := mustReadActor(t, ctx, persistence, actorRef)
+ st := stored.GetStatus()
+ if st.GetState() != tt.wantState {
+ t.Fatalf("actor state = %v, want %v", st.GetState(), tt.wantState)
+ }
+ worker, err := persistence.GetWorker(ctx, wireTestAssignment().GetWorker().GetName())
+ if err != nil {
+ t.Fatalf("GetWorker: %v", err)
+ }
+ if tt.wantState == ateapipb.ActorState_ACTOR_STATE_CRASHED {
+ if st.GetCrash() == nil || st.GetWorkerAssignment() != nil || worker.GetStatus().GetAllocated().GetActors() != 0 {
+ t.Errorf("checkpoint failure did not crash the actor and release its worker: actor %v, worker %v", st, worker.GetStatus())
+ }
+ return
+ }
+ if st.GetCrash() != nil || !proto.Equal(st.GetWorkerAssignment(), actor.GetStatus().GetWorkerAssignment()) {
+ t.Errorf("checkpoint error changed the worker assignment or crashed the actor: %v", st)
+ }
+ if got := worker.GetStatus().GetAllocated().GetActors(); got != 1 {
+ t.Errorf("worker allocated actors = %d, want 1", got)
+ }
+ pending := snapshotAtLatestGeneration(st)
+ if pending.GetUuid() == "" || pending.GetDurableSnapshot().GetStatus() != ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_PENDING {
+ t.Fatalf("snapshot = %v, want a persisted pending snapshot", pending)
+ }
+ first := atelet.checkpointRequest().GetSnapshot()
+ if first.GetSnapshotUuid() != pending.GetUuid() || first.GetGeneration() != pending.GetGeneration() {
+ t.Fatalf("Checkpoint snapshot = %v, want pending snapshot %v", first, pending)
+ }
+ atelet.mu.Lock()
+ atelet.onCheckpoint = nil
+ atelet.mu.Unlock()
+ finished, err := w.SuspendActor(ctx, actorRef, mode.suspendMode)
+ if err != nil {
+ t.Fatalf("SuspendActor retry: %v", err)
+ }
+ if checkpoints, _ := atelet.calls(); checkpoints != 2 {
+ t.Errorf("Checkpoint calls after retry = %d, want 2", checkpoints)
+ }
+ if diff := cmp.Diff(first, atelet.checkpointRequest().GetSnapshot(), protocmp.Transform()); diff != "" {
+ t.Errorf("retry changed the snapshot (-first +retry):\n%s", diff)
+ }
+ if finished.GetStatus().GetState() != ateapipb.ActorState_ACTOR_STATE_SUSPENDED || finished.GetStatus().GetWorkerAssignment() != nil {
+ t.Errorf("retry did not suspend the actor and clear its assignment: %v", finished.GetStatus())
+ }
+ latest := snapshotAtLatestGeneration(finished.GetStatus())
+ if latest.GetUuid() != pending.GetUuid() || latest.GetGeneration() != pending.GetGeneration() {
+ t.Errorf("retry finalized a different snapshot: %v, want %v", latest, pending)
+ }
+ })
+ }
+ }
+}
+
func TestSuspendActor_MarkedSuspending_SnapshotURI(t *testing.T) {
ctx := context.Background()
persistence := newTestPersistence(t)
@@ -593,6 +697,63 @@ func TestSuspendActor_Finalized_PauseReleasesReplacedSnapshot(t *testing.T) {
}
}
+// TestSnapshotLocationChange_LeavesNothingBehind verifies that moving an
+// actor's snapshot location, by changing --snapshot-location, toggling
+// the PerTemplateSnapshotLocation preview gate, or repointing the actor, strands
+// nothing: the next suspend releases the replaced snapshot under its own
+// location, and deleting the actor then collects the new one.
+func TestSnapshotLocationChange_LeavesNothingBehind(t *testing.T) {
+ const (
+ oldLocation = "gs://old-bucket/root"
+ newLocation = "gs://new-bucket/root"
+ )
+ ctx := context.Background()
+ persistence := newTestPersistence(t)
+ w, objects := newFinalizeWorkflow(persistence)
+
+ actorRef := resources.ActorRef{Atespace: "team-a", Name: "actor-1"}
+ actor := storetest.MustCreateActor(t, ctx, persistence, &ateapipb.Actor{
+ Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name},
+ ActorTemplate: &ateapipb.ObjectRef{Atespace: "team-a", Name: "sub-tmpl"},
+ Status: &ateapipb.ActorStatus{State: ateapipb.ActorState_ACTOR_STATE_SUSPENDING},
+ })
+ previous, err := resources.NewActorSnapshotURI(oldLocation, actorRef.Atespace, actor.GetMetadata().GetUid(), "old")
+ if err != nil {
+ t.Fatalf("NewActorSnapshotURI(old): %v", err)
+ }
+ fresh, err := resources.NewActorSnapshotURI(newLocation, actorRef.Atespace, actor.GetMetadata().GetUid(), "2026-01-01t00-00-00z-new")
+ if err != nil {
+ t.Fatalf("NewActorSnapshotURI(new): %v", err)
+ }
+ objects.PutSnapshot(t, previous, "manifest.json")
+ objects.PutSnapshot(t, fresh, "manifest.json")
+ mustUpdateActorStatus(t, ctx, persistence, actor, func(s *ateapipb.ActorStatus) {
+ s.LastAssignedGeneration = 2
+ s.Snapshots = []*ateapipb.Snapshot{
+ newDurableSnapshot(1, ateapipb.SnapshotOwner_SNAPSHOT_OWNER_ACTOR, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", previous.Name(), previous.String(), ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED),
+ newDurableSnapshot(2, ateapipb.SnapshotOwner_SNAPSHOT_OWNER_ACTOR, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", fresh.Name(), fresh.String(), ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_PENDING),
+ }
+ })
+
+ stored, err := w.ensureSuspendedFinalized(ctx, actorRef, suspendSync, nil)
+ if err != nil {
+ t.Fatalf("ensureSuspendedFinalized: %v", err)
+ }
+ if got := objects.Snapshot(t, previous); len(got) != 0 {
+ t.Errorf("snapshot under the old location after the suspend = %v, want released", got)
+ }
+ if len(objects.Snapshot(t, fresh)) == 0 {
+ t.Fatal("the snapshot this suspend wrote under the new location was collected")
+ }
+
+ if err := w.ensureExternalSnapshotsReleased(ctx, stored); err != nil {
+ t.Fatalf("ensureExternalSnapshotsReleased: %v", err)
+ }
+ if got := objects.Snapshot(t, fresh); len(got) != 0 {
+ t.Errorf("snapshot under the new location after the delete = %v, want released", got)
+ }
+}
+
// errObjectStore stands in for object storage being unreachable.
var errObjectStore = errors.New("object storage is unavailable")
@@ -1256,8 +1417,7 @@ func assertRequestNamesLatestSnapshot(t *testing.T, atelet *capturingAtelet, tmp
// TestSuspendActor_AppliesCheckpointState verifies finalize applies the
// snapshot state atelet returns from Checkpoint, with the report rules. A state
-// that conflicts with the record fails the suspend, and leaves the actor
-// SUSPENDING with its worker.
+// that conflicts with the record crashes the actor.
func TestSuspendActor_AppliesCheckpointState(t *testing.T) {
const node = "node-1"
memory := ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY
@@ -1335,7 +1495,7 @@ func TestSuspendActor_AppliesCheckpointState(t *testing.T) {
},
wantSnapshots: []snapshotOutcome{{generation: 1, fidelity: ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS, durable: completed}},
}, {
- name: "sync: a state that conflicts with the record fails the suspend",
+ name: "sync: a state that conflicts with the record crashes the actor",
mode: suspendSync,
state: func(req *ateletpb.CheckpointRequest) *ateletpb.ActorSnapshotState {
snap := proto.CloneOf(req.GetSnapshot())
@@ -1343,9 +1503,9 @@ func TestSuspendActor_AppliesCheckpointState(t *testing.T) {
snap.Object = &ateletpb.ObjectStorage{SnapshotUri: uriOf(9)}
return &ateletpb.ActorSnapshotState{LatestDurable: &ateletpb.SnapshotReport{Snapshot: snap}}
},
- wantCode: codes.InvalidArgument,
+ wantCode: codes.FailedPrecondition,
}, {
- name: "async: a state that conflicts with the record fails the suspend",
+ name: "async: a state that conflicts with the record crashes the actor",
mode: suspendAsync,
status: &ateapipb.ActorStatus{LastAssignedGeneration: 1, Snapshots: []*ateapipb.Snapshot{durable(1, pending)}},
state: func(req *ateletpb.CheckpointRequest) *ateletpb.ActorSnapshotState {
@@ -1354,7 +1514,7 @@ func TestSuspendActor_AppliesCheckpointState(t *testing.T) {
LatestDurable: report(1, uriOf(9), ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY),
}
},
- wantCode: codes.InvalidArgument,
+ wantCode: codes.FailedPrecondition,
}}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
@@ -1390,11 +1550,11 @@ func TestSuspendActor_AppliesCheckpointState(t *testing.T) {
if got := apierror.Code(err); got != tt.wantCode {
t.Fatalf("suspend = %v, want code %v", err, tt.wantCode)
}
- if got := stored.GetStatus().GetState(); got != ateapipb.ActorState_ACTOR_STATE_SUSPENDING {
- t.Errorf("state = %v, want SUSPENDING", got)
+ if got := stored.GetStatus().GetState(); got != ateapipb.ActorState_ACTOR_STATE_CRASHED {
+ t.Errorf("state = %v, want CRASHED", got)
}
- if firstAssignment(t, persistence, wireTestAssignment().GetWorker().GetName()) == nil {
- t.Error("worker released, want it still holding the actor so a retry can call atelet")
+ if got, want := stored.GetStatus().GetCrash().GetMessage(), "suspend failed: "+crashMessageSnapshotStateConflict; got != want {
+ t.Errorf("crash message = %q, want %q", got, want)
}
return
}
diff --git a/cmd/ateapi/internal/controlapi/workflow_tag.go b/cmd/ateapi/internal/controlapi/workflow_tag.go
index e880e9c090..fa18332620 100644
--- a/cmd/ateapi/internal/controlapi/workflow_tag.go
+++ b/cmd/ateapi/internal/controlapi/workflow_tag.go
@@ -22,8 +22,11 @@ import (
"go.ate.dev/substrate/cmd/ateapi/internal/store"
"go.ate.dev/substrate/internal/apierror"
"go.ate.dev/substrate/internal/preview"
+ "go.ate.dev/substrate/internal/proto/ateletpb"
"go.ate.dev/substrate/internal/resources"
"go.ate.dev/substrate/pkg/proto/ateapipb"
+ "google.golang.org/grpc/codes"
+ "google.golang.org/grpc/status"
"google.golang.org/protobuf/proto"
)
@@ -32,18 +35,21 @@ import (
// The tag is given its own copy of that snapshot, so suspending the actor again
// or deleting the actor does not garbage collect the tag's snapshot.
//
-// The tag is built in 4 phases:
-// 1. Reserve the tag's name; the store assigns its UID.
-// 2. Record the snapshot's URI, under the tag's UID, before anything is
+// The tag is built in 6 phases:
+// 1. Check that the name is free and the storage location is usable.
+// 2. Make the actor's latest snapshot durable, if it is not uploaded yet: a
+// paused actor's snapshot may still be uploading.
+// 3. Reserve the tag's name; the store assigns its UID.
+// 4. Record the snapshot's URI, under the tag's UID, before anything is
// copied there.
-// 3. Copy the snapshot to that URI.
-// 4. Finalize: mark the tag's snapshot COMPLETED.
+// 5. Copy the snapshot to that URI.
+// 6. Finalize: mark the tag's snapshot COMPLETED.
//
// The tag captures whichever snapshot the actor holds when the workflow runs.
// An actor keeps no snapshot history, so a suspend that lands first moves what
// gets tagged; that race is inherent to naming an actor rather than a snapshot.
//
-// Not idempotent: the name is taken as soon as phase 1 lands, so a create that
+// Not idempotent: the name is taken as soon as phase 3 lands, so a create that
// dies after it leaves a pending tag and every later create under that name is
// AlreadyExists. To retry, delete the tag, which collects whatever the failed
// attempt stranded, and create it again.
@@ -72,9 +78,21 @@ func (w *ActorWorkflow) TagActorSnapshot(ctx context.Context, tag *ateapipb.Tag)
if err != nil {
return nil, err
}
- snapshot := findLatestDurableSnapshot(actor.GetStatus(), ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED)
+ // Runs before the upload, so a create that cannot succeed fails fast.
+ if err := w.checkTagCreatable(leaseCtx, tagRef, actorTemplate); err != nil {
+ return nil, err
+ }
+ // Runs before the name is taken, so a failed upload leaves no pending tag.
+ actor, err = w.ensureTagSourceSnapshotDurable(leaseCtx, actorRef, actor)
+ if err != nil {
+ return nil, err
+ }
+ snapshot, err := sourceSnapshotForTag(actorRef, actor)
+ if err != nil {
+ return nil, err
+ }
- reserved, err := w.ensureTagReserved(leaseCtx, tagRef, actor, actorTemplate, snapshot, tag)
+ reserved, err := w.ensureTagReserved(leaseCtx, tagRef, actor, snapshot, tag)
if err != nil {
return nil, err
}
@@ -204,7 +222,7 @@ func (w *ActorWorkflow) finalizeTagDeleted(ctx context.Context, tagRef resources
}
// loadActorForTag fetches the actor to tag and its template, and checks that
-// the actor holds an external snapshot a tag can be made from.
+// the actor is suspended.
func (w *ActorWorkflow) loadActorForTag(ctx context.Context, actorRef resources.ActorRef) (_ *ateapipb.Actor, _ *ateapipb.ActorTemplate, err error) {
ctx, done := stepSpan(ctx, "LoadActorForTag")
defer func() { err = done(err) }()
@@ -218,14 +236,149 @@ func (w *ActorWorkflow) loadActorForTag(ctx context.Context, actorRef resources.
if got := actor.GetStatus().GetState(); got != ateapipb.ActorState_ACTOR_STATE_SUSPENDED {
return nil, nil, apierror.FailedPrecondition("Actor %s must be %s to be tagged (got: %v)", actorRef, ateapipb.ActorState_ACTOR_STATE_SUSPENDED, got)
}
+ actorTemplate, err := resolveActorTemplate(ctx, w.store, actor)
+ if err != nil {
+ return nil, nil, err
+ }
+ return actor, actorTemplate, nil
+}
+
+// checkTagCreatable fails a create that cannot succeed before the snapshot is
+// uploaded: a name already taken, or a storage location the copy cannot use.
+// The tag lease is held, so no other create or delete of the name races the
+// lookup.
+func (w *ActorWorkflow) checkTagCreatable(ctx context.Context, tagRef resources.TagRef, actorTemplate *ateapipb.ActorTemplate) (err error) {
+ ctx, done := stepSpan(ctx, "CheckTagCreatable")
+ defer func() { err = done(err) }()
+
+ _, err = w.store.GetTag(ctx, tagRef)
+ switch {
+ case err == nil:
+ return apierror.AlreadyExists("Tag %s already exists; delete it and create it again to retry", tagRef)
+ case !errors.Is(err, store.ErrNotFound):
+ return fmt.Errorf("while getting tag %s: %w", tagRef, err)
+ }
+ location, err := w.snapshotLocation(actorTemplate)
+ if err != nil {
+ return err
+ }
+ if err := resources.ValidateSnapshotLocation(location); err != nil {
+ return fmt.Errorf("invalid storage location for tag %s: %w", tagRef, err)
+ }
+ return nil
+}
+
+// ensureTagSourceSnapshotDurable makes the actor's latest snapshot durable when
+// it is not uploaded yet, the shape a pause leaves until atelet's background
+// upload is recorded. It asks the atelet on the snapshot's node to upload it
+// with PromoteSnapshot, which keeps the local copy so the actor still resumes
+// on that node, and records the upload on the actor. It returns the actor as
+// stored afterwards.
+//
+// A failed upload does not crash the actor: the tag fails and the actor is
+// left as it was.
+func (w *ActorWorkflow) ensureTagSourceSnapshotDurable(ctx context.Context, actorRef resources.ActorRef, actor *ateapipb.Actor) (_ *ateapipb.Actor, err error) {
+ ctx, done := stepSpan(ctx, "PromoteTagSourceSnapshot")
+ defer func() { err = done(err) }()
+
+ snap := latestSnapshot(actor.GetStatus(), func(*ateapipb.Snapshot) bool { return true })
+ if snap.GetDurableSnapshot().GetStatus() == ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED {
+ markSkipped(ctx, "the actor's latest snapshot is already durable")
+ return actor, nil
+ }
+ // For example a PENDING upload whose node was lost. The tag falls back to
+ // the latest durable snapshot, the one the actor would resume from.
+ if snap.GetLocality() == "" {
+ markSkipped(ctx, "the actor's latest snapshot is on no node to upload from")
+ return actor, nil
+ }
+ snapshotURI := snap.GetDurableSnapshot().GetObject().GetSnapshotUri()
+ if snapshotURI == "" {
+ return nil, apierror.Internal("Actor %s holds a node-local snapshot at generation %d but records no URI to upload it to", actorRef, snap.GetGeneration())
+ }
+ // The uploaded snapshot is the one the tag copies, so a snapshot that
+ // records no template fails before the upload rather than after it.
+ if snap.GetActorTemplateUid() == "" {
+ return nil, apierror.Internal("Actor %s holds a snapshot at generation %d but records no template it was built under", actorRef, snap.GetGeneration())
+ }
+
+ ateletConn, err := w.dialer.DialForAteletOnNode(snap.GetLocality())
+ if err != nil {
+ return nil, apierror.Unavailable("while getting atelet conn for node %q: %w", snap.GetLocality(), err)
+ }
+ // The fidelity is the one recorded in the snapshot, not the current
+ // template's: a suspended actor can be repointed, and atelet requires the
+ // fidelity its manifest records.
+ resp, err := ateletpb.NewAteletClient(ateletConn).PromoteSnapshot(ctx, &ateletpb.PromoteSnapshotRequest{
+ Atespace: actor.GetMetadata().GetAtespace(),
+ ActorName: actor.GetMetadata().GetName(),
+ ActorUid: actor.GetMetadata().GetUid(),
+ ActorTemplateAtespace: actor.GetActorTemplate().GetAtespace(),
+ ActorTemplateName: actor.GetActorTemplate().GetName(),
+ Snapshot: &ateletpb.Snapshot{
+ SnapshotUuid: snap.GetUuid(),
+ Generation: snap.GetGeneration(),
+ Fidelity: fidelityToAtelet(snap.GetFidelity()),
+ Object: &ateletpb.ObjectStorage{SnapshotUri: snapshotURI},
+ },
+ })
+ if err != nil {
+ return nil, promoteSnapshotError(ctx, err)
+ }
+
+ // The reported durable snapshot must be the one requested.
+ reported := resp.GetState().GetLatestDurable().GetSnapshot()
+ if reported.GetSnapshotUuid() != snap.GetUuid() || reported.GetGeneration() != snap.GetGeneration() {
+ return nil, apierror.Internal("atelet on node %q was asked to promote Actor %s's snapshot %s at generation %d, but reported %q at generation %d durable",
+ snap.GetLocality(), actorRef, snap.GetUuid(), snap.GetGeneration(), reported.GetSnapshotUuid(), reported.GetGeneration())
+ }
+ // The actor lease is held for the whole tag, so the actor cannot be resumed
+ // or suspended meanwhile and snap is still its newest snapshot here.
+ applied := proto.CloneOf(actor.GetStatus())
+ // The request was valid, so a state that conflicts with the record is
+ // atelet breaking its contract.
+ if err := applyAteletResponseState(ctx, applied, resp.GetState()); err != nil {
+ return nil, apierror.Internal("atelet on node %q reported a state for Actor %s that conflicts with its record: %w", snap.GetLocality(), actorRef, err)
+ }
+ stored, err := w.store.UpdateActor(ctx, actorRef, store.PreconditionFrom(actor), func(toUpdate *ateapipb.Actor) error {
+ // The precondition makes toUpdate the version applied was built from.
+ toUpdate.Status.Snapshots = applied.GetSnapshots()
+ return nil
+ })
+ if err != nil {
+ if errors.Is(err, store.ErrVersionConflict) {
+ return nil, apierror.Aborted("concurrent update conflict, please retry")
+ }
+ return nil, fmt.Errorf("while recording the upload of Actor %s's latest snapshot: %w", actorRef, err)
+ }
+ return stored, nil
+}
+
+// promoteSnapshotError maps an error from atelet's PromoteSnapshot to the
+// error the tag fails with.
+func promoteSnapshotError(ctx context.Context, err error) error {
+ // The caller went away or the lease was lost: report that, not atelet's error.
+ if ctx.Err() != nil {
+ return fmt.Errorf("while calling atelet PromoteSnapshot: %w: %w", err, ctx.Err())
+ }
+ switch status.Code(err) {
+ case codes.Aborted:
+ return apierror.Aborted("while calling atelet PromoteSnapshot: %w", err)
+ case codes.Unavailable, codes.Canceled, codes.DeadlineExceeded:
+ return apierror.Unavailable("while calling atelet PromoteSnapshot: %w", err)
+ // NotFound: the local snapshot is gone and no uploaded copy exists.
+ case codes.FailedPrecondition, codes.NotFound:
+ return apierror.FailedPrecondition("while calling atelet PromoteSnapshot: %w", err)
+ }
+ return apierror.Internal("while calling atelet PromoteSnapshot: %w", err)
+}
+
+// sourceSnapshotForTag returns the actor's latest COMPLETED external snapshot,
+// the one a tag copies.
+func sourceSnapshotForTag(actorRef resources.ActorRef, actor *ateapipb.Actor) (*ateapipb.Snapshot, error) {
snap := findLatestDurableSnapshot(actor.GetStatus(), ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED)
if snap == nil {
- return nil, nil, apierror.FailedPrecondition("Actor %s holds no external snapshot to tag", actorRef)
- }
- // The actor's latest snapshot is local only. It must be uploaded before it can be tagged.
- // TODO: Wait for atelet's background upload, or upload it here, instead of failing.
- if latestSnapshotIsLocalOnly(actor.GetStatus()) {
- return nil, nil, apierror.FailedPrecondition("Actor %s holds no external snapshot of its latest state to tag: its latest snapshot is not uploaded yet", actorRef)
+ return nil, apierror.FailedPrecondition("Actor %s holds no external snapshot to tag", actorRef)
}
// Every way an Actor comes to hold an external snapshot records the
// template its guest state was built under: a suspend through
@@ -233,13 +386,9 @@ func (w *ActorWorkflow) loadActorForTag(ctx context.Context, actorRef resources.
// A snapshot without one is a broken row, and tagging it would mint a tag
// that names no template.
if snap.GetActorTemplateUid() == "" {
- return nil, nil, apierror.Internal("Actor %s holds an external snapshot but records no template it was built under", actorRef)
+ return nil, apierror.Internal("Actor %s holds an external snapshot but records no template it was built under", actorRef)
}
- actorTemplate, err := resolveActorTemplate(ctx, w.store, actor)
- if err != nil {
- return nil, nil, err
- }
- return actor, actorTemplate, nil
+ return snap, nil
}
// ensureTagReserved takes the tag's name. The store assigns the tag's UID on
@@ -251,15 +400,10 @@ func (w *ActorWorkflow) loadActorForTag(ctx context.Context, actorRef resources.
// deciding whether the objects under it still belong to the snapshot being
// tagged, and the row may not even be this actor's; deleting the tag collects
// them and frees the name, so a retry is a delete followed by a create.
-func (w *ActorWorkflow) ensureTagReserved(ctx context.Context, tagRef resources.TagRef, actor *ateapipb.Actor, actorTemplate *ateapipb.ActorTemplate, snapshot *ateapipb.Snapshot, tag *ateapipb.Tag) (_ *ateapipb.Tag, err error) {
+func (w *ActorWorkflow) ensureTagReserved(ctx context.Context, tagRef resources.TagRef, actor *ateapipb.Actor, snapshot *ateapipb.Snapshot, tag *ateapipb.Tag) (_ *ateapipb.Tag, err error) {
ctx, done := stepSpan(ctx, "ReserveTag")
defer func() { err = done(err) }()
- // Checked before the name is taken, so an unusable location fails the
- // create without leaving a pending tag behind.
- if err := resources.ValidateSnapshotLocation(actorTemplate.GetSnapshotConfig().GetStorageLocation()); err != nil {
- return nil, fmt.Errorf("invalid storage location for tag %s: %w", tagRef, err)
- }
tagToCreate := &ateapipb.Tag{
Metadata: &ateapipb.ResourceMetadata{Atespace: tagRef.Atespace, Name: tagRef.Name},
Scope: tag.GetScope(),
@@ -295,7 +439,11 @@ func (w *ActorWorkflow) ensureTagSnapshotRecorded(ctx context.Context, tag *atea
defer func() { err = done(err) }()
tagRef := resources.TagRefFromTag(tag)
- dst, err := resources.NewTagSnapshotURI(actorTemplate.GetSnapshotConfig().GetStorageLocation(), tagRef.Atespace, tag.GetMetadata().GetUid())
+ location, err := w.snapshotLocation(actorTemplate)
+ if err != nil {
+ return nil, resources.SnapshotURI{}, err
+ }
+ dst, err := resources.NewTagSnapshotURI(location, tagRef.Atespace, tag.GetMetadata().GetUid())
if err != nil {
return nil, resources.SnapshotURI{}, fmt.Errorf("while building the snapshot URI for tag %s: %w", tagRef, err)
}
diff --git a/cmd/ateapi/internal/controlapi/workflow_tag_test.go b/cmd/ateapi/internal/controlapi/workflow_tag_test.go
index 25e0d301af..8820c30890 100644
--- a/cmd/ateapi/internal/controlapi/workflow_tag_test.go
+++ b/cmd/ateapi/internal/controlapi/workflow_tag_test.go
@@ -17,6 +17,7 @@ package controlapi
import (
"context"
"errors"
+ "fmt"
"strings"
"sync"
"testing"
@@ -26,9 +27,12 @@ import (
"go.ate.dev/substrate/cmd/ateapi/internal/store/storetest"
"go.ate.dev/substrate/internal/apierror"
"go.ate.dev/substrate/internal/objectstore/objectstoretest"
+ "go.ate.dev/substrate/internal/proto/ateletpb"
"go.ate.dev/substrate/internal/resources"
"go.ate.dev/substrate/pkg/proto/ateapipb"
"google.golang.org/grpc/codes"
+ "google.golang.org/grpc/status"
+ "google.golang.org/protobuf/proto"
"google.golang.org/protobuf/testing/protocmp"
)
@@ -165,10 +169,7 @@ func TestTagActorSnapshot_Preconditions(t *testing.T) {
// under, as every real path to holding one does. Only the case that
// exercises the missing record leaves it false.
builtOnTemplate bool
- // paused adds a node-local snapshot above the external one, the
- // shape a pause leaves behind.
- paused bool
- wantCode codes.Code
+ wantCode codes.Code
}{
{
name: "the actor is not suspended",
@@ -188,14 +189,6 @@ func TestTagActorSnapshot_Preconditions(t *testing.T) {
withSnapshot: true,
wantCode: codes.Internal,
},
- {
- name: "the actor is paused above its external snapshot",
- state: ateapipb.ActorState_ACTOR_STATE_SUSPENDED,
- withSnapshot: true,
- builtOnTemplate: true,
- paused: true,
- wantCode: codes.FailedPrecondition,
- },
}
for _, tt := range tests {
@@ -223,10 +216,6 @@ func TestTagActorSnapshot_Preconditions(t *testing.T) {
s.Snapshots = []*ateapipb.Snapshot{
newDurableSnapshot(1, ateapipb.SnapshotOwner_SNAPSHOT_OWNER_ACTOR, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, tmplUID, uri.Name(), uri.String(), ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED),
}
- if tt.paused {
- s.LastAssignedGeneration = 2
- s.Snapshots = append(s.Snapshots, newPausedSnapshot(t, 2, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, tmplUID, "paused", "node-1"))
- }
})
}
@@ -298,6 +287,299 @@ func TestTagActorSnapshot_TagsNewestExternalSnapshot(t *testing.T) {
}
}
+// TestTagActorSnapshot_PromotesLocalSnapshot verifies that tagging a paused
+// actor whose snapshot is not uploaded yet has the atelet on the snapshot's
+// node upload it first, records the upload on the actor, and tags that
+// snapshot. A failed upload fails the tag without reserving its name and leaves
+// the actor as it was. A create that cannot succeed fails before the upload.
+func TestTagActorSnapshot_PromotesLocalSnapshot(t *testing.T) {
+ // reported is the state atelet answers with once it made snap durable.
+ reported := func(snap *ateapipb.Snapshot) *ateletpb.ActorSnapshotState {
+ report := &ateletpb.SnapshotReport{Snapshot: &ateletpb.Snapshot{
+ SnapshotUuid: snap.GetUuid(),
+ Generation: snap.GetGeneration(),
+ Fidelity: fidelityToAtelet(snap.GetFidelity()),
+ Object: &ateletpb.ObjectStorage{SnapshotUri: snap.GetDurableSnapshot().GetObject().GetSnapshotUri()},
+ }}
+ return &ateletpb.ActorSnapshotState{LatestLocal: report, LatestDurable: proto.CloneOf(report)}
+ }
+
+ tests := []struct {
+ name string
+ // olderDurable seeds a COMPLETED snapshot below the paused one.
+ olderDurable bool
+ // uploaded records the paused snapshot's durable copy COMPLETED.
+ uploaded bool
+ // nodeLost clears the paused snapshot's locality, as when its node
+ // was lost before the upload finished.
+ nodeLost bool
+ // nameTaken seeds a tag under the name the create asks for.
+ nameTaken bool
+ // badLocation points the template at an unusable storage location.
+ badLocation bool
+ // noTemplateUID records no template the paused snapshot was built
+ // under, a broken row.
+ noTemplateUID bool
+ // promoteState builds the state PromoteSnapshot answers with from the
+ // paused snapshot.
+ promoteState func(paused *ateapipb.Snapshot) *ateletpb.ActorSnapshotState
+ promoteErr error
+ wantPromote bool
+ wantCode codes.Code
+ }{
+ {
+ name: "the paused snapshot is above an older durable one",
+ olderDurable: true,
+ promoteState: reported,
+ wantPromote: true,
+ },
+ {
+ name: "the paused snapshot is the actor's only one",
+ promoteState: reported,
+ wantPromote: true,
+ },
+ {
+ name: "the paused snapshot's upload is already recorded",
+ uploaded: true,
+ },
+ {
+ name: "the paused snapshot's node was lost before its upload finished",
+ olderDurable: true,
+ nodeLost: true,
+ },
+ {
+ name: "the name is already taken",
+ nameTaken: true,
+ wantCode: codes.AlreadyExists,
+ },
+ {
+ name: "the storage location is unusable",
+ badLocation: true,
+ wantCode: codes.Internal,
+ },
+ {
+ name: "the paused snapshot records no template",
+ olderDurable: true,
+ noTemplateUID: true,
+ wantCode: codes.Internal,
+ },
+ {
+ name: "atelet is unavailable",
+ promoteErr: apierror.Unavailable("plugin: object store restarting"),
+ wantPromote: true,
+ wantCode: codes.Unavailable,
+ },
+ {
+ name: "the snapshot conflicts with atelet's newer state",
+ promoteErr: apierror.Aborted("stale snapshot generation"),
+ wantPromote: true,
+ wantCode: codes.Aborted,
+ },
+ {
+ name: "the paused snapshot is gone and was never uploaded",
+ promoteErr: apierror.NotFound("local snapshot is gone and no uploaded copy exists: %w", status.Error(codes.NotFound, "manifest not found")),
+ wantPromote: true,
+ wantCode: codes.FailedPrecondition,
+ },
+ {
+ name: "the paused snapshot's fidelity does not match",
+ promoteErr: apierror.FailedPrecondition("pause snapshot captured full; upload asks for memory"),
+ wantPromote: true,
+ wantCode: codes.FailedPrecondition,
+ },
+ {
+ name: "the upload fails with a plugin error atelet does not classify",
+ promoteErr: fmt.Errorf("while uploading snapshot files: %w", status.Error(codes.NotFound, "bucket not found")),
+ wantPromote: true,
+ wantCode: codes.Internal,
+ },
+ {
+ name: "atelet reports no durable snapshot",
+ promoteState: func(*ateapipb.Snapshot) *ateletpb.ActorSnapshotState { return &ateletpb.ActorSnapshotState{} },
+ wantPromote: true,
+ wantCode: codes.Internal,
+ },
+ {
+ name: "atelet reports another snapshot at the generation",
+ promoteState: func(paused *ateapipb.Snapshot) *ateletpb.ActorSnapshotState {
+ state := reported(paused)
+ state.LatestDurable.Snapshot.SnapshotUuid = "another-snapshot"
+ return state
+ },
+ wantPromote: true,
+ wantCode: codes.Internal,
+ },
+ {
+ name: "atelet reports the snapshot at another generation",
+ promoteState: func(paused *ateapipb.Snapshot) *ateletpb.ActorSnapshotState {
+ state := reported(paused)
+ state.LatestDurable.Snapshot.Generation = 1
+ return state
+ },
+ wantPromote: true,
+ wantCode: codes.Internal,
+ },
+ {
+ name: "atelet reports the snapshot uploaded to another URI",
+ promoteState: func(paused *ateapipb.Snapshot) *ateletpb.ActorSnapshotState {
+ state := reported(paused)
+ state.LatestDurable.Snapshot.Object.SnapshotUri += "-elsewhere"
+ return state
+ },
+ wantPromote: true,
+ wantCode: codes.Internal,
+ },
+ {
+ name: "atelet reports the snapshot without a fidelity",
+ promoteState: func(paused *ateapipb.Snapshot) *ateletpb.ActorSnapshotState {
+ state := reported(paused)
+ state.LatestDurable.Snapshot.Fidelity = ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_UNSPECIFIED
+ return state
+ },
+ wantPromote: true,
+ wantCode: codes.Internal,
+ },
+ }
+
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ ctx := context.Background()
+ persistence := newTestPersistence(t)
+ template := seedSubstrateTemplate(t, ctx, persistence, "sub-tmpl")
+ w, atelet, objects := newWireFinalizeWorkflow(t, persistence)
+ tmplUID := template.GetMetadata().GetUid()
+
+ actorRef := resources.ActorRef{Atespace: "team-a", Name: "actor-1"}
+ actor := storetest.MustCreateActor(t, ctx, persistence, &ateapipb.Actor{
+ Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name},
+ ActorTemplate: &ateapipb.ObjectRef{Atespace: "team-a", Name: "sub-tmpl"},
+ Status: &ateapipb.ActorStatus{State: ateapipb.ActorState_ACTOR_STATE_SUSPENDED, AssignedNode: "node-1"},
+ })
+ pausedURI := mustActorSnapshotURI(t, template, actor, "paused")
+ objects.PutSnapshot(t, pausedURI, "manifest.json", "memory.zst")
+ pausedStatus := ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_PENDING
+ if tt.uploaded {
+ pausedStatus = ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED
+ }
+ paused := newDurableSnapshot(2, ateapipb.SnapshotOwner_SNAPSHOT_OWNER_ACTOR, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, tmplUID, pausedURI.Name(), pausedURI.String(), pausedStatus)
+ if !tt.nodeLost {
+ paused.Locality = "node-1"
+ }
+ if tt.noTemplateUID {
+ paused.ActorTemplateUid = ""
+ }
+ actor = mustUpdateActorStatus(t, ctx, persistence, actor, func(s *ateapipb.ActorStatus) {
+ s.LastAssignedGeneration = 2
+ if tt.olderDurable {
+ olderURI := mustActorSnapshotURI(t, template, actor, "older")
+ objects.PutSnapshot(t, olderURI, "manifest.json")
+ s.Snapshots = append(s.Snapshots, newDurableSnapshot(1, ateapipb.SnapshotOwner_SNAPSHOT_OWNER_ACTOR, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, tmplUID, olderURI.Name(), olderURI.String(), ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED))
+ }
+ s.Snapshots = append(s.Snapshots, proto.CloneOf(paused))
+ })
+ tagRef := resources.TagRef{Atespace: "team-a", Name: "v1"}
+ var takenBy *ateapipb.Tag
+ if tt.nameTaken {
+ var err error
+ takenBy, err = persistence.CreateTag(ctx, &ateapipb.Tag{
+ Metadata: &ateapipb.ResourceMetadata{Atespace: tagRef.Atespace, Name: tagRef.Name},
+ Scope: ateapipb.TagScope_TAG_SCOPE_ATESPACE,
+ SourceActor: actorRef.ToObjectRef(),
+ Status: &ateapipb.TagStatus{ActorTemplateUid: tmplUID},
+ })
+ if err != nil {
+ t.Fatalf("CreateTag: %v", err)
+ }
+ }
+ if tt.badLocation {
+ _, err := persistence.UpdateActorTemplate(ctx, resources.ActorTemplateRefFromActorTemplate(template), store.PreconditionFrom(template), func(tmpl *ateapipb.ActorTemplate) error {
+ tmpl.SnapshotConfig.StorageLocation = "gs://ate-snapshots/team-a/?query"
+ return nil
+ })
+ if err != nil {
+ t.Fatalf("UpdateActorTemplate: %v", err)
+ }
+ }
+ if tt.promoteState != nil {
+ atelet.promoteState = tt.promoteState(paused)
+ }
+ atelet.promoteErr = tt.promoteErr
+
+ tag, err := w.TagActorSnapshot(ctx, tagToCreate(actorRef, "v1"))
+ if code := apierror.Code(err); code != tt.wantCode {
+ t.Fatalf("TagActorSnapshot error = %v (code %v), want code %v", err, code, tt.wantCode)
+ }
+
+ gotPromote := atelet.promoteRequest()
+ switch {
+ case !tt.wantPromote && gotPromote != nil:
+ t.Errorf("PromoteSnapshot request = %v, want none", gotPromote)
+ case tt.wantPromote:
+ wantPromote := &ateletpb.PromoteSnapshotRequest{
+ Atespace: "team-a",
+ ActorName: "actor-1",
+ ActorUid: actor.GetMetadata().GetUid(),
+ ActorTemplateAtespace: "team-a",
+ ActorTemplateName: "sub-tmpl",
+ Snapshot: &ateletpb.Snapshot{
+ SnapshotUuid: pausedURI.Name(),
+ Generation: 2,
+ Fidelity: fidelityToAtelet(ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY),
+ Object: &ateletpb.ObjectStorage{SnapshotUri: pausedURI.String()},
+ },
+ }
+ if diff := cmp.Diff(wantPromote, gotPromote, protocmp.Transform()); diff != "" {
+ t.Errorf("PromoteSnapshot request mismatch (-want +got):\n%s", diff)
+ }
+ }
+
+ stored, err := persistence.GetActor(ctx, actorRef)
+ if err != nil {
+ t.Fatalf("GetActor: %v", err)
+ }
+ if tt.wantCode != codes.OK {
+ if diff := cmp.Diff(actor.GetStatus(), stored.GetStatus(), protocmp.Transform()); diff != "" {
+ t.Errorf("a failed promote changed the actor's status (-want +got):\n%s", diff)
+ }
+ got, err := persistence.GetTag(ctx, tagRef)
+ switch {
+ case takenBy != nil:
+ if diff := cmp.Diff(takenBy, got, protocmp.Transform()); diff != "" {
+ t.Errorf("the failed create changed the tag holding the name (-want +got):\n%s", diff)
+ }
+ case !errors.Is(err, store.ErrNotFound):
+ t.Errorf("GetTag = %v, want ErrNotFound: the failed create reserved the name", err)
+ }
+ return
+ }
+
+ tagSnapshot := tagDurableSnapshotURI(tag, ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED)
+ if tt.nodeLost {
+ // Nothing to upload from: the actor is left as it was, and the
+ // tag copied the older durable snapshot.
+ if diff := cmp.Diff(actor.GetStatus(), stored.GetStatus(), protocmp.Transform()); diff != "" {
+ t.Errorf("the actor's status changed (-want +got):\n%s", diff)
+ }
+ if diff := cmp.Diff([]string{"manifest.json"}, objects.Snapshot(t, mustParseSnapshotURI(t, tagSnapshot))); diff != "" {
+ t.Errorf("the tag's external snapshot mismatch (-want +got):\n%s", diff)
+ }
+ return
+ }
+
+ // The paused snapshot is now durable and stays on its node.
+ wantPaused := proto.CloneOf(paused)
+ wantPaused.DurableSnapshot.Status = ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED
+ if diff := cmp.Diff(wantPaused, findSnapshotByGeneration(stored.GetStatus().GetSnapshots(), 2), protocmp.Transform()); diff != "" {
+ t.Errorf("the actor's latest snapshot mismatch (-want +got):\n%s", diff)
+ }
+ // The tag copied the paused snapshot, not the older one.
+ if diff := cmp.Diff([]string{"manifest.json", "memory.zst"}, objects.Snapshot(t, mustParseSnapshotURI(t, tagSnapshot))); diff != "" {
+ t.Errorf("the tag's external snapshot mismatch (-want +got):\n%s", diff)
+ }
+ })
+ }
+}
+
// TestTagActorSnapshot_RecreateAfterCopyFailure verifies what a create that
// dies mid-copy leaves behind, and how a client gets past it. The row survives
// as a pending tag naming the prefix the copy was writing into, so the objects
diff --git a/cmd/ateapi/internal/controlapi/workflow_template_delete.go b/cmd/ateapi/internal/controlapi/workflow_template_delete.go
index acdb37b0aa..2d5245f779 100644
--- a/cmd/ateapi/internal/controlapi/workflow_template_delete.go
+++ b/cmd/ateapi/internal/controlapi/workflow_template_delete.go
@@ -86,7 +86,7 @@ func (w *ActorWorkflow) ensureGoldenActorDeleted(ctx context.Context, goldenRef
ctx, done := stepSpan(ctx, "DeleteGoldenActor")
defer func() { err = done(err) }()
- if _, err := w.DeleteActor(ctx, goldenRef, true, store.DeletePreconditions{}); err != nil {
+ if _, err := w.DeleteActor(ctx, goldenRef, store.DeletePreconditions{}); err != nil {
if apierror.Code(err) == codes.NotFound {
markSkipped(ctx, "golden actor already deleted")
return nil
diff --git a/cmd/ateapi/internal/controlapi/workflow_test.go b/cmd/ateapi/internal/controlapi/workflow_test.go
index 87a09d79fd..fe24540cbd 100644
--- a/cmd/ateapi/internal/controlapi/workflow_test.go
+++ b/cmd/ateapi/internal/controlapi/workflow_test.go
@@ -88,7 +88,7 @@ func TestActorStateChangeRecords(t *testing.T) {
name: "delete marks the actor deleting",
seedState: ateapipb.ActorState_ACTOR_STATE_SUSPENDED,
transition: func(t *testing.T, w *ActorWorkflow, ref resources.ActorRef, actor *ateapipb.Actor, _ *ateapipb.ActorTemplate) {
- if _, err := w.ensureMarkedDeleting(context.Background(), ref, actor, false); err != nil {
+ if _, err := w.ensureMarkedDeleting(context.Background(), ref, actor); err != nil {
t.Fatalf("ensureMarkedDeleting: %v", err)
}
},
@@ -301,7 +301,7 @@ func TestActorStateChangeRecordSkippedOnConflict(t *testing.T) {
}
w := &ActorWorkflow{store: persistence}
- if _, err := w.ensureMarkedDeleting(ctx, actorRef, stale, false); err == nil {
+ if _, err := w.ensureMarkedDeleting(ctx, actorRef, stale); err == nil {
t.Fatal("ensureMarkedDeleting on a stale actor = nil, want a conflict")
}
if len(*records) != 0 {
diff --git a/cmd/ateapi/internal/controlapi/workflow_testutil_test.go b/cmd/ateapi/internal/controlapi/workflow_testutil_test.go
index 4e35ff9891..3520bf9229 100644
--- a/cmd/ateapi/internal/controlapi/workflow_testutil_test.go
+++ b/cmd/ateapi/internal/controlapi/workflow_testutil_test.go
@@ -58,7 +58,7 @@ func newTestActorWorkflow(t *testing.T, st store.Interface, tmplAtespace, tmplNa
}); err != nil && !errors.Is(err, store.ErrAlreadyExists) {
t.Fatalf("create test ActorTemplate: %v", err)
}
- return NewActorWorkflow(st, nil, nil, nil, nil, nil, nil, "", nil, time.Minute, objectstoreplugintest.ControlClient(objectstoretest.New()))
+ return NewActorWorkflow(st, nil, nil, nil, nil, "", nil, time.Minute, objectstoreplugintest.ControlClient(objectstoretest.New()))
}
// newFinalizeWorkflow builds an ActorWorkflow over persistence with an
diff --git a/cmd/ateapi/internal/controlapi/workflow_worker_crash.go b/cmd/ateapi/internal/controlapi/workflow_worker_crash.go
new file mode 100644
index 0000000000..6b426c0576
--- /dev/null
+++ b/cmd/ateapi/internal/controlapi/workflow_worker_crash.go
@@ -0,0 +1,59 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package controlapi
+
+import (
+ "context"
+ "errors"
+ "fmt"
+
+ "go.ate.dev/substrate/cmd/ateapi/internal/store"
+ "go.ate.dev/substrate/internal/apierror"
+ "go.ate.dev/substrate/internal/ateattr"
+ "go.ate.dev/substrate/internal/preview"
+ "go.ate.dev/substrate/internal/resources"
+ "go.ate.dev/substrate/pkg/proto/ateapipb"
+)
+
+// CrashActorReportedByWorker crashes a RUNNING actor whose sandbox the worker
+// hosting it saw die. It holds the actor's lease, so an operation in flight
+// keeps the actor and fails on its own if the sandbox is gone.
+func (w *ActorWorkflow) CrashActorReportedByWorker(ctx context.Context, actorRef resources.ActorRef, actorUID, workerName, message string) (*ateapipb.Actor, error) {
+ ctx = preview.ContextWithOptIn(ctx)
+ leaseCtx, lease, err := w.acquireActorLease(ctx, actorRef)
+ if err != nil {
+ return nil, err
+ }
+ defer lease.Close()
+
+ actor, err := w.store.GetActor(leaseCtx, actorRef)
+ if err != nil {
+ if errors.Is(err, store.ErrNotFound) {
+ return nil, apierror.NotFound("Actor %s not found", actorRef)
+ }
+ return nil, fmt.Errorf("while loading actor %s: %w", actorRef, err)
+ }
+ // rechecked under the lease, the caller's checks raced other operations
+ if actor.GetMetadata().GetUid() != actorUID || actor.GetStatus().GetWorkerAssignment().GetWorker().GetName() != workerName {
+ return nil, apierror.NotFound("Actor %s not found", actorRef)
+ }
+ if state := actor.GetStatus().GetState(); state != ateapipb.ActorState_ACTOR_STATE_RUNNING {
+ return nil, apierror.FailedPrecondition("actor %s is %s, not RUNNING", actorRef, state)
+ }
+ if err := crashActor(leaseCtx, w.store, actorRef, ateattr.OperationUnknown, message); err != nil {
+ return nil, err
+ }
+ return w.store.GetActor(leaseCtx, actorRef)
+}
diff --git a/cmd/ateapi/internal/controlapi/workflow_worker_crash_test.go b/cmd/ateapi/internal/controlapi/workflow_worker_crash_test.go
new file mode 100644
index 0000000000..af7e1aa218
--- /dev/null
+++ b/cmd/ateapi/internal/controlapi/workflow_worker_crash_test.go
@@ -0,0 +1,102 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package controlapi
+
+import (
+ "context"
+ "strings"
+ "testing"
+
+ "google.golang.org/grpc/codes"
+
+ "go.ate.dev/substrate/cmd/ateapi/internal/store"
+ "go.ate.dev/substrate/cmd/ateapi/internal/store/storetest"
+ "go.ate.dev/substrate/internal/apierror"
+ "go.ate.dev/substrate/internal/resources"
+ "go.ate.dev/substrate/pkg/proto/ateapipb"
+)
+
+func TestCrashActorReportedByWorker(t *testing.T) {
+ actorRef := resources.ActorRef{Atespace: "team-a", Name: "actor-1"}
+
+ tests := []struct {
+ name string
+ // mutate edits the seeded RUNNING actor before the report.
+ mutate func(*ateapipb.Actor)
+ // uid and worker the report names; empty means the real ones.
+ uid, worker string
+ wantCode codes.Code
+ }{
+ {name: "running actor is crashed", wantCode: codes.OK},
+ {name: "stale incarnation", uid: "not-the-uid", wantCode: codes.NotFound},
+ {name: "hosted by another worker", worker: "other-worker", wantCode: codes.NotFound},
+ {
+ name: "operation in flight",
+ mutate: func(a *ateapipb.Actor) { a.Status.State = ateapipb.ActorState_ACTOR_STATE_SUSPENDING },
+ wantCode: codes.FailedPrecondition,
+ },
+ }
+ for _, tc := range tests {
+ t.Run(tc.name, func(t *testing.T) {
+ ctx := context.Background()
+ st, cleanup := storetest.SetupTestStore(t)
+ defer cleanup()
+ seedActor(t, ctx, st, actorRef)
+ seedWorker(t, ctx, st, actorRef)
+ actor, err := st.GetActor(ctx, actorRef)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if tc.mutate != nil {
+ if actor, err = st.UpdateActor(ctx, actorRef, store.PreconditionFrom(actor), func(a *ateapipb.Actor) error {
+ tc.mutate(a)
+ return nil
+ }); err != nil {
+ t.Fatal(err)
+ }
+ }
+ uid, worker := tc.uid, tc.worker
+ if uid == "" {
+ uid = actor.GetMetadata().GetUid()
+ }
+ if worker == "" {
+ worker = "uid"
+ }
+
+ w := newMarkOnlyWorkflow(st)
+ got, err := w.CrashActorReportedByWorker(ctx, actorRef, uid, worker, "cloud-hypervisor exited")
+ if code := apierror.Code(err); code != tc.wantCode {
+ t.Fatalf("code = %v (err %v), want %v", code, err, tc.wantCode)
+ }
+ if tc.wantCode != codes.OK {
+ after, err := st.GetActor(ctx, actorRef)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if after.GetStatus().GetState() == ateapipb.ActorState_ACTOR_STATE_CRASHED {
+ t.Errorf("actor crashed by a refused report")
+ }
+ return
+ }
+ assertCrashed(t, ctx, st, actorRef)
+ if msg := got.GetStatus().GetCrash().GetMessage(); !strings.Contains(msg, "cloud-hypervisor exited") {
+ t.Errorf("crash message = %q, want it to carry the report", msg)
+ }
+ if a := firstAssignment(t, st, "uid"); a != nil {
+ t.Errorf("worker assignment = %v, want released", a)
+ }
+ })
+ }
+}
diff --git a/cmd/ateapi/internal/controlapi/workload_spec.go b/cmd/ateapi/internal/controlapi/workload_spec.go
index be06a1ec56..ac23395717 100644
--- a/cmd/ateapi/internal/controlapi/workload_spec.go
+++ b/cmd/ateapi/internal/controlapi/workload_spec.go
@@ -156,18 +156,18 @@ func appendExternalVolumes(workloadSpec *ateletpb.WorkloadSpec, template *ateapi
return nil
}
for _, vol := range template.GetVolumes() {
- if vol.GetExternalVolumeTemplate() != nil {
+ if vol.GetExternal().GetTemplate() != nil {
if !isVolumeMounted(vol.GetName(), template) {
continue
}
if actor == nil {
- return fmt.Errorf("actor is required when externalVolumeTemplate is present")
+ return fmt.Errorf("actor is required when an external volume is present")
}
var storageVolID string
var volType string
var volCtx map[string]string
- for _, dbVol := range actor.GetStatus().GetExternalVolumes() {
+ for _, dbVol := range actor.GetStatus().GetExternalVolumes().GetVolumes() {
if dbVol.GetName() == vol.GetName() {
storageVolID = dbVol.GetStorageVolumeId()
volType = dbVol.GetVolumeType()
diff --git a/cmd/ateapi/internal/controlapi/workload_spec_test.go b/cmd/ateapi/internal/controlapi/workload_spec_test.go
index 925a614201..e781297b7e 100644
--- a/cmd/ateapi/internal/controlapi/workload_spec_test.go
+++ b/cmd/ateapi/internal/controlapi/workload_spec_test.go
@@ -390,9 +390,9 @@ func TestAppendExternalVolumes(t *testing.T) {
Volumes: []*ateapipb.Volume{
{
Name: "vol-1",
- ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{
+ External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{
StorageClassName: "pd-standard",
- },
+ }},
},
{
Name: "vol-2",
@@ -400,9 +400,9 @@ func TestAppendExternalVolumes(t *testing.T) {
},
{
Name: "unmounted-vol",
- ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{
+ External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{
StorageClassName: "pd-standard",
- },
+ }},
},
},
}
@@ -413,14 +413,14 @@ func TestAppendExternalVolumes(t *testing.T) {
Name: "actor-123",
},
Status: &ateapipb.ActorStatus{
- ExternalVolumes: []*ateapipb.ExternalVolume{
+ ExternalVolumes: &ateapipb.ExternalVolumes{Volumes: []*ateapipb.ExternalVolumeStatus{
{
Name: "vol-1",
StorageVolumeId: "vol-gce-pd-123",
VolumeType: "pd-standard",
VolumeContext: map[string]string{"foo": "bar"},
},
- },
+ }},
},
}
@@ -470,7 +470,7 @@ func TestAppendExternalVolumes(t *testing.T) {
Atespace: "space-abc",
Name: "actor-123",
},
- Status: &ateapipb.ActorStatus{ExternalVolumes: []*ateapipb.ExternalVolume{}},
+ Status: &ateapipb.ActorStatus{ExternalVolumes: &ateapipb.ExternalVolumes{Volumes: []*ateapipb.ExternalVolumeStatus{}}},
}
if err := appendExternalVolumes(&ateletpb.WorkloadSpec{}, template, missingActor, nil); err == nil {
t.Errorf("appendExternalVolumes expected error for missing volume, got nil")
diff --git a/cmd/ateapi/internal/defaults/defaults.go b/cmd/ateapi/internal/defaults/defaults.go
index 73849186a5..21ef87abad 100644
--- a/cmd/ateapi/internal/defaults/defaults.go
+++ b/cmd/ateapi/internal/defaults/defaults.go
@@ -51,6 +51,10 @@ func applyActorTemplateDefaults(t *ateapipb.ActorTemplate) {
if t == nil {
return
}
+ // snapshot_config is optional; an omitted one takes every default.
+ if t.SnapshotConfig == nil {
+ t.SnapshotConfig = &ateapipb.SnapshotConfig{}
+ }
applySnapshotConfigDefaults(t.SnapshotConfig)
for _, c := range t.Containers {
applyContainerDefaults(c)
diff --git a/cmd/ateapi/internal/defaults/defaults_test.go b/cmd/ateapi/internal/defaults/defaults_test.go
index 9c82226087..84f281e6b3 100644
--- a/cmd/ateapi/internal/defaults/defaults_test.go
+++ b/cmd/ateapi/internal/defaults/defaults_test.go
@@ -29,6 +29,13 @@ func TestApply(t *testing.T) {
scopeFull = ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY
scopeData = ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES
)
+ // defaultSnapshotConfig is what an omitted snapshot_config defaults to.
+ defaultSnapshotConfig := func() *ateapipb.SnapshotConfig {
+ return &ateapipb.SnapshotConfig{
+ PreferredFidelity: scopeFull,
+ GoldenSnapshotConfig: &ateapipb.GoldenSnapshotConfig{Mode: ateapipb.GoldenSnapshotMode_GOLDEN_SNAPSHOT_MODE_ENABLED},
+ }
+ }
tests := []struct {
name string
in proto.Message
@@ -38,9 +45,9 @@ func TestApply(t *testing.T) {
in: (*ateapipb.ActorTemplate)(nil),
want: (*ateapipb.ActorTemplate)(nil),
}, {
- name: "missing snapshot_config is left for validation",
+ name: "missing snapshot_config gets every default",
in: &ateapipb.ActorTemplate{},
- want: &ateapipb.ActorTemplate{},
+ want: &ateapipb.ActorTemplate{SnapshotConfig: defaultSnapshotConfig()},
}, {
name: "empty snapshot_config gets every default",
in: &ateapipb.ActorTemplate{SnapshotConfig: &ateapipb.SnapshotConfig{}},
@@ -60,13 +67,13 @@ func TestApply(t *testing.T) {
}, {
name: "container without wakeup probe stays without one",
in: &ateapipb.ActorTemplate{Containers: []*ateapipb.Container{{Name: "main"}}},
- want: &ateapipb.ActorTemplate{Containers: []*ateapipb.Container{{Name: "main"}}},
+ want: &ateapipb.ActorTemplate{SnapshotConfig: defaultSnapshotConfig(), Containers: []*ateapipb.Container{{Name: "main"}}},
}, {
name: "wakeup probe gets timeout and path",
in: &ateapipb.ActorTemplate{Containers: []*ateapipb.Container{
{Name: "main", WakeupProbe: &ateapipb.WakeupProbe{HttpGet: &ateapipb.HttpGetAction{Port: 8080}}},
}},
- want: &ateapipb.ActorTemplate{Containers: []*ateapipb.Container{
+ want: &ateapipb.ActorTemplate{SnapshotConfig: defaultSnapshotConfig(), Containers: []*ateapipb.Container{
{Name: "main", WakeupProbe: &ateapipb.WakeupProbe{
HttpGet: &ateapipb.HttpGetAction{Port: 8080, Path: "/"},
TimeoutSeconds: 30,
@@ -77,7 +84,7 @@ func TestApply(t *testing.T) {
in: &ateapipb.ActorTemplate{Containers: []*ateapipb.Container{
{Name: "main", WakeupProbe: &ateapipb.WakeupProbe{}},
}},
- want: &ateapipb.ActorTemplate{Containers: []*ateapipb.Container{
+ want: &ateapipb.ActorTemplate{SnapshotConfig: defaultSnapshotConfig(), Containers: []*ateapipb.Container{
{Name: "main", WakeupProbe: &ateapipb.WakeupProbe{TimeoutSeconds: 30}},
}},
}, {
@@ -88,7 +95,7 @@ func TestApply(t *testing.T) {
TimeoutSeconds: 5,
}},
}},
- want: &ateapipb.ActorTemplate{Containers: []*ateapipb.Container{
+ want: &ateapipb.ActorTemplate{SnapshotConfig: defaultSnapshotConfig(), Containers: []*ateapipb.Container{
{Name: "main", WakeupProbe: &ateapipb.WakeupProbe{
HttpGet: &ateapipb.HttpGetAction{Port: 8080, Path: "/healthz"},
TimeoutSeconds: 5,
@@ -101,7 +108,7 @@ func TestApply(t *testing.T) {
{Name: "b"},
{Name: "c", WakeupProbe: &ateapipb.WakeupProbe{HttpGet: &ateapipb.HttpGetAction{Port: 3}}},
}},
- want: &ateapipb.ActorTemplate{Containers: []*ateapipb.Container{
+ want: &ateapipb.ActorTemplate{SnapshotConfig: defaultSnapshotConfig(), Containers: []*ateapipb.Container{
{Name: "a", WakeupProbe: &ateapipb.WakeupProbe{HttpGet: &ateapipb.HttpGetAction{Port: 1, Path: "/"}, TimeoutSeconds: 30}},
{Name: "b"},
{Name: "c", WakeupProbe: &ateapipb.WakeupProbe{HttpGet: &ateapipb.HttpGetAction{Port: 3, Path: "/"}, TimeoutSeconds: 30}},
diff --git a/cmd/ateapi/internal/scheduling/scheduling.go b/cmd/ateapi/internal/scheduling/scheduling.go
index 3bee872b17..b9930ac531 100644
--- a/cmd/ateapi/internal/scheduling/scheduling.go
+++ b/cmd/ateapi/internal/scheduling/scheduling.go
@@ -236,6 +236,14 @@ func (s *scheduler) Applies(worker *ateapipb.Worker, constraints Constraints) bo
return false
}
+ // The assignment records this runtime and atelet requires it on
+ // Run/Restore, so a worker that has not registered one cannot host
+ // anything yet (see ServiceImpl.CreateWorker: "Nothing is placed on it
+ // in the meantime").
+ if worker.GetStatus().GetDefaultRuntime() == nil {
+ return false
+ }
+
if len(constraints.RequiredNodes) > 0 && !slices.Contains(constraints.RequiredNodes, worker.GetNodeName()) {
return false
}
diff --git a/cmd/ateapi/internal/scheduling/scheduling_test.go b/cmd/ateapi/internal/scheduling/scheduling_test.go
index c42920eced..d279b2c8f7 100644
--- a/cmd/ateapi/internal/scheduling/scheduling_test.go
+++ b/cmd/ateapi/internal/scheduling/scheduling_test.go
@@ -54,6 +54,13 @@ func TestSchedule(t *testing.T) {
},
constraints: Constraints{},
},
+ {
+ name: "worker with no default runtime is never scheduled",
+ fleet: fleet{
+ worker("w-unregistered", "gvisor", "node-a", tierTwo, withRuntimes(nil)),
+ },
+ constraints: Constraints{},
+ },
{
name: "template selector filters",
fleet: fleet{
@@ -474,6 +481,12 @@ func TestApplies(t *testing.T) {
constraints: Constraints{},
want: false,
},
+ {
+ name: "skips worker with no default runtime",
+ worker: worker("w", "gvisor", "node-a", nil, withRuntimes(nil)),
+ constraints: Constraints{},
+ want: false,
+ },
{
name: "matches required runtime via default runtime",
worker: worker("w", "gvisor", "node-a", nil, withRuntimes(sandboxRuntime("gvisor", "amd64", "old"))),
@@ -510,13 +523,21 @@ type fleet []*ateapipb.Worker
func (f fleet) Workers() ([]*ateapipb.Worker, error) { return f, nil }
func worker(pod, class, node string, lbls map[string]string, opts ...func(*ateapipb.Worker)) *ateapipb.Worker {
+ var defRuntime *ateapipb.SandboxRuntime
+ if class != "" {
+ defRuntime = sandboxRuntime(class, "amd64")
+ }
w := &ateapipb.Worker{
WorkerPod: pod,
SandboxClass: class,
NodeName: node,
Labels: lbls,
// A stored Worker always carries a ceiling; CreateWorker reifies one.
- Status: &ateapipb.WorkerStatus{State: ateapipb.WorkerState_WORKER_STATE_ACTIVE, Capacity: &ateapipb.WorkerResources{Actors: 1}},
+ Status: &ateapipb.WorkerStatus{
+ State: ateapipb.WorkerState_WORKER_STATE_ACTIVE,
+ Capacity: &ateapipb.WorkerResources{Actors: 1},
+ DefaultRuntime: defRuntime,
+ },
}
for _, opt := range opts {
opt(w)
diff --git a/cmd/ateapi/internal/store/atepg/atepg.go b/cmd/ateapi/internal/store/atepg/atepg.go
index a84c7218ec..34ada16a68 100644
--- a/cmd/ateapi/internal/store/atepg/atepg.go
+++ b/cmd/ateapi/internal/store/atepg/atepg.go
@@ -38,6 +38,7 @@ import (
"go.ate.dev/substrate/cmd/ateapi/internal/defaults"
"go.ate.dev/substrate/cmd/ateapi/internal/store"
"go.ate.dev/substrate/pkg/proto/ateapipb"
+ "go.opentelemetry.io/otel"
"google.golang.org/protobuf/proto"
"google.golang.org/protobuf/types/known/timestamppb"
)
@@ -272,6 +273,9 @@ func poolConfig(dsn, role string) (*pgxpool.Config, error) {
}
return nil
}
+ // Per-statement trace spans; the watch and probe pools inherit this through
+ // resizedPoolConfig.
+ cfg.ConnConfig.Tracer = newQueryTracer(otel.GetTracerProvider(), cfg.ConnConfig)
usesTLS := cfg.ConnConfig.TLSConfig != nil
for _, fallback := range cfg.ConnConfig.Fallbacks {
usesTLS = usesTLS || fallback.TLSConfig != nil
diff --git a/cmd/ateapi/internal/store/atepg/tracing.go b/cmd/ateapi/internal/store/atepg/tracing.go
new file mode 100644
index 0000000000..666c95fe1f
--- /dev/null
+++ b/cmd/ateapi/internal/store/atepg/tracing.go
@@ -0,0 +1,180 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package atepg
+
+import (
+ "context"
+ "fmt"
+ "strings"
+
+ "github.com/jackc/pgx/v5"
+ "go.opentelemetry.io/otel/attribute"
+ "go.opentelemetry.io/otel/codes"
+ semconv "go.opentelemetry.io/otel/semconv/v1.40.0"
+ "go.opentelemetry.io/otel/trace"
+)
+
+// queryTracer is a pgx QueryTracer that opens one client span per statement,
+// so an RPC trace shows where its time went inside PostgreSQL. Statements are
+// parameterized ($1, $2, ...), so db.query.text carries no argument values.
+//
+// Spans follow the OpenTelemetry database semantic conventions: the span name
+// is the query summary ("SELECT actors", "UPDATE workers", "commit"), and the
+// connection-level attributes are computed once from the pool configuration.
+type queryTracer struct {
+ tracer trace.Tracer
+ // attrs holds db.system.name, server.address, server.port and
+ // db.namespace, which are the same for every statement on the pool.
+ attrs []attribute.KeyValue
+}
+
+var _ pgx.QueryTracer = (*queryTracer)(nil)
+
+// newQueryTracer builds the tracer for pools opened from cc. The tracer is
+// resolved once here rather than per statement.
+func newQueryTracer(tp trace.TracerProvider, cc *pgx.ConnConfig) *queryTracer {
+ return &queryTracer{
+ tracer: tp.Tracer("atepg"),
+ attrs: []attribute.KeyValue{
+ semconv.DBSystemNamePostgreSQL,
+ semconv.ServerAddress(cc.Host),
+ semconv.ServerPort(int(cc.Port)),
+ semconv.DBNamespace(cc.Database),
+ },
+ }
+}
+
+// querySpanKey marks a context whose statement span was opened by
+// TraceQueryStart, so TraceQueryEnd never ends a span it did not start.
+type querySpanKey struct{}
+
+func (t *queryTracer) TraceQueryStart(ctx context.Context, _ *pgx.Conn, data pgx.TraceQueryStartData) context.Context {
+ // Join sampled traces only. Statements issued from background work
+ // (outbox polling, lease maintenance) carry no span, and opening a root
+ // span for each would flood the backend with single-span traces. An
+ // unsampled parent is skipped too: every sampler serverboot installs is
+ // ParentBased, so its children could never be sampled, and skipping
+ // them saves the non-recording span and context wrap per statement.
+ if !trace.SpanContextFromContext(ctx).IsSampled() {
+ return ctx
+ }
+ operation, collection := querySummary(data.SQL)
+ attrs := make([]attribute.KeyValue, 0, len(t.attrs)+4)
+ attrs = append(attrs, t.attrs...)
+ attrs = append(attrs, semconv.DBQueryText(data.SQL))
+ name := "postgresql"
+ if operation != "" {
+ name = operation
+ attrs = append(attrs, semconv.DBOperationName(operation))
+ if collection != "" {
+ name += " " + collection
+ attrs = append(attrs, semconv.DBCollectionName(collection))
+ }
+ attrs = append(attrs, semconv.DBQuerySummary(name))
+ }
+ ctx, span := t.tracer.Start(ctx, name,
+ trace.WithSpanKind(trace.SpanKindClient),
+ trace.WithAttributes(attrs...))
+ return context.WithValue(ctx, querySpanKey{}, span)
+}
+
+func (*queryTracer) TraceQueryEnd(ctx context.Context, _ *pgx.Conn, data pgx.TraceQueryEndData) {
+ span, ok := ctx.Value(querySpanKey{}).(trace.Span)
+ if !ok {
+ return
+ }
+ // pgx.ErrNoRows never arrives here: pgx synthesizes it for the caller
+ // after the rows are closed, so a lookup miss ends the span cleanly.
+ if data.Err != nil {
+ span.RecordError(data.Err)
+ span.SetStatus(codes.Error, data.Err.Error())
+ if code := pgErrCode(data.Err); code != "" {
+ // SQLSTATE is both the server's status code and the most
+ // useful low-cardinality error class.
+ span.SetAttributes(semconv.DBResponseStatusCode(code), semconv.ErrorTypeKey.String(code))
+ } else {
+ span.SetAttributes(semconv.ErrorTypeKey.String(fmt.Sprintf("%T", data.Err)))
+ }
+ }
+ span.End()
+}
+
+// querySummary extracts the db.operation.name and db.collection.name of a
+// statement: the leading keyword, and the single table it acts on. Both are
+// returned as written, without case normalization, as the conventions ask.
+//
+// The store issues one hand-written statement per call, so a keyword scan is
+// enough: the table follows INTO for INSERT, UPDATE for UPDATE, TABLE for
+// LOCK, and the first FROM for SELECT and DELETE. A statement that reads
+// several tables (any JOIN), none (SELECT clock_timestamp()) or only a
+// subquery's (SELECT EXISTS(SELECT 1 FROM t)) has no collection, and DDL and
+// transaction control report only their keyword.
+func querySummary(sql string) (operation, collection string) {
+ fields := strings.Fields(sql)
+ if len(fields) == 0 {
+ return "", ""
+ }
+ operation = fields[0]
+ var marker string
+ switch strings.ToUpper(operation) {
+ case "SELECT", "DELETE":
+ marker = "FROM"
+ case "INSERT":
+ marker = "INTO"
+ case "LOCK":
+ marker = "TABLE"
+ case "UPDATE":
+ return operation, collectionToken(fields, 1)
+ default:
+ return operation, ""
+ }
+ at := -1
+ for i, f := range fields[1:] {
+ switch {
+ case strings.EqualFold(f, "JOIN"):
+ return operation, ""
+ case at < 0 && opensSubquery(f):
+ // A subquery opened before the marker owns the first FROM,
+ // as in SELECT EXISTS(SELECT 1 FROM t): the outer statement
+ // reads no table of its own.
+ return operation, ""
+ case at < 0 && strings.EqualFold(f, marker):
+ at = i + 2
+ }
+ }
+ if at < 0 {
+ return operation, ""
+ }
+ return operation, collectionToken(fields, at)
+}
+
+// opensSubquery reports whether a whitespace-delimited token starts a nested
+// SELECT: "SELECT", "(SELECT" or "EXISTS(SELECT".
+func opensSubquery(tok string) bool {
+ return strings.EqualFold(tok[strings.LastIndexByte(tok, '(')+1:], "SELECT")
+}
+
+// collectionToken returns fields[i] stripped of surrounding punctuation, or
+// "" when it is absent or is not an identifier (a subquery, for instance).
+func collectionToken(fields []string, i int) string {
+ if i >= len(fields) {
+ return ""
+ }
+ tok := strings.Trim(fields[i], "(),;")
+ if tok == "" || strings.EqualFold(tok, "SELECT") {
+ return ""
+ }
+ return tok
+}
diff --git a/cmd/ateapi/internal/store/atepg/tracing_test.go b/cmd/ateapi/internal/store/atepg/tracing_test.go
new file mode 100644
index 0000000000..ad5be15a82
--- /dev/null
+++ b/cmd/ateapi/internal/store/atepg/tracing_test.go
@@ -0,0 +1,345 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package atepg
+
+import (
+ "context"
+ "errors"
+ "slices"
+ "testing"
+
+ "github.com/jackc/pgx/v5"
+ "github.com/jackc/pgx/v5/pgconn"
+ "github.com/jackc/pgx/v5/pgxpool"
+ "go.opentelemetry.io/otel/attribute"
+ "go.opentelemetry.io/otel/codes"
+ sdktrace "go.opentelemetry.io/otel/sdk/trace"
+ "go.opentelemetry.io/otel/sdk/trace/tracetest"
+ semconv "go.opentelemetry.io/otel/semconv/v1.40.0"
+ "go.opentelemetry.io/otel/trace"
+)
+
+// newTestQueryTracer builds a queryTracer against a local recording provider,
+// never touching the global one, so these tests stay parallel-safe.
+func newTestQueryTracer(t *testing.T) (*queryTracer, *sdktrace.TracerProvider, *tracetest.SpanRecorder) {
+ t.Helper()
+ sr := tracetest.NewSpanRecorder()
+ tp := sdktrace.NewTracerProvider(sdktrace.WithSpanProcessor(sr))
+ t.Cleanup(func() { _ = tp.Shutdown(context.Background()) })
+ cc, err := pgx.ParseConfig("postgres://ate@db.example.internal:5433/atedb?sslmode=disable")
+ if err != nil {
+ t.Fatalf("ParseConfig: %v", err)
+ }
+ return newQueryTracer(tp, cc), tp, sr
+}
+
+func spanAttrs(s sdktrace.ReadOnlySpan) map[attribute.Key]attribute.Value {
+ m := make(map[attribute.Key]attribute.Value, len(s.Attributes()))
+ for _, kv := range s.Attributes() {
+ m[kv.Key] = kv.Value
+ }
+ return m
+}
+
+func findSpan(t *testing.T, sr *tracetest.SpanRecorder, name string) sdktrace.ReadOnlySpan {
+ t.Helper()
+ for _, s := range sr.Ended() {
+ if s.Name() == name {
+ return s
+ }
+ }
+ var names []string
+ for _, s := range sr.Ended() {
+ names = append(names, s.Name())
+ }
+ t.Fatalf("no %q span recorded; got %q", name, names)
+ return nil
+}
+
+func TestQueryTracerJoinsParentTrace(t *testing.T) {
+ t.Parallel()
+ qt, tp, sr := newTestQueryTracer(t)
+ ctx, parent := tp.Tracer("test").Start(context.Background(), "parent")
+
+ const sql = "SELECT id FROM actors WHERE name = $1"
+ qctx := qt.TraceQueryStart(ctx, nil, pgx.TraceQueryStartData{SQL: sql})
+ qt.TraceQueryEnd(qctx, nil, pgx.TraceQueryEndData{})
+ parent.End()
+
+ dbSpan := findSpan(t, sr, "SELECT actors")
+ if got, want := dbSpan.Parent().SpanID(), parent.SpanContext().SpanID(); got != want {
+ t.Errorf("db span parent = %s, want the RPC span %s", got, want)
+ }
+ if dbSpan.SpanKind() != trace.SpanKindClient {
+ t.Errorf("db span kind = %v, want client", dbSpan.SpanKind())
+ }
+ if dbSpan.Status().Code != codes.Unset {
+ t.Errorf("status = %v, want Unset", dbSpan.Status().Code)
+ }
+ attrs := spanAttrs(dbSpan)
+ for key, want := range map[attribute.Key]attribute.Value{
+ semconv.DBSystemNameKey: semconv.DBSystemNamePostgreSQL.Value,
+ semconv.ServerAddressKey: attribute.StringValue("db.example.internal"),
+ semconv.ServerPortKey: attribute.IntValue(5433),
+ semconv.DBNamespaceKey: attribute.StringValue("atedb"),
+ semconv.DBQueryTextKey: attribute.StringValue(sql),
+ semconv.DBOperationNameKey: attribute.StringValue("SELECT"),
+ semconv.DBCollectionNameKey: attribute.StringValue("actors"),
+ semconv.DBQuerySummaryKey: attribute.StringValue("SELECT actors"),
+ } {
+ if got, ok := attrs[key]; !ok {
+ t.Errorf("missing %s", key)
+ } else if got != want {
+ t.Errorf("%s = %v, want %v", key, got.String(), want.String())
+ }
+ }
+ for _, key := range []attribute.Key{semconv.ErrorTypeKey, semconv.DBResponseStatusCodeKey} {
+ if _, ok := attrs[key]; ok {
+ t.Errorf("successful statement must not carry %s", key)
+ }
+ }
+}
+
+func TestQueryTracerSkipsWithoutSampledParent(t *testing.T) {
+ t.Parallel()
+ qt, _, sr := newTestQueryTracer(t)
+
+ // No span at all: background work.
+ ctx := context.Background()
+ qctx := qt.TraceQueryStart(ctx, nil, pgx.TraceQueryStartData{SQL: "SELECT 1"})
+ if qctx != ctx {
+ t.Error("TraceQueryStart without a parent span must return the context unchanged")
+ }
+ qt.TraceQueryEnd(qctx, nil, pgx.TraceQueryEndData{})
+
+ // A parent that was not sampled: its children could never be sampled
+ // either, and ending the statement must leave the parent untouched.
+ unsampled := sdktrace.NewTracerProvider(sdktrace.WithSampler(sdktrace.NeverSample()), sdktrace.WithSpanProcessor(sr))
+ ctx, parent := unsampled.Tracer("test").Start(context.Background(), "parent")
+ if parent.SpanContext().IsSampled() {
+ t.Fatal("test setup: parent must not be sampled")
+ }
+ qctx = qt.TraceQueryStart(ctx, nil, pgx.TraceQueryStartData{SQL: "SELECT 1"})
+ if qctx != ctx {
+ t.Error("TraceQueryStart under an unsampled parent must return the context unchanged")
+ }
+ qt.TraceQueryEnd(qctx, nil, pgx.TraceQueryEndData{})
+ if n := len(sr.Ended()); n != 0 {
+ t.Errorf("unsampled statements recorded %d spans, want 0", n)
+ }
+}
+
+// recordOnlySampler records every span without sampling it, the one
+// decision that yields a parent that is recording yet not sampled.
+type recordOnlySampler struct{}
+
+func (recordOnlySampler) ShouldSample(sdktrace.SamplingParameters) sdktrace.SamplingResult {
+ return sdktrace.SamplingResult{Decision: sdktrace.RecordOnly}
+}
+
+func (recordOnlySampler) Description() string { return "RecordOnly" }
+
+func TestQueryTracerLeavesRecordOnlyParentOpen(t *testing.T) {
+ t.Parallel()
+ qt, _, sr := newTestQueryTracer(t)
+ recordOnly := sdktrace.NewTracerProvider(sdktrace.WithSampler(recordOnlySampler{}), sdktrace.WithSpanProcessor(sr))
+ ctx, parent := recordOnly.Tracer("test").Start(context.Background(), "parent")
+ if parent.SpanContext().IsSampled() || !parent.IsRecording() {
+ t.Fatal("test setup: parent must be recording but not sampled")
+ }
+
+ // The statement is skipped because the parent is not sampled. Ending it
+ // must not touch the parent, which is the span the context now carries.
+ qctx := qt.TraceQueryStart(ctx, nil, pgx.TraceQueryStartData{SQL: "SELECT 1"})
+ qt.TraceQueryEnd(qctx, nil, pgx.TraceQueryEndData{})
+
+ if !parent.IsRecording() {
+ t.Error("TraceQueryEnd ended the parent span it did not start")
+ }
+ if n := len(sr.Ended()); n != 0 {
+ t.Errorf("recorded %d ended spans before the parent finished, want 0", n)
+ }
+}
+
+func TestQueryTracerErrorStatus(t *testing.T) {
+ t.Parallel()
+ qt, tp, sr := newTestQueryTracer(t)
+ ctx, parent := tp.Tracer("test").Start(context.Background(), "parent")
+
+ // A PostgreSQL error carries its SQLSTATE as status code and error type.
+ pgErr := &pgconn.PgError{Code: "40P01", Message: "deadlock detected"}
+ qctx := qt.TraceQueryStart(ctx, nil, pgx.TraceQueryStartData{SQL: "UPDATE actors SET state = $1"})
+ qt.TraceQueryEnd(qctx, nil, pgx.TraceQueryEndData{Err: pgErr})
+
+ // Any other failure is typed by its Go type.
+ qctx = qt.TraceQueryStart(ctx, nil, pgx.TraceQueryStartData{SQL: "DELETE FROM workers WHERE name = $1"})
+ qt.TraceQueryEnd(qctx, nil, pgx.TraceQueryEndData{Err: context.DeadlineExceeded})
+ parent.End()
+
+ update := findSpan(t, sr, "UPDATE actors")
+ if update.Status().Code != codes.Error {
+ t.Errorf("failed statement status = %v, want Error", update.Status().Code)
+ }
+ attrs := spanAttrs(update)
+ if got := attrs[semconv.DBResponseStatusCodeKey].AsString(); got != "40P01" {
+ t.Errorf("db.response.status_code = %q, want 40P01", got)
+ }
+ if got := attrs[semconv.ErrorTypeKey].AsString(); got != "40P01" {
+ t.Errorf("error.type = %q, want 40P01", got)
+ }
+ if len(update.Events()) == 0 {
+ t.Error("failed statement recorded no exception event")
+ }
+
+ del := findSpan(t, sr, "DELETE workers")
+ if del.Status().Code != codes.Error {
+ t.Errorf("failed statement status = %v, want Error", del.Status().Code)
+ }
+ attrs = spanAttrs(del)
+ if _, ok := attrs[semconv.DBResponseStatusCodeKey]; ok {
+ t.Error("non-PostgreSQL error must not carry db.response.status_code")
+ }
+ if got := attrs[semconv.ErrorTypeKey].AsString(); got == "" || got == "40P01" {
+ t.Errorf("error.type = %q, want the Go error type", got)
+ }
+}
+
+func TestQuerySummary(t *testing.T) {
+ t.Parallel()
+ for _, tc := range []struct {
+ sql, operation, collection string
+ }{
+ // Statements as the store writes them.
+ {"SELECT proto FROM actors WHERE atespace = $1 AND name = $2", "SELECT", "actors"},
+ {"SELECT proto FROM workers WHERE name = $1 FOR UPDATE", "SELECT", "workers"},
+ {"\n\t\tSELECT atespace, name, proto\n\t\tFROM actor_templates\n\t\tWHERE atespace = $1\n", "SELECT", "actor_templates"},
+ {"INSERT INTO actors (atespace, name, uid, version, proto) VALUES ($1, $2, $3, $4, $5)", "INSERT", "actors"},
+ {"UPDATE actor_templates SET version = $1, proto = $2 WHERE atespace = $3", "UPDATE", "actor_templates"},
+ {"DELETE FROM actor_templates AS t WHERE t.atespace = $1 RETURNING t.proto", "DELETE", "actor_templates"},
+ {"DELETE FROM leases WHERE expires_at <= clock_timestamp()", "DELETE", "leases"},
+ {"INSERT INTO leases (key, token, expires_at) VALUES ($1, $2, clock_timestamp()) ON CONFLICT (key) DO UPDATE SET token = $2", "INSERT", "leases"},
+ {"INSERT INTO worker_outbox_trim (xid) SELECT xid FROM worker_outbox_default ORDER BY xid DESC LIMIT 1", "INSERT", "worker_outbox_trim"},
+ {"SELECT proto FROM actors WHERE name IN (SELECT name FROM actor_templates WHERE atespace = $1)", "SELECT", "actors"},
+ {"LOCK TABLE worker_outbox IN ACCESS EXCLUSIVE MODE", "LOCK", "worker_outbox"},
+ // No single table.
+ {"SELECT clock_timestamp()", "SELECT", ""},
+ {"SELECT pg_advisory_xact_lock(hashtextextended($1, 0))", "SELECT", ""},
+ {"SELECT child.relname FROM pg_inherits i JOIN pg_class child ON child.oid = i.inhrelid", "SELECT", ""},
+ {"SELECT count(*) FROM (SELECT 1) AS sub", "SELECT", ""},
+ {"SELECT EXISTS(SELECT 1 FROM worker_outbox_default)", "SELECT", ""},
+ {"SELECT (SELECT xid FROM worker_outbox_trim) >= $1::xid8", "SELECT", ""},
+ {"CREATE SCHEMA IF NOT EXISTS \"ate\"", "CREATE", ""},
+ {"DROP TABLE worker_outbox_p1", "DROP", ""},
+ // Transaction control as pgx issues it.
+ {"begin", "begin", ""},
+ {"begin isolation level serializable", "begin", ""},
+ {"commit", "commit", ""},
+ {"rollback", "rollback", ""},
+ {"", "", ""},
+ } {
+ op, coll := querySummary(tc.sql)
+ if op != tc.operation || coll != tc.collection {
+ t.Errorf("querySummary(%q) = (%q, %q), want (%q, %q)", tc.sql, op, coll, tc.operation, tc.collection)
+ }
+ }
+}
+
+func TestPoolConfigTracerSharedWithWatchPool(t *testing.T) {
+ t.Parallel()
+ cfg, err := poolConfig("postgres://postgres@localhost:5432/atepg?sslmode=disable", "atepg")
+ if err != nil {
+ t.Fatalf("poolConfig: %v", err)
+ }
+ qt, ok := cfg.ConnConfig.Tracer.(*queryTracer)
+ if !ok {
+ t.Fatalf("pool tracer = %T, want *queryTracer", cfg.ConnConfig.Tracer)
+ }
+ watch := resizedPoolConfig(cfg, watchPoolMaxConns, watchPoolMinConns)
+ if watch.ConnConfig.Tracer != qt {
+ t.Errorf("watch pool tracer = %v, want the same *queryTracer as the main pool", watch.ConnConfig.Tracer)
+ }
+}
+
+// TestQueryTracerAgainstPostgreSQL drives real statements through a traced
+// pool, so the span names, transaction control spans, lookup misses and
+// server errors are what pgx actually delivers rather than what the unit
+// tests feed the tracer by hand.
+func TestQueryTracerAgainstPostgreSQL(t *testing.T) {
+ requirePool(t)
+ ctx := context.Background()
+ sr := tracetest.NewSpanRecorder()
+ tp := sdktrace.NewTracerProvider(sdktrace.WithSpanProcessor(sr))
+ t.Cleanup(func() { _ = tp.Shutdown(ctx) })
+
+ cfg, err := poolConfig(containerDSN, "atepg")
+ if err != nil {
+ t.Fatalf("poolConfig: %v", err)
+ }
+ cfg.ConnConfig.Tracer = newQueryTracer(tp, cfg.ConnConfig)
+ pool, err := pgxpool.NewWithConfig(ctx, cfg)
+ if err != nil {
+ t.Fatalf("opening traced pool: %v", err)
+ }
+ t.Cleanup(pool.Close)
+ // Open the connection outside the rpc span, so the SET ROLE that
+ // AfterConnect issues on a new connection is not among the spans below.
+ if err := pool.Ping(ctx); err != nil {
+ t.Fatalf("Ping: %v", err)
+ }
+
+ ctx, parent := tp.Tracer("test").Start(ctx, "rpc")
+ tx, err := pool.Begin(ctx)
+ if err != nil {
+ t.Fatalf("Begin: %v", err)
+ }
+ var relname string
+ if err := tx.QueryRow(ctx, "SELECT relname FROM pg_class WHERE false").Scan(&relname); !errors.Is(err, pgx.ErrNoRows) {
+ t.Fatalf("lookup miss err = %v, want ErrNoRows", err)
+ }
+ if err := tx.Commit(ctx); err != nil {
+ t.Fatalf("Commit: %v", err)
+ }
+ if _, err := pool.Exec(ctx, "SELECT 1 FROM no_such_table"); err == nil {
+ t.Fatal("querying a missing table succeeded")
+ }
+ parent.End()
+
+ var names []string
+ for _, s := range sr.Ended() {
+ if s.Name() == "rpc" {
+ continue
+ }
+ names = append(names, s.Name())
+ if got, want := s.Parent().SpanID(), parent.SpanContext().SpanID(); got != want {
+ t.Errorf("%s parent = %s, want the rpc span %s", s.Name(), got, want)
+ }
+ }
+ want := []string{"begin", "SELECT pg_class", "commit", "SELECT no_such_table"}
+ if !slices.Equal(names, want) {
+ t.Fatalf("span names = %q, want %q", names, want)
+ }
+
+ miss := findSpan(t, sr, "SELECT pg_class")
+ if miss.Status().Code != codes.Unset {
+ t.Errorf("lookup miss status = %v, want Unset: pgx must not hand ErrNoRows to the tracer", miss.Status().Code)
+ }
+ failed := findSpan(t, sr, "SELECT no_such_table")
+ if failed.Status().Code != codes.Error {
+ t.Errorf("missing table status = %v, want Error", failed.Status().Code)
+ }
+ if got := spanAttrs(failed)[semconv.DBResponseStatusCodeKey].AsString(); got != "42P01" {
+ t.Errorf("db.response.status_code = %q, want 42P01 (undefined_table)", got)
+ }
+}
diff --git a/cmd/ateapi/internal/store/storecontract/contract.go b/cmd/ateapi/internal/store/storecontract/contract.go
index 847e7cc926..b5a8b8ce85 100644
--- a/cmd/ateapi/internal/store/storecontract/contract.go
+++ b/cmd/ateapi/internal/store/storecontract/contract.go
@@ -116,6 +116,12 @@ func newTestActorTemplate(atespace, name string) *ateapipb.ActorTemplate {
},
},
}},
+ // Fully defaulted, as the RPC layer stores it, so a read that
+ // backfills defaults returns what was created.
+ SnapshotConfig: &ateapipb.SnapshotConfig{
+ PreferredFidelity: ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY,
+ GoldenSnapshotConfig: &ateapipb.GoldenSnapshotConfig{Mode: ateapipb.GoldenSnapshotMode_GOLDEN_SNAPSHOT_MODE_ENABLED},
+ },
}
}
diff --git a/cmd/ateapi/internal/workerservice/certificate_test.go b/cmd/ateapi/internal/workerservice/certificate_test.go
index 357aa7f9a0..76f98a871e 100644
--- a/cmd/ateapi/internal/workerservice/certificate_test.go
+++ b/cmd/ateapi/internal/workerservice/certificate_test.go
@@ -127,7 +127,7 @@ func newTestCAPool(t *testing.T) *localca.ConcretePool {
func TestMintWorkerActorCertificate(t *testing.T) {
st, actor := newFakeStoreWithActor("team-a", "my-actor", "3b9f1e77-2c4d-4a80-91be-6d5c8f0a7e21")
caPool := newTestCAPool(t)
- s := New(st, &fakeSuspender{}, testAteletSPIFFEID, caPool, "", nil)
+ s := New(st, &fakeControl{}, testAteletSPIFFEID, caPool, "", nil)
csr, key := generateTestCSR(t)
@@ -180,7 +180,7 @@ func TestMintWorkerActorCertificate(t *testing.T) {
func TestMintWorkerActorCertificate_Errors(t *testing.T) {
st, actor := newFakeStoreWithActor("team-a", "my-actor", "3b9f1e77-2c4d-4a80-91be-6d5c8f0a7e21")
caPool := newTestCAPool(t)
- s := New(st, &fakeSuspender{}, testAteletSPIFFEID, caPool, "", nil)
+ s := New(st, &fakeControl{}, testAteletSPIFFEID, caPool, "", nil)
csr, _ := generateTestCSR(t)
authed := ateletauthtest.ContextWith(ateletauthtest.CertOn(t, "node-1"))
diff --git a/cmd/ateapi/internal/workerservice/crash.go b/cmd/ateapi/internal/workerservice/crash.go
new file mode 100644
index 0000000000..da4e0cfb59
--- /dev/null
+++ b/cmd/ateapi/internal/workerservice/crash.go
@@ -0,0 +1,55 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package workerservice
+
+import (
+ "context"
+ "log/slog"
+
+ "go.ate.dev/substrate/cmd/ateapi/internal/apivalidation"
+ "go.ate.dev/substrate/cmd/ateapi/internal/ateletauth"
+ "go.ate.dev/substrate/internal/resources"
+ "go.ate.dev/substrate/pkg/proto/ateapipb"
+)
+
+// ReportActorCrashed crashes a RUNNING Actor whose sandbox died, on behalf of
+// the Worker hosting it. The caller must be an atelet running on the Worker's
+// node.
+func (s *Server) ReportActorCrashed(ctx context.Context, req *ateapipb.ReportActorCrashedRequest) (*ateapipb.ReportActorCrashedResponse, error) {
+ caller, err := ateletauth.Authenticate(ctx, s.ateletSPIFFEID)
+ if err != nil {
+ return nil, err
+ }
+ if errs := apivalidation.ValidateReportActorCrashedRequest(ctx, req); len(errs) > 0 {
+ return nil, resources.ToAPIError(errs)
+ }
+
+ workerName := req.GetWorker().GetName()
+ actorRef := resources.ActorRefFromObjectRef(req.GetActor())
+ if err := s.authorizeHostedActor(ctx, caller, workerName, actorRef, req.GetActorUid()); err != nil {
+ return nil, err
+ }
+ actor, err := s.control.CrashActorReportedByWorker(ctx, actorRef, req.GetActorUid(), workerName, req.GetMessage())
+ if err != nil {
+ return nil, err
+ }
+ slog.InfoContext(ctx, "Worker reported an actor crash",
+ slog.String("worker", workerName),
+ slog.String("actor", actorRef.String()),
+ slog.String("actor_uid", req.GetActorUid()),
+ slog.String("node", caller.NodeName),
+ slog.String("message", req.GetMessage()))
+ return &ateapipb.ReportActorCrashedResponse{Actor: actor}, nil
+}
diff --git a/cmd/ateapi/internal/workerservice/crash_test.go b/cmd/ateapi/internal/workerservice/crash_test.go
new file mode 100644
index 0000000000..f279a2780a
--- /dev/null
+++ b/cmd/ateapi/internal/workerservice/crash_test.go
@@ -0,0 +1,102 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package workerservice
+
+import (
+ "context"
+ "strings"
+ "testing"
+
+ "github.com/google/go-cmp/cmp"
+ "google.golang.org/grpc/codes"
+
+ "go.ate.dev/substrate/cmd/ateapi/internal/ateletauth/ateletauthtest"
+ "go.ate.dev/substrate/cmd/ateapi/internal/store/storetest"
+ "go.ate.dev/substrate/internal/apierror"
+ "go.ate.dev/substrate/internal/resources"
+ "go.ate.dev/substrate/pkg/proto/ateapipb"
+)
+
+func crashRequest(actorUID string) *ateapipb.ReportActorCrashedRequest {
+ return &ateapipb.ReportActorCrashedRequest{
+ Worker: &ateapipb.ObjectRef{Name: testWorkerName},
+ Actor: &ateapipb.ObjectRef{Atespace: suspendAtespace, Name: suspendActor},
+ ActorUid: actorUID,
+ Message: "cloud-hypervisor exited: signal: killed",
+ }
+}
+
+func TestReportActorCrashed(t *testing.T) {
+ st, cleanup := storetest.SetupTestStore(t)
+ defer cleanup()
+ seedReportedWorker(t, st, testNode, &ateapipb.WorkerResources{Actors: 1})
+ actor := seedHostedActor(t, st, testWorkerName)
+ uid := actor.GetMetadata().GetUid()
+ control := &fakeControl{crashResp: actor}
+ s := New(st, control, testAteletSPIFFEID, nil, "", nil)
+
+ if _, err := s.ReportActorCrashed(ateletauthtest.ContextWith(ateletauthtest.CertOn(t, testNode)), crashRequest(uid)); err != nil {
+ t.Fatalf("ReportActorCrashed() failed: %v", err)
+ }
+ want := []crashCall{{
+ ActorRef: resources.ActorRef{Atespace: suspendAtespace, Name: suspendActor},
+ ActorUID: uid,
+ WorkerName: testWorkerName,
+ Message: "cloud-hypervisor exited: signal: killed",
+ }}
+ if diff := cmp.Diff(want, control.crashCalls); diff != "" {
+ t.Errorf("crash calls mismatch (-want +got):\n%s", diff)
+ }
+}
+
+func TestReportActorCrashed_Refused(t *testing.T) {
+ st, cleanup := storetest.SetupTestStore(t)
+ defer cleanup()
+ seedReportedWorker(t, st, testNode, &ateapipb.WorkerResources{Actors: 1})
+ actor := seedHostedActor(t, st, testWorkerName)
+ uid := actor.GetMetadata().GetUid()
+ authed := ateletauthtest.ContextWith(ateletauthtest.CertOn(t, testNode))
+
+ noMessage := crashRequest(uid)
+ noMessage.Message = ""
+ longMessage := crashRequest(uid)
+ longMessage.Message = strings.Repeat("x", 4097)
+
+ tests := []struct {
+ name string
+ ctx context.Context
+ req *ateapipb.ReportActorCrashedRequest
+ want codes.Code
+ }{
+ {"unauthenticated", ateletauthtest.ContextWith(nil), crashRequest(uid), codes.Unauthenticated},
+ {"worker on another node", ateletauthtest.ContextWith(ateletauthtest.CertOn(t, "some-other-node")), crashRequest(uid), codes.NotFound},
+ {"stale actor incarnation", authed, crashRequest("3b9f1e77-2c4d-4a80-91be-6d5c8f0a7e21"), codes.NotFound},
+ {"no message", authed, noMessage, codes.InvalidArgument},
+ {"message too long", authed, longMessage, codes.InvalidArgument},
+ }
+ for _, tc := range tests {
+ t.Run(tc.name, func(t *testing.T) {
+ control := &fakeControl{}
+ s := New(st, control, testAteletSPIFFEID, nil, "", nil)
+ _, err := s.ReportActorCrashed(tc.ctx, tc.req)
+ if got := apierror.Code(err); got != tc.want {
+ t.Fatalf("code = %v (err %v), want %v", got, err, tc.want)
+ }
+ if len(control.crashCalls) != 0 {
+ t.Errorf("refused report still reached the crash: %v", control.crashCalls)
+ }
+ })
+ }
+}
diff --git a/cmd/ateapi/internal/workerservice/jwt.go b/cmd/ateapi/internal/workerservice/jwt.go
index 4889c43c8c..977fbbcb7c 100644
--- a/cmd/ateapi/internal/workerservice/jwt.go
+++ b/cmd/ateapi/internal/workerservice/jwt.go
@@ -86,6 +86,7 @@ func (s *Server) MintWorkerActorJwt(ctx context.Context, req *ateapipb.MintWorke
Atespace: meta.GetAtespace(),
ActorName: meta.GetName(),
ActorUID: meta.GetUid(),
+ Node: caller.NodeName,
},
}
token, err := s.actorIDJWTPool.SignJWT(claims)
diff --git a/cmd/ateapi/internal/workerservice/jwt_test.go b/cmd/ateapi/internal/workerservice/jwt_test.go
index c27ad58871..66f2779bd1 100644
--- a/cmd/ateapi/internal/workerservice/jwt_test.go
+++ b/cmd/ateapi/internal/workerservice/jwt_test.go
@@ -65,7 +65,7 @@ func decodeJWTClaims(t *testing.T, token string) actoridjwt.WireClaims {
func TestMintWorkerActorJwt(t *testing.T) {
st, actor := newFakeStoreWithActor("team-a", "my-actor", "3b9f1e77-2c4d-4a80-91be-6d5c8f0a7e21")
- s := New(st, &fakeSuspender{}, testAteletSPIFFEID, nil, testActorJWTIssuer, newTestJWTPool(t))
+ s := New(st, &fakeControl{}, testAteletSPIFFEID, nil, testActorJWTIssuer, newTestJWTPool(t))
ctx := ateletauthtest.ContextWith(ateletauthtest.CertOn(t, "node-1"))
before := time.Now().Truncate(time.Second)
@@ -96,7 +96,7 @@ func TestMintWorkerActorJwt(t *testing.T) {
if !ok || kind != actoridjwt.KindWorkerForActor {
t.Fatalf("token kind = %v (ok %v), want KindWorkerForActor under worker-for-actor.ate.dev", kind, ok)
}
- if got, want := *substrate, (actoridjwt.WireSubstrateClaims{Atespace: "team-a", ActorName: "my-actor", ActorUID: actor.GetMetadata().GetUid()}); got != want {
+ if got, want := *substrate, (actoridjwt.WireSubstrateClaims{Atespace: "team-a", ActorName: "my-actor", ActorUID: actor.GetMetadata().GetUid(), Node: "node-1"}); got != want {
t.Errorf("worker-for-actor.ate.dev = %+v, want %+v", got, want)
}
@@ -139,7 +139,7 @@ func uploadPendingStatus(node string) *ateapipb.ActorStatus {
func TestMintWorkerActorJwt_Errors(t *testing.T) {
st, actor := newFakeStoreWithActor("team-a", "my-actor", "3b9f1e77-2c4d-4a80-91be-6d5c8f0a7e21")
- s := New(st, &fakeSuspender{}, testAteletSPIFFEID, nil, testActorJWTIssuer, newTestJWTPool(t))
+ s := New(st, &fakeControl{}, testAteletSPIFFEID, nil, testActorJWTIssuer, newTestJWTPool(t))
authed := ateletauthtest.ContextWith(ateletauthtest.CertOn(t, "node-1"))
valid := func() *ateapipb.MintWorkerActorJwtRequest {
@@ -455,7 +455,7 @@ func TestNodeRefusalLogNamesCredential(t *testing.T) {
st, actor := newFakeStoreWithActor("team-a", "my-actor", "3b9f1e77-2c4d-4a80-91be-6d5c8f0a7e21")
csr, _ := generateTestCSR(t)
- s := New(st, &fakeSuspender{}, testAteletSPIFFEID, newTestCAPool(t), testActorJWTIssuer, newTestJWTPool(t))
+ s := New(st, &fakeControl{}, testAteletSPIFFEID, newTestCAPool(t), testActorJWTIssuer, newTestJWTPool(t))
// The actor's worker runs on node-1.
otherNode := ateletauthtest.ContextWith(ateletauthtest.CertOn(t, "node-2"))
actorRef := &ateapipb.ObjectRef{Atespace: "team-a", Name: "my-actor"}
diff --git a/cmd/ateapi/internal/workerservice/register_test.go b/cmd/ateapi/internal/workerservice/register_test.go
index 2906277774..d6845a60c1 100644
--- a/cmd/ateapi/internal/workerservice/register_test.go
+++ b/cmd/ateapi/internal/workerservice/register_test.go
@@ -61,7 +61,7 @@ func setRequest(actors int32) *ateapipb.RegisterWorkerRequest {
func TestRegisterWorker(t *testing.T) {
st, cleanup := storetest.SetupTestStore(t)
defer cleanup()
- s := New(st, &fakeSuspender{}, testAteletSPIFFEID, nil, "", nil)
+ s := New(st, &fakeControl{}, testAteletSPIFFEID, nil, "", nil)
seedReportedWorker(t, st, testNode, &ateapipb.WorkerResources{Actors: 1, Resources: resources.CPUMemory(2000, 0)})
got, err := s.RegisterWorker(ateletauthtest.ContextWith(ateletauthtest.CertOn(t, testNode)), setRequest(4094))
@@ -85,7 +85,7 @@ func TestRegisterWorker(t *testing.T) {
func TestRegisterWorker_Runtimes(t *testing.T) {
st, cleanup := storetest.SetupTestStore(t)
defer cleanup()
- s := New(st, &fakeSuspender{}, testAteletSPIFFEID, nil, "", nil)
+ s := New(st, &fakeControl{}, testAteletSPIFFEID, nil, "", nil)
authed := ateletauthtest.ContextWith(ateletauthtest.CertOn(t, testNode))
gvisorAMD64, gvisorARM64 := testRuntime("gvisor", "amd64"), testRuntime("gvisor", "arm64")
@@ -220,7 +220,7 @@ func TestRegisterWorker_Runtimes(t *testing.T) {
func TestRegisterWorker_OtherNodeIsNotFound(t *testing.T) {
st, cleanup := storetest.SetupTestStore(t)
defer cleanup()
- s := New(st, &fakeSuspender{}, testAteletSPIFFEID, nil, "", nil)
+ s := New(st, &fakeControl{}, testAteletSPIFFEID, nil, "", nil)
seedReportedWorker(t, st, testNode, &ateapipb.WorkerResources{Actors: 1})
_, err := s.RegisterWorker(ateletauthtest.ContextWith(ateletauthtest.CertOn(t, "some-other-node")), setRequest(4094))
@@ -244,7 +244,7 @@ func TestRegisterWorker_OtherNodeIsNotFound(t *testing.T) {
func TestRegisterWorker_UnchangedDoesNotWrite(t *testing.T) {
st, cleanup := storetest.SetupTestStore(t)
defer cleanup()
- s := New(st, &fakeSuspender{}, testAteletSPIFFEID, nil, "", nil)
+ s := New(st, &fakeControl{}, testAteletSPIFFEID, nil, "", nil)
seeded := seedReportedWorker(t, st, testNode, &ateapipb.WorkerResources{Actors: 4094})
for range 3 {
@@ -267,7 +267,7 @@ func TestRegisterWorker_UnchangedDoesNotWrite(t *testing.T) {
func TestRegisterWorker_SortsAttributes(t *testing.T) {
st, cleanup := storetest.SetupTestStore(t)
defer cleanup()
- s := New(st, &fakeSuspender{}, testAteletSPIFFEID, nil, "", nil)
+ s := New(st, &fakeControl{}, testAteletSPIFFEID, nil, "", nil)
seedReportedWorker(t, st, testNode, &ateapipb.WorkerResources{Actors: 1})
authed := ateletauthtest.ContextWith(ateletauthtest.CertOn(t, testNode))
@@ -310,7 +310,7 @@ func TestRegisterWorker_SortsAttributes(t *testing.T) {
func TestRegisterWorker_Errors(t *testing.T) {
st, cleanup := storetest.SetupTestStore(t)
defer cleanup()
- s := New(st, &fakeSuspender{}, testAteletSPIFFEID, nil, "", nil)
+ s := New(st, &fakeControl{}, testAteletSPIFFEID, nil, "", nil)
seedReportedWorker(t, st, testNode, &ateapipb.WorkerResources{Actors: 1})
authed := ateletauthtest.ContextWith(ateletauthtest.CertOn(t, testNode))
@@ -355,7 +355,7 @@ func TestRegisterWorker_Errors(t *testing.T) {
func TestRegisterWorker_RejectsNonsense(t *testing.T) {
st, cleanup := storetest.SetupTestStore(t)
defer cleanup()
- s := New(st, &fakeSuspender{}, testAteletSPIFFEID, nil, "", nil)
+ s := New(st, &fakeControl{}, testAteletSPIFFEID, nil, "", nil)
seeded := seedReportedWorker(t, st, testNode, &ateapipb.WorkerResources{Actors: 4094})
authed := ateletauthtest.ContextWith(ateletauthtest.CertOn(t, testNode))
diff --git a/cmd/ateapi/internal/workerservice/report.go b/cmd/ateapi/internal/workerservice/report.go
index 15888f5b11..979bbe3cb5 100644
--- a/cmd/ateapi/internal/workerservice/report.go
+++ b/cmd/ateapi/internal/workerservice/report.go
@@ -36,7 +36,7 @@ func (s *Server) ReportActorSnapshotState(ctx context.Context, req *ateapipb.Rep
return nil, resources.ToAPIError(errs)
}
actorRef := resources.ActorRefFromObjectRef(req.GetActor())
- if err := s.suspender.ApplyActorSnapshotState(ctx, actorRef, req.GetActorUid(), caller.NodeName, req.GetState()); err != nil {
+ if err := s.control.ApplyActorSnapshotState(ctx, actorRef, req.GetActorUid(), caller.NodeName, req.GetState()); err != nil {
return nil, err
}
return &ateapipb.ReportActorSnapshotStateResponse{}, nil
diff --git a/cmd/ateapi/internal/workerservice/report_test.go b/cmd/ateapi/internal/workerservice/report_test.go
index e11447e7e3..c2f6c5204f 100644
--- a/cmd/ateapi/internal/workerservice/report_test.go
+++ b/cmd/ateapi/internal/workerservice/report_test.go
@@ -48,17 +48,17 @@ func reportRequest() *ateapipb.ReportActorSnapshotStateRequest {
// A report reaches the control plane with the node from the caller's
// certificate. The handler reads nothing from the store.
func TestReportActorSnapshotState(t *testing.T) {
- suspender := &fakeSuspender{}
- s := New(nil, suspender, testAteletSPIFFEID, nil, "", nil)
+ control := &fakeControl{}
+ s := New(nil, control, testAteletSPIFFEID, nil, "", nil)
req := reportRequest()
if _, err := s.ReportActorSnapshotState(ateletauthtest.ContextWith(ateletauthtest.CertOn(t, testNode)), req); err != nil {
t.Fatalf("ReportActorSnapshotState() failed: %v", err)
}
- if len(suspender.reports) != 1 {
- t.Fatalf("got %d applied reports, want 1", len(suspender.reports))
+ if len(control.reports) != 1 {
+ t.Fatalf("got %d applied reports, want 1", len(control.reports))
}
- got := suspender.reports[0]
+ got := control.reports[0]
if want := (resources.ActorRef{Atespace: suspendAtespace, Name: suspendActor}); got.actorRef != want {
t.Errorf("actorRef = %v, want %v", got.actorRef, want)
}
@@ -92,13 +92,13 @@ func TestReportActorSnapshotState_Refused(t *testing.T) {
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
- suspender := &fakeSuspender{reportErr: tt.controlErr}
- s := New(nil, suspender, testAteletSPIFFEID, nil, "", nil)
+ control := &fakeControl{reportErr: tt.controlErr}
+ s := New(nil, control, testAteletSPIFFEID, nil, "", nil)
_, err := s.ReportActorSnapshotState(tt.ctx, tt.req)
if got := apierror.Code(err); got != tt.want {
t.Fatalf("ReportActorSnapshotState() error = %v, want code %v", err, tt.want)
}
- if applied := len(suspender.reports) > 0; applied != tt.wantApplied {
+ if applied := len(control.reports) > 0; applied != tt.wantApplied {
t.Errorf("report reached the control plane = %v, want %v", applied, tt.wantApplied)
}
})
diff --git a/cmd/ateapi/internal/workerservice/service.go b/cmd/ateapi/internal/workerservice/service.go
index 09ae137522..f295fbf86d 100644
--- a/cmd/ateapi/internal/workerservice/service.go
+++ b/cmd/ateapi/internal/workerservice/service.go
@@ -34,10 +34,10 @@ type Server struct {
// state every request is authorized against.
store store.Interface
- // suspender runs the suspend a Worker asks for. It is the same entry point
- // Control.SuspendActor uses, so a Worker's request is subject to every
+ // control runs the operations a Worker asks for through the same entry
+ // points Control uses, so a Worker's request is subject to every
// precondition a client's would be.
- suspender actorSuspender
+ control actorControl
// ateletSPIFFEID is the identity the calling atelet must present.
ateletSPIFFEID string
@@ -51,19 +51,20 @@ type Server struct {
actorIDJWTPool localjwtauthority.Pool
}
-// actorSuspender is the in-process slice of the Control service this package
+// actorControl is the in-process slice of the Control service this package
// drives. *controlapi.RPCService satisfies it.
-type actorSuspender interface {
+type actorControl interface {
SuspendActor(ctx context.Context, req *ateapipb.SuspendActorRequest) (*ateapipb.SuspendActorResponse, error)
ApplyActorSnapshotState(ctx context.Context, actorRef resources.ActorRef, actorUID, node string, state *ateapipb.ActorSnapshotState) error
+ CrashActorReportedByWorker(ctx context.Context, actorRef resources.ActorRef, actorUID, workerName, message string) (*ateapipb.Actor, error)
}
var _ ateapipb.WorkerServiceServer = (*Server)(nil)
-func New(store store.Interface, suspender actorSuspender, ateletSPIFFEID string, actorIDCAPool localca.Pool, actorJWTIssuer string, actorIDJWTPool localjwtauthority.Pool) *Server {
+func New(store store.Interface, control actorControl, ateletSPIFFEID string, actorIDCAPool localca.Pool, actorJWTIssuer string, actorIDJWTPool localjwtauthority.Pool) *Server {
return &Server{
store: store,
- suspender: suspender,
+ control: control,
ateletSPIFFEID: ateletSPIFFEID,
actorIDCAPool: actorIDCAPool,
actorJWTIssuer: actorJWTIssuer,
diff --git a/cmd/ateapi/internal/workerservice/service_test.go b/cmd/ateapi/internal/workerservice/service_test.go
index 0c29c10103..56d72a0a3c 100644
--- a/cmd/ateapi/internal/workerservice/service_test.go
+++ b/cmd/ateapi/internal/workerservice/service_test.go
@@ -35,24 +35,26 @@ const (
// certificates from ateletauthtest carry.
var testAteletSPIFFEID = installdefaults.SPIFFEID(installdefaults.SystemNamespace, installdefaults.AteletServiceAccount)
-// fakeSuspender stands in for the Control service. What it records is the
-// point of most of the suspend tests: whether the request reached the suspend
-// at all.
-type fakeSuspender struct {
+// fakeControl stands in for the Control service. What it records is the
+// point of most of the tests: whether the request reached the operation at
+// all.
+type fakeControl struct {
calls []*ateapipb.SuspendActorRequest
resp *ateapipb.SuspendActorResponse
err error
- reports []appliedReport
- reportErr error
+ reports []appliedReport
+ reportErr error
+ crashCalls []crashCall
+ crashResp *ateapipb.Actor
}
-func (f *fakeSuspender) SuspendActor(_ context.Context, req *ateapipb.SuspendActorRequest) (*ateapipb.SuspendActorResponse, error) {
+func (f *fakeControl) SuspendActor(_ context.Context, req *ateapipb.SuspendActorRequest) (*ateapipb.SuspendActorResponse, error) {
f.calls = append(f.calls, req)
return f.resp, f.err
}
-func (f *fakeSuspender) ApplyActorSnapshotState(_ context.Context, actorRef resources.ActorRef, actorUID, node string, state *ateapipb.ActorSnapshotState) error {
+func (f *fakeControl) ApplyActorSnapshotState(_ context.Context, actorRef resources.ActorRef, actorUID, node string, state *ateapipb.ActorSnapshotState) error {
f.reports = append(f.reports, appliedReport{actorRef: actorRef, actorUID: actorUID, node: node, state: state})
return f.reportErr
}
@@ -65,6 +67,17 @@ type appliedReport struct {
state *ateapipb.ActorSnapshotState
}
+// crashCall is one CrashActorReportedByWorker call.
+type crashCall struct {
+ ActorRef resources.ActorRef
+ ActorUID, WorkerName, Message string
+}
+
+func (f *fakeControl) CrashActorReportedByWorker(_ context.Context, actorRef resources.ActorRef, actorUID, workerName, message string) (*ateapipb.Actor, error) {
+ f.crashCalls = append(f.crashCalls, crashCall{actorRef, actorUID, workerName, message})
+ return f.crashResp, f.err
+}
+
// seedReportedWorker registers a Worker on nodeName that has already reported
// capacity and its default runtime: identity from the pod, the way the syncer
// writes it, plus the result of an earlier registration. A Worker that has
diff --git a/cmd/ateapi/internal/workerservice/suspend.go b/cmd/ateapi/internal/workerservice/suspend.go
index c59d0d9622..6035fa29e2 100644
--- a/cmd/ateapi/internal/workerservice/suspend.go
+++ b/cmd/ateapi/internal/workerservice/suspend.go
@@ -41,34 +41,7 @@ func (s *Server) RequestActorSuspend(ctx context.Context, req *ateapipb.RequestA
workerName := req.GetWorker().GetName()
actorRef := resources.ActorRefFromObjectRef(req.GetActor())
-
- // Use authoritative state to authorize the request, never the request
- // itself: the caller proves which node it is on, and nothing else.
- worker, err := s.store.GetWorker(ctx, workerName)
- if err != nil {
- if errors.Is(err, store.ErrNotFound) {
- return nil, apierror.NotFound("Worker %s not found", workerName)
- }
- return nil, fmt.Errorf("while fetching worker %s: %w", workerName, err)
- }
- if worker.GetNodeName() != caller.NodeName {
- // Do not disclose Workers on other nodes.
- slog.WarnContext(ctx, "Refusing a suspend request for a worker on another node",
- slog.String("worker", workerName),
- slog.String("worker_node", worker.GetNodeName()),
- slog.String("caller_node", caller.NodeName),
- slog.String("caller_pod", caller.PodName))
- return nil, apierror.NotFound("Worker %s not found", workerName)
- }
-
- actor, err := s.store.GetActor(ctx, actorRef)
- if err != nil {
- if errors.Is(err, store.ErrNotFound) {
- return nil, apierror.NotFound("Actor %s not found", actorRef)
- }
- return nil, fmt.Errorf("while fetching actor %s: %w", actorRef, err)
- }
- if err := checkActorHostedBy(ctx, actor, workerName, req.GetActorUid()); err != nil {
+ if err := s.authorizeHostedActor(ctx, caller, workerName, actorRef, req.GetActorUid()); err != nil {
return nil, err
}
@@ -82,7 +55,7 @@ func (s *Server) RequestActorSuspend(ctx context.Context, req *ateapipb.RequestA
// the workflow to acquire the lease, the Actor can be resumed onto another
// Worker, or deleted and recreated under the same name, and the workflow
// suspends whatever the reference resolves to by then. See #1773.
- resp, err := s.suspender.SuspendActor(ctx, &ateapipb.SuspendActorRequest{Actor: req.GetActor()})
+ resp, err := s.control.SuspendActor(ctx, &ateapipb.SuspendActorRequest{Actor: req.GetActor()})
if err != nil {
return nil, err
}
@@ -94,6 +67,38 @@ func (s *Server) RequestActorSuspend(ctx context.Context, req *ateapipb.RequestA
return &ateapipb.RequestActorSuspendResponse{Actor: resp.GetActor()}, nil
}
+// authorizeHostedActor checks that the calling atelet may speak for actorRef:
+// workerName must be on the caller's node, and the actor must be assigned to it.
+func (s *Server) authorizeHostedActor(ctx context.Context, caller *ateletauth.Caller, workerName string, actorRef resources.ActorRef, actorUID string) error {
+ // Use authoritative state to authorize the request, never the request
+ // itself: the caller proves which node it is on, and nothing else.
+ worker, err := s.store.GetWorker(ctx, workerName)
+ if err != nil {
+ if errors.Is(err, store.ErrNotFound) {
+ return apierror.NotFound("Worker %s not found", workerName)
+ }
+ return fmt.Errorf("while fetching worker %s: %w", workerName, err)
+ }
+ if worker.GetNodeName() != caller.NodeName {
+ // Do not disclose Workers on other nodes.
+ slog.WarnContext(ctx, "Refusing a request for a worker on another node",
+ slog.String("worker", workerName),
+ slog.String("worker_node", worker.GetNodeName()),
+ slog.String("caller_node", caller.NodeName),
+ slog.String("caller_pod", caller.PodName))
+ return apierror.NotFound("Worker %s not found", workerName)
+ }
+
+ actor, err := s.store.GetActor(ctx, actorRef)
+ if err != nil {
+ if errors.Is(err, store.ErrNotFound) {
+ return apierror.NotFound("Actor %s not found", actorRef)
+ }
+ return fmt.Errorf("while fetching actor %s: %w", actorRef, err)
+ }
+ return checkActorHostedBy(ctx, actor, workerName, actorUID)
+}
+
// checkActorHostedBy reports whether the Worker may speak for this Actor: the
// Actor must be assigned to it, and must be the incarnation the Worker thinks
// it is hosting.
@@ -104,9 +109,9 @@ func (s *Server) RequestActorSuspend(ctx context.Context, req *ateapipb.RequestA
func checkActorHostedBy(ctx context.Context, actor *ateapipb.Actor, workerName, actorUID string) error {
actorRef := resources.ActorRefFromActor(actor)
if got := actor.GetMetadata().GetUid(); got != actorUID {
- // The named Actor was deleted and recreated. Suspending the recreation
- // would checkpoint a workload the caller never hosted.
- slog.WarnContext(ctx, "Refusing a suspend request naming a stale actor incarnation",
+ // The named Actor was deleted and recreated, and the caller never
+ // hosted the recreation.
+ slog.WarnContext(ctx, "Refusing a request naming a stale actor incarnation",
slog.String("worker", workerName),
slog.String("actor", actorRef.String()),
slog.String("requested_uid", actorUID),
@@ -116,7 +121,7 @@ func checkActorHostedBy(ctx context.Context, actor *ateapipb.Actor, workerName,
// An Actor with no assignment has no worker to speak for it: it is
// SUSPENDED (paused or not) or CRASHED, and nothing hosts it.
if assigned := actor.GetStatus().GetWorkerAssignment().GetWorker().GetName(); assigned != workerName {
- slog.WarnContext(ctx, "Refusing a suspend request for an actor hosted elsewhere",
+ slog.WarnContext(ctx, "Refusing a request for an actor hosted elsewhere",
slog.String("worker", workerName),
slog.String("actor", actorRef.String()),
slog.String("assigned_worker", assigned))
diff --git a/cmd/ateapi/internal/workerservice/suspend_test.go b/cmd/ateapi/internal/workerservice/suspend_test.go
index 426a406274..1599627c55 100644
--- a/cmd/ateapi/internal/workerservice/suspend_test.go
+++ b/cmd/ateapi/internal/workerservice/suspend_test.go
@@ -77,8 +77,8 @@ func TestRequestActorSuspend(t *testing.T) {
Metadata: actor.GetMetadata(),
Status: &ateapipb.ActorStatus{State: ateapipb.ActorState_ACTOR_STATE_SUSPENDED},
}
- suspender := &fakeSuspender{resp: &ateapipb.SuspendActorResponse{Actor: suspended}}
- s := New(st, suspender, testAteletSPIFFEID, nil, "", nil)
+ control := &fakeControl{resp: &ateapipb.SuspendActorResponse{Actor: suspended}}
+ s := New(st, control, testAteletSPIFFEID, nil, "", nil)
got, err := s.RequestActorSuspend(
ateletauthtest.ContextWith(ateletauthtest.CertOn(t, testNode)),
@@ -89,7 +89,7 @@ func TestRequestActorSuspend(t *testing.T) {
want := []*ateapipb.SuspendActorRequest{{
Actor: &ateapipb.ObjectRef{Atespace: suspendAtespace, Name: suspendActor},
}}
- if diff := cmp.Diff(want, suspender.calls, protocmp.Transform()); diff != "" {
+ if diff := cmp.Diff(want, control.calls, protocmp.Transform()); diff != "" {
t.Errorf("suspend calls mismatch (-want +got):\n%s", diff)
}
// The caller gets the actor as the workflow left it, so a worker can see
@@ -125,8 +125,8 @@ func TestRequestActorSuspend_OnlyForHostedActors(t *testing.T) {
defer cleanup()
seedReportedWorker(t, st, testNode, &ateapipb.WorkerResources{Actors: 1})
actor := seedHostedActor(t, st, tc.assignedWorker)
- suspender := &fakeSuspender{}
- s := New(st, suspender, testAteletSPIFFEID, nil, "", nil)
+ control := &fakeControl{}
+ s := New(st, control, testAteletSPIFFEID, nil, "", nil)
uid := tc.requestedUID
if uid == "" {
@@ -138,8 +138,8 @@ func TestRequestActorSuspend_OnlyForHostedActors(t *testing.T) {
if got := apierror.Code(err); got != codes.NotFound {
t.Fatalf("code = %v (err %v), want NotFound", got, err)
}
- if len(suspender.calls) != 0 {
- t.Errorf("refused request still reached the suspend: %v", suspender.calls)
+ if len(control.calls) != 0 {
+ t.Errorf("refused request still reached the suspend: %v", control.calls)
}
})
}
@@ -151,8 +151,8 @@ func TestRequestActorSuspend_Errors(t *testing.T) {
seedReportedWorker(t, st, testNode, &ateapipb.WorkerResources{Actors: 1})
actor := seedHostedActor(t, st, testWorkerName)
uid := actor.GetMetadata().GetUid()
- suspender := &fakeSuspender{}
- s := New(st, suspender, testAteletSPIFFEID, nil, "", nil)
+ control := &fakeControl{}
+ s := New(st, control, testAteletSPIFFEID, nil, "", nil)
authed := ateletauthtest.ContextWith(ateletauthtest.CertOn(t, testNode))
tests := []struct {
@@ -203,8 +203,8 @@ func TestRequestActorSuspend_Errors(t *testing.T) {
}
})
}
- if len(suspender.calls) != 0 {
- t.Errorf("a malformed or unauthorized request reached the suspend: %v", suspender.calls)
+ if len(control.calls) != 0 {
+ t.Errorf("a malformed or unauthorized request reached the suspend: %v", control.calls)
}
}
@@ -216,8 +216,8 @@ func TestRequestActorSuspend_PassesThroughTheWorkflowsRefusal(t *testing.T) {
defer cleanup()
seedReportedWorker(t, st, testNode, &ateapipb.WorkerResources{Actors: 1})
actor := seedHostedActor(t, st, testWorkerName)
- suspender := &fakeSuspender{err: apierror.FailedPrecondition("Actor is RESUMING")}
- s := New(st, suspender, testAteletSPIFFEID, nil, "", nil)
+ control := &fakeControl{err: apierror.FailedPrecondition("Actor is RESUMING")}
+ s := New(st, control, testAteletSPIFFEID, nil, "", nil)
_, err := s.RequestActorSuspend(
ateletauthtest.ContextWith(ateletauthtest.CertOn(t, testNode)),
diff --git a/cmd/ateapi/main.go b/cmd/ateapi/main.go
index 5fec04b9db..4638889a18 100644
--- a/cmd/ateapi/main.go
+++ b/cmd/ateapi/main.go
@@ -111,6 +111,7 @@ var (
drainTimeout = pflag.Duration("drain-timeout", 15*time.Second, "Deadline for the graceful gRPC drain on shutdown. In-flight RPCs still running past it are forcefully cancelled.")
objectStorePluginSocket = pflag.String("objectstore-plugin-socket", "/run/objectstore-plugin/control.sock", "Unix socket of the control object-store plugin that external snapshots are deleted and copied through.")
+ snapshotLocation = pflag.String("snapshot-location", "", "Base object-storage URI (for example gs://BUCKET/PREFIX/) that actor snapshots are stored under. Required unless the PerTemplateSnapshotLocation preview gate is enabled; with the gate, an actor template's snapshotConfig.storageLocation overrides it.")
templateResyncInterval = pflag.Duration("template-resync-interval", 20*time.Second, fmt.Sprintf("Interval between actor template resyncs. Must be at least %s.", minResyncInterval))
actorWorkflowDeadline = pflag.Duration("actor-workflow-deadline", 5*time.Minute, "Maximum wall-clock duration of a single Resume/Suspend workflow; raise it for slow image registries.")
@@ -169,6 +170,13 @@ func main() {
if errs := resources.ValidateHostPort(field.NewPath("default-egress-gateway-address"), *defaultEgressGatewayAddress); len(errs) > 0 {
serverboot.Fatal(ctx, "Invalid --default-egress-gateway-address", errs.ToAggregate())
}
+ if *snapshotLocation != "" {
+ if err := resources.ValidateSnapshotLocation(*snapshotLocation); err != nil {
+ serverboot.Fatal(ctx, "Invalid --snapshot-location", err)
+ }
+ } else if !preview.IsEnabled(preview.GatePerTemplateSnapshotLocation) {
+ serverboot.Fatal(ctx, "Missing --snapshot-location", fmt.Errorf("required unless the %s preview gate is enabled", preview.GatePerTemplateSnapshotLocation))
+ }
// Kept separate from ctx so that in-progress work (clients, informers) is
// not cancelled the moment SIGTERM arrives. The drainOnShutdown
@@ -280,7 +288,6 @@ func main() {
ateFactory := externalversions.NewSharedInformerFactory(ateClient, 0)
workerPoolLister := ateFactory.Api().V1alpha1().WorkerPools().Lister()
- sandboxConfigLister := ateFactory.Api().V1alpha1().SandboxConfigs().Lister()
csiDriverConfigLister := ateFactory.Api().V1alpha1().CSIDriverConfigs().Lister()
ateletPodInformerFactory, ateletPodInformer := controlapi.AteletInformer(clientset, ateletNamespace)
@@ -343,8 +350,6 @@ func main() {
controlSrv := controlapi.NewRPCService(
persistence,
workerCache,
- workerPoolLister,
- sandboxConfigLister,
csiDriverConfigLister,
storageClassLister,
ateletDialer,
@@ -356,6 +361,7 @@ func main() {
*actorJWTIssuer,
actorIDJWTAuthorityPool,
actorIDCAPool,
+ *snapshotLocation,
)
// Drive stored ActorTemplates through the golden actor flow.
@@ -462,6 +468,7 @@ func loadFlagsFromEnv() error {
{postgresOwnerRole, "ATE_API_POSTGRES_OWNER_ROLE"},
{postgresSchema, "ATE_API_POSTGRES_SCHEMA"},
{actorJWTIssuer, "ATE_API_ACTOR_JWT_ISSUER"},
+ {snapshotLocation, "ATE_API_SNAPSHOT_LOCATION"},
}
for _, o := range overrides {
if *o.flag == "@env" {
@@ -515,6 +522,7 @@ func logFlagValues(ctx context.Context) {
slog.Duration("drain-delay", *drainDelay),
slog.Duration("drain-timeout", *drainTimeout),
slog.Duration("actor-workflow-deadline", *actorWorkflowDeadline),
+ slog.String("snapshot-location", *snapshotLocation),
slog.Any("preview", *previewFlags),
)
}
diff --git a/cmd/ateapi/main_test.go b/cmd/ateapi/main_test.go
index b66a7c234f..5a94f9e225 100644
--- a/cmd/ateapi/main_test.go
+++ b/cmd/ateapi/main_test.go
@@ -128,6 +128,20 @@ func TestLoadFlagsFromEnvActorJWTIssuer(t *testing.T) {
}
}
+func TestLoadFlagsFromEnvSnapshotLocation(t *testing.T) {
+ old := *snapshotLocation
+ t.Cleanup(func() { *snapshotLocation = old })
+ *snapshotLocation = "@env"
+ t.Setenv("ATE_API_SNAPSHOT_LOCATION", "gs://ate-snapshots/prefix/")
+
+ if err := loadFlagsFromEnv(); err != nil {
+ t.Fatal(err)
+ }
+ if *snapshotLocation != "gs://ate-snapshots/prefix/" {
+ t.Errorf("--snapshot-location = %q, want %q", *snapshotLocation, "gs://ate-snapshots/prefix/")
+ }
+}
+
func TestPostgresConnectionAttrNeverLogsThePassword(t *testing.T) {
const password = "hunter2-very-secret"
render := func(connString string) string {
diff --git a/cmd/atecontroller/internal/controllers/workerpool_apply.go b/cmd/atecontroller/internal/controllers/workerpool_apply.go
index 4238d19f9c..b2945a7d90 100644
--- a/cmd/atecontroller/internal/controllers/workerpool_apply.go
+++ b/cmd/atecontroller/internal/controllers/workerpool_apply.go
@@ -58,10 +58,11 @@ const (
)
// workerOTelSettings is the telemetry configuration propagated to worker
-// pods. A zero value leaves the pods without telemetry env.
+// pods. A zero value sets only the worker's resource identity.
type workerOTelSettings struct {
- // Endpoint is the OTLP collector address. Empty disables worker telemetry
- // entirely, so the other fields are ignored.
+ // Endpoint is the OTLP collector address the worker exports to directly, when it
+ // finds no atelet relay. Empty leaves the worker exporting through the relay
+ // only, and exporting nothing without one.
Endpoint string
// MetricExportInterval overrides the SDK's 60s PeriodicReader interval. It is
// the raw OTEL_METRIC_EXPORT_INTERVAL value, i.e. whole milliseconds; the SDK
@@ -81,9 +82,15 @@ type workerOTelSettings struct {
// default and drops the arg, which is dead config on its own.
TracesSampler string
TracesSamplerArg string
- // LogsExporter is the raw OTEL_LOGS_EXPORTER value. otlp sends the usage
- // records over OTLP instead of stdout; empty keeps worker's default, none.
- LogsExporter string
+ // LogsExporter, TracesExporter and MetricsExporter are the
+ // OTEL_LOGS_EXPORTER, OTEL_TRACES_EXPORTER and OTEL_METRICS_EXPORTER set on
+ // the worker, passed through untouched. They are worker-only values rather than
+ // the controller's own: on the control plane, metrics prometheus means
+ // scraped instead of pushed, and ateworker serves no /metrics. Empty keeps
+ // ateworker's defaults: none for logs, otlp for traces and metrics.
+ LogsExporter string
+ TracesExporter string
+ MetricsExporter string
}
// workerPoolLabel names the WorkerPool on each of its worker pods.
@@ -99,8 +106,8 @@ const (
// buildDeploymentApplyConfig constructs the SSA apply configuration for the
// Deployment managed by a WorkerPool. Only fields owned by this controller
-// are declared here. otel, when it carries an endpoint, is propagated to the
-// worker container so it pushes telemetry to that collector.
+// are declared here. otel is propagated to the worker container (see
+// workerContainerEnv).
func buildDeploymentApplyConfig(wp *atev1alpha1.WorkerPool, otel workerOTelSettings, systemNamespace, ateletServiceAccount, routerServiceAccount, dnsEgressPolicy string) *appsv1ac.DeploymentApplyConfiguration {
labels := map[string]string{}
annotations := map[string]string{}
@@ -281,26 +288,24 @@ func buildDeploymentApplyConfig(wp *atev1alpha1.WorkerPool, otel workerOTelSetti
WithSpec(podSpecAC)))
}
-// workerContainerEnv adds the OTLP endpoint and resource identity only when
-// telemetry is configured. Every ref precedes OTEL_RESOURCE_ATTRIBUTES so its
-// $(...) substitutions resolve.
+// workerContainerEnv always sets the resource identity, since the worker
+// exports through atelet's relay whenever atelet has a collector, and adds the
+// OTLP endpoint only when one is configured. Every ref precedes
+// OTEL_RESOURCE_ATTRIBUTES so its $(...) substitutions resolve.
func workerContainerEnv(otel workerOTelSettings) []*corev1ac.EnvVarApplyConfiguration {
- // The pool pair labels every usage record, telemetry export or not. A
- // worker pod runs in its WorkerPool's namespace.
+ // The pool pair also labels every usage record. A worker pod runs in its
+ // WorkerPool's namespace.
envs := []*corev1ac.EnvVarApplyConfiguration{
fieldRefEnv("POD_UID", "metadata.uid"),
fieldRefEnv("POD_NAMESPACE", "metadata.namespace"),
fieldRefEnv("WORKER_POOL_NAME", "metadata.labels['"+workerPoolLabel+"']"),
- }
- if otel.Endpoint == "" {
- return envs
- }
- envs = append(envs,
fieldRefEnv("POD_NAME", "metadata.name"),
fieldRefEnv("NODE_NAME", "spec.nodeName"),
- corev1ac.EnvVar().WithName("OTEL_EXPORTER_OTLP_ENDPOINT").WithValue(otel.Endpoint),
- corev1ac.EnvVar().WithName("OTEL_RESOURCE_ATTRIBUTES").WithValue(workerOTelResourceAttributes),
- )
+ }
+ if otel.Endpoint != "" {
+ envs = append(envs, corev1ac.EnvVar().WithName("OTEL_EXPORTER_OTLP_ENDPOINT").WithValue(otel.Endpoint))
+ }
+ envs = append(envs, corev1ac.EnvVar().WithName("OTEL_RESOURCE_ATTRIBUTES").WithValue(workerOTelResourceAttributes))
if otel.MetricExportInterval != "" {
envs = append(envs, corev1ac.EnvVar().
WithName("OTEL_METRIC_EXPORT_INTERVAL").
@@ -326,6 +331,16 @@ func workerContainerEnv(otel workerOTelSettings) []*corev1ac.EnvVarApplyConfigur
WithValue(otel.TracesSamplerArg))
}
}
+ if otel.TracesExporter != "" {
+ envs = append(envs, corev1ac.EnvVar().
+ WithName("OTEL_TRACES_EXPORTER").
+ WithValue(otel.TracesExporter))
+ }
+ if otel.MetricsExporter != "" {
+ envs = append(envs, corev1ac.EnvVar().
+ WithName("OTEL_METRICS_EXPORTER").
+ WithValue(otel.MetricsExporter))
+ }
return envs
}
diff --git a/cmd/atecontroller/internal/controllers/workerpool_apply_test.go b/cmd/atecontroller/internal/controllers/workerpool_apply_test.go
index 80f28e239d..8143c695bb 100644
--- a/cmd/atecontroller/internal/controllers/workerpool_apply_test.go
+++ b/cmd/atecontroller/internal/controllers/workerpool_apply_test.go
@@ -535,15 +535,16 @@ func TestRolloutStrategy(t *testing.T) {
}
}
-// TestBuildDeploymentApplyConfigOTelEndpoint asserts the OTLP endpoint and the
-// resource identity are set on the worker container only when an endpoint is
-// configured, and that every ref the value substitutes is declared ahead of it.
+// TestBuildDeploymentApplyConfigOTelEndpoint asserts the OTLP endpoint is set on
+// the worker container only when one is configured, the resource identity
+// always, since the worker also exports through atelet's relay, and that every
+// ref the identity substitutes is declared ahead of it.
func TestBuildDeploymentApplyConfigOTelEndpoint(t *testing.T) {
const endpoint = "http://collector.otel-system.svc:4317"
tests := []struct {
- name string
- endpoint string
- wantTelemetry bool
+ name string
+ endpoint string
+ wantEndpoint bool
}{
{"endpoint empty", "", false},
{"endpoint set", endpoint, true},
@@ -560,17 +561,9 @@ func TestBuildDeploymentApplyConfigOTelEndpoint(t *testing.T) {
}
}
- if !tt.wantTelemetry {
- for _, k := range []string{"OTEL_EXPORTER_OTLP_ENDPOINT", "OTEL_RESOURCE_ATTRIBUTES", "OTEL_METRIC_EXPORT_INTERVAL", "OTEL_METRIC_EXPORT_TIMEOUT", "OTEL_LOGS_EXPORTER", "POD_NAME", "NODE_NAME"} {
- if _, ok := env[k]; ok {
- t.Errorf("%s must be absent without an OTLP endpoint", k)
- }
- }
- return
- }
-
- if got := env["OTEL_EXPORTER_OTLP_ENDPOINT"].value; got != endpoint {
- t.Errorf("OTEL_EXPORTER_OTLP_ENDPOINT = %q, want %q", got, endpoint)
+ got, ok := env["OTEL_EXPORTER_OTLP_ENDPOINT"]
+ if ok != tt.wantEndpoint || got.value != tt.endpoint {
+ t.Errorf("OTEL_EXPORTER_OTLP_ENDPOINT = %q (set %t), want %q", got.value, ok, tt.endpoint)
}
resourceAttrs := env["OTEL_RESOURCE_ATTRIBUTES"].value
@@ -598,10 +591,11 @@ func TestBuildDeploymentApplyConfigOTelEndpoint(t *testing.T) {
}
// TestBuildDeploymentApplyConfigMetricExportTuning asserts the export interval and
-// per-export timeout reach the worker container only when each is set alongside an
-// endpoint. worker is invisible to the collector until its first successful export
-// tick, so the kind stack shortens the SDK's 60s interval to keep that gap inside
-// the e2e budget, and its 30s timeout so a failing tick cannot swallow three
+// per-export timeout reach the worker container only when each is set, with or
+// without an endpoint, since the worker also exports through atelet's relay. The
+// worker is invisible to the collector until its first successful export tick,
+// so the kind stack shortens the SDK's 60s interval to keep that gap inside the
+// e2e budget, and its 30s timeout so a failing tick cannot swallow three
// shortened intervals.
func TestBuildDeploymentApplyConfigMetricExportTuning(t *testing.T) {
const endpoint = "http://collector.otel-system.svc:4317"
@@ -631,9 +625,9 @@ func TestBuildDeploymentApplyConfigMetricExportTuning(t *testing.T) {
want: map[string]string{"OTEL_METRIC_EXPORT_TIMEOUT": "10000"},
},
{
- name: "ignored without endpoint",
+ name: "set without endpoint",
otel: workerOTelSettings{MetricExportInterval: "10000", MetricExportTimeout: "10000"},
- want: nil,
+ want: map[string]string{"OTEL_METRIC_EXPORT_INTERVAL": "10000", "OTEL_METRIC_EXPORT_TIMEOUT": "10000"},
},
}
for _, tt := range tests {
@@ -657,7 +651,7 @@ func TestBuildDeploymentApplyConfigMetricExportTuning(t *testing.T) {
}
// TestBuildDeploymentApplyConfigTracesSamplerPropagation asserts the sampler
-// env pair reaches the worker container only alongside an endpoint, and the arg
+// env pair reaches the worker container with or without an endpoint, and the arg
// only alongside a sampler: an arg without a sampler name is dead config the
// SDK ignores.
func TestBuildDeploymentApplyConfigTracesSamplerPropagation(t *testing.T) {
@@ -688,9 +682,9 @@ func TestBuildDeploymentApplyConfigTracesSamplerPropagation(t *testing.T) {
want: nil,
},
{
- name: "ignored without endpoint",
+ name: "set without endpoint",
otel: workerOTelSettings{TracesSampler: "parentbased_traceidratio", TracesSamplerArg: "0.25"},
- want: nil,
+ want: map[string]string{"OTEL_TRACES_SAMPLER": "parentbased_traceidratio", "OTEL_TRACES_SAMPLER_ARG": "0.25"},
},
}
for _, tt := range tests {
@@ -713,6 +707,60 @@ func TestBuildDeploymentApplyConfigTracesSamplerPropagation(t *testing.T) {
}
}
+// TestBuildDeploymentApplyConfigTracesExporterPropagation asserts
+// OTEL_TRACES_EXPORTER reaches the worker container only when set, with or
+// without an endpoint.
+func TestBuildDeploymentApplyConfigTracesExporterPropagation(t *testing.T) {
+ const endpoint = "http://collector.otel-system.svc:4317"
+ tests := []struct {
+ name string
+ otel workerOTelSettings
+ want string
+ wantSet bool
+ }{
+ {name: "unset keeps binary default", otel: workerOTelSettings{Endpoint: endpoint}},
+ {name: "none with endpoint", otel: workerOTelSettings{Endpoint: endpoint, TracesExporter: "none"}, want: "none", wantSet: true},
+ {name: "none without endpoint", otel: workerOTelSettings{TracesExporter: "none"}, want: "none", wantSet: true},
+ }
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ c := buildDeploymentApplyConfig(testWorkerPoolApplyConfig(nil), tt.otel, installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount, egresspolicy.DNSEgressPolicyEnforceInAtunnel).
+ Spec.Template.Spec.Containers[0]
+ got, ok := envByName(c.Env)["OTEL_TRACES_EXPORTER"]
+ if ok != tt.wantSet || got.value != tt.want {
+ t.Errorf("OTEL_TRACES_EXPORTER = %q (set %t), want %q (set %t)", got.value, ok, tt.want, tt.wantSet)
+ }
+ })
+ }
+}
+
+// TestBuildDeploymentApplyConfigMetricsExporterPropagation asserts
+// OTEL_METRICS_EXPORTER reaches the worker container only when set, with or
+// without an endpoint.
+func TestBuildDeploymentApplyConfigMetricsExporterPropagation(t *testing.T) {
+ const endpoint = "http://collector.otel-system.svc:4317"
+ tests := []struct {
+ name string
+ otel workerOTelSettings
+ want string
+ wantSet bool
+ }{
+ {name: "unset keeps binary default", otel: workerOTelSettings{Endpoint: endpoint}},
+ {name: "none with endpoint", otel: workerOTelSettings{Endpoint: endpoint, MetricsExporter: "none"}, want: "none", wantSet: true},
+ {name: "none without endpoint", otel: workerOTelSettings{MetricsExporter: "none"}, want: "none", wantSet: true},
+ }
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ c := buildDeploymentApplyConfig(testWorkerPoolApplyConfig(nil), tt.otel, installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount, egresspolicy.DNSEgressPolicyEnforceInAtunnel).
+ Spec.Template.Spec.Containers[0]
+ got, ok := envByName(c.Env)["OTEL_METRICS_EXPORTER"]
+ if ok != tt.wantSet || got.value != tt.want {
+ t.Errorf("OTEL_METRICS_EXPORTER = %q (set %t), want %q (set %t)", got.value, ok, tt.want, tt.wantSet)
+ }
+ })
+ }
+}
+
type envInfo struct {
index int
value string
@@ -879,6 +927,19 @@ func expectedDeploymentApplyConfig(mutatePodSpec func(*corev1ac.PodSpecApplyConf
WithValueFrom(corev1ac.EnvVarSource().
WithFieldRef(corev1ac.ObjectFieldSelector().
WithFieldPath("metadata.labels['ate.dev/worker-pool']"))),
+ corev1ac.EnvVar().
+ WithName("POD_NAME").
+ WithValueFrom(corev1ac.EnvVarSource().
+ WithFieldRef(corev1ac.ObjectFieldSelector().
+ WithFieldPath("metadata.name"))),
+ corev1ac.EnvVar().
+ WithName("NODE_NAME").
+ WithValueFrom(corev1ac.EnvVarSource().
+ WithFieldRef(corev1ac.ObjectFieldSelector().
+ WithFieldPath("spec.nodeName"))),
+ corev1ac.EnvVar().
+ WithName("OTEL_RESOURCE_ATTRIBUTES").
+ WithValue(workerOTelResourceAttributes),
).
WithVolumeMounts(
corev1ac.VolumeMount().
@@ -1049,7 +1110,8 @@ func TestBuildDeploymentApplyConfigWorkerPoolEnv(t *testing.T) {
}
// TestBuildDeploymentApplyConfigLogsExporter asserts OTEL_LOGS_EXPORTER
-// reaches the worker container only alongside an endpoint.
+// reaches the worker container whenever it is set, with or without an endpoint,
+// since the worker also exports through atelet's relay.
func TestBuildDeploymentApplyConfigLogsExporter(t *testing.T) {
const endpoint = "http://collector.otel-system.svc:4317"
for _, tt := range []struct {
@@ -1059,7 +1121,7 @@ func TestBuildDeploymentApplyConfigLogsExporter(t *testing.T) {
}{
{"set with endpoint", workerOTelSettings{Endpoint: endpoint, LogsExporter: "otlp"}, "otlp"},
{"unset keeps binary default", workerOTelSettings{Endpoint: endpoint}, ""},
- {"ignored without endpoint", workerOTelSettings{LogsExporter: "otlp"}, ""},
+ {"set without endpoint", workerOTelSettings{LogsExporter: "otlp"}, "otlp"},
} {
t.Run(tt.name, func(t *testing.T) {
c := buildDeploymentApplyConfig(testWorkerPoolApplyConfig(nil), tt.otel, installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount, egresspolicy.DNSEgressPolicyEnforceInAtunnel).
diff --git a/cmd/atecontroller/internal/controllers/workerpool_controller.go b/cmd/atecontroller/internal/controllers/workerpool_controller.go
index 96cb8d4f1b..cb654cd987 100644
--- a/cmd/atecontroller/internal/controllers/workerpool_controller.go
+++ b/cmd/atecontroller/internal/controllers/workerpool_controller.go
@@ -72,6 +72,12 @@ type WorkerPoolReconciler struct {
// OTelLogsExporter is the OTEL_LOGS_EXPORTER propagated to worker pods.
// Empty keeps the worker binary's default.
OTelLogsExporter string
+ // OTelTracesExporter is the OTEL_TRACES_EXPORTER propagated to worker pods.
+ // Empty keeps the ateworker binary's default.
+ OTelTracesExporter string
+ // OTelMetricsExporter is the OTEL_METRICS_EXPORTER propagated to worker pods.
+ // Empty keeps the ateworker binary's default.
+ OTelMetricsExporter string
// SystemNamespace is the namespace substrate's control plane runs in, and
// AteletServiceAccount / RouterServiceAccount are the ServiceAccounts those
// components run as. Together they name the SPIFFE identities that atunnel
@@ -332,6 +338,8 @@ func (r *WorkerPoolReconciler) applyDeployment(ctx context.Context, wp *atev1alp
TracesSampler: r.OTelTracesSampler,
TracesSamplerArg: r.OTelTracesSamplerArg,
LogsExporter: r.OTelLogsExporter,
+ TracesExporter: r.OTelTracesExporter,
+ MetricsExporter: r.OTelMetricsExporter,
}, r.SystemNamespace, r.AteletServiceAccount, r.RouterServiceAccount, r.DNSEgressPolicy)
if err := r.Apply(ctx, depAC, client.FieldOwner(workerPoolFieldOwner), client.ForceOwnership); err != nil {
return fmt.Errorf("failed to apply Deployment: %w", err)
diff --git a/cmd/atecontroller/main.go b/cmd/atecontroller/main.go
index e7345b4fc9..6271ba6ade 100644
--- a/cmd/atecontroller/main.go
+++ b/cmd/atecontroller/main.go
@@ -64,7 +64,7 @@ var (
previewFlags = pflag.StringSlice("preview", nil, "Preview gates to enable.")
otelEndpoint = pflag.String("otel-exporter-otlp-endpoint", os.Getenv("OTEL_EXPORTER_OTLP_ENDPOINT"),
- "OTLP endpoint set on worker pods so they push telemetry. Defaults to the controller's own OTEL_EXPORTER_OTLP_ENDPOINT.")
+ "OTLP endpoint set on worker pods, which export to it directly when their node's atelet serves no OTLP relay. Defaults to the controller's own OTEL_EXPORTER_OTLP_ENDPOINT.")
otelMetricExportInterval = pflag.String("otel-metric-export-interval", os.Getenv("OTEL_METRIC_EXPORT_INTERVAL"),
"Metric export interval in milliseconds set on worker pods. Empty keeps the OTel SDK's 60s default. Defaults to the controller's own OTEL_METRIC_EXPORT_INTERVAL.")
@@ -78,8 +78,14 @@ var (
otelTracesSamplerArg = pflag.String("otel-traces-sampler-arg", os.Getenv("OTEL_TRACES_SAMPLER_ARG"),
"Trace sampler argument set on worker pods, ignored unless --otel-traces-sampler is set. Defaults to the controller's own OTEL_TRACES_SAMPLER_ARG.")
- otelLogsExporter = pflag.String("otel-logs-exporter", os.Getenv("OTEL_LOGS_EXPORTER"),
- "Logs exporter set on worker pods. Empty keeps the ateworker binary's default, none. Defaults to the controller's own OTEL_LOGS_EXPORTER.")
+ otelLogsExporter = pflag.String("otel-logs-exporter", os.Getenv("ATE_WORKER_OTEL_LOGS_EXPORTER"),
+ "Logs exporter set on worker pods, with OTEL_LOGS_EXPORTER's values. Empty keeps the ateworker binary's default, none. Defaults to the controller's ATE_WORKER_OTEL_LOGS_EXPORTER, not its own OTEL_LOGS_EXPORTER.")
+
+ otelTracesExporter = pflag.String("otel-traces-exporter", os.Getenv("ATE_WORKER_OTEL_TRACES_EXPORTER"),
+ "Traces exporter set on worker pods, with OTEL_TRACES_EXPORTER's values. Empty keeps the ateworker binary's otlp default. Defaults to the controller's ATE_WORKER_OTEL_TRACES_EXPORTER, not its own OTEL_TRACES_EXPORTER.")
+
+ otelMetricsExporter = pflag.String("otel-metrics-exporter", os.Getenv("ATE_WORKER_OTEL_METRICS_EXPORTER"),
+ "Metrics exporter set on worker pods, with OTEL_METRICS_EXPORTER's values. Empty keeps the ateworker binary's otlp default. Defaults to the controller's ATE_WORKER_OTEL_METRICS_EXPORTER, not its own OTEL_METRICS_EXPORTER.")
ateletServiceAccount = pflag.String("atelet-service-account", installdefaults.AteletServiceAccount, "ServiceAccount atelet runs as. It is the service-account segment of the SPIFFE ID each worker's atunnel expects on the credential broker, so it has to match what the deployment actually creates.")
routerServiceAccount = pflag.String("router-service-account", installdefaults.RouterServiceAccount, "ServiceAccount atenet-router runs as. It is the service-account segment of the SPIFFE ID each worker's atunnel accepts on actor ingress, so it has to match what the deployment actually creates.")
@@ -253,6 +259,8 @@ func main() {
OTelTracesSampler: *otelTracesSampler,
OTelTracesSamplerArg: *otelTracesSamplerArg,
OTelLogsExporter: *otelLogsExporter,
+ OTelTracesExporter: *otelTracesExporter,
+ OTelMetricsExporter: *otelMetricsExporter,
SystemNamespace: systemNamespace,
AteletServiceAccount: *ateletServiceAccount,
RouterServiceAccount: *routerServiceAccount,
diff --git a/cmd/atelet/helm_test.go b/cmd/atelet/helm_test.go
index 78e1f01246..4a05ce288e 100644
--- a/cmd/atelet/helm_test.go
+++ b/cmd/atelet/helm_test.go
@@ -155,7 +155,7 @@ func TestHelmSandboxAssetStorage(t *testing.T) {
"--set", "objectStorePlugin.extraEnv[0].name=AWS_REGION",
"--set", "objectStorePlugin.extraEnv[0].value=us-west-2"}
if !tc.rustfs {
- args = append(args, "--set", "rustfs.enabled=false")
+ args = append(args, "--set", "rustfs.enabled=false", "--set", "ateApi.snapshotLocation=gs://external-snapshots/")
}
out, err := exec.CommandContext(t.Context(), "helm", args...).CombinedOutput()
if err != nil {
diff --git a/cmd/atelet/herderauthz_test.go b/cmd/atelet/herderauthz_test.go
new file mode 100644
index 0000000000..b7c1772684
--- /dev/null
+++ b/cmd/atelet/herderauthz_test.go
@@ -0,0 +1,329 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package main
+
+import (
+ "crypto/ecdsa"
+ "crypto/elliptic"
+ "crypto/rand"
+ "crypto/tls"
+ "crypto/x509"
+ "crypto/x509/pkix"
+ "encoding/pem"
+ "errors"
+ "io"
+ "math/big"
+ "net"
+ "net/url"
+ "os"
+ "path/filepath"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/spiffe/go-spiffe/v2/spiffeid"
+ "go.ate.dev/substrate/internal/installdefaults"
+ "go.ate.dev/substrate/internal/substratex509"
+)
+
+func allowAll(tls.ConnectionState) error { return nil }
+
+// testPodCA is a throwaway stand-in for the podidentity signer: it issues
+// leaf certificates shaped like the ones the signer mints (SPIFFE URI SAN,
+// clientAuth+serverAuth EKU, PodIdentity extension).
+type testPodCA struct {
+ cert *x509.Certificate
+ key *ecdsa.PrivateKey
+}
+
+func newTestPodCA(t *testing.T) *testPodCA {
+ t.Helper()
+ key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
+ if err != nil {
+ t.Fatal(err)
+ }
+ tmpl := &x509.Certificate{
+ SerialNumber: big.NewInt(1),
+ Subject: pkix.Name{CommonName: "test podidentity CA"},
+ NotBefore: time.Now().Add(-time.Minute),
+ NotAfter: time.Now().Add(time.Hour),
+ IsCA: true,
+ BasicConstraintsValid: true,
+ KeyUsage: x509.KeyUsageCertSign,
+ }
+ der, err := x509.CreateCertificate(rand.Reader, tmpl, tmpl, &key.PublicKey, key)
+ if err != nil {
+ t.Fatal(err)
+ }
+ cert, err := x509.ParseCertificate(der)
+ if err != nil {
+ t.Fatal(err)
+ }
+ return &testPodCA{cert: cert, key: key}
+}
+
+// writeTrustBundle writes the CA certificate as a PEM trust bundle.
+func (ca *testPodCA) writeTrustBundle(t *testing.T) string {
+ t.Helper()
+ path := filepath.Join(t.TempDir(), "trust-bundle.pem")
+ if err := os.WriteFile(path, pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: ca.cert.Raw}), 0o600); err != nil {
+ t.Fatal(err)
+ }
+ return path
+}
+
+// issue mints a leaf for the given Pod identity and returns it both as a
+// tls.Certificate and as a credential bundle file on disk.
+func (ca *testPodCA) issue(t *testing.T, id *substratex509.PodIdentity) (tls.Certificate, string) {
+ t.Helper()
+ spiffeURI, err := url.Parse(installdefaults.SPIFFEID(id.Namespace, id.ServiceAccountName))
+ if err != nil {
+ t.Fatal(err)
+ }
+ return ca.issueWithURIs(t, id, []*url.URL{spiffeURI})
+}
+
+// issueWithURIs is issue with explicit URI SANs, for malformed-identity cases.
+func (ca *testPodCA) issueWithURIs(t *testing.T, id *substratex509.PodIdentity, uris []*url.URL) (tls.Certificate, string) {
+ t.Helper()
+ key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
+ if err != nil {
+ t.Fatal(err)
+ }
+ serial, err := rand.Int(rand.Reader, big.NewInt(1<<62))
+ if err != nil {
+ t.Fatal(err)
+ }
+ tmpl := &x509.Certificate{
+ SerialNumber: serial,
+ Subject: pkix.Name{CommonName: rand.Text()},
+ NotBefore: time.Now().Add(-time.Minute),
+ NotAfter: time.Now().Add(time.Hour),
+ URIs: uris,
+ KeyUsage: x509.KeyUsageDigitalSignature,
+ ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth, x509.ExtKeyUsageServerAuth},
+ }
+ if err := substratex509.AddPodIdentityToCertificate(id, tmpl); err != nil {
+ t.Fatal(err)
+ }
+ der, err := x509.CreateCertificate(rand.Reader, tmpl, ca.cert, &key.PublicKey, ca.key)
+ if err != nil {
+ t.Fatal(err)
+ }
+ keyDER, err := x509.MarshalPKCS8PrivateKey(key)
+ if err != nil {
+ t.Fatal(err)
+ }
+ bundle := append(
+ pem.EncodeToMemory(&pem.Block{Type: "PRIVATE KEY", Bytes: keyDER}),
+ pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der})...)
+ path := filepath.Join(t.TempDir(), "credential-bundle.pem")
+ if err := os.WriteFile(path, bundle, 0o600); err != nil {
+ t.Fatal(err)
+ }
+ leaf, err := x509.ParseCertificate(der)
+ if err != nil {
+ t.Fatal(err)
+ }
+ return tls.Certificate{Certificate: [][]byte{der}, PrivateKey: key, Leaf: leaf}, path
+}
+
+func podIdentity(namespace, serviceAccount, nodeName string) *substratex509.PodIdentity {
+ return &substratex509.PodIdentity{
+ Namespace: namespace,
+ ServiceAccountName: serviceAccount,
+ ServiceAccountUID: serviceAccount + "-uid",
+ PodName: serviceAccount + "-pod",
+ PodUID: serviceAccount + "-" + nodeName + "-pod-uid",
+ NodeName: nodeName,
+ NodeUID: nodeName + "-uid",
+ }
+}
+
+// handshake runs a full mTLS handshake between a client presenting clientCert
+// and a server using serverCfg, and returns the server-side handshake error.
+// When the server accepts, it also fails the test unless the client saw a
+// clean close, so a client-side failure cannot pass as acceptance.
+func handshake(t *testing.T, serverCfg *tls.Config, clientCert tls.Certificate, roots *x509.CertPool) error {
+ t.Helper()
+ serverConn, clientConn := net.Pipe()
+ t.Cleanup(func() {
+ serverConn.Close()
+ clientConn.Close()
+ })
+ _ = serverConn.SetDeadline(time.Now().Add(10 * time.Second))
+ _ = clientConn.SetDeadline(time.Now().Add(10 * time.Second))
+
+ clientErr := make(chan error, 1)
+ go func() {
+ c := tls.Client(clientConn, &tls.Config{
+ MinVersion: tls.VersionTLS13,
+ Certificates: []tls.Certificate{clientCert},
+ RootCAs: roots,
+ // The server cert has no DNS SAN, mirroring how ateapi dials atelet
+ // by Pod IP; this test exercises server-side authorization only.
+ InsecureSkipVerify: true,
+ })
+ err := c.Handshake()
+ if err == nil {
+ // In TLS 1.3 the client finishes before the server has checked the
+ // client certificate; a read surfaces the server's verdict.
+ _, err = c.Read(make([]byte, 1))
+ }
+ clientErr <- err
+ c.Close()
+ }()
+
+ s := tls.Server(serverConn, serverCfg)
+ err := s.Handshake()
+ if err == nil {
+ // Unblock the client's read so the goroutine exits.
+ s.Close()
+ }
+ cerr := <-clientErr
+ if err == nil && !errors.Is(cerr, io.EOF) {
+ t.Errorf("server accepted the handshake but the client saw %v, want io.EOF", cerr)
+ }
+ return err
+}
+
+// wantHandshakeErr fails the test unless err is non-nil and mentions want,
+// so a rejection for an unrelated reason (timeout, cipher mismatch) cannot
+// pass as an authorization failure.
+func wantHandshakeErr(t *testing.T, err error, want string) {
+ t.Helper()
+ if err == nil {
+ t.Fatalf("handshake succeeded, want an error containing %q", want)
+ }
+ if !strings.Contains(err.Error(), want) {
+ t.Fatalf("handshake error = %v, want it to contain %q", err, want)
+ }
+}
+
+func TestAteomHerderTLSAdmitsOnlyAteapi(t *testing.T) {
+ ca := newTestPodCA(t)
+ trustBundle := ca.writeTrustBundle(t)
+ roots := x509.NewCertPool()
+ roots.AddCert(ca.cert)
+
+ _, serverBundle := ca.issue(t, podIdentity("ate-system", "atelet", "node-a"))
+ ateapiID := spiffeid.RequireFromString("spiffe://cluster.local/ns/ate-system/sa/ate-api-server")
+ cfg, err := ateletServerTLSConfig(serverBundle, trustBundle, verifyClientSPIFFEID(ateapiID))
+ if err != nil {
+ t.Fatalf("ateletServerTLSConfig() error = %v", err)
+ }
+
+ if err := handshake(t, cfg, mustIssue(t, ca, podIdentity("ate-system", "ate-api-server", "node-b")), roots); err != nil {
+ t.Fatalf("ate-api-server handshake error = %v, want nil", err)
+ }
+
+ // Any pod can get a podidentity cert from the same CA; none of these may
+ // drive actor lifecycles.
+ const notAuthorized = "is not authorized"
+ rejected := []struct {
+ name string
+ client *substratex509.PodIdentity
+ }{
+ {name: "workload pod in another namespace", client: podIdentity("tenant", "default", "node-a")},
+ {name: "ate-api-server SA name in another namespace", client: podIdentity("tenant", "ate-api-server", "node-a")},
+ {name: "other ate-system component", client: podIdentity("ate-system", "atenet-router", "node-a")},
+ {name: "atelet on another node", client: podIdentity("ate-system", "atelet", "node-b")},
+ }
+ for _, tc := range rejected {
+ t.Run(tc.name, func(t *testing.T) {
+ wantHandshakeErr(t, handshake(t, cfg, mustIssue(t, ca, tc.client), roots), notAuthorized)
+ })
+ }
+
+ t.Run("no URI SAN", func(t *testing.T) {
+ cert, _ := ca.issueWithURIs(t, podIdentity("ate-system", "ate-api-server", "node-a"), nil)
+ wantHandshakeErr(t, handshake(t, cfg, cert, roots), "no valid SPIFFE ID")
+ })
+ t.Run("ate-api-server plus a second URI SAN", func(t *testing.T) {
+ id := podIdentity("ate-system", "ate-api-server", "node-a")
+ cert, _ := ca.issueWithURIs(t, id, []*url.URL{
+ mustParseURL(t, installdefaults.SPIFFEID(id.Namespace, id.ServiceAccountName)),
+ mustParseURL(t, "spiffe://cluster.local/ns/tenant/sa/default"),
+ })
+ wantHandshakeErr(t, handshake(t, cfg, cert, roots), "no valid SPIFFE ID")
+ })
+ t.Run("no client certificate", func(t *testing.T) {
+ // RequireAndVerifyClientCert rejects before the SPIFFE check runs.
+ wantHandshakeErr(t, handshake(t, cfg, tls.Certificate{}, roots), "client didn't provide a certificate")
+ })
+}
+
+func mustIssue(t *testing.T, ca *testPodCA, id *substratex509.PodIdentity) tls.Certificate {
+ t.Helper()
+ cert, _ := ca.issue(t, id)
+ return cert
+}
+
+func mustParseURL(t *testing.T, raw string) *url.URL {
+ t.Helper()
+ u, err := url.Parse(raw)
+ if err != nil {
+ t.Fatal(err)
+ }
+ return u
+}
+
+func TestAteomHerderTLSRejectsUntrustedCA(t *testing.T) {
+ ca := newTestPodCA(t)
+ rogue := newTestPodCA(t)
+ roots := x509.NewCertPool()
+ roots.AddCert(ca.cert)
+
+ _, serverBundle := ca.issue(t, podIdentity("ate-system", "atelet", "node-a"))
+ ateapiID := spiffeid.RequireFromString("spiffe://cluster.local/ns/ate-system/sa/ate-api-server")
+ cfg, err := ateletServerTLSConfig(serverBundle, ca.writeTrustBundle(t), verifyClientSPIFFEID(ateapiID))
+ if err != nil {
+ t.Fatalf("ateletServerTLSConfig() error = %v", err)
+ }
+ // Right SPIFFE ID, wrong issuer: the chain check must still apply.
+ forged, _ := rogue.issue(t, podIdentity("ate-system", "ate-api-server", "node-a"))
+ wantHandshakeErr(t, handshake(t, cfg, forged, roots), "unknown authority")
+}
+
+// TestCredentialBrokerTLSEnforcesSameNode is a regression test: the same-node
+// check used to be set on the outer tls.Config, which crypto/tls discards in
+// favor of the GetConfigForClient result, so it never ran during a handshake.
+func TestCredentialBrokerTLSEnforcesSameNode(t *testing.T) {
+ ca := newTestPodCA(t)
+ trustBundle := ca.writeTrustBundle(t)
+ roots := x509.NewCertPool()
+ roots.AddCert(ca.cert)
+
+ ateletID := podIdentity("ate-system", "atelet", "node-a")
+ _, serverBundle := ca.issue(t, ateletID)
+ cfg, err := ateletServerTLSConfig(serverBundle, trustBundle, verifyClientOnSameNode(ateletID))
+ if err != nil {
+ t.Fatalf("ateletServerTLSConfig() error = %v", err)
+ }
+
+ sameNode, _ := ca.issue(t, podIdentity("workers", "default", "node-a"))
+ if err := handshake(t, cfg, sameNode, roots); err != nil {
+ t.Fatalf("same-node worker handshake error = %v, want nil", err)
+ }
+ otherNode, _ := ca.issue(t, podIdentity("workers", "default", "node-b"))
+ wantHandshakeErr(t, handshake(t, cfg, otherNode, roots), "is not on node")
+}
+
+func TestAteletServerTLSConfigRequiresVerifier(t *testing.T) {
+ ca := newTestPodCA(t)
+ if _, err := ateletServerTLSConfig("/nonexistent-cred-bundle.pem", ca.writeTrustBundle(t), nil); err == nil {
+ t.Fatal("ateletServerTLSConfig() with a nil verifier succeeded, want an error")
+ }
+}
diff --git a/cmd/atelet/internal/apivalidation/validate.go b/cmd/atelet/internal/apivalidation/validate.go
index 4e73e734b2..47b737399a 100644
--- a/cmd/atelet/internal/apivalidation/validate.go
+++ b/cmd/atelet/internal/apivalidation/validate.go
@@ -28,6 +28,11 @@ func ValidateRequestActorSuspendRequest(ctx context.Context, req *ateletpb.Reque
return Validate_RequestActorSuspendRequest(ctx, operation.Operation{Type: operation.Create}, nil, req, nil)
}
+// ValidateReportActorCrashedRequest runs the generated validation for req.
+func ValidateReportActorCrashedRequest(ctx context.Context, req *ateletpb.ReportActorCrashedRequest) field.ErrorList {
+ return Validate_ReportActorCrashedRequest(ctx, operation.Operation{Type: operation.Create}, nil, req, nil)
+}
+
// ValidateGetActorEgressPolicyRequest runs the generated validation for
// req.
func ValidateGetActorEgressPolicyRequest(ctx context.Context, req *ateletpb.GetActorEgressPolicyRequest) field.ErrorList {
@@ -85,25 +90,10 @@ func ValidateCustom_EgressGateway_Address(_ context.Context, _ operation.Operati
return resources.ValidateHostPort(fldPath, *value)
}
-func ValidateCustom_SandboxAssets_SandboxClass(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *string) field.ErrorList {
- return resources.ValidateSandboxClass(fldPath, *value)
-}
-
func ValidateCustom_SandboxRuntime_SandboxClass(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *string) field.ErrorList {
return resources.ValidateSandboxClass(fldPath, *value)
}
-func ValidateCustom_SandboxAssets_PauseImage(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *string) field.ErrorList {
- return resources.ValidatePinnedImage(fldPath, *value)
-}
-
-func ValidateCustom_AssetFile_Sha256(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *string) field.ErrorList {
- if err := resources.ValidateAssetHash(*value); err != nil {
- return field.ErrorList{field.Invalid(fldPath, *value, err.Error())}
- }
- return nil
-}
-
func ValidateCustom_ObjectStorage_SnapshotUri(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *string) field.ErrorList {
return validateSnapshotURI(fldPath, *value)
}
diff --git a/cmd/atelet/internal/apivalidation/validation_test.go b/cmd/atelet/internal/apivalidation/validation_test.go
index 1978ec860c..35772d76e1 100644
--- a/cmd/atelet/internal/apivalidation/validation_test.go
+++ b/cmd/atelet/internal/apivalidation/validation_test.go
@@ -27,23 +27,6 @@ import (
const testDigestImage = "example.com/app@sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
-const testSHA256 = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
-
-func validSandboxAssets(mutate ...func(*ateletpb.SandboxAssets)) *ateletpb.SandboxAssets {
- a := &ateletpb.SandboxAssets{
- SandboxClass: "gvisor",
- PauseImage: testDigestImage,
- Assets: map[string]*ateletpb.ArchAssets{
- "amd64": {Files: map[string]*ateletpb.AssetFile{"gvisor": {Url: "gs://bucket/gvisor.tar.zstd", Sha256: testSHA256}}},
- "arm64": {Files: map[string]*ateletpb.AssetFile{"gvisor": {Url: "gs://bucket/gvisor.tar.zstd", Sha256: testSHA256}}},
- },
- }
- for _, m := range mutate {
- m(a)
- }
- return a
-}
-
// createOp is for tests of nested messages, which have no request wrapper.
var createOp = operation.Operation{Type: operation.Create}
@@ -247,7 +230,7 @@ func TestValidateRunRequest(t *testing.T) {
ActorTemplateAtespace: "team-a",
ActorTemplateName: "tmpl-1",
Spec: &ateletpb.WorkloadSpec{Containers: []*ateletpb.Container{{Name: "worker", Image: testDigestImage}}},
- SandboxAssets: validSandboxAssets(),
+ SandboxRuntime: runtimeWithAttributes(1),
CpuMilli: 500,
MemoryBytes: 1 << 30,
}
@@ -319,14 +302,6 @@ func TestValidateRunRequest(t *testing.T) {
name: "missing spec",
obj: valid(func(r *ateletpb.RunRequest) { r.Spec = nil }),
want: field.ErrorList{field.Required(field.NewPath("spec"), "")},
- }, {
- name: "missing sandbox_assets",
- obj: valid(func(r *ateletpb.RunRequest) { r.SandboxAssets = nil }),
- want: field.ErrorList{field.Required(field.NewPath("sandbox_assets"), "")},
- }, {
- name: "sandbox asset errors surface at the sandbox_assets path",
- obj: valid(func(r *ateletpb.RunRequest) { r.SandboxAssets.PauseImage = "registry.k8s.io/pause:3.10.2" }),
- want: field.ErrorList{field.Invalid(field.NewPath("sandbox_assets", "pause_image"), nil, "")},
}, {
name: "invalid container name: path escape",
obj: valid(func(r *ateletpb.RunRequest) {
@@ -392,14 +367,12 @@ func TestValidateRunRequest(t *testing.T) {
r.MemoryBytes = 0
}),
}, {
- name: "valid sandbox_runtime",
- obj: valid(func(r *ateletpb.RunRequest) {
- r.SandboxRuntime = runtimeWithAttributes(1)
- }),
+ name: "missing sandbox_runtime",
+ obj: valid(func(r *ateletpb.RunRequest) { r.SandboxRuntime = nil }),
+ want: field.ErrorList{field.Required(field.NewPath("sandbox_runtime"), "")},
}, {
name: "sandbox_runtime errors surface at their path",
obj: valid(func(r *ateletpb.RunRequest) {
- r.SandboxRuntime = runtimeWithAttributes(1)
r.SandboxRuntime.SandboxClass = "kata"
}),
want: field.ErrorList{field.NotSupported[string](field.NewPath("sandbox_runtime", "sandbox_class"), nil, nil)},
@@ -411,101 +384,6 @@ func TestValidateRunRequest(t *testing.T) {
}
}
-// TestValidateSandboxAssets covers SandboxAssets and the asset files it holds.
-func TestValidateSandboxAssets(t *testing.T) {
- file := func(url, sha string) *ateletpb.ArchAssets {
- return &ateletpb.ArchAssets{Files: map[string]*ateletpb.AssetFile{"gvisor": {Url: url, Sha256: sha}}}
- }
- amd64 := field.NewPath("assets").Key("amd64")
- amd64File := amd64.Child("files").Key("gvisor")
-
- tests := []struct {
- name string
- obj *ateletpb.SandboxAssets
- want field.ErrorList
- }{{
- name: "valid gvisor",
- obj: validSandboxAssets(),
- }, {
- name: "valid microvm",
- obj: validSandboxAssets(func(a *ateletpb.SandboxAssets) { a.SandboxClass = "microvm" }),
- }, {
- name: "missing sandbox_class",
- obj: validSandboxAssets(func(a *ateletpb.SandboxAssets) { a.SandboxClass = "" }),
- want: field.ErrorList{field.Required(field.NewPath("sandbox_class"), "")},
- }, {
- name: "unknown sandbox_class",
- obj: validSandboxAssets(func(a *ateletpb.SandboxAssets) { a.SandboxClass = "kvm" }),
- want: field.ErrorList{field.NotSupported[string](field.NewPath("sandbox_class"), nil, nil)},
- }, {
- name: "sandbox_class is case-sensitive",
- obj: validSandboxAssets(func(a *ateletpb.SandboxAssets) { a.SandboxClass = "GVISOR" }),
- want: field.ErrorList{field.NotSupported[string](field.NewPath("sandbox_class"), nil, nil)},
- }, {
- name: "sandbox_class too long",
- obj: validSandboxAssets(func(a *ateletpb.SandboxAssets) { a.SandboxClass = strings.Repeat("g", 64) }),
- want: field.ErrorList{
- field.TooLong(field.NewPath("sandbox_class"), nil, 63).WithOrigin("maxLength"),
- field.NotSupported[string](field.NewPath("sandbox_class"), nil, nil),
- },
- }, {
- name: "valid: no pause_image (microvm has no pause container)",
- obj: validSandboxAssets(func(a *ateletpb.SandboxAssets) { a.PauseImage = "" }),
- }, {
- name: "pause_image not pinned by digest",
- obj: validSandboxAssets(func(a *ateletpb.SandboxAssets) { a.PauseImage = "registry.k8s.io/pause:3.10.2" }),
- want: field.ErrorList{field.Invalid(field.NewPath("pause_image"), nil, "")},
- }, {
- name: "missing assets",
- obj: validSandboxAssets(func(a *ateletpb.SandboxAssets) { a.Assets = nil }),
- want: field.ErrorList{field.Required(field.NewPath("assets"), "")},
- }, {
- name: "too many architectures",
- obj: validSandboxAssets(func(a *ateletpb.SandboxAssets) {
- for _, arch := range []string{"386", "arm", "loong64", "mips64", "ppc64le", "riscv64", "s390x"} {
- a.Assets[arch] = file("gs://bucket/gvisor.tar.zstd", testSHA256)
- }
- }),
- want: field.ErrorList{field.TooMany(field.NewPath("assets"), 9, 8).WithOrigin("maxProperties")},
- }, {
- name: "architecture with no files",
- obj: validSandboxAssets(func(a *ateletpb.SandboxAssets) { a.Assets["amd64"] = &ateletpb.ArchAssets{} }),
- want: field.ErrorList{field.Required(amd64.Child("files"), "")},
- }, {
- name: "missing url",
- obj: validSandboxAssets(func(a *ateletpb.SandboxAssets) { a.Assets["amd64"] = file("", testSHA256) }),
- want: field.ErrorList{field.Required(amd64File.Child("url"), "")},
- }, {
- name: "missing sha256",
- obj: validSandboxAssets(func(a *ateletpb.SandboxAssets) { a.Assets["amd64"] = file("gs://bucket/gvisor.tar.zstd", "") }),
- want: field.ErrorList{field.Required(amd64File.Child("sha256"), "")},
- }, {
- name: "sha256 too short",
- obj: validSandboxAssets(func(a *ateletpb.SandboxAssets) { a.Assets["amd64"] = file("gs://bucket/gvisor.tar.zstd", "deadbeef") }),
- want: field.ErrorList{field.Invalid(amd64File.Child("sha256"), nil, "")},
- }, {
- name: "sha256 too long",
- obj: validSandboxAssets(func(a *ateletpb.SandboxAssets) {
- a.Assets["amd64"] = file("gs://bucket/gvisor.tar.zstd", testSHA256+"0")
- }),
- want: field.ErrorList{
- field.TooLong(amd64File.Child("sha256"), nil, 64).WithOrigin("maxLength"),
- field.Invalid(amd64File.Child("sha256"), nil, ""),
- },
- }, {
- name: "sha256 with a path escape",
- obj: validSandboxAssets(func(a *ateletpb.SandboxAssets) {
- a.Assets["amd64"] = file("gs://bucket/gvisor.tar.zstd", "../"+testSHA256[3:])
- }),
- want: field.ErrorList{field.Invalid(amd64File.Child("sha256"), nil, "")},
- }}
- for _, tt := range tests {
- t.Run(tt.name, func(t *testing.T) {
- assertValidateErr(t, Validate_SandboxAssets(context.Background(), createOp, nil, tt.obj, nil), tt.want)
- })
- }
-}
-
func TestValidateCheckpointRequest(t *testing.T) {
const snapshotURI = "gs://bucket/root/atespaces/team-a/actors/01234567-89ab-cdef-0123-456789abcdef/snapshots/snap-1"
valid := func(mutate ...func(*ateletpb.CheckpointRequest)) *ateletpb.CheckpointRequest {
@@ -697,30 +575,25 @@ func TestValidateRestoreRequest(t *testing.T) {
Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY,
Object: &ateletpb.ObjectStorage{SnapshotUri: snapshotURI},
},
- SandboxAssets: validSandboxAssets(),
+ SandboxRuntime: runtimeWithAttributes(1),
}
for _, m := range mutate {
m(r)
}
return r
}
- local := func(name string) func(*ateletpb.RestoreRequest) {
- return func(r *ateletpb.RestoreRequest) {
- r.Snapshot.SnapshotUuid = name
- r.Snapshot.Object = nil
- }
- }
tests := []struct {
name string
obj *ateletpb.RestoreRequest
want field.ErrorList
}{{
- name: "valid with object storage",
+ name: "valid",
obj: valid(),
}, {
- name: "valid local only",
- obj: valid(local("pause-snap-1")),
+ name: "missing object storage",
+ obj: valid(func(r *ateletpb.RestoreRequest) { r.Snapshot.Object = nil }),
+ want: field.ErrorList{field.Required(field.NewPath("snapshot", "object"), "")},
}, {
name: "valid volumes fidelity",
obj: valid(func(r *ateletpb.RestoreRequest) {
@@ -783,11 +656,11 @@ func TestValidateRestoreRequest(t *testing.T) {
want: field.ErrorList{field.Required(field.NewPath("snapshot"), "")},
}, {
name: "missing snapshot_uuid",
- obj: valid(local("")),
+ obj: valid(func(r *ateletpb.RestoreRequest) { r.Snapshot.SnapshotUuid = "" }),
want: field.ErrorList{field.Required(field.NewPath("snapshot", "snapshot_uuid"), "")},
}, {
name: "invalid snapshot_uuid: path escape",
- obj: valid(local("../escape")),
+ obj: valid(func(r *ateletpb.RestoreRequest) { r.Snapshot.SnapshotUuid = "../escape" }),
want: field.ErrorList{field.Invalid(field.NewPath("snapshot", "snapshot_uuid"), nil, "").WithOrigin("format=k8s-short-name")},
}, {
name: "missing generation",
@@ -837,24 +710,12 @@ func TestValidateRestoreRequest(t *testing.T) {
obj: valid(func(r *ateletpb.RestoreRequest) { r.EgressGateway = &ateletpb.EgressGateway{} }),
want: field.ErrorList{field.Required(field.NewPath("egress_gateway", "address"), "")},
}, {
- name: "missing sandbox_assets",
- obj: valid(func(r *ateletpb.RestoreRequest) { r.SandboxAssets = nil }),
- want: field.ErrorList{field.Required(field.NewPath("sandbox_assets"), "")},
- }, {
- name: "sandbox_assets errors surface at their path",
- obj: valid(func(r *ateletpb.RestoreRequest) {
- r.SandboxAssets = validSandboxAssets(func(a *ateletpb.SandboxAssets) { a.SandboxClass = "kata" })
- }),
- want: field.ErrorList{field.NotSupported[string](field.NewPath("sandbox_assets", "sandbox_class"), nil, nil)},
- }, {
- name: "valid sandbox_runtime",
- obj: valid(func(r *ateletpb.RestoreRequest) {
- r.SandboxRuntime = runtimeWithAttributes(1)
- }),
+ name: "missing sandbox_runtime",
+ obj: valid(func(r *ateletpb.RestoreRequest) { r.SandboxRuntime = nil }),
+ want: field.ErrorList{field.Required(field.NewPath("sandbox_runtime"), "")},
}, {
name: "sandbox_runtime errors surface at their path",
obj: valid(func(r *ateletpb.RestoreRequest) {
- r.SandboxRuntime = runtimeWithAttributes(1)
r.SandboxRuntime.SandboxClass = "kata"
}),
want: field.ErrorList{field.NotSupported[string](field.NewPath("sandbox_runtime", "sandbox_class"), nil, nil)},
diff --git a/cmd/atelet/internal/apivalidation/zz_generated.validation.go b/cmd/atelet/internal/apivalidation/zz_generated.validation.go
index d69ca8cd5c..b378dde702 100644
--- a/cmd/atelet/internal/apivalidation/zz_generated.validation.go
+++ b/cmd/atelet/internal/apivalidation/zz_generated.validation.go
@@ -171,139 +171,6 @@ func Validate_ActorMetadataItem(
return errs
}
-// Validate_ArchAssets validates an instance of ArchAssets according
-// to declarative validation rules in the API schema.
-func Validate_ArchAssets(
- ctx context.Context, op operation.Operation, fldPath *field.Path,
- obj, oldObj *ateletpb.ArchAssets) (errs field.ErrorList) {
-
- { // field ateletpb.ArchAssets.Files
- fn := func(
- fldPath *field.Path,
- obj, oldObj map[string]*ateletpb.AssetFile,
- oldValueCorrelated bool) (errs field.ErrorList) {
- // don't revalidate unchanged data
- if oldValueCorrelated && op.Type == operation.Update {
- if ateDeepEqual(obj, oldObj) {
- return nil
- }
- }
- // call field-attached validations
- earlyReturn := false
- if e := validate.PtrMapNoNils[string, ateletpb.AssetFile](ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 {
- errs = append(errs, e...)
- earlyReturn = true
- }
- if e := validate.MaxProperties(ctx, op, fldPath, obj, oldObj, 16).MarkShortCircuit(); len(e) != 0 {
- errs = append(errs, e...)
- earlyReturn = true
- }
- if e := validate.RequiredMap(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 {
- errs = append(errs, e...)
- earlyReturn = true
- }
- if earlyReturn {
- return // do not proceed
- }
- if e := validate.EachMapKey(ctx, op, fldPath, obj, oldObj,
- func(ctx context.Context, op operation.Operation, fldPath *field.Path, obj, oldObj *string) field.ErrorList {
- return validate.MaxLength(ctx, op, fldPath, obj, oldObj, 64)
- }); len(e) != 0 {
- errs = append(errs, e...)
- }
- // iterate the map and call the value type's validation function
- if e := validate.EachPtrMapVal(ctx, op, fldPath, obj, oldObj, ateDeepEqual, Validate_AssetFile); len(e) != 0 {
- errs = append(errs, e...)
- }
- return
- }
- oldVal := safe.Field(oldObj,
- func(oldObj *ateletpb.ArchAssets) map[string]*ateletpb.AssetFile {
- return oldObj.Files
- })
- errs = append(errs, fn(fldPath.Child("files"), obj.Files, oldVal, oldObj != nil)...)
- }
-
- return errs
-}
-
-// Validate_AssetFile validates an instance of AssetFile according
-// to declarative validation rules in the API schema.
-func Validate_AssetFile(
- ctx context.Context, op operation.Operation, fldPath *field.Path,
- obj, oldObj *ateletpb.AssetFile) (errs field.ErrorList) {
-
- { // field ateletpb.AssetFile.Url
- fn := func(
- fldPath *field.Path,
- obj, oldObj *string,
- oldValueCorrelated bool) (errs field.ErrorList) {
- // don't revalidate unchanged data
- if oldValueCorrelated && op.Type == operation.Update {
- if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) {
- return nil
- }
- }
- // call field-attached validations
- earlyReturn := false
- if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 {
- errs = append(errs, e...)
- earlyReturn = true
- }
- if earlyReturn {
- return // do not proceed
- }
- if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 2048); len(e) != 0 {
- errs = append(errs, e...)
- }
- return
- }
- oldVal := safe.Field(oldObj,
- func(oldObj *ateletpb.AssetFile) *string {
- return &oldObj.Url
- })
- errs = append(errs, fn(fldPath.Child("url"), &obj.Url, oldVal, oldObj != nil)...)
- }
-
- { // field ateletpb.AssetFile.Sha256
- fn := func(
- fldPath *field.Path,
- obj, oldObj *string,
- oldValueCorrelated bool) (errs field.ErrorList) {
- // don't revalidate unchanged data
- if oldValueCorrelated && op.Type == operation.Update {
- if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) {
- return nil
- }
- }
- // call field-attached validations
- earlyReturn := false
- if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 {
- errs = append(errs, e...)
- earlyReturn = true
- }
- if earlyReturn {
- return // do not proceed
- }
- // custom validation
- if e := ValidateCustom_AssetFile_Sha256(ctx, op, fldPath, obj, oldObj); len(e) != 0 {
- errs = append(errs, e...)
- }
- if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 64); len(e) != 0 {
- errs = append(errs, e...)
- }
- return
- }
- oldVal := safe.Field(oldObj,
- func(oldObj *ateletpb.AssetFile) *string {
- return &oldObj.Sha256
- })
- errs = append(errs, fn(fldPath.Child("sha256"), &obj.Sha256, oldVal, oldObj != nil)...)
- }
-
- return errs
-}
-
// Validate_AttributeEntry validates an instance of AttributeEntry according
// to declarative validation rules in the API schema.
func Validate_AttributeEntry(
@@ -713,21 +580,6 @@ func Validate_CheckpointRequest(
if earlyReturn {
return // do not proceed
}
- func() { // cohort = "object"
- earlyReturn := false
- if e := validate.Subfield(ctx, op, fldPath, obj, oldObj, "object",
- func(o *ateletpb.Snapshot) *ateletpb.ObjectStorage { return o.Object }, ateDeepEqual, validate.RequiredPointer).MarkShortCircuit(); len(e) != 0 {
- errs = append(errs, e...)
- earlyReturn = true
- }
- if e := validate.Subfield(ctx, op, fldPath, obj, oldObj, "object",
- func(o *ateletpb.Snapshot) *ateletpb.ObjectStorage { return o.Object }, ateDeepEqual, validate.OptionalPointer).MarkShortCircuit(); len(e) != 0 {
- earlyReturn = true
- }
- if earlyReturn {
- return // do not proceed
- }
- }()
// call the type's validation function
errs = append(errs, Validate_Snapshot(ctx, op, fldPath, obj, oldObj)...)
return
@@ -2124,21 +1976,6 @@ func Validate_PromoteSnapshotRequest(
if earlyReturn {
return // do not proceed
}
- func() { // cohort = "object"
- earlyReturn := false
- if e := validate.Subfield(ctx, op, fldPath, obj, oldObj, "object",
- func(o *ateletpb.Snapshot) *ateletpb.ObjectStorage { return o.Object }, ateDeepEqual, validate.RequiredPointer).MarkShortCircuit(); len(e) != 0 {
- errs = append(errs, e...)
- earlyReturn = true
- }
- if e := validate.Subfield(ctx, op, fldPath, obj, oldObj, "object",
- func(o *ateletpb.Snapshot) *ateletpb.ObjectStorage { return o.Object }, ateDeepEqual, validate.OptionalPointer).MarkShortCircuit(); len(e) != 0 {
- earlyReturn = true
- }
- if earlyReturn {
- return // do not proceed
- }
- }()
// call the type's validation function
errs = append(errs, Validate_Snapshot(ctx, op, fldPath, obj, oldObj)...)
return
@@ -2264,6 +2101,143 @@ func Validate_RegisterWorkerRequest(
return errs
}
+// Validate_ReportActorCrashedRequest validates an instance of ReportActorCrashedRequest according
+// to declarative validation rules in the API schema.
+func Validate_ReportActorCrashedRequest(
+ ctx context.Context, op operation.Operation, fldPath *field.Path,
+ obj, oldObj *ateletpb.ReportActorCrashedRequest) (errs field.ErrorList) {
+
+ { // field ateletpb.ReportActorCrashedRequest.ActorAtespace
+ fn := func(
+ fldPath *field.Path,
+ obj, oldObj *string,
+ oldValueCorrelated bool) (errs field.ErrorList) {
+ // don't revalidate unchanged data
+ if oldValueCorrelated && op.Type == operation.Update {
+ if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) {
+ return nil
+ }
+ }
+ // call field-attached validations
+ earlyReturn := false
+ if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 {
+ errs = append(errs, e...)
+ earlyReturn = true
+ }
+ if earlyReturn {
+ return // do not proceed
+ }
+ if e := validate.ShortName(ctx, op, fldPath, obj, oldObj); len(e) != 0 {
+ errs = append(errs, e...)
+ }
+ return
+ }
+ oldVal := safe.Field(oldObj,
+ func(oldObj *ateletpb.ReportActorCrashedRequest) *string {
+ return &oldObj.ActorAtespace
+ })
+ errs = append(errs, fn(fldPath.Child("actor_atespace"), &obj.ActorAtespace, oldVal, oldObj != nil)...)
+ }
+
+ { // field ateletpb.ReportActorCrashedRequest.ActorName
+ fn := func(
+ fldPath *field.Path,
+ obj, oldObj *string,
+ oldValueCorrelated bool) (errs field.ErrorList) {
+ // don't revalidate unchanged data
+ if oldValueCorrelated && op.Type == operation.Update {
+ if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) {
+ return nil
+ }
+ }
+ // call field-attached validations
+ earlyReturn := false
+ if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 {
+ errs = append(errs, e...)
+ earlyReturn = true
+ }
+ if earlyReturn {
+ return // do not proceed
+ }
+ if e := validate.ShortName(ctx, op, fldPath, obj, oldObj); len(e) != 0 {
+ errs = append(errs, e...)
+ }
+ return
+ }
+ oldVal := safe.Field(oldObj,
+ func(oldObj *ateletpb.ReportActorCrashedRequest) *string {
+ return &oldObj.ActorName
+ })
+ errs = append(errs, fn(fldPath.Child("actor_name"), &obj.ActorName, oldVal, oldObj != nil)...)
+ }
+
+ { // field ateletpb.ReportActorCrashedRequest.ActorUid
+ fn := func(
+ fldPath *field.Path,
+ obj, oldObj *string,
+ oldValueCorrelated bool) (errs field.ErrorList) {
+ // don't revalidate unchanged data
+ if oldValueCorrelated && op.Type == operation.Update {
+ if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) {
+ return nil
+ }
+ }
+ // call field-attached validations
+ earlyReturn := false
+ if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 {
+ errs = append(errs, e...)
+ earlyReturn = true
+ }
+ if earlyReturn {
+ return // do not proceed
+ }
+ if e := validate.UUID(ctx, op, fldPath, obj, oldObj); len(e) != 0 {
+ errs = append(errs, e...)
+ }
+ return
+ }
+ oldVal := safe.Field(oldObj,
+ func(oldObj *ateletpb.ReportActorCrashedRequest) *string {
+ return &oldObj.ActorUid
+ })
+ errs = append(errs, fn(fldPath.Child("actor_uid"), &obj.ActorUid, oldVal, oldObj != nil)...)
+ }
+
+ { // field ateletpb.ReportActorCrashedRequest.Message
+ fn := func(
+ fldPath *field.Path,
+ obj, oldObj *string,
+ oldValueCorrelated bool) (errs field.ErrorList) {
+ // don't revalidate unchanged data
+ if oldValueCorrelated && op.Type == operation.Update {
+ if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) {
+ return nil
+ }
+ }
+ // call field-attached validations
+ earlyReturn := false
+ if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 {
+ errs = append(errs, e...)
+ earlyReturn = true
+ }
+ if earlyReturn {
+ return // do not proceed
+ }
+ if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 4096); len(e) != 0 {
+ errs = append(errs, e...)
+ }
+ return
+ }
+ oldVal := safe.Field(oldObj,
+ func(oldObj *ateletpb.ReportActorCrashedRequest) *string {
+ return &oldObj.Message
+ })
+ errs = append(errs, fn(fldPath.Child("message"), &obj.Message, oldVal, oldObj != nil)...)
+ }
+
+ return errs
+}
+
// Validate_RequestActorSuspendRequest validates an instance of RequestActorSuspendRequest according
// to declarative validation rules in the API schema.
func Validate_RequestActorSuspendRequest(
@@ -2851,37 +2825,6 @@ func Validate_RestoreRequest(
errs = append(errs, fn(fldPath.Child("memory_bytes"), &obj.MemoryBytes, oldVal, oldObj != nil)...)
}
- { // field ateletpb.RestoreRequest.SandboxAssets
- fn := func(
- fldPath *field.Path,
- obj, oldObj *ateletpb.SandboxAssets,
- oldValueCorrelated bool) (errs field.ErrorList) {
- // don't revalidate unchanged data
- if oldValueCorrelated && op.Type == operation.Update {
- if ateDeepEqual(obj, oldObj) {
- return nil
- }
- }
- // call field-attached validations
- earlyReturn := false
- if e := validate.RequiredPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 {
- errs = append(errs, e...)
- earlyReturn = true
- }
- if earlyReturn {
- return // do not proceed
- }
- // call the type's validation function
- errs = append(errs, Validate_SandboxAssets(ctx, op, fldPath, obj, oldObj)...)
- return
- }
- oldVal := safe.Field(oldObj,
- func(oldObj *ateletpb.RestoreRequest) *ateletpb.SandboxAssets {
- return oldObj.SandboxAssets
- })
- errs = append(errs, fn(fldPath.Child("sandbox_assets"), obj.SandboxAssets, oldVal, oldObj != nil)...)
- }
-
{ // field ateletpb.RestoreRequest.SandboxRuntime
fn := func(
fldPath *field.Path,
@@ -2895,7 +2838,8 @@ func Validate_RestoreRequest(
}
// call field-attached validations
earlyReturn := false
- if e := validate.OptionalPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 {
+ if e := validate.RequiredPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 {
+ errs = append(errs, e...)
earlyReturn = true
}
if earlyReturn {
@@ -3142,37 +3086,6 @@ func Validate_RunRequest(
errs = append(errs, fn(fldPath.Child("spec"), obj.Spec, oldVal, oldObj != nil)...)
}
- { // field ateletpb.RunRequest.SandboxAssets
- fn := func(
- fldPath *field.Path,
- obj, oldObj *ateletpb.SandboxAssets,
- oldValueCorrelated bool) (errs field.ErrorList) {
- // don't revalidate unchanged data
- if oldValueCorrelated && op.Type == operation.Update {
- if ateDeepEqual(obj, oldObj) {
- return nil
- }
- }
- // call field-attached validations
- earlyReturn := false
- if e := validate.RequiredPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 {
- errs = append(errs, e...)
- earlyReturn = true
- }
- if earlyReturn {
- return // do not proceed
- }
- // call the type's validation function
- errs = append(errs, Validate_SandboxAssets(ctx, op, fldPath, obj, oldObj)...)
- return
- }
- oldVal := safe.Field(oldObj,
- func(oldObj *ateletpb.RunRequest) *ateletpb.SandboxAssets {
- return oldObj.SandboxAssets
- })
- errs = append(errs, fn(fldPath.Child("sandbox_assets"), obj.SandboxAssets, oldVal, oldObj != nil)...)
- }
-
{ // field ateletpb.RunRequest.EgressGateway
fn := func(
fldPath *field.Path,
@@ -3281,7 +3194,8 @@ func Validate_RunRequest(
}
// call field-attached validations
earlyReturn := false
- if e := validate.OptionalPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 {
+ if e := validate.RequiredPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 {
+ errs = append(errs, e...)
earlyReturn = true
}
if earlyReturn {
@@ -3301,133 +3215,6 @@ func Validate_RunRequest(
return errs
}
-// Validate_SandboxAssets validates an instance of SandboxAssets according
-// to declarative validation rules in the API schema.
-func Validate_SandboxAssets(
- ctx context.Context, op operation.Operation, fldPath *field.Path,
- obj, oldObj *ateletpb.SandboxAssets) (errs field.ErrorList) {
-
- { // field ateletpb.SandboxAssets.SandboxClass
- fn := func(
- fldPath *field.Path,
- obj, oldObj *string,
- oldValueCorrelated bool) (errs field.ErrorList) {
- // don't revalidate unchanged data
- if oldValueCorrelated && op.Type == operation.Update {
- if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) {
- return nil
- }
- }
- // call field-attached validations
- earlyReturn := false
- if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 {
- errs = append(errs, e...)
- earlyReturn = true
- }
- if earlyReturn {
- return // do not proceed
- }
- // custom validation
- if e := ValidateCustom_SandboxAssets_SandboxClass(ctx, op, fldPath, obj, oldObj); len(e) != 0 {
- errs = append(errs, e...)
- }
- if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 63); len(e) != 0 {
- errs = append(errs, e...)
- }
- return
- }
- oldVal := safe.Field(oldObj,
- func(oldObj *ateletpb.SandboxAssets) *string {
- return &oldObj.SandboxClass
- })
- errs = append(errs, fn(fldPath.Child("sandbox_class"), &obj.SandboxClass, oldVal, oldObj != nil)...)
- }
-
- { // field ateletpb.SandboxAssets.Assets
- fn := func(
- fldPath *field.Path,
- obj, oldObj map[string]*ateletpb.ArchAssets,
- oldValueCorrelated bool) (errs field.ErrorList) {
- // don't revalidate unchanged data
- if oldValueCorrelated && op.Type == operation.Update {
- if ateDeepEqual(obj, oldObj) {
- return nil
- }
- }
- // call field-attached validations
- earlyReturn := false
- if e := validate.PtrMapNoNils[string, ateletpb.ArchAssets](ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 {
- errs = append(errs, e...)
- earlyReturn = true
- }
- if e := validate.MaxProperties(ctx, op, fldPath, obj, oldObj, 8).MarkShortCircuit(); len(e) != 0 {
- errs = append(errs, e...)
- earlyReturn = true
- }
- if e := validate.RequiredMap(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 {
- errs = append(errs, e...)
- earlyReturn = true
- }
- if earlyReturn {
- return // do not proceed
- }
- if e := validate.EachMapKey(ctx, op, fldPath, obj, oldObj,
- func(ctx context.Context, op operation.Operation, fldPath *field.Path, obj, oldObj *string) field.ErrorList {
- return validate.MaxLength(ctx, op, fldPath, obj, oldObj, 16)
- }); len(e) != 0 {
- errs = append(errs, e...)
- }
- // iterate the map and call the value type's validation function
- if e := validate.EachPtrMapVal(ctx, op, fldPath, obj, oldObj, ateDeepEqual, Validate_ArchAssets); len(e) != 0 {
- errs = append(errs, e...)
- }
- return
- }
- oldVal := safe.Field(oldObj,
- func(oldObj *ateletpb.SandboxAssets) map[string]*ateletpb.ArchAssets {
- return oldObj.Assets
- })
- errs = append(errs, fn(fldPath.Child("assets"), obj.Assets, oldVal, oldObj != nil)...)
- }
-
- { // field ateletpb.SandboxAssets.PauseImage
- fn := func(
- fldPath *field.Path,
- obj, oldObj *string,
- oldValueCorrelated bool) (errs field.ErrorList) {
- // don't revalidate unchanged data
- if oldValueCorrelated && op.Type == operation.Update {
- if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) {
- return nil
- }
- }
- // call field-attached validations
- earlyReturn := false
- if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 {
- earlyReturn = true
- }
- if earlyReturn {
- return // do not proceed
- }
- // custom validation
- if e := ValidateCustom_SandboxAssets_PauseImage(ctx, op, fldPath, obj, oldObj); len(e) != 0 {
- errs = append(errs, e...)
- }
- if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 512); len(e) != 0 {
- errs = append(errs, e...)
- }
- return
- }
- oldVal := safe.Field(oldObj,
- func(oldObj *ateletpb.SandboxAssets) *string {
- return &oldObj.PauseImage
- })
- errs = append(errs, fn(fldPath.Child("pause_image"), &obj.PauseImage, oldVal, oldObj != nil)...)
- }
-
- return errs
-}
-
// Validate_SandboxRuntime validates an instance of SandboxRuntime according
// to declarative validation rules in the API schema.
func Validate_SandboxRuntime(
@@ -3661,7 +3448,8 @@ func Validate_Snapshot(
}
// call field-attached validations
earlyReturn := false
- if e := validate.OptionalPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 {
+ if e := validate.RequiredPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 {
+ errs = append(errs, e...)
earlyReturn = true
}
if earlyReturn {
diff --git a/cmd/atelet/internal/filecache/doc.go b/cmd/atelet/internal/filecache/doc.go
index cbea8ef40a..ecdeba9a42 100644
--- a/cmd/atelet/internal/filecache/doc.go
+++ b/cmd/atelet/internal/filecache/doc.go
@@ -34,15 +34,15 @@
//
// # Contracts
//
-// Eviction never invalidates what a caller already received:
+// Eviction never takes what a caller holds:
//
// - GetFileTo serves a hit as a hard link, created in the same locked
-// step that resolves the hit. Evicting the entry later removes only the
-// cache's own link; the consumer's file stays valid.
+// step that resolves the hit. While any consumer's link exists the
+// entry is not evicted: the link is how a consumer declares the entry
+// in use, and removing the cache's own link would free nothing.
// - GetDirTo serves a hit as a directory the consumer owns, in which every
-// file is a hard link, built in that same locked step. The consumer's
-// tree stays valid after eviction, and the entry counts as linked while
-// any of its files is.
+// file is a hard link, built in that same locked step. The entry counts
+// as linked while any of its files is.
// - GetFileCopyTo serves a private copy. The consumer owns the inode and
// may mutate it, and a copy in progress reads a held-open handle, so a
// concurrent eviction cannot corrupt it.
diff --git a/cmd/atelet/internal/filecache/evict.go b/cmd/atelet/internal/filecache/evict.go
index caa15b675b..a6354f4dd9 100644
--- a/cmd/atelet/internal/filecache/evict.go
+++ b/cmd/atelet/internal/filecache/evict.go
@@ -83,20 +83,20 @@ type EvictStats struct {
// run, the entries a real pass would have retired.
Retired int
// FreedBytes counts allocated (on-disk) bytes actually returned to the
- // filesystem: the footprints of retired entries whose data had no
- // consumer hard links left and whose physical removal completed. An
- // entry retired but not removed (a RemoveAll failure, reported in the
- // returned error) is excluded; its bytes sit in a .rm-* dir until
- // SweepDebris. In a dry run this is the would-free estimate.
+ // filesystem: the footprints of retired entries whose physical removal
+ // completed. An entry retired but not removed (a RemoveAll failure,
+ // reported in the returned error) is excluded; its bytes sit in a .rm-*
+ // dir until SweepDebris. In a dry run this is the would-free estimate.
FreedBytes int64
- // PendingBytes counts allocated bytes of retired entries whose data is
- // still hard-linked by a consumer: the cache's claim is gone, but the
- // kernel frees the space only when the last consumer link is removed.
- PendingBytes int64
+ // SkippedLinked counts entries some consumer still hard-links. They are
+ // never evicted: retiring one frees nothing while the link lasts, and
+ // would only make the next caller download a second copy beside it.
+ SkippedLinked int
// SkippedYoung counts entries vetoed by the store's min age.
SkippedYoung int
// SkippedBusy counts entries vetoed at retire time: a hit or a fresh
- // fetch moved their last-use clock after the pass listed them.
+ // fetch moved their last-use clock, or a consumer linked them, after
+ // the pass listed them.
SkippedBusy int
}
@@ -109,13 +109,14 @@ type evictCandidate struct {
linked bool // data is a regular file some consumer still hard-links
}
-// EvictUnused frees cache space until targetBytes of actually-freeable
-// bytes are reclaimed or no eligible entries remain, least-recently-used
-// first. Entries younger than the store's min age are skipped, and a hit or
-// fetch racing the pass vetoes its entry at retire time. Retiring an entry
-// never invalidates files already served from it: a consumer's hard link
-// keeps its bytes (counted as pending until the link goes), so eviction
-// only ever costs the next caller a re-download.
+// EvictUnused frees cache space until targetBytes are reclaimed or no
+// eligible entries remain, least-recently-used first. Only entries no
+// consumer links are eligible: a hard link is how a consumer declares an
+// entry in use, and evicting one would free nothing while the link lasts.
+// Entries younger than the store's min age are skipped too, and a hit or
+// fetch racing the pass vetoes its entry at retire time. Eviction therefore
+// never takes anything a caller holds; it only ever costs the next caller a
+// re-download.
//
// With dryRun, nothing is touched and the stats report what a real pass
// would have chosen. Passes are pressure-driven: the caller decides when
@@ -146,21 +147,20 @@ func (s *Store) EvictUnused(ctx context.Context, targetBytes int64, dryRun bool)
errs = append(errs, err)
}
- // Age gate, then order: entries nobody links first (evicting a linked
- // entry frees nothing now), least recently used within each group.
+ // Linked and age gates, then least recently used first.
now := time.Now()
eligible := candidates[:0]
for _, c := range candidates {
- if now.Sub(c.mtime) < s.minAge {
+ switch {
+ case c.linked:
+ stats.SkippedLinked++
+ case now.Sub(c.mtime) < s.minAge:
stats.SkippedYoung++
- continue
+ default:
+ eligible = append(eligible, c)
}
- eligible = append(eligible, c)
}
sort.Slice(eligible, func(i, j int) bool {
- if eligible[i].linked != eligible[j].linked {
- return !eligible[i].linked
- }
return eligible[i].mtime.Before(eligible[j].mtime)
})
@@ -168,7 +168,7 @@ func (s *Store) EvictUnused(ctx context.Context, targetBytes int64, dryRun bool)
// freed is only credited once physical removal succeeds below.
type retiredEntry struct {
path string
- freed int64 // c.size for unlinked victims, 0 for linked ones
+ freed int64
}
var retired []retiredEntry
var selectedBytes int64
@@ -180,10 +180,6 @@ func (s *Store) EvictUnused(ctx context.Context, targetBytes int64, dryRun bool)
errs = append(errs, err)
break
}
- var freed int64
- if !c.linked {
- freed = c.size
- }
if !dryRun {
rmPath, ok, err := s.retireEntry(c)
if err != nil {
@@ -194,13 +190,12 @@ func (s *Store) EvictUnused(ctx context.Context, targetBytes int64, dryRun bool)
stats.SkippedBusy++
continue
}
- retired = append(retired, retiredEntry{path: rmPath, freed: freed})
+ retired = append(retired, retiredEntry{path: rmPath, freed: c.size})
} else {
- stats.FreedBytes += freed
+ stats.FreedBytes += c.size
}
stats.Retired++
- stats.PendingBytes += c.size - freed
- selectedBytes += freed
+ selectedBytes += c.size
}
// The slow physical deletion happens after all retires, outside hitMu
@@ -329,7 +324,8 @@ func (s *Store) sizeEntry(entryDir string) (size int64, linked bool, err error)
// it retired. It runs inside the key's singleflight (joining any in-flight
// fetch rather than racing it) and takes hitMu exclusively for the final
// re-check and rename, so a hit cannot be served from an entry mid-retire.
-// A moved last-use clock is a veto, not an error.
+// A moved last-use clock, or a link gained since the listing, is a veto,
+// not an error.
func (s *Store) retireEntry(c evictCandidate) (string, bool, error) {
var rmPath string
var retired bool
@@ -348,6 +344,15 @@ func (s *Store) retireEntry(c evictCandidate) (string, bool, error) {
if !fi.ModTime().Equal(c.mtime) {
return nil, nil // hit or refetched since the listing: veto
}
+ // The listing's link check ran lock-free. A hit since then touched
+ // the clock, which the check above catches, but the touch is
+ // best-effort, so ask the files themselves: a linked entry is never
+ // retired, whatever the clock says.
+ if _, linked, err := s.sizeEntry(entryDir); err != nil {
+ return nil, err
+ } else if linked {
+ return nil, nil // linked since the listing: veto
+ }
// Unique suffix: a crashed pass may have left .rm--* behind and
// the key may have been refetched and retired again before a sweep.
p := filepath.Join(s.root, rmPrefix+c.dir+"-"+strconv.FormatInt(time.Now().UnixNano(), 36))
diff --git a/cmd/atelet/internal/filecache/evict_test.go b/cmd/atelet/internal/filecache/evict_test.go
index d5ecc7f11e..7b840e4c2a 100644
--- a/cmd/atelet/internal/filecache/evict_test.go
+++ b/cmd/atelet/internal/filecache/evict_test.go
@@ -128,10 +128,10 @@ func TestEvictUnusedRespectsMinAge(t *testing.T) {
}
}
-func TestEvictUnusedPrefersUnlinkedOverOlderLinked(t *testing.T) {
+func TestEvictUnusedTakesOnlyUnlinkedEntries(t *testing.T) {
s := newTestStore(t, WithMinAge(0))
- // The linked entry is older; pure LRU would take it first. The pass must
- // prefer the unlinked one, whose bytes actually come back.
+ // The linked entry is older; pure LRU would take it first. A link is a
+ // consumer's claim, so the pass takes the unlinked one and nothing else.
linked := cacheEntry(t, s, "linked", "xx", 3*time.Hour, true)
unlinked := cacheEntry(t, s, "unlinked", "yy", time.Hour, false)
@@ -139,18 +139,21 @@ func TestEvictUnusedPrefersUnlinkedOverOlderLinked(t *testing.T) {
if err != nil {
t.Fatalf("EvictUnused: %v", err)
}
- if stats.Retired != 1 || stats.FreedBytes == 0 || stats.PendingBytes != 0 {
- t.Errorf("stats = %+v, want one eviction with freed bytes only", stats)
+ if stats.Retired != 1 || stats.FreedBytes == 0 || stats.SkippedLinked != 1 {
+ t.Errorf("stats = %+v, want one eviction and one linked entry skipped", stats)
}
if entryExists(t, s, unlinked) {
t.Error("unlinked entry survived")
}
if !entryExists(t, s, linked) {
- t.Error("linked entry evicted while an unlinked one satisfied the target")
+ t.Error("linked entry evicted")
}
}
-func TestEvictUnusedLinkedEntryIsSafeForConsumer(t *testing.T) {
+// TestEvictUnusedNeverEvictsALinkedEntry pins the contract a consumer's
+// link buys: however old the entry and however short the pass falls, the
+// entry stays, and the next caller gets a hit rather than a second copy.
+func TestEvictUnusedNeverEvictsALinkedEntry(t *testing.T) {
s := newTestStore(t, WithMinAge(0))
key := URIKey("test://held")
fetch, calls := countingFetcher("held bytes")
@@ -167,23 +170,38 @@ func TestEvictUnusedLinkedEntryIsSafeForConsumer(t *testing.T) {
if err != nil {
t.Fatalf("EvictUnused: %v", err)
}
- if stats.Retired != 1 || stats.FreedBytes != 0 || stats.PendingBytes == 0 {
- t.Errorf("stats = %+v, want one eviction counted as pending bytes", stats)
+ if stats.Retired != 0 || stats.FreedBytes != 0 || stats.SkippedLinked != 1 {
+ t.Errorf("stats = %+v, want nothing evicted and one linked entry skipped", stats)
}
- if entryExists(t, s, key) {
- t.Error("entry still published after eviction")
+ if !entryExists(t, s, key) {
+ t.Error("linked entry evicted")
}
- // The consumer's link is untouched: eviction cost is a re-download,
- // never a broken consumer.
got, err := os.ReadFile(dst)
if err != nil || string(got) != "held bytes" {
- t.Errorf("consumer link after eviction: %q, %v", got, err)
+ t.Errorf("consumer link after the pass: %q, %v", got, err)
}
if err := s.GetFileTo(context.Background(), key, dstPath(t, s, "held2"), fetch); err != nil {
- t.Fatalf("GetFileTo after eviction: %v", err)
+ t.Fatalf("GetFileTo after the pass: %v", err)
+ }
+ if calls.Load() != 1 {
+ t.Errorf("fetch ran %d times, want 1 (the second caller hits)", calls.Load())
+ }
+
+ // Once the consumer's link is gone, the entry is an ordinary candidate.
+ for _, d := range []string{dst, dstPath(t, s, "held2")} {
+ if err := os.Remove(d); err != nil {
+ t.Fatal(err)
+ }
+ }
+ if err := os.Chtimes(s.entryDir(key), stale, stale); err != nil {
+ t.Fatal(err)
+ }
+ stats, err = s.EvictUnused(context.Background(), evictAll, false)
+ if err != nil {
+ t.Fatalf("EvictUnused after unlinking: %v", err)
}
- if calls.Load() != 2 {
- t.Errorf("fetch ran %d times, want 2 (original + post-eviction)", calls.Load())
+ if stats.Retired != 1 || entryExists(t, s, key) {
+ t.Errorf("stats = %+v, entry exists = %v; want the unlinked entry evicted", stats, entryExists(t, s, key))
}
}
@@ -196,8 +214,8 @@ func TestEvictUnusedDryRunTouchesNothing(t *testing.T) {
if err != nil {
t.Fatalf("EvictUnused(dryRun): %v", err)
}
- if stats.Retired != 2 || stats.FreedBytes == 0 || stats.PendingBytes == 0 {
- t.Errorf("stats = %+v, want both entries reported (one freed, one pending)", stats)
+ if stats.Retired != 1 || stats.FreedBytes == 0 || stats.SkippedLinked != 1 {
+ t.Errorf("stats = %+v, want the unlinked entry reported and the linked one skipped", stats)
}
if !entryExists(t, s, a) || !entryExists(t, s, b) {
t.Error("dry run removed entries")
@@ -355,6 +373,46 @@ func TestRetireEntryVetoesWhenLastUseMoved(t *testing.T) {
}
}
+// TestRetireEntryVetoesWhenLinkedSinceListing pins the contract at its
+// last gate: an entry a consumer linked after the pass listed it is not
+// retired even when the link left the last-use clock alone, as a failed
+// best-effort touch would.
+func TestRetireEntryVetoesWhenLinkedSinceListing(t *testing.T) {
+ s := newTestStore(t, WithMinAge(0))
+ key := cacheEntry(t, s, "relinked", "x", time.Hour, false)
+
+ fi, err := os.Stat(s.entryDir(key))
+ if err != nil {
+ t.Fatal(err)
+ }
+ listed := evictCandidate{dir: key.dir, mtime: fi.ModTime(), size: 1}
+
+ // A consumer links the entry out between the listing and the retire,
+ // without the touch a hit makes.
+ held := dstPath(t, s, "held")
+ if err := os.Link(s.dataPath(key), held); err != nil {
+ t.Fatal(err)
+ }
+ rmPath, retired, err := s.retireEntry(listed)
+ if err != nil {
+ t.Fatalf("retireEntry: %v", err)
+ }
+ if retired || rmPath != "" {
+ t.Errorf("retireEntry retired an entry linked since the listing (rmPath=%q)", rmPath)
+ }
+ if !entryExists(t, s, key) {
+ t.Error("linked entry vanished")
+ }
+
+ // The link goes: the same candidate retires.
+ if err := os.Remove(held); err != nil {
+ t.Fatal(err)
+ }
+ if _, retired, err := s.retireEntry(listed); err != nil || !retired {
+ t.Errorf("retireEntry after the link went = retired %v, err %v; want retired", retired, err)
+ }
+}
+
func TestRetireEntryRetiresUnchangedEntry(t *testing.T) {
s := newTestStore(t, WithMinAge(0))
key := cacheEntry(t, s, "stale", "x", time.Hour, false)
@@ -437,7 +495,7 @@ func TestEvictUnusedUnlinkedTreeFreesItsBytes(t *testing.T) {
if err != nil {
t.Fatalf("EvictUnused: %v", err)
}
- if stats.Retired != 1 || stats.FreedBytes != size || stats.PendingBytes != 0 {
+ if stats.Retired != 1 || stats.FreedBytes != size {
t.Errorf("stats = %+v, want one eviction freeing %d bytes", stats, size)
}
if entryExists(t, s, key) {
@@ -446,10 +504,9 @@ func TestEvictUnusedUnlinkedTreeFreesItsBytes(t *testing.T) {
noRetiredLeftovers(t, s)
}
-// TestEvictUnusedPartlyLinkedTreeIsSafeForConsumer pins that a tree counts
-// as linked while a consumer keeps any one of its files, and that evicting
-// it leaves the consumer's files intact.
-func TestEvictUnusedPartlyLinkedTreeIsSafeForConsumer(t *testing.T) {
+// TestEvictUnusedPartlyLinkedTreeIsNotEvicted pins that a tree counts as
+// linked while a consumer keeps any one of its files, and so stays.
+func TestEvictUnusedPartlyLinkedTreeIsNotEvicted(t *testing.T) {
s := newTestStore(t, WithMinAge(0))
key, dst := cacheTree(t, s, "held", time.Hour, []string{"gvisor-bin/LICENSE"})
if _, linked, err := s.sizeEntry(s.entryDir(key)); err != nil || !linked {
@@ -460,11 +517,11 @@ func TestEvictUnusedPartlyLinkedTreeIsSafeForConsumer(t *testing.T) {
if err != nil {
t.Fatalf("EvictUnused: %v", err)
}
- if stats.Retired != 1 || stats.FreedBytes != 0 || stats.PendingBytes == 0 {
- t.Errorf("stats = %+v, want one eviction counted as pending bytes", stats)
+ if stats.Retired != 0 || stats.SkippedLinked != 1 {
+ t.Errorf("stats = %+v, want nothing evicted and the tree skipped as linked", stats)
}
- if entryExists(t, s, key) {
- t.Error("entry still published after eviction")
+ if !entryExists(t, s, key) {
+ t.Error("linked tree evicted")
}
got, err := os.ReadFile(filepath.Join(dst, "gvisor-bin", "LICENSE"))
if err != nil || string(got) != releaseTree["gvisor-bin/LICENSE"].content {
@@ -473,7 +530,7 @@ func TestEvictUnusedPartlyLinkedTreeIsSafeForConsumer(t *testing.T) {
noRetiredLeftovers(t, s)
}
-func TestEvictUnusedPrefersUnlinkedTreeOverOlderLinkedTree(t *testing.T) {
+func TestEvictUnusedTakesOnlyUnlinkedTrees(t *testing.T) {
s := newTestStore(t, WithMinAge(0))
linkedKey, _ := cacheTree(t, s, "old-linked", 3*time.Hour, []string{"runsc"})
unlinkedKey, _ := cacheTree(t, s, "new-unlinked", time.Hour, nil)
diff --git a/cmd/atelet/internal/filecache/getfilecopyto_test.go b/cmd/atelet/internal/filecache/getfilecopyto_test.go
index dc468d530f..4220ad30ac 100644
--- a/cmd/atelet/internal/filecache/getfilecopyto_test.go
+++ b/cmd/atelet/internal/filecache/getfilecopyto_test.go
@@ -140,8 +140,8 @@ func TestGetFileCopyToEvictedEntryRefetches(t *testing.T) {
if err != nil {
t.Fatalf("EvictUnused: %v", err)
}
- if stats.Retired != 1 || stats.PendingBytes != 0 {
- t.Errorf("evicting a copied-out entry: retired=%d pending=%d, want 1/0", stats.Retired, stats.PendingBytes)
+ if stats.Retired != 1 || stats.SkippedLinked != 0 {
+ t.Errorf("evicting a copied-out entry: retired=%d skipped linked=%d, want 1/0", stats.Retired, stats.SkippedLinked)
}
// The earlier copy is untouched, and the next copy refetches.
diff --git a/cmd/atelet/internal/filecachegc/filecachegc.go b/cmd/atelet/internal/filecachegc/filecachegc.go
new file mode 100644
index 0000000000..89753ab157
--- /dev/null
+++ b/cmd/atelet/internal/filecachegc/filecachegc.go
@@ -0,0 +1,317 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package filecachegc runs the eviction loop of a node-local file cache: a
+// filecache store with a byte budget, evicted least recently used first,
+// down to a low-water fraction of the budget, never taking an entry a
+// consumer still links. One loop per cache, one set of flags per cache under
+// the cache's flag prefix, and one set of instruments for all of them,
+// labeled by cache name.
+package filecachegc
+
+import (
+ "context"
+ "fmt"
+ "log/slog"
+ "runtime/debug"
+ "time"
+
+ "github.com/spf13/pflag"
+ "go.opentelemetry.io/otel/metric"
+
+ "go.ate.dev/substrate/cmd/atelet/internal/filecache"
+ "go.ate.dev/substrate/internal/ateattr"
+)
+
+// lowWater is the fraction of the budget a pass evicts down to once the
+// cache is over it, so passes do not fire on every tick while usage hovers
+// at the budget.
+const lowWater = 0.9
+
+const (
+ // shortfallWarnLimit is how many consecutive shortfalls warn before the
+ // loop backs off to shortfallReminderEvery.
+ shortfallWarnLimit = 3
+ // shortfallReminderEvery keeps a persistent shortfall visible without a
+ // line per tick (at the 5m default: roughly hourly).
+ shortfallReminderEvery = 12
+)
+
+const (
+ sizeMetric = "ate.filecache.size"
+ evictionsMetric = "ate.filecache.evictions"
+ gcPassesMetric = "ate.filecache.gc.passes"
+)
+
+// Store is what the loop needs from a *filecache.Store, a seam so a pass's
+// skip and recovery paths are testable without a real cache.
+type Store interface {
+ TotalBytes(ctx context.Context) (int64, error)
+ EvictUnused(ctx context.Context, targetBytes int64, dryRun bool) (filecache.EvictStats, error)
+}
+
+// Flags are one cache's eviction flags: ---gc-period,
+// ---max-bytes and ---gc-dry-run.
+type Flags struct {
+ Period *time.Duration
+ MaxBytes *int64
+ DryRun *bool
+ prefix string
+}
+
+// NewFlags registers the eviction flags of the cache called what, for
+// example "sandbox asset cache", under prefix, for example
+// "sandbox-asset-cache", on fs.
+func NewFlags(fs *pflag.FlagSet, prefix, what string) *Flags {
+ return &Flags{
+ prefix: prefix,
+ Period: fs.Duration(prefix+"-gc-period", 5*time.Minute, fmt.Sprintf("How often to run the %s eviction pass. 0 disables it.", what)),
+ MaxBytes: fs.Int64(prefix+"-max-bytes", 0, fmt.Sprintf("Budget for the %s, in bytes of disk the cached files occupy. A pass that finds the cache over it evicts least recently used entries that nothing links, down to %d%% of it. 0 means no budget: the cache only grows.", what, int(lowWater*100))),
+ DryRun: fs.Bool(prefix+"-gc-dry-run", false, fmt.Sprintf("Compute and log eviction decisions for the %s without deleting anything.", what)),
+ }
+}
+
+// Validate rejects values the loop could not act on.
+func (f *Flags) Validate() error {
+ if *f.Period < 0 {
+ return fmt.Errorf("--%s-gc-period %v must be >= 0", f.prefix, *f.Period)
+ }
+ if *f.MaxBytes < 0 {
+ return fmt.Errorf("--%s-max-bytes %d must be >= 0", f.prefix, *f.MaxBytes)
+ }
+ return nil
+}
+
+// target computes the bytes a pass should free: nothing while the cache is
+// within its budget, otherwise enough to bring it down to the low-water
+// fraction of the budget. A zero budget never asks for anything.
+func target(total, maxBytes int64) int64 {
+ if maxBytes <= 0 || total <= maxBytes {
+ return 0
+ }
+ target := total - int64(float64(maxBytes)*lowWater)
+ if target > total {
+ target = total
+ }
+ return target
+}
+
+// Instruments are the metrics of every file cache on the node, one set
+// labeled by cache name: how much disk each holds, how many entries its
+// passes evict, and how each pass ended. Nil-safe: a nil receiver records
+// nothing.
+type Instruments struct {
+ meter metric.Meter
+ size metric.Int64ObservableUpDownCounter
+ evictions metric.Int64Counter
+ passes metric.Int64Counter
+}
+
+// NewInstruments creates the instruments on meter, once per process.
+func NewInstruments(meter metric.Meter) (*Instruments, error) {
+ size, err := meter.Int64ObservableUpDownCounter(
+ sizeMetric,
+ metric.WithUnit("By"),
+ metric.WithDescription("Bytes of disk a node-local file cache occupies, as the sum of its entries' allocated sizes, by cache."),
+ )
+ if err != nil {
+ return nil, fmt.Errorf("create %s updowncounter: %w", sizeMetric, err)
+ }
+ evictions, err := meter.Int64Counter(
+ evictionsMetric,
+ metric.WithUnit("{entry}"),
+ metric.WithDescription("Number of entries eviction passes removed from a node-local file cache, by cache."),
+ )
+ if err != nil {
+ return nil, fmt.Errorf("create %s counter: %w", evictionsMetric, err)
+ }
+ passes, err := meter.Int64Counter(
+ gcPassesMetric,
+ metric.WithUnit("{pass}"),
+ metric.WithDescription("Number of eviction passes over a node-local file cache that had work to do, by cache and outcome."),
+ )
+ if err != nil {
+ return nil, fmt.Errorf("create %s counter: %w", gcPassesMetric, err)
+ }
+ return &Instruments{meter: meter, size: size, evictions: evictions, passes: passes}, nil
+}
+
+// observe has the size of store reported under name on every collection.
+// The callback walks the store's entries; that is a few hundred stats on a
+// full node. Called whether or not the cache has a budget, so the size has
+// a series on every install.
+func (i *Instruments) observe(name string, store Store) error {
+ if i == nil {
+ return nil
+ }
+ attrs := metric.WithAttributes(ateattr.FileCacheNameKey.String(name))
+ _, err := i.meter.RegisterCallback(func(ctx context.Context, o metric.Observer) error {
+ total, err := store.TotalBytes(ctx)
+ if err != nil {
+ return err
+ }
+ o.ObserveInt64(i.size, total, attrs)
+ return nil
+ }, i.size)
+ if err != nil {
+ return fmt.Errorf("register %s callback for %s: %w", sizeMetric, name, err)
+ }
+ return nil
+}
+
+// recordPass records one pass over name that had work to do, with what it
+// evicted.
+func (i *Instruments) recordPass(ctx context.Context, name, outcome string, evicted int) {
+ if i == nil {
+ return
+ }
+ i.passes.Add(ctx, 1, metric.WithAttributes(ateattr.FileCacheNameKey.String(name), ateattr.FileCacheGCOutcomeKey.String(outcome)))
+ if evicted > 0 {
+ i.evictions.Add(ctx, int64(evicted), metric.WithAttributes(ateattr.FileCacheNameKey.String(name)))
+ }
+}
+
+// gc is one cache's eviction loop: configuration snapshotted from the flags
+// at construction, and the shortfall backoff.
+type gc struct {
+ name string
+ store Store
+ instruments *Instruments
+ period time.Duration
+ maxBytes int64
+ dryRun bool
+
+ consecutiveShortfalls int
+}
+
+// newGC returns the loop for the cache called name, a value of
+// ateattr.FileCacheNameKey.
+func newGC(name string, store Store, flags *Flags, instruments *Instruments) *gc {
+ return &gc{
+ name: name,
+ store: store,
+ instruments: instruments,
+ period: *flags.Period,
+ maxBytes: *flags.MaxBytes,
+ dryRun: *flags.DryRun,
+ }
+}
+
+// Start validates the cache's flags, reports its size through instruments,
+// and runs its eviction loop in the background when the flags give it a
+// budget and a period. Without a budget it warns once: the cache then only
+// grows.
+func Start(ctx context.Context, name string, store Store, flags *Flags, instruments *Instruments) error {
+ if err := flags.Validate(); err != nil {
+ return err
+ }
+ if err := instruments.observe(name, store); err != nil {
+ return err
+ }
+ switch {
+ case *flags.MaxBytes == 0:
+ slog.WarnContext(ctx, "File cache has no budget; it only grows", slog.String("cache", name), slog.String("flag", "--"+flags.prefix+"-max-bytes"))
+ case *flags.Period > 0:
+ go newGC(name, store, flags, instruments).run(ctx)
+ }
+ return nil
+}
+
+// run executes eviction passes on the configured period until ctx is done,
+// the first one immediately so a node booting over budget does not wait a
+// full period. Passes are strictly serialized.
+func (g *gc) run(ctx context.Context) {
+ g.runPass(ctx)
+ ticker := time.NewTicker(g.period)
+ defer ticker.Stop()
+ for {
+ select {
+ case <-ctx.Done():
+ return
+ case <-ticker.C:
+ }
+ g.runPass(ctx)
+ }
+}
+
+// runPass performs one pass. It recovers from panics: this is a background
+// janitor, and a bug here must not take atelet down with it and strand every
+// actor on the node.
+func (g *gc) runPass(ctx context.Context) {
+ defer func() {
+ if r := recover(); r != nil {
+ slog.ErrorContext(ctx, "File cache GC pass panicked; skipping this pass",
+ slog.String("cache", g.name), slog.Any("panic", r), slog.String("stack", string(debug.Stack())))
+ }
+ }()
+
+ total, err := g.store.TotalBytes(ctx)
+ if err != nil {
+ // Fail toward retention: retry next tick.
+ slog.WarnContext(ctx, "File cache GC: sizing the cache failed; skipping this pass", slog.String("cache", g.name), slog.Any("err", err))
+ g.instruments.recordPass(ctx, g.name, ateattr.FileCacheGCOutcomeSkipped, 0)
+ return
+ }
+ target := target(total, g.maxBytes)
+ if target == 0 {
+ // Within budget: nothing to do, nothing to say.
+ g.consecutiveShortfalls = 0
+ return
+ }
+
+ tStart := time.Now()
+ stats, err := g.store.EvictUnused(ctx, target, g.dryRun)
+ attrs := []any{
+ slog.String("cache", g.name),
+ slog.Int64("cache_bytes", total),
+ slog.Int64("max_bytes", g.maxBytes),
+ slog.Int64("target_bytes", target),
+ slog.Int64("freed_bytes", stats.FreedBytes),
+ slog.Int("evicted", stats.Retired),
+ slog.Int("skipped_linked", stats.SkippedLinked),
+ slog.Int("skipped_young", stats.SkippedYoung),
+ slog.Int("skipped_busy", stats.SkippedBusy),
+ slog.Bool("dry_run", g.dryRun),
+ slog.Duration("took", time.Since(tStart)),
+ }
+ // A dry run retires nothing: its Retired is what a real pass would have
+ // taken, which is logged above and must not count as an eviction.
+ evicted := stats.Retired
+ if g.dryRun {
+ evicted = 0
+ }
+ if err != nil {
+ // Per-entry failures; each retries next pass.
+ slog.WarnContext(ctx, "File cache GC pass finished with errors", append(attrs, slog.Any("err", err))...)
+ }
+ if stats.FreedBytes < target {
+ // Everything left is linked, young, or busy: the remaining usage is
+ // not the cache's to free. Warn on the first few, then a periodic
+ // reminder, never a line per tick.
+ g.consecutiveShortfalls++
+ switch {
+ case g.consecutiveShortfalls <= shortfallWarnLimit:
+ slog.WarnContext(ctx, "File cache GC could not reach its budget",
+ append(attrs, slog.Int("consecutive", g.consecutiveShortfalls))...)
+ case g.consecutiveShortfalls%shortfallReminderEvery == 0:
+ slog.WarnContext(ctx, "File cache GC still over its budget; what remains is linked by a consumer, younger than the min age, or busy",
+ append(attrs, slog.Int("consecutive", g.consecutiveShortfalls))...)
+ }
+ g.instruments.recordPass(ctx, g.name, ateattr.FileCacheGCOutcomeShortfall, evicted)
+ return
+ }
+ g.consecutiveShortfalls = 0
+ slog.InfoContext(ctx, "File cache GC pass complete", attrs...)
+ g.instruments.recordPass(ctx, g.name, ateattr.FileCacheGCOutcomeComplete, evicted)
+}
diff --git a/cmd/atelet/internal/filecachegc/filecachegc_test.go b/cmd/atelet/internal/filecachegc/filecachegc_test.go
new file mode 100644
index 0000000000..da8c3759cd
--- /dev/null
+++ b/cmd/atelet/internal/filecachegc/filecachegc_test.go
@@ -0,0 +1,361 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package filecachegc
+
+import (
+ "context"
+ "errors"
+ "testing"
+ "time"
+
+ "github.com/spf13/pflag"
+ sdkmetric "go.opentelemetry.io/otel/sdk/metric"
+ "go.opentelemetry.io/otel/sdk/metric/metricdata"
+
+ "go.ate.dev/substrate/cmd/atelet/internal/filecache"
+ "go.ate.dev/substrate/internal/ateattr"
+)
+
+const testCache = "test_cache"
+
+func TestTarget(t *testing.T) {
+ for _, tc := range []struct {
+ name string
+ total, maxBytes int64
+ want int64
+ }{
+ {"no budget", 500, 0, 0},
+ {"within budget", 500, 1000, 0},
+ {"at budget", 1000, 1000, 0},
+ {"over budget frees down to the low water", 1500, 1000, 600},
+ {"far over budget frees down to the low water", 10_000, 1000, 9100},
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ if got := target(tc.total, tc.maxBytes); got != tc.want {
+ t.Errorf("Target(%d, %d) = %d, want %d", tc.total, tc.maxBytes, got, tc.want)
+ }
+ })
+ }
+}
+
+func TestFlagsValidate(t *testing.T) {
+ f := NewFlags(pflag.NewFlagSet("test", pflag.ContinueOnError), "test-cache", "test cache")
+ for _, tc := range []struct {
+ name string
+ period time.Duration
+ maxBytes int64
+ wantErr bool
+ }{
+ {"defaults", *f.Period, *f.MaxBytes, false},
+ {"disabled period", 0, 1 << 30, false},
+ {"negative period", -time.Second, 1 << 30, true},
+ {"negative budget", time.Minute, -1, true},
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ *f.Period, *f.MaxBytes = tc.period, tc.maxBytes
+ if err := f.Validate(); (err != nil) != tc.wantErr {
+ t.Errorf("Validate() = %v, wantErr %v", err, tc.wantErr)
+ }
+ })
+ }
+}
+
+// fakeStore stands in for the cache: a size to report and a canned eviction
+// result.
+type fakeStore struct {
+ size int64
+ sizeErr error
+ sizeCalls int
+ evictCalls int
+ gotTarget int64
+ gotDryRun bool
+ stats filecache.EvictStats
+ evictErr error
+ panicOnEvict bool
+}
+
+func (f *fakeStore) TotalBytes(context.Context) (int64, error) {
+ f.sizeCalls++
+ return f.size, f.sizeErr
+}
+
+func (f *fakeStore) EvictUnused(_ context.Context, target int64, dryRun bool) (filecache.EvictStats, error) {
+ f.evictCalls++
+ f.gotTarget = target
+ f.gotDryRun = dryRun
+ if f.panicOnEvict {
+ panic("boom")
+ }
+ return f.stats, f.evictErr
+}
+
+// newTestGC returns a loop over fake with instruments on a manual reader, so
+// a test can assert what a pass recorded.
+func newTestGC(t *testing.T, fake *fakeStore, maxBytes int64) (*gc, *sdkmetric.ManualReader) {
+ t.Helper()
+ reader := sdkmetric.NewManualReader()
+ mp := sdkmetric.NewMeterProvider(sdkmetric.WithReader(reader))
+ inst, err := NewInstruments(mp.Meter("atelet"))
+ if err != nil {
+ t.Fatalf("NewInstruments: %v", err)
+ }
+ if err := inst.observe(testCache, fake); err != nil {
+ t.Fatalf("Observe: %v", err)
+ }
+ return &gc{name: testCache, store: fake, instruments: inst, maxBytes: maxBytes, period: time.Hour}, reader
+}
+
+// collectMetric returns the named metric from one collection, or nil if it
+// was not emitted.
+func collectMetric(t *testing.T, reader *sdkmetric.ManualReader, name string) *metricdata.Metrics {
+ t.Helper()
+ var rm metricdata.ResourceMetrics
+ if err := reader.Collect(context.Background(), &rm); err != nil {
+ t.Fatalf("collect: %v", err)
+ }
+ for _, sm := range rm.ScopeMetrics {
+ for i := range sm.Metrics {
+ if sm.Metrics[i].Name == name {
+ return &sm.Metrics[i]
+ }
+ }
+ }
+ return nil
+}
+
+// sumByOutcome returns a counter's data points keyed by the GC outcome,
+// checking that every point carries the cache's name.
+func sumByOutcome(t *testing.T, m *metricdata.Metrics) map[string]int64 {
+ t.Helper()
+ sum, ok := m.Data.(metricdata.Sum[int64])
+ if !ok || !sum.IsMonotonic {
+ t.Fatalf("%s is %T (monotonic=%v), want a monotonic int64 sum", m.Name, m.Data, ok && sum.IsMonotonic)
+ }
+ out := map[string]int64{}
+ for _, dp := range sum.DataPoints {
+ if name, ok := dp.Attributes.Value(ateattr.FileCacheNameKey); !ok || name.AsString() != testCache {
+ t.Fatalf("%s datapoint without the cache's name: %v", m.Name, dp.Attributes.ToSlice())
+ }
+ v, ok := dp.Attributes.Value(ateattr.FileCacheGCOutcomeKey)
+ if !ok {
+ t.Fatalf("%s datapoint without an outcome: %v", m.Name, dp.Attributes.ToSlice())
+ }
+ out[v.AsString()] += dp.Value
+ }
+ return out
+}
+
+func TestPassWithinBudgetDoesNothing(t *testing.T) {
+ fake := &fakeStore{size: 500}
+ g, reader := newTestGC(t, fake, 1000)
+ g.runPass(context.Background())
+ if fake.evictCalls != 0 {
+ t.Errorf("evictCalls = %d, want 0 within budget", fake.evictCalls)
+ }
+ if m := collectMetric(t, reader, gcPassesMetric); m != nil {
+ t.Errorf("a quiet pass recorded %v, want nothing", m)
+ }
+}
+
+func TestPassSkipsWhenSizingFails(t *testing.T) {
+ fake := &fakeStore{sizeErr: errors.New("unreadable")}
+ g, reader := newTestGC(t, fake, 1000)
+ g.runPass(context.Background())
+ if fake.evictCalls != 0 {
+ t.Errorf("evictCalls = %d, want 0 when the cache cannot be sized", fake.evictCalls)
+ }
+ fake.sizeErr = nil // so the size callback does not fail the collection
+ m := collectMetric(t, reader, gcPassesMetric)
+ if m == nil {
+ t.Fatal("skipped pass not recorded")
+ }
+ if got := sumByOutcome(t, m); got[ateattr.FileCacheGCOutcomeSkipped] != 1 || len(got) != 1 {
+ t.Errorf("passes by outcome = %v, want one skipped", got)
+ }
+}
+
+func TestPassEvictsDownToTheLowWater(t *testing.T) {
+ fake := &fakeStore{size: 1500, stats: filecache.EvictStats{Retired: 3, FreedBytes: 600}}
+ g, reader := newTestGC(t, fake, 1000)
+ g.runPass(context.Background())
+ if fake.evictCalls != 1 || fake.gotTarget != 600 || fake.gotDryRun {
+ t.Errorf("evict call: calls=%d target=%d dryRun=%v, want 1/600/false", fake.evictCalls, fake.gotTarget, fake.gotDryRun)
+ }
+ passes := collectMetric(t, reader, gcPassesMetric)
+ if passes == nil {
+ t.Fatal("pass not recorded")
+ }
+ if got := sumByOutcome(t, passes); got[ateattr.FileCacheGCOutcomeComplete] != 1 || len(got) != 1 {
+ t.Errorf("passes by outcome = %v, want one complete", got)
+ }
+ evictions := collectMetric(t, reader, evictionsMetric)
+ if evictions == nil {
+ t.Fatal("evictions not recorded")
+ }
+ sum, ok := evictions.Data.(metricdata.Sum[int64])
+ if !ok || len(sum.DataPoints) != 1 || sum.DataPoints[0].Value != 3 {
+ t.Fatalf("evictions = %+v, want a single datapoint of 3", evictions.Data)
+ }
+ if name, ok := sum.DataPoints[0].Attributes.Value(ateattr.FileCacheNameKey); !ok || name.AsString() != testCache || sum.DataPoints[0].Attributes.Len() != 1 {
+ t.Errorf("evictions labels = %v, want the cache's name only", sum.DataPoints[0].Attributes.ToSlice())
+ }
+ if evictions.Unit != "{entry}" || passes.Unit != "{pass}" {
+ t.Errorf("units = %q / %q, want {entry} / {pass}", evictions.Unit, passes.Unit)
+ }
+}
+
+// TestDryRunCountsNoEvictions pins the dry run's metrics: the pass is
+// recorded with its outcome, since it ran and decided, but the entries it
+// would have taken are not evictions, because none was removed.
+func TestDryRunCountsNoEvictions(t *testing.T) {
+ fake := &fakeStore{size: 1500, stats: filecache.EvictStats{Retired: 3, FreedBytes: 600}}
+ g, reader := newTestGC(t, fake, 1000)
+ g.dryRun = true
+ g.runPass(context.Background())
+ if fake.evictCalls != 1 || !fake.gotDryRun {
+ t.Errorf("evict call: calls=%d dryRun=%v, want 1/true", fake.evictCalls, fake.gotDryRun)
+ }
+ passes := collectMetric(t, reader, gcPassesMetric)
+ if passes == nil {
+ t.Fatal("pass not recorded")
+ }
+ if got := sumByOutcome(t, passes); got[ateattr.FileCacheGCOutcomeComplete] != 1 || len(got) != 1 {
+ t.Errorf("passes by outcome = %v, want one complete", got)
+ }
+ if evictions := collectMetric(t, reader, evictionsMetric); evictions != nil {
+ t.Errorf("evictions = %+v after a dry run, want none recorded", evictions.Data)
+ }
+}
+
+func TestPassShortfall(t *testing.T) {
+ // Everything left is linked: the pass frees less than asked.
+ fake := &fakeStore{size: 1500, stats: filecache.EvictStats{Retired: 1, FreedBytes: 100, SkippedLinked: 4}}
+ g, reader := newTestGC(t, fake, 1000)
+ for range 3 {
+ g.runPass(context.Background())
+ }
+ if g.consecutiveShortfalls != 3 {
+ t.Errorf("consecutiveShortfalls = %d, want 3", g.consecutiveShortfalls)
+ }
+ if got := sumByOutcome(t, collectMetric(t, reader, gcPassesMetric)); got[ateattr.FileCacheGCOutcomeShortfall] != 3 {
+ t.Errorf("passes by outcome = %v, want three shortfalls", got)
+ }
+ // A pass that meets the budget resets the streak.
+ fake.size = 500
+ g.runPass(context.Background())
+ if g.consecutiveShortfalls != 0 {
+ t.Errorf("consecutiveShortfalls after a quiet pass = %d, want 0", g.consecutiveShortfalls)
+ }
+}
+
+// TestSizeIsObservedPerCache pins the size instrument: one series per cache
+// name, read from each store on collection.
+func TestSizeIsObservedPerCache(t *testing.T) {
+ fake := &fakeStore{size: 4096}
+ _, reader := newTestGC(t, fake, 0)
+ m := collectMetric(t, reader, sizeMetric)
+ if m == nil {
+ t.Fatal("size not observed")
+ }
+ sum, ok := m.Data.(metricdata.Sum[int64])
+ if !ok || sum.IsMonotonic || len(sum.DataPoints) != 1 || sum.DataPoints[0].Value != 4096 {
+ t.Fatalf("size = %+v, want a non-monotonic int64 sum of 4096", m.Data)
+ }
+ if m.Unit != "By" {
+ t.Errorf("size unit = %q, want By", m.Unit)
+ }
+ if name, ok := sum.DataPoints[0].Attributes.Value(ateattr.FileCacheNameKey); !ok || name.AsString() != testCache || sum.DataPoints[0].Attributes.Len() != 1 {
+ t.Errorf("size labels = %v, want the cache's name only", sum.DataPoints[0].Attributes.ToSlice())
+ }
+}
+
+// TestTwoCachesShareTheInstruments pins that a second cache on the same
+// instruments gets its own series under its own name.
+func TestTwoCachesShareTheInstruments(t *testing.T) {
+ reader := sdkmetric.NewManualReader()
+ mp := sdkmetric.NewMeterProvider(sdkmetric.WithReader(reader))
+ inst, err := NewInstruments(mp.Meter("atelet"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ a, b := &fakeStore{size: 1}, &fakeStore{size: 2}
+ for name, store := range map[string]*fakeStore{"a": a, "b": b} {
+ if err := inst.observe(name, store); err != nil {
+ t.Fatal(err)
+ }
+ }
+ m := collectMetric(t, reader, sizeMetric)
+ if m == nil {
+ t.Fatal("size not observed")
+ }
+ got := map[string]int64{}
+ for _, dp := range m.Data.(metricdata.Sum[int64]).DataPoints {
+ name, _ := dp.Attributes.Value(ateattr.FileCacheNameKey)
+ got[name.AsString()] = dp.Value
+ }
+ if got["a"] != 1 || got["b"] != 2 || len(got) != 2 {
+ t.Errorf("size by cache = %v, want a=1 b=2", got)
+ }
+}
+
+// TestStartRejectsInvalidFlags pins that Start validates the flags itself,
+// so a caller cannot start a loop on values it would silently ignore: a
+// negative period with a budget would leave the cache unbounded unnoticed.
+func TestStartRejectsInvalidFlags(t *testing.T) {
+ f := NewFlags(pflag.NewFlagSet("test", pflag.ContinueOnError), "test-cache", "test cache")
+ *f.Period, *f.MaxBytes = -time.Second, 1<<30
+ reader := sdkmetric.NewManualReader()
+ inst, err := NewInstruments(sdkmetric.NewMeterProvider(sdkmetric.WithReader(reader)).Meter("atelet"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ fake := &fakeStore{size: 1}
+ if err := Start(context.Background(), testCache, fake, f, inst); err == nil {
+ t.Fatal("Start with a negative period = nil, want an error")
+ }
+ if m := collectMetric(t, reader, sizeMetric); m != nil {
+ t.Error("Start observed the size of a cache whose flags it rejected")
+ }
+}
+
+func TestRunFirstPassIsImmediate(t *testing.T) {
+ // A cancelled context and an hour-long period: the single call can
+ // only be the immediate first pass, never a tick.
+ fake := &fakeStore{size: 1500}
+ g, _ := newTestGC(t, fake, 1000)
+ ctx, cancel := context.WithCancel(context.Background())
+ cancel()
+ g.run(ctx)
+ if fake.evictCalls != 1 {
+ t.Errorf("evictCalls = %d, want exactly 1 (the immediate first pass)", fake.evictCalls)
+ }
+}
+
+func TestRunTicks(t *testing.T) {
+ fake := &fakeStore{size: 1500}
+ g, _ := newTestGC(t, fake, 1000)
+ g.period = 10 * time.Millisecond
+ ctx, cancel := context.WithTimeout(context.Background(), 150*time.Millisecond)
+ defer cancel()
+ g.run(ctx)
+ if fake.evictCalls < 2 {
+ t.Errorf("evictCalls = %d, want >= 2 (first pass plus at least one tick)", fake.evictCalls)
+ }
+}
+
+func TestPassRecoversPanic(t *testing.T) {
+ g, _ := newTestGC(t, &fakeStore{size: 1500, panicOnEvict: true}, 1000)
+ g.runPass(context.Background()) // must not propagate the panic
+}
diff --git a/cmd/atelet/internal/snapshottracker/actor.go b/cmd/atelet/internal/snapshottracker/actor.go
index 13c5a5d6ba..b72a8af303 100644
--- a/cmd/atelet/internal/snapshottracker/actor.go
+++ b/cmd/atelet/internal/snapshottracker/actor.go
@@ -180,6 +180,17 @@ type actorSnapshotsRecord struct {
ReportDueSince time.Time `json:"reportDueSince,omitzero"`
}
+// holdKind is the kind of RPC in progress on an actor, or holdNone. Callers
+// serialize lifecycle RPCs per actor, so at most one holds an actor at once.
+type holdKind int
+
+const (
+ holdNone holdKind = iota
+ holdStart // Run or Restore
+ holdCheckpoint // Checkpoint
+ holdPromote // PromoteSnapshot
+)
+
// actorSnapshots is what atelet tracks of one actor's snapshots on this node.
// Its fields are guarded by the actor's lock in Tracker.locks.
type actorSnapshots struct {
@@ -193,20 +204,13 @@ type actorSnapshots struct {
// A background worker or an RPC call such as Detach runs it; an actor has
// at most one. The entry stays in memory while op is set.
op *actorOp
- // holds counts the Runs, Restores, Checkpoints, and PromoteSnapshots of
- // the actor in progress on this node. While it is nonzero, atelet does not
- // detach the actor or report its state, and the entry stays in memory.
- holds int
- // inFlight holds the snapshots of Checkpoints and PromoteSnapshots of the
- // actor currently in progress on this node. newest includes them, so an
- // older generation cannot start while a newer one is in flight.
- inFlight []*SnapshotRef
- // promotes counts the PromoteSnapshots of the actor in progress on this
- // node. While it is nonzero, atelet does not start a background upload of
- // the actor.
- promotes int
+ // hold records the Run, Restore, Checkpoint, or PromoteSnapshot of the
+ // actor in progress on this node. While set, atelet does not detach the
+ // actor or report its state (nor start a background upload during a
+ // promote), and the entry stays in memory.
+ hold holdKind
// reportDeferred is set when a report of the actor's state is held back
- // while holds is nonzero, so holdLocked queues it once holds reaches zero.
+ // while hold is set, so holdLocked queues it once the RPC ends.
reportDeferred bool
// retryAt holds the actor's next background upload or detach back after
// one failed. The entry stays in memory while retryAt is set.
@@ -393,8 +397,7 @@ func (a *actorSnapshots) opTarget() *SnapshotRef {
}
// newest returns the actor's snapshot with the highest generation, or nil
-// when atelet holds no snapshot for the actor. It includes snapshots whose
-// Checkpoint or PromoteSnapshot is in flight.
+// when atelet holds no snapshot for the actor.
func (a *actorSnapshots) newest() *SnapshotRef {
var newest *SnapshotRef
consider := func(r *SnapshotRef) {
@@ -406,13 +409,10 @@ func (a *actorSnapshots) newest() *SnapshotRef {
consider(a.Uploading)
consider(a.LatestDurable)
consider(a.opTarget())
- for _, r := range a.inFlight {
- consider(r)
- }
return newest
}
-// checkGeneration returns a FailedPrecondition error if snap has a stale
+// checkGeneration returns an Aborted error if snap has a stale
// generation for the actor: below the actor's newest snapshot, or equal to
// it under another snapshot_uuid.
func (a *actorSnapshots) checkGeneration(snap *ateletpb.Snapshot) error {
@@ -422,9 +422,9 @@ func (a *actorSnapshots) checkGeneration(snap *ateletpb.Snapshot) error {
}
switch {
case snap.GetGeneration() < newest.generation():
- return apierror.FailedPrecondition("snapshot %q has generation %d, below the actor's newest snapshot %q at generation %d", snap.GetSnapshotUuid(), snap.GetGeneration(), newest.uuid(), newest.generation())
+ return apierror.Aborted("snapshot %q has generation %d, below the actor's newest snapshot %q at generation %d", snap.GetSnapshotUuid(), snap.GetGeneration(), newest.uuid(), newest.generation())
case snap.GetGeneration() == newest.generation() && snap.GetSnapshotUuid() != newest.uuid():
- return apierror.FailedPrecondition("snapshot %q has generation %d, which the actor's newest snapshot %q already holds", snap.GetSnapshotUuid(), snap.GetGeneration(), newest.uuid())
+ return apierror.Aborted("snapshot %q has generation %d, which the actor's newest snapshot %q already holds", snap.GetSnapshotUuid(), snap.GetGeneration(), newest.uuid())
default:
return nil
}
diff --git a/cmd/atelet/internal/snapshottracker/background.go b/cmd/atelet/internal/snapshottracker/background.go
index cc8a551428..4cc4945841 100644
--- a/cmd/atelet/internal/snapshottracker/background.go
+++ b/cmd/atelet/internal/snapshottracker/background.go
@@ -221,7 +221,7 @@ func (t *Tracker) beginDueWork(actorUID string, kind opKind) func() {
switch kind {
case opDetach:
if due, ok := t.detachDueLocked(a); ok && !due.After(now) {
- if a.holds > 0 {
+ if a.hold != holdNone {
// A Run, Restore, Checkpoint, or PromoteSnapshot of the actor
// is in progress, and queues the actor again when it ends
// (see holdLocked).
@@ -234,7 +234,7 @@ func (t *Tracker) beginDueWork(actorUID string, kind opKind) func() {
}
case opUpload:
if due, ok := t.uploadDueLocked(a); ok && !due.After(now) {
- if a.promotes > 0 {
+ if a.hold == holdPromote {
// A PromoteSnapshot of the actor is in progress, and queues
// the actor again when it ends (see holdLocked).
return nil
@@ -287,7 +287,7 @@ func (t *Tracker) requeueLocked(a *actorSnapshots) {
if due, ok := t.detachDueLocked(a); ok {
t.queues[opDetach].AddAfter(a.actorUID, due.Sub(now))
}
- if !supersededUploading && a.op == nil && a.holds == 0 && a.retryAt.IsZero() && !a.uploadingFailed && !a.unsaved {
+ if !supersededUploading && a.op == nil && a.hold == holdNone && a.retryAt.IsZero() && !a.uploadingFailed && !a.unsaved {
t.dropLocked(a.actorUID)
}
}
@@ -386,16 +386,10 @@ func (t *Tracker) endOp(actorUID string, op *actorOp, opErr error) (*ateletpb.Ac
// operation in progress (whether started by a background worker or an atelet
// RPC), and returns the actor's entry, which may be nil. It releases the
// actor's lock while it waits, so the actor can change meanwhile, and holds
-// it again when it returns. check, if not nil, vets the entry on every pass:
-// an error from it is returned at once, without waiting further.
-func (t *Tracker) waitIdleLocked(ctx context.Context, actorUID string, check func(*actorSnapshots) error) (*actorSnapshots, error) {
+// it again when it returns.
+func (t *Tracker) waitIdleLocked(ctx context.Context, actorUID string) (*actorSnapshots, error) {
for {
a := t.findLocked(actorUID)
- if a != nil && check != nil {
- if err := check(a); err != nil {
- return nil, err
- }
- }
if a == nil || a.op == nil {
return a, nil
}
diff --git a/cmd/atelet/internal/snapshottracker/detach.go b/cmd/atelet/internal/snapshottracker/detach.go
index b2f60b78ad..85a2c8d59c 100644
--- a/cmd/atelet/internal/snapshottracker/detach.go
+++ b/cmd/atelet/internal/snapshottracker/detach.go
@@ -38,12 +38,12 @@ func (t *Tracker) beginDetachLocked(ctx context.Context, a *actorSnapshots, back
// Detach makes the actor's latest local snapshot durable, then deletes the
// actor's local snapshots. It is a no-op when atelet holds no local snapshot
-// for the actor. It fails with Aborted while a Run, Restore, Checkpoint, or
-// PromoteSnapshot of the actor is in progress on this node, or once a Run or
-// Restore of the actor has begun since its latest local snapshot, and with
-// FailedPrecondition when no detach can make the snapshot durable. It runs in
-// the calling goroutine rather than through the background queue, once any
-// operation in progress on the actor has ended.
+// for the actor. It fails with Aborted once a Run or Restore of the actor has
+// begun since its latest local snapshot, and with FailedPrecondition when no
+// detach can make the snapshot durable. It runs in the calling goroutine
+// rather than through the background queue, once any operation in progress on
+// the actor has ended. Callers serialize it with the actor's other lifecycle
+// RPCs (see Tracker).
func (t *Tracker) Detach(ctx context.Context, actorUID string) (*ateletpb.ActorSnapshotState, error) {
if t.store == nil {
return nil, apierror.Unimplemented("atelet has no snapshot store to detach actors with")
@@ -52,7 +52,7 @@ func (t *Tracker) Detach(ctx context.Context, actorUID string) (*ateletpb.ActorS
// Wait for any operation in progress (from a background worker or
// another RPC call) to end first: a detach may leave nothing to detach,
// and an upload of the latest local snapshot nothing to upload.
- a, err := t.waitIdleLocked(ctx, actorUID, checkNotHeld)
+ a, err := t.waitIdleLocked(ctx, actorUID)
if err != nil {
t.unlock(actorUID)
return nil, err
@@ -84,15 +84,6 @@ func (t *Tracker) Detach(ctx context.Context, actorUID string) (*ateletpb.ActorS
return finish()
}
-// checkNotHeld fails with Aborted while a Run, Restore, Checkpoint, or
-// PromoteSnapshot of the actor is in progress on this node.
-func checkNotHeld(a *actorSnapshots) error {
- if a.holds > 0 {
- return apierror.Aborted("actor %s is in use on this node", a.actorUID)
- }
- return nil
-}
-
// finishDetach runs op, a detach of the actor's latest local snapshot: it
// uploads the snapshot unless already durable, then deletes the actor's local
// snapshots. background reports whether a worker runs it, rather than a Detach
@@ -126,8 +117,6 @@ func (t *Tracker) finishDetach(ctx context.Context, actorUID string, op *actorOp
// safe because the snapshot is durable: Restore reads snapshot.object
// when the local copy is unusable. The prune runs with op in progress
// (before op.done is closed), so Run, Restore, and Checkpoint wait for it.
- // TODO: Notify ateapi of the upcoming detach before deleting the actor's
- // local snapshots, so it stops sending Restores that rely on them.
recordDetach := err == nil
if recordDetach {
detachStart = t.clock.Now()
diff --git a/cmd/atelet/internal/snapshottracker/detach_test.go b/cmd/atelet/internal/snapshottracker/detach_test.go
index 64a5e69603..c8d2a0eea3 100644
--- a/cmd/atelet/internal/snapshottracker/detach_test.go
+++ b/cmd/atelet/internal/snapshottracker/detach_test.go
@@ -284,17 +284,6 @@ func TestTrackerDetach(t *testing.T) {
},
wantCode: codes.Aborted,
},
- {
- name: "refused while a checkpoint is in progress",
- setup: func(t *testing.T, tr *Tracker) {
- end, err := tr.BeginCheckpoint(t.Context(), trackedActorUID, testRef("s2", 2, time.Time{}).Snapshot)
- if err != nil {
- t.Fatalf("beginCheckpoint: %v", err)
- }
- t.Cleanup(end)
- },
- wantCode: codes.Aborted,
- },
{
name: "refused when durable generation mismatches local",
setup: func(t *testing.T, tr *Tracker) {
@@ -461,8 +450,8 @@ func TestTrackerStartFailsIfItCannotSave(t *testing.T) {
if got := detachDueOf(tr, trackedActorUID); !got.Equal(due) {
t.Errorf("detachDue after the failed start = %v, want %v kept", got, due)
}
- if holds := tr.actors[trackedActorUID].holds; holds != 0 {
- t.Errorf("holds after the failed start = %d, want none", holds)
+ if hold := tr.actors[trackedActorUID].hold; hold != holdNone {
+ t.Errorf("hold after the failed start = %v, want none", hold)
}
// Once saves work again, the start goes ahead.
@@ -481,56 +470,65 @@ func TestTrackerStartFailsIfItCannotSave(t *testing.T) {
// A restore or a checkpoint that arrives during a detach waits for it to end.
func TestTrackerWaitsForDetach(t *testing.T) {
- store := newFakeSnapshotStore()
- store.setBlocking(true)
- tr := newTestTracker(t, store)
- tr.CommitLocal(trackedActorUID, testRef("s1", 1, tr.clock.Now()))
- results := map[string]<-chan error{"detach": startDetach(t, tr)}
- waitStarted(t, store)
-
- for name, begin := range map[string]func(context.Context, string) (func(), error){
- "beginStart": tr.BeginStart,
- "beginCheckpoint": func(ctx context.Context, uid string) (func(), error) {
- return tr.BeginCheckpoint(ctx, uid, testRef("s2", 2, time.Time{}).Snapshot)
+ for _, tc := range []struct {
+ name string
+ begin func(tr *Tracker, ctx context.Context, uid string) (func(), error)
+ }{
+ {
+ name: "BeginStart",
+ begin: func(tr *Tracker, ctx context.Context, uid string) (func(), error) {
+ return tr.BeginStart(ctx, uid)
+ },
+ },
+ {
+ name: "BeginCheckpoint",
+ begin: func(tr *Tracker, ctx context.Context, uid string) (func(), error) {
+ return tr.BeginCheckpoint(ctx, uid, testRef("s2", 2, time.Time{}).Snapshot)
+ },
},
} {
- result := make(chan error, 1)
- results[name] = result
- go func() {
- end, err := begin(t.Context(), trackedActorUID)
- if err == nil {
- end()
+ t.Run(tc.name, func(t *testing.T) {
+ store := newFakeSnapshotStore()
+ store.setBlocking(true)
+ tr := newTestTracker(t, store)
+ tr.CommitLocal(trackedActorUID, testRef("s1", 1, tr.clock.Now()))
+ detachResult := startDetach(t, tr)
+ waitStarted(t, store)
+
+ result := make(chan error, 1)
+ go func() {
+ end, err := tc.begin(tr, t.Context(), trackedActorUID)
+ if err == nil {
+ end()
+ }
+ result <- err
+ }()
+ select {
+ case err := <-result:
+ t.Fatalf("%s returned (%v) during a detach", tc.name, err)
+ case <-time.After(50 * time.Millisecond):
}
- result <- err
- }()
- }
- for name, result := range results {
- select {
- case err := <-result:
- t.Fatalf("%s returned (%v) during a detach", name, err)
- case <-time.After(50 * time.Millisecond):
- }
- }
- store.release <- nil
- for name, result := range results {
- select {
- case err := <-result:
- if err != nil {
- t.Errorf("%s: %v", name, err)
+ store.release <- nil
+ for name, ch := range map[string]<-chan error{"detach": detachResult, tc.name: result} {
+ select {
+ case err := <-ch:
+ if err != nil {
+ t.Errorf("%s: %v", name, err)
+ }
+ case <-time.After(10 * time.Second):
+ t.Fatalf("%s did not return after the detach", name)
+ }
}
- case <-time.After(10 * time.Second):
- t.Fatalf("%s did not return after the detach", name)
- }
- }
- if got := tr.State(trackedActorUID).GetLatestLocal(); got != nil {
- t.Errorf("latest_local = %v after the detach, want unset", got)
+ if got := tr.State(trackedActorUID).GetLatestLocal(); got != nil {
+ t.Errorf("latest_local = %v after the detach, want unset", got)
+ }
+ })
}
}
// BeginCheckpoint and BeginPromote fail at once during a detach when the
-// requested snapshot is already stale, and fail once the detach ends when a
-// newer snapshot was committed while they waited.
+// requested snapshot is already stale.
func TestTrackerDetachWaitChecksGeneration(t *testing.T) {
store := newFakeSnapshotStore()
store.setBlocking(true)
@@ -541,60 +539,20 @@ func TestTrackerDetachWaitChecksGeneration(t *testing.T) {
stale := testRef("s0", 1, time.Time{})
returnsPromptly(t, "BeginCheckpoint of stale generation during detach", func() {
- if _, err := tr.BeginCheckpoint(t.Context(), trackedActorUID, stale.Snapshot); apierror.Code(err) != codes.FailedPrecondition {
- t.Errorf("BeginCheckpoint = %v, want FailedPrecondition", err)
+ if _, err := tr.BeginCheckpoint(t.Context(), trackedActorUID, stale.Snapshot); apierror.Code(err) != codes.Aborted {
+ t.Errorf("BeginCheckpoint = %v, want Aborted", err)
}
})
returnsPromptly(t, "BeginPromote of stale generation during detach", func() {
- if _, _, err := tr.BeginPromote(t.Context(), trackedActorUID, stale); apierror.Code(err) != codes.FailedPrecondition {
- t.Errorf("BeginPromote = %v, want FailedPrecondition", err)
+ if _, _, err := tr.BeginPromote(t.Context(), trackedActorUID, stale); apierror.Code(err) != codes.Aborted {
+ t.Errorf("BeginPromote = %v, want Aborted", err)
}
})
- superseded := testRef("s2", 3, time.Time{})
- results := make(map[string]chan error)
- for name, begin := range map[string]func() (func(), error){
- "beginCheckpoint": func() (func(), error) {
- return tr.BeginCheckpoint(t.Context(), trackedActorUID, superseded.Snapshot)
- },
- "beginPromote": func() (func(), error) {
- _, end, err := tr.BeginPromote(t.Context(), trackedActorUID, superseded)
- return end, err
- },
- } {
- result := make(chan error, 1)
- results[name] = result
- go func() {
- end, err := begin()
- if err == nil {
- end()
- }
- result <- err
- }()
- }
- for name, result := range results {
- select {
- case err := <-result:
- t.Fatalf("%s returned (%v) during a detach", name, err)
- case <-time.After(50 * time.Millisecond):
- }
- }
-
- tr.CommitDurable(trackedActorUID, testRef("s3", 4, tr.clock.Now()))
store.release <- nil
if err := <-detached; err != nil {
t.Fatalf("detach: %v", err)
}
- for name, result := range results {
- select {
- case err := <-result:
- if apierror.Code(err) != codes.FailedPrecondition {
- t.Errorf("%s = %v, want FailedPrecondition", name, err)
- }
- case <-time.After(10 * time.Second):
- t.Fatalf("%s did not return after the detach", name)
- }
- }
}
// A failed detach keeps the local snapshot, and is retried no sooner than the
@@ -692,20 +650,24 @@ func TestTrackerDetachWithoutStore(t *testing.T) {
// actor as it began, as on the actor's first checkpoint on this node.
func TestTrackerHoldsFirstCheckpoint(t *testing.T) {
store := newFakeSnapshotStore()
- tr := newTestTracker(t, store)
+ tr := newIdleTestTracker(t, store)
s1 := testRef("s1", 1, tr.clock.Now())
end, err := tr.BeginCheckpoint(t.Context(), trackedActorUID, s1.Snapshot)
if err != nil {
t.Fatalf("beginCheckpoint: %v", err)
}
tr.CommitLocal(trackedActorUID, s1)
- if _, err := tr.Detach(t.Context(), trackedActorUID); apierror.Code(err) != codes.Aborted {
- t.Errorf("detach during the checkpoint = %v, want Aborted", err)
+ stepTo(tr, detachDueOf(tr, trackedActorUID))
+ if run := tr.beginDueWork(trackedActorUID, opDetach); run != nil {
+ t.Error("a due detach began during the checkpoint, want it held off")
+ run()
}
end()
- if _, err := tr.Detach(t.Context(), trackedActorUID); err != nil {
- t.Errorf("detach after the checkpoint: %v", err)
+ run := tr.beginDueWork(trackedActorUID, opDetach)
+ if run == nil {
+ t.Fatal("no detach began after the checkpoint, want the due detach")
}
+ run()
}
// A start when no upload is due, or a checkpoint replacing the local snapshot,
@@ -748,31 +710,6 @@ func TestTrackerLiftsBackoffWithDetach(t *testing.T) {
}
}
-// Forget during a Run, Restore, or Checkpoint drops the actor for good: the
-// start's end leaves it forgotten.
-func TestTrackerForgetDuringStart(t *testing.T) {
- store := newFakeSnapshotStore()
- tr := newTestTracker(t, store)
- tr.CommitLocal(trackedActorUID, testRef("s1", 1, tr.clock.Now()))
- end, err := tr.BeginStart(t.Context(), trackedActorUID)
- if err != nil {
- t.Fatalf("beginStart: %v", err)
- }
- if err := tr.Forget(trackedActorUID); err != nil {
- t.Fatalf("Forget: %v", err)
- }
- end()
- if st := tr.State(trackedActorUID); !proto.Equal(st, &ateletpb.ActorSnapshotState{}) {
- t.Errorf("state = %v, want empty", st)
- }
- if _, err := os.Stat(ateletpath.ActorSnapshotStateFile(trackedActorUID)); !os.IsNotExist(err) {
- t.Errorf("state file after the start ended: stat = %v, want none", err)
- }
- if inMemory(tr) {
- t.Error("entry in memory after Forget, want none")
- }
-}
-
// Forget does not wait for a detach in progress: it cancels the detach, which
// fails and leaves the forgotten actor's local snapshots for Terminate to
// prune.
diff --git a/cmd/atelet/internal/snapshottracker/lifecycle.go b/cmd/atelet/internal/snapshottracker/lifecycle.go
index 0144bcc938..07c6c6dcd8 100644
--- a/cmd/atelet/internal/snapshottracker/lifecycle.go
+++ b/cmd/atelet/internal/snapshottracker/lifecycle.go
@@ -16,12 +16,10 @@ package snapshottracker
import (
"context"
- "slices"
"time"
"go.ate.dev/substrate/internal/apierror"
"go.ate.dev/substrate/internal/proto/ateletpb"
- "google.golang.org/protobuf/proto"
)
// CommitLocal records snap, just stored in the actor's local checkpoint
@@ -127,8 +125,7 @@ func (t *Tracker) CommitPromoted(actorUID string, snap SnapshotRef) *ateletpb.Ac
defer t.unlock(actorUID)
a := t.findOrAddLocked(actorUID)
// Skip when snap is already LatestDurable (so a retried promote keeps the
- // original CommitTime and UploadTime) or older than LatestDurable (a
- // newer checkpoint became durable while PromoteSnapshot ran).
+ // original CommitTime and UploadTime) or older than LatestDurable.
if !a.uploadNeeded(snap) {
if a.LatestLocal != nil && a.reportDue() {
t.clearReportDueLocked(a)
@@ -156,15 +153,15 @@ func (t *Tracker) CommitPromoted(actorUID string, snap SnapshotRef) *ateletpb.Ac
}
// BeginStart records that a Run or Restore is starting the actor on this node,
-// once any detach in progress has finished, and returns the func to call when
-// the start ends, whether or not it succeeded. It first unschedules the
-// actor's detach (see markStarted) and saves that, so atelet does not detach
-// the actor while a workload restored from its local snapshot may use it, even
-// if atelet restarts or the start fails partway. A failed Run or Restore
-// leaves the detach unscheduled, since atelet cannot tell whether the workload
-// is running on the worker; the local snapshot stays until the next checkpoint
-// or Terminate on this node. When it cannot save that, it fails with
-// Unavailable, and the caller must not start the actor.
+// once any background detach of the actor in progress has finished, and
+// returns the func to call when the start ends, whether or not it succeeded.
+// It first unschedules the actor's detach (see markStarted) and saves that, so
+// atelet does not detach the actor while a workload restored from its local
+// snapshot may use it, even if atelet restarts or the start fails partway. A
+// failed Run or Restore leaves the detach unscheduled, since atelet cannot
+// tell whether the workload is running on the worker; the local snapshot stays
+// until the next checkpoint or Terminate on this node. When it cannot save
+// that, it fails with Unavailable, and the caller must not start the actor.
func (t *Tracker) BeginStart(ctx context.Context, actorUID string) (end func(), err error) {
t.lock(actorUID)
defer t.unlock(actorUID)
@@ -189,22 +186,20 @@ func (t *Tracker) BeginStart(ctx context.Context, actorUID string) (end func(),
t.resetBackoffLocked(a)
}
}
- return t.holdLocked(actorUID, a, nil, false), nil
+ return t.holdLocked(actorUID, a, holdStart), nil
}
// BeginCheckpoint records that a Checkpoint of snap is in progress on this
-// node, once any detach in progress has finished, and returns the func to call
-// when the checkpoint ends, whether or not it succeeded. It fails with
-// FailedPrecondition when snap has a stale generation for the actor: below the
+// node, once any background detach in progress has finished, and returns the
+// func to call when the checkpoint ends, whether or not it succeeded. It fails
+// with Aborted when snap has a stale generation for the actor: below the
// actor's newest snapshot on this node, or equal to it under another
// snapshot_uuid. atelet does not detach the actor in between, so a detach
// cannot delete the local snapshot the checkpoint writes.
func (t *Tracker) BeginCheckpoint(ctx context.Context, actorUID string, snap *ateletpb.Snapshot) (end func(), err error) {
t.lock(actorUID)
defer t.unlock(actorUID)
- a, err := t.waitForDetachLocked(ctx, actorUID, func(a *actorSnapshots) error {
- return a.checkGeneration(snap)
- })
+ a, err := t.waitForDetachLocked(ctx, actorUID, func(a *actorSnapshots) error { return a.checkGeneration(snap) })
if err != nil {
return nil, err
}
@@ -212,16 +207,16 @@ func (t *Tracker) BeginCheckpoint(ctx context.Context, actorUID string, snap *at
// zero, because the actor's Run or Restore on this node unscheduled the
// detach, or this is a retry of a Checkpoint that already committed its
// snapshot, whose detach stays scheduled.
- return t.holdLocked(actorUID, a, &SnapshotRef{Snapshot: proto.CloneOf(snap)}, false), nil
+ return t.holdLocked(actorUID, a, holdCheckpoint), nil
}
// BeginPromote records that a PromoteSnapshot of snap is in progress on this
-// node, once any detach in progress and any background upload of snap have
-// finished. It fails with FailedPrecondition when snap has a stale generation
-// for the actor or conflicts with LatestDurable. It reports whether snap is
-// already the actor's latest durable snapshot, and returns the func to call
-// when the promote ends. atelet does not start a detach or a background upload
-// of the actor in between.
+// node, once any background detach in progress and any background upload of
+// snap have finished. A stale generation fails with Aborted. A conflicting
+// description of LatestDurable fails with FailedPrecondition. It reports
+// whether snap is already the actor's latest durable snapshot, and returns the
+// func to call when the promote ends. atelet does not start a detach or a
+// background upload of the actor in between.
func (t *Tracker) BeginPromote(ctx context.Context, actorUID string, snap SnapshotRef) (alreadyDurable bool, end func(), err error) {
t.lock(actorUID)
defer t.unlock(actorUID)
@@ -235,45 +230,26 @@ func (t *Tracker) BeginPromote(ctx context.Context, actorUID string, snap Snapsh
return false, nil, err
}
}
- return alreadyDurable, t.holdLocked(actorUID, a, snap.clone(), true), nil
+ return alreadyDurable, t.holdLocked(actorUID, a, holdPromote), nil
}
-// holdLocked counts a Run, Restore, Checkpoint, or PromoteSnapshot of the
+// holdLocked records a Run, Restore, Checkpoint, or PromoteSnapshot of the
// actor in progress on its entry a, which holds detaches and state reports of
-// the actor off (and background uploads when promote is true), and returns the
-// func that ends it and queues the actor again. When snap is non-nil
-// (Checkpoint and PromoteSnapshot), it stays in a.inFlight until end runs, so
-// newest includes it. An actor atelet tracks nothing of yet (a nil a) is held
-// too, so no detach or report can begin on a snapshot its first Checkpoint
-// commits before it ends.
-func (t *Tracker) holdLocked(actorUID string, a *actorSnapshots, snap *SnapshotRef, promote bool) (end func()) {
+// the actor off (and background uploads during a promote), and returns the
+// func that ends it and queues the actor again. An actor atelet tracks nothing
+// of yet (a nil a) is held too, so no detach or report can begin on a snapshot
+// its first Checkpoint commits before it ends.
+func (t *Tracker) holdLocked(actorUID string, a *actorSnapshots, kind holdKind) (end func()) {
if a == nil {
a = &actorSnapshots{actorUID: actorUID}
}
t.addLocked(a)
- a.holds++
- if snap != nil {
- a.inFlight = append(a.inFlight, snap)
- }
- if promote {
- a.promotes++
- }
+ a.hold = kind
return func() {
t.lock(actorUID)
defer t.unlock(actorUID)
- a.holds--
- if snap != nil {
- a.inFlight = slices.DeleteFunc(a.inFlight, func(r *SnapshotRef) bool { return r == snap })
- }
- if promote {
- a.promotes--
- }
- if !t.tracksLocked(a) {
- // Forget (via Terminate) dropped the actor while the Run, Restore,
- // Checkpoint, or PromoteSnapshot was in progress.
- return
- }
- if a.holds == 0 && a.reportDeferred {
+ a.hold = holdNone
+ if a.reportDeferred {
t.queueReportLocked(a)
}
t.requeueLocked(a)
diff --git a/cmd/atelet/internal/snapshottracker/queues_test.go b/cmd/atelet/internal/snapshottracker/queues_test.go
index 9c626d62ee..e144f64ab8 100644
--- a/cmd/atelet/internal/snapshottracker/queues_test.go
+++ b/cmd/atelet/internal/snapshottracker/queues_test.go
@@ -173,10 +173,10 @@ func TestTrackerSameActorUploadAndDetachDue(t *testing.T) {
})
}
-// A Checkpoint hold (a.holds > 0) keeps opDetach from pruning local snapshots
-// underneath the checkpoint, while opUpload still runs in the background. When
-// the hold ends, opDetach prunes the now-durable snapshot without uploading
-// again.
+// A Checkpoint hold (a.hold != holdNone) keeps opDetach from pruning local
+// snapshots underneath the checkpoint, while opUpload still runs in the
+// background. When the hold ends, opDetach prunes the now-durable snapshot
+// without uploading again.
func TestTrackerUploadRunsDuringCheckpointHoldWhileDetachWaits(t *testing.T) {
store := newFakeSnapshotStore()
tr := newIdleTestTracker(t, store)
diff --git a/cmd/atelet/internal/snapshottracker/report.go b/cmd/atelet/internal/snapshottracker/report.go
index ba7bdc8d5e..6778908e3f 100644
--- a/cmd/atelet/internal/snapshottracker/report.go
+++ b/cmd/atelet/internal/snapshottracker/report.go
@@ -124,7 +124,7 @@ func (t *Tracker) queueReportLocked(a *actorSnapshots) {
if t.reporting.reporter == nil || !a.reportDue() {
return
}
- if a.holds > 0 {
+ if a.hold != holdNone {
a.reportDeferred = true
return
}
@@ -255,7 +255,7 @@ func (t *Tracker) beginReport(actorUID string) (resources.ActorRef, *ateletpb.Ac
t.reporting.retries.Forget(actorUID)
return resources.ActorRef{}, nil, time.Time{}, false
}
- if a.holds > 0 {
+ if a.hold != holdNone {
// A Run, Restore, Checkpoint, or PromoteSnapshot of the actor is in
// progress, and queues the report when it ends (see holdLocked).
a.reportDeferred = true
diff --git a/cmd/atelet/internal/snapshottracker/tracker.go b/cmd/atelet/internal/snapshottracker/tracker.go
index 61650c3dbd..93b2448e27 100644
--- a/cmd/atelet/internal/snapshottracker/tracker.go
+++ b/cmd/atelet/internal/snapshottracker/tracker.go
@@ -44,6 +44,10 @@ type Store interface {
// or PromoteSnapshot does any work, and commits never replace a snapshot with
// an older generation. It uploads and detaches them in the background.
//
+// Callers serialize lifecycle RPCs (Run, Restore, Checkpoint, PromoteSnapshot,
+// DetachActor, and Terminate) per actor before calling Tracker, so at most one
+// RPC runs for an actor at a time.
+//
// Background work runs off two delaying workqueues of actor UIDs, one for
// uploads and one for detaches, each drained by a fixed pool of workers of its
// own, so slow uploads cannot hold up a due detach, nor detaches an upload.
@@ -72,8 +76,8 @@ type Store interface {
// operation in progress (see actorOp): a worker that finds one leaves the
// actor to that operation's end, which queues it again.
// 5. An upload leaves LatestLocal on disk so a same-node Restore still
-// reads it. A detach, in the background or via the Detach RPC, makes
-// LatestLocal durable and then deletes all the actor's local snapshots.
+// reads it. A detach, in the background or via Detach, makes LatestLocal
+// durable and then deletes all the actor's local snapshots.
// 6. A Run or Restore unschedules the actor's detach as it begins
// (BeginStart), and saves that before the workload can start, so atelet
// does not detach an actor whose start has begun, even after a restart or
@@ -90,19 +94,20 @@ type Store interface {
// control plane, so it cancels any pending report. Report workers, apart
// from the upload and detach workers, spread saved reports across a
// startup window, send the actor's whole state through the Reporter, and
-// retry with per-actor and
-// queue-wide backoff until the control plane acknowledges or refuses it,
-// a call cancels it, or reporting.giveUpAfter passes. A report waits while
-// a Run, Restore, Checkpoint, or PromoteSnapshot is in progress, and a
-// background change made while a report ran is reported again.
+// retry with per-actor and queue-wide backoff until the control plane
+// acknowledges or refuses it, a call cancels it, or reporting.giveUpAfter
+// passes. A report waits while a Run, Restore, Checkpoint, or
+// PromoteSnapshot is in progress, and a background change made while a
+// report ran is reported again.
//
-// The Detach RPC runs in the calling goroutine rather than on a worker, so an
-// explicit control-plane call does not wait behind other actors' background
-// work. It does wait for the actor's own operation in progress, even an
-// upload of an older snapshot; Run, Restore, and Checkpoint wait for a detach
-// in progress; and PromoteSnapshot waits for a detach or a background upload
-// of the same snapshot. A worker leaves an actor with an operation in
-// progress to that operation's end, which queues the actor again.
+// Detach runs in the calling goroutine rather than on a worker, so an explicit
+// control-plane call does not wait behind other actors' background work. It
+// does wait for the actor's own background operation in progress, even an
+// upload of an older snapshot; Run, Restore, and Checkpoint wait for a
+// background detach in progress; and PromoteSnapshot waits for a background
+// detach or a background upload of the same snapshot. A worker leaves an actor
+// with an operation in progress to that operation's end, which queues the
+// actor again.
type Tracker struct {
// ctx bounds background work, and shuts the queues down when it ends.
ctx context.Context
@@ -143,9 +148,9 @@ type Tracker struct {
// workersDone counts the workers still running.
workersDone sync.WaitGroup
- // locks holds each actor's lock. Work on an actor, including its snapshot
- // state file I/O, holds that actor's lock, so it never holds up another
- // actor.
+ // locks holds each actor's state lock. Short critical sections on an
+ // actor, including its snapshot state file I/O, hold that actor's lock, so
+ // they never hold up another actor.
locks *actorlock.Locks
// mu guards the actors map, and is never held during file I/O.
@@ -153,7 +158,7 @@ type Tracker struct {
// actors maps actorUID to the entries requeueLocked keeps in memory (see
// requeueLocked). requeueLocked drops any other entry, and findLocked then
// reads the state from the file. Read and change it through findLocked,
- // findOrAddLocked, addLocked, dropLocked, tracksLocked, and heldLocked.
+ // findOrAddLocked, addLocked, dropLocked, and heldLocked.
actors map[string]*actorSnapshots
}
@@ -310,14 +315,6 @@ func (t *Tracker) dropLocked(actorUID string) {
delete(t.actors, actorUID)
}
-// tracksLocked reports whether a is still its actor's entry in memory: false
-// once Forget has dropped the actor, even if a new entry has been added since.
-func (t *Tracker) tracksLocked(a *actorSnapshots) bool {
- t.mu.Lock()
- defer t.mu.Unlock()
- return t.actors[a.actorUID] == a
-}
-
// heldLocked reports whether a Run, Restore, Checkpoint, or PromoteSnapshot of
// the actor is in progress. An entry with one stays in memory, so this never
// reads the snapshot state file.
@@ -325,7 +322,7 @@ func (t *Tracker) heldLocked(actorUID string) bool {
t.mu.Lock()
defer t.mu.Unlock()
a := t.actors[actorUID]
- return a != nil && a.holds > 0
+ return a != nil && a.hold != holdNone
}
// uploadDueLocked returns when the actor's background upload is due, as
diff --git a/cmd/atelet/internal/snapshottracker/tracker_test.go b/cmd/atelet/internal/snapshottracker/tracker_test.go
index 9443dcbaa0..aa8ac17286 100644
--- a/cmd/atelet/internal/snapshottracker/tracker_test.go
+++ b/cmd/atelet/internal/snapshottracker/tracker_test.go
@@ -657,7 +657,7 @@ func TestTrackerSkipsUnreadableSnapshotState(t *testing.T) {
// BeginCheckpoint and BeginPromote accept any generation when atelet holds no
// snapshot for the actor, and otherwise reject a generation below the actor's
// newest snapshot or equal to it under another snapshot_uuid with
-// FailedPrecondition.
+// Aborted.
func TestTrackerCheckGeneration(t *testing.T) {
useTempNodeDirs(t)
if err := os.MkdirAll(ateletpath.ActorPath(trackedActorUID), 0o700); err != nil {
@@ -700,9 +700,9 @@ func TestTrackerCheckGeneration(t *testing.T) {
}{
{name: "same snapshot at newest generation", ref: ref("s1", 2), wantCode: codes.OK},
{name: "higher generation", ref: ref("s2", 3), wantCode: codes.OK},
- {name: "generation below newest", ref: ref("s0", 1), wantCode: codes.FailedPrecondition},
- {name: "same uuid at lower generation", ref: ref("s1", 1), wantCode: codes.FailedPrecondition},
- {name: "different uuid at newest generation", ref: ref("other", 2), wantCode: codes.FailedPrecondition},
+ {name: "generation below newest", ref: ref("s0", 1), wantCode: codes.Aborted},
+ {name: "same uuid at lower generation", ref: ref("s1", 1), wantCode: codes.Aborted},
+ {name: "different uuid at newest generation", ref: ref("other", 2), wantCode: codes.Aborted},
} {
t.Run("localNewest/"+tc.name, func(t *testing.T) {
checkBegin(t, tr, tc.ref, tc.wantCode)
@@ -720,8 +720,8 @@ func TestTrackerCheckGeneration(t *testing.T) {
}{
{name: "same snapshot at newest generation", ref: ref("s2", 4), wantCode: codes.OK},
{name: "higher generation", ref: ref("s3", 5), wantCode: codes.OK},
- {name: "generation below newest", ref: ref("s1", 3), wantCode: codes.FailedPrecondition},
- {name: "different uuid at newest generation", ref: ref("other", 4), wantCode: codes.FailedPrecondition},
+ {name: "generation below newest", ref: ref("s1", 3), wantCode: codes.Aborted},
+ {name: "different uuid at newest generation", ref: ref("other", 4), wantCode: codes.Aborted},
} {
t.Run("durableNewest/"+tc.name, func(t *testing.T) {
checkBegin(t, tr, tc.ref, tc.wantCode)
@@ -730,35 +730,8 @@ func TestTrackerCheckGeneration(t *testing.T) {
// Both local (generation 6) and durable (generation 4) are tracked: local is newest.
tr.CommitLocal(trackedActorUID, ref("s3", 6))
- checkBegin(t, tr, ref("s2", 4), codes.FailedPrecondition)
+ checkBegin(t, tr, ref("s2", 4), codes.Aborted)
checkBegin(t, tr, ref("s3", 6), codes.OK)
-
- // While a Checkpoint or PromoteSnapshot holds an uncommitted snapshot at
- // generation 8, it is newest until end() runs.
- endCp, err := tr.BeginCheckpoint(t.Context(), trackedActorUID, snap("s4", 8))
- if err != nil {
- t.Fatalf("BeginCheckpoint(s4@8): %v", err)
- }
- checkBegin(t, tr, ref("s3", 6), codes.FailedPrecondition)
- checkBegin(t, tr, ref("between", 7), codes.FailedPrecondition)
- checkBegin(t, tr, ref("other", 8), codes.FailedPrecondition)
- checkBegin(t, tr, ref("s4", 8), codes.OK)
- checkBegin(t, tr, ref("s5", 9), codes.OK)
- endCp()
- // Ending without committing drops the in-flight generation, so s3@6 is newest again.
- checkBegin(t, tr, ref("s3", 6), codes.OK)
- checkBegin(t, tr, ref("between", 7), codes.OK)
-
- _, endPromote, err := tr.BeginPromote(t.Context(), trackedActorUID, ref("s4", 8))
- if err != nil {
- t.Fatalf("BeginPromote(s4@8): %v", err)
- }
- checkBegin(t, tr, ref("between", 7), codes.FailedPrecondition)
- checkBegin(t, tr, ref("other", 8), codes.FailedPrecondition)
- checkBegin(t, tr, ref("s4", 8), codes.OK)
- endPromote()
- checkBegin(t, tr, ref("between", 7), codes.OK)
-
// If durable has a higher generation than local, durable is newest.
a := &actorSnapshots{
actorUID: trackedActorUID,
@@ -767,11 +740,11 @@ func TestTrackerCheckGeneration(t *testing.T) {
LatestDurable: ref("durable", 5).clone(),
},
}
- if got := apierror.Code(a.checkGeneration(snap("local", 3))); got != codes.FailedPrecondition {
- t.Errorf("checkGeneration(older local@3) code = %v, want FailedPrecondition", got)
+ if got := apierror.Code(a.checkGeneration(snap("local", 3))); got != codes.Aborted {
+ t.Errorf("checkGeneration(older local@3) code = %v, want Aborted", got)
}
- if got := apierror.Code(a.checkGeneration(snap("other", 5))); got != codes.FailedPrecondition {
- t.Errorf("checkGeneration(other@5) code = %v, want FailedPrecondition", got)
+ if got := apierror.Code(a.checkGeneration(snap("other", 5))); got != codes.Aborted {
+ t.Errorf("checkGeneration(other@5) code = %v, want Aborted", got)
}
if got := apierror.Code(a.checkGeneration(snap("durable", 5))); got != codes.OK {
t.Errorf("checkGeneration(durable@5) code = %v, want OK", got)
@@ -779,11 +752,11 @@ func TestTrackerCheckGeneration(t *testing.T) {
// If Uploading holds a newer snapshot, it is newest even while no op runs.
a.Uploading = ref("saved-uploading", 6).clone()
- if got := apierror.Code(a.checkGeneration(snap("durable", 5))); got != codes.FailedPrecondition {
- t.Errorf("checkGeneration(older durable@5) code = %v, want FailedPrecondition", got)
+ if got := apierror.Code(a.checkGeneration(snap("durable", 5))); got != codes.Aborted {
+ t.Errorf("checkGeneration(older durable@5) code = %v, want Aborted", got)
}
- if got := apierror.Code(a.checkGeneration(snap("other", 6))); got != codes.FailedPrecondition {
- t.Errorf("checkGeneration(other@6) code = %v, want FailedPrecondition", got)
+ if got := apierror.Code(a.checkGeneration(snap("other", 6))); got != codes.Aborted {
+ t.Errorf("checkGeneration(other@6) code = %v, want Aborted", got)
}
if got := apierror.Code(a.checkGeneration(snap("saved-uploading", 6))); got != codes.OK {
t.Errorf("checkGeneration(saved-uploading@6) code = %v, want OK", got)
@@ -791,11 +764,11 @@ func TestTrackerCheckGeneration(t *testing.T) {
// If an operation in progress acts on a newer snapshot, its target is newest.
a.op = &actorOp{target: ref("uploading", 7)}
- if got := apierror.Code(a.checkGeneration(snap("saved-uploading", 6))); got != codes.FailedPrecondition {
- t.Errorf("checkGeneration(older saved-uploading@6) code = %v, want FailedPrecondition", got)
+ if got := apierror.Code(a.checkGeneration(snap("saved-uploading", 6))); got != codes.Aborted {
+ t.Errorf("checkGeneration(older saved-uploading@6) code = %v, want Aborted", got)
}
- if got := apierror.Code(a.checkGeneration(snap("other", 7))); got != codes.FailedPrecondition {
- t.Errorf("checkGeneration(other@7) code = %v, want FailedPrecondition", got)
+ if got := apierror.Code(a.checkGeneration(snap("other", 7))); got != codes.Aborted {
+ t.Errorf("checkGeneration(other@7) code = %v, want Aborted", got)
}
if got := apierror.Code(a.checkGeneration(snap("uploading", 7))); got != codes.OK {
t.Errorf("checkGeneration(uploading@7) code = %v, want OK", got)
diff --git a/cmd/atelet/internal/snapshottracker/upload_test.go b/cmd/atelet/internal/snapshottracker/upload_test.go
index 995a8431dc..5f1049a1e3 100644
--- a/cmd/atelet/internal/snapshottracker/upload_test.go
+++ b/cmd/atelet/internal/snapshottracker/upload_test.go
@@ -676,8 +676,8 @@ func TestTrackerBeginPromote(t *testing.T) {
tr.CommitLocal(trackedActorUID, testRef("s2", 2, tr.clock.Now()))
store.release <- nil
waitClosed(t, done, "the upload")
- if err := <-errCh; apierror.Code(err) != codes.FailedPrecondition {
- t.Errorf("BeginPromote = %v, want FailedPrecondition", err)
+ if err := <-errCh; apierror.Code(err) != codes.Aborted {
+ t.Errorf("BeginPromote = %v, want Aborted", err)
}
})
diff --git a/cmd/atelet/lifecycle_test.go b/cmd/atelet/lifecycle_test.go
index 4df2da5f16..e813929a54 100644
--- a/cmd/atelet/lifecycle_test.go
+++ b/cmd/atelet/lifecycle_test.go
@@ -34,6 +34,7 @@ import (
"go.ate.dev/substrate/cmd/atelet/internal/trustbundle"
"go.ate.dev/substrate/internal/apierror"
"go.ate.dev/substrate/internal/ateattr"
+ "go.ate.dev/substrate/internal/ateinterceptors"
"go.ate.dev/substrate/internal/hardware"
"go.ate.dev/substrate/internal/nodepath"
"go.ate.dev/substrate/internal/proto/ateletpb"
@@ -42,6 +43,7 @@ import (
"go.ate.dev/substrate/pkg/api/v1alpha1"
"google.golang.org/grpc"
"google.golang.org/grpc/codes"
+ "google.golang.org/grpc/status"
"google.golang.org/protobuf/proto"
"k8s.io/utils/clock"
clocktesting "k8s.io/utils/clock/testing"
@@ -229,20 +231,12 @@ func runLocalSnapshotGC(t *testing.T, fidelity ateletpb.SnapshotFidelity) {
systemInfoVolumes: newSystemInfoVolumeRefresher(nil, nil),
snapshots: snapshottracker.New(t.Context(), nil, clock.RealClock{}),
}
- sandboxAssets := &ateletpb.SandboxAssets{
- SandboxClass: "gvisor",
- PauseImage: pinnedImage,
- Assets: map[string]*ateletpb.ArchAssets{
- runtime.GOARCH: {Files: map[string]*ateletpb.AssetFile{
- gvisorAssetName: release.proto(),
- }},
- },
- }
+ sandboxRec := testSandboxRecord(release, pinnedImage)
spec := &ateletpb.WorkloadSpec{
Containers: []*ateletpb.Container{{Name: "app", Image: pinnedImage, Command: []string{"/bin/app"}}},
}
- advertiseRuntime(t, s, sandboxAssets)
+ advertiseRuntime(t, s, sandboxRec)
if _, err := s.Run(ctx, &ateletpb.RunRequest{
Atespace: atespace,
ActorName: actorName,
@@ -250,8 +244,8 @@ func runLocalSnapshotGC(t *testing.T, fidelity ateletpb.SnapshotFidelity) {
ActorTemplateAtespace: "default",
ActorTemplateName: "counter",
WorkerPodUid: workerPodUID,
- SandboxAssets: sandboxAssets, SandboxRuntime: runtimeFor(t, sandboxAssets),
- Spec: spec,
+ SandboxRuntime: runtimeFor(t, sandboxRec),
+ Spec: spec,
}); err != nil {
t.Fatalf("Run: %v", err)
}
@@ -288,7 +282,8 @@ func runLocalSnapshotGC(t *testing.T, fidelity ateletpb.SnapshotFidelity) {
t.Errorf("state = %v, want nothing durable", st)
}
- // Resume: no Snapshot.Object; atelet must find that local snapshot.
+ // Resume: atelet must restore from that local snapshot, not from
+ // Snapshot.Object.
if _, err := s.Restore(ctx, &ateletpb.RestoreRequest{
Atespace: atespace,
ActorName: actorName,
@@ -296,13 +291,9 @@ func runLocalSnapshotGC(t *testing.T, fidelity ateletpb.SnapshotFidelity) {
ActorTemplateAtespace: "default",
ActorTemplateName: "counter",
WorkerPodUid: workerPodUID,
- SandboxAssets: sandboxAssets, SandboxRuntime: runtimeFor(t, sandboxAssets),
- Spec: spec,
- Snapshot: &ateletpb.Snapshot{
- SnapshotUuid: snapshotName,
- Generation: 1,
- Fidelity: fidelity,
- },
+ SandboxRuntime: runtimeFor(t, sandboxRec),
+ Spec: spec,
+ Snapshot: lifecycleSnapshot(actorUID, snapshotName, 1, fidelity),
}); err != nil {
t.Fatalf("Restore: %v", err)
}
@@ -395,10 +386,10 @@ func runLocalSnapshotGC(t *testing.T, fidelity ateletpb.SnapshotFidelity) {
}
}
-// TestRestoreUsesRequestSandboxAssets checks that Restore runs the actor with
-// the sandbox assets on the request, not the ones recorded in the snapshot
+// TestRestoreUsesRequestSandboxRuntime checks that Restore runs the actor with
+// the sandbox runtime on the request, not the one recorded in the snapshot
// manifest.
-func TestRestoreUsesRequestSandboxAssets(t *testing.T) {
+func TestRestoreUsesRequestSandboxRuntime(t *testing.T) {
useTempNodeDirs(t)
ctx := t.Context()
@@ -428,22 +419,14 @@ func TestRestoreUsesRequestSandboxAssets(t *testing.T) {
systemInfoVolumes: newSystemInfoVolumeRefresher(nil, nil),
snapshots: snapshottracker.New(t.Context(), nil, clock.RealClock{}),
}
- assetsWithPause := func(pause string) *ateletpb.SandboxAssets {
- return &ateletpb.SandboxAssets{
- SandboxClass: "gvisor",
- PauseImage: pause,
- Assets: map[string]*ateletpb.ArchAssets{
- runtime.GOARCH: {Files: map[string]*ateletpb.AssetFile{
- gvisorAssetName: release.proto(),
- }},
- },
- }
+ recWithPause := func(pause string) *sandboxAssetsRecord {
+ return testSandboxRecord(release, pause)
}
spec := &ateletpb.WorkloadSpec{
Containers: []*ateletpb.Container{{Name: "app", Image: pinnedImage, Command: []string{"/bin/app"}}},
}
- advertiseRuntime(t, s, assetsWithPause(checkpointPause))
+ advertiseRuntime(t, s, recWithPause(checkpointPause))
if _, err := s.Run(ctx, &ateletpb.RunRequest{
Atespace: atespace,
ActorName: actorName,
@@ -451,8 +434,8 @@ func TestRestoreUsesRequestSandboxAssets(t *testing.T) {
ActorTemplateAtespace: "default",
ActorTemplateName: "counter",
WorkerPodUid: workerPodUID,
- SandboxAssets: assetsWithPause(checkpointPause), SandboxRuntime: runtimeFor(t, assetsWithPause(checkpointPause)),
- Spec: spec,
+ SandboxRuntime: runtimeFor(t, recWithPause(checkpointPause)),
+ Spec: spec,
}); err != nil {
t.Fatalf("Run: %v", err)
}
@@ -483,7 +466,7 @@ func TestRestoreUsesRequestSandboxAssets(t *testing.T) {
t.Fatalf("manifest pause image = %q, want %q", manifestRec.PauseImage, checkpointPause)
}
- advertiseRuntime(t, s, assetsWithPause(restorePause))
+ advertiseRuntime(t, s, recWithPause(restorePause))
if _, err := s.Restore(ctx, &ateletpb.RestoreRequest{
Atespace: atespace,
ActorName: actorName,
@@ -491,13 +474,9 @@ func TestRestoreUsesRequestSandboxAssets(t *testing.T) {
ActorTemplateAtespace: "default",
ActorTemplateName: "counter",
WorkerPodUid: workerPodUID,
- SandboxAssets: assetsWithPause(restorePause), SandboxRuntime: runtimeFor(t, assetsWithPause(restorePause)),
- Spec: spec,
- Snapshot: &ateletpb.Snapshot{
- SnapshotUuid: snapshotName,
- Generation: 1,
- Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY,
- },
+ SandboxRuntime: runtimeFor(t, recWithPause(restorePause)),
+ Spec: spec,
+ Snapshot: lifecycleSnapshot(actorUID, snapshotName, 1, ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY),
}); err != nil {
t.Fatalf("Restore: %v", err)
}
@@ -526,7 +505,7 @@ func TestActivationFailureBeforeRegistration(t *testing.T) {
{name: "Run partial reset removes registration", failure: "reset", wantErr: "while removing volume dir"},
{name: "Run record failure removes registration", failure: "record", wantErr: "while recording sandbox assets"},
{name: "Restore partial reset removes registration", restore: true, failure: "reset", wantErr: "while removing volume dir"},
- {name: "Restore manifest failure removes registration", restore: true, failure: "manifest", wantErr: "no snapshot.object to fall back to"},
+ {name: "Restore manifest failure removes registration", restore: true, failure: "manifest", wantErr: "no uploaded copy"},
{name: "Restore asset failure removes registration", restore: true, failure: "asset", wantErr: "asset unavailable"},
} {
t.Run(tc.name, func(t *testing.T) {
@@ -587,28 +566,23 @@ func TestActivationFailureBeforeRegistration(t *testing.T) {
if tc.failure == "asset" {
storage.err = fmt.Errorf("asset unavailable")
}
- s := &Atelet{anonGCSClient: storage, systemInfoVolumes: refresher, snapshots: snapshottracker.New(t.Context(), nil, clock.RealClock{}), sandboxAssetCache: newTestAssetCache(t)}
- assets := &ateletpb.SandboxAssets{
- SandboxClass: "gvisor", PauseImage: testPauseImage,
- Assets: map[string]*ateletpb.ArchAssets{runtime.GOARCH: {Files: map[string]*ateletpb.AssetFile{
- gvisorAssetName: release.proto(),
- }}},
- }
- advertiseRuntime(t, s, assets)
+ // The object store is empty: a restore with no local copy finds
+ // no uploaded copy either.
+ s := newPluginHerder(t, &recordingObjectStorage{})
+ s.anonGCSClient = storage
+ s.systemInfoVolumes = refresher
+ rec := testSandboxRecord(release, testPauseImage)
+ advertiseRuntime(t, s, rec)
if tc.restore {
_, err = s.Restore(ctx, &ateletpb.RestoreRequest{
Atespace: "team-a", ActorName: "actor-1", ActorUid: actorUID, WorkerPodUid: "5f1c0d2e-3a4b-4c5d-8e9f-0a1b2c3d4e5f",
- SandboxAssets: assets, SandboxRuntime: runtimeFor(t, assets), Spec: spec,
- Snapshot: &ateletpb.Snapshot{
- SnapshotUuid: snapshotName,
- Generation: 1,
- Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY,
- },
+ SandboxRuntime: runtimeFor(t, rec), Spec: spec,
+ Snapshot: lifecycleSnapshot(actorUID, snapshotName, 1, ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY),
})
} else {
_, err = s.Run(ctx, &ateletpb.RunRequest{
Atespace: "team-a", ActorName: "actor-1", ActorUid: actorUID, WorkerPodUid: "5f1c0d2e-3a4b-4c5d-8e9f-0a1b2c3d4e5f",
- SandboxAssets: assets, SandboxRuntime: runtimeFor(t, assets), Spec: spec,
+ SandboxRuntime: runtimeFor(t, rec), Spec: spec,
})
}
if err == nil || !strings.Contains(err.Error(), tc.wantErr) {
@@ -690,21 +664,15 @@ func TestRunFailureAfterRegistrationRemovesOwnRegistration(t *testing.T) {
SystemInfo: trustVolumeSpec("ca.pem"),
}},
}
- assets := &ateletpb.SandboxAssets{
- SandboxClass: "gvisor",
- PauseImage: missingPauseImage(t),
- Assets: map[string]*ateletpb.ArchAssets{runtime.GOARCH: {Files: map[string]*ateletpb.AssetFile{
- gvisorAssetName: release.proto(),
- }}},
- }
- advertiseRuntime(t, s, assets)
+ rec := testSandboxRecord(release, missingPauseImage(t))
+ advertiseRuntime(t, s, rec)
_, err := s.Run(t.Context(), &ateletpb.RunRequest{
- Atespace: "team-a",
- ActorName: "actor-run",
- ActorUid: actorUID,
- WorkerPodUid: "5f1c0d2e-3a4b-4c5d-8e9f-0a1b2c3d4e5f",
- SandboxAssets: assets, SandboxRuntime: runtimeFor(t, assets),
- Spec: spec,
+ Atespace: "team-a",
+ ActorName: "actor-run",
+ ActorUid: actorUID,
+ WorkerPodUid: "5f1c0d2e-3a4b-4c5d-8e9f-0a1b2c3d4e5f",
+ SandboxRuntime: runtimeFor(t, rec),
+ Spec: spec,
})
if err == nil || !strings.Contains(err.Error(), "while creating pause OCI bundle") {
t.Fatalf("Run error = %v, want the post-registration OCI preparation failure", err)
@@ -754,26 +722,16 @@ func TestRestoreFailureAfterRegistrationRemovesOwnRegistration(t *testing.T) {
systemInfoVolumes: refresher,
snapshots: snapshottracker.New(t.Context(), nil, clock.RealClock{}),
}
- assets := &ateletpb.SandboxAssets{
- SandboxClass: "gvisor",
- PauseImage: pauseImage,
- Assets: map[string]*ateletpb.ArchAssets{runtime.GOARCH: {Files: map[string]*ateletpb.AssetFile{
- gvisorAssetName: release.proto(),
- }}},
- }
- advertiseRuntime(t, s, assets)
+ rec := testSandboxRecord(release, pauseImage)
+ advertiseRuntime(t, s, rec)
_, err := s.Restore(t.Context(), &ateletpb.RestoreRequest{
- Atespace: "team-a",
- ActorName: "actor-restore-after",
- ActorUid: actorUID,
- WorkerPodUid: "5f1c0d2e-3a4b-4c5d-8e9f-0a1b2c3d4e5f",
- SandboxAssets: assets, SandboxRuntime: runtimeFor(t, assets),
- Spec: spec,
- Snapshot: &ateletpb.Snapshot{
- SnapshotUuid: snapshotName,
- Generation: 1,
- Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY,
- },
+ Atespace: "team-a",
+ ActorName: "actor-restore-after",
+ ActorUid: actorUID,
+ WorkerPodUid: "5f1c0d2e-3a4b-4c5d-8e9f-0a1b2c3d4e5f",
+ SandboxRuntime: runtimeFor(t, rec),
+ Spec: spec,
+ Snapshot: lifecycleSnapshot(actorUID, snapshotName, 1, ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY),
})
if err == nil || !strings.Contains(err.Error(), "while creating pause OCI bundle") {
t.Fatalf("Restore error = %v, want the post-registration OCI preparation failure", err)
@@ -807,15 +765,7 @@ func TestRestorePrefersLocalAndFallsBackToObjectStorage(t *testing.T) {
pinnedImage := pushTestImage(t, host+"/actor:v1", singleFileLayer(t, "bin/app", "app"))
release := newGVisorTestRelease(t)
- sandboxAssets := &ateletpb.SandboxAssets{
- SandboxClass: "gvisor",
- PauseImage: pinnedImage,
- Assets: map[string]*ateletpb.ArchAssets{
- runtime.GOARCH: {Files: map[string]*ateletpb.AssetFile{
- gvisorAssetName: release.proto(),
- }},
- },
- }
+ sandboxRec := testSandboxRecord(release, pinnedImage)
spec := &ateletpb.WorkloadSpec{
Containers: []*ateletpb.Container{{Name: "app", Image: pinnedImage, Command: []string{"/bin/app"}}},
}
@@ -894,7 +844,7 @@ func TestRestorePrefersLocalAndFallsBackToObjectStorage(t *testing.T) {
useTempNodeDirs(t)
localDir := ateletpath.LocalSnapshotDir(actorUID, snapshotName)
tc.prepLocal(t, localDir)
- advertiseRuntime(t, s, sandboxAssets)
+ advertiseRuntime(t, s, sandboxRec)
if _, err := s.Restore(t.Context(), &ateletpb.RestoreRequest{
Atespace: atespace,
@@ -903,8 +853,8 @@ func TestRestorePrefersLocalAndFallsBackToObjectStorage(t *testing.T) {
ActorTemplateAtespace: "default",
ActorTemplateName: "counter",
WorkerPodUid: workerPodUID,
- SandboxAssets: sandboxAssets, SandboxRuntime: runtimeFor(t, sandboxAssets),
- Spec: spec,
+ SandboxRuntime: runtimeFor(t, sandboxRec),
+ Spec: spec,
Snapshot: &ateletpb.Snapshot{
SnapshotUuid: snapshotName,
Generation: 1,
@@ -940,42 +890,26 @@ func TestRestorePrefersLocalAndFallsBackToObjectStorage(t *testing.T) {
}
// TestRestoreWithoutUsableLocalCopy checks that a Restore with no usable local
-// copy and nothing to fall back to fails as NotFound and wraps the local
-// copy's problem, whether the request has no snapshot.object or names one that
-// was never uploaded.
+// copy, naming a snapshot.object that was never uploaded, fails as NotFound
+// and wraps the local copy's problem.
func TestRestoreWithoutUsableLocalCopy(t *testing.T) {
const (
actorUID = "0f6c2b0e-1b7e-4b3f-9a6a-1d2c3e4f5a65"
snapshotName = "pause-snap-1"
)
release := newGVisorTestRelease(t)
- assets := &ateletpb.SandboxAssets{
- SandboxClass: "gvisor", PauseImage: testPauseImage,
- Assets: map[string]*ateletpb.ArchAssets{runtime.GOARCH: {Files: map[string]*ateletpb.AssetFile{
- gvisorAssetName: release.proto(),
- }}},
- }
+ rec := testSandboxRecord(release, testPauseImage)
for _, tc := range []struct {
name string
// unusableLocal writes a manifest that lists a file that is not there.
unusableLocal bool
- object *ateletpb.ObjectStorage
- wantErr string
wantCause error
}{{
- name: "no local copy and no object",
- wantErr: "no snapshot.object to fall back to",
+ name: "no local copy",
wantCause: os.ErrNotExist,
}, {
- name: "unusable local copy and no object",
+ name: "unusable local copy",
unusableLocal: true,
- wantErr: "no snapshot.object to fall back to",
- wantCause: errUnusableLocalSnapshot,
- }, {
- name: "unusable local copy and an object never uploaded",
- unusableLocal: true,
- object: &ateletpb.ObjectStorage{SnapshotUri: pausedSnapshotURI},
- wantErr: "no uploaded copy",
wantCause: errUnusableLocalSnapshot,
}} {
t.Run(tc.name, func(t *testing.T) {
@@ -988,24 +922,19 @@ func TestRestoreWithoutUsableLocalCopy(t *testing.T) {
// The object store is empty: nothing was ever uploaded.
s := newPluginHerder(t, &recordingObjectStorage{})
s.systemInfoVolumes = newSystemInfoVolumeRefresher(nil, nil)
- advertiseRuntime(t, s, assets)
+ advertiseRuntime(t, s, rec)
_, err := s.Restore(t.Context(), &ateletpb.RestoreRequest{
Atespace: "team-a", ActorName: "actor-1", ActorUid: actorUID,
- WorkerPodUid: "5f1c0d2e-3a4b-4c5d-8e9f-0a1b2c3d4e5f",
- SandboxAssets: assets, SandboxRuntime: runtimeFor(t, assets),
- Spec: &ateletpb.WorkloadSpec{},
- Snapshot: &ateletpb.Snapshot{
- SnapshotUuid: snapshotName,
- Generation: 1,
- Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY,
- Object: tc.object,
- },
+ WorkerPodUid: "5f1c0d2e-3a4b-4c5d-8e9f-0a1b2c3d4e5f",
+ SandboxRuntime: runtimeFor(t, rec),
+ Spec: &ateletpb.WorkloadSpec{},
+ Snapshot: lifecycleSnapshot(actorUID, snapshotName, 1, ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY),
})
if code := apierror.Code(err); code != codes.NotFound {
t.Fatalf("Restore error = %v (code %v), want NotFound", err, code)
}
- if !strings.Contains(err.Error(), tc.wantErr) {
- t.Errorf("Restore error = %v, want it to say %q", err, tc.wantErr)
+ if !strings.Contains(err.Error(), "no uploaded copy") {
+ t.Errorf("Restore error = %v, want it to say %q", err, "no uploaded copy")
}
if !errors.Is(err, tc.wantCause) {
t.Errorf("Restore error = %v, want it to wrap %v", err, tc.wantCause)
@@ -1089,7 +1018,7 @@ type pluginActor struct {
s *Atelet
worker *fakeWorker
store *recordingObjectStorage
- assets *ateletpb.SandboxAssets
+ rec *sandboxAssetsRecord
spec *ateletpb.WorkloadSpec
}
@@ -1114,20 +1043,12 @@ func runPluginActor(t *testing.T) *pluginActor {
a.s.sandboxAssetCache = newTestAssetCache(t)
a.s.anonGCSClient = fakeObjectStorage{data: release.tarball}
a.s.systemInfoVolumes = newSystemInfoVolumeRefresher(nil, nil)
- a.assets = &ateletpb.SandboxAssets{
- SandboxClass: "gvisor",
- PauseImage: pinnedImage,
- Assets: map[string]*ateletpb.ArchAssets{
- runtime.GOARCH: {Files: map[string]*ateletpb.AssetFile{
- gvisorAssetName: release.proto(),
- }},
- },
- }
+ a.rec = testSandboxRecord(release, pinnedImage)
a.spec = &ateletpb.WorkloadSpec{
Containers: []*ateletpb.Container{{Name: "app", Image: pinnedImage, Command: []string{"/bin/app"}}},
}
- advertiseRuntime(t, a.s, a.assets)
+ advertiseRuntime(t, a.s, a.rec)
if _, err := a.s.Run(t.Context(), &ateletpb.RunRequest{
Atespace: "ate-demo",
ActorName: "counter",
@@ -1135,8 +1056,8 @@ func runPluginActor(t *testing.T) *pluginActor {
ActorTemplateAtespace: "default",
ActorTemplateName: "counter",
WorkerPodUid: pluginWorkerPodUID,
- SandboxAssets: a.assets, SandboxRuntime: runtimeFor(t, a.assets),
- Spec: a.spec,
+ SandboxRuntime: runtimeFor(t, a.rec),
+ Spec: a.spec,
}); err != nil {
t.Fatalf("Run: %v", err)
}
@@ -1165,7 +1086,8 @@ func (a *pluginActor) checkpoint(t *testing.T, snap *ateletpb.Snapshot, mode ate
}
// restoreLocal resumes the actor from its local snapshot named name, taken at
-// generation.
+// generation. The request names the snapshot's upload location too, as the
+// control plane's does.
func (a *pluginActor) restoreLocal(t *testing.T, name string, generation int32) {
t.Helper()
if _, err := a.s.Restore(t.Context(), &ateletpb.RestoreRequest{
@@ -1175,12 +1097,15 @@ func (a *pluginActor) restoreLocal(t *testing.T, name string, generation int32)
ActorTemplateAtespace: "default",
ActorTemplateName: "counter",
WorkerPodUid: pluginWorkerPodUID,
- SandboxAssets: a.assets, SandboxRuntime: runtimeFor(t, a.assets),
- Spec: a.spec,
- Snapshot: &ateletpb.Snapshot{SnapshotUuid: name, Generation: generation, Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY},
+ SandboxRuntime: runtimeFor(t, a.rec),
+ Spec: a.spec,
+ Snapshot: a.snapshot(name, generation),
}); err != nil {
t.Fatalf("Restore: %v", err)
}
+ if !a.worker.preserveRestoreDir {
+ t.Fatalf("Restore of %s read object storage, want its local copy restored in place", name)
+ }
}
// promote promotes snap and returns the actor's state the response reports.
@@ -1262,6 +1187,114 @@ func TestImmediateCheckpointWithoutLocalCopy(t *testing.T) {
}
}
+// TestTerminateWaitsForInFlightRPC checks that Terminate waits on the
+// per-actor lock while Checkpoint or PromoteSnapshot is in flight, then cleans
+// up the actor's tracked state and local files once that RPC returns.
+func TestTerminateWaitsForInFlightRPC(t *testing.T) {
+ waitTerminateBehind := func(t *testing.T, a *pluginActor) (block func(), finish func()) {
+ t.Helper()
+ var once sync.Once
+ started := make(chan struct{})
+ release := make(chan struct{})
+ block = func() {
+ once.Do(func() {
+ close(started)
+ <-release
+ })
+ }
+ finish = func() {
+ t.Helper()
+ <-started
+ termDone := make(chan error, 1)
+ go func() {
+ _, err := a.s.Terminate(t.Context(), &ateletpb.TerminateRequest{
+ Atespace: "ate-demo",
+ ActorName: "counter",
+ ActorUid: pluginActorUID,
+ WorkerPodUid: pluginWorkerPodUID,
+ Spec: a.spec,
+ })
+ termDone <- err
+ }()
+ select {
+ case err := <-termDone:
+ t.Fatalf("Terminate returned %v while another RPC held the actor lock, want it to wait", err)
+ case <-time.After(50 * time.Millisecond):
+ }
+ close(release)
+ if err := <-termDone; err != nil {
+ t.Fatalf("Terminate: %v", err)
+ }
+ }
+ return block, finish
+ }
+
+ assertCleanedUp := func(t *testing.T, a *pluginActor) {
+ t.Helper()
+ if st := a.s.snapshots.State(pluginActorUID); !proto.Equal(st, &ateletpb.ActorSnapshotState{}) {
+ t.Errorf("tracked state = %v, want none", st)
+ }
+ if _, err := os.Stat(ateletpath.ActorSnapshotStateFile(pluginActorUID)); !os.IsNotExist(err) {
+ t.Errorf("snapshot state file: stat = %v, want none", err)
+ }
+ if _, err := os.Stat(ateletpath.LocalSnapshotDir(pluginActorUID, "snap-1")); !os.IsNotExist(err) {
+ t.Errorf("local snapshot: stat = %v, want none", err)
+ }
+ if _, err := os.Stat(ateletpath.ActorPath(pluginActorUID)); !os.IsNotExist(err) {
+ t.Errorf("actor dir after Terminate: stat = %v, want none", err)
+ }
+ }
+
+ for _, mode := range []ateletpb.UploadMode{
+ ateletpb.UploadMode_UPLOAD_MODE_DEFERRED,
+ ateletpb.UploadMode_UPLOAD_MODE_IMMEDIATE,
+ } {
+ t.Run(mode.String(), func(t *testing.T) {
+ a := runPluginActor(t)
+ block, finish := waitTerminateBehind(t, a)
+ a.worker.onCheckpoint = block
+ cpDone := make(chan error, 1)
+ go func() {
+ _, err := a.checkpoint(t, a.snapshot("snap-1", 1), mode)
+ cpDone <- err
+ }()
+ finish()
+ if err := <-cpDone; err != nil {
+ t.Fatalf("Checkpoint: %v", err)
+ }
+ assertCleanedUp(t, a)
+ })
+ }
+
+ t.Run("PromoteSnapshot", func(t *testing.T) {
+ a := runPluginActor(t)
+ snap := a.snapshot("snap-1", 1)
+ if _, err := a.checkpoint(t, snap, ateletpb.UploadMode_UPLOAD_MODE_DEFERRED); err != nil {
+ t.Fatalf("Checkpoint: %v", err)
+ }
+
+ block, finish := waitTerminateBehind(t, a)
+ a.store.onPut = block
+ promoteDone := make(chan error, 1)
+ go func() {
+ _, err := a.s.PromoteSnapshot(t.Context(), &ateletpb.PromoteSnapshotRequest{
+ Atespace: "ate-demo",
+ ActorName: "counter",
+ ActorUid: pluginActorUID,
+ ActorTemplateAtespace: "default",
+ ActorTemplateName: "counter",
+ Snapshot: snap,
+ })
+ promoteDone <- err
+ }()
+ finish()
+ if err := <-promoteDone; err != nil {
+ t.Fatalf("PromoteSnapshot: %v", err)
+ }
+ assertCleanedUp(t, a)
+ })
+}
+
// TestFailedCheckpointKeepsLocalSnapshot checkpoints a resumed actor again, and
// fails before the new snapshot commits: the pause snapshot stays on disk and
// recorded as the actor's latest local snapshot, so the actor can still be
@@ -1384,20 +1417,12 @@ func TestConcurrentRunsShareOneAssetDownload(t *testing.T) {
systemInfoVolumes: newSystemInfoVolumeRefresher(nil, nil),
snapshots: snapshottracker.New(t.Context(), nil, clock.RealClock{}),
}
- sandboxAssets := &ateletpb.SandboxAssets{
- SandboxClass: "gvisor",
- PauseImage: pinnedImage,
- Assets: map[string]*ateletpb.ArchAssets{
- runtime.GOARCH: {Files: map[string]*ateletpb.AssetFile{
- gvisorAssetName: release.proto(),
- }},
- },
- }
+ sandboxRec := testSandboxRecord(release, pinnedImage)
spec := &ateletpb.WorkloadSpec{
Containers: []*ateletpb.Container{{Name: "app", Image: pinnedImage, Command: []string{"/bin/app"}}},
}
- advertiseRuntime(t, s, sandboxAssets)
+ advertiseRuntime(t, s, sandboxRec)
actorUIDs := []string{"01234567-89ab-cdef-0123-456789abcde1", "01234567-89ab-cdef-0123-456789abcde2"}
var wg sync.WaitGroup
errs := make([]error, len(actorUIDs))
@@ -1410,8 +1435,8 @@ func TestConcurrentRunsShareOneAssetDownload(t *testing.T) {
ActorTemplateAtespace: "default",
ActorTemplateName: "counter",
WorkerPodUid: "5f1c0d2e-3a4b-4c5d-8e9f-0a1b2c3d4e5f",
- SandboxAssets: sandboxAssets, SandboxRuntime: runtimeFor(t, sandboxAssets),
- Spec: spec,
+ SandboxRuntime: runtimeFor(t, sandboxRec),
+ Spec: spec,
})
})
}
@@ -1553,8 +1578,8 @@ func TestFailedRestoreUnschedulesDetach(t *testing.T) {
if _, err := a.checkpoint(t, a.snapshot("pause-snap-1", 1), ateletpb.UploadMode_UPLOAD_MODE_DEFERRED); err != nil {
t.Fatalf("Checkpoint (pause): %v", err)
}
- // A snapshot with neither a local copy nor an object storage location
- // fails the restore once it has begun.
+ // A snapshot with neither a local copy nor an uploaded copy fails the
+ // restore once it has begun.
_, err := a.s.Restore(t.Context(), &ateletpb.RestoreRequest{
Atespace: "ate-demo",
ActorName: "counter",
@@ -1562,9 +1587,9 @@ func TestFailedRestoreUnschedulesDetach(t *testing.T) {
ActorTemplateAtespace: "default",
ActorTemplateName: "counter",
WorkerPodUid: pluginWorkerPodUID,
- SandboxAssets: a.assets, SandboxRuntime: runtimeFor(t, a.assets),
- Spec: a.spec,
- Snapshot: &ateletpb.Snapshot{SnapshotUuid: "missing-snap", Generation: 2, Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY},
+ SandboxRuntime: runtimeFor(t, a.rec),
+ Spec: a.spec,
+ Snapshot: a.snapshot("missing-snap", 2),
})
if apierror.Code(err) != codes.NotFound {
t.Fatalf("Restore of a missing snapshot = %v, want NotFound", err)
@@ -1577,6 +1602,170 @@ func TestFailedRestoreUnschedulesDetach(t *testing.T) {
}
}
+// TestRestoreAfterDetachReadsUpload resumes a paused actor that atelet
+// detached before the control plane recorded the upload, so the control plane
+// still sends the Restore here: atelet restores the actor from the upload the
+// request names.
+func TestRestoreAfterDetachReadsUpload(t *testing.T) {
+ a := runPluginActor(t)
+ a.s.snapshots = snapshottracker.New(t.Context(), &trackerStore{s: a.s}, clock.RealClock{})
+ snap := a.snapshot("pause-snap-1", 1)
+ if _, err := a.checkpoint(t, snap, ateletpb.UploadMode_UPLOAD_MODE_DEFERRED); err != nil {
+ t.Fatalf("Checkpoint (pause): %v", err)
+ }
+ // A Detach call uploads and prunes as a background detach does.
+ st, err := a.s.snapshots.Detach(t.Context(), pluginActorUID)
+ if err != nil {
+ t.Fatalf("Detach: %v", err)
+ }
+ if st.GetLatestLocal() != nil || st.GetLatestDurable().GetSnapshot().GetSnapshotUuid() != "pause-snap-1" {
+ t.Fatalf("state after detach = %v, want pause-snap-1 durable and nothing local", st)
+ }
+
+ if _, err := a.s.Restore(t.Context(), &ateletpb.RestoreRequest{
+ Atespace: "ate-demo",
+ ActorName: "counter",
+ ActorUid: pluginActorUID,
+ ActorTemplateAtespace: "default",
+ ActorTemplateName: "counter",
+ WorkerPodUid: pluginWorkerPodUID,
+ SandboxRuntime: runtimeFor(t, a.rec),
+ Spec: a.spec,
+ Snapshot: snap,
+ }); err != nil {
+ t.Fatalf("Restore of the detached snapshot: %v", err)
+ }
+ if a.worker.preserveRestoreDir {
+ t.Error("Restore restored a local copy in place, want the upload staged")
+ }
+ if got := a.worker.restored["checkpoint.img"]; got != "guest-memory" {
+ t.Errorf("restored checkpoint.img = %q, want the paused snapshot's %q", got, "guest-memory")
+ }
+}
+
+func detachActorRequest() *ateletpb.DetachActorRequest {
+ return &ateletpb.DetachActorRequest{Atespace: "ate-demo", ActorName: "counter", ActorUid: pluginActorUID}
+}
+
+// TestDetachActorDetachesPausedActor detaches a paused actor through the RPC:
+// atelet uploads its local snapshot, deletes the local copy, and returns the
+// new state.
+func TestDetachActorDetachesPausedActor(t *testing.T) {
+ a := runPluginActor(t)
+ a.s.snapshots = snapshottracker.New(t.Context(), &trackerStore{s: a.s}, clock.RealClock{})
+ if _, err := a.checkpoint(t, a.snapshot("pause-snap-1", 1), ateletpb.UploadMode_UPLOAD_MODE_DEFERRED); err != nil {
+ t.Fatalf("Checkpoint (pause): %v", err)
+ }
+ localDir := ateletpath.LocalSnapshotDir(pluginActorUID, "pause-snap-1")
+ if _, err := os.Stat(localDir); err != nil {
+ t.Fatalf("local snapshot dir after the pause: %v", err)
+ }
+ resp, err := a.s.DetachActor(t.Context(), detachActorRequest())
+ if err != nil {
+ t.Fatalf("DetachActor: %v", err)
+ }
+ if st := resp.GetState(); st.GetLatestLocal() != nil || st.GetLatestDurable().GetSnapshot().GetSnapshotUuid() != "pause-snap-1" {
+ t.Errorf("state = %v, want pause-snap-1 durable and nothing local", st)
+ }
+ if _, err := os.Stat(localDir); !errors.Is(err, os.ErrNotExist) {
+ t.Errorf("local snapshot dir after DetachActor: stat err = %v, want not exist", err)
+ }
+}
+
+// TestDetachActorWithoutLocalSnapshot detaches an actor atelet holds no local
+// snapshot for: DetachActor does nothing and returns an empty state.
+func TestDetachActorWithoutLocalSnapshot(t *testing.T) {
+ a := runPluginActor(t)
+ a.s.snapshots = snapshottracker.New(t.Context(), &trackerStore{s: a.s}, clock.RealClock{})
+ resp, err := a.s.DetachActor(t.Context(), detachActorRequest())
+ if err != nil {
+ t.Fatalf("DetachActor: %v", err)
+ }
+ if !proto.Equal(resp.GetState(), &ateletpb.ActorSnapshotState{}) {
+ t.Errorf("state = %v, want empty", resp.GetState())
+ }
+}
+
+// TestDetachActorRejectsRestoredActor detaches an actor restored from its
+// local snapshot: DetachActor fails with Aborted.
+func TestDetachActorRejectsRestoredActor(t *testing.T) {
+ a := runPluginActor(t)
+ a.s.snapshots = snapshottracker.New(t.Context(), &trackerStore{s: a.s}, clock.RealClock{})
+ if _, err := a.checkpoint(t, a.snapshot("pause-snap-1", 1), ateletpb.UploadMode_UPLOAD_MODE_DEFERRED); err != nil {
+ t.Fatalf("Checkpoint (pause): %v", err)
+ }
+ a.restoreLocal(t, "pause-snap-1", 1)
+ if _, err := a.s.DetachActor(t.Context(), detachActorRequest()); apierror.Code(err) != codes.Aborted {
+ t.Errorf("DetachActor of the restored actor = %v, want Aborted", err)
+ }
+}
+
+// TestDetachActorValidatesRequest sends a DetachActor request without an
+// actor UID: atelet rejects it with InvalidArgument.
+func TestDetachActorValidatesRequest(t *testing.T) {
+ a := runPluginActor(t)
+ a.s.snapshots = snapshottracker.New(t.Context(), &trackerStore{s: a.s}, clock.RealClock{})
+ req := detachActorRequest()
+ req.ActorUid = ""
+ if _, err := a.s.DetachActor(t.Context(), req); apierror.Code(err) != codes.InvalidArgument {
+ t.Errorf("DetachActor without actor_uid = %v, want InvalidArgument", err)
+ }
+}
+
+// TestDetachActorWaitsForInFlightCheckpoint checks that DetachActor waits on
+// the per-actor lock while a Checkpoint is in flight, then detaches the
+// snapshot that Checkpoint committed.
+func TestDetachActorWaitsForInFlightCheckpoint(t *testing.T) {
+ a := runPluginActor(t)
+ a.s.snapshots = snapshottracker.New(t.Context(), &trackerStore{s: a.s}, clock.RealClock{})
+ started := make(chan struct{})
+ release := make(chan struct{})
+ var once sync.Once
+ a.worker.onCheckpoint = func() {
+ once.Do(func() {
+ close(started)
+ <-release
+ })
+ }
+ cpErr := make(chan error, 1)
+ go func() {
+ _, err := a.checkpoint(t, a.snapshot("pause-snap-1", 1), ateletpb.UploadMode_UPLOAD_MODE_DEFERRED)
+ cpErr <- err
+ }()
+ select {
+ case <-started:
+ case <-time.After(10 * time.Second):
+ t.Fatal("timed out waiting for the Checkpoint to reach the worker")
+ }
+
+ type detachResult struct {
+ resp *ateletpb.DetachActorResponse
+ err error
+ }
+ detached := make(chan detachResult, 1)
+ go func() {
+ resp, err := a.s.DetachActor(t.Context(), detachActorRequest())
+ detached <- detachResult{resp, err}
+ }()
+ select {
+ case r := <-detached:
+ t.Fatalf("DetachActor returned (%v, %v) while a Checkpoint held the actor lock, want it to wait", r.resp, r.err)
+ case <-time.After(50 * time.Millisecond):
+ }
+
+ close(release)
+ if err := <-cpErr; err != nil {
+ t.Fatalf("Checkpoint: %v", err)
+ }
+ r := <-detached
+ if r.err != nil {
+ t.Fatalf("DetachActor: %v", r.err)
+ }
+ if st := r.resp.GetState(); st.GetLatestLocal() != nil || st.GetLatestDurable().GetSnapshot().GetSnapshotUuid() != "pause-snap-1" {
+ t.Errorf("state = %v, want pause-snap-1 durable and nothing local", st)
+ }
+}
+
// TestInvalidRunKeepsDetachScheduled runs an actor with a request that
// names no sandbox runtime: Run fails before it begins the start, so the
// actor's detach stays scheduled.
@@ -1593,7 +1782,6 @@ func TestInvalidRunKeepsDetachScheduled(t *testing.T) {
ActorTemplateAtespace: "default",
ActorTemplateName: "counter",
WorkerPodUid: pluginWorkerPodUID,
- SandboxAssets: a.assets,
Spec: a.spec,
})
if apierror.Code(err) != codes.InvalidArgument {
@@ -1719,19 +1907,25 @@ func TestPromoteSnapshotFoundAlreadyUploaded(t *testing.T) {
}
}
-// advertiseRuntime makes the runtime that sa names one a worker on this
+// testSandboxRecord is the record of a gVisor runtime whose only asset is
+// release, as the syncer plans it from a SandboxConfig version.
+func testSandboxRecord(release gvisorTestRelease, pauseImage string) *sandboxAssetsRecord {
+ return &sandboxAssetsRecord{
+ SandboxClass: "gvisor",
+ PauseImage: pauseImage,
+ Assets: map[string]assetEntry{gvisorAssetName: release.entry},
+ }
+}
+
+// advertiseRuntime makes the runtime that rec describes one a worker on this
// node advertises, as the syncer would have after planning that worker's
-// pod, so a Run or Restore carrying sa resolves. It gives s an index if it
-// has none.
-func advertiseRuntime(t *testing.T, s *Atelet, sa *ateletpb.SandboxAssets) {
+// pod, so a Run or Restore carrying runtimeFor(rec) resolves. It gives s an
+// index if it has none.
+func advertiseRuntime(t *testing.T, s *Atelet, rec *sandboxAssetsRecord) {
t.Helper()
if s.runtimes == nil {
s.runtimes = newRuntimeIndex()
}
- rec, err := recordFromRequest(sa)
- if err != nil {
- t.Fatalf("recordFromRequest: %v", err)
- }
hash, err := versionBundleHash(rec)
if err != nil {
t.Fatalf("versionBundleHash: %v", err)
@@ -1743,12 +1937,16 @@ func advertiseRuntime(t *testing.T, s *Atelet, sa *ateletpb.SandboxAssets) {
})
}
-// TestRunRejectsUnadvertisedRuntime pins that an actor placed for a runtime
-// no worker on this node advertises is refused before anything is fetched
-// or staged, rather than run on whatever the request describes.
-func TestRunRejectsUnadvertisedRuntime(t *testing.T) {
+// TestRunAndRestoreRejectUnadvertisedRuntime pins that an actor placed for a
+// runtime no worker on this node advertises is refused by Run and Restore
+// alike, before anything is fetched or staged, and that either request
+// without a runtime is invalid.
+func TestRunAndRestoreRejectUnadvertisedRuntime(t *testing.T) {
useTempNodeDirs(t)
- const actorUID = "01234567-89ab-cdef-0123-456789abcdef"
+ const (
+ actorUID = "01234567-89ab-cdef-0123-456789abcdef"
+ workerPodUID = "5f1c0d2e-3a4b-4c5d-8e9f-0a1b2c3d4e5f"
+ )
release := newGVisorTestRelease(t)
var gets atomic.Int32
s := &Atelet{
@@ -1757,58 +1955,68 @@ func TestRunRejectsUnadvertisedRuntime(t *testing.T) {
anonGCSClient: fakeObjectStorage{data: release.tarball, gets: &gets},
systemInfoVolumes: newSystemInfoVolumeRefresher(nil, nil),
}
- assets := &ateletpb.SandboxAssets{
- SandboxClass: "gvisor",
- PauseImage: testPauseImage,
- Assets: map[string]*ateletpb.ArchAssets{runtime.GOARCH: {Files: map[string]*ateletpb.AssetFile{
- gvisorAssetName: release.proto(),
- }}},
- }
- _, err := s.Run(t.Context(), &ateletpb.RunRequest{
- Atespace: "team-a", ActorName: "actor-1", ActorUid: actorUID, WorkerPodUid: "5f1c0d2e-3a4b-4c5d-8e9f-0a1b2c3d4e5f",
- SandboxAssets: assets, SandboxRuntime: runtimeFor(t, assets), Spec: &ateletpb.WorkloadSpec{},
- })
- if apierror.Code(err) != codes.FailedPrecondition || !strings.Contains(err.Error(), "not advertised") {
- t.Fatalf("Run for an unadvertised runtime = %v (code %s), want %s naming the runtime", err, apierror.Code(err), codes.FailedPrecondition)
- }
- if gets.Load() != 0 {
- t.Errorf("assets fetched %d times for a refused Run, want 0", gets.Load())
- }
- if _, err := os.Lstat(ateletpath.ActorPath(actorUID)); !os.IsNotExist(err) {
- t.Errorf("refused Run created the actor dir (lstat err = %v)", err)
- }
+ rec := testSandboxRecord(release, testPauseImage)
- // A request that names no runtime is invalid, whatever its assets say.
- _, err = s.Run(t.Context(), &ateletpb.RunRequest{
- Atespace: "team-a", ActorName: "actor-1", ActorUid: actorUID, WorkerPodUid: "5f1c0d2e-3a4b-4c5d-8e9f-0a1b2c3d4e5f",
- SandboxAssets: assets, Spec: &ateletpb.WorkloadSpec{},
- })
- if apierror.Code(err) != codes.InvalidArgument {
- t.Errorf("Run without sandbox_runtime = %v (code %s), want %s", err, apierror.Code(err), codes.InvalidArgument)
+ for _, tc := range []struct {
+ name string
+ call func(t *testing.T, rt *ateletpb.SandboxRuntime) error
+ }{{
+ name: "Run",
+ call: func(t *testing.T, rt *ateletpb.SandboxRuntime) error {
+ _, err := s.Run(t.Context(), &ateletpb.RunRequest{
+ Atespace: "team-a", ActorName: "actor-1", ActorUid: actorUID, WorkerPodUid: workerPodUID,
+ SandboxRuntime: rt, Spec: &ateletpb.WorkloadSpec{},
+ })
+ return err
+ },
+ }, {
+ name: "Restore",
+ call: func(t *testing.T, rt *ateletpb.SandboxRuntime) error {
+ _, err := s.Restore(t.Context(), &ateletpb.RestoreRequest{
+ Atespace: "team-a", ActorName: "actor-1", ActorUid: actorUID, WorkerPodUid: workerPodUID,
+ SandboxRuntime: rt, Spec: &ateletpb.WorkloadSpec{},
+ Snapshot: lifecycleSnapshot(actorUID, "pause-snap-1", 1, ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY),
+ })
+ return err
+ },
+ }} {
+ t.Run(tc.name, func(t *testing.T) {
+ err := tc.call(t, runtimeFor(t, rec))
+ if apierror.Code(err) != codes.FailedPrecondition || !strings.Contains(err.Error(), "not advertised") {
+ t.Fatalf("%s for an unadvertised runtime = %v (code %s), want %s naming the runtime", tc.name, err, apierror.Code(err), codes.FailedPrecondition)
+ }
+ if gets.Load() != 0 {
+ t.Errorf("assets fetched %d times for a refused %s, want 0", gets.Load(), tc.name)
+ }
+ if _, err := os.Lstat(ateletpath.ActorPath(actorUID)); !os.IsNotExist(err) {
+ t.Errorf("refused %s created the actor dir (lstat err = %v)", tc.name, err)
+ }
+
+ // A request that names no runtime is invalid.
+ if err := tc.call(t, nil); apierror.Code(err) != codes.InvalidArgument {
+ t.Errorf("%s without sandbox_runtime = %v (code %s), want %s", tc.name, err, apierror.Code(err), codes.InvalidArgument)
+ }
+ })
}
- // Once a worker advertises it, the same request resolves.
- advertiseRuntime(t, s, assets)
- if _, err := s.advertisedRuntime(t.Context(), runtimeFor(t, assets)); err != nil {
+ // Once a worker advertises it, the same runtime resolves.
+ advertiseRuntime(t, s, rec)
+ if _, err := s.advertisedRuntime(t.Context(), runtimeFor(t, rec)); err != nil {
t.Errorf("advertisedRuntime after advertising: %v", err)
}
}
// runtimeFor returns the SandboxRuntime ate-api-server sends for an actor
-// placed on a worker that advertises the runtime sa names: the worker's
-// registration, whose sandbox_bundle_hash is the hash of sa's record.
-func runtimeFor(t *testing.T, sa *ateletpb.SandboxAssets) *ateletpb.SandboxRuntime {
+// placed on a worker that advertises the runtime rec describes: the worker's
+// registration, whose sandbox_bundle_hash is the hash of rec.
+func runtimeFor(t *testing.T, rec *sandboxAssetsRecord) *ateletpb.SandboxRuntime {
t.Helper()
- rec, err := recordFromRequest(sa)
- if err != nil {
- t.Fatalf("recordFromRequest: %v", err)
- }
hash, err := versionBundleHash(rec)
if err != nil {
t.Fatalf("versionBundleHash: %v", err)
}
return &ateletpb.SandboxRuntime{
- SandboxClass: sa.GetSandboxClass(),
+ SandboxClass: rec.SandboxClass,
Version: &ateletpb.VersionedSandboxCompat{
SchemaVersion: hardware.SchemaVersionV1,
Attributes: []*ateletpb.AttributeEntry{
@@ -1975,7 +2183,7 @@ func TestCheckpointKeepsSnapshotBeingUploaded(t *testing.T) {
}
// TestCheckpointRejectsStaleGeneration checks that Checkpoint fails with
-// FailedPrecondition when given a generation below the actor's newest snapshot
+// Aborted when given a generation below the actor's newest snapshot
// or equal to it under another snapshot_uuid, before calling worker, pruning
// local checkpoints, or changing tracked state.
func TestCheckpointRejectsStaleGeneration(t *testing.T) {
@@ -1993,9 +2201,13 @@ func TestCheckpointRejectsStaleGeneration(t *testing.T) {
a.snapshot("stale-snap-1", 1),
a.snapshot("other-snap-2", 2),
} {
- _, err := a.checkpoint(t, stale, ateletpb.UploadMode_UPLOAD_MODE_DEFERRED)
- if code := apierror.Code(err); code != codes.FailedPrecondition {
- t.Errorf("Checkpoint(%v) error = %v (code %v), want FailedPrecondition", stale, err, code)
+ _, err := ateinterceptors.InternalServerUnaryInterceptor(t.Context(), nil,
+ &grpc.UnaryServerInfo{FullMethod: ateletpb.Atelet_Checkpoint_FullMethodName},
+ func(context.Context, any) (any, error) {
+ return a.checkpoint(t, stale, ateletpb.UploadMode_UPLOAD_MODE_DEFERRED)
+ })
+ if code := status.Code(err); code != codes.Aborted {
+ t.Errorf("Checkpoint(%v) error = %v (code %v), want Aborted", stale, err, code)
}
if got := a.worker.actorDirs["CheckpointWorkload"]; got != nil {
t.Errorf("Checkpoint(%v) called worker.CheckpointWorkload (%v), want rejected before worker", stale, got)
@@ -2010,7 +2222,7 @@ func TestCheckpointRejectsStaleGeneration(t *testing.T) {
}
// TestPromoteSnapshotRejectsStaleGeneration checks that PromoteSnapshot fails
-// with FailedPrecondition when given a generation below the actor's newest
+// with Aborted when given a generation below the actor's newest
// snapshot or equal to it under another snapshot_uuid, before uploading to
// object storage, pruning local checkpoints, or changing tracked state.
func TestPromoteSnapshotRejectsStaleGeneration(t *testing.T) {
@@ -2028,21 +2240,24 @@ func TestPromoteSnapshotRejectsStaleGeneration(t *testing.T) {
} {
writeLocalSnapshot(t, ateletpath.LocalSnapshotDir(pluginActorUID, stale.GetSnapshotUuid()), sandboxAssetsRecord{
SandboxClass: "gvisor",
- PauseImage: a.assets.GetPauseImage(),
+ PauseImage: a.rec.PauseImage,
SnapshotFiles: []string{"checkpoint.img"},
Fidelity: "MEMORY",
}, map[string]string{"checkpoint.img": "stale-memory"})
- _, err := a.s.PromoteSnapshot(t.Context(), &ateletpb.PromoteSnapshotRequest{
+ req := &ateletpb.PromoteSnapshotRequest{
Atespace: "ate-demo",
ActorName: "counter",
ActorUid: pluginActorUID,
ActorTemplateAtespace: "default",
ActorTemplateName: "counter",
Snapshot: stale,
- })
- if code := apierror.Code(err); code != codes.FailedPrecondition {
- t.Errorf("PromoteSnapshot(%v) error = %v (code %v), want FailedPrecondition", stale, err, code)
+ }
+ _, err := ateinterceptors.InternalServerUnaryInterceptor(t.Context(), req,
+ &grpc.UnaryServerInfo{FullMethod: ateletpb.Atelet_PromoteSnapshot_FullMethodName},
+ func(ctx context.Context, _ any) (any, error) { return a.s.PromoteSnapshot(ctx, req) })
+ if code := status.Code(err); code != codes.Aborted {
+ t.Errorf("PromoteSnapshot(%v) error = %v (code %v), want Aborted", stale, err, code)
}
if got := a.store.keys(); len(got) != 0 {
t.Errorf("objects after rejected PromoteSnapshot(%v) = %v, want nothing uploaded", stale, got)
@@ -2058,7 +2273,8 @@ func TestPromoteSnapshotRejectsStaleGeneration(t *testing.T) {
// TestCheckpointInFlightRejectsOlderGeneration checks that while Checkpoint is
// running on the worker for generation 3, an out-of-order Checkpoint or
-// PromoteSnapshot for generation 2 fails with FailedPrecondition.
+// PromoteSnapshot for generation 2 waits on the per-actor RPC lock and then
+// fails with Aborted.
func TestCheckpointInFlightRejectsOlderGeneration(t *testing.T) {
a := runPluginActor(t)
started := make(chan struct{})
@@ -2083,24 +2299,41 @@ func TestCheckpointInFlightRejectsOlderGeneration(t *testing.T) {
}
stale := a.snapshot("pause-snap-2", 2)
- if _, err := a.checkpoint(t, stale, ateletpb.UploadMode_UPLOAD_MODE_DEFERRED); apierror.Code(err) != codes.FailedPrecondition {
- t.Errorf("Checkpoint(%v) while generation 3 is in flight = %v, want FailedPrecondition", stale, err)
- }
- if _, err := a.s.PromoteSnapshot(t.Context(), &ateletpb.PromoteSnapshotRequest{
- Atespace: "ate-demo",
- ActorName: "counter",
- ActorUid: pluginActorUID,
- ActorTemplateAtespace: "default",
- ActorTemplateName: "counter",
- Snapshot: stale,
- }); apierror.Code(err) != codes.FailedPrecondition {
- t.Errorf("PromoteSnapshot(%v) while generation 3 is in flight = %v, want FailedPrecondition", stale, err)
+ staleCpErr := make(chan error, 1)
+ go func() {
+ _, err := a.checkpoint(t, stale, ateletpb.UploadMode_UPLOAD_MODE_DEFERRED)
+ staleCpErr <- err
+ }()
+ stalePromoteErr := make(chan error, 1)
+ go func() {
+ _, err := a.s.PromoteSnapshot(t.Context(), &ateletpb.PromoteSnapshotRequest{
+ Atespace: "ate-demo",
+ ActorName: "counter",
+ ActorUid: pluginActorUID,
+ ActorTemplateAtespace: "default",
+ ActorTemplateName: "counter",
+ Snapshot: stale,
+ })
+ stalePromoteErr <- err
+ }()
+ select {
+ case err := <-staleCpErr:
+ t.Fatalf("Checkpoint(%v) returned %v while generation 3 was in flight, want it to wait", stale, err)
+ case err := <-stalePromoteErr:
+ t.Fatalf("PromoteSnapshot(%v) returned %v while generation 3 was in flight, want it to wait", stale, err)
+ case <-time.After(50 * time.Millisecond):
}
close(release)
if err := <-cpErr; err != nil {
t.Fatalf("Checkpoint(pause-snap-3): %v", err)
}
+ if err := <-staleCpErr; apierror.Code(err) != codes.Aborted {
+ t.Errorf("Checkpoint(%v) after generation 3 committed = %v, want Aborted", stale, err)
+ }
+ if err := <-stalePromoteErr; apierror.Code(err) != codes.Aborted {
+ t.Errorf("PromoteSnapshot(%v) after generation 3 committed = %v, want Aborted", stale, err)
+ }
if got := a.s.snapshots.State(pluginActorUID).GetLatestLocal().GetSnapshot().GetSnapshotUuid(); got != "pause-snap-3" {
t.Errorf("latest_local = %q, want pause-snap-3", got)
}
@@ -2141,7 +2374,7 @@ func TestCheckpointSkipsStaleCommitAndKeepsNewerLocal(t *testing.T) {
snap3Dir := ateletpath.LocalSnapshotDir(pluginActorUID, snap3.GetSnapshotUuid())
writeLocalSnapshot(t, snap3Dir, sandboxAssetsRecord{
SandboxClass: "gvisor",
- PauseImage: a.assets.GetPauseImage(),
+ PauseImage: a.rec.PauseImage,
SnapshotFiles: []string{"checkpoint.img"},
Fidelity: "MEMORY",
}, map[string]string{"checkpoint.img": "newer-memory"})
diff --git a/cmd/atelet/main.go b/cmd/atelet/main.go
index 4384e43f0c..49c94997a1 100644
--- a/cmd/atelet/main.go
+++ b/cmd/atelet/main.go
@@ -32,13 +32,17 @@ import (
"github.com/google/go-containerregistry/pkg/authn"
"github.com/spf13/pflag"
+ "github.com/spiffe/go-spiffe/v2/spiffeid"
+ "github.com/spiffe/go-spiffe/v2/svid/x509svid"
"go.ate.dev/substrate/cmd/atelet/internal/apivalidation"
"go.ate.dev/substrate/cmd/atelet/internal/ateletpath"
"go.ate.dev/substrate/cmd/atelet/internal/credentialprovider"
"go.ate.dev/substrate/cmd/atelet/internal/filecache"
+ "go.ate.dev/substrate/cmd/atelet/internal/filecachegc"
"go.ate.dev/substrate/cmd/atelet/internal/snapshottracker"
"go.ate.dev/substrate/cmd/atelet/internal/sparsefile"
"go.ate.dev/substrate/cmd/atelet/internal/trustbundle"
+ "go.ate.dev/substrate/internal/actorlock"
"go.ate.dev/substrate/internal/apierror"
"go.ate.dev/substrate/internal/ateapiauth"
"go.ate.dev/substrate/internal/ateattr"
@@ -47,6 +51,7 @@ import (
"go.ate.dev/substrate/internal/clustertrustbundle"
"go.ate.dev/substrate/internal/credbundle"
"go.ate.dev/substrate/internal/imagecache"
+ "go.ate.dev/substrate/internal/installdefaults"
"go.ate.dev/substrate/internal/nodepath"
"go.ate.dev/substrate/internal/ocispec"
"go.ate.dev/substrate/internal/otlprelay"
@@ -94,12 +99,13 @@ var (
metricsListenAddr = pflag.String("metrics-listen-addr", ":9090", "Address and port the prometheus metrics server should listen on.")
healthListenAddr = pflag.String("health-listen-addr", ":9091", "Address and port the /readyz and /healthz probe server should listen on.")
- grpcServerCredBundle = pflag.String("grpc-server-cred-bundle", "/run/podidentity.podcert.ate.dev/credential-bundle.pem", "Credential bundle atelet presents as its gRPC serving certificate.")
- clientCACerts = pflag.String("client-ca-certs", "/run/podidentity.podcert.ate.dev/trust-bundle.pem", "CA bundle used to verify gRPC client certificates.")
- ateapiAddress = pflag.String("ateapi-address", "k8s:///api.ate-system.svc:443", "ateapi gRPC target used by the credential broker.")
- ateapiCAFile = pflag.String("ateapi-ca-file", "/run/servicedns.podcert.ate.dev/trust-bundle.pem", "CA bundle used to verify ateapi.")
- grpcInsecure = pflag.Bool("grpc-insecure", false, "Serve gRPC without transport security. Intended only for local clusters without Pod Certificates.")
- ateapiServerName = pflag.String("ateapi-server-name", "api.ate-system.svc", "DNS name expected on the ateapi certificate.")
+ grpcServerCredBundle = pflag.String("grpc-server-cred-bundle", "/run/podidentity.podcert.ate.dev/credential-bundle.pem", "Credential bundle atelet presents as its gRPC serving certificate.")
+ clientCACerts = pflag.String("client-ca-certs", "/run/podidentity.podcert.ate.dev/trust-bundle.pem", "CA bundle used to verify gRPC client certificates.")
+ herderAllowedSPIFFEID = pflag.String("herder-allowed-spiffe-id", installdefaults.SPIFFEID(installdefaults.NamespaceFromPodEnv(), installdefaults.APIServerServiceAccount), "The only SPIFFE ID allowed to call AteomHerder.")
+ ateapiAddress = pflag.String("ateapi-address", "k8s:///api.ate-system.svc:443", "ateapi gRPC target used by the credential broker.")
+ ateapiCAFile = pflag.String("ateapi-ca-file", "/run/servicedns.podcert.ate.dev/trust-bundle.pem", "CA bundle used to verify ateapi.")
+ ateapiServerName = pflag.String("ateapi-server-name", "api.ate-system.svc", "DNS name expected on the ateapi certificate.")
+ grpcInsecure = pflag.Bool("grpc-insecure", false, "Serve gRPC without transport security. Intended only for local clusters without Pod Certificates.")
// The kubelet already knows how to authenticate to its node's cloud
// registry, via an exec plugin the node ships. Pointing atelet at the same
@@ -209,7 +215,7 @@ func main() {
// directly for its whole life, so the socket should exist before any worker
// pod on this node boots.
if relay, err := otlprelay.NewServer(ctx, *otlpRelaySocket); err != nil {
- slog.ErrorContext(ctx, "Failed to create the OTLP relay; workers will export directly", slog.Any("err", err))
+ slog.ErrorContext(ctx, "Failed to create the OTLP relay; workers fall back to their own OTLP endpoint", slog.Any("err", err))
} else if relay != nil {
// Deferred rather than tied to the drain: the relay carries other
// processes' telemetry, so it should outlive atelet's own RPC serving
@@ -218,7 +224,7 @@ func main() {
go func() {
if err := relay.Serve(ctx); err != nil {
// Not fatal: atelet's actual job does not depend on the relay,
- // and the workers fall back to exporting directly.
+ // and the workers fall back to their own OTLP endpoint.
slog.ErrorContext(ctx, "OTLP relay stopped", slog.Any("err", err))
}
}()
@@ -263,10 +269,18 @@ func main() {
if err := validateSandboxAssetCacheFlags(); err != nil {
serverboot.Fatal(ctx, "Invalid sandbox asset cache flags", err)
}
- sandboxAssetCache, err := openSandboxAssetCache(ctx, *sandboxAssetCacheDir, *sandboxAssetCacheMinAge, *sandboxAssetFetchTimeout)
+ sandboxAssetCache, err := openSandboxAssetCache(ctx, *sandboxAssetCacheDir, *sandboxAssetCacheMinAge, *sandboxAssetFetchTimeout, nodepath.ActorsDir, nodepath.WorkersDir())
if err != nil {
serverboot.Fatal(ctx, "Failed to open sandbox asset cache", err)
}
+ // One set of instruments for every file cache on the node, by name.
+ fileCacheInstruments, err := filecachegc.NewInstruments(otel.Meter("atelet"))
+ if err != nil {
+ serverboot.Fatal(ctx, "Failed to create file cache metrics", err)
+ }
+ if err := filecachegc.Start(ctx, ateattr.FileCacheSandboxAsset, sandboxAssetCache, sandboxAssetCacheGCFlags, fileCacheInstruments); err != nil {
+ serverboot.Fatal(ctx, "Failed to start the sandbox asset cache GC", err)
+ }
// Sandbox assets go through the object-store plugin; a gs:// asset it
// cannot serve, such as one in the public gs://gvisor bucket, is read
@@ -405,7 +419,12 @@ func main() {
if *grpcInsecure {
slog.WarnContext(ctx, "Serving atelet gRPC without transport security")
} else {
- tlsCfg, err := ateletServerTLSConfig(*grpcServerCredBundle, *clientCACerts)
+ herderCaller, err := spiffeid.FromString(*herderAllowedSPIFFEID)
+ if err != nil {
+ serverboot.Fatal(ctx, "Invalid --herder-allowed-spiffe-id", err)
+ }
+ // Any pod can get a podidentity certificate, so also check the caller.
+ tlsCfg, err := ateletServerTLSConfig(*grpcServerCredBundle, *clientCACerts, verifyClientSPIFFEID(herderCaller))
if err != nil {
serverboot.Fatal(ctx, "Failed to build server TLS config", err)
}
@@ -424,8 +443,10 @@ func main() {
nodeName = ateletIdentity.NodeName
serverOpts = ateletServerOptions(credentials.NewTLS(tlsCfg))
- workerFacingTLS := tlsCfg.Clone()
- workerFacingTLS.VerifyConnection = verifyClientOnSameNode(ateletIdentity)
+ workerFacingTLS, err := ateletServerTLSConfig(*grpcServerCredBundle, *clientCACerts, verifyClientOnSameNode(ateletIdentity))
+ if err != nil {
+ serverboot.Fatal(ctx, "Failed to build credential broker TLS config", err)
+ }
if err := os.Remove(nodepath.WorkerSupportSocket); err != nil && !errors.Is(err, os.ErrNotExist) {
serverboot.Fatal(ctx, "Failed to remove stale credential broker socket", err)
}
@@ -487,7 +508,7 @@ func main() {
ateFactory.Api().V1alpha1().WorkerPools(),
ateFactory.Api().V1alpha1().SandboxConfigs(),
wmService,
- nodepath.WorkersDir(), runtimes, prewarmer.enqueue)
+ nodepath.WorkersDir(), runtimes, prewarmer)
if err != nil {
// Without the syncer no runtime is ever indexed, so no actor could run.
serverboot.Fatal(ctx, "Failed to start the worker sandbox runtimes syncer", err)
@@ -542,7 +563,7 @@ func servingCheck(served <-chan struct{}) func(context.Context) error {
// ateletServerOptions are the options of the gRPC server that serves
// Atelet to ate-api-server. Its interceptor logs every request; fields
-// marked debug_redact (env values, asset URLs) are masked in that log.
+// marked debug_redact (env values) are masked in that log.
func ateletServerOptions(creds credentials.TransportCredentials) []grpc.ServerOption {
return []grpc.ServerOption{
grpc.Creds(creds),
@@ -672,10 +693,21 @@ type Atelet struct {
// snapshots tracks each actor's snapshots on this node.
snapshots *snapshottracker.Tracker
+ // actorLocks serializes lifecycle RPCs per actor.
+ actorLocks actorlock.Locks
}
var _ ateletpb.AteletServer = (*Atelet)(nil)
+// lockActor acquires the per-actor RPC lock for actorUID, or returns an error
+// wrapping ctx.Err() if ctx ends while waiting.
+func (s *Atelet) lockActor(ctx context.Context, actorUID string) (func(), error) {
+ if !s.actorLocks.Lock(ctx, actorUID) {
+ return nil, fmt.Errorf("gave up waiting for the actor's lock: %w", ctx.Err())
+ }
+ return func() { s.actorLocks.Unlock(actorUID) }, nil
+}
+
// NewService creates a new WorkersManagerService.
func NewService(
ctx context.Context,
@@ -728,6 +760,11 @@ func (s *Atelet) Run(ctx context.Context, req *ateletpb.RunRequest) (resp *atele
if err != nil {
return nil, err
}
+ unlockActor, err := s.lockActor(ctx, actorUID)
+ if err != nil {
+ return nil, err
+ }
+ defer unlockActor()
endStart, err := s.snapshots.BeginStart(ctx, actorUID)
if err != nil {
return nil, err
@@ -907,7 +944,13 @@ func (s *Atelet) Checkpoint(ctx context.Context, req *ateletpb.CheckpointRequest
snapshotLogAttrs(attribution, op, checkpointDurationMetric, err, phases)...)
}()
- // A detach in progress finishes first, and none starts until the
+ unlockActor, err := s.lockActor(ctx, actorUID)
+ if err != nil {
+ return nil, err
+ }
+ defer unlockActor()
+
+ // A detach in progress finishes first, and no detach starts until the
// checkpoint returns, so it cannot delete the snapshot being written.
endCheckpoint, err := s.snapshots.BeginCheckpoint(ctx, actorUID, req.GetSnapshot())
if err != nil {
@@ -1218,6 +1261,12 @@ func (s *Atelet) PromoteSnapshot(ctx context.Context, req *ateletpb.PromoteSnaps
}
localDir := ateletpath.LocalSnapshotDir(req.GetActorUid(), req.GetSnapshot().GetSnapshotUuid())
+ unlockActor, err := s.lockActor(ctx, req.GetActorUid())
+ if err != nil {
+ return nil, err
+ }
+ defer unlockActor()
+
// A detach or a background upload of this snapshot in progress finishes
// first, and no detach starts until the promote returns.
snap := snapshottracker.SnapshotRef{Snapshot: req.GetSnapshot(), Actor: resources.ActorRef{Atespace: req.GetAtespace(), Name: req.GetActorName()}}
@@ -1264,7 +1313,7 @@ func (s *Atelet) uploadLocalCheckpointDir(ctx context.Context, req *ateletpb.Pro
return "", false, nil
}
if status.Code(fetchErr) == codes.NotFound {
- return "", false, fmt.Errorf("local snapshot %q is gone and no uploaded copy exists: %w",
+ return "", false, apierror.NotFound("local snapshot %q is gone and no uploaded copy exists: %w",
localSnapshotName, fetchErr)
}
return "", false, fmt.Errorf("while probing for an already-uploaded snapshot manifest: %w", fetchErr)
@@ -1378,7 +1427,7 @@ func (s *Atelet) Restore(ctx context.Context, req *ateletpb.RestoreRequest) (res
templateNamespace: req.GetActorTemplateAtespace(),
templateName: req.GetActorTemplateName(),
fidelity: ateattr.SnapshotFidelityValue(req.GetSnapshot().GetFidelity()),
- sandboxClass: req.GetSandboxAssets().GetSandboxClass(),
+ sandboxClass: req.GetSandboxRuntime().GetSandboxClass(),
}
attribution := resources.ActorAttribution{
Ref: actorRef,
@@ -1403,8 +1452,14 @@ func (s *Atelet) Restore(ctx context.Context, req *ateletpb.RestoreRequest) (res
snapshotLogAttrs(attribution, op, restoreDurationMetric, err, phases)...)
}()
- // A detach in progress finishes first; its local copy is then gone, and
- // the restore reads snapshot.object.
+ unlockActor, err := s.lockActor(ctx, actorUID)
+ if err != nil {
+ return nil, err
+ }
+ defer unlockActor()
+
+ // A detach in progress finishes first. If it ran, its local copy is gone,
+ // and the restore reads snapshot.object.
endStart, err := s.snapshots.BeginStart(ctx, actorUID)
if err != nil {
return nil, err
@@ -1463,10 +1518,6 @@ func (s *Atelet) Restore(ctx context.Context, req *ateletpb.RestoreRequest) (res
checkpointDir = localDir
sandboxRec = localRec
} else {
- if durable == nil {
- return nil, apierror.NotFound("snapshot %q has no usable local copy and no snapshot.object to fall back to: %w",
- req.GetSnapshot().GetSnapshotUuid(), localErr)
- }
if errors.Is(localErr, errUnusableLocalSnapshot) {
// The next checkpoint or terminate prunes the unusable copy.
slog.WarnContext(ctx, "Local snapshot is unusable; restoring from object storage",
@@ -1609,6 +1660,25 @@ func (s *Atelet) Restore(ctx context.Context, req *ateletpb.RestoreRequest) (res
return &ateletpb.RestoreResponse{}, nil
}
+// DetachActor makes the actor's latest local snapshot durable, then deletes the
+// actor's local snapshots on this node.
+func (s *Atelet) DetachActor(ctx context.Context, req *ateletpb.DetachActorRequest) (*ateletpb.DetachActorResponse, error) {
+ if errs := apivalidation.ValidateDetachActorRequest(ctx, req); len(errs) > 0 {
+ return nil, resources.ToAPIError(errs)
+ }
+ unlockActor, err := s.lockActor(ctx, req.GetActorUid())
+ if err != nil {
+ return nil, err
+ }
+ defer unlockActor()
+
+ state, err := s.snapshots.Detach(ctx, req.GetActorUid())
+ if err != nil {
+ return nil, err
+ }
+ return &ateletpb.DetachActorResponse{State: state}, nil
+}
+
// Terminate terminates any running workload on worker, unmounts external volumes,
// and resets actor directories on the node.
func (s *Atelet) Terminate(ctx context.Context, req *ateletpb.TerminateRequest) (*ateletpb.TerminateResponse, error) {
@@ -1619,6 +1689,12 @@ func (s *Atelet) Terminate(ctx context.Context, req *ateletpb.TerminateRequest)
actorRef := resources.ActorRef{Atespace: req.GetAtespace(), Name: req.GetActorName()}
actorUID := req.GetActorUid()
+ unlockActor, err := s.lockActor(ctx, actorUID)
+ if err != nil {
+ return nil, err
+ }
+ defer unlockActor()
+
if req.GetWorkerPodUid() != "" {
var assetPaths map[string]string
sandboxRec, err := readSandboxRecord(actorUID)
@@ -2165,8 +2241,13 @@ func removeActorDirs(actorUID string) error {
// ateletServerTLSConfig builds a *tls.Config for a gRPC server that presents the
// credential bundle at servingBundlePath, requires a client certificate
-// chaining to a CA in clientCAPath.
-func ateletServerTLSConfig(servingBundlePath, clientCAPath string) (*tls.Config, error) {
+// chaining to a CA in clientCAPath, and authorizes the client with
+// verifyClient. verifyClient must be set on the config GetConfigForClient
+// returns; crypto/tls ignores the outer config's VerifyConnection.
+func ateletServerTLSConfig(servingBundlePath, clientCAPath string, verifyClient func(tls.ConnectionState) error) (*tls.Config, error) {
+ if verifyClient == nil {
+ return nil, fmt.Errorf("client authorization callback is required")
+ }
loadClientCAs := credbundle.PoolLoader(clientCAPath)
if _, err := loadClientCAs(); err != nil {
return nil, fmt.Errorf("load CA bundle %s: %w", clientCAPath, err)
@@ -2180,15 +2261,34 @@ func ateletServerTLSConfig(servingBundlePath, clientCAPath string) (*tls.Config,
return nil, err
}
return &tls.Config{
- MinVersion: tls.VersionTLS13,
- GetCertificate: serverCert,
- ClientAuth: tls.RequireAndVerifyClientCert,
- ClientCAs: clientCAs,
+ MinVersion: tls.VersionTLS13,
+ GetCertificate: serverCert,
+ ClientAuth: tls.RequireAndVerifyClientCert,
+ ClientCAs: clientCAs,
+ VerifyConnection: verifyClient,
}, nil
},
}, nil
}
+// verifyClientSPIFFEID returns a TLS callback that accepts only a client whose
+// verified certificate carries exactly the SPIFFE ID want.
+func verifyClientSPIFFEID(want spiffeid.ID) func(tls.ConnectionState) error {
+ return func(state tls.ConnectionState) error {
+ if len(state.VerifiedChains) == 0 || len(state.VerifiedChains[0]) == 0 {
+ return fmt.Errorf("verified client certificate is required")
+ }
+ got, err := x509svid.IDFromCert(state.VerifiedChains[0][0])
+ if err != nil {
+ return fmt.Errorf("client certificate has no valid SPIFFE ID: %w", err)
+ }
+ if got != want {
+ return fmt.Errorf("client SPIFFE ID %q is not authorized, want %q", got, want)
+ }
+ return nil
+ }
+}
+
func newKubeClients() (*kubernetes.Clientset, versioned.Interface, error) {
config, err := rest.InClusterConfig()
if err != nil {
diff --git a/cmd/atelet/main_test.go b/cmd/atelet/main_test.go
index 97c4099583..9e292c1312 100644
--- a/cmd/atelet/main_test.go
+++ b/cmd/atelet/main_test.go
@@ -40,6 +40,7 @@ import (
"go.ate.dev/substrate/cmd/atelet/internal/snapshottracker"
"go.ate.dev/substrate/internal/apierror"
"go.ate.dev/substrate/internal/ateattr"
+ "go.ate.dev/substrate/internal/ateinterceptors"
"go.ate.dev/substrate/internal/atelet"
"go.ate.dev/substrate/internal/imagecache"
"go.ate.dev/substrate/internal/ocispec"
@@ -52,6 +53,7 @@ import (
"google.golang.org/grpc"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/credentials/insecure"
+ "google.golang.org/grpc/status"
"google.golang.org/protobuf/testing/protocmp"
"google.golang.org/protobuf/types/known/emptypb"
)
@@ -1305,6 +1307,7 @@ type recordingObjectStorage struct {
mu sync.Mutex
objects map[string][]byte
putErr error
+ onPut func()
}
func (r *recordingObjectStorage) GetObject(_ context.Context, bucket, object string) (io.ReadCloser, error) {
@@ -1318,6 +1321,9 @@ func (r *recordingObjectStorage) GetObject(_ context.Context, bucket, object str
}
func (r *recordingObjectStorage) PutObject(_ context.Context, bucket, object string, reader io.Reader) error {
+ if r.onPut != nil {
+ r.onPut()
+ }
if r.putErr != nil {
return r.putErr
}
@@ -1563,12 +1569,15 @@ func TestUploadLocalCheckpointDir(t *testing.T) {
}
})
- t.Run("gone locally and remotely crashes the actor", func(t *testing.T) {
+ t.Run("gone locally and remotely is NotFound", func(t *testing.T) {
s := newPluginHerder(t, &recordingObjectStorage{})
_, _, err := s.uploadLocalCheckpointDir(ctx, validPromoteSnapshotRequest(), filepath.Join(t.TempDir(), "never-created"), uri)
- if err == nil {
- t.Fatal("uploadLocalCheckpointDir succeeded, want error")
+ // atelet's server interceptor decides the code ate-api-server sees.
+ _, wireErr := ateinterceptors.InternalServerUnaryInterceptor(ctx, nil, &grpc.UnaryServerInfo{FullMethod: "/test"},
+ func(context.Context, any) (any, error) { return nil, err })
+ if got := status.Code(wireErr); got != codes.NotFound {
+ t.Fatalf("code = %v (err %v), want NotFound", got, wireErr)
}
if !strings.Contains(err.Error(), "gone and no uploaded copy exists") {
t.Errorf("error = %v, want it to name the unrecoverable local snapshot", err)
diff --git a/cmd/atelet/manifest_test.go b/cmd/atelet/manifest_test.go
new file mode 100644
index 0000000000..1537ec45c7
--- /dev/null
+++ b/cmd/atelet/manifest_test.go
@@ -0,0 +1,64 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package main
+
+import (
+ "bytes"
+ "os"
+ "strings"
+ "testing"
+
+ "go.ate.dev/substrate/internal/atelet"
+ appsv1 "k8s.io/api/apps/v1"
+ "sigs.k8s.io/yaml"
+)
+
+const ateletManifest = "../../manifests/ate-install/atelet.yaml"
+
+// TestAteletManifestDoesNotExposeHerderOnHostPort pins that the AteomHerder
+// listener is reachable only on atelet's Pod IP. ate-api-server dials the Pod
+// IP, so a hostPort would only add every node IP as a way in.
+func TestAteletManifestDoesNotExposeHerderOnHostPort(t *testing.T) {
+ raw, err := os.ReadFile(ateletManifest)
+ if err != nil {
+ t.Fatal(err)
+ }
+ found := false
+ for _, doc := range bytes.Split(raw, []byte("\n---")) {
+ var ds appsv1.DaemonSet
+ if err := yaml.Unmarshal(doc, &ds); err != nil || ds.Kind != "DaemonSet" || !strings.HasPrefix(ds.Name, "atelet") {
+ continue
+ }
+ for _, c := range ds.Spec.Template.Spec.Containers {
+ for _, p := range c.Ports {
+ if p.Name != "grpc" && p.ContainerPort != atelet.DefaultPort {
+ continue
+ }
+ found = true
+ if p.HostPort != 0 {
+ t.Errorf("container %q port %q sets hostPort %d; AteomHerder must not listen on the node IP", c.Name, p.Name, p.HostPort)
+ }
+ }
+ }
+ // ate-api-server dials atelet's Pod IP; with hostNetwork that would be
+ // the node IP, and removing the hostPort would protect nothing.
+ if ds.Spec.Template.Spec.HostNetwork {
+ t.Errorf("atelet DaemonSet uses hostNetwork, which exposes AteomHerder on the node IP")
+ }
+ }
+ if !found {
+ t.Fatal("found no atelet DaemonSet with a grpc port; the manifest changed shape and this test is checking nothing")
+ }
+}
diff --git a/cmd/atelet/rpc_logging_test.go b/cmd/atelet/rpc_logging_test.go
index 74f52c604a..1326e2570b 100644
--- a/cmd/atelet/rpc_logging_test.go
+++ b/cmd/atelet/rpc_logging_test.go
@@ -47,15 +47,12 @@ func (requestLogHerder) Restore(context.Context, *ateletpb.RestoreRequest) (*ate
return nil, apierror.Unavailable("asset fetch failed")
}
-// TestAteletRequestLogMasksSignedAssetURL sends Run and Restore with a
-// signed https asset URL through a server built with atelet's own options and
-// logger, and checks the request log carries neither the URL's user info nor
-// its query.
-func TestAteletRequestLogMasksSignedAssetURL(t *testing.T) {
- const (
- signedURL = "https://asset-user:userinfo-secret@storage.example.com/bucket/gvisor.tar.zstd?X-Goog-Signature=query-secret&X-Goog-Expires=900"
- sum = "d547d81401461fd1c679c5c4fa0a6c2b8ef7dc3c22ce23c9e25dcc4c69cfd06f"
- )
+// TestAteletRequestLogMasksEnvVarValue sends Run and Restore with an
+// environment variable value through a server built with atelet's own options
+// and logger, and checks the request log masks the value while keeping the
+// variable name.
+func TestAteletRequestLogMasksEnvVarValue(t *testing.T) {
+ const secretVal = "sk-secret-env-value"
var logBuf bytes.Buffer
origLogger := slog.Default()
t.Cleanup(func() { slog.SetDefault(origLogger) })
@@ -75,26 +72,24 @@ func TestAteletRequestLogMasksSignedAssetURL(t *testing.T) {
t.Cleanup(func() { conn.Close() })
client := ateletpb.NewAteletClient(conn)
- assets := &ateletpb.SandboxAssets{
- SandboxClass: "gvisor",
- Assets: map[string]*ateletpb.ArchAssets{
- "amd64": {Files: map[string]*ateletpb.AssetFile{"gvisor": {Url: signedURL, Sha256: sum}}},
- },
+ spec := &ateletpb.WorkloadSpec{
+ Containers: []*ateletpb.Container{{
+ Name: "main",
+ Env: []*ateletpb.EnvVar{{Name: "API_KEY", Value: secretVal}},
+ }},
}
ctx := t.Context()
- if _, err := client.Run(ctx, &ateletpb.RunRequest{ActorUid: "uid-run", SandboxAssets: assets}); err != nil {
+ if _, err := client.Run(ctx, &ateletpb.RunRequest{ActorUid: "uid-run", Spec: spec}); err != nil {
t.Fatalf("Run: %v", err)
}
- if _, err := client.Restore(ctx, &ateletpb.RestoreRequest{ActorUid: "uid-restore", SandboxAssets: assets}); status.Code(err) != codes.Unavailable {
+ if _, err := client.Restore(ctx, &ateletpb.RestoreRequest{ActorUid: "uid-restore", Spec: spec}); status.Code(err) != codes.Unavailable {
t.Fatalf("Restore err = %v, want Unavailable", err)
}
srv.Stop()
gotLog := logBuf.String()
- for _, leak := range []string{"userinfo-secret", "query-secret", "asset-user", "X-Goog-Signature"} {
- if strings.Contains(gotLog, leak) {
- t.Fatalf("request log contains %q from the signed asset URL:\n%s", leak, gotLog)
- }
+ if strings.Contains(gotLog, secretVal) {
+ t.Fatalf("request log contains %q from EnvVar.value:\n%s", secretVal, gotLog)
}
for _, method := range []string{ateletpb.Atelet_Run_FullMethodName, ateletpb.Atelet_Restore_FullMethodName} {
var line string
@@ -106,8 +101,8 @@ func TestAteletRequestLogMasksSignedAssetURL(t *testing.T) {
if line == "" {
t.Fatalf("no request log line for %s:\n%s", method, gotLog)
}
- // The sha256 still names the asset.
- for _, want := range []string{`"url":"` + protoredact.Placeholder + `"`, `"sha256":"` + sum + `"`} {
+ // The env var name is kept while its value is masked.
+ for _, want := range []string{`"value":"` + protoredact.Placeholder + `"`, `"name":"API_KEY"`} {
if !strings.Contains(line, want) {
t.Fatalf("%s request log missing %s:\n%s", method, want, line)
}
diff --git a/cmd/atelet/runtime_prewarm.go b/cmd/atelet/runtime_prewarm.go
index 8d7bd5b9fd..6801ed0706 100644
--- a/cmd/atelet/runtime_prewarm.go
+++ b/cmd/atelet/runtime_prewarm.go
@@ -18,6 +18,7 @@ import (
"context"
"errors"
"fmt"
+ "io/fs"
"log/slog"
"os"
"sync"
@@ -25,7 +26,6 @@ import (
"k8s.io/client-go/util/workqueue"
- "go.ate.dev/substrate/cmd/atelet/internal/ateletpath"
"go.ate.dev/substrate/internal/imagecache"
)
@@ -36,26 +36,44 @@ import (
// them.
const defaultPrewarmAttemptTimeout = 10 * time.Minute
-// prewarmMaxRetries bounds how often one runtime's failed prewarm is retried
-// before it is dropped until a worker advertises it again (or first use,
-// which stays the correctness path). The backoff below caps at 5 minutes,
-// so this covers transient bucket or registry outages of several minutes
+// prewarmMaxRetries bounds how often one failed prewarm is retried before
+// it is dropped until the worker is planned again (or first use, which
+// stays the correctness path). The backoff below caps at 5 minutes, so
+// this covers transient bucket or registry outages of several minutes
// without hammering a permanently broken runtime forever.
const prewarmMaxRetries = 8
-// sandboxAssetFetcher is the one slice of Atelet the prewarmer needs.
-// Prewarming through the same method as the Run/Restore path keeps the two
-// fetches from ever diverging on cache layout or validation.
+// sandboxAssetFetcher is the slice of Atelet the prewarmer and the syncer's
+// probe need. Staging through the same methods as the Run/Restore path
+// keeps the fetches from ever diverging on cache layout or validation.
type sandboxAssetFetcher interface {
ensureSandboxAssets(ctx context.Context, rec *sandboxAssetsRecord, dstDir string) (map[string]string, error)
+ stageSandboxAssets(ctx context.Context, rec *sandboxAssetsRecord, dstDir string) (map[string]string, error)
+}
+
+// prewarmItem is one default runtime of a worker to prewarm: the hash to
+// fetch, and the directory the worker sandbox syncer made for it under the
+// worker's own, to link the assets into.
+type prewarmItem struct {
+ hash string
+ dir string
}
// runtimePrewarmer fetches the default runtime of each worker on this node
// into the asset and image caches before any actor asks for it, so the
// fetch inside the first Run/Restore on the node is a cache hit instead of
// a download+extract on the critical path. The worker sandbox syncer hands
-// it the default runtime hash of every worker it plans; the index holds
-// the record to fetch.
+// it each default runtime a worker advertises, once the worker's file says
+// so, with a directory to stage it into; the index holds the record.
+//
+// The assets are linked into that directory and left there: the cache
+// never evicts a linked entry, so a default runtime stays cached while no
+// actor runs it. The directory is the syncer's to make and to remove: it
+// exists while the worker advertises the runtime as a default, and goes
+// with the worker's directory when the pod does. The prewarmer only fills
+// what exists. A directory gone before or while an item is staged means a
+// later sync or the worker's removal no longer wants it, and the item ends
+// without fetching, retrying, or recreating anything.
//
// Prewarming is purely a latency optimization: every failure is logged and
// left to the on-demand fetch in ensureSandboxAssets and the pull inside
@@ -66,28 +84,20 @@ type runtimePrewarmer struct {
index *runtimeIndex
assets sandboxAssetFetcher
images *imagecache.Store
- // queue carries runtime hashes. It dedupes while an item waits, and it
+ // queue carries items. It dedupes while an item waits, and it
// rate-limits retries after failures. A single worker drains it, which
// also serializes downloads so concurrent prewarms never compete for
// node bandwidth.
- queue workqueue.TypedRateLimitingInterface[string]
- // prewarmed holds the hashes whose prewarm succeeded since atelet
- // started. The queue forgets an item once it is done, and every pod of
- // a pool, every re-plan and every retry of a sync hands the same default
- // over again; enqueue skips what is here, so a runtime is prewarmed
- // once rather than re-staged and logged once per pod. Content-addressed
- // hashes make that safe: the bytes a hash names never change.
- mu sync.Mutex
- prewarmed map[string]bool
+ queue workqueue.TypedRateLimitingInterface[prewarmItem]
// done is closed when the worker goroutine exits; wait blocks on it.
done chan struct{}
// attemptTimeout is defaultPrewarmAttemptTimeout. Shortened in tests.
attemptTimeout time.Duration
- // onFinished, if set, observes the result of each prewarm attempt, by
- // runtime hash, once the worker's call has returned. Tests set it to
- // wait for that call before tearing down what it writes into.
- onFinished func(hash string, err error)
+ // onFinished, if set, observes the result of each prewarm attempt once
+ // the worker's call has returned. Tests set it to wait for that call
+ // before tearing down what it writes into.
+ onFinished func(item prewarmItem, err error)
}
func newRuntimePrewarmer(index *runtimeIndex, assets sandboxAssetFetcher, images *imagecache.Store) *runtimePrewarmer {
@@ -100,31 +110,16 @@ func newRuntimePrewarmer(index *runtimeIndex, assets sandboxAssetFetcher, images
// so back off in seconds and cap in minutes rather than the
// millisecond-based controller default.
queue: workqueue.NewTypedRateLimitingQueue(
- workqueue.NewTypedItemExponentialFailureRateLimiter[string](time.Second, 5*time.Minute)),
- prewarmed: map[string]bool{},
- done: make(chan struct{}),
+ workqueue.NewTypedItemExponentialFailureRateLimiter[prewarmItem](time.Second, 5*time.Minute)),
+ done: make(chan struct{}),
}
}
-// enqueue asks for the runtime advertised under hash to be prewarmed,
-// unless it already was. The syncer calls it as it plans a worker, after
-// registering the hash in the index.
-func (p *runtimePrewarmer) enqueue(hash string) {
- p.mu.Lock()
- already := p.prewarmed[hash]
- p.mu.Unlock()
- if already {
- return
- }
- p.queue.Add(hash)
-}
-
-// markPrewarmed records that hash's prewarm succeeded, so enqueue skips it
-// from now on.
-func (p *runtimePrewarmer) markPrewarmed(hash string) {
- p.mu.Lock()
- defer p.mu.Unlock()
- p.prewarmed[hash] = true
+// enqueue asks for the runtime advertised under hash to be prewarmed into
+// dir, which the syncer created. The syncer calls it once the worker's
+// file advertises the runtime as a default.
+func (p *runtimePrewarmer) enqueue(hash, dir string) {
+ p.queue.Add(prewarmItem{hash: hash, dir: dir})
}
// run drains the queue until ctx ends.
@@ -135,11 +130,11 @@ func (p *runtimePrewarmer) run(ctx context.Context) {
p.queue.ShutDown()
}()
for {
- hash, shutdown := p.queue.Get()
+ item, shutdown := p.queue.Get()
if shutdown {
return
}
- p.process(ctx, hash)
+ p.process(ctx, item)
}
}
@@ -150,68 +145,69 @@ func (p *runtimePrewarmer) wait() {
<-p.done
}
-// process prewarms the runtime under hash, requeueing with backoff on
-// failure.
-func (p *runtimePrewarmer) process(ctx context.Context, hash string) {
- defer p.queue.Done(hash)
- rt, ok := p.index.lookup(hash)
+// process prewarms the item, requeueing with backoff on failure.
+func (p *runtimePrewarmer) process(ctx context.Context, item prewarmItem) {
+ defer p.queue.Done(item)
+ rt, ok := p.index.lookup(item.hash)
if !ok {
// The syncer registers before it enqueues, and the index never
// forgets, so this is a programming error rather than a race.
- slog.ErrorContext(ctx, "Sandbox runtime to prewarm is not in the index", slog.String("runtime_hash", hash))
- p.queue.Forget(hash)
+ slog.ErrorContext(ctx, "Sandbox runtime to prewarm is not in the index", slog.String("runtime_hash", item.hash))
+ p.queue.Forget(item)
return
}
- err := p.prewarm(ctx, hash, rt)
+ err := p.prewarm(ctx, item, rt)
if p.onFinished != nil {
- p.onFinished(hash, err)
+ p.onFinished(item, err)
}
if err == nil {
- p.markPrewarmed(hash)
- p.queue.Forget(hash)
+ p.queue.Forget(item)
return
}
if ctx.Err() != nil {
// Shutting down, not a prewarm failure; drop without retry noise.
return
}
- if retries := p.queue.NumRequeues(hash); retries < prewarmMaxRetries {
+ if retries := p.queue.NumRequeues(item); retries < prewarmMaxRetries {
slog.WarnContext(ctx, "Sandbox runtime prewarm failed; will retry",
- slog.String("runtime_hash", hash), slog.String("config", rt.config), slog.String("version", rt.version),
+ slog.String("runtime_hash", item.hash), slog.String("config", rt.config), slog.String("version", rt.version),
slog.Int("retries", retries), slog.Any("err", err))
- p.queue.AddRateLimited(hash)
+ p.queue.AddRateLimited(item)
return
}
- // Best-effort: give up until a worker advertises it again or first use.
+ // Best-effort: give up until the worker is synced again or first use.
slog.WarnContext(ctx, "Sandbox runtime prewarm failed; giving up",
- slog.String("runtime_hash", hash), slog.String("config", rt.config), slog.String("version", rt.version),
+ slog.String("runtime_hash", item.hash), slog.String("config", rt.config), slog.String("version", rt.version),
slog.Int("retries", prewarmMaxRetries), slog.Any("err", err))
- p.queue.Forget(hash)
+ p.queue.Forget(item)
}
-// prewarm fetches the runtime's assets into the asset cache and its pause
-// image into the image cache. Racing an on-demand ensureSandboxAssets for
-// the same assets is safe: the asset cache runs one download per asset and
-// serves every caller from it, and the image cache collapses concurrent
-// pulls of one digest.
+// prewarm fetches the runtime's assets into the asset cache, linked into
+// the item's directory, and its pause image into the image cache, the two
+// side by side. Racing an on-demand ensureSandboxAssets for the same assets
+// is safe: the asset cache runs one download per asset and serves every
+// caller from it, and the image cache collapses concurrent pulls of one
+// digest. A directory the syncer already filled costs a stat per asset.
//
-// Prewarm has no actor to link the assets into, so it stages them into a
-// scratch directory it removes again; the cache keeps the entries.
-func (p *runtimePrewarmer) prewarm(ctx context.Context, hash string, rt *indexedRuntime) error {
+// Nothing here creates a directory: the staging goes into the item's
+// directory as it is, so a directory removed before or during the work, by
+// a later sync or the worker's removal, makes the next step fail, and the
+// item ends without an error and without recreating anything.
+func (p *runtimePrewarmer) prewarm(ctx context.Context, item prewarmItem, rt *indexedRuntime) error {
ctx, cancel := context.WithTimeout(ctx, p.attemptTimeout)
defer cancel()
t := time.Now()
+ if gone, err := dirGone(item.dir); err != nil || gone {
+ if gone {
+ slog.DebugContext(ctx, "Sandbox runtime prewarm superseded; its directory is gone", slog.String("runtime_hash", item.hash), slog.String("dir", item.dir))
+ }
+ return err
+ }
var assetErr, imageErr error
var wg sync.WaitGroup
wg.Go(func() {
- scratch, err := os.MkdirTemp(ateletpath.SandboxAssetScratchDir, "prewarm-")
- if err != nil {
- assetErr = fmt.Errorf("while creating prewarm scratch dir: %w", err)
- return
- }
- defer func() { _ = os.RemoveAll(scratch) }()
- _, assetErr = p.assets.ensureSandboxAssets(ctx, rt.rec, scratch)
+ _, assetErr = p.assets.stageSandboxAssets(ctx, rt.rec, item.dir)
})
if rt.rec.PauseImage != "" {
wg.Go(func() {
@@ -222,14 +218,31 @@ func (p *runtimePrewarmer) prewarm(ctx context.Context, hash string, rt *indexed
}
wg.Wait()
if err := errors.Join(assetErr, imageErr); err != nil {
+ if gone, gerr := dirGone(item.dir); gerr == nil && gone {
+ slog.DebugContext(ctx, "Sandbox runtime prewarm superseded while staging", slog.String("runtime_hash", item.hash), slog.String("dir", item.dir))
+ return nil
+ }
return err
}
slog.InfoContext(ctx, "Sandbox runtime prewarmed",
- slog.String("runtime_hash", hash),
+ slog.String("runtime_hash", item.hash),
slog.String("config", rt.config),
slog.String("version", rt.version),
slog.Int("assets", len(rt.rec.Assets)),
slog.String("pause_image", rt.rec.PauseImage),
+ slog.String("dir", item.dir),
slog.Duration("duration", time.Since(t)))
return nil
}
+
+// dirGone reports whether dir does not exist.
+func dirGone(dir string) (bool, error) {
+ _, err := os.Stat(dir)
+ if errors.Is(err, fs.ErrNotExist) {
+ return true, nil
+ }
+ if err != nil {
+ return false, fmt.Errorf("while checking the prewarm directory: %w", err)
+ }
+ return false, nil
+}
diff --git a/cmd/atelet/runtime_prewarm_test.go b/cmd/atelet/runtime_prewarm_test.go
index 509791eb23..4dc1d72482 100644
--- a/cmd/atelet/runtime_prewarm_test.go
+++ b/cmd/atelet/runtime_prewarm_test.go
@@ -21,7 +21,12 @@ import (
"fmt"
"io"
"os"
+ "path/filepath"
"runtime"
+ "slices"
+ "strings"
+ "sync"
+ "sync/atomic"
"testing"
"time"
@@ -36,6 +41,7 @@ import (
kubefake "k8s.io/client-go/kubernetes/fake"
"go.ate.dev/substrate/cmd/atelet/internal/ateletpath"
+ "go.ate.dev/substrate/cmd/atelet/internal/filecache"
"go.ate.dev/substrate/internal/imagecache"
"go.ate.dev/substrate/pkg/api/v1alpha1"
atefake "go.ate.dev/substrate/pkg/client/clientset/versioned/fake"
@@ -107,11 +113,17 @@ func pauseImageInTestRegistry(t *testing.T) (ref string, host string) {
// tarball content, and the given pause image as a worker's default, and
// returns its hash.
func indexedTestRuntime(t *testing.T, index *runtimeIndex, content []byte, pauseImage string) string {
+ t.Helper()
+ return indexedTestRuntimeAt(t, index, "gs://bucket/gvisor.tar", content, pauseImage)
+}
+
+// indexedTestRuntimeAt is indexedTestRuntime with the asset at url.
+func indexedTestRuntimeAt(t *testing.T, index *runtimeIndex, url string, content []byte, pauseImage string) string {
t.Helper()
rec := &sandboxAssetsRecord{
SandboxClass: "gvisor",
PauseImage: pauseImage,
- Assets: map[string]assetEntry{gvisorAssetName: {URL: "gs://bucket/gvisor.tar", SHA256: fmt.Sprintf("%x", sha256.Sum256(content))}},
+ Assets: map[string]assetEntry{gvisorAssetName: {URL: url, SHA256: fmt.Sprintf("%x", sha256.Sum256(content))}},
}
hash, err := versionBundleHash(rec)
if err != nil {
@@ -123,6 +135,18 @@ func indexedTestRuntime(t *testing.T, index *runtimeIndex, content []byte, pause
return hash
}
+// defaultItem makes the directory the syncer would make for hash as a
+// default of pod-a under root, so the links land on the cache's
+// filesystem, and returns the item the syncer would hand the prewarmer.
+func defaultItem(t *testing.T, root, hash string) prewarmItem {
+ t.Helper()
+ dir := filepath.Join(root, "workers", "pod-a", defaultRuntimeDirsName, strings.ReplaceAll(hash, ":", "-"))
+ if err := os.MkdirAll(dir, 0o700); err != nil {
+ t.Fatal(err)
+ }
+ return prewarmItem{hash: hash, dir: dir}
+}
+
func newTestImageStore(t *testing.T) *imagecache.Store {
t.Helper()
store, err := imagecache.New(t.TempDir())
@@ -134,78 +158,101 @@ func newTestImageStore(t *testing.T) *imagecache.Store {
// TestRuntimePrewarmProcess covers the worker's failure handling: a failed
// prewarm is requeued with backoff until prewarmMaxRetries and then dropped,
-// and a hash that is not in the index is dropped at once.
+// and an item naming a hash that is not in the index is dropped at once.
func TestRuntimePrewarmProcess(t *testing.T) {
- useTempAssetScratch(t)
ctx := context.Background()
index := newRuntimeIndex()
hash := indexedTestRuntime(t, index, gvisorReleaseTarball(t), "")
- herder := &Atelet{sandboxAssetCache: newTestAssetCache(t), anonGCSClient: fakeObjectStorage{err: errors.New("bucket unavailable")}}
+ root := t.TempDir()
+ herder := &Atelet{sandboxAssetCache: newTestAssetCacheAt(t, root), anonGCSClient: fakeObjectStorage{err: errors.New("bucket unavailable")}}
p := newRuntimePrewarmer(index, herder, newTestImageStore(t))
+ item := defaultItem(t, root, hash)
for i := 1; i <= prewarmMaxRetries; i++ {
- p.process(ctx, hash)
- if got := p.queue.NumRequeues(hash); got != i {
+ p.process(ctx, item)
+ if got := p.queue.NumRequeues(item); got != i {
t.Fatalf("NumRequeues after %d failed prewarms = %d, want %d (requeued with backoff)", i, got, i)
}
}
- p.process(ctx, hash)
- if got := p.queue.NumRequeues(hash); got != 0 {
+ p.process(ctx, item)
+ if got := p.queue.NumRequeues(item); got != 0 {
t.Errorf("NumRequeues after giving up = %d, want 0 (forgotten)", got)
}
- p.process(ctx, "sha256:not-in-the-index")
- if got := p.queue.NumRequeues("sha256:not-in-the-index"); got != 0 {
+ unknown := prewarmItem{hash: "sha256:not-in-the-index", dir: item.dir}
+ p.process(ctx, unknown)
+ if got := p.queue.NumRequeues(unknown); got != 0 {
t.Errorf("NumRequeues for an unknown hash = %d, want 0 (forgotten)", got)
}
}
-// TestRuntimePrewarmOnce pins that a runtime is prewarmed once: a hash
-// whose prewarm succeeded is skipped by every later enqueue, while one whose
-// prewarm failed is queued again.
-func TestRuntimePrewarmOnce(t *testing.T) {
- useTempAssetScratch(t)
+// TestRuntimePrewarmSuperseded pins what a prewarm does once the syncer
+// removed its directory, after a later sync or the worker's removal: no
+// fetch, nothing recreated, and no retry, whether the directory went
+// before the item ran or while it staged.
+func TestRuntimePrewarmSuperseded(t *testing.T) {
ctx := context.Background()
index := newRuntimeIndex()
- release := gvisorReleaseTarball(t)
- good := indexedTestRuntime(t, index, release, "")
- // The store serves the same bytes for every asset, so this one's digest
- // never matches and its prewarm fails.
- bad := indexedTestRuntime(t, index, gvisorReleaseTarballWith(t, "another runsc"), "")
- herder := &Atelet{sandboxAssetCache: newTestAssetCache(t), anonGCSClient: fakeObjectStorage{data: release}}
- p := newRuntimePrewarmer(index, herder, newTestImageStore(t))
+ content := gvisorReleaseTarball(t)
+ hash := indexedTestRuntime(t, index, content, "")
+ root := t.TempDir()
+ herder := &Atelet{sandboxAssetCache: newTestAssetCacheAt(t, root), anonGCSClient: fakeObjectStorage{data: content}}
- drain := func() {
- for p.queue.Len() > 0 {
- hash, _ := p.queue.Get()
- p.process(ctx, hash)
+ t.Run("directory gone before the item runs", func(t *testing.T) {
+ item := defaultItem(t, root, hash)
+ if err := os.RemoveAll(item.dir); err != nil {
+ t.Fatal(err)
}
- }
- p.enqueue(good)
- p.enqueue(good) // a second pod of the pool: collapsed while it waits
- if got := p.queue.Len(); got != 1 {
- t.Fatalf("queue holds %d items after two enqueues of one hash, want 1", got)
- }
- drain()
- p.enqueue(good) // a later pod, re-plan or retry: already prewarmed
- if got := p.queue.Len(); got != 0 {
- t.Errorf("queue holds %d items after enqueueing a prewarmed hash, want 0", got)
- }
+ p := newRuntimePrewarmer(index, herder, newTestImageStore(t))
+ p.process(ctx, item)
+ if got := p.queue.NumRequeues(item); got != 0 {
+ t.Errorf("NumRequeues after a superseded prewarm = %d, want 0 (not retried)", got)
+ }
+ if n := cachedAssets(t, root); n != 0 {
+ t.Errorf("a superseded prewarm fetched %d assets, want 0", n)
+ }
+ if exists(item.dir) {
+ t.Error("a superseded prewarm recreated its directory")
+ }
+ })
- p.enqueue(bad)
- drain()
- p.queue.Forget(bad) // drop the retry backoff for the test
- p.enqueue(bad)
- if got := p.queue.Len(); got != 1 {
- t.Errorf("queue holds %d items after enqueueing a hash whose prewarm failed, want 1 (not skipped)", got)
+ t.Run("worker gone while staging", func(t *testing.T) {
+ item := defaultItem(t, root, hash)
+ removing := &removingFetcher{inner: herder, remove: filepath.Join(root, "workers", "pod-a")}
+ p := newRuntimePrewarmer(index, removing, newTestImageStore(t))
+ p.process(ctx, item)
+ if got := p.queue.NumRequeues(item); got != 0 {
+ t.Errorf("NumRequeues after the worker went mid-staging = %d, want 0 (not retried)", got)
+ }
+ if exists(filepath.Join(root, "workers", "pod-a")) {
+ t.Error("a prewarm recreated the directory of a worker removed while it staged")
+ }
+ })
+}
+
+// removingFetcher removes a directory the moment staging starts, standing in
+// for a worker deleted after the prewarm found its directory in place.
+type removingFetcher struct {
+ inner sandboxAssetFetcher
+ remove string
+}
+
+func (r *removingFetcher) ensureSandboxAssets(ctx context.Context, rec *sandboxAssetsRecord, dstDir string) (map[string]string, error) {
+ return r.inner.ensureSandboxAssets(ctx, rec, dstDir)
+}
+
+func (r *removingFetcher) stageSandboxAssets(ctx context.Context, rec *sandboxAssetsRecord, dstDir string) (map[string]string, error) {
+ if err := os.RemoveAll(r.remove); err != nil {
+ return nil, err
}
+ return r.inner.stageSandboxAssets(ctx, rec, dstDir)
}
// TestRuntimePrewarmFillsBothCaches covers a successful prewarm: the asset
-// lands in the asset cache with nothing left in the scratch dir, and the
-// pause image lands in the image cache.
+// lands in the asset cache, linked into the runtime's directory so an
+// eviction pass keeps it, and the pause image lands in the image cache. A
+// second prewarm of the same directory fetches nothing.
func TestRuntimePrewarmFillsBothCaches(t *testing.T) {
- useTempAssetScratch(t)
ctx := context.Background()
pauseRef, _ := pauseImageInTestRegistry(t)
content := gvisorReleaseTarball(t)
@@ -215,25 +262,54 @@ func TestRuntimePrewarmFillsBothCaches(t *testing.T) {
root := t.TempDir()
images := newTestImageStore(t)
- p := newRuntimePrewarmer(index, &Atelet{sandboxAssetCache: newTestAssetCacheAt(t, root), anonGCSClient: fakeObjectStorage{data: content}}, images)
- if err := p.prewarm(ctx, hash, rt); err != nil {
+ var gets atomic.Int32
+ // No min age, so the only thing standing between the entry and a pass is
+ // the directory's link.
+ s := &Atelet{sandboxAssetCache: newTestAssetCacheAt(t, root, filecache.WithMinAge(0)), anonGCSClient: fakeObjectStorage{data: content, gets: &gets}}
+ p := newRuntimePrewarmer(index, s, images)
+ item := defaultItem(t, root, hash)
+ if err := p.prewarm(ctx, item, rt); err != nil {
t.Fatalf("prewarm: %v", err)
}
if n := cachedAssets(t, root); n != 1 {
t.Errorf("asset cache holds %d assets after prewarm, want 1", n)
}
- if left, err := os.ReadDir(ateletpath.SandboxAssetScratchDir); err != nil || len(left) != 0 {
- t.Errorf("prewarm left %d scratch dirs behind (%v)", len(left), err)
+ if _, err := os.Stat(filepath.Join(item.dir, gvisorAssetName, "runsc")); err != nil {
+ t.Errorf("prewarm did not link the runtime's asset into its directory: %v", err)
}
if _, err := images.EnsureImage(ctx, pauseRef); err != nil {
t.Errorf("pause image not prewarmed into the image cache: %v", err)
}
+ stats, err := s.sandboxAssetCache.EvictUnused(ctx, 1<<40, false)
+ if err != nil {
+ t.Fatalf("EvictUnused: %v", err)
+ }
+ if stats.Retired != 0 || stats.SkippedLinked != 1 || cachedAssets(t, root) != 1 {
+ t.Errorf("after an eviction pass: stats=%+v, cached=%d; want the linked asset kept", stats, cachedAssets(t, root))
+ }
+
+ // A re-sync with the same default hands the same directory over again:
+ // the staging finds every asset in place and downloads nothing.
+ before := gets.Load()
+ if err := p.prewarm(ctx, item, rt); err != nil {
+ t.Fatalf("second prewarm: %v", err)
+ }
+ if got := gets.Load(); got != before {
+ t.Errorf("a prewarm of a complete directory fetched %d times, want 0", got-before)
+ }
+
+ // The worker goes, and its directory with it: the entry is evictable.
+ if err := os.RemoveAll(filepath.Join(root, "workers", "pod-a")); err != nil {
+ t.Fatal(err)
+ }
+ if stats, _ := s.sandboxAssetCache.EvictUnused(ctx, 1<<40, true); stats.Retired != 1 {
+ t.Errorf("after the worker went, a pass would retire %d entries, want 1", stats.Retired)
+ }
}
// TestRuntimePrewarmAssetFailureStillPullsImage pins that the two halves
// are independent: a failing asset fetch does not stop the pause image pull.
func TestRuntimePrewarmAssetFailureStillPullsImage(t *testing.T) {
- useTempAssetScratch(t)
ctx := context.Background()
pauseRef, _ := pauseImageInTestRegistry(t)
index := newRuntimeIndex()
@@ -241,8 +317,9 @@ func TestRuntimePrewarmAssetFailureStillPullsImage(t *testing.T) {
rt, _ := index.lookup(hash)
images := newTestImageStore(t)
- p := newRuntimePrewarmer(index, &Atelet{sandboxAssetCache: newTestAssetCache(t), anonGCSClient: fakeObjectStorage{err: errors.New("bucket unavailable")}}, images)
- if err := p.prewarm(ctx, hash, rt); err == nil {
+ root := t.TempDir()
+ p := newRuntimePrewarmer(index, &Atelet{sandboxAssetCache: newTestAssetCacheAt(t, root), anonGCSClient: fakeObjectStorage{err: errors.New("bucket unavailable")}}, images)
+ if err := p.prewarm(ctx, defaultItem(t, root, hash), rt); err == nil {
t.Error("prewarm returned nil despite the asset fetch failing")
}
if _, err := images.EnsureImage(ctx, pauseRef); err != nil {
@@ -254,15 +331,16 @@ func TestRuntimePrewarmAssetFailureStillPullsImage(t *testing.T) {
// the prewarmer's attempt timeout: the queue has one worker, so an attempt
// that never returned would block every other runtime's prewarm.
func TestRuntimePrewarmAttemptTimeout(t *testing.T) {
- useTempAssetScratch(t)
index := newRuntimeIndex()
hash := indexedTestRuntime(t, index, gvisorReleaseTarball(t), "")
rt, _ := index.lookup(hash)
- p := newRuntimePrewarmer(index, &Atelet{sandboxAssetCache: newTestAssetCache(t), anonGCSClient: hangingObjectStorage{}}, newTestImageStore(t))
+ root := t.TempDir()
+ p := newRuntimePrewarmer(index, &Atelet{sandboxAssetCache: newTestAssetCacheAt(t, root), anonGCSClient: hangingObjectStorage{}}, newTestImageStore(t))
p.attemptTimeout = 50 * time.Millisecond
+ item := defaultItem(t, root, hash)
done := make(chan error, 1)
- go func() { done <- p.prewarm(context.Background(), hash, rt) }()
+ go func() { done <- p.prewarm(context.Background(), item, rt) }()
select {
case err := <-done:
if err == nil {
@@ -273,23 +351,33 @@ func TestRuntimePrewarmAttemptTimeout(t *testing.T) {
}
}
-// TestSyncPodPrewarmsDefaultRuntime pins the hand-off: a worker's default
-// runtime, and only that, is handed to the prewarmer, and before the build
-// fetches anything for its probes, so the probes are the prewarm's head
-// start. A build that then fails has still handed it over.
+// TestSyncPodPrewarmsDefaultRuntime pins the hand-off: once a worker's file
+// is written, its default runtime, and only that, is handed to the
+// prewarmer in a directory of its own under the pod's, and the pod's
+// default-runtime directory is made to match the file: a retired default's
+// directory goes, the current one stays as it is. A build that fails hands
+// nothing over and touches no directory, and the pod's removal takes them
+// all.
func TestSyncPodPrewarmsDefaultRuntime(t *testing.T) {
pod := workerPod("pod-a", "pool")
s := newTestSyncer(t, pod, resolvedPool("pool", "gvisor-release"), multiVersionConfig())
- var prewarmed []string
- s.prewarm = func(hash string) { prewarmed = append(prewarmed, hash) }
+ rec := &recordingPrewarm{}
+ s.prewarm = rec
s.assets = &fakeAssetFetcher{
- fn: func(ctx context.Context, rec *sandboxAssetsRecord) (map[string]string, error) {
- if len(prewarmed) == 0 {
- t.Error("the build fetched for a probe before the default runtime was handed to the prewarmer")
+ fn: func(ctx context.Context, r *sandboxAssetsRecord) (map[string]string, error) {
+ if rec.count() != 0 {
+ t.Error("the default runtime was handed to the prewarmer before the build finished")
}
- return newTestAssetFetcher().ensureSandboxAssets(ctx, rec, "")
+ return newTestAssetFetcher().ensureSandboxAssets(ctx, r, "")
},
}
+ dirs := filepath.Join(s.podDir("pod-a"), defaultRuntimeDirsName)
+ // Left by an earlier sync: a default the pool no longer runs, which must
+ // go, and a stray file in the parent, which must go too.
+ stale := filepath.Join(dirs, "sha256-stale")
+ if err := os.MkdirAll(stale, 0o700); err != nil {
+ t.Fatal(err)
+ }
if err := s.syncPod(context.Background(), pod); err != nil {
t.Fatalf("syncPod: %v", err)
}
@@ -298,42 +386,104 @@ func TestSyncPodPrewarmsDefaultRuntime(t *testing.T) {
t.Fatal("pod-a not indexed")
}
cr, _ := wr.class(v1alpha1.SandboxClassGvisor)
- if len(prewarmed) != 1 || prewarmed[0] != cr.defaultHash {
- t.Errorf("prewarmed %v, want the default runtime %s once", prewarmed, cr.defaultHash)
+ want := []prewarmItem{{hash: cr.defaultHash, dir: s.defaultRuntimeDir("pod-a", cr.defaultHash)}}
+ if got := rec.items(); !slices.Equal(got, want) {
+ t.Errorf("prewarm handed %+v, want %+v", got, want)
+ }
+ if !exists(want[0].dir) {
+ t.Error("the default's directory was not created")
+ }
+ if exists(stale) {
+ t.Error("a retired default's directory survived the sync")
}
if len(cr.hashes) < 2 {
t.Fatalf("config advertises %d runtimes, want several so the test proves only the default is prewarmed", len(cr.hashes))
}
- // A build that fails on the default's probe writes nothing, but the
- // prewarm was queued before it started.
+ // A re-sync with the same default: the directory, and what the
+ // prewarmer put in it, stays; the default is handed over again.
+ marker := filepath.Join(want[0].dir, "staged")
+ if err := os.WriteFile(marker, nil, 0o600); err != nil {
+ t.Fatal(err)
+ }
+ if err := s.syncPod(context.Background(), pod); err != nil {
+ t.Fatalf("syncPod again: %v", err)
+ }
+ if got := rec.items(); len(got) != 2 || got[1] != want[0] {
+ t.Errorf("prewarm handed %+v after a re-sync, want the same item twice", got)
+ }
+ if !exists(marker) {
+ t.Error("a re-sync with the same default emptied its directory")
+ }
+ if err := s.removeDir(context.Background(), "pod-a"); err != nil {
+ t.Fatal(err)
+ }
+ if exists(dirs) {
+ t.Error("the default runtime directories survived the pod's removal")
+ }
+
+ // A build that fails on the default's probe writes nothing, hands
+ // nothing over, and leaves the directories as they were.
s = newTestSyncer(t, pod, resolvedPool("pool", "gvisor-release"), multiVersionConfig())
- prewarmed = nil
- s.prewarm = func(hash string) { prewarmed = append(prewarmed, hash) }
+ rec = &recordingPrewarm{}
+ s.prewarm = rec
s.runCPUFeatures = func(context.Context, string) ([]string, error) { return nil, errors.New("runsc failed") }
+ previous := s.defaultRuntimeDir("pod-a", "sha256:previous")
+ if err := os.MkdirAll(previous, 0o700); err != nil {
+ t.Fatal(err)
+ }
if err := s.syncPod(context.Background(), pod); err == nil {
t.Fatal("syncPod with the default's probe failing = nil, want an error")
}
- if len(prewarmed) != 1 || prewarmed[0] != cr.defaultHash {
- t.Errorf("prewarmed %v after a failed build, want the default runtime %s once", prewarmed, cr.defaultHash)
+ if got := rec.items(); len(got) != 0 {
+ t.Errorf("prewarm handed %+v after a failed build, want nothing", got)
+ }
+ if !exists(previous) {
+ t.Error("a failed build removed the previous default's directory")
}
if _, ok := s.index.worker("pod-a"); ok {
t.Error("a failed build registered the worker")
}
}
+// recordingPrewarm stands in for the prewarmer in syncer tests. Safe to
+// read while a syncer's workers call it.
+type recordingPrewarm struct {
+ mu sync.Mutex
+ enqueued []prewarmItem
+}
+
+func (r *recordingPrewarm) enqueue(hash, dir string) {
+ r.mu.Lock()
+ defer r.mu.Unlock()
+ r.enqueued = append(r.enqueued, prewarmItem{hash: hash, dir: dir})
+}
+
+func (r *recordingPrewarm) items() []prewarmItem {
+ r.mu.Lock()
+ defer r.mu.Unlock()
+ return append([]prewarmItem(nil), r.enqueued...)
+}
+
+func (r *recordingPrewarm) count() int {
+ r.mu.Lock()
+ defer r.mu.Unlock()
+ return len(r.enqueued)
+}
+
// TestDefaultRuntimePrewarmThroughSyncer runs the whole path: a worker pod,
// its pool and SandboxConfig in fake clientsets flow through the informers
// into the syncer, which indexes the pod's runtimes, writes its file, and
// hands its default runtime to the prewarmer, which lands the asset in the
-// cache without any Run/Restore request.
+// cache, linked under the worker's directory, without any Run/Restore
+// request.
func TestDefaultRuntimePrewarmThroughSyncer(t *testing.T) {
- useTempAssetScratch(t)
+ useTempAssetScratch(t) // the syncer's runsc probe stages into it
// The worker's own prewarm call returns only when the asset has landed
// and the pause-image pull has finished. Waiting for the call keeps the
// pull's last writes from racing the TempDir cleanup.
prewarmed := make(chan error, 1)
- onFinished := func(_ string, err error) {
+ onFinished := func(_ prewarmItem, err error) {
// Non-blocking: a failed attempt is retried, and the worker must not
// block on a result nobody reads.
select {
@@ -354,6 +504,7 @@ func TestDefaultRuntimePrewarmThroughSyncer(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
root := t.TempDir()
+ workersDir := filepath.Join(root, "workers") // on the cache's filesystem
index := newRuntimeIndex()
herder := &Atelet{sandboxAssetCache: newTestAssetCacheAt(t, root), anonGCSClient: fakeObjectStorage{data: content}}
p := newRuntimePrewarmer(index, herder, newTestImageStore(t))
@@ -372,7 +523,7 @@ func TestDefaultRuntimePrewarmThroughSyncer(t *testing.T) {
ateFactory.Api().V1alpha1().WorkerPools(),
ateFactory.Api().V1alpha1().SandboxConfigs(),
newScriptAssetFetcher(t), // the syncer's runsc probe; prewarm goes through herder
- t.TempDir(), index, p.enqueue)
+ workersDir, index, p)
if err != nil {
t.Fatalf("startWorkerSandboxSyncer: %v", err)
}
@@ -390,11 +541,16 @@ func TestDefaultRuntimePrewarmThroughSyncer(t *testing.T) {
if n := cachedAssets(t, root); n != 1 {
t.Fatalf("asset cache holds %d assets after prewarm, want 1", n)
}
- // The prewarm is queued before the build probes, so it can finish
- // before the file is written; the syncer catches up on its own.
waitFor(t, "the worker's runtimes file to be written", func() bool { return exists(s.runtimesFile("pod-a")) })
- waitFor(t, "the prewarm's and the probe's scratch dirs to be removed", func() bool {
+ waitFor(t, "the probe's scratch dir to be removed", func() bool {
left, err := os.ReadDir(ateletpath.SandboxAssetScratchDir)
return err == nil && len(left) == 0
})
+ wr, _ := index.worker("pod-a")
+ cr, _ := wr.class(v1alpha1.SandboxClassGvisor)
+ linked := filepath.Join(s.defaultRuntimeDir("pod-a", cr.defaultHash), gvisorAssetName, "runsc")
+ waitFor(t, "the worker's default runtime to be linked under its directory", func() bool { return exists(linked) })
+ if dirs, err := os.ReadDir(filepath.Join(s.podDir("pod-a"), defaultRuntimeDirsName)); err != nil || len(dirs) != 1 {
+ t.Errorf("default runtime directories under the worker's = %v, %v; want exactly one", dirs, err)
+ }
}
diff --git a/cmd/atelet/sandbox_asset_cache.go b/cmd/atelet/sandbox_asset_cache.go
index dccb19363a..78e40b458e 100644
--- a/cmd/atelet/sandbox_asset_cache.go
+++ b/cmd/atelet/sandbox_asset_cache.go
@@ -28,17 +28,15 @@ import (
"go.ate.dev/substrate/cmd/atelet/internal/ateletpath"
"go.ate.dev/substrate/cmd/atelet/internal/filecache"
- "go.ate.dev/substrate/internal/nodepath"
+ "go.ate.dev/substrate/cmd/atelet/internal/filecachegc"
)
var (
- sandboxAssetCacheDir = pflag.String("sandbox-asset-cache-dir", ateletpath.SandboxAssetCacheDir, "Directory for the node-local sandbox asset cache (gVisor releases, micro-VM kernels, VMMs and guest images). Must be on the same filesystem mount as the actor state dirs: an actor's assets are hard links from the cache into its own directory.")
- // TODO: no eviction pass runs on the asset cache yet, so this value is
- // handed to the store and not consulted; set or unset, the cache behaves
- // the same. The flag is declared ahead of the GC pass, which the next
- // change adds and which will read it, so the operator surface does not
- // change when it lands.
- sandboxAssetCacheMinAge = pflag.Duration("sandbox-asset-cache-min-age", time.Hour, "Cached sandbox assets younger than this are never evicted. No eviction pass runs on this cache yet, so the value is recorded for when one does. A running actor's assets are protected by its links regardless; this covers an asset prewarmed for a SandboxConfig that no actor on the node has used yet.")
+ sandboxAssetCacheDir = pflag.String("sandbox-asset-cache-dir", ateletpath.SandboxAssetCacheDir, "Directory for the node-local sandbox asset cache (gVisor releases, micro-VM kernels, VMMs and guest images). Must be on the same filesystem mount as the actor state dirs and the worker dirs: an actor's assets, and a worker's default runtime, are hard links from the cache into their own directories.")
+ sandboxAssetCacheMinAge = pflag.Duration("sandbox-asset-cache-min-age", time.Hour, "Cached sandbox assets younger than this are never evicted. A running actor's assets, and the default runtime of every local worker, are protected by their links regardless; this covers an asset between its fetch and its first link.")
+ // sandboxAssetCacheGCFlags are --sandbox-asset-cache-gc-period,
+ // --sandbox-asset-cache-max-bytes and --sandbox-asset-cache-gc-dry-run.
+ sandboxAssetCacheGCFlags = filecachegc.NewFlags(pflag.CommandLine, "sandbox-asset-cache", "sandbox asset cache")
sandboxAssetFetchTimeout = pflag.Duration("sandbox-asset-fetch-timeout", 5*time.Minute, "Bound on one sandbox asset download (and, for a gVisor release, its extraction). Downloads run detached from the requests waiting on them, so this is the only bound on how long one can run; a Run arriving while a download hangs waits on it until then. Sized for multi-hundred-MiB micro-VM guest images on a busy node.")
)
@@ -56,15 +54,18 @@ func validateSandboxAssetCacheFlags() error {
// openSandboxAssetCache opens the asset cache at dir, sweeps the debris a
// crash may have left, and empties the scratch dir asset fetches without
-// an actor link into. It refuses a dir from which the actor dirs cannot be
-// hard-linked: every hit is such a link, so none could be served.
-func openSandboxAssetCache(ctx context.Context, dir string, minAge, fetchTimeout time.Duration) (*filecache.Store, error) {
+// an actor link into. It refuses a dir from which any of linkDirs cannot be
+// hard-linked: the actors dir, where every hit is such a link, and the
+// workers dir, where each worker's default runtime is kept by them.
+func openSandboxAssetCache(ctx context.Context, dir string, minAge, fetchTimeout time.Duration, linkDirs ...string) (*filecache.Store, error) {
store, err := filecache.New(dir, filecache.WithMinAge(minAge), filecache.WithFetchTimeout(fetchTimeout))
if err != nil {
return nil, fmt.Errorf("while opening sandbox asset cache at %s: %w", dir, err)
}
- if err := canHardLink(dir, nodepath.ActorsDir); err != nil {
- return nil, fmt.Errorf("sandbox asset cache: %w", err)
+ for _, linkDir := range linkDirs {
+ if err := canHardLink(dir, linkDir); err != nil {
+ return nil, fmt.Errorf("sandbox asset cache: %w", err)
+ }
}
stats, err := store.SweepDebris(ctx)
if err != nil {
@@ -85,14 +86,14 @@ func openSandboxAssetCache(ctx context.Context, dir string, minAge, fetchTimeout
}
// canHardLink fails unless a file under cacheDir can be hard-linked into
-// actorsDir, which every cache hit relies on. It links a probe file and
-// removes it again. Nothing short of the probe is exact: Linux refuses a
-// link across two mounts even when they are bind mounts of one filesystem,
-// so equal device numbers do not settle it, and a path check cannot see a
-// mount at all. actorsDir is created if it does not exist yet.
-func canHardLink(cacheDir, actorsDir string) error {
- if err := os.MkdirAll(actorsDir, 0o700); err != nil {
- return fmt.Errorf("while creating %s: %w", actorsDir, err)
+// linkDir, which every cache hit into it relies on. It links a probe file
+// and removes it again. Nothing short of the probe is exact: Linux refuses
+// a link across two mounts even when they are bind mounts of one
+// filesystem, so equal device numbers do not settle it, and a path check
+// cannot see a mount at all. linkDir is created if it does not exist yet.
+func canHardLink(cacheDir, linkDir string) error {
+ if err := os.MkdirAll(linkDir, 0o700); err != nil {
+ return fmt.Errorf("while creating %s: %w", linkDir, err)
}
probe, err := os.CreateTemp(cacheDir, ".linkprobe-")
if err != nil {
@@ -102,12 +103,12 @@ func canHardLink(cacheDir, actorsDir string) error {
if err := probe.Close(); err != nil {
return fmt.Errorf("while closing the link probe: %w", err)
}
- link := filepath.Join(actorsDir, filepath.Base(probe.Name()))
+ link := filepath.Join(linkDir, filepath.Base(probe.Name()))
if err := os.Link(probe.Name(), link); err != nil {
if errors.Is(err, syscall.EXDEV) {
- return fmt.Errorf("%s and %s are on different mounts, so assets cannot be hard-linked into actor dirs; put the cache on the mount that holds %s: %w", cacheDir, actorsDir, actorsDir, err)
+ return fmt.Errorf("%s and %s are on different mounts, so assets cannot be hard-linked under %s; put the cache on the mount that holds it: %w", cacheDir, linkDir, linkDir, err)
}
- return fmt.Errorf("while probing a hard link from %s into %s: %w", cacheDir, actorsDir, err)
+ return fmt.Errorf("while probing a hard link from %s into %s: %w", cacheDir, linkDir, err)
}
return os.Remove(link)
}
diff --git a/cmd/atelet/sandbox_asset_cache_linux_test.go b/cmd/atelet/sandbox_asset_cache_linux_test.go
index 96501eac5c..6867d81b19 100644
--- a/cmd/atelet/sandbox_asset_cache_linux_test.go
+++ b/cmd/atelet/sandbox_asset_cache_linux_test.go
@@ -23,8 +23,6 @@ import (
"syscall"
"testing"
"time"
-
- "go.ate.dev/substrate/internal/nodepath"
)
// otherMount returns a fresh directory on a mount other than the one
@@ -63,13 +61,20 @@ func TestCanHardLinkRejectsAnotherMount(t *testing.T) {
}
}
+// TestOpenSandboxAssetCacheRejectsAnotherMount pins the startup refusal for
+// each directory links go into: the actors dir, and the workers dir where
+// each worker's default runtime is linked.
func TestOpenSandboxAssetCacheRejectsAnotherMount(t *testing.T) {
useTempAssetScratch(t)
- orig := nodepath.ActorsDir
- nodepath.ActorsDir = filepath.Join(otherMount(t), "actors")
- t.Cleanup(func() { nodepath.ActorsDir = orig })
- _, err := openSandboxAssetCache(t.Context(), filepath.Join(t.TempDir(), "cache"), time.Hour, time.Minute)
- if !errors.Is(err, syscall.EXDEV) {
- t.Fatalf("openSandboxAssetCache with the actors dir on another mount = %v, want EXDEV", err)
+ for _, tt := range []struct{ name, actors, workers string }{
+ {"actors dir", filepath.Join(otherMount(t), "actors"), filepath.Join(t.TempDir(), "workers")},
+ {"workers dir", filepath.Join(t.TempDir(), "actors"), filepath.Join(otherMount(t), "workers")},
+ } {
+ t.Run(tt.name, func(t *testing.T) {
+ _, err := openSandboxAssetCache(t.Context(), filepath.Join(t.TempDir(), "cache"), time.Hour, time.Minute, tt.actors, tt.workers)
+ if !errors.Is(err, syscall.EXDEV) {
+ t.Fatalf("openSandboxAssetCache with the %s on another mount = %v, want EXDEV", tt.name, err)
+ }
+ })
}
}
diff --git a/cmd/atelet/sandbox_asset_cache_test.go b/cmd/atelet/sandbox_asset_cache_test.go
index 19871a620c..625e9985f2 100644
--- a/cmd/atelet/sandbox_asset_cache_test.go
+++ b/cmd/atelet/sandbox_asset_cache_test.go
@@ -21,6 +21,7 @@ import (
"time"
"go.ate.dev/substrate/cmd/atelet/internal/ateletpath"
+ "go.ate.dev/substrate/internal/nodepath"
)
func TestValidateSandboxAssetCacheFlags(t *testing.T) {
@@ -62,7 +63,7 @@ func TestOpenSandboxAssetCacheResetsScratch(t *testing.T) {
}
dir := filepath.Join(t.TempDir(), "cache")
- store, err := openSandboxAssetCache(t.Context(), dir, time.Hour, 5*time.Minute)
+ store, err := openSandboxAssetCache(t.Context(), dir, time.Hour, 5*time.Minute, nodepath.ActorsDir, filepath.Join(t.TempDir(), "workers"))
if err != nil {
t.Fatalf("openSandboxAssetCache: %v", err)
}
diff --git a/cmd/atelet/sandbox_assets.go b/cmd/atelet/sandbox_assets.go
index 16b70b5f72..c5043cbecc 100644
--- a/cmd/atelet/sandbox_assets.go
+++ b/cmd/atelet/sandbox_assets.go
@@ -33,7 +33,6 @@ import (
"net/url"
"os"
"path/filepath"
- "runtime"
"slices"
"strings"
"time"
@@ -42,7 +41,6 @@ import (
"go.ate.dev/substrate/cmd/atelet/internal/atomicfile"
"go.ate.dev/substrate/cmd/atelet/internal/filecache"
"go.ate.dev/substrate/internal/apierror"
- "go.ate.dev/substrate/internal/proto/ateletpb"
"go.ate.dev/substrate/internal/resources"
"go.ate.dev/substrate/pkg/api/v1alpha1"
"go.ate.dev/substrate/pkg/objectstorage"
@@ -116,31 +114,6 @@ type sandboxAssetsRecord struct {
Fidelity string `json:"fidelity,omitempty"`
}
-// recordFromRequest projects a request's per-architecture SandboxAssets onto the
-// local node's architecture.
-func recordFromRequest(sa *ateletpb.SandboxAssets) (*sandboxAssetsRecord, error) {
- if sa == nil {
- return nil, fmt.Errorf("missing sandbox_assets")
- }
- arch := runtime.GOARCH
- archAssets := sa.GetAssets()[arch]
- if archAssets == nil || len(archAssets.GetFiles()) == 0 {
- return nil, fmt.Errorf("sandbox_assets has no assets for architecture %q", arch)
- }
- if err := checkPauseImage(sa.GetSandboxClass(), sa.GetPauseImage()); err != nil {
- return nil, fmt.Errorf("invalid sandbox_assets: %w", err)
- }
- rec := &sandboxAssetsRecord{
- SandboxClass: sa.GetSandboxClass(),
- PauseImage: sa.GetPauseImage(),
- Assets: make(map[string]assetEntry, len(archAssets.GetFiles())),
- }
- for name, f := range archAssets.GetFiles() {
- rec.Assets[name] = assetEntry{URL: f.GetUrl(), SHA256: f.GetSha256()}
- }
- return rec, nil
-}
-
// checkPauseImage requires a pause image for gVisor, the only sandbox class
// that runs a pause container.
func checkPauseImage(sandboxClass, pauseImage string) error {
@@ -168,6 +141,14 @@ func (s *Atelet) ensureSandboxAssets(ctx context.Context, rec *sandboxAssetsReco
if err := os.MkdirAll(dstDir, 0o700); err != nil {
return nil, wrapFileSystemErr("while creating sandbox assets dir", err)
}
+ return s.stageSandboxAssets(ctx, rec, dstDir)
+}
+
+// stageSandboxAssets is ensureSandboxAssets into a dstDir that exists: it
+// creates nothing above the assets, so a caller whose dstDir may be removed
+// under it, as the prewarmer's is when its worker goes, fails instead of
+// recreating the directory.
+func (s *Atelet) stageSandboxAssets(ctx context.Context, rec *sandboxAssetsRecord, dstDir string) (map[string]string, error) {
paths := make(map[string]string, len(rec.Assets))
for name, entry := range rec.Assets {
if !plainAssetName(name) {
diff --git a/cmd/atelet/sandbox_assets_test.go b/cmd/atelet/sandbox_assets_test.go
index f7b7d3a23a..64e44ac5f8 100644
--- a/cmd/atelet/sandbox_assets_test.go
+++ b/cmd/atelet/sandbox_assets_test.go
@@ -26,19 +26,16 @@ import (
"io/fs"
"os"
"path/filepath"
- "runtime"
"strconv"
"strings"
"sync"
"sync/atomic"
"testing"
- "github.com/google/go-cmp/cmp"
"github.com/klauspost/compress/zstd"
"go.ate.dev/substrate/cmd/atelet/internal/ateletpath"
"go.ate.dev/substrate/cmd/atelet/internal/filecache"
"go.ate.dev/substrate/internal/apierror"
- "go.ate.dev/substrate/internal/proto/ateletpb"
"google.golang.org/grpc/codes"
)
@@ -186,76 +183,12 @@ func (n nopWriteCloser) Close() error {
return nil
}
-func TestRecordFromRequest(t *testing.T) {
- const sha = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
- valid := func() *ateletpb.SandboxAssets {
- return &ateletpb.SandboxAssets{
- SandboxClass: "gvisor",
- PauseImage: testPauseImage,
- Assets: map[string]*ateletpb.ArchAssets{
- runtime.GOARCH: {Files: map[string]*ateletpb.AssetFile{
- gvisorAssetName: {Url: "gs://bucket/gvisor.tar.zst", Sha256: sha},
- }},
- },
- }
- }
-
- t.Run("valid request projects onto this architecture", func(t *testing.T) {
- got, err := recordFromRequest(valid())
- if err != nil {
- t.Fatalf("recordFromRequest: %v", err)
- }
- want := &sandboxAssetsRecord{
- SandboxClass: "gvisor",
- PauseImage: testPauseImage,
- Assets: map[string]assetEntry{gvisorAssetName: {URL: "gs://bucket/gvisor.tar.zst", SHA256: sha}},
- }
- if diff := cmp.Diff(want, got); diff != "" {
- t.Errorf("recordFromRequest mismatch (-want +got):\n%s", diff)
- }
- })
-
- t.Run("microvm needs no pause image", func(t *testing.T) {
- sa := valid()
- sa.SandboxClass = "microvm"
- sa.PauseImage = ""
- got, err := recordFromRequest(sa)
- if err != nil {
- t.Fatalf("recordFromRequest: %v", err)
- }
- if got.PauseImage != "" {
- t.Errorf("PauseImage = %q, want empty", got.PauseImage)
- }
- })
-
- for _, tc := range []struct {
- name string
- mutate func(*ateletpb.SandboxAssets) *ateletpb.SandboxAssets
- }{
- {"missing", func(*ateletpb.SandboxAssets) *ateletpb.SandboxAssets { return nil }},
- {"no assets for this architecture", func(sa *ateletpb.SandboxAssets) *ateletpb.SandboxAssets {
- sa.Assets = map[string]*ateletpb.ArchAssets{"not-" + runtime.GOARCH: sa.Assets[runtime.GOARCH]}
- return sa
- }},
- {"gvisor without pause image", func(sa *ateletpb.SandboxAssets) *ateletpb.SandboxAssets {
- sa.PauseImage = ""
- return sa
- }},
- } {
- t.Run(tc.name, func(t *testing.T) {
- if _, err := recordFromRequest(tc.mutate(valid())); err == nil {
- t.Error("recordFromRequest succeeded, want an error")
- }
- })
- }
-}
-
// newTestAssetCacheAt opens an asset cache under root/cache. Destinations a
// test links into should live under the same root, on the cache's
// filesystem.
-func newTestAssetCacheAt(t *testing.T, root string) *filecache.Store {
+func newTestAssetCacheAt(t *testing.T, root string, opts ...filecache.Option) *filecache.Store {
t.Helper()
- store, err := filecache.New(filepath.Join(root, "cache"))
+ store, err := filecache.New(filepath.Join(root, "cache"), opts...)
if err != nil {
t.Fatalf("filecache.New: %v", err)
}
@@ -279,8 +212,8 @@ func cachedAssets(t *testing.T, root string) int {
return len(entries)
}
-// useTempAssetScratch points prewarm's scratch dir at a temp dir. Not
-// parallel-safe: the path is process-global.
+// useTempAssetScratch points the scratch dir of fetches without an actor
+// at a temp dir. Not parallel-safe: the path is process-global.
func useTempAssetScratch(t *testing.T) {
t.Helper()
orig := ateletpath.SandboxAssetScratchDir
@@ -357,11 +290,6 @@ func newGVisorTestRelease(t *testing.T) gvisorTestRelease {
}
}
-// proto is the asset as a Run or Restore request carries it.
-func (r gvisorTestRelease) proto() *ateletpb.AssetFile {
- return &ateletpb.AssetFile{Url: r.entry.URL, Sha256: r.entry.SHA256}
-}
-
// stagedRunsc is where an actor's staged copy of the release's runsc sits
// under its sandbox assets dir.
func stagedRunsc(actorUID string) string {
diff --git a/cmd/atelet/sandbox_runtime_index.go b/cmd/atelet/sandbox_runtime_index.go
index d466a51294..0dc8050808 100644
--- a/cmd/atelet/sandbox_runtime_index.go
+++ b/cmd/atelet/sandbox_runtime_index.go
@@ -283,12 +283,11 @@ func (x *runtimeIndex) worker(podUID string) (*workerRuntime, bool) {
// under rt's bundle hash. ate-api-server fills rt from the worker
// assignment, which is a runtime the worker registered from the file this
// node wrote, so the hash resolves as long as the request reached the node
-// the actor was placed on. The request's sandbox_assets are not consulted.
+// the actor was placed on.
//
-// A miss is not served from the request: it means the request reached a
-// node whose workers never advertised the runtime, which must surface. A
-// runtime registered before an atelet restart is no miss: the index is
-// loaded from disk before atelet serves.
+// A miss means the request reached a node whose workers never advertised the
+// runtime, which must surface. A runtime registered before an atelet restart
+// is no miss: the index is loaded from disk before atelet serves.
func (s *Atelet) advertisedRuntime(ctx context.Context, rt *ateletpb.SandboxRuntime) (*sandboxAssetsRecord, error) {
hash, err := bundleHashOf(rt)
if err != nil {
diff --git a/cmd/atelet/tlsconfig_test.go b/cmd/atelet/tlsconfig_test.go
index 2d761e2471..985b7f28a4 100644
--- a/cmd/atelet/tlsconfig_test.go
+++ b/cmd/atelet/tlsconfig_test.go
@@ -21,7 +21,7 @@ import (
)
func TestAteletServerTLSConfigRejectsUnreadableCACerts(t *testing.T) {
- _, err := ateletServerTLSConfig("/nonexistent-cred-bundle.pem", filepath.Join(t.TempDir(), "absent.pem"))
+ _, err := ateletServerTLSConfig("/nonexistent-cred-bundle.pem", filepath.Join(t.TempDir(), "absent.pem"), allowAll)
if err == nil {
t.Fatalf("ateletServerTLSConfig() error = nil, want an error for a missing CA file")
}
@@ -36,7 +36,7 @@ func TestAteletServerTLSConfigReloadsCACertsWithoutRestart(t *testing.T) {
t.Fatalf("WriteFile() error = %v", err)
}
- cfg, err := ateletServerTLSConfig("/nonexistent-cred-bundle.pem", path)
+ cfg, err := ateletServerTLSConfig("/nonexistent-cred-bundle.pem", path, allowAll)
if err != nil {
t.Fatalf("ateletServerTLSConfig() error = %v", err)
}
diff --git a/cmd/atelet/worker_sandbox_syncer.go b/cmd/atelet/worker_sandbox_syncer.go
index df88d4f313..ac2b7582f8 100644
--- a/cmd/atelet/worker_sandbox_syncer.go
+++ b/cmd/atelet/worker_sandbox_syncer.go
@@ -31,10 +31,12 @@ import (
"context"
"errors"
"fmt"
+ "io/fs"
"log/slog"
"math/rand/v2"
"os"
"path/filepath"
+ "strings"
"sync"
"time"
@@ -99,10 +101,11 @@ type workerSandboxSyncer struct {
// index is told each pod's runtimes before its file is written, and
// what the pod advertises once it is.
index *runtimeIndex
- // prewarm, if set, is handed the default runtime hash of every pod
- // whose file was written, so its assets are fetched before an actor
- // asks for them.
- prewarm func(hash string)
+ // prewarm, if set, is handed each default runtime a pod's file
+ // advertises, with a directory under the pod's to link the assets into,
+ // so they are fetched before an actor asks for them and stay cached
+ // while none runs them.
+ prewarm runtimePrewarm
// pods holds this node's worker pods, indexed by podUIDIndex.
pods cache.Indexer
assets sandboxAssetFetcher
@@ -133,7 +136,7 @@ type workerSandboxSyncer struct {
cpuFeatures map[string]string
}
-func newWorkerSandboxSyncer(resolver *workerSandboxResolver, pods cache.Indexer, assets sandboxAssetFetcher, workersDir string, index *runtimeIndex, prewarm func(hash string)) *workerSandboxSyncer {
+func newWorkerSandboxSyncer(resolver *workerSandboxResolver, pods cache.Indexer, assets sandboxAssetFetcher, workersDir string, index *runtimeIndex, prewarm runtimePrewarm) *workerSandboxSyncer {
return &workerSandboxSyncer{
resolver: resolver,
pods: pods,
@@ -155,6 +158,61 @@ func newWorkerSandboxSyncer(resolver *workerSandboxResolver, pods cache.Indexer,
}
}
+// runtimePrewarm is what the syncer asks of the runtime prewarmer: fetch a
+// default runtime and link its assets into the given directory.
+type runtimePrewarm interface {
+ enqueue(hash, dir string)
+}
+
+// defaultRuntimeDirsName is the directory under a worker's own that holds
+// one directory per default runtime the worker advertises, by bundle hash,
+// one per sandbox class, which the prewarmer links the runtime's assets
+// into. The links are what keeps the assets cached while no actor runs
+// them: the cache never evicts a linked entry. The syncer makes a runtime's
+// directory when the worker's file first advertises it as a default and
+// removes it when the file stops; they all go with the worker's directory.
+const defaultRuntimeDirsName = "default-runtime"
+
+// defaultRuntimeDir is the directory for the default runtime hash of the
+// worker pod.
+func (s *workerSandboxSyncer) defaultRuntimeDir(uid, hash string) string {
+ return filepath.Join(s.podDir(uid), defaultRuntimeDirsName, strings.ReplaceAll(hash, ":", "-"))
+}
+
+// syncDefaultRuntimeDirs makes the worker pod's default-runtime directory
+// hold one directory per hash in defaults and nothing else: the others are
+// removed, which unlinks the runtimes the worker no longer advertises, and
+// the missing ones are created empty for the prewarmer to fill. Called
+// once the pod's file advertises defaults, so nothing it removes is a
+// runtime the worker still names.
+func (s *workerSandboxSyncer) syncDefaultRuntimeDirs(uid string, defaults []string) error {
+ parent := filepath.Join(s.podDir(uid), defaultRuntimeDirsName)
+ if err := os.MkdirAll(parent, 0o700); err != nil {
+ return fmt.Errorf("while creating the worker's default runtime dir: %w", err)
+ }
+ want := make(map[string]bool, len(defaults))
+ for _, hash := range defaults {
+ want[filepath.Base(s.defaultRuntimeDir(uid, hash))] = true
+ }
+ entries, err := os.ReadDir(parent)
+ if err != nil {
+ return fmt.Errorf("while listing the worker's default runtime dir: %w", err)
+ }
+ for _, e := range entries {
+ if !want[e.Name()] {
+ if err := os.RemoveAll(filepath.Join(parent, e.Name())); err != nil {
+ return fmt.Errorf("while removing a retired default runtime dir: %w", err)
+ }
+ }
+ }
+ for name := range want {
+ if err := os.Mkdir(filepath.Join(parent, name), 0o700); err != nil && !errors.Is(err, fs.ErrExist) {
+ return fmt.Errorf("while creating a default runtime dir: %w", err)
+ }
+ }
+ return nil
+}
+
// startWorkerSandboxSyncer wires a syncer to the given informers, which the
// caller starts, and runs it until ctx ends. pods must hold only this node's
// worker pods and must not have started yet. Worker directories live under
@@ -167,7 +225,7 @@ func newWorkerSandboxSyncer(resolver *workerSandboxResolver, pods cache.Indexer,
// or a list/watch fails (e.g. Forbidden while an RBAC rollout lags the
// binary), existing files and index entries stay as they are and only
// changes wait.
-func startWorkerSandboxSyncer(ctx context.Context, pods coreinformers.PodInformer, workerPools informersv1alpha1.WorkerPoolInformer, configs informersv1alpha1.SandboxConfigInformer, assets sandboxAssetFetcher, workersDir string, index *runtimeIndex, prewarm func(hash string)) (*workerSandboxSyncer, error) {
+func startWorkerSandboxSyncer(ctx context.Context, pods coreinformers.PodInformer, workerPools informersv1alpha1.WorkerPoolInformer, configs informersv1alpha1.SandboxConfigInformer, assets sandboxAssetFetcher, workersDir string, index *runtimeIndex, prewarm runtimePrewarm) (*workerSandboxSyncer, error) {
if err := pods.Informer().AddIndexers(cache.Indexers{podUIDIndex: func(obj any) ([]string, error) {
pod, ok := obj.(*corev1.Pod)
if !ok {
@@ -238,7 +296,7 @@ func (s *workerSandboxSyncer) workerPoolHandler() cache.ResourceEventHandler {
UpdateFunc: func(oldObj, newObj any) {
oldPool, oldOK := oldObj.(*v1alpha1.WorkerPool)
newPool, newOK := newObj.(*v1alpha1.WorkerPool)
- if oldOK && newOK && poolConfigName(oldPool) != poolConfigName(newPool) {
+ if oldOK && newOK && oldPool.ResolvedSandboxConfig() != newPool.ResolvedSandboxConfig() {
s.enqueuePool(newPool, s.maxJitter)
}
},
@@ -254,7 +312,7 @@ func (s *workerSandboxSyncer) sandboxConfigHandler() cache.ResourceEventHandler
return
}
for _, pool := range pools {
- if poolConfigName(pool) == cfg.Name {
+ if pool.ResolvedSandboxConfig() == cfg.Name {
s.enqueuePool(pool, s.maxJitter)
}
}
diff --git a/cmd/atelet/worker_sandbox_syncer_test.go b/cmd/atelet/worker_sandbox_syncer_test.go
index 84aa2bbbda..38cb0a5b3b 100644
--- a/cmd/atelet/worker_sandbox_syncer_test.go
+++ b/cmd/atelet/worker_sandbox_syncer_test.go
@@ -70,6 +70,10 @@ func (f *fakeAssetFetcher) ensureSandboxAssets(ctx context.Context, rec *sandbox
return f.fn(ctx, rec)
}
+func (f *fakeAssetFetcher) stageSandboxAssets(ctx context.Context, rec *sandboxAssetsRecord, _ string) (map[string]string, error) {
+ return f.fn(ctx, rec)
+}
+
// newTestAssetFetcher returns a sandboxAssetFetcher that maps every asset of
// a record to a path under its digest without fetching anything, for syncers
// whose runsc probe is faked.
diff --git a/cmd/atelet/worker_sandbox_versions.go b/cmd/atelet/worker_sandbox_versions.go
index 7181306bf8..d964c9c4de 100644
--- a/cmd/atelet/worker_sandbox_versions.go
+++ b/cmd/atelet/worker_sandbox_versions.go
@@ -100,19 +100,6 @@ type workerSandboxResolver struct {
configs listersv1alpha1.SandboxConfigLister
}
-// poolConfigName returns the SandboxConfig the atecontroller confirmed for the
-// class the pool's workers run, or "" if it has not resolved one yet. It never
-// returns pendingConfigRef: that config is not confirmed yet.
-func poolConfigName(pool *v1alpha1.WorkerPool) string {
- class := pool.Spec.DefaultSandboxClass()
- for _, c := range pool.Status.SandboxClasses {
- if c.Name == class && c.ConfigRef != nil {
- return c.ConfigRef.Name
- }
- }
- return ""
-}
-
// resolve returns the SandboxConfig of a worker pod on this node. Apart from
// a pod in no pool, every failure can succeed once an informer catches up or
// the atecontroller resolves the pool.
@@ -139,7 +126,7 @@ func (r *workerSandboxResolver) pool(pod *corev1.Pod) (*v1alpha1.WorkerPool, err
// config returns the SandboxConfig the workers of pool run.
func (r *workerSandboxResolver) config(pool *v1alpha1.WorkerPool) (*v1alpha1.SandboxConfig, error) {
- configName := poolConfigName(pool)
+ configName := pool.ResolvedSandboxConfig()
if configName == "" {
return nil, fmt.Errorf("%w: WorkerPool %s/%s has not resolved a SandboxConfig for sandbox class %q", errUnresolved, pool.Namespace, pool.Name, pool.Spec.DefaultSandboxClass())
}
@@ -201,8 +188,8 @@ func planVersions(cfg *v1alpha1.SandboxConfig) ([]versionPlan, error) {
}
// recordForArch projects the assets of one version of a SandboxConfig onto
-// this node's architecture, mirroring recordFromRequest. It reports false when
-// the version lists no assets for this architecture.
+// this node's architecture. It reports false when the version lists no assets
+// for this architecture.
func recordForArch(cfg *v1alpha1.SandboxConfig, ver *v1alpha1.SandboxVersionConfig) (*sandboxAssetsRecord, bool) {
files := ver.Assets[runtime.GOARCH]
if len(files) == 0 {
@@ -474,12 +461,12 @@ func bundleHash(rec *sandboxAssetsRecord, pauseDigest string) string {
// - plan: resolve the pool's config and its versions for this node
// - hash: hash each version's runtime, and make each resolvable by its
// hash before the file can let the worker register it
-// - prewarm: have the default runtime's assets fetched, off this
-// goroutine, so the probe below is its head start and the file never
-// waits on a download
// - build: probe what a snapshot taken on each runtime must find again
// - write: replace the file if its content changed, then record what
// the worker now advertises in the index
+// - prewarm: make the pod's default-runtime directories match what the
+// file now advertises, and have each default's assets fetched and
+// linked into its directory, off this goroutine
//
// A file missing restorable runtimes for a transient reason is written all
// the same, and the errIncomplete from build is returned so finish retries.
@@ -497,17 +484,6 @@ func (s *workerSandboxSyncer) syncPod(ctx context.Context, pod *corev1.Pod) erro
if err := s.index.registerRuntimes(plan.config, hashed); err != nil {
return err
}
- // Queued now rather than after the write: the build fetches and probes
- // every restorable version serially, longest on a fresh node, and the
- // file it ends with is what lets the worker take its first actor. The
- // prewarm gets that whole duration to finish first.
- if s.prewarm != nil {
- for _, v := range hashed {
- if v.isDefault {
- s.prewarm(v.hash)
- }
- }
- }
want, buildErr := s.buildFile(ctx, plan, hashed)
if buildErr != nil && !errors.Is(buildErr, errIncomplete) {
return buildErr
@@ -522,6 +498,24 @@ func (s *workerSandboxSyncer) syncPod(ctx context.Context, pod *corev1.Pod) erro
if err := s.index.register(string(pod.UID), advertisedClass{class: plan.sandboxClass, config: plan.config, hashed: writtenVersions(hashed, want)}); err != nil {
return err
}
+ // Only now, with the file naming its defaults: a sync that failed above
+ // left the file, and so the directories, as they were. For a gVisor
+ // default the build fetched the release for its probe, so the prewarm
+ // is links and the pause image; a micro-VM default downloads here.
+ if s.prewarm != nil {
+ var defaults []string
+ for _, v := range writtenVersions(hashed, want) {
+ if v.isDefault {
+ defaults = append(defaults, v.hash)
+ }
+ }
+ if err := s.syncDefaultRuntimeDirs(string(pod.UID), defaults); err != nil {
+ return err
+ }
+ for _, hash := range defaults {
+ s.prewarm.enqueue(hash, s.defaultRuntimeDir(string(pod.UID), hash))
+ }
+ }
return buildErr
}
diff --git a/cmd/atelet/worker_sandbox_versions_test.go b/cmd/atelet/worker_sandbox_versions_test.go
index fe7c00ea79..7941cc1915 100644
--- a/cmd/atelet/worker_sandbox_versions_test.go
+++ b/cmd/atelet/worker_sandbox_versions_test.go
@@ -155,42 +155,6 @@ func TestBundleHash(t *testing.T) {
}
}
-func TestPoolConfigName(t *testing.T) {
- tests := []struct {
- name string
- pool *v1alpha1.WorkerPool
- want string
- }{{
- name: "confirmed config of the pool's class",
- pool: resolvedPool("pool", "gvisor-default"),
- want: "gvisor-default",
- }, {
- name: "not resolved yet",
- pool: resolvedPool("pool", ""),
- }, {
- name: "only a pending config",
- pool: func() *v1alpha1.WorkerPool {
- pool := resolvedPool("pool", "")
- pool.Status.SandboxClasses[0].PendingConfigRef = &v1alpha1.SandboxConfigReference{Name: "gvisor-next"}
- return pool
- }(),
- }, {
- name: "status resolved another class than the pool's",
- pool: func() *v1alpha1.WorkerPool {
- pool := resolvedPool("pool", "gvisor-default")
- pool.Spec.SandboxClasses[0].Name = v1alpha1.SandboxClassMicroVM
- return pool
- }(),
- }}
- for _, tt := range tests {
- t.Run(tt.name, func(t *testing.T) {
- if got := poolConfigName(tt.pool); got != tt.want {
- t.Errorf("poolConfigName() = %q, want %q", got, tt.want)
- }
- })
- }
-}
-
func TestWorkerSandboxResolverResolve(t *testing.T) {
cfg := gvisorConfig("gvisor-default", "gs://bucket/gvisor", "abc")
diff --git a/cmd/atelet/workersupport.go b/cmd/atelet/workersupport.go
index e899ca52e6..9e4e27e647 100644
--- a/cmd/atelet/workersupport.go
+++ b/cmd/atelet/workersupport.go
@@ -269,3 +269,32 @@ func (s *workerSupportServer) GetActorEgressPolicy(
return &ateletpb.GetActorEgressPolicyResponse{EgressPolicy: toAteletEgressPolicy(policy)}, nil
}
+
+// ReportActorCrashed forwards a worker's report that the sandbox of an actor
+// it hosts died on its own. Authorized like RequestActorSuspend.
+func (s *workerSupportServer) ReportActorCrashed(ctx context.Context, req *ateletpb.ReportActorCrashedRequest) (*ateletpb.ReportActorCrashedResponse, error) {
+ workerIdentity, err := authenticatedWorkerIdentity(ctx)
+ if err != nil {
+ return nil, err
+ }
+ if errs := apivalidation.ValidateReportActorCrashedRequest(ctx, req); len(errs) > 0 {
+ return nil, resources.ToGRPCStatusError(errs)
+ }
+ if _, err := s.workers.ReportActorCrashed(ctx, &ateapipb.ReportActorCrashedRequest{
+ Worker: &ateapipb.ObjectRef{Name: workerIdentity.PodUID},
+ Actor: &ateapipb.ObjectRef{
+ Atespace: req.GetActorAtespace(),
+ Name: req.GetActorName(),
+ },
+ ActorUid: req.GetActorUid(),
+ Message: req.GetMessage(),
+ }); err != nil {
+ return nil, err
+ }
+ slog.InfoContext(ctx, "Forwarded an actor crash report",
+ slog.String("pod_uid", workerIdentity.PodUID),
+ slog.String("actor_atespace", req.GetActorAtespace()),
+ slog.String("actor_name", req.GetActorName()),
+ slog.String("actor_uid", req.GetActorUid()))
+ return &ateletpb.ReportActorCrashedResponse{}, nil
+}
diff --git a/cmd/atelet/workersupport_test.go b/cmd/atelet/workersupport_test.go
index da90205146..892c456b9e 100644
--- a/cmd/atelet/workersupport_test.go
+++ b/cmd/atelet/workersupport_test.go
@@ -240,8 +240,17 @@ func TestRegisterWorker(t *testing.T) {
type fakeSuspendService struct {
ateapipb.WorkerServiceClient
- got []*ateapipb.RequestActorSuspendRequest
- err error
+ got []*ateapipb.RequestActorSuspendRequest
+ gotCrashed []*ateapipb.ReportActorCrashedRequest
+ err error
+}
+
+func (s *fakeSuspendService) ReportActorCrashed(_ context.Context, in *ateapipb.ReportActorCrashedRequest, _ ...grpc.CallOption) (*ateapipb.ReportActorCrashedResponse, error) {
+ if s.err != nil {
+ return nil, s.err
+ }
+ s.gotCrashed = append(s.gotCrashed, in)
+ return &ateapipb.ReportActorCrashedResponse{}, nil
}
func (s *fakeSuspendService) RequestActorSuspend(_ context.Context, in *ateapipb.RequestActorSuspendRequest, _ ...grpc.CallOption) (*ateapipb.RequestActorSuspendResponse, error) {
@@ -472,3 +481,44 @@ func TestGetActorEgressPolicy(t *testing.T) {
})
}
}
+
+func TestReportActorCrashedNamesTheCallingWorker(t *testing.T) {
+ workers := &fakeSuspendService{}
+ svc := &workerSupportServer{workers: workers}
+
+ if _, err := svc.ReportActorCrashed(workerContext(t, "pod-a"), &ateletpb.ReportActorCrashedRequest{
+ ActorAtespace: "team-a",
+ ActorName: "actor-1",
+ ActorUid: "01234567-89ab-cdef-0123-456789abcdef",
+ Message: "cloud-hypervisor exited",
+ }); err != nil {
+ t.Fatalf("ReportActorCrashed() failed: %v", err)
+ }
+ want := []*ateapipb.ReportActorCrashedRequest{{
+ Worker: &ateapipb.ObjectRef{Name: "pod-a"},
+ Actor: &ateapipb.ObjectRef{Atespace: "team-a", Name: "actor-1"},
+ ActorUid: "01234567-89ab-cdef-0123-456789abcdef",
+ Message: "cloud-hypervisor exited",
+ }}
+ if diff := cmp.Diff(want, workers.gotCrashed, protocmp.Transform()); diff != "" {
+ t.Errorf("forwarded request mismatch (-want +got):\n%s", diff)
+ }
+}
+
+func TestReportActorCrashedRequiresACertificate(t *testing.T) {
+ workers := &fakeSuspendService{}
+ svc := &workerSupportServer{workers: workers}
+
+ _, err := svc.ReportActorCrashed(context.Background(), &ateletpb.ReportActorCrashedRequest{
+ ActorAtespace: "team-a",
+ ActorName: "actor-1",
+ ActorUid: "01234567-89ab-cdef-0123-456789abcdef",
+ Message: "cloud-hypervisor exited",
+ })
+ if status.Code(err) != codes.Unauthenticated {
+ t.Errorf("unauthenticated report returned %v, want Unauthenticated", err)
+ }
+ if len(workers.gotCrashed) != 0 {
+ t.Errorf("unauthenticated report was still forwarded: %v", workers.gotCrashed)
+ }
+}
diff --git a/cmd/ateworker-gvisor/crash.go b/cmd/ateworker-gvisor/crash.go
new file mode 100644
index 0000000000..e0bc39ca6e
--- /dev/null
+++ b/cmd/ateworker-gvisor/crash.go
@@ -0,0 +1,88 @@
+//go:build linux
+
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package main
+
+import (
+ "context"
+ "log/slog"
+ "time"
+
+ "go.ate.dev/substrate/internal/ateworker/register"
+ "go.ate.dev/substrate/internal/proto/ateworkerpb"
+ "go.ate.dev/substrate/internal/resources"
+)
+
+// crashReportTimeout bounds how long a crash report keeps retrying.
+const crashReportTimeout = 5 * time.Minute
+
+// crashReporter is the slice of register.CrashReporter the service uses.
+type crashReporter interface {
+ ReportCrashed(ctx context.Context, actor resources.ActorAttribution, message string) error
+}
+
+var _ crashReporter = (*register.CrashReporter)(nil)
+
+// watchSandbox handles the actor's sandbox dying while nothing is stopping it:
+// the actor is torn down here, since the control plane will not reach this
+// worker for it again, and then reported crashed.
+func (s *AteWorkerService) watchSandbox(ctx context.Context, hosted *hostedActor, runscPath string, actorDirs *ateworkerpb.ActorDirs, containers []*ateworkerpb.Container) {
+ attribution := hosted.attribution
+ if hosted.sandbox == nil {
+ slog.WarnContext(ctx, "Cannot watch the sandbox, a crash will go unreported: it was never pinned", slog.String("actorUID", attribution.UID))
+ return
+ }
+ pid := hosted.sandbox.pid
+ exited, err := hosted.sandbox.exited()
+ if err != nil {
+ slog.WarnContext(ctx, "Cannot watch the sandbox, a crash will go unreported", slog.String("actorUID", attribution.UID), slog.Any("err", err))
+ return
+ }
+ go func() {
+ <-exited
+ if s.shuttingDown.Load() {
+ return
+ }
+ ctx := context.Background()
+ const message = "gVisor sandbox exited"
+
+ // stopping is checked under the actor's lock, so an exit during a
+ // checkpoint that then fails is still handled once it gives up.
+ // Not reported under the lock: a suspend in flight holds the actor's
+ // lease while it waits on that lock for its Checkpoint.
+ if !s.locks.Lock(ctx, attribution.UID) {
+ return
+ }
+ ours := s.lookupActor(attribution.UID) == hosted && !hosted.stopping.Load()
+ if ours {
+ slog.WarnContext(ctx, "Sandbox died", slog.String("actorUID", attribution.UID), slog.Int("pid", pid))
+ if err := s.terminateWorkload(ctx, attribution.Ref, attribution.UID, runscPath, actorDirs, containers); err != nil {
+ slog.WarnContext(ctx, "Failed to tear down the dead sandbox", slog.String("actorUID", attribution.UID), slog.Any("err", err))
+ }
+ // not measured first: the sandbox, and its cgroup, are gone
+ s.recordFinalIfEnded(ctx, hosted)
+ }
+ s.locks.Unlock(attribution.UID)
+ if !ours || s.crashReporter == nil {
+ return
+ }
+ reportCtx, cancel := context.WithTimeout(ctx, crashReportTimeout)
+ defer cancel()
+ if err := s.crashReporter.ReportCrashed(reportCtx, attribution, message); err != nil {
+ slog.ErrorContext(ctx, "Failed to report the actor crashed", slog.String("actorUID", attribution.UID), slog.Any("err", err))
+ }
+ }()
+}
diff --git a/cmd/ateworker-gvisor/crash_test.go b/cmd/ateworker-gvisor/crash_test.go
new file mode 100644
index 0000000000..151fd9a16b
--- /dev/null
+++ b/cmd/ateworker-gvisor/crash_test.go
@@ -0,0 +1,88 @@
+//go:build linux
+
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package main
+
+import (
+ "context"
+ "os"
+ "os/exec"
+ "strconv"
+ "testing"
+ "time"
+
+ "go.ate.dev/substrate/internal/actorlock"
+ "go.ate.dev/substrate/internal/ocispec"
+ "go.ate.dev/substrate/internal/proto/ateworkerpb"
+ "go.ate.dev/substrate/internal/resources"
+)
+
+type fakeCrashReporter struct {
+ reports chan string
+}
+
+func (f *fakeCrashReporter) ReportCrashed(_ context.Context, _ resources.ActorAttribution, message string) error {
+ f.reports <- message
+ return nil
+}
+
+// exitingSandbox records a process that exits at once as the actor's sandbox.
+func exitingSandbox(t *testing.T) *ateworkerpb.ActorDirs {
+ t.Helper()
+ cmd := exec.Command("true")
+ if err := cmd.Start(); err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { _ = cmd.Wait() })
+ actorDirs := &ateworkerpb.ActorDirs{RootDir: t.TempDir()}
+ if err := os.MkdirAll(pidFileDir(actorDirs), 0o700); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.WriteFile(pidFilePath(actorDirs, ocispec.PauseContainer), []byte(strconv.Itoa(cmd.Process.Pid)), 0o600); err != nil {
+ t.Fatal(err)
+ }
+ return actorDirs
+}
+
+func TestWatchSandboxIgnoresExpectedExits(t *testing.T) {
+ for _, tc := range []struct {
+ name string
+ setup func(*AteWorkerService, *hostedActor)
+ }{
+ {"stopped by ateworker", func(_ *AteWorkerService, h *hostedActor) { h.stopping.Store(true) }},
+ {"worker shutting down", func(s *AteWorkerService, _ *hostedActor) { s.shuttingDown.Store(true) }},
+ {"actor no longer hosted", func(s *AteWorkerService, _ *hostedActor) { delete(s.actors, "actor-a") }},
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ reporter := &fakeCrashReporter{reports: make(chan string, 1)}
+ actorDirs := exitingSandbox(t)
+ sandbox, err := openSandboxProcess(actorDirs)
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(sandbox.Close)
+ hosted := &hostedActor{attribution: resources.ActorAttribution{UID: "actor-a"}, sandbox: sandbox}
+ s := &AteWorkerService{locks: actorlock.New(), actors: map[string]*hostedActor{"actor-a": hosted}, crashReporter: reporter}
+ tc.setup(s, hosted)
+ s.watchSandbox(context.Background(), hosted, "", actorDirs, nil)
+ select {
+ case msg := <-reporter.reports:
+ t.Fatalf("reported %q for an expected exit", msg)
+ case <-time.After(500 * time.Millisecond):
+ }
+ })
+ }
+}
diff --git a/cmd/ateworker-gvisor/hosted.go b/cmd/ateworker-gvisor/hosted.go
index c603bd07f8..a1c764ebe2 100644
--- a/cmd/ateworker-gvisor/hosted.go
+++ b/cmd/ateworker-gvisor/hosted.go
@@ -46,6 +46,13 @@ type hostedActor struct {
// ctx is canceled when the actor is replaced or unhosted.
ctx context.Context
cancel context.CancelFunc
+ // stopping is set before ateworker itself stops the sandbox, so watchSandbox
+ // can tell that exit from a crash.
+ stopping atomic.Bool
+ // sandbox pins the sandbox process from the moment runsc creates it, so
+ // nothing that waits on it later can be fooled by a reused pid. Nil if it
+ // could not be pinned. Set once, under the actor's lock.
+ sandbox *sandboxProcess
}
// admitActor reserves capacity before network setup. An actor that is already
@@ -57,6 +64,12 @@ func (s *AteWorkerService) admitActor(attribution resources.ActorAttribution) (*
if old, ok := s.actors[attribution.UID]; ok {
stale = old.network
old.cancel()
+ // the old activation's sandbox exiting from here on is not a crash
+ old.stopping.Store(true)
+ if old.sandbox != nil {
+ old.sandbox.Close()
+ old.sandbox = nil
+ }
} else if len(s.actors)+s.draining >= s.maxActors {
return nil, nil, apierror.ResourceExhausted("worker is full: %d actors", s.maxActors)
}
@@ -127,6 +140,12 @@ func (s *AteWorkerService) unhostActor(ctx context.Context, actorUID string) err
s.actorsMu.Unlock()
}()
+ s.actorsMu.Lock()
+ if hosted.sandbox != nil {
+ hosted.sandbox.Close()
+ hosted.sandbox = nil
+ }
+ s.actorsMu.Unlock()
if hosted.network == nil {
return nil
}
@@ -137,6 +156,18 @@ func (s *AteWorkerService) unhostActor(ctx context.Context, actorUID string) err
return nil
}
+// pinSandbox pins the actor's sandbox, which runsc has just created.
+func (s *AteWorkerService) pinSandbox(ctx context.Context, hosted *hostedActor, actorDirs *ateworkerpb.ActorDirs) {
+ sandbox, err := openSandboxProcess(actorDirs)
+ if err != nil {
+ slog.WarnContext(ctx, "Cannot pin the sandbox process", slog.String("actorUID", hosted.attribution.UID), slog.Any("err", err))
+ return
+ }
+ s.actorsMu.Lock()
+ hosted.sandbox = sandbox
+ s.actorsMu.Unlock()
+}
+
// lookupActor returns the actor or nil without taking its lifecycle lock.
func (s *AteWorkerService) lookupActor(actorUID string) *hostedActor {
s.actorsMu.RLock()
diff --git a/cmd/ateworker-gvisor/main.go b/cmd/ateworker-gvisor/main.go
index ce61ef3269..c52a5ef12a 100644
--- a/cmd/ateworker-gvisor/main.go
+++ b/cmd/ateworker-gvisor/main.go
@@ -140,7 +140,8 @@ func do(ctx context.Context) error {
const serviceName = "ateworker-gvisor"
// Export through atelet's node-local relay when it is there, so telemetry
// never touches the worker pod's network. A nil conn means it is not, and
- // the providers fall back to dialing the collector directly.
+ // the providers fall back to OTEL_EXPORTER_OTLP_ENDPOINT, and export
+ // nothing without one.
//
// A relay that cannot be dialed is logged rather than fatal, matching both
// ends of the same decision: Dial already treats an absent socket as a
@@ -150,7 +151,7 @@ func do(ctx context.Context) error {
// over its telemetry route would turn a misconfigured flag into an outage.
relayConn, err := otlprelay.Dial(ctx, *otlpRelaySocket)
if err != nil {
- slog.ErrorContext(ctx, "Failed to connect to the OTLP relay; exporting telemetry directly over the pod network",
+ slog.ErrorContext(ctx, "Failed to connect to the OTLP relay; falling back to the OTLP endpoint, if one is set",
slog.String("socket", *otlpRelaySocket), slog.Any("err", err))
}
if relayConn != nil {
@@ -162,7 +163,7 @@ func do(ctx context.Context) error {
Sampling: serverboot.ResolveTraceSampling(ctx, serverboot.ParentRatioSampling(serverboot.ControlPlaneTraceRatio)),
ExporterConn: relayConn,
// So the spans say which path they took, including when relayConn is nil
- // because the dial above failed and this ateworker is exporting directly.
+ // because the dial above failed and this ateworker exports directly or not at all.
RelayCapable: true,
})
if err != nil {
@@ -185,7 +186,7 @@ func do(ctx context.Context) error {
if err != nil {
serverboot.Fatal(ctx, "Failed to initialize logging", err)
}
- // Nil when the exporter does not include otlp.
+ // Nil when there is nothing to export.
if lp != nil {
defer serverboot.ShutdownProvider("LoggerProvider", lp.Shutdown)
}
@@ -279,16 +280,22 @@ func do(ctx context.Context) error {
// Report retries every failure it can outlast, including the window before
// the Worker record exists; anything that reaches here is a misconfiguration
// no restart-in-place will fix.
+ reportConfig := register.ReportConfig{
+ SocketPath: nodepath.WorkerSupportSocket,
+ CredentialBundlePath: tunnelConfig.CredentialBundle,
+ TrustBundlePath: tunnelConfig.TrustBundle,
+ AteletSPIFFEID: tunnelConfig.BrokerIdentity,
+ Actors: *maxActors,
+ SandboxClass: string(atev1alpha1.SandboxClassGvisor),
+ RuntimesFile: nodepath.WorkerSandboxRuntimesPath(*podUID),
+ }
+ if reporter, err := register.NewCrashReporter(reportConfig); err != nil {
+ slog.ErrorContext(ctx, "Crashed actors will not be reported", slog.Any("err", err))
+ } else {
+ ateworkerService.crashReporter = reporter
+ }
go func() {
- err := register.Report(ctx, register.ReportConfig{
- SocketPath: nodepath.WorkerSupportSocket,
- CredentialBundlePath: tunnelConfig.CredentialBundle,
- TrustBundlePath: tunnelConfig.TrustBundle,
- AteletSPIFFEID: tunnelConfig.BrokerIdentity,
- Actors: *maxActors,
- SandboxClass: string(atev1alpha1.SandboxClassGvisor),
- RuntimesFile: nodepath.WorkerSandboxRuntimesPath(*podUID),
- })
+ err := register.Report(ctx, reportConfig)
if err != nil && ctx.Err() == nil {
serverboot.Fatal(ctx, "Failed to register worker", err)
}
@@ -325,6 +332,9 @@ type AteWorkerService struct {
locks *actorlock.Locks
// Tracks lifecycle RPCs for shutdown cancellation and draining.
inFlight *actorlock.InFlight
+ // crashReporter reports actors whose sandbox died. Nil if it could not be
+ // set up.
+ crashReporter crashReporter
// Guards actors and draining against concurrent stats reads.
actorsMu sync.RWMutex
@@ -679,6 +689,7 @@ func (s *AteWorkerService) RunWorkload(ctx context.Context, req *ateworkerpb.Run
if err := rcmd.cmdCreate(ctx, pw, ocispec.PauseContainer, nil); err != nil {
return nil, fmt.Errorf("while creating pause container: %w", err)
}
+ s.pinSandbox(ctx, hosted, req.GetActorDirs())
if err := rcmd.cmdStart(ctx, pw, ocispec.PauseContainer); err != nil {
return nil, fmt.Errorf("while starting pause container: %w", err)
}
@@ -714,6 +725,7 @@ func (s *AteWorkerService) RunWorkload(ctx context.Context, req *ateworkerpb.Run
s.actorLogger.EmitLifecycleLog(ctx, "Actor started", attribution)
s.setSession(req.GetActorUid(), &workloadSession{rcmd: rcmd, containers: containerNames(req.GetSpec().GetContainers())})
s.recordInitial(ctx, hosted)
+ s.watchSandbox(ctx, hosted, req.GetRunscPath(), req.GetActorDirs(), req.GetSpec().GetContainers())
return &ateworkerpb.RunWorkloadResponse{}, nil
}
@@ -836,14 +848,28 @@ func (s *AteWorkerService) CheckpointWorkload(ctx context.Context, req *ateworke
// rootfs overlays into the rootfs bundle and durable-dir volumes into
// /fs when present.
checkpointFn = func(ctx context.Context) error {
- sandbox, sbErr := openSandboxProcess(req.GetActorDirs())
- if sbErr != nil {
- slog.WarnContext(ctx, "Cannot watch the sandbox exit after checkpoint", slog.Any("err", sbErr))
- } else {
- defer sandbox.Close()
+ var sandbox *sandboxProcess
+ if hosted != nil {
+ hosted.stopping.Store(true)
+ sandbox = hosted.sandbox
+ }
+ if sandbox == nil {
+ // never pinned: fall back to the pidfile, still better than
+ // cleaning up before the sandbox is reaped
+ var err error
+ if sandbox, err = openSandboxProcess(req.GetActorDirs()); err != nil {
+ slog.WarnContext(ctx, "Cannot watch the sandbox exit after checkpoint", slog.Any("err", err))
+ } else {
+ defer sandbox.Close()
+ }
}
err := rcmd.cmdCheckpoint(ctx, ocispec.PauseContainer, checkpointPath, fsPaths)
if err != nil {
+ // the actor may stay RUNNING, so its sandbox dying is a crash
+ // again, including if this checkpoint stopped it
+ if hosted != nil {
+ hosted.stopping.Store(false)
+ }
timing.checkpoint = lap(&tLast)
return fmt.Errorf("while checkpointing pause: %w", err)
}
@@ -1089,6 +1115,7 @@ func (s *AteWorkerService) RestoreWorkload(ctx context.Context, req *ateworkerpb
if err != nil {
return nil, fmt.Errorf("while creating pause container: %w", err)
}
+ s.pinSandbox(ctx, hosted, req.GetActorDirs())
err = rcmd.cmdStart(ctx, pw, ocispec.PauseContainer)
timing.pauseRestore = lap(&tLast)
if err != nil {
@@ -1102,6 +1129,7 @@ func (s *AteWorkerService) RestoreWorkload(ctx context.Context, req *ateworkerpb
if err != nil {
return nil, fmt.Errorf("while creating pause container: %w", err)
}
+ s.pinSandbox(ctx, hosted, req.GetActorDirs())
err = rcmd.cmdRestore(ctx, pw, ocispec.PauseContainer, checkpointDir)
timing.pauseRestore = lap(&tLast)
if err != nil {
@@ -1176,6 +1204,7 @@ func (s *AteWorkerService) RestoreWorkload(ctx context.Context, req *ateworkerpb
s.actorLogger.EmitLifecycleLog(ctx, "Actor restored", attribution)
s.setSession(req.GetActorUid(), &workloadSession{rcmd: rcmd, containers: containerNames(containers)})
s.recordInitial(ctx, hosted)
+ s.watchSandbox(ctx, hosted, req.GetRunscPath(), req.GetActorDirs(), containers)
return &ateworkerpb.RestoreWorkloadResponse{}, nil
}
@@ -1210,6 +1239,9 @@ func (s *AteWorkerService) TerminateWorkload(ctx context.Context, req *ateworker
}
func (s *AteWorkerService) terminateWorkload(ctx context.Context, actorRef resources.ActorRef, actorUID, runscPath string, actorDirs *ateworkerpb.ActorDirs, containers []*ateworkerpb.Container) error {
+ if hosted := s.lookupActor(actorUID); hosted != nil {
+ hosted.stopping.Store(true)
+ }
var errs []error
if err := s.tunnel.Deactivate(ctx, resources.ActorAttribution{Ref: actorRef, UID: actorUID}); err != nil {
errs = append(errs, fmt.Errorf("while deactivating actor networking: %w", err))
diff --git a/cmd/ateworker-gvisor/sandboxexit.go b/cmd/ateworker-gvisor/sandboxexit.go
index 109e0cf221..6bbe327812 100644
--- a/cmd/ateworker-gvisor/sandboxexit.go
+++ b/cmd/ateworker-gvisor/sandboxexit.go
@@ -25,6 +25,8 @@ import (
"golang.org/x/sys/unix"
+ "go.ate.dev/substrate/internal/pidfd"
+
"go.ate.dev/substrate/internal/ocispec"
"go.ate.dev/substrate/internal/proto/ateworkerpb"
)
@@ -43,14 +45,10 @@ type sandboxProcess struct {
// openSandboxProcess pins the sandbox before it can exit, so a reused pid is
// never mistaken for it.
func openSandboxProcess(actorDirs *ateworkerpb.ActorDirs) (*sandboxProcess, error) {
- b, err := os.ReadFile(pidFilePath(actorDirs, ocispec.PauseContainer))
+ pid, err := sandboxPid(actorDirs)
if err != nil {
return nil, err
}
- pid, err := strconv.Atoi(strings.TrimSpace(string(b)))
- if err != nil {
- return nil, fmt.Errorf("parsing sandbox pid %q: %w", b, err)
- }
fd, err := unix.PidfdOpen(pid, 0)
if err != nil {
return nil, fmt.Errorf("opening pidfd for sandbox %d: %w", pid, err)
@@ -58,6 +56,25 @@ func openSandboxProcess(actorDirs *ateworkerpb.ActorDirs) (*sandboxProcess, erro
return &sandboxProcess{pid: pid, pidfd: fd}, nil
}
+// sandboxPid reads the sandbox's pid from the pause container's pid file,
+// which runsc writes for the sandbox.
+func sandboxPid(actorDirs *ateworkerpb.ActorDirs) (int, error) {
+ b, err := os.ReadFile(pidFilePath(actorDirs, ocispec.PauseContainer))
+ if err != nil {
+ return 0, err
+ }
+ pid, err := strconv.Atoi(strings.TrimSpace(string(b)))
+ if err != nil {
+ return 0, fmt.Errorf("parsing sandbox pid %q: %w", b, err)
+ }
+ return pid, nil
+}
+
+// exited returns a channel closed once the sandbox exits.
+func (p *sandboxProcess) exited() (<-chan struct{}, error) {
+ return pidfd.ExitedFD(p.pidfd)
+}
+
func (p *sandboxProcess) Close() {
_ = unix.Close(p.pidfd)
}
diff --git a/cmd/ateworker-gvisor/sandboxexit_test.go b/cmd/ateworker-gvisor/sandboxexit_test.go
index c7b477d8f6..a2067de117 100644
--- a/cmd/ateworker-gvisor/sandboxexit_test.go
+++ b/cmd/ateworker-gvisor/sandboxexit_test.go
@@ -82,3 +82,27 @@ func TestSandboxWaitGoneTimesOutWhileRunning(t *testing.T) {
t.Fatalf("waitGone = %v, want a still present error", err)
}
}
+
+func TestSandboxExited(t *testing.T) {
+ cmd, p := startFakeSandbox(t)
+ exited, err := p.exited()
+ if err != nil {
+ t.Fatal(err)
+ }
+ select {
+ case <-exited:
+ t.Fatal("closed while the sandbox runs")
+ case <-time.After(100 * time.Millisecond):
+ }
+ _ = cmd.Process.Kill()
+ select {
+ case <-exited:
+ case <-time.After(10 * time.Second):
+ t.Fatal("not closed after the sandbox exited")
+ }
+ // the watch uses its own copy, so the sandbox's pidfd still works
+ go func() { _, _ = cmd.Process.Wait() }()
+ if err := p.waitGone(5 * time.Second); err != nil {
+ t.Fatalf("waitGone after the watch = %v", err)
+ }
+}
diff --git a/cmd/ateworker-microvm/checkpoint.go b/cmd/ateworker-microvm/checkpoint.go
index 68de889094..1cb6484eb0 100644
--- a/cmd/ateworker-microvm/checkpoint.go
+++ b/cmd/ateworker-microvm/checkpoint.go
@@ -342,6 +342,9 @@ func (s *AteWorkerService) teardownActor(ctx context.Context, id string, actorDi
// FAILED_PRECONDITION path ("no numbers right now") instead of surfacing a
// closed connection as a failed read.
s.setGuestStats(id, nil)
+ if ra != nil {
+ ra.stopping.Store(true)
+ }
var errs []error
if client != nil {
@@ -370,14 +373,12 @@ func (s *AteWorkerService) teardownActor(ctx context.Context, id string, actorDi
}
// Kill the CH process ateworker launched.
- if ra.chCmd != nil && ra.chCmd.Process != nil {
- _ = ra.chCmd.Process.Kill()
- _, _ = ra.chCmd.Process.Wait()
+ if ra.chProc != nil {
+ ra.chProc.Stop()
}
// Kill the virtiofsd (after CH, its only client).
- if ra.vfsdCmd != nil && ra.vfsdCmd.Process != nil {
- _ = ra.vfsdCmd.Process.Kill()
- _, _ = ra.vfsdCmd.Process.Wait()
+ if ra.vfsdProc != nil {
+ ra.vfsdProc.Stop()
}
}
diff --git a/cmd/ateworker-microvm/crash.go b/cmd/ateworker-microvm/crash.go
new file mode 100644
index 0000000000..d27c589ee8
--- /dev/null
+++ b/cmd/ateworker-microvm/crash.go
@@ -0,0 +1,81 @@
+//go:build linux
+
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package main
+
+import (
+ "context"
+ "fmt"
+ "log/slog"
+ "time"
+
+ "go.ate.dev/substrate/internal/ateworker/register"
+ "go.ate.dev/substrate/internal/proto/ateworkerpb"
+ "go.ate.dev/substrate/internal/resources"
+)
+
+// crashReportTimeout bounds how long a crash report keeps retrying.
+const crashReportTimeout = 5 * time.Minute
+
+// crashReporter is the slice of register.CrashReporter the service uses.
+type crashReporter interface {
+ ReportCrashed(ctx context.Context, actor resources.ActorAttribution, message string) error
+}
+
+var _ crashReporter = (*register.CrashReporter)(nil)
+
+// watchVM handles the actor's VMM or virtiofsd dying while nothing is tearing
+// them down: the actor is torn down here, since the control plane will not
+// reach this worker for it again, and then reported crashed.
+func (s *AteWorkerService) watchVM(attribution resources.ActorAttribution, actorDirs *ateworkerpb.ActorDirs, ra *runningActor) {
+ what, exit := "cloud-hypervisor", ""
+ select {
+ case <-ra.chProc.Done():
+ exit = ra.chProc.Exit()
+ case <-ra.vfsdProc.Done():
+ what, exit = "virtiofsd", ra.vfsdProc.Exit()
+ }
+ if ra.stopping.Load() || s.shuttingDown.Load() {
+ return
+ }
+ ctx := context.Background()
+ message := fmt.Sprintf("%s exited: %s", what, exit)
+ slog.WarnContext(ctx, "Micro-VM died", slog.String("actorUID", attribution.UID), slog.String("reason", message))
+
+ // Not reported under the actor's lock: a suspend in flight holds the
+ // actor's lease while it waits on that lock for its Checkpoint.
+ if !s.locks.Lock(ctx, attribution.UID) {
+ return
+ }
+ ours := s.runningVM(attribution.UID) == ra && !ra.stopping.Load()
+ if ours {
+ hosted := s.lookupActor(attribution.UID)
+ if err := s.terminateWorkload(ctx, attribution, actorDirs); err != nil {
+ slog.WarnContext(ctx, "Failed to tear down the dead micro-VM", slog.String("actorUID", attribution.UID), slog.Any("err", err))
+ }
+ // not measured first: the guest is gone
+ s.recordFinalIfEnded(ctx, hosted)
+ }
+ s.locks.Unlock(attribution.UID)
+ if !ours || s.crashReporter == nil {
+ return
+ }
+ reportCtx, cancel := context.WithTimeout(ctx, crashReportTimeout)
+ defer cancel()
+ if err := s.crashReporter.ReportCrashed(reportCtx, attribution, message); err != nil {
+ slog.ErrorContext(ctx, "Failed to report the actor crashed", slog.String("actorUID", attribution.UID), slog.Any("err", err))
+ }
+}
diff --git a/cmd/ateworker-microvm/crash_test.go b/cmd/ateworker-microvm/crash_test.go
new file mode 100644
index 0000000000..1b5d200c80
--- /dev/null
+++ b/cmd/ateworker-microvm/crash_test.go
@@ -0,0 +1,92 @@
+//go:build linux
+
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package main
+
+import (
+ "context"
+ "os/exec"
+ "testing"
+ "time"
+
+ "go.ate.dev/substrate/cmd/ateworker-microvm/internal/child"
+ "go.ate.dev/substrate/internal/actorlock"
+ "go.ate.dev/substrate/internal/resources"
+)
+
+type fakeCrashReporter struct {
+ reports chan string
+}
+
+func (f *fakeCrashReporter) ReportCrashed(_ context.Context, _ resources.ActorAttribution, message string) error {
+ f.reports <- message
+ return nil
+}
+
+// fakeVM is a running actor whose VMM exits at once and whose virtiofsd runs
+// until the test ends.
+func fakeVM(t *testing.T) *runningActor {
+ t.Helper()
+ chProc, err := child.Start(exec.Command("sh", "-c", "exit 3"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ vfsdProc, err := child.Start(exec.Command("sleep", "30"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(vfsdProc.Stop)
+ return &runningActor{chProc: chProc, vfsdProc: vfsdProc}
+}
+
+func TestWatchVMIgnoresTeardown(t *testing.T) {
+ reporter := &fakeCrashReporter{reports: make(chan string, 1)}
+ s := &AteWorkerService{locks: actorlock.New(), actors: map[string]*hostedActor{}, crashReporter: reporter}
+ ra := fakeVM(t)
+ ra.stopping.Store(true)
+ s.watchVM(resources.ActorAttribution{UID: "actor-a"}, nil, ra)
+ select {
+ case msg := <-reporter.reports:
+ t.Fatalf("reported %q for a VM being torn down", msg)
+ default:
+ }
+}
+
+func TestWatchVMIgnoresAReplacedVM(t *testing.T) {
+ reporter := &fakeCrashReporter{reports: make(chan string, 1)}
+ attribution := resources.ActorAttribution{UID: "actor-a"}
+ // the actor is hosted with a different VM than the one that died
+ s := &AteWorkerService{
+ locks: actorlock.New(),
+ actors: map[string]*hostedActor{"actor-a": {attribution: attribution, vm: &runningActor{}}},
+ crashReporter: reporter,
+ }
+ done := make(chan struct{})
+ go func() {
+ s.watchVM(attribution, nil, fakeVM(t))
+ close(done)
+ }()
+ select {
+ case <-done:
+ case <-time.After(10 * time.Second):
+ t.Fatal("watchVM did not return")
+ }
+ select {
+ case msg := <-reporter.reports:
+ t.Fatalf("reported %q for a VM the actor no longer runs", msg)
+ default:
+ }
+}
diff --git a/cmd/ateworker-microvm/internal/ch/restorefds.go b/cmd/ateworker-microvm/internal/ch/restorefds.go
index 886bc6e889..26abbc02d9 100644
--- a/cmd/ateworker-microvm/internal/ch/restorefds.go
+++ b/cmd/ateworker-microvm/internal/ch/restorefds.go
@@ -29,6 +29,8 @@ import (
"time"
"golang.org/x/sys/unix"
+
+ "go.ate.dev/substrate/cmd/ateworker-microvm/internal/child"
)
// RestoredNet identifies one fd-backed network device in a snapshot and the
@@ -59,7 +61,7 @@ type LaunchVMMOptions struct {
// LaunchVMM starts a cloud-hypervisor process with only an api-socket (no VM)
// and waits until it answers. Use Client.RestoreWithNetFDs to then restore a
// snapshot that has fd-backed net devices. The caller owns cmd.
-func LaunchVMM(ctx context.Context, o LaunchVMMOptions) (*exec.Cmd, *Client, error) {
+func LaunchVMM(ctx context.Context, o LaunchVMMOptions) (*child.Process, *Client, error) {
if o.APISocket == "" {
return nil, nil, fmt.Errorf("LaunchVMMOptions.APISocket is required")
}
@@ -74,16 +76,16 @@ func LaunchVMM(ctx context.Context, o LaunchVMMOptions) (*exec.Cmd, *Client, err
cmd.Stdout = o.Stdout
cmd.Stderr = o.Stderr
cmd.SysProcAttr = o.SysProcAttr
- if err := cmd.Start(); err != nil {
+ proc, err := child.Start(cmd)
+ if err != nil {
return nil, nil, fmt.Errorf("while starting cloud-hypervisor: %w", err)
}
client := NewClient(o.APISocket)
if _, err := client.WaitReady(ctx, 15*time.Second); err != nil {
- _ = cmd.Process.Kill()
- _, _ = cmd.Process.Wait()
+ proc.Stop()
return nil, nil, fmt.Errorf("while waiting for VMM api-socket: %w", err)
}
- return cmd, client, nil
+ return proc, client, nil
}
// RestoreWithNetFDs issues vm.restore for a snapshot dir, passing fresh tap FDs
diff --git a/cmd/ateworker-microvm/internal/child/child.go b/cmd/ateworker-microvm/internal/child/child.go
new file mode 100644
index 0000000000..e5cd4d81c8
--- /dev/null
+++ b/cmd/ateworker-microvm/internal/child/child.go
@@ -0,0 +1,86 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package child runs the long-lived subprocesses ateworker-microvm owns, and reaps
+// each as soon as it exits.
+package child
+
+import (
+ "os"
+ "os/exec"
+ "sync/atomic"
+
+ "go.ate.dev/substrate/internal/pidfd"
+)
+
+// Process is a started subprocess. A goroutine waits on it from the start, so
+// it never lingers as a zombie, whether it is stopped or dies on its own.
+type Process struct {
+ cmd *exec.Cmd
+ done chan struct{}
+ state *os.ProcessState
+ stopping atomic.Bool
+}
+
+// Start starts cmd and begins waiting on it.
+func Start(cmd *exec.Cmd) (*Process, error) {
+ if err := cmd.Start(); err != nil {
+ return nil, err
+ }
+ exited, err := pidfd.Exited(cmd.Process.Pid)
+ if err != nil {
+ _ = cmd.Process.Kill()
+ _, _ = cmd.Process.Wait()
+ return nil, err
+ }
+ p := &Process{cmd: cmd, done: make(chan struct{})}
+ go func() {
+ // Wait only once it has exited, so no thread blocks in wait for the
+ // life of the process. Process.Wait, not cmd.Wait: cmd.Wait also waits
+ // for the output pipes to close, and a grandchild (virtiofsd's sandbox)
+ // can hold them open.
+ <-exited
+ p.state, _ = cmd.Process.Wait()
+ close(p.done)
+ }()
+ return p, nil
+}
+
+// Done is closed once the process has exited and been reaped.
+func (p *Process) Done() <-chan struct{} {
+ return p.done
+}
+
+// Stop kills the process and waits until it is reaped. After Stop, Stopped
+// reports true, so a watcher on Done can tell this exit from a crash.
+func (p *Process) Stop() {
+ p.stopping.Store(true)
+ // Signals go through a pidfd, so a process already reaped is not
+ // mistaken for a reused pid.
+ _ = p.cmd.Process.Kill()
+ <-p.done
+}
+
+// Stopped reports whether Stop was called.
+func (p *Process) Stopped() bool {
+ return p.stopping.Load()
+}
+
+// Exit describes how the process exited. Only valid once Done is closed.
+func (p *Process) Exit() string {
+ if p.state == nil {
+ return "exited"
+ }
+ return p.state.String()
+}
diff --git a/cmd/ateworker-microvm/internal/child/child_test.go b/cmd/ateworker-microvm/internal/child/child_test.go
new file mode 100644
index 0000000000..3d0257d53c
--- /dev/null
+++ b/cmd/ateworker-microvm/internal/child/child_test.go
@@ -0,0 +1,57 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package child
+
+import (
+ "os/exec"
+ "testing"
+ "time"
+)
+
+func TestReapedWhenItDiesOnItsOwn(t *testing.T) {
+ p, err := Start(exec.Command("sh", "-c", "kill -KILL $$"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ select {
+ case <-p.Done():
+ case <-time.After(10 * time.Second):
+ t.Fatal("process was not reaped")
+ }
+ if p.Stopped() {
+ t.Error("Stopped() = true for a process that died on its own")
+ }
+ if got, want := p.Exit(), "signal: killed"; got != want {
+ t.Errorf("Exit() = %q, want %q", got, want)
+ }
+}
+
+func TestStop(t *testing.T) {
+ p, err := Start(exec.Command("sleep", "30"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ p.Stop()
+ select {
+ case <-p.Done():
+ default:
+ t.Fatal("Stop returned before the process was reaped")
+ }
+ if !p.Stopped() {
+ t.Error("Stopped() = false after Stop")
+ }
+ // a second Stop on a reaped process must not block or signal anything
+ p.Stop()
+}
diff --git a/cmd/ateworker-microvm/internal/kata/overlay_linux.go b/cmd/ateworker-microvm/internal/kata/overlay_linux.go
index d989644ba8..ddf2ac6e85 100644
--- a/cmd/ateworker-microvm/internal/kata/overlay_linux.go
+++ b/cmd/ateworker-microvm/internal/kata/overlay_linux.go
@@ -39,6 +39,7 @@ import (
"time"
specs "github.com/opencontainers/runtime-spec/specs-go"
+ "go.ate.dev/substrate/cmd/ateworker-microvm/internal/child"
"go.ate.dev/substrate/cmd/ateworker-microvm/internal/third_party/kata/agentpb"
"go.ate.dev/substrate/internal/ocispec"
"golang.org/x/sys/unix"
@@ -67,21 +68,21 @@ func SharedDir(id string) string {
func VirtiofsdSocketPath(id string) string { return filepath.Join(VMDir(id), "virtiofsd.sock") }
// UpperWorkDirs returns the HOST overlay upperdir and workdir for one container
-// under the actor's rootfs-upper base dir: SIBLING directories, /fs and
-// /work. Both properties are load-bearing — the kernel requires upperdir
+// under the actor's rootfs-upper base dir: SIBLING directories, /fs and
+// /work. Both properties are load-bearing — the kernel requires upperdir
// and workdir on the same filesystem and rejects a nested workdir. The
// snapshot only carries each upperdir's contents (see
// cmd/ateworker-microvm/rootfsupper.go); ateworker creates this layout itself.
// Covered by regression tests.
-func UpperWorkDirs(upperBase, containerID string) (upper, work string) {
- return filepath.Join(upperBase, containerID, "fs"), filepath.Join(upperBase, containerID, "work")
+func UpperWorkDirs(upperBase, name string) (upper, work string) {
+ return filepath.Join(upperBase, name, "fs"), filepath.Join(upperBase, name, "work")
}
// MkdirUpperWorkDirs creates UpperWorkDirs under upperBase, which must exist.
-// Only root uses and the workdir, so they are 0700. The upperdir is
+// Only root uses and the workdir, so they are 0700. The upperdir is
// 0755: overlayfs presents its mode as the container's /.
-func MkdirUpperWorkDirs(upperBase, containerID string) (upper, work string, err error) {
- upper, work = UpperWorkDirs(upperBase, containerID)
+func MkdirUpperWorkDirs(upperBase, name string) (upper, work string, err error) {
+ upper, work = UpperWorkDirs(upperBase, name)
for _, d := range []struct {
path string
mode os.FileMode
@@ -140,7 +141,7 @@ func virtiofsdArgs(o VirtiofsdOptions) []string {
// StartVirtiofsd launches virtiofsd in find-paths migration mode serving o.SharedDir
// on o.SocketPath, and waits for the socket to appear. The returned cmd outlives the
// caller's ctx (CH demand-pages from it under the running VM); the caller owns it.
-func StartVirtiofsd(ctx context.Context, o VirtiofsdOptions) (*exec.Cmd, error) {
+func StartVirtiofsd(ctx context.Context, o VirtiofsdOptions) (*child.Process, error) {
bin := o.Binary
if bin == "" {
bin = "virtiofsd"
@@ -150,15 +151,15 @@ func StartVirtiofsd(ctx context.Context, o VirtiofsdOptions) (*exec.Cmd, error)
cmd.Stdout = o.Log
cmd.Stderr = o.Log
cmd.SysProcAttr = o.SysProcAttr
- if err := cmd.Start(); err != nil {
+ proc, err := child.Start(cmd)
+ if err != nil {
return nil, fmt.Errorf("starting virtiofsd: %w", err)
}
if err := waitForSocket(ctx, o.SocketPath, virtiofsdSocketTimeout); err != nil {
- _ = cmd.Process.Kill()
- _, _ = cmd.Process.Wait()
+ proc.Stop()
return nil, err
}
- return cmd, nil
+ return proc, nil
}
const (
@@ -240,7 +241,7 @@ func StageImageVolume(ctx context.Context, src, id, cid, volumeName string) erro
}
// StageMergedRootfs mounts overlay(lower = the OCI image bundle rootfs, upper/work =
-// the actor's host rootfs-upper dirs for cid) at SharedDir(restoreID)//rootfs —
+// the actor's host rootfs-upper dirs for name) at SharedDir(restoreID)//rootfs —
// the merged tree the ONE virtiofsd serves and the guest runs the container on
// directly. The host kernel owns the overlay (the canonical ext4-upper case: no
// special mount options, whiteouts/opaque markers are ordinary trusted.overlay.*
@@ -249,12 +250,12 @@ func StageImageVolume(ctx context.Context, src, id, cid, volumeName string) erro
// The merged path is identical on every node — find-paths migration re-opens the
// guest's open files by path — given a deterministic image unpack plus the upper
// re-materialized from the snapshot tar (see cmd/ateworker-microvm/rootfsupper.go).
-func StageMergedRootfs(ctx context.Context, bundleRootfs, upperBase, restoreID, cid string) error {
- if cid == "" {
- return fmt.Errorf("StageMergedRootfs: empty container id")
+func StageMergedRootfs(ctx context.Context, bundleRootfs, upperBase, name, restoreID, cid string) error {
+ if name == "" || cid == "" {
+ return fmt.Errorf("StageMergedRootfs: empty container name or id")
}
dst := filepath.Join(SharedDir(restoreID), cid, "rootfs")
- _, work := UpperWorkDirs(upperBase, cid)
+ _, work := UpperWorkDirs(upperBase, name)
// Drop any stale mount first (lazy if busy), then ensure clean mountpoints.
Unmount(dst)
// The workdir is scratch: wipe it so a volatile mount is never refused by a
@@ -265,7 +266,7 @@ func StageMergedRootfs(ctx context.Context, bundleRootfs, upperBase, restoreID,
if err := os.MkdirAll(dst, 0o755); err != nil {
return fmt.Errorf("creating %q: %w", dst, err)
}
- upper, work, err := MkdirUpperWorkDirs(upperBase, cid)
+ upper, work, err := MkdirUpperWorkDirs(upperBase, name)
if err != nil {
return err
}
diff --git a/cmd/ateworker-microvm/main.go b/cmd/ateworker-microvm/main.go
index 0bb2cd9a42..930ae4b93d 100644
--- a/cmd/ateworker-microvm/main.go
+++ b/cmd/ateworker-microvm/main.go
@@ -127,7 +127,8 @@ func do(ctx context.Context) error {
const serviceName = "ateworker-microvm"
// Export through atelet's node-local relay when it is there, so telemetry
// never touches the worker pod's network. A nil conn means it is not, and
- // the providers fall back to dialing the collector directly.
+ // the providers fall back to OTEL_EXPORTER_OTLP_ENDPOINT, and export
+ // nothing without one.
//
// A relay that cannot be dialed is logged rather than fatal, matching both
// ends of the same decision: Dial already treats an absent socket as a
@@ -137,7 +138,7 @@ func do(ctx context.Context) error {
// over its telemetry route would turn a misconfigured flag into an outage.
relayConn, err := otlprelay.Dial(ctx, *otlpRelaySocket)
if err != nil {
- slog.ErrorContext(ctx, "Failed to connect to the OTLP relay; exporting telemetry directly over the pod network",
+ slog.ErrorContext(ctx, "Failed to connect to the OTLP relay; falling back to the OTLP endpoint, if one is set",
slog.String("socket", *otlpRelaySocket), slog.Any("err", err))
}
if relayConn != nil {
@@ -149,7 +150,7 @@ func do(ctx context.Context) error {
Sampling: serverboot.ResolveTraceSampling(ctx, serverboot.ParentRatioSampling(serverboot.ControlPlaneTraceRatio)),
ExporterConn: relayConn,
// So the spans say which path they took, including when relayConn is nil
- // because the dial above failed and this ateworker is exporting directly.
+ // because the dial above failed and this ateworker exports directly or not at all.
RelayCapable: true,
})
if err != nil {
@@ -172,7 +173,7 @@ func do(ctx context.Context) error {
if err != nil {
serverboot.Fatal(ctx, "Failed to initialize logging", err)
}
- // Nil when the exporter does not include otlp.
+ // Nil when there is nothing to export.
if lp != nil {
defer serverboot.ShutdownProvider("LoggerProvider", lp.Shutdown)
}
@@ -278,16 +279,22 @@ func do(ctx context.Context) error {
// Report retries every failure it can outlast, including the window before
// the Worker record exists; anything that reaches here is a misconfiguration
// no restart-in-place will fix.
+ reportConfig := register.ReportConfig{
+ SocketPath: nodepath.WorkerSupportSocket,
+ CredentialBundlePath: tunnelConfig.CredentialBundle,
+ TrustBundlePath: tunnelConfig.TrustBundle,
+ AteletSPIFFEID: tunnelConfig.BrokerIdentity,
+ Actors: *maxActors,
+ SandboxClass: string(atev1alpha1.SandboxClassMicroVM),
+ RuntimesFile: nodepath.WorkerSandboxRuntimesPath(*podUID),
+ }
+ if reporter, err := register.NewCrashReporter(reportConfig); err != nil {
+ slog.ErrorContext(ctx, "Crashed actors will not be reported", slog.Any("err", err))
+ } else {
+ ateworkerService.crashReporter = reporter
+ }
go func() {
- err := register.Report(ctx, register.ReportConfig{
- SocketPath: nodepath.WorkerSupportSocket,
- CredentialBundlePath: tunnelConfig.CredentialBundle,
- TrustBundlePath: tunnelConfig.TrustBundle,
- AteletSPIFFEID: tunnelConfig.BrokerIdentity,
- Actors: *maxActors,
- SandboxClass: string(atev1alpha1.SandboxClassMicroVM),
- RuntimesFile: nodepath.WorkerSandboxRuntimesPath(*podUID),
- })
+ err := register.Report(ctx, reportConfig)
if err != nil && ctx.Err() == nil {
serverboot.Fatal(ctx, "Failed to register worker", err)
}
@@ -344,6 +351,9 @@ type AteWorkerService struct {
locks *actorlock.Locks
// Tracks lifecycle RPCs for shutdown cancellation and draining.
inFlight *actorlock.InFlight
+ // crashReporter reports actors whose micro-VM died. Nil if it could not
+ // be set up.
+ crashReporter crashReporter
// shuttingDown is set once SIGTERM has been received. While true, new workload
// RPCs are rejected with codes.Unavailable so the control plane reschedules.
diff --git a/cmd/ateworker-microvm/restore.go b/cmd/ateworker-microvm/restore.go
index ca50611339..09798cc3ff 100644
--- a/cmd/ateworker-microvm/restore.go
+++ b/cmd/ateworker-microvm/restore.go
@@ -312,14 +312,13 @@ func (s *AteWorkerService) restoreMemoryFidelity(ctx context.Context, p actorBoo
return err
}
defer leaf.Close()
- vfsdCmd, err := s.stageMergedRootfs(ctx, rr, actorUID, p.actorDirs, ctrs, containers, leaf.SysProcAttr())
+ vfsdProc, err := s.stageMergedRootfs(ctx, rr, actorUID, p.actorDirs, ctrs, containers, leaf.SysProcAttr())
if err != nil {
return err
}
defer func() {
- if retErr != nil && vfsdCmd.Process != nil {
- _ = vfsdCmd.Process.Kill()
- _, _ = vfsdCmd.Process.Wait()
+ if retErr != nil {
+ vfsdProc.Stop()
}
}()
@@ -371,7 +370,7 @@ func (s *AteWorkerService) restoreMemoryFidelity(ctx context.Context, p actorBoo
// Relaunch CH and restore with the tap FDs attached (SCM_RIGHTS).
apiSocket := filepath.Join(kata.VMDir(actorUID), "clh-api-restore.sock")
tTap := time.Now()
- chCmd, client, err := ch.LaunchVMM(ctx, ch.LaunchVMMOptions{
+ chProc, client, err := ch.LaunchVMM(ctx, ch.LaunchVMMOptions{
Binary: rr.chBinary, APISocket: apiSocket, Stdout: slogWriter{ctx}, Stderr: slogWriter{ctx},
SysProcAttr: leaf.SysProcAttr(),
})
@@ -379,9 +378,8 @@ func (s *AteWorkerService) restoreMemoryFidelity(ctx context.Context, p actorBoo
return fmt.Errorf("while launching VMM for restore: %w", err)
}
defer func() {
- if retErr != nil && chCmd.Process != nil {
- _ = chCmd.Process.Kill()
- _, _ = chCmd.Process.Wait()
+ if retErr != nil {
+ chProc.Stop()
}
}()
// How guest RAM comes back depends on the VMM (see restoreMemMode), and the rest
@@ -491,7 +489,7 @@ func (s *AteWorkerService) restoreMemoryFidelity(ctx context.Context, p actorBoo
})
ra := &runningActor{
- chCmd: chCmd, vfsdCmd: vfsdCmd,
+ chProc: chProc, vfsdProc: vfsdProc,
apiSocket: apiSocket, baseID: srcID, restoreSourceDir: restoreDir,
preserveRestoreSource: preserveRestoreDir,
snapshotIsSelfContained: memMode == ch.MemRestoreEager,
@@ -505,14 +503,15 @@ func (s *AteWorkerService) restoreMemoryFidelity(ctx context.Context, p actorBoo
// connection dialed after resume: the restored guest's containers are alive, so
// ReadStdout/ReadStderr pick up where they left off.
attribution := p.actorAttribution()
- for _, c := range containers {
- s.startActorLogForwarding(guestAC, attribution, c.GetName(), c.GetName())
+ for _, c := range ctrs {
+ s.startActorLogForwarding(guestAC, attribution, c.id, c.name)
}
if err := s.tunnel.Activate(p.attribution(), s.sandboxDialer(p.actorUID), egress); err != nil {
return err
}
s.setRunningVM(actorUID, ra)
+ go s.watchVM(p.attribution(), p.actorDirs, ra)
// Publish the guest to GetWorkloadStats, past the last error return above
// for the same reason as in coldBootActor. Same client the forwarding above
diff --git a/cmd/ateworker-microvm/rootfsupper.go b/cmd/ateworker-microvm/rootfsupper.go
index 5097135c21..23219183c7 100644
--- a/cmd/ateworker-microvm/rootfsupper.go
+++ b/cmd/ateworker-microvm/rootfsupper.go
@@ -59,7 +59,7 @@ import (
)
// rootfsUpperTarFile is the snapshot file holding the tar of one container's
-// rootfs upperdir (/fs under rootfsUpperDir, see kata.UpperWorkDirs).
+// rootfs upperdir (/fs under rootfsUpperDir, see kata.UpperWorkDirs).
// Entries are relative to the upperdir, plus a root entry: overlayfs presents
// the upperdir's own mode, ownership, and xattrs as the container's /. The workdir is never archived: with
// index=off pinned on the mount (see kata.StageMergedRootfs) a restored
diff --git a/cmd/ateworker-microvm/run.go b/cmd/ateworker-microvm/run.go
index ce61de6c12..6bac045d13 100644
--- a/cmd/ateworker-microvm/run.go
+++ b/cmd/ateworker-microvm/run.go
@@ -23,12 +23,13 @@ import (
"io/fs"
"log/slog"
"os"
- "os/exec"
"path/filepath"
"runtime"
+ "sync/atomic"
"syscall"
"time"
+ "go.ate.dev/substrate/cmd/ateworker-microvm/internal/child"
"go.ate.dev/substrate/internal/workerstats"
"go.ate.dev/substrate/internal/ateworker/actornet"
@@ -59,13 +60,17 @@ type runningActor struct {
baseID string
// ateworker owns this CH process (booted at Run or relaunched at Restore).
- chCmd *exec.Cmd
- // vfsdCmd is the virtiofsd serving the unified share (merged rootfs overlay,
+ chProc *child.Process
+ // vfsdProc is the virtiofsd serving the unified share (merged rootfs overlay,
// durable-dir volumes, and CSI volumes). ateworker owns it; teardownActor
// kills it after the CH process.
- vfsdCmd *exec.Cmd
+ vfsdProc *child.Process
// apiSocket is the CH api-socket for this ateworker-owned VMM.
apiSocket string
+ // stopping is set once teardown starts, so watchVM can tell an expected
+ // exit from a crash: shutting the VM down makes cloud-hypervisor exit, and
+ // virtiofsd exits once its only client is gone.
+ stopping atomic.Bool
// restoreSourceDir is the snapshot dir this actor was OnDemand-restored from
// (CH demand-pages its guest RAM from it). Set when restored via OnDemand.
@@ -166,23 +171,22 @@ const minGuestMemMiB = 128
// workloads to signal and wait on, and it must name what the guest actually
// runs: a container the agent does not know is rejected with InvalidContainerId,
// and graceful shutdown then gives up without ever reaching the workload.
-//
-// Containers run under their bare name.
func workloadIDs(ctrs []actorContainer) []string {
ids := make([]string, 0, len(ctrs))
for _, c := range ctrs {
- ids = append(ids, c.name)
+ ids = append(ids, c.id)
}
return ids
}
// actorContainer is one of the actor's containers prepared for the shared micro-VM:
-// its name (also the kata containerID + the merged rootfs's find-paths subdir), its
+// its name, its kata container ID (also the merged rootfs's find-paths subdir), its
// host OCI bundle and the bundle rootfs that backs the overlay lower, and its OCI
// spec. The writable upper is a host directory (see rootfsupper.go); the host kernel
// merges the two.
type actorContainer struct {
name string
+ id string // see ocispec.GuestContainerID
bundle string
bundleRootfs string
// spec is the container's OCI spec shaped for micro-VM execution.
@@ -489,20 +493,19 @@ func (s *AteWorkerService) coldBootActor(ctx context.Context, p actorBootParams)
return err
}
defer leaf.Close()
- vfsdCmd, err := s.stageMergedRootfs(ctx, rr, actorUID, p.actorDirs, ctrs, containers, leaf.SysProcAttr())
+ vfsdProc, err := s.stageMergedRootfs(ctx, rr, actorUID, p.actorDirs, ctrs, containers, leaf.SysProcAttr())
if err != nil {
return err
}
defer func() {
- if retErr != nil && vfsdCmd.Process != nil {
- _ = vfsdCmd.Process.Kill()
- _, _ = vfsdCmd.Process.Wait()
+ if retErr != nil {
+ vfsdProc.Stop()
}
}()
// Launch a bare VMM (CH + api-socket); ateworker owns this process for teardown.
apiSocket := filepath.Join(kata.VMDir(actorUID), "clh-api.sock")
- chCmd, client, err := ch.LaunchVMM(ctx, ch.LaunchVMMOptions{
+ chProc, client, err := ch.LaunchVMM(ctx, ch.LaunchVMMOptions{
Binary: rr.chBinary,
APISocket: apiSocket,
Stdout: slogWriter{ctx},
@@ -513,9 +516,8 @@ func (s *AteWorkerService) coldBootActor(ctx context.Context, p actorBootParams)
return fmt.Errorf("while launching VMM: %w", err)
}
defer func() {
- if retErr != nil && chCmd.Process != nil {
- _ = chCmd.Process.Kill()
- _, _ = chCmd.Process.Wait()
+ if retErr != nil {
+ chProc.Stop()
}
}()
@@ -601,21 +603,21 @@ func (s *AteWorkerService) coldBootActor(ctx context.Context, p actorBootParams)
slog.Duration("wakeup_probe", time.Since(tContainers)),
slog.Duration("since_boot", time.Since(tBooted)))
- ra := &runningActor{chCmd: chCmd, vfsdCmd: vfsdCmd, apiSocket: apiSocket, baseID: actorUID, guestAgent: ac, workloadIDs: workloadIDs(ctrs)}
+ ra := &runningActor{chProc: chProc, vfsdProc: vfsdProc, apiSocket: apiSocket, baseID: actorUID, guestAgent: ac, workloadIDs: workloadIDs(ctrs)}
if err := s.tunnel.Activate(p.attribution(), s.sandboxDialer(p.actorUID), egress); err != nil {
return err
}
s.setRunningVM(actorUID, ra)
+ go s.watchVM(p.attribution(), p.actorDirs, ra)
// Forward each container's stdout/stderr into the pod logs, keyed by the
- // container id (== the name; see StartRootfsContainer). The goroutines read
- // over ac for the actor's lifetime and exit (io.EOF) when teardownActor
- // closes ac.
+ // container id (see StartRootfsContainer). The goroutines read over ac for
+ // the actor's lifetime and exit (io.EOF) when teardownActor closes ac.
workloadIDs := make([]string, 0, len(ctrs))
attribution := p.actorAttribution()
for _, c := range ctrs {
- s.startActorLogForwarding(ac, attribution, c.name, c.name)
- workloadIDs = append(workloadIDs, c.name)
+ s.startActorLogForwarding(ac, attribution, c.id, c.name)
+ workloadIDs = append(workloadIDs, c.id)
}
// Publish the guest to GetWorkloadStats, past every error return above: a
@@ -644,7 +646,7 @@ func (s *AteWorkerService) buildActorContainers(actorUID string, actorDirs *atew
cn := c.GetName()
bundle := ociBundlePath(actorDirs, cn)
spec := ocispec.Build(ocispec.Options{ActorUID: actorUID, ActorDirs: actorDirs, Container: c})
- if err := ocispec.ShapeMicroVM(spec, ocispec.MicroVMOptions{ActorDirs: actorDirs, ContainerID: cn}); err != nil {
+ if err := ocispec.ShapeMicroVM(spec, ocispec.MicroVMOptions{ActorDirs: actorDirs, ContainerName: cn}); err != nil {
return nil, fmt.Errorf("while shaping the OCI spec for %q: %w", cn, err)
}
// Compose the bundle rootfs from the node's cached image layers (an
@@ -663,6 +665,7 @@ func (s *AteWorkerService) buildActorContainers(actorUID string, actorDirs *atew
}
ctrs[i] = actorContainer{
name: cn,
+ id: ocispec.GuestContainerID(cn),
bundle: bundle,
bundleRootfs: bundleRootfs,
spec: spec,
@@ -681,15 +684,15 @@ func (s *AteWorkerService) buildActorContainers(actorUID string, actorDirs *atew
// wipes SharedDir) and stageRootfsUpper (which owns the upper contents). The returned virtiofsd cmd outlives this call (CH
// demand-pages from it); the caller owns it (tracked on runningActor, killed
// in teardownActor).
-func (s *AteWorkerService) stageMergedRootfs(ctx context.Context, rr resolvedRuntime, id string, actorDirs *ateworkerpb.ActorDirs, ctrs []actorContainer, containers []*ateworkerpb.Container, procAttr *syscall.SysProcAttr) (*exec.Cmd, error) {
+func (s *AteWorkerService) stageMergedRootfs(ctx context.Context, rr resolvedRuntime, id string, actorDirs *ateworkerpb.ActorDirs, ctrs []actorContainer, containers []*ateworkerpb.Container, procAttr *syscall.SysProcAttr) (*child.Process, error) {
upperBase := rootfsUpperDir(actorDirs)
for _, c := range ctrs {
- if err := kata.StageMergedRootfs(ctx, c.bundleRootfs, upperBase, id, c.name); err != nil {
+ if err := kata.StageMergedRootfs(ctx, c.bundleRootfs, upperBase, c.name, id, c.id); err != nil {
return nil, fmt.Errorf("while staging merged rootfs for %q: %w", c.name, err)
}
for _, vm := range c.imageMounts {
src := imagecache.ImageVolumeMountPath(c.bundle, vm.GetName())
- if err := kata.StageImageVolume(ctx, src, id, c.name, vm.GetName()); err != nil {
+ if err := kata.StageImageVolume(ctx, src, id, c.id, vm.GetName()); err != nil {
return nil, fmt.Errorf("while staging image volume %q for %q: %w", vm.GetName(), c.name, err)
}
}
@@ -710,7 +713,7 @@ func (s *AteWorkerService) stageMergedRootfs(ctx context.Context, rr resolvedRun
}
}
vfsdLog, _ := os.OpenFile(virtiofsdLogPath(id), os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0o600)
- vfsdCmd, err := kata.StartVirtiofsd(ctx, kata.VirtiofsdOptions{
+ vfsdProc, err := kata.StartVirtiofsd(ctx, kata.VirtiofsdOptions{
Binary: rr.virtiofsd,
SocketPath: kata.VirtiofsdSocketPath(id),
SharedDir: kata.SharedDir(id),
@@ -720,7 +723,7 @@ func (s *AteWorkerService) stageMergedRootfs(ctx context.Context, rr resolvedRun
if err != nil {
return nil, fmt.Errorf("while starting virtiofsd: %w", err)
}
- return vfsdCmd, nil
+ return vfsdProc, nil
}
// kernelParams returns the guest kernel params
@@ -892,19 +895,19 @@ func (s *AteWorkerService) startActorContainers(ctx context.Context, ac *kata.Ag
}
// startRootfsContainer brings up one container on its host-merged rootfs (the
-// stock kata flow: create + start against shared//rootfs). On failure it
+// stock kata flow: create + start against shared//rootfs). On failure it
// dumps the guest's view of the shared tree.
//
// Its spec binds every declared volume at its mount path.
func (s *AteWorkerService) startRootfsContainer(ctx context.Context, ac *kata.AgentClient, vsockPath string, c actorContainer) error {
cCtx, cancel := context.WithTimeout(ctx, 30*time.Second)
- err := ac.StartRootfsContainer(cCtx, c.name, c.spec)
+ err := ac.StartRootfsContainer(cCtx, c.id, c.spec)
cancel()
if err != nil {
if s.guestDebug {
dump := kata.DebugConsoleDump(ctx, vsockPath,
"echo '== shared/containers =='; ls -la /run/kata-containers/shared/containers/ 2>&1 | head -40; "+
- "echo '== rootfs =='; ls /run/kata-containers/shared/containers/"+c.name+"/rootfs/ 2>&1 | head; "+
+ "echo '== rootfs =='; ls /run/kata-containers/shared/containers/"+c.id+"/rootfs/ 2>&1 | head; "+
"echo '== mounts =='; grep -E 'kata|virtiofs' /proc/mounts 2>&1")
slog.ErrorContext(ctx, "rootfs container failed; dump", slog.String("container", c.name), slog.String("dump", dump))
}
diff --git a/cmd/ateworker-microvm/run_test.go b/cmd/ateworker-microvm/run_test.go
index 48a166324c..ee6b71d55e 100644
--- a/cmd/ateworker-microvm/run_test.go
+++ b/cmd/ateworker-microvm/run_test.go
@@ -21,6 +21,7 @@ import (
"encoding/json"
"errors"
"net"
+ "os"
"path/filepath"
"runtime"
"slices"
@@ -29,6 +30,7 @@ import (
"time"
"go.ate.dev/substrate/cmd/ateworker-microvm/internal/kata"
+ "go.ate.dev/substrate/internal/proto/ateworkerpb"
)
// A vsock socket that has gone missing means cloud-hypervisor stopped the VM
@@ -201,9 +203,20 @@ func TestBuildVMConfigConsole(t *testing.T) {
// does not know with InvalidContainerId — which aborts the whole graceful
// shutdown, so a stale id here silently costs the guest its clean exit.
func TestWorkloadIDs(t *testing.T) {
- ctrs := []actorContainer{{name: "counter"}, {name: "sidecar"}}
+ dirs := &ateworkerpb.ActorDirs{OciBundleDir: t.TempDir()}
+ var containers []*ateworkerpb.Container
+ for _, name := range []string{"counter", "w"} {
+ if err := os.MkdirAll(filepath.Join(ociBundlePath(dirs, name), "rootfs"), 0o755); err != nil {
+ t.Fatal(err)
+ }
+ containers = append(containers, &ateworkerpb.Container{Name: name, ContainerSpec: &ateworkerpb.ContainerSpec{Args: []string{"/app"}}})
+ }
+ ctrs, err := (&AteWorkerService{}).buildActorContainers("uid", dirs, containers)
+ if err != nil {
+ t.Fatal(err)
+ }
got := workloadIDs(ctrs)
- if want := []string{"counter", "sidecar"}; !slices.Equal(got, want) {
+ if want := []string{"c_counter", "c_w"}; !slices.Equal(got, want) {
t.Errorf("workloadIDs() = %v, want %v", got, want)
}
}
diff --git a/cmd/benchmarking/boomer-worker/main.go b/cmd/benchmarking/boomer-worker/main.go
index c36ca3790a..f07b6d5ed2 100644
--- a/cmd/benchmarking/boomer-worker/main.go
+++ b/cmd/benchmarking/boomer-worker/main.go
@@ -37,6 +37,7 @@ import (
// Register user classes via init():
"go.ate.dev/substrate/internal/benchmarking/boomer/agentsession"
+ _ "go.ate.dev/substrate/internal/benchmarking/boomer/ateenv"
_ "go.ate.dev/substrate/internal/benchmarking/boomer/glutton"
_ "go.ate.dev/substrate/internal/benchmarking/boomer/resumepaused"
_ "go.ate.dev/substrate/internal/benchmarking/boomer/sweperf"
diff --git a/cmd/benchmarking/isolate/fake-atelet/herder.go b/cmd/benchmarking/isolate/fake-atelet/herder.go
index 1b58808abe..7cb2b1ef29 100644
--- a/cmd/benchmarking/isolate/fake-atelet/herder.go
+++ b/cmd/benchmarking/isolate/fake-atelet/herder.go
@@ -31,6 +31,7 @@ import (
"go.ate.dev/substrate/internal/resources"
"go.ate.dev/substrate/pkg/proto/ateapipb"
"google.golang.org/grpc"
+ "google.golang.org/protobuf/proto"
)
// placeholderObject is the one object written under each snapshot URI.
@@ -195,7 +196,12 @@ func (h *herder) PromoteSnapshot(ctx context.Context, req *ateletpb.PromoteSnaps
if err := wait(ctx, start, h.delays.promoteSnapshot); err != nil {
return nil, err
}
- return &ateletpb.PromoteSnapshotResponse{}, nil
+ // Like atelet, report the snapshot durable and still kept on the node.
+ report := &ateletpb.SnapshotReport{Snapshot: proto.CloneOf(req.GetSnapshot())}
+ return &ateletpb.PromoteSnapshotResponse{State: &ateletpb.ActorSnapshotState{
+ LatestLocal: report,
+ LatestDurable: proto.CloneOf(report),
+ }}, nil
}
func (h *herder) Terminate(ctx context.Context, _ *ateletpb.TerminateRequest) (*ateletpb.TerminateResponse, error) {
diff --git a/cmd/benchmarking/isolate/fake-atelet/herder_test.go b/cmd/benchmarking/isolate/fake-atelet/herder_test.go
index a1b0a52bdf..1715d67b20 100644
--- a/cmd/benchmarking/isolate/fake-atelet/herder_test.go
+++ b/cmd/benchmarking/isolate/fake-atelet/herder_test.go
@@ -27,10 +27,12 @@ import (
"testing"
"time"
+ "github.com/google/go-cmp/cmp"
"go.ate.dev/substrate/internal/proto/ateletpb"
"go.ate.dev/substrate/pkg/proto/ateapipb"
"google.golang.org/grpc"
"google.golang.org/protobuf/proto"
+ "google.golang.org/protobuf/testing/protocmp"
)
const testSnapshotURI = "gs://bench-bucket/benchmark-workloads/sleep/atespaces/team-a/actors/0f9c1d2e-3b4a-4c5d-8e6f-7a8b9c0d1e2f/snapshots/snap-1"
@@ -204,6 +206,29 @@ func TestHerderWritesNothingForLocalCheckpoint(t *testing.T) {
}
}
+// ate-api-server checks that PromoteSnapshot reports the snapshot it asked for
+// durable, and fails the tag otherwise.
+func TestHerderPromoteSnapshotReportsSnapshotDurable(t *testing.T) {
+ snap := &ateletpb.Snapshot{
+ SnapshotUuid: "snap-1",
+ Generation: 2,
+ Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY,
+ Object: &ateletpb.ObjectStorage{SnapshotUri: testSnapshotURI},
+ }
+ h := &herder{storage: &recordingStorage{}}
+ resp, err := h.PromoteSnapshot(context.Background(), &ateletpb.PromoteSnapshotRequest{Snapshot: snap})
+ if err != nil {
+ t.Fatalf("PromoteSnapshot: %v", err)
+ }
+ want := &ateletpb.ActorSnapshotState{
+ LatestLocal: &ateletpb.SnapshotReport{Snapshot: snap},
+ LatestDurable: &ateletpb.SnapshotReport{Snapshot: snap},
+ }
+ if diff := cmp.Diff(want, resp.GetState(), protocmp.Transform()); diff != "" {
+ t.Errorf("PromoteSnapshot state mismatch (-want +got):\n%s", diff)
+ }
+}
+
func TestHerderFailsWhenPlaceholderCannotBeWritten(t *testing.T) {
h := &herder{storage: &recordingStorage{err: errors.New("bucket unavailable")}}
if _, err := h.Checkpoint(context.Background(), externalCheckpoint(testSnapshotURI)); err == nil {
diff --git a/cmd/benchmarking/isolate/fake-atelet/main.go b/cmd/benchmarking/isolate/fake-atelet/main.go
index bcd956cbf7..5916095b4a 100644
--- a/cmd/benchmarking/isolate/fake-atelet/main.go
+++ b/cmd/benchmarking/isolate/fake-atelet/main.go
@@ -45,6 +45,7 @@ import (
"go.ate.dev/substrate/internal/atelet"
"go.ate.dev/substrate/internal/credbundle"
"go.ate.dev/substrate/internal/installdefaults"
+ "go.ate.dev/substrate/internal/preview"
"go.ate.dev/substrate/internal/proto/ateletpb"
"go.ate.dev/substrate/internal/serverboot"
"go.ate.dev/substrate/internal/version"
@@ -76,6 +77,7 @@ var (
showVersion = pflag.Bool("version", false, "Print version and exit.")
logLevelFlag = pflag.String("log-level", "info", "Minimum log level: debug, info, warn, or error.")
+ previewFlags = pflag.StringSlice("preview", nil, "Preview gates to enable.")
)
func main() {
@@ -90,6 +92,9 @@ func main() {
if err := serverboot.SetLogLevel(*logLevelFlag); err != nil {
serverboot.Fatal(ctx, "Invalid --log-level", err)
}
+ if err := preview.Init(*previewFlags...); err != nil {
+ serverboot.Fatal(ctx, "Invalid --preview", err)
+ }
if err := applyTuningEnv(pflag.CommandLine, os.LookupEnv); err != nil {
serverboot.Fatal(ctx, "Invalid tuning environment", err)
}
diff --git a/cmd/benchmarking/isolate/fake-workersync/main.go b/cmd/benchmarking/isolate/fake-workersync/main.go
index 9a3829cae7..3601144610 100644
--- a/cmd/benchmarking/isolate/fake-workersync/main.go
+++ b/cmd/benchmarking/isolate/fake-workersync/main.go
@@ -48,6 +48,7 @@ import (
"go.ate.dev/substrate/cmd/benchmarking/isolate/internal/fakeworker"
"go.ate.dev/substrate/internal/benchmarking/boomer/boomerutil"
"go.ate.dev/substrate/internal/installdefaults"
+ "go.ate.dev/substrate/internal/preview"
"go.ate.dev/substrate/internal/serverboot"
"go.ate.dev/substrate/internal/version"
atev1alpha1 "go.ate.dev/substrate/pkg/api/v1alpha1"
@@ -80,6 +81,7 @@ var (
showVersion = pflag.Bool("version", false, "Print version and exit.")
logLevelFlag = pflag.String("log-level", "info", "Minimum log level: debug, info, warn, or error.")
+ previewFlags = pflag.StringSlice("preview", nil, "Preview gates to enable.")
)
func main() {
@@ -94,6 +96,9 @@ func main() {
if err := serverboot.SetLogLevel(*logLevelFlag); err != nil {
serverboot.Fatal(ctx, "Invalid --log-level", err)
}
+ if err := preview.Init(*previewFlags...); err != nil {
+ serverboot.Fatal(ctx, "Invalid --preview", err)
+ }
if err := fakeworker.ValidateRun(*run); err != nil {
serverboot.Fatal(ctx, "Invalid --run", err)
}
diff --git a/cmd/kubectl-ate/README.md b/cmd/kubectl-ate/README.md
index 4a0396b821..37c2233135 100644
--- a/cmd/kubectl-ate/README.md
+++ b/cmd/kubectl-ate/README.md
@@ -212,11 +212,8 @@ kubectl ate suspend actor my-actor -a --mode=sync
# A suspended actor without a node-local snapshot is already at its external snapshot, so it cannot be reverted
kubectl ate revert actor my-actor -a
-# Delete an actor (by default, requires the actor to be SUSPENDED or CRASHED).
+# Delete an actor, terminating workloads and detaching volumes.
kubectl ate delete actor my-actor -a
-
-# Delete an actor from any state (e.g. RUNNING), terminating workloads and detaching volumes.
-kubectl ate delete actor my-actor -a --any-state
```
### Actor Snapshots
diff --git a/cmd/kubectl-ate/internal/cmd/actor.go b/cmd/kubectl-ate/internal/cmd/actor.go
index 07e3932930..b83f54b574 100644
--- a/cmd/kubectl-ate/internal/cmd/actor.go
+++ b/cmd/kubectl-ate/internal/cmd/actor.go
@@ -48,7 +48,6 @@ var (
createActorTemplateFlag string
createActorTagFlag string
deleteActorAtespaceFlag string
- deleteActorAnyStateFlag bool
resumeActorAtespaceFlag string
suspendActorAtespaceFlag string
suspendActorModeFlag string
@@ -153,8 +152,7 @@ var deleteActorCmd = &cobra.Command{
actorRef := resources.ActorRef{Atespace: deleteActorAtespaceFlag, Name: args[0]}
_, err = c.ControlClient.DeleteActor(ctx, &ateapipb.DeleteActorRequest{
- Actor: actorRef.ToObjectRef(),
- AnyState: deleteActorAnyStateFlag,
+ Actor: actorRef.ToObjectRef(),
})
if err != nil {
return err
@@ -747,7 +745,6 @@ func init() {
createCmd.AddCommand(createActorCmd)
deleteActorCmd.Flags().StringVarP(&deleteActorAtespaceFlag, "atespace", "a", "", "the actor's atespace")
- deleteActorCmd.Flags().BoolVar(&deleteActorAnyStateFlag, "any-state", false, "delete the actor regardless of current state")
deleteCmd.AddCommand(deleteActorCmd)
resumeActorCmd.Flags().StringVarP(&resumeActorAtespaceFlag, "atespace", "a", "", "the actor's atespace")
diff --git a/cmd/objectstore-plugin/README.md b/cmd/objectstore-plugin/README.md
index 9e0b5d2fcb..a81ca91f2e 100644
--- a/cmd/objectstore-plugin/README.md
+++ b/cmd/objectstore-plugin/README.md
@@ -1,4 +1,58 @@
-# Object-store plugin upload memory
+# objectstore-plugin
+
+The in-tree object-store plugin. It serves the `objectstore.v1` API over a Unix
+socket as a sidecar: `node` next to atelet, moving snapshot files and sandbox
+assets, and `control` next to ate-api-server, copying and deleting whole
+snapshots. `ATE_STORAGE_BACKEND` picks GCS, the default, or S3. See
+`docs/architecture.md` for where it sits.
+
+## Per-actor snapshot credentials on GCS
+
+On GCS, `control` mode also serves a credential exchange, when
+`--exchange-listen` is set: the node plugin presents the JWT atelet handed it
+with a transfer and receives a GCS access token limited to that actor's
+snapshot prefixes, read-write on its own and read-only on the tag it was
+created from. The exchange is private to this plugin, defined in
+`internal/proto/objectstoreexchangepb`.
+
+The JWT is the one ate-api-server mints for atelet through
+`WorkerService.MintWorkerActorJwt`, with the audience named on the
+`actor_jwt` proto field. Its shape is defined in `internal/actoridjwt`, and
+the exchange accepts only the worker-for-actor kind, never an actor's own
+token. Its public keys come from the `jwks` key of the `actor-id-jwt-pool`
+Secret, read through `oidcjwt.FileKeySource`.
+
+Three packages behind the exchange know no storage provider and are what a
+provider's own plugin binary reuses. The rest is this binary's.
+
+Reusable by every provider:
+
+- `internal/oidcjwt`: verifies the JWT against the `jwks` file.
+- `internal/objectstoreplugin/actorlookup`: fetches the actor over the
+ loopback, refuses a recreated one.
+- `internal/objectstoreplugin/actorscope`: turns the actor's status into
+ grants, read-write on its own prefix, read-only on its tag.
+
+This binary's, GCS and S3:
+
+- `internal/proto/objectstoreexchangepb`: the private wire format between the
+ node and control sidecars.
+- `internal/objectstoreplugin/exchange`: the handler, and the `Issuer` interface
+ that is the seam between its backends.
+- `internal/objectstoreplugin/gcsboundary`: grants to Credential Access Boundary
+ rules.
+- `internal/objectstoreplugin/gcsissuer`: the GCS `Issuer`, STS exchange and
+ token cache.
+- `cmd/objectstore-plugin`: the binary and its wiring.
+
+S3 has no issuer: its node plugin ignores the JWT and transfers with the
+pod's own credentials, and the exchange answers `UNIMPLEMENTED` there. A
+provider other than these two brings its own binary in this repository, with
+its own exchange handler and proto shaped for its credential, its own issuer
+and storage client, and imports the three reusable packages. A binary outside
+the repository cannot, since they are under `internal/`.
+
+## Upload memory
The node plugin shares one upload memory budget across all snapshot requests.
`--upload-memory-budget` accepts a Kubernetes quantity and defaults to `512Mi`.
@@ -13,9 +67,12 @@ compression or range parallelism to fit the budget; file size does not need to f
in memory. The sparse format and download readers are unchanged.
With the current buffer sizes, the minimum budget is `384Mi` for GCS and `224Mi`
-for S3. Startup rejects smaller values. The accounting is conservative: GCS
-reserves for a composite upload even when an object ultimately fits in one
-request. Raise the budget to trade memory for more simultaneous uploads.
+for S3. Startup rejects smaller values. Each upload reserves for the path it
+takes: on GCS, a file split into range parts reserves only those parts, and an
+object whose sparse stream is bounded below one request (`64Mi`) reserves that
+request's buffers instead of the composite-upload buffers. Other streamed
+objects still reserve for a composite upload, even when they ultimately fit in
+one request. Raise the budget to trade memory for more simultaneous uploads.
The budget is **not a process memory limit**. The Go runtime, garbage awaiting
collection, connections, downloads, and asset transfers need additional memory.
@@ -24,9 +81,11 @@ the existing `128Mi` request is not a bound on its usage. If setting `GOMEMLIMIT
leave headroom below the container limit for memory outside the Go heap.
A manifest included with data files is uploaded only after all those files
-succeed. The existing two-RPC flow (data, then manifest) also uses the same budget.
+succeed. The existing two-RPC flow (data, then manifest) also uses the same
+budget; a manifest small enough for one request reserves only that request's
+buffers.
-## Validation
+### Validation
```sh
go test -race ./pkg/objectstorage ./internal/objectstoreplugin ./cmd/objectstore-plugin -count=2
diff --git a/cmd/objectstore-plugin/exchange.go b/cmd/objectstore-plugin/exchange.go
new file mode 100644
index 0000000000..26d69e8863
--- /dev/null
+++ b/cmd/objectstore-plugin/exchange.go
@@ -0,0 +1,131 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package main
+
+import (
+ "context"
+ "crypto/tls"
+ "errors"
+ "fmt"
+ "log/slog"
+ "net"
+
+ "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"
+ "google.golang.org/grpc"
+ "google.golang.org/grpc/credentials"
+ "google.golang.org/grpc/health"
+ healthpb "google.golang.org/grpc/health/grpc_health_v1"
+
+ "go.ate.dev/substrate/internal/credbundle"
+ "go.ate.dev/substrate/internal/objectstoreplugin/actorlookup"
+ "go.ate.dev/substrate/internal/objectstoreplugin/exchange"
+ "go.ate.dev/substrate/internal/objectstoreplugin/gcsissuer"
+ "go.ate.dev/substrate/internal/oidcjwt"
+ exchangepb "go.ate.dev/substrate/internal/proto/objectstoreexchangepb"
+ "go.ate.dev/substrate/internal/serverboot"
+ "go.ate.dev/substrate/pkg/objectstorage"
+)
+
+// exchangeConfig is what control mode needs to serve the credential exchange.
+type exchangeConfig struct {
+ listen string
+ credBundle string
+ issuer string
+ jwksFile string
+ ateapi actorlookup.Config
+}
+
+// newExchangeService builds the exchange handler for the configured backend.
+// The returned function closes what the handler holds open, the loopback
+// connection to ate-api-server, and is called at shutdown. GCS serves the
+// exchange. S3 answers UNIMPLEMENTED, since the S3 node plugin never calls
+// it: it ignores actor_jwt and transfers with the pod's own credentials. The
+// backend seam is exchange.Issuer, the one piece of the exchange that is not
+// shared between backends.
+func newExchangeService(ctx context.Context, cfg exchangeConfig) (exchangepb.CredentialExchangeServer, func(), error) {
+ if objectstorage.UsesS3() {
+ return exchangepb.UnimplementedCredentialExchangeServer{}, func() {}, nil
+ }
+ if cfg.issuer == "" {
+ return nil, nil, errors.New("--actor-jwt-issuer is required to serve the exchange on GCS")
+ }
+ if cfg.jwksFile == "" {
+ return nil, nil, errors.New("--actor-jwt-jwks-file is required to serve the exchange on GCS")
+ }
+ // Read the key set once now, so a wrong path fails the start rather
+ // than the first exchange.
+ keys := oidcjwt.FileKeySource(cfg.jwksFile)
+ if _, err := keys.Keys(ctx); err != nil {
+ return nil, nil, fmt.Errorf("while reading the actor JWT key set: %w", err)
+ }
+ verifier := oidcjwt.NewVerifier(cfg.issuer, []string{exchange.ActorJWTAudience}, keys, oidcjwt.WithSubstrateClaims())
+ lookup, err := actorlookup.New(cfg.ateapi)
+ if err != nil {
+ return nil, nil, err
+ }
+ issuer, err := gcsissuer.New(ctx)
+ if err != nil {
+ lookup.Close()
+ return nil, nil, fmt.Errorf("while setting up the GCS credential issuer: %w", err)
+ }
+ return exchange.New(verifier, lookup, issuer), func() { _ = lookup.Close() }, nil
+}
+
+// exchangeTransportCredentials serves the pod's service DNS certificate,
+// re-read on rotation. The exchange runs TLS without client certificates:
+// the JWT is the credential, and proof that the caller is atelet is a later
+// addition.
+func exchangeTransportCredentials(credBundle string) (credentials.TransportCredentials, error) {
+ load := credbundle.Loader(credBundle)
+ if _, err := load(nil); err != nil {
+ return nil, fmt.Errorf("while loading the exchange serving certificate: %w", err)
+ }
+ return credentials.NewTLS(&tls.Config{MinVersion: tls.VersionTLS13, GetCertificate: load}), nil
+}
+
+// serveExchange starts the exchange on its own TLS listener, beside the
+// plugin's Unix socket server, and returns a function that drains it.
+func serveExchange(ctx context.Context, cfg exchangeConfig) (func(), error) {
+ svc, release, err := newExchangeService(ctx, cfg)
+ if err != nil {
+ return nil, err
+ }
+ creds, err := exchangeTransportCredentials(cfg.credBundle)
+ if err != nil {
+ release()
+ return nil, err
+ }
+ lis, err := net.Listen("tcp", cfg.listen)
+ if err != nil {
+ release()
+ return nil, fmt.Errorf("while listening on %s: %w", cfg.listen, err)
+ }
+ srv := grpc.NewServer(grpc.Creds(creds), grpc.StatsHandler(otelgrpc.NewServerHandler()))
+ exchangepb.RegisterCredentialExchangeServer(srv, svc)
+ healthpb.RegisterHealthServer(srv, health.NewServer())
+ go func() {
+ slog.InfoContext(ctx, "Serving credential exchange", slog.String("address", lis.Addr().String()))
+ // GracefulStop makes Serve return nil. Anything else leaves the
+ // node plugins without an exchange while the socket keeps the pod
+ // looking healthy, so the process exits and restarts instead.
+ if err := srv.Serve(lis); err != nil {
+ serverboot.Fatal(ctx, "Credential exchange failed", err)
+ }
+ }()
+ return func() {
+ srv.GracefulStop()
+ release()
+ }, nil
+}
diff --git a/cmd/objectstore-plugin/exchange_test.go b/cmd/objectstore-plugin/exchange_test.go
new file mode 100644
index 0000000000..e2b637c12d
--- /dev/null
+++ b/cmd/objectstore-plugin/exchange_test.go
@@ -0,0 +1,60 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package main
+
+import (
+ "path/filepath"
+ "strings"
+ "testing"
+
+ exchangepb "go.ate.dev/substrate/internal/proto/objectstoreexchangepb"
+)
+
+func TestNewExchangeService(t *testing.T) {
+ t.Run("S3 answers unimplemented", func(t *testing.T) {
+ t.Setenv("ATE_STORAGE_BACKEND", "s3")
+ svc, release, err := newExchangeService(t.Context(), exchangeConfig{})
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer release()
+ if _, ok := svc.(exchangepb.UnimplementedCredentialExchangeServer); !ok {
+ t.Fatalf("S3 exchange is %T, want the unimplemented server", svc)
+ }
+ })
+ for _, tc := range []struct {
+ name string
+ cfg exchangeConfig
+ want string
+ }{
+ {"GCS needs the issuer", exchangeConfig{jwksFile: "jwks.json"}, "--actor-jwt-issuer"},
+ {"GCS needs the JWKS file", exchangeConfig{issuer: "https://idp.ate-system.svc"}, "--actor-jwt-jwks-file"},
+ {"GCS needs a readable JWKS file", exchangeConfig{issuer: "https://idp.ate-system.svc", jwksFile: filepath.Join(t.TempDir(), "absent.json")}, "key set"},
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ t.Setenv("ATE_STORAGE_BACKEND", "gcs")
+ _, _, err := newExchangeService(t.Context(), tc.cfg)
+ if err == nil || !strings.Contains(err.Error(), tc.want) {
+ t.Fatalf("err = %v, want it to name %s", err, tc.want)
+ }
+ })
+ }
+}
+
+func TestExchangeTransportCredentials_MissingBundle(t *testing.T) {
+ if _, err := exchangeTransportCredentials(filepath.Join(t.TempDir(), "absent.pem")); err == nil {
+ t.Fatal("a missing serving bundle was accepted")
+ }
+}
diff --git a/cmd/objectstore-plugin/main.go b/cmd/objectstore-plugin/main.go
index 6aa3469f5c..5b432ea37d 100644
--- a/cmd/objectstore-plugin/main.go
+++ b/cmd/objectstore-plugin/main.go
@@ -32,6 +32,7 @@ import (
"cloud.google.com/go/storage"
"github.com/spf13/pflag"
+ "go.ate.dev/substrate/internal/installdefaults"
"go.ate.dev/substrate/internal/nodepath"
"go.ate.dev/substrate/internal/objectstore"
"go.ate.dev/substrate/internal/objectstoreplugin"
@@ -62,6 +63,15 @@ func main() {
assetStagingDir := flags.String("asset-staging-dir", "/var/lib/objectstore-plugin/asset-staging", "node mode: directory sandbox assets are downloaded into and verified before they are copied below --root; must not be below --root or readable by the caller")
assetStagingCapacity := flags.String("asset-staging-capacity", "8Gi", "node mode: the most sandbox asset downloads in --asset-staging-dir may hold at once, as a Kubernetes quantity; downloads wait for space, and an asset larger than this is refused")
timeout := flags.Duration("timeout", 5*time.Second, "healthcheck mode: how long to wait for the plugin to report that it is serving")
+ var xcfg exchangeConfig
+ flags.StringVar(&xcfg.listen, "exchange-listen", "", "control mode: TCP address to serve the credential exchange on over TLS, for example :8445. Empty leaves it unserved.")
+ flags.StringVar(&xcfg.credBundle, "exchange-cred-bundle", "/run/servicedns.podcert.ate.dev/credential-bundle.pem", "control mode: PEM credential bundle the exchange serves with, the pod's service DNS certificate")
+ flags.StringVar(&xcfg.issuer, "actor-jwt-issuer", "", "control mode: issuer URL of the actor JWTs the exchange accepts, the value ate-api-server runs with")
+ flags.StringVar(&xcfg.jwksFile, "actor-jwt-jwks-file", "/run/actor-id-jwt-jwks/jwks.json", "control mode: JWK set file holding the public keys actor JWTs are verified with")
+ flags.StringVar(&xcfg.ateapi.Address, "ateapi-address", "127.0.0.1:443", "control mode: ate-api-server gRPC address, the loopback in the shared pod")
+ flags.StringVar(&xcfg.ateapi.CAFile, "ateapi-ca-file", "/run/servicedns.podcert.ate.dev/trust-bundle.pem", "control mode: CA bundle used to verify ate-api-server")
+ flags.StringVar(&xcfg.ateapi.ServerName, "ateapi-server-name", installdefaults.APIServiceHost(installdefaults.NamespaceFromPodEnv()), "control mode: DNS name expected on the ate-api-server certificate, the API Service in the pod's namespace")
+ flags.StringVar(&xcfg.ateapi.ClientCredBundle, "ateapi-client-cred-bundle", "/run/podidentity.podcert.ate.dev/credential-bundle.pem", "control mode: PEM credential bundle presented to ate-api-server, the pod identity certificate")
previewFlags := flags.StringSlice("preview", nil, "Preview gates to enable.")
_ = flags.Parse(os.Args[2:])
@@ -93,6 +103,8 @@ func main() {
defer serverboot.ShutdownProvider("TracerProvider", tp.Shutdown)
srv := grpc.NewServer(grpc.StatsHandler(otelgrpc.NewServerHandler()))
+ // stopExchange drains the credential exchange, when control mode serves it.
+ stopExchange := func() {}
switch mode {
case "node":
objects, err := objectstorage.NewFromEnv(ctx)
@@ -126,6 +138,13 @@ func main() {
serverboot.Fatal(ctx, "Failed to set up the object storage backend", err)
}
objectstorev1.RegisterControlProviderServer(srv, objectstoreplugin.NewControlPlugin(store))
+ if xcfg.listen != "" {
+ stop, err := serveExchange(ctx, xcfg)
+ if err != nil {
+ serverboot.Fatal(ctx, "Failed to serve the credential exchange", err)
+ }
+ stopExchange = stop
+ }
default:
fmt.Fprintln(os.Stderr, usage)
os.Exit(2)
@@ -138,12 +157,18 @@ func main() {
}
// Finish in-flight transfers on SIGTERM. The pod's grace period bounds
- // how long that may take.
+ // how long that may take. The exchange, when served, drains at the same
+ // time as the socket rather than after it.
sigCtx, stop := signal.NotifyContext(ctx, syscall.SIGTERM, os.Interrupt)
defer stop()
+ exchangeDrained := make(chan struct{})
go func() {
<-sigCtx.Done()
slog.InfoContext(ctx, "Draining object-store plugin")
+ go func() {
+ stopExchange()
+ close(exchangeDrained)
+ }()
srv.GracefulStop()
}()
@@ -151,6 +176,8 @@ func main() {
if err := srv.Serve(lis); err != nil {
serverboot.Fatal(ctx, "Serve failed", err)
}
+ // Serve returns only once GracefulStop ran, so the exchange is draining.
+ <-exchangeDrained
}
// newObjectStore builds the client the control plugin manages snapshot
diff --git a/demos/counter/README.md b/demos/counter/README.md
index 23db9d7ad4..7a4b99317a 100644
--- a/demos/counter/README.md
+++ b/demos/counter/README.md
@@ -136,6 +136,7 @@ ordinary application metadata.
1. Send an HTTP POST request to increment the counter:
Send to MicroVM
+
```bash
curl -X POST \
-H "ate-target-actor: ate-demo-counter-microvm/my-counter-microvm-1" \
@@ -145,6 +146,7 @@ curl -X POST \
Send to GVisor
+
```bash
curl -X POST \
-H "ate-target-actor: ate-demo-counter/my-counter-1" \
@@ -155,6 +157,7 @@ curl -X POST \
2. Verify that the actor is now in a `RUNNING` state and assigned to a worker pod:
Verify on MicroVM
+
```bash
kubectl ate get actor my-counter-microvm-1 -a ate-demo-counter-microvm
```
@@ -162,6 +165,7 @@ kubectl ate get actor my-counter-microvm-1 -a ate-demo-counter-microvm
Verify on GVisor
+
```bash
kubectl ate get actor my-counter-1 -a ate-demo-counter
```
@@ -170,6 +174,7 @@ kubectl ate get actor my-counter-1 -a ate-demo-counter
3. When finished, you can manually suspend the actor:
Suspend on MicroVM
+
```bash
kubectl ate suspend actor my-counter-microvm-1 -a ate-demo-counter-microvm
```
@@ -177,6 +182,7 @@ kubectl ate suspend actor my-counter-microvm-1 -a ate-demo-counter-microvm
Suspend on GVisor
+
```bash
kubectl ate suspend actor my-counter-1 -a ate-demo-counter
```
@@ -193,6 +199,7 @@ memory, the file counter from the durable volume.
4. To permanently delete the suspended actor:
Delete from MicroVM
+
```bash
kubectl ate delete actor my-counter-microvm-1 -a ate-demo-counter-microvm
```
@@ -200,6 +207,7 @@ kubectl ate delete actor my-counter-microvm-1 -a ate-demo-counter-microvm
Delete from GVisor
+
```bash
kubectl ate delete actor my-counter-1 -a ate-demo-counter
```
@@ -220,6 +228,7 @@ proxy behavior wouldn't do:
Non standard port with MicroVM
+
```bash
curl -p -x http://localhost:8001 \
--proxy-header "ate-target-actor: ate-demo-counter-microvm/my-counter-microvm-1" \
@@ -229,6 +238,7 @@ curl -p -x http://localhost:8001 \
Non standard port with GVisor
+
```bash
curl -p -x http://localhost:8001 \
--proxy-header "ate-target-actor: ate-demo-counter/my-counter-1" \
@@ -264,6 +274,7 @@ To remove the counter demo resources from your cluster, run:
Delete from MicroVM
+
```bash
./hack/install-ate.sh --delete-demo-counter
```
@@ -271,6 +282,7 @@ To remove the counter demo resources from your cluster, run:
Delete from GVisor
+
```bash
./hack/install-ate.sh --delete-demo-counter-microvm
```
diff --git a/demos/counter/counter-microvm-csi-test-template.yaml b/demos/counter/counter-microvm-csi-test-template.yaml
index 6d1e7bb280..80eccea76a 100644
--- a/demos/counter/counter-microvm-csi-test-template.yaml
+++ b/demos/counter/counter-microvm-csi-test-template.yaml
@@ -45,6 +45,7 @@ snapshotConfig:
storageLocation: gs://ate-snapshots/ate-demo-counter-microvm-csi/
volumes:
- name: data
- externalVolumeTemplate:
- capacity: 1Gi
- storageClassName: csi-hostpath-sc
+ external:
+ template:
+ capacity: 1Gi
+ storageClassName: csi-hostpath-sc
diff --git a/demos/sandbox/manual-test-multi-template.yaml b/demos/sandbox/manual-test-multi-template.yaml
index 1e35e877ba..add9312307 100644
--- a/demos/sandbox/manual-test-multi-template.yaml
+++ b/demos/sandbox/manual-test-multi-template.yaml
@@ -47,10 +47,12 @@ snapshotConfig:
storageLocation: gs://ate-snapshots/ate-manual-test-multi/
volumes:
- name: vol-hostpath
- externalVolumeTemplate:
- capacity: 1Gi
- storageClassName: csi-hostpath-sc
+ external:
+ template:
+ capacity: 1Gi
+ storageClassName: csi-hostpath-sc
- name: vol-nfs
- externalVolumeTemplate:
- capacity: 1Gi
- storageClassName: csi-nfs-sc
+ external:
+ template:
+ capacity: 1Gi
+ storageClassName: csi-nfs-sc
diff --git a/docs/api-guide.md b/docs/api-guide.md
index d8c3070740..3255e88734 100644
--- a/docs/api-guide.md
+++ b/docs/api-guide.md
@@ -100,8 +100,8 @@ The `ActorTemplate` defines the code, environment, and state-management policies
| :--- | :--- | :--- |
| `containers` | `[]Container` | **Required.** The workload definition — see [Container Fields](#container-fields) below. Each container may also declare an optional `wakeupProbe` HTTP probe — see [Container Wakeup Probe](#container-wakeup-probe-wakeupprobe). |
| `workerSelector` | `*Selector` | Optional. Gates which `WorkerPool`s actors from this template may use, by matching against each pool's labels (`matchLabels`). If unset, all pools are eligible (subject to the actor's own `worker_selector`). The pool decides the sandbox class, so this is also how a template pins one: if your cluster has pools of more than one sandbox class and the workload needs a particular class, make sure the selector points only at pools of that class. An actor booting with no snapshot and no selector may land on any class, and keeps it from its first checkpoint on. |
-| `snapshotConfig` | `SnapshotConfig` | **Required.** The base object-storage location snapshots are written under, the snapshot fidelity (`preferredFidelity`) every Pause and Suspend captures, and automated golden snapshot creation (`goldenSnapshotConfig.mode`). See [Snapshot Storage Layout](#snapshot-storage-layout). |
-| `volumes` | `[]Volume` | Optional. Volumes the containers may mount, each a `durableDir`, an `externalVolumeTemplate` (see [CSI Volumes Guide](csi-volumes.md)), or a `systemInfo` volume (see [SystemInfo Volumes](#systeminfo-volumes)). Every declared volume must be mounted by at least one container. A `durableDir` is created `0777`, as Kubernetes creates an emptyDir, so a container running as any user can write it. A snapshot restores it with the mode it had, so the container's own chmod survives suspend and resume. |
+| `snapshotConfig` | `SnapshotConfig` | Optional; omitted fields take their defaults. The base object-storage location snapshots are written under (a preview field, ignored without the `PerTemplateSnapshotLocation` gate), the snapshot fidelity (`preferredFidelity`) every Pause and Suspend captures, and automated golden snapshot creation (`goldenSnapshotConfig.mode`). See [Snapshot Storage Layout](#snapshot-storage-layout). |
+| `volumes` | `[]Volume` | Optional. Volumes the containers may mount, each a `durableDir`, an `external` CSI volume source (see [CSI Volumes Guide](csi-volumes.md)), or a `systemInfo` volume (see [SystemInfo Volumes](#systeminfo-volumes)). Every declared volume must be mounted by at least one container. A `durableDir` is created `0777`, as Kubernetes creates an emptyDir, so a container running as any user can write it. A snapshot restores it with the mode it had, so the container's own chmod survives suspend and resume. |
| `resources` | `*ResourceRequirements` | Required. Declares each actor's compute size via `limits`, which must set both `cpu` and `memory` — see [Sandbox Right-Sizing](#sandbox-right-sizing-resources). Immutable, like the rest of the template. |
The sandbox runtime and its binaries (e.g. the gVisor `runsc` binary and `pauseImage`, or a micro-VM kernel/firmware/config) are configured on the [`WorkerPool`](#1-workerpool-the-compute-foundation) via its `sandboxClasses` and the cluster-scoped [`SandboxConfig`](#3-sandboxconfig-the-sandbox-itself) it resolves, rather than on the `ActorTemplate`. An `ActorTemplate` selects its `WorkerPool`s via `workerSelector` (AND'd with the actor's `worker_selector`). When an actor runs on a worker, the worker's active `SandboxRuntime` is stamped onto the actor's assignment and snapshots so subsequent resumes prefer runtime-compatible workers.
@@ -342,6 +342,13 @@ snapshotConfig:
/atespaces//tags/
```
+> [!NOTE]
+> Snapshots are stored under ate-api-server's cluster-wide `--snapshot-location`. `snapshotConfig.storageLocation` is an optional preview field: with the `PerTemplateSnapshotLocation` preview gate (`--preview=PerTemplateSnapshotLocation`), a template that sets it stores its snapshots there instead. Without the gate, the field is ignored. `` above is whichever of the two applies.
+>
+> With the gate enabled, only a client that opts in to preview features (`kubectl ate --preview`, or the `Ate-Preview-Allowed: true` header) may set `storageLocation`. `CreateActorTemplate` fails with `FailedPrecondition` when the template leaves it empty and ate-api-server has no `--snapshot-location`.
+>
+> Changing the flag or toggling the gate moves `` for every template it affects. Snapshots already taken stay where they are and can still be resumed from. An actor's next suspend writes under the new location and releases its previous snapshot from the old one. Until then, the actor cannot change its `actor_template` (see `UpdateActor` below), and deleting it collects its prefix under the old location. If that release fails, the previous snapshot's objects stay under the old location, and deleting the actor later does not collect them.
+
The objects of a snapshot (its manifest, memory image, durable-data tar) are named below it. So for the template above, a snapshot of an actor in atespace `team-a` is stored at `gs://my-bucket/secret-agent/atespaces/team-a/actors/3f8b…/snapshots/f47ac10b-…`, and the template's golden snapshot under `gs://my-bucket/secret-agent/atespaces/team-a/tags/`.
An actor takes a series of snapshots over its life, so it gets a prefix of its
@@ -387,8 +394,8 @@ a `Snapshot` with the following fields:
has one, and the tag can seed actors only once it is `COMPLETED`.
- `sandboxRuntime`: The sandbox runtime and hardware compatibility attributes the
guest state was captured under, copied from `status.workerAssignment.sandboxRuntime`
- when the checkpoint is taken. A resume that restores memory or rootfs state
- (`SNAPSHOT_FIDELITY_MEMORY` or `SNAPSHOT_FIDELITY_ROOTFS`) first schedules onto
+ when the checkpoint is taken. A resume that restores memory state
+ (`SNAPSHOT_FIDELITY_MEMORY`) first schedules onto
an eligible worker whose `defaultRuntime` or `restorableRuntimes` is compatible
with `sandboxRuntime` (same sandbox class, same schema version, and every
attribute recorded on the snapshot present on the worker with the same
@@ -398,17 +405,19 @@ a `Snapshot` with the following fields:
capacity:
- A durable snapshot — the actor's own, a `sourceTag` clone, or the
template's golden snapshot a new actor starts from — falls back to any eligible
- worker and downgrades the restore to `SNAPSHOT_FIDELITY_VOLUMES`, preserving
- the snapshot's volume data while booting the guest fresh under the new
- worker's `defaultRuntime`.
+ worker and downgrades the restore to `SNAPSHOT_FIDELITY_ROOTFS`, preserving
+ the snapshot's rootfs and volume data while booting the guest fresh under the
+ new worker's `defaultRuntime`.
- A local pause checkpoint (with `locality`) is not relaxed; the resume
returns `RESOURCE_EXHAUSTED` and the actor stays paused on its node until a
compatible worker there has room, or until atelet detaches it. A detach
uploads the checkpoint, deletes the local files, and lifts the node pin,
after which the next resume may fall back like a durable snapshot.
- A snapshot without `sandboxRuntime` or with `SNAPSHOT_FIDELITY_VOLUMES` imposes
- no runtime placement constraint.
+ A snapshot without `sandboxRuntime` or with `SNAPSHOT_FIDELITY_ROOTFS` or
+ `SNAPSHOT_FIDELITY_VOLUMES` imposes no runtime placement constraint. Because a
+ `ROOTFS` restore reuses the container's rootfs upper layer over the template's
+ image, pools selected by an `ActorTemplate` must share a CPU architecture.
An `ActorTemplate` references its golden tag with the `ObjectRef` in
`status.goldenSnapshotStatus.goldenTag`. These status fields are server-owned
@@ -541,7 +550,8 @@ its Golden tag and any unfinished Golden actor.
Golden capture follows `preferredFidelity`. MEMORY preserves processes and memory;
VOLUMES preserves startup data in mounted volumes and starts new processes on
-restore. ROOTFS preserves a micro-VM's filesystem and also starts new processes.
+restore. ROOTFS preserves the sandbox's root filesystem writes and also starts new
+processes.
`CreateActor` uses an explicit `sourceTag` when supplied; otherwise it resolves the template's golden tag and records that snapshot on the new actor. If the golden tag is not ready yet, the actor starts without a snapshot and cold-boots even if the tag becomes ready before its first resume. The default does not populate the caller-owned `sourceTag` field.
@@ -602,7 +612,7 @@ Registers a new logical actor in the system.
Replaces the mutable fields of an existing actor with the ones in the request.
* **Request:** `UpdateActorRequest`
* `actor`: `Actor` — the complete replacement actor. `metadata.atespace` and `metadata.name` identify the resource; `metadata.uid` and `metadata.version` are **required** preconditions. `metadata` and `status` are server-owned and whatever the request carries in them is ignored. `source_tag` is immutable.
- * `actor_template` may change only while the actor is `SUSPENDED`. The new template must declare the same volumes and volume mounts. Once the actor owns an external snapshot, the new template must also store snapshots under the same `snapshotConfig.storageLocation`.
+ * `actor_template` may change only while the actor is `SUSPENDED`. The new template must declare the same volumes and volume mounts. Once the actor owns an external snapshot, the new template must also store snapshots under the same location: its `snapshotConfig.storageLocation` with the `PerTemplateSnapshotLocation` preview gate, otherwise ate-api-server's `--snapshot-location` (see [Snapshot Storage Layout](#snapshot-storage-layout)).
* **Response:** the updated `Actor`.
* **Errors:** `INVALID_ARGUMENT` if `uid` or `version` is unset, or if the request changes an immutable field — including by leaving one unset; `FAILED_PRECONDITION` if a change to `actor_template` breaks one of the rules above; `ABORTED` if either guard no longer matches the stored resource.
@@ -621,7 +631,8 @@ Hibernate a running actor, capturing its current RAM and disk state into a snaps
* `options.mode`: (Optional) `DURABILITY_MODE_SYNCHRONOUS` returns once the snapshot is `COMPLETED` in object storage. `DURABILITY_MODE_ASYNCHRONOUS` (a pause) returns once the snapshot is captured on the worker's node, and atelet uploads it in the background. Unset runs an asynchronous suspend.
* **Response:** `SuspendActorResponse` containing the `Actor` object in `ACTOR_STATE_SUSPENDED`, with its snapshot in `status.snapshots`.
* Called on an actor that is already `ACTOR_STATE_SUSPENDED`, it returns the actor as it is, in either mode. atelet uploads a paused actor's node-local snapshot in the background, so `status.snapshots` may still show it with `locality` and a `PENDING` `durableSnapshot`.
-* A successful suspend releases the actor's previous external snapshot: an actor keeps one, and only tags outlive it. To keep the snapshot a suspend just wrote, tag it with `CreateTag` while the actor is still suspended. After an asynchronous suspend, `CreateTag` fails with `FAILED_PRECONDITION` until atelet has uploaded the snapshot. Suspend with `DURABILITY_MODE_SYNCHRONOUS` to tag right away.
+* A snapshot generation conflict returns `ABORTED` without crashing the actor. The actor can remain `ACTOR_STATE_SUSPENDING`, with its worker assignment and previous completed snapshot kept for recovery. Read the actor's latest state before retrying; repeating a stale request cannot resolve a persistent disagreement with atelet.
+* A successful suspend releases the actor's previous external snapshot: an actor keeps one, and only tags outlive it. To keep the snapshot a suspend just wrote, tag it with `CreateTag` while the actor is still suspended. After an asynchronous suspend, `CreateTag` first has atelet upload the snapshot if it has not already, so the call can take as long as that upload. If the upload fails, so does `CreateTag`, and the actor is left as it was.
#### Snapshot lifetime
@@ -647,11 +658,10 @@ Discards an actor's live or crashed execution and transitions it to `ACTOR_STATE
* External volumes are not reverted. Their contents are never part of a snapshot, so a reverted actor comes back with its memory and root filesystem rewound but its volumes exactly as the discarded execution left them.
#### `DeleteActor`
-Removes an actor from the registry and cleans up associated resources.
+Removes an actor in any state from the registry and cleans up associated resources, terminating active workloads, detaching volumes, and releasing worker allocations.
* **Request:** `DeleteActorRequest`
* `actor`: `ObjectRef` of the actor to delete.
* `options`: (Optional) `DeleteOptions`. `uid` and `version` are preconditions checked against the actor as the caller last read it: a mismatch returns `ABORTED`, an omitted guard is skipped. They are checked before the workflow starts, so a guarded delete that fails part-way leaves the actor `ACTOR_STATE_DELETING` at a higher version. Retry it with the `uid` guard alone, or re-read first.
- * `any_state`: (Optional) If `true`, allows deleting the actor from any state (e.g. `RUNNING`), terminating active workloads, detaching volumes, and releasing worker allocations. By default (`false`), only actors in `ACTOR_STATE_SUSPENDED` (paused ones included) or `ACTOR_STATE_CRASHED` (or already `ACTOR_STATE_DELETING`) can be deleted.
* **Response:** the deleted `Actor`, as it was immediately before removal.
* Deleting an actor also deletes the external snapshot it owns, along with one an interrupted suspend left behind. Snapshots it only borrows from a tag are left alone, and its tags are unaffected — they hold their own copies.
@@ -669,34 +679,32 @@ Query the physical resource pool.
## 7. Advanced: Actor Identity Credentials
-Workloads can exchange their ephemeral Kubernetes credentials for stable **Actor Identity** credentials that persist even as the process migrates between different physical workers. This is distinct from the `actorMetadata` data source described under [SystemInfo Volumes](#systeminfo-volumes), which only tells an actor its own identity fields (name, atespace, uid).
+ate-api-server mints credentials that identify an actor to services outside Substrate. They name the logical actor, so they stay the same as the actor moves between workers. This is distinct from the `actorMetadata` data source described under [SystemInfo Volumes](#systeminfo-volumes), which only tells an actor its own identity fields (name, atespace, uid).
+
+Both RPCs are on `ateapi.Control`. With authorization on, as ate-setup installs it by default, only the egress gateway may call them; see [authorization.md](authorization.md). It calls `MintActorJwt` when an egress policy injects an actor JWT into a request.
+
+### `MintActorJwt`
-### Service: `ateapi.ActorIdentity`
-* **`MintJWT`:** Generates an OIDC-compatible JWT identifying the Substrate Actor.
-* **`MintCert`:** Signs a Certificate Signing Request (CSR) to provide an mTLS identity for the actor.
+* **Request:** `actor`, the actor's atespace and name; `audiences`, one to 16 values for the token's `aud`; and `expiration_seconds`, the lifetime, from 300 to 3600. The actor must exist, but it doesn't need to be running.
+* **Response:** `actor_jwt`, the signed token, and `expire_time`, which equals its `exp` claim.
-Both RPCs identify the actor the same way the rest of the API does, by `atespace` and `actor_name`.
+The token's header carries `typ: JWT`, the algorithm of the pool's active signing key (ES256 or RS256), and that key's ID as `kid`. Its claims are:
-#### Who may call `MintCert` and `MintJWT`
+* `iss`: ate-api-server's `--actor-jwt-issuer`. ate-setup sets it from the `actorJWT.issuer` setting, which defaults to `https://idp..svc`.
+* `sub`: `actor//`.
+* `aud`: the requested audiences, always a JSON array.
+* `exp`, `nbf`, `iat`: Unix seconds. `nbf` is five minutes before `iat`, to allow for clock skew.
+* `jti`: a random token ID.
+* `actor.ate.dev`: an object with the actor's `atespace`, `actorName`, and `actorUID`.
-`MintCert` and `MintJWT` are not callable by actors directly. They must be called over mTLS with a
-Pod Certificate, and the broker only signs a CSR when all of the following hold:
+Relying parties find the signing keys through OpenID Connect Discovery at `/.well-known/openid-configuration`, which ate-idp-server serves in every install.
-1. The client certificate identifies the **`atelet`** service account
- (`spiffe://cluster.local/ns/ate-system/sa/atelet`) and carries a Pod Identity
- extension, which pins the calling atelet to a node.
-2. The requested actor is **currently running**, per the actor database.
-3. The worker Pod hosting that actor is on the **same node** as the calling
- atelet, and is still assigned to that actor.
+### `MintActorCertificate`
-An atelet is therefore confined to minting credentials for the actors it is
-actually hosting. Callers that fail any of these checks receive
-`PERMISSION_DENIED` with no detail, so the RPC cannot be used to discover
-whether an actor exists or where it is running. An actor that exists but is
-suspended, paused, or crashed yields `FAILED_PRECONDITION`.
+* **Request:** `actor`; `actor_uid`, which must match the actor's current UID, so that a deleted and recreated actor isn't mistaken for the old one (`ABORTED` otherwise); and `certificate_signing_request`, a DER-encoded CSR. The signer checks the CSR's signature and uses only its public key.
+* **Response:** `actor_certificates`, DER certificates with the leaf first, followed by any intermediates.
-The minted leaf certificate carries the SPIFFE URI
-`spiffe://substrate-actor.local/atespace/${atespace}/actor/${actor_name}`.
+The caller must connect with a client certificate; a bearer token is not enough. The leaf is valid for one hour, allows client and server authentication, carries the SPIFFE URI `spiffe://substrate-actor.local/actor//`, and has the `ActorIdentity` X.509 extension with the actor's atespace, name, and UID.
---
@@ -705,7 +713,7 @@ The minted leaf certificate carries the SPIFFE URI
Agent Substrate is designed to be the foundational execution layer for any agentic framework.
### Agent Development Kit (ADK)
-Substrate provides native support for ADK-compatible identities. Workloads can use the `ActorIdentity` service to mint JWTs that align with ADK's security model, ensuring seamless integration with ADK-managed tools and memory.
+An ADK agent running as an actor can have the egress gateway attach an actor JWT ([section 7](#7-advanced-actor-identity-credentials)) to its outbound requests, so the tools and services it calls can tell which actor is calling.
### LangChain
Substrate is an ideal runtime for stateful LangChain agents. By defining a LangChain agent as an `ActorTemplate`, you can preserve the agent's internal "thought process" and conversation history in memory across hibernations, while sandboxing its tool execution for security.
diff --git a/docs/api-style-guide.md b/docs/api-style-guide.md
index 8a733ad102..484576c71c 100644
--- a/docs/api-style-guide.md
+++ b/docs/api-style-guide.md
@@ -622,7 +622,7 @@ message Volume {
// +k8s:format=k8s-short-name
string name = 1;
- // Exactly one of durable_dir / external_volume_template / image must be set.
+ // Exactly one of durable_dir / external / image must be set.
//
// +k8s:optional
// +k8s:unionMember
@@ -630,7 +630,7 @@ message Volume {
// +k8s:optional
// +k8s:unionMember
- ExternalVolumeTemplate external_volume_template = 3;
+ ExternalVolumeSource external = 7;
// +k8s:optional
// +k8s:unionMember
diff --git a/docs/architecture.md b/docs/architecture.md
index 6583be2059..1bda9e6bca 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -97,8 +97,6 @@ started to tackle yet, but which we know will need to be solved eventually.
* Peer to peer state sharing: Relying on data locality presents a risk of
data loss. We probably need some form of state-sharing to protect against
that, which needs to be balanced with saving state to a "permanent" store.
- * Control plane authn/z: We will need to ensure that the control plane is secure, and that only
- authorized users and agents can interact with it.
* Identity and policy: We know that agents' needs for identity are very
different than traditional workload identity and end-user credentials.
* There are certainly more that we haven't thought of yet!
@@ -317,6 +315,10 @@ manage actor lifecycles.
* **Workflow Engine**: Orchestrates the multi-step Resume/Suspend sequences
(lock acquisition, storage download, sandbox restore).
+ * **Authentication & Authorization**: [Authenticates](authentication.md)
+ every RPC and [authorizes](authorization.md) it per atespace with an
+ embedded OpenFGA model.
+
### Node Supervisor (`atelet` + `ateworker`)
The node-level subsystem manages the physical execution of sandboxes and the movement of snapshots.
@@ -329,6 +331,8 @@ The node-level subsystem manages the physical execution of sandboxes and the mov
* **Storage Mover**: External snapshots move through an *object-store plugin*, a sidecar that serves the `objectstore.v1` gRPC API over a Unix socket in its pod. `atelet`'s sidecar (`NodeProvider`) uploads and fetches a snapshot's files; `ate-api-server`'s sidecar (`ControlProvider`) copies and deletes whole snapshots. Neither main process talks to object storage for snapshots. The in-tree plugin, `cmd/objectstore-plugin`, backs the API with GCS or S3; `ATE_STORAGE_BACKEND` on the sidecar (unset or `gcs`, or `s3` with the `AWS_*` variables) selects which, see `manifests/ate-install/atelet.yaml`, `ate-api-server.yaml`, and the kind overlay for an S3 example. The same `atelet` sidecar serves `AssetProvider`, which fetches every sandbox asset; `atelet` reads a `gs://` asset anonymously only when the plugin cannot serve it, which covers public buckets such as `gs://gvisor`. The plugin stages each download where `atelet` cannot read it, and copies it into `atelet`'s cache only if its sha256 matches; it refuses any URL under a snapshot location. `atelet` verifies the sha256 again and has no object storage credentials or settings of its own.
+ * **Sandbox Asset Cache**: `atelet` keeps one node-local, content-addressed cache of the sandbox binaries every actor runs on and gives each actor hard links into it, prewarms each local worker's default runtime and keeps it linked under the worker's directory, and evicts to a byte budget without taking anything an actor or a worker still uses. See the [Sandbox Asset Cache](sandbox-asset-cache.md) page.
+
### Sandbox Classes
A `WorkerPool` selects a **sandbox class** (`spec.sandboxClasses[].name`), and each class has a matching `ateworker` herder image. The sandbox binaries themselves are not baked into the worker image — they, and the pause image holding the sandbox's namespaces, come at runtime from the cluster-scoped [`SandboxConfig`](api-guide.md#3-sandboxconfig-the-sandbox-itself) the `WorkerPool` resolves for its class (an explicit `configRef`, else the class default) and are pinned into each snapshot's manifest so restores stay reproducible across runtime upgrades.
@@ -508,7 +512,7 @@ tags cannot be deleted until they are.
An actor in `ACTOR_STATE_RUNNING` or `ACTOR_STATE_CRASHED`, or a paused actor (`ACTOR_STATE_SUSPENDED` with a node-local latest snapshot), can be reverted back to `ACTOR_STATE_SUSPENDED` at its durable snapshot via `RevertActor`. Reverting terminates any active sandbox, discards any local pause checkpoint or partial in-progress snapshot, and preserves the actor's last completed external snapshot so a subsequent `ResumeActor` restores from that checkpoint.
-By default, only actors in `ACTOR_STATE_SUSPENDED` or `ACTOR_STATE_CRASHED` state can be deleted from the Control Plane. With the `any_state` flag enabled, an actor in any state (such as `ACTOR_STATE_RUNNING`) can be deleted directly; the workflow terminates the running containers on the worker, detaches mounted volumes, and frees the worker assignment before deleting the record.
+An actor in any state (such as `ACTOR_STATE_RUNNING`) can be deleted from the Control Plane. The workflow terminates the running containers on the worker, detaches mounted volumes, and frees the worker assignment before deleting the record.
After deletion, the state of the actor (i.e., memory+disk snapshots) is garbage collected.
@@ -543,12 +547,22 @@ Agent Substrate is built on a **Defense-in-Depth** model:
security context even as they migrate across physical nodes or code
versions.
- * **Request Authorization**: The system currently performs **Identity-Aware
- Routing** by extracting and validating the `ate-target-actor` header at
- the gateway. This ensures requests are only
- routed to recognized, registered actors.
- Pluggable, granular authorization policies are planned for future
- milestones.
+ * **API Authorization**: With enforcement on, the default in installs from
+ `ate-setup`, every `ate-api-server` RPC is authorized. Users
+ and automation get owner, editor, or viewer roles on the global scope or
+ on an atespace through AccessPolicy (`kubectl ate access-policy`), and
+ Substrate's components hold fixed system roles bound to their pod
+ certificates. The atespace is the finest boundary today. See
+ [authorization.md](authorization.md) for the roles, the check behind each
+ RPC, and known gaps.
+
+ * **Ingress Authentication and Routing**: The gateway routes each request to
+ the actor its `ate-target-actor` header names, and only to recognized,
+ registered actors. With `--ingress-auth-mode=static-mtls`, it accepts only
+ clients whose SPIFFE ID is on a static allowlist; any listed client may
+ reach any actor, as per-actor ingress authorization is not yet supported.
+ The default mode, `deprecated-insecure`, also serves plaintext listeners
+ that accept any client.
* **Network Policy**: Agent Substrate leverages standard Kubernetes
NetworkPolicy for connectivity control. Policies can be applied at the
diff --git a/docs/authentication.md b/docs/authentication.md
index e7c6c0fd56..4bccc4fa52 100644
--- a/docs/authentication.md
+++ b/docs/authentication.md
@@ -4,7 +4,6 @@
configured with the file passed to `--authentication-config`:
```yaml
-actorIdentityJWTProvider: kubernetes
jwtProviders:
- name: kubernetes
issuer: https://kubernetes.default.svc.cluster.local
@@ -23,11 +22,10 @@ Provider names and issuers must be unique. `issuer` must be an HTTPS URL and
matches. `certificateAuthorityFile` and `discoveryTokenFile` are optional and
are needed for OIDC discovery against some private Kubernetes API servers.
-`actorIdentityJWTProvider` identifies the provider allowed to call
-`ActorIdentity.MintJWT`. What an authenticated caller may do is decided by
-authorization; see [authorization.md](authorization.md). Its principal ID does
-not record which provider issued the token, so only configure providers you
-trust not to issue a token whose subject is another principal's ID.
+What an authenticated caller may do is decided by authorization; see
+[authorization.md](authorization.md). Its principal ID does not record which
+provider issued the token, so only configure providers you trust not to issue
+a token whose subject is another principal's ID.
## Google Cloud CLI tokens
diff --git a/docs/authorization.md b/docs/authorization.md
index 323775e71a..1b4e16355d 100644
--- a/docs/authorization.md
+++ b/docs/authorization.md
@@ -232,8 +232,7 @@ certificate.
| `RequestActorSuspend` | `atelet` on the node of the worker hosting the actor |
| `MintWorkerActorJwt` | `atelet` on the node hosting the actor |
| `MintWorkerActorCertificate` | `atelet` on the node of the worker the actor is assigned to |
-
-`ReportActorSnapshotState` has a rule but is not served yet.
+| `ReportActorSnapshotState` | `atelet` on the node the actor is placed on or holds a local snapshot of it |
### Every RPC
diff --git a/docs/csi-volumes.md b/docs/csi-volumes.md
index 388d7ce2f2..645beb9369 100644
--- a/docs/csi-volumes.md
+++ b/docs/csi-volumes.md
@@ -16,7 +16,7 @@ do not need it.
In Kubernetes, volumes are reconciled asynchronously via standard Kubernetes objects (e.g. `PersistentVolumeClaim`, `PersistentVolume`). Agent Substrate takes a different approach tailored for actor lifecycle operations:
-* **No PV or PVC Objects:** External volumes are declaratively defined in the [`ActorTemplate`](api-guide.md#2-actortemplate-the-workload-blueprint) via `externalVolumeTemplate` and provisioned dynamically for each actor instance. Volume operations are coupled directly with the actor lifecycle.
+* **No PV or PVC Objects:** External volumes are declaratively defined in the [`ActorTemplate`](api-guide.md#2-actortemplate-the-workload-blueprint) via `external.template` and provisioned dynamically for each actor instance. Volume operations are coupled directly with the actor lifecycle.
* **Direct Network-Based CSI Controller:** The Substrate control plane (`ateapi`) communicates directly with the CSI Controller gRPC service over the network (via TCP or DNS endpoints, optionally secured with TLS/mTLS).
---
@@ -74,7 +74,7 @@ External volumes are declared on the `ActorTemplate` resource. For complete deta
To attach a CSI volume to an actor:
-1. Define the volume under `volumes` with an `externalVolumeTemplate`.
+1. Define the volume under `volumes` with an `external.template`.
2. Mount the volume inside one or more containers under `containers[].volumeMounts`.
#### `volumes[]`
@@ -82,14 +82,15 @@ To attach a CSI volume to an actor:
```yaml
volumes:
- name: my-data-volume
- externalVolumeTemplate:
- capacity: 10Gi
- storageClassName: standard-rwx
+ external:
+ template:
+ capacity: 10Gi
+ storageClassName: standard-rwx
```
* `name`: Unique DNS-label-compliant volume name.
-* `externalVolumeTemplate.capacity`: Quantity string representing the requested volume size (e.g. `1Gi`, `50Gi`).
-* `externalVolumeTemplate.storageClassName`: Name of a Kubernetes `StorageClass` present in the cluster whose `provisioner` matches a registered `CSIDriverConfig`.
+* `external.template.capacity`: Quantity string representing the requested volume size (e.g. `1Gi`, `50Gi`).
+* `external.template.storageClassName`: Name of a Kubernetes `StorageClass` present in the cluster whose `provisioner` matches a registered `CSIDriverConfig`.
#### `containers[].volumeMounts[]`
@@ -197,7 +198,8 @@ snapshotConfig:
storageLocation: gs://my-snapshots-bucket/stateful-agent
volumes:
- name: shared-storage
- externalVolumeTemplate:
- capacity: 5Gi
- storageClassName: csi-nfs-sc
+ external:
+ template:
+ capacity: 5Gi
+ storageClassName: csi-nfs-sc
```
diff --git a/docs/dev/best-practices/otel-collector.md b/docs/dev/best-practices/otel-collector.md
index 5851a154c3..01448f2de2 100644
--- a/docs/dev/best-practices/otel-collector.md
+++ b/docs/dev/best-practices/otel-collector.md
@@ -426,6 +426,9 @@ that defaults to it, so you should not normally need either:
`ate-controller`'s `--otel-exporter-otlp-endpoint`, which it also propagates to
the worker pods it creates, and `atenet-router`'s
`--otlp-collector-address`, which is what its Envoy is given over xDS.
+`ateworker` uses the propagated endpoint only when its node's `atelet` serves no
+OTLP relay. It exports through the relay whenever `atelet` has a collector,
+and exports nothing when it has neither.
Rather than editing the base manifests, prefer a kustomize overlay that
replaces the `ate-otel-config` ConfigMap — see
@@ -439,7 +442,8 @@ lifecycle records and the ateworkers' usage samples, through
`serverboot.InitLogging`. `OTEL_LOGS_EXPORTER=otlp` sends them over OTLP
instead of stdout, and `otlp,console` to both. The kind overlay selects
`otlp,console`; the Helm chart selects `otlp` when `otel.logs.enabled` is true
-and a logs endpoint resolves. See
+and a logs endpoint resolves. Both also set `ATE_WORKER_OTEL_LOGS_EXPORTER`
+for the ateworkers. See
[the same records over OTLP](../../observability.md#the-same-records-over-otlp).
Everything else is stdout. `serverboot.InitLogger` writes structured JSON there,
diff --git a/docs/dev/best-practices/tracing.md b/docs/dev/best-practices/tracing.md
index a13aa2a9a9..1e9edae29e 100644
--- a/docs/dev/best-practices/tracing.md
+++ b/docs/dev/best-practices/tracing.md
@@ -64,6 +64,8 @@ Samplers are resolved with `serverboot.ResolveTraceSampling`, which applies the
All defaults are `ParentBased`, so a request that arrives already sampled stays sampled on every hop, and one that arrives explicitly unsampled stays unsampled. Only parentless requests are subject to the ratio, at whichever component roots the trace.
+The `ateapi` store follows the same rule and has no switch of its own. Its pgx tracer opens one client span per PostgreSQL statement, `begin` and `commit` included, only when the request is already sampled. `always_off` on `ateapi` therefore silences store spans together with everything else, and statements from background work never produce any.
+
An invalid sampler name, or a missing or unparsable ratio arg, keeps the component default and logs a warning. This deliberately diverges from the OTel SDK's own env handling, which falls back to 100% sampling on invalid input and reads a missing arg as ratio 1.0.
In agentgateway mode the data plane root fraction lives in the agentgateway ConfigMap (`randomSampling`, same 0.01 default). Unlike Envoy's `RandomSampling`, it is static config that env overrides on the router do not reach, so adjust both together.
diff --git a/docs/examples/substrate-install.yaml b/docs/examples/substrate-install.yaml
index 6d8f2bb98f..7d45899d03 100644
--- a/docs/examples/substrate-install.yaml
+++ b/docs/examples/substrate-install.yaml
@@ -50,6 +50,7 @@ images:
# Snapshot storage.
storage:
bucketName: ate-snapshots
+ snapshotLocation: gs://ate-snapshots/
ateapi:
postgres:
diff --git a/docs/metrics/registry/metrics.yaml b/docs/metrics/registry/metrics.yaml
index 35117811f9..0e97bc2412 100644
--- a/docs/metrics/registry/metrics.yaml
+++ b/docs/metrics/registry/metrics.yaml
@@ -569,6 +569,50 @@ groups:
value: timeout
brief: The time limit of the caller ended.
+ - id: registry.ate.filecache
+ type: attribute_group
+ brief: >
+ The labels of the node-local file caches: the stores atelet serves
+ hard links from, such as the sandbox asset cache. They run on one
+ store and one eviction loop, so one set of instruments describes them,
+ told apart by name.
+ attributes:
+ - id: ate.filecache.name
+ stability: development
+ brief: Which file cache on the node the measurement is of.
+ type:
+ members:
+ - id: sandbox_asset
+ stability: development
+ value: sandbox_asset
+ brief: >
+ The sandbox asset cache: the runsc binaries, the kernels and
+ the guest images the workers run actors with. Each actor on
+ the node links its assets from this one cache.
+ - id: ate.filecache.gc.outcome
+ stability: development
+ brief: >
+ The result of one eviction pass that had work to do. A pass that
+ finds the cache within its budget records nothing.
+ type:
+ members:
+ - id: complete
+ stability: development
+ value: complete
+ brief: The pass freed what the budget asked.
+ - id: shortfall
+ stability: development
+ value: shortfall
+ brief: >
+ The pass freed less than the budget asked. What remains is
+ linked by a consumer, such as an actor or a worker's default
+ runtime, younger than the min age, or busy. The cache cannot
+ free it.
+ - id: skipped
+ stability: development
+ value: skipped
+ brief: The pass could not size the cache and removed nothing. It retries on the next tick.
+
- id: registry.ate.stats
type: attribute_group
brief: The labels of the resource measurements of the actors.
@@ -1297,6 +1341,79 @@ groups:
The permitted values are 401, 403, 404, 429, 500, 502, 503 and 504.
Each other status, and each failure with no status, reports _OTHER.
+ - id: metric.ate.filecache.size
+ type: metric
+ metric_name: ate.filecache.size
+ instrument: updowncounter
+ unit: By
+ stability: development
+ brief: The bytes of disk that a file cache on the node occupies, by cache.
+ note: >
+ atelet reads the value on each collection as the sum of the allocated
+ size of each entry in the cache. Compare it with the budget that the
+ cache's max-bytes flag sets, --sandbox-asset-cache-max-bytes for the
+ sandbox asset cache. A value that stays over the budget with shortfall
+ passes means that the cache's consumers hold more than the budget: for
+ the sandbox asset cache, the actors and the workers' default runtimes.
+ The budget is too small, or the node hosts too many runtimes.
+ annotations:
+ substrate:
+ emitted_by: [atelet]
+ golden_signals: [saturation]
+ code_anchor: cmd/atelet/internal/filecachegc/filecachegc.go
+ cuj: The disk of a worker node fills. Is it one of the file caches?
+ attributes:
+ - ref: ate.filecache.name
+ requirement_level: required
+
+ - id: metric.ate.filecache.evictions
+ type: metric
+ metric_name: ate.filecache.evictions
+ instrument: counter
+ unit: "{entry}"
+ stability: development
+ brief: The number of entries that the eviction passes removed from a file cache on the node, by cache.
+ note: >
+ An entry is one cached file or tree, such as a runsc binary or a
+ micro-VM kernel. A pass removes only an entry that nothing links,
+ oldest first, and a dry run removes none. A rate that stays above zero
+ with a size at the budget means that the cache thrashes: an entry
+ leaves and returns on each pass, and each return pays for a download.
+ annotations:
+ substrate:
+ emitted_by: [atelet]
+ golden_signals: [saturation]
+ code_anchor: cmd/atelet/internal/filecachegc/filecachegc.go
+ cuj: Actors download their runtime again and again on one node. Does the cache evict it?
+ attributes:
+ - ref: ate.filecache.name
+ requirement_level: required
+
+ - id: metric.ate.filecache.gc.passes
+ type: metric
+ metric_name: ate.filecache.gc.passes
+ instrument: counter
+ unit: "{pass}"
+ stability: development
+ brief: The number of eviction passes over a file cache on the node that had work to do, by cache and outcome.
+ note: >
+ A pass that finds the cache within its budget is not counted. Thus a
+ series that grows means that the cache is over its budget. Read the
+ outcome: complete means that the passes keep up; shortfall means that
+ what remains is not the cache's to free; skipped means that atelet
+ cannot read the cache directory.
+ annotations:
+ substrate:
+ emitted_by: [atelet]
+ golden_signals: [saturation, errors]
+ code_anchor: cmd/atelet/internal/filecachegc/filecachegc.go
+ cuj: A file cache stays over its budget. Does the collector run, and what stops it?
+ attributes:
+ - ref: ate.filecache.name
+ requirement_level: required
+ - ref: ate.filecache.gc.outcome
+ requirement_level: required
+
# ---------------------------------------------------------------------------
# Metrics. atecontroller
# ---------------------------------------------------------------------------
diff --git a/docs/observability.md b/docs/observability.md
index 1cc0362e45..7cfa393373 100644
--- a/docs/observability.md
+++ b/docs/observability.md
@@ -208,7 +208,7 @@ The actor still reports `RUNNING` after this record; nothing moves it to `CRASHE
#### The same records over OTLP
-With `OTEL_LOGS_EXPORTER=otlp` these records go out as OTLP log events instead of stdout, so a collector reads them without knowing substrate's stdout envelope. Unset or `none` keeps them on stdout. The Helm chart selects `otlp` when `otel.logs.enabled` is true and a logs endpoint resolves. `otlp,console` writes both, as kind does for debugging. `console` here is the component's own JSON log line, not the OTel SDK's console exporter format. The value is a comma-separated list: names are trimmed, lowercased, and counted once, and an unknown or empty item logs a warning and is skipped while the rest apply. `none` with an exporter, or a value that names no known exporter, logs a warning and keeps the default, `none`. ateapi emits lifecycle events and the ateworkers usage samples, on the same switch. The ateworkers export through [the ateworker relay](#the-ateworker-otlp-relay), which carries logs, traces, and metrics. atecontroller copies `OTEL_LOGS_EXPORTER` into each worker pod, so the same selection applies to ateapi and the ateworkers.
+With `OTEL_LOGS_EXPORTER=otlp` these records go out as OTLP log events instead of stdout, so a collector reads them without knowing substrate's stdout envelope. Unset or `none` keeps them on stdout. The Helm chart selects `otlp` when `otel.logs.enabled` is true and a logs endpoint resolves, and sets `ATE_WORKER_OTEL_LOGS_EXPORTER` for the ateworkers too. `otlp,console` writes both, as kind does for debugging. `console` here is the component's own JSON log line, not the OTel SDK's console exporter format. The value is a comma-separated list: names are trimmed, lowercased, and counted once, and an unknown or empty item logs a warning and is skipped while the rest apply. `none` with an exporter, or a value that names no known exporter, logs a warning and keeps the default, `none`. ateapi emits the lifecycle events and the ateworkers the usage samples, each on its own switch; the ateworkers export through [the ateworker relay](#the-ateworker-otlp-relay), which carries logs, traces, and metrics. atecontroller sets each worker pod's `OTEL_LOGS_EXPORTER` from `ATE_WORKER_OTEL_LOGS_EXPORTER` (or its `--otel-logs-exporter` flag), not from its own value, so the kind ConfigMap sets both to `otlp,console`.
Three `event.name` values, which is the OTLP LogRecord's own field rather than an attribute:
@@ -296,6 +296,9 @@ Agent Substrate emits foundational OpenTelemetry system and server metrics to mo
| `atelet.snapshot.upload.overdue.actors` | atelet | up/down counter | actors on the node with an undurable local snapshot past `--snapshot-upload-delay` waiting for an upload to finish, bucketed by time overdue (label `ate.snapshot.upload_overdue_band`: `under_1m`, `1m_to_5m`, `5m_to_15m`, `over_15m`) |
| `atelet.snapshot.undurable.age.max` | atelet | gauge | age of the oldest local snapshot on the node that is not durable yet, counted from when the actor's state stopped being durable, or 0 when every local snapshot is durable; a healthy node stays below `--snapshot-upload-delay` plus one upload |
| `ate.imagecache.requests` | atelet | counter | image lookups in the node-local image cache, by outcome (`ate.imagecache.outcome`), with `error.type` on the `error` outcome. A miss pays for the pull and the unpack, so the hit ratio per node is a leading indicator of resume latency |
+| `ate.filecache.size` | atelet | up/down counter | bytes of disk a node-local file cache occupies, by `ate.filecache.name`, read on each collection; compare with `--sandbox-asset-cache-max-bytes` |
+| `ate.filecache.evictions` | atelet | counter | entries the eviction passes removed from a node-local file cache, by `ate.filecache.name`; a steady rate at the budget means the cache thrashes and actors pay for downloads again |
+| `ate.filecache.gc.passes` | atelet | counter | eviction passes over a node-local file cache, by `ate.filecache.name`, that had work to do, by outcome (`ate.filecache.gc.outcome`: `complete`, `shortfall`, `skipped`); a pass within budget is not counted |
The table lists the OpenTelemetry instrument names. How a name appears in a query depends on the backend (Cloud Monitoring (GMP) / Kind collector).
@@ -312,6 +315,9 @@ For `ate.imagecache.requests`:
* `ate.imagecache.outcome` is `hit` when the node holds a complete image record — every layer directory the record names is present — and `miss` when the lookup must pull. A failed lookup is neither: `error` is a failed lookup whatever the cause, and `cancelled` or `timeout` is the caller giving up, as on `ate.router.outcome`. So the hit ratio is `hit / (hit + miss)`, with failures and abandoned lookups out of the denominator.
* `error.type` is present only on the `error` outcome, and carries the registry's own HTTP status for its rejection, from a fixed set: `401`, `403`, `404`, `429`, `500`, `502`, `503`, `504`. The set is an allow-list because the registry client reports whatever the remote returned. Each other status, and each failure that carries no status, reports `_OTHER`.
+For the `ate.filecache.*` instruments, one series per cache under `ate.filecache.name`, today `sandbox_asset` for the sandbox asset cache (see the [Sandbox Asset Cache](sandbox-asset-cache.md) page):
+* The cache evicts only what nothing links: an actor's staged assets and each worker's default runtime are hard links into the cache, and an entry with a link stays whatever the pressure. `ate.filecache.gc.outcome` is `shortfall` when a pass could not reach the budget for that reason; the size then measures what the actors and the defaults on the node need, not waste. `skipped` means atelet could not read the cache directory and did nothing.
+
`ate.workerpool.namespace` and `ate.workerpool.name` identify a pool together, on every instrument that names one. A WorkerPool is a namespaced resource, so the name on its own merges same-named pools from different namespaces into one series. The pair means that capacity (`ate.workerpool.workers`, `ate.workerpool.desired_workers`, `ate.workerpool.ready_workers`) joins to demand (`ate.scheduler.assignment.duration`, `ate.actor.lifecycle.operation.duration`, `ate.actor.crashes`) by pool.
**No keys** means the operation has no pool. The actor-centric instruments omit the pair, so a crash before the actor reached a worker, or the `no_capacity` outcome, names no pool. No instrument sends the pair with both keys empty.
@@ -382,6 +388,8 @@ A queue that has never processed an item bridges as an exponential histogram wit
ateapi, atelet, atenet-router and the credential provider also serve every instrument on their Prometheus `/metrics` endpoint. A cluster that scrapes those endpoints sets `OTEL_METRICS_EXPORTER=prometheus` on the components, so each series reaches the backend once. With `prometheus` the components install no OTLP metric reader and keep the Prometheus one; traces and logs are unaffected. `none` stops the push too. For now `/metrics` stays on for every value, including `otlp` and `none`. This is temporary until the probes move off `:9090`; `otlp` and `none` will then turn it off ([#2067](https://github.com/agent-substrate/substrate/issues/2067)). The variable takes a comma-separated list, with the same rules as `OTEL_LOGS_EXPORTER`. Unset is `otlp,prometheus`, or `otlp` alone on ateworker. `otlp` keeps the push, and `prometheus` or `none` stops it. ateworker serves no `/metrics`, so there `prometheus` is skipped as unknown with a warning, and the push stays on unless the variable is `none`.
+`ate-controller` does not propagate its own value to the worker pods: ateworker serves no `/metrics`, and `none` would leave it exporting no metrics at all. To set the variable on ateworker, set `ATE_WORKER_OTEL_METRICS_EXPORTER` in `ate-otel-config` (or the controller's `--otel-metrics-exporter` flag).
+
atecontroller's `:8080` listener follows the variable. When the variable is unset or `otlp,prometheus`, atecontroller pushes every metric over OTLP, and `:8080` serves the controller-runtime families and the `ate.workerpool.*` instruments. With `otlp`, atecontroller pushes every metric and closes `:8080`. With `prometheus`, atecontroller does not push, and `:8080` serves the controller-runtime families and the `ate.workerpool.*` instruments. With `none`, atecontroller does not push and closes `:8080`.
### Local Metrics with Prometheus (Kind Cluster)
@@ -412,7 +420,7 @@ Distributed tracing tracks the end-to-end flow of requests as they pass through
Agent Substrate samples traces by default. Each component roots parentless requests at a per-component ratio (10% on the control plane components, 1% at the atenet router), overridable per component through the standard `OTEL_TRACES_SAMPLER` / `OTEL_TRACES_SAMPLER_ARG` environment variables. Every component uses a parent based sampler, so a client can also force a request to be traced end to end (e.g. via the `--trace` flag). Agent Substrate leverages OpenTelemetry (OTel) for context propagation across the call stack. Each traced request generates a unique trace hash/ID, which you can use to inspect the detailed request lifecycle and span hierarchy inside Google Cloud Trace or Jaeger. See the per-component defaults table in [Tracing Best Practices](dev/best-practices/tracing.md).
-A component with `OTEL_TRACES_EXPORTER=none` exports no spans. It still samples and propagates trace context, so the traces of the components around it stay whole. The variable takes a comma-separated list, with the same rules as `OTEL_LOGS_EXPORTER`, and knows `otlp` (the default) and `none`.
+A component with `OTEL_TRACES_EXPORTER=none` exports no spans. It still samples and propagates trace context, so the traces of the components around it stay whole. The variable takes a comma-separated list, with the same rules as `OTEL_LOGS_EXPORTER`, and knows `otlp` (the default) and `none`. It does not cover the worker pods: set `ATE_WORKER_OTEL_TRACES_EXPORTER` in `ate-otel-config` (or `ate-controller`'s `--otel-traces-exporter` flag) for them. The atelet relay forwards ateworker's spans as they are and does not apply atelet's own value.
### Local Tracing with Jaeger (Kind Cluster)
@@ -440,6 +448,8 @@ To visualize traces locally:
> ateworker carries no manual spans — its only instrumentation is the `otelgrpc` interceptor on the gRPC surface `atelet` calls. So it produces a span for an actor lifecycle operation (`suspend`, `resume`) and nothing at all for a read like `kubectl ate get actor`. Its sampler is parent based, so a lifecycle command is traced end to end into ateworker whenever `ateapi` roots a sampled trace, which the kind overlay makes unconditional; the per-component ratio never enters into it. To check whether ateworker exported its spans through the [OTLP relay](#the-ateworker-otlp-relay) rather than falling back to direct network egress, inspect the span's resource attributes: `ate.otlp.relay` will be set to `"relay"` (instead of `"direct"`).
+> `ateapi` traces its PostgreSQL store. Every statement issued while serving a sampled request becomes a client span under the RPC span, named by its query summary (`SELECT actors`, `UPDATE workers`), with the parameterized statement in `db.query.text` and never the argument values. Transaction control appears as well, as `begin` and `commit` spans, so commit latency is visible. Store spans only join a trace that is already sampled: they follow `ateapi`'s sampler and have no switch of their own, and background work such as outbox polling and lease upkeep produces none.
+
> **Developer Guide:** For detailed instructions on configuring OpenTelemetry tracer providers, middleware, and exporters in your servers or clients, please refer to the [Tracing Best Practices](dev/best-practices/tracing.md) guide.
---
@@ -472,11 +482,11 @@ ateworker ──OTLP/gRPC over unix socket──► atelet relay ──OTLP/gRPC
The socket sits in the `BasePath` hostPath already mounted into both, so nothing new is mounted. `atelet` is a DaemonSet, so every ateworker on a node shares one relay, and the many per-pod collector connections collapse into one per node. Four things motivate it: the worker pod runs untrusted agent code and will not need egress to the collector once direct fallback is phased out; the connection count drops; ateworker's own telemetry stays clear of the transparent egress redirect it installs for the actor; and `atelet` outlives the worker pod, so spans still queued at teardown are not lost with it.
-The relay is best-effort. If the socket is absent when ateworker starts — `atelet` not up yet, `--otlp-relay-socket=""`, or no collector configured for the relay to forward to — ateworker logs it and exports directly to `OTEL_EXPORTER_OTLP_ENDPOINT` as before. That fallback is decided once at startup, not per export, and is stamped on telemetry as the `ate.otlp.relay` resource attribute (`relay` vs `direct`).
+The relay is best-effort. If the socket is absent when ateworker starts — `atelet` not up yet, `--otlp-relay-socket=""`, or no collector configured for the relay to forward to — ateworker logs it and exports directly to `OTEL_EXPORTER_OTLP_ENDPOINT`. Without that variable either, ateworker exports no traces, metrics, or logs, rather than dialing the SDK default `localhost:4317`, where no collector runs, and logs a warning per signal with `reason="no OTLP relay or endpoint"`. That fallback is decided once at startup, not per export, and is stamped on telemetry as the `ate.otlp.relay` resource attribute (`relay` vs `direct`). On an install that sets no endpoint for `ate-controller` to propagate, an ateworker that starts before its node's relay socket exists therefore exports nothing until the worker pod restarts.
> **Note on Network Egress Lockdown:** Complete network policy lockdown of worker pod egress to the collector is planned as a Phase 2 milestone once the relay path is fully proven and direct fallback is deprecated. While the fallback path remains active, worker pods retain network egress to the collector and `ate-controller` continues to inject `OTEL_EXPORTER_OTLP_ENDPOINT`.
-For verified ateworker sources, the relay forwards each request verbatim rather than decoding and re-exporting, which is what keeps every ateworker its own service in Jaeger/GCP Trace instead of being absorbed into `atelet`'s. `ate-controller` injects `k8s.pod.name`, `k8s.namespace.name`, `k8s.pod.uid`, `k8s.node.name`, and `service.instance.id` directly into `OTEL_RESOURCE_ATTRIBUTES` via the Kubernetes Downward API; because the relay preserves resources verbatim, Kubernetes attributes remain intact even though the TCP connection to the collector originates from `atelet` rather than the worker pod IP (bypassing reliance on collector-side IP-based `k8sattributes` enrichment).
+For verified ateworker sources, the relay forwards each request verbatim rather than decoding and re-exporting, which is what keeps every ateworker its own service in Jaeger/GCP Trace instead of being absorbed into `atelet`'s. `ate-controller` injects `k8s.pod.name`, `k8s.namespace.name`, `k8s.pod.uid`, `k8s.node.name`, and `service.instance.id` directly into `OTEL_RESOURCE_ATTRIBUTES` via the Kubernetes Downward API, whether or not it has an endpoint to propagate; because the relay preserves resources verbatim, Kubernetes attributes remain intact even though the TCP connection to the collector originates from `atelet` rather than the worker pod IP (bypassing reliance on collector-side IP-based `k8sattributes` enrichment).
Verbatim forwarding is restricted to known ateworker sources and refuses anything else with `PermissionDenied`. Actor telemetry is what that excludes: actors share a hostname (`actor`) and an interior IP, so their series merge unless identity is injected from outside the actor ([#761](https://github.com/agent-substrate/substrate/issues/761)) — a rewrite, which will be implemented as an explicit rewriting path alongside this forwarder.
diff --git a/docs/operator-install.md b/docs/operator-install.md
index 08bfc93d68..104a7d4e25 100644
--- a/docs/operator-install.md
+++ b/docs/operator-install.md
@@ -319,6 +319,31 @@ demos, the [benchmarks](../benchmarking/README.md#settings), and the
| `record.dir` | `ATE_RECORD_DIR` | `--record-dir` | — | (global) |
| `rolloutTimeout` | `ATE_INSTALL_ROLLOUT_TIMEOUT` | `--rollout-timeout` | `1m0s` | (global) |
| `storage.bucketName` | `BUCKET_NAME` | `--storage-bucket-name` | — | (global) |
+| `storage.snapshotLocation` | `SNAPSHOT_LOCATION` | `--snapshot-location` | — | (global) |
+
+## Snapshot storage
+
+ate-api-server stores every actor and tag snapshot under a cluster-wide
+`--snapshot-location`, an object-storage URI such as `gs://BUCKET/PREFIX/`.
+Set it with `SNAPSHOT_LOCATION` (or ate-setup's `--snapshot-location`). It has
+no default, and ate-setup fails before deploying without it unless the
+`PerTemplateSnapshotLocation` preview gate is enabled. If you deploy ate-api-server
+some other way, pass the flag yourself; without the gate, the server refuses to
+start without it.
+
+With `--kind`, ate-setup defaults `storage.snapshotLocation` to
+`gs:///` (`gs://ate-snapshots/`, the bucket the in-cluster
+rustfs creates) and ignores an ambient `SNAPSHOT_LOCATION`. On GKE, ate-setup
+does not create the bucket or grant access to it: run
+`go run ./tools/setup-gcp create bucket` and
+`go run ./tools/setup-gcp create iam` first, or `bootstrap` for both (see
+[tools/setup-gcp](../tools/setup-gcp/README.md)).
+
+With the `PerTemplateSnapshotLocation` preview gate, an ActorTemplate may set
+`snapshotConfig.storageLocation` to store its snapshots elsewhere; templates
+that leave it empty use `--snapshot-location`. Without the gate,
+ate-api-server ignores `snapshotConfig.storageLocation`. See
+[Snapshot Storage Layout](api-guide.md#snapshot-storage-layout).
## Secrets
diff --git a/docs/sandbox-asset-cache.md b/docs/sandbox-asset-cache.md
new file mode 100644
index 0000000000..4b1976fab3
--- /dev/null
+++ b/docs/sandbox-asset-cache.md
@@ -0,0 +1,220 @@
+# Sandbox Asset Cache (atelet)
+
+## Summary
+
+Every actor runs on sandbox binaries that come from a cluster-scoped
+`SandboxConfig`: a gVisor release, or a micro-VM kernel, VMM and guest image.
+`atelet` keeps **one node-local, content-addressed cache** of these assets and
+gives each actor **hard links** into it. Nothing is copied per actor, an asset
+is downloaded once per node however many actors and workers use it, and the
+cache evicts to a byte budget without ever taking an asset something still
+uses.
+
+This page describes what is on disk, how an actor gets its assets, how
+`atelet` decides which assets a node needs, and how eviction and its metrics
+behave. For the design history see the node-local artifact cache proposal in
+issue #690.
+
+## What is on disk
+
+Everything lives under `/var/lib/ate`, on one filesystem, because hard links
+cannot cross mounts.
+
+| Path | What it holds |
+|---|---|
+| `sandbox-asset-cache/` | The cache (`--sandbox-asset-cache-dir`). `entries//data` is a cached file or extracted tree, published read-only with execute bits kept; `tmp/` holds in-flight downloads; `.rm-*` are retired entries awaiting removal. An entry directory's mtime is its last-use clock. |
+| `actors//sandbox-assets/` | The actor's assets, each a hard link into the cache, named by asset name. The only location a worker runs a sandbox binary from; the worker checks that a request names the actor's own directory. `sandbox-assets.staging/` beside it exists only while a Run fetches. |
+| `workers//sandbox-runtimes.json` | What each worker advertises to the control plane, written by `atelet`. |
+| `workers//default-runtime//` | The worker's default runtimes, one directory per default its file advertises, by bundle hash, one per sandbox class, linked in from the cache by the prewarmer. These links are what keeps a default runtime cached while no actor runs it; the syncer removes a directory when the file stops naming its runtime, and they all go with the worker's directory when the pod does. |
+| `sandbox-asset-scratch/` | Short-lived destinations for the one fetch that has no actor and no worker directory to link into: the `runsc` binary the worker sandbox syncer probes for CPU features. Wiped at startup. |
+| `sandbox-runtime-index.json` | The runtime index serialized: every runtime a worker on this node has advertised, by hash, with the asset record the hash was computed from. |
+
+A cache entry is keyed by the asset's sha256, so the same binaries under
+another name or from another bucket are one entry. The store never
+invalidates: changed content arrives under a new key.
+
+## How an actor gets its assets
+
+1. ate-api-server places the actor on a worker and names, in `Run` or
+ `Restore`, the **sandbox runtime** that worker registered. Its
+ `sandbox_bundle_hash` attribute is the hash of one `SandboxConfig`
+ version's assets and pause image.
+2. `atelet` resolves that hash in its **runtime index** to the asset record
+ the hash was computed from. A hash the index does not hold fails the
+ request with `FailedPrecondition`: the actor reached a node whose workers
+ never advertised that runtime, which is a placement error to surface, not
+ something to serve from whatever the request describes.
+3. Each asset is looked up in the cache by its sha256. A miss downloads it
+ through the object-store plugin's asset provider (anonymously from a
+ public `gs://` bucket when the plugin cannot serve it), verifies the
+ sha256 twice, extracts a gVisor release, and publishes the entry
+ atomically. Concurrent callers for one asset share one download.
+4. The asset is **hard-linked** into the actor's `sandbox-assets/`
+ directory. `Run` stages into `sandbox-assets.staging/` first and swaps the
+ set in only after the actor's previous state is reset, so a failed fetch
+ leaves a live activation as it was. `Restore` resets the directory and
+ stages into it. `Checkpoint` leaves the links in place: the actor stays
+ resumable on the node.
+
+The link is also the cache's only signal that an entry is in use.
+
+## How a node knows which runtimes it needs
+
+The **worker sandbox syncer** watches this node's worker pods, their
+`WorkerPool`s and `SandboxConfig`s. For each worker it plans the config its
+pool runs, hashes every enabled version, and:
+
+- **registers** every hash in the runtime index, with its record, and writes
+ the index file durably, *before* anything else, so a hash is resolvable
+ before the worker can register it with the control plane;
+- probes what a snapshot taken on each runtime must find again (CPU features
+ for gVisor, host CPU identity for micro-VM) and writes the worker's
+ `sandbox-runtimes.json`, from which the worker registers;
+- records what the worker now advertises, which is what the written file
+ says;
+- makes the worker's `default-runtime/` directory match the file's
+ **default** runtimes, and hands each to the **prewarmer**, which fetches
+ its assets into the cache, links them into the runtime's directory, and
+ pulls its pause image, so the first actor on the node finds a warm cache.
+
+A `SandboxConfig` or `WorkerPool` change registers the affected pools'
+runtimes at once and rewrites their files after a jitter of up to thirty
+seconds, so the fleet does not re-register in one burst. After an `atelet`
+restart the index loads from its file before the port opens, so a request for
+a runtime a worker registered earlier resolves with no wait on the syncer. The
+in-memory map is the read path; the file is write-through persistence.
+
+Only default runtimes are prewarmed and linked. The first actor restored on
+another enabled version a worker advertises pays that version's download.
+
+### The default runtime directories
+
+A worker's default runtime must stay cached while no actor runs it, and the
+cache has one way to say "in use": a hard link. So the prewarmer leaves its
+links in a directory that belongs to the worker, and the syncer decides
+which directories exist. Nothing counts references and nothing prunes.
+
+- Once a worker's file is written and registered, the syncer makes
+ `default-runtime/` hold one directory per default the file names, by
+ bundle hash, and nothing else: the directory of a default the file no
+ longer names is removed, which unlinks that runtime, and an empty one is
+ created for each default that lacks one. Then it hands the prewarmer one
+ item per default, the hash and its directory.
+- The hand-over comes after the build on purpose. A build that fails, as a
+ permanent `runsc` probe failure does, leaves the file and so the
+ directories as they were, with the previous default still linked and still
+ advertised. The prewarm starts late by only a little: a gVisor default's
+ release is fetched by the build's probe, so the prewarm is links and the
+ pause image pull, and a micro-VM build takes milliseconds.
+- The prewarm fills what exists and creates nothing. It checks that its
+ directory is there, stages into it, and on an error checks again: a
+ directory gone, after a later sync or the pod's removal, means the item
+ was superseded, and it ends without a retry and without recreating
+ anything. A directory the syncer already filled costs a stat per asset, so
+ a re-sync with an unchanged default downloads nothing and leaves the links
+ alone.
+- A failed download retries with backoff and gives up after eight attempts.
+ The directory stays, empty or partial, until the next sync; an actor
+ placed on that default meanwhile fetches on demand and shares the
+ download.
+- A pod's removal removes its whole directory, links included. After a
+ restart the directories of pods still on the node keep protecting their
+ defaults with no wait on the syncer, and the startup sweep removes the
+ directories of pods that are gone.
+
+The old default is not kept while the new one downloads: by then the file
+names the new default and the worker has registered it, so nothing new is
+placed on the old one. Actors already running on it hold their own links,
+and restores of snapshots taken on it go through the on-demand path like
+every non-default version. Several pods of one pool each hold their own
+links to the same entries; the entry becomes evictable when the last of
+them goes. One queue and one worker drain the prewarm items in order, so
+defaults download one at a time across the node and never compete for its
+bandwidth.
+
+## Eviction
+
+With a budget set, an eviction pass runs every
+`--sandbox-asset-cache-gc-period`. It sizes the cache, and when that exceeds
+`--sandbox-asset-cache-max-bytes` it asks the store to free down to 90% of
+the budget. The store applies three rules to
+each entry, in order:
+
+1. **Linked, never evicted.** Any hard link from an actor's directory or a
+ worker's default runtime directory keeps the entry. Retiring it would free
+ nothing while the link lasts and would make the next caller download a
+ second copy beside it. The link count is checked again under the cache's
+ lock right before an entry is retired, so a link gained during the pass
+ vetoes it.
+2. **Younger than `--sandbox-asset-cache-min-age`, skipped.** This covers the
+ moment between an entry's publication and its first link.
+3. **Otherwise eligible, oldest first**, until the freed bytes reach the
+ target.
+
+Removal happens after all retires, outside the cache's locks: an entry is
+renamed to `.rm-*` and deleted. A hit racing the pass vetoes its entry.
+
+| Flag | Default | Meaning |
+|---|---|---|
+| `--sandbox-asset-cache-dir` | `/var/lib/ate/sandbox-asset-cache` | Where the cache lives. Must share a mount with the actor directories. |
+| `--sandbox-asset-cache-min-age` | `1h` | Entries younger than this are never evicted. |
+| `--sandbox-asset-fetch-timeout` | `5m` | Bound on one download, and for a gVisor release its extraction. |
+| `--sandbox-asset-cache-gc-period` | `5m` | How often a pass runs. `0` disables passes. |
+| `--sandbox-asset-cache-max-bytes` | `0` | The budget. `0` means no budget: the cache only grows, and `atelet` warns so at startup. |
+| `--sandbox-asset-cache-gc-dry-run` | `false` | Log what a pass would evict without deleting. |
+
+## Metrics
+
+The eviction loop and its instruments are shared by every file cache on the
+node (`cmd/atelet/internal/filecachegc`), so the instruments carry the cache
+in `ate.filecache.name`; this cache is `sandbox_asset`.
+
+| Instrument | Type | What it says |
+|---|---|---|
+| `ate.filecache.size` | up/down counter, bytes, by `ate.filecache.name` | Disk the cache occupies, read on every collection. Compare with the budget. |
+| `ate.filecache.evictions` | counter, entries, by `ate.filecache.name` | Entries passes removed; a dry run counts none. A steady rate with the size at the budget means the cache thrashes: a runtime leaves and returns on each pass, and each return pays for a download. |
+| `ate.filecache.gc.passes` | counter, by `ate.filecache.name` and `ate.filecache.gc.outcome` | Passes that had work to do. `complete`: the pass reached its target. `shortfall`: it freed less than asked. `skipped`: it could not size the cache and did nothing. A pass within budget is not counted. |
+
+A **shortfall** is not a fault of the pass. Everything left is linked by an
+actor or by a worker for its default, younger than the min age, or busy. The
+size then measures what the actors and defaults on this node need, not
+waste; the answer is a larger budget or fewer runtimes on the node. The pass
+warns on the first few shortfalls and then about hourly.
+
+The eviction and pass counters only appear once a pass had work to do, so on
+an install without a budget only the size instrument has a series. The e2e
+metrics suite asserts that one.
+
+## Runbook: a node is over budget
+
+1. Read `ate.filecache.size` for `ate.filecache.name=sandbox_asset` against
+ `--sandbox-asset-cache-max-bytes`, and `ate.filecache.gc.passes` by
+ outcome for the node.
+2. `shortfall`: list `/var/lib/ate/workers/*/default-runtime/` for the
+ defaults the node's workers hold, and
+ `ls -l /var/lib/ate/actors/*/sandbox-assets/` for what actors hold. If
+ those account for the size, raise the budget or reduce the runtimes the
+ node's pools run. Do not delete either by hand: the next sync rebuilds a
+ worker's links, and a running actor's links are its runtime.
+3. `skipped`: `atelet`'s log names the error from sizing the cache directory.
+4. No passes at all while over budget: the period is `0`, or `atelet` is
+ older than the budget flag. The startup log line `Sandbox asset cache has
+ no budget` says the budget is unset.
+5. To see what a pass would take without taking it, restart `atelet` with
+ `--sandbox-asset-cache-gc-dry-run` and read the pass log lines.
+
+## Not covered here, deliberately
+
+- **Snapshots.** The same `filecache` store is designed to serve shared
+ snapshot restores, the golden and clone images many actors start from.
+ That milestone is separate and not yet merged.
+- **One GC driver for both caches.** The image layer cache has its own
+ eviction loop with the same shape but no metrics. Sharing the driver would
+ give it the pass and eviction instruments; it is a follow-up.
+- **Which runtime an actor gets.** The `WorkerPool` resolves the
+ `SandboxConfig`, the worker registers the config's versions, and
+ ate-api-server sends the runtime the assigned worker registered. An
+ `ActorTemplate` has no say in it. The cache serves whatever runtime the
+ request names and has no part in choosing it.
+- **Releasing index entries.** The runtime index never forgets a hash; a node
+ sees a handful of configs in its lifetime, and the file stays small.
diff --git a/docs/security.md b/docs/security.md
index afbcf3a180..22ac45f46a 100644
--- a/docs/security.md
+++ b/docs/security.md
@@ -2,7 +2,7 @@
Substrate delivers zero trust for agents. Every agent has an identity, but no credentials, no open network, and no way out of the sandbox.
-Agent Substrate executes untrusted actors at scale in a multi-tenant environment where workloads are dynamic and memory states migrate continuously across physical infrastructure. All security controls must operate within Substrate's cold-start and resume latency budget. This document defines the high-level security principles. More detailed security risks are documented [in the threat model](threat-model.md).
+Agent Substrate executes untrusted actors at scale in a multi-tenant environment where workloads are dynamic and memory states migrate continuously across physical infrastructure. All security controls operate within Substrate's cold-start and resume latency budget. This document defines the high-level security principles. For identity and credential configuration, see the [Authentication Guide](authentication.md). Detailed threat scenarios and mitigating invariants are documented [in the threat model](threat-model.md).
## Security Scope
@@ -44,39 +44,39 @@ Substrate is designed to safely run mutually hostile, untrusted actors from disp
| **Per-Actor Storage Segregation** | Storage and persistent volume state must be strictly segregated per actor. Actors cannot traverse, enumerate, or access persistence belonging to other actors. Restoring a snapshot must re-validate the actor's current tenant authorization before execution resumes. | Met (Segregation)
Future (Per-Actor Encryption, Restore Authorization) |
| **Per-Actor Network Isolation** | Guest network stacks are segregated per actor. Lateral peer-to-peer network communication between co-located actors on the same host or across the cluster is blocked by default; inter-actor communication must route through authenticated, policy-enforced gateways. | Met (Private netns, atunnel) |
| **Delete Guarantee** | Reset, scrub, and sanitize local ephemeral volumes, cached state, and worker runtime environments between actor scheduling cycles to prevent cross-tenant data leakage. | In Progress (atelet resetActorDirs) |
-| **Node Blast Radius Containment** | A compromise of an individual sandbox guest or node supervisor restricts impact to the actors and identities actively co-located on that node. | Future |
-| **Atespace Separation** | The Substrate equivalent of a Kubernetes namespace that is used to group resources and partition object storage snapshot paths. It does not yet enforce runtime API authorization, inter-atespace network policies, or resource quotas, but an embedded OpenFGA relationship-based access control (ReBAC) model is actively being integrated. | In-Progress (OpenFGA) |
+| **Node Blast Radius Containment** | A compromise of an individual sandbox guest or node supervisor restricts impact to the actors and identities actively co-located on that node. | In Progress (Node-attested PodIdentity, checkActorOnNode) |
+| **Atespace Separation** | The Substrate equivalent of a Kubernetes namespace that is used to group resources and partition object storage snapshot paths. OpenFGA relationship-based access control (ReBAC) is integrated for API authorization, but it does not yet enforce inter-atespace network policies or resource quotas. | In-Progress (OpenFGA ReBAC; network policies and quotas pending) |
Other forms of tenancy:
- **[Possible, but not currently supported] Multiple Substrate installations inside a single Kubernetes cluster**, with nodes separated by taints/tolerations. This gets you Substrate API separation but gives up some node bin-packing efficiency. Care needs to be taken at the Kubernetes layer to keep everything separate.
- **[May be possible, but not currently supported] Attaching multiple Kubernetes clusters to a single Substrate installation**. This may be possible with some work today, but is not supported. We’re expecting a single large K8s cluster to be capable of running 100s of millions of mostly-idle agents.
- **[Out of Scope] Multi-tenancy inside an actor**. There is no meaningful security boundary inside an actor. Actors are designed to be cheap and efficient enough to run many, including one per human user, so that it’s possible to run them without handling content for multiple users inside the same actor.
-- **[Not Supported] Per-Actor ingress/egress**. Actor-facing traffic proxies (`atenet-router` for ingress, `atenet-egress` for egress) are shared across all actors in the substrate installation by default. Due to the high cardinality and rapid lifecycle of actors, dedicated 1:1 proxies per actor are operationally and economically infeasible. While we may partition proxies further in the future to reduce sharing, by atespace for example, proxies will remain multi-tenant by default.
+- **[Not Currently Supported] Per-Actor ingress/egress**. Actor-facing traffic proxies (`atenet-router` for ingress, `atenet-egress` for egress) are shared across all actors in the substrate installation by default. Dedicated proxy-per-actor won't scale to large numbers of actors. Multiple proxies, such as per-atespace, could be used in the future.
### Identity
| Principle | Description | Status |
| :--- | :--- | :--- |
| **Zero Ambient Credentials** | Default actor sandboxes execute completely credential-free. Actors may hold credential identifiers, encrypted credentials, or unprivileged tokens that need to be hydrated or decrypted out-of-band by Substrate infrastructure before accessing external services. | Met (no credentials inside sandbox) |
-| **Distinct Workload Identity** | Every actor receives a short lifetime (1h) SPIFFE ID X.509 certificate. It is possible to distinguish an actor that was deleted and then re-created with the same name using their UID. | Met (X.509)
In-Progress (OIDC discovery) |
-| **Universal Component mTLS** | All control plane and data plane components (`ate-api-server`, `atelet`, `ateworker`, routers, tunnels) authenticate and communicate exclusively over mutual TLS using node- and service-attested identities. Unauthenticated traffic is rejected, exempting only local unauthenticated health/liveness probes (`/healthz`, `/readyz`). | Met (Mutual TLS)
In-Progress (Authz) |
+| **Distinct Workload Identity** | Every actor receives a short lifetime (1h) SPIFFE ID X.509 certificate and on-demand actor JWTs for external and egress authentication. It is possible to distinguish an actor that was deleted and then re-created with the same name using their UID. | Met (X.509, on-demand JWTs, OIDC discovery) |
+| **Universal Component mTLS** | All control plane and data plane components (`ate-api-server`, `atelet`, `ateworker`, routers, tunnels) authenticate and communicate exclusively over mutual TLS using node- and service-attested identities. Unauthenticated traffic is rejected, exempting only local unauthenticated health/liveness probes (`/healthz`, `/readyz`). | Met (Mutual TLS) |
| **Sub-Actor Provenance** | Delegated sub-actors inherit cryptographically signed lineage. Every tool invocation, data access, and egress transaction can be deterministically resolved to the root actor and execution chain. This provenance/hierarchy needs to be recorded in the system that calls substrate. | Out of Scope |
### Storage
| Principle | Description | Status |
| :--- | :--- | :--- |
-| **Untrusted Storage** | Persistence layers are treated as untrusted environments. Remote storage is protected with encryption. Local snapshot caching on storage attached to the node is multi-tenant and protected from cross-tenant attacks with encryption and strict filesystem permissions. | Future (Storage is currently partitioned but is unencrypted) |
-| **Dual-Party Snapshot Authorization** | Reading or writing snapshot storage requires dual-identity verification proving both the actor's identity and the node supervisor (`atelet`) operating on its behalf. Sandboxed guests cannot directly interact with snapshot storage. | Future |
+| **Untrusted Storage** | Persistence layers are treated as untrusted environments. Remote storage is protected with encryption. Local snapshot caching on storage attached to the node is multi-tenant and protected from cross-tenant attacks with encryption and strict filesystem permissions. | Met (Plugin isolation, SHA-256 asset verification)
Future (Per-Actor Encryption) |
+| **Dual-Party Snapshot Authorization** | Reading or writing snapshot storage requires dual-identity verification proving both the actor's identity and the node supervisor (`atelet`) operating on its behalf. Sandboxed guests cannot directly interact with snapshot storage. | Met (Supervisor-only storage access)
In Progress (Per-actor token exchange) |
### Network
| Principle | Description | Status |
| :--- | :--- | :--- |
-| **No Access by Default** | Sandboxed actors and worker nodes launch with default-deny networking and have zero network or API access to control plane services (`ate-api-server`, Kubernetes API servers, PostgreSQL state store). DNS is currently allowed outbound by default, work on that is in progress. Every control plane pod in `ate-system` also denies [ingress by default](../manifests/ate-install/ate-system-networkpolicies.yaml), admitting only the flows the control plane needs. | Met (Loopback, private netns)
In-Progress (DNS filtering, see [#2075](https://github.com/agent-substrate/substrate/issues/2075)) |
-| **Authenticated and Policy-Enforced Ingress** | Inbound invocations to actors terminate at authenticated edge routers (`atenet-router`), which authenticate callers and verify invocation policies before triggering on-demand actor cold-start or resume. | Met (atenet-router)
In-Progress (authentication) |
-| **Fail-Closed Policy-Enforced Egress Gateway** | Outbound traffic is funneled through the multi-tenant egress proxy (`atenet-egress`), which validates actor identity, enforces destination host allowlists, executes TLS inspection or prompt filtering, and dynamically injects target credentials. If an egress policy enforcement point (PEP) is unconfigured/fails egress is dropped. Unsupported protocols are dropped, see [Egress Policy Table](https://docs.google.com/document/d/1XLCo9SMROddQzwsyEhdBNWuLYFtWWZWfYvGlIO1BcUI/edit?resourcekey=0-st2SqKp-O68nv5LeIUfA-g&tab=t.0#bookmark=id.pjnob1gppauk) for specifics. | Met (atenet-egress)
In-Progress (fail closed, UDP currently passthrough) |
+| **No Access by Default** | Sandboxed actors and worker nodes launch with default-deny networking and have zero network or API access to control plane services (`ate-api-server`, Kubernetes API servers, PostgreSQL state store). DNS is filtered against allowed policy hostnames. Every control plane pod in `ate-system` also denies [ingress by default](../manifests/ate-install/ate-system-networkpolicies.yaml), admitting only the flows the control plane needs. | Met (Loopback, private netns, link-local block, DNS relay filtering, ate-system NetworkPolicies) |
+| **Authenticated and Policy-Enforced Ingress** | Inbound invocations to actors terminate at authenticated edge routers (`atenet-router`), which authenticate callers and verify invocation policies before triggering on-demand actor cold-start or resume. | Met (atenet-router static-mtls, atunnel mTLS) |
+| **Fail-Closed Policy-Enforced Egress Gateway** | Outbound traffic is funneled through the multi-tenant egress proxy (`atenet-egress`), which validates actor identity, enforces destination host allowlists, executes TLS inspection or prompt filtering, and dynamically injects target credentials. If an egress policy enforcement point (PEP) is unconfigured/fails egress is dropped. Unsupported protocols are dropped, see [Egress Policy Table](https://docs.google.com/document/d/1XLCo9SMROddQzwsyEhdBNWuLYFtWWZWfYvGlIO1BcUI/edit?resourcekey=0-st2SqKp-O68nv5LeIUfA-g&tab=t.0#bookmark=id.pjnob1gppauk) for specifics. | Met (atenet-egress fail-closed PEP, DNS filtering, non-TCP drop) |
### Authorization
@@ -90,5 +90,5 @@ Other forms of tenancy:
| Principle | Description | Status |
| :--- | :--- | :--- |
-| **Structured Audit Trail** | All lifecycle transitions (start, suspend, resume, terminate), snapshot access, gateway routing events, and policy violations generate structured telemetry linked to cryptographic actor identity and parent lineage. Token log leaks are prevented with sanitization. | In-Progress (OTLP logging, shared principle)
Future (enhanced identity attribution, log leak protection) |
-| **Dynamic Quarantine & Snapshot Tainting** | The control plane supports immediate revocation and network blackholing of misbehaving/compromised actors. Suspended states from compromised actors receive an immutable cryptographic taint, preventing resumption across the fleet. | Future |
+| **Structured Audit Trail** | All lifecycle transitions (start, suspend, resume, terminate), snapshot access, gateway routing events, and policy violations generate structured telemetry linked to cryptographic actor identity and parent lineage. Token log leaks are prevented with sanitization. | Met (OTLP Weaver registry, credential redaction) |
+| **Dynamic Quarantine & Snapshot Tainting** | The control plane supports immediate revocation and network blackholing of misbehaving/compromised actors. Suspended states from compromised actors receive an immutable cryptographic taint, preventing resumption across the fleet. | Met (Dynamic egress revocation, lifecycle halt)
Future (Snapshot tainting) |
diff --git a/docs/threat-model.md b/docs/threat-model.md
index 65db8822d0..acda297a8f 100644
--- a/docs/threat-model.md
+++ b/docs/threat-model.md
@@ -1,31 +1,18 @@
# Substrate Threat Model
-[Michael Taufen](mailto:mtaufen@google.com), [Vikas Kumar](mailto:skvikas@google.com), [Oleg Mitrofanov](mailto:gooleg@google.com)
-
-Last updated: Jun 25, 2026
+Last updated: Oct 14, 2026
# Overview
-Substrate is an early, fast moving product. It is full of debate and subject to change, including major architectural changes. It also has little to no security hardening at this time. Therefore, this threat model is focused on how to achieve security with respect to the general problem Substrate is trying to solve: Fast orchestration of a stateful AI agents. The threat model uses Substrate's current implementation and roadmap as a reference point, but focuses on threats that emerge from this overall class of system, rather than threats specific to the current implementation.
-
-# Intended Outcome
-
-* The suggestions from this threat model should be added to Substrate's official roadmap after review and agreement with the community.
-* Security review SKILLs for AI-assisted security review should be extracted from this threat model and used for continuous review on the upstream [agent-substrate/substrate](https://github.com/agent-substrate/substrate) repository.
+Agent Substrate orchestrates stateful AI agents at high scale and density on Kubernetes, multiplexing untrusted, mutually hostile actors across a shared pool of worker Pods while maintaining sub-second cold-start and snapshot resume latency. Because actors execute untrusted code and migrate across physical infrastructure, Substrate enforces VM-grade sandboxing (`gVisor` and `Cloud Hypervisor` microVMs), per-actor cryptographic identity (`SPIFFE` X.509 certificates and JWTs), zero ambient credentials inside sandboxes, default-deny networking with authenticated ingress and egress proxies, and relationship-based access control (`OpenFGA`).
-# Goals
+This threat model catalogs the security threats inherent to high-density stateful agent orchestration and defines the mitigating invariants and suggested mechanisms required to neutralize each threat. See the [Security Guide](security.md) for the high-level security architecture, principles, and implementation status.
-* Identify threats relevant to any system trying to achieve the same goal, with roughly the same basic building blocks as Substrate (Kubernetes, containers, sandboxes, snapshots, and lightweight "actors" dynamically scheduled to reusable "worker" Pods).
-* Identify general "mitigating invariants" which, if implemented, eliminate or significantly decrease the severity of the threat.
-* Suggest *possible* implementation approaches, which may change if Substrate changes.
-* Prioritize the threats, so that resources can be spent of reducing the biggest risks first.
+# Scope
-# Non-Goals
-
-* This threat model does not attempt to predict Substrate's future architectural decisions.
-* This threat model does not demand any particular approach to mitigation.
-* While shown in diagrams for reference, this threat model does not focus on AI frameworks that may be used on top of Substrate.
+* While shown in diagrams for reference, this threat model does not cover application-layer agent frameworks or harnesses running inside Substrate actors.
+* This threat model does not cover vulnerabilities contained entirely within the underlying cloud provider, host Linux kernel, Kubernetes control plane, or upstream sandbox runtimes (`gVisor`, `Cloud Hypervisor`).
# Architecture of Substrate
@@ -33,28 +20,33 @@ Substrate is an early, fast moving product. It is full of debate and subject to
# Key Components
-* **ate-api-server:** Actor creation and scheduling and credential issuance.
-* **atelet:** Per-node daemon, performs snapshot/resume.
-* **ateworker:** Per-worker Pod sidecar, running inside the worker Pod. It sets up "interior" sandboxes in the worker Pod and manages sandbox lifecycle, including image pulls. It currently uses gvisor but Substrate will support multiple microvm solutions.
-* **Worker:** Preprovisioned Pods that actors get scheduled to.
-* **Actor:** The core compute primitive, gets scheduled to/from worker via Run for cold start and Resume for snapshot resume.
-* **Actor Network:** `ateworker` creates a private point-to-point veth network for the active Actor inside a worker Pod. The Actor is not served directly from the worker Pod's port 80; ingress enters through `atunnel`'s authenticated listeners: mTLS `CONNECT` on port 8443, which the router uses by default, and HTTPS on port 443.
-* **Actor Routing:** Requests sent to `atenet-router` identify the target Actor with `ate-target-actor: /`. Host and authority values are application metadata and do not select the Actor.
-* **atenet-router:** Substrate runs Envoy with an `ext_proc` external processor to handle Actor ingress. With `--ingress-auth-mode=static-mtls`, Envoy accepts connections only on its TLS listeners and only from clients presenting a certificate that chains to the configured client CA bundle and carries one of a static list of SPIFFE IDs; any listed client may reach any Actor. The default, `deprecated-insecure`, rejects a TLS client whose certificate fails those checks but accepts one that presents none, and also serves plaintext listeners that accept any client. The ext_proc reads and validates `ate-target-actor`, calls the Substrate API to resume the Actor and obtain its current worker assignment, and selects that worker as a dynamic backend. It overwrites the routing header before forwarding. The router then connects with mTLS to `atunnel` and accepts it only with the SPIFFE ID of the assigned worker pod's namespace and ServiceAccount, so another pod running as that ServiceAccount in that namespace can still pose as the worker; `atunnel` validates the router identity and authorizes the header against the Actor currently assigned to that worker.
-* **Object Storage:** Used to store actor snapshots.
-* **Filesystem support:** Container local filesystem is saved in snapshots, future integrations likely to include networked storage.
-* **Substrate Database:** PostgreSQL. On GCP, PostgreSQL may be Cloud SQL reached through the Cloud SQL Auth Proxy sidecar (see [tools/setup-gcp/cloud-sql.md](../tools/setup-gcp/cloud-sql.md)); that path adds two egress flows from ate-api-server: HTTPS to `sqladmin.googleapis.com:443` and the proxy tunnel to the instance IP on port 3307, both authenticated via IAM and encrypted.
-* **Kubernetes:** The underlying infrastructure that Substrate runs on is expected to be Kubernetes.
+* **ate-api-server:** Control plane server managing actor lifecycle, scheduling, OpenFGA authorization, and credential issuance.
+* **ate-controller:** Kubernetes controller managing worker pool deployments and network policies.
+* **podcertcontroller:** Kubernetes controller issuing short-lived mTLS certificates for pod identity, service DNS, and database access.
+* **atelet:** Per-node supervisor daemon coordinating sandbox lifecycle, checkpoint/restore, and node-attested credentials.
+* **ateworker:** Per-worker Pod runtime daemon managing interior sandboxes (`gVisor` or `Cloud Hypervisor` microVMs).
+* **Worker:** Pre-provisioned Pods that host active actors.
+* **Actor:** The core stateful compute primitive, grouped into atespaces and scheduled onto workers via Run or Resume.
+* **Actor Network:** Private per-actor network namespace and point-to-point veth link behind `atunnel`'s authenticated listener on port 443.
+* **Actor Routing:** Ingress requests identify the target actor via `ate-target-actor: /`.
+* **atenet-router:** Envoy ingress proxy with an `ext_proc` processor that authenticates callers over mTLS, resolves worker assignments, and routes to `atunnel`.
+* **atenet-egress:** Multi-tenant Envoy egress forward proxy that enforces declarative egress policies and injects credentials.
+* **Object Storage:** Stores partitioned actor snapshots, accessed through the out-of-process `objectstore-plugin`.
+* **Filesystem Support:** Container rootfs overlays and working directories captured in snapshots and wiped on actor teardown.
+* **Substrate Database:** PostgreSQL (or Cloud SQL on GCP), accessed over TLS by `ate-api-server`.
+* **Kubernetes:** The underlying cluster infrastructure providing node isolation and network policies.
# Threats and Mitigations
**Table Schema:**
-* **Priority:** Critical, High, Medium, Low
-* **Threats:** Expected risks in a naïve implementation of Substrate.
-* **Mitigating Invariants:** High level properties which, if true, mitigate the threat.
-* **Suggested Concrete Mitigations:** Specific options for implementing the mitigating invariants, based on current understanding of Substrate.
-* **Notes:** Additional relevant information.
+* **Priority:** Critical, High, Medium, Low
+* **Threats:** Expected risks in a naïve implementation of Substrate.
+* **Mitigating Invariants:** High-level properties which, if true, mitigate the threat.
+* **Suggested Concrete Mitigations:** Specific options and architectural mechanisms for implementing the mitigating invariants.
+* **Notes:** Contextual notes, edge cases, and design considerations.
+
+Implementation status is intentionally not tracked in this table. High-level principle status is maintained in the [Security Guide](security.md), while threat-by-threat mitigation quality is dynamically evaluated against the codebase by automated security review agents in CI (see `.agents/skills/security-status-report/`). Recording static status claims in this reference catalog would risk biasing automated evaluators away from independently verifying the code.
## Attacks from External Networks
diff --git a/docs/upgrade.md b/docs/upgrade.md
index ed65431c8d..f687e61f52 100644
--- a/docs/upgrade.md
+++ b/docs/upgrade.md
@@ -14,14 +14,12 @@ If an upgrade goes wrong, you can roll back to the release you upgraded from, an
These rules and the step order below come from the [version skew policy](version-skew-policy.md).
-
-
| Step | What changes | What running actors see | How long |
|---|---|---|---|
| 1 | podcertificate-controller | Nothing | A minute |
| 2 | atelet, one node at a time | Nothing | Seconds to about 6 minutes per node |
| 3 | workers | `SIGTERM`, then a 30-minute window to be suspended before `SIGKILL` | Seconds to hours per pool |
-| 4 | ate-api-server, the CRDs and ate-controller, atenet, SandboxConfig, then the rest of the system bundle | API calls, long requests and outbound connections can be cut once. See step 4 | A few minutes |
+| 4 | ate-api-server and ate-idp-server, the CRDs and ate-controller, atenet, SandboxConfig, then the rest of the system bundle | API calls, long requests and outbound connections can be cut once. See step 4 | A few minutes |
Rollback is the same list in reverse, from the old release. An actor that crashes along the way goes back to its last snapshot with `kubectl ate revert actor -a `.
@@ -176,17 +174,18 @@ What to expect:
### Step 4. Upgrade the Substrate control plane
-#### 4a. ate-api-server
+#### 4a. ate-api-server and ate-idp-server
The first new API server pod migrates the database schema, and rolling back does not undo that. Back up the database first: with `pg_dump` for the bundled one, or an on-demand backup for Cloud SQL. With an external database, check that `ATE_API_POSTGRES_READ_WRITE_CONNECTION_STRING` is set in this shell: without it, `deploy apiserver` points the API server at the bundled database.
```bash
go run ./cmd/ate-setup deploy apiserver --config ~/ate-upgrade/new.yaml
kubectl -n ate-system rollout status deploy/ate-api-server
+kubectl -n ate-system rollout status deploy/ate-idp-server
kubectl ate get actors -A
```
-- **Done:** rollout status prints `successfully rolled out` and `kubectl ate get actors -A` lists your actors.
+- **Done:** both print `successfully rolled out` and `kubectl ate get actors -A` lists your actors.
- **Stuck:** rollout status does not finish, or a new pod keeps restarting. [Roll back step 4](#roll-back-step-4).
- **API clients:** an API call can fail once with a retriable error while an API server pod restarts. Retry it.
diff --git a/docs/version-skew-policy.md b/docs/version-skew-policy.md
index 68663bcf1a..da8436baf0 100644
--- a/docs/version-skew-policy.md
+++ b/docs/version-skew-policy.md
@@ -19,6 +19,7 @@ Patch releases add no API fields, public or internal, and no schema migrations.
## Components
* `ate-api-server`: serves the public API, calls `atelet`, talks to the database.
+* `ate-idp-server`: publishes the keys that verify actor JWTs. It shares only the actor JWT key pool with `ate-api-server`, and is upgraded with it.
* `ate-controller`: renders WorkerPools into Deployments, registers workers with `ate-api-server`.
* `atelet`: one pod per node (DaemonSet), drives the `ateworker` processes on its node.
* Workers: the pods of a WorkerPool, running `ateworker` and the actors.
@@ -115,7 +116,7 @@ To move from N to N+1, in this order, waiting for each step to finish:
1. `podcertificate-controller`.
2. `atelet`, node by node.
3. Every WorkerPool's `spec.workerImage`, which rolls its workers.
-4. `ate-api-server`. The first N+1 replica migrates the schema.
+4. `ate-api-server` and `ate-idp-server`. The first N+1 `ate-api-server` replica migrates the schema.
5. The CRDs, then `ate-controller`.
6. `atenet`.
7. The SandboxConfig admission policy and the default SandboxConfig.
diff --git a/hack/ate-dev-env.sh.example b/hack/ate-dev-env.sh.example
index 20a56f4f5a..da90304479 100644
--- a/hack/ate-dev-env.sh.example
+++ b/hack/ate-dev-env.sh.example
@@ -45,6 +45,8 @@ export ENABLE_NESTED_VIRTUALIZATION=true
export KUBECTL_CONTEXT=
export BUCKET_NAME=snapshot-substrate-test-${PROJECT_ID}
+# Cluster-wide location ate-api-server stores snapshots under.
+export SNAPSHOT_LOCATION="gs://${BUCKET_NAME}/"
export KO_DOCKER_REPO="gcr.io/${PROJECT_ID}/ate-images"
diff --git a/hack/install-ate.sh b/hack/install-ate.sh
index f4823e1da4..54020f8b89 100755
--- a/hack/install-ate.sh
+++ b/hack/install-ate.sh
@@ -135,7 +135,7 @@ usage() {
echo "Infrastructure components:"
echo ""
echo " --deploy-atelet Deploy atelet only"
- echo " --deploy-ate-apiserver Deploy ate-api-server only"
+ echo " --deploy-ate-apiserver Deploy ate-api-server and ate-idp-server"
echo " --deploy-atenet Deploy atenet only"
echo " --delete-atenet Delete atenet only"
echo ""
diff --git a/hack/run-e2e-kind.sh b/hack/run-e2e-kind.sh
index d23c29d83d..5585228b9d 100755
--- a/hack/run-e2e-kind.sh
+++ b/hack/run-e2e-kind.sh
@@ -35,7 +35,10 @@ export KO_DOCKER_REPO="${KO_DOCKER_REPO:-localhost:5001}"
# only build fixtures for the host arch, like install-ate-kind.sh
export KO_DEFAULTPLATFORMS="${KO_DEFAULTPLATFORMS:-linux/$(go env GOARCH)}"
# default bucket name for local deployment (served by in-cluster rustfs)
-export BUCKET_NAME="${BUCKET_NAME:-ate-snapshots}"
+export BUCKET_NAME="ate-snapshots"
+# cluster-wide snapshot location ate-api-server is installed with; matches
+# the kind profile's default in ate-setup
+export SNAPSHOT_LOCATION="gs://${BUCKET_NAME}/"
# target the local Kind cluster's context
KIND_CLUSTER_NAME="${KIND_CLUSTER_NAME:-kind}"
export KUBECTL_CONTEXT="${KUBECTL_CONTEXT:-kind-${KIND_CLUSTER_NAME}}"
diff --git a/internal/actoridjwt/actoridjwt.go b/internal/actoridjwt/actoridjwt.go
index b3bb9df14e..71c4a6d7a1 100644
--- a/internal/actoridjwt/actoridjwt.go
+++ b/internal/actoridjwt/actoridjwt.go
@@ -16,6 +16,7 @@ package actoridjwt
import (
"encoding/json"
+ "errors"
"fmt"
"time"
)
@@ -84,6 +85,11 @@ type SubstrateClaims struct {
Atespace string
ActorName string
ActorUID string
+ // Node is the node ate-api-server minted a worker-for-actor token for,
+ // the one holding the actor at that moment. A relying party compares it
+ // with where the actor is when the token is used. Empty on an actor's
+ // own token, which is not tied to a node.
+ Node string
}
type WireClaims struct {
@@ -118,6 +124,7 @@ type WireSubstrateClaims struct {
Atespace string `json:"atespace,omitempty"`
ActorName string `json:"actorName,omitempty"`
ActorUID string `json:"actorUID,omitempty"`
+ Node string `json:"node,omitempty"`
}
func ClaimsToWire(claims *Claims) (*WireClaims, error) {
@@ -139,9 +146,13 @@ func ClaimsToWire(claims *Claims) (*WireClaims, error) {
Atespace: claims.Substrate.Atespace,
ActorName: claims.Substrate.ActorName,
ActorUID: claims.Substrate.ActorUID,
+ Node: claims.Substrate.Node,
}
switch claims.Kind {
case KindActor:
+ if substrate.Node != "" {
+ return nil, errors.New("an actor's own token carries no node")
+ }
wire.Actor = substrate
case KindWorkerForActor:
wire.WorkerForActor = substrate
diff --git a/internal/actoridjwt/actoridjwt_test.go b/internal/actoridjwt/actoridjwt_test.go
index 8cd928b844..6a5a7f9ad8 100644
--- a/internal/actoridjwt/actoridjwt_test.go
+++ b/internal/actoridjwt/actoridjwt_test.go
@@ -68,6 +68,26 @@ func TestClaimsToWire_ClaimNamePerKind(t *testing.T) {
}
}
+func TestClaimsToWire_NodeOnWorkerTokensOnly(t *testing.T) {
+ claims := Claims{
+ Kind: KindWorkerForActor,
+ Subject: "x",
+ Audiences: []string{"a"},
+ Substrate: SubstrateClaims{Atespace: "team-a", ActorName: "my-actor", ActorUID: "u", Node: "node-1"},
+ }
+ wire, err := ClaimsToWire(&claims)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if got := wire.WorkerForActor.Node; got != "node-1" {
+ t.Errorf("worker-for-actor node = %q, want node-1", got)
+ }
+ claims.Kind = KindActor
+ if _, err := ClaimsToWire(&claims); err == nil {
+ t.Error("ClaimsToWire accepted a node on an actor's own token")
+ }
+}
+
func TestClaimsToWire_RejectsUnspecifiedKind(t *testing.T) {
for _, kind := range []Kind{KindUnspecified, Kind(99)} {
if _, err := ClaimsToWire(&Claims{Kind: kind, Subject: "x", Audiences: []string{"a"}}); err == nil {
diff --git a/internal/actorlock/actorlock.go b/internal/actorlock/actorlock.go
index b822f6559e..269159c17b 100644
--- a/internal/actorlock/actorlock.go
+++ b/internal/actorlock/actorlock.go
@@ -40,6 +40,9 @@ func (m *CancelableMutex) Unlock() { m.ch <- struct{}{} }
// LockContext returns false on cancellation without acquiring the mutex.
func (m *CancelableMutex) LockContext(ctx context.Context) bool {
+ if ctx.Err() != nil {
+ return false
+ }
select {
case <-m.ch:
return true
@@ -48,7 +51,7 @@ func (m *CancelableMutex) LockContext(ctx context.Context) bool {
}
}
-// Locks serializes lifecycle RPCs per actor.
+// Locks serializes lifecycle RPCs per actor. A zero value is ready to use.
type Locks struct {
mu sync.Mutex
// References include holders and waiters; unused entries are removed.
@@ -67,7 +70,13 @@ func New() *Locks {
// Lock returns false on cancellation without acquiring the actor's lock.
func (l *Locks) Lock(ctx context.Context, actorUID string) bool {
+ if ctx.Err() != nil {
+ return false
+ }
l.mu.Lock()
+ if l.held == nil {
+ l.held = map[string]*heldLock{}
+ }
entry, ok := l.held[actorUID]
if !ok {
entry = &heldLock{mutex: NewCancelableMutex()}
diff --git a/internal/actorlock/actorlock_test.go b/internal/actorlock/actorlock_test.go
index 726d13b8f4..590b752c27 100644
--- a/internal/actorlock/actorlock_test.go
+++ b/internal/actorlock/actorlock_test.go
@@ -161,3 +161,22 @@ func TestActorLocksForgetIdleActors(t *testing.T) {
t.Errorf("locks retained for %d idle actors, want none", len(locks.held))
}
}
+
+// A zero-value Locks works without New, and an already-canceled context never
+// takes a free lock.
+func TestActorLocksZeroValueAndCanceledContext(t *testing.T) {
+ var locks Locks
+ canceled, cancel := context.WithCancel(context.Background())
+ cancel()
+ if locks.Lock(canceled, "actor-a") {
+ locks.Unlock("actor-a")
+ t.Fatal("Lock with an already-canceled context succeeded")
+ }
+ if locks.Busy("actor-a") {
+ t.Fatal("Busy(actor-a) = true after canceled Lock")
+ }
+ if !locks.Lock(context.Background(), "actor-a") {
+ t.Fatal("Lock on zero-value Locks failed")
+ }
+ locks.Unlock("actor-a")
+}
diff --git a/internal/ateattr/ateattr.go b/internal/ateattr/ateattr.go
index 5f37b59d3a..5c5097cf76 100644
--- a/internal/ateattr/ateattr.go
+++ b/internal/ateattr/ateattr.go
@@ -161,6 +161,8 @@ const (
SnapshotFidelityKey = attribute.Key("ate.snapshot.fidelity")
SnapshotPhaseKey = attribute.Key("ate.snapshot.phase")
ImageCacheOutcomeKey = attribute.Key("ate.imagecache.outcome")
+ FileCacheNameKey = attribute.Key("ate.filecache.name")
+ FileCacheGCOutcomeKey = attribute.Key("ate.filecache.gc.outcome")
SchedulerOutcomeKey = attribute.Key("ate.scheduler.outcome")
RouterResumeKey = attribute.Key("ate.router.resume")
RouterOutcomeKey = attribute.Key("ate.router.outcome")
@@ -237,6 +239,21 @@ const (
ImageCacheOutcomeTimeout = "timeout"
)
+// Values for FileCacheNameKey.
+const (
+ // FileCacheSandboxAsset is the sandbox asset cache: the runsc binaries,
+ // kernels and guest images the workers run actors with.
+ FileCacheSandboxAsset = "sandbox_asset"
+)
+
+// Values for FileCacheGCOutcomeKey. A pass is quiet when the cache is within
+// its budget and nothing is asked of it; those are not recorded.
+const (
+ FileCacheGCOutcomeComplete = "complete" // the pass freed what the budget asked
+ FileCacheGCOutcomeShortfall = "shortfall" // it freed what it could, less than asked
+ FileCacheGCOutcomeSkipped = "skipped" // it could not size the cache and did nothing
+)
+
// Values for RouterOutcomeKey. RouterOutcomeNoCapacity means that no worker
// that meets the constraints of the actor had room. scheduling.ErrNoCapacity
// reports the same condition. It is a capacity signal, not a defect. A request
diff --git a/internal/ateinterceptors/ateinterceptors.go b/internal/ateinterceptors/ateinterceptors.go
index 16717f59d5..238d2d9488 100644
--- a/internal/ateinterceptors/ateinterceptors.go
+++ b/internal/ateinterceptors/ateinterceptors.go
@@ -16,8 +16,10 @@ package ateinterceptors
import (
"context"
+ "fmt"
"log/slog"
"strconv"
+ "strings"
"time"
"go.ate.dev/substrate/internal/apierror"
@@ -26,6 +28,7 @@ import (
"google.golang.org/grpc/codes"
"google.golang.org/grpc/metadata"
"google.golang.org/grpc/status"
+ "google.golang.org/protobuf/proto"
)
// ServerElapsedTrailer carries the server's handler duration in microseconds,
@@ -58,7 +61,7 @@ func ServerUnaryInterceptor(ctx context.Context, req any, info *grpc.UnaryServer
slog.InfoContext(ctx, "Handle RPC",
slog.String("method", info.FullMethod),
slog.Any("req", req),
- slog.Any("resp", resp),
+ slog.Any("resp", respForLog(info.FullMethod, resp, err)),
slog.Any("err", err),
slog.String("elapsed-time", elapsed.String()),
slog.Any("principal", pInfo),
@@ -94,7 +97,7 @@ func InternalServerUnaryInterceptor(ctx context.Context, req any, info *grpc.Una
slog.InfoContext(ctx, "Handle RPC",
slog.String("method", info.FullMethod),
slog.Any("req", req),
- slog.Any("resp", resp),
+ slog.Any("resp", respForLog(info.FullMethod, resp, err)),
slog.Any("err", err),
slog.String("elapsed-time", time.Since(startTime).String()),
)
@@ -106,3 +109,22 @@ func InternalServerUnaryInterceptor(ctx context.Context, req any, info *grpc.Una
return resp, err
}
+
+// respForLog keeps list responses out of the log. A page of a thousand actors
+// is tens of kilobytes per call, and at scale those pages rotate the lifecycle
+// lines out of the container log. A failed call's response is logged as the
+// handler returned it: the generated handlers return a typed nil, which logs
+// as null.
+func respForLog(method string, resp any, err error) any {
+ if err != nil {
+ return resp
+ }
+ name := method[strings.LastIndex(method, "/")+1:]
+ if !strings.HasPrefix(name, "List") {
+ return resp
+ }
+ if m, ok := resp.(proto.Message); ok {
+ return fmt.Sprintf("<%s response elided, %d bytes>", name, proto.Size(m))
+ }
+ return resp
+}
diff --git a/internal/ateinterceptors/ateinterceptors_test.go b/internal/ateinterceptors/ateinterceptors_test.go
index 34ebc9847f..b3c2fa729a 100644
--- a/internal/ateinterceptors/ateinterceptors_test.go
+++ b/internal/ateinterceptors/ateinterceptors_test.go
@@ -444,14 +444,46 @@ func TestInternalServerUnaryInterceptorLogMasksBytesFields(t *testing.T) {
}
func TestServerUnaryInterceptorLogsNilResponseOnHandlerError(t *testing.T) {
+ tests := []struct {
+ name string
+ method string
+ resp any
+ }{
+ {name: "nil", method: "/ateapi.Control/MintActorJwt", resp: nil},
+ // The generated handlers return a typed nil on error; a List call must
+ // not log it as an elided response.
+ {name: "typed nil List response", method: "/ateapi.Control/ListActors", resp: (*ateapipb.ListActorsResponse)(nil)},
+ }
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ log := captureDefaultLog(t)
+ _, err := ServerUnaryInterceptor(context.Background(), "request", &grpc.UnaryServerInfo{FullMethod: tt.method}, func(ctx context.Context, req interface{}) (interface{}, error) {
+ return tt.resp, apierror.PermissionDenied("no")
+ })
+ if status.Code(err) != codes.PermissionDenied {
+ t.Fatalf("err = %v", err)
+ }
+ if !strings.Contains(log.String(), `"resp":null`) {
+ t.Errorf("nil response should log as null: %s", log.String())
+ }
+ })
+ }
+}
+
+func TestServerUnaryInterceptorElidesListResponses(t *testing.T) {
log := captureDefaultLog(t)
- _, err := ServerUnaryInterceptor(context.Background(), &ateapipb.MintActorJwtRequest{}, &grpc.UnaryServerInfo{FullMethod: "/ateapi.Control/MintActorJwt"}, func(ctx context.Context, req interface{}) (interface{}, error) {
- return nil, apierror.PermissionDenied("no")
+
+ resp := &ateapipb.ListActorsResponse{Actors: []*ateapipb.Actor{{Metadata: &ateapipb.ResourceMetadata{Name: "gym-0123456789ab"}}}}
+ got, err := ServerUnaryInterceptor(context.Background(), &ateapipb.ListActorsRequest{Atespace: "nemo-gym"}, &grpc.UnaryServerInfo{FullMethod: "/ateapi.Control/ListActors"}, func(ctx context.Context, req interface{}) (interface{}, error) {
+ return resp, nil
})
- if status.Code(err) != codes.PermissionDenied {
- t.Fatalf("err = %v", err)
+ if err != nil {
+ t.Fatalf("ServerUnaryInterceptor failed: %v", err)
}
- if !strings.Contains(log.String(), `"resp":null`) {
- t.Errorf("nil response should log as null: %s", log.String())
+ if gotLog := log.String(); strings.Contains(gotLog, "gym-0123456789ab") || !strings.Contains(gotLog, "ListActors response elided") {
+ t.Fatalf("list response not elided from the log: %s", gotLog)
+ }
+ if len(got.(*ateapipb.ListActorsResponse).GetActors()) != 1 {
+ t.Fatalf("interceptor mutated the response returned to the client")
}
}
diff --git a/internal/ateinterceptors/rejectunknown_test.go b/internal/ateinterceptors/rejectunknown_test.go
index 87d1dcdb7e..1290faa689 100644
--- a/internal/ateinterceptors/rejectunknown_test.go
+++ b/internal/ateinterceptors/rejectunknown_test.go
@@ -22,10 +22,10 @@ import (
"google.golang.org/grpc/codes"
"google.golang.org/protobuf/encoding/protowire"
"google.golang.org/protobuf/proto"
+ "google.golang.org/protobuf/types/known/structpb"
"k8s.io/apimachinery/pkg/util/validation/field"
"go.ate.dev/substrate/internal/apierror"
- "go.ate.dev/substrate/internal/proto/ateletpb"
"go.ate.dev/substrate/pkg/proto/ateapipb"
)
@@ -127,13 +127,13 @@ func TestFindUnknownFields(t *testing.T) {
{
name: "unknown field inside a map value message",
in: func() proto.Message {
- return &ateletpb.SandboxAssets{
- Assets: map[string]*ateletpb.ArchAssets{
- "amd64": withUnknown(&ateletpb.ArchAssets{}, 9999),
+ return &structpb.Struct{
+ Fields: map[string]*structpb.Value{
+ "amd64": withUnknown(&structpb.Value{}, 9999),
},
}
},
- want: field.ErrorList{field.Invalid(field.NewPath("assets").Key("amd64"), field.OmitValueType{}, "")},
+ want: field.ErrorList{field.Invalid(field.NewPath("fields").Key("amd64"), field.OmitValueType{}, "")},
},
{
name: "scalar map values cannot carry unknown fields",
diff --git a/internal/ateworker/register/crash.go b/internal/ateworker/register/crash.go
new file mode 100644
index 0000000000..b2f38dfc05
--- /dev/null
+++ b/internal/ateworker/register/crash.go
@@ -0,0 +1,118 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package register
+
+import (
+ "context"
+ "crypto/tls"
+ "fmt"
+ "log/slog"
+ "strings"
+ "time"
+
+ "google.golang.org/grpc"
+ "google.golang.org/grpc/codes"
+ "google.golang.org/grpc/status"
+
+ "go.ate.dev/substrate/internal/ateletdial"
+ "go.ate.dev/substrate/internal/proto/ateletpb"
+ "go.ate.dev/substrate/internal/resources"
+)
+
+// maxCrashMessageBytes matches the control plane's limit on a crash message.
+const maxCrashMessageBytes = 4096
+
+// CrashReporter tells the control plane, through the node-local atelet, that
+// the sandbox of an actor this ateworker hosts died on its own.
+type CrashReporter struct {
+ socketPath string
+ tlsConfig *tls.Config
+}
+
+// NewCrashReporter loads the ateworker's credentials once, at startup. Only the
+// connection fields of cfg are used.
+func NewCrashReporter(cfg ReportConfig) (*CrashReporter, error) {
+ tlsConfig, err := ateletdial.TLSConfig(cfg.CredentialBundlePath, cfg.TrustBundlePath, cfg.AteletSPIFFEID)
+ if err != nil {
+ return nil, fmt.Errorf("crash reporter: %w", err)
+ }
+ return &CrashReporter{socketPath: cfg.SocketPath, tlsConfig: tlsConfig}, nil
+}
+
+// ReportCrashed reports that actor's sandbox died, retrying until the control
+// plane answers or ctx ends. A crash stays true, so unlike a suspend request it
+// is worth retrying for as long as the caller allows. A refusal, the actor
+// having moved on or an operation in flight owning it, is an answer and is
+// logged rather than returned.
+func (r *CrashReporter) ReportCrashed(ctx context.Context, actor resources.ActorAttribution, message string) error {
+ conn, err := ateletdial.Dial(r.socketPath, r.tlsConfig)
+ if err != nil {
+ return fmt.Errorf("dial atelet: %w", err)
+ }
+ defer conn.Close()
+ return reportCrashedWithRetry(ctx, conn, actor, message, initialReportBackoff)
+}
+
+func reportCrashedWithRetry(ctx context.Context, conn grpc.ClientConnInterface, actor resources.ActorAttribution, message string, backoff time.Duration) error {
+ req := &ateletpb.ReportActorCrashedRequest{
+ ActorAtespace: actor.Ref.Atespace,
+ ActorName: actor.Ref.Name,
+ ActorUid: actor.UID,
+ Message: truncateMessage(message),
+ }
+ for {
+ callCtx, cancel := context.WithTimeout(ctx, reportTimeout)
+ _, err := ateletpb.NewWorkerSupportClient(conn).ReportActorCrashed(callCtx, req)
+ cancel()
+ switch status.Code(err) {
+ case codes.OK:
+ return nil
+ case codes.NotFound, codes.FailedPrecondition:
+ slog.InfoContext(ctx, "Control plane declined the actor crash report",
+ slog.String("actor", actor.Ref.String()),
+ slog.String("actor_uid", actor.UID),
+ slog.Any("err", err))
+ return nil
+ case codes.Unavailable, codes.Aborted, codes.ResourceExhausted, codes.DeadlineExceeded:
+ default:
+ return fmt.Errorf("reporting actor crash: %w", err)
+ }
+ slog.WarnContext(ctx, "Retrying actor crash report", slog.Duration("in", backoff), slog.Any("err", err))
+ select {
+ case <-ctx.Done():
+ return fmt.Errorf("reporting actor crash: %w (last error: %w)", ctx.Err(), err)
+ case <-time.After(backoff):
+ }
+ backoff = min(backoff*2, maxReportBackoff)
+ }
+}
+
+// truncateMessage keeps message within the control plane's limit, on a rune
+// boundary.
+func truncateMessage(message string) string {
+ message = strings.ToValidUTF8(message, "�")
+ if len(message) <= maxCrashMessageBytes {
+ return message
+ }
+ cut := maxCrashMessageBytes
+ for cut > 0 && !utf8RuneStart(message[cut]) {
+ cut--
+ }
+ return message[:cut]
+}
+
+func utf8RuneStart(b byte) bool {
+ return b&0xC0 != 0x80
+}
diff --git a/internal/ateworker/register/crash_test.go b/internal/ateworker/register/crash_test.go
new file mode 100644
index 0000000000..c6beb01ebe
--- /dev/null
+++ b/internal/ateworker/register/crash_test.go
@@ -0,0 +1,104 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package register
+
+import (
+ "context"
+ "strings"
+ "testing"
+ "time"
+
+ "google.golang.org/grpc"
+ "google.golang.org/grpc/codes"
+ "google.golang.org/grpc/status"
+
+ "go.ate.dev/substrate/internal/proto/ateletpb"
+ "go.ate.dev/substrate/internal/resources"
+)
+
+// fakeConn answers ReportActorCrashed with the queued errors, then OK.
+type fakeConn struct {
+ grpc.ClientConnInterface
+ errs []error
+ got []*ateletpb.ReportActorCrashedRequest
+}
+
+func (c *fakeConn) Invoke(_ context.Context, _ string, args, _ any, _ ...grpc.CallOption) error {
+ c.got = append(c.got, args.(*ateletpb.ReportActorCrashedRequest))
+ if len(c.errs) == 0 {
+ return nil
+ }
+ err := c.errs[0]
+ c.errs = c.errs[1:]
+ return err
+}
+
+var testActor = resources.ActorAttribution{
+ Ref: resources.ActorRef{Atespace: "team-a", Name: "actor-1"},
+ UID: "01234567-89ab-cdef-0123-456789abcdef",
+}
+
+func TestReportCrashed(t *testing.T) {
+ tests := []struct {
+ name string
+ errs []error
+ wantErr bool
+ wantCalls int
+ }{
+ {name: "accepted", wantCalls: 1},
+ {name: "retries until delivered", errs: []error{status.Error(codes.Unavailable, ""), status.Error(codes.Aborted, "")}, wantCalls: 3},
+ {name: "refusal is an answer", errs: []error{status.Error(codes.FailedPrecondition, "")}, wantCalls: 1},
+ {name: "bad request is not retried", errs: []error{status.Error(codes.InvalidArgument, "")}, wantErr: true, wantCalls: 1},
+ }
+ for _, tc := range tests {
+ t.Run(tc.name, func(t *testing.T) {
+ conn := &fakeConn{errs: tc.errs}
+ err := reportCrashedWithRetry(context.Background(), conn, testActor, "cloud-hypervisor exited", time.Millisecond)
+ if (err != nil) != tc.wantErr {
+ t.Fatalf("err = %v, wantErr %v", err, tc.wantErr)
+ }
+ if len(conn.got) != tc.wantCalls {
+ t.Fatalf("calls = %d, want %d", len(conn.got), tc.wantCalls)
+ }
+ req := conn.got[0]
+ if req.GetActorAtespace() != "team-a" || req.GetActorName() != "actor-1" || req.GetActorUid() != testActor.UID || req.GetMessage() != "cloud-hypervisor exited" {
+ t.Errorf("request = %v", req)
+ }
+ })
+ }
+}
+
+func TestReportCrashedGivesUpWithContext(t *testing.T) {
+ ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)
+ defer cancel()
+ conn := &fakeConn{errs: make([]error, 1000)}
+ for i := range conn.errs {
+ conn.errs[i] = status.Error(codes.Unavailable, "")
+ }
+ if err := reportCrashedWithRetry(ctx, conn, testActor, "x", time.Millisecond); err == nil {
+ t.Fatal("err = nil, want the context error")
+ }
+}
+
+func TestTruncateMessage(t *testing.T) {
+ long := strings.Repeat("é", maxCrashMessageBytes)
+ got := truncateMessage(long)
+ if len(got) > maxCrashMessageBytes {
+ t.Errorf("len = %d, want <= %d", len(got), maxCrashMessageBytes)
+ }
+ if !strings.HasPrefix(long, got) || strings.ContainsRune(got, '�') {
+ t.Errorf("truncation split a rune")
+ }
+}
diff --git a/internal/benchmarking/boomer/agentsession/agentsession.go b/internal/benchmarking/boomer/agentsession/agentsession.go
index 6abe8c64b5..76ee8b5fe4 100644
--- a/internal/benchmarking/boomer/agentsession/agentsession.go
+++ b/internal/benchmarking/boomer/agentsession/agentsession.go
@@ -906,7 +906,7 @@ func (u *sessionUser) suspendAndDelete(ctx context.Context) {
_, _ = u.cfg.APIStub.SuspendActor(suspendCtx, &ateapipb.SuspendActorRequest{Actor: u.ref()})
cancel()
_ = u.tracedCall(ctx, "DeleteActor", func(callCtx context.Context, tr *metadata.MD) error {
- _, err := u.cfg.APIStub.DeleteActor(callCtx, &ateapipb.DeleteActorRequest{Actor: u.ref(), AnyState: true}, grpc.Trailer(tr))
+ _, err := u.cfg.APIStub.DeleteActor(callCtx, &ateapipb.DeleteActorRequest{Actor: u.ref()}, grpc.Trailer(tr))
return err
})
bmetrics.UpdateUsers(agentSessionUserClass, -1)
diff --git a/internal/benchmarking/boomer/ateenv/knobs_test.go b/internal/benchmarking/boomer/ateenv/knobs_test.go
new file mode 100644
index 0000000000..c5dced4cc1
--- /dev/null
+++ b/internal/benchmarking/boomer/ateenv/knobs_test.go
@@ -0,0 +1,74 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package ateenv
+
+import (
+ "reflect"
+ "testing"
+ "time"
+
+ "go.ate.dev/substrate/internal/benchmarking/boomer/dynconfig"
+)
+
+func TestAteEnvCodec(t *testing.T) {
+ h := dynconfig.NewHolder(ateEnvCodec)
+ initial := ateEnvKnobs{
+ WaitTime: dynconfig.WaitTime{MaxWait: dynconfig.Seconds(500 * time.Millisecond)},
+ MaxPingsPerWake: 1,
+ }
+ if got := h.Load(); !reflect.DeepEqual(got, initial) {
+ t.Fatalf("fresh holder = %+v, want %+v", got, initial)
+ }
+
+ valid := `{
+ "min_wait_time": 0.2, "max_wait_time": 1.0,
+ "min_live_time": 9, "max_live_time": 14,
+ "resume_mode": "explicit", "lifecycle_mode": "pause",
+ "max_pings_per_wake": 2
+ }`
+ if _, _, err := h.Apply([]byte(valid)); err != nil {
+ t.Fatalf("Apply(valid): %v", err)
+ }
+ want := ateEnvKnobs{
+ WaitTime: dynconfig.WaitTime{MinWait: dynconfig.Seconds(200 * time.Millisecond), MaxWait: dynconfig.Seconds(time.Second)},
+ Lifecycle: dynconfig.Lifecycle{ResumeMode: dynconfig.ResumeModeExplicit, LifecycleMode: dynconfig.LifecycleModePause},
+ MinLive: dynconfig.Seconds(9 * time.Second),
+ MaxLive: dynconfig.Seconds(14 * time.Second),
+ MaxPingsPerWake: 2,
+ }
+ if got := h.Load(); !reflect.DeepEqual(got, want) {
+ t.Errorf("decoded =\n %+v, want\n %+v", got, want)
+ }
+
+ for name, payload := range map[string]string{
+ "negative min wait": `{"min_wait_time": -1}`,
+ "negative max wait": `{"max_wait_time": -1}`,
+ "max wait below min wait": `{"min_wait_time": 2, "max_wait_time": 1}`,
+ "negative min live": `{"min_live_time": -1}`,
+ "negative max live": `{"max_live_time": -1}`,
+ "max live below min live": `{"min_live_time": 14, "max_live_time": 9}`,
+ "invalid resume mode": `{"resume_mode": "invalid_mode"}`,
+ "invalid lifecycle mode": `{"lifecycle_mode": "invalid_lifecycle"}`,
+ } {
+ t.Run(name, func(t *testing.T) {
+ if _, _, err := h.Apply([]byte(payload)); err == nil {
+ t.Errorf("Apply(%s) took the payload, want a refusal", payload)
+ }
+ if got := h.Load(); !reflect.DeepEqual(got, want) {
+ t.Errorf("a refused payload changed the config to %+v", got)
+ }
+ })
+ }
+}
diff --git a/internal/benchmarking/boomer/ateenv/lifecycle.go b/internal/benchmarking/boomer/ateenv/lifecycle.go
new file mode 100644
index 0000000000..86741bf58c
--- /dev/null
+++ b/internal/benchmarking/boomer/ateenv/lifecycle.go
@@ -0,0 +1,831 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package ateenv implements the boomer-Go user class for the ate-env benchmark
+// workload (see benchmarking/locust/tests/ate_env.py for the locust harness stub).
+package ateenv
+
+import (
+ "bytes"
+ "context"
+ "errors"
+ "fmt"
+ "io"
+ "log/slog"
+ "math/rand/v2"
+ "net"
+ "net/url"
+ "slices"
+ "strings"
+ "sync"
+ "time"
+
+ "github.com/google/uuid"
+ "go.ate.dev/substrate/internal/ateinterceptors"
+ "go.ate.dev/substrate/internal/atenet"
+ "go.ate.dev/substrate/internal/benchmarking/boomer/boomerutil"
+ "go.ate.dev/substrate/internal/benchmarking/boomer/dynconfig"
+ bmetrics "go.ate.dev/substrate/internal/benchmarking/boomer/metrics"
+ "go.ate.dev/substrate/internal/benchmarking/boomer/userclass"
+ ateenvv1alpha "go.ate.dev/substrate/internal/proto/ateenv/v1alpha"
+ "go.ate.dev/substrate/pkg/proto/ateapipb"
+ "go.opentelemetry.io/otel"
+ "go.opentelemetry.io/otel/attribute"
+ "google.golang.org/grpc"
+ "google.golang.org/grpc/codes"
+ "google.golang.org/grpc/credentials/insecure"
+ "google.golang.org/grpc/metadata"
+ "google.golang.org/grpc/status"
+)
+
+const (
+ userClass = "AteEnvUser"
+ templateName = "ate-env"
+ templateAtespace = "benchmark-workloads"
+ expectedStdout = "hello world\n"
+ execTimeout = 30 * time.Second
+
+ maxConsecutiveFailures = 3
+ minPingGap = 200 * time.Millisecond
+ maxPingGap = 1 * time.Second
+)
+
+type ateEnvKnobs struct {
+ dynconfig.WaitTime
+ dynconfig.Lifecycle
+ MinLive dynconfig.Seconds `json:"min_live_time"`
+ MaxLive dynconfig.Seconds `json:"max_live_time"`
+ MaxPingsPerWake int `json:"max_pings_per_wake"`
+}
+
+var ateEnvCodec = dynconfig.Typed[ateEnvKnobs]{
+ Defaults: ateEnvKnobs{
+ WaitTime: dynconfig.WaitTime{MaxWait: dynconfig.Seconds(500 * time.Millisecond)},
+ MaxPingsPerWake: 1,
+ },
+ Validate: func(k ateEnvKnobs) error {
+ if err := k.WaitTime.Validate(); err != nil {
+ return err
+ }
+ if err := k.Lifecycle.Validate(); err != nil {
+ return err
+ }
+ if k.MinLive < 0 {
+ return fmt.Errorf("min_live_time cannot be negative: %v", k.MinLive.Duration())
+ }
+ if k.MaxLive < 0 {
+ return fmt.Errorf("max_live_time cannot be negative: %v", k.MaxLive.Duration())
+ }
+ if k.MaxLive < k.MinLive {
+ return fmt.Errorf("max_live_time (%v) cannot be less than min_live_time (%v)", k.MaxLive.Duration(), k.MinLive.Duration())
+ }
+ return nil
+ },
+}
+
+func init() {
+ userclass.Add(userclass.Entry{
+ Name: "ate_env",
+ LocustFile: "ate_env.py",
+ UserClass: userClass,
+ Config: ateEnvCodec,
+ Init: initAteEnv,
+ })
+}
+
+// routerGRPCTarget converts an HTTP/HTTPS router URL into a dns:/// gRPC dial target.
+func routerGRPCTarget(rawURL string) (string, error) {
+ if rawURL == "" {
+ rawURL = "http://atenet-router.ate-system.svc.cluster.local"
+ }
+ if !strings.Contains(rawURL, "://") {
+ rawURL = "http://" + rawURL
+ }
+ u, err := url.Parse(rawURL)
+ if err != nil {
+ return "", err
+ }
+ if strings.EqualFold(u.Scheme, "https") {
+ return "", fmt.Errorf("router URL %q uses https, which is not supported by the plaintext gRPC dialer", rawURL)
+ }
+ host := u.Hostname()
+ if host == "" {
+ return "", fmt.Errorf("router URL %q has empty host", rawURL)
+ }
+ port := u.Port()
+ if port == "" {
+ port = "80"
+ }
+ return "dns:///" + net.JoinHostPort(host, port), nil
+}
+
+// initAteEnv creates a runtime tied to cfg and returns a boomer-compatible task
+// function plus a Shutdown hook the caller should run before exit (it
+// suspend+deletes every actor this worker created and closes router connections).
+func initAteEnv(cfg *userclass.Config) (taskFn func(), shutdown func(context.Context)) {
+ if cfg.Tracer == nil {
+ cfg.Tracer = otel.Tracer("substrate-boomer/ateenv")
+ }
+ rt := &taskRuntime{
+ cfg: cfg,
+ stopCh: make(chan struct{}),
+ }
+ return rt.iterate, rt.shutdown
+}
+
+type taskRuntime struct {
+ cfg *userclass.Config
+ users sync.Map // goroutineID → *ateEnvUser
+ dialProcessClient func() (ateenvv1alpha.ProcessServiceClient, io.Closer, error)
+ sleepFn func(time.Duration) bool
+ stopMu sync.RWMutex
+ stopCh chan struct{}
+ stopping bool
+ iterWG sync.WaitGroup
+}
+
+func (r *taskRuntime) stopChan() <-chan struct{} {
+ r.stopMu.RLock()
+ ch := r.stopCh
+ r.stopMu.RUnlock()
+ if ch != nil {
+ return ch
+ }
+ r.stopMu.Lock()
+ defer r.stopMu.Unlock()
+ if r.stopCh == nil {
+ r.stopCh = make(chan struct{})
+ if r.stopping {
+ close(r.stopCh)
+ }
+ }
+ return r.stopCh
+}
+
+func (r *taskRuntime) isStopping() bool {
+ r.stopMu.RLock()
+ defer r.stopMu.RUnlock()
+ return r.stopping
+}
+
+func (r *taskRuntime) tryStartIter() bool {
+ r.stopMu.RLock()
+ defer r.stopMu.RUnlock()
+ if r.stopping {
+ return false
+ }
+ r.iterWG.Add(1)
+ return true
+}
+
+func (r *taskRuntime) markStopping() {
+ r.stopMu.Lock()
+ if !r.stopping {
+ r.stopping = true
+ if r.stopCh != nil {
+ close(r.stopCh)
+ }
+ }
+ r.stopMu.Unlock()
+}
+
+func (r *taskRuntime) sleep(d time.Duration) bool {
+ if r.sleepFn != nil {
+ return r.sleepFn(d)
+ }
+ if d <= 0 {
+ return !r.isStopping()
+ }
+ t := time.NewTimer(d)
+ defer t.Stop()
+ select {
+ case <-t.C:
+ return true
+ case <-r.stopChan():
+ return false
+ }
+}
+
+// iterate is the task function boomer calls in a loop on each VU goroutine.
+func (r *taskRuntime) iterate() {
+ if !r.tryStartIter() {
+ return
+ }
+ defer func() {
+ r.iterWG.Done()
+ r.sleep(r.dynamicWait())
+ }()
+
+ gid := boomerutil.GoroutineID()
+ val, loaded := r.users.Load(gid)
+ if !loaded {
+ u, err := r.startUser(context.Background())
+ if err != nil {
+ slog.Warn("ate_env on_start failed; goroutine will retry next iter",
+ slog.String("err", err.Error()))
+ return
+ }
+ val, _ = r.users.LoadOrStore(gid, u)
+ }
+ user := val.(*ateEnvUser)
+
+ actor := user.nextActor()
+ if actor == nil {
+ return
+ }
+
+ ctx := context.Background()
+
+ // resume()'s failure path replaces a crashed actor on the spot, so
+ // reaching here means the replacement's create failed last time. Retry
+ // it, or the VU runs a slot short for the rest of the run.
+ if actor.isCrashed() {
+ user.replaceActor(ctx, actor)
+ return
+ }
+
+ // Going straight back to resume on a stranded actor would turn every
+ // retryable hibernate failure into a FailedPrecondition and cost the
+ // actor anyway. SuspendActor is re-entrant in either mode, so finish
+ // the hibernate and pick the cycle up next iteration.
+ if actor.isHibernatePending() {
+ if err := actor.hibernate(ctx); err != nil {
+ if actor.noteFailure(err) {
+ user.replaceActor(ctx, actor)
+ }
+ } else {
+ actor.noteSuccess()
+ }
+ return
+ }
+
+ if !actor.isRunning() || dynconfig.Get[ateEnvKnobs](r.cfg.Dyn).LifecycleMode != dynconfig.LifecycleModeNone {
+ if err := actor.resume(ctx); err != nil {
+ if actor.noteFailure(err) {
+ user.replaceActor(ctx, actor)
+ }
+ return
+ }
+ }
+
+ // Live window (--min/--max-live-time): exec HelloWorld, then hold the actor
+ // running until the window closes. At most one exec failure per wake is
+ // counted toward replacement (after hibernate completes), so a multi-ping
+ // wake cannot trigger replacement from a single transient blip.
+ deadline := time.Now().Add(r.liveWait())
+ maxPings := max(dynconfig.Get[ateEnvKnobs](r.cfg.Dyn).MaxPingsPerWake, 1)
+ var execErr error
+ for sent := 0; sent < maxPings; sent++ {
+ if sent > 0 {
+ gap := minPingGap + time.Duration(rand.Float64()*float64(maxPingGap-minPingGap))
+ if time.Now().Add(gap).After(deadline) {
+ break
+ }
+ if !r.sleep(gap) {
+ break
+ }
+ }
+ if err := actor.execHelloWorld(ctx); err != nil {
+ execErr = err
+ } else {
+ execErr = nil
+ actor.noteSuccess()
+ }
+ }
+ if remaining := time.Until(deadline); remaining > 0 {
+ r.sleep(remaining)
+ }
+ if err := actor.hibernate(ctx); err != nil {
+ if actor.noteFailure(err) {
+ user.replaceActor(ctx, actor)
+ }
+ return
+ }
+ if execErr != nil {
+ if actor.noteExecFailure(execErr) {
+ user.replaceActor(ctx, actor)
+ }
+ return
+ }
+ actor.noteSuccess()
+}
+
+func (r *taskRuntime) startUser(ctx context.Context) (*ateEnvUser, error) {
+ n := r.cfg.ActorsPerUser
+ if n < 1 {
+ n = 1
+ }
+
+ dialFn := r.dialProcessClient
+ if dialFn == nil {
+ target, err := routerGRPCTarget(r.cfg.RouterURL)
+ if err != nil {
+ return nil, err
+ }
+ dialFn = func() (ateenvv1alpha.ProcessServiceClient, io.Closer, error) {
+ conn, err := grpc.NewClient(
+ target,
+ grpc.WithTransportCredentials(insecure.NewCredentials()),
+ grpc.WithDisableServiceConfig(),
+ )
+ if err != nil {
+ return nil, nil, err
+ }
+ return ateenvv1alpha.NewProcessServiceClient(conn), conn, nil
+ }
+ }
+
+ client, closer, err := dialFn()
+ if err != nil {
+ return nil, fmt.Errorf("dial router gRPC: %w", err)
+ }
+
+ u := &ateEnvUser{
+ routerConn: closer,
+ actors: make([]*ateEnvActor, 0, n),
+ }
+ bmetrics.UpdateUsers(userClass, 1)
+
+ var lastCreateErr error
+ for i := 0; i < n; i++ {
+ a := &ateEnvActor{
+ cfg: r.cfg,
+ actorName: "sb-" + uuid.NewString(),
+ firstResume: true,
+ processClient: client,
+ }
+ if i == 0 {
+ if err := a.ensureAtespace(ctx); err != nil {
+ if closer != nil {
+ _ = closer.Close()
+ }
+ bmetrics.UpdateUsers(userClass, -1)
+ return nil, err
+ }
+ }
+ if err := a.create(ctx); err != nil {
+ lastCreateErr = err
+ slog.Warn("ate_env create failed partway; using actors created so far",
+ slog.String("atespace", r.cfg.Atespace),
+ slog.Int("wanted", n),
+ slog.Int("got", len(u.actors)),
+ slog.String("err", err.Error()))
+ break
+ }
+ u.actors = append(u.actors, a)
+ }
+ if len(u.actors) == 0 {
+ if closer != nil {
+ _ = closer.Close()
+ }
+ bmetrics.UpdateUsers(userClass, -1)
+ return nil, fmt.Errorf("no actors created: %w", lastCreateErr)
+ }
+ return u, nil
+}
+
+// shutdown stops new iterations, waits for any in-flight iterate() calls to
+// finish (capped at half of ctx's remaining deadline so cleanup still has time
+// to run), hibernates (if still running) and deletes every actor this worker
+// created, then closes each VU's gRPC channel.
+func (r *taskRuntime) shutdown(ctx context.Context) {
+ r.markStopping()
+ waitCtx := ctx
+ if deadline, ok := ctx.Deadline(); ok {
+ var cancel context.CancelFunc
+ waitCtx, cancel = context.WithTimeout(ctx, max(time.Until(deadline)/2, 0))
+ defer cancel()
+ }
+ done := make(chan struct{})
+ go func() {
+ r.iterWG.Wait()
+ close(done)
+ }()
+ select {
+ case <-done:
+ case <-waitCtx.Done():
+ }
+ r.users.Range(func(_, val any) bool {
+ u := val.(*ateEnvUser)
+ for _, a := range u.snapshotActors() {
+ if a.isRunning() {
+ if dynconfig.Get[ateEnvKnobs](r.cfg.Dyn).LifecycleMode == dynconfig.LifecycleModePause {
+ _ = a.pause(ctx)
+ } else {
+ _ = a.suspend(ctx)
+ }
+ }
+ a.delete(ctx)
+ }
+ if u.routerConn != nil {
+ _ = u.routerConn.Close()
+ }
+ bmetrics.UpdateUsers(userClass, -1)
+ return true
+ })
+}
+
+func (r *taskRuntime) dynamicWait() time.Duration {
+ return dynconfig.Get[ateEnvKnobs](r.cfg.Dyn).WaitTime.Draw()
+}
+
+func (r *taskRuntime) liveWait() time.Duration {
+ knobs := dynconfig.Get[ateEnvKnobs](r.cfg.Dyn)
+ return dynconfig.Uniform(knobs.MinLive.Duration(), knobs.MaxLive.Duration())
+}
+
+// ateEnvUser represents one virtual user (goroutine).
+type ateEnvUser struct {
+ mu sync.Mutex
+ routerConn io.Closer
+ actors []*ateEnvActor
+ nextIdx int
+}
+
+func (u *ateEnvUser) nextActor() *ateEnvActor {
+ u.mu.Lock()
+ defer u.mu.Unlock()
+ if len(u.actors) == 0 {
+ return nil
+ }
+ a := u.actors[u.nextIdx]
+ u.nextIdx = (u.nextIdx + 1) % len(u.actors)
+ return a
+}
+
+func (u *ateEnvUser) snapshotActors() []*ateEnvActor {
+ u.mu.Lock()
+ defer u.mu.Unlock()
+ return slices.Clone(u.actors)
+}
+
+func (u *ateEnvUser) replaceActor(ctx context.Context, broken *ateEnvActor) {
+ if broken.markDeleted() {
+ broken.delete(ctx)
+ }
+ replacement := &ateEnvActor{
+ cfg: broken.cfg,
+ actorName: "sb-" + uuid.NewString(),
+ firstResume: true,
+ processClient: broken.processClient,
+ }
+ if err := replacement.create(ctx); err != nil {
+ slog.Warn("ate_env actor replacement create failed; will retry on next failure",
+ slog.String("old_actor", broken.actorName),
+ slog.String("err", err.Error()))
+ return
+ }
+ u.mu.Lock()
+ defer u.mu.Unlock()
+ for i, a := range u.actors {
+ if a == broken {
+ u.actors[i] = replacement
+ return
+ }
+ }
+}
+
+// ateEnvActor tracks the state of a single actor within a VU.
+type ateEnvActor struct {
+ mu sync.Mutex
+ cfg *userclass.Config
+ actorName string
+ firstResume bool
+ actorRunning bool
+ crashed bool
+ hibernatePending bool
+ deleted bool
+ consecutiveFailures int
+ processClient ateenvv1alpha.ProcessServiceClient
+ execTimeout time.Duration
+}
+
+func (u *ateEnvActor) isRunning() bool {
+ u.mu.Lock()
+ defer u.mu.Unlock()
+ return u.actorRunning
+}
+
+func (u *ateEnvActor) isHibernatePending() bool {
+ u.mu.Lock()
+ defer u.mu.Unlock()
+ return u.hibernatePending
+}
+
+func (u *ateEnvActor) isCrashed() bool {
+ u.mu.Lock()
+ defer u.mu.Unlock()
+ return u.crashed
+}
+
+func (u *ateEnvActor) markDeleted() bool {
+ u.mu.Lock()
+ defer u.mu.Unlock()
+ if u.deleted {
+ return false
+ }
+ u.deleted = true
+ return true
+}
+
+func (u *ateEnvActor) noteFailure(err error) bool {
+ switch boomerutil.ClassifyLifecycleFailure(err) {
+ case boomerutil.ReplaceNow:
+ return true
+ case boomerutil.RetryLater:
+ return false
+ }
+ return u.countFailure(err)
+}
+
+func (u *ateEnvActor) noteExecFailure(err error) bool {
+ if s, ok := status.FromError(err); ok {
+ switch s.Code() {
+ case codes.Canceled, codes.ResourceExhausted, codes.InvalidArgument,
+ codes.PermissionDenied, codes.Unauthenticated, codes.Unimplemented:
+ return false
+ }
+ }
+ return u.countFailure(err)
+}
+
+func (u *ateEnvActor) countFailure(err error) bool {
+ u.mu.Lock()
+ u.consecutiveFailures++
+ n := u.consecutiveFailures
+ u.mu.Unlock()
+ if n < maxConsecutiveFailures {
+ return false
+ }
+ slog.Warn("ate_env actor failed repeatedly; replacing it",
+ slog.String("actor", u.actorName),
+ slog.Int("consecutive_failures", n),
+ slog.String("err", err.Error()))
+ return true
+}
+
+func (u *ateEnvActor) noteSuccess() {
+ u.mu.Lock()
+ u.consecutiveFailures = 0
+ u.mu.Unlock()
+}
+
+func (u *ateEnvActor) ref() *ateapipb.ObjectRef {
+ return &ateapipb.ObjectRef{Atespace: u.cfg.Atespace, Name: u.actorName}
+}
+
+func (u *ateEnvActor) ensureAtespace(ctx context.Context) error {
+ return u.tracedCall(ctx, "CreateAtespace", func(callCtx context.Context, tr *metadata.MD) error {
+ _, err := u.cfg.APIStub.CreateAtespace(callCtx, &ateapipb.CreateAtespaceRequest{
+ Atespace: &ateapipb.Atespace{
+ Metadata: &ateapipb.ResourceMetadata{
+ Name: u.cfg.Atespace,
+ },
+ },
+ }, grpc.Trailer(tr))
+ if err == nil {
+ return nil
+ }
+ if s, ok := status.FromError(err); ok && s.Code() == codes.AlreadyExists {
+ return nil
+ }
+ return err
+ })
+}
+
+func (u *ateEnvActor) create(ctx context.Context) error {
+ return u.tracedCall(ctx, "CreateActor", func(callCtx context.Context, tr *metadata.MD) error {
+ _, err := u.cfg.APIStub.CreateActor(callCtx, &ateapipb.CreateActorRequest{
+ Actor: &ateapipb.Actor{
+ Metadata: &ateapipb.ResourceMetadata{Atespace: u.cfg.Atespace, Name: u.actorName},
+ ActorTemplate: &ateapipb.ObjectRef{Atespace: templateAtespace, Name: templateName},
+ },
+ }, grpc.Trailer(tr))
+ return err
+ })
+}
+
+func (u *ateEnvActor) resume(ctx context.Context) error {
+ u.mu.Lock()
+ first := u.firstResume
+ u.mu.Unlock()
+ metricName := "ResumeActor"
+ if first {
+ metricName = "ResumeActorFirstResume"
+ }
+ err := u.tracedCall(ctx, metricName, func(callCtx context.Context, tr *metadata.MD) error {
+ return boomerutil.RetryOnConflict(callCtx, func() error {
+ _, err := u.cfg.APIStub.ResumeActor(callCtx, &ateapipb.ResumeActorRequest{
+ Actor: u.ref(),
+ }, grpc.Trailer(tr))
+ return err
+ })
+ })
+ if err != nil {
+ if boomerutil.IsCrashed(err) {
+ u.mu.Lock()
+ u.crashed = true
+ u.mu.Unlock()
+ bmetrics.RecordFailure("actor", "CrashCount", userClass, 0, "actor entered ACTOR_STATE_CRASHED")
+ slog.Warn("ate_env actor crashed; will stop sending requests",
+ slog.String("actor", u.actorName),
+ slog.String("err", err.Error()))
+ }
+ return err
+ }
+ u.mu.Lock()
+ u.firstResume = false
+ u.actorRunning = true
+ u.mu.Unlock()
+ return nil
+}
+
+func (u *ateEnvActor) hibernate(ctx context.Context) error {
+ switch dynconfig.Get[ateEnvKnobs](u.cfg.Dyn).LifecycleMode {
+ case dynconfig.LifecycleModeNone:
+ return nil
+ case dynconfig.LifecycleModePause:
+ return u.pause(ctx)
+ default:
+ return u.suspend(ctx)
+ }
+}
+
+func (u *ateEnvActor) pause(ctx context.Context) error {
+ err := u.tracedCall(ctx, "SuspendActorAsync", func(callCtx context.Context, tr *metadata.MD) error {
+ _, err := u.cfg.APIStub.SuspendActor(callCtx, &ateapipb.SuspendActorRequest{
+ Actor: u.ref(),
+ Options: &ateapipb.SuspendOptions{Mode: ateapipb.DurabilityMode_DURABILITY_MODE_ASYNCHRONOUS},
+ }, grpc.Trailer(tr))
+ return err
+ })
+ u.mu.Lock()
+ u.actorRunning = false
+ u.hibernatePending = err != nil
+ u.mu.Unlock()
+ return err
+}
+
+func (u *ateEnvActor) suspend(ctx context.Context) error {
+ err := u.tracedCall(ctx, "SuspendActor", func(callCtx context.Context, tr *metadata.MD) error {
+ _, err := u.cfg.APIStub.SuspendActor(callCtx, &ateapipb.SuspendActorRequest{
+ Actor: u.ref(),
+ Options: &ateapipb.SuspendOptions{Mode: ateapipb.DurabilityMode_DURABILITY_MODE_SYNCHRONOUS},
+ }, grpc.Trailer(tr))
+ return err
+ })
+ u.mu.Lock()
+ u.actorRunning = false
+ u.hibernatePending = err != nil
+ u.mu.Unlock()
+ return err
+}
+
+func (u *ateEnvActor) delete(ctx context.Context) {
+ _ = u.tracedCall(ctx, "DeleteActor", func(callCtx context.Context, tr *metadata.MD) error {
+ _, err := u.cfg.APIStub.DeleteActor(callCtx, &ateapipb.DeleteActorRequest{
+ Actor: u.ref(),
+ }, grpc.Trailer(tr))
+ return err
+ })
+}
+
+func (u *ateEnvActor) tracedCall(ctx context.Context, name string, do func(context.Context, *metadata.MD) error) error {
+ ctx, span := u.cfg.Tracer.Start(ctx, name)
+ defer span.End()
+
+ start := time.Now()
+ var tr metadata.MD
+ err := do(ctx, &tr)
+ latency := time.Since(start)
+
+ if serverLatency, source := boomerutil.ElapsedFromMD(tr, ateinterceptors.ServerElapsedTrailer, 0); source == boomerutil.SourceServer {
+ span.SetAttributes(attribute.Float64("server.elapsed_ms", boomerutil.MsFloat(serverLatency)))
+ }
+ boomerutil.LogSampledTrace(span, name, latency, boomerutil.SourceClient, err)
+ if err != nil {
+ bmetrics.RecordFailure("grpc", name, userClass, latency, err.Error())
+ return err
+ }
+ bmetrics.RecordSuccess("grpc", name, userClass, latency, 0)
+ return nil
+}
+
+// metadataCarrier adapts metadata.MD to OpenTelemetry's TextMapCarrier.
+type metadataCarrier metadata.MD
+
+func (m metadataCarrier) Get(key string) string {
+ vals := metadata.MD(m).Get(key)
+ if len(vals) == 0 {
+ return ""
+ }
+ return vals[0]
+}
+
+func (m metadataCarrier) Set(key string, value string) {
+ metadata.MD(m).Set(key, value)
+}
+
+func (m metadataCarrier) Keys() []string {
+ keys := make([]string, 0, len(m))
+ for k := range m {
+ keys = append(keys, k)
+ }
+ return keys
+}
+
+// execHelloWorld executes "echo hello world" inside the actor container via
+// StartProcess and drains StreamProcessOutput until completion.
+func (u *ateEnvActor) execHelloWorld(ctx context.Context) error {
+ timeout := u.execTimeout
+ if timeout <= 0 {
+ timeout = execTimeout
+ }
+ ctx, cancel := context.WithTimeout(ctx, timeout)
+ defer cancel()
+
+ ctx, span := u.cfg.Tracer.Start(ctx, "AteEnvExec")
+ defer span.End()
+
+ start := time.Now()
+
+ md := metadata.MD{}
+ otel.GetTextMapPropagator().Inject(ctx, metadataCarrier(md))
+ md.Set(atenet.TargetActorHeader, u.cfg.Atespace+"/"+u.actorName)
+ callCtx := metadata.NewOutgoingContext(ctx, md)
+
+ fail := func(err error) error {
+ clientLatency := time.Since(start)
+ boomerutil.LogSampledTrace(span, "AteEnvExec", clientLatency, boomerutil.SourceClient, err)
+ bmetrics.RecordFailure("grpc", "AteEnvExec", userClass, clientLatency, err.Error())
+ return err
+ }
+
+ startResp, err := u.processClient.StartProcess(callCtx, &ateenvv1alpha.StartProcessRequest{
+ Command: []string{"echo", "hello world"},
+ })
+ if err != nil {
+ return fail(fmt.Errorf("StartProcess: %w", err))
+ }
+ pid := startResp.GetProcessId()
+ if pid == "" {
+ return fail(errors.New("StartProcess returned empty process_id"))
+ }
+
+ stream, err := u.processClient.StreamProcessOutput(callCtx, &ateenvv1alpha.StreamProcessOutputRequest{
+ ProcessId: pid,
+ Follow: true,
+ })
+ if err != nil {
+ return fail(fmt.Errorf("StreamProcessOutput: %w", err))
+ }
+
+ var (
+ stdoutBuf bytes.Buffer
+ exitProc *ateenvv1alpha.Process
+ )
+ for {
+ msg, err := stream.Recv()
+ if errors.Is(err, io.EOF) {
+ break
+ }
+ if err != nil {
+ return fail(fmt.Errorf("StreamProcessOutput.Recv: %w", err))
+ }
+ if chunk := msg.GetStdout(); len(chunk) > 0 {
+ stdoutBuf.Write(chunk)
+ }
+ if exit := msg.GetExit(); exit != nil {
+ exitProc = exit
+ }
+ }
+
+ if exitProc == nil {
+ return fail(errors.New("stream closed without exit event"))
+ }
+ if state := exitProc.GetState(); state != ateenvv1alpha.ProcessState_PROCESS_STATE_EXITED {
+ return fail(fmt.Errorf("unexpected process state: %v, want PROCESS_STATE_EXITED", state))
+ }
+ if code := exitProc.GetExitCode(); code != 0 {
+ return fail(fmt.Errorf("unexpected exit code: %d, want 0", code))
+ }
+ if got := stdoutBuf.String(); got != expectedStdout {
+ return fail(fmt.Errorf("stdout mismatch: got %q, want %q", got, expectedStdout))
+ }
+
+ clientLatency := time.Since(start)
+ boomerutil.LogSampledTrace(span, "AteEnvExec", clientLatency, boomerutil.SourceClient, nil)
+ bmetrics.RecordSuccess("grpc", "AteEnvExec", userClass, clientLatency, int64(stdoutBuf.Len()))
+ return nil
+}
diff --git a/internal/benchmarking/boomer/ateenv/lifecycle_test.go b/internal/benchmarking/boomer/ateenv/lifecycle_test.go
new file mode 100644
index 0000000000..2a27b4b929
--- /dev/null
+++ b/internal/benchmarking/boomer/ateenv/lifecycle_test.go
@@ -0,0 +1,886 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package ateenv
+
+import (
+ "context"
+ "io"
+ "net"
+ "slices"
+ "strings"
+ "sync"
+ "testing"
+ "time"
+
+ "go.ate.dev/substrate/internal/atenet"
+ "go.ate.dev/substrate/internal/benchmarking/boomer/boomerutil"
+ "go.ate.dev/substrate/internal/benchmarking/boomer/dynconfig"
+ "go.ate.dev/substrate/internal/benchmarking/boomer/userclass"
+ ateenvv1alpha "go.ate.dev/substrate/internal/proto/ateenv/v1alpha"
+ "go.ate.dev/substrate/pkg/proto/ateapipb"
+ "go.opentelemetry.io/otel"
+ "go.opentelemetry.io/otel/propagation"
+ sdktrace "go.opentelemetry.io/otel/sdk/trace"
+ "google.golang.org/grpc"
+ "google.golang.org/grpc/codes"
+ "google.golang.org/grpc/credentials/insecure"
+ "google.golang.org/grpc/metadata"
+ "google.golang.org/grpc/status"
+ "google.golang.org/grpc/test/bufconn"
+)
+
+// fakeControlClient implements ateapipb.ControlClient for testing.
+type fakeControlClient struct {
+ ateapipb.ControlClient
+ mu sync.Mutex
+ calls []string
+ suspendReqs []*ateapipb.SuspendActorRequest
+ resumeErrs []error
+ suspendErrs []error
+}
+
+func nextErr(errs *[]error) error {
+ if len(*errs) == 0 {
+ return nil
+ }
+ err := (*errs)[0]
+ *errs = (*errs)[1:]
+ return err
+}
+
+func (f *fakeControlClient) CreateAtespace(ctx context.Context, in *ateapipb.CreateAtespaceRequest, opts ...grpc.CallOption) (*ateapipb.Atespace, error) {
+ if err := ctx.Err(); err != nil {
+ return nil, err
+ }
+ f.mu.Lock()
+ defer f.mu.Unlock()
+ f.calls = append(f.calls, "CreateAtespace")
+ return &ateapipb.Atespace{}, nil
+}
+
+func (f *fakeControlClient) CreateActor(ctx context.Context, in *ateapipb.CreateActorRequest, opts ...grpc.CallOption) (*ateapipb.Actor, error) {
+ if err := ctx.Err(); err != nil {
+ return nil, err
+ }
+ f.mu.Lock()
+ defer f.mu.Unlock()
+ f.calls = append(f.calls, "CreateActor")
+ return &ateapipb.Actor{}, nil
+}
+
+func (f *fakeControlClient) ResumeActor(ctx context.Context, in *ateapipb.ResumeActorRequest, opts ...grpc.CallOption) (*ateapipb.ResumeActorResponse, error) {
+ if err := ctx.Err(); err != nil {
+ return nil, err
+ }
+ f.mu.Lock()
+ defer f.mu.Unlock()
+ f.calls = append(f.calls, "ResumeActor")
+ if err := nextErr(&f.resumeErrs); err != nil {
+ return nil, err
+ }
+ return &ateapipb.ResumeActorResponse{}, nil
+}
+
+func (f *fakeControlClient) SuspendActor(ctx context.Context, in *ateapipb.SuspendActorRequest, opts ...grpc.CallOption) (*ateapipb.SuspendActorResponse, error) {
+ if err := ctx.Err(); err != nil {
+ return nil, err
+ }
+ f.mu.Lock()
+ defer f.mu.Unlock()
+ f.calls = append(f.calls, suspendCallName(in))
+ f.suspendReqs = append(f.suspendReqs, in)
+ if err := nextErr(&f.suspendErrs); err != nil {
+ return nil, err
+ }
+ return &ateapipb.SuspendActorResponse{}, nil
+}
+
+// suspendCallName is the name a SuspendActor call is recorded under: an
+// async suspend is told apart from a sync one.
+func suspendCallName(in *ateapipb.SuspendActorRequest) string {
+ if in.GetOptions().GetMode() == ateapipb.DurabilityMode_DURABILITY_MODE_ASYNCHRONOUS {
+ return "SuspendActor(async)"
+ }
+ return "SuspendActor"
+}
+
+func (f *fakeControlClient) DeleteActor(ctx context.Context, in *ateapipb.DeleteActorRequest, opts ...grpc.CallOption) (*ateapipb.Actor, error) {
+ if err := ctx.Err(); err != nil {
+ return nil, err
+ }
+ f.mu.Lock()
+ defer f.mu.Unlock()
+ f.calls = append(f.calls, "DeleteActor")
+ return &ateapipb.Actor{}, nil
+}
+
+func (f *fakeControlClient) recordedCalls() []string {
+ f.mu.Lock()
+ defer f.mu.Unlock()
+ return append([]string(nil), f.calls...)
+}
+
+func (f *fakeControlClient) recordedSuspendRequests() []*ateapipb.SuspendActorRequest {
+ f.mu.Lock()
+ defer f.mu.Unlock()
+ return append([]*ateapipb.SuspendActorRequest(nil), f.suspendReqs...)
+}
+
+// fakeProcessServer implements ateenvv1alpha.ProcessServiceServer for testing.
+type fakeProcessServer struct {
+ ateenvv1alpha.UnimplementedProcessServiceServer
+ mu sync.Mutex
+ startReqs []*ateenvv1alpha.StartProcessRequest
+ streamReqs []*ateenvv1alpha.StreamProcessOutputRequest
+ incomingMD []metadata.MD
+ startErr error
+ streamErr error
+ emptyPID bool
+ omitExit bool
+ blockStream bool
+ customStdout *string
+ stdoutChunks []string
+ customExitState *ateenvv1alpha.ProcessState
+ customExitCode *int32
+}
+
+func (s *fakeProcessServer) StartProcess(ctx context.Context, in *ateenvv1alpha.StartProcessRequest) (*ateenvv1alpha.Process, error) {
+ s.mu.Lock()
+ defer s.mu.Unlock()
+ if md, ok := metadata.FromIncomingContext(ctx); ok {
+ s.incomingMD = append(s.incomingMD, md.Copy())
+ }
+ s.startReqs = append(s.startReqs, in)
+ if s.startErr != nil {
+ return nil, s.startErr
+ }
+ pid := "proc-1"
+ if s.emptyPID {
+ pid = ""
+ }
+ return &ateenvv1alpha.Process{
+ ProcessId: pid,
+ Command: in.GetCommand(),
+ State: ateenvv1alpha.ProcessState_PROCESS_STATE_RUNNING,
+ }, nil
+}
+
+func (s *fakeProcessServer) StreamProcessOutput(in *ateenvv1alpha.StreamProcessOutputRequest, stream grpc.ServerStreamingServer[ateenvv1alpha.ProcessOutput]) error {
+ s.mu.Lock()
+ if md, ok := metadata.FromIncomingContext(stream.Context()); ok {
+ s.incomingMD = append(s.incomingMD, md.Copy())
+ }
+ s.streamReqs = append(s.streamReqs, in)
+ streamErr := s.streamErr
+ omitExit := s.omitExit
+ blockStream := s.blockStream
+ stdout := expectedStdout
+ if s.customStdout != nil {
+ stdout = *s.customStdout
+ }
+ chunks := append([]string(nil), s.stdoutChunks...)
+ exitState := ateenvv1alpha.ProcessState_PROCESS_STATE_EXITED
+ if s.customExitState != nil {
+ exitState = *s.customExitState
+ }
+ exitCode := int32(0)
+ if s.customExitCode != nil {
+ exitCode = *s.customExitCode
+ }
+ s.mu.Unlock()
+
+ if blockStream {
+ <-stream.Context().Done()
+ return stream.Context().Err()
+ }
+ if streamErr != nil {
+ return streamErr
+ }
+
+ if len(chunks) > 0 {
+ for _, chunk := range chunks {
+ if err := stream.Send(&ateenvv1alpha.ProcessOutput{
+ Output: &ateenvv1alpha.ProcessOutput_Stdout{Stdout: []byte(chunk)},
+ }); err != nil {
+ return err
+ }
+ }
+ } else if len(stdout) > 0 {
+ if err := stream.Send(&ateenvv1alpha.ProcessOutput{
+ Output: &ateenvv1alpha.ProcessOutput_Stdout{Stdout: []byte(stdout)},
+ }); err != nil {
+ return err
+ }
+ }
+
+ if !omitExit {
+ if err := stream.Send(&ateenvv1alpha.ProcessOutput{
+ Output: &ateenvv1alpha.ProcessOutput_Exit{
+ Exit: &ateenvv1alpha.Process{
+ ProcessId: in.GetProcessId(),
+ State: exitState,
+ ExitCode: exitCode,
+ },
+ },
+ }); err != nil {
+ return err
+ }
+ }
+ return nil
+}
+
+type closerTracker struct {
+ io.Closer
+ closed bool
+ mu sync.Mutex
+}
+
+func (c *closerTracker) Close() error {
+ c.mu.Lock()
+ defer c.mu.Unlock()
+ c.closed = true
+ if c.Closer != nil {
+ return c.Closer.Close()
+ }
+ return nil
+}
+
+func (c *closerTracker) isClosed() bool {
+ c.mu.Lock()
+ defer c.mu.Unlock()
+ return c.closed
+}
+
+// testFixture sets up an in-memory gRPC server with fakeProcessServer.
+type testFixture struct {
+ srv *fakeProcessServer
+ grpcServer *grpc.Server
+ listener *bufconn.Listener
+ client ateenvv1alpha.ProcessServiceClient
+ closerTracker *closerTracker
+}
+
+func newTestFixture(t *testing.T, srv *fakeProcessServer) *testFixture {
+ t.Helper()
+ lis := bufconn.Listen(1 << 20)
+ grpcSrv := grpc.NewServer()
+ ateenvv1alpha.RegisterProcessServiceServer(grpcSrv, srv)
+ go func() {
+ _ = grpcSrv.Serve(lis)
+ }()
+
+ conn, err := grpc.NewClient(
+ "passthrough:///bufnet",
+ grpc.WithContextDialer(func(context.Context, string) (net.Conn, error) {
+ return lis.Dial()
+ }),
+ grpc.WithTransportCredentials(insecure.NewCredentials()),
+ )
+ if err != nil {
+ t.Fatalf("failed to dial bufnet: %v", err)
+ }
+
+ ct := &closerTracker{Closer: conn}
+ t.Cleanup(func() {
+ _ = conn.Close()
+ grpcSrv.Stop()
+ _ = lis.Close()
+ })
+
+ return &testFixture{
+ srv: srv,
+ grpcServer: grpcSrv,
+ listener: lis,
+ client: ateenvv1alpha.NewProcessServiceClient(conn),
+ closerTracker: ct,
+ }
+}
+
+func newTestRuntime(t *testing.T, ctrl *fakeControlClient, procSrv *fakeProcessServer, mode string) (*taskRuntime, *testFixture) {
+ t.Helper()
+ fix := newTestFixture(t, procSrv)
+ cfg := &userclass.Config{
+ APIStub: ctrl,
+ Atespace: "bench-test",
+ ActorsPerUser: 1,
+ Tracer: otel.Tracer("test"),
+ Dyn: dynconfig.Static(ateEnvKnobs{
+ Lifecycle: dynconfig.Lifecycle{LifecycleMode: mode},
+ }),
+ }
+ rt := &taskRuntime{
+ cfg: cfg,
+ dialProcessClient: func() (ateenvv1alpha.ProcessServiceClient, io.Closer, error) {
+ return fix.client, fix.closerTracker, nil
+ },
+ }
+ return rt, fix
+}
+
+func TestRouterGRPCTarget(t *testing.T) {
+ tests := []struct {
+ input string
+ want string
+ }{
+ {
+ input: "",
+ want: "dns:///atenet-router.ate-system.svc.cluster.local:80",
+ },
+ {
+ input: "http://atenet-router.ate-system.svc.cluster.local",
+ want: "dns:///atenet-router.ate-system.svc.cluster.local:80",
+ },
+ {
+ input: "http://atenet-router.ate-system.svc.cluster.local:8080",
+ want: "dns:///atenet-router.ate-system.svc.cluster.local:8080",
+ },
+ {
+ input: "router.example.internal:9000",
+ want: "dns:///router.example.internal:9000",
+ },
+ }
+
+ for _, tt := range tests {
+ t.Run(tt.input, func(t *testing.T) {
+ got, err := routerGRPCTarget(tt.input)
+ if err != nil {
+ t.Fatalf("routerGRPCTarget(%q) unexpected err: %v", tt.input, err)
+ }
+ if got != tt.want {
+ t.Errorf("routerGRPCTarget(%q) = %q, want %q", tt.input, got, tt.want)
+ }
+ })
+ }
+
+ if _, err := routerGRPCTarget("https://atenet-router.ate-system.svc.cluster.local"); err == nil {
+ t.Errorf("routerGRPCTarget(https) succeeded, want error")
+ }
+ if _, err := routerGRPCTarget("http:///missing-host"); err == nil {
+ t.Errorf("routerGRPCTarget(empty host) succeeded, want error")
+ }
+}
+
+func TestAteEnvIterate_SuspendMode(t *testing.T) {
+ ctrl := &fakeControlClient{}
+ srv := &fakeProcessServer{}
+ rt, fix := newTestRuntime(t, ctrl, srv, dynconfig.LifecycleModeSuspend)
+
+ rt.iterate()
+
+ calls := ctrl.recordedCalls()
+ if !slices.Contains(calls, "CreateAtespace") {
+ t.Errorf("expected CreateAtespace in calls, got: %v", calls)
+ }
+ if !slices.Contains(calls, "CreateActor") {
+ t.Errorf("expected CreateActor in calls, got: %v", calls)
+ }
+ if !slices.Contains(calls, "ResumeActor") {
+ t.Errorf("expected ResumeActor in calls, got: %v", calls)
+ }
+ if !slices.Contains(calls, "SuspendActor") {
+ t.Errorf("expected SuspendActor in calls, got: %v", calls)
+ }
+ if slices.Contains(calls, "SuspendActor(async)") {
+ t.Errorf("did not expect SuspendActor(async) in suspend mode calls, got: %v", calls)
+ }
+ for _, req := range ctrl.recordedSuspendRequests() {
+ if got := req.GetOptions().GetMode(); got != ateapipb.DurabilityMode_DURABILITY_MODE_SYNCHRONOUS {
+ t.Errorf("suspend mode = %v, want %v", got, ateapipb.DurabilityMode_DURABILITY_MODE_SYNCHRONOUS)
+ }
+ }
+
+ fix.srv.mu.Lock()
+ defer fix.srv.mu.Unlock()
+ if len(fix.srv.startReqs) != 1 {
+ t.Fatalf("expected 1 StartProcess request, got %d", len(fix.srv.startReqs))
+ }
+ wantCmd := []string{"echo", "hello world"}
+ if !slices.Equal(fix.srv.startReqs[0].GetCommand(), wantCmd) {
+ t.Errorf("command = %v, want %v", fix.srv.startReqs[0].GetCommand(), wantCmd)
+ }
+ if len(fix.srv.streamReqs) != 1 {
+ t.Fatalf("expected 1 StreamProcessOutput request, got %d", len(fix.srv.streamReqs))
+ }
+ if fix.srv.streamReqs[0].GetProcessId() != "proc-1" {
+ t.Errorf("stream process_id = %q, want 'proc-1'", fix.srv.streamReqs[0].GetProcessId())
+ }
+ if !fix.srv.streamReqs[0].GetFollow() {
+ t.Errorf("stream follow = false, want true")
+ }
+}
+
+func TestAteEnvIterate_PauseMode(t *testing.T) {
+ ctrl := &fakeControlClient{}
+ srv := &fakeProcessServer{}
+ rt, _ := newTestRuntime(t, ctrl, srv, dynconfig.LifecycleModePause)
+
+ rt.iterate()
+
+ calls := ctrl.recordedCalls()
+ if !slices.Contains(calls, "SuspendActor(async)") {
+ t.Errorf("expected SuspendActor(async) in calls, got: %v", calls)
+ }
+ if slices.Contains(calls, "SuspendActor") {
+ t.Errorf("did not expect SuspendActor in pause mode calls, got: %v", calls)
+ }
+}
+
+func TestAteEnvIterate_NoneMode(t *testing.T) {
+ ctrl := &fakeControlClient{}
+ srv := &fakeProcessServer{}
+ rt, _ := newTestRuntime(t, ctrl, srv, dynconfig.LifecycleModeNone)
+
+ rt.iterate()
+ rt.iterate()
+
+ calls := ctrl.recordedCalls()
+ if slices.Contains(calls, "SuspendActor") || slices.Contains(calls, "SuspendActor(async)") {
+ t.Errorf("did not expect a sync or async SuspendActor during iterate in none mode, got: %v", calls)
+ }
+ if got := countCalls(calls, "ResumeActor"); got != 1 {
+ t.Errorf("ResumeActor count across 2 iterations = %d, want 1 (first activation only); calls = %v", got, calls)
+ }
+
+ rt.shutdown(context.Background())
+ afterShutdown := ctrl.recordedCalls()
+ if len(afterShutdown) < 2 || afterShutdown[len(afterShutdown)-2] != "SuspendActor" || afterShutdown[len(afterShutdown)-1] != "DeleteActor" {
+ t.Errorf("shutdown in none mode must end with [SuspendActor, DeleteActor], got %v", afterShutdown)
+ }
+}
+
+func TestAteEnvExec_MetadataPropagation(t *testing.T) {
+ prevProp := otel.GetTextMapPropagator()
+ otel.SetTextMapPropagator(propagation.TraceContext{})
+ t.Cleanup(func() {
+ otel.SetTextMapPropagator(prevProp)
+ })
+ tp := sdktrace.NewTracerProvider(sdktrace.WithSampler(sdktrace.AlwaysSample()))
+ t.Cleanup(func() {
+ _ = tp.Shutdown(context.Background())
+ })
+
+ ctrl := &fakeControlClient{}
+ srv := &fakeProcessServer{
+ stdoutChunks: []string{"hello ", "world\n"},
+ }
+ rt, fix := newTestRuntime(t, ctrl, srv, dynconfig.LifecycleModeSuspend)
+ rt.cfg.Tracer = tp.Tracer("test-sampled")
+
+ rt.iterate()
+
+ fix.srv.mu.Lock()
+ defer fix.srv.mu.Unlock()
+ if len(fix.srv.incomingMD) < 2 {
+ t.Fatalf("expected at least 2 incoming metadata captures, got %d", len(fix.srv.incomingMD))
+ }
+
+ for i, md := range fix.srv.incomingMD {
+ vals := md.Get(atenet.TargetActorHeader)
+ if len(vals) != 1 {
+ t.Errorf("call %d: expected exactly 1 %q header, got %v", i, atenet.TargetActorHeader, vals)
+ } else if !strings.HasPrefix(vals[0], "bench-test/sb-") {
+ t.Errorf("call %d: invalid target actor header: %v", i, vals)
+ }
+ if tpVals := md.Get("traceparent"); len(tpVals) != 1 || tpVals[0] == "" {
+ t.Errorf("call %d: expected sampled traceparent header in metadata, got %v", i, tpVals)
+ }
+ }
+}
+
+func TestAteEnvExec_ValidationFailures(t *testing.T) {
+ tests := []struct {
+ name string
+ setupSrv func(*fakeProcessServer)
+ wantErr string
+ }{
+ {
+ name: "start process error",
+ setupSrv: func(s *fakeProcessServer) {
+ s.startErr = status.Error(codes.Internal, "failed to fork")
+ },
+ wantErr: "StartProcess:",
+ },
+ {
+ name: "empty process_id",
+ setupSrv: func(s *fakeProcessServer) {
+ s.emptyPID = true
+ },
+ wantErr: "StartProcess returned empty process_id",
+ },
+ {
+ name: "stream process output error",
+ setupSrv: func(s *fakeProcessServer) {
+ s.streamErr = status.Error(codes.Unavailable, "guest down")
+ },
+ wantErr: "StreamProcessOutput.Recv:",
+ },
+ {
+ name: "stream closed without exit event",
+ setupSrv: func(s *fakeProcessServer) {
+ s.omitExit = true
+ },
+ wantErr: "stream closed without exit event",
+ },
+ {
+ name: "non-zero exit code",
+ setupSrv: func(s *fakeProcessServer) {
+ code := int32(1)
+ s.customExitCode = &code
+ },
+ wantErr: "unexpected exit code: 1, want 0",
+ },
+ {
+ name: "unexpected process state",
+ setupSrv: func(s *fakeProcessServer) {
+ state := ateenvv1alpha.ProcessState_PROCESS_STATE_RUNNING
+ s.customExitState = &state
+ },
+ wantErr: "unexpected process state: PROCESS_STATE_RUNNING, want PROCESS_STATE_EXITED",
+ },
+ {
+ name: "stdout mismatch",
+ setupSrv: func(s *fakeProcessServer) {
+ mismatch := "hello mars\n"
+ s.customStdout = &mismatch
+ },
+ wantErr: "stdout mismatch:",
+ },
+ }
+
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ ctrl := &fakeControlClient{}
+ srv := &fakeProcessServer{}
+ tt.setupSrv(srv)
+ rt, fix := newTestRuntime(t, ctrl, srv, dynconfig.LifecycleModeSuspend)
+
+ actor := &ateEnvActor{
+ cfg: rt.cfg,
+ actorName: "sb-test",
+ processClient: fix.client,
+ }
+ err := actor.execHelloWorld(context.Background())
+ if err == nil || !strings.Contains(err.Error(), tt.wantErr) {
+ t.Fatalf("execHelloWorld() err = %v, want substring %q", err, tt.wantErr)
+ }
+
+ // iterate should execute and not panic or crash the runtime
+ rt.iterate()
+
+ calls := ctrl.recordedCalls()
+ // Control-plane lifecycle should still execute normally
+ if !slices.Contains(calls, "ResumeActor") || !slices.Contains(calls, "SuspendActor") {
+ t.Errorf("expected ResumeActor and SuspendActor in calls, got: %v", calls)
+ }
+ // A single exec failure is below maxConsecutiveFailures, so actor is not replaced yet.
+ if slices.Contains(calls, "DeleteActor") {
+ t.Errorf("did not expect DeleteActor on single exec failure, got: %v", calls)
+ }
+ })
+ }
+}
+
+func TestAteEnvExec_Timeout(t *testing.T) {
+ ctrl := &fakeControlClient{}
+ srv := &fakeProcessServer{blockStream: true}
+ rt, fix := newTestRuntime(t, ctrl, srv, dynconfig.LifecycleModeSuspend)
+
+ actor := &ateEnvActor{
+ cfg: rt.cfg,
+ actorName: "sb-timeout",
+ processClient: fix.client,
+ execTimeout: 20 * time.Millisecond,
+ }
+ err := actor.execHelloWorld(context.Background())
+ if err == nil || status.Code(err) != codes.DeadlineExceeded {
+ t.Fatalf("execHelloWorld() on hung stream err = %v, want DeadlineExceeded", err)
+ }
+}
+
+func TestAteEnvIterate_RetriesStrandedHibernate(t *testing.T) {
+ ctrl := &fakeControlClient{
+ suspendErrs: []error{status.Error(codes.Unavailable, "ate-api-server restarting")},
+ }
+ srv := &fakeProcessServer{}
+ rt, _ := newTestRuntime(t, ctrl, srv, dynconfig.LifecycleModeSuspend)
+
+ rt.iterate() // resume, exec, suspend -> suspend fails
+ before := ctrl.recordedCalls()
+ rt.iterate() // must re-drive suspend, not resume
+
+ after := ctrl.recordedCalls()
+ if got := after[len(after)-1]; got != "SuspendActor" {
+ t.Errorf("last call = %q, want SuspendActor; calls = %v", got, after)
+ }
+ resumeBefore := countCalls(before, "ResumeActor")
+ resumeAfter := countCalls(after, "ResumeActor")
+ if resumeBefore != resumeAfter {
+ t.Errorf("ResumeActor count increased from %d to %d (stranded actor should not resume)", resumeBefore, resumeAfter)
+ }
+}
+
+func TestAteEnvIterate_ReplacesActorOnTerminalResumeFailure(t *testing.T) {
+ ctrl := &fakeControlClient{
+ resumeErrs: []error{status.Error(codes.FailedPrecondition, "AssignWorker prerequisite not met")},
+ }
+ srv := &fakeProcessServer{}
+ rt, _ := newTestRuntime(t, ctrl, srv, dynconfig.LifecycleModeSuspend)
+
+ rt.iterate()
+
+ calls := ctrl.recordedCalls()
+ if got := countCalls(calls, "CreateActor"); got != 2 {
+ t.Errorf("CreateActor count = %d, want 2 (initial + replacement); calls = %v", got, calls)
+ }
+ if got := countCalls(calls, "DeleteActor"); got != 1 {
+ t.Errorf("DeleteActor count = %d, want 1; calls = %v", got, calls)
+ }
+}
+
+func TestAteEnvIterate_ReplacesActorOnRepeatedExecFailures(t *testing.T) {
+ ctrl := &fakeControlClient{}
+ srv := &fakeProcessServer{
+ startErr: status.Error(codes.Unavailable, "guest wedged"),
+ }
+ rt, _ := newTestRuntime(t, ctrl, srv, dynconfig.LifecycleModeSuspend)
+
+ for i := 0; i < maxConsecutiveFailures; i++ {
+ rt.iterate()
+ }
+
+ calls := ctrl.recordedCalls()
+ if got := countCalls(calls, "CreateActor"); got != 2 {
+ t.Errorf("CreateActor count = %d, want 2 (initial + replacement); calls = %v", got, calls)
+ }
+ if got := countCalls(calls, "SuspendActor"); got != maxConsecutiveFailures {
+ t.Errorf("SuspendActor count = %d, want %d (hibernated before replacement); calls = %v", got, maxConsecutiveFailures, calls)
+ }
+ if got := countCalls(calls, "DeleteActor"); got != 1 {
+ t.Errorf("DeleteActor count = %d, want 1; calls = %v", got, calls)
+ }
+}
+
+func TestAteEnvIterate_CountsAtMostOneExecFailurePerWake(t *testing.T) {
+ ctrl := &fakeControlClient{}
+ srv := &fakeProcessServer{
+ startErr: status.Error(codes.Unavailable, "transient router blip"),
+ }
+ rt, _ := newTestRuntime(t, ctrl, srv, dynconfig.LifecycleModeSuspend)
+ rt.cfg.Dyn = dynconfig.Static(ateEnvKnobs{
+ Lifecycle: dynconfig.Lifecycle{LifecycleMode: dynconfig.LifecycleModeSuspend},
+ MinLive: dynconfig.Seconds(5 * time.Second),
+ MaxLive: dynconfig.Seconds(5 * time.Second),
+ MaxPingsPerWake: maxConsecutiveFailures,
+ })
+ rt.sleepFn = func(time.Duration) bool { return true }
+
+ // A single wake with MaxPingsPerWake == maxConsecutiveFailures failed pings
+ // must only count as 1 failed wake, not replace the actor immediately.
+ rt.iterate()
+
+ calls := ctrl.recordedCalls()
+ if got := countCalls(calls, "CreateActor"); got != 1 {
+ t.Errorf("CreateActor count after 1 wake = %d, want 1; calls = %v", got, calls)
+ }
+ if got := countCalls(calls, "DeleteActor"); got != 0 {
+ t.Errorf("DeleteActor count after 1 wake = %d, want 0; calls = %v", got, calls)
+ }
+}
+
+func TestAteEnvIterate_DoesNotReplaceOnResourceExhaustedExecFailure(t *testing.T) {
+ ctrl := &fakeControlClient{}
+ srv := &fakeProcessServer{
+ startErr: status.Error(codes.ResourceExhausted, "rate limit exceeded"),
+ }
+ rt, _ := newTestRuntime(t, ctrl, srv, dynconfig.LifecycleModeSuspend)
+
+ for i := 0; i < maxConsecutiveFailures+1; i++ {
+ rt.iterate()
+ }
+
+ calls := ctrl.recordedCalls()
+ if got := countCalls(calls, "CreateActor"); got != 1 {
+ t.Errorf("CreateActor count = %d, want 1 (ResourceExhausted should not replace actor); calls = %v", got, calls)
+ }
+ if got := countCalls(calls, "DeleteActor"); got != 0 {
+ t.Errorf("DeleteActor count = %d, want 0; calls = %v", got, calls)
+ }
+}
+
+func TestAteEnvShutdown_PauseModeRunningActor(t *testing.T) {
+ ctrl := &fakeControlClient{}
+ srv := &fakeProcessServer{}
+ rt, fix := newTestRuntime(t, ctrl, srv, dynconfig.LifecycleModePause)
+
+ u := &ateEnvUser{
+ routerConn: fix.closerTracker,
+ actors: []*ateEnvActor{{
+ cfg: rt.cfg,
+ actorName: "running-actor",
+ actorRunning: true,
+ processClient: fix.client,
+ }},
+ }
+ rt.users.Store(boomerutil.GoroutineID(), u)
+
+ rt.shutdown(context.Background())
+
+ calls := ctrl.recordedCalls()
+ if len(calls) != 2 || calls[0] != "SuspendActor(async)" || calls[1] != "DeleteActor" {
+ t.Errorf("recordedCalls = %v, want [SuspendActor(async) DeleteActor]", calls)
+ }
+ if !fix.closerTracker.isClosed() {
+ t.Errorf("router gRPC connection was not closed on shutdown")
+ }
+}
+
+func TestAteEnvShutdown_StoppedActor(t *testing.T) {
+ ctrl := &fakeControlClient{}
+ srv := &fakeProcessServer{}
+ rt, fix := newTestRuntime(t, ctrl, srv, dynconfig.LifecycleModeSuspend)
+
+ u := &ateEnvUser{
+ routerConn: fix.closerTracker,
+ actors: []*ateEnvActor{{
+ cfg: rt.cfg,
+ actorName: "stopped-actor",
+ actorRunning: false,
+ processClient: fix.client,
+ }},
+ }
+ rt.users.Store(boomerutil.GoroutineID(), u)
+
+ rt.shutdown(context.Background())
+
+ calls := ctrl.recordedCalls()
+ if len(calls) != 1 || calls[0] != "DeleteActor" {
+ t.Errorf("recordedCalls = %v, want [DeleteActor]", calls)
+ }
+ if !fix.closerTracker.isClosed() {
+ t.Errorf("router gRPC connection was not closed on shutdown")
+ }
+}
+
+func TestAteEnvShutdown_WaitsForInFlightIterate(t *testing.T) {
+ ctrl := &fakeControlClient{}
+ srv := &fakeProcessServer{}
+ rt, fix := newTestRuntime(t, ctrl, srv, dynconfig.LifecycleModeSuspend)
+ rt.cfg.Dyn = dynconfig.Static(ateEnvKnobs{
+ Lifecycle: dynconfig.Lifecycle{LifecycleMode: dynconfig.LifecycleModeSuspend},
+ WaitTime: dynconfig.WaitTime{MinWait: dynconfig.Seconds(5 * time.Second), MaxWait: dynconfig.Seconds(5 * time.Second)},
+ MinLive: dynconfig.Seconds(5 * time.Second),
+ MaxLive: dynconfig.Seconds(5 * time.Second),
+ MaxPingsPerWake: 1,
+ })
+
+ iterDone := make(chan struct{})
+ go func() {
+ defer close(iterDone)
+ rt.iterate()
+ }()
+
+ // Wait until StartProcess has been invoked inside iterate().
+ deadline := time.Now().Add(2 * time.Second)
+ for {
+ fix.srv.mu.Lock()
+ n := len(fix.srv.startReqs)
+ fix.srv.mu.Unlock()
+ if n > 0 {
+ break
+ }
+ if time.Now().After(deadline) {
+ t.Fatal("timed out waiting for iterate() to reach StartProcess")
+ }
+ time.Sleep(5 * time.Millisecond)
+ }
+
+ // shutdown should interrupt the 5s liveWait/dynamicWait sleep, wait for
+ // iterate()'s SuspendActor to finish, and then DeleteActor once without
+ // triggering any replacement CreateActor.
+ start := time.Now()
+ rt.shutdown(context.Background())
+ <-iterDone
+ if elapsed := time.Since(start); elapsed > 2*time.Second {
+ t.Errorf("shutdown took %v, expected live/wait sleeps to be interrupted", elapsed)
+ }
+
+ calls := ctrl.recordedCalls()
+ if got := countCalls(calls, "CreateActor"); got != 1 {
+ t.Errorf("CreateActor count = %d, want 1; calls = %v", got, calls)
+ }
+ if got := countCalls(calls, "SuspendActor"); got != 1 {
+ t.Errorf("SuspendActor count = %d, want 1; calls = %v", got, calls)
+ }
+ if got := countCalls(calls, "DeleteActor"); got != 1 {
+ t.Errorf("DeleteActor count = %d, want 1; calls = %v", got, calls)
+ }
+}
+
+func TestAteEnvShutdown_RespectsContextCancellation(t *testing.T) {
+ ctrl := &fakeControlClient{}
+ srv := &fakeProcessServer{blockStream: true}
+ rt, fix := newTestRuntime(t, ctrl, srv, dynconfig.LifecycleModeSuspend)
+
+ iterDone := make(chan struct{})
+ go func() {
+ defer close(iterDone)
+ rt.iterate()
+ }()
+
+ // Wait until StreamProcessOutput is blocked inside iterate().
+ deadline := time.Now().Add(2 * time.Second)
+ for {
+ fix.srv.mu.Lock()
+ n := len(fix.srv.streamReqs)
+ fix.srv.mu.Unlock()
+ if n > 0 {
+ break
+ }
+ if time.Now().After(deadline) {
+ t.Fatal("timed out waiting for iterate() to block in StreamProcessOutput")
+ }
+ time.Sleep(5 * time.Millisecond)
+ }
+
+ ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
+ defer cancel()
+
+ start := time.Now()
+ rt.shutdown(ctx)
+ if elapsed := time.Since(start); elapsed > 2*time.Second {
+ t.Errorf("shutdown ignored ctx cancellation and took %v", elapsed)
+ }
+ // Closing routerConn during shutdown unblocks the stuck stream so iterate()
+ // also exits cleanly.
+ <-iterDone
+
+ calls := ctrl.recordedCalls()
+ if !slices.Contains(calls, "SuspendActor") {
+ t.Errorf("expected SuspendActor to succeed before ctx expired, got: %v", calls)
+ }
+ if !slices.Contains(calls, "DeleteActor") {
+ t.Errorf("expected DeleteActor to succeed before ctx expired, got: %v", calls)
+ }
+}
+
+func countCalls(calls []string, name string) int {
+ n := 0
+ for _, c := range calls {
+ if c == name {
+ n++
+ }
+ }
+ return n
+}
diff --git a/internal/benchmarking/boomer/glutton/durdir.go b/internal/benchmarking/boomer/glutton/durdir.go
index 7adc227a06..d6cbc1ee7e 100644
--- a/internal/benchmarking/boomer/glutton/durdir.go
+++ b/internal/benchmarking/boomer/glutton/durdir.go
@@ -257,8 +257,7 @@ func (u *durDirUser) hibernateAndDelete(ctx context.Context, dynCfg durDirKnobs)
func (u *durDirUser) delete(ctx context.Context) {
_ = u.tracedCall(ctx, "DeleteActor", func(callCtx context.Context, tr *metadata.MD) error {
_, err := u.cfg.APIStub.DeleteActor(callCtx, &ateapipb.DeleteActorRequest{
- Actor: u.ref(),
- AnyState: true,
+ Actor: u.ref(),
}, grpc.Trailer(tr))
return err
})
diff --git a/internal/benchmarking/boomer/glutton/durdir_test.go b/internal/benchmarking/boomer/glutton/durdir_test.go
index a1ab0524ff..a0b7ce763b 100644
--- a/internal/benchmarking/boomer/glutton/durdir_test.go
+++ b/internal/benchmarking/boomer/glutton/durdir_test.go
@@ -298,10 +298,6 @@ func TestDurDirShutdownHibernatesBeforeDelete(t *testing.T) {
if got := suspends[0].GetOptions().GetMode(); got != tc.wantMode {
t.Errorf("suspend mode = %v, want %v", got, tc.wantMode)
}
- reqs := fakeCtrl.recordedDeleteRequests()
- if len(reqs) == 0 || !reqs[0].GetAnyState() {
- t.Errorf("DeleteActor must set AnyState=true, got %v", reqs)
- }
})
}
}
diff --git a/internal/benchmarking/boomer/glutton/lifecycle.go b/internal/benchmarking/boomer/glutton/lifecycle.go
index e50af663bc..af34d78c6e 100644
--- a/internal/benchmarking/boomer/glutton/lifecycle.go
+++ b/internal/benchmarking/boomer/glutton/lifecycle.go
@@ -503,8 +503,7 @@ func (u *gluttonActor) suspend(ctx context.Context) error {
func (u *gluttonActor) delete(ctx context.Context) {
_ = u.tracedCall(ctx, "DeleteActor", func(callCtx context.Context, tr *metadata.MD) error {
_, err := u.cfg.APIStub.DeleteActor(callCtx, &ateapipb.DeleteActorRequest{
- Actor: u.ref(),
- AnyState: true,
+ Actor: u.ref(),
}, grpc.Trailer(tr))
return err
})
diff --git a/internal/benchmarking/boomer/glutton/lifecycle_test.go b/internal/benchmarking/boomer/glutton/lifecycle_test.go
index 1aa434c9d3..8a6d71c517 100644
--- a/internal/benchmarking/boomer/glutton/lifecycle_test.go
+++ b/internal/benchmarking/boomer/glutton/lifecycle_test.go
@@ -134,13 +134,9 @@ func TestGluttonShutdown_PauseModeRunningActor(t *testing.T) {
if len(calls) < 2 || calls[len(calls)-2] != "SuspendActor(async)" || calls[len(calls)-1] != "DeleteActor" {
t.Errorf("recordedCalls must end with [SuspendActor(async), DeleteActor], got %v", calls)
}
- reqs := fakeCtrl.recordedDeleteRequests()
- if len(reqs) == 0 || !reqs[0].GetAnyState() {
- t.Errorf("DeleteActor must set AnyState=true, got %v", reqs)
- }
}
-func TestGluttonShutdown_DeleteSetsAnyState(t *testing.T) {
+func TestGluttonShutdown_StoppedActorOnlyDeleted(t *testing.T) {
srv := &fake.Server{}
fakeCtrl := &fakeControlClient{}
cfg := newTestConfig(t, srv, &userclass.Config{
@@ -165,10 +161,6 @@ func TestGluttonShutdown_DeleteSetsAnyState(t *testing.T) {
if len(calls) != 1 || calls[0] != "DeleteActor" {
t.Errorf("expected only DeleteActor call, got %v", calls)
}
- reqs := fakeCtrl.recordedDeleteRequests()
- if len(reqs) == 0 || !reqs[0].GetAnyState() {
- t.Errorf("DeleteActor must set AnyState=true, got %v", reqs)
- }
}
// newReplacementRuntime builds a one-actor-per-VU suspend-mode runtime whose
diff --git a/internal/benchmarking/boomer/glutton/spawn.go b/internal/benchmarking/boomer/glutton/spawn.go
index 28175db539..4ae40c6aee 100644
--- a/internal/benchmarking/boomer/glutton/spawn.go
+++ b/internal/benchmarking/boomer/glutton/spawn.go
@@ -483,7 +483,6 @@ func (r *spawnRuntime) deleteActor(ctx context.Context, name string) error {
Atespace: r.cfg.Atespace,
Name: name,
},
- AnyState: true,
}, grpc.Trailer(tr))
return err
})
diff --git a/internal/benchmarking/boomer/resumepaused/resumepaused.go b/internal/benchmarking/boomer/resumepaused/resumepaused.go
index 020080686b..e6364311db 100644
--- a/internal/benchmarking/boomer/resumepaused/resumepaused.go
+++ b/internal/benchmarking/boomer/resumepaused/resumepaused.go
@@ -334,8 +334,8 @@ func (r *runtime) pause(ctx context.Context, suffix, name string) error {
// the ones in flight, and a delete that races one of them loses the actor
// lease and fails with Aborted. The running ones get half of ctx's remaining
// time; any still running then are canceled, so a hung RPC cannot spend the
-// time the deletes need. DeleteActor with AnyState removes an actor in any
-// state, so a canceled cycle or setup still leaves nothing behind.
+// time the deletes need. DeleteActor removes an actor in any state, so a
+// canceled cycle or setup still leaves nothing behind.
func (r *runtime) shutdown(ctx context.Context) {
r.stopped.Store(true)
r.cycleMu.Lock()
@@ -407,8 +407,7 @@ func (r *runtime) delete(ctx context.Context, name string) error {
}
err = r.call(ctx, "DeleteActor", func(ctx context.Context) error {
_, err := r.cfg.APIStub.DeleteActor(ctx, &ateapipb.DeleteActorRequest{
- Actor: r.ref(name),
- AnyState: true,
+ Actor: r.ref(name),
})
return err
})
diff --git a/internal/benchmarking/boomer/sweperf/sweperf.go b/internal/benchmarking/boomer/sweperf/sweperf.go
index 878e84c5e6..b104a88119 100644
--- a/internal/benchmarking/boomer/sweperf/sweperf.go
+++ b/internal/benchmarking/boomer/sweperf/sweperf.go
@@ -416,11 +416,11 @@ func (r *sweperfRuntime) startUser(ctx context.Context) (_ *sweperfUser, err err
// retireUser deletes the actor of a session that finished its trajectory and
// unbinds it from goroutine gid, so the next iterate starts a fresh actor. The
-// final cycle leaves the actor running, and DeleteActor with AnyState removes
-// it from any state, so no suspend is sent first. The delete runs detached
-// from the run context so shutdown cannot cut it short. A failed delete is
-// retried once, with a fresh timeout; if that also fails the actor is left
-// behind and logged.
+// final cycle leaves the actor running, and DeleteActor removes it from any
+// state, so no suspend is sent first. The delete runs detached from the run
+// context so shutdown cannot cut it short. A failed delete is retried once,
+// with a fresh timeout; if that also fails the actor is left behind and
+// logged.
func (r *sweperfRuntime) retireUser(gid int64, u *sweperfUser) {
r.users.CompareAndDelete(gid, u)
if !u.cleanedUp.CompareAndSwap(false, true) {
@@ -623,17 +623,15 @@ func (u *sweperfUser) delete(ctx context.Context) error {
return u.tracedCall(ctx, "DeleteActor", func(callCtx context.Context, tr *metadata.MD) error {
_, err := u.cfg.APIStub.DeleteActor(callCtx, &ateapipb.DeleteActorRequest{
Actor: u.ref(),
- // Teardown discards the actor, so skip the SUSPENDED precondition.
- AnyState: true,
}, grpc.Trailer(tr))
return err
})
}
// teardown releases the actor and the worker it holds, and decrements the
-// user gauge. Like retireUser, it deletes with AnyState and sends no suspend
-// first. Safe to call more than once and from several goroutines: only the
-// first call does anything.
+// user gauge. Like retireUser, it sends no suspend before deleting. Safe to
+// call more than once and from several goroutines: only the first call does
+// anything.
func (u *sweperfUser) teardown(ctx context.Context) {
if !u.cleanedUp.CompareAndSwap(false, true) {
return
diff --git a/internal/benchmarking/boomer/sweperf/sweperf_test.go b/internal/benchmarking/boomer/sweperf/sweperf_test.go
index 1c63c66c33..c4476642d8 100644
--- a/internal/benchmarking/boomer/sweperf/sweperf_test.go
+++ b/internal/benchmarking/boomer/sweperf/sweperf_test.go
@@ -108,8 +108,6 @@ type fakeControlClient struct {
// Make ResumeActor take time and report a smaller server-side elapsed.
resumeDelay time.Duration
resumeElapsedUs string
- // AnyState carried by the most recent DeleteActor request.
- deleteAnyState bool
// beforeDelete, if set, runs at the start of each DeleteActor, outside mu.
beforeDelete func()
}
@@ -175,7 +173,6 @@ func (f *fakeControlClient) DeleteActor(ctx context.Context, in *ateapipb.Delete
f.mu.Lock()
defer f.mu.Unlock()
f.calls = append(f.calls, "DeleteActor")
- f.deleteAnyState = in.GetAnyState()
if err := ctx.Err(); err != nil {
return nil, err
}
@@ -847,10 +844,6 @@ func TestShutdownDeletesWithoutSuspend(t *testing.T) {
if got, want := fakeCtrl.recordedCalls(), []string{"DeleteActor"}; !reflect.DeepEqual(got, want) {
t.Errorf("recordedCalls = %v, want %v", got, want)
}
- // Without AnyState the server refuses to delete an actor caught mid-cycle.
- if !fakeCtrl.deleteAnyState {
- t.Error("DeleteActor sent AnyState=false; an actor not yet SUSPENDED would leak its worker")
- }
}
func TestTracedCallRecordsBothLatencies(t *testing.T) {
diff --git a/internal/e2e/collector_metrics.go b/internal/e2e/collector_metrics.go
index 45aad928c5..f8519f3fd1 100644
--- a/internal/e2e/collector_metrics.go
+++ b/internal/e2e/collector_metrics.go
@@ -59,6 +59,7 @@ var PlatformMetricPrefixes = []string{
"atelet_snapshot_upload_overdue_actors",
"atelet_snapshot_undurable_age_max",
"atenet_router_route_duration",
+ "ate_filecache_size",
}
// ScrapeAgentGatewayRouterMetrics reads the AgentGateway router's native
diff --git a/internal/e2e/fixtures/testserver/kill.go b/internal/e2e/fixtures/testserver/kill.go
new file mode 100644
index 0000000000..574b7a565d
--- /dev/null
+++ b/internal/e2e/fixtures/testserver/kill.go
@@ -0,0 +1,77 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package main
+
+import (
+ "errors"
+ "fmt"
+ "os"
+ "path/filepath"
+ "strconv"
+ "strings"
+ "syscall"
+
+ "github.com/spf13/cobra"
+)
+
+func newKillCmd() *cobra.Command {
+ var contains string
+ cmd := &cobra.Command{
+ Use: "kill",
+ Short: "SIGKILL every process whose command line contains a string, as a crash would.",
+ Args: cobra.NoArgs,
+ RunE: func(_ *cobra.Command, _ []string) error {
+ n, err := killMatching("/proc", contains, os.Getpid())
+ if err != nil {
+ return err
+ }
+ fmt.Printf("killed %d\n", n)
+ if n == 0 {
+ return fmt.Errorf("no process command line contains %q", contains)
+ }
+ return nil
+ },
+ }
+ cmd.Flags().StringVar(&contains, "cmdline-contains", "", "kill processes whose command line contains this")
+ _ = cmd.MarkFlagRequired("cmdline-contains")
+ return cmd
+}
+
+// killMatching kills the processes under procDir whose command line contains
+// contains, other than self, and reports how many it killed.
+func killMatching(procDir, contains string, self int) (int, error) {
+ entries, err := os.ReadDir(procDir)
+ if err != nil {
+ return 0, err
+ }
+ killed := 0
+ for _, e := range entries {
+ pid, err := strconv.Atoi(e.Name())
+ if err != nil || pid == self {
+ continue
+ }
+ cmdline, err := os.ReadFile(filepath.Join(procDir, e.Name(), "cmdline"))
+ if err != nil || !strings.Contains(string(cmdline), contains) {
+ continue
+ }
+ // already gone counts: a gofer or virtiofsd can exit on its own once
+ // the process it serves is killed
+ if err := syscall.Kill(pid, syscall.SIGKILL); err != nil && !errors.Is(err, syscall.ESRCH) {
+ return killed, fmt.Errorf("killing %d: %w", pid, err)
+ }
+ killed++
+ }
+ return killed, nil
+}
diff --git a/internal/e2e/fixtures/testserver/kill_test.go b/internal/e2e/fixtures/testserver/kill_test.go
new file mode 100644
index 0000000000..d2017e3b27
--- /dev/null
+++ b/internal/e2e/fixtures/testserver/kill_test.go
@@ -0,0 +1,82 @@
+//go:build linux
+
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package main
+
+import (
+ "os"
+ "os/exec"
+ "path/filepath"
+ "strconv"
+ "testing"
+ "time"
+)
+
+func TestKillMatching(t *testing.T) {
+ cmd := exec.Command("sleep", "31337")
+ if err := cmd.Start(); err != nil {
+ t.Fatal(err)
+ }
+ exited := make(chan struct{})
+ go func() { _ = cmd.Wait(); close(exited) }()
+
+ n, err := killMatching("/proc", "sleep\x0031337", os.Getpid())
+ if err != nil {
+ t.Fatal(err)
+ }
+ if n != 1 {
+ t.Fatalf("killed %d, want 1", n)
+ }
+ select {
+ case <-exited:
+ case <-time.After(10 * time.Second):
+ t.Fatal("matching process still running")
+ }
+}
+
+func TestKillMatchingSkipsSelf(t *testing.T) {
+ // the test binary's own command line contains its name
+ n, err := killMatching("/proc", os.Args[0], os.Getpid())
+ if err != nil {
+ t.Fatal(err)
+ }
+ if n != 0 {
+ t.Fatalf("killed %d, want 0", n)
+ }
+}
+
+func TestKillMatchingCountsAProcessAlreadyGone(t *testing.T) {
+ cmd := exec.Command("true")
+ if err := cmd.Run(); err != nil {
+ t.Fatal(err)
+ }
+ // a process listed but reaped by the time it is killed
+ proc := t.TempDir()
+ dir := filepath.Join(proc, strconv.Itoa(cmd.Process.Pid))
+ if err := os.Mkdir(dir, 0o700); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.WriteFile(filepath.Join(dir, "cmdline"), []byte("virtiofsd\x00actor-uid"), 0o600); err != nil {
+ t.Fatal(err)
+ }
+ n, err := killMatching(proc, "actor-uid", os.Getpid())
+ if err != nil {
+ t.Fatal(err)
+ }
+ if n != 1 {
+ t.Fatalf("killed %d, want 1", n)
+ }
+}
diff --git a/internal/e2e/fixtures/testserver/main.go b/internal/e2e/fixtures/testserver/main.go
index c17ae28af9..10a3af7a09 100644
--- a/internal/e2e/fixtures/testserver/main.go
+++ b/internal/e2e/fixtures/testserver/main.go
@@ -45,7 +45,7 @@ func main() {
Use: "testserver",
Short: "Multi-mode helper server for the egress e2e suites.",
}
- root.AddCommand(newGRPCCmd(), newHTTPCmd(), newEgressProbeCmd(), newWebsocketCmd())
+ root.AddCommand(newGRPCCmd(), newHTTPCmd(), newEgressProbeCmd(), newKillCmd(), newWebsocketCmd())
if err := root.Execute(); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
diff --git a/internal/e2e/sandboxkill.go b/internal/e2e/sandboxkill.go
new file mode 100644
index 0000000000..4ec51c02b9
--- /dev/null
+++ b/internal/e2e/sandboxkill.go
@@ -0,0 +1,103 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package e2e
+
+import (
+ "context"
+ "fmt"
+ "os"
+ "path/filepath"
+ "strings"
+ "testing"
+ "time"
+
+ corev1 "k8s.io/api/core/v1"
+ metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+ "k8s.io/utils/ptr"
+)
+
+// testserverImage resolves the e2e testserver image, whose kill subcommand
+// does the killing.
+func testserverImage(t *testing.T) string {
+ t.Helper()
+ manifest := filepath.Join(t.TempDir(), "image.yaml")
+ if err := os.WriteFile(manifest, []byte("image: ko://go.ate.dev/substrate/internal/e2e/fixtures/testserver\n"), 0o600); err != nil {
+ t.Fatal(err)
+ }
+ // the first line; ko appends a document separator
+ line, _, _ := strings.Cut(string(koResolve(t, manifest)), "\n")
+ image := strings.TrimSpace(strings.TrimPrefix(line, "image:"))
+ if image == "" {
+ t.Fatal("ko resolved no testserver image")
+ }
+ return image
+}
+
+// KillActorSandbox kills the actor's sandbox processes on its worker, the way a
+// crash would, from an ephemeral container sharing the worker's PID namespace.
+func KillActorSandbox(ctx context.Context, t *testing.T, clients *Clients, namespace, podName, actorUID string) {
+ t.Helper()
+ pods := clients.K8s.CoreV1().Pods(namespace)
+ pod, err := pods.Get(ctx, podName, metav1.GetOptions{})
+ if err != nil {
+ t.Fatalf("getting worker pod %s/%s: %v", namespace, podName, err)
+ }
+ name := fmt.Sprintf("kill-sandbox-%d", len(pod.Spec.EphemeralContainers))
+ pod.Spec.EphemeralContainers = append(pod.Spec.EphemeralContainers, corev1.EphemeralContainer{
+ TargetContainerName: "worker",
+ EphemeralContainerCommon: corev1.EphemeralContainerCommon{
+ Name: name,
+ Image: testserverImage(t),
+ // kills the runsc sandbox and gofer, or cloud-hypervisor and
+ // virtiofsd: each names the actor UID in its command line
+ Args: []string{"kill", "--cmdline-contains", actorUID},
+ // root, like the sandbox processes, which also run unconfined, and
+ // the default AppArmor profile, enforced on GKE, may not signal them
+ SecurityContext: &corev1.SecurityContext{
+ RunAsUser: ptr.To[int64](0),
+ AppArmorProfile: &corev1.AppArmorProfile{Type: corev1.AppArmorProfileTypeUnconfined},
+ },
+ },
+ })
+ if _, err := pods.UpdateEphemeralContainers(ctx, podName, pod, metav1.UpdateOptions{}); err != nil {
+ t.Fatalf("adding ephemeral container to %s/%s: %v", namespace, podName, err)
+ }
+
+ deadline := time.Now().Add(2 * time.Minute)
+ for time.Now().Before(deadline) {
+ pod, err := pods.Get(ctx, podName, metav1.GetOptions{})
+ if err != nil {
+ t.Fatalf("getting worker pod %s/%s: %v", namespace, podName, err)
+ }
+ for _, st := range pod.Status.EphemeralContainerStatuses {
+ if st.Name != name {
+ continue
+ }
+ if w := st.State.Waiting; w != nil && w.Reason != "" && w.Reason != "ContainerCreating" && w.Reason != "PodInitializing" {
+ t.Fatalf("kill container for actor %s cannot start: %s: %s", actorUID, w.Reason, w.Message)
+ }
+ if st.State.Terminated == nil {
+ continue
+ }
+ if code := st.State.Terminated.ExitCode; code != 0 {
+ t.Fatalf("killing the sandbox of actor %s exited %d", actorUID, code)
+ }
+ t.Logf("Killed the sandbox of actor %s on %s/%s", actorUID, namespace, podName)
+ return
+ }
+ time.Sleep(time.Second)
+ }
+ t.Fatalf("timed out killing the sandbox of actor %s", actorUID)
+}
diff --git a/internal/e2e/suites/authz/security_regression_test.go b/internal/e2e/suites/authz/security_regression_test.go
index 7d80979e4a..31b0e420ee 100644
--- a/internal/e2e/suites/authz/security_regression_test.go
+++ b/internal/e2e/suites/authz/security_regression_test.go
@@ -304,8 +304,24 @@ func testWorkerServiceRequiresAtelet(t *testing.T, ctx context.Context, f *fixtu
})
return err
}
- // TODO(#2570): also check ReportActorSnapshotState once it is served. Until
- // then every caller gets UNIMPLEMENTED, before anything authenticates it.
+ // This handler also validates the request before authenticating the
+ // caller, so the request must be valid.
+ mintWorkerJWT := func(ctx context.Context, api *ateclient.Client) error {
+ _, err := api.WorkerService().MintWorkerActorJwt(ctx, &ateapipb.MintWorkerActorJwtRequest{
+ Actor: f.actor,
+ ActorUid: "00000000-0000-4000-8000-000000000000",
+ Audiences: []string{"authz-e2e"},
+ })
+ return err
+ }
+ reportSnapshotState := func(ctx context.Context, api *ateclient.Client) error {
+ _, err := api.WorkerService().ReportActorSnapshotState(ctx, &ateapipb.ReportActorSnapshotStateRequest{
+ Actor: f.actor,
+ ActorUid: "00000000-0000-4000-8000-000000000000",
+ State: &ateapipb.ActorSnapshotState{},
+ })
+ return err
+ }
expectAll(t, ctx, []call{
{"RegisterWorker", f.bootstrap, registerWorker, rejected},
@@ -314,5 +330,9 @@ func testWorkerServiceRequiresAtelet(t *testing.T, ctx context.Context, f *fixtu
{"RequestActorSuspend", ateletToken, requestSuspend, rejected},
{"MintWorkerActorCertificate", f.bootstrap, mintWorkerCert, rejected},
{"MintWorkerActorCertificate", ateletToken, mintWorkerCert, rejected},
+ {"MintWorkerActorJwt", f.bootstrap, mintWorkerJWT, rejected},
+ {"MintWorkerActorJwt", ateletToken, mintWorkerJWT, rejected},
+ {"ReportActorSnapshotState", f.bootstrap, reportSnapshotState, rejected},
+ {"ReportActorSnapshotState", ateletToken, reportSnapshotState, rejected},
})
}
diff --git a/internal/e2e/suites/combinedvolumes/combinedvolumes_test.go b/internal/e2e/suites/combinedvolumes/combinedvolumes_test.go
index d46373d338..e26ce7606b 100644
--- a/internal/e2e/suites/combinedvolumes/combinedvolumes_test.go
+++ b/internal/e2e/suites/combinedvolumes/combinedvolumes_test.go
@@ -234,10 +234,10 @@ func createTemplate(ctx context.Context, t *testing.T, clients *e2e.Clients, ns
&ateapipb.VolumeMount{Name: extVolume, MountPath: extPathB})
tmpl.Volumes = append(tmpl.Volumes, &ateapipb.Volume{
Name: extVolume,
- ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{
+ External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{
Capacity: extCapacity,
StorageClassName: storageClass,
- },
+ }},
})
},
})
diff --git a/internal/e2e/suites/demo/demo_test.go b/internal/e2e/suites/demo/demo_test.go
index 0aa113e8a8..8009a27cb9 100644
--- a/internal/e2e/suites/demo/demo_test.go
+++ b/internal/e2e/suites/demo/demo_test.go
@@ -77,12 +77,12 @@ func TestActorLifecycle(t *testing.T) {
f: suspendActor,
},
{
- name: "DeleteActorAnyState",
- f: deleteActorAnyState,
+ name: "DeleteRunningActor",
+ f: deleteRunningActor,
},
{
- name: "DeletePausedActorAnyState",
- f: deletePausedActorAnyState,
+ name: "DeletePausedActor",
+ f: deletePausedActor,
},
{
name: "RevertResumeActor",
@@ -188,6 +188,7 @@ func TestActorSnapshotLifecycle(t *testing.T) {
if diff := cmp.Diff(assignment.GetSandboxRuntime(), snapshotStamp, protocmp.Transform()); diff != "" {
t.Fatalf("suspended snapshot stamp mismatch (-assignment +snapshot):\n%s", diff)
}
+ assertSnapshotUnderLocation(t, snapshotURI, wantSnapshotLocation(t, at))
// The tag is given its own copy of the snapshot the suspend just wrote, so
// it survives the Actor being suspended again or deleted.
tagToUpdate, err := clients.SubstrateAPI.CreateTag(ctx, &ateapipb.CreateTagRequest{
@@ -209,6 +210,7 @@ func TestActorSnapshotLifecycle(t *testing.T) {
if diff := cmp.Diff(snapshotStamp, tagToUpdate.GetStatus().GetSnapshot().GetSandboxRuntime(), protocmp.Transform()); diff != "" {
t.Fatalf("tag snapshot stamp mismatch (-actor snapshot +tag):\n%s", diff)
}
+ assertSnapshotUnderLocation(t, tagSnapshotURI(tagToUpdate), wantSnapshotLocation(t, at))
listed, err := clients.SubstrateAPI.ListTags(ctx, &ateapipb.ListTagsRequest{Atespace: demoAtespace})
if err != nil {
t.Fatalf("failed to list Tags: %v", err)
@@ -400,7 +402,7 @@ func TestExternalVolumeLifecycle(t *testing.T) {
}
}
-func TestDeleteActorAnyStateWithExternalVolume(t *testing.T) {
+func TestDeleteRunningActorWithExternalVolume(t *testing.T) {
e2e.RequirePreview(t, preview.GateExternalVolumes)
t.Parallel()
ctx := context.Background()
@@ -412,7 +414,7 @@ func TestDeleteActorAnyStateWithExternalVolume(t *testing.T) {
t.Fatalf("failed to initialize ActorTemplate: %v", err)
}
- actorName := "anystate-delete-extvol-" + nsObj.Name
+ actorName := "delete-running-extvol-" + nsObj.Name
t.Logf("Creating Actor %q...", actorName)
if _, err := clients.SubstrateAPI.CreateActor(ctx, &ateapipb.CreateActorRequest{Actor: &ateapipb.Actor{
@@ -438,22 +440,21 @@ func TestDeleteActorAnyStateWithExternalVolume(t *testing.T) {
if err != nil {
t.Fatalf("failed to get actor: %v", err)
}
- if len(actor.GetStatus().GetExternalVolumes()) == 0 {
+ if len(actor.GetStatus().GetExternalVolumes().GetVolumes()) == 0 {
t.Fatalf("expected actor to have volumes, got 0")
}
- for _, vol := range actor.GetStatus().GetExternalVolumes() {
+ for _, vol := range actor.GetStatus().GetExternalVolumes().GetVolumes() {
if vol.StorageVolumeId == "" {
t.Fatalf("expected volume %q to be provisioned, got no storage volume ID", vol.Name)
}
}
- // Delete the running actor with any-state
- t.Logf("Deleting running Actor %q with any-state...", actorName)
+ // Delete the running actor
+ t.Logf("Deleting running Actor %q...", actorName)
if _, err := clients.SubstrateAPI.DeleteActor(ctx, &ateapipb.DeleteActorRequest{
- Actor: &ateapipb.ObjectRef{Atespace: demoAtespace, Name: actorName},
- AnyState: true,
+ Actor: &ateapipb.ObjectRef{Atespace: demoAtespace, Name: actorName},
}); err != nil {
- t.Fatalf("failed to delete Actor with any-state: %v", err)
+ t.Fatalf("failed to delete Actor: %v", err)
}
// Verify deletion in store
@@ -548,8 +549,7 @@ func TestExternalVolume_NodeMigration(t *testing.T) {
// Clean up by deleting the actor
t.Logf("Deleting Actor %q...", actorName)
if _, err := clients.SubstrateAPI.DeleteActor(ctx, &ateapipb.DeleteActorRequest{
- Actor: &ateapipb.ObjectRef{Atespace: demoAtespace, Name: actorName},
- AnyState: true,
+ Actor: &ateapipb.ObjectRef{Atespace: demoAtespace, Name: actorName},
}); err != nil {
t.Fatalf("failed to delete Actor: %v", err)
}
@@ -616,8 +616,7 @@ func runActorLifecycleTestCase(t *testing.T, prefix string, createTemplate func(
t.Logf("Successfully created Actor: %s", createResp.GetMetadata().GetName())
defer func() {
clients.SubstrateAPI.DeleteActor(ctx, &ateapipb.DeleteActorRequest{
- Actor: &ateapipb.ObjectRef{Atespace: demoAtespace, Name: actorID},
- AnyState: true,
+ Actor: &ateapipb.ObjectRef{Atespace: demoAtespace, Name: actorID},
})
}()
@@ -817,6 +816,41 @@ func runtimeAttribute(rt *ateapipb.SandboxRuntime, key string) string {
return ""
}
+// wantSnapshotLocation is where ate-api-server stores tmpl's snapshots: the
+// template's storageLocation with the PerTemplateSnapshotLocation preview gate,
+// otherwise $SNAPSHOT_LOCATION, the --snapshot-location ate-setup installs it
+// with.
+func wantSnapshotLocation(t *testing.T, tmpl *ateapipb.ActorTemplate) string {
+ t.Helper()
+ if location := tmpl.GetSnapshotConfig().GetStorageLocation(); location != "" && preview.IsEnabled(preview.GatePerTemplateSnapshotLocation) {
+ return location
+ }
+ env, err := e2e.CheckEnv("SNAPSHOT_LOCATION")
+ if err != nil {
+ t.Fatalf("CheckEnv failed: %v", err)
+ }
+ return env["SNAPSHOT_LOCATION"]
+}
+
+// assertSnapshotUnderLocation fails t unless snapshotURI is stored under
+// location. It compares owner prefixes, so location may be spelled with or
+// without a trailing slash.
+func assertSnapshotUnderLocation(t *testing.T, snapshotURI, location string) {
+ t.Helper()
+ uri, err := resources.ParseSnapshotURI(snapshotURI)
+ if err != nil {
+ t.Fatalf("ParseSnapshotURI(%q): %v", snapshotURI, err)
+ }
+ want, err := uri.Owner().Prefix(location)
+ if err != nil {
+ t.Fatalf("snapshot owner prefix under %q: %v", location, err)
+ }
+ if got := uri.OwnerPrefix(); got != want {
+ t.Errorf("snapshot %q is stored under %q, want %q (PerTemplateSnapshotLocation preview gate enabled: %t)",
+ snapshotURI, got, want, preview.IsEnabled(preview.GatePerTemplateSnapshotLocation))
+ }
+}
+
func tagSnapshotURI(tag *ateapipb.Tag) string {
return tag.GetStatus().GetSnapshot().GetDurableSnapshot().GetObject().GetSnapshotUri()
}
@@ -1064,7 +1098,7 @@ func revertActor(ctx context.Context, t *testing.T, clients *e2e.Clients, nsObj
t.Cleanup(func() {
cleanupCtx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
defer cancel()
- _, _ = clients.SubstrateAPI.DeleteActor(cleanupCtx, &ateapipb.DeleteActorRequest{Actor: actorRef, AnyState: true})
+ _, _ = clients.SubstrateAPI.DeleteActor(cleanupCtx, &ateapipb.DeleteActorRequest{Actor: actorRef})
})
if _, err := clients.SubstrateAPI.CreateActor(ctx, &ateapipb.CreateActorRequest{Actor: &ateapipb.Actor{
@@ -1175,16 +1209,16 @@ func revertActor(ctx context.Context, t *testing.T, clients *e2e.Clients, nsObj
}
validateCounterResponse(t, resp, "after revert from paused", 2, 2)
- if _, err := clients.SubstrateAPI.DeleteActor(ctx, &ateapipb.DeleteActorRequest{Actor: actorRef, AnyState: true}); err != nil {
+ if _, err := clients.SubstrateAPI.DeleteActor(ctx, &ateapipb.DeleteActorRequest{Actor: actorRef}); err != nil {
t.Fatalf("failed to delete Actor: %v", err)
}
return nil
}
-func deleteActorAnyState(ctx context.Context, t *testing.T, clients *e2e.Clients, nsObj *e2e.Namespace, at *ateapipb.ActorTemplate) error {
- actorName := "anystate-delete-actor-" + nsObj.Name
+func deleteRunningActor(ctx context.Context, t *testing.T, clients *e2e.Clients, nsObj *e2e.Namespace, at *ateapipb.ActorTemplate) error {
+ actorName := "delete-running-actor-" + nsObj.Name
- // 1. Creating an actor
+ // 1. Creating and resuming the actor
t.Logf("Creating Actor %q...", actorName)
if _, err := clients.SubstrateAPI.CreateActor(ctx, &ateapipb.CreateActorRequest{Actor: &ateapipb.Actor{
Metadata: &ateapipb.ResourceMetadata{Atespace: demoAtespace, Name: actorName},
@@ -1194,25 +1228,6 @@ func deleteActorAnyState(ctx context.Context, t *testing.T, clients *e2e.Clients
}
waitForActorState(ctx, t, clients, actorName, ateapipb.ActorState_ACTOR_STATE_SUSPENDED)
- // Verify that any-state delete on a SUSPENDED actor succeeds
- t.Logf("Attempting any-state delete on suspended Actor %q (should succeed)...", actorName)
- if _, err := clients.SubstrateAPI.DeleteActor(ctx, &ateapipb.DeleteActorRequest{
- Actor: &ateapipb.ObjectRef{Atespace: demoAtespace, Name: actorName},
- AnyState: true,
- }); err != nil {
- t.Fatalf("expected any-state delete on suspended actor to succeed, got %v", err)
- }
-
- // 2. Re-creating and Resuming the actor
- t.Logf("Re-creating Actor %q...", actorName)
- if _, err := clients.SubstrateAPI.CreateActor(ctx, &ateapipb.CreateActorRequest{Actor: &ateapipb.Actor{
- Metadata: &ateapipb.ResourceMetadata{Atespace: demoAtespace, Name: actorName},
- ActorTemplate: e2e.TemplateRef(at),
- }}); err != nil {
- t.Fatalf("failed to create Actor: %v", err)
- }
- waitForActorState(ctx, t, clients, actorName, ateapipb.ActorState_ACTOR_STATE_SUSPENDED)
-
t.Logf("Resuming Actor %q...", actorName)
if _, err := e2e.ResumeActorAwaitCapacity(t, ctx, clients, &ateapipb.ResumeActorRequest{
Actor: &ateapipb.ObjectRef{Atespace: demoAtespace, Name: actorName},
@@ -1221,39 +1236,29 @@ func deleteActorAnyState(ctx context.Context, t *testing.T, clients *e2e.Clients
}
waitForActorState(ctx, t, clients, actorName, ateapipb.ActorState_ACTOR_STATE_RUNNING)
- // 3. Call the actor to ensure workload is actively serving on worker
+ // 2. Call the actor to ensure workload is actively serving on worker
resp, err := callActor(t, resources.ActorRef{Atespace: demoAtespace, Name: actorName})
if err != nil {
t.Fatalf("failed to call actor: %v", err)
}
validateCounterResponse(t, resp, "after creation", 1, 1)
- // 4. Attempt standard DeleteActor without any-state on running actor (should fail with FailedPrecondition)
- t.Logf("Attempting standard delete on running Actor %q (should fail)...", actorName)
- if _, err := clients.SubstrateAPI.DeleteActor(ctx, &ateapipb.DeleteActorRequest{
- Actor: &ateapipb.ObjectRef{Atespace: demoAtespace, Name: actorName},
- AnyState: false,
- }); status.Code(err) != codes.FailedPrecondition {
- t.Fatalf("expected FailedPrecondition when standard deleting running actor, got %v", err)
- }
-
- // 5. Delete the running actor with any-state
- t.Logf("Deleting running Actor %q with any-state...", actorName)
+ // 3. Delete the running actor
+ t.Logf("Deleting running Actor %q...", actorName)
if _, err := clients.SubstrateAPI.DeleteActor(ctx, &ateapipb.DeleteActorRequest{
- Actor: &ateapipb.ObjectRef{Atespace: demoAtespace, Name: actorName},
- AnyState: true,
+ Actor: &ateapipb.ObjectRef{Atespace: demoAtespace, Name: actorName},
}); err != nil {
- t.Fatalf("failed to delete Actor with any-state: %v", err)
+ t.Fatalf("failed to delete Actor: %v", err)
}
- // 6. Verify deletion in store
+ // 4. Verify deletion in store
if _, err := clients.SubstrateAPI.GetActor(ctx, &ateapipb.GetActorRequest{
Actor: &ateapipb.ObjectRef{Atespace: demoAtespace, Name: actorName},
}); status.Code(err) != codes.NotFound {
t.Fatalf("expected actor %q to be NotFound after delete, got err: %v", actorName, err)
}
- // 7. Verify actor name can be immediately reused
+ // 5. Verify actor name can be immediately reused
t.Logf("Re-creating Actor %q to verify name reuse...", actorName)
if _, err := clients.SubstrateAPI.CreateActor(ctx, &ateapipb.CreateActorRequest{Actor: &ateapipb.Actor{
Metadata: &ateapipb.ResourceMetadata{Atespace: demoAtespace, Name: actorName},
@@ -1271,8 +1276,8 @@ func deleteActorAnyState(ctx context.Context, t *testing.T, clients *e2e.Clients
return nil
}
-func deletePausedActorAnyState(ctx context.Context, t *testing.T, clients *e2e.Clients, nsObj *e2e.Namespace, at *ateapipb.ActorTemplate) error {
- actorName := "anystate-delete-paused-actor-" + nsObj.Name
+func deletePausedActor(ctx context.Context, t *testing.T, clients *e2e.Clients, nsObj *e2e.Namespace, at *ateapipb.ActorTemplate) error {
+ actorName := "delete-paused-actor-" + nsObj.Name
// 1. Creating an actor
t.Logf("Creating Actor %q...", actorName)
@@ -1303,13 +1308,12 @@ func deletePausedActorAnyState(ctx context.Context, t *testing.T, clients *e2e.C
}
waitForActorPaused(ctx, t, clients, actorName)
- // 4. Delete the paused actor with any-state
- t.Logf("Deleting paused Actor %q with any-state...", actorName)
+ // 4. Delete the paused actor
+ t.Logf("Deleting paused Actor %q...", actorName)
if _, err := clients.SubstrateAPI.DeleteActor(ctx, &ateapipb.DeleteActorRequest{
- Actor: &ateapipb.ObjectRef{Atespace: demoAtespace, Name: actorName},
- AnyState: true,
+ Actor: &ateapipb.ObjectRef{Atespace: demoAtespace, Name: actorName},
}); err != nil {
- t.Fatalf("failed to delete Actor with any-state: %v", err)
+ t.Fatalf("failed to delete Actor: %v", err)
}
// 5. Verify deletion in store
@@ -1389,10 +1393,10 @@ func createActorTemplateWithExternalVolume(ctx context.Context, t *testing.T, cl
if !hasExtVol {
at.Volumes = append(at.Volumes, &ateapipb.Volume{
Name: "external-data",
- ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{
+ External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{
Capacity: "1Gi",
StorageClassName: scName,
- },
+ }},
})
}
}
@@ -1773,6 +1777,83 @@ func TestWorkerPodDeletion(t *testing.T) {
t.Errorf("worker for pod %s/%s was not cleaned up within 30s. Current workers: %v", podNamespace, podName, lastWorkers)
}
+// A sandbox that dies on its own is reported by the worker: the actor goes
+// CRASHED without any operation on it failing, its worker is freed, and it can
+// be reverted and resumed.
+func TestSandboxCrashIsReported(t *testing.T) {
+ t.Parallel()
+ nsObj := e2e.CreateNamespace(t)
+ ctx := context.Background()
+ clients := e2e.GetClients()
+
+ at, err := createActorTemplate(ctx, t, clients, nsObj, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY)
+ if err != nil {
+ t.Fatalf("failed to initialize ActorTemplate: %v", err)
+ }
+
+ actorName := "sandbox-crash-actor-" + nsObj.Name
+ actorRef := &ateapipb.ObjectRef{Atespace: demoAtespace, Name: actorName}
+ t.Cleanup(func() {
+ cleanupCtx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
+ defer cancel()
+ _, _ = clients.SubstrateAPI.DeleteActor(cleanupCtx, &ateapipb.DeleteActorRequest{Actor: actorRef})
+ })
+
+ if _, err := clients.SubstrateAPI.CreateActor(ctx, &ateapipb.CreateActorRequest{Actor: &ateapipb.Actor{
+ Metadata: &ateapipb.ResourceMetadata{Atespace: demoAtespace, Name: actorName},
+ ActorTemplate: e2e.TemplateRef(at),
+ }}); err != nil {
+ t.Fatalf("failed to create Actor: %v", err)
+ }
+ waitForActorState(ctx, t, clients, actorName, ateapipb.ActorState_ACTOR_STATE_SUSPENDED)
+ if _, err := e2e.ResumeActorAwaitCapacity(t, ctx, clients, &ateapipb.ResumeActorRequest{Actor: actorRef}); err != nil {
+ t.Fatalf("failed to resume Actor: %v", err)
+ }
+ waitForActorState(ctx, t, clients, actorName, ateapipb.ActorState_ACTOR_STATE_RUNNING)
+
+ actor, err := clients.SubstrateAPI.GetActor(ctx, &ateapipb.GetActorRequest{Actor: actorRef})
+ if err != nil {
+ t.Fatalf("failed to get Actor: %v", err)
+ }
+ assignment := actor.GetStatus().GetWorkerAssignment()
+ if assignment.GetWorkerPod() == "" || assignment.GetWorkerNamespace() == "" {
+ t.Fatalf("running actor has no pod recorded: %v", assignment)
+ }
+
+ e2e.KillActorSandbox(ctx, t, clients, assignment.GetWorkerNamespace(), assignment.GetWorkerPod(), actor.GetMetadata().GetUid())
+ waitForActorStateWithTimeout(ctx, t, clients, actorName, ateapipb.ActorState_ACTOR_STATE_CRASHED, 2*time.Minute)
+
+ crashed, err := clients.SubstrateAPI.GetActor(ctx, &ateapipb.GetActorRequest{Actor: actorRef})
+ if err != nil {
+ t.Fatalf("failed to get crashed Actor: %v", err)
+ }
+ if msg := crashed.GetStatus().GetCrash().GetMessage(); !strings.Contains(msg, "exited") {
+ t.Errorf("crash message = %q, want the worker's report of the sandbox exiting", msg)
+ }
+ if crashed.GetStatus().GetWorkerAssignment() != nil {
+ t.Errorf("worker assignment survived the crash: %v", crashed.GetStatus().GetWorkerAssignment())
+ }
+ // the worker pod survives, unlike TestRevertCrashedActor
+ if _, err := clients.K8s.CoreV1().Pods(assignment.GetWorkerNamespace()).Get(ctx, assignment.GetWorkerPod(), metav1.GetOptions{}); err != nil {
+ t.Errorf("worker pod %s/%s is gone: %v", assignment.GetWorkerNamespace(), assignment.GetWorkerPod(), err)
+ }
+
+ reverted, err := clients.SubstrateAPI.RevertActor(ctx, &ateapipb.RevertActorRequest{Actor: actorRef})
+ if err != nil {
+ t.Fatalf("RevertActor from CRASHED: %v", err)
+ }
+ if got := reverted.GetActor().GetStatus().GetState(); got != ateapipb.ActorState_ACTOR_STATE_SUSPENDED {
+ t.Fatalf("state after revert = %v, want SUSPENDED", got)
+ }
+ if _, err := e2e.ResumeActorAwaitCapacity(t, ctx, clients, &ateapipb.ResumeActorRequest{Actor: actorRef}); err != nil {
+ t.Fatalf("failed to resume the reverted Actor: %v", err)
+ }
+ waitForActorState(ctx, t, clients, actorName, ateapipb.ActorState_ACTOR_STATE_RUNNING)
+ if _, err := callActor(t, resources.ActorRef{Atespace: demoAtespace, Name: actorName}); err != nil {
+ t.Fatalf("failed to call the reverted Actor: %v", err)
+ }
+}
+
func TestRevertCrashedActor(t *testing.T) {
t.Parallel()
nsObj := e2e.CreateNamespace(t)
@@ -1789,7 +1870,7 @@ func TestRevertCrashedActor(t *testing.T) {
t.Cleanup(func() {
cleanupCtx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
defer cancel()
- _, _ = clients.SubstrateAPI.DeleteActor(cleanupCtx, &ateapipb.DeleteActorRequest{Actor: actorRef, AnyState: true})
+ _, _ = clients.SubstrateAPI.DeleteActor(cleanupCtx, &ateapipb.DeleteActorRequest{Actor: actorRef})
})
if _, err := clients.SubstrateAPI.CreateActor(ctx, &ateapipb.CreateActorRequest{Actor: &ateapipb.Actor{
diff --git a/internal/e2e/suites/demo/golden_test.go b/internal/e2e/suites/demo/golden_test.go
index 23f1de0796..b11e83cd95 100644
--- a/internal/e2e/suites/demo/golden_test.go
+++ b/internal/e2e/suites/demo/golden_test.go
@@ -90,7 +90,7 @@ func TestGoldenSnapshotLifecycle(t *testing.T) {
t.Cleanup(func() {
cleanupCtx, cancel := context.WithTimeout(context.Background(), time.Minute)
defer cancel()
- _, _ = clients.SubstrateAPI.DeleteActor(cleanupCtx, &ateapipb.DeleteActorRequest{Actor: actorRef, AnyState: true})
+ _, _ = clients.SubstrateAPI.DeleteActor(cleanupCtx, &ateapipb.DeleteActorRequest{Actor: actorRef})
})
actor, err := clients.SubstrateAPI.CreateActor(ctx, &ateapipb.CreateActorRequest{Actor: &ateapipb.Actor{
Metadata: &ateapipb.ResourceMetadata{Atespace: atespace, Name: actorRef.Name}, ActorTemplate: templateRef,
@@ -110,7 +110,7 @@ func TestGoldenSnapshotLifecycle(t *testing.T) {
}
validateCounterResponse(t, response, "golden restore", 1, 1)
// Delete the borrower before collecting the template's snapshot.
- if _, err := clients.SubstrateAPI.DeleteActor(ctx, &ateapipb.DeleteActorRequest{Actor: actorRef, AnyState: true}); err != nil {
+ if _, err := clients.SubstrateAPI.DeleteActor(ctx, &ateapipb.DeleteActorRequest{Actor: actorRef}); err != nil {
t.Fatal(err)
}
if _, err := clients.SubstrateAPI.DeleteActorTemplate(ctx, &ateapipb.DeleteActorTemplateRequest{ActorTemplate: templateRef}); err != nil {
diff --git a/internal/e2e/suites/egresscredinject/egresscredinject_test.go b/internal/e2e/suites/egresscredinject/egresscredinject_test.go
index 5d92f13c6d..92a237c7f3 100644
--- a/internal/e2e/suites/egresscredinject/egresscredinject_test.go
+++ b/internal/e2e/suites/egresscredinject/egresscredinject_test.go
@@ -16,12 +16,14 @@
// EgressPolicy https rule with a replace_headers effect makes the egress
// gateway's MITM leg resolve the credential through the
// k8s-credential-provider and replace the actor's placeholder header with it
-// before re-originating upstream. See TestActorEgressCredentialInjection for
-// the proof structure and how to run this locally.
+// before re-originating upstream. The same rule also injects an actor JWT
+// minted by ateapi. See TestActorEgressCredentialInjection for the proof
+// structure and how to run this locally.
package egresscredinject
import (
"context"
+ "encoding/base64"
"encoding/json"
"io"
"net/http"
@@ -31,6 +33,7 @@ import (
"testing"
"time"
+ "go.ate.dev/substrate/internal/actoridjwt"
"go.ate.dev/substrate/internal/e2e"
"go.ate.dev/substrate/internal/resources"
"go.ate.dev/substrate/pkg/proto/ateapipb"
@@ -59,12 +62,23 @@ const (
unauthorizedHost = "unauthorized.egresscredinject.invalid"
)
-// placeholder is the Authorization value the actor sends. replace_headers
-// replaces a header only when the request carries it, so every fetch that
-// should trigger injection sends it.
+// placeholder is the value the actor sends in each injected header.
+// replace_headers replaces a header only when the request carries it, so every
+// fetch that should trigger injection sends it.
const placeholder = "Bearer actor-placeholder"
-var withPlaceholder = []string{"header=" + url.QueryEscape("Authorization:"+placeholder)}
+// The echo rules also replace actorJWTHeader with an actor JWT for
+// actorJWTAudience.
+const (
+ actorJWTHeader = "X-Actor-Token"
+ actorJWTAudience = "https://egresscredinject.invalid"
+ actorJWTLifetime = 600
+)
+
+var withPlaceholder = []string{
+ "header=" + url.QueryEscape("Authorization:"+placeholder),
+ "header=" + url.QueryEscape(actorJWTHeader+":"+placeholder),
+}
// TestActorEgressCredentialInjection proves the injected credential reaches
// the upstream, and that every way injection can go wrong lands on the
@@ -74,10 +88,11 @@ var withPlaceholder = []string{"header=" + url.QueryEscape("Authorization:"+plac
// injected "Authorization: Bearer " among the headers the origin
// received — the on-the-wire proof, not an inference from a status code —
// and not the placeholder, so an actor cannot choose the value that
-// leaves.
-// - no header: the same fetch without the placeholder echoes no
-// Authorization at all — the gateway replaces only a header the request
-// carries, and does not add one.
+// leaves. The same fetch echoes an actor JWT naming this actor in
+// actorJWTHeader.
+// - no header: the same fetch without the placeholders echoes neither
+// header — the gateway replaces only a header the request carries, and
+// does not add one.
// - cleartext: the same fetch over plain HTTP, allowed by an http rule
// with the same effect, echoes the injected credential too.
// - fail closed: a credential the policy requires but the provider will
@@ -114,6 +129,12 @@ func TestActorEgressCredentialInjection(t *testing.T) {
const id = "probe-credinject"
createAndResumeActor(t, ctx, clients, id, echoHost)
waitForActorState(t, ctx, clients, id, ateapipb.ActorState_ACTOR_STATE_RUNNING)
+ actor, err := clients.SubstrateAPI.GetActor(ctx, &ateapipb.GetActorRequest{
+ Actor: &ateapipb.ObjectRef{Atespace: probeNamespace, Name: id},
+ })
+ if err != nil {
+ t.Fatalf("GetActor %q: %v", id, err)
+ }
rc, err := e2e.NewRouterClient(ctx)
if err != nil {
@@ -138,15 +159,18 @@ func TestActorEgressCredentialInjection(t *testing.T) {
// The upstream must receive the policy's credential instead of the
// actor's placeholder.
wantHeader := "Bearer " + e2e.CredentialInjectionToken
- replaced := fetchEcho(t, ctx, rc, id, origin, withPlaceholder)
- if got := assertEchoedAuthorization(t, "injection fetch", replaced); got != wantHeader {
+ replaced := assertEchoedHeaders(t, "injection fetch", fetchEcho(t, ctx, rc, id, origin, withPlaceholder))
+ if got := replaced["Authorization"]; got != wantHeader {
t.Errorf("upstream received Authorization %q, want the injected %q", got, wantHeader)
}
+ checkActorJWT(t, replaced[actorJWTHeader], actor.GetMetadata())
// No header is added when the actor did not send one.
- unasked := fetchEcho(t, ctx, rc, id, origin, nil)
- if got := assertEchoedAuthorization(t, "fetch without the header", unasked); got != "" {
- t.Errorf("upstream received Authorization %q on a request that did not carry it, want none", got)
+ unasked := assertEchoedHeaders(t, "fetch without the headers", fetchEcho(t, ctx, rc, id, origin, nil))
+ for _, h := range []string{"Authorization", actorJWTHeader} {
+ if got, ok := unasked[h]; ok {
+ t.Errorf("upstream received %s %q on a request that did not carry it, want none", h, got)
+ }
}
})
}
@@ -235,9 +259,9 @@ func decodeEchoedHeaders(t *testing.T, step, body string) map[string]string {
return echoed.Headers
}
-// assertEchoedAuthorization fails on any transport- or HTTP-level failure of
-// an echo fetch and returns the Authorization value the upstream received.
-func assertEchoedAuthorization(t *testing.T, step string, resp fetchResponse) string {
+// assertEchoedHeaders fails on any transport- or HTTP-level failure of an echo
+// fetch and returns the headers the upstream received.
+func assertEchoedHeaders(t *testing.T, step string, resp fetchResponse) map[string]string {
t.Helper()
if resp.Error != "" {
t.Fatalf("%s: TLS through the MITM egress gateway failed: %s", step, resp.Error)
@@ -245,7 +269,46 @@ func assertEchoedAuthorization(t *testing.T, step string, resp fetchResponse) st
if resp.Status != "200" {
t.Fatalf("%s: status %s, want 200 (an injection failure would deny with 403/500/503; is the provider deployed, the gateway installed with --credential-provider='{\"name\":\"k8s.io\"}', and hack/setup-e2e-egress-tls-kind.sh run so the gateway trusts the echo origin's certificate?) body %q", step, resp.Status, resp.Body)
}
- return decodeEchoedHeaders(t, step, resp.Body)["Authorization"]
+ return decodeEchoedHeaders(t, step, resp.Body)
+}
+
+// checkActorJWT checks the claims of an injected actor JWT against the actor
+// and the echo rules' actor_jwt source. It does not verify the signature.
+func checkActorJWT(t *testing.T, jwt string, actor *ateapipb.ResourceMetadata) {
+ t.Helper()
+ parts := strings.Split(jwt, ".")
+ if len(parts) != 3 {
+ t.Errorf("upstream received %s %q, want an actor JWT", actorJWTHeader, jwt)
+ return
+ }
+ payload, err := base64.RawURLEncoding.DecodeString(parts[1])
+ if err != nil {
+ t.Errorf("decoding the actor JWT payload: %v", err)
+ return
+ }
+ var claims actoridjwt.WireClaims
+ if err := json.Unmarshal(payload, &claims); err != nil {
+ t.Errorf("decoding the actor JWT claims: %v (payload %s)", err, payload)
+ return
+ }
+ var aud []string
+ if err := json.Unmarshal(claims.Audiences, &aud); err != nil {
+ t.Errorf("decoding the actor JWT aud claim: %v (payload %s)", err, payload)
+ }
+
+ if want := "actor/" + actor.GetAtespace() + "/" + actor.GetName(); claims.Subject != want {
+ t.Errorf("actor JWT sub is %q, want %q", claims.Subject, want)
+ }
+ if !slices.Equal(aud, []string{actorJWTAudience}) {
+ t.Errorf("actor JWT aud is %q, want [%q]", aud, actorJWTAudience)
+ }
+ if got := claims.Expiration - claims.IssuedAt; got != actorJWTLifetime {
+ t.Errorf("actor JWT lives %vs (exp - iat), want %ds", got, actorJWTLifetime)
+ }
+ want := actoridjwt.WireSubstrateClaims{Atespace: actor.GetAtespace(), ActorName: actor.GetName(), ActorUID: actor.GetUid()}
+ if kind, got, ok := claims.Substrate(); !ok || kind != actoridjwt.KindActor || *got != want {
+ t.Errorf("actor JWT payload is %s, want only an actor.ate.dev claim of %+v", payload, want)
+ }
}
type fetchResponse struct {
@@ -323,10 +386,11 @@ func createAndResumeActor(t *testing.T, ctx context.Context, clients *e2e.Client
})
// Exercise each credential outcome on both HTTP and HTTPS: the echo
// origin's credential resolves, and each fail-closed host's cannot.
- rules := []*ateapipb.EgressRule{
- e2e.EgressInjectHeader("Authorization", "Bearer ", e2e.CredentialInjectionURI, echoHost, echoHost+".cluster.local"),
- e2e.EgressInjectHeaderHTTP("Authorization", "Bearer ", e2e.CredentialInjectionURI, echoHost, echoHost+".cluster.local"),
- }
+ echoHTTPSRule := e2e.EgressInjectHeader("Authorization", "Bearer ", e2e.CredentialInjectionURI, echoHost, echoHost+".cluster.local")
+ echoHTTPSRule.Https.Effects.ReplaceHeaders = append(echoHTTPSRule.Https.Effects.ReplaceHeaders, actorJWTInjection())
+ echoHTTPRule := e2e.EgressInjectHeaderHTTP("Authorization", "Bearer ", e2e.CredentialInjectionURI, echoHost, echoHost+".cluster.local")
+ echoHTTPRule.Http.Effects.ReplaceHeaders = append(echoHTTPRule.Http.Effects.ReplaceHeaders, actorJWTInjection())
+ rules := []*ateapipb.EgressRule{echoHTTPSRule, echoHTTPRule}
for _, credential := range []struct {
host string
uri string
@@ -347,6 +411,16 @@ func createAndResumeActor(t *testing.T, ctx context.Context, clients *e2e.Client
}
}
+func actorJWTInjection() *ateapipb.CredentialHeader {
+ return &ateapipb.CredentialHeader{
+ Header: actorJWTHeader,
+ ActorJwt: &ateapipb.ActorJwtSource{
+ Audiences: []string{actorJWTAudience},
+ ExpirationSeconds: actorJWTLifetime,
+ },
+ }
+}
+
func waitForActorState(t *testing.T, ctx context.Context, clients *e2e.Clients, actorName string, want ateapipb.ActorState) {
t.Helper()
deadline := time.Now().Add(60 * time.Second)
diff --git a/internal/imagecache/README.md b/internal/imagecache/README.md
index e366534174..7b9b5861e0 100644
--- a/internal/imagecache/README.md
+++ b/internal/imagecache/README.md
@@ -214,9 +214,10 @@ percentage as `df` reports it.
(after a freshness re-check under the same lock the cache-hit path
holds), then retire each layer the removal left unreferenced. If any
layer must be kept — still referenced by another record, rooted by a
- spec, younger than min-age, or its retirement failed — the record is
- restored byte-exact and the image simply is not evicted this pass: a
- layer is never left on disk without a record explaining it.
+ spec, pinned by an in-flight pull, younger than min-age, or its
+ retirement failed — the record is restored byte-exact and the image
+ simply is not evicted this pass: a layer is never left on disk without
+ a record explaining it.
The pass reaches layers **only through records**: a layer is deleted
exactly when its last referencing record goes, never by an independent
@@ -229,14 +230,16 @@ refcounts and roots computed from partial data would retire layers that
unread records still reference or running actors still mount. Dry-run
mutates nothing at all — not even the lazy size-file backfill.
-**In-flight pulls need no separate protection.** `EnsureImage` writes
-the image record **before** unpacking (the way Go allocates black during
-GC and containerd creates its ingest record before the bytes land), so
-every layer a pull produces is referenced — and kept fresh by a
-per-layer progress touch — from before it exists on disk. An interrupted
-pull's record is resumable progress, not garbage: the next pull of that
-digest re-fetches only the missing layers, and a pull that never resumes
-ages out through ordinary LRU.
+**In-flight pulls.** `EnsureImage` writes the image record **before**
+unpacking (the way Go allocates black during GC and containerd creates
+its ingest record before the bytes land), so every layer a pull produces
+is referenced — and kept fresh by a per-layer progress touch — from
+before it exists on disk. The pull also pins each of its layers until it
+returns, and eviction never retires a pinned layer, so a layer the pull
+has unpacked or reused cannot be taken from under it even if its record
+ages out meanwhile. An interrupted pull's record is resumable progress,
+not garbage: the next pull of that digest re-fetches only the missing
+layers, and a pull that never resumes ages out through ordinary LRU.
**Startup recovery.** A layer no record references can only be crash
debris (eviction interrupted between record-delete and layer-rename) or
@@ -247,12 +250,13 @@ fails to read. There is no online whole-pool scan (ext4's split: bounded
recovery at mount, fsck offline).
**Deletion is two-phase.** A layer is atomically renamed to `.rm-*`
-inside the layer's singleflight (one `rename(2)` — eviction can never
-stall a pull), then removed asynchronously; a crash in between leaves
-the dir for the startup sweep. This matters because the kernel offers no
-protection here: deleting a directory that is a live overlay lowerdir in
-another mount namespace succeeds silently, leaves the overlay's behavior
-undefined, and doesn't even free the space until the mount goes away.
+under the pin lock, and never while a pull has it pinned (a pull waits
+on eviction for at most one stat and one `rename(2)`), then removed
+asynchronously; a crash in between leaves the dir for the startup sweep.
+This matters because the kernel offers no protection here: deleting a
+directory that is a live overlay lowerdir in another mount namespace
+succeeds silently, leaves the overlay's behavior undefined, and doesn't
+even free the space until the mount goes away.
Deleting the cache root by hand (while no actors are starting) remains
safe — the store re-pulls whatever is missing.
diff --git a/internal/imagecache/gc.go b/internal/imagecache/gc.go
index b17e02f643..313bd23c6a 100644
--- a/internal/imagecache/gc.go
+++ b/internal/imagecache/gc.go
@@ -38,7 +38,7 @@ package imagecache
//
// Deletion is two-phase: the only steps that contend with the pull path
// are one os.Remove of a record and one rename of a layer dir to a ".rm-*"
-// name inside the layer's singleflight (see retireLayer); the slow
+// name under the pin lock (see retireLayer); the slow
// RemoveAll of multi-GB trees happens afterwards, on dirs nothing can
// reach by diffID. A crash in between leaves a ".rm-*" dir for the
// startup sweep.
@@ -590,14 +590,15 @@ func (s *Store) sweepOrphanLayers(ctx context.Context, roots RootSet, refcount m
// Sizing is read-only: even a size-file backfill would break dry-run's
// mutate-nothing contract.
func (s *Store) dryRunRetire(hex string, cutoff time.Time) (int64, retireStatus) {
- dir := filepath.Join(s.layersDir(), hex)
- fi, err := os.Stat(dir)
- if err != nil {
- return 0, retireGone
- }
- if fi.ModTime().After(cutoff) {
+ // A pinned layer is not reclaimable, landed or not, so a dry run must
+ // not count it.
+ if s.pinned(hex) {
return 0, retireVetoed
}
+ dir := filepath.Join(s.layersDir(), hex)
+ if st, _, _ := retirable(dir, cutoff); st != retireRetired {
+ return 0, st
+ }
size, err := s.layerSizeReadOnly(dir)
if err != nil {
size = 0
diff --git a/internal/imagecache/imagecache.go b/internal/imagecache/imagecache.go
index edcaaff37b..9693cf1c8a 100644
--- a/internal/imagecache/imagecache.go
+++ b/internal/imagecache/imagecache.go
@@ -224,6 +224,13 @@ type Store struct {
// (removeStaleRecord), so a hit's last-use touch and eviction's final
// re-check can never interleave. Uncontended except during a pass.
hitMu sync.RWMutex
+
+ // pins counts the in-flight pulls using each layer, by diffID hex;
+ // retireLayer never retires a pinned layer. Held only across a pin, an
+ // unpin, or a retirement's stat and rename. Entries are deleted at
+ // zero, so the map holds only layers pulls are using right now.
+ pinMu sync.Mutex
+ pins map[string]int
}
// Option configures a Store.
@@ -305,7 +312,7 @@ type imageRecord struct {
// startup recovery: verifying the layout version and sweeping temp dirs left
// by unpacks that were in flight when a previous atelet died.
func New(root string, opts ...Option) (*Store, error) {
- s := &Store{root: root, minAge: defaultMinAge, pullTimeout: defaultPullTimeout}
+ s := &Store{root: root, minAge: defaultMinAge, pullTimeout: defaultPullTimeout, pins: map[string]int{}}
for _, o := range opts {
o(s)
}
@@ -547,8 +554,9 @@ func (s *Store) cachedImage(digest v1.Hash) (*Image, error) {
// so an interrupted pull's record is just resumable progress.
func (s *Store) pull(ctx context.Context, parsedRef name.Reference, digest v1.Hash) (*Image, error) {
// Re-check under the flight lock: a racing EnsureImage may have completed
- // the pull between our cache miss and winning the singleflight slot.
- if img, err := s.cachedImage(digest); err != nil {
+ // the pull between our cache miss and winning the singleflight slot. As
+ // a hit, so the record is touched and its layers are kept past min-age.
+ if img, err := s.cachedImageHit(digest); err != nil {
return nil, err
} else if img != nil {
return img, nil
@@ -587,9 +595,15 @@ func (s *Store) pullWith(ctx context.Context, parsedRef name.Reference, digest v
return nil, fmt.Errorf("image %s config lists %d diffIDs but manifest has %d layers", digest, len(cfgFile.RootFS.DiffIDs), len(layers))
}
diffIDs := make([]string, len(layers))
+ hexes := make([]string, len(layers))
for i, d := range cfgFile.RootFS.DiffIDs {
diffIDs[i] = d.String()
+ hexes[i] = d.Hex
}
+ // Pinned until this pull returns: eviction never retires a pinned
+ // layer, so nothing this pull has unpacked or reused can be taken out
+ // from under it, whatever happens to its record meanwhile.
+ defer s.pinLayers(hexes)()
// The full diffID list is known from the config before any unpack; make
// every layer of this pull referenced before it can exist.
@@ -628,10 +642,11 @@ func (s *Store) pullWith(ctx context.Context, parsedRef name.Reference, digest v
}
layerDirs[i] = dir
// Each completed layer refreshes the record's mtime: a pull
- // making progress stays fresh indefinitely; a wedged one ages
- // into ordinary LRU eviction. The twin is not touched — the
- // primary keeps the layers referenced, and the final rewrite
- // recreates the twin if it ages out mid-pull.
+ // making progress stays fresh indefinitely. A wedged one can
+ // become a candidate, but its pins veto retiring any layer only
+ // it references, and the veto restores the record. The twin is
+ // not touched — the final rewrite recreates it if it ages out
+ // mid-pull.
s.touchRecord(digest)
return nil
})
@@ -641,8 +656,8 @@ func (s *Store) pullWith(ctx context.Context, parsedRef name.Reference, digest v
}
// Rewrite the record: eviction may legitimately remove it mid-pull
- // (no progress for min-age reads as wedged), and success must never
- // leave the just-unpacked layers unreferenced.
+ // (no progress for min-age reads as wedged), and the pins that kept
+ // the layers meanwhile lapse when this pull returns.
if err := s.writeRecord(digest, rec); err != nil {
return nil, fmt.Errorf("while rewriting image record after unpack: %w", err)
}
@@ -671,33 +686,60 @@ func (s *Store) pullWith(ctx context.Context, parsedRef name.Reference, digest v
return &Image{Digest: digest, Config: cfgFile.Config, LayerDirs: layerDirs}, nil
}
+// pinLayers marks the layers as in use by an in-flight pull and returns the
+// matching unpin. Taken by each pull for all of its layers before any
+// ensureLayer call, outside the layer flight, so a pull that joins another
+// pull's flight is protected by its own pin once the leader's lapses.
+func (s *Store) pinLayers(hexes []string) (unpin func()) {
+ s.pinMu.Lock()
+ for _, hex := range hexes {
+ s.pins[hex]++
+ }
+ s.pinMu.Unlock()
+ return func() {
+ s.pinMu.Lock()
+ defer s.pinMu.Unlock()
+ for _, hex := range hexes {
+ if s.pins[hex]--; s.pins[hex] == 0 {
+ delete(s.pins, hex)
+ }
+ }
+ }
+}
+
+// pinned reports whether an in-flight pull holds the layer. A snapshot:
+// retirement itself re-checks under pinMu instead.
+func (s *Store) pinned(hex string) bool {
+ s.pinMu.Lock()
+ defer s.pinMu.Unlock()
+ return s.pins[hex] > 0
+}
+
// ensureLayer makes the unpacked tree for diffID present in the pool,
// collapsing concurrent requests for the same layer across images.
+//
+// The caller must hold a pin on the layer (see pinLayers) from before this
+// call until it no longer relies on the returned dir. The pin, not the
+// flight, is what keeps eviction from retiring the layer meanwhile.
func (s *Store) ensureLayer(ctx context.Context, diffID v1.Hash, layer v1.Layer) (string, error) {
dir := s.layerDir(diffID)
- for {
- result, err, _ := s.layerSF.Do(layerFlightKey(diffID.Hex), func() (any, error) {
- if _, err := os.Stat(filepath.Join(dir, layerFSDirName)); err == nil {
- // Refresh the dir mtime inside the flight: retireLayer re-checks
- // the mtime in this same flight, so a layer reused here can
- // never be renamed away between this stat and the image record
- // that will re-reference it.
- now := time.Now()
- if err := os.Chtimes(dir, now, now); err != nil {
- slog.WarnContext(ctx, "Failed to refresh layer mtime on reuse", slog.String("diffid", diffID.String()), slog.Any("err", err))
- }
- return dir, nil
+ _, err, _ := s.layerSF.Do(diffID.String(), func() (any, error) {
+ if _, err := os.Stat(filepath.Join(dir, layerFSDirName)); err == nil {
+ // Refresh the dir mtime so eviction's LRU order sees the reuse.
+ // The pull's pin, not this stamp, is what keeps the layer from
+ // being retired before the image record re-references it.
+ now := time.Now()
+ if err := os.Chtimes(dir, now, now); err != nil {
+ slog.WarnContext(ctx, "Failed to refresh layer mtime on reuse", slog.String("diffid", diffID.String()), slog.Any("err", err))
}
- return dir, s.unpackLayerToPool(ctx, diffID, layer)
- })
- if err != nil {
- return "", err
- }
- // A joined retirement returns nil, so the layer still needs a pull.
- if result != nil {
- return dir, nil
+ return nil, nil
}
+ return nil, s.unpackLayerToPool(ctx, diffID, layer)
+ })
+ if err != nil {
+ return "", err
}
+ return dir, nil
}
// unpackLayerToPool streams the layer (download → decompress → untar) into a
diff --git a/internal/imagecache/retire.go b/internal/imagecache/retire.go
index 5d909dc9e5..e630e59a6c 100644
--- a/internal/imagecache/retire.go
+++ b/internal/imagecache/retire.go
@@ -15,8 +15,8 @@
package imagecache
// Two-phase layer deletion: eviction renames a layer dir aside (one
-// rename(2) inside the layer singleflight — the only step that contends
-// with the pull path) and the slow RemoveAll of the renamed-aside tree
+// rename(2) under the pin lock — the only step that contends with the
+// pull path) and the slow RemoveAll of the renamed-aside tree
// happens afterwards, outside all locks. A crash in between leaves a
// ".rm-*" dir for the startup sweep. Nothing here needs privileges:
// retirement is rename/chmod/unlink, which plain root can do even on
@@ -40,26 +40,25 @@ const retiredPrefix = ".rm-"
// exact race being exercised.
const logMsgLayerRetireVetoed = "Image cache layer retirement vetoed: recently used"
+// logMsgLayerRetirePinned is its counterpart for a layer an in-flight pull
+// is using, which can hold a layer for up to the pull timeout.
+const logMsgLayerRetirePinned = "Image cache layer retirement vetoed: in use by a pull"
+
// retireStatus reports what retireLayer did with a layer.
type retireStatus int
const (
- // retireGone: no dir under the layer's final name — already removed, or
- // mid-unpack (only the commit rename creates the final name). Nothing
- // stranded.
+ // retireGone: no dir under the layer's final name and no pull pinning
+ // it — already removed. Nothing stranded.
retireGone retireStatus = iota
- // retireVetoed: the layer stays — fresh mtime, in-flight reuse, or a
- // failed rename.
+ // retireVetoed: the layer stays — pinned by an in-flight pull (landed
+ // or still unpacking), fresh mtime, or a failed rename.
retireVetoed
// retireRetired: renamed to a ".rm-*" name, gone from the pool; the
// caller removes the renamed dir afterwards.
retireRetired
)
-// layerFlightKey is the singleflight key shared by ensureLayer and
-// retireLayer; the retire/reuse interlock depends on both using it.
-func layerFlightKey(hex string) string { return "sha256:" + hex }
-
// isLayerDirName reports whether name is a well-formed sha256 layer
// directory name. Callers enumerate directories and read hexes out of
// records, so they can encounter anything an operator (or a corrupt
@@ -78,70 +77,67 @@ func isLayerDirName(name string) bool {
// retireLayer evicts a layer by renaming its dir to a ".rm-*" name and
// returns the renamed path; the caller deletes it afterwards. A layer
-// with an mtime after cutoff is vetoed and left in place.
+// pinned by an in-flight pull, or with an mtime after cutoff, is vetoed
+// and left in place.
//
-// The mtime check and the rename run inside the layer singleflight — the
-// same flight in which ensureLayer refreshes the mtime — so a retirement
-// and a reuse cannot interleave: whichever runs second sees the first.
+// A fresh layer is ruled out before taking pinMu, which every pull takes to
+// pin, and the logging happens outside it. A missing layer that a pull has
+// pinned is still unpacking, so it is vetoed to keep the record.
func (s *Store) retireLayer(hex string, cutoff time.Time) (string, retireStatus, error) {
if !isLayerDirName(hex) {
return "", retireVetoed, fmt.Errorf("not a layer dir name: %q", hex)
}
dir := filepath.Join(s.layersDir(), hex)
+ if st, mtime, err := retirable(dir, cutoff); st != retireRetired {
+ if st == retireGone && s.pinned(hex) {
+ return "", retireVetoed, nil
+ }
+ if st == retireVetoed && err == nil {
+ slog.Info(logMsgLayerRetireVetoed, slog.String("diffid", hex), slog.Time("last_used", mtime))
+ }
+ return "", st, err
+ }
+ dst, st, pins, err := s.renameAsideUnlessPinned(dir, hex, cutoff)
+ if pins > 0 {
+ slog.Info(logMsgLayerRetirePinned, slog.String("diffid", hex), slog.Int("pulls", pins))
+ }
+ return dst, st, err
+}
+
+// renameAsideUnlessPinned is retireLayer's critical section. Under pinMu
+// the pin check, a re-stat and the rename run as one unit: either a pin
+// lands first and the retirement is vetoed, or the rename lands first and
+// the pull finds the dir missing and unpacks it again. Every pull waits on
+// this lock, so it does only syscalls; a layer refreshed since the
+// caller's preflight is vetoed without a log.
+func (s *Store) renameAsideUnlessPinned(dir, hex string, cutoff time.Time) (dst string, st retireStatus, pins int, err error) {
+ s.pinMu.Lock()
+ defer s.pinMu.Unlock()
+ if pins = s.pins[hex]; pins > 0 {
+ return "", retireVetoed, pins, nil
+ }
+ if st, _, err := retirable(dir, cutoff); st != retireRetired {
+ return "", st, 0, err
+ }
+ dst = filepath.Join(s.layersDir(), fmt.Sprintf("%s%s-%d", retiredPrefix, hex[:12], time.Now().UnixNano()))
+ if err := os.Rename(dir, dst); err != nil {
+ return "", retireVetoed, 0, fmt.Errorf("while retiring layer %s: %w", hex, err)
+ }
+ return dst, retireRetired, 0, nil
+}
- // Pre-flight, outside the singleflight: a missing dir or a fresh mtime
- // means nothing to do, and no reason to enter the flight and block
- // behind an in-progress download. Both checks are re-run inside the
- // flight before the rename, so this is a fast path, not the
- // correctness path.
+// retirable reports retireRetired when dir exists and is older than cutoff,
+// retireGone when it is missing, and retireVetoed otherwise, with the dir's
+// mtime when it was read.
+func retirable(dir string, cutoff time.Time) (retireStatus, time.Time, error) {
fi, err := os.Stat(dir)
if errors.Is(err, os.ErrNotExist) {
- return "", retireGone, nil
+ return retireGone, time.Time{}, nil
} else if err != nil {
- return "", retireVetoed, err
+ return retireVetoed, time.Time{}, err
}
if fi.ModTime().After(cutoff) {
- slog.Info(logMsgLayerRetireVetoed, slog.String("diffid", hex), slog.Time("last_used", fi.ModTime()))
- return "", retireVetoed, nil
- }
-
- var retired string
- status := retireGone
- ran := false
- _, err, _ = s.layerSF.Do(layerFlightKey(hex), func() (any, error) {
- ran = true
- fi, err := os.Stat(dir)
- if errors.Is(err, os.ErrNotExist) {
- status = retireGone
- return nil, nil
- } else if err != nil {
- status = retireVetoed
- return nil, err
- }
- // A concurrent ensureLayer touched the dir if it reused this layer
- // since the pre-flight stat.
- if fi.ModTime().After(cutoff) {
- slog.Info(logMsgLayerRetireVetoed, slog.String("diffid", hex), slog.Time("last_used", fi.ModTime()))
- status = retireVetoed
- return nil, nil
- }
- dst := filepath.Join(s.layersDir(), fmt.Sprintf("%s%s-%d", retiredPrefix, hex[:12], time.Now().UnixNano()))
- if err := os.Rename(dir, dst); err != nil {
- status = retireVetoed
- return nil, fmt.Errorf("while retiring layer %s: %w", hex, err)
- }
- retired = dst
- status = retireRetired
- return nil, nil
- })
- if !ran {
- // Our closure never executed: Do joined a flight already in progress
- // (an ensureLayer reuse or another retire), so status/retired are
- // stale zero values. Concurrent activity on the layer is a veto.
- return "", retireVetoed, nil
- }
- if err != nil {
- return "", status, err
+ return retireVetoed, fi.ModTime(), nil
}
- return retired, status, nil
+ return retireRetired, fi.ModTime(), nil
}
diff --git a/internal/imagecache/retire_test.go b/internal/imagecache/retire_test.go
index 2abf3efa28..ef62fa4904 100644
--- a/internal/imagecache/retire_test.go
+++ b/internal/imagecache/retire_test.go
@@ -17,27 +17,17 @@ package imagecache
import (
"archive/tar"
"context"
+ "math"
"os"
"path/filepath"
"strings"
"sync"
"testing"
- "testing/synctest"
"time"
v1 "github.com/google/go-containerregistry/pkg/v1"
)
-// The retire/reuse interlock depends on retireLayer and ensureLayer using
-// the same singleflight key; pin the key format to diffID.String()'s.
-func TestLayerFlightKeyMatchesDiffIDString(t *testing.T) {
- hex := strings.Repeat("ab", 32)
- want := v1.Hash{Algorithm: "sha256", Hex: hex}.String()
- if got := layerFlightKey(hex); got != want {
- t.Errorf("layerFlightKey = %q, want %q", got, want)
- }
-}
-
func TestRetireLayerStatuses(t *testing.T) {
store := newTestStore(t)
hex := strings.Repeat("ab", 32)
@@ -50,6 +40,16 @@ func TestRetireLayerStatuses(t *testing.T) {
t.Errorf("retireLayer(absent) = %v, %v; want retireGone, nil", st, err)
}
+ // Absent but pinned: still unpacking, so vetoed to keep its record.
+ unpin := store.pinLayers([]string{hex})
+ if _, st, err := store.retireLayer(hex, time.Now()); err != nil || st != retireVetoed {
+ t.Errorf("retireLayer(absent, pinned) = %v, %v; want retireVetoed, nil", st, err)
+ }
+ if _, st := store.dryRunRetire(hex, time.Now()); st != retireVetoed {
+ t.Errorf("dryRunRetire(absent, pinned) = %v; want retireVetoed", st)
+ }
+ unpin()
+
dir := filepath.Join(store.layersDir(), hex)
if err := os.MkdirAll(filepath.Join(dir, layerFSDirName), 0o700); err != nil {
t.Fatal(err)
@@ -144,9 +144,9 @@ func TestSweepLeavesNonPrefixedEntries(t *testing.T) {
}
// TestRetireLayerVsEnsureImageRace races retirement against pulls of an
-// image using the same layer. The singleflight serializes the retire
-// rename against unpack, and the in-flight mtime touch turns concurrent
-// reuse into a veto — so neither side may ever error. Run with -race.
+// image using the same layer. A pull pins its layers, so a retirement
+// racing it is vetoed or finds nothing to take — neither side may ever
+// error. Run with -race.
func TestRetireLayerVsEnsureImageRace(t *testing.T) {
_, host := newTestRegistry(t)
ref := host + "/test/retire-race:latest"
@@ -203,39 +203,197 @@ func TestRetireLayerVsEnsureImageRace(t *testing.T) {
}
}
-func TestEnsureLayerJoinsRetirement(t *testing.T) {
- synctest.Test(t, func(t *testing.T) {
- store := newTestStore(t)
- layer := layerFromEntries(t, []tarEntry{
- {name: "f", typeflag: tar.TypeReg, mode: 0o644, body: "hi"},
- })
- diffID, err := layer.DiffID()
- if err != nil {
- t.Fatal(err)
- }
+// A pinned layer is never retired, however old; the veto lifts with the
+// last unpin.
+func TestRetireLayerVetoedWhilePinned(t *testing.T) {
+ store := newTestStore(t)
+ hex := strings.Repeat("ab", 32)
+ dir := filepath.Join(store.layersDir(), hex)
+ if err := os.MkdirAll(filepath.Join(dir, layerFSDirName), 0o700); err != nil {
+ t.Fatal(err)
+ }
+ backdate(t, dir, 3*time.Hour)
- // Hold a retirement flight while the pull joins it.
- release := make(chan struct{})
- go func() {
- _, _, _ = store.layerSF.Do(layerFlightKey(diffID.Hex), func() (any, error) {
- <-release
- return nil, nil
- })
- }()
- synctest.Wait()
- var dir string
- go func() {
- dir, err = store.ensureLayer(context.Background(), diffID, layer)
- }()
- synctest.Wait()
- close(release)
- synctest.Wait()
-
- if err != nil {
- t.Fatal(err)
+ unpinA, unpinB := store.pinLayers([]string{hex}), store.pinLayers([]string{hex})
+ unpinA()
+ if _, st, err := store.retireLayer(hex, time.Now()); err != nil || st != retireVetoed {
+ t.Fatalf("retireLayer(pinned) = %v, %v; want retireVetoed, nil", st, err)
+ }
+ if _, err := os.Stat(filepath.Join(dir, layerFSDirName)); err != nil {
+ t.Fatalf("vetoed layer was disturbed: %v", err)
+ }
+
+ unpinB()
+ if store.pinned(hex) {
+ t.Fatal("layer still pinned after its last unpin")
+ }
+ if _, st, err := store.retireLayer(hex, time.Now()); err != nil || st != retireRetired {
+ t.Fatalf("retireLayer(unpinned) = %v, %v; want retireRetired, nil", st, err)
+ }
+}
+
+// The layers a pull has already landed survive an eviction pass that runs
+// while the pull is still downloading the rest — even one that removes the
+// pull's record as wedged. Without pins the free layer here is unreferenced
+// the moment the record goes, and the pull fails its final re-verify with
+// "layer dir vanished during pull".
+func TestPullPinsLayersAgainstEviction(t *testing.T) {
+ reg := newGatedRegistry(t)
+ free, gated, _ := gatedTestLayers(t)
+ ref := reg.host + "/test/pinned:latest"
+ pushImage(t, ref, v1.Config{}, free, gated)
+ store := newTestStore(t)
+ release := reg.gate(t, gated)
+
+ done := make(chan error, 1)
+ var img *Image
+ go func() {
+ var err error
+ img, err = store.EnsureImage(context.Background(), ref)
+ done <- err
+ }()
+ freeDir := layerDirOf(t, store, free)
+ waitFor(t, "free layer to land", func() bool {
+ _, err := os.Stat(filepath.Join(freeDir, layerFSDirName))
+ return err == nil
+ })
+
+ // Age everything past min-age, so the pass removes the record as
+ // wedged and would retire the free layer as unreferenced. The pull
+ // touches the record once after the free layer lands, with nothing to
+ // observe the touch by, so a backdate can race it and leave the record
+ // fresh. A pass that skipped the record as fresh changed nothing, so
+ // repeat until both passes reach retirement.
+ var dry, stats EvictStats
+ waitFor(t, "the aged record to reach retirement", func() bool {
+ var err error
+ backdateStore(t, store, 3*time.Hour)
+ if dry, err = store.EvictUnused(context.Background(), math.MaxInt64, true); err != nil {
+ t.Fatalf("EvictUnused(dry run): %v", err)
}
- if _, err := os.Stat(filepath.Join(dir, layerFSDirName, "f")); err != nil {
- t.Fatalf("layer was not unpacked after retirement: %v", err)
+ if stats, err = store.EvictUnused(context.Background(), math.MaxInt64, false); err != nil {
+ t.Fatalf("EvictUnused: %v", err)
}
+ return dry.SkippedFresh == 0 && stats.SkippedFresh == 0
})
+ if dry.FreedBytes != 0 || dry.EvictedLayers != 0 {
+ t.Errorf("dry run counted the pinned layer as reclaimable: freed=%d layers=%d", dry.FreedBytes, dry.EvictedLayers)
+ }
+ if stats.Candidates != 1 {
+ t.Fatalf("candidates=%d; the record must reach retirement for this test to mean anything", stats.Candidates)
+ }
+ if _, err := os.Stat(filepath.Join(freeDir, layerFSDirName)); err != nil {
+ t.Fatalf("pinned layer was retired mid-pull: %v", err)
+ }
+ if stats.EvictedLayers != 0 {
+ t.Errorf("EvictedLayers = %d, want 0: only pinned layers were eligible", stats.EvictedLayers)
+ }
+
+ release()
+ if err := <-done; err != nil {
+ t.Fatalf("EnsureImage across a mid-pull eviction pass: %v", err)
+ }
+ for _, dir := range img.LayerDirs {
+ if _, err := os.Stat(filepath.Join(dir, layerFSDirName)); err != nil {
+ t.Errorf("returned layer dir unusable: %v", err)
+ }
+ }
+ freeDiff, err := free.DiffID()
+ if err != nil {
+ t.Fatal(err)
+ }
+ if store.pinned(freeDiff.Hex) {
+ t.Error("pin outlived its pull")
+ }
+}
+
+// A pull must pin before it enters the layer flight. A pull that joins
+// another's flight runs no closure, so a pin taken inside one would never
+// happen for it, and its layer would be unprotected the moment the leader
+// returned.
+func TestPullPinsBeforeJoiningLayerFlight(t *testing.T) {
+ _, host := newTestRegistry(t)
+ store := newTestStore(t)
+ layer := layerFromEntries(t, []tarEntry{
+ {name: "f", typeflag: tar.TypeReg, mode: 0o644, body: strings.Repeat("p", 512)},
+ })
+ ref := host + "/test/pin-before-join:latest"
+ pushImage(t, ref, v1.Config{}, layer)
+ diffID, err := layer.DiffID()
+ if err != nil {
+ t.Fatal(err)
+ }
+
+ // Lead the layer's flight ourselves, and do its real work only once
+ // released, so the pull below has to join and wait.
+ held, release := make(chan struct{}), make(chan struct{})
+ var once sync.Once
+ t.Cleanup(func() { once.Do(func() { close(release) }) })
+ go func() {
+ _, _, _ = store.layerSF.Do(diffID.String(), func() (any, error) {
+ close(held)
+ <-release
+ return nil, store.unpackLayerToPool(context.Background(), diffID, layer)
+ })
+ }()
+ <-held
+
+ done := make(chan error, 1)
+ go func() {
+ _, err := store.EnsureImage(context.Background(), ref)
+ done <- err
+ }()
+ waitFor(t, "the pull to pin its layer", func() bool { return store.pinned(diffID.Hex) })
+ select {
+ case err := <-done:
+ t.Fatalf("pull returned while its layer flight was still held: %v", err)
+ default:
+ }
+
+ once.Do(func() { close(release) })
+ if err := <-done; err != nil {
+ t.Fatalf("EnsureImage: %v", err)
+ }
+ if store.pinned(diffID.Hex) {
+ t.Error("pin outlived its pull")
+ }
+}
+
+// A fresh layer is ruled out before pinMu is taken, so eviction passes over
+// it without blocking pulls of unrelated images.
+func TestRetireLayerPreflightSkipsPinLock(t *testing.T) {
+ store := newTestStore(t)
+ fresh := layerFromEntries(t, []tarEntry{
+ {name: "f", typeflag: tar.TypeReg, mode: 0o644, body: "fresh"},
+ })
+ freshID, err := fresh.DiffID()
+ if err != nil {
+ t.Fatal(err)
+ }
+ if _, err := store.ensureLayer(context.Background(), freshID, fresh); err != nil {
+ t.Fatal(err)
+ }
+ cutoff := time.Now().Add(-time.Hour)
+
+ // Held for the whole check, so a retireLayer that takes pinMu blocks
+ // until the timeout below.
+ store.pinMu.Lock()
+ defer store.pinMu.Unlock()
+ type result struct {
+ st retireStatus
+ err error
+ }
+ results := make(chan result, 1)
+ go func() {
+ _, st, err := store.retireLayer(freshID.Hex, cutoff)
+ results <- result{st, err}
+ }()
+ select {
+ case got := <-results:
+ if got.st != retireVetoed || got.err != nil {
+ t.Errorf("retireLayer(fresh) = %v, %v; want retireVetoed", got.st, got.err)
+ }
+ case <-time.After(5 * time.Second):
+ t.Fatal("retireLayer took pinMu for a fresh layer")
+ }
}
diff --git a/internal/installdefaults/installdefaults.go b/internal/installdefaults/installdefaults.go
index fa8f7098d7..ed0aad3190 100644
--- a/internal/installdefaults/installdefaults.go
+++ b/internal/installdefaults/installdefaults.go
@@ -81,6 +81,12 @@ func ActorJWTIssuer(namespace string) string {
}).String()
}
+// APIServiceHost returns the in-cluster DNS name of ate-api-server's Service
+// in namespace, the name on its serving certificate.
+func APIServiceHost(namespace string) string {
+ return APIServiceName + "." + namespace + ".svc"
+}
+
// ClientPrincipal returns the principal ID ate-api-server sees for a bearer
// token minted from ClientServiceAccount in namespace: the Kubernetes
// ServiceAccount username in the token's subject.
diff --git a/internal/installdefaults/installdefaults_test.go b/internal/installdefaults/installdefaults_test.go
index 4c29998cff..2329431e20 100644
--- a/internal/installdefaults/installdefaults_test.go
+++ b/internal/installdefaults/installdefaults_test.go
@@ -88,6 +88,17 @@ func TestAPIServerSPIFFEID(t *testing.T) {
}
}
+func TestAPIServiceHost(t *testing.T) {
+ for _, tt := range []struct{ namespace, want string }{
+ {"ate-system", "api.ate-system.svc"},
+ {"substrate-prod", "api.substrate-prod.svc"},
+ } {
+ if got := APIServiceHost(tt.namespace); got != tt.want {
+ t.Errorf("APIServiceHost(%q) = %q, want %q", tt.namespace, got, tt.want)
+ }
+ }
+}
+
func TestActorJWTIssuer(t *testing.T) {
tests := []struct {
name string
diff --git a/internal/objectstoreplugin/actorlookup/lookup.go b/internal/objectstoreplugin/actorlookup/lookup.go
new file mode 100644
index 0000000000..e57a8c9b50
--- /dev/null
+++ b/internal/objectstoreplugin/actorlookup/lookup.go
@@ -0,0 +1,104 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package actorlookup fetches the actor an exchange request names from
+// ate-api-server. The control plugin is a sidecar in the ate-api-server pod, so
+// it dials the loopback address and presents the pod identity certificate,
+// which ate-api-server accepts as an mTLS client.
+package actorlookup
+
+import (
+ "context"
+ "errors"
+ "fmt"
+
+ "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"
+ "google.golang.org/grpc"
+
+ "go.ate.dev/substrate/internal/actoridjwt"
+ "go.ate.dev/substrate/internal/ateapiauth"
+ "go.ate.dev/substrate/pkg/proto/ateapipb"
+)
+
+// ErrStale is returned when the actor at the name a JWT carries has a
+// different UID: the actor was deleted and recreated since the JWT was minted.
+var ErrStale = errors.New("actor UID does not match the JWT")
+
+// Config says how to reach ate-api-server. Every file is re-read on each
+// handshake, so certificate rotation needs no restart.
+type Config struct {
+ // Address is the gRPC target, 127.0.0.1:443 in the pod.
+ Address string
+ // CAFile verifies the serving certificate.
+ CAFile string
+ // ServerName is the DNS name expected on the serving certificate, since
+ // Address is not a name the certificate carries.
+ ServerName string
+ // ClientCredBundle is the PEM bundle with the pod identity certificate
+ // and key.
+ ClientCredBundle string
+}
+
+// Lookup resolves actors over one connection to ate-api-server.
+type Lookup struct {
+ conn *grpc.ClientConn
+ client ateapipb.ControlClient
+}
+
+// New dials ate-api-server. The connection is lazy, so a server that is not up
+// yet fails the first lookup rather than New.
+func New(cfg Config) (*Lookup, error) {
+ if cfg.Address == "" {
+ return nil, errors.New("actorlookup: Address is required")
+ }
+ opts, err := ateapiauth.DialOptions(ateapiauth.ClientConfig{
+ CAFile: cfg.CAFile,
+ ServerName: cfg.ServerName,
+ ClientCredBundle: cfg.ClientCredBundle,
+ })
+ if err != nil {
+ return nil, err
+ }
+ // Carry the exchange's trace into GetActor, as the other ate-api-server
+ // clients do.
+ opts = append(opts, grpc.WithStatsHandler(otelgrpc.NewClientHandler()))
+ conn, err := grpc.NewClient(cfg.Address, opts...)
+ if err != nil {
+ return nil, fmt.Errorf("actorlookup: while creating the ate-api-server client: %w", err)
+ }
+ return &Lookup{conn: conn, client: ateapipb.NewControlClient(conn)}, nil
+}
+
+// Actor returns the actor the claims name. The claims come from a verified
+// JWT, so the atespace and name are trusted, but the actor may have been
+// recreated since, which the UID detects. A gRPC status error from
+// ate-api-server, such as NotFound, is returned unchanged.
+func (l *Lookup) Actor(ctx context.Context, claims *actoridjwt.WireSubstrateClaims) (*ateapipb.Actor, error) {
+ if claims == nil || claims.Atespace == "" || claims.ActorName == "" || claims.ActorUID == "" {
+ return nil, errors.New("actorlookup: claims lack atespace, actor name, or UID")
+ }
+ actor, err := l.client.GetActor(ctx, &ateapipb.GetActorRequest{
+ Actor: &ateapipb.ObjectRef{Atespace: claims.Atespace, Name: claims.ActorName},
+ })
+ if err != nil {
+ return nil, err
+ }
+ if actor.GetMetadata().GetUid() != claims.ActorUID {
+ return nil, ErrStale
+ }
+ return actor, nil
+}
+
+// Close closes the connection.
+func (l *Lookup) Close() error { return l.conn.Close() }
diff --git a/internal/objectstoreplugin/actorlookup/lookup_test.go b/internal/objectstoreplugin/actorlookup/lookup_test.go
new file mode 100644
index 0000000000..c3d81be4fe
--- /dev/null
+++ b/internal/objectstoreplugin/actorlookup/lookup_test.go
@@ -0,0 +1,279 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package actorlookup
+
+import (
+ "context"
+ "crypto/ecdsa"
+ "crypto/elliptic"
+ "crypto/rand"
+ "crypto/tls"
+ "crypto/x509"
+ "crypto/x509/pkix"
+ "encoding/pem"
+ "errors"
+ "math/big"
+ "net"
+ "net/url"
+ "os"
+ "path/filepath"
+ "testing"
+ "time"
+
+ "google.golang.org/grpc"
+ "google.golang.org/grpc/codes"
+ "google.golang.org/grpc/credentials"
+ "google.golang.org/grpc/peer"
+ "google.golang.org/grpc/status"
+
+ "go.ate.dev/substrate/internal/actoridjwt"
+ "go.ate.dev/substrate/pkg/proto/ateapipb"
+)
+
+const serverName = "api.ate-system.svc"
+
+// fakeControl serves GetActor for one actor and records the client identity
+// it saw.
+type fakeControl struct {
+ ateapipb.UnimplementedControlServer
+ actor *ateapipb.Actor
+ peerURIs []*url.URL
+}
+
+func (f *fakeControl) GetActor(ctx context.Context, req *ateapipb.GetActorRequest) (*ateapipb.Actor, error) {
+ if p, ok := peerCert(ctx); ok {
+ f.peerURIs = p.URIs
+ }
+ if req.GetActor().GetAtespace() != f.actor.GetMetadata().GetAtespace() || req.GetActor().GetName() != f.actor.GetMetadata().GetName() {
+ return nil, status.Errorf(codes.NotFound, "no actor %s/%s", req.GetActor().GetAtespace(), req.GetActor().GetName())
+ }
+ return f.actor, nil
+}
+
+func peerCert(ctx context.Context) (*x509.Certificate, bool) {
+ p, ok := peer.FromContext(ctx)
+ if !ok {
+ return nil, false
+ }
+ info, ok := p.AuthInfo.(credentials.TLSInfo)
+ if !ok || len(info.State.PeerCertificates) == 0 {
+ return nil, false
+ }
+ return info.State.PeerCertificates[0], true
+}
+
+// startServer serves fake over mTLS on the loopback and returns a Config that
+// reaches it with a client certificate carrying spiffeID.
+func startServer(t *testing.T, fake *fakeControl, spiffeID string) Config {
+ t.Helper()
+ ca := newTestCA(t)
+ dir := t.TempDir()
+ caFile := filepath.Join(dir, "ca.pem")
+ if err := os.WriteFile(caFile, pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: ca.certDER}), 0o600); err != nil {
+ t.Fatal(err)
+ }
+ bundle := filepath.Join(dir, "client.pem")
+ if err := os.WriteFile(bundle, ca.issueClientBundle(t, spiffeID), 0o600); err != nil {
+ t.Fatal(err)
+ }
+ pool := x509.NewCertPool()
+ pool.AddCert(ca.cert)
+ srv := grpc.NewServer(grpc.Creds(credentials.NewTLS(&tls.Config{
+ MinVersion: tls.VersionTLS13,
+ Certificates: []tls.Certificate{ca.issueServerCert(t)},
+ ClientAuth: tls.RequireAndVerifyClientCert,
+ ClientCAs: pool,
+ })))
+ ateapipb.RegisterControlServer(srv, fake)
+ lis, err := net.Listen("tcp", "127.0.0.1:0")
+ if err != nil {
+ t.Fatal(err)
+ }
+ go func() { _ = srv.Serve(lis) }()
+ t.Cleanup(srv.Stop)
+ return Config{Address: lis.Addr().String(), CAFile: caFile, ServerName: serverName, ClientCredBundle: bundle}
+}
+
+func testActor(uid string) *ateapipb.Actor {
+ return &ateapipb.Actor{Metadata: &ateapipb.ResourceMetadata{Atespace: "team-a", Name: "agent-1", Uid: uid}}
+}
+
+func TestActor(t *testing.T) {
+ const spiffeID = "spiffe://ate.dev/ns/ate-system/sa/ate-api-server"
+ fake := &fakeControl{actor: testActor("uid-1")}
+ l, err := New(startServer(t, fake, spiffeID))
+ if err != nil {
+ t.Fatalf("New: %v", err)
+ }
+ defer l.Close()
+
+ claims := &actoridjwt.WireSubstrateClaims{Atespace: "team-a", ActorName: "agent-1", ActorUID: "uid-1"}
+ actor, err := l.Actor(t.Context(), claims)
+ if err != nil {
+ t.Fatalf("Actor: %v", err)
+ }
+ if actor.GetMetadata().GetUid() != "uid-1" {
+ t.Errorf("got actor %v", actor.GetMetadata())
+ }
+ if len(fake.peerURIs) != 1 || fake.peerURIs[0].String() != spiffeID {
+ t.Errorf("server saw client identity %v, want %s", fake.peerURIs, spiffeID)
+ }
+}
+
+func TestActor_Errors(t *testing.T) {
+ fake := &fakeControl{actor: testActor("uid-1")}
+ l, err := New(startServer(t, fake, "spiffe://ate.dev/ns/ate-system/sa/ate-api-server"))
+ if err != nil {
+ t.Fatalf("New: %v", err)
+ }
+ defer l.Close()
+
+ for _, tc := range []struct {
+ name string
+ claims *actoridjwt.WireSubstrateClaims
+ code codes.Code
+ is error
+ }{
+ {"recreated actor", &actoridjwt.WireSubstrateClaims{Atespace: "team-a", ActorName: "agent-1", ActorUID: "uid-0"}, codes.Unknown, ErrStale},
+ {"unknown actor", &actoridjwt.WireSubstrateClaims{Atespace: "team-a", ActorName: "agent-2", ActorUID: "uid-2"}, codes.NotFound, nil},
+ {"incomplete claims", &actoridjwt.WireSubstrateClaims{Atespace: "team-a", ActorName: "agent-1"}, codes.Unknown, nil},
+ {"nil claims", nil, codes.Unknown, nil},
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ _, err := l.Actor(t.Context(), tc.claims)
+ if err == nil {
+ t.Fatal("got an actor, want an error")
+ }
+ if tc.is != nil && !errors.Is(err, tc.is) {
+ t.Errorf("err = %v, want %v", err, tc.is)
+ }
+ if status.Code(err) != tc.code {
+ t.Errorf("code = %v, want %v (%v)", status.Code(err), tc.code, err)
+ }
+ })
+ }
+}
+
+func TestActor_RefusesUnverifiedServer(t *testing.T) {
+ // A server whose certificate the CA file does not sign must be refused,
+ // otherwise a wrong loopback peer could answer for ate-api-server.
+ cfg := startServer(t, &fakeControl{actor: testActor("uid-1")}, "spiffe://ate.dev/x")
+ other := startServer(t, &fakeControl{actor: testActor("uid-1")}, "spiffe://ate.dev/x")
+ cfg.Address = other.Address
+ l, err := New(cfg)
+ if err != nil {
+ t.Fatalf("New: %v", err)
+ }
+ defer l.Close()
+ ctx, cancel := context.WithTimeout(t.Context(), 5*time.Second)
+ defer cancel()
+ _, err = l.Actor(ctx, &actoridjwt.WireSubstrateClaims{Atespace: "team-a", ActorName: "agent-1", ActorUID: "uid-1"})
+ if status.Code(err) != codes.Unavailable {
+ t.Fatalf("err = %v, want Unavailable from a failed handshake", err)
+ }
+}
+
+func TestNew_RequiresAddress(t *testing.T) {
+ if _, err := New(Config{CAFile: "x", ClientCredBundle: "y"}); err == nil {
+ t.Fatal("New accepted an empty Address")
+ }
+}
+
+type testCA struct {
+ cert *x509.Certificate
+ certDER []byte
+ key *ecdsa.PrivateKey
+}
+
+func generateKey(t *testing.T) *ecdsa.PrivateKey {
+ t.Helper()
+ key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
+ if err != nil {
+ t.Fatal(err)
+ }
+ return key
+}
+
+func newTestCA(t *testing.T) *testCA {
+ t.Helper()
+ key := generateKey(t)
+ template := &x509.Certificate{
+ SerialNumber: big.NewInt(1),
+ Subject: pkix.Name{CommonName: "test-ca"},
+ NotBefore: time.Now().Add(-time.Hour),
+ NotAfter: time.Now().Add(time.Hour),
+ IsCA: true,
+ KeyUsage: x509.KeyUsageCertSign,
+ BasicConstraintsValid: true,
+ }
+ der, err := x509.CreateCertificate(rand.Reader, template, template, &key.PublicKey, key)
+ if err != nil {
+ t.Fatal(err)
+ }
+ cert, err := x509.ParseCertificate(der)
+ if err != nil {
+ t.Fatal(err)
+ }
+ return &testCA{cert: cert, certDER: der, key: key}
+}
+
+func (ca *testCA) issueClientBundle(t *testing.T, spiffeID string) []byte {
+ t.Helper()
+ uri, err := url.Parse(spiffeID)
+ if err != nil {
+ t.Fatal(err)
+ }
+ key := generateKey(t)
+ template := &x509.Certificate{
+ SerialNumber: big.NewInt(2),
+ Subject: pkix.Name{CommonName: "test-client"},
+ NotBefore: time.Now().Add(-time.Hour),
+ NotAfter: time.Now().Add(time.Hour),
+ URIs: []*url.URL{uri},
+ KeyUsage: x509.KeyUsageDigitalSignature,
+ ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth},
+ }
+ der, err := x509.CreateCertificate(rand.Reader, template, ca.cert, &key.PublicKey, ca.key)
+ if err != nil {
+ t.Fatal(err)
+ }
+ keyDER, err := x509.MarshalPKCS8PrivateKey(key)
+ if err != nil {
+ t.Fatal(err)
+ }
+ return append(
+ pem.EncodeToMemory(&pem.Block{Type: "PRIVATE KEY", Bytes: keyDER}),
+ pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der})...,
+ )
+}
+
+func (ca *testCA) issueServerCert(t *testing.T) tls.Certificate {
+ t.Helper()
+ key := generateKey(t)
+ template := &x509.Certificate{
+ SerialNumber: big.NewInt(3),
+ Subject: pkix.Name{CommonName: "test-server"},
+ NotBefore: time.Now().Add(-time.Hour),
+ NotAfter: time.Now().Add(time.Hour),
+ DNSNames: []string{serverName},
+ KeyUsage: x509.KeyUsageDigitalSignature,
+ ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
+ }
+ der, err := x509.CreateCertificate(rand.Reader, template, ca.cert, &key.PublicKey, ca.key)
+ if err != nil {
+ t.Fatal(err)
+ }
+ return tls.Certificate{Certificate: [][]byte{der}, PrivateKey: key}
+}
diff --git a/internal/objectstoreplugin/actorscope/actorscope.go b/internal/objectstoreplugin/actorscope/actorscope.go
index 89e2e0a85d..ffef87b397 100644
--- a/internal/objectstoreplugin/actorscope/actorscope.go
+++ b/internal/objectstoreplugin/actorscope/actorscope.go
@@ -13,9 +13,9 @@
// limitations under the License.
// Package actorscope derives the object-storage prefixes an actor's transfers
-// may touch from the actor record, with the access each needs. It is the
-// cloud-neutral half of the credential exchange. A provider turns the grants
-// into a scoped credential, for GCS a Credential Access Boundary.
+// may touch from the actor record, with the access each needs. It knows no
+// storage provider: a provider's credential exchange turns the grants into
+// whatever scoped credential its storage supports.
package actorscope
import (
diff --git a/internal/objectstoreplugin/credential/credential.go b/internal/objectstoreplugin/credential/credential.go
new file mode 100644
index 0000000000..78022dbe62
--- /dev/null
+++ b/internal/objectstoreplugin/credential/credential.go
@@ -0,0 +1,26 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package credential holds the credential type the exchange hands out and
+// its issuers produce. It is a leaf so an issuer does not depend on the
+// handler that serves it.
+package credential
+
+import "time"
+
+// Token is a storage credential and when it expires.
+type Token struct {
+ AccessToken string
+ Expiry time.Time
+}
diff --git a/internal/objectstoreplugin/exchange/exchange.go b/internal/objectstoreplugin/exchange/exchange.go
new file mode 100644
index 0000000000..d3b3e5cda2
--- /dev/null
+++ b/internal/objectstoreplugin/exchange/exchange.go
@@ -0,0 +1,169 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package exchange serves this plugin's credential exchange. The node plugin
+// presents the worker-for-actor JWT atelet gave it and gets back a GCS access
+// token scoped to that actor's snapshot prefixes. The handler verifies the
+// JWT with oidcjwt, looks the actor up with actorlookup, derives the grants
+// with actorscope, and hands them to an Issuer. Those three packages are
+// provider-neutral and are what another provider's plugin reuses. This
+// package and its proto are not: they are shaped for this binary's
+// credential, and the Issuer interface is the seam between its backends.
+package exchange
+
+import (
+ "context"
+ "errors"
+ "slices"
+ "time"
+
+ "google.golang.org/grpc/codes"
+ "google.golang.org/grpc/status"
+ "google.golang.org/protobuf/types/known/timestamppb"
+
+ "go.ate.dev/substrate/internal/actoridjwt"
+ "go.ate.dev/substrate/internal/objectstoreplugin/actorlookup"
+ "go.ate.dev/substrate/internal/objectstoreplugin/actorscope"
+ "go.ate.dev/substrate/internal/objectstoreplugin/credential"
+ "go.ate.dev/substrate/internal/oidcjwt"
+ exchangepb "go.ate.dev/substrate/internal/proto/objectstoreexchangepb"
+ "go.ate.dev/substrate/internal/resources"
+ "go.ate.dev/substrate/pkg/proto/ateapipb"
+)
+
+// ActorJWTAudience is the audience of the objectstore plugin API. atelet
+// requests it on the worker-for-actor JWT it hands to whichever node plugin is
+// installed, and every exchange serving this API requires it. A token minted
+// for another audience is refused.
+const ActorJWTAudience = "objectstore-plugin.ate.dev"
+
+// Issuer mints a credential limited to grants for the actor. force skips any
+// cache the issuer keeps. An error carrying a gRPC status code other than
+// Unavailable is passed to the caller unchanged, for failures the same request
+// would hit again, and one wrapping the context's error keeps the caller's
+// code. Any other error is reported as Unavailable. The GCS issuer is
+// gcsissuer.
+type Issuer interface {
+ Issue(ctx context.Context, actorUID string, grants []actorscope.Grant, force bool) (credential.Token, error)
+}
+
+// Lookup resolves the actor a verified JWT names. actorlookup implements it.
+type Lookup interface {
+ Actor(ctx context.Context, claims *actoridjwt.WireSubstrateClaims) (*ateapipb.Actor, error)
+}
+
+// Verifier checks a JWT and returns its claims. oidcjwt.Verifier implements
+// it, built with WithSubstrateClaims so the token's kind is known.
+type Verifier interface {
+ Verify(ctx context.Context, jwt string, now time.Time) (*oidcjwt.Claims, error)
+}
+
+// Server implements CredentialExchange.
+type Server struct {
+ exchangepb.UnimplementedCredentialExchangeServer
+ verifier Verifier
+ lookup Lookup
+ issuer Issuer
+ now func() time.Time
+}
+
+// New returns a Server. The verifier must accept ActorJWTAudience.
+func New(verifier Verifier, lookup Lookup, issuer Issuer) *Server {
+ return &Server{verifier: verifier, lookup: lookup, issuer: issuer, now: time.Now}
+}
+
+// ExchangeActorJwt is the exchange. Only a worker-for-actor token is accepted:
+// an actor's own token has the same shape and can be minted with this
+// audience, and actor code must not obtain credentials for its own snapshots.
+func (s *Server) ExchangeActorJwt(ctx context.Context, req *exchangepb.ExchangeActorJwtRequest) (*exchangepb.ExchangeActorJwtResponse, error) {
+ if req.GetActorJwt() == "" {
+ return nil, status.Error(codes.InvalidArgument, "actor_jwt is required")
+ }
+ claims, err := s.verifier.Verify(ctx, req.GetActorJwt(), s.now())
+ if err != nil {
+ return nil, status.Errorf(codes.Unauthenticated, "actor_jwt is not valid: %v", err)
+ }
+ if claims.Kind != actoridjwt.KindWorkerForActor {
+ return nil, status.Error(codes.PermissionDenied, "only a worker-for-actor token may be exchanged")
+ }
+ if sub := claims.Substrate; sub == nil || sub.Atespace == "" || sub.ActorName == "" || sub.ActorUID == "" || sub.Node == "" {
+ return nil, status.Error(codes.Unauthenticated, "actor_jwt must carry the actor's atespace, name and UID and the minting node, and one of them is empty")
+ }
+ actor, err := s.lookup.Actor(ctx, claims.Substrate)
+ switch {
+ case errors.Is(err, actorlookup.ErrStale), status.Code(err) == codes.NotFound:
+ return nil, status.Error(codes.NotFound, "actor not found")
+ case status.Code(err) == codes.InvalidArgument:
+ // ate-api-server refused the actor reference the token carries.
+ return nil, status.Errorf(codes.Unauthenticated, "actor_jwt names an invalid actor: %v", err)
+ case status.Code(err) == codes.PermissionDenied, status.Code(err) == codes.Unauthenticated:
+ // The exchange's own identity lacks access. Retrying cannot help.
+ return nil, status.Errorf(codes.FailedPrecondition, "the exchange may not read actors: %v", err)
+ case status.Code(err) == codes.Canceled, status.Code(err) == codes.DeadlineExceeded:
+ // The caller gave up, which says nothing about ate-api-server.
+ return nil, err
+ case err != nil:
+ return nil, status.Errorf(codes.Unavailable, "while looking up the actor: %v", err)
+ }
+ if err := actorStillOnNode(actor, claims.Substrate.Node); err != nil {
+ return nil, err
+ }
+ grants, err := actorscope.FromActor(actor)
+ if err != nil {
+ return nil, status.Errorf(codes.Internal, "while deriving the actor's storage grants: %v", err)
+ }
+ if len(grants) == 0 {
+ return nil, status.Error(codes.PermissionDenied, "actor has no snapshot location to grant")
+ }
+ tok, err := s.issuer.Issue(ctx, actor.GetMetadata().GetUid(), grants, req.GetForce())
+ if err != nil {
+ return nil, status.Errorf(issueCode(err), "while issuing the credential: %v", err)
+ }
+ return &exchangepb.ExchangeActorJwtResponse{AccessToken: tok.AccessToken, ExpireTime: timestamppb.New(tok.Expiry)}, nil
+}
+
+// issueCode is the code the caller sees for an issuer error. A context error
+// keeps the caller's code, a gRPC status keeps its own, and anything else is
+// a backend failure worth retrying.
+func issueCode(err error) codes.Code {
+ switch {
+ case errors.Is(err, context.Canceled):
+ return codes.Canceled
+ case errors.Is(err, context.DeadlineExceeded):
+ return codes.DeadlineExceeded
+ case status.Code(err) == codes.Unknown:
+ return codes.Unavailable
+ }
+ return status.Code(err)
+}
+
+// actorStillOnNode refuses a token minted for a node the actor has since
+// left, or for an actor in a state with no transfer. ate-api-server checked
+// the node when it minted, and the token outlives that check by up to an
+// hour. status.assigned_node names the node holding the actor: a resume sets
+// it to the worker's node, it stays set while the actor is SUSPENDED and its
+// local snapshot remains on that node, and it is cleared when the node
+// detaches the actor, by a revert and by a delete, or replaced by a resume
+// onto another node. So a token from a node the actor moved away from, or
+// one kept past the actor's last transfer there, no longer matches.
+func actorStillOnNode(actor *ateapipb.Actor, node string) error {
+ st := actor.GetStatus()
+ if !slices.Contains(resources.TransferStates, st.GetState()) {
+ return status.Errorf(codes.PermissionDenied, "actor is %s, no transfer is due", st.GetState())
+ }
+ if st.GetAssignedNode() != node {
+ return status.Error(codes.PermissionDenied, "actor is no longer on the token's node")
+ }
+ return nil
+}
diff --git a/internal/objectstoreplugin/exchange/exchange_test.go b/internal/objectstoreplugin/exchange/exchange_test.go
new file mode 100644
index 0000000000..5776cc8a99
--- /dev/null
+++ b/internal/objectstoreplugin/exchange/exchange_test.go
@@ -0,0 +1,346 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package exchange
+
+import (
+ "context"
+ "crypto/ecdsa"
+ "crypto/rand"
+ "crypto/sha256"
+ "encoding/base64"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "os"
+ "path/filepath"
+ "slices"
+ "testing"
+ "time"
+
+ "google.golang.org/grpc/codes"
+ "google.golang.org/grpc/status"
+
+ "go.ate.dev/substrate/internal/actoridjwt"
+ "go.ate.dev/substrate/internal/localjwtauthority"
+ "go.ate.dev/substrate/internal/objectstoreplugin/actorlookup"
+ "go.ate.dev/substrate/internal/objectstoreplugin/actorscope"
+ "go.ate.dev/substrate/internal/objectstoreplugin/credential"
+ "go.ate.dev/substrate/internal/oidcjwt"
+ exchangepb "go.ate.dev/substrate/internal/proto/objectstoreexchangepb"
+ "go.ate.dev/substrate/internal/resources"
+ "go.ate.dev/substrate/pkg/proto/ateapipb"
+)
+
+const (
+ issuer = "https://idp.ate-system.svc"
+ atespace = "team-a"
+ name = "bot"
+ uid = "9f1e0b2a-9e4f-4c3d-8a6b-5f2e1d0c9b8a"
+ tagURI = "gs://snaps/root/atespaces/ate-golden/tags/7a3c0b2a-9e4f-4c3d-8a6b-5f2e1d0c9b8a"
+)
+
+// signer mints real JWTs from a pool whose public keys the verifier reads.
+type signer struct {
+ pool *localjwtauthority.ConcretePool
+ now time.Time
+ // node is the node the signed tokens are minted for.
+ node string
+}
+
+func newSigner(t *testing.T) (*signer, *oidcjwt.Verifier) {
+ t.Helper()
+ auth, err := localjwtauthority.GenerateAuthority("ES256", "k1")
+ if err != nil {
+ t.Fatal(err)
+ }
+ pool := &localjwtauthority.ConcretePool{Authorities: []*localjwtauthority.Authority{auth}, ActiveForSigning: "k1"}
+ jwks, err := pool.JWKS()
+ if err != nil {
+ t.Fatal(err)
+ }
+ path := filepath.Join(t.TempDir(), "jwks.json")
+ if err := os.WriteFile(path, jwks, 0o600); err != nil {
+ t.Fatal(err)
+ }
+ v := oidcjwt.NewVerifier(issuer, []string{ActorJWTAudience}, oidcjwt.FileKeySource(path), oidcjwt.WithSubstrateClaims())
+ return &signer{pool: pool, now: time.Now(), node: "node-1"}, v
+}
+
+func (s *signer) jwt(t *testing.T, kind actoridjwt.Kind, audience string, actorUID string) string {
+ t.Helper()
+ substrate := actoridjwt.SubstrateClaims{Atespace: atespace, ActorName: name, ActorUID: actorUID}
+ if kind == actoridjwt.KindWorkerForActor {
+ // Only a worker-for-actor token is tied to a node.
+ substrate.Node = s.node
+ }
+ jwt, err := s.pool.SignJWT(&actoridjwt.Claims{
+ Kind: kind,
+ Issuer: issuer,
+ Subject: kind.SubjectPrefix() + "/" + atespace + "/" + name,
+ Audiences: []string{audience},
+ Expiration: s.now.Add(time.Hour),
+ NotBefore: s.now.Add(-time.Minute),
+ IssuedAt: s.now,
+ Substrate: substrate,
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+ return jwt
+}
+
+// fakeLookup answers for one actor and records the claims it was asked for.
+type fakeLookup struct {
+ actor *ateapipb.Actor
+ err error
+}
+
+func (f *fakeLookup) Actor(_ context.Context, claims *actoridjwt.WireSubstrateClaims) (*ateapipb.Actor, error) {
+ if f.err != nil {
+ return nil, f.err
+ }
+ if claims.ActorUID != f.actor.GetMetadata().GetUid() {
+ return nil, actorlookup.ErrStale
+ }
+ return f.actor, nil
+}
+
+// fakeIssuer records what it was asked to mint.
+type fakeIssuer struct {
+ err error
+ calls int
+ uid string
+ grants []actorscope.Grant
+ force bool
+ expiry time.Time
+}
+
+func (f *fakeIssuer) Issue(_ context.Context, actorUID string, grants []actorscope.Grant, force bool) (credential.Token, error) {
+ f.calls++
+ f.uid, f.grants, f.force = actorUID, grants, force
+ if f.err != nil {
+ return credential.Token{}, f.err
+ }
+ return credential.Token{AccessToken: "downscoped", Expiry: f.expiry}, nil
+}
+
+// rawJWT signs an arbitrary claim set with the signer's key, for tokens
+// actoridjwt refuses to build.
+func (s *signer) rawJWT(t *testing.T, claims map[string]any) string {
+ t.Helper()
+ header, err := json.Marshal(map[string]string{"alg": "ES256", "typ": "JWT", "kid": "k1"})
+ if err != nil {
+ t.Fatal(err)
+ }
+ payload, err := json.Marshal(claims)
+ if err != nil {
+ t.Fatal(err)
+ }
+ input := base64.RawURLEncoding.EncodeToString(header) + "." + base64.RawURLEncoding.EncodeToString(payload)
+ key := s.pool.Authorities[0].SigningKey.(*ecdsa.PrivateKey)
+ digest := sha256.Sum256([]byte(input))
+ r, sg, err := ecdsa.Sign(rand.Reader, key, digest[:])
+ if err != nil {
+ t.Fatal(err)
+ }
+ sig := make([]byte, 64)
+ r.FillBytes(sig[:32])
+ sg.FillBytes(sig[32:])
+ return input + "." + base64.RawURLEncoding.EncodeToString(sig)
+}
+
+// rawClaims is a valid worker-for-actor claim set, for rawJWT to vary.
+func (s *signer) rawClaims() map[string]any {
+ substrate := map[string]string{"atespace": atespace, "actorName": name, "actorUID": uid, "node": s.node}
+ return map[string]any{
+ "iss": issuer, "sub": "worker-for-actor/" + atespace + "/" + name, "aud": ActorJWTAudience,
+ "exp": s.now.Add(time.Hour).Unix(), "nbf": s.now.Add(-time.Minute).Unix(), "iat": s.now.Unix(),
+ "worker-for-actor.ate.dev": substrate,
+ }
+}
+
+// actorWithTag is a running actor on node-1 that still carries the tag
+// generation it was created from.
+func actorWithTag() *ateapipb.Actor {
+ return &ateapipb.Actor{
+ Metadata: &ateapipb.ResourceMetadata{Atespace: atespace, Name: name, Uid: uid},
+ Status: &ateapipb.ActorStatus{State: ateapipb.ActorState_ACTOR_STATE_RUNNING, AssignedNode: "node-1", Snapshots: []*ateapipb.Snapshot{{
+ Generation: 1, Owner: ateapipb.SnapshotOwner_SNAPSHOT_OWNER_TAG,
+ DurableSnapshot: &ateapipb.SnapshotStorage{
+ Status: ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED,
+ Object: &ateapipb.ObjectSnapshot{SnapshotUri: tagURI},
+ },
+ }}},
+ }
+}
+
+// actorLocalOnly is a paused actor whose only snapshot is on node-1.
+func actorLocalOnly() *ateapipb.Actor {
+ return &ateapipb.Actor{
+ Metadata: &ateapipb.ResourceMetadata{Atespace: atespace, Name: name, Uid: uid},
+ Status: &ateapipb.ActorStatus{State: ateapipb.ActorState_ACTOR_STATE_SUSPENDED, AssignedNode: "node-1",
+ Snapshots: []*ateapipb.Snapshot{{Generation: 1, Owner: ateapipb.SnapshotOwner_SNAPSHOT_OWNER_ACTOR, Locality: "node-1"}}},
+ }
+}
+
+// actorIn returns actorWithTag in the given state on the given node.
+func actorIn(state ateapipb.ActorState, node string) *ateapipb.Actor {
+ a := actorWithTag()
+ a.Status.State, a.Status.AssignedNode = state, node
+ return a
+}
+
+// TestExchangeActorJwt_NodeBinding pins which actor states and placements a
+// token minted on node-1 is good for.
+func TestExchangeActorJwt_NodeBinding(t *testing.T) {
+ s, v := newSigner(t)
+ for _, tc := range []struct {
+ name string
+ actor *ateapipb.Actor
+ want codes.Code
+ }{
+ {"resuming on the token's node", actorIn(ateapipb.ActorState_ACTOR_STATE_RESUMING, "node-1"), codes.OK},
+ {"running on the token's node", actorIn(ateapipb.ActorState_ACTOR_STATE_RUNNING, "node-1"), codes.OK},
+ {"suspending on the token's node", actorIn(ateapipb.ActorState_ACTOR_STATE_SUSPENDING, "node-1"), codes.OK},
+ {"suspended with its snapshot on the token's node", actorIn(ateapipb.ActorState_ACTOR_STATE_SUSPENDED, "node-1"), codes.OK},
+ {"resumed on another node", actorIn(ateapipb.ActorState_ACTOR_STATE_RUNNING, "node-2"), codes.PermissionDenied},
+ {"detached from the node", actorIn(ateapipb.ActorState_ACTOR_STATE_SUSPENDED, ""), codes.PermissionDenied},
+ {"crashed on the token's node", actorIn(ateapipb.ActorState_ACTOR_STATE_CRASHED, "node-1"), codes.PermissionDenied},
+ {"deleting on the token's node", actorIn(ateapipb.ActorState_ACTOR_STATE_DELETING, "node-1"), codes.PermissionDenied},
+ {"reverting on the token's node", actorIn(ateapipb.ActorState_ACTOR_STATE_REVERTING, "node-1"), codes.PermissionDenied},
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ issuer := &fakeIssuer{expiry: s.now.Add(time.Hour)}
+ srv := New(v, &fakeLookup{actor: tc.actor}, issuer)
+ srv.now = func() time.Time { return s.now }
+ _, err := srv.ExchangeActorJwt(t.Context(), &exchangepb.ExchangeActorJwtRequest{ActorJwt: s.jwt(t, actoridjwt.KindWorkerForActor, ActorJWTAudience, uid)})
+ if status.Code(err) != tc.want {
+ t.Fatalf("code = %v (%v), want %v", status.Code(err), err, tc.want)
+ }
+ if wantCalls := map[bool]int{true: 1, false: 0}[tc.want == codes.OK]; issuer.calls != wantCalls {
+ t.Errorf("issuer was called %d times, want %d", issuer.calls, wantCalls)
+ }
+ })
+ }
+ t.Run("token without a node", func(t *testing.T) {
+ // Refused before the lookup: a lookup that would answer NotFound must
+ // not be reached.
+ s.node = ""
+ defer func() { s.node = "node-1" }()
+ srv := New(v, &fakeLookup{err: status.Error(codes.NotFound, "no such actor")}, &fakeIssuer{})
+ srv.now = func() time.Time { return s.now }
+ _, err := srv.ExchangeActorJwt(t.Context(), &exchangepb.ExchangeActorJwtRequest{ActorJwt: s.jwt(t, actoridjwt.KindWorkerForActor, ActorJWTAudience, uid)})
+ if status.Code(err) != codes.Unauthenticated {
+ t.Fatalf("code = %v (%v), want Unauthenticated", status.Code(err), err)
+ }
+ })
+}
+
+func TestExchangeActorJwt(t *testing.T) {
+ s, v := newSigner(t)
+ expiry := s.now.Add(50 * time.Minute).Truncate(time.Second)
+ issuerOK := &fakeIssuer{expiry: expiry}
+ srv := New(v, &fakeLookup{actor: actorWithTag()}, issuerOK)
+ srv.now = func() time.Time { return s.now }
+
+ for _, force := range []bool{false, true} {
+ resp, err := srv.ExchangeActorJwt(t.Context(), &exchangepb.ExchangeActorJwtRequest{ActorJwt: s.jwt(t, actoridjwt.KindWorkerForActor, ActorJWTAudience, uid), Force: force})
+ if err != nil {
+ t.Fatalf("force=%v: %v", force, err)
+ }
+ if resp.GetAccessToken() != "downscoped" || !resp.GetExpireTime().AsTime().Equal(expiry) {
+ t.Errorf("force=%v: response = %v", force, resp)
+ }
+ if issuerOK.uid != uid || issuerOK.force != force {
+ t.Errorf("force=%v: issuer asked for uid=%s force=%v", force, issuerOK.uid, issuerOK.force)
+ }
+ // Read-write on the actor's own prefix at the tag's location, read-only
+ // on the tag it was created from.
+ want := []actorscope.Grant{
+ {Prefix: storagePrefix(t, "gs://snaps/root/atespaces/"+atespace+"/actors/"+uid), Access: actorscope.ReadWrite},
+ {Prefix: storagePrefix(t, tagURI), Access: actorscope.ReadOnly},
+ }
+ if !slices.Equal(issuerOK.grants, want) {
+ t.Errorf("force=%v: issuer asked for grants %v, want %v", force, issuerOK.grants, want)
+ }
+ }
+ if issuerOK.calls != 2 {
+ t.Errorf("issuer calls = %d, want 2", issuerOK.calls)
+ }
+}
+
+func storagePrefix(t *testing.T, uri string) resources.StoragePrefix {
+ t.Helper()
+ p, err := resources.ParseStoragePrefix(uri)
+ if err != nil {
+ t.Fatal(err)
+ }
+ return p
+}
+
+func TestExchangeActorJwt_Errors(t *testing.T) {
+ s, v := newSigner(t)
+ stranger, _ := newSigner(t)
+ good := s.jwt(t, actoridjwt.KindWorkerForActor, ActorJWTAudience, uid)
+ noClaim := s.rawClaims()
+ delete(noClaim, "worker-for-actor.ate.dev")
+ bothClaims := s.rawClaims()
+ bothClaims["actor.ate.dev"] = bothClaims["worker-for-actor.ate.dev"]
+ noUID := s.rawClaims()
+ noUID["worker-for-actor.ate.dev"] = map[string]string{"atespace": atespace, "actorName": name}
+ for _, tc := range []struct {
+ name string
+ jwt string
+ lookup *fakeLookup
+ issuer *fakeIssuer
+ want codes.Code
+ }{
+ {"empty jwt", "", &fakeLookup{actor: actorWithTag()}, &fakeIssuer{}, codes.InvalidArgument},
+ {"bad signature", stranger.jwt(t, actoridjwt.KindWorkerForActor, ActorJWTAudience, uid), &fakeLookup{actor: actorWithTag()}, &fakeIssuer{}, codes.Unauthenticated},
+ {"wrong audience", s.jwt(t, actoridjwt.KindWorkerForActor, "other", uid), &fakeLookup{actor: actorWithTag()}, &fakeIssuer{}, codes.Unauthenticated},
+ {"actor's own token", s.jwt(t, actoridjwt.KindActor, ActorJWTAudience, uid), &fakeLookup{actor: actorWithTag()}, &fakeIssuer{}, codes.PermissionDenied},
+ {"no substrate claim", s.rawJWT(t, noClaim), &fakeLookup{actor: actorWithTag()}, &fakeIssuer{}, codes.PermissionDenied},
+ {"both substrate claims", s.rawJWT(t, bothClaims), &fakeLookup{actor: actorWithTag()}, &fakeIssuer{}, codes.Unauthenticated},
+ {"claim without the actor uid", s.rawJWT(t, noUID), &fakeLookup{actor: actorWithTag()}, &fakeIssuer{}, codes.Unauthenticated},
+ {"ate-api-server rejects the actor ref", good, &fakeLookup{err: status.Error(codes.InvalidArgument, "bad name")}, &fakeIssuer{}, codes.Unauthenticated},
+ {"caller gave up during the lookup", good, &fakeLookup{err: status.Error(codes.Canceled, "canceled")}, &fakeIssuer{}, codes.Canceled},
+ {"caller's deadline passed during the lookup", good, &fakeLookup{err: status.Error(codes.DeadlineExceeded, "deadline")}, &fakeIssuer{}, codes.DeadlineExceeded},
+ {"recreated actor", s.jwt(t, actoridjwt.KindWorkerForActor, ActorJWTAudience, "old-uid"), &fakeLookup{actor: actorWithTag()}, &fakeIssuer{}, codes.NotFound},
+ {"actor gone", good, &fakeLookup{err: status.Error(codes.NotFound, "no such actor")}, &fakeIssuer{}, codes.NotFound},
+ {"ate-api-server down", good, &fakeLookup{err: status.Error(codes.Unavailable, "connection refused")}, &fakeIssuer{}, codes.Unavailable},
+ {"exchange lacks the actor read role", good, &fakeLookup{err: status.Error(codes.PermissionDenied, "not allowed")}, &fakeIssuer{}, codes.FailedPrecondition},
+ {"no snapshot location", good, &fakeLookup{actor: actorLocalOnly()}, &fakeIssuer{}, codes.PermissionDenied},
+ {"issuer failing", good, &fakeLookup{actor: actorWithTag()}, &fakeIssuer{err: errors.New("STS 503")}, codes.Unavailable},
+ {"issuer cannot express the grants", good, &fakeLookup{actor: actorWithTag()}, &fakeIssuer{err: status.Error(codes.Internal, "bad boundary")}, codes.Internal},
+ {"caller gave up at STS", good, &fakeLookup{actor: actorWithTag()}, &fakeIssuer{err: fmt.Errorf("Post sts: %w", context.Canceled)}, codes.Canceled},
+ {"caller's deadline passed at STS", good, &fakeLookup{actor: actorWithTag()}, &fakeIssuer{err: fmt.Errorf("Post sts: %w", context.DeadlineExceeded)}, codes.DeadlineExceeded},
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ srv := New(v, tc.lookup, tc.issuer)
+ srv.now = func() time.Time { return s.now }
+ _, err := srv.ExchangeActorJwt(t.Context(), &exchangepb.ExchangeActorJwtRequest{ActorJwt: tc.jwt})
+ if status.Code(err) != tc.want {
+ t.Fatalf("code = %v (%v), want %v", status.Code(err), err, tc.want)
+ }
+ wantCalls := 0
+ if tc.issuer.err != nil {
+ wantCalls = 1
+ }
+ if tc.issuer.calls != wantCalls {
+ t.Errorf("issuer was called %d times, want %d", tc.issuer.calls, wantCalls)
+ }
+ })
+ }
+}
diff --git a/internal/objectstoreplugin/gcsissuer/issuer.go b/internal/objectstoreplugin/gcsissuer/issuer.go
index e48c88b909..22b71db891 100644
--- a/internal/objectstoreplugin/gcsissuer/issuer.go
+++ b/internal/objectstoreplugin/gcsissuer/issuer.go
@@ -35,9 +35,12 @@ import (
"golang.org/x/oauth2"
"golang.org/x/oauth2/google"
"golang.org/x/oauth2/google/downscope"
+ "google.golang.org/grpc/codes"
+ "google.golang.org/grpc/status"
"k8s.io/utils/lru"
"go.ate.dev/substrate/internal/objectstoreplugin/actorscope"
+ "go.ate.dev/substrate/internal/objectstoreplugin/credential"
"go.ate.dev/substrate/internal/objectstoreplugin/gcsboundary"
)
@@ -58,12 +61,6 @@ const (
stsTimeout = 30 * time.Second
)
-// Token is a downscoped GCS access token and when it expires.
-type Token struct {
- AccessToken string
- Expiry time.Time
-}
-
// Issuer exchanges the pod's credential for per-actor downscoped tokens and
// caches them by actor UID.
type Issuer struct {
@@ -76,7 +73,7 @@ type Issuer struct {
// entry is a cached token with the grants it was minted for, so a request can
// tell whether the token covers it.
type entry struct {
- token Token
+ token credential.Token
grants []actorscope.Grant
}
@@ -140,11 +137,14 @@ func New(ctx context.Context, opts ...Option) (*Issuer, error) {
// widen when a never-suspended actor is repointed to another storage
// location, and the cached token lacks it, a mint. So the common path costs
// nothing and a repoint never hands out a token GCS would refuse.
-func (i *Issuer) Issue(ctx context.Context, actorUID string, grants []actorscope.Grant, force bool) (Token, error) {
+//
+// A grant set the access boundary cannot express fails with codes.Internal.
+// Every other failure is an STS or network error and carries no code.
+func (i *Issuer) Issue(ctx context.Context, actorUID string, grants []actorscope.Grant, force bool) (credential.Token, error) {
if len(grants) == 0 {
// Checked before the cache: an empty set is covered by any cached
// token, but an actor with no grants must get nothing.
- return Token{}, errors.New("actor has no storage grants")
+ return credential.Token{}, status.Error(codes.Internal, "actor has no storage grants")
}
if !force {
if v, ok := i.cache.Get(actorUID); ok {
@@ -155,7 +155,7 @@ func (i *Issuer) Issue(ctx context.Context, actorUID string, grants []actorscope
}
tok, err := i.mint(ctx, grants)
if err != nil {
- return Token{}, err
+ return credential.Token{}, err
}
if tok.Expiry.Sub(i.now()) >= minServeLifetime {
i.cache.Add(actorUID, entry{token: tok, grants: slices.Clone(grants)})
@@ -178,10 +178,10 @@ func covers(cached, requested []actorscope.Grant) bool {
return true
}
-func (i *Issuer) mint(ctx context.Context, grants []actorscope.Grant) (Token, error) {
+func (i *Issuer) mint(ctx context.Context, grants []actorscope.Grant) (credential.Token, error) {
rules, err := gcsboundary.FromGrants(grants)
if err != nil {
- return Token{}, fmt.Errorf("while building the access boundary: %w", err)
+ return credential.Token{}, status.Errorf(codes.Internal, "while building the access boundary: %v", err)
}
// The downscope package reads its HTTP client from the context but sends
// the STS request without one, so the caller's deadline is attached by the
@@ -192,21 +192,21 @@ func (i *Issuer) mint(ctx context.Context, grants []actorscope.Grant) (Token, er
})
ts, err := downscope.NewTokenSource(ctx, downscope.DownscopingConfig{RootSource: i.source, Rules: rules})
if err != nil {
- return Token{}, fmt.Errorf("while building the access boundary: %w", err)
+ return credential.Token{}, status.Errorf(codes.Internal, "while building the access boundary: %v", err)
}
tok, err := ts.Token()
if err != nil {
- return Token{}, fmt.Errorf("while exchanging at STS: %w", err)
+ return credential.Token{}, fmt.Errorf("while exchanging at STS: %w", err)
}
if tok.AccessToken == "" {
- return Token{}, errors.New("STS returned no access token")
+ return credential.Token{}, errors.New("STS returned no access token")
}
if tok.Expiry.IsZero() {
// Without an expiry the node plugin could not know when to stop using
// the token.
- return Token{}, errors.New("downscoped token has no expiry")
+ return credential.Token{}, errors.New("downscoped token has no expiry")
}
- return Token{AccessToken: tok.AccessToken, Expiry: tok.Expiry}, nil
+ return credential.Token{AccessToken: tok.AccessToken, Expiry: tok.Expiry}, nil
}
// contextTransport sends every request with ctx, so a cancelled or expired
diff --git a/internal/objectstoreplugin/gcsissuer/issuer_test.go b/internal/objectstoreplugin/gcsissuer/issuer_test.go
index ea07e37baf..1bfd687ebe 100644
--- a/internal/objectstoreplugin/gcsissuer/issuer_test.go
+++ b/internal/objectstoreplugin/gcsissuer/issuer_test.go
@@ -28,6 +28,8 @@ import (
"github.com/google/go-cmp/cmp"
"golang.org/x/oauth2"
"golang.org/x/oauth2/google/downscope"
+ "google.golang.org/grpc/codes"
+ "google.golang.org/grpc/status"
"go.ate.dev/substrate/internal/objectstoreplugin/actorscope"
"go.ate.dev/substrate/internal/objectstoreplugin/gcsboundary"
@@ -337,12 +339,15 @@ func TestIssue_Errors(t *testing.T) {
source oauth2.TokenSource
grants []actorscope.Grant
want string
+ // code is Internal for a failure the same request would hit again
+ // and Unknown for an STS or source failure, which carries no code.
+ code codes.Code
}{
- {"source without expiry", &fakeSTS{t: t}, staticSource(time.Time{}), testGrants, "no expiry"},
- {"STS refuses", &fakeSTS{t: t, status: 400, body: `{"error":"invalid_grant"}`}, staticSource(now.Add(time.Hour)), testGrants, "400"},
- {"STS answers without a token", &fakeSTS{t: t, body: `{}`}, staticSource(now.Add(time.Hour)), testGrants, "no access token"},
- {"no grants", &fakeSTS{t: t}, staticSource(now.Add(time.Hour)), nil, "no storage grants"},
- {"too many grants", &fakeSTS{t: t}, staticSource(now.Add(time.Hour)), manyGrants(11), "greater than 10"},
+ {"source without expiry", &fakeSTS{t: t}, staticSource(time.Time{}), testGrants, "no expiry", codes.Unknown},
+ {"STS refuses", &fakeSTS{t: t, status: 400, body: `{"error":"invalid_grant"}`}, staticSource(now.Add(time.Hour)), testGrants, "400", codes.Unknown},
+ {"STS answers without a token", &fakeSTS{t: t, body: `{}`}, staticSource(now.Add(time.Hour)), testGrants, "no access token", codes.Unknown},
+ {"no grants", &fakeSTS{t: t}, staticSource(now.Add(time.Hour)), nil, "no storage grants", codes.Internal},
+ {"too many grants", &fakeSTS{t: t}, staticSource(now.Add(time.Hour)), manyGrants(11), "greater than 10", codes.Internal},
} {
t.Run(tc.name, func(t *testing.T) {
i := newTestIssuer(t, tc.sts, tc.source, clock)
@@ -350,6 +355,9 @@ func TestIssue_Errors(t *testing.T) {
if err == nil || !strings.Contains(err.Error(), tc.want) {
t.Fatalf("err = %v, want it to mention %q", err, tc.want)
}
+ if status.Code(err) != tc.code {
+ t.Errorf("code = %v, want %v", status.Code(err), tc.code)
+ }
})
}
}
diff --git a/internal/ocispec/microvm.go b/internal/ocispec/microvm.go
index fc11c2047e..d446cc359b 100644
--- a/internal/ocispec/microvm.go
+++ b/internal/ocispec/microvm.go
@@ -39,12 +39,21 @@ const (
ShareVolumes = "volumes"
)
+// GuestContainerID returns the kata container ID for a container name.
+//
+// kata requires IDs of at least two characters, and a name can be one.
+// '_' is not valid in a name, so the ID can't collide with another name or
+// the Share* dirs. It must only depend on the name, since a restored guest
+// keeps the IDs it booted with.
+// https://github.com/kata-containers/kata-containers/blob/4.1.0/src/libs/kata-sys-util/src/validate.rs#L17-L22
+func GuestContainerID(name string) string { return "c_" + name }
+
// MicroVMOptions describes the micro-VM context of one actor container.
type MicroVMOptions struct {
// ActorDirs are the actor's directories; volume bind sources under them are
// staged into the share.
- ActorDirs *ateworkerpb.ActorDirs
- ContainerID string
+ ActorDirs *ateworkerpb.ActorDirs
+ ContainerName string
}
// ShapeMicroVM replaces host system mounts with guest mounts, repoints volume
@@ -58,7 +67,7 @@ func ShapeMicroVM(spec *specs.Spec, o MicroVMOptions) error {
if m.Type != "bind" {
continue
}
- src, err := guestVolumeSource(m.Source, o.ActorDirs, o.ContainerID)
+ src, err := guestVolumeSource(m.Source, o.ActorDirs, o.ContainerName)
if err != nil {
return fmt.Errorf("mount %q: %w", m.Destination, err)
}
@@ -112,12 +121,12 @@ func mergeKataResources(from *specs.LinuxResources) *specs.LinuxResources {
}
// guestVolumeSource maps a volume's host directory to its guest path.
-func guestVolumeSource(hostPath string, actorDirs *ateworkerpb.ActorDirs, containerID string) (string, error) {
+func guestVolumeSource(hostPath string, actorDirs *ateworkerpb.ActorDirs, containerName string) (string, error) {
for _, staged := range []struct{ host, guest string }{
{actorDirs.GetDurableDirVolumeMountsDir(), path.Join(GuestSharedDir, ShareDurable)},
{actorDirs.GetExternalVolumeMountsDir(), path.Join(GuestSharedDir, ShareCSI)},
{actorDirs.GetSystemInfoVolumeRootsDir(), path.Join(GuestSharedDir, ShareSystemInfo)},
- {imagecache.ImageVolumeMountPath(filepath.Join(actorDirs.GetOciBundleDir(), containerID), ""), path.Join(GuestSharedDir, containerID, ShareVolumes)},
+ {imagecache.ImageVolumeMountPath(filepath.Join(actorDirs.GetOciBundleDir(), containerName), ""), path.Join(GuestSharedDir, GuestContainerID(containerName), ShareVolumes)},
} {
if rel, ok := strings.CutPrefix(hostPath, staged.host+"/"); ok {
return path.Join(staged.guest, rel), nil
diff --git a/internal/ocispec/microvm_test.go b/internal/ocispec/microvm_test.go
index 905a0f792e..60110eb21b 100644
--- a/internal/ocispec/microvm_test.go
+++ b/internal/ocispec/microvm_test.go
@@ -15,12 +15,31 @@
package ocispec
import (
+ "regexp"
+ "slices"
+ "strings"
"testing"
"github.com/opencontainers/runtime-spec/specs-go"
"go.ate.dev/substrate/internal/proto/ateworkerpb"
)
+// kata's verify_id, see GuestContainerID
+var kataID = regexp.MustCompile(`^[a-zA-Z0-9][a-zA-Z0-9._-]+$`)
+
+func TestGuestContainerID(t *testing.T) {
+ shares := []string{ShareDurable, ShareCSI, ShareSystemInfo}
+ for _, name := range append([]string{"w", "app", strings.Repeat("a", 63)}, shares...) {
+ id := GuestContainerID(name)
+ if !kataID.MatchString(id) {
+ t.Errorf("GuestContainerID(%q) = %q, which kata rejects", name, id)
+ }
+ if slices.Contains(shares, id) {
+ t.Errorf("GuestContainerID(%q) = %q, which collides with a share dir", name, id)
+ }
+ }
+}
+
// The device allowlist and CPU shares from guestResources are the proven-good
// kata shape. Overlaying a memory limit must not drop them: a container without
// the allowlist fails in ways that do not point back here.
@@ -125,7 +144,7 @@ func TestShapeMicroVM_KeepsDeclaredContainerLimits(t *testing.T) {
Args: []string{"/app"},
Resources: &ateworkerpb.ResourceLimits{MemoryBytes: declared},
}}})
- if err := ShapeMicroVM(spec, MicroVMOptions{ActorDirs: parityActorDirs, ContainerID: "app"}); err != nil {
+ if err := ShapeMicroVM(spec, MicroVMOptions{ActorDirs: parityActorDirs, ContainerName: "app"}); err != nil {
t.Fatalf("ShapeMicroVM() = %v", err)
}
@@ -141,7 +160,7 @@ func TestShapeMicroVM_KeepsDeclaredContainerLimits(t *testing.T) {
// RAM is the real ceiling, and a cap equal to the whole guest can never bind.
func TestShapeMicroVM_LeavesUndeclaredContainerUnlimited(t *testing.T) {
spec := Build(Options{Container: &ateworkerpb.Container{ContainerSpec: &ateworkerpb.ContainerSpec{Args: []string{"/app"}}}})
- if err := ShapeMicroVM(spec, MicroVMOptions{ActorDirs: parityActorDirs, ContainerID: "app"}); err != nil {
+ if err := ShapeMicroVM(spec, MicroVMOptions{ActorDirs: parityActorDirs, ContainerName: "app"}); err != nil {
t.Fatalf("ShapeMicroVM() = %v", err)
}
diff --git a/internal/ocispec/parity_test.go b/internal/ocispec/parity_test.go
index 2f7f9d9d92..e341285761 100644
--- a/internal/ocispec/parity_test.go
+++ b/internal/ocispec/parity_test.go
@@ -80,7 +80,7 @@ func TestShapers_PreserveEveryVolumeMount(t *testing.T) {
}, {
runtime: "microvm",
shape: func(s *specs.Spec) error {
- return ShapeMicroVM(s, MicroVMOptions{ActorDirs: parityActorDirs, ContainerID: "app"})
+ return ShapeMicroVM(s, MicroVMOptions{ActorDirs: parityActorDirs, ContainerName: "app"})
},
}} {
t.Run(tc.runtime, func(t *testing.T) {
@@ -109,14 +109,14 @@ func TestShapers_PreserveEveryVolumeMount(t *testing.T) {
// ShapeMicroVM rewrites bind sources to their guest share paths.
func TestShapeMicroVM_TranslatesSourcesIntoTheShare(t *testing.T) {
spec := Build(parityOptions)
- if err := ShapeMicroVM(spec, MicroVMOptions{ActorDirs: parityActorDirs, ContainerID: "app"}); err != nil {
+ if err := ShapeMicroVM(spec, MicroVMOptions{ActorDirs: parityActorDirs, ContainerName: "app"}); err != nil {
t.Fatalf("ShapeMicroVM() = %v", err)
}
for _, tc := range []struct{ dest, wantSource string }{
{"/var/data", GuestSharedDir + "/" + ShareDurable + "/data"},
{"/run/ate", GuestSharedDir + "/" + ShareSystemInfo + "/sysinfo"},
{"/mnt/csi", GuestSharedDir + "/" + ShareCSI + "/csi"},
- {"/ate", GuestSharedDir + "/app/" + ShareVolumes + "/agent"},
+ {"/ate", GuestSharedDir + "/c_app/" + ShareVolumes + "/agent"},
} {
if got := mountFor(t, spec, tc.dest).Source; got != tc.wantSource {
t.Errorf("%s source = %q, want %q", tc.dest, got, tc.wantSource)
@@ -132,7 +132,7 @@ func TestShapeMicroVM_TranslatesSourcesIntoTheShare(t *testing.T) {
func TestShapeMicroVM_UnstagedSourceIsAnError(t *testing.T) {
spec := Build(Options{Container: &ateworkerpb.Container{ContainerSpec: &ateworkerpb.ContainerSpec{Args: []string{"/app"}}}})
spec.Mounts = append(spec.Mounts, specs.Mount{Destination: "/mnt/new", Type: "bind", Source: "/var/lib/ate/new-kind/x"})
- if err := ShapeMicroVM(spec, MicroVMOptions{ActorDirs: parityActorDirs, ContainerID: "app"}); err == nil {
+ if err := ShapeMicroVM(spec, MicroVMOptions{ActorDirs: parityActorDirs, ContainerName: "app"}); err == nil {
t.Fatal("ShapeMicroVM() = nil, want an error for a bind that is not staged into the share")
}
}
diff --git a/internal/otlprelay/client.go b/internal/otlprelay/client.go
index 55b0466e8a..db6650a72a 100644
--- a/internal/otlprelay/client.go
+++ b/internal/otlprelay/client.go
@@ -30,11 +30,11 @@ import (
// socket, to be handed to the OTLP exporters via serverboot's ExporterConn.
//
// It returns (nil, nil) when sockPath is empty or absent, which the caller reads
-// as "export directly instead". The existence check is what makes the fallback
-// deterministic at startup: grpc.NewClient is lazy, so a connection to a missing
-// socket would be created happily and only fail later, per export, with the
-// telemetry already lost. Losing spans is not worth failing worker over either,
-// hence a fallback rather than an error.
+// as "fall back to the OTLP endpoint". The existence check is what makes the
+// fallback deterministic at startup: grpc.NewClient is lazy, so a connection to
+// a missing socket would be created happily and only fail later, per export,
+// with the telemetry already lost. Losing spans is not worth failing a worker
+// over either, hence a fallback rather than an error.
//
// The connection is plaintext by design. A unix socket cannot leave the node, so
// there is no transport to protect; access is controlled by the socket's file
@@ -48,7 +48,7 @@ func Dial(ctx context.Context, sockPath string) (*grpc.ClientConn, error) {
}
if _, err := os.Stat(sockPath); err != nil {
if errors.Is(err, fs.ErrNotExist) {
- slog.WarnContext(ctx, "OTLP relay socket absent, exporting telemetry directly over the pod network",
+ slog.WarnContext(ctx, "OTLP relay socket absent, falling back to the OTLP endpoint, if one is set",
slog.String("socket", sockPath))
return nil, nil
}
diff --git a/internal/pidfd/doc.go b/internal/pidfd/doc.go
new file mode 100644
index 0000000000..65ba3d8601
--- /dev/null
+++ b/internal/pidfd/doc.go
@@ -0,0 +1,20 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package pidfd watches for processes to exit without holding a thread per
+// process, so a worker can watch thousands of them.
+//
+// It needs Linux 5.3+ for pidfd_open. Before 5.10, which added
+// PIDFD_NONBLOCK, each watch falls back to a blocking poll on its own thread.
+package pidfd
diff --git a/internal/pidfd/pidfd.go b/internal/pidfd/pidfd.go
new file mode 100644
index 0000000000..43858cf1b0
--- /dev/null
+++ b/internal/pidfd/pidfd.go
@@ -0,0 +1,108 @@
+//go:build linux
+
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package pidfd
+
+import (
+ "errors"
+ "log/slog"
+ "os"
+ "time"
+
+ "golang.org/x/sys/unix"
+)
+
+// Exited returns a channel that is closed once pid exits. pid is pinned when
+// Exited is called, so a later reuse of the number is never mistaken for it.
+// A process already gone yields a closed channel.
+func Exited(pid int) (<-chan struct{}, error) {
+ done := make(chan struct{})
+ fd, err := unix.PidfdOpen(pid, unix.PIDFD_NONBLOCK)
+ if errors.Is(err, unix.EINVAL) {
+ // PIDFD_NONBLOCK is Linux 5.10+; watch falls back to a blocking poll
+ fd, err = unix.PidfdOpen(pid, 0)
+ }
+ if errors.Is(err, unix.ESRCH) {
+ close(done)
+ return done, nil
+ }
+ if err != nil {
+ return nil, err
+ }
+ return watch(fd, done)
+}
+
+// ExitedFD is Exited for a process already pinned by pidfd. It watches a
+// duplicate, so the caller keeps pidfd and may close it at any time.
+func ExitedFD(pidfd int) (<-chan struct{}, error) {
+ fd, err := unix.FcntlInt(uintptr(pidfd), unix.F_DUPFD_CLOEXEC, 0)
+ if err != nil {
+ return nil, err
+ }
+ // the duplicate shares the file status flags, so this makes pidfd
+ // nonblocking too, which signaling it does not mind
+ if err := unix.SetNonblock(fd, true); err != nil {
+ unix.Close(fd)
+ return nil, err
+ }
+ return watch(fd, make(chan struct{}))
+}
+
+// watch closes done once the nonblocking pidfd fd turns readable, and takes
+// ownership of fd.
+func watch(fd int, done chan struct{}) (<-chan struct{}, error) {
+ // A nonblocking fd handed to os.NewFile joins the runtime's netpoller, so
+ // waiting on it parks a goroutine rather than a thread.
+ f := os.NewFile(uintptr(fd), "pidfd")
+ rc, err := f.SyscallConn()
+ if err != nil {
+ f.Close()
+ return nil, err
+ }
+ go func() {
+ defer close(done)
+ defer f.Close()
+ // The pidfd turns readable when the process exits. Check the fd itself
+ // rather than trusting a wakeup: Read drops a readiness edge that
+ // arrived before it started waiting.
+ err := rc.Read(func(fd uintptr) bool {
+ return readable(int(fd), 0)
+ })
+ if err != nil {
+ // the netpoller cannot watch this fd, so wait in a blocking poll:
+ // it holds a thread, but never reports an exit that did not happen
+ slog.Warn("pidfd: falling back to a blocking poll", slog.Any("err", err))
+ _ = rc.Control(func(fd uintptr) {
+ for !readable(int(fd), -1) {
+ time.Sleep(time.Second)
+ }
+ })
+ }
+ }()
+ return done, nil
+}
+
+// readable polls fd for POLLIN for up to timeoutMs, -1 meaning forever.
+func readable(fd, timeoutMs int) bool {
+ fds := []unix.PollFd{{Fd: int32(fd), Events: unix.POLLIN}}
+ for {
+ n, err := unix.Poll(fds, timeoutMs)
+ if err == unix.EINTR {
+ continue
+ }
+ return err == nil && n > 0
+ }
+}
diff --git a/internal/pidfd/pidfd_other.go b/internal/pidfd/pidfd_other.go
new file mode 100644
index 0000000000..cafea744cc
--- /dev/null
+++ b/internal/pidfd/pidfd_other.go
@@ -0,0 +1,29 @@
+//go:build !linux
+
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package pidfd
+
+import "errors"
+
+// Exited is not supported off Linux.
+func Exited(pid int) (<-chan struct{}, error) {
+ return nil, errors.ErrUnsupported
+}
+
+// ExitedFD is not supported off Linux.
+func ExitedFD(pidfd int) (<-chan struct{}, error) {
+ return nil, errors.ErrUnsupported
+}
diff --git a/internal/pidfd/pidfd_test.go b/internal/pidfd/pidfd_test.go
new file mode 100644
index 0000000000..7e739c3f11
--- /dev/null
+++ b/internal/pidfd/pidfd_test.go
@@ -0,0 +1,182 @@
+//go:build linux
+
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package pidfd
+
+import (
+ "os"
+ "os/exec"
+ "strconv"
+ "strings"
+ "testing"
+ "time"
+
+ "golang.org/x/sys/unix"
+)
+
+func TestExited(t *testing.T) {
+ cmd := exec.Command("sleep", "30")
+ if err := cmd.Start(); err != nil {
+ t.Fatal(err)
+ }
+ defer cmd.Wait()
+ done, err := Exited(cmd.Process.Pid)
+ if err != nil {
+ t.Fatal(err)
+ }
+ select {
+ case <-done:
+ t.Fatal("closed while the process runs")
+ case <-time.After(100 * time.Millisecond):
+ }
+ _ = cmd.Process.Kill()
+ select {
+ case <-done:
+ case <-time.After(10 * time.Second):
+ t.Fatal("not closed after the process exited")
+ }
+}
+
+func TestExitedAlreadyGone(t *testing.T) {
+ cmd := exec.Command("true")
+ if err := cmd.Run(); err != nil {
+ t.Fatal(err)
+ }
+ done, err := Exited(cmd.Process.Pid)
+ if err != nil {
+ t.Fatal(err)
+ }
+ select {
+ case <-done:
+ case <-time.After(10 * time.Second):
+ t.Fatal("not closed for a reaped process")
+ }
+}
+
+func TestExitedFD(t *testing.T) {
+ cmd := exec.Command("sleep", "30")
+ if err := cmd.Start(); err != nil {
+ t.Fatal(err)
+ }
+ defer cmd.Wait()
+ fd, err := unix.PidfdOpen(cmd.Process.Pid, 0)
+ if err != nil {
+ t.Fatal(err)
+ }
+ done, err := ExitedFD(fd)
+ if err != nil {
+ t.Fatal(err)
+ }
+ // the caller's pidfd is its own to close
+ unix.Close(fd)
+ _ = cmd.Process.Kill()
+ select {
+ case <-done:
+ case <-time.After(10 * time.Second):
+ t.Fatal("not closed after the process exited")
+ }
+}
+
+// A blocking pidfd cannot join the netpoller. The watch must still wait for
+// the exit, not report one straight away.
+func TestWatchFallsBackToABlockingPoll(t *testing.T) {
+ cmd := exec.Command("sleep", "30")
+ if err := cmd.Start(); err != nil {
+ t.Fatal(err)
+ }
+ defer cmd.Wait()
+ fd, err := unix.PidfdOpen(cmd.Process.Pid, 0)
+ if err != nil {
+ t.Fatal(err)
+ }
+ done, err := watch(fd, make(chan struct{}))
+ if err != nil {
+ t.Fatal(err)
+ }
+ select {
+ case <-done:
+ t.Fatal("closed while the process runs")
+ case <-time.After(200 * time.Millisecond):
+ }
+ _ = cmd.Process.Kill()
+ select {
+ case <-done:
+ case <-time.After(10 * time.Second):
+ t.Fatal("not closed after the process exited")
+ }
+}
+
+// A process that exits while the watcher is starting must still be seen.
+func TestExitedRacingExit(t *testing.T) {
+ for range 200 {
+ cmd := exec.Command("true")
+ if err := cmd.Start(); err != nil {
+ t.Fatal(err)
+ }
+ done, err := Exited(cmd.Process.Pid)
+ if err != nil {
+ t.Fatal(err)
+ }
+ select {
+ case <-done:
+ case <-time.After(10 * time.Second):
+ t.Fatal("exit missed")
+ }
+ _ = cmd.Wait()
+ }
+}
+
+// Watching many processes must not cost a thread each.
+func TestExitedHoldsNoThreads(t *testing.T) {
+ var cmds []*exec.Cmd
+ defer func() {
+ for _, c := range cmds {
+ _ = c.Process.Kill()
+ _ = c.Wait()
+ }
+ }()
+ before := threads(t)
+ for range 200 {
+ c := exec.Command("sleep", "30")
+ if err := c.Start(); err != nil {
+ t.Fatal(err)
+ }
+ cmds = append(cmds, c)
+ if _, err := Exited(c.Process.Pid); err != nil {
+ t.Fatal(err)
+ }
+ }
+ time.Sleep(100 * time.Millisecond)
+ if after := threads(t); after-before > 20 {
+ t.Errorf("threads grew from %d to %d watching 200 processes", before, after)
+ }
+}
+
+func threads(t *testing.T) int {
+ t.Helper()
+ b, err := os.ReadFile("/proc/self/status")
+ if err != nil {
+ t.Fatal(err)
+ }
+ for _, line := range strings.Split(string(b), "\n") {
+ if v, ok := strings.CutPrefix(line, "Threads:"); ok {
+ n, _ := strconv.Atoi(strings.TrimSpace(v))
+ return n
+ }
+ }
+ t.Fatal("no Threads line")
+ return 0
+}
diff --git a/internal/preview/preview.go b/internal/preview/preview.go
index 968315596c..6a6ceb96e2 100644
--- a/internal/preview/preview.go
+++ b/internal/preview/preview.go
@@ -41,6 +41,11 @@ type Gate string
const (
// GateExternalVolumes controls whether external volumes are supported.
GateExternalVolumes Gate = "ExternalVolumes"
+
+ // GatePerTemplateSnapshotLocation controls whether an ActorTemplate's
+ // snapshotConfig.storage_location is honored. When disabled, snapshots
+ // are stored under ate-api-server's --snapshot-location.
+ GatePerTemplateSnapshotLocation Gate = "PerTemplateSnapshotLocation"
)
// All is the pseudo-gate name that enables every known gate.
@@ -50,6 +55,7 @@ const All = "ALL"
// default and enabled via the --preview flag.
var realKnownGates = []Gate{
GateExternalVolumes,
+ GatePerTemplateSnapshotLocation,
}
var globalKnownGates []Gate // all known gates
diff --git a/internal/proto/ateenv/v1alpha/guest.pb.go b/internal/proto/ateenv/v1alpha/guest.pb.go
new file mode 100644
index 0000000000..eab5979d55
--- /dev/null
+++ b/internal/proto/ateenv/v1alpha/guest.pb.go
@@ -0,0 +1,1248 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Copied from github.com/agent-substrate/env/proto/ateenv/v1alpha/guest.proto
+// at commit 1c08a7a578afd896373511a83beb242aec1425c0 (matching the default
+// ATE_ENV_IMAGE digest in benchmarking/workloads/deploy.sh and documented in
+// benchmarking/workloads/manifests/ate-env-template.yaml.tmpl) to avoid adding
+// a cross-repo Go module dependency on agent-substrate/env.
+//
+// Minimal Protocol Buffers definition for the Agent Substrate Guest Data Plane.
+// Exposes asynchronous process execution with streaming I/O and signals, and
+// streaming file I/O.
+
+// Code generated by protoc-gen-go. DO NOT EDIT.
+// versions:
+// protoc-gen-go v1.36.11-devel
+// protoc v4.25.3
+// source: guest.proto
+
+package ateenvv1alpha
+
+import (
+ protoreflect "google.golang.org/protobuf/reflect/protoreflect"
+ protoimpl "google.golang.org/protobuf/runtime/protoimpl"
+ durationpb "google.golang.org/protobuf/types/known/durationpb"
+ timestamppb "google.golang.org/protobuf/types/known/timestamppb"
+ reflect "reflect"
+ sync "sync"
+ unsafe "unsafe"
+)
+
+const (
+ // Verify that this generated code is sufficiently up-to-date.
+ _ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion)
+ // Verify that runtime/protoimpl is sufficiently up-to-date.
+ _ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20)
+)
+
+// Lifecycle state of a process.
+type ProcessState int32
+
+const (
+ ProcessState_PROCESS_STATE_UNSPECIFIED ProcessState = 0
+ // The process is running (or stopped by SIGSTOP) and has not been reaped.
+ ProcessState_PROCESS_STATE_RUNNING ProcessState = 1
+ // The process has exited; see Process.exit_code.
+ ProcessState_PROCESS_STATE_EXITED ProcessState = 2
+)
+
+// Enum value maps for ProcessState.
+var (
+ ProcessState_name = map[int32]string{
+ 0: "PROCESS_STATE_UNSPECIFIED",
+ 1: "PROCESS_STATE_RUNNING",
+ 2: "PROCESS_STATE_EXITED",
+ }
+ ProcessState_value = map[string]int32{
+ "PROCESS_STATE_UNSPECIFIED": 0,
+ "PROCESS_STATE_RUNNING": 1,
+ "PROCESS_STATE_EXITED": 2,
+ }
+)
+
+func (x ProcessState) Enum() *ProcessState {
+ p := new(ProcessState)
+ *p = x
+ return p
+}
+
+func (x ProcessState) String() string {
+ return protoimpl.X.EnumStringOf(x.Descriptor(), protoreflect.EnumNumber(x))
+}
+
+func (ProcessState) Descriptor() protoreflect.EnumDescriptor {
+ return file_guest_proto_enumTypes[0].Descriptor()
+}
+
+func (ProcessState) Type() protoreflect.EnumType {
+ return &file_guest_proto_enumTypes[0]
+}
+
+func (x ProcessState) Number() protoreflect.EnumNumber {
+ return protoreflect.EnumNumber(x)
+}
+
+// Deprecated: Use ProcessState.Descriptor instead.
+func (ProcessState) EnumDescriptor() ([]byte, []int) {
+ return file_guest_proto_rawDescGZIP(), []int{0}
+}
+
+// POSIX signals that can be sent to a process. Values match the Linux signal
+// numbers on x86/arm.
+type Signal int32
+
+const (
+ Signal_SIGNAL_UNSPECIFIED Signal = 0
+ Signal_SIGNAL_HUP Signal = 1
+ Signal_SIGNAL_INT Signal = 2
+ Signal_SIGNAL_QUIT Signal = 3
+ Signal_SIGNAL_ILL Signal = 4
+ Signal_SIGNAL_TRAP Signal = 5
+ Signal_SIGNAL_ABRT Signal = 6
+ Signal_SIGNAL_BUS Signal = 7
+ Signal_SIGNAL_FPE Signal = 8
+ Signal_SIGNAL_KILL Signal = 9
+ Signal_SIGNAL_USR1 Signal = 10
+ Signal_SIGNAL_SEGV Signal = 11
+ Signal_SIGNAL_USR2 Signal = 12
+ Signal_SIGNAL_PIPE Signal = 13
+ Signal_SIGNAL_ALRM Signal = 14
+ Signal_SIGNAL_TERM Signal = 15
+ Signal_SIGNAL_CHLD Signal = 17
+ Signal_SIGNAL_CONT Signal = 18
+ Signal_SIGNAL_STOP Signal = 19
+ Signal_SIGNAL_TSTP Signal = 20
+ Signal_SIGNAL_TTIN Signal = 21
+ Signal_SIGNAL_TTOU Signal = 22
+ Signal_SIGNAL_URG Signal = 23
+ Signal_SIGNAL_XCPU Signal = 24
+ Signal_SIGNAL_XFSZ Signal = 25
+ Signal_SIGNAL_VTALRM Signal = 26
+ Signal_SIGNAL_PROF Signal = 27
+ Signal_SIGNAL_WINCH Signal = 28
+ Signal_SIGNAL_IO Signal = 29
+ Signal_SIGNAL_SYS Signal = 31
+)
+
+// Enum value maps for Signal.
+var (
+ Signal_name = map[int32]string{
+ 0: "SIGNAL_UNSPECIFIED",
+ 1: "SIGNAL_HUP",
+ 2: "SIGNAL_INT",
+ 3: "SIGNAL_QUIT",
+ 4: "SIGNAL_ILL",
+ 5: "SIGNAL_TRAP",
+ 6: "SIGNAL_ABRT",
+ 7: "SIGNAL_BUS",
+ 8: "SIGNAL_FPE",
+ 9: "SIGNAL_KILL",
+ 10: "SIGNAL_USR1",
+ 11: "SIGNAL_SEGV",
+ 12: "SIGNAL_USR2",
+ 13: "SIGNAL_PIPE",
+ 14: "SIGNAL_ALRM",
+ 15: "SIGNAL_TERM",
+ 17: "SIGNAL_CHLD",
+ 18: "SIGNAL_CONT",
+ 19: "SIGNAL_STOP",
+ 20: "SIGNAL_TSTP",
+ 21: "SIGNAL_TTIN",
+ 22: "SIGNAL_TTOU",
+ 23: "SIGNAL_URG",
+ 24: "SIGNAL_XCPU",
+ 25: "SIGNAL_XFSZ",
+ 26: "SIGNAL_VTALRM",
+ 27: "SIGNAL_PROF",
+ 28: "SIGNAL_WINCH",
+ 29: "SIGNAL_IO",
+ 31: "SIGNAL_SYS",
+ }
+ Signal_value = map[string]int32{
+ "SIGNAL_UNSPECIFIED": 0,
+ "SIGNAL_HUP": 1,
+ "SIGNAL_INT": 2,
+ "SIGNAL_QUIT": 3,
+ "SIGNAL_ILL": 4,
+ "SIGNAL_TRAP": 5,
+ "SIGNAL_ABRT": 6,
+ "SIGNAL_BUS": 7,
+ "SIGNAL_FPE": 8,
+ "SIGNAL_KILL": 9,
+ "SIGNAL_USR1": 10,
+ "SIGNAL_SEGV": 11,
+ "SIGNAL_USR2": 12,
+ "SIGNAL_PIPE": 13,
+ "SIGNAL_ALRM": 14,
+ "SIGNAL_TERM": 15,
+ "SIGNAL_CHLD": 17,
+ "SIGNAL_CONT": 18,
+ "SIGNAL_STOP": 19,
+ "SIGNAL_TSTP": 20,
+ "SIGNAL_TTIN": 21,
+ "SIGNAL_TTOU": 22,
+ "SIGNAL_URG": 23,
+ "SIGNAL_XCPU": 24,
+ "SIGNAL_XFSZ": 25,
+ "SIGNAL_VTALRM": 26,
+ "SIGNAL_PROF": 27,
+ "SIGNAL_WINCH": 28,
+ "SIGNAL_IO": 29,
+ "SIGNAL_SYS": 31,
+ }
+)
+
+func (x Signal) Enum() *Signal {
+ p := new(Signal)
+ *p = x
+ return p
+}
+
+func (x Signal) String() string {
+ return protoimpl.X.EnumStringOf(x.Descriptor(), protoreflect.EnumNumber(x))
+}
+
+func (Signal) Descriptor() protoreflect.EnumDescriptor {
+ return file_guest_proto_enumTypes[1].Descriptor()
+}
+
+func (Signal) Type() protoreflect.EnumType {
+ return &file_guest_proto_enumTypes[1]
+}
+
+func (x Signal) Number() protoreflect.EnumNumber {
+ return protoreflect.EnumNumber(x)
+}
+
+// Deprecated: Use Signal.Descriptor instead.
+func (Signal) EnumDescriptor() ([]byte, []int) {
+ return file_guest_proto_rawDescGZIP(), []int{1}
+}
+
+// The Process resource: identity, lifecycle state, and exit status.
+type Process struct {
+ state protoimpl.MessageState `protogen:"open.v1"`
+ // Unique process identifier assigned by the guest.
+ ProcessId string `protobuf:"bytes,1,opt,name=process_id,json=processId,proto3" json:"process_id,omitempty"`
+ // Command binary and arguments the process was started with.
+ Command []string `protobuf:"bytes,2,rep,name=command,proto3" json:"command,omitempty"`
+ // Operating system PID inside the container.
+ Pid int32 `protobuf:"varint,3,opt,name=pid,proto3" json:"pid,omitempty"`
+ // Current lifecycle state.
+ State ProcessState `protobuf:"varint,4,opt,name=state,proto3,enum=ateenv.v1alpha.ProcessState" json:"state,omitempty"`
+ // Exit status, valid once state is EXITED: the process's exit code, or
+ // 128 + signal number if it was terminated by a signal (e.g. 137 for
+ // SIGKILL, 143 for SIGTERM).
+ ExitCode int32 `protobuf:"varint,5,opt,name=exit_code,json=exitCode,proto3" json:"exit_code,omitempty"`
+ // Timestamp when the process started.
+ StartedAt *timestamppb.Timestamp `protobuf:"bytes,6,opt,name=started_at,json=startedAt,proto3" json:"started_at,omitempty"`
+ // Timestamp when the process exited. Unset while RUNNING.
+ FinishedAt *timestamppb.Timestamp `protobuf:"bytes,7,opt,name=finished_at,json=finishedAt,proto3" json:"finished_at,omitempty"`
+ unknownFields protoimpl.UnknownFields
+ sizeCache protoimpl.SizeCache
+}
+
+func (x *Process) Reset() {
+ *x = Process{}
+ mi := &file_guest_proto_msgTypes[0]
+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+ ms.StoreMessageInfo(mi)
+}
+
+func (x *Process) String() string {
+ return protoimpl.X.MessageStringOf(x)
+}
+
+func (*Process) ProtoMessage() {}
+
+func (x *Process) ProtoReflect() protoreflect.Message {
+ mi := &file_guest_proto_msgTypes[0]
+ if x != nil {
+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+ if ms.LoadMessageInfo() == nil {
+ ms.StoreMessageInfo(mi)
+ }
+ return ms
+ }
+ return mi.MessageOf(x)
+}
+
+// Deprecated: Use Process.ProtoReflect.Descriptor instead.
+func (*Process) Descriptor() ([]byte, []int) {
+ return file_guest_proto_rawDescGZIP(), []int{0}
+}
+
+func (x *Process) GetProcessId() string {
+ if x != nil {
+ return x.ProcessId
+ }
+ return ""
+}
+
+func (x *Process) GetCommand() []string {
+ if x != nil {
+ return x.Command
+ }
+ return nil
+}
+
+func (x *Process) GetPid() int32 {
+ if x != nil {
+ return x.Pid
+ }
+ return 0
+}
+
+func (x *Process) GetState() ProcessState {
+ if x != nil {
+ return x.State
+ }
+ return ProcessState_PROCESS_STATE_UNSPECIFIED
+}
+
+func (x *Process) GetExitCode() int32 {
+ if x != nil {
+ return x.ExitCode
+ }
+ return 0
+}
+
+func (x *Process) GetStartedAt() *timestamppb.Timestamp {
+ if x != nil {
+ return x.StartedAt
+ }
+ return nil
+}
+
+func (x *Process) GetFinishedAt() *timestamppb.Timestamp {
+ if x != nil {
+ return x.FinishedAt
+ }
+ return nil
+}
+
+// Request to start a process.
+type StartProcessRequest struct {
+ state protoimpl.MessageState `protogen:"open.v1"`
+ // Command binary and arguments to execute (e.g. ["pytest", "tests/"] or ["sh", "-c", "ls -la"]).
+ Command []string `protobuf:"bytes,1,rep,name=command,proto3" json:"command,omitempty"`
+ // Working directory inside the container (defaults to the guest workspace).
+ Cwd string `protobuf:"bytes,2,opt,name=cwd,proto3" json:"cwd,omitempty"`
+ // Environment variables to set for the process, on top of the guest's environment.
+ Env map[string]string `protobuf:"bytes,3,rep,name=env,proto3" json:"env,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"`
+ // If true, stdin is an open pipe fed by WriteProcessInput. If false, stdin
+ // reads as empty (/dev/null) and WriteProcessInput is rejected.
+ Stdin bool `protobuf:"varint,4,opt,name=stdin,proto3" json:"stdin,omitempty"`
+ // Maximum wall-clock run time. On expiry the process group receives SIGKILL.
+ // Unset or zero applies the guest's default timeout.
+ Timeout *durationpb.Duration `protobuf:"bytes,5,opt,name=timeout,proto3" json:"timeout,omitempty"`
+ unknownFields protoimpl.UnknownFields
+ sizeCache protoimpl.SizeCache
+}
+
+func (x *StartProcessRequest) Reset() {
+ *x = StartProcessRequest{}
+ mi := &file_guest_proto_msgTypes[1]
+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+ ms.StoreMessageInfo(mi)
+}
+
+func (x *StartProcessRequest) String() string {
+ return protoimpl.X.MessageStringOf(x)
+}
+
+func (*StartProcessRequest) ProtoMessage() {}
+
+func (x *StartProcessRequest) ProtoReflect() protoreflect.Message {
+ mi := &file_guest_proto_msgTypes[1]
+ if x != nil {
+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+ if ms.LoadMessageInfo() == nil {
+ ms.StoreMessageInfo(mi)
+ }
+ return ms
+ }
+ return mi.MessageOf(x)
+}
+
+// Deprecated: Use StartProcessRequest.ProtoReflect.Descriptor instead.
+func (*StartProcessRequest) Descriptor() ([]byte, []int) {
+ return file_guest_proto_rawDescGZIP(), []int{1}
+}
+
+func (x *StartProcessRequest) GetCommand() []string {
+ if x != nil {
+ return x.Command
+ }
+ return nil
+}
+
+func (x *StartProcessRequest) GetCwd() string {
+ if x != nil {
+ return x.Cwd
+ }
+ return ""
+}
+
+func (x *StartProcessRequest) GetEnv() map[string]string {
+ if x != nil {
+ return x.Env
+ }
+ return nil
+}
+
+func (x *StartProcessRequest) GetStdin() bool {
+ if x != nil {
+ return x.Stdin
+ }
+ return false
+}
+
+func (x *StartProcessRequest) GetTimeout() *durationpb.Duration {
+ if x != nil {
+ return x.Timeout
+ }
+ return nil
+}
+
+// Request to get a Process.
+type GetProcessRequest struct {
+ state protoimpl.MessageState `protogen:"open.v1"`
+ // Identifier of the process to inspect.
+ ProcessId string `protobuf:"bytes,1,opt,name=process_id,json=processId,proto3" json:"process_id,omitempty"`
+ unknownFields protoimpl.UnknownFields
+ sizeCache protoimpl.SizeCache
+}
+
+func (x *GetProcessRequest) Reset() {
+ *x = GetProcessRequest{}
+ mi := &file_guest_proto_msgTypes[2]
+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+ ms.StoreMessageInfo(mi)
+}
+
+func (x *GetProcessRequest) String() string {
+ return protoimpl.X.MessageStringOf(x)
+}
+
+func (*GetProcessRequest) ProtoMessage() {}
+
+func (x *GetProcessRequest) ProtoReflect() protoreflect.Message {
+ mi := &file_guest_proto_msgTypes[2]
+ if x != nil {
+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+ if ms.LoadMessageInfo() == nil {
+ ms.StoreMessageInfo(mi)
+ }
+ return ms
+ }
+ return mi.MessageOf(x)
+}
+
+// Deprecated: Use GetProcessRequest.ProtoReflect.Descriptor instead.
+func (*GetProcessRequest) Descriptor() ([]byte, []int) {
+ return file_guest_proto_rawDescGZIP(), []int{2}
+}
+
+func (x *GetProcessRequest) GetProcessId() string {
+ if x != nil {
+ return x.ProcessId
+ }
+ return ""
+}
+
+// Request to stream output from a process.
+type StreamProcessOutputRequest struct {
+ state protoimpl.MessageState `protogen:"open.v1"`
+ // Identifier of the process to stream output from.
+ ProcessId string `protobuf:"bytes,1,opt,name=process_id,json=processId,proto3" json:"process_id,omitempty"`
+ // Byte offset to start reading stdout from (defaults to 0 / beginning).
+ // Bytes before the offset are never delivered, so an offset past the end
+ // of the spool suppresses stdout entirely.
+ StdoutOffset int64 `protobuf:"varint,2,opt,name=stdout_offset,json=stdoutOffset,proto3" json:"stdout_offset,omitempty"`
+ // Byte offset to start reading stderr from (defaults to 0 / beginning).
+ // Same semantics as stdout_offset.
+ StderrOffset int64 `protobuf:"varint,3,opt,name=stderr_offset,json=stderrOffset,proto3" json:"stderr_offset,omitempty"`
+ // If true, the stream stays open and delivers new output until the process
+ // exits. If false, the output spooled so far is returned and the stream ends.
+ Follow bool `protobuf:"varint,4,opt,name=follow,proto3" json:"follow,omitempty"`
+ unknownFields protoimpl.UnknownFields
+ sizeCache protoimpl.SizeCache
+}
+
+func (x *StreamProcessOutputRequest) Reset() {
+ *x = StreamProcessOutputRequest{}
+ mi := &file_guest_proto_msgTypes[3]
+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+ ms.StoreMessageInfo(mi)
+}
+
+func (x *StreamProcessOutputRequest) String() string {
+ return protoimpl.X.MessageStringOf(x)
+}
+
+func (*StreamProcessOutputRequest) ProtoMessage() {}
+
+func (x *StreamProcessOutputRequest) ProtoReflect() protoreflect.Message {
+ mi := &file_guest_proto_msgTypes[3]
+ if x != nil {
+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+ if ms.LoadMessageInfo() == nil {
+ ms.StoreMessageInfo(mi)
+ }
+ return ms
+ }
+ return mi.MessageOf(x)
+}
+
+// Deprecated: Use StreamProcessOutputRequest.ProtoReflect.Descriptor instead.
+func (*StreamProcessOutputRequest) Descriptor() ([]byte, []int) {
+ return file_guest_proto_rawDescGZIP(), []int{3}
+}
+
+func (x *StreamProcessOutputRequest) GetProcessId() string {
+ if x != nil {
+ return x.ProcessId
+ }
+ return ""
+}
+
+func (x *StreamProcessOutputRequest) GetStdoutOffset() int64 {
+ if x != nil {
+ return x.StdoutOffset
+ }
+ return 0
+}
+
+func (x *StreamProcessOutputRequest) GetStderrOffset() int64 {
+ if x != nil {
+ return x.StderrOffset
+ }
+ return 0
+}
+
+func (x *StreamProcessOutputRequest) GetFollow() bool {
+ if x != nil {
+ return x.Follow
+ }
+ return false
+}
+
+// One message on a process output stream. Exactly one field is set.
+type ProcessOutput struct {
+ state protoimpl.MessageState `protogen:"open.v1"`
+ // Types that are valid to be assigned to Output:
+ //
+ // *ProcessOutput_Stdout
+ // *ProcessOutput_Stderr
+ // *ProcessOutput_Exit
+ Output isProcessOutput_Output `protobuf_oneof:"output"`
+ unknownFields protoimpl.UnknownFields
+ sizeCache protoimpl.SizeCache
+}
+
+func (x *ProcessOutput) Reset() {
+ *x = ProcessOutput{}
+ mi := &file_guest_proto_msgTypes[4]
+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+ ms.StoreMessageInfo(mi)
+}
+
+func (x *ProcessOutput) String() string {
+ return protoimpl.X.MessageStringOf(x)
+}
+
+func (*ProcessOutput) ProtoMessage() {}
+
+func (x *ProcessOutput) ProtoReflect() protoreflect.Message {
+ mi := &file_guest_proto_msgTypes[4]
+ if x != nil {
+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+ if ms.LoadMessageInfo() == nil {
+ ms.StoreMessageInfo(mi)
+ }
+ return ms
+ }
+ return mi.MessageOf(x)
+}
+
+// Deprecated: Use ProcessOutput.ProtoReflect.Descriptor instead.
+func (*ProcessOutput) Descriptor() ([]byte, []int) {
+ return file_guest_proto_rawDescGZIP(), []int{4}
+}
+
+func (x *ProcessOutput) GetOutput() isProcessOutput_Output {
+ if x != nil {
+ return x.Output
+ }
+ return nil
+}
+
+func (x *ProcessOutput) GetStdout() []byte {
+ if x != nil {
+ if x, ok := x.Output.(*ProcessOutput_Stdout); ok {
+ return x.Stdout
+ }
+ }
+ return nil
+}
+
+func (x *ProcessOutput) GetStderr() []byte {
+ if x != nil {
+ if x, ok := x.Output.(*ProcessOutput_Stderr); ok {
+ return x.Stderr
+ }
+ }
+ return nil
+}
+
+func (x *ProcessOutput) GetExit() *Process {
+ if x != nil {
+ if x, ok := x.Output.(*ProcessOutput_Exit); ok {
+ return x.Exit
+ }
+ }
+ return nil
+}
+
+type isProcessOutput_Output interface {
+ isProcessOutput_Output()
+}
+
+type ProcessOutput_Stdout struct {
+ // A chunk of standard output.
+ Stdout []byte `protobuf:"bytes,1,opt,name=stdout,proto3,oneof"`
+}
+
+type ProcessOutput_Stderr struct {
+ // A chunk of standard error.
+ Stderr []byte `protobuf:"bytes,2,opt,name=stderr,proto3,oneof"`
+}
+
+type ProcessOutput_Exit struct {
+ // Final message: the process has exited and all output has been delivered.
+ // Absent if the stream ends while the process is still running (follow=false).
+ Exit *Process `protobuf:"bytes,3,opt,name=exit,proto3,oneof"`
+}
+
+func (*ProcessOutput_Stdout) isProcessOutput_Output() {}
+
+func (*ProcessOutput_Stderr) isProcessOutput_Output() {}
+
+func (*ProcessOutput_Exit) isProcessOutput_Output() {}
+
+// One chunk of standard input for a process.
+type WriteProcessInputRequest struct {
+ state protoimpl.MessageState `protogen:"open.v1"`
+ // Identifier of the target process. Required on the first message; ignored afterwards.
+ ProcessId string `protobuf:"bytes,1,opt,name=process_id,json=processId,proto3" json:"process_id,omitempty"`
+ // Bytes to write to stdin. May be empty, e.g. to only close stdin.
+ Data []byte `protobuf:"bytes,2,opt,name=data,proto3" json:"data,omitempty"`
+ // If true, stdin is closed (EOF) after data is written. No further input is accepted.
+ Close bool `protobuf:"varint,3,opt,name=close,proto3" json:"close,omitempty"`
+ unknownFields protoimpl.UnknownFields
+ sizeCache protoimpl.SizeCache
+}
+
+func (x *WriteProcessInputRequest) Reset() {
+ *x = WriteProcessInputRequest{}
+ mi := &file_guest_proto_msgTypes[5]
+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+ ms.StoreMessageInfo(mi)
+}
+
+func (x *WriteProcessInputRequest) String() string {
+ return protoimpl.X.MessageStringOf(x)
+}
+
+func (*WriteProcessInputRequest) ProtoMessage() {}
+
+func (x *WriteProcessInputRequest) ProtoReflect() protoreflect.Message {
+ mi := &file_guest_proto_msgTypes[5]
+ if x != nil {
+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+ if ms.LoadMessageInfo() == nil {
+ ms.StoreMessageInfo(mi)
+ }
+ return ms
+ }
+ return mi.MessageOf(x)
+}
+
+// Deprecated: Use WriteProcessInputRequest.ProtoReflect.Descriptor instead.
+func (*WriteProcessInputRequest) Descriptor() ([]byte, []int) {
+ return file_guest_proto_rawDescGZIP(), []int{5}
+}
+
+func (x *WriteProcessInputRequest) GetProcessId() string {
+ if x != nil {
+ return x.ProcessId
+ }
+ return ""
+}
+
+func (x *WriteProcessInputRequest) GetData() []byte {
+ if x != nil {
+ return x.Data
+ }
+ return nil
+}
+
+func (x *WriteProcessInputRequest) GetClose() bool {
+ if x != nil {
+ return x.Close
+ }
+ return false
+}
+
+// Response confirming the input write.
+type WriteProcessInputResponse struct {
+ state protoimpl.MessageState `protogen:"open.v1"`
+ // Total number of bytes written to stdin across all messages in the stream.
+ BytesWritten int64 `protobuf:"varint,1,opt,name=bytes_written,json=bytesWritten,proto3" json:"bytes_written,omitempty"`
+ unknownFields protoimpl.UnknownFields
+ sizeCache protoimpl.SizeCache
+}
+
+func (x *WriteProcessInputResponse) Reset() {
+ *x = WriteProcessInputResponse{}
+ mi := &file_guest_proto_msgTypes[6]
+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+ ms.StoreMessageInfo(mi)
+}
+
+func (x *WriteProcessInputResponse) String() string {
+ return protoimpl.X.MessageStringOf(x)
+}
+
+func (*WriteProcessInputResponse) ProtoMessage() {}
+
+func (x *WriteProcessInputResponse) ProtoReflect() protoreflect.Message {
+ mi := &file_guest_proto_msgTypes[6]
+ if x != nil {
+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+ if ms.LoadMessageInfo() == nil {
+ ms.StoreMessageInfo(mi)
+ }
+ return ms
+ }
+ return mi.MessageOf(x)
+}
+
+// Deprecated: Use WriteProcessInputResponse.ProtoReflect.Descriptor instead.
+func (*WriteProcessInputResponse) Descriptor() ([]byte, []int) {
+ return file_guest_proto_rawDescGZIP(), []int{6}
+}
+
+func (x *WriteProcessInputResponse) GetBytesWritten() int64 {
+ if x != nil {
+ return x.BytesWritten
+ }
+ return 0
+}
+
+// Request to send a signal to a process.
+type SignalProcessRequest struct {
+ state protoimpl.MessageState `protogen:"open.v1"`
+ // Identifier of the target process.
+ ProcessId string `protobuf:"bytes,1,opt,name=process_id,json=processId,proto3" json:"process_id,omitempty"`
+ // Signal to deliver to the process group.
+ Signal Signal `protobuf:"varint,2,opt,name=signal,proto3,enum=ateenv.v1alpha.Signal" json:"signal,omitempty"`
+ unknownFields protoimpl.UnknownFields
+ sizeCache protoimpl.SizeCache
+}
+
+func (x *SignalProcessRequest) Reset() {
+ *x = SignalProcessRequest{}
+ mi := &file_guest_proto_msgTypes[7]
+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+ ms.StoreMessageInfo(mi)
+}
+
+func (x *SignalProcessRequest) String() string {
+ return protoimpl.X.MessageStringOf(x)
+}
+
+func (*SignalProcessRequest) ProtoMessage() {}
+
+func (x *SignalProcessRequest) ProtoReflect() protoreflect.Message {
+ mi := &file_guest_proto_msgTypes[7]
+ if x != nil {
+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+ if ms.LoadMessageInfo() == nil {
+ ms.StoreMessageInfo(mi)
+ }
+ return ms
+ }
+ return mi.MessageOf(x)
+}
+
+// Deprecated: Use SignalProcessRequest.ProtoReflect.Descriptor instead.
+func (*SignalProcessRequest) Descriptor() ([]byte, []int) {
+ return file_guest_proto_rawDescGZIP(), []int{7}
+}
+
+func (x *SignalProcessRequest) GetProcessId() string {
+ if x != nil {
+ return x.ProcessId
+ }
+ return ""
+}
+
+func (x *SignalProcessRequest) GetSignal() Signal {
+ if x != nil {
+ return x.Signal
+ }
+ return Signal_SIGNAL_UNSPECIFIED
+}
+
+// Request to read a file from the container.
+type ReadFileRequest struct {
+ state protoimpl.MessageState `protogen:"open.v1"`
+ // Absolute or workspace-relative path of the file to read.
+ Path string `protobuf:"bytes,1,opt,name=path,proto3" json:"path,omitempty"`
+ // Optional POSIX file mode permission (e.g. 0644). When set and the file
+ // does not exist, it is created empty with this mode (parent directories
+ // included) and the stream ends without chunks. When zero, a missing file
+ // is a NOT_FOUND error. Existing files are never modified.
+ Mode uint32 `protobuf:"varint,2,opt,name=mode,proto3" json:"mode,omitempty"`
+ unknownFields protoimpl.UnknownFields
+ sizeCache protoimpl.SizeCache
+}
+
+func (x *ReadFileRequest) Reset() {
+ *x = ReadFileRequest{}
+ mi := &file_guest_proto_msgTypes[8]
+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+ ms.StoreMessageInfo(mi)
+}
+
+func (x *ReadFileRequest) String() string {
+ return protoimpl.X.MessageStringOf(x)
+}
+
+func (*ReadFileRequest) ProtoMessage() {}
+
+func (x *ReadFileRequest) ProtoReflect() protoreflect.Message {
+ mi := &file_guest_proto_msgTypes[8]
+ if x != nil {
+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+ if ms.LoadMessageInfo() == nil {
+ ms.StoreMessageInfo(mi)
+ }
+ return ms
+ }
+ return mi.MessageOf(x)
+}
+
+// Deprecated: Use ReadFileRequest.ProtoReflect.Descriptor instead.
+func (*ReadFileRequest) Descriptor() ([]byte, []int) {
+ return file_guest_proto_rawDescGZIP(), []int{8}
+}
+
+func (x *ReadFileRequest) GetPath() string {
+ if x != nil {
+ return x.Path
+ }
+ return ""
+}
+
+func (x *ReadFileRequest) GetMode() uint32 {
+ if x != nil {
+ return x.Mode
+ }
+ return 0
+}
+
+// Streamed chunk of file data.
+type ReadFileResponse struct {
+ state protoimpl.MessageState `protogen:"open.v1"`
+ // Raw binary or text chunk of the file.
+ Chunk []byte `protobuf:"bytes,1,opt,name=chunk,proto3" json:"chunk,omitempty"`
+ unknownFields protoimpl.UnknownFields
+ sizeCache protoimpl.SizeCache
+}
+
+func (x *ReadFileResponse) Reset() {
+ *x = ReadFileResponse{}
+ mi := &file_guest_proto_msgTypes[9]
+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+ ms.StoreMessageInfo(mi)
+}
+
+func (x *ReadFileResponse) String() string {
+ return protoimpl.X.MessageStringOf(x)
+}
+
+func (*ReadFileResponse) ProtoMessage() {}
+
+func (x *ReadFileResponse) ProtoReflect() protoreflect.Message {
+ mi := &file_guest_proto_msgTypes[9]
+ if x != nil {
+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+ if ms.LoadMessageInfo() == nil {
+ ms.StoreMessageInfo(mi)
+ }
+ return ms
+ }
+ return mi.MessageOf(x)
+}
+
+// Deprecated: Use ReadFileResponse.ProtoReflect.Descriptor instead.
+func (*ReadFileResponse) Descriptor() ([]byte, []int) {
+ return file_guest_proto_rawDescGZIP(), []int{9}
+}
+
+func (x *ReadFileResponse) GetChunk() []byte {
+ if x != nil {
+ return x.Chunk
+ }
+ return nil
+}
+
+// Streamed request chunk for writing data to a file. path, mode and
+// seek_offset are read from the first message only and ignored on all
+// subsequent messages; every message may carry a chunk.
+type WriteFileRequest struct {
+ state protoimpl.MessageState `protogen:"open.v1"`
+ // Absolute or workspace-relative path of the file to write. Required on the
+ // first message.
+ Path string `protobuf:"bytes,1,opt,name=path,proto3" json:"path,omitempty"`
+ // Optional POSIX file mode permission (e.g. 0644 or 0755), applied when the
+ // file is created.
+ Mode uint32 `protobuf:"varint,2,opt,name=mode,proto3" json:"mode,omitempty"`
+ // Byte offset at which writing starts. Zero, the default, truncates the
+ // file and rewrites it from the beginning. A positive offset keeps existing
+ // content, begins writing at the offset (extending the file with zero bytes
+ // if it is shorter), and leaves bytes after the written range in place.
+ SeekOffset int64 `protobuf:"varint,3,opt,name=seek_offset,json=seekOffset,proto3" json:"seek_offset,omitempty"`
+ // Raw binary or text chunk to write to the file.
+ Chunk []byte `protobuf:"bytes,4,opt,name=chunk,proto3" json:"chunk,omitempty"`
+ unknownFields protoimpl.UnknownFields
+ sizeCache protoimpl.SizeCache
+}
+
+func (x *WriteFileRequest) Reset() {
+ *x = WriteFileRequest{}
+ mi := &file_guest_proto_msgTypes[10]
+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+ ms.StoreMessageInfo(mi)
+}
+
+func (x *WriteFileRequest) String() string {
+ return protoimpl.X.MessageStringOf(x)
+}
+
+func (*WriteFileRequest) ProtoMessage() {}
+
+func (x *WriteFileRequest) ProtoReflect() protoreflect.Message {
+ mi := &file_guest_proto_msgTypes[10]
+ if x != nil {
+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+ if ms.LoadMessageInfo() == nil {
+ ms.StoreMessageInfo(mi)
+ }
+ return ms
+ }
+ return mi.MessageOf(x)
+}
+
+// Deprecated: Use WriteFileRequest.ProtoReflect.Descriptor instead.
+func (*WriteFileRequest) Descriptor() ([]byte, []int) {
+ return file_guest_proto_rawDescGZIP(), []int{10}
+}
+
+func (x *WriteFileRequest) GetPath() string {
+ if x != nil {
+ return x.Path
+ }
+ return ""
+}
+
+func (x *WriteFileRequest) GetMode() uint32 {
+ if x != nil {
+ return x.Mode
+ }
+ return 0
+}
+
+func (x *WriteFileRequest) GetSeekOffset() int64 {
+ if x != nil {
+ return x.SeekOffset
+ }
+ return 0
+}
+
+func (x *WriteFileRequest) GetChunk() []byte {
+ if x != nil {
+ return x.Chunk
+ }
+ return nil
+}
+
+// Response confirming the write operation.
+type WriteFileResponse struct {
+ state protoimpl.MessageState `protogen:"open.v1"`
+ // Total number of bytes written across all stream chunks.
+ BytesWritten int64 `protobuf:"varint,1,opt,name=bytes_written,json=bytesWritten,proto3" json:"bytes_written,omitempty"`
+ unknownFields protoimpl.UnknownFields
+ sizeCache protoimpl.SizeCache
+}
+
+func (x *WriteFileResponse) Reset() {
+ *x = WriteFileResponse{}
+ mi := &file_guest_proto_msgTypes[11]
+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+ ms.StoreMessageInfo(mi)
+}
+
+func (x *WriteFileResponse) String() string {
+ return protoimpl.X.MessageStringOf(x)
+}
+
+func (*WriteFileResponse) ProtoMessage() {}
+
+func (x *WriteFileResponse) ProtoReflect() protoreflect.Message {
+ mi := &file_guest_proto_msgTypes[11]
+ if x != nil {
+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+ if ms.LoadMessageInfo() == nil {
+ ms.StoreMessageInfo(mi)
+ }
+ return ms
+ }
+ return mi.MessageOf(x)
+}
+
+// Deprecated: Use WriteFileResponse.ProtoReflect.Descriptor instead.
+func (*WriteFileResponse) Descriptor() ([]byte, []int) {
+ return file_guest_proto_rawDescGZIP(), []int{11}
+}
+
+func (x *WriteFileResponse) GetBytesWritten() int64 {
+ if x != nil {
+ return x.BytesWritten
+ }
+ return 0
+}
+
+var File_guest_proto protoreflect.FileDescriptor
+
+const file_guest_proto_rawDesc = "" +
+ "\n" +
+ "\vguest.proto\x12\x0eateenv.v1alpha\x1a\x1egoogle/protobuf/duration.proto\x1a\x1fgoogle/protobuf/timestamp.proto\"\x9d\x02\n" +
+ "\aProcess\x12\x1d\n" +
+ "\n" +
+ "process_id\x18\x01 \x01(\tR\tprocessId\x12\x18\n" +
+ "\acommand\x18\x02 \x03(\tR\acommand\x12\x10\n" +
+ "\x03pid\x18\x03 \x01(\x05R\x03pid\x122\n" +
+ "\x05state\x18\x04 \x01(\x0e2\x1c.ateenv.v1alpha.ProcessStateR\x05state\x12\x1b\n" +
+ "\texit_code\x18\x05 \x01(\x05R\bexitCode\x129\n" +
+ "\n" +
+ "started_at\x18\x06 \x01(\v2\x1a.google.protobuf.TimestampR\tstartedAt\x12;\n" +
+ "\vfinished_at\x18\a \x01(\v2\x1a.google.protobuf.TimestampR\n" +
+ "finishedAt\"\x84\x02\n" +
+ "\x13StartProcessRequest\x12\x18\n" +
+ "\acommand\x18\x01 \x03(\tR\acommand\x12\x10\n" +
+ "\x03cwd\x18\x02 \x01(\tR\x03cwd\x12>\n" +
+ "\x03env\x18\x03 \x03(\v2,.ateenv.v1alpha.StartProcessRequest.EnvEntryR\x03env\x12\x14\n" +
+ "\x05stdin\x18\x04 \x01(\bR\x05stdin\x123\n" +
+ "\atimeout\x18\x05 \x01(\v2\x19.google.protobuf.DurationR\atimeout\x1a6\n" +
+ "\bEnvEntry\x12\x10\n" +
+ "\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
+ "\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"2\n" +
+ "\x11GetProcessRequest\x12\x1d\n" +
+ "\n" +
+ "process_id\x18\x01 \x01(\tR\tprocessId\"\x9d\x01\n" +
+ "\x1aStreamProcessOutputRequest\x12\x1d\n" +
+ "\n" +
+ "process_id\x18\x01 \x01(\tR\tprocessId\x12#\n" +
+ "\rstdout_offset\x18\x02 \x01(\x03R\fstdoutOffset\x12#\n" +
+ "\rstderr_offset\x18\x03 \x01(\x03R\fstderrOffset\x12\x16\n" +
+ "\x06follow\x18\x04 \x01(\bR\x06follow\"|\n" +
+ "\rProcessOutput\x12\x18\n" +
+ "\x06stdout\x18\x01 \x01(\fH\x00R\x06stdout\x12\x18\n" +
+ "\x06stderr\x18\x02 \x01(\fH\x00R\x06stderr\x12-\n" +
+ "\x04exit\x18\x03 \x01(\v2\x17.ateenv.v1alpha.ProcessH\x00R\x04exitB\b\n" +
+ "\x06output\"c\n" +
+ "\x18WriteProcessInputRequest\x12\x1d\n" +
+ "\n" +
+ "process_id\x18\x01 \x01(\tR\tprocessId\x12\x12\n" +
+ "\x04data\x18\x02 \x01(\fR\x04data\x12\x14\n" +
+ "\x05close\x18\x03 \x01(\bR\x05close\"@\n" +
+ "\x19WriteProcessInputResponse\x12#\n" +
+ "\rbytes_written\x18\x01 \x01(\x03R\fbytesWritten\"e\n" +
+ "\x14SignalProcessRequest\x12\x1d\n" +
+ "\n" +
+ "process_id\x18\x01 \x01(\tR\tprocessId\x12.\n" +
+ "\x06signal\x18\x02 \x01(\x0e2\x16.ateenv.v1alpha.SignalR\x06signal\"9\n" +
+ "\x0fReadFileRequest\x12\x12\n" +
+ "\x04path\x18\x01 \x01(\tR\x04path\x12\x12\n" +
+ "\x04mode\x18\x02 \x01(\rR\x04mode\"(\n" +
+ "\x10ReadFileResponse\x12\x14\n" +
+ "\x05chunk\x18\x01 \x01(\fR\x05chunk\"q\n" +
+ "\x10WriteFileRequest\x12\x12\n" +
+ "\x04path\x18\x01 \x01(\tR\x04path\x12\x12\n" +
+ "\x04mode\x18\x02 \x01(\rR\x04mode\x12\x1f\n" +
+ "\vseek_offset\x18\x03 \x01(\x03R\n" +
+ "seekOffset\x12\x14\n" +
+ "\x05chunk\x18\x04 \x01(\fR\x05chunk\"8\n" +
+ "\x11WriteFileResponse\x12#\n" +
+ "\rbytes_written\x18\x01 \x01(\x03R\fbytesWritten*b\n" +
+ "\fProcessState\x12\x1d\n" +
+ "\x19PROCESS_STATE_UNSPECIFIED\x10\x00\x12\x19\n" +
+ "\x15PROCESS_STATE_RUNNING\x10\x01\x12\x18\n" +
+ "\x14PROCESS_STATE_EXITED\x10\x02*\x87\x04\n" +
+ "\x06Signal\x12\x16\n" +
+ "\x12SIGNAL_UNSPECIFIED\x10\x00\x12\x0e\n" +
+ "\n" +
+ "SIGNAL_HUP\x10\x01\x12\x0e\n" +
+ "\n" +
+ "SIGNAL_INT\x10\x02\x12\x0f\n" +
+ "\vSIGNAL_QUIT\x10\x03\x12\x0e\n" +
+ "\n" +
+ "SIGNAL_ILL\x10\x04\x12\x0f\n" +
+ "\vSIGNAL_TRAP\x10\x05\x12\x0f\n" +
+ "\vSIGNAL_ABRT\x10\x06\x12\x0e\n" +
+ "\n" +
+ "SIGNAL_BUS\x10\a\x12\x0e\n" +
+ "\n" +
+ "SIGNAL_FPE\x10\b\x12\x0f\n" +
+ "\vSIGNAL_KILL\x10\t\x12\x0f\n" +
+ "\vSIGNAL_USR1\x10\n" +
+ "\x12\x0f\n" +
+ "\vSIGNAL_SEGV\x10\v\x12\x0f\n" +
+ "\vSIGNAL_USR2\x10\f\x12\x0f\n" +
+ "\vSIGNAL_PIPE\x10\r\x12\x0f\n" +
+ "\vSIGNAL_ALRM\x10\x0e\x12\x0f\n" +
+ "\vSIGNAL_TERM\x10\x0f\x12\x0f\n" +
+ "\vSIGNAL_CHLD\x10\x11\x12\x0f\n" +
+ "\vSIGNAL_CONT\x10\x12\x12\x0f\n" +
+ "\vSIGNAL_STOP\x10\x13\x12\x0f\n" +
+ "\vSIGNAL_TSTP\x10\x14\x12\x0f\n" +
+ "\vSIGNAL_TTIN\x10\x15\x12\x0f\n" +
+ "\vSIGNAL_TTOU\x10\x16\x12\x0e\n" +
+ "\n" +
+ "SIGNAL_URG\x10\x17\x12\x0f\n" +
+ "\vSIGNAL_XCPU\x10\x18\x12\x0f\n" +
+ "\vSIGNAL_XFSZ\x10\x19\x12\x11\n" +
+ "\rSIGNAL_VTALRM\x10\x1a\x12\x0f\n" +
+ "\vSIGNAL_PROF\x10\x1b\x12\x10\n" +
+ "\fSIGNAL_WINCH\x10\x1c\x12\r\n" +
+ "\tSIGNAL_IO\x10\x1d\x12\x0e\n" +
+ "\n" +
+ "SIGNAL_SYS\x10\x1f2\xc8\x03\n" +
+ "\x0eProcessService\x12L\n" +
+ "\fStartProcess\x12#.ateenv.v1alpha.StartProcessRequest\x1a\x17.ateenv.v1alpha.Process\x12H\n" +
+ "\n" +
+ "GetProcess\x12!.ateenv.v1alpha.GetProcessRequest\x1a\x17.ateenv.v1alpha.Process\x12b\n" +
+ "\x13StreamProcessOutput\x12*.ateenv.v1alpha.StreamProcessOutputRequest\x1a\x1d.ateenv.v1alpha.ProcessOutput0\x01\x12j\n" +
+ "\x11WriteProcessInput\x12(.ateenv.v1alpha.WriteProcessInputRequest\x1a).ateenv.v1alpha.WriteProcessInputResponse(\x01\x12N\n" +
+ "\rSignalProcess\x12$.ateenv.v1alpha.SignalProcessRequest\x1a\x17.ateenv.v1alpha.Process2\xb8\x01\n" +
+ "\x11FileSystemService\x12O\n" +
+ "\bReadFile\x12\x1f.ateenv.v1alpha.ReadFileRequest\x1a .ateenv.v1alpha.ReadFileResponse0\x01\x12R\n" +
+ "\tWriteFile\x12 .ateenv.v1alpha.WriteFileRequest\x1a!.ateenv.v1alpha.WriteFileResponse(\x01BBZ@go.ate.dev/substrate/internal/proto/ateenv/v1alpha;ateenvv1alphab\x06proto3"
+
+var (
+ file_guest_proto_rawDescOnce sync.Once
+ file_guest_proto_rawDescData []byte
+)
+
+func file_guest_proto_rawDescGZIP() []byte {
+ file_guest_proto_rawDescOnce.Do(func() {
+ file_guest_proto_rawDescData = protoimpl.X.CompressGZIP(unsafe.Slice(unsafe.StringData(file_guest_proto_rawDesc), len(file_guest_proto_rawDesc)))
+ })
+ return file_guest_proto_rawDescData
+}
+
+var file_guest_proto_enumTypes = make([]protoimpl.EnumInfo, 2)
+var file_guest_proto_msgTypes = make([]protoimpl.MessageInfo, 13)
+var file_guest_proto_goTypes = []any{
+ (ProcessState)(0), // 0: ateenv.v1alpha.ProcessState
+ (Signal)(0), // 1: ateenv.v1alpha.Signal
+ (*Process)(nil), // 2: ateenv.v1alpha.Process
+ (*StartProcessRequest)(nil), // 3: ateenv.v1alpha.StartProcessRequest
+ (*GetProcessRequest)(nil), // 4: ateenv.v1alpha.GetProcessRequest
+ (*StreamProcessOutputRequest)(nil), // 5: ateenv.v1alpha.StreamProcessOutputRequest
+ (*ProcessOutput)(nil), // 6: ateenv.v1alpha.ProcessOutput
+ (*WriteProcessInputRequest)(nil), // 7: ateenv.v1alpha.WriteProcessInputRequest
+ (*WriteProcessInputResponse)(nil), // 8: ateenv.v1alpha.WriteProcessInputResponse
+ (*SignalProcessRequest)(nil), // 9: ateenv.v1alpha.SignalProcessRequest
+ (*ReadFileRequest)(nil), // 10: ateenv.v1alpha.ReadFileRequest
+ (*ReadFileResponse)(nil), // 11: ateenv.v1alpha.ReadFileResponse
+ (*WriteFileRequest)(nil), // 12: ateenv.v1alpha.WriteFileRequest
+ (*WriteFileResponse)(nil), // 13: ateenv.v1alpha.WriteFileResponse
+ nil, // 14: ateenv.v1alpha.StartProcessRequest.EnvEntry
+ (*timestamppb.Timestamp)(nil), // 15: google.protobuf.Timestamp
+ (*durationpb.Duration)(nil), // 16: google.protobuf.Duration
+}
+var file_guest_proto_depIdxs = []int32{
+ 0, // 0: ateenv.v1alpha.Process.state:type_name -> ateenv.v1alpha.ProcessState
+ 15, // 1: ateenv.v1alpha.Process.started_at:type_name -> google.protobuf.Timestamp
+ 15, // 2: ateenv.v1alpha.Process.finished_at:type_name -> google.protobuf.Timestamp
+ 14, // 3: ateenv.v1alpha.StartProcessRequest.env:type_name -> ateenv.v1alpha.StartProcessRequest.EnvEntry
+ 16, // 4: ateenv.v1alpha.StartProcessRequest.timeout:type_name -> google.protobuf.Duration
+ 2, // 5: ateenv.v1alpha.ProcessOutput.exit:type_name -> ateenv.v1alpha.Process
+ 1, // 6: ateenv.v1alpha.SignalProcessRequest.signal:type_name -> ateenv.v1alpha.Signal
+ 3, // 7: ateenv.v1alpha.ProcessService.StartProcess:input_type -> ateenv.v1alpha.StartProcessRequest
+ 4, // 8: ateenv.v1alpha.ProcessService.GetProcess:input_type -> ateenv.v1alpha.GetProcessRequest
+ 5, // 9: ateenv.v1alpha.ProcessService.StreamProcessOutput:input_type -> ateenv.v1alpha.StreamProcessOutputRequest
+ 7, // 10: ateenv.v1alpha.ProcessService.WriteProcessInput:input_type -> ateenv.v1alpha.WriteProcessInputRequest
+ 9, // 11: ateenv.v1alpha.ProcessService.SignalProcess:input_type -> ateenv.v1alpha.SignalProcessRequest
+ 10, // 12: ateenv.v1alpha.FileSystemService.ReadFile:input_type -> ateenv.v1alpha.ReadFileRequest
+ 12, // 13: ateenv.v1alpha.FileSystemService.WriteFile:input_type -> ateenv.v1alpha.WriteFileRequest
+ 2, // 14: ateenv.v1alpha.ProcessService.StartProcess:output_type -> ateenv.v1alpha.Process
+ 2, // 15: ateenv.v1alpha.ProcessService.GetProcess:output_type -> ateenv.v1alpha.Process
+ 6, // 16: ateenv.v1alpha.ProcessService.StreamProcessOutput:output_type -> ateenv.v1alpha.ProcessOutput
+ 8, // 17: ateenv.v1alpha.ProcessService.WriteProcessInput:output_type -> ateenv.v1alpha.WriteProcessInputResponse
+ 2, // 18: ateenv.v1alpha.ProcessService.SignalProcess:output_type -> ateenv.v1alpha.Process
+ 11, // 19: ateenv.v1alpha.FileSystemService.ReadFile:output_type -> ateenv.v1alpha.ReadFileResponse
+ 13, // 20: ateenv.v1alpha.FileSystemService.WriteFile:output_type -> ateenv.v1alpha.WriteFileResponse
+ 14, // [14:21] is the sub-list for method output_type
+ 7, // [7:14] is the sub-list for method input_type
+ 7, // [7:7] is the sub-list for extension type_name
+ 7, // [7:7] is the sub-list for extension extendee
+ 0, // [0:7] is the sub-list for field type_name
+}
+
+func init() { file_guest_proto_init() }
+func file_guest_proto_init() {
+ if File_guest_proto != nil {
+ return
+ }
+ file_guest_proto_msgTypes[4].OneofWrappers = []any{
+ (*ProcessOutput_Stdout)(nil),
+ (*ProcessOutput_Stderr)(nil),
+ (*ProcessOutput_Exit)(nil),
+ }
+ type x struct{}
+ out := protoimpl.TypeBuilder{
+ File: protoimpl.DescBuilder{
+ GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
+ RawDescriptor: unsafe.Slice(unsafe.StringData(file_guest_proto_rawDesc), len(file_guest_proto_rawDesc)),
+ NumEnums: 2,
+ NumMessages: 13,
+ NumExtensions: 0,
+ NumServices: 2,
+ },
+ GoTypes: file_guest_proto_goTypes,
+ DependencyIndexes: file_guest_proto_depIdxs,
+ EnumInfos: file_guest_proto_enumTypes,
+ MessageInfos: file_guest_proto_msgTypes,
+ }.Build()
+ File_guest_proto = out.File
+ file_guest_proto_goTypes = nil
+ file_guest_proto_depIdxs = nil
+}
diff --git a/internal/proto/ateenv/v1alpha/guest.proto b/internal/proto/ateenv/v1alpha/guest.proto
new file mode 100644
index 0000000000..0efa545358
--- /dev/null
+++ b/internal/proto/ateenv/v1alpha/guest.proto
@@ -0,0 +1,268 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Copied from github.com/agent-substrate/env/proto/ateenv/v1alpha/guest.proto
+// at commit 1c08a7a578afd896373511a83beb242aec1425c0 (matching the default
+// ATE_ENV_IMAGE digest in benchmarking/workloads/deploy.sh and documented in
+// benchmarking/workloads/manifests/ate-env-template.yaml.tmpl) to avoid adding
+// a cross-repo Go module dependency on agent-substrate/env.
+//
+// Minimal Protocol Buffers definition for the Agent Substrate Guest Data Plane.
+// Exposes asynchronous process execution with streaming I/O and signals, and
+// streaming file I/O.
+
+syntax = "proto3";
+
+package ateenv.v1alpha;
+
+option go_package = "go.ate.dev/substrate/internal/proto/ateenv/v1alpha;ateenvv1alpha";
+
+import "google/protobuf/duration.proto";
+import "google/protobuf/timestamp.proto";
+
+// ============================================================================
+// --- SERVICES ---
+// ============================================================================
+
+// ProcessService manages the lifecycle and I/O of processes running inside
+// the container.
+//
+// A process is started asynchronously and addressed by its process_id. Output
+// is spooled by the guest and can be streamed live or replayed from an offset.
+// Input is fed through a client stream, and any POSIX signal can be delivered
+// to the process group.
+service ProcessService {
+ // StartProcess launches a process in the background and returns its
+ // Process resource immediately.
+ rpc StartProcess(StartProcessRequest) returns (Process);
+
+ // GetProcess returns the current state of a process.
+ rpc GetProcess(GetProcessRequest) returns (Process);
+
+ // StreamProcessOutput streams stdout and stderr from a process. Once the
+ // process has exited and all output has been delivered, the stream ends
+ // with a final exit message carrying the finished Process. Following with
+ // no interest in output (offsets past the end of the spool) is the way to
+ // wait for a process to exit.
+ rpc StreamProcessOutput(StreamProcessOutputRequest) returns (stream ProcessOutput);
+
+ // WriteProcessInput feeds bytes to the stdin of a process started with
+ // stdin enabled. Multiple calls may be made over the life of a process;
+ // stdin stays open until a message sets close, or the process exits.
+ rpc WriteProcessInput(stream WriteProcessInputRequest) returns (WriteProcessInputResponse);
+
+ // SignalProcess delivers a signal to the process and its process group and
+ // returns the process state right after delivery. Use StreamProcessOutput
+ // to observe the effect.
+ rpc SignalProcess(SignalProcessRequest) returns (Process);
+}
+
+// FileSystemService provides streaming file reading and writing capabilities inside
+// the container rootfs/workspace to prevent memory exhaustion (OOM).
+service FileSystemService {
+ // ReadFile streams the binary or text contents of a file in chunks.
+ rpc ReadFile(ReadFileRequest) returns (stream ReadFileResponse);
+
+ // WriteFile streams binary or text chunks directly to a target file.
+ rpc WriteFile(stream WriteFileRequest) returns (WriteFileResponse);
+}
+
+// ============================================================================
+// --- PROCESS SERVICE MESSAGES ---
+// ============================================================================
+
+// Lifecycle state of a process.
+enum ProcessState {
+ PROCESS_STATE_UNSPECIFIED = 0;
+ // The process is running (or stopped by SIGSTOP) and has not been reaped.
+ PROCESS_STATE_RUNNING = 1;
+ // The process has exited; see Process.exit_code.
+ PROCESS_STATE_EXITED = 2;
+}
+
+// POSIX signals that can be sent to a process. Values match the Linux signal
+// numbers on x86/arm.
+enum Signal {
+ SIGNAL_UNSPECIFIED = 0;
+ SIGNAL_HUP = 1;
+ SIGNAL_INT = 2;
+ SIGNAL_QUIT = 3;
+ SIGNAL_ILL = 4;
+ SIGNAL_TRAP = 5;
+ SIGNAL_ABRT = 6;
+ SIGNAL_BUS = 7;
+ SIGNAL_FPE = 8;
+ SIGNAL_KILL = 9;
+ SIGNAL_USR1 = 10;
+ SIGNAL_SEGV = 11;
+ SIGNAL_USR2 = 12;
+ SIGNAL_PIPE = 13;
+ SIGNAL_ALRM = 14;
+ SIGNAL_TERM = 15;
+ SIGNAL_CHLD = 17;
+ SIGNAL_CONT = 18;
+ SIGNAL_STOP = 19;
+ SIGNAL_TSTP = 20;
+ SIGNAL_TTIN = 21;
+ SIGNAL_TTOU = 22;
+ SIGNAL_URG = 23;
+ SIGNAL_XCPU = 24;
+ SIGNAL_XFSZ = 25;
+ SIGNAL_VTALRM = 26;
+ SIGNAL_PROF = 27;
+ SIGNAL_WINCH = 28;
+ SIGNAL_IO = 29;
+ SIGNAL_SYS = 31;
+}
+
+// The Process resource: identity, lifecycle state, and exit status.
+message Process {
+ // Unique process identifier assigned by the guest.
+ string process_id = 1;
+ // Command binary and arguments the process was started with.
+ repeated string command = 2;
+ // Operating system PID inside the container.
+ int32 pid = 3;
+ // Current lifecycle state.
+ ProcessState state = 4;
+ // Exit status, valid once state is EXITED: the process's exit code, or
+ // 128 + signal number if it was terminated by a signal (e.g. 137 for
+ // SIGKILL, 143 for SIGTERM).
+ int32 exit_code = 5;
+ // Timestamp when the process started.
+ google.protobuf.Timestamp started_at = 6;
+ // Timestamp when the process exited. Unset while RUNNING.
+ google.protobuf.Timestamp finished_at = 7;
+}
+
+// Request to start a process.
+message StartProcessRequest {
+ // Command binary and arguments to execute (e.g. ["pytest", "tests/"] or ["sh", "-c", "ls -la"]).
+ repeated string command = 1;
+ // Working directory inside the container (defaults to the guest workspace).
+ string cwd = 2;
+ // Environment variables to set for the process, on top of the guest's environment.
+ map env = 3;
+ // If true, stdin is an open pipe fed by WriteProcessInput. If false, stdin
+ // reads as empty (/dev/null) and WriteProcessInput is rejected.
+ bool stdin = 4;
+ // Maximum wall-clock run time. On expiry the process group receives SIGKILL.
+ // Unset or zero applies the guest's default timeout.
+ google.protobuf.Duration timeout = 5;
+}
+
+// Request to get a Process.
+message GetProcessRequest {
+ // Identifier of the process to inspect.
+ string process_id = 1;
+}
+
+// Request to stream output from a process.
+message StreamProcessOutputRequest {
+ // Identifier of the process to stream output from.
+ string process_id = 1;
+ // Byte offset to start reading stdout from (defaults to 0 / beginning).
+ // Bytes before the offset are never delivered, so an offset past the end
+ // of the spool suppresses stdout entirely.
+ int64 stdout_offset = 2;
+ // Byte offset to start reading stderr from (defaults to 0 / beginning).
+ // Same semantics as stdout_offset.
+ int64 stderr_offset = 3;
+ // If true, the stream stays open and delivers new output until the process
+ // exits. If false, the output spooled so far is returned and the stream ends.
+ bool follow = 4;
+}
+
+// One message on a process output stream. Exactly one field is set.
+message ProcessOutput {
+ oneof output {
+ // A chunk of standard output.
+ bytes stdout = 1;
+ // A chunk of standard error.
+ bytes stderr = 2;
+ // Final message: the process has exited and all output has been delivered.
+ // Absent if the stream ends while the process is still running (follow=false).
+ Process exit = 3;
+ }
+}
+
+// One chunk of standard input for a process.
+message WriteProcessInputRequest {
+ // Identifier of the target process. Required on the first message; ignored afterwards.
+ string process_id = 1;
+ // Bytes to write to stdin. May be empty, e.g. to only close stdin.
+ bytes data = 2;
+ // If true, stdin is closed (EOF) after data is written. No further input is accepted.
+ bool close = 3;
+}
+
+// Response confirming the input write.
+message WriteProcessInputResponse {
+ // Total number of bytes written to stdin across all messages in the stream.
+ int64 bytes_written = 1;
+}
+
+// Request to send a signal to a process.
+message SignalProcessRequest {
+ // Identifier of the target process.
+ string process_id = 1;
+ // Signal to deliver to the process group.
+ Signal signal = 2;
+}
+
+// ============================================================================
+// --- FILESYSTEM SERVICE MESSAGES ---
+// ============================================================================
+
+// Request to read a file from the container.
+message ReadFileRequest {
+ // Absolute or workspace-relative path of the file to read.
+ string path = 1;
+ // Optional POSIX file mode permission (e.g. 0644). When set and the file
+ // does not exist, it is created empty with this mode (parent directories
+ // included) and the stream ends without chunks. When zero, a missing file
+ // is a NOT_FOUND error. Existing files are never modified.
+ uint32 mode = 2;
+}
+
+// Streamed chunk of file data.
+message ReadFileResponse {
+ // Raw binary or text chunk of the file.
+ bytes chunk = 1;
+}
+
+// Streamed request chunk for writing data to a file. path, mode and
+// seek_offset are read from the first message only and ignored on all
+// subsequent messages; every message may carry a chunk.
+message WriteFileRequest {
+ // Absolute or workspace-relative path of the file to write. Required on the
+ // first message.
+ string path = 1;
+ // Optional POSIX file mode permission (e.g. 0644 or 0755), applied when the
+ // file is created.
+ uint32 mode = 2;
+ // Byte offset at which writing starts. Zero, the default, truncates the
+ // file and rewrites it from the beginning. A positive offset keeps existing
+ // content, begins writing at the offset (extending the file with zero bytes
+ // if it is shorter), and leaves bytes after the written range in place.
+ int64 seek_offset = 3;
+ // Raw binary or text chunk to write to the file.
+ bytes chunk = 4;
+}
+
+// Response confirming the write operation.
+message WriteFileResponse {
+ // Total number of bytes written across all stream chunks.
+ int64 bytes_written = 1;
+}
diff --git a/internal/proto/ateenv/v1alpha/guest_grpc.pb.go b/internal/proto/ateenv/v1alpha/guest_grpc.pb.go
new file mode 100644
index 0000000000..8e5e209f3c
--- /dev/null
+++ b/internal/proto/ateenv/v1alpha/guest_grpc.pb.go
@@ -0,0 +1,484 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Copied from github.com/agent-substrate/env/proto/ateenv/v1alpha/guest.proto
+// at commit 1c08a7a578afd896373511a83beb242aec1425c0 (matching the default
+// ATE_ENV_IMAGE digest in benchmarking/workloads/deploy.sh and documented in
+// benchmarking/workloads/manifests/ate-env-template.yaml.tmpl) to avoid adding
+// a cross-repo Go module dependency on agent-substrate/env.
+//
+// Minimal Protocol Buffers definition for the Agent Substrate Guest Data Plane.
+// Exposes asynchronous process execution with streaming I/O and signals, and
+// streaming file I/O.
+
+// Code generated by protoc-gen-go-grpc. DO NOT EDIT.
+// versions:
+// - protoc-gen-go-grpc v1.6.2
+// - protoc v4.25.3
+// source: guest.proto
+
+package ateenvv1alpha
+
+import (
+ context "context"
+ grpc "google.golang.org/grpc"
+ codes "google.golang.org/grpc/codes"
+ status "google.golang.org/grpc/status"
+)
+
+// This is a compile-time assertion to ensure that this generated file
+// is compatible with the grpc package it is being compiled against.
+// Requires gRPC-Go v1.64.0 or later.
+const _ = grpc.SupportPackageIsVersion9
+
+const (
+ ProcessService_StartProcess_FullMethodName = "/ateenv.v1alpha.ProcessService/StartProcess"
+ ProcessService_GetProcess_FullMethodName = "/ateenv.v1alpha.ProcessService/GetProcess"
+ ProcessService_StreamProcessOutput_FullMethodName = "/ateenv.v1alpha.ProcessService/StreamProcessOutput"
+ ProcessService_WriteProcessInput_FullMethodName = "/ateenv.v1alpha.ProcessService/WriteProcessInput"
+ ProcessService_SignalProcess_FullMethodName = "/ateenv.v1alpha.ProcessService/SignalProcess"
+)
+
+// ProcessServiceClient is the client API for ProcessService service.
+//
+// For semantics around ctx use and closing/ending streaming RPCs, please refer to https://pkg.go.dev/google.golang.org/grpc/?tab=doc#ClientConn.NewStream.
+//
+// ProcessService manages the lifecycle and I/O of processes running inside
+// the container.
+//
+// A process is started asynchronously and addressed by its process_id. Output
+// is spooled by the guest and can be streamed live or replayed from an offset.
+// Input is fed through a client stream, and any POSIX signal can be delivered
+// to the process group.
+type ProcessServiceClient interface {
+ // StartProcess launches a process in the background and returns its
+ // Process resource immediately.
+ StartProcess(ctx context.Context, in *StartProcessRequest, opts ...grpc.CallOption) (*Process, error)
+ // GetProcess returns the current state of a process.
+ GetProcess(ctx context.Context, in *GetProcessRequest, opts ...grpc.CallOption) (*Process, error)
+ // StreamProcessOutput streams stdout and stderr from a process. Once the
+ // process has exited and all output has been delivered, the stream ends
+ // with a final exit message carrying the finished Process. Following with
+ // no interest in output (offsets past the end of the spool) is the way to
+ // wait for a process to exit.
+ StreamProcessOutput(ctx context.Context, in *StreamProcessOutputRequest, opts ...grpc.CallOption) (grpc.ServerStreamingClient[ProcessOutput], error)
+ // WriteProcessInput feeds bytes to the stdin of a process started with
+ // stdin enabled. Multiple calls may be made over the life of a process;
+ // stdin stays open until a message sets close, or the process exits.
+ WriteProcessInput(ctx context.Context, opts ...grpc.CallOption) (grpc.ClientStreamingClient[WriteProcessInputRequest, WriteProcessInputResponse], error)
+ // SignalProcess delivers a signal to the process and its process group and
+ // returns the process state right after delivery. Use StreamProcessOutput
+ // to observe the effect.
+ SignalProcess(ctx context.Context, in *SignalProcessRequest, opts ...grpc.CallOption) (*Process, error)
+}
+
+type processServiceClient struct {
+ cc grpc.ClientConnInterface
+}
+
+func NewProcessServiceClient(cc grpc.ClientConnInterface) ProcessServiceClient {
+ return &processServiceClient{cc}
+}
+
+func (c *processServiceClient) StartProcess(ctx context.Context, in *StartProcessRequest, opts ...grpc.CallOption) (*Process, error) {
+ cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...)
+ out := new(Process)
+ err := c.cc.Invoke(ctx, ProcessService_StartProcess_FullMethodName, in, out, cOpts...)
+ if err != nil {
+ return nil, err
+ }
+ return out, nil
+}
+
+func (c *processServiceClient) GetProcess(ctx context.Context, in *GetProcessRequest, opts ...grpc.CallOption) (*Process, error) {
+ cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...)
+ out := new(Process)
+ err := c.cc.Invoke(ctx, ProcessService_GetProcess_FullMethodName, in, out, cOpts...)
+ if err != nil {
+ return nil, err
+ }
+ return out, nil
+}
+
+func (c *processServiceClient) StreamProcessOutput(ctx context.Context, in *StreamProcessOutputRequest, opts ...grpc.CallOption) (grpc.ServerStreamingClient[ProcessOutput], error) {
+ cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...)
+ stream, err := c.cc.NewStream(ctx, &ProcessService_ServiceDesc.Streams[0], ProcessService_StreamProcessOutput_FullMethodName, cOpts...)
+ if err != nil {
+ return nil, err
+ }
+ x := &grpc.GenericClientStream[StreamProcessOutputRequest, ProcessOutput]{ClientStream: stream}
+ if err := x.ClientStream.SendMsg(in); err != nil {
+ return nil, err
+ }
+ if err := x.ClientStream.CloseSend(); err != nil {
+ return nil, err
+ }
+ return x, nil
+}
+
+// This type alias is provided for backwards compatibility with existing code that references the prior non-generic stream type by name.
+type ProcessService_StreamProcessOutputClient = grpc.ServerStreamingClient[ProcessOutput]
+
+func (c *processServiceClient) WriteProcessInput(ctx context.Context, opts ...grpc.CallOption) (grpc.ClientStreamingClient[WriteProcessInputRequest, WriteProcessInputResponse], error) {
+ cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...)
+ stream, err := c.cc.NewStream(ctx, &ProcessService_ServiceDesc.Streams[1], ProcessService_WriteProcessInput_FullMethodName, cOpts...)
+ if err != nil {
+ return nil, err
+ }
+ x := &grpc.GenericClientStream[WriteProcessInputRequest, WriteProcessInputResponse]{ClientStream: stream}
+ return x, nil
+}
+
+// This type alias is provided for backwards compatibility with existing code that references the prior non-generic stream type by name.
+type ProcessService_WriteProcessInputClient = grpc.ClientStreamingClient[WriteProcessInputRequest, WriteProcessInputResponse]
+
+func (c *processServiceClient) SignalProcess(ctx context.Context, in *SignalProcessRequest, opts ...grpc.CallOption) (*Process, error) {
+ cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...)
+ out := new(Process)
+ err := c.cc.Invoke(ctx, ProcessService_SignalProcess_FullMethodName, in, out, cOpts...)
+ if err != nil {
+ return nil, err
+ }
+ return out, nil
+}
+
+// ProcessServiceServer is the server API for ProcessService service.
+// All implementations must embed UnimplementedProcessServiceServer
+// for forward compatibility.
+//
+// ProcessService manages the lifecycle and I/O of processes running inside
+// the container.
+//
+// A process is started asynchronously and addressed by its process_id. Output
+// is spooled by the guest and can be streamed live or replayed from an offset.
+// Input is fed through a client stream, and any POSIX signal can be delivered
+// to the process group.
+type ProcessServiceServer interface {
+ // StartProcess launches a process in the background and returns its
+ // Process resource immediately.
+ StartProcess(context.Context, *StartProcessRequest) (*Process, error)
+ // GetProcess returns the current state of a process.
+ GetProcess(context.Context, *GetProcessRequest) (*Process, error)
+ // StreamProcessOutput streams stdout and stderr from a process. Once the
+ // process has exited and all output has been delivered, the stream ends
+ // with a final exit message carrying the finished Process. Following with
+ // no interest in output (offsets past the end of the spool) is the way to
+ // wait for a process to exit.
+ StreamProcessOutput(*StreamProcessOutputRequest, grpc.ServerStreamingServer[ProcessOutput]) error
+ // WriteProcessInput feeds bytes to the stdin of a process started with
+ // stdin enabled. Multiple calls may be made over the life of a process;
+ // stdin stays open until a message sets close, or the process exits.
+ WriteProcessInput(grpc.ClientStreamingServer[WriteProcessInputRequest, WriteProcessInputResponse]) error
+ // SignalProcess delivers a signal to the process and its process group and
+ // returns the process state right after delivery. Use StreamProcessOutput
+ // to observe the effect.
+ SignalProcess(context.Context, *SignalProcessRequest) (*Process, error)
+ mustEmbedUnimplementedProcessServiceServer()
+}
+
+// UnimplementedProcessServiceServer must be embedded to have
+// forward compatible implementations.
+//
+// NOTE: this should be embedded by value instead of pointer to avoid a nil
+// pointer dereference when methods are called.
+type UnimplementedProcessServiceServer struct{}
+
+func (UnimplementedProcessServiceServer) StartProcess(context.Context, *StartProcessRequest) (*Process, error) {
+ return nil, status.Error(codes.Unimplemented, "method StartProcess not implemented")
+}
+func (UnimplementedProcessServiceServer) GetProcess(context.Context, *GetProcessRequest) (*Process, error) {
+ return nil, status.Error(codes.Unimplemented, "method GetProcess not implemented")
+}
+func (UnimplementedProcessServiceServer) StreamProcessOutput(*StreamProcessOutputRequest, grpc.ServerStreamingServer[ProcessOutput]) error {
+ return status.Error(codes.Unimplemented, "method StreamProcessOutput not implemented")
+}
+func (UnimplementedProcessServiceServer) WriteProcessInput(grpc.ClientStreamingServer[WriteProcessInputRequest, WriteProcessInputResponse]) error {
+ return status.Error(codes.Unimplemented, "method WriteProcessInput not implemented")
+}
+func (UnimplementedProcessServiceServer) SignalProcess(context.Context, *SignalProcessRequest) (*Process, error) {
+ return nil, status.Error(codes.Unimplemented, "method SignalProcess not implemented")
+}
+func (UnimplementedProcessServiceServer) mustEmbedUnimplementedProcessServiceServer() {}
+func (UnimplementedProcessServiceServer) testEmbeddedByValue() {}
+
+// UnsafeProcessServiceServer may be embedded to opt out of forward compatibility for this service.
+// Use of this interface is not recommended, as added methods to ProcessServiceServer will
+// result in compilation errors.
+type UnsafeProcessServiceServer interface {
+ mustEmbedUnimplementedProcessServiceServer()
+}
+
+func RegisterProcessServiceServer(s grpc.ServiceRegistrar, srv ProcessServiceServer) {
+ // If the following call panics, it indicates UnimplementedProcessServiceServer was
+ // embedded by pointer and is nil. This will cause panics if an
+ // unimplemented method is ever invoked, so we test this at initialization
+ // time to prevent it from happening at runtime later due to I/O.
+ if t, ok := srv.(interface{ testEmbeddedByValue() }); ok {
+ t.testEmbeddedByValue()
+ }
+ s.RegisterService(&ProcessService_ServiceDesc, srv)
+}
+
+func _ProcessService_StartProcess_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
+ in := new(StartProcessRequest)
+ if err := dec(in); err != nil {
+ return nil, err
+ }
+ if interceptor == nil {
+ return srv.(ProcessServiceServer).StartProcess(ctx, in)
+ }
+ info := &grpc.UnaryServerInfo{
+ Server: srv,
+ FullMethod: ProcessService_StartProcess_FullMethodName,
+ }
+ handler := func(ctx context.Context, req interface{}) (interface{}, error) {
+ return srv.(ProcessServiceServer).StartProcess(ctx, req.(*StartProcessRequest))
+ }
+ return interceptor(ctx, in, info, handler)
+}
+
+func _ProcessService_GetProcess_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
+ in := new(GetProcessRequest)
+ if err := dec(in); err != nil {
+ return nil, err
+ }
+ if interceptor == nil {
+ return srv.(ProcessServiceServer).GetProcess(ctx, in)
+ }
+ info := &grpc.UnaryServerInfo{
+ Server: srv,
+ FullMethod: ProcessService_GetProcess_FullMethodName,
+ }
+ handler := func(ctx context.Context, req interface{}) (interface{}, error) {
+ return srv.(ProcessServiceServer).GetProcess(ctx, req.(*GetProcessRequest))
+ }
+ return interceptor(ctx, in, info, handler)
+}
+
+func _ProcessService_StreamProcessOutput_Handler(srv interface{}, stream grpc.ServerStream) error {
+ m := new(StreamProcessOutputRequest)
+ if err := stream.RecvMsg(m); err != nil {
+ return err
+ }
+ return srv.(ProcessServiceServer).StreamProcessOutput(m, &grpc.GenericServerStream[StreamProcessOutputRequest, ProcessOutput]{ServerStream: stream})
+}
+
+// This type alias is provided for backwards compatibility with existing code that references the prior non-generic stream type by name.
+type ProcessService_StreamProcessOutputServer = grpc.ServerStreamingServer[ProcessOutput]
+
+func _ProcessService_WriteProcessInput_Handler(srv interface{}, stream grpc.ServerStream) error {
+ return srv.(ProcessServiceServer).WriteProcessInput(&grpc.GenericServerStream[WriteProcessInputRequest, WriteProcessInputResponse]{ServerStream: stream})
+}
+
+// This type alias is provided for backwards compatibility with existing code that references the prior non-generic stream type by name.
+type ProcessService_WriteProcessInputServer = grpc.ClientStreamingServer[WriteProcessInputRequest, WriteProcessInputResponse]
+
+func _ProcessService_SignalProcess_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
+ in := new(SignalProcessRequest)
+ if err := dec(in); err != nil {
+ return nil, err
+ }
+ if interceptor == nil {
+ return srv.(ProcessServiceServer).SignalProcess(ctx, in)
+ }
+ info := &grpc.UnaryServerInfo{
+ Server: srv,
+ FullMethod: ProcessService_SignalProcess_FullMethodName,
+ }
+ handler := func(ctx context.Context, req interface{}) (interface{}, error) {
+ return srv.(ProcessServiceServer).SignalProcess(ctx, req.(*SignalProcessRequest))
+ }
+ return interceptor(ctx, in, info, handler)
+}
+
+// ProcessService_ServiceDesc is the grpc.ServiceDesc for ProcessService service.
+// It's only intended for direct use with grpc.RegisterService,
+// and not to be introspected or modified (even as a copy)
+var ProcessService_ServiceDesc = grpc.ServiceDesc{
+ ServiceName: "ateenv.v1alpha.ProcessService",
+ HandlerType: (*ProcessServiceServer)(nil),
+ Methods: []grpc.MethodDesc{
+ {
+ MethodName: "StartProcess",
+ Handler: _ProcessService_StartProcess_Handler,
+ },
+ {
+ MethodName: "GetProcess",
+ Handler: _ProcessService_GetProcess_Handler,
+ },
+ {
+ MethodName: "SignalProcess",
+ Handler: _ProcessService_SignalProcess_Handler,
+ },
+ },
+ Streams: []grpc.StreamDesc{
+ {
+ StreamName: "StreamProcessOutput",
+ Handler: _ProcessService_StreamProcessOutput_Handler,
+ ServerStreams: true,
+ },
+ {
+ StreamName: "WriteProcessInput",
+ Handler: _ProcessService_WriteProcessInput_Handler,
+ ClientStreams: true,
+ },
+ },
+ Metadata: "guest.proto",
+}
+
+const (
+ FileSystemService_ReadFile_FullMethodName = "/ateenv.v1alpha.FileSystemService/ReadFile"
+ FileSystemService_WriteFile_FullMethodName = "/ateenv.v1alpha.FileSystemService/WriteFile"
+)
+
+// FileSystemServiceClient is the client API for FileSystemService service.
+//
+// For semantics around ctx use and closing/ending streaming RPCs, please refer to https://pkg.go.dev/google.golang.org/grpc/?tab=doc#ClientConn.NewStream.
+//
+// FileSystemService provides streaming file reading and writing capabilities inside
+// the container rootfs/workspace to prevent memory exhaustion (OOM).
+type FileSystemServiceClient interface {
+ // ReadFile streams the binary or text contents of a file in chunks.
+ ReadFile(ctx context.Context, in *ReadFileRequest, opts ...grpc.CallOption) (grpc.ServerStreamingClient[ReadFileResponse], error)
+ // WriteFile streams binary or text chunks directly to a target file.
+ WriteFile(ctx context.Context, opts ...grpc.CallOption) (grpc.ClientStreamingClient[WriteFileRequest, WriteFileResponse], error)
+}
+
+type fileSystemServiceClient struct {
+ cc grpc.ClientConnInterface
+}
+
+func NewFileSystemServiceClient(cc grpc.ClientConnInterface) FileSystemServiceClient {
+ return &fileSystemServiceClient{cc}
+}
+
+func (c *fileSystemServiceClient) ReadFile(ctx context.Context, in *ReadFileRequest, opts ...grpc.CallOption) (grpc.ServerStreamingClient[ReadFileResponse], error) {
+ cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...)
+ stream, err := c.cc.NewStream(ctx, &FileSystemService_ServiceDesc.Streams[0], FileSystemService_ReadFile_FullMethodName, cOpts...)
+ if err != nil {
+ return nil, err
+ }
+ x := &grpc.GenericClientStream[ReadFileRequest, ReadFileResponse]{ClientStream: stream}
+ if err := x.ClientStream.SendMsg(in); err != nil {
+ return nil, err
+ }
+ if err := x.ClientStream.CloseSend(); err != nil {
+ return nil, err
+ }
+ return x, nil
+}
+
+// This type alias is provided for backwards compatibility with existing code that references the prior non-generic stream type by name.
+type FileSystemService_ReadFileClient = grpc.ServerStreamingClient[ReadFileResponse]
+
+func (c *fileSystemServiceClient) WriteFile(ctx context.Context, opts ...grpc.CallOption) (grpc.ClientStreamingClient[WriteFileRequest, WriteFileResponse], error) {
+ cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...)
+ stream, err := c.cc.NewStream(ctx, &FileSystemService_ServiceDesc.Streams[1], FileSystemService_WriteFile_FullMethodName, cOpts...)
+ if err != nil {
+ return nil, err
+ }
+ x := &grpc.GenericClientStream[WriteFileRequest, WriteFileResponse]{ClientStream: stream}
+ return x, nil
+}
+
+// This type alias is provided for backwards compatibility with existing code that references the prior non-generic stream type by name.
+type FileSystemService_WriteFileClient = grpc.ClientStreamingClient[WriteFileRequest, WriteFileResponse]
+
+// FileSystemServiceServer is the server API for FileSystemService service.
+// All implementations must embed UnimplementedFileSystemServiceServer
+// for forward compatibility.
+//
+// FileSystemService provides streaming file reading and writing capabilities inside
+// the container rootfs/workspace to prevent memory exhaustion (OOM).
+type FileSystemServiceServer interface {
+ // ReadFile streams the binary or text contents of a file in chunks.
+ ReadFile(*ReadFileRequest, grpc.ServerStreamingServer[ReadFileResponse]) error
+ // WriteFile streams binary or text chunks directly to a target file.
+ WriteFile(grpc.ClientStreamingServer[WriteFileRequest, WriteFileResponse]) error
+ mustEmbedUnimplementedFileSystemServiceServer()
+}
+
+// UnimplementedFileSystemServiceServer must be embedded to have
+// forward compatible implementations.
+//
+// NOTE: this should be embedded by value instead of pointer to avoid a nil
+// pointer dereference when methods are called.
+type UnimplementedFileSystemServiceServer struct{}
+
+func (UnimplementedFileSystemServiceServer) ReadFile(*ReadFileRequest, grpc.ServerStreamingServer[ReadFileResponse]) error {
+ return status.Error(codes.Unimplemented, "method ReadFile not implemented")
+}
+func (UnimplementedFileSystemServiceServer) WriteFile(grpc.ClientStreamingServer[WriteFileRequest, WriteFileResponse]) error {
+ return status.Error(codes.Unimplemented, "method WriteFile not implemented")
+}
+func (UnimplementedFileSystemServiceServer) mustEmbedUnimplementedFileSystemServiceServer() {}
+func (UnimplementedFileSystemServiceServer) testEmbeddedByValue() {}
+
+// UnsafeFileSystemServiceServer may be embedded to opt out of forward compatibility for this service.
+// Use of this interface is not recommended, as added methods to FileSystemServiceServer will
+// result in compilation errors.
+type UnsafeFileSystemServiceServer interface {
+ mustEmbedUnimplementedFileSystemServiceServer()
+}
+
+func RegisterFileSystemServiceServer(s grpc.ServiceRegistrar, srv FileSystemServiceServer) {
+ // If the following call panics, it indicates UnimplementedFileSystemServiceServer was
+ // embedded by pointer and is nil. This will cause panics if an
+ // unimplemented method is ever invoked, so we test this at initialization
+ // time to prevent it from happening at runtime later due to I/O.
+ if t, ok := srv.(interface{ testEmbeddedByValue() }); ok {
+ t.testEmbeddedByValue()
+ }
+ s.RegisterService(&FileSystemService_ServiceDesc, srv)
+}
+
+func _FileSystemService_ReadFile_Handler(srv interface{}, stream grpc.ServerStream) error {
+ m := new(ReadFileRequest)
+ if err := stream.RecvMsg(m); err != nil {
+ return err
+ }
+ return srv.(FileSystemServiceServer).ReadFile(m, &grpc.GenericServerStream[ReadFileRequest, ReadFileResponse]{ServerStream: stream})
+}
+
+// This type alias is provided for backwards compatibility with existing code that references the prior non-generic stream type by name.
+type FileSystemService_ReadFileServer = grpc.ServerStreamingServer[ReadFileResponse]
+
+func _FileSystemService_WriteFile_Handler(srv interface{}, stream grpc.ServerStream) error {
+ return srv.(FileSystemServiceServer).WriteFile(&grpc.GenericServerStream[WriteFileRequest, WriteFileResponse]{ServerStream: stream})
+}
+
+// This type alias is provided for backwards compatibility with existing code that references the prior non-generic stream type by name.
+type FileSystemService_WriteFileServer = grpc.ClientStreamingServer[WriteFileRequest, WriteFileResponse]
+
+// FileSystemService_ServiceDesc is the grpc.ServiceDesc for FileSystemService service.
+// It's only intended for direct use with grpc.RegisterService,
+// and not to be introspected or modified (even as a copy)
+var FileSystemService_ServiceDesc = grpc.ServiceDesc{
+ ServiceName: "ateenv.v1alpha.FileSystemService",
+ HandlerType: (*FileSystemServiceServer)(nil),
+ Methods: []grpc.MethodDesc{},
+ Streams: []grpc.StreamDesc{
+ {
+ StreamName: "ReadFile",
+ Handler: _FileSystemService_ReadFile_Handler,
+ ServerStreams: true,
+ },
+ {
+ StreamName: "WriteFile",
+ Handler: _FileSystemService_WriteFile_Handler,
+ ClientStreams: true,
+ },
+ },
+ Metadata: "guest.proto",
+}
diff --git a/internal/proto/ateletpb/atelet.pb.go b/internal/proto/ateletpb/atelet.pb.go
index 5f085380aa..3ae5fa13fe 100644
--- a/internal/proto/ateletpb/atelet.pb.go
+++ b/internal/proto/ateletpb/atelet.pb.go
@@ -167,9 +167,6 @@ const (
// atelet reports the actor's state with
// WorkerService.ReportActorSnapshotState once such an upload makes a newer
// snapshot durable, and once such a detach has deleted the local snapshots.
- //
- // atelet never uploads or detaches a snapshot it cannot make durable, such
- // as one without `snapshot.object`.
UploadMode_UPLOAD_MODE_DEFERRED UploadMode = 1
// Upload the snapshot synchronously before Checkpoint returns, and keep a
// local copy if atelet can: failing to store it does not fail the
@@ -815,6 +812,122 @@ func (*RequestActorSuspendResponse) Descriptor() ([]byte, []int) {
return file_atelet_proto_rawDescGZIP(), []int{9}
}
+type ReportActorCrashedRequest struct {
+ state protoimpl.MessageState `protogen:"open.v1"`
+ // The actor the calling worker is hosting, as for RequestActorSuspend.
+ //
+ // +k8s:required
+ // +k8s:format=k8s-short-name
+ ActorAtespace string `protobuf:"bytes,1,opt,name=actor_atespace,json=actorAtespace,proto3" json:"actor_atespace,omitempty"`
+ // +k8s:required
+ // +k8s:format=k8s-short-name
+ ActorName string `protobuf:"bytes,2,opt,name=actor_name,json=actorName,proto3" json:"actor_name,omitempty"`
+ // +k8s:required
+ // +k8s:format=k8s-uuid
+ ActorUid string `protobuf:"bytes,3,opt,name=actor_uid,json=actorUid,proto3" json:"actor_uid,omitempty"`
+ // What the worker saw, e.g. how the sandbox exited.
+ //
+ // +k8s:required
+ // +k8s:maxLength=4096
+ Message string `protobuf:"bytes,4,opt,name=message,proto3" json:"message,omitempty"`
+ unknownFields protoimpl.UnknownFields
+ sizeCache protoimpl.SizeCache
+}
+
+func (x *ReportActorCrashedRequest) Reset() {
+ *x = ReportActorCrashedRequest{}
+ mi := &file_atelet_proto_msgTypes[10]
+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+ ms.StoreMessageInfo(mi)
+}
+
+func (x *ReportActorCrashedRequest) String() string {
+ return protoimpl.X.MessageStringOf(x)
+}
+
+func (*ReportActorCrashedRequest) ProtoMessage() {}
+
+func (x *ReportActorCrashedRequest) ProtoReflect() protoreflect.Message {
+ mi := &file_atelet_proto_msgTypes[10]
+ if x != nil {
+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+ if ms.LoadMessageInfo() == nil {
+ ms.StoreMessageInfo(mi)
+ }
+ return ms
+ }
+ return mi.MessageOf(x)
+}
+
+// Deprecated: Use ReportActorCrashedRequest.ProtoReflect.Descriptor instead.
+func (*ReportActorCrashedRequest) Descriptor() ([]byte, []int) {
+ return file_atelet_proto_rawDescGZIP(), []int{10}
+}
+
+func (x *ReportActorCrashedRequest) GetActorAtespace() string {
+ if x != nil {
+ return x.ActorAtespace
+ }
+ return ""
+}
+
+func (x *ReportActorCrashedRequest) GetActorName() string {
+ if x != nil {
+ return x.ActorName
+ }
+ return ""
+}
+
+func (x *ReportActorCrashedRequest) GetActorUid() string {
+ if x != nil {
+ return x.ActorUid
+ }
+ return ""
+}
+
+func (x *ReportActorCrashedRequest) GetMessage() string {
+ if x != nil {
+ return x.Message
+ }
+ return ""
+}
+
+type ReportActorCrashedResponse struct {
+ state protoimpl.MessageState `protogen:"open.v1"`
+ unknownFields protoimpl.UnknownFields
+ sizeCache protoimpl.SizeCache
+}
+
+func (x *ReportActorCrashedResponse) Reset() {
+ *x = ReportActorCrashedResponse{}
+ mi := &file_atelet_proto_msgTypes[11]
+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+ ms.StoreMessageInfo(mi)
+}
+
+func (x *ReportActorCrashedResponse) String() string {
+ return protoimpl.X.MessageStringOf(x)
+}
+
+func (*ReportActorCrashedResponse) ProtoMessage() {}
+
+func (x *ReportActorCrashedResponse) ProtoReflect() protoreflect.Message {
+ mi := &file_atelet_proto_msgTypes[11]
+ if x != nil {
+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+ if ms.LoadMessageInfo() == nil {
+ ms.StoreMessageInfo(mi)
+ }
+ return ms
+ }
+ return mi.MessageOf(x)
+}
+
+// Deprecated: Use ReportActorCrashedResponse.ProtoReflect.Descriptor instead.
+func (*ReportActorCrashedResponse) Descriptor() ([]byte, []int) {
+ return file_atelet_proto_rawDescGZIP(), []int{11}
+}
+
type GetActorEgressPolicyRequest struct {
state protoimpl.MessageState `protogen:"open.v1"`
// The actor the calling worker is hosting. atelet reads its policy from the
@@ -832,7 +945,7 @@ type GetActorEgressPolicyRequest struct {
func (x *GetActorEgressPolicyRequest) Reset() {
*x = GetActorEgressPolicyRequest{}
- mi := &file_atelet_proto_msgTypes[10]
+ mi := &file_atelet_proto_msgTypes[12]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -844,7 +957,7 @@ func (x *GetActorEgressPolicyRequest) String() string {
func (*GetActorEgressPolicyRequest) ProtoMessage() {}
func (x *GetActorEgressPolicyRequest) ProtoReflect() protoreflect.Message {
- mi := &file_atelet_proto_msgTypes[10]
+ mi := &file_atelet_proto_msgTypes[12]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -857,7 +970,7 @@ func (x *GetActorEgressPolicyRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use GetActorEgressPolicyRequest.ProtoReflect.Descriptor instead.
func (*GetActorEgressPolicyRequest) Descriptor() ([]byte, []int) {
- return file_atelet_proto_rawDescGZIP(), []int{10}
+ return file_atelet_proto_rawDescGZIP(), []int{12}
}
func (x *GetActorEgressPolicyRequest) GetActorAtespace() string {
@@ -885,7 +998,7 @@ type GetActorEgressPolicyResponse struct {
func (x *GetActorEgressPolicyResponse) Reset() {
*x = GetActorEgressPolicyResponse{}
- mi := &file_atelet_proto_msgTypes[11]
+ mi := &file_atelet_proto_msgTypes[13]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -897,7 +1010,7 @@ func (x *GetActorEgressPolicyResponse) String() string {
func (*GetActorEgressPolicyResponse) ProtoMessage() {}
func (x *GetActorEgressPolicyResponse) ProtoReflect() protoreflect.Message {
- mi := &file_atelet_proto_msgTypes[11]
+ mi := &file_atelet_proto_msgTypes[13]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -910,7 +1023,7 @@ func (x *GetActorEgressPolicyResponse) ProtoReflect() protoreflect.Message {
// Deprecated: Use GetActorEgressPolicyResponse.ProtoReflect.Descriptor instead.
func (*GetActorEgressPolicyResponse) Descriptor() ([]byte, []int) {
- return file_atelet_proto_rawDescGZIP(), []int{11}
+ return file_atelet_proto_rawDescGZIP(), []int{13}
}
func (x *GetActorEgressPolicyResponse) GetEgressPolicy() *EgressPolicy {
@@ -934,7 +1047,7 @@ type EgressPolicy struct {
func (x *EgressPolicy) Reset() {
*x = EgressPolicy{}
- mi := &file_atelet_proto_msgTypes[12]
+ mi := &file_atelet_proto_msgTypes[14]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -946,7 +1059,7 @@ func (x *EgressPolicy) String() string {
func (*EgressPolicy) ProtoMessage() {}
func (x *EgressPolicy) ProtoReflect() protoreflect.Message {
- mi := &file_atelet_proto_msgTypes[12]
+ mi := &file_atelet_proto_msgTypes[14]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -959,7 +1072,7 @@ func (x *EgressPolicy) ProtoReflect() protoreflect.Message {
// Deprecated: Use EgressPolicy.ProtoReflect.Descriptor instead.
func (*EgressPolicy) Descriptor() ([]byte, []int) {
- return file_atelet_proto_rawDescGZIP(), []int{12}
+ return file_atelet_proto_rawDescGZIP(), []int{14}
}
func (x *EgressPolicy) GetRules() []*EgressRule {
@@ -986,7 +1099,7 @@ type EgressRule struct {
func (x *EgressRule) Reset() {
*x = EgressRule{}
- mi := &file_atelet_proto_msgTypes[13]
+ mi := &file_atelet_proto_msgTypes[15]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -998,7 +1111,7 @@ func (x *EgressRule) String() string {
func (*EgressRule) ProtoMessage() {}
func (x *EgressRule) ProtoReflect() protoreflect.Message {
- mi := &file_atelet_proto_msgTypes[13]
+ mi := &file_atelet_proto_msgTypes[15]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -1011,7 +1124,7 @@ func (x *EgressRule) ProtoReflect() protoreflect.Message {
// Deprecated: Use EgressRule.ProtoReflect.Descriptor instead.
func (*EgressRule) Descriptor() ([]byte, []int) {
- return file_atelet_proto_rawDescGZIP(), []int{13}
+ return file_atelet_proto_rawDescGZIP(), []int{15}
}
func (x *EgressRule) GetHttp() *EgressDestinations {
@@ -1049,7 +1162,7 @@ type EgressDestinations struct {
func (x *EgressDestinations) Reset() {
*x = EgressDestinations{}
- mi := &file_atelet_proto_msgTypes[14]
+ mi := &file_atelet_proto_msgTypes[16]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -1061,7 +1174,7 @@ func (x *EgressDestinations) String() string {
func (*EgressDestinations) ProtoMessage() {}
func (x *EgressDestinations) ProtoReflect() protoreflect.Message {
- mi := &file_atelet_proto_msgTypes[14]
+ mi := &file_atelet_proto_msgTypes[16]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -1074,7 +1187,7 @@ func (x *EgressDestinations) ProtoReflect() protoreflect.Message {
// Deprecated: Use EgressDestinations.ProtoReflect.Descriptor instead.
func (*EgressDestinations) Descriptor() ([]byte, []int) {
- return file_atelet_proto_rawDescGZIP(), []int{14}
+ return file_atelet_proto_rawDescGZIP(), []int{16}
}
func (x *EgressDestinations) GetHostnames() []string {
@@ -1105,7 +1218,7 @@ type EgressPorts struct {
func (x *EgressPorts) Reset() {
*x = EgressPorts{}
- mi := &file_atelet_proto_msgTypes[15]
+ mi := &file_atelet_proto_msgTypes[17]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -1117,7 +1230,7 @@ func (x *EgressPorts) String() string {
func (*EgressPorts) ProtoMessage() {}
func (x *EgressPorts) ProtoReflect() protoreflect.Message {
- mi := &file_atelet_proto_msgTypes[15]
+ mi := &file_atelet_proto_msgTypes[17]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -1130,7 +1243,7 @@ func (x *EgressPorts) ProtoReflect() protoreflect.Message {
// Deprecated: Use EgressPorts.ProtoReflect.Descriptor instead.
func (*EgressPorts) Descriptor() ([]byte, []int) {
- return file_atelet_proto_rawDescGZIP(), []int{15}
+ return file_atelet_proto_rawDescGZIP(), []int{17}
}
func (x *EgressPorts) GetAll() *EgressAllPorts {
@@ -1156,7 +1269,7 @@ type EgressAllPorts struct {
func (x *EgressAllPorts) Reset() {
*x = EgressAllPorts{}
- mi := &file_atelet_proto_msgTypes[16]
+ mi := &file_atelet_proto_msgTypes[18]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -1168,7 +1281,7 @@ func (x *EgressAllPorts) String() string {
func (*EgressAllPorts) ProtoMessage() {}
func (x *EgressAllPorts) ProtoReflect() protoreflect.Message {
- mi := &file_atelet_proto_msgTypes[16]
+ mi := &file_atelet_proto_msgTypes[18]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -1181,7 +1294,7 @@ func (x *EgressAllPorts) ProtoReflect() protoreflect.Message {
// Deprecated: Use EgressAllPorts.ProtoReflect.Descriptor instead.
func (*EgressAllPorts) Descriptor() ([]byte, []int) {
- return file_atelet_proto_rawDescGZIP(), []int{16}
+ return file_atelet_proto_rawDescGZIP(), []int{18}
}
type MintActorCertificateRequest struct {
@@ -1190,29 +1303,29 @@ type MintActorCertificateRequest struct {
//
// +k8s:required
// +k8s:format=k8s-short-name
- ActorAtespace string `protobuf:"bytes,3,opt,name=actor_atespace,json=actorAtespace,proto3" json:"actor_atespace,omitempty"`
+ ActorAtespace string `protobuf:"bytes,1,opt,name=actor_atespace,json=actorAtespace,proto3" json:"actor_atespace,omitempty"`
// +k8s:required
// +k8s:format=k8s-short-name
- ActorName string `protobuf:"bytes,4,opt,name=actor_name,json=actorName,proto3" json:"actor_name,omitempty"`
+ ActorName string `protobuf:"bytes,2,opt,name=actor_name,json=actorName,proto3" json:"actor_name,omitempty"`
// The UID of the actor --- used to guard against deletion and recreation of
// an actor with the same name.
//
// +k8s:required
// +k8s:format=k8s-uuid
- ActorUid string `protobuf:"bytes,5,opt,name=actor_uid,json=actorUid,proto3" json:"actor_uid,omitempty"`
+ ActorUid string `protobuf:"bytes,3,opt,name=actor_uid,json=actorUid,proto3" json:"actor_uid,omitempty"`
// DER-encoded PKCS #10 certificate signing request. Atunnel retains the
// corresponding private key.
//
// +k8s:required
// +k8s:maxBytes=16384 # matches the control-plane MintActorCertificateRequest bound this is forwarded to
- CertificateSigningRequest []byte `protobuf:"bytes,1,opt,name=certificate_signing_request,json=certificateSigningRequest,proto3" json:"certificate_signing_request,omitempty"`
+ CertificateSigningRequest []byte `protobuf:"bytes,4,opt,name=certificate_signing_request,json=certificateSigningRequest,proto3" json:"certificate_signing_request,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *MintActorCertificateRequest) Reset() {
*x = MintActorCertificateRequest{}
- mi := &file_atelet_proto_msgTypes[17]
+ mi := &file_atelet_proto_msgTypes[19]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -1224,7 +1337,7 @@ func (x *MintActorCertificateRequest) String() string {
func (*MintActorCertificateRequest) ProtoMessage() {}
func (x *MintActorCertificateRequest) ProtoReflect() protoreflect.Message {
- mi := &file_atelet_proto_msgTypes[17]
+ mi := &file_atelet_proto_msgTypes[19]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -1237,7 +1350,7 @@ func (x *MintActorCertificateRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use MintActorCertificateRequest.ProtoReflect.Descriptor instead.
func (*MintActorCertificateRequest) Descriptor() ([]byte, []int) {
- return file_atelet_proto_rawDescGZIP(), []int{17}
+ return file_atelet_proto_rawDescGZIP(), []int{19}
}
func (x *MintActorCertificateRequest) GetActorAtespace() string {
@@ -1278,7 +1391,7 @@ type MintActorCertificateResponse struct {
func (x *MintActorCertificateResponse) Reset() {
*x = MintActorCertificateResponse{}
- mi := &file_atelet_proto_msgTypes[18]
+ mi := &file_atelet_proto_msgTypes[20]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -1290,7 +1403,7 @@ func (x *MintActorCertificateResponse) String() string {
func (*MintActorCertificateResponse) ProtoMessage() {}
func (x *MintActorCertificateResponse) ProtoReflect() protoreflect.Message {
- mi := &file_atelet_proto_msgTypes[18]
+ mi := &file_atelet_proto_msgTypes[20]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -1303,7 +1416,7 @@ func (x *MintActorCertificateResponse) ProtoReflect() protoreflect.Message {
// Deprecated: Use MintActorCertificateResponse.ProtoReflect.Descriptor instead.
func (*MintActorCertificateResponse) Descriptor() ([]byte, []int) {
- return file_atelet_proto_rawDescGZIP(), []int{18}
+ return file_atelet_proto_rawDescGZIP(), []int{20}
}
func (x *MintActorCertificateResponse) GetActorCertificates() [][]byte {
@@ -1349,7 +1462,7 @@ type TerminateRequest struct {
func (x *TerminateRequest) Reset() {
*x = TerminateRequest{}
- mi := &file_atelet_proto_msgTypes[19]
+ mi := &file_atelet_proto_msgTypes[21]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -1361,7 +1474,7 @@ func (x *TerminateRequest) String() string {
func (*TerminateRequest) ProtoMessage() {}
func (x *TerminateRequest) ProtoReflect() protoreflect.Message {
- mi := &file_atelet_proto_msgTypes[19]
+ mi := &file_atelet_proto_msgTypes[21]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -1374,7 +1487,7 @@ func (x *TerminateRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use TerminateRequest.ProtoReflect.Descriptor instead.
func (*TerminateRequest) Descriptor() ([]byte, []int) {
- return file_atelet_proto_rawDescGZIP(), []int{19}
+ return file_atelet_proto_rawDescGZIP(), []int{21}
}
func (x *TerminateRequest) GetWorkerPodUid() string {
@@ -1434,7 +1547,7 @@ type TerminateResponse struct {
func (x *TerminateResponse) Reset() {
*x = TerminateResponse{}
- mi := &file_atelet_proto_msgTypes[20]
+ mi := &file_atelet_proto_msgTypes[22]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -1446,7 +1559,7 @@ func (x *TerminateResponse) String() string {
func (*TerminateResponse) ProtoMessage() {}
func (x *TerminateResponse) ProtoReflect() protoreflect.Message {
- mi := &file_atelet_proto_msgTypes[20]
+ mi := &file_atelet_proto_msgTypes[22]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -1459,7 +1572,7 @@ func (x *TerminateResponse) ProtoReflect() protoreflect.Message {
// Deprecated: Use TerminateResponse.ProtoReflect.Descriptor instead.
func (*TerminateResponse) Descriptor() ([]byte, []int) {
- return file_atelet_proto_rawDescGZIP(), []int{20}
+ return file_atelet_proto_rawDescGZIP(), []int{22}
}
type RunRequest struct {
@@ -1488,12 +1601,6 @@ type RunRequest struct {
ActorTemplateName string `protobuf:"bytes,6,opt,name=actor_template_name,json=actorTemplateName,proto3" json:"actor_template_name,omitempty"`
// +k8s:required
Spec *WorkloadSpec `protobuf:"bytes,7,opt,name=spec,proto3" json:"spec,omitempty"`
- // The sandbox binaries to use for booting this actor from scratch. atelet
- // fetches the relevant assets and records them with the actor's on-node state
- // so a later Checkpoint can pin the same version into the snapshot manifest.
- //
- // +k8s:required
- SandboxAssets *SandboxAssets `protobuf:"bytes,8,opt,name=sandbox_assets,json=sandboxAssets,proto3" json:"sandbox_assets,omitempty"`
// When absent the actor has no egress: its TCP is captured and refused.
//
// +k8s:optional
@@ -1510,13 +1617,11 @@ type RunRequest struct {
// +k8s:minimum=0
MemoryBytes int64 `protobuf:"varint,11,opt,name=memory_bytes,json=memoryBytes,proto3" json:"memory_bytes,omitempty"` // Memory limit in bytes.
// The runtime the assigned worker runs this actor under, from
- // WorkerAssignment.sandbox_runtime.
- //
- // TODO: atelet still runs with sandbox_assets; a follow-up resolves the
- // SandboxConfig version from this field, makes it required, and removes
- // sandbox_assets.
+ // WorkerAssignment.sandbox_runtime. atelet resolves the pause image and
+ // sandbox binaries through its advertised-runtime index and records the
+ // runtime for the actor so a later Checkpoint pins the snapshot to it.
//
- // +k8s:optional
+ // +k8s:required
SandboxRuntime *SandboxRuntime `protobuf:"bytes,12,opt,name=sandbox_runtime,json=sandboxRuntime,proto3" json:"sandbox_runtime,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
@@ -1524,7 +1629,7 @@ type RunRequest struct {
func (x *RunRequest) Reset() {
*x = RunRequest{}
- mi := &file_atelet_proto_msgTypes[21]
+ mi := &file_atelet_proto_msgTypes[23]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -1536,7 +1641,7 @@ func (x *RunRequest) String() string {
func (*RunRequest) ProtoMessage() {}
func (x *RunRequest) ProtoReflect() protoreflect.Message {
- mi := &file_atelet_proto_msgTypes[21]
+ mi := &file_atelet_proto_msgTypes[23]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -1549,7 +1654,7 @@ func (x *RunRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use RunRequest.ProtoReflect.Descriptor instead.
func (*RunRequest) Descriptor() ([]byte, []int) {
- return file_atelet_proto_rawDescGZIP(), []int{21}
+ return file_atelet_proto_rawDescGZIP(), []int{23}
}
func (x *RunRequest) GetWorkerPodUid() string {
@@ -1601,13 +1706,6 @@ func (x *RunRequest) GetSpec() *WorkloadSpec {
return nil
}
-func (x *RunRequest) GetSandboxAssets() *SandboxAssets {
- if x != nil {
- return x.SandboxAssets
- }
- return nil
-}
-
func (x *RunRequest) GetEgressGateway() *EgressGateway {
if x != nil {
return x.EgressGateway
@@ -1637,216 +1735,34 @@ func (x *RunRequest) GetSandboxRuntime() *SandboxRuntime {
}
// EgressGateway configures tunneled egress for one actor activation.
-type EgressGateway struct {
- state protoimpl.MessageState `protogen:"open.v1"`
- // address is required and identifies the remote gateway as an IP address or
- // DNS name followed by its port.
- //
- // +k8s:required
- // +k8s:maxLength=261 # a bracketed or 253-character host, ':' and a 5-digit port
- // +k8s:customValidation # host:port shape
- Address string `protobuf:"bytes,1,opt,name=address,proto3" json:"address,omitempty"`
- unknownFields protoimpl.UnknownFields
- sizeCache protoimpl.SizeCache
-}
-
-func (x *EgressGateway) Reset() {
- *x = EgressGateway{}
- mi := &file_atelet_proto_msgTypes[22]
- ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
- ms.StoreMessageInfo(mi)
-}
-
-func (x *EgressGateway) String() string {
- return protoimpl.X.MessageStringOf(x)
-}
-
-func (*EgressGateway) ProtoMessage() {}
-
-func (x *EgressGateway) ProtoReflect() protoreflect.Message {
- mi := &file_atelet_proto_msgTypes[22]
- if x != nil {
- ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
- if ms.LoadMessageInfo() == nil {
- ms.StoreMessageInfo(mi)
- }
- return ms
- }
- return mi.MessageOf(x)
-}
-
-// Deprecated: Use EgressGateway.ProtoReflect.Descriptor instead.
-func (*EgressGateway) Descriptor() ([]byte, []int) {
- return file_atelet_proto_rawDescGZIP(), []int{22}
-}
-
-func (x *EgressGateway) GetAddress() string {
- if x != nil {
- return x.Address
- }
- return ""
-}
-
-// AssetFile is one content-addressed file atelet fetches for a sandbox runtime
-// (e.g. the gVisor release tarball).
-type AssetFile struct {
- state protoimpl.MessageState `protogen:"open.v1"`
- // URL to download the asset from. atelet fetches every URL through the
- // node's object-store plugin, and reads a gs:// URL anonymously only when
- // the plugin cannot serve it.
- // Masked in logged requests, since a signed URL carries its credential in
- // the query.
- //
- // +k8s:required
- // +k8s:maxLength=2048 # matches the SandboxConfig asset url's bound
- Url string `protobuf:"bytes,1,opt,name=url,proto3" json:"url,omitempty"`
- // Lower-case hex SHA256; names the cached file and verifies its integrity.
- //
- // +k8s:required
- // +k8s:maxLength=64
- // +k8s:customValidation # 64 hex characters
- Sha256 string `protobuf:"bytes,2,opt,name=sha256,proto3" json:"sha256,omitempty"`
- unknownFields protoimpl.UnknownFields
- sizeCache protoimpl.SizeCache
-}
-
-func (x *AssetFile) Reset() {
- *x = AssetFile{}
- mi := &file_atelet_proto_msgTypes[23]
- ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
- ms.StoreMessageInfo(mi)
-}
-
-func (x *AssetFile) String() string {
- return protoimpl.X.MessageStringOf(x)
-}
-
-func (*AssetFile) ProtoMessage() {}
-
-func (x *AssetFile) ProtoReflect() protoreflect.Message {
- mi := &file_atelet_proto_msgTypes[23]
- if x != nil {
- ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
- if ms.LoadMessageInfo() == nil {
- ms.StoreMessageInfo(mi)
- }
- return ms
- }
- return mi.MessageOf(x)
-}
-
-// Deprecated: Use AssetFile.ProtoReflect.Descriptor instead.
-func (*AssetFile) Descriptor() ([]byte, []int) {
- return file_atelet_proto_rawDescGZIP(), []int{23}
-}
-
-func (x *AssetFile) GetUrl() string {
- if x != nil {
- return x.Url
- }
- return ""
-}
-
-func (x *AssetFile) GetSha256() string {
- if x != nil {
- return x.Sha256
- }
- return ""
-}
-
-// ArchAssets is the set of assets for a single architecture, keyed by asset
-// name (a wrapper message because proto map values cannot themselves be maps).
-type ArchAssets struct {
- state protoimpl.MessageState `protogen:"open.v1"`
- // +k8s:required
- // +k8s:maxProperties=16
- // +k8s:eachKey=+k8s:maxLength=64
- Files map[string]*AssetFile `protobuf:"bytes,1,rep,name=files,proto3" json:"files,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"` // asset name -> file
- unknownFields protoimpl.UnknownFields
- sizeCache protoimpl.SizeCache
-}
-
-func (x *ArchAssets) Reset() {
- *x = ArchAssets{}
- mi := &file_atelet_proto_msgTypes[24]
- ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
- ms.StoreMessageInfo(mi)
-}
-
-func (x *ArchAssets) String() string {
- return protoimpl.X.MessageStringOf(x)
-}
-
-func (*ArchAssets) ProtoMessage() {}
-
-func (x *ArchAssets) ProtoReflect() protoreflect.Message {
- mi := &file_atelet_proto_msgTypes[24]
- if x != nil {
- ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
- if ms.LoadMessageInfo() == nil {
- ms.StoreMessageInfo(mi)
- }
- return ms
- }
- return mi.MessageOf(x)
-}
-
-// Deprecated: Use ArchAssets.ProtoReflect.Descriptor instead.
-func (*ArchAssets) Descriptor() ([]byte, []int) {
- return file_atelet_proto_rawDescGZIP(), []int{24}
-}
-
-func (x *ArchAssets) GetFiles() map[string]*AssetFile {
- if x != nil {
- return x.Files
- }
- return nil
-}
-
-// SandboxAssets is the generic, backend-agnostic description of the sandbox
-// an actor runs in: a sandbox class, the pause image holding its namespaces,
-// and binaries keyed first by architecture (GOARCH) and then by asset name.
-// atelet's backend code interprets the asset names (gVisor expects "gvisor",
-// the release tarball).
-type SandboxAssets struct {
+type EgressGateway struct {
state protoimpl.MessageState `protogen:"open.v1"`
- // +k8s:required
- // +k8s:maxLength=63 # guardrail; the class names are a few characters
- // +k8s:customValidation # gvisor or microvm
- SandboxClass string `protobuf:"bytes,1,opt,name=sandbox_class,json=sandboxClass,proto3" json:"sandbox_class,omitempty"` // e.g. "gvisor"
- // +k8s:required
- // +k8s:maxProperties=8 # one entry per architecture
- // +k8s:eachKey=+k8s:maxLength=16
- Assets map[string]*ArchAssets `protobuf:"bytes,2,rep,name=assets,proto3" json:"assets,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"` // arch -> {name -> file}
- // pause_image is the image for the sandbox's root container. Like the
- // binaries above it is sandbox configuration, not workload configuration,
- // and atelet pins it into the snapshot manifest so a restore rebuilds the
- // sandbox from the same image. Empty for sandboxes without a pause
- // container (microvm).
+ // address is required and identifies the remote gateway as an IP address or
+ // DNS name followed by its port.
//
- // +k8s:optional
- // +k8s:maxLength=512
- // +k8s:customValidation # must be a well-formed image reference, pinned by digest
- PauseImage string `protobuf:"bytes,3,opt,name=pause_image,json=pauseImage,proto3" json:"pause_image,omitempty"`
+ // +k8s:required
+ // +k8s:maxLength=261 # a bracketed or 253-character host, ':' and a 5-digit port
+ // +k8s:customValidation # host:port shape
+ Address string `protobuf:"bytes,1,opt,name=address,proto3" json:"address,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
-func (x *SandboxAssets) Reset() {
- *x = SandboxAssets{}
- mi := &file_atelet_proto_msgTypes[25]
+func (x *EgressGateway) Reset() {
+ *x = EgressGateway{}
+ mi := &file_atelet_proto_msgTypes[24]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
-func (x *SandboxAssets) String() string {
+func (x *EgressGateway) String() string {
return protoimpl.X.MessageStringOf(x)
}
-func (*SandboxAssets) ProtoMessage() {}
+func (*EgressGateway) ProtoMessage() {}
-func (x *SandboxAssets) ProtoReflect() protoreflect.Message {
- mi := &file_atelet_proto_msgTypes[25]
+func (x *EgressGateway) ProtoReflect() protoreflect.Message {
+ mi := &file_atelet_proto_msgTypes[24]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -1857,28 +1773,14 @@ func (x *SandboxAssets) ProtoReflect() protoreflect.Message {
return mi.MessageOf(x)
}
-// Deprecated: Use SandboxAssets.ProtoReflect.Descriptor instead.
-func (*SandboxAssets) Descriptor() ([]byte, []int) {
- return file_atelet_proto_rawDescGZIP(), []int{25}
-}
-
-func (x *SandboxAssets) GetSandboxClass() string {
- if x != nil {
- return x.SandboxClass
- }
- return ""
-}
-
-func (x *SandboxAssets) GetAssets() map[string]*ArchAssets {
- if x != nil {
- return x.Assets
- }
- return nil
+// Deprecated: Use EgressGateway.ProtoReflect.Descriptor instead.
+func (*EgressGateway) Descriptor() ([]byte, []int) {
+ return file_atelet_proto_rawDescGZIP(), []int{24}
}
-func (x *SandboxAssets) GetPauseImage() string {
+func (x *EgressGateway) GetAddress() string {
if x != nil {
- return x.PauseImage
+ return x.Address
}
return ""
}
@@ -1904,7 +1806,7 @@ type WorkloadSpec struct {
func (x *WorkloadSpec) Reset() {
*x = WorkloadSpec{}
- mi := &file_atelet_proto_msgTypes[26]
+ mi := &file_atelet_proto_msgTypes[25]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -1916,7 +1818,7 @@ func (x *WorkloadSpec) String() string {
func (*WorkloadSpec) ProtoMessage() {}
func (x *WorkloadSpec) ProtoReflect() protoreflect.Message {
- mi := &file_atelet_proto_msgTypes[26]
+ mi := &file_atelet_proto_msgTypes[25]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -1929,7 +1831,7 @@ func (x *WorkloadSpec) ProtoReflect() protoreflect.Message {
// Deprecated: Use WorkloadSpec.ProtoReflect.Descriptor instead.
func (*WorkloadSpec) Descriptor() ([]byte, []int) {
- return file_atelet_proto_rawDescGZIP(), []int{26}
+ return file_atelet_proto_rawDescGZIP(), []int{25}
}
func (x *WorkloadSpec) GetContainers() []*Container {
@@ -1954,7 +1856,7 @@ type DurableDirVolumeSource struct {
func (x *DurableDirVolumeSource) Reset() {
*x = DurableDirVolumeSource{}
- mi := &file_atelet_proto_msgTypes[27]
+ mi := &file_atelet_proto_msgTypes[26]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -1966,7 +1868,7 @@ func (x *DurableDirVolumeSource) String() string {
func (*DurableDirVolumeSource) ProtoMessage() {}
func (x *DurableDirVolumeSource) ProtoReflect() protoreflect.Message {
- mi := &file_atelet_proto_msgTypes[27]
+ mi := &file_atelet_proto_msgTypes[26]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -1979,7 +1881,7 @@ func (x *DurableDirVolumeSource) ProtoReflect() protoreflect.Message {
// Deprecated: Use DurableDirVolumeSource.ProtoReflect.Descriptor instead.
func (*DurableDirVolumeSource) Descriptor() ([]byte, []int) {
- return file_atelet_proto_rawDescGZIP(), []int{27}
+ return file_atelet_proto_rawDescGZIP(), []int{26}
}
type ExternalVolumeSource struct {
@@ -1988,11 +1890,11 @@ type ExternalVolumeSource struct {
// sends volumes that finished provisioning.
//
// +k8s:required
- // +k8s:maxLength=256 # matches ExternalVolume.storage_volume_id's bound
+ // +k8s:maxLength=256 # matches ExternalVolumeStatus.storage_volume_id's bound
// +k8s:customValidation # no control characters
StorageVolumeId string `protobuf:"bytes,1,opt,name=storage_volume_id,json=storageVolumeId,proto3" json:"storage_volume_id,omitempty"`
// +k8s:optional
- // +k8s:maxLength=253 # matches ExternalVolume.volume_type's bound
+ // +k8s:maxLength=253 # matches ExternalVolumeStatus.volume_type's bound
// +k8s:customValidation # optional "substrate.io/" prefix + DNS subdomain
VolumeType string `protobuf:"bytes,2,opt,name=volume_type,json=volumeType,proto3" json:"volume_type,omitempty"`
// volume_context is the driver's provisioning metadata. Its entries are the
@@ -2013,7 +1915,7 @@ type ExternalVolumeSource struct {
func (x *ExternalVolumeSource) Reset() {
*x = ExternalVolumeSource{}
- mi := &file_atelet_proto_msgTypes[28]
+ mi := &file_atelet_proto_msgTypes[27]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -2025,7 +1927,7 @@ func (x *ExternalVolumeSource) String() string {
func (*ExternalVolumeSource) ProtoMessage() {}
func (x *ExternalVolumeSource) ProtoReflect() protoreflect.Message {
- mi := &file_atelet_proto_msgTypes[28]
+ mi := &file_atelet_proto_msgTypes[27]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -2038,7 +1940,7 @@ func (x *ExternalVolumeSource) ProtoReflect() protoreflect.Message {
// Deprecated: Use ExternalVolumeSource.ProtoReflect.Descriptor instead.
func (*ExternalVolumeSource) Descriptor() ([]byte, []int) {
- return file_atelet_proto_rawDescGZIP(), []int{28}
+ return file_atelet_proto_rawDescGZIP(), []int{27}
}
func (x *ExternalVolumeSource) GetStorageVolumeId() string {
@@ -2083,7 +1985,7 @@ type ImageVolumeSource struct {
func (x *ImageVolumeSource) Reset() {
*x = ImageVolumeSource{}
- mi := &file_atelet_proto_msgTypes[29]
+ mi := &file_atelet_proto_msgTypes[28]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -2095,7 +1997,7 @@ func (x *ImageVolumeSource) String() string {
func (*ImageVolumeSource) ProtoMessage() {}
func (x *ImageVolumeSource) ProtoReflect() protoreflect.Message {
- mi := &file_atelet_proto_msgTypes[29]
+ mi := &file_atelet_proto_msgTypes[28]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -2108,7 +2010,7 @@ func (x *ImageVolumeSource) ProtoReflect() protoreflect.Message {
// Deprecated: Use ImageVolumeSource.ProtoReflect.Descriptor instead.
func (*ImageVolumeSource) Descriptor() ([]byte, []int) {
- return file_atelet_proto_rawDescGZIP(), []int{29}
+ return file_atelet_proto_rawDescGZIP(), []int{28}
}
func (x *ImageVolumeSource) GetReference() string {
@@ -2140,7 +2042,7 @@ type ActorMetadataItem struct {
func (x *ActorMetadataItem) Reset() {
*x = ActorMetadataItem{}
- mi := &file_atelet_proto_msgTypes[30]
+ mi := &file_atelet_proto_msgTypes[29]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -2152,7 +2054,7 @@ func (x *ActorMetadataItem) String() string {
func (*ActorMetadataItem) ProtoMessage() {}
func (x *ActorMetadataItem) ProtoReflect() protoreflect.Message {
- mi := &file_atelet_proto_msgTypes[30]
+ mi := &file_atelet_proto_msgTypes[29]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -2165,7 +2067,7 @@ func (x *ActorMetadataItem) ProtoReflect() protoreflect.Message {
// Deprecated: Use ActorMetadataItem.ProtoReflect.Descriptor instead.
func (*ActorMetadataItem) Descriptor() ([]byte, []int) {
- return file_atelet_proto_rawDescGZIP(), []int{30}
+ return file_atelet_proto_rawDescGZIP(), []int{29}
}
func (x *ActorMetadataItem) GetField() ActorMetadataField {
@@ -2201,7 +2103,7 @@ type ActorMetadataDataSource struct {
func (x *ActorMetadataDataSource) Reset() {
*x = ActorMetadataDataSource{}
- mi := &file_atelet_proto_msgTypes[31]
+ mi := &file_atelet_proto_msgTypes[30]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -2213,7 +2115,7 @@ func (x *ActorMetadataDataSource) String() string {
func (*ActorMetadataDataSource) ProtoMessage() {}
func (x *ActorMetadataDataSource) ProtoReflect() protoreflect.Message {
- mi := &file_atelet_proto_msgTypes[31]
+ mi := &file_atelet_proto_msgTypes[30]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -2226,7 +2128,7 @@ func (x *ActorMetadataDataSource) ProtoReflect() protoreflect.Message {
// Deprecated: Use ActorMetadataDataSource.ProtoReflect.Descriptor instead.
func (*ActorMetadataDataSource) Descriptor() ([]byte, []int) {
- return file_atelet_proto_rawDescGZIP(), []int{31}
+ return file_atelet_proto_rawDescGZIP(), []int{30}
}
func (x *ActorMetadataDataSource) GetItems() []*ActorMetadataItem {
@@ -2253,14 +2155,14 @@ type TrustBundleDataSource struct {
// +k8s:listType=set
// +k8s:eachVal=+k8s:minLength=1
// +k8s:eachVal=+k8s:maxLength=253 # matches the template's bundle-name bound
- Names []string `protobuf:"bytes,3,rep,name=names,proto3" json:"names,omitempty"`
+ Names []string `protobuf:"bytes,2,rep,name=names,proto3" json:"names,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *TrustBundleDataSource) Reset() {
*x = TrustBundleDataSource{}
- mi := &file_atelet_proto_msgTypes[32]
+ mi := &file_atelet_proto_msgTypes[31]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -2272,7 +2174,7 @@ func (x *TrustBundleDataSource) String() string {
func (*TrustBundleDataSource) ProtoMessage() {}
func (x *TrustBundleDataSource) ProtoReflect() protoreflect.Message {
- mi := &file_atelet_proto_msgTypes[32]
+ mi := &file_atelet_proto_msgTypes[31]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -2285,7 +2187,7 @@ func (x *TrustBundleDataSource) ProtoReflect() protoreflect.Message {
// Deprecated: Use TrustBundleDataSource.ProtoReflect.Descriptor instead.
func (*TrustBundleDataSource) Descriptor() ([]byte, []int) {
- return file_atelet_proto_rawDescGZIP(), []int{32}
+ return file_atelet_proto_rawDescGZIP(), []int{31}
}
func (x *TrustBundleDataSource) GetPath() string {
@@ -2320,7 +2222,7 @@ type SystemInfoDataSource struct {
func (x *SystemInfoDataSource) Reset() {
*x = SystemInfoDataSource{}
- mi := &file_atelet_proto_msgTypes[33]
+ mi := &file_atelet_proto_msgTypes[32]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -2332,7 +2234,7 @@ func (x *SystemInfoDataSource) String() string {
func (*SystemInfoDataSource) ProtoMessage() {}
func (x *SystemInfoDataSource) ProtoReflect() protoreflect.Message {
- mi := &file_atelet_proto_msgTypes[33]
+ mi := &file_atelet_proto_msgTypes[32]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -2345,7 +2247,7 @@ func (x *SystemInfoDataSource) ProtoReflect() protoreflect.Message {
// Deprecated: Use SystemInfoDataSource.ProtoReflect.Descriptor instead.
func (*SystemInfoDataSource) Descriptor() ([]byte, []int) {
- return file_atelet_proto_rawDescGZIP(), []int{33}
+ return file_atelet_proto_rawDescGZIP(), []int{32}
}
func (x *SystemInfoDataSource) GetActorMetadata() *ActorMetadataDataSource {
@@ -2378,7 +2280,7 @@ type SystemInfoVolumeSource struct {
func (x *SystemInfoVolumeSource) Reset() {
*x = SystemInfoVolumeSource{}
- mi := &file_atelet_proto_msgTypes[34]
+ mi := &file_atelet_proto_msgTypes[33]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -2390,7 +2292,7 @@ func (x *SystemInfoVolumeSource) String() string {
func (*SystemInfoVolumeSource) ProtoMessage() {}
func (x *SystemInfoVolumeSource) ProtoReflect() protoreflect.Message {
- mi := &file_atelet_proto_msgTypes[34]
+ mi := &file_atelet_proto_msgTypes[33]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -2403,7 +2305,7 @@ func (x *SystemInfoVolumeSource) ProtoReflect() protoreflect.Message {
// Deprecated: Use SystemInfoVolumeSource.ProtoReflect.Descriptor instead.
func (*SystemInfoVolumeSource) Descriptor() ([]byte, []int) {
- return file_atelet_proto_rawDescGZIP(), []int{34}
+ return file_atelet_proto_rawDescGZIP(), []int{33}
}
func (x *SystemInfoVolumeSource) GetDataSources() []*SystemInfoDataSource {
@@ -2439,7 +2341,7 @@ type Volume struct {
func (x *Volume) Reset() {
*x = Volume{}
- mi := &file_atelet_proto_msgTypes[35]
+ mi := &file_atelet_proto_msgTypes[34]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -2451,7 +2353,7 @@ func (x *Volume) String() string {
func (*Volume) ProtoMessage() {}
func (x *Volume) ProtoReflect() protoreflect.Message {
- mi := &file_atelet_proto_msgTypes[35]
+ mi := &file_atelet_proto_msgTypes[34]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -2464,7 +2366,7 @@ func (x *Volume) ProtoReflect() protoreflect.Message {
// Deprecated: Use Volume.ProtoReflect.Descriptor instead.
func (*Volume) Descriptor() ([]byte, []int) {
- return file_atelet_proto_rawDescGZIP(), []int{35}
+ return file_atelet_proto_rawDescGZIP(), []int{34}
}
func (x *Volume) GetName() string {
@@ -2521,7 +2423,7 @@ type VolumeMount struct {
func (x *VolumeMount) Reset() {
*x = VolumeMount{}
- mi := &file_atelet_proto_msgTypes[36]
+ mi := &file_atelet_proto_msgTypes[35]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -2533,7 +2435,7 @@ func (x *VolumeMount) String() string {
func (*VolumeMount) ProtoMessage() {}
func (x *VolumeMount) ProtoReflect() protoreflect.Message {
- mi := &file_atelet_proto_msgTypes[36]
+ mi := &file_atelet_proto_msgTypes[35]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -2546,7 +2448,7 @@ func (x *VolumeMount) ProtoReflect() protoreflect.Message {
// Deprecated: Use VolumeMount.ProtoReflect.Descriptor instead.
func (*VolumeMount) Descriptor() ([]byte, []int) {
- return file_atelet_proto_rawDescGZIP(), []int{36}
+ return file_atelet_proto_rawDescGZIP(), []int{35}
}
func (x *VolumeMount) GetName() string {
@@ -2611,7 +2513,7 @@ type Container struct {
func (x *Container) Reset() {
*x = Container{}
- mi := &file_atelet_proto_msgTypes[37]
+ mi := &file_atelet_proto_msgTypes[36]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -2623,7 +2525,7 @@ func (x *Container) String() string {
func (*Container) ProtoMessage() {}
func (x *Container) ProtoReflect() protoreflect.Message {
- mi := &file_atelet_proto_msgTypes[37]
+ mi := &file_atelet_proto_msgTypes[36]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -2636,7 +2538,7 @@ func (x *Container) ProtoReflect() protoreflect.Message {
// Deprecated: Use Container.ProtoReflect.Descriptor instead.
func (*Container) Descriptor() ([]byte, []int) {
- return file_atelet_proto_rawDescGZIP(), []int{37}
+ return file_atelet_proto_rawDescGZIP(), []int{36}
}
func (x *Container) GetName() string {
@@ -2713,7 +2615,7 @@ type SecurityContext struct {
func (x *SecurityContext) Reset() {
*x = SecurityContext{}
- mi := &file_atelet_proto_msgTypes[38]
+ mi := &file_atelet_proto_msgTypes[37]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -2725,7 +2627,7 @@ func (x *SecurityContext) String() string {
func (*SecurityContext) ProtoMessage() {}
func (x *SecurityContext) ProtoReflect() protoreflect.Message {
- mi := &file_atelet_proto_msgTypes[38]
+ mi := &file_atelet_proto_msgTypes[37]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -2738,7 +2640,7 @@ func (x *SecurityContext) ProtoReflect() protoreflect.Message {
// Deprecated: Use SecurityContext.ProtoReflect.Descriptor instead.
func (*SecurityContext) Descriptor() ([]byte, []int) {
- return file_atelet_proto_rawDescGZIP(), []int{38}
+ return file_atelet_proto_rawDescGZIP(), []int{37}
}
func (x *SecurityContext) GetCapabilities() *Capabilities {
@@ -2768,7 +2670,7 @@ type Capabilities struct {
func (x *Capabilities) Reset() {
*x = Capabilities{}
- mi := &file_atelet_proto_msgTypes[39]
+ mi := &file_atelet_proto_msgTypes[38]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -2780,7 +2682,7 @@ func (x *Capabilities) String() string {
func (*Capabilities) ProtoMessage() {}
func (x *Capabilities) ProtoReflect() protoreflect.Message {
- mi := &file_atelet_proto_msgTypes[39]
+ mi := &file_atelet_proto_msgTypes[38]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -2793,7 +2695,7 @@ func (x *Capabilities) ProtoReflect() protoreflect.Message {
// Deprecated: Use Capabilities.ProtoReflect.Descriptor instead.
func (*Capabilities) Descriptor() ([]byte, []int) {
- return file_atelet_proto_rawDescGZIP(), []int{39}
+ return file_atelet_proto_rawDescGZIP(), []int{38}
}
func (x *Capabilities) GetAdd() []string {
@@ -2831,7 +2733,7 @@ type ResourceLimits struct {
func (x *ResourceLimits) Reset() {
*x = ResourceLimits{}
- mi := &file_atelet_proto_msgTypes[40]
+ mi := &file_atelet_proto_msgTypes[39]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -2843,7 +2745,7 @@ func (x *ResourceLimits) String() string {
func (*ResourceLimits) ProtoMessage() {}
func (x *ResourceLimits) ProtoReflect() protoreflect.Message {
- mi := &file_atelet_proto_msgTypes[40]
+ mi := &file_atelet_proto_msgTypes[39]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -2856,7 +2758,7 @@ func (x *ResourceLimits) ProtoReflect() protoreflect.Message {
// Deprecated: Use ResourceLimits.ProtoReflect.Descriptor instead.
func (*ResourceLimits) Descriptor() ([]byte, []int) {
- return file_atelet_proto_rawDescGZIP(), []int{40}
+ return file_atelet_proto_rawDescGZIP(), []int{39}
}
func (x *ResourceLimits) GetMemoryBytes() int64 {
@@ -2890,7 +2792,7 @@ type EnvVar struct {
func (x *EnvVar) Reset() {
*x = EnvVar{}
- mi := &file_atelet_proto_msgTypes[41]
+ mi := &file_atelet_proto_msgTypes[40]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -2902,7 +2804,7 @@ func (x *EnvVar) String() string {
func (*EnvVar) ProtoMessage() {}
func (x *EnvVar) ProtoReflect() protoreflect.Message {
- mi := &file_atelet_proto_msgTypes[41]
+ mi := &file_atelet_proto_msgTypes[40]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -2915,7 +2817,7 @@ func (x *EnvVar) ProtoReflect() protoreflect.Message {
// Deprecated: Use EnvVar.ProtoReflect.Descriptor instead.
func (*EnvVar) Descriptor() ([]byte, []int) {
- return file_atelet_proto_rawDescGZIP(), []int{41}
+ return file_atelet_proto_rawDescGZIP(), []int{40}
}
func (x *EnvVar) GetName() string {
@@ -2950,7 +2852,7 @@ type WakeupProbe struct {
func (x *WakeupProbe) Reset() {
*x = WakeupProbe{}
- mi := &file_atelet_proto_msgTypes[42]
+ mi := &file_atelet_proto_msgTypes[41]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -2962,7 +2864,7 @@ func (x *WakeupProbe) String() string {
func (*WakeupProbe) ProtoMessage() {}
func (x *WakeupProbe) ProtoReflect() protoreflect.Message {
- mi := &file_atelet_proto_msgTypes[42]
+ mi := &file_atelet_proto_msgTypes[41]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -2975,7 +2877,7 @@ func (x *WakeupProbe) ProtoReflect() protoreflect.Message {
// Deprecated: Use WakeupProbe.ProtoReflect.Descriptor instead.
func (*WakeupProbe) Descriptor() ([]byte, []int) {
- return file_atelet_proto_rawDescGZIP(), []int{42}
+ return file_atelet_proto_rawDescGZIP(), []int{41}
}
func (x *WakeupProbe) GetHttpGet() *HttpGetAction {
@@ -3013,7 +2915,7 @@ type HttpGetAction struct {
func (x *HttpGetAction) Reset() {
*x = HttpGetAction{}
- mi := &file_atelet_proto_msgTypes[43]
+ mi := &file_atelet_proto_msgTypes[42]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -3025,7 +2927,7 @@ func (x *HttpGetAction) String() string {
func (*HttpGetAction) ProtoMessage() {}
func (x *HttpGetAction) ProtoReflect() protoreflect.Message {
- mi := &file_atelet_proto_msgTypes[43]
+ mi := &file_atelet_proto_msgTypes[42]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -3038,7 +2940,7 @@ func (x *HttpGetAction) ProtoReflect() protoreflect.Message {
// Deprecated: Use HttpGetAction.ProtoReflect.Descriptor instead.
func (*HttpGetAction) Descriptor() ([]byte, []int) {
- return file_atelet_proto_rawDescGZIP(), []int{43}
+ return file_atelet_proto_rawDescGZIP(), []int{42}
}
func (x *HttpGetAction) GetPath() string {
@@ -3063,7 +2965,7 @@ type RunResponse struct {
func (x *RunResponse) Reset() {
*x = RunResponse{}
- mi := &file_atelet_proto_msgTypes[44]
+ mi := &file_atelet_proto_msgTypes[43]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -3075,7 +2977,7 @@ func (x *RunResponse) String() string {
func (*RunResponse) ProtoMessage() {}
func (x *RunResponse) ProtoReflect() protoreflect.Message {
- mi := &file_atelet_proto_msgTypes[44]
+ mi := &file_atelet_proto_msgTypes[43]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -3088,7 +2990,7 @@ func (x *RunResponse) ProtoReflect() protoreflect.Message {
// Deprecated: Use RunResponse.ProtoReflect.Descriptor instead.
func (*RunResponse) Descriptor() ([]byte, []int) {
- return file_atelet_proto_rawDescGZIP(), []int{44}
+ return file_atelet_proto_rawDescGZIP(), []int{43}
}
type ObjectStorage struct {
@@ -3106,7 +3008,7 @@ type ObjectStorage struct {
func (x *ObjectStorage) Reset() {
*x = ObjectStorage{}
- mi := &file_atelet_proto_msgTypes[45]
+ mi := &file_atelet_proto_msgTypes[44]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -3118,7 +3020,7 @@ func (x *ObjectStorage) String() string {
func (*ObjectStorage) ProtoMessage() {}
func (x *ObjectStorage) ProtoReflect() protoreflect.Message {
- mi := &file_atelet_proto_msgTypes[45]
+ mi := &file_atelet_proto_msgTypes[44]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -3131,7 +3033,7 @@ func (x *ObjectStorage) ProtoReflect() protoreflect.Message {
// Deprecated: Use ObjectStorage.ProtoReflect.Descriptor instead.
func (*ObjectStorage) Descriptor() ([]byte, []int) {
- return file_atelet_proto_rawDescGZIP(), []int{45}
+ return file_atelet_proto_rawDescGZIP(), []int{44}
}
func (x *ObjectStorage) GetSnapshotUri() string {
@@ -3174,14 +3076,10 @@ type Snapshot struct {
// +k8s:minimum=1
// +k8s:maximum=3 # keep this in sync with the SnapshotFidelity enum
Fidelity SnapshotFidelity `protobuf:"varint,3,opt,name=fidelity,proto3,enum=atelet.SnapshotFidelity" json:"fidelity,omitempty"`
- // The snapshot's durable copy in object storage. Required on Checkpoint
- // and PromoteSnapshot: atelet uploads there, or reads from there when it
- // holds no usable local copy.
+ // The snapshot's durable copy in object storage: atelet uploads there, or
+ // reads from there when it holds no usable local copy.
//
- // TODO: Temporarily, this is optional on RestoreRequest while the control
- // plane does not set it when restoring a paused actor.
- //
- // +k8s:optional
+ // +k8s:required
Object *ObjectStorage `protobuf:"bytes,4,opt,name=object,proto3" json:"object,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
@@ -3189,7 +3087,7 @@ type Snapshot struct {
func (x *Snapshot) Reset() {
*x = Snapshot{}
- mi := &file_atelet_proto_msgTypes[46]
+ mi := &file_atelet_proto_msgTypes[45]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -3201,7 +3099,7 @@ func (x *Snapshot) String() string {
func (*Snapshot) ProtoMessage() {}
func (x *Snapshot) ProtoReflect() protoreflect.Message {
- mi := &file_atelet_proto_msgTypes[46]
+ mi := &file_atelet_proto_msgTypes[45]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -3214,7 +3112,7 @@ func (x *Snapshot) ProtoReflect() protoreflect.Message {
// Deprecated: Use Snapshot.ProtoReflect.Descriptor instead.
func (*Snapshot) Descriptor() ([]byte, []int) {
- return file_atelet_proto_rawDescGZIP(), []int{46}
+ return file_atelet_proto_rawDescGZIP(), []int{45}
}
func (x *Snapshot) GetSnapshotUuid() string {
@@ -3261,7 +3159,7 @@ type SnapshotReport struct {
func (x *SnapshotReport) Reset() {
*x = SnapshotReport{}
- mi := &file_atelet_proto_msgTypes[47]
+ mi := &file_atelet_proto_msgTypes[46]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -3273,7 +3171,7 @@ func (x *SnapshotReport) String() string {
func (*SnapshotReport) ProtoMessage() {}
func (x *SnapshotReport) ProtoReflect() protoreflect.Message {
- mi := &file_atelet_proto_msgTypes[47]
+ mi := &file_atelet_proto_msgTypes[46]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -3286,7 +3184,7 @@ func (x *SnapshotReport) ProtoReflect() protoreflect.Message {
// Deprecated: Use SnapshotReport.ProtoReflect.Descriptor instead.
func (*SnapshotReport) Descriptor() ([]byte, []int) {
- return file_atelet_proto_rawDescGZIP(), []int{47}
+ return file_atelet_proto_rawDescGZIP(), []int{46}
}
func (x *SnapshotReport) GetSnapshot() *Snapshot {
@@ -3342,7 +3240,7 @@ type ActorSnapshotState struct {
func (x *ActorSnapshotState) Reset() {
*x = ActorSnapshotState{}
- mi := &file_atelet_proto_msgTypes[48]
+ mi := &file_atelet_proto_msgTypes[47]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -3354,7 +3252,7 @@ func (x *ActorSnapshotState) String() string {
func (*ActorSnapshotState) ProtoMessage() {}
func (x *ActorSnapshotState) ProtoReflect() protoreflect.Message {
- mi := &file_atelet_proto_msgTypes[48]
+ mi := &file_atelet_proto_msgTypes[47]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -3367,7 +3265,7 @@ func (x *ActorSnapshotState) ProtoReflect() protoreflect.Message {
// Deprecated: Use ActorSnapshotState.ProtoReflect.Descriptor instead.
func (*ActorSnapshotState) Descriptor() ([]byte, []int) {
- return file_atelet_proto_rawDescGZIP(), []int{48}
+ return file_atelet_proto_rawDescGZIP(), []int{47}
}
func (x *ActorSnapshotState) GetLatestLocal() *SnapshotReport {
@@ -3411,7 +3309,7 @@ type UploadConfig struct {
func (x *UploadConfig) Reset() {
*x = UploadConfig{}
- mi := &file_atelet_proto_msgTypes[49]
+ mi := &file_atelet_proto_msgTypes[48]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -3423,7 +3321,7 @@ func (x *UploadConfig) String() string {
func (*UploadConfig) ProtoMessage() {}
func (x *UploadConfig) ProtoReflect() protoreflect.Message {
- mi := &file_atelet_proto_msgTypes[49]
+ mi := &file_atelet_proto_msgTypes[48]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -3436,7 +3334,7 @@ func (x *UploadConfig) ProtoReflect() protoreflect.Message {
// Deprecated: Use UploadConfig.ProtoReflect.Descriptor instead.
func (*UploadConfig) Descriptor() ([]byte, []int) {
- return file_atelet_proto_rawDescGZIP(), []int{49}
+ return file_atelet_proto_rawDescGZIP(), []int{48}
}
func (x *UploadConfig) GetMode() UploadMode {
@@ -3487,7 +3385,6 @@ type CheckpointRequest struct {
// workload.
//
// +k8s:required
- // +k8s:subfield(object)=+k8s:required
Snapshot *Snapshot `protobuf:"bytes,8,opt,name=snapshot,proto3" json:"snapshot,omitempty"`
// When the snapshot becomes durable.
//
@@ -3499,7 +3396,7 @@ type CheckpointRequest struct {
func (x *CheckpointRequest) Reset() {
*x = CheckpointRequest{}
- mi := &file_atelet_proto_msgTypes[50]
+ mi := &file_atelet_proto_msgTypes[49]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -3511,7 +3408,7 @@ func (x *CheckpointRequest) String() string {
func (*CheckpointRequest) ProtoMessage() {}
func (x *CheckpointRequest) ProtoReflect() protoreflect.Message {
- mi := &file_atelet_proto_msgTypes[50]
+ mi := &file_atelet_proto_msgTypes[49]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -3524,7 +3421,7 @@ func (x *CheckpointRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use CheckpointRequest.ProtoReflect.Descriptor instead.
func (*CheckpointRequest) Descriptor() ([]byte, []int) {
- return file_atelet_proto_rawDescGZIP(), []int{50}
+ return file_atelet_proto_rawDescGZIP(), []int{49}
}
func (x *CheckpointRequest) GetWorkerPodUid() string {
@@ -3608,7 +3505,7 @@ type CheckpointResponse struct {
func (x *CheckpointResponse) Reset() {
*x = CheckpointResponse{}
- mi := &file_atelet_proto_msgTypes[51]
+ mi := &file_atelet_proto_msgTypes[50]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -3620,7 +3517,7 @@ func (x *CheckpointResponse) String() string {
func (*CheckpointResponse) ProtoMessage() {}
func (x *CheckpointResponse) ProtoReflect() protoreflect.Message {
- mi := &file_atelet_proto_msgTypes[51]
+ mi := &file_atelet_proto_msgTypes[50]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -3633,7 +3530,7 @@ func (x *CheckpointResponse) ProtoReflect() protoreflect.Message {
// Deprecated: Use CheckpointResponse.ProtoReflect.Descriptor instead.
func (*CheckpointResponse) Descriptor() ([]byte, []int) {
- return file_atelet_proto_rawDescGZIP(), []int{51}
+ return file_atelet_proto_rawDescGZIP(), []int{50}
}
func (x *CheckpointResponse) GetState() *ActorSnapshotState {
@@ -3675,16 +3572,16 @@ type RestoreRequest struct {
// detaching the actor, it waits for the detach to finish and then reads
// `snapshot.object`. Fails if neither copy exists: that generation is lost.
//
- // TODO: Temporarily, the control plane does not set `snapshot.object` when
- // it restores a paused actor, so atelet has nothing to fall back to if that
- // actor's local copy is missing or unusable.
+ // `snapshot.object` is required even while the control plane has not
+ // recorded the snapshot's upload complete: atelet may have uploaded the
+ // snapshot and detached the actor before its report of the upload landed.
//
// +k8s:required
Snapshot *Snapshot `protobuf:"bytes,8,opt,name=snapshot,proto3" json:"snapshot,omitempty"`
// When absent the actor has no egress: its TCP is captured and refused.
//
// +k8s:optional
- EgressGateway *EgressGateway `protobuf:"bytes,13,opt,name=egress_gateway,json=egressGateway,proto3,oneof" json:"egress_gateway,omitempty"`
+ EgressGateway *EgressGateway `protobuf:"bytes,9,opt,name=egress_gateway,json=egressGateway,proto3,oneof" json:"egress_gateway,omitempty"`
// The actor's declared size, from the ActorTemplate's resource limits. For
// gVisor and micro-VM VOLUMES restores the sandbox is (re)sized to these;
// for a MEMORY micro-VM restore the size baked into the snapshot wins. Zero
@@ -3693,31 +3590,25 @@ type RestoreRequest struct {
// +k8s:optional
// +k8s:minimum=0
// +k8s:maximum=999999 # the control plane caps cpu limits strictly below 1000 cores
- CpuMilli int64 `protobuf:"varint,14,opt,name=cpu_milli,json=cpuMilli,proto3" json:"cpu_milli,omitempty"` // CPU limit in millicores (1000 = one core).
+ CpuMilli int64 `protobuf:"varint,10,opt,name=cpu_milli,json=cpuMilli,proto3" json:"cpu_milli,omitempty"` // CPU limit in millicores (1000 = one core).
// +k8s:optional
// +k8s:minimum=0
- MemoryBytes int64 `protobuf:"varint,15,opt,name=memory_bytes,json=memoryBytes,proto3" json:"memory_bytes,omitempty"` // Memory limit in bytes.
- // The sandbox binaries and pause image to restore with, resolved from the
- // assigned worker pool's SandboxConfig.
- //
- // +k8s:required
- SandboxAssets *SandboxAssets `protobuf:"bytes,16,opt,name=sandbox_assets,json=sandboxAssets,proto3" json:"sandbox_assets,omitempty"`
+ MemoryBytes int64 `protobuf:"varint,11,opt,name=memory_bytes,json=memoryBytes,proto3" json:"memory_bytes,omitempty"` // Memory limit in bytes.
// The runtime the assigned worker restores this actor under, from
- // WorkerAssignment.sandbox_runtime.
+ // WorkerAssignment.sandbox_runtime. atelet resolves the pause image and
+ // sandbox binaries through its advertised-runtime index and records the
+ // runtime for the actor so the next Checkpoint pins the snapshot to what ran
+ // this activation, not what the restored snapshot was taken under.
//
- // TODO: atelet still restores with sandbox_assets; a follow-up resolves the
- // SandboxConfig version from this field, makes it required, and removes
- // sandbox_assets.
- //
- // +k8s:optional
- SandboxRuntime *SandboxRuntime `protobuf:"bytes,17,opt,name=sandbox_runtime,json=sandboxRuntime,proto3" json:"sandbox_runtime,omitempty"`
+ // +k8s:required
+ SandboxRuntime *SandboxRuntime `protobuf:"bytes,12,opt,name=sandbox_runtime,json=sandboxRuntime,proto3" json:"sandbox_runtime,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *RestoreRequest) Reset() {
*x = RestoreRequest{}
- mi := &file_atelet_proto_msgTypes[52]
+ mi := &file_atelet_proto_msgTypes[51]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -3729,7 +3620,7 @@ func (x *RestoreRequest) String() string {
func (*RestoreRequest) ProtoMessage() {}
func (x *RestoreRequest) ProtoReflect() protoreflect.Message {
- mi := &file_atelet_proto_msgTypes[52]
+ mi := &file_atelet_proto_msgTypes[51]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -3742,7 +3633,7 @@ func (x *RestoreRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use RestoreRequest.ProtoReflect.Descriptor instead.
func (*RestoreRequest) Descriptor() ([]byte, []int) {
- return file_atelet_proto_rawDescGZIP(), []int{52}
+ return file_atelet_proto_rawDescGZIP(), []int{51}
}
func (x *RestoreRequest) GetWorkerPodUid() string {
@@ -3822,13 +3713,6 @@ func (x *RestoreRequest) GetMemoryBytes() int64 {
return 0
}
-func (x *RestoreRequest) GetSandboxAssets() *SandboxAssets {
- if x != nil {
- return x.SandboxAssets
- }
- return nil
-}
-
func (x *RestoreRequest) GetSandboxRuntime() *SandboxRuntime {
if x != nil {
return x.SandboxRuntime
@@ -3844,7 +3728,7 @@ type RestoreResponse struct {
func (x *RestoreResponse) Reset() {
*x = RestoreResponse{}
- mi := &file_atelet_proto_msgTypes[53]
+ mi := &file_atelet_proto_msgTypes[52]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -3856,7 +3740,7 @@ func (x *RestoreResponse) String() string {
func (*RestoreResponse) ProtoMessage() {}
func (x *RestoreResponse) ProtoReflect() protoreflect.Message {
- mi := &file_atelet_proto_msgTypes[53]
+ mi := &file_atelet_proto_msgTypes[52]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -3869,7 +3753,7 @@ func (x *RestoreResponse) ProtoReflect() protoreflect.Message {
// Deprecated: Use RestoreResponse.ProtoReflect.Descriptor instead.
func (*RestoreResponse) Descriptor() ([]byte, []int) {
- return file_atelet_proto_rawDescGZIP(), []int{53}
+ return file_atelet_proto_rawDescGZIP(), []int{52}
}
type PromoteSnapshotRequest struct {
@@ -3895,7 +3779,6 @@ type PromoteSnapshotRequest struct {
// it is, or will be, uploaded.
//
// +k8s:required
- // +k8s:subfield(object)=+k8s:required
Snapshot *Snapshot `protobuf:"bytes,6,opt,name=snapshot,proto3" json:"snapshot,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
@@ -3903,7 +3786,7 @@ type PromoteSnapshotRequest struct {
func (x *PromoteSnapshotRequest) Reset() {
*x = PromoteSnapshotRequest{}
- mi := &file_atelet_proto_msgTypes[54]
+ mi := &file_atelet_proto_msgTypes[53]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -3915,7 +3798,7 @@ func (x *PromoteSnapshotRequest) String() string {
func (*PromoteSnapshotRequest) ProtoMessage() {}
func (x *PromoteSnapshotRequest) ProtoReflect() protoreflect.Message {
- mi := &file_atelet_proto_msgTypes[54]
+ mi := &file_atelet_proto_msgTypes[53]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -3928,7 +3811,7 @@ func (x *PromoteSnapshotRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use PromoteSnapshotRequest.ProtoReflect.Descriptor instead.
func (*PromoteSnapshotRequest) Descriptor() ([]byte, []int) {
- return file_atelet_proto_rawDescGZIP(), []int{54}
+ return file_atelet_proto_rawDescGZIP(), []int{53}
}
func (x *PromoteSnapshotRequest) GetAtespace() string {
@@ -3984,7 +3867,7 @@ type PromoteSnapshotResponse struct {
func (x *PromoteSnapshotResponse) Reset() {
*x = PromoteSnapshotResponse{}
- mi := &file_atelet_proto_msgTypes[55]
+ mi := &file_atelet_proto_msgTypes[54]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -3996,7 +3879,7 @@ func (x *PromoteSnapshotResponse) String() string {
func (*PromoteSnapshotResponse) ProtoMessage() {}
func (x *PromoteSnapshotResponse) ProtoReflect() protoreflect.Message {
- mi := &file_atelet_proto_msgTypes[55]
+ mi := &file_atelet_proto_msgTypes[54]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -4009,7 +3892,7 @@ func (x *PromoteSnapshotResponse) ProtoReflect() protoreflect.Message {
// Deprecated: Use PromoteSnapshotResponse.ProtoReflect.Descriptor instead.
func (*PromoteSnapshotResponse) Descriptor() ([]byte, []int) {
- return file_atelet_proto_rawDescGZIP(), []int{55}
+ return file_atelet_proto_rawDescGZIP(), []int{54}
}
func (x *PromoteSnapshotResponse) GetState() *ActorSnapshotState {
@@ -4038,7 +3921,7 @@ type DetachActorRequest struct {
func (x *DetachActorRequest) Reset() {
*x = DetachActorRequest{}
- mi := &file_atelet_proto_msgTypes[56]
+ mi := &file_atelet_proto_msgTypes[55]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -4050,7 +3933,7 @@ func (x *DetachActorRequest) String() string {
func (*DetachActorRequest) ProtoMessage() {}
func (x *DetachActorRequest) ProtoReflect() protoreflect.Message {
- mi := &file_atelet_proto_msgTypes[56]
+ mi := &file_atelet_proto_msgTypes[55]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -4063,7 +3946,7 @@ func (x *DetachActorRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use DetachActorRequest.ProtoReflect.Descriptor instead.
func (*DetachActorRequest) Descriptor() ([]byte, []int) {
- return file_atelet_proto_rawDescGZIP(), []int{56}
+ return file_atelet_proto_rawDescGZIP(), []int{55}
}
func (x *DetachActorRequest) GetAtespace() string {
@@ -4098,7 +3981,7 @@ type DetachActorResponse struct {
func (x *DetachActorResponse) Reset() {
*x = DetachActorResponse{}
- mi := &file_atelet_proto_msgTypes[57]
+ mi := &file_atelet_proto_msgTypes[56]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -4110,7 +3993,7 @@ func (x *DetachActorResponse) String() string {
func (*DetachActorResponse) ProtoMessage() {}
func (x *DetachActorResponse) ProtoReflect() protoreflect.Message {
- mi := &file_atelet_proto_msgTypes[57]
+ mi := &file_atelet_proto_msgTypes[56]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -4123,7 +4006,7 @@ func (x *DetachActorResponse) ProtoReflect() protoreflect.Message {
// Deprecated: Use DetachActorResponse.ProtoReflect.Descriptor instead.
func (*DetachActorResponse) Descriptor() ([]byte, []int) {
- return file_atelet_proto_rawDescGZIP(), []int{57}
+ return file_atelet_proto_rawDescGZIP(), []int{56}
}
func (x *DetachActorResponse) GetState() *ActorSnapshotState {
@@ -4167,7 +4050,14 @@ const file_atelet_proto_rawDesc = "" +
"\n" +
"actor_name\x18\x02 \x01(\tR\tactorName\x12\x1b\n" +
"\tactor_uid\x18\x03 \x01(\tR\bactorUid\"\x1d\n" +
- "\x1bRequestActorSuspendResponse\"c\n" +
+ "\x1bRequestActorSuspendResponse\"\x98\x01\n" +
+ "\x19ReportActorCrashedRequest\x12%\n" +
+ "\x0eactor_atespace\x18\x01 \x01(\tR\ractorAtespace\x12\x1d\n" +
+ "\n" +
+ "actor_name\x18\x02 \x01(\tR\tactorName\x12\x1b\n" +
+ "\tactor_uid\x18\x03 \x01(\tR\bactorUid\x12\x18\n" +
+ "\amessage\x18\x04 \x01(\tR\amessage\"\x1c\n" +
+ "\x1aReportActorCrashedResponse\"c\n" +
"\x1bGetActorEgressPolicyRequest\x12%\n" +
"\x0eactor_atespace\x18\x01 \x01(\tR\ractorAtespace\x12\x1d\n" +
"\n" +
@@ -4189,11 +4079,11 @@ const file_atelet_proto_rawDesc = "" +
"\anumbers\x18\x02 \x03(\x05R\anumbers\"\x10\n" +
"\x0eEgressAllPorts\"\xc0\x01\n" +
"\x1bMintActorCertificateRequest\x12%\n" +
- "\x0eactor_atespace\x18\x03 \x01(\tR\ractorAtespace\x12\x1d\n" +
+ "\x0eactor_atespace\x18\x01 \x01(\tR\ractorAtespace\x12\x1d\n" +
"\n" +
- "actor_name\x18\x04 \x01(\tR\tactorName\x12\x1b\n" +
- "\tactor_uid\x18\x05 \x01(\tR\bactorUid\x12>\n" +
- "\x1bcertificate_signing_request\x18\x01 \x01(\fR\x19certificateSigningRequest\"M\n" +
+ "actor_name\x18\x02 \x01(\tR\tactorName\x12\x1b\n" +
+ "\tactor_uid\x18\x03 \x01(\tR\bactorUid\x12>\n" +
+ "\x1bcertificate_signing_request\x18\x04 \x01(\fR\x19certificateSigningRequest\"M\n" +
"\x1cMintActorCertificateResponse\x12-\n" +
"\x12actor_certificates\x18\x01 \x03(\fR\x11actorCertificates\"\xa2\x02\n" +
"\x10TerminateRequest\x12$\n" +
@@ -4205,7 +4095,7 @@ const file_atelet_proto_rawDesc = "" +
"\x17actor_template_atespace\x18\x05 \x01(\tR\x15actorTemplateAtespace\x12.\n" +
"\x13actor_template_name\x18\x06 \x01(\tR\x11actorTemplateName\x12(\n" +
"\x04spec\x18\a \x01(\v2\x14.atelet.WorkloadSpecR\x04spec\"\x13\n" +
- "\x11TerminateResponse\"\xb1\x04\n" +
+ "\x11TerminateResponse\"\xf3\x03\n" +
"\n" +
"RunRequest\x12$\n" +
"\x0eworker_pod_uid\x18\x01 \x01(\tR\fworkerPodUid\x12\x1a\n" +
@@ -4215,8 +4105,7 @@ const file_atelet_proto_rawDesc = "" +
"\tactor_uid\x18\x04 \x01(\tR\bactorUid\x126\n" +
"\x17actor_template_atespace\x18\x05 \x01(\tR\x15actorTemplateAtespace\x12.\n" +
"\x13actor_template_name\x18\x06 \x01(\tR\x11actorTemplateName\x12(\n" +
- "\x04spec\x18\a \x01(\v2\x14.atelet.WorkloadSpecR\x04spec\x12<\n" +
- "\x0esandbox_assets\x18\b \x01(\v2\x15.atelet.SandboxAssetsR\rsandboxAssets\x12A\n" +
+ "\x04spec\x18\a \x01(\v2\x14.atelet.WorkloadSpecR\x04spec\x12A\n" +
"\x0eegress_gateway\x18\t \x01(\v2\x15.atelet.EgressGatewayH\x00R\regressGateway\x88\x01\x01\x12\x1b\n" +
"\tcpu_milli\x18\n" +
" \x01(\x03R\bcpuMilli\x12!\n" +
@@ -4224,25 +4113,7 @@ const file_atelet_proto_rawDesc = "" +
"\x0fsandbox_runtime\x18\f \x01(\v2\x16.atelet.SandboxRuntimeR\x0esandboxRuntimeB\x11\n" +
"\x0f_egress_gateway\")\n" +
"\rEgressGateway\x12\x18\n" +
- "\aaddress\x18\x01 \x01(\tR\aaddress\":\n" +
- "\tAssetFile\x12\x15\n" +
- "\x03url\x18\x01 \x01(\tB\x03\x80\x01\x01R\x03url\x12\x16\n" +
- "\x06sha256\x18\x02 \x01(\tR\x06sha256\"\x8e\x01\n" +
- "\n" +
- "ArchAssets\x123\n" +
- "\x05files\x18\x01 \x03(\v2\x1d.atelet.ArchAssets.FilesEntryR\x05files\x1aK\n" +
- "\n" +
- "FilesEntry\x12\x10\n" +
- "\x03key\x18\x01 \x01(\tR\x03key\x12'\n" +
- "\x05value\x18\x02 \x01(\v2\x11.atelet.AssetFileR\x05value:\x028\x01\"\xdf\x01\n" +
- "\rSandboxAssets\x12#\n" +
- "\rsandbox_class\x18\x01 \x01(\tR\fsandboxClass\x129\n" +
- "\x06assets\x18\x02 \x03(\v2!.atelet.SandboxAssets.AssetsEntryR\x06assets\x12\x1f\n" +
- "\vpause_image\x18\x03 \x01(\tR\n" +
- "pauseImage\x1aM\n" +
- "\vAssetsEntry\x12\x10\n" +
- "\x03key\x18\x01 \x01(\tR\x03key\x12(\n" +
- "\x05value\x18\x02 \x01(\v2\x12.atelet.ArchAssetsR\x05value:\x028\x01\"k\n" +
+ "\aaddress\x18\x01 \x01(\tR\aaddress\"k\n" +
"\fWorkloadSpec\x121\n" +
"\n" +
"containers\x18\x01 \x03(\v2\x11.atelet.ContainerR\n" +
@@ -4267,10 +4138,10 @@ const file_atelet_proto_rawDesc = "" +
"\x05field\x18\x01 \x01(\x0e2\x1a.atelet.ActorMetadataFieldR\x05field\x12\x12\n" +
"\x04path\x18\x02 \x01(\tR\x04path\"J\n" +
"\x17ActorMetadataDataSource\x12/\n" +
- "\x05items\x18\x01 \x03(\v2\x19.atelet.ActorMetadataItemR\x05items\"M\n" +
+ "\x05items\x18\x01 \x03(\v2\x19.atelet.ActorMetadataItemR\x05items\"A\n" +
"\x15TrustBundleDataSource\x12\x12\n" +
"\x04path\x18\x01 \x01(\tR\x04path\x12\x14\n" +
- "\x05names\x18\x03 \x03(\tR\x05namesJ\x04\b\x02\x10\x03R\x04name\"\xa0\x01\n" +
+ "\x05names\x18\x02 \x03(\tR\x05names\"\xa0\x01\n" +
"\x14SystemInfoDataSource\x12F\n" +
"\x0eactor_metadata\x18\x01 \x01(\v2\x1f.atelet.ActorMetadataDataSourceR\ractorMetadata\x12@\n" +
"\ftrust_bundle\x18\x02 \x01(\v2\x1d.atelet.TrustBundleDataSourceR\vtrustBundle\"Y\n" +
@@ -4349,7 +4220,7 @@ const file_atelet_proto_rawDesc = "" +
"\bsnapshot\x18\b \x01(\v2\x10.atelet.SnapshotR\bsnapshot\x129\n" +
"\rupload_config\x18\t \x01(\v2\x14.atelet.UploadConfigR\fuploadConfig\"F\n" +
"\x12CheckpointResponse\x120\n" +
- "\x05state\x18\x01 \x01(\v2\x1a.atelet.ActorSnapshotStateR\x05state\"\xe3\x04\n" +
+ "\x05state\x18\x01 \x01(\v2\x1a.atelet.ActorSnapshotStateR\x05state\"\xa5\x04\n" +
"\x0eRestoreRequest\x12$\n" +
"\x0eworker_pod_uid\x18\x01 \x01(\tR\fworkerPodUid\x12\x1a\n" +
"\batespace\x18\x02 \x01(\tR\batespace\x12\x1d\n" +
@@ -4360,11 +4231,11 @@ const file_atelet_proto_rawDesc = "" +
"\x13actor_template_name\x18\x06 \x01(\tR\x11actorTemplateName\x12(\n" +
"\x04spec\x18\a \x01(\v2\x14.atelet.WorkloadSpecR\x04spec\x12,\n" +
"\bsnapshot\x18\b \x01(\v2\x10.atelet.SnapshotR\bsnapshot\x12A\n" +
- "\x0eegress_gateway\x18\r \x01(\v2\x15.atelet.EgressGatewayH\x00R\regressGateway\x88\x01\x01\x12\x1b\n" +
- "\tcpu_milli\x18\x0e \x01(\x03R\bcpuMilli\x12!\n" +
- "\fmemory_bytes\x18\x0f \x01(\x03R\vmemoryBytes\x12<\n" +
- "\x0esandbox_assets\x18\x10 \x01(\v2\x15.atelet.SandboxAssetsR\rsandboxAssets\x12?\n" +
- "\x0fsandbox_runtime\x18\x11 \x01(\v2\x16.atelet.SandboxRuntimeR\x0esandboxRuntimeB\x11\n" +
+ "\x0eegress_gateway\x18\t \x01(\v2\x15.atelet.EgressGatewayH\x00R\regressGateway\x88\x01\x01\x12\x1b\n" +
+ "\tcpu_milli\x18\n" +
+ " \x01(\x03R\bcpuMilli\x12!\n" +
+ "\fmemory_bytes\x18\v \x01(\x03R\vmemoryBytes\x12?\n" +
+ "\x0fsandbox_runtime\x18\f \x01(\v2\x16.atelet.SandboxRuntimeR\x0esandboxRuntimeB\x11\n" +
"\x0f_egress_gateway\"\x11\n" +
"\x0fRestoreResponse\"\x86\x02\n" +
"\x16PromoteSnapshotRequest\x12\x1a\n" +
@@ -4398,12 +4269,13 @@ const file_atelet_proto_rawDesc = "" +
"UploadMode\x12\x1b\n" +
"\x17UPLOAD_MODE_UNSPECIFIED\x10\x00\x12\x18\n" +
"\x14UPLOAD_MODE_DEFERRED\x10\x01\x12\x19\n" +
- "\x15UPLOAD_MODE_IMMEDIATE\x10\x022\x8e\x03\n" +
+ "\x15UPLOAD_MODE_IMMEDIATE\x10\x022\xed\x03\n" +
"\rWorkerSupport\x12c\n" +
"\x14MintActorCertificate\x12#.atelet.MintActorCertificateRequest\x1a$.atelet.MintActorCertificateResponse\"\x00\x12Q\n" +
"\x0eRegisterWorker\x12\x1d.atelet.RegisterWorkerRequest\x1a\x1e.atelet.RegisterWorkerResponse\"\x00\x12`\n" +
"\x13RequestActorSuspend\x12\".atelet.RequestActorSuspendRequest\x1a#.atelet.RequestActorSuspendResponse\"\x00\x12c\n" +
- "\x14GetActorEgressPolicy\x12#.atelet.GetActorEgressPolicyRequest\x1a$.atelet.GetActorEgressPolicyResponse\"\x002\xa3\x03\n" +
+ "\x14GetActorEgressPolicy\x12#.atelet.GetActorEgressPolicyRequest\x1a$.atelet.GetActorEgressPolicyResponse\"\x00\x12]\n" +
+ "\x12ReportActorCrashed\x12!.atelet.ReportActorCrashedRequest\x1a\".atelet.ReportActorCrashedResponse\"\x002\xa3\x03\n" +
"\x06Atelet\x120\n" +
"\x03Run\x12\x12.atelet.RunRequest\x1a\x13.atelet.RunResponse\"\x00\x12E\n" +
"\n" +
@@ -4426,7 +4298,7 @@ func file_atelet_proto_rawDescGZIP() []byte {
}
var file_atelet_proto_enumTypes = make([]protoimpl.EnumInfo, 3)
-var file_atelet_proto_msgTypes = make([]protoimpl.MessageInfo, 62)
+var file_atelet_proto_msgTypes = make([]protoimpl.MessageInfo, 59)
var file_atelet_proto_goTypes = []any{
(ActorMetadataField)(0), // 0: atelet.ActorMetadataField
(SnapshotFidelity)(0), // 1: atelet.SnapshotFidelity
@@ -4441,59 +4313,56 @@ var file_atelet_proto_goTypes = []any{
(*RegisterWorkerResponse)(nil), // 10: atelet.RegisterWorkerResponse
(*RequestActorSuspendRequest)(nil), // 11: atelet.RequestActorSuspendRequest
(*RequestActorSuspendResponse)(nil), // 12: atelet.RequestActorSuspendResponse
- (*GetActorEgressPolicyRequest)(nil), // 13: atelet.GetActorEgressPolicyRequest
- (*GetActorEgressPolicyResponse)(nil), // 14: atelet.GetActorEgressPolicyResponse
- (*EgressPolicy)(nil), // 15: atelet.EgressPolicy
- (*EgressRule)(nil), // 16: atelet.EgressRule
- (*EgressDestinations)(nil), // 17: atelet.EgressDestinations
- (*EgressPorts)(nil), // 18: atelet.EgressPorts
- (*EgressAllPorts)(nil), // 19: atelet.EgressAllPorts
- (*MintActorCertificateRequest)(nil), // 20: atelet.MintActorCertificateRequest
- (*MintActorCertificateResponse)(nil), // 21: atelet.MintActorCertificateResponse
- (*TerminateRequest)(nil), // 22: atelet.TerminateRequest
- (*TerminateResponse)(nil), // 23: atelet.TerminateResponse
- (*RunRequest)(nil), // 24: atelet.RunRequest
- (*EgressGateway)(nil), // 25: atelet.EgressGateway
- (*AssetFile)(nil), // 26: atelet.AssetFile
- (*ArchAssets)(nil), // 27: atelet.ArchAssets
- (*SandboxAssets)(nil), // 28: atelet.SandboxAssets
- (*WorkloadSpec)(nil), // 29: atelet.WorkloadSpec
- (*DurableDirVolumeSource)(nil), // 30: atelet.DurableDirVolumeSource
- (*ExternalVolumeSource)(nil), // 31: atelet.ExternalVolumeSource
- (*ImageVolumeSource)(nil), // 32: atelet.ImageVolumeSource
- (*ActorMetadataItem)(nil), // 33: atelet.ActorMetadataItem
- (*ActorMetadataDataSource)(nil), // 34: atelet.ActorMetadataDataSource
- (*TrustBundleDataSource)(nil), // 35: atelet.TrustBundleDataSource
- (*SystemInfoDataSource)(nil), // 36: atelet.SystemInfoDataSource
- (*SystemInfoVolumeSource)(nil), // 37: atelet.SystemInfoVolumeSource
- (*Volume)(nil), // 38: atelet.Volume
- (*VolumeMount)(nil), // 39: atelet.VolumeMount
- (*Container)(nil), // 40: atelet.Container
- (*SecurityContext)(nil), // 41: atelet.SecurityContext
- (*Capabilities)(nil), // 42: atelet.Capabilities
- (*ResourceLimits)(nil), // 43: atelet.ResourceLimits
- (*EnvVar)(nil), // 44: atelet.EnvVar
- (*WakeupProbe)(nil), // 45: atelet.WakeupProbe
- (*HttpGetAction)(nil), // 46: atelet.HttpGetAction
- (*RunResponse)(nil), // 47: atelet.RunResponse
- (*ObjectStorage)(nil), // 48: atelet.ObjectStorage
- (*Snapshot)(nil), // 49: atelet.Snapshot
- (*SnapshotReport)(nil), // 50: atelet.SnapshotReport
- (*ActorSnapshotState)(nil), // 51: atelet.ActorSnapshotState
- (*UploadConfig)(nil), // 52: atelet.UploadConfig
- (*CheckpointRequest)(nil), // 53: atelet.CheckpointRequest
- (*CheckpointResponse)(nil), // 54: atelet.CheckpointResponse
- (*RestoreRequest)(nil), // 55: atelet.RestoreRequest
- (*RestoreResponse)(nil), // 56: atelet.RestoreResponse
- (*PromoteSnapshotRequest)(nil), // 57: atelet.PromoteSnapshotRequest
- (*PromoteSnapshotResponse)(nil), // 58: atelet.PromoteSnapshotResponse
- (*DetachActorRequest)(nil), // 59: atelet.DetachActorRequest
- (*DetachActorResponse)(nil), // 60: atelet.DetachActorResponse
- nil, // 61: atelet.ArchAssets.FilesEntry
- nil, // 62: atelet.SandboxAssets.AssetsEntry
- nil, // 63: atelet.ExternalVolumeSource.VolumeContextEntry
- nil, // 64: atelet.ExternalVolumeSource.PublishContextEntry
- (*timestamppb.Timestamp)(nil), // 65: google.protobuf.Timestamp
+ (*ReportActorCrashedRequest)(nil), // 13: atelet.ReportActorCrashedRequest
+ (*ReportActorCrashedResponse)(nil), // 14: atelet.ReportActorCrashedResponse
+ (*GetActorEgressPolicyRequest)(nil), // 15: atelet.GetActorEgressPolicyRequest
+ (*GetActorEgressPolicyResponse)(nil), // 16: atelet.GetActorEgressPolicyResponse
+ (*EgressPolicy)(nil), // 17: atelet.EgressPolicy
+ (*EgressRule)(nil), // 18: atelet.EgressRule
+ (*EgressDestinations)(nil), // 19: atelet.EgressDestinations
+ (*EgressPorts)(nil), // 20: atelet.EgressPorts
+ (*EgressAllPorts)(nil), // 21: atelet.EgressAllPorts
+ (*MintActorCertificateRequest)(nil), // 22: atelet.MintActorCertificateRequest
+ (*MintActorCertificateResponse)(nil), // 23: atelet.MintActorCertificateResponse
+ (*TerminateRequest)(nil), // 24: atelet.TerminateRequest
+ (*TerminateResponse)(nil), // 25: atelet.TerminateResponse
+ (*RunRequest)(nil), // 26: atelet.RunRequest
+ (*EgressGateway)(nil), // 27: atelet.EgressGateway
+ (*WorkloadSpec)(nil), // 28: atelet.WorkloadSpec
+ (*DurableDirVolumeSource)(nil), // 29: atelet.DurableDirVolumeSource
+ (*ExternalVolumeSource)(nil), // 30: atelet.ExternalVolumeSource
+ (*ImageVolumeSource)(nil), // 31: atelet.ImageVolumeSource
+ (*ActorMetadataItem)(nil), // 32: atelet.ActorMetadataItem
+ (*ActorMetadataDataSource)(nil), // 33: atelet.ActorMetadataDataSource
+ (*TrustBundleDataSource)(nil), // 34: atelet.TrustBundleDataSource
+ (*SystemInfoDataSource)(nil), // 35: atelet.SystemInfoDataSource
+ (*SystemInfoVolumeSource)(nil), // 36: atelet.SystemInfoVolumeSource
+ (*Volume)(nil), // 37: atelet.Volume
+ (*VolumeMount)(nil), // 38: atelet.VolumeMount
+ (*Container)(nil), // 39: atelet.Container
+ (*SecurityContext)(nil), // 40: atelet.SecurityContext
+ (*Capabilities)(nil), // 41: atelet.Capabilities
+ (*ResourceLimits)(nil), // 42: atelet.ResourceLimits
+ (*EnvVar)(nil), // 43: atelet.EnvVar
+ (*WakeupProbe)(nil), // 44: atelet.WakeupProbe
+ (*HttpGetAction)(nil), // 45: atelet.HttpGetAction
+ (*RunResponse)(nil), // 46: atelet.RunResponse
+ (*ObjectStorage)(nil), // 47: atelet.ObjectStorage
+ (*Snapshot)(nil), // 48: atelet.Snapshot
+ (*SnapshotReport)(nil), // 49: atelet.SnapshotReport
+ (*ActorSnapshotState)(nil), // 50: atelet.ActorSnapshotState
+ (*UploadConfig)(nil), // 51: atelet.UploadConfig
+ (*CheckpointRequest)(nil), // 52: atelet.CheckpointRequest
+ (*CheckpointResponse)(nil), // 53: atelet.CheckpointResponse
+ (*RestoreRequest)(nil), // 54: atelet.RestoreRequest
+ (*RestoreResponse)(nil), // 55: atelet.RestoreResponse
+ (*PromoteSnapshotRequest)(nil), // 56: atelet.PromoteSnapshotRequest
+ (*PromoteSnapshotResponse)(nil), // 57: atelet.PromoteSnapshotResponse
+ (*DetachActorRequest)(nil), // 58: atelet.DetachActorRequest
+ (*DetachActorResponse)(nil), // 59: atelet.DetachActorResponse
+ nil, // 60: atelet.ExternalVolumeSource.VolumeContextEntry
+ nil, // 61: atelet.ExternalVolumeSource.PublishContextEntry
+ (*timestamppb.Timestamp)(nil), // 62: google.protobuf.Timestamp
}
var file_atelet_proto_depIdxs = []int32{
7, // 0: atelet.RegisterWorkerRequest.capacity:type_name -> atelet.WorkerResources
@@ -4503,89 +4372,85 @@ var file_atelet_proto_depIdxs = []int32{
6, // 4: atelet.VersionedSandboxCompat.attributes:type_name -> atelet.AttributeEntry
8, // 5: atelet.WorkerResources.resources:type_name -> atelet.Resources
9, // 6: atelet.Resources.limits:type_name -> atelet.Limits
- 15, // 7: atelet.GetActorEgressPolicyResponse.egress_policy:type_name -> atelet.EgressPolicy
- 16, // 8: atelet.EgressPolicy.rules:type_name -> atelet.EgressRule
- 17, // 9: atelet.EgressRule.http:type_name -> atelet.EgressDestinations
- 17, // 10: atelet.EgressRule.https:type_name -> atelet.EgressDestinations
- 17, // 11: atelet.EgressRule.tls_passthrough:type_name -> atelet.EgressDestinations
- 18, // 12: atelet.EgressDestinations.ports:type_name -> atelet.EgressPorts
- 19, // 13: atelet.EgressPorts.all:type_name -> atelet.EgressAllPorts
- 29, // 14: atelet.TerminateRequest.spec:type_name -> atelet.WorkloadSpec
- 29, // 15: atelet.RunRequest.spec:type_name -> atelet.WorkloadSpec
- 28, // 16: atelet.RunRequest.sandbox_assets:type_name -> atelet.SandboxAssets
- 25, // 17: atelet.RunRequest.egress_gateway:type_name -> atelet.EgressGateway
- 4, // 18: atelet.RunRequest.sandbox_runtime:type_name -> atelet.SandboxRuntime
- 61, // 19: atelet.ArchAssets.files:type_name -> atelet.ArchAssets.FilesEntry
- 62, // 20: atelet.SandboxAssets.assets:type_name -> atelet.SandboxAssets.AssetsEntry
- 40, // 21: atelet.WorkloadSpec.containers:type_name -> atelet.Container
- 38, // 22: atelet.WorkloadSpec.volumes:type_name -> atelet.Volume
- 63, // 23: atelet.ExternalVolumeSource.volume_context:type_name -> atelet.ExternalVolumeSource.VolumeContextEntry
- 64, // 24: atelet.ExternalVolumeSource.publish_context:type_name -> atelet.ExternalVolumeSource.PublishContextEntry
- 0, // 25: atelet.ActorMetadataItem.field:type_name -> atelet.ActorMetadataField
- 33, // 26: atelet.ActorMetadataDataSource.items:type_name -> atelet.ActorMetadataItem
- 34, // 27: atelet.SystemInfoDataSource.actor_metadata:type_name -> atelet.ActorMetadataDataSource
- 35, // 28: atelet.SystemInfoDataSource.trust_bundle:type_name -> atelet.TrustBundleDataSource
- 36, // 29: atelet.SystemInfoVolumeSource.data_sources:type_name -> atelet.SystemInfoDataSource
- 30, // 30: atelet.Volume.durable_dir:type_name -> atelet.DurableDirVolumeSource
- 31, // 31: atelet.Volume.external:type_name -> atelet.ExternalVolumeSource
- 37, // 32: atelet.Volume.system_info:type_name -> atelet.SystemInfoVolumeSource
- 32, // 33: atelet.Volume.image:type_name -> atelet.ImageVolumeSource
- 44, // 34: atelet.Container.env:type_name -> atelet.EnvVar
- 45, // 35: atelet.Container.wakeup_probe:type_name -> atelet.WakeupProbe
- 39, // 36: atelet.Container.volume_mounts:type_name -> atelet.VolumeMount
- 41, // 37: atelet.Container.security_context:type_name -> atelet.SecurityContext
- 43, // 38: atelet.Container.resources:type_name -> atelet.ResourceLimits
- 42, // 39: atelet.SecurityContext.capabilities:type_name -> atelet.Capabilities
- 46, // 40: atelet.WakeupProbe.http_get:type_name -> atelet.HttpGetAction
- 1, // 41: atelet.Snapshot.fidelity:type_name -> atelet.SnapshotFidelity
- 48, // 42: atelet.Snapshot.object:type_name -> atelet.ObjectStorage
- 49, // 43: atelet.SnapshotReport.snapshot:type_name -> atelet.Snapshot
- 65, // 44: atelet.SnapshotReport.local_commit_time:type_name -> google.protobuf.Timestamp
- 65, // 45: atelet.SnapshotReport.durable_complete_time:type_name -> google.protobuf.Timestamp
- 50, // 46: atelet.ActorSnapshotState.latest_local:type_name -> atelet.SnapshotReport
- 50, // 47: atelet.ActorSnapshotState.uploading:type_name -> atelet.SnapshotReport
- 50, // 48: atelet.ActorSnapshotState.latest_durable:type_name -> atelet.SnapshotReport
- 65, // 49: atelet.ActorSnapshotState.upload_due_time:type_name -> google.protobuf.Timestamp
- 2, // 50: atelet.UploadConfig.mode:type_name -> atelet.UploadMode
- 29, // 51: atelet.CheckpointRequest.spec:type_name -> atelet.WorkloadSpec
- 49, // 52: atelet.CheckpointRequest.snapshot:type_name -> atelet.Snapshot
- 52, // 53: atelet.CheckpointRequest.upload_config:type_name -> atelet.UploadConfig
- 51, // 54: atelet.CheckpointResponse.state:type_name -> atelet.ActorSnapshotState
- 29, // 55: atelet.RestoreRequest.spec:type_name -> atelet.WorkloadSpec
- 49, // 56: atelet.RestoreRequest.snapshot:type_name -> atelet.Snapshot
- 25, // 57: atelet.RestoreRequest.egress_gateway:type_name -> atelet.EgressGateway
- 28, // 58: atelet.RestoreRequest.sandbox_assets:type_name -> atelet.SandboxAssets
- 4, // 59: atelet.RestoreRequest.sandbox_runtime:type_name -> atelet.SandboxRuntime
- 49, // 60: atelet.PromoteSnapshotRequest.snapshot:type_name -> atelet.Snapshot
- 51, // 61: atelet.PromoteSnapshotResponse.state:type_name -> atelet.ActorSnapshotState
- 51, // 62: atelet.DetachActorResponse.state:type_name -> atelet.ActorSnapshotState
- 26, // 63: atelet.ArchAssets.FilesEntry.value:type_name -> atelet.AssetFile
- 27, // 64: atelet.SandboxAssets.AssetsEntry.value:type_name -> atelet.ArchAssets
- 20, // 65: atelet.WorkerSupport.MintActorCertificate:input_type -> atelet.MintActorCertificateRequest
- 3, // 66: atelet.WorkerSupport.RegisterWorker:input_type -> atelet.RegisterWorkerRequest
- 11, // 67: atelet.WorkerSupport.RequestActorSuspend:input_type -> atelet.RequestActorSuspendRequest
- 13, // 68: atelet.WorkerSupport.GetActorEgressPolicy:input_type -> atelet.GetActorEgressPolicyRequest
- 24, // 69: atelet.Atelet.Run:input_type -> atelet.RunRequest
- 53, // 70: atelet.Atelet.Checkpoint:input_type -> atelet.CheckpointRequest
- 55, // 71: atelet.Atelet.Restore:input_type -> atelet.RestoreRequest
- 57, // 72: atelet.Atelet.PromoteSnapshot:input_type -> atelet.PromoteSnapshotRequest
- 59, // 73: atelet.Atelet.DetachActor:input_type -> atelet.DetachActorRequest
- 22, // 74: atelet.Atelet.Terminate:input_type -> atelet.TerminateRequest
- 21, // 75: atelet.WorkerSupport.MintActorCertificate:output_type -> atelet.MintActorCertificateResponse
- 10, // 76: atelet.WorkerSupport.RegisterWorker:output_type -> atelet.RegisterWorkerResponse
- 12, // 77: atelet.WorkerSupport.RequestActorSuspend:output_type -> atelet.RequestActorSuspendResponse
- 14, // 78: atelet.WorkerSupport.GetActorEgressPolicy:output_type -> atelet.GetActorEgressPolicyResponse
- 47, // 79: atelet.Atelet.Run:output_type -> atelet.RunResponse
- 54, // 80: atelet.Atelet.Checkpoint:output_type -> atelet.CheckpointResponse
- 56, // 81: atelet.Atelet.Restore:output_type -> atelet.RestoreResponse
- 58, // 82: atelet.Atelet.PromoteSnapshot:output_type -> atelet.PromoteSnapshotResponse
- 60, // 83: atelet.Atelet.DetachActor:output_type -> atelet.DetachActorResponse
- 23, // 84: atelet.Atelet.Terminate:output_type -> atelet.TerminateResponse
- 75, // [75:85] is the sub-list for method output_type
- 65, // [65:75] is the sub-list for method input_type
- 65, // [65:65] is the sub-list for extension type_name
- 65, // [65:65] is the sub-list for extension extendee
- 0, // [0:65] is the sub-list for field type_name
+ 17, // 7: atelet.GetActorEgressPolicyResponse.egress_policy:type_name -> atelet.EgressPolicy
+ 18, // 8: atelet.EgressPolicy.rules:type_name -> atelet.EgressRule
+ 19, // 9: atelet.EgressRule.http:type_name -> atelet.EgressDestinations
+ 19, // 10: atelet.EgressRule.https:type_name -> atelet.EgressDestinations
+ 19, // 11: atelet.EgressRule.tls_passthrough:type_name -> atelet.EgressDestinations
+ 20, // 12: atelet.EgressDestinations.ports:type_name -> atelet.EgressPorts
+ 21, // 13: atelet.EgressPorts.all:type_name -> atelet.EgressAllPorts
+ 28, // 14: atelet.TerminateRequest.spec:type_name -> atelet.WorkloadSpec
+ 28, // 15: atelet.RunRequest.spec:type_name -> atelet.WorkloadSpec
+ 27, // 16: atelet.RunRequest.egress_gateway:type_name -> atelet.EgressGateway
+ 4, // 17: atelet.RunRequest.sandbox_runtime:type_name -> atelet.SandboxRuntime
+ 39, // 18: atelet.WorkloadSpec.containers:type_name -> atelet.Container
+ 37, // 19: atelet.WorkloadSpec.volumes:type_name -> atelet.Volume
+ 60, // 20: atelet.ExternalVolumeSource.volume_context:type_name -> atelet.ExternalVolumeSource.VolumeContextEntry
+ 61, // 21: atelet.ExternalVolumeSource.publish_context:type_name -> atelet.ExternalVolumeSource.PublishContextEntry
+ 0, // 22: atelet.ActorMetadataItem.field:type_name -> atelet.ActorMetadataField
+ 32, // 23: atelet.ActorMetadataDataSource.items:type_name -> atelet.ActorMetadataItem
+ 33, // 24: atelet.SystemInfoDataSource.actor_metadata:type_name -> atelet.ActorMetadataDataSource
+ 34, // 25: atelet.SystemInfoDataSource.trust_bundle:type_name -> atelet.TrustBundleDataSource
+ 35, // 26: atelet.SystemInfoVolumeSource.data_sources:type_name -> atelet.SystemInfoDataSource
+ 29, // 27: atelet.Volume.durable_dir:type_name -> atelet.DurableDirVolumeSource
+ 30, // 28: atelet.Volume.external:type_name -> atelet.ExternalVolumeSource
+ 36, // 29: atelet.Volume.system_info:type_name -> atelet.SystemInfoVolumeSource
+ 31, // 30: atelet.Volume.image:type_name -> atelet.ImageVolumeSource
+ 43, // 31: atelet.Container.env:type_name -> atelet.EnvVar
+ 44, // 32: atelet.Container.wakeup_probe:type_name -> atelet.WakeupProbe
+ 38, // 33: atelet.Container.volume_mounts:type_name -> atelet.VolumeMount
+ 40, // 34: atelet.Container.security_context:type_name -> atelet.SecurityContext
+ 42, // 35: atelet.Container.resources:type_name -> atelet.ResourceLimits
+ 41, // 36: atelet.SecurityContext.capabilities:type_name -> atelet.Capabilities
+ 45, // 37: atelet.WakeupProbe.http_get:type_name -> atelet.HttpGetAction
+ 1, // 38: atelet.Snapshot.fidelity:type_name -> atelet.SnapshotFidelity
+ 47, // 39: atelet.Snapshot.object:type_name -> atelet.ObjectStorage
+ 48, // 40: atelet.SnapshotReport.snapshot:type_name -> atelet.Snapshot
+ 62, // 41: atelet.SnapshotReport.local_commit_time:type_name -> google.protobuf.Timestamp
+ 62, // 42: atelet.SnapshotReport.durable_complete_time:type_name -> google.protobuf.Timestamp
+ 49, // 43: atelet.ActorSnapshotState.latest_local:type_name -> atelet.SnapshotReport
+ 49, // 44: atelet.ActorSnapshotState.uploading:type_name -> atelet.SnapshotReport
+ 49, // 45: atelet.ActorSnapshotState.latest_durable:type_name -> atelet.SnapshotReport
+ 62, // 46: atelet.ActorSnapshotState.upload_due_time:type_name -> google.protobuf.Timestamp
+ 2, // 47: atelet.UploadConfig.mode:type_name -> atelet.UploadMode
+ 28, // 48: atelet.CheckpointRequest.spec:type_name -> atelet.WorkloadSpec
+ 48, // 49: atelet.CheckpointRequest.snapshot:type_name -> atelet.Snapshot
+ 51, // 50: atelet.CheckpointRequest.upload_config:type_name -> atelet.UploadConfig
+ 50, // 51: atelet.CheckpointResponse.state:type_name -> atelet.ActorSnapshotState
+ 28, // 52: atelet.RestoreRequest.spec:type_name -> atelet.WorkloadSpec
+ 48, // 53: atelet.RestoreRequest.snapshot:type_name -> atelet.Snapshot
+ 27, // 54: atelet.RestoreRequest.egress_gateway:type_name -> atelet.EgressGateway
+ 4, // 55: atelet.RestoreRequest.sandbox_runtime:type_name -> atelet.SandboxRuntime
+ 48, // 56: atelet.PromoteSnapshotRequest.snapshot:type_name -> atelet.Snapshot
+ 50, // 57: atelet.PromoteSnapshotResponse.state:type_name -> atelet.ActorSnapshotState
+ 50, // 58: atelet.DetachActorResponse.state:type_name -> atelet.ActorSnapshotState
+ 22, // 59: atelet.WorkerSupport.MintActorCertificate:input_type -> atelet.MintActorCertificateRequest
+ 3, // 60: atelet.WorkerSupport.RegisterWorker:input_type -> atelet.RegisterWorkerRequest
+ 11, // 61: atelet.WorkerSupport.RequestActorSuspend:input_type -> atelet.RequestActorSuspendRequest
+ 15, // 62: atelet.WorkerSupport.GetActorEgressPolicy:input_type -> atelet.GetActorEgressPolicyRequest
+ 13, // 63: atelet.WorkerSupport.ReportActorCrashed:input_type -> atelet.ReportActorCrashedRequest
+ 26, // 64: atelet.Atelet.Run:input_type -> atelet.RunRequest
+ 52, // 65: atelet.Atelet.Checkpoint:input_type -> atelet.CheckpointRequest
+ 54, // 66: atelet.Atelet.Restore:input_type -> atelet.RestoreRequest
+ 56, // 67: atelet.Atelet.PromoteSnapshot:input_type -> atelet.PromoteSnapshotRequest
+ 58, // 68: atelet.Atelet.DetachActor:input_type -> atelet.DetachActorRequest
+ 24, // 69: atelet.Atelet.Terminate:input_type -> atelet.TerminateRequest
+ 23, // 70: atelet.WorkerSupport.MintActorCertificate:output_type -> atelet.MintActorCertificateResponse
+ 10, // 71: atelet.WorkerSupport.RegisterWorker:output_type -> atelet.RegisterWorkerResponse
+ 12, // 72: atelet.WorkerSupport.RequestActorSuspend:output_type -> atelet.RequestActorSuspendResponse
+ 16, // 73: atelet.WorkerSupport.GetActorEgressPolicy:output_type -> atelet.GetActorEgressPolicyResponse
+ 14, // 74: atelet.WorkerSupport.ReportActorCrashed:output_type -> atelet.ReportActorCrashedResponse
+ 46, // 75: atelet.Atelet.Run:output_type -> atelet.RunResponse
+ 53, // 76: atelet.Atelet.Checkpoint:output_type -> atelet.CheckpointResponse
+ 55, // 77: atelet.Atelet.Restore:output_type -> atelet.RestoreResponse
+ 57, // 78: atelet.Atelet.PromoteSnapshot:output_type -> atelet.PromoteSnapshotResponse
+ 59, // 79: atelet.Atelet.DetachActor:output_type -> atelet.DetachActorResponse
+ 25, // 80: atelet.Atelet.Terminate:output_type -> atelet.TerminateResponse
+ 70, // [70:81] is the sub-list for method output_type
+ 59, // [59:70] is the sub-list for method input_type
+ 59, // [59:59] is the sub-list for extension type_name
+ 59, // [59:59] is the sub-list for extension extendee
+ 0, // [0:59] is the sub-list for field type_name
}
func init() { file_atelet_proto_init() }
@@ -4593,15 +4458,15 @@ func file_atelet_proto_init() {
if File_atelet_proto != nil {
return
}
- file_atelet_proto_msgTypes[21].OneofWrappers = []any{}
- file_atelet_proto_msgTypes[52].OneofWrappers = []any{}
+ file_atelet_proto_msgTypes[23].OneofWrappers = []any{}
+ file_atelet_proto_msgTypes[51].OneofWrappers = []any{}
type x struct{}
out := protoimpl.TypeBuilder{
File: protoimpl.DescBuilder{
GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
RawDescriptor: unsafe.Slice(unsafe.StringData(file_atelet_proto_rawDesc), len(file_atelet_proto_rawDesc)),
NumEnums: 3,
- NumMessages: 62,
+ NumMessages: 59,
NumExtensions: 0,
NumServices: 2,
},
diff --git a/internal/proto/ateletpb/atelet.proto b/internal/proto/ateletpb/atelet.proto
index c00ee06159..74417fa926 100644
--- a/internal/proto/ateletpb/atelet.proto
+++ b/internal/proto/ateletpb/atelet.proto
@@ -51,6 +51,11 @@ service WorkerSupport {
// Control.GetActorEgressPolicy. An error is the control plane's, passed
// through as it was given.
rpc GetActorEgressPolicy(GetActorEgressPolicyRequest) returns (GetActorEgressPolicyResponse) {}
+
+ // ReportActorCrashed reports that the sandbox of the actor this worker
+ // hosts died on its own, for atelet to forward to the control plane's
+ // WorkerService, which this mirrors.
+ rpc ReportActorCrashed(ReportActorCrashedRequest) returns (ReportActorCrashedResponse) {}
}
message RegisterWorkerRequest {
@@ -213,6 +218,31 @@ message RequestActorSuspendRequest {
message RequestActorSuspendResponse {
}
+message ReportActorCrashedRequest {
+ // The actor the calling worker is hosting, as for RequestActorSuspend.
+ //
+ // +k8s:required
+ // +k8s:format=k8s-short-name
+ string actor_atespace = 1;
+
+ // +k8s:required
+ // +k8s:format=k8s-short-name
+ string actor_name = 2;
+
+ // +k8s:required
+ // +k8s:format=k8s-uuid
+ string actor_uid = 3;
+
+ // What the worker saw, e.g. how the sandbox exited.
+ //
+ // +k8s:required
+ // +k8s:maxLength=4096
+ string message = 4;
+}
+
+message ReportActorCrashedResponse {
+}
+
message GetActorEgressPolicyRequest {
// The actor the calling worker is hosting. atelet reads its policy from the
// control plane by name.
@@ -283,25 +313,25 @@ message MintActorCertificateRequest {
//
// +k8s:required
// +k8s:format=k8s-short-name
- string actor_atespace = 3;
+ string actor_atespace = 1;
// +k8s:required
// +k8s:format=k8s-short-name
- string actor_name = 4;
+ string actor_name = 2;
// The UID of the actor --- used to guard against deletion and recreation of
// an actor with the same name.
//
// +k8s:required
// +k8s:format=k8s-uuid
- string actor_uid = 5;
+ string actor_uid = 3;
// DER-encoded PKCS #10 certificate signing request. Atunnel retains the
// corresponding private key.
//
// +k8s:required
// +k8s:maxBytes=16384 # matches the control-plane MintActorCertificateRequest bound this is forwarded to
- bytes certificate_signing_request = 1;
+ bytes certificate_signing_request = 4;
}
message MintActorCertificateResponse {
@@ -313,6 +343,10 @@ message MintActorCertificateResponse {
//
// Called by ate-api-server over cluster networking. ate-api-server
// authenticates with its k8s pod identity mTLS certificate.
+//
+// atelet runs at most one Run, Restore, Checkpoint, PromoteSnapshot,
+// DetachActor, or Terminate of an actor at a time. A call waits for the one in
+// progress, up to the call's deadline.
service Atelet {
// Run tells atelet to create a new containerized workload from scratch on a
// worker.
@@ -348,12 +382,9 @@ service Atelet {
// report a DetachActor call, whose caller reads the state from the response.
// Waits for an operation already in progress on the actor, such as an
// upload or a detach. Does nothing when atelet holds no local snapshot for
- // the actor. Returns ABORTED while a Run, Restore, or Checkpoint of the
- // actor is in progress or the actor runs on this node, and
+ // the actor. Returns ABORTED when the actor runs on this node, and
// FAILED_PRECONDITION when atelet cannot make the latest local snapshot
// durable, such as when it has no `snapshot.object`.
- //
- // TODO: Temporarily, atelet returns UNIMPLEMENTED.
rpc DetachActor(DetachActorRequest) returns (DetachActorResponse) {}
// Terminate tells atelet to terminate/kill any running workload for an actor,
@@ -433,13 +464,6 @@ message RunRequest {
// +k8s:required
WorkloadSpec spec = 7;
- // The sandbox binaries to use for booting this actor from scratch. atelet
- // fetches the relevant assets and records them with the actor's on-node state
- // so a later Checkpoint can pin the same version into the snapshot manifest.
- //
- // +k8s:required
- SandboxAssets sandbox_assets = 8;
-
// When absent the actor has no egress: its TCP is captured and refused.
//
// +k8s:optional
@@ -459,13 +483,11 @@ message RunRequest {
int64 memory_bytes = 11; // Memory limit in bytes.
// The runtime the assigned worker runs this actor under, from
- // WorkerAssignment.sandbox_runtime.
- //
- // TODO: atelet still runs with sandbox_assets; a follow-up resolves the
- // SandboxConfig version from this field, makes it required, and removes
- // sandbox_assets.
+ // WorkerAssignment.sandbox_runtime. atelet resolves the pause image and
+ // sandbox binaries through its advertised-runtime index and records the
+ // runtime for the actor so a later Checkpoint pins the snapshot to it.
//
- // +k8s:optional
+ // +k8s:required
SandboxRuntime sandbox_runtime = 12;
}
@@ -480,64 +502,6 @@ message EgressGateway {
string address = 1;
}
-// AssetFile is one content-addressed file atelet fetches for a sandbox runtime
-// (e.g. the gVisor release tarball).
-message AssetFile {
- // URL to download the asset from. atelet fetches every URL through the
- // node's object-store plugin, and reads a gs:// URL anonymously only when
- // the plugin cannot serve it.
- // Masked in logged requests, since a signed URL carries its credential in
- // the query.
- //
- // +k8s:required
- // +k8s:maxLength=2048 # matches the SandboxConfig asset url's bound
- string url = 1 [ debug_redact = true ];
-
- // Lower-case hex SHA256; names the cached file and verifies its integrity.
- //
- // +k8s:required
- // +k8s:maxLength=64
- // +k8s:customValidation # 64 hex characters
- string sha256 = 2;
-}
-
-// ArchAssets is the set of assets for a single architecture, keyed by asset
-// name (a wrapper message because proto map values cannot themselves be maps).
-message ArchAssets {
- // +k8s:required
- // +k8s:maxProperties=16
- // +k8s:eachKey=+k8s:maxLength=64
- map files = 1; // asset name -> file
-}
-
-// SandboxAssets is the generic, backend-agnostic description of the sandbox
-// an actor runs in: a sandbox class, the pause image holding its namespaces,
-// and binaries keyed first by architecture (GOARCH) and then by asset name.
-// atelet's backend code interprets the asset names (gVisor expects "gvisor",
-// the release tarball).
-message SandboxAssets {
- // +k8s:required
- // +k8s:maxLength=63 # guardrail; the class names are a few characters
- // +k8s:customValidation # gvisor or microvm
- string sandbox_class = 1; // e.g. "gvisor"
-
- // +k8s:required
- // +k8s:maxProperties=8 # one entry per architecture
- // +k8s:eachKey=+k8s:maxLength=16
- map assets = 2; // arch -> {name -> file}
-
- // pause_image is the image for the sandbox's root container. Like the
- // binaries above it is sandbox configuration, not workload configuration,
- // and atelet pins it into the snapshot manifest so a restore rebuilds the
- // sandbox from the same image. Empty for sandboxes without a pause
- // container (microvm).
- //
- // +k8s:optional
- // +k8s:maxLength=512
- // +k8s:customValidation # must be a well-formed image reference, pinned by digest
- string pause_image = 3;
-}
-
// WorkloadSpec parallels Pod, but with far fewer configurable fields.
//
// +k8s:customValidation # every volume mount names a declared volume
@@ -563,12 +527,12 @@ message ExternalVolumeSource {
// sends volumes that finished provisioning.
//
// +k8s:required
- // +k8s:maxLength=256 # matches ExternalVolume.storage_volume_id's bound
+ // +k8s:maxLength=256 # matches ExternalVolumeStatus.storage_volume_id's bound
// +k8s:customValidation # no control characters
string storage_volume_id = 1;
// +k8s:optional
- // +k8s:maxLength=253 # matches ExternalVolume.volume_type's bound
+ // +k8s:maxLength=253 # matches ExternalVolumeStatus.volume_type's bound
// +k8s:customValidation # optional "substrate.io/" prefix + DNS subdomain
string volume_type = 2;
@@ -653,9 +617,7 @@ message TrustBundleDataSource {
// +k8s:listType=set
// +k8s:eachVal=+k8s:minLength=1
// +k8s:eachVal=+k8s:maxLength=253 # matches the template's bundle-name bound
- repeated string names = 3;
- reserved 2;
- reserved "name";
+ repeated string names = 2;
}
// SystemInfoDataSource selects exactly one projection to place in the
@@ -922,14 +884,10 @@ message Snapshot {
// +k8s:maximum=3 # keep this in sync with the SnapshotFidelity enum
SnapshotFidelity fidelity = 3;
- // The snapshot's durable copy in object storage. Required on Checkpoint
- // and PromoteSnapshot: atelet uploads there, or reads from there when it
- // holds no usable local copy.
+ // The snapshot's durable copy in object storage: atelet uploads there, or
+ // reads from there when it holds no usable local copy.
//
- // TODO: Temporarily, this is optional on RestoreRequest while the control
- // plane does not set it when restoring a paused actor.
- //
- // +k8s:optional
+ // +k8s:required
ObjectStorage object = 4;
}
@@ -1000,9 +958,6 @@ enum UploadMode {
// atelet reports the actor's state with
// WorkerService.ReportActorSnapshotState once such an upload makes a newer
// snapshot durable, and once such a detach has deleted the local snapshots.
- //
- // atelet never uploads or detaches a snapshot it cannot make durable, such
- // as one without `snapshot.object`.
UPLOAD_MODE_DEFERRED = 1;
// Upload the snapshot synchronously before Checkpoint returns, and keep a
@@ -1059,7 +1014,6 @@ message CheckpointRequest {
// workload.
//
// +k8s:required
- // +k8s:subfield(object)=+k8s:required
Snapshot snapshot = 8;
// When the snapshot becomes durable.
@@ -1119,9 +1073,9 @@ message RestoreRequest {
// detaching the actor, it waits for the detach to finish and then reads
// `snapshot.object`. Fails if neither copy exists: that generation is lost.
//
- // TODO: Temporarily, the control plane does not set `snapshot.object` when
- // it restores a paused actor, so atelet has nothing to fall back to if that
- // actor's local copy is missing or unusable.
+ // `snapshot.object` is required even while the control plane has not
+ // recorded the snapshot's upload complete: atelet may have uploaded the
+ // snapshot and detached the actor before its report of the upload landed.
//
// +k8s:required
Snapshot snapshot = 8;
@@ -1129,7 +1083,7 @@ message RestoreRequest {
// When absent the actor has no egress: its TCP is captured and refused.
//
// +k8s:optional
- optional EgressGateway egress_gateway = 13;
+ optional EgressGateway egress_gateway = 9;
// The actor's declared size, from the ActorTemplate's resource limits. For
// gVisor and micro-VM VOLUMES restores the sandbox is (re)sized to these;
@@ -1139,27 +1093,20 @@ message RestoreRequest {
// +k8s:optional
// +k8s:minimum=0
// +k8s:maximum=999999 # the control plane caps cpu limits strictly below 1000 cores
- int64 cpu_milli = 14; // CPU limit in millicores (1000 = one core).
+ int64 cpu_milli = 10; // CPU limit in millicores (1000 = one core).
// +k8s:optional
// +k8s:minimum=0
- int64 memory_bytes = 15; // Memory limit in bytes.
-
- // The sandbox binaries and pause image to restore with, resolved from the
- // assigned worker pool's SandboxConfig.
- //
- // +k8s:required
- SandboxAssets sandbox_assets = 16;
+ int64 memory_bytes = 11; // Memory limit in bytes.
// The runtime the assigned worker restores this actor under, from
- // WorkerAssignment.sandbox_runtime.
+ // WorkerAssignment.sandbox_runtime. atelet resolves the pause image and
+ // sandbox binaries through its advertised-runtime index and records the
+ // runtime for the actor so the next Checkpoint pins the snapshot to what ran
+ // this activation, not what the restored snapshot was taken under.
//
- // TODO: atelet still restores with sandbox_assets; a follow-up resolves the
- // SandboxConfig version from this field, makes it required, and removes
- // sandbox_assets.
- //
- // +k8s:optional
- SandboxRuntime sandbox_runtime = 17;
+ // +k8s:required
+ SandboxRuntime sandbox_runtime = 12;
}
message RestoreResponse {
@@ -1192,7 +1139,6 @@ message PromoteSnapshotRequest {
// it is, or will be, uploaded.
//
// +k8s:required
- // +k8s:subfield(object)=+k8s:required
Snapshot snapshot = 6;
}
diff --git a/internal/proto/ateletpb/atelet_grpc.pb.go b/internal/proto/ateletpb/atelet_grpc.pb.go
index 5a35d73bbe..e4f6a5f3b9 100644
--- a/internal/proto/ateletpb/atelet_grpc.pb.go
+++ b/internal/proto/ateletpb/atelet_grpc.pb.go
@@ -37,6 +37,7 @@ const (
WorkerSupport_RegisterWorker_FullMethodName = "/atelet.WorkerSupport/RegisterWorker"
WorkerSupport_RequestActorSuspend_FullMethodName = "/atelet.WorkerSupport/RequestActorSuspend"
WorkerSupport_GetActorEgressPolicy_FullMethodName = "/atelet.WorkerSupport/GetActorEgressPolicy"
+ WorkerSupport_ReportActorCrashed_FullMethodName = "/atelet.WorkerSupport/ReportActorCrashed"
)
// WorkerSupportClient is the client API for WorkerSupport service.
@@ -71,6 +72,10 @@ type WorkerSupportClient interface {
// Control.GetActorEgressPolicy. An error is the control plane's, passed
// through as it was given.
GetActorEgressPolicy(ctx context.Context, in *GetActorEgressPolicyRequest, opts ...grpc.CallOption) (*GetActorEgressPolicyResponse, error)
+ // ReportActorCrashed reports that the sandbox of the actor this worker
+ // hosts died on its own, for atelet to forward to the control plane's
+ // WorkerService, which this mirrors.
+ ReportActorCrashed(ctx context.Context, in *ReportActorCrashedRequest, opts ...grpc.CallOption) (*ReportActorCrashedResponse, error)
}
type workerSupportClient struct {
@@ -121,6 +126,16 @@ func (c *workerSupportClient) GetActorEgressPolicy(ctx context.Context, in *GetA
return out, nil
}
+func (c *workerSupportClient) ReportActorCrashed(ctx context.Context, in *ReportActorCrashedRequest, opts ...grpc.CallOption) (*ReportActorCrashedResponse, error) {
+ cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...)
+ out := new(ReportActorCrashedResponse)
+ err := c.cc.Invoke(ctx, WorkerSupport_ReportActorCrashed_FullMethodName, in, out, cOpts...)
+ if err != nil {
+ return nil, err
+ }
+ return out, nil
+}
+
// WorkerSupportServer is the server API for WorkerSupport service.
// All implementations must embed UnimplementedWorkerSupportServer
// for forward compatibility.
@@ -153,6 +168,10 @@ type WorkerSupportServer interface {
// Control.GetActorEgressPolicy. An error is the control plane's, passed
// through as it was given.
GetActorEgressPolicy(context.Context, *GetActorEgressPolicyRequest) (*GetActorEgressPolicyResponse, error)
+ // ReportActorCrashed reports that the sandbox of the actor this worker
+ // hosts died on its own, for atelet to forward to the control plane's
+ // WorkerService, which this mirrors.
+ ReportActorCrashed(context.Context, *ReportActorCrashedRequest) (*ReportActorCrashedResponse, error)
mustEmbedUnimplementedWorkerSupportServer()
}
@@ -175,6 +194,9 @@ func (UnimplementedWorkerSupportServer) RequestActorSuspend(context.Context, *Re
func (UnimplementedWorkerSupportServer) GetActorEgressPolicy(context.Context, *GetActorEgressPolicyRequest) (*GetActorEgressPolicyResponse, error) {
return nil, status.Error(codes.Unimplemented, "method GetActorEgressPolicy not implemented")
}
+func (UnimplementedWorkerSupportServer) ReportActorCrashed(context.Context, *ReportActorCrashedRequest) (*ReportActorCrashedResponse, error) {
+ return nil, status.Error(codes.Unimplemented, "method ReportActorCrashed not implemented")
+}
func (UnimplementedWorkerSupportServer) mustEmbedUnimplementedWorkerSupportServer() {}
func (UnimplementedWorkerSupportServer) testEmbeddedByValue() {}
@@ -268,6 +290,24 @@ func _WorkerSupport_GetActorEgressPolicy_Handler(srv interface{}, ctx context.Co
return interceptor(ctx, in, info, handler)
}
+func _WorkerSupport_ReportActorCrashed_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
+ in := new(ReportActorCrashedRequest)
+ if err := dec(in); err != nil {
+ return nil, err
+ }
+ if interceptor == nil {
+ return srv.(WorkerSupportServer).ReportActorCrashed(ctx, in)
+ }
+ info := &grpc.UnaryServerInfo{
+ Server: srv,
+ FullMethod: WorkerSupport_ReportActorCrashed_FullMethodName,
+ }
+ handler := func(ctx context.Context, req interface{}) (interface{}, error) {
+ return srv.(WorkerSupportServer).ReportActorCrashed(ctx, req.(*ReportActorCrashedRequest))
+ }
+ return interceptor(ctx, in, info, handler)
+}
+
// WorkerSupport_ServiceDesc is the grpc.ServiceDesc for WorkerSupport service.
// It's only intended for direct use with grpc.RegisterService,
// and not to be introspected or modified (even as a copy)
@@ -291,6 +331,10 @@ var WorkerSupport_ServiceDesc = grpc.ServiceDesc{
MethodName: "GetActorEgressPolicy",
Handler: _WorkerSupport_GetActorEgressPolicy_Handler,
},
+ {
+ MethodName: "ReportActorCrashed",
+ Handler: _WorkerSupport_ReportActorCrashed_Handler,
+ },
},
Streams: []grpc.StreamDesc{},
Metadata: "atelet.proto",
@@ -313,6 +357,10 @@ const (
//
// Called by ate-api-server over cluster networking. ate-api-server
// authenticates with its k8s pod identity mTLS certificate.
+//
+// atelet runs at most one Run, Restore, Checkpoint, PromoteSnapshot,
+// DetachActor, or Terminate of an actor at a time. A call waits for the one in
+// progress, up to the call's deadline.
type AteletClient interface {
// Run tells atelet to create a new containerized workload from scratch on a
// worker.
@@ -344,12 +392,9 @@ type AteletClient interface {
// report a DetachActor call, whose caller reads the state from the response.
// Waits for an operation already in progress on the actor, such as an
// upload or a detach. Does nothing when atelet holds no local snapshot for
- // the actor. Returns ABORTED while a Run, Restore, or Checkpoint of the
- // actor is in progress or the actor runs on this node, and
+ // the actor. Returns ABORTED when the actor runs on this node, and
// FAILED_PRECONDITION when atelet cannot make the latest local snapshot
// durable, such as when it has no `snapshot.object`.
- //
- // TODO: Temporarily, atelet returns UNIMPLEMENTED.
DetachActor(ctx context.Context, in *DetachActorRequest, opts ...grpc.CallOption) (*DetachActorResponse, error)
// Terminate tells atelet to terminate/kill any running workload for an actor,
// unmount its volumes, and clean up actor state on the node.
@@ -432,6 +477,10 @@ func (c *ateletClient) Terminate(ctx context.Context, in *TerminateRequest, opts
//
// Called by ate-api-server over cluster networking. ate-api-server
// authenticates with its k8s pod identity mTLS certificate.
+//
+// atelet runs at most one Run, Restore, Checkpoint, PromoteSnapshot,
+// DetachActor, or Terminate of an actor at a time. A call waits for the one in
+// progress, up to the call's deadline.
type AteletServer interface {
// Run tells atelet to create a new containerized workload from scratch on a
// worker.
@@ -463,12 +512,9 @@ type AteletServer interface {
// report a DetachActor call, whose caller reads the state from the response.
// Waits for an operation already in progress on the actor, such as an
// upload or a detach. Does nothing when atelet holds no local snapshot for
- // the actor. Returns ABORTED while a Run, Restore, or Checkpoint of the
- // actor is in progress or the actor runs on this node, and
+ // the actor. Returns ABORTED when the actor runs on this node, and
// FAILED_PRECONDITION when atelet cannot make the latest local snapshot
// durable, such as when it has no `snapshot.object`.
- //
- // TODO: Temporarily, atelet returns UNIMPLEMENTED.
DetachActor(context.Context, *DetachActorRequest) (*DetachActorResponse, error)
// Terminate tells atelet to terminate/kill any running workload for an actor,
// unmount its volumes, and clean up actor state on the node.
diff --git a/internal/proto/ateworkerpb/ateworker.pb.go b/internal/proto/ateworkerpb/ateworker.pb.go
index fdc172232b..3e4d6df0c0 100644
--- a/internal/proto/ateworkerpb/ateworker.pb.go
+++ b/internal/proto/ateworkerpb/ateworker.pb.go
@@ -484,14 +484,14 @@ type RunWorkloadRequest struct {
// the gVisor runtime, which uses runsc_path.
//
// +k8s:optional
- // +k8s:maxProperties=16 # matches atelet.ArchAssets.files
- // +k8s:eachKey=+k8s:maxLength=64 # matches atelet.ArchAssets.files
+ // +k8s:maxProperties=16
+ // +k8s:eachKey=+k8s:maxLength=64
// +k8s:eachVal=+k8s:maxLength=4096 # PATH_MAX; a host path, not a contract
RuntimeAssetPaths map[string]string `protobuf:"bytes,8,rep,name=runtime_asset_paths,json=runtimeAssetPaths,proto3" json:"runtime_asset_paths,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"`
// When absent the actor has no egress: its TCP is captured and refused.
//
// +k8s:optional
- EgressGateway *EgressGateway `protobuf:"bytes,10,opt,name=egress_gateway,json=egressGateway,proto3,oneof" json:"egress_gateway,omitempty"`
+ EgressGateway *EgressGateway `protobuf:"bytes,9,opt,name=egress_gateway,json=egressGateway,proto3,oneof" json:"egress_gateway,omitempty"`
// The actor's declared size, from the ActorTemplate's resource limits. The worker
// sizes the sandbox to these (cgroup caps via the OCI spec, and for the
// micro-VM the VM's vCPU count and memory). Zero means "unset": keep the
@@ -500,12 +500,12 @@ type RunWorkloadRequest struct {
// +k8s:optional
// +k8s:minimum=1
// +k8s:maximum=999999 # matches atelet.RunRequest.cpu_milli, the control plane caps cpu limits strictly below 1000 cores
- CpuMilli int64 `protobuf:"varint,11,opt,name=cpu_milli,json=cpuMilli,proto3" json:"cpu_milli,omitempty"` // CPU limit in millicores (1000 = one core).
+ CpuMilli int64 `protobuf:"varint,10,opt,name=cpu_milli,json=cpuMilli,proto3" json:"cpu_milli,omitempty"` // CPU limit in millicores (1000 = one core).
// +k8s:optional
// +k8s:minimum=1
- MemoryBytes int64 `protobuf:"varint,12,opt,name=memory_bytes,json=memoryBytes,proto3" json:"memory_bytes,omitempty"` // Memory limit in bytes.
+ MemoryBytes int64 `protobuf:"varint,11,opt,name=memory_bytes,json=memoryBytes,proto3" json:"memory_bytes,omitempty"` // Memory limit in bytes.
// +k8s:required
- ActorDirs *ActorDirs `protobuf:"bytes,13,opt,name=actor_dirs,json=actorDirs,proto3" json:"actor_dirs,omitempty"`
+ ActorDirs *ActorDirs `protobuf:"bytes,12,opt,name=actor_dirs,json=actorDirs,proto3" json:"actor_dirs,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
@@ -1248,9 +1248,9 @@ type CheckpointWorkloadRequest struct {
// +k8s:required
// +k8s:minimum=1
// +k8s:maximum=3 # keep this in sync with the SnapshotFidelity enum
- Fidelity SnapshotFidelity `protobuf:"varint,10,opt,name=fidelity,proto3,enum=ateworker.SnapshotFidelity" json:"fidelity,omitempty"`
+ Fidelity SnapshotFidelity `protobuf:"varint,8,opt,name=fidelity,proto3,enum=ateworker.SnapshotFidelity" json:"fidelity,omitempty"`
// +k8s:required
- ActorDirs *ActorDirs `protobuf:"bytes,11,opt,name=actor_dirs,json=actorDirs,proto3" json:"actor_dirs,omitempty"`
+ ActorDirs *ActorDirs `protobuf:"bytes,9,opt,name=actor_dirs,json=actorDirs,proto3" json:"actor_dirs,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
@@ -1434,20 +1434,20 @@ type RestoreWorkloadRequest struct {
// atelet fetched it to (see RunWorkloadRequest). Empty for gVisor.
//
// +k8s:optional
- // +k8s:maxProperties=16 # matches atelet.ArchAssets.files
- // +k8s:eachKey=+k8s:maxLength=64 # matches atelet.ArchAssets.files
+ // +k8s:maxProperties=16
+ // +k8s:eachKey=+k8s:maxLength=64
// +k8s:eachVal=+k8s:maxLength=4096 # PATH_MAX; a host path, not a contract
- RuntimeAssetPaths map[string]string `protobuf:"bytes,9,rep,name=runtime_asset_paths,json=runtimeAssetPaths,proto3" json:"runtime_asset_paths,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"`
+ RuntimeAssetPaths map[string]string `protobuf:"bytes,8,rep,name=runtime_asset_paths,json=runtimeAssetPaths,proto3" json:"runtime_asset_paths,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"`
// Fidelity to restore from the snapshot.
//
// +k8s:required
// +k8s:minimum=1
// +k8s:maximum=3 # keep this in sync with the SnapshotFidelity enum
- Fidelity SnapshotFidelity `protobuf:"varint,10,opt,name=fidelity,proto3,enum=ateworker.SnapshotFidelity" json:"fidelity,omitempty"`
+ Fidelity SnapshotFidelity `protobuf:"varint,9,opt,name=fidelity,proto3,enum=ateworker.SnapshotFidelity" json:"fidelity,omitempty"`
// When absent the actor has no egress: its TCP is captured and refused.
//
// +k8s:optional
- EgressGateway *EgressGateway `protobuf:"bytes,12,opt,name=egress_gateway,json=egressGateway,proto3,oneof" json:"egress_gateway,omitempty"`
+ EgressGateway *EgressGateway `protobuf:"bytes,10,opt,name=egress_gateway,json=egressGateway,proto3,oneof" json:"egress_gateway,omitempty"`
// The actor's declared size, from the ActorTemplate's resource limits. Used to
// (re)size the sandbox on a VOLUMES restore (fresh guest container). On a
// MEMORY micro-VM restore the size baked into the snapshot is authoritative and
@@ -1456,17 +1456,17 @@ type RestoreWorkloadRequest struct {
// +k8s:optional
// +k8s:minimum=1
// +k8s:maximum=999999 # matches atelet.RestoreRequest.cpu_milli, the control plane caps cpu limits strictly below 1000 cores
- CpuMilli int64 `protobuf:"varint,14,opt,name=cpu_milli,json=cpuMilli,proto3" json:"cpu_milli,omitempty"` // CPU limit in millicores (1000 = one core).
+ CpuMilli int64 `protobuf:"varint,11,opt,name=cpu_milli,json=cpuMilli,proto3" json:"cpu_milli,omitempty"` // CPU limit in millicores (1000 = one core).
// +k8s:optional
// +k8s:minimum=1
- MemoryBytes int64 `protobuf:"varint,15,opt,name=memory_bytes,json=memoryBytes,proto3" json:"memory_bytes,omitempty"` // Memory limit in bytes.
+ MemoryBytes int64 `protobuf:"varint,12,opt,name=memory_bytes,json=memoryBytes,proto3" json:"memory_bytes,omitempty"` // Memory limit in bytes.
// +k8s:required
- ActorDirs *ActorDirs `protobuf:"bytes,16,opt,name=actor_dirs,json=actorDirs,proto3" json:"actor_dirs,omitempty"`
+ ActorDirs *ActorDirs `protobuf:"bytes,13,opt,name=actor_dirs,json=actorDirs,proto3" json:"actor_dirs,omitempty"`
// actor_dirs.restore_dir is a preserved local snapshot, not a scratch copy:
// the worker must not delete or modify files in it.
//
// +k8s:optional
- PreserveRestoreDir bool `protobuf:"varint,17,opt,name=preserve_restore_dir,json=preserveRestoreDir,proto3" json:"preserve_restore_dir,omitempty"`
+ PreserveRestoreDir bool `protobuf:"varint,14,opt,name=preserve_restore_dir,json=preserveRestoreDir,proto3" json:"preserve_restore_dir,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
@@ -2031,12 +2031,12 @@ const file_ateworker_proto_rawDesc = "" +
"runsc_path\x18\x06 \x01(\tR\trunscPath\x12+\n" +
"\x04spec\x18\a \x01(\v2\x17.ateworker.WorkloadSpecR\x04spec\x12d\n" +
"\x13runtime_asset_paths\x18\b \x03(\v24.ateworker.RunWorkloadRequest.RuntimeAssetPathsEntryR\x11runtimeAssetPaths\x12D\n" +
- "\x0eegress_gateway\x18\n" +
- " \x01(\v2\x18.ateworker.EgressGatewayH\x00R\regressGateway\x88\x01\x01\x12\x1b\n" +
- "\tcpu_milli\x18\v \x01(\x03R\bcpuMilli\x12!\n" +
- "\fmemory_bytes\x18\f \x01(\x03R\vmemoryBytes\x123\n" +
+ "\x0eegress_gateway\x18\t \x01(\v2\x18.ateworker.EgressGatewayH\x00R\regressGateway\x88\x01\x01\x12\x1b\n" +
+ "\tcpu_milli\x18\n" +
+ " \x01(\x03R\bcpuMilli\x12!\n" +
+ "\fmemory_bytes\x18\v \x01(\x03R\vmemoryBytes\x123\n" +
"\n" +
- "actor_dirs\x18\r \x01(\v2\x14.ateworker.ActorDirsR\tactorDirs\x1aD\n" +
+ "actor_dirs\x18\f \x01(\v2\x14.ateworker.ActorDirsR\tactorDirs\x1aD\n" +
"\x16RuntimeAssetPathsEntry\x12\x10\n" +
"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01B\x11\n" +
@@ -2085,10 +2085,9 @@ const file_ateworker_proto_rawDesc = "" +
"\n" +
"runsc_path\x18\x06 \x01(\tR\trunscPath\x12+\n" +
"\x04spec\x18\a \x01(\v2\x17.ateworker.WorkloadSpecR\x04spec\x127\n" +
- "\bfidelity\x18\n" +
- " \x01(\x0e2\x1b.ateworker.SnapshotFidelityR\bfidelity\x123\n" +
+ "\bfidelity\x18\b \x01(\x0e2\x1b.ateworker.SnapshotFidelityR\bfidelity\x123\n" +
"\n" +
- "actor_dirs\x18\v \x01(\v2\x14.ateworker.ActorDirsR\tactorDirs\"s\n" +
+ "actor_dirs\x18\t \x01(\v2\x14.ateworker.ActorDirsR\tactorDirs\"s\n" +
"\x1aCheckpointWorkloadResponse\x12%\n" +
"\x0esnapshot_files\x18\x01 \x03(\tR\rsnapshotFiles\x12.\n" +
"\x13data_snapshot_files\x18\x02 \x03(\tR\x11dataSnapshotFiles\"\x8d\x06\n" +
@@ -2102,15 +2101,15 @@ const file_ateworker_proto_rawDesc = "" +
"\n" +
"runsc_path\x18\x06 \x01(\tR\trunscPath\x12+\n" +
"\x04spec\x18\a \x01(\v2\x17.ateworker.WorkloadSpecR\x04spec\x12h\n" +
- "\x13runtime_asset_paths\x18\t \x03(\v28.ateworker.RestoreWorkloadRequest.RuntimeAssetPathsEntryR\x11runtimeAssetPaths\x127\n" +
- "\bfidelity\x18\n" +
- " \x01(\x0e2\x1b.ateworker.SnapshotFidelityR\bfidelity\x12D\n" +
- "\x0eegress_gateway\x18\f \x01(\v2\x18.ateworker.EgressGatewayH\x00R\regressGateway\x88\x01\x01\x12\x1b\n" +
- "\tcpu_milli\x18\x0e \x01(\x03R\bcpuMilli\x12!\n" +
- "\fmemory_bytes\x18\x0f \x01(\x03R\vmemoryBytes\x123\n" +
+ "\x13runtime_asset_paths\x18\b \x03(\v28.ateworker.RestoreWorkloadRequest.RuntimeAssetPathsEntryR\x11runtimeAssetPaths\x127\n" +
+ "\bfidelity\x18\t \x01(\x0e2\x1b.ateworker.SnapshotFidelityR\bfidelity\x12D\n" +
+ "\x0eegress_gateway\x18\n" +
+ " \x01(\v2\x18.ateworker.EgressGatewayH\x00R\regressGateway\x88\x01\x01\x12\x1b\n" +
+ "\tcpu_milli\x18\v \x01(\x03R\bcpuMilli\x12!\n" +
+ "\fmemory_bytes\x18\f \x01(\x03R\vmemoryBytes\x123\n" +
"\n" +
- "actor_dirs\x18\x10 \x01(\v2\x14.ateworker.ActorDirsR\tactorDirs\x120\n" +
- "\x14preserve_restore_dir\x18\x11 \x01(\bR\x12preserveRestoreDir\x1aD\n" +
+ "actor_dirs\x18\r \x01(\v2\x14.ateworker.ActorDirsR\tactorDirs\x120\n" +
+ "\x14preserve_restore_dir\x18\x0e \x01(\bR\x12preserveRestoreDir\x1aD\n" +
"\x16RuntimeAssetPathsEntry\x12\x10\n" +
"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01B\x11\n" +
diff --git a/internal/proto/ateworkerpb/ateworker.proto b/internal/proto/ateworkerpb/ateworker.proto
index 4e07f820f7..8b2e32abdc 100644
--- a/internal/proto/ateworkerpb/ateworker.proto
+++ b/internal/proto/ateworkerpb/ateworker.proto
@@ -224,15 +224,15 @@ message RunWorkloadRequest {
// the gVisor runtime, which uses runsc_path.
//
// +k8s:optional
- // +k8s:maxProperties=16 # matches atelet.ArchAssets.files
- // +k8s:eachKey=+k8s:maxLength=64 # matches atelet.ArchAssets.files
+ // +k8s:maxProperties=16
+ // +k8s:eachKey=+k8s:maxLength=64
// +k8s:eachVal=+k8s:maxLength=4096 # PATH_MAX; a host path, not a contract
map runtime_asset_paths = 8;
// When absent the actor has no egress: its TCP is captured and refused.
//
// +k8s:optional
- optional EgressGateway egress_gateway = 10;
+ optional EgressGateway egress_gateway = 9;
// The actor's declared size, from the ActorTemplate's resource limits. The worker
// sizes the sandbox to these (cgroup caps via the OCI spec, and for the
@@ -242,12 +242,12 @@ message RunWorkloadRequest {
// +k8s:optional
// +k8s:minimum=1
// +k8s:maximum=999999 # matches atelet.RunRequest.cpu_milli, the control plane caps cpu limits strictly below 1000 cores
- int64 cpu_milli = 11; // CPU limit in millicores (1000 = one core).
+ int64 cpu_milli = 10; // CPU limit in millicores (1000 = one core).
// +k8s:optional
// +k8s:minimum=1
- int64 memory_bytes = 12; // Memory limit in bytes.
+ int64 memory_bytes = 11; // Memory limit in bytes.
// +k8s:required
- ActorDirs actor_dirs = 13;
+ ActorDirs actor_dirs = 12;
}
// EgressGateway configures tunneled egress for one actor activation.
@@ -459,9 +459,9 @@ message CheckpointWorkloadRequest {
// +k8s:required
// +k8s:minimum=1
// +k8s:maximum=3 # keep this in sync with the SnapshotFidelity enum
- SnapshotFidelity fidelity = 10;
+ SnapshotFidelity fidelity = 8;
// +k8s:required
- ActorDirs actor_dirs = 11;
+ ActorDirs actor_dirs = 9;
}
message CheckpointWorkloadResponse {
@@ -506,22 +506,22 @@ message RestoreWorkloadRequest {
// atelet fetched it to (see RunWorkloadRequest). Empty for gVisor.
//
// +k8s:optional
- // +k8s:maxProperties=16 # matches atelet.ArchAssets.files
- // +k8s:eachKey=+k8s:maxLength=64 # matches atelet.ArchAssets.files
+ // +k8s:maxProperties=16
+ // +k8s:eachKey=+k8s:maxLength=64
// +k8s:eachVal=+k8s:maxLength=4096 # PATH_MAX; a host path, not a contract
- map runtime_asset_paths = 9;
+ map runtime_asset_paths = 8;
// Fidelity to restore from the snapshot.
//
// +k8s:required
// +k8s:minimum=1
// +k8s:maximum=3 # keep this in sync with the SnapshotFidelity enum
- SnapshotFidelity fidelity = 10;
+ SnapshotFidelity fidelity = 9;
// When absent the actor has no egress: its TCP is captured and refused.
//
// +k8s:optional
- optional EgressGateway egress_gateway = 12;
+ optional EgressGateway egress_gateway = 10;
// The actor's declared size, from the ActorTemplate's resource limits. Used to
// (re)size the sandbox on a VOLUMES restore (fresh guest container). On a
@@ -531,18 +531,18 @@ message RestoreWorkloadRequest {
// +k8s:optional
// +k8s:minimum=1
// +k8s:maximum=999999 # matches atelet.RestoreRequest.cpu_milli, the control plane caps cpu limits strictly below 1000 cores
- int64 cpu_milli = 14; // CPU limit in millicores (1000 = one core).
+ int64 cpu_milli = 11; // CPU limit in millicores (1000 = one core).
// +k8s:optional
// +k8s:minimum=1
- int64 memory_bytes = 15; // Memory limit in bytes.
+ int64 memory_bytes = 12; // Memory limit in bytes.
// +k8s:required
- ActorDirs actor_dirs = 16;
+ ActorDirs actor_dirs = 13;
// actor_dirs.restore_dir is a preserved local snapshot, not a scratch copy:
// the worker must not delete or modify files in it.
//
// +k8s:optional
- bool preserve_restore_dir = 17;
+ bool preserve_restore_dir = 14;
}
message RestoreWorkloadResponse {
diff --git a/internal/proto/objectstoreexchangepb/exchange.pb.go b/internal/proto/objectstoreexchangepb/exchange.pb.go
new file mode 100644
index 0000000000..4c4973ad1e
--- /dev/null
+++ b/internal/proto/objectstoreexchangepb/exchange.pb.go
@@ -0,0 +1,217 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Code generated by protoc-gen-go. DO NOT EDIT.
+// versions:
+// protoc-gen-go v1.36.11-devel
+// protoc v4.25.3
+// source: exchange.proto
+
+package objectstoreexchangepb
+
+import (
+ protoreflect "google.golang.org/protobuf/reflect/protoreflect"
+ protoimpl "google.golang.org/protobuf/runtime/protoimpl"
+ timestamppb "google.golang.org/protobuf/types/known/timestamppb"
+ reflect "reflect"
+ sync "sync"
+ unsafe "unsafe"
+)
+
+const (
+ // Verify that this generated code is sufficiently up-to-date.
+ _ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion)
+ // Verify that runtime/protoimpl is sufficiently up-to-date.
+ _ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20)
+)
+
+type ExchangeActorJwtRequest struct {
+ state protoimpl.MessageState `protogen:"open.v1"`
+ // A worker-for-actor JWT minted by ate-api-server for the audience the
+ // plugin requires. It names the node it was minted for, and is good only
+ // while the actor is on that node.
+ ActorJwt string `protobuf:"bytes,1,opt,name=actor_jwt,json=actorJwt,proto3" json:"actor_jwt,omitempty"`
+ // force skips the control plugin's token cache and mints a fresh credential
+ // from the actor's current state. The node plugin sets it after storage
+ // refused a credential it obtained earlier.
+ Force bool `protobuf:"varint,2,opt,name=force,proto3" json:"force,omitempty"`
+ unknownFields protoimpl.UnknownFields
+ sizeCache protoimpl.SizeCache
+}
+
+func (x *ExchangeActorJwtRequest) Reset() {
+ *x = ExchangeActorJwtRequest{}
+ mi := &file_exchange_proto_msgTypes[0]
+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+ ms.StoreMessageInfo(mi)
+}
+
+func (x *ExchangeActorJwtRequest) String() string {
+ return protoimpl.X.MessageStringOf(x)
+}
+
+func (*ExchangeActorJwtRequest) ProtoMessage() {}
+
+func (x *ExchangeActorJwtRequest) ProtoReflect() protoreflect.Message {
+ mi := &file_exchange_proto_msgTypes[0]
+ if x != nil {
+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+ if ms.LoadMessageInfo() == nil {
+ ms.StoreMessageInfo(mi)
+ }
+ return ms
+ }
+ return mi.MessageOf(x)
+}
+
+// Deprecated: Use ExchangeActorJwtRequest.ProtoReflect.Descriptor instead.
+func (*ExchangeActorJwtRequest) Descriptor() ([]byte, []int) {
+ return file_exchange_proto_rawDescGZIP(), []int{0}
+}
+
+func (x *ExchangeActorJwtRequest) GetActorJwt() string {
+ if x != nil {
+ return x.ActorJwt
+ }
+ return ""
+}
+
+func (x *ExchangeActorJwtRequest) GetForce() bool {
+ if x != nil {
+ return x.Force
+ }
+ return false
+}
+
+type ExchangeActorJwtResponse struct {
+ state protoimpl.MessageState `protogen:"open.v1"`
+ // The GCS OAuth access token.
+ AccessToken string `protobuf:"bytes,1,opt,name=access_token,json=accessToken,proto3" json:"access_token,omitempty"`
+ // When the credential expires.
+ ExpireTime *timestamppb.Timestamp `protobuf:"bytes,2,opt,name=expire_time,json=expireTime,proto3" json:"expire_time,omitempty"`
+ unknownFields protoimpl.UnknownFields
+ sizeCache protoimpl.SizeCache
+}
+
+func (x *ExchangeActorJwtResponse) Reset() {
+ *x = ExchangeActorJwtResponse{}
+ mi := &file_exchange_proto_msgTypes[1]
+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+ ms.StoreMessageInfo(mi)
+}
+
+func (x *ExchangeActorJwtResponse) String() string {
+ return protoimpl.X.MessageStringOf(x)
+}
+
+func (*ExchangeActorJwtResponse) ProtoMessage() {}
+
+func (x *ExchangeActorJwtResponse) ProtoReflect() protoreflect.Message {
+ mi := &file_exchange_proto_msgTypes[1]
+ if x != nil {
+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+ if ms.LoadMessageInfo() == nil {
+ ms.StoreMessageInfo(mi)
+ }
+ return ms
+ }
+ return mi.MessageOf(x)
+}
+
+// Deprecated: Use ExchangeActorJwtResponse.ProtoReflect.Descriptor instead.
+func (*ExchangeActorJwtResponse) Descriptor() ([]byte, []int) {
+ return file_exchange_proto_rawDescGZIP(), []int{1}
+}
+
+func (x *ExchangeActorJwtResponse) GetAccessToken() string {
+ if x != nil {
+ return x.AccessToken
+ }
+ return ""
+}
+
+func (x *ExchangeActorJwtResponse) GetExpireTime() *timestamppb.Timestamp {
+ if x != nil {
+ return x.ExpireTime
+ }
+ return nil
+}
+
+var File_exchange_proto protoreflect.FileDescriptor
+
+const file_exchange_proto_rawDesc = "" +
+ "\n" +
+ "\x0eexchange.proto\x12\x13objectstoreexchange\x1a\x1fgoogle/protobuf/timestamp.proto\"Q\n" +
+ "\x17ExchangeActorJwtRequest\x12 \n" +
+ "\tactor_jwt\x18\x01 \x01(\tB\x03\x80\x01\x01R\bactorJwt\x12\x14\n" +
+ "\x05force\x18\x02 \x01(\bR\x05force\"\x7f\n" +
+ "\x18ExchangeActorJwtResponse\x12&\n" +
+ "\faccess_token\x18\x01 \x01(\tB\x03\x80\x01\x01R\vaccessToken\x12;\n" +
+ "\vexpire_time\x18\x02 \x01(\v2\x1a.google.protobuf.TimestampR\n" +
+ "expireTime2\x85\x01\n" +
+ "\x12CredentialExchange\x12o\n" +
+ "\x10ExchangeActorJwt\x12,.objectstoreexchange.ExchangeActorJwtRequest\x1a-.objectstoreexchange.ExchangeActorJwtResponseB;Z9go.ate.dev/substrate/internal/proto/objectstoreexchangepbb\x06proto3"
+
+var (
+ file_exchange_proto_rawDescOnce sync.Once
+ file_exchange_proto_rawDescData []byte
+)
+
+func file_exchange_proto_rawDescGZIP() []byte {
+ file_exchange_proto_rawDescOnce.Do(func() {
+ file_exchange_proto_rawDescData = protoimpl.X.CompressGZIP(unsafe.Slice(unsafe.StringData(file_exchange_proto_rawDesc), len(file_exchange_proto_rawDesc)))
+ })
+ return file_exchange_proto_rawDescData
+}
+
+var file_exchange_proto_msgTypes = make([]protoimpl.MessageInfo, 2)
+var file_exchange_proto_goTypes = []any{
+ (*ExchangeActorJwtRequest)(nil), // 0: objectstoreexchange.ExchangeActorJwtRequest
+ (*ExchangeActorJwtResponse)(nil), // 1: objectstoreexchange.ExchangeActorJwtResponse
+ (*timestamppb.Timestamp)(nil), // 2: google.protobuf.Timestamp
+}
+var file_exchange_proto_depIdxs = []int32{
+ 2, // 0: objectstoreexchange.ExchangeActorJwtResponse.expire_time:type_name -> google.protobuf.Timestamp
+ 0, // 1: objectstoreexchange.CredentialExchange.ExchangeActorJwt:input_type -> objectstoreexchange.ExchangeActorJwtRequest
+ 1, // 2: objectstoreexchange.CredentialExchange.ExchangeActorJwt:output_type -> objectstoreexchange.ExchangeActorJwtResponse
+ 2, // [2:3] is the sub-list for method output_type
+ 1, // [1:2] is the sub-list for method input_type
+ 1, // [1:1] is the sub-list for extension type_name
+ 1, // [1:1] is the sub-list for extension extendee
+ 0, // [0:1] is the sub-list for field type_name
+}
+
+func init() { file_exchange_proto_init() }
+func file_exchange_proto_init() {
+ if File_exchange_proto != nil {
+ return
+ }
+ type x struct{}
+ out := protoimpl.TypeBuilder{
+ File: protoimpl.DescBuilder{
+ GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
+ RawDescriptor: unsafe.Slice(unsafe.StringData(file_exchange_proto_rawDesc), len(file_exchange_proto_rawDesc)),
+ NumEnums: 0,
+ NumMessages: 2,
+ NumExtensions: 0,
+ NumServices: 1,
+ },
+ GoTypes: file_exchange_proto_goTypes,
+ DependencyIndexes: file_exchange_proto_depIdxs,
+ MessageInfos: file_exchange_proto_msgTypes,
+ }.Build()
+ File_exchange_proto = out.File
+ file_exchange_proto_goTypes = nil
+ file_exchange_proto_depIdxs = nil
+}
diff --git a/internal/proto/objectstoreexchangepb/exchange.proto b/internal/proto/objectstoreexchangepb/exchange.proto
new file mode 100644
index 0000000000..46809af738
--- /dev/null
+++ b/internal/proto/objectstoreexchangepb/exchange.proto
@@ -0,0 +1,63 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+syntax = "proto3";
+
+package objectstoreexchange;
+
+import "google/protobuf/timestamp.proto";
+
+option go_package = "go.ate.dev/substrate/internal/proto/objectstoreexchangepb";
+
+// CredentialExchange is served by the objectstore control plugin, next to
+// ate-api-server, and called by the node plugin before a snapshot transfer. It
+// turns a worker-for-actor JWT into a storage credential scoped to that
+// actor's snapshot prefixes. It is private to the objectstore plugin.
+service CredentialExchange {
+ // ExchangeActorJwt verifies actor_jwt and returns a credential limited to
+ // the actor's snapshot prefixes. Errors:
+ // - INVALID_ARGUMENT: actor_jwt is empty.
+ // - UNAUTHENTICATED: actor_jwt fails signature, audience or expiry, or
+ // does not name an actor.
+ // - PERMISSION_DENIED: actor_jwt is not a worker-for-actor token, the
+ // actor has no snapshot location, the actor has left the node the token
+ // was minted for, or the actor is in a state with no transfer.
+ // - NOT_FOUND: the actor is gone or was recreated.
+ // - FAILED_PRECONDITION: the exchange is not allowed to read actors.
+ // - INTERNAL: the actor's grants cannot be derived or expressed as an
+ // access boundary.
+ // - UNAVAILABLE: ate-api-server or the credential issuer cannot be reached.
+ // - CANCELLED, DEADLINE_EXCEEDED: the caller gave up first.
+ rpc ExchangeActorJwt(ExchangeActorJwtRequest) returns (ExchangeActorJwtResponse);
+}
+
+message ExchangeActorJwtRequest {
+ // A worker-for-actor JWT minted by ate-api-server for the audience the
+ // plugin requires. It names the node it was minted for, and is good only
+ // while the actor is on that node.
+ string actor_jwt = 1 [ debug_redact = true ];
+
+ // force skips the control plugin's token cache and mints a fresh credential
+ // from the actor's current state. The node plugin sets it after storage
+ // refused a credential it obtained earlier.
+ bool force = 2;
+}
+
+message ExchangeActorJwtResponse {
+ // The GCS OAuth access token.
+ string access_token = 1 [ debug_redact = true ];
+
+ // When the credential expires.
+ google.protobuf.Timestamp expire_time = 2;
+}
diff --git a/internal/proto/objectstoreexchangepb/exchange_grpc.pb.go b/internal/proto/objectstoreexchangepb/exchange_grpc.pb.go
new file mode 100644
index 0000000000..696d42a354
--- /dev/null
+++ b/internal/proto/objectstoreexchangepb/exchange_grpc.pb.go
@@ -0,0 +1,173 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Code generated by protoc-gen-go-grpc. DO NOT EDIT.
+// versions:
+// - protoc-gen-go-grpc v1.6.2
+// - protoc v4.25.3
+// source: exchange.proto
+
+package objectstoreexchangepb
+
+import (
+ context "context"
+ grpc "google.golang.org/grpc"
+ codes "google.golang.org/grpc/codes"
+ status "google.golang.org/grpc/status"
+)
+
+// This is a compile-time assertion to ensure that this generated file
+// is compatible with the grpc package it is being compiled against.
+// Requires gRPC-Go v1.64.0 or later.
+const _ = grpc.SupportPackageIsVersion9
+
+const (
+ CredentialExchange_ExchangeActorJwt_FullMethodName = "/objectstoreexchange.CredentialExchange/ExchangeActorJwt"
+)
+
+// CredentialExchangeClient is the client API for CredentialExchange service.
+//
+// For semantics around ctx use and closing/ending streaming RPCs, please refer to https://pkg.go.dev/google.golang.org/grpc/?tab=doc#ClientConn.NewStream.
+//
+// CredentialExchange is served by the objectstore control plugin, next to
+// ate-api-server, and called by the node plugin before a snapshot transfer. It
+// turns a worker-for-actor JWT into a storage credential scoped to that
+// actor's snapshot prefixes. It is private to the objectstore plugin.
+type CredentialExchangeClient interface {
+ // ExchangeActorJwt verifies actor_jwt and returns a credential limited to
+ // the actor's snapshot prefixes. Errors:
+ // - INVALID_ARGUMENT: actor_jwt is empty.
+ // - UNAUTHENTICATED: actor_jwt fails signature, audience or expiry, or
+ // does not name an actor.
+ // - PERMISSION_DENIED: actor_jwt is not a worker-for-actor token, the
+ // actor has no snapshot location, the actor has left the node the token
+ // was minted for, or the actor is in a state with no transfer.
+ // - NOT_FOUND: the actor is gone or was recreated.
+ // - FAILED_PRECONDITION: the exchange is not allowed to read actors.
+ // - INTERNAL: the actor's grants cannot be derived or expressed as an
+ // access boundary.
+ // - UNAVAILABLE: ate-api-server or the credential issuer cannot be reached.
+ // - CANCELLED, DEADLINE_EXCEEDED: the caller gave up first.
+ ExchangeActorJwt(ctx context.Context, in *ExchangeActorJwtRequest, opts ...grpc.CallOption) (*ExchangeActorJwtResponse, error)
+}
+
+type credentialExchangeClient struct {
+ cc grpc.ClientConnInterface
+}
+
+func NewCredentialExchangeClient(cc grpc.ClientConnInterface) CredentialExchangeClient {
+ return &credentialExchangeClient{cc}
+}
+
+func (c *credentialExchangeClient) ExchangeActorJwt(ctx context.Context, in *ExchangeActorJwtRequest, opts ...grpc.CallOption) (*ExchangeActorJwtResponse, error) {
+ cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...)
+ out := new(ExchangeActorJwtResponse)
+ err := c.cc.Invoke(ctx, CredentialExchange_ExchangeActorJwt_FullMethodName, in, out, cOpts...)
+ if err != nil {
+ return nil, err
+ }
+ return out, nil
+}
+
+// CredentialExchangeServer is the server API for CredentialExchange service.
+// All implementations must embed UnimplementedCredentialExchangeServer
+// for forward compatibility.
+//
+// CredentialExchange is served by the objectstore control plugin, next to
+// ate-api-server, and called by the node plugin before a snapshot transfer. It
+// turns a worker-for-actor JWT into a storage credential scoped to that
+// actor's snapshot prefixes. It is private to the objectstore plugin.
+type CredentialExchangeServer interface {
+ // ExchangeActorJwt verifies actor_jwt and returns a credential limited to
+ // the actor's snapshot prefixes. Errors:
+ // - INVALID_ARGUMENT: actor_jwt is empty.
+ // - UNAUTHENTICATED: actor_jwt fails signature, audience or expiry, or
+ // does not name an actor.
+ // - PERMISSION_DENIED: actor_jwt is not a worker-for-actor token, the
+ // actor has no snapshot location, the actor has left the node the token
+ // was minted for, or the actor is in a state with no transfer.
+ // - NOT_FOUND: the actor is gone or was recreated.
+ // - FAILED_PRECONDITION: the exchange is not allowed to read actors.
+ // - INTERNAL: the actor's grants cannot be derived or expressed as an
+ // access boundary.
+ // - UNAVAILABLE: ate-api-server or the credential issuer cannot be reached.
+ // - CANCELLED, DEADLINE_EXCEEDED: the caller gave up first.
+ ExchangeActorJwt(context.Context, *ExchangeActorJwtRequest) (*ExchangeActorJwtResponse, error)
+ mustEmbedUnimplementedCredentialExchangeServer()
+}
+
+// UnimplementedCredentialExchangeServer must be embedded to have
+// forward compatible implementations.
+//
+// NOTE: this should be embedded by value instead of pointer to avoid a nil
+// pointer dereference when methods are called.
+type UnimplementedCredentialExchangeServer struct{}
+
+func (UnimplementedCredentialExchangeServer) ExchangeActorJwt(context.Context, *ExchangeActorJwtRequest) (*ExchangeActorJwtResponse, error) {
+ return nil, status.Error(codes.Unimplemented, "method ExchangeActorJwt not implemented")
+}
+func (UnimplementedCredentialExchangeServer) mustEmbedUnimplementedCredentialExchangeServer() {}
+func (UnimplementedCredentialExchangeServer) testEmbeddedByValue() {}
+
+// UnsafeCredentialExchangeServer may be embedded to opt out of forward compatibility for this service.
+// Use of this interface is not recommended, as added methods to CredentialExchangeServer will
+// result in compilation errors.
+type UnsafeCredentialExchangeServer interface {
+ mustEmbedUnimplementedCredentialExchangeServer()
+}
+
+func RegisterCredentialExchangeServer(s grpc.ServiceRegistrar, srv CredentialExchangeServer) {
+ // If the following call panics, it indicates UnimplementedCredentialExchangeServer was
+ // embedded by pointer and is nil. This will cause panics if an
+ // unimplemented method is ever invoked, so we test this at initialization
+ // time to prevent it from happening at runtime later due to I/O.
+ if t, ok := srv.(interface{ testEmbeddedByValue() }); ok {
+ t.testEmbeddedByValue()
+ }
+ s.RegisterService(&CredentialExchange_ServiceDesc, srv)
+}
+
+func _CredentialExchange_ExchangeActorJwt_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
+ in := new(ExchangeActorJwtRequest)
+ if err := dec(in); err != nil {
+ return nil, err
+ }
+ if interceptor == nil {
+ return srv.(CredentialExchangeServer).ExchangeActorJwt(ctx, in)
+ }
+ info := &grpc.UnaryServerInfo{
+ Server: srv,
+ FullMethod: CredentialExchange_ExchangeActorJwt_FullMethodName,
+ }
+ handler := func(ctx context.Context, req interface{}) (interface{}, error) {
+ return srv.(CredentialExchangeServer).ExchangeActorJwt(ctx, req.(*ExchangeActorJwtRequest))
+ }
+ return interceptor(ctx, in, info, handler)
+}
+
+// CredentialExchange_ServiceDesc is the grpc.ServiceDesc for CredentialExchange service.
+// It's only intended for direct use with grpc.RegisterService,
+// and not to be introspected or modified (even as a copy)
+var CredentialExchange_ServiceDesc = grpc.ServiceDesc{
+ ServiceName: "objectstoreexchange.CredentialExchange",
+ HandlerType: (*CredentialExchangeServer)(nil),
+ Methods: []grpc.MethodDesc{
+ {
+ MethodName: "ExchangeActorJwt",
+ Handler: _CredentialExchange_ExchangeActorJwt_Handler,
+ },
+ },
+ Streams: []grpc.StreamDesc{},
+ Metadata: "exchange.proto",
+}
diff --git a/internal/protoredact/protoredact_test.go b/internal/protoredact/protoredact_test.go
index ce8057df31..928b601961 100644
--- a/internal/protoredact/protoredact_test.go
+++ b/internal/protoredact/protoredact_test.go
@@ -23,10 +23,12 @@ import (
"sync"
"testing"
+ ateenvv1alpha "go.ate.dev/substrate/internal/proto/ateenv/v1alpha"
"go.ate.dev/substrate/internal/proto/ateletpb"
"go.ate.dev/substrate/internal/proto/ateworkerpb"
"go.ate.dev/substrate/internal/proto/glutton"
"go.ate.dev/substrate/internal/proto/grpcechopb"
+ "go.ate.dev/substrate/internal/proto/objectstoreexchangepb"
"go.ate.dev/substrate/internal/protoredact"
"go.ate.dev/substrate/pkg/proto/ateapipb"
"go.ate.dev/substrate/pkg/proto/ateconfigpb"
@@ -39,56 +41,35 @@ import (
"google.golang.org/protobuf/types/dynamicpb"
)
-// TestRedactedRecursesIntoRealMapFields walks real messages whose maps hold
-// messages (atelet sandbox assets) and strings (ateapi selectors): map values
-// are visited without panicking, labeled fields in them are masked, and
-// non-sensitive content is left intact.
-func TestRedactedRecursesIntoRealMapFields(t *testing.T) {
+// TestRedactedSkipsUnlabeledScalarMaps pins the reachability short-circuit on
+// a real message: a map has no field that could carry
+// debug_redact, so redact skips it instead of descending into values that are
+// not messages (which would panic), while a labeled field elsewhere in the
+// same message is still masked and the original is left alone.
+func TestRedactedSkipsUnlabeledScalarMaps(t *testing.T) {
run := &ateletpb.RunRequest{
- SandboxAssets: &ateletpb.SandboxAssets{
- SandboxClass: "gvisor",
- Assets: map[string]*ateletpb.ArchAssets{
- "amd64": {Files: map[string]*ateletpb.AssetFile{
- "runsc": {Url: "https://assets.example/runsc", Sha256: "abc123"},
- }},
- },
+ Spec: &ateletpb.WorkloadSpec{
+ Volumes: []*ateletpb.Volume{{
+ Name: "data",
+ External: &ateletpb.ExternalVolumeSource{VolumeContext: map[string]string{"fsType": "ext4"}},
+ }},
+ Containers: []*ateletpb.Container{{
+ Name: "main",
+ Env: []*ateletpb.EnvVar{{Name: "API_KEY", Value: "sk-secret"}},
+ }},
},
- Spec: &ateletpb.WorkloadSpec{Containers: []*ateletpb.Container{{
- Name: "main",
- Env: []*ateletpb.EnvVar{{Name: "API_KEY", Value: "sk-secret"}},
- }}},
}
redacted, _ := protoredact.Redacted(run)
got := redacted.(*ateletpb.RunRequest)
if v := got.GetSpec().GetContainers()[0].GetEnv()[0].GetValue(); v != protoredact.Placeholder {
t.Fatalf("env value = %q, want placeholder", v)
}
- // Inside the map value, the labeled url is masked and the sha256 is kept.
- file := got.GetSandboxAssets().GetAssets()["amd64"].GetFiles()["runsc"]
- if u := file.GetUrl(); u != protoredact.Placeholder {
- t.Fatalf("asset url in a map value = %q, want placeholder", u)
- }
- if sum := file.GetSha256(); sum != "abc123" {
- t.Fatalf("map-of-message content was altered: sha256 = %q", sum)
+ if fsType := got.GetSpec().GetVolumes()[0].GetExternal().GetVolumeContext()["fsType"]; fsType != "ext4" {
+ t.Fatalf("scalar map was altered: fsType = %q", fsType)
}
if run.GetSpec().GetContainers()[0].GetEnv()[0].GetValue() != "sk-secret" {
t.Fatal("original mutated")
}
-
- tpl := &ateapipb.ActorTemplate{
- WorkerSelector: &ateapipb.Selector{MatchLabels: map[string]string{"workload": "agent"}},
- Containers: []*ateapipb.Container{{
- Name: "c", Env: []*ateapipb.EnvVar{{Name: "TOKEN", Value: "t0p"}},
- }},
- }
- redactedTpl, _ := protoredact.Redacted(tpl)
- gotTpl := redactedTpl.(*ateapipb.ActorTemplate)
- if gotTpl.GetWorkerSelector().GetMatchLabels()["workload"] != "agent" {
- t.Fatal("string map was altered")
- }
- if gotTpl.GetContainers()[0].GetEnv()[0].GetValue() != protoredact.Placeholder {
- t.Fatal("EnvVar.value not masked")
- }
}
// redactTestSchema builds, at test time, a schema that exercises every branch
@@ -379,12 +360,14 @@ func TestNeedsRedactionIsSafeUnderConcurrentFirstUse(t *testing.T) {
// the .proto files on disk.
var ourProtoFiles = []protoreflect.FileDescriptor{
ateapipb.File_ateapi_proto,
+ ateenvv1alpha.File_guest_proto,
ateletpb.File_atelet_proto,
ateworkerpb.File_ateworker_proto,
ateconfigpb.File_apiconfig_proto,
credproviderpb.File_credprovider_proto,
glutton.File_glutton_proto,
grpcechopb.File_grpcecho_proto,
+ objectstoreexchangepb.File_exchange_proto,
objectstorev1.File_objectstore_proto,
}
@@ -417,16 +400,17 @@ func isDebugRedact(fd protoreflect.FieldDescriptor) bool {
// change is reviewed as a deliberate decision about what the logs may show.
func TestDebugRedactFieldsArePinned(t *testing.T) {
want := map[string]bool{
- "ateapi.EnvVar.value": true,
- "ateapi.MintActorJwtResponse.actor_jwt": true,
- "ateapi.MintWorkerActorJwtResponse.actor_jwt": true,
- "atelet.AssetFile.url": true,
- "atelet.EnvVar.value": true,
- "ateworker.ContainerSpec.env": true,
- "credprovider.FetchSecretResponse.opaque_bytes": true,
- "objectstore.v1.FetchAssetRequest.asset_uri": true,
- "objectstore.v1.FetchSnapshotRequest.actor_jwt": true,
- "objectstore.v1.UploadSnapshotRequest.actor_jwt": true,
+ "ateapi.EnvVar.value": true,
+ "ateapi.MintActorJwtResponse.actor_jwt": true,
+ "ateapi.MintWorkerActorJwtResponse.actor_jwt": true,
+ "atelet.EnvVar.value": true,
+ "ateworker.ContainerSpec.env": true,
+ "credprovider.FetchSecretResponse.opaque_bytes": true,
+ "objectstore.v1.FetchAssetRequest.asset_uri": true,
+ "objectstore.v1.FetchSnapshotRequest.actor_jwt": true,
+ "objectstore.v1.UploadSnapshotRequest.actor_jwt": true,
+ "objectstoreexchange.ExchangeActorJwtRequest.actor_jwt": true,
+ "objectstoreexchange.ExchangeActorJwtResponse.access_token": true,
}
got := map[string]bool{}
forEachField(func(fd protoreflect.FieldDescriptor) {
diff --git a/internal/resources/actorstate.go b/internal/resources/actorstate.go
new file mode 100644
index 0000000000..e2c0e76b4a
--- /dev/null
+++ b/internal/resources/actorstate.go
@@ -0,0 +1,35 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package resources
+
+import "go.ate.dev/substrate/pkg/proto/ateapipb"
+
+// TransferStates are the states in which a node may be moving the actor's
+// snapshot between itself and object storage:
+// - RESUMING: a resume fetches the snapshot.
+// - RUNNING: the actor keeps the credentials its resume minted.
+// - SUSPENDING: a suspend uploads the snapshot.
+// - SUSPENDED: a paused actor's background upload runs.
+//
+// In every other state no node acts for the actor, so no node may hold a
+// credential for its snapshots. The mint's host check and the objectstore
+// plugin's credential exchange both refuse the other states, and read the
+// list here so they cannot disagree.
+var TransferStates = []ateapipb.ActorState{
+ ateapipb.ActorState_ACTOR_STATE_RESUMING,
+ ateapipb.ActorState_ACTOR_STATE_RUNNING,
+ ateapipb.ActorState_ACTOR_STATE_SUSPENDING,
+ ateapipb.ActorState_ACTOR_STATE_SUSPENDED,
+}
diff --git a/internal/resources/actorstate_test.go b/internal/resources/actorstate_test.go
new file mode 100644
index 0000000000..f96fa385f9
--- /dev/null
+++ b/internal/resources/actorstate_test.go
@@ -0,0 +1,49 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package resources
+
+import (
+ "slices"
+ "testing"
+
+ "go.ate.dev/substrate/pkg/proto/ateapipb"
+)
+
+// TestTransferStatesClassifyEveryState fails when a state is added to the
+// enum without deciding whether a node transfers snapshots in it, since the
+// mint's host check and the credential exchange both refuse unlisted states.
+func TestTransferStatesClassifyEveryState(t *testing.T) {
+ want := map[ateapipb.ActorState]bool{
+ ateapipb.ActorState_ACTOR_STATE_UNSPECIFIED: false,
+ ateapipb.ActorState_ACTOR_STATE_RESUMING: true,
+ ateapipb.ActorState_ACTOR_STATE_RUNNING: true,
+ ateapipb.ActorState_ACTOR_STATE_SUSPENDING: true,
+ ateapipb.ActorState_ACTOR_STATE_SUSPENDED: true,
+ ateapipb.ActorState_ACTOR_STATE_CRASHED: false,
+ ateapipb.ActorState_ACTOR_STATE_REVERTING: false,
+ ateapipb.ActorState_ACTOR_STATE_DELETING: false,
+ }
+ for value, name := range ateapipb.ActorState_name {
+ state := ateapipb.ActorState(value)
+ transfers, classified := want[state]
+ if !classified {
+ t.Errorf("%s is not classified: decide whether a node transfers snapshots in it and add it to this test", name)
+ continue
+ }
+ if got := slices.Contains(TransferStates, state); got != transfers {
+ t.Errorf("TransferStates contains %s = %v, want %v", name, got, transfers)
+ }
+ }
+}
diff --git a/internal/resources/validate.go b/internal/resources/validate.go
index 3298cdd8b2..b1f7ebf760 100644
--- a/internal/resources/validate.go
+++ b/internal/resources/validate.go
@@ -15,7 +15,6 @@
package resources
import (
- "encoding/hex"
"fmt"
"net"
"net/netip"
@@ -210,25 +209,12 @@ func ValidateSandboxAssetsDir(actorUID, dir string, fldPath *field.Path) field.E
return nil
}
-// ValidateAssetHash ensures a sandbox asset's SHA-256 hash is exactly 64 hex
-// characters before it becomes the asset's cache key, so a malformed hash is
-// rejected as the request's fault rather than failing deep inside the cache.
-func ValidateAssetHash(sha256Hash string) error {
- if len(sha256Hash) != 64 {
- return fmt.Errorf("invalid asset sha256 hash: want 64 hex chars, got %d", len(sha256Hash))
- }
- // Same decoder atelet's digest comparison uses.
- if _, err := hex.DecodeString(sha256Hash); err != nil {
- return fmt.Errorf("invalid asset sha256 hash %q: must be hex", sha256Hash)
- }
- return nil
-}
-
-// ValidateSnapshotLocation ensures an ActorTemplate's snapshotConfig.location
-// is a well-formed URI with a bucket, so a bad location fails fast instead of
-// deep inside an object-storage call. It deliberately does not restrict the
-// scheme: the storage layer only uses the host (bucket) and path, and which
-// schemes are acceptable is a storage-backend policy, not a per-RPC one.
+// ValidateSnapshotLocation ensures an ActorTemplate's
+// snapshotConfig.storageLocation is a well-formed URI with a bucket, so a bad
+// location fails fast instead of deep inside an object-storage call. It
+// deliberately does not restrict the scheme: the storage layer only uses the
+// host (bucket) and path, and which schemes are acceptable is a
+// storage-backend policy, not a per-RPC one.
//
// This validates the base that many snapshots share, not any one snapshot's
// URI; SnapshotURI is the type for the latter, and it applies this check when
diff --git a/internal/resources/validate_test.go b/internal/resources/validate_test.go
index 676106d8e0..f9aa293f5b 100644
--- a/internal/resources/validate_test.go
+++ b/internal/resources/validate_test.go
@@ -145,31 +145,6 @@ func TestValidateWorkerPodUID(t *testing.T) {
}
}
-func TestValidateAssetHash(t *testing.T) {
- const valid = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
-
- tests := []struct {
- name string
- hash string
- wantErr bool
- }{
- {"valid lowercase", valid, false},
- {"valid uppercase", strings.ToUpper(valid), false},
- {"empty", "", true},
- {"too short", "abc123", true},
- {"too long", valid + "00", true},
- {"separator", strings.Repeat("a", 60) + "/../", true},
- {"non-hex", strings.Repeat("g", 64), true},
- }
- for _, tt := range tests {
- t.Run(tt.name, func(t *testing.T) {
- if err := ValidateAssetHash(tt.hash); (err != nil) != tt.wantErr {
- t.Errorf("ValidateAssetHash(%q) err = %v, wantErr %v", tt.hash, err, tt.wantErr)
- }
- })
- }
-}
-
func TestValidateSnapshotLocation(t *testing.T) {
tests := []struct {
name string
diff --git a/internal/serverboot/logging.go b/internal/serverboot/logging.go
index f9a7f9af57..8dc334ec2e 100644
--- a/internal/serverboot/logging.go
+++ b/internal/serverboot/logging.go
@@ -58,9 +58,11 @@ type LoggingOptions struct {
// through internal/actorevent, and sets whether those records also go to stdout
// (console).
//
-// It returns (nil, nil) when the exporter does not include otlp, and leaves the
-// OTel global alone: emitting then costs one Enabled check. Guard the shutdown
-// on a nil provider, unlike InitTracing which always returns one.
+// It returns (nil, nil) when the exporter does not include otlp, or when a
+// relay-capable component has neither a relay connection nor an OTLP endpoint
+// (see hasOTLPDestination), and leaves the OTel global alone: emitting then
+// costs one Enabled check. Guard the shutdown on a nil provider, unlike
+// InitTracing which always returns one.
//
// The processor batches. These records sit on the actor resume and suspend
// path, and exporting inside the emit call would put a blocking gRPC round trip
@@ -76,8 +78,11 @@ func InitLogging(ctx context.Context, opts LoggingOptions) (*sdklog.LoggerProvid
return nil, fmt.Errorf("LoggingOptions.Exporter is required")
}
actorevent.SetConsole(opts.Exporter.Has(ExporterConsole))
- if !opts.Exporter.Has(ExporterOTLP) {
- slog.InfoContext(ctx, "OTLP logs export disabled", slog.String("exporter", opts.Exporter.String()))
+ if reason, level := otlpExportBlocked(opts.Exporter.Has(ExporterOTLP),
+ hasOTLPDestination(logsEndpointEnv, opts.RelayCapable, opts.ExporterConn)); reason != "" {
+ slog.Log(ctx, level, "OTLP logs export disabled",
+ slog.String("exporter", opts.Exporter.String()),
+ slog.String("reason", reason))
return nil, nil
}
diff --git a/internal/serverboot/serverboot.go b/internal/serverboot/serverboot.go
index f261760e4d..a32f343854 100644
--- a/internal/serverboot/serverboot.go
+++ b/internal/serverboot/serverboot.go
@@ -28,6 +28,7 @@ import (
"net/http"
"os"
"slices"
+ "strings"
"sync"
"sync/atomic"
"time"
@@ -118,7 +119,8 @@ func newResource(ctx context.Context, serviceName string, extraAttrs ...attribut
//
// The distinction matters because a nil conn on a relay-capable component is
// exactly the degraded case worth alerting on: the worker asked for the relay,
-// could not dial it, and is now exporting over the worker pod's own network.
+// could not dial it, and is now exporting over the worker pod's own network,
+// or not at all without an OTLP endpoint (see hasOTLPDestination).
// Collapsing that into the same "no attribute" bucket as atecontroller would
// hide it.
func relayAttrs(relayCapable bool, conn *grpc.ClientConn) []attribute.KeyValue {
@@ -183,7 +185,9 @@ func InitTracing(ctx context.Context, opts TracingOptions) (*sdktrace.TracerProv
// Without an exporter the provider still samples and propagates trace
// context, so a component that exports nothing keeps the traces of the
// components around it whole.
- export := tracesPushEnabled(ctx)
+ reason, level := otlpExportBlocked(tracesPushEnabled(ctx),
+ hasOTLPDestination(tracesEndpointEnv, opts.RelayCapable, opts.ExporterConn))
+ export := reason == ""
if export {
expOpts := []otlptracegrpc.Option{
// GKE managed traces doesn't support validating the TLS certs of the collector.
@@ -204,17 +208,58 @@ func InitTracing(ctx context.Context, opts TracingOptions) (*sdktrace.TracerProv
tp := sdktrace.NewTracerProvider(tpOpts...)
otel.SetTracerProvider(tp)
otel.SetTextMapPropagator(propagation.TraceContext{})
- slog.InfoContext(ctx, "Tracing initialized",
+ attrs := []any{
slog.String("sampler", opts.Sampling.Sampler().Description()),
- slog.Bool("export", export))
+ slog.Bool("export", export),
+ }
+ if !export {
+ attrs = append(attrs, slog.String("reason", reason))
+ }
+ slog.Log(ctx, level, "Tracing initialized", attrs...)
return tp, nil
}
const (
metricsExporterEnv = "OTEL_METRICS_EXPORTER"
tracesExporterEnv = "OTEL_TRACES_EXPORTER"
+
+ otlpEndpointEnv = "OTEL_EXPORTER_OTLP_ENDPOINT"
+ tracesEndpointEnv = "OTEL_EXPORTER_OTLP_TRACES_ENDPOINT"
+ metricsEndpointEnv = "OTEL_EXPORTER_OTLP_METRICS_ENDPOINT"
+ logsEndpointEnv = "OTEL_EXPORTER_OTLP_LOGS_ENDPOINT"
+)
+
+const (
+ reasonOTLPNotSelected = "otlp exporter not selected"
+ reasonNoDestination = "no OTLP relay or endpoint"
)
+// otlpExportBlocked returns why a signal does not export over OTLP, empty when
+// it does, and the level to log that at: Warn when otlp is selected but has
+// nowhere to go, since the component then loses that signal.
+func otlpExportBlocked(selected, hasDestination bool) (string, slog.Level) {
+ switch {
+ case !selected:
+ return reasonOTLPNotSelected, slog.LevelInfo
+ case !hasDestination:
+ return reasonNoDestination, slog.LevelWarn
+ default:
+ return "", slog.LevelInfo
+ }
+}
+
+// hasOTLPDestination reports whether a component has a collector to export
+// to. A relay-capable component (ateworker) without a relay connection
+// needs an OTLP endpoint, the generic one or signalEndpointEnv: a worker pod
+// runs no collector beside it, so the SDK's localhost:4317 default reaches
+// nothing. Every other component keeps that default.
+func hasOTLPDestination(signalEndpointEnv string, relayCapable bool, conn *grpc.ClientConn) bool {
+ if !relayCapable || conn != nil {
+ return true
+ }
+ return strings.TrimSpace(os.Getenv(otlpEndpointEnv)) != "" || strings.TrimSpace(os.Getenv(signalEndpointEnv)) != ""
+}
+
// metricsExporters applies OTEL_METRICS_EXPORTER with the shared exporter
// rules. prometheus, the specification's pull exporter, is known only when
// pullServed, i.e. the component serves its own /metrics; elsewhere it is
@@ -380,7 +425,9 @@ func newMeterProvider(ctx context.Context, serviceName string, push, relayCapabl
return nil, fmt.Errorf("create metric resource: %w", err)
}
opts := []sdkmetric.Option{sdkmetric.WithResource(res), sdkmetric.WithView(dropRPCServerAddress)}
- if push {
+ reason, level := otlpExportBlocked(push, hasOTLPDestination(metricsEndpointEnv, relayCapable, conn))
+ export := reason == ""
+ if export {
expOpts := []otlpmetricgrpc.Option{
// GKE managed metrics doesn't support validating the TLS certs of the collector.
otlpmetricgrpc.WithInsecure(),
@@ -405,6 +452,11 @@ func newMeterProvider(ctx context.Context, serviceName string, push, relayCapabl
}
mp := sdkmetric.NewMeterProvider(opts...)
otel.SetMeterProvider(mp)
+ attrs := []any{slog.Bool("otlp_export", export)}
+ if !export {
+ attrs = append(attrs, slog.String("reason", reason))
+ }
+ slog.Log(ctx, level, "Metrics initialized", attrs...)
return mp, nil
}
diff --git a/internal/serverboot/serverboot_test.go b/internal/serverboot/serverboot_test.go
index 33f3e48509..180652df8f 100644
--- a/internal/serverboot/serverboot_test.go
+++ b/internal/serverboot/serverboot_test.go
@@ -18,6 +18,7 @@ import (
"bytes"
"context"
"errors"
+ "log"
"log/slog"
"net"
"net/http"
@@ -406,12 +407,12 @@ func TestHTTPServerBoundsHeaderRead(t *testing.T) {
}
func TestInitMetricsPushOnlyHasNoPrometheusSurface(t *testing.T) {
- mp, err := InitMetricsPushOnlyVia(context.Background(), "test-pushonly", nil)
+ mp, err := InitMetricsPushOnlyVia(context.Background(), "test-pushonly", lazyConn(t))
if err != nil {
t.Fatalf("InitMetricsPushOnlyVia: %v", err)
}
// Bound shutdown: the periodic reader would otherwise block flushing to the
- // unreachable default OTLP endpoint until the export timeout.
+ // unreachable connection until the export timeout.
t.Cleanup(func() {
ctx, cancel := context.WithTimeout(context.Background(), time.Second)
defer cancel()
@@ -475,6 +476,16 @@ func (c *metricsCollector) flush(t *testing.T, mp *sdkmetric.MeterProvider) map[
// startMetricsCollector serves a metricsCollector on a loopback port, points
// OTEL_EXPORTER_OTLP_ENDPOINT at it, and sets OTEL_METRICS_EXPORTER.
func startMetricsCollector(t *testing.T, exporter string) *metricsCollector {
+ t.Helper()
+ collector, addr := serveMetricsCollector(t)
+ t.Setenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://"+addr)
+ t.Setenv(metricsExporterEnv, exporter)
+ return collector
+}
+
+// serveMetricsCollector serves a metricsCollector on a loopback port and
+// returns it with its address.
+func serveMetricsCollector(t *testing.T) (*metricsCollector, string) {
t.Helper()
ln, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
@@ -485,9 +496,7 @@ func startMetricsCollector(t *testing.T, exporter string) *metricsCollector {
colmetricspb.RegisterMetricsServiceServer(srv, collector)
go func() { _ = srv.Serve(ln) }()
t.Cleanup(srv.Stop)
- t.Setenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://"+ln.Addr().String())
- t.Setenv(metricsExporterEnv, exporter)
- return collector
+ return collector, ln.Addr().String()
}
func addOne(t *testing.T, mp *sdkmetric.MeterProvider, name string) {
@@ -925,8 +934,9 @@ func (c *traceCollector) Export(_ context.Context, req *coltracepb.ExportTraceSe
}
// startTraceCollector serves a traceCollector on a loopback port, points
-// OTEL_EXPORTER_OTLP_ENDPOINT at it, and sets OTEL_TRACES_EXPORTER.
-func startTraceCollector(t *testing.T, exporter string) *traceCollector {
+// OTEL_EXPORTER_OTLP_ENDPOINT at it, and sets OTEL_TRACES_EXPORTER. It returns
+// the collector and its address.
+func startTraceCollector(t *testing.T, exporter string) (*traceCollector, string) {
t.Helper()
ln, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
@@ -939,14 +949,16 @@ func startTraceCollector(t *testing.T, exporter string) *traceCollector {
t.Cleanup(srv.Stop)
t.Setenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://"+ln.Addr().String())
t.Setenv(tracesExporterEnv, exporter)
- return collector
+ return collector, ln.Addr().String()
}
// endOneSpan ends a sampled span, then flushes and shuts the provider down, so
-// a span exporter, if there is one, has exported it.
-func endOneSpan(t *testing.T, serviceName string) {
+// a span exporter, if there is one, has exported it. opts.Sampling is set to
+// sample every span.
+func endOneSpan(t *testing.T, opts TracingOptions) {
t.Helper()
- tp, err := InitTracing(t.Context(), TracingOptions{ServiceName: serviceName, Sampling: ParentRatioSampling(1)})
+ opts.Sampling = ParentRatioSampling(1)
+ tp, err := InitTracing(t.Context(), opts)
if err != nil {
t.Fatalf("InitTracing: %v", err)
}
@@ -962,8 +974,8 @@ func endOneSpan(t *testing.T, serviceName string) {
}
func TestInitTracingExportsOverOTLPByDefault(t *testing.T) {
- collector := startTraceCollector(t, "")
- endOneSpan(t, "test-traces-default")
+ collector, _ := startTraceCollector(t, "")
+ endOneSpan(t, TracingOptions{ServiceName: "test-traces-default"})
if got := collector.spans.Load(); got != 1 {
t.Errorf("the collector received %d span(s), want 1", got)
}
@@ -972,9 +984,135 @@ func TestInitTracingExportsOverOTLPByDefault(t *testing.T) {
// With OTEL_TRACES_EXPORTER=none the span is still sampled, so the context a
// component propagates keeps its sampled flag, and nothing is exported.
func TestInitTracingExporterNoneExportsNothing(t *testing.T) {
- collector := startTraceCollector(t, "none")
- endOneSpan(t, "test-traces-none")
+ collector, _ := startTraceCollector(t, "none")
+ endOneSpan(t, TracingOptions{ServiceName: "test-traces-none"})
if got := collector.spans.Load(); got != 0 {
t.Errorf("OTEL_TRACES_EXPORTER=none still exported %d span(s)", got)
}
}
+
+// ateworker exports through the relay without an OTLP endpoint of its own: the
+// connection is the destination.
+func TestInitTracingRelayCapableExportsThroughConnWithoutEndpoint(t *testing.T) {
+ collector, addr := startTraceCollector(t, "")
+ t.Setenv(otlpEndpointEnv, "")
+ conn, err := grpc.NewClient("passthrough:///"+addr, grpc.WithTransportCredentials(insecure.NewCredentials()))
+ if err != nil {
+ t.Fatalf("grpc.NewClient: %v", err)
+ }
+ t.Cleanup(func() { conn.Close() })
+ endOneSpan(t, TracingOptions{ServiceName: "test-traces-relay", ExporterConn: conn, RelayCapable: true})
+ if got := collector.spans.Load(); got != 1 {
+ t.Errorf("the collector received %d span(s) through the connection, want 1", got)
+ }
+}
+
+func TestHasOTLPDestination(t *testing.T) {
+ conn := lazyConn(t)
+ tests := []struct {
+ name string
+ relayCapable bool
+ conn *grpc.ClientConn
+ generic, signal string
+ want bool
+ }{
+ {name: "not relay capable", want: true},
+ {name: "relay connection", relayCapable: true, conn: conn, want: true},
+ {name: "generic endpoint", relayCapable: true, generic: "http://collector:4317", want: true},
+ {name: "signal endpoint", relayCapable: true, signal: "http://collector:4317", want: true},
+ {name: "neither", relayCapable: true, want: false},
+ {name: "blank endpoints", relayCapable: true, generic: " ", signal: " ", want: false},
+ }
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ t.Setenv(otlpEndpointEnv, tt.generic)
+ t.Setenv(tracesEndpointEnv, tt.signal)
+ if got := hasOTLPDestination(tracesEndpointEnv, tt.relayCapable, tt.conn); got != tt.want {
+ t.Errorf("hasOTLPDestination = %t, want %t", got, tt.want)
+ }
+ })
+ }
+}
+
+// Without a relay or an endpoint, InitLogging returns no provider for ateworker
+// rather than one that dials localhost:4317.
+func TestInitLoggingRelayCapableWithoutDestinationReturnsNil(t *testing.T) {
+ t.Setenv(otlpEndpointEnv, "")
+ t.Setenv(logsEndpointEnv, "")
+ lp, err := InitLogging(t.Context(), LoggingOptions{ServiceName: "test-logs-nowhere", Exporter: Exporters{ExporterOTLP: true}, RelayCapable: true})
+ if err != nil {
+ t.Fatalf("InitLogging: %v", err)
+ }
+ if lp != nil {
+ t.Error("InitLogging returned a provider with nowhere to export")
+ }
+}
+
+// captureLogs sends slog's default logger to the returned buffer for the rest
+// of the test.
+func captureLogs(t *testing.T) *bytes.Buffer {
+ t.Helper()
+ // slog.SetDefault also points the log package at the handler, and setting
+ // slog back does not undo that, so restore all three.
+ prev, prevOut, prevFlags := slog.Default(), log.Writer(), log.Flags()
+ t.Cleanup(func() {
+ slog.SetDefault(prev)
+ log.SetOutput(prevOut)
+ log.SetFlags(prevFlags)
+ })
+ var buf bytes.Buffer
+ slog.SetDefault(slog.New(slog.NewTextHandler(&buf, nil)))
+ return &buf
+}
+
+// Without a relay or an endpoint, ateworker installs no span exporter rather than
+// one that dials localhost:4317, and still samples.
+func TestInitTracingRelayCapableWithoutDestinationExportsNothing(t *testing.T) {
+ t.Setenv(otlpEndpointEnv, "")
+ t.Setenv(tracesEndpointEnv, "")
+ t.Setenv(tracesExporterEnv, "")
+ logs := captureLogs(t)
+ endOneSpan(t, TracingOptions{ServiceName: "test-traces-nowhere", RelayCapable: true})
+ if !strings.Contains(logs.String(), `level=WARN msg="Tracing initialized"`) || !strings.Contains(logs.String(), `export=false reason="no OTLP relay or endpoint"`) {
+ t.Errorf("InitTracing installed a span exporter with nowhere to export:\n%s", logs.String())
+ }
+}
+
+// Without a relay or an endpoint, ateworker installs no OTLP metric reader rather
+// than one that dials localhost:4317.
+func TestInitMetricsPushOnlyWithoutDestinationExportsNothing(t *testing.T) {
+ t.Setenv(otlpEndpointEnv, "")
+ t.Setenv(metricsEndpointEnv, "")
+ t.Setenv(metricsExporterEnv, "")
+ logs := captureLogs(t)
+ mp, err := InitMetricsPushOnlyVia(t.Context(), "test-metrics-nowhere", nil)
+ if err != nil {
+ t.Fatalf("InitMetricsPushOnlyVia: %v", err)
+ }
+ t.Cleanup(func() { _ = mp.Shutdown(context.Background()) })
+ if !strings.Contains(logs.String(), `level=WARN msg="Metrics initialized" otlp_export=false reason="no OTLP relay or endpoint"`) {
+ t.Errorf("InitMetricsPushOnlyVia installed an OTLP reader with nowhere to export:\n%s", logs.String())
+ }
+}
+
+// ateworker exports metrics through the relay without an OTLP endpoint of its own:
+// the connection is the destination.
+func TestInitMetricsPushOnlyExportsThroughConnWithoutEndpoint(t *testing.T) {
+ collector, addr := serveMetricsCollector(t)
+ t.Setenv(otlpEndpointEnv, "")
+ t.Setenv(metricsEndpointEnv, "")
+ t.Setenv(metricsExporterEnv, "")
+ conn, err := grpc.NewClient("passthrough:///"+addr, grpc.WithTransportCredentials(insecure.NewCredentials()))
+ if err != nil {
+ t.Fatalf("grpc.NewClient: %v", err)
+ }
+ t.Cleanup(func() { conn.Close() })
+ mp, err := InitMetricsPushOnlyVia(t.Context(), "test-metrics-relay", conn)
+ if err != nil {
+ t.Fatalf("InitMetricsPushOnlyVia: %v", err)
+ }
+ addOne(t, mp, "ate.test.relay.count")
+ if got := collector.flush(t, mp); got["ate.test.relay.count"] != 1 {
+ t.Errorf("the collector received %v through the connection, want ate.test.relay.count once", got)
+ }
+}
diff --git a/manifests/ate-install/ate-api-server-envvars.yaml b/manifests/ate-install/ate-api-server-envvars.yaml
index f733d225df..9db2735b0b 100644
--- a/manifests/ate-install/ate-api-server-envvars.yaml
+++ b/manifests/ate-install/ate-api-server-envvars.yaml
@@ -21,5 +21,6 @@ metadata:
name: ate-api-server-envvars
namespace: ate-system
data:
+ ATE_API_SNAPSHOT_LOCATION: "gs://ate-snapshots/"
ATE_API_POSTGRES_SCHEMA: "substrate"
ATE_API_ACTOR_JWT_ISSUER: "https://idp.ate-system.svc"
diff --git a/manifests/ate-install/ate-api-server.yaml b/manifests/ate-install/ate-api-server.yaml
index b3fcaab851..11b4e6bcbd 100644
--- a/manifests/ate-install/ate-api-server.yaml
+++ b/manifests/ate-install/ate-api-server.yaml
@@ -23,10 +23,7 @@ rules:
resources: ["pods"]
verbs: ["get", "watch", "list"]
- apiGroups: ["ate.dev"]
- resources: ["workerpools", "sandboxconfigs", "csidriverconfigs"]
- verbs: ["get", "watch", "list"]
-- apiGroups: ["ate.dev"]
- resources: ["workerpools"]
+ resources: ["workerpools", "csidriverconfigs"]
verbs: ["get", "watch", "list"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
@@ -94,6 +91,19 @@ spec:
args:
- control
- --socket=/run/objectstore-plugin/control.sock
+ # The credential exchange. On GCS, node plugins trade an actor JWT for
+ # a token scoped to that actor's snapshots. On S3 it answers
+ # UNIMPLEMENTED and the node plugins never call it. It serves over TLS
+ # on its own port with the pod's service DNS certificate, verifies
+ # JWTs with the public keys mounted below, and reaches ate-api-server
+ # over the loopback with the pod identity certificate. The issuer must
+ # be the one ate-api-server mints with, so both read it from the same
+ # ConfigMap key.
+ - --exchange-listen=:8445
+ - --actor-jwt-issuer=$(ATE_API_ACTOR_JWT_ISSUER)
+ ports:
+ - name: exchange
+ containerPort: 8445
# The plugin listens on a Unix socket, which a gRPC probe cannot
# reach, and the image has no shell, so the binary checks itself.
startupProbe:
@@ -118,12 +128,36 @@ spec:
# set on it.
- name: ATE_STORAGE_BACKEND
value: "gcs"
+ - name: ATE_API_ACTOR_JWT_ISSUER
+ valueFrom:
+ configMapKeyRef:
+ name: ate-api-server-envvars
+ key: ATE_API_ACTOR_JWT_ISSUER
+ # The exchange verifies ate-api-server's certificate against the API
+ # Service name in this namespace.
+ - name: POD_NAMESPACE
+ valueFrom:
+ fieldRef:
+ fieldPath: metadata.namespace
envFrom:
- configMapRef:
name: ate-otel-config
volumeMounts:
- name: objectstore-plugin
mountPath: /run/objectstore-plugin
+ # For the exchange: its serving certificate, the client certificate it
+ # dials ate-api-server with, and the public keys of the actor JWT
+ # pool. Only the jwks item is projected, never the pool itself, so the
+ # sidecar cannot mint JWTs.
+ - name: "servicedns"
+ mountPath: "/run/servicedns.podcert.ate.dev"
+ readOnly: true
+ - name: "podidentity"
+ mountPath: "/run/podidentity.podcert.ate.dev"
+ readOnly: true
+ - name: "actor-id-jwt-jwks"
+ mountPath: "/run/actor-id-jwt-jwks"
+ readOnly: true
containers:
- name: ate-api-server
image: ko://go.ate.dev/substrate/cmd/ateapi
@@ -151,6 +185,7 @@ spec:
- --drain-delay=13s
- --drain-timeout=15s
- --objectstore-plugin-socket=/run/objectstore-plugin/control.sock
+ - --snapshot-location=@env
env:
- name: POD_NAME
valueFrom:
@@ -311,6 +346,17 @@ spec:
items:
- key: "pool"
path: "pool.json"
+ # The same Secret's public keys alone, for the objectstore-plugin
+ # sidecar's JWT verification. ate-setup and kubectl ate admin keep the
+ # jwks key in step with pool.
+ - name: "actor-id-jwt-jwks"
+ projected:
+ sources:
+ - secret:
+ name: "actor-id-jwt-pool"
+ items:
+ - key: "jwks"
+ path: "jwks.json"
- name: "actor-id-ca-pool"
projected:
sources:
@@ -362,3 +408,9 @@ spec:
protocol: TCP
port: 443
targetPort: 443
+ # The objectstore-plugin sidecar's credential exchange, called by the node
+ # plugins beside atelet.
+ - name: exchange
+ protocol: TCP
+ port: 8445
+ targetPort: 8445
diff --git a/manifests/ate-install/ate-controller.yaml b/manifests/ate-install/ate-controller.yaml
index c1464e01a8..f5aeaba137 100644
--- a/manifests/ate-install/ate-controller.yaml
+++ b/manifests/ate-install/ate-controller.yaml
@@ -123,7 +123,8 @@ spec:
value: k8s.namespace.name=$(POD_NAMESPACE),k8s.pod.name=$(POD_NAME),k8s.pod.uid=$(POD_UID),service.instance.id=$(POD_UID)
# OTLP exporter settings are shared by every control plane component;
# see manifests/ate-install/ate-otel-config.yaml. The controller also
- # propagates these to the worker pods it creates.
+ # propagates these to the worker pods it creates, except the exporter
+ # selection, which it takes from the ATE_WORKER_OTEL_*_EXPORTER keys.
envFrom:
- configMapRef:
name: ate-otel-config
diff --git a/manifests/ate-install/ate-otel-config.yaml b/manifests/ate-install/ate-otel-config.yaml
index 0c65ca7ea2..0ee44e5053 100644
--- a/manifests/ate-install/ate-otel-config.yaml
+++ b/manifests/ate-install/ate-otel-config.yaml
@@ -17,7 +17,10 @@
# atenet-egress), which consume it via `envFrom`. ate-controller additionally propagates these values
# to the worker pods it creates, which have no other way to receive them,
# and atenet-router's --otlp-collector-address defaults to the endpoint below,
-# so Envoy's own spans follow it too.
+# so Envoy's own spans follow it too. The exporter selection is the exception:
+# ateworker takes ATE_WORKER_OTEL_LOGS_EXPORTER, ATE_WORKER_OTEL_TRACES_EXPORTER
+# and ATE_WORKER_OTEL_METRICS_EXPORTER, not the control plane's
+# OTEL_*_EXPORTER values.
#
# This file holds the GKE values. The kind overlay supplies a ConfigMap of the
# same name with the in-cluster collector address; see
diff --git a/manifests/ate-install/ate-system-networkpolicies.yaml b/manifests/ate-install/ate-system-networkpolicies.yaml
index 159f4fd947..d0bd200ea1 100644
--- a/manifests/ate-install/ate-system-networkpolicies.yaml
+++ b/manifests/ate-install/ate-system-networkpolicies.yaml
@@ -80,6 +80,15 @@ spec:
ports:
- protocol: TCP
port: 443
+ # The objectstore-plugin sidecar's credential exchange is for the node
+ # plugins beside atelet and nothing else.
+ - from:
+ - podSelector:
+ matchLabels:
+ app: atelet
+ ports:
+ - protocol: TCP
+ port: 8445
---
# The bundled postgres is reached only by ate-api-server. The installer runs
# psql through `kubectl exec`, and the probes are exec probes, so neither needs
diff --git a/manifests/ate-install/atelet.yaml b/manifests/ate-install/atelet.yaml
index e10e601042..4350367758 100644
--- a/manifests/ate-install/atelet.yaml
+++ b/manifests/ate-install/atelet.yaml
@@ -282,7 +282,6 @@ spec:
ports:
- name: grpc
containerPort: 21998
- hostPort: 21998
- name: prometheus
containerPort: 9090
protocol: TCP
diff --git a/manifests/ate-install/components/agentgateway/kustomization.yaml b/manifests/ate-install/components/agentgateway/kustomization.yaml
index 646d21958e..824096d643 100644
--- a/manifests/ate-install/components/agentgateway/kustomization.yaml
+++ b/manifests/ate-install/components/agentgateway/kustomization.yaml
@@ -42,7 +42,7 @@ patches:
path: /spec/template/spec/containers/0
value:
name: agentgateway
- image: ghcr.io/kagent-dev/substrate/agentgateway:3554db38a0b2@sha256:309f6811560635471958db67b0f68dd083d84048dffc681967599d159ebfc83d
+ image: ghcr.io/kagent-dev/substrate/agentgateway:b736d06f88eb@sha256:39898817ed3e94b78570d20862fec06f32142fe8dee8691070128e74b5290407
args:
- -f
- /etc/agentgateway/config.yaml
@@ -118,7 +118,7 @@ patches:
path: /spec/template/spec/containers/0
value:
name: agentgateway
- image: ghcr.io/kagent-dev/substrate/agentgateway:3554db38a0b2@sha256:309f6811560635471958db67b0f68dd083d84048dffc681967599d159ebfc83d
+ image: ghcr.io/kagent-dev/substrate/agentgateway:b736d06f88eb@sha256:39898817ed3e94b78570d20862fec06f32142fe8dee8691070128e74b5290407
args:
- -f
- /etc/agentgateway/config.yaml
diff --git a/manifests/ate-install/components/fake-data-plane/kustomization.yaml b/manifests/ate-install/components/fake-data-plane/kustomization.yaml
index 9039c816c1..dce35f26f9 100644
--- a/manifests/ate-install/components/fake-data-plane/kustomization.yaml
+++ b/manifests/ate-install/components/fake-data-plane/kustomization.yaml
@@ -130,6 +130,14 @@ patches:
- op: add
path: /spec/ingress/0/from/0/podSelector/matchExpressions/0/values/-
value: fake-workersync
+ # Grant fake-workersync the controller authz role so it can manage Workers.
+ - target:
+ kind: Deployment
+ name: ate-api-server
+ patch: |-
+ - op: add
+ path: /spec/template/spec/containers/0/args/-
+ value: --controller-spiffe-id=spiffe://cluster.local/ns/ate-system/sa/fake-workersync
# fake-workersync stands in for ate-controller, which must not run beside
# it: its WorkerPool controller would make real worker pods for the same
# pools, and its worker syncer deletes every Worker with no live pod. Its
diff --git a/manifests/ate-install/kind/ate-otel-config.yaml b/manifests/ate-install/kind/ate-otel-config.yaml
index ce58ab76b6..78ca799ab1 100644
--- a/manifests/ate-install/kind/ate-otel-config.yaml
+++ b/manifests/ate-install/kind/ate-otel-config.yaml
@@ -30,10 +30,12 @@
# the worker pods it creates, which have no other way to receive them.
# Kind only; production keeps the SDK defaults.
#
-# OTEL_LOGS_EXPORTER=otlp sends the actor events, ateapi's lifecycle records
-# and the ateworkers' usage samples, over OTLP instead of stdout; kind adds console
-# to keep them in kubectl logs too. serverboot defaults it to none, so the base
-# ConfigMap leaves it unset and every other environment keeps them on stdout.
+# OTEL_LOGS_EXPORTER=otlp sends the actor events and ateapi's lifecycle records
+# over OTLP instead of stdout; kind adds console to keep them in kubectl logs
+# too. serverboot defaults it to none, so the base ConfigMap leaves it unset and
+# every other environment keeps them on stdout. ATE_WORKER_OTEL_LOGS_EXPORTER is
+# the same setting for the worker pods and their usage samples, which
+# ate-controller sets from it.
apiVersion: v1
kind: ConfigMap
metadata:
@@ -44,3 +46,4 @@ data:
OTEL_METRIC_EXPORT_INTERVAL: "10000"
OTEL_METRIC_EXPORT_TIMEOUT: "10000"
OTEL_LOGS_EXPORTER: otlp,console
+ ATE_WORKER_OTEL_LOGS_EXPORTER: otlp,console
diff --git a/pkg/api/v1alpha1/workerpool_types_test.go b/pkg/api/v1alpha1/workerpool_types_test.go
new file mode 100644
index 0000000000..d51bfd2e0a
--- /dev/null
+++ b/pkg/api/v1alpha1/workerpool_types_test.go
@@ -0,0 +1,85 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package v1alpha1
+
+import "testing"
+
+func TestResolvedSandboxConfig(t *testing.T) {
+ tests := []struct {
+ name string
+ pool WorkerPool
+ want string
+ }{{
+ name: "confirmed config of the pool's class",
+ pool: WorkerPool{
+ Spec: WorkerPoolSpec{
+ SandboxClasses: []WorkerPoolSandboxClass{{Name: SandboxClassGvisor}},
+ },
+ Status: WorkerPoolStatus{
+ SandboxClasses: []WorkerPoolSandboxClassStatus{{
+ Name: SandboxClassGvisor,
+ ConfigRef: &SandboxConfigReference{Name: "gvisor-default"},
+ }},
+ },
+ },
+ want: "gvisor-default",
+ }, {
+ name: "not resolved yet",
+ pool: WorkerPool{
+ Spec: WorkerPoolSpec{
+ SandboxClasses: []WorkerPoolSandboxClass{{Name: SandboxClassGvisor}},
+ },
+ Status: WorkerPoolStatus{
+ SandboxClasses: []WorkerPoolSandboxClassStatus{{Name: SandboxClassGvisor}},
+ },
+ },
+ }, {
+ name: "only a pending config",
+ pool: WorkerPool{
+ Spec: WorkerPoolSpec{
+ SandboxClasses: []WorkerPoolSandboxClass{{Name: SandboxClassGvisor}},
+ },
+ Status: WorkerPoolStatus{
+ SandboxClasses: []WorkerPoolSandboxClassStatus{{
+ Name: SandboxClassGvisor,
+ PendingConfigRef: &SandboxConfigReference{Name: "gvisor-next"},
+ }},
+ },
+ },
+ }, {
+ name: "status resolved another class than the pool's",
+ pool: WorkerPool{
+ Spec: WorkerPoolSpec{
+ SandboxClasses: []WorkerPoolSandboxClass{{Name: SandboxClassMicroVM}},
+ },
+ Status: WorkerPoolStatus{
+ SandboxClasses: []WorkerPoolSandboxClassStatus{{
+ Name: SandboxClassGvisor,
+ ConfigRef: &SandboxConfigReference{Name: "gvisor-default"},
+ }},
+ },
+ },
+ }, {
+ name: "empty sandboxClasses in spec",
+ pool: WorkerPool{},
+ }}
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ if got := tt.pool.ResolvedSandboxConfig(); got != tt.want {
+ t.Errorf("ResolvedSandboxConfig() = %q, want %q", got, tt.want)
+ }
+ })
+ }
+}
diff --git a/pkg/objectstorage/gcs.go b/pkg/objectstorage/gcs.go
index 1f6de99360..327af424f5 100644
--- a/pkg/objectstorage/gcs.go
+++ b/pkg/objectstorage/gcs.go
@@ -21,7 +21,7 @@ import (
"errors"
"fmt"
"io"
- "sync"
+ "sync/atomic"
"time"
"cloud.google.com/go/storage"
@@ -58,19 +58,16 @@ func describeGCSError(err error) (ServiceError, bool) {
type gcsClient struct {
client *storage.Client
- // opts are how client was built, so the pool below is built the same way.
- // A pooled client that authenticates differently from the one that opened
- // an object fails partway through reading it.
- opts []option.ClientOption
- // pool holds extra clients so concurrent parts get their own connections;
- // built on first use by uploadClient.
- poolOnce sync.Once
- pool []*storage.Client
+ // pool holds extra clients so concurrent transfers get their own
+ // connections, see uploadClient. next is the round-robin position for
+ // single-request transfers, see nextClient.
+ pool []*storage.Client
+ next atomic.Uint64
}
// NewGCSClient returns a GCS-backed ObjectStorage. It builds its own
-// storage.Client from opts (and more from the same opts for the part-upload
-// pool, see uploadClient) rather than accepting one, because it installs a
+// storage.Client from opts (and more from the same opts for the connection
+// pool, see newPool) rather than accepting one, because it installs a
// RetryAlways policy (see setRetry) that is only safe for this package's own
// operations and must not leak onto a client shared with other code.
func NewGCSClient(ctx context.Context, opts ...option.ClientOption) (ObjectStorage, error) {
@@ -79,7 +76,7 @@ func NewGCSClient(ctx context.Context, opts ...option.ClientOption) (ObjectStora
return nil, err
}
setRetry(client)
- return &gcsClient{client: client, opts: opts}, nil
+ return &gcsClient{client: client, pool: newPool(ctx, opts)}, nil
}
// setRetry makes every operation on c retry transient errors (408, 429, 5xx)
@@ -131,9 +128,10 @@ func (g *gcsClient) PutObject(ctx context.Context, bucket, object string, reader
return g.putComposite(ctx, bucket, object, bytes.NewReader(head[:n]), reader)
}
-// putSingle writes the whole body in one resumable request.
+// putSingle writes the whole body in one resumable request, on a pooled client so
+// the small objects a node uploads concurrently do not queue on one connection.
func (g *gcsClient) putSingle(ctx context.Context, bucket, object string, reader io.Reader) error {
- wc := g.client.Bucket(bucket).Object(object).NewWriter(ctx)
+ wc := g.nextClient().Bucket(bucket).Object(object).NewWriter(ctx)
wc.ChunkSize = uploadChunkSize
// io.Copy reports local read errors; wc.Close() reports the actual
// GCS upload (auth, permissions, transient). Join both so the caller
diff --git a/pkg/objectstorage/gcs_retry_test.go b/pkg/objectstorage/gcs_retry_test.go
index 192bffbdfb..fe1c089c3f 100644
--- a/pkg/objectstorage/gcs_retry_test.go
+++ b/pkg/objectstorage/gcs_retry_test.go
@@ -25,6 +25,19 @@ import (
"testing"
)
+// fakeGCSClient returns a production-configured client bound to srv.
+func fakeGCSClient(t *testing.T, srv *httptest.Server) *gcsClient {
+ t.Helper()
+ t.Setenv("STORAGE_EMULATOR_HOST", srv.URL)
+ store, err := NewGCSClient(context.Background())
+ if err != nil {
+ t.Fatalf("storage client: %v", err)
+ }
+ g := store.(*gcsClient)
+ t.Cleanup(func() { g.client.Close() })
+ return g
+}
+
// A transient 429 on an upload must be retried, not surfaced: GCS sheds write
// bursts with rateLimitExceeded while it scales a bucket's key ranges, and the
// default client policy (RetryIdempotent) would fail the whole snapshot on the
@@ -68,14 +81,8 @@ func TestPutObjectRetriesTransient429(t *testing.T) {
defer srv.Close()
srvURL = srv.URL
- // The storage client routes everything, uploads included, at the emulator.
- t.Setenv("STORAGE_EMULATOR_HOST", srv.URL)
+ store := fakeGCSClient(t, srv)
ctx := context.Background()
- store, err := NewGCSClient(ctx)
- if err != nil {
- t.Fatalf("storage client: %v", err)
- }
- defer store.(*gcsClient).client.Close()
if err := store.PutObject(ctx, "snapshots", "snap/pages.img.zstd", strings.NewReader("payload")); err != nil {
t.Fatalf("PutObject after a transient 429: %v", err)
diff --git a/pkg/objectstorage/gcscompose.go b/pkg/objectstorage/gcscompose.go
index c093de762b..545339dfd4 100644
--- a/pkg/objectstorage/gcscompose.go
+++ b/pkg/objectstorage/gcscompose.go
@@ -25,36 +25,55 @@ import (
"cloud.google.com/go/storage"
"golang.org/x/sync/errgroup"
+ "google.golang.org/api/option"
)
// uploadPoolSize is how many storage.Clients the parallel part upload spreads its
// parts over. One client keeps a single HTTP/2 connection per host and multiplexes
// every part onto it, so parts that should be independent share one TCP stream:
// measured on a GKE worker node at 8 streams, 334 MB/s shared against 518 MB/s with a
-// connection each.
+// connection each. The same holds for the many small objects a node moves at once:
+// each snapshot below the part and range thresholds is a single request, so the
+// single-request paths draw on this pool too (see nextClient).
const uploadPoolSize = 8
-// uploadClient returns the client part i should use, or the default client if the pool
-// could not be built (a pool failure costs throughput, not correctness).
-func (g *gcsClient) uploadClient(ctx context.Context, i int) *storage.Client {
- g.poolOnce.Do(func() {
- for range uploadPoolSize {
- // The clients outlive this call, so they must not hold its cancellation.
- c, err := storage.NewClient(context.WithoutCancel(ctx), g.opts...)
- if err != nil {
- slog.WarnContext(ctx, "Falling back to one client for part uploads", slog.Any("err", err))
- return
- }
- setRetry(c)
- g.pool = append(g.pool, c)
+// newPool builds the uploadPoolSize clients transfers are spread over, from the
+// same opts as the client they stand in for: a pooled client that authenticates
+// differently from the one that opened an object fails partway through reading
+// it. It is built with that client rather than on first use, so a burst of
+// transfers on a fresh node does not wait for it. A pool that could not be
+// completed is used as far as it got, down to none: that costs throughput, not
+// correctness.
+func newPool(ctx context.Context, opts []option.ClientOption) []*storage.Client {
+ pool := make([]*storage.Client, 0, uploadPoolSize)
+ for range uploadPoolSize {
+ c, err := storage.NewClient(ctx, opts...)
+ if err != nil {
+ slog.WarnContext(ctx, "GCS connection pool is short; transfers share fewer connections",
+ slog.Int("built", len(pool)), slog.Int("want", uploadPoolSize), slog.Any("err", err))
+ break
}
- })
+ setRetry(c)
+ pool = append(pool, c)
+ }
+ return pool
+}
+
+// uploadClient returns the client part i should use, or the default client if there
+// is no pool.
+func (g *gcsClient) uploadClient(i int) *storage.Client {
if len(g.pool) == 0 {
return g.client
}
return g.pool[i%len(g.pool)]
}
+// nextClient hands out pooled clients round-robin, for a transfer that is one
+// request and so has no part or range index of its own.
+func (g *gcsClient) nextClient() *storage.Client {
+ return g.uploadClient(int(g.next.Add(1) - 1))
+}
+
// One stream to GCS tops out near 100 MiB/s however it is chunked; several do not
// (measured at 300 MiB: 82-107 MiB/s on one stream, 233-257 on four). So a large object
// is cut into parts, uploaded concurrently, and composed server-side. The parts are
@@ -75,24 +94,13 @@ const (
// into object. Only for objects past uploadCompositeMin; putSingle handles the rest.
func (g *gcsClient) putComposite(ctx context.Context, bucket, object string, head io.Reader, rest io.Reader) (err error) {
bkt := g.client.Bucket(bucket)
- // A run id keeps concurrent or retried uploads of the same object from colliding on
- // part names, and makes leftovers from a crash identifiable.
- var idBytes [8]byte
- if _, err := rand.Read(idBytes[:]); err != nil {
- return fmt.Errorf("while naming upload parts: %w", err)
+ runID, err := newRunID()
+ if err != nil {
+ return err
}
- runID := hex.EncodeToString(idBytes[:])
var parts []*storage.ObjectHandle
- defer func() {
- // Parts are scratch either way, and deleting them must not mask the real error.
- for _, p := range parts {
- if delErr := p.Delete(context.WithoutCancel(ctx)); delErr != nil &&
- !errors.Is(delErr, storage.ErrObjectNotExist) && err == nil {
- err = fmt.Errorf("while removing upload part %q: %w", p.ObjectName(), delErr)
- }
- }
- }()
+ defer func() { err = removeScratch(ctx, parts, err) }()
// The stream is read in order here and whole parts handed to the uploaders; buffers
// cycle through free so the peak stays at uploadConcurrency of them.
@@ -115,7 +123,7 @@ func (g *gcsClient) putComposite(ctx context.Context, bucket, object string, hea
if n > 0 {
part := bkt.Object(fmt.Sprintf("%s.part-%s-%04d", object, runID, i))
parts = append(parts, part)
- upPart := g.uploadClient(ctx, i).Bucket(bucket).Object(part.ObjectName())
+ upPart := g.uploadClient(i).Bucket(bucket).Object(part.ObjectName())
data := buf[:n]
g2.Go(func() error {
defer func() { free <- buf }()
@@ -137,6 +145,29 @@ func (g *gcsClient) putComposite(ctx context.Context, bucket, object string, hea
return composeAll(ctx, bkt, object, parts, runID)
}
+// newRunID returns an id that keeps concurrent or retried uploads of the same object
+// from colliding on part names, and makes leftovers from a crash identifiable.
+func newRunID() (string, error) {
+ var b [8]byte
+ if _, err := rand.Read(b[:]); err != nil {
+ return "", fmt.Errorf("while naming upload parts: %w", err)
+ }
+ return hex.EncodeToString(b[:]), nil
+}
+
+// removeScratch deletes objs, which are scratch whether or not the upload succeeded.
+// It returns err, or the first delete failure if err is nil, so cleanup never masks
+// the real error.
+func removeScratch(ctx context.Context, objs []*storage.ObjectHandle, err error) error {
+ for _, o := range objs {
+ if delErr := o.Delete(context.WithoutCancel(ctx)); delErr != nil &&
+ !errors.Is(delErr, storage.ErrObjectNotExist) && err == nil {
+ err = fmt.Errorf("while removing scratch object %q: %w", o.ObjectName(), delErr)
+ }
+ }
+ return err
+}
+
// writeObject writes data to obj in one request.
func writeObject(ctx context.Context, obj *storage.ObjectHandle, data []byte, size int) error {
w := obj.NewWriter(ctx)
@@ -150,11 +181,15 @@ func writeObject(ctx context.Context, obj *storage.ObjectHandle, data []byte, si
}
// composeAll composes parts into object, folding in rounds when there are more parts
-// than GCS accepts in one call. Intermediates are cleaned up as they are consumed.
-func composeAll(ctx context.Context, bkt *storage.BucketHandle, object string, parts []*storage.ObjectHandle, runID string) error {
+// than GCS accepts in one call. The intermediates it creates are removed on return; the
+// caller removes parts.
+func composeAll(ctx context.Context, bkt *storage.BucketHandle, object string, parts []*storage.ObjectHandle, runID string) (err error) {
if len(parts) == 0 {
return fmt.Errorf("no parts to compose into %q", object)
}
+ var mids []*storage.ObjectHandle
+ defer func() { err = removeScratch(ctx, mids, err) }()
+
round := 0
for len(parts) > maxComposeSources {
var next []*storage.ObjectHandle
@@ -166,14 +201,11 @@ func composeAll(ctx context.Context, bkt *storage.BucketHandle, object string, p
continue
}
mid := bkt.Object(fmt.Sprintf("%s.compose-%s-%d-%04d", object, runID, round, i))
+ // Before Run: it can fail after GCS committed mid.
+ mids = append(mids, mid)
if _, err := mid.ComposerFrom(group...).Run(ctx); err != nil {
return fmt.Errorf("while composing parts of %q: %w", object, err)
}
- // The sources are dead once folded; the caller's deferred cleanup only
- // knows about the leaf parts.
- for _, s := range group {
- _ = s.Delete(context.WithoutCancel(ctx))
- }
next = append(next, mid)
}
parts = next
diff --git a/pkg/objectstorage/gcscompose_test.go b/pkg/objectstorage/gcscompose_test.go
index e081e1831f..8fe5b2a5b0 100644
--- a/pkg/objectstorage/gcscompose_test.go
+++ b/pkg/objectstorage/gcscompose_test.go
@@ -18,10 +18,16 @@ import (
"bytes"
"context"
"crypto/sha256"
+ "encoding/json"
+ "fmt"
"io"
"math/rand/v2"
+ "net/http"
+ "net/http/httptest"
+ "net/url"
"os"
"strings"
+ "sync"
"testing"
"cloud.google.com/go/storage"
@@ -121,3 +127,84 @@ func TestPutObjectCompositeRoundTrip(t *testing.T) {
}
}
}
+
+// fakeObjects is an in-memory GCS that serves only what composeAll sends: compose
+// (concatenation, capped at maxComposeSources like GCS) and delete.
+type fakeObjects struct {
+ mu sync.Mutex
+ objs map[string][]byte
+}
+
+func (f *fakeObjects) ServeHTTP(w http.ResponseWriter, r *http.Request) {
+ rest, ok := strings.CutPrefix(r.URL.EscapedPath(), "/storage/v1/b/test/o/")
+ if !ok {
+ http.NotFound(w, r)
+ return
+ }
+ f.mu.Lock()
+ defer f.mu.Unlock()
+ switch {
+ case r.Method == http.MethodPost && strings.HasSuffix(rest, "/compose"):
+ name, _ := url.PathUnescape(strings.TrimSuffix(rest, "/compose"))
+ var req struct {
+ SourceObjects []struct{ Name string } `json:"sourceObjects"`
+ }
+ if err := json.NewDecoder(r.Body).Decode(&req); err != nil || len(req.SourceObjects) > maxComposeSources {
+ http.Error(w, "bad compose", http.StatusBadRequest)
+ return
+ }
+ var data []byte
+ for _, src := range req.SourceObjects {
+ b, ok := f.objs[src.Name]
+ if !ok {
+ http.NotFound(w, r)
+ return
+ }
+ data = append(data, b...)
+ }
+ f.objs[name] = data
+ w.Header().Set("Content-Type", "application/json")
+ _ = json.NewEncoder(w).Encode(map[string]string{"name": name, "bucket": "test"})
+ case r.Method == http.MethodDelete:
+ name, _ := url.PathUnescape(rest)
+ if _, ok := f.objs[name]; !ok {
+ http.NotFound(w, r)
+ return
+ }
+ delete(f.objs, name)
+ w.WriteHeader(http.StatusNoContent)
+ default:
+ http.NotFound(w, r)
+ }
+}
+
+// Folding more than maxComposeSources parts goes through intermediate objects, and
+// none may outlive the compose, including the ones the final compose consumes.
+func TestComposeAllRemovesIntermediates(t *testing.T) {
+ const partSize = 1 << 10
+ want := body((maxComposeSources + 8) * partSize)
+ fake := &fakeObjects{objs: map[string][]byte{}}
+ srv := httptest.NewServer(fake)
+ defer srv.Close()
+ bkt := fakeGCSClient(t, srv).client.Bucket("test")
+
+ var parts []*storage.ObjectHandle
+ for i := 0; i < len(want); i += partSize {
+ name := fmt.Sprintf("fold/obj.part-run-%04d", i/partSize)
+ fake.objs[name] = want[i : i+partSize]
+ parts = append(parts, bkt.Object(name))
+ }
+
+ if err := composeAll(context.Background(), bkt, "fold/obj", parts, "run"); err != nil {
+ t.Fatalf("composeAll: %v", err)
+ }
+
+ if got := fake.objs["fold/obj"]; !bytes.Equal(got, want) {
+ t.Errorf("composed %d bytes differ from the %d uploaded", len(got), len(want))
+ }
+ for name := range fake.objs {
+ if strings.Contains(name, ".compose-") {
+ t.Errorf("composeAll left intermediate %q behind", name)
+ }
+ }
+}
diff --git a/pkg/objectstorage/gcspool_test.go b/pkg/objectstorage/gcspool_test.go
index d9ed6e3cca..e43412de84 100644
--- a/pkg/objectstorage/gcspool_test.go
+++ b/pkg/objectstorage/gcspool_test.go
@@ -16,8 +16,13 @@ package objectstorage
import (
"context"
+ "io"
+ "net/http"
+ "strings"
+ "sync/atomic"
"testing"
+ "cloud.google.com/go/storage"
"google.golang.org/api/option"
)
@@ -46,7 +51,7 @@ func TestPooledClientsAreBuiltLikeTheClientTheyStandIn(t *testing.T) {
}
defer g.client.Close()
- pooled := g.uploadClient(ctx, 0)
+ pooled := g.uploadClient(0)
if pooled == nil {
t.Fatal("uploadClient returned nil")
}
@@ -58,8 +63,76 @@ func TestPooledClientsAreBuiltLikeTheClientTheyStandIn(t *testing.T) {
t.Errorf("pool holds %d clients, want %d", len(g.pool), uploadPoolSize)
}
for i := range uploadPoolSize {
- if c := g.uploadClient(ctx, i); c == nil || c == g.client {
+ if c := g.uploadClient(i); c == nil || c == g.client {
t.Errorf("uploadClient(%d) did not return a pooled connection", i)
}
}
}
+
+// recordingClient is a storage.Client whose every request is counted, standing
+// in for one pooled connection.
+func recordingClient(t *testing.T, hits *atomic.Int32) *storage.Client {
+ t.Helper()
+ client, err := storage.NewClient(t.Context(), option.WithHTTPClient(&http.Client{Transport: roundTripFunc(func(req *http.Request) (*http.Response, error) {
+ hits.Add(1)
+ if req.Method == http.MethodGet {
+ // A whole small object in one range: the reader sees its size and
+ // stays a single request.
+ return &http.Response{
+ StatusCode: http.StatusPartialContent,
+ Header: http.Header{"Content-Range": {"bytes 0-6/7"}, "Content-Type": {"application/octet-stream"}},
+ ContentLength: 7,
+ Body: io.NopCloser(strings.NewReader("payload")),
+ }, nil
+ }
+ _, _ = io.Copy(io.Discard, req.Body)
+ return &http.Response{
+ StatusCode: http.StatusOK,
+ Header: http.Header{"Content-Type": {"application/json"}},
+ Body: io.NopCloser(strings.NewReader(`{"name":"object","bucket":"snapshots"}`)),
+ }, nil
+ })}))
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { client.Close() })
+ return client
+}
+
+// TestSingleRequestTransfersSpreadOverThePool is the point of the pool for a node
+// turning over actors: a compressed snapshot is usually below the part and range
+// thresholds, so it is one request, and without this every snapshot the node moves
+// at once would multiplex onto the wrapped client's one connection.
+func TestSingleRequestTransfersSpreadOverThePool(t *testing.T) {
+ var wrapped atomic.Int32
+ g := &gcsClient{client: recordingClient(t, &wrapped)}
+ hits := make([]atomic.Int32, 3)
+ for i := range hits {
+ g.pool = append(g.pool, recordingClient(t, &hits[i]))
+ }
+
+ ctx := t.Context()
+ for i := range 2 * len(hits) {
+ if i%2 == 0 {
+ rc, err := g.GetObject(ctx, "snapshots", "object")
+ if err != nil {
+ t.Fatalf("GetObject %d: %v", i, err)
+ }
+ if _, err := io.Copy(io.Discard, rc); err != nil {
+ t.Fatalf("reading object %d: %v", i, err)
+ }
+ rc.Close()
+ } else if err := g.PutObject(ctx, "snapshots", "object", strings.NewReader("payload")); err != nil {
+ t.Fatalf("PutObject %d: %v", i, err)
+ }
+ }
+
+ if n := wrapped.Load(); n != 0 {
+ t.Errorf("the wrapped client served %d requests, want 0: small transfers bypassed the pool", n)
+ }
+ for i := range hits {
+ if n := hits[i].Load(); n != 2 {
+ t.Errorf("pooled client %d served %d requests, want 2: transfers were not spread round-robin", i, n)
+ }
+ }
+}
diff --git a/pkg/objectstorage/gcsranged.go b/pkg/objectstorage/gcsranged.go
index 63cf352301..bc939def5e 100644
--- a/pkg/objectstorage/gcsranged.go
+++ b/pkg/objectstorage/gcsranged.go
@@ -28,8 +28,11 @@ import (
func (g *gcsClient) GetObject(ctx context.Context, bucket, object string) (io.ReadCloser, error) {
ctx, cancel := context.WithCancel(ctx)
// The first chunk doubles as the size probe: a range read reports the whole
- // object's size in its attrs, so nothing pays an extra round trip for it.
- head, err := g.client.Bucket(bucket).Object(object).NewRangeReader(ctx, 0, downloadChunkSize)
+ // object's size in its attrs, so nothing pays an extra round trip for it. It
+ // goes out on a pooled client because for most snapshots it is the whole
+ // download, and a node restoring several at once must not queue them all on
+ // one connection.
+ head, err := g.nextClient().Bucket(bucket).Object(object).NewRangeReader(ctx, 0, downloadChunkSize)
if err != nil {
cancel()
if errors.Is(err, storage.ErrObjectNotExist) || errors.Is(err, storage.ErrBucketNotExist) {
@@ -63,7 +66,7 @@ func (r *gcsReader) Close() error {
// multiplex onto the one HTTP/2 connection a single storage.Client holds.
func (g *gcsClient) fetchRange(bucket, object string) fetchRangeFunc {
return func(ctx context.Context, i int, off, n int64, buf []byte) error {
- rc, err := g.uploadClient(ctx, i).Bucket(bucket).Object(object).NewRangeReader(ctx, off, n)
+ rc, err := g.uploadClient(i).Bucket(bucket).Object(object).NewRangeReader(ctx, off, n)
if err != nil {
return fmt.Errorf("while opening range %d+%d of %q: %w", off, n, object, err)
}
diff --git a/pkg/objectstorage/gcsranged_test.go b/pkg/objectstorage/gcsranged_test.go
index 24b30166eb..9d9fa23fe9 100644
--- a/pkg/objectstorage/gcsranged_test.go
+++ b/pkg/objectstorage/gcsranged_test.go
@@ -44,8 +44,7 @@ func TestGCSGetObjectCloseCancelsInitialRequest(t *testing.T) {
t.Fatal(err)
}
defer client.Close()
- g := &gcsClient{client: client}
- g.poolOnce.Do(func() {}) // Use the same test client for every range.
+ g := &gcsClient{client: client} // No pool: the test client serves every range.
r, err := g.GetObject(t.Context(), "snapshots", "object")
if err != nil {
t.Fatal(err)
diff --git a/pkg/objectstorage/gcssparse.go b/pkg/objectstorage/gcssparse.go
index b19f2f52a9..16224b1993 100644
--- a/pkg/objectstorage/gcssparse.go
+++ b/pkg/objectstorage/gcssparse.go
@@ -16,8 +16,6 @@ package objectstorage
import (
"context"
- "crypto/rand"
- "encoding/hex"
"errors"
"fmt"
"os"
@@ -61,31 +59,23 @@ func (g *gcsClient) PutSparseFile(ctx context.Context, bucket, object string, f
}
groups := planSparseParts(exts, populated, n)
- var idBytes [8]byte
- if _, err := rand.Read(idBytes[:]); err != nil {
- return res, fmt.Errorf("while naming upload parts: %w", err)
+ runID, err := newRunID()
+ if err != nil {
+ return res, err
}
- runID := hex.EncodeToString(idBytes[:])
bkt := g.client.Bucket(bucket)
parts := make([]*storage.ObjectHandle, len(groups))
for i := range groups {
parts[i] = bkt.Object(fmt.Sprintf("%s.part-%s-%04d", object, runID, i))
}
- defer func() {
- for _, p := range parts {
- if delErr := p.Delete(context.WithoutCancel(ctx)); delErr != nil &&
- !errors.Is(delErr, storage.ErrObjectNotExist) && err == nil {
- err = fmt.Errorf("while removing upload part %q: %w", p.ObjectName(), delErr)
- }
- }
- }()
+ defer func() { err = removeScratch(ctx, parts, err) }()
grp, gctx := errgroup.WithContext(ctx)
for i, ranges := range groups {
grp.Go(func() error {
// Each part uploads over its own connection; see uploadClient.
- obj := g.uploadClient(gctx, i).Bucket(bucket).Object(parts[i].ObjectName())
+ obj := g.uploadClient(i).Bucket(bucket).Object(parts[i].ObjectName())
w := obj.NewWriter(gctx)
w.ChunkSize = partWriterChunk
if err := writeSparsePart(contextWriter{ctx: gctx, dst: w}, f, size, ranges, i == 0, i == len(groups)-1); err != nil {
diff --git a/pkg/objectstorage/upload.go b/pkg/objectstorage/upload.go
index ce116acfd3..d5ffd15459 100644
--- a/pkg/objectstorage/upload.go
+++ b/pkg/objectstorage/upload.go
@@ -107,7 +107,7 @@ func (u *Uploader) plan(content io.Reader) (uploadPlan, error) {
if err != nil {
return uploadPlan{}, err
}
- _, populated, err := sparseExtents(f, info.Size())
+ exts, populated, err := sparseExtents(f, info.Size())
_, seekErr := f.Seek(position, io.SeekStart)
if err != nil {
return uploadPlan{}, err
@@ -118,6 +118,13 @@ func (u *Uploader) plan(content io.Reader) (uploadPlan, error) {
// Holes never enter the compressor or range uploads.
workers = min(workers, populated/parZstdChunk+1)
parts = min(sparsePartCount(populated), int(u.budget.capacity/sparseUploadMemory))
+ if _, ok := u.client.(sparseFilePutter); ok && parts >= 2 {
+ // PutSparseFile splits a file into parts whenever it has two or
+ // more (see errSparseTooSmall), so the streaming fallback never
+ // runs: reserve only what the parts hold.
+ return uploadPlan{workers: int(workers), parts: parts, memory: int64(parts) * sparseUploadMemory}, nil
+ }
+ buffers = streamBufferMemory(u.client, sparseEncodedBound(populated, len(exts)))
}
memory := buffers + workers*parallelWorkerMemory
if _, ok := u.client.(sparseFilePutter); ok {
@@ -128,6 +135,25 @@ func (u *Uploader) plan(content io.Reader) (uploadPlan, error) {
return uploadPlan{workers: int(workers), parts: parts, memory: memory}, nil
}
+// sparseEncodedBound bounds the size of the sparse-extent stream for a file
+// whose populated bytes lie in the given number of extents: zstd's worst-case
+// expansion plus the format's header, per-extent frames, and per-chunk zstd
+// frame headers.
+func sparseEncodedBound(populated int64, extents int) int64 {
+ return populated + populated/128 + int64(extents)*16 + (populated/parZstdChunk+1)*64 + 64
+}
+
+// streamBufferMemory is the backend buffering for a streamed object of at
+// most size bytes. GCS sends an object smaller than uploadCompositeMin in one
+// request: it holds PutObject's lookahead and the client's chunk buffer for
+// that request (see putSingle), never the composite parts.
+func streamBufferMemory(client ObjectStorage, size int64) int64 {
+ if _, ok := client.(*gcsClient); ok && size < uploadCompositeMin {
+ return uploadCompositeMin + uploadChunkSize
+ }
+ return uploadBufferMemory(client)
+}
+
// SendFile compresses and uploads an already-open snapshot file. The caller keeps
// ownership of the file. Waiting for memory is canceled with ctx.
func (u *Uploader) SendFile(ctx context.Context, url string, file *os.File) error {
@@ -147,6 +173,13 @@ func (u *Uploader) Send(ctx context.Context, url string, content io.Reader) erro
return err
}
memory := uploadBufferMemory(u.client)
+ if f, ok := content.(*os.File); ok {
+ // A manifest is a few KiB: size its reservation by the file rather
+ // than by the largest object the backend could buffer.
+ if info, err := f.Stat(); err == nil && info.Mode().IsRegular() {
+ memory = streamBufferMemory(u.client, info.Size())
+ }
+ }
if err := u.budget.reserved.Acquire(ctx, memory); err != nil {
return err
}
diff --git a/pkg/objectstorage/upload_test.go b/pkg/objectstorage/upload_test.go
index c0136d14d6..199dc8b366 100644
--- a/pkg/objectstorage/upload_test.go
+++ b/pkg/objectstorage/upload_test.go
@@ -52,11 +52,16 @@ func TestUploaderBudgetFitsBothPaths(t *testing.T) {
if plan.workers < 1 || plan.workers > runtime.GOMAXPROCS(0) || plan.memory > capacity {
t.Fatalf("%T: invalid plan %+v for capacity %d", client, plan, capacity)
}
- if plan.memory < uploadBufferMemory(client)+int64(plan.workers)*parallelWorkerMemory {
- t.Fatalf("%T: streaming pipeline under-reserved: %+v", client, plan)
+ if _, ok := client.(sparseFilePutter); ok && plan.parts >= 2 {
+ // Only the range-parallel path runs.
+ if int64(plan.parts)*sparseUploadMemory > plan.memory {
+ t.Fatalf("%T: sparse pipeline under-reserved: %+v", client, plan)
+ }
+ continue
}
- if _, ok := client.(sparseFilePutter); ok && (plan.parts < 1 || int64(plan.parts)*sparseUploadMemory > plan.memory) {
- t.Fatalf("%T: sparse pipeline under-reserved: %+v", client, plan)
+ // An all-holes file streams a few bytes.
+ if plan.memory < streamBufferMemory(client, sparseEncodedBound(0, 0))+int64(plan.workers)*parallelWorkerMemory {
+ t.Fatalf("%T: streaming pipeline under-reserved: %+v", client, plan)
}
}
}
@@ -70,10 +75,10 @@ func TestUploaderPlansFromPopulatedBytes(t *testing.T) {
populatedMiB int
gcs, s3 uploadPlan
}{
- {"all_holes", 0, uploadPlan{1, 1, 384 << 20}, uploadPlan{1, 1, 224 << 20}},
- {"small", 1, uploadPlan{1, 1, 384 << 20}, uploadPlan{1, 1, 224 << 20}},
- {"two_chunks", 9, uploadPlan{2, 1, 448 << 20}, uploadPlan{2, 1, 288 << 20}},
- {"multiple_ranges", 192, uploadPlan{3, 3, 512 << 20}, uploadPlan{5, 3, 480 << 20}},
+ {"all_holes", 0, uploadPlan{1, 1, 192 << 20}, uploadPlan{1, 1, 224 << 20}},
+ {"small", 1, uploadPlan{1, 1, 192 << 20}, uploadPlan{1, 1, 224 << 20}},
+ {"two_chunks", 9, uploadPlan{2, 1, 256 << 20}, uploadPlan{2, 1, 288 << 20}},
+ {"multiple_ranges", 192, uploadPlan{3, 3, 144 << 20}, uploadPlan{5, 3, 480 << 20}},
} {
t.Run(tc.name, func(t *testing.T) {
offsets := make([]int64, tc.populatedMiB)
@@ -113,6 +118,40 @@ func TestUploaderPlansFromPopulatedBytes(t *testing.T) {
}
}
+func TestUploaderReservesOnlyTheChosenPath(t *testing.T) {
+ previous := runtime.GOMAXPROCS(192)
+ defer runtime.GOMAXPROCS(previous)
+ offsets := make([]int64, 292)
+ for i := range offsets {
+ offsets[i] = int64(i) << 20
+ }
+ ranged := sparseFileWith(t, 2<<30, offsets)
+ small := sparseFileWith(t, 2<<30, []int64{0})
+ for _, capacity := range []int64{DefaultUploadMemory, 16 << 30} {
+ u, err := NewUploader(&gcsClient{}, capacity)
+ if err != nil {
+ t.Fatal(err)
+ }
+ // 292 MiB populated: four ranged parts, whatever the budget or GOMAXPROCS.
+ got, err := u.plan(ranged)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if got.parts != 4 || got.memory != 4*sparseUploadMemory {
+ t.Errorf("capacity %d: ranged plan = %+v, want 4 parts reserving %d", capacity, got, 4*sparseUploadMemory)
+ }
+ // 1 MiB populated: one request, so PutObject's lookahead, the client's
+ // chunk buffer, and one worker.
+ got, err = u.plan(small)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if want := uploadCompositeMin + uploadChunkSize + parallelWorkerMemory; got.memory != want {
+ t.Errorf("capacity %d: small plan = %+v, want %d", capacity, got, want)
+ }
+ }
+}
+
func TestUploaderSmallSparseFilesShareBudget(t *testing.T) {
previous := runtime.GOMAXPROCS(2)
defer runtime.GOMAXPROCS(previous)
diff --git a/pkg/proto/ateapipb/ateapi.pb.go b/pkg/proto/ateapipb/ateapi.pb.go
index b7280154ae..4efc4a0434 100644
--- a/pkg/proto/ateapipb/ateapi.pb.go
+++ b/pkg/proto/ateapipb/ateapi.pb.go
@@ -262,9 +262,9 @@ const (
ActorState_ACTOR_STATE_RUNNING ActorState = 2
ActorState_ACTOR_STATE_SUSPENDING ActorState = 3
ActorState_ACTOR_STATE_SUSPENDED ActorState = 4
- ActorState_ACTOR_STATE_CRASHED ActorState = 7
- ActorState_ACTOR_STATE_DELETING ActorState = 8
- ActorState_ACTOR_STATE_REVERTING ActorState = 9 // Keep this in sync with ActorStatus.state's maximum.
+ ActorState_ACTOR_STATE_CRASHED ActorState = 5
+ ActorState_ACTOR_STATE_DELETING ActorState = 6
+ ActorState_ACTOR_STATE_REVERTING ActorState = 7 // Keep this in sync with ActorStatus.state's maximum.
)
// Enum value maps for ActorState.
@@ -275,9 +275,9 @@ var (
2: "ACTOR_STATE_RUNNING",
3: "ACTOR_STATE_SUSPENDING",
4: "ACTOR_STATE_SUSPENDED",
- 7: "ACTOR_STATE_CRASHED",
- 8: "ACTOR_STATE_DELETING",
- 9: "ACTOR_STATE_REVERTING",
+ 5: "ACTOR_STATE_CRASHED",
+ 6: "ACTOR_STATE_DELETING",
+ 7: "ACTOR_STATE_REVERTING",
}
ActorState_value = map[string]int32{
"ACTOR_STATE_UNSPECIFIED": 0,
@@ -285,9 +285,9 @@ var (
"ACTOR_STATE_RUNNING": 2,
"ACTOR_STATE_SUSPENDING": 3,
"ACTOR_STATE_SUSPENDED": 4,
- "ACTOR_STATE_CRASHED": 7,
- "ACTOR_STATE_DELETING": 8,
- "ACTOR_STATE_REVERTING": 9,
+ "ACTOR_STATE_CRASHED": 5,
+ "ACTOR_STATE_DELETING": 6,
+ "ACTOR_STATE_REVERTING": 7,
}
)
@@ -1027,7 +1027,61 @@ func (x *ResourceMetadata) GetUpdateTime() *timestamppb.Timestamp {
return nil
}
-type ExternalVolume struct {
+// ExternalVolumes is the external volume state recorded on an actor, wrapped
+// so later additions do not reshape ActorStatus.
+type ExternalVolumes struct {
+ state protoimpl.MessageState `protogen:"open.v1"`
+ // volumes has one entry per external volume of the template. Each template
+ // volume appears at most once.
+ //
+ // +k8s:optional
+ // +k8s:maxItems=32 # matches the template's volumes bound
+ // +k8s:listType=map
+ // +k8s:listMapKey=name
+ Volumes []*ExternalVolumeStatus `protobuf:"bytes,1,rep,name=volumes,proto3" json:"volumes,omitempty"`
+ unknownFields protoimpl.UnknownFields
+ sizeCache protoimpl.SizeCache
+}
+
+func (x *ExternalVolumes) Reset() {
+ *x = ExternalVolumes{}
+ mi := &file_ateapi_proto_msgTypes[5]
+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+ ms.StoreMessageInfo(mi)
+}
+
+func (x *ExternalVolumes) String() string {
+ return protoimpl.X.MessageStringOf(x)
+}
+
+func (*ExternalVolumes) ProtoMessage() {}
+
+func (x *ExternalVolumes) ProtoReflect() protoreflect.Message {
+ mi := &file_ateapi_proto_msgTypes[5]
+ if x != nil {
+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+ if ms.LoadMessageInfo() == nil {
+ ms.StoreMessageInfo(mi)
+ }
+ return ms
+ }
+ return mi.MessageOf(x)
+}
+
+// Deprecated: Use ExternalVolumes.ProtoReflect.Descriptor instead.
+func (*ExternalVolumes) Descriptor() ([]byte, []int) {
+ return file_ateapi_proto_rawDescGZIP(), []int{5}
+}
+
+func (x *ExternalVolumes) GetVolumes() []*ExternalVolumeStatus {
+ if x != nil {
+ return x.Volumes
+ }
+ return nil
+}
+
+// ExternalVolumeStatus is one actor's view of an external volume it mounts.
+type ExternalVolumeStatus struct {
state protoimpl.MessageState `protogen:"open.v1"`
// Name of the volume specified in the actor template. Template volume
// names are DNS labels.
@@ -1068,26 +1122,26 @@ type ExternalVolume struct {
//
// +k8s:optional
// +k8s:customValidation # at most 40 KiB in total, ten times the CSI limit for map fields
- VolumeContext map[string]string `protobuf:"bytes,5,rep,name=volume_context,json=volumeContext,proto3" json:"volume_context,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"`
+ VolumeContext map[string]string `protobuf:"bytes,4,rep,name=volume_context,json=volumeContext,proto3" json:"volume_context,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
-func (x *ExternalVolume) Reset() {
- *x = ExternalVolume{}
- mi := &file_ateapi_proto_msgTypes[5]
+func (x *ExternalVolumeStatus) Reset() {
+ *x = ExternalVolumeStatus{}
+ mi := &file_ateapi_proto_msgTypes[6]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
-func (x *ExternalVolume) String() string {
+func (x *ExternalVolumeStatus) String() string {
return protoimpl.X.MessageStringOf(x)
}
-func (*ExternalVolume) ProtoMessage() {}
+func (*ExternalVolumeStatus) ProtoMessage() {}
-func (x *ExternalVolume) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[5]
+func (x *ExternalVolumeStatus) ProtoReflect() protoreflect.Message {
+ mi := &file_ateapi_proto_msgTypes[6]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -1098,33 +1152,33 @@ func (x *ExternalVolume) ProtoReflect() protoreflect.Message {
return mi.MessageOf(x)
}
-// Deprecated: Use ExternalVolume.ProtoReflect.Descriptor instead.
-func (*ExternalVolume) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{5}
+// Deprecated: Use ExternalVolumeStatus.ProtoReflect.Descriptor instead.
+func (*ExternalVolumeStatus) Descriptor() ([]byte, []int) {
+ return file_ateapi_proto_rawDescGZIP(), []int{6}
}
-func (x *ExternalVolume) GetName() string {
+func (x *ExternalVolumeStatus) GetName() string {
if x != nil {
return x.Name
}
return ""
}
-func (x *ExternalVolume) GetStorageVolumeId() string {
+func (x *ExternalVolumeStatus) GetStorageVolumeId() string {
if x != nil {
return x.StorageVolumeId
}
return ""
}
-func (x *ExternalVolume) GetVolumeType() string {
+func (x *ExternalVolumeStatus) GetVolumeType() string {
if x != nil {
return x.VolumeType
}
return ""
}
-func (x *ExternalVolume) GetVolumeContext() map[string]string {
+func (x *ExternalVolumeStatus) GetVolumeContext() map[string]string {
if x != nil {
return x.VolumeContext
}
@@ -1142,7 +1196,7 @@ type Actor struct {
// +k8s:required
// +k8s:subfield(atespace)=+k8s:required
// +k8s:mutable
- ActorTemplate *ObjectRef `protobuf:"bytes,4,opt,name=actor_template,json=actorTemplate,proto3" json:"actor_template,omitempty"`
+ ActorTemplate *ObjectRef `protobuf:"bytes,2,opt,name=actor_template,json=actorTemplate,proto3" json:"actor_template,omitempty"`
// worker_selector is the per-actor placement constraint. The scheduler
// evaluates the AND of this selector and the template's workerSelector to
// find eligible pools. It narrows the template's selection; pinning a
@@ -1150,7 +1204,7 @@ type Actor struct {
// Changes take effect on the next ResumeActor call.
//
// +k8s:optional
- WorkerSelector *Selector `protobuf:"bytes,5,opt,name=worker_selector,json=workerSelector,proto3" json:"worker_selector,omitempty"`
+ WorkerSelector *Selector `protobuf:"bytes,3,opt,name=worker_selector,json=workerSelector,proto3" json:"worker_selector,omitempty"`
// The tag specified by the caller at CreateActor to seed this Actor from an
// ActorSnapshot. Unset if the Actor was not created from a snapshot. Set
// once at creation and immutable afterward.
@@ -1158,19 +1212,19 @@ type Actor struct {
// +k8s:optional
// +k8s:subfield(atespace)=+k8s:required
// +k8s:immutable
- SourceTag *ObjectRef `protobuf:"bytes,6,opt,name=source_tag,json=sourceTag,proto3" json:"source_tag,omitempty"`
+ SourceTag *ObjectRef `protobuf:"bytes,4,opt,name=source_tag,json=sourceTag,proto3" json:"source_tag,omitempty"`
// status is the system-managed state of the Actor. It is updated by the
// server and ignored on input. It is always present in output.
//
// +k8s:optional
- Status *ActorStatus `protobuf:"bytes,7,opt,name=status,proto3" json:"status,omitempty"`
+ Status *ActorStatus `protobuf:"bytes,5,opt,name=status,proto3" json:"status,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *Actor) Reset() {
*x = Actor{}
- mi := &file_ateapi_proto_msgTypes[6]
+ mi := &file_ateapi_proto_msgTypes[7]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -1182,7 +1236,7 @@ func (x *Actor) String() string {
func (*Actor) ProtoMessage() {}
func (x *Actor) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[6]
+ mi := &file_ateapi_proto_msgTypes[7]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -1195,7 +1249,7 @@ func (x *Actor) ProtoReflect() protoreflect.Message {
// Deprecated: Use Actor.ProtoReflect.Descriptor instead.
func (*Actor) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{6}
+ return file_ateapi_proto_rawDescGZIP(), []int{7}
}
func (x *Actor) GetMetadata() *ResourceMetadata {
@@ -1273,7 +1327,7 @@ type EgressPolicy struct {
func (x *EgressPolicy) Reset() {
*x = EgressPolicy{}
- mi := &file_ateapi_proto_msgTypes[7]
+ mi := &file_ateapi_proto_msgTypes[8]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -1285,7 +1339,7 @@ func (x *EgressPolicy) String() string {
func (*EgressPolicy) ProtoMessage() {}
func (x *EgressPolicy) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[7]
+ mi := &file_ateapi_proto_msgTypes[8]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -1298,7 +1352,7 @@ func (x *EgressPolicy) ProtoReflect() protoreflect.Message {
// Deprecated: Use EgressPolicy.ProtoReflect.Descriptor instead.
func (*EgressPolicy) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{7}
+ return file_ateapi_proto_rawDescGZIP(), []int{8}
}
func (x *EgressPolicy) GetMetadata() *ResourceMetadata {
@@ -1348,7 +1402,7 @@ type EgressRule struct {
func (x *EgressRule) Reset() {
*x = EgressRule{}
- mi := &file_ateapi_proto_msgTypes[8]
+ mi := &file_ateapi_proto_msgTypes[9]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -1360,7 +1414,7 @@ func (x *EgressRule) String() string {
func (*EgressRule) ProtoMessage() {}
func (x *EgressRule) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[8]
+ mi := &file_ateapi_proto_msgTypes[9]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -1373,7 +1427,7 @@ func (x *EgressRule) ProtoReflect() protoreflect.Message {
// Deprecated: Use EgressRule.ProtoReflect.Descriptor instead.
func (*EgressRule) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{8}
+ return file_ateapi_proto_rawDescGZIP(), []int{9}
}
func (x *EgressRule) GetHttp() *HttpRule {
@@ -1441,7 +1495,7 @@ type HttpRule struct {
func (x *HttpRule) Reset() {
*x = HttpRule{}
- mi := &file_ateapi_proto_msgTypes[9]
+ mi := &file_ateapi_proto_msgTypes[10]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -1453,7 +1507,7 @@ func (x *HttpRule) String() string {
func (*HttpRule) ProtoMessage() {}
func (x *HttpRule) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[9]
+ mi := &file_ateapi_proto_msgTypes[10]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -1466,7 +1520,7 @@ func (x *HttpRule) ProtoReflect() protoreflect.Message {
// Deprecated: Use HttpRule.ProtoReflect.Descriptor instead.
func (*HttpRule) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{9}
+ return file_ateapi_proto_rawDescGZIP(), []int{10}
}
func (x *HttpRule) GetHostnames() []string {
@@ -1526,7 +1580,7 @@ type HttpsRule struct {
func (x *HttpsRule) Reset() {
*x = HttpsRule{}
- mi := &file_ateapi_proto_msgTypes[10]
+ mi := &file_ateapi_proto_msgTypes[11]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -1538,7 +1592,7 @@ func (x *HttpsRule) String() string {
func (*HttpsRule) ProtoMessage() {}
func (x *HttpsRule) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[10]
+ mi := &file_ateapi_proto_msgTypes[11]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -1551,7 +1605,7 @@ func (x *HttpsRule) ProtoReflect() protoreflect.Message {
// Deprecated: Use HttpsRule.ProtoReflect.Descriptor instead.
func (*HttpsRule) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{10}
+ return file_ateapi_proto_rawDescGZIP(), []int{11}
}
func (x *HttpsRule) GetHostnames() []string {
@@ -1602,7 +1656,7 @@ type TlsPassthroughRule struct {
func (x *TlsPassthroughRule) Reset() {
*x = TlsPassthroughRule{}
- mi := &file_ateapi_proto_msgTypes[11]
+ mi := &file_ateapi_proto_msgTypes[12]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -1614,7 +1668,7 @@ func (x *TlsPassthroughRule) String() string {
func (*TlsPassthroughRule) ProtoMessage() {}
func (x *TlsPassthroughRule) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[11]
+ mi := &file_ateapi_proto_msgTypes[12]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -1627,7 +1681,7 @@ func (x *TlsPassthroughRule) ProtoReflect() protoreflect.Message {
// Deprecated: Use TlsPassthroughRule.ProtoReflect.Descriptor instead.
func (*TlsPassthroughRule) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{11}
+ return file_ateapi_proto_rawDescGZIP(), []int{12}
}
func (x *TlsPassthroughRule) GetHostnames() []string {
@@ -1668,7 +1722,7 @@ type Ports struct {
func (x *Ports) Reset() {
*x = Ports{}
- mi := &file_ateapi_proto_msgTypes[12]
+ mi := &file_ateapi_proto_msgTypes[13]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -1680,7 +1734,7 @@ func (x *Ports) String() string {
func (*Ports) ProtoMessage() {}
func (x *Ports) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[12]
+ mi := &file_ateapi_proto_msgTypes[13]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -1693,7 +1747,7 @@ func (x *Ports) ProtoReflect() protoreflect.Message {
// Deprecated: Use Ports.ProtoReflect.Descriptor instead.
func (*Ports) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{12}
+ return file_ateapi_proto_rawDescGZIP(), []int{13}
}
func (x *Ports) GetAll() *AllPorts {
@@ -1719,7 +1773,7 @@ type AllPorts struct {
func (x *AllPorts) Reset() {
*x = AllPorts{}
- mi := &file_ateapi_proto_msgTypes[13]
+ mi := &file_ateapi_proto_msgTypes[14]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -1731,7 +1785,7 @@ func (x *AllPorts) String() string {
func (*AllPorts) ProtoMessage() {}
func (x *AllPorts) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[13]
+ mi := &file_ateapi_proto_msgTypes[14]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -1744,7 +1798,7 @@ func (x *AllPorts) ProtoReflect() protoreflect.Message {
// Deprecated: Use AllPorts.ProtoReflect.Descriptor instead.
func (*AllPorts) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{13}
+ return file_ateapi_proto_rawDescGZIP(), []int{14}
}
// HttpRuleEffects contains effects applied by a matching HTTP or HTTPS rule.
@@ -1778,7 +1832,7 @@ type HttpRuleEffects struct {
func (x *HttpRuleEffects) Reset() {
*x = HttpRuleEffects{}
- mi := &file_ateapi_proto_msgTypes[14]
+ mi := &file_ateapi_proto_msgTypes[15]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -1790,7 +1844,7 @@ func (x *HttpRuleEffects) String() string {
func (*HttpRuleEffects) ProtoMessage() {}
func (x *HttpRuleEffects) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[14]
+ mi := &file_ateapi_proto_msgTypes[15]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -1803,7 +1857,7 @@ func (x *HttpRuleEffects) ProtoReflect() protoreflect.Message {
// Deprecated: Use HttpRuleEffects.ProtoReflect.Descriptor instead.
func (*HttpRuleEffects) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{14}
+ return file_ateapi_proto_rawDescGZIP(), []int{15}
}
func (x *HttpRuleEffects) GetReplaceHeaders() []*CredentialHeader {
@@ -1853,7 +1907,7 @@ type CredentialHeader struct {
func (x *CredentialHeader) Reset() {
*x = CredentialHeader{}
- mi := &file_ateapi_proto_msgTypes[15]
+ mi := &file_ateapi_proto_msgTypes[16]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -1865,7 +1919,7 @@ func (x *CredentialHeader) String() string {
func (*CredentialHeader) ProtoMessage() {}
func (x *CredentialHeader) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[15]
+ mi := &file_ateapi_proto_msgTypes[16]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -1878,7 +1932,7 @@ func (x *CredentialHeader) ProtoReflect() protoreflect.Message {
// Deprecated: Use CredentialHeader.ProtoReflect.Descriptor instead.
func (*CredentialHeader) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{15}
+ return file_ateapi_proto_rawDescGZIP(), []int{16}
}
func (x *CredentialHeader) GetHeader() string {
@@ -1935,7 +1989,7 @@ type ActorJwtSource struct {
func (x *ActorJwtSource) Reset() {
*x = ActorJwtSource{}
- mi := &file_ateapi_proto_msgTypes[16]
+ mi := &file_ateapi_proto_msgTypes[17]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -1947,7 +2001,7 @@ func (x *ActorJwtSource) String() string {
func (*ActorJwtSource) ProtoMessage() {}
func (x *ActorJwtSource) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[16]
+ mi := &file_ateapi_proto_msgTypes[17]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -1960,7 +2014,7 @@ func (x *ActorJwtSource) ProtoReflect() protoreflect.Message {
// Deprecated: Use ActorJwtSource.ProtoReflect.Descriptor instead.
func (*ActorJwtSource) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{16}
+ return file_ateapi_proto_rawDescGZIP(), []int{17}
}
func (x *ActorJwtSource) GetAudiences() []string {
@@ -1983,7 +2037,7 @@ type ActorStatus struct {
//
// +k8s:required
// +k8s:minimum=1
- // +k8s:maximum=9 # keep this in sync with the ActorState enum
+ // +k8s:maximum=7 # keep this in sync with the ActorState enum
State ActorState `protobuf:"varint,1,opt,name=state,proto3,enum=ateapi.ActorState" json:"state,omitempty"`
// worker_assignment points at the worker currently hosting this Actor.
// Unset whenever the Actor has no worker (SUSPENDED, CRASHED).
@@ -1991,28 +2045,25 @@ type ActorStatus struct {
// +k8s:optional
// +k8s:update=NoModify # can be set and cleared, but not changed in place
WorkerAssignment *WorkerAssignment `protobuf:"bytes,2,opt,name=worker_assignment,json=workerAssignment,proto3" json:"worker_assignment,omitempty"`
- // Volumes attached to the actor. These volumes only live as long as the actor.
- // They are deleted when the actor is deleted. Each template volume
- // appears at most once.
+ // external_volumes is the state of the external volumes attached to the
+ // actor. These volumes only live as long as the actor and are deleted with
+ // it.
//
// +ate:preview=ExternalVolumes
// +k8s:optional
- // +k8s:maxItems=32 # matches the template's volumes bound
- // +k8s:listType=map
- // +k8s:listMapKey=name
- ExternalVolumes []*ExternalVolume `protobuf:"bytes,7,rep,name=external_volumes,json=externalVolumes,proto3" json:"external_volumes,omitempty"`
+ ExternalVolumes *ExternalVolumes `protobuf:"bytes,3,opt,name=external_volumes,json=externalVolumes,proto3" json:"external_volumes,omitempty"`
// crash records why and when the Actor entered CRASHED. It is set with the
// CRASHED state and cleared when a revert returns the Actor to SUSPENDED.
//
// +k8s:optional
- Crash *ActorCrash `protobuf:"bytes,9,opt,name=crash,proto3" json:"crash,omitempty"`
+ Crash *ActorCrash `protobuf:"bytes,4,opt,name=crash,proto3" json:"crash,omitempty"`
// assigned_node is the Kubernetes node the Actor is currently attached to
// (hosting its active worker or its node-local snapshots). Unset when the
// Actor is detached from any node.
//
// +k8s:optional
// +k8s:format=k8s-long-name
- AssignedNode string `protobuf:"bytes,10,opt,name=assigned_node,json=assignedNode,proto3" json:"assigned_node,omitempty"`
+ AssignedNode string `protobuf:"bytes,5,opt,name=assigned_node,json=assignedNode,proto3" json:"assigned_node,omitempty"`
// snapshots holds the Actor's active and in-progress snapshot generations.
//
// +k8s:optional
@@ -2020,7 +2071,7 @@ type ActorStatus struct {
// +k8s:listType=atomic
// +k8s:unique=map
// +k8s:listMapKey=generation
- Snapshots []*Snapshot `protobuf:"bytes,11,rep,name=snapshots,proto3" json:"snapshots,omitempty"`
+ Snapshots []*Snapshot `protobuf:"bytes,6,rep,name=snapshots,proto3" json:"snapshots,omitempty"`
// last_assigned_generation tracks the last snapshot generation assigned for
// this Actor during a Suspend or Pause request.
// Even during Revert, the generation is not decreased to keep track of snapshots
@@ -2030,14 +2081,14 @@ type ActorStatus struct {
// +k8s:alpha(since: "0.0")=+k8s:monotonic # TODO: get rid of alpha prefix
// +k8s:minimum=0
// +k8s:update=NoUnset
- LastAssignedGeneration int32 `protobuf:"varint,12,opt,name=last_assigned_generation,json=lastAssignedGeneration,proto3" json:"last_assigned_generation,omitempty"`
+ LastAssignedGeneration int32 `protobuf:"varint,7,opt,name=last_assigned_generation,json=lastAssignedGeneration,proto3" json:"last_assigned_generation,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *ActorStatus) Reset() {
*x = ActorStatus{}
- mi := &file_ateapi_proto_msgTypes[17]
+ mi := &file_ateapi_proto_msgTypes[18]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -2049,7 +2100,7 @@ func (x *ActorStatus) String() string {
func (*ActorStatus) ProtoMessage() {}
func (x *ActorStatus) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[17]
+ mi := &file_ateapi_proto_msgTypes[18]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -2062,7 +2113,7 @@ func (x *ActorStatus) ProtoReflect() protoreflect.Message {
// Deprecated: Use ActorStatus.ProtoReflect.Descriptor instead.
func (*ActorStatus) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{17}
+ return file_ateapi_proto_rawDescGZIP(), []int{18}
}
func (x *ActorStatus) GetState() ActorState {
@@ -2079,7 +2130,7 @@ func (x *ActorStatus) GetWorkerAssignment() *WorkerAssignment {
return nil
}
-func (x *ActorStatus) GetExternalVolumes() []*ExternalVolume {
+func (x *ActorStatus) GetExternalVolumes() *ExternalVolumes {
if x != nil {
return x.ExternalVolumes
}
@@ -2132,7 +2183,7 @@ type ActorCrash struct {
func (x *ActorCrash) Reset() {
*x = ActorCrash{}
- mi := &file_ateapi_proto_msgTypes[18]
+ mi := &file_ateapi_proto_msgTypes[19]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -2144,7 +2195,7 @@ func (x *ActorCrash) String() string {
func (*ActorCrash) ProtoMessage() {}
func (x *ActorCrash) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[18]
+ mi := &file_ateapi_proto_msgTypes[19]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -2157,7 +2208,7 @@ func (x *ActorCrash) ProtoReflect() protoreflect.Message {
// Deprecated: Use ActorCrash.ProtoReflect.Descriptor instead.
func (*ActorCrash) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{18}
+ return file_ateapi_proto_rawDescGZIP(), []int{19}
}
func (x *ActorCrash) GetMessage() string {
@@ -2250,7 +2301,7 @@ type WorkerAssignment struct {
func (x *WorkerAssignment) Reset() {
*x = WorkerAssignment{}
- mi := &file_ateapi_proto_msgTypes[19]
+ mi := &file_ateapi_proto_msgTypes[20]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -2262,7 +2313,7 @@ func (x *WorkerAssignment) String() string {
func (*WorkerAssignment) ProtoMessage() {}
func (x *WorkerAssignment) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[19]
+ mi := &file_ateapi_proto_msgTypes[20]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -2275,7 +2326,7 @@ func (x *WorkerAssignment) ProtoReflect() protoreflect.Message {
// Deprecated: Use WorkerAssignment.ProtoReflect.Descriptor instead.
func (*WorkerAssignment) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{19}
+ return file_ateapi_proto_rawDescGZIP(), []int{20}
}
func (x *WorkerAssignment) GetWorker() *ObjectRef {
@@ -2375,7 +2426,7 @@ type TagStatus struct {
func (x *TagStatus) Reset() {
*x = TagStatus{}
- mi := &file_ateapi_proto_msgTypes[20]
+ mi := &file_ateapi_proto_msgTypes[21]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -2387,7 +2438,7 @@ func (x *TagStatus) String() string {
func (*TagStatus) ProtoMessage() {}
func (x *TagStatus) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[20]
+ mi := &file_ateapi_proto_msgTypes[21]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -2400,7 +2451,7 @@ func (x *TagStatus) ProtoReflect() protoreflect.Message {
// Deprecated: Use TagStatus.ProtoReflect.Descriptor instead.
func (*TagStatus) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{20}
+ return file_ateapi_proto_rawDescGZIP(), []int{21}
}
func (x *TagStatus) GetSnapshot() *Snapshot {
@@ -2452,7 +2503,7 @@ type Tag struct {
func (x *Tag) Reset() {
*x = Tag{}
- mi := &file_ateapi_proto_msgTypes[21]
+ mi := &file_ateapi_proto_msgTypes[22]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -2464,7 +2515,7 @@ func (x *Tag) String() string {
func (*Tag) ProtoMessage() {}
func (x *Tag) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[21]
+ mi := &file_ateapi_proto_msgTypes[22]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -2477,7 +2528,7 @@ func (x *Tag) ProtoReflect() protoreflect.Message {
// Deprecated: Use Tag.ProtoReflect.Descriptor instead.
func (*Tag) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{21}
+ return file_ateapi_proto_rawDescGZIP(), []int{22}
}
func (x *Tag) GetMetadata() *ResourceMetadata {
@@ -2523,7 +2574,7 @@ type Atespace struct {
func (x *Atespace) Reset() {
*x = Atespace{}
- mi := &file_ateapi_proto_msgTypes[22]
+ mi := &file_ateapi_proto_msgTypes[23]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -2535,7 +2586,7 @@ func (x *Atespace) String() string {
func (*Atespace) ProtoMessage() {}
func (x *Atespace) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[22]
+ mi := &file_ateapi_proto_msgTypes[23]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -2548,7 +2599,7 @@ func (x *Atespace) ProtoReflect() protoreflect.Message {
// Deprecated: Use Atespace.ProtoReflect.Descriptor instead.
func (*Atespace) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{22}
+ return file_ateapi_proto_rawDescGZIP(), []int{23}
}
func (x *Atespace) GetMetadata() *ResourceMetadata {
@@ -2581,7 +2632,7 @@ type ObjectRef struct {
func (x *ObjectRef) Reset() {
*x = ObjectRef{}
- mi := &file_ateapi_proto_msgTypes[23]
+ mi := &file_ateapi_proto_msgTypes[24]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -2593,7 +2644,7 @@ func (x *ObjectRef) String() string {
func (*ObjectRef) ProtoMessage() {}
func (x *ObjectRef) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[23]
+ mi := &file_ateapi_proto_msgTypes[24]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -2606,7 +2657,7 @@ func (x *ObjectRef) ProtoReflect() protoreflect.Message {
// Deprecated: Use ObjectRef.ProtoReflect.Descriptor instead.
func (*ObjectRef) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{23}
+ return file_ateapi_proto_rawDescGZIP(), []int{24}
}
func (x *ObjectRef) GetAtespace() string {
@@ -2651,7 +2702,10 @@ type ActorTemplate struct {
// +k8s:listType=map
// +k8s:listMapKey=name
Volumes []*Volume `protobuf:"bytes,4,rep,name=volumes,proto3" json:"volumes,omitempty"`
- // +k8s:required
+ // snapshot_config selects what actor snapshots capture and where they are
+ // stored. Optional: when omitted, every field takes its default.
+ //
+ // +k8s:optional
SnapshotConfig *SnapshotConfig `protobuf:"bytes,5,opt,name=snapshot_config,json=snapshotConfig,proto3" json:"snapshot_config,omitempty"`
// resources is the actor's compute size. Both cpu and memory limits are
// required: the scheduler books them against worker capacity, and a
@@ -2659,16 +2713,16 @@ type ActorTemplate struct {
//
// +k8s:required
// +k8s:customValidation # cpu and memory limits are both set
- Resources *Resources `protobuf:"bytes,7,opt,name=resources,proto3" json:"resources,omitempty"`
+ Resources *Resources `protobuf:"bytes,6,opt,name=resources,proto3" json:"resources,omitempty"`
// +k8s:optional
- Status *ActorTemplateStatus `protobuf:"bytes,8,opt,name=status,proto3" json:"status,omitempty"`
+ Status *ActorTemplateStatus `protobuf:"bytes,7,opt,name=status,proto3" json:"status,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *ActorTemplate) Reset() {
*x = ActorTemplate{}
- mi := &file_ateapi_proto_msgTypes[24]
+ mi := &file_ateapi_proto_msgTypes[25]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -2680,7 +2734,7 @@ func (x *ActorTemplate) String() string {
func (*ActorTemplate) ProtoMessage() {}
func (x *ActorTemplate) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[24]
+ mi := &file_ateapi_proto_msgTypes[25]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -2693,7 +2747,7 @@ func (x *ActorTemplate) ProtoReflect() protoreflect.Message {
// Deprecated: Use ActorTemplate.ProtoReflect.Descriptor instead.
func (*ActorTemplate) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{24}
+ return file_ateapi_proto_rawDescGZIP(), []int{25}
}
func (x *ActorTemplate) GetMetadata() *ResourceMetadata {
@@ -2762,7 +2816,7 @@ type Resources struct {
func (x *Resources) Reset() {
*x = Resources{}
- mi := &file_ateapi_proto_msgTypes[25]
+ mi := &file_ateapi_proto_msgTypes[26]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -2774,7 +2828,7 @@ func (x *Resources) String() string {
func (*Resources) ProtoMessage() {}
func (x *Resources) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[25]
+ mi := &file_ateapi_proto_msgTypes[26]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -2787,7 +2841,7 @@ func (x *Resources) ProtoReflect() protoreflect.Message {
// Deprecated: Use Resources.ProtoReflect.Descriptor instead.
func (*Resources) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{25}
+ return file_ateapi_proto_rawDescGZIP(), []int{26}
}
func (x *Resources) GetLimits() []*Limits {
@@ -2815,7 +2869,7 @@ type Limits struct {
func (x *Limits) Reset() {
*x = Limits{}
- mi := &file_ateapi_proto_msgTypes[26]
+ mi := &file_ateapi_proto_msgTypes[27]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -2827,7 +2881,7 @@ func (x *Limits) String() string {
func (*Limits) ProtoMessage() {}
func (x *Limits) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[26]
+ mi := &file_ateapi_proto_msgTypes[27]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -2840,7 +2894,7 @@ func (x *Limits) ProtoReflect() protoreflect.Message {
// Deprecated: Use Limits.ProtoReflect.Descriptor instead.
func (*Limits) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{26}
+ return file_ateapi_proto_rawDescGZIP(), []int{27}
}
func (x *Limits) GetName() string {
@@ -2881,7 +2935,7 @@ type GoldenSnapshotStatus struct {
func (x *GoldenSnapshotStatus) Reset() {
*x = GoldenSnapshotStatus{}
- mi := &file_ateapi_proto_msgTypes[27]
+ mi := &file_ateapi_proto_msgTypes[28]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -2893,7 +2947,7 @@ func (x *GoldenSnapshotStatus) String() string {
func (*GoldenSnapshotStatus) ProtoMessage() {}
func (x *GoldenSnapshotStatus) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[27]
+ mi := &file_ateapi_proto_msgTypes[28]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -2906,7 +2960,7 @@ func (x *GoldenSnapshotStatus) ProtoReflect() protoreflect.Message {
// Deprecated: Use GoldenSnapshotStatus.ProtoReflect.Descriptor instead.
func (*GoldenSnapshotStatus) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{27}
+ return file_ateapi_proto_rawDescGZIP(), []int{28}
}
func (x *GoldenSnapshotStatus) GetGoldenTag() *ObjectRef {
@@ -2940,7 +2994,7 @@ type ActorTemplateStatus struct {
func (x *ActorTemplateStatus) Reset() {
*x = ActorTemplateStatus{}
- mi := &file_ateapi_proto_msgTypes[28]
+ mi := &file_ateapi_proto_msgTypes[29]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -2952,7 +3006,7 @@ func (x *ActorTemplateStatus) String() string {
func (*ActorTemplateStatus) ProtoMessage() {}
func (x *ActorTemplateStatus) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[28]
+ mi := &file_ateapi_proto_msgTypes[29]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -2965,7 +3019,7 @@ func (x *ActorTemplateStatus) ProtoReflect() protoreflect.Message {
// Deprecated: Use ActorTemplateStatus.ProtoReflect.Descriptor instead.
func (*ActorTemplateStatus) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{28}
+ return file_ateapi_proto_rawDescGZIP(), []int{29}
}
func (x *ActorTemplateStatus) GetGoldenSnapshotStatus() *GoldenSnapshotStatus {
@@ -2980,9 +3034,12 @@ func (x *ActorTemplateStatus) GetGoldenSnapshotStatus() *GoldenSnapshotStatus {
type SnapshotConfig struct {
state protoimpl.MessageState `protogen:"open.v1"`
// storage_location is the base object-storage URI snapshots of actors on
- // this version are stored under. Required.
+ // this version are stored under. If unset, ate-api-server's
+ // --snapshot-location is used. Requires the PerTemplateSnapshotLocation preview
+ // gate; ignored without it.
//
- // +k8s:required
+ // +ate:preview=PerTemplateSnapshotLocation
+ // +k8s:optional
// +k8s:maxLength=1024
// +k8s:customValidation # Validate URI
StorageLocation string `protobuf:"bytes,1,opt,name=storage_location,json=storageLocation,proto3" json:"storage_location,omitempty"`
@@ -3002,14 +3059,14 @@ type SnapshotConfig struct {
// Golden snapshots are enabled when this config is omitted.
//
// +k8s:optional
- GoldenSnapshotConfig *GoldenSnapshotConfig `protobuf:"bytes,5,opt,name=golden_snapshot_config,json=goldenSnapshotConfig,proto3" json:"golden_snapshot_config,omitempty"`
+ GoldenSnapshotConfig *GoldenSnapshotConfig `protobuf:"bytes,3,opt,name=golden_snapshot_config,json=goldenSnapshotConfig,proto3" json:"golden_snapshot_config,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *SnapshotConfig) Reset() {
*x = SnapshotConfig{}
- mi := &file_ateapi_proto_msgTypes[29]
+ mi := &file_ateapi_proto_msgTypes[30]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -3021,7 +3078,7 @@ func (x *SnapshotConfig) String() string {
func (*SnapshotConfig) ProtoMessage() {}
func (x *SnapshotConfig) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[29]
+ mi := &file_ateapi_proto_msgTypes[30]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -3034,7 +3091,7 @@ func (x *SnapshotConfig) ProtoReflect() protoreflect.Message {
// Deprecated: Use SnapshotConfig.ProtoReflect.Descriptor instead.
func (*SnapshotConfig) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{29}
+ return file_ateapi_proto_rawDescGZIP(), []int{30}
}
func (x *SnapshotConfig) GetStorageLocation() string {
@@ -3074,7 +3131,7 @@ type GoldenSnapshotConfig struct {
func (x *GoldenSnapshotConfig) Reset() {
*x = GoldenSnapshotConfig{}
- mi := &file_ateapi_proto_msgTypes[30]
+ mi := &file_ateapi_proto_msgTypes[31]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -3086,7 +3143,7 @@ func (x *GoldenSnapshotConfig) String() string {
func (*GoldenSnapshotConfig) ProtoMessage() {}
func (x *GoldenSnapshotConfig) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[30]
+ mi := &file_ateapi_proto_msgTypes[31]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -3099,7 +3156,7 @@ func (x *GoldenSnapshotConfig) ProtoReflect() protoreflect.Message {
// Deprecated: Use GoldenSnapshotConfig.ProtoReflect.Descriptor instead.
func (*GoldenSnapshotConfig) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{30}
+ return file_ateapi_proto_rawDescGZIP(), []int{31}
}
func (x *GoldenSnapshotConfig) GetMode() GoldenSnapshotMode {
@@ -3179,7 +3236,7 @@ type Container struct {
func (x *Container) Reset() {
*x = Container{}
- mi := &file_ateapi_proto_msgTypes[31]
+ mi := &file_ateapi_proto_msgTypes[32]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -3191,7 +3248,7 @@ func (x *Container) String() string {
func (*Container) ProtoMessage() {}
func (x *Container) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[31]
+ mi := &file_ateapi_proto_msgTypes[32]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -3204,7 +3261,7 @@ func (x *Container) ProtoReflect() protoreflect.Message {
// Deprecated: Use Container.ProtoReflect.Descriptor instead.
func (*Container) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{31}
+ return file_ateapi_proto_rawDescGZIP(), []int{32}
}
func (x *Container) GetName() string {
@@ -3284,7 +3341,7 @@ type SecurityContext struct {
func (x *SecurityContext) Reset() {
*x = SecurityContext{}
- mi := &file_ateapi_proto_msgTypes[32]
+ mi := &file_ateapi_proto_msgTypes[33]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -3296,7 +3353,7 @@ func (x *SecurityContext) String() string {
func (*SecurityContext) ProtoMessage() {}
func (x *SecurityContext) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[32]
+ mi := &file_ateapi_proto_msgTypes[33]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -3309,7 +3366,7 @@ func (x *SecurityContext) ProtoReflect() protoreflect.Message {
// Deprecated: Use SecurityContext.ProtoReflect.Descriptor instead.
func (*SecurityContext) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{32}
+ return file_ateapi_proto_rawDescGZIP(), []int{33}
}
func (x *SecurityContext) GetCapabilities() *Capabilities {
@@ -3347,7 +3404,7 @@ type Capabilities struct {
func (x *Capabilities) Reset() {
*x = Capabilities{}
- mi := &file_ateapi_proto_msgTypes[33]
+ mi := &file_ateapi_proto_msgTypes[34]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -3359,7 +3416,7 @@ func (x *Capabilities) String() string {
func (*Capabilities) ProtoMessage() {}
func (x *Capabilities) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[33]
+ mi := &file_ateapi_proto_msgTypes[34]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -3372,7 +3429,7 @@ func (x *Capabilities) ProtoReflect() protoreflect.Message {
// Deprecated: Use Capabilities.ProtoReflect.Descriptor instead.
func (*Capabilities) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{33}
+ return file_ateapi_proto_rawDescGZIP(), []int{34}
}
func (x *Capabilities) GetAdd() []string {
@@ -3410,7 +3467,7 @@ type EnvVar struct {
func (x *EnvVar) Reset() {
*x = EnvVar{}
- mi := &file_ateapi_proto_msgTypes[34]
+ mi := &file_ateapi_proto_msgTypes[35]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -3422,7 +3479,7 @@ func (x *EnvVar) String() string {
func (*EnvVar) ProtoMessage() {}
func (x *EnvVar) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[34]
+ mi := &file_ateapi_proto_msgTypes[35]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -3435,7 +3492,7 @@ func (x *EnvVar) ProtoReflect() protoreflect.Message {
// Deprecated: Use EnvVar.ProtoReflect.Descriptor instead.
func (*EnvVar) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{34}
+ return file_ateapi_proto_rawDescGZIP(), []int{35}
}
func (x *EnvVar) GetName() string {
@@ -3475,7 +3532,7 @@ type WakeupProbe struct {
func (x *WakeupProbe) Reset() {
*x = WakeupProbe{}
- mi := &file_ateapi_proto_msgTypes[35]
+ mi := &file_ateapi_proto_msgTypes[36]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -3487,7 +3544,7 @@ func (x *WakeupProbe) String() string {
func (*WakeupProbe) ProtoMessage() {}
func (x *WakeupProbe) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[35]
+ mi := &file_ateapi_proto_msgTypes[36]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -3500,7 +3557,7 @@ func (x *WakeupProbe) ProtoReflect() protoreflect.Message {
// Deprecated: Use WakeupProbe.ProtoReflect.Descriptor instead.
func (*WakeupProbe) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{35}
+ return file_ateapi_proto_rawDescGZIP(), []int{36}
}
func (x *WakeupProbe) GetHttpGet() *HttpGetAction {
@@ -3540,7 +3597,7 @@ type HttpGetAction struct {
func (x *HttpGetAction) Reset() {
*x = HttpGetAction{}
- mi := &file_ateapi_proto_msgTypes[36]
+ mi := &file_ateapi_proto_msgTypes[37]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -3552,7 +3609,7 @@ func (x *HttpGetAction) String() string {
func (*HttpGetAction) ProtoMessage() {}
func (x *HttpGetAction) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[36]
+ mi := &file_ateapi_proto_msgTypes[37]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -3565,7 +3622,7 @@ func (x *HttpGetAction) ProtoReflect() protoreflect.Message {
// Deprecated: Use HttpGetAction.ProtoReflect.Descriptor instead.
func (*HttpGetAction) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{36}
+ return file_ateapi_proto_rawDescGZIP(), []int{37}
}
func (x *HttpGetAction) GetPath() string {
@@ -3589,32 +3646,34 @@ type Volume struct {
// +k8s:required
// +k8s:format=k8s-short-name
Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"`
- // Exactly one of durable_dir / external_volume_template / image /
- // system_info must be set.
+ // Exactly one of durable_dir / external / image / system_info must be
+ // set.
//
// +k8s:optional
// +k8s:unionMember
DurableDir *DurableDirVolumeSource `protobuf:"bytes,2,opt,name=durable_dir,json=durableDir,proto3" json:"durable_dir,omitempty"`
+ // external is a CSI volume: provisioned for this actor from a template.
+ //
// +ate:preview=ExternalVolumes
// +k8s:ifDisabled(ExternalVolumes)=+k8s:forbidden
// +k8s:optional
// +k8s:unionMember
- ExternalVolumeTemplate *ExternalVolumeTemplate `protobuf:"bytes,3,opt,name=external_volume_template,json=externalVolumeTemplate,proto3" json:"external_volume_template,omitempty"`
+ External *ExternalVolumeSource `protobuf:"bytes,3,opt,name=external,proto3" json:"external,omitempty"`
// +k8s:optional
// +k8s:unionMember
- SystemInfo *SystemInfoVolumeSource `protobuf:"bytes,5,opt,name=system_info,json=systemInfo,proto3" json:"system_info,omitempty"`
+ SystemInfo *SystemInfoVolumeSource `protobuf:"bytes,4,opt,name=system_info,json=systemInfo,proto3" json:"system_info,omitempty"`
// image mounts the contents of an OCI image, read-only.
//
// +k8s:optional
// +k8s:unionMember
- Image *ImageVolumeSource `protobuf:"bytes,6,opt,name=image,proto3" json:"image,omitempty"`
+ Image *ImageVolumeSource `protobuf:"bytes,5,opt,name=image,proto3" json:"image,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *Volume) Reset() {
*x = Volume{}
- mi := &file_ateapi_proto_msgTypes[37]
+ mi := &file_ateapi_proto_msgTypes[38]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -3626,7 +3685,7 @@ func (x *Volume) String() string {
func (*Volume) ProtoMessage() {}
func (x *Volume) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[37]
+ mi := &file_ateapi_proto_msgTypes[38]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -3639,7 +3698,7 @@ func (x *Volume) ProtoReflect() protoreflect.Message {
// Deprecated: Use Volume.ProtoReflect.Descriptor instead.
func (*Volume) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{37}
+ return file_ateapi_proto_rawDescGZIP(), []int{38}
}
func (x *Volume) GetName() string {
@@ -3656,9 +3715,9 @@ func (x *Volume) GetDurableDir() *DurableDirVolumeSource {
return nil
}
-func (x *Volume) GetExternalVolumeTemplate() *ExternalVolumeTemplate {
+func (x *Volume) GetExternal() *ExternalVolumeSource {
if x != nil {
- return x.ExternalVolumeTemplate
+ return x.External
}
return nil
}
@@ -3677,25 +3736,75 @@ func (x *Volume) GetImage() *ImageVolumeSource {
return nil
}
-// ImageVolumeSource mounts the contents of an OCI image, read-only. The
-// reference must include a digest: changing the image invalidates
-// snapshots.
-type ImageVolumeSource struct {
+// ExternalVolumeSource is a CSI volume provisioned for the actor from a template.
+type ExternalVolumeSource struct {
state protoimpl.MessageState `protogen:"open.v1"`
- // reference is the OCI image reference. Must include a digest
- // (e.g. "name@sha256:...").
+ // template provisions a volume for this actor alone; it is deleted with the
+ // actor.
//
- // +k8s:required
- // +k8s:maxLength=512
- // +k8s:customValidation # must be a well-formed image reference, pinned by digest
- Reference string `protobuf:"bytes,1,opt,name=reference,proto3" json:"reference,omitempty"`
+ // +k8s:optional
+ // +k8s:unionMember
+ Template *ExternalVolumeTemplate `protobuf:"bytes,1,opt,name=template,proto3" json:"template,omitempty"`
+ unknownFields protoimpl.UnknownFields
+ sizeCache protoimpl.SizeCache
+}
+
+func (x *ExternalVolumeSource) Reset() {
+ *x = ExternalVolumeSource{}
+ mi := &file_ateapi_proto_msgTypes[39]
+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+ ms.StoreMessageInfo(mi)
+}
+
+func (x *ExternalVolumeSource) String() string {
+ return protoimpl.X.MessageStringOf(x)
+}
+
+func (*ExternalVolumeSource) ProtoMessage() {}
+
+func (x *ExternalVolumeSource) ProtoReflect() protoreflect.Message {
+ mi := &file_ateapi_proto_msgTypes[39]
+ if x != nil {
+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+ if ms.LoadMessageInfo() == nil {
+ ms.StoreMessageInfo(mi)
+ }
+ return ms
+ }
+ return mi.MessageOf(x)
+}
+
+// Deprecated: Use ExternalVolumeSource.ProtoReflect.Descriptor instead.
+func (*ExternalVolumeSource) Descriptor() ([]byte, []int) {
+ return file_ateapi_proto_rawDescGZIP(), []int{39}
+}
+
+func (x *ExternalVolumeSource) GetTemplate() *ExternalVolumeTemplate {
+ if x != nil {
+ return x.Template
+ }
+ return nil
+}
+
+// ImageVolumeSource mounts the contents of an OCI image, read-only. The
+// reference must include a digest: changing the image invalidates
+// snapshots.
+type ImageVolumeSource struct {
+ state protoimpl.MessageState `protogen:"open.v1"`
+ // reference is the OCI image reference. Must include a digest
+ // (e.g. "name@sha256:...").
+ //
+ // +k8s:required
+ // +k8s:maxLength=512
+ // +k8s:customValidation # must be a well-formed image reference, pinned by digest
+ Reference string `protobuf:"bytes,1,opt,name=reference,proto3" json:"reference,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *ImageVolumeSource) Reset() {
*x = ImageVolumeSource{}
- mi := &file_ateapi_proto_msgTypes[38]
+ mi := &file_ateapi_proto_msgTypes[40]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -3707,7 +3816,7 @@ func (x *ImageVolumeSource) String() string {
func (*ImageVolumeSource) ProtoMessage() {}
func (x *ImageVolumeSource) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[38]
+ mi := &file_ateapi_proto_msgTypes[40]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -3720,7 +3829,7 @@ func (x *ImageVolumeSource) ProtoReflect() protoreflect.Message {
// Deprecated: Use ImageVolumeSource.ProtoReflect.Descriptor instead.
func (*ImageVolumeSource) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{38}
+ return file_ateapi_proto_rawDescGZIP(), []int{40}
}
func (x *ImageVolumeSource) GetReference() string {
@@ -3740,7 +3849,7 @@ type DurableDirVolumeSource struct {
func (x *DurableDirVolumeSource) Reset() {
*x = DurableDirVolumeSource{}
- mi := &file_ateapi_proto_msgTypes[39]
+ mi := &file_ateapi_proto_msgTypes[41]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -3752,7 +3861,7 @@ func (x *DurableDirVolumeSource) String() string {
func (*DurableDirVolumeSource) ProtoMessage() {}
func (x *DurableDirVolumeSource) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[39]
+ mi := &file_ateapi_proto_msgTypes[41]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -3765,7 +3874,7 @@ func (x *DurableDirVolumeSource) ProtoReflect() protoreflect.Message {
// Deprecated: Use DurableDirVolumeSource.ProtoReflect.Descriptor instead.
func (*DurableDirVolumeSource) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{39}
+ return file_ateapi_proto_rawDescGZIP(), []int{41}
}
// ExternalVolumeTemplate provisions an external volume per actor; the volume
@@ -3791,7 +3900,7 @@ type ExternalVolumeTemplate struct {
func (x *ExternalVolumeTemplate) Reset() {
*x = ExternalVolumeTemplate{}
- mi := &file_ateapi_proto_msgTypes[40]
+ mi := &file_ateapi_proto_msgTypes[42]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -3803,7 +3912,7 @@ func (x *ExternalVolumeTemplate) String() string {
func (*ExternalVolumeTemplate) ProtoMessage() {}
func (x *ExternalVolumeTemplate) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[40]
+ mi := &file_ateapi_proto_msgTypes[42]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -3816,7 +3925,7 @@ func (x *ExternalVolumeTemplate) ProtoReflect() protoreflect.Message {
// Deprecated: Use ExternalVolumeTemplate.ProtoReflect.Descriptor instead.
func (*ExternalVolumeTemplate) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{40}
+ return file_ateapi_proto_rawDescGZIP(), []int{42}
}
func (x *ExternalVolumeTemplate) GetCapacity() string {
@@ -3853,7 +3962,7 @@ type SystemInfoVolumeSource struct {
func (x *SystemInfoVolumeSource) Reset() {
*x = SystemInfoVolumeSource{}
- mi := &file_ateapi_proto_msgTypes[41]
+ mi := &file_ateapi_proto_msgTypes[43]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -3865,7 +3974,7 @@ func (x *SystemInfoVolumeSource) String() string {
func (*SystemInfoVolumeSource) ProtoMessage() {}
func (x *SystemInfoVolumeSource) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[41]
+ mi := &file_ateapi_proto_msgTypes[43]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -3878,7 +3987,7 @@ func (x *SystemInfoVolumeSource) ProtoReflect() protoreflect.Message {
// Deprecated: Use SystemInfoVolumeSource.ProtoReflect.Descriptor instead.
func (*SystemInfoVolumeSource) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{41}
+ return file_ateapi_proto_rawDescGZIP(), []int{43}
}
func (x *SystemInfoVolumeSource) GetDataSources() []*SystemInfoDataSource {
@@ -3904,7 +4013,7 @@ type SystemInfoDataSource struct {
func (x *SystemInfoDataSource) Reset() {
*x = SystemInfoDataSource{}
- mi := &file_ateapi_proto_msgTypes[42]
+ mi := &file_ateapi_proto_msgTypes[44]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -3916,7 +4025,7 @@ func (x *SystemInfoDataSource) String() string {
func (*SystemInfoDataSource) ProtoMessage() {}
func (x *SystemInfoDataSource) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[42]
+ mi := &file_ateapi_proto_msgTypes[44]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -3929,7 +4038,7 @@ func (x *SystemInfoDataSource) ProtoReflect() protoreflect.Message {
// Deprecated: Use SystemInfoDataSource.ProtoReflect.Descriptor instead.
func (*SystemInfoDataSource) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{42}
+ return file_ateapi_proto_rawDescGZIP(), []int{44}
}
func (x *SystemInfoDataSource) GetActorMetadata() *ActorMetadataDataSource {
@@ -3966,7 +4075,7 @@ type ActorMetadataDataSource struct {
func (x *ActorMetadataDataSource) Reset() {
*x = ActorMetadataDataSource{}
- mi := &file_ateapi_proto_msgTypes[43]
+ mi := &file_ateapi_proto_msgTypes[45]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -3978,7 +4087,7 @@ func (x *ActorMetadataDataSource) String() string {
func (*ActorMetadataDataSource) ProtoMessage() {}
func (x *ActorMetadataDataSource) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[43]
+ mi := &file_ateapi_proto_msgTypes[45]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -3991,7 +4100,7 @@ func (x *ActorMetadataDataSource) ProtoReflect() protoreflect.Message {
// Deprecated: Use ActorMetadataDataSource.ProtoReflect.Descriptor instead.
func (*ActorMetadataDataSource) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{43}
+ return file_ateapi_proto_rawDescGZIP(), []int{45}
}
func (x *ActorMetadataDataSource) GetItems() []*ActorMetadataItem {
@@ -4023,7 +4132,7 @@ type ActorMetadataItem struct {
func (x *ActorMetadataItem) Reset() {
*x = ActorMetadataItem{}
- mi := &file_ateapi_proto_msgTypes[44]
+ mi := &file_ateapi_proto_msgTypes[46]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -4035,7 +4144,7 @@ func (x *ActorMetadataItem) String() string {
func (*ActorMetadataItem) ProtoMessage() {}
func (x *ActorMetadataItem) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[44]
+ mi := &file_ateapi_proto_msgTypes[46]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -4048,7 +4157,7 @@ func (x *ActorMetadataItem) ProtoReflect() protoreflect.Message {
// Deprecated: Use ActorMetadataItem.ProtoReflect.Descriptor instead.
func (*ActorMetadataItem) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{44}
+ return file_ateapi_proto_rawDescGZIP(), []int{46}
}
func (x *ActorMetadataItem) GetField() ActorMetadataField {
@@ -4085,7 +4194,7 @@ type TrustBundleDataSource struct {
// +k8s:eachVal=+k8s:minLength=1
// +k8s:eachVal=+k8s:maxLength=253
// +k8s:listType=set
- Names []string `protobuf:"bytes,3,rep,name=names,proto3" json:"names,omitempty"`
+ Names []string `protobuf:"bytes,1,rep,name=names,proto3" json:"names,omitempty"`
// path must be a clean relative Unix path: at most 16 '/'-separated
// segments, none of them empty, '.' or '..', and no NUL byte.
//
@@ -4100,7 +4209,7 @@ type TrustBundleDataSource struct {
func (x *TrustBundleDataSource) Reset() {
*x = TrustBundleDataSource{}
- mi := &file_ateapi_proto_msgTypes[45]
+ mi := &file_ateapi_proto_msgTypes[47]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -4112,7 +4221,7 @@ func (x *TrustBundleDataSource) String() string {
func (*TrustBundleDataSource) ProtoMessage() {}
func (x *TrustBundleDataSource) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[45]
+ mi := &file_ateapi_proto_msgTypes[47]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -4125,7 +4234,7 @@ func (x *TrustBundleDataSource) ProtoReflect() protoreflect.Message {
// Deprecated: Use TrustBundleDataSource.ProtoReflect.Descriptor instead.
func (*TrustBundleDataSource) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{45}
+ return file_ateapi_proto_rawDescGZIP(), []int{47}
}
func (x *TrustBundleDataSource) GetNames() []string {
@@ -4164,7 +4273,7 @@ type VolumeMount struct {
func (x *VolumeMount) Reset() {
*x = VolumeMount{}
- mi := &file_ateapi_proto_msgTypes[46]
+ mi := &file_ateapi_proto_msgTypes[48]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -4176,7 +4285,7 @@ func (x *VolumeMount) String() string {
func (*VolumeMount) ProtoMessage() {}
func (x *VolumeMount) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[46]
+ mi := &file_ateapi_proto_msgTypes[48]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -4189,7 +4298,7 @@ func (x *VolumeMount) ProtoReflect() protoreflect.Message {
// Deprecated: Use VolumeMount.ProtoReflect.Descriptor instead.
func (*VolumeMount) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{46}
+ return file_ateapi_proto_rawDescGZIP(), []int{48}
}
func (x *VolumeMount) GetName() string {
@@ -4218,7 +4327,7 @@ type CreateAtespaceRequest struct {
func (x *CreateAtespaceRequest) Reset() {
*x = CreateAtespaceRequest{}
- mi := &file_ateapi_proto_msgTypes[47]
+ mi := &file_ateapi_proto_msgTypes[49]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -4230,7 +4339,7 @@ func (x *CreateAtespaceRequest) String() string {
func (*CreateAtespaceRequest) ProtoMessage() {}
func (x *CreateAtespaceRequest) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[47]
+ mi := &file_ateapi_proto_msgTypes[49]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -4243,7 +4352,7 @@ func (x *CreateAtespaceRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use CreateAtespaceRequest.ProtoReflect.Descriptor instead.
func (*CreateAtespaceRequest) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{47}
+ return file_ateapi_proto_rawDescGZIP(), []int{49}
}
func (x *CreateAtespaceRequest) GetAtespace() *Atespace {
@@ -4264,7 +4373,7 @@ type GetAtespaceRequest struct {
func (x *GetAtespaceRequest) Reset() {
*x = GetAtespaceRequest{}
- mi := &file_ateapi_proto_msgTypes[48]
+ mi := &file_ateapi_proto_msgTypes[50]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -4276,7 +4385,7 @@ func (x *GetAtespaceRequest) String() string {
func (*GetAtespaceRequest) ProtoMessage() {}
func (x *GetAtespaceRequest) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[48]
+ mi := &file_ateapi_proto_msgTypes[50]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -4289,7 +4398,7 @@ func (x *GetAtespaceRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use GetAtespaceRequest.ProtoReflect.Descriptor instead.
func (*GetAtespaceRequest) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{48}
+ return file_ateapi_proto_rawDescGZIP(), []int{50}
}
func (x *GetAtespaceRequest) GetAtespace() *ObjectRef {
@@ -4311,7 +4420,7 @@ type ListAtespacesRequest struct {
func (x *ListAtespacesRequest) Reset() {
*x = ListAtespacesRequest{}
- mi := &file_ateapi_proto_msgTypes[49]
+ mi := &file_ateapi_proto_msgTypes[51]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -4323,7 +4432,7 @@ func (x *ListAtespacesRequest) String() string {
func (*ListAtespacesRequest) ProtoMessage() {}
func (x *ListAtespacesRequest) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[49]
+ mi := &file_ateapi_proto_msgTypes[51]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -4336,7 +4445,7 @@ func (x *ListAtespacesRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use ListAtespacesRequest.ProtoReflect.Descriptor instead.
func (*ListAtespacesRequest) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{49}
+ return file_ateapi_proto_rawDescGZIP(), []int{51}
}
func (x *ListAtespacesRequest) GetOptions() *ListOptions {
@@ -4358,7 +4467,7 @@ type ListAtespacesResponse struct {
func (x *ListAtespacesResponse) Reset() {
*x = ListAtespacesResponse{}
- mi := &file_ateapi_proto_msgTypes[50]
+ mi := &file_ateapi_proto_msgTypes[52]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -4370,7 +4479,7 @@ func (x *ListAtespacesResponse) String() string {
func (*ListAtespacesResponse) ProtoMessage() {}
func (x *ListAtespacesResponse) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[50]
+ mi := &file_ateapi_proto_msgTypes[52]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -4383,7 +4492,7 @@ func (x *ListAtespacesResponse) ProtoReflect() protoreflect.Message {
// Deprecated: Use ListAtespacesResponse.ProtoReflect.Descriptor instead.
func (*ListAtespacesResponse) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{50}
+ return file_ateapi_proto_rawDescGZIP(), []int{52}
}
func (x *ListAtespacesResponse) GetAtespaces() []*Atespace {
@@ -4415,7 +4524,7 @@ type DeleteAtespaceRequest struct {
func (x *DeleteAtespaceRequest) Reset() {
*x = DeleteAtespaceRequest{}
- mi := &file_ateapi_proto_msgTypes[51]
+ mi := &file_ateapi_proto_msgTypes[53]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -4427,7 +4536,7 @@ func (x *DeleteAtespaceRequest) String() string {
func (*DeleteAtespaceRequest) ProtoMessage() {}
func (x *DeleteAtespaceRequest) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[51]
+ mi := &file_ateapi_proto_msgTypes[53]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -4440,7 +4549,7 @@ func (x *DeleteAtespaceRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use DeleteAtespaceRequest.ProtoReflect.Descriptor instead.
func (*DeleteAtespaceRequest) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{51}
+ return file_ateapi_proto_rawDescGZIP(), []int{53}
}
func (x *DeleteAtespaceRequest) GetAtespace() *ObjectRef {
@@ -4471,7 +4580,7 @@ type CreateActorTemplateRequest struct {
func (x *CreateActorTemplateRequest) Reset() {
*x = CreateActorTemplateRequest{}
- mi := &file_ateapi_proto_msgTypes[52]
+ mi := &file_ateapi_proto_msgTypes[54]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -4483,7 +4592,7 @@ func (x *CreateActorTemplateRequest) String() string {
func (*CreateActorTemplateRequest) ProtoMessage() {}
func (x *CreateActorTemplateRequest) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[52]
+ mi := &file_ateapi_proto_msgTypes[54]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -4496,7 +4605,7 @@ func (x *CreateActorTemplateRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use CreateActorTemplateRequest.ProtoReflect.Descriptor instead.
func (*CreateActorTemplateRequest) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{52}
+ return file_ateapi_proto_rawDescGZIP(), []int{54}
}
func (x *CreateActorTemplateRequest) GetActorTemplate() *ActorTemplate {
@@ -4517,7 +4626,7 @@ type GetActorTemplateRequest struct {
func (x *GetActorTemplateRequest) Reset() {
*x = GetActorTemplateRequest{}
- mi := &file_ateapi_proto_msgTypes[53]
+ mi := &file_ateapi_proto_msgTypes[55]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -4529,7 +4638,7 @@ func (x *GetActorTemplateRequest) String() string {
func (*GetActorTemplateRequest) ProtoMessage() {}
func (x *GetActorTemplateRequest) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[53]
+ mi := &file_ateapi_proto_msgTypes[55]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -4542,7 +4651,7 @@ func (x *GetActorTemplateRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use GetActorTemplateRequest.ProtoReflect.Descriptor instead.
func (*GetActorTemplateRequest) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{53}
+ return file_ateapi_proto_rawDescGZIP(), []int{55}
}
func (x *GetActorTemplateRequest) GetActorTemplate() *ObjectRef {
@@ -4570,7 +4679,7 @@ type ListActorTemplatesRequest struct {
func (x *ListActorTemplatesRequest) Reset() {
*x = ListActorTemplatesRequest{}
- mi := &file_ateapi_proto_msgTypes[54]
+ mi := &file_ateapi_proto_msgTypes[56]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -4582,7 +4691,7 @@ func (x *ListActorTemplatesRequest) String() string {
func (*ListActorTemplatesRequest) ProtoMessage() {}
func (x *ListActorTemplatesRequest) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[54]
+ mi := &file_ateapi_proto_msgTypes[56]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -4595,7 +4704,7 @@ func (x *ListActorTemplatesRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use ListActorTemplatesRequest.ProtoReflect.Descriptor instead.
func (*ListActorTemplatesRequest) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{54}
+ return file_ateapi_proto_rawDescGZIP(), []int{56}
}
func (x *ListActorTemplatesRequest) GetAtespace() string {
@@ -4625,7 +4734,7 @@ type ListActorTemplatesResponse struct {
func (x *ListActorTemplatesResponse) Reset() {
*x = ListActorTemplatesResponse{}
- mi := &file_ateapi_proto_msgTypes[55]
+ mi := &file_ateapi_proto_msgTypes[57]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -4637,7 +4746,7 @@ func (x *ListActorTemplatesResponse) String() string {
func (*ListActorTemplatesResponse) ProtoMessage() {}
func (x *ListActorTemplatesResponse) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[55]
+ mi := &file_ateapi_proto_msgTypes[57]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -4650,7 +4759,7 @@ func (x *ListActorTemplatesResponse) ProtoReflect() protoreflect.Message {
// Deprecated: Use ListActorTemplatesResponse.ProtoReflect.Descriptor instead.
func (*ListActorTemplatesResponse) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{55}
+ return file_ateapi_proto_rawDescGZIP(), []int{57}
}
func (x *ListActorTemplatesResponse) GetActorTemplates() []*ActorTemplate {
@@ -4682,7 +4791,7 @@ type DeleteActorTemplateRequest struct {
func (x *DeleteActorTemplateRequest) Reset() {
*x = DeleteActorTemplateRequest{}
- mi := &file_ateapi_proto_msgTypes[56]
+ mi := &file_ateapi_proto_msgTypes[58]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -4694,7 +4803,7 @@ func (x *DeleteActorTemplateRequest) String() string {
func (*DeleteActorTemplateRequest) ProtoMessage() {}
func (x *DeleteActorTemplateRequest) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[56]
+ mi := &file_ateapi_proto_msgTypes[58]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -4707,7 +4816,7 @@ func (x *DeleteActorTemplateRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use DeleteActorTemplateRequest.ProtoReflect.Descriptor instead.
func (*DeleteActorTemplateRequest) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{56}
+ return file_ateapi_proto_rawDescGZIP(), []int{58}
}
func (x *DeleteActorTemplateRequest) GetActorTemplate() *ObjectRef {
@@ -4735,7 +4844,7 @@ type GetActorRequest struct {
func (x *GetActorRequest) Reset() {
*x = GetActorRequest{}
- mi := &file_ateapi_proto_msgTypes[57]
+ mi := &file_ateapi_proto_msgTypes[59]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -4747,7 +4856,7 @@ func (x *GetActorRequest) String() string {
func (*GetActorRequest) ProtoMessage() {}
func (x *GetActorRequest) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[57]
+ mi := &file_ateapi_proto_msgTypes[59]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -4760,7 +4869,7 @@ func (x *GetActorRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use GetActorRequest.ProtoReflect.Descriptor instead.
func (*GetActorRequest) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{57}
+ return file_ateapi_proto_rawDescGZIP(), []int{59}
}
func (x *GetActorRequest) GetActor() *ObjectRef {
@@ -4783,7 +4892,7 @@ type CreateActorRequest struct {
func (x *CreateActorRequest) Reset() {
*x = CreateActorRequest{}
- mi := &file_ateapi_proto_msgTypes[58]
+ mi := &file_ateapi_proto_msgTypes[60]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -4795,7 +4904,7 @@ func (x *CreateActorRequest) String() string {
func (*CreateActorRequest) ProtoMessage() {}
func (x *CreateActorRequest) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[58]
+ mi := &file_ateapi_proto_msgTypes[60]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -4808,7 +4917,7 @@ func (x *CreateActorRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use CreateActorRequest.ProtoReflect.Descriptor instead.
func (*CreateActorRequest) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{58}
+ return file_ateapi_proto_rawDescGZIP(), []int{60}
}
func (x *CreateActorRequest) GetActor() *Actor {
@@ -4839,7 +4948,7 @@ type UpdateActorRequest struct {
func (x *UpdateActorRequest) Reset() {
*x = UpdateActorRequest{}
- mi := &file_ateapi_proto_msgTypes[59]
+ mi := &file_ateapi_proto_msgTypes[61]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -4851,7 +4960,7 @@ func (x *UpdateActorRequest) String() string {
func (*UpdateActorRequest) ProtoMessage() {}
func (x *UpdateActorRequest) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[59]
+ mi := &file_ateapi_proto_msgTypes[61]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -4864,7 +4973,7 @@ func (x *UpdateActorRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use UpdateActorRequest.ProtoReflect.Descriptor instead.
func (*UpdateActorRequest) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{59}
+ return file_ateapi_proto_rawDescGZIP(), []int{61}
}
func (x *UpdateActorRequest) GetActor() *Actor {
@@ -4889,7 +4998,7 @@ type SuspendActorRequest struct {
func (x *SuspendActorRequest) Reset() {
*x = SuspendActorRequest{}
- mi := &file_ateapi_proto_msgTypes[60]
+ mi := &file_ateapi_proto_msgTypes[62]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -4901,7 +5010,7 @@ func (x *SuspendActorRequest) String() string {
func (*SuspendActorRequest) ProtoMessage() {}
func (x *SuspendActorRequest) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[60]
+ mi := &file_ateapi_proto_msgTypes[62]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -4914,7 +5023,7 @@ func (x *SuspendActorRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use SuspendActorRequest.ProtoReflect.Descriptor instead.
func (*SuspendActorRequest) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{60}
+ return file_ateapi_proto_rawDescGZIP(), []int{62}
}
func (x *SuspendActorRequest) GetActor() *ObjectRef {
@@ -4940,7 +5049,7 @@ type SuspendActorResponse struct {
func (x *SuspendActorResponse) Reset() {
*x = SuspendActorResponse{}
- mi := &file_ateapi_proto_msgTypes[61]
+ mi := &file_ateapi_proto_msgTypes[63]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -4952,7 +5061,7 @@ func (x *SuspendActorResponse) String() string {
func (*SuspendActorResponse) ProtoMessage() {}
func (x *SuspendActorResponse) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[61]
+ mi := &file_ateapi_proto_msgTypes[63]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -4965,7 +5074,7 @@ func (x *SuspendActorResponse) ProtoReflect() protoreflect.Message {
// Deprecated: Use SuspendActorResponse.ProtoReflect.Descriptor instead.
func (*SuspendActorResponse) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{61}
+ return file_ateapi_proto_rawDescGZIP(), []int{63}
}
func (x *SuspendActorResponse) GetActor() *Actor {
@@ -4991,7 +5100,7 @@ type SuspendOptions struct {
func (x *SuspendOptions) Reset() {
*x = SuspendOptions{}
- mi := &file_ateapi_proto_msgTypes[62]
+ mi := &file_ateapi_proto_msgTypes[64]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -5003,7 +5112,7 @@ func (x *SuspendOptions) String() string {
func (*SuspendOptions) ProtoMessage() {}
func (x *SuspendOptions) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[62]
+ mi := &file_ateapi_proto_msgTypes[64]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -5016,7 +5125,7 @@ func (x *SuspendOptions) ProtoReflect() protoreflect.Message {
// Deprecated: Use SuspendOptions.ProtoReflect.Descriptor instead.
func (*SuspendOptions) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{62}
+ return file_ateapi_proto_rawDescGZIP(), []int{64}
}
func (x *SuspendOptions) GetMode() DurabilityMode {
@@ -5037,7 +5146,7 @@ type ResumeActorRequest struct {
func (x *ResumeActorRequest) Reset() {
*x = ResumeActorRequest{}
- mi := &file_ateapi_proto_msgTypes[63]
+ mi := &file_ateapi_proto_msgTypes[65]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -5049,7 +5158,7 @@ func (x *ResumeActorRequest) String() string {
func (*ResumeActorRequest) ProtoMessage() {}
func (x *ResumeActorRequest) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[63]
+ mi := &file_ateapi_proto_msgTypes[65]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -5062,7 +5171,7 @@ func (x *ResumeActorRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use ResumeActorRequest.ProtoReflect.Descriptor instead.
func (*ResumeActorRequest) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{63}
+ return file_ateapi_proto_rawDescGZIP(), []int{65}
}
func (x *ResumeActorRequest) GetActor() *ObjectRef {
@@ -5084,7 +5193,7 @@ type ResumeActorResponse struct {
func (x *ResumeActorResponse) Reset() {
*x = ResumeActorResponse{}
- mi := &file_ateapi_proto_msgTypes[64]
+ mi := &file_ateapi_proto_msgTypes[66]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -5096,7 +5205,7 @@ func (x *ResumeActorResponse) String() string {
func (*ResumeActorResponse) ProtoMessage() {}
func (x *ResumeActorResponse) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[64]
+ mi := &file_ateapi_proto_msgTypes[66]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -5109,7 +5218,7 @@ func (x *ResumeActorResponse) ProtoReflect() protoreflect.Message {
// Deprecated: Use ResumeActorResponse.ProtoReflect.Descriptor instead.
func (*ResumeActorResponse) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{64}
+ return file_ateapi_proto_rawDescGZIP(), []int{66}
}
func (x *ResumeActorResponse) GetActor() *Actor {
@@ -5139,7 +5248,7 @@ type RevertActorRequest struct {
func (x *RevertActorRequest) Reset() {
*x = RevertActorRequest{}
- mi := &file_ateapi_proto_msgTypes[65]
+ mi := &file_ateapi_proto_msgTypes[67]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -5151,7 +5260,7 @@ func (x *RevertActorRequest) String() string {
func (*RevertActorRequest) ProtoMessage() {}
func (x *RevertActorRequest) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[65]
+ mi := &file_ateapi_proto_msgTypes[67]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -5164,7 +5273,7 @@ func (x *RevertActorRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use RevertActorRequest.ProtoReflect.Descriptor instead.
func (*RevertActorRequest) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{65}
+ return file_ateapi_proto_rawDescGZIP(), []int{67}
}
func (x *RevertActorRequest) GetActor() *ObjectRef {
@@ -5186,7 +5295,7 @@ type RevertActorResponse struct {
func (x *RevertActorResponse) Reset() {
*x = RevertActorResponse{}
- mi := &file_ateapi_proto_msgTypes[66]
+ mi := &file_ateapi_proto_msgTypes[68]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -5198,7 +5307,7 @@ func (x *RevertActorResponse) String() string {
func (*RevertActorResponse) ProtoMessage() {}
func (x *RevertActorResponse) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[66]
+ mi := &file_ateapi_proto_msgTypes[68]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -5211,7 +5320,7 @@ func (x *RevertActorResponse) ProtoReflect() protoreflect.Message {
// Deprecated: Use RevertActorResponse.ProtoReflect.Descriptor instead.
func (*RevertActorResponse) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{66}
+ return file_ateapi_proto_rawDescGZIP(), []int{68}
}
func (x *RevertActorResponse) GetActor() *Actor {
@@ -5226,21 +5335,17 @@ type DeleteActorRequest struct {
// +k8s:required
// +k8s:subfield(atespace)=+k8s:required
Actor *ObjectRef `protobuf:"bytes,1,opt,name=actor,proto3" json:"actor,omitempty"`
- // If true, delete the actor regardless of its state.
- //
- // +k8s:optional
- AnyState bool `protobuf:"varint,2,opt,name=any_state,json=anyState,proto3" json:"any_state,omitempty"`
// Optional per-delete preconditions.
//
// +k8s:optional
- Options *DeleteOptions `protobuf:"bytes,3,opt,name=options,proto3" json:"options,omitempty"`
+ Options *DeleteOptions `protobuf:"bytes,2,opt,name=options,proto3" json:"options,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *DeleteActorRequest) Reset() {
*x = DeleteActorRequest{}
- mi := &file_ateapi_proto_msgTypes[67]
+ mi := &file_ateapi_proto_msgTypes[69]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -5252,7 +5357,7 @@ func (x *DeleteActorRequest) String() string {
func (*DeleteActorRequest) ProtoMessage() {}
func (x *DeleteActorRequest) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[67]
+ mi := &file_ateapi_proto_msgTypes[69]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -5265,7 +5370,7 @@ func (x *DeleteActorRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use DeleteActorRequest.ProtoReflect.Descriptor instead.
func (*DeleteActorRequest) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{67}
+ return file_ateapi_proto_rawDescGZIP(), []int{69}
}
func (x *DeleteActorRequest) GetActor() *ObjectRef {
@@ -5275,13 +5380,6 @@ func (x *DeleteActorRequest) GetActor() *ObjectRef {
return nil
}
-func (x *DeleteActorRequest) GetAnyState() bool {
- if x != nil {
- return x.AnyState
- }
- return false
-}
-
func (x *DeleteActorRequest) GetOptions() *DeleteOptions {
if x != nil {
return x.Options
@@ -5309,7 +5407,7 @@ type GetActorEgressPolicyRequest struct {
func (x *GetActorEgressPolicyRequest) Reset() {
*x = GetActorEgressPolicyRequest{}
- mi := &file_ateapi_proto_msgTypes[68]
+ mi := &file_ateapi_proto_msgTypes[70]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -5321,7 +5419,7 @@ func (x *GetActorEgressPolicyRequest) String() string {
func (*GetActorEgressPolicyRequest) ProtoMessage() {}
func (x *GetActorEgressPolicyRequest) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[68]
+ mi := &file_ateapi_proto_msgTypes[70]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -5334,7 +5432,7 @@ func (x *GetActorEgressPolicyRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use GetActorEgressPolicyRequest.ProtoReflect.Descriptor instead.
func (*GetActorEgressPolicyRequest) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{68}
+ return file_ateapi_proto_rawDescGZIP(), []int{70}
}
func (x *GetActorEgressPolicyRequest) GetActor() *ObjectRef {
@@ -5372,7 +5470,7 @@ type CreateActorEgressPolicyRequest struct {
func (x *CreateActorEgressPolicyRequest) Reset() {
*x = CreateActorEgressPolicyRequest{}
- mi := &file_ateapi_proto_msgTypes[69]
+ mi := &file_ateapi_proto_msgTypes[71]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -5384,7 +5482,7 @@ func (x *CreateActorEgressPolicyRequest) String() string {
func (*CreateActorEgressPolicyRequest) ProtoMessage() {}
func (x *CreateActorEgressPolicyRequest) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[69]
+ mi := &file_ateapi_proto_msgTypes[71]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -5397,7 +5495,7 @@ func (x *CreateActorEgressPolicyRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use CreateActorEgressPolicyRequest.ProtoReflect.Descriptor instead.
func (*CreateActorEgressPolicyRequest) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{69}
+ return file_ateapi_proto_rawDescGZIP(), []int{71}
}
func (x *CreateActorEgressPolicyRequest) GetActor() *ObjectRef {
@@ -5435,7 +5533,7 @@ type UpdateActorEgressPolicyRequest struct {
func (x *UpdateActorEgressPolicyRequest) Reset() {
*x = UpdateActorEgressPolicyRequest{}
- mi := &file_ateapi_proto_msgTypes[70]
+ mi := &file_ateapi_proto_msgTypes[72]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -5447,7 +5545,7 @@ func (x *UpdateActorEgressPolicyRequest) String() string {
func (*UpdateActorEgressPolicyRequest) ProtoMessage() {}
func (x *UpdateActorEgressPolicyRequest) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[70]
+ mi := &file_ateapi_proto_msgTypes[72]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -5460,7 +5558,7 @@ func (x *UpdateActorEgressPolicyRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use UpdateActorEgressPolicyRequest.ProtoReflect.Descriptor instead.
func (*UpdateActorEgressPolicyRequest) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{70}
+ return file_ateapi_proto_rawDescGZIP(), []int{72}
}
func (x *UpdateActorEgressPolicyRequest) GetActor() *ObjectRef {
@@ -5501,7 +5599,7 @@ type DeleteActorEgressPolicyRequest struct {
func (x *DeleteActorEgressPolicyRequest) Reset() {
*x = DeleteActorEgressPolicyRequest{}
- mi := &file_ateapi_proto_msgTypes[71]
+ mi := &file_ateapi_proto_msgTypes[73]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -5513,7 +5611,7 @@ func (x *DeleteActorEgressPolicyRequest) String() string {
func (*DeleteActorEgressPolicyRequest) ProtoMessage() {}
func (x *DeleteActorEgressPolicyRequest) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[71]
+ mi := &file_ateapi_proto_msgTypes[73]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -5526,7 +5624,7 @@ func (x *DeleteActorEgressPolicyRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use DeleteActorEgressPolicyRequest.ProtoReflect.Descriptor instead.
func (*DeleteActorEgressPolicyRequest) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{71}
+ return file_ateapi_proto_rawDescGZIP(), []int{73}
}
func (x *DeleteActorEgressPolicyRequest) GetActor() *ObjectRef {
@@ -5561,7 +5659,7 @@ type GetTagRequest struct {
func (x *GetTagRequest) Reset() {
*x = GetTagRequest{}
- mi := &file_ateapi_proto_msgTypes[72]
+ mi := &file_ateapi_proto_msgTypes[74]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -5573,7 +5671,7 @@ func (x *GetTagRequest) String() string {
func (*GetTagRequest) ProtoMessage() {}
func (x *GetTagRequest) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[72]
+ mi := &file_ateapi_proto_msgTypes[74]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -5586,7 +5684,7 @@ func (x *GetTagRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use GetTagRequest.ProtoReflect.Descriptor instead.
func (*GetTagRequest) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{72}
+ return file_ateapi_proto_rawDescGZIP(), []int{74}
}
func (x *GetTagRequest) GetTag() *ObjectRef {
@@ -5604,7 +5702,7 @@ type MintActorJwtRequest struct {
// Must be a valid actor that currently exists according to the actor store.
//
// +k8s:required
- Actor *ObjectRef `protobuf:"bytes,5,opt,name=actor,proto3" json:"actor,omitempty"`
+ Actor *ObjectRef `protobuf:"bytes,1,opt,name=actor,proto3" json:"actor,omitempty"`
// The audiences the minted JWT is bound to. Tokens are only issued with
// audience bindings, so at least one is required.
//
@@ -5612,20 +5710,20 @@ type MintActorJwtRequest struct {
// +k8s:maxItems=16 # guardrail; tokens realistically bind a handful of audiences
// +k8s:listType=set
// +k8s:eachVal=+k8s:maxLength=512 # audiences are caller-defined URIs; bound only
- Audiences []string `protobuf:"bytes,1,rep,name=audiences,proto3" json:"audiences,omitempty"`
+ Audiences []string `protobuf:"bytes,2,rep,name=audiences,proto3" json:"audiences,omitempty"`
// Lifetime of the JWT, in seconds.
//
// +k8s:required
// +k8s:minimum=300
// +k8s:maximum=3600
- ExpirationSeconds int64 `protobuf:"varint,8,opt,name=expiration_seconds,json=expirationSeconds,proto3" json:"expiration_seconds,omitempty"`
+ ExpirationSeconds int64 `protobuf:"varint,3,opt,name=expiration_seconds,json=expirationSeconds,proto3" json:"expiration_seconds,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *MintActorJwtRequest) Reset() {
*x = MintActorJwtRequest{}
- mi := &file_ateapi_proto_msgTypes[73]
+ mi := &file_ateapi_proto_msgTypes[75]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -5637,7 +5735,7 @@ func (x *MintActorJwtRequest) String() string {
func (*MintActorJwtRequest) ProtoMessage() {}
func (x *MintActorJwtRequest) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[73]
+ mi := &file_ateapi_proto_msgTypes[75]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -5650,7 +5748,7 @@ func (x *MintActorJwtRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use MintActorJwtRequest.ProtoReflect.Descriptor instead.
func (*MintActorJwtRequest) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{73}
+ return file_ateapi_proto_rawDescGZIP(), []int{75}
}
func (x *MintActorJwtRequest) GetActor() *ObjectRef {
@@ -5705,7 +5803,7 @@ type MintActorJwtResponse struct {
func (x *MintActorJwtResponse) Reset() {
*x = MintActorJwtResponse{}
- mi := &file_ateapi_proto_msgTypes[74]
+ mi := &file_ateapi_proto_msgTypes[76]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -5717,7 +5815,7 @@ func (x *MintActorJwtResponse) String() string {
func (*MintActorJwtResponse) ProtoMessage() {}
func (x *MintActorJwtResponse) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[74]
+ mi := &file_ateapi_proto_msgTypes[76]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -5730,7 +5828,7 @@ func (x *MintActorJwtResponse) ProtoReflect() protoreflect.Message {
// Deprecated: Use MintActorJwtResponse.ProtoReflect.Descriptor instead.
func (*MintActorJwtResponse) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{74}
+ return file_ateapi_proto_rawDescGZIP(), []int{76}
}
func (x *MintActorJwtResponse) GetActorJwt() string {
@@ -5755,27 +5853,27 @@ type MintActorCertificateRequest struct {
// Must be a valid actor that currently exists according to the actor store.
//
// +k8s:required
- Actor *ObjectRef `protobuf:"bytes,6,opt,name=actor,proto3" json:"actor,omitempty"`
+ Actor *ObjectRef `protobuf:"bytes,1,opt,name=actor,proto3" json:"actor,omitempty"`
// The UID of the actor --- used to guard against deletion and recreation of
// an actor with the same name.
//
// +k8s:required
// +k8s:format=k8s-uuid
- ActorUid string `protobuf:"bytes,7,opt,name=actor_uid,json=actorUid,proto3" json:"actor_uid,omitempty"`
+ ActorUid string `protobuf:"bytes,2,opt,name=actor_uid,json=actorUid,proto3" json:"actor_uid,omitempty"`
// Request contains DER encoded bytes of a x509 certificate signing request.
// The signer will ignore the contents of the CSR except to extract the
// subject public key.
//
// +k8s:required
// +k8s:maxBytes=16384
- CertificateSigningRequest []byte `protobuf:"bytes,2,opt,name=certificate_signing_request,json=certificateSigningRequest,proto3" json:"certificate_signing_request,omitempty"`
+ CertificateSigningRequest []byte `protobuf:"bytes,3,opt,name=certificate_signing_request,json=certificateSigningRequest,proto3" json:"certificate_signing_request,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *MintActorCertificateRequest) Reset() {
*x = MintActorCertificateRequest{}
- mi := &file_ateapi_proto_msgTypes[75]
+ mi := &file_ateapi_proto_msgTypes[77]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -5787,7 +5885,7 @@ func (x *MintActorCertificateRequest) String() string {
func (*MintActorCertificateRequest) ProtoMessage() {}
func (x *MintActorCertificateRequest) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[75]
+ mi := &file_ateapi_proto_msgTypes[77]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -5800,7 +5898,7 @@ func (x *MintActorCertificateRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use MintActorCertificateRequest.ProtoReflect.Descriptor instead.
func (*MintActorCertificateRequest) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{75}
+ return file_ateapi_proto_rawDescGZIP(), []int{77}
}
func (x *MintActorCertificateRequest) GetActor() *ObjectRef {
@@ -5837,7 +5935,7 @@ type MintActorCertificateResponse struct {
func (x *MintActorCertificateResponse) Reset() {
*x = MintActorCertificateResponse{}
- mi := &file_ateapi_proto_msgTypes[76]
+ mi := &file_ateapi_proto_msgTypes[78]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -5849,7 +5947,7 @@ func (x *MintActorCertificateResponse) String() string {
func (*MintActorCertificateResponse) ProtoMessage() {}
func (x *MintActorCertificateResponse) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[76]
+ mi := &file_ateapi_proto_msgTypes[78]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -5862,7 +5960,7 @@ func (x *MintActorCertificateResponse) ProtoReflect() protoreflect.Message {
// Deprecated: Use MintActorCertificateResponse.ProtoReflect.Descriptor instead.
func (*MintActorCertificateResponse) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{76}
+ return file_ateapi_proto_rawDescGZIP(), []int{78}
}
func (x *MintActorCertificateResponse) GetActorCertificates() [][]byte {
@@ -5889,7 +5987,7 @@ type ListTagsRequest struct {
func (x *ListTagsRequest) Reset() {
*x = ListTagsRequest{}
- mi := &file_ateapi_proto_msgTypes[77]
+ mi := &file_ateapi_proto_msgTypes[79]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -5901,7 +5999,7 @@ func (x *ListTagsRequest) String() string {
func (*ListTagsRequest) ProtoMessage() {}
func (x *ListTagsRequest) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[77]
+ mi := &file_ateapi_proto_msgTypes[79]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -5914,7 +6012,7 @@ func (x *ListTagsRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use ListTagsRequest.ProtoReflect.Descriptor instead.
func (*ListTagsRequest) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{77}
+ return file_ateapi_proto_rawDescGZIP(), []int{79}
}
func (x *ListTagsRequest) GetAtespace() string {
@@ -5941,7 +6039,7 @@ type ListTagsResponse struct {
func (x *ListTagsResponse) Reset() {
*x = ListTagsResponse{}
- mi := &file_ateapi_proto_msgTypes[78]
+ mi := &file_ateapi_proto_msgTypes[80]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -5953,7 +6051,7 @@ func (x *ListTagsResponse) String() string {
func (*ListTagsResponse) ProtoMessage() {}
func (x *ListTagsResponse) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[78]
+ mi := &file_ateapi_proto_msgTypes[80]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -5966,7 +6064,7 @@ func (x *ListTagsResponse) ProtoReflect() protoreflect.Message {
// Deprecated: Use ListTagsResponse.ProtoReflect.Descriptor instead.
func (*ListTagsResponse) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{78}
+ return file_ateapi_proto_rawDescGZIP(), []int{80}
}
func (x *ListTagsResponse) GetTags() []*Tag {
@@ -6011,7 +6109,7 @@ type CreateTagRequest struct {
func (x *CreateTagRequest) Reset() {
*x = CreateTagRequest{}
- mi := &file_ateapi_proto_msgTypes[79]
+ mi := &file_ateapi_proto_msgTypes[81]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -6023,7 +6121,7 @@ func (x *CreateTagRequest) String() string {
func (*CreateTagRequest) ProtoMessage() {}
func (x *CreateTagRequest) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[79]
+ mi := &file_ateapi_proto_msgTypes[81]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -6036,7 +6134,7 @@ func (x *CreateTagRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use CreateTagRequest.ProtoReflect.Descriptor instead.
func (*CreateTagRequest) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{79}
+ return file_ateapi_proto_rawDescGZIP(), []int{81}
}
func (x *CreateTagRequest) GetTag() *Tag {
@@ -6066,7 +6164,7 @@ type UpdateTagRequest struct {
func (x *UpdateTagRequest) Reset() {
*x = UpdateTagRequest{}
- mi := &file_ateapi_proto_msgTypes[80]
+ mi := &file_ateapi_proto_msgTypes[82]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -6078,7 +6176,7 @@ func (x *UpdateTagRequest) String() string {
func (*UpdateTagRequest) ProtoMessage() {}
func (x *UpdateTagRequest) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[80]
+ mi := &file_ateapi_proto_msgTypes[82]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -6091,7 +6189,7 @@ func (x *UpdateTagRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use UpdateTagRequest.ProtoReflect.Descriptor instead.
func (*UpdateTagRequest) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{80}
+ return file_ateapi_proto_rawDescGZIP(), []int{82}
}
func (x *UpdateTagRequest) GetTag() *Tag {
@@ -6116,7 +6214,7 @@ type DeleteTagRequest struct {
func (x *DeleteTagRequest) Reset() {
*x = DeleteTagRequest{}
- mi := &file_ateapi_proto_msgTypes[81]
+ mi := &file_ateapi_proto_msgTypes[83]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -6128,7 +6226,7 @@ func (x *DeleteTagRequest) String() string {
func (*DeleteTagRequest) ProtoMessage() {}
func (x *DeleteTagRequest) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[81]
+ mi := &file_ateapi_proto_msgTypes[83]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -6141,7 +6239,7 @@ func (x *DeleteTagRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use DeleteTagRequest.ProtoReflect.Descriptor instead.
func (*DeleteTagRequest) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{81}
+ return file_ateapi_proto_rawDescGZIP(), []int{83}
}
func (x *DeleteTagRequest) GetTag() *ObjectRef {
@@ -6180,7 +6278,7 @@ type DeleteOptions struct {
func (x *DeleteOptions) Reset() {
*x = DeleteOptions{}
- mi := &file_ateapi_proto_msgTypes[82]
+ mi := &file_ateapi_proto_msgTypes[84]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -6192,7 +6290,7 @@ func (x *DeleteOptions) String() string {
func (*DeleteOptions) ProtoMessage() {}
func (x *DeleteOptions) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[82]
+ mi := &file_ateapi_proto_msgTypes[84]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -6205,7 +6303,7 @@ func (x *DeleteOptions) ProtoReflect() protoreflect.Message {
// Deprecated: Use DeleteOptions.ProtoReflect.Descriptor instead.
func (*DeleteOptions) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{82}
+ return file_ateapi_proto_rawDescGZIP(), []int{84}
}
func (x *DeleteOptions) GetVersion() int64 {
@@ -6244,7 +6342,7 @@ type ListOptions struct {
func (x *ListOptions) Reset() {
*x = ListOptions{}
- mi := &file_ateapi_proto_msgTypes[83]
+ mi := &file_ateapi_proto_msgTypes[85]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -6256,7 +6354,7 @@ func (x *ListOptions) String() string {
func (*ListOptions) ProtoMessage() {}
func (x *ListOptions) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[83]
+ mi := &file_ateapi_proto_msgTypes[85]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -6269,7 +6367,7 @@ func (x *ListOptions) ProtoReflect() protoreflect.Message {
// Deprecated: Use ListOptions.ProtoReflect.Descriptor instead.
func (*ListOptions) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{83}
+ return file_ateapi_proto_rawDescGZIP(), []int{85}
}
func (x *ListOptions) GetPageSize() int32 {
@@ -6305,7 +6403,7 @@ type ListWorkerActorAssignmentsRequest struct {
func (x *ListWorkerActorAssignmentsRequest) Reset() {
*x = ListWorkerActorAssignmentsRequest{}
- mi := &file_ateapi_proto_msgTypes[84]
+ mi := &file_ateapi_proto_msgTypes[86]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -6317,7 +6415,7 @@ func (x *ListWorkerActorAssignmentsRequest) String() string {
func (*ListWorkerActorAssignmentsRequest) ProtoMessage() {}
func (x *ListWorkerActorAssignmentsRequest) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[84]
+ mi := &file_ateapi_proto_msgTypes[86]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -6330,7 +6428,7 @@ func (x *ListWorkerActorAssignmentsRequest) ProtoReflect() protoreflect.Message
// Deprecated: Use ListWorkerActorAssignmentsRequest.ProtoReflect.Descriptor instead.
func (*ListWorkerActorAssignmentsRequest) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{84}
+ return file_ateapi_proto_rawDescGZIP(), []int{86}
}
func (x *ListWorkerActorAssignmentsRequest) GetWorker() *ObjectRef {
@@ -6360,7 +6458,7 @@ type ListWorkerActorAssignmentsResponse struct {
func (x *ListWorkerActorAssignmentsResponse) Reset() {
*x = ListWorkerActorAssignmentsResponse{}
- mi := &file_ateapi_proto_msgTypes[85]
+ mi := &file_ateapi_proto_msgTypes[87]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -6372,7 +6470,7 @@ func (x *ListWorkerActorAssignmentsResponse) String() string {
func (*ListWorkerActorAssignmentsResponse) ProtoMessage() {}
func (x *ListWorkerActorAssignmentsResponse) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[85]
+ mi := &file_ateapi_proto_msgTypes[87]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -6385,7 +6483,7 @@ func (x *ListWorkerActorAssignmentsResponse) ProtoReflect() protoreflect.Message
// Deprecated: Use ListWorkerActorAssignmentsResponse.ProtoReflect.Descriptor instead.
func (*ListWorkerActorAssignmentsResponse) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{85}
+ return file_ateapi_proto_rawDescGZIP(), []int{87}
}
func (x *ListWorkerActorAssignmentsResponse) GetActorAssignments() []*ActorAssignment {
@@ -6414,7 +6512,7 @@ type ListWorkersRequest struct {
func (x *ListWorkersRequest) Reset() {
*x = ListWorkersRequest{}
- mi := &file_ateapi_proto_msgTypes[86]
+ mi := &file_ateapi_proto_msgTypes[88]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -6426,7 +6524,7 @@ func (x *ListWorkersRequest) String() string {
func (*ListWorkersRequest) ProtoMessage() {}
func (x *ListWorkersRequest) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[86]
+ mi := &file_ateapi_proto_msgTypes[88]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -6439,7 +6537,7 @@ func (x *ListWorkersRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use ListWorkersRequest.ProtoReflect.Descriptor instead.
func (*ListWorkersRequest) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{86}
+ return file_ateapi_proto_rawDescGZIP(), []int{88}
}
func (x *ListWorkersRequest) GetOptions() *ListOptions {
@@ -6461,7 +6559,7 @@ type ListWorkersResponse struct {
func (x *ListWorkersResponse) Reset() {
*x = ListWorkersResponse{}
- mi := &file_ateapi_proto_msgTypes[87]
+ mi := &file_ateapi_proto_msgTypes[89]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -6473,7 +6571,7 @@ func (x *ListWorkersResponse) String() string {
func (*ListWorkersResponse) ProtoMessage() {}
func (x *ListWorkersResponse) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[87]
+ mi := &file_ateapi_proto_msgTypes[89]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -6486,7 +6584,7 @@ func (x *ListWorkersResponse) ProtoReflect() protoreflect.Message {
// Deprecated: Use ListWorkersResponse.ProtoReflect.Descriptor instead.
func (*ListWorkersResponse) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{87}
+ return file_ateapi_proto_rawDescGZIP(), []int{89}
}
func (x *ListWorkersResponse) GetWorkers() []*Worker {
@@ -6516,7 +6614,7 @@ type GetWorkerRequest struct {
func (x *GetWorkerRequest) Reset() {
*x = GetWorkerRequest{}
- mi := &file_ateapi_proto_msgTypes[88]
+ mi := &file_ateapi_proto_msgTypes[90]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -6528,7 +6626,7 @@ func (x *GetWorkerRequest) String() string {
func (*GetWorkerRequest) ProtoMessage() {}
func (x *GetWorkerRequest) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[88]
+ mi := &file_ateapi_proto_msgTypes[90]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -6541,7 +6639,7 @@ func (x *GetWorkerRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use GetWorkerRequest.ProtoReflect.Descriptor instead.
func (*GetWorkerRequest) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{88}
+ return file_ateapi_proto_rawDescGZIP(), []int{90}
}
func (x *GetWorkerRequest) GetWorker() *ObjectRef {
@@ -6563,7 +6661,7 @@ type CreateWorkerRequest struct {
func (x *CreateWorkerRequest) Reset() {
*x = CreateWorkerRequest{}
- mi := &file_ateapi_proto_msgTypes[89]
+ mi := &file_ateapi_proto_msgTypes[91]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -6575,7 +6673,7 @@ func (x *CreateWorkerRequest) String() string {
func (*CreateWorkerRequest) ProtoMessage() {}
func (x *CreateWorkerRequest) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[89]
+ mi := &file_ateapi_proto_msgTypes[91]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -6588,7 +6686,7 @@ func (x *CreateWorkerRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use CreateWorkerRequest.ProtoReflect.Descriptor instead.
func (*CreateWorkerRequest) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{89}
+ return file_ateapi_proto_rawDescGZIP(), []int{91}
}
func (x *CreateWorkerRequest) GetWorker() *Worker {
@@ -6623,7 +6721,7 @@ type UpdateWorkerRequest struct {
func (x *UpdateWorkerRequest) Reset() {
*x = UpdateWorkerRequest{}
- mi := &file_ateapi_proto_msgTypes[90]
+ mi := &file_ateapi_proto_msgTypes[92]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -6635,7 +6733,7 @@ func (x *UpdateWorkerRequest) String() string {
func (*UpdateWorkerRequest) ProtoMessage() {}
func (x *UpdateWorkerRequest) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[90]
+ mi := &file_ateapi_proto_msgTypes[92]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -6648,7 +6746,7 @@ func (x *UpdateWorkerRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use UpdateWorkerRequest.ProtoReflect.Descriptor instead.
func (*UpdateWorkerRequest) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{90}
+ return file_ateapi_proto_rawDescGZIP(), []int{92}
}
func (x *UpdateWorkerRequest) GetWorker() *Worker {
@@ -6675,7 +6773,7 @@ type DeleteWorkerRequest struct {
func (x *DeleteWorkerRequest) Reset() {
*x = DeleteWorkerRequest{}
- mi := &file_ateapi_proto_msgTypes[91]
+ mi := &file_ateapi_proto_msgTypes[93]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -6687,7 +6785,7 @@ func (x *DeleteWorkerRequest) String() string {
func (*DeleteWorkerRequest) ProtoMessage() {}
func (x *DeleteWorkerRequest) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[91]
+ mi := &file_ateapi_proto_msgTypes[93]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -6700,7 +6798,7 @@ func (x *DeleteWorkerRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use DeleteWorkerRequest.ProtoReflect.Descriptor instead.
func (*DeleteWorkerRequest) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{91}
+ return file_ateapi_proto_rawDescGZIP(), []int{93}
}
func (x *DeleteWorkerRequest) GetWorker() *ObjectRef {
@@ -6730,7 +6828,7 @@ type DrainWorkerRequest struct {
func (x *DrainWorkerRequest) Reset() {
*x = DrainWorkerRequest{}
- mi := &file_ateapi_proto_msgTypes[92]
+ mi := &file_ateapi_proto_msgTypes[94]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -6742,7 +6840,7 @@ func (x *DrainWorkerRequest) String() string {
func (*DrainWorkerRequest) ProtoMessage() {}
func (x *DrainWorkerRequest) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[92]
+ mi := &file_ateapi_proto_msgTypes[94]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -6755,7 +6853,7 @@ func (x *DrainWorkerRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use DrainWorkerRequest.ProtoReflect.Descriptor instead.
func (*DrainWorkerRequest) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{92}
+ return file_ateapi_proto_rawDescGZIP(), []int{94}
}
func (x *DrainWorkerRequest) GetWorker() *ObjectRef {
@@ -6776,7 +6874,7 @@ type DrainWorkerResponse struct {
func (x *DrainWorkerResponse) Reset() {
*x = DrainWorkerResponse{}
- mi := &file_ateapi_proto_msgTypes[93]
+ mi := &file_ateapi_proto_msgTypes[95]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -6788,7 +6886,7 @@ func (x *DrainWorkerResponse) String() string {
func (*DrainWorkerResponse) ProtoMessage() {}
func (x *DrainWorkerResponse) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[93]
+ mi := &file_ateapi_proto_msgTypes[95]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -6801,7 +6899,7 @@ func (x *DrainWorkerResponse) ProtoReflect() protoreflect.Message {
// Deprecated: Use DrainWorkerResponse.ProtoReflect.Descriptor instead.
func (*DrainWorkerResponse) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{93}
+ return file_ateapi_proto_rawDescGZIP(), []int{95}
}
func (x *DrainWorkerResponse) GetWorker() *Worker {
@@ -6825,7 +6923,7 @@ type MarkWorkerAvailableRequest struct {
func (x *MarkWorkerAvailableRequest) Reset() {
*x = MarkWorkerAvailableRequest{}
- mi := &file_ateapi_proto_msgTypes[94]
+ mi := &file_ateapi_proto_msgTypes[96]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -6837,7 +6935,7 @@ func (x *MarkWorkerAvailableRequest) String() string {
func (*MarkWorkerAvailableRequest) ProtoMessage() {}
func (x *MarkWorkerAvailableRequest) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[94]
+ mi := &file_ateapi_proto_msgTypes[96]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -6850,7 +6948,7 @@ func (x *MarkWorkerAvailableRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use MarkWorkerAvailableRequest.ProtoReflect.Descriptor instead.
func (*MarkWorkerAvailableRequest) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{94}
+ return file_ateapi_proto_rawDescGZIP(), []int{96}
}
func (x *MarkWorkerAvailableRequest) GetWorker() *ObjectRef {
@@ -6871,7 +6969,7 @@ type MarkWorkerAvailableResponse struct {
func (x *MarkWorkerAvailableResponse) Reset() {
*x = MarkWorkerAvailableResponse{}
- mi := &file_ateapi_proto_msgTypes[95]
+ mi := &file_ateapi_proto_msgTypes[97]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -6883,7 +6981,7 @@ func (x *MarkWorkerAvailableResponse) String() string {
func (*MarkWorkerAvailableResponse) ProtoMessage() {}
func (x *MarkWorkerAvailableResponse) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[95]
+ mi := &file_ateapi_proto_msgTypes[97]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -6896,7 +6994,7 @@ func (x *MarkWorkerAvailableResponse) ProtoReflect() protoreflect.Message {
// Deprecated: Use MarkWorkerAvailableResponse.ProtoReflect.Descriptor instead.
func (*MarkWorkerAvailableResponse) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{95}
+ return file_ateapi_proto_rawDescGZIP(), []int{97}
}
func (x *MarkWorkerAvailableResponse) GetWorker() *Worker {
@@ -6920,7 +7018,7 @@ type MarkWorkerUnavailableRequest struct {
func (x *MarkWorkerUnavailableRequest) Reset() {
*x = MarkWorkerUnavailableRequest{}
- mi := &file_ateapi_proto_msgTypes[96]
+ mi := &file_ateapi_proto_msgTypes[98]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -6932,7 +7030,7 @@ func (x *MarkWorkerUnavailableRequest) String() string {
func (*MarkWorkerUnavailableRequest) ProtoMessage() {}
func (x *MarkWorkerUnavailableRequest) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[96]
+ mi := &file_ateapi_proto_msgTypes[98]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -6945,7 +7043,7 @@ func (x *MarkWorkerUnavailableRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use MarkWorkerUnavailableRequest.ProtoReflect.Descriptor instead.
func (*MarkWorkerUnavailableRequest) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{96}
+ return file_ateapi_proto_rawDescGZIP(), []int{98}
}
func (x *MarkWorkerUnavailableRequest) GetWorker() *ObjectRef {
@@ -6966,7 +7064,7 @@ type MarkWorkerUnavailableResponse struct {
func (x *MarkWorkerUnavailableResponse) Reset() {
*x = MarkWorkerUnavailableResponse{}
- mi := &file_ateapi_proto_msgTypes[97]
+ mi := &file_ateapi_proto_msgTypes[99]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -6978,7 +7076,7 @@ func (x *MarkWorkerUnavailableResponse) String() string {
func (*MarkWorkerUnavailableResponse) ProtoMessage() {}
func (x *MarkWorkerUnavailableResponse) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[97]
+ mi := &file_ateapi_proto_msgTypes[99]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -6991,7 +7089,7 @@ func (x *MarkWorkerUnavailableResponse) ProtoReflect() protoreflect.Message {
// Deprecated: Use MarkWorkerUnavailableResponse.ProtoReflect.Descriptor instead.
func (*MarkWorkerUnavailableResponse) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{97}
+ return file_ateapi_proto_rawDescGZIP(), []int{99}
}
func (x *MarkWorkerUnavailableResponse) GetWorker() *Worker {
@@ -7020,7 +7118,7 @@ type ListActorsRequest struct {
func (x *ListActorsRequest) Reset() {
*x = ListActorsRequest{}
- mi := &file_ateapi_proto_msgTypes[98]
+ mi := &file_ateapi_proto_msgTypes[100]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -7032,7 +7130,7 @@ func (x *ListActorsRequest) String() string {
func (*ListActorsRequest) ProtoMessage() {}
func (x *ListActorsRequest) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[98]
+ mi := &file_ateapi_proto_msgTypes[100]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -7045,7 +7143,7 @@ func (x *ListActorsRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use ListActorsRequest.ProtoReflect.Descriptor instead.
func (*ListActorsRequest) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{98}
+ return file_ateapi_proto_rawDescGZIP(), []int{100}
}
func (x *ListActorsRequest) GetAtespace() string {
@@ -7074,7 +7172,7 @@ type ListActorsResponse struct {
func (x *ListActorsResponse) Reset() {
*x = ListActorsResponse{}
- mi := &file_ateapi_proto_msgTypes[99]
+ mi := &file_ateapi_proto_msgTypes[101]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -7086,7 +7184,7 @@ func (x *ListActorsResponse) String() string {
func (*ListActorsResponse) ProtoMessage() {}
func (x *ListActorsResponse) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[99]
+ mi := &file_ateapi_proto_msgTypes[101]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -7099,7 +7197,7 @@ func (x *ListActorsResponse) ProtoReflect() protoreflect.Message {
// Deprecated: Use ListActorsResponse.ProtoReflect.Descriptor instead.
func (*ListActorsResponse) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{99}
+ return file_ateapi_proto_rawDescGZIP(), []int{101}
}
func (x *ListActorsResponse) GetActors() []*Actor {
@@ -7172,14 +7270,14 @@ type Worker struct {
// +k8s:maxLength=2048
// +k8s:customValidation # a SPIFFE ID
// +k8s:immutable
- WorkerSpiffeId string `protobuf:"bytes,13,opt,name=worker_spiffe_id,json=workerSpiffeId,proto3" json:"worker_spiffe_id,omitempty"`
+ WorkerSpiffeId string `protobuf:"bytes,8,opt,name=worker_spiffe_id,json=workerSpiffeId,proto3" json:"worker_spiffe_id,omitempty"`
// sandbox_class mirrors the WorkerPool's default sandbox class
//
// +k8s:optional
// +k8s:maxLength=63
// +k8s:customValidation # gvisor or microvm
// +k8s:immutable
- SandboxClass string `protobuf:"bytes,8,opt,name=sandbox_class,json=sandboxClass,proto3" json:"sandbox_class,omitempty"`
+ SandboxClass string `protobuf:"bytes,9,opt,name=sandbox_class,json=sandboxClass,proto3" json:"sandbox_class,omitempty"`
// labels mirror the WorkerPool object's Kubernetes labels, which selectors
// match against.
// Kubernetes does not cap an object's label count, so the bound below is
@@ -7189,7 +7287,7 @@ type Worker struct {
// +k8s:maxProperties=64
// +k8s:eachKey=+k8s:format=k8s-label-key
// +k8s:eachVal=+k8s:format=k8s-label-value
- Labels map[string]string `protobuf:"bytes,9,rep,name=labels,proto3" json:"labels,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"`
+ Labels map[string]string `protobuf:"bytes,10,rep,name=labels,proto3" json:"labels,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"`
// epoch counts the runs of the worker pod's worker container: 1 for its first
// run and one more for each restart. 0 means it has not been reported.
//
@@ -7202,21 +7300,21 @@ type Worker struct {
// +k8s:minimum=0
// +k8s:update=NoUnset
// +k8s:customValidation
- Epoch int64 `protobuf:"varint,12,opt,name=epoch,proto3" json:"epoch,omitempty"`
+ Epoch int64 `protobuf:"varint,11,opt,name=epoch,proto3" json:"epoch,omitempty"`
// Output-only server-managed state. Absent from Create/Update request
// payloads; whatever a request carries here is ignored. A client moves
// state only through DrainWorker, MarkWorkerAvailable and
// MarkWorkerUnavailable, and the assignments are the scheduler's.
//
// +k8s:optional
- Status *WorkerStatus `protobuf:"bytes,11,opt,name=status,proto3" json:"status,omitempty"`
+ Status *WorkerStatus `protobuf:"bytes,12,opt,name=status,proto3" json:"status,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *Worker) Reset() {
*x = Worker{}
- mi := &file_ateapi_proto_msgTypes[100]
+ mi := &file_ateapi_proto_msgTypes[102]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -7228,7 +7326,7 @@ func (x *Worker) String() string {
func (*Worker) ProtoMessage() {}
func (x *Worker) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[100]
+ mi := &file_ateapi_proto_msgTypes[102]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -7241,7 +7339,7 @@ func (x *Worker) ProtoReflect() protoreflect.Message {
// Deprecated: Use Worker.ProtoReflect.Descriptor instead.
func (*Worker) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{100}
+ return file_ateapi_proto_rawDescGZIP(), []int{102}
}
func (x *Worker) GetMetadata() *ResourceMetadata {
@@ -7377,7 +7475,7 @@ type WorkerStatus struct {
func (x *WorkerStatus) Reset() {
*x = WorkerStatus{}
- mi := &file_ateapi_proto_msgTypes[101]
+ mi := &file_ateapi_proto_msgTypes[103]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -7389,7 +7487,7 @@ func (x *WorkerStatus) String() string {
func (*WorkerStatus) ProtoMessage() {}
func (x *WorkerStatus) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[101]
+ mi := &file_ateapi_proto_msgTypes[103]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -7402,7 +7500,7 @@ func (x *WorkerStatus) ProtoReflect() protoreflect.Message {
// Deprecated: Use WorkerStatus.ProtoReflect.Descriptor instead.
func (*WorkerStatus) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{101}
+ return file_ateapi_proto_rawDescGZIP(), []int{103}
}
func (x *WorkerStatus) GetState() WorkerState {
@@ -7470,7 +7568,7 @@ type SandboxRuntime struct {
func (x *SandboxRuntime) Reset() {
*x = SandboxRuntime{}
- mi := &file_ateapi_proto_msgTypes[102]
+ mi := &file_ateapi_proto_msgTypes[104]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -7482,7 +7580,7 @@ func (x *SandboxRuntime) String() string {
func (*SandboxRuntime) ProtoMessage() {}
func (x *SandboxRuntime) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[102]
+ mi := &file_ateapi_proto_msgTypes[104]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -7495,7 +7593,7 @@ func (x *SandboxRuntime) ProtoReflect() protoreflect.Message {
// Deprecated: Use SandboxRuntime.ProtoReflect.Descriptor instead.
func (*SandboxRuntime) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{102}
+ return file_ateapi_proto_rawDescGZIP(), []int{104}
}
func (x *SandboxRuntime) GetSandboxClass() string {
@@ -7546,7 +7644,7 @@ type VersionedSandboxCompat struct {
func (x *VersionedSandboxCompat) Reset() {
*x = VersionedSandboxCompat{}
- mi := &file_ateapi_proto_msgTypes[103]
+ mi := &file_ateapi_proto_msgTypes[105]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -7558,7 +7656,7 @@ func (x *VersionedSandboxCompat) String() string {
func (*VersionedSandboxCompat) ProtoMessage() {}
func (x *VersionedSandboxCompat) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[103]
+ mi := &file_ateapi_proto_msgTypes[105]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -7571,7 +7669,7 @@ func (x *VersionedSandboxCompat) ProtoReflect() protoreflect.Message {
// Deprecated: Use VersionedSandboxCompat.ProtoReflect.Descriptor instead.
func (*VersionedSandboxCompat) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{103}
+ return file_ateapi_proto_rawDescGZIP(), []int{105}
}
func (x *VersionedSandboxCompat) GetSchemaVersion() string {
@@ -7607,7 +7705,7 @@ type AttributeEntry struct {
func (x *AttributeEntry) Reset() {
*x = AttributeEntry{}
- mi := &file_ateapi_proto_msgTypes[104]
+ mi := &file_ateapi_proto_msgTypes[106]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -7619,7 +7717,7 @@ func (x *AttributeEntry) String() string {
func (*AttributeEntry) ProtoMessage() {}
func (x *AttributeEntry) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[104]
+ mi := &file_ateapi_proto_msgTypes[106]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -7632,7 +7730,7 @@ func (x *AttributeEntry) ProtoReflect() protoreflect.Message {
// Deprecated: Use AttributeEntry.ProtoReflect.Descriptor instead.
func (*AttributeEntry) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{104}
+ return file_ateapi_proto_rawDescGZIP(), []int{106}
}
func (x *AttributeEntry) GetKey() string {
@@ -7671,7 +7769,7 @@ type WorkerResources struct {
func (x *WorkerResources) Reset() {
*x = WorkerResources{}
- mi := &file_ateapi_proto_msgTypes[105]
+ mi := &file_ateapi_proto_msgTypes[107]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -7683,7 +7781,7 @@ func (x *WorkerResources) String() string {
func (*WorkerResources) ProtoMessage() {}
func (x *WorkerResources) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[105]
+ mi := &file_ateapi_proto_msgTypes[107]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -7696,7 +7794,7 @@ func (x *WorkerResources) ProtoReflect() protoreflect.Message {
// Deprecated: Use WorkerResources.ProtoReflect.Descriptor instead.
func (*WorkerResources) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{105}
+ return file_ateapi_proto_rawDescGZIP(), []int{107}
}
func (x *WorkerResources) GetResources() *Resources {
@@ -7725,7 +7823,7 @@ type ActorAssignment struct {
//
// +k8s:required
// +k8s:beta(since: "0.0")=+k8s:subfield(atespace)=+k8s:forbidden # TODO: get rid of beta prefix
- Metadata *ResourceMetadata `protobuf:"bytes,6,opt,name=metadata,proto3" json:"metadata,omitempty"`
+ Metadata *ResourceMetadata `protobuf:"bytes,1,opt,name=metadata,proto3" json:"metadata,omitempty"`
// actor is the Actor that is assigned to the Worker.
//
// +k8s:required
@@ -7737,19 +7835,19 @@ type ActorAssignment struct {
// The resources allocated to the Actor when it was assigned.
//
// +k8s:optional
- Resources *Resources `protobuf:"bytes,5,opt,name=resources,proto3" json:"resources,omitempty"`
+ Resources *Resources `protobuf:"bytes,4,opt,name=resources,proto3" json:"resources,omitempty"`
// worker_epoch is the Worker's epoch when the Actor was bound.
//
// +k8s:optional
// +k8s:minimum=0
- WorkerEpoch int64 `protobuf:"varint,7,opt,name=worker_epoch,json=workerEpoch,proto3" json:"worker_epoch,omitempty"`
+ WorkerEpoch int64 `protobuf:"varint,5,opt,name=worker_epoch,json=workerEpoch,proto3" json:"worker_epoch,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *ActorAssignment) Reset() {
*x = ActorAssignment{}
- mi := &file_ateapi_proto_msgTypes[106]
+ mi := &file_ateapi_proto_msgTypes[108]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -7761,7 +7859,7 @@ func (x *ActorAssignment) String() string {
func (*ActorAssignment) ProtoMessage() {}
func (x *ActorAssignment) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[106]
+ mi := &file_ateapi_proto_msgTypes[108]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -7774,7 +7872,7 @@ func (x *ActorAssignment) ProtoReflect() protoreflect.Message {
// Deprecated: Use ActorAssignment.ProtoReflect.Descriptor instead.
func (*ActorAssignment) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{106}
+ return file_ateapi_proto_rawDescGZIP(), []int{108}
}
func (x *ActorAssignment) GetMetadata() *ResourceMetadata {
@@ -7845,7 +7943,7 @@ type RegisterWorkerRequest struct {
func (x *RegisterWorkerRequest) Reset() {
*x = RegisterWorkerRequest{}
- mi := &file_ateapi_proto_msgTypes[107]
+ mi := &file_ateapi_proto_msgTypes[109]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -7857,7 +7955,7 @@ func (x *RegisterWorkerRequest) String() string {
func (*RegisterWorkerRequest) ProtoMessage() {}
func (x *RegisterWorkerRequest) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[107]
+ mi := &file_ateapi_proto_msgTypes[109]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -7870,7 +7968,7 @@ func (x *RegisterWorkerRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use RegisterWorkerRequest.ProtoReflect.Descriptor instead.
func (*RegisterWorkerRequest) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{107}
+ return file_ateapi_proto_rawDescGZIP(), []int{109}
}
func (x *RegisterWorkerRequest) GetWorker() *ObjectRef {
@@ -7911,7 +8009,7 @@ type RegisterWorkerResponse struct {
func (x *RegisterWorkerResponse) Reset() {
*x = RegisterWorkerResponse{}
- mi := &file_ateapi_proto_msgTypes[108]
+ mi := &file_ateapi_proto_msgTypes[110]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -7923,7 +8021,7 @@ func (x *RegisterWorkerResponse) String() string {
func (*RegisterWorkerResponse) ProtoMessage() {}
func (x *RegisterWorkerResponse) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[108]
+ mi := &file_ateapi_proto_msgTypes[110]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -7936,7 +8034,7 @@ func (x *RegisterWorkerResponse) ProtoReflect() protoreflect.Message {
// Deprecated: Use RegisterWorkerResponse.ProtoReflect.Descriptor instead.
func (*RegisterWorkerResponse) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{108}
+ return file_ateapi_proto_rawDescGZIP(), []int{110}
}
func (x *RegisterWorkerResponse) GetWorker() *Worker {
@@ -7983,7 +8081,7 @@ type MintWorkerActorCertificateRequest struct {
func (x *MintWorkerActorCertificateRequest) Reset() {
*x = MintWorkerActorCertificateRequest{}
- mi := &file_ateapi_proto_msgTypes[109]
+ mi := &file_ateapi_proto_msgTypes[111]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -7995,7 +8093,7 @@ func (x *MintWorkerActorCertificateRequest) String() string {
func (*MintWorkerActorCertificateRequest) ProtoMessage() {}
func (x *MintWorkerActorCertificateRequest) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[109]
+ mi := &file_ateapi_proto_msgTypes[111]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -8008,7 +8106,7 @@ func (x *MintWorkerActorCertificateRequest) ProtoReflect() protoreflect.Message
// Deprecated: Use MintWorkerActorCertificateRequest.ProtoReflect.Descriptor instead.
func (*MintWorkerActorCertificateRequest) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{109}
+ return file_ateapi_proto_rawDescGZIP(), []int{111}
}
func (x *MintWorkerActorCertificateRequest) GetActor() *ObjectRef {
@@ -8052,7 +8150,7 @@ type MintWorkerActorCertificateResponse struct {
func (x *MintWorkerActorCertificateResponse) Reset() {
*x = MintWorkerActorCertificateResponse{}
- mi := &file_ateapi_proto_msgTypes[110]
+ mi := &file_ateapi_proto_msgTypes[112]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -8064,7 +8162,7 @@ func (x *MintWorkerActorCertificateResponse) String() string {
func (*MintWorkerActorCertificateResponse) ProtoMessage() {}
func (x *MintWorkerActorCertificateResponse) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[110]
+ mi := &file_ateapi_proto_msgTypes[112]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -8077,7 +8175,7 @@ func (x *MintWorkerActorCertificateResponse) ProtoReflect() protoreflect.Message
// Deprecated: Use MintWorkerActorCertificateResponse.ProtoReflect.Descriptor instead.
func (*MintWorkerActorCertificateResponse) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{110}
+ return file_ateapi_proto_rawDescGZIP(), []int{112}
}
func (x *MintWorkerActorCertificateResponse) GetActorCertificates() [][]byte {
@@ -8118,7 +8216,7 @@ type MintWorkerActorJwtRequest struct {
func (x *MintWorkerActorJwtRequest) Reset() {
*x = MintWorkerActorJwtRequest{}
- mi := &file_ateapi_proto_msgTypes[111]
+ mi := &file_ateapi_proto_msgTypes[113]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -8130,7 +8228,7 @@ func (x *MintWorkerActorJwtRequest) String() string {
func (*MintWorkerActorJwtRequest) ProtoMessage() {}
func (x *MintWorkerActorJwtRequest) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[111]
+ mi := &file_ateapi_proto_msgTypes[113]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -8143,7 +8241,7 @@ func (x *MintWorkerActorJwtRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use MintWorkerActorJwtRequest.ProtoReflect.Descriptor instead.
func (*MintWorkerActorJwtRequest) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{111}
+ return file_ateapi_proto_rawDescGZIP(), []int{113}
}
func (x *MintWorkerActorJwtRequest) GetActor() *ObjectRef {
@@ -8176,7 +8274,9 @@ type MintWorkerActorJwtResponse struct {
//
// * sub: `worker-for-actor//`.
// * `worker-for-actor.ate.dev`: A JSON object with the actor's `atespace`,
- // `actorName`, and `actorUID`.
+ // `actorName`, and `actorUID`, and the `node` the token was minted for,
+ // the one hosting the actor at that moment. A relying party refuses the
+ // token once the actor has left that node.
//
// A relying party that trusts actor JWTs must not accept this token as the
// actor, and vice versa.
@@ -8189,7 +8289,7 @@ type MintWorkerActorJwtResponse struct {
func (x *MintWorkerActorJwtResponse) Reset() {
*x = MintWorkerActorJwtResponse{}
- mi := &file_ateapi_proto_msgTypes[112]
+ mi := &file_ateapi_proto_msgTypes[114]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -8201,7 +8301,7 @@ func (x *MintWorkerActorJwtResponse) String() string {
func (*MintWorkerActorJwtResponse) ProtoMessage() {}
func (x *MintWorkerActorJwtResponse) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[112]
+ mi := &file_ateapi_proto_msgTypes[114]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -8214,7 +8314,7 @@ func (x *MintWorkerActorJwtResponse) ProtoReflect() protoreflect.Message {
// Deprecated: Use MintWorkerActorJwtResponse.ProtoReflect.Descriptor instead.
func (*MintWorkerActorJwtResponse) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{112}
+ return file_ateapi_proto_rawDescGZIP(), []int{114}
}
func (x *MintWorkerActorJwtResponse) GetActorJwt() string {
@@ -8262,7 +8362,7 @@ type RequestActorSuspendRequest struct {
func (x *RequestActorSuspendRequest) Reset() {
*x = RequestActorSuspendRequest{}
- mi := &file_ateapi_proto_msgTypes[113]
+ mi := &file_ateapi_proto_msgTypes[115]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -8274,7 +8374,7 @@ func (x *RequestActorSuspendRequest) String() string {
func (*RequestActorSuspendRequest) ProtoMessage() {}
func (x *RequestActorSuspendRequest) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[113]
+ mi := &file_ateapi_proto_msgTypes[115]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -8287,7 +8387,7 @@ func (x *RequestActorSuspendRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use RequestActorSuspendRequest.ProtoReflect.Descriptor instead.
func (*RequestActorSuspendRequest) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{113}
+ return file_ateapi_proto_rawDescGZIP(), []int{115}
}
func (x *RequestActorSuspendRequest) GetWorker() *ObjectRef {
@@ -8321,7 +8421,7 @@ type RequestActorSuspendResponse struct {
func (x *RequestActorSuspendResponse) Reset() {
*x = RequestActorSuspendResponse{}
- mi := &file_ateapi_proto_msgTypes[114]
+ mi := &file_ateapi_proto_msgTypes[116]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -8333,7 +8433,7 @@ func (x *RequestActorSuspendResponse) String() string {
func (*RequestActorSuspendResponse) ProtoMessage() {}
func (x *RequestActorSuspendResponse) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[114]
+ mi := &file_ateapi_proto_msgTypes[116]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -8346,7 +8446,7 @@ func (x *RequestActorSuspendResponse) ProtoReflect() protoreflect.Message {
// Deprecated: Use RequestActorSuspendResponse.ProtoReflect.Descriptor instead.
func (*RequestActorSuspendResponse) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{114}
+ return file_ateapi_proto_rawDescGZIP(), []int{116}
}
func (x *RequestActorSuspendResponse) GetActor() *Actor {
@@ -8356,6 +8456,137 @@ func (x *RequestActorSuspendResponse) GetActor() *Actor {
return nil
}
+type ReportActorCrashedRequest struct {
+ state protoimpl.MessageState `protogen:"open.v1"`
+ // The Worker making the report. atespace is always empty; Workers are
+ // global-scoped. atelet fills this in from its caller's certificate.
+ //
+ // +k8s:required
+ // +k8s:beta(since: "0.0")=+k8s:subfield(atespace)=+k8s:forbidden # TODO: get rid of beta prefix
+ Worker *ObjectRef `protobuf:"bytes,1,opt,name=worker,proto3" json:"worker,omitempty"`
+ // The Actor whose sandbox died. It must be assigned to worker.
+ //
+ // +k8s:required
+ // +k8s:subfield(atespace)=+k8s:required
+ Actor *ObjectRef `protobuf:"bytes,2,opt,name=actor,proto3" json:"actor,omitempty"`
+ // The UID of the Actor the Worker believes it is hosting, as for
+ // RequestActorSuspend.
+ //
+ // +k8s:required
+ // +k8s:format=k8s-uuid
+ ActorUid string `protobuf:"bytes,3,opt,name=actor_uid,json=actorUid,proto3" json:"actor_uid,omitempty"`
+ // What the Worker saw, recorded in the Actor's crash message.
+ //
+ // +k8s:required
+ // +k8s:maxLength=4096
+ Message string `protobuf:"bytes,4,opt,name=message,proto3" json:"message,omitempty"`
+ unknownFields protoimpl.UnknownFields
+ sizeCache protoimpl.SizeCache
+}
+
+func (x *ReportActorCrashedRequest) Reset() {
+ *x = ReportActorCrashedRequest{}
+ mi := &file_ateapi_proto_msgTypes[117]
+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+ ms.StoreMessageInfo(mi)
+}
+
+func (x *ReportActorCrashedRequest) String() string {
+ return protoimpl.X.MessageStringOf(x)
+}
+
+func (*ReportActorCrashedRequest) ProtoMessage() {}
+
+func (x *ReportActorCrashedRequest) ProtoReflect() protoreflect.Message {
+ mi := &file_ateapi_proto_msgTypes[117]
+ if x != nil {
+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+ if ms.LoadMessageInfo() == nil {
+ ms.StoreMessageInfo(mi)
+ }
+ return ms
+ }
+ return mi.MessageOf(x)
+}
+
+// Deprecated: Use ReportActorCrashedRequest.ProtoReflect.Descriptor instead.
+func (*ReportActorCrashedRequest) Descriptor() ([]byte, []int) {
+ return file_ateapi_proto_rawDescGZIP(), []int{117}
+}
+
+func (x *ReportActorCrashedRequest) GetWorker() *ObjectRef {
+ if x != nil {
+ return x.Worker
+ }
+ return nil
+}
+
+func (x *ReportActorCrashedRequest) GetActor() *ObjectRef {
+ if x != nil {
+ return x.Actor
+ }
+ return nil
+}
+
+func (x *ReportActorCrashedRequest) GetActorUid() string {
+ if x != nil {
+ return x.ActorUid
+ }
+ return ""
+}
+
+func (x *ReportActorCrashedRequest) GetMessage() string {
+ if x != nil {
+ return x.Message
+ }
+ return ""
+}
+
+type ReportActorCrashedResponse struct {
+ state protoimpl.MessageState `protogen:"open.v1"`
+ // The Actor as the report left it.
+ Actor *Actor `protobuf:"bytes,1,opt,name=actor,proto3" json:"actor,omitempty"`
+ unknownFields protoimpl.UnknownFields
+ sizeCache protoimpl.SizeCache
+}
+
+func (x *ReportActorCrashedResponse) Reset() {
+ *x = ReportActorCrashedResponse{}
+ mi := &file_ateapi_proto_msgTypes[118]
+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+ ms.StoreMessageInfo(mi)
+}
+
+func (x *ReportActorCrashedResponse) String() string {
+ return protoimpl.X.MessageStringOf(x)
+}
+
+func (*ReportActorCrashedResponse) ProtoMessage() {}
+
+func (x *ReportActorCrashedResponse) ProtoReflect() protoreflect.Message {
+ mi := &file_ateapi_proto_msgTypes[118]
+ if x != nil {
+ ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+ if ms.LoadMessageInfo() == nil {
+ ms.StoreMessageInfo(mi)
+ }
+ return ms
+ }
+ return mi.MessageOf(x)
+}
+
+// Deprecated: Use ReportActorCrashedResponse.ProtoReflect.Descriptor instead.
+func (*ReportActorCrashedResponse) Descriptor() ([]byte, []int) {
+ return file_ateapi_proto_rawDescGZIP(), []int{118}
+}
+
+func (x *ReportActorCrashedResponse) GetActor() *Actor {
+ if x != nil {
+ return x.Actor
+ }
+ return nil
+}
+
type ReportActorSnapshotStateRequest struct {
state protoimpl.MessageState `protogen:"open.v1"`
// The Actor the report is about.
@@ -8377,7 +8608,7 @@ type ReportActorSnapshotStateRequest struct {
func (x *ReportActorSnapshotStateRequest) Reset() {
*x = ReportActorSnapshotStateRequest{}
- mi := &file_ateapi_proto_msgTypes[115]
+ mi := &file_ateapi_proto_msgTypes[119]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -8389,7 +8620,7 @@ func (x *ReportActorSnapshotStateRequest) String() string {
func (*ReportActorSnapshotStateRequest) ProtoMessage() {}
func (x *ReportActorSnapshotStateRequest) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[115]
+ mi := &file_ateapi_proto_msgTypes[119]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -8402,7 +8633,7 @@ func (x *ReportActorSnapshotStateRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use ReportActorSnapshotStateRequest.ProtoReflect.Descriptor instead.
func (*ReportActorSnapshotStateRequest) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{115}
+ return file_ateapi_proto_rawDescGZIP(), []int{119}
}
func (x *ReportActorSnapshotStateRequest) GetActor() *ObjectRef {
@@ -8434,7 +8665,7 @@ type ReportActorSnapshotStateResponse struct {
func (x *ReportActorSnapshotStateResponse) Reset() {
*x = ReportActorSnapshotStateResponse{}
- mi := &file_ateapi_proto_msgTypes[116]
+ mi := &file_ateapi_proto_msgTypes[120]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -8446,7 +8677,7 @@ func (x *ReportActorSnapshotStateResponse) String() string {
func (*ReportActorSnapshotStateResponse) ProtoMessage() {}
func (x *ReportActorSnapshotStateResponse) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[116]
+ mi := &file_ateapi_proto_msgTypes[120]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -8459,7 +8690,7 @@ func (x *ReportActorSnapshotStateResponse) ProtoReflect() protoreflect.Message {
// Deprecated: Use ReportActorSnapshotStateResponse.ProtoReflect.Descriptor instead.
func (*ReportActorSnapshotStateResponse) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{116}
+ return file_ateapi_proto_rawDescGZIP(), []int{120}
}
// ActorSnapshotState is what atelet holds for one Actor on its node. It
@@ -8493,7 +8724,7 @@ type ActorSnapshotState struct {
func (x *ActorSnapshotState) Reset() {
*x = ActorSnapshotState{}
- mi := &file_ateapi_proto_msgTypes[117]
+ mi := &file_ateapi_proto_msgTypes[121]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -8505,7 +8736,7 @@ func (x *ActorSnapshotState) String() string {
func (*ActorSnapshotState) ProtoMessage() {}
func (x *ActorSnapshotState) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[117]
+ mi := &file_ateapi_proto_msgTypes[121]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -8518,7 +8749,7 @@ func (x *ActorSnapshotState) ProtoReflect() protoreflect.Message {
// Deprecated: Use ActorSnapshotState.ProtoReflect.Descriptor instead.
func (*ActorSnapshotState) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{117}
+ return file_ateapi_proto_rawDescGZIP(), []int{121}
}
func (x *ActorSnapshotState) GetLatestLocal() *SnapshotReport {
@@ -8590,7 +8821,7 @@ type SnapshotReport struct {
func (x *SnapshotReport) Reset() {
*x = SnapshotReport{}
- mi := &file_ateapi_proto_msgTypes[118]
+ mi := &file_ateapi_proto_msgTypes[122]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -8602,7 +8833,7 @@ func (x *SnapshotReport) String() string {
func (*SnapshotReport) ProtoMessage() {}
func (x *SnapshotReport) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[118]
+ mi := &file_ateapi_proto_msgTypes[122]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -8615,7 +8846,7 @@ func (x *SnapshotReport) ProtoReflect() protoreflect.Message {
// Deprecated: Use SnapshotReport.ProtoReflect.Descriptor instead.
func (*SnapshotReport) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{118}
+ return file_ateapi_proto_rawDescGZIP(), []int{122}
}
func (x *SnapshotReport) GetUuid() string {
@@ -8685,7 +8916,7 @@ type AccessPolicy struct {
func (x *AccessPolicy) Reset() {
*x = AccessPolicy{}
- mi := &file_ateapi_proto_msgTypes[119]
+ mi := &file_ateapi_proto_msgTypes[123]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -8697,7 +8928,7 @@ func (x *AccessPolicy) String() string {
func (*AccessPolicy) ProtoMessage() {}
func (x *AccessPolicy) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[119]
+ mi := &file_ateapi_proto_msgTypes[123]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -8710,7 +8941,7 @@ func (x *AccessPolicy) ProtoReflect() protoreflect.Message {
// Deprecated: Use AccessPolicy.ProtoReflect.Descriptor instead.
func (*AccessPolicy) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{119}
+ return file_ateapi_proto_rawDescGZIP(), []int{123}
}
func (x *AccessPolicy) GetMetadata() *ResourceMetadata {
@@ -8752,7 +8983,7 @@ type Binding struct {
func (x *Binding) Reset() {
*x = Binding{}
- mi := &file_ateapi_proto_msgTypes[120]
+ mi := &file_ateapi_proto_msgTypes[124]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -8764,7 +8995,7 @@ func (x *Binding) String() string {
func (*Binding) ProtoMessage() {}
func (x *Binding) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[120]
+ mi := &file_ateapi_proto_msgTypes[124]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -8777,7 +9008,7 @@ func (x *Binding) ProtoReflect() protoreflect.Message {
// Deprecated: Use Binding.ProtoReflect.Descriptor instead.
func (*Binding) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{120}
+ return file_ateapi_proto_rawDescGZIP(), []int{124}
}
func (x *Binding) GetRole() string {
@@ -8809,7 +9040,7 @@ type GetGlobalAccessPolicyRequest struct {
func (x *GetGlobalAccessPolicyRequest) Reset() {
*x = GetGlobalAccessPolicyRequest{}
- mi := &file_ateapi_proto_msgTypes[121]
+ mi := &file_ateapi_proto_msgTypes[125]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -8821,7 +9052,7 @@ func (x *GetGlobalAccessPolicyRequest) String() string {
func (*GetGlobalAccessPolicyRequest) ProtoMessage() {}
func (x *GetGlobalAccessPolicyRequest) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[121]
+ mi := &file_ateapi_proto_msgTypes[125]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -8834,7 +9065,7 @@ func (x *GetGlobalAccessPolicyRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use GetGlobalAccessPolicyRequest.ProtoReflect.Descriptor instead.
func (*GetGlobalAccessPolicyRequest) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{121}
+ return file_ateapi_proto_rawDescGZIP(), []int{125}
}
func (x *GetGlobalAccessPolicyRequest) GetName() string {
@@ -8858,7 +9089,7 @@ type CreateGlobalAccessPolicyRequest struct {
func (x *CreateGlobalAccessPolicyRequest) Reset() {
*x = CreateGlobalAccessPolicyRequest{}
- mi := &file_ateapi_proto_msgTypes[122]
+ mi := &file_ateapi_proto_msgTypes[126]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -8870,7 +9101,7 @@ func (x *CreateGlobalAccessPolicyRequest) String() string {
func (*CreateGlobalAccessPolicyRequest) ProtoMessage() {}
func (x *CreateGlobalAccessPolicyRequest) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[122]
+ mi := &file_ateapi_proto_msgTypes[126]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -8883,7 +9114,7 @@ func (x *CreateGlobalAccessPolicyRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use CreateGlobalAccessPolicyRequest.ProtoReflect.Descriptor instead.
func (*CreateGlobalAccessPolicyRequest) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{122}
+ return file_ateapi_proto_rawDescGZIP(), []int{126}
}
func (x *CreateGlobalAccessPolicyRequest) GetAccessPolicy() *AccessPolicy {
@@ -8907,7 +9138,7 @@ type UpdateGlobalAccessPolicyRequest struct {
func (x *UpdateGlobalAccessPolicyRequest) Reset() {
*x = UpdateGlobalAccessPolicyRequest{}
- mi := &file_ateapi_proto_msgTypes[123]
+ mi := &file_ateapi_proto_msgTypes[127]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -8919,7 +9150,7 @@ func (x *UpdateGlobalAccessPolicyRequest) String() string {
func (*UpdateGlobalAccessPolicyRequest) ProtoMessage() {}
func (x *UpdateGlobalAccessPolicyRequest) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[123]
+ mi := &file_ateapi_proto_msgTypes[127]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -8932,7 +9163,7 @@ func (x *UpdateGlobalAccessPolicyRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use UpdateGlobalAccessPolicyRequest.ProtoReflect.Descriptor instead.
func (*UpdateGlobalAccessPolicyRequest) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{123}
+ return file_ateapi_proto_rawDescGZIP(), []int{127}
}
func (x *UpdateGlobalAccessPolicyRequest) GetAccessPolicy() *AccessPolicy {
@@ -8962,7 +9193,7 @@ type GetAtespaceAccessPolicyRequest struct {
func (x *GetAtespaceAccessPolicyRequest) Reset() {
*x = GetAtespaceAccessPolicyRequest{}
- mi := &file_ateapi_proto_msgTypes[124]
+ mi := &file_ateapi_proto_msgTypes[128]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -8974,7 +9205,7 @@ func (x *GetAtespaceAccessPolicyRequest) String() string {
func (*GetAtespaceAccessPolicyRequest) ProtoMessage() {}
func (x *GetAtespaceAccessPolicyRequest) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[124]
+ mi := &file_ateapi_proto_msgTypes[128]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -8987,7 +9218,7 @@ func (x *GetAtespaceAccessPolicyRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use GetAtespaceAccessPolicyRequest.ProtoReflect.Descriptor instead.
func (*GetAtespaceAccessPolicyRequest) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{124}
+ return file_ateapi_proto_rawDescGZIP(), []int{128}
}
func (x *GetAtespaceAccessPolicyRequest) GetAtespace() *ObjectRef {
@@ -9023,7 +9254,7 @@ type CreateAtespaceAccessPolicyRequest struct {
func (x *CreateAtespaceAccessPolicyRequest) Reset() {
*x = CreateAtespaceAccessPolicyRequest{}
- mi := &file_ateapi_proto_msgTypes[125]
+ mi := &file_ateapi_proto_msgTypes[129]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -9035,7 +9266,7 @@ func (x *CreateAtespaceAccessPolicyRequest) String() string {
func (*CreateAtespaceAccessPolicyRequest) ProtoMessage() {}
func (x *CreateAtespaceAccessPolicyRequest) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[125]
+ mi := &file_ateapi_proto_msgTypes[129]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -9048,7 +9279,7 @@ func (x *CreateAtespaceAccessPolicyRequest) ProtoReflect() protoreflect.Message
// Deprecated: Use CreateAtespaceAccessPolicyRequest.ProtoReflect.Descriptor instead.
func (*CreateAtespaceAccessPolicyRequest) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{125}
+ return file_ateapi_proto_rawDescGZIP(), []int{129}
}
func (x *CreateAtespaceAccessPolicyRequest) GetAtespace() *ObjectRef {
@@ -9084,7 +9315,7 @@ type UpdateAtespaceAccessPolicyRequest struct {
func (x *UpdateAtespaceAccessPolicyRequest) Reset() {
*x = UpdateAtespaceAccessPolicyRequest{}
- mi := &file_ateapi_proto_msgTypes[126]
+ mi := &file_ateapi_proto_msgTypes[130]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -9096,7 +9327,7 @@ func (x *UpdateAtespaceAccessPolicyRequest) String() string {
func (*UpdateAtespaceAccessPolicyRequest) ProtoMessage() {}
func (x *UpdateAtespaceAccessPolicyRequest) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[126]
+ mi := &file_ateapi_proto_msgTypes[130]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -9109,7 +9340,7 @@ func (x *UpdateAtespaceAccessPolicyRequest) ProtoReflect() protoreflect.Message
// Deprecated: Use UpdateAtespaceAccessPolicyRequest.ProtoReflect.Descriptor instead.
func (*UpdateAtespaceAccessPolicyRequest) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{126}
+ return file_ateapi_proto_rawDescGZIP(), []int{130}
}
func (x *UpdateAtespaceAccessPolicyRequest) GetAtespace() *ObjectRef {
@@ -9150,7 +9381,7 @@ type DeleteAtespaceAccessPolicyRequest struct {
func (x *DeleteAtespaceAccessPolicyRequest) Reset() {
*x = DeleteAtespaceAccessPolicyRequest{}
- mi := &file_ateapi_proto_msgTypes[127]
+ mi := &file_ateapi_proto_msgTypes[131]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -9162,7 +9393,7 @@ func (x *DeleteAtespaceAccessPolicyRequest) String() string {
func (*DeleteAtespaceAccessPolicyRequest) ProtoMessage() {}
func (x *DeleteAtespaceAccessPolicyRequest) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[127]
+ mi := &file_ateapi_proto_msgTypes[131]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -9175,7 +9406,7 @@ func (x *DeleteAtespaceAccessPolicyRequest) ProtoReflect() protoreflect.Message
// Deprecated: Use DeleteAtespaceAccessPolicyRequest.ProtoReflect.Descriptor instead.
func (*DeleteAtespaceAccessPolicyRequest) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{127}
+ return file_ateapi_proto_rawDescGZIP(), []int{131}
}
func (x *DeleteAtespaceAccessPolicyRequest) GetAtespace() *ObjectRef {
@@ -9221,7 +9452,7 @@ type ResourceAnnotation struct {
func (x *ResourceAnnotation) Reset() {
*x = ResourceAnnotation{}
- mi := &file_ateapi_proto_msgTypes[128]
+ mi := &file_ateapi_proto_msgTypes[132]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -9233,7 +9464,7 @@ func (x *ResourceAnnotation) String() string {
func (*ResourceAnnotation) ProtoMessage() {}
func (x *ResourceAnnotation) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[128]
+ mi := &file_ateapi_proto_msgTypes[132]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -9246,7 +9477,7 @@ func (x *ResourceAnnotation) ProtoReflect() protoreflect.Message {
// Deprecated: Use ResourceAnnotation.ProtoReflect.Descriptor instead.
func (*ResourceAnnotation) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{128}
+ return file_ateapi_proto_rawDescGZIP(), []int{132}
}
func (x *ResourceAnnotation) GetParents() []string {
@@ -9289,7 +9520,7 @@ type MethodAnnotation struct {
func (x *MethodAnnotation) Reset() {
*x = MethodAnnotation{}
- mi := &file_ateapi_proto_msgTypes[129]
+ mi := &file_ateapi_proto_msgTypes[133]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -9301,7 +9532,7 @@ func (x *MethodAnnotation) String() string {
func (*MethodAnnotation) ProtoMessage() {}
func (x *MethodAnnotation) ProtoReflect() protoreflect.Message {
- mi := &file_ateapi_proto_msgTypes[129]
+ mi := &file_ateapi_proto_msgTypes[133]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -9314,7 +9545,7 @@ func (x *MethodAnnotation) ProtoReflect() protoreflect.Message {
// Deprecated: Use MethodAnnotation.ProtoReflect.Descriptor instead.
func (*MethodAnnotation) Descriptor() ([]byte, []int) {
- return file_ateapi_proto_rawDescGZIP(), []int{129}
+ return file_ateapi_proto_rawDescGZIP(), []int{133}
}
func (x *MethodAnnotation) GetResource() string {
@@ -9389,23 +9620,25 @@ const file_ateapi_proto_rawDesc = "" +
"\vcreate_time\x18\x05 \x01(\v2\x1a.google.protobuf.TimestampR\n" +
"createTime\x12;\n" +
"\vupdate_time\x18\x06 \x01(\v2\x1a.google.protobuf.TimestampR\n" +
- "updateTime\"\x85\x02\n" +
- "\x0eExternalVolume\x12\x12\n" +
+ "updateTime\"I\n" +
+ "\x0fExternalVolumes\x126\n" +
+ "\avolumes\x18\x01 \x03(\v2\x1c.ateapi.ExternalVolumeStatusR\avolumes\"\x91\x02\n" +
+ "\x14ExternalVolumeStatus\x12\x12\n" +
"\x04name\x18\x01 \x01(\tR\x04name\x12*\n" +
"\x11storage_volume_id\x18\x02 \x01(\tR\x0fstorageVolumeId\x12\x1f\n" +
"\vvolume_type\x18\x03 \x01(\tR\n" +
- "volumeType\x12P\n" +
- "\x0evolume_context\x18\x05 \x03(\v2).ateapi.ExternalVolume.VolumeContextEntryR\rvolumeContext\x1a@\n" +
+ "volumeType\x12V\n" +
+ "\x0evolume_context\x18\x04 \x03(\v2/.ateapi.ExternalVolumeStatus.VolumeContextEntryR\rvolumeContext\x1a@\n" +
"\x12VolumeContextEntry\x12\x10\n" +
"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"\xa1\x02\n" +
"\x05Actor\x124\n" +
"\bmetadata\x18\x01 \x01(\v2\x18.ateapi.ResourceMetadataR\bmetadata\x128\n" +
- "\x0eactor_template\x18\x04 \x01(\v2\x11.ateapi.ObjectRefR\ractorTemplate\x129\n" +
- "\x0fworker_selector\x18\x05 \x01(\v2\x10.ateapi.SelectorR\x0eworkerSelector\x120\n" +
+ "\x0eactor_template\x18\x02 \x01(\v2\x11.ateapi.ObjectRefR\ractorTemplate\x129\n" +
+ "\x0fworker_selector\x18\x03 \x01(\v2\x10.ateapi.SelectorR\x0eworkerSelector\x120\n" +
"\n" +
- "source_tag\x18\x06 \x01(\v2\x11.ateapi.ObjectRefR\tsourceTag\x12+\n" +
- "\x06status\x18\a \x01(\v2\x13.ateapi.ActorStatusR\x06status:\x0e\x82\xb5\x18\n" +
+ "source_tag\x18\x04 \x01(\v2\x11.ateapi.ObjectRefR\tsourceTag\x12+\n" +
+ "\x06status\x18\x05 \x01(\v2\x13.ateapi.ActorStatusR\x06status:\x0e\x82\xb5\x18\n" +
"\x18\x01\"\x06Actors\"\x8d\x01\n" +
"\fEgressPolicy\x124\n" +
"\bmetadata\x18\x01 \x01(\v2\x18.ateapi.ResourceMetadataR\bmetadata\x12(\n" +
@@ -9441,16 +9674,15 @@ const file_ateapi_proto_rawDesc = "" +
"\tactor_jwt\x18\x04 \x01(\v2\x16.ateapi.ActorJwtSourceR\bactorJwt\"]\n" +
"\x0eActorJwtSource\x12\x1c\n" +
"\taudiences\x18\x01 \x03(\tR\taudiences\x12-\n" +
- "\x12expiration_seconds\x18\x02 \x01(\x03R\x11expirationSeconds\"\xfa\x02\n" +
+ "\x12expiration_seconds\x18\x02 \x01(\x03R\x11expirationSeconds\"\xfb\x02\n" +
"\vActorStatus\x12(\n" +
"\x05state\x18\x01 \x01(\x0e2\x12.ateapi.ActorStateR\x05state\x12E\n" +
- "\x11worker_assignment\x18\x02 \x01(\v2\x18.ateapi.WorkerAssignmentR\x10workerAssignment\x12A\n" +
- "\x10external_volumes\x18\a \x03(\v2\x16.ateapi.ExternalVolumeR\x0fexternalVolumes\x12(\n" +
- "\x05crash\x18\t \x01(\v2\x12.ateapi.ActorCrashR\x05crash\x12#\n" +
- "\rassigned_node\x18\n" +
- " \x01(\tR\fassignedNode\x12.\n" +
- "\tsnapshots\x18\v \x03(\v2\x10.ateapi.SnapshotR\tsnapshots\x128\n" +
- "\x18last_assigned_generation\x18\f \x01(\x05R\x16lastAssignedGeneration\"a\n" +
+ "\x11worker_assignment\x18\x02 \x01(\v2\x18.ateapi.WorkerAssignmentR\x10workerAssignment\x12B\n" +
+ "\x10external_volumes\x18\x03 \x01(\v2\x17.ateapi.ExternalVolumesR\x0fexternalVolumes\x12(\n" +
+ "\x05crash\x18\x04 \x01(\v2\x12.ateapi.ActorCrashR\x05crash\x12#\n" +
+ "\rassigned_node\x18\x05 \x01(\tR\fassignedNode\x12.\n" +
+ "\tsnapshots\x18\x06 \x03(\v2\x10.ateapi.SnapshotR\tsnapshots\x128\n" +
+ "\x18last_assigned_generation\x18\a \x01(\x05R\x16lastAssignedGeneration\"a\n" +
"\n" +
"ActorCrash\x12\x18\n" +
"\amessage\x18\x01 \x01(\tR\amessage\x129\n" +
@@ -9491,8 +9723,8 @@ const file_ateapi_proto_rawDesc = "" +
"containers\x12(\n" +
"\avolumes\x18\x04 \x03(\v2\x0e.ateapi.VolumeR\avolumes\x12?\n" +
"\x0fsnapshot_config\x18\x05 \x01(\v2\x16.ateapi.SnapshotConfigR\x0esnapshotConfig\x12/\n" +
- "\tresources\x18\a \x01(\v2\x11.ateapi.ResourcesR\tresources\x123\n" +
- "\x06status\x18\b \x01(\v2\x1b.ateapi.ActorTemplateStatusR\x06status:\x16\x82\xb5\x18\x12\x18\x01\"\x0eActorTemplates\"3\n" +
+ "\tresources\x18\x06 \x01(\v2\x11.ateapi.ResourcesR\tresources\x123\n" +
+ "\x06status\x18\a \x01(\v2\x1b.ateapi.ActorTemplateStatusR\x06status:\x16\x82\xb5\x18\x12\x18\x01\"\x0eActorTemplates\"3\n" +
"\tResources\x12&\n" +
"\x06limits\x18\x01 \x03(\v2\x0e.ateapi.LimitsR\x06limits\"8\n" +
"\x06Limits\x12\x12\n" +
@@ -9508,7 +9740,7 @@ const file_ateapi_proto_rawDesc = "" +
"\x0eSnapshotConfig\x12)\n" +
"\x10storage_location\x18\x01 \x01(\tR\x0fstorageLocation\x12G\n" +
"\x12preferred_fidelity\x18\x02 \x01(\x0e2\x18.ateapi.SnapshotFidelityR\x11preferredFidelity\x12R\n" +
- "\x16golden_snapshot_config\x18\x05 \x01(\v2\x1c.ateapi.GoldenSnapshotConfigR\x14goldenSnapshotConfig\"F\n" +
+ "\x16golden_snapshot_config\x18\x03 \x01(\v2\x1c.ateapi.GoldenSnapshotConfigR\x14goldenSnapshotConfig\"F\n" +
"\x14GoldenSnapshotConfig\x12.\n" +
"\x04mode\x18\x01 \x01(\x0e2\x1a.ateapi.GoldenSnapshotModeR\x04mode\"\xec\x02\n" +
"\tContainer\x12\x12\n" +
@@ -9534,15 +9766,17 @@ const file_ateapi_proto_rawDesc = "" +
"\x0ftimeout_seconds\x18\x02 \x01(\x05R\x0etimeoutSeconds\"7\n" +
"\rHttpGetAction\x12\x12\n" +
"\x04path\x18\x01 \x01(\tR\x04path\x12\x12\n" +
- "\x04port\x18\x02 \x01(\x05R\x04port\"\xa9\x02\n" +
+ "\x04port\x18\x02 \x01(\x05R\x04port\"\x89\x02\n" +
"\x06Volume\x12\x12\n" +
"\x04name\x18\x01 \x01(\tR\x04name\x12?\n" +
"\vdurable_dir\x18\x02 \x01(\v2\x1e.ateapi.DurableDirVolumeSourceR\n" +
- "durableDir\x12X\n" +
- "\x18external_volume_template\x18\x03 \x01(\v2\x1e.ateapi.ExternalVolumeTemplateR\x16externalVolumeTemplate\x12?\n" +
- "\vsystem_info\x18\x05 \x01(\v2\x1e.ateapi.SystemInfoVolumeSourceR\n" +
+ "durableDir\x128\n" +
+ "\bexternal\x18\x03 \x01(\v2\x1c.ateapi.ExternalVolumeSourceR\bexternal\x12?\n" +
+ "\vsystem_info\x18\x04 \x01(\v2\x1e.ateapi.SystemInfoVolumeSourceR\n" +
"systemInfo\x12/\n" +
- "\x05image\x18\x06 \x01(\v2\x19.ateapi.ImageVolumeSourceR\x05image\"1\n" +
+ "\x05image\x18\x05 \x01(\v2\x19.ateapi.ImageVolumeSourceR\x05image\"R\n" +
+ "\x14ExternalVolumeSource\x12:\n" +
+ "\btemplate\x18\x01 \x01(\v2\x1e.ateapi.ExternalVolumeTemplateR\btemplate\"1\n" +
"\x11ImageVolumeSource\x12\x1c\n" +
"\treference\x18\x01 \x01(\tR\treference\"\x18\n" +
"\x16DurableDirVolumeSource\"b\n" +
@@ -9558,10 +9792,10 @@ const file_ateapi_proto_rawDesc = "" +
"\x05items\x18\x01 \x03(\v2\x19.ateapi.ActorMetadataItemR\x05items\"Y\n" +
"\x11ActorMetadataItem\x120\n" +
"\x05field\x18\x01 \x01(\x0e2\x1a.ateapi.ActorMetadataFieldR\x05field\x12\x12\n" +
- "\x04path\x18\x02 \x01(\tR\x04path\"M\n" +
+ "\x04path\x18\x02 \x01(\tR\x04path\"A\n" +
"\x15TrustBundleDataSource\x12\x14\n" +
- "\x05names\x18\x03 \x03(\tR\x05names\x12\x12\n" +
- "\x04path\x18\x02 \x01(\tR\x04pathJ\x04\b\x01\x10\x02R\x04name\"@\n" +
+ "\x05names\x18\x01 \x03(\tR\x05names\x12\x12\n" +
+ "\x04path\x18\x02 \x01(\tR\x04path\"@\n" +
"\vVolumeMount\x12\x12\n" +
"\x04name\x18\x01 \x01(\tR\x04name\x12\x1d\n" +
"\n" +
@@ -9612,11 +9846,10 @@ const file_ateapi_proto_rawDesc = "" +
"\x12RevertActorRequest\x12'\n" +
"\x05actor\x18\x01 \x01(\v2\x11.ateapi.ObjectRefR\x05actor\":\n" +
"\x13RevertActorResponse\x12#\n" +
- "\x05actor\x18\x01 \x01(\v2\r.ateapi.ActorR\x05actor\"\x8b\x01\n" +
+ "\x05actor\x18\x01 \x01(\v2\r.ateapi.ActorR\x05actor\"n\n" +
"\x12DeleteActorRequest\x12'\n" +
- "\x05actor\x18\x01 \x01(\v2\x11.ateapi.ObjectRefR\x05actor\x12\x1b\n" +
- "\tany_state\x18\x02 \x01(\bR\banyState\x12/\n" +
- "\aoptions\x18\x03 \x01(\v2\x15.ateapi.DeleteOptionsR\aoptions\"Z\n" +
+ "\x05actor\x18\x01 \x01(\v2\x11.ateapi.ObjectRefR\x05actor\x12/\n" +
+ "\aoptions\x18\x02 \x01(\v2\x15.ateapi.DeleteOptionsR\aoptions\"Z\n" +
"\x1bGetActorEgressPolicyRequest\x12'\n" +
"\x05actor\x18\x01 \x01(\v2\x11.ateapi.ObjectRefR\x05actor\x12\x12\n" +
"\x04name\x18\x02 \x01(\tR\x04name\"\x84\x01\n" +
@@ -9633,17 +9866,17 @@ const file_ateapi_proto_rawDesc = "" +
"\rGetTagRequest\x12#\n" +
"\x03tag\x18\x01 \x01(\v2\x11.ateapi.ObjectRefR\x03tag\"\x8b\x01\n" +
"\x13MintActorJwtRequest\x12'\n" +
- "\x05actor\x18\x05 \x01(\v2\x11.ateapi.ObjectRefR\x05actor\x12\x1c\n" +
- "\taudiences\x18\x01 \x03(\tR\taudiences\x12-\n" +
- "\x12expiration_seconds\x18\b \x01(\x03R\x11expirationSeconds\"u\n" +
+ "\x05actor\x18\x01 \x01(\v2\x11.ateapi.ObjectRefR\x05actor\x12\x1c\n" +
+ "\taudiences\x18\x02 \x03(\tR\taudiences\x12-\n" +
+ "\x12expiration_seconds\x18\x03 \x01(\x03R\x11expirationSeconds\"u\n" +
"\x14MintActorJwtResponse\x12 \n" +
"\tactor_jwt\x18\x01 \x01(\tB\x03\x80\x01\x01R\bactorJwt\x12;\n" +
"\vexpire_time\x18\x02 \x01(\v2\x1a.google.protobuf.TimestampR\n" +
"expireTime\"\xa3\x01\n" +
"\x1bMintActorCertificateRequest\x12'\n" +
- "\x05actor\x18\x06 \x01(\v2\x11.ateapi.ObjectRefR\x05actor\x12\x1b\n" +
- "\tactor_uid\x18\a \x01(\tR\bactorUid\x12>\n" +
- "\x1bcertificate_signing_request\x18\x02 \x01(\fR\x19certificateSigningRequest\"M\n" +
+ "\x05actor\x18\x01 \x01(\v2\x11.ateapi.ObjectRefR\x05actor\x12\x1b\n" +
+ "\tactor_uid\x18\x02 \x01(\tR\bactorUid\x12>\n" +
+ "\x1bcertificate_signing_request\x18\x03 \x01(\fR\x19certificateSigningRequest\"M\n" +
"\x1cMintActorCertificateResponse\x12-\n" +
"\x12actor_certificates\x18\x01 \x03(\fR\x11actorCertificates\"\\\n" +
"\x0fListTagsRequest\x12\x1a\n" +
@@ -9714,11 +9947,12 @@ const file_ateapi_proto_rawDesc = "" +
"\x0eworker_pod_uid\x18\x05 \x01(\tR\fworkerPodUid\x12\x1b\n" +
"\tnode_name\x18\x06 \x01(\tR\bnodeName\x12\x10\n" +
"\x03ips\x18\a \x03(\tR\x03ips\x12(\n" +
- "\x10worker_spiffe_id\x18\r \x01(\tR\x0eworkerSpiffeId\x12#\n" +
- "\rsandbox_class\x18\b \x01(\tR\fsandboxClass\x122\n" +
- "\x06labels\x18\t \x03(\v2\x1a.ateapi.Worker.LabelsEntryR\x06labels\x12\x14\n" +
- "\x05epoch\x18\f \x01(\x03R\x05epoch\x12,\n" +
- "\x06status\x18\v \x01(\v2\x14.ateapi.WorkerStatusR\x06status\x1a9\n" +
+ "\x10worker_spiffe_id\x18\b \x01(\tR\x0eworkerSpiffeId\x12#\n" +
+ "\rsandbox_class\x18\t \x01(\tR\fsandboxClass\x122\n" +
+ "\x06labels\x18\n" +
+ " \x03(\v2\x1a.ateapi.Worker.LabelsEntryR\x06labels\x12\x14\n" +
+ "\x05epoch\x18\v \x01(\x03R\x05epoch\x12,\n" +
+ "\x06status\x18\f \x01(\v2\x14.ateapi.WorkerStatusR\x06status\x1a9\n" +
"\vLabelsEntry\x12\x10\n" +
"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01:\x0f\x82\xb5\x18\v\x18\x02\"\aWorkers\"\xd6\x02\n" +
@@ -9744,11 +9978,11 @@ const file_ateapi_proto_rawDesc = "" +
"\tresources\x18\x01 \x01(\v2\x11.ateapi.ResourcesR\tresources\x12\x16\n" +
"\x06actors\x18\x02 \x01(\x05R\x06actors\"\x81\x02\n" +
"\x0fActorAssignment\x124\n" +
- "\bmetadata\x18\x06 \x01(\v2\x18.ateapi.ResourceMetadataR\bmetadata\x12'\n" +
+ "\bmetadata\x18\x01 \x01(\v2\x18.ateapi.ResourceMetadataR\bmetadata\x12'\n" +
"\x05actor\x18\x02 \x01(\v2\x11.ateapi.ObjectRefR\x05actor\x12\x1b\n" +
"\tactor_uid\x18\x03 \x01(\tR\bactorUid\x12/\n" +
- "\tresources\x18\x05 \x01(\v2\x11.ateapi.ResourcesR\tresources\x12!\n" +
- "\fworker_epoch\x18\a \x01(\x03R\vworkerEpoch:\x1e\x82\xb5\x18\x1a\n" +
+ "\tresources\x18\x04 \x01(\v2\x11.ateapi.ResourcesR\tresources\x12!\n" +
+ "\fworker_epoch\x18\x05 \x01(\x03R\vworkerEpoch:\x1e\x82\xb5\x18\x1a\n" +
"\x06Worker\"\x10ActorAssignments\"\x81\x02\n" +
"\x15RegisterWorkerRequest\x12)\n" +
"\x06worker\x18\x01 \x01(\v2\x11.ateapi.ObjectRefR\x06worker\x123\n" +
@@ -9777,6 +10011,13 @@ const file_ateapi_proto_rawDesc = "" +
"\x05actor\x18\x02 \x01(\v2\x11.ateapi.ObjectRefR\x05actor\x12\x1b\n" +
"\tactor_uid\x18\x03 \x01(\tR\bactorUid\"B\n" +
"\x1bRequestActorSuspendResponse\x12#\n" +
+ "\x05actor\x18\x01 \x01(\v2\r.ateapi.ActorR\x05actor\"\xa6\x01\n" +
+ "\x19ReportActorCrashedRequest\x12)\n" +
+ "\x06worker\x18\x01 \x01(\v2\x11.ateapi.ObjectRefR\x06worker\x12'\n" +
+ "\x05actor\x18\x02 \x01(\v2\x11.ateapi.ObjectRefR\x05actor\x12\x1b\n" +
+ "\tactor_uid\x18\x03 \x01(\tR\bactorUid\x12\x18\n" +
+ "\amessage\x18\x04 \x01(\tR\amessage\"A\n" +
+ "\x1aReportActorCrashedResponse\x12#\n" +
"\x05actor\x18\x01 \x01(\v2\r.ateapi.ActorR\x05actor\"\x99\x01\n" +
"\x1fReportActorSnapshotStateRequest\x12'\n" +
"\x05actor\x18\x01 \x01(\v2\x11.ateapi.ObjectRefR\x05actor\x12\x1b\n" +
@@ -9855,9 +10096,9 @@ const file_ateapi_proto_rawDesc = "" +
"\x13ACTOR_STATE_RUNNING\x10\x02\x12\x1a\n" +
"\x16ACTOR_STATE_SUSPENDING\x10\x03\x12\x19\n" +
"\x15ACTOR_STATE_SUSPENDED\x10\x04\x12\x17\n" +
- "\x13ACTOR_STATE_CRASHED\x10\a\x12\x18\n" +
- "\x14ACTOR_STATE_DELETING\x10\b\x12\x19\n" +
- "\x15ACTOR_STATE_REVERTING\x10\t*\x7f\n" +
+ "\x13ACTOR_STATE_CRASHED\x10\x05\x12\x18\n" +
+ "\x14ACTOR_STATE_DELETING\x10\x06\x12\x19\n" +
+ "\x15ACTOR_STATE_REVERTING\x10\a*\x7f\n" +
"\x12GoldenSnapshotMode\x12$\n" +
" GOLDEN_SNAPSHOT_MODE_UNSPECIFIED\x10\x00\x12 \n" +
"\x1cGOLDEN_SNAPSHOT_MODE_ENABLED\x10\x01\x12!\n" +
@@ -9971,13 +10212,14 @@ const file_ateapi_proto_rawDesc = "" +
"\x1aUpdateAtespaceAccessPolicy\x12).ateapi.UpdateAtespaceAccessPolicyRequest\x1a\x14.ateapi.AccessPolicy\"\x12\x82\xb5\x18\x0e\n" +
"\fAccessPolicy\x12q\n" +
"\x1aDeleteAtespaceAccessPolicy\x12).ateapi.DeleteAtespaceAccessPolicyRequest\x1a\x14.ateapi.AccessPolicy\"\x12\x82\xb5\x18\x0e\n" +
- "\fAccessPolicy2\x81\x04\n" +
+ "\fAccessPolicy2\xde\x04\n" +
"\rWorkerService\x12O\n" +
"\x0eRegisterWorker\x12\x1d.ateapi.RegisterWorkerRequest\x1a\x1e.ateapi.RegisterWorkerResponse\x12s\n" +
"\x1aMintWorkerActorCertificate\x12).ateapi.MintWorkerActorCertificateRequest\x1a*.ateapi.MintWorkerActorCertificateResponse\x12[\n" +
"\x12MintWorkerActorJwt\x12!.ateapi.MintWorkerActorJwtRequest\x1a\".ateapi.MintWorkerActorJwtResponse\x12^\n" +
"\x13RequestActorSuspend\x12\".ateapi.RequestActorSuspendRequest\x1a#.ateapi.RequestActorSuspendResponse\x12m\n" +
- "\x18ReportActorSnapshotState\x12'.ateapi.ReportActorSnapshotStateRequest\x1a(.ateapi.ReportActorSnapshotStateResponse:Y\n" +
+ "\x18ReportActorSnapshotState\x12'.ateapi.ReportActorSnapshotStateRequest\x1a(.ateapi.ReportActorSnapshotStateResponse\x12[\n" +
+ "\x12ReportActorCrashed\x12!.ateapi.ReportActorCrashedRequest\x1a\".ateapi.ReportActorCrashedResponse:Y\n" +
"\bresource\x12\x1f.google.protobuf.MessageOptions\x18І\x03 \x01(\v2\x1a.ateapi.ResourceAnnotationR\bresource:R\n" +
"\x06method\x12\x1e.google.protobuf.MethodOptions\x18І\x03 \x01(\v2\x18.ateapi.MethodAnnotationR\x06methodB)Z'go.ate.dev/substrate/pkg/proto/ateapipbb\x06proto3"
@@ -9994,7 +10236,7 @@ func file_ateapi_proto_rawDescGZIP() []byte {
}
var file_ateapi_proto_enumTypes = make([]protoimpl.EnumInfo, 10)
-var file_ateapi_proto_msgTypes = make([]protoimpl.MessageInfo, 133)
+var file_ateapi_proto_msgTypes = make([]protoimpl.MessageInfo, 137)
var file_ateapi_proto_goTypes = []any{
(SnapshotOwner)(0), // 0: ateapi.SnapshotOwner
(SnapshotStorageStatus)(0), // 1: ateapi.SnapshotStorageStatus
@@ -10011,137 +10253,141 @@ var file_ateapi_proto_goTypes = []any{
(*Snapshot)(nil), // 12: ateapi.Snapshot
(*Selector)(nil), // 13: ateapi.Selector
(*ResourceMetadata)(nil), // 14: ateapi.ResourceMetadata
- (*ExternalVolume)(nil), // 15: ateapi.ExternalVolume
- (*Actor)(nil), // 16: ateapi.Actor
- (*EgressPolicy)(nil), // 17: ateapi.EgressPolicy
- (*EgressRule)(nil), // 18: ateapi.EgressRule
- (*HttpRule)(nil), // 19: ateapi.HttpRule
- (*HttpsRule)(nil), // 20: ateapi.HttpsRule
- (*TlsPassthroughRule)(nil), // 21: ateapi.TlsPassthroughRule
- (*Ports)(nil), // 22: ateapi.Ports
- (*AllPorts)(nil), // 23: ateapi.AllPorts
- (*HttpRuleEffects)(nil), // 24: ateapi.HttpRuleEffects
- (*CredentialHeader)(nil), // 25: ateapi.CredentialHeader
- (*ActorJwtSource)(nil), // 26: ateapi.ActorJwtSource
- (*ActorStatus)(nil), // 27: ateapi.ActorStatus
- (*ActorCrash)(nil), // 28: ateapi.ActorCrash
- (*WorkerAssignment)(nil), // 29: ateapi.WorkerAssignment
- (*TagStatus)(nil), // 30: ateapi.TagStatus
- (*Tag)(nil), // 31: ateapi.Tag
- (*Atespace)(nil), // 32: ateapi.Atespace
- (*ObjectRef)(nil), // 33: ateapi.ObjectRef
- (*ActorTemplate)(nil), // 34: ateapi.ActorTemplate
- (*Resources)(nil), // 35: ateapi.Resources
- (*Limits)(nil), // 36: ateapi.Limits
- (*GoldenSnapshotStatus)(nil), // 37: ateapi.GoldenSnapshotStatus
- (*ActorTemplateStatus)(nil), // 38: ateapi.ActorTemplateStatus
- (*SnapshotConfig)(nil), // 39: ateapi.SnapshotConfig
- (*GoldenSnapshotConfig)(nil), // 40: ateapi.GoldenSnapshotConfig
- (*Container)(nil), // 41: ateapi.Container
- (*SecurityContext)(nil), // 42: ateapi.SecurityContext
- (*Capabilities)(nil), // 43: ateapi.Capabilities
- (*EnvVar)(nil), // 44: ateapi.EnvVar
- (*WakeupProbe)(nil), // 45: ateapi.WakeupProbe
- (*HttpGetAction)(nil), // 46: ateapi.HttpGetAction
- (*Volume)(nil), // 47: ateapi.Volume
- (*ImageVolumeSource)(nil), // 48: ateapi.ImageVolumeSource
- (*DurableDirVolumeSource)(nil), // 49: ateapi.DurableDirVolumeSource
- (*ExternalVolumeTemplate)(nil), // 50: ateapi.ExternalVolumeTemplate
- (*SystemInfoVolumeSource)(nil), // 51: ateapi.SystemInfoVolumeSource
- (*SystemInfoDataSource)(nil), // 52: ateapi.SystemInfoDataSource
- (*ActorMetadataDataSource)(nil), // 53: ateapi.ActorMetadataDataSource
- (*ActorMetadataItem)(nil), // 54: ateapi.ActorMetadataItem
- (*TrustBundleDataSource)(nil), // 55: ateapi.TrustBundleDataSource
- (*VolumeMount)(nil), // 56: ateapi.VolumeMount
- (*CreateAtespaceRequest)(nil), // 57: ateapi.CreateAtespaceRequest
- (*GetAtespaceRequest)(nil), // 58: ateapi.GetAtespaceRequest
- (*ListAtespacesRequest)(nil), // 59: ateapi.ListAtespacesRequest
- (*ListAtespacesResponse)(nil), // 60: ateapi.ListAtespacesResponse
- (*DeleteAtespaceRequest)(nil), // 61: ateapi.DeleteAtespaceRequest
- (*CreateActorTemplateRequest)(nil), // 62: ateapi.CreateActorTemplateRequest
- (*GetActorTemplateRequest)(nil), // 63: ateapi.GetActorTemplateRequest
- (*ListActorTemplatesRequest)(nil), // 64: ateapi.ListActorTemplatesRequest
- (*ListActorTemplatesResponse)(nil), // 65: ateapi.ListActorTemplatesResponse
- (*DeleteActorTemplateRequest)(nil), // 66: ateapi.DeleteActorTemplateRequest
- (*GetActorRequest)(nil), // 67: ateapi.GetActorRequest
- (*CreateActorRequest)(nil), // 68: ateapi.CreateActorRequest
- (*UpdateActorRequest)(nil), // 69: ateapi.UpdateActorRequest
- (*SuspendActorRequest)(nil), // 70: ateapi.SuspendActorRequest
- (*SuspendActorResponse)(nil), // 71: ateapi.SuspendActorResponse
- (*SuspendOptions)(nil), // 72: ateapi.SuspendOptions
- (*ResumeActorRequest)(nil), // 73: ateapi.ResumeActorRequest
- (*ResumeActorResponse)(nil), // 74: ateapi.ResumeActorResponse
- (*RevertActorRequest)(nil), // 75: ateapi.RevertActorRequest
- (*RevertActorResponse)(nil), // 76: ateapi.RevertActorResponse
- (*DeleteActorRequest)(nil), // 77: ateapi.DeleteActorRequest
- (*GetActorEgressPolicyRequest)(nil), // 78: ateapi.GetActorEgressPolicyRequest
- (*CreateActorEgressPolicyRequest)(nil), // 79: ateapi.CreateActorEgressPolicyRequest
- (*UpdateActorEgressPolicyRequest)(nil), // 80: ateapi.UpdateActorEgressPolicyRequest
- (*DeleteActorEgressPolicyRequest)(nil), // 81: ateapi.DeleteActorEgressPolicyRequest
- (*GetTagRequest)(nil), // 82: ateapi.GetTagRequest
- (*MintActorJwtRequest)(nil), // 83: ateapi.MintActorJwtRequest
- (*MintActorJwtResponse)(nil), // 84: ateapi.MintActorJwtResponse
- (*MintActorCertificateRequest)(nil), // 85: ateapi.MintActorCertificateRequest
- (*MintActorCertificateResponse)(nil), // 86: ateapi.MintActorCertificateResponse
- (*ListTagsRequest)(nil), // 87: ateapi.ListTagsRequest
- (*ListTagsResponse)(nil), // 88: ateapi.ListTagsResponse
- (*CreateTagRequest)(nil), // 89: ateapi.CreateTagRequest
- (*UpdateTagRequest)(nil), // 90: ateapi.UpdateTagRequest
- (*DeleteTagRequest)(nil), // 91: ateapi.DeleteTagRequest
- (*DeleteOptions)(nil), // 92: ateapi.DeleteOptions
- (*ListOptions)(nil), // 93: ateapi.ListOptions
- (*ListWorkerActorAssignmentsRequest)(nil), // 94: ateapi.ListWorkerActorAssignmentsRequest
- (*ListWorkerActorAssignmentsResponse)(nil), // 95: ateapi.ListWorkerActorAssignmentsResponse
- (*ListWorkersRequest)(nil), // 96: ateapi.ListWorkersRequest
- (*ListWorkersResponse)(nil), // 97: ateapi.ListWorkersResponse
- (*GetWorkerRequest)(nil), // 98: ateapi.GetWorkerRequest
- (*CreateWorkerRequest)(nil), // 99: ateapi.CreateWorkerRequest
- (*UpdateWorkerRequest)(nil), // 100: ateapi.UpdateWorkerRequest
- (*DeleteWorkerRequest)(nil), // 101: ateapi.DeleteWorkerRequest
- (*DrainWorkerRequest)(nil), // 102: ateapi.DrainWorkerRequest
- (*DrainWorkerResponse)(nil), // 103: ateapi.DrainWorkerResponse
- (*MarkWorkerAvailableRequest)(nil), // 104: ateapi.MarkWorkerAvailableRequest
- (*MarkWorkerAvailableResponse)(nil), // 105: ateapi.MarkWorkerAvailableResponse
- (*MarkWorkerUnavailableRequest)(nil), // 106: ateapi.MarkWorkerUnavailableRequest
- (*MarkWorkerUnavailableResponse)(nil), // 107: ateapi.MarkWorkerUnavailableResponse
- (*ListActorsRequest)(nil), // 108: ateapi.ListActorsRequest
- (*ListActorsResponse)(nil), // 109: ateapi.ListActorsResponse
- (*Worker)(nil), // 110: ateapi.Worker
- (*WorkerStatus)(nil), // 111: ateapi.WorkerStatus
- (*SandboxRuntime)(nil), // 112: ateapi.SandboxRuntime
- (*VersionedSandboxCompat)(nil), // 113: ateapi.VersionedSandboxCompat
- (*AttributeEntry)(nil), // 114: ateapi.AttributeEntry
- (*WorkerResources)(nil), // 115: ateapi.WorkerResources
- (*ActorAssignment)(nil), // 116: ateapi.ActorAssignment
- (*RegisterWorkerRequest)(nil), // 117: ateapi.RegisterWorkerRequest
- (*RegisterWorkerResponse)(nil), // 118: ateapi.RegisterWorkerResponse
- (*MintWorkerActorCertificateRequest)(nil), // 119: ateapi.MintWorkerActorCertificateRequest
- (*MintWorkerActorCertificateResponse)(nil), // 120: ateapi.MintWorkerActorCertificateResponse
- (*MintWorkerActorJwtRequest)(nil), // 121: ateapi.MintWorkerActorJwtRequest
- (*MintWorkerActorJwtResponse)(nil), // 122: ateapi.MintWorkerActorJwtResponse
- (*RequestActorSuspendRequest)(nil), // 123: ateapi.RequestActorSuspendRequest
- (*RequestActorSuspendResponse)(nil), // 124: ateapi.RequestActorSuspendResponse
- (*ReportActorSnapshotStateRequest)(nil), // 125: ateapi.ReportActorSnapshotStateRequest
- (*ReportActorSnapshotStateResponse)(nil), // 126: ateapi.ReportActorSnapshotStateResponse
- (*ActorSnapshotState)(nil), // 127: ateapi.ActorSnapshotState
- (*SnapshotReport)(nil), // 128: ateapi.SnapshotReport
- (*AccessPolicy)(nil), // 129: ateapi.AccessPolicy
- (*Binding)(nil), // 130: ateapi.Binding
- (*GetGlobalAccessPolicyRequest)(nil), // 131: ateapi.GetGlobalAccessPolicyRequest
- (*CreateGlobalAccessPolicyRequest)(nil), // 132: ateapi.CreateGlobalAccessPolicyRequest
- (*UpdateGlobalAccessPolicyRequest)(nil), // 133: ateapi.UpdateGlobalAccessPolicyRequest
- (*GetAtespaceAccessPolicyRequest)(nil), // 134: ateapi.GetAtespaceAccessPolicyRequest
- (*CreateAtespaceAccessPolicyRequest)(nil), // 135: ateapi.CreateAtespaceAccessPolicyRequest
- (*UpdateAtespaceAccessPolicyRequest)(nil), // 136: ateapi.UpdateAtespaceAccessPolicyRequest
- (*DeleteAtespaceAccessPolicyRequest)(nil), // 137: ateapi.DeleteAtespaceAccessPolicyRequest
- (*ResourceAnnotation)(nil), // 138: ateapi.ResourceAnnotation
- (*MethodAnnotation)(nil), // 139: ateapi.MethodAnnotation
- nil, // 140: ateapi.Selector.MatchLabelsEntry
- nil, // 141: ateapi.ExternalVolume.VolumeContextEntry
- nil, // 142: ateapi.Worker.LabelsEntry
- (*timestamppb.Timestamp)(nil), // 143: google.protobuf.Timestamp
- (*descriptorpb.MessageOptions)(nil), // 144: google.protobuf.MessageOptions
- (*descriptorpb.MethodOptions)(nil), // 145: google.protobuf.MethodOptions
+ (*ExternalVolumes)(nil), // 15: ateapi.ExternalVolumes
+ (*ExternalVolumeStatus)(nil), // 16: ateapi.ExternalVolumeStatus
+ (*Actor)(nil), // 17: ateapi.Actor
+ (*EgressPolicy)(nil), // 18: ateapi.EgressPolicy
+ (*EgressRule)(nil), // 19: ateapi.EgressRule
+ (*HttpRule)(nil), // 20: ateapi.HttpRule
+ (*HttpsRule)(nil), // 21: ateapi.HttpsRule
+ (*TlsPassthroughRule)(nil), // 22: ateapi.TlsPassthroughRule
+ (*Ports)(nil), // 23: ateapi.Ports
+ (*AllPorts)(nil), // 24: ateapi.AllPorts
+ (*HttpRuleEffects)(nil), // 25: ateapi.HttpRuleEffects
+ (*CredentialHeader)(nil), // 26: ateapi.CredentialHeader
+ (*ActorJwtSource)(nil), // 27: ateapi.ActorJwtSource
+ (*ActorStatus)(nil), // 28: ateapi.ActorStatus
+ (*ActorCrash)(nil), // 29: ateapi.ActorCrash
+ (*WorkerAssignment)(nil), // 30: ateapi.WorkerAssignment
+ (*TagStatus)(nil), // 31: ateapi.TagStatus
+ (*Tag)(nil), // 32: ateapi.Tag
+ (*Atespace)(nil), // 33: ateapi.Atespace
+ (*ObjectRef)(nil), // 34: ateapi.ObjectRef
+ (*ActorTemplate)(nil), // 35: ateapi.ActorTemplate
+ (*Resources)(nil), // 36: ateapi.Resources
+ (*Limits)(nil), // 37: ateapi.Limits
+ (*GoldenSnapshotStatus)(nil), // 38: ateapi.GoldenSnapshotStatus
+ (*ActorTemplateStatus)(nil), // 39: ateapi.ActorTemplateStatus
+ (*SnapshotConfig)(nil), // 40: ateapi.SnapshotConfig
+ (*GoldenSnapshotConfig)(nil), // 41: ateapi.GoldenSnapshotConfig
+ (*Container)(nil), // 42: ateapi.Container
+ (*SecurityContext)(nil), // 43: ateapi.SecurityContext
+ (*Capabilities)(nil), // 44: ateapi.Capabilities
+ (*EnvVar)(nil), // 45: ateapi.EnvVar
+ (*WakeupProbe)(nil), // 46: ateapi.WakeupProbe
+ (*HttpGetAction)(nil), // 47: ateapi.HttpGetAction
+ (*Volume)(nil), // 48: ateapi.Volume
+ (*ExternalVolumeSource)(nil), // 49: ateapi.ExternalVolumeSource
+ (*ImageVolumeSource)(nil), // 50: ateapi.ImageVolumeSource
+ (*DurableDirVolumeSource)(nil), // 51: ateapi.DurableDirVolumeSource
+ (*ExternalVolumeTemplate)(nil), // 52: ateapi.ExternalVolumeTemplate
+ (*SystemInfoVolumeSource)(nil), // 53: ateapi.SystemInfoVolumeSource
+ (*SystemInfoDataSource)(nil), // 54: ateapi.SystemInfoDataSource
+ (*ActorMetadataDataSource)(nil), // 55: ateapi.ActorMetadataDataSource
+ (*ActorMetadataItem)(nil), // 56: ateapi.ActorMetadataItem
+ (*TrustBundleDataSource)(nil), // 57: ateapi.TrustBundleDataSource
+ (*VolumeMount)(nil), // 58: ateapi.VolumeMount
+ (*CreateAtespaceRequest)(nil), // 59: ateapi.CreateAtespaceRequest
+ (*GetAtespaceRequest)(nil), // 60: ateapi.GetAtespaceRequest
+ (*ListAtespacesRequest)(nil), // 61: ateapi.ListAtespacesRequest
+ (*ListAtespacesResponse)(nil), // 62: ateapi.ListAtespacesResponse
+ (*DeleteAtespaceRequest)(nil), // 63: ateapi.DeleteAtespaceRequest
+ (*CreateActorTemplateRequest)(nil), // 64: ateapi.CreateActorTemplateRequest
+ (*GetActorTemplateRequest)(nil), // 65: ateapi.GetActorTemplateRequest
+ (*ListActorTemplatesRequest)(nil), // 66: ateapi.ListActorTemplatesRequest
+ (*ListActorTemplatesResponse)(nil), // 67: ateapi.ListActorTemplatesResponse
+ (*DeleteActorTemplateRequest)(nil), // 68: ateapi.DeleteActorTemplateRequest
+ (*GetActorRequest)(nil), // 69: ateapi.GetActorRequest
+ (*CreateActorRequest)(nil), // 70: ateapi.CreateActorRequest
+ (*UpdateActorRequest)(nil), // 71: ateapi.UpdateActorRequest
+ (*SuspendActorRequest)(nil), // 72: ateapi.SuspendActorRequest
+ (*SuspendActorResponse)(nil), // 73: ateapi.SuspendActorResponse
+ (*SuspendOptions)(nil), // 74: ateapi.SuspendOptions
+ (*ResumeActorRequest)(nil), // 75: ateapi.ResumeActorRequest
+ (*ResumeActorResponse)(nil), // 76: ateapi.ResumeActorResponse
+ (*RevertActorRequest)(nil), // 77: ateapi.RevertActorRequest
+ (*RevertActorResponse)(nil), // 78: ateapi.RevertActorResponse
+ (*DeleteActorRequest)(nil), // 79: ateapi.DeleteActorRequest
+ (*GetActorEgressPolicyRequest)(nil), // 80: ateapi.GetActorEgressPolicyRequest
+ (*CreateActorEgressPolicyRequest)(nil), // 81: ateapi.CreateActorEgressPolicyRequest
+ (*UpdateActorEgressPolicyRequest)(nil), // 82: ateapi.UpdateActorEgressPolicyRequest
+ (*DeleteActorEgressPolicyRequest)(nil), // 83: ateapi.DeleteActorEgressPolicyRequest
+ (*GetTagRequest)(nil), // 84: ateapi.GetTagRequest
+ (*MintActorJwtRequest)(nil), // 85: ateapi.MintActorJwtRequest
+ (*MintActorJwtResponse)(nil), // 86: ateapi.MintActorJwtResponse
+ (*MintActorCertificateRequest)(nil), // 87: ateapi.MintActorCertificateRequest
+ (*MintActorCertificateResponse)(nil), // 88: ateapi.MintActorCertificateResponse
+ (*ListTagsRequest)(nil), // 89: ateapi.ListTagsRequest
+ (*ListTagsResponse)(nil), // 90: ateapi.ListTagsResponse
+ (*CreateTagRequest)(nil), // 91: ateapi.CreateTagRequest
+ (*UpdateTagRequest)(nil), // 92: ateapi.UpdateTagRequest
+ (*DeleteTagRequest)(nil), // 93: ateapi.DeleteTagRequest
+ (*DeleteOptions)(nil), // 94: ateapi.DeleteOptions
+ (*ListOptions)(nil), // 95: ateapi.ListOptions
+ (*ListWorkerActorAssignmentsRequest)(nil), // 96: ateapi.ListWorkerActorAssignmentsRequest
+ (*ListWorkerActorAssignmentsResponse)(nil), // 97: ateapi.ListWorkerActorAssignmentsResponse
+ (*ListWorkersRequest)(nil), // 98: ateapi.ListWorkersRequest
+ (*ListWorkersResponse)(nil), // 99: ateapi.ListWorkersResponse
+ (*GetWorkerRequest)(nil), // 100: ateapi.GetWorkerRequest
+ (*CreateWorkerRequest)(nil), // 101: ateapi.CreateWorkerRequest
+ (*UpdateWorkerRequest)(nil), // 102: ateapi.UpdateWorkerRequest
+ (*DeleteWorkerRequest)(nil), // 103: ateapi.DeleteWorkerRequest
+ (*DrainWorkerRequest)(nil), // 104: ateapi.DrainWorkerRequest
+ (*DrainWorkerResponse)(nil), // 105: ateapi.DrainWorkerResponse
+ (*MarkWorkerAvailableRequest)(nil), // 106: ateapi.MarkWorkerAvailableRequest
+ (*MarkWorkerAvailableResponse)(nil), // 107: ateapi.MarkWorkerAvailableResponse
+ (*MarkWorkerUnavailableRequest)(nil), // 108: ateapi.MarkWorkerUnavailableRequest
+ (*MarkWorkerUnavailableResponse)(nil), // 109: ateapi.MarkWorkerUnavailableResponse
+ (*ListActorsRequest)(nil), // 110: ateapi.ListActorsRequest
+ (*ListActorsResponse)(nil), // 111: ateapi.ListActorsResponse
+ (*Worker)(nil), // 112: ateapi.Worker
+ (*WorkerStatus)(nil), // 113: ateapi.WorkerStatus
+ (*SandboxRuntime)(nil), // 114: ateapi.SandboxRuntime
+ (*VersionedSandboxCompat)(nil), // 115: ateapi.VersionedSandboxCompat
+ (*AttributeEntry)(nil), // 116: ateapi.AttributeEntry
+ (*WorkerResources)(nil), // 117: ateapi.WorkerResources
+ (*ActorAssignment)(nil), // 118: ateapi.ActorAssignment
+ (*RegisterWorkerRequest)(nil), // 119: ateapi.RegisterWorkerRequest
+ (*RegisterWorkerResponse)(nil), // 120: ateapi.RegisterWorkerResponse
+ (*MintWorkerActorCertificateRequest)(nil), // 121: ateapi.MintWorkerActorCertificateRequest
+ (*MintWorkerActorCertificateResponse)(nil), // 122: ateapi.MintWorkerActorCertificateResponse
+ (*MintWorkerActorJwtRequest)(nil), // 123: ateapi.MintWorkerActorJwtRequest
+ (*MintWorkerActorJwtResponse)(nil), // 124: ateapi.MintWorkerActorJwtResponse
+ (*RequestActorSuspendRequest)(nil), // 125: ateapi.RequestActorSuspendRequest
+ (*RequestActorSuspendResponse)(nil), // 126: ateapi.RequestActorSuspendResponse
+ (*ReportActorCrashedRequest)(nil), // 127: ateapi.ReportActorCrashedRequest
+ (*ReportActorCrashedResponse)(nil), // 128: ateapi.ReportActorCrashedResponse
+ (*ReportActorSnapshotStateRequest)(nil), // 129: ateapi.ReportActorSnapshotStateRequest
+ (*ReportActorSnapshotStateResponse)(nil), // 130: ateapi.ReportActorSnapshotStateResponse
+ (*ActorSnapshotState)(nil), // 131: ateapi.ActorSnapshotState
+ (*SnapshotReport)(nil), // 132: ateapi.SnapshotReport
+ (*AccessPolicy)(nil), // 133: ateapi.AccessPolicy
+ (*Binding)(nil), // 134: ateapi.Binding
+ (*GetGlobalAccessPolicyRequest)(nil), // 135: ateapi.GetGlobalAccessPolicyRequest
+ (*CreateGlobalAccessPolicyRequest)(nil), // 136: ateapi.CreateGlobalAccessPolicyRequest
+ (*UpdateGlobalAccessPolicyRequest)(nil), // 137: ateapi.UpdateGlobalAccessPolicyRequest
+ (*GetAtespaceAccessPolicyRequest)(nil), // 138: ateapi.GetAtespaceAccessPolicyRequest
+ (*CreateAtespaceAccessPolicyRequest)(nil), // 139: ateapi.CreateAtespaceAccessPolicyRequest
+ (*UpdateAtespaceAccessPolicyRequest)(nil), // 140: ateapi.UpdateAtespaceAccessPolicyRequest
+ (*DeleteAtespaceAccessPolicyRequest)(nil), // 141: ateapi.DeleteAtespaceAccessPolicyRequest
+ (*ResourceAnnotation)(nil), // 142: ateapi.ResourceAnnotation
+ (*MethodAnnotation)(nil), // 143: ateapi.MethodAnnotation
+ nil, // 144: ateapi.Selector.MatchLabelsEntry
+ nil, // 145: ateapi.ExternalVolumeStatus.VolumeContextEntry
+ nil, // 146: ateapi.Worker.LabelsEntry
+ (*timestamppb.Timestamp)(nil), // 147: google.protobuf.Timestamp
+ (*descriptorpb.MessageOptions)(nil), // 148: google.protobuf.MessageOptions
+ (*descriptorpb.MethodOptions)(nil), // 149: google.protobuf.MethodOptions
}
var file_ateapi_proto_depIdxs = []int32{
1, // 0: ateapi.SnapshotStorage.status:type_name -> ateapi.SnapshotStorageStatus
@@ -10149,286 +10395,293 @@ var file_ateapi_proto_depIdxs = []int32{
0, // 2: ateapi.Snapshot.owner:type_name -> ateapi.SnapshotOwner
2, // 3: ateapi.Snapshot.fidelity:type_name -> ateapi.SnapshotFidelity
11, // 4: ateapi.Snapshot.durable_snapshot:type_name -> ateapi.SnapshotStorage
- 112, // 5: ateapi.Snapshot.sandbox_runtime:type_name -> ateapi.SandboxRuntime
- 140, // 6: ateapi.Selector.match_labels:type_name -> ateapi.Selector.MatchLabelsEntry
- 143, // 7: ateapi.ResourceMetadata.create_time:type_name -> google.protobuf.Timestamp
- 143, // 8: ateapi.ResourceMetadata.update_time:type_name -> google.protobuf.Timestamp
- 141, // 9: ateapi.ExternalVolume.volume_context:type_name -> ateapi.ExternalVolume.VolumeContextEntry
- 14, // 10: ateapi.Actor.metadata:type_name -> ateapi.ResourceMetadata
- 33, // 11: ateapi.Actor.actor_template:type_name -> ateapi.ObjectRef
- 13, // 12: ateapi.Actor.worker_selector:type_name -> ateapi.Selector
- 33, // 13: ateapi.Actor.source_tag:type_name -> ateapi.ObjectRef
- 27, // 14: ateapi.Actor.status:type_name -> ateapi.ActorStatus
- 14, // 15: ateapi.EgressPolicy.metadata:type_name -> ateapi.ResourceMetadata
- 18, // 16: ateapi.EgressPolicy.rules:type_name -> ateapi.EgressRule
- 19, // 17: ateapi.EgressRule.http:type_name -> ateapi.HttpRule
- 20, // 18: ateapi.EgressRule.https:type_name -> ateapi.HttpsRule
- 21, // 19: ateapi.EgressRule.tls_passthrough:type_name -> ateapi.TlsPassthroughRule
- 22, // 20: ateapi.HttpRule.ports:type_name -> ateapi.Ports
- 24, // 21: ateapi.HttpRule.effects:type_name -> ateapi.HttpRuleEffects
- 22, // 22: ateapi.HttpsRule.ports:type_name -> ateapi.Ports
- 24, // 23: ateapi.HttpsRule.effects:type_name -> ateapi.HttpRuleEffects
- 22, // 24: ateapi.TlsPassthroughRule.ports:type_name -> ateapi.Ports
- 23, // 25: ateapi.Ports.all:type_name -> ateapi.AllPorts
- 25, // 26: ateapi.HttpRuleEffects.replace_headers:type_name -> ateapi.CredentialHeader
- 26, // 27: ateapi.CredentialHeader.actor_jwt:type_name -> ateapi.ActorJwtSource
- 4, // 28: ateapi.ActorStatus.state:type_name -> ateapi.ActorState
- 29, // 29: ateapi.ActorStatus.worker_assignment:type_name -> ateapi.WorkerAssignment
- 15, // 30: ateapi.ActorStatus.external_volumes:type_name -> ateapi.ExternalVolume
- 28, // 31: ateapi.ActorStatus.crash:type_name -> ateapi.ActorCrash
- 12, // 32: ateapi.ActorStatus.snapshots:type_name -> ateapi.Snapshot
- 143, // 33: ateapi.ActorCrash.crash_time:type_name -> google.protobuf.Timestamp
- 33, // 34: ateapi.WorkerAssignment.worker:type_name -> ateapi.ObjectRef
- 112, // 35: ateapi.WorkerAssignment.sandbox_runtime:type_name -> ateapi.SandboxRuntime
- 12, // 36: ateapi.TagStatus.snapshot:type_name -> ateapi.Snapshot
- 14, // 37: ateapi.Tag.metadata:type_name -> ateapi.ResourceMetadata
- 30, // 38: ateapi.Tag.status:type_name -> ateapi.TagStatus
- 3, // 39: ateapi.Tag.scope:type_name -> ateapi.TagScope
- 33, // 40: ateapi.Tag.source_actor:type_name -> ateapi.ObjectRef
- 14, // 41: ateapi.Atespace.metadata:type_name -> ateapi.ResourceMetadata
- 14, // 42: ateapi.ActorTemplate.metadata:type_name -> ateapi.ResourceMetadata
- 13, // 43: ateapi.ActorTemplate.worker_selector:type_name -> ateapi.Selector
- 41, // 44: ateapi.ActorTemplate.containers:type_name -> ateapi.Container
- 47, // 45: ateapi.ActorTemplate.volumes:type_name -> ateapi.Volume
- 39, // 46: ateapi.ActorTemplate.snapshot_config:type_name -> ateapi.SnapshotConfig
- 35, // 47: ateapi.ActorTemplate.resources:type_name -> ateapi.Resources
- 38, // 48: ateapi.ActorTemplate.status:type_name -> ateapi.ActorTemplateStatus
- 36, // 49: ateapi.Resources.limits:type_name -> ateapi.Limits
- 33, // 50: ateapi.GoldenSnapshotStatus.golden_tag:type_name -> ateapi.ObjectRef
- 143, // 51: ateapi.GoldenSnapshotStatus.snapshot_time:type_name -> google.protobuf.Timestamp
- 37, // 52: ateapi.ActorTemplateStatus.golden_snapshot_status:type_name -> ateapi.GoldenSnapshotStatus
- 2, // 53: ateapi.SnapshotConfig.preferred_fidelity:type_name -> ateapi.SnapshotFidelity
- 40, // 54: ateapi.SnapshotConfig.golden_snapshot_config:type_name -> ateapi.GoldenSnapshotConfig
- 5, // 55: ateapi.GoldenSnapshotConfig.mode:type_name -> ateapi.GoldenSnapshotMode
- 44, // 56: ateapi.Container.env:type_name -> ateapi.EnvVar
- 45, // 57: ateapi.Container.wakeup_probe:type_name -> ateapi.WakeupProbe
- 56, // 58: ateapi.Container.volume_mounts:type_name -> ateapi.VolumeMount
- 42, // 59: ateapi.Container.security_context:type_name -> ateapi.SecurityContext
- 35, // 60: ateapi.Container.resources:type_name -> ateapi.Resources
- 43, // 61: ateapi.SecurityContext.capabilities:type_name -> ateapi.Capabilities
- 46, // 62: ateapi.WakeupProbe.http_get:type_name -> ateapi.HttpGetAction
- 49, // 63: ateapi.Volume.durable_dir:type_name -> ateapi.DurableDirVolumeSource
- 50, // 64: ateapi.Volume.external_volume_template:type_name -> ateapi.ExternalVolumeTemplate
- 51, // 65: ateapi.Volume.system_info:type_name -> ateapi.SystemInfoVolumeSource
- 48, // 66: ateapi.Volume.image:type_name -> ateapi.ImageVolumeSource
- 52, // 67: ateapi.SystemInfoVolumeSource.data_sources:type_name -> ateapi.SystemInfoDataSource
- 53, // 68: ateapi.SystemInfoDataSource.actor_metadata:type_name -> ateapi.ActorMetadataDataSource
- 55, // 69: ateapi.SystemInfoDataSource.trust_bundle:type_name -> ateapi.TrustBundleDataSource
- 54, // 70: ateapi.ActorMetadataDataSource.items:type_name -> ateapi.ActorMetadataItem
- 6, // 71: ateapi.ActorMetadataItem.field:type_name -> ateapi.ActorMetadataField
- 32, // 72: ateapi.CreateAtespaceRequest.atespace:type_name -> ateapi.Atespace
- 33, // 73: ateapi.GetAtespaceRequest.atespace:type_name -> ateapi.ObjectRef
- 93, // 74: ateapi.ListAtespacesRequest.options:type_name -> ateapi.ListOptions
- 32, // 75: ateapi.ListAtespacesResponse.atespaces:type_name -> ateapi.Atespace
- 33, // 76: ateapi.DeleteAtespaceRequest.atespace:type_name -> ateapi.ObjectRef
- 92, // 77: ateapi.DeleteAtespaceRequest.options:type_name -> ateapi.DeleteOptions
- 34, // 78: ateapi.CreateActorTemplateRequest.actor_template:type_name -> ateapi.ActorTemplate
- 33, // 79: ateapi.GetActorTemplateRequest.actor_template:type_name -> ateapi.ObjectRef
- 93, // 80: ateapi.ListActorTemplatesRequest.options:type_name -> ateapi.ListOptions
- 34, // 81: ateapi.ListActorTemplatesResponse.actor_templates:type_name -> ateapi.ActorTemplate
- 33, // 82: ateapi.DeleteActorTemplateRequest.actor_template:type_name -> ateapi.ObjectRef
- 92, // 83: ateapi.DeleteActorTemplateRequest.options:type_name -> ateapi.DeleteOptions
- 33, // 84: ateapi.GetActorRequest.actor:type_name -> ateapi.ObjectRef
- 16, // 85: ateapi.CreateActorRequest.actor:type_name -> ateapi.Actor
- 16, // 86: ateapi.UpdateActorRequest.actor:type_name -> ateapi.Actor
- 33, // 87: ateapi.SuspendActorRequest.actor:type_name -> ateapi.ObjectRef
- 72, // 88: ateapi.SuspendActorRequest.options:type_name -> ateapi.SuspendOptions
- 16, // 89: ateapi.SuspendActorResponse.actor:type_name -> ateapi.Actor
- 7, // 90: ateapi.SuspendOptions.mode:type_name -> ateapi.DurabilityMode
- 33, // 91: ateapi.ResumeActorRequest.actor:type_name -> ateapi.ObjectRef
- 16, // 92: ateapi.ResumeActorResponse.actor:type_name -> ateapi.Actor
- 33, // 93: ateapi.RevertActorRequest.actor:type_name -> ateapi.ObjectRef
- 16, // 94: ateapi.RevertActorResponse.actor:type_name -> ateapi.Actor
- 33, // 95: ateapi.DeleteActorRequest.actor:type_name -> ateapi.ObjectRef
- 92, // 96: ateapi.DeleteActorRequest.options:type_name -> ateapi.DeleteOptions
- 33, // 97: ateapi.GetActorEgressPolicyRequest.actor:type_name -> ateapi.ObjectRef
- 33, // 98: ateapi.CreateActorEgressPolicyRequest.actor:type_name -> ateapi.ObjectRef
- 17, // 99: ateapi.CreateActorEgressPolicyRequest.egress_policy:type_name -> ateapi.EgressPolicy
- 33, // 100: ateapi.UpdateActorEgressPolicyRequest.actor:type_name -> ateapi.ObjectRef
- 17, // 101: ateapi.UpdateActorEgressPolicyRequest.egress_policy:type_name -> ateapi.EgressPolicy
- 33, // 102: ateapi.DeleteActorEgressPolicyRequest.actor:type_name -> ateapi.ObjectRef
- 92, // 103: ateapi.DeleteActorEgressPolicyRequest.options:type_name -> ateapi.DeleteOptions
- 33, // 104: ateapi.GetTagRequest.tag:type_name -> ateapi.ObjectRef
- 33, // 105: ateapi.MintActorJwtRequest.actor:type_name -> ateapi.ObjectRef
- 143, // 106: ateapi.MintActorJwtResponse.expire_time:type_name -> google.protobuf.Timestamp
- 33, // 107: ateapi.MintActorCertificateRequest.actor:type_name -> ateapi.ObjectRef
- 93, // 108: ateapi.ListTagsRequest.options:type_name -> ateapi.ListOptions
- 31, // 109: ateapi.ListTagsResponse.tags:type_name -> ateapi.Tag
- 31, // 110: ateapi.CreateTagRequest.tag:type_name -> ateapi.Tag
- 31, // 111: ateapi.UpdateTagRequest.tag:type_name -> ateapi.Tag
- 33, // 112: ateapi.DeleteTagRequest.tag:type_name -> ateapi.ObjectRef
- 92, // 113: ateapi.DeleteTagRequest.options:type_name -> ateapi.DeleteOptions
- 33, // 114: ateapi.ListWorkerActorAssignmentsRequest.worker:type_name -> ateapi.ObjectRef
- 93, // 115: ateapi.ListWorkerActorAssignmentsRequest.options:type_name -> ateapi.ListOptions
- 116, // 116: ateapi.ListWorkerActorAssignmentsResponse.actor_assignments:type_name -> ateapi.ActorAssignment
- 93, // 117: ateapi.ListWorkersRequest.options:type_name -> ateapi.ListOptions
- 110, // 118: ateapi.ListWorkersResponse.workers:type_name -> ateapi.Worker
- 33, // 119: ateapi.GetWorkerRequest.worker:type_name -> ateapi.ObjectRef
- 110, // 120: ateapi.CreateWorkerRequest.worker:type_name -> ateapi.Worker
- 110, // 121: ateapi.UpdateWorkerRequest.worker:type_name -> ateapi.Worker
- 33, // 122: ateapi.DeleteWorkerRequest.worker:type_name -> ateapi.ObjectRef
- 92, // 123: ateapi.DeleteWorkerRequest.options:type_name -> ateapi.DeleteOptions
- 33, // 124: ateapi.DrainWorkerRequest.worker:type_name -> ateapi.ObjectRef
- 110, // 125: ateapi.DrainWorkerResponse.worker:type_name -> ateapi.Worker
- 33, // 126: ateapi.MarkWorkerAvailableRequest.worker:type_name -> ateapi.ObjectRef
- 110, // 127: ateapi.MarkWorkerAvailableResponse.worker:type_name -> ateapi.Worker
- 33, // 128: ateapi.MarkWorkerUnavailableRequest.worker:type_name -> ateapi.ObjectRef
- 110, // 129: ateapi.MarkWorkerUnavailableResponse.worker:type_name -> ateapi.Worker
- 93, // 130: ateapi.ListActorsRequest.options:type_name -> ateapi.ListOptions
- 16, // 131: ateapi.ListActorsResponse.actors:type_name -> ateapi.Actor
- 14, // 132: ateapi.Worker.metadata:type_name -> ateapi.ResourceMetadata
- 142, // 133: ateapi.Worker.labels:type_name -> ateapi.Worker.LabelsEntry
- 111, // 134: ateapi.Worker.status:type_name -> ateapi.WorkerStatus
- 8, // 135: ateapi.WorkerStatus.state:type_name -> ateapi.WorkerState
- 115, // 136: ateapi.WorkerStatus.capacity:type_name -> ateapi.WorkerResources
- 115, // 137: ateapi.WorkerStatus.allocated:type_name -> ateapi.WorkerResources
- 112, // 138: ateapi.WorkerStatus.default_runtime:type_name -> ateapi.SandboxRuntime
- 112, // 139: ateapi.WorkerStatus.restorable_runtimes:type_name -> ateapi.SandboxRuntime
- 113, // 140: ateapi.SandboxRuntime.version:type_name -> ateapi.VersionedSandboxCompat
- 114, // 141: ateapi.VersionedSandboxCompat.attributes:type_name -> ateapi.AttributeEntry
- 35, // 142: ateapi.WorkerResources.resources:type_name -> ateapi.Resources
- 14, // 143: ateapi.ActorAssignment.metadata:type_name -> ateapi.ResourceMetadata
- 33, // 144: ateapi.ActorAssignment.actor:type_name -> ateapi.ObjectRef
- 35, // 145: ateapi.ActorAssignment.resources:type_name -> ateapi.Resources
- 33, // 146: ateapi.RegisterWorkerRequest.worker:type_name -> ateapi.ObjectRef
- 115, // 147: ateapi.RegisterWorkerRequest.capacity:type_name -> ateapi.WorkerResources
- 112, // 148: ateapi.RegisterWorkerRequest.default_runtime:type_name -> ateapi.SandboxRuntime
- 112, // 149: ateapi.RegisterWorkerRequest.restorable_runtimes:type_name -> ateapi.SandboxRuntime
- 110, // 150: ateapi.RegisterWorkerResponse.worker:type_name -> ateapi.Worker
- 33, // 151: ateapi.MintWorkerActorCertificateRequest.actor:type_name -> ateapi.ObjectRef
- 33, // 152: ateapi.MintWorkerActorCertificateRequest.worker:type_name -> ateapi.ObjectRef
- 33, // 153: ateapi.MintWorkerActorJwtRequest.actor:type_name -> ateapi.ObjectRef
- 143, // 154: ateapi.MintWorkerActorJwtResponse.expire_time:type_name -> google.protobuf.Timestamp
- 33, // 155: ateapi.RequestActorSuspendRequest.worker:type_name -> ateapi.ObjectRef
- 33, // 156: ateapi.RequestActorSuspendRequest.actor:type_name -> ateapi.ObjectRef
- 16, // 157: ateapi.RequestActorSuspendResponse.actor:type_name -> ateapi.Actor
- 33, // 158: ateapi.ReportActorSnapshotStateRequest.actor:type_name -> ateapi.ObjectRef
- 127, // 159: ateapi.ReportActorSnapshotStateRequest.state:type_name -> ateapi.ActorSnapshotState
- 128, // 160: ateapi.ActorSnapshotState.latest_local:type_name -> ateapi.SnapshotReport
- 128, // 161: ateapi.ActorSnapshotState.uploading:type_name -> ateapi.SnapshotReport
- 128, // 162: ateapi.ActorSnapshotState.latest_durable:type_name -> ateapi.SnapshotReport
- 143, // 163: ateapi.ActorSnapshotState.upload_due_time:type_name -> google.protobuf.Timestamp
- 2, // 164: ateapi.SnapshotReport.fidelity:type_name -> ateapi.SnapshotFidelity
- 10, // 165: ateapi.SnapshotReport.object:type_name -> ateapi.ObjectSnapshot
- 143, // 166: ateapi.SnapshotReport.local_commit_time:type_name -> google.protobuf.Timestamp
- 143, // 167: ateapi.SnapshotReport.durable_complete_time:type_name -> google.protobuf.Timestamp
- 14, // 168: ateapi.AccessPolicy.metadata:type_name -> ateapi.ResourceMetadata
- 130, // 169: ateapi.AccessPolicy.bindings:type_name -> ateapi.Binding
- 129, // 170: ateapi.CreateGlobalAccessPolicyRequest.access_policy:type_name -> ateapi.AccessPolicy
- 129, // 171: ateapi.UpdateGlobalAccessPolicyRequest.access_policy:type_name -> ateapi.AccessPolicy
- 33, // 172: ateapi.GetAtespaceAccessPolicyRequest.atespace:type_name -> ateapi.ObjectRef
- 33, // 173: ateapi.CreateAtespaceAccessPolicyRequest.atespace:type_name -> ateapi.ObjectRef
- 129, // 174: ateapi.CreateAtespaceAccessPolicyRequest.access_policy:type_name -> ateapi.AccessPolicy
- 33, // 175: ateapi.UpdateAtespaceAccessPolicyRequest.atespace:type_name -> ateapi.ObjectRef
- 129, // 176: ateapi.UpdateAtespaceAccessPolicyRequest.access_policy:type_name -> ateapi.AccessPolicy
- 33, // 177: ateapi.DeleteAtespaceAccessPolicyRequest.atespace:type_name -> ateapi.ObjectRef
- 92, // 178: ateapi.DeleteAtespaceAccessPolicyRequest.options:type_name -> ateapi.DeleteOptions
- 9, // 179: ateapi.ResourceAnnotation.scope:type_name -> ateapi.Scope
- 144, // 180: ateapi.resource:extendee -> google.protobuf.MessageOptions
- 145, // 181: ateapi.method:extendee -> google.protobuf.MethodOptions
- 138, // 182: ateapi.resource:type_name -> ateapi.ResourceAnnotation
- 139, // 183: ateapi.method:type_name -> ateapi.MethodAnnotation
- 67, // 184: ateapi.Control.GetActor:input_type -> ateapi.GetActorRequest
- 68, // 185: ateapi.Control.CreateActor:input_type -> ateapi.CreateActorRequest
- 69, // 186: ateapi.Control.UpdateActor:input_type -> ateapi.UpdateActorRequest
- 70, // 187: ateapi.Control.SuspendActor:input_type -> ateapi.SuspendActorRequest
- 73, // 188: ateapi.Control.ResumeActor:input_type -> ateapi.ResumeActorRequest
- 75, // 189: ateapi.Control.RevertActor:input_type -> ateapi.RevertActorRequest
- 77, // 190: ateapi.Control.DeleteActor:input_type -> ateapi.DeleteActorRequest
- 78, // 191: ateapi.Control.GetActorEgressPolicy:input_type -> ateapi.GetActorEgressPolicyRequest
- 79, // 192: ateapi.Control.CreateActorEgressPolicy:input_type -> ateapi.CreateActorEgressPolicyRequest
- 80, // 193: ateapi.Control.UpdateActorEgressPolicy:input_type -> ateapi.UpdateActorEgressPolicyRequest
- 81, // 194: ateapi.Control.DeleteActorEgressPolicy:input_type -> ateapi.DeleteActorEgressPolicyRequest
- 83, // 195: ateapi.Control.MintActorJwt:input_type -> ateapi.MintActorJwtRequest
- 85, // 196: ateapi.Control.MintActorCertificate:input_type -> ateapi.MintActorCertificateRequest
- 89, // 197: ateapi.Control.CreateTag:input_type -> ateapi.CreateTagRequest
- 82, // 198: ateapi.Control.GetTag:input_type -> ateapi.GetTagRequest
- 87, // 199: ateapi.Control.ListTags:input_type -> ateapi.ListTagsRequest
- 90, // 200: ateapi.Control.UpdateTag:input_type -> ateapi.UpdateTagRequest
- 91, // 201: ateapi.Control.DeleteTag:input_type -> ateapi.DeleteTagRequest
- 96, // 202: ateapi.Control.ListWorkers:input_type -> ateapi.ListWorkersRequest
- 98, // 203: ateapi.Control.GetWorker:input_type -> ateapi.GetWorkerRequest
- 99, // 204: ateapi.Control.CreateWorker:input_type -> ateapi.CreateWorkerRequest
- 100, // 205: ateapi.Control.UpdateWorker:input_type -> ateapi.UpdateWorkerRequest
- 101, // 206: ateapi.Control.DeleteWorker:input_type -> ateapi.DeleteWorkerRequest
- 102, // 207: ateapi.Control.DrainWorker:input_type -> ateapi.DrainWorkerRequest
- 104, // 208: ateapi.Control.MarkWorkerAvailable:input_type -> ateapi.MarkWorkerAvailableRequest
- 106, // 209: ateapi.Control.MarkWorkerUnavailable:input_type -> ateapi.MarkWorkerUnavailableRequest
- 94, // 210: ateapi.Control.ListWorkerActorAssignments:input_type -> ateapi.ListWorkerActorAssignmentsRequest
- 108, // 211: ateapi.Control.ListActors:input_type -> ateapi.ListActorsRequest
- 57, // 212: ateapi.Control.CreateAtespace:input_type -> ateapi.CreateAtespaceRequest
- 58, // 213: ateapi.Control.GetAtespace:input_type -> ateapi.GetAtespaceRequest
- 59, // 214: ateapi.Control.ListAtespaces:input_type -> ateapi.ListAtespacesRequest
- 61, // 215: ateapi.Control.DeleteAtespace:input_type -> ateapi.DeleteAtespaceRequest
- 62, // 216: ateapi.Control.CreateActorTemplate:input_type -> ateapi.CreateActorTemplateRequest
- 63, // 217: ateapi.Control.GetActorTemplate:input_type -> ateapi.GetActorTemplateRequest
- 64, // 218: ateapi.Control.ListActorTemplates:input_type -> ateapi.ListActorTemplatesRequest
- 66, // 219: ateapi.Control.DeleteActorTemplate:input_type -> ateapi.DeleteActorTemplateRequest
- 131, // 220: ateapi.Control.GetGlobalAccessPolicy:input_type -> ateapi.GetGlobalAccessPolicyRequest
- 132, // 221: ateapi.Control.CreateGlobalAccessPolicy:input_type -> ateapi.CreateGlobalAccessPolicyRequest
- 133, // 222: ateapi.Control.UpdateGlobalAccessPolicy:input_type -> ateapi.UpdateGlobalAccessPolicyRequest
- 134, // 223: ateapi.Control.GetAtespaceAccessPolicy:input_type -> ateapi.GetAtespaceAccessPolicyRequest
- 135, // 224: ateapi.Control.CreateAtespaceAccessPolicy:input_type -> ateapi.CreateAtespaceAccessPolicyRequest
- 136, // 225: ateapi.Control.UpdateAtespaceAccessPolicy:input_type -> ateapi.UpdateAtespaceAccessPolicyRequest
- 137, // 226: ateapi.Control.DeleteAtespaceAccessPolicy:input_type -> ateapi.DeleteAtespaceAccessPolicyRequest
- 117, // 227: ateapi.WorkerService.RegisterWorker:input_type -> ateapi.RegisterWorkerRequest
- 119, // 228: ateapi.WorkerService.MintWorkerActorCertificate:input_type -> ateapi.MintWorkerActorCertificateRequest
- 121, // 229: ateapi.WorkerService.MintWorkerActorJwt:input_type -> ateapi.MintWorkerActorJwtRequest
- 123, // 230: ateapi.WorkerService.RequestActorSuspend:input_type -> ateapi.RequestActorSuspendRequest
- 125, // 231: ateapi.WorkerService.ReportActorSnapshotState:input_type -> ateapi.ReportActorSnapshotStateRequest
- 16, // 232: ateapi.Control.GetActor:output_type -> ateapi.Actor
- 16, // 233: ateapi.Control.CreateActor:output_type -> ateapi.Actor
- 16, // 234: ateapi.Control.UpdateActor:output_type -> ateapi.Actor
- 71, // 235: ateapi.Control.SuspendActor:output_type -> ateapi.SuspendActorResponse
- 74, // 236: ateapi.Control.ResumeActor:output_type -> ateapi.ResumeActorResponse
- 76, // 237: ateapi.Control.RevertActor:output_type -> ateapi.RevertActorResponse
- 16, // 238: ateapi.Control.DeleteActor:output_type -> ateapi.Actor
- 17, // 239: ateapi.Control.GetActorEgressPolicy:output_type -> ateapi.EgressPolicy
- 17, // 240: ateapi.Control.CreateActorEgressPolicy:output_type -> ateapi.EgressPolicy
- 17, // 241: ateapi.Control.UpdateActorEgressPolicy:output_type -> ateapi.EgressPolicy
- 17, // 242: ateapi.Control.DeleteActorEgressPolicy:output_type -> ateapi.EgressPolicy
- 84, // 243: ateapi.Control.MintActorJwt:output_type -> ateapi.MintActorJwtResponse
- 86, // 244: ateapi.Control.MintActorCertificate:output_type -> ateapi.MintActorCertificateResponse
- 31, // 245: ateapi.Control.CreateTag:output_type -> ateapi.Tag
- 31, // 246: ateapi.Control.GetTag:output_type -> ateapi.Tag
- 88, // 247: ateapi.Control.ListTags:output_type -> ateapi.ListTagsResponse
- 31, // 248: ateapi.Control.UpdateTag:output_type -> ateapi.Tag
- 31, // 249: ateapi.Control.DeleteTag:output_type -> ateapi.Tag
- 97, // 250: ateapi.Control.ListWorkers:output_type -> ateapi.ListWorkersResponse
- 110, // 251: ateapi.Control.GetWorker:output_type -> ateapi.Worker
- 110, // 252: ateapi.Control.CreateWorker:output_type -> ateapi.Worker
- 110, // 253: ateapi.Control.UpdateWorker:output_type -> ateapi.Worker
- 110, // 254: ateapi.Control.DeleteWorker:output_type -> ateapi.Worker
- 103, // 255: ateapi.Control.DrainWorker:output_type -> ateapi.DrainWorkerResponse
- 105, // 256: ateapi.Control.MarkWorkerAvailable:output_type -> ateapi.MarkWorkerAvailableResponse
- 107, // 257: ateapi.Control.MarkWorkerUnavailable:output_type -> ateapi.MarkWorkerUnavailableResponse
- 95, // 258: ateapi.Control.ListWorkerActorAssignments:output_type -> ateapi.ListWorkerActorAssignmentsResponse
- 109, // 259: ateapi.Control.ListActors:output_type -> ateapi.ListActorsResponse
- 32, // 260: ateapi.Control.CreateAtespace:output_type -> ateapi.Atespace
- 32, // 261: ateapi.Control.GetAtespace:output_type -> ateapi.Atespace
- 60, // 262: ateapi.Control.ListAtespaces:output_type -> ateapi.ListAtespacesResponse
- 32, // 263: ateapi.Control.DeleteAtespace:output_type -> ateapi.Atespace
- 34, // 264: ateapi.Control.CreateActorTemplate:output_type -> ateapi.ActorTemplate
- 34, // 265: ateapi.Control.GetActorTemplate:output_type -> ateapi.ActorTemplate
- 65, // 266: ateapi.Control.ListActorTemplates:output_type -> ateapi.ListActorTemplatesResponse
- 34, // 267: ateapi.Control.DeleteActorTemplate:output_type -> ateapi.ActorTemplate
- 129, // 268: ateapi.Control.GetGlobalAccessPolicy:output_type -> ateapi.AccessPolicy
- 129, // 269: ateapi.Control.CreateGlobalAccessPolicy:output_type -> ateapi.AccessPolicy
- 129, // 270: ateapi.Control.UpdateGlobalAccessPolicy:output_type -> ateapi.AccessPolicy
- 129, // 271: ateapi.Control.GetAtespaceAccessPolicy:output_type -> ateapi.AccessPolicy
- 129, // 272: ateapi.Control.CreateAtespaceAccessPolicy:output_type -> ateapi.AccessPolicy
- 129, // 273: ateapi.Control.UpdateAtespaceAccessPolicy:output_type -> ateapi.AccessPolicy
- 129, // 274: ateapi.Control.DeleteAtespaceAccessPolicy:output_type -> ateapi.AccessPolicy
- 118, // 275: ateapi.WorkerService.RegisterWorker:output_type -> ateapi.RegisterWorkerResponse
- 120, // 276: ateapi.WorkerService.MintWorkerActorCertificate:output_type -> ateapi.MintWorkerActorCertificateResponse
- 122, // 277: ateapi.WorkerService.MintWorkerActorJwt:output_type -> ateapi.MintWorkerActorJwtResponse
- 124, // 278: ateapi.WorkerService.RequestActorSuspend:output_type -> ateapi.RequestActorSuspendResponse
- 126, // 279: ateapi.WorkerService.ReportActorSnapshotState:output_type -> ateapi.ReportActorSnapshotStateResponse
- 232, // [232:280] is the sub-list for method output_type
- 184, // [184:232] is the sub-list for method input_type
- 182, // [182:184] is the sub-list for extension type_name
- 180, // [180:182] is the sub-list for extension extendee
- 0, // [0:180] is the sub-list for field type_name
+ 114, // 5: ateapi.Snapshot.sandbox_runtime:type_name -> ateapi.SandboxRuntime
+ 144, // 6: ateapi.Selector.match_labels:type_name -> ateapi.Selector.MatchLabelsEntry
+ 147, // 7: ateapi.ResourceMetadata.create_time:type_name -> google.protobuf.Timestamp
+ 147, // 8: ateapi.ResourceMetadata.update_time:type_name -> google.protobuf.Timestamp
+ 16, // 9: ateapi.ExternalVolumes.volumes:type_name -> ateapi.ExternalVolumeStatus
+ 145, // 10: ateapi.ExternalVolumeStatus.volume_context:type_name -> ateapi.ExternalVolumeStatus.VolumeContextEntry
+ 14, // 11: ateapi.Actor.metadata:type_name -> ateapi.ResourceMetadata
+ 34, // 12: ateapi.Actor.actor_template:type_name -> ateapi.ObjectRef
+ 13, // 13: ateapi.Actor.worker_selector:type_name -> ateapi.Selector
+ 34, // 14: ateapi.Actor.source_tag:type_name -> ateapi.ObjectRef
+ 28, // 15: ateapi.Actor.status:type_name -> ateapi.ActorStatus
+ 14, // 16: ateapi.EgressPolicy.metadata:type_name -> ateapi.ResourceMetadata
+ 19, // 17: ateapi.EgressPolicy.rules:type_name -> ateapi.EgressRule
+ 20, // 18: ateapi.EgressRule.http:type_name -> ateapi.HttpRule
+ 21, // 19: ateapi.EgressRule.https:type_name -> ateapi.HttpsRule
+ 22, // 20: ateapi.EgressRule.tls_passthrough:type_name -> ateapi.TlsPassthroughRule
+ 23, // 21: ateapi.HttpRule.ports:type_name -> ateapi.Ports
+ 25, // 22: ateapi.HttpRule.effects:type_name -> ateapi.HttpRuleEffects
+ 23, // 23: ateapi.HttpsRule.ports:type_name -> ateapi.Ports
+ 25, // 24: ateapi.HttpsRule.effects:type_name -> ateapi.HttpRuleEffects
+ 23, // 25: ateapi.TlsPassthroughRule.ports:type_name -> ateapi.Ports
+ 24, // 26: ateapi.Ports.all:type_name -> ateapi.AllPorts
+ 26, // 27: ateapi.HttpRuleEffects.replace_headers:type_name -> ateapi.CredentialHeader
+ 27, // 28: ateapi.CredentialHeader.actor_jwt:type_name -> ateapi.ActorJwtSource
+ 4, // 29: ateapi.ActorStatus.state:type_name -> ateapi.ActorState
+ 30, // 30: ateapi.ActorStatus.worker_assignment:type_name -> ateapi.WorkerAssignment
+ 15, // 31: ateapi.ActorStatus.external_volumes:type_name -> ateapi.ExternalVolumes
+ 29, // 32: ateapi.ActorStatus.crash:type_name -> ateapi.ActorCrash
+ 12, // 33: ateapi.ActorStatus.snapshots:type_name -> ateapi.Snapshot
+ 147, // 34: ateapi.ActorCrash.crash_time:type_name -> google.protobuf.Timestamp
+ 34, // 35: ateapi.WorkerAssignment.worker:type_name -> ateapi.ObjectRef
+ 114, // 36: ateapi.WorkerAssignment.sandbox_runtime:type_name -> ateapi.SandboxRuntime
+ 12, // 37: ateapi.TagStatus.snapshot:type_name -> ateapi.Snapshot
+ 14, // 38: ateapi.Tag.metadata:type_name -> ateapi.ResourceMetadata
+ 31, // 39: ateapi.Tag.status:type_name -> ateapi.TagStatus
+ 3, // 40: ateapi.Tag.scope:type_name -> ateapi.TagScope
+ 34, // 41: ateapi.Tag.source_actor:type_name -> ateapi.ObjectRef
+ 14, // 42: ateapi.Atespace.metadata:type_name -> ateapi.ResourceMetadata
+ 14, // 43: ateapi.ActorTemplate.metadata:type_name -> ateapi.ResourceMetadata
+ 13, // 44: ateapi.ActorTemplate.worker_selector:type_name -> ateapi.Selector
+ 42, // 45: ateapi.ActorTemplate.containers:type_name -> ateapi.Container
+ 48, // 46: ateapi.ActorTemplate.volumes:type_name -> ateapi.Volume
+ 40, // 47: ateapi.ActorTemplate.snapshot_config:type_name -> ateapi.SnapshotConfig
+ 36, // 48: ateapi.ActorTemplate.resources:type_name -> ateapi.Resources
+ 39, // 49: ateapi.ActorTemplate.status:type_name -> ateapi.ActorTemplateStatus
+ 37, // 50: ateapi.Resources.limits:type_name -> ateapi.Limits
+ 34, // 51: ateapi.GoldenSnapshotStatus.golden_tag:type_name -> ateapi.ObjectRef
+ 147, // 52: ateapi.GoldenSnapshotStatus.snapshot_time:type_name -> google.protobuf.Timestamp
+ 38, // 53: ateapi.ActorTemplateStatus.golden_snapshot_status:type_name -> ateapi.GoldenSnapshotStatus
+ 2, // 54: ateapi.SnapshotConfig.preferred_fidelity:type_name -> ateapi.SnapshotFidelity
+ 41, // 55: ateapi.SnapshotConfig.golden_snapshot_config:type_name -> ateapi.GoldenSnapshotConfig
+ 5, // 56: ateapi.GoldenSnapshotConfig.mode:type_name -> ateapi.GoldenSnapshotMode
+ 45, // 57: ateapi.Container.env:type_name -> ateapi.EnvVar
+ 46, // 58: ateapi.Container.wakeup_probe:type_name -> ateapi.WakeupProbe
+ 58, // 59: ateapi.Container.volume_mounts:type_name -> ateapi.VolumeMount
+ 43, // 60: ateapi.Container.security_context:type_name -> ateapi.SecurityContext
+ 36, // 61: ateapi.Container.resources:type_name -> ateapi.Resources
+ 44, // 62: ateapi.SecurityContext.capabilities:type_name -> ateapi.Capabilities
+ 47, // 63: ateapi.WakeupProbe.http_get:type_name -> ateapi.HttpGetAction
+ 51, // 64: ateapi.Volume.durable_dir:type_name -> ateapi.DurableDirVolumeSource
+ 49, // 65: ateapi.Volume.external:type_name -> ateapi.ExternalVolumeSource
+ 53, // 66: ateapi.Volume.system_info:type_name -> ateapi.SystemInfoVolumeSource
+ 50, // 67: ateapi.Volume.image:type_name -> ateapi.ImageVolumeSource
+ 52, // 68: ateapi.ExternalVolumeSource.template:type_name -> ateapi.ExternalVolumeTemplate
+ 54, // 69: ateapi.SystemInfoVolumeSource.data_sources:type_name -> ateapi.SystemInfoDataSource
+ 55, // 70: ateapi.SystemInfoDataSource.actor_metadata:type_name -> ateapi.ActorMetadataDataSource
+ 57, // 71: ateapi.SystemInfoDataSource.trust_bundle:type_name -> ateapi.TrustBundleDataSource
+ 56, // 72: ateapi.ActorMetadataDataSource.items:type_name -> ateapi.ActorMetadataItem
+ 6, // 73: ateapi.ActorMetadataItem.field:type_name -> ateapi.ActorMetadataField
+ 33, // 74: ateapi.CreateAtespaceRequest.atespace:type_name -> ateapi.Atespace
+ 34, // 75: ateapi.GetAtespaceRequest.atespace:type_name -> ateapi.ObjectRef
+ 95, // 76: ateapi.ListAtespacesRequest.options:type_name -> ateapi.ListOptions
+ 33, // 77: ateapi.ListAtespacesResponse.atespaces:type_name -> ateapi.Atespace
+ 34, // 78: ateapi.DeleteAtespaceRequest.atespace:type_name -> ateapi.ObjectRef
+ 94, // 79: ateapi.DeleteAtespaceRequest.options:type_name -> ateapi.DeleteOptions
+ 35, // 80: ateapi.CreateActorTemplateRequest.actor_template:type_name -> ateapi.ActorTemplate
+ 34, // 81: ateapi.GetActorTemplateRequest.actor_template:type_name -> ateapi.ObjectRef
+ 95, // 82: ateapi.ListActorTemplatesRequest.options:type_name -> ateapi.ListOptions
+ 35, // 83: ateapi.ListActorTemplatesResponse.actor_templates:type_name -> ateapi.ActorTemplate
+ 34, // 84: ateapi.DeleteActorTemplateRequest.actor_template:type_name -> ateapi.ObjectRef
+ 94, // 85: ateapi.DeleteActorTemplateRequest.options:type_name -> ateapi.DeleteOptions
+ 34, // 86: ateapi.GetActorRequest.actor:type_name -> ateapi.ObjectRef
+ 17, // 87: ateapi.CreateActorRequest.actor:type_name -> ateapi.Actor
+ 17, // 88: ateapi.UpdateActorRequest.actor:type_name -> ateapi.Actor
+ 34, // 89: ateapi.SuspendActorRequest.actor:type_name -> ateapi.ObjectRef
+ 74, // 90: ateapi.SuspendActorRequest.options:type_name -> ateapi.SuspendOptions
+ 17, // 91: ateapi.SuspendActorResponse.actor:type_name -> ateapi.Actor
+ 7, // 92: ateapi.SuspendOptions.mode:type_name -> ateapi.DurabilityMode
+ 34, // 93: ateapi.ResumeActorRequest.actor:type_name -> ateapi.ObjectRef
+ 17, // 94: ateapi.ResumeActorResponse.actor:type_name -> ateapi.Actor
+ 34, // 95: ateapi.RevertActorRequest.actor:type_name -> ateapi.ObjectRef
+ 17, // 96: ateapi.RevertActorResponse.actor:type_name -> ateapi.Actor
+ 34, // 97: ateapi.DeleteActorRequest.actor:type_name -> ateapi.ObjectRef
+ 94, // 98: ateapi.DeleteActorRequest.options:type_name -> ateapi.DeleteOptions
+ 34, // 99: ateapi.GetActorEgressPolicyRequest.actor:type_name -> ateapi.ObjectRef
+ 34, // 100: ateapi.CreateActorEgressPolicyRequest.actor:type_name -> ateapi.ObjectRef
+ 18, // 101: ateapi.CreateActorEgressPolicyRequest.egress_policy:type_name -> ateapi.EgressPolicy
+ 34, // 102: ateapi.UpdateActorEgressPolicyRequest.actor:type_name -> ateapi.ObjectRef
+ 18, // 103: ateapi.UpdateActorEgressPolicyRequest.egress_policy:type_name -> ateapi.EgressPolicy
+ 34, // 104: ateapi.DeleteActorEgressPolicyRequest.actor:type_name -> ateapi.ObjectRef
+ 94, // 105: ateapi.DeleteActorEgressPolicyRequest.options:type_name -> ateapi.DeleteOptions
+ 34, // 106: ateapi.GetTagRequest.tag:type_name -> ateapi.ObjectRef
+ 34, // 107: ateapi.MintActorJwtRequest.actor:type_name -> ateapi.ObjectRef
+ 147, // 108: ateapi.MintActorJwtResponse.expire_time:type_name -> google.protobuf.Timestamp
+ 34, // 109: ateapi.MintActorCertificateRequest.actor:type_name -> ateapi.ObjectRef
+ 95, // 110: ateapi.ListTagsRequest.options:type_name -> ateapi.ListOptions
+ 32, // 111: ateapi.ListTagsResponse.tags:type_name -> ateapi.Tag
+ 32, // 112: ateapi.CreateTagRequest.tag:type_name -> ateapi.Tag
+ 32, // 113: ateapi.UpdateTagRequest.tag:type_name -> ateapi.Tag
+ 34, // 114: ateapi.DeleteTagRequest.tag:type_name -> ateapi.ObjectRef
+ 94, // 115: ateapi.DeleteTagRequest.options:type_name -> ateapi.DeleteOptions
+ 34, // 116: ateapi.ListWorkerActorAssignmentsRequest.worker:type_name -> ateapi.ObjectRef
+ 95, // 117: ateapi.ListWorkerActorAssignmentsRequest.options:type_name -> ateapi.ListOptions
+ 118, // 118: ateapi.ListWorkerActorAssignmentsResponse.actor_assignments:type_name -> ateapi.ActorAssignment
+ 95, // 119: ateapi.ListWorkersRequest.options:type_name -> ateapi.ListOptions
+ 112, // 120: ateapi.ListWorkersResponse.workers:type_name -> ateapi.Worker
+ 34, // 121: ateapi.GetWorkerRequest.worker:type_name -> ateapi.ObjectRef
+ 112, // 122: ateapi.CreateWorkerRequest.worker:type_name -> ateapi.Worker
+ 112, // 123: ateapi.UpdateWorkerRequest.worker:type_name -> ateapi.Worker
+ 34, // 124: ateapi.DeleteWorkerRequest.worker:type_name -> ateapi.ObjectRef
+ 94, // 125: ateapi.DeleteWorkerRequest.options:type_name -> ateapi.DeleteOptions
+ 34, // 126: ateapi.DrainWorkerRequest.worker:type_name -> ateapi.ObjectRef
+ 112, // 127: ateapi.DrainWorkerResponse.worker:type_name -> ateapi.Worker
+ 34, // 128: ateapi.MarkWorkerAvailableRequest.worker:type_name -> ateapi.ObjectRef
+ 112, // 129: ateapi.MarkWorkerAvailableResponse.worker:type_name -> ateapi.Worker
+ 34, // 130: ateapi.MarkWorkerUnavailableRequest.worker:type_name -> ateapi.ObjectRef
+ 112, // 131: ateapi.MarkWorkerUnavailableResponse.worker:type_name -> ateapi.Worker
+ 95, // 132: ateapi.ListActorsRequest.options:type_name -> ateapi.ListOptions
+ 17, // 133: ateapi.ListActorsResponse.actors:type_name -> ateapi.Actor
+ 14, // 134: ateapi.Worker.metadata:type_name -> ateapi.ResourceMetadata
+ 146, // 135: ateapi.Worker.labels:type_name -> ateapi.Worker.LabelsEntry
+ 113, // 136: ateapi.Worker.status:type_name -> ateapi.WorkerStatus
+ 8, // 137: ateapi.WorkerStatus.state:type_name -> ateapi.WorkerState
+ 117, // 138: ateapi.WorkerStatus.capacity:type_name -> ateapi.WorkerResources
+ 117, // 139: ateapi.WorkerStatus.allocated:type_name -> ateapi.WorkerResources
+ 114, // 140: ateapi.WorkerStatus.default_runtime:type_name -> ateapi.SandboxRuntime
+ 114, // 141: ateapi.WorkerStatus.restorable_runtimes:type_name -> ateapi.SandboxRuntime
+ 115, // 142: ateapi.SandboxRuntime.version:type_name -> ateapi.VersionedSandboxCompat
+ 116, // 143: ateapi.VersionedSandboxCompat.attributes:type_name -> ateapi.AttributeEntry
+ 36, // 144: ateapi.WorkerResources.resources:type_name -> ateapi.Resources
+ 14, // 145: ateapi.ActorAssignment.metadata:type_name -> ateapi.ResourceMetadata
+ 34, // 146: ateapi.ActorAssignment.actor:type_name -> ateapi.ObjectRef
+ 36, // 147: ateapi.ActorAssignment.resources:type_name -> ateapi.Resources
+ 34, // 148: ateapi.RegisterWorkerRequest.worker:type_name -> ateapi.ObjectRef
+ 117, // 149: ateapi.RegisterWorkerRequest.capacity:type_name -> ateapi.WorkerResources
+ 114, // 150: ateapi.RegisterWorkerRequest.default_runtime:type_name -> ateapi.SandboxRuntime
+ 114, // 151: ateapi.RegisterWorkerRequest.restorable_runtimes:type_name -> ateapi.SandboxRuntime
+ 112, // 152: ateapi.RegisterWorkerResponse.worker:type_name -> ateapi.Worker
+ 34, // 153: ateapi.MintWorkerActorCertificateRequest.actor:type_name -> ateapi.ObjectRef
+ 34, // 154: ateapi.MintWorkerActorCertificateRequest.worker:type_name -> ateapi.ObjectRef
+ 34, // 155: ateapi.MintWorkerActorJwtRequest.actor:type_name -> ateapi.ObjectRef
+ 147, // 156: ateapi.MintWorkerActorJwtResponse.expire_time:type_name -> google.protobuf.Timestamp
+ 34, // 157: ateapi.RequestActorSuspendRequest.worker:type_name -> ateapi.ObjectRef
+ 34, // 158: ateapi.RequestActorSuspendRequest.actor:type_name -> ateapi.ObjectRef
+ 17, // 159: ateapi.RequestActorSuspendResponse.actor:type_name -> ateapi.Actor
+ 34, // 160: ateapi.ReportActorCrashedRequest.worker:type_name -> ateapi.ObjectRef
+ 34, // 161: ateapi.ReportActorCrashedRequest.actor:type_name -> ateapi.ObjectRef
+ 17, // 162: ateapi.ReportActorCrashedResponse.actor:type_name -> ateapi.Actor
+ 34, // 163: ateapi.ReportActorSnapshotStateRequest.actor:type_name -> ateapi.ObjectRef
+ 131, // 164: ateapi.ReportActorSnapshotStateRequest.state:type_name -> ateapi.ActorSnapshotState
+ 132, // 165: ateapi.ActorSnapshotState.latest_local:type_name -> ateapi.SnapshotReport
+ 132, // 166: ateapi.ActorSnapshotState.uploading:type_name -> ateapi.SnapshotReport
+ 132, // 167: ateapi.ActorSnapshotState.latest_durable:type_name -> ateapi.SnapshotReport
+ 147, // 168: ateapi.ActorSnapshotState.upload_due_time:type_name -> google.protobuf.Timestamp
+ 2, // 169: ateapi.SnapshotReport.fidelity:type_name -> ateapi.SnapshotFidelity
+ 10, // 170: ateapi.SnapshotReport.object:type_name -> ateapi.ObjectSnapshot
+ 147, // 171: ateapi.SnapshotReport.local_commit_time:type_name -> google.protobuf.Timestamp
+ 147, // 172: ateapi.SnapshotReport.durable_complete_time:type_name -> google.protobuf.Timestamp
+ 14, // 173: ateapi.AccessPolicy.metadata:type_name -> ateapi.ResourceMetadata
+ 134, // 174: ateapi.AccessPolicy.bindings:type_name -> ateapi.Binding
+ 133, // 175: ateapi.CreateGlobalAccessPolicyRequest.access_policy:type_name -> ateapi.AccessPolicy
+ 133, // 176: ateapi.UpdateGlobalAccessPolicyRequest.access_policy:type_name -> ateapi.AccessPolicy
+ 34, // 177: ateapi.GetAtespaceAccessPolicyRequest.atespace:type_name -> ateapi.ObjectRef
+ 34, // 178: ateapi.CreateAtespaceAccessPolicyRequest.atespace:type_name -> ateapi.ObjectRef
+ 133, // 179: ateapi.CreateAtespaceAccessPolicyRequest.access_policy:type_name -> ateapi.AccessPolicy
+ 34, // 180: ateapi.UpdateAtespaceAccessPolicyRequest.atespace:type_name -> ateapi.ObjectRef
+ 133, // 181: ateapi.UpdateAtespaceAccessPolicyRequest.access_policy:type_name -> ateapi.AccessPolicy
+ 34, // 182: ateapi.DeleteAtespaceAccessPolicyRequest.atespace:type_name -> ateapi.ObjectRef
+ 94, // 183: ateapi.DeleteAtespaceAccessPolicyRequest.options:type_name -> ateapi.DeleteOptions
+ 9, // 184: ateapi.ResourceAnnotation.scope:type_name -> ateapi.Scope
+ 148, // 185: ateapi.resource:extendee -> google.protobuf.MessageOptions
+ 149, // 186: ateapi.method:extendee -> google.protobuf.MethodOptions
+ 142, // 187: ateapi.resource:type_name -> ateapi.ResourceAnnotation
+ 143, // 188: ateapi.method:type_name -> ateapi.MethodAnnotation
+ 69, // 189: ateapi.Control.GetActor:input_type -> ateapi.GetActorRequest
+ 70, // 190: ateapi.Control.CreateActor:input_type -> ateapi.CreateActorRequest
+ 71, // 191: ateapi.Control.UpdateActor:input_type -> ateapi.UpdateActorRequest
+ 72, // 192: ateapi.Control.SuspendActor:input_type -> ateapi.SuspendActorRequest
+ 75, // 193: ateapi.Control.ResumeActor:input_type -> ateapi.ResumeActorRequest
+ 77, // 194: ateapi.Control.RevertActor:input_type -> ateapi.RevertActorRequest
+ 79, // 195: ateapi.Control.DeleteActor:input_type -> ateapi.DeleteActorRequest
+ 80, // 196: ateapi.Control.GetActorEgressPolicy:input_type -> ateapi.GetActorEgressPolicyRequest
+ 81, // 197: ateapi.Control.CreateActorEgressPolicy:input_type -> ateapi.CreateActorEgressPolicyRequest
+ 82, // 198: ateapi.Control.UpdateActorEgressPolicy:input_type -> ateapi.UpdateActorEgressPolicyRequest
+ 83, // 199: ateapi.Control.DeleteActorEgressPolicy:input_type -> ateapi.DeleteActorEgressPolicyRequest
+ 85, // 200: ateapi.Control.MintActorJwt:input_type -> ateapi.MintActorJwtRequest
+ 87, // 201: ateapi.Control.MintActorCertificate:input_type -> ateapi.MintActorCertificateRequest
+ 91, // 202: ateapi.Control.CreateTag:input_type -> ateapi.CreateTagRequest
+ 84, // 203: ateapi.Control.GetTag:input_type -> ateapi.GetTagRequest
+ 89, // 204: ateapi.Control.ListTags:input_type -> ateapi.ListTagsRequest
+ 92, // 205: ateapi.Control.UpdateTag:input_type -> ateapi.UpdateTagRequest
+ 93, // 206: ateapi.Control.DeleteTag:input_type -> ateapi.DeleteTagRequest
+ 98, // 207: ateapi.Control.ListWorkers:input_type -> ateapi.ListWorkersRequest
+ 100, // 208: ateapi.Control.GetWorker:input_type -> ateapi.GetWorkerRequest
+ 101, // 209: ateapi.Control.CreateWorker:input_type -> ateapi.CreateWorkerRequest
+ 102, // 210: ateapi.Control.UpdateWorker:input_type -> ateapi.UpdateWorkerRequest
+ 103, // 211: ateapi.Control.DeleteWorker:input_type -> ateapi.DeleteWorkerRequest
+ 104, // 212: ateapi.Control.DrainWorker:input_type -> ateapi.DrainWorkerRequest
+ 106, // 213: ateapi.Control.MarkWorkerAvailable:input_type -> ateapi.MarkWorkerAvailableRequest
+ 108, // 214: ateapi.Control.MarkWorkerUnavailable:input_type -> ateapi.MarkWorkerUnavailableRequest
+ 96, // 215: ateapi.Control.ListWorkerActorAssignments:input_type -> ateapi.ListWorkerActorAssignmentsRequest
+ 110, // 216: ateapi.Control.ListActors:input_type -> ateapi.ListActorsRequest
+ 59, // 217: ateapi.Control.CreateAtespace:input_type -> ateapi.CreateAtespaceRequest
+ 60, // 218: ateapi.Control.GetAtespace:input_type -> ateapi.GetAtespaceRequest
+ 61, // 219: ateapi.Control.ListAtespaces:input_type -> ateapi.ListAtespacesRequest
+ 63, // 220: ateapi.Control.DeleteAtespace:input_type -> ateapi.DeleteAtespaceRequest
+ 64, // 221: ateapi.Control.CreateActorTemplate:input_type -> ateapi.CreateActorTemplateRequest
+ 65, // 222: ateapi.Control.GetActorTemplate:input_type -> ateapi.GetActorTemplateRequest
+ 66, // 223: ateapi.Control.ListActorTemplates:input_type -> ateapi.ListActorTemplatesRequest
+ 68, // 224: ateapi.Control.DeleteActorTemplate:input_type -> ateapi.DeleteActorTemplateRequest
+ 135, // 225: ateapi.Control.GetGlobalAccessPolicy:input_type -> ateapi.GetGlobalAccessPolicyRequest
+ 136, // 226: ateapi.Control.CreateGlobalAccessPolicy:input_type -> ateapi.CreateGlobalAccessPolicyRequest
+ 137, // 227: ateapi.Control.UpdateGlobalAccessPolicy:input_type -> ateapi.UpdateGlobalAccessPolicyRequest
+ 138, // 228: ateapi.Control.GetAtespaceAccessPolicy:input_type -> ateapi.GetAtespaceAccessPolicyRequest
+ 139, // 229: ateapi.Control.CreateAtespaceAccessPolicy:input_type -> ateapi.CreateAtespaceAccessPolicyRequest
+ 140, // 230: ateapi.Control.UpdateAtespaceAccessPolicy:input_type -> ateapi.UpdateAtespaceAccessPolicyRequest
+ 141, // 231: ateapi.Control.DeleteAtespaceAccessPolicy:input_type -> ateapi.DeleteAtespaceAccessPolicyRequest
+ 119, // 232: ateapi.WorkerService.RegisterWorker:input_type -> ateapi.RegisterWorkerRequest
+ 121, // 233: ateapi.WorkerService.MintWorkerActorCertificate:input_type -> ateapi.MintWorkerActorCertificateRequest
+ 123, // 234: ateapi.WorkerService.MintWorkerActorJwt:input_type -> ateapi.MintWorkerActorJwtRequest
+ 125, // 235: ateapi.WorkerService.RequestActorSuspend:input_type -> ateapi.RequestActorSuspendRequest
+ 129, // 236: ateapi.WorkerService.ReportActorSnapshotState:input_type -> ateapi.ReportActorSnapshotStateRequest
+ 127, // 237: ateapi.WorkerService.ReportActorCrashed:input_type -> ateapi.ReportActorCrashedRequest
+ 17, // 238: ateapi.Control.GetActor:output_type -> ateapi.Actor
+ 17, // 239: ateapi.Control.CreateActor:output_type -> ateapi.Actor
+ 17, // 240: ateapi.Control.UpdateActor:output_type -> ateapi.Actor
+ 73, // 241: ateapi.Control.SuspendActor:output_type -> ateapi.SuspendActorResponse
+ 76, // 242: ateapi.Control.ResumeActor:output_type -> ateapi.ResumeActorResponse
+ 78, // 243: ateapi.Control.RevertActor:output_type -> ateapi.RevertActorResponse
+ 17, // 244: ateapi.Control.DeleteActor:output_type -> ateapi.Actor
+ 18, // 245: ateapi.Control.GetActorEgressPolicy:output_type -> ateapi.EgressPolicy
+ 18, // 246: ateapi.Control.CreateActorEgressPolicy:output_type -> ateapi.EgressPolicy
+ 18, // 247: ateapi.Control.UpdateActorEgressPolicy:output_type -> ateapi.EgressPolicy
+ 18, // 248: ateapi.Control.DeleteActorEgressPolicy:output_type -> ateapi.EgressPolicy
+ 86, // 249: ateapi.Control.MintActorJwt:output_type -> ateapi.MintActorJwtResponse
+ 88, // 250: ateapi.Control.MintActorCertificate:output_type -> ateapi.MintActorCertificateResponse
+ 32, // 251: ateapi.Control.CreateTag:output_type -> ateapi.Tag
+ 32, // 252: ateapi.Control.GetTag:output_type -> ateapi.Tag
+ 90, // 253: ateapi.Control.ListTags:output_type -> ateapi.ListTagsResponse
+ 32, // 254: ateapi.Control.UpdateTag:output_type -> ateapi.Tag
+ 32, // 255: ateapi.Control.DeleteTag:output_type -> ateapi.Tag
+ 99, // 256: ateapi.Control.ListWorkers:output_type -> ateapi.ListWorkersResponse
+ 112, // 257: ateapi.Control.GetWorker:output_type -> ateapi.Worker
+ 112, // 258: ateapi.Control.CreateWorker:output_type -> ateapi.Worker
+ 112, // 259: ateapi.Control.UpdateWorker:output_type -> ateapi.Worker
+ 112, // 260: ateapi.Control.DeleteWorker:output_type -> ateapi.Worker
+ 105, // 261: ateapi.Control.DrainWorker:output_type -> ateapi.DrainWorkerResponse
+ 107, // 262: ateapi.Control.MarkWorkerAvailable:output_type -> ateapi.MarkWorkerAvailableResponse
+ 109, // 263: ateapi.Control.MarkWorkerUnavailable:output_type -> ateapi.MarkWorkerUnavailableResponse
+ 97, // 264: ateapi.Control.ListWorkerActorAssignments:output_type -> ateapi.ListWorkerActorAssignmentsResponse
+ 111, // 265: ateapi.Control.ListActors:output_type -> ateapi.ListActorsResponse
+ 33, // 266: ateapi.Control.CreateAtespace:output_type -> ateapi.Atespace
+ 33, // 267: ateapi.Control.GetAtespace:output_type -> ateapi.Atespace
+ 62, // 268: ateapi.Control.ListAtespaces:output_type -> ateapi.ListAtespacesResponse
+ 33, // 269: ateapi.Control.DeleteAtespace:output_type -> ateapi.Atespace
+ 35, // 270: ateapi.Control.CreateActorTemplate:output_type -> ateapi.ActorTemplate
+ 35, // 271: ateapi.Control.GetActorTemplate:output_type -> ateapi.ActorTemplate
+ 67, // 272: ateapi.Control.ListActorTemplates:output_type -> ateapi.ListActorTemplatesResponse
+ 35, // 273: ateapi.Control.DeleteActorTemplate:output_type -> ateapi.ActorTemplate
+ 133, // 274: ateapi.Control.GetGlobalAccessPolicy:output_type -> ateapi.AccessPolicy
+ 133, // 275: ateapi.Control.CreateGlobalAccessPolicy:output_type -> ateapi.AccessPolicy
+ 133, // 276: ateapi.Control.UpdateGlobalAccessPolicy:output_type -> ateapi.AccessPolicy
+ 133, // 277: ateapi.Control.GetAtespaceAccessPolicy:output_type -> ateapi.AccessPolicy
+ 133, // 278: ateapi.Control.CreateAtespaceAccessPolicy:output_type -> ateapi.AccessPolicy
+ 133, // 279: ateapi.Control.UpdateAtespaceAccessPolicy:output_type -> ateapi.AccessPolicy
+ 133, // 280: ateapi.Control.DeleteAtespaceAccessPolicy:output_type -> ateapi.AccessPolicy
+ 120, // 281: ateapi.WorkerService.RegisterWorker:output_type -> ateapi.RegisterWorkerResponse
+ 122, // 282: ateapi.WorkerService.MintWorkerActorCertificate:output_type -> ateapi.MintWorkerActorCertificateResponse
+ 124, // 283: ateapi.WorkerService.MintWorkerActorJwt:output_type -> ateapi.MintWorkerActorJwtResponse
+ 126, // 284: ateapi.WorkerService.RequestActorSuspend:output_type -> ateapi.RequestActorSuspendResponse
+ 130, // 285: ateapi.WorkerService.ReportActorSnapshotState:output_type -> ateapi.ReportActorSnapshotStateResponse
+ 128, // 286: ateapi.WorkerService.ReportActorCrashed:output_type -> ateapi.ReportActorCrashedResponse
+ 238, // [238:287] is the sub-list for method output_type
+ 189, // [189:238] is the sub-list for method input_type
+ 187, // [187:189] is the sub-list for extension type_name
+ 185, // [185:187] is the sub-list for extension extendee
+ 0, // [0:185] is the sub-list for field type_name
}
func init() { file_ateapi_proto_init() }
@@ -10442,7 +10695,7 @@ func file_ateapi_proto_init() {
GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
RawDescriptor: unsafe.Slice(unsafe.StringData(file_ateapi_proto_rawDesc), len(file_ateapi_proto_rawDesc)),
NumEnums: 10,
- NumMessages: 133,
+ NumMessages: 137,
NumExtensions: 2,
NumServices: 2,
},
diff --git a/pkg/proto/ateapipb/ateapi.proto b/pkg/proto/ateapipb/ateapi.proto
index 8eefdd647e..9fe2be328d 100644
--- a/pkg/proto/ateapipb/ateapi.proto
+++ b/pkg/proto/ateapipb/ateapi.proto
@@ -57,8 +57,8 @@ service Control {
option (method).resource = "Actor";
}
- // Delete an actor. Only suspended or crashed actors can be deleted unless
- // any_state is set.
+ // Delete an actor. A running actor's workload is terminated, its volumes
+ // detached, and its worker released before the record is removed.
rpc DeleteActor(DeleteActorRequest) returns (Actor) {
option (method).resource = "Actor";
}
@@ -102,7 +102,8 @@ service Control {
// Tag the external snapshot a suspended Actor holds. The tag gets its own
// copy of that snapshot, so suspending or deleting the Actor afterwards
- // cannot collect it.
+ // cannot collect it. When the Actor's latest snapshot is not uploaded yet,
+ // atelet uploads it first, so the call can take as long as that upload.
rpc CreateTag(CreateTagRequest) returns (Tag) {
option (method).resource = "Tag";
}
@@ -478,7 +479,21 @@ message ResourceMetadata {
google.protobuf.Timestamp update_time = 6;
}
-message ExternalVolume {
+// ExternalVolumes is the external volume state recorded on an actor, wrapped
+// so later additions do not reshape ActorStatus.
+message ExternalVolumes {
+ // volumes has one entry per external volume of the template. Each template
+ // volume appears at most once.
+ //
+ // +k8s:optional
+ // +k8s:maxItems=32 # matches the template's volumes bound
+ // +k8s:listType=map
+ // +k8s:listMapKey=name
+ repeated ExternalVolumeStatus volumes = 1;
+}
+
+// ExternalVolumeStatus is one actor's view of an external volume it mounts.
+message ExternalVolumeStatus {
// Name of the volume specified in the actor template. Template volume
// names are DNS labels.
//
@@ -521,7 +536,7 @@ message ExternalVolume {
//
// +k8s:optional
// +k8s:customValidation # at most 40 KiB in total, ten times the CSI limit for map fields
- map volume_context = 5;
+ map volume_context = 4;
}
message Actor {
@@ -540,7 +555,7 @@ message Actor {
// +k8s:required
// +k8s:subfield(atespace)=+k8s:required
// +k8s:mutable
- ObjectRef actor_template = 4;
+ ObjectRef actor_template = 2;
// worker_selector is the per-actor placement constraint. The scheduler
// evaluates the AND of this selector and the template's workerSelector to
@@ -549,7 +564,7 @@ message Actor {
// Changes take effect on the next ResumeActor call.
//
// +k8s:optional
- Selector worker_selector = 5;
+ Selector worker_selector = 3;
// The tag specified by the caller at CreateActor to seed this Actor from an
// ActorSnapshot. Unset if the Actor was not created from a snapshot. Set
@@ -558,13 +573,13 @@ message Actor {
// +k8s:optional
// +k8s:subfield(atespace)=+k8s:required
// +k8s:immutable
- ObjectRef source_tag = 6;
+ ObjectRef source_tag = 4;
// status is the system-managed state of the Actor. It is updated by the
// server and ignored on input. It is always present in output.
//
// +k8s:optional
- ActorStatus status = 7;
+ ActorStatus status = 5;
}
// EgressPolicy is an egress policy resource nested under an Actor. An Actor has
@@ -852,9 +867,9 @@ enum ActorState {
ACTOR_STATE_RUNNING = 2;
ACTOR_STATE_SUSPENDING = 3;
ACTOR_STATE_SUSPENDED = 4;
- ACTOR_STATE_CRASHED = 7;
- ACTOR_STATE_DELETING = 8;
- ACTOR_STATE_REVERTING = 9;
+ ACTOR_STATE_CRASHED = 5;
+ ACTOR_STATE_DELETING = 6;
+ ACTOR_STATE_REVERTING = 7;
// Keep this in sync with ActorStatus.state's maximum.
}
@@ -863,7 +878,7 @@ message ActorStatus {
//
// +k8s:required
// +k8s:minimum=1
- // +k8s:maximum=9 # keep this in sync with the ActorState enum
+ // +k8s:maximum=7 # keep this in sync with the ActorState enum
ActorState state = 1;
// worker_assignment points at the worker currently hosting this Actor.
@@ -873,22 +888,19 @@ message ActorStatus {
// +k8s:update=NoModify # can be set and cleared, but not changed in place
WorkerAssignment worker_assignment = 2;
- // Volumes attached to the actor. These volumes only live as long as the actor.
- // They are deleted when the actor is deleted. Each template volume
- // appears at most once.
+ // external_volumes is the state of the external volumes attached to the
+ // actor. These volumes only live as long as the actor and are deleted with
+ // it.
//
// +ate:preview=ExternalVolumes
// +k8s:optional
- // +k8s:maxItems=32 # matches the template's volumes bound
- // +k8s:listType=map
- // +k8s:listMapKey=name
- repeated ExternalVolume external_volumes = 7;
+ ExternalVolumes external_volumes = 3;
// crash records why and when the Actor entered CRASHED. It is set with the
// CRASHED state and cleared when a revert returns the Actor to SUSPENDED.
//
// +k8s:optional
- ActorCrash crash = 9;
+ ActorCrash crash = 4;
// assigned_node is the Kubernetes node the Actor is currently attached to
// (hosting its active worker or its node-local snapshots). Unset when the
@@ -896,7 +908,7 @@ message ActorStatus {
//
// +k8s:optional
// +k8s:format=k8s-long-name
- string assigned_node = 10;
+ string assigned_node = 5;
// snapshots holds the Actor's active and in-progress snapshot generations.
//
@@ -905,7 +917,7 @@ message ActorStatus {
// +k8s:listType=atomic
// +k8s:unique=map
// +k8s:listMapKey=generation
- repeated Snapshot snapshots = 11;
+ repeated Snapshot snapshots = 6;
// last_assigned_generation tracks the last snapshot generation assigned for
// this Actor during a Suspend or Pause request.
@@ -916,7 +928,7 @@ message ActorStatus {
// +k8s:alpha(since: "0.0")=+k8s:monotonic # TODO: get rid of alpha prefix
// +k8s:minimum=0
// +k8s:update=NoUnset
- int32 last_assigned_generation = 12;
+ int32 last_assigned_generation = 7;
}
// ActorCrash describes the failure that moved an Actor to CRASHED.
@@ -1144,7 +1156,10 @@ message ActorTemplate {
// +k8s:listMapKey=name
repeated Volume volumes = 4;
- // +k8s:required
+ // snapshot_config selects what actor snapshots capture and where they are
+ // stored. Optional: when omitted, every field takes its default.
+ //
+ // +k8s:optional
SnapshotConfig snapshot_config = 5;
// resources is the actor's compute size. Both cpu and memory limits are
@@ -1153,10 +1168,10 @@ message ActorTemplate {
//
// +k8s:required
// +k8s:customValidation # cpu and memory limits are both set
- Resources resources = 7;
+ Resources resources = 6;
// +k8s:optional
- ActorTemplateStatus status = 8;
+ ActorTemplateStatus status = 7;
}
message Resources {
@@ -1215,9 +1230,12 @@ message ActorTemplateStatus {
// stored.
message SnapshotConfig {
// storage_location is the base object-storage URI snapshots of actors on
- // this version are stored under. Required.
+ // this version are stored under. If unset, ate-api-server's
+ // --snapshot-location is used. Requires the PerTemplateSnapshotLocation preview
+ // gate; ignored without it.
//
- // +k8s:required
+ // +ate:preview=PerTemplateSnapshotLocation
+ // +k8s:optional
// +k8s:maxLength=1024
// +k8s:customValidation # Validate URI
string storage_location = 1;
@@ -1239,7 +1257,7 @@ message SnapshotConfig {
// Golden snapshots are enabled when this config is omitted.
//
// +k8s:optional
- GoldenSnapshotConfig golden_snapshot_config = 5;
+ GoldenSnapshotConfig golden_snapshot_config = 3;
}
// GoldenSnapshotMode selects whether a template builds a golden snapshot.
@@ -1428,28 +1446,40 @@ message Volume {
// +k8s:format=k8s-short-name
string name = 1;
- // Exactly one of durable_dir / external_volume_template / image /
- // system_info must be set.
+ // Exactly one of durable_dir / external / image / system_info must be
+ // set.
//
// +k8s:optional
// +k8s:unionMember
DurableDirVolumeSource durable_dir = 2;
+ // external is a CSI volume: provisioned for this actor from a template.
+ //
// +ate:preview=ExternalVolumes
// +k8s:ifDisabled(ExternalVolumes)=+k8s:forbidden
// +k8s:optional
// +k8s:unionMember
- ExternalVolumeTemplate external_volume_template = 3;
+ ExternalVolumeSource external = 3;
// +k8s:optional
// +k8s:unionMember
- SystemInfoVolumeSource system_info = 5;
+ SystemInfoVolumeSource system_info = 4;
// image mounts the contents of an OCI image, read-only.
//
// +k8s:optional
// +k8s:unionMember
- ImageVolumeSource image = 6;
+ ImageVolumeSource image = 5;
+}
+
+// ExternalVolumeSource is a CSI volume provisioned for the actor from a template.
+message ExternalVolumeSource {
+ // template provisions a volume for this actor alone; it is deleted with the
+ // actor.
+ //
+ // +k8s:optional
+ // +k8s:unionMember
+ ExternalVolumeTemplate template = 1;
}
// ImageVolumeSource mounts the contents of an OCI image, read-only. The
@@ -1576,7 +1606,7 @@ message TrustBundleDataSource {
// +k8s:eachVal=+k8s:minLength=1
// +k8s:eachVal=+k8s:maxLength=253
// +k8s:listType=set
- repeated string names = 3;
+ repeated string names = 1;
// path must be a clean relative Unix path: at most 16 '/'-separated
// segments, none of them empty, '.' or '..', and no NUL byte.
@@ -1586,9 +1616,6 @@ message TrustBundleDataSource {
// +k8s:maxLength=255
// +k8s:customValidation # projected-path rule shared with atelet, see internal/volumepath
string path = 2;
-
- reserved 1;
- reserved "name";
}
// VolumeMount mounts a named Volume into a container.
@@ -1804,15 +1831,10 @@ message DeleteActorRequest {
// +k8s:subfield(atespace)=+k8s:required
ObjectRef actor = 1;
- // If true, delete the actor regardless of its state.
- //
- // +k8s:optional
- bool any_state = 2;
-
// Optional per-delete preconditions.
//
// +k8s:optional
- DeleteOptions options = 3;
+ DeleteOptions options = 2;
}
// GetActorEgressPolicyRequest identifies an Actor's egress policy.
@@ -1899,7 +1921,7 @@ message MintActorJwtRequest {
// Must be a valid actor that currently exists according to the actor store.
//
// +k8s:required
- ObjectRef actor = 5;
+ ObjectRef actor = 1;
// The audiences the minted JWT is bound to. Tokens are only issued with
// audience bindings, so at least one is required.
@@ -1908,14 +1930,14 @@ message MintActorJwtRequest {
// +k8s:maxItems=16 # guardrail; tokens realistically bind a handful of audiences
// +k8s:listType=set
// +k8s:eachVal=+k8s:maxLength=512 # audiences are caller-defined URIs; bound only
- repeated string audiences = 1;
+ repeated string audiences = 2;
// Lifetime of the JWT, in seconds.
//
// +k8s:required
// +k8s:minimum=300
// +k8s:maximum=3600
- int64 expiration_seconds = 8;
+ int64 expiration_seconds = 3;
}
// Response message for MintActorJwt.
@@ -1952,14 +1974,14 @@ message MintActorCertificateRequest {
// Must be a valid actor that currently exists according to the actor store.
//
// +k8s:required
- ObjectRef actor = 6;
+ ObjectRef actor = 1;
// The UID of the actor --- used to guard against deletion and recreation of
// an actor with the same name.
//
// +k8s:required
// +k8s:format=k8s-uuid
- string actor_uid = 7;
+ string actor_uid = 2;
// Request contains DER encoded bytes of a x509 certificate signing request.
// The signer will ignore the contents of the CSR except to extract the
@@ -1967,7 +1989,7 @@ message MintActorCertificateRequest {
//
// +k8s:required
// +k8s:maxBytes=16384
- bytes certificate_signing_request = 2;
+ bytes certificate_signing_request = 3;
}
// Response message for MintActorCertificate.
@@ -2296,7 +2318,7 @@ message Worker {
// +k8s:maxLength=2048
// +k8s:customValidation # a SPIFFE ID
// +k8s:immutable
- string worker_spiffe_id = 13;
+ string worker_spiffe_id = 8;
// sandbox_class mirrors the WorkerPool's default sandbox class
//
@@ -2304,7 +2326,7 @@ message Worker {
// +k8s:maxLength=63
// +k8s:customValidation # gvisor or microvm
// +k8s:immutable
- string sandbox_class = 8;
+ string sandbox_class = 9;
// labels mirror the WorkerPool object's Kubernetes labels, which selectors
// match against.
@@ -2315,7 +2337,7 @@ message Worker {
// +k8s:maxProperties=64
// +k8s:eachKey=+k8s:format=k8s-label-key
// +k8s:eachVal=+k8s:format=k8s-label-value
- map labels = 9;
+ map labels = 10;
// epoch counts the runs of the worker pod's worker container: 1 for its first
// run and one more for each restart. 0 means it has not been reported.
@@ -2329,7 +2351,7 @@ message Worker {
// +k8s:minimum=0
// +k8s:update=NoUnset
// +k8s:customValidation
- int64 epoch = 12;
+ int64 epoch = 11;
// Output-only server-managed state. Absent from Create/Update request
// payloads; whatever a request carries here is ignored. A client moves
@@ -2337,7 +2359,7 @@ message Worker {
// MarkWorkerUnavailable, and the assignments are the scheduler's.
//
// +k8s:optional
- WorkerStatus status = 11;
+ WorkerStatus status = 12;
}
enum WorkerState {
@@ -2503,7 +2525,7 @@ message ActorAssignment {
//
// +k8s:required
// +k8s:beta(since: "0.0")=+k8s:subfield(atespace)=+k8s:forbidden # TODO: get rid of beta prefix
- ResourceMetadata metadata = 6;
+ ResourceMetadata metadata = 1;
// actor is the Actor that is assigned to the Worker.
//
@@ -2518,13 +2540,13 @@ message ActorAssignment {
// The resources allocated to the Actor when it was assigned.
//
// +k8s:optional
- Resources resources = 5;
+ Resources resources = 4;
// worker_epoch is the Worker's epoch when the Actor was bound.
//
// +k8s:optional
// +k8s:minimum=0
- int64 worker_epoch = 7;
+ int64 worker_epoch = 5;
}
// WorkerService is how a Worker tells the control plane what it observes. It is
@@ -2602,6 +2624,18 @@ service WorkerService {
// node. Returns FAILED_PRECONDITION or INVALID_ARGUMENT when the report
// conflicts with the record; don't retry.
rpc ReportActorSnapshotState(ReportActorSnapshotStateRequest) returns (ReportActorSnapshotStateResponse);
+
+ // ReportActorCrashed tells the control plane that the sandbox of an Actor
+ // the calling Worker hosts died on its own. Nothing else notices: the
+ // control plane only learns of a dead sandbox when an operation on the Actor
+ // fails.
+ //
+ // A RUNNING Actor is moved to ACTOR_STATE_CRASHED and its Worker is
+ // released. An Actor in any other state is left alone and the report is
+ // refused as FailedPrecondition: an operation in flight owns the Actor and
+ // fails on its own if the sandbox is gone. Authorized like
+ // RequestActorSuspend.
+ rpc ReportActorCrashed(ReportActorCrashedRequest) returns (ReportActorCrashedResponse);
}
message RegisterWorkerRequest {
@@ -2719,7 +2753,9 @@ message MintWorkerActorJwtResponse {
//
// * sub: `worker-for-actor//`.
// * `worker-for-actor.ate.dev`: A JSON object with the actor's `atespace`,
- // `actorName`, and `actorUID`.
+ // `actorName`, and `actorUID`, and the `node` the token was minted for,
+ // the one hosting the actor at that moment. A relying party refuses the
+ // token once the actor has left that node.
//
// A relying party that trusts actor JWTs must not accept this token as the
// actor, and vice versa.
@@ -2762,6 +2798,39 @@ message RequestActorSuspendResponse {
Actor actor = 1;
}
+message ReportActorCrashedRequest {
+ // The Worker making the report. atespace is always empty; Workers are
+ // global-scoped. atelet fills this in from its caller's certificate.
+ //
+ // +k8s:required
+ // +k8s:beta(since: "0.0")=+k8s:subfield(atespace)=+k8s:forbidden # TODO: get rid of beta prefix
+ ObjectRef worker = 1;
+
+ // The Actor whose sandbox died. It must be assigned to worker.
+ //
+ // +k8s:required
+ // +k8s:subfield(atespace)=+k8s:required
+ ObjectRef actor = 2;
+
+ // The UID of the Actor the Worker believes it is hosting, as for
+ // RequestActorSuspend.
+ //
+ // +k8s:required
+ // +k8s:format=k8s-uuid
+ string actor_uid = 3;
+
+ // What the Worker saw, recorded in the Actor's crash message.
+ //
+ // +k8s:required
+ // +k8s:maxLength=4096
+ string message = 4;
+}
+
+message ReportActorCrashedResponse {
+ // The Actor as the report left it.
+ Actor actor = 1;
+}
+
message ReportActorSnapshotStateRequest {
// The Actor the report is about.
//
diff --git a/pkg/proto/ateapipb/ateapi_grpc.pb.go b/pkg/proto/ateapipb/ateapi_grpc.pb.go
index a40dd7f2f2..c21225257c 100644
--- a/pkg/proto/ateapipb/ateapi_grpc.pb.go
+++ b/pkg/proto/ateapipb/ateapi_grpc.pb.go
@@ -100,8 +100,8 @@ type ControlClient interface {
// snapshot, can be reverted. A suspended actor without a node-local snapshot
// is already at its durable snapshot, so there is nothing to revert.
RevertActor(ctx context.Context, in *RevertActorRequest, opts ...grpc.CallOption) (*RevertActorResponse, error)
- // Delete an actor. Only suspended or crashed actors can be deleted unless
- // any_state is set.
+ // Delete an actor. A running actor's workload is terminated, its volumes
+ // detached, and its worker released before the record is removed.
DeleteActor(ctx context.Context, in *DeleteActorRequest, opts ...grpc.CallOption) (*Actor, error)
// Get the egress policy resource nested under an Actor.
GetActorEgressPolicy(ctx context.Context, in *GetActorEgressPolicyRequest, opts ...grpc.CallOption) (*EgressPolicy, error)
@@ -124,7 +124,8 @@ type ControlClient interface {
MintActorCertificate(ctx context.Context, in *MintActorCertificateRequest, opts ...grpc.CallOption) (*MintActorCertificateResponse, error)
// Tag the external snapshot a suspended Actor holds. The tag gets its own
// copy of that snapshot, so suspending or deleting the Actor afterwards
- // cannot collect it.
+ // cannot collect it. When the Actor's latest snapshot is not uploaded yet,
+ // atelet uploads it first, so the call can take as long as that upload.
CreateTag(ctx context.Context, in *CreateTagRequest, opts ...grpc.CallOption) (*Tag, error)
// Get a Tag.
GetTag(ctx context.Context, in *GetTagRequest, opts ...grpc.CallOption) (*Tag, error)
@@ -664,8 +665,8 @@ type ControlServer interface {
// snapshot, can be reverted. A suspended actor without a node-local snapshot
// is already at its durable snapshot, so there is nothing to revert.
RevertActor(context.Context, *RevertActorRequest) (*RevertActorResponse, error)
- // Delete an actor. Only suspended or crashed actors can be deleted unless
- // any_state is set.
+ // Delete an actor. A running actor's workload is terminated, its volumes
+ // detached, and its worker released before the record is removed.
DeleteActor(context.Context, *DeleteActorRequest) (*Actor, error)
// Get the egress policy resource nested under an Actor.
GetActorEgressPolicy(context.Context, *GetActorEgressPolicyRequest) (*EgressPolicy, error)
@@ -688,7 +689,8 @@ type ControlServer interface {
MintActorCertificate(context.Context, *MintActorCertificateRequest) (*MintActorCertificateResponse, error)
// Tag the external snapshot a suspended Actor holds. The tag gets its own
// copy of that snapshot, so suspending or deleting the Actor afterwards
- // cannot collect it.
+ // cannot collect it. When the Actor's latest snapshot is not uploaded yet,
+ // atelet uploads it first, so the call can take as long as that upload.
CreateTag(context.Context, *CreateTagRequest) (*Tag, error)
// Get a Tag.
GetTag(context.Context, *GetTagRequest) (*Tag, error)
@@ -1890,6 +1892,7 @@ const (
WorkerService_MintWorkerActorJwt_FullMethodName = "/ateapi.WorkerService/MintWorkerActorJwt"
WorkerService_RequestActorSuspend_FullMethodName = "/ateapi.WorkerService/RequestActorSuspend"
WorkerService_ReportActorSnapshotState_FullMethodName = "/ateapi.WorkerService/ReportActorSnapshotState"
+ WorkerService_ReportActorCrashed_FullMethodName = "/ateapi.WorkerService/ReportActorCrashed"
)
// WorkerServiceClient is the client API for WorkerService service.
@@ -1967,6 +1970,17 @@ type WorkerServiceClient interface {
// node. Returns FAILED_PRECONDITION or INVALID_ARGUMENT when the report
// conflicts with the record; don't retry.
ReportActorSnapshotState(ctx context.Context, in *ReportActorSnapshotStateRequest, opts ...grpc.CallOption) (*ReportActorSnapshotStateResponse, error)
+ // ReportActorCrashed tells the control plane that the sandbox of an Actor
+ // the calling Worker hosts died on its own. Nothing else notices: the
+ // control plane only learns of a dead sandbox when an operation on the Actor
+ // fails.
+ //
+ // A RUNNING Actor is moved to ACTOR_STATE_CRASHED and its Worker is
+ // released. An Actor in any other state is left alone and the report is
+ // refused as FailedPrecondition: an operation in flight owns the Actor and
+ // fails on its own if the sandbox is gone. Authorized like
+ // RequestActorSuspend.
+ ReportActorCrashed(ctx context.Context, in *ReportActorCrashedRequest, opts ...grpc.CallOption) (*ReportActorCrashedResponse, error)
}
type workerServiceClient struct {
@@ -2027,6 +2041,16 @@ func (c *workerServiceClient) ReportActorSnapshotState(ctx context.Context, in *
return out, nil
}
+func (c *workerServiceClient) ReportActorCrashed(ctx context.Context, in *ReportActorCrashedRequest, opts ...grpc.CallOption) (*ReportActorCrashedResponse, error) {
+ cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...)
+ out := new(ReportActorCrashedResponse)
+ err := c.cc.Invoke(ctx, WorkerService_ReportActorCrashed_FullMethodName, in, out, cOpts...)
+ if err != nil {
+ return nil, err
+ }
+ return out, nil
+}
+
// WorkerServiceServer is the server API for WorkerService service.
// All implementations must embed UnimplementedWorkerServiceServer
// for forward compatibility.
@@ -2102,6 +2126,17 @@ type WorkerServiceServer interface {
// node. Returns FAILED_PRECONDITION or INVALID_ARGUMENT when the report
// conflicts with the record; don't retry.
ReportActorSnapshotState(context.Context, *ReportActorSnapshotStateRequest) (*ReportActorSnapshotStateResponse, error)
+ // ReportActorCrashed tells the control plane that the sandbox of an Actor
+ // the calling Worker hosts died on its own. Nothing else notices: the
+ // control plane only learns of a dead sandbox when an operation on the Actor
+ // fails.
+ //
+ // A RUNNING Actor is moved to ACTOR_STATE_CRASHED and its Worker is
+ // released. An Actor in any other state is left alone and the report is
+ // refused as FailedPrecondition: an operation in flight owns the Actor and
+ // fails on its own if the sandbox is gone. Authorized like
+ // RequestActorSuspend.
+ ReportActorCrashed(context.Context, *ReportActorCrashedRequest) (*ReportActorCrashedResponse, error)
mustEmbedUnimplementedWorkerServiceServer()
}
@@ -2127,6 +2162,9 @@ func (UnimplementedWorkerServiceServer) RequestActorSuspend(context.Context, *Re
func (UnimplementedWorkerServiceServer) ReportActorSnapshotState(context.Context, *ReportActorSnapshotStateRequest) (*ReportActorSnapshotStateResponse, error) {
return nil, status.Error(codes.Unimplemented, "method ReportActorSnapshotState not implemented")
}
+func (UnimplementedWorkerServiceServer) ReportActorCrashed(context.Context, *ReportActorCrashedRequest) (*ReportActorCrashedResponse, error) {
+ return nil, status.Error(codes.Unimplemented, "method ReportActorCrashed not implemented")
+}
func (UnimplementedWorkerServiceServer) mustEmbedUnimplementedWorkerServiceServer() {}
func (UnimplementedWorkerServiceServer) testEmbeddedByValue() {}
@@ -2238,6 +2276,24 @@ func _WorkerService_ReportActorSnapshotState_Handler(srv interface{}, ctx contex
return interceptor(ctx, in, info, handler)
}
+func _WorkerService_ReportActorCrashed_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
+ in := new(ReportActorCrashedRequest)
+ if err := dec(in); err != nil {
+ return nil, err
+ }
+ if interceptor == nil {
+ return srv.(WorkerServiceServer).ReportActorCrashed(ctx, in)
+ }
+ info := &grpc.UnaryServerInfo{
+ Server: srv,
+ FullMethod: WorkerService_ReportActorCrashed_FullMethodName,
+ }
+ handler := func(ctx context.Context, req interface{}) (interface{}, error) {
+ return srv.(WorkerServiceServer).ReportActorCrashed(ctx, req.(*ReportActorCrashedRequest))
+ }
+ return interceptor(ctx, in, info, handler)
+}
+
// WorkerService_ServiceDesc is the grpc.ServiceDesc for WorkerService service.
// It's only intended for direct use with grpc.RegisterService,
// and not to be introspected or modified (even as a copy)
@@ -2265,6 +2321,10 @@ var WorkerService_ServiceDesc = grpc.ServiceDesc{
MethodName: "ReportActorSnapshotState",
Handler: _WorkerService_ReportActorSnapshotState_Handler,
},
+ {
+ MethodName: "ReportActorCrashed",
+ Handler: _WorkerService_ReportActorCrashed_Handler,
+ },
},
Streams: []grpc.StreamDesc{},
Metadata: "ateapi.proto",
diff --git a/pkg/proto/objectstorepb/v1/objectstore.pb.go b/pkg/proto/objectstorepb/v1/objectstore.pb.go
index 9a59e63c57..aa34678e55 100644
--- a/pkg/proto/objectstorepb/v1/objectstore.pb.go
+++ b/pkg/proto/objectstorepb/v1/objectstore.pb.go
@@ -41,8 +41,12 @@ type FetchSnapshotRequest struct {
// The snapshot to read, e.g. gs:///.
SnapshotUri string `protobuf:"bytes,1,opt,name=snapshot_uri,json=snapshotUri,proto3" json:"snapshot_uri,omitempty"`
// The JWT of the actor that owns the snapshot, for plugins that authorize
- // storage access per actor. Substrate does not set it yet, so it is
- // currently always empty.
+ // storage access per actor. It is a worker-for-actor token ate-api-server
+ // minted for the node hosting the actor, with the audience
+ // "objectstore-plugin.ate.dev". That audience belongs to this API, so every
+ // plugin serving it verifies the same value. The token names the node it
+ // was minted for and is refused once the actor has left it. Substrate does
+ // not set the field yet, so it is currently always empty.
ActorJwt string `protobuf:"bytes,2,opt,name=actor_jwt,json=actorJwt,proto3" json:"actor_jwt,omitempty"`
// Full local path of the directory the files are written into. It must
// exist.
@@ -153,8 +157,12 @@ type UploadSnapshotRequest struct {
// The snapshot to write, e.g. gs:///.
SnapshotUri string `protobuf:"bytes,1,opt,name=snapshot_uri,json=snapshotUri,proto3" json:"snapshot_uri,omitempty"`
// The JWT of the actor that owns the snapshot, for plugins that authorize
- // storage access per actor. Substrate does not set it yet, so it is
- // currently always empty.
+ // storage access per actor. It is a worker-for-actor token ate-api-server
+ // minted for the node hosting the actor, with the audience
+ // "objectstore-plugin.ate.dev". That audience belongs to this API, so every
+ // plugin serving it verifies the same value. The token names the node it
+ // was minted for and is refused once the actor has left it. Substrate does
+ // not set the field yet, so it is currently always empty.
ActorJwt string `protobuf:"bytes,2,opt,name=actor_jwt,json=actorJwt,proto3" json:"actor_jwt,omitempty"`
// Full local path of the directory the files are read from.
LocalPath string `protobuf:"bytes,3,opt,name=local_path,json=localPath,proto3" json:"local_path,omitempty"`
diff --git a/pkg/proto/objectstorepb/v1/objectstore.proto b/pkg/proto/objectstorepb/v1/objectstore.proto
index f14141d6b2..e35fb9fae1 100644
--- a/pkg/proto/objectstorepb/v1/objectstore.proto
+++ b/pkg/proto/objectstorepb/v1/objectstore.proto
@@ -54,8 +54,12 @@ message FetchSnapshotRequest {
string snapshot_uri = 1;
// The JWT of the actor that owns the snapshot, for plugins that authorize
- // storage access per actor. Substrate does not set it yet, so it is
- // currently always empty.
+ // storage access per actor. It is a worker-for-actor token ate-api-server
+ // minted for the node hosting the actor, with the audience
+ // "objectstore-plugin.ate.dev". That audience belongs to this API, so every
+ // plugin serving it verifies the same value. The token names the node it
+ // was minted for and is refused once the actor has left it. Substrate does
+ // not set the field yet, so it is currently always empty.
string actor_jwt = 2 [ debug_redact = true ];
// Full local path of the directory the files are written into. It must
@@ -74,8 +78,12 @@ message UploadSnapshotRequest {
string snapshot_uri = 1;
// The JWT of the actor that owns the snapshot, for plugins that authorize
- // storage access per actor. Substrate does not set it yet, so it is
- // currently always empty.
+ // storage access per actor. It is a worker-for-actor token ate-api-server
+ // minted for the node hosting the actor, with the audience
+ // "objectstore-plugin.ate.dev". That audience belongs to this API, so every
+ // plugin serving it verifies the same value. The token names the node it
+ // was minted for and is refused once the actor has left it. Substrate does
+ // not set the field yet, so it is currently always empty.
string actor_jwt = 2 [ debug_redact = true ];
// Full local path of the directory the files are read from.
diff --git a/tools/apitool/exemptions.json b/tools/apitool/exemptions.json
index d1283b1cec..c9e5c86eae 100644
--- a/tools/apitool/exemptions.json
+++ b/tools/apitool/exemptions.json
@@ -1,9 +1,4 @@
[
- {
- "rule": "delete-request-shape",
- "subject": "ateapi.Control.DeleteActor",
- "message": "request field \"any_state\" is not allowed - want only \"actor\" ateapi.ObjectRef and optional \"options\" ateapi.DeleteOptions"
- },
{
"rule": "documented",
"subject": "ateapi.Actor",
@@ -39,11 +34,6 @@
"subject": "ateapi.ActorTemplate.containers",
"message": "field has no doc comment"
},
- {
- "rule": "documented",
- "subject": "ateapi.ActorTemplate.snapshot_config",
- "message": "field has no doc comment"
- },
{
"rule": "documented",
"subject": "ateapi.ActorTemplate.status",
@@ -149,11 +139,6 @@
"subject": "ateapi.DrainWorkerRequest",
"message": "message has no doc comment"
},
- {
- "rule": "documented",
- "subject": "ateapi.ExternalVolume",
- "message": "message has no doc comment"
- },
{
"rule": "documented",
"subject": "ateapi.GetActorRequest",
@@ -339,11 +324,6 @@
"subject": "ateapi.Volume",
"message": "message has no doc comment"
},
- {
- "rule": "documented",
- "subject": "ateapi.Volume.external_volume_template",
- "message": "field has no doc comment"
- },
{
"rule": "documented",
"subject": "ateapi.Volume.system_info",
@@ -383,10 +363,5 @@
"rule": "documented",
"subject": "ateapi.WorkerStatus.state",
"message": "field has no doc comment"
- },
- {
- "rule": "resource-metadata",
- "subject": "ateapi.ActorAssignment",
- "message": "\"metadata\" field is number 6, want 1"
}
]
diff --git a/tools/apitool/internal/model/model.go b/tools/apitool/internal/model/model.go
index a40f551890..aee7cb5d31 100644
--- a/tools/apitool/internal/model/model.go
+++ b/tools/apitool/internal/model/model.go
@@ -178,7 +178,7 @@ type Field struct {
// TypeDisplay is always a ready-to-render string. For example, "string"
// for ResourceMetadata.name, "int64" for ResourceMetadata.version,
// "repeated ExternalVolume" for ActorStatus.external_volumes,
- // "map" for SandboxAssets.assets, "ActorState" for
+ // "map" for Selector.match_labels, "ActorState" for
// ActorStatus.state, or "google.protobuf.Timestamp" for
// ResourceMetadata.create_time.
TypeDisplay string
@@ -203,11 +203,11 @@ type Field struct {
// MapValueKind and MapValueFullName mirror Kind/TypeFullName, but for a
// map field's value type. Both are empty for a field that isn't a map.
- // For example, for SandboxAssets.assets (`map
- // assets = 2;`), MapValueKind is MessageKind and MapValueFullName is
- // "ateapi.ArchAssets"; for Selector.match_labels (`map
+ // For example, for Selector.match_labels (`map
// match_labels = 1;`), MapValueKind is StringKind and MapValueFullName
- // is empty, since the map's value is a scalar.
+ // is empty, since the map's value is a scalar; for a map with message
+ // values, MapValueKind is MessageKind and MapValueFullName is the
+ // message's full name.
MapValueKind protoreflect.Kind
MapValueFullName protoreflect.FullName
diff --git a/tools/setup-gcp/README.md b/tools/setup-gcp/README.md
index 81803576db..35cac2714d 100644
--- a/tools/setup-gcp/README.md
+++ b/tools/setup-gcp/README.md
@@ -66,7 +66,8 @@ go run ./tools/setup-gcp enable apis [flags]
Creates a GKE cluster configured for Agent Substrate (with Workload Identity,
required Kubernetes beta APIs, and Managed OpenTelemetry enabled, and the
-Filestore CSI driver disabled).
+Filestore CSI driver disabled). An existing cluster keeps its Filestore
+add-on setting.
> [!WARNING]
> Agent Substrate requires PodCertificateRequest and ClusterTrustBundle APIs.
diff --git a/tools/setup-gcp/cmd/cluster.go b/tools/setup-gcp/cmd/cluster.go
index c3386272de..9225a7c356 100644
--- a/tools/setup-gcp/cmd/cluster.go
+++ b/tools/setup-gcp/cmd/cluster.go
@@ -107,6 +107,7 @@ func buildCreateClusterRequest(parent string, cfg *Config) *containerpb.CreateCl
Subnetwork: cfg.Subnetwork,
NetworkConfig: networkConfig,
ManagedOpentelemetryConfig: &containerpb.ManagedOpenTelemetryConfig{Scope: containerpb.ManagedOpenTelemetryConfig_COLLECTION_AND_INSTRUMENTATION_COMPONENTS.Enum()},
+ // Only new clusters; an existing cluster's add-on is left alone.
AddonsConfig: &containerpb.AddonsConfig{
GcpFilestoreCsiDriverConfig: &containerpb.GcpFilestoreCsiDriverConfig{
Enabled: false,
@@ -116,10 +117,6 @@ func buildCreateClusterRequest(parent string, cfg *Config) *containerpb.CreateCl
}
}
-func filestoreCsiDriverEnabled(cluster *containerpb.Cluster) bool {
- return cluster.GetAddonsConfig().GetGcpFilestoreCsiDriverConfig().GetEnabled()
-}
-
// nestedVirtualizationEnabled reports whether any node pool exposes /dev/kvm,
// which micro-VM workers need. Any pool, not every pool: a cluster that keeps
// its KVM nodes in a second pool added by hand is a normal arrangement.
@@ -305,31 +302,6 @@ func createClusterIdempotent(ctx context.Context, cfg *Config) error {
slog.Info("Cluster ManagedOpentelemetryConfig match perfectly.", slog.String("cluster", cfg.ClusterName))
}
- if filestoreCsiDriverEnabled(cluster) {
- slog.Info("Mismatch in Filestore CSI driver config",
- slog.Bool("current", true),
- slog.Bool("expected", false))
- slog.Info("Updating cluster to disable Filestore CSI driver...")
- op, err := client.UpdateCluster(ctx, &containerpb.UpdateClusterRequest{
- Name: clusterName,
- Update: &containerpb.ClusterUpdate{
- DesiredAddonsConfig: &containerpb.AddonsConfig{
- GcpFilestoreCsiDriverConfig: &containerpb.GcpFilestoreCsiDriverConfig{
- Enabled: false,
- },
- },
- },
- })
- if err != nil {
- return fmt.Errorf("update cluster filestore csi driver: %w", err)
- }
- if err := waitContainerOperation(ctx, client, op.Name, cfg); err != nil {
- return err
- }
- } else {
- slog.Info("Cluster Filestore CSI driver match perfectly.", slog.String("cluster", cfg.ClusterName))
- }
-
// Nested virtualization is fixed at node creation, so an existing cluster
// cannot be updated into it: the pool has to be recreated. Report the drift
// rather than let the flag look applied when it was not.
diff --git a/tools/setup-gcp/cmd/cluster_test.go b/tools/setup-gcp/cmd/cluster_test.go
index ab46f4451a..e59d98e2fa 100644
--- a/tools/setup-gcp/cmd/cluster_test.go
+++ b/tools/setup-gcp/cmd/cluster_test.go
@@ -335,62 +335,6 @@ func TestNodeSysctlsConfigured(t *testing.T) {
}
}
-func TestFilestoreCsiDriverEnabled(t *testing.T) {
- tests := []struct {
- name string
- cluster *containerpb.Cluster
- want bool
- }{
- {
- name: "nil cluster",
- cluster: nil,
- want: false,
- },
- {
- name: "nil addons config",
- cluster: &containerpb.Cluster{},
- want: false,
- },
- {
- name: "nil filestore config",
- cluster: &containerpb.Cluster{
- AddonsConfig: &containerpb.AddonsConfig{},
- },
- want: false,
- },
- {
- name: "filestore disabled",
- cluster: &containerpb.Cluster{
- AddonsConfig: &containerpb.AddonsConfig{
- GcpFilestoreCsiDriverConfig: &containerpb.GcpFilestoreCsiDriverConfig{
- Enabled: false,
- },
- },
- },
- want: false,
- },
- {
- name: "filestore enabled",
- cluster: &containerpb.Cluster{
- AddonsConfig: &containerpb.AddonsConfig{
- GcpFilestoreCsiDriverConfig: &containerpb.GcpFilestoreCsiDriverConfig{
- Enabled: true,
- },
- },
- },
- want: true,
- },
- }
-
- for _, tt := range tests {
- t.Run(tt.name, func(t *testing.T) {
- if got := filestoreCsiDriverEnabled(tt.cluster); got != tt.want {
- t.Errorf("filestoreCsiDriverEnabled() = %v, want %v", got, tt.want)
- }
- })
- }
-}
-
func TestValidateBootDisk(t *testing.T) {
tests := []struct {
name string