Repository navigation
Sync upstream resource API and sandbox lifecycle updates - #92
Merged
Merged
Conversation
…rate#2831) Background uploads and detaches ran under atelet's lifetime context with no per-call deadline. A wedged object-store plugin call or stalled stream could occupy a background worker indefinitely. Each background upload attempt, including the one a background detach runs first, now runs under a timeout that scales based on the snapshot size. --snapshot-upload-timeout is the initial timeout value --snapshot-upload-est-throughput-mib is used to add on additional time based off of the snapshot size When an upload exceeds the timeout, atelet cancels that attempt, logs a warning, and retries the actor after the exponential backoff. Foreground Detach RPCs remain bounded by their caller's context. Part of agent-substrate#2069 and agent-substrate#2571 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
) Part of agent-substrate#1660. - `docs/egress-credential-injection.md` gains an Actor JWTs section: the `actorJwt` policy fields and their limits, the claims a minted token carries, and how long the gateway reuses a token before minting another. - The outcome table gains rows for actor JWT entries: 403 when the actor no longer exists, 503 when ateapi is unreachable, and 500 (Envoy) or 403 (agentgateway) for any other mint failure. The intro and "How it works" mention the ateapi path. - An operational note says tokens follow the actor's name, so a recreated actor can get its predecessor's token until the refresh point. The metrics from agent-substrate#2317 aren't mentioned, since they aren't on main yet. Testing: docs only. The YAML example parses with `kubectl ate`'s policy parser, and each documented limit matches ateapi's validation, both checked with throwaway tests. - [ ] Tests pass - [x] Appropriate changes to documentation are included in the PR
… maintainer feedback (agent-substrate#2812) Agent-written PRs keep getting the same review feedback. I had agents mine the last few months of review comments (my own and other maintainers') for recurring themes, and turned the ones we didn't already cover into rules. This was manually reviewed, iterated with adversarial agent reviews, and manually edited again. - AGENTS.md: new "Rules for contributors" section up top: AI disclosure, self-review before asking for review, how to write comments/commits/descriptions (with the concrete smell to avoid), plus scope, code and test rules - CONTRIBUTING.md / COLLABORATING.md: small fixes can go straight to a PR, larger changes need an issue first, search open PRs before starting, and tests are for behavior rather than coverage - Workload isolation now mentions micro-VM and points at docs/security.md (only mentioned gvisor) - PR template: much shorter so it fits in the GitHub editor, with a "why", and an explicit AI disclosure section. The lowest quality PRs have lacked disclosure in-line with our guidelines. The PR template (AI disclosure line, a "why" prompt) will be a follow-up. AI-assisted: agents mined the review history and drafted the rules, I edited and reviewed all of it. - [x] Appropriate changes to documentation are included in the PR - [x] Test pass
…e#2906) A restore mints the actor certificate over a fresh connection to the node-local credential broker. Under a wide restore wave the node can refuse or drop that connection; the mint failed with Unavailable, the restore failed, and a healthy actor ended CRASHED. `MintWorkerCertificate` now retries up to four times with a short back-off while the error is Unavailable; every other error stays final. The TODO in the file about keeping one broker connection stands; this change only stops a transient failure from crashing the actor. Tests: two Unavailable answers then success (three broker calls); a FailedPrecondition is not retried. The retry test fails on the old code. `go test -race ./internal/atunnel/egress/` passes; the tunnel and micro-VM worker packages pass on Linux. Seen on bare metal in a wave of 1536 restores at once: four restores crashed this way and nothing else in the wave failed. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR (none needed)
…ubstrate#2743) Under bursts of concurrent micro-VM restores on one node, a freshly unpaused guest can take more than 5s to schedule its `kata-agent` and answer the `ReseedRandomDev` ttrpc call, even after `dialAgentRetry` completed the hybrid-vsock handshake. The 5s timeout in `reseedGuestCRNG` then fails `RestoreWorkload` and crashes the actor. This PR gives `reseedGuestCRNG` a single 60s deadline (`reseedTimeout`) and adds unit tests in `cmd/ateworker-microvm/restore_test.go` for the deadline and error propagation. It also adds a `reseed` duration (the kata-agent dial plus the reseed, the head of `wakeup_probe`) to the `Actor restore phases` log, from agent-substrate#2878. Why 60s rather than 15s (from @dims in agent-substrate#2878): on a heavily loaded host, the agent dial before the reseed already redials for up to 15s, and the slowest actors take more than 15s from resume to a reseeded guest. The deadline is a cap, not a wait: a healthy guest answers in milliseconds, so 60s costs nothing on the normal path and still fails a dead guest within a minute. #### How I tested it - `go test -race ./cmd/ateworker-microvm/...` - `./hack/verify/gofmt.sh` and `./hack/verify/boilerplate.sh` - In the 5,000-actor `nano-personal-agent` micro-VM benchmark campaign, a 15s reseed timeout plus one retry took these crashes from ~1 in 500–1,000 restores to 0 in later runs. - From agent-substrate#2878: on a nested-virtualization host, a burst of 64 restores crashed 3 at 5s and ran clean at 60s. On bare metal, 5,976 restores in waves of up to 384 reseeded in 0.00s median and 0.28s max. - [x] Tests pass - [ ] Appropriate changes to documentation are included in the PR Co-authored-by: Davanum Srinivas <davanum@gmail.com>
…gent-substrate#2904) ## Summary Today's `teardownActor` flow: 1. calls `client.Shutdown()`: two HTTP calls, `PUT /api/v1/vm.shutdown` and `PUT /api/v1/vmm.shutdown`, sent over the chv REST socket. 2. calls `ra.chProc.Stop()`, which SIGKILLs chv and waits until it is reaped. 3. calls `cleanupSandboxState`, which kills any leftover processes for the actor. Step 1 is unnecessary, since steps 2 and 3 already kill the processes. ## Why it is safe - Checkpoint already finishes before `teardownActor` runs. - `vm.shutdown` doesn't notify the guest, so to the guest it is the same as SIGKILL. - When `ra == nil`, there is no `ra.chProc`, but step 3 still cleans up the processes. ## Testing - `go vet` and `go test ./cmd/ateworker-microvm/...` pass. - Run → Checkpoint (MEMORY) → Restore: the restored guest resumes where it stopped (a 1 Hz counter continues at N+1). chv and virtiofsd are gone after they are killed, before `cleanupSandboxState` runs. - `ra == nil`: SIGKILLed ateworker with a restored VM running, restarted it, and sent `TerminateWorkload`. Step 2 was skipped, and `cleanupSandboxState` killed the orphaned chv and virtiofsd. - [x] Tests pass - [ ] Appropriate changes to documentation are included in the PR
…nt-substrate#2881) A suspended actor with a local snapshot is pinned to its node. While that node was full, its resume failed with `ResourceExhausted` even when other nodes had room, until atelet detached the actor on its own after `--snapshot-detach-delay`. Resume now detaches the actor itself, with `DetachActor` from agent-substrate#2852, and restores it on another node. Part of agent-substrate#589. This covers a full node. A node that is gone, the scale-down case, is still open. #### How the move works Assignment still runs pinned to the node first. When it finds no room there, and a worker on another node has room, resume: 1. Calls `DetachActor` on the pinned node. atelet uploads the latest snapshot and deletes the local copies. 2. Records the detach, which marks the upload `COMPLETED` and clears the locality and `assigned_node`. 3. Assigns a worker on any node and restores from the durable snapshot. `DetachActor` is synchronous. If the snapshot isn't durable yet, the resume waits for atelet to upload it. #### What the restore keeps On a worker of the same runtime, the restore keeps the snapshot's fidelity. On a worker of another runtime, a MEMORY snapshot restores at ROOTFS, as for any durable-only actor since agent-substrate#2880. Process memory is lost, but rootfs writes and volumes are kept. ROOTFS and VOLUMES snapshots keep their fidelity on any worker of the same sandbox class. A template can't yet refuse the downgrade; agent-substrate#2818 would add that. #### When the actor doesn't move - No worker anywhere has room. The resume returns `ResourceExhausted` as before, without calling atelet. - The detach fails. The resume fails, and the actor stays SUSPENDED on its node, not crashed, since it can still resume there. - The node is gone. `DetachActor` can't reach atelet, so the resume returns `Unavailable` and the actor stays pinned. ```mermaid sequenceDiagram participant C as client participant A as ate-api-server participant L1 as atelet on node1 (full) participant O as object storage participant L2 as atelet on node2 C->>A: ResumeActor A->>A: assign on node1: no room A->>A: node2 has room A->>L1: DetachActor L1->>O: upload latest snapshot L1->>L1: delete local copies L1-->>A: state with latest_durable only A->>A: mark COMPLETED, clear locality and assigned_node A->>A: reload restore source, assign on node2 A->>L2: Restore(durable snapshot) L2->>O: download L2-->>A: ok A-->>C: RUNNING on node2 ``` #### Design choices - **Any runtime counts as room.** - Memory is an optimization for GA, so a worker that can't restore the snapshot's process memory still triggers the move, and the actor restores there at ROOTFS. - Assignment still prefers a worker that keeps it. - **No claim before the detach.** - Claiming a worker first would leave the claim and the snapshot record disagreeing if ate-api-server crashed before the detach. - In this order, every step ends in a state resume already handles. - Two actors detached for one free place race for it, and the loser has paid an upload, but it ends durable-only and its next resume schedules it onto any node without another detach. #### Also worth knowing - A concurrent update fails the record write with `Aborted`, and the next resume detaches again; atelet returns the same state. If the node has room by then, the resume restores there from the upload instead, and the record keeps the upload `PENDING`, because the detach cancelled atelet's report. - `DetachActor` errors map as the tag workflow maps `PromoteSnapshot` errors, plus atelet's `Aborted` passed through. - A resume that detaches records two scheduler samples, `no_capacity` then `assigned`. A dashboard that reads `no_capacity` as a failed resume will count these. - No API changes. - `TestResumeActor_RelocatesAfterDetach` checked `ResourceExhausted` at a point where node2 already had room, which now detaches. The check moved to before node2 exists. - The functional fake's `DetachActor` returns the named actor's last checkpointed snapshot, not the last of any actor.
…gent-substrate#2912) A bare number in tests.yaml (`actorCPU: 1`) parses as an int. The orchestrator appended it to the deploy command unconverted, and the run helper quotes every argument with shlex before executing, which rejects a non-string with `expected string or bytes-like object, got 'int'`. The workload deploy then never started. Convert the value with `str()`, as the other numeric knobs already are, and add a test that an integer actorCPU renders as `--actor-cpu 1`.
…oad buffers (agent-substrate#2835) Part of agent-substrate#2553. Related to agent-substrate#2502 and agent-substrate#2708. Large GCS snapshots (`PutSparseFile` → `writeSparsePart`) previously compressed each 64 MiB part as one unsized streaming zstd frame with freshly allocated buffers. `agent-substrate#2502`'s `parZstdReader` only parallelizes frames whose size is declared or bounded at `≤ 16 MiB`; a part written as one streaming frame of 64 MiB or more is neither, so restore goes serial from the first part. Meanwhile, switching parts to the worker-pool `parZstd` with one worker uses ~77 MiB peak per part (2.4× over `agent-substrate#2708`'s 32 MiB `serialWorkerMemory` budget). This PR: - Adds an inline mode to `parZstd` for `workers == 1`: no goroutines; each `≤ 8 MiB` chunk is compressed on the caller's goroutine into a frame that declares its size. `writeSparsePart` uses `newParZstd(dst, 1)`, and small files get it too. - Pools the 8 MiB chunk, 8.5 MiB frame, and 1 MiB sparse copy buffers, and removes unused `newSerialZstd`. - Makes `Write` and a second `Close` after `Close` return `os.ErrClosed`. - Adds `BenchmarkWriteSparsePart`, `BenchmarkDecodeSparse`, and format tests, including one that pins the output byte for byte to per-chunk `EncodeAll`. ### Results (`-count=10`, 512 MiB populated / 2 GiB sparse) | Benchmark | `main` | `agent-substrate#2502` as-is | **this PR** | |---|---|---|---| | **Write 64 MiB part:** time / peak heap (32 MiB budget) | 148 ms / 9.8 MiB | 133 ms / **76.9 MiB** ❌ | 144 ms / **26.4 MiB** ✅ | | **Write 64 MiB part:** frames with content size | 0 / 1 | 8 / 9 | **9 / 9** | | **Read 512 MiB:** time (this PR + `agent-substrate#2502`: 254 ms, 3.8×) | 971 ms | 285 ms (3.4×) | **552 ms (1.76×)** | | **`ConcurrentSnapshotUploads` (`shared_512MiB`):** `B/op` / peak heap¹ | 1,326 / 272 MiB | 1,326 / 286 MiB | **70 / 196 MiB** | - **Serial restore is 1.76× faster even without `agent-substrate#2502`:** sized frames decode straight into a buffer of the declared size (`runtime.memmove` drops from 40% to 9% of decode CPU). - **Stress case (`all_80_admitted`)¹:** batch time roughly halves (`57.6 → ~29 ms`) and peak heap drops about a third (`1,994 → 1,330 MiB`). - **Cluster A/B (`sweperf`, 150u on `c3-highcpu-192-metal`, with `agent-substrate#2502`):** microVM `restore_golden_download` −31% at p50 (`1,095 → 752 ms`) and −63% at p95 (`3,419 → 1,255 ms`); `restore_p50_s` −29% (`945 → 669 ms`). ¹ This PR's upload numbers are from a later session on the same machine than the `main` and `agent-substrate#2502` columns, so treat the deltas as approximate. ### Compatibility & follow-up - Snapshot format is unchanged; old and new snapshots decode byte-exact on both readers. - `agent-substrate#2502`'s `newParZstd(dst, 1)` in `writeSparsePart` lands as-is; only `TestReadZstdFrameBoundsUnsizedSmallFrames` needs to build its unsized frame with `zstd.NewWriter(nil).EncodeAll`. - Follow-up (now that `agent-substrate#2848` is merged): 1-worker uploads could reserve `serialWorkerMemory` instead of `parallelWorkerMemory`. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR (none needed)
…l ate update actor (agent-substrate#2814) Fixes agent-substrate#2758 1. Allows repointing `actor_template` on any `SUSPENDED` actor even when it holds a node-local snapshot. With async suspend, users have no way to control whether a suspended actor is still holding a node-local snapshot ("paused" vs "suspended" to durable storage), so rejecting template upgrades when a local snapshot is present makes `UpdateActor` very hard for users to use in practice. 2. Adds the `kubectl ate update actor` subcommand to update an existing Actor's `actor_template` and/or `worker_selector` via the `UpdateActor` RPC (read-modify-write over `GetActor` + `UpdateActor`).
…he fleet had no room for (agent-substrate#2915) Locust spawns the fleet in one ramp and every session begins at step 1, so the sessions reach each later step together, in a wave the ramp alone spreads over seconds. With 1300 sessions pinned 13 to a node and a step that dwells for 47 s, the wave asked every worker for more slots at once than it had, and ateapi answered about half the ResumeActor calls with "no worker has room for the actor". A session that got that answer dropped the step until its next think gap, so the fleet's full moments showed up as failed rows and skipped steps rather than as wake latency. --agentsession-phase-in D makes each VU wait a uniform random share of D before creating its actor, once, so the fleet spreads out over the window and stays spread through the script. A wake the fleet had no room for (ResumeActor ResourceExhausted or Unavailable, or the router's 503, 504 and 429 faces of them) is now retried with 50 ms to 1 s backoff, as glutton's spawn path does, for --agentsession-resume-retry-budget (default 60 s). Each attempt is its own ResumeActor or WakeFirstTouch sample; a new TimeToWake row records the whole wake from the first attempt to the ping that found the actor serving, so the wait for a slot is measured instead of lost. When the budget runs out the step is given up as before. WakeFirstTouch keeps its meaning, so earlier runs stay comparable. - [ x ] Tests pass - [ x ] Appropriate changes to documentation are included in the PR Written with help of AI. I have reviewed the code.
…atic-mtls ingress auth mode (agent-substrate#2390) This PR can be broken into the following parts: * Defining a way for e2e tests running in Github Actions to get a client certificate the router understands. (The test code creates a PodCertificateRequest that doesn't correspond to any real pod). * Switching the router deployed for e2e tests into static-mtls mode. * Adding a new e2e test to verify static-mtls mode * Converting all other e2e tests that speak to the router to present a client certificate.
…gent-substrate#2093) The syncer now deregisters the Worker of a pod whose WorkerPool is gone, and reconciles a pool's pods when the pool is deleted. It waits for the pool cache to sync first. Fixes agent-substrate#2054 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR (none needed)
…memoryBytes, capacityBytes (agent-substrate#2769) This replaces the resource.Quantity strings in the API with plain integers, so ate-api-server no longer parses quantities and every hop compares numbers. Removes a performance bottleneck in scheduling and makes the API ~unambiguous. This came about after a lot of back and forth with @thockin and @juli4n about the tradeoffs. - ActorTemplate and container limits: `resources.limits` is now `{cpuMillicores, memoryBytes}` instead of a name/quantity list - Worker capacity/allocated and actor assignments use the same `ResourceList` - external volumes: `capacity` is now `capacityBytes` - the atelet and ateworker RPCs carry the same `Resources` type and field names, instead of `cpu_milli` / `ResourceLimits.cpu_millis` it's still `limits`, we'll add `requests` next to it later. `kubectl ate get workers` still prints memory with binary suffixes. templates and docs are converted, with the friendly unit as a comment (`memoryBytes: 536870912 # 512Mi`). benchmarking/workloads/deploy.sh still takes `--actor-cpu 250m` / `--actor-memory 1536Mi` and converts. #### Breaking change ActorTemplate manifests must be updated: ```yaml resources: limits: cpuMillicores: 1000 memoryBytes: 536870912 # 512Mi ``` and external volumes use `capacityBytes: 1073741824` instead of `capacity: 1Gi`. no compatibility with the old form, per our pre-1.0 policy. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR This PR was AI assisted. I have reviewed the changes.
Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
EItanya
force-pushed
the
sync/upstream-main-20261020
branch
from
October 20, 2026 18:39
b51e642 to
6029331
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
13 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bring the fork up to upstream main at
9ee109bab(14 commits), preserving both histories and the agentgateway configuration.Includes integer resource limits and volume capacities, worker deregistration on pool deletion, snapshot upload bounds and compression improvements, actor template updates, and ingress test changes. The merge retains both duplicate-header test cases, updates the fork's router-client callers, adds ReplicaSet read access to the Helm controller role, and sets the Helm E2E ingress mode for agentgateway.
Breaking change: upgrade components and API clients together. Actor templates now use
resources.limits.cpuMillicoresandmemoryBytes; external volume templates usecapacityBytes.Validation: 36 Helm tests, chart lint, manifest rendering, CRD parity, controller RBAC comparison, and focused Go race checks passed. The networking, ingress-auth, and credential suites compile; local cluster E2E was not run. Full local Go tests and repository verification were blocked by temporary disk exhaustion, including a serial Go retry. CI's first run identified one missed router-client caller in the credential suite; that is fixed and compiles locally. CI is rerunning on the fix.
AI use: Codex prepared the merge, reviewed the integration changes, and ran the local checks.
Merge with a merge commit to retain upstream ancestry.