Sync upstream resource API and sandbox lifecycle updates

.github/PULL_REQUEST_TEMPLATE.md

@@ -1,25 +1,13 @@
-> The PR title becomes the release note. Write it for users: what changed for them, not how the code changed.
+<!-- The PR title becomes the release note. Write it for users. -->
 
-#### Breaking change
+Why is this needed? (Or link the issue. Open one first for features, API, install or CI changes.)
 
-<!--
+Fixes #
 
-If users must act when they upgrade, describe what they must do, and ask for the `breaking-change` label.
-Otherwise delete this section.
+<!-- e.g. "AI-assisted, I reviewed and tested it." No AI co-author trailers. -->
+AI use:
 
-What is a breaking change? Something that requires customer code/configuration changes to make their agents work.
-But more broadly it might include any behavior change that customers were relying on (Hyrum’s law). Examples of
-breaking changes:
-
-- Moving from 1:1 pods to actors to multiple actors in one pod.
-- Changing actor primary keys or identity formats.
-- Adding authorization or authentication where none was required before.
-
--->
-
-Fixes #<issue_number_goes_here>
-
-> It's a good idea to open an issue first for discussion.
+<!-- Breaking change? Describe what users must do on upgrade and ask for the `breaking-change` label. -->
 
 - [ ] Tests pass
-- [ ] Appropriate changes to documentation are included in the PR
+- [ ] Docs updated

.github/workflows/helm-e2e.yaml

@@ -25,6 +25,7 @@ jobs:
     env:
       VERSION: helm-e2e
       E2E_ATENET_DATAPLANE: agentgateway
+      E2E_INGRESS_AUTH_MODE: deprecated-insecure
       E2E_CREDENTIAL_PROVIDER: "1"
       # Retain external-volume coverage after the preview gate is introduced.
       ATE_PREVIEW: ALL

.github/workflows/pr-workflow.yaml

@@ -173,6 +173,15 @@ jobs:
       # --preview, so tests of a disabled gate skip themselves (see
       # internal/e2e/preview.go).
       ATE_PREVIEW: ${{ matrix.preview == 'on' && 'ALL' || '' }}
+      # Job-wide for the same reason, and E2E_INGRESS_AUTH_MODE has to match
+      # it so the suites dial the router's mTLS listeners. The agentgateway
+      # dataplane does not support static-mtls.
+      ATE_INGRESS_AUTH_MODE: ${{ matrix.dataplane == 'agentgateway' && 'deprecated-insecure' || 'static-mtls' }}
+      E2E_INGRESS_AUTH_MODE: ${{ matrix.dataplane == 'agentgateway' && 'deprecated-insecure' || 'static-mtls' }}
+      # The suites' router client (internal/e2e/ingressauth.go). Only e2e
+      # installs allowlist it, so the suites can reach actors through the
+      # router's mTLS listeners.
+      ATE_INGRESS_ALLOWED_SPIFFE_IDS: spiffe://cluster.local/ns/ate-e2e-router-client/sa/router-client
       # Appended to the lane and artifact names: e2e-gvisor runs without
       # preview features, e2e-gvisor-preview with them.
       LANE_SUFFIX: ${{ matrix.preview == 'on' && '-preview' || '' }}

AGENTS.md

@@ -1,5 +1,60 @@
 # Agent Substrate
 
+## Rules for contributors
+
+- Read the "Using AI" section of `COLLABORATING.md`. You are responsible for everything your agent submits.
+- Say in the PR description that you used AI, and how. Don't add AI co-author or assisted-by trailers. Submit commits and PRs from your own account. We can't accept contributions from agent or bot accounts (a human must sign the contribution agreement).
+- Before asking for review, review your own diff. For substantial changes, also have a fresh agent session review it adversarially, then validate its findings and fix the confirmed issues.
+- **Be brief.** Code comments, commit messages, PR descriptions and issues should be short, plain English, and read the way an experienced maintainer writes by hand. Reviewer time and agent context windows are limited.
+  - `COLLABORATING.md` doesn't allow AI-generated commit messages, because they have usually been low quality and unreviewed. If you are asked to write one anyway, hold it to the same standard, and the person submitting it must review it.
+  - Write short, direct sentences with one idea each, subject first.
+  - Use the real identifiers from the code. Don't coin new names, and avoid noun stacks like "the store-wide pin-lock pre-flight".
+  - Prefer verbs to abstract nouns: "evicts" instead of "performs eviction".
+  - Say each thing once. Don't restate the diff, open with "This PR …" or "This commit …", or repeat the same explanation in several comments.
+  - Explain why only when the code doesn't make it obvious, in a clause or a line.
+  - Avoid:
+    - em-dashes, and chains of semicolons
+    - headings, bold, or test plans in commit messages
+    - epigrams and contrast pairs, like "X, not Y" or "So treat ENOENT as what it is."
+    - numbers added for color, like "measured at a flat 51ms"
+    - giving code motives or editorializing about it, like "wants", "decides", "is responsible for", "lands", "owns", or "X is what makes Y". Plain active voice is fine: "the reaper collects" is how Go doc comments read.
+    - filler words: load-bearing, footgun, surfaced, robust, comprehensive, seamless, leverage
+    - "No behavior change" unless the diff is a pure refactor
+    - long sentences that pile up clauses, or put the subject after a long lead-in
+
+### Scope
+
+- Keep each PR to one change. Send refactors, cleanups and follow-ups as separate PRs. For how to batch small fixes, see "Sizing PRs for review" in `CONTRIBUTING.md`.
+- Follow "Contribution process" in `CONTRIBUTING.md`: open an issue first for anything beyond a small fix, and search open PRs and issues before you start.
+- Start the PR description with why the change is needed, or link the issue. Only write "Fixes #N" if the PR fully fixes it, and keep the description in sync with the final diff.
+- Rebase onto upstream `main` before opening a PR, and don't merge main into your branch. This file may have changed there. Follow that version.
+- When updating a PR, keep rebases and other changes in separate pushes, so reviewers can see what changed:
+  - To address review comments or test failures, push the fix without rebasing.
+  - To rebase, push only the rebase.
+- After a rebase, check your diff against what main renamed or rewrote since you branched: packages, the module path, helpers you call.
+
+### Code
+
+- Search the repo and its dependencies before writing a helper, cache, client, flag set or test fixture. Reuse or extend what exists, and handle a case the way nearby code already does.
+- Build only what the change needs. Don't add options, interfaces, wrappers, flags, exports or API fields for later. Prefer a sensible default to a new knob. Delete code your change makes unused.
+- Pass dependencies through constructors or functional options. Don't add callback hooks, setters or package globals just so tests can swap behavior.
+- Fail loudly. Ignore only the specific errors you expect, like ENOENT. Don't guard against states the code can't reach, and don't fall back silently to an insecure or no-op mode.
+- Comment the final state of the code. Bugs that only existed while you were writing the change, and links to planning docs outside the repository, are noise to the next reader.
+- Check every claim in comments, docs and the PR description against the code. Don't claim a guarantee the code doesn't enforce. When you change behavior, update the comments, docs and scripts elsewhere that describe it. Link to existing docs instead of copying them.
+- Find the root cause before fixing a symptom. Back performance claims with a measurement.
+- Keep atelet and the worker runtimes opaque to each other. The only knowledge they may share is `internal/proto/ateworkerpb` and the paths in `internal/nodepath`. Using the same library code is fine; depending on how the other side is built is not. We want atelet to roll ahead of workers eventually, so don't widen that interface casually. Until v1.0.0 nothing enforces this; see Backward Compatibility.
+- Actor features work on both gVisor and micro-VM, with e2e coverage for both, unless the issue or PR description scopes the change to one runtime.
+- Don't assume kind, GKE, amd64 or a newer kernel than the repo already needs. Dev scripts must work on arm64 macOS. Don't change host-wide state.
+- Read the `CI` environment variable only inside a named fail-closed predicate, never inline. See Testing Instructions 5.
+- Before changing an API, read `docs/api-style-guide.md` and `docs/api-validation.md`.
+- Pin image references you add or update by digest, and look the digest up yourself: `go run github.com/google/go-containerregistry/cmd/crane@latest digest <image>`.
+
+### Tests
+
+- Test new behavior in ways that help future changes. Update tests when you change existing behavior. Don't add tests just for coverage. Every test costs CI time on every PR.
+- Make sure each new test fails without your change. Follow the Testing section of `docs/code-style-guide.md`; prefer adding rows to an existing table test.
+- Don't use fixed sleeps or wall-clock waits. Run new tests locally with `-count=3 -shuffle=on`.
+
 ## Project Overview
 
 Agent Substrate is a system built on top of Kubernetes which manages agent-like workloads to achieve higher scale and efficiency than Kubernetes alone can offer, with lower latency.
@@ -58,7 +113,6 @@ The dev entry points are the scripts under `hack/` and the `cmd/ate-setup` insta
 - **Copyright Headers**: All files must contain appropriate copyright and license headers. See templates in `hack/boilerplate/`.
 - **Modularity**: Submit small, focused Pull Requests that touch a limited part of the codebase for easier reviews and rebasing.
 - **Go Modules**: Ensure `go.mod` is clean. Run `go mod tidy` if adding or removing dependencies.
-- **Comments**: Keep them brief and to the point. Comment the final state of the code, not the path taken to it — a problem that only existed partway through writing the change is noise to the next reader, as is a pointer to a scratch or planning file that isn't in the repository.
 - **Spelling**: American English. `golangci-lint` runs `misspell` with `locale: US`, so British spellings fail lint.
 
 ## Backward Compatibility
@@ -88,11 +142,11 @@ See the [metric registry](docs/observability.md#the-metric-registry) section of
 
 ## Testing Instructions
 
-1. Write tests for all new code. We will not merge code that lacks tests.
+1. Write tests for new behavior. Update tests when you change existing behavior. Consider the value of tests for future changes.
 2. Ensure changes do not break existing tests.
 3. Run `go test -race ./...` and `hack/verify-all.sh` locally before requesting a code review to catch common issues like missed copyright headers or formatting drift.
 4. For end-to-end tests involving the actual infrastructure, ensure you have a running cluster (setup via `hack/ate-dev-env.sh.example` and `go run ./tools/setup-gcp bootstrap`).
-5. A test that skips when a precondition is missing — Docker, a cluster, an artifact directory — must **fail** on it in CI, because a skip and a pass are the same exit code. Resolve strictness through a named predicate rather than an inline `os.Getenv("CI")`; `cmd/ateapi/internal/store/dockerenv.Required()` is the reference implementation. Prove the strict branch red before merging: a guard only ever observed passing is not known to guard anything. See `docs/dev/best-practices/ci-fail-closed.md`.
+5. A test that skips when a precondition is missing (Docker, a cluster, an artifact directory) must **fail** on it in CI, because a skip and a pass are the same exit code. Check for CI with a named predicate like `cmd/ateapi/internal/store/dockerenv.Required()` instead of an inline `os.Getenv("CI")`. Make sure the CI branch actually fails before merging. See `docs/dev/best-practices/ci-fail-closed.md`.
 
 ## Security Considerations
 

CONTRIBUTING.md

@@ -47,11 +47,12 @@ to review and update than large PRs that touch many different parts of the
 codebase.
 
 2) While we welcome new contributors, we are really focused on the minimal
-capabilities needed to make this project useful. Before you start a new
-contribution, please discuss it with us first (if there is an issue open,
-comment there and if not, open one). We want to make sure that your work is
-aligned with our near-term goals for the project and that we are not
-duplicating work that is already in flight.
+capabilities needed to make this project useful. Small fixes and cleanups can
+go straight to a PR. Before you start anything larger, please discuss it with
+us first (if there is an issue open, comment there and if not, open one), and
+search open PRs so we are not duplicating work that is already in flight. We
+want to make sure that your work is aligned with our near-term goals for the
+project.
 
 3) PRs which are not aligned with our near-term goals may be closed without
 extensive review. We are not trying to be discouraging, but we need to make
@@ -85,8 +86,11 @@ All submissions, including submissions by project members, require review. We
 use [GitHub pull requests](https://docs.github.com/articles/about-pull-requests)
 for this purpose.
 
-All code changes should be accompanied by tests. We will not merge code that
-does not have tests, and we will not merge code that causes tests to fail.
+Changes that add or alter behavior should come with tests, and existing tests
+should be updated when behavior they cover changes. We will not merge code that
+causes tests to fail. Every test runs in CI on every PR, so please don't add
+tests whose only purpose is to raise coverage numbers. Tests should be
+inherently valuable for future changes.
 
 Follow the [PostgreSQL schema evolution rules](docs/dev/postgresql-schema-evolution.md) for each application schema change.
 

benchmarking/README.md

@@ -379,6 +379,20 @@ deployment fails loudly instead of showing up as OOM-flaky steps.
 * `--agentsession-snapshot-steps` — comma-separated 1-based step numbers after
   which the actor is hibernated (for example, `3,10,20,30,40,50,61`); between
   all other steps the actor stays resident.
+* `--agentsession-phase-in` — seconds over which session starts are spread:
+  each user waits a uniform random share of it before creating its actor
+  (default 0, every session starts at once). Locust's ramp spreads the fleet
+  over seconds and every session begins at step 1, so without it the
+  sessions reach each later step together; a step that dwells longer than
+  the ramp then asks every worker for more slots at once than it has. Set
+  it to a few think gaps for density runs.
+* `--agentsession-resume-retry-budget` — seconds a step keeps retrying a
+  wake the fleet had no room for (ResumeActor `ResourceExhausted` or
+  `Unavailable`, router 503/504/429) with 50ms-to-1s backoff before giving
+  the step up until its next think gap (default 60; `0` gives the wake one
+  attempt). Each attempt is its own
+  `ResumeActor` or `WakeFirstTouch` sample; the whole wait lands in
+  `TimeToWake`.
 
 #### Writing an agent-session script
 
@@ -476,6 +490,15 @@ To go back to a built-in variant, redeploy without the flag.
 * `WakeFirstTouch`: latency of a dedicated ping sent before each step's ops —
   in implicit mode that ping is what triggers the parked wake, so this row
   **is** the user-visible wake latency, unpolluted by the step's own work.
+* `TimeToWake`: the whole wake of a parked actor, from the first
+  `ResumeActor` attempt (in implicit mode, the first wake ping) to the ping
+  that found the actor serving, including every retry while the fleet had no
+  room. In implicit mode a first-attempt wake is the `WakeFirstTouch` sample;
+  in explicit mode it is `WakeFirstTouch` plus `ResumeActor`. A failed row is
+  a wake that failed: a step given up after
+  `--agentsession-resume-retry-budget`, or one that failed at once with an
+  error that is not a capacity error, such as a crashed actor or a router
+  404 or 500.
 * `Step_<name>` (e.g. `Step_06_first_build`): wall time of that step's ops,
   think gap excluded.
 * `SuspendActor` / `ResumeActor` / `CreateActor` / `DeleteActor`: control-plane

benchmarking/automation/orchestrator.py

@@ -456,8 +456,9 @@ def deploy_workloads(
     # Empty keeps the default in workloads/deploy.sh (1 CPU). The scheduler
     # books the limit against worker CPU capacity, so dense suites set
     # actorCPU in tests.yaml to fit more actors per worker.
+    # str() because a whole number of cores (actorCPU: 1) parses as an int.
     if actor_cpu:
-        cmd += ["--actor-cpu", actor_cpu]
+        cmd += ["--actor-cpu", str(actor_cpu)]
     # Empty leaves the worker pods unsized; set workerMemory in tests.yaml to
     # control how many workers the scheduler packs onto a node.
     if worker_memory:

benchmarking/automation/test_orchestrator.py

@@ -106,6 +106,13 @@ def test_worker_memory(self, run):
         i = cmd.index("--worker-memory")
         self.assertEqual(cmd[i + 1], "12Gi")
 
+    @mock.patch("orchestrator.run")
+    def test_whole_core_actor_cpu_is_a_string(self, run):
+        orchestrator.deploy_workloads(actor_cpu=1)
+        cmd = run.call_args.args[0]
+        i = cmd.index("--actor-cpu")
+        self.assertEqual(cmd[i + 1], "1")
+
     @mock.patch("orchestrator.run")
     def test_all_options(self, run):
         orchestrator.deploy_workloads(

benchmarking/locust/common/agentsession_config.py

@@ -56,6 +56,33 @@ def add_agentsession_arguments(parser: LocustArgumentParser) -> None:
         ),
         include_in_web_ui=True,
     )
+    parser.add_argument(
+        "--agentsession-phase-in",
+        type=float,
+        default=0.0,
+        env_var="LOCUST_AGENTSESSION_PHASE_IN",
+        help=(
+            "Seconds over which session starts are spread: each user waits a "
+            "uniform random share of this before creating its actor, so a "
+            "fleet spawned in one ramp does not reach every later step in a "
+            "single wave (default: 0, every session starts at once)"
+        ),
+        include_in_web_ui=True,
+    )
+    parser.add_argument(
+        "--agentsession-resume-retry-budget",
+        type=float,
+        default=60.0,
+        env_var="LOCUST_AGENTSESSION_RESUME_RETRY_BUDGET",
+        help=(
+            "Seconds a step keeps retrying a wake the fleet had no room for "
+            "(ResumeActor ResourceExhausted or Unavailable, router 503/504/429) "
+            "before giving the step up until its next think gap. The wait "
+            "lands in the TimeToWake row. 0 gives the wake one attempt "
+            "(default: 60)"
+        ),
+        include_in_web_ui=True,
+    )
     parser.add_argument(
         "--agentsession-snapshot-steps",
         type=str,

benchmarking/locust/common/boomer_config.py

@@ -79,6 +79,8 @@ class that consumes it.
     "--agentsession-script-file": str,
     "--agentsession-think-scale": float,
     "--agentsession-snapshot-steps": str,
+    "--agentsession-phase-in": float,
+    "--agentsession-resume-retry-budget": float,
     "--total-actors": int,
     "--spawn-concurrency": int,
     "--actor-deadline": float,

benchmarking/locust/unit_tests/test_boomer_config.py

@@ -49,6 +49,17 @@ def test_build_config_json_snapshot_steps(self):
             parsed["agentsession_snapshot_steps"], "3,10,20,30,40,50,61"
         )
 
+    def test_build_config_json_phase_in_and_retry_budget(self):
+        raw = boomer_config.build_config_json([
+            "--agentsession-phase-in",
+            "300",
+            "--agentsession-resume-retry-budget",
+            "0",
+        ])
+        parsed = json.loads(raw)
+        self.assertEqual(parsed["agentsession_phase_in"], 300.0)
+        self.assertEqual(parsed["agentsession_resume_retry_budget"], 0.0)
+
 
 if __name__ == "__main__":
     unittest.main()

benchmarking/nighthawk-egress/traffic-driver-template.yaml.tmpl

@@ -61,10 +61,8 @@ volumes:
 # Sandbox size: headroom so the load generator is not the bottleneck.
 resources:
   limits:
-  - name: cpu
-    quantity: "2"
-  - name: memory
-    quantity: 1Gi
+    cpuMillicores: 2000
+    memoryBytes: 1073741824 # 1Gi
 snapshotConfig:
   preferredFidelity: SNAPSHOT_FIDELITY_MEMORY
   storageLocation: gs://${BUCKET_NAME}/nighthawk/traffic-driver/