diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index dc1746e4ef..7509dc0cee 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -1,25 +1,13 @@ -> The PR title becomes the release note. Write it for users: what changed for them, not how the code changed. + -#### Breaking change +Why is this needed? (Or link the issue. Open one first for features, API, install or CI changes.) - +AI use: -What is a breaking change? Something that requires customer code/configuration changes to make their agents work. -But more broadly it might include any behavior change that customers were relying on (Hyrum’s law). Examples of -breaking changes: - -- Moving from 1:1 pods to actors to multiple actors in one pod. -- Changing actor primary keys or identity formats. -- Adding authorization or authentication where none was required before. - ---> - -Fixes # - -> It's a good idea to open an issue first for discussion. + - [ ] Tests pass -- [ ] Appropriate changes to documentation are included in the PR +- [ ] Docs updated diff --git a/.github/workflows/helm-e2e.yaml b/.github/workflows/helm-e2e.yaml index c7de0b1280..aaa3c1431a 100644 --- a/.github/workflows/helm-e2e.yaml +++ b/.github/workflows/helm-e2e.yaml @@ -25,6 +25,7 @@ jobs: env: VERSION: helm-e2e E2E_ATENET_DATAPLANE: agentgateway + E2E_INGRESS_AUTH_MODE: deprecated-insecure E2E_CREDENTIAL_PROVIDER: "1" # Retain external-volume coverage after the preview gate is introduced. ATE_PREVIEW: ALL diff --git a/.github/workflows/pr-workflow.yaml b/.github/workflows/pr-workflow.yaml index c2e9871f00..0323c01a0c 100644 --- a/.github/workflows/pr-workflow.yaml +++ b/.github/workflows/pr-workflow.yaml @@ -173,6 +173,15 @@ jobs: # --preview, so tests of a disabled gate skip themselves (see # internal/e2e/preview.go). ATE_PREVIEW: ${{ matrix.preview == 'on' && 'ALL' || '' }} + # Job-wide for the same reason, and E2E_INGRESS_AUTH_MODE has to match + # it so the suites dial the router's mTLS listeners. The agentgateway + # dataplane does not support static-mtls. + ATE_INGRESS_AUTH_MODE: ${{ matrix.dataplane == 'agentgateway' && 'deprecated-insecure' || 'static-mtls' }} + E2E_INGRESS_AUTH_MODE: ${{ matrix.dataplane == 'agentgateway' && 'deprecated-insecure' || 'static-mtls' }} + # The suites' router client (internal/e2e/ingressauth.go). Only e2e + # installs allowlist it, so the suites can reach actors through the + # router's mTLS listeners. + ATE_INGRESS_ALLOWED_SPIFFE_IDS: spiffe://cluster.local/ns/ate-e2e-router-client/sa/router-client # Appended to the lane and artifact names: e2e-gvisor runs without # preview features, e2e-gvisor-preview with them. LANE_SUFFIX: ${{ matrix.preview == 'on' && '-preview' || '' }} diff --git a/AGENTS.md b/AGENTS.md index 6254f37216..1784a630ed 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,5 +1,60 @@ # Agent Substrate +## Rules for contributors + +- Read the "Using AI" section of `COLLABORATING.md`. You are responsible for everything your agent submits. +- Say in the PR description that you used AI, and how. Don't add AI co-author or assisted-by trailers. Submit commits and PRs from your own account. We can't accept contributions from agent or bot accounts (a human must sign the contribution agreement). +- Before asking for review, review your own diff. For substantial changes, also have a fresh agent session review it adversarially, then validate its findings and fix the confirmed issues. +- **Be brief.** Code comments, commit messages, PR descriptions and issues should be short, plain English, and read the way an experienced maintainer writes by hand. Reviewer time and agent context windows are limited. + - `COLLABORATING.md` doesn't allow AI-generated commit messages, because they have usually been low quality and unreviewed. If you are asked to write one anyway, hold it to the same standard, and the person submitting it must review it. + - Write short, direct sentences with one idea each, subject first. + - Use the real identifiers from the code. Don't coin new names, and avoid noun stacks like "the store-wide pin-lock pre-flight". + - Prefer verbs to abstract nouns: "evicts" instead of "performs eviction". + - Say each thing once. Don't restate the diff, open with "This PR …" or "This commit …", or repeat the same explanation in several comments. + - Explain why only when the code doesn't make it obvious, in a clause or a line. + - Avoid: + - em-dashes, and chains of semicolons + - headings, bold, or test plans in commit messages + - epigrams and contrast pairs, like "X, not Y" or "So treat ENOENT as what it is." + - numbers added for color, like "measured at a flat 51ms" + - giving code motives or editorializing about it, like "wants", "decides", "is responsible for", "lands", "owns", or "X is what makes Y". Plain active voice is fine: "the reaper collects" is how Go doc comments read. + - filler words: load-bearing, footgun, surfaced, robust, comprehensive, seamless, leverage + - "No behavior change" unless the diff is a pure refactor + - long sentences that pile up clauses, or put the subject after a long lead-in + +### Scope + +- Keep each PR to one change. Send refactors, cleanups and follow-ups as separate PRs. For how to batch small fixes, see "Sizing PRs for review" in `CONTRIBUTING.md`. +- Follow "Contribution process" in `CONTRIBUTING.md`: open an issue first for anything beyond a small fix, and search open PRs and issues before you start. +- Start the PR description with why the change is needed, or link the issue. Only write "Fixes #N" if the PR fully fixes it, and keep the description in sync with the final diff. +- Rebase onto upstream `main` before opening a PR, and don't merge main into your branch. This file may have changed there. Follow that version. +- When updating a PR, keep rebases and other changes in separate pushes, so reviewers can see what changed: + - To address review comments or test failures, push the fix without rebasing. + - To rebase, push only the rebase. +- After a rebase, check your diff against what main renamed or rewrote since you branched: packages, the module path, helpers you call. + +### Code + +- Search the repo and its dependencies before writing a helper, cache, client, flag set or test fixture. Reuse or extend what exists, and handle a case the way nearby code already does. +- Build only what the change needs. Don't add options, interfaces, wrappers, flags, exports or API fields for later. Prefer a sensible default to a new knob. Delete code your change makes unused. +- Pass dependencies through constructors or functional options. Don't add callback hooks, setters or package globals just so tests can swap behavior. +- Fail loudly. Ignore only the specific errors you expect, like ENOENT. Don't guard against states the code can't reach, and don't fall back silently to an insecure or no-op mode. +- Comment the final state of the code. Bugs that only existed while you were writing the change, and links to planning docs outside the repository, are noise to the next reader. +- Check every claim in comments, docs and the PR description against the code. Don't claim a guarantee the code doesn't enforce. When you change behavior, update the comments, docs and scripts elsewhere that describe it. Link to existing docs instead of copying them. +- Find the root cause before fixing a symptom. Back performance claims with a measurement. +- Keep atelet and the worker runtimes opaque to each other. The only knowledge they may share is `internal/proto/ateworkerpb` and the paths in `internal/nodepath`. Using the same library code is fine; depending on how the other side is built is not. We want atelet to roll ahead of workers eventually, so don't widen that interface casually. Until v1.0.0 nothing enforces this; see Backward Compatibility. +- Actor features work on both gVisor and micro-VM, with e2e coverage for both, unless the issue or PR description scopes the change to one runtime. +- Don't assume kind, GKE, amd64 or a newer kernel than the repo already needs. Dev scripts must work on arm64 macOS. Don't change host-wide state. +- Read the `CI` environment variable only inside a named fail-closed predicate, never inline. See Testing Instructions 5. +- Before changing an API, read `docs/api-style-guide.md` and `docs/api-validation.md`. +- Pin image references you add or update by digest, and look the digest up yourself: `go run github.com/google/go-containerregistry/cmd/crane@latest digest `. + +### Tests + +- Test new behavior in ways that help future changes. Update tests when you change existing behavior. Don't add tests just for coverage. Every test costs CI time on every PR. +- Make sure each new test fails without your change. Follow the Testing section of `docs/code-style-guide.md`; prefer adding rows to an existing table test. +- Don't use fixed sleeps or wall-clock waits. Run new tests locally with `-count=3 -shuffle=on`. + ## Project Overview Agent Substrate is a system built on top of Kubernetes which manages agent-like workloads to achieve higher scale and efficiency than Kubernetes alone can offer, with lower latency. @@ -58,7 +113,6 @@ The dev entry points are the scripts under `hack/` and the `cmd/ate-setup` insta - **Copyright Headers**: All files must contain appropriate copyright and license headers. See templates in `hack/boilerplate/`. - **Modularity**: Submit small, focused Pull Requests that touch a limited part of the codebase for easier reviews and rebasing. - **Go Modules**: Ensure `go.mod` is clean. Run `go mod tidy` if adding or removing dependencies. -- **Comments**: Keep them brief and to the point. Comment the final state of the code, not the path taken to it — a problem that only existed partway through writing the change is noise to the next reader, as is a pointer to a scratch or planning file that isn't in the repository. - **Spelling**: American English. `golangci-lint` runs `misspell` with `locale: US`, so British spellings fail lint. ## Backward Compatibility @@ -88,11 +142,11 @@ See the [metric registry](docs/observability.md#the-metric-registry) section of ## Testing Instructions -1. Write tests for all new code. We will not merge code that lacks tests. +1. Write tests for new behavior. Update tests when you change existing behavior. Consider the value of tests for future changes. 2. Ensure changes do not break existing tests. 3. Run `go test -race ./...` and `hack/verify-all.sh` locally before requesting a code review to catch common issues like missed copyright headers or formatting drift. 4. For end-to-end tests involving the actual infrastructure, ensure you have a running cluster (setup via `hack/ate-dev-env.sh.example` and `go run ./tools/setup-gcp bootstrap`). -5. A test that skips when a precondition is missing — Docker, a cluster, an artifact directory — must **fail** on it in CI, because a skip and a pass are the same exit code. Resolve strictness through a named predicate rather than an inline `os.Getenv("CI")`; `cmd/ateapi/internal/store/dockerenv.Required()` is the reference implementation. Prove the strict branch red before merging: a guard only ever observed passing is not known to guard anything. See `docs/dev/best-practices/ci-fail-closed.md`. +5. A test that skips when a precondition is missing (Docker, a cluster, an artifact directory) must **fail** on it in CI, because a skip and a pass are the same exit code. Check for CI with a named predicate like `cmd/ateapi/internal/store/dockerenv.Required()` instead of an inline `os.Getenv("CI")`. Make sure the CI branch actually fails before merging. See `docs/dev/best-practices/ci-fail-closed.md`. ## Security Considerations diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 7cb826aa83..b8af87a341 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -47,11 +47,12 @@ to review and update than large PRs that touch many different parts of the codebase. 2) While we welcome new contributors, we are really focused on the minimal -capabilities needed to make this project useful. Before you start a new -contribution, please discuss it with us first (if there is an issue open, -comment there and if not, open one). We want to make sure that your work is -aligned with our near-term goals for the project and that we are not -duplicating work that is already in flight. +capabilities needed to make this project useful. Small fixes and cleanups can +go straight to a PR. Before you start anything larger, please discuss it with +us first (if there is an issue open, comment there and if not, open one), and +search open PRs so we are not duplicating work that is already in flight. We +want to make sure that your work is aligned with our near-term goals for the +project. 3) PRs which are not aligned with our near-term goals may be closed without extensive review. We are not trying to be discouraging, but we need to make @@ -85,8 +86,11 @@ All submissions, including submissions by project members, require review. We use [GitHub pull requests](https://docs.github.com/articles/about-pull-requests) for this purpose. -All code changes should be accompanied by tests. We will not merge code that -does not have tests, and we will not merge code that causes tests to fail. +Changes that add or alter behavior should come with tests, and existing tests +should be updated when behavior they cover changes. We will not merge code that +causes tests to fail. Every test runs in CI on every PR, so please don't add +tests whose only purpose is to raise coverage numbers. Tests should be +inherently valuable for future changes. Follow the [PostgreSQL schema evolution rules](docs/dev/postgresql-schema-evolution.md) for each application schema change. diff --git a/benchmarking/README.md b/benchmarking/README.md index 51475da90e..58a6679a76 100644 --- a/benchmarking/README.md +++ b/benchmarking/README.md @@ -379,6 +379,20 @@ deployment fails loudly instead of showing up as OOM-flaky steps. * `--agentsession-snapshot-steps` — comma-separated 1-based step numbers after which the actor is hibernated (for example, `3,10,20,30,40,50,61`); between all other steps the actor stays resident. +* `--agentsession-phase-in` — seconds over which session starts are spread: + each user waits a uniform random share of it before creating its actor + (default 0, every session starts at once). Locust's ramp spreads the fleet + over seconds and every session begins at step 1, so without it the + sessions reach each later step together; a step that dwells longer than + the ramp then asks every worker for more slots at once than it has. Set + it to a few think gaps for density runs. +* `--agentsession-resume-retry-budget` — seconds a step keeps retrying a + wake the fleet had no room for (ResumeActor `ResourceExhausted` or + `Unavailable`, router 503/504/429) with 50ms-to-1s backoff before giving + the step up until its next think gap (default 60; `0` gives the wake one + attempt). Each attempt is its own + `ResumeActor` or `WakeFirstTouch` sample; the whole wait lands in + `TimeToWake`. #### Writing an agent-session script @@ -476,6 +490,15 @@ To go back to a built-in variant, redeploy without the flag. * `WakeFirstTouch`: latency of a dedicated ping sent before each step's ops — in implicit mode that ping is what triggers the parked wake, so this row **is** the user-visible wake latency, unpolluted by the step's own work. +* `TimeToWake`: the whole wake of a parked actor, from the first + `ResumeActor` attempt (in implicit mode, the first wake ping) to the ping + that found the actor serving, including every retry while the fleet had no + room. In implicit mode a first-attempt wake is the `WakeFirstTouch` sample; + in explicit mode it is `WakeFirstTouch` plus `ResumeActor`. A failed row is + a wake that failed: a step given up after + `--agentsession-resume-retry-budget`, or one that failed at once with an + error that is not a capacity error, such as a crashed actor or a router + 404 or 500. * `Step_` (e.g. `Step_06_first_build`): wall time of that step's ops, think gap excluded. * `SuspendActor` / `ResumeActor` / `CreateActor` / `DeleteActor`: control-plane diff --git a/benchmarking/automation/orchestrator.py b/benchmarking/automation/orchestrator.py index 0940c4d79c..97af6142dc 100644 --- a/benchmarking/automation/orchestrator.py +++ b/benchmarking/automation/orchestrator.py @@ -456,8 +456,9 @@ def deploy_workloads( # Empty keeps the default in workloads/deploy.sh (1 CPU). The scheduler # books the limit against worker CPU capacity, so dense suites set # actorCPU in tests.yaml to fit more actors per worker. + # str() because a whole number of cores (actorCPU: 1) parses as an int. if actor_cpu: - cmd += ["--actor-cpu", actor_cpu] + cmd += ["--actor-cpu", str(actor_cpu)] # Empty leaves the worker pods unsized; set workerMemory in tests.yaml to # control how many workers the scheduler packs onto a node. if worker_memory: diff --git a/benchmarking/automation/test_orchestrator.py b/benchmarking/automation/test_orchestrator.py index 004785f011..aef06afed9 100644 --- a/benchmarking/automation/test_orchestrator.py +++ b/benchmarking/automation/test_orchestrator.py @@ -106,6 +106,13 @@ def test_worker_memory(self, run): i = cmd.index("--worker-memory") self.assertEqual(cmd[i + 1], "12Gi") + @mock.patch("orchestrator.run") + def test_whole_core_actor_cpu_is_a_string(self, run): + orchestrator.deploy_workloads(actor_cpu=1) + cmd = run.call_args.args[0] + i = cmd.index("--actor-cpu") + self.assertEqual(cmd[i + 1], "1") + @mock.patch("orchestrator.run") def test_all_options(self, run): orchestrator.deploy_workloads( diff --git a/benchmarking/locust/common/agentsession_config.py b/benchmarking/locust/common/agentsession_config.py index a6ea37fe3e..9e43dfcbdb 100644 --- a/benchmarking/locust/common/agentsession_config.py +++ b/benchmarking/locust/common/agentsession_config.py @@ -56,6 +56,33 @@ def add_agentsession_arguments(parser: LocustArgumentParser) -> None: ), include_in_web_ui=True, ) + parser.add_argument( + "--agentsession-phase-in", + type=float, + default=0.0, + env_var="LOCUST_AGENTSESSION_PHASE_IN", + help=( + "Seconds over which session starts are spread: each user waits a " + "uniform random share of this before creating its actor, so a " + "fleet spawned in one ramp does not reach every later step in a " + "single wave (default: 0, every session starts at once)" + ), + include_in_web_ui=True, + ) + parser.add_argument( + "--agentsession-resume-retry-budget", + type=float, + default=60.0, + env_var="LOCUST_AGENTSESSION_RESUME_RETRY_BUDGET", + help=( + "Seconds a step keeps retrying a wake the fleet had no room for " + "(ResumeActor ResourceExhausted or Unavailable, router 503/504/429) " + "before giving the step up until its next think gap. The wait " + "lands in the TimeToWake row. 0 gives the wake one attempt " + "(default: 60)" + ), + include_in_web_ui=True, + ) parser.add_argument( "--agentsession-snapshot-steps", type=str, diff --git a/benchmarking/locust/common/boomer_config.py b/benchmarking/locust/common/boomer_config.py index 6f9cfeac92..41151c5a6a 100644 --- a/benchmarking/locust/common/boomer_config.py +++ b/benchmarking/locust/common/boomer_config.py @@ -79,6 +79,8 @@ class that consumes it. "--agentsession-script-file": str, "--agentsession-think-scale": float, "--agentsession-snapshot-steps": str, + "--agentsession-phase-in": float, + "--agentsession-resume-retry-budget": float, "--total-actors": int, "--spawn-concurrency": int, "--actor-deadline": float, diff --git a/benchmarking/locust/unit_tests/test_boomer_config.py b/benchmarking/locust/unit_tests/test_boomer_config.py index 93455361df..58e08fd594 100644 --- a/benchmarking/locust/unit_tests/test_boomer_config.py +++ b/benchmarking/locust/unit_tests/test_boomer_config.py @@ -49,6 +49,17 @@ def test_build_config_json_snapshot_steps(self): parsed["agentsession_snapshot_steps"], "3,10,20,30,40,50,61" ) + def test_build_config_json_phase_in_and_retry_budget(self): + raw = boomer_config.build_config_json([ + "--agentsession-phase-in", + "300", + "--agentsession-resume-retry-budget", + "0", + ]) + parsed = json.loads(raw) + self.assertEqual(parsed["agentsession_phase_in"], 300.0) + self.assertEqual(parsed["agentsession_resume_retry_budget"], 0.0) + if __name__ == "__main__": unittest.main() diff --git a/benchmarking/nighthawk-egress/traffic-driver-template.yaml.tmpl b/benchmarking/nighthawk-egress/traffic-driver-template.yaml.tmpl index 2395016880..acaad876e7 100644 --- a/benchmarking/nighthawk-egress/traffic-driver-template.yaml.tmpl +++ b/benchmarking/nighthawk-egress/traffic-driver-template.yaml.tmpl @@ -61,10 +61,8 @@ volumes: # Sandbox size: headroom so the load generator is not the bottleneck. resources: limits: - - name: cpu - quantity: "2" - - name: memory - quantity: 1Gi + cpuMillicores: 2000 + memoryBytes: 1073741824 # 1Gi snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://${BUCKET_NAME}/nighthawk/traffic-driver/ diff --git a/benchmarking/workloads/deploy.sh b/benchmarking/workloads/deploy.sh index f7f73aae2f..c18328dfca 100755 --- a/benchmarking/workloads/deploy.sh +++ b/benchmarking/workloads/deploy.sh @@ -153,6 +153,40 @@ run_kubectl_ate() { "${KUBECTL_ATE_BIN}" "$@" } +# cpu_millicores converts a Kubernetes cpu quantity like 250m or 1.5 to +# millicores, which is what ActorTemplates take +cpu_millicores() { + local q="$1" + if [[ "${q}" =~ ^([0-9]+)m$ ]]; then + echo "${BASH_REMATCH[1]}" + elif [[ "${q}" =~ ^([0-9]+)(\.([0-9]{1,3}))?$ ]]; then + local frac="${BASH_REMATCH[3]}000" + echo $((10#${BASH_REMATCH[1]} * 1000 + 10#${frac:0:3})) + else + echo "Error: --actor-cpu ${q} is not a cpu quantity like 250m or 1.5" >&2 + return 1 + fi +} + +# memory_bytes converts a memory quantity like 256Mi or 1Gi to bytes, which is +# what ActorTemplates take +memory_bytes() { + local q="$1" + if [[ "${q}" =~ ^([0-9]+)(Ki|Mi|Gi|Ti)?$ ]]; then + local shift=0 + case "${BASH_REMATCH[2]}" in + Ki) shift=10 ;; + Mi) shift=20 ;; + Gi) shift=30 ;; + Ti) shift=40 ;; + esac + echo $((10#${BASH_REMATCH[1]} << shift)) + else + echo "Error: --actor-memory ${q} is not a memory quantity like 256Mi or 1Gi" >&2 + return 1 + fi +} + substitute() { local manifest="$1" local worker_resources="" @@ -165,7 +199,8 @@ substitute() { -e "s|\${SANDBOX_CLASS}|${SANDBOX_CLASS}|g" \ -e "s|\${OTLP_ENDPOINT}|${OTLP_ENDPOINT}|g" \ -e "s|\${ACTOR_MEMORY}|${ACTOR_MEMORY}|g" \ - -e "s|\${ACTOR_CPU}|${ACTOR_CPU}|g" \ + -e "s|\${ACTOR_MEMORY_BYTES}|${ACTOR_MEMORY_BYTES}|g" \ + -e "s|\${ACTOR_CPU_MILLICORES}|${ACTOR_CPU_MILLICORES}|g" \ -e "s|\${WORKER_RESOURCES}|${worker_resources}|g" \ -e "s|\${ATE_ENV_IMAGE}|${ATE_ENV_IMAGE:-}|g" \ -e "s|\${STORAGE_CLASS_NAME}|${STORAGE_CLASS_NAME}|g" \ @@ -275,6 +310,8 @@ deploy() { echo "Error: --fake-atelet-env needs substrate installed with --fake-data-plane" >&2 exit 1 fi + ACTOR_CPU_MILLICORES="$(cpu_millicores "${ACTOR_CPU}")" + ACTOR_MEMORY_BYTES="$(memory_bytes "${ACTOR_MEMORY}")" echo "Deploying workloads (worker_count=${WORKER_COUNT}, actor_memory=${ACTOR_MEMORY}, actor_cpu=${ACTOR_CPU}, worker_memory=${WORKER_MEMORY:-unset}, storage_class=${STORAGE_CLASS_NAME:-none}, otlp_endpoint=${OTLP_ENDPOINT})..." substitute "${POOL_MANIFEST}" | hack/run-tool.sh ko apply -f - echo "Waiting for worker pool to be ready (timeout: ${WAIT_TIMEOUT_SECS}s)..." diff --git a/benchmarking/workloads/manifests/ate-env-template.yaml.tmpl b/benchmarking/workloads/manifests/ate-env-template.yaml.tmpl index e25388ef55..7b5605c58f 100644 --- a/benchmarking/workloads/manifests/ate-env-template.yaml.tmpl +++ b/benchmarking/workloads/manifests/ate-env-template.yaml.tmpl @@ -42,10 +42,8 @@ containers: port: 80 resources: limits: - - name: cpu - quantity: "${ACTOR_CPU}" - - name: memory - quantity: "${ACTOR_MEMORY}" + cpuMillicores: "${ACTOR_CPU_MILLICORES}" + memoryBytes: "${ACTOR_MEMORY_BYTES}" snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://${BUCKET_NAME}/benchmark-workloads/ate-env/ diff --git a/benchmarking/workloads/manifests/glutton-durdir-data-template.yaml.tmpl b/benchmarking/workloads/manifests/glutton-durdir-data-template.yaml.tmpl index 75bb218164..ee169fdeed 100644 --- a/benchmarking/workloads/manifests/glutton-durdir-data-template.yaml.tmpl +++ b/benchmarking/workloads/manifests/glutton-durdir-data-template.yaml.tmpl @@ -46,10 +46,8 @@ volumes: # Sandbox size: see the note on the sleep template. resources: limits: - - name: cpu - quantity: "${ACTOR_CPU}" - - name: memory - quantity: "${ACTOR_MEMORY}" + cpuMillicores: "${ACTOR_CPU_MILLICORES}" + memoryBytes: "${ACTOR_MEMORY_BYTES}" snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_VOLUMES storageLocation: gs://${BUCKET_NAME}/benchmark-workloads/glutton-durdir-data/ diff --git a/benchmarking/workloads/manifests/glutton-durdir-full-template.yaml.tmpl b/benchmarking/workloads/manifests/glutton-durdir-full-template.yaml.tmpl index 4f40c33618..30aacdc7fb 100644 --- a/benchmarking/workloads/manifests/glutton-durdir-full-template.yaml.tmpl +++ b/benchmarking/workloads/manifests/glutton-durdir-full-template.yaml.tmpl @@ -43,10 +43,8 @@ volumes: # Sandbox size: see the note on the sleep template. resources: limits: - - name: cpu - quantity: "${ACTOR_CPU}" - - name: memory - quantity: "${ACTOR_MEMORY}" + cpuMillicores: "${ACTOR_CPU_MILLICORES}" + memoryBytes: "${ACTOR_MEMORY_BYTES}" snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://${BUCKET_NAME}/benchmark-workloads/glutton-durdir-full/ diff --git a/benchmarking/workloads/manifests/glutton-storage-template.yaml.tmpl b/benchmarking/workloads/manifests/glutton-storage-template.yaml.tmpl index 7145925134..0ccd569269 100644 --- a/benchmarking/workloads/manifests/glutton-storage-template.yaml.tmpl +++ b/benchmarking/workloads/manifests/glutton-storage-template.yaml.tmpl @@ -26,7 +26,7 @@ volumes: - name: external-storage external: template: - capacity: 1Gi + capacityBytes: 1073741824 # 1Gi storageClassName: ${STORAGE_CLASS_NAME} containers: - name: glutton-storage @@ -53,10 +53,8 @@ containers: port: 80 resources: limits: - - name: cpu - quantity: "${ACTOR_CPU}" - - name: memory - quantity: "${ACTOR_MEMORY}" + cpuMillicores: "${ACTOR_CPU_MILLICORES}" + memoryBytes: "${ACTOR_MEMORY_BYTES}" snapshotConfig: onCommit: SNAPSHOT_CONTENT_SCOPE_FULL storageLocation: gs://${BUCKET_NAME}/benchmark-workloads/glutton-storage/ diff --git a/benchmarking/workloads/manifests/glutton-template.yaml.tmpl b/benchmarking/workloads/manifests/glutton-template.yaml.tmpl index 4749253b32..46fd41e7a5 100644 --- a/benchmarking/workloads/manifests/glutton-template.yaml.tmpl +++ b/benchmarking/workloads/manifests/glutton-template.yaml.tmpl @@ -55,10 +55,8 @@ containers: # exercise glutton's WriteRAM path. resources: limits: - - name: cpu - quantity: "${ACTOR_CPU}" - - name: memory - quantity: "${ACTOR_MEMORY}" + cpuMillicores: "${ACTOR_CPU_MILLICORES}" + memoryBytes: "${ACTOR_MEMORY_BYTES}" snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://${BUCKET_NAME}/benchmark-workloads/glutton/ diff --git a/benchmarking/workloads/manifests/kernelmem-template.yaml.tmpl b/benchmarking/workloads/manifests/kernelmem-template.yaml.tmpl index 2849c9a494..e970a15195 100644 --- a/benchmarking/workloads/manifests/kernelmem-template.yaml.tmpl +++ b/benchmarking/workloads/manifests/kernelmem-template.yaml.tmpl @@ -33,10 +33,8 @@ containers: # each, mostly page tables and kernel state). resources: limits: - - name: cpu - quantity: "1" - - name: memory - quantity: 8Gi + cpuMillicores: 1000 + memoryBytes: 8589934592 # 8Gi snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://${BUCKET_NAME}/benchmark-workloads/kernelmem/ diff --git a/benchmarking/workloads/manifests/sleep-template.yaml.tmpl b/benchmarking/workloads/manifests/sleep-template.yaml.tmpl index a4692ff5d3..d7e04f67aa 100644 --- a/benchmarking/workloads/manifests/sleep-template.yaml.tmpl +++ b/benchmarking/workloads/manifests/sleep-template.yaml.tmpl @@ -30,10 +30,8 @@ containers: # 128Mi guest floor). resources: limits: - - name: cpu - quantity: "${ACTOR_CPU}" - - name: memory - quantity: "${ACTOR_MEMORY}" + cpuMillicores: "${ACTOR_CPU_MILLICORES}" + memoryBytes: "${ACTOR_MEMORY_BYTES}" snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://${BUCKET_NAME}/benchmark-workloads/sleep/ diff --git a/benchmarking/workloads/manifests/sweperf/generated/sweperf-astropy-7336-template.yaml.tmpl b/benchmarking/workloads/manifests/sweperf/generated/sweperf-astropy-7336-template.yaml.tmpl index 55ee16942d..33bb0d7738 100644 --- a/benchmarking/workloads/manifests/sweperf/generated/sweperf-astropy-7336-template.yaml.tmpl +++ b/benchmarking/workloads/manifests/sweperf/generated/sweperf-astropy-7336-template.yaml.tmpl @@ -43,10 +43,8 @@ containers: # the measured execution time, so keep it fixed across runs. resources: limits: - - name: cpu - quantity: "2" - - name: memory - quantity: 4Gi + cpuMillicores: 2000 + memoryBytes: 4294967296 # 4Gi snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://${BUCKET_NAME}/benchmark-workloads/sweperf-astropy-7336/ diff --git a/benchmarking/workloads/manifests/sweperf/generated/sweperf-django-11099-template.yaml.tmpl b/benchmarking/workloads/manifests/sweperf/generated/sweperf-django-11099-template.yaml.tmpl index da8965ea68..af94623150 100644 --- a/benchmarking/workloads/manifests/sweperf/generated/sweperf-django-11099-template.yaml.tmpl +++ b/benchmarking/workloads/manifests/sweperf/generated/sweperf-django-11099-template.yaml.tmpl @@ -43,10 +43,8 @@ containers: # the measured execution time, so keep it fixed across runs. resources: limits: - - name: cpu - quantity: "2" - - name: memory - quantity: 4Gi + cpuMillicores: 2000 + memoryBytes: 4294967296 # 4Gi snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://${BUCKET_NAME}/benchmark-workloads/sweperf-django-11099/ diff --git a/benchmarking/workloads/manifests/sweperf/generated/sweperf-matplotlib-20826-template.yaml.tmpl b/benchmarking/workloads/manifests/sweperf/generated/sweperf-matplotlib-20826-template.yaml.tmpl index 16d29b9ce7..040e0486ff 100644 --- a/benchmarking/workloads/manifests/sweperf/generated/sweperf-matplotlib-20826-template.yaml.tmpl +++ b/benchmarking/workloads/manifests/sweperf/generated/sweperf-matplotlib-20826-template.yaml.tmpl @@ -43,10 +43,8 @@ containers: # the measured execution time, so keep it fixed across runs. resources: limits: - - name: cpu - quantity: "2" - - name: memory - quantity: 4Gi + cpuMillicores: 2000 + memoryBytes: 4294967296 # 4Gi snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://${BUCKET_NAME}/benchmark-workloads/sweperf-matplotlib-20826/ diff --git a/benchmarking/workloads/manifests/sweperf/generated/sweperf-sphinx-8638-template.yaml.tmpl b/benchmarking/workloads/manifests/sweperf/generated/sweperf-sphinx-8638-template.yaml.tmpl index 555db42def..1a8264c447 100644 --- a/benchmarking/workloads/manifests/sweperf/generated/sweperf-sphinx-8638-template.yaml.tmpl +++ b/benchmarking/workloads/manifests/sweperf/generated/sweperf-sphinx-8638-template.yaml.tmpl @@ -43,10 +43,8 @@ containers: # the measured execution time, so keep it fixed across runs. resources: limits: - - name: cpu - quantity: "2" - - name: memory - quantity: 4Gi + cpuMillicores: 2000 + memoryBytes: 4294967296 # 4Gi snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://${BUCKET_NAME}/benchmark-workloads/sweperf-sphinx-8638/ diff --git a/benchmarking/workloads/manifests/sweperf/generated/sweperf-sympy-13647-template.yaml.tmpl b/benchmarking/workloads/manifests/sweperf/generated/sweperf-sympy-13647-template.yaml.tmpl index 77459b205c..e869b8c2ee 100644 --- a/benchmarking/workloads/manifests/sweperf/generated/sweperf-sympy-13647-template.yaml.tmpl +++ b/benchmarking/workloads/manifests/sweperf/generated/sweperf-sympy-13647-template.yaml.tmpl @@ -43,10 +43,8 @@ containers: # the measured execution time, so keep it fixed across runs. resources: limits: - - name: cpu - quantity: "2" - - name: memory - quantity: 4Gi + cpuMillicores: 2000 + memoryBytes: 4294967296 # 4Gi snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://${BUCKET_NAME}/benchmark-workloads/sweperf-sympy-13647/ diff --git a/benchmarking/workloads/manifests/sweperf/sweperf-template.yaml.j2 b/benchmarking/workloads/manifests/sweperf/sweperf-template.yaml.j2 index 9fb8514a40..9fc60b1653 100644 --- a/benchmarking/workloads/manifests/sweperf/sweperf-template.yaml.j2 +++ b/benchmarking/workloads/manifests/sweperf/sweperf-template.yaml.j2 @@ -43,10 +43,8 @@ containers: # the measured execution time, so keep it fixed across runs. resources: limits: - - name: cpu - quantity: "2" - - name: memory - quantity: 4Gi + cpuMillicores: 2000 + memoryBytes: 4294967296 # 4Gi snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://${BUCKET_NAME}/benchmark-workloads/{{ name }}/ diff --git a/benchmarking/workloads/manifests/usermem-template.yaml.tmpl b/benchmarking/workloads/manifests/usermem-template.yaml.tmpl index e717ec174a..29a34a25fd 100644 --- a/benchmarking/workloads/manifests/usermem-template.yaml.tmpl +++ b/benchmarking/workloads/manifests/usermem-template.yaml.tmpl @@ -31,10 +31,8 @@ containers: # Sandbox size: the 1 GiB stress allocation plus headroom. resources: limits: - - name: cpu - quantity: "1" - - name: memory - quantity: 1536Mi + cpuMillicores: 1000 + memoryBytes: 1610612736 # 1536Mi snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://${BUCKET_NAME}/benchmark-workloads/usermem/ diff --git a/charts/substrate/templates/role.yaml b/charts/substrate/templates/role.yaml index 5ff03a5ff6..669fcc25b4 100644 --- a/charts/substrate/templates/role.yaml +++ b/charts/substrate/templates/role.yaml @@ -54,6 +54,12 @@ rules: - patch - update - watch +- apiGroups: + - apps + resources: + - replicasets + verbs: + - get - apiGroups: - ate.dev resources: diff --git a/cmd/ate-setup/commands.md b/cmd/ate-setup/commands.md index ae716ff032..a32674c2a2 100644 --- a/cmd/ate-setup/commands.md +++ b/cmd/ate-setup/commands.md @@ -31,6 +31,7 @@ a pre-scan pass, so they may appear anywhere on its command line. | `--ateapi-postgres-storage-class NAME` | `ATE_API_POSTGRES_STORAGE_CLASS=NAME` | StorageClass for the bundled PostgreSQL volume instead of the cluster default, such as `dynamic-rwo` on GKE so C4 and N4 nodes can attach it. An existing install must delete the postgres StatefulSet and its claim first | | `--cordon-control-plane` | `--cordon-control-plane` | Keep the control plane off the worker nodes. Assumes a small shared pool labeled and tainted `ate.dev/workloadType=ate-control-plane:NoSchedule`, across which each workload's replicas are spread, and a one-node pool labeled and tainted `ate.dev/workloadType=ate-postgres:NoSchedule` for postgres alone. `ATE_INSTALL_CORDON_CONTROL_PLANE=true` when the flag is absent | | `--fake-data-plane` | `--fake-data-plane` | Install the fake data plane for control-plane benchmarks: fake-atelet takes atelet's place in its DaemonSet, taking its objectstore-plugin's storage settings, ate-controller is scaled to zero, and fake-workersync honors every WorkerPool with fake Workers on the fake-atelet nodes. Reaches `deploy ate-system`, `deploy atelet`, and `deploy ate-controller`; only `deploy ate-system` switches a cluster between the real and fake data planes, and the other two refuse a data plane other than the installed one. `deploy ate-system --fake-data-plane` refuses a cluster whose WorkerPools have worker pods. fake-atelet runs with its defaults; a benchmark tunes it through the optional `fake-atelet-tuning` ConfigMap, whose `FAKE_ATELET_*` variables set its flags, and which no install creates or resets. Actors do not exist; never use it on a cluster that serves real actors. `ATE_INSTALL_FAKE_DATA_PLANE=true` when the flag is absent | +| `--ingress-allowed-spiffe-ids IDS` | `--ingress-allowed-spiffe-ids IDS`, or `ATE_INGRESS_ALLOWED_SPIFFE_IDS=IDS` | Required by `deploy ate-system` and `deploy atenet` with `--atenet-dataplane=envoy`. Comma-separated SPIFFE IDs (exact match) of the clients the ingress router lets reach actors over its TLS ports; any listed client may reach any actor. Appended to the router as its `--ingress-allowed-spiffe-ids`. The podidentity signer issues `spiffe://cluster.local/ns//sa/` | | `--experimental-additional-egress-extproc-service NS/SVC:PORT` | `--experimental-additional-egress-extproc-service NS/SVC:PORT` | External processor authorization filter | | `--credential-provider JSON` | `--credential-provider JSON`, or `ATE_CREDENTIAL_PROVIDER` | Required by `deploy ate-system` and `deploy atenet`. A JSON object: `{"name":"k8s.io"}` deploys and uses the bundled Kubernetes Secrets provider, with a NetworkPolicy that admits only the egress gateway; `{"enabled":false}` turns egress credential injection off; `{"name":"","address":":"}` uses a provider you deploy yourself. See [`docs/egress-credential-injection.md`](../../docs/egress-credential-injection.md) | | `--ateapi-log-level LEVEL` | `--ateapi-log-level LEVEL`, or `ATE_API_LOG_LEVEL=LEVEL` | ate-api-server log level: `debug`, `info`, `warn`, or `error`. Written to the `ate-api-config` ConfigMap on every deploy when set; unset keeps the installed level (`info` on a new install). The server picks up a change without restarting | diff --git a/cmd/ate-setup/internal/config/config.go b/cmd/ate-setup/internal/config/config.go index a5040e7228..e497c6ebbf 100644 --- a/cmd/ate-setup/internal/config/config.go +++ b/cmd/ate-setup/internal/config/config.go @@ -223,6 +223,11 @@ type Config struct { // IngressAuthStaticMTLS. IngressAuthMode string + // IngressAllowedSPIFFEIDs are the SPIFFE IDs the ingress router lets + // reach actors over its TLS listeners (ATE_INGRESS_ALLOWED_SPIFFE_IDS). + // See RequireIngressAllowedSPIFFEIDs. + IngressAllowedSPIFFEIDs []string + // AdditionalEgressExtprocService is the optional NS/SVC:PORT external processor filter. AdditionalEgressExtprocService string @@ -279,6 +284,18 @@ func (c *Config) Resolved() *Resolved { return c.resolved } +// RequireIngressAllowedSPIFFEIDs fails when the install would deploy the +// Envoy ingress router without an allowlist. That router authenticates +// clients on its TLS listeners, and refuses to start with no SPIFFE ID to +// match them against. The agentgateway router is configured statically and +// takes none. +func (c *Config) RequireIngressAllowedSPIFFEIDs() error { + if c.Router == RouterAgentgateway || len(c.IngressAllowedSPIFFEIDs) > 0 { + return nil + } + return c.Resolved().Require("ingressAllowedSPIFFEIDs") +} + // SetResolved attaches resolved settings to a Config built directly. Load // does this itself; the callers here are tests and the demo test harness. func (c *Config) SetResolved(r *Resolved) { c.resolved = r } @@ -484,6 +501,10 @@ func (c *Config) ScriptEnv() []string { if c.IngressAuthMode != "" && c.IngressAuthMode != IngressAuthDeprecatedInsecure { merged["ATE_INGRESS_AUTH_MODE"] = c.IngressAuthMode } + delete(merged, "ATE_INGRESS_ALLOWED_SPIFFE_IDS") + if len(c.IngressAllowedSPIFFEIDs) > 0 { + merged["ATE_INGRESS_ALLOWED_SPIFFE_IDS"] = strings.Join(c.IngressAllowedSPIFFEIDs, ",") + } if c.AdditionalEgressExtprocService != "" { merged["ATE_ADDITIONAL_EGRESS_EXTPROC_SERVICE"] = c.AdditionalEgressExtprocService } diff --git a/cmd/ate-setup/internal/config/config_test.go b/cmd/ate-setup/internal/config/config_test.go index 19996c0b2e..1e887b4f45 100644 --- a/cmd/ate-setup/internal/config/config_test.go +++ b/cmd/ate-setup/internal/config/config_test.go @@ -15,6 +15,7 @@ package config import ( + "errors" "os" "path/filepath" "runtime" @@ -63,6 +64,7 @@ func loadEnv(t *testing.T) { "ATE_CREDENTIAL_PROVIDER", "ATE_IMAGE_REPO", "ATE_IMAGE_TAG", + "ATE_INGRESS_ALLOWED_SPIFFE_IDS", "ATE_INGRESS_AUTH_MODE", "ATE_INSTALL_CLUSTER_SIZE", "ATE_INSTALL_CORDON_CONTROL_PLANE", @@ -283,6 +285,74 @@ func TestLoadIngressAuthMode(t *testing.T) { } } +func TestLoadIngressAllowedSPIFFEIDs(t *testing.T) { + const a, b = "spiffe://cluster.local/ns/a/sa/a", "spiffe://cluster.local/ns/b/sa/b" + for _, tc := range []struct { + name string + env string + want []string + wantErr string + }{ + {name: "unset", want: nil}, + {name: "one", env: a, want: []string{a}}, + {name: "several, with stray separators", env: " " + a + ",," + b + ", ", want: []string{a, b}}, + {name: "not a SPIFFE ID", env: a + ",https://example.com/x", wantErr: "ingressAllowedSPIFFEIDs must be a comma-separated list of SPIFFE IDs"}, + } { + t.Run(tc.name, func(t *testing.T) { + loadEnv(t) + if tc.env != "" { + t.Setenv("ATE_INGRESS_ALLOWED_SPIFFE_IDS", tc.env) + } + cfg, err := Load(Options{}) + if tc.wantErr != "" { + if err == nil || !strings.Contains(err.Error(), tc.wantErr) { + t.Fatalf("Load() error = %v, want one containing %q", err, tc.wantErr) + } + return + } + if err != nil { + t.Fatalf("Load() error = %v", err) + } + if !slices.Equal(cfg.IngressAllowedSPIFFEIDs, tc.want) { + t.Errorf("IngressAllowedSPIFFEIDs = %q, want %q", cfg.IngressAllowedSPIFFEIDs, tc.want) + } + got, exported := scriptEnvMap(t, cfg)["ATE_INGRESS_ALLOWED_SPIFFE_IDS"] + if want := strings.Join(tc.want, ","); exported != (want != "") || got != want { + t.Errorf("ScriptEnv() ATE_INGRESS_ALLOWED_SPIFFE_IDS = %q (exported %v), want %q", got, exported, want) + } + }) + } +} + +// Only the Envoy router authenticates clients against the allowlist, and it +// refuses to start without one. +func TestRequireIngressAllowedSPIFFEIDs(t *testing.T) { + for _, tc := range []struct { + name string + cfg Config + wantErr bool + }{ + {name: "envoy without IDs", cfg: Config{Router: RouterEnvoy}, wantErr: true}, + {name: "envoy static-mtls without IDs", cfg: Config{Router: RouterEnvoy, IngressAuthMode: IngressAuthStaticMTLS}, wantErr: true}, + {name: "envoy with IDs", cfg: Config{Router: RouterEnvoy, IngressAllowedSPIFFEIDs: []string{"spiffe://cluster.local/ns/a/sa/a"}}}, + {name: "agentgateway without IDs", cfg: Config{Router: RouterAgentgateway}}, + } { + t.Run(tc.name, func(t *testing.T) { + err := tc.cfg.RequireIngressAllowedSPIFFEIDs() + if !tc.wantErr { + if err != nil { + t.Fatalf("RequireIngressAllowedSPIFFEIDs() = %v, want nil", err) + } + return + } + var required *RequiredError + if !errors.As(err, &required) || required.Setting.Key != "ingressAllowedSPIFFEIDs" { + t.Fatalf("RequireIngressAllowedSPIFFEIDs() = %v, want a RequiredError for ingressAllowedSPIFFEIDs", err) + } + }) + } +} + func TestLoadDockerBuildFlags(t *testing.T) { loadEnv(t) t.Setenv("DOCKER_BUILD_FLAGS", " --cache-from type=gha --cache-to type=gha,mode=max ") diff --git a/cmd/ate-setup/internal/config/coverage_test.go b/cmd/ate-setup/internal/config/coverage_test.go index 036b0739f0..23ad45729f 100644 --- a/cmd/ate-setup/internal/config/coverage_test.go +++ b/cmd/ate-setup/internal/config/coverage_test.go @@ -92,6 +92,8 @@ func valueFor(key string) string { return ClusterSizeSize0 case "ingressAuthMode": return IngressAuthStaticMTLS + case "ingressAllowedSPIFFEIDs": + return "spiffe://cluster.local/ns/a/sa/b" case "ateapi.postgres.cloudsql.ipType": return CloudSQLIPTypePSC case "atenet.egress.additionalExtprocService": diff --git a/cmd/ate-setup/internal/config/load.go b/cmd/ate-setup/internal/config/load.go index f819645b50..0861321817 100644 --- a/cmd/ate-setup/internal/config/load.go +++ b/cmd/ate-setup/internal/config/load.go @@ -22,6 +22,7 @@ import ( "strings" "github.com/spf13/pflag" + "github.com/spiffe/go-spiffe/v2/spiffeid" "go.ate.dev/substrate/cmd/ate-setup/internal/images" "go.ate.dev/substrate/internal/actoridjwt" @@ -275,6 +276,7 @@ func buildConfig(root string, env map[string]string, r *Resolved) (*Config, erro CordonControlPlane: r.Bool("cordonControlPlane"), FakeDataPlane: r.Bool("fakeDataPlane.enabled"), IngressAuthMode: r.String("ingressAuthMode"), + IngressAllowedSPIFFEIDs: splitList(r.String("ingressAllowedSPIFFEIDs")), AdditionalEgressExtprocService: r.String("atenet.egress.additionalExtprocService"), CredentialProviderJSON: r.String("atenet.egress.credentialProvider"), AnthropicAPIKey: r.String("demo.anthropicAPIKey"), @@ -422,6 +424,15 @@ func validateResolved(cfg *Config, r *Resolved) error { return &InvalidError{Value: authMode, Want: IngressAuthDeprecatedInsecure + " or " + IngressAuthStaticMTLS} } + // The router rejects a malformed ID at startup; catch it before the + // rollout does. + for _, id := range cfg.IngressAllowedSPIFFEIDs { + if _, err := spiffeid.FromString(id); err != nil { + ids, _ := r.Value("ingressAllowedSPIFFEIDs") + return &InvalidError{Value: ids, Want: fmt.Sprintf("a comma-separated list of SPIFFE IDs (%q: %v)", id, err)} + } + } + switch cfg.APILogLevel { case "", "debug", "info", "warn", "error": default: diff --git a/cmd/ate-setup/internal/config/setting.go b/cmd/ate-setup/internal/config/setting.go index 2a5fd61496..f7d2cfd1ce 100644 --- a/cmd/ate-setup/internal/config/setting.go +++ b/cmd/ate-setup/internal/config/setting.go @@ -304,9 +304,19 @@ var Registry = []Setting{ Key: "ingressAuthMode", Env: "ATE_INGRESS_AUTH_MODE", Flag: "ingress-auth-mode", Kind: KindString, Default: IngressAuthDeprecatedInsecure, Usage: "How the ingress router authenticates clients: deprecated-insecure or static-mtls. Both check a " + - "client certificate on the router's TLS ports against the podidentity CA and the SPIFFE IDs listed in " + - "manifests/ate-install/atenet-router.yaml; deprecated-insecure also lets through TLS clients without one " + - "and serves plaintext ports, while static-mtls requires the certificate and --atenet-dataplane=envoy", + "client certificate on the router's TLS ports against the podidentity CA and --ingress-allowed-spiffe-ids; " + + "deprecated-insecure also lets through TLS clients without one and serves plaintext ports, while " + + "static-mtls requires the certificate and --atenet-dataplane=envoy", + }, + { + // Comma-separated, as the router's own flag is. The registry has no + // list kind, so the value is carried as written and split where it is + // used. + Key: "ingressAllowedSPIFFEIDs", Env: "ATE_INGRESS_ALLOWED_SPIFFE_IDS", Flag: "ingress-allowed-spiffe-ids", + Kind: KindString, + Usage: "SPIFFE IDs (exact match, comma-separated) of the clients the ingress router lets reach actors over " + + "its TLS ports; any listed client may reach any actor. Required to deploy the router with " + + "--atenet-dataplane=envoy. The podidentity signer issues spiffe://cluster.local/ns//sa/", }, { Key: "otlpEndpoint", Env: "ATE_OTLP_ENDPOINT", Flag: "otlp-endpoint", Kind: KindString, diff --git a/cmd/ate-setup/internal/demos/counter/counter.go b/cmd/ate-setup/internal/demos/counter/counter.go index ec00f4fcdf..a3405a618d 100644 --- a/cmd/ate-setup/internal/demos/counter/counter.go +++ b/cmd/ate-setup/internal/demos/counter/counter.go @@ -42,7 +42,7 @@ func (d *demo) externalVolumeValues(e *steps.Env) map[string]string { "EXTERNAL_VOLUMES": "- name: external-data\n" + " external:\n" + " template:\n" + - " capacity: 1Gi\n" + + " capacityBytes: 1073741824 # 1Gi\n" + " storageClassName: " + sc, } } diff --git a/cmd/ate-setup/internal/steps/deploy.go b/cmd/ate-setup/internal/steps/deploy.go index c116307137..174821109c 100644 --- a/cmd/ate-setup/internal/steps/deploy.go +++ b/cmd/ate-setup/internal/steps/deploy.go @@ -68,6 +68,10 @@ func (e *Env) DeployAteSystem(ctx context.Context, opts DeployOptions) error { if err != nil { return err } + // Likewise a router that would refuse to start without an allowlist. + if err := e.Cfg.RequireIngressAllowedSPIFFEIDs(); err != nil { + return err + } // Likewise the CSI request, even though it is only acted on partway // through: a driver this cluster cannot run is worth knowing before the // control plane goes up, not after. @@ -527,6 +531,9 @@ func (e *Env) DeployAtenet(ctx context.Context) error { if err != nil { return err } + if err := e.Cfg.RequireIngressAllowedSPIFFEIDs(); err != nil { + return err + } if err := e.EnsureCRDs(ctx); err != nil { return err } diff --git a/cmd/ate-setup/internal/steps/overlay.go b/cmd/ate-setup/internal/steps/overlay.go index 597d280d15..396cd4d89c 100644 --- a/cmd/ate-setup/internal/steps/overlay.go +++ b/cmd/ate-setup/internal/steps/overlay.go @@ -151,6 +151,10 @@ func (e *Env) renderSystemManifests(ctx context.Context) ([]byte, error) { return nil, err } if e.Cfg.Router != config.RouterAgentgateway { + manifest, err = e.patchAtenetRouterAllowedSPIFFEIDs(manifest) + if err != nil { + return nil, err + } imageReference, err := e.dockerfileImage(ctx, envoyDataplaneImage, envoyDataplaneDockefile) if err != nil { return nil, err @@ -170,6 +174,10 @@ func (e *Env) renderAtenetRouterManifest(ctx context.Context) ([]byte, error) { if err != nil { return nil, err } + manifest, err = e.patchAtenetRouterAllowedSPIFFEIDs(manifest) + if err != nil { + return nil, err + } imageReference, err := e.dockerfileImage(ctx, envoyDataplaneImage, envoyDataplaneDockefile) if err != nil { return nil, err @@ -241,10 +249,29 @@ func (e *Env) renderAtenetEgressManifest(ctx context.Context, provider config.Cr return e.ResolveManifestBytes(ctx, rendered) } -// patchAtenetEgressEnvoyConcurrency uses a kustomize JSON6902 patch to append -// --concurrency to the envoy container's args in the -// atenet-egress Deployment. +// patchAtenetEgressEnvoyConcurrency appends --concurrency to the +// envoy container's args in the atenet-egress Deployment. func patchAtenetEgressEnvoyConcurrency(raw []byte, concurrency int) ([]byte, error) { + return appendContainerArgs(raw, "atenet-egress", "envoy", "--concurrency", strconv.Itoa(concurrency)) +} + +// patchAtenetRouterAllowedSPIFFEIDs appends --ingress-allowed-spiffe-ids to the +// atenet-router container's args in the Envoy router Deployment. The base +// manifest carries no allowlist, because which clients may reach actors is the +// install's to decide. +func (e *Env) patchAtenetRouterAllowedSPIFFEIDs(raw []byte) ([]byte, error) { + if err := e.Cfg.RequireIngressAllowedSPIFFEIDs(); err != nil { + return nil, err + } + return appendContainerArgs(raw, "atenet-router", "atenet-router", + "--ingress-allowed-spiffe-ids="+strings.Join(e.Cfg.IngressAllowedSPIFFEIDs, ",")) +} + +// appendContainerArgs uses a kustomize JSON6902 patch to append args to the +// first container of the named Deployment in ate-system. The patch first +// tests that the first container is the one named, so a reordered manifest +// fails the render instead of patching the wrong container. +func appendContainerArgs(raw []byte, deployment, container string, args ...string) ([]byte, error) { dir, err := os.MkdirTemp("", "ate-setup-kustomize-") if err != nil { return nil, fmt.Errorf("while creating a scratch kustomization: %w", err) @@ -254,7 +281,8 @@ func patchAtenetEgressEnvoyConcurrency(raw []byte, concurrency int) ([]byte, err if err := os.WriteFile(filepath.Join(dir, "manifest.yaml"), raw, 0o600); err != nil { return nil, fmt.Errorf("while staging a manifest for kustomize: %w", err) } - kustomization := fmt.Sprintf(`apiVersion: kustomize.config.k8s.io/v1beta1 + var kustomization strings.Builder + fmt.Fprintf(&kustomization, `apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - manifest.yaml @@ -263,20 +291,20 @@ patches: group: apps version: v1 kind: Deployment - name: atenet-egress + name: %s namespace: ate-system patch: |- - op: test path: /spec/template/spec/containers/0/name - value: envoy - - op: add - path: /spec/template/spec/containers/0/args/- - value: --concurrency - - op: add + value: %q +`, deployment, container) + for _, arg := range args { + fmt.Fprintf(&kustomization, ` - op: add path: /spec/template/spec/containers/0/args/- value: %q -`, strconv.Itoa(concurrency)) - if err := os.WriteFile(filepath.Join(dir, "kustomization.yaml"), []byte(kustomization), 0o600); err != nil { +`, arg) + } + if err := os.WriteFile(filepath.Join(dir, "kustomization.yaml"), []byte(kustomization.String()), 0o600); err != nil { return nil, fmt.Errorf("while writing a scratch kustomization: %w", err) } return kustomize.Build(dir) diff --git a/cmd/ate-setup/internal/steps/overlay_test.go b/cmd/ate-setup/internal/steps/overlay_test.go index be583dbef0..619022b42e 100644 --- a/cmd/ate-setup/internal/steps/overlay_test.go +++ b/cmd/ate-setup/internal/steps/overlay_test.go @@ -16,6 +16,7 @@ package steps import ( "context" + "errors" "fmt" "os" "reflect" @@ -837,6 +838,73 @@ func TestRenderRouterStaticMTLS(t *testing.T) { } } +// Both paths that deliver the Envoy router pass it the install's allowlist, +// whatever the overlay and auth mode, and refuse to render without one. +func TestRenderRouterAllowedSPIFFEIDs(t *testing.T) { + const a, b = "spiffe://cluster.local/ns/a/sa/a", "spiffe://cluster.local/ns/b/sa/b" + src := images.Source{Repo: "example.com/substrate", Tag: "v1.2.3"} + resolver := images.NewPrebuilt(src, func(context.Context, string) (string, error) { + return "sha256:2222222222222222222222222222222222222222222222222222222222222222", nil + }) + for _, tc := range []struct { + name string + cfg config.Config + render func(e *Env) ([]byte, error) + }{ + { + name: "base bundle", + cfg: config.Config{}, + render: func(e *Env) ([]byte, error) { return e.renderSystemManifests(t.Context()) }, + }, + { + name: "kind bundle static-mtls", + cfg: config.Config{Kind: true, IngressAuthMode: config.IngressAuthStaticMTLS}, + render: func(e *Env) ([]byte, error) { return e.renderSystemManifests(t.Context()) }, + }, + { + name: "router file", + cfg: config.Config{}, + render: func(e *Env) ([]byte, error) { return e.renderAtenetRouterManifest(t.Context()) }, + }, + { + name: "router file static-mtls", + cfg: config.Config{IngressAuthMode: config.IngressAuthStaticMTLS}, + render: func(e *Env) ([]byte, error) { return e.renderAtenetRouterManifest(t.Context()) }, + }, + } { + t.Run(tc.name, func(t *testing.T) { + cfg := tc.cfg + cfg.Root = repoRoot(t) + cfg.Router = config.RouterEnvoy + cfg.Images = src + cfg.IngressAllowedSPIFFEIDs = []string{a, b} + rendered, err := tc.render(&Env{Cfg: &cfg, resolver: resolver}) + if err != nil { + t.Fatalf("render: %v", err) + } + var got []string + for _, arg := range routerArgs(t, rendered) { + if strings.HasPrefix(arg, "--ingress-allowed-spiffe-ids") { + got = append(got, arg) + } + } + if want := []string{"--ingress-allowed-spiffe-ids=" + a + "," + b}; !slices.Equal(got, want) { + t.Errorf("atenet-router allowlist args = %q, want %q", got, want) + } + + t.Run("without IDs", func(t *testing.T) { + cfg := cfg + cfg.IngressAllowedSPIFFEIDs = nil + _, err := tc.render(&Env{Cfg: &cfg, resolver: resolver}) + var required *config.RequiredError + if !errors.As(err, &required) { + t.Fatalf("render error = %v, want a RequiredError", err) + } + }) + }) + } +} + // routerArgs returns the atenet-router container's args from a rendered // manifest. func routerArgs(t *testing.T, manifest []byte) []string { @@ -1147,7 +1215,8 @@ func TestRenderAtenetEgressManifestPrebuilt(t *testing.T) { src := images.Source{Repo: "example.com/substrate", Tag: "v1.2.3"} var looked []string e := &Env{ - Cfg: &config.Config{Root: repoRoot(t), Router: config.RouterEnvoy, Images: src}, + Cfg: &config.Config{Root: repoRoot(t), Router: config.RouterEnvoy, Images: src, + IngressAllowedSPIFFEIDs: []string{"spiffe://cluster.local/ns/a/sa/a"}}, resolver: images.NewPrebuilt(src, func(_ context.Context, ref string) (string, error) { looked = append(looked, ref) return digest, nil diff --git a/cmd/ate-setup/internal/steps/substrate_test.go b/cmd/ate-setup/internal/steps/substrate_test.go index 5728c2eb90..8feaaba3ea 100644 --- a/cmd/ate-setup/internal/steps/substrate_test.go +++ b/cmd/ate-setup/internal/steps/substrate_test.go @@ -43,10 +43,11 @@ func TestActorTemplateFromManifestDemos(t *testing.T) { // The values benchmarking/workloads/deploy.sh substitutes, beyond the // ${BUCKET_NAME} every installer provides. benchmarkValues := map[string]string{ - "${ACTOR_CPU}": "1", - "${ACTOR_MEMORY}": "256Mi", - "${ATE_ENV_IMAGE}": "gcr.io/example/ate-env-guest@sha256:1d38934ef1dac71fcb1f1286de734e06ab2e19747279bccbd37bfe69e0d5bc72", - "${OTLP_ENDPOINT}": "http://otel-collector.otel-system:4317", + "${ACTOR_CPU_MILLICORES}": "1000", + "${ACTOR_MEMORY_BYTES}": "268435456", + "${ACTOR_MEMORY}": "256Mi", + "${ATE_ENV_IMAGE}": "gcr.io/example/ate-env-guest@sha256:1d38934ef1dac71fcb1f1286de734e06ab2e19747279bccbd37bfe69e0d5bc72", + "${OTLP_ENDPOINT}": "http://otel-collector.otel-system:4317", } tests := []struct { diff --git a/cmd/ate-setup/shim_test.go b/cmd/ate-setup/shim_test.go index d3430bbe54..9c76926869 100644 --- a/cmd/ate-setup/shim_test.go +++ b/cmd/ate-setup/shim_test.go @@ -214,6 +214,13 @@ func TestShimTranslatesFlags(t *testing.T) { name: "cluster profile flags are forwarded in either value form", args: []string{"--deploy-ate-system", "--cluster-size", "size10", "--cordon-control-plane"}, want: []string{"--cluster-size=size10 --cordon-control-plane deploy ate-system --setup-csi=none"}, + }, { + name: "--ingress-allowed-spiffe-ids is forwarded in either value form", + args: []string{"--deploy-ate-system", "--ingress-allowed-spiffe-ids", "spiffe://cluster.local/ns/a/sa/a", "--deploy-atenet", "--ingress-allowed-spiffe-ids=spiffe://cluster.local/ns/b/sa/b"}, + want: []string{ + "--ingress-allowed-spiffe-ids=spiffe://cluster.local/ns/a/sa/a --ingress-allowed-spiffe-ids=spiffe://cluster.local/ns/b/sa/b deploy ate-system --setup-csi=none", + "--ingress-allowed-spiffe-ids=spiffe://cluster.local/ns/a/sa/a --ingress-allowed-spiffe-ids=spiffe://cluster.local/ns/b/sa/b deploy atenet", + }, }, { name: "--fake-data-plane is forwarded to every action", args: []string{"--deploy-ate-system", "--fake-data-plane", "--deploy-atelet"}, diff --git a/cmd/ateapi/internal/apivalidation/actor_template.go b/cmd/ateapi/internal/apivalidation/actor_template.go index 7f20e9322b..2931bb2a71 100644 --- a/cmd/ateapi/internal/apivalidation/actor_template.go +++ b/cmd/ateapi/internal/apivalidation/actor_template.go @@ -22,7 +22,6 @@ import ( "go.ate.dev/substrate/internal/resources" "go.ate.dev/substrate/pkg/proto/ateapipb" "k8s.io/apimachinery/pkg/api/operation" - "k8s.io/apimachinery/pkg/api/resource" "k8s.io/apimachinery/pkg/util/sets" "k8s.io/apimachinery/pkg/util/validation/field" ) @@ -84,16 +83,13 @@ func ValidateCustom_CreateActorTemplateRequest_ActorTemplate(_ context.Context, // ValidateCustom_ActorTemplate_Resources requires both a cpu and a memory // limit, which the scheduler books against worker capacity. -func ValidateCustom_ActorTemplate_Resources(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *ateapipb.Resources) field.ErrorList { - set := sets.New[string]() - for _, l := range value.GetLimits() { - set.Insert(l.GetName()) - } +func ValidateCustom_ActorTemplate_Resources(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *ateapipb.ResourceRequirements) field.ErrorList { var errs field.ErrorList - for _, name := range []string{resources.ResourceCPU, resources.ResourceMemory} { - if !set.Has(name) { - errs = append(errs, field.Required(fldPath.Child("limits"), "must set a "+name+" limit")) - } + if value.GetLimits().GetCpuMillicores() == 0 { + errs = append(errs, field.Required(fldPath.Child("limits", "cpu_millicores"), "")) + } + if value.GetLimits().GetMemoryBytes() == 0 { + errs = append(errs, field.Required(fldPath.Child("limits", "memory_bytes"), "")) } return errs } @@ -167,13 +163,6 @@ func ValidateCustom_Container_Image(_ context.Context, _ operation.Operation, fl return resources.ValidatePinnedImage(fldPath, *value) } -func ValidateCustom_ExternalVolumeTemplate_Capacity(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *string) field.ErrorList { - if _, err := resource.ParseQuantity(*value); err != nil { - return field.ErrorList{field.Invalid(fldPath, *value, fmt.Sprintf("must be a Kubernetes resource quantity: %v", err))} - } - return nil -} - // ValidateCustom_SnapshotConfig_StorageLocation ensures an // ActorTemplate's snapshotConfig.location is a well-formed // URI with a bucket, so a bad location fails fast. diff --git a/cmd/ateapi/internal/apivalidation/actor_template_test.go b/cmd/ateapi/internal/apivalidation/actor_template_test.go index 52bf65d114..480e86f3a9 100644 --- a/cmd/ateapi/internal/apivalidation/actor_template_test.go +++ b/cmd/ateapi/internal/apivalidation/actor_template_test.go @@ -159,25 +159,25 @@ func TestValidateCreateActorTemplateRequest(t *testing.T) { })}, field.ErrorList{field.Required(field.NewPath("actor_template", "resources"), "")}, }, { - "missing cpu limit", + "missing cpu", &ateapipb.CreateActorTemplateRequest{ActorTemplate: validActorTemplate(func(tmpl *ateapipb.ActorTemplate) { - tmpl.Resources = resources.CPUMemory(0, 512<<20) + tmpl.Resources = &ateapipb.ResourceRequirements{Limits: resources.CPUMemory(0, 512<<20)} })}, - field.ErrorList{field.Required(field.NewPath("actor_template", "resources", "limits"), "must set a cpu limit")}, + field.ErrorList{field.Required(field.NewPath("actor_template", "resources", "limits", "cpu_millicores"), "")}, }, { - "missing memory limit", + "missing memory", &ateapipb.CreateActorTemplateRequest{ActorTemplate: validActorTemplate(func(tmpl *ateapipb.ActorTemplate) { - tmpl.Resources = resources.CPUMemory(1000, 0) + tmpl.Resources = &ateapipb.ResourceRequirements{Limits: resources.CPUMemory(1000, 0)} })}, - field.ErrorList{field.Required(field.NewPath("actor_template", "resources", "limits"), "must set a memory limit")}, + field.ErrorList{field.Required(field.NewPath("actor_template", "resources", "limits", "memory_bytes"), "")}, }, { - "empty limits", + "empty resources", &ateapipb.CreateActorTemplateRequest{ActorTemplate: validActorTemplate(func(tmpl *ateapipb.ActorTemplate) { - tmpl.Resources = &ateapipb.Resources{} + tmpl.Resources = &ateapipb.ResourceRequirements{} })}, field.ErrorList{ - field.Required(field.NewPath("actor_template", "resources", "limits"), "must set a cpu limit"), - field.Required(field.NewPath("actor_template", "resources", "limits"), "must set a memory limit"), + field.Required(field.NewPath("actor_template", "resources", "limits", "cpu_millicores"), ""), + field.Required(field.NewPath("actor_template", "resources", "limits", "memory_bytes"), ""), }, }} for _, tt := range tests { @@ -190,7 +190,7 @@ func TestValidateCreateActorTemplateRequest(t *testing.T) { func TestValidateCreateActorTemplateRequest_PreviewRestricted(t *testing.T) { setPreviewForTest(t, "ALL") req := &ateapipb.CreateActorTemplateRequest{ActorTemplate: validActorTemplate(func(tmpl *ateapipb.ActorTemplate) { - tmpl.Volumes = []*ateapipb.Volume{{Name: "data", External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{Capacity: "10Gi", StorageClassName: "fast-ssd"}}}} + tmpl.Volumes = []*ateapipb.Volume{{Name: "data", External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{CapacityBytes: 10 << 30, StorageClassName: "fast-ssd"}}}} })} t.Run("allowed", func(t *testing.T) { @@ -827,7 +827,7 @@ func TestValidateActorTemplate(t *testing.T) { }, { name: "valid external volume template", mutate: func(tmpl *ateapipb.ActorTemplate) { - tmpl.Volumes = []*ateapipb.Volume{{Name: "data", External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{Capacity: "10Gi", StorageClassName: "fast-ssd"}}}} + tmpl.Volumes = []*ateapipb.Volume{{Name: "data", External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{CapacityBytes: 10 << 30, StorageClassName: "fast-ssd"}}}} }, want: field.ErrorList{field.Forbidden(field.NewPath("volumes").Index(0).Child("external"), "")}, wantWithPreview: map[string]field.ErrorList{"ALL": nil}, @@ -837,102 +837,57 @@ func TestValidateActorTemplate(t *testing.T) { tmpl.Volumes = []*ateapipb.Volume{{Name: "data", External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{StorageClassName: "fast-ssd"}}}} }, want: field.ErrorList{field.Forbidden(field.NewPath("volumes").Index(0).Child("external"), "")}, - wantWithPreview: map[string]field.ErrorList{"ALL": {field.Required(field.NewPath("volumes").Index(0).Child("external", "template", "capacity"), "")}}, + wantWithPreview: map[string]field.ErrorList{"ALL": {field.Required(field.NewPath("volumes").Index(0).Child("external", "template", "capacity_bytes"), "")}}, }, { - name: "external volume template malformed capacity", + name: "external volume template negative capacity", mutate: func(tmpl *ateapipb.ActorTemplate) { - tmpl.Volumes = []*ateapipb.Volume{{Name: "data", External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{Capacity: "ten gigs", StorageClassName: "fast-ssd"}}}} + tmpl.Volumes = []*ateapipb.Volume{{Name: "data", External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{CapacityBytes: -1, StorageClassName: "fast-ssd"}}}} }, want: field.ErrorList{field.Forbidden(field.NewPath("volumes").Index(0).Child("external"), "")}, - wantWithPreview: map[string]field.ErrorList{"ALL": {field.Invalid(field.NewPath("volumes").Index(0).Child("external", "template", "capacity"), nil, "")}}, - }, { - name: "external volume template capacity at the length bound", - mutate: func(tmpl *ateapipb.ActorTemplate) { - tmpl.Volumes = []*ateapipb.Volume{{Name: "data", External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{Capacity: strings.Repeat("1", 30) + "Gi", StorageClassName: "fast-ssd"}}}} - }, - want: field.ErrorList{field.Forbidden(field.NewPath("volumes").Index(0).Child("external"), "")}, - wantWithPreview: map[string]field.ErrorList{"ALL": nil}, - }, { - name: "external volume template capacity too long", - mutate: func(tmpl *ateapipb.ActorTemplate) { - tmpl.Volumes = []*ateapipb.Volume{{Name: "data", External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{Capacity: strings.Repeat("1", 31) + "Gi", StorageClassName: "fast-ssd"}}}} - }, - want: field.ErrorList{field.Forbidden(field.NewPath("volumes").Index(0).Child("external"), "")}, - wantWithPreview: map[string]field.ErrorList{"ALL": {field.TooLong(field.NewPath("volumes").Index(0).Child("external", "template", "capacity"), nil, 32).WithOrigin("maxLength")}}, + wantWithPreview: map[string]field.ErrorList{"ALL": {field.Invalid(field.NewPath("volumes").Index(0).Child("external", "template", "capacity_bytes"), nil, "").WithOrigin("minimum")}}, }, { name: "external volume template missing storage_class_name", mutate: func(tmpl *ateapipb.ActorTemplate) { - tmpl.Volumes = []*ateapipb.Volume{{Name: "data", External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{Capacity: "10Gi"}}}} + tmpl.Volumes = []*ateapipb.Volume{{Name: "data", External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{CapacityBytes: 10 << 30}}}} }, want: field.ErrorList{field.Forbidden(field.NewPath("volumes").Index(0).Child("external"), "")}, wantWithPreview: map[string]field.ErrorList{"ALL": {field.Required(field.NewPath("volumes").Index(0).Child("external", "template", "storage_class_name"), "")}}, }, { name: "external volume template invalid storage_class_name", mutate: func(tmpl *ateapipb.ActorTemplate) { - tmpl.Volumes = []*ateapipb.Volume{{Name: "data", External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{Capacity: "10Gi", StorageClassName: "Fast SSD"}}}} + tmpl.Volumes = []*ateapipb.Volume{{Name: "data", External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{CapacityBytes: 10 << 30, StorageClassName: "Fast SSD"}}}} }, want: field.ErrorList{field.Forbidden(field.NewPath("volumes").Index(0).Child("external"), "")}, wantWithPreview: map[string]field.ErrorList{"ALL": {field.Invalid(field.NewPath("volumes").Index(0).Child("external", "template", "storage_class_name"), nil, "").WithOrigin("format=k8s-long-name")}}, }, { name: "valid resources", mutate: func(tmpl *ateapipb.ActorTemplate) { - tmpl.Containers[0].Resources = &ateapipb.Resources{Limits: []*ateapipb.Limits{ - {Name: "cpu", Quantity: "500m"}, {Name: "memory", Quantity: "2Gi"}, - }} + tmpl.Containers[0].Resources = &ateapipb.ResourceRequirements{Limits: resources.CPUMemory(500, 2<<30)} }, }, { - name: "limit missing name", + name: "negative cpu", mutate: func(tmpl *ateapipb.ActorTemplate) { - tmpl.Containers[0].Resources = &ateapipb.Resources{Limits: []*ateapipb.Limits{{Quantity: "2Gi"}}} - }, - want: field.ErrorList{ - field.Required(field.NewPath("containers").Index(0).Child("resources", "limits").Index(0).Child("name"), ""), - field.NotSupported[string](field.NewPath("containers").Index(0).Child("resources", "limits").Index(0).Child("name"), nil, nil), + tmpl.Containers[0].Resources = &ateapipb.ResourceRequirements{Limits: &ateapipb.Resources{CpuMillicores: -1}} }, + want: field.ErrorList{field.Invalid(field.NewPath("containers").Index(0).Child("resources", "limits", "cpu_millicores"), nil, "").WithOrigin("minimum")}, }, { - name: "limit missing quantity", + name: "negative memory", mutate: func(tmpl *ateapipb.ActorTemplate) { - tmpl.Containers[0].Resources = &ateapipb.Resources{Limits: []*ateapipb.Limits{{Name: "cpu"}}} + tmpl.Containers[0].Resources = &ateapipb.ResourceRequirements{Limits: &ateapipb.Resources{MemoryBytes: -1}} }, - want: field.ErrorList{field.Required(field.NewPath("containers").Index(0).Child("resources", "limits").Index(0).Child("quantity"), "")}, + want: field.ErrorList{field.Invalid(field.NewPath("containers").Index(0).Child("resources", "limits", "memory_bytes"), nil, "").WithOrigin("minimum")}, }, { - name: "unsupported limit name", + name: "cpu above maximum", mutate: func(tmpl *ateapipb.ActorTemplate) { - tmpl.Containers[0].Resources = &ateapipb.Resources{Limits: []*ateapipb.Limits{{Name: "gpu", Quantity: "1"}}} + tmpl.Containers[0].Resources = &ateapipb.ResourceRequirements{Limits: &ateapipb.Resources{CpuMillicores: 1000000}} }, - want: field.ErrorList{field.NotSupported[string](field.NewPath("containers").Index(0).Child("resources", "limits").Index(0).Child("name"), nil, nil)}, - }, { - name: "duplicate limit name", - mutate: func(tmpl *ateapipb.ActorTemplate) { - tmpl.Containers[0].Resources = &ateapipb.Resources{Limits: []*ateapipb.Limits{ - {Name: "cpu", Quantity: "1"}, {Name: "cpu", Quantity: "2"}, - }} - }, - want: field.ErrorList{field.Duplicate(field.NewPath("containers").Index(0).Child("resources", "limits").Index(1), nil)}, - }, { - name: "malformed limit quantity", - mutate: func(tmpl *ateapipb.ActorTemplate) { - tmpl.Containers[0].Resources = &ateapipb.Resources{Limits: []*ateapipb.Limits{{Name: "memory", Quantity: "two gigs"}}} - }, - want: field.ErrorList{field.Invalid(field.NewPath("containers").Index(0).Child("resources", "limits").Index(0).Child("quantity"), nil, "")}, - }, { - name: "too many limits", - mutate: func(tmpl *ateapipb.ActorTemplate) { - tmpl.Containers[0].Resources = &ateapipb.Resources{Limits: []*ateapipb.Limits{ - {Name: "cpu", Quantity: "1"}, {Name: "memory", Quantity: "1Gi"}, {Name: "cpu", Quantity: "2"}, - }} - }, - // maxItems short-circuits the per-item and uniqueness checks. - want: field.ErrorList{field.TooMany(field.NewPath("containers").Index(0).Child("resources", "limits"), 3, 2).WithOrigin("maxItems")}, + want: field.ErrorList{field.Invalid(field.NewPath("containers").Index(0).Child("resources", "limits", "cpu_millicores"), nil, "").WithOrigin("maximum")}, }, { name: "template-level resources validated too", mutate: func(tmpl *ateapipb.ActorTemplate) { - tmpl.Resources = &ateapipb.Resources{Limits: []*ateapipb.Limits{{Name: "cpu", Quantity: "1"}, {Name: "gpu", Quantity: "1"}}} - }, - want: field.ErrorList{ - field.NotSupported[string](field.NewPath("resources", "limits").Index(1).Child("name"), nil, nil), - field.Required(field.NewPath("resources", "limits"), "must set a memory limit"), + tmpl.Resources = &ateapipb.ResourceRequirements{Limits: &ateapipb.Resources{CpuMillicores: 1000, MemoryBytes: -1}} }, + want: field.ErrorList{field.Invalid(field.NewPath("resources", "limits", "memory_bytes"), nil, "").WithOrigin("minimum")}, }} for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { @@ -960,7 +915,7 @@ func validActorTemplate(mutations ...func(*ateapipb.ActorTemplate)) *ateapipb.Ac StorageLocation: "gs://my-bucket/snapshots", PreferredFidelity: ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, }, - Resources: resources.CPUMemory(1000, 512<<20), + Resources: &ateapipb.ResourceRequirements{Limits: resources.CPUMemory(1000, 512<<20)}, } for _, m := range mutations { m(template) diff --git a/cmd/ateapi/internal/apivalidation/common.go b/cmd/ateapi/internal/apivalidation/common.go index 4e4cafcfc1..8ebf87cbca 100644 --- a/cmd/ateapi/internal/apivalidation/common.go +++ b/cmd/ateapi/internal/apivalidation/common.go @@ -91,12 +91,6 @@ func ValidateCustom_ResourceMetadata(_ context.Context, _ operation.Operation, f return errs } -// ValidateCustom_Limits validates one limit with resources.ValidateLimit. -// Presence and uniqueness of names are enforced by tags. -func ValidateCustom_Limits(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *ateapipb.Limits) field.ErrorList { - return resources.ValidateLimit(fldPath, value.GetName(), value.GetQuantity()) -} - // validateDualStackIPs checks that each entry is a valid, canonical IP and // that the list holds at most one IPv4 and one IPv6 address, matching the // Kubernetes rule for pod IPs. Presence and length are enforced by tags. diff --git a/cmd/ateapi/internal/apivalidation/worker_test.go b/cmd/ateapi/internal/apivalidation/worker_test.go index f9f198cdb6..a0b60ea8ea 100644 --- a/cmd/ateapi/internal/apivalidation/worker_test.go +++ b/cmd/ateapi/internal/apivalidation/worker_test.go @@ -354,10 +354,8 @@ func TestValidateRegisterWorkerRequest(t *testing.T) { r := &ateapipb.RegisterWorkerRequest{ Worker: workerRef(apiWorkerName), Capacity: &ateapipb.WorkerResources{ - Actors: 10, - Resources: &ateapipb.Resources{ - Limits: []*ateapipb.Limits{{Name: "cpu", Quantity: "2"}, {Name: "memory", Quantity: "4Gi"}}, - }, + Actors: 10, + CpuMillicores: 2000, MemoryBytes: 4 << 30, }, DefaultRuntime: validRuntime(), } @@ -416,11 +414,9 @@ func TestValidateRegisterWorkerRequest(t *testing.T) { req: valid(func(r *ateapipb.RegisterWorkerRequest) { r.Capacity.Actors = -1 }), want: field.ErrorList{field.Invalid(field.NewPath("capacity", "actors"), nil, "").WithOrigin("minimum")}, }, { - name: "invalid limit quantity", - req: valid(func(r *ateapipb.RegisterWorkerRequest) { - r.Capacity.Resources.Limits = []*ateapipb.Limits{{Name: "cpu", Quantity: "lots"}} - }), - want: field.ErrorList{field.Invalid(field.NewPath("capacity", "resources", "limits").Index(0).Child("quantity"), nil, "")}, + name: "negative capacity.memory_bytes", + req: valid(func(r *ateapipb.RegisterWorkerRequest) { r.Capacity.MemoryBytes = -1 }), + want: field.ErrorList{field.Invalid(field.NewPath("capacity", "memory_bytes"), nil, "").WithOrigin("minimum")}, }, { name: "missing default_runtime", req: valid(func(r *ateapipb.RegisterWorkerRequest) { r.DefaultRuntime = nil }), diff --git a/cmd/ateapi/internal/apivalidation/zz_generated.validation.go b/cmd/ateapi/internal/apivalidation/zz_generated.validation.go index 9067a434e4..e10bfb8a0f 100644 --- a/cmd/ateapi/internal/apivalidation/zz_generated.validation.go +++ b/cmd/ateapi/internal/apivalidation/zz_generated.validation.go @@ -1224,7 +1224,7 @@ func Validate_ActorTemplate( { // field ateapipb.ActorTemplate.Resources fn := func( fldPath *field.Path, - obj, oldObj *ateapipb.Resources, + obj, oldObj *ateapipb.ResourceRequirements, oldValueCorrelated bool) (errs field.ErrorList) { // don't revalidate unchanged data if oldValueCorrelated && op.Type == operation.Update { @@ -1246,11 +1246,11 @@ func Validate_ActorTemplate( errs = append(errs, e...) } // call the type's validation function - errs = append(errs, Validate_Resources(ctx, op, fldPath, obj, oldObj)...) + errs = append(errs, Validate_ResourceRequirements(ctx, op, fldPath, obj, oldObj)...) return } oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.ActorTemplate) *ateapipb.Resources { + func(oldObj *ateapipb.ActorTemplate) *ateapipb.ResourceRequirements { return oldObj.Resources }) errs = append(errs, fn(fldPath.Child("resources"), obj.Resources, oldVal, oldObj != nil)...) @@ -1946,7 +1946,7 @@ func Validate_Container( { // field ateapipb.Container.Resources fn := func( fldPath *field.Path, - obj, oldObj *ateapipb.Resources, + obj, oldObj *ateapipb.ResourceRequirements, oldValueCorrelated bool) (errs field.ErrorList) { // don't revalidate unchanged data if oldValueCorrelated && op.Type == operation.Update { @@ -1963,11 +1963,11 @@ func Validate_Container( return // do not proceed } // call the type's validation function - errs = append(errs, Validate_Resources(ctx, op, fldPath, obj, oldObj)...) + errs = append(errs, Validate_ResourceRequirements(ctx, op, fldPath, obj, oldObj)...) return } oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.Container) *ateapipb.Resources { + func(oldObj *ateapipb.Container) *ateapipb.ResourceRequirements { return oldObj.Resources }) errs = append(errs, fn(fldPath.Child("resources"), obj.Resources, oldVal, oldObj != nil)...) @@ -3892,10 +3892,10 @@ func Validate_ExternalVolumeTemplate( ctx context.Context, op operation.Operation, fldPath *field.Path, obj, oldObj *ateapipb.ExternalVolumeTemplate) (errs field.ErrorList) { - { // field ateapipb.ExternalVolumeTemplate.Capacity + { // field ateapipb.ExternalVolumeTemplate.CapacityBytes fn := func( fldPath *field.Path, - obj, oldObj *string, + obj, oldObj *int64, oldValueCorrelated bool) (errs field.ErrorList) { // don't revalidate unchanged data if oldValueCorrelated && op.Type == operation.Update { @@ -3912,20 +3912,16 @@ func Validate_ExternalVolumeTemplate( if earlyReturn { return // do not proceed } - // custom validation - if e := ValidateCustom_ExternalVolumeTemplate_Capacity(ctx, op, fldPath, obj, oldObj); len(e) != 0 { - errs = append(errs, e...) - } - if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 32); len(e) != 0 { + if e := validate.Minimum(ctx, op, fldPath, obj, oldObj, 1); len(e) != 0 { errs = append(errs, e...) } return } oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.ExternalVolumeTemplate) *string { - return &oldObj.Capacity + func(oldObj *ateapipb.ExternalVolumeTemplate) *int64 { + return &oldObj.CapacityBytes }) - errs = append(errs, fn(fldPath.Child("capacity"), &obj.Capacity, oldVal, oldObj != nil)...) + errs = append(errs, fn(fldPath.Child("capacity_bytes"), &obj.CapacityBytes, oldVal, oldObj != nil)...) } { // field ateapipb.ExternalVolumeTemplate.StorageClassName @@ -5094,84 +5090,6 @@ func Validate_ImageVolumeSource( return errs } -// Validate_Limits validates an instance of Limits according -// to declarative validation rules in the API schema. -func Validate_Limits( - ctx context.Context, op operation.Operation, fldPath *field.Path, - obj, oldObj *ateapipb.Limits) (errs field.ErrorList) { - - // custom validation - if e := ValidateCustom_Limits(ctx, op, fldPath, obj, oldObj); len(e) != 0 { - errs = append(errs, e...) - } - - { // field ateapipb.Limits.Name - fn := func( - fldPath *field.Path, - obj, oldObj *string, - oldValueCorrelated bool) (errs field.ErrorList) { - // don't revalidate unchanged data - if oldValueCorrelated && op.Type == operation.Update { - if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { - return nil - } - } - // call field-attached validations - earlyReturn := false - if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { - errs = append(errs, e...) - earlyReturn = true - } - if earlyReturn { - return // do not proceed - } - if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 16); len(e) != 0 { - errs = append(errs, e...) - } - return - } - oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.Limits) *string { - return &oldObj.Name - }) - errs = append(errs, fn(fldPath.Child("name"), &obj.Name, oldVal, oldObj != nil)...) - } - - { // field ateapipb.Limits.Quantity - fn := func( - fldPath *field.Path, - obj, oldObj *string, - oldValueCorrelated bool) (errs field.ErrorList) { - // don't revalidate unchanged data - if oldValueCorrelated && op.Type == operation.Update { - if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { - return nil - } - } - // call field-attached validations - earlyReturn := false - if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { - errs = append(errs, e...) - earlyReturn = true - } - if earlyReturn { - return // do not proceed - } - if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 32); len(e) != 0 { - errs = append(errs, e...) - } - return - } - oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.Limits) *string { - return &oldObj.Quantity - }) - errs = append(errs, fn(fldPath.Child("quantity"), &obj.Quantity, oldVal, oldObj != nil)...) - } - - return errs -} - // Validate_ListActorTemplatesRequest validates an instance of ListActorTemplatesRequest according // to declarative validation rules in the API schema. func Validate_ListActorTemplatesRequest( @@ -7289,56 +7207,114 @@ func Validate_ResourceMetadata( return errs } +// Validate_ResourceRequirements validates an instance of ResourceRequirements according +// to declarative validation rules in the API schema. +func Validate_ResourceRequirements( + ctx context.Context, op operation.Operation, fldPath *field.Path, + obj, oldObj *ateapipb.ResourceRequirements) (errs field.ErrorList) { + + { // field ateapipb.ResourceRequirements.Limits + fn := func( + fldPath *field.Path, + obj, oldObj *ateapipb.Resources, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.OptionalPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // call the type's validation function + errs = append(errs, Validate_Resources(ctx, op, fldPath, obj, oldObj)...) + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateapipb.ResourceRequirements) *ateapipb.Resources { + return oldObj.Limits + }) + errs = append(errs, fn(fldPath.Child("limits"), obj.Limits, oldVal, oldObj != nil)...) + } + + return errs +} + // Validate_Resources validates an instance of Resources according // to declarative validation rules in the API schema. func Validate_Resources( ctx context.Context, op operation.Operation, fldPath *field.Path, obj, oldObj *ateapipb.Resources) (errs field.ErrorList) { - { // field ateapipb.Resources.Limits + { // field ateapipb.Resources.CpuMillicores fn := func( fldPath *field.Path, - obj, oldObj []*ateapipb.Limits, + obj, oldObj *int64, oldValueCorrelated bool) (errs field.ErrorList) { // don't revalidate unchanged data if oldValueCorrelated && op.Type == operation.Update { - if ateDeepEqual(obj, oldObj) { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { return nil } } // call field-attached validations earlyReturn := false - if e := validate.PtrSliceNoNils[ateapipb.Limits](ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { - errs = append(errs, e...) + if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { earlyReturn = true } - if e := validate.MaxItems(ctx, op, fldPath, obj, oldObj, 2).MarkShortCircuit(); len(e) != 0 { + if earlyReturn { + return // do not proceed + } + if e := validate.Maximum(ctx, op, fldPath, obj, oldObj, 999999); len(e) != 0 { errs = append(errs, e...) - earlyReturn = true } - if e := validate.OptionalSlice(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + if e := validate.Minimum(ctx, op, fldPath, obj, oldObj, 0); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateapipb.Resources) *int64 { + return &oldObj.CpuMillicores + }) + errs = append(errs, fn(fldPath.Child("cpu_millicores"), &obj.CpuMillicores, oldVal, oldObj != nil)...) + } + + { // field ateapipb.Resources.MemoryBytes + fn := func( + fldPath *field.Path, + obj, oldObj *int64, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { earlyReturn = true } if earlyReturn { return // do not proceed } - // lists with map semantics require unique keys - if e := validate.PtrSliceUnique(ctx, op, fldPath, obj, oldObj, - func(a *ateapipb.Limits, b *ateapipb.Limits) bool { return a.Name == b.Name }); len(e) != 0 { - errs = append(errs, e...) - } - // iterate the list and call the type's validation function - if e := validate.EachPtrSliceVal(ctx, op, fldPath, obj, oldObj, - func(a *ateapipb.Limits, b *ateapipb.Limits) bool { return a.Name == b.Name }, ateDeepEqual, Validate_Limits); len(e) != 0 { + if e := validate.Minimum(ctx, op, fldPath, obj, oldObj, 0); len(e) != 0 { errs = append(errs, e...) } return } oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.Resources) []*ateapipb.Limits { - return oldObj.Limits + func(oldObj *ateapipb.Resources) *int64 { + return &oldObj.MemoryBytes }) - errs = append(errs, fn(fldPath.Child("limits"), obj.Limits, oldVal, oldObj != nil)...) + errs = append(errs, fn(fldPath.Child("memory_bytes"), &obj.MemoryBytes, oldVal, oldObj != nil)...) } return errs @@ -10628,34 +10604,69 @@ func Validate_WorkerResources( ctx context.Context, op operation.Operation, fldPath *field.Path, obj, oldObj *ateapipb.WorkerResources) (errs field.ErrorList) { - { // field ateapipb.WorkerResources.Resources + { // field ateapipb.WorkerResources.CpuMillicores fn := func( fldPath *field.Path, - obj, oldObj *ateapipb.Resources, + obj, oldObj *int64, oldValueCorrelated bool) (errs field.ErrorList) { // don't revalidate unchanged data if oldValueCorrelated && op.Type == operation.Update { - if ateDeepEqual(obj, oldObj) { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { return nil } } // call field-attached validations earlyReturn := false - if e := validate.OptionalPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { earlyReturn = true } if earlyReturn { return // do not proceed } - // call the type's validation function - errs = append(errs, Validate_Resources(ctx, op, fldPath, obj, oldObj)...) + if e := validate.Maximum(ctx, op, fldPath, obj, oldObj, 999999); len(e) != 0 { + errs = append(errs, e...) + } + if e := validate.Minimum(ctx, op, fldPath, obj, oldObj, 0); len(e) != 0 { + errs = append(errs, e...) + } return } oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.WorkerResources) *ateapipb.Resources { - return oldObj.Resources + func(oldObj *ateapipb.WorkerResources) *int64 { + return &oldObj.CpuMillicores }) - errs = append(errs, fn(fldPath.Child("resources"), obj.Resources, oldVal, oldObj != nil)...) + errs = append(errs, fn(fldPath.Child("cpu_millicores"), &obj.CpuMillicores, oldVal, oldObj != nil)...) + } + + { // field ateapipb.WorkerResources.MemoryBytes + fn := func( + fldPath *field.Path, + obj, oldObj *int64, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.Minimum(ctx, op, fldPath, obj, oldObj, 0); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateapipb.WorkerResources) *int64 { + return &oldObj.MemoryBytes + }) + errs = append(errs, fn(fldPath.Child("memory_bytes"), &obj.MemoryBytes, oldVal, oldObj != nil)...) } { // field ateapipb.WorkerResources.Actors diff --git a/cmd/ateapi/internal/controlapi/actor.go b/cmd/ateapi/internal/controlapi/actor.go index 3bef57aeb6..35e1fef471 100644 --- a/cmd/ateapi/internal/controlapi/actor.go +++ b/cmd/ateapi/internal/controlapi/actor.go @@ -305,13 +305,6 @@ func (s *ServiceImpl) UpdateActor(ctx context.Context, actorRef resources.ActorR return apierror.FailedPrecondition( "actor must be %s to change its actor template (got: %s)", ateapipb.ActorState_ACTOR_STATE_SUSPENDED, state) } - // A paused actor resumes from its node-local snapshot, uploaded or - // not, which was captured from the old template's sandbox. The - // snapshot stays local until atelet detaches it from the node. - if latestSnapshotIsLocal(oldVal.GetStatus()) { - return apierror.FailedPrecondition( - "actor must hold no node-local snapshot to change its actor template: wait for its snapshot to be uploaded and detached from the node, or revert it") - } newTemplate, err := resolveActorTemplate(ctx, s.store, newVal) if err != nil { return err @@ -398,7 +391,9 @@ func validateTemplateVolumesUnchanged(oldTemplate, newTemplate *ateapipb.ActorTe // Deleting an actor collects everything under its external snapshot prefix. If // the location prefix ever changes, we risk leaking the snapshots under the old prefix. func (s *ServiceImpl) validateSnapshotLocationUnchanged(actor *ateapipb.Actor, newTemplate *ateapipb.ActorTemplate) error { - snap := findLatestDurableSnapshot(actor.GetStatus(), ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED) + snap := latestSnapshot(actor.GetStatus(), func(snapshot *ateapipb.Snapshot) bool { + return snapshot.GetDurableSnapshot().GetObject().GetSnapshotUri() != "" + }) currentSnapshotURI := snap.GetDurableSnapshot().GetObject().GetSnapshotUri() if currentSnapshotURI == "" { return nil diff --git a/cmd/ateapi/internal/controlapi/actor_sizing_test.go b/cmd/ateapi/internal/controlapi/actor_sizing_test.go deleted file mode 100644 index d6cec28580..0000000000 --- a/cmd/ateapi/internal/controlapi/actor_sizing_test.go +++ /dev/null @@ -1,68 +0,0 @@ -// Copyright 2026 Google LLC -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package controlapi - -import ( - "testing" - - "go.ate.dev/substrate/pkg/proto/ateapipb" -) - -// TestActorResourceLimits covers the actor-side extraction: the CPU/memory limits -// an ActorTemplate declares become the sandbox size and the scheduling floor. -func TestActorResourceLimits(t *testing.T) { - tests := []struct { - name string - resources *ateapipb.Resources - wantCPU int64 - wantMemory int64 - }{ - { - name: "nil resources yields zero", - resources: nil, - wantCPU: 0, - wantMemory: 0, - }, - { - name: "cpu and memory limits are read", - resources: &ateapipb.Resources{Limits: []*ateapipb.Limits{ - {Name: "cpu", Quantity: "2"}, - {Name: "memory", Quantity: "4Gi"}, - }}, - wantCPU: 2000, - wantMemory: 4 << 30, - }, - { - name: "millicpu is preserved", - resources: &ateapipb.Resources{Limits: []*ateapipb.Limits{ - {Name: "cpu", Quantity: "1500m"}, - }}, - wantCPU: 1500, - wantMemory: 0, - }, - } - for _, tc := range tests { - t.Run(tc.name, func(t *testing.T) { - tmpl := &ateapipb.ActorTemplate{Resources: tc.resources} - cpu, mem, err := actorResourceLimits(tmpl) - if err != nil { - t.Fatalf("actorResourceLimits() error: %v", err) - } - if cpu != tc.wantCPU || mem != tc.wantMemory { - t.Fatalf("actorResourceLimits() = (%d, %d), want (%d, %d)", cpu, mem, tc.wantCPU, tc.wantMemory) - } - }) - } -} diff --git a/cmd/ateapi/internal/controlapi/actor_template_test.go b/cmd/ateapi/internal/controlapi/actor_template_test.go index 857915f7c4..bef9a961c0 100644 --- a/cmd/ateapi/internal/controlapi/actor_template_test.go +++ b/cmd/ateapi/internal/controlapi/actor_template_test.go @@ -45,7 +45,7 @@ func validActorTemplate(mutations ...func(*ateapipb.ActorTemplate)) *ateapipb.Ac StorageLocation: "gs://my-bucket/snapshots", PreferredFidelity: ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, }, - Resources: resources.CPUMemory(1000, 512<<20), + Resources: &ateapipb.ResourceRequirements{Limits: resources.CPUMemory(1000, 512<<20)}, } for _, m := range mutations { m(template) @@ -165,7 +165,7 @@ func TestCreateActorTemplateIgnoresServerOwnedFields(t *testing.T) { tmpl.Metadata.Version = 42 tmpl.WorkerSelector = &ateapipb.Selector{MatchLabels: map[string]string{"pool": "default"}} tmpl.Containers = []*ateapipb.Container{{Name: "main", Image: "example.com/app:v1@sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"}} - tmpl.Resources = &ateapipb.Resources{Limits: []*ateapipb.Limits{{Name: "cpu", Quantity: "1"}, {Name: "memory", Quantity: "1Gi"}}} + tmpl.Resources = &ateapipb.ResourceRequirements{Limits: resources.CPUMemory(1000, 1<<30)} // Server-owned status a client must not be able to set. tmpl.Status = &ateapipb.ActorTemplateStatus{ GoldenSnapshotStatus: &ateapipb.GoldenSnapshotStatus{ diff --git a/cmd/ateapi/internal/controlapi/actor_test.go b/cmd/ateapi/internal/controlapi/actor_test.go index d0cca17b95..3faf3001bf 100644 --- a/cmd/ateapi/internal/controlapi/actor_test.go +++ b/cmd/ateapi/internal/controlapi/actor_test.go @@ -274,7 +274,7 @@ func TestUpdateActor_RepointTemplate(t *testing.T) { }}, Volumes: tmpl.volumes, SnapshotConfig: &ateapipb.SnapshotConfig{StorageLocation: "gs://my-bucket/snapshots"}, - Resources: resources.CPUMemory(1000, 512<<20), + Resources: &ateapipb.ResourceRequirements{Limits: resources.CPUMemory(1000, 512<<20)}, }); err != nil { t.Fatalf("creating template %s: %v", name, err) } @@ -361,9 +361,8 @@ func TestUpdateActor_RepointTemplate(t *testing.T) { t.Fatalf("UpdateActor repointing a running actor = %v, want FailedPrecondition (err: %v)", got, err) } - // Repointing a paused actor is rejected, even once atelet has uploaded its - // snapshot: it resumes from the node-local copy, captured under the old - // template. + // Repointing a paused actor is allowed while suspended, even when it holds + // a node-local snapshot. for _, name := range []string{"paused-actor", "uploaded-paused-actor"} { paused := storetest.MustCreateActor(t, ctx, persistence, &ateapipb.Actor{ Metadata: &ateapipb.ResourceMetadata{Atespace: testAtespace, Name: name}, @@ -383,12 +382,15 @@ func TestUpdateActor_RepointTemplate(t *testing.T) { s.LastAssignedGeneration = 1 s.Snapshots = []*ateapipb.Snapshot{snap} }) - _, err = svc.UpdateActor(ctx, &ateapipb.UpdateActorRequest{Actor: &ateapipb.Actor{ + repointedPaused, err := svc.UpdateActor(ctx, &ateapipb.UpdateActorRequest{Actor: &ateapipb.Actor{ Metadata: paused.GetMetadata(), ActorTemplate: &ateapipb.ObjectRef{Atespace: testAtespace, Name: "tmpl-b"}, }}) - if got := apierror.Code(err); got != codes.FailedPrecondition { - t.Fatalf("UpdateActor repointing %s = %v, want FailedPrecondition (err: %v)", name, got, err) + if err != nil { + t.Fatalf("UpdateActor repointing %s failed: %v", name, err) + } + if got, want := repointedPaused.GetActorTemplate().GetName(), "tmpl-b"; got != want { + t.Errorf("%s actor_template.name = %q, want %q", name, got, want) } } @@ -463,10 +465,16 @@ func TestUpdateActor_RepointTemplateStorageLocation(t *testing.T) { } tests := []struct { - name string - template string - snapshot func(*testing.T, *ateapipb.Actor) string - repointTo string + name string + template string + snapshot func(*testing.T, *ateapipb.Actor) string + // pending leaves the snapshot's upload PENDING, as a pause does. + pending bool + // lastAssignedGen overrides status.LastAssignedGeneration when > 0 + // (e.g. after a reverted pause or crashed suspend pruned a higher + // pending generation). + lastAssignedGen int32 + repointTo string // gateOff disables the PerTemplateSnapshotLocation preview gate, so // every template resolves to clusterSnapshotLocation. gateOff bool @@ -500,6 +508,30 @@ func TestUpdateActor_RepointTemplateStorageLocation(t *testing.T) { repointTo: "different-location", wantCode: codes.FailedPrecondition, }, + { + name: "pending owned upload moves to the same location", + template: "same-location", + snapshot: snapshotOwnedByActor, + pending: true, + repointTo: "same-location-slash", + wantCode: codes.OK, + }, + { + name: "pending owned upload cannot move to another location", + template: "same-location", + snapshot: snapshotOwnedByActor, + pending: true, + repointTo: "different-location", + wantCode: codes.FailedPrecondition, + }, + { + name: "owned snapshot after reverted later generation cannot move to another location", + template: "same-location", + snapshot: snapshotOwnedByActor, + lastAssignedGen: 2, + repointTo: "different-location", + wantCode: codes.FailedPrecondition, + }, { name: "snapshot location repoint is checked even when old template is gone", template: "gone", @@ -563,10 +595,17 @@ func TestUpdateActor_RepointTemplateStorageLocation(t *testing.T) { }) if tt.snapshot != nil { uri := tt.snapshot(t, actor) + storageStatus := ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED + if tt.pending { + storageStatus = ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_PENDING + } actor = mustUpdateActorStatus(t, ctx, persistence, actor, func(s *ateapipb.ActorStatus) { s.LastAssignedGeneration = 1 + if tt.lastAssignedGen > 0 { + s.LastAssignedGeneration = tt.lastAssignedGen + } s.Snapshots = []*ateapipb.Snapshot{ - newDurableSnapshot(1, ateapipb.SnapshotOwner_SNAPSHOT_OWNER_ACTOR, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", "snapshot", uri, ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED), + newDurableSnapshot(1, ateapipb.SnapshotOwner_SNAPSHOT_OWNER_ACTOR, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", "snapshot", uri, storageStatus), } }) } diff --git a/cmd/ateapi/internal/controlapi/functionaltest/actor_template_test.go b/cmd/ateapi/internal/controlapi/functionaltest/actor_template_test.go index d86759c0b0..047996805c 100644 --- a/cmd/ateapi/internal/controlapi/functionaltest/actor_template_test.go +++ b/cmd/ateapi/internal/controlapi/functionaltest/actor_template_test.go @@ -44,7 +44,7 @@ func TestActorTemplateCRUD(t *testing.T) { Metadata: &ateapipb.ResourceMetadata{Atespace: testAtespace, Name: "tmpl-a"}, Containers: []*ateapipb.Container{{Name: "main", Image: "example.com/app:v1@sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"}}, SnapshotConfig: &ateapipb.SnapshotConfig{StorageLocation: "gs://my-bucket/snapshots"}, - Resources: &ateapipb.Resources{Limits: []*ateapipb.Limits{{Name: "cpu", Quantity: "1"}, {Name: "memory", Quantity: "1Gi"}}}, + Resources: &ateapipb.ResourceRequirements{Limits: resources.CPUMemory(1000, 1<<30)}, // Server-owned status on the request is ignored. Status: &ateapipb.ActorTemplateStatus{ GoldenSnapshotStatus: &ateapipb.GoldenSnapshotStatus{ @@ -64,7 +64,7 @@ func TestActorTemplateCRUD(t *testing.T) { PreferredFidelity: ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, GoldenSnapshotConfig: &ateapipb.GoldenSnapshotConfig{Mode: ateapipb.GoldenSnapshotMode_GOLDEN_SNAPSHOT_MODE_ENABLED}, }, - Resources: &ateapipb.Resources{Limits: []*ateapipb.Limits{{Name: "cpu", Quantity: "1"}, {Name: "memory", Quantity: "1Gi"}}}, + Resources: &ateapipb.ResourceRequirements{Limits: resources.CPUMemory(1000, 1<<30)}, Status: &ateapipb.ActorTemplateStatus{}, } if diff := cmp.Diff(want, created, protocmp.Transform(), ignoreUID, ignoreTimestamps); diff != "" { @@ -76,7 +76,7 @@ func TestActorTemplateCRUD(t *testing.T) { Metadata: &ateapipb.ResourceMetadata{Atespace: testAtespace, Name: "tmpl-a"}, Containers: []*ateapipb.Container{{Name: "main", Image: "example.com/app:v1@sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"}}, SnapshotConfig: &ateapipb.SnapshotConfig{StorageLocation: "gs://my-bucket/snapshots"}, - Resources: resources.CPUMemory(1000, 512<<20), + Resources: &ateapipb.ResourceRequirements{Limits: resources.CPUMemory(1000, 512<<20)}, }, }) assertGrpcError(t, err, codes.AlreadyExists, "ActorTemplate "+testAtespace+"/tmpl-a already exists") @@ -237,7 +237,7 @@ func TestActorLifecycleWithoutGoldenSnapshot(t *testing.T) { }, Containers: []*ateapipb.Container{{Name: "main", Image: "main@sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", Command: []string{"/main"}}}, WorkerSelector: &ateapipb.Selector{MatchLabels: map[string]string{poolLabelKey: ns}}, - Resources: resources.CPUMemory(1000, 512<<20), + Resources: &ateapipb.ResourceRequirements{Limits: resources.CPUMemory(1000, 512<<20)}, }}) if err != nil { t.Fatal(err) @@ -407,9 +407,9 @@ func TestCreateActorTemplate_ExternalVolumesPreview(t *testing.T) { }}, Volumes: []*ateapipb.Volume{{ Name: "data", - External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{StorageClassName: "standard", Capacity: "1Gi"}}, + External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{StorageClassName: "standard", CapacityBytes: 1 << 30}}, }}, - Resources: &ateapipb.Resources{Limits: []*ateapipb.Limits{{Name: "cpu", Quantity: "1"}, {Name: "memory", Quantity: "1Gi"}}}, + Resources: &ateapipb.ResourceRequirements{Limits: resources.CPUMemory(1000, 1<<30)}, SnapshotConfig: &ateapipb.SnapshotConfig{StorageLocation: "gs://my-bucket/snapshots"}, }, }) @@ -458,7 +458,7 @@ func TestCreateActorTemplate_PerTemplateSnapshotLocationPreview(t *testing.T) { Name: "main", Image: "example.com/app:v1@sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", }}, - Resources: &ateapipb.Resources{Limits: []*ateapipb.Limits{{Name: "cpu", Quantity: "1"}, {Name: "memory", Quantity: "1Gi"}}}, + Resources: &ateapipb.ResourceRequirements{Limits: resources.CPUMemory(1000, 1<<30)}, SnapshotConfig: &ateapipb.SnapshotConfig{StorageLocation: tt.location}, }, }) diff --git a/cmd/ateapi/internal/controlapi/functionaltest/actor_test.go b/cmd/ateapi/internal/controlapi/functionaltest/actor_test.go index bae2613d37..da875809a9 100644 --- a/cmd/ateapi/internal/controlapi/functionaltest/actor_test.go +++ b/cmd/ateapi/internal/controlapi/functionaltest/actor_test.go @@ -123,7 +123,7 @@ func TestCreateActor_WithExternalVolumes(t *testing.T) { Name: "ext-vol-1", External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{ StorageClassName: "standard", - Capacity: "10Gi", + CapacityBytes: 10 << 30, }}, }, } @@ -184,7 +184,7 @@ func TestCreateActor_ExternalVolumesWithoutOptIn(t *testing.T) { createTemplateWithVolumes(t, tc, ns, []*ateapipb.Volume{{ Name: "ext-vol-1", - External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{StorageClassName: "standard", Capacity: "10Gi"}}, + External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{StorageClassName: "standard", CapacityBytes: 10 << 30}}, }}, []*ateapipb.VolumeMount{{Name: "ext-vol-1", MountPath: "/data"}}) @@ -216,7 +216,7 @@ func TestActorLifecycle_ExternalVolumesWithoutOptIn(t *testing.T) { createTemplateWithVolumes(t, tc, ns, []*ateapipb.Volume{{ Name: "data-vol", - External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{StorageClassName: "fast", Capacity: "20Gi"}}, + External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{StorageClassName: "fast", CapacityBytes: 20 << 30}}, }}, []*ateapipb.VolumeMount{{Name: "data-vol", MountPath: "/mnt/data"}}) createWorkerPod(t, tc, ns, "worker-1", "node1", "pool1") @@ -278,7 +278,7 @@ func TestCreateActor_SubstrateTemplateRef(t *testing.T) { Metadata: &ateapipb.ResourceMetadata{Atespace: testAtespace, Name: "sub-tmpl"}, Containers: []*ateapipb.Container{{Name: "main", Image: "example.com/app:v1@sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"}}, SnapshotConfig: &ateapipb.SnapshotConfig{StorageLocation: "gs://my-bucket/snapshots"}, - Resources: resources.CPUMemory(1000, 512<<20), + Resources: &ateapipb.ResourceRequirements{Limits: resources.CPUMemory(1000, 512<<20)}, }, }); err != nil { t.Fatalf("CreateActorTemplate failed: %v", err) @@ -407,10 +407,10 @@ func TestCreateActor_RejectsSnapshotWithExternalVolumes(t *testing.T) { Volumes: []*ateapipb.Volume{{ Name: "data", External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{ - Capacity: "1Gi", StorageClassName: "standard", + CapacityBytes: 1 << 30, StorageClassName: "standard", }}, }}, - Resources: resources.CPUMemory(1000, 512<<20), + Resources: &ateapipb.ResourceRequirements{Limits: resources.CPUMemory(1000, 512<<20)}, }, }) if err != nil { @@ -840,7 +840,7 @@ func TestUpdateActor_RepointTemplate(t *testing.T) { }}, Volumes: tmpl.volumes, SnapshotConfig: &ateapipb.SnapshotConfig{StorageLocation: "gs://my-bucket/snapshots"}, - Resources: resources.CPUMemory(1000, 512<<20), + Resources: &ateapipb.ResourceRequirements{Limits: resources.CPUMemory(1000, 512<<20)}, }, }); err != nil { t.Fatalf("CreateActorTemplate %s failed: %v", name, err) @@ -1246,7 +1246,7 @@ func TestDeleteActor_Running(t *testing.T) { Name: "vol1", External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{ StorageClassName: "standard", - Capacity: "10Gi", + CapacityBytes: 10 << 30, }}, }, }, []*ateapipb.VolumeMount{ @@ -1734,7 +1734,7 @@ func TestActorLifecycle_WithExternalVolumes(t *testing.T) { Name: "data-vol", External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{ StorageClassName: "fast", - Capacity: "20Gi", + CapacityBytes: 20 << 30, }}, }, } @@ -1884,14 +1884,14 @@ func TestResumeActor_VolumeCreationFailure(t *testing.T) { Name: "succ-vol1", External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{ StorageClassName: "standard", - Capacity: "10Gi", + CapacityBytes: 10 << 30, }}, }, { Name: "fail-vol2", External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{ StorageClassName: "standard", - Capacity: "10Gi", + CapacityBytes: 10 << 30, }}, }, } @@ -2028,14 +2028,14 @@ func TestResumeActor_VolumeCreationRetrySuccess(t *testing.T) { Name: "succ-vol1", External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{ StorageClassName: "standard", - Capacity: "10Gi", + CapacityBytes: 10 << 30, }}, }, { Name: "retry-vol2", External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{ StorageClassName: "standard", - Capacity: "10Gi", + CapacityBytes: 10 << 30, }}, }, } @@ -2189,7 +2189,7 @@ func TestResumeActor_VolumeAttachFailureAndRetry(t *testing.T) { Name: "vol1", External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{ StorageClassName: "standard", - Capacity: "10Gi", + CapacityBytes: 10 << 30, }}, }, } @@ -2292,7 +2292,7 @@ func TestResumeActor_VolumeAttachFailure_DeleteActor(t *testing.T) { Name: "vol1", External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{ StorageClassName: "standard", - Capacity: "10Gi", + CapacityBytes: 10 << 30, }}, }, } @@ -2442,14 +2442,14 @@ func TestResumeActor_MultiVolumePartialAttachFailure_Retry(t *testing.T) { Name: "vol1", External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{ StorageClassName: "standard", - Capacity: "10Gi", + CapacityBytes: 10 << 30, }}, }, { Name: "vol2", External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{ StorageClassName: "standard", - Capacity: "10Gi", + CapacityBytes: 10 << 30, }}, }, } @@ -2614,7 +2614,7 @@ func TestSuspendActor_VolumeDetachFailure_RetrySuccess(t *testing.T) { Name: "vol1", External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{ StorageClassName: "standard", - Capacity: "10Gi", + CapacityBytes: 10 << 30, }}, }, } @@ -2724,7 +2724,7 @@ func TestSuspendActor_VolumeDetachFailure_DeleteActor(t *testing.T) { Name: "vol1", External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{ StorageClassName: "standard", - Capacity: "10Gi", + CapacityBytes: 10 << 30, }}, }, } @@ -2815,7 +2815,7 @@ func TestSuspendActorAsync_VolumeLifecycle_DetachResumeAndMigrate(t *testing.T) Name: "vol1", External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{ StorageClassName: "standard", - Capacity: "10Gi", + CapacityBytes: 10 << 30, }}, }, } @@ -3005,7 +3005,7 @@ func TestSuspendActorAsync_VolumeDetachFailure_RetrySuccess(t *testing.T) { Name: "vol1", External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{ StorageClassName: "standard", - Capacity: "10Gi", + CapacityBytes: 10 << 30, }}, }, } @@ -3187,10 +3187,10 @@ func TestResumeActor(t *testing.T) { Status: &ateapipb.WorkerStatus{ State: ateapipb.WorkerState_WORKER_STATE_ACTIVE, // Only the ceiling the worker reported. - Capacity: &ateapipb.WorkerResources{Actors: 1, Resources: resources.CPUMemory(1000, 512<<20)}, + Capacity: &ateapipb.WorkerResources{Actors: 1, CpuMillicores: 1000, MemoryBytes: 512 << 20}, // All a listing reports of the assignments: one actor and the // compute its template declares. - Allocated: &ateapipb.WorkerResources{Actors: 1, Resources: resources.CPUMemory(1000, 512<<20)}, + Allocated: &ateapipb.WorkerResources{Actors: 1, CpuMillicores: 1000, MemoryBytes: 512 << 20}, DefaultRuntime: defaultWorkerRuntime("gvisor"), }, } @@ -3279,7 +3279,7 @@ func createDataFidelityTemplate(t *testing.T, tc *testContext, ns string) *ateap WorkerSelector: &ateapipb.Selector{ MatchLabels: map[string]string{poolLabelKey: ns}, }, - Resources: resources.CPUMemory(1000, 512<<20), + Resources: &ateapipb.ResourceRequirements{Limits: resources.CPUMemory(1000, 512<<20)}, }, }) if err != nil { @@ -5337,14 +5337,13 @@ func TestResumeActor_RelocatesAfterDetach(t *testing.T) { }); err != nil { t.Fatalf("ResumeActor(%s) failed: %v", relocated, err) } - createWorkerPod(t, tc, ns, "worker-2", "node2", "pool1") - setupAteletOnNode(t, tc, "atelet-node2", "node2") - - // Capacity exhaustion is ResourceExhausted. + // With no room on any node, the resume fails with ResourceExhausted. _, err = tc.client.ResumeActor(context.Background(), &ateapipb.ResumeActorRequest{ Actor: &ateapipb.ObjectRef{Atespace: testAtespace, Name: pinned}, }) assertGrpcError(t, err, codes.ResourceExhausted, "no worker has room for the actor") + createWorkerPod(t, tc, ns, "worker-2", "node2", "pool1") + setupAteletOnNode(t, tc, "atelet-node2", "node2") // node1's atelet reports the paused snapshot uploaded and detached: a report // with only latest_durable. @@ -5391,6 +5390,79 @@ func TestResumeActor_RelocatesAfterDetach(t *testing.T) { } } +// TestResumeActor_DetachesFromFullNode resumes a paused actor whose node is +// full while another has room: it is detached and restored on the other node. +func TestResumeActor_DetachesFromFullNode(t *testing.T) { + ns := namespaceForTest("ns-resume-detach") + tc := setupTest(t, ns) + defer tc.cleanup() + + createTemplate(t, tc, ns) + workerName := createWorkerPod(t, tc, ns, "worker-1", "node1", "pool1") + + const pinned, squatter = "actor-pinned", "actor-squatter" + for _, name := range []string{pinned, squatter} { + if _, err := tc.client.CreateActor(context.Background(), &ateapipb.CreateActorRequest{Actor: &ateapipb.Actor{ + Metadata: &ateapipb.ResourceMetadata{Atespace: testAtespace, Name: name}, + ActorTemplate: &ateapipb.ObjectRef{Atespace: testAtespace, Name: "tmpl1"}, + }}); err != nil { + t.Fatalf("CreateActor(%s) failed: %v", name, err) + } + } + if _, err := tc.client.ResumeActor(context.Background(), &ateapipb.ResumeActorRequest{ + Actor: &ateapipb.ObjectRef{Atespace: testAtespace, Name: pinned}, + }); err != nil { + t.Fatalf("ResumeActor(%s) failed: %v", pinned, err) + } + if _, err := tc.client.SuspendActor(context.Background(), &ateapipb.SuspendActorRequest{ + Actor: &ateapipb.ObjectRef{Atespace: testAtespace, Name: pinned}, + Options: &ateapipb.SuspendOptions{Mode: ateapipb.DurabilityMode_DURABILITY_MODE_ASYNCHRONOUS}, + }); err != nil { + t.Fatalf("SuspendActor(%s) failed: %v", pinned, err) + } + paused, err := tc.client.GetActor(context.Background(), &ateapipb.GetActorRequest{ + Actor: &ateapipb.ObjectRef{Atespace: testAtespace, Name: pinned}, + }) + if err != nil { + t.Fatalf("GetActor(%s) failed: %v", pinned, err) + } + ls := localSnapshot(paused.GetStatus()) + if ls == nil || paused.GetStatus().GetAssignedNode() != "node1" { + t.Fatalf("paused actor = %v, want a local snapshot on node1", paused.GetStatus()) + } + waitForWorkerAvailable(t, tc, workerName) + if _, err := tc.client.ResumeActor(context.Background(), &ateapipb.ResumeActorRequest{ + Actor: &ateapipb.ObjectRef{Atespace: testAtespace, Name: squatter}, + }); err != nil { + t.Fatalf("ResumeActor(%s) failed: %v", squatter, err) + } + createWorkerPod(t, tc, ns, "worker-2", "node2", "pool1") + setupAteletOnNode(t, tc, "atelet-node2", "node2") + + resumed, err := tc.client.ResumeActor(context.Background(), &ateapipb.ResumeActorRequest{ + Actor: &ateapipb.ObjectRef{Atespace: testAtespace, Name: pinned}, + }) + if err != nil { + t.Fatalf("ResumeActor(%s) on a full node failed: %v", pinned, err) + } + if got := resumed.GetActor().GetStatus().GetWorkerAssignment().GetWorkerPod(); got != "worker-2" { + t.Errorf("resumed onto worker %q, want worker-2 (the worker on node2)", got) + } + tc.fakeAtelet.Lock.Lock() + detach := proto.CloneOf(tc.fakeAtelet.DetachRequest) + tc.fakeAtelet.Lock.Unlock() + if got := detach.GetActorUid(); got != paused.GetMetadata().GetUid() { + t.Errorf("DetachActor actor_uid = %q, want %q", got, paused.GetMetadata().GetUid()) + } + restore := tc.fakeAtelet.lastRestoreRequest() + if got := restore.GetSnapshot().GetSnapshotUuid(); got != ls.GetUuid() { + t.Errorf("Restore snapshot = %q, want the detached snapshot %q", got, ls.GetUuid()) + } + if got := restore.GetSnapshot().GetObject().GetSnapshotUri(); got != ls.GetDurableSnapshot().GetObject().GetSnapshotUri() { + t.Errorf("Restore snapshot URI = %q, want %q", got, ls.GetDurableSnapshot().GetObject().GetSnapshotUri()) + } +} + // TestLifecycleOpPoolAttributesOnSuccess is the regression test for #957: a // successful suspend and pause must stamp the pool they ran on. Both recorded // the histogram from a defer that read the finalized record, whose assignment diff --git a/cmd/ateapi/internal/controlapi/functionaltest/common_test.go b/cmd/ateapi/internal/controlapi/functionaltest/common_test.go index b48290c3b0..7aef251900 100644 --- a/cmd/ateapi/internal/controlapi/functionaltest/common_test.go +++ b/cmd/ateapi/internal/controlapi/functionaltest/common_test.go @@ -479,7 +479,7 @@ func createTemplateWithContainersAndVolumes(t *testing.T, tc *testContext, ns st WorkerSelector: &ateapipb.Selector{ MatchLabels: map[string]string{poolLabelKey: ns}, }, - Resources: resources.CPUMemory(1000, 512<<20), + Resources: &ateapipb.ResourceRequirements{Limits: resources.CPUMemory(1000, 512<<20)}, }, }) if err != nil { @@ -628,7 +628,7 @@ func createTemplateWithSelector(t *testing.T, tc *testContext, name string, sele {Name: "main", Image: "main@sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", Command: []string{"/main"}}, }, WorkerSelector: selector, - Resources: resources.CPUMemory(1000, 512<<20), + Resources: &ateapipb.ResourceRequirements{Limits: resources.CPUMemory(1000, 512<<20)}, }, }) if err != nil { diff --git a/cmd/ateapi/internal/controlapi/functionaltest/main_test.go b/cmd/ateapi/internal/controlapi/functionaltest/main_test.go index 7fb90c6a46..939b736507 100644 --- a/cmd/ateapi/internal/controlapi/functionaltest/main_test.go +++ b/cmd/ateapi/internal/controlapi/functionaltest/main_test.go @@ -125,6 +125,8 @@ type FakeAteletServer struct { CheckpointCalled bool CheckpointRequest *ateletpb.CheckpointRequest FailCheckpoint error + // checkpointed is each actor's last checkpointed snapshot, by UID. + checkpointed map[string]*ateletpb.Snapshot RestoreCalled bool RestoreRequest *ateletpb.RestoreRequest @@ -135,6 +137,10 @@ type FakeAteletServer struct { PromoteRequest *ateletpb.PromoteSnapshotRequest FailPromote error + DetachCalled bool + DetachRequest *ateletpb.DetachActorRequest + FailDetach error + TerminateCalled bool TerminateRequest *ateletpb.TerminateRequest FailTerminate error @@ -177,6 +183,7 @@ func (f *FakeAteletServer) Reset() { f.CheckpointCalled = false f.CheckpointRequest = nil f.FailCheckpoint = nil + f.checkpointed = nil f.RestoreCalled = false f.RestoreRequest = nil @@ -187,6 +194,10 @@ func (f *FakeAteletServer) Reset() { f.PromoteRequest = nil f.FailPromote = nil + f.DetachCalled = false + f.DetachRequest = nil + f.FailDetach = nil + f.TerminateCalled = false f.TerminateRequest = nil f.FailTerminate = nil @@ -214,6 +225,29 @@ func (f *FakeAteletServer) PromoteSnapshot(ctx context.Context, req *ateletpb.Pr }}, nil } +// DetachActor uploads the actor's last checkpointed snapshot and reports it +// durable, with nothing left on the node. +func (f *FakeAteletServer) DetachActor(ctx context.Context, req *ateletpb.DetachActorRequest) (*ateletpb.DetachActorResponse, error) { + f.Lock.Lock() + defer f.Lock.Unlock() + + f.DetachCalled = true + f.DetachRequest = proto.CloneOf(req) + if f.FailDetach != nil { + return nil, f.FailDetach + } + snap := f.checkpointed[req.GetActorUid()] + if snap == nil { + return &ateletpb.DetachActorResponse{State: &ateletpb.ActorSnapshotState{}}, nil + } + if err := f.writeSnapshot(snap.GetObject().GetSnapshotUri()); err != nil { + return nil, err + } + return &ateletpb.DetachActorResponse{State: &ateletpb.ActorSnapshotState{ + LatestDurable: &ateletpb.SnapshotReport{Snapshot: proto.CloneOf(snap)}, + }}, nil +} + func (f *FakeAteletServer) Run(ctx context.Context, req *ateletpb.RunRequest) (*ateletpb.RunResponse, error) { f.Lock.Lock() defer f.Lock.Unlock() @@ -236,6 +270,10 @@ func (f *FakeAteletServer) Checkpoint(ctx context.Context, req *ateletpb.Checkpo if f.FailCheckpoint != nil { return nil, f.FailCheckpoint } + if f.checkpointed == nil { + f.checkpointed = map[string]*ateletpb.Snapshot{} + } + f.checkpointed[req.GetActorUid()] = proto.CloneOf(req.GetSnapshot()) // Only an immediate upload is in object storage when Checkpoint returns; // atelet does not upload deferred snapshots yet. diff --git a/cmd/ateapi/internal/controlapi/functionaltest/worker_contention_test.go b/cmd/ateapi/internal/controlapi/functionaltest/worker_contention_test.go index 29abc66a13..9ac8e83c88 100644 --- a/cmd/ateapi/internal/controlapi/functionaltest/worker_contention_test.go +++ b/cmd/ateapi/internal/controlapi/functionaltest/worker_contention_test.go @@ -25,7 +25,6 @@ import ( "go.ate.dev/substrate/cmd/ateapi/internal/store" "go.ate.dev/substrate/internal/hardware" - "go.ate.dev/substrate/internal/resources" "go.ate.dev/substrate/pkg/proto/ateapipb" ) @@ -185,8 +184,8 @@ func reportWorkerCapacity(t *testing.T, tc *testContext, name string, actors int } if _, err := tc.persistence.UpdateWorker(ctx, name, store.PreconditionFrom(worker), func(toUpdate *ateapipb.Worker) error { toUpdate.Status.Capacity = &ateapipb.WorkerResources{ - Actors: actors, - Resources: resources.CPUMemory(int64(actors)*1000, int64(actors)*512<<20), + Actors: actors, + CpuMillicores: int64(actors) * 1000, MemoryBytes: int64(actors) * 512 << 20, } if toUpdate.Status.DefaultRuntime == nil { toUpdate.Status.DefaultRuntime = defaultWorkerRuntime(toUpdate.GetSandboxClass()) diff --git a/cmd/ateapi/internal/controlapi/functionaltest/worker_test.go b/cmd/ateapi/internal/controlapi/functionaltest/worker_test.go index d5fbebaf0d..87573a4ea7 100644 --- a/cmd/ateapi/internal/controlapi/functionaltest/worker_test.go +++ b/cmd/ateapi/internal/controlapi/functionaltest/worker_test.go @@ -21,7 +21,6 @@ import ( "time" "github.com/google/go-cmp/cmp" - "go.ate.dev/substrate/internal/resources" "go.ate.dev/substrate/pkg/proto/ateapipb" "google.golang.org/grpc/codes" "google.golang.org/protobuf/testing/protocmp" @@ -169,7 +168,7 @@ func TestListWorkers(t *testing.T) { Labels: map[string]string{"foo": "bar"}, Status: &ateapipb.WorkerStatus{ State: ateapipb.WorkerState_WORKER_STATE_ACTIVE, - Capacity: &ateapipb.WorkerResources{Actors: 1, Resources: resources.CPUMemory(1000, 512<<20)}, + Capacity: &ateapipb.WorkerResources{Actors: 1, CpuMillicores: 1000, MemoryBytes: 512 << 20}, DefaultRuntime: defaultWorkerRuntime("gvisor"), }, }, diff --git a/cmd/ateapi/internal/controlapi/volumes.go b/cmd/ateapi/internal/controlapi/volumes.go index dba16763d5..218958cfc9 100644 --- a/cmd/ateapi/internal/controlapi/volumes.go +++ b/cmd/ateapi/internal/controlapi/volumes.go @@ -121,9 +121,9 @@ func createActorVolumes(ctx context.Context, registry VolumePluginRegistry, scLi } resp, volErr := plugin.CreateVolume(ctx, volume.CreateVolumeRequest{ - Name: actVolID, - Capacity: specVol.GetExternal().GetTemplate().GetCapacity(), - Parameters: sc.Parameters, + Name: actVolID, + CapacityBytes: specVol.GetExternal().GetTemplate().GetCapacityBytes(), + Parameters: sc.Parameters, }) if volErr != nil { return resultVolumes, apierror.Internal("failed to create volume %q: %v", specVol.GetName(), volErr) diff --git a/cmd/ateapi/internal/controlapi/workflow_resume.go b/cmd/ateapi/internal/controlapi/workflow_resume.go index bd15d0b7ef..1a96d5c32c 100644 --- a/cmd/ateapi/internal/controlapi/workflow_resume.go +++ b/cmd/ateapi/internal/controlapi/workflow_resume.go @@ -33,7 +33,6 @@ import ( "go.ate.dev/substrate/internal/volume" "go.ate.dev/substrate/pkg/proto/ateapipb" "google.golang.org/protobuf/proto" - "k8s.io/apimachinery/pkg/api/resource" "k8s.io/apimachinery/pkg/labels" "k8s.io/apimachinery/pkg/util/wait" ) @@ -122,7 +121,20 @@ func (w *ActorWorkflow) ResumeActor(ctx context.Context, actorRef resources.Acto actor = created var worker *ateapipb.Worker var assigned *ateapipb.Actor - if assigned, worker, err = w.ensureWorkerAssigned(leaseCtx, actorRef, actor, actorTemplate); err != nil { + assigned, worker, err = w.ensureWorkerAssigned(leaseCtx, actorRef, actor, actorTemplate) + if err != nil && shouldDetachFromNode(actor, err) { + var detached *ateapipb.Actor + if detached, err = w.detachFromFullNode(leaseCtx, actorRef, actor, actorTemplate, err); err != nil { + return nil, false, err + } + actor = detached + // Resolve the restore source again, from the uploaded snapshot. + if src, err = resolveResumeSource(actorRef, actor, actorTemplate); err != nil { + return nil, false, err + } + assigned, worker, err = w.ensureWorkerAssigned(leaseCtx, actorRef, actor, actorTemplate) + } + if err != nil { return nil, false, err } actor = assigned @@ -167,27 +179,42 @@ func (w *ActorWorkflow) loadActorForResume(ctx context.Context, actorRef resourc if err != nil { return nil, nil, src, err } + if src, err = resolveResumeSource(actorRef, actor, actorTemplate); err != nil { + return nil, nil, src, err + } + return actor, actorTemplate, src, nil +} + +// resolveResumeSource resolves the boot source for the actor's pending restore. +func resolveResumeSource(actorRef resources.ActorRef, actor *ateapipb.Actor, actorTemplate *ateapipb.ActorTemplate) (resumeSnapshotSource, error) { + var src resumeSnapshotSource + var err error src.RequiredRuntime = requiredSnapshotRuntime(actor, actorTemplate) if snap := findLatestDurableSnapshot(actor.GetStatus(), ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED); snap != nil { st := snap.GetDurableSnapshot() if src.SnapshotURI, err = resources.ParseSnapshotURI(st.GetObject().GetSnapshotUri()); err != nil { - return nil, nil, src, apierror.DataLoss("Actor %s external snapshot: %v", actorRef, err) + return src, apierror.DataLoss("Actor %s external snapshot: %v", actorRef, err) } src.SnapshotUUID = snap.GetUuid() src.Generation = snap.GetGeneration() src.Fidelity = snap.GetFidelity() - capturedUnder := snap.GetActorTemplateUid() - src.TemplateReplaced = capturedUnder != "" && capturedUnder != actorTemplate.GetMetadata().GetUid() + src.TemplateReplaced = templateReplaced(snap, actorTemplate) } // A local restore also names the local snapshot's upload, so a URI that // does not parse fails here too, before a worker is claimed. if local := findLatestLocalSnapshot(actor.GetStatus()); local != nil { if _, err = resources.ParseSnapshotURI(local.GetDurableSnapshot().GetObject().GetSnapshotUri()); err != nil { - return nil, nil, src, apierror.DataLoss("Actor %s local snapshot: %v", actorRef, err) + return src, apierror.DataLoss("Actor %s local snapshot: %v", actorRef, err) } } + return src, nil +} - return actor, actorTemplate, src, nil +// templateReplaced reports whether snap was captured under a different +// ActorTemplate than tmpl. An unstamped snapshot is treated as matching. +func templateReplaced(snap *ateapipb.Snapshot, tmpl *ateapipb.ActorTemplate) bool { + capturedUnder := snap.GetActorTemplateUid() + return capturedUnder != "" && capturedUnder != tmpl.GetMetadata().GetUid() } // ensureVolumesCreated provisions any actor volumes that have no storage @@ -397,10 +424,7 @@ func (w *ActorWorkflow) validateAssignedWorker(ctx context.Context, actorRef res } return nil, apierror.Aborted("actor %s crashed", actorRef) } - constraints, err := schedulingConstraints(actor, actorTemplate) - if err != nil { - return nil, err - } + constraints := schedulingConstraints(actor, actorTemplate) if !w.appliesWithFallback(worker, constraints, canRelaxRuntime(actor, constraints)) { slog.ErrorContext(ctx, "crashing actor because previously assigned worker is not eligible anymore") // If that worker's pool is no longer eligible (e.g. the actor's @@ -480,10 +504,7 @@ func (w *ActorWorkflow) assignWorkerAttempt(ctx context.Context, actorRef resour } }() - constraints, err := schedulingConstraints(actor, actorTemplate) - if err != nil { - return nil, nil, err - } + constraints := schedulingConstraints(actor, actorTemplate) snapRuntime := constraints.RequiredRuntime class = constraints.SandboxClass relaxable := canRelaxRuntime(actor, constraints) @@ -501,7 +522,7 @@ func (w *ActorWorkflow) assignWorkerAttempt(ctx context.Context, actorRef resour if err != nil { if errors.Is(err, scheduling.ErrNoCapacity) { outcome = ateattr.SchedulerOutcomeNoCapacity - return nil, nil, apierror.ResourceExhausted("no worker has room for the actor") + return nil, nil, apierror.ResourceExhausted("%w", errNoWorkerRoom) } return nil, nil, err } @@ -584,6 +605,10 @@ func (w *ActorWorkflow) assignWorkerAttempt(ctx context.Context, actorRef resour return storedActor, bound, nil } +// errNoWorkerRoom reports that no worker has room for the Actor under its +// constraints. +var errNoWorkerRoom = errors.New("no worker has room for the actor") + // errWorkerFilledUp reports that the Worker the scheduler picked would not take // the Actor once the store asked under its row lock. Retryable: the next // attempt re-runs scheduling. @@ -622,38 +647,10 @@ func toAteletSandboxRuntime(in *ateapipb.SandboxRuntime) *ateletpb.SandboxRuntim return out } -// actorResourceLimits returns the actor's declared CPU (millicores) and memory -// (bytes) limits from its ActorTemplate, or 0 for a dimension the template did -// not set. These size the sandbox, which takes the two scalars the runtimes -// understand rather than the named set placement accounts in. -func actorResourceLimits(tmpl *ateapipb.ActorTemplate) (cpuMilli, memBytes int64, err error) { - for _, limit := range tmpl.GetResources().GetLimits() { - q, perr := resource.ParseQuantity(limit.GetQuantity()) - if perr != nil { - return 0, 0, fmt.Errorf("invalid template resource limit %s=%q: %w", limit.GetName(), limit.GetQuantity(), perr) - } - switch limit.GetName() { - case "cpu": - cpuMilli = q.MilliValue() - case "memory": - memBytes = q.Value() - } - } - return cpuMilli, memBytes, nil -} - -func schedulingConstraints(actor *ateapipb.Actor, tmpl *ateapipb.ActorTemplate) (scheduling.Constraints, error) { - // Canonicalized here, the one place an assignment's booked resources are - // decided, so what is recorded is sorted however the template was authored. - limits, err := resources.ParseQuantities(tmpl.GetResources()) - if err != nil { - return scheduling.Constraints{}, fmt.Errorf("invalid template resource limits: %w", err) - } +func schedulingConstraints(actor *ateapipb.Actor, tmpl *ateapipb.ActorTemplate) scheduling.Constraints { var requiredNodes []string - // A paused Actor stays pinned to its node until atelet detaches the local - // snapshot, so it cannot resume while that node is full. - // TODO: when the node has no room, detach the Actor there and resume it on - // another node from the durable snapshot. + // A suspended Actor with a local snapshot stays pinned to its node until + // atelet detaches it. When the node is full, ResumeActor detaches it. if node := actor.GetStatus().GetAssignedNode(); node != "" { requiredNodes = []string{node} } @@ -662,12 +659,12 @@ func schedulingConstraints(actor *ateapipb.Actor, tmpl *ateapipb.ActorTemplate) RequiredRuntime: hardware.NewRequirement(requiredSnapshotRuntime(actor, tmpl)), ActorSelector: labels.SelectorFromSet(labels.Set(actor.GetWorkerSelector().GetMatchLabels())), RequiredNodes: requiredNodes, - Limits: limits.Proto(), + Limits: tmpl.GetResources().GetLimits(), } if sel := tmpl.GetWorkerSelector(); sel != nil { c.TemplateSelector = labels.SelectorFromSet(labels.Set(sel.GetMatchLabels())) } - return c, nil + return c } // snapshotSandboxClass returns the sandbox class of the snapshot ResumeActor @@ -695,14 +692,16 @@ func requiredSnapshotRuntime(actor *ateapipb.Actor, tmpl *ateapipb.ActorTemplate if localSnap.GetFidelity() != ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY { return nil } + if templateReplaced(localSnap, tmpl) { + return nil + } return localSnap.GetSandboxRuntime() } if snap := findLatestDurableSnapshot(actor.GetStatus(), ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED); snap != nil { if snap.GetFidelity() != ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY { return nil } - capturedUnder := snap.GetActorTemplateUid() - if capturedUnder != "" && capturedUnder != tmpl.GetMetadata().GetUid() { + if templateReplaced(snap, tmpl) { return nil } return snap.GetSandboxRuntime() @@ -798,10 +797,7 @@ func (w *ActorWorkflow) ensureAteletRestored(ctx context.Context, actorRef resou // The actor's declared limits ride the RPC down to the sandbox so it is sized // to the actor (replacing the worker-pod downward-API approach). - cpuMilli, memBytes, err := actorResourceLimits(actorTemplate) - if err != nil { - return tele, err - } + actorResources := toAteletResources(actorTemplate.GetResources()) // The assignment is what the next checkpoint is stamped from, so it is also // what decides how this restore runs. @@ -812,6 +808,9 @@ func (w *ActorWorkflow) ensureAteletRestored(ctx context.Context, actorRef resou tele.SnapshotKind = ateattr.SnapshotKindLocal fidelity := fidelityToAtelet(localSnap.GetFidelity()) + if templateReplaced(localSnap, actorTemplate) { + fidelity = ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES + } req := &ateletpb.RestoreRequest{ WorkerPodUid: assignment.GetWorkerPodUid(), Atespace: actor.GetMetadata().GetAtespace(), @@ -833,8 +832,7 @@ func (w *ActorWorkflow) ensureAteletRestored(ctx context.Context, actorRef resou SandboxRuntime: sandboxRuntime, ActorUid: actor.GetMetadata().Uid, EgressGateway: egressGateway, - CpuMilli: cpuMilli, - MemoryBytes: memBytes, + Resources: actorResources, } tele.WireFidelity = ateattr.SnapshotFidelityValue(fidelity) @@ -875,8 +873,7 @@ func (w *ActorWorkflow) ensureAteletRestored(ctx context.Context, actorRef resou SandboxRuntime: sandboxRuntime, ActorUid: actor.GetMetadata().Uid, EgressGateway: egressGateway, - CpuMilli: cpuMilli, - MemoryBytes: memBytes, + Resources: actorResources, } if _, err = client.Restore(ctx, req); err != nil { return tele, handleAteletError(ctx, w.store, actorRef, ateattr.OperationResume, "Restore", false, err) @@ -896,8 +893,7 @@ func (w *ActorWorkflow) ensureAteletRestored(ctx context.Context, actorRef resou Spec: workloadSpec, ActorUid: actor.GetMetadata().Uid, EgressGateway: egressGateway, - CpuMilli: cpuMilli, - MemoryBytes: memBytes, + Resources: actorResources, } if _, err = client.Run(ctx, req); err != nil { return tele, handleAteletError(ctx, w.store, actorRef, ateattr.OperationResume, "Run", false, err) diff --git a/cmd/ateapi/internal/controlapi/workflow_resume_detach.go b/cmd/ateapi/internal/controlapi/workflow_resume_detach.go new file mode 100644 index 0000000000..e80e8fe167 --- /dev/null +++ b/cmd/ateapi/internal/controlapi/workflow_resume_detach.go @@ -0,0 +1,159 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package controlapi + +import ( + "context" + "errors" + "fmt" + "log/slog" + + "go.ate.dev/substrate/cmd/ateapi/internal/scheduling" + "go.ate.dev/substrate/cmd/ateapi/internal/store" + "go.ate.dev/substrate/internal/apierror" + "go.ate.dev/substrate/internal/hardware" + "go.ate.dev/substrate/internal/proto/ateletpb" + "go.ate.dev/substrate/internal/resources" + "go.ate.dev/substrate/pkg/proto/ateapipb" + "google.golang.org/grpc/codes" + "google.golang.org/grpc/status" + "google.golang.org/protobuf/proto" +) + +// shouldDetachFromNode reports whether a SUSPENDED actor's assignment failed +// for lack of room on the node it is pinned to. +func shouldDetachFromNode(actor *ateapipb.Actor, assignErr error) bool { + return errors.Is(assignErr, errNoWorkerRoom) && + actor.GetStatus().GetState() == ateapipb.ActorState_ACTOR_STATE_SUSPENDED && + actor.GetStatus().GetAssignedNode() != "" +} + +// detachFromFullNode detaches a SUSPENDED actor from its full node when +// another node has room, so the resume restores it there from the durable +// snapshot, and returns the actor as stored. A worker of another runtime +// counts: it restores the snapshot at ROOTFS, without its process memory, and +// assignment still prefers one that keeps it. With no room anywhere it returns +// assignErr. A failed detach leaves the actor on its node. +func (w *ActorWorkflow) detachFromFullNode(ctx context.Context, actorRef resources.ActorRef, actor *ateapipb.Actor, actorTemplate *ateapipb.ActorTemplate, assignErr error) (_ *ateapipb.Actor, err error) { + ctx, done := stepSpan(ctx, "DetachFromFullNode") + defer func() { err = done(err) }() + + node := actor.GetStatus().GetAssignedNode() + // Assignment may have retried on a newer version than actor. + if actor, err = w.loadPinnedActor(ctx, actorRef, actor.GetMetadata().GetUid(), node); err != nil { + return nil, err + } + constraints := schedulingConstraints(actor, actorTemplate) + constraints.RequiredNodes = nil + constraints.RequiredRuntime = hardware.Requirement{} + if _, err := w.scheduler.Schedule(ctx, constraints); err != nil { + if errors.Is(err, scheduling.ErrNoCapacity) { + return nil, assignErr + } + return nil, err + } + + // The place is not claimed first, so two actors detached for it race in + // their assignments; the loser is left durable-only. + ateletConn, err := w.dialer.DialForAteletOnNode(node) + if err != nil { + return nil, apierror.Unavailable("while getting atelet conn for node %q: %w", node, err) + } + resp, err := ateletpb.NewAteletClient(ateletConn).DetachActor(ctx, &ateletpb.DetachActorRequest{ + Atespace: actor.GetMetadata().GetAtespace(), + ActorName: actor.GetMetadata().GetName(), + ActorUid: actor.GetMetadata().GetUid(), + }) + if err != nil { + return nil, detachActorError(ctx, err) + } + state := resp.GetState() + if state.GetLatestLocal() != nil || state.GetUploading() != nil { + return nil, apierror.Internal("atelet on node %q detached Actor %s but still reports a local snapshot", node, actorRef) + } + + // A conflict fails the resume with Aborted. The next resume detaches + // again, and atelet returns the same state. + stored, err := w.recordDetach(ctx, actorRef, actor, node, state) + if err != nil { + return nil, err + } + slog.InfoContext(ctx, "Detached actor from its full node to resume it on another node", + slog.Any("actor", actorRef), + slog.String("node", node)) + return stored, nil +} + +// loadPinnedActor reads the actor, and fails with Aborted unless it is still +// the SUSPENDED incarnation uid, pinned to node. +func (w *ActorWorkflow) loadPinnedActor(ctx context.Context, actorRef resources.ActorRef, uid, node string) (*ateapipb.Actor, error) { + actor, err := w.store.GetActor(ctx, actorRef) + if err != nil { + return nil, fmt.Errorf("while reading Actor %s: %w", actorRef, err) + } + if actor.GetMetadata().GetUid() != uid || + actor.GetStatus().GetState() != ateapipb.ActorState_ACTOR_STATE_SUSPENDED || + actor.GetStatus().GetAssignedNode() != node { + return nil, apierror.Aborted("Actor %s changed during the resume, please retry", actorRef) + } + return actor, nil +} + +// recordDetach records the state atelet on node returned from DetachActor on +// the actor as read, and returns it as stored. It fails with +// FailedPrecondition if the latest copy on node was not durable. +func (w *ActorWorkflow) recordDetach(ctx context.Context, actorRef resources.ActorRef, actor *ateapipb.Actor, node string, state *ateletpb.ActorSnapshotState) (*ateapipb.Actor, error) { + applied := proto.CloneOf(actor.GetStatus()) + if err := applyLatestDurable(ctx, applied, snapshotReportFromAtelet(state.GetLatestDurable())); err != nil { + return nil, apierror.Internal("atelet on node %q reported a state for Actor %s that conflicts with its record: %w", node, actorRef, err) + } + // Under the lease, atelet deleted every copy on node, including any newer + // than the snapshot it reports durable, and reports none when it holds no + // entry for the actor. + if err := applyDetach(applied, node, applied.GetLastAssignedGeneration()); err != nil { + return nil, err + } + stored, err := w.store.UpdateActor(ctx, actorRef, store.PreconditionFrom(actor), func(toUpdate *ateapipb.Actor) error { + // The precondition makes toUpdate the version applied was built from. + toUpdate.Status.Snapshots = applied.GetSnapshots() + toUpdate.Status.AssignedNode = applied.GetAssignedNode() + return nil + }) + if err != nil { + if errors.Is(err, store.ErrVersionConflict) { + return nil, apierror.Aborted("concurrent update conflict, please retry") + } + return nil, fmt.Errorf("while recording the detach of Actor %s from node %q: %w", actorRef, node, err) + } + return stored, nil +} + +// detachActorError maps a DetachActor error to the one the resume fails with. +func detachActorError(ctx context.Context, err error) error { + // The caller went away or the lease was lost: report that, not atelet's error. + if ctx.Err() != nil { + return fmt.Errorf("while calling atelet DetachActor: %w: %w", err, ctx.Err()) + } + switch status.Code(err) { + case codes.Unavailable, codes.Canceled, codes.DeadlineExceeded: + return apierror.Unavailable("while calling atelet DetachActor: %w", err) + case codes.Aborted: + return apierror.Aborted("while calling atelet DetachActor: %w", err) + // NotFound: the local snapshot is gone and no uploaded copy exists. + case codes.FailedPrecondition, codes.NotFound: + return apierror.FailedPrecondition("while calling atelet DetachActor: %w", err) + } + return apierror.Internal("while calling atelet DetachActor: %w", err) +} diff --git a/cmd/ateapi/internal/controlapi/workflow_resume_detach_test.go b/cmd/ateapi/internal/controlapi/workflow_resume_detach_test.go new file mode 100644 index 0000000000..e11902b4e0 --- /dev/null +++ b/cmd/ateapi/internal/controlapi/workflow_resume_detach_test.go @@ -0,0 +1,466 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package controlapi + +import ( + "context" + "errors" + "testing" + "time" + + "go.ate.dev/substrate/cmd/ateapi/internal/scheduling" + "go.ate.dev/substrate/cmd/ateapi/internal/store" + "go.ate.dev/substrate/cmd/ateapi/internal/store/storetest" + "go.ate.dev/substrate/cmd/ateapi/internal/workercache" + "go.ate.dev/substrate/internal/apierror" + "go.ate.dev/substrate/internal/installdefaults" + "go.ate.dev/substrate/internal/proto/ateletpb" + "go.ate.dev/substrate/internal/resources" + "go.ate.dev/substrate/pkg/proto/ateapipb" + "google.golang.org/grpc/codes" + "google.golang.org/protobuf/proto" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" +) + +// newDetachTestWorkflow schedules over the workers already in st and dials the +// capturing atelet on node-1. +func newDetachTestWorkflow(t *testing.T, ctx context.Context, st store.Interface) (*ActorWorkflow, *capturingAtelet) { + t.Helper() + w, atelet := newWireCaptureWorkflow(t, st) + cacheCtx, cancel := context.WithCancel(ctx) + t.Cleanup(cancel) + wc := workercache.New(st, time.Minute) + if err := wc.Start(cacheCtx); err != nil { + t.Fatalf("workercache.Start: %v", err) + } + w.workerCache = wc + w.scheduler = scheduling.New(wc) + return w, atelet +} + +// fillWorker binds another actor to worker, using up its one place. +func fillWorker(t *testing.T, st store.Interface, worker *ateapipb.Worker) { + t.Helper() + seedAssignment(t, st, worker.GetMetadata().GetName(), &ateapipb.ActorAssignment{ + Actor: &ateapipb.ObjectRef{Atespace: "team-a", Name: "other"}, + ActorUid: "other-uid", + }) +} + +// detachedState is atelet's state after detaching snap. +func detachedState(snap *ateapipb.Snapshot) *ateletpb.ActorSnapshotState { + return &ateletpb.ActorSnapshotState{LatestDurable: &ateletpb.SnapshotReport{Snapshot: &ateletpb.Snapshot{ + SnapshotUuid: snap.GetUuid(), + Generation: snap.GetGeneration(), + Fidelity: fidelityToAtelet(snap.GetFidelity()), + Object: &ateletpb.ObjectStorage{SnapshotUri: snap.GetDurableSnapshot().GetObject().GetSnapshotUri()}, + }}} +} + +func TestShouldDetachFromNode(t *testing.T) { + noRoom := apierror.ResourceExhausted("%w", errNoWorkerRoom) + pinned := func(state ateapipb.ActorState, node string) *ateapipb.Actor { + return &ateapipb.Actor{Status: &ateapipb.ActorStatus{State: state, AssignedNode: node}} + } + tests := []struct { + name string + actor *ateapipb.Actor + err error + want bool + }{ + {"pinned suspended actor with no room", pinned(ateapipb.ActorState_ACTOR_STATE_SUSPENDED, "node-1"), noRoom, true}, + {"no room wrapped by a step", pinned(ateapipb.ActorState_ACTOR_STATE_SUSPENDED, "node-1"), errors.Join(noRoom), true}, + {"assigned", pinned(ateapipb.ActorState_ACTOR_STATE_SUSPENDED, "node-1"), nil, false}, + {"another error", pinned(ateapipb.ActorState_ACTOR_STATE_SUSPENDED, "node-1"), apierror.Aborted("conflict"), false}, + {"not pinned", pinned(ateapipb.ActorState_ACTOR_STATE_SUSPENDED, ""), noRoom, false}, + {"resuming", pinned(ateapipb.ActorState_ACTOR_STATE_RESUMING, "node-1"), noRoom, false}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if got := shouldDetachFromNode(tt.actor, tt.err); got != tt.want { + t.Errorf("shouldDetachFromNode() = %v, want %v", got, tt.want) + } + }) + } +} + +// TestDetachFromFullNode runs the step for an actor pinned to a full node. +func TestDetachFromFullNode(t *testing.T) { + actorRef := resources.ActorRef{Atespace: "team-a", Name: "id1"} + suspended := ateapipb.ActorState_ACTOR_STATE_SUSPENDED + const uuid = "pause-snap-1" + noRoom := apierror.ResourceExhausted("%w", errNoWorkerRoom) + amd64RT := testRuntime("amd64", "") + arm64RT := testRuntime("arm64", "") + + tests := []struct { + name string + // node is the actor's full node. Only node-1 has an atelet. + node string + // otherNode adds a free worker on node-2 with otherRuntime, or the + // test runtime when unset. + otherNode bool + otherRuntime *ateapipb.SandboxRuntime + // snapRuntime stamps the MEMORY snapshot. + snapRuntime *ateapipb.SandboxRuntime + // completed records the snapshot's upload COMPLETED. + completed bool + // before updates the actor after it was read and before the step + // runs, and during while atelet detaches it. + before, during func(*ateapipb.Actor) + // detachState builds DetachActor's reply from the snapshot. + detachState func(*ateapipb.Snapshot) *ateletpb.ActorSnapshotState + detachErr error + wantDetached bool + wantCode codes.Code + wantCall bool + }{{ + name: "another node has room: detaches", + node: "node-1", + otherNode: true, + detachState: detachedState, + wantDetached: true, + wantCall: true, + }, { + name: "another node can restore the snapshot's runtime: detaches", + node: "node-1", + otherNode: true, + otherRuntime: amd64RT, + snapRuntime: amd64RT, + detachState: detachedState, + wantDetached: true, + wantCall: true, + }, { + name: "actor updated after it was read: detaches", + node: "node-1", + otherNode: true, + before: func(*ateapipb.Actor) {}, + detachState: detachedState, + wantDetached: true, + wantCall: true, + }, { + name: "selector changed after the actor was read: no room under the new one", + node: "node-1", + otherNode: true, + before: func(a *ateapipb.Actor) { + a.WorkerSelector = &ateapipb.Selector{MatchLabels: map[string]string{"tier": "paid"}} + }, + wantCode: codes.ResourceExhausted, + }, { + name: "actor updated while atelet detaches: Aborted, record unchanged", + node: "node-1", + otherNode: true, + during: func(*ateapipb.Actor) {}, + detachState: detachedState, + wantCode: codes.Aborted, + wantCall: true, + }, { + name: "atelet holds no entry, upload COMPLETED: detaches", + node: "node-1", + otherNode: true, + completed: true, + detachState: func(*ateapipb.Snapshot) *ateletpb.ActorSnapshotState { return &ateletpb.ActorSnapshotState{} }, + wantDetached: true, + wantCall: true, + }, { + name: "atelet holds no entry, upload PENDING: FailedPrecondition, record unchanged", + node: "node-1", + otherNode: true, + detachState: func(*ateapipb.Snapshot) *ateletpb.ActorSnapshotState { return &ateletpb.ActorSnapshotState{} }, + wantCode: codes.FailedPrecondition, + wantCall: true, + }, { + name: "no node has room: the assignment's error, without detaching", + node: "node-1", + wantCode: codes.ResourceExhausted, + }, { + name: "room only on another runtime: detaches", + node: "node-1", + otherNode: true, + otherRuntime: arm64RT, + snapRuntime: amd64RT, + detachState: detachedState, + wantDetached: true, + wantCall: true, + }, { + name: "no atelet on the node: Unavailable, actor stays on its node", + node: "node-3", + otherNode: true, + wantCode: codes.Unavailable, + }, { + name: "atelet refuses: Aborted, actor stays on its node", + node: "node-1", + otherNode: true, + detachErr: apierror.Aborted("actor is in use on this node"), + wantCode: codes.Aborted, + wantCall: true, + }, { + name: "atelet unreachable: Unavailable, actor stays on its node", + node: "node-1", + otherNode: true, + detachErr: apierror.Unavailable("connection refused"), + wantCode: codes.Unavailable, + wantCall: true, + }, { + name: "atelet keeps a local copy: Internal, actor stays on its node", + node: "node-1", + otherNode: true, + detachState: func(snap *ateapipb.Snapshot) *ateletpb.ActorSnapshotState { + st := detachedState(snap) + st.LatestLocal = st.GetLatestDurable() + return st + }, + wantCode: codes.Internal, + wantCall: true, + }, { + name: "returned state conflicts with the record: Internal, actor stays on its node", + node: "node-1", + otherNode: true, + detachState: func(snap *ateapipb.Snapshot) *ateletpb.ActorSnapshotState { + st := detachedState(snap) + st.LatestDurable.Snapshot.Object.SnapshotUri = someActorSnapshotURI(t, testStorageLocation, "team-a", "other") + return st + }, + wantCode: codes.Internal, + wantCall: true, + }} + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + ctx := context.Background() + st := newTestPersistence(t) + fillWorker(t, st, seedRuntimeWorker(t, ctx, st, "pod-"+tt.node, tt.node, testSandboxRuntime())) + if tt.otherNode { + otherRuntime := tt.otherRuntime + if otherRuntime == nil { + otherRuntime = testSandboxRuntime() + } + seedRuntimeWorker(t, ctx, st, "pod-2", "node-2", otherRuntime) + } + snap := newPausedSnapshot(t, 1, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", uuid, tt.node) + snap.SandboxRuntime = tt.snapRuntime + if tt.completed { + snap.DurableSnapshot.Status = ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED + } + actor := seedActorWithSnapshot(t, ctx, st, suspended, tt.node, snap) + w, atelet := newDetachTestWorkflow(t, ctx, st) + if tt.detachState != nil { + atelet.detachState = tt.detachState(snap) + } + atelet.detachErr = tt.detachErr + // update writes mutate to the stored actor, moving its version on. + update := func(mutate func(*ateapipb.Actor)) { + t.Helper() + cur, err := st.GetActor(ctx, actorRef) + if err != nil { + t.Fatalf("GetActor: %v", err) + } + if _, err := st.UpdateActor(ctx, actorRef, store.PreconditionFrom(cur), func(toUpdate *ateapipb.Actor) error { + mutate(toUpdate) + return nil + }); err != nil { + t.Fatalf("UpdateActor: %v", err) + } + } + if tt.before != nil { + update(tt.before) + } + if tt.during != nil { + atelet.onDetach = func() { update(tt.during) } + } + // The record unchanged by a failed step is the one as last written. + want, err := st.GetActor(ctx, actorRef) + if err != nil { + t.Fatalf("GetActor: %v", err) + } + + stored, err := w.detachFromFullNode(ctx, actorRef, actor, &ateapipb.ActorTemplate{}, noRoom) + if got := apierror.Code(err); got != tt.wantCode { + t.Fatalf("detachFromFullNode() = %v, want code %v", err, tt.wantCode) + } + req := atelet.detachRequest() + if (req != nil) != tt.wantCall { + t.Fatalf("DetachActor request = %v, want a call: %v", req, tt.wantCall) + } + if req != nil { + want := &ateletpb.DetachActorRequest{Atespace: "team-a", ActorName: "id1", ActorUid: actor.GetMetadata().GetUid()} + if !proto.Equal(req, want) { + t.Errorf("DetachActor request = %v, want %v", req, want) + } + } + + after, err := st.GetActor(ctx, actorRef) + if err != nil { + t.Fatalf("GetActor: %v", err) + } + if !tt.wantDetached { + if !proto.Equal(after.GetStatus(), want.GetStatus()) { + t.Errorf("status = %v, want unchanged %v", after.GetStatus(), want.GetStatus()) + } + return + } + if !proto.Equal(stored, after) { + t.Errorf("returned actor = %v, want the stored %v", stored, after) + } + if got := after.GetStatus().GetAssignedNode(); got != "" { + t.Errorf("AssignedNode = %q, want none", got) + } + got := after.GetStatus().GetSnapshots() + if len(got) != 1 || got[0].GetLocality() != "" || + got[0].GetDurableSnapshot().GetStatus() != ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED { + t.Errorf("snapshots = %v, want %s durable only and COMPLETED", got, uuid) + } + }) + } +} + +// TestResumeActor_DetachesFromFullNode resumes an actor pinned to node-1 with +// room on node-2: it moves to node-2 only when node-1 is full. +func TestResumeActor_DetachesFromFullNode(t *testing.T) { + tests := []struct { + name string + node1 bool + wantNode string + }{ + {name: "node-1 full: detaches and resumes on node-2", wantNode: "node-2"}, + {name: "node-1 has room: resumes there from the local copy", node1: true, wantNode: "node-1"}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + ctx := context.Background() + st := newTestPersistence(t) + actorRef := resources.ActorRef{Atespace: "team-a", Name: "id1"} + + storetest.MustCreateAtespace(t, ctx, st, "ns") + if _, err := st.CreateActorTemplate(ctx, &ateapipb.ActorTemplate{ + Metadata: &ateapipb.ResourceMetadata{Atespace: "ns", Name: "tmpl1"}, + SnapshotConfig: &ateapipb.SnapshotConfig{StorageLocation: testStorageLocation}, + }); err != nil { + t.Fatalf("create template: %v", err) + } + workers := map[string]*ateapipb.Worker{ + "node-1": seedRuntimeWorker(t, ctx, st, "pod-1", "node-1", testSandboxRuntime()), + "node-2": seedRuntimeWorker(t, ctx, st, "pod-2", "node-2", testSandboxRuntime()), + } + if !tt.node1 { + fillWorker(t, st, workers["node-1"]) + } + snap := newPausedSnapshot(t, 1, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", "pause-snap-1", "node-1") + seedWorkflowActor(t, ctx, st, actorRef, "ns", "tmpl1", ateapipb.ActorState_ACTOR_STATE_SUSPENDED, func(a *ateapipb.Actor) { + a.Status.AssignedNode = "node-1" + a.Status.LastAssignedGeneration = 1 + a.Status.Snapshots = []*ateapipb.Snapshot{proto.CloneOf(snap)} + }) + + w, atelet := newDetachTestWorkflow(t, ctx, st) + addAteletOnNode(t, w, "node-2") + atelet.detachState = detachedState(snap) + + got, resumed, err := w.ResumeActor(ctx, actorRef) + if err != nil { + t.Fatalf("ResumeActor: %v", err) + } + if !resumed || got.GetStatus().GetState() != ateapipb.ActorState_ACTOR_STATE_RUNNING { + t.Fatalf("ResumeActor = (state %v, resumed %v), want RUNNING and resumed", got.GetStatus().GetState(), resumed) + } + detached := atelet.detachRequest() != nil + if wantDetach := tt.wantNode != "node-1"; detached != wantDetach { + t.Errorf("DetachActor called = %v, want %v", detached, wantDetach) + } + if node := got.GetStatus().GetAssignedNode(); node != tt.wantNode { + t.Errorf("AssignedNode = %q, want %s", node, tt.wantNode) + } + restore, run := atelet.requests() + if restore == nil || run != nil { + t.Fatalf("atelet requests = (restore=%v, run=%v), want exactly one Restore", restore, run) + } + if got, want := restore.GetWorkerPodUid(), workers[tt.wantNode].GetWorkerPodUid(); got != want { + t.Errorf("Restore worker = %q, want %s's %q", got, tt.wantNode, want) + } + want := &ateletpb.Snapshot{ + SnapshotUuid: snap.GetUuid(), + Generation: snap.GetGeneration(), + Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, + Object: &ateletpb.ObjectStorage{SnapshotUri: snap.GetDurableSnapshot().GetObject().GetSnapshotUri()}, + } + if !proto.Equal(restore.GetSnapshot(), want) { + t.Errorf("Restore snapshot = %v, want %v", restore.GetSnapshot(), want) + } + }) + } +} + +// addAteletOnNode routes node's atelet to the same capturing fake as node-1's. +func addAteletOnNode(t *testing.T, w *ActorWorkflow, node string) { + t.Helper() + cached, ok := w.dialer.ateletConns.Get("atelet-uid") + if !ok { + t.Fatal("no cached conn for the fake atelet") + } + uid := types.UID("atelet-uid-" + node) + if err := w.dialer.ateletIndexer.Add(&corev1.Pod{ + ObjectMeta: metav1.ObjectMeta{Namespace: installdefaults.SystemNamespace, Name: "atelet-" + node, UID: uid}, + Spec: corev1.PodSpec{NodeName: node}, + Status: corev1.PodStatus{PodIPs: []corev1.PodIP{{IP: "10.0.0.2"}}}, + }); err != nil { + t.Fatalf("adding atelet pod on %s: %v", node, err) + } + w.dialer.ateletConns.Add(string(uid), &ateletConn{ip: "10.0.0.2", conn: cached.(*ateletConn).conn}) +} + +// TestDetachFromFullNode_ReentryRecordsAfterConflict loses the record write +// to a concurrent update, then runs the step again, as the next resume does: +// the second attempt detaches again and records the detach. +func TestDetachFromFullNode_ReentryRecordsAfterConflict(t *testing.T) { + ctx := context.Background() + st := newTestPersistence(t) + actorRef := resources.ActorRef{Atespace: "team-a", Name: "id1"} + fillWorker(t, st, seedRuntimeWorker(t, ctx, st, "pod-node-1", "node-1", testSandboxRuntime())) + seedRuntimeWorker(t, ctx, st, "pod-2", "node-2", testSandboxRuntime()) + snap := newPausedSnapshot(t, 1, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", "pause-snap-1", "node-1") + actor := seedActorWithSnapshot(t, ctx, st, ateapipb.ActorState_ACTOR_STATE_SUSPENDED, "node-1", snap) + w, atelet := newDetachTestWorkflow(t, ctx, st) + atelet.detachState = detachedState(snap) + noRoom := apierror.ResourceExhausted("%w", errNoWorkerRoom) + + // A concurrent update while atelet detaches. + atelet.onDetach = func() { + cur, err := st.GetActor(ctx, actorRef) + if err != nil { + t.Errorf("GetActor: %v", err) + return + } + if _, err := st.UpdateActor(ctx, actorRef, store.PreconditionFrom(cur), func(*ateapipb.Actor) error { return nil }); err != nil { + t.Errorf("UpdateActor: %v", err) + } + } + if _, err := w.detachFromFullNode(ctx, actorRef, actor, &ateapipb.ActorTemplate{}, noRoom); apierror.Code(err) != codes.Aborted { + t.Fatalf("first detachFromFullNode() = %v, want Aborted", err) + } + + atelet.onDetach = nil + stored, err := w.detachFromFullNode(ctx, actorRef, actor, &ateapipb.ActorTemplate{}, noRoom) + if err != nil { + t.Fatalf("second detachFromFullNode() = %v, want success", err) + } + if got := stored.GetStatus().GetAssignedNode(); got != "" { + t.Errorf("AssignedNode = %q, want none", got) + } + if got := stored.GetStatus().GetSnapshots(); len(got) != 1 || got[0].GetLocality() != "" || + got[0].GetDurableSnapshot().GetStatus() != ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED { + t.Errorf("snapshots = %v, want pause-snap-1 durable only and COMPLETED", got) + } +} diff --git a/cmd/ateapi/internal/controlapi/workflow_resume_test.go b/cmd/ateapi/internal/controlapi/workflow_resume_test.go index 6d12b6b0e4..15a527569e 100644 --- a/cmd/ateapi/internal/controlapi/workflow_resume_test.go +++ b/cmd/ateapi/internal/controlapi/workflow_resume_test.go @@ -1096,6 +1096,14 @@ type capturingAtelet struct { promoteState *ateletpb.ActorSnapshotState // promoteErr, when set, fails PromoteSnapshot after it is recorded. promoteErr error + // detach is the last DetachActor request, and detachState the snapshot + // state DetachActor returns. detachErr, when set, fails DetachActor after + // it is recorded. + detach *ateletpb.DetachActorRequest + detachState *ateletpb.ActorSnapshotState + detachErr error + // onDetach, when set, runs before DetachActor returns. + onDetach func() } func (f *capturingAtelet) Restore(ctx context.Context, req *ateletpb.RestoreRequest) (*ateletpb.RestoreResponse, error) { @@ -1139,6 +1147,28 @@ func (f *capturingAtelet) PromoteSnapshot(ctx context.Context, req *ateletpb.Pro return &ateletpb.PromoteSnapshotResponse{State: proto.CloneOf(f.promoteState)}, nil } +func (f *capturingAtelet) DetachActor(ctx context.Context, req *ateletpb.DetachActorRequest) (*ateletpb.DetachActorResponse, error) { + f.mu.Lock() + hook := f.onDetach + f.mu.Unlock() + if hook != nil { + hook() + } + f.mu.Lock() + defer f.mu.Unlock() + f.detach = proto.CloneOf(req) + if f.detachErr != nil { + return nil, f.detachErr + } + return &ateletpb.DetachActorResponse{State: proto.CloneOf(f.detachState)}, nil +} + +func (f *capturingAtelet) detachRequest() *ateletpb.DetachActorRequest { + f.mu.Lock() + defer f.mu.Unlock() + return proto.CloneOf(f.detach) +} + // calls returns how many Checkpoint and PromoteSnapshot RPCs were received. func (f *capturingAtelet) calls() (checkpoints, promotes int) { f.mu.Lock() @@ -1493,9 +1523,6 @@ func TestResumeActor_AteletWireRequest(t *testing.T) { }, }, { - // The repoint permutation without a durable snapshot is - // deliberately not pinned: its wire scope is in flux while the - // resume-source resolution is being reworked. name: "14 local snapshot built on the current template stays Full", actor: actorSeed{ localSnapshot: localSnap(localSnapshotName, fullScope), @@ -1507,6 +1534,30 @@ func TestResumeActor_AteletWireRequest(t *testing.T) { scope: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, }, }, + { + name: "14a repointed actor's Full local snapshot drops to Data", + actor: actorSeed{ + localSnapshot: localSnap(localSnapshotName, fullScope), + tmplUID: "mismatch", + }, + want: restoreWant{ + snapshotName: localSnapshotName, + snapshotURI: pauseUploadURI, + scope: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, + }, + }, + { + name: "14b repointed actor's Rootfs local snapshot drops to Data", + actor: actorSeed{ + localSnapshot: localSnap(localSnapshotName, rootfsScope), + tmplUID: "mismatch", + }, + want: restoreWant{ + snapshotName: localSnapshotName, + snapshotURI: pauseUploadURI, + scope: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, + }, + }, { name: "15 Data pause snapshot restores locally as Data", actor: actorSeed{ @@ -1548,12 +1599,14 @@ func TestResumeActor_AteletWireRequest(t *testing.T) { }, { // An older external snapshot's template mismatch does not affect a - // local pause restore: templates can only be updated while - // SUSPENDED, so a pause checkpoint is always from the current - // template. + // local pause restore captured under the current template. name: "18 local snapshot ignores an older external snapshot's template mismatch", actor: actorSeed{ - localSnapshot: localSnap(localSnapshotName, fullScope), + localSnapshot: func() *ateapipb.Snapshot { + s := localSnap(localSnapshotName, fullScope) + s.ActorTemplateUid = "current" + return s + }(), externalSnapshot: extSnap(actorURI, fullScope), tmplUID: "mismatch", }, @@ -1742,6 +1795,8 @@ func TestResumeActor_AteletWireRequest(t *testing.T) { loc.Generation = gen if loc.ActorTemplateUid == "" { loc.ActorTemplateUid = uid + } else if loc.ActorTemplateUid == "current" { + loc.ActorTemplateUid = createdTmpl.GetMetadata().GetUid() } a.Status.Snapshots = append(a.Status.Snapshots, loc) } @@ -2762,6 +2817,16 @@ func TestRequiredSnapshotRuntime(t *testing.T) { snapshots: []*ateapipb.Snapshot{local(1, memory, localRT)}, want: localRT, }, + { + name: "local checkpoint captured under a replaced template imposes no constraint", + snapshots: []*ateapipb.Snapshot{ + func() *ateapipb.Snapshot { + s := newPausedSnapshot(t, 1, memory, "old-tmpl-uid", "local", "node-1") + s.SandboxRuntime = localRT + return s + }(), + }, + }, { // ensureAteletRestored restores a local checkpoint before any // durable snapshot, so the constraint follows it regardless of diff --git a/cmd/ateapi/internal/controlapi/workload_spec.go b/cmd/ateapi/internal/controlapi/workload_spec.go index ac23395717..3aa46b8966 100644 --- a/cmd/ateapi/internal/controlapi/workload_spec.go +++ b/cmd/ateapi/internal/controlapi/workload_spec.go @@ -19,31 +19,17 @@ import ( "go.ate.dev/substrate/internal/proto/ateletpb" "go.ate.dev/substrate/pkg/proto/ateapipb" - "k8s.io/apimachinery/pkg/api/resource" ) -// toAteletResources resolves a container's declared limits into the scalars -// atelet and the workers consume; everything downstream compares numbers. -// Returns nil when the container declares no limits, so the OCI spec stays -// untouched for templates that do not use them. -func toAteletResources(r *ateapipb.Resources) (*ateletpb.ResourceLimits, error) { - out := &ateletpb.ResourceLimits{} - for _, limit := range r.GetLimits() { - q, err := resource.ParseQuantity(limit.GetQuantity()) - if err != nil { - return nil, fmt.Errorf("invalid container resource limit %s=%q: %w", limit.GetName(), limit.GetQuantity(), err) - } - switch limit.GetName() { - case "cpu": - out.CpuMillis = q.MilliValue() - case "memory": - out.MemoryBytes = q.Value() - } - } - if out.MemoryBytes == 0 && out.CpuMillis == 0 { - return nil, nil +// toAteletResources copies declared limits into atelet's form. Returns nil +// when no limits are declared, so the OCI spec stays untouched for templates +// that do not use them. +func toAteletResources(r *ateapipb.ResourceRequirements) *ateletpb.ResourceRequirements { + l := r.GetLimits() + if l.GetMemoryBytes() == 0 && l.GetCpuMillicores() == 0 { + return nil } - return out, nil + return &ateletpb.ResourceRequirements{Limits: &ateletpb.Resources{CpuMillicores: l.GetCpuMillicores(), MemoryBytes: l.GetMemoryBytes()}} } // workloadSpecFromActorTemplate builds a WorkloadSpec from the template; @@ -116,10 +102,7 @@ func workloadSpecFromActorTemplate(actorTemplate *ateapipb.ActorTemplate, actor } for _, ctr := range actorTemplate.GetContainers() { - ctrResources, err := toAteletResources(ctr.GetResources()) - if err != nil { - return nil, err - } + ctrResources := toAteletResources(ctr.GetResources()) ateletCtr := &ateletpb.Container{ Name: ctr.GetName(), Image: ctr.GetImage(), diff --git a/cmd/ateapi/internal/controlapi/workload_spec_test.go b/cmd/ateapi/internal/controlapi/workload_spec_test.go index e781297b7e..0b085a0e5a 100644 --- a/cmd/ateapi/internal/controlapi/workload_spec_test.go +++ b/cmd/ateapi/internal/controlapi/workload_spec_test.go @@ -537,65 +537,42 @@ func TestWorkloadSpecFromActorTemplatePropagatesSecurityContext(t *testing.T) { func TestToAteletResources(t *testing.T) { tests := []struct { - name string - in *ateapipb.Resources - want *ateletpb.ResourceLimits - wantErr bool + name string + in *ateapipb.ResourceRequirements + want *ateletpb.ResourceRequirements }{{ name: "nil resources", in: nil, want: nil, }, { - name: "empty limits", - in: &ateapipb.Resources{}, + name: "empty resources", + in: &ateapipb.ResourceRequirements{}, want: nil, }, { name: "memory only", - in: &ateapipb.Resources{Limits: []*ateapipb.Limits{{Name: "memory", Quantity: "256Mi"}}}, - want: &ateletpb.ResourceLimits{MemoryBytes: 268435456}, + in: &ateapipb.ResourceRequirements{Limits: &ateapipb.Resources{MemoryBytes: 256 << 20}}, + want: &ateletpb.ResourceRequirements{Limits: &ateletpb.Resources{MemoryBytes: 268435456}}, }, { name: "cpu only", - in: &ateapipb.Resources{Limits: []*ateapipb.Limits{{Name: "cpu", Quantity: "200m"}}}, - want: &ateletpb.ResourceLimits{CpuMillis: 200}, + in: &ateapipb.ResourceRequirements{Limits: &ateapipb.Resources{CpuMillicores: 200}}, + want: &ateletpb.ResourceRequirements{Limits: &ateletpb.Resources{CpuMillicores: 200}}, }, { name: "both", - in: &ateapipb.Resources{Limits: []*ateapipb.Limits{ - {Name: "cpu", Quantity: "1"}, - {Name: "memory", Quantity: "1Gi"}, - }}, - want: &ateletpb.ResourceLimits{MemoryBytes: 1073741824, CpuMillis: 1000}, - }, { - name: "invalid quantity", - in: &ateapipb.Resources{Limits: []*ateapipb.Limits{{Name: "cpu", Quantity: "banana"}}}, - wantErr: true, + in: &ateapipb.ResourceRequirements{Limits: &ateapipb.Resources{CpuMillicores: 1000, MemoryBytes: 1 << 30}}, + want: &ateletpb.ResourceRequirements{Limits: &ateletpb.Resources{MemoryBytes: 1073741824, CpuMillicores: 1000}}, }} for _, tc := range tests { t.Run(tc.name, func(t *testing.T) { - got, err := toAteletResources(tc.in) - if tc.wantErr { - if err == nil { - t.Fatal("toAteletResources() error = nil, want parse error") - } - return - } - if err != nil { - t.Fatalf("toAteletResources() failed: %v", err) - } + got := toAteletResources(tc.in) if tc.want == nil { if got != nil { t.Fatalf("toAteletResources() = %v, want nil", got) } return } - if got == nil { - t.Fatalf("toAteletResources() = nil, want %v", tc.want) - } - if got.GetMemoryBytes() != tc.want.GetMemoryBytes() { - t.Errorf("memory_bytes = %d, want %d", got.GetMemoryBytes(), tc.want.GetMemoryBytes()) - } - if got.GetCpuMillis() != tc.want.GetCpuMillis() { - t.Errorf("cpu_millis = %d, want %d", got.GetCpuMillis(), tc.want.GetCpuMillis()) + if diff := cmp.Diff(tc.want, got, protocmp.Transform()); diff != "" { + t.Errorf("toAteletResources() mismatch (-want +got):\n%s", diff) } }) } @@ -610,12 +587,9 @@ func TestWorkloadSpecFromActorTemplatePropagatesResources(t *testing.T) { Metadata: &ateapipb.ResourceMetadata{Atespace: "agent-ns", Name: "tmpl-limits"}, Containers: []*ateapipb.Container{ { - Name: "limited", - Image: "main", - Resources: &ateapipb.Resources{Limits: []*ateapipb.Limits{ - {Name: "cpu", Quantity: "200m"}, - {Name: "memory", Quantity: "256Mi"}, - }}, + Name: "limited", + Image: "main", + Resources: &ateapipb.ResourceRequirements{Limits: &ateapipb.Resources{CpuMillicores: 200, MemoryBytes: 256 << 20}}, }, {Name: "unlimited", Image: "main"}, }, @@ -631,11 +605,11 @@ func TestWorkloadSpecFromActorTemplatePropagatesResources(t *testing.T) { if limited == nil { t.Fatal("limited container Resources = nil, want the declared limits") } - if limited.GetMemoryBytes() != 268435456 { - t.Errorf("memory_bytes = %d, want 268435456", limited.GetMemoryBytes()) + if got := limited.GetLimits().GetMemoryBytes(); got != 268435456 { + t.Errorf("memory_bytes = %d, want 268435456", got) } - if limited.GetCpuMillis() != 200 { - t.Errorf("cpu_millis = %d, want 200", limited.GetCpuMillis()) + if got := limited.GetLimits().GetCpuMillicores(); got != 200 { + t.Errorf("cpu_millicores = %d, want 200", got) } if r := got.GetContainers()[1].GetResources(); r != nil { t.Errorf("unlimited container Resources = %v, want nil", r) diff --git a/cmd/ateapi/internal/scheduling/scheduling.go b/cmd/ateapi/internal/scheduling/scheduling.go index b9930ac531..f7f4f91dd3 100644 --- a/cmd/ateapi/internal/scheduling/scheduling.go +++ b/cmd/ateapi/internal/scheduling/scheduling.go @@ -53,8 +53,8 @@ type Constraints struct { // to specific node VMs. RequiredNodes []string - // Limits are the actor's declared resource limits, named as a Worker names - // the capacity it reports, so the two subtract. + // Limits are the actor's declared resource limits, in the units a Worker + // reports its capacity in, so the two subtract. Limits *ateapipb.Resources } @@ -118,17 +118,12 @@ func (s *scheduler) Schedule(ctx context.Context, constraints Constraints) (*ate return nil, fmt.Errorf("while listing workers: %w", err) } - want, err := resources.ParseQuantities(constraints.Limits) - if err != nil { - return nil, fmt.Errorf("while parsing actor resource limits: %w", err) - } - var candidates []candidate for _, worker := range workers { if !s.Applies(worker, constraints) { continue } - if cand, ok := checkRoom(worker, want); ok { + if cand, ok := checkRoom(worker, constraints.Limits); ok { candidates = append(candidates, cand) } } @@ -192,30 +187,23 @@ func lessLoaded(a, b *candidate) bool { } // workerResourceUtilization returns the highest allocated/capacity ratio across -// compute dimensions, or 1 when capacity is unreported or quantities fail to parse. +// compute dimensions, or 1 when capacity is unreported. func workerResourceUtilization(w *ateapipb.Worker) float64 { - capQ, err := resources.ParseQuantities(w.GetStatus().GetCapacity().GetResources()) - if err != nil || len(capQ) == 0 { - return 1 - } - usedQ, err := resources.ParseQuantities(w.GetStatus().GetAllocated().GetResources()) - if err != nil { - return 1 - } - return quantitiesUtilization(capQ, usedQ) + return utilization(w.GetStatus().GetCapacity(), w.GetStatus().GetAllocated()) } -func quantitiesUtilization(capQ, usedQ resources.Quantities) float64 { +func utilization(capacity, used *ateapipb.WorkerResources) float64 { var ( maxRatio float64 hasPositive bool ) - for name, capVal := range capQ { - if capFloat := capVal.AsApproximateFloat64(); capFloat > 0 { + for _, d := range [][2]int64{ + {capacity.GetCpuMillicores(), used.GetCpuMillicores()}, + {capacity.GetMemoryBytes(), used.GetMemoryBytes()}, + } { + if d[0] > 0 { hasPositive = true - if usedVal, ok := usedQ[name]; ok { - maxRatio = max(maxRatio, usedVal.AsApproximateFloat64()/capFloat) - } + maxRatio = max(maxRatio, float64(d[1])/float64(d[0])) } } if !hasPositive { @@ -262,20 +250,12 @@ func (s *scheduler) Applies(worker *ateapipb.Worker, constraints Constraints) bo // HasRoom reports whether what the worker has left admits one more actor of // this size. A dimension the worker does not report is unconstrained, so // placement is never blocked by missing data. -// -// A worker whose recorded capacity or allocation will not parse is treated as -// having no room: it is the only answer that cannot overcommit a worker whose -// true occupancy is unreadable. func (s *scheduler) HasRoom(worker *ateapipb.Worker, constraints Constraints) bool { - want, err := resources.ParseQuantities(constraints.Limits) - if err != nil { - return false - } - _, ok := checkRoom(worker, want) + _, ok := checkRoom(worker, constraints.Limits) return ok } -func checkRoom(worker *ateapipb.Worker, want resources.Quantities) (candidate, bool) { +func checkRoom(worker *ateapipb.Worker, want *ateapipb.Resources) (candidate, bool) { capacity := worker.GetStatus().GetCapacity() used := worker.GetStatus().GetAllocated() @@ -285,24 +265,12 @@ func checkRoom(worker *ateapipb.Worker, want resources.Quantities) (candidate, b return candidate{}, false } - if len(want) == 0 { + if want.GetCpuMillicores() == 0 && want.GetMemoryBytes() == 0 { return candidate{worker: worker, resUtil: -1}, true } - free, err := resources.ParseQuantities(capacity.GetResources()) - if err != nil { - return candidate{}, false - } - if free == nil { - free = resources.Quantities{} - } - allocated, err := resources.ParseQuantities(used.GetResources()) - if err != nil { - return candidate{}, false - } - resUtil := quantitiesUtilization(free, allocated) - free.Sub(allocated) - if !free.Covers(want) { + if !resources.Covers(resources.Free(capacity, used), want) { return candidate{}, false } + resUtil := utilization(capacity, used) return candidate{worker: worker, resUtil: resUtil}, true } diff --git a/cmd/ateapi/internal/scheduling/scheduling_test.go b/cmd/ateapi/internal/scheduling/scheduling_test.go index d279b2c8f7..36aa42547a 100644 --- a/cmd/ateapi/internal/scheduling/scheduling_test.go +++ b/cmd/ateapi/internal/scheduling/scheduling_test.go @@ -295,22 +295,6 @@ func TestSchedule(t *testing.T) { constraints: Constraints{}, wantPod: "w-known", }, - { - name: "prefers worker with known resource utilization over malformed allocated quantities", - fleet: fleet{ - worker("w-malformed", "gvisor", "node-a", tierTwo, withCapacity(4000, 8<<30), withMaxActors(4), - func(w *ateapipb.Worker) { - w.Status.Allocated = &ateapipb.WorkerResources{ - Actors: 1, - Resources: &ateapipb.Resources{Limits: []*ateapipb.Limits{{Name: resources.ResourceCPU, Quantity: "not-a-quantity"}}}, - } - }), - worker("w-known", "gvisor", "node-b", tierTwo, withCapacity(4000, 8<<30), withMaxActors(4), - assignedFor("demo", "b", resources.CPUMemory(1000, 2<<30))), - }, - constraints: Constraints{}, - wantPod: "w-known", - }, { name: "breaks utilization ties using remaining actor slots", fleet: fleet{ @@ -585,12 +569,8 @@ func assignedFor(atespace, name string, took *ateapipb.Resources) func(*ateapipb if w.Status == nil { w.Status = &ateapipb.WorkerStatus{} } - allocated, err := resources.AddToAllocated(w.Status.Allocated, + w.Status.Allocated = resources.AddToAllocated(w.Status.Allocated, &ateapipb.ActorAssignment{ActorUid: atespace + "/" + name, Resources: took}, +1) - if err != nil { - panic(err) - } - w.Status.Allocated = allocated } } @@ -611,7 +591,8 @@ func withCapacity(cpuMilli, memBytes int64) func(*ateapipb.Worker) { if w.Status.Capacity == nil { w.Status.Capacity = &ateapipb.WorkerResources{} } - w.Status.Capacity.Resources = resources.CPUMemory(cpuMilli, memBytes) + w.Status.Capacity.CpuMillicores = cpuMilli + w.Status.Capacity.MemoryBytes = memBytes } } @@ -691,17 +672,3 @@ func TestSchedulePowerOfTwoChoicesShiftedSecondSample(t *testing.T) { t.Fatalf("Schedule() = %q, want %q", got.GetWorkerPod(), "w-2") } } - -func TestScheduleMalformedLimitsIsNotNoCapacity(t *testing.T) { - f := fleet{worker("w-1", "gvisor", "node-a", nil)} - constraints := Constraints{ - Limits: &ateapipb.Resources{Limits: []*ateapipb.Limits{{Name: resources.ResourceCPU, Quantity: "not-a-quantity"}}}, - } - _, err := New(f, WithIntn(firstIntn)).Schedule(context.Background(), constraints) - if err == nil { - t.Fatal("Schedule() error = nil, want a parse error") - } - if errors.Is(err, ErrNoCapacity) { - t.Fatalf("Schedule() error = %v, want an error other than ErrNoCapacity", err) - } -} diff --git a/cmd/ateapi/internal/store/atepg/worker_assignment.go b/cmd/ateapi/internal/store/atepg/worker_assignment.go index b7e49f489f..9439982d18 100644 --- a/cmd/ateapi/internal/store/atepg/worker_assignment.go +++ b/cmd/ateapi/internal/store/atepg/worker_assignment.go @@ -116,11 +116,7 @@ func (p *Persistence) BindActorToWorker(ctx context.Context, workerName string, return nil, err } } - allocated, err := resources.AddToAllocated(worker.Status.Allocated, assignment, +1) - if err != nil { - return nil, err - } - worker.Status.Allocated = allocated + worker.Status.Allocated = resources.AddToAllocated(worker.Status.Allocated, assignment, +1) if err := saveWorker(ctx, tx, worker); err != nil { return nil, err } @@ -138,11 +134,7 @@ func (p *Persistence) BindActorToWorker(ctx context.Context, workerName string, // Subtract before adding: the Actor is already counted, and its // declared size may have changed. - allocated, err := resources.AddToAllocated(worker.Status.Allocated, previous, -1) - if err != nil { - return nil, err - } - worker.Status.Allocated = allocated + worker.Status.Allocated = resources.AddToAllocated(worker.Status.Allocated, previous, -1) // Admit against the Worker without the old reservation. An // ActorTemplate is mutable, so a replacement can be larger than what @@ -153,10 +145,7 @@ func (p *Persistence) BindActorToWorker(ctx context.Context, workerName string, } } - if allocated, err = resources.AddToAllocated(allocated, assignment, +1); err != nil { - return nil, err - } - worker.Status.Allocated = allocated + worker.Status.Allocated = resources.AddToAllocated(worker.Status.Allocated, assignment, +1) // A rebind updates the assignment already recorded, so re-stamping it // as a create would make a retried claim look like a new subresource. @@ -214,11 +203,7 @@ func (p *Persistence) ReleaseActorFromWorker(ctx context.Context, workerName str if worker.Status == nil { worker.Status = &ateapipb.WorkerStatus{} } - allocated, err := resources.AddToAllocated(worker.Status.Allocated, assignment, -1) - if err != nil { - return nil, err - } - worker.Status.Allocated = allocated + worker.Status.Allocated = resources.AddToAllocated(worker.Status.Allocated, assignment, -1) if err := saveWorker(ctx, tx, worker); err != nil { return nil, err } diff --git a/cmd/ateapi/internal/store/storecontract/contract.go b/cmd/ateapi/internal/store/storecontract/contract.go index b5a8b8ce85..8cd37f1d8b 100644 --- a/cmd/ateapi/internal/store/storecontract/contract.go +++ b/cmd/ateapi/internal/store/storecontract/contract.go @@ -135,7 +135,7 @@ func newTestWorker(name, pod string) *ateapipb.Worker { WorkerPod: pod, WorkerPodUid: testWorkerPodUID, Status: &ateapipb.WorkerStatus{ - Capacity: &ateapipb.WorkerResources{Resources: resources.CPUMemory(2000, 4<<30)}, + Capacity: &ateapipb.WorkerResources{CpuMillicores: 2000, MemoryBytes: 4 << 30}, }, } } @@ -1882,13 +1882,13 @@ func runWorkerContractTests(t *testing.T, setup func(t *testing.T) store.Interfa defer wg.Done() // Claimants contend over the allocation total, which is what a // claim actually moves on the Worker record now that the - // assignments themselves are separate records. CpuMilli carries + // assignments themselves are separate records. The cpu limit carries // the claimant's number so the winner is identifiable. _, err := s.UpdateWorker(ctx, testWorkerName, store.PreconditionFrom(created), func(toUpdate *ateapipb.Worker) error { if toUpdate.GetStatus().GetAllocated().GetActors() > 0 { return errTaken } - toUpdate.Status.Allocated = &ateapipb.WorkerResources{Actors: 1, Resources: resources.CPUMemory(int64(i)+1, 0)} + toUpdate.Status.Allocated = &ateapipb.WorkerResources{Actors: 1, CpuMillicores: int64(i) + 1, MemoryBytes: 0} return nil }) switch { @@ -2183,7 +2183,7 @@ func runWorkerAssignmentContractTests(t *testing.T, setup func(t *testing.T) sto if err != nil { t.Fatalf("GetWorker failed: %v", err) } - want := &ateapipb.WorkerResources{Actors: 1, Resources: resources.CPUMemory(500, 1<<20)} + want := &ateapipb.WorkerResources{Actors: 1, CpuMillicores: 500, MemoryBytes: 1 << 20} if diff := cmp.Diff(want, worker.GetStatus().GetAllocated(), protocmp.Transform()); diff != "" { t.Errorf("allocated mismatch (-want +got):\n%s", diff) } @@ -2216,7 +2216,7 @@ func runWorkerAssignmentContractTests(t *testing.T, setup func(t *testing.T) sto if err != nil { t.Fatalf("GetWorker failed: %v", err) } - want := &ateapipb.WorkerResources{Actors: 1, Resources: resources.CPUMemory(250, 1<<21)} + want := &ateapipb.WorkerResources{Actors: 1, CpuMillicores: 250, MemoryBytes: 1 << 21} if diff := cmp.Diff(want, worker.GetStatus().GetAllocated(), protocmp.Transform()); diff != "" { t.Errorf("allocated mismatch after rebinding the same actor (-want +got):\n%s", diff) } @@ -2331,7 +2331,7 @@ func runWorkerAssignmentContractTests(t *testing.T, setup func(t *testing.T) sto if err != nil { t.Fatalf("GetWorker failed: %v", err) } - want := &ateapipb.WorkerResources{Actors: 1, Resources: resources.CPUMemory(500, 0)} + want := &ateapipb.WorkerResources{Actors: 1, CpuMillicores: 500, MemoryBytes: 0} if diff := cmp.Diff(want, worker.GetStatus().GetAllocated(), protocmp.Transform()); diff != "" { t.Errorf("a refused replacement changed the allocation (-want +got):\n%s", diff) } @@ -2418,7 +2418,7 @@ func runWorkerAssignmentContractTests(t *testing.T, setup func(t *testing.T) sto if err != nil { t.Fatalf("GetWorker failed: %v", err) } - want := &ateapipb.WorkerResources{Actors: 1, Resources: resources.CPUMemory(500, 1<<20)} + want := &ateapipb.WorkerResources{Actors: 1, CpuMillicores: 500, MemoryBytes: 1 << 20} if diff := cmp.Diff(want, first.GetStatus().GetAllocated(), protocmp.Transform()); diff != "" { t.Errorf("first worker's allocation changed (-want +got):\n%s", diff) } @@ -2482,15 +2482,12 @@ func runWorkerAssignmentContractTests(t *testing.T, setup func(t *testing.T) sto t.Fatalf("ListWorkerAssignments(%s) failed: %v", name, err) } assignments := assignmentsPage.Items - want, err := resources.SumAllocated(assignments) - if err != nil { - t.Fatalf("%s: summing assignments: %v", name, err) - } + want := resources.SumAllocated(assignments) // A Worker that never took anything has no allocation at all, while // a sum over no assignments is a zeroed message; the two say the // same thing, so compare what they mean rather than the messages. got := worker.GetStatus().GetAllocated() - if got.GetActors() != want.GetActors() || !proto.Equal(got.GetResources(), want.GetResources()) { + if !sameAllocation(got, want) { t.Errorf("%s: allocation %v disagrees with the %d assignments it holds (%v)", name, got, len(assignments), want) } @@ -2524,7 +2521,7 @@ func runWorkerAssignmentContractTests(t *testing.T, setup func(t *testing.T) sto } // The returned Worker is what the caller feeds the cache, so it has to // be the post-release state, not the copy that went in. - wantReleased := &ateapipb.WorkerResources{Actors: 1, Resources: resources.CPUMemory(250, 1<<21)} + wantReleased := &ateapipb.WorkerResources{Actors: 1, CpuMillicores: 250, MemoryBytes: 1 << 21} if diff := cmp.Diff(wantReleased, released.GetStatus().GetAllocated(), protocmp.Transform()); diff != "" { t.Errorf("returned worker's allocated mismatch (-want +got):\n%s", diff) } @@ -2540,7 +2537,7 @@ func runWorkerAssignmentContractTests(t *testing.T, setup func(t *testing.T) sto if err != nil { t.Fatalf("GetWorker failed: %v", err) } - want := &ateapipb.WorkerResources{Actors: 1, Resources: resources.CPUMemory(250, 1<<21)} + want := &ateapipb.WorkerResources{Actors: 1, CpuMillicores: 250, MemoryBytes: 1 << 21} if diff := cmp.Diff(want, worker.GetStatus().GetAllocated(), protocmp.Transform()); diff != "" { t.Errorf("allocated mismatch after release (-want +got):\n%s", diff) } @@ -2765,10 +2762,7 @@ func runWorkerAssignmentContractTests(t *testing.T, setup func(t *testing.T) sto t.Fatalf("ListWorkerAssignments failed: %v", err) } assignments := assignmentsPage.Items - want, err := resources.SumAllocated(assignments) - if err != nil { - t.Fatalf("summing assignments: %v", err) - } + want := resources.SumAllocated(assignments) if diff := cmp.Diff(want, worker.GetStatus().GetAllocated(), protocmp.Transform()); diff != "" { t.Fatalf("after step %d (actor %s) the total no longer matches the assignments (-want +got):\n%s", step, actorUID, diff) } @@ -2790,7 +2784,7 @@ func runWorkerAssignmentContractTests(t *testing.T, setup func(t *testing.T) sto if err != nil { t.Fatalf("GetWorker failed: %v", err) } - if got := worker.GetStatus().GetAllocated(); got.GetActors() != 0 || got.GetResources() != nil { + if got := worker.GetStatus().GetAllocated(); got.GetActors() != 0 || got.GetCpuMillicores() != 0 || got.GetMemoryBytes() != 0 { t.Errorf("after releasing everything the total is %v, want all zero", got) } }) @@ -3426,3 +3420,9 @@ func runUnknownFieldContractTests(t *testing.T, setup func(t *testing.T) store.I assertPruned(t, "DeleteWorker", created, deleted) }) } + +// sameAllocation compares two allocation totals by what they hold, so an +// absent total and a zeroed one are the same. +func sameAllocation(a, b *ateapipb.WorkerResources) bool { + return a.GetActors() == b.GetActors() && a.GetCpuMillicores() == b.GetCpuMillicores() && a.GetMemoryBytes() == b.GetMemoryBytes() +} diff --git a/cmd/ateapi/internal/workerservice/register_test.go b/cmd/ateapi/internal/workerservice/register_test.go index d6845a60c1..c1cdaf9cc4 100644 --- a/cmd/ateapi/internal/workerservice/register_test.go +++ b/cmd/ateapi/internal/workerservice/register_test.go @@ -26,7 +26,6 @@ import ( "go.ate.dev/substrate/cmd/ateapi/internal/ateletauth/ateletauthtest" "go.ate.dev/substrate/cmd/ateapi/internal/store/storetest" "go.ate.dev/substrate/internal/apierror" - "go.ate.dev/substrate/internal/resources" "go.ate.dev/substrate/pkg/proto/ateapipb" "google.golang.org/grpc/codes" ) @@ -62,7 +61,7 @@ func TestRegisterWorker(t *testing.T) { st, cleanup := storetest.SetupTestStore(t) defer cleanup() s := New(st, &fakeControl{}, testAteletSPIFFEID, nil, "", nil) - seedReportedWorker(t, st, testNode, &ateapipb.WorkerResources{Actors: 1, Resources: resources.CPUMemory(2000, 0)}) + seedReportedWorker(t, st, testNode, &ateapipb.WorkerResources{Actors: 1, CpuMillicores: 2000, MemoryBytes: 0}) got, err := s.RegisterWorker(ateletauthtest.ContextWith(ateletauthtest.CertOn(t, testNode)), setRequest(4094)) if err != nil { @@ -74,8 +73,8 @@ func TestRegisterWorker(t *testing.T) { // A report replaces what is recorded. The Worker reports everything it has, // so a dimension this one leaves out is one it no longer supplies -- keeping // the old value would advertise compute nothing claims to have. - if got := got.GetWorker().GetStatus().GetCapacity().GetResources(); got != nil { - t.Errorf("capacity resources = %v, want the report's own (none)", got) + if c := got.GetWorker().GetStatus().GetCapacity(); c.GetCpuMillicores() != 0 || c.GetMemoryBytes() != 0 { + t.Errorf("capacity = %v, want the report's own (no cpu or memory)", c) } } @@ -365,10 +364,7 @@ func TestRegisterWorker_RejectsNonsense(t *testing.T) { }{ {"negative ceiling", &ateapipb.WorkerResources{Actors: -1}}, {"int32 underflow", &ateapipb.WorkerResources{Actors: -2147483648}}, - {"negative quantity", &ateapipb.WorkerResources{Resources: resources.CPUMemory(-1, 0)}}, - {"unparseable quantity", &ateapipb.WorkerResources{ - Resources: &ateapipb.Resources{Limits: []*ateapipb.Limits{{Name: "cpu", Quantity: "lots"}}}, - }}, + {"negative quantity", &ateapipb.WorkerResources{CpuMillicores: -1, MemoryBytes: 0}}, } { t.Run(tc.name, func(t *testing.T) { _, err := s.RegisterWorker(authed, &ateapipb.RegisterWorkerRequest{ diff --git a/cmd/atecontroller/internal/controllers/gen.go b/cmd/atecontroller/internal/controllers/gen.go index c3e0b0ea52..ea7432a5bc 100644 --- a/cmd/atecontroller/internal/controllers/gen.go +++ b/cmd/atecontroller/internal/controllers/gen.go @@ -17,10 +17,12 @@ package controllers // RBAC needed by atecontroller components outside this package, which // controller-gen (paths="./...") does not scan: // - internal/workersync's pod informer lists and watches worker pods, and -// the syncer deletes worker pods that reach a terminal phase. +// the syncer deletes worker pods that reach a terminal phase or outlive +// their pool, and reads their ReplicaSets to tell the latter apart. // - internal/k8sresolver watches ateapi's EndpointSlices to dial it. // //+kubebuilder:rbac:groups=core,resources=pods,verbs=get;list;watch;delete +//+kubebuilder:rbac:groups=apps,resources=replicasets,verbs=get //+kubebuilder:rbac:groups=discovery.k8s.io,resources=endpointslices,verbs=get;list;watch,namespace=ate-system //go:generate bash ../../../../hack/gen-rbac.sh diff --git a/cmd/atecontroller/internal/workersync/syncer.go b/cmd/atecontroller/internal/workersync/syncer.go index b38595d078..7f485bc26b 100644 --- a/cmd/atecontroller/internal/workersync/syncer.go +++ b/cmd/atecontroller/internal/workersync/syncer.go @@ -33,6 +33,7 @@ import ( apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/wait" + appsv1client "k8s.io/client-go/kubernetes/typed/apps/v1" corev1client "k8s.io/client-go/kubernetes/typed/core/v1" "k8s.io/client-go/tools/cache" "k8s.io/client-go/util/workqueue" @@ -98,6 +99,7 @@ func (k workerKey) logAttrs() []any { type WorkerPoolSyncer struct { client ateapipb.ControlClient pods corev1client.PodsGetter + apps appsv1client.AppsV1Interface workerInformer cache.SharedIndexInformer workerPoolInformer cache.SharedIndexInformer queue workqueue.TypedRateLimitingInterface[workerKey] @@ -111,11 +113,13 @@ type WorkerPoolSyncer struct { } // NewWorkerPoolSyncer creates a new WorkerPoolSyncer. pods is used to delete -// worker pods that have reached a terminal phase. -func NewWorkerPoolSyncer(client ateapipb.ControlClient, pods corev1client.PodsGetter, workerInformer, workerPoolInformer cache.SharedIndexInformer, workerCount int) *WorkerPoolSyncer { +// worker pods that have reached a terminal phase or outlived their pool, and +// apps to tell the latter from pods the garbage collector is still removing. +func NewWorkerPoolSyncer(client ateapipb.ControlClient, pods corev1client.PodsGetter, apps appsv1client.AppsV1Interface, workerInformer, workerPoolInformer cache.SharedIndexInformer, workerCount int) *WorkerPoolSyncer { return &WorkerPoolSyncer{ client: client, pods: pods, + apps: apps, workerInformer: workerInformer, workerPoolInformer: workerPoolInformer, queue: workqueue.NewTypedRateLimitingQueue(workqueue.DefaultTypedControllerRateLimiter[workerKey]()), @@ -147,18 +151,8 @@ func (s *WorkerPoolSyncer) Start(ctx context.Context) { s.enqueuePod(newPod) }, DeleteFunc: func(obj interface{}) { - var pod *corev1.Pod - switch t := obj.(type) { - case *corev1.Pod: - pod = t - case cache.DeletedFinalStateUnknown: - var ok bool - pod, ok = t.Obj.(*corev1.Pod) - if !ok { - slog.ErrorContext(ctx, "Failed to cast DeletedFinalStateUnknown object to Pod") - return - } - default: + pod, ok := deletedObject[*corev1.Pod](obj) + if !ok { slog.ErrorContext(ctx, "Unknown object type in delete handler", slog.Any("obj", obj)) return } @@ -168,11 +162,16 @@ func (s *WorkerPoolSyncer) Start(ctx context.Context) { s.workerPoolInformer.AddEventHandler(cache.ResourceEventHandlerFuncs{ AddFunc: s.enqueueWorkerPool, UpdateFunc: func(_, obj interface{}) { s.enqueueWorkerPool(obj) }, + // Orphaned pods outlive their pool; reconcile them now, not at the + // next resync. + DeleteFunc: s.enqueueWorkerPool, }) go func() { defer s.queue.ShutDown() - if !cache.WaitForCacheSync(ctx.Done(), s.workerInformer.HasSynced) { + // The pool cache must be warm too: reconciling a pod against an empty + // one would read every pool as missing and deregister every Worker. + if !cache.WaitForCacheSync(ctx.Done(), s.workerInformer.HasSynced, s.workerPoolInformer.HasSynced) { slog.ErrorContext(ctx, "Syncer: failed to sync informer cache") return } @@ -192,9 +191,20 @@ func (s *WorkerPoolSyncer) Start(ctx context.Context) { }() } +// deletedObject returns the T that a delete event carries. An informer that +// missed the delete itself delivers a cache.DeletedFinalStateUnknown holding the +// last state it knew of, so that is unwrapped first. +func deletedObject[T any](obj interface{}) (T, bool) { + if tombstone, ok := obj.(cache.DeletedFinalStateUnknown); ok { + obj = tombstone.Obj + } + t, ok := obj.(T) + return t, ok +} + // enqueueWorkerPool schedules every current pod in pool. func (s *WorkerPoolSyncer) enqueueWorkerPool(obj interface{}) { - pool, ok := obj.(*atev1alpha1.WorkerPool) + pool, ok := deletedObject[*atev1alpha1.WorkerPool](obj) if !ok { slog.Error("Syncer: unexpected WorkerPool informer object", slog.Any("obj", obj)) return @@ -274,9 +284,10 @@ func (s *WorkerPoolSyncer) reconcile(ctx context.Context, key workerKey) error { } // A terminal pod is never Ready again, so left alone its Worker would stay // UNAVAILABLE, and the Actors bound to it registered, for as long as the pod - // object lingers. + // object lingers. The API server deletes a terminal pod without a grace + // period. if isPodTerminal(pod) { - return s.deleteTerminalPod(ctx, key, pod) + return s.deleteWorkerPod(ctx, key, pod, "phase "+string(pod.Status.Phase)) } return s.createOrUpdateWorker(ctx, key, pod) } @@ -307,7 +318,7 @@ func (s *WorkerPoolSyncer) createOrUpdateWorker(ctx context.Context, key workerK return fmt.Errorf("getting WorkerPool %s/%s: %w", key.namespace, poolName, err) } if !exists { - return fmt.Errorf("getting WorkerPool %s/%s: not found", key.namespace, poolName) + return s.reconcileMissingPool(ctx, key, pod, poolName) } pool, ok := poolObject.(*atev1alpha1.WorkerPool) if !ok { @@ -441,21 +452,20 @@ func isPodTerminal(pod *corev1.Pod) bool { return pod.Status.Phase == corev1.PodSucceeded || pod.Status.Phase == corev1.PodFailed } -// deleteTerminalPod deletes a worker pod that has reached a terminal phase. -// The API server deletes a terminal pod without a grace period, and the -// resulting Pod Deleted event deregisters the Worker and releases its Actors -// through reconcileDeadWorker. The Worker is marked DRAINING first so the -// scheduler stops routing to it even while a failed delete is being retried. +// deleteWorkerPod gracefully deletes a worker pod; the Pod Deleted event then +// deregisters the Worker through reconcileDeadWorker. The Worker is marked +// DRAINING first so the scheduler stops routing to it even if the delete is +// retried. // // The delete is preconditioned on the key's UID so it can never remove a // same-named replacement. A pod already gone, or replaced, is the state this // drives towards, so NotFound and Conflict are success. -func (s *WorkerPoolSyncer) deleteTerminalPod(ctx context.Context, key workerKey, pod *corev1.Pod) error { +func (s *WorkerPoolSyncer) deleteWorkerPod(ctx context.Context, key workerKey, pod *corev1.Pod, reason string) error { if err := s.markWorkerDraining(ctx, key); err != nil { return err } - slog.InfoContext(ctx, "Syncer: deleting worker pod (terminal phase)", - append(key.logAttrs(), slog.String("phase", string(pod.Status.Phase)))...) + slog.InfoContext(ctx, "Syncer: deleting worker pod", + append(key.logAttrs(), slog.String("reason", reason))...) uid := pod.UID err := s.pods.Pods(key.namespace).Delete(ctx, key.name, metav1.DeleteOptions{ Preconditions: &metav1.Preconditions{UID: &uid}, @@ -464,11 +474,85 @@ func (s *WorkerPoolSyncer) deleteTerminalPod(ctx context.Context, key workerKey, return nil } if err != nil { - return fmt.Errorf("deleting terminal pod: %w", err) + return fmt.Errorf("deleting pod (%s): %w", reason, err) } return nil } +// reconcileMissingPool handles a live pod whose WorkerPool is not in the cache. +// The Worker is never deleted outright, which would crash its Actors. A pod +// the garbage collector is still removing is left to it. An orphaned pod is +// deleted gracefully; a replacement from its ReplicaSet has no Worker, so is +// left alone. +func (s *WorkerPoolSyncer) reconcileMissingPool(ctx context.Context, key workerKey, pod *corev1.Pod, poolName string) error { + _, err := s.client.GetWorker(ctx, &ateapipb.GetWorkerRequest{Worker: key.workerRef()}) + if status.Code(err) == codes.NotFound { + return nil + } + if err != nil { + return fmt.Errorf("getting worker: %w", err) + } + pending, err := s.poolCleanupPending(ctx, pod) + if err != nil { + return err + } + if pending { + slog.InfoContext(ctx, "Syncer: leaving worker to cascading delete (pool not found)", + append(key.logAttrs(), slog.String("workerPool", key.namespace+"/"+poolName))...) + return nil + } + return s.deleteWorkerPod(ctx, key, pod, "orphaned from deleted pool "+poolName) +} + +// poolCleanupPending reports whether the garbage collector will still remove +// pod. It walks the controller chain pod -> ReplicaSet -> Deployment -> +// WorkerPool: a gone or deleting owner means collection is pending, a cut link +// means the pod was orphaned. +func (s *WorkerPoolSyncer) poolCleanupPending(ctx context.Context, pod *corev1.Pod) (bool, error) { + var obj metav1.Object = pod + for _, kind := range []string{"ReplicaSet", "Deployment"} { + ref := metav1.GetControllerOf(obj) + if ref == nil || ref.Kind != kind { + return false, nil + } + owner, err := s.getOwner(ctx, pod.Namespace, ref) + if err != nil { + return false, fmt.Errorf("getting %s %s/%s: %w", ref.Kind, pod.Namespace, ref.Name, err) + } + if owner == nil || owner.GetDeletionTimestamp() != nil { + return true, nil + } + obj = owner + } + ref := metav1.GetControllerOf(obj) + return ref != nil && ref.Kind == "WorkerPool", nil +} + +// getOwner reads the ReplicaSet or Deployment ref names, or nil if it is gone +// or was replaced under the same name. +func (s *WorkerPoolSyncer) getOwner(ctx context.Context, namespace string, ref *metav1.OwnerReference) (metav1.Object, error) { + var owner metav1.Object + var err error + switch ref.Kind { + case "ReplicaSet": + owner, err = s.apps.ReplicaSets(namespace).Get(ctx, ref.Name, metav1.GetOptions{}) + case "Deployment": + owner, err = s.apps.Deployments(namespace).Get(ctx, ref.Name, metav1.GetOptions{}) + default: + return nil, fmt.Errorf("unsupported owner kind %q", ref.Kind) + } + if apierrors.IsNotFound(err) { + return nil, nil + } + if err != nil { + return nil, err + } + if owner.GetUID() != ref.UID { + return nil, nil + } + return owner, nil +} + // podIPs returns the pod's IP addresses, one per IP family, in the order // Kubernetes reports them. func podIPs(pod *corev1.Pod) []string { diff --git a/cmd/atecontroller/internal/workersync/syncer_test.go b/cmd/atecontroller/internal/workersync/syncer_test.go index dcc37ab113..f331bfecba 100644 --- a/cmd/atecontroller/internal/workersync/syncer_test.go +++ b/cmd/atecontroller/internal/workersync/syncer_test.go @@ -29,6 +29,7 @@ import ( "go.ate.dev/substrate/pkg/proto/ateapipb" "google.golang.org/grpc/codes" "google.golang.org/grpc/status" + appsv1 "k8s.io/api/apps/v1" corev1 "k8s.io/api/core/v1" apierrors "k8s.io/apimachinery/pkg/api/errors" "k8s.io/apimachinery/pkg/api/resource" @@ -39,6 +40,7 @@ import ( "k8s.io/client-go/kubernetes/fake" k8stesting "k8s.io/client-go/testing" "k8s.io/client-go/tools/cache" + "k8s.io/utils/ptr" ) // testPodUID is the pod UID most syncer tests give their single worker pod. @@ -102,54 +104,62 @@ func registeredWorker(ns, poolName, podName, uid, ip string) *ateapipb.Worker { } } -// newWorkerPoolInformer builds the WorkerPool informer the syncer reads, and -// returns its indexer so a test can seed and mutate pools synchronously rather -// than starting a factory and waiting for a watch to deliver them. -func newWorkerPoolInformer(t *testing.T, initPools ...*atev1alpha1.WorkerPool) (cache.SharedIndexInformer, cache.Indexer) { - t.Helper() - //nolint:staticcheck // NewSimpleClientset is the only available fake clientset for versioned CRDs. - pools := externalversions.NewSharedInformerFactory(atefake.NewSimpleClientset(), 0).Api().V1alpha1().WorkerPools() - indexer := pools.Informer().GetIndexer() +// newWorkerPoolInformer builds the WorkerPool informer the syncer reads, backed +// by a fake clientset that holds initPools. Nothing is cached until the factory +// starts. +func newWorkerPoolInformer(initPools ...*atev1alpha1.WorkerPool) (*atefake.Clientset, externalversions.SharedInformerFactory, cache.SharedIndexInformer) { + objs := make([]runtime.Object, 0, len(initPools)) for _, pool := range initPools { - if err := indexer.Add(pool); err != nil { - t.Fatalf("seeding WorkerPool %s/%s: %v", pool.Namespace, pool.Name, err) - } + objs = append(objs, pool) } - return pools.Informer(), indexer + //nolint:staticcheck // NewSimpleClientset is the only available fake clientset for versioned CRDs. + client := atefake.NewSimpleClientset(objs...) + factory := externalversions.NewSharedInformerFactory(client, 0) + return client, factory, factory.Api().V1alpha1().WorkerPools().Informer() } // setupSyncerTest wires a running syncer to a fake Control API and a fake -// Kubernetes, driving it end to end through pod events and the queue. -func setupSyncerTest(t *testing.T, ctx context.Context, api *fakeControl, initPools ...*atev1alpha1.WorkerPool) *fake.Clientset { +// Kubernetes, driving it end to end through pod and pool events and the queue. +// It returns the fake Kubernetes and the fake clientset the pools live in. +func setupSyncerTest(t *testing.T, ctx context.Context, api *fakeControl, initPools ...*atev1alpha1.WorkerPool) (*fake.Clientset, *atefake.Clientset) { t.Helper() //nolint:staticcheck // NewSimpleClientset is what the informer machinery takes. fakeK8s := fake.NewSimpleClientset() workerFactory, workerInformer := WorkerPodInformer(fakeK8s) - workerPoolInformer, _ := newWorkerPoolInformer(t, initPools...) + poolClient, poolFactory, workerPoolInformer := newWorkerPoolInformer(initPools...) - // Start before the factory: the informer's initial list is what seeds the + // Start before the factories: the informer's initial list is what seeds the // queue with the pods that already exist. - NewWorkerPoolSyncer(api, fakeK8s.CoreV1(), workerInformer, workerPoolInformer, 2).Start(ctx) + NewWorkerPoolSyncer(api, fakeK8s.CoreV1(), fakeK8s.AppsV1(), workerInformer, workerPoolInformer, 2).Start(ctx) workerFactory.Start(ctx.Done()) + poolFactory.Start(ctx.Done()) workerFactory.WaitForCacheSync(ctx.Done()) + poolFactory.WaitForCacheSync(ctx.Done()) - return fakeK8s + return fakeK8s, poolClient } // setupReconcileTest builds a syncer whose pod and pool caches can be seeded // directly, for tests that drive reconcile synchronously without starting // factories or worker goroutines. It returns those caches alongside the syncer. -// The syncer's pod client is a fake clientset that the caches do not watch. +// The syncer's pod and apps clients are a fake clientset that the caches do not +// watch. func setupReconcileTest(t *testing.T, api *fakeControl, initPools ...*atev1alpha1.WorkerPool) (*WorkerPoolSyncer, cache.Indexer, cache.Indexer) { t.Helper() //nolint:staticcheck // NewSimpleClientset is what the informer machinery takes. fakeK8s := fake.NewSimpleClientset() _, workerInformer := WorkerPodInformer(fakeK8s) - workerPoolInformer, poolIndexer := newWorkerPoolInformer(t, initPools...) + _, _, workerPoolInformer := newWorkerPoolInformer() + poolIndexer := workerPoolInformer.GetIndexer() + for _, pool := range initPools { + if err := poolIndexer.Add(pool); err != nil { + t.Fatalf("seeding WorkerPool %s/%s: %v", pool.Namespace, pool.Name, err) + } + } - return NewWorkerPoolSyncer(api, fakeK8s.CoreV1(), workerInformer, workerPoolInformer, 2), workerInformer.GetIndexer(), poolIndexer + return NewWorkerPoolSyncer(api, fakeK8s.CoreV1(), fakeK8s.AppsV1(), workerInformer, workerPoolInformer, 2), workerInformer.GetIndexer(), poolIndexer } // seedPod puts a pod in the syncer's cache as though the informer had delivered @@ -191,7 +201,7 @@ func TestSyncer_Lifecycle(t *testing.T) { ns, podName, poolName := "ns-syncer-lifecycle", "worker-unit-1", "pool1" api := newFakeControl() - fakeK8s := setupSyncerTest(t, ctx, api, workerPool(ns, poolName, "gvisor", map[string]string{"foo": "bar"})) + fakeK8s, _ := setupSyncerTest(t, ctx, api, workerPool(ns, poolName, "gvisor", map[string]string{"foo": "bar"})) // A pod with no IP is not registered: there is nothing to route to yet. pod := workerPod(ns, podName, poolName, testPodUID, "") @@ -245,7 +255,7 @@ func TestSyncer_OmittedFields(t *testing.T) { ns, podName, poolName := "ns-syncer-omitted", "worker-unit-1", "pool1" api := newFakeControl() - fakeK8s := setupSyncerTest(t, ctx, api, workerPool(ns, poolName, "", nil)) + fakeK8s, _ := setupSyncerTest(t, ctx, api, workerPool(ns, poolName, "", nil)) pod := workerPod(ns, podName, poolName, testPodUID, "127.0.0.1") if _, err := fakeK8s.CoreV1().Pods(ns).Create(ctx, pod, metav1.CreateOptions{}); err != nil { @@ -272,7 +282,7 @@ func TestSyncer_DoesNotInferCapacityFromThePod(t *testing.T) { ns, podName, poolName := "ns-syncer-capacity", "worker-unit-1", "pool1" api := newFakeControl() - fakeK8s := setupSyncerTest(t, ctx, api, workerPool(ns, poolName, "gvisor", nil)) + fakeK8s, _ := setupSyncerTest(t, ctx, api, workerPool(ns, poolName, "gvisor", nil)) pod := workerPod(ns, podName, poolName, testPodUID, "127.0.0.1") pod.Spec.Containers = append(pod.Spec.Containers, corev1.Container{ @@ -662,36 +672,325 @@ func TestSyncer_SandboxClassDriftLeavesWorkerAlone(t *testing.T) { } } -// TestSyncer_RequeueOnMissingWorkerPool verifies that a pod whose WorkerPool is -// not yet in the lister is requeued rather than dropped, and converges once the -// pool appears. -func TestSyncer_RequeueOnMissingWorkerPool(t *testing.T) { +// TestSyncer_MissingWorkerPool verifies that a pod whose WorkerPool is not in +// the lister gets no Worker, and that the Worker comes once the pool does. +func TestSyncer_MissingWorkerPool(t *testing.T) { ctx := context.Background() - ns, podName, poolName := "ns-syncer-latepool", "worker-late-1", "pool-late" + ns, podName, poolName := "ns-syncer-missingpool", "worker-orphan-1", "pool-orphan" + pool := workerPool(ns, poolName, "gvisor", nil) api := newFakeControl() // The pod is there; its pool is not yet. s, pods, poolIndexer := setupReconcileTest(t, api) key := seedPod(t, pods, workerPod(ns, podName, poolName, testPodUID, "10.0.0.5")) - if err := s.reconcile(ctx, key); err == nil { - t.Fatal("reconcile succeeded with no WorkerPool, want an error so the key requeues") - } + mustReconcile(t, ctx, s, key) if got := api.names(); len(got) != 0 { t.Errorf("registry holds %v, want nothing written before the pool exists", got) } - if err := poolIndexer.Add(workerPool(ns, poolName, "gvisor", nil)); err != nil { + // The pool appears and enqueues its pods, which then register. + if err := poolIndexer.Add(pool); err != nil { t.Fatalf("adding pool: %v", err) } + s.enqueueWorkerPool(pool) + if got := s.queue.Len(); got != 1 { + t.Fatalf("queued workers = %d after the pool appeared, want 1", got) + } mustReconcile(t, ctx, s, key) - if got := api.get(testPodUID).GetSandboxClass(); got != "gvisor" { t.Errorf("worker sandbox class = %q, want gvisor", got) } } +// owned sets owner as the controller of obj. +func owned(obj, owner metav1.Object, kind string) { + obj.SetOwnerReferences([]metav1.OwnerReference{{ + Kind: kind, Name: owner.GetName(), UID: owner.GetUID(), Controller: ptr.To(true), + }}) +} + +// poolChain returns the Deployment and ReplicaSet that own pod, linked as the +// WorkerPool controller links them, ending at a WorkerPool of the given name. +func poolChain(pod *corev1.Pod, poolName string) (*appsv1.Deployment, *appsv1.ReplicaSet) { + dep := &appsv1.Deployment{ObjectMeta: metav1.ObjectMeta{Name: poolName, Namespace: pod.Namespace, UID: "dep-uid"}} + owned(dep, &metav1.ObjectMeta{Name: poolName, UID: "pool-uid"}, "WorkerPool") + rs := &appsv1.ReplicaSet{ObjectMeta: metav1.ObjectMeta{Name: poolName + "-abc", Namespace: pod.Namespace, UID: "rs-uid"}} + owned(rs, dep, "Deployment") + owned(pod, rs, "ReplicaSet") + return dep, rs +} + +// TestSyncer_DeletedWorkerPool verifies what happens to a registered Worker +// whose pod is live but whose WorkerPool is gone. The Worker is never deleted +// outright, which would crash its Actors: a pod the garbage collector is still +// removing is left alone, and one that outlives its pool is deleted gracefully. +func TestSyncer_DeletedWorkerPool(t *testing.T) { + ns, poolName, podName, ip := "ns-deleted-pool", "pool1", "worker-1", "10.0.0.5" + + for _, tc := range []struct { + name string + // shape edits the chain, which starts intact, before it is stored. + shape func(pod *corev1.Pod, rs *appsv1.ReplicaSet, dep *appsv1.Deployment) (store []runtime.Object) + // deleted is whether the pod is expected to be deleted. + deleted bool + }{ + { + name: "background cascade", + shape: func(_ *corev1.Pod, rs *appsv1.ReplicaSet, dep *appsv1.Deployment) []runtime.Object { + return []runtime.Object{rs, dep} + }, + }, + { + name: "background cascade, deployment collected", + shape: func(_ *corev1.Pod, rs *appsv1.ReplicaSet, _ *appsv1.Deployment) []runtime.Object { + return []runtime.Object{rs} + }, + }, + { + name: "background cascade, replicaset collected", + shape: func(*corev1.Pod, *appsv1.ReplicaSet, *appsv1.Deployment) []runtime.Object { + return nil + }, + }, + { + name: "replicaset replaced under its name", + shape: func(_ *corev1.Pod, rs *appsv1.ReplicaSet, dep *appsv1.Deployment) []runtime.Object { + rs.UID = "new-uid" + return []runtime.Object{rs, dep} + }, + }, + { + name: "deployment deleting, no longer owned by the pool", + shape: func(_ *corev1.Pod, rs *appsv1.ReplicaSet, dep *appsv1.Deployment) []runtime.Object { + dep.OwnerReferences = nil + dep.DeletionTimestamp = ptr.To(metav1.Now()) + return []runtime.Object{rs, dep} + }, + }, + { + name: "replicaset deleting, deployment orphaned", + shape: func(_ *corev1.Pod, rs *appsv1.ReplicaSet, dep *appsv1.Deployment) []runtime.Object { + dep.OwnerReferences = nil + rs.DeletionTimestamp = ptr.To(metav1.Now()) + return []runtime.Object{rs, dep} + }, + }, + { + name: "orphaned deployment", + deleted: true, + shape: func(_ *corev1.Pod, rs *appsv1.ReplicaSet, dep *appsv1.Deployment) []runtime.Object { + dep.OwnerReferences = nil + return []runtime.Object{rs, dep} + }, + }, + { + name: "orphaned replicaset", + deleted: true, + shape: func(_ *corev1.Pod, rs *appsv1.ReplicaSet, dep *appsv1.Deployment) []runtime.Object { + rs.OwnerReferences = nil + return []runtime.Object{rs, dep} + }, + }, + { + name: "orphaned pod", + deleted: true, + shape: func(pod *corev1.Pod, rs *appsv1.ReplicaSet, dep *appsv1.Deployment) []runtime.Object { + pod.OwnerReferences = nil + return []runtime.Object{rs, dep} + }, + }, + } { + t.Run(tc.name, func(t *testing.T) { + ctx := context.Background() + api := newFakeControl() + api.put(registeredWorker(ns, poolName, podName, testPodUID, ip)) + s, pods, _ := setupReconcileTest(t, api) + + pod := workerPod(ns, podName, poolName, testPodUID, ip) + dep, rs := poolChain(pod, poolName) + objs := tc.shape(pod, rs, dep) + //nolint:staticcheck // NewSimpleClientset is the fake the syncer's clients take. + fakeK8s := fake.NewSimpleClientset(append(objs, pod.DeepCopy())...) + s.pods, s.apps = fakeK8s.CoreV1(), fakeK8s.AppsV1() + mustReconcile(t, ctx, s, seedPod(t, pods, pod)) + + got := api.get(testPodUID) + if got == nil { + t.Fatal("worker deleted outright, want it left to the pod's own deletion") + } + _, err := fakeK8s.CoreV1().Pods(ns).Get(ctx, podName, metav1.GetOptions{}) + if gone := apierrors.IsNotFound(err); gone != tc.deleted { + t.Errorf("pod deleted = %v, want %v", gone, tc.deleted) + } + wantState := ateapipb.WorkerState_WORKER_STATE_ACTIVE + if tc.deleted { + wantState = ateapipb.WorkerState_WORKER_STATE_DRAINING + } + if state := got.GetStatus().GetState(); state != wantState { + t.Errorf("worker state = %v, want %v", state, wantState) + } + }) + } +} + +// TestSyncer_DeletedWorkerPool_NotReadyPod verifies that an orphaned pod that is +// not Ready is still cleaned up. +func TestSyncer_DeletedWorkerPool_NotReadyPod(t *testing.T) { + ctx := context.Background() + ns, poolName, podName, ip := "ns-deleted-pool-notready", "pool1", "worker-1", "10.0.0.5" + + api := newFakeControl() + api.put(registeredWorker(ns, poolName, podName, testPodUID, ip)) + s, pods, _ := setupReconcileTest(t, api) + + pod := workerPod(ns, podName, poolName, testPodUID, ip) + pod.Status.Conditions = nil + //nolint:staticcheck // NewSimpleClientset is the fake the syncer's clients take. + fakeK8s := fake.NewSimpleClientset(pod.DeepCopy()) + s.pods, s.apps = fakeK8s.CoreV1(), fakeK8s.AppsV1() + mustReconcile(t, ctx, s, seedPod(t, pods, pod)) + + if _, err := fakeK8s.CoreV1().Pods(ns).Get(ctx, podName, metav1.GetOptions{}); !apierrors.IsNotFound(err) { + t.Errorf("get pod = %v, want NotFound", err) + } + if got := api.get(testPodUID).GetStatus().GetState(); got != ateapipb.WorkerState_WORKER_STATE_DRAINING { + t.Errorf("worker state = %v, want DRAINING", got) + } +} + +// TestSyncer_DeletedWorkerPool_Errors verifies that a failed read of the pod's +// owners requeues the key and deletes nothing, rather than guessing the pod was +// orphaned. +func TestSyncer_DeletedWorkerPool_Errors(t *testing.T) { + ctx := context.Background() + ns, poolName, podName, ip := "ns-deleted-pool-err", "pool1", "worker-1", "10.0.0.5" + + api := newFakeControl() + api.put(registeredWorker(ns, poolName, podName, testPodUID, ip)) + s, pods, _ := setupReconcileTest(t, api) + + pod := workerPod(ns, podName, poolName, testPodUID, ip) + poolChain(pod, poolName) + //nolint:staticcheck // NewSimpleClientset is the fake the syncer's clients take. + fakeK8s := fake.NewSimpleClientset(pod.DeepCopy()) + fakeK8s.PrependReactor("get", "replicasets", func(k8stesting.Action) (bool, runtime.Object, error) { + return true, nil, errors.New("apiserver unavailable") + }) + s.pods, s.apps = fakeK8s.CoreV1(), fakeK8s.AppsV1() + + if err := s.reconcile(ctx, seedPod(t, pods, pod)); err == nil { + t.Fatal("reconcile succeeded despite failing to read the pod's owners, want an error so the key requeues") + } + if _, err := fakeK8s.CoreV1().Pods(ns).Get(ctx, podName, metav1.GetOptions{}); err != nil { + t.Errorf("get pod: %v, want it left alone", err) + } + if got := api.get(testPodUID).GetStatus().GetState(); got != ateapipb.WorkerState_WORKER_STATE_ACTIVE { + t.Errorf("worker state = %v, want ACTIVE", got) + } +} + +// TestSyncer_DeletedWorkerPool_ReplacementPod verifies that the pod a surviving +// ReplicaSet starts in place of an orphaned one is left alone: it has no Worker, +// so deleting it would only make the ReplicaSet start another. +func TestSyncer_DeletedWorkerPool_ReplacementPod(t *testing.T) { + ctx := context.Background() + ns, poolName, podName := "ns-deleted-pool-repl", "pool1", "worker-2" + + api := newFakeControl() + s, pods, _ := setupReconcileTest(t, api) + + pod := workerPod(ns, podName, poolName, otherPodUID, "10.0.0.6") + //nolint:staticcheck // NewSimpleClientset is the fake the syncer's clients take. + fakeK8s := fake.NewSimpleClientset(pod.DeepCopy()) + s.pods, s.apps = fakeK8s.CoreV1(), fakeK8s.AppsV1() + mustReconcile(t, ctx, s, seedPod(t, pods, pod)) + + if _, err := fakeK8s.CoreV1().Pods(ns).Get(ctx, podName, metav1.GetOptions{}); err != nil { + t.Errorf("get pod: %v, want it left alone", err) + } + if got := api.names(); len(got) != 0 { + t.Errorf("registry holds %v, want nothing written", got) + } +} + +// TestSyncer_OrphanedPodsOfDeletedPool_ViaInformer verifies that deleting a +// WorkerPool without deleting its pods, as kubectl delete --cascade=orphan +// does, shuts their Workers down through the pods' own deletion without waiting +// for a pod event. +func TestSyncer_OrphanedPodsOfDeletedPool_ViaInformer(t *testing.T) { + ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) + defer cancel() + + ns, podName, poolName := "ns-syncer-orphaned", "worker-orphaned-1", "pool1" + + api := newFakeControl() + fakeK8s, poolClient := setupSyncerTest(t, ctx, api, workerPool(ns, poolName, "gvisor", nil)) + + pod := workerPod(ns, podName, poolName, testPodUID, "10.0.0.7") + if _, err := fakeK8s.CoreV1().Pods(ns).Create(ctx, pod, metav1.CreateOptions{}); err != nil { + t.Fatalf("create pod: %v", err) + } + waitForWorker(t, ctx, api, testPodUID, func(w *ateapipb.Worker) bool { + return w.GetStatus().GetState() == ateapipb.WorkerState_WORKER_STATE_ACTIVE + }) + + if err := poolClient.ApiV1alpha1().WorkerPools(ns).Delete(ctx, poolName, metav1.DeleteOptions{}); err != nil { + t.Fatalf("delete pool: %v", err) + } + waitForWorker(t, ctx, api, testPodUID, func(w *ateapipb.Worker) bool { return w == nil }) + + if _, err := fakeK8s.CoreV1().Pods(ns).Get(ctx, podName, metav1.GetOptions{}); !apierrors.IsNotFound(err) { + t.Errorf("get pod = %v, want NotFound", err) + } +} + +// TestSyncer_StartWaitsForWorkerPoolCache verifies that no pod is reconciled +// before the WorkerPool cache has synced: against an empty cache every pool +// reads as missing, which would shut down every live Worker at startup. +func TestSyncer_StartWaitsForWorkerPoolCache(t *testing.T) { + ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) + defer cancel() + + ns, podName, poolName := "ns-syncer-poolsync", "worker-poolsync-1", "pool1" + poolLabels := map[string]string{"foo": "bar"} + + api := newFakeControl() + api.put(registeredWorker(ns, poolName, podName, testPodUID, "10.0.0.8")) + + //nolint:staticcheck // NewSimpleClientset is what the informer machinery takes. + fakeK8s := fake.NewSimpleClientset(workerPod(ns, podName, poolName, testPodUID, "10.0.0.8")) + workerFactory, workerInformer := WorkerPodInformer(fakeK8s) + _, poolFactory, workerPoolInformer := newWorkerPoolInformer(workerPool(ns, poolName, "", poolLabels)) + + NewWorkerPoolSyncer(api, fakeK8s.CoreV1(), fakeK8s.AppsV1(), workerInformer, workerPoolInformer, 2).Start(ctx) + workerFactory.Start(ctx.Done()) + workerFactory.WaitForCacheSync(ctx.Done()) + + // The pod cache is warm and the pool cache is not. There is no state to wait + // for, so this gives the syncer a window to get it wrong in. + err := wait.PollUntilContextTimeout(ctx, 20*time.Millisecond, 300*time.Millisecond, true, func(context.Context) (bool, error) { + if api.get(testPodUID).GetStatus().GetState() != ateapipb.WorkerState_WORKER_STATE_ACTIVE { + return false, errors.New("worker shut down before the WorkerPool cache synced") + } + return false, nil + }) + if !errors.Is(err, context.DeadlineExceeded) { + t.Fatalf("while the WorkerPool cache is cold: %v", err) + } + + // Once the pool cache syncs, the pod is reconciled against the pool, which + // copies the pool's labels onto the surviving Worker. + poolFactory.Start(ctx.Done()) + got := waitForWorker(t, ctx, api, testPodUID, func(w *ateapipb.Worker) bool { + return w != nil && maps.Equal(w.GetLabels(), poolLabels) + }) + if got == nil { + t.Fatal("worker gone after the WorkerPool cache synced") + } +} + // TestEnqueueWorkerPool verifies that a WorkerPool change requeues every pod // in that pool, so its Worker record receives updated labels promptly. func TestEnqueueWorkerPool(t *testing.T) { @@ -722,6 +1021,25 @@ func TestEnqueueWorkerPool(t *testing.T) { } } +// TestEnqueueWorkerPool_DeletedFinalStateUnknown verifies that a pool delete the +// informer missed, delivered as a tombstone, still requeues the pool's pods. +func TestEnqueueWorkerPool_DeletedFinalStateUnknown(t *testing.T) { + api := newFakeControl() + s, pods, _ := setupReconcileTest(t, api) + pool := workerPool("ns-pool-tombstone", "pool-a", "gvisor", nil) + seedPod(t, pods, workerPod(pool.Namespace, "worker-1", pool.Name, testPodUID, "10.0.0.1")) + + s.enqueueWorkerPool(cache.DeletedFinalStateUnknown{Key: pool.Namespace + "/" + pool.Name, Obj: pool}) + if got := s.queue.Len(); got != 1 { + t.Errorf("queued workers = %d, want 1", got) + } + + s.enqueueWorkerPool(cache.DeletedFinalStateUnknown{Key: "ns/other", Obj: &corev1.Pod{}}) + if got := s.queue.Len(); got != 1 { + t.Errorf("queued workers = %d after a tombstone of the wrong type, want it unchanged at 1", got) + } +} + // TestSyncer_SoftDelete_ViaInformer walks the whole transition through the // informer rather than seeding an already-registered worker: a pod is // registered ACTIVE, then enters graceful termination and flips to DRAINING @@ -733,7 +1051,7 @@ func TestSyncer_SoftDelete_ViaInformer(t *testing.T) { ns, podName, poolName := "ns-syncer-softdelete", "worker-soft-1", "pool1" api := newFakeControl() - fakeK8s := setupSyncerTest(t, ctx, api, workerPool(ns, poolName, "gvisor", nil)) + fakeK8s, _ := setupSyncerTest(t, ctx, api, workerPool(ns, poolName, "gvisor", nil)) pod := workerPod(ns, podName, poolName, testPodUID, "10.0.0.6") if _, err := fakeK8s.CoreV1().Pods(ns).Create(ctx, pod, metav1.CreateOptions{}); err != nil { @@ -840,7 +1158,7 @@ func TestSyncer_TerminalPod_ViaInformer(t *testing.T) { ns, podName, poolName := "ns-syncer-terminal", "worker-terminal-1", "pool1" api := newFakeControl() - fakeK8s := setupSyncerTest(t, ctx, api, workerPool(ns, poolName, "gvisor", nil)) + fakeK8s, _ := setupSyncerTest(t, ctx, api, workerPool(ns, poolName, "gvisor", nil)) pod := workerPod(ns, podName, poolName, testPodUID, "10.0.0.21") if _, err := fakeK8s.CoreV1().Pods(ns).Create(ctx, pod, metav1.CreateOptions{}); err != nil { diff --git a/cmd/atecontroller/main.go b/cmd/atecontroller/main.go index 6271ba6ade..4af6bb9c5a 100644 --- a/cmd/atecontroller/main.go +++ b/cmd/atecontroller/main.go @@ -322,7 +322,7 @@ func main() { // Start registers the informer event handlers, so it has to run before the // factory does: the initial list then synthesizes an Add for every pod that // already exists, and no explicit startup re-list is needed. - workersync.NewWorkerPoolSyncer(ateapiClient, k8sClient.CoreV1(), workerPodInformer, workerPoolInformer.Informer(), *workerSyncConcurrency).Start(runCtx) + workersync.NewWorkerPoolSyncer(ateapiClient, k8sClient.CoreV1(), k8sClient.AppsV1(), workerPodInformer, workerPoolInformer.Informer(), *workerSyncConcurrency).Start(runCtx) workerPodInformerFactory.Start(runCtx.Done()) ateFactory.Start(runCtx.Done()) diff --git a/cmd/atelet/internal/apivalidation/validate.go b/cmd/atelet/internal/apivalidation/validate.go index 47b737399a..0f07acd06d 100644 --- a/cmd/atelet/internal/apivalidation/validate.go +++ b/cmd/atelet/internal/apivalidation/validate.go @@ -79,13 +79,6 @@ func ValidateRegisterWorkerRequest(ctx context.Context, req *ateletpb.RegisterWo return Validate_RegisterWorkerRequest(ctx, operation.Operation{Type: operation.Create}, nil, req, nil) } -// ValidateCustom_Limits validates one limit with resources.ValidateLimit, the -// rule the control plane applies to its own Limits. Presence and uniqueness -// of names are enforced by tags. -func ValidateCustom_Limits(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *ateletpb.Limits) field.ErrorList { - return resources.ValidateLimit(fldPath, value.GetName(), value.GetQuantity()) -} - func ValidateCustom_EgressGateway_Address(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *string) field.ErrorList { return resources.ValidateHostPort(fldPath, *value) } diff --git a/cmd/atelet/internal/apivalidation/validation_test.go b/cmd/atelet/internal/apivalidation/validation_test.go index 35772d76e1..fff8d0b2b6 100644 --- a/cmd/atelet/internal/apivalidation/validation_test.go +++ b/cmd/atelet/internal/apivalidation/validation_test.go @@ -231,8 +231,7 @@ func TestValidateRunRequest(t *testing.T) { ActorTemplateName: "tmpl-1", Spec: &ateletpb.WorkloadSpec{Containers: []*ateletpb.Container{{Name: "worker", Image: testDigestImage}}}, SandboxRuntime: runtimeWithAttributes(1), - CpuMilli: 500, - MemoryBytes: 1 << 30, + Resources: &ateletpb.ResourceRequirements{Limits: &ateletpb.Resources{CpuMillicores: 500, MemoryBytes: 1 << 30}}, } for _, m := range mutate { m(r) @@ -346,25 +345,32 @@ func TestValidateRunRequest(t *testing.T) { field.Invalid(field.NewPath("egress_gateway", "address"), nil, ""), }, }, { - name: "negative cpu_milli", - obj: valid(func(r *ateletpb.RunRequest) { r.CpuMilli = -1 }), - want: field.ErrorList{field.Invalid(field.NewPath("cpu_milli"), nil, "").WithOrigin("minimum")}, + name: "negative cpu_millicores", + obj: valid(func(r *ateletpb.RunRequest) { + r.Resources = &ateletpb.ResourceRequirements{Limits: &ateletpb.Resources{CpuMillicores: -1}} + }), + want: field.ErrorList{field.Invalid(field.NewPath("resources", "limits", "cpu_millicores"), nil, "").WithOrigin("minimum")}, }, { - name: "cpu_milli at the bound", - obj: valid(func(r *ateletpb.RunRequest) { r.CpuMilli = 999999 }), + name: "cpu_millicores at the bound", + obj: valid(func(r *ateletpb.RunRequest) { + r.Resources = &ateletpb.ResourceRequirements{Limits: &ateletpb.Resources{CpuMillicores: 999999}} + }), }, { - name: "cpu_milli over the bound", - obj: valid(func(r *ateletpb.RunRequest) { r.CpuMilli = 1000000 }), - want: field.ErrorList{field.Invalid(field.NewPath("cpu_milli"), nil, "").WithOrigin("maximum")}, + name: "cpu_millicores over the bound", + obj: valid(func(r *ateletpb.RunRequest) { + r.Resources = &ateletpb.ResourceRequirements{Limits: &ateletpb.Resources{CpuMillicores: 1000000}} + }), + want: field.ErrorList{field.Invalid(field.NewPath("resources", "limits", "cpu_millicores"), nil, "").WithOrigin("maximum")}, }, { name: "negative memory_bytes", - obj: valid(func(r *ateletpb.RunRequest) { r.MemoryBytes = -1 }), - want: field.ErrorList{field.Invalid(field.NewPath("memory_bytes"), nil, "").WithOrigin("minimum")}, + obj: valid(func(r *ateletpb.RunRequest) { + r.Resources = &ateletpb.ResourceRequirements{Limits: &ateletpb.Resources{MemoryBytes: -1}} + }), + want: field.ErrorList{field.Invalid(field.NewPath("resources", "limits", "memory_bytes"), nil, "").WithOrigin("minimum")}, }, { name: "unset sizes are allowed", obj: valid(func(r *ateletpb.RunRequest) { - r.CpuMilli = 0 - r.MemoryBytes = 0 + r.Resources = nil }), }, { name: "missing sandbox_runtime", @@ -602,8 +608,7 @@ func TestValidateRestoreRequest(t *testing.T) { }, { name: "valid size and egress", obj: valid(func(r *ateletpb.RestoreRequest) { - r.CpuMilli = 999999 - r.MemoryBytes = 1 << 30 + r.Resources = &ateletpb.ResourceRequirements{Limits: &ateletpb.Resources{CpuMillicores: 999999, MemoryBytes: 1 << 30}} r.EgressGateway = &ateletpb.EgressGateway{Address: "10.0.0.1:15001"} }), }, { @@ -694,17 +699,23 @@ func TestValidateRestoreRequest(t *testing.T) { r.Snapshot.Fidelity = ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS }), }, { - name: "negative cpu_milli", - obj: valid(func(r *ateletpb.RestoreRequest) { r.CpuMilli = -1 }), - want: field.ErrorList{field.Invalid(field.NewPath("cpu_milli"), nil, "").WithOrigin("minimum")}, + name: "negative cpu_millicores", + obj: valid(func(r *ateletpb.RestoreRequest) { + r.Resources = &ateletpb.ResourceRequirements{Limits: &ateletpb.Resources{CpuMillicores: -1}} + }), + want: field.ErrorList{field.Invalid(field.NewPath("resources", "limits", "cpu_millicores"), nil, "").WithOrigin("minimum")}, }, { - name: "cpu_milli too large", - obj: valid(func(r *ateletpb.RestoreRequest) { r.CpuMilli = 1000000 }), - want: field.ErrorList{field.Invalid(field.NewPath("cpu_milli"), nil, "").WithOrigin("maximum")}, + name: "cpu_millicores too large", + obj: valid(func(r *ateletpb.RestoreRequest) { + r.Resources = &ateletpb.ResourceRequirements{Limits: &ateletpb.Resources{CpuMillicores: 1000000}} + }), + want: field.ErrorList{field.Invalid(field.NewPath("resources", "limits", "cpu_millicores"), nil, "").WithOrigin("maximum")}, }, { name: "negative memory_bytes", - obj: valid(func(r *ateletpb.RestoreRequest) { r.MemoryBytes = -1 }), - want: field.ErrorList{field.Invalid(field.NewPath("memory_bytes"), nil, "").WithOrigin("minimum")}, + obj: valid(func(r *ateletpb.RestoreRequest) { + r.Resources = &ateletpb.ResourceRequirements{Limits: &ateletpb.Resources{MemoryBytes: -1}} + }), + want: field.ErrorList{field.Invalid(field.NewPath("resources", "limits", "memory_bytes"), nil, "").WithOrigin("minimum")}, }, { name: "empty egress gateway address", obj: valid(func(r *ateletpb.RestoreRequest) { r.EgressGateway = &ateletpb.EgressGateway{} }), @@ -1012,13 +1023,13 @@ func TestValidateRegisterWorkerRequest(t *testing.T) { }, } } - withLimits := func(limits ...*ateletpb.Limits) *ateletpb.RegisterWorkerRequest { + withCapacity := func(c *ateletpb.WorkerResources) *ateletpb.RegisterWorkerRequest { return &ateletpb.RegisterWorkerRequest{ - Capacity: &ateletpb.WorkerResources{Resources: &ateletpb.Resources{Limits: limits}}, + Capacity: c, DefaultRuntime: testRuntime(), } } - limitsPath := field.NewPath("capacity", "resources", "limits") + capacityPath := field.NewPath("capacity") defaultRuntimePath := field.NewPath("default_runtime") restorablePath := field.NewPath("restorable_runtimes") @@ -1244,9 +1255,7 @@ func TestValidateRegisterWorkerRequest(t *testing.T) { }, { name: "full capacity", obj: &ateletpb.RegisterWorkerRequest{ - Capacity: &ateletpb.WorkerResources{Actors: 4, Resources: &ateletpb.Resources{ - Limits: []*ateletpb.Limits{{Name: "cpu", Quantity: "4"}, {Name: "memory", Quantity: "8Gi"}}, - }}, + Capacity: &ateletpb.WorkerResources{Actors: 4, CpuMillicores: 4000, MemoryBytes: 8 << 30}, DefaultRuntime: testRuntime(), }, }, { @@ -1254,51 +1263,17 @@ func TestValidateRegisterWorkerRequest(t *testing.T) { obj: &ateletpb.RegisterWorkerRequest{Capacity: &ateletpb.WorkerResources{Actors: -1}, DefaultRuntime: testRuntime()}, want: field.ErrorList{field.Invalid(field.NewPath("capacity", "actors"), nil, "").WithOrigin("minimum")}, }, { - name: "unsupported resource name", - obj: withLimits(&ateletpb.Limits{Name: "gpu", Quantity: "1"}), - want: field.ErrorList{field.NotSupported[string](limitsPath.Index(0).Child("name"), nil, nil)}, - }, { - name: "missing resource name", - obj: withLimits(&ateletpb.Limits{Quantity: "1"}), - want: field.ErrorList{ - field.Required(limitsPath.Index(0).Child("name"), ""), - field.NotSupported[string](limitsPath.Index(0).Child("name"), nil, nil), - }, + name: "negative cpu_millicores", + obj: withCapacity(&ateletpb.WorkerResources{CpuMillicores: -1}), + want: field.ErrorList{field.Invalid(capacityPath.Child("cpu_millicores"), nil, "").WithOrigin("minimum")}, }, { - name: "resource name too long", - obj: withLimits(&ateletpb.Limits{Name: strings.Repeat("x", 17), Quantity: "1"}), - want: field.ErrorList{ - field.TooLong(limitsPath.Index(0).Child("name"), nil, 16).WithOrigin("maxLength"), - field.NotSupported[string](limitsPath.Index(0).Child("name"), nil, nil), - }, + name: "cpu_millicores above the cap", + obj: withCapacity(&ateletpb.WorkerResources{CpuMillicores: 1000000}), + want: field.ErrorList{field.Invalid(capacityPath.Child("cpu_millicores"), nil, "").WithOrigin("maximum")}, }, { - name: "quantity too long", - obj: withLimits(&ateletpb.Limits{Name: "memory", Quantity: "1." + strings.Repeat("0", 31)}), - want: field.ErrorList{field.TooLong(limitsPath.Index(0).Child("quantity"), nil, 32).WithOrigin("maxLength")}, - }, { - name: "duplicate resource name", - obj: withLimits(&ateletpb.Limits{Name: "cpu", Quantity: "1"}, &ateletpb.Limits{Name: "cpu", Quantity: "2"}), - want: field.ErrorList{field.Duplicate(limitsPath.Index(1), nil)}, - }, { - name: "missing quantity", - obj: withLimits(&ateletpb.Limits{Name: "cpu"}), - want: field.ErrorList{field.Required(limitsPath.Index(0).Child("quantity"), "")}, - }, { - name: "malformed quantity", - obj: withLimits(&ateletpb.Limits{Name: "cpu", Quantity: "not-a-quantity"}), - want: field.ErrorList{field.Invalid(limitsPath.Index(0).Child("quantity"), nil, "")}, - }, { - name: "too many limits", - obj: withLimits( - &ateletpb.Limits{Name: "cpu", Quantity: "1"}, - &ateletpb.Limits{Name: "memory", Quantity: "1Gi"}, - &ateletpb.Limits{Name: "cpu", Quantity: "2"}, - ), - want: field.ErrorList{field.TooMany(limitsPath, 3, 2).WithOrigin("maxItems")}, - }, { - name: "nil limit entry", - obj: withLimits(nil), - want: field.ErrorList{field.Required(limitsPath.Index(0), "")}, + name: "negative memory_bytes", + obj: withCapacity(&ateletpb.WorkerResources{MemoryBytes: -1}), + want: field.ErrorList{field.Invalid(capacityPath.Child("memory_bytes"), nil, "").WithOrigin("minimum")}, }} for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { @@ -1376,9 +1351,9 @@ func TestValidateWorkloadSpec(t *testing.T) { }, { name: "container errors surface at the container's path", obj: valid(func(s *ateletpb.WorkloadSpec) { - s.Containers[0].Resources = &ateletpb.ResourceLimits{CpuMillis: -1} + s.Containers[0].Resources = &ateletpb.ResourceRequirements{Limits: &ateletpb.Resources{CpuMillicores: -1}} }), - want: field.ErrorList{field.Invalid(field.NewPath("containers").Index(0).Child("resources", "cpu_millis"), nil, "").WithOrigin("minimum")}, + want: field.ErrorList{field.Invalid(field.NewPath("containers").Index(0).Child("resources", "limits", "cpu_millicores"), nil, "").WithOrigin("minimum")}, }, { name: "mount references a declared volume", obj: valid(func(s *ateletpb.WorkloadSpec) { @@ -1690,7 +1665,7 @@ func TestValidateContainer(t *testing.T) { c.SecurityContext = &ateletpb.SecurityContext{Capabilities: &ateletpb.Capabilities{Add: add, Drop: drop}} }) } - limits := func(r *ateletpb.ResourceLimits) *ateletpb.Container { + limits := func(r *ateletpb.ResourceRequirements) *ateletpb.Container { return valid(func(c *ateletpb.Container) { c.Resources = r }) } @@ -1850,28 +1825,28 @@ func TestValidateContainer(t *testing.T) { want: field.ErrorList{field.Duplicate(capsPath.Child("add").Index(1), nil)}, }, - // ResourceLimits. + // Resources. { name: "valid limits", - obj: limits(&ateletpb.ResourceLimits{MemoryBytes: 1 << 30, CpuMillis: 500}), + obj: limits(&ateletpb.ResourceRequirements{Limits: &ateletpb.Resources{MemoryBytes: 1 << 30, CpuMillicores: 500}}), }, { name: "limits: zero means unset", - obj: limits(&ateletpb.ResourceLimits{}), + obj: limits(&ateletpb.ResourceRequirements{Limits: &ateletpb.Resources{}}), }, { name: "limits: negative memory_bytes", - obj: limits(&ateletpb.ResourceLimits{MemoryBytes: -1}), - want: field.ErrorList{field.Invalid(field.NewPath("resources", "memory_bytes"), nil, "").WithOrigin("minimum")}, + obj: limits(&ateletpb.ResourceRequirements{Limits: &ateletpb.Resources{MemoryBytes: -1}}), + want: field.ErrorList{field.Invalid(field.NewPath("resources", "limits", "memory_bytes"), nil, "").WithOrigin("minimum")}, }, { name: "limits: negative cpu_millis", - obj: limits(&ateletpb.ResourceLimits{CpuMillis: -1}), - want: field.ErrorList{field.Invalid(field.NewPath("resources", "cpu_millis"), nil, "").WithOrigin("minimum")}, + obj: limits(&ateletpb.ResourceRequirements{Limits: &ateletpb.Resources{CpuMillicores: -1}}), + want: field.ErrorList{field.Invalid(field.NewPath("resources", "limits", "cpu_millicores"), nil, "").WithOrigin("minimum")}, }, { name: "limits: cpu_millis at the cap", - obj: limits(&ateletpb.ResourceLimits{CpuMillis: 999999}), + obj: limits(&ateletpb.ResourceRequirements{Limits: &ateletpb.Resources{CpuMillicores: 999999}}), }, { name: "limits: cpu_millis above the cap", - obj: limits(&ateletpb.ResourceLimits{CpuMillis: 1000000}), - want: field.ErrorList{field.Invalid(field.NewPath("resources", "cpu_millis"), nil, "").WithOrigin("maximum")}, + obj: limits(&ateletpb.ResourceRequirements{Limits: &ateletpb.Resources{CpuMillicores: 1000000}}), + want: field.ErrorList{field.Invalid(field.NewPath("resources", "limits", "cpu_millicores"), nil, "").WithOrigin("maximum")}, }, } for _, tt := range tests { diff --git a/cmd/atelet/internal/apivalidation/zz_generated.validation.go b/cmd/atelet/internal/apivalidation/zz_generated.validation.go index b378dde702..d26f538dd1 100644 --- a/cmd/atelet/internal/apivalidation/zz_generated.validation.go +++ b/cmd/atelet/internal/apivalidation/zz_generated.validation.go @@ -934,7 +934,7 @@ func Validate_Container( { // field ateletpb.Container.Resources fn := func( fldPath *field.Path, - obj, oldObj *ateletpb.ResourceLimits, + obj, oldObj *ateletpb.ResourceRequirements, oldValueCorrelated bool) (errs field.ErrorList) { // don't revalidate unchanged data if oldValueCorrelated && op.Type == operation.Update { @@ -951,11 +951,11 @@ func Validate_Container( return // do not proceed } // call the type's validation function - errs = append(errs, Validate_ResourceLimits(ctx, op, fldPath, obj, oldObj)...) + errs = append(errs, Validate_ResourceRequirements(ctx, op, fldPath, obj, oldObj)...) return } oldVal := safe.Field(oldObj, - func(oldObj *ateletpb.Container) *ateletpb.ResourceLimits { + func(oldObj *ateletpb.Container) *ateletpb.ResourceRequirements { return oldObj.Resources }) errs = append(errs, fn(fldPath.Child("resources"), obj.Resources, oldVal, oldObj != nil)...) @@ -1532,84 +1532,6 @@ func Validate_ImageVolumeSource( return errs } -// Validate_Limits validates an instance of Limits according -// to declarative validation rules in the API schema. -func Validate_Limits( - ctx context.Context, op operation.Operation, fldPath *field.Path, - obj, oldObj *ateletpb.Limits) (errs field.ErrorList) { - - // custom validation - if e := ValidateCustom_Limits(ctx, op, fldPath, obj, oldObj); len(e) != 0 { - errs = append(errs, e...) - } - - { // field ateletpb.Limits.Name - fn := func( - fldPath *field.Path, - obj, oldObj *string, - oldValueCorrelated bool) (errs field.ErrorList) { - // don't revalidate unchanged data - if oldValueCorrelated && op.Type == operation.Update { - if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { - return nil - } - } - // call field-attached validations - earlyReturn := false - if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { - errs = append(errs, e...) - earlyReturn = true - } - if earlyReturn { - return // do not proceed - } - if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 16); len(e) != 0 { - errs = append(errs, e...) - } - return - } - oldVal := safe.Field(oldObj, - func(oldObj *ateletpb.Limits) *string { - return &oldObj.Name - }) - errs = append(errs, fn(fldPath.Child("name"), &obj.Name, oldVal, oldObj != nil)...) - } - - { // field ateletpb.Limits.Quantity - fn := func( - fldPath *field.Path, - obj, oldObj *string, - oldValueCorrelated bool) (errs field.ErrorList) { - // don't revalidate unchanged data - if oldValueCorrelated && op.Type == operation.Update { - if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { - return nil - } - } - // call field-attached validations - earlyReturn := false - if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { - errs = append(errs, e...) - earlyReturn = true - } - if earlyReturn { - return // do not proceed - } - if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 32); len(e) != 0 { - errs = append(errs, e...) - } - return - } - oldVal := safe.Field(oldObj, - func(oldObj *ateletpb.Limits) *string { - return &oldObj.Quantity - }) - errs = append(errs, fn(fldPath.Child("quantity"), &obj.Quantity, oldVal, oldObj != nil)...) - } - - return errs -} - // Validate_MintActorCertificateRequest validates an instance of MintActorCertificateRequest according // to declarative validation rules in the API schema. func Validate_MintActorCertificateRequest( @@ -2343,44 +2265,52 @@ func Validate_RequestActorSuspendRequest( return errs } -// Validate_ResourceLimits validates an instance of ResourceLimits according +// Validate_ResourceRequirements validates an instance of ResourceRequirements according // to declarative validation rules in the API schema. -func Validate_ResourceLimits( +func Validate_ResourceRequirements( ctx context.Context, op operation.Operation, fldPath *field.Path, - obj, oldObj *ateletpb.ResourceLimits) (errs field.ErrorList) { + obj, oldObj *ateletpb.ResourceRequirements) (errs field.ErrorList) { - { // field ateletpb.ResourceLimits.MemoryBytes + { // field ateletpb.ResourceRequirements.Limits fn := func( fldPath *field.Path, - obj, oldObj *int64, + obj, oldObj *ateletpb.Resources, oldValueCorrelated bool) (errs field.ErrorList) { // don't revalidate unchanged data if oldValueCorrelated && op.Type == operation.Update { - if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + if ateDeepEqual(obj, oldObj) { return nil } } // call field-attached validations earlyReturn := false - if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + if e := validate.OptionalPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { earlyReturn = true } if earlyReturn { return // do not proceed } - if e := validate.Minimum(ctx, op, fldPath, obj, oldObj, 0); len(e) != 0 { - errs = append(errs, e...) - } + // call the type's validation function + errs = append(errs, Validate_Resources(ctx, op, fldPath, obj, oldObj)...) return } oldVal := safe.Field(oldObj, - func(oldObj *ateletpb.ResourceLimits) *int64 { - return &oldObj.MemoryBytes + func(oldObj *ateletpb.ResourceRequirements) *ateletpb.Resources { + return oldObj.Limits }) - errs = append(errs, fn(fldPath.Child("memory_bytes"), &obj.MemoryBytes, oldVal, oldObj != nil)...) + errs = append(errs, fn(fldPath.Child("limits"), obj.Limits, oldVal, oldObj != nil)...) } - { // field ateletpb.ResourceLimits.CpuMillis + return errs +} + +// Validate_Resources validates an instance of Resources according +// to declarative validation rules in the API schema. +func Validate_Resources( + ctx context.Context, op operation.Operation, fldPath *field.Path, + obj, oldObj *ateletpb.Resources) (errs field.ErrorList) { + + { // field ateletpb.Resources.CpuMillicores fn := func( fldPath *field.Path, obj, oldObj *int64, @@ -2408,65 +2338,41 @@ func Validate_ResourceLimits( return } oldVal := safe.Field(oldObj, - func(oldObj *ateletpb.ResourceLimits) *int64 { - return &oldObj.CpuMillis + func(oldObj *ateletpb.Resources) *int64 { + return &oldObj.CpuMillicores }) - errs = append(errs, fn(fldPath.Child("cpu_millis"), &obj.CpuMillis, oldVal, oldObj != nil)...) + errs = append(errs, fn(fldPath.Child("cpu_millicores"), &obj.CpuMillicores, oldVal, oldObj != nil)...) } - return errs -} - -// Validate_Resources validates an instance of Resources according -// to declarative validation rules in the API schema. -func Validate_Resources( - ctx context.Context, op operation.Operation, fldPath *field.Path, - obj, oldObj *ateletpb.Resources) (errs field.ErrorList) { - - { // field ateletpb.Resources.Limits + { // field ateletpb.Resources.MemoryBytes fn := func( fldPath *field.Path, - obj, oldObj []*ateletpb.Limits, + obj, oldObj *int64, oldValueCorrelated bool) (errs field.ErrorList) { // don't revalidate unchanged data if oldValueCorrelated && op.Type == operation.Update { - if ateDeepEqual(obj, oldObj) { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { return nil } } // call field-attached validations earlyReturn := false - if e := validate.PtrSliceNoNils[ateletpb.Limits](ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { - errs = append(errs, e...) - earlyReturn = true - } - if e := validate.MaxItems(ctx, op, fldPath, obj, oldObj, 2).MarkShortCircuit(); len(e) != 0 { - errs = append(errs, e...) - earlyReturn = true - } - if e := validate.OptionalSlice(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { earlyReturn = true } if earlyReturn { return // do not proceed } - // lists with map semantics require unique keys - if e := validate.PtrSliceUnique(ctx, op, fldPath, obj, oldObj, - func(a *ateletpb.Limits, b *ateletpb.Limits) bool { return a.Name == b.Name }); len(e) != 0 { - errs = append(errs, e...) - } - // iterate the list and call the type's validation function - if e := validate.EachPtrSliceVal(ctx, op, fldPath, obj, oldObj, - func(a *ateletpb.Limits, b *ateletpb.Limits) bool { return a.Name == b.Name }, ateDeepEqual, Validate_Limits); len(e) != 0 { + if e := validate.Minimum(ctx, op, fldPath, obj, oldObj, 0); len(e) != 0 { errs = append(errs, e...) } return } oldVal := safe.Field(oldObj, - func(oldObj *ateletpb.Resources) []*ateletpb.Limits { - return oldObj.Limits + func(oldObj *ateletpb.Resources) *int64 { + return &oldObj.MemoryBytes }) - errs = append(errs, fn(fldPath.Child("limits"), obj.Limits, oldVal, oldObj != nil)...) + errs = append(errs, fn(fldPath.Child("memory_bytes"), &obj.MemoryBytes, oldVal, oldObj != nil)...) } return errs @@ -2760,69 +2666,34 @@ func Validate_RestoreRequest( errs = append(errs, fn(fldPath.Child("egress_gateway"), obj.EgressGateway, oldVal, oldObj != nil)...) } - { // field ateletpb.RestoreRequest.CpuMilli - fn := func( - fldPath *field.Path, - obj, oldObj *int64, - oldValueCorrelated bool) (errs field.ErrorList) { - // don't revalidate unchanged data - if oldValueCorrelated && op.Type == operation.Update { - if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { - return nil - } - } - // call field-attached validations - earlyReturn := false - if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { - earlyReturn = true - } - if earlyReturn { - return // do not proceed - } - if e := validate.Maximum(ctx, op, fldPath, obj, oldObj, 999999); len(e) != 0 { - errs = append(errs, e...) - } - if e := validate.Minimum(ctx, op, fldPath, obj, oldObj, 0); len(e) != 0 { - errs = append(errs, e...) - } - return - } - oldVal := safe.Field(oldObj, - func(oldObj *ateletpb.RestoreRequest) *int64 { - return &oldObj.CpuMilli - }) - errs = append(errs, fn(fldPath.Child("cpu_milli"), &obj.CpuMilli, oldVal, oldObj != nil)...) - } - - { // field ateletpb.RestoreRequest.MemoryBytes + { // field ateletpb.RestoreRequest.Resources fn := func( fldPath *field.Path, - obj, oldObj *int64, + obj, oldObj *ateletpb.ResourceRequirements, oldValueCorrelated bool) (errs field.ErrorList) { // don't revalidate unchanged data if oldValueCorrelated && op.Type == operation.Update { - if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + if ateDeepEqual(obj, oldObj) { return nil } } // call field-attached validations earlyReturn := false - if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + if e := validate.OptionalPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { earlyReturn = true } if earlyReturn { return // do not proceed } - if e := validate.Minimum(ctx, op, fldPath, obj, oldObj, 0); len(e) != 0 { - errs = append(errs, e...) - } + // call the type's validation function + errs = append(errs, Validate_ResourceRequirements(ctx, op, fldPath, obj, oldObj)...) return } oldVal := safe.Field(oldObj, - func(oldObj *ateletpb.RestoreRequest) *int64 { - return &oldObj.MemoryBytes + func(oldObj *ateletpb.RestoreRequest) *ateletpb.ResourceRequirements { + return oldObj.Resources }) - errs = append(errs, fn(fldPath.Child("memory_bytes"), &obj.MemoryBytes, oldVal, oldObj != nil)...) + errs = append(errs, fn(fldPath.Child("resources"), obj.Resources, oldVal, oldObj != nil)...) } { // field ateletpb.RestoreRequest.SandboxRuntime @@ -3116,69 +2987,34 @@ func Validate_RunRequest( errs = append(errs, fn(fldPath.Child("egress_gateway"), obj.EgressGateway, oldVal, oldObj != nil)...) } - { // field ateletpb.RunRequest.CpuMilli - fn := func( - fldPath *field.Path, - obj, oldObj *int64, - oldValueCorrelated bool) (errs field.ErrorList) { - // don't revalidate unchanged data - if oldValueCorrelated && op.Type == operation.Update { - if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { - return nil - } - } - // call field-attached validations - earlyReturn := false - if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { - earlyReturn = true - } - if earlyReturn { - return // do not proceed - } - if e := validate.Maximum(ctx, op, fldPath, obj, oldObj, 999999); len(e) != 0 { - errs = append(errs, e...) - } - if e := validate.Minimum(ctx, op, fldPath, obj, oldObj, 0); len(e) != 0 { - errs = append(errs, e...) - } - return - } - oldVal := safe.Field(oldObj, - func(oldObj *ateletpb.RunRequest) *int64 { - return &oldObj.CpuMilli - }) - errs = append(errs, fn(fldPath.Child("cpu_milli"), &obj.CpuMilli, oldVal, oldObj != nil)...) - } - - { // field ateletpb.RunRequest.MemoryBytes + { // field ateletpb.RunRequest.Resources fn := func( fldPath *field.Path, - obj, oldObj *int64, + obj, oldObj *ateletpb.ResourceRequirements, oldValueCorrelated bool) (errs field.ErrorList) { // don't revalidate unchanged data if oldValueCorrelated && op.Type == operation.Update { - if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + if ateDeepEqual(obj, oldObj) { return nil } } // call field-attached validations earlyReturn := false - if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + if e := validate.OptionalPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { earlyReturn = true } if earlyReturn { return // do not proceed } - if e := validate.Minimum(ctx, op, fldPath, obj, oldObj, 0); len(e) != 0 { - errs = append(errs, e...) - } + // call the type's validation function + errs = append(errs, Validate_ResourceRequirements(ctx, op, fldPath, obj, oldObj)...) return } oldVal := safe.Field(oldObj, - func(oldObj *ateletpb.RunRequest) *int64 { - return &oldObj.MemoryBytes + func(oldObj *ateletpb.RunRequest) *ateletpb.ResourceRequirements { + return oldObj.Resources }) - errs = append(errs, fn(fldPath.Child("memory_bytes"), &obj.MemoryBytes, oldVal, oldObj != nil)...) + errs = append(errs, fn(fldPath.Child("resources"), obj.Resources, oldVal, oldObj != nil)...) } { // field ateletpb.RunRequest.SandboxRuntime @@ -4416,34 +4252,69 @@ func Validate_WorkerResources( ctx context.Context, op operation.Operation, fldPath *field.Path, obj, oldObj *ateletpb.WorkerResources) (errs field.ErrorList) { - { // field ateletpb.WorkerResources.Resources + { // field ateletpb.WorkerResources.CpuMillicores fn := func( fldPath *field.Path, - obj, oldObj *ateletpb.Resources, + obj, oldObj *int64, oldValueCorrelated bool) (errs field.ErrorList) { // don't revalidate unchanged data if oldValueCorrelated && op.Type == operation.Update { - if ateDeepEqual(obj, oldObj) { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { return nil } } // call field-attached validations earlyReturn := false - if e := validate.OptionalPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { earlyReturn = true } if earlyReturn { return // do not proceed } - // call the type's validation function - errs = append(errs, Validate_Resources(ctx, op, fldPath, obj, oldObj)...) + if e := validate.Maximum(ctx, op, fldPath, obj, oldObj, 999999); len(e) != 0 { + errs = append(errs, e...) + } + if e := validate.Minimum(ctx, op, fldPath, obj, oldObj, 0); len(e) != 0 { + errs = append(errs, e...) + } return } oldVal := safe.Field(oldObj, - func(oldObj *ateletpb.WorkerResources) *ateletpb.Resources { - return oldObj.Resources + func(oldObj *ateletpb.WorkerResources) *int64 { + return &oldObj.CpuMillicores }) - errs = append(errs, fn(fldPath.Child("resources"), obj.Resources, oldVal, oldObj != nil)...) + errs = append(errs, fn(fldPath.Child("cpu_millicores"), &obj.CpuMillicores, oldVal, oldObj != nil)...) + } + + { // field ateletpb.WorkerResources.MemoryBytes + fn := func( + fldPath *field.Path, + obj, oldObj *int64, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.Minimum(ctx, op, fldPath, obj, oldObj, 0); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.WorkerResources) *int64 { + return &oldObj.MemoryBytes + }) + errs = append(errs, fn(fldPath.Child("memory_bytes"), &obj.MemoryBytes, oldVal, oldObj != nil)...) } { // field ateletpb.WorkerResources.Actors diff --git a/cmd/atelet/internal/filecache/evict.go b/cmd/atelet/internal/filecache/evict.go index a6354f4dd9..ae875f6352 100644 --- a/cmd/atelet/internal/filecache/evict.go +++ b/cmd/atelet/internal/filecache/evict.go @@ -26,23 +26,16 @@ import ( "strings" "syscall" "time" -) -// allocatedSize returns the bytes a file actually occupies on disk -// (st_blocks), not its logical length. The cache's biggest artifacts are -// sparse guest memory images, whose logical size can exceed their footprint -// by orders of magnitude, and every byte figure the store reports is -// compared against real disk usage (statfs watermarks, byte budgets). -func allocatedSize(info fs.FileInfo) int64 { - if st, ok := info.Sys().(*syscall.Stat_t); ok { - return st.Blocks * 512 - } - return info.Size() -} + "go.ate.dev/substrate/cmd/atelet/internal/sparsefile" +) // TotalBytes sums the allocated (on-disk) bytes of all published entries' -// regular files. It is the GC driver's usage measure against the store's -// byte budget. +// regular files (see sparsefile.AllocatedBytes), not their logical lengths: +// the cache's biggest artifacts are sparse guest memory images, and every byte +// figure the store reports is compared against real disk usage (statfs +// watermarks, byte budgets). It is the GC driver's usage measure against the +// store's byte budget. func (s *Store) TotalBytes(ctx context.Context) (int64, error) { var total int64 err := filepath.WalkDir(s.entriesDir(), func(path string, d fs.DirEntry, err error) error { @@ -66,7 +59,7 @@ func (s *Store) TotalBytes(ctx context.Context) (int64, error) { } return err } - total += allocatedSize(info) + total += sparsefile.AllocatedBytes(info) return nil }) if err != nil { @@ -105,7 +98,7 @@ type EvictStats struct { type evictCandidate struct { dir string // entry dir name (the key hash) mtime time.Time - size int64 // allocated bytes (see allocatedSize), not logical length + size int64 // allocated bytes (see sparsefile.AllocatedBytes), not logical length linked bool // data is a regular file some consumer still hard-links } @@ -305,7 +298,7 @@ func (s *Store) sizeEntry(entryDir string) (size int64, linked bool, err error) } return err } - size += allocatedSize(info) + size += sparsefile.AllocatedBytes(info) if path == dataPath || strings.HasPrefix(path, dataPath+string(filepath.Separator)) { if st, ok := info.Sys().(*syscall.Stat_t); ok && st.Nlink > 1 { linked = true diff --git a/cmd/atelet/internal/snapshottracker/background.go b/cmd/atelet/internal/snapshottracker/background.go index 4cc4945841..eef10454c6 100644 --- a/cmd/atelet/internal/snapshottracker/background.go +++ b/cmd/atelet/internal/snapshottracker/background.go @@ -24,8 +24,9 @@ import ( ) // Background work settings. All but the Default ones, which atelet's -// --snapshot-upload-delay, --snapshot-detach-delay, --snapshot-upload-workers, -// and --snapshot-detach-workers override, are fixed until they become flags or +// --snapshot-upload-delay, --snapshot-detach-delay, --snapshot-upload-timeout, +// --snapshot-upload-est-throughput-mib, --snapshot-upload-workers, and +// --snapshot-detach-workers override, are fixed until they become flags or // per-request settings on UploadConfig. const ( // DefaultUploadDelay is how long by default after the oldest @@ -35,6 +36,22 @@ const ( // DefaultDetachDelay is how long an actor stays suspended on this node by // default, its latest snapshot kept locally, before atelet detaches it. DefaultDetachDelay = 30 * time.Minute + // DefaultUploadTimeout is how long one background upload attempt, + // including the one a background detach runs first, may run by default, + // on top of the time its snapshot takes at the estimated upload + // throughput, before atelet cancels it. It keeps a wedged object-store + // call from holding a background worker forever. + DefaultUploadTimeout = 2 * time.Minute + // DefaultUploadEstThroughput, in bytes per second of a snapshot's + // allocated local size, is how fast atelet expects a background upload to + // run by default. Each attempt's timeout allows for the snapshot to upload + // at this rate, so a large snapshot is not cut off. + DefaultUploadEstThroughput = 100 << 20 + // maxUploadTimeoutDoublings caps how many times consecutive failures of + // an actor's background work double its next upload attempt's timeout: + // at most 8 times the first attempt's, so an upload slower than the + // estimated throughput still finishes. + maxUploadTimeoutDoublings = 3 // snapshotRetryDelay is how long atelet holds an actor's background work // back after a failure. Each further consecutive failure doubles it, up to // maxSnapshotRetryDelay. diff --git a/cmd/atelet/internal/snapshottracker/background_test.go b/cmd/atelet/internal/snapshottracker/background_test.go index 18facfc447..cba3521552 100644 --- a/cmd/atelet/internal/snapshottracker/background_test.go +++ b/cmd/atelet/internal/snapshottracker/background_test.go @@ -441,3 +441,86 @@ func TestBackgroundDetachFailureLogsWarning(t *testing.T) { t.Errorf("logged %v, want %v", got, want) } } + +// fakeClockTimeout's contexts end with DeadlineExceeded when the fake clock +// reaches their deadline, and with Canceled when the parent or the returned +// cancel func ends first. +func TestFakeClockTimeout(t *testing.T) { + t.Run("clock reaches the deadline", func(t *testing.T) { + clk := clocktesting.NewFakeClock(time.Unix(1_700_000_000, 0)) + ctx, cancel := fakeClockTimeout(clk)(context.Background(), time.Minute) + defer cancel() + if got, ok := ctx.Deadline(); !ok || !got.Equal(clk.Now().Add(time.Minute)) { + t.Fatalf("Deadline() = (%v, %v), want %v", got, ok, clk.Now().Add(time.Minute)) + } + clk.Step(time.Minute) + waitClosed(t, ctx.Done(), "fake clock timeout") + if got := ctx.Err(); !errors.Is(got, context.DeadlineExceeded) { + t.Errorf("Err() = %v, want DeadlineExceeded", got) + } + }) + + t.Run("parent canceled first", func(t *testing.T) { + clk := clocktesting.NewFakeClock(time.Unix(1_700_000_000, 0)) + parent, cancelParent := context.WithCancel(context.Background()) + ctx, cancel := fakeClockTimeout(clk)(parent, time.Minute) + defer cancel() + cancelParent() + waitClosed(t, ctx.Done(), "parent cancel") + if got := ctx.Err(); !errors.Is(got, context.Canceled) { + t.Errorf("Err() = %v, want Canceled", got) + } + }) + + t.Run("canceled first", func(t *testing.T) { + clk := clocktesting.NewFakeClock(time.Unix(1_700_000_000, 0)) + ctx, cancel := fakeClockTimeout(clk)(context.Background(), time.Minute) + cancel() + waitClosed(t, ctx.Done(), "cancel") + if got := ctx.Err(); !errors.Is(got, context.Canceled) { + t.Errorf("Err() = %v, want Canceled", got) + } + }) +} + +// Outside tests, a background upload attempt times out on real time, whatever +// clock the tracker measures due times on. +func TestUploadTimeoutRunsOnRealTime(t *testing.T) { + tr := New(t.Context(), nil, clocktesting.NewFakeClock(time.Unix(1_700_000_000, 0))) + ctx, cancel := tr.withTimeout(t.Context(), time.Hour) + defer cancel() + if got, ok := ctx.Deadline(); !ok || time.Until(got) < 59*time.Minute { + t.Errorf("Deadline() = (%v, %v), want an hour from now", got, ok) + } +} + +// A timed-out background upload logs a warning rather than an info +// cancellation, because only the upload attempt timed out while the operation +// context itself was not canceled. +func TestBackgroundUploadTimeoutLogsWarning(t *testing.T) { + store := newFakeSnapshotStore() + store.setBlocking(true) + tr := newTestTracker(t, store) + + h := &recordingHandler{} + defaultLogger := slog.Default() + slog.SetDefault(slog.New(h)) + t.Cleanup(func() { slog.SetDefault(defaultLogger) }) + + tr.CommitLocal(trackedActorUID, testRef("s1", 1, tr.clock.Now())) + done := startUpload(t, tr, store) + stepTo(tr, tr.clock.Now().Add(testUploadTimeout)) + waitClosed(t, done, "the timed-out upload") + + h.mu.Lock() + defer h.mu.Unlock() + var got []string + for _, r := range h.records { + if strings.HasPrefix(r.Message, "Snapshot operation") { + got = append(got, r.Level.String()+" "+r.Message) + } + } + if want := []string{"WARN Snapshot operation failed"}; !slices.Equal(got, want) { + t.Errorf("logged %v, want %v", got, want) + } +} diff --git a/cmd/atelet/internal/snapshottracker/detach_test.go b/cmd/atelet/internal/snapshottracker/detach_test.go index c8d2a0eea3..5c01ad25af 100644 --- a/cmd/atelet/internal/snapshottracker/detach_test.go +++ b/cmd/atelet/internal/snapshottracker/detach_test.go @@ -893,3 +893,56 @@ func TestTrackerDetachActsForTheActor(t *testing.T) { } }) } + +// A background detach whose upload runs past the upload timeout fails, backs +// off, and keeps the local snapshot to retry; a foreground Detach RPC is +// bounded by its caller's context instead. +func TestTrackerDetachUploadTimeout(t *testing.T) { + t.Run("background", func(t *testing.T) { + store := newFakeSnapshotStore() + store.setBlocking(true) + tr := newIdleTestTracker(t, store) + setWorkers(tr, 0, DefaultDetachWorkers) + tr.Start() + tr.CommitLocal(trackedActorUID, testRef("s1", 1, tr.clock.Now())) + due := detachDueOf(tr, trackedActorUID) + stepTo(tr, due) + waitStarted(t, store) + done := opDone(t, tr, trackedActorUID) + + stepTo(tr, tr.clock.Now().Add(testUploadTimeout)) + waitClosed(t, done, "the timed-out background detach") + + retryAt := tr.clock.Now().Add(testRetryDelay) + if got := detachDueOf(tr, trackedActorUID); !got.Equal(retryAt) { + t.Errorf("detach due after timeout = %v, want %v", got, retryAt) + } + if got := uuidOf(tr.State(trackedActorUID).GetLatestLocal()); got != "s1" { + t.Errorf("latest_local after timeout = %q, want s1 kept", got) + } + + store.setBlocking(false) + detachNow(t, tr) + if got := tr.State(trackedActorUID).GetLatestLocal(); got != nil { + t.Errorf("latest_local after retry = %v, want unset", got) + } + }) + + t.Run("foreground ignores background upload timeout", func(t *testing.T) { + store := newFakeSnapshotStore() + store.setBlocking(true) + tr := newTestTracker(t, store) + tr.CommitLocal(trackedActorUID, testRef("s1", 1, tr.clock.Now())) + + detached := startDetach(t, tr) + waitStarted(t, store) + if d := store.lastDeadline(); !d.IsZero() { + t.Errorf("foreground Detach's upload has deadline %v, want none beyond its caller's", d) + } + + store.release <- nil + if err := <-detached; err != nil { + t.Fatalf("Detach: %v", err) + } + }) +} diff --git a/cmd/atelet/internal/snapshottracker/localcheckpoints.go b/cmd/atelet/internal/snapshottracker/localcheckpoints.go index dd6648d17c..cee515d2d6 100644 --- a/cmd/atelet/internal/snapshottracker/localcheckpoints.go +++ b/cmd/atelet/internal/snapshottracker/localcheckpoints.go @@ -18,12 +18,14 @@ import ( "context" "errors" "fmt" + "io/fs" "log/slog" "os" "path/filepath" "slices" "go.ate.dev/substrate/cmd/atelet/internal/ateletpath" + "go.ate.dev/substrate/cmd/atelet/internal/sparsefile" ) // PruneLocalCheckpoints removes every local snapshot of the actor except those @@ -77,3 +79,26 @@ func removeLocalCheckpoint(ctx context.Context, actorUID, name string) error { slog.InfoContext(ctx, "removed local checkpoint", slog.String("path", path)) return nil } + +// localSnapshotSize returns the disk space allocated to the files of one local +// snapshot of the actor, nested ones included. Allocated blocks track the data +// an upload of the snapshot sends, where apparent sizes would count a sparse +// memory image up to the guest RAM ceiling. A missing snapshot, or a file that +// vanishes during the walk, counts nothing. On error it returns what it +// measured so far. +func localSnapshotSize(actorUID, name string) (int64, error) { + var size int64 + err := filepath.WalkDir(ateletpath.LocalSnapshotDir(actorUID, name), func(_ string, d fs.DirEntry, err error) error { + if err == nil && d.Type().IsRegular() { + var info fs.FileInfo + if info, err = d.Info(); err == nil { + size += sparsefile.AllocatedBytes(info) + } + } + if errors.Is(err, fs.ErrNotExist) { + return nil + } + return err + }) + return size, err +} diff --git a/cmd/atelet/internal/snapshottracker/localcheckpoints_test.go b/cmd/atelet/internal/snapshottracker/localcheckpoints_test.go index 350321c652..780b1684b7 100644 --- a/cmd/atelet/internal/snapshottracker/localcheckpoints_test.go +++ b/cmd/atelet/internal/snapshottracker/localcheckpoints_test.go @@ -16,10 +16,13 @@ package snapshottracker import ( "context" + "math/rand/v2" "os" "path/filepath" "strings" "testing" + + "go.ate.dev/substrate/cmd/atelet/internal/ateletpath" ) func writeSnapshotDir(t *testing.T, dir, prefix string) { @@ -33,6 +36,54 @@ func writeSnapshotDir(t *testing.T, dir, prefix string) { } } +// writeDenseFile writes n bytes of incompressible data to path, so all of them +// are allocated on disk. +func writeDenseFile(t *testing.T, path string, n int) { + t.Helper() + data := make([]byte, n) + _, _ = rand.NewChaCha8([32]byte{}).Read(data) + if err := os.MkdirAll(filepath.Dir(path), 0o700); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(path, data, 0o600); err != nil { + t.Fatal(err) + } +} + +// localSnapshotSize counts the disk space allocated to every file of a local +// snapshot, nested ones included, so a sparse image counts only its data, and +// a missing snapshot counts nothing. +func TestLocalSnapshotSize(t *testing.T) { + useTempNodeDirs(t) + if got, err := localSnapshotSize(trackedActorUID, "s1"); got != 0 || err != nil { + t.Fatalf("localSnapshotSize of a missing snapshot = %d, %v; want 0, nil", got, err) + } + + dir := ateletpath.LocalSnapshotDir(trackedActorUID, "s1") + const dense = 96 << 10 + writeDenseFile(t, filepath.Join(dir, "pages.img"), 64<<10) + writeDenseFile(t, filepath.Join(dir, "fs", "multitar.img"), 32<<10) + // A sparse image as large as a guest's RAM, with no data in it. + sparse, err := os.Create(filepath.Join(dir, "memory.img")) + if err != nil { + t.Fatal(err) + } + if err := sparse.Truncate(1 << 30); err != nil { + t.Fatal(err) + } + if err := sparse.Close(); err != nil { + t.Fatal(err) + } + + got, err := localSnapshotSize(trackedActorUID, "s1") + if err != nil { + t.Fatalf("localSnapshotSize: %v", err) + } + if got < dense || got > 2*dense { + t.Errorf("localSnapshotSize = %d, want about %d, the dense files' size: the sparse image's holes must not count", got, dense) + } +} + func TestPruneRemovesEverySnapshot(t *testing.T) { dir := t.TempDir() writeSnapshotDir(t, dir, "pause-1") diff --git a/cmd/atelet/internal/snapshottracker/metrics.go b/cmd/atelet/internal/snapshottracker/metrics.go index 914f17b041..a2db45a833 100644 --- a/cmd/atelet/internal/snapshottracker/metrics.go +++ b/cmd/atelet/internal/snapshottracker/metrics.go @@ -71,10 +71,16 @@ func classifyOverdueBand(overdue time.Duration) overdueBand { } } -// opDurationBuckets cover background uploads and detaches: deleting local -// snapshots or uploading a small volumes snapshot takes milliseconds, while -// uploading a multi-GiB memory snapshot runs for tens of seconds. -var opDurationBuckets = []float64{0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 20, 30, 60} +// uploadDurationBuckets cover background uploads: uploading a small volumes +// snapshot takes milliseconds, uploading a multi-GiB memory snapshot runs for +// tens of seconds, and an upload of a large snapshot can run for minutes +// before its timeout, which is at least --snapshot-upload-timeout. +var uploadDurationBuckets = []float64{0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 20, 30, 60, 120, 300, 600} + +// detachDurationBuckets cover deleting an actor's local snapshots during a +// background detach, which takes milliseconds to seconds. The upload a detach +// runs first is recorded on the upload histogram. +var detachDurationBuckets = []float64{0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 20, 30, 60} // overdueBuckets cover how late background work starts: within 100ms of its // due time while a worker is free, and up to an hour late while every worker @@ -103,7 +109,7 @@ func NewInstruments(meter metric.Meter) (*Instruments, error) { uploadDurationMetric, metric.WithUnit("s"), metric.WithDescription("Duration of one background upload of a local snapshot to object storage on atelet."), - metric.WithExplicitBucketBoundaries(opDurationBuckets...), + metric.WithExplicitBucketBoundaries(uploadDurationBuckets...), ) if err != nil { return nil, fmt.Errorf("create %s histogram: %w", uploadDurationMetric, err) @@ -112,7 +118,7 @@ func NewInstruments(meter metric.Meter) (*Instruments, error) { detachDurationMetric, metric.WithUnit("s"), metric.WithDescription("Duration of deleting an actor's local snapshots during a background detach on atelet, excluding any snapshot upload."), - metric.WithExplicitBucketBoundaries(opDurationBuckets...), + metric.WithExplicitBucketBoundaries(detachDurationBuckets...), ) if err != nil { return nil, fmt.Errorf("create %s histogram: %w", detachDurationMetric, err) @@ -243,10 +249,10 @@ func outcomeAttrs(ctx context.Context, target SnapshotRef, err error) []attribut return attrs } -// errorType returns the error.type of a failed upload or detach: Canceled -// once ctx has ended, as when Terminate or an atelet shutdown cancels the -// operation, and otherwise the code an apierror or a plugin's gRPC status -// carries, or Internal. +// errorType returns the error.type of a failed upload or detach: Canceled or +// DeadlineExceeded once ctx has ended (when Terminate, an atelet shutdown, or +// the background upload timeout ends it), and otherwise the code an apierror +// or a plugin's gRPC status carries, or Internal. func errorType(ctx context.Context, err error) string { if ctxErr := ctx.Err(); ctxErr != nil { return status.FromContextError(ctxErr).Code().String() diff --git a/cmd/atelet/internal/snapshottracker/metrics_test.go b/cmd/atelet/internal/snapshottracker/metrics_test.go index d30c9b2f4f..0accc34b07 100644 --- a/cmd/atelet/internal/snapshottracker/metrics_test.go +++ b/cmd/atelet/internal/snapshottracker/metrics_test.go @@ -258,6 +258,8 @@ func TestUploadDurationSkipsNonUploadingPaths(t *testing.T) { func TestErrorType(t *testing.T) { canceled, cancel := context.WithCancel(context.Background()) cancel() + expired, cancelExpired := context.WithDeadline(context.Background(), time.Unix(0, 0)) + defer cancelExpired() for _, tc := range []struct { name string ctx context.Context @@ -270,6 +272,12 @@ func TestErrorType(t *testing.T) { err: fmt.Errorf("while uploading snapshot files: %w", status.Error(codes.Canceled, "context canceled")), want: codes.Canceled, }, + { + name: "deadline exceeded", + ctx: expired, + err: context.DeadlineExceeded, + want: codes.DeadlineExceeded, + }, { name: "apierror", ctx: context.Background(), @@ -787,3 +795,26 @@ func TestUndurableActorsInstrumentsMultipleActors(t *testing.T) { t.Errorf("after dropping the oldest actor: %+v, want %+v", got, want) } } + +// A background upload that times out records error.type DeadlineExceeded and +// the timeout duration. +func TestUploadDurationTimeoutRecordsDeadlineExceeded(t *testing.T) { + inst, reader := newTestInstruments(t) + store := newFakeSnapshotStore() + store.setBlocking(true) + useActorDir(t) + tr := newStoreTracker(t, store, WithInstruments(inst)) + tr.Start() + + tr.CommitLocal(trackedActorUID, testRef("s1", 1, tr.clock.Now())) + done := startUpload(t, tr, store) + stepTo(tr, tr.clock.Now().Add(testUploadTimeout)) + waitClosed(t, done, "the timed-out upload") + + if got, want := recordedErrorTypes(t, reader), []string{codes.DeadlineExceeded.String()}; !slices.Equal(got, want) { + t.Errorf("recorded error.type = %v, want %v", got, want) + } + if hist := histogramOf(t, reader, uploadDurationMetric); len(hist.DataPoints) != 1 || hist.DataPoints[0].Sum != testUploadTimeout.Seconds() { + t.Errorf("upload histogram = %+v, want one upload of %v", hist.DataPoints, testUploadTimeout) + } +} diff --git a/cmd/atelet/internal/snapshottracker/tracker.go b/cmd/atelet/internal/snapshottracker/tracker.go index 93b2448e27..3bf91e6291 100644 --- a/cmd/atelet/internal/snapshottracker/tracker.go +++ b/cmd/atelet/internal/snapshottracker/tracker.go @@ -124,6 +124,21 @@ type Tracker struct { // this node before atelet detaches it: DefaultDetachDelay unless // WithDetachDelay sets it. detachDelay time.Duration + // uploadTimeout is how long one background upload attempt may run, on + // top of the time its snapshot takes at uploadEstThroughput (see + // uploadTimeoutFor), before it is canceled and retried: + // DefaultUploadTimeout unless WithUploadTimeout sets it. Zero or less + // turns the timeout off. + uploadTimeout time.Duration + // uploadEstThroughput, in bytes per second of a snapshot's allocated + // local size, is how fast atelet expects a background upload to run (see + // uploadTimeoutFor): DefaultUploadEstThroughput unless + // WithUploadEstThroughput sets it. + uploadEstThroughput int64 + // withTimeout derives the context a background upload attempt runs under: + // context.WithTimeout, unless a test on a fake clock replaces it with one + // that times out on that clock. + withTimeout func(context.Context, time.Duration) (context.Context, context.CancelFunc) // instruments records the background uploads and detaches (how late they // start, how long they take, and how they end) and reports // undurableActorsTally. Nil without WithInstruments. @@ -133,7 +148,9 @@ type Tracker struct { // instruments. undurableActorsTally *undurableActorsTally // retries says how long an actor's background work is held back after a - // failure: longer with each consecutive failure for the actor. + // failure: longer with each consecutive failure for the actor. The count + // of those failures also lengthens its next upload attempt's timeout (see + // uploadTimeoutFor). retries workqueue.TypedRateLimiter[string] // workers is how many workers drain each kind's queue: DefaultUploadWorkers // and DefaultDetachWorkers unless WithUploadWorkers and WithDetachWorkers @@ -181,6 +198,28 @@ func WithDetachDelay(d time.Duration) Option { } } +// WithUploadTimeout sets how long one background upload attempt may run, on +// top of the time its snapshot takes at the estimated upload throughput (see +// WithUploadEstThroughput), before atelet cancels and retries it. Each +// consecutive failure doubles the next attempt's timeout, to at most 8 times +// the first's. Zero or less turns the timeout off, the time for the +// snapshot's size included. +func WithUploadTimeout(d time.Duration) Option { + return func(t *Tracker) { + t.uploadTimeout = d + } +} + +// WithUploadEstThroughput sets how fast, in bytes per second of a snapshot's +// allocated local size, atelet expects a background upload to run. Each +// attempt's timeout allows for the snapshot to upload at this rate, on top of +// WithUploadTimeout's. Zero or less adds no time for the snapshot's size. +func WithUploadEstThroughput(bytesPerSecond int64) Option { + return func(t *Tracker) { + t.uploadEstThroughput = bytesPerSecond + } +} + // WithUploadWorkers sets how many background uploads run at once on this node. func WithUploadWorkers(n int) Option { return func(t *Tracker) { @@ -234,16 +273,19 @@ func WithReportGiveUpAfter(d time.Duration) Option { // schedules no uploads or detaches. func New(ctx context.Context, store Store, clk clock.WithTicker, opts ...Option) *Tracker { t := &Tracker{ - ctx: ctx, - store: store, - clock: clk, - uploadDelay: DefaultUploadDelay, - detachDelay: DefaultDetachDelay, - retries: workqueue.NewTypedItemExponentialFailureRateLimiter[string](snapshotRetryDelay, maxSnapshotRetryDelay), - workers: [numOpKinds]int{opUpload: DefaultUploadWorkers, opDetach: DefaultDetachWorkers}, - reporting: newReportQueue(clk), - locks: actorlock.New(), - actors: map[string]*actorSnapshots{}, + ctx: ctx, + store: store, + clock: clk, + uploadDelay: DefaultUploadDelay, + detachDelay: DefaultDetachDelay, + uploadTimeout: DefaultUploadTimeout, + uploadEstThroughput: DefaultUploadEstThroughput, + withTimeout: context.WithTimeout, + retries: workqueue.NewTypedItemExponentialFailureRateLimiter[string](snapshotRetryDelay, maxSnapshotRetryDelay), + workers: [numOpKinds]int{opUpload: DefaultUploadWorkers, opDetach: DefaultDetachWorkers}, + reporting: newReportQueue(clk), + locks: actorlock.New(), + actors: map[string]*actorSnapshots{}, } for k := range t.queues { t.queues[k] = workqueue.NewTypedDelayingQueueWithConfig(workqueue.TypedDelayingQueueConfig[string]{Clock: clk}) diff --git a/cmd/atelet/internal/snapshottracker/tracker_test.go b/cmd/atelet/internal/snapshottracker/tracker_test.go index aa8ac17286..cb44505c7d 100644 --- a/cmd/atelet/internal/snapshottracker/tracker_test.go +++ b/cmd/atelet/internal/snapshottracker/tracker_test.go @@ -35,6 +35,7 @@ import ( "google.golang.org/protobuf/proto" "google.golang.org/protobuf/types/known/timestamppb" "k8s.io/client-go/util/workqueue" + "k8s.io/utils/clock" clocktesting "k8s.io/utils/clock/testing" ) @@ -107,8 +108,12 @@ func uuidOf(r *ateletpb.SnapshotReport) string { const ( testUploadDelay = time.Hour testDetachDelay = 4 * time.Hour + testUploadTimeout = 45 * time.Minute testRetryDelay = 30 * time.Minute testMaxRetryDelay = 2 * time.Hour + // testUploadEstThroughput lets a test's local copy of a few MiB add whole + // seconds to its upload timeout. + testUploadEstThroughput = 1 << 20 ) // fakeSnapshotStore is a Store that records its calls. A call for a URI it @@ -125,6 +130,9 @@ type fakeSnapshotStore struct { uploaded map[string]bool blocking bool err error + // deadline is the deadline of the latest call's context: zero if it has + // none. + deadline time.Time } func newFakeSnapshotStore() *fakeSnapshotStore { @@ -158,10 +166,19 @@ func (f *fakeSnapshotStore) actorOf(uuid string) resources.ActorRef { return f.actors[uuid] } +// lastDeadline returns the deadline of the latest call's context: zero if it +// has none. +func (f *fakeSnapshotStore) lastDeadline() time.Time { + f.mu.Lock() + defer f.mu.Unlock() + return f.deadline +} + func (f *fakeSnapshotStore) UploadLocalSnapshot(ctx context.Context, _ string, ref SnapshotRef) (bool, error) { f.mu.Lock() f.calls = append(f.calls, ref.uuid()) f.actors[ref.uuid()] = ref.Actor + f.deadline, _ = ctx.Deadline() blocking, err := f.blocking, f.err f.mu.Unlock() f.started <- ref.uuid() @@ -189,13 +206,16 @@ func (f *fakeSnapshotStore) UploadLocalSnapshot(ctx context.Context, _ string, r // newStoreTracker returns a tracker over store on a fake clock, with the test // delays and opts, whose workers have not started: background work starts only -// once a test advances the clock to when it is due (see stepTo). Its workers -// stop once the test ends, before the node directories are restored. +// once a test advances the clock to when it is due (see stepTo), and an upload +// attempt times out once a test advances the clock past its deadline. Its +// workers stop once the test ends, before the node directories are restored. func newStoreTracker(t *testing.T, store Store, opts ...Option) *Tracker { t.Helper() - opts = append([]Option{WithUploadDelay(testUploadDelay), WithDetachDelay(testDetachDelay)}, opts...) - tr := New(t.Context(), store, clocktesting.NewFakeClock(time.Unix(1_700_000_000, 0)), opts...) + opts = append([]Option{WithUploadDelay(testUploadDelay), WithDetachDelay(testDetachDelay), WithUploadTimeout(testUploadTimeout), WithUploadEstThroughput(testUploadEstThroughput)}, opts...) + clk := clocktesting.NewFakeClock(time.Unix(1_700_000_000, 0)) + tr := New(t.Context(), store, clk, opts...) tr.retries = workqueue.NewTypedItemExponentialFailureRateLimiter[string](testRetryDelay, testMaxRetryDelay) + tr.withTimeout = fakeClockTimeout(clk) t.Cleanup(func() { ctx, cancel := context.WithTimeout(context.Background(), time.Minute) defer cancel() @@ -206,6 +226,66 @@ func newStoreTracker(t *testing.T, store Store, opts ...Option) *Tracker { return tr } +// fakeClockTimeout returns a context.WithTimeout that times out on clk: its +// contexts end with context.DeadlineExceeded once clk reaches their deadline, +// which Deadline reports in clk's time. Like context.WithTimeout's, its cancel +// stops the timer before returning, so an upload that has ended leaves no +// waiter on clk for tests that count them. +func fakeClockTimeout(clk clock.Clock) func(context.Context, time.Duration) (context.Context, context.CancelFunc) { + return func(parent context.Context, d time.Duration) (context.Context, context.CancelFunc) { + c := &clockTimerCtx{Context: parent, deadline: clk.Now().Add(d), done: make(chan struct{})} + if err := parent.Err(); err != nil { + c.finish(err) + return c, func() {} + } + timer := clk.NewTimer(d) + go func() { + select { + case <-parent.Done(): + timer.Stop() + c.finish(parent.Err()) + case <-timer.C(): + c.finish(context.DeadlineExceeded) + case <-c.done: + } + }() + return c, func() { + timer.Stop() + c.finish(context.Canceled) + } + } +} + +// clockTimerCtx is a context fakeClockTimeout returns. +type clockTimerCtx struct { + context.Context + deadline time.Time + done chan struct{} + + mu sync.Mutex + err error +} + +func (c *clockTimerCtx) Deadline() (time.Time, bool) { return c.deadline, true } +func (c *clockTimerCtx) Done() <-chan struct{} { return c.done } + +func (c *clockTimerCtx) Err() error { + c.mu.Lock() + defer c.mu.Unlock() + return c.err +} + +// finish ends c with err, unless it has ended already. +func (c *clockTimerCtx) finish(err error) { + c.mu.Lock() + defer c.mu.Unlock() + if c.err != nil { + return + } + c.err = err + close(c.done) +} + // newIdleTestTracker returns a tracker like newStoreTracker's, on fresh node // directories that hold the tracked actor's directory. func newIdleTestTracker(t *testing.T, store Store) *Tracker { diff --git a/cmd/atelet/internal/snapshottracker/upload.go b/cmd/atelet/internal/snapshottracker/upload.go index 0b7864ab26..3fad1c546b 100644 --- a/cmd/atelet/internal/snapshottracker/upload.go +++ b/cmd/atelet/internal/snapshottracker/upload.go @@ -17,6 +17,7 @@ package snapshottracker import ( "context" "log/slog" + "math" "time" ) @@ -65,17 +66,55 @@ func (t *Tracker) cleanupSupersededUploadingLocked(a *actorSnapshots) func() { } // uploadSnapshot uploads target from its local copy to object storage and, -// when background is true, records the background upload metric. +// when background is true, bounds the attempt by uploadTimeoutFor, unless +// uploadTimeout is zero or less, and records the background upload metric. func (t *Tracker) uploadSnapshot(ctx context.Context, actorUID string, target SnapshotRef, background bool) (uploaded bool, elapsed time.Duration, err error) { + uploadCtx := ctx + if background && t.uploadTimeout > 0 { + var cancel context.CancelFunc + uploadCtx, cancel = t.withTimeout(ctx, t.uploadTimeoutFor(ctx, actorUID, target)) + defer cancel() + } start := t.clock.Now() - uploaded, err = t.store.UploadLocalSnapshot(ctx, actorUID, target) + uploaded, err = t.store.UploadLocalSnapshot(uploadCtx, actorUID, target) elapsed = t.clock.Since(start) if background { - t.instruments.recordUpload(ctx, target, elapsed, uploaded, err) + t.instruments.recordUpload(uploadCtx, target, elapsed, uploaded, err) } return uploaded, elapsed, err } +// uploadTimeoutFor returns how long a background upload attempt of target may +// run: uploadTimeout, plus the time target's local copy takes to upload at +// uploadEstThroughput, doubled for each consecutive failure of the actor's +// background work, up to maxUploadTimeoutDoublings times. A copy that is +// gone, as when the upload only probes for a committed manifest, adds nothing, +// as does an uploadEstThroughput of zero or less. A failure to measure the +// copy is logged, and the timeout allows for what was measured. +func (t *Tracker) uploadTimeoutFor(ctx context.Context, actorUID string, target SnapshotRef) time.Duration { + timeout := t.uploadTimeout + if t.uploadEstThroughput > 0 { + size, err := localSnapshotSize(actorUID, target.uuid()) + if err != nil { + slog.WarnContext(ctx, "Failed to measure a local snapshot to bound its upload", slog.String("actorUID", actorUID), slog.String("snapshot", target.uuid()), slog.Any("err", err)) + } + timeout += uploadAllowance(size, t.uploadEstThroughput) + } + n := min(t.retries.NumRequeues(actorUID), maxUploadTimeoutDoublings) + // Saturate: a doubled timeout that wrapped negative would end at once. + return min(timeout, time.Duration(math.MaxInt64)>>n) << n +} + +// uploadAllowance returns how long size bytes take to upload at +// bytesPerSecond, rounded up to a whole second: zero when either is zero or +// less. +func uploadAllowance(size, bytesPerSecond int64) time.Duration { + if size <= 0 || bytesPerSecond <= 0 { + return 0 + } + return time.Duration((size-1)/bytesPerSecond+1) * time.Second +} + // backgroundUpload runs op, a background upload of the actor's nextUploadTarget // when it began, on a worker. Once the upload succeeds or a newer snapshot // replaces target, it removes target's local copy if LatestLocal moved on (see diff --git a/cmd/atelet/internal/snapshottracker/upload_test.go b/cmd/atelet/internal/snapshottracker/upload_test.go index 5f1049a1e3..698993091f 100644 --- a/cmd/atelet/internal/snapshottracker/upload_test.go +++ b/cmd/atelet/internal/snapshottracker/upload_test.go @@ -17,7 +17,9 @@ package snapshottracker import ( "context" "errors" + "math" "os" + "path/filepath" "slices" "testing" "time" @@ -1233,3 +1235,186 @@ func TestTrackerOlderLocalDoesNotSupersedeUploading(t *testing.T) { t.Errorf("s2 local dir after upload: %v, want it kept", err) } } + +// A background upload that runs past the upload timeout is canceled, kept in +// Uploading, and retried once the retry delay has passed. +func TestTrackerBackgroundUploadTimeout(t *testing.T) { + store := newFakeSnapshotStore() + store.setBlocking(true) + tr := newTestTracker(t, store) + t0 := tr.clock.Now() + tr.CommitLocal(trackedActorUID, testRef("s1", 1, t0)) + + done := startUpload(t, tr, store) + stepTo(tr, tr.clock.Now().Add(testUploadTimeout)) + waitClosed(t, done, "the timed-out upload") + + st := tr.State(trackedActorUID) + if st.GetLatestDurable() != nil { + t.Errorf("latest_durable after timeout = %v, want unset", st.GetLatestDurable()) + } + if got := uuidOf(st.GetUploading()); got != "s1" { + t.Errorf("uploading after timeout = %q, want s1 kept to retry", got) + } + if got, want := timeOf(st.GetUploadDueTime()), tr.clock.Now().Add(testRetryDelay); !got.Equal(want) { + t.Errorf("upload_due_time after timeout = %v, want %v", got, want) + } + + uploadNow(t, tr, store, nil) + if got := uuidOf(tr.State(trackedActorUID).GetLatestDurable()); got != "s1" { + t.Errorf("latest_durable after retry = %q, want s1", got) + } +} + +// A background upload, and the upload a background detach runs first, may run +// uploadTimeout plus the time the snapshot's local copy takes at +// uploadEstThroughput, so a large snapshot is not cut off. +func TestTrackerUploadTimeoutScalesWithSnapshotSize(t *testing.T) { + // writeLocalCopy writes a local copy of s1 of testUploadEstThroughput + // bytes, and returns how long an upload of it may run. The filesystem + // decides how much space the copy takes, so that is measured: it takes at + // least a second at testUploadEstThroughput. + writeLocalCopy := func(t *testing.T) time.Duration { + t.Helper() + writeDenseFile(t, filepath.Join(ateletpath.LocalSnapshotDir(trackedActorUID, "s1"), "pages.img"), testUploadEstThroughput) + size, err := localSnapshotSize(trackedActorUID, "s1") + if err != nil { + t.Fatalf("localSnapshotSize: %v", err) + } + scaled := testUploadTimeout + uploadAllowance(size, testUploadEstThroughput) + if scaled < testUploadTimeout+time.Second { + t.Fatalf("a local copy of %d bytes may upload for %v, want at least %v", size, scaled, testUploadTimeout+time.Second) + } + return scaled + } + + t.Run("upload", func(t *testing.T) { + store := newFakeSnapshotStore() + store.setBlocking(true) + tr := newTestTracker(t, store) + scaled := writeLocalCopy(t) + tr.CommitLocal(trackedActorUID, testRef("s1", 1, tr.clock.Now())) + + done := startUpload(t, tr, store) + start := tr.clock.Now() + if got := store.lastDeadline().Sub(start); got != scaled { + t.Errorf("upload may run %v, want %v", got, scaled) + } + stepTo(tr, start.Add(scaled)) + waitClosed(t, done, "the timed-out upload") + }) + + t.Run("detach", func(t *testing.T) { + store := newFakeSnapshotStore() + store.setBlocking(true) + tr := newIdleTestTracker(t, store) + setWorkers(tr, 0, DefaultDetachWorkers) + tr.Start() + scaled := writeLocalCopy(t) + tr.CommitLocal(trackedActorUID, testRef("s1", 1, tr.clock.Now())) + + stepTo(tr, detachDueOf(tr, trackedActorUID)) + waitStarted(t, store) + if got := store.lastDeadline().Sub(tr.clock.Now()); got != scaled { + t.Errorf("the detach's upload may run %v, want %v", got, scaled) + } + }) + + t.Run("no local copy", func(t *testing.T) { + store := newFakeSnapshotStore() + store.setBlocking(true) + tr := newTestTracker(t, store) + tr.CommitLocal(trackedActorUID, testRef("s1", 1, tr.clock.Now())) + + startUpload(t, tr, store) + if got := store.lastDeadline().Sub(tr.clock.Now()); got != testUploadTimeout { + t.Errorf("upload may run %v, want %v", got, testUploadTimeout) + } + }) + + t.Run("zero throughput adds nothing", func(t *testing.T) { + store := newFakeSnapshotStore() + store.setBlocking(true) + useActorDir(t) + tr := newStoreTracker(t, store, WithUploadEstThroughput(0)) + tr.Start() + writeLocalCopy(t) + tr.CommitLocal(trackedActorUID, testRef("s1", 1, tr.clock.Now())) + + startUpload(t, tr, store) + if got := store.lastDeadline().Sub(tr.clock.Now()); got != testUploadTimeout { + t.Errorf("upload may run %v, want %v", got, testUploadTimeout) + } + }) +} + +// Each consecutive failure of an actor's background work doubles its next +// upload attempt's timeout, to at most 8 times the first's, so an upload +// slower than the estimated throughput still finishes. Once the actor's work +// succeeds, its next upload gets the first attempt's timeout again. +func TestTrackerUploadTimeoutGrowsWithFailures(t *testing.T) { + store := newFakeSnapshotStore() + store.setBlocking(true) + tr := newIdleTestTracker(t, store) + // Without detach workers, no detach runs between the attempts. + setWorkers(tr, DefaultUploadWorkers, 0) + tr.Start() + tr.CommitLocal(trackedActorUID, testRef("s1", 1, tr.clock.Now())) + + for attempt, factor := range []time.Duration{1, 2, 4, 8, 8} { + done := startUpload(t, tr, store) + start := tr.clock.Now() + want := factor * testUploadTimeout + if got := store.lastDeadline().Sub(start); got != want { + t.Fatalf("attempt %d may run %v, want %v", attempt+1, got, want) + } + stepTo(tr, start.Add(want)) + waitClosed(t, done, "the timed-out upload") + } + + uploadNow(t, tr, store, nil) + tr.CommitLocal(trackedActorUID, testRef("s2", 2, tr.clock.Now())) + startUpload(t, tr, store) + if got := store.lastDeadline().Sub(tr.clock.Now()); got != testUploadTimeout { + t.Errorf("the upload after a success may run %v, want %v", got, testUploadTimeout) + } +} + +// A timeout too long to double saturates rather than wrapping negative, which +// would time the upload out at once. +func TestUploadTimeoutForSaturates(t *testing.T) { + useActorDir(t) + tr := newStoreTracker(t, newFakeSnapshotStore(), WithUploadTimeout(math.MaxInt64)) + for range maxUploadTimeoutDoublings { + tr.retries.When(trackedActorUID) + } + if got := tr.uploadTimeoutFor(t.Context(), trackedActorUID, testRef("s1", 1, tr.clock.Now())); got < math.MaxInt64/2 { + t.Errorf("uploadTimeoutFor = %v, want it to saturate near the longest time.Duration", got) + } +} + +// uploadAllowance rounds the time a snapshot takes to upload up to a whole +// second, and allows none for an empty snapshot or a throughput of zero or +// less. +func TestUploadAllowance(t *testing.T) { + const mib = 1 << 20 + for _, tc := range []struct { + size, bytesPerSecond int64 + want time.Duration + }{ + {size: 0, bytesPerSecond: 100 * mib, want: 0}, + {size: 100 * mib, bytesPerSecond: 0, want: 0}, + {size: 100 * mib, bytesPerSecond: -1, want: 0}, + {size: 1, bytesPerSecond: 100 * mib, want: time.Second}, + {size: 100 * mib, bytesPerSecond: 100 * mib, want: time.Second}, + {size: 100*mib + 1, bytesPerSecond: 100 * mib, want: 2 * time.Second}, + // 102.4 seconds. + {size: 10 << 30, bytesPerSecond: 100 * mib, want: 103 * time.Second}, + // Too large to multiply by time.Second before dividing. + {size: 1 << 40, bytesPerSecond: mib, want: mib * time.Second}, + } { + if got := uploadAllowance(tc.size, tc.bytesPerSecond); got != tc.want { + t.Errorf("uploadAllowance(%d, %d) = %v, want %v", tc.size, tc.bytesPerSecond, got, tc.want) + } + } +} diff --git a/cmd/atelet/internal/sparsefile/sparsefile.go b/cmd/atelet/internal/sparsefile/sparsefile.go index 9fb0d66472..a7ded46ef9 100644 --- a/cmd/atelet/internal/sparsefile/sparsefile.go +++ b/cmd/atelet/internal/sparsefile/sparsefile.go @@ -12,7 +12,8 @@ // See the License for the specific language governing permissions and // limitations under the License. -// Package sparsefile copies files while preserving holes. +// Package sparsefile copies files while preserving holes, and measures the +// disk space a file's data takes. // // The biggest thing atelet copies is a guest memory image, which is mostly // unallocated: a plain io.Copy reads holes as zeroes and writes them as @@ -25,7 +26,9 @@ import ( "errors" "fmt" "io" + "io/fs" "os" + "syscall" "golang.org/x/sys/unix" ) @@ -38,6 +41,17 @@ var errSparseUnsupported = errors.New("filesystem cannot report holes") // range in the kernel, so the caller should copy through userspace instead. var errKernelCopyUnsupported = errors.New("kernel range copy unsupported") +// AllocatedBytes returns the disk space allocated to info (st_blocks * 512) +// rather than its apparent size, which for a sparse guest memory image is the +// guest RAM ceiling, often orders of magnitude more than the data it holds. +// ext4, XFS, and btrfs count delayed allocations before writeback. +func AllocatedBytes(info fs.FileInfo) int64 { + if st, ok := info.Sys().(*syscall.Stat_t); ok { + return int64(st.Blocks) * 512 + } + return info.Size() +} + // CopyFile copies src to dst, preserving holes where it can, and returns the number of // logical bytes copied. dst is created (truncated if it exists), like os.Create. func CopyFile(src, dst string) (int64, error) { diff --git a/cmd/atelet/internal/sparsefile/sparsefile_test.go b/cmd/atelet/internal/sparsefile/sparsefile_test.go index fc38fe4945..b58f169698 100644 --- a/cmd/atelet/internal/sparsefile/sparsefile_test.go +++ b/cmd/atelet/internal/sparsefile/sparsefile_test.go @@ -315,3 +315,73 @@ func TestCopyHandles(t *testing.T) { } } } + +// patternBytes returns a byte slice of length n filled with a repeating +// non-zero byte pattern so test data is incompressible and not treated as zero +// blocks. +func patternBytes(n int64) []byte { + b := make([]byte, n) + for i := range b { + b[i] = byte(i%251 + 1) + } + return b +} + +func TestAllocatedBytesSparseImage(t *testing.T) { + const ( + apparent = 2 << 30 // 2 GiB + written = 1 << 20 // 1 MiB + ) + + path := filepath.Join(t.TempDir(), "memory-ranges") + f, err := os.Create(path) + if err != nil { + t.Fatalf("create sparse image: %v", err) + } + if _, err := f.Write(patternBytes(written)); err != nil { + f.Close() + t.Fatalf("write initial pages: %v", err) + } + if err := f.Truncate(apparent); err != nil { + f.Close() + t.Fatalf("extend sparse image: %v", err) + } + if err := f.Close(); err != nil { + t.Fatalf("close sparse image: %v", err) + } + + fi, err := os.Lstat(path) + if err != nil { + t.Fatalf("lstat sparse image: %v", err) + } + if fi.Size() != apparent { + t.Fatalf("fi.Size() = %d, want %d", fi.Size(), int64(apparent)) + } + + got := AllocatedBytes(fi) + if got >= apparent/2 { + t.Skipf("file did not end up sparse (%d of %d bytes allocated); "+ + "this filesystem cannot report holes", got, int64(apparent)) + } + if got == 0 || got > written*16 { + t.Errorf("AllocatedBytes() = %d, want > 0 and << %d (wrote %d)", got, int64(apparent), int64(written)) + } +} + +func TestAllocatedBytesDenseImage(t *testing.T) { + const size = 64 << 10 + + path := filepath.Join(t.TempDir(), "state.json") + if err := os.WriteFile(path, patternBytes(size), 0o600); err != nil { + t.Fatalf("write dense image: %v", err) + } + fi, err := os.Lstat(path) + if err != nil { + t.Fatalf("lstat dense image: %v", err) + } + + got := AllocatedBytes(fi) + if got < size || got > size*2 { + t.Errorf("AllocatedBytes() = %d, want ~%d", got, size) + } +} diff --git a/cmd/atelet/main.go b/cmd/atelet/main.go index 49c94997a1..59fc49d785 100644 --- a/cmd/atelet/main.go +++ b/cmd/atelet/main.go @@ -131,6 +131,8 @@ var ( objectStorePluginSocket = pflag.String("objectstore-plugin-socket", "/run/objectstore-plugin/node.sock", "Unix socket of the node object-store plugin that external snapshots are fetched and uploaded through, and every sandbox asset is fetched through. A gs:// asset the plugin cannot serve is then read anonymously.") snapshotUploadDelay = pflag.Duration("snapshot-upload-delay", snapshottracker.DefaultUploadDelay, "How long after an actor's oldest snapshot not yet in object storage was taken atelet uploads the actor's latest local snapshot in the background; later while an upload is in progress or failing.") snapshotDetachDelay = pflag.Duration("snapshot-detach-delay", snapshottracker.DefaultDetachDelay, "How long an actor stays suspended on this node, its latest snapshot kept locally, before atelet detaches it in the background.") + snapshotUploadTimeout = pflag.Duration("snapshot-upload-timeout", snapshottracker.DefaultUploadTimeout, "How long one background snapshot upload, including the one a background detach runs first, may run before atelet cancels and retries it, on top of the time the snapshot takes at --snapshot-upload-est-throughput-mib. Each consecutive failure doubles the next attempt's timeout, to at most 8 times the first's. 0 turns the timeout off.") + snapshotUploadEstMiBps = pflag.Int("snapshot-upload-est-throughput-mib", snapshottracker.DefaultUploadEstThroughput>>20, "Estimated background snapshot upload throughput, in MiB per second of the snapshot's allocated size: each upload attempt may run --snapshot-upload-timeout plus the time the snapshot takes at this rate. 0 adds no time for the snapshot's size.") snapshotUploadWorkers = pflag.Int("snapshot-upload-workers", snapshottracker.DefaultUploadWorkers, "How many background snapshot uploads atelet runs at once.") snapshotDetachWorkers = pflag.Int("snapshot-detach-workers", snapshottracker.DefaultDetachWorkers, "How many background detaches atelet runs at once.") snapshotReportWorkers = pflag.Int("snapshot-report-workers", snapshottracker.DefaultReportWorkers, "How many background snapshot state reports atelet runs at once. 0 disables background reporting.") @@ -380,6 +382,8 @@ func main() { snapshots := snapshottracker.New(trackerCtx, store, clock.RealClock{}, snapshottracker.WithUploadDelay(*snapshotUploadDelay), snapshottracker.WithDetachDelay(*snapshotDetachDelay), + snapshottracker.WithUploadTimeout(*snapshotUploadTimeout), + snapshottracker.WithUploadEstThroughput(int64(*snapshotUploadEstMiBps)<<20), snapshottracker.WithUploadWorkers(*snapshotUploadWorkers), snapshottracker.WithDetachWorkers(*snapshotDetachWorkers), snapshottracker.WithInstruments(trackerInstruments), @@ -583,10 +587,18 @@ func rejectStorageEnv() error { return nil } +// maxSnapshotUploadEstMiBps bounds --snapshot-upload-est-throughput-mib at +// 1 TiB/s, far above any upload, so its value in bytes per second cannot +// overflow. +const maxSnapshotUploadEstMiBps = 1 << 20 + // validateSnapshotDelayFlags checks --snapshot-upload-delay, -// --snapshot-detach-delay, and --snapshot-report-give-up-after: all three must -// be positive, and the upload delay must not exceed the detach delay, since a -// detach uploads the latest local snapshot anyway. +// --snapshot-detach-delay, --snapshot-upload-timeout, +// --snapshot-upload-est-throughput-mib, and --snapshot-report-give-up-after. +// The delays must be positive, and the upload delay must not exceed the detach +// delay, since a detach uploads the latest local snapshot anyway. The upload +// timeout must not be negative, and the upload throughput must be between 0 +// and maxSnapshotUploadEstMiBps: 0 turns either off. func validateSnapshotDelayFlags() error { if *snapshotUploadDelay <= 0 { return fmt.Errorf("--snapshot-upload-delay %v must be > 0", *snapshotUploadDelay) @@ -594,6 +606,12 @@ func validateSnapshotDelayFlags() error { if *snapshotDetachDelay <= 0 { return fmt.Errorf("--snapshot-detach-delay %v must be > 0", *snapshotDetachDelay) } + if *snapshotUploadTimeout < 0 { + return fmt.Errorf("--snapshot-upload-timeout %v must be >= 0", *snapshotUploadTimeout) + } + if *snapshotUploadEstMiBps < 0 || *snapshotUploadEstMiBps > maxSnapshotUploadEstMiBps { + return fmt.Errorf("--snapshot-upload-est-throughput-mib %d must be between 0 and %d", *snapshotUploadEstMiBps, maxSnapshotUploadEstMiBps) + } if *snapshotUploadDelay > *snapshotDetachDelay { return fmt.Errorf("--snapshot-upload-delay %v must not exceed --snapshot-detach-delay %v", *snapshotUploadDelay, *snapshotDetachDelay) } @@ -855,8 +873,7 @@ func (s *Atelet) Run(ctx context.Context, req *ateletpb.RunRequest) (resp *atele ActorUid: actorUID, ActorDirs: ateletpath.ActorDirs(actorUID), EgressGateway: toWorkerEgressGateway(req.GetEgressGateway()), - CpuMilli: req.GetCpuMilli(), - MemoryBytes: req.GetMemoryBytes(), + Resources: toAteworkerResources(req.GetResources()), }); err != nil { return nil, fmt.Errorf("while calling worker RunWorkload: %w", err) } @@ -894,16 +911,6 @@ func recordSnapshotSize(ctx context.Context, file string, size int64, templateAt )) } -// allocatedBytes returns the disk space allocated to info (st_blocks * 512) -// rather than its apparent size, which for sparse snapshot images reflects the -// guest RAM ceiling; ext4/XFS/btrfs include delalloc blocks before writeback. -func allocatedBytes(info os.FileInfo) int64 { - if st, ok := info.Sys().(*syscall.Stat_t); ok { - return int64(st.Blocks) * 512 - } - return info.Size() -} - func (s *Atelet) Checkpoint(ctx context.Context, req *ateletpb.CheckpointRequest) (_ *ateletpb.CheckpointResponse, err error) { if errs := apivalidation.ValidateCheckpointRequest(ctx, req); len(errs) > 0 { return nil, resources.ToAPIError(errs) @@ -1160,7 +1167,7 @@ func (s *Atelet) moveLocalCheckpoint(ctx context.Context, req *ateletpb.Checkpoi if !info.Mode().IsRegular() { return fmt.Errorf("checkpoint file %s is not a regular file", fileName) } - recordSnapshotSize(ctx, fileName, allocatedBytes(info), req.GetActorTemplateAtespace(), req.GetActorTemplateName()) + recordSnapshotSize(ctx, fileName, sparsefile.AllocatedBytes(info), req.GetActorTemplateAtespace(), req.GetActorTemplateName()) if err := root.MkdirAll(filepath.Dir(dst), 0o700); err != nil { return fmt.Errorf("while creating parent directory for %s: %w", dst, err) @@ -1640,8 +1647,7 @@ func (s *Atelet) Restore(ctx context.Context, req *ateletpb.RestoreRequest) (res ActorDirs: actorDirs, PreserveRestoreDir: directLocal, EgressGateway: toWorkerEgressGateway(req.GetEgressGateway()), - CpuMilli: req.GetCpuMilli(), - MemoryBytes: req.GetMemoryBytes(), + Resources: toAteworkerResources(req.GetResources()), }) dWorker = time.Since(tWorker) if err != nil { @@ -1950,7 +1956,7 @@ func (s *Atelet) prepareOCIBundles( Args: args, Env: resolveActorEnv(imageConfig, envs), Capabilities: resolveCapabilities(ctr.GetSecurityContext().GetCapabilities()), - Resources: toWorkerResourceLimits(ctr.GetResources()), + Resources: toAteworkerResources(ctr.GetResources()), } return nil }) @@ -2039,11 +2045,15 @@ func buildWorkerWorkloadSpec(spec *ateletpb.WorkloadSpec, containerSpecs []*atew return out, nil } -func toWorkerResourceLimits(limits *ateletpb.ResourceLimits) *ateworkerpb.ResourceLimits { - if limits == nil { +func toAteworkerResources(r *ateletpb.ResourceRequirements) *ateworkerpb.ResourceRequirements { + if r == nil { return nil } - return &ateworkerpb.ResourceLimits{MemoryBytes: limits.GetMemoryBytes(), CpuMillis: limits.GetCpuMillis()} + out := &ateworkerpb.ResourceRequirements{} + if l := r.GetLimits(); l != nil { + out.Limits = &ateworkerpb.Resources{CpuMillicores: l.GetCpuMillicores(), MemoryBytes: l.GetMemoryBytes()} + } + return out } func toWorkerEgressGateway(gateway *ateletpb.EgressGateway) *ateworkerpb.EgressGateway { diff --git a/cmd/atelet/main_test.go b/cmd/atelet/main_test.go index 9e292c1312..9fb7a4e906 100644 --- a/cmd/atelet/main_test.go +++ b/cmd/atelet/main_test.go @@ -82,39 +82,73 @@ func TestPortFlagDefault(t *testing.T) { } } -// The snapshot delay flags must be positive, and the upload delay must not -// exceed the detach delay, which the defaults satisfy too. +// The snapshot delay flags must be positive, the upload timeout must not be +// negative (0 turns it off), and the upload delay must not exceed the detach +// delay, which the defaults satisfy too. func TestValidateSnapshotDelayFlags(t *testing.T) { - setFlags := func(upload, detach, giveUp time.Duration) { - *snapshotUploadDelay, *snapshotDetachDelay, *snapshotReportGiveUpAfter = upload, detach, giveUp + setFlags := func(upload, detach, timeout, giveUp time.Duration) { + *snapshotUploadDelay, *snapshotDetachDelay, *snapshotUploadTimeout, *snapshotReportGiveUpAfter = upload, detach, timeout, giveUp } t.Cleanup(func() { - setFlags(snapshottracker.DefaultUploadDelay, snapshottracker.DefaultDetachDelay, snapshottracker.DefaultReportGiveUpAfter) + setFlags(snapshottracker.DefaultUploadDelay, snapshottracker.DefaultDetachDelay, snapshottracker.DefaultUploadTimeout, snapshottracker.DefaultReportGiveUpAfter) }) for _, tc := range []struct { - name string - upload, detach, giveUp time.Duration - wantErr bool + name string + upload, detach, timeout, giveUp time.Duration + wantErr bool }{ - {"defaults", snapshottracker.DefaultUploadDelay, snapshottracker.DefaultDetachDelay, snapshottracker.DefaultReportGiveUpAfter, false}, - {"upload sooner than detach", time.Minute, time.Hour, time.Minute, false}, - {"upload as late as detach", time.Hour, time.Hour, time.Minute, false}, - {"upload later than detach", time.Hour, time.Minute, time.Minute, true}, - {"zero upload delay", 0, time.Hour, time.Minute, true}, - {"negative detach delay", time.Minute, -time.Minute, time.Minute, true}, - {"zero report give-up delay", time.Minute, time.Hour, 0, true}, - {"negative report give-up delay", time.Minute, time.Hour, -time.Minute, true}, + {"defaults", snapshottracker.DefaultUploadDelay, snapshottracker.DefaultDetachDelay, snapshottracker.DefaultUploadTimeout, snapshottracker.DefaultReportGiveUpAfter, false}, + {"upload sooner than detach", time.Minute, time.Hour, 5 * time.Minute, time.Minute, false}, + {"upload as late as detach", time.Hour, time.Hour, 5 * time.Minute, time.Minute, false}, + {"upload later than detach", time.Hour, time.Minute, 5 * time.Minute, time.Minute, true}, + {"zero upload delay", 0, time.Hour, 5 * time.Minute, time.Minute, true}, + {"negative detach delay", time.Minute, -time.Minute, 5 * time.Minute, time.Minute, true}, + {"zero upload timeout", time.Minute, time.Hour, 0, time.Minute, false}, + {"negative upload timeout", time.Minute, time.Hour, -time.Minute, time.Minute, true}, + {"zero report give-up delay", time.Minute, time.Hour, 5 * time.Minute, 0, true}, + {"negative report give-up delay", time.Minute, time.Hour, 5 * time.Minute, -time.Minute, true}, } { t.Run(tc.name, func(t *testing.T) { - setFlags(tc.upload, tc.detach, tc.giveUp) + setFlags(tc.upload, tc.detach, tc.timeout, tc.giveUp) if err := validateSnapshotDelayFlags(); (err != nil) != tc.wantErr { - t.Errorf("upload=%v detach=%v giveUp=%v: err = %v, wantErr %v", tc.upload, tc.detach, tc.giveUp, err, tc.wantErr) + t.Errorf("upload=%v detach=%v timeout=%v giveUp=%v: err = %v, wantErr %v", tc.upload, tc.detach, tc.timeout, tc.giveUp, err, tc.wantErr) } }) } } +// --snapshot-upload-est-throughput-mib must be between 0, which adds no time +// for a snapshot's size, and maxSnapshotUploadEstMiBps, so its rate in bytes +// cannot overflow, and defaults to the tracker's default throughput in MiB/s. +func TestValidateSnapshotUploadEstThroughput(t *testing.T) { + t.Cleanup(func() { *snapshotUploadEstMiBps = snapshottracker.DefaultUploadEstThroughput >> 20 }) + for _, tc := range []struct { + mibps int + wantErr bool + }{ + {snapshottracker.DefaultUploadEstThroughput >> 20, false}, + {1, false}, + {0, false}, + {maxSnapshotUploadEstMiBps, false}, + {maxSnapshotUploadEstMiBps + 1, true}, + {-1, true}, + } { + *snapshotUploadEstMiBps = tc.mibps + if err := validateSnapshotDelayFlags(); (err != nil) != tc.wantErr { + t.Errorf("--snapshot-upload-est-throughput-mib=%d: err = %v, wantErr %v", tc.mibps, err, tc.wantErr) + } + } + + f := pflag.Lookup("snapshot-upload-est-throughput-mib") + if f == nil { + t.Fatal("no --snapshot-upload-est-throughput-mib flag registered") + } + if want := "100"; f.DefValue != want { + t.Errorf("--snapshot-upload-est-throughput-mib default = %q, want %q", f.DefValue, want) + } +} + // Upload and detach worker flags must be at least 1, report workers must be at // least 0, and each flag defaults to the tracker's default count. func TestValidateSnapshotWorkerFlags(t *testing.T) { @@ -374,7 +408,7 @@ func TestPrepareOCIBundlesFillsContainerSpec(t *testing.T) { SecurityContext: &ateletpb.SecurityContext{ Capabilities: &ateletpb.Capabilities{Add: []string{"NET_ADMIN"}, Drop: []string{"KILL"}}, }, - Resources: &ateletpb.ResourceLimits{MemoryBytes: 1 << 30, CpuMillis: 500}, + Resources: &ateletpb.ResourceRequirements{Limits: &ateletpb.Resources{MemoryBytes: 1 << 30, CpuMillicores: 500}}, }}, } @@ -387,7 +421,7 @@ func TestPrepareOCIBundlesFillsContainerSpec(t *testing.T) { Args: []string{"/bin/app", "serve"}, Env: []string{"GREETING=hello", "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"}, Capabilities: []string{"CAP_AUDIT_WRITE", "CAP_NET_ADMIN", "CAP_NET_BIND_SERVICE"}, - Resources: &ateworkerpb.ResourceLimits{MemoryBytes: 1 << 30, CpuMillis: 500}, + Resources: &ateworkerpb.ResourceRequirements{Limits: &ateworkerpb.Resources{MemoryBytes: 1 << 30, CpuMillicores: 500}}, }} if diff := cmp.Diff(want, got, protocmp.Transform()); diff != "" { t.Errorf("prepareOCIBundles container specs mismatch (-want +got):\n%s", diff) diff --git a/cmd/atelet/metrics_test.go b/cmd/atelet/metrics_test.go index 4145e6e91e..757ea85f05 100644 --- a/cmd/atelet/metrics_test.go +++ b/cmd/atelet/metrics_test.go @@ -17,8 +17,6 @@ package main import ( "context" "errors" - "os" - "path/filepath" "testing" "time" @@ -428,73 +426,3 @@ func TestCheckpointSnapshotKind(t *testing.T) { }) } } - -// patternBytes returns a byte slice of length n filled with a repeating -// non-zero byte pattern so test data is incompressible and not treated as zero -// blocks. -func patternBytes(n int64) []byte { - b := make([]byte, n) - for i := range b { - b[i] = byte(i%251 + 1) - } - return b -} - -func TestAllocatedBytesSparseImage(t *testing.T) { - const ( - apparent = 2 << 30 // 2 GiB - written = 1 << 20 // 1 MiB - ) - - path := filepath.Join(t.TempDir(), "memory-ranges") - f, err := os.Create(path) - if err != nil { - t.Fatalf("create sparse image: %v", err) - } - if _, err := f.Write(patternBytes(written)); err != nil { - f.Close() - t.Fatalf("write initial pages: %v", err) - } - if err := f.Truncate(apparent); err != nil { - f.Close() - t.Fatalf("extend sparse image: %v", err) - } - if err := f.Close(); err != nil { - t.Fatalf("close sparse image: %v", err) - } - - fi, err := os.Lstat(path) - if err != nil { - t.Fatalf("lstat sparse image: %v", err) - } - if fi.Size() != apparent { - t.Fatalf("fi.Size() = %d, want %d", fi.Size(), int64(apparent)) - } - - got := allocatedBytes(fi) - if got >= apparent/2 { - t.Skipf("file did not end up sparse (%d of %d bytes allocated); "+ - "this filesystem cannot report holes", got, int64(apparent)) - } - if got == 0 || got > written*16 { - t.Errorf("allocatedBytes() = %d, want > 0 and << %d (wrote %d)", got, int64(apparent), int64(written)) - } -} - -func TestAllocatedBytesDenseImage(t *testing.T) { - const size = 64 << 10 - - path := filepath.Join(t.TempDir(), "state.json") - if err := os.WriteFile(path, patternBytes(size), 0o600); err != nil { - t.Fatalf("write dense image: %v", err) - } - fi, err := os.Lstat(path) - if err != nil { - t.Fatalf("lstat dense image: %v", err) - } - - got := allocatedBytes(fi) - if got < size || got > size*2 { - t.Errorf("allocatedBytes() = %d, want ~%d", got, size) - } -} diff --git a/cmd/atelet/workersupport.go b/cmd/atelet/workersupport.go index 9e4e27e647..be9f3c32de 100644 --- a/cmd/atelet/workersupport.go +++ b/cmd/atelet/workersupport.go @@ -146,14 +146,11 @@ func toWorkerResources(in *ateletpb.WorkerResources) *ateapipb.WorkerResources { if in == nil { return nil } - out := &ateapipb.WorkerResources{Actors: in.GetActors()} - if r := in.GetResources(); r != nil { - out.Resources = &ateapipb.Resources{} - for _, l := range r.GetLimits() { - out.Resources.Limits = append(out.Resources.Limits, &ateapipb.Limits{Name: l.GetName(), Quantity: l.GetQuantity()}) - } + return &ateapipb.WorkerResources{ + CpuMillicores: in.GetCpuMillicores(), + MemoryBytes: in.GetMemoryBytes(), + Actors: in.GetActors(), } - return out } // toAteAPISandboxRuntime converts atelet's SandboxRuntime to the control diff --git a/cmd/atelet/workersupport_test.go b/cmd/atelet/workersupport_test.go index 892c456b9e..974b9e9d14 100644 --- a/cmd/atelet/workersupport_test.go +++ b/cmd/atelet/workersupport_test.go @@ -163,12 +163,10 @@ func TestRegisterWorker(t *testing.T) { }{{ name: "records what the worker says", req: &ateletpb.RegisterWorkerRequest{ - Capacity: &ateletpb.WorkerResources{Actors: 4, Resources: &ateletpb.Resources{ - Limits: []*ateletpb.Limits{{Name: "cpu", Quantity: "2"}, {Name: "memory", Quantity: "4Gi"}}, - }}, + Capacity: &ateletpb.WorkerResources{Actors: 4, CpuMillicores: 2000, MemoryBytes: 4 << 30}, DefaultRuntime: reqDefault, }, - wantForwarded: forwarded(&ateapipb.WorkerResources{Actors: 4, Resources: resources.CPUMemory(2000, 4294967296)}), + wantForwarded: forwarded(&ateapipb.WorkerResources{Actors: 4, CpuMillicores: 2000, MemoryBytes: 4294967296}), }, { name: "forwards restorable runtimes", req: &ateletpb.RegisterWorkerRequest{ @@ -182,15 +180,13 @@ func TestRegisterWorker(t *testing.T) { req: &ateletpb.RegisterWorkerRequest{Capacity: &ateletpb.WorkerResources{Actors: 1}, DefaultRuntime: reqDefault}, wantForwarded: forwarded(&ateapipb.WorkerResources{Actors: 1}), }, { - name: "forwards empty resources", - req: &ateletpb.RegisterWorkerRequest{Capacity: &ateletpb.WorkerResources{Resources: &ateletpb.Resources{}}, DefaultRuntime: reqDefault}, - wantForwarded: forwarded(&ateapipb.WorkerResources{Resources: &ateapipb.Resources{}}), + name: "forwards empty capacity", + req: &ateletpb.RegisterWorkerRequest{Capacity: &ateletpb.WorkerResources{}, DefaultRuntime: reqDefault}, + wantForwarded: forwarded(&ateapipb.WorkerResources{}), }, { name: "rejects invalid capacity without forwarding", req: &ateletpb.RegisterWorkerRequest{ - Capacity: &ateletpb.WorkerResources{Resources: &ateletpb.Resources{ - Limits: []*ateletpb.Limits{{Name: "gpu", Quantity: "1"}}, - }}, + Capacity: &ateletpb.WorkerResources{CpuMillicores: -1}, DefaultRuntime: reqDefault, }, wantCode: codes.InvalidArgument, diff --git a/cmd/atenet/internal/router/ingressauthmanifest_test.go b/cmd/atenet/internal/router/ingressauthmanifest_test.go index caf4051c03..b657bbb736 100644 --- a/cmd/atenet/internal/router/ingressauthmanifest_test.go +++ b/cmd/atenet/internal/router/ingressauthmanifest_test.go @@ -35,7 +35,9 @@ const routerStaticMTLSComponentPath = "../../../../manifests/ate-install/compone // resulting flags. The component appends to an argument list it does not own, // and the base manifest's TLS listeners need the client auth settings, so a // new listener in the manifest, or a new startup check here, would otherwise -// surface only as a router that refuses to start in the e2e cluster. +// surface only as a router that refuses to start in the e2e cluster. The +// manifest carries no allowlist, so the test appends one as ate-setup does +// (TestRenderRouterAllowedSPIFFEIDs pins that ate-setup appends it). func TestRouterIngressAuthManifestsAreValid(t *testing.T) { for _, tc := range []struct { name string @@ -46,7 +48,8 @@ func TestRouterIngressAuthManifestsAreValid(t *testing.T) { {name: "router-static-mtls", componentPath: routerStaticMTLSComponentPath, wantMode: IngressAuthStaticMTLS}, } { t.Run(tc.name, func(t *testing.T) { - args := renderedRouterArgs(t, tc.componentPath) + args := append(renderedRouterArgs(t, tc.componentPath), + "--ingress-allowed-spiffe-ids=spiffe://cluster.local/ns/test/sa/client") var cfg routerConfig fs := pflag.NewFlagSet("router", pflag.ContinueOnError) diff --git a/cmd/ateworker-gvisor/main.go b/cmd/ateworker-gvisor/main.go index c52a5ef12a..660c48054e 100644 --- a/cmd/ateworker-gvisor/main.go +++ b/cmd/ateworker-gvisor/main.go @@ -644,7 +644,7 @@ func (s *AteWorkerService) RunWorkload(ctx context.Context, req *ateworkerpb.Run path: req.GetRunscPath(), actorUID: req.GetActorUid(), actorDirs: req.GetActorDirs(), - size: sizing.FromLimits(req.GetCpuMilli(), req.GetMemoryBytes()), + size: sizing.FromLimits(req.GetResources().GetLimits()), durableVolumes: durableVolumeNames(req.GetSpec()), containers: req.GetSpec().GetContainers(), flags: s.flags, @@ -1061,7 +1061,7 @@ func (s *AteWorkerService) RestoreWorkload(ctx context.Context, req *ateworkerpb path: req.GetRunscPath(), actorUID: req.GetActorUid(), actorDirs: req.GetActorDirs(), - size: sizing.FromLimits(req.GetCpuMilli(), req.GetMemoryBytes()), + size: sizing.FromLimits(req.GetResources().GetLimits()), durableVolumes: durableVolumeNames(req.GetSpec()), containers: req.GetSpec().GetContainers(), flags: s.flags, diff --git a/cmd/ateworker-microvm/checkpoint.go b/cmd/ateworker-microvm/checkpoint.go index 1cb6484eb0..27a0a77092 100644 --- a/cmd/ateworker-microvm/checkpoint.go +++ b/cmd/ateworker-microvm/checkpoint.go @@ -335,7 +335,7 @@ func listFiles(dir string) ([]string, error) { // teardownActor stops the ateworker-owned CH VMM for an actor. ra may be // nil (e.g. ateworker restarted and lost in-memory state). -func (s *AteWorkerService) teardownActor(ctx context.Context, id string, actorDirs *ateworkerpb.ActorDirs, ra *runningActor, client *ch.Client) error { +func (s *AteWorkerService) teardownActor(ctx context.Context, id string, actorDirs *ateworkerpb.ActorDirs, ra *runningActor) error { // Stop offering the guest to GetWorkloadStats first, before anything below // makes it stop answering. Clearing it here rather than alongside the // attribution is what keeps a poll that lands mid-teardown on the @@ -347,18 +347,6 @@ func (s *AteWorkerService) teardownActor(ctx context.Context, id string, actorDi } var errs []error - if client != nil { - tShutdown := time.Now() - shutCtx, cancel := context.WithTimeout(ctx, 10*time.Second) - if err := client.Shutdown(shutCtx); err != nil { - if !errors.Is(err, os.ErrNotExist) { - slog.WarnContext(ctx, "CH shutdown returned error during teardown", slog.Any("err", err)) - } - } - cancel() - slog.InfoContext(ctx, "CH API shutdown done", slog.Duration("took", time.Since(tShutdown))) - } - if ra != nil { // Close the kata-agent client kept open for stdout/stderr forwarding. This // fails the forwarding goroutines' in-flight ReadStdout/ReadStderr calls, so @@ -441,11 +429,11 @@ func (s *AteWorkerService) TerminateWorkload(ctx context.Context, req *ateworker // Restore. Ending the activation also stops a late recordInitial from writing. func (s *AteWorkerService) endPreviousActivation(ctx context.Context, actorUID string, actorDirs *ateworkerpb.ActorDirs) error { old := s.lookupActor(actorUID) - if s.runningVM(actorUID) != nil { + if ra := s.runningVM(actorUID); ra != nil { if old != nil { s.readFinal(ctx, old) } - if err := s.stopActorVM(ctx, actorUID, actorDirs); err != nil { + if err := s.teardownActor(ctx, actorUID, actorDirs, ra); err != nil { return fmt.Errorf("while stopping the actor's previous micro-VM: %w", err) } } @@ -455,16 +443,6 @@ func (s *AteWorkerService) endPreviousActivation(ctx context.Context, actorUID s return nil } -// stopActorVM tears down the actor's micro-VM, if any, keeping it hosted. -func (s *AteWorkerService) stopActorVM(ctx context.Context, actorUID string, actorDirs *ateworkerpb.ActorDirs) error { - ra := s.runningVM(actorUID) - chSocket := kata.CLHSocketPath(actorUID) - if ra != nil && ra.apiSocket != "" { - chSocket = ra.apiSocket - } - return s.teardownActor(ctx, actorUID, actorDirs, ra, ch.NewClient(chSocket)) -} - func (s *AteWorkerService) terminateWorkload(ctx context.Context, actor resources.ActorAttribution, actorDirs *ateworkerpb.ActorDirs) error { var errs []error if err := s.tunnel.Deactivate(ctx, actor); err != nil { @@ -472,7 +450,7 @@ func (s *AteWorkerService) terminateWorkload(ctx context.Context, actor resource } actorUID := actor.UID - if err := s.stopActorVM(ctx, actorUID, actorDirs); err != nil { + if err := s.teardownActor(ctx, actorUID, actorDirs, s.runningVM(actorUID)); err != nil { errs = append(errs, fmt.Errorf("while tearing down actor: %w", err)) } // Remove attribution after teardown; a failed checkpoint may leave the VM running. diff --git a/cmd/ateworker-microvm/internal/ch/ch.go b/cmd/ateworker-microvm/internal/ch/ch.go index 38be196b03..109f10ab74 100644 --- a/cmd/ateworker-microvm/internal/ch/ch.go +++ b/cmd/ateworker-microvm/internal/ch/ch.go @@ -125,12 +125,6 @@ func (c *Client) Snapshot(ctx context.Context, destDir string) error { return c.api.put(ctx, "/api/v1/vm.snapshot", snapshotConfig{DestinationURL: SnapshotURL(destDir)}) } -// Shutdown best-effort tears down the VM and the VMM process behind the socket. -func (c *Client) Shutdown(ctx context.Context) error { - _ = c.api.put(ctx, "/api/v1/vm.shutdown", nil) - return c.api.put(ctx, "/api/v1/vmm.shutdown", nil) -} - // SnapshotURL returns the file:// URL cloud-hypervisor expects for a snapshot // destination or restore source directory. func SnapshotURL(dir string) string { return "file://" + dir } diff --git a/cmd/ateworker-microvm/restore.go b/cmd/ateworker-microvm/restore.go index 09798cc3ff..25243ab3dc 100644 --- a/cmd/ateworker-microvm/restore.go +++ b/cmd/ateworker-microvm/restore.go @@ -67,17 +67,27 @@ func restoreMemMode(ctx context.Context, info ch.VMMInfo) string { return ch.MemRestoreEager } +// reseedTimeout bounds ReseedRandomDev. It needs to cover a whole resume wave, not a +// single quiet guest. +const reseedTimeout = 60 * time.Second + +// randomDevReseeder is the part of *kata.AgentClient that reseedGuestCRNG +// uses, so tests can fake it. +type randomDevReseeder interface { + ReseedRandomDev(ctx context.Context, data []byte) error +} + // reseedGuestCRNG mixes fresh, per-restore entropy into the restored guest's kernel // CRNG through the kata-agent (see the call site in restoreMemoryFidelity for why a restore // needs this). The nonce is a throwaway; its only job is to differ between restores so // that clones of one snapshot diverge instead of producing identical randomness. The // caller owns ac: it stays open for log forwarding and guest stats. -func reseedGuestCRNG(ctx context.Context, ac *kata.AgentClient) error { +func reseedGuestCRNG(ctx context.Context, ac randomDevReseeder) error { nonce, err := newReseedNonce() if err != nil { return err } - rctx, cancel := context.WithTimeout(ctx, 5*time.Second) + rctx, cancel := context.WithTimeout(ctx, reseedTimeout) defer cancel() return ac.ReseedRandomDev(rctx, nonce) } @@ -142,7 +152,7 @@ func (s *AteWorkerService) RestoreWorkload(ctx context.Context, req *ateworkerpb containers: req.GetSpec().GetContainers(), assetPaths: req.GetRuntimeAssetPaths(), egressGateway: req.GetEgressGateway(), - size: sizing.FromLimits(req.GetCpuMilli(), req.GetMemoryBytes()), + size: sizing.FromLimits(req.GetResources().GetLimits()), } restoreDir := p.actorDirs.GetRestoreDir() durableDir := p.actorDirs.GetDurableDirVolumeMountsDir() @@ -447,6 +457,7 @@ func (s *AteWorkerService) restoreMemoryFidelity(ctx context.Context, p actorBoo if err := reseedGuestCRNG(ctx, guestAC); err != nil { return fmt.Errorf("while reseeding guest CRNG: %w", err) } + tReseeded := time.Now() // Block until every wakeup-probe-enabled container reports 200. if err := wakeupprobe.WaitAll(ctx, containers, actornet.ActorVethIP, wakeupprobe.DialFunc(s.sandboxDialer(actorUID))); err != nil { @@ -456,7 +467,8 @@ func (s *AteWorkerService) restoreMemoryFidelity(ctx context.Context, p actorBoo // Where a resume goes. Like the boot phases, this used to be a single total, // which hid that a first (cold) restore and a later (warm) one differ by more // than 5x on the same actor. upper/lowers is the host reassembling the rootfs; - // vm_restore is cloud-hypervisor reading guest RAM back. + // vm_restore is cloud-hypervisor reading guest RAM back; reseed is the kata-agent dial + // and the CRNG reseed, the head of wakeup_probe. dWakeupProbe := time.Since(tResume) dTotal := time.Since(tStart) slog.InfoContext(ctx, "Actor restore phases", slog.String("id", actorUID), @@ -469,6 +481,7 @@ func (s *AteWorkerService) restoreMemoryFidelity(ctx context.Context, p actorBoo slog.Duration("vmm_launch", tLaunch.Sub(tTap)), slog.Duration("vm_restore", tVMRestore.Sub(tLaunch)), slog.Duration("resume", tResume.Sub(tVMRestore)), + slog.Duration("reseed", tReseeded.Sub(tResume)), slog.Duration("wakeup_probe", dWakeupProbe), slog.Duration("total", dTotal)) // The joinable per-actor record the benchmarking tooling aggregates. The diff --git a/cmd/ateworker-microvm/restore_test.go b/cmd/ateworker-microvm/restore_test.go index 8572dba7a1..926885a76b 100644 --- a/cmd/ateworker-microvm/restore_test.go +++ b/cmd/ateworker-microvm/restore_test.go @@ -20,6 +20,7 @@ import ( "bytes" "context" "encoding/json" + "errors" "os" "path/filepath" "strings" @@ -490,6 +491,54 @@ func TestNewReseedNonce(t *testing.T) { } } +type fakeReseeder func(ctx context.Context, data []byte) error + +func (f fakeReseeder) ReseedRandomDev(ctx context.Context, data []byte) error { + return f(ctx, data) +} + +func TestReseedGuestCRNG(t *testing.T) { + t.Run("sends a fresh nonce once, bounded by reseedTimeout", func(t *testing.T) { + var calls int + var gotNonce []byte + var gotDeadline time.Time + start := time.Now() + err := reseedGuestCRNG(t.Context(), fakeReseeder(func(ctx context.Context, data []byte) error { + calls++ + gotNonce = append([]byte(nil), data...) + gotDeadline, _ = ctx.Deadline() + return nil + })) + if err != nil { + t.Fatalf("reseedGuestCRNG: %v", err) + } + if calls != 1 { + t.Errorf("calls = %d, want 1", calls) + } + if len(gotNonce) != 32 { + t.Errorf("nonce length = %d, want 32", len(gotNonce)) + } + if gotDeadline.IsZero() || gotDeadline.Before(start.Add(reseedTimeout)) || gotDeadline.After(time.Now().Add(reseedTimeout)) { + t.Errorf("call deadline = %v, want about %v after the call started at %v", gotDeadline, reseedTimeout, start) + } + }) + + t.Run("returns the agent's error without retrying", func(t *testing.T) { + wantErr := errors.New("agent busy") + var calls int + err := reseedGuestCRNG(t.Context(), fakeReseeder(func(context.Context, []byte) error { + calls++ + return wantErr + })) + if !errors.Is(err, wantErr) { + t.Errorf("reseedGuestCRNG error = %v, want %v", err, wantErr) + } + if calls != 1 { + t.Errorf("calls = %d, want 1", calls) + } + }) +} + func TestRPCsRejectUntrustedRuntimeAssetPaths(t *testing.T) { s := &AteWorkerService{} ctx := context.Background() diff --git a/cmd/ateworker-microvm/run.go b/cmd/ateworker-microvm/run.go index 6bac045d13..33f5bd913e 100644 --- a/cmd/ateworker-microvm/run.go +++ b/cmd/ateworker-microvm/run.go @@ -268,7 +268,7 @@ func (s *AteWorkerService) RunWorkload(ctx context.Context, req *ateworkerpb.Run containers: req.GetSpec().GetContainers(), assetPaths: req.GetRuntimeAssetPaths(), egressGateway: req.GetEgressGateway(), - size: sizing.FromLimits(req.GetCpuMilli(), req.GetMemoryBytes()), + size: sizing.FromLimits(req.GetResources().GetLimits()), } attribution := p.actorAttribution() diff --git a/cmd/benchmarking/isolate/fake-atelet/relay_test.go b/cmd/benchmarking/isolate/fake-atelet/relay_test.go index ba0d2c51ca..5e44dea2c2 100644 --- a/cmd/benchmarking/isolate/fake-atelet/relay_test.go +++ b/cmd/benchmarking/isolate/fake-atelet/relay_test.go @@ -49,11 +49,8 @@ func report() *ateapipb.RegisterWorkerRequest { return &ateapipb.RegisterWorkerRequest{ Worker: &ateapipb.ObjectRef{Name: "fake-r1-c52b9bd3-0"}, Capacity: &ateapipb.WorkerResources{ - Actors: 1000, - Resources: &ateapipb.Resources{Limits: []*ateapipb.Limits{ - {Name: "cpu", Quantity: "64"}, - {Name: "memory", Quantity: "256Gi"}, - }}, + Actors: 1000, + CpuMillicores: 64000, MemoryBytes: 256 << 30, }, } } diff --git a/cmd/benchmarking/isolate/fake-workersync/controller.go b/cmd/benchmarking/isolate/fake-workersync/controller.go index 22a7d161dd..23e76e6ed0 100644 --- a/cmd/benchmarking/isolate/fake-workersync/controller.go +++ b/cmd/benchmarking/isolate/fake-workersync/controller.go @@ -30,7 +30,6 @@ import ( "github.com/google/uuid" "go.ate.dev/substrate/cmd/benchmarking/isolate/internal/fakeworker" "go.ate.dev/substrate/internal/hardware" - "go.ate.dev/substrate/internal/resources" atev1alpha1 "go.ate.dev/substrate/pkg/api/v1alpha1" "go.ate.dev/substrate/pkg/proto/ateapipb" "golang.org/x/sync/errgroup" @@ -637,7 +636,7 @@ func (t template) capacity(allocatable corev1.ResourceList) *ateapipb.WorkerReso if memory == 0 { memory = allocatable.Memory().Value() } - return &ateapipb.WorkerResources{Actors: int32(t.actors), Resources: resources.CPUMemory(cpu, memory)} + return &ateapipb.WorkerResources{Actors: int32(t.actors), CpuMillicores: cpu, MemoryBytes: memory} } // fits reports whether w was made from wp's current template, so a real pool diff --git a/cmd/benchmarking/isolate/fake-workersync/controller_test.go b/cmd/benchmarking/isolate/fake-workersync/controller_test.go index 543f5d49f2..f8c74bc81f 100644 --- a/cmd/benchmarking/isolate/fake-workersync/controller_test.go +++ b/cmd/benchmarking/isolate/fake-workersync/controller_test.go @@ -414,9 +414,8 @@ func poolNames(name string, n int) []string { } func wantCapacity(actors int32, cpu, mem string) *ateapipb.WorkerResources { - return &ateapipb.WorkerResources{Actors: actors, Resources: &ateapipb.Resources{Limits: []*ateapipb.Limits{ - {Name: "cpu", Quantity: cpu}, {Name: "memory", Quantity: mem}, - }}} + c, m := resource.MustParse(cpu), resource.MustParse(mem) + return &ateapipb.WorkerResources{Actors: actors, CpuMillicores: c.MilliValue(), MemoryBytes: m.Value()} } func TestReconcileHonorsReplicasAndLimits(t *testing.T) { @@ -1223,7 +1222,7 @@ func TestCapacityFormatsLikeWorker(t *testing.T) { if want := wantCapacity(1000, "500m", "1e9"); !proto.Equal(got, want) { t.Errorf("capacity = %v, want %v", got, want) } - if none := capacity(pool("p", 1, nil)); none.GetResources() != nil || none.GetActors() != 1000 { + if none := capacity(pool("p", 1, nil)); none.GetCpuMillicores() != 0 || none.GetMemoryBytes() != 0 || none.GetActors() != 1000 { t.Errorf("no limits and no allocatable: %v; want 1000 actors and no resources", none) } } diff --git a/cmd/kubectl-ate/README.md b/cmd/kubectl-ate/README.md index 37c2233135..1f6ba2c12d 100644 --- a/cmd/kubectl-ate/README.md +++ b/cmd/kubectl-ate/README.md @@ -111,7 +111,7 @@ kubectl ate get workers -l |---|---| | `ATESPACE` | The atespace the actor belongs to. Part of the actor's identity; folded into the storage key as `actor::`. | | `NAME` | The actor's name. User-provided for application actors; UUID for the golden actor that each template materialises while building its golden tag. | -| `TEMPLATE` | The `ActorTemplate` the actor was created from, displayed as `/`. | +| `TEMPLATE` | The `ActorTemplate` the actor references, displayed as `/`. | | `STATE` | Current lifecycle state (`ACTOR_STATE_RESUMING`, `ACTOR_STATE_RUNNING`, `ACTOR_STATE_SUSPENDING`, `ACTOR_STATE_SUSPENDED`, `ACTOR_STATE_CRASHED`, `ACTOR_STATE_DELETING`, `ACTOR_STATE_REVERTING`). | | `WORKER POD` | The worker pod (namespace/name) currently hosting the actor. Empty while suspended. | | `WORKER IP` | The pod IP of that worker. Empty while suspended. | @@ -160,7 +160,8 @@ kubectl ate delete atespace An **actor template** describes what an actor runs: containers, volumes, snapshot policy, and worker selection. Templates live in an atespace and are immutable — there is no update; delete and recreate to change -one. +one (or create a new template and repoint suspended actors to it with +`kubectl ate update actor --template`). ```bash # Create a template from a manifest (protojson-shaped ateapipb.ActorTemplate, @@ -198,6 +199,11 @@ Manage the execution state of your workloads. # Create a new actor from an ActorTemplate. kubectl ate create actor my-actor --template= -a +# Update an actor's template (requires the actor to be SUSPENDED) or worker selector. +kubectl ate update actor my-actor --template= -a +kubectl ate update actor my-actor --worker-selector=tier=paid,env=prod -a +kubectl ate update actor my-actor --worker-selector="" -a + # Resume an actor (assigns it to a free worker and restores its state) kubectl ate resume actor my-actor -a diff --git a/cmd/kubectl-ate/internal/cmd/actor.go b/cmd/kubectl-ate/internal/cmd/actor.go index b83f54b574..1304adb860 100644 --- a/cmd/kubectl-ate/internal/cmd/actor.go +++ b/cmd/kubectl-ate/internal/cmd/actor.go @@ -37,24 +37,30 @@ import ( "google.golang.org/grpc/codes" "google.golang.org/grpc/status" corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/validate/content" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/client-go/kubernetes" ) +const maxWorkerSelectorLabels = 10 + var ( - getActorAtespaceFlag string - getActorAllAtespacesFlag bool - createActorAtespaceFlag string - createActorTemplateFlag string - createActorTagFlag string - deleteActorAtespaceFlag string - resumeActorAtespaceFlag string - suspendActorAtespaceFlag string - suspendActorModeFlag string - logsActorAtespaceFlag string - logsActorFollowFlag bool - logsActorContainerFlag string - logsActorSourceFlag string + getActorAtespaceFlag string + getActorAllAtespacesFlag bool + createActorAtespaceFlag string + createActorTemplateFlag string + createActorTagFlag string + updateActorAtespaceFlag string + updateActorTemplateFlag string + updateActorWorkerSelectorFlag string + deleteActorAtespaceFlag string + resumeActorAtespaceFlag string + suspendActorAtespaceFlag string + suspendActorModeFlag string + logsActorAtespaceFlag string + logsActorFollowFlag bool + logsActorContainerFlag string + logsActorSourceFlag string ) var getActorsCmd = &cobra.Command{ @@ -138,6 +144,59 @@ var createActorCmd = &cobra.Command{ }, } +var updateActorCmd = &cobra.Command{ + Use: "actor ", + Short: "Update an actor", + Args: exactlyOneArg("", "atespace"), + RunE: func(cmd *cobra.Command, args []string) error { + templateSet := cmd.Flags().Changed("template") + workerSelectorSet := cmd.Flags().Changed("worker-selector") + if !templateSet && !workerSelectorSet { + return usageErrorf("at least one of --template or --worker-selector is required") + } + + var templateRef *ateapipb.ObjectRef + if templateSet { + ref, err := parseAtespacedName(updateActorTemplateFlag, updateActorAtespaceFlag) + if err != nil { + return wrapUsageError(err) + } + templateRef = ref + } + + var selector *ateapipb.Selector + if workerSelectorSet { + s, err := parseWorkerSelector(updateActorWorkerSelectorFlag) + if err != nil { + return wrapUsageError(err) + } + selector = s + } + + ctx := cmd.Context() + apiClient, err := newAPIClient(cmd) + if err != nil { + return fmt.Errorf("failed to connect to ate-api-server: %w", err) + } + defer apiClient.Close() + + actorRef := resources.ActorRef{Atespace: updateActorAtespaceFlag, Name: args[0]} + resp, err := updateActor(ctx, apiClient, actorRef.ToObjectRef(), func(actor *ateapipb.Actor) { + if templateSet { + actor.ActorTemplate = templateRef + } + if workerSelectorSet { + actor.WorkerSelector = selector // nil clears the selector + } + }) + if err != nil { + return err + } + + return printer.PrintActorTo(cmd.OutOrStdout(), resp, outputFmt) + }, +} + var deleteActorCmd = &cobra.Command{ Use: "actor ", Short: "Delete an actor", @@ -269,6 +328,55 @@ func buildCreateActorRequest(actorName, atespace, template, tag string) (*ateapi return &ateapipb.CreateActorRequest{Actor: actor}, nil } +type actorUpdater interface { + GetActor(ctx context.Context, in *ateapipb.GetActorRequest, opts ...grpc.CallOption) (*ateapipb.Actor, error) + UpdateActor(ctx context.Context, in *ateapipb.UpdateActorRequest, opts ...grpc.CallOption) (*ateapipb.Actor, error) +} + +func updateActor(ctx context.Context, client actorUpdater, ref *ateapipb.ObjectRef, mutate func(*ateapipb.Actor)) (*ateapipb.Actor, error) { + actor, err := client.GetActor(ctx, &ateapipb.GetActorRequest{Actor: ref}) + if err != nil { + return nil, fmt.Errorf("failed to get actor %q: %w", ref.GetName(), err) + } + mutate(actor) + + resp, err := client.UpdateActor(ctx, &ateapipb.UpdateActorRequest{Actor: actor}) + if err != nil { + return nil, fmt.Errorf("failed to update actor: %w", err) + } + return resp, nil +} + +// parseWorkerSelector parses a comma-separated key=value label selector into a +// Selector proto. An empty string returns nil, which clears the actor's +// worker_selector on update. +func parseWorkerSelector(raw string) (*ateapipb.Selector, error) { + if raw == "" { + return nil, nil + } + matchLabels := make(map[string]string) + for entry := range strings.SplitSeq(raw, ",") { + key, val, ok := strings.Cut(entry, "=") + if !ok { + return nil, fmt.Errorf("invalid worker selector %q: entry %q must be in key=value format", raw, entry) + } + if errs := content.IsLabelKey(key); len(errs) > 0 { + return nil, fmt.Errorf("invalid worker selector %q: invalid label key %q: %s", raw, key, strings.Join(errs, "; ")) + } + if errs := content.IsLabelValue(val); len(errs) > 0 { + return nil, fmt.Errorf("invalid worker selector %q: invalid label value %q for key %q: %s", raw, val, key, strings.Join(errs, "; ")) + } + if _, exists := matchLabels[key]; exists { + return nil, fmt.Errorf("invalid worker selector %q: duplicate label key %q", raw, key) + } + matchLabels[key] = val + } + if len(matchLabels) > maxWorkerSelectorLabels { + return nil, fmt.Errorf("invalid worker selector %q: at most %d labels are allowed, got %d", raw, maxWorkerSelectorLabels, len(matchLabels)) + } + return &ateapipb.Selector{MatchLabels: matchLabels}, nil +} + // AteAPIClient abstracts the gRPC client calls. type AteAPIClient interface { GetActor(ctx context.Context, in *ateapipb.GetActorRequest, opts ...grpc.CallOption) (*ateapipb.Actor, error) @@ -744,6 +852,11 @@ func init() { createActorCmd.Flags().StringVar(&createActorTagFlag, "tag", "", "tag to initialize the actor from, as /, or just to use the actor's own atespace (see --atespace)") createCmd.AddCommand(createActorCmd) + updateActorCmd.Flags().StringVarP(&updateActorAtespaceFlag, "atespace", "a", "", "the actor's atespace") + updateActorCmd.Flags().StringVar(&updateActorTemplateFlag, "template", "", "actor-template to repoint the actor to, as /, or just to use the actor's own atespace (see --atespace)") + updateActorCmd.Flags().StringVar(&updateActorWorkerSelectorFlag, "worker-selector", "", "comma-separated key=value label selector for worker pools, or empty string to clear") + updateCmd.AddCommand(updateActorCmd) + deleteActorCmd.Flags().StringVarP(&deleteActorAtespaceFlag, "atespace", "a", "", "the actor's atespace") deleteCmd.AddCommand(deleteActorCmd) diff --git a/cmd/kubectl-ate/internal/cmd/actor_test.go b/cmd/kubectl-ate/internal/cmd/actor_test.go index a6d0954687..4c5db51c3e 100644 --- a/cmd/kubectl-ate/internal/cmd/actor_test.go +++ b/cmd/kubectl-ate/internal/cmd/actor_test.go @@ -30,6 +30,7 @@ import ( "google.golang.org/grpc" "google.golang.org/grpc/codes" "google.golang.org/grpc/status" + "google.golang.org/protobuf/proto" "google.golang.org/protobuf/testing/protocmp" corev1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" @@ -173,6 +174,221 @@ func TestSuspendActorModeFlagDefault(t *testing.T) { } } +func TestParseWorkerSelector(t *testing.T) { + tests := []struct { + name string + raw string + want *ateapipb.Selector + wantErr string + }{ + { + name: "empty string clears selector", + raw: "", + want: nil, + }, + { + name: "single label", + raw: "tier=paid", + want: &ateapipb.Selector{MatchLabels: map[string]string{"tier": "paid"}}, + }, + { + name: "multiple labels with prefixed key and empty value", + raw: "ate.dev/pool=fast,tier=paid,empty=", + want: &ateapipb.Selector{MatchLabels: map[string]string{ + "ate.dev/pool": "fast", + "tier": "paid", + "empty": "", + }}, + }, + { + name: "exactly 10 labels is allowed", + raw: "k1=v1,k2=v2,k3=v3,k4=v4,k5=v5,k6=v6,k7=v7,k8=v8,k9=v9,k10=v10", + want: &ateapipb.Selector{MatchLabels: map[string]string{ + "k1": "v1", "k2": "v2", "k3": "v3", "k4": "v4", "k5": "v5", + "k6": "v6", "k7": "v7", "k8": "v8", "k9": "v9", "k10": "v10", + }}, + }, + { + name: "more than 10 labels is rejected", + raw: "k1=v1,k2=v2,k3=v3,k4=v4,k5=v5,k6=v6,k7=v7,k8=v8,k9=v9,k10=v10,k11=v11", + wantErr: "at most 10 labels are allowed, got 11", + }, + { + name: "missing equals sign", + raw: "tier", + wantErr: `entry "tier" must be in key=value format`, + }, + { + name: "trailing comma produces empty entry", + raw: "tier=paid,", + wantErr: `entry "" must be in key=value format`, + }, + { + name: "duplicate key", + raw: "tier=free,tier=paid", + wantErr: `duplicate label key "tier"`, + }, + { + name: "invalid label key", + raw: "bad key!=paid", + wantErr: `invalid label key "bad key!"`, + }, + { + name: "invalid label value", + raw: "tier=not valid!", + wantErr: `invalid label value "not valid!" for key "tier"`, + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + got, err := parseWorkerSelector(tc.raw) + if tc.wantErr != "" { + if err == nil || !strings.Contains(err.Error(), tc.wantErr) { + t.Fatalf("parseWorkerSelector(%q) error = %v, want substring %q", tc.raw, err, tc.wantErr) + } + return + } + if err != nil { + t.Fatalf("parseWorkerSelector(%q) unexpected error: %v", tc.raw, err) + } + if diff := cmp.Diff(tc.want, got, protocmp.Transform()); diff != "" { + t.Errorf("parseWorkerSelector(%q) mismatch (-want +got):\n%s", tc.raw, diff) + } + }) + } +} + +type fakeActorUpdater struct { + actor *ateapipb.Actor + getErr error + updateErr error + + getCalls int + updateReq *ateapipb.UpdateActorRequest +} + +func (f *fakeActorUpdater) GetActor(_ context.Context, _ *ateapipb.GetActorRequest, _ ...grpc.CallOption) (*ateapipb.Actor, error) { + f.getCalls++ + if f.getErr != nil { + return nil, f.getErr + } + return proto.Clone(f.actor).(*ateapipb.Actor), nil +} + +func (f *fakeActorUpdater) UpdateActor(_ context.Context, in *ateapipb.UpdateActorRequest, _ ...grpc.CallOption) (*ateapipb.Actor, error) { + f.updateReq = in + if f.updateErr != nil { + return nil, f.updateErr + } + updated := proto.Clone(in.GetActor()).(*ateapipb.Actor) + updated.Metadata.Version++ + return updated, nil +} + +func TestUpdateActor(t *testing.T) { + baseActor := func() *ateapipb.Actor { + return &ateapipb.Actor{ + Metadata: &ateapipb.ResourceMetadata{ + Atespace: "demo", + Name: "my-counter", + Uid: "8ba9b6ee-2e1a-4c0f-9f4e-2f0a0f1c3d55", + Version: 3, + }, + ActorTemplate: &ateapipb.ObjectRef{Atespace: "demo", Name: "counter-v1"}, + WorkerSelector: &ateapipb.Selector{MatchLabels: map[string]string{"tier": "free"}}, + SourceTag: &ateapipb.ObjectRef{Atespace: "demo", Name: "seed-tag"}, + Status: &ateapipb.ActorStatus{State: ateapipb.ActorState_ACTOR_STATE_SUSPENDED}, + } + } + + tests := []struct { + name string + mutate func(*ateapipb.Actor) + wantTemplate *ateapipb.ObjectRef + wantSelector *ateapipb.Selector + }{ + { + name: "repoint template keeps worker selector", + mutate: func(a *ateapipb.Actor) { + a.ActorTemplate = &ateapipb.ObjectRef{Atespace: "shared", Name: "counter-v2"} + }, + wantTemplate: &ateapipb.ObjectRef{Atespace: "shared", Name: "counter-v2"}, + wantSelector: &ateapipb.Selector{MatchLabels: map[string]string{"tier": "free"}}, + }, + { + name: "update worker selector keeps template", + mutate: func(a *ateapipb.Actor) { + a.WorkerSelector = &ateapipb.Selector{MatchLabels: map[string]string{"tier": "paid", "env": "prod"}} + }, + wantTemplate: &ateapipb.ObjectRef{Atespace: "demo", Name: "counter-v1"}, + wantSelector: &ateapipb.Selector{MatchLabels: map[string]string{"tier": "paid", "env": "prod"}}, + }, + { + name: "nil worker selector clears existing selector", + mutate: func(a *ateapipb.Actor) { a.WorkerSelector = nil }, + wantTemplate: &ateapipb.ObjectRef{Atespace: "demo", Name: "counter-v1"}, + wantSelector: nil, + }, + { + name: "update both template and worker selector", + mutate: func(a *ateapipb.Actor) { + a.ActorTemplate = &ateapipb.ObjectRef{Atespace: "demo", Name: "counter-v2"} + a.WorkerSelector = &ateapipb.Selector{MatchLabels: map[string]string{"tier": "paid"}} + }, + wantTemplate: &ateapipb.ObjectRef{Atespace: "demo", Name: "counter-v2"}, + wantSelector: &ateapipb.Selector{MatchLabels: map[string]string{"tier": "paid"}}, + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + client := &fakeActorUpdater{actor: baseActor()} + ref := &ateapipb.ObjectRef{Atespace: "demo", Name: "my-counter"} + + got, err := updateActor(context.Background(), client, ref, tc.mutate) + if err != nil { + t.Fatalf("updateActor() unexpected error: %v", err) + } + if client.getCalls != 1 { + t.Errorf("GetActor called %d times, want 1", client.getCalls) + } + + want := baseActor() + want.ActorTemplate = tc.wantTemplate + want.WorkerSelector = tc.wantSelector + want.Metadata.Version = 4 + if diff := cmp.Diff(want, got, protocmp.Transform()); diff != "" { + t.Errorf("updated actor mismatch (-want +got):\n%s", diff) + } + }) + } + + t.Run("GetActor error is propagated", func(t *testing.T) { + client := &fakeActorUpdater{getErr: status.Error(codes.NotFound, "not found")} + ref := &ateapipb.ObjectRef{Atespace: "demo", Name: "my-counter"} + _, err := updateActor(context.Background(), client, ref, func(*ateapipb.Actor) {}) + if status.Code(err) != codes.NotFound { + t.Fatalf("updateActor() error = %v, want NotFound", err) + } + if client.updateReq != nil { + t.Errorf("UpdateActor called after GetActor failed") + } + }) + + t.Run("UpdateActor error is propagated", func(t *testing.T) { + client := &fakeActorUpdater{ + actor: baseActor(), + updateErr: status.Error(codes.FailedPrecondition, "actor must be SUSPENDED"), + } + ref := &ateapipb.ObjectRef{Atespace: "demo", Name: "my-counter"} + _, err := updateActor(context.Background(), client, ref, func(*ateapipb.Actor) {}) + if status.Code(err) != codes.FailedPrecondition { + t.Fatalf("updateActor() error = %v, want FailedPrecondition", err) + } + }) +} + func TestFilterAndDisplayLogLine(t *testing.T) { tests := []struct { name string diff --git a/cmd/kubectl-ate/internal/cmd/actortemplate_test.go b/cmd/kubectl-ate/internal/cmd/actortemplate_test.go index c84f9c4d46..0374fdb129 100644 --- a/cmd/kubectl-ate/internal/cmd/actortemplate_test.go +++ b/cmd/kubectl-ate/internal/cmd/actortemplate_test.go @@ -52,10 +52,8 @@ containers: mountPath: /home/counter resources: limits: - - name: cpu - quantity: "1" - - name: memory - quantity: 512Mi + cpuMillicores: 1000 + memoryBytes: 536870912 snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://ate-snapshots/ate-demo-counter/ @@ -82,10 +80,7 @@ func TestActorTemplateFromManifest(t *testing.T) { }, VolumeMounts: []*ateapipb.VolumeMount{{Name: "data", MountPath: "/home/counter"}}, }}, - Resources: &ateapipb.Resources{Limits: []*ateapipb.Limits{ - {Name: "cpu", Quantity: "1"}, - {Name: "memory", Quantity: "512Mi"}, - }}, + Resources: &ateapipb.ResourceRequirements{Limits: &ateapipb.Resources{CpuMillicores: 1000, MemoryBytes: 512 << 20}}, SnapshotConfig: &ateapipb.SnapshotConfig{ PreferredFidelity: ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, StorageLocation: "gs://ate-snapshots/ate-demo-counter/", @@ -125,6 +120,7 @@ func TestActorTemplateFromManifest_Errors(t *testing.T) { {name: "empty", manifest: ""}, {name: "unknown field", manifest: "metadata: {atespace: a, name: n}\nsandboxClass: gvisor\n"}, {name: "bad enum", manifest: "snapshotConfig: {preferredFidelity: memory}\n"}, + {name: "resources limits list", manifest: "resources: {limits: [{name: cpu, quantity: \"1\"}]}\n"}, {name: "crd shape", manifest: "apiVersion: ate.dev/v1alpha1\nkind: ActorTemplate\nmetadata: {name: counter}\n"}, {name: "not yaml", manifest: "\t{"}, } diff --git a/cmd/kubectl-ate/internal/cmd/validation_test.go b/cmd/kubectl-ate/internal/cmd/validation_test.go index ec8dcf66f9..5e9ebbbe19 100644 --- a/cmd/kubectl-ate/internal/cmd/validation_test.go +++ b/cmd/kubectl-ate/internal/cmd/validation_test.go @@ -198,6 +198,18 @@ func TestCLIUsageErrorsGoToStderrWithUsage(t *testing.T) { wantErrorLine: `Error: expected 1 argument, received 2 ("a1", "a2")`, wantUsageLine: "kubectl-ate create actor [flags]", }, + { + name: "update actor with nothing", + args: []string{"update", "actor"}, + wantErrorLine: "Error: missing required argument and required flag -a/--atespace", + wantUsageLine: "kubectl-ate update actor [flags]", + }, + { + name: "update actor missing -a", + args: []string{"update", "actor", "my-actor", "--template", "counter"}, + wantErrorLine: "Error: missing required flag -a/--atespace", + wantUsageLine: "kubectl-ate update actor [flags]", + }, { name: "delete actor with nothing", args: []string{"delete", "actor"}, @@ -398,6 +410,24 @@ func TestCLIUsageErrorsGoToStderrWithUsage(t *testing.T) { wantErrorLine: `Error: invalid --source "bogus"`, wantUsageLine: "kubectl-ate logs actor [flags]", }, + { + name: "update actor without --template or --worker-selector", + args: []string{"update", "actor", "my-actor", "-a", "space-1"}, + wantErrorLine: "Error: at least one of --template or --worker-selector is required", + wantUsageLine: "kubectl-ate update actor [flags]", + }, + { + name: "update actor with malformed --template", + args: []string{"update", "actor", "my-actor", "-a", "space-1", "--template", "a/b/c"}, + wantErrorLine: `Error: malformed reference "a/b/c" (expected or /)`, + wantUsageLine: "kubectl-ate update actor [flags]", + }, + { + name: "update actor with invalid --worker-selector", + args: []string{"update", "actor", "my-actor", "-a", "space-1", "--worker-selector", "===bad"}, + wantErrorLine: `Error: invalid worker selector "===bad"`, + wantUsageLine: "kubectl-ate update actor [flags]", + }, { name: "create tag with invalid --scope", args: []string{"create", "tag", "t1", "-a", "space-1", "--actor", "a1", "--scope=bogus"}, diff --git a/cmd/kubectl-ate/internal/cmd/worker_test.go b/cmd/kubectl-ate/internal/cmd/worker_test.go index adc5c39d8f..0e8a927730 100644 --- a/cmd/kubectl-ate/internal/cmd/worker_test.go +++ b/cmd/kubectl-ate/internal/cmd/worker_test.go @@ -161,10 +161,10 @@ func TestGetWorkersRunner_Filters(t *testing.T) { "space-b": {actorOn("space-b", "actor-b", "worker-3")}, }} - header := "NAME POOL STATE ACTORS CPU MEMORY POD AGE\n" - row1 := "worker-1 counter WORKER_STATE_ACTIVE 1/1 - - ns-1/pod-1 5m\n" - row2 := "worker-2 other WORKER_STATE_ACTIVE 0/0 - - ns-1/pod-2 3h\n" - row3 := "worker-3 counter WORKER_STATE_ACTIVE 1/1 - - ns-2/pod-3 3d\n" + header := "NAME POOL STATE ACTORS CPU(MILLICORES) MEMORY POD AGE\n" + row1 := "worker-1 counter WORKER_STATE_ACTIVE 1/1 - - ns-1/pod-1 5m\n" + row2 := "worker-2 other WORKER_STATE_ACTIVE 0/0 - - ns-1/pod-2 3h\n" + row3 := "worker-3 counter WORKER_STATE_ACTIVE 1/1 - - ns-2/pod-3 3d\n" tests := []struct { name string @@ -178,7 +178,7 @@ func TestGetWorkersRunner_Filters(t *testing.T) { {name: "namespace", namespace: "ns-1", expected: header + row1 + row2}, {name: "atespace", atespace: "space-a", expected: header + row1}, // With no matching rows the tabwriter sizes columns to the header alone. - {name: "atespace excludes free workers", atespace: "no-such-space", expected: "NAME POOL STATE ACTORS CPU MEMORY POD AGE\n"}, + {name: "atespace excludes free workers", atespace: "no-such-space", expected: "NAME POOL STATE ACTORS CPU(MILLICORES) MEMORY POD AGE\n"}, {name: "selector", selector: "ate.dev/worker-pool=counter", expected: header + row1 + row3}, {name: "sandbox class", sandboxClass: "microvm", expected: header + row1}, {name: "sandbox class gvisor", sandboxClass: "gvisor", expected: header + row2 + row3}, @@ -230,13 +230,9 @@ func TestGetWorkersRunner_GetByName(t *testing.T) { WorkerPool: "counter", WorkerPod: "pod-1", Status: &ateapipb.WorkerStatus{ - State: ateapipb.WorkerState_WORKER_STATE_ACTIVE, - Capacity: &ateapipb.WorkerResources{Actors: 4, Resources: &ateapipb.Resources{Limits: []*ateapipb.Limits{ - {Name: "cpu", Quantity: "2"}, {Name: "memory", Quantity: "4Gi"}, - }}}, - Allocated: &ateapipb.WorkerResources{Actors: 1, Resources: &ateapipb.Resources{Limits: []*ateapipb.Limits{ - {Name: "cpu", Quantity: "500m"}, {Name: "memory", Quantity: "1Gi"}, - }}}, + State: ateapipb.WorkerState_WORKER_STATE_ACTIVE, + Capacity: &ateapipb.WorkerResources{Actors: 4, CpuMillicores: 2000, MemoryBytes: 4 << 30}, + Allocated: &ateapipb.WorkerResources{Actors: 1, CpuMillicores: 500, MemoryBytes: 1 << 30}, }, }, "worker-2": { @@ -259,9 +255,9 @@ func TestGetWorkersRunner_GetByName(t *testing.T) { t.Fatalf("Run() unexpected error: %v", err) } - expected := "NAME POOL STATE ACTORS CPU MEMORY POD AGE\n" + - "worker-1 counter WORKER_STATE_ACTIVE 1/4 500m/2 1Gi/4Gi ns-1/pod-1 5m\n" + - "worker-2 counter WORKER_STATE_DRAINING 0/0 - - ns-1/pod-2 3d\n" + expected := "NAME POOL STATE ACTORS CPU(MILLICORES) MEMORY POD AGE\n" + + "worker-1 counter WORKER_STATE_ACTIVE 1/4 500/2000 1Gi/4Gi ns-1/pod-1 5m\n" + + "worker-2 counter WORKER_STATE_DRAINING 0/0 - - ns-1/pod-2 3d\n" if diff := cmp.Diff(expected, buf.String()); diff != "" { t.Errorf("output mismatch (-want +got):\n%s", diff) } diff --git a/cmd/kubectl-ate/internal/printer/printer.go b/cmd/kubectl-ate/internal/printer/printer.go index c41c6d6663..c1fa8504c4 100644 --- a/cmd/kubectl-ate/internal/printer/printer.go +++ b/cmd/kubectl-ate/internal/printer/printer.go @@ -20,6 +20,7 @@ import ( "fmt" "io" "slices" + "strconv" "strings" "text/tabwriter" "time" @@ -28,6 +29,7 @@ import ( "google.golang.org/protobuf/encoding/protojson" "google.golang.org/protobuf/proto" "google.golang.org/protobuf/types/known/timestamppb" + "k8s.io/apimachinery/pkg/api/resource" "k8s.io/apimachinery/pkg/util/duration" "sigs.k8s.io/yaml" ) @@ -55,7 +57,7 @@ func sortActors(actors []*ateapipb.Actor) { }) } -// actorTemplateDisplay renders the template an actor was created from, in +// actorTemplateDisplay renders the template an actor references, in // "/" form. func actorTemplateDisplay(a *ateapipb.Actor) string { ref := a.GetActorTemplate() @@ -122,30 +124,23 @@ func workerActorsColumn(worker *ateapipb.Worker) string { return fmt.Sprintf("%d/%d", worker.GetStatus().GetAllocated().GetActors(), worker.GetStatus().GetCapacity().GetActors()) } -// resourceLimit returns the quantity of the named limit (e.g. "cpu", -// "memory"), or "" if it isn't set. -func resourceLimit(res *ateapipb.Resources, name string) string { - for _, limit := range res.GetLimits() { - if limit.GetName() == name { - return limit.GetQuantity() - } +// workerResourceColumn renders one resource of a Worker as allocated/capacity +// (e.g. "500/2000"). "-" if the Worker has no capacity for it, since there is +// then nothing to compare against. +func workerResourceColumn(allocated, capacity int64, format func(int64) string) string { + if capacity == 0 { + return "-" } - return "" + return format(allocated) + "/" + format(capacity) } -// workerResourceColumn renders a Worker's named resource limit as -// allocated/capacity (e.g. "500m/2"). "-" if the Worker has no capacity -// limit for that resource, since there is then nothing to compare against. -func workerResourceColumn(worker *ateapipb.Worker, name string) string { - capacity := resourceLimit(worker.GetStatus().GetCapacity().GetResources(), name) - if capacity == "" { - return "-" - } - allocated := resourceLimit(worker.GetStatus().GetAllocated().GetResources(), name) - if allocated == "" { - allocated = "0" - } - return allocated + "/" + capacity +// formatBytes renders bytes with binary suffixes, e.g. 2<<30 as "2Gi" +func formatBytes(n int64) string { + return resource.NewQuantity(n, resource.BinarySI).String() +} + +func formatInt(n int64) string { + return strconv.FormatInt(n, 10) } // PrintWorkersTo prints a slice of workers to the provided writer. @@ -156,7 +151,7 @@ func PrintWorkersTo(out io.Writer, workers []*ateapipb.Worker, format string) er return printProto(out, &ateapipb.ListWorkersResponse{Workers: workers}, format) case "table": w := tabwriter.NewWriter(out, 0, 0, 3, ' ', 0) - fmt.Fprintln(w, "NAME\tPOOL\tSTATE\tACTORS\tCPU\tMEMORY\tPOD\tAGE") + fmt.Fprintln(w, "NAME\tPOOL\tSTATE\tACTORS\tCPU(MILLICORES)\tMEMORY\tPOD\tAGE") for _, worker := range workers { name := worker.GetMetadata().GetName() pool := worker.GetWorkerPool() @@ -165,7 +160,10 @@ func PrintWorkersTo(out io.Writer, workers []*ateapipb.Worker, format string) er age := formatAge(worker.GetMetadata().GetCreateTime()) fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n", name, pool, state, workerActorsColumn(worker), - workerResourceColumn(worker, "cpu"), workerResourceColumn(worker, "memory"), + workerResourceColumn(worker.GetStatus().GetAllocated().GetCpuMillicores(), + worker.GetStatus().GetCapacity().GetCpuMillicores(), formatInt), + workerResourceColumn(worker.GetStatus().GetAllocated().GetMemoryBytes(), + worker.GetStatus().GetCapacity().GetMemoryBytes(), formatBytes), pod, age) } return w.Flush() diff --git a/cmd/kubectl-ate/internal/printer/printer_test.go b/cmd/kubectl-ate/internal/printer/printer_test.go index 3137bc69b8..d3f6e9ab88 100644 --- a/cmd/kubectl-ate/internal/printer/printer_test.go +++ b/cmd/kubectl-ate/internal/printer/printer_test.go @@ -283,13 +283,9 @@ func TestPrintWorkersTo_Table(t *testing.T) { WorkerPod: "pod-1", SandboxClass: "gvisor", Status: &ateapipb.WorkerStatus{ - State: ateapipb.WorkerState_WORKER_STATE_ACTIVE, - Capacity: &ateapipb.WorkerResources{Actors: 4, Resources: &ateapipb.Resources{Limits: []*ateapipb.Limits{ - {Name: "cpu", Quantity: "2"}, {Name: "memory", Quantity: "4Gi"}, - }}}, - Allocated: &ateapipb.WorkerResources{Actors: 1, Resources: &ateapipb.Resources{Limits: []*ateapipb.Limits{ - {Name: "cpu", Quantity: "500m"}, {Name: "memory", Quantity: "1Gi"}, - }}}, + State: ateapipb.WorkerState_WORKER_STATE_ACTIVE, + Capacity: &ateapipb.WorkerResources{Actors: 4, CpuMillicores: 2000, MemoryBytes: 4 << 30}, + Allocated: &ateapipb.WorkerResources{Actors: 1, CpuMillicores: 500, MemoryBytes: 1 << 30}, }, }, } @@ -299,8 +295,8 @@ func TestPrintWorkersTo_Table(t *testing.T) { } output := buf.String() - expected := `NAME POOL STATE ACTORS CPU MEMORY POD AGE -worker-1 pool-1 WORKER_STATE_ACTIVE 1/4 500m/2 1Gi/4Gi default/pod-1 3d + expected := `NAME POOL STATE ACTORS CPU(MILLICORES) MEMORY POD AGE +worker-1 pool-1 WORKER_STATE_ACTIVE 1/4 500/2000 1Gi/4Gi default/pod-1 3d ` if diff := cmp.Diff(expected, output); diff != "" { t.Errorf("output mismatch (-want +got):\n%s", diff) @@ -330,8 +326,8 @@ func TestPrintWorkersTo_Table_Free(t *testing.T) { } output := buf.String() - expected := `NAME POOL STATE ACTORS CPU MEMORY POD AGE - pool-1 WORKER_STATE_UNSPECIFIED 0/0 - - default/pod-1 56y + expected := `NAME POOL STATE ACTORS CPU(MILLICORES) MEMORY POD AGE + pool-1 WORKER_STATE_UNSPECIFIED 0/0 - - default/pod-1 56y ` if diff := cmp.Diff(expected, output); diff != "" { t.Errorf("output mismatch (-want +got):\n%s", diff) @@ -368,10 +364,10 @@ func TestPrintWorkersTo_Table_Sorted(t *testing.T) { t.Fatalf("unexpected error: %v", err) } - expected := `NAME POOL STATE ACTORS CPU MEMORY POD AGE -worker-a pool-1 WORKER_STATE_UNSPECIFIED 0/0 - - default/pod-a 5m -worker-z pool-1 WORKER_STATE_UNSPECIFIED 0/0 - - default/pod-z 5m -worker-o pool-2 WORKER_STATE_UNSPECIFIED 0/0 - - other/pod-1 5m + expected := `NAME POOL STATE ACTORS CPU(MILLICORES) MEMORY POD AGE +worker-a pool-1 WORKER_STATE_UNSPECIFIED 0/0 - - default/pod-a 5m +worker-z pool-1 WORKER_STATE_UNSPECIFIED 0/0 - - default/pod-z 5m +worker-o pool-2 WORKER_STATE_UNSPECIFIED 0/0 - - other/pod-1 5m ` if diff := cmp.Diff(expected, buf.String()); diff != "" { t.Errorf("output mismatch (-want +got):\n%s", diff) diff --git a/demos/autoscaled-workerpool/autoscaled-workerpool-template.yaml.tmpl b/demos/autoscaled-workerpool/autoscaled-workerpool-template.yaml.tmpl index 3eeda134bf..81558161b3 100644 --- a/demos/autoscaled-workerpool/autoscaled-workerpool-template.yaml.tmpl +++ b/demos/autoscaled-workerpool/autoscaled-workerpool-template.yaml.tmpl @@ -39,10 +39,8 @@ containers: # node's allocatable and these only need to fit the node. resources: limits: - - name: cpu - quantity: "1" - - name: memory - quantity: 512Mi + cpuMillicores: 1000 + memoryBytes: 536870912 # 512Mi snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_VOLUMES storageLocation: gs://${BUCKET_NAME}/ate-demo-autoscaled-workerpool/ diff --git a/demos/claude-code-multiplex/agent-luna-template.yaml.tmpl b/demos/claude-code-multiplex/agent-luna-template.yaml.tmpl index ce154f3d8b..0507117c0d 100644 --- a/demos/claude-code-multiplex/agent-luna-template.yaml.tmpl +++ b/demos/claude-code-multiplex/agent-luna-template.yaml.tmpl @@ -44,10 +44,8 @@ containers: # running tools. resources: limits: - - name: cpu - quantity: "1" - - name: memory - quantity: 2Gi + cpuMillicores: 1000 + memoryBytes: 2147483648 # 2Gi snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://${BUCKET_NAME}/claude-multiplex-demo/ diff --git a/demos/claude-code-multiplex/agent-mars-template.yaml.tmpl b/demos/claude-code-multiplex/agent-mars-template.yaml.tmpl index 19b48d2a47..7afab88f99 100644 --- a/demos/claude-code-multiplex/agent-mars-template.yaml.tmpl +++ b/demos/claude-code-multiplex/agent-mars-template.yaml.tmpl @@ -36,10 +36,8 @@ containers: # Sandbox size: see agent-luna-template.yaml.tmpl. resources: limits: - - name: cpu - quantity: "1" - - name: memory - quantity: 2Gi + cpuMillicores: 1000 + memoryBytes: 2147483648 # 2Gi snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://${BUCKET_NAME}/claude-multiplex-demo/ diff --git a/demos/claude-code-multiplex/agent-orion-template.yaml.tmpl b/demos/claude-code-multiplex/agent-orion-template.yaml.tmpl index 8ecc7ac2f2..5b1c886885 100644 --- a/demos/claude-code-multiplex/agent-orion-template.yaml.tmpl +++ b/demos/claude-code-multiplex/agent-orion-template.yaml.tmpl @@ -36,10 +36,8 @@ containers: # Sandbox size: see agent-luna-template.yaml.tmpl. resources: limits: - - name: cpu - quantity: "1" - - name: memory - quantity: 2Gi + cpuMillicores: 1000 + memoryBytes: 2147483648 # 2Gi snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://${BUCKET_NAME}/claude-multiplex-demo/ diff --git a/demos/counter/counter-microvm-csi-test-template.yaml b/demos/counter/counter-microvm-csi-test-template.yaml index 80eccea76a..1dfc37bc54 100644 --- a/demos/counter/counter-microvm-csi-test-template.yaml +++ b/demos/counter/counter-microvm-csi-test-template.yaml @@ -36,10 +36,8 @@ containers: # worker limits, so these only need to fit the node. resources: limits: - - name: cpu - quantity: "1" - - name: memory - quantity: 512Mi + cpuMillicores: 1000 + memoryBytes: 536870912 # 512Mi snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_VOLUMES storageLocation: gs://ate-snapshots/ate-demo-counter-microvm-csi/ @@ -47,5 +45,5 @@ volumes: - name: data external: template: - capacity: 1Gi + capacityBytes: 1073741824 # 1Gi storageClassName: csi-hostpath-sc diff --git a/demos/counter/counter-microvm-template.yaml.tmpl b/demos/counter/counter-microvm-template.yaml.tmpl index c5d7dde843..1a1dd448f2 100644 --- a/demos/counter/counter-microvm-template.yaml.tmpl +++ b/demos/counter/counter-microvm-template.yaml.tmpl @@ -52,13 +52,11 @@ containers: mountPath: /home/counter # Sandbox size: vCPUs + guest RAM at (or below) the pool's per-worker # capacity, leaving VMM-reserve headroom under the worker memory limit -# (see internal/sizing). Quantities are strings in the proto. +# (see internal/sizing). resources: limits: - - name: cpu - quantity: "1" - - name: memory - quantity: 512Mi + cpuMillicores: 1000 + memoryBytes: 536870912 # 512Mi snapshotConfig: # MEMORY, matching the gVisor variant: a suspend captures guest memory, so # the in-RAM count continues from where it left off — the round-trip this diff --git a/demos/counter/counter-template.yaml.tmpl b/demos/counter/counter-template.yaml.tmpl index fabdfe60f3..43c6b33640 100644 --- a/demos/counter/counter-template.yaml.tmpl +++ b/demos/counter/counter-template.yaml.tmpl @@ -46,13 +46,10 @@ ${VALIDATE_EXISTING_FILE_PATH_ARG} ${EXTERNAL_VOLUME_MOUNTS} # Sandbox size: an actor occupies its whole worker, so size it at (or below) # the pool's per-worker capacity (see counter.yaml.tmpl and internal/sizing). -# Quantities are strings in the proto. resources: limits: - - name: cpu - quantity: "1" - - name: memory - quantity: 512Mi + cpuMillicores: 1000 + memoryBytes: 536870912 # 512Mi snapshotConfig: # MEMORY, so a suspend captures process memory as well as the durable # volumes and the tutorial's in-memory counter really does continue diff --git a/demos/egress/egress-microvm-template.yaml.tmpl b/demos/egress/egress-microvm-template.yaml.tmpl index 888ee22b48..767b4d8450 100644 --- a/demos/egress/egress-microvm-template.yaml.tmpl +++ b/demos/egress/egress-microvm-template.yaml.tmpl @@ -63,13 +63,10 @@ volumes: - egress-mitm.ate.dev path: trust-bundle.pem # Sandbox size: ateworker applies these to the VM (see internal/sizing). -# Quantities are strings in the proto. resources: limits: - - name: cpu - quantity: "1" - - name: memory - quantity: 512Mi + cpuMillicores: 1000 + memoryBytes: 536870912 # 512Mi snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://${BUCKET_NAME}/ate-demo-egress-microvm/ diff --git a/demos/egress/egress-template.yaml.tmpl b/demos/egress/egress-template.yaml.tmpl index 23480ead7e..4d544c1443 100644 --- a/demos/egress/egress-template.yaml.tmpl +++ b/demos/egress/egress-template.yaml.tmpl @@ -57,10 +57,8 @@ volumes: # node's allocatable and these only need to fit the node. resources: limits: - - name: cpu - quantity: "1" - - name: memory - quantity: 512Mi + cpuMillicores: 1000 + memoryBytes: 536870912 # 512Mi snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://${BUCKET_NAME}/ate-demo-egress/ diff --git a/demos/jupyter/jupyter-template.yaml.tmpl b/demos/jupyter/jupyter-template.yaml.tmpl index 452d6a7b2c..66c345f8ed 100644 --- a/demos/jupyter/jupyter-template.yaml.tmpl +++ b/demos/jupyter/jupyter-template.yaml.tmpl @@ -43,10 +43,8 @@ containers: # server. resources: limits: - - name: cpu - quantity: "2" - - name: memory - quantity: 4Gi + cpuMillicores: 2000 + memoryBytes: 4294967296 # 4Gi snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://${BUCKET_NAME}/ate-demo-jupyter/ diff --git a/demos/multi-template/counter-template.yaml.tmpl b/demos/multi-template/counter-template.yaml.tmpl index ea1815b247..4cc31c5488 100644 --- a/demos/multi-template/counter-template.yaml.tmpl +++ b/demos/multi-template/counter-template.yaml.tmpl @@ -31,10 +31,8 @@ containers: # node's allocatable and these only need to fit the node. resources: limits: - - name: cpu - quantity: "1" - - name: memory - quantity: 512Mi + cpuMillicores: 1000 + memoryBytes: 536870912 # 512Mi snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://${BUCKET_NAME}/ate-demo-multi-template-counter/ diff --git a/demos/multi-template/fspersist-template.yaml.tmpl b/demos/multi-template/fspersist-template.yaml.tmpl index 4215abe360..411a091e77 100644 --- a/demos/multi-template/fspersist-template.yaml.tmpl +++ b/demos/multi-template/fspersist-template.yaml.tmpl @@ -31,10 +31,8 @@ containers: # node's allocatable and these only need to fit the node. resources: limits: - - name: cpu - quantity: "1" - - name: memory - quantity: 512Mi + cpuMillicores: 1000 + memoryBytes: 536870912 # 512Mi snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://${BUCKET_NAME}/ate-demo-multi-template-fspersist/ diff --git a/demos/parking/parking-template.yaml.tmpl b/demos/parking/parking-template.yaml.tmpl index 80c9b67f32..9d808864d2 100644 --- a/demos/parking/parking-template.yaml.tmpl +++ b/demos/parking/parking-template.yaml.tmpl @@ -30,10 +30,8 @@ containers: # node's allocatable and these only need to fit the node. resources: limits: - - name: cpu - quantity: "1" - - name: memory - quantity: 512Mi + cpuMillicores: 1000 + memoryBytes: 536870912 # 512Mi snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://${BUCKET_NAME}/ate-demo-parking/ diff --git a/demos/sandbox/manual-test-multi-template.yaml b/demos/sandbox/manual-test-multi-template.yaml index add9312307..d64dd70601 100644 --- a/demos/sandbox/manual-test-multi-template.yaml +++ b/demos/sandbox/manual-test-multi-template.yaml @@ -38,10 +38,8 @@ containers: # node's allocatable and these only need to fit the node. resources: limits: - - name: cpu - quantity: "1" - - name: memory - quantity: 512Mi + cpuMillicores: 1000 + memoryBytes: 536870912 # 512Mi snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://ate-snapshots/ate-manual-test-multi/ @@ -49,10 +47,10 @@ volumes: - name: vol-hostpath external: template: - capacity: 1Gi + capacityBytes: 1073741824 # 1Gi storageClassName: csi-hostpath-sc - name: vol-nfs external: template: - capacity: 1Gi + capacityBytes: 1073741824 # 1Gi storageClassName: csi-nfs-sc diff --git a/demos/sandbox/sandbox-template.yaml.tmpl b/demos/sandbox/sandbox-template.yaml.tmpl index 87ad72eb58..b63a1c9f36 100644 --- a/demos/sandbox/sandbox-template.yaml.tmpl +++ b/demos/sandbox/sandbox-template.yaml.tmpl @@ -30,13 +30,10 @@ containers: value: "80" # Sandbox size: an actor occupies its whole worker, so size it at (or below) # the pool's per-worker capacity (see sandbox.yaml.tmpl and internal/sizing). -# Quantities are strings in the proto. resources: limits: - - name: cpu - quantity: "2" - - name: memory - quantity: 1Gi + cpuMillicores: 2000 + memoryBytes: 1073741824 # 1Gi snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://${BUCKET_NAME}/ate-demo-sandbox/ diff --git a/docs/api-guide.md b/docs/api-guide.md index 3255e88734..0c79a76109 100644 --- a/docs/api-guide.md +++ b/docs/api-guide.md @@ -102,7 +102,7 @@ The `ActorTemplate` defines the code, environment, and state-management policies | `workerSelector` | `*Selector` | Optional. Gates which `WorkerPool`s actors from this template may use, by matching against each pool's labels (`matchLabels`). If unset, all pools are eligible (subject to the actor's own `worker_selector`). The pool decides the sandbox class, so this is also how a template pins one: if your cluster has pools of more than one sandbox class and the workload needs a particular class, make sure the selector points only at pools of that class. An actor booting with no snapshot and no selector may land on any class, and keeps it from its first checkpoint on. | | `snapshotConfig` | `SnapshotConfig` | Optional; omitted fields take their defaults. The base object-storage location snapshots are written under (a preview field, ignored without the `PerTemplateSnapshotLocation` gate), the snapshot fidelity (`preferredFidelity`) every Pause and Suspend captures, and automated golden snapshot creation (`goldenSnapshotConfig.mode`). See [Snapshot Storage Layout](#snapshot-storage-layout). | | `volumes` | `[]Volume` | Optional. Volumes the containers may mount, each a `durableDir`, an `external` CSI volume source (see [CSI Volumes Guide](csi-volumes.md)), or a `systemInfo` volume (see [SystemInfo Volumes](#systeminfo-volumes)). Every declared volume must be mounted by at least one container. A `durableDir` is created `0777`, as Kubernetes creates an emptyDir, so a container running as any user can write it. A snapshot restores it with the mode it had, so the container's own chmod survives suspend and resume. | -| `resources` | `*ResourceRequirements` | Required. Declares each actor's compute size via `limits`, which must set both `cpu` and `memory` — see [Sandbox Right-Sizing](#sandbox-right-sizing-resources). Immutable, like the rest of the template. | +| `resources` | `*ResourceRequirements` | Required. Declares each actor's compute size via `limits`, which must set both `cpuMillicores` and `memoryBytes` — see [Sandbox Right-Sizing](#sandbox-right-sizing-resources). Immutable, like the rest of the template. | The sandbox runtime and its binaries (e.g. the gVisor `runsc` binary and `pauseImage`, or a micro-VM kernel/firmware/config) are configured on the [`WorkerPool`](#1-workerpool-the-compute-foundation) via its `sandboxClasses` and the cluster-scoped [`SandboxConfig`](#3-sandboxconfig-the-sandbox-itself) it resolves, rather than on the `ActorTemplate`. An `ActorTemplate` selects its `WorkerPool`s via `workerSelector` (AND'd with the actor's `worker_selector`). When an actor runs on a worker, the worker's active `SandboxRuntime` is stamped onto the actor's assignment and snapshots so subsequent resumes prefer runtime-compatible workers. @@ -111,8 +111,8 @@ The sandbox runtime and its binaries (e.g. the gVisor `runsc` binary and `pauseI Unlike a Pod, an actor is sized by its **`limits`** (CPU and Memory): the size is a property of the template, baked into snapshots, so it lives on the immutable `ActorTemplate`. Both are required. Declared limits do two things: 1. **Size the sandbox.** The limits are supplied to the sandbox over the actor RPCs (control plane → atelet → ateworker): - - **gVisor (`ateworker-gvisor`)** — applied to the container OCI spec: `limits.cpu` sets the cgroup v2 CPU quota (`cpu.max`) and the Sentry vCPU count (`--cpu-num-from-quota`); `limits.memory` sets the cgroup v2 memory limit (`memory.max`) and bounds the virtual total memory the sandbox reports (so JVM/Go do not over-allocate from host RAM). - - **Micro-VM (`ateworker-microvm`)** — `limits.cpu` sets Cloud Hypervisor `BootVcpus` / `MaxVcpus` (rounded up to whole vCPUs); `limits.memory` sets guest RAM, reserving a small configurable margin (default 256 MiB, `--vmm-mem-reserve-mib`) for the VMM and virtiofsd so the pod cgroup does not OOM. + - **gVisor (`ateworker-gvisor`)** — applied to the container OCI spec: `limits.cpuMillicores` sets the cgroup v2 CPU quota (`cpu.max`) and the Sentry vCPU count (`--cpu-num-from-quota`); `limits.memoryBytes` sets the cgroup v2 memory limit (`memory.max`) and bounds the virtual total memory the sandbox reports (so JVM/Go do not over-allocate from host RAM). + - **Micro-VM (`ateworker-microvm`)** — `limits.cpuMillicores` sets Cloud Hypervisor `BootVcpus` / `MaxVcpus` (rounded up to whole vCPUs); `limits.memoryBytes` sets guest RAM, reserving a small configurable margin (default 256 MiB, `--vmm-mem-reserve-mib`) for the VMM and virtiofsd so the pod cgroup does not OOM. 2. **Gate scheduling.** An actor is only placed on a `WorkerPool` whose [worker capacity](#worker-capacity-spectemplateresources) is `>=` these limits. `requests` are not consulted today (an actor occupies its whole worker). Because the size is baked into snapshots, a **micro-VM MEMORY-fidelity restore reuses the size in the snapshot**; changing an actor's limits takes effect on its next cold boot. @@ -276,7 +276,7 @@ A container that exceeds its memory limit is OOM-killed on its own; the actor's Per-container limits are micro-VM only today. gVisor applies cgroup limits at the sandbox level: one sentry backs every container in the actor, so a per-container cgroup is created and then stays empty ([google/gvisor#190](https://github.com/google/gvisor/issues/190)). On a gVisor worker the per-container limits are accepted but not enforced; only the sandbox-level size applies. -These limits subdivide the sandbox that [`resources`](#sandbox-right-sizing-resources) already sized; a container that declares none is bounded by the guest as a whole, not by a copy of the actor's total. A micro-VM guest's memory is `resources.limits.memory` minus the VMM reserve, and its vCPU count is `resources.limits.cpu` rounded up to whole cores. A container limit above either ceiling can never bind, so the actor fails to start with an error naming both the limit and the ceiling. +These limits subdivide the sandbox that [`resources`](#sandbox-right-sizing-resources) already sized; a container that declares none is bounded by the guest as a whole, not by a copy of the actor's total. A micro-VM guest's memory is `resources.limits.memoryBytes` minus the VMM reserve, and its vCPU count is `resources.limits.cpuMillicores` rounded up to whole cores. A container limit above either ceiling can never bind, so the actor fails to start with an error naming both the limit and the ceiling. Each limit is validated on its own at apply, but the sum across the actor's containers is only checked when the actor first runs, against the real guest size. A template whose limits do not fit is accepted by the API server and fails on its first actor. @@ -325,10 +325,8 @@ workerSelector: # The actor's size, booked against worker capacity. resources: limits: - - name: cpu - quantity: "1" - - name: memory - quantity: 1Gi + cpuMillicores: 1000 + memoryBytes: 1073741824 # 1Gi snapshotConfig: storageLocation: gs://my-bucket/secret-agent ``` @@ -590,7 +588,7 @@ The node can cut the 30 minutes short. GKE node upgrades and cluster autoscaler ## 5. Best Practices * **Startup Logic:** Place expensive initialization (loading large models, establishing baseline connections) in your application's entry point. When golden snapshots are enabled, this initialization can be captured in the Golden Snapshot. Actors that cold-boot run initialization again. * **Placement:** The `WorkerPool` decides the sandbox class (`sandboxClasses[].name`), so use the template's `workerSelector` to target the pools whose class your workload needs — pool selection is by label match, not by namespace or RBAC. Without a selector an actor may land on any pool. -* **Version Management:** When updating code, create a new `ActorTemplate` (e.g. `v2`). Substrate treats each template as an immutable state root. +* **Version Management:** When updating code, create a new `ActorTemplate` (e.g. `v2`). Substrate treats each template as an immutable state root. Suspended actors can be repointed to the new template via `UpdateActor` (`kubectl ate update actor --template`). * **Eviction:** Give every actor a `SIGTERM` handler that does not exit, and have the actor suspended within 30 minutes of it. See [Eviction](#eviction). --- @@ -606,13 +604,16 @@ Registers a new logical actor in the system. * **Request:** `CreateActorRequest` * `actor`: `Actor` — the actor to create. Its `metadata` carries the atespace and name (name must be a DNS-1123 label); the `actor_template` ref (atespace + name) selects the `ActorTemplate`. * `actor.source_tag`: (Optional) `ObjectRef` of a `Tag` to seed the actor from. The tag must be taken under the same `ActorTemplate`, and either in the actor's own atespace or `PUBLISHED`. Nothing is copied: the new actor's `status.snapshots` points at the tag's snapshot, under the tag's prefix, until its own first suspend. + * `actor.worker_selector`: (Optional) Per-actor `WorkerSelector` merged with the template's `worker_selector` at resume time (actor keys override template keys). * **Response:** the initialized `Actor`. #### `UpdateActor` -Replaces the mutable fields of an existing actor with the ones in the request. +Replaces the mutable fields of an existing actor with the ones in the request (`kubectl ate update actor`). * **Request:** `UpdateActorRequest` * `actor`: `Actor` — the complete replacement actor. `metadata.atespace` and `metadata.name` identify the resource; `metadata.uid` and `metadata.version` are **required** preconditions. `metadata` and `status` are server-owned and whatever the request carries in them is ignored. `source_tag` is immutable. * `actor_template` may change only while the actor is `SUSPENDED`. The new template must declare the same volumes and volume mounts. Once the actor owns an external snapshot, the new template must also store snapshots under the same location: its `snapshotConfig.storageLocation` with the `PerTemplateSnapshotLocation` preview gate, otherwise ate-api-server's `--snapshot-location` (see [Snapshot Storage Layout](#snapshot-storage-layout)). + * `worker_selector` may be updated or cleared (`nil`) in any state; the updated selector takes effect on the next `ResumeActor`. + * A paused actor stays pinned to its node until atelet detaches it (see `locality` in [Snapshot Storage Layout](#snapshot-storage-layout)). Its next `ResumeActor` therefore needs a worker on that node that matches both the new template's `workerSelector` and the actor's updated `worker_selector`. If no worker there matches, `ResumeActor` returns `RESOURCE_EXHAUSTED` until the detach lifts the pin. * **Response:** the updated `Actor`. * **Errors:** `INVALID_ARGUMENT` if `uid` or `version` is unset, or if the request changes an immutable field — including by leaving one unset; `FAILED_PRECONDITION` if a change to `actor_template` breaks one of the rules above; `ABORTED` if either guard no longer matches the stored resource. diff --git a/docs/csi-volumes.md b/docs/csi-volumes.md index 645beb9369..0ae8acb403 100644 --- a/docs/csi-volumes.md +++ b/docs/csi-volumes.md @@ -84,12 +84,12 @@ volumes: - name: my-data-volume external: template: - capacity: 10Gi + capacityBytes: 10737418240 # 10Gi storageClassName: standard-rwx ``` * `name`: Unique DNS-label-compliant volume name. -* `external.template.capacity`: Quantity string representing the requested volume size (e.g. `1Gi`, `50Gi`). +* `external.template.capacityBytes`: The requested volume size in bytes (e.g. `1073741824` for 1Gi). * `external.template.storageClassName`: Name of a Kubernetes `StorageClass` present in the cluster whose `provisioner` matches a registered `CSIDriverConfig`. #### `containers[].volumeMounts[]` @@ -190,16 +190,14 @@ containers: port: 8080 resources: limits: - - name: cpu - quantity: "1" - - name: memory - quantity: 1Gi + cpuMillicores: 1000 + memoryBytes: 1073741824 # 1Gi snapshotConfig: storageLocation: gs://my-snapshots-bucket/stateful-agent volumes: - name: shared-storage external: template: - capacity: 5Gi + capacityBytes: 5368709120 # 5Gi storageClassName: csi-nfs-sc ``` diff --git a/docs/dev/release-notes.md b/docs/dev/release-notes.md index 626ebee27e..d9f6d153ad 100644 --- a/docs/dev/release-notes.md +++ b/docs/dev/release-notes.md @@ -37,7 +37,7 @@ The generated notes list each pull request as ` by @author in <link>`. Wr | `multi actor worker support` | `Run more than one actor on a worker` | | `Fix #1234` | `Fix the kind install on arm64 hosts` | -For a breaking change, describe the upgrade step in the pull request's "Breaking change" section. The release manager uses it to write the note. +For a breaking change, describe the upgrade step in the pull request description and add the `breaking-change` label. The release manager uses it to write the note. ## Cut a release diff --git a/docs/egress-credential-injection.md b/docs/egress-credential-injection.md index 5177ac18fa..d402b017a7 100644 --- a/docs/egress-credential-injection.md +++ b/docs/egress-credential-injection.md @@ -14,11 +14,17 @@ The actor never holds the secret. It sends the header with a placeholder value, which the gateway discards, so it cannot read, snapshot, or exfiltrate the secret, nor choose the value that leaves the cluster. +The gateway can also inject an **actor JWT** in place of a stored secret: a +token that ateapi mints for the actor making the request, which an upstream +that trusts Substrate's issuer can verify. See [Actor JWTs](#actor-jwts). + ## When you need this Actors call APIs that need bearer tokens or API keys, and those secrets must -stay out of the actor's filesystem, environment, and snapshots. +stay out of the actor's filesystem, environment, and snapshots. Other APIs +need to know which actor is calling and accept a signed token that says so; +for those, inject an actor JWT. Injection is supported on HTTP and HTTPS requests allowed by an `http` or `https` rule with credential replacement effects. For HTTPS, the actor must @@ -32,11 +38,13 @@ flowchart LR gateway["Egress gateway<br/>applies the EgressPolicy"] provider["Credential provider"] store[("Secret store")] + ateapi["ateapi"] upstream["Upstream API"] actor -->|"HTTPS request,<br/>placeholder header"| gateway - gateway -->|"fetch credential<br/>for this actor"| provider + gateway -->|"credentialUri:<br/>fetch credential<br/>for this actor"| provider provider -->|"authorize, then read"| store + gateway -->|"actorJwt:<br/>mint a JWT<br/>for this actor"| ateapi gateway -->|"request with<br/>the credential"| upstream ``` @@ -55,6 +63,10 @@ flowchart LR 4. The gateway replaces the placeholder with the credential and sends the request on to the origin over a new TLS connection. +For an actor JWT, steps 2 and 3 go to ateapi instead of a credential provider: +the gateway asks ateapi to mint a token for the actor, or reuses one it minted +earlier. + The reference k8s secret provider, `cmd/credential-provider/kubernetes-secrets`, resolves Kubernetes Secrets and is the one [For cluster admins](#for-cluster-admins) deploys. @@ -104,6 +116,44 @@ interpret. A URI the provider refuses denies the request with 403. `secretmanager.googleapis.com`) — see its [README](../internal/plugins/gcp-secret-manager/README.md#credential-uris). +### Actor JWTs + +An entry can set `actorJwt` in place of `credentialUri`. The gateway then puts +the prefix and a JWT that ateapi minted for the actor into the header. No +credential provider is involved, so this works on a gateway installed with +`--credential-provider='{"enabled":false}'`. + +```yaml +rules: +- https: + hostnames: ["api.example.com"] + effects: + replaceHeaders: + - header: Authorization + prefix: "Bearer " + actorJwt: + audiences: ["https://api.example.com"] + expirationSeconds: 900 +``` + +* `audiences` become the token's `aud` claim: 1 to 16 distinct values, each + up to 512 characters. Use the value the upstream checks the audience + against. +* `expirationSeconds` is the token's lifetime, from 300 to 3600 seconds, with + no default. +* An entry sets exactly one of `credentialUri` and `actorJwt`. + +The token's `sub` is `actor/<atespace>/<actor-name>`, and its `iss` is the +issuer the install gave ateapi (`ACTOR_JWT_ISSUER`). It also names the actor's +atespace, name, and UID. ateapi signs it with the install's actor JWT signing +key, which `kubectl ate admin` rotates (see the +[kubectl-ate README](../cmd/kubectl-ate/README.md)). + +The gateway reuses a token for the same actor, audiences, and lifetime until a +third of its lifetime is left, so the upstream always gets a token with at +least that much time remaining: 300 seconds for the 900-second lifetime above. +Concurrent requests that need the same token share one mint. + ## What the gateway does | Situation | Outcome | @@ -115,6 +165,10 @@ interpret. A URI the provider refuses denies the request with 403. | Provider unreachable or timed out | **503**, fail closed but retryable | | Provider returns an empty credential, or one containing control characters | **503**, fail closed | | URI names a provider class this gateway does not serve; unusable header name; unparseable URI | **500** (Envoy) or **403** (agentgateway), fail closed | +| Request decided by an `http` or `https` rule, carries the header of an `actorJwt` entry, mint succeeds | Header value replaced with the prefix and the actor's JWT, with or without a provider | +| `actorJwt` entry, the actor no longer exists | **403**, fail closed | +| `actorJwt` entry, ateapi unreachable or timed out | **503**, fail closed but retryable | +| `actorJwt` entry, any other mint failure | **500** (Envoy) or **403** (agentgateway), fail closed | A request that does not carry the header skips injection. On an HTTP or HTTPS request that carries it, any failure to produce the credential, including @@ -207,7 +261,8 @@ kubectl apply -f manifests/egress-credential-injection/sample-secret.yaml **3. The actors.** Actors can now add `replaceHeaders` effects to their `https` rules, as described under [The policy](#the-policy). Each such actor also needs the projected egress trust bundle to do TLS through the gateway at -all — see [egress-trust-bundle.md](egress-trust-bundle.md). +all — see [egress-trust-bundle.md](egress-trust-bundle.md). Actor JWT entries +need no provider; ateapi signs them with a key that every install creates. ### Verify @@ -236,6 +291,11 @@ and recreated, the channel can sit in connect backoff for up to a couple of minutes before re-resolving; injection fails closed with 503 (deliberately retryable) until it reconnects. +**Actor JWTs follow the actor's name.** The gateway caches tokens by actor +name. An actor that is deleted and recreated under the same name can get its +predecessor's token until that token's refresh point, up to two thirds of its +lifetime, and that token's UID claim names the old actor. + **Scope of the reference provider's RBAC.** The sample deployment grants read on Secrets cluster-wide so the policy may name any namespace; a production deployment should scope this to the namespaces the provider is allowed to diff --git a/docs/metrics/registry/metrics.yaml b/docs/metrics/registry/metrics.yaml index 0e97bc2412..0fe9265ae9 100644 --- a/docs/metrics/registry/metrics.yaml +++ b/docs/metrics/registry/metrics.yaml @@ -1061,15 +1061,18 @@ groups: atelet uploads deferred snapshots in the background once the upload delay or the detach delay elapses. A probe that finds the manifest committed already without uploading is not recorded. An upload that Terminate or an - atelet shutdown cancels records error.type Canceled. Synchronous uploads - from UPLOAD_MODE_IMMEDIATE checkpoints and PromoteSnapshot are recorded - on ate.actor.checkpoint.duration instead. + atelet shutdown cancels records error.type Canceled, and one that runs + past its timeout (--snapshot-upload-timeout plus the time the snapshot + takes at --snapshot-upload-est-throughput-mib, doubled for each + consecutive failure to at most 8 times that) records DeadlineExceeded. + Synchronous uploads from UPLOAD_MODE_IMMEDIATE checkpoints and + PromoteSnapshot are recorded on ate.actor.checkpoint.duration instead. annotations: substrate: emitted_by: [atelet] golden_signals: [latency, errors] code_anchor: cmd/atelet/internal/snapshottracker/metrics.go - buckets: [0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 20, 30, 60] + buckets: [0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 20, 30, 60, 120, 300, 600] cuj: Are background snapshot uploads slow or failing on a node? attributes: - ref: ate.snapshot.fidelity diff --git a/docs/operator-install.md b/docs/operator-install.md index 104a7d4e25..f4f8c2d45b 100644 --- a/docs/operator-install.md +++ b/docs/operator-install.md @@ -306,6 +306,7 @@ demos, the [benchmarks](../benchmarking/README.md#settings), and the | `gcp.projectID` | `PROJECT_ID` | `--gcp-project-id` | — | (global) | | `images.repo` | `ATE_IMAGE_REPO` | `--image-repo` | — | (global) | | `images.tag` | `ATE_IMAGE_TAG` | `--image-tag` | — | (global) | +| `ingressAllowedSPIFFEIDs` | `ATE_INGRESS_ALLOWED_SPIFFE_IDS` | `--ingress-allowed-spiffe-ids` | — | (global) | | `ingressAuthMode` | `ATE_INGRESS_AUTH_MODE` | `--ingress-auth-mode` | `deprecated-insecure` | (global) | | `kindCluster.enabled` | `ATE_INSTALL_KIND` | `--kind` | `false` | (global) | | `kindCluster.name` | `KIND_CLUSTER_NAME` | `--kind-cluster-name` | `kind` | (global) | diff --git a/hack/install-ate.sh b/hack/install-ate.sh index 54020f8b89..984942bfff 100755 --- a/hack/install-ate.sh +++ b/hack/install-ate.sh @@ -107,9 +107,11 @@ usage() { echo " --ingress-auth-mode=deprecated-insecure|static-mtls" echo " How the ingress router authenticates clients (default: deprecated-insecure)." echo " Both check a client certificate on the TLS ports against the podidentity CA and" - echo " the SPIFFE IDs listed in manifests/ate-install/atenet-router.yaml." + echo " --ingress-allowed-spiffe-ids." echo " deprecated-insecure also accepts TLS clients without one and serves plaintext" echo " ports; static-mtls requires the certificate and --atenet-dataplane=envoy." + echo " --ingress-allowed-spiffe-ids=IDS SPIFFE IDs (comma-separated) of the clients the ingress router lets reach actors" + echo " over its TLS ports. Required with --atenet-dataplane=envoy" echo " --rollout-timeout DURATION Per-workload readiness wait timeout, kubectl-style Go duration (default: 60s)" echo " --otlp-endpoint URL Send all control plane telemetry to URL, not to the cluster default (see benchmarking/telemetry/README.md)" echo " --ateapi-log-level LEVEL ate-api-server log level: debug, info, warn, or error" @@ -340,6 +342,14 @@ for ((i = 0; i < ${#prescan_args[@]}; i++)); do fi GLOBAL_FLAGS+=("--ingress-auth-mode=${prescan_args[$((i + 1))]}") ;; + --ingress-allowed-spiffe-ids=*) GLOBAL_FLAGS+=("${prescan_args[i]}") ;; + --ingress-allowed-spiffe-ids) + if (( i + 1 >= ${#prescan_args[@]} )); then + echo "Error: --ingress-allowed-spiffe-ids requires a comma-separated list of SPIFFE IDs" >&2 + exit 1 + fi + GLOBAL_FLAGS+=("--ingress-allowed-spiffe-ids=${prescan_args[$((i + 1))]}") + ;; --rollout-timeout=*) GLOBAL_FLAGS+=("${prescan_args[i]}") ;; --rollout-timeout) if (( i + 1 >= ${#prescan_args[@]} )); then @@ -404,7 +414,7 @@ while [[ "$#" -gt 0 ]]; do # Captured in the pre-scan above; matched here only so the `*)` branch does # not reject them, and so a separated value is consumed with its flag. --atenet-dataplane|--podcert-workers-per-signer|--rollout-timeout|--otlp-endpoint) shift ;; - --cluster-size|--ingress-auth-mode) shift ;; + --cluster-size|--ingress-auth-mode|--ingress-allowed-spiffe-ids) shift ;; --experimental-additional-egress-extproc-service) shift ;; --credential-provider) shift ;; --ateapi-log-level) shift ;; @@ -412,6 +422,7 @@ while [[ "$#" -gt 0 ]]; do --atenet-dataplane=*|--podcert-workers-per-signer=*|--rollout-timeout=*|--otlp-endpoint=*) ;; --fake-data-plane|--fake-data-plane=*) ;; --cluster-size=*|--cordon-control-plane|--cordon-control-plane=*|--ingress-auth-mode=*) ;; + --ingress-allowed-spiffe-ids=*) ;; --experimental-additional-egress-extproc-service=*) ;; --credential-provider=*) ;; --ateapi-log-level=*) ;; diff --git a/internal/ateworker/register/register.go b/internal/ateworker/register/register.go index 582a0bac57..c22fd918b9 100644 --- a/internal/ateworker/register/register.go +++ b/internal/ateworker/register/register.go @@ -35,13 +35,11 @@ import ( "go.ate.dev/substrate/internal/ateletdial" "go.ate.dev/substrate/internal/hardware" "go.ate.dev/substrate/internal/proto/ateletpb" - "go.ate.dev/substrate/internal/resources" "go.ate.dev/substrate/internal/sandboxruntimes" "go.ate.dev/substrate/internal/workercapacity" "google.golang.org/grpc/codes" "google.golang.org/grpc/status" "google.golang.org/protobuf/proto" - "k8s.io/apimachinery/pkg/api/resource" ) const ( @@ -67,11 +65,9 @@ func FromFiles(actors int) *ateletpb.WorkerResources { func fromDir(dir string, actors int) *ateletpb.WorkerResources { return &ateletpb.WorkerResources{ - Actors: int32(actors), - Resources: cpuMemory( - readLimit(filepath.Join(dir, workercapacity.CPULimitFile)), - readLimit(filepath.Join(dir, workercapacity.MemoryLimitFile)), - ), + CpuMillicores: readLimit(filepath.Join(dir, workercapacity.CPULimitFile)), + MemoryBytes: readLimit(filepath.Join(dir, workercapacity.MemoryLimitFile)), + Actors: int32(actors), } } @@ -107,22 +103,6 @@ func buildRequest(capacity *ateletpb.WorkerResources, sandboxClass string, f *sa return req } -// cpuMemory is the Resources for those two dimensions. A zero dimension is -// left out, which the control plane reads as none of it; nil when both are. -func cpuMemory(cpuMilli, memoryBytes int64) *ateletpb.Resources { - var limits []*ateletpb.Limits - if cpuMilli != 0 { - limits = append(limits, &ateletpb.Limits{Name: resources.ResourceCPU, Quantity: resource.NewMilliQuantity(cpuMilli, resource.DecimalSI).String()}) - } - if memoryBytes != 0 { - limits = append(limits, &ateletpb.Limits{Name: resources.ResourceMemory, Quantity: resource.NewQuantity(memoryBytes, resource.BinarySI).String()}) - } - if len(limits) == 0 { - return nil - } - return &ateletpb.Resources{Limits: limits} -} - func readLimit(path string) int64 { contents, err := os.ReadFile(path) if err != nil { diff --git a/internal/ateworker/register/register_test.go b/internal/ateworker/register/register_test.go index d04166998e..d6210d271f 100644 --- a/internal/ateworker/register/register_test.go +++ b/internal/ateworker/register/register_test.go @@ -47,9 +47,7 @@ func TestFromFiles(t *testing.T) { memory string want *ateletpb.WorkerResources }{ - {name: "limits set", cpu: "2000", memory: "4294967296", want: &ateletpb.WorkerResources{Actors: testActors, Resources: &ateletpb.Resources{ - Limits: []*ateletpb.Limits{{Name: "cpu", Quantity: "2"}, {Name: "memory", Quantity: "4Gi"}}, - }}}, + {name: "limits set", cpu: "2000", memory: "4294967296", want: &ateletpb.WorkerResources{Actors: testActors, CpuMillicores: 2000, MemoryBytes: 4294967296}}, {name: "unparseable is none", cpu: "2Gi", memory: "", want: &ateletpb.WorkerResources{Actors: testActors}}, {name: "negative is none", cpu: "-1", memory: "-1", want: &ateletpb.WorkerResources{Actors: testActors}}, } { diff --git a/internal/ateworkervalidation/validation_test.go b/internal/ateworkervalidation/validation_test.go index 4144941827..5eaed65bbb 100644 --- a/internal/ateworkervalidation/validation_test.go +++ b/internal/ateworkervalidation/validation_test.go @@ -67,8 +67,7 @@ func validRunWorkloadRequest(mutate ...func(*ateworkerpb.RunWorkloadRequest)) *a Spec: fullSpec(), RuntimeAssetPaths: map[string]string{"kata-kernel": "/opt/kernel", "virtiofsd": "/opt/virtiofsd"}, EgressGateway: &ateworkerpb.EgressGateway{Address: "gateway:443"}, - CpuMilli: 1500, - MemoryBytes: 1 << 30, + Resources: &ateworkerpb.ResourceRequirements{Limits: &ateworkerpb.Resources{CpuMillicores: 1500, MemoryBytes: 1 << 30}}, ActorDirs: fullActorDirs(), } for _, m := range mutate { @@ -92,7 +91,7 @@ func TestValidateRunWorkloadRequest(t *testing.T) { }, { name: "valid: gVisor shape, no assets, no egress, unset size", obj: valid(func(r *ateworkerpb.RunWorkloadRequest) { - r.RuntimeAssetPaths, r.EgressGateway, r.CpuMilli, r.MemoryBytes = nil, nil, 0, 0 + r.RuntimeAssetPaths, r.EgressGateway, r.Resources = nil, nil, nil }), }, { name: "missing atespace", @@ -148,17 +147,23 @@ func TestValidateRunWorkloadRequest(t *testing.T) { }), want: field.ErrorList{field.TooLong(assets.Key("kata-kernel"), nil, 4096).WithOrigin("maxLength")}, }, { - name: "negative cpu_milli", - obj: valid(func(r *ateworkerpb.RunWorkloadRequest) { r.CpuMilli = -1 }), - want: field.ErrorList{field.Invalid(field.NewPath("cpu_milli"), nil, "").WithOrigin("minimum")}, + name: "negative cpu_millicores", + obj: valid(func(r *ateworkerpb.RunWorkloadRequest) { + r.Resources = &ateworkerpb.ResourceRequirements{Limits: &ateworkerpb.Resources{CpuMillicores: -1}} + }), + want: field.ErrorList{field.Invalid(field.NewPath("resources", "limits", "cpu_millicores"), nil, "").WithOrigin("minimum")}, }, { - name: "cpu_milli past atelet's bound", - obj: valid(func(r *ateworkerpb.RunWorkloadRequest) { r.CpuMilli = 1000000 }), - want: field.ErrorList{field.Invalid(field.NewPath("cpu_milli"), nil, "").WithOrigin("maximum")}, + name: "cpu_millicores past atelet's bound", + obj: valid(func(r *ateworkerpb.RunWorkloadRequest) { + r.Resources = &ateworkerpb.ResourceRequirements{Limits: &ateworkerpb.Resources{CpuMillicores: 1000000}} + }), + want: field.ErrorList{field.Invalid(field.NewPath("resources", "limits", "cpu_millicores"), nil, "").WithOrigin("maximum")}, }, { name: "negative memory_bytes", - obj: valid(func(r *ateworkerpb.RunWorkloadRequest) { r.MemoryBytes = -1 }), - want: field.ErrorList{field.Invalid(field.NewPath("memory_bytes"), nil, "").WithOrigin("minimum")}, + obj: valid(func(r *ateworkerpb.RunWorkloadRequest) { + r.Resources = &ateworkerpb.ResourceRequirements{Limits: &ateworkerpb.Resources{MemoryBytes: -1}} + }), + want: field.ErrorList{field.Invalid(field.NewPath("resources", "limits", "memory_bytes"), nil, "").WithOrigin("minimum")}, }, { name: "empty egress_gateway", obj: valid(func(r *ateworkerpb.RunWorkloadRequest) { r.EgressGateway = &ateworkerpb.EgressGateway{} }), @@ -591,8 +596,7 @@ func validRestoreWorkloadRequest(mutate ...func(*ateworkerpb.RestoreWorkloadRequ RuntimeAssetPaths: map[string]string{"kata-kernel": "/opt/kernel"}, Fidelity: ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, EgressGateway: &ateworkerpb.EgressGateway{Address: "gateway:443"}, - CpuMilli: 1500, - MemoryBytes: 1 << 30, + Resources: &ateworkerpb.ResourceRequirements{Limits: &ateworkerpb.Resources{CpuMillicores: 1500, MemoryBytes: 1 << 30}}, ActorDirs: fullActorDirs(), } for _, m := range mutate { @@ -645,13 +649,17 @@ func TestValidateRestoreWorkloadRequest(t *testing.T) { obj: valid(func(r *ateworkerpb.RestoreWorkloadRequest) { r.ActorDirs = nil }), want: field.ErrorList{field.Required(field.NewPath("actor_dirs"), "")}, }, { - name: "negative cpu_milli", - obj: valid(func(r *ateworkerpb.RestoreWorkloadRequest) { r.CpuMilli = -1 }), - want: field.ErrorList{field.Invalid(field.NewPath("cpu_milli"), nil, "").WithOrigin("minimum")}, + name: "negative cpu_millicores", + obj: valid(func(r *ateworkerpb.RestoreWorkloadRequest) { + r.Resources = &ateworkerpb.ResourceRequirements{Limits: &ateworkerpb.Resources{CpuMillicores: -1}} + }), + want: field.ErrorList{field.Invalid(field.NewPath("resources", "limits", "cpu_millicores"), nil, "").WithOrigin("minimum")}, }, { - name: "cpu_milli past atelet's bound", - obj: valid(func(r *ateworkerpb.RestoreWorkloadRequest) { r.CpuMilli = 1000000 }), - want: field.ErrorList{field.Invalid(field.NewPath("cpu_milli"), nil, "").WithOrigin("maximum")}, + name: "cpu_millicores past atelet's bound", + obj: valid(func(r *ateworkerpb.RestoreWorkloadRequest) { + r.Resources = &ateworkerpb.ResourceRequirements{Limits: &ateworkerpb.Resources{CpuMillicores: 1000000}} + }), + want: field.ErrorList{field.Invalid(field.NewPath("resources", "limits", "cpu_millicores"), nil, "").WithOrigin("maximum")}, }, { name: "runtime asset path too long", obj: valid(func(r *ateworkerpb.RestoreWorkloadRequest) { diff --git a/internal/ateworkervalidation/zz_generated.validation.go b/internal/ateworkervalidation/zz_generated.validation.go index acf75a2af9..eb633428cf 100644 --- a/internal/ateworkervalidation/zz_generated.validation.go +++ b/internal/ateworkervalidation/zz_generated.validation.go @@ -882,6 +882,49 @@ func Validate_Container( return errs } +// Validate_ContainerSpec validates an instance of ContainerSpec according +// to declarative validation rules in the API schema. +func Validate_ContainerSpec( + ctx context.Context, op operation.Operation, fldPath *field.Path, + obj, oldObj *ateworkerpb.ContainerSpec) (errs field.ErrorList) { + + // field ateworkerpb.ContainerSpec.Args has no validation + // field ateworkerpb.ContainerSpec.Env has no validation + // field ateworkerpb.ContainerSpec.Capabilities has no validation + + { // field ateworkerpb.ContainerSpec.Resources + fn := func( + fldPath *field.Path, + obj, oldObj *ateworkerpb.ResourceRequirements, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.OptionalPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // call the type's validation function + errs = append(errs, Validate_ResourceRequirements(ctx, op, fldPath, obj, oldObj)...) + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateworkerpb.ContainerSpec) *ateworkerpb.ResourceRequirements { + return oldObj.Resources + }) + errs = append(errs, fn(fldPath.Child("resources"), obj.Resources, oldVal, oldObj != nil)...) + } + + return errs +} + // Validate_EgressGateway validates an instance of EgressGateway according // to declarative validation rules in the API schema. func Validate_EgressGateway( @@ -1048,6 +1091,119 @@ func Validate_HttpGetAction( return errs } +// Validate_ResourceRequirements validates an instance of ResourceRequirements according +// to declarative validation rules in the API schema. +func Validate_ResourceRequirements( + ctx context.Context, op operation.Operation, fldPath *field.Path, + obj, oldObj *ateworkerpb.ResourceRequirements) (errs field.ErrorList) { + + { // field ateworkerpb.ResourceRequirements.Limits + fn := func( + fldPath *field.Path, + obj, oldObj *ateworkerpb.Resources, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.OptionalPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // call the type's validation function + errs = append(errs, Validate_Resources(ctx, op, fldPath, obj, oldObj)...) + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateworkerpb.ResourceRequirements) *ateworkerpb.Resources { + return oldObj.Limits + }) + errs = append(errs, fn(fldPath.Child("limits"), obj.Limits, oldVal, oldObj != nil)...) + } + + return errs +} + +// Validate_Resources validates an instance of Resources according +// to declarative validation rules in the API schema. +func Validate_Resources( + ctx context.Context, op operation.Operation, fldPath *field.Path, + obj, oldObj *ateworkerpb.Resources) (errs field.ErrorList) { + + { // field ateworkerpb.Resources.CpuMillicores + fn := func( + fldPath *field.Path, + obj, oldObj *int64, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.Maximum(ctx, op, fldPath, obj, oldObj, 999999); len(e) != 0 { + errs = append(errs, e...) + } + if e := validate.Minimum(ctx, op, fldPath, obj, oldObj, 0); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateworkerpb.Resources) *int64 { + return &oldObj.CpuMillicores + }) + errs = append(errs, fn(fldPath.Child("cpu_millicores"), &obj.CpuMillicores, oldVal, oldObj != nil)...) + } + + { // field ateworkerpb.Resources.MemoryBytes + fn := func( + fldPath *field.Path, + obj, oldObj *int64, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.Minimum(ctx, op, fldPath, obj, oldObj, 0); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateworkerpb.Resources) *int64 { + return &oldObj.MemoryBytes + }) + errs = append(errs, fn(fldPath.Child("memory_bytes"), &obj.MemoryBytes, oldVal, oldObj != nil)...) + } + + return errs +} + // Validate_RestoreWorkloadRequest validates an instance of RestoreWorkloadRequest according // to declarative validation rules in the API schema. func Validate_RestoreWorkloadRequest( @@ -1405,69 +1561,34 @@ func Validate_RestoreWorkloadRequest( errs = append(errs, fn(fldPath.Child("egress_gateway"), obj.EgressGateway, oldVal, oldObj != nil)...) } - { // field ateworkerpb.RestoreWorkloadRequest.CpuMilli - fn := func( - fldPath *field.Path, - obj, oldObj *int64, - oldValueCorrelated bool) (errs field.ErrorList) { - // don't revalidate unchanged data - if oldValueCorrelated && op.Type == operation.Update { - if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { - return nil - } - } - // call field-attached validations - earlyReturn := false - if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { - earlyReturn = true - } - if earlyReturn { - return // do not proceed - } - if e := validate.Maximum(ctx, op, fldPath, obj, oldObj, 999999); len(e) != 0 { - errs = append(errs, e...) - } - if e := validate.Minimum(ctx, op, fldPath, obj, oldObj, 1); len(e) != 0 { - errs = append(errs, e...) - } - return - } - oldVal := safe.Field(oldObj, - func(oldObj *ateworkerpb.RestoreWorkloadRequest) *int64 { - return &oldObj.CpuMilli - }) - errs = append(errs, fn(fldPath.Child("cpu_milli"), &obj.CpuMilli, oldVal, oldObj != nil)...) - } - - { // field ateworkerpb.RestoreWorkloadRequest.MemoryBytes + { // field ateworkerpb.RestoreWorkloadRequest.Resources fn := func( fldPath *field.Path, - obj, oldObj *int64, + obj, oldObj *ateworkerpb.ResourceRequirements, oldValueCorrelated bool) (errs field.ErrorList) { // don't revalidate unchanged data if oldValueCorrelated && op.Type == operation.Update { - if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + if ateDeepEqual(obj, oldObj) { return nil } } // call field-attached validations earlyReturn := false - if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + if e := validate.OptionalPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { earlyReturn = true } if earlyReturn { return // do not proceed } - if e := validate.Minimum(ctx, op, fldPath, obj, oldObj, 1); len(e) != 0 { - errs = append(errs, e...) - } + // call the type's validation function + errs = append(errs, Validate_ResourceRequirements(ctx, op, fldPath, obj, oldObj)...) return } oldVal := safe.Field(oldObj, - func(oldObj *ateworkerpb.RestoreWorkloadRequest) *int64 { - return &oldObj.MemoryBytes + func(oldObj *ateworkerpb.RestoreWorkloadRequest) *ateworkerpb.ResourceRequirements { + return oldObj.Resources }) - errs = append(errs, fn(fldPath.Child("memory_bytes"), &obj.MemoryBytes, oldVal, oldObj != nil)...) + errs = append(errs, fn(fldPath.Child("resources"), obj.Resources, oldVal, oldObj != nil)...) } { // field ateworkerpb.RestoreWorkloadRequest.ActorDirs @@ -1854,69 +1975,34 @@ func Validate_RunWorkloadRequest( errs = append(errs, fn(fldPath.Child("egress_gateway"), obj.EgressGateway, oldVal, oldObj != nil)...) } - { // field ateworkerpb.RunWorkloadRequest.CpuMilli + { // field ateworkerpb.RunWorkloadRequest.Resources fn := func( fldPath *field.Path, - obj, oldObj *int64, + obj, oldObj *ateworkerpb.ResourceRequirements, oldValueCorrelated bool) (errs field.ErrorList) { // don't revalidate unchanged data if oldValueCorrelated && op.Type == operation.Update { - if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { - return nil - } - } - // call field-attached validations - earlyReturn := false - if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { - earlyReturn = true - } - if earlyReturn { - return // do not proceed - } - if e := validate.Maximum(ctx, op, fldPath, obj, oldObj, 999999); len(e) != 0 { - errs = append(errs, e...) - } - if e := validate.Minimum(ctx, op, fldPath, obj, oldObj, 1); len(e) != 0 { - errs = append(errs, e...) - } - return - } - oldVal := safe.Field(oldObj, - func(oldObj *ateworkerpb.RunWorkloadRequest) *int64 { - return &oldObj.CpuMilli - }) - errs = append(errs, fn(fldPath.Child("cpu_milli"), &obj.CpuMilli, oldVal, oldObj != nil)...) - } - - { // field ateworkerpb.RunWorkloadRequest.MemoryBytes - fn := func( - fldPath *field.Path, - obj, oldObj *int64, - oldValueCorrelated bool) (errs field.ErrorList) { - // don't revalidate unchanged data - if oldValueCorrelated && op.Type == operation.Update { - if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + if ateDeepEqual(obj, oldObj) { return nil } } // call field-attached validations earlyReturn := false - if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + if e := validate.OptionalPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { earlyReturn = true } if earlyReturn { return // do not proceed } - if e := validate.Minimum(ctx, op, fldPath, obj, oldObj, 1); len(e) != 0 { - errs = append(errs, e...) - } + // call the type's validation function + errs = append(errs, Validate_ResourceRequirements(ctx, op, fldPath, obj, oldObj)...) return } oldVal := safe.Field(oldObj, - func(oldObj *ateworkerpb.RunWorkloadRequest) *int64 { - return &oldObj.MemoryBytes + func(oldObj *ateworkerpb.RunWorkloadRequest) *ateworkerpb.ResourceRequirements { + return oldObj.Resources }) - errs = append(errs, fn(fldPath.Child("memory_bytes"), &obj.MemoryBytes, oldVal, oldObj != nil)...) + errs = append(errs, fn(fldPath.Child("resources"), obj.Resources, oldVal, oldObj != nil)...) } { // field ateworkerpb.RunWorkloadRequest.ActorDirs diff --git a/internal/atunnel/egress/credential.go b/internal/atunnel/egress/credential.go index 5cf3a7b5be..70fea58c92 100644 --- a/internal/atunnel/egress/credential.go +++ b/internal/atunnel/egress/credential.go @@ -28,6 +28,8 @@ import ( "go.ate.dev/substrate/internal/ateletdial" "go.ate.dev/substrate/internal/proto/ateletpb" + "google.golang.org/grpc/codes" + "google.golang.org/grpc/status" ) // BrokerCertificateSource owns atunnel's actor private key and obtains the @@ -101,14 +103,37 @@ func NewBrokerCertificateSource(cfg BrokerConfig) (*BrokerCertificateSource, err }, nil } +// mintAttempts bounds the retries of a mint whose broker connection came back +// Unavailable: a node under a restore wave can refuse or drop the fresh connection, and +// a failed mint fails the restore that needs the certificate. +const mintAttempts = 4 + // MintWorkerCertificate requests and installs a fresh certificate for the source's existing -// actor key. It returns the new expiry for renewal scheduling. +// actor key. It returns the new expiry for renewal scheduling. A mint that fails with +// Unavailable is retried a few times with a short back-off; every other error is final. func (s *BrokerCertificateSource) MintWorkerCertificate(ctx context.Context) (time.Time, error) { csr, err := x509.CreateCertificateRequest(rand.Reader, &x509.CertificateRequest{}, s.privateKey) if err != nil { return time.Time{}, fmt.Errorf("egress: create actor CSR: %w", err) } + for attempt := 1; ; attempt++ { + expiresAt, err := s.mintOnce(ctx, csr) + if err == nil { + return expiresAt, nil + } + if attempt >= mintAttempts || status.Code(err) != codes.Unavailable { + return time.Time{}, err + } + select { + case <-ctx.Done(): + return time.Time{}, err + case <-time.After(time.Duration(attempt) * 500 * time.Millisecond): + } + } +} +// mintOnce is one connection to the credential broker and one mint over it. +func (s *BrokerCertificateSource) mintOnce(ctx context.Context, csr []byte) (time.Time, error) { // TODO(identity): We should not be re-establishing a gRPC connection for // every mint request. Do it once at atunnel startup. diff --git a/internal/atunnel/egress/credential_test.go b/internal/atunnel/egress/credential_test.go index 22f96bc462..a48138a0aa 100644 --- a/internal/atunnel/egress/credential_test.go +++ b/internal/atunnel/egress/credential_test.go @@ -88,9 +88,18 @@ type workerSupportStub struct { ca *testCA lifetime time.Duration publicKeys chan []byte + // failFirst mints fail with failCode before the stub answers; calls counts every mint. + failFirst int + failCode codes.Code + calls int } func (s *workerSupportStub) MintActorCertificate(_ context.Context, req *ateletpb.MintActorCertificateRequest) (*ateletpb.MintActorCertificateResponse, error) { + s.calls++ + if s.failFirst > 0 { + s.failFirst-- + return nil, status.Error(s.failCode, "broker busy") + } if req.GetActorUid() != "actor-uid" { return nil, status.Error(codes.FailedPrecondition, "unexpected actor UID") } @@ -223,3 +232,33 @@ func issueTestPodCertificate(t *testing.T, ca *testCA, identity *substratex509.P } return cert } + +func TestBrokerCertificateSourceRetriesUnavailable(t *testing.T) { + source, broker := newTestBrokerCertificateSource(t, testAteletIdentity("node-a"), time.Hour) + broker.failFirst, broker.failCode = 2, codes.Unavailable + ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + + if _, err := source.MintWorkerCertificate(ctx); err != nil { + t.Fatalf("MintWorkerCertificate after two Unavailable answers: %v", err) + } + receiveWithin(t, broker.publicKeys, "certificate public key") + if broker.calls != 3 { + t.Fatalf("broker calls = %d, want 3 (two Unavailable, one mint)", broker.calls) + } +} + +func TestBrokerCertificateSourceDoesNotRetryOtherErrors(t *testing.T) { + source, broker := newTestBrokerCertificateSource(t, testAteletIdentity("node-a"), time.Hour) + broker.failFirst, broker.failCode = 1, codes.FailedPrecondition + ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + + _, err := source.MintWorkerCertificate(ctx) + if status.Code(err) != codes.FailedPrecondition { + t.Fatalf("MintWorkerCertificate error = %v, want FailedPrecondition", err) + } + if broker.calls != 1 { + t.Fatalf("broker calls = %d, want 1 (no retry)", broker.calls) + } +} diff --git a/internal/benchmarking/boomer/agentsession/agentsession.go b/internal/benchmarking/boomer/agentsession/agentsession.go index 76ee8b5fe4..c1bbbbd6c4 100644 --- a/internal/benchmarking/boomer/agentsession/agentsession.go +++ b/internal/benchmarking/boomer/agentsession/agentsession.go @@ -23,6 +23,11 @@ // Per-step timings land in locust stats as Step_<name>. The first request // after each suspend is additionally recorded as WakeFirstTouch — in // implicit resume mode that row IS the parking wake latency distribution. +// TimeToWake is the whole wake, from the first ResumeActor attempt to that +// first request, including the time a session spent retrying while the +// fleet had no room for its actor (--agentsession-resume-retry-budget). +// --agentsession-phase-in staggers session starts so a fleet spawned in one +// ramp does not reach every later step in a single wave. package agentsession import ( @@ -39,6 +44,7 @@ import ( "strconv" "strings" "sync" + "sync/atomic" "time" "github.com/google/uuid" @@ -59,7 +65,6 @@ import ( "google.golang.org/grpc/metadata" "google.golang.org/grpc/status" "google.golang.org/protobuf/proto" - "k8s.io/apimachinery/pkg/api/resource" ) const ( @@ -80,8 +85,25 @@ const ( // maxConsecutiveStepFailures is how many ReplaceIfPersistent failures in // a row a session tolerates before its actor is deleted and recreated. maxConsecutiveStepFailures = 3 + + // defaultResumeRetryBudget is how long a step keeps retrying a wake the + // fleet had no room for when --agentsession-resume-retry-budget is unset; + // a budget of 0 gives the wake one attempt. + defaultResumeRetryBudget = 60 * time.Second + // The backoff between wake retries, the same window as glutton's spawn + // path: short enough that a slot freed by another session's suspend is + // taken within a second. Each wait is lengthened by up to this fraction + // of itself, so sessions that hit a full fleet in the same instant + // drift apart over their attempts instead of retrying in step. + wakeRetryInitialBackoff = 50 * time.Millisecond + wakeRetryMaxBackoff = time.Second + wakeRetryJitterFraction = 0.1 ) +// errStopping is startUser's answer once shutdown has begun: the sweep has +// passed, so no session may start. +var errStopping = errors.New("agentsession: shutting down, no session started") + // burnRate is the per-goroutine sha256 iteration rate BurnCPU reports. // BurnCPU runs for a fixed wall-clock duration, so Step_* latency stays flat // when the sandbox is CPU-starved; this rate is what drops. @@ -116,9 +138,18 @@ type knobs struct { // SnapshotSteps is a comma-separated list of 1-based step numbers // after which the actor is hibernated; "" hibernates after every step. SnapshotSteps string `json:"agentsession_snapshot_steps"` + // PhaseIn spreads session starts: each VU waits a uniform random share + // of it before creating its actor, so a fleet does not move through the + // script in lockstep waves; 0 starts every session at once. + PhaseIn dynconfig.Seconds `json:"agentsession_phase_in"` + // ResumeRetryBudget bounds how long a step keeps retrying a wake the + // fleet had no room for before it gives up on the step; 0 gives the + // wake one attempt. Unset, it is defaultResumeRetryBudget. + ResumeRetryBudget dynconfig.Seconds `json:"agentsession_resume_retry_budget"` } var knobsCodec = dynconfig.Typed[knobs]{ + Defaults: knobs{ResumeRetryBudget: dynconfig.Seconds(defaultResumeRetryBudget)}, Validate: func(k knobs) error { if err := k.Lifecycle.Validate(); err != nil { return err @@ -126,6 +157,12 @@ var knobsCodec = dynconfig.Typed[knobs]{ if k.ThinkScale < 0 { return fmt.Errorf("agentsession_think_scale cannot be negative: %f", k.ThinkScale) } + if k.PhaseIn < 0 { + return fmt.Errorf("agentsession_phase_in cannot be negative: %v", k.PhaseIn) + } + if k.ResumeRetryBudget < 0 { + return fmt.Errorf("agentsession_resume_retry_budget cannot be negative: %v", k.ResumeRetryBudget) + } if steps, err := parseSnapshotSteps(k.SnapshotSteps); err != nil { return err } else if len(steps) > 0 && k.LifecycleMode == dynconfig.LifecycleModeNone { @@ -190,6 +227,19 @@ func makeIngestBuf(steps []Step) []byte { type runtime struct { cfg *userclass.Config users sync.Map // goroutineID -> *sessionUser + // phasedIn holds the goroutines that have served their phase-in wait, + // so a start that failed and is retried does not wait a second time. + phasedIn sync.Map // goroutineID -> struct{} + // stopping is set by shutdown. A VU that comes out of its phase-in + // wait after that starts no session, and one whose start finishes + // after that cleans up its own, since the sweep has passed. + stopping atomic.Bool + // starting counts the starts past their phase-in wait that iterate has + // not yet stored or cleaned up, so shutdown can wait for them: main + // exits when it returns. startMu orders setting stopping against + // joining the count, so no start slips in between the two. + startMu sync.Mutex + starting sync.WaitGroup // loaded is what the knobs currently select, resolved by loadScript when // a session starts and re-resolved when the knobs name something else or @@ -322,23 +372,10 @@ func (r *runtime) checkTemplateMemory(ctx context.Context, script *Script) error } // memoryLimit reads the template's memory limit in bytes. found is false -// when the template sets none; a limit that does not parse is an error. +// when the template sets none. func memoryLimit(tmpl *ateapipb.ActorTemplate) (limit int64, found bool, err error) { - for _, l := range tmpl.GetResources().GetLimits() { - if l.GetName() != "memory" { - continue - } - q, err := resource.ParseQuantity(l.GetQuantity()) - if err != nil { - return 0, true, fmt.Errorf("memory limit %q: %w", l.GetQuantity(), err) - } - n, ok := q.AsInt64() - if !ok { - return 0, true, fmt.Errorf("memory limit %q is not a whole byte count", l.GetQuantity()) - } - return n, true, nil - } - return 0, false, nil + limit = tmpl.GetResources().GetLimits().GetMemoryBytes() + return limit, limit != 0, nil } // think returns the suspended "LLM thinking" gap before step s, which is the @@ -353,6 +390,32 @@ func (r *runtime) think(s Step) time.Duration { return time.Duration(float64(s.Think) * scale * jitter) } +// phaseIn makes a VU that has no session yet wait its share of +// --agentsession-phase-in before it starts one. Locust spawns the fleet in +// one ramp and every session begins at step 1, so without it the sessions +// reach each later step together, in a wave the spawn ramp alone spreads +// over seconds: with a step that dwells for longer than that, the wave +// asks every worker for more slots at once than it has. The wait is drawn +// once per VU; a start that fails and is retried does not wait again. +func (r *runtime) phaseIn(gid int64) { + if _, done := r.phasedIn.LoadOrStore(gid, struct{}{}); done { + return + } + if d := r.phaseInDelay(); d > 0 { + time.Sleep(d) + } +} + +// phaseInDelay draws one VU's wait: uniform in [0, --agentsession-phase-in), +// or none when the window is unset. +func (r *runtime) phaseInDelay() time.Duration { + window := dynconfig.Get[knobs](r.cfg.Dyn).PhaseIn.Duration() + if window <= 0 { + return 0 + } + return time.Duration(mathrand.Int64N(int64(window))) +} + // iterate is the boomer task function: one scripted step per call. The VU's // session is created lazily on first use and wraps back to step 1 after the // script completes, so one actor keeps producing suspend/resume cycles for @@ -368,7 +431,10 @@ func (r *runtime) iterate() { time.Sleep(2 * time.Second) return } - u, err := r.startUser(context.Background(), loaded) + u, err := r.startUser(context.Background(), gid, loaded) + if errors.Is(err, errStopping) { + return + } if err != nil { slog.Warn("agentsession start failed; goroutine will retry next iter", slog.String("err", err.Error())) @@ -376,6 +442,21 @@ func (r *runtime) iterate() { return } val, _ = r.users.LoadOrStore(gid, u) + stopping := r.stopping.Load() + if stopping { + // shutdown ran while the session started, and its sweep may + // or may not have seen the entry. Deleting the entry claims + // the cleanup, on either side, so the actor is suspended and + // deleted exactly once and does not outlive the run. + if _, ok := r.users.LoadAndDelete(gid); ok { + u.suspendAndDelete(context.Background()) + } + } + // The start is complete, cleanup included; shutdown may return. + r.starting.Done() + if stopping { + return + } } u := val.(*sessionUser) @@ -403,8 +484,14 @@ func (r *runtime) iterate() { // startUser creates the session's actor, waits for its sandbox to serve, // then suspends it so the first scripted step begins — like every later -// step — with a wake from suspension. -func (r *runtime) startUser(ctx context.Context, loaded *loadedScript) (*sessionUser, error) { +// step — with a wake from suspension. The VU's phase-in wait comes after +// the preflight checks, so a refusal that a redeploy fixes a minute into +// the run does not leave every VU whose wait elapsed meanwhile starting +// together once the check clears. It returns errStopping when shutdown +// began during the wait. A start that gets past the wait is counted in +// starting until iterate has stored or cleaned up the session; a start +// that fails leaves the count itself. +func (r *runtime) startUser(ctx context.Context, gid int64, loaded *loadedScript) (*sessionUser, error) { if err := r.checkTemplateMemory(ctx, loaded.Script); err != nil { return nil, err } @@ -412,6 +499,26 @@ func (r *runtime) startUser(ctx context.Context, loaded *loadedScript) (*session if err := checkSnapshotSteps(dyn, loaded.Script); err != nil { return nil, err } + r.phaseIn(gid) + r.startMu.Lock() + if r.stopping.Load() { + r.startMu.Unlock() + return nil, errStopping + } + r.starting.Add(1) + r.startMu.Unlock() + u, err := r.createSession(ctx, dyn, loaded) + if err != nil { + r.starting.Done() + return nil, err + } + return u, nil +} + +// createSession is the part of a start that touches the cluster: it creates +// the actor, waits for it to serve, and parks it unless the session stays +// resident. +func (r *runtime) createSession(ctx context.Context, dyn knobs, loaded *loadedScript) (*sessionUser, error) { u := &sessionUser{ cfg: r.cfg, actorName: "agent-" + uuid.NewString(), @@ -468,16 +575,30 @@ func checkSnapshotSteps(dyn knobs, script *Script) error { const shutdownConcurrency = 64 // shutdown suspends and deletes every session's actor, shutdownConcurrency -// at a time, until done or ctx expires. +// at a time, until done or ctx expires. It first marks the runtime as +// stopping, so a VU that is still waiting out its phase-in starts no +// session once it wakes, and one whose start finishes during the sweep +// cleans up its own session (see iterate). After the sweep it waits for +// those in-flight starts, within the same ctx, since main exits once it +// returns. func (r *runtime) shutdown(ctx context.Context) { + r.startMu.Lock() + r.stopping.Store(true) + r.startMu.Unlock() var wg sync.WaitGroup sem := make(chan struct{}, shutdownConcurrency) - r.users.Range(func(_, val any) bool { + r.users.Range(func(key, val any) bool { select { case sem <- struct{}{}: case <-ctx.Done(): return false } + // Deleting the entry claims its cleanup; a VU that stored its + // session during the sweep claims its own the same way. + if _, ok := r.users.LoadAndDelete(key); !ok { + <-sem + return true + } wg.Add(1) go func(u *sessionUser) { defer wg.Done() @@ -487,6 +608,16 @@ func (r *runtime) shutdown(ctx context.Context) { return true }) wg.Wait() + + started := make(chan struct{}) + go func() { + r.starting.Wait() + close(started) + }() + select { + case <-started: + case <-ctx.Done(): + } } // sessionUser is one coding-agent session: a single actor plus its progress @@ -624,22 +755,42 @@ func (u *sessionUser) runStep(ctx context.Context, step Step) bool { dyn := dynconfig.Get[knobs](u.cfg.Dyn) wasParked := !u.awake && dyn.LifecycleMode != dynconfig.LifecycleModeNone + // A parked actor's wake, from the first attempt to the ping that finds it + // serving, is one TimeToWake sample. A fleet with no room for the actor + // is retried within the budget, so the wait for a slot lands in that row + // as latency instead of vanishing into failed rows and a skipped step. + wakeStart := time.Now() + deadline := wakeStart.Add(dyn.ResumeRetryBudget.Duration()) if wasParked && dyn.ResumeMode == dynconfig.ResumeModeExplicit { - if err := u.resume(ctx); err != nil { + if err := retryWhileFull(ctx, deadline, func() error { return u.resume(ctx) }); err != nil { u.noteFailure(err) + u.recordTimeToWake(wakeStart, err) + if isCapacityError(err) { + slog.Warn("agent session found no room within the retry budget", + slog.String("actor", u.actorName), + slog.String("step", step.Name), + slog.Duration("waited", time.Since(wakeStart)), + slog.String("err", err.Error())) + } return false } } - wakeStart := time.Now() - if err := u.execOp(ctx, ping()); err != nil { + var err error + if wasParked { + err = retryWhileFull(ctx, deadline, func() error { return u.wakePing(ctx) }) + } else { + err = u.execOp(ctx, ping()) + } + if err != nil { u.noteFailure(err) if wasParked { - bmetrics.RecordFailure("http", "WakeFirstTouch", agentSessionUserClass, time.Since(wakeStart), err.Error()) + u.recordTimeToWake(wakeStart, err) } slog.Warn("agent session wake failed", slog.String("actor", u.actorName), slog.String("step", step.Name), + slog.Duration("waited", time.Since(wakeStart)), slog.String("err", err.Error())) if wasParked || u.shouldHibernateAfterStep(dyn) { u.hibernate(ctx) @@ -648,7 +799,7 @@ func (u *sessionUser) runStep(ctx context.Context, step Step) bool { } u.awake = true if wasParked { - bmetrics.RecordSuccess("http", "WakeFirstTouch", agentSessionUserClass, time.Since(wakeStart), 0) + u.recordTimeToWake(wakeStart, nil) } metricName := "Step_" + step.Name @@ -677,6 +828,71 @@ func (u *sessionUser) runStep(ctx context.Context, step Step) bool { return true } +// wakePing sends the ping that measures a parked actor's wake, recorded as +// a WakeFirstTouch sample of its own: in explicit resume mode the round trip +// after ResumeActor returned, in implicit mode the parked request itself. +func (u *sessionUser) wakePing(ctx context.Context) error { + start := time.Now() + if err := u.execOp(ctx, ping()); err != nil { + bmetrics.RecordFailure("http", "WakeFirstTouch", agentSessionUserClass, time.Since(start), err.Error()) + return err + } + bmetrics.RecordSuccess("http", "WakeFirstTouch", agentSessionUserClass, time.Since(start), 0) + return nil +} + +// recordTimeToWake books one step's wake as a TimeToWake sample: the time +// from the first ResumeActor attempt (in implicit mode, the first wake ping) +// to the ping that found the actor serving, retries included. +func (u *sessionUser) recordTimeToWake(start time.Time, err error) { + if err != nil { + bmetrics.RecordFailure("summary", "TimeToWake", agentSessionUserClass, time.Since(start), err.Error()) + return + } + bmetrics.RecordSuccess("summary", "TimeToWake", agentSessionUserClass, time.Since(start), 0) +} + +// retryWhileFull runs call and, while it fails with a capacity error and +// the next attempt would still land before deadline, waits with exponential +// backoff and runs it again. Each attempt records its own sample inside +// call, as glutton's spawn path does. Any other error returns at once. +func retryWhileFull(ctx context.Context, deadline time.Time, call func() error) error { + delay := wakeRetryInitialBackoff + for { + err := call() + if err == nil || !isCapacityError(err) { + return err + } + wait := delay + time.Duration(mathrand.Float64()*wakeRetryJitterFraction*float64(delay)) + if time.Now().Add(wait).After(deadline) { + return err + } + select { + case <-time.After(wait): + case <-ctx.Done(): + return err + } + delay = min(delay*2, wakeRetryMaxBackoff) + } +} + +// isCapacityError reports whether err says the fleet or the control plane +// had no room for the actor just now: ateapi's ResourceExhausted ("no +// worker has room for the actor") or Unavailable, or the router's 503, 504 +// and 429 faces of them. The actor itself is fine, and the next try may find +// a slot another session's suspend has freed. +func isCapacityError(err error) bool { + var he *httpError + if errors.As(err, &he) { + return classifyHTTP(he.status) == boomerutil.RetryLater + } + switch status.Code(err) { + case codes.ResourceExhausted, codes.Unavailable: + return true + } + return false +} + // execOp performs one scripted op as a glutton RPC through the router. func (u *sessionUser) execOp(ctx context.Context, o op) error { switch o.kind { @@ -860,7 +1076,11 @@ func (u *sessionUser) resume(ctx context.Context) error { if boomerutil.IsCrashed(err) { bmetrics.RecordFailure("actor", "CrashCount", agentSessionUserClass, 0, "actor entered ACTOR_STATE_CRASHED") } - slog.Error("ResumeActor failed", slog.String("actor", u.actorName), slog.String("err", err.Error())) + // A full fleet is retried by the caller and reported once when the + // budget runs out; a line per attempt would flood the log. + if !isCapacityError(err) { + slog.Error("ResumeActor failed", slog.String("actor", u.actorName), slog.String("err", err.Error())) + } } return err } diff --git a/internal/benchmarking/boomer/agentsession/agentsession_test.go b/internal/benchmarking/boomer/agentsession/agentsession_test.go index 4f6c4c80b5..79c8aff2f0 100644 --- a/internal/benchmarking/boomer/agentsession/agentsession_test.go +++ b/internal/benchmarking/boomer/agentsession/agentsession_test.go @@ -17,6 +17,7 @@ package agentsession import ( "context" "errors" + "io" "net/http" "os" "path/filepath" @@ -30,6 +31,7 @@ import ( "time" "github.com/prometheus/client_golang/prometheus" + "go.ate.dev/substrate/internal/benchmarking/boomer/boomerutil" "go.ate.dev/substrate/internal/benchmarking/boomer/dynconfig" "go.ate.dev/substrate/internal/benchmarking/boomer/userclass" "go.ate.dev/substrate/internal/benchmarking/glutton/fake" @@ -339,6 +341,91 @@ func TestLoadScriptFollowsTheKnob(t *testing.T) { } } +// A VU still in its phase-in wait when the run ends wakes after shutdown +// has swept the sessions. It must not start one then, or the actor would +// outlive the run with nothing left to delete it. +func TestIterateAfterShutdownStartsNoSession(t *testing.T) { + ctl := &fakeControlClient{} + u := newTestUser(t, &fake.Server{}, ctl, knobs{}) + rt := &runtime{cfg: u.cfg} + rt.shutdown(context.Background()) + rt.iterate() + if got := countCalls(ctl.recordedCalls(), "CreateActor"); got != 0 { + t.Fatalf("CreateActor calls after shutdown = %d, want 0", got) + } + if _, ok := rt.users.Load(boomerutil.GoroutineID()); ok { + t.Error("a session was stored after shutdown") + } +} + +// A VU that is inside startUser when shutdown runs is not in users yet, so +// the sweep cannot see its session. shutdown waits for the start, and the +// VU finds stopping set once the session is stored and cleans it up itself, +// exactly once, before shutdown returns and the process exits. +func TestShutdownWaitsForSessionStartedDuringIt(t *testing.T) { + ctl := &fakeControlClient{createStarted: make(chan struct{}, 1), createGate: make(chan struct{})} + u := newTestUser(t, &fake.Server{}, ctl, knobs{}) + rt := &runtime{cfg: u.cfg} + + iterated := make(chan struct{}) + go func() { + defer close(iterated) + rt.iterate() + }() + <-ctl.createStarted + shutdownDone := make(chan struct{}) + go func() { + defer close(shutdownDone) + rt.shutdown(context.Background()) // the sweep sees no session yet + }() + select { + case <-shutdownDone: + t.Fatal("shutdown returned with a session start in flight") + case <-time.After(100 * time.Millisecond): + } + close(ctl.createGate) + <-shutdownDone + + // Checked as soon as shutdown returns: that is when main exits. + if got := countCalls(ctl.recordedCalls(), "DeleteActor"); got != 1 { + t.Errorf("DeleteActor calls when shutdown returned = %d, want the session started during it cleaned up once", got) + } + left := 0 + rt.users.Range(func(any, any) bool { left++; return true }) + if left != 0 { + t.Errorf("%d sessions left in users after shutdown, want 0", left) + } + <-iterated +} + +// The wait for in-flight starts is bounded by shutdown's context, as the +// sweep is, so a start that never finishes cannot hold the process past +// boomer's shutdown budget. +func TestShutdownStopsWaitingForStartsWhenCtxExpires(t *testing.T) { + ctl := &fakeControlClient{createStarted: make(chan struct{}, 1), createGate: make(chan struct{})} + u := newTestUser(t, &fake.Server{}, ctl, knobs{}) + rt := &runtime{cfg: u.cfg} + + iterated := make(chan struct{}) + go func() { + defer close(iterated) + rt.iterate() + }() + <-ctl.createStarted + ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond) + defer cancel() + start := time.Now() + rt.shutdown(ctx) + if waited := time.Since(start); waited > 2*time.Second { + t.Errorf("shutdown waited %v on a start that never finished, want to stop at the context", waited) + } + if got := countCalls(ctl.recordedCalls(), "DeleteActor"); got != 0 { + t.Errorf("DeleteActor calls = %d while the start was still gated, want 0", got) + } + close(ctl.createGate) + <-iterated +} + // TestShutdownFansOut: at thousands of sessions a serial suspend+delete // sweep leaks most actors before boomer's shutdown budget runs out, so the // hook must run sessions concurrently and still clean up every one. @@ -419,29 +506,29 @@ func TestLoadScriptPrefersFile(t *testing.T) { } // TestStartUserChecksTemplateMemory: a template smaller than the script's -// floor, or with a limit that does not parse, is refused before any actor -// is created; a big enough one, or one with no limit, proceeds. +// floor is refused before any actor is created; a big enough one, or one with +// no limit, proceeds. func TestStartUserChecksTemplateMemory(t *testing.T) { script, err := Load(DefaultScript) if err != nil { t.Fatal(err) } for _, tc := range []struct { - name, memory string - wantStart bool - wantErr string + name string + memory int64 + wantStart bool + wantErr string }{ - {"too small", "512Mi", false, "min_actor_memory"}, - {"unparseable", "1Gi-ish", false, "memory limit"}, - {"exact", "1Gi", true, ""}, - {"no limit", "", true, ""}, + {"too small", 512 << 20, false, "min_actor_memory"}, + {"exact", 1 << 30, true, ""}, + {"no limit", 0, true, ""}, } { t.Run(tc.name, func(t *testing.T) { ctl := &fakeControlClient{templateMemory: tc.memory} u := newTestUser(t, &fake.Server{}, ctl, knobs{}) rt := &runtime{cfg: u.cfg} - started, err := rt.startUser(context.Background(), &loadedScript{Script: script, ingestBuf: makeIngestBuf(script.Steps)}) + started, err := rt.startUser(context.Background(), 0, &loadedScript{Script: script, ingestBuf: makeIngestBuf(script.Steps)}) created := countCalls(ctl.recordedCalls(), "CreateActor") > 0 if tc.wantStart { if err != nil || !created { @@ -470,14 +557,14 @@ func TestTemplateMemoryRefusalExpires(t *testing.T) { if err != nil { t.Fatal(err) } - ctl := &fakeControlClient{templateMemory: "512Mi"} + ctl := &fakeControlClient{templateMemory: 512 << 20} u := newTestUser(t, &fake.Server{}, ctl, knobs{}) rt := &runtime{cfg: u.cfg} if err := rt.checkTemplateMemory(context.Background(), script); err == nil { t.Fatal("512Mi template passed the check") } - ctl.templateMemory = "1Gi" // the operator redeployed + ctl.templateMemory = 1 << 30 // the operator redeployed if err := rt.checkTemplateMemory(context.Background(), script); err == nil { t.Fatal("refusal must stand for templateRecheckInterval") } @@ -496,14 +583,14 @@ func TestTemplateMemoryRefusalExpires(t *testing.T) { // TestRunningSessionKeepsItsScript: changing the knob affects sessions that // start afterwards, not one already walking its steps. func TestRunningSessionKeepsItsScript(t *testing.T) { - ctl := &fakeControlClient{templateMemory: "1Gi"} + ctl := &fakeControlClient{templateMemory: 1 << 30} u := newTestUser(t, &fake.Server{}, ctl, knobs{}) rt := &runtime{cfg: u.cfg} first, err := rt.loadScript() if err != nil { t.Fatal(err) } - session, err := rt.startUser(context.Background(), first) + session, err := rt.startUser(context.Background(), 0, first) if err != nil { t.Fatal(err) } @@ -559,14 +646,19 @@ type fakeControlClient struct { suspendReqs []*ateapipb.SuspendActorRequest // sawDeadline is set when a call arrived with a context deadline. sawDeadline bool - // templateMemory is the memory limit GetActorTemplate reports; "" means + // templateMemory is the memory limit GetActorTemplate reports in bytes; 0 means // the template sets none. - templateMemory string + templateMemory int64 // deleteDelay stalls each DeleteActor; inFlight and maxInFlight count // concurrent DeleteActor calls, to prove shutdown fans out. deleteDelay time.Duration inFlight atomic.Int32 maxInFlight atomic.Int32 + // createStarted, when set, receives once a CreateActor has begun, and + // createGate, when set, holds every CreateActor until it is closed, so + // a test can run shutdown while a session start is in flight. + createStarted chan struct{} + createGate chan struct{} } func nextErr(errs *[]error) error { @@ -590,8 +682,8 @@ func (f *fakeControlClient) record(ctx context.Context, name string) { func (f *fakeControlClient) GetActorTemplate(ctx context.Context, in *ateapipb.GetActorTemplateRequest, opts ...grpc.CallOption) (*ateapipb.ActorTemplate, error) { f.record(ctx, "GetActorTemplate") tmpl := &ateapipb.ActorTemplate{} - if f.templateMemory != "" { - tmpl.Resources = &ateapipb.Resources{Limits: []*ateapipb.Limits{{Name: "memory", Quantity: f.templateMemory}}} + if f.templateMemory != 0 { + tmpl.Resources = &ateapipb.ResourceRequirements{Limits: &ateapipb.Resources{MemoryBytes: f.templateMemory}} } return tmpl, nil } @@ -603,6 +695,15 @@ func (f *fakeControlClient) CreateAtespace(ctx context.Context, in *ateapipb.Cre func (f *fakeControlClient) CreateActor(ctx context.Context, in *ateapipb.CreateActorRequest, opts ...grpc.CallOption) (*ateapipb.Actor, error) { f.record(ctx, "CreateActor") + if f.createStarted != nil { + select { + case f.createStarted <- struct{}{}: + default: + } + } + if f.createGate != nil { + <-f.createGate + } return &ateapipb.Actor{}, nil } @@ -779,7 +880,8 @@ func TestRunStep_ReplacesCrashedActorImmediately(t *testing.T) { // a state nothing the driver can call moves the actor out of. func TestRunStep_ReplacesStuckActorAfterFailedWake(t *testing.T) { ctl := &fakeControlClient{suspendErrs: []error{status.Error(codes.FailedPrecondition, "MarkSuspending prerequisite not met (got: ACTOR_STATE_CRASHED)")}} - u := newTestUser(t, &fake.Server{Status: 503}, ctl, knobs{}) + // A zero budget gives the wake one attempt: the router never recovers here. + u := newTestUser(t, &fake.Server{Status: 503}, ctl, knobs{ResumeRetryBudget: 0}) if u.runStep(context.Background(), pingStep) { t.Fatal("runStep = true with a failing router") @@ -798,7 +900,9 @@ func TestRunStep_KeepsActorThroughCapacityShortage(t *testing.T) { errs[i] = status.Error(codes.ResourceExhausted, "no worker has room for the actor") } ctl := &fakeControlClient{resumeErrs: errs} - u := newTestUser(t, &fake.Server{}, ctl, knobs{Lifecycle: dynconfig.Lifecycle{ResumeMode: dynconfig.ResumeModeExplicit}}) + // A budget too short for a second attempt: this test is about the actor + // being kept, not about the retry. + u := newTestUser(t, &fake.Server{}, ctl, knobs{Lifecycle: dynconfig.Lifecycle{ResumeMode: dynconfig.ResumeModeExplicit}, ResumeRetryBudget: 0}) for range rounds { if u.runStep(context.Background(), pingStep) { @@ -808,6 +912,212 @@ func TestRunStep_KeepsActorThroughCapacityShortage(t *testing.T) { if u.broken || u.consecutiveFailures != 0 { t.Errorf("broken=%v consecutiveFailures=%d after capacity errors, want false/0", u.broken, u.consecutiveFailures) } + if got := countCalls(ctl.recordedCalls(), "ResumeActor"); got != rounds { + t.Errorf("ResumeActor calls = %d with a budget that fits one attempt, want %d", got, rounds) + } +} + +// A fleet with no room is retried within the budget, as glutton's spawn path +// does: the step goes through once a slot opens, with one ResumeActor sample +// per attempt. +func TestRunStep_RetriesWakeWhileFleetIsFull(t *testing.T) { + const full = 3 + errs := make([]error, full) + for i := range errs { + errs[i] = status.Error(codes.ResourceExhausted, "no worker has room for the actor") + } + ctl := &fakeControlClient{resumeErrs: errs} + srv := &fake.Server{} + u := newTestUser(t, srv, ctl, knobs{Lifecycle: dynconfig.Lifecycle{ResumeMode: dynconfig.ResumeModeExplicit}, ResumeRetryBudget: dynconfig.Seconds(10 * time.Second)}) + + // One TimeToWake row for the whole wake, not one per attempt. + expectRows(t, "TimeToWake", 1, 0, func() { + if !u.runStep(context.Background(), pingStep) { + t.Fatal("runStep = false; the resume succeeded within the budget and the step must count") + } + }) + if got := countCalls(ctl.recordedCalls(), "ResumeActor"); got != full+1 { + t.Errorf("ResumeActor calls = %d, want %d failed attempts and one success", got, full+1) + } + if u.broken || u.consecutiveFailures != 0 { + t.Errorf("broken=%v consecutiveFailures=%d, want false/0", u.broken, u.consecutiveFailures) + } + // The wake ping and the step's one op: nothing reached the actor while + // the resume was being retried. + if got := len(srv.RecordedPaths()); got != 2 { + t.Errorf("actor requests = %d, want 2", got) + } +} + +// When the budget runs out the step is given up as before: the actor is +// kept, nothing counts toward replacement, and the session retries the step +// after its next think gap. +func TestRunStep_GivesUpWakeAfterRetryBudget(t *testing.T) { + errs := make([]error, 100) + for i := range errs { + errs[i] = status.Error(codes.ResourceExhausted, "no worker has room for the actor") + } + ctl := &fakeControlClient{resumeErrs: errs} + srv := &fake.Server{} + const budget = 200 * time.Millisecond + u := newTestUser(t, srv, ctl, knobs{Lifecycle: dynconfig.Lifecycle{ResumeMode: dynconfig.ResumeModeExplicit}, ResumeRetryBudget: dynconfig.Seconds(budget)}) + + start := time.Now() + // The given-up wake is one failed TimeToWake row. + expectRows(t, "TimeToWake", 1, 1, func() { + if u.runStep(context.Background(), pingStep) { + t.Fatal("runStep = true while every resume fails") + } + }) + if waited := time.Since(start); waited > budget+wakeRetryMaxBackoff { + t.Errorf("runStep took %v, want at most the budget plus one backoff", waited) + } + attempts := countCalls(ctl.recordedCalls(), "ResumeActor") + // Backoff 50ms, 100ms, ... fits two or three waits in 200ms. + if attempts < 2 || attempts > 5 { + t.Errorf("ResumeActor attempts = %d, want a few within a %v budget", attempts, budget) + } + if u.broken || u.consecutiveFailures != 0 { + t.Errorf("broken=%v consecutiveFailures=%d, want false/0", u.broken, u.consecutiveFailures) + } + if got := len(srv.RecordedPaths()); got != 0 { + t.Errorf("actor requests = %d, want none while the resume never succeeded", got) + } + if calls := ctl.recordedCalls(); calls[len(calls)-1] == "SuspendActor" { + t.Error("SuspendActor issued after a resume that never succeeded") + } +} + +// capacityThenServe fails the first n requests with a router capacity status +// and passes the rest to the actor, the implicit-mode shape of a full fleet. +type capacityThenServe struct { + next http.RoundTripper + status int + n int + mu sync.Mutex + failed int +} + +func (c *capacityThenServe) RoundTrip(req *http.Request) (*http.Response, error) { + c.mu.Lock() + fail := c.failed < c.n + if fail { + c.failed++ + } + c.mu.Unlock() + if !fail { + return c.next.RoundTrip(req) + } + return &http.Response{ + StatusCode: c.status, + Status: http.StatusText(c.status), + Body: io.NopCloser(strings.NewReader("no worker has room for the actor")), + Header: http.Header{}, + Request: req, + }, nil +} + +// In implicit mode the capacity error is the router's 503 on the wake ping; +// it is retried the same way, and the step counts once the ping lands. +func TestRunStep_RetriesWakePingOnRouterCapacityError(t *testing.T) { + srv := &fake.Server{} + ctl := &fakeControlClient{} + u := newTestUser(t, srv, ctl, knobs{ResumeRetryBudget: dynconfig.Seconds(10 * time.Second)}) + rt := &capacityThenServe{next: u.cfg.HTTPClient.Transport, status: http.StatusServiceUnavailable, n: 2} + u.cfg.HTTPClient = &http.Client{Transport: rt} + + expectRows(t, "TimeToWake", 1, 0, func() { + if !u.runStep(context.Background(), pingStep) { + t.Fatal("runStep = false; the wake ping succeeded within the budget and the step must count") + } + }) + if rt.failed != 2 { + t.Errorf("router failures served = %d, want 2", rt.failed) + } + // The wake ping that got through and the step's one op. + if got := len(srv.RecordedPaths()); got != 2 { + t.Errorf("actor requests = %d, want 2", got) + } + if u.broken || u.consecutiveFailures != 0 { + t.Errorf("broken=%v consecutiveFailures=%d, want false/0", u.broken, u.consecutiveFailures) + } +} + +func TestIsCapacityError(t *testing.T) { + cases := []struct { + name string + err error + want bool + }{ + {"no room", status.Error(codes.ResourceExhausted, "no worker has room for the actor"), true}, + {"api server restarting", status.Error(codes.Unavailable, "connection refused"), true}, + {"router 503", &httpError{route: "/ping", status: http.StatusServiceUnavailable}, true}, + {"router 504", &httpError{route: "/ping", status: http.StatusGatewayTimeout}, true}, + {"router 429", &httpError{route: "/ping", status: http.StatusTooManyRequests}, true}, + {"router 404", &httpError{route: "/ping", status: http.StatusNotFound}, false}, + {"router 500", &httpError{route: "/ping", status: http.StatusInternalServerError}, false}, + {"crashed", status.Error(codes.Aborted, "actor benchmark/agent-test crashed"), false}, + {"deadline", status.Error(codes.DeadlineExceeded, "context deadline exceeded"), false}, + {"plain error", errors.New("dial tcp: connection refused"), false}, + {"nil", nil, false}, + } + for _, tc := range cases { + if got := isCapacityError(tc.err); got != tc.want { + t.Errorf("%s: isCapacityError = %v, want %v", tc.name, got, tc.want) + } + } +} + +// A capacity error that stops being one is returned at once. +func TestRetryWhileFull_StopsOnOtherErrors(t *testing.T) { + calls := 0 + boom := errors.New("sandbox exploded") + err := retryWhileFull(context.Background(), time.Now().Add(time.Minute), func() error { + calls++ + if calls == 1 { + return status.Error(codes.ResourceExhausted, "no worker has room for the actor") + } + return boom + }) + if !errors.Is(err, boom) || calls != 2 { + t.Errorf("err=%v calls=%d, want the second error after two calls", err, calls) + } +} + +func TestPhaseInDelay(t *testing.T) { + off := &runtime{cfg: &userclass.Config{Dyn: dynconfig.Static(knobs{})}} + if got := off.phaseInDelay(); got != 0 { + t.Fatalf("phaseInDelay = %v with no window, want 0", got) + } + + const window = 10 * time.Second + r := &runtime{cfg: &userclass.Config{Dyn: dynconfig.Static(knobs{PhaseIn: dynconfig.Seconds(window)})}} + lo, hi := window, time.Duration(0) + for range 200 { + got := r.phaseInDelay() + if got < 0 || got >= window { + t.Fatalf("phaseInDelay = %v, want within [0, %v)", got, window) + } + lo, hi = min(lo, got), max(hi, got) + } + if hi-lo < window/4 { + t.Errorf("200 draws spanned only %v of a %v window; want them spread", hi-lo, window) + } +} + +// A VU waits once: a start that failed and is retried does not wait again. +func TestPhaseInWaitsOnce(t *testing.T) { + const window = 40 * time.Millisecond + r := &runtime{cfg: &userclass.Config{Dyn: dynconfig.Static(knobs{PhaseIn: dynconfig.Seconds(window)})}} + r.phaseIn(7) + start := time.Now() + r.phaseIn(7) + if again := time.Since(start); again > window/4 { + t.Errorf("second phaseIn waited %v, want none", again) + } + if _, ok := r.phasedIn.Load(int64(7)); !ok { + t.Error("phasedIn does not record the goroutine") + } } // A router 503 is the fleet being full or the control plane being busy, @@ -817,7 +1127,9 @@ func TestRunStep_KeepsActorThroughRouterCapacityErrors(t *testing.T) { for _, status := range []int{503, 504, 429} { t.Run(strconv.Itoa(status), func(t *testing.T) { ctl := &fakeControlClient{} - u := newTestUser(t, &fake.Server{Status: status}, ctl, knobs{}) + // A zero budget gives the wake one attempt: this test is about + // the actor being kept, not about the retry. + u := newTestUser(t, &fake.Server{Status: status}, ctl, knobs{ResumeRetryBudget: 0}) for range maxConsecutiveStepFailures + 2 { if u.runStep(context.Background(), pingStep) { t.Fatal("runStep = true with a failing router") @@ -914,39 +1226,66 @@ func TestDwell(t *testing.T) { } } -// wakeFirstTouchCount returns the cumulative WakeFirstTouch count (success + -// failure) recorded in Prometheus so far. -func wakeFirstTouchCount(t *testing.T) float64 { +// rowCount returns the cumulative count of the locust stats row `name` +// recorded in Prometheus so far, and how many of those were failures. +func rowCount(t *testing.T, name string) (total, failures float64) { t.Helper() mfs, err := prometheus.DefaultGatherer.Gather() if err != nil { t.Fatal(err) } - var total float64 for _, mf := range mfs { if mf.GetName() != "locust_requests_total" { continue } for _, m := range mf.GetMetric() { - var isWake bool + var isRow, failed bool for _, lp := range m.GetLabel() { - if lp.GetName() == "name" && lp.GetValue() == "WakeFirstTouch" { - isWake = true + switch lp.GetName() { + case "name": + isRow = lp.GetValue() == name + case "status": + failed = lp.GetValue() == "failure" } } - if isWake { - total += m.GetCounter().GetValue() + if !isRow { + continue + } + total += m.GetCounter().GetValue() + if failed { + failures += m.GetCounter().GetValue() } } } + return total, failures +} + +// wakeFirstTouchCount returns the cumulative WakeFirstTouch count (success + +// failure) recorded in Prometheus so far. +func wakeFirstTouchCount(t *testing.T) float64 { + t.Helper() + total, _ := rowCount(t, "WakeFirstTouch") return total } +// expectRows runs fn and checks it recorded wantTotal rows named `name`, +// wantFailed of them failures. +func expectRows(t *testing.T, name string, wantTotal, wantFailed float64, fn func()) { + t.Helper() + totalBefore, failedBefore := rowCount(t, name) + fn() + total, failed := rowCount(t, name) + if total-totalBefore != wantTotal || failed-failedBefore != wantFailed { + t.Errorf("%s rows recorded = %v (%v failed), want %v (%v failed)", + name, total-totalBefore, failed-failedBefore, wantTotal, wantFailed) + } +} + // TestLifecycleModeNone: in resident mode, startUser and runStep never park // or explicitly resume the actor, never record WakeFirstTouch, and // suspendAndDelete still suspends before deleting on teardown. func TestLifecycleModeNone(t *testing.T) { - ctl := &fakeControlClient{templateMemory: "1Gi"} + ctl := &fakeControlClient{templateMemory: 1 << 30} u := newTestUser(t, &fake.Server{}, ctl, knobs{ Lifecycle: dynconfig.Lifecycle{ LifecycleMode: dynconfig.LifecycleModeNone, @@ -958,7 +1297,7 @@ func TestLifecycleModeNone(t *testing.T) { if err != nil { t.Fatal(err) } - session, err := rt.startUser(context.Background(), loaded) + session, err := rt.startUser(context.Background(), 0, loaded) if err != nil { t.Fatal(err) } @@ -966,11 +1305,14 @@ func TestLifecycleModeNone(t *testing.T) { t.Error("session.awake = false after startUser in LifecycleModeNone") } wakesBefore := wakeFirstTouchCount(t) - for range 3 { - if !session.runStep(context.Background(), pingStep) { - t.Fatal("runStep = false in LifecycleModeNone") + // A resident step is no wake, so it records no TimeToWake either. + expectRows(t, "TimeToWake", 0, 0, func() { + for range 3 { + if !session.runStep(context.Background(), pingStep) { + t.Fatal("runStep = false in LifecycleModeNone") + } } - } + }) if got := wakeFirstTouchCount(t) - wakesBefore; got != 0 { t.Errorf("WakeFirstTouch recorded %v times in LifecycleModeNone, want 0", got) } @@ -995,7 +1337,7 @@ func TestLifecycleModeNone(t *testing.T) { // steps that actually wake from a parked state. A step number past the // script length is rejected in startUser before CreateActor. func TestSnapshotSteps(t *testing.T) { - ctl := &fakeControlClient{templateMemory: "1Gi"} + ctl := &fakeControlClient{templateMemory: 1 << 30} u := newTestUser(t, &fake.Server{}, ctl, knobs{ Lifecycle: dynconfig.Lifecycle{ LifecycleMode: dynconfig.LifecycleModePause, @@ -1008,7 +1350,7 @@ func TestSnapshotSteps(t *testing.T) { if err != nil { t.Fatal(err) } - session, err := rt.startUser(context.Background(), loaded) + session, err := rt.startUser(context.Background(), 0, loaded) if err != nil { t.Fatal(err) } @@ -1047,13 +1389,13 @@ func TestSnapshotSteps(t *testing.T) { } // A step number beyond len(loaded.Steps) is rejected before CreateActor. - badCtl := &fakeControlClient{templateMemory: "1Gi"} + badCtl := &fakeControlClient{templateMemory: 1 << 30} badUser := newTestUser(t, &fake.Server{}, badCtl, knobs{ Lifecycle: dynconfig.Lifecycle{LifecycleMode: dynconfig.LifecycleModePause}, SnapshotSteps: "2,99", }) badRT := &runtime{cfg: badUser.cfg} - if _, err := badRT.startUser(context.Background(), loaded); err == nil || !strings.Contains(err.Error(), "exceeds script") { + if _, err := badRT.startUser(context.Background(), 0, loaded); err == nil || !strings.Contains(err.Error(), "exceeds script") { t.Errorf("startUser with out-of-range snapshot step = %v, want error mentioning 'exceeds script'", err) } if got := countCalls(badCtl.recordedCalls(), "CreateActor"); got != 0 { diff --git a/internal/benchmarking/boomer/agentsession/knobs_test.go b/internal/benchmarking/boomer/agentsession/knobs_test.go index b85bb83c4e..4182716888 100644 --- a/internal/benchmarking/boomer/agentsession/knobs_test.go +++ b/internal/benchmarking/boomer/agentsession/knobs_test.go @@ -15,6 +15,7 @@ package agentsession import ( "testing" + "time" "go.ate.dev/substrate/internal/benchmarking/boomer/dynconfig" ) @@ -23,30 +24,36 @@ import ( // the values it cannot run on. func TestKnobsCodec(t *testing.T) { h := dynconfig.NewHolder(knobsCodec) - if got := dynconfig.Get[knobs](h); got != (knobs{}) { - t.Fatalf("fresh holder = %+v, want the zero defaults", got) + if got := dynconfig.Get[knobs](h); got != knobsCodec.Defaults || got.ResumeRetryBudget.Duration() != defaultResumeRetryBudget { + t.Fatalf("fresh holder = %+v, want the defaults with a %v retry budget", got, defaultResumeRetryBudget) } if _, _, err := h.Apply([]byte(`{ "resume_mode": "explicit", "lifecycle_mode": "pause", "agentsession_script": "coding-session", "agentsession_script_file": "/etc/agentsession/script.yaml", "agentsession_think_scale": 0.5, - "agentsession_snapshot_steps": "3,10,20,30,40,50,61" + "agentsession_snapshot_steps": "3,10,20,30,40,50,61", + "agentsession_phase_in": 300, + "agentsession_resume_retry_budget": 45.5 }`)); err != nil { t.Fatalf("Apply(valid): %v", err) } want := knobs{ - Lifecycle: dynconfig.Lifecycle{ResumeMode: dynconfig.ResumeModeExplicit, LifecycleMode: dynconfig.LifecycleModePause}, - Script: "coding-session", - ScriptFile: "/etc/agentsession/script.yaml", - ThinkScale: 0.5, - SnapshotSteps: "3,10,20,30,40,50,61", + Lifecycle: dynconfig.Lifecycle{ResumeMode: dynconfig.ResumeModeExplicit, LifecycleMode: dynconfig.LifecycleModePause}, + Script: "coding-session", + ScriptFile: "/etc/agentsession/script.yaml", + ThinkScale: 0.5, + SnapshotSteps: "3,10,20,30,40,50,61", + PhaseIn: dynconfig.Seconds(5 * time.Minute), + ResumeRetryBudget: dynconfig.Seconds(45500 * time.Millisecond), } if got := dynconfig.Get[knobs](h); got != want { t.Errorf("decoded =\n %+v, want\n %+v", got, want) } for name, payload := range map[string]string{ "negative think scale": `{"agentsession_think_scale": -1}`, + "negative phase-in": `{"agentsession_phase_in": -1}`, + "negative resume retry budget": `{"agentsession_resume_retry_budget": -1}`, "invalid resume mode": `{"resume_mode": "invalid_mode"}`, "invalid snapshot steps entry": `{"agentsession_snapshot_steps": "3,zero,10"}`, "non-positive snapshot steps entry": `{"agentsession_snapshot_steps": "0,10"}`, diff --git a/internal/e2e/fixtures/capabilities/capabilities-templates.yaml.tmpl b/internal/e2e/fixtures/capabilities/capabilities-templates.yaml.tmpl index c132736f3c..9f895b1b86 100644 --- a/internal/e2e/fixtures/capabilities/capabilities-templates.yaml.tmpl +++ b/internal/e2e/fixtures/capabilities/capabilities-templates.yaml.tmpl @@ -38,10 +38,8 @@ containers: timeoutSeconds: 60 resources: limits: - - name: cpu - quantity: "1" - - name: memory - quantity: 512Mi + cpuMillicores: 1000 + memoryBytes: 536870912 # 512Mi snapshotConfig: storageLocation: gs://${BUCKET_NAME}/ate-e2e-caps-default${FIXTURE_SUFFIX}/ @@ -74,9 +72,7 @@ containers: timeoutSeconds: 60 resources: limits: - - name: cpu - quantity: "1" - - name: memory - quantity: 512Mi + cpuMillicores: 1000 + memoryBytes: 536870912 # 512Mi snapshotConfig: storageLocation: gs://${BUCKET_NAME}/ate-e2e-caps-exact${FIXTURE_SUFFIX}/ diff --git a/internal/e2e/fixtures/probe/probe-multiactor-template.yaml.tmpl b/internal/e2e/fixtures/probe/probe-multiactor-template.yaml.tmpl index 01a35f7cc2..55138cdd5a 100644 --- a/internal/e2e/fixtures/probe/probe-multiactor-template.yaml.tmpl +++ b/internal/e2e/fixtures/probe/probe-multiactor-template.yaml.tmpl @@ -48,9 +48,7 @@ volumes: # Allow multiple actors to fit on one worker. resources: limits: - - name: cpu - quantity: "1" - - name: memory - quantity: 256Mi + cpuMillicores: 1000 + memoryBytes: 268435456 # 256Mi snapshotConfig: storageLocation: gs://${BUCKET_NAME}/ate-e2e${FIXTURE_SUFFIX}/ diff --git a/internal/e2e/fixtures/probe/probe-parking-template.yaml.tmpl b/internal/e2e/fixtures/probe/probe-parking-template.yaml.tmpl index 92f7f7ce29..9260f41647 100644 --- a/internal/e2e/fixtures/probe/probe-parking-template.yaml.tmpl +++ b/internal/e2e/fixtures/probe/probe-parking-template.yaml.tmpl @@ -44,12 +44,9 @@ volumes: path: actor-id # One actor's worth of CPU, matching the pool's per-worker limit in # probe-parking.yaml.tmpl: what makes the worker full after one placement. -# Quantities are strings in the proto. resources: limits: - - name: cpu - quantity: "1" - - name: memory - quantity: 512Mi + cpuMillicores: 1000 + memoryBytes: 536870912 # 512Mi snapshotConfig: storageLocation: gs://${BUCKET_NAME}/ate-e2e${FIXTURE_SUFFIX}/ diff --git a/internal/e2e/fixtures/probe/probe-sized-template.yaml.tmpl b/internal/e2e/fixtures/probe/probe-sized-template.yaml.tmpl index aea06d8029..4341b99270 100644 --- a/internal/e2e/fixtures/probe/probe-sized-template.yaml.tmpl +++ b/internal/e2e/fixtures/probe/probe-sized-template.yaml.tmpl @@ -38,12 +38,9 @@ containers: # advertises capacity, gate scheduling). CPU=2 makes NumCPU() inside the # sandbox a distinct, assertable value — the vCPU count of the micro-VM guest, # or what runsc --cpu-num-from-quota provisions the gVisor sentry with. -# Quantities are strings in the proto. resources: limits: - - name: cpu - quantity: "2" - - name: memory - quantity: 512Mi + cpuMillicores: 2000 + memoryBytes: 536870912 # 512Mi snapshotConfig: storageLocation: gs://${BUCKET_NAME}/ate-e2e${FIXTURE_SUFFIX}/ diff --git a/internal/e2e/fixtures/probe/probe-template.yaml.tmpl b/internal/e2e/fixtures/probe/probe-template.yaml.tmpl index 29e2bf9486..c2521de3f4 100644 --- a/internal/e2e/fixtures/probe/probe-template.yaml.tmpl +++ b/internal/e2e/fixtures/probe/probe-template.yaml.tmpl @@ -58,9 +58,7 @@ volumes: ${TEMPLATE_TRUST_BUNDLE} resources: limits: - - name: cpu - quantity: "1" - - name: memory - quantity: 512Mi + cpuMillicores: 1000 + memoryBytes: 536870912 # 512Mi snapshotConfig: storageLocation: gs://${BUCKET_NAME}/ate-e2e-probe${FIXTURE_SUFFIX}/ diff --git a/internal/e2e/fixtures/security/atespace-template.yaml.tmpl b/internal/e2e/fixtures/security/atespace-template.yaml.tmpl index 0cda6a3407..5dd69b2cd7 100644 --- a/internal/e2e/fixtures/security/atespace-template.yaml.tmpl +++ b/internal/e2e/fixtures/security/atespace-template.yaml.tmpl @@ -72,9 +72,7 @@ volumes: durableDir: {} resources: limits: - - name: cpu - quantity: "1" - - name: memory - quantity: 512Mi + cpuMillicores: 1000 + memoryBytes: 536870912 # 512Mi snapshotConfig: storageLocation: gs://${BUCKET_NAME}/ate-e2e${FIXTURE_SUFFIX}/ diff --git a/internal/e2e/fixtures/testserver/grpcecho-template.yaml.tmpl b/internal/e2e/fixtures/testserver/grpcecho-template.yaml.tmpl index e3d85ac91f..4d6e94e1cb 100644 --- a/internal/e2e/fixtures/testserver/grpcecho-template.yaml.tmpl +++ b/internal/e2e/fixtures/testserver/grpcecho-template.yaml.tmpl @@ -47,9 +47,7 @@ containers: timeoutSeconds: 60 resources: limits: - - name: cpu - quantity: "1" - - name: memory - quantity: 512Mi + cpuMillicores: 1000 + memoryBytes: 536870912 # 512Mi snapshotConfig: storageLocation: gs://${BUCKET_NAME}/ate-e2e-grpcecho${FIXTURE_SUFFIX}/ diff --git a/internal/e2e/fixtures/testserver/websocket-template.yaml.tmpl b/internal/e2e/fixtures/testserver/websocket-template.yaml.tmpl index c9166a545a..2f0cd18b60 100644 --- a/internal/e2e/fixtures/testserver/websocket-template.yaml.tmpl +++ b/internal/e2e/fixtures/testserver/websocket-template.yaml.tmpl @@ -33,9 +33,7 @@ containers: timeoutSeconds: 30 resources: limits: - - name: cpu - quantity: "1" - - name: memory - quantity: 512Mi + cpuMillicores: 1000 + memoryBytes: 536870912 # 512Mi snapshotConfig: storageLocation: gs://${BUCKET_NAME}/ate-e2e-websocket${FIXTURE_SUFFIX}/ diff --git a/internal/e2e/ingressauth.go b/internal/e2e/ingressauth.go new file mode 100644 index 0000000000..99f1a278fb --- /dev/null +++ b/internal/e2e/ingressauth.go @@ -0,0 +1,380 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package e2e + +import ( + "context" + "crypto/ecdsa" + "crypto/elliptic" + "crypto/rand" + "crypto/tls" + "crypto/x509" + "encoding/pem" + "fmt" + "os" + "strings" + "sync" + "testing" + "time" + + "go.ate.dev/substrate/internal/clustertrustbundle" + "go.ate.dev/substrate/internal/installdefaults" + certsv1 "k8s.io/api/certificates/v1" + certsv1beta1 "k8s.io/api/certificates/v1beta1" + corev1 "k8s.io/api/core/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/util/uuid" + "k8s.io/apimachinery/pkg/util/wait" + "k8s.io/client-go/kubernetes" + "k8s.io/utils/ptr" +) + +// IngressAuthModeEnv names the ingress router's --ingress-auth-mode the +// suites run against: IngressAuthDeprecatedInsecure or IngressAuthStaticMTLS. +// It has to match how the router was installed (ATE_INGRESS_AUTH_MODE for +// ate-setup); unset means deprecated-insecure. +const IngressAuthModeEnv = "E2E_INGRESS_AUTH_MODE" + +const ( + IngressAuthDeprecatedInsecure = "deprecated-insecure" + IngressAuthStaticMTLS = "static-mtls" +) + +// IngressAuthMode returns the router authentication mode the suites expect. +func IngressAuthMode() string { + if mode := os.Getenv(IngressAuthModeEnv); mode != "" { + return mode + } + return IngressAuthDeprecatedInsecure +} + +// IngressAuthModeRequired reports whether IngressAuthModeEnv must be set +// explicitly. In CI an unset mode would quietly skip the ingress +// authentication suite, so it is an error there; REQUIRE_INGRESS_AUTH_MODE +// makes it one locally too. +func IngressAuthModeRequired() bool { + return os.Getenv("CI") == "true" || os.Getenv("REQUIRE_INGRESS_AUTH_MODE") == "true" +} + +// IngressAuthModeSet reports whether IngressAuthModeEnv is set. +func IngressAuthModeSet() bool { return os.Getenv(IngressAuthModeEnv) != "" } + +// IngressMTLS reports whether the router requires mTLS client certificates, +// and so whether RouterClient dials its TLS listeners as RouterClientIdentity. +func IngressMTLS() bool { return IngressAuthMode() == IngressAuthStaticMTLS } + +// The router client's identity. +const ( + RouterClientNamespace = "ate-e2e-router-client" + RouterClientServiceAccount = "router-client" +) + +// RouterClientSPIFFEID is the SPIFFE ID the podidentity signer issues for +// serviceAccount in RouterClientNamespace. +func RouterClientSPIFFEID(serviceAccount string) string { + return installdefaults.SPIFFEID(RouterClientNamespace, serviceAccount) +} + +const ( + // podIdentitySignerName issues the SPIFFE client certificates the router + // verifies. + podIdentitySignerName = "podidentity.podcert.ate.dev/identity" + // routerClientCertTimeout bounds the wait for the signer to issue. + routerClientCertTimeout = 2 * time.Minute +) + +var ( + routerClientCertsMu sync.Mutex + routerClientCerts = map[string]tls.Certificate{} +) + +// RouterClientCertificate returns a certificate the podidentity signer issued +// for serviceAccount in RouterClientNamespace. Each ServiceAccount's +// certificate is requested once per test process, and lasts longer than any +// suite. +func RouterClientCertificate(t *testing.T, ctx context.Context, serviceAccount string) tls.Certificate { + t.Helper() + routerClientCertsMu.Lock() + defer routerClientCertsMu.Unlock() + if cert, ok := routerClientCerts[serviceAccount]; ok { + return cert + } + + clientset := GetClients().K8s + if err := ensureRouterClientNamespace(ctx, clientset); err != nil { + t.Fatalf("preparing namespace %s: %v", RouterClientNamespace, err) + } + cert, err := requestPodIdentityCertificate(ctx, clientset, RouterClientNamespace, serviceAccount, routerClientCertTimeout) + if err != nil { + t.Fatalf("requesting a podidentity certificate for %s/%s: %v", RouterClientNamespace, serviceAccount, err) + } + routerClientCerts[serviceAccount] = cert + return cert +} + +// requestPodIdentityCertificate creates a PodCertificateRequest for +// serviceAccount in namespace, waits for the podidentity signer to issue it, +// and deletes it again. The key never leaves this process. +// +// The pod, ServiceAccount and node the request names do not exist: +// kube-apiserver only checks them for requests a kubelet creates, and the +// signer takes the identity from the request. +func requestPodIdentityCertificate(ctx context.Context, clientset kubernetes.Interface, namespace, serviceAccount string, timeout time.Duration) (tls.Certificate, error) { + key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + if err != nil { + return tls.Certificate{}, fmt.Errorf("generating key: %w", err) + } + csr, err := x509.CreateCertificateRequest(rand.Reader, &x509.CertificateRequest{}, key) + if err != nil { + return tls.Certificate{}, fmt.Errorf("creating stub CSR: %w", err) + } + + // Unique per call: suites run as concurrent processes and share the + // namespace. rand.Text is upper-case base32; object names are lower case. + name := "e2e-" + serviceAccount + "-" + strings.ToLower(rand.Text()[:8]) + pcr := &certsv1.PodCertificateRequest{ + ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: namespace}, + Spec: certsv1.PodCertificateRequestSpec{ + SignerName: podIdentitySignerName, + PodName: name, + PodUID: uuid.NewUUID(), + ServiceAccountName: serviceAccount, + ServiceAccountUID: uuid.NewUUID(), + NodeName: "e2e-test-process", + NodeUID: uuid.NewUUID(), + MaxExpirationSeconds: ptr.To(int32(24 * 60 * 60)), + StubPKCS10Request: csr, + }, + } + pcrs, err := podCertificateRequests(clientset, namespace) + if err != nil { + return tls.Certificate{}, err + } + if err := pcrs.create(ctx, pcr); err != nil { + return tls.Certificate{}, fmt.Errorf("creating PodCertificateRequest %s/%s: %w", namespace, name, err) + } + defer func() { + // A leftover request is harmless: kube-apiserver garbage collects + // old PodCertificateRequests. + delCtx, cancel := context.WithTimeout(context.Background(), 30*time.Second) + defer cancel() + if err := pcrs.delete(delCtx, name); err != nil && !apierrors.IsNotFound(err) { + fmt.Fprintf(os.Stderr, "Failed to delete PodCertificateRequest %s/%s: %v\n", namespace, name, err) + } + }() + + var chainPEM string + err = wait.PollUntilContextTimeout(ctx, 500*time.Millisecond, timeout, true, func(ctx context.Context) (bool, error) { + status, err := pcrs.status(ctx, name) + if err != nil { + return false, err + } + for _, c := range status.Conditions { + if c.Status != metav1.ConditionTrue { + continue + } + switch c.Type { + case certsv1.PodCertificateRequestConditionTypeIssued: + chainPEM = status.CertificateChain + return true, nil + case certsv1.PodCertificateRequestConditionTypeDenied, certsv1.PodCertificateRequestConditionTypeFailed: + return false, fmt.Errorf("%s: %s: %s", c.Type, c.Reason, c.Message) + } + } + return false, nil + }) + if err != nil { + return tls.Certificate{}, fmt.Errorf("waiting for PodCertificateRequest %s/%s to be issued: %w", namespace, name, err) + } + + cert := tls.Certificate{PrivateKey: key} + for rest := []byte(chainPEM); ; { + var block *pem.Block + block, rest = pem.Decode(rest) + if block == nil { + break + } + cert.Certificate = append(cert.Certificate, block.Bytes) + } + if len(cert.Certificate) == 0 { + return tls.Certificate{}, fmt.Errorf("PodCertificateRequest %s/%s was issued with no certificates", namespace, name) + } + leaf, err := x509.ParseCertificate(cert.Certificate[0]) + if err != nil { + return tls.Certificate{}, fmt.Errorf("parsing the issued leaf certificate: %w", err) + } + if !key.PublicKey.Equal(leaf.PublicKey) { + return tls.Certificate{}, fmt.Errorf("the issued certificate does not carry the requested public key") + } + if want := installdefaults.SPIFFEID(namespace, serviceAccount); len(leaf.URIs) != 1 || leaf.URIs[0].String() != want { + return tls.Certificate{}, fmt.Errorf("the podidentity signer issued a certificate with URI SANs %v, want exactly %s", leaf.URIs, want) + } + cert.Leaf = leaf + return cert, nil +} + +// pcrClient creates, reads and deletes PodCertificateRequests through +// whichever of certificates.k8s.io v1 and v1beta1 the cluster serves, +// preferring v1, as the podcertificate controller does. +type pcrClient struct { + clientset kubernetes.Interface + namespace string + v1 bool +} + +func podCertificateRequests(clientset kubernetes.Interface, namespace string) (*pcrClient, error) { + for _, version := range []string{"v1", "v1beta1"} { + resources, err := clientset.Discovery().ServerResourcesForGroupVersion("certificates.k8s.io/" + version) + if apierrors.IsNotFound(err) { + continue + } + if err != nil { + return nil, fmt.Errorf("discovering PodCertificateRequest %s: %w", version, err) + } + for _, r := range resources.APIResources { + if r.Name == "podcertificaterequests" { + return &pcrClient{clientset: clientset, namespace: namespace, v1: version == "v1"}, nil + } + } + } + return nil, fmt.Errorf("the cluster serves neither v1 nor v1beta1 PodCertificateRequest") +} + +func (c *pcrClient) create(ctx context.Context, pcr *certsv1.PodCertificateRequest) error { + if c.v1 { + _, err := c.clientset.CertificatesV1().PodCertificateRequests(c.namespace).Create(ctx, pcr, metav1.CreateOptions{}) + return err + } + beta := &certsv1beta1.PodCertificateRequest{ + ObjectMeta: pcr.ObjectMeta, + Spec: certsv1beta1.PodCertificateRequestSpec{ + SignerName: pcr.Spec.SignerName, + PodName: pcr.Spec.PodName, + PodUID: pcr.Spec.PodUID, + ServiceAccountName: pcr.Spec.ServiceAccountName, + ServiceAccountUID: pcr.Spec.ServiceAccountUID, + NodeName: pcr.Spec.NodeName, + NodeUID: pcr.Spec.NodeUID, + MaxExpirationSeconds: pcr.Spec.MaxExpirationSeconds, + StubPKCS10Request: pcr.Spec.StubPKCS10Request, + }, + } + _, err := c.clientset.CertificatesV1beta1().PodCertificateRequests(c.namespace).Create(ctx, beta, metav1.CreateOptions{}) + return err +} + +func (c *pcrClient) status(ctx context.Context, name string) (certsv1.PodCertificateRequestStatus, error) { + if c.v1 { + pcr, err := c.clientset.CertificatesV1().PodCertificateRequests(c.namespace).Get(ctx, name, metav1.GetOptions{}) + if err != nil { + return certsv1.PodCertificateRequestStatus{}, err + } + return pcr.Status, nil + } + pcr, err := c.clientset.CertificatesV1beta1().PodCertificateRequests(c.namespace).Get(ctx, name, metav1.GetOptions{}) + if err != nil { + return certsv1.PodCertificateRequestStatus{}, err + } + return certsv1.PodCertificateRequestStatus(pcr.Status), nil +} + +func (c *pcrClient) delete(ctx context.Context, name string) error { + if c.v1 { + return c.clientset.CertificatesV1().PodCertificateRequests(c.namespace).Delete(ctx, name, metav1.DeleteOptions{}) + } + return c.clientset.CertificatesV1beta1().PodCertificateRequests(c.namespace).Delete(ctx, name, metav1.DeleteOptions{}) +} + +// ensureRouterClientNamespace creates RouterClientNamespace, leaving it in +// place if it already exists. The namespace carries NamespaceLabel, so +// hack/cleanup-e2e.sh removes it with the rest. +func ensureRouterClientNamespace(ctx context.Context, clientset kubernetes.Interface) error { + _, err := clientset.CoreV1().Namespaces().Create(ctx, &corev1.Namespace{ + ObjectMeta: metav1.ObjectMeta{ + Name: RouterClientNamespace, + Labels: map[string]string{NamespaceLabel: "true"}, + }, + }, metav1.CreateOptions{}) + if err != nil && !apierrors.IsAlreadyExists(err) { + return fmt.Errorf("creating namespace: %w", err) + } + return nil +} + +// RouterServerName is the DNS name on the router's servicedns serving +// certificate. +func RouterServerName() string { + return fmt.Sprintf("%s.%s.svc", ResourceName("atenet-router"), SystemNamespace()) +} + +// RouterTLSConfig returns a TLS config for the router's TLS listeners that +// verifies the router's serving certificate and presents clientCert, which +// may be nil to present none. The caller sets NextProtos. +func RouterTLSConfig(ctx context.Context, clientCert *tls.Certificate) (*tls.Config, error) { + roots, err := serviceDNSTrustPool(ctx, GetClients().K8s) + if err != nil { + return nil, fmt.Errorf("loading the servicedns trust bundle: %w", err) + } + cfg := &tls.Config{ + MinVersion: tls.VersionTLS12, + RootCAs: roots, + ServerName: RouterServerName(), + } + if clientCert != nil { + cfg.Certificates = []tls.Certificate{*clientCert} + } + return cfg, nil +} + +const ( + // serviceDNSSignerName and liveBundleSelector mirror the + // clusterTrustBundle projected-volume sources that in-cluster clients + // mount to verify ateapi's serving cert. + serviceDNSSignerName = "servicedns.podcert.ate.dev/identity" + liveBundleSelector = "podcert.ate.dev/canarying=live" +) + +// serviceDNSTrustPool returns the live trust bundle of the servicedns signer, +// which issues the serving certificates of substrate's in-cluster Services +// (ateapi and atenet-router among them). +func serviceDNSTrustPool(ctx context.Context, clientset kubernetes.Interface) (*x509.CertPool, error) { + bundles, err := clustertrustbundle.NewClient(clientset, nil) + if err != nil { + return nil, fmt.Errorf("failed to discover ClusterTrustBundle API: %w", err) + } + ctbs, err := bundles.List(ctx, metav1.ListOptions{ + LabelSelector: liveBundleSelector, + }) + if err != nil { + return nil, fmt.Errorf("failed to list ClusterTrustBundles: %w", err) + } + + pool := x509.NewCertPool() + found := false + for _, ctb := range ctbs.Items { + if ctb.Spec.SignerName != serviceDNSSignerName { + continue + } + if !pool.AppendCertsFromPEM([]byte(ctb.Spec.TrustBundle)) { + return nil, fmt.Errorf("ClusterTrustBundle %q contains no valid certificates", ctb.ObjectMeta.Name) + } + found = true + } + if !found { + return nil, fmt.Errorf("no live ClusterTrustBundle found for signer %q", serviceDNSSignerName) + } + return pool, nil +} diff --git a/internal/e2e/ingressauth_test.go b/internal/e2e/ingressauth_test.go new file mode 100644 index 0000000000..d97dc14efe --- /dev/null +++ b/internal/e2e/ingressauth_test.go @@ -0,0 +1,242 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package e2e + +import ( + "context" + "crypto/ecdsa" + "crypto/elliptic" + "crypto/rand" + "crypto/x509" + "encoding/pem" + "math/big" + "net/url" + "strings" + "testing" + "time" + + "go.ate.dev/substrate/internal/installdefaults" + certsv1 "k8s.io/api/certificates/v1" + certsv1beta1 "k8s.io/api/certificates/v1beta1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/client-go/kubernetes/fake" + k8stesting "k8s.io/client-go/testing" +) + +// fakePodIdentitySigner answers PodCertificateRequests the way the podidentity +// signer does, signing the stub CSR's key with a throwaway CA. +type fakePodIdentitySigner struct { + t *testing.T + ca *x509.Certificate + caKey *ecdsa.PrivateKey + uri func(spec certsv1.PodCertificateRequestSpec) string + denied bool + // got is the spec of the last request created. + got certsv1.PodCertificateRequestSpec +} + +func newFakePodIdentitySigner(t *testing.T) *fakePodIdentitySigner { + t.Helper() + caKey, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + if err != nil { + t.Fatal(err) + } + template := &x509.Certificate{ + SerialNumber: big.NewInt(1), + NotBefore: time.Now().Add(-time.Minute), + NotAfter: time.Now().Add(time.Hour), + IsCA: true, + BasicConstraintsValid: true, + KeyUsage: x509.KeyUsageCertSign, + } + der, err := x509.CreateCertificate(rand.Reader, template, template, &caKey.PublicKey, caKey) + if err != nil { + t.Fatal(err) + } + ca, err := x509.ParseCertificate(der) + if err != nil { + t.Fatal(err) + } + return &fakePodIdentitySigner{ + t: t, ca: ca, caKey: caKey, + uri: func(spec certsv1.PodCertificateRequestSpec) string { + // The namespace is not in the spec; the tests use one namespace. + return installdefaults.SPIFFEID(RouterClientNamespace, spec.ServiceAccountName) + }, + } +} + +func (s *fakePodIdentitySigner) status(spec certsv1.PodCertificateRequestSpec) certsv1.PodCertificateRequestStatus { + s.t.Helper() + s.got = spec + if s.denied { + return certsv1.PodCertificateRequestStatus{Conditions: []metav1.Condition{{ + Type: certsv1.PodCertificateRequestConditionTypeDenied, Status: metav1.ConditionTrue, + Reason: "UnsupportedKeyType", Message: "no", + }}} + } + csr, err := x509.ParseCertificateRequest(spec.StubPKCS10Request) + if err != nil { + s.t.Fatalf("parsing the stub CSR: %v", err) + } + uri, err := url.Parse(s.uri(spec)) + if err != nil { + s.t.Fatal(err) + } + der, err := x509.CreateCertificate(rand.Reader, &x509.Certificate{ + SerialNumber: big.NewInt(2), + NotBefore: time.Now().Add(-time.Minute), + NotAfter: time.Now().Add(time.Hour), + URIs: []*url.URL{uri}, + ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth}, + }, s.ca, csr.PublicKey, s.caKey) + if err != nil { + s.t.Fatal(err) + } + return certsv1.PodCertificateRequestStatus{ + Conditions: []metav1.Condition{{ + Type: certsv1.PodCertificateRequestConditionTypeIssued, Status: metav1.ConditionTrue, + Reason: "Reason", Message: "Issued", + }}, + CertificateChain: string(pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der})), + } +} + +// install serves PodCertificateRequest at version and has s answer every +// request created through it. +func (s *fakePodIdentitySigner) install(kc *fake.Clientset, version string) { + kc.Resources = []*metav1.APIResourceList{{ + GroupVersion: "certificates.k8s.io/" + version, + APIResources: []metav1.APIResource{{Name: "podcertificaterequests", Namespaced: true}}, + }} + // Returning false hands the mutated object on to the object tracker. + kc.PrependReactor("create", "podcertificaterequests", func(action k8stesting.Action) (bool, runtime.Object, error) { + switch pcr := action.(k8stesting.CreateAction).GetObject().(type) { + case *certsv1.PodCertificateRequest: + pcr.Status = s.status(pcr.Spec) + case *certsv1beta1.PodCertificateRequest: + pcr.Status = certsv1beta1.PodCertificateRequestStatus(s.status(certsv1.PodCertificateRequestSpec{ + SignerName: pcr.Spec.SignerName, + ServiceAccountName: pcr.Spec.ServiceAccountName, + StubPKCS10Request: pcr.Spec.StubPKCS10Request, + })) + default: + s.t.Fatalf("unexpected PodCertificateRequest type %T", pcr) + } + return false, nil, nil + }) +} + +func TestRequestPodIdentityCertificate(t *testing.T) { + for _, version := range []string{"v1", "v1beta1"} { + t.Run(version, func(t *testing.T) { + kc := fake.NewSimpleClientset() + signer := newFakePodIdentitySigner(t) + signer.install(kc, version) + + cert, err := requestPodIdentityCertificate(context.Background(), kc, RouterClientNamespace, "router-client", time.Second) + if err != nil { + t.Fatalf("requestPodIdentityCertificate: %v", err) + } + + if signer.got.SignerName != podIdentitySignerName { + t.Errorf("signerName = %q, want %q", signer.got.SignerName, podIdentitySignerName) + } + if signer.got.ServiceAccountName != "router-client" { + t.Errorf("serviceAccountName = %q, want router-client", signer.got.ServiceAccountName) + } + if want := RouterClientSPIFFEID("router-client"); cert.Leaf == nil || len(cert.Leaf.URIs) != 1 || cert.Leaf.URIs[0].String() != want { + t.Fatalf("leaf URIs = %v, want [%s]", cert.Leaf.URIs, want) + } + key, ok := cert.PrivateKey.(*ecdsa.PrivateKey) + if !ok || !key.PublicKey.Equal(cert.Leaf.PublicKey) { + t.Errorf("the private key does not match the leaf certificate") + } + + // The request is deleted once the certificate is in hand. + var remaining int + if version == "v1" { + list, err := kc.CertificatesV1().PodCertificateRequests(RouterClientNamespace).List(context.Background(), metav1.ListOptions{}) + if err != nil { + t.Fatal(err) + } + remaining = len(list.Items) + } else { + list, err := kc.CertificatesV1beta1().PodCertificateRequests(RouterClientNamespace).List(context.Background(), metav1.ListOptions{}) + if err != nil { + t.Fatal(err) + } + remaining = len(list.Items) + } + if remaining != 0 { + t.Errorf("%d PodCertificateRequests left behind, want 0", remaining) + } + }) + } +} + +func TestRequestPodIdentityCertificateErrors(t *testing.T) { + for _, tc := range []struct { + name string + setup func(t *testing.T, kc *fake.Clientset) + wantErr string + }{ + { + name: "denied", + setup: func(t *testing.T, kc *fake.Clientset) { + s := newFakePodIdentitySigner(t) + s.denied = true + s.install(kc, "v1") + }, + wantErr: "Denied", + }, + { + name: "wrong SPIFFE ID", + setup: func(t *testing.T, kc *fake.Clientset) { + s := newFakePodIdentitySigner(t) + s.uri = func(certsv1.PodCertificateRequestSpec) string { + return "spiffe://cluster.local/ns/other/sa/router-client" + } + s.install(kc, "v1") + }, + wantErr: "URI SANs", + }, + { + name: "never issued", + setup: func(t *testing.T, kc *fake.Clientset) { + kc.Resources = []*metav1.APIResourceList{{ + GroupVersion: "certificates.k8s.io/v1", + APIResources: []metav1.APIResource{{Name: "podcertificaterequests", Namespaced: true}}, + }} + }, + wantErr: "to be issued", + }, + { + name: "not served", + setup: func(*testing.T, *fake.Clientset) {}, + wantErr: "neither v1 nor v1beta1", + }, + } { + t.Run(tc.name, func(t *testing.T) { + kc := fake.NewSimpleClientset() + tc.setup(t, kc) + _, err := requestPodIdentityCertificate(context.Background(), kc, RouterClientNamespace, "router-client", 100*time.Millisecond) + if err == nil || !strings.Contains(err.Error(), tc.wantErr) { + t.Fatalf("requestPodIdentityCertificate error = %v, want one containing %q", err, tc.wantErr) + } + }) + } +} diff --git a/internal/e2e/router_client.go b/internal/e2e/router_client.go index b46ebd16e1..ee5361c499 100644 --- a/internal/e2e/router_client.go +++ b/internal/e2e/router_client.go @@ -18,6 +18,7 @@ import ( "bufio" "bytes" "context" + "crypto/tls" "fmt" "io" "net" @@ -26,6 +27,7 @@ import ( "strconv" "strings" "sync" + "testing" "time" "go.ate.dev/substrate/internal/ateclient" @@ -36,23 +38,32 @@ import ( "k8s.io/client-go/rest" ) +// atenet-router's Service ports (see manifests/ate-install/atenet-router.yaml). +// The CONNECT ports are distinct listeners from the HTTP ones Get/PostJSON +// use: the HTTP listeners never enable the CONNECT method, only the +// connect_terminate ones do. In static-mtls mode only the TLS ports serve. const ( - // routerConnectServicePort is atenet-router's Service port for - // CONNECT-tunneled traffic (see manifests/ate-install/atenet-router.yaml). - // It is a distinct listener from the plain HTTP one Get/PostJSON use: - // atenet-router's ingress_http_listener never enables the CONNECT method, - // only connect_terminate does. - routerConnectServicePort = 8081 + routerHTTPServicePort = 80 + routerHTTPSServicePort = 443 + routerConnectServicePort = 8081 + routerConnectTLSServicePort = 8444 ) // RouterClient sends HTTP requests to actors through the ingress atenet-router, the // same way real traffic arrives (so the request is routed and, if needed, the // actor is resumed). It port-forwards the router Service, mirroring the // approach in internal/ateclient. +// +// When the router requires mTLS (see IngressMTLS), the client dials the TLS +// listeners and presents the certificate the podidentity signer issued to +// RouterClientServiceAccount, which the router allowlists. type RouterClient struct { + address string baseURL string http *http.Client stop func() + // tlsConfig is nil when the router accepts plaintext. + tlsConfig *tls.Config // config/clientset are retained to lazily open a second port-forward, to // routerConnectServicePort, only if Connect is ever called -- most callers @@ -68,7 +79,8 @@ type RouterClient struct { // NewRouterClient establishes a port-forward to the ingress atenet-router. Call Close // to tear it down. -func NewRouterClient(ctx context.Context) (*RouterClient, error) { +func NewRouterClient(t *testing.T, ctx context.Context) (*RouterClient, error) { + t.Helper() config, err := ateclient.LoadKubeConfig(KubeConfig, KubeContext) if err != nil { return nil, fmt.Errorf("loading kubeconfig: %w", err) @@ -78,15 +90,31 @@ func NewRouterClient(ctx context.Context) (*RouterClient, error) { return nil, fmt.Errorf("creating k8s client: %w", err) } - localPort, stop, err := portforward.ServicePortForward(ctx, config, clientset, SystemNamespace(), ResourceName("atenet-router"), 80) + scheme, servicePort := "http", routerHTTPServicePort + var tlsConfig *tls.Config + if IngressMTLS() { + cert := RouterClientCertificate(t, ctx, RouterClientServiceAccount) + tlsConfig, err = RouterTLSConfig(ctx, &cert) + if err != nil { + return nil, err + } + scheme, servicePort = "https", routerHTTPSServicePort + } + + localPort, stop, err := portforward.ServicePortForward(ctx, config, clientset, SystemNamespace(), ResourceName("atenet-router"), int32(servicePort)) if err != nil { return nil, err } + address := fmt.Sprintf("127.0.0.1:%d", localPort) + transport := http.DefaultTransport.(*http.Transport).Clone() + transport.TLSClientConfig = tlsConfig return &RouterClient{ - baseURL: fmt.Sprintf("http://127.0.0.1:%d", localPort), - http: &http.Client{Timeout: 30 * time.Second}, + address: address, + baseURL: scheme + "://" + address, + http: &http.Client{Transport: transport, Timeout: 30 * time.Second}, stop: stop, + tlsConfig: tlsConfig, config: config, clientset: clientset, }, nil @@ -100,34 +128,60 @@ func (c *RouterClient) Close() { } } -// BaseURL returns the local router port-forward address. +// BaseURL returns the local router port-forward address, as an http:// or, +// when the router requires mTLS, an https:// URL. func (c *RouterClient) BaseURL() string { return c.baseURL } +// Address returns the local router port-forward's host:port, for a caller that +// has to control the protocol it speaks to the router itself. +func (c *RouterClient) Address() string { + return c.address +} + +// TLSConfig returns a copy of the TLS config the client dials the router +// with, or nil when the router accepts plaintext. +func (c *RouterClient) TLSConfig() *tls.Config { + if c.tlsConfig == nil { + return nil + } + return c.tlsConfig.Clone() +} + // Get issues GET path to actor through the router. The caller must close the body. func (c *RouterClient) Get(ctx context.Context, actorRef resources.ActorRef, path string) (*http.Response, error) { - return c.request(ctx, http.MethodGet, actorRef, path, nil) + return c.Do(ctx, http.MethodGet, actorRef, path, nil) } // PostJSON issues a POST with a JSON body to an Actor through the router. The // caller must close the response body. func (c *RouterClient) PostJSON(ctx context.Context, actorRef resources.ActorRef, path string, body []byte) (*http.Response, error) { - return c.request(ctx, http.MethodPost, actorRef, path, bytes.NewReader(body)) + return c.do(ctx, http.MethodPost, actorRef, path, bytes.NewReader(body), "application/json") } -func (c *RouterClient) request(ctx context.Context, method string, actorRef resources.ActorRef, path string, body io.Reader) (*http.Response, error) { +// Do issues a method request for path to actorRef through the router. The +// caller must close the response body. +func (c *RouterClient) Do(ctx context.Context, method string, actorRef resources.ActorRef, path string, body io.Reader) (*http.Response, error) { + return c.do(ctx, method, actorRef, path, body, "") +} + +func (c *RouterClient) do(ctx context.Context, method string, actorRef resources.ActorRef, path string, body io.Reader, contentType string) (*http.Response, error) { req, err := http.NewRequestWithContext(ctx, method, c.baseURL+path, body) if err != nil { return nil, err } - if method == http.MethodPost { - req.Header.Set("Content-Type", "application/json") + if contentType != "" { + req.Header.Set("Content-Type", contentType) } req.Header.Set(atenet.TargetActorHeader, actorRef.String()) return c.http.Do(req) } +func (c *RouterClient) DoReq(req *http.Request) (*http.Response, error) { + return c.http.Do(req) +} + // Connect opens a CONNECT tunnel through the router to port on actorRef, // exercising atenet-router's arbitrary-port ingress support: the target port // travels in the CONNECT authority (e.g. "my-actor.team-a...:9090"), the same @@ -149,10 +203,20 @@ func (c *RouterClient) ConnectWithHeaders(ctx context.Context, actorRef resource return nil, err } - rawConn, err := (&net.Dialer{}).DialContext(ctx, "tcp", c.connectAddr) + var rawConn net.Conn + tcpConn, err := (&net.Dialer{}).DialContext(ctx, "tcp", c.connectAddr) if err != nil { return nil, fmt.Errorf("connecting to router's CONNECT listener: %w", err) } + rawConn = tcpConn + if c.tlsConfig != nil { + tlsConn := tls.Client(tcpConn, c.tlsConfig.Clone()) + if err := tlsConn.HandshakeContext(ctx); err != nil { + _ = tcpConn.Close() + return nil, fmt.Errorf("TLS handshake with router's CONNECT listener: %w", err) + } + rawConn = tlsConn + } destination := net.JoinHostPort(actorRef.Name, strconv.Itoa(port)) req := &http.Request{ @@ -194,7 +258,11 @@ func (c *RouterClient) ConnectWithHeaders(ctx context.Context, actorRef resource // in one test don't each pay for a fresh port-forward. func (c *RouterClient) ensureConnectPortForward(ctx context.Context) error { c.connectOnce.Do(func() { - localPort, stop, err := portforward.ServicePortForward(ctx, c.config, c.clientset, SystemNamespace(), ResourceName("atenet-router"), routerConnectServicePort) + servicePort := routerConnectServicePort + if c.tlsConfig != nil { + servicePort = routerConnectTLSServicePort + } + localPort, stop, err := portforward.ServicePortForward(ctx, c.config, c.clientset, SystemNamespace(), ResourceName("atenet-router"), int32(servicePort)) if err != nil { c.connectErr = fmt.Errorf("port-forwarding to the router's CONNECT listener: %w", err) return diff --git a/internal/e2e/router_client_test.go b/internal/e2e/router_client_test.go index f01aa04fc4..4e288001a7 100644 --- a/internal/e2e/router_client_test.go +++ b/internal/e2e/router_client_test.go @@ -16,10 +16,19 @@ package e2e import ( "context" + "crypto/ecdsa" + "crypto/elliptic" + "crypto/rand" + "crypto/tls" + "crypto/x509" "io" + "math/big" "net/http" + "net/http/httptest" + "net/url" "strings" "testing" + "time" "go.ate.dev/substrate/internal/atenet" "go.ate.dev/substrate/internal/resources" @@ -72,3 +81,99 @@ type testRoundTripper func(*http.Request) (*http.Response, error) func (f testRoundTripper) RoundTrip(request *http.Request) (*http.Response, error) { return f(request) } + +// With the router in static-mtls mode, Connect has to complete a TLS +// handshake, presenting the client certificate, before it sends the CONNECT. +func TestRouterClientConnectTLS(t *testing.T) { + clientCert := selfSignedClientCert(t, "spiffe://cluster.local/ns/test/sa/client") + clientCAs := x509.NewCertPool() + clientCAs.AddCert(clientCert.Leaf) + + var gotPeer string + server := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodConnect { + t.Errorf("method = %q, want CONNECT", r.Method) + } + if r.Host != "fetcher:9090" { + t.Errorf("CONNECT authority = %q, want fetcher:9090", r.Host) + } + if got := r.Header.Get(atenet.TargetActorHeader); got != "demo/fetcher" { + t.Errorf("target actor = %q, want demo/fetcher", got) + } + if len(r.TLS.PeerCertificates) > 0 && len(r.TLS.PeerCertificates[0].URIs) > 0 { + gotPeer = r.TLS.PeerCertificates[0].URIs[0].String() + } + conn, buf, err := w.(http.Hijacker).Hijack() + if err != nil { + t.Errorf("hijacking: %v", err) + return + } + defer conn.Close() + _, _ = buf.WriteString("HTTP/1.1 200 OK\r\n\r\ntunneled") + _ = buf.Flush() + })) + server.TLS = &tls.Config{ClientAuth: tls.RequireAndVerifyClientCert, ClientCAs: clientCAs} + server.StartTLS() + defer server.Close() + + tlsConfig := &tls.Config{ + RootCAs: x509.NewCertPool(), + Certificates: []tls.Certificate{clientCert}, + ServerName: "example.com", + } + tlsConfig.RootCAs.AddCert(server.Certificate()) + client := &RouterClient{tlsConfig: tlsConfig, connectAddr: server.Listener.Addr().String()} + client.connectOnce.Do(func() {}) + + conn, err := client.Connect(context.Background(), resources.ActorRef{Atespace: "demo", Name: "fetcher"}, 9090) + if err != nil { + t.Fatalf("Connect: %v", err) + } + defer conn.Close() + if _, ok := conn.(*bufferedConn).Conn.(*tls.Conn); !ok { + t.Errorf("Connect returned a %T tunnel, want one over TLS", conn.(*bufferedConn).Conn) + } + got, err := io.ReadAll(conn) + if err != nil { + t.Fatalf("reading the tunnel: %v", err) + } + if string(got) != "tunneled" { + t.Errorf("tunnel carried %q, want %q", got, "tunneled") + } + if gotPeer != "spiffe://cluster.local/ns/test/sa/client" { + t.Errorf("router saw client identity %q, want the presented certificate's", gotPeer) + } +} + +// selfSignedClientCert returns a self-signed client certificate carrying uri +// as its only URI SAN. +func selfSignedClientCert(t *testing.T, uri string) tls.Certificate { + t.Helper() + key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + if err != nil { + t.Fatal(err) + } + u, err := url.Parse(uri) + if err != nil { + t.Fatal(err) + } + template := &x509.Certificate{ + SerialNumber: big.NewInt(1), + NotBefore: time.Now().Add(-time.Minute), + NotAfter: time.Now().Add(time.Hour), + URIs: []*url.URL{u}, + KeyUsage: x509.KeyUsageDigitalSignature | x509.KeyUsageCertSign, + ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth}, + BasicConstraintsValid: true, + IsCA: true, + } + der, err := x509.CreateCertificate(rand.Reader, template, template, &key.PublicKey, key) + if err != nil { + t.Fatal(err) + } + leaf, err := x509.ParseCertificate(der) + if err != nil { + t.Fatal(err) + } + return tls.Certificate{Certificate: [][]byte{der}, PrivateKey: key, Leaf: leaf} +} diff --git a/internal/e2e/sandbox_test.go b/internal/e2e/sandbox_test.go index ca78b69f97..862d6a4ce2 100644 --- a/internal/e2e/sandbox_test.go +++ b/internal/e2e/sandbox_test.go @@ -110,17 +110,6 @@ func renderTemplates(t *testing.T, relPath string) []*ateapipb.ActorTemplate { return decodeSubstrateTemplates(t, rendered) } -// memoryLimit returns the template's spec-level memory limit quantity, "" if -// none is declared. -func memoryLimit(tmpl *ateapipb.ActorTemplate) string { - for _, l := range tmpl.GetResources().GetLimits() { - if l.GetName() == "memory" { - return l.GetQuantity() - } - } - return "" -} - // TestRenderSubstrateFixtures_GVisor pins the default rendering: every // micro-VM block is gone, no placeholder survives, and the templates select // their fixture's pool. @@ -183,7 +172,7 @@ func TestRenderSubstrateFixtures_MicroVM(t *testing.T) { for _, tmpl := range templates { name := tmpl.GetMetadata().GetName() // The API rejects a template without limits. - if memoryLimit(tmpl) == "" { + if tmpl.GetResources().GetLimits().GetMemoryBytes() == 0 { t.Errorf("template %s declares no memory limit", name) } location := tmpl.GetSnapshotConfig().GetStorageLocation() diff --git a/internal/e2e/suites/capabilities/capabilities_test.go b/internal/e2e/suites/capabilities/capabilities_test.go index 0709fa87ea..525e734de5 100644 --- a/internal/e2e/suites/capabilities/capabilities_test.go +++ b/internal/e2e/suites/capabilities/capabilities_test.go @@ -83,7 +83,7 @@ func TestActorCapabilities(t *testing.T) { actor := tt.template + "-actor" createAndResumeActor(t, ctx, clients, namespace, tt.template, actor) - rc, err := e2e.NewRouterClient(ctx) + rc, err := e2e.NewRouterClient(t, ctx) if err != nil { t.Fatalf("NewRouterClient: %v", err) } diff --git a/internal/e2e/suites/combinedvolumes/combinedvolumes_test.go b/internal/e2e/suites/combinedvolumes/combinedvolumes_test.go index e26ce7606b..b494549a02 100644 --- a/internal/e2e/suites/combinedvolumes/combinedvolumes_test.go +++ b/internal/e2e/suites/combinedvolumes/combinedvolumes_test.go @@ -64,7 +64,7 @@ const ( extVolume = "external" extPathA = "/mnt/ate-external-a" extPathB = "/mnt/ate-external-b" - extCapacity = "1Gi" + extCapacity = 1 << 30 // probeWrittenContent is the fixed string the probe's /writefile writes. probeWrittenContent = "written by probe" @@ -235,7 +235,7 @@ func createTemplate(ctx context.Context, t *testing.T, clients *e2e.Clients, ns tmpl.Volumes = append(tmpl.Volumes, &ateapipb.Volume{ Name: extVolume, External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{ - Capacity: extCapacity, + CapacityBytes: extCapacity, StorageClassName: storageClass, }}, }) @@ -358,7 +358,7 @@ func TestCombinedVolumes(t *testing.T) { t.Fatalf("ResumeActor: %v", err) } - router, err := e2e.NewRouterClient(ctx) + router, err := e2e.NewRouterClient(t, ctx) if err != nil { t.Fatalf("NewRouterClient: %v", err) } diff --git a/internal/e2e/suites/credentials/credentials_test.go b/internal/e2e/suites/credentials/credentials_test.go index df8ba0fd28..086d8a5b70 100644 --- a/internal/e2e/suites/credentials/credentials_test.go +++ b/internal/e2e/suites/credentials/credentials_test.go @@ -70,7 +70,7 @@ func TestKubernetesCredentialInjection(t *testing.T) { }) // Match the DNS query's fully qualified name in the egress policy. host := "credential-origin." + namespace + ".svc.cluster.local" - router, err := e2e.NewRouterClient(ctx) + router, err := e2e.NewRouterClient(t, ctx) require.NoError(t, err) t.Cleanup(router.Close) // Keep the successful actor alive through the cache-isolation check. diff --git a/internal/e2e/suites/demo/demo_test.go b/internal/e2e/suites/demo/demo_test.go index 8009a27cb9..d3887798c5 100644 --- a/internal/e2e/suites/demo/demo_test.go +++ b/internal/e2e/suites/demo/demo_test.go @@ -27,8 +27,6 @@ import ( "time" "github.com/google/go-cmp/cmp" - "go.ate.dev/substrate/internal/ateclient" - "go.ate.dev/substrate/internal/atenet" "go.ate.dev/substrate/internal/e2e" "go.ate.dev/substrate/internal/hardware" "go.ate.dev/substrate/internal/preview" @@ -39,9 +37,6 @@ import ( "google.golang.org/grpc/status" "google.golang.org/protobuf/testing/protocmp" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" - "k8s.io/apimachinery/pkg/labels" - "k8s.io/client-go/tools/portforward" - "k8s.io/client-go/transport/spdy" ) const demoAtespace = "demo" @@ -1394,7 +1389,7 @@ func createActorTemplateWithExternalVolume(ctx context.Context, t *testing.T, cl at.Volumes = append(at.Volumes, &ateapipb.Volume{ Name: "external-data", External: &ateapipb.ExternalVolumeSource{Template: &ateapipb.ExternalVolumeTemplate{ - Capacity: "1Gi", + CapacityBytes: 1 << 30, StorageClassName: scName, }}, }) @@ -1554,79 +1549,15 @@ func callActorPath(t *testing.T, actorRef resources.ActorRef, method, path strin func callActorPathOnce(t *testing.T, actorRef resources.ActorRef, method, path string) (string, error) { t.Helper() - clients := e2e.GetClients() - - svc, err := clients.K8s.CoreV1().Services(e2e.SystemNamespace()).Get(context.Background(), e2e.ResourceName("atenet-router"), metav1.GetOptions{}) - if err != nil { - return "", fmt.Errorf("failed to get atenet-router service: %w", err) - } - - selector := labels.SelectorFromSet(svc.Spec.Selector).String() - pods, err := clients.K8s.CoreV1().Pods(e2e.SystemNamespace()).List(context.Background(), metav1.ListOptions{LabelSelector: selector}) - if err != nil { - return "", fmt.Errorf("failed to list atenet-router pods: %w", err) - } - if len(pods.Items) == 0 { - return "", fmt.Errorf("no atenet-router pods found") - } - targetPod := pods.Items[0] - - config, err := ateclient.LoadKubeConfig(e2e.KubeConfig, e2e.KubeContext) - if err != nil { - return "", fmt.Errorf("failed to load kubeconfig: %w", err) - } - - reqConfig := clients.K8s.CoreV1().RESTClient().Post(). - Resource("pods"). - Namespace(targetPod.Namespace). - Name(targetPod.Name). - SubResource("portforward") - - transport, upgrader, err := spdy.RoundTripperFor(config) - if err != nil { - return "", fmt.Errorf("failed to create SPDY transport: %w", err) - } - - dialer := spdy.NewDialer(upgrader, &http.Client{Transport: transport}, http.MethodPost, reqConfig.URL()) - - stopCh := make(chan struct{}) - readyCh := make(chan struct{}) - defer close(stopCh) - - fw, err := portforward.New(dialer, []string{"0:8080"}, stopCh, readyCh, io.Discard, io.Discard) - if err != nil { - return "", fmt.Errorf("failed to create port forwarder: %w", err) - } - - errCh := make(chan error, 1) - go func() { - if err := fw.ForwardPorts(); err != nil { - errCh <- err - } - }() - - select { - case <-readyCh: - case err := <-errCh: - return "", fmt.Errorf("port forwarding failed: %w", err) - case <-time.After(10 * time.Second): - return "", fmt.Errorf("timeout waiting for port-forward") - } - - forwardedPorts, err := fw.GetPorts() - if err != nil || len(forwardedPorts) == 0 { - return "", fmt.Errorf("failed to get forwarded ports: %w", err) - } - localPort := forwardedPorts[0].Local - - reqHttp, err := http.NewRequest(method, fmt.Sprintf("http://127.0.0.1:%d%s", localPort, path), nil) + router, err := e2e.NewRouterClient(t, context.Background()) if err != nil { - return "", fmt.Errorf("failed to create request: %w", err) + return "", fmt.Errorf("connecting to atenet-router: %w", err) } - reqHttp.Header.Set(atenet.TargetActorHeader, actorRef.String()) + defer router.Close() - httpClient := &http.Client{Timeout: 15 * time.Second} - resp, err := httpClient.Do(reqHttp) + ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second) + defer cancel() + resp, err := router.Do(ctx, method, actorRef, path, nil) if err != nil { return "", fmt.Errorf("failed to do request: %w", err) } diff --git a/internal/e2e/suites/egresscredinject/egresscredinject_test.go b/internal/e2e/suites/egresscredinject/egresscredinject_test.go index 92a237c7f3..f0d0750471 100644 --- a/internal/e2e/suites/egresscredinject/egresscredinject_test.go +++ b/internal/e2e/suites/egresscredinject/egresscredinject_test.go @@ -136,7 +136,7 @@ func TestActorEgressCredentialInjection(t *testing.T) { t.Fatalf("GetActor %q: %v", id, err) } - rc, err := e2e.NewRouterClient(ctx) + rc, err := e2e.NewRouterClient(t, ctx) if err != nil { t.Fatalf("NewRouterClient: %v", err) } diff --git a/internal/e2e/suites/egressmitm/egressmitm_test.go b/internal/e2e/suites/egressmitm/egressmitm_test.go index 7f69a458ae..014a4c1a88 100644 --- a/internal/e2e/suites/egressmitm/egressmitm_test.go +++ b/internal/e2e/suites/egressmitm/egressmitm_test.go @@ -99,7 +99,7 @@ func TestActorEgressMITMTrust(t *testing.T) { createAndResumeActor(t, ctx, clients, id, mitm, passthrough) waitForActorState(t, ctx, clients, id, ateapipb.ActorState_ACTOR_STATE_RUNNING) - rc, err := e2e.NewRouterClient(ctx) + rc, err := e2e.NewRouterClient(t, ctx) if err != nil { t.Fatalf("NewRouterClient: %v", err) } diff --git a/internal/e2e/suites/identity/identity_test.go b/internal/e2e/suites/identity/identity_test.go index aff525db91..690e8155c0 100644 --- a/internal/e2e/suites/identity/identity_test.go +++ b/internal/e2e/suites/identity/identity_test.go @@ -96,7 +96,7 @@ func TestActorIdentity_AfterRestore_IsOwnID_NotGolden(t *testing.T) { createAndResumeActor(t, ctx, clients, id) } - rc, err := e2e.NewRouterClient(ctx) + rc, err := e2e.NewRouterClient(t, ctx) if err != nil { t.Fatalf("NewRouterClient: %v", err) } diff --git a/internal/e2e/suites/ingressauth/ingressauth_test.go b/internal/e2e/suites/ingressauth/ingressauth_test.go new file mode 100644 index 0000000000..16509ccc4c --- /dev/null +++ b/internal/e2e/suites/ingressauth/ingressauth_test.go @@ -0,0 +1,315 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// Package ingressauth checks atenet-router's static-mtls client +// authentication: who the ingress listeners let through, and who they turn +// away. The other suites cover what happens once a client is let through, +// since in static-mtls mode every one of them reaches its actors as the +// allowlisted e2e router client. +package ingressauth + +import ( + "bufio" + "context" + "crypto/ecdsa" + "crypto/elliptic" + "crypto/rand" + "crypto/tls" + "crypto/x509" + "errors" + "fmt" + "math/big" + "net" + "net/http" + "net/url" + "testing" + "time" + + "go.ate.dev/substrate/internal/ateclient" + "go.ate.dev/substrate/internal/atenet" + "go.ate.dev/substrate/internal/e2e" + "go.ate.dev/substrate/internal/portforward" + "go.ate.dev/substrate/internal/resources" + "k8s.io/client-go/kubernetes" +) + +// unlistedServiceAccount runs in e2e.RouterClientNamespace, so the +// podidentity signer issues it a certificate from the same CA as the +// allowlisted router client, under a SPIFFE ID the router does not list. +const unlistedServiceAccount = "router-client-unlisted" + +// targetActor names an actor that does not exist. A client the router lets +// through gets an HTTP response, whatever its status; one it turns away gets +// none, because the TLS handshake fails first. +var targetActor = resources.ActorRef{Atespace: "ate-e2e-ingressauth", Name: "nobody"} + +// atenet-router's Service ports; see manifests/ate-install/atenet-router.yaml. +const ( + httpServicePort = 80 + httpsServicePort = 443 + connectServicePort = 8081 + connectTLSServicePort = 8444 +) + +// requireStaticMTLS skips the test unless the router was installed with +// --ingress-auth-mode=static-mtls. +// +// Only needed until agent-gateway supports static-mtls mode as well. +func requireStaticMTLS(t *testing.T) { + t.Helper() + if !e2e.IngressAuthModeSet() && e2e.IngressAuthModeRequired() { + t.Fatalf("%s is unset and required (CI or REQUIRE_INGRESS_AUTH_MODE is set): set it to %q or %q to match how atenet-router was installed (ATE_INGRESS_AUTH_MODE)", + e2e.IngressAuthModeEnv, e2e.IngressAuthStaticMTLS, e2e.IngressAuthDeprecatedInsecure) + } + if !e2e.IngressMTLS() { + t.Skipf("atenet-router does not require client certificates (%s=%q); install with --ingress-auth-mode=%s and set %s=%s to run this suite", + e2e.IngressAuthModeEnv, e2e.IngressAuthMode(), e2e.IngressAuthStaticMTLS, e2e.IngressAuthModeEnv, e2e.IngressAuthStaticMTLS) + } +} + +// tlsListener is one of the router's TLS ingress listeners and how to send it +// one request. +type tlsListener struct { + name string + servicePort int32 + // request sends one request for targetActor over a TLS connection + // presenting cert (nil for none), and returns the HTTP status. An error + // means no HTTP response came back. + request func(ctx context.Context, address string, cfg *tls.Config) (int, error) +} + +// tlsListeners are both TLS ingress listeners. Every check runs against both: +// they share a transport socket, but nothing else would notice one of them +// losing it. +var tlsListeners = []tlsListener{ + {name: "https", servicePort: httpsServicePort, request: httpsRequest}, + {name: "connect-tls", servicePort: connectTLSServicePort, request: connectRequest}, +} + +func forwardRouterPort(t *testing.T, ctx context.Context, servicePort int32) string { + t.Helper() + config, err := ateclient.LoadKubeConfig(e2e.KubeConfig, e2e.KubeContext) + if err != nil { + t.Fatalf("loading kubeconfig: %v", err) + } + clientset, err := kubernetes.NewForConfig(config) + if err != nil { + t.Fatalf("creating k8s client: %v", err) + } + localPort, stop, err := portforward.ServicePortForward(ctx, config, clientset, e2e.SystemNamespace(), e2e.ResourceName("atenet-router"), servicePort) + if err != nil { + t.Fatalf("port-forwarding to atenet-router port %d: %v", servicePort, err) + } + t.Cleanup(stop) + return fmt.Sprintf("127.0.0.1:%d", localPort) +} + +func httpsRequest(ctx context.Context, address string, cfg *tls.Config) (int, error) { + transport := http.DefaultTransport.(*http.Transport).Clone() + transport.TLSClientConfig = cfg + transport.DisableKeepAlives = true + client := &http.Client{Transport: transport, Timeout: 30 * time.Second} + req, err := http.NewRequestWithContext(ctx, http.MethodGet, "https://"+address+"/", nil) + if err != nil { + return 0, err + } + req.Header.Set(atenet.TargetActorHeader, targetActor.String()) + resp, err := client.Do(req) + if err != nil { + return 0, err + } + resp.Body.Close() + return resp.StatusCode, nil +} + +func connectRequest(ctx context.Context, address string, cfg *tls.Config) (int, error) { + ctx, cancel := context.WithTimeout(ctx, 30*time.Second) + defer cancel() + conn, err := (&tls.Dialer{Config: cfg}).DialContext(ctx, "tcp", address) + if err != nil { + return 0, err + } + defer conn.Close() + if deadline, ok := ctx.Deadline(); ok { + _ = conn.SetDeadline(deadline) + } + destination := net.JoinHostPort(targetActor.Name, "80") + req := &http.Request{ + Method: http.MethodConnect, + URL: &url.URL{Host: destination}, + Host: destination, + Header: http.Header{atenet.TargetActorHeader: []string{targetActor.String()}}, + } + if err := req.Write(conn); err != nil { + return 0, err + } + resp, err := http.ReadResponse(bufio.NewReader(conn), req) + if err != nil { + return 0, err + } + resp.Body.Close() + return resp.StatusCode, nil +} + +// routerTLSConfig is the TLS config a client presenting cert dials the router +// with. It presents cert unconditionally: Go's client otherwise withholds a +// certificate that does not chain to a CA the server's CertificateRequest +// names, which would turn the untrusted-CA check into a no-certificate one. +func routerTLSConfig(t *testing.T, ctx context.Context, cert *tls.Certificate) *tls.Config { + t.Helper() + cfg, err := e2e.RouterTLSConfig(ctx, nil) + if err != nil { + t.Fatalf("building the router TLS config: %v", err) + } + if cert != nil { + cfg.GetClientCertificate = func(*tls.CertificateRequestInfo) (*tls.Certificate, error) { return cert, nil } + } + return cfg +} + +// isRemoteTLSAlert reports whether err is a TLS alert the peer sent, which +// crypto/tls reports as a *net.OpError with Op "remote error". +func isRemoteTLSAlert(err error) bool { + var opErr *net.OpError + return errors.As(err, &opErr) && opErr.Op == "remote error" +} + +// TestIngressAuthentication checks each kind of client against both TLS +// listeners. +// +// Each refused client gets a port-forward of its own, opened fresh and first +// shown to carry the allowlisted client through: a port-forward tears down its +// whole tunnel once one forwarded connection fails, so a shared one would turn +// every check after the first refusal into a dial error. The refusal itself +// has to be a TLS alert from the router, which a broken tunnel cannot fake. +func TestIngressAuthentication(t *testing.T) { + requireStaticMTLS(t) + ctx := context.Background() + + allowed := e2e.RouterClientCertificate(t, ctx, e2e.RouterClientServiceAccount) + unlisted := e2e.RouterClientCertificate(t, ctx, unlistedServiceAccount) + foreign := foreignCertificate(t, e2e.RouterClientSPIFFEID(e2e.RouterClientServiceAccount)) + + for _, listener := range tlsListeners { + t.Run(listener.name, func(t *testing.T) { + for _, tc := range []struct { + name string + cert *tls.Certificate + }{ + {name: "no client certificate", cert: nil}, + {name: "podidentity certificate for an unlisted SPIFFE ID", cert: &unlisted}, + {name: "allowlisted SPIFFE ID signed by an untrusted CA", cert: &foreign}, + } { + t.Run(tc.name, func(t *testing.T) { + address := forwardRouterPort(t, ctx, listener.servicePort) + + status, err := listener.request(ctx, address, routerTLSConfig(t, ctx, &allowed)) + if err != nil { + t.Fatalf("allowlisted client %s got no response: %v", e2e.RouterClientSPIFFEID(e2e.RouterClientServiceAccount), err) + } + t.Logf("allowlisted client let through (status %d)", status) + + status, err = listener.request(ctx, address, routerTLSConfig(t, ctx, tc.cert)) + if err == nil { + t.Fatalf("router let the client through (status %d); want the TLS handshake refused", status) + } + if !isRemoteTLSAlert(err) { + t.Fatalf("request failed without a TLS alert from the router, so the refusal is unproven: %v", err) + } + t.Logf("refused: %v", err) + }) + } + }) + } +} + +// TestPlaintextListenersDisabled checks that static-mtls leaves nothing +// listening on the plaintext ports, which could not authenticate a client. +// The Service still publishes them; the router container no longer serves +// them, so a request through either gets no response. +func TestPlaintextListenersDisabled(t *testing.T) { + requireStaticMTLS(t) + ctx := context.Background() + + for _, tc := range []struct { + name string + port int32 + }{ + {name: "http", port: httpServicePort}, + {name: "connect", port: connectServicePort}, + } { + t.Run(tc.name, func(t *testing.T) { + address := forwardRouterPort(t, ctx, tc.port) + client := &http.Client{Timeout: 30 * time.Second} + req, err := http.NewRequestWithContext(ctx, http.MethodGet, "http://"+address+"/", nil) + if err != nil { + t.Fatal(err) + } + req.Header.Set(atenet.TargetActorHeader, targetActor.String()) + resp, err := client.Do(req) + if err == nil { + resp.Body.Close() + t.Fatalf("plaintext port %d answered with status %d; want nothing listening", tc.port, resp.StatusCode) + } + t.Logf("no response: %v", err) + }) + } +} + +// foreignCertificate returns a client certificate carrying spiffeID, signed +// by a throwaway CA the router does not trust. +func foreignCertificate(t *testing.T, spiffeID string) tls.Certificate { + t.Helper() + caKey, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + if err != nil { + t.Fatal(err) + } + caTemplate := &x509.Certificate{ + SerialNumber: big.NewInt(1), + NotBefore: time.Now().Add(-time.Minute), + NotAfter: time.Now().Add(time.Hour), + KeyUsage: x509.KeyUsageCertSign, + BasicConstraintsValid: true, + IsCA: true, + } + caDER, err := x509.CreateCertificate(rand.Reader, caTemplate, caTemplate, &caKey.PublicKey, caKey) + if err != nil { + t.Fatal(err) + } + ca, err := x509.ParseCertificate(caDER) + if err != nil { + t.Fatal(err) + } + + key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + if err != nil { + t.Fatal(err) + } + id, err := url.Parse(spiffeID) + if err != nil { + t.Fatal(err) + } + leafDER, err := x509.CreateCertificate(rand.Reader, &x509.Certificate{ + SerialNumber: big.NewInt(2), + NotBefore: time.Now().Add(-time.Minute), + NotAfter: time.Now().Add(time.Hour), + URIs: []*url.URL{id}, + KeyUsage: x509.KeyUsageDigitalSignature, + ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth}, + }, ca, &key.PublicKey, caKey) + if err != nil { + t.Fatal(err) + } + return tls.Certificate{Certificate: [][]byte{leafDER, caDER}, PrivateKey: key} +} diff --git a/internal/e2e/suites/ingressauth/testmain_test.go b/internal/e2e/suites/ingressauth/testmain_test.go new file mode 100644 index 0000000000..275bd09861 --- /dev/null +++ b/internal/e2e/suites/ingressauth/testmain_test.go @@ -0,0 +1,24 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package ingressauth + +import ( + "os" + "testing" + + "go.ate.dev/substrate/internal/e2e" +) + +func TestMain(m *testing.M) { os.Exit(e2e.RunTestMain(m)) } diff --git a/internal/e2e/suites/metrics/metrics_test.go b/internal/e2e/suites/metrics/metrics_test.go index a4b401ebc8..2971911e90 100644 --- a/internal/e2e/suites/metrics/metrics_test.go +++ b/internal/e2e/suites/metrics/metrics_test.go @@ -74,7 +74,7 @@ func TestPlatformMetricsEmitted(t *testing.T) { resume(t, ctx, clients, actorID) // Drive request through the router so the dataplane emits atenet_router_route_duration. - rClient, err := e2e.NewRouterClient(ctx) + rClient, err := e2e.NewRouterClient(t, ctx) if err != nil { t.Fatalf("NewRouterClient: %v", err) } diff --git a/internal/e2e/suites/multiactor/multiactor_test.go b/internal/e2e/suites/multiactor/multiactor_test.go index 5daf2ccda4..c495a3845f 100644 --- a/internal/e2e/suites/multiactor/multiactor_test.go +++ b/internal/e2e/suites/multiactor/multiactor_test.go @@ -67,7 +67,7 @@ func TestTwoActorsShareOneWorker(t *testing.T) { pods[names[0]], pods[names[1]]) } - rc, err := e2e.NewRouterClient(ctx) + rc, err := e2e.NewRouterClient(t, ctx) if err != nil { t.Fatalf("NewRouterClient: %v", err) } diff --git a/internal/e2e/suites/networking/grpcingress_test.go b/internal/e2e/suites/networking/grpcingress_test.go index ce3f1127b5..447f83d38c 100644 --- a/internal/e2e/suites/networking/grpcingress_test.go +++ b/internal/e2e/suites/networking/grpcingress_test.go @@ -25,16 +25,14 @@ import ( "time" "google.golang.org/grpc" + "google.golang.org/grpc/credentials" "google.golang.org/grpc/credentials/insecure" "google.golang.org/grpc/metadata" - "go.ate.dev/substrate/internal/ateclient" "go.ate.dev/substrate/internal/atenet" "go.ate.dev/substrate/internal/e2e" - "go.ate.dev/substrate/internal/portforward" "go.ate.dev/substrate/internal/proto/grpcechopb" "go.ate.dev/substrate/internal/resources" - "k8s.io/client-go/kubernetes" ) // grpcEchoFixtureManifests name the fixture this suite installs to get a @@ -61,10 +59,11 @@ func TestIngressProtocolDowngrade(t *testing.T) { _, actorName, _ := createAndResumeSubstrateActor(t, ctx, "protodowngrade", e2e.SubstrateCounterFixture()) actorRef := resources.ActorRef{Atespace: networkingAtespace, Name: actorName} - base := "http://" + routerAddress(t, ctx) - - h1 := &http.Client{Timeout: 30 * time.Second} - h2c := &http.Client{Transport: h2cTransport(), Timeout: 30 * time.Second} + rc := mustRouterClient(t, ctx) + t.Cleanup(rc.Close) + base := rc.BaseURL() + h1 := &http.Client{Transport: routerTransport(rc, false), Timeout: 30 * time.Second} + h2 := &http.Client{Transport: routerTransport(rc, true), Timeout: 30 * time.Second} request := func(client *http.Client, method, path, contentType string) (*http.Response, error) { req, err := http.NewRequestWithContext(ctx, method, base+path, http.NoBody) @@ -85,9 +84,9 @@ func TestIngressProtocolDowngrade(t *testing.T) { }) t.Run("h2 client reaches h1-only actor", func(t *testing.T) { - resp, err := request(h2c, http.MethodGet, "/readyz", "") + resp, err := request(h2, http.MethodGet, "/readyz", "") if err != nil { - t.Fatalf("h2c request through ingress: %v", err) + t.Fatalf("h2 request through ingress: %v", err) } body, _ := io.ReadAll(resp.Body) resp.Body.Close() @@ -95,15 +94,15 @@ func TestIngressProtocolDowngrade(t *testing.T) { t.Errorf("downstream proto = %s, want HTTP/2.0 (the client really negotiated h2)", resp.Proto) } if resp.StatusCode != http.StatusOK { - t.Fatalf("h2c GET = %d (body %q), want 200: non-gRPC HTTP/2 must be downgraded for HTTP/1.1-only actors", resp.StatusCode, body) + t.Fatalf("h2 GET = %d (body %q), want 200: non-gRPC HTTP/2 must be downgraded for HTTP/1.1-only actors", resp.StatusCode, body) } if !strings.Contains(string(body), "ok") { - t.Errorf("h2c GET body = %q, want the actor's health payload", body) + t.Errorf("h2 GET body = %q, want the actor's health payload", body) } }) t.Run("grpc to non-grpc actor fails loudly", func(t *testing.T) { - resp, err := request(h2c, http.MethodPost, "/count", "application/grpc") + resp, err := request(h2, http.MethodPost, "/count", "application/grpc") if err != nil { t.Fatalf("gRPC-shaped request through ingress: %v", err) } @@ -152,12 +151,16 @@ func TestIngressGRPC(t *testing.T) { atenet.TargetActorHeader, actorRef.String(), ) - // Cleartext h2c to the router's HTTP port. Explicit metadata identifies the - // Actor; the conventional actor authority remains application metadata. The - // h2 ALPN offer is about the *TLS* listener; nothing here needs it. - conn, err := grpc.NewClient(routerAddress(t, ctx), - grpc.WithTransportCredentials(insecure.NewCredentials()), - ) + // Cleartext h2c to the router's HTTP port, or h2 negotiated by ALPN on its + // TLS port when the router requires mTLS. Explicit metadata identifies the + // Actor; the conventional actor authority remains application metadata. + rc := mustRouterClient(t, ctx) + t.Cleanup(rc.Close) + creds := insecure.NewCredentials() + if tlsConfig := rc.TLSConfig(); tlsConfig != nil { + creds = credentials.NewTLS(tlsConfig) + } + conn, err := grpc.NewClient(rc.Address(), grpc.WithTransportCredentials(creds)) if err != nil { t.Fatalf("creating the gRPC client for %s: %v", actorRef, err) } @@ -297,37 +300,31 @@ func waitForGRPCRouteReady(t *testing.T, ctx context.Context, client grpcechopb. } } -// routerAddress port-forwards to atenet-router's HTTP port and returns the -// local host:port, torn down when the test ends. +// routerTransport returns an HTTP transport for rc's router listener that +// speaks HTTP/2 when http2 is set and HTTP/1.1 otherwise. Against the plaintext +// listener HTTP/2 is cleartext h2c by prior knowledge; against the TLS +// listener the router requires in static-mtls mode, it is negotiated by ALPN. // -// e2e.RouterClient is not usable here: it speaks HTTP/1.1 only, and the whole -// point of both tests in this file is to control the protocol the client -// negotiates with the router. -func routerAddress(t *testing.T, ctx context.Context) string { - t.Helper() - config, err := ateclient.LoadKubeConfig(e2e.KubeConfig, e2e.KubeContext) - if err != nil { - t.Fatalf("loading kubeconfig: %v", err) - } - clientset, err := kubernetes.NewForConfig(config) - if err != nil { - t.Fatalf("creating k8s client: %v", err) - } - localPort, stop, err := portforward.ServicePortForward(ctx, config, clientset, e2e.SystemNamespace(), e2e.ResourceName("atenet-router"), 80) - if err != nil { - t.Fatalf("port-forwarding to the router: %v", err) - } - t.Cleanup(stop) - return fmt.Sprintf("127.0.0.1:%d", localPort) -} - -// h2cTransport is an HTTP transport that speaks cleartext HTTP/2 by prior -// knowledge, so a test can reach the router's plain HTTP port as an h2 client -// without any ALPN negotiation. -func h2cTransport() *http.Transport { +// e2e.RouterClient's own client is not usable here: it speaks HTTP/1.1 only, +// and the whole point of both tests in this file is to control the protocol +// the client negotiates with the router. +func routerTransport(rc *e2e.RouterClient, http2 bool) *http.Transport { transport := http.DefaultTransport.(*http.Transport).Clone() protocols := new(http.Protocols) - protocols.SetUnencryptedHTTP2(true) + if tlsConfig := rc.TLSConfig(); tlsConfig != nil { + transport.TLSClientConfig = tlsConfig + if http2 { + protocols.SetHTTP2(true) + } else { + protocols.SetHTTP1(true) + } + } else { + if http2 { + protocols.SetUnencryptedHTTP2(true) + } else { + protocols.SetHTTP1(true) + } + } transport.Protocols = protocols return transport } diff --git a/internal/e2e/suites/networking/networking_test.go b/internal/e2e/suites/networking/networking_test.go index 48ce302d14..574de7c2de 100644 --- a/internal/e2e/suites/networking/networking_test.go +++ b/internal/e2e/suites/networking/networking_test.go @@ -42,6 +42,7 @@ const networkingAtespace = "networking-e2e" // egressFixture returns the unified egress demo deployed by the E2E workflow. func egressFixture() e2e.Fixture { return e2e.EgressFixture() } + func TestActorDirectAccess(t *testing.T) { ctx := context.Background() _, actorName, actor := createAndResumeSubstrateActor(t, ctx, "direct", e2e.SubstrateCounterFixture()) @@ -58,7 +59,7 @@ func TestActorDirectAccess(t *testing.T) { }) t.Logf("Actor access through ingress succeeded; body: %s", body) }) - t.Run("duplicate actor header rejected on plain HTTP", func(t *testing.T) { + t.Run("duplicate actor header rejected", func(t *testing.T) { actorRef := resources.ActorRef{Atespace: networkingAtespace, Name: actorName} for _, secondActor := range []string{actorRef.String(), "untrusted/actor"} { t.Run(secondActor, func(t *testing.T) { @@ -68,14 +69,14 @@ func TestActorDirectAccess(t *testing.T) { } req.Header.Add(atenet.TargetActorHeader, actorRef.String()) req.Header.Add(atenet.TargetActorHeader, secondActor) - resp, err := (&http.Client{Timeout: 10 * time.Second}).Do(req) + resp, err := router.DoReq(req) if err != nil { t.Fatalf("sending request with duplicate actor headers: %v", err) } defer resp.Body.Close() if resp.StatusCode != http.StatusNotFound { body, _ := io.ReadAll(resp.Body) - t.Fatalf("duplicate actor header on plain HTTP returned %d, want 404; body: %s", resp.StatusCode, body) + t.Fatalf("duplicate actor header returned %d, want 404; body: %s", resp.StatusCode, body) } }) } @@ -527,7 +528,7 @@ func createAndResume(t *testing.T, ctx context.Context, prefix string, actor *at func mustRouterClient(t *testing.T, ctx context.Context) *e2e.RouterClient { t.Helper() - router, err := e2e.NewRouterClient(ctx) + router, err := e2e.NewRouterClient(t, ctx) if err != nil { t.Fatalf("NewRouterClient: %v", err) } diff --git a/internal/e2e/suites/networking/websocketingress_test.go b/internal/e2e/suites/networking/websocketingress_test.go index 48dedc97d8..5a497badf5 100644 --- a/internal/e2e/suites/networking/websocketingress_test.go +++ b/internal/e2e/suites/networking/websocketingress_test.go @@ -59,8 +59,10 @@ func TestWebsocketIngressPing(t *testing.T) { rc := mustRouterClient(t, ctx) defer rc.Close() - // Convert http://127.0.0.1:<port> to ws://127.0.0.1:<port>/ws - wsURLStr := strings.Replace(rc.BaseURL(), "http://", "ws://", 1) + "/ws" + // Convert http(s)://127.0.0.1:<port> to ws(s)://127.0.0.1:<port>/ws + wsURLStr := "ws" + strings.TrimPrefix(rc.BaseURL(), "http") + "/ws" + dialer := *websocket.DefaultDialer + dialer.TLSClientConfig = rc.TLSConfig() u, err := url.Parse(wsURLStr) if err != nil { t.Fatalf("parse ws URL: %v", err) @@ -76,7 +78,7 @@ func TestWebsocketIngressPing(t *testing.T) { deadline := time.Now().Add(30 * time.Second) for { var resp *http.Response - c, resp, err = websocket.DefaultDialer.DialContext(ctx, u.String(), header) + c, resp, err = dialer.DialContext(ctx, u.String(), header) if err == nil { break } diff --git a/internal/e2e/suites/networkpolicy/networkpolicy_test.go b/internal/e2e/suites/networkpolicy/networkpolicy_test.go index c357e81cce..b7de552548 100644 --- a/internal/e2e/suites/networkpolicy/networkpolicy_test.go +++ b/internal/e2e/suites/networkpolicy/networkpolicy_test.go @@ -181,7 +181,7 @@ func TestNetworkPolicyDataPlaneEnforcement(t *testing.T) { // === Positive Data Plane Verification (Authorized Ingress) === t.Log("=== Verifying authorized ingress via atenet-router ===") - rc, err := e2e.NewRouterClient(ctx) + rc, err := e2e.NewRouterClient(t, ctx) if err != nil { t.Fatalf("NewRouterClient: %v", err) } diff --git a/internal/e2e/suites/parking/parking_test.go b/internal/e2e/suites/parking/parking_test.go index 2cd23ab812..0eb61b0c1a 100644 --- a/internal/e2e/suites/parking/parking_test.go +++ b/internal/e2e/suites/parking/parking_test.go @@ -58,7 +58,7 @@ func TestRequestParking(t *testing.T) { createActor(ctx, t, clients, at, name) } - router, err := e2e.NewRouterClient(ctx) + router, err := e2e.NewRouterClient(t, ctx) if err != nil { t.Fatalf("creating router client: %v", err) } diff --git a/internal/e2e/suites/security/crossatespace_test.go b/internal/e2e/suites/security/crossatespace_test.go index bca1f30b83..43e467dcc6 100644 --- a/internal/e2e/suites/security/crossatespace_test.go +++ b/internal/e2e/suites/security/crossatespace_test.go @@ -104,7 +104,7 @@ func TestCrossAtespaceIsolation(t *testing.T) { ns := deployAtespaces(t, ctx, clients, env["BUCKET_NAME"]) - rc, err := e2e.NewRouterClient(ctx) + rc, err := e2e.NewRouterClient(t, ctx) if err != nil { t.Fatalf("NewRouterClient: %v", err) } diff --git a/internal/e2e/suites/sizing/sizing_test.go b/internal/e2e/suites/sizing/sizing_test.go index 7cc179b066..fec9b7cc81 100644 --- a/internal/e2e/suites/sizing/sizing_test.go +++ b/internal/e2e/suites/sizing/sizing_test.go @@ -74,7 +74,7 @@ func TestActorSizing_SandboxObservesDeclaredLimits(t *testing.T) { const id = "sized-actor" createAndResumeActor(t, ctx, clients, id) - rc, err := e2e.NewRouterClient(ctx) + rc, err := e2e.NewRouterClient(t, ctx) if err != nil { t.Fatalf("NewRouterClient: %v", err) } diff --git a/internal/ocispec/microvm_test.go b/internal/ocispec/microvm_test.go index 60110eb21b..74bdbb8b08 100644 --- a/internal/ocispec/microvm_test.go +++ b/internal/ocispec/microvm_test.go @@ -142,7 +142,7 @@ func TestShapeMicroVM_KeepsDeclaredContainerLimits(t *testing.T) { const declared = 64 * 1024 * 1024 spec := Build(Options{Container: &ateworkerpb.Container{ContainerSpec: &ateworkerpb.ContainerSpec{ Args: []string{"/app"}, - Resources: &ateworkerpb.ResourceLimits{MemoryBytes: declared}, + Resources: &ateworkerpb.ResourceRequirements{Limits: &ateworkerpb.Resources{MemoryBytes: declared}}, }}}) if err := ShapeMicroVM(spec, MicroVMOptions{ActorDirs: parityActorDirs, ContainerName: "app"}); err != nil { t.Fatalf("ShapeMicroVM() = %v", err) diff --git a/internal/ocispec/ocispec.go b/internal/ocispec/ocispec.go index 2557d43594..14e76e852f 100644 --- a/internal/ocispec/ocispec.go +++ b/internal/ocispec/ocispec.go @@ -60,15 +60,16 @@ const ( // ociResources maps the resolved limits onto the OCI spec's linux.resources. // Returns nil when nothing is set, so the spec carries no linux.resources at // all for a container that declares no limits. -func ociResources(r *ateworkerpb.ResourceLimits) *specs.LinuxResources { - if r == nil || (r.GetMemoryBytes() <= 0 && r.GetCpuMillis() <= 0) { +func ociResources(r *ateworkerpb.ResourceRequirements) *specs.LinuxResources { + l := r.GetLimits() + if l.GetMemoryBytes() <= 0 && l.GetCpuMillicores() <= 0 { return nil } out := &specs.LinuxResources{} - if b := r.GetMemoryBytes(); b > 0 { + if b := l.GetMemoryBytes(); b > 0 { out.Memory = &specs.LinuxMemory{Limit: &b} } - if m := r.GetCpuMillis(); m > 0 { + if m := l.GetCpuMillicores(); m > 0 { quota := max(m*cpuQuotaPeriodUS/1000, cpuQuotaMinUS) period := uint64(cpuQuotaPeriodUS) out.CPU = &specs.LinuxCPU{Quota: "a, Period: &period} diff --git a/internal/ocispec/ocispec_test.go b/internal/ocispec/ocispec_test.go index a4edb2b4a8..1d5438f2ca 100644 --- a/internal/ocispec/ocispec_test.go +++ b/internal/ocispec/ocispec_test.go @@ -184,11 +184,11 @@ func TestOCIResources(t *testing.T) { if got := ociResources(nil); got != nil { t.Errorf("ociResources(nil) = %v, want nil", got) } - if got := ociResources(&ateworkerpb.ResourceLimits{}); got != nil { + if got := ociResources(&ateworkerpb.ResourceRequirements{Limits: &ateworkerpb.Resources{}}); got != nil { t.Errorf("ociResources(zero) = %v, want nil so the spec is unchanged", got) } - got := ociResources(&ateworkerpb.ResourceLimits{MemoryBytes: 268435456, CpuMillis: 200}) + got := ociResources(&ateworkerpb.ResourceRequirements{Limits: &ateworkerpb.Resources{MemoryBytes: 268435456, CpuMillicores: 200}}) if got == nil { t.Fatal("ociResources() = nil, want limits") } @@ -215,7 +215,7 @@ func TestOCIResources(t *testing.T) { // EINVAL at container create. func TestOCIResources_ClampsQuotaToKernelMinimum(t *testing.T) { for _, millis := range []int64{1, 5, 9} { - got := ociResources(&ateworkerpb.ResourceLimits{CpuMillis: millis}) + got := ociResources(&ateworkerpb.ResourceRequirements{Limits: &ateworkerpb.Resources{CpuMillicores: millis}}) if got == nil || got.CPU == nil || got.CPU.Quota == nil { t.Fatalf("cpu=%dm: ociResources() = %v, want a quota", millis, got) } @@ -225,7 +225,7 @@ func TestOCIResources_ClampsQuotaToKernelMinimum(t *testing.T) { } } // At the floor the quota is exact, not clamped. - got := ociResources(&ateworkerpb.ResourceLimits{CpuMillis: 10}) + got := ociResources(&ateworkerpb.ResourceRequirements{Limits: &ateworkerpb.Resources{CpuMillicores: 10}}) if *got.CPU.Quota != cpuQuotaMinUS { t.Errorf("cpu=10m: quota = %d, want exactly %d", *got.CPU.Quota, cpuQuotaMinUS) } @@ -234,7 +234,7 @@ func TestOCIResources_ClampsQuotaToKernelMinimum(t *testing.T) { // A negative limit must not produce a non-nil but empty LinuxResources, which // would put a bare "resources": {} into the spec. func TestOCIResources_NegativeIsUnset(t *testing.T) { - if got := ociResources(&ateworkerpb.ResourceLimits{MemoryBytes: -1, CpuMillis: -1}); got != nil { + if got := ociResources(&ateworkerpb.ResourceRequirements{Limits: &ateworkerpb.Resources{MemoryBytes: -1, CpuMillicores: -1}}); got != nil { t.Errorf("ociResources(negative) = %+v, want nil", got) } } @@ -250,7 +250,7 @@ func TestBuild_NoResourcesLeavesLinuxUntouched(t *testing.T) { func TestBuild_ResourcesApplied(t *testing.T) { spec := Build(Options{Container: &ateworkerpb.Container{ContainerSpec: &ateworkerpb.ContainerSpec{ Args: []string{"/app"}, - Resources: &ateworkerpb.ResourceLimits{MemoryBytes: 67108864}, + Resources: &ateworkerpb.ResourceRequirements{Limits: &ateworkerpb.Resources{MemoryBytes: 67108864}}, }}}) if spec.Linux.Resources == nil || spec.Linux.Resources.Memory == nil { t.Fatalf("Linux.Resources = %v, want a memory limit", spec.Linux.Resources) diff --git a/internal/proto/ateletpb/atelet.pb.go b/internal/proto/ateletpb/atelet.pb.go index 3ae5fa13fe..b174379c53 100644 --- a/internal/proto/ateletpb/atelet.pb.go +++ b/internal/proto/ateletpb/atelet.pb.go @@ -496,17 +496,26 @@ func (x *AttributeEntry) GetValue() string { // WorkerResources and is validated by the same rules. type WorkerResources struct { state protoimpl.MessageState `protogen:"open.v1"` - // Resources are matched by name against actors' resources for placement. + // cpu_millicores is CPU in millicores (1000 = one core). Compared against + // actors' resource limits for placement. // // +k8s:optional - Resources *Resources `protobuf:"bytes,1,opt,name=resources,proto3" json:"resources,omitempty"` + // +k8s:minimum=0 + // +k8s:maximum=999999 # less than 1000 cores + CpuMillicores int64 `protobuf:"varint,1,opt,name=cpu_millicores,json=cpuMillicores,proto3" json:"cpu_millicores,omitempty"` + // memory_bytes is memory in bytes. Compared against actors' resource + // limits for placement. + // + // +k8s:optional + // +k8s:minimum=0 + MemoryBytes int64 `protobuf:"varint,2,opt,name=memory_bytes,json=memoryBytes,proto3" json:"memory_bytes,omitempty"` // A count of Actors. Workers can use this to limit the number of Actors // that can be placed on them, constraining costs not captured in the // resources above. // // +k8s:optional // +k8s:minimum=1 - Actors int32 `protobuf:"varint,2,opt,name=actors,proto3" json:"actors,omitempty"` + Actors int32 `protobuf:"varint,3,opt,name=actors,proto3" json:"actors,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache } @@ -541,11 +550,18 @@ func (*WorkerResources) Descriptor() ([]byte, []int) { return file_atelet_proto_rawDescGZIP(), []int{4} } -func (x *WorkerResources) GetResources() *Resources { +func (x *WorkerResources) GetCpuMillicores() int64 { if x != nil { - return x.Resources + return x.CpuMillicores } - return nil + return 0 +} + +func (x *WorkerResources) GetMemoryBytes() int64 { + if x != nil { + return x.MemoryBytes + } + return 0 } func (x *WorkerResources) GetActors() int32 { @@ -555,35 +571,31 @@ func (x *WorkerResources) GetActors() int32 { return 0 } -type Resources struct { +// ResourceRequirements are the compute resources a workload may use. +type ResourceRequirements struct { state protoimpl.MessageState `protogen:"open.v1"` - // limits is the maximum amount of compute resources allowed. Only "cpu" - // and "memory" are supported, each at most once, and each quantity must be - // greater than zero; the cpu limit must be less than 1000 cores. + // limits is the maximum amount of compute resources allowed. // // +k8s:optional - // +k8s:maxItems=2 - // +k8s:listType=map - // +k8s:listMapKey=name - Limits []*Limits `protobuf:"bytes,1,rep,name=limits,proto3" json:"limits,omitempty"` + Limits *Resources `protobuf:"bytes,1,opt,name=limits,proto3" json:"limits,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache } -func (x *Resources) Reset() { - *x = Resources{} +func (x *ResourceRequirements) Reset() { + *x = ResourceRequirements{} mi := &file_atelet_proto_msgTypes[5] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } -func (x *Resources) String() string { +func (x *ResourceRequirements) String() string { return protoimpl.X.MessageStringOf(x) } -func (*Resources) ProtoMessage() {} +func (*ResourceRequirements) ProtoMessage() {} -func (x *Resources) ProtoReflect() protoreflect.Message { +func (x *ResourceRequirements) ProtoReflect() protoreflect.Message { mi := &file_atelet_proto_msgTypes[5] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) @@ -595,48 +607,50 @@ func (x *Resources) ProtoReflect() protoreflect.Message { return mi.MessageOf(x) } -// Deprecated: Use Resources.ProtoReflect.Descriptor instead. -func (*Resources) Descriptor() ([]byte, []int) { +// Deprecated: Use ResourceRequirements.ProtoReflect.Descriptor instead. +func (*ResourceRequirements) Descriptor() ([]byte, []int) { return file_atelet_proto_rawDescGZIP(), []int{5} } -func (x *Resources) GetLimits() []*Limits { +func (x *ResourceRequirements) GetLimits() *Resources { if x != nil { return x.Limits } return nil } -// +k8s:customValidation # names, quantity parse/positivity, cpu bound -type Limits struct { +// Resources is an amount of each compute resource. +type Resources struct { state protoimpl.MessageState `protogen:"open.v1"` - // +k8s:required - // +k8s:maxLength=16 # the Limits hook restricts values to cpu/memory - Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"` - // quantity is in Kubernetes resource.Quantity string form (e.g. "500m", - // "2Gi"). + // cpu_millicores is CPU in millicores (1000 = one core). 0 means none. // - // +k8s:required - // +k8s:maxLength=32 # the Limits hook requires a parseable quantity - Quantity string `protobuf:"bytes,2,opt,name=quantity,proto3" json:"quantity,omitempty"` + // +k8s:optional + // +k8s:minimum=0 + // +k8s:maximum=999999 # less than 1000 cores + CpuMillicores int64 `protobuf:"varint,1,opt,name=cpu_millicores,json=cpuMillicores,proto3" json:"cpu_millicores,omitempty"` + // memory_bytes is memory in bytes. 0 means none. + // + // +k8s:optional + // +k8s:minimum=0 + MemoryBytes int64 `protobuf:"varint,2,opt,name=memory_bytes,json=memoryBytes,proto3" json:"memory_bytes,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache } -func (x *Limits) Reset() { - *x = Limits{} +func (x *Resources) Reset() { + *x = Resources{} mi := &file_atelet_proto_msgTypes[6] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } -func (x *Limits) String() string { +func (x *Resources) String() string { return protoimpl.X.MessageStringOf(x) } -func (*Limits) ProtoMessage() {} +func (*Resources) ProtoMessage() {} -func (x *Limits) ProtoReflect() protoreflect.Message { +func (x *Resources) ProtoReflect() protoreflect.Message { mi := &file_atelet_proto_msgTypes[6] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) @@ -648,23 +662,23 @@ func (x *Limits) ProtoReflect() protoreflect.Message { return mi.MessageOf(x) } -// Deprecated: Use Limits.ProtoReflect.Descriptor instead. -func (*Limits) Descriptor() ([]byte, []int) { +// Deprecated: Use Resources.ProtoReflect.Descriptor instead. +func (*Resources) Descriptor() ([]byte, []int) { return file_atelet_proto_rawDescGZIP(), []int{6} } -func (x *Limits) GetName() string { +func (x *Resources) GetCpuMillicores() int64 { if x != nil { - return x.Name + return x.CpuMillicores } - return "" + return 0 } -func (x *Limits) GetQuantity() string { +func (x *Resources) GetMemoryBytes() int64 { if x != nil { - return x.Quantity + return x.MemoryBytes } - return "" + return 0 } type RegisterWorkerResponse struct { @@ -1607,15 +1621,10 @@ type RunRequest struct { EgressGateway *EgressGateway `protobuf:"bytes,9,opt,name=egress_gateway,json=egressGateway,proto3,oneof" json:"egress_gateway,omitempty"` // The actor's declared size, from the ActorTemplate's resource limits. atelet // passes these through to the sandbox so it is sized to the actor (not the - // whole host or worker pod). Zero means "unset": keep the runtime default. + // whole host or worker pod). A zero limit keeps the runtime default. // // +k8s:optional - // +k8s:minimum=0 - // +k8s:maximum=999999 # the control plane caps cpu limits strictly below 1000 cores - CpuMilli int64 `protobuf:"varint,10,opt,name=cpu_milli,json=cpuMilli,proto3" json:"cpu_milli,omitempty"` // CPU limit in millicores (1000 = one core). - // +k8s:optional - // +k8s:minimum=0 - MemoryBytes int64 `protobuf:"varint,11,opt,name=memory_bytes,json=memoryBytes,proto3" json:"memory_bytes,omitempty"` // Memory limit in bytes. + Resources *ResourceRequirements `protobuf:"bytes,10,opt,name=resources,proto3" json:"resources,omitempty"` // The runtime the assigned worker runs this actor under, from // WorkerAssignment.sandbox_runtime. atelet resolves the pause image and // sandbox binaries through its advertised-runtime index and records the @@ -1713,18 +1722,11 @@ func (x *RunRequest) GetEgressGateway() *EgressGateway { return nil } -func (x *RunRequest) GetCpuMilli() int64 { +func (x *RunRequest) GetResources() *ResourceRequirements { if x != nil { - return x.CpuMilli - } - return 0 -} - -func (x *RunRequest) GetMemoryBytes() int64 { - if x != nil { - return x.MemoryBytes + return x.Resources } - return 0 + return nil } func (x *RunRequest) GetSandboxRuntime() *SandboxRuntime { @@ -2502,11 +2504,11 @@ type Container struct { VolumeMounts []*VolumeMount `protobuf:"bytes,6,rep,name=volume_mounts,json=volumeMounts,proto3" json:"volume_mounts,omitempty"` // +k8s:optional SecurityContext *SecurityContext `protobuf:"bytes,8,opt,name=security_context,json=securityContext,proto3" json:"security_context,omitempty"` - // resources are the cgroup limits for this container, resolved by - // ate-api-server from the ActorTemplate. Unset means no limits. + // resources are the cgroup limits for this container, from the + // ActorTemplate. Unset means no limits. // // +k8s:optional - Resources *ResourceLimits `protobuf:"bytes,9,opt,name=resources,proto3" json:"resources,omitempty"` + Resources *ResourceRequirements `protobuf:"bytes,9,opt,name=resources,proto3" json:"resources,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache } @@ -2597,7 +2599,7 @@ func (x *Container) GetSecurityContext() *SecurityContext { return nil } -func (x *Container) GetResources() *ResourceLimits { +func (x *Container) GetResources() *ResourceRequirements { if x != nil { return x.Resources } @@ -2712,69 +2714,6 @@ func (x *Capabilities) GetDrop() []string { return nil } -// ResourceLimits are the cgroup limits for one container. Scalars rather than a -// map so each resource.Quantity is parsed exactly once, in ate-api-server. -type ResourceLimits struct { - state protoimpl.MessageState `protogen:"open.v1"` - // memory_bytes is the memory limit in bytes. 0 means unset. - // - // +k8s:optional - // +k8s:minimum=0 - MemoryBytes int64 `protobuf:"varint,1,opt,name=memory_bytes,json=memoryBytes,proto3" json:"memory_bytes,omitempty"` - // cpu_millis is the CPU limit in milli-cores (1000 = one core). 0 means unset. - // - // +k8s:optional - // +k8s:minimum=0 - // +k8s:maximum=999999 # the control plane caps cpu limits strictly below 1000 cores - CpuMillis int64 `protobuf:"varint,2,opt,name=cpu_millis,json=cpuMillis,proto3" json:"cpu_millis,omitempty"` - unknownFields protoimpl.UnknownFields - sizeCache protoimpl.SizeCache -} - -func (x *ResourceLimits) Reset() { - *x = ResourceLimits{} - mi := &file_atelet_proto_msgTypes[39] - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - ms.StoreMessageInfo(mi) -} - -func (x *ResourceLimits) String() string { - return protoimpl.X.MessageStringOf(x) -} - -func (*ResourceLimits) ProtoMessage() {} - -func (x *ResourceLimits) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[39] - if x != nil { - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - if ms.LoadMessageInfo() == nil { - ms.StoreMessageInfo(mi) - } - return ms - } - return mi.MessageOf(x) -} - -// Deprecated: Use ResourceLimits.ProtoReflect.Descriptor instead. -func (*ResourceLimits) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{39} -} - -func (x *ResourceLimits) GetMemoryBytes() int64 { - if x != nil { - return x.MemoryBytes - } - return 0 -} - -func (x *ResourceLimits) GetCpuMillis() int64 { - if x != nil { - return x.CpuMillis - } - return 0 -} - type EnvVar struct { state protoimpl.MessageState `protogen:"open.v1"` // name may be any printable ASCII character except '='. @@ -2792,7 +2731,7 @@ type EnvVar struct { func (x *EnvVar) Reset() { *x = EnvVar{} - mi := &file_atelet_proto_msgTypes[40] + mi := &file_atelet_proto_msgTypes[39] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -2804,7 +2743,7 @@ func (x *EnvVar) String() string { func (*EnvVar) ProtoMessage() {} func (x *EnvVar) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[40] + mi := &file_atelet_proto_msgTypes[39] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -2817,7 +2756,7 @@ func (x *EnvVar) ProtoReflect() protoreflect.Message { // Deprecated: Use EnvVar.ProtoReflect.Descriptor instead. func (*EnvVar) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{40} + return file_atelet_proto_rawDescGZIP(), []int{39} } func (x *EnvVar) GetName() string { @@ -2852,7 +2791,7 @@ type WakeupProbe struct { func (x *WakeupProbe) Reset() { *x = WakeupProbe{} - mi := &file_atelet_proto_msgTypes[41] + mi := &file_atelet_proto_msgTypes[40] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -2864,7 +2803,7 @@ func (x *WakeupProbe) String() string { func (*WakeupProbe) ProtoMessage() {} func (x *WakeupProbe) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[41] + mi := &file_atelet_proto_msgTypes[40] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -2877,7 +2816,7 @@ func (x *WakeupProbe) ProtoReflect() protoreflect.Message { // Deprecated: Use WakeupProbe.ProtoReflect.Descriptor instead. func (*WakeupProbe) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{41} + return file_atelet_proto_rawDescGZIP(), []int{40} } func (x *WakeupProbe) GetHttpGet() *HttpGetAction { @@ -2915,7 +2854,7 @@ type HttpGetAction struct { func (x *HttpGetAction) Reset() { *x = HttpGetAction{} - mi := &file_atelet_proto_msgTypes[42] + mi := &file_atelet_proto_msgTypes[41] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -2927,7 +2866,7 @@ func (x *HttpGetAction) String() string { func (*HttpGetAction) ProtoMessage() {} func (x *HttpGetAction) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[42] + mi := &file_atelet_proto_msgTypes[41] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -2940,7 +2879,7 @@ func (x *HttpGetAction) ProtoReflect() protoreflect.Message { // Deprecated: Use HttpGetAction.ProtoReflect.Descriptor instead. func (*HttpGetAction) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{42} + return file_atelet_proto_rawDescGZIP(), []int{41} } func (x *HttpGetAction) GetPath() string { @@ -2965,7 +2904,7 @@ type RunResponse struct { func (x *RunResponse) Reset() { *x = RunResponse{} - mi := &file_atelet_proto_msgTypes[43] + mi := &file_atelet_proto_msgTypes[42] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -2977,7 +2916,7 @@ func (x *RunResponse) String() string { func (*RunResponse) ProtoMessage() {} func (x *RunResponse) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[43] + mi := &file_atelet_proto_msgTypes[42] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -2990,7 +2929,7 @@ func (x *RunResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use RunResponse.ProtoReflect.Descriptor instead. func (*RunResponse) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{43} + return file_atelet_proto_rawDescGZIP(), []int{42} } type ObjectStorage struct { @@ -3008,7 +2947,7 @@ type ObjectStorage struct { func (x *ObjectStorage) Reset() { *x = ObjectStorage{} - mi := &file_atelet_proto_msgTypes[44] + mi := &file_atelet_proto_msgTypes[43] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -3020,7 +2959,7 @@ func (x *ObjectStorage) String() string { func (*ObjectStorage) ProtoMessage() {} func (x *ObjectStorage) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[44] + mi := &file_atelet_proto_msgTypes[43] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -3033,7 +2972,7 @@ func (x *ObjectStorage) ProtoReflect() protoreflect.Message { // Deprecated: Use ObjectStorage.ProtoReflect.Descriptor instead. func (*ObjectStorage) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{44} + return file_atelet_proto_rawDescGZIP(), []int{43} } func (x *ObjectStorage) GetSnapshotUri() string { @@ -3087,7 +3026,7 @@ type Snapshot struct { func (x *Snapshot) Reset() { *x = Snapshot{} - mi := &file_atelet_proto_msgTypes[45] + mi := &file_atelet_proto_msgTypes[44] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -3099,7 +3038,7 @@ func (x *Snapshot) String() string { func (*Snapshot) ProtoMessage() {} func (x *Snapshot) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[45] + mi := &file_atelet_proto_msgTypes[44] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -3112,7 +3051,7 @@ func (x *Snapshot) ProtoReflect() protoreflect.Message { // Deprecated: Use Snapshot.ProtoReflect.Descriptor instead. func (*Snapshot) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{45} + return file_atelet_proto_rawDescGZIP(), []int{44} } func (x *Snapshot) GetSnapshotUuid() string { @@ -3159,7 +3098,7 @@ type SnapshotReport struct { func (x *SnapshotReport) Reset() { *x = SnapshotReport{} - mi := &file_atelet_proto_msgTypes[46] + mi := &file_atelet_proto_msgTypes[45] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -3171,7 +3110,7 @@ func (x *SnapshotReport) String() string { func (*SnapshotReport) ProtoMessage() {} func (x *SnapshotReport) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[46] + mi := &file_atelet_proto_msgTypes[45] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -3184,7 +3123,7 @@ func (x *SnapshotReport) ProtoReflect() protoreflect.Message { // Deprecated: Use SnapshotReport.ProtoReflect.Descriptor instead. func (*SnapshotReport) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{46} + return file_atelet_proto_rawDescGZIP(), []int{45} } func (x *SnapshotReport) GetSnapshot() *Snapshot { @@ -3240,7 +3179,7 @@ type ActorSnapshotState struct { func (x *ActorSnapshotState) Reset() { *x = ActorSnapshotState{} - mi := &file_atelet_proto_msgTypes[47] + mi := &file_atelet_proto_msgTypes[46] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -3252,7 +3191,7 @@ func (x *ActorSnapshotState) String() string { func (*ActorSnapshotState) ProtoMessage() {} func (x *ActorSnapshotState) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[47] + mi := &file_atelet_proto_msgTypes[46] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -3265,7 +3204,7 @@ func (x *ActorSnapshotState) ProtoReflect() protoreflect.Message { // Deprecated: Use ActorSnapshotState.ProtoReflect.Descriptor instead. func (*ActorSnapshotState) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{47} + return file_atelet_proto_rawDescGZIP(), []int{46} } func (x *ActorSnapshotState) GetLatestLocal() *SnapshotReport { @@ -3309,7 +3248,7 @@ type UploadConfig struct { func (x *UploadConfig) Reset() { *x = UploadConfig{} - mi := &file_atelet_proto_msgTypes[48] + mi := &file_atelet_proto_msgTypes[47] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -3321,7 +3260,7 @@ func (x *UploadConfig) String() string { func (*UploadConfig) ProtoMessage() {} func (x *UploadConfig) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[48] + mi := &file_atelet_proto_msgTypes[47] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -3334,7 +3273,7 @@ func (x *UploadConfig) ProtoReflect() protoreflect.Message { // Deprecated: Use UploadConfig.ProtoReflect.Descriptor instead. func (*UploadConfig) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{48} + return file_atelet_proto_rawDescGZIP(), []int{47} } func (x *UploadConfig) GetMode() UploadMode { @@ -3396,7 +3335,7 @@ type CheckpointRequest struct { func (x *CheckpointRequest) Reset() { *x = CheckpointRequest{} - mi := &file_atelet_proto_msgTypes[49] + mi := &file_atelet_proto_msgTypes[48] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -3408,7 +3347,7 @@ func (x *CheckpointRequest) String() string { func (*CheckpointRequest) ProtoMessage() {} func (x *CheckpointRequest) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[49] + mi := &file_atelet_proto_msgTypes[48] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -3421,7 +3360,7 @@ func (x *CheckpointRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use CheckpointRequest.ProtoReflect.Descriptor instead. func (*CheckpointRequest) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{49} + return file_atelet_proto_rawDescGZIP(), []int{48} } func (x *CheckpointRequest) GetWorkerPodUid() string { @@ -3505,7 +3444,7 @@ type CheckpointResponse struct { func (x *CheckpointResponse) Reset() { *x = CheckpointResponse{} - mi := &file_atelet_proto_msgTypes[50] + mi := &file_atelet_proto_msgTypes[49] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -3517,7 +3456,7 @@ func (x *CheckpointResponse) String() string { func (*CheckpointResponse) ProtoMessage() {} func (x *CheckpointResponse) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[50] + mi := &file_atelet_proto_msgTypes[49] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -3530,7 +3469,7 @@ func (x *CheckpointResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use CheckpointResponse.ProtoReflect.Descriptor instead. func (*CheckpointResponse) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{50} + return file_atelet_proto_rawDescGZIP(), []int{49} } func (x *CheckpointResponse) GetState() *ActorSnapshotState { @@ -3588,12 +3527,7 @@ type RestoreRequest struct { // means "unset": keep the runtime default. // // +k8s:optional - // +k8s:minimum=0 - // +k8s:maximum=999999 # the control plane caps cpu limits strictly below 1000 cores - CpuMilli int64 `protobuf:"varint,10,opt,name=cpu_milli,json=cpuMilli,proto3" json:"cpu_milli,omitempty"` // CPU limit in millicores (1000 = one core). - // +k8s:optional - // +k8s:minimum=0 - MemoryBytes int64 `protobuf:"varint,11,opt,name=memory_bytes,json=memoryBytes,proto3" json:"memory_bytes,omitempty"` // Memory limit in bytes. + Resources *ResourceRequirements `protobuf:"bytes,10,opt,name=resources,proto3" json:"resources,omitempty"` // The runtime the assigned worker restores this actor under, from // WorkerAssignment.sandbox_runtime. atelet resolves the pause image and // sandbox binaries through its advertised-runtime index and records the @@ -3608,7 +3542,7 @@ type RestoreRequest struct { func (x *RestoreRequest) Reset() { *x = RestoreRequest{} - mi := &file_atelet_proto_msgTypes[51] + mi := &file_atelet_proto_msgTypes[50] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -3620,7 +3554,7 @@ func (x *RestoreRequest) String() string { func (*RestoreRequest) ProtoMessage() {} func (x *RestoreRequest) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[51] + mi := &file_atelet_proto_msgTypes[50] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -3633,7 +3567,7 @@ func (x *RestoreRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use RestoreRequest.ProtoReflect.Descriptor instead. func (*RestoreRequest) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{51} + return file_atelet_proto_rawDescGZIP(), []int{50} } func (x *RestoreRequest) GetWorkerPodUid() string { @@ -3699,18 +3633,11 @@ func (x *RestoreRequest) GetEgressGateway() *EgressGateway { return nil } -func (x *RestoreRequest) GetCpuMilli() int64 { - if x != nil { - return x.CpuMilli - } - return 0 -} - -func (x *RestoreRequest) GetMemoryBytes() int64 { +func (x *RestoreRequest) GetResources() *ResourceRequirements { if x != nil { - return x.MemoryBytes + return x.Resources } - return 0 + return nil } func (x *RestoreRequest) GetSandboxRuntime() *SandboxRuntime { @@ -3728,7 +3655,7 @@ type RestoreResponse struct { func (x *RestoreResponse) Reset() { *x = RestoreResponse{} - mi := &file_atelet_proto_msgTypes[52] + mi := &file_atelet_proto_msgTypes[51] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -3740,7 +3667,7 @@ func (x *RestoreResponse) String() string { func (*RestoreResponse) ProtoMessage() {} func (x *RestoreResponse) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[52] + mi := &file_atelet_proto_msgTypes[51] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -3753,7 +3680,7 @@ func (x *RestoreResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use RestoreResponse.ProtoReflect.Descriptor instead. func (*RestoreResponse) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{52} + return file_atelet_proto_rawDescGZIP(), []int{51} } type PromoteSnapshotRequest struct { @@ -3786,7 +3713,7 @@ type PromoteSnapshotRequest struct { func (x *PromoteSnapshotRequest) Reset() { *x = PromoteSnapshotRequest{} - mi := &file_atelet_proto_msgTypes[53] + mi := &file_atelet_proto_msgTypes[52] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -3798,7 +3725,7 @@ func (x *PromoteSnapshotRequest) String() string { func (*PromoteSnapshotRequest) ProtoMessage() {} func (x *PromoteSnapshotRequest) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[53] + mi := &file_atelet_proto_msgTypes[52] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -3811,7 +3738,7 @@ func (x *PromoteSnapshotRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use PromoteSnapshotRequest.ProtoReflect.Descriptor instead. func (*PromoteSnapshotRequest) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{53} + return file_atelet_proto_rawDescGZIP(), []int{52} } func (x *PromoteSnapshotRequest) GetAtespace() string { @@ -3867,7 +3794,7 @@ type PromoteSnapshotResponse struct { func (x *PromoteSnapshotResponse) Reset() { *x = PromoteSnapshotResponse{} - mi := &file_atelet_proto_msgTypes[54] + mi := &file_atelet_proto_msgTypes[53] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -3879,7 +3806,7 @@ func (x *PromoteSnapshotResponse) String() string { func (*PromoteSnapshotResponse) ProtoMessage() {} func (x *PromoteSnapshotResponse) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[54] + mi := &file_atelet_proto_msgTypes[53] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -3892,7 +3819,7 @@ func (x *PromoteSnapshotResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use PromoteSnapshotResponse.ProtoReflect.Descriptor instead. func (*PromoteSnapshotResponse) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{54} + return file_atelet_proto_rawDescGZIP(), []int{53} } func (x *PromoteSnapshotResponse) GetState() *ActorSnapshotState { @@ -3921,7 +3848,7 @@ type DetachActorRequest struct { func (x *DetachActorRequest) Reset() { *x = DetachActorRequest{} - mi := &file_atelet_proto_msgTypes[55] + mi := &file_atelet_proto_msgTypes[54] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -3933,7 +3860,7 @@ func (x *DetachActorRequest) String() string { func (*DetachActorRequest) ProtoMessage() {} func (x *DetachActorRequest) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[55] + mi := &file_atelet_proto_msgTypes[54] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -3946,7 +3873,7 @@ func (x *DetachActorRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use DetachActorRequest.ProtoReflect.Descriptor instead. func (*DetachActorRequest) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{55} + return file_atelet_proto_rawDescGZIP(), []int{54} } func (x *DetachActorRequest) GetAtespace() string { @@ -3981,7 +3908,7 @@ type DetachActorResponse struct { func (x *DetachActorResponse) Reset() { *x = DetachActorResponse{} - mi := &file_atelet_proto_msgTypes[56] + mi := &file_atelet_proto_msgTypes[55] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -3993,7 +3920,7 @@ func (x *DetachActorResponse) String() string { func (*DetachActorResponse) ProtoMessage() {} func (x *DetachActorResponse) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[56] + mi := &file_atelet_proto_msgTypes[55] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -4006,7 +3933,7 @@ func (x *DetachActorResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use DetachActorResponse.ProtoReflect.Descriptor instead. func (*DetachActorResponse) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{56} + return file_atelet_proto_rawDescGZIP(), []int{55} } func (x *DetachActorResponse) GetState() *ActorSnapshotState { @@ -4035,15 +3962,16 @@ const file_atelet_proto_rawDesc = "" + "attributes\"8\n" + "\x0eAttributeEntry\x12\x10\n" + "\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" + - "\x05value\x18\x02 \x01(\tR\x05value\"Z\n" + - "\x0fWorkerResources\x12/\n" + - "\tresources\x18\x01 \x01(\v2\x11.atelet.ResourcesR\tresources\x12\x16\n" + - "\x06actors\x18\x02 \x01(\x05R\x06actors\"3\n" + - "\tResources\x12&\n" + - "\x06limits\x18\x01 \x03(\v2\x0e.atelet.LimitsR\x06limits\"8\n" + - "\x06Limits\x12\x12\n" + - "\x04name\x18\x01 \x01(\tR\x04name\x12\x1a\n" + - "\bquantity\x18\x02 \x01(\tR\bquantity\"\x18\n" + + "\x05value\x18\x02 \x01(\tR\x05value\"s\n" + + "\x0fWorkerResources\x12%\n" + + "\x0ecpu_millicores\x18\x01 \x01(\x03R\rcpuMillicores\x12!\n" + + "\fmemory_bytes\x18\x02 \x01(\x03R\vmemoryBytes\x12\x16\n" + + "\x06actors\x18\x03 \x01(\x05R\x06actors\"A\n" + + "\x14ResourceRequirements\x12)\n" + + "\x06limits\x18\x01 \x01(\v2\x11.atelet.ResourcesR\x06limits\"U\n" + + "\tResources\x12%\n" + + "\x0ecpu_millicores\x18\x01 \x01(\x03R\rcpuMillicores\x12!\n" + + "\fmemory_bytes\x18\x02 \x01(\x03R\vmemoryBytes\"\x18\n" + "\x16RegisterWorkerResponse\"\x7f\n" + "\x1aRequestActorSuspendRequest\x12%\n" + "\x0eactor_atespace\x18\x01 \x01(\tR\ractorAtespace\x12\x1d\n" + @@ -4095,7 +4023,7 @@ const file_atelet_proto_rawDesc = "" + "\x17actor_template_atespace\x18\x05 \x01(\tR\x15actorTemplateAtespace\x12.\n" + "\x13actor_template_name\x18\x06 \x01(\tR\x11actorTemplateName\x12(\n" + "\x04spec\x18\a \x01(\v2\x14.atelet.WorkloadSpecR\x04spec\"\x13\n" + - "\x11TerminateResponse\"\xf3\x03\n" + + "\x11TerminateResponse\"\xef\x03\n" + "\n" + "RunRequest\x12$\n" + "\x0eworker_pod_uid\x18\x01 \x01(\tR\fworkerPodUid\x12\x1a\n" + @@ -4106,10 +4034,9 @@ const file_atelet_proto_rawDesc = "" + "\x17actor_template_atespace\x18\x05 \x01(\tR\x15actorTemplateAtespace\x12.\n" + "\x13actor_template_name\x18\x06 \x01(\tR\x11actorTemplateName\x12(\n" + "\x04spec\x18\a \x01(\v2\x14.atelet.WorkloadSpecR\x04spec\x12A\n" + - "\x0eegress_gateway\x18\t \x01(\v2\x15.atelet.EgressGatewayH\x00R\regressGateway\x88\x01\x01\x12\x1b\n" + - "\tcpu_milli\x18\n" + - " \x01(\x03R\bcpuMilli\x12!\n" + - "\fmemory_bytes\x18\v \x01(\x03R\vmemoryBytes\x12?\n" + + "\x0eegress_gateway\x18\t \x01(\v2\x15.atelet.EgressGatewayH\x00R\regressGateway\x88\x01\x01\x12:\n" + + "\tresources\x18\n" + + " \x01(\v2\x1c.atelet.ResourceRequirementsR\tresources\x12?\n" + "\x0fsandbox_runtime\x18\f \x01(\v2\x16.atelet.SandboxRuntimeR\x0esandboxRuntimeB\x11\n" + "\x0f_egress_gateway\")\n" + "\rEgressGateway\x12\x18\n" + @@ -4158,7 +4085,7 @@ const file_atelet_proto_rawDesc = "" + "\vVolumeMount\x12\x12\n" + "\x04name\x18\x01 \x01(\tR\x04name\x12\x1d\n" + "\n" + - "mount_path\x18\x02 \x01(\tR\tmountPath\"\xf1\x02\n" + + "mount_path\x18\x02 \x01(\tR\tmountPath\"\xf7\x02\n" + "\tContainer\x12\x12\n" + "\x04name\x18\x01 \x01(\tR\x04name\x12\x14\n" + "\x05image\x18\x02 \x01(\tR\x05image\x12\x18\n" + @@ -4167,17 +4094,13 @@ const file_atelet_proto_rawDesc = "" + "\x03env\x18\x04 \x03(\v2\x0e.atelet.EnvVarR\x03env\x126\n" + "\fwakeup_probe\x18\x05 \x01(\v2\x13.atelet.WakeupProbeR\vwakeupProbe\x128\n" + "\rvolume_mounts\x18\x06 \x03(\v2\x13.atelet.VolumeMountR\fvolumeMounts\x12B\n" + - "\x10security_context\x18\b \x01(\v2\x17.atelet.SecurityContextR\x0fsecurityContext\x124\n" + - "\tresources\x18\t \x01(\v2\x16.atelet.ResourceLimitsR\tresources\"K\n" + + "\x10security_context\x18\b \x01(\v2\x17.atelet.SecurityContextR\x0fsecurityContext\x12:\n" + + "\tresources\x18\t \x01(\v2\x1c.atelet.ResourceRequirementsR\tresources\"K\n" + "\x0fSecurityContext\x128\n" + "\fcapabilities\x18\x01 \x01(\v2\x14.atelet.CapabilitiesR\fcapabilities\"4\n" + "\fCapabilities\x12\x10\n" + "\x03add\x18\x01 \x03(\tR\x03add\x12\x12\n" + - "\x04drop\x18\x02 \x03(\tR\x04drop\"R\n" + - "\x0eResourceLimits\x12!\n" + - "\fmemory_bytes\x18\x01 \x01(\x03R\vmemoryBytes\x12\x1d\n" + - "\n" + - "cpu_millis\x18\x02 \x01(\x03R\tcpuMillis\"7\n" + + "\x04drop\x18\x02 \x03(\tR\x04drop\"7\n" + "\x06EnvVar\x12\x12\n" + "\x04name\x18\x01 \x01(\tR\x04name\x12\x19\n" + "\x05value\x18\x02 \x01(\tB\x03\x80\x01\x01R\x05value\"h\n" + @@ -4220,7 +4143,7 @@ const file_atelet_proto_rawDesc = "" + "\bsnapshot\x18\b \x01(\v2\x10.atelet.SnapshotR\bsnapshot\x129\n" + "\rupload_config\x18\t \x01(\v2\x14.atelet.UploadConfigR\fuploadConfig\"F\n" + "\x12CheckpointResponse\x120\n" + - "\x05state\x18\x01 \x01(\v2\x1a.atelet.ActorSnapshotStateR\x05state\"\xa5\x04\n" + + "\x05state\x18\x01 \x01(\v2\x1a.atelet.ActorSnapshotStateR\x05state\"\xa1\x04\n" + "\x0eRestoreRequest\x12$\n" + "\x0eworker_pod_uid\x18\x01 \x01(\tR\fworkerPodUid\x12\x1a\n" + "\batespace\x18\x02 \x01(\tR\batespace\x12\x1d\n" + @@ -4231,10 +4154,9 @@ const file_atelet_proto_rawDesc = "" + "\x13actor_template_name\x18\x06 \x01(\tR\x11actorTemplateName\x12(\n" + "\x04spec\x18\a \x01(\v2\x14.atelet.WorkloadSpecR\x04spec\x12,\n" + "\bsnapshot\x18\b \x01(\v2\x10.atelet.SnapshotR\bsnapshot\x12A\n" + - "\x0eegress_gateway\x18\t \x01(\v2\x15.atelet.EgressGatewayH\x00R\regressGateway\x88\x01\x01\x12\x1b\n" + - "\tcpu_milli\x18\n" + - " \x01(\x03R\bcpuMilli\x12!\n" + - "\fmemory_bytes\x18\v \x01(\x03R\vmemoryBytes\x12?\n" + + "\x0eegress_gateway\x18\t \x01(\v2\x15.atelet.EgressGatewayH\x00R\regressGateway\x88\x01\x01\x12:\n" + + "\tresources\x18\n" + + " \x01(\v2\x1c.atelet.ResourceRequirementsR\tresources\x12?\n" + "\x0fsandbox_runtime\x18\f \x01(\v2\x16.atelet.SandboxRuntimeR\x0esandboxRuntimeB\x11\n" + "\x0f_egress_gateway\"\x11\n" + "\x0fRestoreResponse\"\x86\x02\n" + @@ -4298,7 +4220,7 @@ func file_atelet_proto_rawDescGZIP() []byte { } var file_atelet_proto_enumTypes = make([]protoimpl.EnumInfo, 3) -var file_atelet_proto_msgTypes = make([]protoimpl.MessageInfo, 59) +var file_atelet_proto_msgTypes = make([]protoimpl.MessageInfo, 58) var file_atelet_proto_goTypes = []any{ (ActorMetadataField)(0), // 0: atelet.ActorMetadataField (SnapshotFidelity)(0), // 1: atelet.SnapshotFidelity @@ -4308,8 +4230,8 @@ var file_atelet_proto_goTypes = []any{ (*VersionedSandboxCompat)(nil), // 5: atelet.VersionedSandboxCompat (*AttributeEntry)(nil), // 6: atelet.AttributeEntry (*WorkerResources)(nil), // 7: atelet.WorkerResources - (*Resources)(nil), // 8: atelet.Resources - (*Limits)(nil), // 9: atelet.Limits + (*ResourceRequirements)(nil), // 8: atelet.ResourceRequirements + (*Resources)(nil), // 9: atelet.Resources (*RegisterWorkerResponse)(nil), // 10: atelet.RegisterWorkerResponse (*RequestActorSuspendRequest)(nil), // 11: atelet.RequestActorSuspendRequest (*RequestActorSuspendResponse)(nil), // 12: atelet.RequestActorSuspendResponse @@ -4342,27 +4264,26 @@ var file_atelet_proto_goTypes = []any{ (*Container)(nil), // 39: atelet.Container (*SecurityContext)(nil), // 40: atelet.SecurityContext (*Capabilities)(nil), // 41: atelet.Capabilities - (*ResourceLimits)(nil), // 42: atelet.ResourceLimits - (*EnvVar)(nil), // 43: atelet.EnvVar - (*WakeupProbe)(nil), // 44: atelet.WakeupProbe - (*HttpGetAction)(nil), // 45: atelet.HttpGetAction - (*RunResponse)(nil), // 46: atelet.RunResponse - (*ObjectStorage)(nil), // 47: atelet.ObjectStorage - (*Snapshot)(nil), // 48: atelet.Snapshot - (*SnapshotReport)(nil), // 49: atelet.SnapshotReport - (*ActorSnapshotState)(nil), // 50: atelet.ActorSnapshotState - (*UploadConfig)(nil), // 51: atelet.UploadConfig - (*CheckpointRequest)(nil), // 52: atelet.CheckpointRequest - (*CheckpointResponse)(nil), // 53: atelet.CheckpointResponse - (*RestoreRequest)(nil), // 54: atelet.RestoreRequest - (*RestoreResponse)(nil), // 55: atelet.RestoreResponse - (*PromoteSnapshotRequest)(nil), // 56: atelet.PromoteSnapshotRequest - (*PromoteSnapshotResponse)(nil), // 57: atelet.PromoteSnapshotResponse - (*DetachActorRequest)(nil), // 58: atelet.DetachActorRequest - (*DetachActorResponse)(nil), // 59: atelet.DetachActorResponse - nil, // 60: atelet.ExternalVolumeSource.VolumeContextEntry - nil, // 61: atelet.ExternalVolumeSource.PublishContextEntry - (*timestamppb.Timestamp)(nil), // 62: google.protobuf.Timestamp + (*EnvVar)(nil), // 42: atelet.EnvVar + (*WakeupProbe)(nil), // 43: atelet.WakeupProbe + (*HttpGetAction)(nil), // 44: atelet.HttpGetAction + (*RunResponse)(nil), // 45: atelet.RunResponse + (*ObjectStorage)(nil), // 46: atelet.ObjectStorage + (*Snapshot)(nil), // 47: atelet.Snapshot + (*SnapshotReport)(nil), // 48: atelet.SnapshotReport + (*ActorSnapshotState)(nil), // 49: atelet.ActorSnapshotState + (*UploadConfig)(nil), // 50: atelet.UploadConfig + (*CheckpointRequest)(nil), // 51: atelet.CheckpointRequest + (*CheckpointResponse)(nil), // 52: atelet.CheckpointResponse + (*RestoreRequest)(nil), // 53: atelet.RestoreRequest + (*RestoreResponse)(nil), // 54: atelet.RestoreResponse + (*PromoteSnapshotRequest)(nil), // 55: atelet.PromoteSnapshotRequest + (*PromoteSnapshotResponse)(nil), // 56: atelet.PromoteSnapshotResponse + (*DetachActorRequest)(nil), // 57: atelet.DetachActorRequest + (*DetachActorResponse)(nil), // 58: atelet.DetachActorResponse + nil, // 59: atelet.ExternalVolumeSource.VolumeContextEntry + nil, // 60: atelet.ExternalVolumeSource.PublishContextEntry + (*timestamppb.Timestamp)(nil), // 61: google.protobuf.Timestamp } var file_atelet_proto_depIdxs = []int32{ 7, // 0: atelet.RegisterWorkerRequest.capacity:type_name -> atelet.WorkerResources @@ -4370,23 +4291,23 @@ var file_atelet_proto_depIdxs = []int32{ 4, // 2: atelet.RegisterWorkerRequest.restorable_runtimes:type_name -> atelet.SandboxRuntime 5, // 3: atelet.SandboxRuntime.version:type_name -> atelet.VersionedSandboxCompat 6, // 4: atelet.VersionedSandboxCompat.attributes:type_name -> atelet.AttributeEntry - 8, // 5: atelet.WorkerResources.resources:type_name -> atelet.Resources - 9, // 6: atelet.Resources.limits:type_name -> atelet.Limits - 17, // 7: atelet.GetActorEgressPolicyResponse.egress_policy:type_name -> atelet.EgressPolicy - 18, // 8: atelet.EgressPolicy.rules:type_name -> atelet.EgressRule - 19, // 9: atelet.EgressRule.http:type_name -> atelet.EgressDestinations - 19, // 10: atelet.EgressRule.https:type_name -> atelet.EgressDestinations - 19, // 11: atelet.EgressRule.tls_passthrough:type_name -> atelet.EgressDestinations - 20, // 12: atelet.EgressDestinations.ports:type_name -> atelet.EgressPorts - 21, // 13: atelet.EgressPorts.all:type_name -> atelet.EgressAllPorts - 28, // 14: atelet.TerminateRequest.spec:type_name -> atelet.WorkloadSpec - 28, // 15: atelet.RunRequest.spec:type_name -> atelet.WorkloadSpec - 27, // 16: atelet.RunRequest.egress_gateway:type_name -> atelet.EgressGateway + 9, // 5: atelet.ResourceRequirements.limits:type_name -> atelet.Resources + 17, // 6: atelet.GetActorEgressPolicyResponse.egress_policy:type_name -> atelet.EgressPolicy + 18, // 7: atelet.EgressPolicy.rules:type_name -> atelet.EgressRule + 19, // 8: atelet.EgressRule.http:type_name -> atelet.EgressDestinations + 19, // 9: atelet.EgressRule.https:type_name -> atelet.EgressDestinations + 19, // 10: atelet.EgressRule.tls_passthrough:type_name -> atelet.EgressDestinations + 20, // 11: atelet.EgressDestinations.ports:type_name -> atelet.EgressPorts + 21, // 12: atelet.EgressPorts.all:type_name -> atelet.EgressAllPorts + 28, // 13: atelet.TerminateRequest.spec:type_name -> atelet.WorkloadSpec + 28, // 14: atelet.RunRequest.spec:type_name -> atelet.WorkloadSpec + 27, // 15: atelet.RunRequest.egress_gateway:type_name -> atelet.EgressGateway + 8, // 16: atelet.RunRequest.resources:type_name -> atelet.ResourceRequirements 4, // 17: atelet.RunRequest.sandbox_runtime:type_name -> atelet.SandboxRuntime 39, // 18: atelet.WorkloadSpec.containers:type_name -> atelet.Container 37, // 19: atelet.WorkloadSpec.volumes:type_name -> atelet.Volume - 60, // 20: atelet.ExternalVolumeSource.volume_context:type_name -> atelet.ExternalVolumeSource.VolumeContextEntry - 61, // 21: atelet.ExternalVolumeSource.publish_context:type_name -> atelet.ExternalVolumeSource.PublishContextEntry + 59, // 20: atelet.ExternalVolumeSource.volume_context:type_name -> atelet.ExternalVolumeSource.VolumeContextEntry + 60, // 21: atelet.ExternalVolumeSource.publish_context:type_name -> atelet.ExternalVolumeSource.PublishContextEntry 0, // 22: atelet.ActorMetadataItem.field:type_name -> atelet.ActorMetadataField 32, // 23: atelet.ActorMetadataDataSource.items:type_name -> atelet.ActorMetadataItem 33, // 24: atelet.SystemInfoDataSource.actor_metadata:type_name -> atelet.ActorMetadataDataSource @@ -4396,61 +4317,62 @@ var file_atelet_proto_depIdxs = []int32{ 30, // 28: atelet.Volume.external:type_name -> atelet.ExternalVolumeSource 36, // 29: atelet.Volume.system_info:type_name -> atelet.SystemInfoVolumeSource 31, // 30: atelet.Volume.image:type_name -> atelet.ImageVolumeSource - 43, // 31: atelet.Container.env:type_name -> atelet.EnvVar - 44, // 32: atelet.Container.wakeup_probe:type_name -> atelet.WakeupProbe + 42, // 31: atelet.Container.env:type_name -> atelet.EnvVar + 43, // 32: atelet.Container.wakeup_probe:type_name -> atelet.WakeupProbe 38, // 33: atelet.Container.volume_mounts:type_name -> atelet.VolumeMount 40, // 34: atelet.Container.security_context:type_name -> atelet.SecurityContext - 42, // 35: atelet.Container.resources:type_name -> atelet.ResourceLimits + 8, // 35: atelet.Container.resources:type_name -> atelet.ResourceRequirements 41, // 36: atelet.SecurityContext.capabilities:type_name -> atelet.Capabilities - 45, // 37: atelet.WakeupProbe.http_get:type_name -> atelet.HttpGetAction + 44, // 37: atelet.WakeupProbe.http_get:type_name -> atelet.HttpGetAction 1, // 38: atelet.Snapshot.fidelity:type_name -> atelet.SnapshotFidelity - 47, // 39: atelet.Snapshot.object:type_name -> atelet.ObjectStorage - 48, // 40: atelet.SnapshotReport.snapshot:type_name -> atelet.Snapshot - 62, // 41: atelet.SnapshotReport.local_commit_time:type_name -> google.protobuf.Timestamp - 62, // 42: atelet.SnapshotReport.durable_complete_time:type_name -> google.protobuf.Timestamp - 49, // 43: atelet.ActorSnapshotState.latest_local:type_name -> atelet.SnapshotReport - 49, // 44: atelet.ActorSnapshotState.uploading:type_name -> atelet.SnapshotReport - 49, // 45: atelet.ActorSnapshotState.latest_durable:type_name -> atelet.SnapshotReport - 62, // 46: atelet.ActorSnapshotState.upload_due_time:type_name -> google.protobuf.Timestamp + 46, // 39: atelet.Snapshot.object:type_name -> atelet.ObjectStorage + 47, // 40: atelet.SnapshotReport.snapshot:type_name -> atelet.Snapshot + 61, // 41: atelet.SnapshotReport.local_commit_time:type_name -> google.protobuf.Timestamp + 61, // 42: atelet.SnapshotReport.durable_complete_time:type_name -> google.protobuf.Timestamp + 48, // 43: atelet.ActorSnapshotState.latest_local:type_name -> atelet.SnapshotReport + 48, // 44: atelet.ActorSnapshotState.uploading:type_name -> atelet.SnapshotReport + 48, // 45: atelet.ActorSnapshotState.latest_durable:type_name -> atelet.SnapshotReport + 61, // 46: atelet.ActorSnapshotState.upload_due_time:type_name -> google.protobuf.Timestamp 2, // 47: atelet.UploadConfig.mode:type_name -> atelet.UploadMode 28, // 48: atelet.CheckpointRequest.spec:type_name -> atelet.WorkloadSpec - 48, // 49: atelet.CheckpointRequest.snapshot:type_name -> atelet.Snapshot - 51, // 50: atelet.CheckpointRequest.upload_config:type_name -> atelet.UploadConfig - 50, // 51: atelet.CheckpointResponse.state:type_name -> atelet.ActorSnapshotState + 47, // 49: atelet.CheckpointRequest.snapshot:type_name -> atelet.Snapshot + 50, // 50: atelet.CheckpointRequest.upload_config:type_name -> atelet.UploadConfig + 49, // 51: atelet.CheckpointResponse.state:type_name -> atelet.ActorSnapshotState 28, // 52: atelet.RestoreRequest.spec:type_name -> atelet.WorkloadSpec - 48, // 53: atelet.RestoreRequest.snapshot:type_name -> atelet.Snapshot + 47, // 53: atelet.RestoreRequest.snapshot:type_name -> atelet.Snapshot 27, // 54: atelet.RestoreRequest.egress_gateway:type_name -> atelet.EgressGateway - 4, // 55: atelet.RestoreRequest.sandbox_runtime:type_name -> atelet.SandboxRuntime - 48, // 56: atelet.PromoteSnapshotRequest.snapshot:type_name -> atelet.Snapshot - 50, // 57: atelet.PromoteSnapshotResponse.state:type_name -> atelet.ActorSnapshotState - 50, // 58: atelet.DetachActorResponse.state:type_name -> atelet.ActorSnapshotState - 22, // 59: atelet.WorkerSupport.MintActorCertificate:input_type -> atelet.MintActorCertificateRequest - 3, // 60: atelet.WorkerSupport.RegisterWorker:input_type -> atelet.RegisterWorkerRequest - 11, // 61: atelet.WorkerSupport.RequestActorSuspend:input_type -> atelet.RequestActorSuspendRequest - 15, // 62: atelet.WorkerSupport.GetActorEgressPolicy:input_type -> atelet.GetActorEgressPolicyRequest - 13, // 63: atelet.WorkerSupport.ReportActorCrashed:input_type -> atelet.ReportActorCrashedRequest - 26, // 64: atelet.Atelet.Run:input_type -> atelet.RunRequest - 52, // 65: atelet.Atelet.Checkpoint:input_type -> atelet.CheckpointRequest - 54, // 66: atelet.Atelet.Restore:input_type -> atelet.RestoreRequest - 56, // 67: atelet.Atelet.PromoteSnapshot:input_type -> atelet.PromoteSnapshotRequest - 58, // 68: atelet.Atelet.DetachActor:input_type -> atelet.DetachActorRequest - 24, // 69: atelet.Atelet.Terminate:input_type -> atelet.TerminateRequest - 23, // 70: atelet.WorkerSupport.MintActorCertificate:output_type -> atelet.MintActorCertificateResponse - 10, // 71: atelet.WorkerSupport.RegisterWorker:output_type -> atelet.RegisterWorkerResponse - 12, // 72: atelet.WorkerSupport.RequestActorSuspend:output_type -> atelet.RequestActorSuspendResponse - 16, // 73: atelet.WorkerSupport.GetActorEgressPolicy:output_type -> atelet.GetActorEgressPolicyResponse - 14, // 74: atelet.WorkerSupport.ReportActorCrashed:output_type -> atelet.ReportActorCrashedResponse - 46, // 75: atelet.Atelet.Run:output_type -> atelet.RunResponse - 53, // 76: atelet.Atelet.Checkpoint:output_type -> atelet.CheckpointResponse - 55, // 77: atelet.Atelet.Restore:output_type -> atelet.RestoreResponse - 57, // 78: atelet.Atelet.PromoteSnapshot:output_type -> atelet.PromoteSnapshotResponse - 59, // 79: atelet.Atelet.DetachActor:output_type -> atelet.DetachActorResponse - 25, // 80: atelet.Atelet.Terminate:output_type -> atelet.TerminateResponse - 70, // [70:81] is the sub-list for method output_type - 59, // [59:70] is the sub-list for method input_type - 59, // [59:59] is the sub-list for extension type_name - 59, // [59:59] is the sub-list for extension extendee - 0, // [0:59] is the sub-list for field type_name + 8, // 55: atelet.RestoreRequest.resources:type_name -> atelet.ResourceRequirements + 4, // 56: atelet.RestoreRequest.sandbox_runtime:type_name -> atelet.SandboxRuntime + 47, // 57: atelet.PromoteSnapshotRequest.snapshot:type_name -> atelet.Snapshot + 49, // 58: atelet.PromoteSnapshotResponse.state:type_name -> atelet.ActorSnapshotState + 49, // 59: atelet.DetachActorResponse.state:type_name -> atelet.ActorSnapshotState + 22, // 60: atelet.WorkerSupport.MintActorCertificate:input_type -> atelet.MintActorCertificateRequest + 3, // 61: atelet.WorkerSupport.RegisterWorker:input_type -> atelet.RegisterWorkerRequest + 11, // 62: atelet.WorkerSupport.RequestActorSuspend:input_type -> atelet.RequestActorSuspendRequest + 15, // 63: atelet.WorkerSupport.GetActorEgressPolicy:input_type -> atelet.GetActorEgressPolicyRequest + 13, // 64: atelet.WorkerSupport.ReportActorCrashed:input_type -> atelet.ReportActorCrashedRequest + 26, // 65: atelet.Atelet.Run:input_type -> atelet.RunRequest + 51, // 66: atelet.Atelet.Checkpoint:input_type -> atelet.CheckpointRequest + 53, // 67: atelet.Atelet.Restore:input_type -> atelet.RestoreRequest + 55, // 68: atelet.Atelet.PromoteSnapshot:input_type -> atelet.PromoteSnapshotRequest + 57, // 69: atelet.Atelet.DetachActor:input_type -> atelet.DetachActorRequest + 24, // 70: atelet.Atelet.Terminate:input_type -> atelet.TerminateRequest + 23, // 71: atelet.WorkerSupport.MintActorCertificate:output_type -> atelet.MintActorCertificateResponse + 10, // 72: atelet.WorkerSupport.RegisterWorker:output_type -> atelet.RegisterWorkerResponse + 12, // 73: atelet.WorkerSupport.RequestActorSuspend:output_type -> atelet.RequestActorSuspendResponse + 16, // 74: atelet.WorkerSupport.GetActorEgressPolicy:output_type -> atelet.GetActorEgressPolicyResponse + 14, // 75: atelet.WorkerSupport.ReportActorCrashed:output_type -> atelet.ReportActorCrashedResponse + 45, // 76: atelet.Atelet.Run:output_type -> atelet.RunResponse + 52, // 77: atelet.Atelet.Checkpoint:output_type -> atelet.CheckpointResponse + 54, // 78: atelet.Atelet.Restore:output_type -> atelet.RestoreResponse + 56, // 79: atelet.Atelet.PromoteSnapshot:output_type -> atelet.PromoteSnapshotResponse + 58, // 80: atelet.Atelet.DetachActor:output_type -> atelet.DetachActorResponse + 25, // 81: atelet.Atelet.Terminate:output_type -> atelet.TerminateResponse + 71, // [71:82] is the sub-list for method output_type + 60, // [60:71] is the sub-list for method input_type + 60, // [60:60] is the sub-list for extension type_name + 60, // [60:60] is the sub-list for extension extendee + 0, // [0:60] is the sub-list for field type_name } func init() { file_atelet_proto_init() } @@ -4459,14 +4381,14 @@ func file_atelet_proto_init() { return } file_atelet_proto_msgTypes[23].OneofWrappers = []any{} - file_atelet_proto_msgTypes[51].OneofWrappers = []any{} + file_atelet_proto_msgTypes[50].OneofWrappers = []any{} type x struct{} out := protoimpl.TypeBuilder{ File: protoimpl.DescBuilder{ GoPackagePath: reflect.TypeOf(x{}).PkgPath(), RawDescriptor: unsafe.Slice(unsafe.StringData(file_atelet_proto_rawDesc), len(file_atelet_proto_rawDesc)), NumEnums: 3, - NumMessages: 59, + NumMessages: 58, NumExtensions: 0, NumServices: 2, }, diff --git a/internal/proto/ateletpb/atelet.proto b/internal/proto/ateletpb/atelet.proto index 74417fa926..495a4e0cae 100644 --- a/internal/proto/ateletpb/atelet.proto +++ b/internal/proto/ateletpb/atelet.proto @@ -151,10 +151,20 @@ message AttributeEntry { // WorkerResources is what a worker can supply. It mirrors the control plane's // WorkerResources and is validated by the same rules. message WorkerResources { - // Resources are matched by name against actors' resources for placement. + // cpu_millicores is CPU in millicores (1000 = one core). Compared against + // actors' resource limits for placement. // // +k8s:optional - Resources resources = 1; + // +k8s:minimum=0 + // +k8s:maximum=999999 # less than 1000 cores + int64 cpu_millicores = 1; + + // memory_bytes is memory in bytes. Compared against actors' resource + // limits for placement. + // + // +k8s:optional + // +k8s:minimum=0 + int64 memory_bytes = 2; // A count of Actors. Workers can use this to limit the number of Actors // that can be placed on them, constraining costs not captured in the @@ -162,33 +172,31 @@ message WorkerResources { // // +k8s:optional // +k8s:minimum=1 - int32 actors = 2; + int32 actors = 3; } -message Resources { - // limits is the maximum amount of compute resources allowed. Only "cpu" - // and "memory" are supported, each at most once, and each quantity must be - // greater than zero; the cpu limit must be less than 1000 cores. +// ResourceRequirements are the compute resources a workload may use. +message ResourceRequirements { + // limits is the maximum amount of compute resources allowed. // // +k8s:optional - // +k8s:maxItems=2 - // +k8s:listType=map - // +k8s:listMapKey=name - repeated Limits limits = 1; + Resources limits = 1; } -// +k8s:customValidation # names, quantity parse/positivity, cpu bound -message Limits { - // +k8s:required - // +k8s:maxLength=16 # the Limits hook restricts values to cpu/memory - string name = 1; +// Resources is an amount of each compute resource. +message Resources { + // cpu_millicores is CPU in millicores (1000 = one core). 0 means none. + // + // +k8s:optional + // +k8s:minimum=0 + // +k8s:maximum=999999 # less than 1000 cores + int64 cpu_millicores = 1; - // quantity is in Kubernetes resource.Quantity string form (e.g. "500m", - // "2Gi"). + // memory_bytes is memory in bytes. 0 means none. // - // +k8s:required - // +k8s:maxLength=32 # the Limits hook requires a parseable quantity - string quantity = 2; + // +k8s:optional + // +k8s:minimum=0 + int64 memory_bytes = 2; } message RegisterWorkerResponse { @@ -471,16 +479,10 @@ message RunRequest { // The actor's declared size, from the ActorTemplate's resource limits. atelet // passes these through to the sandbox so it is sized to the actor (not the - // whole host or worker pod). Zero means "unset": keep the runtime default. + // whole host or worker pod). A zero limit keeps the runtime default. // // +k8s:optional - // +k8s:minimum=0 - // +k8s:maximum=999999 # the control plane caps cpu limits strictly below 1000 cores - int64 cpu_milli = 10; // CPU limit in millicores (1000 = one core). - - // +k8s:optional - // +k8s:minimum=0 - int64 memory_bytes = 11; // Memory limit in bytes. + ResourceRequirements resources = 10; // The runtime the assigned worker runs this actor under, from // WorkerAssignment.sandbox_runtime. atelet resolves the pause image and @@ -729,11 +731,11 @@ message Container { // +k8s:optional SecurityContext security_context = 8; - // resources are the cgroup limits for this container, resolved by - // ate-api-server from the ActorTemplate. Unset means no limits. + // resources are the cgroup limits for this container, from the + // ActorTemplate. Unset means no limits. // // +k8s:optional - ResourceLimits resources = 9; + ResourceRequirements resources = 9; } // SecurityContext holds security settings for a container's process. @@ -758,23 +760,6 @@ message Capabilities { repeated string drop = 2; } -// ResourceLimits are the cgroup limits for one container. Scalars rather than a -// map so each resource.Quantity is parsed exactly once, in ate-api-server. -message ResourceLimits { - // memory_bytes is the memory limit in bytes. 0 means unset. - // - // +k8s:optional - // +k8s:minimum=0 - int64 memory_bytes = 1; - - // cpu_millis is the CPU limit in milli-cores (1000 = one core). 0 means unset. - // - // +k8s:optional - // +k8s:minimum=0 - // +k8s:maximum=999999 # the control plane caps cpu limits strictly below 1000 cores - int64 cpu_millis = 2; -} - message EnvVar { // name may be any printable ASCII character except '='. // @@ -1091,13 +1076,7 @@ message RestoreRequest { // means "unset": keep the runtime default. // // +k8s:optional - // +k8s:minimum=0 - // +k8s:maximum=999999 # the control plane caps cpu limits strictly below 1000 cores - int64 cpu_milli = 10; // CPU limit in millicores (1000 = one core). - - // +k8s:optional - // +k8s:minimum=0 - int64 memory_bytes = 11; // Memory limit in bytes. + ResourceRequirements resources = 10; // The runtime the assigned worker restores this actor under, from // WorkerAssignment.sandbox_runtime. atelet resolves the pause image and diff --git a/internal/proto/ateworkerpb/ateworker.pb.go b/internal/proto/ateworkerpb/ateworker.pb.go index 3e4d6df0c0..322b7d1648 100644 --- a/internal/proto/ateworkerpb/ateworker.pb.go +++ b/internal/proto/ateworkerpb/ateworker.pb.go @@ -498,12 +498,7 @@ type RunWorkloadRequest struct { // runtime default (unlimited for gVisor, the worker's own default for the micro-VM). // // +k8s:optional - // +k8s:minimum=1 - // +k8s:maximum=999999 # matches atelet.RunRequest.cpu_milli, the control plane caps cpu limits strictly below 1000 cores - CpuMilli int64 `protobuf:"varint,10,opt,name=cpu_milli,json=cpuMilli,proto3" json:"cpu_milli,omitempty"` // CPU limit in millicores (1000 = one core). - // +k8s:optional - // +k8s:minimum=1 - MemoryBytes int64 `protobuf:"varint,11,opt,name=memory_bytes,json=memoryBytes,proto3" json:"memory_bytes,omitempty"` // Memory limit in bytes. + Resources *ResourceRequirements `protobuf:"bytes,10,opt,name=resources,proto3" json:"resources,omitempty"` // +k8s:required ActorDirs *ActorDirs `protobuf:"bytes,12,opt,name=actor_dirs,json=actorDirs,proto3" json:"actor_dirs,omitempty"` unknownFields protoimpl.UnknownFields @@ -603,18 +598,11 @@ func (x *RunWorkloadRequest) GetEgressGateway() *EgressGateway { return nil } -func (x *RunWorkloadRequest) GetCpuMilli() int64 { +func (x *RunWorkloadRequest) GetResources() *ResourceRequirements { if x != nil { - return x.CpuMilli - } - return 0 -} - -func (x *RunWorkloadRequest) GetMemoryBytes() int64 { - if x != nil { - return x.MemoryBytes + return x.Resources } - return 0 + return nil } func (x *RunWorkloadRequest) GetActorDirs() *ActorDirs { @@ -878,7 +866,9 @@ type ContainerSpec struct { // capabilities is the container's capability set, CAP_-prefixed. Capabilities []string `protobuf:"bytes,3,rep,name=capabilities,proto3" json:"capabilities,omitempty"` // resources are the container's own declared limits, if any. - Resources *ResourceLimits `protobuf:"bytes,4,opt,name=resources,proto3" json:"resources,omitempty"` + // + // +k8s:optional + Resources *ResourceRequirements `protobuf:"bytes,4,opt,name=resources,proto3" json:"resources,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache } @@ -934,38 +924,38 @@ func (x *ContainerSpec) GetCapabilities() []string { return nil } -func (x *ContainerSpec) GetResources() *ResourceLimits { +func (x *ContainerSpec) GetResources() *ResourceRequirements { if x != nil { return x.Resources } return nil } -// ResourceLimits are a container's declared limits. -type ResourceLimits struct { +// ResourceRequirements mirrors atelet's ResourceRequirements. +type ResourceRequirements struct { state protoimpl.MessageState `protogen:"open.v1"` - // memory_bytes is the memory limit in bytes. 0 means unset. - MemoryBytes int64 `protobuf:"varint,1,opt,name=memory_bytes,json=memoryBytes,proto3" json:"memory_bytes,omitempty"` - // cpu_millis is the CPU limit in milli-cores (1000 = one core). 0 means unset. - CpuMillis int64 `protobuf:"varint,2,opt,name=cpu_millis,json=cpuMillis,proto3" json:"cpu_millis,omitempty"` + // limits is the maximum amount of compute resources allowed. + // + // +k8s:optional + Limits *Resources `protobuf:"bytes,1,opt,name=limits,proto3" json:"limits,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache } -func (x *ResourceLimits) Reset() { - *x = ResourceLimits{} +func (x *ResourceRequirements) Reset() { + *x = ResourceRequirements{} mi := &file_ateworker_proto_msgTypes[8] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } -func (x *ResourceLimits) String() string { +func (x *ResourceRequirements) String() string { return protoimpl.X.MessageStringOf(x) } -func (*ResourceLimits) ProtoMessage() {} +func (*ResourceRequirements) ProtoMessage() {} -func (x *ResourceLimits) ProtoReflect() protoreflect.Message { +func (x *ResourceRequirements) ProtoReflect() protoreflect.Message { mi := &file_ateworker_proto_msgTypes[8] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) @@ -977,21 +967,76 @@ func (x *ResourceLimits) ProtoReflect() protoreflect.Message { return mi.MessageOf(x) } -// Deprecated: Use ResourceLimits.ProtoReflect.Descriptor instead. -func (*ResourceLimits) Descriptor() ([]byte, []int) { +// Deprecated: Use ResourceRequirements.ProtoReflect.Descriptor instead. +func (*ResourceRequirements) Descriptor() ([]byte, []int) { return file_ateworker_proto_rawDescGZIP(), []int{8} } -func (x *ResourceLimits) GetMemoryBytes() int64 { +func (x *ResourceRequirements) GetLimits() *Resources { if x != nil { - return x.MemoryBytes + return x.Limits + } + return nil +} + +// Resources is an amount of each compute resource. +type Resources struct { + state protoimpl.MessageState `protogen:"open.v1"` + // cpu_millicores is CPU in millicores (1000 = one core). 0 means none. + // + // +k8s:optional + // +k8s:minimum=0 + // +k8s:maximum=999999 # less than 1000 cores + CpuMillicores int64 `protobuf:"varint,1,opt,name=cpu_millicores,json=cpuMillicores,proto3" json:"cpu_millicores,omitempty"` + // memory_bytes is memory in bytes. 0 means none. + // + // +k8s:optional + // +k8s:minimum=0 + MemoryBytes int64 `protobuf:"varint,2,opt,name=memory_bytes,json=memoryBytes,proto3" json:"memory_bytes,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache +} + +func (x *Resources) Reset() { + *x = Resources{} + mi := &file_ateworker_proto_msgTypes[9] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *Resources) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*Resources) ProtoMessage() {} + +func (x *Resources) ProtoReflect() protoreflect.Message { + mi := &file_ateworker_proto_msgTypes[9] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use Resources.ProtoReflect.Descriptor instead. +func (*Resources) Descriptor() ([]byte, []int) { + return file_ateworker_proto_rawDescGZIP(), []int{9} +} + +func (x *Resources) GetCpuMillicores() int64 { + if x != nil { + return x.CpuMillicores } return 0 } -func (x *ResourceLimits) GetCpuMillis() int64 { +func (x *Resources) GetMemoryBytes() int64 { if x != nil { - return x.CpuMillis + return x.MemoryBytes } return 0 } @@ -1018,7 +1063,7 @@ type VolumeMount struct { func (x *VolumeMount) Reset() { *x = VolumeMount{} - mi := &file_ateworker_proto_msgTypes[9] + mi := &file_ateworker_proto_msgTypes[10] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1030,7 +1075,7 @@ func (x *VolumeMount) String() string { func (*VolumeMount) ProtoMessage() {} func (x *VolumeMount) ProtoReflect() protoreflect.Message { - mi := &file_ateworker_proto_msgTypes[9] + mi := &file_ateworker_proto_msgTypes[10] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1043,7 +1088,7 @@ func (x *VolumeMount) ProtoReflect() protoreflect.Message { // Deprecated: Use VolumeMount.ProtoReflect.Descriptor instead. func (*VolumeMount) Descriptor() ([]byte, []int) { - return file_ateworker_proto_rawDescGZIP(), []int{9} + return file_ateworker_proto_rawDescGZIP(), []int{10} } func (x *VolumeMount) GetName() string { @@ -1078,7 +1123,7 @@ type WakeupProbe struct { func (x *WakeupProbe) Reset() { *x = WakeupProbe{} - mi := &file_ateworker_proto_msgTypes[10] + mi := &file_ateworker_proto_msgTypes[11] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1090,7 +1135,7 @@ func (x *WakeupProbe) String() string { func (*WakeupProbe) ProtoMessage() {} func (x *WakeupProbe) ProtoReflect() protoreflect.Message { - mi := &file_ateworker_proto_msgTypes[10] + mi := &file_ateworker_proto_msgTypes[11] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1103,7 +1148,7 @@ func (x *WakeupProbe) ProtoReflect() protoreflect.Message { // Deprecated: Use WakeupProbe.ProtoReflect.Descriptor instead. func (*WakeupProbe) Descriptor() ([]byte, []int) { - return file_ateworker_proto_rawDescGZIP(), []int{10} + return file_ateworker_proto_rawDescGZIP(), []int{11} } func (x *WakeupProbe) GetHttpGet() *HttpGetAction { @@ -1141,7 +1186,7 @@ type HttpGetAction struct { func (x *HttpGetAction) Reset() { *x = HttpGetAction{} - mi := &file_ateworker_proto_msgTypes[11] + mi := &file_ateworker_proto_msgTypes[12] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1153,7 +1198,7 @@ func (x *HttpGetAction) String() string { func (*HttpGetAction) ProtoMessage() {} func (x *HttpGetAction) ProtoReflect() protoreflect.Message { - mi := &file_ateworker_proto_msgTypes[11] + mi := &file_ateworker_proto_msgTypes[12] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1166,7 +1211,7 @@ func (x *HttpGetAction) ProtoReflect() protoreflect.Message { // Deprecated: Use HttpGetAction.ProtoReflect.Descriptor instead. func (*HttpGetAction) Descriptor() ([]byte, []int) { - return file_ateworker_proto_rawDescGZIP(), []int{11} + return file_ateworker_proto_rawDescGZIP(), []int{12} } func (x *HttpGetAction) GetPath() string { @@ -1191,7 +1236,7 @@ type RunWorkloadResponse struct { func (x *RunWorkloadResponse) Reset() { *x = RunWorkloadResponse{} - mi := &file_ateworker_proto_msgTypes[12] + mi := &file_ateworker_proto_msgTypes[13] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1203,7 +1248,7 @@ func (x *RunWorkloadResponse) String() string { func (*RunWorkloadResponse) ProtoMessage() {} func (x *RunWorkloadResponse) ProtoReflect() protoreflect.Message { - mi := &file_ateworker_proto_msgTypes[12] + mi := &file_ateworker_proto_msgTypes[13] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1216,7 +1261,7 @@ func (x *RunWorkloadResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use RunWorkloadResponse.ProtoReflect.Descriptor instead. func (*RunWorkloadResponse) Descriptor() ([]byte, []int) { - return file_ateworker_proto_rawDescGZIP(), []int{12} + return file_ateworker_proto_rawDescGZIP(), []int{13} } type CheckpointWorkloadRequest struct { @@ -1257,7 +1302,7 @@ type CheckpointWorkloadRequest struct { func (x *CheckpointWorkloadRequest) Reset() { *x = CheckpointWorkloadRequest{} - mi := &file_ateworker_proto_msgTypes[13] + mi := &file_ateworker_proto_msgTypes[14] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1269,7 +1314,7 @@ func (x *CheckpointWorkloadRequest) String() string { func (*CheckpointWorkloadRequest) ProtoMessage() {} func (x *CheckpointWorkloadRequest) ProtoReflect() protoreflect.Message { - mi := &file_ateworker_proto_msgTypes[13] + mi := &file_ateworker_proto_msgTypes[14] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1282,7 +1327,7 @@ func (x *CheckpointWorkloadRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use CheckpointWorkloadRequest.ProtoReflect.Descriptor instead. func (*CheckpointWorkloadRequest) Descriptor() ([]byte, []int) { - return file_ateworker_proto_rawDescGZIP(), []int{13} + return file_ateworker_proto_rawDescGZIP(), []int{14} } func (x *CheckpointWorkloadRequest) GetAtespace() string { @@ -1363,7 +1408,7 @@ type CheckpointWorkloadResponse struct { func (x *CheckpointWorkloadResponse) Reset() { *x = CheckpointWorkloadResponse{} - mi := &file_ateworker_proto_msgTypes[14] + mi := &file_ateworker_proto_msgTypes[15] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1375,7 +1420,7 @@ func (x *CheckpointWorkloadResponse) String() string { func (*CheckpointWorkloadResponse) ProtoMessage() {} func (x *CheckpointWorkloadResponse) ProtoReflect() protoreflect.Message { - mi := &file_ateworker_proto_msgTypes[14] + mi := &file_ateworker_proto_msgTypes[15] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1388,7 +1433,7 @@ func (x *CheckpointWorkloadResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use CheckpointWorkloadResponse.ProtoReflect.Descriptor instead. func (*CheckpointWorkloadResponse) Descriptor() ([]byte, []int) { - return file_ateworker_proto_rawDescGZIP(), []int{14} + return file_ateworker_proto_rawDescGZIP(), []int{15} } func (x *CheckpointWorkloadResponse) GetSnapshotFiles() []string { @@ -1454,12 +1499,7 @@ type RestoreWorkloadRequest struct { // these are ignored. Zero means "unset": keep the runtime default. // // +k8s:optional - // +k8s:minimum=1 - // +k8s:maximum=999999 # matches atelet.RestoreRequest.cpu_milli, the control plane caps cpu limits strictly below 1000 cores - CpuMilli int64 `protobuf:"varint,11,opt,name=cpu_milli,json=cpuMilli,proto3" json:"cpu_milli,omitempty"` // CPU limit in millicores (1000 = one core). - // +k8s:optional - // +k8s:minimum=1 - MemoryBytes int64 `protobuf:"varint,12,opt,name=memory_bytes,json=memoryBytes,proto3" json:"memory_bytes,omitempty"` // Memory limit in bytes. + Resources *ResourceRequirements `protobuf:"bytes,11,opt,name=resources,proto3" json:"resources,omitempty"` // +k8s:required ActorDirs *ActorDirs `protobuf:"bytes,13,opt,name=actor_dirs,json=actorDirs,proto3" json:"actor_dirs,omitempty"` // actor_dirs.restore_dir is a preserved local snapshot, not a scratch copy: @@ -1473,7 +1513,7 @@ type RestoreWorkloadRequest struct { func (x *RestoreWorkloadRequest) Reset() { *x = RestoreWorkloadRequest{} - mi := &file_ateworker_proto_msgTypes[15] + mi := &file_ateworker_proto_msgTypes[16] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1485,7 +1525,7 @@ func (x *RestoreWorkloadRequest) String() string { func (*RestoreWorkloadRequest) ProtoMessage() {} func (x *RestoreWorkloadRequest) ProtoReflect() protoreflect.Message { - mi := &file_ateworker_proto_msgTypes[15] + mi := &file_ateworker_proto_msgTypes[16] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1498,7 +1538,7 @@ func (x *RestoreWorkloadRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use RestoreWorkloadRequest.ProtoReflect.Descriptor instead. func (*RestoreWorkloadRequest) Descriptor() ([]byte, []int) { - return file_ateworker_proto_rawDescGZIP(), []int{15} + return file_ateworker_proto_rawDescGZIP(), []int{16} } func (x *RestoreWorkloadRequest) GetAtespace() string { @@ -1571,18 +1611,11 @@ func (x *RestoreWorkloadRequest) GetEgressGateway() *EgressGateway { return nil } -func (x *RestoreWorkloadRequest) GetCpuMilli() int64 { +func (x *RestoreWorkloadRequest) GetResources() *ResourceRequirements { if x != nil { - return x.CpuMilli - } - return 0 -} - -func (x *RestoreWorkloadRequest) GetMemoryBytes() int64 { - if x != nil { - return x.MemoryBytes + return x.Resources } - return 0 + return nil } func (x *RestoreWorkloadRequest) GetActorDirs() *ActorDirs { @@ -1607,7 +1640,7 @@ type RestoreWorkloadResponse struct { func (x *RestoreWorkloadResponse) Reset() { *x = RestoreWorkloadResponse{} - mi := &file_ateworker_proto_msgTypes[16] + mi := &file_ateworker_proto_msgTypes[17] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1619,7 +1652,7 @@ func (x *RestoreWorkloadResponse) String() string { func (*RestoreWorkloadResponse) ProtoMessage() {} func (x *RestoreWorkloadResponse) ProtoReflect() protoreflect.Message { - mi := &file_ateworker_proto_msgTypes[16] + mi := &file_ateworker_proto_msgTypes[17] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1632,7 +1665,7 @@ func (x *RestoreWorkloadResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use RestoreWorkloadResponse.ProtoReflect.Descriptor instead. func (*RestoreWorkloadResponse) Descriptor() ([]byte, []int) { - return file_ateworker_proto_rawDescGZIP(), []int{16} + return file_ateworker_proto_rawDescGZIP(), []int{17} } type GetWorkloadStatsRequest struct { @@ -1651,7 +1684,7 @@ type GetWorkloadStatsRequest struct { func (x *GetWorkloadStatsRequest) Reset() { *x = GetWorkloadStatsRequest{} - mi := &file_ateworker_proto_msgTypes[17] + mi := &file_ateworker_proto_msgTypes[18] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1663,7 +1696,7 @@ func (x *GetWorkloadStatsRequest) String() string { func (*GetWorkloadStatsRequest) ProtoMessage() {} func (x *GetWorkloadStatsRequest) ProtoReflect() protoreflect.Message { - mi := &file_ateworker_proto_msgTypes[17] + mi := &file_ateworker_proto_msgTypes[18] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1676,7 +1709,7 @@ func (x *GetWorkloadStatsRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use GetWorkloadStatsRequest.ProtoReflect.Descriptor instead. func (*GetWorkloadStatsRequest) Descriptor() ([]byte, []int) { - return file_ateworker_proto_rawDescGZIP(), []int{17} + return file_ateworker_proto_rawDescGZIP(), []int{18} } func (x *GetWorkloadStatsRequest) GetActorUid() string { @@ -1741,7 +1774,7 @@ type WorkloadStatsSample struct { func (x *WorkloadStatsSample) Reset() { *x = WorkloadStatsSample{} - mi := &file_ateworker_proto_msgTypes[18] + mi := &file_ateworker_proto_msgTypes[19] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1753,7 +1786,7 @@ func (x *WorkloadStatsSample) String() string { func (*WorkloadStatsSample) ProtoMessage() {} func (x *WorkloadStatsSample) ProtoReflect() protoreflect.Message { - mi := &file_ateworker_proto_msgTypes[18] + mi := &file_ateworker_proto_msgTypes[19] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1766,7 +1799,7 @@ func (x *WorkloadStatsSample) ProtoReflect() protoreflect.Message { // Deprecated: Use WorkloadStatsSample.ProtoReflect.Descriptor instead. func (*WorkloadStatsSample) Descriptor() ([]byte, []int) { - return file_ateworker_proto_rawDescGZIP(), []int{18} + return file_ateworker_proto_rawDescGZIP(), []int{19} } func (x *WorkloadStatsSample) GetAtespace() string { @@ -1869,7 +1902,7 @@ type GetWorkloadStatsResponse struct { func (x *GetWorkloadStatsResponse) Reset() { *x = GetWorkloadStatsResponse{} - mi := &file_ateworker_proto_msgTypes[19] + mi := &file_ateworker_proto_msgTypes[20] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1881,7 +1914,7 @@ func (x *GetWorkloadStatsResponse) String() string { func (*GetWorkloadStatsResponse) ProtoMessage() {} func (x *GetWorkloadStatsResponse) ProtoReflect() protoreflect.Message { - mi := &file_ateworker_proto_msgTypes[19] + mi := &file_ateworker_proto_msgTypes[20] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1894,7 +1927,7 @@ func (x *GetWorkloadStatsResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use GetWorkloadStatsResponse.ProtoReflect.Descriptor instead. func (*GetWorkloadStatsResponse) Descriptor() ([]byte, []int) { - return file_ateworker_proto_rawDescGZIP(), []int{19} + return file_ateworker_proto_rawDescGZIP(), []int{20} } func (x *GetWorkloadStatsResponse) GetSample() *WorkloadStatsSample { @@ -1912,7 +1945,7 @@ type GetActiveWorkloadStatsRequest struct { func (x *GetActiveWorkloadStatsRequest) Reset() { *x = GetActiveWorkloadStatsRequest{} - mi := &file_ateworker_proto_msgTypes[20] + mi := &file_ateworker_proto_msgTypes[21] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1924,7 +1957,7 @@ func (x *GetActiveWorkloadStatsRequest) String() string { func (*GetActiveWorkloadStatsRequest) ProtoMessage() {} func (x *GetActiveWorkloadStatsRequest) ProtoReflect() protoreflect.Message { - mi := &file_ateworker_proto_msgTypes[20] + mi := &file_ateworker_proto_msgTypes[21] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1937,7 +1970,7 @@ func (x *GetActiveWorkloadStatsRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use GetActiveWorkloadStatsRequest.ProtoReflect.Descriptor instead. func (*GetActiveWorkloadStatsRequest) Descriptor() ([]byte, []int) { - return file_ateworker_proto_rawDescGZIP(), []int{20} + return file_ateworker_proto_rawDescGZIP(), []int{21} } type GetActiveWorkloadStatsResponse struct { @@ -1957,7 +1990,7 @@ type GetActiveWorkloadStatsResponse struct { func (x *GetActiveWorkloadStatsResponse) Reset() { *x = GetActiveWorkloadStatsResponse{} - mi := &file_ateworker_proto_msgTypes[21] + mi := &file_ateworker_proto_msgTypes[22] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1969,7 +2002,7 @@ func (x *GetActiveWorkloadStatsResponse) String() string { func (*GetActiveWorkloadStatsResponse) ProtoMessage() {} func (x *GetActiveWorkloadStatsResponse) ProtoReflect() protoreflect.Message { - mi := &file_ateworker_proto_msgTypes[21] + mi := &file_ateworker_proto_msgTypes[22] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1982,7 +2015,7 @@ func (x *GetActiveWorkloadStatsResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use GetActiveWorkloadStatsResponse.ProtoReflect.Descriptor instead. func (*GetActiveWorkloadStatsResponse) Descriptor() ([]byte, []int) { - return file_ateworker_proto_rawDescGZIP(), []int{21} + return file_ateworker_proto_rawDescGZIP(), []int{22} } func (x *GetActiveWorkloadStatsResponse) GetSamples() []*WorkloadStatsSample { @@ -2019,7 +2052,7 @@ const file_ateworker_proto_rawDesc = "" + "\x04spec\x18\a \x01(\v2\x17.ateworker.WorkloadSpecR\x04spec\x123\n" + "\n" + "actor_dirs\x18\b \x01(\v2\x14.ateworker.ActorDirsR\tactorDirs\"\x1b\n" + - "\x19TerminateWorkloadResponse\"\x9a\x05\n" + + "\x19TerminateWorkloadResponse\"\x99\x05\n" + "\x12RunWorkloadRequest\x12\x1a\n" + "\batespace\x18\x01 \x01(\tR\batespace\x12\x1d\n" + "\n" + @@ -2031,10 +2064,9 @@ const file_ateworker_proto_rawDesc = "" + "runsc_path\x18\x06 \x01(\tR\trunscPath\x12+\n" + "\x04spec\x18\a \x01(\v2\x17.ateworker.WorkloadSpecR\x04spec\x12d\n" + "\x13runtime_asset_paths\x18\b \x03(\v24.ateworker.RunWorkloadRequest.RuntimeAssetPathsEntryR\x11runtimeAssetPaths\x12D\n" + - "\x0eegress_gateway\x18\t \x01(\v2\x18.ateworker.EgressGatewayH\x00R\regressGateway\x88\x01\x01\x12\x1b\n" + - "\tcpu_milli\x18\n" + - " \x01(\x03R\bcpuMilli\x12!\n" + - "\fmemory_bytes\x18\v \x01(\x03R\vmemoryBytes\x123\n" + + "\x0eegress_gateway\x18\t \x01(\v2\x18.ateworker.EgressGatewayH\x00R\regressGateway\x88\x01\x01\x12=\n" + + "\tresources\x18\n" + + " \x01(\v2\x1f.ateworker.ResourceRequirementsR\tresources\x123\n" + "\n" + "actor_dirs\x18\f \x01(\v2\x14.ateworker.ActorDirsR\tactorDirs\x1aD\n" + "\x16RuntimeAssetPathsEntry\x12\x10\n" + @@ -2054,16 +2086,17 @@ const file_ateworker_proto_rawDesc = "" + "\x16external_volume_mounts\x18\x04 \x03(\v2\x16.ateworker.VolumeMountR\x14externalVolumeMounts\x12Q\n" + "\x19system_info_volume_mounts\x18\x05 \x03(\v2\x16.ateworker.VolumeMountR\x16systemInfoVolumeMounts\x12F\n" + "\x13image_volume_mounts\x18\x06 \x03(\v2\x16.ateworker.VolumeMountR\x11imageVolumeMounts\x12?\n" + - "\x0econtainer_spec\x18\a \x01(\v2\x18.ateworker.ContainerSpecR\rcontainerSpec\"\x97\x01\n" + + "\x0econtainer_spec\x18\a \x01(\v2\x18.ateworker.ContainerSpecR\rcontainerSpec\"\x9d\x01\n" + "\rContainerSpec\x12\x12\n" + "\x04args\x18\x01 \x03(\tR\x04args\x12\x15\n" + "\x03env\x18\x02 \x03(\tB\x03\x80\x01\x01R\x03env\x12\"\n" + - "\fcapabilities\x18\x03 \x03(\tR\fcapabilities\x127\n" + - "\tresources\x18\x04 \x01(\v2\x19.ateworker.ResourceLimitsR\tresources\"R\n" + - "\x0eResourceLimits\x12!\n" + - "\fmemory_bytes\x18\x01 \x01(\x03R\vmemoryBytes\x12\x1d\n" + - "\n" + - "cpu_millis\x18\x02 \x01(\x03R\tcpuMillis\"@\n" + + "\fcapabilities\x18\x03 \x03(\tR\fcapabilities\x12=\n" + + "\tresources\x18\x04 \x01(\v2\x1f.ateworker.ResourceRequirementsR\tresources\"D\n" + + "\x14ResourceRequirements\x12,\n" + + "\x06limits\x18\x01 \x01(\v2\x14.ateworker.ResourcesR\x06limits\"U\n" + + "\tResources\x12%\n" + + "\x0ecpu_millicores\x18\x01 \x01(\x03R\rcpuMillicores\x12!\n" + + "\fmemory_bytes\x18\x02 \x01(\x03R\vmemoryBytes\"@\n" + "\vVolumeMount\x12\x12\n" + "\x04name\x18\x01 \x01(\tR\x04name\x12\x1d\n" + "\n" + @@ -2090,7 +2123,7 @@ const file_ateworker_proto_rawDesc = "" + "actor_dirs\x18\t \x01(\v2\x14.ateworker.ActorDirsR\tactorDirs\"s\n" + "\x1aCheckpointWorkloadResponse\x12%\n" + "\x0esnapshot_files\x18\x01 \x03(\tR\rsnapshotFiles\x12.\n" + - "\x13data_snapshot_files\x18\x02 \x03(\tR\x11dataSnapshotFiles\"\x8d\x06\n" + + "\x13data_snapshot_files\x18\x02 \x03(\tR\x11dataSnapshotFiles\"\x8c\x06\n" + "\x16RestoreWorkloadRequest\x12\x1a\n" + "\batespace\x18\x01 \x01(\tR\batespace\x12\x1d\n" + "\n" + @@ -2104,9 +2137,8 @@ const file_ateworker_proto_rawDesc = "" + "\x13runtime_asset_paths\x18\b \x03(\v28.ateworker.RestoreWorkloadRequest.RuntimeAssetPathsEntryR\x11runtimeAssetPaths\x127\n" + "\bfidelity\x18\t \x01(\x0e2\x1b.ateworker.SnapshotFidelityR\bfidelity\x12D\n" + "\x0eegress_gateway\x18\n" + - " \x01(\v2\x18.ateworker.EgressGatewayH\x00R\regressGateway\x88\x01\x01\x12\x1b\n" + - "\tcpu_milli\x18\v \x01(\x03R\bcpuMilli\x12!\n" + - "\fmemory_bytes\x18\f \x01(\x03R\vmemoryBytes\x123\n" + + " \x01(\v2\x18.ateworker.EgressGatewayH\x00R\regressGateway\x88\x01\x01\x12=\n" + + "\tresources\x18\v \x01(\v2\x1f.ateworker.ResourceRequirementsR\tresources\x123\n" + "\n" + "actor_dirs\x18\r \x01(\v2\x14.ateworker.ActorDirsR\tactorDirs\x120\n" + "\x14preserve_restore_dir\x18\x0e \x01(\bR\x12preserveRestoreDir\x1aD\n" + @@ -2168,7 +2200,7 @@ func file_ateworker_proto_rawDescGZIP() []byte { } var file_ateworker_proto_enumTypes = make([]protoimpl.EnumInfo, 2) -var file_ateworker_proto_msgTypes = make([]protoimpl.MessageInfo, 24) +var file_ateworker_proto_msgTypes = make([]protoimpl.MessageInfo, 25) var file_ateworker_proto_goTypes = []any{ (SnapshotFidelity)(0), // 0: ateworker.SnapshotFidelity (StatsSource)(0), // 1: ateworker.StatsSource @@ -2180,67 +2212,71 @@ var file_ateworker_proto_goTypes = []any{ (*WorkloadSpec)(nil), // 7: ateworker.WorkloadSpec (*Container)(nil), // 8: ateworker.Container (*ContainerSpec)(nil), // 9: ateworker.ContainerSpec - (*ResourceLimits)(nil), // 10: ateworker.ResourceLimits - (*VolumeMount)(nil), // 11: ateworker.VolumeMount - (*WakeupProbe)(nil), // 12: ateworker.WakeupProbe - (*HttpGetAction)(nil), // 13: ateworker.HttpGetAction - (*RunWorkloadResponse)(nil), // 14: ateworker.RunWorkloadResponse - (*CheckpointWorkloadRequest)(nil), // 15: ateworker.CheckpointWorkloadRequest - (*CheckpointWorkloadResponse)(nil), // 16: ateworker.CheckpointWorkloadResponse - (*RestoreWorkloadRequest)(nil), // 17: ateworker.RestoreWorkloadRequest - (*RestoreWorkloadResponse)(nil), // 18: ateworker.RestoreWorkloadResponse - (*GetWorkloadStatsRequest)(nil), // 19: ateworker.GetWorkloadStatsRequest - (*WorkloadStatsSample)(nil), // 20: ateworker.WorkloadStatsSample - (*GetWorkloadStatsResponse)(nil), // 21: ateworker.GetWorkloadStatsResponse - (*GetActiveWorkloadStatsRequest)(nil), // 22: ateworker.GetActiveWorkloadStatsRequest - (*GetActiveWorkloadStatsResponse)(nil), // 23: ateworker.GetActiveWorkloadStatsResponse - nil, // 24: ateworker.RunWorkloadRequest.RuntimeAssetPathsEntry - nil, // 25: ateworker.RestoreWorkloadRequest.RuntimeAssetPathsEntry + (*ResourceRequirements)(nil), // 10: ateworker.ResourceRequirements + (*Resources)(nil), // 11: ateworker.Resources + (*VolumeMount)(nil), // 12: ateworker.VolumeMount + (*WakeupProbe)(nil), // 13: ateworker.WakeupProbe + (*HttpGetAction)(nil), // 14: ateworker.HttpGetAction + (*RunWorkloadResponse)(nil), // 15: ateworker.RunWorkloadResponse + (*CheckpointWorkloadRequest)(nil), // 16: ateworker.CheckpointWorkloadRequest + (*CheckpointWorkloadResponse)(nil), // 17: ateworker.CheckpointWorkloadResponse + (*RestoreWorkloadRequest)(nil), // 18: ateworker.RestoreWorkloadRequest + (*RestoreWorkloadResponse)(nil), // 19: ateworker.RestoreWorkloadResponse + (*GetWorkloadStatsRequest)(nil), // 20: ateworker.GetWorkloadStatsRequest + (*WorkloadStatsSample)(nil), // 21: ateworker.WorkloadStatsSample + (*GetWorkloadStatsResponse)(nil), // 22: ateworker.GetWorkloadStatsResponse + (*GetActiveWorkloadStatsRequest)(nil), // 23: ateworker.GetActiveWorkloadStatsRequest + (*GetActiveWorkloadStatsResponse)(nil), // 24: ateworker.GetActiveWorkloadStatsResponse + nil, // 25: ateworker.RunWorkloadRequest.RuntimeAssetPathsEntry + nil, // 26: ateworker.RestoreWorkloadRequest.RuntimeAssetPathsEntry } var file_ateworker_proto_depIdxs = []int32{ 7, // 0: ateworker.TerminateWorkloadRequest.spec:type_name -> ateworker.WorkloadSpec 2, // 1: ateworker.TerminateWorkloadRequest.actor_dirs:type_name -> ateworker.ActorDirs 7, // 2: ateworker.RunWorkloadRequest.spec:type_name -> ateworker.WorkloadSpec - 24, // 3: ateworker.RunWorkloadRequest.runtime_asset_paths:type_name -> ateworker.RunWorkloadRequest.RuntimeAssetPathsEntry + 25, // 3: ateworker.RunWorkloadRequest.runtime_asset_paths:type_name -> ateworker.RunWorkloadRequest.RuntimeAssetPathsEntry 6, // 4: ateworker.RunWorkloadRequest.egress_gateway:type_name -> ateworker.EgressGateway - 2, // 5: ateworker.RunWorkloadRequest.actor_dirs:type_name -> ateworker.ActorDirs - 8, // 6: ateworker.WorkloadSpec.containers:type_name -> ateworker.Container - 12, // 7: ateworker.Container.wakeup_probe:type_name -> ateworker.WakeupProbe - 11, // 8: ateworker.Container.durable_dir_volume_mounts:type_name -> ateworker.VolumeMount - 11, // 9: ateworker.Container.external_volume_mounts:type_name -> ateworker.VolumeMount - 11, // 10: ateworker.Container.system_info_volume_mounts:type_name -> ateworker.VolumeMount - 11, // 11: ateworker.Container.image_volume_mounts:type_name -> ateworker.VolumeMount - 9, // 12: ateworker.Container.container_spec:type_name -> ateworker.ContainerSpec - 10, // 13: ateworker.ContainerSpec.resources:type_name -> ateworker.ResourceLimits - 13, // 14: ateworker.WakeupProbe.http_get:type_name -> ateworker.HttpGetAction - 7, // 15: ateworker.CheckpointWorkloadRequest.spec:type_name -> ateworker.WorkloadSpec - 0, // 16: ateworker.CheckpointWorkloadRequest.fidelity:type_name -> ateworker.SnapshotFidelity - 2, // 17: ateworker.CheckpointWorkloadRequest.actor_dirs:type_name -> ateworker.ActorDirs - 7, // 18: ateworker.RestoreWorkloadRequest.spec:type_name -> ateworker.WorkloadSpec - 25, // 19: ateworker.RestoreWorkloadRequest.runtime_asset_paths:type_name -> ateworker.RestoreWorkloadRequest.RuntimeAssetPathsEntry - 0, // 20: ateworker.RestoreWorkloadRequest.fidelity:type_name -> ateworker.SnapshotFidelity - 6, // 21: ateworker.RestoreWorkloadRequest.egress_gateway:type_name -> ateworker.EgressGateway - 2, // 22: ateworker.RestoreWorkloadRequest.actor_dirs:type_name -> ateworker.ActorDirs - 1, // 23: ateworker.WorkloadStatsSample.source:type_name -> ateworker.StatsSource - 20, // 24: ateworker.GetWorkloadStatsResponse.sample:type_name -> ateworker.WorkloadStatsSample - 20, // 25: ateworker.GetActiveWorkloadStatsResponse.samples:type_name -> ateworker.WorkloadStatsSample - 5, // 26: ateworker.Worker.RunWorkload:input_type -> ateworker.RunWorkloadRequest - 15, // 27: ateworker.Worker.CheckpointWorkload:input_type -> ateworker.CheckpointWorkloadRequest - 17, // 28: ateworker.Worker.RestoreWorkload:input_type -> ateworker.RestoreWorkloadRequest - 19, // 29: ateworker.Worker.GetWorkloadStats:input_type -> ateworker.GetWorkloadStatsRequest - 22, // 30: ateworker.Worker.GetActiveWorkloadStats:input_type -> ateworker.GetActiveWorkloadStatsRequest - 3, // 31: ateworker.Worker.TerminateWorkload:input_type -> ateworker.TerminateWorkloadRequest - 14, // 32: ateworker.Worker.RunWorkload:output_type -> ateworker.RunWorkloadResponse - 16, // 33: ateworker.Worker.CheckpointWorkload:output_type -> ateworker.CheckpointWorkloadResponse - 18, // 34: ateworker.Worker.RestoreWorkload:output_type -> ateworker.RestoreWorkloadResponse - 21, // 35: ateworker.Worker.GetWorkloadStats:output_type -> ateworker.GetWorkloadStatsResponse - 23, // 36: ateworker.Worker.GetActiveWorkloadStats:output_type -> ateworker.GetActiveWorkloadStatsResponse - 4, // 37: ateworker.Worker.TerminateWorkload:output_type -> ateworker.TerminateWorkloadResponse - 32, // [32:38] is the sub-list for method output_type - 26, // [26:32] is the sub-list for method input_type - 26, // [26:26] is the sub-list for extension type_name - 26, // [26:26] is the sub-list for extension extendee - 0, // [0:26] is the sub-list for field type_name + 10, // 5: ateworker.RunWorkloadRequest.resources:type_name -> ateworker.ResourceRequirements + 2, // 6: ateworker.RunWorkloadRequest.actor_dirs:type_name -> ateworker.ActorDirs + 8, // 7: ateworker.WorkloadSpec.containers:type_name -> ateworker.Container + 13, // 8: ateworker.Container.wakeup_probe:type_name -> ateworker.WakeupProbe + 12, // 9: ateworker.Container.durable_dir_volume_mounts:type_name -> ateworker.VolumeMount + 12, // 10: ateworker.Container.external_volume_mounts:type_name -> ateworker.VolumeMount + 12, // 11: ateworker.Container.system_info_volume_mounts:type_name -> ateworker.VolumeMount + 12, // 12: ateworker.Container.image_volume_mounts:type_name -> ateworker.VolumeMount + 9, // 13: ateworker.Container.container_spec:type_name -> ateworker.ContainerSpec + 10, // 14: ateworker.ContainerSpec.resources:type_name -> ateworker.ResourceRequirements + 11, // 15: ateworker.ResourceRequirements.limits:type_name -> ateworker.Resources + 14, // 16: ateworker.WakeupProbe.http_get:type_name -> ateworker.HttpGetAction + 7, // 17: ateworker.CheckpointWorkloadRequest.spec:type_name -> ateworker.WorkloadSpec + 0, // 18: ateworker.CheckpointWorkloadRequest.fidelity:type_name -> ateworker.SnapshotFidelity + 2, // 19: ateworker.CheckpointWorkloadRequest.actor_dirs:type_name -> ateworker.ActorDirs + 7, // 20: ateworker.RestoreWorkloadRequest.spec:type_name -> ateworker.WorkloadSpec + 26, // 21: ateworker.RestoreWorkloadRequest.runtime_asset_paths:type_name -> ateworker.RestoreWorkloadRequest.RuntimeAssetPathsEntry + 0, // 22: ateworker.RestoreWorkloadRequest.fidelity:type_name -> ateworker.SnapshotFidelity + 6, // 23: ateworker.RestoreWorkloadRequest.egress_gateway:type_name -> ateworker.EgressGateway + 10, // 24: ateworker.RestoreWorkloadRequest.resources:type_name -> ateworker.ResourceRequirements + 2, // 25: ateworker.RestoreWorkloadRequest.actor_dirs:type_name -> ateworker.ActorDirs + 1, // 26: ateworker.WorkloadStatsSample.source:type_name -> ateworker.StatsSource + 21, // 27: ateworker.GetWorkloadStatsResponse.sample:type_name -> ateworker.WorkloadStatsSample + 21, // 28: ateworker.GetActiveWorkloadStatsResponse.samples:type_name -> ateworker.WorkloadStatsSample + 5, // 29: ateworker.Worker.RunWorkload:input_type -> ateworker.RunWorkloadRequest + 16, // 30: ateworker.Worker.CheckpointWorkload:input_type -> ateworker.CheckpointWorkloadRequest + 18, // 31: ateworker.Worker.RestoreWorkload:input_type -> ateworker.RestoreWorkloadRequest + 20, // 32: ateworker.Worker.GetWorkloadStats:input_type -> ateworker.GetWorkloadStatsRequest + 23, // 33: ateworker.Worker.GetActiveWorkloadStats:input_type -> ateworker.GetActiveWorkloadStatsRequest + 3, // 34: ateworker.Worker.TerminateWorkload:input_type -> ateworker.TerminateWorkloadRequest + 15, // 35: ateworker.Worker.RunWorkload:output_type -> ateworker.RunWorkloadResponse + 17, // 36: ateworker.Worker.CheckpointWorkload:output_type -> ateworker.CheckpointWorkloadResponse + 19, // 37: ateworker.Worker.RestoreWorkload:output_type -> ateworker.RestoreWorkloadResponse + 22, // 38: ateworker.Worker.GetWorkloadStats:output_type -> ateworker.GetWorkloadStatsResponse + 24, // 39: ateworker.Worker.GetActiveWorkloadStats:output_type -> ateworker.GetActiveWorkloadStatsResponse + 4, // 40: ateworker.Worker.TerminateWorkload:output_type -> ateworker.TerminateWorkloadResponse + 35, // [35:41] is the sub-list for method output_type + 29, // [29:35] is the sub-list for method input_type + 29, // [29:29] is the sub-list for extension type_name + 29, // [29:29] is the sub-list for extension extendee + 0, // [0:29] is the sub-list for field type_name } func init() { file_ateworker_proto_init() } @@ -2249,14 +2285,14 @@ func file_ateworker_proto_init() { return } file_ateworker_proto_msgTypes[3].OneofWrappers = []any{} - file_ateworker_proto_msgTypes[15].OneofWrappers = []any{} + file_ateworker_proto_msgTypes[16].OneofWrappers = []any{} type x struct{} out := protoimpl.TypeBuilder{ File: protoimpl.DescBuilder{ GoPackagePath: reflect.TypeOf(x{}).PkgPath(), RawDescriptor: unsafe.Slice(unsafe.StringData(file_ateworker_proto_rawDesc), len(file_ateworker_proto_rawDesc)), NumEnums: 2, - NumMessages: 24, + NumMessages: 25, NumExtensions: 0, NumServices: 1, }, diff --git a/internal/proto/ateworkerpb/ateworker.proto b/internal/proto/ateworkerpb/ateworker.proto index 8b2e32abdc..b31556c3d7 100644 --- a/internal/proto/ateworkerpb/ateworker.proto +++ b/internal/proto/ateworkerpb/ateworker.proto @@ -240,12 +240,7 @@ message RunWorkloadRequest { // runtime default (unlimited for gVisor, the worker's own default for the micro-VM). // // +k8s:optional - // +k8s:minimum=1 - // +k8s:maximum=999999 # matches atelet.RunRequest.cpu_milli, the control plane caps cpu limits strictly below 1000 cores - int64 cpu_milli = 10; // CPU limit in millicores (1000 = one core). - // +k8s:optional - // +k8s:minimum=1 - int64 memory_bytes = 11; // Memory limit in bytes. + ResourceRequirements resources = 10; // +k8s:required ActorDirs actor_dirs = 12; } @@ -349,15 +344,33 @@ message ContainerSpec { repeated string capabilities = 3; // resources are the container's own declared limits, if any. - ResourceLimits resources = 4; + // + // +k8s:optional + ResourceRequirements resources = 4; } -// ResourceLimits are a container's declared limits. -message ResourceLimits { - // memory_bytes is the memory limit in bytes. 0 means unset. - int64 memory_bytes = 1; - // cpu_millis is the CPU limit in milli-cores (1000 = one core). 0 means unset. - int64 cpu_millis = 2; +// ResourceRequirements mirrors atelet's ResourceRequirements. +message ResourceRequirements { + // limits is the maximum amount of compute resources allowed. + // + // +k8s:optional + Resources limits = 1; +} + +// Resources is an amount of each compute resource. +message Resources { + // cpu_millicores is CPU in millicores (1000 = one core). 0 means none. + // + // +k8s:optional + // +k8s:minimum=0 + // +k8s:maximum=999999 # less than 1000 cores + int64 cpu_millicores = 1; + + // memory_bytes is memory in bytes. 0 means none. + // + // +k8s:optional + // +k8s:minimum=0 + int64 memory_bytes = 2; } // VolumeMount is one volume mounted into a container. @@ -529,12 +542,7 @@ message RestoreWorkloadRequest { // these are ignored. Zero means "unset": keep the runtime default. // // +k8s:optional - // +k8s:minimum=1 - // +k8s:maximum=999999 # matches atelet.RestoreRequest.cpu_milli, the control plane caps cpu limits strictly below 1000 cores - int64 cpu_milli = 11; // CPU limit in millicores (1000 = one core). - // +k8s:optional - // +k8s:minimum=1 - int64 memory_bytes = 12; // Memory limit in bytes. + ResourceRequirements resources = 11; // +k8s:required ActorDirs actor_dirs = 13; diff --git a/internal/resources/compute.go b/internal/resources/compute.go new file mode 100644 index 0000000000..4a854ede25 --- /dev/null +++ b/internal/resources/compute.go @@ -0,0 +1,40 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package resources + +import ( + "go.ate.dev/substrate/pkg/proto/ateapipb" +) + +// The same ateapipb.Resources is what an Actor's limits ask for, what a Worker +// supplies, and the sum of what a Worker's Actors hold, so they subtract. +// +// A zero dimension is none of that resource. A Worker reports every dimension +// it has, so a zero in its capacity is one it cannot supply at all. + +// CPUMemory returns the Resources for cpuMilli and memoryBytes, or nil when +// both are zero, so an emptied total compares equal to an absent one. +func CPUMemory(cpuMilli, memoryBytes int64) *ateapipb.Resources { + if cpuMilli == 0 && memoryBytes == 0 { + return nil + } + return &ateapipb.Resources{CpuMillicores: cpuMilli, MemoryBytes: memoryBytes} +} + +// Covers reports whether have leaves room for want in every dimension. A +// dimension want leaves at zero asks for nothing. +func Covers(have, want *ateapipb.Resources) bool { + return have.GetCpuMillicores() >= want.GetCpuMillicores() && have.GetMemoryBytes() >= want.GetMemoryBytes() +} diff --git a/internal/resources/compute_test.go b/internal/resources/compute_test.go new file mode 100644 index 0000000000..c6aa605015 --- /dev/null +++ b/internal/resources/compute_test.go @@ -0,0 +1,78 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package resources + +import ( + "testing" + + "go.ate.dev/substrate/pkg/proto/ateapipb" +) + +func TestCovers(t *testing.T) { + tests := []struct { + name string + have *ateapipb.Resources + want *ateapipb.Resources + ok bool + }{ + { + name: "enough of every dimension", + have: CPUMemory(4000, 8<<30), + want: CPUMemory(2000, 4<<30), + ok: true, + }, + { + name: "exactly enough", + have: CPUMemory(2000, 0), + want: CPUMemory(2000, 0), + ok: true, + }, + { + name: "short in one dimension", + have: CPUMemory(4000, 1<<30), + want: CPUMemory(2000, 4<<30), + ok: false, + }, + { + // A Worker reports everything it has, so a dimension it left at + // zero is one it has none of. + name: "a dimension the worker never reported is none of it", + have: CPUMemory(4000, 0), + want: CPUMemory(2000, 1<<20), + ok: false, + }, + { + name: "an actor asking for nothing fits anywhere", + have: nil, + want: nil, + ok: true, + }, + { + // What is left goes negative once a Worker is overcommitted, and + // nothing more may be placed on it. + name: "an overcommitted dimension covers nothing", + have: CPUMemory(-500, 0), + want: CPUMemory(1000, 0), + ok: false, + }, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + if got := Covers(tc.have, tc.want); got != tc.ok { + t.Errorf("Covers() = %v, want %v", got, tc.ok) + } + }) + } +} diff --git a/internal/resources/quantities.go b/internal/resources/quantities.go deleted file mode 100644 index 76c97362f7..0000000000 --- a/internal/resources/quantities.go +++ /dev/null @@ -1,149 +0,0 @@ -// Copyright 2026 Google LLC -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package resources - -import ( - "fmt" - "sort" - - "go.ate.dev/substrate/pkg/proto/ateapipb" - "k8s.io/apimachinery/pkg/api/resource" -) - -// Quantities is the parsed form of an ateapipb.Resources: what an Actor asks -// for, what a Worker supplies, or the sum of what a Worker's Actors hold. The -// same type serves all three so they subtract, which is the point of naming -// Worker capacity the way an ActorTemplate names its limits. -// -// An absent name is none of that resource. A Worker reports every dimension it -// has, so a name missing from its capacity is one it cannot supply at all. -type Quantities map[string]resource.Quantity - -// ParseQuantities reads the wire form. It errors on a quantity it cannot parse -// rather than skipping it, so a malformed limit cannot silently become -// unconstrained. -func ParseQuantities(r *ateapipb.Resources) (Quantities, error) { - if len(r.GetLimits()) == 0 { - return nil, nil - } - out := make(Quantities, len(r.GetLimits())) - for _, limit := range r.GetLimits() { - q, err := resource.ParseQuantity(limit.GetQuantity()) - if err != nil { - return nil, fmt.Errorf("resource %s has an invalid quantity %q: %w", limit.GetName(), limit.GetQuantity(), err) - } - if existing, ok := out[limit.GetName()]; ok { - existing.Add(q) - out[limit.GetName()] = existing - continue - } - out[limit.GetName()] = q - } - return out, nil -} - -// Proto is the wire form, sorted by name. Sorting is what lets proto.Equal -// decide whether a report or a recomputed total actually changed anything; -// unsorted, equal sets would compare unequal and churn the record. -// -// A dimension that has reached zero is dropped: it constrains nothing, and -// keeping it would make an emptied total compare unequal to an absent one. -func (q Quantities) Proto() *ateapipb.Resources { - if len(q) == 0 { - return nil - } - names := make([]string, 0, len(q)) - for name, quantity := range q { - if quantity.IsZero() { - continue - } - names = append(names, name) - } - if len(names) == 0 { - return nil - } - sort.Strings(names) - out := &ateapipb.Resources{Limits: make([]*ateapipb.Limits, 0, len(names))} - for _, name := range names { - quantity := q[name] - out.Limits = append(out.Limits, &ateapipb.Limits{Name: name, Quantity: quantity.String()}) - } - return out -} - -// ResourceCPU and ResourceMemory are the two dimensions everything declares -// today, named as Kubernetes names them so an ActorTemplate's limits and a -// Worker's capacity meet under the same keys. -const ( - ResourceCPU = "cpu" - ResourceMemory = "memory" -) - -// CPUMemory is the Resources for those two dimensions, in the units the -// runtimes deal in. A zero dimension is omitted, which reads as unconstrained. -func CPUMemory(cpuMilli, memoryBytes int64) *ateapipb.Resources { - q := Quantities{} - if cpuMilli != 0 { - q[ResourceCPU] = *resource.NewMilliQuantity(cpuMilli, resource.DecimalSI) - } - if memoryBytes != 0 { - q[ResourceMemory] = *resource.NewQuantity(memoryBytes, resource.BinarySI) - } - return q.Proto() -} - -// Add adds other into q, dimension by dimension. -func (q Quantities) Add(other Quantities) { - for name, quantity := range other { - existing, ok := q[name] - if !ok { - q[name] = quantity.DeepCopy() - continue - } - existing.Add(quantity) - q[name] = existing - } -} - -// Sub subtracts other from q, dimension by dimension. -func (q Quantities) Sub(other Quantities) { - for name, quantity := range other { - existing, ok := q[name] - if !ok { - neg := quantity.DeepCopy() - neg.Neg() - q[name] = neg - continue - } - existing.Sub(quantity) - q[name] = existing - } -} - -// Covers reports whether q leaves room for want in every dimension want names. -// -// A dimension q does not name is none of it, not any amount of it: a Worker -// reports everything it has, so silence about GPUs means it has no GPUs and -// cannot take an Actor asking for one. A dimension want does not name asks for -// nothing. -func (q Quantities) Covers(want Quantities) bool { - for name, need := range want { - have := q[name] // absent reads as the zero quantity - if have.Cmp(need) < 0 { - return false - } - } - return true -} diff --git a/internal/resources/quantities_test.go b/internal/resources/quantities_test.go deleted file mode 100644 index c141566259..0000000000 --- a/internal/resources/quantities_test.go +++ /dev/null @@ -1,177 +0,0 @@ -// Copyright 2026 Google LLC -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package resources - -import ( - "testing" - - "go.ate.dev/substrate/pkg/proto/ateapipb" - "google.golang.org/protobuf/proto" - "k8s.io/apimachinery/pkg/api/resource" -) - -func limits(pairs ...string) *ateapipb.Resources { - out := &ateapipb.Resources{} - for i := 0; i < len(pairs); i += 2 { - out.Limits = append(out.Limits, &ateapipb.Limits{Name: pairs[i], Quantity: pairs[i+1]}) - } - return out -} - -// The whole reason Proto sorts. Capacity is compared with proto.Equal to decide -// whether a report changed anything, and a repeated field compares positionally: -// unsorted, two equal sets would look different and write on every report. -func TestProtoIsOrderIndependent(t *testing.T) { - a, err := ParseQuantities(limits("cpu", "2", "memory", "8Gi")) - if err != nil { - t.Fatalf("ParseQuantities: %v", err) - } - b, err := ParseQuantities(limits("memory", "8Gi", "cpu", "2")) - if err != nil { - t.Fatalf("ParseQuantities: %v", err) - } - if !proto.Equal(a.Proto(), b.Proto()) { - t.Errorf("the same set in two orders round-trips unequal:\n%v\n%v", a.Proto(), b.Proto()) - } -} - -// A dimension that reaches zero is dropped, so a Worker whose Actors all left -// compares equal to one that never held any. -func TestProtoDropsEmptyDimensions(t *testing.T) { - q := Quantities{"cpu": resource.MustParse("0")} - if got := q.Proto(); got != nil { - t.Errorf("Proto() = %v for an all-zero set, want nil", got) - } - if got := (Quantities{}).Proto(); got != nil { - t.Errorf("Proto() = %v for an empty set, want nil", got) - } -} - -func TestParseQuantities(t *testing.T) { - t.Run("an unparseable quantity is an error, not a skip", func(t *testing.T) { - if _, err := ParseQuantities(limits("cpu", "banana")); err == nil { - t.Error("ParseQuantities() = nil error, want one") - } - }) - - t.Run("nothing declared parses to nothing", func(t *testing.T) { - got, err := ParseQuantities(nil) - if err != nil || got != nil { - t.Errorf("ParseQuantities(nil) = %v, %v, want nil, nil", got, err) - } - }) - - t.Run("a repeated name sums rather than shadowing", func(t *testing.T) { - got, err := ParseQuantities(limits("cpu", "1", "cpu", "500m")) - if err != nil { - t.Fatalf("ParseQuantities: %v", err) - } - cpu := got["cpu"] - if want := resource.MustParse("1500m"); cpu.Cmp(want) != 0 { - t.Errorf("cpu = %v, want %v", cpu, want) - } - }) -} - -func TestQuantitiesAddAndSub(t *testing.T) { - q := Quantities{} - q.Add(Quantities{"cpu": resource.MustParse("2"), "memory": resource.MustParse("4Gi")}) - q.Add(Quantities{"cpu": resource.MustParse("1")}) - cpu := q["cpu"] - if want := resource.MustParse("3"); cpu.Cmp(want) != 0 { - t.Errorf("cpu after adds = %v, want %v", cpu, want) - } - - q.Sub(Quantities{"cpu": resource.MustParse("3"), "memory": resource.MustParse("4Gi")}) - if got := q.Proto(); got != nil { - t.Errorf("Proto() = %v after subtracting everything back out, want nil", got) - } -} - -func TestQuantitiesCovers(t *testing.T) { - tests := []struct { - name string - have Quantities - want Quantities - ok bool - }{ - { - name: "enough of every dimension", - have: Quantities{"cpu": resource.MustParse("4"), "memory": resource.MustParse("8Gi")}, - want: Quantities{"cpu": resource.MustParse("2"), "memory": resource.MustParse("4Gi")}, - ok: true, - }, - { - name: "exactly enough", - have: Quantities{"cpu": resource.MustParse("2")}, - want: Quantities{"cpu": resource.MustParse("2")}, - ok: true, - }, - { - name: "short in one dimension", - have: Quantities{"cpu": resource.MustParse("4"), "memory": resource.MustParse("1Gi")}, - want: Quantities{"cpu": resource.MustParse("2"), "memory": resource.MustParse("4Gi")}, - ok: false, - }, - { - // A Worker reports everything it has, so a dimension it never - // reported is one it has none of. Asking for a GPU must not land on - // a Worker that never said it had one. - name: "a dimension the worker never reported is none of it", - have: Quantities{"cpu": resource.MustParse("4")}, - want: Quantities{"cpu": resource.MustParse("2"), "nvidia.com/gpu": resource.MustParse("1")}, - ok: false, - }, - { - // Asking for zero of something absent is still satisfiable. - name: "asking for none of an absent dimension fits", - have: Quantities{"cpu": resource.MustParse("4")}, - want: Quantities{"nvidia.com/gpu": resource.MustParse("0")}, - ok: true, - }, - { - name: "an actor asking for nothing fits anywhere", - have: Quantities{"cpu": resource.MustParse("0")}, - want: nil, - ok: true, - }, - { - // What is left goes negative once a Worker is overcommitted, and - // nothing more may be placed on it. - name: "an overcommitted dimension covers nothing", - have: Quantities{"cpu": *resource.NewMilliQuantity(-500, resource.DecimalSI)}, - want: Quantities{"cpu": resource.MustParse("1")}, - ok: false, - }, - } - for _, tc := range tests { - t.Run(tc.name, func(t *testing.T) { - if got := tc.have.Covers(tc.want); got != tc.ok { - t.Errorf("Covers() = %v, want %v", got, tc.ok) - } - }) - } -} - -func TestCPUMemory(t *testing.T) { - got := CPUMemory(2500, 8<<30) - want := limits("cpu", "2500m", "memory", "8Gi") - if !proto.Equal(got, want) { - t.Errorf("CPUMemory() = %v, want %v", got, want) - } - if got := CPUMemory(0, 0); got != nil { - t.Errorf("CPUMemory(0, 0) = %v, want nil", got) - } -} diff --git a/internal/resources/validate.go b/internal/resources/validate.go index b1f7ebf760..ccf5b58b31 100644 --- a/internal/resources/validate.go +++ b/internal/resources/validate.go @@ -35,7 +35,6 @@ import ( "google.golang.org/grpc/codes" "google.golang.org/grpc/status" "google.golang.org/protobuf/proto" - "k8s.io/apimachinery/pkg/api/resource" "k8s.io/apimachinery/pkg/api/validate/content" "k8s.io/apimachinery/pkg/util/validation" "k8s.io/apimachinery/pkg/util/validation/field" @@ -299,45 +298,6 @@ func ValidateUUID(uuid string, fldPath *field.Path) field.ErrorList { return nil } -// cpuLimitMax bounds cpu limits. The wire carries ceil(cores × 1000) -// millicores and caps it at 999999 (strictly below 1000 cores), so a limit -// above 999.999 cores rounds up out of range. -var cpuLimitMax = resource.MustParse("999.999") - -// memoryLimitMax bounds memory limits at what the wire's int64 byte count -// holds; resource.Quantity.Value wraps silently past it. -var memoryLimitMax = resource.MustParse("9223372036854775807") - -// ValidateLimit validates one resource limit entry at fldPath: only cpu and -// memory are supported, the quantity must be greater than zero, and it must -// fit the scalars the control plane converts it to (cpuLimitMax and -// memoryLimitMax). The bounds compare Quantities rather than the converted -// integers, which wrap for very large values. An empty quantity is left to -// the required tag. -func ValidateLimit(fldPath *field.Path, name, quantity string) field.ErrorList { - if name != ResourceCPU && name != ResourceMemory { - return field.ErrorList{field.NotSupported(fldPath.Child("name"), name, []string{ResourceCPU, ResourceMemory})} - } - if quantity == "" { - return nil - } - q, err := resource.ParseQuantity(quantity) - if err != nil { - return field.ErrorList{field.Invalid(fldPath.Child("quantity"), quantity, fmt.Sprintf("must be a Kubernetes resource quantity: %v", err))} - } - var errs field.ErrorList - if q.Sign() <= 0 { - errs = append(errs, field.Invalid(fldPath.Child("quantity"), quantity, "must be greater than zero")) - } - switch { - case name == ResourceCPU && q.Cmp(cpuLimitMax) > 0: - errs = append(errs, field.Invalid(fldPath.Child("quantity"), quantity, "cpu limit must be at most 999.999 cores")) - case name == ResourceMemory && q.Cmp(memoryLimitMax) > 0: - errs = append(errs, field.Invalid(fldPath.Child("quantity"), quantity, "memory limit must be at most 9223372036854775807 bytes")) - } - return errs -} - // ValidatePinnedImage requires a well-formed OCI image reference pinned by // digest (e.g. "name@sha256:..."): changing the image content under a fixed // reference invalidates snapshots. It parses with the same grammar the diff --git a/internal/resources/validate_test.go b/internal/resources/validate_test.go index f9aa293f5b..aaa8ef0a59 100644 --- a/internal/resources/validate_test.go +++ b/internal/resources/validate_test.go @@ -277,38 +277,6 @@ func TestValidateSandboxClass(t *testing.T) { } } -func TestValidateLimit(t *testing.T) { - path := field.NewPath("limits").Index(0) - quantityPath := path.Child("quantity") - tests := []struct { - name string - limit string - quantity string - want field.ErrorList - }{ - {name: "cpu below the bound", limit: "cpu", quantity: "999"}, - {name: "cpu at the millicore bound", limit: "cpu", quantity: "999.999"}, - {name: "memory", limit: "memory", quantity: "1Gi"}, - {name: "memory at the int64 bound", limit: "memory", quantity: "9223372036854775807"}, - {name: "missing quantity left to tags", limit: "cpu"}, - {name: "unsupported name", limit: "gpu", quantity: "1", want: field.ErrorList{field.NotSupported[string](path.Child("name"), nil, nil)}}, - {name: "malformed quantity", limit: "cpu", quantity: "x", want: field.ErrorList{field.Invalid(quantityPath, nil, "")}}, - {name: "zero quantity", limit: "memory", quantity: "0", want: field.ErrorList{field.Invalid(quantityPath, nil, "")}}, - {name: "negative quantity", limit: "memory", quantity: "-1", want: field.ErrorList{field.Invalid(quantityPath, nil, "")}}, - {name: "cpu at 1000 cores", limit: "cpu", quantity: "1000", want: field.ErrorList{field.Invalid(quantityPath, nil, "")}}, - {name: "cpu rounds up to 1000 cores in millicores", limit: "cpu", quantity: "999.9995", want: field.ErrorList{field.Invalid(quantityPath, nil, "")}}, - {name: "cpu far beyond int64 millicores", limit: "cpu", quantity: "1e30", want: field.ErrorList{field.Invalid(quantityPath, nil, "")}}, - {name: "memory past the int64 bound", limit: "memory", quantity: "9223372036854775808", want: field.ErrorList{field.Invalid(quantityPath, nil, "")}}, - {name: "memory rounds up past the int64 bound", limit: "memory", quantity: "9223372036854775807.5", want: field.ErrorList{field.Invalid(quantityPath, nil, "")}}, - {name: "memory far beyond int64 bytes", limit: "memory", quantity: "1e30", want: field.ErrorList{field.Invalid(quantityPath, nil, "")}}, - } - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - field.ErrorMatcher{}.ByType().ByField().Test(t, tt.want, ValidateLimit(path, tt.limit, tt.quantity)) - }) - } -} - func testActorDirs() *ateworkerpb.ActorDirs { return &ateworkerpb.ActorDirs{ RootDir: "/node/actors/a", diff --git a/internal/resources/worker.go b/internal/resources/worker.go index 2ba5dd0d4b..fe949f23e3 100644 --- a/internal/resources/worker.go +++ b/internal/resources/worker.go @@ -15,53 +15,44 @@ package resources import ( - "fmt" - "go.ate.dev/substrate/pkg/proto/ateapipb" ) // AddToAllocated adjusts allocation by an assignment; sign is 1 or -1. // It returns nil when allocation reaches zero. -func AddToAllocated(total *ateapipb.WorkerResources, assignment *ateapipb.ActorAssignment, sign int64) (*ateapipb.WorkerResources, error) { - held, err := ParseQuantities(total.GetResources()) - if err != nil { - return nil, fmt.Errorf("allocated: %w", err) - } - booked, err := ParseQuantities(assignment.GetResources()) - if err != nil { - return nil, fmt.Errorf("assignment for actor %s: %w", assignment.GetActorUid(), err) - } - if held == nil { - held = Quantities{} - } - if sign > 0 { - held.Add(booked) - } else { - held.Sub(booked) +func AddToAllocated(total *ateapipb.WorkerResources, assignment *ateapipb.ActorAssignment, sign int64) *ateapipb.WorkerResources { + booked := assignment.GetResources() + out := &ateapipb.WorkerResources{ + CpuMillicores: total.GetCpuMillicores() + sign*booked.GetCpuMillicores(), + MemoryBytes: total.GetMemoryBytes() + sign*booked.GetMemoryBytes(), + Actors: total.GetActors() + int32(sign), } - - actors := total.GetActors() + int32(sign) - resources := held.Proto() - if actors == 0 && resources == nil { - return nil, nil + if out.CpuMillicores == 0 && out.MemoryBytes == 0 && out.Actors == 0 { + return nil } - return &ateapipb.WorkerResources{Actors: actors, Resources: resources}, nil + return out } // SumAllocated is what a set of assignments takes from a Worker, or nil for // none. Rebuilds the total rather than adjusting it, which is what the checks // holding AddToAllocated to the assignments it counts compare against. -func SumAllocated(assignments []*ateapipb.ActorAssignment) (*ateapipb.WorkerResources, error) { +func SumAllocated(assignments []*ateapipb.ActorAssignment) *ateapipb.WorkerResources { if len(assignments) == 0 { - return nil, nil + return nil } - total := Quantities{} + total := &ateapipb.WorkerResources{Actors: int32(len(assignments))} for _, assignment := range assignments { - booked, err := ParseQuantities(assignment.GetResources()) - if err != nil { - return nil, fmt.Errorf("assignment for actor %s: %w", assignment.GetActorUid(), err) - } - total.Add(booked) + total.CpuMillicores += assignment.GetResources().GetCpuMillicores() + total.MemoryBytes += assignment.GetResources().GetMemoryBytes() + } + return total +} + +// Free is what a Worker has left after what its Actors hold. It goes negative +// once a Worker is overcommitted. +func Free(capacity, allocated *ateapipb.WorkerResources) *ateapipb.Resources { + return &ateapipb.Resources{ + CpuMillicores: capacity.GetCpuMillicores() - allocated.GetCpuMillicores(), + MemoryBytes: capacity.GetMemoryBytes() - allocated.GetMemoryBytes(), } - return &ateapipb.WorkerResources{Actors: int32(len(assignments)), Resources: total.Proto()}, nil } diff --git a/internal/resources/worker_test.go b/internal/resources/worker_test.go index f56c309695..55f0aad869 100644 --- a/internal/resources/worker_test.go +++ b/internal/resources/worker_test.go @@ -31,77 +31,48 @@ func assignment(uid string, cpu, mem int64) *ateapipb.ActorAssignment { } } -// mustAdd is AddToAllocated where the fixtures are known to parse. -func mustAdd(t *testing.T, total *ateapipb.WorkerResources, a *ateapipb.ActorAssignment, sign int64) *ateapipb.WorkerResources { - t.Helper() - got, err := AddToAllocated(total, a, sign) - if err != nil { - t.Fatalf("AddToAllocated(%v, %d): %v", a, sign, err) - } - return got -} - // The total moves by one assignment's worth in each direction, and a Worker // back to holding nothing carries no allocation at all rather than a zeroed // message: emptied and never-filled have to be the same record. func TestAddToAllocated(t *testing.T) { var total *ateapipb.WorkerResources - total = mustAdd(t, total, assignment("a", 1000, 1<<30), +1) - total = mustAdd(t, total, assignment("b", 500, 2<<30), +1) + total = AddToAllocated(total, assignment("a", 1000, 1<<30), +1) + total = AddToAllocated(total, assignment("b", 500, 2<<30), +1) - want := &ateapipb.WorkerResources{Actors: 2, Resources: CPUMemory(1500, 3<<30)} + want := &ateapipb.WorkerResources{Actors: 2, CpuMillicores: 1500, MemoryBytes: 3 << 30} if diff := cmp.Diff(want, total, protocmp.Transform()); diff != "" { t.Errorf("allocated mismatch (-want +got):\n%s", diff) } - total = mustAdd(t, total, assignment("b", 500, 2<<30), -1) - want = &ateapipb.WorkerResources{Actors: 1, Resources: CPUMemory(1000, 1<<30)} + total = AddToAllocated(total, assignment("b", 500, 2<<30), -1) + want = &ateapipb.WorkerResources{Actors: 1, CpuMillicores: 1000, MemoryBytes: 1 << 30} if diff := cmp.Diff(want, total, protocmp.Transform()); diff != "" { t.Errorf("allocated after release mismatch (-want +got):\n%s", diff) } - if total = mustAdd(t, total, assignment("a", 1000, 1<<30), -1); total != nil { + if total = AddToAllocated(total, assignment("a", 1000, 1<<30), -1); total != nil { t.Errorf("allocated = %v for a Worker holding nothing, want nil", total) } } // An Actor that declared no limits reserves nothing but still costs a slot. func TestAddToAllocatedCountsAnActorWithoutResources(t *testing.T) { - total := mustAdd(t, nil, &ateapipb.ActorAssignment{ActorUid: "a"}, +1) + total := AddToAllocated(nil, &ateapipb.ActorAssignment{ActorUid: "a"}, +1) want := &ateapipb.WorkerResources{Actors: 1} if diff := cmp.Diff(want, total, protocmp.Transform()); diff != "" { t.Errorf("allocated mismatch (-want +got):\n%s", diff) } } -// A quantity that will not parse is refused rather than counted as nothing: -// silently booking zero would overcommit the Worker. -func TestAddToAllocatedRejectsAnUnparseableQuantity(t *testing.T) { - bad := &ateapipb.ActorAssignment{ - ActorUid: "a", - Resources: &ateapipb.Resources{Limits: []*ateapipb.Limits{{Name: "cpu", Quantity: "two"}}}, - } - if _, err := AddToAllocated(nil, bad, +1); err == nil { - t.Fatal("AddToAllocated() = nil error for an unparseable quantity, want one") - } -} - // SumAllocated rebuilds what AddToAllocated adjusts; the contract tests hold the // two to each other, so they have to agree on the empty case as well. func TestSumAllocated(t *testing.T) { - got, err := SumAllocated(nil) - if err != nil { - t.Fatalf("SumAllocated(nil): %v", err) - } - if got != nil { + if got := SumAllocated(nil); got != nil { t.Errorf("SumAllocated(nil) = %v, want nil", got) } - got, err = SumAllocated([]*ateapipb.ActorAssignment{assignment("a", 1000, 1<<30), assignment("b", 500, 0)}) - if err != nil { - t.Fatalf("SumAllocated(): %v", err) - } - want := &ateapipb.WorkerResources{Actors: 2, Resources: CPUMemory(1500, 1<<30)} + got := SumAllocated([]*ateapipb.ActorAssignment{assignment("a", 1000, 1<<30), assignment("b", 500, 0)}) + want := &ateapipb.WorkerResources{Actors: 2, CpuMillicores: 1500, MemoryBytes: 1 << 30} if diff := cmp.Diff(want, got, protocmp.Transform()); diff != "" { t.Errorf("SumAllocated() mismatch (-want +got):\n%s", diff) } diff --git a/internal/sizing/sizing.go b/internal/sizing/sizing.go index 8122981c19..1f0b1d6bb3 100644 --- a/internal/sizing/sizing.go +++ b/internal/sizing/sizing.go @@ -24,6 +24,7 @@ package sizing import ( specs "github.com/opencontainers/runtime-spec/specs-go" + "go.ate.dev/substrate/internal/proto/ateworkerpb" ) const ( @@ -41,17 +42,11 @@ type SandboxSize struct { MemoryBytes int64 } -// FromLimits builds a SandboxSize from an actor's declared limits (millicores and -// bytes) as carried on the worker RPCs. It is runtime-agnostic; both ateworker-gvisor -// and ateworker-microvm call it. Negative values are clamped to zero ("unset"). -func FromLimits(milliCPU, memoryBytes int64) SandboxSize { - if milliCPU < 0 { - milliCPU = 0 - } - if memoryBytes < 0 { - memoryBytes = 0 - } - return SandboxSize{MilliCPU: milliCPU, MemoryBytes: memoryBytes} +// FromLimits builds a SandboxSize from an actor's declared limits as carried +// on the worker RPCs. It is runtime-agnostic; both ateworker-gvisor and +// ateworker-microvm call it. Negative values are clamped to zero ("unset"). +func FromLimits(l *ateworkerpb.Resources) SandboxSize { + return SandboxSize{MilliCPU: max(l.GetCpuMillicores(), 0), MemoryBytes: max(l.GetMemoryBytes(), 0)} } // VCPUs converts the CPU limit to a whole vCPU count for the micro-VM, rounding diff --git a/internal/sizing/sizing_test.go b/internal/sizing/sizing_test.go index 1c9ac24fc5..b0b8dedc63 100644 --- a/internal/sizing/sizing_test.go +++ b/internal/sizing/sizing_test.go @@ -15,20 +15,21 @@ package sizing import ( + "go.ate.dev/substrate/internal/proto/ateworkerpb" "testing" specs "github.com/opencontainers/runtime-spec/specs-go" ) func TestFromLimits(t *testing.T) { - got := FromLimits(1500, 2147483648) + got := FromLimits(&ateworkerpb.Resources{CpuMillicores: 1500, MemoryBytes: 2147483648}) if got.MilliCPU != 1500 || got.MemoryBytes != 2147483648 { t.Fatalf("FromLimits() = %+v", got) } } func TestFromLimitsClampsNegative(t *testing.T) { - got := FromLimits(-5, -1) + got := FromLimits(&ateworkerpb.Resources{CpuMillicores: -5, MemoryBytes: -1}) if got.MilliCPU != 0 || got.MemoryBytes != 0 { t.Fatalf("FromLimits() = %+v, want zero", got) } diff --git a/internal/volume/csi/plugin.go b/internal/volume/csi/plugin.go index 4b348680cb..49f6e9601a 100644 --- a/internal/volume/csi/plugin.go +++ b/internal/volume/csi/plugin.go @@ -31,7 +31,6 @@ import ( v1alpha1 "go.ate.dev/substrate/pkg/api/v1alpha1" "google.golang.org/grpc/codes" "google.golang.org/grpc/status" - "k8s.io/apimachinery/pkg/api/resource" ) const ( @@ -168,16 +167,10 @@ func (p *Plugin) DriverName(ctx context.Context) (string, error) { // CreateVolume maps to CSI Controller CreateVolume. func (p *Plugin) CreateVolume(ctx context.Context, req volume.CreateVolumeRequest) (volume.CreateVolumeResponse, error) { - qty, err := resource.ParseQuantity(req.Capacity) - if err != nil { - return volume.CreateVolumeResponse{}, fmt.Errorf("failed to parse capacity %q: %w", req.Capacity, err) - } - capBytes := qty.Value() - csiReq := &csi.CreateVolumeRequest{ Name: req.Name, CapacityRange: &csi.CapacityRange{ - RequiredBytes: capBytes, + RequiredBytes: req.CapacityBytes, }, VolumeCapabilities: getStandardCapabilities(), Parameters: req.Parameters, diff --git a/internal/volume/csi/plugin_test.go b/internal/volume/csi/plugin_test.go index 8da7583864..c47bb4dabd 100644 --- a/internal/volume/csi/plugin_test.go +++ b/internal/volume/csi/plugin_test.go @@ -224,8 +224,8 @@ func TestPlugin_CreateVolume(t *testing.T) { ctx := context.Background() resp, err := plugin.CreateVolume(ctx, volume.CreateVolumeRequest{ - Name: "test-vol", - Capacity: "1Gi", + Name: "test-vol", + CapacityBytes: 1 << 30, }) if err != nil { t.Fatalf("CreateVolume failed: %v", err) diff --git a/internal/volume/mock.go b/internal/volume/mock.go index 8fad98b3ee..88bf952da3 100644 --- a/internal/volume/mock.go +++ b/internal/volume/mock.go @@ -64,7 +64,7 @@ func (p *MockVolumePlugin) DriverName(ctx context.Context) (string, error) { // CreateVolume simulates volume provisioning. func (p *MockVolumePlugin) CreateVolume(ctx context.Context, req CreateVolumeRequest) (CreateVolumeResponse, error) { volumeID := "mock-vol-" + req.Name - slog.InfoContext(ctx, "MockVolumePlugin.CreateVolume", slog.String("name", req.Name), slog.String("capacity", req.Capacity), slog.String("volumeID", volumeID)) + slog.InfoContext(ctx, "MockVolumePlugin.CreateVolume", slog.String("name", req.Name), slog.Int64("capacity_bytes", req.CapacityBytes), slog.String("volumeID", volumeID)) return CreateVolumeResponse{VolumeID: volumeID, VolumeContext: req.Parameters}, nil } diff --git a/internal/volume/plugin.go b/internal/volume/plugin.go index 20e898700f..e003f48c28 100644 --- a/internal/volume/plugin.go +++ b/internal/volume/plugin.go @@ -22,8 +22,8 @@ import ( type CreateVolumeRequest struct { // Name is the name to provision the volume under. Name string - // Capacity is the requested size as a Kubernetes resource.Quantity string. - Capacity string + // CapacityBytes is the requested size in bytes. + CapacityBytes int64 // Parameters are the driver-specific parameters from the StorageClass. Parameters map[string]string } diff --git a/internal/workercapacity/workercapacity.go b/internal/workercapacity/workercapacity.go index 8ebab0d1bd..2080b74756 100644 --- a/internal/workercapacity/workercapacity.go +++ b/internal/workercapacity/workercapacity.go @@ -13,11 +13,11 @@ // limitations under the License. // Package workercapacity names the files the atecontroller projects into the -// worker container from the downward API, in milli-cores and bytes. +// worker container from the downward API, in millicores and bytes. package workercapacity const ( MountPath = "/run/worker-capacity" - CPULimitFile = "cpu_milli" + CPULimitFile = "cpu_millicores" MemoryLimitFile = "memory_bytes" ) diff --git a/manifests/ate-install/atenet-router.yaml b/manifests/ate-install/atenet-router.yaml index 638864efc2..4d4b8e1aca 100644 --- a/manifests/ate-install/atenet-router.yaml +++ b/manifests/ate-install/atenet-router.yaml @@ -226,21 +226,17 @@ spec: # Client auth on the TLS listeners (--port-https, --port-connect-tls). # A client certificate must chain to the podidentity # ClusterTrustBundle, which the envoy container mounts too, and carry - # one of the SPIFFE IDs below. The default --ingress-auth-mode, + # one of the allowed SPIFFE IDs. The default --ingress-auth-mode, # deprecated-insecure, only checks a certificate a client chooses to # present, and keeps the plaintext listeners open; # components/router-static-mtls makes the certificate mandatory. # - # Allowed SPIFFE IDs (exact match), one per client workload: - # - # spiffe://cluster.local/ns/ate-e2e-router-client/sa/router-client - # The e2e suites' router client (see internal/e2e/ingressauth.go). - # - # Add an ID for every in-cluster workload that sends traffic to actors - # through the router. The podidentity signer issues - # spiffe://cluster.local/ns/<namespace>/sa/<service-account>. + # The allowed SPIFFE IDs are the install's choice, so this manifest + # carries none: cmd/ate-setup appends --ingress-allowed-spiffe-ids from + # its own --ingress-allowed-spiffe-ids (ATE_INGRESS_ALLOWED_SPIFFE_IDS), + # and the router refuses to start without it. - "--ingress-client-ca-file=/run/podidentity.podcert.ate.dev/trust-bundle.pem" - - "--ingress-allowed-spiffe-ids=spiffe://cluster.local/ns/ate-e2e-router-client/sa/router-client" + # - "--ingress-allowed-spiffe-ids=spiffe://cluster.local/ns/ate-e2e-router-client/sa/router-client" # Client auth to ateapi (mtls): verify the serving cert against the # servicedns trust bundle and present the podidentity client cert. - "--ateapi-address=k8s:///api.ate-system.svc:443" diff --git a/manifests/ate-install/components/router-static-mtls/kustomization.yaml b/manifests/ate-install/components/router-static-mtls/kustomization.yaml index ea57dcff67..663e8f4151 100644 --- a/manifests/ate-install/components/router-static-mtls/kustomization.yaml +++ b/manifests/ate-install/components/router-static-mtls/kustomization.yaml @@ -24,9 +24,9 @@ # two listeners cannot authenticate a client, and the router refuses to start # in static-mtls mode while either is enabled. # -# The client CA bundle and the allowed SPIFFE IDs come from the base manifest -# (atenet-router.yaml), where the TLS listeners already check any certificate a -# client presents. +# The client CA bundle comes from the base manifest (atenet-router.yaml), where +# the TLS listeners already check any certificate a client presents, and the +# allowed SPIFFE IDs from ate-setup's --ingress-allowed-spiffe-ids. # TODO(ahmedtd): All of this code (and atenet ingress's support for plaintext # listeners) should be removed when the deprecated-insecure mode is removed. diff --git a/manifests/ate-install/generated/role.yaml b/manifests/ate-install/generated/role.yaml index 3298e88aa2..9e2ab2e7b1 100644 --- a/manifests/ate-install/generated/role.yaml +++ b/manifests/ate-install/generated/role.yaml @@ -54,6 +54,12 @@ rules: - patch - update - watch +- apiGroups: + - apps + resources: + - replicasets + verbs: + - get - apiGroups: - ate.dev resources: diff --git a/manifests/ate-install/role.yaml b/manifests/ate-install/role.yaml index f393cb2d9c..e7efb0504b 100644 --- a/manifests/ate-install/role.yaml +++ b/manifests/ate-install/role.yaml @@ -56,6 +56,12 @@ rules: - patch - update - watch +- apiGroups: + - apps + resources: + - replicasets + verbs: + - get - apiGroups: - ate.dev resources: diff --git a/pkg/objectstorage/parzstd.go b/pkg/objectstorage/parzstd.go index 894017c3aa..7802703d36 100644 --- a/pkg/objectstorage/parzstd.go +++ b/pkg/objectstorage/parzstd.go @@ -16,6 +16,7 @@ package objectstorage import ( "io" + "os" "runtime" "sync" @@ -39,11 +40,70 @@ const ( // parZstdQueue is how many chunks may be outstanding per worker, so the producer // can run ahead while workers compress. parZstdQueue = 2 + // parZstdFrameCap is the capacity of a pooled output buffer: one chunk's frame + // even when the chunk does not compress. + parZstdFrameCap = parZstdChunk + parZstdChunk/16 ) -// parZstd is an io.WriteCloser that compresses what it is given as parallel zstd -// frames, written to dst in order. Close flushes the tail and reports the first -// error from any worker or from dst. +// Chunk and frame buffers are pooled across uploads rather than allocated for each +// one: a snapshot otherwise allocates tens of MiB of fresh buffers per upload, and a +// node checkpointing many actors at once spends its time page-faulting them in. Only +// buffers of the pooled capacity go back, so a pool never hands out something smaller +// than its callers expect. Encoders are not pooled: an Encoder makes an internal +// channel on first use, which ties it to the synctest bubble (in tests) that first +// used it. +var ( + parZstdChunkPool = sync.Pool{New: func() any { b := make([]byte, 0, parZstdChunk); return &b }} + parZstdFramePool = sync.Pool{New: func() any { b := make([]byte, 0, parZstdFrameCap); return &b }} +) + +// newChunkEncoder returns an encoder for EncodeAll use. Single-segment frames always +// declare their content size, even the short last frame of a stream (zstd otherwise +// leaves it out under 256 bytes), so a reader can size every frame without having +// to bound it; a chunk is at most parZstdChunk, so the frame never needs a larger +// window than that. +func newChunkEncoder() *zstd.Encoder { + enc, err := zstd.NewWriter(nil, + zstd.WithEncoderLevel(zstd.SpeedFastest), + zstd.WithEncoderConcurrency(1), + zstd.WithSingleSegment(true)) + if err != nil { + // NewWriter only fails on bad options, which are compile-time constants here. + panic(err) + } + return enc +} + +func getChunk() []byte { return (*parZstdChunkPool.Get().(*[]byte))[:0] } +func getFrame() []byte { return (*parZstdFramePool.Get().(*[]byte))[:0] } + +func putChunk(b []byte) { + if cap(b) == parZstdChunk { + b = b[:0] + parZstdChunkPool.Put(&b) + } +} + +func putFrame(b []byte) { + if cap(b) == parZstdFrameCap { + b = b[:0] + parZstdFramePool.Put(&b) + } +} + +// parZstd is an io.WriteCloser that compresses what it is given as independent zstd +// frames of at most parZstdChunk input bytes, each declaring its decompressed size, +// written to dst in order. Sized frames can be split and decoded in parallel on +// restore. Close flushes the tail and reports the first error from any worker or +// from dst. +// +// With more than one worker, chunks are compressed on a pool of goroutines. With one +// (inline mode) there is no pool: each chunk is compressed on the caller's goroutine +// when it fills. Inline mode holds one pooled chunk, one encoder, and a pooled frame +// only once a full chunk is compressed, which keeps an upload part within +// serialWorkerMemory; it suits callers that already run one encoder per parallel +// unit of work (an upload part) and small files. Inline Write returns dst's error +// as soon as it happens, so a failed upload stops reading its source. type parZstd struct { dst io.Writer workers int @@ -55,6 +115,11 @@ type parZstd struct { done chan struct{} err error workerWG sync.WaitGroup + + // Inline mode (workers == 1) only. + inline bool + enc *zstd.Encoder + frame []byte } type parZstdJob struct { @@ -63,10 +128,21 @@ type parZstdJob struct { } // newParZstd starts the worker pool, capped at GOMAXPROCS (workers <= 0 asks for it). +// One worker, after the cap, means inline mode with no goroutines, so with +// GOMAXPROCS=1 every caller compresses inline. func newParZstd(dst io.Writer, workers int) *parZstd { if workers <= 0 || workers > runtime.GOMAXPROCS(0) { workers = runtime.GOMAXPROCS(0) } + if workers == 1 { + return &parZstd{ + dst: dst, + workers: 1, + inline: true, + enc: newChunkEncoder(), + buf: getChunk(), + } + } p := &parZstd{ dst: dst, workers: workers, @@ -76,7 +152,7 @@ func newParZstd(dst io.Writer, workers int) *parZstd { done: make(chan struct{}), } for range workers * parZstdQueue { - p.free <- make([]byte, 0, parZstdChunk) + p.free <- getChunk() } for range workers { p.workerWG.Go(p.worker) @@ -89,16 +165,19 @@ func newParZstd(dst io.Writer, workers int) *parZstd { // worker compresses whole chunks. Each holds its own encoder: the encoders are // single-shot EncodeAll users, so one per worker keeps their state private. func (p *parZstd) worker() { - enc, err := zstd.NewWriter(nil, - zstd.WithEncoderLevel(zstd.SpeedFastest), - zstd.WithEncoderConcurrency(1)) - if err != nil { - // NewWriter only fails on bad options, which are compile-time constants here. - panic(err) - } + enc := newChunkEncoder() defer enc.Close() for j := range p.jobs { - j.out <- enc.EncodeAll(j.buf, make([]byte, 0, len(j.buf)+len(j.buf)/16)) + // Only a full chunk takes a pooled frame. A partial one (a small file, or + // the tail of a stream) gets a buffer its own size, so many small uploads + // at once don't each hold a full-size frame. + var dst []byte + if len(j.buf) == parZstdChunk { + dst = getFrame() + } else { + dst = make([]byte, 0, len(j.buf)+len(j.buf)/16) + } + j.out <- enc.EncodeAll(j.buf, dst) p.free <- j.buf[:0] } } @@ -112,10 +191,17 @@ func (p *parZstd) writer() { if p.err == nil { _, p.err = p.dst.Write(frame) } + putFrame(frame) } } func (p *parZstd) Write(b []byte) (int, error) { + if p.buf == nil { + return 0, os.ErrClosed + } + if p.inline { + return p.writeInline(b) + } total := len(b) for len(b) > 0 { n := min(cap(p.buf)-len(p.buf), len(b)) @@ -128,35 +214,100 @@ func (p *parZstd) Write(b []byte) (int, error) { return total, nil } -// flush dispatches the current chunk and takes a fresh buffer. +// writeInline is Write in inline mode. It stops at the first error from dst, so the +// caller stops feeding a failed upload. (With workers, dst's error is only known to +// the writer goroutine and surfaces from Close.) +func (p *parZstd) writeInline(b []byte) (int, error) { + total := len(b) + for len(b) > 0 && p.err == nil { + n := min(cap(p.buf)-len(p.buf), len(b)) + p.buf = append(p.buf, b[:n]...) + b = b[n:] + if len(p.buf) == cap(p.buf) { + p.flush() + } + } + if p.err != nil { + return total - len(b), p.err + } + return total, nil +} + +// flush dispatches the current chunk and takes a fresh buffer or, inline, +// compresses and writes it here. func (p *parZstd) flush() { if len(p.buf) == 0 { return } + if p.inline { + p.flushInline() + return + } out := make(chan []byte, 1) p.ordered <- out p.jobs <- parZstdJob{buf: p.buf, out: out} p.buf = <-p.free } +// flushInline compresses the buffered chunk into one frame and writes it. Only a +// full chunk takes the pooled frame, which is then kept for the rest of the stream; +// a stream that is only a partial chunk (a small file) gets a frame its own size. +// Write flushes only full chunks, so a partial one is always the tail, from Close. +func (p *parZstd) flushInline() { + if p.err != nil { + return + } + out := p.frame + if out == nil { + if len(p.buf) == parZstdChunk { + p.frame = getFrame() + out = p.frame + } else { + out = make([]byte, 0, len(p.buf)+len(p.buf)/16) + } + } + out = p.enc.EncodeAll(p.buf, out[:0]) + _, p.err = p.dst.Write(out) + p.buf = p.buf[:0] +} + +// Close flushes the tail, returns the buffers to their pools and reports the first +// error. Later calls to Write or Close return that error, or os.ErrClosed. func (p *parZstd) Close() error { + if p.buf == nil { + return p.err + } p.flush() + if p.inline { + p.enc.Close() + putChunk(p.buf) + putFrame(p.frame) + p.enc, p.buf, p.frame = nil, nil, nil + return p.markClosed() + } close(p.jobs) close(p.ordered) <-p.done p.workerWG.Wait() - return p.err + // Every chunk is idle now: the one in hand and the rest back in free. + putChunk(p.buf) + p.buf = nil + for { + select { + case b := <-p.free: + putChunk(b) + default: + return p.markClosed() + } + } } -// newSerialZstd is a plain streaming encoder, for callers that are already running one -// per parallel unit of work (an upload part) and so need no fan-out of their own. -func newSerialZstd(dst io.Writer) *zstd.Encoder { - // NewWriter only errors on bad options, which are compile-time constants here. - enc, err := zstd.NewWriter(dst, - zstd.WithEncoderLevel(zstd.SpeedFastest), - zstd.WithEncoderConcurrency(1)) - if err != nil { - panic(err) +// markClosed returns the first error and records os.ErrClosed if there was none, so +// a stray Write after Close fails fast instead of spinning on the nil buffer. +func (p *parZstd) markClosed() error { + err := p.err + if err == nil { + p.err = os.ErrClosed } - return enc + return err } diff --git a/pkg/objectstorage/parzstd_test.go b/pkg/objectstorage/parzstd_test.go new file mode 100644 index 0000000000..43137f1e75 --- /dev/null +++ b/pkg/objectstorage/parzstd_test.go @@ -0,0 +1,285 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package objectstorage + +import ( + "bytes" + "errors" + "io" + "math/rand" + "os" + "path/filepath" + "runtime" + "sync" + "testing" + + "github.com/klauspost/compress/zstd" +) + +func randomBytes(seed int64, n int) []byte { + b := make([]byte, n) + rng := rand.New(rand.NewSource(seed)) + fillSnapshotData(rng, b) + return b +} + +// writeInPieces feeds data to w in uneven writes, so chunk boundaries fall inside them. +func writeInPieces(t *testing.T, w io.Writer, data []byte) { + t.Helper() + rng := rand.New(rand.NewSource(int64(len(data)))) + for len(data) > 0 { + n := min(len(data), 1+rng.Intn(3<<20)) + if _, err := w.Write(data[:n]); err != nil { + t.Fatal(err) + } + data = data[n:] + } +} + +func zstdDecodeAll(t *testing.T, stream []byte) []byte { + t.Helper() + zr, err := zstd.NewReader(nil) + if err != nil { + t.Fatal(err) + } + defer zr.Close() + out, err := zr.DecodeAll(stream, nil) + if err != nil { + t.Fatal(err) + } + return out +} + +// withGOMAXPROCS runs the rest of the test with at least n Ps, so newParZstd(w, n) +// really starts n workers instead of falling back to inline mode on a small runner. +func withGOMAXPROCS(t *testing.T, n int) { + t.Helper() + if prev := runtime.GOMAXPROCS(0); prev < n { + runtime.GOMAXPROCS(n) + t.Cleanup(func() { runtime.GOMAXPROCS(prev) }) + } +} + +// TestParZstdInlineWritesSizedFrames checks parZstd's inline mode (one worker) +// round-trips and cuts its output into frames of at most parZstdChunk that each +// declare their size, which is what lets a restore decode them in parallel. +func TestParZstdInlineWritesSizedFrames(t *testing.T) { + for _, n := range []int{0, 1, parZstdChunk - 1, parZstdChunk, parZstdChunk + 1, 3*parZstdChunk + 4321} { + data := randomBytes(int64(n), n) + var out bytes.Buffer + zw := newParZstd(&out, 1) + if !zw.inline { + t.Fatal("newParZstd(w, 1) is not inline") + } + writeInPieces(t, zw, data) + if err := zw.Close(); err != nil { + t.Fatal(err) + } + if got := zstdDecodeAll(t, out.Bytes()); !bytes.Equal(got, data) { + t.Fatalf("n=%d: round trip differs (%d vs %d bytes)", n, len(got), len(data)) + } + frames, sized, maxSize := zstdFrameStats(t, out.Bytes()) + if want := (n + parZstdChunk - 1) / parZstdChunk; frames != want { + t.Errorf("n=%d: %d frames, want %d", n, frames, want) + } + if sized != frames { + t.Errorf("n=%d: %d of %d frames declare their size", n, sized, frames) + } + if maxSize > parZstdChunk { + t.Errorf("n=%d: frame of %d bytes, want at most %d", n, maxSize, parZstdChunk) + } + } +} + +// TestParZstdMatchesChunkedEncodeAll pins parZstd's output, in both modes, to +// EncodeAll of each parZstdChunk of input with newChunkEncoder, concatenated: the +// sized-frame format parallel restore relies on. +func TestParZstdMatchesChunkedEncodeAll(t *testing.T) { + withGOMAXPROCS(t, 2) + data := randomBytes(7, 2*parZstdChunk+parZstdChunk/2+123) + enc := newChunkEncoder() + defer enc.Close() + var want []byte + for rest := data; len(rest) > 0; { + n := min(len(rest), parZstdChunk) + want = enc.EncodeAll(rest[:n], want) + rest = rest[n:] + } + for _, workers := range []int{1, 2} { + var out bytes.Buffer + zw := newParZstd(&out, workers) + writeInPieces(t, zw, data) + if err := zw.Close(); err != nil { + t.Fatal(err) + } + if !bytes.Equal(out.Bytes(), want) { + t.Errorf("workers=%d: output differs from chunked EncodeAll (%d vs %d bytes)", workers, out.Len(), len(want)) + } + } +} + +// TestPooledEncodersDoNotLeakData runs uploads back to back and side by side with +// different data through both modes. Pooled buffers are reused, so a later upload +// must never carry bytes from an earlier one. +func TestPooledEncodersDoNotLeakData(t *testing.T) { + withGOMAXPROCS(t, 2) + encoders := map[string]func(io.Writer) io.WriteCloser{ + "inline": func(w io.Writer) io.WriteCloser { return newParZstd(w, 1) }, + "par": func(w io.Writer) io.WriteCloser { return newParZstd(w, 2) }, + } + for name, newEnc := range encoders { + t.Run(name, func(t *testing.T) { + upload := func(t *testing.T, seed int64, n int) { + data := randomBytes(seed, n) + var out bytes.Buffer + zw := newEnc(&out) + writeInPieces(t, zw, data) + if err := zw.Close(); err != nil { + t.Error(err) + return + } + if got := zstdDecodeAll(t, out.Bytes()); !bytes.Equal(got, data) { + t.Errorf("seed %d: decoded %d bytes, want %d matching the input", seed, len(got), len(data)) + } + } + // A big upload fills the pooled buffers; a smaller one after it must not + // pick up the rest of them. + upload(t, 1, 3*parZstdChunk+parZstdChunk/2) + upload(t, 2, parZstdChunk+parZstdChunk/4) + upload(t, 3, 100) + var wg sync.WaitGroup + for i := range 6 { + wg.Go(func() { upload(t, int64(10+i), (i+1)*parZstdChunk/2) }) + } + wg.Wait() + }) + } +} + +type failAfter struct { + n int + err error +} + +func (f *failAfter) Write(p []byte) (int, error) { + if f.n == 0 { + return 0, f.err + } + f.n-- + return len(p), nil +} + +// TestParZstdInlineReportsWriteErrors checks a failing destination stops the inline +// encoder and surfaces from Write and Close, and that Close is safe to call twice. +func TestParZstdInlineReportsWriteErrors(t *testing.T) { + boom := errors.New("boom") + zw := newParZstd(&failAfter{n: 1, err: boom}, 1) + data := randomBytes(5, 3*parZstdChunk) + var werr error + for len(data) > 0 && werr == nil { + n := min(len(data), 1<<20) + _, werr = zw.Write(data[:n]) + data = data[n:] + } + if !errors.Is(werr, boom) { + t.Errorf("Write error = %v, want %v", werr, boom) + } + if err := zw.Close(); !errors.Is(err, boom) { + t.Errorf("Close error = %v, want %v", err, boom) + } + if err := zw.Close(); !errors.Is(err, boom) { + t.Errorf("second Close error = %v, want %v", err, boom) + } +} + +// TestWriteSparseZstdOneWorkerDstFails checks the streamed path with one (inline) +// worker returns the destination's error, whether it comes from Close (a tail-only +// stream) or from Write (a full chunk, after which fail() closes the encoder). The +// 9 MiB source is random, so its first frame is a full-size incompressible chunk. +func TestWriteSparseZstdOneWorkerDstFails(t *testing.T) { + boom := errors.New("boom") + for _, tc := range []struct { + name string + size int + }{ + {"tail_4KiB", 4 << 10}, + {"chunk_9MiB", 9 << 20}, + } { + t.Run(tc.name, func(t *testing.T) { + data := make([]byte, tc.size) + rand.New(rand.NewSource(int64(tc.size))).Read(data) + path := filepath.Join(t.TempDir(), "src") + if err := os.WriteFile(path, data, 0o600); err != nil { + t.Fatal(err) + } + src, err := os.Open(path) + if err != nil { + t.Fatal(err) + } + defer src.Close() + // The first write (magic + version) succeeds; the first frame fails. + if _, _, err := writeSparseZstd(t.Context(), &failAfter{n: 1, err: boom}, src, 1); !errors.Is(err, boom) { + t.Errorf("writeSparseZstd error = %v, want %v", err, boom) + } + }) + } +} + +// TestSparsePartsWriteSizedFrames checks the upload parts are made of sized frames +// of at most parZstdChunk, several per part, rather than one streaming frame each. +func TestSparsePartsWriteSizedFrames(t *testing.T) { + const logical, populated, extent = 80 << 20, 40 << 20, 4 << 20 + src := writeSnapshotSource(t, t.TempDir(), logical, populated, extent) + exts, pop, err := sparseExtents(src, logical) + if err != nil { + t.Fatal(err) + } + groups := planSparseParts(exts, pop, 2) + frames, sized, maxSize := zstdFrameStats(t, encodeSnapshot(t, src, logical, groups)) + if frames <= len(groups) { + t.Errorf("%d frames for %d parts of %d MiB, want several per part", frames, len(groups), pop>>20/int64(len(groups))) + } + if sized != frames { + t.Errorf("%d of %d frames declare their size", sized, frames) + } + if maxSize > parZstdChunk { + t.Errorf("frame of %d bytes, want at most %d", maxSize, parZstdChunk) + } +} + +// TestWriteAfterClose checks Write and a second Close fail fast with os.ErrClosed in +// both modes, rather than spinning on (inline) or panicking over (workers) the +// released state. +func TestWriteAfterClose(t *testing.T) { + withGOMAXPROCS(t, 2) + for _, workers := range []int{1, 2} { + zw := newParZstd(io.Discard, workers) + if got := zw.inline; got != (workers == 1) { + t.Fatalf("workers=%d: inline = %v", workers, got) + } + if _, err := zw.Write([]byte("x")); err != nil { + t.Fatal(err) + } + if err := zw.Close(); err != nil { + t.Fatal(err) + } + if _, err := zw.Write([]byte("x")); !errors.Is(err, os.ErrClosed) { + t.Errorf("workers=%d: Write after Close = %v, want os.ErrClosed", workers, err) + } + if err := zw.Close(); !errors.Is(err, os.ErrClosed) { + t.Errorf("workers=%d: second Close = %v, want os.ErrClosed", workers, err) + } + } +} diff --git a/pkg/objectstorage/sparsebench_test.go b/pkg/objectstorage/sparsebench_test.go new file mode 100644 index 0000000000..5fa4ff7bcd --- /dev/null +++ b/pkg/objectstorage/sparsebench_test.go @@ -0,0 +1,422 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package objectstorage + +// Benchmarks and a compatibility test for the sparse-part snapshot path, the one +// big snapshots take on GCS (PutSparseFile -> writeSparsePart, and readSparseZstd on +// restore). They only use functions that exist whatever encoder or decoder backs +// that path, so the same file measures every variant of it. + +import ( + "bytes" + "encoding/binary" + "fmt" + "io" + "math/rand" + "os" + "path/filepath" + "runtime" + "sync" + "sync/atomic" + "testing" + "time" + + "github.com/klauspost/compress/zstd" + "golang.org/x/sys/unix" +) + +const ( + // benchLogical and benchPopulated shape the benchmark snapshot like a guest + // memory image: mostly holes, with the resident set spread across it. + benchLogical = 2 << 30 + benchPopulated = 512 << 20 + benchExtent = 8 << 20 +) + +// snapshotPageText is the source of the compressible pages in snapshotData. +var snapshotPageText = func() []byte { + words := []string{"func", "return", "error", "nil", "context", "string", "struct", + "0x00007f3a", "import", "package", "self", "def", "None", "true", "false", "{", "}", + "\n\t", "for", "range", "if", "else", "len", "make", "[]byte", "int64", "README"} + rng := rand.New(rand.NewSource(3)) + var b bytes.Buffer + for b.Len() < 64<<10 { + b.WriteString(words[rng.Intn(len(words))]) + b.WriteByte(' ') + } + return b.Bytes()[:64<<10] +}() + +// fillSnapshotData fills p with a fixed-seed mix of 4 KiB pages: a quarter zero, +// a quarter random, and half text-like, so it compresses roughly like guest memory. +func fillSnapshotData(rng *rand.Rand, p []byte) { + for page := 0; page < len(p); page += 4096 { + pg := p[page:min(page+4096, len(p))] + switch rng.Intn(4) { + case 0: + clear(pg) + case 1: + rng.Read(pg) + default: + off := rng.Intn(len(snapshotPageText) - len(pg)) + copy(pg, snapshotPageText[off:]) + } + } +} + +// writeSnapshotSource creates a sparse file of logical bytes whose populated bytes +// are spread over extents of extent bytes, evenly spaced, and returns it open. +func writeSnapshotSource(tb testing.TB, dir string, logical, populated, extent int64) *os.File { + tb.Helper() + f, err := os.Create(filepath.Join(dir, "snapshot")) + if err != nil { + tb.Fatal(err) + } + tb.Cleanup(func() { f.Close() }) + if err := f.Truncate(logical); err != nil { + tb.Fatal(err) + } + n := populated / extent + stride := logical / n + rng := rand.New(rand.NewSource(1)) + buf := make([]byte, extent) + for i := range n { + fillSnapshotData(rng, buf) + if _, err := f.WriteAt(buf, i*stride); err != nil { + tb.Fatal(err) + } + } + return f +} + +// snapshotParts plans src the way PutSparseFile does. +func snapshotParts(tb testing.TB, src *os.File, logical int64) (groups [][]extentRange, populated int64) { + tb.Helper() + exts, populated, err := sparseExtents(src, logical) + if err != nil { + tb.Fatal(err) + } + return planSparseParts(exts, populated, sparsePartCount(populated)), populated +} + +func rangesBytes(ranges []extentRange) (n int64) { + for _, r := range ranges { + n += r.length + } + return n +} + +// encodeSnapshot concatenates every part's writeSparsePart output, as the composed +// GCS object would hold them. +func encodeSnapshot(tb testing.TB, src *os.File, logical int64, groups [][]extentRange) []byte { + tb.Helper() + var out bytes.Buffer + for i, g := range groups { + if err := writeSparsePart(&out, src, logical, g, i == 0, i == len(groups)-1); err != nil { + tb.Fatalf("part %d: %v", i, err) + } + } + return out.Bytes() +} + +// encodeSnapshotOldFormat writes the same parts the way main wrote them before parts +// were split into sized frames: one streaming zstd frame per part, with no content +// size. Restores must keep decoding snapshots written like this. +func encodeSnapshotOldFormat(tb testing.TB, src *os.File, logical int64, groups [][]extentRange) []byte { + tb.Helper() + var out bytes.Buffer + buf := make([]byte, 1<<20) + for i, g := range groups { + if i == 0 { + out.WriteString(sparseMagic) + binary.Write(&out, binary.LittleEndian, sparseVersion) + } + zw, err := zstd.NewWriter(&out, zstd.WithEncoderLevel(zstd.SpeedFastest), zstd.WithEncoderConcurrency(1)) + if err != nil { + tb.Fatal(err) + } + if i == 0 { + binary.Write(zw, binary.LittleEndian, logical) + } + for _, e := range g { + binary.Write(zw, binary.LittleEndian, e.off) + binary.Write(zw, binary.LittleEndian, e.length) + for at, end := e.off, e.off+e.length; at < end; { + n, err := src.ReadAt(buf[:min(int64(len(buf)), end-at)], at) + if err != nil { + tb.Fatal(err) + } + zw.Write(buf[:n]) + at += int64(n) + } + } + if i == len(groups)-1 { + binary.Write(zw, binary.LittleEndian, sparseEndOffset) + } + if err := zw.Close(); err != nil { + tb.Fatal(err) + } + } + return out.Bytes() +} + +// zstdFrameStats walks the zstd frames in stream (skipping the sparse header when +// present) and reports how many there are, how many declare their content size, and +// the largest declared size. +func zstdFrameStats(tb testing.TB, stream []byte) (frames, sized int, maxSize uint64) { + tb.Helper() + if bytes.HasPrefix(stream, []byte(sparseMagic)) { + stream = stream[len(sparseMagic)+4:] + } + for len(stream) > 0 { + var h zstd.Header + if err := h.Decode(stream); err != nil { + tb.Fatalf("frame %d header: %v", frames, err) + } + if h.Skippable { + stream = stream[8+int(h.SkippableSize):] + continue + } + frames++ + if h.HasFCS { + sized++ + maxSize = max(maxSize, h.FrameContentSize) + } + pos := h.HeaderSize + for { + if pos+3 > len(stream) { + tb.Fatalf("frame %d: truncated block header", frames) + } + bh := uint32(stream[pos]) | uint32(stream[pos+1])<<8 | uint32(stream[pos+2])<<16 + pos += 3 + if (bh>>1)&3 == 1 { // RLE block: one byte repeated + pos++ + } else { + pos += int(bh >> 3) + } + if bh&1 == 1 { + break + } + } + if h.HasCheckSum { + pos += 4 + } + stream = stream[pos:] + } + return frames, sized, maxSize +} + +// peakHeapMiB runs fn and reports how far the heap grew above where it started, +// sampled every millisecond. Two GCs first empty the sync.Pools too, so buffers fn +// takes from a pool count against it. +func peakHeapMiB(fn func()) float64 { + runtime.GC() + runtime.GC() + var base runtime.MemStats + runtime.ReadMemStats(&base) + var peak atomic.Uint64 + peak.Store(base.HeapAlloc) + sample := func() { + var s runtime.MemStats + runtime.ReadMemStats(&s) + if s.HeapAlloc > peak.Load() { + peak.Store(s.HeapAlloc) + } + } + stop, stopped := make(chan struct{}), make(chan struct{}) + go func() { + defer close(stopped) + tick := time.NewTicker(time.Millisecond) + defer tick.Stop() + for { + select { + case <-stop: + return + case <-tick.C: + sample() + } + } + }() + fn() + close(stop) + <-stopped + sample() + return float64(peak.Load()-base.HeapAlloc) / (1 << 20) +} + +// cpuTime is the process's user+system CPU time. +func cpuTime(tb testing.TB) time.Duration { + var ru unix.Rusage + if err := unix.Getrusage(unix.RUSAGE_SELF, &ru); err != nil { + tb.Fatal(err) + } + return time.Duration(ru.Utime.Nano() + ru.Stime.Nano()) +} + +type byteCounter struct{ n atomic.Int64 } + +func (c *byteCounter) Write(p []byte) (int, error) { + c.n.Add(int64(len(p))) + return len(p), nil +} + +// BenchmarkWriteSparsePart measures the suspend side: compressing one 64 MiB part +// (the unit the upload memory budget reserves for), and a whole 512 MiB snapshot +// written as concurrent parts like PutSparseFile. +func BenchmarkWriteSparsePart(b *testing.B) { + src := writeSnapshotSource(b, b.TempDir(), benchLogical, benchPopulated, benchExtent) + groups, populated := snapshotParts(b, src, benchLogical) + if len(groups) < 3 { + b.Fatalf("got %d parts, want at least 3", len(groups)) + } + + b.Run("part=64MiB", func(b *testing.B) { + part := groups[1] // a middle part: no header, no end sentinel + write := func(w io.Writer) { + if err := writeSparsePart(w, src, benchLogical, part, false, false); err != nil { + b.Fatal(err) + } + } + var out bytes.Buffer + write(&out) + frames, sized, maxSize := zstdFrameStats(b, out.Bytes()) + var peak float64 + for range 3 { + peak = max(peak, peakHeapMiB(func() { write(io.Discard) })) + } + b.SetBytes(rangesBytes(part)) + b.ReportAllocs() + cpu := cpuTime(b) + b.ResetTimer() + for range b.N { + write(io.Discard) + } + b.StopTimer() + b.ReportMetric(float64(cpuTime(b)-cpu)/float64(b.Elapsed()), "cpu-cores") + b.ReportMetric(peak, "peak-heap-MiB") + b.ReportMetric(float64(out.Len())/float64(rangesBytes(part)), "ratio") + b.ReportMetric(float64(frames), "frames") + b.ReportMetric(float64(sized)/float64(frames), "sized-frac") + b.ReportMetric(float64(maxSize)/(1<<20), "max-frame-MiB") + }) + + b.Run(fmt.Sprintf("snapshot=%dMiB", populated>>20), func(b *testing.B) { + write := func() int64 { + var c byteCounter + var wg sync.WaitGroup + for i, g := range groups { + wg.Go(func() { + if err := writeSparsePart(&c, src, benchLogical, g, i == 0, i == len(groups)-1); err != nil { + b.Error(err) + } + }) + } + wg.Wait() + return c.n.Load() + } + compressed := write() + peak := peakHeapMiB(func() { write() }) + b.SetBytes(populated) + b.ReportAllocs() + cpu := cpuTime(b) + b.ResetTimer() + for range b.N { + write() + } + b.StopTimer() + b.ReportMetric(float64(cpuTime(b)-cpu)/float64(b.Elapsed()), "cpu-cores") + b.ReportMetric(peak, "peak-heap-MiB") + b.ReportMetric(float64(compressed)/float64(populated), "ratio") + }) +} + +// BenchmarkDecodeSparse measures the resume side: decoding a 512 MiB snapshot through +// readSparseZstd. "own" is the stream this build writes; "old-format" is one unsized +// frame per part, as snapshots already in buckets were written. +func BenchmarkDecodeSparse(b *testing.B) { + dir := b.TempDir() + src := writeSnapshotSource(b, dir, benchLogical, benchPopulated, benchExtent) + groups, populated := snapshotParts(b, src, benchLogical) + for _, tc := range []struct { + name string + encode func(testing.TB, *os.File, int64, [][]extentRange) []byte + }{ + {"own", encodeSnapshot}, + {"old-format", encodeSnapshotOldFormat}, + } { + b.Run(tc.name, func(b *testing.B) { + stream := tc.encode(b, src, benchLogical, groups) + dst, err := os.Create(filepath.Join(dir, "restored-"+tc.name)) + if err != nil { + b.Fatal(err) + } + defer os.Remove(dst.Name()) + defer dst.Close() + decode := func() { + size, err := readSparseZstd(dst, bytes.NewReader(stream[len(sparseMagic):])) + if err != nil { + b.Fatal(err) + } + if size != benchLogical { + b.Fatalf("restored %d bytes, want %d", size, benchLogical) + } + } + decode() + peak := peakHeapMiB(decode) + b.SetBytes(populated) + b.ReportAllocs() + cpu := cpuTime(b) + b.ResetTimer() + for range b.N { + decode() + } + b.StopTimer() + b.ReportMetric(float64(cpuTime(b)-cpu)/float64(b.Elapsed()), "cpu-cores") + b.ReportMetric(peak, "peak-heap-MiB") + b.ReportMetric(float64(len(stream))/float64(populated), "ratio") + }) + } +} + +// TestSparseSnapshotFormatsDecode checks that a multi-part snapshot decodes byte-exact +// through readSparseZstd both as this build writes it and in the older one-frame-per- +// part layout, so restores keep working across an upgrade or a rollback. +func TestSparseSnapshotFormatsDecode(t *testing.T) { + const logical, populated, extent = 80 << 20, 40 << 20, 4 << 20 + src := writeSnapshotSource(t, t.TempDir(), logical, populated, extent) + exts, pop, err := sparseExtents(src, logical) + if err != nil { + t.Fatal(err) + } + groups := planSparseParts(exts, pop, 2) // 20 MiB per part: several chunks each + want := make([]byte, logical) + if n, err := src.ReadAt(want, 0); n != logical { + t.Fatalf("read %d source bytes: %v", n, err) + } + for _, tc := range []struct { + name string + encode func(testing.TB, *os.File, int64, [][]extentRange) []byte + }{ + {"own", encodeSnapshot}, + {"old-format", encodeSnapshotOldFormat}, + } { + t.Run(tc.name, func(t *testing.T) { + stream := tc.encode(t, src, logical, groups) + if got := decodeSparse(t, bytes.NewReader(stream)); !bytes.Equal(got, want) { + t.Errorf("decoded snapshot differs from the source (%d vs %d bytes)", len(got), len(want)) + } + }) + } +} diff --git a/pkg/objectstorage/sparseparts.go b/pkg/objectstorage/sparseparts.go index 5a579b2b5d..89aa7f2607 100644 --- a/pkg/objectstorage/sparseparts.go +++ b/pkg/objectstorage/sparseparts.go @@ -19,6 +19,7 @@ import ( "fmt" "io" "os" + "sync" "golang.org/x/sys/unix" ) @@ -110,6 +111,9 @@ func sparsePartCount(populated int64) int { return min(n, sparseMaxParts) } +// sparseCopyPool holds the 1 MiB buffers parts read the source through. +var sparseCopyPool = sync.Pool{New: func() any { b := make([]byte, 1<<20); return &b }} + // writeSparsePart writes one part's share of the sparse-extent stream: the header and // totalSize when it is the first part, its own extents, and the end sentinel when it is // the last. Reads use ReadAt so every part can read the same file concurrently. @@ -122,14 +126,18 @@ func writeSparsePart(dst io.Writer, src *os.File, size int64, ranges []extentRan return err } } - zw := newSerialZstd(dst) + // Sized frames rather than one streaming frame per part, so a restore can split + // the part and decode its frames in parallel. + zw := newParZstd(dst, 1) if first { if err := binary.Write(zw, binary.LittleEndian, size); err != nil { zw.Close() return err } } - buf := make([]byte, 1<<20) + bp := sparseCopyPool.Get().(*[]byte) + defer sparseCopyPool.Put(bp) + buf := *bp for _, e := range ranges { if err := binary.Write(zw, binary.LittleEndian, e.off); err != nil { zw.Close() diff --git a/pkg/objectstorage/upload.go b/pkg/objectstorage/upload.go index d5ffd15459..bca454f9d1 100644 --- a/pkg/objectstorage/upload.go +++ b/pkg/objectstorage/upload.go @@ -33,7 +33,9 @@ const ( // zstd state. Include the writer's frame and the producer's pending frame // even when there is only one worker. parallelWorkerMemory int64 = (parZstdQueue*3 + 2) * parZstdChunk - // A serial encoder has a 4 MiB window, history, blocks, and a copy buffer. + // A part's encoder (parZstd inline, workers == 1) holds an input chunk, its + // output frame, the encoder's history for that chunk, and a copy buffer: about + // 26 MiB with 8 MiB chunks (BenchmarkWriteSparsePart/part=64MiB). serialWorkerMemory int64 = 32 << 20 sparseUploadMemory = serialWorkerMemory + partWriterChunk ) diff --git a/pkg/proto/ateapipb/ateapi.pb.go b/pkg/proto/ateapipb/ateapi.pb.go index 4efc4a0434..c39bf6f695 100644 --- a/pkg/proto/ateapipb/ateapi.pb.go +++ b/pkg/proto/ateapipb/ateapi.pb.go @@ -2713,7 +2713,7 @@ type ActorTemplate struct { // // +k8s:required // +k8s:customValidation # cpu and memory limits are both set - Resources *Resources `protobuf:"bytes,6,opt,name=resources,proto3" json:"resources,omitempty"` + Resources *ResourceRequirements `protobuf:"bytes,6,opt,name=resources,proto3" json:"resources,omitempty"` // +k8s:optional Status *ActorTemplateStatus `protobuf:"bytes,7,opt,name=status,proto3" json:"status,omitempty"` unknownFields protoimpl.UnknownFields @@ -2785,7 +2785,7 @@ func (x *ActorTemplate) GetSnapshotConfig() *SnapshotConfig { return nil } -func (x *ActorTemplate) GetResources() *Resources { +func (x *ActorTemplate) GetResources() *ResourceRequirements { if x != nil { return x.Resources } @@ -2799,35 +2799,31 @@ func (x *ActorTemplate) GetStatus() *ActorTemplateStatus { return nil } -type Resources struct { +// ResourceRequirements are the compute resources a workload may use. +type ResourceRequirements struct { state protoimpl.MessageState `protogen:"open.v1"` - // limits is the maximum amount of compute resources allowed. Only "cpu" - // and "memory" are supported, each at most once, and each quantity must be - // greater than zero; the cpu limit must be less than 1000 cores. + // limits is the maximum amount of compute resources allowed. // // +k8s:optional - // +k8s:maxItems=2 - // +k8s:listType=map - // +k8s:listMapKey=name - Limits []*Limits `protobuf:"bytes,1,rep,name=limits,proto3" json:"limits,omitempty"` + Limits *Resources `protobuf:"bytes,1,opt,name=limits,proto3" json:"limits,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache } -func (x *Resources) Reset() { - *x = Resources{} +func (x *ResourceRequirements) Reset() { + *x = ResourceRequirements{} mi := &file_ateapi_proto_msgTypes[26] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } -func (x *Resources) String() string { +func (x *ResourceRequirements) String() string { return protoimpl.X.MessageStringOf(x) } -func (*Resources) ProtoMessage() {} +func (*ResourceRequirements) ProtoMessage() {} -func (x *Resources) ProtoReflect() protoreflect.Message { +func (x *ResourceRequirements) ProtoReflect() protoreflect.Message { mi := &file_ateapi_proto_msgTypes[26] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) @@ -2839,48 +2835,50 @@ func (x *Resources) ProtoReflect() protoreflect.Message { return mi.MessageOf(x) } -// Deprecated: Use Resources.ProtoReflect.Descriptor instead. -func (*Resources) Descriptor() ([]byte, []int) { +// Deprecated: Use ResourceRequirements.ProtoReflect.Descriptor instead. +func (*ResourceRequirements) Descriptor() ([]byte, []int) { return file_ateapi_proto_rawDescGZIP(), []int{26} } -func (x *Resources) GetLimits() []*Limits { +func (x *ResourceRequirements) GetLimits() *Resources { if x != nil { return x.Limits } return nil } -// +k8s:customValidation # names, quantity parse/positivity, cpu bound -type Limits struct { +// Resources is an amount of each compute resource. +type Resources struct { state protoimpl.MessageState `protogen:"open.v1"` - // +k8s:required - // +k8s:maxLength=16 # the Limits hook restricts values to cpu/memory - Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"` - // quantity is in Kubernetes resource.Quantity string form (e.g. "500m", - // "2Gi"). + // cpu_millicores is CPU in millicores (1000 = one core). 0 means none. // - // +k8s:required - // +k8s:maxLength=32 # the Limits hook requires a parseable quantity - Quantity string `protobuf:"bytes,2,opt,name=quantity,proto3" json:"quantity,omitempty"` + // +k8s:optional + // +k8s:minimum=0 + // +k8s:maximum=999999 # less than 1000 cores + CpuMillicores int64 `protobuf:"varint,1,opt,name=cpu_millicores,json=cpuMillicores,proto3" json:"cpu_millicores,omitempty"` + // memory_bytes is memory in bytes. 0 means none. + // + // +k8s:optional + // +k8s:minimum=0 + MemoryBytes int64 `protobuf:"varint,2,opt,name=memory_bytes,json=memoryBytes,proto3" json:"memory_bytes,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache } -func (x *Limits) Reset() { - *x = Limits{} +func (x *Resources) Reset() { + *x = Resources{} mi := &file_ateapi_proto_msgTypes[27] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } -func (x *Limits) String() string { +func (x *Resources) String() string { return protoimpl.X.MessageStringOf(x) } -func (*Limits) ProtoMessage() {} +func (*Resources) ProtoMessage() {} -func (x *Limits) ProtoReflect() protoreflect.Message { +func (x *Resources) ProtoReflect() protoreflect.Message { mi := &file_ateapi_proto_msgTypes[27] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) @@ -2892,23 +2890,23 @@ func (x *Limits) ProtoReflect() protoreflect.Message { return mi.MessageOf(x) } -// Deprecated: Use Limits.ProtoReflect.Descriptor instead. -func (*Limits) Descriptor() ([]byte, []int) { +// Deprecated: Use Resources.ProtoReflect.Descriptor instead. +func (*Resources) Descriptor() ([]byte, []int) { return file_ateapi_proto_rawDescGZIP(), []int{27} } -func (x *Limits) GetName() string { +func (x *Resources) GetCpuMillicores() int64 { if x != nil { - return x.Name + return x.CpuMillicores } - return "" + return 0 } -func (x *Limits) GetQuantity() string { +func (x *Resources) GetMemoryBytes() int64 { if x != nil { - return x.Quantity + return x.MemoryBytes } - return "" + return 0 } type GoldenSnapshotStatus struct { @@ -3226,10 +3224,10 @@ type Container struct { // +k8s:optional SecurityContext *SecurityContext `protobuf:"bytes,8,opt,name=security_context,json=securityContext,proto3" json:"security_context,omitempty"` // resources carries this container's compute limits, enforced inside the - // actor's sandbox. Only cpu and memory limits are supported. + // actor's sandbox. A limit of 0 leaves that resource unlimited. // // +k8s:optional - Resources *Resources `protobuf:"bytes,9,opt,name=resources,proto3" json:"resources,omitempty"` + Resources *ResourceRequirements `protobuf:"bytes,9,opt,name=resources,proto3" json:"resources,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache } @@ -3320,7 +3318,7 @@ func (x *Container) GetSecurityContext() *SecurityContext { return nil } -func (x *Container) GetResources() *Resources { +func (x *Container) GetResources() *ResourceRequirements { if x != nil { return x.Resources } @@ -3881,13 +3879,11 @@ func (*DurableDirVolumeSource) Descriptor() ([]byte, []int) { // lives only as long as the actor. type ExternalVolumeTemplate struct { state protoimpl.MessageState `protogen:"open.v1"` - // capacity of the volume to create, in Kubernetes resource.Quantity string - // form (e.g. "10Gi"). Required. + // capacity_bytes is the size of the volume to create, in bytes. Required. // // +k8s:required - // +k8s:maxLength=32 # matches Limits.quantity's bound - // +k8s:customValidation # must parse as a resource.Quantity - Capacity string `protobuf:"bytes,1,opt,name=capacity,proto3" json:"capacity,omitempty"` + // +k8s:minimum=1 + CapacityBytes int64 `protobuf:"varint,1,opt,name=capacity_bytes,json=capacityBytes,proto3" json:"capacity_bytes,omitempty"` // storage_class_name names the cluster-scoped Kubernetes StorageClass to // create the volume from. Required. // @@ -3928,11 +3924,11 @@ func (*ExternalVolumeTemplate) Descriptor() ([]byte, []int) { return file_ateapi_proto_rawDescGZIP(), []int{42} } -func (x *ExternalVolumeTemplate) GetCapacity() string { +func (x *ExternalVolumeTemplate) GetCapacityBytes() int64 { if x != nil { - return x.Capacity + return x.CapacityBytes } - return "" + return 0 } func (x *ExternalVolumeTemplate) GetStorageClassName() string { @@ -7750,11 +7746,19 @@ func (x *AttributeEntry) GetValue() string { // WorkerResources represents a set of resources. type WorkerResources struct { state protoimpl.MessageState `protogen:"open.v1"` - // Resources match the Actor's definition of resources. - // They are matched by name and used for placement. + // cpu_millicores is CPU in millicores (1000 = one core). Compared against + // the Actor's resource limits for placement. // // +k8s:optional - Resources *Resources `protobuf:"bytes,1,opt,name=resources,proto3" json:"resources,omitempty"` + // +k8s:minimum=0 + // +k8s:maximum=999999 # less than 1000 cores + CpuMillicores int64 `protobuf:"varint,1,opt,name=cpu_millicores,json=cpuMillicores,proto3" json:"cpu_millicores,omitempty"` + // memory_bytes is memory in bytes. Compared against the Actor's resource + // limits for placement. + // + // +k8s:optional + // +k8s:minimum=0 + MemoryBytes int64 `protobuf:"varint,2,opt,name=memory_bytes,json=memoryBytes,proto3" json:"memory_bytes,omitempty"` // A count of Actors. Workers can use this to limit the number of Actors // that can be placed on them, constraining the costs of other things that // are not captured in the resources above, but which still have costs - @@ -7762,7 +7766,7 @@ type WorkerResources struct { // // +k8s:optional // +k8s:minimum=1 - Actors int32 `protobuf:"varint,2,opt,name=actors,proto3" json:"actors,omitempty"` + Actors int32 `protobuf:"varint,3,opt,name=actors,proto3" json:"actors,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache } @@ -7797,11 +7801,18 @@ func (*WorkerResources) Descriptor() ([]byte, []int) { return file_ateapi_proto_rawDescGZIP(), []int{107} } -func (x *WorkerResources) GetResources() *Resources { +func (x *WorkerResources) GetCpuMillicores() int64 { if x != nil { - return x.Resources + return x.CpuMillicores } - return nil + return 0 +} + +func (x *WorkerResources) GetMemoryBytes() int64 { + if x != nil { + return x.MemoryBytes + } + return 0 } func (x *WorkerResources) GetActors() int32 { @@ -9714,7 +9725,7 @@ const file_ateapi_proto_rawDesc = "" + "\bmetadata\x18\x01 \x01(\v2\x18.ateapi.ResourceMetadataR\bmetadata:\x11\x82\xb5\x18\r\x18\x02\"\tAtespaces\";\n" + "\tObjectRef\x12\x1a\n" + "\batespace\x18\x01 \x01(\tR\batespace\x12\x12\n" + - "\x04name\x18\x02 \x01(\tR\x04name\"\x9c\x03\n" + + "\x04name\x18\x02 \x01(\tR\x04name\"\xa7\x03\n" + "\rActorTemplate\x124\n" + "\bmetadata\x18\x01 \x01(\v2\x18.ateapi.ResourceMetadataR\bmetadata\x129\n" + "\x0fworker_selector\x18\x02 \x01(\v2\x10.ateapi.SelectorR\x0eworkerSelector\x121\n" + @@ -9722,14 +9733,14 @@ const file_ateapi_proto_rawDesc = "" + "containers\x18\x03 \x03(\v2\x11.ateapi.ContainerR\n" + "containers\x12(\n" + "\avolumes\x18\x04 \x03(\v2\x0e.ateapi.VolumeR\avolumes\x12?\n" + - "\x0fsnapshot_config\x18\x05 \x01(\v2\x16.ateapi.SnapshotConfigR\x0esnapshotConfig\x12/\n" + - "\tresources\x18\x06 \x01(\v2\x11.ateapi.ResourcesR\tresources\x123\n" + - "\x06status\x18\a \x01(\v2\x1b.ateapi.ActorTemplateStatusR\x06status:\x16\x82\xb5\x18\x12\x18\x01\"\x0eActorTemplates\"3\n" + - "\tResources\x12&\n" + - "\x06limits\x18\x01 \x03(\v2\x0e.ateapi.LimitsR\x06limits\"8\n" + - "\x06Limits\x12\x12\n" + - "\x04name\x18\x01 \x01(\tR\x04name\x12\x1a\n" + - "\bquantity\x18\x02 \x01(\tR\bquantity\"\xae\x01\n" + + "\x0fsnapshot_config\x18\x05 \x01(\v2\x16.ateapi.SnapshotConfigR\x0esnapshotConfig\x12:\n" + + "\tresources\x18\x06 \x01(\v2\x1c.ateapi.ResourceRequirementsR\tresources\x123\n" + + "\x06status\x18\a \x01(\v2\x1b.ateapi.ActorTemplateStatusR\x06status:\x16\x82\xb5\x18\x12\x18\x01\"\x0eActorTemplates\"A\n" + + "\x14ResourceRequirements\x12)\n" + + "\x06limits\x18\x01 \x01(\v2\x11.ateapi.ResourcesR\x06limits\"U\n" + + "\tResources\x12%\n" + + "\x0ecpu_millicores\x18\x01 \x01(\x03R\rcpuMillicores\x12!\n" + + "\fmemory_bytes\x18\x02 \x01(\x03R\vmemoryBytes\"\xae\x01\n" + "\x14GoldenSnapshotStatus\x120\n" + "\n" + "golden_tag\x18\x01 \x01(\v2\x11.ateapi.ObjectRefR\tgoldenTag\x12?\n" + @@ -9742,7 +9753,7 @@ const file_ateapi_proto_rawDesc = "" + "\x12preferred_fidelity\x18\x02 \x01(\x0e2\x18.ateapi.SnapshotFidelityR\x11preferredFidelity\x12R\n" + "\x16golden_snapshot_config\x18\x03 \x01(\v2\x1c.ateapi.GoldenSnapshotConfigR\x14goldenSnapshotConfig\"F\n" + "\x14GoldenSnapshotConfig\x12.\n" + - "\x04mode\x18\x01 \x01(\x0e2\x1a.ateapi.GoldenSnapshotModeR\x04mode\"\xec\x02\n" + + "\x04mode\x18\x01 \x01(\x0e2\x1a.ateapi.GoldenSnapshotModeR\x04mode\"\xf7\x02\n" + "\tContainer\x12\x12\n" + "\x04name\x18\x01 \x01(\tR\x04name\x12\x14\n" + "\x05image\x18\x02 \x01(\tR\x05image\x12\x18\n" + @@ -9751,8 +9762,8 @@ const file_ateapi_proto_rawDesc = "" + "\x03env\x18\x05 \x03(\v2\x0e.ateapi.EnvVarR\x03env\x126\n" + "\fwakeup_probe\x18\x06 \x01(\v2\x13.ateapi.WakeupProbeR\vwakeupProbe\x128\n" + "\rvolume_mounts\x18\a \x03(\v2\x13.ateapi.VolumeMountR\fvolumeMounts\x12B\n" + - "\x10security_context\x18\b \x01(\v2\x17.ateapi.SecurityContextR\x0fsecurityContext\x12/\n" + - "\tresources\x18\t \x01(\v2\x11.ateapi.ResourcesR\tresources\"K\n" + + "\x10security_context\x18\b \x01(\v2\x17.ateapi.SecurityContextR\x0fsecurityContext\x12:\n" + + "\tresources\x18\t \x01(\v2\x1c.ateapi.ResourceRequirementsR\tresources\"K\n" + "\x0fSecurityContext\x128\n" + "\fcapabilities\x18\x01 \x01(\v2\x14.ateapi.CapabilitiesR\fcapabilities\"4\n" + "\fCapabilities\x12\x10\n" + @@ -9779,9 +9790,9 @@ const file_ateapi_proto_rawDesc = "" + "\btemplate\x18\x01 \x01(\v2\x1e.ateapi.ExternalVolumeTemplateR\btemplate\"1\n" + "\x11ImageVolumeSource\x12\x1c\n" + "\treference\x18\x01 \x01(\tR\treference\"\x18\n" + - "\x16DurableDirVolumeSource\"b\n" + - "\x16ExternalVolumeTemplate\x12\x1a\n" + - "\bcapacity\x18\x01 \x01(\tR\bcapacity\x12,\n" + + "\x16DurableDirVolumeSource\"m\n" + + "\x16ExternalVolumeTemplate\x12%\n" + + "\x0ecapacity_bytes\x18\x01 \x01(\x03R\rcapacityBytes\x12,\n" + "\x12storage_class_name\x18\x02 \x01(\tR\x10storageClassName\"Y\n" + "\x16SystemInfoVolumeSource\x12?\n" + "\fdata_sources\x18\x01 \x03(\v2\x1c.ateapi.SystemInfoDataSourceR\vdataSources\"\xa0\x01\n" + @@ -9973,10 +9984,11 @@ const file_ateapi_proto_rawDesc = "" + "attributes\"8\n" + "\x0eAttributeEntry\x12\x10\n" + "\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" + - "\x05value\x18\x02 \x01(\tR\x05value\"Z\n" + - "\x0fWorkerResources\x12/\n" + - "\tresources\x18\x01 \x01(\v2\x11.ateapi.ResourcesR\tresources\x12\x16\n" + - "\x06actors\x18\x02 \x01(\x05R\x06actors\"\x81\x02\n" + + "\x05value\x18\x02 \x01(\tR\x05value\"s\n" + + "\x0fWorkerResources\x12%\n" + + "\x0ecpu_millicores\x18\x01 \x01(\x03R\rcpuMillicores\x12!\n" + + "\fmemory_bytes\x18\x02 \x01(\x03R\vmemoryBytes\x12\x16\n" + + "\x06actors\x18\x03 \x01(\x05R\x06actors\"\x81\x02\n" + "\x0fActorAssignment\x124\n" + "\bmetadata\x18\x01 \x01(\v2\x18.ateapi.ResourceMetadataR\bmetadata\x12'\n" + "\x05actor\x18\x02 \x01(\v2\x11.ateapi.ObjectRefR\x05actor\x12\x1b\n" + @@ -10274,8 +10286,8 @@ var file_ateapi_proto_goTypes = []any{ (*Atespace)(nil), // 33: ateapi.Atespace (*ObjectRef)(nil), // 34: ateapi.ObjectRef (*ActorTemplate)(nil), // 35: ateapi.ActorTemplate - (*Resources)(nil), // 36: ateapi.Resources - (*Limits)(nil), // 37: ateapi.Limits + (*ResourceRequirements)(nil), // 36: ateapi.ResourceRequirements + (*Resources)(nil), // 37: ateapi.Resources (*GoldenSnapshotStatus)(nil), // 38: ateapi.GoldenSnapshotStatus (*ActorTemplateStatus)(nil), // 39: ateapi.ActorTemplateStatus (*SnapshotConfig)(nil), // 40: ateapi.SnapshotConfig @@ -10438,9 +10450,9 @@ var file_ateapi_proto_depIdxs = []int32{ 42, // 45: ateapi.ActorTemplate.containers:type_name -> ateapi.Container 48, // 46: ateapi.ActorTemplate.volumes:type_name -> ateapi.Volume 40, // 47: ateapi.ActorTemplate.snapshot_config:type_name -> ateapi.SnapshotConfig - 36, // 48: ateapi.ActorTemplate.resources:type_name -> ateapi.Resources + 36, // 48: ateapi.ActorTemplate.resources:type_name -> ateapi.ResourceRequirements 39, // 49: ateapi.ActorTemplate.status:type_name -> ateapi.ActorTemplateStatus - 37, // 50: ateapi.Resources.limits:type_name -> ateapi.Limits + 37, // 50: ateapi.ResourceRequirements.limits:type_name -> ateapi.Resources 34, // 51: ateapi.GoldenSnapshotStatus.golden_tag:type_name -> ateapi.ObjectRef 147, // 52: ateapi.GoldenSnapshotStatus.snapshot_time:type_name -> google.protobuf.Timestamp 38, // 53: ateapi.ActorTemplateStatus.golden_snapshot_status:type_name -> ateapi.GoldenSnapshotStatus @@ -10451,7 +10463,7 @@ var file_ateapi_proto_depIdxs = []int32{ 46, // 58: ateapi.Container.wakeup_probe:type_name -> ateapi.WakeupProbe 58, // 59: ateapi.Container.volume_mounts:type_name -> ateapi.VolumeMount 43, // 60: ateapi.Container.security_context:type_name -> ateapi.SecurityContext - 36, // 61: ateapi.Container.resources:type_name -> ateapi.Resources + 36, // 61: ateapi.Container.resources:type_name -> ateapi.ResourceRequirements 44, // 62: ateapi.SecurityContext.capabilities:type_name -> ateapi.Capabilities 47, // 63: ateapi.WakeupProbe.http_get:type_name -> ateapi.HttpGetAction 51, // 64: ateapi.Volume.durable_dir:type_name -> ateapi.DurableDirVolumeSource @@ -10534,154 +10546,153 @@ var file_ateapi_proto_depIdxs = []int32{ 114, // 141: ateapi.WorkerStatus.restorable_runtimes:type_name -> ateapi.SandboxRuntime 115, // 142: ateapi.SandboxRuntime.version:type_name -> ateapi.VersionedSandboxCompat 116, // 143: ateapi.VersionedSandboxCompat.attributes:type_name -> ateapi.AttributeEntry - 36, // 144: ateapi.WorkerResources.resources:type_name -> ateapi.Resources - 14, // 145: ateapi.ActorAssignment.metadata:type_name -> ateapi.ResourceMetadata - 34, // 146: ateapi.ActorAssignment.actor:type_name -> ateapi.ObjectRef - 36, // 147: ateapi.ActorAssignment.resources:type_name -> ateapi.Resources - 34, // 148: ateapi.RegisterWorkerRequest.worker:type_name -> ateapi.ObjectRef - 117, // 149: ateapi.RegisterWorkerRequest.capacity:type_name -> ateapi.WorkerResources - 114, // 150: ateapi.RegisterWorkerRequest.default_runtime:type_name -> ateapi.SandboxRuntime - 114, // 151: ateapi.RegisterWorkerRequest.restorable_runtimes:type_name -> ateapi.SandboxRuntime - 112, // 152: ateapi.RegisterWorkerResponse.worker:type_name -> ateapi.Worker - 34, // 153: ateapi.MintWorkerActorCertificateRequest.actor:type_name -> ateapi.ObjectRef - 34, // 154: ateapi.MintWorkerActorCertificateRequest.worker:type_name -> ateapi.ObjectRef - 34, // 155: ateapi.MintWorkerActorJwtRequest.actor:type_name -> ateapi.ObjectRef - 147, // 156: ateapi.MintWorkerActorJwtResponse.expire_time:type_name -> google.protobuf.Timestamp - 34, // 157: ateapi.RequestActorSuspendRequest.worker:type_name -> ateapi.ObjectRef - 34, // 158: ateapi.RequestActorSuspendRequest.actor:type_name -> ateapi.ObjectRef - 17, // 159: ateapi.RequestActorSuspendResponse.actor:type_name -> ateapi.Actor - 34, // 160: ateapi.ReportActorCrashedRequest.worker:type_name -> ateapi.ObjectRef - 34, // 161: ateapi.ReportActorCrashedRequest.actor:type_name -> ateapi.ObjectRef - 17, // 162: ateapi.ReportActorCrashedResponse.actor:type_name -> ateapi.Actor - 34, // 163: ateapi.ReportActorSnapshotStateRequest.actor:type_name -> ateapi.ObjectRef - 131, // 164: ateapi.ReportActorSnapshotStateRequest.state:type_name -> ateapi.ActorSnapshotState - 132, // 165: ateapi.ActorSnapshotState.latest_local:type_name -> ateapi.SnapshotReport - 132, // 166: ateapi.ActorSnapshotState.uploading:type_name -> ateapi.SnapshotReport - 132, // 167: ateapi.ActorSnapshotState.latest_durable:type_name -> ateapi.SnapshotReport - 147, // 168: ateapi.ActorSnapshotState.upload_due_time:type_name -> google.protobuf.Timestamp - 2, // 169: ateapi.SnapshotReport.fidelity:type_name -> ateapi.SnapshotFidelity - 10, // 170: ateapi.SnapshotReport.object:type_name -> ateapi.ObjectSnapshot - 147, // 171: ateapi.SnapshotReport.local_commit_time:type_name -> google.protobuf.Timestamp - 147, // 172: ateapi.SnapshotReport.durable_complete_time:type_name -> google.protobuf.Timestamp - 14, // 173: ateapi.AccessPolicy.metadata:type_name -> ateapi.ResourceMetadata - 134, // 174: ateapi.AccessPolicy.bindings:type_name -> ateapi.Binding - 133, // 175: ateapi.CreateGlobalAccessPolicyRequest.access_policy:type_name -> ateapi.AccessPolicy - 133, // 176: ateapi.UpdateGlobalAccessPolicyRequest.access_policy:type_name -> ateapi.AccessPolicy - 34, // 177: ateapi.GetAtespaceAccessPolicyRequest.atespace:type_name -> ateapi.ObjectRef - 34, // 178: ateapi.CreateAtespaceAccessPolicyRequest.atespace:type_name -> ateapi.ObjectRef - 133, // 179: ateapi.CreateAtespaceAccessPolicyRequest.access_policy:type_name -> ateapi.AccessPolicy - 34, // 180: ateapi.UpdateAtespaceAccessPolicyRequest.atespace:type_name -> ateapi.ObjectRef - 133, // 181: ateapi.UpdateAtespaceAccessPolicyRequest.access_policy:type_name -> ateapi.AccessPolicy - 34, // 182: ateapi.DeleteAtespaceAccessPolicyRequest.atespace:type_name -> ateapi.ObjectRef - 94, // 183: ateapi.DeleteAtespaceAccessPolicyRequest.options:type_name -> ateapi.DeleteOptions - 9, // 184: ateapi.ResourceAnnotation.scope:type_name -> ateapi.Scope - 148, // 185: ateapi.resource:extendee -> google.protobuf.MessageOptions - 149, // 186: ateapi.method:extendee -> google.protobuf.MethodOptions - 142, // 187: ateapi.resource:type_name -> ateapi.ResourceAnnotation - 143, // 188: ateapi.method:type_name -> ateapi.MethodAnnotation - 69, // 189: ateapi.Control.GetActor:input_type -> ateapi.GetActorRequest - 70, // 190: ateapi.Control.CreateActor:input_type -> ateapi.CreateActorRequest - 71, // 191: ateapi.Control.UpdateActor:input_type -> ateapi.UpdateActorRequest - 72, // 192: ateapi.Control.SuspendActor:input_type -> ateapi.SuspendActorRequest - 75, // 193: ateapi.Control.ResumeActor:input_type -> ateapi.ResumeActorRequest - 77, // 194: ateapi.Control.RevertActor:input_type -> ateapi.RevertActorRequest - 79, // 195: ateapi.Control.DeleteActor:input_type -> ateapi.DeleteActorRequest - 80, // 196: ateapi.Control.GetActorEgressPolicy:input_type -> ateapi.GetActorEgressPolicyRequest - 81, // 197: ateapi.Control.CreateActorEgressPolicy:input_type -> ateapi.CreateActorEgressPolicyRequest - 82, // 198: ateapi.Control.UpdateActorEgressPolicy:input_type -> ateapi.UpdateActorEgressPolicyRequest - 83, // 199: ateapi.Control.DeleteActorEgressPolicy:input_type -> ateapi.DeleteActorEgressPolicyRequest - 85, // 200: ateapi.Control.MintActorJwt:input_type -> ateapi.MintActorJwtRequest - 87, // 201: ateapi.Control.MintActorCertificate:input_type -> ateapi.MintActorCertificateRequest - 91, // 202: ateapi.Control.CreateTag:input_type -> ateapi.CreateTagRequest - 84, // 203: ateapi.Control.GetTag:input_type -> ateapi.GetTagRequest - 89, // 204: ateapi.Control.ListTags:input_type -> ateapi.ListTagsRequest - 92, // 205: ateapi.Control.UpdateTag:input_type -> ateapi.UpdateTagRequest - 93, // 206: ateapi.Control.DeleteTag:input_type -> ateapi.DeleteTagRequest - 98, // 207: ateapi.Control.ListWorkers:input_type -> ateapi.ListWorkersRequest - 100, // 208: ateapi.Control.GetWorker:input_type -> ateapi.GetWorkerRequest - 101, // 209: ateapi.Control.CreateWorker:input_type -> ateapi.CreateWorkerRequest - 102, // 210: ateapi.Control.UpdateWorker:input_type -> ateapi.UpdateWorkerRequest - 103, // 211: ateapi.Control.DeleteWorker:input_type -> ateapi.DeleteWorkerRequest - 104, // 212: ateapi.Control.DrainWorker:input_type -> ateapi.DrainWorkerRequest - 106, // 213: ateapi.Control.MarkWorkerAvailable:input_type -> ateapi.MarkWorkerAvailableRequest - 108, // 214: ateapi.Control.MarkWorkerUnavailable:input_type -> ateapi.MarkWorkerUnavailableRequest - 96, // 215: ateapi.Control.ListWorkerActorAssignments:input_type -> ateapi.ListWorkerActorAssignmentsRequest - 110, // 216: ateapi.Control.ListActors:input_type -> ateapi.ListActorsRequest - 59, // 217: ateapi.Control.CreateAtespace:input_type -> ateapi.CreateAtespaceRequest - 60, // 218: ateapi.Control.GetAtespace:input_type -> ateapi.GetAtespaceRequest - 61, // 219: ateapi.Control.ListAtespaces:input_type -> ateapi.ListAtespacesRequest - 63, // 220: ateapi.Control.DeleteAtespace:input_type -> ateapi.DeleteAtespaceRequest - 64, // 221: ateapi.Control.CreateActorTemplate:input_type -> ateapi.CreateActorTemplateRequest - 65, // 222: ateapi.Control.GetActorTemplate:input_type -> ateapi.GetActorTemplateRequest - 66, // 223: ateapi.Control.ListActorTemplates:input_type -> ateapi.ListActorTemplatesRequest - 68, // 224: ateapi.Control.DeleteActorTemplate:input_type -> ateapi.DeleteActorTemplateRequest - 135, // 225: ateapi.Control.GetGlobalAccessPolicy:input_type -> ateapi.GetGlobalAccessPolicyRequest - 136, // 226: ateapi.Control.CreateGlobalAccessPolicy:input_type -> ateapi.CreateGlobalAccessPolicyRequest - 137, // 227: ateapi.Control.UpdateGlobalAccessPolicy:input_type -> ateapi.UpdateGlobalAccessPolicyRequest - 138, // 228: ateapi.Control.GetAtespaceAccessPolicy:input_type -> ateapi.GetAtespaceAccessPolicyRequest - 139, // 229: ateapi.Control.CreateAtespaceAccessPolicy:input_type -> ateapi.CreateAtespaceAccessPolicyRequest - 140, // 230: ateapi.Control.UpdateAtespaceAccessPolicy:input_type -> ateapi.UpdateAtespaceAccessPolicyRequest - 141, // 231: ateapi.Control.DeleteAtespaceAccessPolicy:input_type -> ateapi.DeleteAtespaceAccessPolicyRequest - 119, // 232: ateapi.WorkerService.RegisterWorker:input_type -> ateapi.RegisterWorkerRequest - 121, // 233: ateapi.WorkerService.MintWorkerActorCertificate:input_type -> ateapi.MintWorkerActorCertificateRequest - 123, // 234: ateapi.WorkerService.MintWorkerActorJwt:input_type -> ateapi.MintWorkerActorJwtRequest - 125, // 235: ateapi.WorkerService.RequestActorSuspend:input_type -> ateapi.RequestActorSuspendRequest - 129, // 236: ateapi.WorkerService.ReportActorSnapshotState:input_type -> ateapi.ReportActorSnapshotStateRequest - 127, // 237: ateapi.WorkerService.ReportActorCrashed:input_type -> ateapi.ReportActorCrashedRequest - 17, // 238: ateapi.Control.GetActor:output_type -> ateapi.Actor - 17, // 239: ateapi.Control.CreateActor:output_type -> ateapi.Actor - 17, // 240: ateapi.Control.UpdateActor:output_type -> ateapi.Actor - 73, // 241: ateapi.Control.SuspendActor:output_type -> ateapi.SuspendActorResponse - 76, // 242: ateapi.Control.ResumeActor:output_type -> ateapi.ResumeActorResponse - 78, // 243: ateapi.Control.RevertActor:output_type -> ateapi.RevertActorResponse - 17, // 244: ateapi.Control.DeleteActor:output_type -> ateapi.Actor - 18, // 245: ateapi.Control.GetActorEgressPolicy:output_type -> ateapi.EgressPolicy - 18, // 246: ateapi.Control.CreateActorEgressPolicy:output_type -> ateapi.EgressPolicy - 18, // 247: ateapi.Control.UpdateActorEgressPolicy:output_type -> ateapi.EgressPolicy - 18, // 248: ateapi.Control.DeleteActorEgressPolicy:output_type -> ateapi.EgressPolicy - 86, // 249: ateapi.Control.MintActorJwt:output_type -> ateapi.MintActorJwtResponse - 88, // 250: ateapi.Control.MintActorCertificate:output_type -> ateapi.MintActorCertificateResponse - 32, // 251: ateapi.Control.CreateTag:output_type -> ateapi.Tag - 32, // 252: ateapi.Control.GetTag:output_type -> ateapi.Tag - 90, // 253: ateapi.Control.ListTags:output_type -> ateapi.ListTagsResponse - 32, // 254: ateapi.Control.UpdateTag:output_type -> ateapi.Tag - 32, // 255: ateapi.Control.DeleteTag:output_type -> ateapi.Tag - 99, // 256: ateapi.Control.ListWorkers:output_type -> ateapi.ListWorkersResponse - 112, // 257: ateapi.Control.GetWorker:output_type -> ateapi.Worker - 112, // 258: ateapi.Control.CreateWorker:output_type -> ateapi.Worker - 112, // 259: ateapi.Control.UpdateWorker:output_type -> ateapi.Worker - 112, // 260: ateapi.Control.DeleteWorker:output_type -> ateapi.Worker - 105, // 261: ateapi.Control.DrainWorker:output_type -> ateapi.DrainWorkerResponse - 107, // 262: ateapi.Control.MarkWorkerAvailable:output_type -> ateapi.MarkWorkerAvailableResponse - 109, // 263: ateapi.Control.MarkWorkerUnavailable:output_type -> ateapi.MarkWorkerUnavailableResponse - 97, // 264: ateapi.Control.ListWorkerActorAssignments:output_type -> ateapi.ListWorkerActorAssignmentsResponse - 111, // 265: ateapi.Control.ListActors:output_type -> ateapi.ListActorsResponse - 33, // 266: ateapi.Control.CreateAtespace:output_type -> ateapi.Atespace - 33, // 267: ateapi.Control.GetAtespace:output_type -> ateapi.Atespace - 62, // 268: ateapi.Control.ListAtespaces:output_type -> ateapi.ListAtespacesResponse - 33, // 269: ateapi.Control.DeleteAtespace:output_type -> ateapi.Atespace - 35, // 270: ateapi.Control.CreateActorTemplate:output_type -> ateapi.ActorTemplate - 35, // 271: ateapi.Control.GetActorTemplate:output_type -> ateapi.ActorTemplate - 67, // 272: ateapi.Control.ListActorTemplates:output_type -> ateapi.ListActorTemplatesResponse - 35, // 273: ateapi.Control.DeleteActorTemplate:output_type -> ateapi.ActorTemplate - 133, // 274: ateapi.Control.GetGlobalAccessPolicy:output_type -> ateapi.AccessPolicy - 133, // 275: ateapi.Control.CreateGlobalAccessPolicy:output_type -> ateapi.AccessPolicy - 133, // 276: ateapi.Control.UpdateGlobalAccessPolicy:output_type -> ateapi.AccessPolicy - 133, // 277: ateapi.Control.GetAtespaceAccessPolicy:output_type -> ateapi.AccessPolicy - 133, // 278: ateapi.Control.CreateAtespaceAccessPolicy:output_type -> ateapi.AccessPolicy - 133, // 279: ateapi.Control.UpdateAtespaceAccessPolicy:output_type -> ateapi.AccessPolicy - 133, // 280: ateapi.Control.DeleteAtespaceAccessPolicy:output_type -> ateapi.AccessPolicy - 120, // 281: ateapi.WorkerService.RegisterWorker:output_type -> ateapi.RegisterWorkerResponse - 122, // 282: ateapi.WorkerService.MintWorkerActorCertificate:output_type -> ateapi.MintWorkerActorCertificateResponse - 124, // 283: ateapi.WorkerService.MintWorkerActorJwt:output_type -> ateapi.MintWorkerActorJwtResponse - 126, // 284: ateapi.WorkerService.RequestActorSuspend:output_type -> ateapi.RequestActorSuspendResponse - 130, // 285: ateapi.WorkerService.ReportActorSnapshotState:output_type -> ateapi.ReportActorSnapshotStateResponse - 128, // 286: ateapi.WorkerService.ReportActorCrashed:output_type -> ateapi.ReportActorCrashedResponse - 238, // [238:287] is the sub-list for method output_type - 189, // [189:238] is the sub-list for method input_type - 187, // [187:189] is the sub-list for extension type_name - 185, // [185:187] is the sub-list for extension extendee - 0, // [0:185] is the sub-list for field type_name + 14, // 144: ateapi.ActorAssignment.metadata:type_name -> ateapi.ResourceMetadata + 34, // 145: ateapi.ActorAssignment.actor:type_name -> ateapi.ObjectRef + 37, // 146: ateapi.ActorAssignment.resources:type_name -> ateapi.Resources + 34, // 147: ateapi.RegisterWorkerRequest.worker:type_name -> ateapi.ObjectRef + 117, // 148: ateapi.RegisterWorkerRequest.capacity:type_name -> ateapi.WorkerResources + 114, // 149: ateapi.RegisterWorkerRequest.default_runtime:type_name -> ateapi.SandboxRuntime + 114, // 150: ateapi.RegisterWorkerRequest.restorable_runtimes:type_name -> ateapi.SandboxRuntime + 112, // 151: ateapi.RegisterWorkerResponse.worker:type_name -> ateapi.Worker + 34, // 152: ateapi.MintWorkerActorCertificateRequest.actor:type_name -> ateapi.ObjectRef + 34, // 153: ateapi.MintWorkerActorCertificateRequest.worker:type_name -> ateapi.ObjectRef + 34, // 154: ateapi.MintWorkerActorJwtRequest.actor:type_name -> ateapi.ObjectRef + 147, // 155: ateapi.MintWorkerActorJwtResponse.expire_time:type_name -> google.protobuf.Timestamp + 34, // 156: ateapi.RequestActorSuspendRequest.worker:type_name -> ateapi.ObjectRef + 34, // 157: ateapi.RequestActorSuspendRequest.actor:type_name -> ateapi.ObjectRef + 17, // 158: ateapi.RequestActorSuspendResponse.actor:type_name -> ateapi.Actor + 34, // 159: ateapi.ReportActorCrashedRequest.worker:type_name -> ateapi.ObjectRef + 34, // 160: ateapi.ReportActorCrashedRequest.actor:type_name -> ateapi.ObjectRef + 17, // 161: ateapi.ReportActorCrashedResponse.actor:type_name -> ateapi.Actor + 34, // 162: ateapi.ReportActorSnapshotStateRequest.actor:type_name -> ateapi.ObjectRef + 131, // 163: ateapi.ReportActorSnapshotStateRequest.state:type_name -> ateapi.ActorSnapshotState + 132, // 164: ateapi.ActorSnapshotState.latest_local:type_name -> ateapi.SnapshotReport + 132, // 165: ateapi.ActorSnapshotState.uploading:type_name -> ateapi.SnapshotReport + 132, // 166: ateapi.ActorSnapshotState.latest_durable:type_name -> ateapi.SnapshotReport + 147, // 167: ateapi.ActorSnapshotState.upload_due_time:type_name -> google.protobuf.Timestamp + 2, // 168: ateapi.SnapshotReport.fidelity:type_name -> ateapi.SnapshotFidelity + 10, // 169: ateapi.SnapshotReport.object:type_name -> ateapi.ObjectSnapshot + 147, // 170: ateapi.SnapshotReport.local_commit_time:type_name -> google.protobuf.Timestamp + 147, // 171: ateapi.SnapshotReport.durable_complete_time:type_name -> google.protobuf.Timestamp + 14, // 172: ateapi.AccessPolicy.metadata:type_name -> ateapi.ResourceMetadata + 134, // 173: ateapi.AccessPolicy.bindings:type_name -> ateapi.Binding + 133, // 174: ateapi.CreateGlobalAccessPolicyRequest.access_policy:type_name -> ateapi.AccessPolicy + 133, // 175: ateapi.UpdateGlobalAccessPolicyRequest.access_policy:type_name -> ateapi.AccessPolicy + 34, // 176: ateapi.GetAtespaceAccessPolicyRequest.atespace:type_name -> ateapi.ObjectRef + 34, // 177: ateapi.CreateAtespaceAccessPolicyRequest.atespace:type_name -> ateapi.ObjectRef + 133, // 178: ateapi.CreateAtespaceAccessPolicyRequest.access_policy:type_name -> ateapi.AccessPolicy + 34, // 179: ateapi.UpdateAtespaceAccessPolicyRequest.atespace:type_name -> ateapi.ObjectRef + 133, // 180: ateapi.UpdateAtespaceAccessPolicyRequest.access_policy:type_name -> ateapi.AccessPolicy + 34, // 181: ateapi.DeleteAtespaceAccessPolicyRequest.atespace:type_name -> ateapi.ObjectRef + 94, // 182: ateapi.DeleteAtespaceAccessPolicyRequest.options:type_name -> ateapi.DeleteOptions + 9, // 183: ateapi.ResourceAnnotation.scope:type_name -> ateapi.Scope + 148, // 184: ateapi.resource:extendee -> google.protobuf.MessageOptions + 149, // 185: ateapi.method:extendee -> google.protobuf.MethodOptions + 142, // 186: ateapi.resource:type_name -> ateapi.ResourceAnnotation + 143, // 187: ateapi.method:type_name -> ateapi.MethodAnnotation + 69, // 188: ateapi.Control.GetActor:input_type -> ateapi.GetActorRequest + 70, // 189: ateapi.Control.CreateActor:input_type -> ateapi.CreateActorRequest + 71, // 190: ateapi.Control.UpdateActor:input_type -> ateapi.UpdateActorRequest + 72, // 191: ateapi.Control.SuspendActor:input_type -> ateapi.SuspendActorRequest + 75, // 192: ateapi.Control.ResumeActor:input_type -> ateapi.ResumeActorRequest + 77, // 193: ateapi.Control.RevertActor:input_type -> ateapi.RevertActorRequest + 79, // 194: ateapi.Control.DeleteActor:input_type -> ateapi.DeleteActorRequest + 80, // 195: ateapi.Control.GetActorEgressPolicy:input_type -> ateapi.GetActorEgressPolicyRequest + 81, // 196: ateapi.Control.CreateActorEgressPolicy:input_type -> ateapi.CreateActorEgressPolicyRequest + 82, // 197: ateapi.Control.UpdateActorEgressPolicy:input_type -> ateapi.UpdateActorEgressPolicyRequest + 83, // 198: ateapi.Control.DeleteActorEgressPolicy:input_type -> ateapi.DeleteActorEgressPolicyRequest + 85, // 199: ateapi.Control.MintActorJwt:input_type -> ateapi.MintActorJwtRequest + 87, // 200: ateapi.Control.MintActorCertificate:input_type -> ateapi.MintActorCertificateRequest + 91, // 201: ateapi.Control.CreateTag:input_type -> ateapi.CreateTagRequest + 84, // 202: ateapi.Control.GetTag:input_type -> ateapi.GetTagRequest + 89, // 203: ateapi.Control.ListTags:input_type -> ateapi.ListTagsRequest + 92, // 204: ateapi.Control.UpdateTag:input_type -> ateapi.UpdateTagRequest + 93, // 205: ateapi.Control.DeleteTag:input_type -> ateapi.DeleteTagRequest + 98, // 206: ateapi.Control.ListWorkers:input_type -> ateapi.ListWorkersRequest + 100, // 207: ateapi.Control.GetWorker:input_type -> ateapi.GetWorkerRequest + 101, // 208: ateapi.Control.CreateWorker:input_type -> ateapi.CreateWorkerRequest + 102, // 209: ateapi.Control.UpdateWorker:input_type -> ateapi.UpdateWorkerRequest + 103, // 210: ateapi.Control.DeleteWorker:input_type -> ateapi.DeleteWorkerRequest + 104, // 211: ateapi.Control.DrainWorker:input_type -> ateapi.DrainWorkerRequest + 106, // 212: ateapi.Control.MarkWorkerAvailable:input_type -> ateapi.MarkWorkerAvailableRequest + 108, // 213: ateapi.Control.MarkWorkerUnavailable:input_type -> ateapi.MarkWorkerUnavailableRequest + 96, // 214: ateapi.Control.ListWorkerActorAssignments:input_type -> ateapi.ListWorkerActorAssignmentsRequest + 110, // 215: ateapi.Control.ListActors:input_type -> ateapi.ListActorsRequest + 59, // 216: ateapi.Control.CreateAtespace:input_type -> ateapi.CreateAtespaceRequest + 60, // 217: ateapi.Control.GetAtespace:input_type -> ateapi.GetAtespaceRequest + 61, // 218: ateapi.Control.ListAtespaces:input_type -> ateapi.ListAtespacesRequest + 63, // 219: ateapi.Control.DeleteAtespace:input_type -> ateapi.DeleteAtespaceRequest + 64, // 220: ateapi.Control.CreateActorTemplate:input_type -> ateapi.CreateActorTemplateRequest + 65, // 221: ateapi.Control.GetActorTemplate:input_type -> ateapi.GetActorTemplateRequest + 66, // 222: ateapi.Control.ListActorTemplates:input_type -> ateapi.ListActorTemplatesRequest + 68, // 223: ateapi.Control.DeleteActorTemplate:input_type -> ateapi.DeleteActorTemplateRequest + 135, // 224: ateapi.Control.GetGlobalAccessPolicy:input_type -> ateapi.GetGlobalAccessPolicyRequest + 136, // 225: ateapi.Control.CreateGlobalAccessPolicy:input_type -> ateapi.CreateGlobalAccessPolicyRequest + 137, // 226: ateapi.Control.UpdateGlobalAccessPolicy:input_type -> ateapi.UpdateGlobalAccessPolicyRequest + 138, // 227: ateapi.Control.GetAtespaceAccessPolicy:input_type -> ateapi.GetAtespaceAccessPolicyRequest + 139, // 228: ateapi.Control.CreateAtespaceAccessPolicy:input_type -> ateapi.CreateAtespaceAccessPolicyRequest + 140, // 229: ateapi.Control.UpdateAtespaceAccessPolicy:input_type -> ateapi.UpdateAtespaceAccessPolicyRequest + 141, // 230: ateapi.Control.DeleteAtespaceAccessPolicy:input_type -> ateapi.DeleteAtespaceAccessPolicyRequest + 119, // 231: ateapi.WorkerService.RegisterWorker:input_type -> ateapi.RegisterWorkerRequest + 121, // 232: ateapi.WorkerService.MintWorkerActorCertificate:input_type -> ateapi.MintWorkerActorCertificateRequest + 123, // 233: ateapi.WorkerService.MintWorkerActorJwt:input_type -> ateapi.MintWorkerActorJwtRequest + 125, // 234: ateapi.WorkerService.RequestActorSuspend:input_type -> ateapi.RequestActorSuspendRequest + 129, // 235: ateapi.WorkerService.ReportActorSnapshotState:input_type -> ateapi.ReportActorSnapshotStateRequest + 127, // 236: ateapi.WorkerService.ReportActorCrashed:input_type -> ateapi.ReportActorCrashedRequest + 17, // 237: ateapi.Control.GetActor:output_type -> ateapi.Actor + 17, // 238: ateapi.Control.CreateActor:output_type -> ateapi.Actor + 17, // 239: ateapi.Control.UpdateActor:output_type -> ateapi.Actor + 73, // 240: ateapi.Control.SuspendActor:output_type -> ateapi.SuspendActorResponse + 76, // 241: ateapi.Control.ResumeActor:output_type -> ateapi.ResumeActorResponse + 78, // 242: ateapi.Control.RevertActor:output_type -> ateapi.RevertActorResponse + 17, // 243: ateapi.Control.DeleteActor:output_type -> ateapi.Actor + 18, // 244: ateapi.Control.GetActorEgressPolicy:output_type -> ateapi.EgressPolicy + 18, // 245: ateapi.Control.CreateActorEgressPolicy:output_type -> ateapi.EgressPolicy + 18, // 246: ateapi.Control.UpdateActorEgressPolicy:output_type -> ateapi.EgressPolicy + 18, // 247: ateapi.Control.DeleteActorEgressPolicy:output_type -> ateapi.EgressPolicy + 86, // 248: ateapi.Control.MintActorJwt:output_type -> ateapi.MintActorJwtResponse + 88, // 249: ateapi.Control.MintActorCertificate:output_type -> ateapi.MintActorCertificateResponse + 32, // 250: ateapi.Control.CreateTag:output_type -> ateapi.Tag + 32, // 251: ateapi.Control.GetTag:output_type -> ateapi.Tag + 90, // 252: ateapi.Control.ListTags:output_type -> ateapi.ListTagsResponse + 32, // 253: ateapi.Control.UpdateTag:output_type -> ateapi.Tag + 32, // 254: ateapi.Control.DeleteTag:output_type -> ateapi.Tag + 99, // 255: ateapi.Control.ListWorkers:output_type -> ateapi.ListWorkersResponse + 112, // 256: ateapi.Control.GetWorker:output_type -> ateapi.Worker + 112, // 257: ateapi.Control.CreateWorker:output_type -> ateapi.Worker + 112, // 258: ateapi.Control.UpdateWorker:output_type -> ateapi.Worker + 112, // 259: ateapi.Control.DeleteWorker:output_type -> ateapi.Worker + 105, // 260: ateapi.Control.DrainWorker:output_type -> ateapi.DrainWorkerResponse + 107, // 261: ateapi.Control.MarkWorkerAvailable:output_type -> ateapi.MarkWorkerAvailableResponse + 109, // 262: ateapi.Control.MarkWorkerUnavailable:output_type -> ateapi.MarkWorkerUnavailableResponse + 97, // 263: ateapi.Control.ListWorkerActorAssignments:output_type -> ateapi.ListWorkerActorAssignmentsResponse + 111, // 264: ateapi.Control.ListActors:output_type -> ateapi.ListActorsResponse + 33, // 265: ateapi.Control.CreateAtespace:output_type -> ateapi.Atespace + 33, // 266: ateapi.Control.GetAtespace:output_type -> ateapi.Atespace + 62, // 267: ateapi.Control.ListAtespaces:output_type -> ateapi.ListAtespacesResponse + 33, // 268: ateapi.Control.DeleteAtespace:output_type -> ateapi.Atespace + 35, // 269: ateapi.Control.CreateActorTemplate:output_type -> ateapi.ActorTemplate + 35, // 270: ateapi.Control.GetActorTemplate:output_type -> ateapi.ActorTemplate + 67, // 271: ateapi.Control.ListActorTemplates:output_type -> ateapi.ListActorTemplatesResponse + 35, // 272: ateapi.Control.DeleteActorTemplate:output_type -> ateapi.ActorTemplate + 133, // 273: ateapi.Control.GetGlobalAccessPolicy:output_type -> ateapi.AccessPolicy + 133, // 274: ateapi.Control.CreateGlobalAccessPolicy:output_type -> ateapi.AccessPolicy + 133, // 275: ateapi.Control.UpdateGlobalAccessPolicy:output_type -> ateapi.AccessPolicy + 133, // 276: ateapi.Control.GetAtespaceAccessPolicy:output_type -> ateapi.AccessPolicy + 133, // 277: ateapi.Control.CreateAtespaceAccessPolicy:output_type -> ateapi.AccessPolicy + 133, // 278: ateapi.Control.UpdateAtespaceAccessPolicy:output_type -> ateapi.AccessPolicy + 133, // 279: ateapi.Control.DeleteAtespaceAccessPolicy:output_type -> ateapi.AccessPolicy + 120, // 280: ateapi.WorkerService.RegisterWorker:output_type -> ateapi.RegisterWorkerResponse + 122, // 281: ateapi.WorkerService.MintWorkerActorCertificate:output_type -> ateapi.MintWorkerActorCertificateResponse + 124, // 282: ateapi.WorkerService.MintWorkerActorJwt:output_type -> ateapi.MintWorkerActorJwtResponse + 126, // 283: ateapi.WorkerService.RequestActorSuspend:output_type -> ateapi.RequestActorSuspendResponse + 130, // 284: ateapi.WorkerService.ReportActorSnapshotState:output_type -> ateapi.ReportActorSnapshotStateResponse + 128, // 285: ateapi.WorkerService.ReportActorCrashed:output_type -> ateapi.ReportActorCrashedResponse + 237, // [237:286] is the sub-list for method output_type + 188, // [188:237] is the sub-list for method input_type + 186, // [186:188] is the sub-list for extension type_name + 184, // [184:186] is the sub-list for extension extendee + 0, // [0:184] is the sub-list for field type_name } func init() { file_ateapi_proto_init() } diff --git a/pkg/proto/ateapipb/ateapi.proto b/pkg/proto/ateapipb/ateapi.proto index 9fe2be328d..fcc3a8976b 100644 --- a/pkg/proto/ateapipb/ateapi.proto +++ b/pkg/proto/ateapipb/ateapi.proto @@ -1168,36 +1168,34 @@ message ActorTemplate { // // +k8s:required // +k8s:customValidation # cpu and memory limits are both set - Resources resources = 6; + ResourceRequirements resources = 6; // +k8s:optional ActorTemplateStatus status = 7; } -message Resources { - // limits is the maximum amount of compute resources allowed. Only "cpu" - // and "memory" are supported, each at most once, and each quantity must be - // greater than zero; the cpu limit must be less than 1000 cores. +// ResourceRequirements are the compute resources a workload may use. +message ResourceRequirements { + // limits is the maximum amount of compute resources allowed. // // +k8s:optional - // +k8s:maxItems=2 - // +k8s:listType=map - // +k8s:listMapKey=name - repeated Limits limits = 1; + Resources limits = 1; } -// +k8s:customValidation # names, quantity parse/positivity, cpu bound -message Limits { - // +k8s:required - // +k8s:maxLength=16 # the Limits hook restricts values to cpu/memory - string name = 1; +// Resources is an amount of each compute resource. +message Resources { + // cpu_millicores is CPU in millicores (1000 = one core). 0 means none. + // + // +k8s:optional + // +k8s:minimum=0 + // +k8s:maximum=999999 # less than 1000 cores + int64 cpu_millicores = 1; - // quantity is in Kubernetes resource.Quantity string form (e.g. "500m", - // "2Gi"). + // memory_bytes is memory in bytes. 0 means none. // - // +k8s:required - // +k8s:maxLength=32 # the Limits hook requires a parseable quantity - string quantity = 2; + // +k8s:optional + // +k8s:minimum=0 + int64 memory_bytes = 2; } message GoldenSnapshotStatus { @@ -1345,10 +1343,10 @@ message Container { SecurityContext security_context = 8; // resources carries this container's compute limits, enforced inside the - // actor's sandbox. Only cpu and memory limits are supported. + // actor's sandbox. A limit of 0 leaves that resource unlimited. // // +k8s:optional - Resources resources = 9; + ResourceRequirements resources = 9; } // SecurityContext holds security settings for a container's process, @@ -1502,13 +1500,11 @@ message DurableDirVolumeSource {} // ExternalVolumeTemplate provisions an external volume per actor; the volume // lives only as long as the actor. message ExternalVolumeTemplate { - // capacity of the volume to create, in Kubernetes resource.Quantity string - // form (e.g. "10Gi"). Required. + // capacity_bytes is the size of the volume to create, in bytes. Required. // // +k8s:required - // +k8s:maxLength=32 # matches Limits.quantity's bound - // +k8s:customValidation # must parse as a resource.Quantity - string capacity = 1; + // +k8s:minimum=1 + int64 capacity_bytes = 1; // storage_class_name names the cluster-scoped Kubernetes StorageClass to // create the volume from. Required. @@ -2493,11 +2489,20 @@ message AttributeEntry { // WorkerResources represents a set of resources. message WorkerResources { - // Resources match the Actor's definition of resources. - // They are matched by name and used for placement. + // cpu_millicores is CPU in millicores (1000 = one core). Compared against + // the Actor's resource limits for placement. // // +k8s:optional - Resources resources = 1; + // +k8s:minimum=0 + // +k8s:maximum=999999 # less than 1000 cores + int64 cpu_millicores = 1; + + // memory_bytes is memory in bytes. Compared against the Actor's resource + // limits for placement. + // + // +k8s:optional + // +k8s:minimum=0 + int64 memory_bytes = 2; // A count of Actors. Workers can use this to limit the number of Actors // that can be placed on them, constraining the costs of other things that @@ -2506,7 +2511,7 @@ message WorkerResources { // // +k8s:optional // +k8s:minimum=1 - int32 actors = 2; + int32 actors = 3; } // ActorAssignment binds an Actor to a Worker. This is a subresource of the diff --git a/tools/apitool/exemptions.json b/tools/apitool/exemptions.json index c9e5c86eae..58bfc2b44d 100644 --- a/tools/apitool/exemptions.json +++ b/tools/apitool/exemptions.json @@ -189,16 +189,6 @@ "subject": "ateapi.GoldenSnapshotStatus", "message": "message has no doc comment" }, - { - "rule": "documented", - "subject": "ateapi.Limits", - "message": "message has no doc comment" - }, - { - "rule": "documented", - "subject": "ateapi.Limits.name", - "message": "field has no doc comment" - }, { "rule": "documented", "subject": "ateapi.ListActorTemplatesRequest", @@ -254,11 +244,6 @@ "subject": "ateapi.ListWorkersResponse", "message": "message has no doc comment" }, - { - "rule": "documented", - "subject": "ateapi.Resources", - "message": "message has no doc comment" - }, { "rule": "documented", "subject": "ateapi.ResumeActorRequest",