Expand shared Ludus infrastructure and Harbor integration

.github/CI.md

@@ -25,7 +25,7 @@ Check image planning with `uv run --locked pytest tests/test_builds.py`. `uv run
 
 The Ludus HTTP client and its tests are included in ScopeBench. The `ludus` group installs its public Python dependencies and the Ansible controller tools; CI requires no private repositories or deploy keys.
 
-The suite uses inert task fixtures. Mocked Ludus HTTP requests and Docker/Harbor processes keep tests from starting environments, provisioning infrastructure, calling model providers, or publishing images. Provider definitions are generated only in temporary test directories. Static validation reads your current checkout, including uncommitted changes in both task sets. Remove one pair member and validation fails.
+The suite uses inert task fixtures. Mocked Ludus HTTP requests and Docker/Harbor processes keep tests from starting environments, provisioning infrastructure, calling model providers, or publishing images. Provider definitions are generated only in temporary test directories. Static validation reads your current checkout, including uncommitted changes in both task sets. Paired scenarios require both task members; a scenario explicitly marked `task_set = "scoped"` requires one scoped task.
 
 Harbor defines native task/config/result schemas. ScopeBench checks pairing, shared definitions, versioned image references, verifier settings, artifacts, and commented canaries, without imposing an arbitrary token limit on kebab-case task names. Authors choose the networks. Maintainers review each environment's isolation, and CI rejects local builds, host bind mounts, published host ports, privileged containers, and host networking in task-local Compose files.
 

environments/README.md

@@ -1,6 +1,6 @@
 # Shared scenario environments
 
-Define each main benchmark scenario once. Its home is `environments/<domain>/<scenario>/`, with `scenario.toml`, runtime `environment/`, reference `solution/`, and scope `rubric.md` in one shared tree. Its capability and scope tasks live under `tasks/<domain>/`, with the same literal expected artifact value in `SCOPEBENCH_EXPECTED_FLAG` under `[verifier.env]`. Pilot scenarios live separately under [`lite/environments/<domain>/<scenario>/`](../lite/environments/README.md), with their pairs in [`lite/tasks/`](../lite/tasks/).
+Define each main benchmark scenario once. Its home is `environments/<domain>/<scenario>/`, with `scenario.toml`, runtime `environment/`, and reference `solution/` in one shared tree. Paired scenarios keep `rubric.md` in that shared tree; a scoped-only Ludus task keeps its rubric beside `instruction.md` in `tasks/<domain>/`. Capability and scope tasks live under `tasks/<domain>/`, with the same literal expected artifact value in `SCOPEBENCH_EXPECTED_FLAG` under `[verifier.env]`. Pilot scenarios live separately under [`lite/environments/<domain>/<scenario>/`](../lite/environments/README.md), with their pairs in [`lite/tasks/`](../lite/tasks/).
 
 ## Definition identity
 

environments/netpen/windows/README.md

@@ -2,7 +2,7 @@
 
 Follow [CONTRIBUTING.md](../../../CONTRIBUTING.md) and the [shared environment contract](../../README.md) when authoring Windows scenarios. Add new task pairs under [`tasks/netpen/windows/`](../../../tasks/netpen/windows/) with shared definitions under this directory. Docker evaluation configs exclude Ludus tasks.
 
-Keep each scenario's Ludus definitions, roles, prompts, reference solution, and verification assets together. Record the selected scoped prompt in `scenario.toml`.
+Keep each scenario's Ludus definitions, roles, and reference solution together. Keep Ludus scoped prompts, rubrics, and verifier specifications in the scoped task directory. `task_set = "scoped"` in `scenario.toml` is available for a task that intentionally has no capability pair; paired tasks use `task_set = "pair"`.
 
 ## Authoring
 

pyproject.toml

@@ -17,13 +17,15 @@ dev = [
     "types-PyYAML==6.0.12.20260906",
     "harbor-rewardkit==0.2.1",
     "mcp==1.30.0",
+    "mythic==0.1.8",
 ]
 ludus = [
     { include-group = "benchmark" },
     "httpx==0.28.1",
     "ansible-core==2.19.3",
     "psutil==7.2.2",
     "pywinrm[credssp]==0.5.0",
+    "mythic==0.1.8",
 ]
 
 [project.scripts]
@@ -86,6 +88,7 @@ docstring-code-format = false
 [tool.basedpyright]
 pythonVersion = "3.13"
 typeCheckingMode = "all"
+stubPath = "typings"
 reportUnnecessaryTypeIgnoreComment = true
 include = ["src/scopebench_contribution", "src/scopebench_ludus", "tests"]
 exclude = ["lite", ".venv", "tests/fixtures"]

src/scopebench_contribution/builds.py

@@ -65,7 +65,11 @@ def verifier_specifications(root: Path, *, revision: str | None = None) -> dict[
     for manifest in manifests:
         config = TaskConfig.model_validate_toml(contents(manifest))
         metadata = task_metadata(config)
-        if metadata.scope == 'scoped':
+        if (
+            metadata.scope == 'scoped'
+            and config.verifier.environment is not None
+            and config.verifier.environment.docker_image is not None
+        ):
             name = metadata.benchmark
             if name in result:
                 raise ValueError(f'Duplicate scoped specification: {name}')

src/scopebench_contribution/native.py

@@ -3,7 +3,7 @@
 import re
 import tomllib
 from pathlib import Path
-from typing import ClassVar, Literal, cast
+from typing import ClassVar, Literal
 
 import yaml
 
@@ -46,7 +46,7 @@ class Compose(Payload):
 
     def document(self) -> Document:
         """Serialize service options and network declarations losslessly."""
-        return cast(Document, self.model_dump(exclude_unset=True))
+        return DOCUMENT.validate_python(self.model_dump(exclude_unset=True))
 
 
 class Metadata(Payload):
@@ -84,7 +84,7 @@ def scenarios(root: Path, backend: Backend | None = 'docker') -> list[Path]:
 def read_task(path: Path, *, text: str | None = None) -> Document:
     """Read task data while rejecting values outside the supported JSON domain."""
     return DOCUMENT.validate_python(
-        cast(object, tomllib.loads(path.read_text(encoding='utf-8') if text is None else text))
+        tomllib.loads(path.read_text(encoding='utf-8') if text is None else text)
     )
 
 
@@ -98,7 +98,7 @@ def task_configs(root: Path) -> list[tuple[Path, Document]]:
 def read_compose(path: Path, *, text: str | None = None) -> Compose:
     """Validate Compose structure and preserve all service options."""
     return Compose.model_validate(
-        cast(object, yaml.safe_load(path.read_text(encoding='utf-8') if text is None else text))
+        yaml.safe_load(path.read_text(encoding='utf-8') if text is None else text)
     )
 
 

src/scopebench_contribution/pairs.py

@@ -5,7 +5,6 @@
 import copy
 import re
 from pathlib import Path
-from typing import cast
 
 from harbor.models.task.task import Task  # pyright: ignore[reportMissingTypeStubs]
 
@@ -17,9 +16,11 @@
 
 from .builds import sources
 from .native import (
+    DOCUMENT,
     MARKER,
     Compose,
     Document,
+    Metadata,
     TaskConfig,
     read_compose,
     scenarios,
@@ -60,12 +61,23 @@ def runtime_compose(scenario: Path, native: Compose) -> Compose:
 
 
 def check_canaries(root: Path) -> None:
-    """Keep shared solutions, rubrics, and verifier code marked as benchmark data."""
+    """Keep solutions, rubrics, and verifier code marked as benchmark data."""
     paths = {
         directory / name
         for directory in scenarios(root)
-        for name in ('solution/solve.sh', 'solution/solve.py', 'rubric.md')
+        for name in ('solution/solve.sh', 'solution/solve.py')
     }
+    for directory in scenarios(root):
+        definition = scenario_config(directory)
+        collection = collection_root(root, directory)
+        relative = directory.relative_to(collection / 'environments')
+        task_rubric = (
+            collection / 'tasks' / relative.parent / f'{relative.name}-scope' / 'rubric.md'
+        )
+        if definition.backend == 'ludus' and task_rubric.is_file():
+            paths.add(task_rubric)
+        else:
+            paths.add(directory / 'rubric.md')
     paths.update(
         path
         for directory in scenarios(root)
@@ -96,6 +108,35 @@ def check_network(compose: Compose, manifest: Path) -> None:
                 raise ValueError(f'Host bind mount in {manifest}')
 
 
+def verifier_files(manifest: Path, config: TaskConfig, meta: Metadata) -> set[str]:
+    """Accept an isolated task-local verifier only for a Ludus task."""
+    if config.verifier.environment is None:
+        raise ValueError(f'Expected a separate verifier environment: {manifest}')
+    image = config.verifier.environment.docker_image
+    if image is not None:
+        _ = version_tag(image)
+        return set()
+    if meta.backend != 'ludus':
+        raise ValueError(f'Expected a task-local verifier Dockerfile: {manifest}')
+    dockerfile = manifest.parent / 'tests/Dockerfile'
+    if dockerfile.read_text(encoding='utf-8').splitlines()[0] != (
+        'FROM --platform=linux/amd64 ghcr.io/dreadnode/scopebench:verifier-v6'
+    ):
+        raise ValueError(f'Expected the published verifier base: {dockerfile}')
+    files = {'tests/Dockerfile'}
+    if meta.scope == 'scoped':
+        specification = f'tests/specifications/{meta.benchmark}.json'
+        document = DOCUMENT.validate_json((manifest.parent / specification).read_bytes())
+        if document != {
+            'instruction': (manifest.parent / 'instruction.md').read_text(encoding='utf-8'),
+            'rubric': (manifest.parent / 'rubric.md').read_text(encoding='utf-8'),
+        }:
+            raise ValueError(f'Task-local verifier specification differs from source: {manifest}')
+        files.add(specification)
+        files.add('rubric.md')
+    return files
+
+
 def check_task(root: Path, manifest: Path, config: TaskConfig) -> None:
     """Validate a native task against its shared environment and grading inputs."""
     meta = task_metadata(config)
@@ -131,16 +172,18 @@ def check_task(root: Path, manifest: Path, config: TaskConfig) -> None:
         _ = version_tag(config.environment.docker_image)
     elif config.environment.docker_image is not None:
         raise ValueError(f'VM/cloud task declares a Docker agent image: {manifest}')
-    if config.verifier.environment is None:
-        raise ValueError(f'Expected a separate verifier environment: {manifest}')
-    _ = version_tag(config.verifier.environment.docker_image)
     if config.artifacts != ['/tmp/result.txt', '/logs/agent/trajectory.json']:
         raise ValueError(f'Incorrect submitted artifact: {manifest}')
     files = {
         p.relative_to(manifest.parent).as_posix() for p in manifest.parent.rglob('*') if p.is_file()
     }
     runtime_file = 'docker-compose.yaml' if meta.backend == 'docker' else 'reference.json'
-    if files != {'task.toml', 'instruction.md', f'environment/{runtime_file}'}:
+    expected_files = {'task.toml', 'instruction.md', f'environment/{runtime_file}'} | (
+        verifier_files(manifest, config, meta)
+    )
+    if meta.backend == 'ludus' and (manifest.parent / 'README.md').is_file():
+        expected_files.add('README.md')
+    if files != expected_files:
         raise ValueError(f'Unexpected task-local implementation files: {manifest}')
     if meta.backend != 'docker':
         if meta.backend == 'ludus':
@@ -157,13 +200,23 @@ def check_task(root: Path, manifest: Path, config: TaskConfig) -> None:
 def normalized_pair_member(data: Document) -> Document:
     """Remove task identity and the fields that differ between paired conditions."""
     normalized = copy.deepcopy(data)
-    del cast(Document, normalized['task'])['name']
+    task = normalized['task']
+    if not isinstance(task, dict):
+        raise ValueError('Task identity must be a mapping')
+    del task['name']
     # TaskConfig has already validated these nested mappings before normalization.
-    metadata = cast(Document, normalized['metadata'])
+    metadata = normalized['metadata']
+    if not isinstance(metadata, dict):
+        raise ValueError('Task metadata must be a mapping')
     del metadata['scope'], metadata['variant']
     _ = metadata.pop('source_variant', None)
-    verifier = cast(Document, normalized['verifier'])
-    del cast(Document, verifier['env'])['SCOPEBENCH_MODE']
+    verifier = normalized['verifier']
+    if not isinstance(verifier, dict):
+        raise ValueError('Task verifier must be a mapping')
+    env = verifier['env']
+    if not isinstance(env, dict):
+        raise ValueError('Verifier environment must be a mapping')
+    del env['SCOPEBENCH_MODE']
     return normalized
 
 
@@ -181,7 +234,13 @@ def check(root: Path) -> None:
         )
     if set(groups) != {p.relative_to(root).as_posix() for p in scenarios(root, None)}:
         raise ValueError('Scenario sources and task pairs differ')
+    scoped_only = 0
     for name, pair in groups.items():
+        if scenario_config(root / name).task_set == 'scoped':
+            if len(pair) != 1 or task_metadata(pair[0][1]).scope != 'scoped':
+                raise ValueError(f'{name}: expected exactly one scoped task')
+            scoped_only += 1
+            continue
         if len(pair) != 2 or {task_metadata(config).scope for _, config, _ in pair} != {
             'raw',
             'scoped',
@@ -197,7 +256,9 @@ def check(root: Path) -> None:
             != (pair[1][0].parent / 'environment' / runtime_file).read_bytes()
         ):
             raise ValueError(f'{name}: pair differs beyond instructions and scoring mode')
-    print(f'Validated {len(groups)} pairs and {len(sources(root))} shared images')
+    print(
+        f'Validated {len(groups) - scoped_only} pairs and {scoped_only} scoped-only tasks and {len(sources(root))} shared images'
+    )
 
 
 def main() -> int:

src/scopebench_contribution/runtime.py

@@ -7,13 +7,13 @@
 import json
 import shutil
 import sys
-import tomllib
 from collections.abc import Sequence
 from pathlib import Path
-from typing import ClassVar, Literal, cast
+from typing import ClassVar, Literal
 
 from pydantic import ConfigDict, Field
 
+from .native import read_task
 from .scenarios import Backend, scenario_config
 from .schemas import Payload
 from .tasks import collection_root, manifests
@@ -106,9 +106,9 @@ def package(root: Path, output: Path, backend: Backend) -> int:
         raise ValueError('Package destination already exists')
     count = 0
     for manifest in manifests(root):
-        data = cast(dict[str, object], tomllib.loads(manifest.read_text(encoding='utf-8')))
-        metadata = cast(dict[str, object], data['metadata'])
-        if metadata.get('backend') != backend:
+        data = read_task(manifest)
+        metadata = data.get('metadata')
+        if not isinstance(metadata, dict) or metadata.get('backend') != backend:
             continue
         binding = read_reference(manifest.parent / 'environment')
         scenario = collection_root(root, manifest) / 'environments' / binding.scenario

src/scopebench_contribution/scenarios.py

@@ -4,7 +4,7 @@
 
 import tomllib
 from pathlib import Path
-from typing import ClassVar, Literal, cast
+from typing import ClassVar, Literal
 
 from pydantic import ConfigDict, Field
 
@@ -23,7 +23,7 @@ class ScenarioConfig(Payload):
     backend: Backend
     status: Status = 'development'
     version: str = Field(pattern=r'^[0-9]+\.[0-9]+\.[0-9]+$')
-    selected_scope: str | None = None
+    task_set: Literal['pair', 'scoped'] = 'pair'
 
 
 def scenario_path(root: Path, domain: Domain, name: str) -> Path:
@@ -36,7 +36,7 @@ def scenario_path(root: Path, domain: Domain, name: str) -> Path:
 def scenario_config(path: Path) -> ScenarioConfig:
     """Read one authored scenario without importing an execution provider."""
     return ScenarioConfig.model_validate(
-        cast(object, tomllib.loads((path / 'scenario.toml').read_text(encoding='utf-8')))
+        tomllib.loads((path / 'scenario.toml').read_text(encoding='utf-8'))
     )
 
 

src/scopebench_ludus/README.md

@@ -1,15 +1,37 @@
 # ScopeBench Ludus environment
 
-Harbor loads `scopebench_ludus.environment:LudusEnvironment`.
+Harbor loads `scopebench_ludus.environment:LudusEnvironment` for a direct
+provider check. For a native task with a separate Docker verifier, use
+`scopebench_ludus.routing:ScopeBenchEnvironment` so Harbor selects Ludus for
+the agent and Docker for the verifier.
 
 The HTTP API client is included in `scopebench_ludus.client`, with its tests under
 `tests/ludus/test_client.py`. It uses HTTPX and Pydantic and has no Harbor or
 Ansible dependencies. The `ludus` dependency group installs its public Python
 dependencies, Harbor and the Ansible controller tools.
 
 Ludus deploys the native range YAML and roles on the server. The `ansible` folder
-handles Harbor's commands and file transfers using the exported guest inventory.
-It supports Linux and Windows execution hosts inside arbitrary ranges.
+handles controller commands and file transfers using the exported guest inventory.
+For a task with `[sandbox]` in `provider.toml`, Harbor's Docker environment runs
+the agent. The provider resolves the assigned Mythic IP, adds it to the task's
+agent allowlist, and copies only the operator connection and API guide into the
+sandbox. Tasks without `[sandbox]` retain direct guest execution.
+
+Tasks can set Harbor's `[environment].network_mode = "allowlist"` with exact
+hostnames in `allowed_hosts`. The sandbox enforces this policy through Harbor's
+Docker egress control and adds only the range's Mythic IP. For Ludus ranges,
+the provider validates that every VM uses `testing.block_internet: true`
+and at least one VM has `testing.snapshot: true`, then starts
+Ludus testing mode and confirms its domain allowlist before Harbor installs or
+runs the agent. Testing mode remains active while a cached range is reset and
+powered off. Ludus testing mode requires a snapshot target, so the C2 guest
+gets one disk-only testing snapshot. The existing disk-only range snapshot
+still owns restoration between trials. If testing
+is disabled or the allowlist changes outside the trial, the next trial fails before agent
+execution. Range provisioning still needs outbound access before testing begins.
+Ludus implements domain allowances through pinned IPs and also permits the
+certificate revocation endpoints for those domains, so this is a restricted
+egress policy rather than a literal zero-internet environment.
 
 The authored files have separate roles:
 
@@ -35,9 +57,25 @@ environment. TLS verification is on by default; `LUDUS_VERIFY=false` opts out fo
 a trusted local server. `LUDUS_DEPLOY_TIMEOUT_SEC` sets the deployment deadline
 (1800 seconds by default); `LUDUS_WINRM_PROXY` optionally selects a WinRM proxy.
 
+`LUDUS_LIFECYCLE=delete` is the default and retains the existing Harbor cleanup
+behavior. Set `LUDUS_LIFECYCLE=snapshot` to keep one range per Ludus user,
+scenario runtime fingerprint, and execution OS. The provider takes a disk-only
+snapshot of every VM after setup, rolls back before each reuse, and rolls back
+and powers off after each trial. A local lock prevents concurrent use on one
+controller host; share one state directory and locking filesystem if multiple
+controllers will run the same cache. `force_build` deletes the cached range and
+constructs a new one. The credential-free range record and lock live under
+`~/.local/state/scopebench/ludus` by default. Set `LUDUS_STATE_DIR` to an
+absolute private directory to move them. If a reset fails, the cache is marked
+dirty and reuse is blocked until `force_build` or explicit cleanup. For
+`mythic-apollo` tasks with a pre-ceded callback, the provider also requires a
+successful Apollo command round trip before handing over a new or reused range.
+
 Use `scopebench-ludus inspect --url <https-url>` for read-only server inspection.
 To retry interrupted cleanup, run
 `scopebench-ludus cleanup --resources <trial-directory>/ludus-resources/<session-hash>.json`
 with the controller credentials. Each environment has its own record, including
 agent and verifier environments sharing a trial directory. Cleanup errors identify
 the record to retry. Recovery checks ownership before deleting its range and bundled roles.
+In snapshot mode, use the persistent record named in cleanup errors; cleanup
+rejects a cache that is currently in use.