feat(computer): record a computer's display (computer-recording/v1)

AGENTS.md

@@ -107,7 +107,7 @@ and deserialized through a registry.
 |---|---|
 | `env/` | `Env` base class: a subclass declares `type`, implements `from_dict` and `async deploy(...) -> DeployedEnv`, and may override `reset` and `load_file_artifact_universe`. Built-ins in `env/envs/`: `MCPServerEnv` (`mcp_server`), `MultiEnv` (`multi`, MCP servers plus websites), `ServiceDBEnv` (`service_db`, PostgreSQL), `WebsiteEnv` (`website`), `GatewayEnv` (`gateway_server`), `ComputerDriverEnv` (`computer_driver`, a computer driver's files and what its class declares for each platform it supports, read at put in a process of its own, `agentenv_protocol.computer._declare`, where the driver's own dependencies needn't be installed; its deploy is refused until agent-env creates computers). `env/gateway/gateway.py` is the env-side gateway server, container-only code. While its clock is armed it serves agents `get_time` (`env/gateway/get_time.py`), which answers in UTC and, given an IANA `time_zone`, also in that zone; executors' working `hours` (`env/gateway/triggers.py`) take a fixed `utc_offset` or an IANA `time_zone`. |
 | `artifact/` | Immutable versioned artifacts; every `put()` creates a new version. The document goes to the document store, binary payloads to the object store. Built-ins: `CliArtifact` (`cli`), `FileArtifact` (`file`), `FileArtifactUniverse` (`file_artifact_universe`), `DockerImageArtifact` (`docker_image`), `EnvironmentArtifact` (`environment`), `EnvironmentUniverseArtifact` (`environment_universe`), `SkillArtifact` (`skill`); `[artifacts] type_aliases` maps another stored spelling of a type onto one of them. A `DockerImageArtifact` holds a tar.gz of its image, or, written by `put_ref` (`--image-ref` on `env mcp-server put` and `a2a-agent put`), only a registry reference with its tag pinned to a digest, which sandboxes pull, or a published image's tag with its `image_digest` and its build context, which sandboxes pull by the digest and build from the context when they can't (the infra images, and a bundle agent built from a shipped agent's sources, `Plan.published`), or, written by `put_context`, only its build context, which a VM sandbox builds before running it (`VmSandbox.load_docker_images`) and Modal builds before creating its container (`ModalSandboxProvider.prepare_image`, outside a fallback chain's deadline); anything else that runs images by name (a container on the local provider, a lone server on any provider but Modal) refuses such an image. A build context (`utils/docker/context.py`) is what `docker build` would send, `.dockerignore` applied, written so equal files give equal bytes and an equal `source_digest`. |
-| `task/`, `task_step/` | A `Task` holds its `TaskStep`s inline; `Task.run()` executes them as a DAG. `depends_on` (None means all prior steps) gates a step, independent steps run concurrently, `fail_task_on_error` makes a failure fatal or tolerated, `retry_config` rolls a failed span back through the step journal and re-dispatches it. Built-in steps live in `task_step/task_steps/` (`deploy_env`, `deploy_agent`, `prompt_agent`, the verifiers under `verifiers/`, and more). `task/episode/` runs a task with some of its agents played from outside, as a training loop does (`EpisodeSpec`). |
+| `task/`, `task_step/` | A `Task` holds its `TaskStep`s inline; `Task.run()` executes them as a DAG. `depends_on` (None means all prior steps) gates a step, independent steps run concurrently, `fail_task_on_error` makes a failure fatal or tolerated, `retry_config` rolls a failed span back through the step journal and re-dispatches it. Built-in steps live in `task_step/task_steps/` (`deploy_env`, `deploy_agent`, `prompt_agent`, the verifiers under `verifiers/`, `start_computer_recording` and `end_computer_recording`, whose open recordings `teardown_run`, `teardown_sandboxes` and `reset_env` end first, and more). `task/episode/` runs a task with some of its agents played from outside, as a training loop does (`EpisodeSpec`). |
 | `store/` | Four store ABCs with local and cloud implementations: `DocumentStore` (SQLite, MongoDB, DynamoDB, Firestore with MongoDB compatibility), `ObjectStore` (filesystem, S3, Cloud Storage), `ImageStore` (local OCI registry, ECR), `SecretStore` (env vars or file, AWS Secrets Manager, Google Cloud Secret Manager). `VersionedEntityStore` implements the shared versioned get/put logic, `QueryBuilder` is the immutable chained query API, `store/base.py` holds the error types. `store/routing.py` is the `@local` namespace: while the CLI or a `build_app` server runs, an `@local/…` id and what an `@local` task's run records go to local stores beside the configured ones, and that run's writes to the configured stores are refused. A new backend must pass the conformance kits in `tst/store/`. |
 | `config/` | The `Config` singleton (`get_config`, `configure`, `reset_config`) in `config/runtime.py`, file discovery in `config/loader.py`, and `load_impl`, which resolves `module:Class` pointers. `agent_env.store` re-exports the config names for compatibility. |
 | `providers/` | `providers/sandbox_providers/` holds the sandbox providers `local`, `modal`, `modal_vm`, `e2b`, `sail_vm` (the `sail` extra), `smol_vm` (the `smol` extra), `tensorlake` (the `tensorlake` extra), `tenki` (the `tenki` extra), `vercel` (the `vercel` extra), `boxd` (the `boxd` extra), `daytona` (the `daytona` extra); `[sandbox] default` and `agent_default` accept a comma-separated fallback chain, whose deploys skip a provider that can't run them. A sandbox provider declares what it runs (`ON_THIS_MACHINE`, and for VMs on this machine the address their requests to it arrive at, `REACHES_HOST_AT`; `CONTAINER_ACCEPTS`, `PRIVATE_NETWORK`; `SUPPORTED_SANDBOX_MODES`, derived from whether it implements `create_sandbox` and `create_vm`; and `SANDBOX_ACCEPTS`, the images an agent's sandbox runs, which defaults to those a VM loads when it implements `create_vm` and to `CONTAINER_ACCEPTS` otherwise, so only a provider that implements `create_vm` yet gives an agent a container, like the local one, declares it), each sandbox carries the `platform` of the images it runs (linux/amd64 unless its provider sets another when it creates it; None, any, unchecked, for a local sandbox), and code outside `providers/sandbox_providers/` reads those, never which provider it has. `providers/env_providers/` holds the environment providers: `EnvironmentProvider` (an env's containers and state store), `EnvironmentGatewayProvider` (`gateway`, an env's default), which renders a docker-compose for the gateway and its MCP servers inside the sandbox, and `EnvironmentServerProvider` (`server`), one MCP server in its own container with no gateway in front and no state store; `providers/env_state/` holds env-state providers (`local_postgres` built in). |
@@ -120,7 +120,7 @@ and deserialized through a registry.
 | `explorer/` | Optional local web UI: `agent-env up`, binds loopback `:8234`. |
 | `cli/` | Click CLI with the groups `a2a-agent`, `artifact`, `bundle`, `config`, `env`, `eval`, `plugin`, `run`, `task`, `up`. `task run`, `task run-batch` and `eval run` hand their runs to the configured `[runner]` and follow them. |
 | `agents/` | The `agents` bundle: the agents agent-env ships, one folder each under `agents/agents/`, with a smoke task each under `agents/tasks/`. Its Claude Code, `@local/agentenv-framework/agents/claude-code`, is the default agent. An agent's Python packages are its folder's `requirements.txt`, installed in its image, never in agent-env's own environment. A run that pulls one of its agents reads the whole bundle, so `tst/unit/bundle/shipped_bundles_test.py` parses and checks every shipped bundle. |
-| `drivers/` | The `drivers` bundle: the computer drivers agent-env ships as `computer_driver` envs, the reference driver (X11 on Linux) aliased `reference`. It has no tasks: a bundle with neither tasks nor evals is refused unless its `aliases.toml` names something for other bundles to use. `tst/integration/computer/` serves it on a scaffold desktop image and runs the conformance probe and an agent against it. |
+| `drivers/` | The `drivers` bundle: the computer drivers agent-env ships as `computer_driver` envs, the reference driver (X11 on Linux, recording through `x11grab`) aliased `reference`. It has no tasks: a bundle with neither tasks nor evals is refused unless its `aliases.toml` names something for other bundles to use. `tst/integration/computer/` serves it on a scaffold desktop image and runs the conformance probe and an agent against it. |
 | `examples/` | The example bundles agent-env ships (`hello`, and `echo-mcp`, which needs Docker). Tests run them, but they are examples for users, not fixtures. Every shipped bundle is registered under `agent_env.bundles` in `pyproject.toml`, `<name> = "<package>"` shipping `src/<package as a path>/<name>/`. |
 
 ## Extension points

packages/agentenv-protocol/README.md

@@ -367,6 +367,7 @@ pixels), at the display's size times its scale. Input operations may return an `
 | `@key_down`, `@key_up` | both or none | `computer-keyboard/v1` and its two tools |
 | `@run_in_session` | no; a Linux default | `computer-exec/v1` |
 | `@write_files` | no; a Linux default | `data/add` |
+| `@record` | no, with `recording = Recording()` | the stream `computer-recording/v1` records from; without it the base class polls `@capture` |
 
 The class is checked when it is defined: a missing required operation, half a group, two methods
 for one operation that some platform could both match, or features that don't match the
@@ -398,8 +399,9 @@ driver has them, through the `served_tools`, `served_extensions` and `served_ope
 - **Framework extensions.** `computer-ready/v1` (`status`: real checks of the display and a
   capture, plus the driver's own `ready_checks()`; tools refuse `device_unavailable` until they
   pass once), `computer-display/v1`, `computer-spaces/v1` (the default space, written into the tool
-  descriptions), `computer-keyboard/v1` and `computer-exec/v1` (`run {argv, cwd?, env?, timeout_s?,
-  request_id}`: output capped per stream, a repeated `request_id` answered with the first result).
+  descriptions), `computer-keyboard/v1`, `computer-exec/v1` (`run {argv, cwd?, env?, timeout_s?,
+  request_id}`: output capped per stream, a repeated `request_id` answered with the first result) and,
+  with `recording`, `computer-recording/v1` (below).
 - **The data plane.** `data/add` takes file parts that name their path, relative to the session
   user's home, under the `data-objects/v1` marker (`{"path": ...}`), as inline bytes, an https URL,
   or a `transfers.ReadObject` read grant under the marker's `read`, and writes them atomically.
@@ -425,6 +427,44 @@ names the caller whose latest frame and others' input a call uses.
 `MyDriver.declaration(platform)` returns the card the class serves on a `Platform`, built without
 running the driver, its operations or its dependencies.
 
+### Recording the display
+
+A driver that declares `recording` serves `computer-recording/v1`: video of the display, as MP4
+(H.264) segments, for the framework to start and end around the work it wants on record. Encoding
+needs PyAV, from the `recording` extra (`pip install 'agentenv-framework-protocol[computer,recording]'`).
+
+```python
+from agentenv_protocol.computer import RecordedFrame, Recording, record
+
+
+class MyDriver(ComputerDriverEnvironment):
+    recording = Recording(max_fps=30)      # max_concurrent=2, max_duration_s=3600, max_segment_s=300
+
+    @record
+    async def stream(self, hints):         # hints.scope, fps, max_edge, cursor, max_segment_s, directory
+        async for image, captured_at in my_screen_stream(hints.fps):
+            yield RecordedFrame(image, captured_at)   # a Pillow image or a PyAV VideoFrame
+```
+
+- **The stream.** `@record` yields `RecordedFrame`s from a stream the OS pushes, which the base class
+  scales, encodes and splits into segments, each frame timed by its `captured_at`; or
+  `RecordedSegment`s, files a native recorder finished under `hints.directory`, which it only hashes,
+  uploads and lists. The base class closes the stream at `end`. A driver without `@record` records by
+  the base class polling `@capture`, with a pointer drawn at `@pointer_position`: the slow path, at
+  most 15 fps.
+- **The methods.** `start {request_id, scope?, fps?, max_edge?, cursor?, upload?}` returns
+  `{recording_id, started_at, format}`; `end {recording_id, upload?, reason?}` closes the last segment
+  and returns the manifest. A repeated `request_id`, or a repeated `end`, answers the same. A start
+  past `max_concurrent` is refused `limit_exceeded`; `max_duration_s`, free disk under 512 MB or a stream
+  failing for 60 s end a recording by itself.
+- **Segments.** A segment closes at `max_segment_s`, when the frame size changes or after a gap in the
+  stream. Each uploads as it closes under `start`'s `write_namespace` grant, at
+  `<recording_id>/<index>.mp4`, and is then deleted; `end`'s own grant uploads any that missed. With no
+  grant they stay under `~/.agentenv-recordings/`.
+- **The manifest.** Its `segments` carry each one's path, `started_at` and `ended_at` (its first and
+  last frames' capture times), size, frame count, byte size, SHA-256 and whether it uploaded, and
+  `ended_by` says why the recording ended: `end`, `reset`, `teardown`, `limit` or `error`.
+
 ### Checking a driver: the conformance probe
 
 The probe checks a running driver against the contract through its card, tools and extensions. It
@@ -447,7 +487,7 @@ The CLI exits 0 when no check failed, 1 when one did and 2 when it can't reach t
 | Level | Checks | Needs on the computer |
 |---|---|---|
 | `basic` | P0 ready and a screenshot; P1 the card and tool schemas; P2 native frames, regions and JPEG; P5 refusals and their codes; X the declared extensions (`computer-display`, `computer-spaces`, `computer-keyboard`, `computer-exec`, `data/get`, per-role tool access, the pointer) | nothing |
-| `full` | `basic`, then P3 colours land where the mapping says; P4 clicks show their effect; P5 refusals change nothing; P6 typed and pressed keys arrive; P7 scrolls and drags arrive; P9 commands and `data/add` run as one unprivileged user; X pointer and keyboard input, `data/add` | Linux, `python3` with Tk, a 1280x800 screen or larger, `computer-exec` and `data/add` |
+| `full` | `basic`, then P3 colours land where the mapping says; P4 clicks show their effect; P5 refusals change nothing; P6 typed and pressed keys arrive; P7 scrolls and drags arrive; P9 commands and `data/add` run as one unprivileged user; X pointer and keyboard input, `data/add`, and a recording's segments matching their files with no gap, a repeated `request_id` joining it and `max_concurrent` held | Linux, `python3` with Tk, a 1280x800 screen or larger, `computer-exec` and `data/add` |
 
 At `full` the probe writes the Fixture Desk, a Tk window (`conformance/fixture.py`), into
 `~/.agentenv-probe` with `data/add` and starts it with `computer-exec`. It aims at the boxes the

packages/agentenv-protocol/pyproject.toml

@@ -41,6 +41,10 @@ computer = [
     "Pillow>=10.4",
     "uvicorn>=0.30",
 ]
+# A computer driver that records: computer-recording/v1 encodes H.264 with PyAV.
+recording = [
+    "av>=17",
+]
 dev = [
     "a2a-sdk[http-server]>=0.3.26,<0.4",
     "python-multipart>=0.0.13",
@@ -50,6 +54,7 @@ dev = [
     "mcp>=1.25,<2",
     "fastmcp>=3.4,<4",  # the standalone-fastmcp mount path
     "Pillow>=10.4",  # the computer extra's frame crops and encodings
+    "av>=17",  # the recording extra's encoder, and the tests' decoder
 ]
 [project.urls]
 Homepage = "https://www.agentenvframework.com"

packages/agentenv-protocol/src/agentenv_protocol/computer/__init__.py

@@ -25,6 +25,7 @@
     COMPUTER_KEYBOARD,
     COMPUTER_POINTER,
     COMPUTER_READY,
+    COMPUTER_RECORDING,
     COMPUTER_SPACES,
 )
 from .operations import (
@@ -44,24 +45,26 @@
     pointer_position,
     pointer_up,
     press,
+    record,
     run_in_session,
     scroll,
     set_display,
     type_text,
     write_files,
 )
 from .platform import Platform
+from .recording import RecordedFrame, RecordedSegment, RecordHints, Recording
 from .settings import SETTINGS_ENV, ComputerSettings
 
 __all__ = [
     "ComputerDriverEnvironment", "computer_driver", "DEFAULT_IDENTITY",
     "Display", "Image", "Effect", "ExecResult", "ReadyCheck", "Platform", "ComputerSettings", "SETTINGS_ENV",
     "display", "set_display", "capture", "click", "drag", "scroll", "type_text", "press",
     "pointer_move", "pointer_down", "pointer_up", "pointer_position", "key_down", "key_up",
-    "run_in_session", "write_files",
+    "run_in_session", "write_files", "record", "Recording", "RecordHints", "RecordedFrame", "RecordedSegment",
     "ComputerError", "InvalidArgument", "OutOfBounds", "StaleFrame", "InputBusy", "PermissionDenied", "Unsupported",
     "BackgroundUnavailable", "TargetUnavailable", "DeviceUnavailable", "LimitExceeded", "Timeout", "DeliveryFailed",
     "Internal",
     "COMPUTER", "COMPUTER_READY", "COMPUTER_DISPLAY", "COMPUTER_SPACES", "COMPUTER_POINTER", "COMPUTER_KEYBOARD",
-    "COMPUTER_EXEC",
+    "COMPUTER_EXEC", "COMPUTER_RECORDING",
 ]

packages/agentenv-protocol/src/agentenv_protocol/computer/card.py

@@ -5,6 +5,7 @@
 
 from typing import TYPE_CHECKING
 
+from .recording import recording_params
 from ..types import METHOD_ADD, EnvironmentExtension, EnvironmentTool, IntakeDeclaration, IntakeFormat, intake_extension
 
 if TYPE_CHECKING:
@@ -17,6 +18,7 @@
 COMPUTER_POINTER = "urn:agentenv:computer-pointer/v1"
 COMPUTER_KEYBOARD = "urn:agentenv:computer-keyboard/v1"
 COMPUTER_EXEC = "urn:agentenv:computer-exec/v1"
+COMPUTER_RECORDING = "urn:agentenv:computer-recording/v1"
 
 MAX_EXEC_TIMEOUT_S = 3600
 KEYBOARD_RANGES = {"tap_hold_ms": (0, 1000), "type_delay_ms": (0, 1000), "max_hold_ms": (0, 600_000)}
@@ -48,6 +50,8 @@ def served_extensions(driver: ComputerDriverEnvironment,
         need = _EXTENSION_NEEDS.get(extension.uri)
         if need is not None and not driver.has(need):
             continue
+        if extension.uri == COMPUTER_RECORDING and driver.recording is None:
+            continue
         dropped = {op for (uri, op), need in _METHOD_NEEDS.items() if uri == extension.uri and not driver.has(need)}
         if dropped:
             methods = {op: entry for op, entry in extension.params["methods"].items() if op not in dropped}
@@ -70,6 +74,8 @@ def extension_params(driver: ComputerDriverEnvironment, uri: str) -> dict:
                 "layouts": list(driver.layouts)}
     if uri == COMPUTER_EXEC:
         return {"max_output_bytes": driver.settings.exec_output_bytes, "max_timeout_s": MAX_EXEC_TIMEOUT_S}
+    if uri == COMPUTER_RECORDING and driver.recording is not None:
+        return recording_params(driver.recording, driver.has("record"))
     return {}
 
 

packages/agentenv-protocol/src/agentenv_protocol/computer/checks.py

@@ -5,6 +5,7 @@
 from ..errors import AgentEnvError
 from .keys import canonical
 from .operations import declared
+from .recording import check_recording
 
 BUTTONS = ("left", "right", "middle", "back", "forward")
 
@@ -38,5 +39,8 @@ def check_features(cls: type) -> None:
             problems.append(f"keys: {e.message}")
     if ("set_display" in declared(cls)) != bool(cls.display_modes):
         problems.append("display_modes and @set_display come together")
+    problems += check_recording(cls.recording)
+    if "record" in declared(cls) and cls.recording is None:
+        problems.append("@record needs recording = Recording()")
     if problems:
         raise TypeError(f"{cls.__qualname__}: " + "; ".join(problems))