[chruffins]

Summary

PR 4 of 6. Adds macOS stopped snapshots, same-instance memory snapshots and restore, snapshot schedules, and stopped clones. Base is #500, now merged to main.

• Stopped snapshots and cold clones require a confirmed graceful stop. A clone gets a fresh Apple machine identifier, a MAC assigned under the fork admission lock, and an SSH host key rekey on first start. A failed rekey stops the clone and is retried on the next start.
• Memory snapshots of Running or Standby instances validate the exact VZ configuration before pausing. Disk, auxiliary storage and machine state are captured while the VM is down. A formerly Running source resumes afterwards, even if the request is cancelled.
• Same-instance restore validates the payload before anything is replaced, then swaps metadata and storage in together, with crash recovery.
• Interrupted restore: if the API dies mid-restore, the leftover VM comes up Paused. POST /restore resumes it and finishes the restore.
• Snapshot schedules use the existing controller and retention. They run only where macOS save/restore is supported.
• Boot admission: start and restore check machine identity and MAC uniqueness under one boot lock.
• Warm memory forks are rejected. VZ restores saved state only into an identical configuration (MAC, machine identifier, NIC and auxiliary storage), and Apple documents concurrent VMs sharing an identifier as undefined. Repeated same-instance restore is the warm start.

Existing and new flows

Pseudocode for each flow this PR changes. The Existing path is main. Unchanged marks branches that behave as before for Linux.

Stop and the graceful-stop receipt

Existing: stopInstance -> tryGracefulGuestShutdown -> fallback hypervisor shutdown -> save metadata
          (no receipt field; stopped snapshots and forks are not gated on how the stop ended)
New:      stopInstance -> tryGracefulGuestShutdown -> fallback hypervisor shutdown
          if macOS:
              clear any old vz-shim stop receipt, send the shutdown RPC
              tryGracefulGuestShutdown returns true only if the VMM it signalled exits
                and the shim wrote its stop receipt (the guest stopped the VM itself)
              pid == 0 (VMM already gone, nothing signalled): macOS returns false (Linux still true)
              StopGraceful = that result                 (no separate pre-stop liveness probe)
          save metadata

Start

Existing: startInstance -> checkMacOSMACAvailable -> startAndBootVM
          startAndBootVM -> admitMacOSBoot (socket paths, identity, MAC under macOSBootMu) -> StartVM
New:      startInstance -> checkMacOSMACAvailable -> startAndBootVM
          startAndBootVM -> admitMacOSBoot                  (unchanged)
          if macOS: clearMacOSStopReceipt (persisted)       (new; before StartVM)
          StartVM
          A start rejected at admission keeps the receipt; the disk was not touched.

Stopped snapshot and memory snapshot create

Existing: createSnapshot -> admitGuestOperation refuses macOS ("not implemented") -> capture
New:      createSnapshot -> admitGuestOperation (guest-type check only)
          validateMacOSSnapshotCreate:
              Stopped:  requires StopGraceful
              Standby:  rejects enabled compression; validates VZ support
          Running source (memory kind):
              validateMacOSSaveRestoreSupport (VZ configuration probe, before pausing)
              pause (waits for raw Paused) -> capture disk, aux and machine state while VMM is down
              cancellation after pause: resume with bounded cleanup contexts
          Stopped source (stopped kind): existing capture path
          macOS compression policy resolves to disabled (no codec job)

Restore from snapshot (restoreSnapshot)

Existing: validate source -> admitGuestOperation refuses macOS -> replaceInstanceWithSnapshotPayload -> save metadata
New:      validateMacOSSnapshotRestore (kind, hardware model, machine identifier, MAC match)
          validateMacOSSnapshotRestoreState (source must be Stopped or Standby)
          admitGuestOperation (guest-type check only)
          if macOS:
              validateMacOSMemorySnapshotPayload (manifest, storage, bundle; before any swap)
              prepareMacOSSnapshotRestoreMetadata (clears restart block; manual stop only for Stopped target)
              replaceMacOSInstanceWithSnapshotPayload
                stage payload + metadata in guests/<id>/restore-staging/staged
                retire instance files to previous/, rename staged -> installing (commit), install
                crash before commit: roll back; after: finish install (server start or next restore)
              metadata is already committed by the staged write; no second save
          else (Linux, unchanged):
              replaceInstanceWithSnapshotPayload -> save metadata

In-place restore (RestoreInstance and restoreInstance)

Existing: RestoreInstance
              Running/Initializing -> return
              Paused               -> resume -> return
          restoreInstance
              admitGuestOperation refuses macOS -> resolve image -> network (if enabled) -> start VM
              consumed snapshot: remove best-effort
New:      RestoreInstance
              Running/Initializing and macOS and checkpoint on disk -> finishMacOSResume -> return
              Paused and macOS                                      -> retire checkpoint -> resume -> finishMacOSResume -> return
              Linux                                                 -> unchanged
          restoreInstance
              admitGuestOperation (guest-type check only) -> admitMacOSBoot (boot lock)
              if macOS: skip image resolve (boot disk and aux are cloned at create); no TAP
              network: usesLinuxNetwork() (NetworkEnabled && MacOS == nil)
              if macOS: once the shim has loaded the VM, retire the checkpoint before resume
                        (failure: shut the VM down, keep the checkpoint)
              consumed snapshot after resume: Linux unchanged

Cold clone (fork)

Existing: forkInstance -> admitGuestOperation refuses macOS -> fork from stopped or standby
New:      forkInstance -> admitGuestOperation (guest-type check only)
          if macOS and (source not Stopped or no StopGraceful): refuse
          forkInstanceFromStoppedOrStandby:
              name check and TAP only when usesLinuxNetwork()
              if macOS: prepareMacOSClone (fresh machine identifier, IdentityRekeyPending, MAC cleared)
          saveForkMetadata:
              name check (Linux network only)
              if macOS: assignMacOSCloneMAC under the fork admission lock
              save
          first start of the clone: rekeyMacOSCloneOnStart (failure stops the clone, flag stays set)

Schedules

Existing: SetSnapshotSchedule / RunSnapshotSchedules -> admitGuestOperation refuses macOS
New:      admitGuestOperation (guest-type check only)
          validateMacOSSnapshotSchedule: requires save/restore support; probes live configuration
          run: Stopped source -> stopped snapshot; Running -> memory snapshot then resume;
               Standby -> memory snapshot; ineligible or failed run -> last_error, interval consumed

Change size

40 files, +2219 / −112 at 2703576 (about 2331 churn, so above the 500-line threshold).

Approximate meaningful implementation LOC: about 870 added and 95 deleted. Excluded: about 1,200 added test lines across 20 files (41 new test functions), about 115 doc lines, and mechanical churn (the admitGuestOperation signature change across call sites, the resumeReady* to stateReady* rename, and removal of the darwin build tag from shimconfig).

Chunk	Files	Approx. impl. LOC	Purpose
Memory snapshot capture, restore and recovery	snapshot_macos.go (new)	+364	payload validation, staged swap with rollback, crash recovery, finishMacOSResume
Clone identity and admission	macos_identity.go (new)	+246	identity rebind, MAC under the fork lock, SSH rekey, receipt clearing, boot identity/MAC checks (moved from macos.go)
macOS branch wiring on existing paths	snapshot.go, restore.go, standby.go, fork.go, snapshot_schedule.go	+122 / −39	macOS branches on existing snapshot, restore, standby, fork and schedule paths
VZ configuration probe and pause	vz/client.go, cmd/vz-shim/server.go, hypervisor/tracing.go, hypervisor/snapshot_support.go	+64 / −7	probe endpoint, pause waits for raw Paused, tracing passthrough
Lifecycle	stop.go, start.go, create.go, manager.go, types.go, macos.go, update.go, vm_config_validation.go, snapshot_compression.go	+66 / −50	receipt, startup staging recovery, new fields, removal of the macOS "not implemented" refusal list in macos.go
Network	lib/network/allocate.go	+11	NewUniqueMAC, so macOS clones share the allocator's uniqueness rule

Behavior

Linux. Behavior changes in one place, as fixes. If a standby fails after the VMM has stopped, the source is restored before the error is returned, instead of being left in Standby. This covers snapshot create, the public standby request, and fork from a running source. The restore uses the record standby reached, not the pre-standby disk copy. Otherwise behavior is unchanged.

macOS.

• Explicit enabled compression on memory snapshots and standby is rejected. Linux server compression defaults do not apply.
• A restored VM is a new run. Restoring a stopped snapshot to Running clears the source's restart block. Restoring with an explicit Stopped target sets the manual-stop block.
• Start clears the stop receipt before StartVM, after boot admission, so a rejected start keeps it.
• A restore retried while the shim is still loading the VM returns 409 and keeps the checkpoint.
• A loaded checkpoint is retired (renamed, then removed) before the guest resumes, so a crash after resume can't restore old memory onto a newer disk. A resume that fails after that leaves the instance Stopped without a graceful receipt.
• Restore staging lives in guests/<id>/restore-staging/. The instance directory is never renamed, so deleting an instance always deletes its staging.
• A stop is graceful only if the vz-shim recorded that the guest powered the VM off itself (vz-guest-stopped, written only when the shim didn't force the stop). VMM exit alone no longer counts. A shim started by an older hypeman never writes the receipt, so its guest needs a fresh boot and stop before a stopped snapshot or clone.
• A start retried after an API exit finishes a pending clone rekey before it returns for a Running or Initializing clone. A failed rekey keeps the flag set.

Testing

• go build ./lib/...: passes. Run with empty placeholder files for the make-generated embedded binaries (cloud-hypervisor, guest-agent, caddy, vz-shim), which aren't built in this environment.
• go vet ./lib/instances/: passes. gofmt -l lib cmd: clean.
• go test ./lib/instances/ -run 'TestMacOS|TestRejectedMacOSStartKeepsStopReceipt|TestClearMacOSStopReceipt|TestFinishMacOS|TestAssignMacOSCloneMAC|TestGracefulShutdown' -count=1: ok.
• go test ./lib/instances/ -run 'TestRestoreKeepsCheckpointWhileShimLoads|TestStartRetryFinishesCloneRekey|TestRecoverSource' -count=1: ok. The first two fail without their fixes. TestRecoverSourceAfterFailedStandby fails if the recovery skips saving the standby record, so it guards the stale-disk fix. Recovery is tested at the helper level only. The write-failure injection that reproduces a late standby failure end to end was not built.
• Not run on hardware: the late-standby recovery on a real Mac. the Mac still runs the macOS guests 5/6: guest-port forwarding #514 hypeman build.
• go test ./lib/hypervisor/vz/ -count=1: ok.
• go test ./lib/instances/ -count=1 (full package): 24 integration tests fail in this sandbox. Causes: no bridge permission (create bridge: operation not permitted), no mkfs.erofs (image builds fail), and no VMs. No failures are in the new macOS unit tests. I did not run a baseline on main for this package.
• go test ./lib/hypervisor/ -count=1: 2 failures, both also present on main. TestSocketCacheKeyChangesWhenSocketIsRecreated fails 3/3 on main. TestResolveProcessPIDDuringProcessChurn fails intermittently on main and failed on main once.
• Not run: darwin-tagged tests (snapshot_macos_darwin_test.go, and macos_identity_test.go, which is darwin && arm64). GOOS=darwin go vet cannot build here because of the vz cgo dependencies.
• Native QA (Apple silicon, macOS 27.0.1 host and guests): six runs plus a VZ restore spike, as recorded by the author before this PR's review remediation. Covered: memory checkpoint, change and restore (same PID and boot session), repeated restore, Standby and Stopped targets, cancellation after pause, API crash mid-restore, schedules with retention, cold forks with distinct identities, and the VZ restore-compatibility spike behind the warm-fork decision.
• TestMacOSRestoreRecoveryRollsBackOrFinishes, TestMacOSRestoreStagingIsDeletedWithInstance: crash before and after the swap commits, and a deleted instance is not recreated.
• TestMacOSRestoreRetiresCheckpointBeforeResume, TestMacOSRestoreKeepsCheckpointWhenItCannotBeRetired: both fail with the old order.
• TestMacOSGracefulShutdownRequiresGuestStopReceipt: VMM exit without a receipt is not graceful, and an earlier run's receipt never counts. Fails on the old stop code.
• Darwin type-check (vz-shim, lib/hypervisor/vz, lib/instances, including test files): clean. The shim's receipt write has not run on a Mac.
• Full suite at b21ccd5 vs 234278b: same failures, plus TestEnsureSystemFiles, which hit the Docker Hub rate limit.
• Not re-run natively: b21ccd5 (staging location, checkpoint retirement, stop receipt), and the review remediation commits (8e04e57, 2703576). They change stop-receipt derivation, receipt clearing, the restart block on restore, compression rejection on standby, and the restore metadata write.

Caveats / Not included

• Linux follow-up (pre-existing, not changed here): a restore to Running on Linux keeps the source's manual_stop restart block. Both the stopped-snapshot path (snapshot.go, startInstance) and restoreInstance skip clearing RestartStatus. An instance with a restart policy that was stopped, snapshotted, and restored to Running won't auto-restart after a crash. main has the same path, so this isn't a regression. Confirmed by reading; not reproduced at runtime. The macOS path is fixed in this PR.
• macOS standby ignores CompressionDelay silently. Explicit compression is rejected.
• Not supported: memory-snapshot compression, warm memory forks, and snapshots carried to another host or macOS account (VZ saved state is tied to both).
• Clones change machine identifier, MAC and SSH host keys only. Hostname and accounts are copied.

Review focus

• lib/instances/snapshot_macos.go (staged swap, recovery, finishMacOSResume).
• lib/instances/macos_identity.go (clone identity, MAC assignment, rekey teardown).
• lib/instances/stop.go (receipt derivation, the pid == 0 macOS branch).

Stack

1. macOS guests 1/6: VZ runtime, machine images and instance lifecycle #498: VZ runtime, machine images and instance lifecycle (merged)
2. macOS guests 2/6: OCI machine-image integration #499: OCI machine images (merged)
3. macOS guests 3/6: Darwin guest service, readiness and exec #500: Darwin guest agent, readiness and exec (merged; base of this PR)
4. macOS guests 4/6: snapshots, schedules and stopped forks #506: snapshots, schedules and stopped clones (this PR)
5. macOS guests 5/6: guest-port forwarding #514: guest-port forwarding (open; base is this PR's branch)
6. macOS guests 6/6: experimental machine-image builds #503: machine-image builds, not needed for v0 and can land later (open; base is macOS guests 5/6: guest-port forwarding #514's branch)

Shared Linux files get small hooks such as if stored.MacOS != nil. macOS logic lives in its own files (macos*.go, *_darwin.go, snapshot_macos.go).

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit 8a79efe. Configure here.}

[hiroTamada]

reviewed at 2703576 — three recovery gaps worth addressing:

P1 — don't consume a checkpoint while its shim is still loading

• lib/instances/manager.go:649–656 — StateInitializing also covers a surviving shim that hasn't opened its control socket yet (macos.go:377–390), not just a running VM waiting for its agent. retrying RestoreInstance in that interval calls finishMacOSResume, removes the checkpoint, and returns success before loading/resumption has completed. if loading subsequently fails, the local checkpoint is gone. confirm the live VM has resumed before consuming it; keep the checkpoint and return an in-progress/conflict response for a pre-listener shim. reproduced with the process-lookup test seam: restore returned nil and deleted the checkpoint while the simulated shim was still pre-listener.

P2 — recover the snapshot source after a late standby failure

• lib/instances/snapshot.go:108–111, lib/instances/standby.go:269–272 — standby can save the checkpoint and shut down the VMM, then fail writing metadata. capture immediately returns that error before setting restoreSource, leaving the previously running guest in standby. arm source recovery once the transition has happened, including late failures inside standbyInstance. reproduced by denying metadata writes after mocked shutdown: capture failed and the source stayed in Standby. the shared Linux orchestration already has this gap; this PR newly exposes it for macOS.

P2 — finish pending clone rekeys on start retries

• lib/instances/start.go:234–236, lib/instances/manager.go:737–743 — if the API exits after clone boot but before rekey completes, the VMM survives with IdentityRekeyPending=true. a start retry returns early for Running/Initializing without reaching the rekey hook, so the clone can keep its source's SSH host keys while start reports success. handle pending rekeys before that fast path returns, retaining teardown-on-failure behavior. reproduced for both Running and Initializing: start returned nil with the pending flag unchanged.

validation: focused existing macOS unit tests passed; additional mocked regression tests reproduced all three findings. tests ran in an isolated source copy with compile-only placeholders for generated embedded binaries. no native VZ execution, identifier generation, or real guest SSH rekey was tested.

[chruffins]

@hiroTamada thanks for the review. All three findings are addressed in edb1be8 and 234278b.

• P1, restore consumes a checkpoint while the shim loads: restore returns 409 and keeps the checkpoint until the control socket answers. TestRestoreKeepsCheckpointWhileShimLoads fails without the fix.
• P2b, clone rekey skipped on start retry: start runs the pending rekey before its fast path returns, and a failed rekey keeps the flag set. TestStartRetryFinishesCloneRekey fails without the fix.
• P2a, source left in Standby after a late standby failure: now covered. Standby returns the record it reached after stopping the VMM. Recovery saves that record before restoring, so the restore no longer reads the pre-standby disk copy. TestRecoverSourceAfterFailedStandby fails if the record save is removed. TestRecoverSourceKeepsStandbyWhenRecordCannotBeSaved covers a failed save.

Still not covered: an end-to-end run that injects a real metadata-write fault. That is unit-level coverage only. Hardware validation of this path is pending.

[sjmiller609]

Primary finding

This PR is introducing a new directory restore-staging like this:

<data-dir>/
├── guests/
│   └── <instance-id>/
...
│
└── restore-staging/                 # NEW external temporary storage
    └── <instance-id>/
        ├── staged/                 # incoming snapshot + metadata
        └── previous/               # old guest directory

but I think that any guest-specific data should always be inside of the guest directory, so that a deletion of the guest dir always results in full wipe of any guest specific data.

storage ownership/deletion: keep a stable guest directory with active storage and restore staging inside it; leftover external backups can resurrect deleted instances.
e.g.:

guests/<instance-id>/
...
└── restore-staging/
    ├── staged/
    └── previous/

Other, less important

stale memory restore: invalidate abandoned checkpoints before resuming; otherwise a later crash can expose old memory for restore against newer disks.
graceful-stop verification: require confirmed clean shutdown before allowing stopped snapshots/clones; VMM exit alone currently counts, even after failed shutdown requests.

[chruffins]

@sjmiller609 thanks, all three addressed in b21ccd5:

• restore staging: moved to guests/<id>/restore-staging/{staged,previous,installing}. The guest dir is never renamed; files are swapped within it, and renaming staged → installing commits. A crash before commit rolls back, after it finishes the install. Deleting the guest dir always wipes staging, so recovery can't resurrect a deleted instance.
• stale memory restore: the checkpoint is retired (atomic rename, then removed) once the shim has loaded the VM and before resume, on both the normal restore and Paused-resume paths. If retiring fails, the VM is shut down and the checkpoint kept.
• graceful-stop verification: the vz-shim writes a stop receipt only when the guest powers the VM off itself, not when the shim forces the stop. A macOS stop counts as graceful only with VMM exit + that receipt; it's cleared before each shutdown request and consumed after.

Tests for each; the checkpoint and receipt tests fail on the old code. Not yet run natively on a Mac.