From 570f328b8c8f654adddd445c5d501ec9f7229ca3 Mon Sep 17 00:00:00 2001 From: Mikola Lysenko Date: Fri, 9 Oct 2026 12:43:22 -0400 Subject: [PATCH 1/3] WIP: Fix berry restore registry lookup sources (#1017) Co-Authored-By: Claude Opus 5.5 (1M context) From d1221abb4b30d6a21ec29ebc1361d480aeb27170 Mon Sep 17 00:00:00 2001 From: Mikola Lysenko Date: Fri, 9 Oct 2026 13:55:08 -0400 Subject: [PATCH 2/3] Read berry restore registry like yarn does Hosted yarn berry rollback, remove and the takeover restore read only the project .yarnrc.yml's top-level npmRegistryServer. A mirror set through npmScopes, YARN_NPM_REGISTRY_SERVER, a parent-directory or home .yarnrc.yml, or a ${VAR:-default} value was ignored. The restore then dropped the mirror's __archiveUrl binding without a warning, and the next cold immutable install failed YN0035. The restore now resolves each package's registry the way yarn merges its settings, and warns upstream_registry_fallback when the registry cannot be determined. Fixes #1017 Co-Authored-By: Claude Opus 5.5 (1M context) --- crates/socket-patch-cli/CLI_CONTRACT.md | 4 +- .../tests/in_process_redirect.rs | 186 +++++++++ .../src/patch/redirect/upstream/npm.rs | 387 ++++++++++++++++-- 3 files changed, 544 insertions(+), 33 deletions(-) diff --git a/crates/socket-patch-cli/CLI_CONTRACT.md b/crates/socket-patch-cli/CLI_CONTRACT.md index bb4592f4b..13b3cee11 100644 --- a/crates/socket-patch-cli/CLI_CONTRACT.md +++ b/crates/socket-patch-cli/CLI_CONTRACT.md @@ -959,7 +959,7 @@ v5.0 replaces v4's per-purl reverts and whole-ledger reverse replay (`revert_rem * **Scope.** The hosted pins are what lockfile discovery finds — `(purl, patch uuid, files wiring it)`, recognized only on `https://patch.socket.dev` or the `--patch-server-url` / `SOCKET_PATCH_SERVER_URL` origin. A scoped rollback (paths / identifiers / `--ecosystems`) restores exactly the pins in scope; each pin restores or refuses on its own (there is no whole-ledger replay, and a pre-v5 ledger's edits are never replayed). A pin discovery cannot see is out of reach: a lockless cargo `registry = "socket-patch-"` pin, a nuget exact-id mapping with no `packages.lock.json`, a gem wired only in the `Gemfile` (pre-bundler-2.6 mixed state) — restore those files from version control. * **What a restore does.** Every file wiring the pin is rewritten back to the DEFAULT UPSTREAM registry entry for `name@version`, re-resolving whatever the entry pins (tarball URL, integrity, checksum, hashes) from the public registry; only the hosted entries change and every other byte stays the file's own. A pin is **all-or-nothing**: refused in one of its files, it is restored in none of them, so no pin is left half hosted. Nothing reaches disk until every pin has resolved, and `--dry-run` resolves exactly like a wet run — registry lookups included — and skips only the write. Per format: - * **npm family** — `package-lock.json` / `npm-shrinkwrap.json`, `yarn.lock` (classic and berry), `pnpm-lock.yaml` / `shrinkwrap.yaml`, `bun.lock`: resolution + integrity (+ shasum where recorded) from the npm registry's version document (`SOCKET_NPM_REGISTRY`); a yarn berry lock whose `.yarnrc.yml` names another `npmRegistryServer` reads that registry's document instead, so a mirror's off-path `dist.tarball` keeps its `::__archiveUrl=` binding, and a pnpm lock whose sibling settings name a registry reads that registry's document — `.npmrc` `registry` (or, for a scoped name, `@scope:registry`); on pnpm 10 a pnpm-workspace.yaml `registries` map instead when present; on pnpm 11+ (or an unknown major) pnpm-workspace.yaml `registries."@scope"` / `registry` / `registries.default` too, ahead of the matching `.npmrc` key — so a mirror's `tarball:` comes back as pnpm recorded it and a URL conventional under that registry stays derived (falling back to the default registry, with `upstream_registry_fallback`, when the mirror can't be read; a value holding an unexpanded `${VAR}` is read as unset). Whether a restored pnpm entry gets its `tarball:` back follows `lockfileIncludeTarballUrl` as the pnpm that wrote the lock read it (#902), from the strongest evidence available: (1) the lock's own unpinned registry resolutions (a bare one proves it off; a URL pnpm could have derived proves it on; pnpm 11+'s env lockfile document does not count); else (2) the settings file the installed pnpm major reads (`node_modules/.modules.yaml` `packageManager`, else package.json `packageManager`; a pre-9 lock or shrinkwrap means pnpm <= 8): `.npmrc` `lockfile-include-tarball-url` on pnpm <= 9, pnpm-workspace.yaml `lockfileIncludeTarballUrl` on pnpm >= 11 (also assumed for a lock carrying an env lockfile document), the workspace file then `.npmrc` on pnpm 10; else (3) pnpm 10's reading. A Rush lock (`common/config/rush/pnpm-lock.yaml` or a subspace lock, with `rush.json` at the Rush root) takes its pnpm major from rush.json `pnpmVersion` instead of tier 2's install record and package.json pin. A 9.0 lock may come from pnpm 9, 10 or 11+, so when tier 3's reading differs from pnpm 9's (`.npmrc` only) or pnpm >= 11's (pnpm-workspace.yaml only) — e.g. `.npmrc` on with the workspace file silent, or the workspace file setting it with `.npmrc` silent or disagreeing — the restore follows pnpm 10 but warns `upstream_pnpm_tarball_setting_guessed` (once per lock, naming the entries); a URL pnpm records anyway (not derivable from the registry) never warns. A `bun.lock` 4-tuple's registry slot is rebuilt the way Bun writes it (#992): `""` for a package from registry.npmjs.org, otherwise the full tarball URL — Bun 1.1.39–1.3.6 read `""` as npmjs whatever the project configures. The registry is the one Bun resolves the package against: a scope's `.npmrc` `@scope:registry` or `bunfig.toml` `[install.scopes]` entry, else `BUN_CONFIG_REGISTRY` / `NPM_CONFIG_REGISTRY`, the `.npmrc` `registry`, then `bunfig.toml` `[install] registry`; its version document's `dist.tarball` fills the slot, and when it can't be read (`upstream_registry_fallback`) the default registry's conventional URL is re-based on it. The `bun.lockb` takeover restore records the same URL. Side settings: a project `.npmrc` that is exactly `allow-remote=all\n` is deleted once no root npm lock entry is hosted, otherwise a remaining top-level `allow-remote=all` warns `npm_allow_remote_left`; a `pnpm-workspace.yaml` that is exactly the scaffold hosted mode creates is deleted once `pnpm-lock.yaml` is no longer hosted, otherwise a remaining `trustLockfile: true` warns `pnpm_trust_lockfile_left`. **`bun.lockb` (binary)**: `rollback` and `remove` refuse it (the checkout remedy). The hosted → vendored takeover and the eject DO restore it, since the vendor ledger then records the rebuilt record as its pre-vendor original: the native codec turns each hosted remote-tarball record back into Bun's npm registry record for `name@version` (the registry's `dist.tarball` + `dist.integrity`, the package metadata hash re-derived, the hosted URL string dropped from the string pool). The hosted rewrite keeps the registry record's inactive bytes (padding, semver) in the tarball record, so a lock it wrote comes back byte for byte — early writers' uninitialized padding included; a record without them (an older socket-patch or a Bun re-save) is rebuilt the way Bun writes one, and refused for a prerelease/build version. A lock the hosted rewrite had to normalize is marked in the root package's resolution value bytes (which no Bun reader reads): a binary format 1 lock it promoted to format 2 is demoted back to its exact format-1 bytes (verified by promoting it again, otherwise refused), and a lock whose workspace dependency behaviors it normalized is refused with the `git checkout -- bun.lockb` remedy. + * **npm family** — `package-lock.json` / `npm-shrinkwrap.json`, `yarn.lock` (classic and berry), `pnpm-lock.yaml` / `shrinkwrap.yaml`, `bun.lock`: resolution + integrity (+ shasum where recorded) from the npm registry's version document (`SOCKET_NPM_REGISTRY`); a yarn berry lock whose settings name another registry reads that registry's document instead — the registry yarn itself resolves the package against (#1017): a scoped package's `npmScopes..npmRegistryServer`, else `YARN_NPM_REGISTRY_SERVER`, else `npmRegistryServer`, each from the closest `.yarnrc.yml` (`YARN_RC_FILENAME`) that sets it among the lock's directory, every parent directory and then the home directory, with `${VAR}` / `${VAR:-default}` / `${VAR-default}` expanded as yarn expands them (a reference yarn cannot expand, or an `npmScopes` written as a flow mapping, falls back to the default registry with `upstream_registry_fallback`), so a mirror's off-path `dist.tarball` keeps its `::__archiveUrl=` binding, and a pnpm lock whose sibling settings name a registry reads that registry's document — `.npmrc` `registry` (or, for a scoped name, `@scope:registry`); on pnpm 10 a pnpm-workspace.yaml `registries` map instead when present; on pnpm 11+ (or an unknown major) pnpm-workspace.yaml `registries."@scope"` / `registry` / `registries.default` too, ahead of the matching `.npmrc` key — so a mirror's `tarball:` comes back as pnpm recorded it and a URL conventional under that registry stays derived (falling back to the default registry, with `upstream_registry_fallback`, when the mirror can't be read; a value holding an unexpanded `${VAR}` is read as unset). Whether a restored pnpm entry gets its `tarball:` back follows `lockfileIncludeTarballUrl` as the pnpm that wrote the lock read it (#902), from the strongest evidence available: (1) the lock's own unpinned registry resolutions (a bare one proves it off; a URL pnpm could have derived proves it on; pnpm 11+'s env lockfile document does not count); else (2) the settings file the installed pnpm major reads (`node_modules/.modules.yaml` `packageManager`, else package.json `packageManager`; a pre-9 lock or shrinkwrap means pnpm <= 8): `.npmrc` `lockfile-include-tarball-url` on pnpm <= 9, pnpm-workspace.yaml `lockfileIncludeTarballUrl` on pnpm >= 11 (also assumed for a lock carrying an env lockfile document), the workspace file then `.npmrc` on pnpm 10; else (3) pnpm 10's reading. A Rush lock (`common/config/rush/pnpm-lock.yaml` or a subspace lock, with `rush.json` at the Rush root) takes its pnpm major from rush.json `pnpmVersion` instead of tier 2's install record and package.json pin. A 9.0 lock may come from pnpm 9, 10 or 11+, so when tier 3's reading differs from pnpm 9's (`.npmrc` only) or pnpm >= 11's (pnpm-workspace.yaml only) — e.g. `.npmrc` on with the workspace file silent, or the workspace file setting it with `.npmrc` silent or disagreeing — the restore follows pnpm 10 but warns `upstream_pnpm_tarball_setting_guessed` (once per lock, naming the entries); a URL pnpm records anyway (not derivable from the registry) never warns. A `bun.lock` 4-tuple's registry slot is rebuilt the way Bun writes it (#992): `""` for a package from registry.npmjs.org, otherwise the full tarball URL — Bun 1.1.39–1.3.6 read `""` as npmjs whatever the project configures. The registry is the one Bun resolves the package against: a scope's `.npmrc` `@scope:registry` or `bunfig.toml` `[install.scopes]` entry, else `BUN_CONFIG_REGISTRY` / `NPM_CONFIG_REGISTRY`, the `.npmrc` `registry`, then `bunfig.toml` `[install] registry`; its version document's `dist.tarball` fills the slot, and when it can't be read (`upstream_registry_fallback`) the default registry's conventional URL is re-based on it. The `bun.lockb` takeover restore records the same URL. Side settings: a project `.npmrc` that is exactly `allow-remote=all\n` is deleted once no root npm lock entry is hosted, otherwise a remaining top-level `allow-remote=all` warns `npm_allow_remote_left`; a `pnpm-workspace.yaml` that is exactly the scaffold hosted mode creates is deleted once `pnpm-lock.yaml` is no longer hosted, otherwise a remaining `trustLockfile: true` warns `pnpm_trust_lockfile_left`. **`bun.lockb` (binary)**: `rollback` and `remove` refuse it (the checkout remedy). The hosted → vendored takeover and the eject DO restore it, since the vendor ledger then records the rebuilt record as its pre-vendor original: the native codec turns each hosted remote-tarball record back into Bun's npm registry record for `name@version` (the registry's `dist.tarball` + `dist.integrity`, the package metadata hash re-derived, the hosted URL string dropped from the string pool). The hosted rewrite keeps the registry record's inactive bytes (padding, semver) in the tarball record, so a lock it wrote comes back byte for byte — early writers' uninitialized padding included; a record without them (an older socket-patch or a Bun re-save) is rebuilt the way Bun writes one, and refused for a prerelease/build version. A lock the hosted rewrite had to normalize is marked in the root package's resolution value bytes (which no Bun reader reads): a binary format 1 lock it promoted to format 2 is demoted back to its exact format-1 bytes (verified by promoting it again, otherwise refused), and a lock whose workspace dependency behaviors it normalized is refused with the `git checkout -- bun.lockb` remedy. * **vlt** — `vlt-lock.json`: slot [2] from the registry's `dist.integrity`, slot [3] per the lock's own convention (see the vlt hosted-mode contract); every hosted instance of the pin together. * **cargo** — `Cargo.lock` back on crates.io (source + the sparse index's checksum, `SOCKET_CRATES_INDEX`); every `Cargo.toml` declaration loses its `registry = "socket-patch-"` pin (the shorthand the rewriter produced collapses back); every `[registries.socket-patch-]` block no manifest or lock still references leaves the project cargo config — including a superseded patch generation's block an earlier re-pin left behind (#864). A declaration it cannot unpin refuses. * **golang** — the hosted `replace` and the socket module's go.sum lines go; the upstream module's two go.sum lines come back, hashed from the module proxy (`SOCKET_GOPROXY`, else `GOPROXY` / `GONOPROXY` / `GOPRIVATE` as go reads them) and cross-checked against the checksum database (`SOCKET_GOSUMDB_URL`, else `sum.golang.org` unless `GOSUMDB=off` / `GONOSUMDB` / `GOPRIVATE` say go would not ask it). A `replace` the user had before the hosted run is not recorded anywhere, so the restore lands on the plain upstream module. @@ -1281,7 +1281,7 @@ Every `--json` invocation emits a single JSON object that follows the **unified | `manifest_write_failed` | rollback `warnings[]` | rollback (v5.0): the post-rollback manifest update could not be written; no entries were removed (`manifest.removedEntries: []`) and the run exits `partial_failure` 1. | | `npm_allow_remote_left` / `pnpm_trust_lockfile_left` | rollback/remove `warnings[]`; vendor advisory event (takeover) | upstream restore (v5.0): no npm-family lock entry is hosted any more, but the project `.npmrc` keeps a top-level `allow-remote=all` (resp. `pnpm-workspace.yaml` keeps `trustLockfile: true`) in a file that is not exactly what hosted mode creates; the file is left untouched (v5 records no provenance), remove the line if nothing else needs it. A file that is exactly hosted mode's own is deleted silently. | | `maven_trusted_checksums_left` / `nuget_default_config_left` / `upstream_uv_override_removed` | rollback/remove `warnings[]`; vendor advisory event (takeover) | upstream restore (v5.0): `.mvn` config keeps the trusted-checksums resolver lines because it holds more than hosted mode writes; `nuget.config` now holds only the nuget.org source (delete it if hosted mode created it); a transitive `override-dependencies` entry hosted mode added to `pyproject.toml` was removed. | -| `upstream_registry_fallback` | rollback/remove `warnings[]`; vendor advisory event (takeover) | upstream restore: a yarn berry, pnpm or vlt entry is restored from the version document of the registry the project resolves it against (`.yarnrc.yml` `npmRegistryServer`, the pnpm lock's sibling `.npmrc` `registry` / `@scope:registry` or pnpm-workspace.yaml `registry` / `registries`, vlt's node registry); that registry could not be read (e.g. it needs credentials), so the default registry's document was used and the restored tarball URL may not be the mirror's. | +| `upstream_registry_fallback` | rollback/remove `warnings[]`; vendor advisory event (takeover) | upstream restore: a yarn berry, pnpm or vlt entry is restored from the version document of the registry the project resolves it against (yarn berry: `npmScopes` / `YARN_NPM_REGISTRY_SERVER` / `npmRegistryServer` from the `.yarnrc.yml` chain, as above, the pnpm lock's sibling `.npmrc` `registry` / `@scope:registry` or pnpm-workspace.yaml `registry` / `registries`, vlt's node registry); that registry could not be read (e.g. it needs credentials) or, for yarn berry, could not be determined (an rc value referencing an unset variable with no default, a flow-style `npmScopes`), so the default registry's document was used and the restored tarball URL may not be the mirror's. | | `upstream_gem_stale_cache` | rollback/remove `warnings[]`; vendor advisory event (takeover) | upstream restore (#1260): a restored gem's `-.gem` is still in Bundler's cache dir (`cache_path`, default `vendor/cache`, resolved as for the Gem stale-install guard) and its sha256 is not the upstream one (the restored `CHECKSUMS` entry, else the rubygems.org compact index), or it could not be checked (`--offline`, a registry error). Bundler installs from that dir first, so a `bundle cache` taken while the hosted pin was live makes every later install fail on the upstream checksum (exit 37) or, on bundler < 2.6 frozen installs, keep installing the patched bytes. The detail names the file. Remedy: delete it, then run `bundle cache` to cache the upstream gem in its place (or `bundle install` if the project does not commit its cache; with the cache dir committed, a frozen install reads only the cache). Read-only: the restore never deletes it. Not raised for an archive whose sha256 matches upstream. | | `upstream_pnpm_tarball_setting_guessed` | rollback/remove `warnings[]`; vendor advisory event (takeover) | upstream restore (#902): nothing showed which pnpm wrote a hosted `pnpm-lock.yaml` (no unpinned registry entry that shows the setting, no `node_modules/.modules.yaml` install record, no package.json `packageManager` pin; for a Rush lock, no rush.json `pnpmVersion`), so its entries were restored with or without `tarball:` by pnpm 10's reading of `lockfileIncludeTarballUrl` (pnpm-workspace.yaml, else `.npmrc` `lockfile-include-tarball-url`), and pnpm 9 (which reads only `.npmrc`) or pnpm >= 11 (which reads only pnpm-workspace.yaml) would have read it the other way. The detail names the lock, the setting followed, the pnpm that disagrees and the entries. Remedy: pin the pnpm (package.json `packageManager`, or reinstall so the install record names it; rush.json `pnpmVersion` for Rush), or give the two files the same value so every pnpm reads it alike; a rollback or remove can then be redone by restoring the lock from version control and re-running. Not raised when evidence decided, when both files read the same on every pnpm, or when the tarball is one pnpm records regardless. | | `legacy_redirect_ledger_kept` | rollback `warnings[]` (+ remove stderr) | v5.0: a pre-v5 `.socket/vendor/redirect-state.json` could not be deleted once no hosted pin was left; the file is inert (never read for planning). Never flips the exit. | diff --git a/crates/socket-patch-cli/tests/in_process_redirect.rs b/crates/socket-patch-cli/tests/in_process_redirect.rs index 799267eeb..1e822c091 100644 --- a/crates/socket-patch-cli/tests/in_process_redirect.rs +++ b/crates/socket-patch-cli/tests/in_process_redirect.rs @@ -6055,3 +6055,189 @@ async fn hosted_json_reports_an_unpinned_row_for_a_granted_patch_nothing_pins() assert_eq!(rows[0]["action"], "unpinned", "{env:#}"); assert_eq!(rows[0]["errorCode"], "redirect_unconfirmed", "{env:#}"); } + +// ── #1017: the berry restore reads the registry from every yarn source ────── + +/// Where a #1017 cell configures the project's mirror. +#[derive(Debug, Clone, Copy)] +enum MirrorSource { + /// `YARN_NPM_REGISTRY_SERVER`, no key in any rc. + Env, + /// A parent directory's `.yarnrc.yml`. + ParentRc, + /// The home directory's `.yarnrc.yml` (the project is not under it). + HomeRc, + /// The project rc's `npmRegistryServer: "${UNSET:-}"`. + Interpolated, +} + +/// #1017: #908's mirror restore, with the mirror set the other ways yarn +/// reads it. Each cell pins the project hosted, rolls it back, and needs +/// the mirror's `::__archiveUrl=` binding back byte-exactly — reading the +/// default registry instead silently writes a bare `name@npm:` locator +/// whose conventional tarball the mirror 404s. +#[tokio::test] +#[serial] +async fn yarn_berry_rollback_reads_the_registry_from_every_yarn_source() { + for source in [ + MirrorSource::Env, + MirrorSource::ParentRc, + MirrorSource::HomeRc, + MirrorSource::Interpolated, + ] { + let server = MockServer::start().await; + mock_discovery(&server).await; + let hosted_url = HOSTED_URL.replace("http://patch.test", &server.uri()); + mock_reference_with_berry_url(&server, &hosted_url).await; + mock_view(&server).await; + let integrity = vlt_hosted_common::sha512_sri(&upstream_tarball()); + mock_npm_registry_advertising( + &server, + &integrity, + &format!( + "{}/npm-registry/{NAME}/-/{NAME}-{VERSION}.tgz", + server.uri() + ), + ) + .await; + let mirror = format!("{}/mirror", server.uri()); + let advertised = format!("{}/cdn/files/{NAME}-{VERSION}.tgz", server.uri()); + Mock::given(method("GET")) + .and(path(format!("/mirror/{NAME}/{VERSION}"))) + .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ + "name": NAME, + "version": VERSION, + "dist": { "tarball": advertised, "integrity": integrity }, + }))) + .mount(&server) + .await; + let binding = |t: &str| { + t.replace( + &format!("resolution: \"{NAME}@npm:{VERSION}\""), + &format!( + "resolution: \"{NAME}@npm:{VERSION}::__archiveUrl={}\"", + socket_patch_core::utils::uri::encode_uri_component(&advertised) + ), + ) + }; + + let tmp = tempfile::tempdir().unwrap(); + let project = tmp.path().join("work").join("proj"); + let home = tmp.path().join("home"); + std::fs::create_dir_all(&project).unwrap(); + std::fs::create_dir_all(&home).unwrap(); + write_berry_project_spelled(&project, binding); + let mut project_rc = "nodeLinker: node-modules\n".to_string(); + let mut envs: Vec<(&str, String)> = vec![("HOME", home.display().to_string())]; + match source { + MirrorSource::Env => envs.push(("YARN_NPM_REGISTRY_SERVER", mirror.clone())), + MirrorSource::ParentRc => std::fs::write( + tmp.path().join("work").join(".yarnrc.yml"), + format!("npmRegistryServer: \"{mirror}\"\n"), + ) + .unwrap(), + MirrorSource::HomeRc => std::fs::write( + home.join(".yarnrc.yml"), + format!("npmRegistryServer: \"{mirror}\"\n"), + ) + .unwrap(), + MirrorSource::Interpolated => project_rc.push_str(&format!( + "npmRegistryServer: \"${{SOCKET_PATCH_TEST_UNSET_REG:-{mirror}}}\"\n" + )), + } + std::fs::write(project.join(".yarnrc.yml"), &project_rc).unwrap(); + let lock_path = project.join("yarn.lock"); + let pristine = std::fs::read_to_string(&lock_path).unwrap(); + + let env = run_redirect_subprocess_with( + &project, + &server.uri(), + &["--patch-server-url", &server.uri()], + ); + assert_eq!(env["redirect"]["redirected"], 1, "{source:?}: {env:#}"); + + let out = scrubbed_cli() + .args([ + "rollback", + "--json", + "--yes", + "--patch-server-url", + &server.uri(), + "--cwd", + project.to_str().unwrap(), + ]) + .env( + "SOCKET_NPM_REGISTRY", + format!("{}/npm-registry", server.uri()), + ) + .env_remove("YARN_NPM_REGISTRY_SERVER") + .env_remove("YARN_RC_FILENAME") + .env_remove("SOCKET_PATCH_TEST_UNSET_REG") + .envs(envs.iter().map(|(k, v)| (*k, v.as_str()))) + .output() + .unwrap(); + let env: serde_json::Value = serde_json::from_slice(&out.stdout).unwrap_or_else(|e| { + panic!( + "{source:?}: rollback stdout must be JSON: {e}\n{}\n{}", + String::from_utf8_lossy(&out.stdout), + String::from_utf8_lossy(&out.stderr) + ) + }); + assert_eq!(out.status.code(), Some(0), "{source:?}: rollback: {env:#}"); + let restored = std::fs::read_to_string(&lock_path).unwrap(); + let checksum = berry_checksum_of(&restored); + assert_eq!( + restored, + pristine.replace( + &format!("10c0/{}", "3".repeat(128)), + &format!("10c0/{checksum}") + ), + "{source:?}: rollback keeps the mirror's __archiveUrl binding" + ); + } +} + +/// #1017: a registry yarn itself cannot resolve (an rc reference to an +/// unset variable with no default) is not silently replaced by the default +/// registry: the restore says so with `upstream_registry_fallback`. +#[tokio::test] +#[serial] +async fn yarn_berry_rollback_warns_when_the_registry_cannot_be_known() { + let server = MockServer::start().await; + mock_discovery(&server).await; + let hosted_url = HOSTED_URL.replace("http://patch.test", &server.uri()); + mock_reference_with_berry_url(&server, &hosted_url).await; + mock_view(&server).await; + let conventional = format!( + "{}/npm-registry/{NAME}/-/{NAME}-{VERSION}.tgz", + server.uri() + ); + mock_npm_registry_advertising( + &server, + &vlt_hosted_common::sha512_sri(&upstream_tarball()), + &conventional, + ) + .await; + let tmp = tempfile::tempdir().unwrap(); + write_berry_project(tmp.path()); + let env = run_redirect_subprocess_with( + tmp.path(), + &server.uri(), + &["--patch-server-url", &server.uri()], + ); + assert_eq!(env["redirect"]["redirected"], 1, "{env:#}"); + std::fs::write( + tmp.path().join(".yarnrc.yml"), + "nodeLinker: node-modules\nnpmRegistryServer: \"${SOCKET_PATCH_TEST_UNSET_REG}\"\n", + ) + .unwrap(); + let (code, env) = rollback_json_with_origin(tmp.path(), &server, &server.uri()); + assert_eq!(code, Some(0), "rollback: {env:#}"); + let codes: Vec<&str> = env["warnings"] + .as_array() + .into_iter() + .flatten() + .filter_map(|w| w["code"].as_str()) + .collect(); + assert!(codes.contains(&"upstream_registry_fallback"), "{env:#}"); +} diff --git a/crates/socket-patch-core/src/patch/redirect/upstream/npm.rs b/crates/socket-patch-core/src/patch/redirect/upstream/npm.rs index c3d0d26e7..0e3ba1efb 100644 --- a/crates/socket-patch-core/src/patch/redirect/upstream/npm.rs +++ b/crates/socket-patch-core/src/patch/redirect/upstream/npm.rs @@ -519,21 +519,214 @@ fn berry_registry_locator( use crate::formats::pnpm::workspace::yaml_top_level_value; use crate::formats::text::strip_bom; -/// The registry a berry restore reads `name`'s version document from: -/// `.yarnrc.yml`'s `npmRegistryServer`. A scoped package may resolve -/// against an `npmScopes` registry instead, so with such a block present -/// it keeps the default registry's document. -fn berry_lookup_registry(yarnrc: Option<&str>, name: &str) -> Option { - let text = yarnrc?; - // `top_level_key` skips the first line's BOM (`formats::text`). - let has_scopes = text +/// The registry yarn berry resolves `name` against (`Ok(None)`: yarn's +/// default registry), read the way yarn merges its settings (#1017): +/// +/// - a scoped `@scope/name` takes `npmScopes..npmRegistryServer` +/// from the highest-precedence rc file that sets it; +/// - otherwise (and for a scope without its own server) the +/// `YARN_NPM_REGISTRY_SERVER` environment variable, then the +/// highest-precedence rc file's top-level `npmRegistryServer`. +/// +/// `rcs` are the `.yarnrc.yml` texts yarn reads, highest precedence first +/// ([`BerryRegistrySettings::read`]): the lock directory's, each parent +/// directory's up to the filesystem root, then the home directory's. A +/// value's `${VAR}` / `${VAR:-default}` / `${VAR-default}` references are +/// expanded through `var` as yarn expands them. `Err` names why the +/// registry cannot be known: a reference to an unset variable with no +/// default (yarn refuses to run), or an `npmScopes` this reader cannot +/// follow (a flow mapping). +fn berry_lookup_registry( + rcs: &[String], + env_registry: Option<&str>, + name: &str, + var: &dyn Fn(&str) -> Option, +) -> Result, String> { + if let Some((scope, _)) = name.strip_prefix('@').and_then(|rest| rest.split_once('/')) { + let keys = [scope.to_string(), format!("@{scope}")]; + for rc in rcs { + for key in &keys { + if let Some(value) = + yaml_path_value(rc, &["npmScopes", key.as_str(), "npmRegistryServer"])? + { + return yarn_expand_env(&value, var).map(Some); + } + } + } + } + if let Some(env) = env_registry.map(str::trim).filter(|v| !v.is_empty()) { + return Ok(Some(env.to_string())); + } + for rc in rcs { + if let Some(value) = yaml_path_value(rc, &["npmRegistryServer"])? { + return yarn_expand_env(&value, var).map(Some); + } + } + Ok(None) +} + +/// The scalar at `path` in a YAML settings file of block mappings (the +/// last occurrence of each key wins, quotes removed). `Ok(None)` when a +/// key on the path is absent or null; `Err` when a mapping on the path is +/// written in flow style, which this reader does not follow. +fn yaml_path_value(text: &str, path: &[&str]) -> Result, String> { + // `top_level_key` skips the first line's BOM (one, as yarn's parser). + let lines: Vec = text .lines() - .filter_map(crate::formats::pnpm::workspace::top_level_key) - .any(|(key, _)| key == "npmScopes"); - if has_scopes && name.starts_with('@') { - return None; + .map(|l| l.strip_suffix('\r').unwrap_or(l).to_string()) + .collect(); + yaml_path_in(&lines, path) +} + +fn yaml_path_in(lines: &[String], path: &[&str]) -> Result, String> { + use crate::formats::pnpm::workspace::top_level_key; + let Some((key, rest)) = path.split_first() else { + return Ok(None); + }; + let Some((at, value)) = lines.iter().enumerate().rev().find_map(|(i, line)| { + top_level_key(line) + .filter(|(k, _)| k == key) + .map(|(_, value)| (i, value.to_string())) + }) else { + return Ok(None); + }; + if matches!(value.as_str(), "~" | "null") { + return Ok(None); + } + if rest.is_empty() { + let value = value.trim_matches(['"', '\'']).to_string(); + return Ok((!value.is_empty()).then_some(value)); + } + if !value.is_empty() { + return Err(format!( + "`{}` is not written as a block mapping", + path.first().copied().unwrap_or_default() + )); + } + let end = lines[at + 1..] + .iter() + .position(|l| !l.is_empty() && !l.starts_with([' ', '\t', '#'])) + .map_or(lines.len(), |p| at + 1 + p); + let children = &lines[at + 1..end]; + let Some(indent) = children.iter().find_map(|line| { + let content = line.trim_start_matches(' '); + (!content.trim().is_empty() && !content.starts_with('#')) + .then(|| line.len() - content.len()) + }) else { + return Ok(None); + }; + let dedented: Vec = children + .iter() + .map(|line| match line.get(indent..) { + Some(rest) if line[..indent].trim().is_empty() => rest.to_string(), + _ => String::new(), + }) + .collect(); + yaml_path_in(&dedented, rest) +} + +/// A `.yarnrc.yml` value with its environment references expanded as yarn +/// expands them: `${NAME}`, `${NAME-fallback}` (the fallback when `NAME` +/// is unset) and `${NAME:-fallback}` (also when it is empty). A reference +/// to an unset variable with no fallback is the error yarn stops on. +fn yarn_expand_env(value: &str, var: &dyn Fn(&str) -> Option) -> Result { + let mut out = String::with_capacity(value.len()); + let mut rest = value; + while let Some(at) = rest.find("${") { + out.push_str(&rest[..at]); + let body = &rest[at + 2..]; + let name_len = body + .find(|c: char| !(c.is_ascii_alphanumeric() || c == '_')) + .unwrap_or(body.len()); + let name = &body[..name_len]; + let mut tail = &body[name_len..]; + let colon = tail.starts_with(':'); + if colon { + tail = &tail[1..]; + } + let fallback = match tail.strip_prefix('-') { + Some(after) => match after.find('}') { + Some(close) => { + tail = &after[close..]; + Some(&after[..close]) + } + None => None, + }, + None => None, + }; + let Some(after) = tail.strip_prefix('}').filter(|_| !name.is_empty()) else { + // Not a reference yarn recognizes: kept literally. + out.push_str("${"); + rest = body; + continue; + }; + let set = var(name); + let expanded = match (set, fallback) { + (Some(v), _) if !v.is_empty() => v, + (Some(v), _) if !colon => v, + (_, Some(fallback)) => fallback.to_string(), + (_, None) => return Err(format!("the environment variable {name} is not set")), + }; + out.push_str(&expanded); + rest = after; + } + out.push_str(rest); + Ok(out) +} + +/// The yarn berry settings that decide which registry a package resolves +/// against: the `.yarnrc.yml` texts yarn reads for a lock, highest +/// precedence first, and `YARN_NPM_REGISTRY_SERVER`. +struct BerryRegistrySettings { + rcs: Vec, + env_registry: Option, +} + +impl BerryRegistrySettings { + /// Yarn reads the rc file (`YARN_RC_FILENAME`, default `.yarnrc.yml`) + /// of the project directory and of every parent directory up to the + /// filesystem root (a closer one wins), then the home directory's, + /// below them all. + async fn read(view: &View<'_>, dir_prefix: &str) -> Self { + let rc_name = std::env::var("YARN_RC_FILENAME") + .ok() + .filter(|v| !v.trim().is_empty()) + .unwrap_or_else(|| ".yarnrc.yml".to_string()); + let root = view.root().join(dir_prefix); + let root = tokio::fs::canonicalize(&root).await.unwrap_or(root); + let mut read: Vec = + root.ancestors().map(|d| d.join(&rc_name)).collect(); + let home = std::env::var_os("HOME") + .or_else(|| std::env::var_os("USERPROFILE")) + .filter(|h| !h.is_empty()) + .map(std::path::PathBuf::from); + if let Some(home) = home { + let home = tokio::fs::canonicalize(&home).await.unwrap_or(home); + let rc = home.join(&rc_name); + if !read.contains(&rc) { + read.push(rc); + } + } + let mut rcs = Vec::new(); + for path in read { + if let Ok(text) = crate::utils::fs::read_regular_to_string(&path).await { + rcs.push(text); + } + } + BerryRegistrySettings { + rcs, + env_registry: std::env::var("YARN_NPM_REGISTRY_SERVER").ok(), + } + } + + fn registry(&self, name: &str) -> Result, String> { + berry_lookup_registry( + &self.rcs, + self.env_registry.as_deref(), + name, + &|key: &str| std::env::var(key).ok(), + ) } - yaml_top_level_value(text, "npmRegistryServer") } async fn restore_berry( @@ -689,10 +882,30 @@ async fn restore_berry( .iter() .map(|h| (h.uuid.clone(), h.name.clone(), h.version.clone())) .collect(); - let project_registry = yarnrc - .as_deref() - .and_then(|text| yaml_top_level_value(text, "npmRegistryServer")); - let registry = |name: &str| berry_lookup_registry(yarnrc.as_deref(), name); + // The registry yarn resolves each package against, read from every + // settings source yarn merges (#1017); one it cannot be known for is + // restored from the default registry's document, with a warning. + let settings = BerryRegistrySettings::read(view, &dir_prefix).await; + let mut registries: BTreeMap> = BTreeMap::new(); + for hit in &hits { + if registries.contains_key(&hit.name) { + continue; + } + let registry = settings.registry(&hit.name).unwrap_or_else(|why| { + result.warnings.push(( + "upstream_registry_fallback", + format!( + "{}@{}: the registry yarn resolves it against could not be determined \ + ({why}), so the entry was restored from the default registry's version \ + document; check its tarball URL against the project's registry", + hit.name, hit.version + ), + )); + None + }); + registries.insert(hit.name.clone(), registry); + } + let registry = |name: &str| registries.get(name).cloned().flatten(); let dists = fetch_dists_on(&wanted, registry, ctx, result).await; let mut changed = false; let mut moved: Vec = Vec::new(); @@ -732,7 +945,12 @@ async fn restore_berry( }; let resolution = format!( " resolution: \"{}\"", - berry_registry_locator(project_registry.as_deref(), &name, &version, &dist.tarball) + berry_registry_locator( + registries.get(&name).and_then(Option::as_deref), + &name, + &version, + &dist.tarball + ) ); let mut lines = stanza_lines(&blocks[idx]); if let Some(pinned) = with_body_field(&lines, "resolution", &resolution) { @@ -2108,7 +2326,7 @@ mod tests { bun_registry_slot, bun_tarball_url, expand_url_env, modules_yaml_pnpm_major, non_default_registry, package_json_pnpm_major, pnpm_include_tarball, pnpm_lookup_registry, pnpm_tarball_guess_warning, registry_derives_tarball, rush_json_pnpm_major, rush_lock_root, - split_userinfo, yaml_top_level_value, PnpmTarballGuess, ProjectDist, + split_userinfo, yaml_top_level_value, yarn_expand_env, PnpmTarballGuess, ProjectDist, }; use crate::patch::redirect::upstream::client::NpmDist; @@ -2898,17 +3116,17 @@ mod tests { /// content, so the first key is not `npmScopes`. #[test] fn berry_scopes_probe_reads_past_one_bom_only() { + let lookup = |rc: &str, name: &str| { + berry_lookup_registry(&[rc.to_string()], None, name, &|_: &str| None).unwrap() + }; let rc = "npmScopes:\n s:\n npmRegistryServer: https://s.example\n\ npmRegistryServer: https://m.example/\n"; for bom in ["", "\u{feff}"] { let rc = format!("{bom}{rc}"); - assert_eq!(berry_lookup_registry(Some(&rc), "@s/a"), None); + assert_eq!(lookup(&rc, "@s/a").as_deref(), Some("https://s.example")); } let rc = format!("\u{feff}\u{feff}{rc}"); - assert_eq!( - berry_lookup_registry(Some(&rc), "@s/a").as_deref(), - Some("https://m.example/") - ); + assert_eq!(lookup(&rc, "@s/a").as_deref(), Some("https://m.example/")); let rc = "npmRegistryServer: https://m.example/\n"; for bom in ["", "\u{feff}"] { assert_eq!( @@ -2922,24 +3140,131 @@ mod tests { ); } + /// #1017: yarn resolves a package's registry from every settings + /// source it merges, not only the project rc's top-level key. #[test] - fn berry_reads_the_project_registry_except_for_npm_scopes() { + fn berry_reads_the_registry_from_every_yarn_settings_source() { + let none = |_: &str| None::; + let lookup = |rcs: &[&str], env: Option<&str>, name: &str| { + let rcs: Vec = rcs.iter().map(|s| s.to_string()).collect(); + berry_lookup_registry(&rcs, env, name, &none) + }; let rc = "npmRegistryServer: \"https://m.example/npm/\"\n"; assert_eq!( - berry_lookup_registry(Some(rc), "a").as_deref(), + lookup(&[rc], None, "a").unwrap().as_deref(), + Some("https://m.example/npm/") + ); + assert_eq!( + lookup(&[rc], None, "@s/a").unwrap().as_deref(), + Some("https://m.example/npm/") + ); + assert_eq!(lookup(&[], None, "a").unwrap(), None); + + // npmScopes: the scope's own server, else the top-level one. + let scoped = format!( + "{rc}npmScopes:\n s:\n npmAlwaysAuth: true\n npmRegistryServer: \ + \"https://s.example\"\n t:\n npmAlwaysAuth: true\n" + ); + assert_eq!( + lookup(&[&scoped], None, "a").unwrap().as_deref(), + Some("https://m.example/npm/") + ); + assert_eq!( + lookup(&[&scoped], None, "@s/a").unwrap().as_deref(), + Some("https://s.example") + ); + assert_eq!( + lookup(&[&scoped], None, "@t/a").unwrap().as_deref(), Some("https://m.example/npm/") ); assert_eq!( - berry_lookup_registry(Some(rc), "@s/a").as_deref(), + lookup(&[&scoped], None, "@u/a").unwrap().as_deref(), Some("https://m.example/npm/") ); - let scoped = format!("{rc}npmScopes:\n s:\n npmRegistryServer: https://s.example\n"); + // A scope beats the env registry; the env registry beats every rc. + let env = Some("https://env.example"); + assert_eq!( + lookup(&[&scoped], env, "@s/a").unwrap().as_deref(), + Some("https://s.example") + ); + assert_eq!( + lookup(&[&scoped], env, "a").unwrap().as_deref(), + Some("https://env.example") + ); + assert_eq!( + lookup(&[], env, "a").unwrap().as_deref(), + Some("https://env.example") + ); + assert_eq!( + lookup(&[rc], Some(" "), "a").unwrap().as_deref(), + Some("https://m.example/npm/") + ); + + // Layers: the closer rc wins per key; a farther one (a parent + // directory's, the home one) fills in what the closer one lacks. + let parent = "npmRegistryServer: https://parent.example\n"; + let home = "npmScopes:\n s:\n npmRegistryServer: https://home-s.example\n"; + let project = "nodeLinker: node-modules\n"; + assert_eq!( + lookup(&[project, parent, home], None, "a") + .unwrap() + .as_deref(), + Some("https://parent.example") + ); + assert_eq!( + lookup(&[project, parent, home], None, "@s/a") + .unwrap() + .as_deref(), + Some("https://home-s.example") + ); assert_eq!( - berry_lookup_registry(Some(&scoped), "a").as_deref(), + lookup(&[rc, parent], None, "a").unwrap().as_deref(), Some("https://m.example/npm/") ); - assert_eq!(berry_lookup_registry(Some(&scoped), "@s/a"), None); - assert_eq!(berry_lookup_registry(None, "a"), None); + + // A flow-style npmScopes is not guessed at for a scoped name. + let flow = "npmScopes: {s: {npmRegistryServer: https://s.example}}\n"; + assert!(lookup(&[flow], None, "@s/a").is_err()); + assert_eq!( + lookup(&[flow, parent], None, "a").unwrap().as_deref(), + Some("https://parent.example") + ); + } + + #[test] + fn berry_registry_values_expand_env_references_like_yarn() { + let var = |name: &str| match name { + "REG" => Some("https://reg.example".to_string()), + "EMPTY" => Some(String::new()), + _ => None, + }; + for (value, want) in [ + ("${REG}", Ok("https://reg.example")), + ("${REG:-https://d.example}", Ok("https://reg.example")), + ("${UNSET:-https://d.example}", Ok("https://d.example")), + ("${UNSET-https://d.example}", Ok("https://d.example")), + ("${EMPTY:-https://d.example}", Ok("https://d.example")), + ("${EMPTY-https://d.example}", Ok("")), + ("${EMPTY}", Ok("")), + ("https://h/${REG}/x", Ok("https://h/https://reg.example/x")), + ("plain $ {REG} ${", Ok("plain $ {REG} ${")), + ("${UNSET}", Err(())), + ] { + assert_eq!( + yarn_expand_env(value, &var).map_err(|_| ()), + want.map(str::to_string), + "{value}" + ); + } + let rc = "npmRegistryServer: \"${REG:-http://127.0.0.1:8792}\"\n"; + assert_eq!( + berry_lookup_registry(&[rc.to_string()], None, "a", &|_: &str| None) + .unwrap() + .as_deref(), + Some("http://127.0.0.1:8792") + ); + let rc = "npmRegistryServer: \"${REG}\"\n"; + assert!(berry_lookup_registry(&[rc.to_string()], None, "a", &|_: &str| None).is_err()); } #[test] From e5e3b494dcef545a43a049af3bc2363630c1f26c Mon Sep 17 00:00:00 2001 From: Mikola Lysenko Date: Fri, 9 Oct 2026 14:19:33 -0400 Subject: [PATCH 3/3] Never expand set env vars into a registry URL A checked-in .yarnrc.yml could name any variable of the socket-patch process (a token) in npmRegistryServer. The restore would then request, and on a failed lookup print, a URL carrying it. A reference to a variable that is set is now treated as an undeterminable registry. The restore falls back to the default registry with upstream_registry_fallback, as the Bun and pnpm settings readers already do. An unset variable's default is still read the way yarn reads it. The hosted-unwind test runner also clears YARN_NPM_REGISTRY_SERVER and YARN_RC_FILENAME, so an ambient value can't steer the fixtures. Co-Authored-By: Claude Opus 5.5 (1M context) --- crates/socket-patch-cli/CLI_CONTRACT.md | 4 +-- .../tests/in_process_redirect.rs | 4 +++ .../src/patch/redirect/upstream/npm.rs | 28 +++++++++++++++---- 3 files changed, 28 insertions(+), 8 deletions(-) diff --git a/crates/socket-patch-cli/CLI_CONTRACT.md b/crates/socket-patch-cli/CLI_CONTRACT.md index 13b3cee11..2efabb478 100644 --- a/crates/socket-patch-cli/CLI_CONTRACT.md +++ b/crates/socket-patch-cli/CLI_CONTRACT.md @@ -959,7 +959,7 @@ v5.0 replaces v4's per-purl reverts and whole-ledger reverse replay (`revert_rem * **Scope.** The hosted pins are what lockfile discovery finds — `(purl, patch uuid, files wiring it)`, recognized only on `https://patch.socket.dev` or the `--patch-server-url` / `SOCKET_PATCH_SERVER_URL` origin. A scoped rollback (paths / identifiers / `--ecosystems`) restores exactly the pins in scope; each pin restores or refuses on its own (there is no whole-ledger replay, and a pre-v5 ledger's edits are never replayed). A pin discovery cannot see is out of reach: a lockless cargo `registry = "socket-patch-"` pin, a nuget exact-id mapping with no `packages.lock.json`, a gem wired only in the `Gemfile` (pre-bundler-2.6 mixed state) — restore those files from version control. * **What a restore does.** Every file wiring the pin is rewritten back to the DEFAULT UPSTREAM registry entry for `name@version`, re-resolving whatever the entry pins (tarball URL, integrity, checksum, hashes) from the public registry; only the hosted entries change and every other byte stays the file's own. A pin is **all-or-nothing**: refused in one of its files, it is restored in none of them, so no pin is left half hosted. Nothing reaches disk until every pin has resolved, and `--dry-run` resolves exactly like a wet run — registry lookups included — and skips only the write. Per format: - * **npm family** — `package-lock.json` / `npm-shrinkwrap.json`, `yarn.lock` (classic and berry), `pnpm-lock.yaml` / `shrinkwrap.yaml`, `bun.lock`: resolution + integrity (+ shasum where recorded) from the npm registry's version document (`SOCKET_NPM_REGISTRY`); a yarn berry lock whose settings name another registry reads that registry's document instead — the registry yarn itself resolves the package against (#1017): a scoped package's `npmScopes..npmRegistryServer`, else `YARN_NPM_REGISTRY_SERVER`, else `npmRegistryServer`, each from the closest `.yarnrc.yml` (`YARN_RC_FILENAME`) that sets it among the lock's directory, every parent directory and then the home directory, with `${VAR}` / `${VAR:-default}` / `${VAR-default}` expanded as yarn expands them (a reference yarn cannot expand, or an `npmScopes` written as a flow mapping, falls back to the default registry with `upstream_registry_fallback`), so a mirror's off-path `dist.tarball` keeps its `::__archiveUrl=` binding, and a pnpm lock whose sibling settings name a registry reads that registry's document — `.npmrc` `registry` (or, for a scoped name, `@scope:registry`); on pnpm 10 a pnpm-workspace.yaml `registries` map instead when present; on pnpm 11+ (or an unknown major) pnpm-workspace.yaml `registries."@scope"` / `registry` / `registries.default` too, ahead of the matching `.npmrc` key — so a mirror's `tarball:` comes back as pnpm recorded it and a URL conventional under that registry stays derived (falling back to the default registry, with `upstream_registry_fallback`, when the mirror can't be read; a value holding an unexpanded `${VAR}` is read as unset). Whether a restored pnpm entry gets its `tarball:` back follows `lockfileIncludeTarballUrl` as the pnpm that wrote the lock read it (#902), from the strongest evidence available: (1) the lock's own unpinned registry resolutions (a bare one proves it off; a URL pnpm could have derived proves it on; pnpm 11+'s env lockfile document does not count); else (2) the settings file the installed pnpm major reads (`node_modules/.modules.yaml` `packageManager`, else package.json `packageManager`; a pre-9 lock or shrinkwrap means pnpm <= 8): `.npmrc` `lockfile-include-tarball-url` on pnpm <= 9, pnpm-workspace.yaml `lockfileIncludeTarballUrl` on pnpm >= 11 (also assumed for a lock carrying an env lockfile document), the workspace file then `.npmrc` on pnpm 10; else (3) pnpm 10's reading. A Rush lock (`common/config/rush/pnpm-lock.yaml` or a subspace lock, with `rush.json` at the Rush root) takes its pnpm major from rush.json `pnpmVersion` instead of tier 2's install record and package.json pin. A 9.0 lock may come from pnpm 9, 10 or 11+, so when tier 3's reading differs from pnpm 9's (`.npmrc` only) or pnpm >= 11's (pnpm-workspace.yaml only) — e.g. `.npmrc` on with the workspace file silent, or the workspace file setting it with `.npmrc` silent or disagreeing — the restore follows pnpm 10 but warns `upstream_pnpm_tarball_setting_guessed` (once per lock, naming the entries); a URL pnpm records anyway (not derivable from the registry) never warns. A `bun.lock` 4-tuple's registry slot is rebuilt the way Bun writes it (#992): `""` for a package from registry.npmjs.org, otherwise the full tarball URL — Bun 1.1.39–1.3.6 read `""` as npmjs whatever the project configures. The registry is the one Bun resolves the package against: a scope's `.npmrc` `@scope:registry` or `bunfig.toml` `[install.scopes]` entry, else `BUN_CONFIG_REGISTRY` / `NPM_CONFIG_REGISTRY`, the `.npmrc` `registry`, then `bunfig.toml` `[install] registry`; its version document's `dist.tarball` fills the slot, and when it can't be read (`upstream_registry_fallback`) the default registry's conventional URL is re-based on it. The `bun.lockb` takeover restore records the same URL. Side settings: a project `.npmrc` that is exactly `allow-remote=all\n` is deleted once no root npm lock entry is hosted, otherwise a remaining top-level `allow-remote=all` warns `npm_allow_remote_left`; a `pnpm-workspace.yaml` that is exactly the scaffold hosted mode creates is deleted once `pnpm-lock.yaml` is no longer hosted, otherwise a remaining `trustLockfile: true` warns `pnpm_trust_lockfile_left`. **`bun.lockb` (binary)**: `rollback` and `remove` refuse it (the checkout remedy). The hosted → vendored takeover and the eject DO restore it, since the vendor ledger then records the rebuilt record as its pre-vendor original: the native codec turns each hosted remote-tarball record back into Bun's npm registry record for `name@version` (the registry's `dist.tarball` + `dist.integrity`, the package metadata hash re-derived, the hosted URL string dropped from the string pool). The hosted rewrite keeps the registry record's inactive bytes (padding, semver) in the tarball record, so a lock it wrote comes back byte for byte — early writers' uninitialized padding included; a record without them (an older socket-patch or a Bun re-save) is rebuilt the way Bun writes one, and refused for a prerelease/build version. A lock the hosted rewrite had to normalize is marked in the root package's resolution value bytes (which no Bun reader reads): a binary format 1 lock it promoted to format 2 is demoted back to its exact format-1 bytes (verified by promoting it again, otherwise refused), and a lock whose workspace dependency behaviors it normalized is refused with the `git checkout -- bun.lockb` remedy. + * **npm family** — `package-lock.json` / `npm-shrinkwrap.json`, `yarn.lock` (classic and berry), `pnpm-lock.yaml` / `shrinkwrap.yaml`, `bun.lock`: resolution + integrity (+ shasum where recorded) from the npm registry's version document (`SOCKET_NPM_REGISTRY`); a yarn berry lock whose settings name another registry reads that registry's document instead — the registry yarn itself resolves the package against (#1017): a scoped package's `npmScopes..npmRegistryServer`, else `YARN_NPM_REGISTRY_SERVER`, else `npmRegistryServer`, each from the closest `.yarnrc.yml` (`YARN_RC_FILENAME`) that sets it among the lock's directory, every parent directory and then the home directory, with a `${VAR:-default}` / `${VAR-default}` reference to an unset variable read as its default, as yarn reads it (a reference to a variable that is set is not expanded — the rc file must not choose which of the process's variables lands in a requested URL — and, like a reference yarn cannot expand or an `npmScopes` written as a flow mapping, falls back to the default registry with `upstream_registry_fallback`), so a mirror's off-path `dist.tarball` keeps its `::__archiveUrl=` binding, and a pnpm lock whose sibling settings name a registry reads that registry's document — `.npmrc` `registry` (or, for a scoped name, `@scope:registry`); on pnpm 10 a pnpm-workspace.yaml `registries` map instead when present; on pnpm 11+ (or an unknown major) pnpm-workspace.yaml `registries."@scope"` / `registry` / `registries.default` too, ahead of the matching `.npmrc` key — so a mirror's `tarball:` comes back as pnpm recorded it and a URL conventional under that registry stays derived (falling back to the default registry, with `upstream_registry_fallback`, when the mirror can't be read; a value holding an unexpanded `${VAR}` is read as unset). Whether a restored pnpm entry gets its `tarball:` back follows `lockfileIncludeTarballUrl` as the pnpm that wrote the lock read it (#902), from the strongest evidence available: (1) the lock's own unpinned registry resolutions (a bare one proves it off; a URL pnpm could have derived proves it on; pnpm 11+'s env lockfile document does not count); else (2) the settings file the installed pnpm major reads (`node_modules/.modules.yaml` `packageManager`, else package.json `packageManager`; a pre-9 lock or shrinkwrap means pnpm <= 8): `.npmrc` `lockfile-include-tarball-url` on pnpm <= 9, pnpm-workspace.yaml `lockfileIncludeTarballUrl` on pnpm >= 11 (also assumed for a lock carrying an env lockfile document), the workspace file then `.npmrc` on pnpm 10; else (3) pnpm 10's reading. A Rush lock (`common/config/rush/pnpm-lock.yaml` or a subspace lock, with `rush.json` at the Rush root) takes its pnpm major from rush.json `pnpmVersion` instead of tier 2's install record and package.json pin. A 9.0 lock may come from pnpm 9, 10 or 11+, so when tier 3's reading differs from pnpm 9's (`.npmrc` only) or pnpm >= 11's (pnpm-workspace.yaml only) — e.g. `.npmrc` on with the workspace file silent, or the workspace file setting it with `.npmrc` silent or disagreeing — the restore follows pnpm 10 but warns `upstream_pnpm_tarball_setting_guessed` (once per lock, naming the entries); a URL pnpm records anyway (not derivable from the registry) never warns. A `bun.lock` 4-tuple's registry slot is rebuilt the way Bun writes it (#992): `""` for a package from registry.npmjs.org, otherwise the full tarball URL — Bun 1.1.39–1.3.6 read `""` as npmjs whatever the project configures. The registry is the one Bun resolves the package against: a scope's `.npmrc` `@scope:registry` or `bunfig.toml` `[install.scopes]` entry, else `BUN_CONFIG_REGISTRY` / `NPM_CONFIG_REGISTRY`, the `.npmrc` `registry`, then `bunfig.toml` `[install] registry`; its version document's `dist.tarball` fills the slot, and when it can't be read (`upstream_registry_fallback`) the default registry's conventional URL is re-based on it. The `bun.lockb` takeover restore records the same URL. Side settings: a project `.npmrc` that is exactly `allow-remote=all\n` is deleted once no root npm lock entry is hosted, otherwise a remaining top-level `allow-remote=all` warns `npm_allow_remote_left`; a `pnpm-workspace.yaml` that is exactly the scaffold hosted mode creates is deleted once `pnpm-lock.yaml` is no longer hosted, otherwise a remaining `trustLockfile: true` warns `pnpm_trust_lockfile_left`. **`bun.lockb` (binary)**: `rollback` and `remove` refuse it (the checkout remedy). The hosted → vendored takeover and the eject DO restore it, since the vendor ledger then records the rebuilt record as its pre-vendor original: the native codec turns each hosted remote-tarball record back into Bun's npm registry record for `name@version` (the registry's `dist.tarball` + `dist.integrity`, the package metadata hash re-derived, the hosted URL string dropped from the string pool). The hosted rewrite keeps the registry record's inactive bytes (padding, semver) in the tarball record, so a lock it wrote comes back byte for byte — early writers' uninitialized padding included; a record without them (an older socket-patch or a Bun re-save) is rebuilt the way Bun writes one, and refused for a prerelease/build version. A lock the hosted rewrite had to normalize is marked in the root package's resolution value bytes (which no Bun reader reads): a binary format 1 lock it promoted to format 2 is demoted back to its exact format-1 bytes (verified by promoting it again, otherwise refused), and a lock whose workspace dependency behaviors it normalized is refused with the `git checkout -- bun.lockb` remedy. * **vlt** — `vlt-lock.json`: slot [2] from the registry's `dist.integrity`, slot [3] per the lock's own convention (see the vlt hosted-mode contract); every hosted instance of the pin together. * **cargo** — `Cargo.lock` back on crates.io (source + the sparse index's checksum, `SOCKET_CRATES_INDEX`); every `Cargo.toml` declaration loses its `registry = "socket-patch-"` pin (the shorthand the rewriter produced collapses back); every `[registries.socket-patch-]` block no manifest or lock still references leaves the project cargo config — including a superseded patch generation's block an earlier re-pin left behind (#864). A declaration it cannot unpin refuses. * **golang** — the hosted `replace` and the socket module's go.sum lines go; the upstream module's two go.sum lines come back, hashed from the module proxy (`SOCKET_GOPROXY`, else `GOPROXY` / `GONOPROXY` / `GOPRIVATE` as go reads them) and cross-checked against the checksum database (`SOCKET_GOSUMDB_URL`, else `sum.golang.org` unless `GOSUMDB=off` / `GONOSUMDB` / `GOPRIVATE` say go would not ask it). A `replace` the user had before the hosted run is not recorded anywhere, so the restore lands on the plain upstream module. @@ -1281,7 +1281,7 @@ Every `--json` invocation emits a single JSON object that follows the **unified | `manifest_write_failed` | rollback `warnings[]` | rollback (v5.0): the post-rollback manifest update could not be written; no entries were removed (`manifest.removedEntries: []`) and the run exits `partial_failure` 1. | | `npm_allow_remote_left` / `pnpm_trust_lockfile_left` | rollback/remove `warnings[]`; vendor advisory event (takeover) | upstream restore (v5.0): no npm-family lock entry is hosted any more, but the project `.npmrc` keeps a top-level `allow-remote=all` (resp. `pnpm-workspace.yaml` keeps `trustLockfile: true`) in a file that is not exactly what hosted mode creates; the file is left untouched (v5 records no provenance), remove the line if nothing else needs it. A file that is exactly hosted mode's own is deleted silently. | | `maven_trusted_checksums_left` / `nuget_default_config_left` / `upstream_uv_override_removed` | rollback/remove `warnings[]`; vendor advisory event (takeover) | upstream restore (v5.0): `.mvn` config keeps the trusted-checksums resolver lines because it holds more than hosted mode writes; `nuget.config` now holds only the nuget.org source (delete it if hosted mode created it); a transitive `override-dependencies` entry hosted mode added to `pyproject.toml` was removed. | -| `upstream_registry_fallback` | rollback/remove `warnings[]`; vendor advisory event (takeover) | upstream restore: a yarn berry, pnpm or vlt entry is restored from the version document of the registry the project resolves it against (yarn berry: `npmScopes` / `YARN_NPM_REGISTRY_SERVER` / `npmRegistryServer` from the `.yarnrc.yml` chain, as above, the pnpm lock's sibling `.npmrc` `registry` / `@scope:registry` or pnpm-workspace.yaml `registry` / `registries`, vlt's node registry); that registry could not be read (e.g. it needs credentials) or, for yarn berry, could not be determined (an rc value referencing an unset variable with no default, a flow-style `npmScopes`), so the default registry's document was used and the restored tarball URL may not be the mirror's. | +| `upstream_registry_fallback` | rollback/remove `warnings[]`; vendor advisory event (takeover) | upstream restore: a yarn berry, pnpm or vlt entry is restored from the version document of the registry the project resolves it against (yarn berry: `npmScopes` / `YARN_NPM_REGISTRY_SERVER` / `npmRegistryServer` from the `.yarnrc.yml` chain, as above, the pnpm lock's sibling `.npmrc` `registry` / `@scope:registry` or pnpm-workspace.yaml `registry` / `registries`, vlt's node registry); that registry could not be read (e.g. it needs credentials) or, for yarn berry, could not be determined (an rc value referencing an environment variable that is set, or an unset one with no default; a flow-style `npmScopes`), so the default registry's document was used and the restored tarball URL may not be the mirror's. | | `upstream_gem_stale_cache` | rollback/remove `warnings[]`; vendor advisory event (takeover) | upstream restore (#1260): a restored gem's `-.gem` is still in Bundler's cache dir (`cache_path`, default `vendor/cache`, resolved as for the Gem stale-install guard) and its sha256 is not the upstream one (the restored `CHECKSUMS` entry, else the rubygems.org compact index), or it could not be checked (`--offline`, a registry error). Bundler installs from that dir first, so a `bundle cache` taken while the hosted pin was live makes every later install fail on the upstream checksum (exit 37) or, on bundler < 2.6 frozen installs, keep installing the patched bytes. The detail names the file. Remedy: delete it, then run `bundle cache` to cache the upstream gem in its place (or `bundle install` if the project does not commit its cache; with the cache dir committed, a frozen install reads only the cache). Read-only: the restore never deletes it. Not raised for an archive whose sha256 matches upstream. | | `upstream_pnpm_tarball_setting_guessed` | rollback/remove `warnings[]`; vendor advisory event (takeover) | upstream restore (#902): nothing showed which pnpm wrote a hosted `pnpm-lock.yaml` (no unpinned registry entry that shows the setting, no `node_modules/.modules.yaml` install record, no package.json `packageManager` pin; for a Rush lock, no rush.json `pnpmVersion`), so its entries were restored with or without `tarball:` by pnpm 10's reading of `lockfileIncludeTarballUrl` (pnpm-workspace.yaml, else `.npmrc` `lockfile-include-tarball-url`), and pnpm 9 (which reads only `.npmrc`) or pnpm >= 11 (which reads only pnpm-workspace.yaml) would have read it the other way. The detail names the lock, the setting followed, the pnpm that disagrees and the entries. Remedy: pin the pnpm (package.json `packageManager`, or reinstall so the install record names it; rush.json `pnpmVersion` for Rush), or give the two files the same value so every pnpm reads it alike; a rollback or remove can then be redone by restoring the lock from version control and re-running. Not raised when evidence decided, when both files read the same on every pnpm, or when the tarball is one pnpm records regardless. | | `legacy_redirect_ledger_kept` | rollback `warnings[]` (+ remove stderr) | v5.0: a pre-v5 `.socket/vendor/redirect-state.json` could not be deleted once no hosted pin was left; the file is inert (never read for planning). Never flips the exit. | diff --git a/crates/socket-patch-cli/tests/in_process_redirect.rs b/crates/socket-patch-cli/tests/in_process_redirect.rs index 1e822c091..4eacf397d 100644 --- a/crates/socket-patch-cli/tests/in_process_redirect.rs +++ b/crates/socket-patch-cli/tests/in_process_redirect.rs @@ -1783,6 +1783,10 @@ fn hosted_unwind_json( "SOCKET_NPM_REGISTRY", format!("{}/npm-registry", registry.uri()), ) + // The yarn berry restore reads these like yarn does (#1017): an + // ambient value must not steer the fixtures' registry. + .env_remove("YARN_NPM_REGISTRY_SERVER") + .env_remove("YARN_RC_FILENAME") .output() .unwrap_or_else(|e| panic!("run socket-patch {}: {e}", command[0])); let env_json: serde_json::Value = serde_json::from_slice(&out.stdout).unwrap_or_else(|e| { diff --git a/crates/socket-patch-core/src/patch/redirect/upstream/npm.rs b/crates/socket-patch-core/src/patch/redirect/upstream/npm.rs index 0e3ba1efb..fd118787f 100644 --- a/crates/socket-patch-core/src/patch/redirect/upstream/npm.rs +++ b/crates/socket-patch-core/src/patch/redirect/upstream/npm.rs @@ -625,10 +625,17 @@ fn yaml_path_in(lines: &[String], path: &[&str]) -> Result, Strin yaml_path_in(&dedented, rest) } -/// A `.yarnrc.yml` value with its environment references expanded as yarn -/// expands them: `${NAME}`, `${NAME-fallback}` (the fallback when `NAME` +/// A `.yarnrc.yml` registry value with its environment references read as +/// yarn reads them: `${NAME}`, `${NAME-fallback}` (the fallback when `NAME` /// is unset) and `${NAME:-fallback}` (also when it is empty). A reference /// to an unset variable with no fallback is the error yarn stops on. +/// +/// A reference to a variable that IS set is never expanded: the rc file +/// (possibly a checked-in, lower-trust one) would choose which of this +/// process's variables — a token, say — lands in a URL the restore +/// requests and may print. That is `Err` too, so the restore falls back to +/// the default registry with `upstream_registry_fallback`, as for the Bun +/// and pnpm settings, which never expand an arbitrary variable either. fn yarn_expand_env(value: &str, var: &dyn Fn(&str) -> Option) -> Result { let mut out = String::with_capacity(value.len()); let mut rest = value; @@ -662,7 +669,12 @@ fn yarn_expand_env(value: &str, var: &dyn Fn(&str) -> Option) -> Result< }; let set = var(name); let expanded = match (set, fallback) { - (Some(v), _) if !v.is_empty() => v, + (Some(v), _) if !v.is_empty() => { + return Err(format!( + "it references the environment variable {name}, which socket-patch does \ + not expand into a registry URL" + )) + } (Some(v), _) if !colon => v, (_, Some(fallback)) => fallback.to_string(), (_, None) => return Err(format!("the environment variable {name} is not set")), @@ -3239,14 +3251,18 @@ mod tests { _ => None, }; for (value, want) in [ - ("${REG}", Ok("https://reg.example")), - ("${REG:-https://d.example}", Ok("https://reg.example")), + // A set variable is never expanded (the rc file must not pick + // which of the process's variables lands in a requested URL). + ("${REG}", Err(())), + ("${REG:-https://d.example}", Err(())), + ("${REG-https://d.example}", Err(())), ("${UNSET:-https://d.example}", Ok("https://d.example")), ("${UNSET-https://d.example}", Ok("https://d.example")), ("${EMPTY:-https://d.example}", Ok("https://d.example")), ("${EMPTY-https://d.example}", Ok("")), ("${EMPTY}", Ok("")), - ("https://h/${REG}/x", Ok("https://h/https://reg.example/x")), + ("https://h/${REG}/x", Err(())), + ("https://h/${UNSET:-m}/x", Ok("https://h/m/x")), ("plain $ {REG} ${", Ok("plain $ {REG} ${")), ("${UNSET}", Err(())), ] {