diff --git a/crates/socket-patch-cli/CLI_CONTRACT.md b/crates/socket-patch-cli/CLI_CONTRACT.md index b9921c653..bcbca4bb9 100644 --- a/crates/socket-patch-cli/CLI_CONTRACT.md +++ b/crates/socket-patch-cli/CLI_CONTRACT.md @@ -1017,7 +1017,7 @@ v5.0 replaces v4's per-purl reverts and whole-ledger reverse replay (`revert_rem * **Scope.** The hosted pins are what lockfile discovery finds — `(purl, patch uuid, files wiring it)`, recognized only on `https://patch.socket.dev` or the `--patch-server-url` / `SOCKET_PATCH_SERVER_URL` origin. A scoped rollback (paths / identifiers / `--ecosystems`) restores exactly the pins in scope; each pin restores or refuses on its own (there is no whole-ledger replay, and a pre-v5 ledger's edits are never replayed). A pin discovery cannot see is out of reach: a lockless cargo `registry = "socket-patch-"` pin, a nuget exact-id mapping with no `packages.lock.json`, a gem wired only in the `Gemfile` (pre-bundler-2.6 mixed state) — restore those files from version control. * **What a restore does.** Every file wiring the pin is rewritten back to the DEFAULT UPSTREAM registry entry for `name@version`, re-resolving whatever the entry pins (tarball URL, integrity, checksum, hashes) from the public registry; only the hosted entries change and every other byte stays the file's own. A pin is **all-or-nothing**: refused in one of its files, it is restored in none of them, so no pin is left half hosted. Nothing reaches disk until every pin has resolved, and `--dry-run` resolves exactly like a wet run — registry lookups included — and skips only the write. Per format: - * **npm family** — `package-lock.json` / `npm-shrinkwrap.json`, `yarn.lock` (classic and berry), `pnpm-lock.yaml` / `shrinkwrap.yaml`, `bun.lock`: resolution + integrity (+ shasum where recorded) from the npm registry's version document (`SOCKET_NPM_REGISTRY`); a yarn berry lock whose `.yarnrc.yml` names another `npmRegistryServer` reads that registry's document instead, so a mirror's off-path `dist.tarball` keeps its `::__archiveUrl=` binding (a berry entry the hosted pin keyed by its tarball URL also takes the version document's `bin` back, in place of the served tarball's own spelling the pin wrote, as yarn writes the `npm:` entry, #1131), and a pnpm lock whose sibling settings name a registry reads that registry's document — `.npmrc` `registry` (or, for a scoped name, `@scope:registry`); on pnpm 10 a pnpm-workspace.yaml `registries` map instead when present; on pnpm 11+ (or an unknown major) pnpm-workspace.yaml `registries."@scope"` / `registry` / `registries.default` too, ahead of the matching `.npmrc` key — so a mirror's `tarball:` comes back as pnpm recorded it and a URL conventional under that registry stays derived (falling back to the default registry, with `upstream_registry_fallback`, when the mirror can't be read; a value holding an unexpanded `${VAR}` is read as unset). Whether a restored pnpm entry gets its `tarball:` back follows `lockfileIncludeTarballUrl` as the pnpm that wrote the lock read it (#902), from the strongest evidence available: (1) the lock's own unpinned registry resolutions (a bare one proves it off; a URL pnpm could have derived proves it on; pnpm 11+'s env lockfile document does not count); else (2) the settings file the installed pnpm major reads (`node_modules/.modules.yaml` `packageManager`, else package.json `packageManager`; a pre-9 lock or shrinkwrap means pnpm <= 8): `.npmrc` `lockfile-include-tarball-url` on pnpm <= 9, pnpm-workspace.yaml `lockfileIncludeTarballUrl` on pnpm >= 11 (also assumed for a lock carrying an env lockfile document), the workspace file then `.npmrc` on pnpm 10; else (3) pnpm 10's reading. A Rush lock (`common/config/rush/pnpm-lock.yaml` or a subspace lock, with `rush.json` at the Rush root) takes its pnpm major from rush.json `pnpmVersion` instead of tier 2's install record and package.json pin. A 9.0 lock may come from pnpm 9, 10 or 11+, so when tier 3's reading differs from pnpm 9's (`.npmrc` only) or pnpm >= 11's (pnpm-workspace.yaml only) — e.g. `.npmrc` on with the workspace file silent, or the workspace file setting it with `.npmrc` silent or disagreeing — the restore follows pnpm 10 but warns `upstream_pnpm_tarball_setting_guessed` (once per lock, naming the entries); a URL pnpm records anyway (not derivable from the registry) never warns. A `bun.lock` 4-tuple's registry slot is rebuilt the way Bun writes it (#992): `""` for a package from registry.npmjs.org, otherwise the full tarball URL — Bun 1.1.39–1.3.6 read `""` as npmjs whatever the project configures. The registry is the one Bun resolves the package against: a scope's `.npmrc` `@scope:registry` or `bunfig.toml` `[install.scopes]` entry, else `BUN_CONFIG_REGISTRY` / `NPM_CONFIG_REGISTRY`, then the `.npmrc` `registry` or `bunfig.toml` `[install] registry`. Bun reads these from the files beside the lock and from the user's own (#1276): `$XDG_CONFIG_HOME/.npmrc` when it exists, else `~/.npmrc` (never `NPM_CONFIG_USERCONFIG`), and the global bunfig `$XDG_CONFIG_HOME/.bunfig.toml` when `XDG_CONFIG_HOME` is set, else `~/.bunfig.toml`. A key set beside the lock wins over the user's own file of the same kind. Between the two kinds, Bun ≤ 1.3 takes any `.npmrc` over any bunfig and Bun ≥ 1.4 any bunfig over any `.npmrc`. A lockfileVersion-2 `bun.lock` is Bun ≥ 1.4's (Bun 1.3 ignores it); when the kinds disagree on a package of a lockfileVersion-0/1 `bun.lock` (which Bun 1.4 keeps as is) or a `bun.lockb`, the restore refuses that pin rather than guess. Only the conventional token variables (`NPM_TOKEN`, `NODE_AUTH_TOKEN`, `BUN_AUTH_TOKEN`) expand in the project's files; the user's own files expand any variable, as Bun does. Its version document's `dist.tarball` fills the slot, and when it can't be read (`upstream_registry_fallback`) the default registry's conventional URL is re-based on it. The `bun.lockb` takeover restore records the same URL. Side settings: a project `.npmrc` that is exactly `allow-remote=all\n` is deleted once no root npm lock entry is hosted, otherwise a remaining top-level `allow-remote=all` warns `npm_allow_remote_left`; a `pnpm-workspace.yaml` that is exactly the scaffold hosted mode creates is deleted once `pnpm-lock.yaml` is no longer hosted, otherwise a remaining `trustLockfile: true` warns `pnpm_trust_lockfile_left`. **`bun.lockb` (binary)**: `rollback` and `remove` refuse it (the checkout remedy). The hosted → vendored takeover and the eject DO restore it, since the vendor ledger then records the rebuilt record as its pre-vendor original: the native codec turns each hosted remote-tarball record back into Bun's npm registry record for `name@version` (the registry's `dist.tarball` + `dist.integrity`, the package metadata hash re-derived, the hosted URL string dropped from the string pool). The hosted rewrite keeps the registry record's inactive bytes (padding, semver) in the tarball record, so a lock it wrote comes back byte for byte — early writers' uninitialized padding included; a record without them (an older socket-patch or a Bun re-save) is rebuilt the way Bun writes one, and refused for a prerelease/build version. A lock the hosted rewrite had to normalize is marked in the root package's resolution value bytes (which no Bun reader reads): a binary format 1 lock it promoted to format 2 is demoted back to its exact format-1 bytes (verified by promoting it again, otherwise refused), and a lock whose workspace dependency behaviors it normalized is refused with the `git checkout -- bun.lockb` remedy. + * **npm family** — `package-lock.json` / `npm-shrinkwrap.json`, `yarn.lock` (classic and berry), `pnpm-lock.yaml` / `shrinkwrap.yaml`, `bun.lock`: resolution + integrity (+ shasum where recorded) from the npm registry's version document (`SOCKET_NPM_REGISTRY`); a yarn berry lock whose settings name another registry reads that registry's document instead — the registry yarn itself resolves the package against (#1017): a scoped package's `npmScopes..npmRegistryServer`, else `YARN_NPM_REGISTRY_SERVER`, else `npmRegistryServer`, each from the closest `.yarnrc.yml` (`YARN_RC_FILENAME`) that sets it among the lock's directory, every parent directory and then the home directory, with a `${VAR:-default}` / `${VAR-default}` reference to an unset variable read as its default, as yarn reads it (a reference to a variable that is set is not expanded — the rc file must not choose which of the process's variables lands in a requested URL — and, like a reference yarn cannot expand or an `npmScopes` written as a flow mapping, falls back to the default registry with `upstream_registry_fallback`), so a mirror's off-path `dist.tarball` keeps its `::__archiveUrl=` binding (a berry entry the hosted pin keyed by its tarball URL also takes the version document's `bin` back, in place of the served tarball's own spelling the pin wrote, as yarn writes the `npm:` entry, #1131), and a pnpm lock whose sibling settings name a registry reads that registry's document — `.npmrc` `registry` (or, for a scoped name, `@scope:registry`); on pnpm 10 a pnpm-workspace.yaml `registries` map instead when present; on pnpm 11+ (or an unknown major) pnpm-workspace.yaml `registries."@scope"` / `registry` / `registries.default` too, ahead of the matching `.npmrc` key — so a mirror's `tarball:` comes back as pnpm recorded it and a URL conventional under that registry stays derived (falling back to the default registry, with `upstream_registry_fallback`, when the mirror can't be read; a value holding an unexpanded `${VAR}` is read as unset). Whether a restored pnpm entry gets its `tarball:` back follows `lockfileIncludeTarballUrl` as the pnpm that wrote the lock read it (#902), from the strongest evidence available: (1) the lock's own unpinned registry resolutions (a bare one proves it off; a URL pnpm could have derived proves it on; pnpm 11+'s env lockfile document does not count); else (2) the settings file the installed pnpm major reads (`node_modules/.modules.yaml` `packageManager`, else package.json `packageManager`; a pre-9 lock or shrinkwrap means pnpm <= 8): `.npmrc` `lockfile-include-tarball-url` on pnpm <= 9, pnpm-workspace.yaml `lockfileIncludeTarballUrl` on pnpm >= 11 (also assumed for a lock carrying an env lockfile document), the workspace file then `.npmrc` on pnpm 10; else (3) pnpm 10's reading. A Rush lock (`common/config/rush/pnpm-lock.yaml` or a subspace lock, with `rush.json` at the Rush root) takes its pnpm major from rush.json `pnpmVersion` instead of tier 2's install record and package.json pin. A 9.0 lock may come from pnpm 9, 10 or 11+, so when tier 3's reading differs from pnpm 9's (`.npmrc` only) or pnpm >= 11's (pnpm-workspace.yaml only) — e.g. `.npmrc` on with the workspace file silent, or the workspace file setting it with `.npmrc` silent or disagreeing — the restore follows pnpm 10 but warns `upstream_pnpm_tarball_setting_guessed` (once per lock, naming the entries); a URL pnpm records anyway (not derivable from the registry) never warns. A `bun.lock` 4-tuple's registry slot is rebuilt the way Bun writes it (#992): `""` for a package from registry.npmjs.org, otherwise the full tarball URL — Bun 1.1.39–1.3.6 read `""` as npmjs whatever the project configures. The registry is the one Bun resolves the package against: a scope's `.npmrc` `@scope:registry` or `bunfig.toml` `[install.scopes]` entry, else `BUN_CONFIG_REGISTRY` / `NPM_CONFIG_REGISTRY`, then the `.npmrc` `registry` or `bunfig.toml` `[install] registry`. Bun reads these from the files beside the lock and from the user's own (#1276): `$XDG_CONFIG_HOME/.npmrc` when it exists, else `~/.npmrc` (never `NPM_CONFIG_USERCONFIG`), and the global bunfig `$XDG_CONFIG_HOME/.bunfig.toml` when `XDG_CONFIG_HOME` is set, else `~/.bunfig.toml`. A key set beside the lock wins over the user's own file of the same kind. Between the two kinds, Bun ≤ 1.3 takes any `.npmrc` over any bunfig and Bun ≥ 1.4 any bunfig over any `.npmrc`. A lockfileVersion-2 `bun.lock` is Bun ≥ 1.4's (Bun 1.3 ignores it); when the kinds disagree on a package of a lockfileVersion-0/1 `bun.lock` (which Bun 1.4 keeps as is) or a `bun.lockb`, the restore refuses that pin rather than guess. Only the conventional token variables (`NPM_TOKEN`, `NODE_AUTH_TOKEN`, `BUN_AUTH_TOKEN`) expand in the project's files; the user's own files expand any variable, as Bun does. Its version document's `dist.tarball` fills the slot, and when it can't be read (`upstream_registry_fallback`) the default registry's conventional URL is re-based on it. The `bun.lockb` takeover restore records the same URL. Side settings: a project `.npmrc` that is exactly `allow-remote=all\n` is deleted once no root npm lock entry is hosted, otherwise a remaining top-level `allow-remote=all` warns `npm_allow_remote_left`; a `pnpm-workspace.yaml` that is exactly the scaffold hosted mode creates is deleted once `pnpm-lock.yaml` is no longer hosted, otherwise a remaining `trustLockfile: true` warns `pnpm_trust_lockfile_left`. **`bun.lockb` (binary)**: `rollback` and `remove` refuse it (the checkout remedy). The hosted → vendored takeover and the eject DO restore it, since the vendor ledger then records the rebuilt record as its pre-vendor original: the native codec turns each hosted remote-tarball record back into Bun's npm registry record for `name@version` (the registry's `dist.tarball` + `dist.integrity`, the package metadata hash re-derived, the hosted URL string dropped from the string pool). The hosted rewrite keeps the registry record's inactive bytes (padding, semver) in the tarball record, so a lock it wrote comes back byte for byte — early writers' uninitialized padding included; a record without them (an older socket-patch or a Bun re-save) is rebuilt the way Bun writes one, and refused for a prerelease/build version. A lock the hosted rewrite had to normalize is marked in the root package's resolution value bytes (which no Bun reader reads): a binary format 1 lock it promoted to format 2 is demoted back to its exact format-1 bytes (verified by promoting it again, otherwise refused), and a lock whose workspace dependency behaviors it normalized is refused with the `git checkout -- bun.lockb` remedy. * **vlt** — `vlt-lock.json`: slot [2] from the registry's `dist.integrity`, slot [3] per the lock's own convention (see the vlt hosted-mode contract); every hosted instance of the pin together. * **cargo** — `Cargo.lock` back on crates.io (source + the sparse index's checksum, `SOCKET_CRATES_INDEX`). When the lock already holds a crates.io block for the same `name@version` (a later dependent locked its own copy, #679), the Socket block merges into it instead: it is dropped (with a v1 lock's `[metadata]` line), and dependents' references take the spelling cargo writes, so no duplicate block is left that cargo cannot parse (#863), and no index lookup is needed; every `Cargo.toml` declaration loses its `registry = "socket-patch-"` pin (the shorthand the rewriter produced collapses back); every `[registries.socket-patch-]` block no manifest or lock still references leaves the project cargo config — including a superseded patch generation's block an earlier re-pin left behind (#864). A declaration it cannot unpin refuses. * **golang** — the hosted `replace` and the socket module's go.sum lines go; the upstream module's two go.sum lines come back, hashed from the module proxy (`SOCKET_GOPROXY`, else `GOPROXY` / `GONOPROXY` / `GOPRIVATE` as go reads them) and cross-checked against the checksum database (`SOCKET_GOSUMDB_URL`, else `sum.golang.org` unless `GOSUMDB=off` / `GONOSUMDB` / `GOPRIVATE` say go would not ask it). A `replace` the user had before the hosted run is not recorded anywhere, so the restore lands on the plain upstream module. @@ -1338,7 +1338,7 @@ Every `--json` invocation emits a single JSON object that follows the **unified | `manifest_write_failed` | rollback `warnings[]` | rollback (v5.0): the post-rollback manifest update could not be written; no entries were removed (`manifest.removedEntries: []`) and the run exits `partial_failure` 1. | | `npm_allow_remote_left` / `pnpm_trust_lockfile_left` | rollback/remove `warnings[]`; vendor advisory event (takeover) | upstream restore (v5.0): no npm-family lock entry is hosted any more, but the project `.npmrc` keeps a top-level `allow-remote=all` (resp. `pnpm-workspace.yaml` keeps `trustLockfile: true`) in a file that is not exactly what hosted mode creates; the file is left untouched (v5 records no provenance), remove the line if nothing else needs it. A file that is exactly hosted mode's own is deleted silently. | | `maven_trusted_checksums_left` / `nuget_default_config_left` / `upstream_uv_override_removed` | rollback/remove `warnings[]`; vendor advisory event (takeover) | upstream restore (v5.0): `.mvn` config keeps the trusted-checksums resolver lines because it holds more than hosted mode writes; `nuget.config` now holds only the nuget.org source (delete it if hosted mode created it); a transitive `override-dependencies` entry hosted mode added to `pyproject.toml` (the one under its `# socket-patch hosted` comment) was removed; a user-authored override is never removed and never reported. | -| `upstream_registry_fallback` | rollback/remove `warnings[]`; vendor advisory event (takeover) | upstream restore: a yarn berry, pnpm or vlt entry is restored from the version document of the registry the project resolves it against (`.yarnrc.yml` `npmRegistryServer`, the pnpm lock's sibling `.npmrc` `registry` / `@scope:registry` or pnpm-workspace.yaml `registry` / `registries`, vlt's node registry); that registry could not be read (e.g. it needs credentials), so the default registry's document was used and the restored tarball URL may not be the mirror's. | +| `upstream_registry_fallback` | rollback/remove `warnings[]`; vendor advisory event (takeover) | upstream restore: a yarn berry, pnpm or vlt entry is restored from the version document of the registry the project resolves it against (yarn berry: `npmScopes` / `YARN_NPM_REGISTRY_SERVER` / `npmRegistryServer` from the `.yarnrc.yml` chain, as above, the pnpm lock's sibling `.npmrc` `registry` / `@scope:registry` or pnpm-workspace.yaml `registry` / `registries`, vlt's node registry); that registry could not be read (e.g. it needs credentials) or, for yarn berry, could not be determined (an rc value referencing an environment variable that is set, or an unset one with no default; a flow-style `npmScopes`), so the default registry's document was used and the restored tarball URL may not be the mirror's. | | `upstream_gem_stale_cache` | rollback/remove `warnings[]`; vendor advisory event (takeover) | upstream restore (#1260): a restored gem's `-.gem` is still in Bundler's cache dir (`cache_path`, default `vendor/cache`, resolved as for the Gem stale-install guard) and its sha256 is not the upstream one (the restored `CHECKSUMS` entry, else the rubygems.org compact index), or it could not be checked (`--offline`, a registry error). Bundler installs from that dir first, so a `bundle cache` taken while the hosted pin was live makes every later install fail on the upstream checksum (exit 37) or, on bundler < 2.6 frozen installs, keep installing the patched bytes. The detail names the file. Remedy: delete it, then run `bundle cache` to cache the upstream gem in its place (or `bundle install` if the project does not commit its cache; with the cache dir committed, a frozen install reads only the cache). Read-only: the restore never deletes it. Not raised for an archive whose sha256 matches upstream. | | `upstream_pnpm_tarball_setting_guessed` | rollback/remove `warnings[]`; vendor advisory event (takeover) | upstream restore (#902): nothing showed which pnpm wrote a hosted `pnpm-lock.yaml` (no unpinned registry entry that shows the setting, no `node_modules/.modules.yaml` install record, no package.json `packageManager` pin; for a Rush lock, no rush.json `pnpmVersion`), so its entries were restored with or without `tarball:` by pnpm 10's reading of `lockfileIncludeTarballUrl` (pnpm-workspace.yaml, else `.npmrc` `lockfile-include-tarball-url`), and pnpm 9 (which reads only `.npmrc`) or pnpm >= 11 (which reads only pnpm-workspace.yaml) would have read it the other way. The detail names the lock, the setting followed, the pnpm that disagrees and the entries. Remedy: pin the pnpm (package.json `packageManager`, or reinstall so the install record names it; rush.json `pnpmVersion` for Rush), or give the two files the same value so every pnpm reads it alike; a rollback or remove can then be redone by restoring the lock from version control and re-running. Not raised when evidence decided, when both files read the same on every pnpm, or when the tarball is one pnpm records regardless. | | `yarn_berry_node_gyp_unresolved` | rollback/remove `warnings[]`; vendor advisory event (takeover) | upstream restore (#737): yarn's npm resolver gives the restored registry entry an implicit `node-gyp: "npm:latest"` dependency (the version document runs `node-gyp` in a script without declaring it), which the hosted pin dropped because yarn never gives a tarball-locator entry one, and the lock no longer has a `node-gyp@npm:latest` entry (the pin dropped the subtree only node-gyp reached). The entry is restored without it; a plain `yarn install --immutable` accepts that, a hardened or `--refresh-lockfile` install reports the lock modified. Remedy: run `yarn install` once. Not raised while another entry keeps `node-gyp@npm:latest` in the lock: the dependency is then restored too. | diff --git a/crates/socket-patch-cli/tests/in_process_redirect.rs b/crates/socket-patch-cli/tests/in_process_redirect.rs index ac24f4e2d..26c229f38 100644 --- a/crates/socket-patch-cli/tests/in_process_redirect.rs +++ b/crates/socket-patch-cli/tests/in_process_redirect.rs @@ -1968,6 +1968,10 @@ fn hosted_unwind_json( "SOCKET_NPM_REGISTRY", format!("{}/npm-registry", registry.uri()), ) + // The yarn berry restore reads these like yarn does (#1017): an + // ambient value must not steer the fixtures' registry. + .env_remove("YARN_NPM_REGISTRY_SERVER") + .env_remove("YARN_RC_FILENAME") .output() .unwrap_or_else(|e| panic!("run socket-patch {}: {e}", command[0])); let env_json: serde_json::Value = serde_json::from_slice(&out.stdout).unwrap_or_else(|e| { @@ -6252,6 +6256,192 @@ async fn hosted_json_reports_an_unpinned_row_for_a_granted_patch_nothing_pins() assert_eq!(rows[0]["errorCode"], "redirect_unconfirmed", "{env:#}"); } +// ── #1017: the berry restore reads the registry from every yarn source ────── + +/// Where a #1017 cell configures the project's mirror. +#[derive(Debug, Clone, Copy)] +enum MirrorSource { + /// `YARN_NPM_REGISTRY_SERVER`, no key in any rc. + Env, + /// A parent directory's `.yarnrc.yml`. + ParentRc, + /// The home directory's `.yarnrc.yml` (the project is not under it). + HomeRc, + /// The project rc's `npmRegistryServer: "${UNSET:-}"`. + Interpolated, +} + +/// #1017: #908's mirror restore, with the mirror set the other ways yarn +/// reads it. Each cell pins the project hosted, rolls it back, and needs +/// the mirror's `::__archiveUrl=` binding back byte-exactly — reading the +/// default registry instead silently writes a bare `name@npm:` locator +/// whose conventional tarball the mirror 404s. +#[tokio::test] +#[serial] +async fn yarn_berry_rollback_reads_the_registry_from_every_yarn_source() { + for source in [ + MirrorSource::Env, + MirrorSource::ParentRc, + MirrorSource::HomeRc, + MirrorSource::Interpolated, + ] { + let server = MockServer::start().await; + mock_discovery(&server).await; + let hosted_url = HOSTED_URL.replace("http://patch.test", &server.uri()); + mock_reference_with_berry_url(&server, &hosted_url).await; + mock_view(&server).await; + let integrity = vlt_hosted_common::sha512_sri(&upstream_tarball()); + mock_npm_registry_advertising( + &server, + &integrity, + &format!( + "{}/npm-registry/{NAME}/-/{NAME}-{VERSION}.tgz", + server.uri() + ), + ) + .await; + let mirror = format!("{}/mirror", server.uri()); + let advertised = format!("{}/cdn/files/{NAME}-{VERSION}.tgz", server.uri()); + Mock::given(method("GET")) + .and(path(format!("/mirror/{NAME}/{VERSION}"))) + .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ + "name": NAME, + "version": VERSION, + "dist": { "tarball": advertised, "integrity": integrity }, + }))) + .mount(&server) + .await; + let binding = |t: &str| { + t.replace( + &format!("resolution: \"{NAME}@npm:{VERSION}\""), + &format!( + "resolution: \"{NAME}@npm:{VERSION}::__archiveUrl={}\"", + socket_patch_core::utils::uri::encode_uri_component(&advertised) + ), + ) + }; + + let tmp = tempfile::tempdir().unwrap(); + let project = tmp.path().join("work").join("proj"); + let home = tmp.path().join("home"); + std::fs::create_dir_all(&project).unwrap(); + std::fs::create_dir_all(&home).unwrap(); + write_berry_project_spelled(&project, binding); + let mut project_rc = "nodeLinker: node-modules\n".to_string(); + let mut envs: Vec<(&str, String)> = vec![("HOME", home.display().to_string())]; + match source { + MirrorSource::Env => envs.push(("YARN_NPM_REGISTRY_SERVER", mirror.clone())), + MirrorSource::ParentRc => std::fs::write( + tmp.path().join("work").join(".yarnrc.yml"), + format!("npmRegistryServer: \"{mirror}\"\n"), + ) + .unwrap(), + MirrorSource::HomeRc => std::fs::write( + home.join(".yarnrc.yml"), + format!("npmRegistryServer: \"{mirror}\"\n"), + ) + .unwrap(), + MirrorSource::Interpolated => project_rc.push_str(&format!( + "npmRegistryServer: \"${{SOCKET_PATCH_TEST_UNSET_REG:-{mirror}}}\"\n" + )), + } + std::fs::write(project.join(".yarnrc.yml"), &project_rc).unwrap(); + let lock_path = project.join("yarn.lock"); + let pristine = std::fs::read_to_string(&lock_path).unwrap(); + + let env = run_redirect_subprocess_with( + &project, + &server.uri(), + &["--patch-server-url", &server.uri()], + ); + assert_eq!(env["redirect"]["redirected"], 1, "{source:?}: {env:#}"); + + let out = scrubbed_cli() + .args([ + "rollback", + "--json", + "--yes", + "--patch-server-url", + &server.uri(), + "--cwd", + project.to_str().unwrap(), + ]) + .env( + "SOCKET_NPM_REGISTRY", + format!("{}/npm-registry", server.uri()), + ) + .env_remove("YARN_NPM_REGISTRY_SERVER") + .env_remove("YARN_RC_FILENAME") + .env_remove("SOCKET_PATCH_TEST_UNSET_REG") + .envs(envs.iter().map(|(k, v)| (*k, v.as_str()))) + .output() + .unwrap(); + let env: serde_json::Value = serde_json::from_slice(&out.stdout).unwrap_or_else(|e| { + panic!( + "{source:?}: rollback stdout must be JSON: {e}\n{}\n{}", + String::from_utf8_lossy(&out.stdout), + String::from_utf8_lossy(&out.stderr) + ) + }); + assert_eq!(out.status.code(), Some(0), "{source:?}: rollback: {env:#}"); + let restored = std::fs::read_to_string(&lock_path).unwrap(); + let checksum = berry_checksum_of(&restored); + assert_eq!( + restored, + pristine.replace( + &format!("10c0/{}", "3".repeat(128)), + &format!("10c0/{checksum}") + ), + "{source:?}: rollback keeps the mirror's __archiveUrl binding" + ); + } +} + +/// #1017: a registry yarn itself cannot resolve (an rc reference to an +/// unset variable with no default) is not silently replaced by the default +/// registry: the restore says so with `upstream_registry_fallback`. +#[tokio::test] +#[serial] +async fn yarn_berry_rollback_warns_when_the_registry_cannot_be_known() { + let server = MockServer::start().await; + mock_discovery(&server).await; + let hosted_url = HOSTED_URL.replace("http://patch.test", &server.uri()); + mock_reference_with_berry_url(&server, &hosted_url).await; + mock_view(&server).await; + let conventional = format!( + "{}/npm-registry/{NAME}/-/{NAME}-{VERSION}.tgz", + server.uri() + ); + mock_npm_registry_advertising( + &server, + &vlt_hosted_common::sha512_sri(&upstream_tarball()), + &conventional, + ) + .await; + let tmp = tempfile::tempdir().unwrap(); + write_berry_project(tmp.path()); + let env = run_redirect_subprocess_with( + tmp.path(), + &server.uri(), + &["--patch-server-url", &server.uri()], + ); + assert_eq!(env["redirect"]["redirected"], 1, "{env:#}"); + std::fs::write( + tmp.path().join(".yarnrc.yml"), + "nodeLinker: node-modules\nnpmRegistryServer: \"${SOCKET_PATCH_TEST_UNSET_REG}\"\n", + ) + .unwrap(); + let (code, env) = rollback_json_with_origin(tmp.path(), &server, &server.uri()); + assert_eq!(code, Some(0), "rollback: {env:#}"); + let codes: Vec<&str> = env["warnings"] + .as_array() + .into_iter() + .flatten() + .filter_map(|w| w["code"].as_str()) + .collect(); + assert!(codes.contains(&"upstream_registry_fallback"), "{env:#}"); +} + // ── #1131: the restored `npm:` entry takes the registry's `bin:` back ─────── /// #1131: a hosted berry pin writes the served tarball's own `bin:` diff --git a/crates/socket-patch-core/src/patch/redirect/upstream/npm.rs b/crates/socket-patch-core/src/patch/redirect/upstream/npm.rs index 7dcda2d82..0dc36b8ed 100644 --- a/crates/socket-patch-core/src/patch/redirect/upstream/npm.rs +++ b/crates/socket-patch-core/src/patch/redirect/upstream/npm.rs @@ -480,21 +480,226 @@ fn berry_registry_locator( use crate::formats::pnpm::workspace::yaml_top_level_value; use crate::formats::text::strip_bom; -/// The registry a berry restore reads `name`'s version document from: -/// `.yarnrc.yml`'s `npmRegistryServer`. A scoped package may resolve -/// against an `npmScopes` registry instead, so with such a block present -/// it keeps the default registry's document. -fn berry_lookup_registry(yarnrc: Option<&str>, name: &str) -> Option { - let text = yarnrc?; - // `top_level_key` skips the first line's BOM (`formats::text`). - let has_scopes = text +/// The registry yarn berry resolves `name` against (`Ok(None)`: yarn's +/// default registry), read the way yarn merges its settings (#1017): +/// +/// - a scoped `@scope/name` takes `npmScopes..npmRegistryServer` +/// from the highest-precedence rc file that sets it; +/// - otherwise (and for a scope without its own server) the +/// `YARN_NPM_REGISTRY_SERVER` environment variable, then the +/// highest-precedence rc file's top-level `npmRegistryServer`. +/// +/// `rcs` are the `.yarnrc.yml` texts yarn reads, highest precedence first +/// ([`BerryRegistrySettings::read`]): the lock directory's, each parent +/// directory's up to the filesystem root, then the home directory's. A +/// value's `${VAR}` / `${VAR:-default}` / `${VAR-default}` references are +/// expanded through `var` as yarn expands them. `Err` names why the +/// registry cannot be known: a reference to an unset variable with no +/// default (yarn refuses to run), or an `npmScopes` this reader cannot +/// follow (a flow mapping). +fn berry_lookup_registry( + rcs: &[String], + env_registry: Option<&str>, + name: &str, + var: &dyn Fn(&str) -> Option, +) -> Result, String> { + if let Some((scope, _)) = name.strip_prefix('@').and_then(|rest| rest.split_once('/')) { + let keys = [scope.to_string(), format!("@{scope}")]; + for rc in rcs { + for key in &keys { + if let Some(value) = + yaml_path_value(rc, &["npmScopes", key.as_str(), "npmRegistryServer"])? + { + return yarn_expand_env(&value, var).map(Some); + } + } + } + } + if let Some(env) = env_registry.map(str::trim).filter(|v| !v.is_empty()) { + return Ok(Some(env.to_string())); + } + for rc in rcs { + if let Some(value) = yaml_path_value(rc, &["npmRegistryServer"])? { + return yarn_expand_env(&value, var).map(Some); + } + } + Ok(None) +} + +/// The scalar at `path` in a YAML settings file of block mappings (the +/// last occurrence of each key wins, quotes removed). `Ok(None)` when a +/// key on the path is absent or null; `Err` when a mapping on the path is +/// written in flow style, which this reader does not follow. +fn yaml_path_value(text: &str, path: &[&str]) -> Result, String> { + // `top_level_key` skips the first line's BOM (one, as yarn's parser). + let lines: Vec = text .lines() - .filter_map(crate::formats::pnpm::workspace::top_level_key) - .any(|(key, _)| key == "npmScopes"); - if has_scopes && name.starts_with('@') { - return None; + .map(|l| l.strip_suffix('\r').unwrap_or(l).to_string()) + .collect(); + yaml_path_in(&lines, path) +} + +fn yaml_path_in(lines: &[String], path: &[&str]) -> Result, String> { + use crate::formats::pnpm::workspace::top_level_key; + let Some((key, rest)) = path.split_first() else { + return Ok(None); + }; + let Some((at, value)) = lines.iter().enumerate().rev().find_map(|(i, line)| { + top_level_key(line) + .filter(|(k, _)| k == key) + .map(|(_, value)| (i, value.to_string())) + }) else { + return Ok(None); + }; + if matches!(value.as_str(), "~" | "null") { + return Ok(None); + } + if rest.is_empty() { + let value = value.trim_matches(['"', '\'']).to_string(); + return Ok((!value.is_empty()).then_some(value)); + } + if !value.is_empty() { + return Err(format!( + "`{}` is not written as a block mapping", + path.first().copied().unwrap_or_default() + )); + } + let end = lines[at + 1..] + .iter() + .position(|l| !l.is_empty() && !l.starts_with([' ', '\t', '#'])) + .map_or(lines.len(), |p| at + 1 + p); + let children = &lines[at + 1..end]; + let Some(indent) = children.iter().find_map(|line| { + let content = line.trim_start_matches(' '); + (!content.trim().is_empty() && !content.starts_with('#')) + .then(|| line.len() - content.len()) + }) else { + return Ok(None); + }; + let dedented: Vec = children + .iter() + .map(|line| match line.get(indent..) { + Some(rest) if line[..indent].trim().is_empty() => rest.to_string(), + _ => String::new(), + }) + .collect(); + yaml_path_in(&dedented, rest) +} + +/// A `.yarnrc.yml` registry value with its environment references read as +/// yarn reads them: `${NAME}`, `${NAME-fallback}` (the fallback when `NAME` +/// is unset) and `${NAME:-fallback}` (also when it is empty). A reference +/// to an unset variable with no fallback is the error yarn stops on. +/// +/// A reference to a variable that IS set is never expanded: the rc file +/// (possibly a checked-in, lower-trust one) would choose which of this +/// process's variables — a token, say — lands in a URL the restore +/// requests and may print. That is `Err` too, so the restore falls back to +/// the default registry with `upstream_registry_fallback`, as for the Bun +/// and pnpm settings, which never expand an arbitrary variable either. +fn yarn_expand_env(value: &str, var: &dyn Fn(&str) -> Option) -> Result { + let mut out = String::with_capacity(value.len()); + let mut rest = value; + while let Some(at) = rest.find("${") { + out.push_str(&rest[..at]); + let body = &rest[at + 2..]; + let name_len = body + .find(|c: char| !(c.is_ascii_alphanumeric() || c == '_')) + .unwrap_or(body.len()); + let name = &body[..name_len]; + let mut tail = &body[name_len..]; + let colon = tail.starts_with(':'); + if colon { + tail = &tail[1..]; + } + let fallback = match tail.strip_prefix('-') { + Some(after) => match after.find('}') { + Some(close) => { + tail = &after[close..]; + Some(&after[..close]) + } + None => None, + }, + None => None, + }; + let Some(after) = tail.strip_prefix('}').filter(|_| !name.is_empty()) else { + // Not a reference yarn recognizes: kept literally. + out.push_str("${"); + rest = body; + continue; + }; + let set = var(name); + let expanded = match (set, fallback) { + (Some(v), _) if !v.is_empty() => { + return Err(format!( + "it references the environment variable {name}, which socket-patch does \ + not expand into a registry URL" + )) + } + (Some(v), _) if !colon => v, + (_, Some(fallback)) => fallback.to_string(), + (_, None) => return Err(format!("the environment variable {name} is not set")), + }; + out.push_str(&expanded); + rest = after; + } + out.push_str(rest); + Ok(out) +} + +/// The yarn berry settings that decide which registry a package resolves +/// against: the `.yarnrc.yml` texts yarn reads for a lock, highest +/// precedence first, and `YARN_NPM_REGISTRY_SERVER`. +struct BerryRegistrySettings { + rcs: Vec, + env_registry: Option, +} + +impl BerryRegistrySettings { + /// Yarn reads the rc file (`YARN_RC_FILENAME`, default `.yarnrc.yml`) + /// of the project directory and of every parent directory up to the + /// filesystem root (a closer one wins), then the home directory's, + /// below them all. + async fn read(view: &View<'_>, dir_prefix: &str) -> Self { + let rc_name = std::env::var("YARN_RC_FILENAME") + .ok() + .filter(|v| !v.trim().is_empty()) + .unwrap_or_else(|| ".yarnrc.yml".to_string()); + let root = view.root().join(dir_prefix); + let root = tokio::fs::canonicalize(&root).await.unwrap_or(root); + let mut read: Vec = + root.ancestors().map(|d| d.join(&rc_name)).collect(); + let home = std::env::var_os("HOME") + .or_else(|| std::env::var_os("USERPROFILE")) + .filter(|h| !h.is_empty()) + .map(std::path::PathBuf::from); + if let Some(home) = home { + let home = tokio::fs::canonicalize(&home).await.unwrap_or(home); + let rc = home.join(&rc_name); + if !read.contains(&rc) { + read.push(rc); + } + } + let mut rcs = Vec::new(); + for path in read { + if let Ok(text) = crate::utils::fs::read_regular_to_string(&path).await { + rcs.push(text); + } + } + BerryRegistrySettings { + rcs, + env_registry: std::env::var("YARN_NPM_REGISTRY_SERVER").ok(), + } + } + + fn registry(&self, name: &str) -> Result, String> { + berry_lookup_registry( + &self.rcs, + self.env_registry.as_deref(), + name, + &|key: &str| std::env::var(key).ok(), + ) } - yaml_top_level_value(text, "npmRegistryServer") } async fn restore_berry( @@ -654,10 +859,30 @@ async fn restore_berry( .iter() .map(|h| (h.uuid.clone(), h.name.clone(), h.version.clone())) .collect(); - let project_registry = yarnrc - .as_deref() - .and_then(|text| yaml_top_level_value(text, "npmRegistryServer")); - let registry = |name: &str| berry_lookup_registry(yarnrc.as_deref(), name); + // The registry yarn resolves each package against, read from every + // settings source yarn merges (#1017); one it cannot be known for is + // restored from the default registry's document, with a warning. + let settings = BerryRegistrySettings::read(view, &dir_prefix).await; + let mut registries: BTreeMap> = BTreeMap::new(); + for hit in &hits { + if registries.contains_key(&hit.name) { + continue; + } + let registry = settings.registry(&hit.name).unwrap_or_else(|why| { + result.warnings.push(( + "upstream_registry_fallback", + format!( + "{}@{}: the registry yarn resolves it against could not be determined \ + ({why}), so the entry was restored from the default registry's version \ + document; check its tarball URL against the project's registry", + hit.name, hit.version + ), + )); + None + }); + registries.insert(hit.name.clone(), registry); + } + let registry = |name: &str| registries.get(name).cloned().flatten(); let dists = fetch_dists_on(&wanted, registry, ctx, result).await; let mut changed = false; let mut moved: Vec = Vec::new(); @@ -696,8 +921,12 @@ async fn restore_berry( let Some(dist) = dists.get(&(name.clone(), version.clone())).map(|d| &d.dist) else { continue; }; - let locator = - berry_registry_locator(project_registry.as_deref(), &name, &version, &dist.tarball); + let locator = berry_registry_locator( + registries.get(&name).and_then(Option::as_deref), + &name, + &version, + &dist.tarball, + ); let resolution = format!(" resolution: \"{locator}\""); let mut lines = stanza_lines(&blocks[idx]); if let Some(pinned) = with_body_field(&lines, "resolution", &resolution) { @@ -2360,7 +2589,7 @@ mod tests { bun_registry_slot, bun_tarball_url, expand_url_env, modules_yaml_pnpm_major, non_default_registry, package_json_pnpm_major, pnpm_include_tarball, pnpm_lookup_registry, pnpm_tarball_guess_warning, registry_derives_tarball, rush_json_pnpm_major, rush_lock_root, - split_userinfo, yaml_top_level_value, PnpmTarballGuess, ProjectDist, + split_userinfo, yaml_top_level_value, yarn_expand_env, PnpmTarballGuess, ProjectDist, }; use crate::patch::redirect::upstream::client::NpmDist; @@ -3409,17 +3638,17 @@ mod tests { /// content, so the first key is not `npmScopes`. #[test] fn berry_scopes_probe_reads_past_one_bom_only() { + let lookup = |rc: &str, name: &str| { + berry_lookup_registry(&[rc.to_string()], None, name, &|_: &str| None).unwrap() + }; let rc = "npmScopes:\n s:\n npmRegistryServer: https://s.example\n\ npmRegistryServer: https://m.example/\n"; for bom in ["", "\u{feff}"] { let rc = format!("{bom}{rc}"); - assert_eq!(berry_lookup_registry(Some(&rc), "@s/a"), None); + assert_eq!(lookup(&rc, "@s/a").as_deref(), Some("https://s.example")); } let rc = format!("\u{feff}\u{feff}{rc}"); - assert_eq!( - berry_lookup_registry(Some(&rc), "@s/a").as_deref(), - Some("https://m.example/") - ); + assert_eq!(lookup(&rc, "@s/a").as_deref(), Some("https://m.example/")); let rc = "npmRegistryServer: https://m.example/\n"; for bom in ["", "\u{feff}"] { assert_eq!( @@ -3433,24 +3662,135 @@ mod tests { ); } + /// #1017: yarn resolves a package's registry from every settings + /// source it merges, not only the project rc's top-level key. #[test] - fn berry_reads_the_project_registry_except_for_npm_scopes() { + fn berry_reads_the_registry_from_every_yarn_settings_source() { + let none = |_: &str| None::; + let lookup = |rcs: &[&str], env: Option<&str>, name: &str| { + let rcs: Vec = rcs.iter().map(|s| s.to_string()).collect(); + berry_lookup_registry(&rcs, env, name, &none) + }; let rc = "npmRegistryServer: \"https://m.example/npm/\"\n"; assert_eq!( - berry_lookup_registry(Some(rc), "a").as_deref(), + lookup(&[rc], None, "a").unwrap().as_deref(), Some("https://m.example/npm/") ); assert_eq!( - berry_lookup_registry(Some(rc), "@s/a").as_deref(), + lookup(&[rc], None, "@s/a").unwrap().as_deref(), Some("https://m.example/npm/") ); - let scoped = format!("{rc}npmScopes:\n s:\n npmRegistryServer: https://s.example\n"); + assert_eq!(lookup(&[], None, "a").unwrap(), None); + + // npmScopes: the scope's own server, else the top-level one. + let scoped = format!( + "{rc}npmScopes:\n s:\n npmAlwaysAuth: true\n npmRegistryServer: \ + \"https://s.example\"\n t:\n npmAlwaysAuth: true\n" + ); assert_eq!( - berry_lookup_registry(Some(&scoped), "a").as_deref(), + lookup(&[&scoped], None, "a").unwrap().as_deref(), Some("https://m.example/npm/") ); - assert_eq!(berry_lookup_registry(Some(&scoped), "@s/a"), None); - assert_eq!(berry_lookup_registry(None, "a"), None); + assert_eq!( + lookup(&[&scoped], None, "@s/a").unwrap().as_deref(), + Some("https://s.example") + ); + assert_eq!( + lookup(&[&scoped], None, "@t/a").unwrap().as_deref(), + Some("https://m.example/npm/") + ); + assert_eq!( + lookup(&[&scoped], None, "@u/a").unwrap().as_deref(), + Some("https://m.example/npm/") + ); + // A scope beats the env registry; the env registry beats every rc. + let env = Some("https://env.example"); + assert_eq!( + lookup(&[&scoped], env, "@s/a").unwrap().as_deref(), + Some("https://s.example") + ); + assert_eq!( + lookup(&[&scoped], env, "a").unwrap().as_deref(), + Some("https://env.example") + ); + assert_eq!( + lookup(&[], env, "a").unwrap().as_deref(), + Some("https://env.example") + ); + assert_eq!( + lookup(&[rc], Some(" "), "a").unwrap().as_deref(), + Some("https://m.example/npm/") + ); + + // Layers: the closer rc wins per key; a farther one (a parent + // directory's, the home one) fills in what the closer one lacks. + let parent = "npmRegistryServer: https://parent.example\n"; + let home = "npmScopes:\n s:\n npmRegistryServer: https://home-s.example\n"; + let project = "nodeLinker: node-modules\n"; + assert_eq!( + lookup(&[project, parent, home], None, "a") + .unwrap() + .as_deref(), + Some("https://parent.example") + ); + assert_eq!( + lookup(&[project, parent, home], None, "@s/a") + .unwrap() + .as_deref(), + Some("https://home-s.example") + ); + assert_eq!( + lookup(&[rc, parent], None, "a").unwrap().as_deref(), + Some("https://m.example/npm/") + ); + + // A flow-style npmScopes is not guessed at for a scoped name. + let flow = "npmScopes: {s: {npmRegistryServer: https://s.example}}\n"; + assert!(lookup(&[flow], None, "@s/a").is_err()); + assert_eq!( + lookup(&[flow, parent], None, "a").unwrap().as_deref(), + Some("https://parent.example") + ); + } + + #[test] + fn berry_registry_values_expand_env_references_like_yarn() { + let var = |name: &str| match name { + "REG" => Some("https://reg.example".to_string()), + "EMPTY" => Some(String::new()), + _ => None, + }; + for (value, want) in [ + // A set variable is never expanded (the rc file must not pick + // which of the process's variables lands in a requested URL). + ("${REG}", Err(())), + ("${REG:-https://d.example}", Err(())), + ("${REG-https://d.example}", Err(())), + ("${UNSET:-https://d.example}", Ok("https://d.example")), + ("${UNSET-https://d.example}", Ok("https://d.example")), + ("${EMPTY:-https://d.example}", Ok("https://d.example")), + ("${EMPTY-https://d.example}", Ok("")), + ("${EMPTY}", Ok("")), + ("https://h/${REG}/x", Err(())), + ("https://h/${UNSET:-m}/x", Ok("https://h/m/x")), + ("plain $ {REG} ${", Ok("plain $ {REG} ${")), + ("${UNSET}", Err(())), + ] { + assert_eq!( + yarn_expand_env(value, &var).map_err(|_| ()), + want.map(str::to_string), + "{value}" + ); + } + let rc = "npmRegistryServer: \"${REG:-http://127.0.0.1:8792}\"\n"; + assert_eq!( + berry_lookup_registry(&[rc.to_string()], None, "a", &|_: &str| None) + .unwrap() + .as_deref(), + Some("http://127.0.0.1:8792") + ); + let rc = "npmRegistryServer: \"${REG}\"\n"; + assert!(berry_lookup_registry(&[rc.to_string()], None, "a", &|_: &str| None).is_err()); } #[test]