diff --git a/crates/socket-patch-cli/CLI_CONTRACT.md b/crates/socket-patch-cli/CLI_CONTRACT.md index edc6d674d..b9921c653 100644 --- a/crates/socket-patch-cli/CLI_CONTRACT.md +++ b/crates/socket-patch-cli/CLI_CONTRACT.md @@ -1459,6 +1459,7 @@ Every `--json` invocation emits a single JSON object that follows the **unified | `redirect_vlt_lock_unsupported` | `redirect.warnings[]` (warning) | scan/get `--mode hosted` (vlt): `vlt-lock.json` has a `lockfileVersion` other than absent, `0` or `1` (decided on the raw JSON token), is not a JSON object, starts with a UTF-8 BOM, or its `nodes` section is not vlt's one-node-per-line layout. Nothing rewritten; also refuses a vendored → hosted takeover of a `flavor: "vlt"` entry before its revert (`redirect.skipped[].reason`). Exit 0. | | `redirect_takeover_kept_vendored` | `redirect.warnings[]` (warning) | scan/get `--mode hosted`: a vendored → hosted takeover the hosted rewrite would not pin was retracted (see **Staged takeover**): the package keeps its vendored wiring, ledger entry and artifact byte-identical and stays patched. The detail names the cause code, which is also the purl's `redirect.skipped[].reason`. Exit 0. Replaces v5.0-pre `redirect_takeover_unpatched`, which reported a package left unpatched in both modes and is no longer emitted. | | `redirect_takeover_not_pinned` | `redirect.skipped[].reason` | scan/get `--mode hosted`: the skip reason of a retracted takeover when no rewriter warning names the cause. | +| `redirect_hatch_lock_regenerated` | `redirect.warnings[]` (warning) | scan/get `--mode hosted` (pypi / Hatch 1.17+): the project's Hatch environments are locked (`lock-envs = true`, an environment's `locked = true` or `lock-filename`), so its `pylock*.toml` is derived from pyproject and Hatch regenerates it on the next sync. The package is wired in the Hatch declarations as well as in the lock (rewriting only the lock was discarded by Hatch, #479); run `hatch dep lock` to refresh the lock (`hatch dep lock --check` reports it stale until then). Vendored routes such a project to the Hatch lane and its pip-installer refusal. | | `redirect_requirements_direct_reference` | `redirect.warnings[]` (warning) | scan/get `--mode hosted` (pypi / requirements.txt): the root `requirements.txt` installs the patched release from a PEP 508 direct reference the user wrote (`name @ ` whose archive names that release: a `files.pythonhosted.org` file, a private mirror, an internal fork build, a `file://` path), not from socket-patch's own hosted artifact. It is the user's own source choice, so the line is left byte-for-byte and the package is not redirected (#542). Before v5.0 the line was swapped for the hosted build, and rollback then wrote a plain `name==version` index pin. | | `redirect_requirements_takeover_unreachable` | `redirect.warnings[]` (warning) | scan/get `--mode hosted` (pypi / requirements.txt): a vendored → hosted takeover of a package that vendored mode wired through a pin in a `-r` include, or through a `(transitive)` line it appended to the root `requirements.txt`. Hosted mode only rewrites an existing pin in the root `requirements.txt`, so the staged takeover is retracted (wet and `--dry-run`): the vendored wiring, ledger entry and wheel stay byte-identical, the purl is skipped with this code as `redirect.skipped[].reason`, and nothing is redirected for it. Exit 0. The detail names the remedy and its reach: run `socket-patch vendor --revert` (it reverts EVERY vendored package in the project, not just this one), move the pin from the include into the root `requirements.txt` and delete it from the include (or, for a `(transitive)` line, add an exact `==` pin to the root file), then re-run `scan --mode hosted`. | | `redirect_uv_takeover_version_unreachable` | `redirect.warnings[]` (warning) | scan/get `--mode hosted` (pypi / uv): a vendored → hosted takeover of a package whose recorded pre-vendor `uv.lock` entry is at another version than the patch (vendored uv pins the lock entry down to the patch's version, for example when the lock resolved a newer release). Reverting would bring the lock's own version back, and hosted mode only pins the version the lock resolves, so the staged takeover is retracted (wet and `--dry-run`): the vendored wiring, ledger entry and wheel stay byte-identical, the purl is skipped with this code as `redirect.skipped[].reason`, and nothing is redirected for it. Exit 0. The detail names the remedy and its reach: run `socket-patch vendor --revert` (it reverts EVERY vendored package in the project, not just this one), make the project resolve the patch's version (for example an exact `==` requirement) and re-lock, then re-run `scan --mode hosted`. | diff --git a/crates/socket-patch-cli/tests/mode_migration_pypi.rs b/crates/socket-patch-cli/tests/mode_migration_pypi.rs index 1b148bfff..66423d6d6 100644 --- a/crates/socket-patch-cli/tests/mode_migration_pypi.rs +++ b/crates/socket-patch-cli/tests/mode_migration_pypi.rs @@ -2071,6 +2071,64 @@ async fn pipenv_hosted_to_vendored_names_the_unpatched_requirements() { ); } +// ── #479: a Hatch-derived pylock.toml ──────────────────────────────────── + +/// The pylock `hatch lock` writes for a locked environment (uv locker), +/// pinning six from PyPI. +const HATCH_PYLOCK: &str = "lock-version = \"1.0\"\ncreated-by = \"uv\"\nrequires-python = \">=3.9\"\n\n[[packages]]\nname = \"six\"\nversion = \"1.16.0\"\nindex = \"https://pypi.org/simple\"\nsdist = { url = \"https://files.pythonhosted.org/packages/71/39/six-1.16.0.tar.gz\", upload-time = 2021-05-05T14:18:18Z, size = 34041, hashes = { sha256 = \"SDIST_SHA\" } }\nwheels = [{ url = \"https://files.pythonhosted.org/packages/d9/5a/six-1.16.0-py2.py3-none-any.whl\", upload-time = 2021-05-05T14:18:17Z, size = 11053, hashes = { sha256 = \"WHEEL_SHA\" } }]\n"; + +const HATCH_LOCKED_PYPROJECT: &str = "[build-system]\nrequires = [\"hatchling\"]\nbuild-backend = \"hatchling.build\"\n\n[project]\nname = \"app\"\nversion = \"0.1.0\"\ndependencies = [\"six==1.16.0\"]\n\n[tool.hatch.envs.default]\nlocked = true\ninstaller = \"uv\"\n"; + +/// #479: Hatch 1.17+ derives `pylock.toml` from pyproject for a locked +/// environment and regenerates it on the next sync, so a scan that rewrote +/// only the lock (success, no warning) never patched a Hatch environment. +/// Hosted must wire pyproject too (with a warning to re-lock); vendored must route to the Hatch lane, +/// whose uv-installer refusal applies, and write nothing. +#[tokio::test] +async fn hatch_locked_env_pylock_wires_pyproject() { + let pylock = HATCH_PYLOCK + .replace("SDIST_SHA", SDIST_SHA) + .replace("WHEEL_SHA", WHEEL_SHA); + let server = MockServer::start().await; + let hosted_url = mount_hosted_api(&server, true).await; + + let (_tmp, root) = project(); + std::fs::write(root.join("pyproject.toml"), HATCH_LOCKED_PYPROJECT).unwrap(); + std::fs::write(root.join("pylock.toml"), &pylock).unwrap(); + let (code, env) = hosted_scan(&root, &server); + assert_eq!(code, 0, "hosted: {env:#}"); + assert_eq!(env["redirect"]["redirected"], 1, "{env:#}"); + let pyproject = std::fs::read_to_string(root.join("pyproject.toml")).unwrap(); + assert!( + pyproject.contains(&format!("six @ {hosted_url}")), + "pyproject is wired:\n{pyproject}" + ); + let lock = std::fs::read_to_string(root.join("pylock.toml")).unwrap(); + assert!( + lock.contains(&hosted_url), + "the lock stays consistent with pyproject until Hatch regenerates it:\n{lock}" + ); + assert!( + env.to_string().contains("redirect_hatch_lock_regenerated"), + "{env:#}" + ); + let (_tmp, root) = project(); + std::fs::write(root.join("pyproject.toml"), HATCH_LOCKED_PYPROJECT).unwrap(); + std::fs::write(root.join("pylock.toml"), &pylock).unwrap(); + stage_manifest(&root); + let (code, env) = run_cli(&root, &["vendor"], &[]); + assert_ne!(code, 0, "vendored: {env:#}"); + assert!(env.to_string().contains("pip installer"), "{env:#}"); + assert_eq!( + std::fs::read_to_string(root.join("pyproject.toml")).unwrap(), + HATCH_LOCKED_PYPROJECT + ); + assert_eq!( + std::fs::read_to_string(root.join("pylock.toml")).unwrap(), + pylock + ); +} + // ── #604: PEP 440-equivalent lock-only pins ────────────────────────────── /// #604: on a fresh checkout (no venv), `six==1.16` (or `==1.16.0.0`, diff --git a/crates/socket-patch-core/src/patch/redirect/mod.rs b/crates/socket-patch-core/src/patch/redirect/mod.rs index 06afda7d9..2faee8377 100644 --- a/crates/socket-patch-core/src/patch/redirect/mod.rs +++ b/crates/socket-patch-core/src/patch/redirect/mod.rs @@ -1024,14 +1024,25 @@ fn rewrite_hatch( .filter(|dep| dep.ecosystem == "pypi") .map(|dep| dep.patch_uuid.clone()), ); + // A pylock Hatch derives from pyproject (locked environments, #479) is + // not an install source of its own: Hatch regenerates it from the + // pyproject, so this lane wires pyproject too (the python-lock lane + // still rewrites the lock, keeping the two consistent until Hatch + // regenerates it from the wired pyproject). if files.keys().any(|file| { matches!( file.as_str(), "uv.lock" | "poetry.lock" | "pdm.lock" | "Pipfile.lock" - ) || crate::utils::python_lock::is_python_lock_name(file) + ) || (crate::utils::python_lock::is_python_lock_name(file) + && !crate::utils::hatch::is_hatch_lock(files, file)) }) { return; } + let hatch_locks: Vec<&str> = files + .keys() + .filter(|file| crate::utils::hatch::is_hatch_lock(files, file)) + .map(String::as_str) + .collect(); if files.contains_key("requirements.txt") { result .confirmed_hatch_uuids @@ -1061,6 +1072,20 @@ fn rewrite_hatch( match crate::utils::hatch::rewrite(¤t, &dep.name, &dep.version, &url) { Ok(edits) => { result.confirmed_hatch_uuids.insert(dep.patch_uuid.clone()); + if !edits.is_empty() && !hatch_locks.is_empty() { + result.warnings.push(RewriteWarning { + code: "redirect_hatch_lock_regenerated".into(), + detail: format!( + "{}@{}: wired in the Hatch configuration as well as in {}, which \ + Hatch derives from it (locked environments) and regenerates on \ + the next environment sync; run `hatch dep lock` to refresh it \ + now (`hatch dep lock --check` reports it stale until then)", + dep.name, + dep.version, + hatch_locks.join(", ") + ), + }); + } for (path, new) in edits { result.edits.push(FileEdit { path: path.clone(), @@ -24420,6 +24445,77 @@ mod hatch_tests { } } + /// The pylock `hatch lock` (Hatch 1.17+, `created-by = "uv"`) writes for + /// a locked environment, pinning urllib3 from PyPI. + const HATCH_PYLOCK: &str = "lock-version = \"1.0\"\ncreated-by = \"uv\"\nrequires-python = \">=3.9\"\n\n[[packages]]\nname = \"urllib3\"\nversion = \"1.26.18\"\nindex = \"https://pypi.org/simple\"\nwheels = [{ url = \"https://files.pythonhosted.org/packages/b0/urllib3-1.26.18-py2.py3-none-any.whl\", upload-time = 2023-10-17T17:46:21Z, size = 143835, hashes = { sha256 = \"bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb\" } }]\n"; + + /// #479: a Hatch project whose environments are locked carries a + /// `pylock*.toml` Hatch DERIVES from pyproject (and regenerates on the + /// next sync), so hosted mode wires pyproject — the declaration Hatch + /// locks and installs from — beside the lock, with a warning, instead + /// of rewriting only the lock (which Hatch then threw away). + #[test] + fn hatch_locked_env_pylock_also_wires_pyproject() { + let head = "[build-system]\nrequires = [\"hatchling\"]\nbuild-backend = \"hatchling.build\"\n\n[project]\nname = \"app\"\nversion = \"0.1.0\"\ndependencies = [\"urllib3==1.26.18\"]\n"; + for (config, lock) in [ + ( + "\n[tool.hatch.envs.default]\nlocked = true\ninstaller = \"uv\"\n", + "pylock.toml", + ), + ("\n[tool.hatch]\nlock-envs = true\n", "pylock.toml"), + ( + "\n[tool.hatch.envs.test]\nlocked = true\n", + "pylock.test.toml", + ), + ] { + let files: BTreeMap = [ + ("pyproject.toml".to_string(), format!("{head}{config}")), + (lock.to_string(), HATCH_PYLOCK.to_string()), + ] + .into_iter() + .collect(); + let result = rewrite_registry_redirect(&files, &[patch()]); + assert!( + result.confirmed_hatch_uuids.contains("test-uuid"), + "{config}: {:?}", + result.warnings + ); + assert!( + result.files["pyproject.toml"].contains(&format!( + "urllib3 @ {}#sha256={}", + patch().artifact_url, + "a".repeat(64) + )), + "{config}" + ); + // The lock stays consistent with pyproject until Hatch + // regenerates it from the wired declaration. + assert!( + result.files[lock].contains(&patch().artifact_url), + "{config}" + ); + assert!( + result + .warnings + .iter() + .any(|w| w.code == "redirect_hatch_lock_regenerated" && w.detail.contains(lock)), + "{config}: {:?}", + result.warnings + ); + } + // Without locked environments a pylock is its own install source + // (`uv export`, `pip lock`): the lock lane keeps it, pyproject stays. + let files: BTreeMap = [ + ("pyproject.toml".to_string(), head.to_string()), + ("pylock.toml".to_string(), HATCH_PYLOCK.to_string()), + ] + .into_iter() + .collect(); + let result = rewrite_registry_redirect(&files, &[patch()]); + assert!(!result.files.contains_key("pyproject.toml")); + assert!(result.confirmed_hatch_uuids.is_empty()); + } + #[test] fn hatch_confirmation_ignores_inactive_sources_and_comments() { let dep = patch(); diff --git a/crates/socket-patch-core/src/utils/hatch.rs b/crates/socket-patch-core/src/utils/hatch.rs index fd8bf7a22..6ba38b4c2 100644 --- a/crates/socket-patch-core/src/utils/hatch.rs +++ b/crates/socket-patch-core/src/utils/hatch.rs @@ -28,6 +28,51 @@ pub fn is_hatch(files: &BTreeMap) -> bool { }) } +/// Whether the project's Hatch configuration locks environments (Hatch +/// 1.17+: `lock-envs = true`, or an environment with `locked = true` or a +/// `lock-filename`). Read the way Hatch merges its configuration: a +/// hatch.toml TOP-LEVEL key replaces pyproject's `[tool.hatch]` key of the +/// same name, so a hatch.toml `envs` table hides every pyproject +/// environment and a hatch.toml `lock-envs` overrides pyproject's (the rule +/// [`environment_specs`] applies to dependencies). +fn locks_environments(files: &BTreeMap) -> bool { + let external = parsed(files, HATCH_FILES[1]); + let project = parsed(files, HATCH_FILES[0]); + let key = |name: &str| -> Option { + if let Some(item) = external.as_ref().and_then(|doc| doc.get(name)) { + return Some(item.clone()); + } + project + .as_ref() + .and_then(|doc| doc.get("tool")) + .and_then(|tool| tool.get("hatch")) + .and_then(|hatch| hatch.get(name)) + .cloned() + }; + key("lock-envs").and_then(|item| item.as_bool()) == Some(true) + || key("envs") + .as_ref() + .and_then(Item::as_table_like) + .is_some_and(|envs| { + envs.iter().any(|(_, env)| { + env.get("locked").and_then(Item::as_bool) == Some(true) + || env.get("lock-filename").is_some() + }) + }) +} + +/// Whether `path` is a PEP 751 lock Hatch derives from pyproject (#479): +/// a `pylock.toml` / `pylock..toml` in a Hatch project whose +/// environments are locked. Hatch regenerates it from pyproject whenever +/// the dependency hash changes, so it is not an independent install +/// source: the Hatch wiring (pyproject) is what must be rewritten, and a +/// rewrite of only the lock is thrown away by the next environment sync. +pub fn is_hatch_lock(files: &BTreeMap, path: &str) -> bool { + crate::utils::python_lock::is_pep751_lock_name(path) + && is_hatch(files) + && locks_environments(files) +} + /// One PEP 508 dependency string Hatch installs. pub(crate) struct HatchSpec<'d> { /// `pyproject.toml` or `hatch.toml` (a [`HATCH_FILES`] entry). @@ -518,6 +563,42 @@ pub fn plan( mod tests { use super::*; + /// #479 (Bugbot): Hatch merges hatch.toml over `[tool.hatch]` by + /// top-level key, so a shadowed `locked` / `lock-envs` does not make a + /// pylock Hatch-derived, and one set in hatch.toml does. + #[test] + fn hatch_lock_detection_follows_the_config_merge() { + let head = "[build-system]\nrequires = [\"hatchling\"]\nbuild-backend = \"hatchling.build\"\n\n[project]\nname = \"app\"\n"; + let files = |pyproject: &str, hatch: Option<&str>| -> BTreeMap { + let mut files = + BTreeMap::from([("pyproject.toml".to_string(), format!("{head}{pyproject}"))]); + if let Some(hatch) = hatch { + files.insert("hatch.toml".to_string(), hatch.to_string()); + } + files + }; + let locked = "\n[tool.hatch.envs.default]\nlocked = true\n"; + assert!(is_hatch_lock(&files(locked, None), "pylock.toml")); + assert!(!is_hatch_lock(&files(locked, None), "uv.lock")); + // A hatch.toml `envs` table replaces pyproject's environments. + assert!(!is_hatch_lock( + &files(locked, Some("[envs.default]\ninstaller = \"uv\"\n")), + "pylock.toml" + )); + assert!(is_hatch_lock( + &files("", Some("[envs.test]\nlocked = true\n")), + "pylock.test.toml" + )); + // A hatch.toml `lock-envs` overrides pyproject's. + let lock_envs = "\n[tool.hatch]\nlock-envs = true\n"; + assert!(is_hatch_lock(&files(lock_envs, None), "pylock.toml")); + assert!(!is_hatch_lock( + &files(lock_envs, Some("lock-envs = false\n")), + "pylock.toml" + )); + assert!(!is_hatch_lock(&files("", None), "pylock.toml")); + } + fn files(text: &str) -> BTreeMap { [("pyproject.toml".into(), text.into())] .into_iter() diff --git a/crates/socket-patch-core/src/vendor/pypi.rs b/crates/socket-patch-core/src/vendor/pypi.rs index 5338b912e..a81b73952 100644 --- a/crates/socket-patch-core/src/vendor/pypi.rs +++ b/crates/socket-patch-core/src/vendor/pypi.rs @@ -310,10 +310,20 @@ async fn detect_pypi_flavor( tool_locks.contains(&lock) })); } + // #479: a pylock Hatch derives from pyproject (locked environments) is + // regenerated by Hatch on the next sync, so it is not a lock to wire on + // its own; the project routes to the Hatch lane (and its guards). + let mut hatch_files = std::collections::BTreeMap::new(); + for file in crate::utils::hatch::HATCH_FILES { + if let Ok(text) = read_regular_to_string(&project_root.join(file)).await { + hatch_files.insert(file.to_string(), text); + } + } let additional_locks: Vec = crate::utils::python_lock::python_lock_paths(project_root) .map_err(|error| ("pypi_lock_read_failed", error.to_string()))? .into_iter() .filter(|path| path != "uv.lock") + .filter(|path| !crate::utils::hatch::is_hatch_lock(&hatch_files, path)) .collect(); // #912 / #1122: a pylock beside a `Pipfile` belongs to Pipenv, not to a // PEP 751 installer. With a `Pipfile.lock` Pipenv installs from that @@ -2805,6 +2815,42 @@ mod tests { } } + /// #479: in a Hatch project with locked environments, the pylock is + /// Hatch's own derived lock: it routes to the Hatch flavor (whose guards + /// refuse the uv installer locked environments use) instead of the + /// python-lock lane, which wired only the lock Hatch then regenerated. + /// Without locked environments the same pylock still routes to it. + #[tokio::test] + async fn hatch_locked_env_pylock_routes_to_hatch() { + let head = "[build-system]\nrequires = [\"hatchling\"]\nbuild-backend = \"hatchling.build\"\n\n[project]\nname = \"app\"\nversion = \"0.1.0\"\ndependencies = [\"six==1.16.0\"]\n"; + for (lock, config) in [ + ( + "pylock.toml", + "\n[tool.hatch.envs.default]\nlocked = true\ninstaller = \"uv\"\n", + ), + ("pylock.test.toml", "\n[tool.hatch]\nlock-envs = true\n"), + ] { + let tmp = tempfile::tempdir().unwrap(); + touch(tmp.path(), lock, PIPENV_PYLOCK).await; + touch(tmp.path(), "pyproject.toml", head).await; + let (selected, _) = detect_pypi_flavor(tmp.path(), Some(("six", "1.16.0"))) + .await + .unwrap(); + assert_eq!(selected, PypiFlavor::PythonLocks, "{lock}: no locked envs"); + touch(tmp.path(), "pyproject.toml", &format!("{head}{config}")).await; + let (selected, warnings) = detect_pypi_flavor(tmp.path(), Some(("six", "1.16.0"))) + .await + .unwrap(); + assert_eq!(selected, PypiFlavor::Hatch, "{lock}"); + assert!( + warnings + .iter() + .all(|w| w.code != "pypi_unmatched_lockfiles"), + "{lock}: {warnings:?}" + ); + } + } + /// #1120: a `requirements.txt` exported beside the governing lock in /// UTF-16 (`uv export > requirements.txt` in Windows PowerShell 5.1) /// is installed by pip and uv like its UTF-8 twin, so it is a loud diff --git a/docs/testing/hatch.md b/docs/testing/hatch.md index 405c69e37..0dd27b61e 100644 --- a/docs/testing/hatch.md +++ b/docs/testing/hatch.md @@ -17,6 +17,18 @@ Hatch 0.x continues to use the requirements/pip backend. Existing lockfile and requirements routing retains precedence over Hatchling's build marker. A build backend alone does not change which existing pip inputs are wired. +Hatch 1.17+ locked environments (`lock-envs = true`, an environment's +`locked = true` or `lock-filename`) write a `pylock.toml` / +`pylock..toml` that Hatch derives from pyproject and regenerates on the +next sync whenever the dependency hash changes (#479). Such a lock is not an +install source of its own: hosted mode wires the Hatch declarations as well +as the lock (warning `redirect_hatch_lock_regenerated`; run `hatch dep lock` +to refresh the lock, as `hatch dep lock --check` reports it stale until +then), and vendored mode routes the project to the Hatch lane, whose +pip-installer requirement refuses the uv installer locked environments use. +A `pylock.toml` in a Hatch project without locked environments (`uv export`, +`pip lock`) is still wired as a standalone lock. + A range, transitive-only declaration, dynamic dependency metadata, custom environment plugin, source table or conditional override is refused before writing. Use agent mode (`scan --mode agent` + `socket-patch apply` in CI) for these shapes. Hosted PEP 735 groups are