Lock-only scans still drop a requirements.txt pip decodes through a PEP 263 coding line (latin-1): "No pypi packages found", exit 0, while the same project with a venv refuses candidate_file_unreadable #1119

[agent] Found by the scheduled pip / requirements.txt bug-hunt routine (ledger #309).

Summary

#724 (fix for #721) made lock-only discovery decode a requirements file by its BOM, the way pip does, and made hosted mode refuse a candidate file it can't decode (candidate_file_unreadable). pip's auto_decode has one more rule: with no BOM it honours a PEP 263 # -*- coding: <enc> -*- line in the first two lines. utils::requirements::decode returns None for such a file, and requirements_tree in lock-only discovery reads None as "no requirements file". So on a fresh checkout (no venv), scan prints No pypi packages found. and exits 0 with nothing pinned, and pip goes on to install the unpatched pin.

The same file in a project whose venv holds the package is refused loudly (candidate_file_unreadable, exit 1), so the refusal only fires when something else discovered the package first. A -r include in that encoding is dropped silently as well.

I raised this case in a comment on #721 before #724 merged (#721 (comment)). #721 was then closed with only the BOM cases fixed, so I'm filing the remainder here.

Impact

A CI job that runs socket-patch scan --mode hosted (or --mode vendored) on a fresh checkout before pip install -r requirements.txt reports success with nothing found. The install then pulls the unpatched release, and nothing in the output or the --json envelope says that a requirements file was skipped. The likeliest real-world trigger is a requirements file with a non-ASCII comment (a maintainer name, say) in a legacy encoding. pip accepts it whenever it carries a coding line.

Repro (Linux, main b96a785)

# Run in an empty directory, with VIRTUAL_ENV pointing at an EMPTY venv (fresh checkout / lock-only).
printf '# -*- coding: latin-1 -*-\n# Maintainer: Jos\xe9\nsix==1.16.0\n' > requirements.txt

pip install --no-deps --target t -r requirements.txt   # pip 20.3.4 / 24.0 / 26.2.1: rc=0, installs six 1.16.0
# (without the coding line, pip fails: UnicodeDecodeError, so the coding line is what makes the file valid)

socket-patch scan --mode hosted --yes --ecosystems pypi
# -> "No pypi packages found."  exit 0, requirements.txt unchanged
socket-patch scan --mode hosted --yes --json --ecosystems pypi
# -> {"status":"success","scannedPackages":0, ..., "redirect":{"redirected":0,"patches":[],"skipped":[],"warnings":[]}}
socket-patch scan --mode vendored --yes --json --ecosystems pypi
# -> status success, scannedPackages 0, exit 0

# Same file, but with a .venv holding six==1.16.0:
socket-patch scan --mode hosted --yes --json --ecosystems pypi
# -> exit 1, "requirements.txt is not UTF-8 text ... re-save it as UTF-8 and re-run; nothing was written"

# Include variant (lock-only): requirements.txt = "-r dev.txt", dev.txt = the latin-1 file above
# -> "No pypi packages found." exit 0; pip installs six from dev.txt.

socket-patch vex on the same lock-only checkout does warn cannot read requirements.txt: stream did not contain valid UTF-8, so the scan is the only silent path.

The patch API was a local mock serving a patch for pkg:pypi/six@1.16.0. Scans with a venv find and offer it.

Expected vs actual

• Expected: lock-only discovery decodes the file the way pip does, BOM first and then the PEP 263 coding line (the Fix UTF-16 requirements.txt silently skipped (#721) #724 commit message says it "decodes requirements.txt and its -r includes by BOM the way pip does"). Failing that, it should refuse or warn by name, like the venv path's candidate_file_unreadable and vex's lockfile_unreadable. A silent success contradicts the reason Hosted and lock-only scans treat a UTF-16 requirements.txt (what Windows PowerShell's pip freeze > writes) as absent: exit 0, no warning, and pip keeps installing the unpatched pin #721 was fixed: a requirements file pip installs from must not be treated as absent.
• Actual: exit 0, No pypi packages found., an empty redirect block and no warning, for both the root file and a -r include.

OS × version

First bad: this never worked. Before #724 every non-UTF-8 file was skipped (#721); #724 fixed only the BOM encodings.

Suspect code

• crates/socket-patch-core/src/utils/requirements.rs:26 (decode): BOMs, then String::from_utf8, with no PEP 263 coding-line step (pip: pip/_internal/utils/encoding.py::auto_decode).
• crates/socket-patch-core/src/vendor/lock_inventory/pypi.rs:733 (requirements_tree): read(ROOT)? turns an undecodable root into None (no requirements), and an undecodable include is continued, with no diagnostic in either case.

No probe runs: the sandbox can't delete probe branches, and the defect is in OS-independent decoding.

Backlog review — 2026-10-08

Priority: P1 → P2. Lock-only discovery misses a PEP 263 non-UTF8 requirements file. Retain support for the uncommon encoding; no false VEX evidence is claimed.