Diff-aware detection and clone baseline: gate CI on new duplication only

[kucherenko]

Summary

Umbrella issue for a family of long-standing requests that all reduce to the same capability: distinguish new duplication from pre-existing duplication, and gate CI on the new part only. Consolidates #879, #938, #588, #468, and the earlier #254 into one coherent design for the Rust v5 engine.

Design revision (2026-08-19): an earlier draft proposed two parallel mechanisms — a stateless git-diff mode and a stateful baseline mode. This revision reduces the design to one mechanism: the clone-fingerprint baseline. Diff-vs-ref checking is expressible as an ephemeral baseline built from the base ref, so it becomes a thin convenience flag in a later phase rather than a parallel implementation.

The requests being consolidated

Issue	Request	Core need
#879	Detect and fail on clones introduced by a Git diff	PR gate that ignores legacy duplication
#938	ThresholdReporter tolerates a full clone-instance swap under the aggregate percentage	Baseline of clone fingerprints; fail when the clone set gains members, not only when the percentage rises
#588	Check one specific file (or the files being modified) against the whole repo	Scan the full source set but report only clones touching a focus set of files
#254 (closed)	Compare commit-diff files against all sources	Same as #879; the git diff --name-only | xargs jscpd workaround misses clones between changed and unchanged files
#468	thresholdAbsolute — gate on clone count, immune to "add filler code to dilute the percentage"	A gate that is not skewed by adding new clean code — satisfied by count-based gating on new clones

All five fail the same way today: detection and gating operate on the whole corpus at once, with a single aggregate percentage as the only exit signal. There is no notion of which clones are new relative to a recorded baseline.

Design: one mechanism, the clone-fingerprint baseline

A clone is new or known, decided by membership in a set of fingerprints of accepted clones. The fingerprint set normally lives in a committed .jscpd-baseline.json; a later phase can build it ephemerally from a git ref instead. Both feed the same gating and reporting pipeline.

Phase 1 — committed baseline (closes #938, #468, and the CI-gate core of #879/#254)

cpd --baseline .jscpd-baseline.json --fail-on-new-clones .   # gate against recorded state
cpd --baseline .jscpd-baseline.json --update-baseline .      # absorb current state

• A committed .jscpd-baseline.json stores fingerprints of accepted clones. A clone whose fingerprint is absent from the baseline is new → fails the run even when total.percentage stays under --threshold. Removed clones are fine (the baseline shrinks on next update). This closes the ThresholdReporter gates only on aggregate total.percentage, silently tolerating a full instance swap #938 instance-swap repro: a swap is one removal plus one addition, and the addition fires.
• Fingerprint multiplicity (from ThresholdReporter gates only on aggregate total.percentage, silently tolerating a full instance swap #938 discussion): store a count per fingerprint, so removing one instance and adding an identical one elsewhere is not invisible.
• Non-silent regeneration (from ThresholdReporter gates only on aggregate total.percentage, silently tolerating a full instance swap #938 discussion): --update-baseline prints added/removed fingerprint counts, so baseline growth is visible in CI logs and PR review.
• --fail-on-new-clones[=N] exits non-zero when more than N (default 0) new clones are found, independently of and composable with the existing --threshold. Count-based gating on new clones is what Consider a thresholdAbsolute Option #468 actually wants: it cannot be gamed by adding filler code.
• Conflict-friendly file format: versioned ("version": 1), sorted, stable, one fingerprint per line — merge conflicts resolve trivially and baseline diffs are reviewable in PRs.

The PR-gate workflow needs no git logic at all: main carries the baseline, a PR run fails on any fingerprint not in it. Because fingerprints are content-hash based (not line numbers), unrelated edits in the same files don't fire; editing inside an existing clone changes its fingerprint and flags it as new — exactly the semantics the earlier diff-mode draft approximated with line-range overlap, without the overlap-definition ambiguity. Whole classes of diff-mode edge cases disappear: no shallow-checkout problem (the baseline is a file in the checkout), no fetch-depth: 0 docs, no git rename detection, no "which mode wins when both are active".

Fingerprint design question (settle early)

The earlier draft proposed fingerprint = content hash of the duplicated fragments + file-pair paths. Path-bearing fingerprints make a pure rename look like all-new clones (the diff-mode draft compensated with git rename detection, now dropped). Preferred alternative: content-hash-only fingerprints with a multiplicity count — a rename is invisible (content and count unchanged), while a genuinely new occurrence still bumps the count and fires. The gate doesn't need "which file pair", and the report still shows paths from the current scan. Decide and document before freezing the format.

Phase 2 (optional, later) — ephemeral baseline from a git ref (stateless variant of #879/#254)

cpd --baseline-from-ref origin/main --fail-on-new-clones .

Sugar over phase 1: read the base ref's tree in-process (gitoxide, already a dependency for blame — no shelling out, no worktree), scan it in memory to build a transient fingerprint set, then compare exactly as phase 1 does. Same fingerprint engine, same gating, same reporting; ~no new concepts.

Known cost, accepted: this scans the corpus twice (base tree + HEAD) instead of once plus a cheap git diff. It only affects users who choose not to commit a baseline; the Rust engine is fast enough for this to be acceptable. The shallow-checkout caveat returns for this flag only (the base ref's blobs must be present) — fail with a clear message and document the fix.

Decoupled — --focus (#588)

--focus <path,...> is orthogonal to newness: it's a report filter (full-corpus scan, report only clones touching the given files), not a newness criterion. It survives as its own small flag and can land independently of, before, or after the baseline work — or be split into its own issue.

Shared reporting surface

Baseline comparison populates fields that already exist in the data model and are currently unused in the Rust engine (StatRow.new_clones, StatRow.new_duplicated_lines; per-clone isNew in the v4 TypeScript schema):

• console / console-full: new clones flagged (e.g. Clone found (javascript) [NEW]), summary line gains "N new".
• json: per-clone isNew plus the existing newClones / newDuplicatedLines statistic fields — additive, schema-compatible.
• sarif: new clones at level error, known ones stay warning, so GitHub code scanning highlights only the regression.
• openmetrics (OpenMetrics reporter #422): jscpd_new_clones / jscpd_new_duplicated_lines gauges for GitLab MR metrics.
• threshold / exit gate: --fail-on-new-clones composes with the existing --threshold.

Non-goals (tracked separately)

• Historical trend reporting across commits (Walk back through the history and collect stats #77) — a different feature (walk history, emit stats per revision), though the baseline file format could serve it later.
• Detection-time performance work: the design deliberately scans the full corpus and filters afterwards; speeding up scans by scanning fewer files is exactly the trap Is jscpd support comparasions between commit diffs and all sources? #254/Feature: Detect and fail on clones introduced by a Git diff #879 identified.
• Line-range git-diff overlap marking (the earlier draft's "diff mode"): superseded by fingerprint comparison; not planned as a separate mechanism.

Acceptance criteria

Phase 1

• Fingerprint engine: content-hash-based clone fingerprints with multiplicity (format question above settled and documented)
• --baseline <file> marks clones absent from the baseline as new (full-corpus scan)
• --update-baseline regenerates the baseline and prints added/removed counts
• --fail-on-new-clones[=N] exits non-zero on new clones, independent of --threshold
• Baseline file: versioned, sorted, stable, merge-friendly
• isNew / newClones / newDuplicatedLines populated in json; new-clone markers in console, sarif (error level), openmetrics
• Docs: CI recipes for GitHub Actions and GitLab CI (committed-baseline PR gate on new duplication only)

Phase 2 (optional)

• --baseline-from-ref <git-ref> builds an ephemeral baseline from the base ref's tree (via a temporary git worktree — see implementation note below)
• Clear error message for missing base refs in shallow checkouts, with documented fix

Decoupled

• --focus <paths> reports only clones touching the given files (may be split into its own issue)

Closes / supersedes: #879, #938, #588, #468 (each can be closed against this once the corresponding checkbox lands).

[kucherenko]

Phase 1 implemented in 251544b (on feat/422-openmetrics-reporter, since the openmetrics gauges build on that reporter).

Fingerprint question settled: content-hash-only with multiplicity — the fingerprint is snippet_pair_hash of the two duplicated snippets, the same value the SARIF reporter already emits as partialFingerprints["jscpdCloneHash/v1"], formatted as 16 hex chars. No file paths in the fingerprint, so pure renames don't fire; the per-fingerprint count catches added identical occurrences. Baseline file is {"version": 1, "fingerprints": {"<hash>": <count>}}, sorted one per line.

Behavioral decisions made during implementation:

• A missing baseline file is a hard error with a hint to run --update-baseline (a gate that silently passes is worse than a failing one); --update-baseline creates the file on first run.
• --update-baseline and --fail-on-new-clones without --baseline error out rather than no-op.
• --fail-on-new-clones is skipped during --update-baseline runs (the run just absorbed its clones), but the report still shows what was new relative to the previous baseline, and the added/removed counts print to stderr.
• The GitHub Action gained dedicated baseline / update-baseline / fail-on-new-clones inputs.

Test coverage: 11 unit tests on the baseline module, reporter tests for isNew/sarif-error/openmetrics gauges, 9 CLI integration tests for the create → pass → fail → allow-N workflow and flag validation. Docs (options, CI recipes for GitHub Actions + GitLab CI, openmetrics gauges) are staged for jscpd.dev.

Phase 2 (--baseline-from-ref) and --focus remain open.

[kucherenko]

Phase 2 implemented in 4a72a70 (same branch as phase 1).

Implementation note — git worktree instead of gitoxide: the design sketch assumed gitoxide was already a dependency, but the engine actually shells out to the git binary for blame, so --baseline-from-ref follows that precedent: git worktree add --detach into a temp dir, scan the base tree with the identical RunConfig, fingerprint its clones into an in-memory baseline (baseline::apply_in_memory, reusing all phase 1 machinery), then git worktree remove --force (with worktree prune fallback). No new dependency, no state left in .git — the integration test asserts git worktree list is clean afterwards.

Behavior:

• Scan paths are remapped into the base tree; paths that don't exist in the base ref count as new code (empty contribution to the baseline).
• Missing ref → hard error naming the ref and hinting git fetch origin main / fetch-depth: 0 for shallow checkouts.
• Conflicts with --baseline and --update-baseline (clap-level and config-mix validation); --fail-on-new-clones accepts either baseline source.
• Config key baselineFromRef; GitHub Action input baseline-from-ref (docs recommend fetch-depth: 0).

Tests: 5 new CLI integration tests against a real temp git repo (pass vs HEAD, fail on uncommitted new pair, missing-ref hint, flag conflict, console (1 new) marking) plus an apply_in_memory unit test — 825 workspace tests green.

Remaining: --focus <paths> (decoupled — candidate for its own issue).