[NachiketPatil26]

Fixes #5374

Copyright detection spends most of its extra time in gibberish detection and markup stripping, which ran on nearly every line. This keeps the detections exactly the same and skips work that can't change the result:

• is_candidate() now checks for years and copyright markers first and only runs gibberish detection on lines that have one. A line without either was never a candidate anyway, so the result is the same.
• strip_known_markup_from_text() returns early with only whitespace normalized when the line has no < or >, no skipped attribute (href=, @end, mailto:, ...) and none of the bare w3.org tags. In that case keep_tag() keeps every token, so the full split gives the same string.
• detect_gibberish() no longer normalizes the line before calling avg_transition_prob(), which normalizes it again through ngram().

Results

llvm/lib from llvm-project 18.1.8 (same corpus as the issue), single process, detect_copyrights() on every file, macOS arm64, Python 3.12:

	develop	this PR
723-file sample, best of 3	23.6 s	18.5 s (-22%)
all 4,335 files	145.1 s	117.7 s (-19%)

The JSON dumps of all detections are byte-identical (142 detections on the full tree).

I also ran the old and new versions of strip_known_markup_from_text(), detect_gibberish() and is_candidate() side by side on every distinct line of llvm/lib, tests/cluecode/data and tests/textcode/data, plus 200k random lines mixing tags, MAILTO:/maİlto:, w3.org URLs and non-ASCII characters (about 2M lines in total): no differences.

pytest --test-suite=all tests/cluecode tests/textcode: 5352 passed, 17 xfailed. I added tests for the markup fast path and for is_candidate() still rejecting gibberish lines that have a year and a marker.

Tasks

• Reviewed contribution guidelines
• PR is descriptively titled 📑 and links the original issue above 🔗
• Tests pass -- look for a green checkbox ✔️ a few minutes after opening your PR
• Commits are in uniquely-named feature branch and has no merge conflicts 📁
• Updated documentation pages (if applicable)
• Updated CHANGELOG.rst (if applicable)

[knowprasad]

Tested this locally against a sample tree and verified the test suite. The ~20% speedup on candidate selection is a great optimization, and skipping the redundant normalize() call in detect_gibberish() makes total sense given ngram() handles it.

Two small notes from reading the diff:

• In is_candidate(), lowered_line = prepared_line.lower() is allocated before checking copyrights_hint.years(prepared_line). If you check years() first or move the lowercase conversion into the statement_markers check, lines with year hits avoid allocating lowered_line entirely.

• Regarding CI: the Azure pipeline failure on license_validate_ignorables_2 is pre-existing and failing across develop and recent PRs (like Fix MPL version in RabbitMQ Java client license rule #5387), so that's not introduced by your changes.

Clean patch and solid benchmarks.