Skip to content

extend project URLs -> external references #800

Open
#1066
Open
extend project URLs -> external references#800
#1066
Labels
enhancementNew feature or requestgood first issueGood for newcomershacktoberfesthelp wantedExtra attention is needed

Description

@jkowalleck
jkowalleck
opened on Sep 25, 2024

Is your feature request related to a problem? Please describe.

This tool detects project URLs and emits them in the SBOM results as "external references".
The URL types are detected based on patterns - see

cyclonedx-python/cyclonedx_py/_internal/utils/cdx.py

Lines 85 to 104 in 721f12d

__known_ulr_labels: Dict[str, ExternalReferenceType] = {
# see https://peps.python.org/pep-0345/#project-url-multiple-use
# see https://github.com/pypi/warehouse/issues/5947#issuecomment-699660629
'bugtracker': ExternalReferenceType.ISSUE_TRACKER,
'issuetracker': ExternalReferenceType.ISSUE_TRACKER,
'issues': ExternalReferenceType.ISSUE_TRACKER,
'bugreports': ExternalReferenceType.ISSUE_TRACKER,
'tracker': ExternalReferenceType.ISSUE_TRACKER,
'home': ExternalReferenceType.WEBSITE,
'homepage': ExternalReferenceType.WEBSITE,
'download': ExternalReferenceType.DISTRIBUTION,
'documentation': ExternalReferenceType.DOCUMENTATION,
'docs': ExternalReferenceType.DOCUMENTATION,
'changelog': ExternalReferenceType.RELEASE_NOTES,
'changes': ExternalReferenceType.RELEASE_NOTES,
# 'source': ExternalReferenceType.SOURCE-DISTRIBUTION,
'repository': ExternalReferenceType.VCS,
'github': ExternalReferenceType.VCS,
'chat': ExternalReferenceType.CHAT,
}

it appears that PyPI also documented the way they detect URL types: https://docs.pypi.org/project_metadata/#icons

It would be great if the PyPI heuristics could be applied, too.

Describe the solution you'd like

Have the heuristics from PyPI apply to URL type detections, so that emitted external references follow the de facto standards, too.

These heuristics work on the URL name, as well on the URL itself.

Additional context

PyPI's type classification according to https://docs.pypi.org/project_metadata/#icons

Activity

  1. jkowalleck
    added
    enhancementNew feature or request
    help wantedExtra attention is needed
    good first issueGood for newcomers
    hacktoberfest
    on Sep 25, 2024

  2. eeshs commented on Aug 30, 2025

    @eeshs
    eeshs
    on Aug 30, 2025

    Hello. I am a looking to make my first open source contribution and think this is a good issue for me to start with. I would like to begin working on this issue if nobody else is on it currently.

  3. jkowalleck commented on Sep 2, 2025

    @jkowalleck
    jkowalleck
    on Sep 2, 2025
    MemberAuthor

    @eeshs , sure - go ahead.
    if you need any mentoring, let me know.
    please open a draft pull request as soon as possible, so that i can assist, if needed.

  4. arun2dot0 commented on Jun 20, 2026

    @arun2dot0
    arun2dot0
    on Jun 20, 2026

    I'd like to pick this up. The issue was kindly assigned to @eeshs back in Aug/Sep 2025, but there hasn't been a PR or branch in the ~9 months since, so I'm assuming it's available again — @eeshs, apologies if you're still on it, just say the word and I'll back off.

    I've opened a draft PR early (as @jkowalleck requested in this thread) so the approach can be reviewed before I finish: #1066.

    Approach

    The current implementation classifies a project URL into an ExternalReferenceType using only the URL label (an exact-match dict in cyclonedx_py/_internal/utils/cdx.py). I'm extending it to use PyPI's documented heuristics (https://docs.pypi.org/project_metadata/#icons) — matching on both the label (exact + prefix, honoring PyPI's * semantics) and the URL host (domain + subdomains).

    Precedence (label-first): exact label → label prefix → host suffix → host subdomain-prefix → OTHER. The label expresses the author's explicit intent; the host only fills gaps. So e.g. a Funding label pointing at github.com stays OTHER rather than flipping to VCS.

    Expandability / data-vs-logic: all rules live in a new data-only module url_classifiers.py as four declarative tables; the matcher in cdx.py is pure logic and doesn't change when rules are added. Adding a host or label is a one-line data edit.

    Mapping decisions worth a sanity check

    CycloneDX's ExternalReferenceType enum is narrower than PyPI's icon set, so a few categories need a judgment call:

    PyPI category → CycloneDX type note
    Funding / Sponsor / Donation / Donate OTHER no CycloneDX funding type; OTHER over forcing WEBSITE
    Discord / Slack / Gitter CHAT splits PyPI's flat "social" bucket…
    Reddit / YouTube / Twitter-X / Mastodon / Bluesky SOCIAL …using CycloneDX's CHAT vs SOCIAL distinction
    AppVeyor / CircleCI / Codecov / Coveralls / Travis BUILD_SYSTEM
    github.com / github.io / gitlab.com / bitbucket.org VCS
    *.readthedocs.io, docs./documentation. subdomains DOCUMENTATION
    pypi.org / pypi.io / pypi.python.org DISTRIBUTION
    google.com (unmapped → OTHER) too ambiguous to classify

    Status

    • ✅ Task 1 — data module + label matching (exact + prefix), unit-tested, flake8/isort/mypy clean.
    • ⬜ Task 2 — host matching (suffix + subdomain-prefix) with label-first precedence.
    • ⬜ Task 3 — pass the URL through the three callers (poetry, pep621, packaging metadata) + reconcile snapshots.

    @jkowalleck — does the mapping table above look right to you, especially Funding → OTHER? Happy to adjust before I complete Tasks 2–3.

  5. jkowalleck
    linked a pull request that will close this issuefeat: apply PyPI heuristics to project-URL classification (#800) #1066on Jul 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestgood first issueGood for newcomershacktoberfesthelp wantedExtra attention is needed

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions