Add cuDNN discovery and NCCL header support - #2680
Conversation
github-actions
Bot
added
the
cuda.pathfinder
There was a problem hiding this comment.
Pull request overview
This pull request extends cuda_pathfinder’s component discovery catalog to cover cuDNN (shared libraries + headers) and NCCL headers, and wires those additions into the test/dependency setup used for validating wheel and conda layouts.
Changes:
- Add cuDNN dynamic-library descriptor metadata for Linux and Windows, plus a catalog test to validate expected wheel layout metadata.
- Add header descriptor metadata for cuDNN (Linux/Windows) and NCCL (Linux-only), plus a test to validate header metadata against expected wheel layouts.
- Update pathfinder conda test-environment helpers and
cuda_pathfindertest dependency groups to include cuDNN (and NCCL on Linux).
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
toolshed/conda_create_for_pathfinder_testing.sh |
Add cudnn and nccl to the Linux conda-based test environment packages. |
toolshed/conda_create_for_pathfinder_testing.ps1 |
Add cudnn to the Windows conda-based test environment packages. |
cuda_pathfinder/tests/test_find_nvidia_headers.py |
Extend non-CTK distribution metadata patterns for cuDNN/NCCL and add a header descriptor layout test. |
cuda_pathfinder/tests/test_descriptor_catalog.py |
Add a descriptor-catalog test validating cuDNN wheel layout metadata. |
cuda_pathfinder/pyproject.toml |
Add nvidia-cudnn-cu12 / nvidia-cudnn-cu13 to the CUDA test dependency groups. |
cuda_pathfinder/cuda/pathfinder/_headers/header_descriptor_catalog.py |
Add header descriptor specs for cudnn and nccl. |
cuda_pathfinder/cuda/pathfinder/_dynamic_libs/descriptor_catalog.py |
Add a cuDNN dynamic-library descriptor spec for discovery on Linux/Windows. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
- cuDNN: added dynamic-library loading and header discovery. - NCCL: dynamic-library loading already existed; added the missing header discovery.
19a6448 to
3d1eab9
Compare
|
/ok to test ddaa78f |
|
@DEKHTIARJonathan I pushed four commits (one of them adding WoA support) and triggered the CI here. I'll do the WoA testing on the internal repo. |
|
I forgot to add: I didn't fully review myself yet. |
|
|
/ok to test |
@rwgk, there was an error processing your request: See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/1/ |
|
/ok to test 39f25a0 |
|
codex gpt-5.6-sol ultra PR 2680: first successful public CI log analysisAnalyzed against public CI run 32522149396, commit 39f25a0 ( Executive conclusionThe public run is fully green: 100 jobs succeeded, two intentionally gated CUDA Core API checks were skipped, no job failed or was cancelled, and the final aggregate status job succeeded. The run provides strong, genuinely unmocked integration evidence for the main Pathfinder contract introduced or affected by PR 2680:
This is sufficient real integration coverage for the pip-wheel discovery and native-load claims on the three public platforms above. It is not sufficient real evidence for Windows on ARM, standalone My merge-readiness judgment is therefore:
MethodI treated a test as "real" only when it used an installed NVIDIA artifact and the production discovery or native-loading code without replacing that operation with a mock. I used three evidence levels:
The audit used the completed GitHub Actions metadata and all 65 Linux/Windows GPU test logs, then cross-checked the relevant workflow and test implementation at the run SHA. Representative job links are included below. Run overview
The 24 wheel-build jobs comprise eight each for Linux x86_64, Linux AArch64, and Windows x64. They cover Python 3.10 through 3.15, including selected 3.14t and 3.15t free-threaded builds. GPU test matrixEvery one of these 65 jobs successfully completed all three relevant steps:
The test jobs span Python 3.10, 3.11, 3.12, 3.13, 3.14, 3.14t, 3.15, and 3.15t where selected by the platform matrix. Windows additionally spans MCDM, TCC, and WDDM driver modes. An important qualification: Why the dynamic-library test is genuinely realThe relevant test is data-driven over the descriptor-derived supported-library tables in For each library,
Those assertions are in Strict mode is also meaningful. The expected-failure helper at Real Windows x64 evidenceAll 21 Windows GPU test logs were checked. Every strict run reported:
Representative CUDA 12 evidence is job 96899599802, Python 3.10 / CUDA 12.9 / RTX 2080 WDDM. Its first permissive phase reports cuDNN, cublasLt, and NVRTC as not found. The dependency step then installs Representative CUDA 13 evidence is job 96899599699, Python 3.14t / CUDA 13.3 / A100 MCDM. It installs Normal Windows jobs report Real Linux evidenceAll 44 Linux GPU test logs were checked: 26 x86_64 and 18 AArch64. Every log showed the same useful before/after transition:
This transition is particularly good evidence: it rules out a pre-existing system cuDNN or NCCL installation masking an error in wheel discovery. Across those 44 jobs:
Representative jobs:
The 44 strict Linux repetitions total 65,128 passed test instances and 212 expected skips. Those are matrix repetitions, not 65,128 unique tests; the value is breadth across environments rather than independent test-case count. Real header evidenceThe relevant data-driven test is
The cuDNN distribution matcher is deliberately limited to backend packages ( This is strong evidence for the actual header-wheel layouts on Windows x64, Linux x86_64, and Linux AArch64. It does not compile a translation unit or validate transitive includes. Dependency-loading evidenceThe cuDNN descriptor declares required The strict environments contain real cublasLt and NVRTC wheels, and every platform/family log shows both can be loaded as real native libraries. Therefore the run proves:
The per-cuDNN child payload does not emit a nested trace for each dependency. The source call graph guarantees the attempt, while the separate real load records establish artifact availability. The optional-NVRTC-absent case remains callback-based unit coverage rather than a real-binary integration scenario. Coverage of the CI-failure fixThe previous public run failed on two Windows-host path-string assumptions. In representative Windows job 96899599699, both corrected tests pass in both Pathfinder phases:
The first now exercises the host's real The two recurring Pathfinder warnings in representative jobs are pytest configuration/marker warnings for Coverage map for PR 2680 behavior
Residual risks and recommended follow-ups1. Native Windows ARM64 remains the largest gapPublic CI has no Windows ARM64 job. The WoA orchestration test at Minimum high-value follow-up: on a native WoA runner, provision a pinned/checksummed official ARM64 cuDNN archive plus compatible cublasLt, set 2. Standalone and system installation roots are not real-testedNo public job uses a real standalone cuDNN archive, a real Program Files installation, Linux cuDNN DEB/RPM headers, a real Recommended follow-ups, in priority order:
3. Native attachment is not product functionalityThe real dynamic test proves that the top-level cuDNN/NCCL library can be attached and that Pathfinder reports the right file. It does not resolve a product symbol, query a version, create a cuDNN handle, or run a convolution/graph operation. In particular, it may not force every component that cuDNN loads lazily. A lightweight 4. Version breadth is environmental, not contractualThe requirements correctly constrain cuDNN to major 9, but the resolver selected only Final assessmentThis is a high-quality green run for a discovery/loading library. The most important paths are not merely mocked: the CI installs the built Pathfinder wheel and current NVIDIA wheels, observes the products absent before installation, discovers real headers afterward, and attaches real libraries through fresh-process native loader calls under strict failure semantics. Coverage is broad across 65 GPU environments and both CUDA-major package families. Accordingly, the public CI is sufficient to support merging the Windows x64 and Linux wheel behavior in PR 2680 and to demonstrate that the two Windows portability regressions are fixed. The result should not be summarized as "all supported layouts and architectures are real-tested." Windows ARM64, standalone/system roots, conda, optional-NVRTC absence, header compilation, and post-load product behavior remain explicitly outside the real integration evidence. A native WoA archive smoke is the one follow-up most likely to change confidence materially. |
kkraus14
left a comment
kkraus14
left a comment
There was a problem hiding this comment.
Non-blocking follow-ups from a local review of the current head. CI and the local suite are green; the inline comments cover forward-compatible Windows DLL matching, DLL-directory registration for an already-loaded standalone cuDNN, and Linux product-root dynamic discovery. I omitted the native Windows ARM64 wheel-dependency point because public CUDA 13.4/WoA packaging is still pending.
| desc: LibDescriptor, | ||
| target_arch: str | None = None, | ||
| ) -> str | None: | ||
| for dll_basename in reversed(cast(tuple[str, ...], desc.windows_dlls)): |
There was a problem hiding this comment.
Non-blocking: making every Windows filesystem search exact fixes the cuDNN sidecar ambiguity, but it also removes the previous forward-compatible wildcard lookup for versioned DLL names. For example, the catalog currently ends at cupti64_2026.3.0.dll, so a newer CUPTI DLL in a wheel, conda env, or toolkit root will be ignored until Pathfinder is updated. Could exact-name selection be an opt-in descriptor policy (enabled for cuDNN), while retaining wildcard discovery for descriptors such as CUPTI?
There was a problem hiding this comment.
Addressed in two commits:
c1cd585 — Restore forward-compatible Windows DLL discovery
It restored wildcard discovery while allowing cuDNN to require exact names.
The implementation was subsequently refined by:
e5c91aa — Constrain Windows DLL fallback matching
That second commit established the final policy: exact known names first, with explicit descriptor-specific fallback globs—currently only cupti64_*.dll.
| arm64=("bin/arm64",), | ||
| ), | ||
| program_files_root_globs_windows=WindowsSearchDirs.x64_only("NVIDIA/CUDNN/v9.*"), | ||
| requires_add_dll_directory=True, |
There was a problem hiding this comment.
Non-blocking edge case: this side effect is skipped when cudnn64_9.dll was already loaded from a late root such as CUDNN_PATH or Program Files. _load_lib_no_cache() passes find is not None to the already-loaded check, but find contains only the early wheel/conda result; the function then returns before late-root discovery. That can leave cuDNN's lazily loaded component DLLs undiscoverable. Could the already-loaded path register its resolved module directory whenever requires_add_dll_directory is set, independent of the early-find boolean, with a regression test for a preloaded standalone install?
| return cast(tuple[str, ...], desc.anchor_rel_dirs_linux) | ||
|
|
||
| def install_root_env_vars(self, _desc: LibDescriptor) -> tuple[str, ...]: | ||
| return () |
There was a problem hiding this comment.
Non-blocking: these no-op Linux hooks make the new product-root dynamic search Windows-only. Header discovery accepts CUDNN_PATH and NCCL_HOME, but dynamic discovery will not inspect locations such as $CUDNN_PATH/lib{,64} or $NCCL_HOME/{lib,build/lib} unless users also configure the OS loader path. Could we add Linux product-root metadata/relative directories and focused tests so header and library discovery stay aligned?
|
Thanks @kkraus14! I still haven't fully reviewed myself. I'm working on making the internal CI more complete. I have a fully successful internal CI run, but (codex analysis):
I believe it's fixable; and it's a good value, long-term. |
PR 2680: final ctk-next CI analysis with real WoA cuDNN coverageAnalyzed against the final green CI for ctk-next PR 537, which mirrors the product changes in public PR 2680 and adds ctk-next-only provisioning and verification for native Windows ARM64. Executive conclusionThe intended ctk-next matrix is fully green. Four platform workflows ran on Linux x86_64, Linux AArch64, Windows x64, and native Windows ARM64. Each completed its source/runbook job, wheel-build job, and isolated installed-wheel test job: 12/12 active platform jobs succeeded. Bandit also succeeded. The complete check rollup contains 13 successes and 17 checks intentionally skipped by branch or policy routing; there were no failures or cancellations. This result complements the already-posted public CI analysis for PR 2680:
MethodI used the same evidence standard as the public report:
For the new WoA conclusion, only the first level is used as proof. The dedicated strict tests and semantic smoke use the production Pathfinder public APIs, real ARM64 PE files, the native Windows loader, a real driver, and a real GB10 GPU. Their relevant discovery and loading operations are not mocked. I inspected the final check inventory, all four completed platform workflows, both WoA test-bearing job logs, and the exact workflow and verification source at the tested ctk-next head. Run inventoryAll four active platform workflows ran at the same final ctk-next PR 537 head:
Bandit also passed. Across the final PR check rollup, 13 checks succeeded and 17 were skipped. The skipped generic CI, security, labeling, and PR-policy checks are expected for this ctk-next ref-routing arrangement and should not be counted as test coverage. The four three-job platform workflows are the active matrix. Matrix and Pathfinder resultsAll platform jobs use CUDA 13.4.1, kitpick 011, and Python 3.13. Linux test-bearing jobs run in Ubuntu 24.04 containers with driver 615.62. Windows reports Python 3.13.14 with the native architecture for each runner. The full Pathfinder suite ran twice on every platform: once from the editable/source runbook installation and once from the built wheel in the isolated wheel-test environment.
The eight full-suite executions total 11,714 passed test instances and 48 skips. These totals reflect repeated source/wheel and platform coverage, not 11,714 unique tests. In addition, each WoA test-bearing job ran four dedicated strict real-artifact Pathfinder checks. Those eight additional test instances all passed. Each job then ran the semantic native cuDNN verification script successfully. The wheel jobs create a fresh virtual environment, install the locally built Pathfinder, bindings, and core wheels, move the checkout package trees aside, and fail if What ctk-next CI proves with real artifactsCUDA 13.4 and production Pathfinder loadingThe logs contain broad production-loader evidence against the real CUDA 13.4 kitpick:
This directly exercises the shared install-root/search refactoring under CUDA 13.4 and shows that the new cuDNN metadata and dependency handling do not regress established CUDA library discovery on either Windows architecture. Native Windows ARM64The WoA workflow is genuinely native, not x64 emulation with a patched platform string:
The full Pathfinder suite passes twice on this native host. Architecture detection, descriptor invariants, Windows path handling, source packaging, and the installed wheel therefore all execute successfully under real ARM64 Python and Windows filesystem semantics. Real WoA cuDNN, cuBLASLt, and NVRTC coverageArtifact provisioningThe successful source/runbook job had a cuDNN cache miss. It therefore exercised the complete cold path:
This matters because it proves discovery through the new descriptor-driven The CUDA 13.4 mini-toolkit contains real ARM64 The installed-wheel job restored the checksum-keyed cuDNN cache produced by the source job and re-ran the required-file and exact-DLL-set validation. The two jobs therefore cover both the cold download/extraction path and the subsequent cache-restore path. Dedicated strict Pathfinder checksBoth the source/runbook environment and the isolated installed-wheel environment set: Each then runs four targeted cases, all of which pass:
The dynamic-library cases run production Pathfinder in fresh subprocesses and use the native Windows loader. They do not mock the search context, dependency callback, loader, or DLL. The logged absolute paths are the intended artifacts: The cuDNN result cannot succeed merely by finding the shim. Its descriptor first loads required cuBLASLt and attempts optional NVRTC, registers the cuDNN directory for native dependency resolution, and then attaches the real ARM64 cuDNN DLL. Semantic native API smokeAfter the strict pytest cases, both environments run a stronger direct verification through the production Pathfinder APIs. It proves all of the following:
This goes beyond a top-level Source and installed-wheel repetitionThe entire strict and semantic sequence succeeds twice:
The installed-wheel job additionally runs the broader native package suites successfully. Its aggregate result, including the four targeted Pathfinder checks, is 6,265 passed, 414 skipped, two expected xfails, and zero failures, plus 29 successful subtests. This is useful corroboration that the real cuDNN path coexists with the complete native wheel stack rather than succeeding only in a narrowly constructed loader process. Combined public and ctk-next coverageThe two runs now cover complementary axes, including the intersection that was missing from the first ctk-next report:
The most accurate single-sentence combined summary is:
|
|
/ok to test 6dbbe8d |
|
/ok to test e5c91aa |
|
/ok to test 3319fc3 |
|
/ok to test 30a905ad0d4c4fcce383d673a562e15aadd0bd48 |
@rwgk, there was an error processing your request: See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/ |
|
/ok to test 30a905a |
|
/ok to test 23ba0f1 |
|
|
For completeness, companion comment for commit 955b5e7: The first version mixed installation roots with library directories: This follow-up gives each descriptor field one responsibility:
Root-based searches now share one root -> anchor -> DLL path, try every declared anchor before failing, and match the descriptor's exact DLL candidates so a cuDNN component DLL cannot be mistaken for |
|
For completeness, companion comment for commit ddaa78f: NVIDIA now publishes a CUDA 13 cuDNN 9.25 standalone archive for Windows ARM64 with
I intentionally did not add a The ctk-next PR 537 WoA job subsequently completed this validation using the real archive on native Windows ARM64. It verified strict Pathfinder discovery and loading, real cuBLASLt and NVRTC dependency resolution, ARM64 PE identity, cuDNN component-compatibility checks, and successful cudnnCreate/cudnnDestroy from both source and installed-wheel environments. This validates the loader and DLL-directory behavior beyond the simulated tests; it is not a numerical cuDNN workload. |
Summary
This PR adds
cuda-pathfindercoverage for cuDNN 9 and NCCL. It adds cuDNN dynamic-library and header discovery, and adds the missing NCCL header discovery alongside NCCL's existing dynamic-library and device-bitcode support. The explicitly declared layouts cover released wheels, Conda test environments, selected conventional standalone and system locations, and the verified Windows ARM64 cuDNN archive layout.Completing that goal required more than adding filenames to the catalogs. cuDNN has required and conditional runtime dependencies, Windows standalone installations need product-specific roots and dependent-DLL directory registration, and cuDNN's component DLL names make unconstrained wildcard matching unsafe. The implementation therefore adds reusable internal metadata for optional dependencies, product installation roots, architecture-specific Windows layouts, and descriptor-authorized DLL fallback globs.
Reviewing those paths also exposed several pre-existing inconsistencies: CUPTI's Windows names were stored in the opposite order from the catalog convention, Windows already-loaded detection did not use the same newest-first order as native loading, and the directory of an already-loaded DLL whose descriptor requires dependent-directory registration was registered only when a potentially unrelated early wheel/Conda search had succeeded. Those issues are fixed here. Finally, the PR contains a hook-local Cython 3.2.9 pin for an unrelated
cython-lintincompatibility that appeared when Cython 3.3.0 was released while this work was in CI.No new public Python API is introduced. Existing
cuda.pathfinderAPIs recognize additional products and installation layouts; the new fields and helpers are internal discovery metadata.Additional changes at a glance
CUDNN_PATHandNCCL_HOMEdiscovery for dynamic libraries and headers, plus standard Linux include layouts and versioned Windows Program Files cuDNN installs.cupti64_*.dll.cython-linthook to Cython 3.2.9 untilcython-lintsupports Cython 3.3.cuda-pathfinderv1.7.0 release and restores the missing 1.6.1 documentation-version switcher entry.Primary goal: cuDNN 9 discovery and NCCL header coverage
The original request was to add cuDNN discovery and complete the NCCL header surface in
cuda-pathfinder.libcudnn.so.9cudnn64_9.dllCUDNN_PATHcudnn.hcudnn.hincludelayoutlibnccl.so.2support; this PR addsNCCL_HOMElayoutsnccl.hThe manually created Pathfinder test environments are kept aligned with that scope: the Conda helpers provision cuDNN on Linux and Windows and NCCL on Linux.
cuDNN dynamic-library discovery
The cuDNN descriptor recognizes the cuDNN 9 ABI names and the released wheel layouts:
libcudnn.so.9undernvidia/cudnn/lib.cudnn64_9.dllundernvidia/cudnn/bin.The descriptor is deliberately major-version-specific. The test dependency groups therefore use
nvidia-cudnn-cu12>=9,<10andnvidia-cudnn-cu13>=9,<10; an installed cuDNN 8 or a future cuDNN 10 package must not be treated as evidence that this cuDNN 9 descriptor is valid.cuDNN and NCCL header discovery
The existing header APIs now locate:
CUDNN_PATH/include, standard Linux include directories, Linux's running multiarch include directory, and versioned Windows Program Files installs.NCCL_HOME/include,NCCL_HOME/build/include,/usr/include, and/usr/local/include.NCCL_HOME/build/includecovers a source or build tree, whileNCCL_HOME/includecovers an installed or extracted prefix. The Windows cuDNN system search uses versionedv9.*roots and numeric-aware ordering, sov9.10correctly sorts newer thanv9.9.cuDNN runtime dependencies
cuDNN does not fit the existing all-or-nothing dependency model:
dependencies=("cublasLt",)and fails if cuBLASLt cannot be loaded.optional_dependencies=("nvrtc",).optional_dependencieshas intentionally narrow semantics. A plainDynamicLibNotFoundErrormeans that an optional runtime component is absent and may be ignored. An unknown descriptor, a platform-availability error, or a dependency that was found but failed to load remains an error. In other words, optional means that absence is acceptable; it does not mean that a broken installation is acceptable.Required dependencies are loaded first, followed by optional dependencies. This ordering also ensures that cuBLASLt is available before Pathfinder attaches cuDNN.
Product installation roots and native layouts
cuDNN and NCCL are separately packaged products, so treating every layout as if it were rooted at
CUDA_PATHis insufficient. This PR adds descriptor-driven installation-root metadata and a shared search step for conventional product variables.CUDNN_PATHon Linuxlib,lib64includeCUDNN_PATHon Windows x64bin/x64,binincludeCUDNN_PATHon Windows ARM64bin/arm64includeNCCL_HOMEon Linuxlib,lib64,build/libinclude,build/includeNVIDIA/CUDNN/v9.*roots with the x64 cuDNN bin layoutsNVIDIA/CUDNN/v9.*/includeThe metadata separates installation roots from the relative layouts beneath those roots. It is also platform- and Windows-architecture-specific. Product variables and Program Files reuse the same root-traversal machinery while retaining source-specific layout metadata. This prevents a verified ARM64 archive layout from leaking into unverified Conda,
CUDA_PATH, or Program Files dynamic-library searches.Dynamic descriptors express those responsibilities separately through
install_root_env_vars_linux/windows,install_root_env_rel_dirs_linux/windows, andprogram_files_root_globs_windows. Header descriptors similarly declareproduct_root_env_vars, platform-specific system directories, and the Linux multiarch include policy. Search results retain the winning environment variable infound_via, so callers can distinguishCUDNN_PATHandNCCL_HOMEresults from wheel, Conda, CUDA, or system results.Root traversal tries every configured relative directory until it finds the requested primary library. Merely finding an earlier directory is not success; for example, an existing
bin/x64withoutcudnn64_9.dlldoes not prevent checking the next supported layout. Version-bearing roots and fallback filenames use shared numeric-aware ordering rather than lexical ordering.The NCCL ordering is intentional:
libis NCCL's canonical installed, archive, and build-directory layout.lib64supports legitimate GNUInstallDirs/CMake and RPM-style installed prefixes.build/libsupports an NCCL source checkout.There is no speculative
build/lib64: NCCL's in-tree build emitsbuild/lib. Ordinary distro multiarch installs remain the native loader's responsibility rather than introducing a broad architecture glob here.Search precedence and relationship to #1054
These product variables are conventional-layout fallbacks, not developer overrides.
CUDA_PATH/CUDA_HOME, Program Files, then the CTK-root canary.This is useful groundwork adjacent to #1054, but it does not implement that issue's highest-priority, fail-loud developer override. A custom NVSHMEM build still cannot override an installed NVIDIA wheel, so #1054 remains open for that separately designed capability.
Windows ARM64 cuDNN support
NVIDIA now publishes a CUDA 13 cuDNN 9.25 standalone Windows ARM64 archive with
bin/arm64/cudnn64_9.dlland architecture-independent headers underinclude. This PR supports that verified topology throughCUDNN_PATH.The scope is intentionally narrow:
win_arm64cuDNN wheel.CUDA_PATH, and Program Files dynamic-library layouts remain x64-only because equivalent ARM64 layouts were not verified.CUDNN_PATH/bin/arm64, so adding ARM64 support does not silently claim unsupported installation forms.This should be described as end-to-end Pathfinder discovery and loading support for the official standalone archive layout, not as exhaustive cuDNN product support on every Windows ARM64 packaging channel.
Already-loaded Windows DLL directory registration
Windows descriptors such as NVRTC, cuFFT, and cuDNN use
requires_add_dll_directorybecause the primary DLL may later resolve sibling or component DLLs from its own directory.The old already-loaded path gated this side effect on
have_abs_path. That name was misleading: the boolean did not say whether Pathfinder knew the resident module's absolute path. Windows already resolved that authoritative path withGetModuleFileNameW; the boolean only said whether the earlier wheel/Conda filesystem search had found some candidate.As a result, a recognized cuDNN DLL that was already resident from a late or external location such as
CUDNN_PATHor Program Files could be returned before those roots were searched and without registering the directory containing its lazy component DLLs.This PR removes the phase-derived boolean. If a recognized resident DLL's descriptor sets
requires_add_dll_directory, Pathfinder now registers the directory of the module that is actually loaded before loading dependencies, regardless of how that module entered the process. This changes runtime behavior only for the affected Windows already-loaded branch; newly loaded absolute paths already registered their directory.Safe Windows DLL matching and forward compatibility
Directory-based Windows discovery previously relied on a generic
<descriptor-name>*.dllpattern. That is unsafe for cuDNN becausecudnn*.dllalso matches component DLLs; loading a sidecar instead ofcudnn64_9.dllis incorrect. The same generic policy has sibling-name collisions such ascublas/cublasLt,cufft/cufftw,cusolver/cusolverMg,cusparse/cusparseLt, andcutensor/cutensorMg.Removing wildcard matching globally would create a different regression: CUPTI embeds a full release version in its Windows basename, so a future
cupti64_*.dllwould become undiscoverable until the catalog was updated.The final policy makes that tradeoff explicit:
windows_dll_fallback_globs.Only CUPTI currently opts in, with
windows_dll_fallback_globs=("cupti64_*.dll",). cuDNN remains exact-only. Known CUPTI names deliberately win over an unlisted fallback match.The descriptor-aware filesystem helper also applies PE-architecture validation for descriptors that request it, including site-packages candidates.
The wildcard is limited to filesystem-directory discovery. Native Windows name search and already-loaded detection require concrete catalogued basenames, so the exact table remains necessary and an unknown wildcard-only DLL that is already resident is outside this fallback's guarantees.
Consistent newest-first Windows DLL selection
The established Windows catalog convention stores versioned DLL names from oldest to newest. Runtime lookup reverses that order so the newest supported name wins.
Two pre-existing inconsistencies became visible during this work:
check_if_already_loaded_from_elsewhere()scanning the catalog forward. If multiple known generations were already resident, that path could return the oldest one even though normal loading preferred the newest.This PR stores CUPTI names in canonical oldest-to-newest order and centralizes Windows runtime traversal in
_candidate_dll_names(). Both native system loading and already-loaded detection now consume the same newest-to-oldest sequence. Normal single-version installations are unchanged; the fix makes multi-version behavior deterministic and consistent.Catalog, packaging, and test-environment correctness
The expanded product coverage exposed several smaller assumptions that needed to be made explicit:
nvidia-cudnn-cu12andnvidia-cudnn-cu13. Broadnvidia-cudnn-*matching incorrectly treated packages such asnvidia-cudnn-frontendandnvidia-cudnn-jit-*as proof thatnvidia/cudnn/include/cudnn.hexists.Coincidental
cython-lintinfrastructure fixCython 3.3.0 was released while a clean Windows pre-commit environment for this PR was being created. The repository freezes
cython-lintat v0.19.0, but that package declares an unboundedCython>=0.29.32dependency. The fresh hook therefore resolved Cython 3.3.0.Cython 3.3 changed its dictionary-comprehension AST, while released
cython-lintversions still accessed the removedDictComprehensionAppendNode.value_exprattribute. The hook crashed on an unchanged, valid dictionary comprehension incuda_core; it did not report a lint violation and the crash was unrelated to this PR's Pathfinder changes.The fix adds
Cython==3.2.9only to the isolatedcython-linthook'sadditional_dependencies, matching the repository's existing stub-generation hook pin. It does not constrain CUDA Python builds, runtime dependencies, wheels, or users. The pin can be removed after a releasedcython-lintsupports Cython 3.3 and the hook passes from a clean cache.Validation
Validation combines focused structural tests with real, non-mocked artifact loading:
CUDNN_PATH/CUDA_PATHresolution, ARM64 PE identity, the complete expected cuDNN component set,cudnnGetVersion, Graph/Ops/CNN/Adv compatibility checks, and successfulcudnnCreate/cudnnDestroyon a real GB10 GPU.This is comprehensive coverage of Pathfinder's discovery and loading contract for the verified layouts. It is not an exhaustive cuDNN or cuBLASLt numerical workload suite, nor does it claim every packaging channel or version/architecture combination.
The real-artifact runs establish the platform and artifact coverage described above. The later review-follow-up loader policies are additionally guarded by focused regression tests.