fix(python): resolve names re-exported through package namespaces

[DeusData]

What

A Python package exposes names its submodules define. It re-exports them in its __init__.py, either explicitly or through a star import:

# django/db/models/__init__.py
from django.db.models.fields import *        # CharField, IntegerField, ...
# tests/common.py (home-assistant)
from pytest_homeassistant_custom_component.common import MockConfigEntry
# sqlalchemy/testing/__init__.py
from .config import combinations

A caller writes models.CharField(...), from tests.common import MockConfigEntry or @testing.combinations(...). The LSP looked for such a name among the package's own definitions only. The package defines none of them, so the call fell through to the name-based fallbacks. Those pick the first same-named QN under the package (import_map_suffix) or any same-named definition (suffix_match). For sqlalchemy, @testing.combinations resolved to _NullFixtureFunctions.combinations instead of testing.config.combinations.

Extraction has already recorded each module's ordered namespace facts since de1819e: own definitions, imports, star imports, __all__, shadowing assignments, deletions and unknown statements. This change consumes them:

• Records (pass_lsp_cross.c). cbm_pxc_collect_all_defs turns each complete namespace-fact stream into PyReexport / PyStarImport / PyAll records next to the module's definitions. An import-only __init__.py has no definitions, but still gets its records. The def array is sized for the records up front. Import targets come from the same import map the resolver binds imports with. An ambiguous or unresolved import gets no record, which leaves that name unknown.

• Fold (py_lsp.c, type_registry.c). The shared Python registry folds the records once: after finalize, before the seal. The result is a sorted export table, module.name to the definition QN, plus the sorted list of modules. Lookups are binary searches and never mutate it. There is no lookup in the registration loop and no tail scan. The fold is order-deterministic: modules are sorted, records stay in source order, and a cycle contributes what it has so far.

• Conservative states. A fact stream that is not complete publishes nothing for its module. Each of these removes the names it covers, so the old fallbacks keep handling them:

  • a shadowing assignment, a del, or a name a TYPE_CHECKING else-branch imports again;
  • an unknown statement (a top-level try: / with / call), which removes the whole namespace.

  A star whose source cannot be enumerated publishes nothing. A literal __all__ restricts a star. Without one, a star exports the public names.

• Consult (py_lsp.c). The export table is checked on a module-attribute miss and on from pkg import Y when Y is no definition of pkg. That covers calls, expression types and callable values. A re-export under another name (from .x import Orig as Renamed) joins as lsp_import_alias, with the written name as reason.

• Incremental. An incremental re-index of a closure of at most 8 files (CLOSURE_BUDGET_FLOOR_FILES) resolves with a per-file selection of definitions rather than the shared registry. That selection now follows the selected re-export records to the modules they name, transitively, so the re-exported definitions are in the per-file registry too (pxc_mark_reexport_targets). Both full-index drivers always build the shared Python registry and never reach this path.

No index-format change. Other languages are unchanged.

Known limit: sql.cast, sql.select, sql.and_ (sqlalchemy)

These names travel sqlalchemy.sql → sql.expression → sql._elements_constructors / _selectable_constructors. Two namespace barriers break the chain:

• lib/sqlalchemy/sql/__init__.py ends with a top-level call, __go(locals()) (line 156). A top-level call is an unknown EFFECT: it can rebind any name of the module. So the whole sqlalchemy.sql namespace is withdrawn, and its 108 re-exports are not published.
• lib/sqlalchemy/sql/expression.py has attribute docstrings after its imports (nullsfirst = nulls_first followed by """Synonym for ..."""). The namespace facts (de1819e) prove only plain one-line strings safe. A triple-quoted string statement is therefore an unknown EFFECT, and it withdraws the 149 re-exports before it.

sql.cast, sql.select and sql.and_ therefore still resolve by the old fallback (ColumnElement.cast, FunctionElement.select, BooleanClauseList.and_). This is deliberate: proving which top-level calls and string statements are harmless is a separate, narrower change and a follow-up. @testing.combinations is behind neither barrier and is fixed (12 edges).

Evidence

Measured at 12b3b84, merged with #2680 on main ea5c1f7, against ea5c1f7 + #2680. The two trees differ by this change only. Both are clean production builds. Corpus: 26 Python repos (manifest v1), macOS arm64, --repeat 2, every repeat byte-identical (noise 0). The later commits are the incremental test leg, the per-file selection above (not reached by a full index), and a merge of main.

type	base	this PR	+added	-removed	same edge, new attributes
CALLS	1,355,199	1,376,407	26,100	4,892	44,050
TESTS	510,618	530,988	22,909	2,539	0
CALL_REFERENCE	26,924	27,532	608	0	0
HTTP_CALLS	7,582	8,162	580	0	0
USAGE	1,672,624	1,672,077	0	547	0

SEMANTICALLY_RELATED (derived) moves +558/-416. Every other edge type is identical.

CALLS.

• Every added edge is lsp_*:

  strategy	added
  lsp_method	20,440
  lsp_callable_alias	3,248
  lsp_constructor	1,569
  lsp_operator_dunder	738
  lsp_module_attr	77
  lsp_import_alias	20

• 44,050 existing edges keep their target and move from a name guess to an lsp_* strategy.

• Of the 4,892 removed edges, 4,882 are same-line retargets away from weak guesses: suffix_match 4,720 and import_map_suffix 102.

Judged precision: seeded samples, read with their source snippets.

• home-assistant CALLS, 20/20. MockConfigEntry is re-exported by tests.common from pytest-homeassistant-custom-component, which alone accounts for 9,801 .add_to_hass calls.
• CALLS in the other repos, 20/20. Examples: django reverse / Template / forms.*, sympy functions, scrapy LinkExtractor to LxmlLinkExtractor, pandas register_matplotlib_converters.
• TESTS, 10/10.

Other types:

• TESTS -2,539: these follow the CALLS retargets. 2,538 of the affected tests gain a TESTS edge to the correct target.
• USAGE -547:
  • 320 come back as CALL_REFERENCE with the same source and target.
  • 161 are scikit-learn array_api_compat modules (arange = get_xp(cp)(_aliases.arange)). Their reference to _aliases.arange had been bound to the module's own variable. It is now a CALL_REFERENCE to common._aliases.arange.
  • The other 66 were cross-module name guesses, for example a django forms test bound to AbstractBaseUser.clean.

Tests

• pipeline_python_package_reexports_resolve_by_lsp. Shapes:

  • explicit re-export, star re-export and __all__-listed name, both as a module attribute and through from pkg import;
  • a re-export under another name, a chain over three modules, and an import cycle;
  • a django-shaped db.models star package, a variable typed through a re-export, and a TYPE_CHECKING re-export whose else-branch binds other names only.

  Controls that must not resolve by the LSP: a name __all__ leaves out, a private name under a star, a re-export the module shadows later, a call before a top-level try: barrier, and a TYPE_CHECKING re-export whose else-branch binds the same name again (a compat try import). Four modes: sequential driver, parallel driver, and each followed by an incremental re-index of the caller alone. RED without the change, by name for the call that fails.

• python_namespace_facts_type_checking_alternatives (extraction): a name the else-branch of a TYPE_CHECKING block imports again becomes a SHADOW fact with a valid reason, and a star or unreadable alternative makes the namespace unknown.

• Affected suites on the merged tree pass (4,099 tests): complexity, extraction, extraction_imports, extraction_inheritance, grammar_imports, registry, pipeline, pipeline_semantic_manifest_repro, cross_repo, py_lsp, py_lsp_stress, py_lsp_scale, py_lsp_bench, the go / c / php / cs / ts / java / kotlin / rust LSP suites, lsp_resolution_probe, edge_imports, edge_structural, edge_types_probe, convergence_probe, matrix_known_classes, parallel, incremental, integration, index_resilience.

• Lint: clang-format clean, the memory-core linter shows no growth, and cppcheck 2.20 reports nothing new. py_lsp.c has the same findings as on main.