Repository navigation
fix(python): extract defs nested in a function body and resolve bare names lexically (#1548) - #2686
Merged
Merged
Conversation
Why
The Python def walk stopped at every function_definition: it extracted
the def and never looked at its body. Every def or class declared inside
a function -- closures, decorator wrappers, the route handlers of a Flask
or FastAPI app factory (`@app.route` inside create_app), local classes and
their methods -- was absent from the graph. Blocks at module level (if /
try / for / with) were transparent, so only function bodies were blind.
The calls inside those defs were not re-attributed either: the call-scope
walk named a nested def by its class or module only (`pkg.mod.nested`, a
QN no node had), so the edges fell back to the file node, and inbound
traces of the called function silently lost the caller. A def nested in a
method got the method's own QN (`Tree.visit` for the `def visit` inside
method Tree.visit), so its calls, recursion included, landed on the
method. On an app-factory codebase this removes the whole HTTP layer.
Fix
- Extraction (extract_defs.c): after a Python def is extracted, its body
is walked with the def's QN as the enclosing scope; methods' bodies are
walked under the method QN. A nested def or class is qualified by the
def it sits in -- pkg.mod.create_app.index, pkg.mod.Tree.visit.visit,
pkg.mod.create_app.Settings.value -- the QN the Python LSP already gives
the calls inside it (py_process_function). Decorated nested defs keep
their decorators, so their route_path mints the Route and HANDLES.
Lambdas and comprehensions hold no statements and stay as they are.
- Call scope (extract_unified.c): for Python the nearest named scope, a
def or a class, prefixes a nested def or class, so every call, usage and
env read inside a nested def carries the nested def's QN.
- A def nested in a function is a local of it (Python LEGB), so it must
not become a global name: a Python symbol whose QN extends a
Function/Method node is registered for exact lookups only, never in the
short-name index (registry.c cbm_registry_add_for_language, used by the
full, parallel and incremental registration sites alike). Without this,
django's per-test `class MyModelAdmin` and every decorator's `def
wrapper` became unique_name / suffix_match targets for unrelated calls.
The incremental seed derives each stored file's language the way
discovery does (name, then shebang), so it keeps the same symbols out.
The import symbol-name fallback skips such locals too (pass_pkgmap.c),
and the Python LSP no longer binds a def-local class module-wide by its
short name (py_lsp.c).
- A bare call now resolves lexically first (registry.c
cbm_registry_resolve_lexical, strategy lexical_scope): the def or class
bound in the nearest enclosing function, skipping class bodies, stopping
at the module. The sequential and parallel call passes use it
identically, and so does the handler lookup of a call-registered route:
`app.add_url_rule("/ping", "ping", ping)` in create_app reaches the
local ping.
Proof (real code; every node and edge diffed, main 1f9b4db vs this)
- pallets/flask examples/tutorial (flaskr, create_app factory): the
nested `hello` handler is now create_app.hello with HANDLES to
/hello; 4 nested defs/classes added, 0 nodes and 0 edges removed.
- pallets/flask d73fa1c (tests declare handlers inside test functions):
Function 532 -> 1068, Method 318 -> 381, Class 130 -> 209. The +678
equals the 678 unique nested def/class QNs Python's own ast finds.
Handler HANDLES (decorators and add_url_rule) 32 -> 296. 0 nodes
removed, 0 duplicate QNs.
The 94 removed CALLS: 90 moved from the file node onto the nested def
that makes the call, 4 replaced by the lexically bound local.
- django fb11376 (local, 2,927 .py files): nodes 55,483 -> 60,052,
Function 1,992 -> 3,027, Method 28,850 -> 29,491, Class 8,187 ->
11,073. 0 nodes removed and 0 duplicate QNs. Every added
Function/Method/Class sits under a Function/Method of the same file,
and they match ast's 4,567 nested QNs except for 5 methods declared in
an `if` inside a local class body (class-body `if` is not walked for
any class). CALLS 62,430 -> 63,815. Every added CALLS edge has a nested
def at one end: 1,015 from nested defs, 1,219 lexical_scope into a
local, and none from outside the local's function. Of the 849 removed:
507 moved from the file node onto the nested def, 334 replaced by the
local the name binds, and 8 were false on main, all checked (e.g. the
recursive `get_actions` nested in call_command had been suffix-matched
to ModelAdmin.get_actions). HANDLES unchanged (135).
- Samples checked against the source:
AbstractBaseUser.check_password.setter -> set_password / save;
View.as_view.view -> setup / dispatch;
Apps.lazy_model_operation -> lazy_model_operation.apply_next_model
(line 426);
InstrumentedASGIRequest.META -> the local InstrumentedDict (line 279);
flask test_provide_automatic_options_arg_disable.index -> GET /.
Tests
- extraction: extract_python_nested_def_qualified_by_enclosing_def,
_calls_attributed_to_it, _decorated_nested_route_handler,
_nested_class, _nested_def_named_like_its_method,
_lambda_and_comprehension_stay_in_enclosing_def (control),
_nested_def_env_access_attributed_to_it (all _issue1548).
- pipeline: pipeline_python_app_factory_routes, _nested_def_is_function_
local, _local_classes_stay_in_their_def, _import_never_binds_a_local_def,
_nested_defs_parallel (>= 50 files, CBM_WORKERS=4),
_nested_def_local_after_incremental (all _issue1548).
- registry: registry_python_function_local_is_exact_only,
registry_resolve_lexical_walks_enclosing_defs (both _issue1548).
The extraction and pipeline tests were RED on main 3/3 for the asserted
reason and are green with the fix; the registry tests cover the new
registry API. Each production piece was reverted on its own, and each
revert turns at least one of these tests RED.
Fixes #1548
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
(cherry picked from commit 62fda43fd66200879db62a58903a32dab0dd1a20)
…after it Re-port onto current main (agreed with lane L7): - The Python LSP binds every def/class a function body defines as a local before walking the body (LEGB): an undecorated def as an exact callable alias to its nested QN, anything else as a hard shadow, so no call falls through to a same-named module symbol. Defs inside the body's compound statements count too, and their bodies are now walked. - Classes declared in a function are walked under the function's QN (py_process_class_in), so super()/self calls in their methods resolve. - cbm_registry_resolve_lexical moves AFTER the LSP override: both resolvers call one entry point, cbm_registry_resolve_call_scoped, at the same place (before the language-aware chain). It never uses the per-file resolve cache. - Main's cbm_registry_add_lang carries the def-local exclusion, so every registration site (full, parallel, incremental seed) applies it; the separate cbm_registry_add_for_language and the second incremental language derivation are dropped. - The pass_pkgmap.c def-local import skip is its own continue. Tests: local-vs-module shadowing in both orders, a decorated local, and a function-local class's super()/self calls, sequential and parallel. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Whitespace only; the token stream is unchanged. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A package __init__.py can define a function whose QN equals a submodule's (poetry: layouts/__init__.py `def layout` is the QN prefix of every symbol in layouts/layout.py). The prefix walks that recognise a def declared inside a function body accepted that other file's function as the enclosing def, so class Layout and its methods read as locals of layout(): they left the short-name index and the import symbol fallback, and the answer depended on which file registered first (discovery order on a full index, node-id order on an incremental one). Every walk now stops at the symbol's own module: - registration (pass_definitions.c, pass_parallel.c, and the incremental seed) passes the defining file's symbol prefix length, cbm_pipeline_fqn_symbol_prefix_len, computed without allocating, to cbm_registry_add_lang_in_module; cbm_registry_add_lang keeps its signature and does no local detection; - cbm_registry_resolve_lexical stops at the caller's module QN; - the import fallback's local test only counts an enclosing def in the same file. Also moves the #1548 registry tests out of the test they had been inserted into. Tests: registry_python_local_bound_at_own_file_issue1548 (both registration orders), fqn_symbol_prefix_len_matches_fqn_compute, pipeline_python_init_function_shares_submodule_qn{,_parallel}_issue1548 (src/ layout: the import and calls onto Layout survive, then again after an incremental re-index). Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
…the LSP At a compound statement's join the Python LSP invalidates what the branch bound, so the local def under if True: is resolved by the registry's lexical_scope step; pin that producer instead of an lsp_ strategy. Found at the branch's first build. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
…ategy Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
pylsp_nested_and_local_classbody_calls_join_extracted_caller asserted that a class declared in a function body is NOT a def, which is the behaviour #1548 fixes. Their class-body calls still join the enclosing function; that part is unchanged. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Since #1548 a def declared inside a Python test function is extracted, so its calls start at the helper instead of the test FILE node, whose name passed the test-name check. 4,641 of 6,315 TESTS edges lost on the corpus were this shape. TESTS for a call from such a helper is now attributed to the nearest enclosing test function of the same file; a helper nested in a non-test function stays without TESTS, like a module-level helper. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Conflict in pass_definitions.c: process_def gained main's link_parents and this branch's module_prefix_len; both kept. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
cppcheck 2.20 variableScope on defs_before; same behaviour. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
…the Smart CI selection fix #2684) Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
This was referenced Oct 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Python functions declare functions and classes inside their bodies: closures, app-factory route
handlers (
def create_app(): ... @app.route("/") def index(): ...), decorator wrappers, test-localclasses. They were never extracted, so their calls hung on the enclosing function or the FILE node, their
route handlers vanished (no HANDLES), and a bare call to a local name was bound by a name guess to a
same-named definition somewhere else in the project.
extract_defs.c,extract_unified.c): a def or class nested in a Python function body,including inside its compound statements, is extracted with the QN
<enclosing function>.<name>.py_lsp.c): a function's nested defs and classes are bound as locals of that function(LEGB); nested defs are walked with their own scope; a class declared in a function is walked under the
function's QN.
registry.c): a def declared inside a function body of its own file is registered for exactlookups only and kept out of the short-name index, so no weak strategy offers it to code elsewhere. The
check is bounded at the symbol's own file (
cbm_pipeline_fqn_symbol_prefix_len): a package__init__.pyfunction can share a submodule's QN (poetry:layouts/__init__.pydef layoutvslayouts/layout.py). Every registration path (sequential, parallel, incremental seed) goes throughcbm_registry_add_lang_in_module;cbm_registry_add_langkeeps its signature and does no localdetection.
lexical_scopestep runs after the LSP in both call resolvers (one entry point,cbm_registry_resolve_call_scoped) and for route-handler arguments: a bare name resolves to the defbound in the nearest enclosing function, stopping at the caller's module; exact lookups only.
pass_pkgmap.c): the symbol-name fallback never binds an import to a function-localdef of the same file.
pass_tests.c): a call from a def nested inside a Python test function is attributed to thenearest enclosing test function of the same file (before, such calls hung on the test FILE node).
No index-format change.
Evidence
Base origin/main 9f85f10 vs this head, 26 Python repos (manifest v1), production builds, macOS arm64,
--repeat 2, every repeat byte-identical.Sampled and read, per stratum:
callbacks): correct in the samples (test-local classes inherit
BaseModel, djangoModel,Entity,BaseOperator; local callbacks are referenced). One weak case: a base that is a function-local variable(
Base = declarative_base()) still resolves by name.enclosing function; the removed CALLS / USAGE / HTTP_CALLS from
.pyfile sources reappear from thenested defs with the same targets.
lexical_scope+2,731,lsp_*through the LSP's local binding); the removedunique_name/suffix_matchedges were wrongguesses for those names (test-local
C(),Model(...),SelectApp(),post_image()bound tounrelated same-named definitions).
homepage/endpointwins over a same-named module function or method.removed = 1,374 from function sources (1,367 of them wrong targets that are gone because the call now
binds the test-local def) + 4,941 from test FILE nodes, which existed only because a call inside a
test's nested helper hung on the file. With the enclosing-test-function attribution (this head), 4,128
of those 4,941 (83.5%) are TESTS again, now from the test function; net TESTS +5,972 vs base. Residual:
813 (0.16% of 505,546) are calls from helpers nested in NON-test functions of test files (fixtures,
setup helpers, snapshot makers, private test-class helpers); like a module-level fixture or helper, such
code is not a TESTS source.
inbound CALLS 6 / IMPORTS 2 / INHERITS 1 / USAGE 3 are identical to base; without the own-file bound the
2 IMPORTS were lost.
incridentical on every edge type;incr-n5/incr-seq-n5show+39 DEFINES from edited
__init__.pyFile nodes to the synthetic<python-builtins>stubs, and baseshows the identical +39: a pre-existing incremental defect, not changed here and tracked separately
(repro: re-index after editing a package
__init__.pyadds DEFINES from that File node to the 13builtins stubs, 13 x 3 edited files on poetry; a full index has none).
Exposed approximation (documented, not changed here)
route node, and route nodes are keyed by method + path globally, not per app. The newly extracted
test-local handlers (e.g. a camera test's
stream_handleron its own test server) now pair with othertests' HTTP calls to the same path; 10 of 10 sampled are links between separate apps. A follow-up for the
route model: pair caller and handler only within the same app / registration scope.
Tests
local classes, imports never binding a local def, local-first shadowing in both orders, nested classes
in defs), all on the sequential and parallel resolvers.
registry_python_local_bound_at_own_file_issue1548(both registration orders) andpipeline_python_init_function_shares_submodule_qn{,_parallel}_issue1548(src/ layout: the import andcalls onto
Layoutsurvive a full index and an incremental re-index): RED with the own-file boundremoved.
pipeline_python_nested_helper_keeps_tests{,_parallel}_issue1548: RED with the TESTS attributionreverted.
pylsp_nested_and_local_classbody_calls_join_extracted_callerasserted that a class declared in afunction body is not a def -- the behaviour Python: defs nested inside a function are never extracted — app-factory (create_app) codebases lose their entire HTTP layer, and inbound traces silently return zero callers #1548 fixes; it now asserts the local classes are defs.
Its call-site expectations (class-body calls join the enclosing function) are unchanged.
if True:is bound bylexical_scope, not by the LSP: at acompound statement's join the LSP invalidates what the branch bound. The asserted target is unchanged
(the local def, never the module def); a later LSP walk of compound statements may move it to an
lsp_*strategy deliberately.lsp_resolution_probe, incremental) pass.
Closes #1548.