Skip to content

fix(python): extract defs nested in a function body and resolve bare names lexically (#1548) - #2686

Merged
DeusData merged 11 commits into
mainfrom
fix/issue-1548-python-nested-defs-v2
Oct 18, 2026
Merged

DeusData merged 11 commits into
mainfrom
fix/issue-1548-python-nested-defs-v2

Conversation

@DeusData

Copy link
Copy Markdown
Owner

What

Python functions declare functions and classes inside their bodies: closures, app-factory route
handlers (def create_app(): ... @app.route("/") def index(): ...), decorator wrappers, test-local
classes. They were never extracted, so their calls hung on the enclosing function or the FILE node, their
route handlers vanished (no HANDLES), and a bare call to a local name was bound by a name guess to a
same-named definition somewhere else in the project.

  • Extraction (extract_defs.c, extract_unified.c): a def or class nested in a Python function body,
    including inside its compound statements, is extracted with the QN <enclosing function>.<name>.
  • Python LSP (py_lsp.c): a function's nested defs and classes are bound as locals of that function
    (LEGB); nested defs are walked with their own scope; a class declared in a function is walked under the
    function's QN.
  • Registry (registry.c): a def declared inside a function body of its own file is registered for exact
    lookups only and kept out of the short-name index, so no weak strategy offers it to code elsewhere. The
    check is bounded at the symbol's own file (cbm_pipeline_fqn_symbol_prefix_len): a package
    __init__.py function can share a submodule's QN (poetry: layouts/__init__.py def layout vs
    layouts/layout.py). Every registration path (sequential, parallel, incremental seed) goes through
    cbm_registry_add_lang_in_module; cbm_registry_add_lang keeps its signature and does no local
    detection.
  • Resolution: a new lexical_scope step runs after the LSP in both call resolvers (one entry point,
    cbm_registry_resolve_call_scoped) and for route-handler arguments: a bare name resolves to the def
    bound in the nearest enclosing function, stopping at the caller's module; exact lookups only.
  • Import fallback (pass_pkgmap.c): the symbol-name fallback never binds an import to a function-local
    def of the same file.
  • TESTS (pass_tests.c): a call from a def nested inside a Python test function is attributed to the
    nearest enclosing test function of the same file (before, such calls hung on the test FILE node).

No index-format change.

Evidence

Base origin/main 9f85f10 vs this head, 26 Python repos (manifest v1), production builds, macOS arm64,
--repeat 2, every repeat byte-identical.

type base this PR +added -removed
DEFINES 2,461,759 2,515,174 53,415 0
CALLS 1,326,684 1,354,959 47,539 19,264
USAGE 1,654,225 1,672,257 34,974 16,942
CALL_REFERENCE 14,970 26,924 11,965 11
DECORATES 233,412 243,846 10,434 0
INHERITS 68,723 78,598 9,875 0
DEFINES_METHOD 389,610 397,865 8,255 0
OVERRIDE 79,984 82,497 2,513 0
TESTS 505,546 511,518 12,284 6,312
HANDLES 1,740 2,328 598 10
DATA_FLOWS 0 80 80 0

Sampled and read, per stratum:

  • New definitions and their own edges (DEFINES, DECORATES, INHERITS to their bases, CALL_REFERENCE as
    callbacks): correct in the samples (test-local classes inherit BaseModel, django Model, Entity,
    BaseOperator; local callbacks are referenced). One weak case: a base that is a function-local variable
    (Base = declarative_base()) still resolves by name.
  • Re-attribution: calls inside nested defs now start at the nested def instead of the FILE node or the
    enclosing function; the removed CALLS / USAGE / HTTP_CALLS from .py file sources reappear from the
    nested defs with the same targets.
  • Local-first binding: bare calls to a locally defined name bind the local def (lexical_scope +2,731,
    lsp_* through the LSP's local binding); the removed unique_name / suffix_match edges were wrong
    guesses for those names (test-local C(), Model(...), SelectApp(), post_image() bound to
    unrelated same-named definitions).
  • HANDLES +598: route handlers declared inside app factories and test functions; a local homepage /
    endpoint wins over a same-named module function or method.
  • TESTS, classified on the extraction change alone (the commit before the TESTS attribution): 6,315
    removed = 1,374 from function sources (1,367 of them wrong targets that are gone because the call now
    binds the test-local def) + 4,941 from test FILE nodes, which existed only because a call inside a
    test's nested helper hung on the file. With the enclosing-test-function attribution (this head), 4,128
    of those 4,941 (83.5%) are TESTS again, now from the test function; net TESTS +5,972 vs base. Residual:
    813 (0.16% of 505,546) are calls from helpers nested in NON-test functions of test files (fixtures,
    setup helpers, snapshot makers, private test-class helpers); like a module-level fixture or helper, such
    code is not a TESTS source.
  • L7's collision check (poetry, the 11 defs whose nearest callable QN prefix lives in another file): their
    inbound CALLS 6 / IMPORTS 2 / INHERITS 1 / USAGE 3 are identical to base; without the own-file bound the
    2 IMPORTS were lost.
  • Full == incremental (poetry): plain incr identical on every edge type; incr-n5 / incr-seq-n5 show
    +39 DEFINES from edited __init__.py File nodes to the synthetic <python-builtins> stubs, and base
    shows the identical +39: a pre-existing incremental defect, not changed here and tracked separately
    (repro: re-index after editing a package __init__.py adds DEFINES from that File node to the 13
    builtins stubs, 13 x 3 edited files on poetry; a full index has none).

Exposed approximation (documented, not changed here)

  • DATA_FLOWS +80 (base 0): DATA_FLOWS pairs any HTTP_CALLS caller with any HANDLES handler of the same
    route node, and route nodes are keyed by method + path globally, not per app. The newly extracted
    test-local handlers (e.g. a camera test's stream_handler on its own test server) now pair with other
    tests' HTTP calls to the same path; 10 of 10 sampled are links between separate apps. A follow-up for the
    route model: pair caller and handler only within the same app / registration scope.

Tests

Closes #1548.

Why
The Python def walk stopped at every function_definition: it extracted
the def and never looked at its body. Every def or class declared inside
a function -- closures, decorator wrappers, the route handlers of a Flask
or FastAPI app factory (`@app.route` inside create_app), local classes and
their methods -- was absent from the graph. Blocks at module level (if /
try / for / with) were transparent, so only function bodies were blind.
The calls inside those defs were not re-attributed either: the call-scope
walk named a nested def by its class or module only (`pkg.mod.nested`, a
QN no node had), so the edges fell back to the file node, and inbound
traces of the called function silently lost the caller. A def nested in a
method got the method's own QN (`Tree.visit` for the `def visit` inside
method Tree.visit), so its calls, recursion included, landed on the
method. On an app-factory codebase this removes the whole HTTP layer.

Fix
- Extraction (extract_defs.c): after a Python def is extracted, its body
  is walked with the def's QN as the enclosing scope; methods' bodies are
  walked under the method QN. A nested def or class is qualified by the
  def it sits in -- pkg.mod.create_app.index, pkg.mod.Tree.visit.visit,
  pkg.mod.create_app.Settings.value -- the QN the Python LSP already gives
  the calls inside it (py_process_function). Decorated nested defs keep
  their decorators, so their route_path mints the Route and HANDLES.
  Lambdas and comprehensions hold no statements and stay as they are.
- Call scope (extract_unified.c): for Python the nearest named scope, a
  def or a class, prefixes a nested def or class, so every call, usage and
  env read inside a nested def carries the nested def's QN.
- A def nested in a function is a local of it (Python LEGB), so it must
  not become a global name: a Python symbol whose QN extends a
  Function/Method node is registered for exact lookups only, never in the
  short-name index (registry.c cbm_registry_add_for_language, used by the
  full, parallel and incremental registration sites alike). Without this,
  django's per-test `class MyModelAdmin` and every decorator's `def
  wrapper` became unique_name / suffix_match targets for unrelated calls.
  The incremental seed derives each stored file's language the way
  discovery does (name, then shebang), so it keeps the same symbols out.
  The import symbol-name fallback skips such locals too (pass_pkgmap.c),
  and the Python LSP no longer binds a def-local class module-wide by its
  short name (py_lsp.c).
- A bare call now resolves lexically first (registry.c
  cbm_registry_resolve_lexical, strategy lexical_scope): the def or class
  bound in the nearest enclosing function, skipping class bodies, stopping
  at the module. The sequential and parallel call passes use it
  identically, and so does the handler lookup of a call-registered route:
  `app.add_url_rule("/ping", "ping", ping)` in create_app reaches the
  local ping.

Proof (real code; every node and edge diffed, main 1f9b4db vs this)
- pallets/flask examples/tutorial (flaskr, create_app factory): the
  nested `hello` handler is now create_app.hello with HANDLES to
  /hello; 4 nested defs/classes added, 0 nodes and 0 edges removed.
- pallets/flask d73fa1c (tests declare handlers inside test functions):
  Function 532 -> 1068, Method 318 -> 381, Class 130 -> 209. The +678
  equals the 678 unique nested def/class QNs Python's own ast finds.
  Handler HANDLES (decorators and add_url_rule) 32 -> 296. 0 nodes
  removed, 0 duplicate QNs.
  The 94 removed CALLS: 90 moved from the file node onto the nested def
  that makes the call, 4 replaced by the lexically bound local.
- django fb11376 (local, 2,927 .py files): nodes 55,483 -> 60,052,
  Function 1,992 -> 3,027, Method 28,850 -> 29,491, Class 8,187 ->
  11,073. 0 nodes removed and 0 duplicate QNs. Every added
  Function/Method/Class sits under a Function/Method of the same file,
  and they match ast's 4,567 nested QNs except for 5 methods declared in
  an `if` inside a local class body (class-body `if` is not walked for
  any class). CALLS 62,430 -> 63,815. Every added CALLS edge has a nested
  def at one end: 1,015 from nested defs, 1,219 lexical_scope into a
  local, and none from outside the local's function. Of the 849 removed:
  507 moved from the file node onto the nested def, 334 replaced by the
  local the name binds, and 8 were false on main, all checked (e.g. the
  recursive `get_actions` nested in call_command had been suffix-matched
  to ModelAdmin.get_actions). HANDLES unchanged (135).
- Samples checked against the source:
  AbstractBaseUser.check_password.setter -> set_password / save;
  View.as_view.view -> setup / dispatch;
  Apps.lazy_model_operation -> lazy_model_operation.apply_next_model
  (line 426);
  InstrumentedASGIRequest.META -> the local InstrumentedDict (line 279);
  flask test_provide_automatic_options_arg_disable.index -> GET /.

Tests
- extraction: extract_python_nested_def_qualified_by_enclosing_def,
  _calls_attributed_to_it, _decorated_nested_route_handler,
  _nested_class, _nested_def_named_like_its_method,
  _lambda_and_comprehension_stay_in_enclosing_def (control),
  _nested_def_env_access_attributed_to_it (all _issue1548).
- pipeline: pipeline_python_app_factory_routes, _nested_def_is_function_
  local, _local_classes_stay_in_their_def, _import_never_binds_a_local_def,
  _nested_defs_parallel (>= 50 files, CBM_WORKERS=4),
  _nested_def_local_after_incremental (all _issue1548).
- registry: registry_python_function_local_is_exact_only,
  registry_resolve_lexical_walks_enclosing_defs (both _issue1548).
The extraction and pipeline tests were RED on main 3/3 for the asserted
reason and are green with the fix; the registry tests cover the new
registry API. Each production piece was reverted on its own, and each
revert turns at least one of these tests RED.

Fixes #1548
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
(cherry picked from commit 62fda43fd66200879db62a58903a32dab0dd1a20)
…after it

Re-port onto current main (agreed with lane L7):
- The Python LSP binds every def/class a function body defines as a local
  before walking the body (LEGB): an undecorated def as an exact callable
  alias to its nested QN, anything else as a hard shadow, so no call falls
  through to a same-named module symbol. Defs inside the body's compound
  statements count too, and their bodies are now walked.
- Classes declared in a function are walked under the function's QN
  (py_process_class_in), so super()/self calls in their methods resolve.
- cbm_registry_resolve_lexical moves AFTER the LSP override: both resolvers
  call one entry point, cbm_registry_resolve_call_scoped, at the same place
  (before the language-aware chain). It never uses the per-file resolve cache.
- Main's cbm_registry_add_lang carries the def-local exclusion, so every
  registration site (full, parallel, incremental seed) applies it; the
  separate cbm_registry_add_for_language and the second incremental language
  derivation are dropped.
- The pass_pkgmap.c def-local import skip is its own continue.

Tests: local-vs-module shadowing in both orders, a decorated local, and a
function-local class's super()/self calls, sequential and parallel.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Whitespace only; the token stream is unchanged.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A package __init__.py can define a function whose QN equals a submodule's
(poetry: layouts/__init__.py `def layout` is the QN prefix of every symbol in
layouts/layout.py). The prefix walks that recognise a def declared inside a
function body accepted that other file's function as the enclosing def, so
class Layout and its methods read as locals of layout(): they left the
short-name index and the import symbol fallback, and the answer depended on
which file registered first (discovery order on a full index, node-id order on
an incremental one).

Every walk now stops at the symbol's own module:
- registration (pass_definitions.c, pass_parallel.c, and the incremental seed)
  passes the defining file's symbol prefix length,
  cbm_pipeline_fqn_symbol_prefix_len, computed without allocating, to
  cbm_registry_add_lang_in_module; cbm_registry_add_lang keeps its signature and
  does no local detection;
- cbm_registry_resolve_lexical stops at the caller's module QN;
- the import fallback's local test only counts an enclosing def in the same file.

Also moves the #1548 registry tests out of the test they had been inserted into.

Tests: registry_python_local_bound_at_own_file_issue1548 (both registration
orders), fqn_symbol_prefix_len_matches_fqn_compute,
pipeline_python_init_function_shares_submodule_qn{,_parallel}_issue1548 (src/
layout: the import and calls onto Layout survive, then again after an
incremental re-index).

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
…the LSP

At a compound statement's join the Python LSP invalidates what the branch bound, so the local def under if True: is resolved by the registry's lexical_scope step; pin that producer instead of an lsp_ strategy. Found at the branch's first build.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
…ategy

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
pylsp_nested_and_local_classbody_calls_join_extracted_caller asserted that a class declared in a function body is NOT a def, which is the behaviour #1548 fixes. Their class-body calls still join the enclosing function; that part is unchanged.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Since #1548 a def declared inside a Python test function is extracted, so its calls start at the helper instead of the test FILE node, whose name passed the test-name check. 4,641 of 6,315 TESTS edges lost on the corpus were this shape. TESTS for a call from such a helper is now attributed to the nearest enclosing test function of the same file; a helper nested in a non-test function stays without TESTS, like a module-level helper.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Conflict in pass_definitions.c: process_def gained main's link_parents and this branch's module_prefix_len; both kept.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
cppcheck 2.20 variableScope on defs_before; same behaviour.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
…the Smart CI selection fix #2684)

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant