Skip to content

fix(pipeline): size definition properties for every field - #2687

Open
DeusData wants to merge 3 commits into
mainfrom
fix/def-props-buffer-every-field
Open

DeusData wants to merge 3 commits into
mainfrom
fix/def-props-buffer-every-field

Conversation

@DeusData

Copy link
Copy Markdown
Owner

What

build_def_props (in pass_definitions.c and its twin in pass_parallel.c) appends a definition's fields atomically: a field that does not fit is dropped whole. The buffer was sized only for the docstring, variants and variant_key, and every other field shared the remaining 2 KB.

On wide definitions (long signatures, decorator lists, parameter lists), that left no room for the three fields appended last:

  • fp: the MinHash fingerprint;
  • sp: the structural profile;
  • bt: the body tokens.

They were dropped silently. Such definitions got no SIMILAR_TO edge and fewer SEMANTICALLY_RELATED edges. Any newly added property pushed more of them out; #1692's route_path / route_method dropped fp on 12 long-docstring jellyfin controller actions and cost 34 SIMILAR_TO edges.

pd_props_buf / pp_props_buf (twins, kept in lockstep) now count:

  • the fixed metrics prefix;
  • the whole serialized form of every field build_def_props appends;
  • the reserve that the fp / sp / bt room checks use.

Definitions that fit keep the stack buffer. The field order is unchanged, so definitions that already fit produce byte-identical properties.

Consumers of these properties are pass_similarity, pass_semantic_edges and semantic search; MCP's search_graph already filters fp / sp / bt out of agent output. None of them depends on the fields being absent, so this is not an index format change.

Evidence

Corpus run against current main (354ccec), all PHP and C# corpus repos (PHP 27 par / 26 seq, C# 29 par / 29 seq; the heavy roslyn/runtime pair excluded):

language SIMILAR_TO (identical on par and seq) SEMANTICALLY_RELATED par seq
PHP +964 / -276 (net +688) +583 / -709 +502 / -532
C# +239 / -49 (net +190) +1,174 / -1,180 +1,244 / -1,218
  • No other edge type moves on either venue.
  • 324 of the 325 removed SIMILAR_TO edges are displacements under the per-node edge cap: one endpoint gained a better-ranked neighbour once more definitions carried a fingerprint. The remaining one fits the LSH candidate cap.
  • SEMANTICALLY_RELATED is TF-IDF with corpus-wide document frequencies plus a threshold. Recovered body tokens shift IDF across the repo, so some near-threshold pairs cross either way.

Test: parallel_wide_definition_keeps_fp_sp_bt_f14, both resolvers.

  • A Python function with 50 annotated parameters loses fp, sp and bt without the fix (props 1,904 B) and keeps all three with it (2,600 B).
  • A control with the same body, small, has all three either way (1,026 B, unchanged).

Suites run locally: parallel, pipeline (+ conventions / frozen / semantic-manifest), semantic, incremental, store_search: 819 passed. cppcheck 2.20 (the CI version) and clang-format are clean on the changed files.

A definition's properties buffer was sized for its docstring, variants and
variant_key only; every other field had to share the remaining 2 KB. Fields
that do not fit are dropped whole, so a long signature, decorator list or
parameter list silently squeezed out the MinHash fingerprint (fp), the
structural profile (sp) and the body tokens (bt) appended behind them. Those
definitions got no SIMILAR_TO edge and fewer SEMANTICALLY_RELATED edges, and
any newly added property (route_path, route_method) pushed more of them out.

pd_props_buf and pp_props_buf (twins, kept in lockstep) now count the fixed
metrics prefix plus the whole serialized form of every field
build_def_props appends, including the reserve the fp/sp/bt room checks
use. Definitions that fit keep the stack buffer.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
…every-field

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
…every-field

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant