Research: TikNib-inspired interpretable feature-delta triage for OwnIR regressions (bounded spike) #439

Classification

• Type: bounded research / diagnostic-triage evaluation spike
• Priority: P2 / backlog; no production integration by default
• Area: Roslyn → OwnIR/CFG/call facts → ownership summaries → diagnostic comparison
• Prior art: SoftSec-KAIST/TikNib; Kim et al., Revisiting Binary Code Similarity Analysis using Interpretable Feature Engineering and Lessons Learned (IEEE TSE, online 2022 / volume 49, 2023); BinKit
• Related Own.NET: Research: semantic mutation testing harness for Own.NET analyzer coverage (bounded pilot) #405 (validated defect mutants and negative controls), Research: refactoring-aware metamorphic regression testing for ownership analysis (RefactoringMiner prior art) #432 (refactoring/semantic correspondence), Research: semantic subgroup analysis for Own.NET diagnostic quality (bounded pilot) #427 (semantic subgroup diagnostic-quality reporting), Research: SootUp-inspired OwnIR/ICFG semantic-fidelity and incomplete-call audit (bounded spike) #430 (source→OwnIR/ICFG fidelity), P-022 diagnostics: normalized AnalysisTrace and first-divergence minimizer for Python↔Rust shadow mode #269 (first divergence), Research: evaluate Bifrost for C# code intelligence, agent navigation, and evidence contracts #426 (evidence/completeness)
• Default decision: do not embed/port TikNib, adopt IDA, add ML, change ownership semantics, or introduce a production similarity score.

Motivation and transferable lesson

TikNib is a binary-function similarity research tool, not a C# ownership analyzer. Its reusable idea is to compare interpretable structural features under controlled compilation changes, independently evaluate which features remain stable/useful, and investigate surprising differences for errors in the underlying analysis representation. Its own README warns that its relative-difference similarity model is not necessarily the best BCSA method.

Own.NET has a different problem: when a validated C# transformation leaves ownership semantics unchanged but a diagnostic changes, can a small set of interpretable source/OwnIR features help localize the first wrong representation or inference step, more effectively than raw diffs/traces alone? Conversely, can verified bug-inducing edits reveal that an apparently stable feature vector is missing crucial ownership information?

This is about error explanation and triage value, not matching arbitrary similar programs, proving semantic equivalence, improving soundness by resemblance, or chasing ROC AUC.

Boundaries vs existing issues

• Research: semantic mutation testing harness for Own.NET analyzer coverage (bounded pilot) #405 generates and validates defect-inducing mutations; reuse its oracle/results. Do not build new mutation operators or a harness here.
• Research: refactoring-aware metamorphic regression testing for ownership analysis (RefactoringMiner prior art) #432 validates semantics-preserving refactorings and source correspondence; reuse its pairs/mappings. Do not build AST differencing or a refactoring engine here.
• Research: semantic subgroup analysis for Own.NET diagnostic quality (bounded pilot) #427 owns per-semantic-subgroup TP/FN/FP reporting. This issue instead tests whether low-cost structural/ownership feature deltas explain and accelerate debugging a specific divergence. Do not add a second dashboard.
• Research: SootUp-inspired OwnIR/ICFG semantic-fidelity and incomplete-call audit (bounded spike) #430 / P-022 diagnostics: normalized AnalysisTrace and first-divergence minimizer for Python↔Rust shadow mode #269 own representation-fidelity / first-divergence investigation. This spike may consume their snapshots/traces, not redefine their contracts.
• Research: evaluate Bifrost for C# code intelligence, agent navigation, and evidence contracts #426 owns uncertainty/completeness evidence; avoid interpreting absent facts, truncated calls or failed extraction as proved safety.

Proposed bounded experiment

0. Preregister cases and baseline

1. Pin Own.NET commit, extractor/engine/toolchain, source/case IDs, expected ownership outcomes, commands and provenance.
2. Select 10–20 already reviewed C# pairs, preferentially from Research: semantic mutation testing harness for Own.NET analyzer coverage (bounded pilot) #405/Research: refactoring-aware metamorphic regression testing for ownership analysis (RefactoringMiner prior art) #432/Research: SootUp-inspired OwnIR/ICFG semantic-fidelity and incomplete-call audit (bounded spike) #430: validated equivalent refactorings (clean and known-positive), plus several independently verified non-equivalent bug mutations. If those cases do not yet exist, curate a smaller manual subset and document the dependency instead of recreating their infrastructure.
3. Reserve some pairs for blind triage/evaluation; predeclare what counts as a useful explanation. Classify invalid/oracle-unresolved/unsupported/infra cases separately and exclude them from success denominators.
4. Baseline: record whether a simple diagnostic diff + existing OwnIR trace can already identify the earliest causal divergence, and effort needed per case. Avoid invented precision estimates.

1. Small, explicitly non-normative feature inventory

Extract features offline from existing artifacts, per comparable symbol/region where possible, with a provenance map:

• CFG: block/edge/exit counts, branch/loop shape, reachability of cleanup exits (only where emitted facts support it).
• Calls: direct/indirect/unresolved call boundaries, callee summaries present/missing, relevant argument/receiver mappings.
• Ownership: represented acquire/alias/use/escape/transfer/release events, guarded vs unguarded release, resource identity/summary flags where actually available.
• Extraction/completeness: missing/unsupported constructs, unknown target or truncated analysis, not zero-valued “safe” features.

Start with at most 8–12 cheap features. Preserve missing vs zero vs unknown, case/region IDs, and source→OwnIR evidence. Mark unsupported metrics as unsupported; never fabricate an event or CFG edge from syntax. Normalization must respect one-to-many extract/inline mappings; naive method-count equality is not a semantics oracle.

2. Sensitivity + triage comparison

For each reviewed pair, compare:

1. Ground truth first: expected source-level ownership behavior and applicable diagnostic(s), independently verified.
2. Raw baseline: source/OwnIR/diagnostic diff and existing first-divergence workflow.
3. Feature-assisted triage: per-feature deltas with source/evidence links, hypothesized cause, and manually verified first causal divergence.

Check both directions:

• Equivalent control: diagnostic/evidence differences may be a real regression, a legitimate relocation, a reporting difference, or an unresolved mapping. Feature drift alone is not a bug.
• Defect mutation: a changed feature is not a successful detection; the actual expected diagnostic at the relevant resource/site determines detection. Identical feature vectors do not establish semantic equivalence.

Record explanatory utility: number of independently confirmed issues/representation losses surfaced, correct vs misleading suspected stage, time/steps to verified root cause versus the basic diff workflow, setup/running overhead, and feature stability per predeclared category with raw counts. Do not optimize feature weights to the evaluation set or report an inflated AUC from tiny curated cases.

3. Investigate disagreements

Manually inspect a few high-information disagreements: (a) feature delta with no justified semantic/diagnostic change; (b) legitimate mutation with almost no feature delta; (c) changed diagnostic despite a reviewed equivalent transformation; (d) loss of an OwnIR fact without a reported diagnostic difference. Identify the earliest substantiated error stage (source correspondence, Roslyn extraction, OwnIR lowering/CFG/call frontier, summary/transfer, diagnostic mapping, oracle/unsupported).

One correctly minimized novel regression is more valuable than a large resemblance score. Forward any concrete causal bug to the relevant existing owner rather than broadening this spike.

Deliverables / acceptance criteria

• Source-backed TikNib note with exact version/revision when executed; explicitly separate binary-analysis results from untested C#/OwnIR hypotheses.
• Pinned case ledger (target 10–20 reviewed pairs, or smaller with documented availability), verified positive/negative/equivalent controls, exclusions and reproduction commands.
• Minimal offline feature extractor/report (or manual table first), reusing existing Research: semantic mutation testing harness for Own.NET analyzer coverage (bounded pilot) #405/Research: refactoring-aware metamorphic regression testing for ownership analysis (RefactoringMiner prior art) #432/Research: SootUp-inspired OwnIR/ICFG semantic-fidelity and incomplete-call audit (bounded spike) #430 artifacts; 8–12 documented features with provenance and missing/unknown policy.
• Baseline vs feature-assisted comparison on identical cases, with verified first divergence, effort/cost, false leads, and no leakage from development to held-out cases.
• At least two manually adjudicated feature/diagnostic disagreements, including a non-equivalent mutation control.
• Final GO / STOP / RESEARCH ONLY recommendation grounded in incremental triage utility, maintenance cost and independently verified findings.

Stop/promotion gate

Timebox to a small offline prototype. STOP if the same failures are as easy to localize from the existing source/OwnIR/diagnostic diff, if features are mostly unstable or uninformative, or if a trustworthy source-level oracle/correspondence is unavailable. No new framework merely to produce attractive plots.

A follow-up implementation issue is justified only for demonstrated improvement (e.g. a previously untracked, independently confirmed regression or reproducible, material reduction in root-cause triage work), with negative controls and a clear owner. No change to analyzer verdicts, runtime proof claims, default CI, or the Rust/Python cores in this task.

Sources

• TikNib README and feature inventory
• Paper / DOI
• BinKit dataset build infrastructure

Desired outcome: determine whether interpretable feature-delta analysis makes Own.NET better at diagnosing real mistakes, not better at scoring vaguely similar code.