Skip to content

Borrow target names during deduplication - #582

Merged
tinder-maxwellelliott merged 1 commit into
Tinder:masterfrom
tamird:tamird/borrow-deduplicated-target-names
Oct 27, 2026
Merged

tinder-maxwellelliott merged 1 commit into
Tinder:masterfrom
tamird:tamird/borrow-deduplicated-target-names

Conversation

@tamird

@tamird tamird commented Oct 27, 2026

Copy link
Copy Markdown
Contributor

deduplicate_targets clones each target name into its seen set, even though the input already owns the name. Borrow the names to record one keep flag per input, then retain the selected targets in place. The test puts a nameless target and a distinct target before the later duplicate and checks the complete first payload and output order.

On the 160,000-target public mixed fixture, the old set copies 160,000 nonempty strings totaling 3,060,000 UTF-8 bytes. The new pass stores 160,000 one-byte flags and visits the vector a second time. All 14 full generations produced identical hash and edge files. Median process wall was 2.338 s before and 2.245 s after. Median peak RSS was 286.3 MiB before and 294.4 MiB after. Both differences were within run-to-run variation. Fixture and full-output reproduction

I also timed just the helper on those decoded targets in a release build on macOS. In two fresh processes the medians were 13.46 → 11.23 ms and 22.78 → 17.43 ms. All six pairs in each process favored borrowing. The second process had substantially wider variation. The 192 Rust tests, 21 CLI tests, formatting and Clippy checks passed.

Codex for Tamir.

Reproduce the helper comparison

Generate mixed.pb with the public fixture above. On this branch, add the following temporary test to the existing tests module in src/bazel.rs. It discards the first pair, places each implementation first in three of the six measured pairs, and times only the helper. Input cloning and complete output comparison are outside the interval.

BAZEL_DIFF_PUBLIC_DEDUP_WIRE=/absolute/path/to/mixed.pb \
  bazel run --config=release --lockfile_mode=error //src:rust_tests -- \
  --ignored --exact bazel::tests::compare_public_target_deduplication --nocapture
    #[test]
    #[ignore]
    fn compare_public_target_deduplication() {
        use std::time::Instant;

        fn owning(targets: Vec<Target>) -> Vec<Target> {
            let mut seen = HashSet::new();
            targets
                .into_iter()
                .filter(|target| {
                    target_name(target).is_some_and(|name| seen.insert(name.to_owned()))
                })
                .collect()
        }

        let wire = std::env::var_os("BAZEL_DIFF_PUBLIC_DEDUP_WIRE").unwrap();
        let targets: Vec<Target> = decode_delimited(Path::new(&wire)).unwrap();
        let names: HashSet<&str> = targets.iter().filter_map(target_name).collect();
        assert_eq!(targets.len(), 160_000);
        assert_eq!(names.len(), targets.len());
        assert!(!names.contains(""));
        println!(
            "DEDUP_CENSUS {} {} {}",
            targets.len(),
            names.len(),
            names.iter().map(|name| name.len()).sum::<usize>()
        );
        drop(names);
        let pairs = [false, false, true, true, false, false, true];
        for (pair, swapped) in pairs.into_iter().enumerate() {
            for borrowed in [swapped, !swapped] {
                let input = targets.clone();
                let start = Instant::now();
                let result = if borrowed {
                    deduplicate_targets(input).unwrap()
                } else {
                    owning(input)
                };
                let elapsed = start.elapsed();
                assert_eq!(result, targets);
                println!(
                    "DEDUP_HELPER {pair} {} {}",
                    if borrowed { "candidate" } else { "control" },
                    elapsed.as_nanos()
                );
            }
        }
    }

Record the keep decisions while the seen set can borrow the input names, then retain those targets in place. Exercise a nonadjacent duplicate and a target with no name in the order check.
@tinder-maxwellelliott
tinder-maxwellelliott merged commit d75875e into Tinder:master Oct 27, 2026
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants