[ET-VK][test] Run operator benchmarks from one etvk_bench binary

[SS-JIA]

Stack from ghstack (oldest at bottom):

• -> [ET-VK][test] Run operator benchmarks from one etvk_bench binary #23598
• [ET-VK][test] Move operator benchmark binaries under ops/benchmark #23597
• [ET-VK][gemm] Nest linear and matmul sources by scheme #23493
• [ET-VK][convolution] Nest convolution sources by route #23492
• [ET-VK][binary] Nest binary operator sources #23491
• [ET-VK][codegen] Support nested shader directories #23490

Ulterior Motive

Make the Vulkan operator benchmarks easy to drive during optimization work: group related test cases (such as one model's bottleneck shapes) into named sets, run any subset from one binary, and later sweep implementations and dispatch parameters.

Rationale

What: Replace the 24 per-operator test+benchmark binaries with one entry point, etvk_bench, that runs test case sets registered by files under test/ops/benchmark/cases/<op>/. Split each operator's code into shared helpers, CPU references, and one file per set, and split each binary's default case list into correctness and performance sets.

Why: Each binary hardcoded one mixed list of correctness-sized and performance-sized cases, so running a subset meant editing code, and modes were spread across ad hoc CLI flags, environment variables, and #ifdef DEBUG_MODE.

Details

• Framework
  • framework/registry.{h,cpp}: TestCaseSet (test cases, reference, FLOP calculator, run counts, or a custom test for checks that are not test cases) and REGISTER_TEST_CASE_SET("<op>", "<set>") { ... }. The registered body runs only when the set is selected, since building test cases may query the device. execute_test_cases() takes the cases directly instead of a generator.
  • etvk_bench.cpp parses gflags: --list, --op=<op>[,...] (a parent directory such as q8ta selects every operator beneath it), --set=<set>[,...], --filter='<a>;<b>|<c>' (cases whose name contains every ;-separated substring of at least one |-separated alternative), --chain=<n>, --nogpu_timestamps, --print_output, --print_latencies, --debug, --device_info, and --helpshort. It runs every selected set, continues past failures, and exits nonzero if any set failed. gflags comes from external_deps in Buck and, in the standalone CMake project, from the main ExecuTorch build tree (as the Llama runner does).
  • --device-info replaces the test_coopmat_probe binary (framework/device_info.{h,cpp}, renamed from it).
• Case directories: each cases/<op>/ has <op>.h (configs and shared declarations), <op>.cpp (config to TestCase builders, CPU references, and FLOP calculators; q8ta/conv2d splits these by route into conv2d{,_dw,_pw}.cpp), and a test_sets/ folder whose files register the sets. A set file holds the set's test case data (configs, shapes, and the dtypes, storage types, layouts, and route selectors to sweep), the loop that turns it into test cases (including device gating and rules that derive variants from a config), and the returned TestCaseSet. A set gets its own <set>_set.cpp; sets that share generation code share one <group>_sets.cpp, named for their common prefix (dw_sets.cpp, pw_sets.cpp, fpa_sets.cpp, coopmat_sets.cpp, scenex_sets.cpp) or, for unprefixed sets such as correctness and performance, for the operator (linear_sets.cpp). Each directory's code is in its own namespace (prototyping::<op>), since operators reuse names such as LinearConfig and now link into one binary.
• Sets
  • Each binary's default list becomes correctness (its ACCU cases) and performance (its PERF cases), prefixed by route for secondary binaries in a directory (dw_, pw_, fpa_, coopmat_). Every case is registered in exactly one set, so the default lists no longer repeat the cases of the mode sets below.
  • Modes become sets:
    • test_q8ta_conv2d --im2col-path=*, --narrow-workgroups-only, --scenex-regular, and --scenex-grouped → im2col (each edge config under the signed, unsigned, and auto selectors), narrow_workgroup, scenex_regular, scenex_grouped.
    • Its streaming cases and dynamic shrink test → streaming_im2col and streaming_dynamic_shrink; its startup unsigned dot checks → unsigned_dot_selection.
    • test_q8ta_conv2d_pw --pw-path=* folds into pw_correctness and pw_performance, which run each config under the default, signed, unsigned, and auto selectors; test_q8ta_conv2d_dw narrow workgroups → dw_narrow_workgroup.
    • SDPA_DECODE_ONLY → sdpa:decode (sdpa:performance keeps only prefill); COOPMAT_BENCH_CORRECTNESS_ONLY → q4gsw_linear:coopmat_correctness; SDPA_NO_CHAIN → --chain=1.
    • #ifdef DEBUG_MODE and never-called easy case lists → debug. The qdq easy list now skips the legacy 4W4C path for 5D/6D shapes, as the default list does. The never-called q8csw_conv2d easy list is removed: it could not run, since it never set an operator name.
• Dropped: per-binary banners, the coopmat benchmark's tiled-vs-coopmat summary table, and the redundant query pool initialization in test_add.
• Build: one etvk_bench target in Buck and CMake replaces the per-binary targets.
• CI: each Skycastle (GPU host, Eureka) and OSS workflow job keeps its name and runs etvk_bench with the --op/--set selection that covers its old binary's run. The default test_q8ta_conv2d job adds im2col and narrow_workgroup, the _dw job adds dw_narrow_workgroup, and test_sdpa adds decode; the six --im2col-path/--pw-path jobs are removed, since test_q8ta_conv2d and test_q8ta_conv2d_pw now cover them.
• New files are recorded as copies of the binary they came from.

CLANGTIDY findings in the moved case code are fixed (argument comments that did not match parameter names, a shadowed loop variable, narrowing, a C array). Test case construction, references, tolerances, and case names are otherwise unchanged.

Differential Revision: D123707389

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/23598

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure

As of commit 9f46f53 with merge base bdb489e ():

NEW FAILURE - The following job has failed:

• Lint / lintrunner (gh)
  >>> Lint for backends/vulkan/test/ops/benchmark/cases/q4gsw_linear/test_sets/q4gsw_linear_sets.cpp:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.