From 91e6f42996e45afb3b488393ca1a09d4cc5757a1 Mon Sep 17 00:00:00 2001 From: drowaudio Date: Sun, 7 Jun 2026 16:35:59 +0100 Subject: [PATCH 01/21] 2.0.0: Rewrite CLI parsing onto CLI11 + a JSON settings pipeline (#175) * Bump pluginval to C++23 and add magic_args + nlohmann/json via CPM Prepares for the CLI parser refactor onto magic_args + a JSON-merge settings pipeline. - target_compile_features: cxx_std_20 -> cxx_std_23 (magic_args requires C++23) - CPMAddPackage magic_args v0.2.1 (header-only INTERFACE: magic_args::magic_args) - CPMAddPackage nlohmann/json 3.12.0 (nlohmann_json::nlohmann_json) - link both into the pluginval target Verified: clean build + link under C++23 with juce_recommended_warning_flags on JUCE 8.0.13 (macOS/AppleClang); binary runs. Co-Authored-By: Claude Opus 4.8 * Replace hand-rolled CLI parser with a JSON-merge settings pipeline Introduces a single parser-agnostic settings struct and a layered JSON pipeline that replaces the bespoke juce::ArgumentList parser. Pipeline: each input layer (--config file, environment, CLI) becomes a sparse nlohmann::json object; layers are merged with merge_patch in precedence order (defaults < config < env < CLI, CLI wins) and deserialised in one step into PluginvalSettings, which converts to the existing PluginTests::Options at the JUCE boundary. - PluginvalSettings.h: unified std-typed struct + toPluginTestOptions() + fromPluginTestOptions(); NLOHMANN_DEFINE_TYPE..._WITH_DEFAULT so missing keys fall back to defaults; RealtimeCheck enum<->string mapping. - SettingsSerializer.{h,cpp}: JSON load/save + the comma-list, hex-seed and disabled-tests(file-or-list) coercions shared by the CLI/env layers. - SettingsParser.{h,cpp}: preprocess (deprecation rewrite, macOS flag strip, implicit --validate), magic_args parse of the flat options into the sparse CLI layer, env layer, merge, and the child-process handoff. - Child process round-trip now uses an authoritative base64 JSON arg (--config-base64) instead of per-flag re-serialisation, avoiding quote/tokenisation hazards. Round-trip asserted in tests. - CommandLine.{h,cpp}: thin adapter; old manual parsers/env-merge/help text deleted. magic_args drives --help (auto usage + env-var trailer). - New --config option; precedence covered by tests. - CommandLineTests.cpp rewritten against the new pipeline (defaults, parser, hex seed, comma lists, rtcheck, path resolution, implicit validate, env precedence, config precedence, round-trip). - CMake: macOS deployment target 10.11 -> 13.3 (std::format in magic_args requires libc++ floating-point to_chars, available on 13.3+); add new sources. All --run-tests pass. Co-Authored-By: Claude Opus 4.8 * Regenerate CLI docs and update architecture notes - docs/Command line options.md regenerated from magic_args --help output - CLAUDE.md: document the JSON-merge settings pipeline, new source files, magic_args + nlohmann/json deps, C++23, and the macOS 13.3 deployment target Co-Authored-By: Claude Opus 4.8 * Swap magic_args for CLI11: bind options directly to the settings struct Addresses review feedback that the magic_args version duplicated the settings into a parallel PluginvalCliArgs struct plus manual CLI->JSON and ENV->JSON bridges (5-6 edit sites per new option), and forced cxx_std_23 + a macOS 13.3 deployment target (std::format). CLI11 binds each option directly to a PluginvalSettings member, with the environment variable on the same line via ->envname(). Adding an option is now three edits: a struct member, a nlohmann macro entry, and one add_option(...) line. No parallel struct, no cliArgsToJson, no envLayer. - SettingsParser: rewritten on CLI11; comma lists via ->delimiter(','), enum via CheckedTransformer, hex/int seed via a small callback. --config seeds the struct before parse so precedence stays defaults CLI11 2.6.2; revert cxx_std_23 -> cxx_std_20 and CMAKE_OSX_DEPLOYMENT_TARGET 13.3 -> 10.11. - Docs/CLAUDE.md updated. Verified: full build (validator ON, C++20, macOS 10.11), --run-tests pass, real --validate SUCCESS, --config precedence covered by tests. Co-Authored-By: Claude Opus 4.8 * Fix CI failures from the CLI11 refactor Three regressions surfaced by CI (develop is green): - Linux build: std::int64_t (== long on LP64) vs juce::int64 (== long long) made expectEquals template deduction fail. Cast the settings int64 fields to juce::int64 in CommandLineTests. (macOS compiled because there the two types are identical.) - "Pluginval as Dependency" Verify: verify_pluginval runs `pluginval --help | grep "JUCE v"`. CLI11's auto help didn't print the JUCE version; add it to the help footer. - Windows --run-tests exit 127: runUnitTests() called the [[noreturn]] juce::ConsoleApplication::fail(), which throws; outside a ConsoleApplication command handler that throw was uncaught -> std::terminate. Set the application return value directly instead. Co-Authored-By: Claude Opus 4.8 * Fix Linux X11 macro clash; print unit-test failures to stdout - Linux build: include before the JUCE headers. JUCE pulls in X11 on Linux (JUCE_GUI_BASICS_INCLUDE_XHEADERS), whose `#define Success 0` clashed with CLI11's CLI::ExitCodes::Success enumerator. macOS/Windows don't use X11, so only Linux was affected. - runUnitTests now prints each failed test (name + messages) to stdout. juce::UnitTestRunner logs via juce::Logger, which on a GUI app (Windows) doesn't reach the console, so CI couldn't show which tests failed. Co-Authored-By: Claude Opus 4.8 * Fix Windows test: compare raw parsed paths, not juce::File-normalised "Command line parser" asserted opts.dataFile/outputDir.getFullPathName() against Unix-style literals; on Windows juce::File normalises those to the current drive (e.g. "D:\path\to\file"), failing the comparison. Assert the raw parsed settings strings instead, which are identical on all platforms. Co-Authored-By: Claude Opus 4.8 * Revise precedence to defaults < env < config < CLI Reworks the layering so a --config file (deliberate, per-invocation) ranks above ambient environment variables, and makes --config repeatable. - Environment layer: env-var names are now derived from the registered CLI11 options (--strictness-level -> STRICTNESS_LEVEL) instead of CLI11's ->envname(), so there is no hand-maintained env table and env support is automatic for every option. A synthetic "--name=value" argv is built from the environment and parsed by CLI11, reusing all its coercion (comma lists, enum transformer, hex-seed callback). Flags accept --flag=1/0. - --config: repeatable; each JSON file is merge_patch-ed in command-line order (later files win per key). Applied between the env and CLI layers, so it beats env and loses to explicit CLI options. (Inline JSON dropped for now to avoid command-line re-tokenisation hazards; file paths only.) - The env layer reads through an injectable EnvProvider again, so the unit tests no longer mutate the real process environment (also removes the Windows _putenv_s fragility). - Tests updated for the new precedence + a repeatable-config test. - Help footer, docs and CLAUDE.md updated. Verified end-to-end: env(6) -> 6; env(6)+config(2) -> 2; +CLI(9) -> 9. Co-Authored-By: Claude Opus 4.8 * Add 2.0.0 changelist, harden --config-base64, add subcommands handoff - CHANGELIST.md: 2.0.0 entry summarising the CLI11/JSON parser rewrite, --config, the new precedence, and the breaking changes (VERSION not bumped). - --config-base64 (internal parent->child handoff) now errors if combined with any option other than --validate, instead of silently ignoring it. Test added. - SUBCOMMANDS_HANDOFF.md: handoff doc for the deferred subcommand restructure (validate/run-tests/strictness-help with deprecated flat-flag aliases). Co-Authored-By: Claude Opus 4.8 * Audit CLAUDE.md: remove non-existent --validate-in-process CLI flag - --validate-in-process is not a CLI option (launchInProcess is GUI/internal only); the new strict parser would now error on it. Removed from the options list and corrected the process-model notes (GUI uses child processes; the CLI --validate runs in-process with signal-handler crash reporting). - Note the staged 2.0.0 CHANGELIST entry vs the un-bumped VERSION. Co-Authored-By: Claude Opus 4.8 * Compile EditorTests and ExtremeTests (were present but not built) Source/tests/EditorTests.cpp and Source/tests/ExtremeTests.cpp were never added to SourceFiles, so their self-registering PluginTest instances were never compiled in and the tests never ran. Add them to the build. Re-enables: "Editor stress" (L6), "Editor DPI Awareness" (L3, Windows only), "Allocations during process" (L9), "Process called with a larger than prepared block size". Verified they compile and pass a real strictness-10 validation. Co-Authored-By: Claude Opus 4.8 * Fix Windows-only compile error in EditorDPITest Qualify juce::ScopedDPIAwarenessDisabler. The EditorDPITest block is guarded by #if JUCE_WINDOWS && JUCE_WIN_PER_MONITOR_DPI_AWARE, so this latent bug only surfaced now that EditorTests.cpp is compiled (and only on Windows). Co-Authored-By: Claude Opus 4.8 --------- Co-authored-by: Claude Opus 4.8 --- CHANGELIST.md | 11 + CLAUDE.md | 50 ++- CMakeLists.txt | 13 +- SUBCOMMANDS_HANDOFF.md | 89 ++++++ Source/CommandLine.cpp | 561 +++++----------------------------- Source/CommandLine.h | 4 +- Source/CommandLineTests.cpp | 322 ++++++++++++++----- Source/PluginvalSettings.h | 142 +++++++++ Source/SettingsParser.cpp | 409 +++++++++++++++++++++++++ Source/SettingsParser.h | 79 +++++ Source/SettingsSerializer.cpp | 68 +++++ Source/SettingsSerializer.h | 50 +++ Source/tests/EditorTests.cpp | 2 +- docs/Command line options.md | 137 ++++----- 14 files changed, 1282 insertions(+), 655 deletions(-) create mode 100644 SUBCOMMANDS_HANDOFF.md create mode 100644 Source/PluginvalSettings.h create mode 100644 Source/SettingsParser.cpp create mode 100644 Source/SettingsParser.h create mode 100644 Source/SettingsSerializer.cpp create mode 100644 Source/SettingsSerializer.h diff --git a/CHANGELIST.md b/CHANGELIST.md index fb4e7fd6..5e5140b5 100644 --- a/CHANGELIST.md +++ b/CHANGELIST.md @@ -1,5 +1,16 @@ # pluginval Change List +### 2.0.0 +- Replaced the hand-rolled command-line parser with CLI11 and a single JSON-based settings pipeline +- Added `--config ` to load settings from JSON (repeatable; later files win per key) +- Settings precedence is now (lowest to highest): defaults, environment variables, `--config`, command-line options +- Environment variable support is now automatic for every option (including `DISABLED_TESTS`) +- **Breaking:** unrecognised options now produce an error instead of being ignored +- **Breaking:** invalid option values now error instead of silently falling back (e.g. `--rtcheck banana`, `--strictness-level abc`) +- **Breaking:** falsey flag environment variables now mean off (e.g. `SKIP_GUI_TESTS=0` no longer enables skipping) +- `--help` output is now generated by CLI11 +- Enabled the editor stress and extreme (real-time allocation / oversized block) tests that were present in the source tree but had not been compiled into the build + ### 1.0.5 - Added drag-and-drop of plug-in files onto the main window, with two drop zones to either validate the plug-in or add it to the plugin list [#170] - Added static linking to the Windows runtime so it should run on more Windows systems (particularly non-dev machines) diff --git a/CLAUDE.md b/CLAUDE.md index fcec6889..4acbea1c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -4,7 +4,7 @@ **pluginval** is a cross-platform audio plugin validator and tester application developed by Tracktion Corporation. It tests VST, VST3, AU (Audio Unit), LV2, and LADSPA plugins for compatibility and stability with host applications. -- **Version**: 1.0.4 (see `VERSION` file) +- **Version**: 1.0.4 (see `VERSION` file; a 2.0.0 entry is staged in `CHANGELIST.md` but `VERSION` is not yet bumped) - **License**: GPLv3 - **Framework**: Built on JUCE (v8.0.x) - **Language**: C++20 @@ -80,7 +80,10 @@ pluginval/ │ ├── MainComponent.cpp/h # GUI main window component │ ├── Validator.cpp/h # Core validation orchestration │ ├── PluginTests.cpp/h # Test framework and base classes -│ ├── CommandLine.cpp/h # CLI argument parsing +│ ├── CommandLine.cpp/h # Thin CLI adapter (delegates to SettingsParser) +│ ├── PluginvalSettings.h # Unified settings struct + JSON mapping + toPluginTestOptions() +│ ├── SettingsParser.cpp/h # CLI/env/config -> merged JSON -> settings; child-process handoff +│ ├── SettingsSerializer.cpp/h # JSON load/save + value coercions (comma lists, hex seed) │ ├── CrashHandler.cpp/h # Crash reporting utilities │ ├── TestUtilities.cpp/h # Helper functions for tests │ ├── RTCheck.h # Real-time safety checking macros @@ -188,6 +191,41 @@ VST2_SDK_DIR=/path/to/vst2sdk cmake -B Builds/Debug . - Auto-registers via static instance pattern - Defines requirements (thread, GUI needs) +### CLI Settings Pipeline + +Command-line parsing centres on one plain settings struct (`PluginvalSettings`) +that CLI11 binds to directly. A single instance is filled by successive layers, +**lowest to highest precedence: defaults → environment → `--config` → CLI** +(in `SettingsParser::parseTokens`): + +1. **preprocess** the raw command line — rewrite the deprecated `strictnessLevel`, + strip the macOS `-NSDocumentRevisionsDebugMode YES` flag, and insert an + implicit `--validate` when the last argument is a bare plugin path. +2. **Environment layer.** Env-var names are *derived* from the registered + options (`--strictness-level` → `STRICTNESS_LEVEL`), so there is no separate + env table. A synthetic `--name=value` argv is built from the environment and + parsed by CLI11, reusing all its coercion. +3. **`--config` layer.** Repeatable; each JSON file is `merge_patch`-ed in + command-line order (later files win per key). Beats the environment. +4. **CLI layer.** The real arguments are parsed last and beat everything; + CLI11 only overwrites a member when its option was actually provided. + +`configureApp()` registers every option (bound to the struct) and is used for +both the env pass and the CLI pass. Comma lists use `->delimiter(',')`, the enum +uses a `CheckedTransformer`, and the hex/int seed is a small callback. +`PluginvalSettings::toPluginTestOptions()` converts to the JUCE-flavoured +`PluginTests::Options` at the boundary. + +Adding a new option is three edits: a struct member, an entry in the nlohmann +macro list, and one `add_option(...)` line — its environment variable then works +automatically. `SettingsSerializer` handles JSON load/save plus the two +remaining conversions (hex seed, disabled-tests file). + +The child validation process receives a fully-resolved, **authoritative** +settings set via a base64-encoded JSON argument (`--config-base64`), avoiding +per-flag re-serialisation and command-line quoting hazards. `--help`/`--version` +are handled by CLI11 (auto usage + a footer with the env-var/commands notes). + ### Test Framework Tests are self-registering. To find all tests, look for static instances: @@ -327,9 +365,9 @@ Basic usage: Key options: - `--validate [path]` - Validate plugin at path +- `--config [file.json]` - Load a full settings set from JSON (overridden by env vars and CLI options) - `--strictness-level [1-10]` - Test thoroughness (default: 5) - `--skip-gui-tests` - Skip GUI tests (for headless CI) -- `--validate-in-process` - Don't use child process (for debugging) - `--timeout-ms [ms]` - Test timeout (default: 30000, -1 for none) - `--verbose` - Enable verbose logging - `--output-dir [dir]` - Directory for log files @@ -374,6 +412,8 @@ add_pluginval_tests(MyPluginTarget ### External - **JUCE** (v8.0.x) - Audio application framework (git submodule) - **magic_enum** (v0.9.7) - Enum reflection (fetched via CPM) +- **CLI11** (v2.6.2) - CLI argument parsing, header-only (fetched via CPM) +- **nlohmann/json** (3.12.0) - JSON settings layering/serialisation (fetched via CPM) - **rtcheck** (optional, macOS) - Real-time safety checking (fetched via CPM) - **VST3 SDK** (v3.7.x) - Steinberg VST3 SDK for embedded validator (fetched via CPM, optional) @@ -439,6 +479,6 @@ Run internal tests via CLI: - Always test changes on multiple platforms when possible - VST3 plugins have specific threading requirements - use the `*OnMessageThreadIfVST3` helpers -- Child process validation is the default and recommended for production use -- In-process validation (`--validate-in-process`) is useful for debugging but a crashing plugin will crash pluginval +- The GUI runs each validation in a separate child process for crash isolation (the default) +- The CLI `--validate` path runs in-process; a crashing plugin will terminate pluginval, and the signal handler reports it as a failure rather than a pass - Real-time safety checking is only available on macOS currently (uses rtcheck library) diff --git a/CMakeLists.txt b/CMakeLists.txt index f2ac5bb7..20e02b69 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -61,6 +61,8 @@ option(JUCE_ENABLE_MODULE_SOURCE_GROUPS "Enable Module Source Groups" ON) include(cmake/CPM.cmake) CPMAddPackage("gh:Neargye/magic_enum#v0.9.7") +CPMAddPackage("gh:CLIUtils/CLI11@2.6.2") +CPMAddPackage("gh:nlohmann/json@3.12.0") if(PLUGINVAL_ENABLE_RTCHECK) CPMAddPackage("gh:Tracktion/rtcheck#main") @@ -120,6 +122,9 @@ set(SourceFiles Source/CrashHandler.h Source/MainComponent.h Source/PluginTests.h + Source/PluginvalSettings.h + Source/SettingsParser.h + Source/SettingsSerializer.h Source/TestUtilities.h Source/Validator.h Source/CommandLine.cpp @@ -127,9 +132,13 @@ set(SourceFiles Source/Main.cpp Source/MainComponent.cpp Source/PluginTests.cpp + Source/SettingsParser.cpp + Source/SettingsSerializer.cpp Source/tests/BasicTests.cpp Source/tests/LocaleTest.cpp Source/tests/BusTests.cpp + Source/tests/EditorTests.cpp + Source/tests/ExtremeTests.cpp Source/tests/ParameterFuzzTests.cpp Source/TestUtilities.cpp Source/Validator.cpp) @@ -176,7 +185,9 @@ target_link_libraries(pluginval PRIVATE juce::juce_audio_processors juce::juce_audio_utils juce::juce_recommended_warning_flags - magic_enum) + magic_enum + CLI11::CLI11 + nlohmann_json::nlohmann_json) if (${CMAKE_SYSTEM_NAME} MATCHES "Linux") target_link_libraries(pluginval PRIVATE diff --git a/SUBCOMMANDS_HANDOFF.md b/SUBCOMMANDS_HANDOFF.md new file mode 100644 index 00000000..3f6ef277 --- /dev/null +++ b/SUBCOMMANDS_HANDOFF.md @@ -0,0 +1,89 @@ +# HANDOFF: Restructure pluginval's CLI into subcommands + +## Goal +Turn the current "mode" flags into proper subcommands, each with its own +argument set, while keeping the old flat flags working as **deprecated aliases +for one release**: + +| New (target) | Old (keep as deprecated alias) | +|---|---| +| `pluginval validate [options] ` (the **default**) | `pluginval --validate ` / `pluginval ` | +| `pluginval run-tests` | `pluginval --run-tests` | +| `pluginval strictness-help [level]` | `pluginval --strictness-help [level]` | +| `pluginval --version` / `pluginval --help` | (unchanged) | + +All settings options (`--strictness-level`, `--config`, `--rtcheck`, …), +environment variables, and the JSON precedence pipeline belong to the `validate` +subcommand and are otherwise unchanged. + +This was deliberately deferred from the CLI11 refactor PR (#174). The pipeline +and `PluginvalSettings` were designed to be reused unchanged. + +## Read these first (don't trust this summary — verify against the files) +- `Source/SettingsParser.cpp/.h` — the parser. Key pieces to reuse: + - `preprocess()` — deprecation rewrite, macOS flag strip, implicit `--validate`, tokenise. + - `parseTokens(tokens, env)` — env → `--config` → CLI layering into one `PluginvalSettings`. **This is the `validate` body.** + - `configureApp(app, s)` — registers every option on a `CLI::App`, used for both the env and CLI passes. + - `createChildProcessCommandLine()` — the parent→child base64 handoff (`--config-base64 --validate `). + - `isCommandLine(tokens)` — what makes `shouldPerformCommandLine` return true. +- `Source/CommandLine.cpp` — `performCommandLine()` is the dispatcher today: + token-scans for `--run-tests` / `--strictness-help`, otherwise runs `parseTokens` for validate; `--help`/`--version` go through CLI11. Also `runUnitTests()`, `printStrictnessHelp()`. +- `Source/Main.cpp` — calls `shouldPerformCommandLine()` then `performCommandLine()` with `getCommandLineParameters()` (a single `juce::String`). +- `Source/CommandLineTests.cpp` — the test contract. + +## Recommended approach: a thin `argv[1]` verb dispatcher (not CLI11 subcommands) +The existing `parseTokens` pipeline (env/config/CLI layering via two `CLI::App` +passes) is the hard part and already works. Rather than re-express it inside +CLI11's native `add_subcommand` machinery (which complicates the env/config +layering because the options live on the subcommand), peel the verb off the +front and route: + +1. In a new `dispatch()` step (in `SettingsParser` or `CommandLine.cpp`): + - Tokenise the command line (reuse `preprocess` minus the implicit-validate step, or add a pre-step). + - Look at the first non-option token: + - `validate` → strip it, run the existing validate pipeline on the rest. + - `run-tests` → `runUnitTests()`. + - `strictness-help` → `printStrictnessHelp(level)`. + - otherwise → **default to validate** (this preserves `pluginval ` and `pluginval --strictness-level 5 `). +2. Each verb keeps its own small set of expected args. `validate` reuses + `configureApp`/`parseTokens` verbatim. `run-tests` takes none. + `strictness-help` takes an optional level. + +This keeps `parseTokens` and the precedence layering untouched — the subcommand +work is purely a routing layer in front of it. + +(If you prefer CLI11-native subcommands instead: put `configureApp` options on a +`validate` subcommand, and make `buildEnvArgv`/the env pass target that +subcommand's options. Doable, but more invasive for no functional gain here.) + +## Deprecated-alias behaviour (one release) +Keep the old flat forms working, but print a one-line notice to stderr pointing +at the new syntax, e.g.: +- `pluginval --validate x` → run validate; warn `"--validate is deprecated; use 'pluginval validate x'"`. +- `pluginval --run-tests` → warn `"use 'pluginval run-tests'"`. +- `pluginval --strictness-help` → warn `"use 'pluginval strictness-help'"`. +- `pluginval ` (bare path) → **no warning** (still the documented shorthand for `validate`). + +Gate the warnings behind detection of the old flag so the new subcommand form is +silent. Remove the aliases in the release after next; note it in `CHANGELIST.md`. + +## Files to touch +- `Source/SettingsParser.{h,cpp}` — add the verb dispatch + a `Command` result (validate/run-tests/strictness-help/help/version), or expose a `dispatch()` that returns which verb + the remaining tokens. Keep `parseTokens` as the validate body. +- `Source/CommandLine.cpp` — `performCommandLine()` routes on the verb; emit the deprecation notices for old flat flags. `shouldPerformCommandLine()` must also recognise the bare verbs (`validate`/`run-tests`/`strictness-help`) in addition to the old flags. +- `Source/CommandLineTests.cpp` — add tests: each subcommand; default-to-validate; bare-path shorthand; every deprecated alias still works (and warns); `run-tests`/`strictness-help` arg handling. +- `docs/Command line options.md` — regenerate (`pluginval --help`); CLI11 can show per-subcommand help if you go native, otherwise hand-format the verb list. +- `CHANGELIST.md` — note the subcommand syntax + the deprecation. +- `CLAUDE.md` — update the "CLI Settings Pipeline" section to mention the verb layer. + +## Gotchas / decisions to make +- **Child-process handoff.** `createChildProcessCommandLine()` emits `--config-base64 --validate `. Decide whether the child invocation becomes `validate --config-base64 …` or stays flat. Simplest: keep it flat and have the dispatcher treat a leading `--config-base64`/`--validate` as the (deprecated, unwarned-for-internal) validate path. Note `--config-base64` is now hardened to reject being combined with non-`--validate` options — keep that working under whichever form you choose. +- **`shouldPerformCommandLine`** is what flips pluginval into CLI (vs GUI) mode in `Main.cpp`. It must return true for `pluginval run-tests` etc., not just the old flags. +- **`--help` scope.** With the dispatcher, `pluginval --help` is the top-level help (list verbs + the validate options). Consider `pluginval validate --help` for the full option list. CLI11-native subcommands give this for free. +- **Implicit validate** currently lives in `preprocess`. With an explicit `validate` verb, make sure `pluginval ` (no verb) still resolves to validate, and `pluginval validate ` doesn't double-insert `--validate`. +- **Reconcile** a positional plugin path under `validate` (e.g. `pluginval validate `) with the existing `--validate ` option — pick one canonical form (recommend the positional for the new syntax, mapping it onto `s.validatePath`). + +## Verify +- `pluginval run-tests` passes the full unit suite (it must, it's how CI runs tests). +- `pluginval validate --strictness-level 10 ` and `pluginval ` both validate. +- Every deprecated alias produces identical behaviour to before (plus a notice). +- CI matrix green (Linux/macOS/Windows build + dependency). Remember `.github/workflows/build.yaml` uses `--run-tests` and `--strictness-level 10 --validate …`; update those to the new syntax **and** keep an alias test, or the deprecation will fire in CI. diff --git a/Source/CommandLine.cpp b/Source/CommandLine.cpp index 55e8094c..f152d274 100644 --- a/Source/CommandLine.cpp +++ b/Source/CommandLine.cpp @@ -16,6 +16,11 @@ #include "Validator.h" #include "CrashHandler.h" #include "PluginTests.h" +#include "PluginvalSettings.h" +#include "SettingsParser.h" + +#include +#include #if JUCE_MAC #include @@ -23,8 +28,6 @@ #include #endif -#include - //============================================================================== static void exitWithError (const juce::String& error) { @@ -115,305 +118,7 @@ void CommandLineValidator::validate (const juce::String& fileOrID, PluginTests:: }); } - -//============================================================================== -//============================================================================== -namespace -{ - juce::ArgumentList::Argument getArgumentAfterOption (const juce::ArgumentList& args, juce::StringRef option) - { - for (int i = 0; i < args.size() - 1; ++i) - if (args[i] == option) - return args[i + 1]; - - return {}; - } - - juce::var getOptionValue (const juce::ArgumentList& args, juce::StringRef option, juce::var defaultValue, juce::StringRef errorMessage) - { - if (args.containsOption (option)) - { - const auto nextArg = getArgumentAfterOption (args, option); - - if (nextArg.isShortOption() || nextArg.isLongOption()) - juce::ConsoleApplication::fail (errorMessage, -1); - - return nextArg.text; - } - - return defaultValue; - } - - int getStrictnessLevel (const juce::ArgumentList& args) - { - return juce::jlimit (1, 10, (int) getOptionValue (args, "--strictness-level", 5, "Missing strictness level argument! (Must be between 1 - 10)")); - } - - juce::int64 getRandomSeed (const juce::ArgumentList& args) - { - const juce::String seedString = getOptionValue (args, "--random-seed", "0", "Missing random seed argument!").toString(); - - if (! seedString.containsOnly ("x-0123456789acbdef")) - juce::ConsoleApplication::fail ("Invalid random seed argument!", -1); - - if (seedString.startsWith ("0x")) - return seedString.getHexValue64(); - - return seedString.getLargeIntValue(); - } - - juce::int64 getTimeout (const juce::ArgumentList& args) - { - return getOptionValue (args, "--timeout-ms", 30000, "Missing timeout-ms level argument!"); - } - - int getNumRepeats (const juce::ArgumentList& args) - { - return juce::jmax (1, (int) getOptionValue (args, "--repeat", 1, "Missing repeat argument! (Must be greater than 0)")); - } - - juce::File getDataFile (const juce::ArgumentList& args) - { - return getOptionValue (args, "--data-file", {}, "Missing data-file path argument!").toString(); - } - - juce::File getOutputDir (const juce::ArgumentList& args) - { - return getOptionValue (args, "--output-dir", {}, "Missing output-dir path argument!").toString(); - } - - juce::String getOutputFilename (const juce::ArgumentList& args) - { - return getOptionValue (args, "--output-filename", {}, "Missing output-filename path argument!").toString(); - } - - std::vector getSampleRates (const juce::ArgumentList& args) - { - juce::StringArray input = juce::StringArray::fromTokens (getOptionValue (args, - "--sample-rates", - juce::String ("44100,48000,96000"), - "Missing sample rate list argument!") - .toString(), - ",", - "\""); - std::vector output; - - for (juce::String sr : input) - output.push_back (sr.getDoubleValue()); - - return output; - } - - std::vector getBlockSizes (const juce::ArgumentList& args) - { - juce::StringArray input = juce::StringArray::fromTokens (getOptionValue (args, - "--block-sizes", - juce::String ("64,128,256,512,1024"), - "Missing block size list argument!") - .toString(), - ",", - "\""); - std::vector output; - - for (juce::String sr : input) - output.push_back (sr.getIntValue()); - - return output; - } - - juce::StringArray getDisabledTests (const juce::ArgumentList& args) - { - const auto value = getOptionValue (args, "--disabled-tests", {}, "Missing disabled-tests path argument!").toString(); - - if (juce::File::isAbsolutePath (value)) - { - const juce::File disabledTestsFile (value); - - juce::StringArray disabledTests; - disabledTestsFile.readLines (disabledTests); - - return disabledTests; - } - - return juce::StringArray::fromTokens (value, ",", ""); - } - - bool isPluginArgument (juce::String arg) - { - juce::AudioPluginFormatManager formatManager; - #if JUCE_VERSION >= 0x08000B - juce::addDefaultFormatsToManager (formatManager); - #else - formatManager.addDefaultFormats(); - #endif - - for (auto format : formatManager.getFormats()) - if (format->fileMightContainThisPluginType (arg)) - return true; - - // The above will check if the file actually exists which isn't really what we want for CLI parsing - if (auto f = juce::File::createFileWithoutCheckingPath (arg); - f.hasFileExtension (".vst3") - #if JUCE_PLUGINHOST_VST - || f.hasFileExtension (".dll") - #endif - ) - return true; - - return false; - } -} - -//============================================================================== -struct Option -{ - const char* name; - bool requiresValue; -}; - -static juce::String getEnvironmentVariableName (Option opt) -{ - return juce::String (opt.name).trimCharactersAtStart ("-").replace ("-", "_").toUpperCase(); -} - -static Option possibleOptions[] = -{ - { "--strictness-level", true }, - { "--random-seed", true }, - { "--timeout-ms", true }, - { "--verbose", true }, - { "--skip-gui-tests", false }, - { "--data-file", true }, - { "--output-dir", true }, - { "--output-filename", true }, - { "--repeat", true }, - { "--randomise", false }, - { "--sample-rates", true }, - { "--block-sizes", true }, - { "--rtcheck", false }, -}; - -static juce::StringArray mergeEnvironmentVariables (juce::StringArray args, std::function environmentVariableProvider = [] (const juce::String& name, const juce::String& defaultValue) { return juce::SystemStats::getEnvironmentVariable (name, defaultValue); }) -{ - for (auto arg : possibleOptions) - { - auto envVarName = getEnvironmentVariableName (arg); - auto envVarValue = environmentVariableProvider (envVarName, {}); - - if (envVarValue.isNotEmpty()) - { - const int index = args.indexOf (arg.name); - - if (index != -1) - { - std::cout << "Skipping environment variable " << envVarName << " due to " << arg.name << " set" << std::endl; - continue; - } - - if (arg.requiresValue) - args.insert (0, envVarValue); - - args.insert (0, arg.name); - } - } - - return args; -} - - -//============================================================================== //============================================================================== -static juce::String getHelpMessage() -{ - const juce::String appName (juce::JUCEApplication::getInstance()->getApplicationName()); - const juce::String juceVersion (juce::SystemStats::getJUCEVersion()); - - return juce::String (R"(//============================================================================== -)" + appName + R"( -)" + juceVersion + R"( - -Description: - Validate plugins to test compatibility with hosts and verify plugin API conformance - -Usage: - --version - Print pluginval version. - --validate [pathToPlugin] - Validates the plugin at the given path. - N.B. the "--validate" flag is optional if the path is the last argument. - This enables you to validate a plugin with simply "pluginval path_to_plugin". - - --sample-rates [list of comma separated sample rates] - If specified, sets the list of sample rates at which tests will be executed - (default=44100,48000,96000) - --block-sizes [list of comma separated block sizes] - If specified, sets the list of block sizes at which tests will be executed - (default=64,128,256,512,1024) - --random-seed [hex or int] - Sets the random seed to use for the tests. Useful for replicating test - environments. - --data-file [pathToFile] - If specified, sets a path to a data file which can be used by tests to - configure themselves. This can be useful for things like known audio output. - - --strictness-level [1-10] - Sets the strictness level to use. A minimum level of 5 (also the default) - is recommended for compatibility. - Higher levels include longer, more thorough tests such as fuzzing. - --strictness-help [level] - Lists all tests that run at the given strictness level (default: 5). - --timeout-ms [numMilliseconds] - Sets a timout which will stop validation with an error if no output from any - test has happened for this number of ms. - By default this is 30s but can be set to "-1" (must be quoted) to never timeout. - --rtcheck [empty, disabled, enabled or relaxed] - Turns on real-time safety checks using rtcheck (macOS and Linux only). - relaxed mode doesn't run the checks for the first processing block as a lot of plugins - use this to allocate or initialise thread-locals (which can allocate) - - --repeat [num repeats] - If specified repeats the tests a given number of times. Note that this does - not delete and re-instantiate the plugin for each repeat. - --randomise - If specified, the tests are run in a random order per repeat. - - --skip-gui-tests - If specified, avoids tests that create GUI windows, which can cause problems - on headless CI systems. - --disabled-tests [pathToFile] - If specified, sets a path to a file that should have the names of disabled - tests on each row. - - --output-dir [pathToDir] - If specified, sets a directory to store the log files. This can be useful - for continuous integration. - --output-filename [filename] - If specified, sets a filename for the log files (within 'output-dir' or - (lacking that) the current directory. - By default, the name is constructed from the plugin metainformation - --verbose - If specified, outputs additional logging information. It can be useful to - turn this off when building with CI to avoid huge log files. - -Exit code: - 0 if all tests complete successfully - 1 if there are any errors - -Additionally, you can specify any of the command line options as environment -variables by removing prefix dashes, converting internal dashes to underscores -and capitalising all letters, a.g. - "--skip-gui-tests" > "SKIP_GUI_TESTS=1" - "--timeout-ms 30000" > "TIMEOUT_MS=30000" -Specifying specific command-line options will override any environment variables -set for that option. -)"); -} - -static juce::String getVersionText() -{ - return juce::String ("pluginval") + " - " + VERSION; -} - static void printStrictnessHelp (int level) { level = juce::jlimit (1, 10, level); @@ -449,231 +154,115 @@ static void printStrictnessHelp (int level) std::cout << std::endl; } -static int getNumTestFailures (juce::UnitTestRunner& testRunner) -{ - int numFailures = 0; - - for (int i = 0; i < testRunner.getNumResults(); ++i) - if (auto result = testRunner.getResult (i)) - numFailures += result->failures; - - return numFailures; -} - static void runUnitTests() { juce::UnitTestRunner testRunner; testRunner.runTestsInCategory ("pluginval"); - const int numFailures = getNumTestFailures (testRunner); - if (numFailures > 0) - juce::ConsoleApplication::fail (juce::String (numFailures) + " tests failed!!!"); -} + int numFailures = 0; -//============================================================================== -static juce::ArgumentList createCommandLineArgs (juce::String commandLine) -{ - if (commandLine.contains ("strictnessLevel")) + // Print failures to stdout: juce::UnitTestRunner logs via juce::Logger, which + // on a GUI app (e.g. Windows) doesn't reach the console. + for (int i = 0; i < testRunner.getNumResults(); ++i) { - std::cout << "!!! WARNING:\n\t\"strictnessLevel\" is deprecated and will be removed in a future version.\n" - << "\tPlease use --strictness-level instead\n\n"; - } - - commandLine = commandLine.replace ("strictnessLevel", "strictness-level") - .replace ("-NSDocumentRevisionsDebugMode YES", "") - .trim(); - - const auto exe = juce::File::getSpecialLocation (juce::File::currentExecutableFile); - - juce::StringArray args; - args.addTokens (commandLine, true); - args = mergeEnvironmentVariables (args); - args.trim(); - - for (auto& s : args) - s = s.unquoted(); + if (auto* result = testRunner.getResult (i)) + { + numFailures += result->failures; - // If only a plugin path is supplied as the last arg, add an implicit --validate - // option for it so the rest of the CLI works - juce::ArgumentList argList (exe.getFullPathName(), args); + if (result->failures > 0) + { + std::cout << "!!! FAILED: " << result->unitTestName << " / " << result->subcategoryName << std::endl; - if (argList.size() > 0) - { - const bool hasValidateOrOtherCommand = argList.containsOption ("--validate") - || argList.containsOption ("--help|-h") - || argList.containsOption ("--version") - || argList.containsOption ("--run-tests"); - - if (! hasValidateOrOtherCommand) - if (isPluginArgument (argList.arguments.getLast().text)) - argList.arguments.insert (argList.arguments.size() - 1, { "--validate" }); + for (const auto& message : result->messages) + std::cout << " " << message << std::endl; + } + } } - return argList; -} - -static void performCommandLine (CommandLineValidator& validator, const juce::ArgumentList& args) -{ - hideDockIcon(); - - juce::ConsoleApplication cli; - cli.addVersionCommand ("--version", getVersionText()); - cli.addHelpCommand ("--help|-h", getHelpMessage(), true); - cli.addCommand ({ "--validate", - "--validate [pathToPlugin]", - "Validates the file (or IDs for AUs).", juce::String(), - [&validator] (const auto& validatorArgs) - { - auto [fileOrIDToValidate, options] = parseCommandLine (validatorArgs); - validator.validate (fileOrIDToValidate, options); - }}); - cli.addCommand ({ "--run-tests", - "--run-tests", - "Runs the internal unit tests.", juce::String(), - [] (const auto&) { runUnitTests(); }}); - cli.addCommand ({ "--strictness-help", - "--strictness-help [level]", - "Lists all tests that run at the given strictness level.", juce::String(), - [] (const auto& commandArgs) - { - int level = 5; - auto arg = getArgumentAfterOption (commandArgs, "--strictness-help"); - if (arg.text.isNotEmpty() && ! arg.isShortOption() && ! arg.isLongOption()) - level = arg.text.getIntValue(); - printStrictnessHelp (level); - }}); - - if (const auto retValue = cli.findAndRunCommand (args); retValue != 0) + // Set the return value directly rather than juce::ConsoleApplication::fail(), + // which throws and would terminate the process when called outside a + // ConsoleApplication command handler. + if (numFailures > 0) { - juce::JUCEApplication::getInstance()->setApplicationReturnValue (retValue); - juce::JUCEApplication::getInstance()->quit(); + std::cout << numFailures << " tests failed!!!" << std::endl; + juce::JUCEApplication::getInstance()->setApplicationReturnValue (1); } - - // --validate runs async so will quit itself when done - if (! args.containsOption ("--validate")) - juce::JUCEApplication::getInstance()->quit(); -} - -//============================================================================== -void performCommandLine (CommandLineValidator& validator, const juce::String& commandLine) -{ - performCommandLine (validator, createCommandLineArgs (commandLine)); -} - -bool shouldPerformCommandLine (const juce::String& commandLine) -{ - const auto args = createCommandLineArgs (commandLine); - return args.containsOption ("--help|-h") - || args.containsOption ("--version") - || args.containsOption ("--validate") - || args.containsOption ("--run-tests") - || args.containsOption ("--strictness-help"); } //============================================================================== //============================================================================== -std::pair parseCommandLine (const juce::ArgumentList& args) +std::pair parseCommandLine (const juce::String& commandLine) { - auto fileOrID = getOptionValue (args, "--validate", "", "Expected a plugin path for the --validate option").toString(); - - // in the case of a path (vs. ID), grab the full path - // getCurrentWorkingDirectory is needed to handle relative paths - // It preserves absolute paths and first checks for ~ on Mac/Windows - if (fileOrID.contains ("~") || fileOrID.contains (".")) - fileOrID = juce::File::getCurrentWorkingDirectory().getChildFile(fileOrID).getFullPathName(); - - PluginTests::Options options; - options.strictnessLevel = getStrictnessLevel (args); - options.randomSeed = getRandomSeed (args); - options.timeoutMs = getTimeout (args); - options.verbose = args.containsOption ("--verbose"); - options.numRepeats = getNumRepeats (args); - options.randomiseTestOrder = args.containsOption ("--randomise"); - options.dataFile = getDataFile (args); - options.outputDir = getOutputDir (args); - options.outputFilename = getOutputFilename (args); - options.withGUI = ! args.containsOption ("--skip-gui-tests"); - options.disabledTests = getDisabledTests (args); - options.sampleRates = getSampleRates (args); - options.blockSizes = getBlockSizes (args); - options.realtimeCheck = magic_enum::enum_cast (getOptionValue (args, "--rtcheck", "", "Expected one of [disabled, enabled, relaxed]").toString().toStdString()) - .value_or (RealtimeCheck::disabled); - - return { fileOrID, options }; + const auto settings = settings_parser::parse (commandLine); + return { juce::String (settings.validatePath), settings.toPluginTestOptions() }; } -std::pair parseCommandLine (const juce::String& cmd) +juce::StringArray createCommandLine (juce::String fileOrID, PluginTests::Options options) { - return parseCommandLine (createCommandLineArgs (cmd)); + return settings_parser::createChildProcessCommandLine (fileOrID, options); } -juce::StringArray createCommandLine (juce::String fileOrID, PluginTests::Options options) +//============================================================================== +void performCommandLine (CommandLineValidator& validator, const juce::String& commandLine) { - juce::StringArray args (juce::File::getSpecialLocation (juce::File::currentExecutableFile).getFullPathName()); - const PluginTests::Options defaults; - - if (options.strictnessLevel != defaults.strictnessLevel) - args.addArray ({ "--strictness-level", juce::String (options.strictnessLevel) }); - - if (options.randomSeed != defaults.randomSeed) - args.addArray ({ "--random-seed", juce::String (options.randomSeed) }); - - if (options.timeoutMs != defaults.timeoutMs) - args.addArray ({ "--timeout-ms", juce::String (options.timeoutMs) }); - - if (options.verbose) - args.add ("--verbose"); - - if (! options.withGUI) - args.add ("--skip-gui-tests"); - - if (options.numRepeats != defaults.numRepeats) - args.addArray ({ "--repeat", juce::String (options.numRepeats) }); - - if (options.randomiseTestOrder) - args.add ("--randomise"); - - if (options.dataFile != defaults.dataFile) - args.addArray ({ "--data-file", options.dataFile.getFullPathName() }); - - if (options.outputDir != defaults.outputDir) - args.addArray ({ "--output-dir", options.outputDir.getFullPathName() }); + hideDockIcon(); - if (options.outputFilename != defaults.outputFilename) - args.addArray ({ "--output-filename", options.outputFilename }); + auto& app = *juce::JUCEApplication::getInstance(); + const auto tokens = settings_parser::preprocess (commandLine); - if (options.disabledTests != defaults.disabledTests) - args.addArray ({ "--disabled-tests", options.disabledTests.joinIntoString (",") }); + if (tokens.contains ("--run-tests")) + { + runUnitTests(); + app.quit(); + return; + } - if (! options.sampleRates.empty()) + if (tokens.contains ("--strictness-help")) { - juce::StringArray sampleRates; + int level = 5; - for (auto rate : options.sampleRates) - sampleRates.add (juce::String (rate)); + if (const auto idx = tokens.indexOf ("--strictness-help"); idx >= 0 && idx + 1 < tokens.size()) + if (const auto next = tokens[idx + 1]; ! next.startsWith ("-")) + level = next.getIntValue(); - args.addArray ({ "--sample-rates", sampleRates.joinIntoString (",") }); + printStrictnessHelp (level); + app.quit(); + return; } - if (! options.blockSizes.empty()) + // Otherwise this is a validation run (explicit or implicit --validate). + // CLI11 handles --help/--version and parse errors. + try { - juce::StringArray blockSizes; + const auto result = settings_parser::parseTokens (tokens); - for (auto size : options.blockSizes) - blockSizes.add (juce::String (size)); + if (result.handled) + { + app.setApplicationReturnValue (result.exitCode); + app.quit(); + return; + } - args.addArray ({ "--block-sizes", blockSizes.joinIntoString (",") }); - } + const auto fileOrID = juce::String (result.settings.validatePath); - if (auto rtCheckMode = options.realtimeCheck; - rtCheckMode != RealtimeCheck::disabled) + if (fileOrID.isEmpty()) + { + exitWithError ("*** FAILED: No plugin path or ID specified to validate"); + return; + } + + // --validate runs async so will quit itself when done + validator.validate (fileOrID, result.settings.toPluginTestOptions()); + } + catch (const std::exception& e) { - args.addArray ({ "--rtcheck", std::string (magic_enum::enum_name (rtCheckMode)) }); + exitWithError (juce::String ("*** FAILED: ") + e.what()); } +} - args.addArray ({ "--validate", fileOrID }); - - return args; +bool shouldPerformCommandLine (const juce::String& commandLine) +{ + return settings_parser::isCommandLine (settings_parser::preprocess (commandLine)); } //============================================================================== diff --git a/Source/CommandLine.h b/Source/CommandLine.h index 644eb55a..c19f69f1 100644 --- a/Source/CommandLine.h +++ b/Source/CommandLine.h @@ -34,6 +34,8 @@ void performCommandLine (CommandLineValidator&, const juce::String& commandLine) bool shouldPerformCommandLine (const juce::String& commandLine); //============================================================================== +/** Parses a command line into the plugin path/ID and resolved test options. */ std::pair parseCommandLine (const juce::String&); -std::pair parseCommandLine (const juce::ArgumentList&); + +/** Serialises options for the child validation process. */ juce::StringArray createCommandLine (juce::String fileOrID, PluginTests::Options); diff --git a/Source/CommandLineTests.cpp b/Source/CommandLineTests.cpp index 9be45214..e537a296 100644 --- a/Source/CommandLineTests.cpp +++ b/Source/CommandLineTests.cpp @@ -12,6 +12,7 @@ ==============================================================================*/ +#include struct CommandLineTests : public juce::UnitTest { @@ -20,146 +21,315 @@ struct CommandLineTests : public juce::UnitTest { } - void runTest() override + static settings_parser::EnvProvider emptyEnv() { + return [] (const juce::String&) { return juce::String(); }; + } - beginTest ("Merge environment variables"); - { - juce::StringPairArray envVars; - envVars.set ("STRICTNESS_LEVEL", "5"); - envVars.set ("RANDOM_SEED", "1234"); - envVars.set ("TIMEOUT_MS", "30000"); - envVars.set ("VERBOSE", "1"); - envVars.set ("REPEAT", "10"); - envVars.set ("RANDOMISE", "1"); - envVars.set ("SKIP_GUI_TESTS", "1"); - envVars.set ("DATA_FILE", ""); - envVars.set ("OUTPUT_DIR", ""); - - const auto merged = mergeEnvironmentVariables (juce::String(), [&envVars] (const juce::String& n, const juce::String& def) { return envVars.getValue (n, def); }).joinIntoString (" "); - expect (merged.contains ("--strictness-level 5")); - expect (merged.contains ("--random-seed 1234")); - expect (merged.contains ("--timeout-ms 30000")); - expect (merged.contains ("--verbose")); - expect (merged.contains ("--repeat 10")); - expect (merged.contains ("--randomise")); - expect (merged.contains ("--skip-gui-tests")); - expect (merged.contains ("--data-file ")); - expect (merged.contains ("--output-dir ")); - } + static settings_parser::EnvProvider envFrom (std::map vars) + { + return [vars = std::move (vars)] (const juce::String& name) + { + const auto it = vars.find (name); + return it != vars.end() ? it->second : juce::String(); + }; + } + + static PluginvalSettings parse (const juce::String& cmd, settings_parser::EnvProvider env) + { + return settings_parser::parse (cmd, env); + } + + static PluginvalSettings parse (const juce::String& cmd) + { + return settings_parser::parse (cmd, emptyEnv()); + } + void runTest() override + { beginTest ("Command line defaults"); { - juce::ArgumentList args ({}, ""); - expectEquals (getStrictnessLevel (args), 5); - expectEquals (getRandomSeed (args), (juce::int64) 0); - expectEquals (getTimeout (args), (juce::int64) 30000); - expectEquals (getNumRepeats (args), 1); - expectEquals (getOptionValue (args, "--data-file", {}, "Missing data-file path argument!").toString(), juce::String()); - expectEquals (getOptionValue (args, "--output-dir", {}, "Missing output-dir path argument!").toString(), juce::String()); + const auto opts = parse ("").toPluginTestOptions(); + expectEquals (opts.strictnessLevel, 5); + expectEquals (opts.randomSeed, (juce::int64) 0); + expectEquals (opts.timeoutMs, (juce::int64) 30000); + expectEquals (opts.numRepeats, 1); + expect (opts.verbose == false); + expect (opts.randomiseTestOrder == false); + expect (opts.withGUI == true); + expect (opts.realtimeCheck == RealtimeCheck::disabled); + expect (opts.dataFile == juce::File()); + expect (opts.outputDir == juce::File()); + expect (opts.sampleRates == PluginvalSettings::defaultSampleRates()); + expect (opts.blockSizes == PluginvalSettings::defaultBlockSizes()); } beginTest ("Command line parser"); { - juce::ArgumentList args ({}, "--strictness-level 7 --random-seed 1234 --timeout-ms 20000 --repeat 11 --data-file /path/to/file --output-dir /path/to/dir --validate /path/to/plugin"); - expectEquals (getStrictnessLevel (args), 7); - expectEquals (getRandomSeed (args), (juce::int64) 1234); - expectEquals (getTimeout (args), (juce::int64) 20000); - expectEquals (getNumRepeats (args), 11); - expectEquals (getOptionValue (args, "--data-file", {}, "Missing data-file path argument!").toString(),juce::String ("/path/to/file")); - expectEquals (getOptionValue (args, "--output-dir", {}, "Missing output-dir path argument!").toString(),juce::String ("/path/to/dir")); - expectEquals (getOptionValue (args, "--validate", {}, "Missing validate argument!").toString(),juce::String ("/path/to/plugin")); + const auto settings = parse ("--strictness-level 7 --random-seed 1234 --timeout-ms 20000 --repeat 11 " + "--data-file /path/to/file --output-dir /path/to/dir --validate /path/to/plugin"); + const auto opts = settings.toPluginTestOptions(); + expectEquals (opts.strictnessLevel, 7); + expectEquals (opts.randomSeed, (juce::int64) 1234); + expectEquals (opts.timeoutMs, (juce::int64) 20000); + expectEquals (opts.numRepeats, 11); + // Compare the raw parsed strings: juce::File would normalise these to the + // current drive on Windows (e.g. "D:\path\to\file"). + expectEquals (juce::String (settings.dataFile), juce::String ("/path/to/file")); + expectEquals (juce::String (settings.outputDir), juce::String ("/path/to/dir")); + expectEquals (juce::String (settings.validatePath), juce::String ("/path/to/plugin")); + } + + beginTest ("Negative timeout"); + { + expectEquals ((juce::int64) parse ("--timeout-ms -1 --validate x").timeoutMs, (juce::int64) -1); + } + + beginTest ("Command line random (hex and int)"); + { + expectEquals ((juce::int64) parse ("--random-seed 0x7f2da1 --validate x").randomSeed, (juce::int64) 8334753); + expectEquals ((juce::int64) parse ("--random-seed 0x692bc1f --validate x").randomSeed, (juce::int64) 110279711); + expectEquals ((juce::int64) parse ("--random-seed 1234 --validate x").randomSeed, (juce::int64) 1234); + } + + beginTest ("Comma-separated lists"); + { + const auto opts = parse ("--sample-rates 22050,44100 --block-sizes 32,64,128 --validate x").toPluginTestOptions(); + expect (opts.sampleRates == std::vector ({ 22050.0, 44100.0 })); + expect (opts.blockSizes == std::vector ({ 32, 64, 128 })); + } + + beginTest ("rtcheck enum parsing"); + { + expect (parse ("--rtcheck relaxed --validate x").realtimeCheck == RealtimeCheck::relaxed); + expect (parse ("--rtcheck enabled --validate x").realtimeCheck == RealtimeCheck::enabled); + expect (parse ("--validate x").realtimeCheck == RealtimeCheck::disabled); } beginTest ("Handles an absolute path to the plugin"); { const auto homeDir = juce::File::getSpecialLocation (juce::File::userHomeDirectory).getFullPathName(); - const auto commandLineString = "--validate " + homeDir + "/path/to/MyPlugin"; - const auto args = createCommandLineArgs (commandLineString); - expectEquals (parseCommandLine (args).first, homeDir + "/path/to/MyPlugin"); + expectEquals (juce::String (parse ("--validate " + homeDir + "/path/to/MyPlugin").validatePath), + homeDir + "/path/to/MyPlugin"); } beginTest ("Handles a quoted absolute path to the plugin"); { const auto homeDir = juce::File::getSpecialLocation (juce::File::userHomeDirectory).getFullPathName(); const auto pathToQuote = homeDir + "/path/to/MyPlugin"; - const auto commandLineString = "--validate " + pathToQuote.quoted(); - const auto args = createCommandLineArgs (commandLineString); - expectEquals (parseCommandLine (args).first, homeDir + "/path/to/MyPlugin"); + expectEquals (juce::String (parse ("--validate " + pathToQuote.quoted()).validatePath), + homeDir + "/path/to/MyPlugin"); } beginTest ("Handles a relative path"); { const auto currentDir = juce::File::getCurrentWorkingDirectory(); - const auto args = createCommandLineArgs ("--validate MyPlugin.vst3"); - expectEquals (parseCommandLine (args).first, currentDir.getChildFile ("MyPlugin.vst3").getFullPathName()); + expectEquals (juce::String (parse ("--validate MyPlugin.vst3").validatePath), + currentDir.getChildFile ("MyPlugin.vst3").getFullPathName()); } beginTest ("Handles a quoted relative path with spaces to the plugin"); { const auto currentDir = juce::File::getCurrentWorkingDirectory(); - const auto args = createCommandLineArgs (R"(--validate "My Plugin.vst3")"); - expectEquals (parseCommandLine (args).first, currentDir.getChildFile ("My Plugin.vst3").getFullPathName()); + expectEquals (juce::String (parse (R"(--validate "My Plugin.vst3")").validatePath), + currentDir.getChildFile ("My Plugin.vst3").getFullPathName()); } - #if !JUCE_WINDOWS - + #if !JUCE_WINDOWS beginTest ("Handles a relative path with ./ to the plugin"); { const auto currentDir = juce::File::getCurrentWorkingDirectory().getFullPathName(); - const auto commandLineString = "--validate ./path/to/MyPlugin"; - const auto args = createCommandLineArgs(commandLineString); - expectEquals (parseCommandLine (args).first, currentDir + "/path/to/MyPlugin"); + expectEquals (juce::String (parse ("--validate ./path/to/MyPlugin").validatePath), + currentDir + "/path/to/MyPlugin"); } beginTest ("Handles a home directory relative path to the plugin"); { - const auto commandLineString = "--validate ~/path/to/MyPlugin"; - const auto args = createCommandLineArgs(commandLineString); - expectEquals (parseCommandLine (args).first, juce::File::getSpecialLocation (juce::File::userHomeDirectory).getFullPathName() + "/path/to/MyPlugin"); + expectEquals (juce::String (parse ("--validate ~/path/to/MyPlugin").validatePath), + juce::File::getSpecialLocation (juce::File::userHomeDirectory).getFullPathName() + "/path/to/MyPlugin"); } beginTest ("Handles quoted strings, spaces, and home directory relative path to the plugin"); { - const auto commandLineString = R"(--data-file "~/path/to/My File" --output-dir "~/path/to/My Directory" --validate "~/path/to/My Plugin")"; - const auto args = createCommandLineArgs(commandLineString); - expectEquals (parseCommandLine (args).first, juce::File::getSpecialLocation (juce::File::userHomeDirectory).getFullPathName() + "/path/to/My Plugin"); + const auto cmd = R"(--data-file "~/path/to/My File" --output-dir "~/path/to/My Directory" --validate "~/path/to/My Plugin")"; + expectEquals (juce::String (parse (cmd).validatePath), + juce::File::getSpecialLocation (juce::File::userHomeDirectory).getFullPathName() + "/path/to/My Plugin"); } - #endif + #endif beginTest ("Implicit validate with a relative path"); { const auto currentDir = juce::File::getCurrentWorkingDirectory(); - const auto args = createCommandLineArgs ("MyPlugin.vst3"); - expectEquals (parseCommandLine (args).first, currentDir.getChildFile ("MyPlugin.vst3").getFullPathName()); + expectEquals (juce::String (parse ("MyPlugin.vst3").validatePath), + currentDir.getChildFile ("MyPlugin.vst3").getFullPathName()); } beginTest ("Doesn't alter component IDs"); { - const auto commandLineString = "--validate MyPluginID"; - const auto args = createCommandLineArgs(commandLineString); - expectEquals (parseCommandLine (args).first,juce::String ("MyPluginID")); + expectEquals (juce::String (parse ("--validate MyPluginID").validatePath), juce::String ("MyPluginID")); } - beginTest ("Command line random"); + beginTest ("Allows for other options after explicit --validate"); { - expectEquals (getRandomSeed (juce::ArgumentList ({}, "--random-seed 0x7f2da1")), (juce::int64) 8334753); - expectEquals (getRandomSeed (juce::ArgumentList ({}, "--random-seed 0x692bc1f")), (juce::int64) 110279711); + const auto currentDir = juce::File::getCurrentWorkingDirectory(); + const auto settings = parse ("--validate MyPlugin.vst3 --randomise"); + expectEquals (juce::String (settings.validatePath), currentDir.getChildFile ("MyPlugin.vst3").getFullPathName()); + expect (settings.randomiseTestOrder); } - beginTest ("Implicit validate options"); + beginTest ("Should perform command line"); { juce::TemporaryFile temp ("path_to_file.vst3"); expect (temp.getFile().create()); expect (shouldPerformCommandLine (temp.getFile().getFullPathName())); + expect (shouldPerformCommandLine ("--run-tests")); + expect (shouldPerformCommandLine ("--version")); + expect (! shouldPerformCommandLine ("")); } - beginTest ("Allows for other options after explicit --validate"); + beginTest ("Environment variables"); { - const auto currentDir = juce::File::getCurrentWorkingDirectory(); - const auto args = createCommandLineArgs ("--validate MyPlugin.vst3 --randomise"); - expectEquals (parseCommandLine (args).first, currentDir.getChildFile ("MyPlugin.vst3").getFullPathName()); - expect (parseCommandLine(args).second.randomiseTestOrder); + const auto env = envFrom ({ + { "STRICTNESS_LEVEL", "7" }, + { "RANDOM_SEED", "1234" }, + { "TIMEOUT_MS", "20000" }, + { "VERBOSE", "1" }, + { "REPEAT", "11" }, + { "RANDOMISE", "1" }, + { "SKIP_GUI_TESTS", "1" }, + { "DATA_FILE", "/path/to/file" }, + { "OUTPUT_DIR", "/path/to/dir" }, + { "SAMPLE_RATES", "22050,44100" }, + { "BLOCK_SIZES", "32,64" }, + { "RTCHECK", "relaxed" }, + }); + + const auto settings = parse ("--validate x", env); + const auto opts = settings.toPluginTestOptions(); + expectEquals (opts.strictnessLevel, 7); + expectEquals (opts.randomSeed, (juce::int64) 1234); + expectEquals (opts.timeoutMs, (juce::int64) 20000); + expect (opts.verbose); + expectEquals (opts.numRepeats, 11); + expect (opts.randomiseTestOrder); + expect (opts.withGUI == false); + expectEquals (juce::String (settings.dataFile), juce::String ("/path/to/file")); + expect (opts.sampleRates == std::vector ({ 22050.0, 44100.0 })); + expect (opts.blockSizes == std::vector ({ 32, 64 })); + expect (opts.realtimeCheck == RealtimeCheck::relaxed); + } + + beginTest ("Command line overrides environment variables"); + { + const auto env = envFrom ({ { "STRICTNESS_LEVEL", "3" } }); + expectEquals (parse ("--validate x", env).strictnessLevel, 3); // env only + expectEquals (parse ("--strictness-level 9 --validate x", env).strictnessLevel, 9); // CLI wins + } + + beginTest ("Precedence: CLI > --config > env > defaults"); + { + juce::TemporaryFile configFile (".json"); + configFile.getFile().replaceWithText (R"({ "strictnessLevel": 2, "timeoutMs": 12345 })"); + const auto cfg = "--config " + configFile.getFile().getFullPathName().quoted(); + + const auto env6 = envFrom ({ { "STRICTNESS_LEVEL", "6" } }); + + // env beats defaults + expectEquals (parse ("--validate x", env6).strictnessLevel, 6); + + // config alone + { + const auto s = parse (cfg + " --validate x"); + expectEquals (s.strictnessLevel, 2); + expectEquals ((juce::int64) s.timeoutMs, (juce::int64) 12345); + } + + // config beats env + { + const auto s = parse (cfg + " --validate x", env6); + expectEquals (s.strictnessLevel, 2); // config wins over env + expectEquals ((juce::int64) s.timeoutMs, (juce::int64) 12345); + } + + // CLI beats config and env + { + const auto s = parse (cfg + " --strictness-level 9 --validate x", env6); + expectEquals (s.strictnessLevel, 9); + expectEquals ((juce::int64) s.timeoutMs, (juce::int64) 12345); + } + } + + beginTest ("Repeatable --config merges per key, last wins"); + { + juce::TemporaryFile baseFile (".json"); + baseFile.getFile().replaceWithText (R"({ "strictnessLevel": 2, "timeoutMs": 11111 })"); + + juce::TemporaryFile overrideFile (".json"); + overrideFile.getFile().replaceWithText (R"({ "strictnessLevel": 8 })"); + + const auto cmd = "--config " + baseFile.getFile().getFullPathName().quoted() + + " --config " + overrideFile.getFile().getFullPathName().quoted() + + " --validate x"; + + const auto s = parse (cmd); + expectEquals (s.strictnessLevel, 8); // overridden by the second file + expectEquals ((juce::int64) s.timeoutMs, (juce::int64) 11111); // untouched, kept from the first + } + + beginTest ("Child-process command line round-trip"); + { + PluginTests::Options opts; + opts.strictnessLevel = 8; + opts.randomSeed = 8334753; + opts.timeoutMs = 15000; + opts.verbose = true; + opts.numRepeats = 3; + opts.randomiseTestOrder = true; + opts.withGUI = false; + opts.outputFilename = "log.txt"; + opts.disabledTests = juce::StringArray ({ "Test A", "Test B" }); + opts.sampleRates = { 44100.0, 96000.0 }; + opts.blockSizes = { 64, 512 }; + opts.realtimeCheck = RealtimeCheck::relaxed; + + const juce::String fileOrID = "/some/dir/MyPlugin.vst3"; + + const auto args = createCommandLine (fileOrID, opts); + + juce::StringArray childArgs (args); + childArgs.remove (0); // drop the executable path + const auto childCommandLine = childArgs.joinIntoString (" "); + + const auto [fileOrID2, opts2] = parseCommandLine (childCommandLine); + + auto expected = PluginvalSettings::fromPluginTestOptions (opts, fileOrID); + expected.validatePath = settings_parser::resolvePluginPath (fileOrID).toStdString(); + + expect (PluginvalSettings::fromPluginTestOptions (opts2, fileOrID2) == expected); + } + + beginTest ("--config-base64 rejects extra options"); + { + PluginTests::Options opts; + opts.strictnessLevel = 7; + + juce::StringArray childArgs (createCommandLine ("/some/MyPlugin.vst3", opts)); + childArgs.remove (0); // drop the executable path + + // The legitimate parent -> child handoff parses fine. + { + const auto r = settings_parser::parseTokens (childArgs, emptyEnv()); + expect (! r.handled); + expectEquals (r.settings.strictnessLevel, 7); + } + + // Combining it with any other option is rejected. + { + childArgs.addArray ({ "--strictness-level", "9" }); + const auto r = settings_parser::parseTokens (childArgs, emptyEnv()); + expect (r.handled); + expectEquals (r.exitCode, 1); + } } } }; diff --git a/Source/PluginvalSettings.h b/Source/PluginvalSettings.h new file mode 100644 index 00000000..e8a82558 --- /dev/null +++ b/Source/PluginvalSettings.h @@ -0,0 +1,142 @@ +/*============================================================================== + + Copyright 2018 by Tracktion Corporation. + For more information visit www.tracktion.com + + You may also use this code under the terms of the GPL v3 (see + www.gnu.org/licenses). + + pluginval IS PROVIDED "AS IS" WITHOUT ANY WARRANTY, AND ALL WARRANTIES, WHETHER + EXPRESSED OR IMPLIED, INCLUDING MERCHANTABILITY AND FITNESS FOR PURPOSE, ARE + DISCLAIMED. + + ==============================================================================*/ + +#pragma once + +#include "PluginTests.h" +#include + +#include +#include +#include + +//============================================================================== +/** Maps the RealtimeCheck enum to/from its JSON string representation. + An unrecognised/empty string deserialises to the first entry (disabled), + preserving the historical "empty -> disabled" behaviour. +*/ +NLOHMANN_JSON_SERIALIZE_ENUM (RealtimeCheck, +{ + { RealtimeCheck::disabled, "disabled" }, + { RealtimeCheck::enabled, "enabled" }, + { RealtimeCheck::relaxed, "relaxed" }, +}) + +//============================================================================== +/** + The single, parser-agnostic settings struct. + + Every field uses std types so it can be (de)serialised by nlohmann/json and + is decoupled from JUCE. JSON keys mirror the field names below. Conversion to + the JUCE-flavoured PluginTests::Options happens only at the boundary via + toPluginTestOptions(). +*/ +struct PluginvalSettings +{ + int strictnessLevel = 5; + std::int64_t randomSeed = 0; + std::int64_t timeoutMs = 30000; + bool verbose = false; + int numRepeats = 1; + bool randomiseTestOrder = false; + bool skipGuiTests = false; + std::string dataFile; + std::string outputDir; + std::string outputFilename; + std::vector disabledTests; + std::vector sampleRates; + std::vector blockSizes; + RealtimeCheck realtimeCheck = RealtimeCheck::disabled; + std::string validatePath; + + bool operator== (const PluginvalSettings&) const = default; + + //============================================================================== + /** The default sample rates used when none are specified. */ + static std::vector defaultSampleRates() { return { 44100.0, 48000.0, 96000.0 }; } + /** The default block sizes used when none are specified. */ + static std::vector defaultBlockSizes() { return { 64, 128, 256, 512, 1024 }; } + + //============================================================================== + /** Converts to the JUCE-flavoured options consumed by PluginTests. */ + PluginTests::Options toPluginTestOptions() const + { + PluginTests::Options o; + o.strictnessLevel = juce::jlimit (1, 10, strictnessLevel); + o.randomSeed = randomSeed; + o.timeoutMs = timeoutMs; + o.verbose = verbose; + o.numRepeats = juce::jmax (1, numRepeats); + o.randomiseTestOrder = randomiseTestOrder; + o.withGUI = ! skipGuiTests; + o.dataFile = juce::String (dataFile); + o.outputDir = juce::String (outputDir); + o.outputFilename = juce::String (outputFilename); + o.disabledTests = toStringArray (disabledTests); + o.sampleRates = sampleRates.empty() ? defaultSampleRates() : sampleRates; + o.blockSizes = blockSizes.empty() ? defaultBlockSizes() : blockSizes; + o.realtimeCheck = realtimeCheck; + return o; + } + + /** Builds settings from a resolved PluginTests::Options + plugin path. + Used to serialise options for the child validation process. + */ + static PluginvalSettings fromPluginTestOptions (const PluginTests::Options& o, const juce::String& fileOrID) + { + PluginvalSettings s; + s.strictnessLevel = o.strictnessLevel; + s.randomSeed = o.randomSeed; + s.timeoutMs = o.timeoutMs; + s.verbose = o.verbose; + s.numRepeats = o.numRepeats; + s.randomiseTestOrder = o.randomiseTestOrder; + s.skipGuiTests = ! o.withGUI; + s.dataFile = o.dataFile.getFullPathName().toStdString(); + s.outputDir = o.outputDir.getFullPathName().toStdString(); + s.outputFilename = o.outputFilename.toStdString(); + s.disabledTests = fromStringArray (o.disabledTests); + s.sampleRates = o.sampleRates; + s.blockSizes = o.blockSizes; + s.realtimeCheck = o.realtimeCheck; + s.validatePath = fileOrID.toStdString(); + return s; + } + +private: + static juce::StringArray toStringArray (const std::vector& v) + { + juce::StringArray a; + for (const auto& s : v) + a.add (juce::String (s)); + return a; + } + + static std::vector fromStringArray (const juce::StringArray& a) + { + std::vector v; + for (const auto& s : a) + v.push_back (s.toStdString()); + return v; + } +}; + +//============================================================================== +// to_json/from_json with per-field defaults: keys missing from the JSON fall +// back to a default-constructed PluginvalSettings rather than value-zero. This +// makes "defaults" a free layer and lets sparse JSON layers merge cleanly. +NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE_WITH_DEFAULT (PluginvalSettings, + strictnessLevel, randomSeed, timeoutMs, verbose, numRepeats, + randomiseTestOrder, skipGuiTests, dataFile, outputDir, outputFilename, + disabledTests, sampleRates, blockSizes, realtimeCheck, validatePath) diff --git a/Source/SettingsParser.cpp b/Source/SettingsParser.cpp new file mode 100644 index 00000000..244286aa --- /dev/null +++ b/Source/SettingsParser.cpp @@ -0,0 +1,409 @@ +/*============================================================================== + + Copyright 2018 by Tracktion Corporation. + For more information visit www.tracktion.com + + You may also use this code under the terms of the GPL v3 (see + www.gnu.org/licenses). + + pluginval IS PROVIDED "AS IS" WITHOUT ANY WARRANTY, AND ALL WARRANTIES, WHETHER + EXPRESSED OR IMPLIED, INCLUDING MERCHANTABILITY AND FITNESS FOR PURPOSE, ARE + DISCLAIMED. + + ==============================================================================*/ + +// CLI11 must be included before the JUCE headers: on Linux JUCE pulls in the X11 +// headers (JUCE_GUI_BASICS_INCLUDE_XHEADERS), which #define Success/None/Bool/etc. +// and would clash with CLI11's CLI::ExitCodes::Success enumerator. +#include + +#include "SettingsParser.h" +#include "SettingsSerializer.h" + +#include +#include +#include +#include +#include +#include + +namespace settings_parser +{ + juce::String systemEnv (const juce::String& name) + { + return juce::SystemStats::getEnvironmentVariable (name, {}); + } + + juce::String getVersionString() + { + return juce::String ("pluginval") + " - " + VERSION; + } + + //============================================================================== + namespace + { + bool isPluginArgument (const juce::String& arg) + { + juce::AudioPluginFormatManager formatManager; + #if JUCE_VERSION >= 0x08000B + juce::addDefaultFormatsToManager (formatManager); + #else + formatManager.addDefaultFormats(); + #endif + + for (auto format : formatManager.getFormats()) + if (format->fileMightContainThisPluginType (arg)) + return true; + + // The above checks the file exists, which isn't what we want for CLI parsing + if (auto f = juce::File::createFileWithoutCheckingPath (arg); + f.hasFileExtension (".vst3") + #if JUCE_PLUGINHOST_VST + || f.hasFileExtension (".dll") + #endif + ) + return true; + + return false; + } + + juce::String valueForOption (const juce::StringArray& tokens, juce::StringRef option) + { + const juce::String prefix = juce::String (option) + "="; + + for (int i = 0; i < tokens.size(); ++i) + { + if (tokens[i] == option) + return i + 1 < tokens.size() ? tokens[i + 1] : juce::String(); + + if (tokens[i].startsWith (prefix)) + return tokens[i].substring (prefix.length()); + } + + return {}; + } + + /** Returns every value supplied for an option, in command-line order. */ + juce::StringArray allValuesForOption (const juce::StringArray& tokens, juce::StringRef option) + { + juce::StringArray out; + const juce::String prefix = juce::String (option) + "="; + + for (int i = 0; i < tokens.size(); ++i) + { + if (tokens[i] == option) + { + if (i + 1 < tokens.size()) + out.add (tokens[i + 1]); + } + else if (tokens[i].startsWith (prefix)) + { + out.add (tokens[i].substring (prefix.length())); + } + } + + return out; + } + + bool hasOption (const juce::StringArray& tokens, juce::StringRef option) + { + const juce::String prefix = juce::String (option) + "="; + + for (const auto& t : tokens) + if (t == option || t.startsWith (prefix)) + return true; + + return false; + } + + juce::String decodeBase64 (const juce::String& b64) + { + juce::MemoryOutputStream mos; + juce::Base64::convertFromBase64 (mos, b64); + return mos.toString(); + } + + juce::String getFooterText() + { + return juce::SystemStats::getJUCEVersion() + "\n\n" + juce::String ( +R"(Other commands: + --run-tests Run the internal unit tests. + --strictness-help [level] List all tests that run at the given strictness level. + +Exit code: + 0 if all tests complete successfully + 1 if there are any errors + +You can also specify any option as an environment variable by removing the prefix +dashes, converting internal dashes to underscores and capitalising, e.g. + "--skip-gui-tests" -> "SKIP_GUI_TESTS=1" + "--timeout-ms 30000" -> "TIMEOUT_MS=30000" +Precedence (lowest to highest): defaults, environment variables, --config, command-line options. +--config is repeatable; later files win per key.)"); + } + + std::string toEnvName (const std::string& longName) + { + return juce::String (longName).toUpperCase().replace ("-", "_").toStdString(); + } + + /** Loads one --config JSON file. */ + nlohmann::json loadConfigFile (const juce::String& path) + { + const juce::File file (path); + + if (! file.existsAsFile()) + throw std::runtime_error (("--config file not found: " + path).toStdString()); + + return nlohmann::json::parse (file.loadFileAsString().toStdString()); + } + + /** Registers every option, bound to s. No ->envname(): the environment is a + separate, lower-precedence layer (see parseTokens). */ + void configureApp (CLI::App& app, PluginvalSettings& s) + { + app.set_version_flag ("--version", getVersionString().toStdString()); + app.footer (getFooterText().toStdString()); + + // Accepted so the CLI parse doesn't error on --config; the values are + // handled manually (repeatable, inline-or-file) in parseTokens. + app.add_option_function> ("--config", + [] (const std::vector&) {}, + "Path to a JSON settings file. Repeatable; later files win per key.")->take_all(); + + app.add_option ("--validate", s.validatePath, "Validates the plugin at the given path (or AU id)."); + app.add_option ("--strictness-level", s.strictnessLevel, "Strictness level 1-10 (default 5)."); + app.add_option ("--timeout-ms", s.timeoutMs, "Test timeout in ms (default 30000, -1 to never timeout)."); + app.add_option ("--repeat", s.numRepeats, "Number of times to repeat the tests."); + app.add_flag ("--randomise", s.randomiseTestOrder, "Run the tests in a random order per repeat."); + app.add_flag ("--verbose", s.verbose, "Output additional logging information."); + app.add_flag ("--skip-gui-tests", s.skipGuiTests, "Avoid tests that create GUI windows (for headless CI)."); + app.add_option ("--sample-rates", s.sampleRates, "Comma-separated sample rates (default 44100,48000,96000).")->delimiter (','); + app.add_option ("--block-sizes", s.blockSizes, "Comma-separated block sizes (default 64,128,256,512,1024).")->delimiter (','); + app.add_option ("--data-file", s.dataFile, "Path to a data file tests can use to configure themselves."); + app.add_option ("--output-dir", s.outputDir, "Directory in which to write the log files."); + app.add_option ("--output-filename", s.outputFilename, "Filename to write logs into."); + + app.add_option_function ("--disabled-tests", + [&s] (const std::string& v) { s.disabledTests = settings_serializer::disabledTestsToList (juce::String (v)); }, + "Comma-separated test names, or a path to a file listing them."); + + app.add_option_function ("--random-seed", + [&s] (const std::string& v) + { + try { s.randomSeed = settings_serializer::parseRandomSeed (juce::String (v)); } + catch (const std::exception& e) { throw CLI::ValidationError ("--random-seed", e.what()); } + }, + "Random seed (hex 0x.. or int) for replicable test runs."); + + app.add_option ("--rtcheck", s.realtimeCheck, "Real-time safety checks: disabled, enabled or relaxed.") + ->transform (CLI::CheckedTransformer (std::map { + { "disabled", RealtimeCheck::disabled }, + { "enabled", RealtimeCheck::enabled }, + { "relaxed", RealtimeCheck::relaxed } }, CLI::ignore_case)); + } + + /** Builds a synthetic argv from the environment by deriving an env-var name + from each registered option (e.g. --strictness-level -> STRICTNESS_LEVEL). + CLI11 then parses and coerces it like any other argument. */ + std::vector buildEnvArgv (const CLI::App& app, const EnvProvider& env) + { + std::vector argv { "pluginval" }; + + for (const auto* opt : app.get_options()) + { + const auto& lnames = opt->get_lnames(); + + if (lnames.empty()) + continue; + + const auto& lname = lnames.front(); + + // Skip the meta options that shouldn't be environment-driven. + if (lname == "help" || lname == "version" || lname == "config" || lname == "validate") + continue; + + if (const auto value = env (juce::String (toEnvName (lname))); value.isNotEmpty()) + argv.push_back ("--" + lname + "=" + value.toStdString()); // works for flags too (--flag=1/0) + } + + return argv; + } + + /** Runs one parse pass. Returns an exit code if the parse was "handled" + (help/version/error), or nullopt on success. */ + std::optional runParse (CLI::App& app, const std::vector& argv) + { + std::vector cargv; + cargv.reserve (argv.size()); + + for (const auto& a : argv) + cargv.push_back (a.c_str()); + + try + { + app.parse ((int) cargv.size(), cargv.data()); + } + catch (const CLI::ParseError& e) + { + return app.exit (e); + } + + return std::nullopt; + } + } + + //============================================================================== + juce::StringArray preprocess (const juce::String& commandLineIn) + { + if (commandLineIn.contains ("strictnessLevel")) + { + std::cout << "!!! WARNING:\n\t\"strictnessLevel\" is deprecated and will be removed in a future version.\n" + << "\tPlease use --strictness-level instead\n\n"; + } + + const auto commandLine = commandLineIn.replace ("strictnessLevel", "strictness-level") + .replace ("-NSDocumentRevisionsDebugMode YES", "") + .trim(); + + juce::StringArray args; + args.addTokens (commandLine, true); + args.trim(); + + for (auto& s : args) + s = s.unquoted(); + + // If only a plugin path is supplied as the last arg, add an implicit + // --validate option for it so the rest of the CLI works. + if (args.size() > 0) + { + const bool hasCommand = hasOption (args, "--validate") + || args.contains ("--help") || args.contains ("-h") + || args.contains ("--version") + || args.contains ("--run-tests") + || hasOption (args, "--config-base64"); + + if (! hasCommand && isPluginArgument (args[args.size() - 1])) + args.insert (args.size() - 1, "--validate"); + } + + return args; + } + + bool isCommandLine (const juce::StringArray& tokens) + { + return tokens.contains ("--help") || tokens.contains ("-h") + || tokens.contains ("--version") + || hasOption (tokens, "--validate") + || tokens.contains ("--run-tests") + || tokens.contains ("--strictness-help") + || hasOption (tokens, "--config-base64"); + } + + juce::String resolvePluginPath (const juce::String& raw) + { + // Resolve relative/home paths against the working directory. Absolute + // paths and bare component IDs (no '.' or '~') are left untouched. + if (raw.contains ("~") || raw.contains (".")) + return juce::File::getCurrentWorkingDirectory().getChildFile (raw).getFullPathName(); + + return raw; + } + + //============================================================================== + ParseResult parseTokens (const juce::StringArray& tokens, const EnvProvider& env) + { + ParseResult result; + auto& s = result.settings; + + // --config-base64 carries a fully-resolved settings set from the parent + // process and is authoritative. It is for internal use only and must not be + // combined with other options (which would otherwise be silently ignored); + // the only companion allowed is --validate. + if (const auto b64 = valueForOption (tokens, "--config-base64"); b64.isNotEmpty()) + { + for (const auto& token : tokens) + { + if (! token.startsWith ("-") || token == "-") + continue; // a value, not an option + + if (const auto name = token.upToFirstOccurrenceOf ("=", false, false); + name != "--config-base64" && name != "--validate") + { + std::cerr << "*** FAILED: --config-base64 is for internal use and cannot be combined " + << "with other options (got " << name << ")" << std::endl; + result.exitCode = 1; + result.handled = true; + return result; + } + } + + s = settings_serializer::fromJsonString (decodeBase64 (b64).toStdString()); + s.validatePath = resolvePluginPath (juce::String (s.validatePath)).toStdString(); + return result; + } + + // 1. Environment layer (env-var names derived from the registered options). + { + CLI::App envApp; + configureApp (envApp, s); + + if (const auto code = runParse (envApp, buildEnvArgv (envApp, env)); code) + { + result.exitCode = *code; + result.handled = true; + return result; + } + } + + // 2. --config layer: repeatable, inline JSON or file, merged per key in + // command-line order (last wins). Beats the environment, loses to CLI. + if (const auto configs = allValuesForOption (tokens, "--config"); ! configs.isEmpty()) + { + auto merged = settings_serializer::toJson (s); + + for (const auto& source : configs) + merged.merge_patch (loadConfigFile (source)); + + s = merged.get(); + } + + // 3. Command-line layer: the individual options beat everything. + { + CLI::App cliApp { "Validate plugins to test compatibility with hosts and verify plugin API conformance" }; + configureApp (cliApp, s); + + std::vector argv { "pluginval" }; + + for (const auto& t : tokens) + argv.push_back (t.toStdString()); + + if (const auto code = runParse (cliApp, argv); code) + { + result.exitCode = *code; + result.handled = true; + return result; + } + } + + s.validatePath = resolvePluginPath (juce::String (s.validatePath)).toStdString(); + return result; + } + + PluginvalSettings parse (const juce::String& commandLine, const EnvProvider& env) + { + return parseTokens (preprocess (commandLine), env).settings; + } + + //============================================================================== + juce::StringArray createChildProcessCommandLine (const juce::String& fileOrID, const PluginTests::Options& options) + { + const auto settings = PluginvalSettings::fromPluginTestOptions (options, fileOrID); + const auto jsonString = settings_serializer::toJson (settings).dump(); + const auto b64 = juce::Base64::toBase64 (juce::String (jsonString)); + + juce::StringArray args (juce::File::getSpecialLocation (juce::File::currentExecutableFile).getFullPathName()); + args.addArray ({ "--config-base64", b64, "--validate", fileOrID }); + return args; + } +} diff --git a/Source/SettingsParser.h b/Source/SettingsParser.h new file mode 100644 index 00000000..3cd6ebc6 --- /dev/null +++ b/Source/SettingsParser.h @@ -0,0 +1,79 @@ +/*============================================================================== + + Copyright 2018 by Tracktion Corporation. + For more information visit www.tracktion.com + + You may also use this code under the terms of the GPL v3 (see + www.gnu.org/licenses). + + pluginval IS PROVIDED "AS IS" WITHOUT ANY WARRANTY, AND ALL WARRANTIES, WHETHER + EXPRESSED OR IMPLIED, INCLUDING MERCHANTABILITY AND FITNESS FOR PURPOSE, ARE + DISCLAIMED. + + ==============================================================================*/ + +#pragma once + +#include "PluginvalSettings.h" +#include "PluginTests.h" + +#include + +/** + The command-line -> settings pipeline. + + A single PluginvalSettings is filled by successive layers, lowest to highest + precedence: hardcoded defaults, then environment variables, then --config + (repeatable JSON files, later files win per key), then the individual CLI options. + CLI11 binds each option to a member and also provides the coercion used for + the environment layer (env names are derived from the registered options, so + there is no separate env table). Conversion to the JUCE-flavoured + PluginTests::Options happens at the boundary via toPluginTestOptions(). +*/ +namespace settings_parser +{ + //============================================================================== + /** Returns the value of an environment variable, or "" if unset. */ + using EnvProvider = std::function; + juce::String systemEnv (const juce::String& name); + + //============================================================================== + /** Tokenises + preprocesses a raw command line: rewrites the deprecated + "strictnessLevel", strips the macOS "-NSDocumentRevisionsDebugMode YES" + flag, and inserts an implicit "--validate" when the last argument is a + bare plugin path. Returns argv tokens (without the program name). + */ + juce::StringArray preprocess (const juce::String& commandLine); + + /** True if the tokens contain a recognised command that triggers CLI mode. */ + bool isCommandLine (const juce::StringArray& tokens); + + /** Resolves a relative/home plugin path against the working directory, + leaving absolute paths and bare component IDs untouched. + */ + juce::String resolvePluginPath (const juce::String& raw); + + //============================================================================== + /** The outcome of parsing the option tokens. */ + struct ParseResult + { + PluginvalSettings settings; + bool handled = false; /**< true if --help/--version/parse-error was handled and the app should exit. */ + int exitCode = 0; /**< the exit code to use when handled is true. */ + }; + + /** Parses preprocessed option tokens into settings (and handles --help/--version). */ + ParseResult parseTokens (const juce::StringArray& tokens, const EnvProvider& env = systemEnv); + + /** Convenience: preprocess + parse from a raw command line, returning settings. */ + PluginvalSettings parse (const juce::String& commandLine, const EnvProvider& env = systemEnv); + + //============================================================================== + /** Serialises options for the child validation process as a base64 JSON + handoff: { exe, --config-base64 , --validate }. + */ + juce::StringArray createChildProcessCommandLine (const juce::String& fileOrID, const PluginTests::Options&); + + /** The "--version" output string. */ + juce::String getVersionString(); +} diff --git a/Source/SettingsSerializer.cpp b/Source/SettingsSerializer.cpp new file mode 100644 index 00000000..a390b4de --- /dev/null +++ b/Source/SettingsSerializer.cpp @@ -0,0 +1,68 @@ +/*============================================================================== + + Copyright 2018 by Tracktion Corporation. + For more information visit www.tracktion.com + + You may also use this code under the terms of the GPL v3 (see + www.gnu.org/licenses). + + pluginval IS PROVIDED "AS IS" WITHOUT ANY WARRANTY, AND ALL WARRANTIES, WHETHER + EXPRESSED OR IMPLIED, INCLUDING MERCHANTABILITY AND FITNESS FOR PURPOSE, ARE + DISCLAIMED. + + ==============================================================================*/ + +#include "SettingsSerializer.h" + +#include + +namespace settings_serializer +{ + nlohmann::json toJson (const PluginvalSettings& s) + { + return nlohmann::json (s); + } + + PluginvalSettings fromJsonString (const std::string& jsonString) + { + return nlohmann::json::parse (jsonString).get(); + } + + PluginvalSettings fromJsonFile (const juce::File& file) + { + return fromJsonString (file.loadFileAsString().toStdString()); + } + + //============================================================================== + std::int64_t parseRandomSeed (const juce::String& raw) + { + if (! raw.containsOnly ("x-0123456789acbdef")) + throw std::invalid_argument ("Invalid random seed argument!"); + + if (raw.startsWith ("0x")) + return raw.getHexValue64(); + + return raw.getLargeIntValue(); + } + + std::vector disabledTestsToList (const juce::String& raw) + { + std::vector out; + + if (juce::File::isAbsolutePath (raw)) + { + juce::StringArray lines; + juce::File (raw).readLines (lines); + + for (const auto& line : lines) + out.push_back (line.toStdString()); + + return out; + } + + for (const auto& token : juce::StringArray::fromTokens (raw, ",", "")) + out.push_back (token.toStdString()); + + return out; + } +} diff --git a/Source/SettingsSerializer.h b/Source/SettingsSerializer.h new file mode 100644 index 00000000..a5c9336b --- /dev/null +++ b/Source/SettingsSerializer.h @@ -0,0 +1,50 @@ +/*============================================================================== + + Copyright 2018 by Tracktion Corporation. + For more information visit www.tracktion.com + + You may also use this code under the terms of the GPL v3 (see + www.gnu.org/licenses). + + pluginval IS PROVIDED "AS IS" WITHOUT ANY WARRANTY, AND ALL WARRANTIES, WHETHER + EXPRESSED OR IMPLIED, INCLUDING MERCHANTABILITY AND FITNESS FOR PURPOSE, ARE + DISCLAIMED. + + ==============================================================================*/ + +#pragma once + +#include "PluginvalSettings.h" +#include + +#include +#include +#include + +/** + JSON (de)serialisation for PluginvalSettings plus the couple of value + conversions the CLI parser still needs (hex seed, disabled-tests file). +*/ +namespace settings_serializer +{ + //============================================================================== + /** Full JSON representation of a settings set (all fields present). */ + nlohmann::json toJson (const PluginvalSettings&); + + /** Parses a complete settings set from a JSON string (missing keys -> defaults). */ + PluginvalSettings fromJsonString (const std::string&); + + /** Parses a complete settings set from a JSON file (missing keys -> defaults). */ + PluginvalSettings fromJsonFile (const juce::File&); + + //============================================================================== + /** Parses a random seed from "0x..." hex or a decimal integer, preserving the + historical character-set validation. Throws std::invalid_argument if invalid. + */ + std::int64_t parseRandomSeed (const juce::String& raw); + + /** Resolves --disabled-tests: an absolute path is read line-by-line, otherwise + the value is treated as a comma-separated list. + */ + std::vector disabledTestsToList (const juce::String& raw); +} diff --git a/Source/tests/EditorTests.cpp b/Source/tests/EditorTests.cpp index 25fc7c41..8eb66345 100644 --- a/Source/tests/EditorTests.cpp +++ b/Source/tests/EditorTests.cpp @@ -63,7 +63,7 @@ struct EditorDPITest : public PluginTest { if (instance.hasEditor()) { - ScopedDPIAwarenessDisabler scopedDPIAwarenessDisabler; + juce::ScopedDPIAwarenessDisabler scopedDPIAwarenessDisabler; { ut.logMessage ("Testing opening Editor with DPI Awareness disabled"); diff --git a/docs/Command line options.md b/docs/Command line options.md index 4af7339f..4c824692 100644 --- a/docs/Command line options.md +++ b/docs/Command line options.md @@ -1,85 +1,52 @@ -Hello rtcheck! -//============================================================================== -pluginval -JUCE v8.0.10 - -Description: - Validate plugins to test compatibility with hosts and verify plugin API conformance - -Usage: - --version - Print pluginval version. - --validate [pathToPlugin] - Validates the plugin at the given path. - N.B. the "--validate" flag is optional if the path is the last argument. - This enables you to validate a plugin with simply "pluginval path_to_plugin". - - --sample-rates [list of comma separated sample rates] - If specified, sets the list of sample rates at which tests will be executed - (default=44100,48000,96000) - --block-sizes [list of comma separated block sizes] - If specified, sets the list of block sizes at which tests will be executed - (default=64,128,256,512,1024) - --random-seed [hex or int] - Sets the random seed to use for the tests. Useful for replicating test - environments. - --data-file [pathToFile] - If specified, sets a path to a data file which can be used by tests to - configure themselves. This can be useful for things like known audio output. - - --strictness-level [1-10] - Sets the strictness level to use. A minimum level of 5 (also the default) - is recomended for compatibility. - Higher levels include longer, more thorough tests such as fuzzing. - --timeout-ms [numMilliseconds] - Sets a timout which will stop validation with an error if no output from any - test has happened for this number of ms. - By default this is 30s but can be set to "-1" (must be quoted) to never timeout. - --rtcheck [empty, disabled, enabled or relaxed] - Turns on real-time saftey checks using rtcheck (macOS and Linux only). - relaxed mode doesn't run the checks for the first processing block as a lot of plugins - use this to allocate or initialise thread-locals (which can allocate) - - --repeat [num repeats] - If specified repeats the tests a given number of times. Note that this does - not delete and re-instantiate the plugin for each repeat. - --randomise - If specified, the tests are run in a random order per repeat. - - --skip-gui-tests - If specified, avoids tests that create GUI windows, which can cause problems - on headless CI systems. - --disabled-tests [pathToFile] - If specified, sets a path to a file that should have the names of disabled - tests on each row. - --vst3validator [pathToValidator] - If specified, this will run the VST3 validator as part of the test process. - - --output-dir [pathToDir] - If specified, sets a directory to store the log files. This can be useful - for continuous integration. - --output-filename [filename] - If specified, sets a filename for the log files (within 'output-dir' or - (lacking that) the current directory. - By default, the name is constructed from the plugin metainformation - --verbose - If specified, outputs additional logging information. It can be useful to - turn this off when building with CI to avoid huge log files. - -Exit code: - 0 if all tests complete successfully - 1 if there are any errors - -Additionally, you can specify any of the command line options as environment -variables by removing prefix dashes, converting internal dashes to underscores -and capitalising all letters, a.g. - "--skip-gui-tests" > "SKIP_GUI_TESTS=1" - "--timeout-ms 30000" > "TIMEOUT_MS=30000" -Specifying specific command-line options will override any environment variables -set for that option. - - pluginval --version Prints the current version number - pluginval --help|-h Prints the list of commands - pluginval --validate [pathToPlugin] Validates the file (or IDs for AUs). - pluginval --run-tests Runs the internal unit tests. - +Validate plugins to test compatibility with hosts and verify plugin API +conformance + + +pluginval [OPTIONS] + + +OPTIONS: + -h, --help Print this help message and exit + --version Display program version information and exit + --config TEXT ... Path to a JSON settings file. Repeatable; later files win per + key. + --validate TEXT Validates the plugin at the given path (or AU id). + --strictness-level INT + Strictness level 1-10 (default 5). + --timeout-ms INT Test timeout in ms (default 30000, -1 to never timeout). + --repeat INT Number of times to repeat the tests. + --randomise Run the tests in a random order per repeat. + --verbose Output additional logging information. + --skip-gui-tests Avoid tests that create GUI windows (for headless CI). + --sample-rates FLOAT ... + Comma-separated sample rates (default 44100,48000,96000). + --block-sizes INT ... + Comma-separated block sizes (default 64,128,256,512,1024). + --data-file TEXT Path to a data file tests can use to configure themselves. + --output-dir TEXT Directory in which to write the log files. + --output-filename TEXT + Filename to write logs into. + --disabled-tests TEXT + Comma-separated test names, or a path to a file listing them. + --random-seed TEXT Random seed (hex 0x.. or int) for replicable test runs. + --rtcheck ENUM:value in {disabled->0,enabled->1,relaxed->2} OR {0,1,2} + Real-time safety checks: disabled, enabled or relaxed. + +JUCE v8.0.13 + +Other commands: +--run-tests Run the internal unit tests. +--strictness-help [level] List all tests that run at the given strictness level. + +Exit code: +0 if all tests complete successfully +1 if there are any errors + +You can also specify any option as an environment variable by removing the +prefix +dashes, converting internal dashes to underscores and capitalising, e.g. +"--skip-gui-tests" -> "SKIP_GUI_TESTS=1" +"--timeout-ms 30000" -> "TIMEOUT_MS=30000" +Precedence (lowest to highest): defaults, environment variables, --config, +command-line options. +--config is repeatable; later files win per key. From c8142660b92d4ba5f6312d6a65c301ea022b0484 Mon Sep 17 00:00:00 2001 From: drowaudio Date: Mon, 8 Jun 2026 11:23:25 +0100 Subject: [PATCH 02/21] v2: `Restructure CLI into validate/run-tests/strictness-help subcommands, keeping flat flags as deprecated aliases` (#176) --- .github/workflows/build.yaml | 10 +-- CHANGELIST.md | 2 + CLAUDE.md | 46 +++++++++++-- CMakeLists.txt | 3 +- Source/CommandLine.cpp | 33 ++++++---- Source/CommandLineTests.cpp | 106 ++++++++++++++++++++++++++++++ Source/SettingsParser.cpp | 116 +++++++++++++++++++++++++++++++-- Source/SettingsParser.h | 47 +++++++++++-- docs/Adding pluginval to CI.md | 6 +- docs/Command line options.md | 25 ++++--- tests/AddPluginvalTests.cmake | 3 +- tests/windows_tests.bat | 4 +- 12 files changed, 351 insertions(+), 50 deletions(-) diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index fa5d16df..02725283 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -104,7 +104,7 @@ jobs: run: | pwd ls -ahl - ${{ matrix.test-binary }} --run-tests + ${{ matrix.test-binary }} run-tests - name: Cache JUCE example plugin binaries id: cache-plugins @@ -127,9 +127,9 @@ jobs: - name: Validate JUCE Plugin examples (VST3) shell: bash run: | - # Paths must be single quoted for bash not to escape the Windows backslash character \ used in absolute paths - ${{ env.APP_DIR }}/${{ matrix.test-binary }} --strictness-level 10 --validate '${{ env.PLUGIN_CACHE_PATH }}/DSPModulePluginDemo_artefacts/Release/VST3/DSPModulePluginDemo.vst3' - ${{ env.APP_DIR }}/${{ matrix.test-binary }} --strictness-level 10 --validate '${{ env.PLUGIN_CACHE_PATH }}/MultiOutSynthPlugin_artefacts/Release/VST3/MultiOutSynthPlugin.vst3' + # Paths must be single quoted for bash not to escape the Windows backslash character \ used in absolute paths + ${{ env.APP_DIR }}/${{ matrix.test-binary }} validate --strictness-level 10 '${{ env.PLUGIN_CACHE_PATH }}/DSPModulePluginDemo_artefacts/Release/VST3/DSPModulePluginDemo.vst3' + ${{ env.APP_DIR }}/${{ matrix.test-binary }} validate --strictness-level 10 '${{ env.PLUGIN_CACHE_PATH }}/MultiOutSynthPlugin_artefacts/Release/VST3/MultiOutSynthPlugin.vst3' - name: Validate JUCE Plugin examples (AU) shell: bash @@ -141,6 +141,8 @@ jobs: mkdir -p ~/Library/Audio/Plug-Ins/Components/ cp -R ${{ env.PLUGIN_CACHE_PATH }}/DSPModulePluginDemo_artefacts/Release/AU/DSPModulePluginDemo.component ~/Library/Audio/Plug-Ins/Components/ killall -9 AudioComponentRegistrar # kick the AU registrar + # Intentionally uses the deprecated "--validate" flat flag to keep the + # backwards-compatible alias covered (this step is continue-on-error). ${{ env.APP_DIR }}/${{ matrix.test-binary }} --strictness-level 10 --validate ~/Library/Audio/Plug-Ins/Components/DSPModulePluginDemo.component - name: Codesign (macOS) diff --git a/CHANGELIST.md b/CHANGELIST.md index 5e5140b5..6afdf839 100644 --- a/CHANGELIST.md +++ b/CHANGELIST.md @@ -1,6 +1,8 @@ # pluginval Change List ### 2.0.0 +- Restructured the command line into subcommands: `pluginval validate [options] ` (the default), `pluginval run-tests` and `pluginval strictness-help [level]`. The plugin path is now a positional argument of `validate` +- **Deprecated:** the flat flags `--validate `, `--run-tests` and `--strictness-help [level]` still work as aliases (with a one-line notice) but will be removed in a future version. `pluginval ` remains a silent shorthand for `validate` - Replaced the hand-rolled command-line parser with CLI11 and a single JSON-based settings pipeline - Added `--config ` to load settings from JSON (repeatable; later files win per key) - Settings precedence is now (lowest to highest): defaults, environment variables, `--config`, command-line options diff --git a/CLAUDE.md b/CLAUDE.md index 4acbea1c..97aa0061 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -193,6 +193,38 @@ VST2_SDK_DIR=/path/to/vst2sdk cmake -B Builds/Debug . ### CLI Settings Pipeline +#### Subcommand dispatch layer + +The command line is structured into subcommands, peeled off by a thin verb +dispatcher in front of the settings pipeline (it does **not** use CLI11-native +subcommands, so the env/config/CLI layering below is untouched): + +- `pluginval validate [options] ` — the default; `` is a + positional argument. `pluginval ` and `pluginval [options] ` + (no verb) also resolve to validate. +- `pluginval run-tests` — runs the internal unit tests. +- `pluginval strictness-help [level]` — lists tests at a strictness level. + +`settings_parser::dispatch()` (in `SettingsParser.cpp`) takes the `tokenise()`d +command line and returns a `DispatchResult { command, validateTokens, +deprecatedAlias, strictnessLevel }`. For `validate` it strips the verb and hands +`validateTokens` to `parseTokens` unchanged. `CommandLine.cpp`'s +`performCommandLine()` switches on `command` and emits a one-line stderr notice +when `deprecatedAlias` is set. + +The old flat flags (`--validate `, `--run-tests`, `--strictness-help`) +are kept as **deprecated aliases** that route to the same commands with +`deprecatedAlias = true` (the bare-path shorthand and the internal child handoff +stay silent). `preprocess()` is now `tokenise()` + `insertImplicitValidate()`. +The child process is launched with the explicit verb: +`validate --config-base64 `. + +Note: `--config` is parsed manually (it is stripped from the tokens fed to the +CLI11 pass) so its greedy CLI11 vector parsing can't swallow the positional +plugin path; it stays registered only so it appears in `--help`. + +#### Settings layering + Command-line parsing centres on one plain settings struct (`PluginvalSettings`) that CLI11 binds to directly. A single instance is filled by successive layers, **lowest to highest precedence: defaults → environment → `--config` → CLI** @@ -360,11 +392,17 @@ ut.logVerboseMessage("Detail message"); // Only with --verbose flag Basic usage: ```bash -./pluginval --strictness-level 5 /path/to/plugin.vst3 +./pluginval validate --strictness-level 5 /path/to/plugin.vst3 ``` -Key options: -- `--validate [path]` - Validate plugin at path +Commands: +- `validate [options] ` - Validate the plugin at the given path/AU id (the default; `./pluginval ` also works) +- `run-tests` - Run the internal unit tests +- `strictness-help [level]` - List the tests that run at a strictness level + +The flat flags `--validate `, `--run-tests` and `--strictness-help [level]` are deprecated aliases. + +Key options (for `validate`): - `--config [file.json]` - Load a full settings set from JSON (overridden by env vars and CLI options) - `--strictness-level [1-10]` - Test thoroughness (default: 5) - `--skip-gui-tests` - Skip GUI tests (for headless CI) @@ -434,7 +472,7 @@ Debug unit tests run automatically in debug builds: Run internal tests via CLI: ```bash -./pluginval --run-tests +./pluginval run-tests ``` ## Release Process diff --git a/CMakeLists.txt b/CMakeLists.txt index 20e02b69..4a7eab78 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -250,8 +250,9 @@ else() # TODO: This doesn't populate the executable in clion add_custom_target(${CMAKE_PROJECT_NAME}_pluginval_cli COMMAND $ - --validate ${artefact} + validate --strictness-level 10 + ${artefact} DEPENDS pluginval ${PLUGINVAL_TARGET} COMMENT "Run pluginval CLI with strict validation") endif() diff --git a/Source/CommandLine.cpp b/Source/CommandLine.cpp index f152d274..282c9b50 100644 --- a/Source/CommandLine.cpp +++ b/Source/CommandLine.cpp @@ -203,38 +203,47 @@ juce::StringArray createCommandLine (juce::String fileOrID, PluginTests::Options } //============================================================================== +static void warnDeprecated (const juce::String& oldForm, const juce::String& newForm) +{ + std::cerr << "!!! WARNING: " << oldForm << " is deprecated; use '" << newForm + << "' instead. It will be removed in a future version." << std::endl; +} + void performCommandLine (CommandLineValidator& validator, const juce::String& commandLine) { hideDockIcon(); auto& app = *juce::JUCEApplication::getInstance(); - const auto tokens = settings_parser::preprocess (commandLine); + const auto routed = settings_parser::dispatch (settings_parser::tokenise (commandLine)); - if (tokens.contains ("--run-tests")) + if (routed.command == settings_parser::Command::runTests) { + if (routed.deprecatedAlias) + warnDeprecated ("--run-tests", "pluginval run-tests"); + runUnitTests(); app.quit(); return; } - if (tokens.contains ("--strictness-help")) + if (routed.command == settings_parser::Command::strictnessHelp) { - int level = 5; - - if (const auto idx = tokens.indexOf ("--strictness-help"); idx >= 0 && idx + 1 < tokens.size()) - if (const auto next = tokens[idx + 1]; ! next.startsWith ("-")) - level = next.getIntValue(); + if (routed.deprecatedAlias) + warnDeprecated ("--strictness-help", "pluginval strictness-help"); - printStrictnessHelp (level); + printStrictnessHelp (routed.strictnessLevel); app.quit(); return; } - // Otherwise this is a validation run (explicit or implicit --validate). - // CLI11 handles --help/--version and parse errors. + // Otherwise this is a validation run (positional plugin, or explicit/implicit + // --validate). CLI11 handles --help/--version and parse errors. + if (routed.deprecatedAlias) + warnDeprecated ("--validate", "pluginval validate "); + try { - const auto result = settings_parser::parseTokens (tokens); + const auto result = settings_parser::parseTokens (routed.validateTokens); if (result.handled) { diff --git a/Source/CommandLineTests.cpp b/Source/CommandLineTests.cpp index e537a296..b19a8106 100644 --- a/Source/CommandLineTests.cpp +++ b/Source/CommandLineTests.cpp @@ -331,6 +331,112 @@ struct CommandLineTests : public juce::UnitTest expectEquals (r.exitCode, 1); } } + + beginTest ("Subcommand: validate with a positional plugin path"); + { + const auto currentDir = juce::File::getCurrentWorkingDirectory(); + + // Plugin path as a positional after the verb. + expectEquals (juce::String (parse ("validate MyPlugin.vst3").validatePath), + currentDir.getChildFile ("MyPlugin.vst3").getFullPathName()); + + // Options before the positional plugin path. + const auto s = parse ("validate --strictness-level 8 MyPlugin.vst3"); + expectEquals (s.strictnessLevel, 8); + expectEquals (juce::String (s.validatePath), currentDir.getChildFile ("MyPlugin.vst3").getFullPathName()); + + // A bare AU component id positional is left untouched. + expectEquals (juce::String (parse ("validate MyPluginID").validatePath), juce::String ("MyPluginID")); + } + + beginTest ("Subcommand: validate with --config before the positional plugin"); + { + juce::TemporaryFile configFile (".json"); + configFile.getFile().replaceWithText (R"({ "strictnessLevel": 2 })"); + + const auto currentDir = juce::File::getCurrentWorkingDirectory(); + const auto cmd = "validate --config " + configFile.getFile().getFullPathName().quoted() + " MyPlugin.vst3"; + + const auto s = parse (cmd); + expectEquals (s.strictnessLevel, 2); // --config didn't swallow the plugin path + expectEquals (juce::String (s.validatePath), currentDir.getChildFile ("MyPlugin.vst3").getFullPathName()); + } + + beginTest ("Subcommand dispatch routing (new verbs do not warn)"); + { + using settings_parser::Command; + + { + const auto d = settings_parser::dispatch (settings_parser::tokenise ("run-tests")); + expect (d.command == Command::runTests); + expect (! d.deprecatedAlias); + } + { + const auto d = settings_parser::dispatch (settings_parser::tokenise ("strictness-help 7")); + expect (d.command == Command::strictnessHelp); + expectEquals (d.strictnessLevel, 7); + expect (! d.deprecatedAlias); + } + { + const auto d = settings_parser::dispatch (settings_parser::tokenise ("strictness-help")); + expect (d.command == Command::strictnessHelp); + expectEquals (d.strictnessLevel, 5); // default level + } + { + const auto d = settings_parser::dispatch (settings_parser::tokenise ("validate MyPlugin.vst3")); + expect (d.command == Command::validate); + expect (! d.deprecatedAlias); + } + } + + beginTest ("Deprecated flat flags still route, flagged as deprecated"); + { + using settings_parser::Command; + + { + const auto d = settings_parser::dispatch (settings_parser::tokenise ("--run-tests")); + expect (d.command == Command::runTests); + expect (d.deprecatedAlias); + } + { + const auto d = settings_parser::dispatch (settings_parser::tokenise ("--strictness-help 9")); + expect (d.command == Command::strictnessHelp); + expectEquals (d.strictnessLevel, 9); + expect (d.deprecatedAlias); + } + { + const auto d = settings_parser::dispatch (settings_parser::tokenise ("--validate x")); + expect (d.command == Command::validate); + expect (d.deprecatedAlias); + } + } + + beginTest ("Bare-path shorthand and child handoff do not warn"); + { + juce::TemporaryFile temp ("path_to_file.vst3"); + expect (temp.getFile().create()); + + // Bare plugin path -> validate, no deprecation warning. + const auto bare = settings_parser::dispatch (settings_parser::tokenise (temp.getFile().getFullPathName())); + expect (bare.command == settings_parser::Command::validate); + expect (! bare.deprecatedAlias); + + // The internal child handoff uses the explicit verb -> no warning. + PluginTests::Options opts; + juce::StringArray childArgs (createCommandLine ("/some/MyPlugin.vst3", opts)); + childArgs.remove (0); // drop the executable path + const auto child = settings_parser::dispatch (childArgs); + expect (child.command == settings_parser::Command::validate); + expect (! child.deprecatedAlias); + } + + beginTest ("Should perform command line recognises subcommands"); + { + expect (shouldPerformCommandLine ("run-tests")); + expect (shouldPerformCommandLine ("strictness-help")); + expect (shouldPerformCommandLine ("validate MyPlugin.vst3")); + expect (shouldPerformCommandLine ("validate MyPluginID")); + } } }; diff --git a/Source/SettingsParser.cpp b/Source/SettingsParser.cpp index 244286aa..e93cb76e 100644 --- a/Source/SettingsParser.cpp +++ b/Source/SettingsParser.cpp @@ -126,9 +126,13 @@ namespace settings_parser juce::String getFooterText() { return juce::SystemStats::getJUCEVersion() + "\n\n" + juce::String ( -R"(Other commands: - --run-tests Run the internal unit tests. - --strictness-help [level] List all tests that run at the given strictness level. +R"(Commands: + validate [options] Validate the plugin at the given path or AU id (the default). + run-tests Run the internal unit tests. + strictness-help [level] List all tests that run at the given strictness level. + +The flat flags --validate , --run-tests and --strictness-help [level] are +deprecated aliases for the commands above and will be removed in a future version. Exit code: 0 if all tests complete successfully @@ -172,6 +176,9 @@ Precedence (lowest to highest): defaults, environment variables, --config, comma "Path to a JSON settings file. Repeatable; later files win per key.")->take_all(); app.add_option ("--validate", s.validatePath, "Validates the plugin at the given path (or AU id)."); + // The new canonical form: "pluginval validate ". Bound to the + // same member as --validate; whichever is supplied sets the path. + app.add_option ("plugin", s.validatePath, "Plugin path or AU id to validate."); app.add_option ("--strictness-level", s.strictnessLevel, "Strictness level 1-10 (default 5)."); app.add_option ("--timeout-ms", s.timeoutMs, "Test timeout in ms (default 30000, -1 to never timeout)."); app.add_option ("--repeat", s.numRepeats, "Number of times to repeat the tests."); @@ -254,7 +261,7 @@ Precedence (lowest to highest): defaults, environment variables, --config, comma } //============================================================================== - juce::StringArray preprocess (const juce::String& commandLineIn) + juce::StringArray tokenise (const juce::String& commandLineIn) { if (commandLineIn.contains ("strictnessLevel")) { @@ -273,6 +280,11 @@ Precedence (lowest to highest): defaults, environment variables, --config, comma for (auto& s : args) s = s.unquoted(); + return args; + } + + void insertImplicitValidate (juce::StringArray& args) + { // If only a plugin path is supplied as the last arg, add an implicit // --validate option for it so the rest of the CLI works. if (args.size() > 0) @@ -286,12 +298,25 @@ Precedence (lowest to highest): defaults, environment variables, --config, comma if (! hasCommand && isPluginArgument (args[args.size() - 1])) args.insert (args.size() - 1, "--validate"); } + } + juce::StringArray preprocess (const juce::String& commandLine) + { + auto args = tokenise (commandLine); + insertImplicitValidate (args); return args; } bool isCommandLine (const juce::StringArray& tokens) { + if (! tokens.isEmpty()) + { + const auto& verb = tokens.getReference (0); + + if (verb == "validate" || verb == "run-tests" || verb == "strictness-help") + return true; + } + return tokens.contains ("--help") || tokens.contains ("-h") || tokens.contains ("--version") || hasOption (tokens, "--validate") @@ -300,6 +325,73 @@ Precedence (lowest to highest): defaults, environment variables, --config, comma || hasOption (tokens, "--config-base64"); } + //============================================================================== + DispatchResult dispatch (const juce::StringArray& tokensIn) + { + DispatchResult result; + + // The optional [level] argument that follows the strictness-help command/flag. + const auto levelAfter = [] (const juce::StringArray& tokens, int idx) + { + if (idx >= 0 && idx + 1 < tokens.size()) + if (const auto& next = tokens.getReference (idx + 1); ! next.startsWith ("-")) + return juce::jlimit (1, 10, next.getIntValue()); + + return 5; + }; + + // 1. Explicit subcommand verbs (the new, non-deprecated syntax). + if (! tokensIn.isEmpty()) + { + const auto& verb = tokensIn.getReference (0); + + if (verb == "validate") + { + result.command = Command::validate; + result.validateTokens = tokensIn; + result.validateTokens.remove (0); // the plugin path is captured by the positional CLI option + return result; + } + + if (verb == "run-tests") + { + result.command = Command::runTests; + return result; + } + + if (verb == "strictness-help") + { + result.command = Command::strictnessHelp; + result.strictnessLevel = levelAfter (tokensIn, 0); + return result; + } + } + + // 2. Deprecated flat command flags. + if (tokensIn.contains ("--run-tests")) + { + result.command = Command::runTests; + result.deprecatedAlias = true; + return result; + } + + if (tokensIn.contains ("--strictness-help")) + { + result.command = Command::strictnessHelp; + result.deprecatedAlias = true; + result.strictnessLevel = levelAfter (tokensIn, tokensIn.indexOf ("--strictness-help")); + return result; + } + + // 3. Default: validate. An explicit --validate flag (but not the bare-path + // shorthand, nor the internal --config-base64 handoff) is a deprecated alias. + result.command = Command::validate; + result.deprecatedAlias = hasOption (tokensIn, "--validate") && ! hasOption (tokensIn, "--config-base64"); + result.validateTokens = tokensIn; + insertImplicitValidate (result.validateTokens); + return result; + } + juce::String resolvePluginPath (const juce::String& raw) { // Resolve relative/home paths against the working directory. Absolute @@ -375,8 +467,18 @@ Precedence (lowest to highest): defaults, environment variables, --config, comma std::vector argv { "pluginval" }; - for (const auto& t : tokens) + // --config is handled manually above; drop it (and its value) here so + // its greedy CLI11 vector parsing can't swallow the positional plugin + // path. It stays registered purely so it appears in --help. + for (int i = 0; i < tokens.size(); ++i) + { + const auto& t = tokens.getReference (i); + + if (t == "--config") { ++i; continue; } // skip the flag and its value + if (t.startsWith ("--config=")) continue; // skip the inline form + argv.push_back (t.toStdString()); + } if (const auto code = runParse (cliApp, argv); code) { @@ -392,7 +494,7 @@ Precedence (lowest to highest): defaults, environment variables, --config, comma PluginvalSettings parse (const juce::String& commandLine, const EnvProvider& env) { - return parseTokens (preprocess (commandLine), env).settings; + return parseTokens (dispatch (tokenise (commandLine)).validateTokens, env).settings; } //============================================================================== @@ -403,7 +505,7 @@ Precedence (lowest to highest): defaults, environment variables, --config, comma const auto b64 = juce::Base64::toBase64 (juce::String (jsonString)); juce::StringArray args (juce::File::getSpecialLocation (juce::File::currentExecutableFile).getFullPathName()); - args.addArray ({ "--config-base64", b64, "--validate", fileOrID }); + args.addArray ({ "validate", "--config-base64", b64, fileOrID }); return args; } } diff --git a/Source/SettingsParser.h b/Source/SettingsParser.h index 3cd6ebc6..15089faa 100644 --- a/Source/SettingsParser.h +++ b/Source/SettingsParser.h @@ -38,14 +38,24 @@ namespace settings_parser juce::String systemEnv (const juce::String& name); //============================================================================== - /** Tokenises + preprocesses a raw command line: rewrites the deprecated - "strictnessLevel", strips the macOS "-NSDocumentRevisionsDebugMode YES" - flag, and inserts an implicit "--validate" when the last argument is a - bare plugin path. Returns argv tokens (without the program name). + /** Tokenises a raw command line: rewrites the deprecated "strictnessLevel", + strips the macOS "-NSDocumentRevisionsDebugMode YES" flag, splits and + unquotes. Returns argv tokens (without the program name) exactly as the + user typed them - it does NOT insert an implicit "--validate". */ + juce::StringArray tokenise (const juce::String& commandLine); + + /** Inserts an implicit "--validate" when the last token is a bare plugin path + (and no command is already present). This is the documented shorthand for + "pluginval ". + */ + void insertImplicitValidate (juce::StringArray& tokens); + + /** Convenience: tokenise + insertImplicitValidate. */ juce::StringArray preprocess (const juce::String& commandLine); - /** True if the tokens contain a recognised command that triggers CLI mode. */ + /** True if the tokens contain a recognised command (subcommand verb or legacy + flag) that triggers CLI mode. */ bool isCommandLine (const juce::StringArray& tokens); /** Resolves a relative/home plugin path against the working directory, @@ -53,6 +63,31 @@ namespace settings_parser */ juce::String resolvePluginPath (const juce::String& raw); + //============================================================================== + /** The subcommand selected on the command line. */ + enum class Command + { + validate, /**< Validate a plugin (the default when no verb is given). */ + runTests, /**< Run the internal unit tests. */ + strictnessHelp /**< List the tests that run at a given strictness level. */ + }; + + /** The outcome of routing tokens to a subcommand. */ + struct DispatchResult + { + Command command = Command::validate; + juce::StringArray validateTokens; /**< Verb-stripped option tokens to feed parseTokens (validate only). */ + bool deprecatedAlias = false; /**< A legacy flat flag (--validate/--run-tests/--strictness-help) was used. */ + int strictnessLevel = 5; /**< The level for the strictnessHelp command. */ + }; + + /** Routes tokenised input to a subcommand, peeling the leading verb. The + validate token-layering in parseTokens is left untouched; the plugin path + is captured by a positional CLI option there. Legacy flat flags map onto + the matching command with deprecatedAlias set. + */ + DispatchResult dispatch (const juce::StringArray& tokens); + //============================================================================== /** The outcome of parsing the option tokens. */ struct ParseResult @@ -70,7 +105,7 @@ namespace settings_parser //============================================================================== /** Serialises options for the child validation process as a base64 JSON - handoff: { exe, --config-base64 , --validate }. + handoff: { exe, validate, --config-base64 , }. */ juce::StringArray createChildProcessCommandLine (const juce::String& fileOrID, const PluginTests::Options&); diff --git a/docs/Adding pluginval to CI.md b/docs/Adding pluginval to CI.md index e7e96de7..3baa06b9 100644 --- a/docs/Adding pluginval to CI.md +++ b/docs/Adding pluginval to CI.md @@ -19,21 +19,21 @@ In these examples the following is assumed: ```sh $ curl -L "https://github.com/Tracktion/pluginval/releases/latest/download/pluginval_macOS.zip" -o pluginval.zip $ unzip pluginval -$ pluginval.app/Contents/MacOS/pluginval --validate-in-process --output-dir "./bin" "" || exit 1 +$ pluginval.app/Contents/MacOS/pluginval validate --output-dir "./bin" "" || exit 1 ``` ##### Linux ```sh $ curl -L "https://github.com/Tracktion/pluginval/releases/latest/download/pluginval_Linux.zip" -o pluginval.zip $ unzip pluginval -$ ./pluginval --validate-in-process --output-dir "./bin" "" || exit 1 +$ ./pluginval validate --output-dir "./bin" "" || exit 1 ``` ##### Windows ```sh > powershell -Command "[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12; Invoke-WebRequest https://github.com/Tracktion/pluginval/releases/latest/download/pluginval_Windows.zip -OutFile pluginval.zip" > powershell -Command "Expand-Archive pluginval.zip -DestinationPath ." -> pluginval.exe --validate-in-process --output-dir "./bin" "" +> pluginval.exe validate --output-dir "./bin" "" > if %ERRORLEVEL% neq 0 exit /b 1 ``` diff --git a/docs/Command line options.md b/docs/Command line options.md index 4c824692..d87244b8 100644 --- a/docs/Command line options.md +++ b/docs/Command line options.md @@ -2,10 +2,22 @@ Validate plugins to test compatibility with hosts and verify plugin API conformance -pluginval [OPTIONS] +pluginval [OPTIONS] [plugin] -OPTIONS: +COMMANDS: + validate [options] Validate the plugin at the given path or AU id. + This is the default, so "pluginval " and + "pluginval [options] " also work. + run-tests Run the internal unit tests. + strictness-help [level] List all tests that run at the given strictness level. + +The flat flags --validate , --run-tests and --strictness-help [level] +are deprecated aliases for the commands above and will be removed in a future +version. + + +OPTIONS (for the validate command): -h, --help Print this help message and exit --version Display program version information and exit --config TEXT ... Path to a JSON settings file. Repeatable; later files win per @@ -32,19 +44,12 @@ OPTIONS: --rtcheck ENUM:value in {disabled->0,enabled->1,relaxed->2} OR {0,1,2} Real-time safety checks: disabled, enabled or relaxed. -JUCE v8.0.13 - -Other commands: ---run-tests Run the internal unit tests. ---strictness-help [level] List all tests that run at the given strictness level. - Exit code: 0 if all tests complete successfully 1 if there are any errors You can also specify any option as an environment variable by removing the -prefix -dashes, converting internal dashes to underscores and capitalising, e.g. +prefix dashes, converting internal dashes to underscores and capitalising, e.g. "--skip-gui-tests" -> "SKIP_GUI_TESTS=1" "--timeout-ms 30000" -> "TIMEOUT_MS=30000" Precedence (lowest to highest): defaults, environment variables, --config, diff --git a/tests/AddPluginvalTests.cmake b/tests/AddPluginvalTests.cmake index c4850712..133a9775 100644 --- a/tests/AddPluginvalTests.cmake +++ b/tests/AddPluginvalTests.cmake @@ -81,13 +81,14 @@ function (add_pluginval_tests pluginTarget) add_test (NAME "${test_name}" COMMAND "${PLUGINVAL_PROGRAM}" + validate --strictness-level "${PLUGINVAL_STRICTNESS}" --sample-rates "${sample_rates}" --block-sizes "${block_sizes}" --repeat "${PLUGINVAL_REPEATS}" --randomise - --validate "${plugin_artefact}" ${log_dir_arg} + "${plugin_artefact}" ) set_tests_properties ( diff --git a/tests/windows_tests.bat b/tests/windows_tests.bat index 327d4766..e8ac04d7 100755 --- a/tests/windows_tests.bat +++ b/tests/windows_tests.bat @@ -64,11 +64,11 @@ exit /B %ERRORLEVEL% "%MSBUILD_EXE%" %PLUGIN_NAME%.sln /p:VisualStudioVersion=15.0 /m /t:Build /p:Configuration=Release /p:Platform=x64 /p:PreferredToolArchitecture=x64 /p:TreatWarningsAsErrors=true :: Test in process - call "%PLUGINVAL_EXE%" --validate-in-process --strictness-level 5 --validate %PLUGIN_VST3% + call "%PLUGINVAL_EXE%" validate --strictness-level 5 %PLUGIN_VST3% if %ERRORLEVEL% NEQ 0 exit 1 :: Test out of process - :: call "%PLUGINVAL_EXE%" --strictness-level 5 --validate %PLUGIN_VST3% + :: call "%PLUGINVAL_EXE%" validate --strictness-level 5 %PLUGIN_VST3% :: if %ERRORLEVEL% NEQ 0 exit 1 exit /B 0 From 247e42ab62bd79eee588a935e60e8ad74a9776b8 Mon Sep 17 00:00:00 2001 From: David Rowland Date: Mon, 8 Jun 2026 11:44:27 +0100 Subject: [PATCH 03/21] v2: Removed a planning doc --- ROADMAP.md | 2 +- SUBCOMMANDS_HANDOFF.md | 89 ------------------------------------------ 2 files changed, 1 insertion(+), 90 deletions(-) delete mode 100644 SUBCOMMANDS_HANDOFF.md diff --git a/ROADMAP.md b/ROADMAP.md index e7d181b5..5a63ca77 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -5,7 +5,7 @@ There are lots of ways pluginval can be improved, some of these are listed below You can help get there by issuing PRs or [funding](FUNDING.md) development. - [x] Integration of real-time safety checking (via rtcheck) -- [ ] Improved command-line handling and json config file support +- [x] Improved command-line handling and json config file support - [ ] Acceptance testing using input files and reference output files - [ ] Improved stack trace/crash reporting - [ ] Automatic integration of Asan/Tsan on platforms that allow it diff --git a/SUBCOMMANDS_HANDOFF.md b/SUBCOMMANDS_HANDOFF.md deleted file mode 100644 index 3f6ef277..00000000 --- a/SUBCOMMANDS_HANDOFF.md +++ /dev/null @@ -1,89 +0,0 @@ -# HANDOFF: Restructure pluginval's CLI into subcommands - -## Goal -Turn the current "mode" flags into proper subcommands, each with its own -argument set, while keeping the old flat flags working as **deprecated aliases -for one release**: - -| New (target) | Old (keep as deprecated alias) | -|---|---| -| `pluginval validate [options] ` (the **default**) | `pluginval --validate ` / `pluginval ` | -| `pluginval run-tests` | `pluginval --run-tests` | -| `pluginval strictness-help [level]` | `pluginval --strictness-help [level]` | -| `pluginval --version` / `pluginval --help` | (unchanged) | - -All settings options (`--strictness-level`, `--config`, `--rtcheck`, …), -environment variables, and the JSON precedence pipeline belong to the `validate` -subcommand and are otherwise unchanged. - -This was deliberately deferred from the CLI11 refactor PR (#174). The pipeline -and `PluginvalSettings` were designed to be reused unchanged. - -## Read these first (don't trust this summary — verify against the files) -- `Source/SettingsParser.cpp/.h` — the parser. Key pieces to reuse: - - `preprocess()` — deprecation rewrite, macOS flag strip, implicit `--validate`, tokenise. - - `parseTokens(tokens, env)` — env → `--config` → CLI layering into one `PluginvalSettings`. **This is the `validate` body.** - - `configureApp(app, s)` — registers every option on a `CLI::App`, used for both the env and CLI passes. - - `createChildProcessCommandLine()` — the parent→child base64 handoff (`--config-base64 --validate `). - - `isCommandLine(tokens)` — what makes `shouldPerformCommandLine` return true. -- `Source/CommandLine.cpp` — `performCommandLine()` is the dispatcher today: - token-scans for `--run-tests` / `--strictness-help`, otherwise runs `parseTokens` for validate; `--help`/`--version` go through CLI11. Also `runUnitTests()`, `printStrictnessHelp()`. -- `Source/Main.cpp` — calls `shouldPerformCommandLine()` then `performCommandLine()` with `getCommandLineParameters()` (a single `juce::String`). -- `Source/CommandLineTests.cpp` — the test contract. - -## Recommended approach: a thin `argv[1]` verb dispatcher (not CLI11 subcommands) -The existing `parseTokens` pipeline (env/config/CLI layering via two `CLI::App` -passes) is the hard part and already works. Rather than re-express it inside -CLI11's native `add_subcommand` machinery (which complicates the env/config -layering because the options live on the subcommand), peel the verb off the -front and route: - -1. In a new `dispatch()` step (in `SettingsParser` or `CommandLine.cpp`): - - Tokenise the command line (reuse `preprocess` minus the implicit-validate step, or add a pre-step). - - Look at the first non-option token: - - `validate` → strip it, run the existing validate pipeline on the rest. - - `run-tests` → `runUnitTests()`. - - `strictness-help` → `printStrictnessHelp(level)`. - - otherwise → **default to validate** (this preserves `pluginval ` and `pluginval --strictness-level 5 `). -2. Each verb keeps its own small set of expected args. `validate` reuses - `configureApp`/`parseTokens` verbatim. `run-tests` takes none. - `strictness-help` takes an optional level. - -This keeps `parseTokens` and the precedence layering untouched — the subcommand -work is purely a routing layer in front of it. - -(If you prefer CLI11-native subcommands instead: put `configureApp` options on a -`validate` subcommand, and make `buildEnvArgv`/the env pass target that -subcommand's options. Doable, but more invasive for no functional gain here.) - -## Deprecated-alias behaviour (one release) -Keep the old flat forms working, but print a one-line notice to stderr pointing -at the new syntax, e.g.: -- `pluginval --validate x` → run validate; warn `"--validate is deprecated; use 'pluginval validate x'"`. -- `pluginval --run-tests` → warn `"use 'pluginval run-tests'"`. -- `pluginval --strictness-help` → warn `"use 'pluginval strictness-help'"`. -- `pluginval ` (bare path) → **no warning** (still the documented shorthand for `validate`). - -Gate the warnings behind detection of the old flag so the new subcommand form is -silent. Remove the aliases in the release after next; note it in `CHANGELIST.md`. - -## Files to touch -- `Source/SettingsParser.{h,cpp}` — add the verb dispatch + a `Command` result (validate/run-tests/strictness-help/help/version), or expose a `dispatch()` that returns which verb + the remaining tokens. Keep `parseTokens` as the validate body. -- `Source/CommandLine.cpp` — `performCommandLine()` routes on the verb; emit the deprecation notices for old flat flags. `shouldPerformCommandLine()` must also recognise the bare verbs (`validate`/`run-tests`/`strictness-help`) in addition to the old flags. -- `Source/CommandLineTests.cpp` — add tests: each subcommand; default-to-validate; bare-path shorthand; every deprecated alias still works (and warns); `run-tests`/`strictness-help` arg handling. -- `docs/Command line options.md` — regenerate (`pluginval --help`); CLI11 can show per-subcommand help if you go native, otherwise hand-format the verb list. -- `CHANGELIST.md` — note the subcommand syntax + the deprecation. -- `CLAUDE.md` — update the "CLI Settings Pipeline" section to mention the verb layer. - -## Gotchas / decisions to make -- **Child-process handoff.** `createChildProcessCommandLine()` emits `--config-base64 --validate `. Decide whether the child invocation becomes `validate --config-base64 …` or stays flat. Simplest: keep it flat and have the dispatcher treat a leading `--config-base64`/`--validate` as the (deprecated, unwarned-for-internal) validate path. Note `--config-base64` is now hardened to reject being combined with non-`--validate` options — keep that working under whichever form you choose. -- **`shouldPerformCommandLine`** is what flips pluginval into CLI (vs GUI) mode in `Main.cpp`. It must return true for `pluginval run-tests` etc., not just the old flags. -- **`--help` scope.** With the dispatcher, `pluginval --help` is the top-level help (list verbs + the validate options). Consider `pluginval validate --help` for the full option list. CLI11-native subcommands give this for free. -- **Implicit validate** currently lives in `preprocess`. With an explicit `validate` verb, make sure `pluginval ` (no verb) still resolves to validate, and `pluginval validate ` doesn't double-insert `--validate`. -- **Reconcile** a positional plugin path under `validate` (e.g. `pluginval validate `) with the existing `--validate ` option — pick one canonical form (recommend the positional for the new syntax, mapping it onto `s.validatePath`). - -## Verify -- `pluginval run-tests` passes the full unit suite (it must, it's how CI runs tests). -- `pluginval validate --strictness-level 10 ` and `pluginval ` both validate. -- Every deprecated alias produces identical behaviour to before (plus a notice). -- CI matrix green (Linux/macOS/Windows build + dependency). Remember `.github/workflows/build.yaml` uses `--run-tests` and `--strictness-level 10 --validate …`; update those to the new syntax **and** keep an alias test, or the deprecation will fire in CI. From febe6af0343b054a4cfc1a6ccd2f9da2fd471b61 Mon Sep 17 00:00:00 2001 From: David Rowland Date: Tue, 16 Jun 2026 14:21:21 +0100 Subject: [PATCH 04/21] Renamed "Source" dir to "source" for consistency --- {Source => source}/CommandLine.cpp | 0 {Source => source}/CommandLine.h | 0 {Source => source}/CommandLineTests.cpp | 0 {Source => source}/CrashHandler.cpp | 0 {Source => source}/CrashHandler.h | 0 {Source => source}/Main.cpp | 0 {Source => source}/MainComponent.cpp | 0 {Source => source}/MainComponent.h | 0 {Source => source}/PluginTests.cpp | 0 {Source => source}/PluginTests.h | 0 {Source => source}/PluginvalLookAndFeel.h | 0 {Source => source}/PluginvalSettings.h | 0 {Source => source}/RTCheck.h | 0 {Source => source}/SettingsParser.cpp | 0 {Source => source}/SettingsParser.h | 0 {Source => source}/SettingsSerializer.cpp | 0 {Source => source}/SettingsSerializer.h | 0 {Source => source}/StrictnessInfoPopup.h | 0 {Source => source}/TestUtilities.cpp | 0 {Source => source}/TestUtilities.h | 0 {Source => source}/Validator.cpp | 0 {Source => source}/Validator.h | 0 {Source => source}/binarydata/icon.png | Bin {Source => source}/binarydata/icon.svg | 0 {Source => source}/tests/BasicTests.cpp | 0 {Source => source}/tests/BusTests.cpp | 0 {Source => source}/tests/EditorTests.cpp | 0 {Source => source}/tests/ExtremeTests.cpp | 0 {Source => source}/tests/LocaleTest.cpp | 0 {Source => source}/tests/ParameterFuzzTests.cpp | 0 .../vst3validator/VST3ValidatorRunner.cpp | 0 .../vst3validator/VST3ValidatorRunner.h | 0 32 files changed, 0 insertions(+), 0 deletions(-) rename {Source => source}/CommandLine.cpp (100%) rename {Source => source}/CommandLine.h (100%) rename {Source => source}/CommandLineTests.cpp (100%) rename {Source => source}/CrashHandler.cpp (100%) rename {Source => source}/CrashHandler.h (100%) rename {Source => source}/Main.cpp (100%) rename {Source => source}/MainComponent.cpp (100%) rename {Source => source}/MainComponent.h (100%) rename {Source => source}/PluginTests.cpp (100%) rename {Source => source}/PluginTests.h (100%) rename {Source => source}/PluginvalLookAndFeel.h (100%) rename {Source => source}/PluginvalSettings.h (100%) rename {Source => source}/RTCheck.h (100%) rename {Source => source}/SettingsParser.cpp (100%) rename {Source => source}/SettingsParser.h (100%) rename {Source => source}/SettingsSerializer.cpp (100%) rename {Source => source}/SettingsSerializer.h (100%) rename {Source => source}/StrictnessInfoPopup.h (100%) rename {Source => source}/TestUtilities.cpp (100%) rename {Source => source}/TestUtilities.h (100%) rename {Source => source}/Validator.cpp (100%) rename {Source => source}/Validator.h (100%) rename {Source => source}/binarydata/icon.png (100%) rename {Source => source}/binarydata/icon.svg (100%) rename {Source => source}/tests/BasicTests.cpp (100%) rename {Source => source}/tests/BusTests.cpp (100%) rename {Source => source}/tests/EditorTests.cpp (100%) rename {Source => source}/tests/ExtremeTests.cpp (100%) rename {Source => source}/tests/LocaleTest.cpp (100%) rename {Source => source}/tests/ParameterFuzzTests.cpp (100%) rename {Source => source}/vst3validator/VST3ValidatorRunner.cpp (100%) rename {Source => source}/vst3validator/VST3ValidatorRunner.h (100%) diff --git a/Source/CommandLine.cpp b/source/CommandLine.cpp similarity index 100% rename from Source/CommandLine.cpp rename to source/CommandLine.cpp diff --git a/Source/CommandLine.h b/source/CommandLine.h similarity index 100% rename from Source/CommandLine.h rename to source/CommandLine.h diff --git a/Source/CommandLineTests.cpp b/source/CommandLineTests.cpp similarity index 100% rename from Source/CommandLineTests.cpp rename to source/CommandLineTests.cpp diff --git a/Source/CrashHandler.cpp b/source/CrashHandler.cpp similarity index 100% rename from Source/CrashHandler.cpp rename to source/CrashHandler.cpp diff --git a/Source/CrashHandler.h b/source/CrashHandler.h similarity index 100% rename from Source/CrashHandler.h rename to source/CrashHandler.h diff --git a/Source/Main.cpp b/source/Main.cpp similarity index 100% rename from Source/Main.cpp rename to source/Main.cpp diff --git a/Source/MainComponent.cpp b/source/MainComponent.cpp similarity index 100% rename from Source/MainComponent.cpp rename to source/MainComponent.cpp diff --git a/Source/MainComponent.h b/source/MainComponent.h similarity index 100% rename from Source/MainComponent.h rename to source/MainComponent.h diff --git a/Source/PluginTests.cpp b/source/PluginTests.cpp similarity index 100% rename from Source/PluginTests.cpp rename to source/PluginTests.cpp diff --git a/Source/PluginTests.h b/source/PluginTests.h similarity index 100% rename from Source/PluginTests.h rename to source/PluginTests.h diff --git a/Source/PluginvalLookAndFeel.h b/source/PluginvalLookAndFeel.h similarity index 100% rename from Source/PluginvalLookAndFeel.h rename to source/PluginvalLookAndFeel.h diff --git a/Source/PluginvalSettings.h b/source/PluginvalSettings.h similarity index 100% rename from Source/PluginvalSettings.h rename to source/PluginvalSettings.h diff --git a/Source/RTCheck.h b/source/RTCheck.h similarity index 100% rename from Source/RTCheck.h rename to source/RTCheck.h diff --git a/Source/SettingsParser.cpp b/source/SettingsParser.cpp similarity index 100% rename from Source/SettingsParser.cpp rename to source/SettingsParser.cpp diff --git a/Source/SettingsParser.h b/source/SettingsParser.h similarity index 100% rename from Source/SettingsParser.h rename to source/SettingsParser.h diff --git a/Source/SettingsSerializer.cpp b/source/SettingsSerializer.cpp similarity index 100% rename from Source/SettingsSerializer.cpp rename to source/SettingsSerializer.cpp diff --git a/Source/SettingsSerializer.h b/source/SettingsSerializer.h similarity index 100% rename from Source/SettingsSerializer.h rename to source/SettingsSerializer.h diff --git a/Source/StrictnessInfoPopup.h b/source/StrictnessInfoPopup.h similarity index 100% rename from Source/StrictnessInfoPopup.h rename to source/StrictnessInfoPopup.h diff --git a/Source/TestUtilities.cpp b/source/TestUtilities.cpp similarity index 100% rename from Source/TestUtilities.cpp rename to source/TestUtilities.cpp diff --git a/Source/TestUtilities.h b/source/TestUtilities.h similarity index 100% rename from Source/TestUtilities.h rename to source/TestUtilities.h diff --git a/Source/Validator.cpp b/source/Validator.cpp similarity index 100% rename from Source/Validator.cpp rename to source/Validator.cpp diff --git a/Source/Validator.h b/source/Validator.h similarity index 100% rename from Source/Validator.h rename to source/Validator.h diff --git a/Source/binarydata/icon.png b/source/binarydata/icon.png similarity index 100% rename from Source/binarydata/icon.png rename to source/binarydata/icon.png diff --git a/Source/binarydata/icon.svg b/source/binarydata/icon.svg similarity index 100% rename from Source/binarydata/icon.svg rename to source/binarydata/icon.svg diff --git a/Source/tests/BasicTests.cpp b/source/tests/BasicTests.cpp similarity index 100% rename from Source/tests/BasicTests.cpp rename to source/tests/BasicTests.cpp diff --git a/Source/tests/BusTests.cpp b/source/tests/BusTests.cpp similarity index 100% rename from Source/tests/BusTests.cpp rename to source/tests/BusTests.cpp diff --git a/Source/tests/EditorTests.cpp b/source/tests/EditorTests.cpp similarity index 100% rename from Source/tests/EditorTests.cpp rename to source/tests/EditorTests.cpp diff --git a/Source/tests/ExtremeTests.cpp b/source/tests/ExtremeTests.cpp similarity index 100% rename from Source/tests/ExtremeTests.cpp rename to source/tests/ExtremeTests.cpp diff --git a/Source/tests/LocaleTest.cpp b/source/tests/LocaleTest.cpp similarity index 100% rename from Source/tests/LocaleTest.cpp rename to source/tests/LocaleTest.cpp diff --git a/Source/tests/ParameterFuzzTests.cpp b/source/tests/ParameterFuzzTests.cpp similarity index 100% rename from Source/tests/ParameterFuzzTests.cpp rename to source/tests/ParameterFuzzTests.cpp diff --git a/Source/vst3validator/VST3ValidatorRunner.cpp b/source/vst3validator/VST3ValidatorRunner.cpp similarity index 100% rename from Source/vst3validator/VST3ValidatorRunner.cpp rename to source/vst3validator/VST3ValidatorRunner.cpp diff --git a/Source/vst3validator/VST3ValidatorRunner.h b/source/vst3validator/VST3ValidatorRunner.h similarity index 100% rename from Source/vst3validator/VST3ValidatorRunner.h rename to source/vst3validator/VST3ValidatorRunner.h From 8b405367212e43d61506ab10e9f832b86a5a5ee8 Mon Sep 17 00:00:00 2001 From: David Rowland Date: Tue, 16 Jun 2026 14:53:38 +0100 Subject: [PATCH 05/21] Acceptance testing: Initial commit --- .github/workflows/acceptance.yml | 70 +++ CLAUDE.md | 57 ++- CMakeLists.txt | 78 +-- README.md | 1 + docs/Acceptance testing.md | 75 +++ docs/Command line options.md | 3 + source/CommandLine.cpp | 17 + source/CommandLineTests.cpp | 69 +++ source/SettingsParser.cpp | 20 +- source/SettingsParser.h | 4 +- source/acceptance/AcceptanceTest.cpp | 449 ++++++++++++++++++ source/acceptance/AcceptanceTest.h | 56 +++ source/acceptance/ReferenceComparator.cpp | 122 +++++ source/acceptance/ReferenceComparator.h | 60 +++ source/acceptance/TestConfig.cpp | 208 ++++++++ source/acceptance/TestConfig.h | 104 ++++ source/acceptance/TestReporter.cpp | 110 +++++ source/acceptance/TestReporter.h | 73 +++ tests/acceptance/Acceptance testing design.md | 390 +++++++++++++++ tests/acceptance/CMakeLists.txt | 36 ++ tests/acceptance/gain-half.json.in | 16 + tests/acceptance/inputs/sine-full.wav | Bin 0 -> 96104 bytes tests/acceptance/refs/gain-half.wav | Bin 0 -> 96104 bytes tests/acceptance/refs/gain-half.wav.json | 23 + tests/acceptance/refs/sine-440.wav | Bin 0 -> 96104 bytes tests/acceptance/refs/sine-440.wav.json | 23 + tests/acceptance/refs/square-220.wav | Bin 0 -> 96104 bytes tests/acceptance/refs/square-220.wav.json | 23 + tests/acceptance/refs/square-state.state | Bin 0 -> 12 bytes tests/acceptance/refs/square-state.wav | Bin 0 -> 96104 bytes tests/acceptance/refs/square-state.wav.json | 23 + tests/acceptance/sine-440.json.in | 15 + tests/acceptance/square-220.json.in | 18 + tests/acceptance/square-state.json.in | 17 + tests/test_plugins/gain/CMakeLists.txt | 30 ++ tests/test_plugins/gain/GainPlugin.cpp | 56 +++ tests/test_plugins/gain/GainPlugin.h | 68 +++ .../tone_generator/CMakeLists.txt | 32 ++ .../tone_generator/ToneGeneratorPlugin.cpp | 114 +++++ .../tone_generator/ToneGeneratorPlugin.h | 76 +++ 40 files changed, 2501 insertions(+), 35 deletions(-) create mode 100644 .github/workflows/acceptance.yml create mode 100644 docs/Acceptance testing.md create mode 100644 source/acceptance/AcceptanceTest.cpp create mode 100644 source/acceptance/AcceptanceTest.h create mode 100644 source/acceptance/ReferenceComparator.cpp create mode 100644 source/acceptance/ReferenceComparator.h create mode 100644 source/acceptance/TestConfig.cpp create mode 100644 source/acceptance/TestConfig.h create mode 100644 source/acceptance/TestReporter.cpp create mode 100644 source/acceptance/TestReporter.h create mode 100644 tests/acceptance/Acceptance testing design.md create mode 100644 tests/acceptance/CMakeLists.txt create mode 100644 tests/acceptance/gain-half.json.in create mode 100644 tests/acceptance/inputs/sine-full.wav create mode 100644 tests/acceptance/refs/gain-half.wav create mode 100644 tests/acceptance/refs/gain-half.wav.json create mode 100644 tests/acceptance/refs/sine-440.wav create mode 100644 tests/acceptance/refs/sine-440.wav.json create mode 100644 tests/acceptance/refs/square-220.wav create mode 100644 tests/acceptance/refs/square-220.wav.json create mode 100644 tests/acceptance/refs/square-state.state create mode 100644 tests/acceptance/refs/square-state.wav create mode 100644 tests/acceptance/refs/square-state.wav.json create mode 100644 tests/acceptance/sine-440.json.in create mode 100644 tests/acceptance/square-220.json.in create mode 100644 tests/acceptance/square-state.json.in create mode 100644 tests/test_plugins/gain/CMakeLists.txt create mode 100644 tests/test_plugins/gain/GainPlugin.cpp create mode 100644 tests/test_plugins/gain/GainPlugin.h create mode 100644 tests/test_plugins/tone_generator/CMakeLists.txt create mode 100644 tests/test_plugins/tone_generator/ToneGeneratorPlugin.cpp create mode 100644 tests/test_plugins/tone_generator/ToneGeneratorPlugin.h diff --git a/.github/workflows/acceptance.yml b/.github/workflows/acceptance.yml new file mode 100644 index 00000000..a216337b --- /dev/null +++ b/.github/workflows/acceptance.yml @@ -0,0 +1,70 @@ +name: Acceptance tests + +# Builds the in-repo dogfood plugins and runs the acceptance (golden-file) +# self-tests via CTest. This is independent of Tracktion's main private build +# pipeline; it exists so the `pluginval test` feature has a public, reproducible +# regression check across the three desktop platforms. + +on: + push: + branches: [ master, develop, v2 ] + pull_request: + workflow_dispatch: + +concurrency: + group: acceptance-${{ github.ref }} + cancel-in-progress: true + +jobs: + acceptance: + name: ${{ matrix.os }} + runs-on: ${{ matrix.os }} + strategy: + fail-fast: false + matrix: + os: [ ubuntu-latest, macos-latest, windows-latest ] + + env: + # Cache CPM's downloads (JUCE etc.) so reruns don't re-fetch them. + CPM_SOURCE_CACHE: ${{ github.workspace }}/.cpm-cache + + steps: + - uses: actions/checkout@v4 + + - name: Cache CPM sources + uses: actions/cache@v4 + with: + path: ${{ github.workspace }}/.cpm-cache + key: cpm-${{ matrix.os }}-${{ hashFiles('CMakeLists.txt', 'cmake/CPM.cmake') }} + restore-keys: cpm-${{ matrix.os }}- + + - name: Install Linux dependencies + if: runner.os == 'Linux' + run: | + sudo apt-get update + sudo apt-get install -y \ + libasound2-dev libx11-dev libxext-dev libxinerama-dev libxrandr-dev \ + libxcursor-dev libxcomposite-dev libfreetype6-dev libfontconfig1-dev \ + libgl1-mesa-dev libcurl4-openssl-dev ninja-build xvfb + + # The acceptance tests host the dogfood plugins via JUCE's own VST3 hosting, + # so the embedded VST3 validator (and its heavy SDK build) isn't needed here. + - name: Configure + run: > + cmake -B build + -DCMAKE_BUILD_TYPE=Release + -DPLUGINVAL_BUILD_TEST_PLUGINS=ON + -DPLUGINVAL_VST3_VALIDATOR=OFF + + - name: Build + run: cmake --build build --config Release + + - name: Run acceptance tests (Linux) + if: runner.os == 'Linux' + working-directory: build + run: xvfb-run --auto-servernum ctest -C Release --output-on-failure -R pluginval.acceptance + + - name: Run acceptance tests (macOS / Windows) + if: runner.os != 'Linux' + working-directory: build + run: ctest -C Release --output-on-failure -R pluginval.acceptance diff --git a/CLAUDE.md b/CLAUDE.md index 97aa0061..58b11511 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -75,7 +75,7 @@ Replace `` with the ID from step 2. ``` pluginval/ -├── Source/ # Main application source code +├── source/ # Main application source code │ ├── Main.cpp # Application entry point │ ├── MainComponent.cpp/h # GUI main window component │ ├── Validator.cpp/h # Core validation orchestration @@ -93,6 +93,11 @@ pluginval/ │ ├── vst3validator/ # Embedded VST3 validator integration │ │ ├── VST3ValidatorRunner.h │ │ └── VST3ValidatorRunner.cpp +│ ├── acceptance/ # `pluginval test` golden-file subsystem (parallel to validate) +│ │ ├── TestConfig.cpp/h # snake_case JSON config struct (independent of PluginvalSettings) +│ │ ├── AcceptanceTest.cpp/h # plugin load + state + input + render; record-or-compare orchestrator +│ │ ├── ReferenceComparator.cpp/h # Comparator interface + registry + SampleComparator +│ │ └── TestReporter.cpp/h # text + JSON result, exit code │ └── tests/ # Individual test implementations │ ├── BasicTests.cpp # Core plugin tests (info, state, audio) │ ├── BusTests.cpp # Audio bus configuration tests @@ -108,6 +113,9 @@ pluginval/ ├── tests/ │ ├── AddPluginvalTests.cmake # CMake module for CTest integration │ ├── test_plugins/ # Test plugin files +│ │ ├── tone_generator/ # Deterministic dogfood generator (PLUGINVAL_BUILD_TEST_PLUGINS) +│ │ └── gain/ # Deterministic dogfood gain effect (for the input.audio path) +│ ├── acceptance/ # Acceptance self-tests: configs (*.json.in), inputs/, checked-in refs/ WAVs │ ├── mac_tests/ # macOS-specific tests │ └── windows_tests.bat # Windows test scripts ├── docs/ # Documentation @@ -150,6 +158,7 @@ cmake --build Builds/Debug --config Debug | `WITH_ADDRESS_SANITIZER` | Enable AddressSanitizer | OFF | | `WITH_THREAD_SANITIZER` | Enable ThreadSanitizer | OFF | | `VST2_SDK_DIR` | Path to VST2 SDK (env var) | - | +| `PLUGINVAL_BUILD_TEST_PLUGINS` | Build the in-repo dogfood plugins + acceptance CTest self-tests | OFF | ### Enabling VST2 Support @@ -258,6 +267,46 @@ settings set via a base64-encoded JSON argument (`--config-base64`), avoiding per-flag re-serialisation and command-line quoting hazards. `--help`/`--version` are handled by CLI11 (auto usage + a footer with the env-var/commands notes). +### Acceptance Testing (`pluginval test`) + +A **parallel subsystem** to validate, in `source/acceptance/`. It answers "does +this plugin produce the expected output for a known input + state?" — a +deterministic *render + golden-file comparison*, not a unit-test pass/fail. Full +spec: `tests/acceptance/Acceptance testing design.md`; end-user guide: +`docs/Acceptance testing.md`. + +- **CLI**: `pluginval test `. A new `Command::test` is recognised + by `settings_parser::dispatch()` (captures the positional config path into + `DispatchResult::testConfigPath`), `isCommandLine()` and `getFooterText()`; + `CommandLine.cpp`'s `performCommandLine()` has a `Command::test` branch that + runs the acceptance runner **synchronously on the message thread** and quits. +- **Config**: `acceptance::TestConfig` (`TestConfig.cpp/h`) — std-typed struct, + **snake_case JSON keys** mapped via explicit `to_json`/`from_json` (members + stay camelCase). It is **independent** of `PluginvalSettings` / the `--config` + layering: the test config is a positional argument loaded standalone. +- **Flow** (`AcceptanceTest.cpp`): load plugin → apply `state.file` then + `state.parameters` (normalised, matched by index / case-insensitive name or + paramID) → feed `input.audio`/`input.midi` or silence → render a fixed + duration block-by-block (reusing the `AudioProcessingTest` shape + the + VST3-safe helpers in `TestUtilities.h`). If no reference exists it **records** + one (32-bit float WAV + `.wav.json` sidecar manifest); otherwise it + **compares** and writes a diff WAV on failure. Exit `0`/`1`. +- **Comparators** (`ReferenceComparator.cpp/h`): pluggable `Comparator` + + `createComparator(name)` registry. v1 ships only `sample` (per-sample abs-diff + tolerance, default one 16-bit LSB = `1/32768`; `0` = bit-exact). Adding + `spectrum`/`crosscorr`/etc. is one registry entry, no config/runner changes. +- **Dogfood + self-tests**: two minimal deterministic `juce_add_plugin` targets + behind `PLUGINVAL_BUILD_TEST_PLUGINS` — `tests/test_plugins/tone_generator/` + (closed-form sine/square generator, phase resets on `prepareToPlay`) and + `tests/test_plugins/gain/` (a gain effect, used to dogfood the `input.audio` + path: it gains a checked-in full-height sine and is compared bit-exact). + `tests/acceptance/` holds checked-in configs (`*.json.in`, the plugin paths + + input dir substituted at configure time), `inputs/` and reference WAVs, run via + CTest (`pluginval.acceptance.*`: sine-440, square-220, square-state, gain-half). + Phase 2 items (config-array multiplexing, + automation, playhead, extra comparators, child-process isolation) are notes + only. + ### Test Framework Tests are self-registering. To find all tests, look for static instances: @@ -298,12 +347,12 @@ The VST3 validator (Steinberg's vstvalidator) is embedded into pluginval when bu 1. The VST3 SDK is fetched via CPM during CMake configure 2. The SDK's own `validator` target is built as a separate executable 3. A CMake script (`cmake/GenerateBinaryHeader.cmake`) converts the compiled binary into a C byte array header -4. `VST3ValidatorRunner` (`Source/vst3validator/`) extracts the embedded binary to a temp file on first use +4. `VST3ValidatorRunner` (`source/vst3validator/`) extracts the embedded binary to a temp file on first use 5. When the `VST3validator` test runs, it spawns the extracted validator as a subprocess **Key files:** - `cmake/GenerateBinaryHeader.cmake` — binary-to-C-header conversion script -- `Source/vst3validator/VST3ValidatorRunner.h/cpp` — extracts embedded binary, returns `juce::File` +- `source/vst3validator/VST3ValidatorRunner.h/cpp` — extracts embedded binary, returns `juce::File` **Disabling embedded validator:** ```bash @@ -486,7 +535,7 @@ Run internal tests via CLI: ## Common Tasks for AI Assistants ### Finding Where Tests Are Defined -- All test classes are in `Source/tests/*.cpp` +- All test classes are in `source/tests/*.cpp` - Search for `static.*Test.*Test;` to find registrations - Each test subclasses `PluginTest` diff --git a/CMakeLists.txt b/CMakeLists.txt index 4a7eab78..6a333660 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -28,6 +28,10 @@ endif() option(WITH_ADDRESS_SANITIZER "Enable Address Sanitizer" OFF) option(WITH_THREAD_SANITIZER "Enable Thread Sanitizer" OFF) +# Builds the in-repo dogfood plugins (e.g. the tone generator) used by the +# acceptance self-tests. Off for normal release builds. +option(PLUGINVAL_BUILD_TEST_PLUGINS "Build the in-repo test plugins used by the acceptance self-tests" OFF) + message(STATUS "Sanitizers: ASan=${WITH_ADDRESS_SANITIZER} TSan=${WITH_THREAD_SANITIZER}") if (WITH_ADDRESS_SANITIZER) if (MSVC) @@ -105,7 +109,7 @@ endif() juce_add_gui_app(pluginval BUNDLE_ID com.Tracktion.pluginval COMPANY_NAME Tracktion - ICON_BIG "${CMAKE_CURRENT_SOURCE_DIR}/Source/binarydata/icon.png" + ICON_BIG "${CMAKE_CURRENT_SOURCE_DIR}/source/binarydata/icon.png" HARDENED_RUNTIME_ENABLED TRUE HARDENED_RUNTIME_OPTIONS com.apple.security.cs.allow-unsigned-executable-memory com.apple.security.cs.disable-library-validation com.apple.security.get-task-allow) @@ -118,42 +122,50 @@ set_target_properties(pluginval PROPERTIES CXX_VISIBILITY_PRESET hidden) set(SourceFiles - Source/CommandLine.h - Source/CrashHandler.h - Source/MainComponent.h - Source/PluginTests.h - Source/PluginvalSettings.h - Source/SettingsParser.h - Source/SettingsSerializer.h - Source/TestUtilities.h - Source/Validator.h - Source/CommandLine.cpp - Source/CrashHandler.cpp - Source/Main.cpp - Source/MainComponent.cpp - Source/PluginTests.cpp - Source/SettingsParser.cpp - Source/SettingsSerializer.cpp - Source/tests/BasicTests.cpp - Source/tests/LocaleTest.cpp - Source/tests/BusTests.cpp - Source/tests/EditorTests.cpp - Source/tests/ExtremeTests.cpp - Source/tests/ParameterFuzzTests.cpp - Source/TestUtilities.cpp - Source/Validator.cpp) + source/CommandLine.h + source/CrashHandler.h + source/MainComponent.h + source/PluginTests.h + source/PluginvalSettings.h + source/SettingsParser.h + source/SettingsSerializer.h + source/TestUtilities.h + source/Validator.h + source/acceptance/TestConfig.h + source/acceptance/TestConfig.cpp + source/acceptance/AcceptanceTest.h + source/acceptance/AcceptanceTest.cpp + source/acceptance/ReferenceComparator.h + source/acceptance/ReferenceComparator.cpp + source/acceptance/TestReporter.h + source/acceptance/TestReporter.cpp + source/CommandLine.cpp + source/CrashHandler.cpp + source/Main.cpp + source/MainComponent.cpp + source/PluginTests.cpp + source/SettingsParser.cpp + source/SettingsSerializer.cpp + source/tests/BasicTests.cpp + source/tests/LocaleTest.cpp + source/tests/BusTests.cpp + source/tests/EditorTests.cpp + source/tests/ExtremeTests.cpp + source/tests/ParameterFuzzTests.cpp + source/TestUtilities.cpp + source/Validator.cpp) target_sources(pluginval PRIVATE ${SourceFiles}) -source_group(TREE ${CMAKE_CURRENT_SOURCE_DIR}/Source PREFIX Source FILES ${SourceFiles}) +source_group(TREE ${CMAKE_CURRENT_SOURCE_DIR}/source PREFIX Source FILES ${SourceFiles}) # Add VST3 validator runner sources to pluginval (extracts embedded binary at runtime) if(PLUGINVAL_VST3_VALIDATOR) set(VST3ValidatorFiles - Source/vst3validator/VST3ValidatorRunner.h - Source/vst3validator/VST3ValidatorRunner.cpp + source/vst3validator/VST3ValidatorRunner.h + source/vst3validator/VST3ValidatorRunner.cpp ) target_sources(pluginval PRIVATE ${VST3ValidatorFiles}) - source_group("Source/vst3validator" FILES ${VST3ValidatorFiles}) + source_group("source/vst3validator" FILES ${VST3ValidatorFiles}) endif() if (DEFINED ENV{VST2_SDK_DIR}) @@ -256,3 +268,11 @@ else() DEPENDS pluginval ${PLUGINVAL_TARGET} COMMENT "Run pluginval CLI with strict validation") endif() + +# In-repo dogfood plugins + acceptance self-tests. +if (PLUGINVAL_BUILD_TEST_PLUGINS) + enable_testing() + add_subdirectory(tests/test_plugins/tone_generator) + add_subdirectory(tests/test_plugins/gain) + add_subdirectory(tests/acceptance) +endif() diff --git a/README.md b/README.md index 4ad3a92a..144a6bc9 100644 --- a/README.md +++ b/README.md @@ -102,6 +102,7 @@ This means you can check the exit code on your various CI and mark builds a fail - [Testing plugins with pluginval]() - [Debugging a failed validation]() - [Adding pluginval to CI]() + - [Acceptance testing]() ### Contributing If you would like to contribute to the project please do! It's very simple to add tests, simply: diff --git a/docs/Acceptance testing.md b/docs/Acceptance testing.md new file mode 100644 index 00000000..95c0a6a7 --- /dev/null +++ b/docs/Acceptance testing.md @@ -0,0 +1,75 @@ +# Acceptance testing + +Acceptance testing answers a different question from normal validation. Instead of +"does this plugin conform to the host API and behave safely?" it asks **"does this +plugin still produce the output I expect for a known input and state?"** — a +deterministic *render + golden-file comparison*. It is ideal for catching +accidental DSP changes (a wrong coefficient, a refactor that shifts the output) in +your own plugins from CI. + +It is driven by a small JSON config and a dedicated command: + +``` +pluginval test myPlugin-default.json +``` + +- The **first** run renders the plugin and, finding no reference, **records** one + (a `.wav` plus a `.wav.json` manifest) next to the config and reports success. +- **Subsequent** runs render again and **compare** against that reference, + exiting `0` on a match and `1` on a mismatch (writing a diff `.wav` to help you + see what changed). + +So the workflow is: write a config, run once to record the reference, **commit the +config and the reference**, then let CI run the same command on every change. + +### A minimal config + +```json +{ + "name": "myReverb-default", + "plugin": "/path/to/MyReverb.vst3", + "input": { "audio": "inputs/drums.wav" }, + "reference": "refs/myReverb-default.wav", + "state": { "parameters": { "Mix": 0.5, "Decay": 0.8 } }, + "sample_rate": 48000, + "block_size": 512, + "render_duration": 2.0, + "comparison": { "sample": 1e-6 } +} +``` + +JSON keys are `snake_case`. The most useful fields: + +| Field | Notes | +|---|---| +| `plugin` | Path to the plugin (or an AU identifier). **Required.** | +| `input.audio` / `input.midi` | Input files to feed it. Omit both for silence (e.g. instruments driven only by MIDI, or generators). | +| `state.parameters` | A map of parameter name (or index) → **normalised** value (`0`–`1`), applied before rendering. | +| `state.file` | A binary `getStateInformation` blob to restore first (e.g. a captured preset). Applied before `state.parameters`. | +| `reference` | The golden `.wav`. Defaults to `.wav` next to the config. | +| `render_duration` | Seconds to render. If omitted, the input audio's length is used. | +| `comparison` | How to compare. `{ "sample": }` is a per-sample absolute-difference tolerance (`0` = bit-exact); the default is one 16-bit LSB. | + +### Determinism matters + +Acceptance testing only works for output that is reproducible from a fixed input +and state. A plugin with free-running randomness can't be golden-tested reliably. +For the same reason, references are only safely **portable across platforms** when +you allow a tolerance — exact per-sample matches rarely survive different CPUs and +floating-point libraries. If a reference recorded on one OS fails on another, raise +the `sample` tolerance (or use a more tolerant comparison method as they are added). + +### Running from CI + +`pluginval test` is just a command that returns an exit code, so any CI system can +run it the same way it runs your other checks: + +```bash +pluginval test tests/acceptance/myReverb-default.json +``` + +A non-zero exit fails the build. Commit the config and its reference `.wav` +alongside your project so every run compares against the same golden file. + +For the complete schema, the comparator design and the planned roadmap, see the +[design document](<../tests/acceptance/Acceptance testing design.md>). diff --git a/docs/Command line options.md b/docs/Command line options.md index d87244b8..6c4c66a5 100644 --- a/docs/Command line options.md +++ b/docs/Command line options.md @@ -11,6 +11,9 @@ COMMANDS: "pluginval [options] " also work. run-tests Run the internal unit tests. strictness-help [level] List all tests that run at the given strictness level. + test Run a deterministic acceptance (golden-file) test + from a config. Records a reference on first run, + compares against it afterwards (exit 0/1). The flat flags --validate , --run-tests and --strictness-help [level] are deprecated aliases for the commands above and will be removed in a future diff --git a/source/CommandLine.cpp b/source/CommandLine.cpp index 282c9b50..7b46af1c 100644 --- a/source/CommandLine.cpp +++ b/source/CommandLine.cpp @@ -18,6 +18,7 @@ #include "PluginTests.h" #include "PluginvalSettings.h" #include "SettingsParser.h" +#include "acceptance/AcceptanceTest.h" #include #include @@ -236,6 +237,22 @@ void performCommandLine (CommandLineValidator& validator, const juce::String& co return; } + if (routed.command == settings_parser::Command::test) + { + if (routed.testConfigPath.isEmpty()) + { + exitWithError ("*** FAILED: No acceptance-test config specified (usage: pluginval test )"); + return; + } + + // The acceptance runner needs the message thread for the VST3-safe + // lifecycle helpers, and we are already on it here, so run synchronously. + const auto configFile = juce::File::getCurrentWorkingDirectory().getChildFile (routed.testConfigPath); + app.setApplicationReturnValue (acceptance::runTestFile (configFile)); + app.quit(); + return; + } + // Otherwise this is a validation run (positional plugin, or explicit/implicit // --validate). CLI11 handles --help/--version and parse errors. if (routed.deprecatedAlias) diff --git a/source/CommandLineTests.cpp b/source/CommandLineTests.cpp index b19a8106..237007e0 100644 --- a/source/CommandLineTests.cpp +++ b/source/CommandLineTests.cpp @@ -436,6 +436,75 @@ struct CommandLineTests : public juce::UnitTest expect (shouldPerformCommandLine ("strictness-help")); expect (shouldPerformCommandLine ("validate MyPlugin.vst3")); expect (shouldPerformCommandLine ("validate MyPluginID")); + expect (shouldPerformCommandLine ("test config.json")); + } + + beginTest ("Subcommand: test captures the positional config path"); + { + using settings_parser::Command; + + { + const auto d = settings_parser::dispatch (settings_parser::tokenise ("test config.json")); + expect (d.command == Command::test); + expect (! d.deprecatedAlias); + expectEquals (d.testConfigPath, juce::String ("config.json")); + } + { + // The config path is the first non-option token after the verb. + const auto d = settings_parser::dispatch (settings_parser::tokenise ("test ./refs/sine.json")); + expect (d.command == Command::test); + expectEquals (d.testConfigPath, juce::String ("./refs/sine.json")); + } + { + // Missing positional -> empty path (the runner reports the usage error). + const auto d = settings_parser::dispatch (settings_parser::tokenise ("test")); + expect (d.command == Command::test); + expect (d.testConfigPath.isEmpty()); + } + } + + beginTest ("Acceptance TestConfig JSON parsing (snake_case keys, defaults)"); + { + const auto json = R"({ + "name": "myReverb-default", + "plugin": "/path/to/Plugin.vst3", + "input": { "audio": "in.wav" }, + "state": { "parameters": { "Mix": 0.5, "3": 1.0 } }, + "sample_rate": 48000, + "block_size": 256, + "render_duration": 2.0 + })"; + + const auto config = nlohmann::json::parse (json).get(); + + expectEquals (config.getName(), juce::String ("myReverb-default")); + expectEquals (juce::String (config.plugin), juce::String ("/path/to/Plugin.vst3")); + expectEquals (juce::String (config.inputAudio), juce::String ("in.wav")); + expect (config.inputMidi.empty()); + expectEquals (config.sampleRate, 48000.0); + expectEquals (config.blockSize, 256); + expect (config.renderDuration.has_value()); + expectEquals (*config.renderDuration, 2.0); + expectEquals ((int) config.stateParameters.size(), 2); + expectEquals (config.stateParameters.at ("Mix"), 0.5); + expectEquals (config.stateParameters.at ("3"), 1.0); + + // Omitted comparison falls back to one 16-bit LSB. + const auto comparison = config.getComparison(); + expect (comparison.contains ("sample")); + expectEquals (comparison["sample"].get(), 1.0 / 32768.0); + } + + beginTest ("Acceptance TestConfig omitted render_duration and explicit comparison"); + { + const auto json = R"({ + "plugin": "Plugin.vst3", + "comparison": { "sample": 0.0 } + })"; + + const auto config = nlohmann::json::parse (json).get(); + expect (! config.renderDuration.has_value()); + expectEquals (config.getComparison()["sample"].get(), 0.0); } } }; diff --git a/source/SettingsParser.cpp b/source/SettingsParser.cpp index e93cb76e..61daea8d 100644 --- a/source/SettingsParser.cpp +++ b/source/SettingsParser.cpp @@ -130,6 +130,7 @@ R"(Commands: validate [options] Validate the plugin at the given path or AU id (the default). run-tests Run the internal unit tests. strictness-help [level] List all tests that run at the given strictness level. + test Run a deterministic acceptance (golden-file) test from a config. The flat flags --validate , --run-tests and --strictness-help [level] are deprecated aliases for the commands above and will be removed in a future version. @@ -313,7 +314,7 @@ Precedence (lowest to highest): defaults, environment variables, --config, comma { const auto& verb = tokens.getReference (0); - if (verb == "validate" || verb == "run-tests" || verb == "strictness-help") + if (verb == "validate" || verb == "run-tests" || verb == "strictness-help" || verb == "test") return true; } @@ -365,6 +366,23 @@ Precedence (lowest to highest): defaults, environment variables, --config, comma result.strictnessLevel = levelAfter (tokensIn, 0); return result; } + + if (verb == "test") + { + result.command = Command::test; + + // The positional acceptance-test config (first non-option token after the verb). + for (int i = 1; i < tokensIn.size(); ++i) + { + if (! tokensIn.getReference (i).startsWith ("-")) + { + result.testConfigPath = tokensIn.getReference (i); + break; + } + } + + return result; + } } // 2. Deprecated flat command flags. diff --git a/source/SettingsParser.h b/source/SettingsParser.h index 15089faa..1898fb4b 100644 --- a/source/SettingsParser.h +++ b/source/SettingsParser.h @@ -69,7 +69,8 @@ namespace settings_parser { validate, /**< Validate a plugin (the default when no verb is given). */ runTests, /**< Run the internal unit tests. */ - strictnessHelp /**< List the tests that run at a given strictness level. */ + strictnessHelp, /**< List the tests that run at a given strictness level. */ + test /**< Run a deterministic acceptance (golden-file) test from a config. */ }; /** The outcome of routing tokens to a subcommand. */ @@ -79,6 +80,7 @@ namespace settings_parser juce::StringArray validateTokens; /**< Verb-stripped option tokens to feed parseTokens (validate only). */ bool deprecatedAlias = false; /**< A legacy flat flag (--validate/--run-tests/--strictness-help) was used. */ int strictnessLevel = 5; /**< The level for the strictnessHelp command. */ + juce::String testConfigPath; /**< The positional config.json for the test command. */ }; /** Routes tokenised input to a subcommand, peeling the leading verb. The diff --git a/source/acceptance/AcceptanceTest.cpp b/source/acceptance/AcceptanceTest.cpp new file mode 100644 index 00000000..dc907660 --- /dev/null +++ b/source/acceptance/AcceptanceTest.cpp @@ -0,0 +1,449 @@ +/*============================================================================== + + Copyright 2018 by Tracktion Corporation. + For more information visit www.tracktion.com + + You may also use this code under the terms of the GPL v3 (see + www.gnu.org/licenses). + + pluginval IS PROVIDED "AS IS" WITHOUT ANY WARRANTY, AND ALL WARRANTIES, WHETHER + EXPRESSED OR IMPLIED, INCLUDING MERCHANTABILITY AND FITNESS FOR PURPOSE, ARE + DISCLAIMED. + + ==============================================================================*/ + +#include "AcceptanceTest.h" +#include "../TestUtilities.h" + +#include + +#include +#include +#include + +namespace acceptance +{ + +//============================================================================== +namespace +{ + juce::String archString() + { + #if defined (__aarch64__) || defined (_M_ARM64) + return "arm64"; + #elif defined (__x86_64__) || defined (_M_X64) + return "x86_64"; + #else + return "unknown"; + #endif + } + + //============================================================================== + std::unique_ptr loadPlugin (juce::AudioPluginFormatManager& formatManager, + const juce::String& pathOrID, + double sampleRate, int blockSize) + { + juce::KnownPluginList list; + juce::OwnedArray found; + list.scanAndAddDragAndDroppedFiles (formatManager, juce::StringArray (pathOrID), found); + + if (found.isEmpty()) + throw std::runtime_error (("no plugin found at: " + pathOrID + + " (missing/damaged binary, an incompatible format, or an AU not registered with macOS)").toStdString()); + + juce::String error; + auto instance = std::unique_ptr ( + formatManager.createPluginInstance (*found.getFirst(), sampleRate, blockSize, error)); + + if (instance == nullptr) + throw std::runtime_error (("failed to create plugin instance: " + error).toStdString()); + + return instance; + } + + //============================================================================== + void applyState (juce::AudioPluginInstance& instance, const TestConfig& config) + { + // Binary state first (the full plugin state), then the parameter map as + // overrides (see the precedence rule in the spec). + if (const auto stateFile = config.getStateFile(); stateFile != juce::File()) + { + if (! stateFile.existsAsFile()) + throw std::runtime_error (("state.file not found: " + stateFile.getFullPathName()).toStdString()); + + juce::MemoryBlock state; + stateFile.loadFileAsData (state); + callSetStateInformationOnMessageThreadIfVST3 (instance, state); + } + + for (const auto& [key, value] : config.stateParameters) + { + const juce::String name (key); + juce::AudioProcessorParameter* match = nullptr; + + if (name.containsOnly ("0123456789") && name.isNotEmpty()) + { + match = instance.getParameters()[name.getIntValue()]; + } + else + { + // Match the display name (case-insensitively), or the JUCE paramID + // when the host exposes parameters as AudioProcessorParameterWithID. + for (auto* p : instance.getParameters()) + { + if (auto* withID = dynamic_cast (p)) + if (withID->paramID.equalsIgnoreCase (name)) + { + match = p; + break; + } + + if (p->getName (512).equalsIgnoreCase (name)) + { + match = p; + break; + } + } + } + + if (match == nullptr) + throw std::runtime_error (("state.parameters: no parameter named or indexed \"" + name + "\"").toStdString()); + + match->setValueNotifyingHost ((float) value); + } + } + + //============================================================================== + juce::AudioBuffer readWav (juce::AudioFormatManager& formatManager, const juce::File& file) + { + std::unique_ptr reader (formatManager.createReaderFor (file)); + + if (reader == nullptr) + throw std::runtime_error (("could not read audio file: " + file.getFullPathName()).toStdString()); + + juce::AudioBuffer buffer ((int) reader->numChannels, (int) reader->lengthInSamples); + reader->read (&buffer, 0, (int) reader->lengthInSamples, 0, true, true); + return buffer; + } + + void writeFloatWav (const juce::File& file, const juce::AudioBuffer& buffer, double sampleRate) + { + file.getParentDirectory().createDirectory(); + file.deleteFile(); + + std::unique_ptr stream (file.createOutputStream()); + + if (stream == nullptr) + throw std::runtime_error (("could not open for writing: " + file.getFullPathName()).toStdString()); + + juce::WavAudioFormat wav; + const auto options = juce::AudioFormatWriterOptions() + .withSampleRate (sampleRate) + .withNumChannels (buffer.getNumChannels()) + .withBitsPerSample (32) + .withSampleFormat (juce::AudioFormatWriterOptions::SampleFormat::floatingPoint); + + std::unique_ptr writer (wav.createWriterFor (stream, options)); + + if (writer == nullptr) + throw std::runtime_error (("could not create WAV writer for: " + file.getFullPathName()).toStdString()); + + writer->writeFromAudioSampleBuffer (buffer, 0, buffer.getNumSamples()); + } + + //============================================================================== + /** Builds a single MidiBuffer with events at absolute sample positions. */ + juce::MidiBuffer loadMidi (const juce::File& file, double sampleRate) + { + juce::FileInputStream stream (file); + + if (! stream.openedOk()) + throw std::runtime_error (("could not read MIDI file: " + file.getFullPathName()).toStdString()); + + juce::MidiFile midiFile; + + if (! midiFile.readFrom (stream)) + throw std::runtime_error (("could not parse MIDI file: " + file.getFullPathName()).toStdString()); + + midiFile.convertTimestampTicksToSeconds(); + + juce::MidiBuffer buffer; + + for (int t = 0; t < midiFile.getNumTracks(); ++t) + if (const auto* track = midiFile.getTrack (t)) + for (const auto* event : *track) + { + const auto sample = (int) std::lround (event->message.getTimeStamp() * sampleRate); + buffer.addEvent (event->message, sample); + } + + return buffer; + } + + //============================================================================== + juce::String configHash (const TestConfig& config) + { + auto j = nlohmann::json {}; + to_json (j, config); + j.erase ("reference"); // the hash must be independent of where the reference lives + + // A small, stable (cross-platform) FNV-1a 64-bit hash. The hash is only + // used for an informational "stale reference" warning, so juce_cryptography + // (MD5) isn't worth pulling in. + const auto dump = j.dump(); + std::uint64_t hash = 1469598103934665603ull; + + for (const auto c : dump) + { + hash ^= (std::uint64_t) (unsigned char) c; + hash *= 1099511628211ull; + } + + return juce::String::toHexString ((juce::int64) hash); + } +} + +//============================================================================== +RenderedAudio renderPlugin (const TestConfig& config) +{ + const auto sampleRate = config.sampleRate; + const auto blockSize = juce::jmax (1, config.blockSize); + + juce::AudioPluginFormatManager formatManager; + #if JUCE_VERSION >= 0x08000B + juce::addDefaultFormatsToManager (formatManager); + #else + formatManager.addDefaultFormats(); + #endif + + auto instance = loadPlugin (formatManager, config.getPluginPathOrID(), sampleRate, blockSize); + const auto description = instance->getPluginDescription(); + + // 1. State (file then parameter overrides), applied before prepareToPlay. + applyState (*instance, config); + + // 2. Input audio / MIDI (or silence). + juce::AudioFormatManager audioFormats; + audioFormats.registerBasicFormats(); + + juce::AudioBuffer inputAudio; + if (const auto audioFile = config.getInputAudioFile(); audioFile != juce::File()) + { + if (! audioFile.existsAsFile()) + throw std::runtime_error (("input.audio not found: " + audioFile.getFullPathName()).toStdString()); + + inputAudio = readWav (audioFormats, audioFile); + } + + juce::MidiBuffer midi; + if (const auto midiFile = config.getInputMidiFile(); midiFile != juce::File()) + { + if (! midiFile.existsAsFile()) + throw std::runtime_error (("input.midi not found: " + midiFile.getFullPathName()).toStdString()); + + midi = loadMidi (midiFile, sampleRate); + } + + // 3. Render length. + int numSamples = 0; + if (config.renderDuration) + numSamples = (int) std::lround (*config.renderDuration * sampleRate); + else if (inputAudio.getNumSamples() > 0) + numSamples = inputAudio.getNumSamples(); + + if (numSamples <= 0) + throw std::runtime_error ("render_duration is required when there is no input.audio"); + + // 4. Prepare and render block by block. + callPrepareToPlayOnMessageThreadIfVST3 (*instance, sampleRate, blockSize); + + const int numInputChannels = instance->getTotalNumInputChannels(); + const int numOutputChannels = juce::jmax (1, instance->getTotalNumOutputChannels()); + const int channelsRequired = juce::jmax (numInputChannels, numOutputChannels); + + juce::AudioBuffer output (numOutputChannels, numSamples); + output.clear(); + + juce::AudioBuffer block (channelsRequired, blockSize); + + for (int pos = 0; pos < numSamples; pos += blockSize) + { + const int thisBlock = juce::jmin (blockSize, numSamples - pos); + + block.clear(); + + // Copy the input audio slice into the block (silence past its end). + for (int c = 0; c < juce::jmin (numInputChannels, inputAudio.getNumChannels()); ++c) + { + const int available = juce::jmax (0, juce::jmin (thisBlock, inputAudio.getNumSamples() - pos)); + if (available > 0) + block.copyFrom (c, 0, inputAudio, c, pos, available); + } + + juce::MidiBuffer blockMidi; + blockMidi.addEvents (midi, pos, thisBlock, -pos); + + juce::AudioBuffer proc (block.getArrayOfWritePointers(), channelsRequired, thisBlock); + instance->processBlock (proc, blockMidi); + + for (int c = 0; c < numOutputChannels; ++c) + output.copyFrom (c, pos, proc, c, 0, thisBlock); + } + + callReleaseResourcesOnMessageThreadIfVST3 (*instance); + instance.reset(); + + return { std::move (output), sampleRate, blockSize, description }; +} + +//============================================================================== +TestResult runTest (const TestConfig& config) +{ + const auto name = config.getName(); + + try + { + const auto rendered = renderPlugin (config); + const auto referenceFile = config.getReferenceFile(); + + TestResult result; + result.name = name; + result.referenceFile = referenceFile; + + // Record mode: no reference yet -> write the float WAV + JSON sidecar. + if (! referenceFile.existsAsFile()) + { + writeFloatWav (referenceFile, rendered.buffer, rendered.sampleRate); + + nlohmann::json manifest; + manifest["plugin"] = { + { "name", rendered.description.name.toStdString() }, + { "manufacturer", rendered.description.manufacturerName.toStdString() }, + { "version", rendered.description.version.toStdString() }, + { "format", rendered.description.pluginFormatName.toStdString() }, + { "uid", rendered.description.createIdentifierString().toStdString() } + }; + manifest["render"] = { + { "sample_rate", rendered.sampleRate }, + { "block_size", rendered.blockSize }, + { "num_channels", rendered.buffer.getNumChannels() }, + { "num_samples", rendered.buffer.getNumSamples() }, + { "length_seconds", rendered.buffer.getNumSamples() / rendered.sampleRate } + }; + manifest["pluginval_version"] = VERSION; + manifest["config_hash"] = configHash (config).toStdString(); + manifest["created_on"] = { + { "os", juce::SystemStats::getOperatingSystemName().toStdString() }, + { "arch", archString().toStdString() }, + { "date", juce::Time::getCurrentTime().toISO8601 (true).toStdString() } + }; + + config.getReferenceSidecarFile().replaceWithText (manifest.dump (2)); + + result.outcome = TestResult::Outcome::referenceCreated; + return result; + } + + // Compare mode: warn (don't fail) if the stored config hash differs. + if (const auto sidecar = config.getReferenceSidecarFile(); sidecar.existsAsFile()) + { + try + { + const auto manifest = nlohmann::json::parse (sidecar.loadFileAsString().toStdString()); + if (manifest.contains ("config_hash") + && manifest["config_hash"].get() != configHash (config).toStdString()) + { + std::cout << " WARNING: reference was recorded from a different config (stale?): " + << sidecar.getFullPathName() << std::endl; + } + } + catch (const std::exception&) { /* a malformed sidecar is non-fatal */ } + } + + juce::AudioFormatManager formats; + formats.registerBasicFormats(); + const auto reference = readWav (formats, referenceFile); + + bool allPassed = true; + + const auto comparisonConfig = config.getComparison(); + + for (auto it = comparisonConfig.begin(); it != comparisonConfig.end(); ++it) + { + const juce::String method (it.key()); + auto comparator = createComparator (method); + + if (comparator == nullptr) + return TestResult::makeError (name, "unknown comparison method: " + method); + + const auto result2 = comparator->compare (reference, rendered.buffer, it.value()); + allPassed = allPassed && result2.passed; + result.comparisons.emplace_back (method, result2); + } + + result.outcome = allPassed ? TestResult::Outcome::passed : TestResult::Outcome::failed; + + if (! allPassed) + { + // Write a diff WAV (output - reference) over the overlapping region. + const int channels = juce::jmin (reference.getNumChannels(), rendered.buffer.getNumChannels()); + const int samples = juce::jmin (reference.getNumSamples(), rendered.buffer.getNumSamples()); + + if (channels > 0 && samples > 0) + { + juce::AudioBuffer diff (channels, samples); + + for (int c = 0; c < channels; ++c) + { + diff.copyFrom (c, 0, rendered.buffer, c, 0, samples); + diff.addFrom (c, 0, reference, c, 0, samples, -1.0f); + } + + const auto diffFile = config.getDiffFile(); + writeFloatWav (diffFile, diff, rendered.sampleRate); + result.diffFile = diffFile; + } + } + + return result; + } + catch (const std::exception& e) + { + return TestResult::makeError (name, e.what()); + } +} + +//============================================================================== +int runTestFile (const juce::File& configFile) +{ + std::vector configs; + + try + { + configs = TestConfig::loadFromFile (configFile); + } + catch (const std::exception& e) + { + const auto result = TestResult::makeError (configFile.getFileNameWithoutExtension(), e.what()); + reporter::report (result); + return reporter::exitCode (result); + } + + if (configs.empty()) + { + const auto result = TestResult::makeError (configFile.getFileNameWithoutExtension(), "no test configs in file"); + reporter::report (result); + return reporter::exitCode (result); + } + + // v1 runs the first entry only; the parser already accepts arrays for the + // phase-2 multiplexing extension. + if (configs.size() > 1) + std::cout << "Note: " << configs.size() << " configs found; v1 runs the first only." << std::endl; + + const auto result = runTest (configs.front()); + reporter::report (result); + return reporter::exitCode (result); +} + +} // namespace acceptance diff --git a/source/acceptance/AcceptanceTest.h b/source/acceptance/AcceptanceTest.h new file mode 100644 index 00000000..fe666619 --- /dev/null +++ b/source/acceptance/AcceptanceTest.h @@ -0,0 +1,56 @@ +/*============================================================================== + + Copyright 2018 by Tracktion Corporation. + For more information visit www.tracktion.com + + You may also use this code under the terms of the GPL v3 (see + www.gnu.org/licenses). + + pluginval IS PROVIDED "AS IS" WITHOUT ANY WARRANTY, AND ALL WARRANTIES, WHETHER + EXPRESSED OR IMPLIED, INCLUDING MERCHANTABILITY AND FITNESS FOR PURPOSE, ARE + DISCLAIMED. + + ==============================================================================*/ + +#pragma once + +#include "TestConfig.h" +#include "TestReporter.h" + +#include + +namespace acceptance +{ + +//============================================================================== +/** A rendered buffer plus the metadata needed for the reference manifest. */ +struct RenderedAudio +{ + juce::AudioBuffer buffer; + double sampleRate = 0.0; + int blockSize = 0; + juce::PluginDescription description; +}; + +//============================================================================== +/** Loads the plugin, applies state (file then parameters), feeds the input + (audio / MIDI / silence) and renders a fixed duration into a single buffer. + + Must be called on the message thread (it uses the VST3-safe lifecycle + helpers and creates the plugin instance directly). Throws std::runtime_error + on any setup / render failure. +*/ +RenderedAudio renderPlugin (const TestConfig&); + +//============================================================================== +/** Runs a single resolved config end-to-end: render, then record-or-compare, + producing a TestResult. Never throws - setup failures become an error result. */ +TestResult runTest (const TestConfig&); + +/** Loads the config file (first entry only for v1), runs it, reports the result + to stdout and returns the process exit code (0 success, 1 failure). + + Must be called on the message thread. */ +int runTestFile (const juce::File& configFile); + +} // namespace acceptance diff --git a/source/acceptance/ReferenceComparator.cpp b/source/acceptance/ReferenceComparator.cpp new file mode 100644 index 00000000..9853267c --- /dev/null +++ b/source/acceptance/ReferenceComparator.cpp @@ -0,0 +1,122 @@ +/*============================================================================== + + Copyright 2018 by Tracktion Corporation. + For more information visit www.tracktion.com + + You may also use this code under the terms of the GPL v3 (see + www.gnu.org/licenses). + + pluginval IS PROVIDED "AS IS" WITHOUT ANY WARRANTY, AND ALL WARRANTIES, WHETHER + EXPRESSED OR IMPLIED, INCLUDING MERCHANTABILITY AND FITNESS FOR PURPOSE, ARE + DISCLAIMED. + + ==============================================================================*/ + +#include "ReferenceComparator.h" + +#include + +namespace acceptance +{ + +//============================================================================== +/** + Per-sample absolute-difference comparator. + + config is the value under the "sample" key: a bare number giving the + tolerance (0 == bit-exact), or an object { "tolerance": }. Channel + and length mismatches fail outright. +*/ +struct SampleComparator : public Comparator +{ + juce::String getName() const override { return "sample"; } + + static double toleranceFrom (const nlohmann::json& config) + { + if (config.is_number()) + return config.get(); + + if (config.is_object()) + if (auto t = config.find ("tolerance"); t != config.end() && t->is_number()) + return t->get(); + + return 1.0 / 32768.0; + } + + ComparisonResult compare (const juce::AudioBuffer& reference, + const juce::AudioBuffer& output, + const nlohmann::json& config) override + { + const double tolerance = toleranceFrom (config); + + ComparisonResult r; + r.details["tolerance"] = tolerance; + + if (reference.getNumChannels() != output.getNumChannels() + || reference.getNumSamples() != output.getNumSamples()) + { + r.passed = false; + r.summary = "size mismatch: reference " + + juce::String (reference.getNumChannels()) + "ch x " + juce::String (reference.getNumSamples()) + " samples, " + + "output " + juce::String (output.getNumChannels()) + "ch x " + juce::String (output.getNumSamples()) + " samples"; + r.details["reference_channels"] = reference.getNumChannels(); + r.details["reference_samples"] = reference.getNumSamples(); + r.details["output_channels"] = output.getNumChannels(); + r.details["output_samples"] = output.getNumSamples(); + return r; + } + + double maxAbsDiff = 0.0; + int firstFailChannel = -1, firstFailSample = -1; + + for (int c = 0; c < reference.getNumChannels(); ++c) + { + const auto* ref = reference.getReadPointer (c); + const auto* out = output.getReadPointer (c); + + for (int s = 0; s < reference.getNumSamples(); ++s) + { + const double diff = std::abs ((double) out[s] - (double) ref[s]); + + if (diff > maxAbsDiff) + maxAbsDiff = diff; + + if (diff > tolerance && firstFailChannel < 0) + { + firstFailChannel = c; + firstFailSample = s; + } + } + } + + r.score = maxAbsDiff; + r.passed = maxAbsDiff <= tolerance; + r.details["max_abs_diff"] = maxAbsDiff; + + if (r.passed) + { + r.summary = "max abs diff " + juce::String (maxAbsDiff) + " <= tolerance " + juce::String (tolerance); + } + else + { + r.summary = "max abs diff " + juce::String (maxAbsDiff) + " > tolerance " + juce::String (tolerance) + + " (first at channel " + juce::String (firstFailChannel) + ", sample " + juce::String (firstFailSample) + ")"; + r.details["first_fail_channel"] = firstFailChannel; + r.details["first_fail_sample"] = firstFailSample; + } + + return r; + } +}; + +//============================================================================== +std::unique_ptr createComparator (const juce::String& name) +{ + if (name == "sample") + return std::make_unique(); + + // Future: "peakrms", "spectrum", "crosscorr", "fingerprint" register here. + return {}; +} + +} // namespace acceptance diff --git a/source/acceptance/ReferenceComparator.h b/source/acceptance/ReferenceComparator.h new file mode 100644 index 00000000..680c9d45 --- /dev/null +++ b/source/acceptance/ReferenceComparator.h @@ -0,0 +1,60 @@ +/*============================================================================== + + Copyright 2018 by Tracktion Corporation. + For more information visit www.tracktion.com + + You may also use this code under the terms of the GPL v3 (see + www.gnu.org/licenses). + + pluginval IS PROVIDED "AS IS" WITHOUT ANY WARRANTY, AND ALL WARRANTIES, WHETHER + EXPRESSED OR IMPLIED, INCLUDING MERCHANTABILITY AND FITNESS FOR PURPOSE, ARE + DISCLAIMED. + + ==============================================================================*/ + +#pragma once + +#include +#include + +#include + +namespace acceptance +{ + +//============================================================================== +/** The outcome of a single comparator run. */ +struct ComparisonResult +{ + bool passed = false; + double score = 0.0; /**< Method-specific metric, e.g. max abs diff. */ + juce::String summary; /**< Human-readable one-liner. */ + nlohmann::json details; /**< Structured detail for the machine-readable report. */ +}; + +//============================================================================== +/** + Pluggable comparison method. v1 ships only "sample"; further methods + (spectrum, crosscorr, fingerprint) register later with no changes to config + parsing or the runner. +*/ +struct Comparator +{ + virtual ~Comparator() = default; + + /** The method name, e.g. "sample". */ + virtual juce::String getName() const = 0; + + /** Compares a freshly rendered buffer against the reference. config is the + value sitting under this comparator's key in the config's "comparison" + map (a bare number for "sample", or an object for richer methods). */ + virtual ComparisonResult compare (const juce::AudioBuffer& reference, + const juce::AudioBuffer& output, + const nlohmann::json& config) = 0; +}; + +//============================================================================== +/** Creates a comparator by name, or nullptr if the name is unknown. */ +std::unique_ptr createComparator (const juce::String& name); + +} // namespace acceptance diff --git a/source/acceptance/TestConfig.cpp b/source/acceptance/TestConfig.cpp new file mode 100644 index 00000000..a3292c29 --- /dev/null +++ b/source/acceptance/TestConfig.cpp @@ -0,0 +1,208 @@ +/*============================================================================== + + Copyright 2018 by Tracktion Corporation. + For more information visit www.tracktion.com + + You may also use this code under the terms of the GPL v3 (see + www.gnu.org/licenses). + + pluginval IS PROVIDED "AS IS" WITHOUT ANY WARRANTY, AND ALL WARRANTIES, WHETHER + EXPRESSED OR IMPLIED, INCLUDING MERCHANTABILITY AND FITNESS FOR PURPOSE, ARE + DISCLAIMED. + + ==============================================================================*/ + +#include "TestConfig.h" + +#include + +namespace acceptance +{ + +//============================================================================== +namespace +{ + /** The default comparison when none is supplied: one 16-bit LSB. */ + nlohmann::json defaultComparison() + { + return nlohmann::json { { "sample", 1.0 / 32768.0 } }; + } + + /** Reads a string member from a (possibly absent) nested object. */ + std::string getNestedString (const nlohmann::json& j, const char* outer, const char* inner) + { + if (auto o = j.find (outer); o != j.end() && o->is_object()) + if (auto i = o->find (inner); i != o->end() && i->is_string()) + return i->get(); + + return {}; + } +} + +//============================================================================== +void to_json (nlohmann::json& j, const TestConfig& c) +{ + j = nlohmann::json::object(); + j["name"] = c.name; + j["plugin"] = c.plugin; + + if (! c.inputAudio.empty() || ! c.inputMidi.empty()) + { + auto input = nlohmann::json::object(); + if (! c.inputAudio.empty()) input["audio"] = c.inputAudio; + if (! c.inputMidi.empty()) input["midi"] = c.inputMidi; + j["input"] = input; + } + + if (! c.reference.empty()) + j["reference"] = c.reference; + + if (! c.stateFile.empty() || ! c.stateParameters.empty()) + { + auto state = nlohmann::json::object(); + if (! c.stateFile.empty()) state["file"] = c.stateFile; + if (! c.stateParameters.empty()) state["parameters"] = c.stateParameters; + j["state"] = state; + } + + j["sample_rate"] = c.sampleRate; + j["block_size"] = c.blockSize; + + if (c.renderDuration) + j["render_duration"] = *c.renderDuration; + + if (! c.comparison.is_null()) + j["comparison"] = c.comparison; +} + +void from_json (const nlohmann::json& j, TestConfig& c) +{ + c.name = j.value ("name", std::string()); + c.plugin = j.value ("plugin", std::string()); + c.reference = j.value ("reference", std::string()); + + c.inputAudio = getNestedString (j, "input", "audio"); + c.inputMidi = getNestedString (j, "input", "midi"); + + c.stateFile = getNestedString (j, "state", "file"); + + if (auto state = j.find ("state"); state != j.end() && state->is_object()) + if (auto params = state->find ("parameters"); params != state->end() && params->is_object()) + c.stateParameters = params->get>(); + + c.sampleRate = j.value ("sample_rate", 44100.0); + c.blockSize = j.value ("block_size", 512); + + if (auto d = j.find ("render_duration"); d != j.end() && d->is_number()) + c.renderDuration = d->get(); + + if (auto comp = j.find ("comparison"); comp != j.end() && ! comp->is_null()) + c.comparison = *comp; +} + +//============================================================================== +nlohmann::json TestConfig::getComparison() const +{ + return comparison.is_null() ? defaultComparison() : comparison; +} + +juce::String TestConfig::getPluginPathOrID() const +{ + const juce::String raw (plugin); + + // Resolve relative/home paths against the working directory; leave absolute + // paths and bare component IDs (no '.' or '~') untouched. Mirrors + // settings_parser::resolvePluginPath. + if (raw.contains ("~") || raw.contains (".")) + return juce::File::getCurrentWorkingDirectory().getChildFile (raw).getFullPathName(); + + return raw; +} + +juce::File TestConfig::resolveAgainstConfigDir (const std::string& path) const +{ + if (path.empty()) + return {}; + + const juce::String s (path); + + if (juce::File::isAbsolutePath (s)) + return juce::File (s); + + const auto base = configDir == juce::File() ? juce::File::getCurrentWorkingDirectory() : configDir; + return base.getChildFile (s); +} + +juce::String TestConfig::getName() const +{ + if (! name.empty()) + return juce::String (name); + + return getReferenceFile().getFileNameWithoutExtension(); +} + +juce::File TestConfig::getReferenceFile() const +{ + if (! reference.empty()) + return resolveAgainstConfigDir (reference); + + const auto base = configDir == juce::File() ? juce::File::getCurrentWorkingDirectory() : configDir; + return base.getChildFile (juce::String (name) + ".wav"); +} + +juce::File TestConfig::getReferenceSidecarFile() const +{ + const auto ref = getReferenceFile(); + return ref.getSiblingFile (ref.getFileName() + ".json"); +} + +juce::File TestConfig::getDiffFile() const +{ + const auto ref = getReferenceFile(); + return ref.getSiblingFile (ref.getFileNameWithoutExtension() + "-diff.wav"); +} + +juce::File TestConfig::getInputAudioFile() const { return resolveAgainstConfigDir (inputAudio); } +juce::File TestConfig::getInputMidiFile() const { return resolveAgainstConfigDir (inputMidi); } +juce::File TestConfig::getStateFile() const { return resolveAgainstConfigDir (stateFile); } + +//============================================================================== +std::vector TestConfig::loadFromFile (const juce::File& file) +{ + if (! file.existsAsFile()) + throw std::runtime_error (("test config not found: " + file.getFullPathName()).toStdString()); + + nlohmann::json j; + + try + { + j = nlohmann::json::parse (file.loadFileAsString().toStdString()); + } + catch (const std::exception& e) + { + throw std::runtime_error (("failed to parse test config " + file.getFullPathName() + ": " + e.what()).toStdString()); + } + + std::vector configs; + + const auto addOne = [&] (const nlohmann::json& entry) + { + auto c = entry.get(); + c.configDir = file.getParentDirectory(); + configs.push_back (std::move (c)); + }; + + if (j.is_array()) + { + for (const auto& entry : j) + addOne (entry); + } + else + { + addOne (j); + } + + return configs; +} + +} // namespace acceptance diff --git a/source/acceptance/TestConfig.h b/source/acceptance/TestConfig.h new file mode 100644 index 00000000..fdf0dae7 --- /dev/null +++ b/source/acceptance/TestConfig.h @@ -0,0 +1,104 @@ +/*============================================================================== + + Copyright 2018 by Tracktion Corporation. + For more information visit www.tracktion.com + + You may also use this code under the terms of the GPL v3 (see + www.gnu.org/licenses). + + pluginval IS PROVIDED "AS IS" WITHOUT ANY WARRANTY, AND ALL WARRANTIES, WHETHER + EXPRESSED OR IMPLIED, INCLUDING MERCHANTABILITY AND FITNESS FOR PURPOSE, ARE + DISCLAIMED. + + ==============================================================================*/ + +#pragma once + +#include +#include + +#include +#include +#include +#include + +namespace acceptance +{ + +//============================================================================== +/** + The definition of a single acceptance test. + + This is a plain, std-typed struct deserialised from the positional + of the "pluginval test" command. JSON keys are snake_case + (e.g. sample_rate); the C++ members stay JUCE-style camelCase, so the mapping + is done explicitly in the per-struct to_json / from_json (see TestConfig.cpp) + rather than the bare NLOHMANN macro. + + This config is completely independent of PluginvalSettings / the --config + settings layering used by the validate command. +*/ +struct TestConfig +{ + std::string name; /**< Labels results; derives the default reference path. */ + std::string plugin; /**< Plugin path or AU id. Required. */ + std::string inputAudio; /**< input.audio: path to an input audio file, or empty for silence. */ + std::string inputMidi; /**< input.midi: path to a .mid file, or empty for none. */ + std::string reference; /**< The golden file. Empty -> derived from name. */ + std::string stateFile; /**< state.file: binary getStateInformation blob. */ + std::map stateParameters; /**< state.parameters: name-or-index -> normalised value. */ + double sampleRate = 44100.0; + int blockSize = 512; + std::optional renderDuration; /**< Seconds. Unset -> derive from the input length. */ + nlohmann::json comparison; /**< Map of comparator name -> sub-config. Empty -> default. */ + + //============================================================================== + /** The directory the config was loaded from. Relative reference / input / + state paths resolve against this; the plugin path resolves against the + working directory (per the spec). Not part of the JSON. */ + juce::File configDir; + + //============================================================================== + /** Returns the resolved comparison map, substituting the default + ({ "sample": 1/32768 }) when none was supplied. */ + nlohmann::json getComparison() const; + + /** The plugin path or AU id, resolved against the working directory. */ + juce::String getPluginPathOrID() const; + + /** The golden reference file (explicit, or /.wav). */ + juce::File getReferenceFile() const; + + /** The reference's JSON sidecar (.json). */ + juce::File getReferenceSidecarFile() const; + + /** The diff WAV written next to the reference on a comparison failure. */ + juce::File getDiffFile() const; + + /** Resolved input audio file, or an empty File if none. */ + juce::File getInputAudioFile() const; + + /** Resolved input MIDI file, or an empty File if none. */ + juce::File getInputMidiFile() const; + + /** Resolved state file, or an empty File if none. */ + juce::File getStateFile() const; + + /** The effective test name (explicit, or the reference's basename). */ + juce::String getName() const; + + //============================================================================== + /** Loads one or more configs from a JSON file. The top level may be a single + object or an array of objects (phase 2 multiplexing); v1 callers use the + first entry. Throws std::runtime_error on a load / parse error. */ + static std::vector loadFromFile (const juce::File&); + +private: + juce::File resolveAgainstConfigDir (const std::string& path) const; +}; + +//============================================================================== +void to_json (nlohmann::json&, const TestConfig&); +void from_json (const nlohmann::json&, TestConfig&); + +} // namespace acceptance diff --git a/source/acceptance/TestReporter.cpp b/source/acceptance/TestReporter.cpp new file mode 100644 index 00000000..a95c100e --- /dev/null +++ b/source/acceptance/TestReporter.cpp @@ -0,0 +1,110 @@ +/*============================================================================== + + Copyright 2018 by Tracktion Corporation. + For more information visit www.tracktion.com + + You may also use this code under the terms of the GPL v3 (see + www.gnu.org/licenses). + + pluginval IS PROVIDED "AS IS" WITHOUT ANY WARRANTY, AND ALL WARRANTIES, WHETHER + EXPRESSED OR IMPLIED, INCLUDING MERCHANTABILITY AND FITNESS FOR PURPOSE, ARE + DISCLAIMED. + + ==============================================================================*/ + +#include "TestReporter.h" + +#include + +namespace acceptance::reporter +{ + +//============================================================================== +static const char* toString (TestResult::Outcome o) +{ + switch (o) + { + case TestResult::Outcome::referenceCreated: return "reference-created"; + case TestResult::Outcome::passed: return "passed"; + case TestResult::Outcome::failed: return "failed"; + case TestResult::Outcome::error: return "error"; + } + + return "error"; +} + +nlohmann::json toJson (const TestResult& r) +{ + nlohmann::json j; + j["name"] = r.name.toStdString(); + j["outcome"] = toString (r.outcome); + + if (r.message.isNotEmpty()) + j["message"] = r.message.toStdString(); + + if (r.referenceFile != juce::File()) + j["reference"] = r.referenceFile.getFullPathName().toStdString(); + + if (r.diffFile != juce::File()) + j["diff"] = r.diffFile.getFullPathName().toStdString(); + + if (! r.comparisons.empty()) + { + auto comparisons = nlohmann::json::object(); + + for (const auto& [method, result] : r.comparisons) + { + nlohmann::json c; + c["passed"] = result.passed; + c["score"] = result.score; + c["summary"] = result.summary.toStdString(); + if (! result.details.is_null()) + c["details"] = result.details; + + comparisons[method.toStdString()] = c; + } + + j["comparisons"] = comparisons; + } + + return j; +} + +void report (const TestResult& r) +{ + switch (r.outcome) + { + case TestResult::Outcome::referenceCreated: + std::cout << "Reference created: " << r.referenceFile.getFullPathName() << std::endl; + break; + + case TestResult::Outcome::passed: + std::cout << "PASSED: " << r.name << std::endl; + break; + + case TestResult::Outcome::failed: + std::cout << "*** FAILED: " << r.name << std::endl; + break; + + case TestResult::Outcome::error: + std::cout << "*** ERROR: " << r.name << ": " << r.message << std::endl; + break; + } + + for (const auto& [method, result] : r.comparisons) + std::cout << " [" << method << "] " << (result.passed ? "ok" : "FAIL") << ": " << result.summary << std::endl; + + if (r.diffFile != juce::File()) + std::cout << " diff written to: " << r.diffFile.getFullPathName() << std::endl; + + // The structured result for machine consumers. + std::cout << toJson (r).dump (2) << std::endl; +} + +int exitCode (const TestResult& r) +{ + return (r.outcome == TestResult::Outcome::referenceCreated + || r.outcome == TestResult::Outcome::passed) ? 0 : 1; +} + +} // namespace acceptance::reporter diff --git a/source/acceptance/TestReporter.h b/source/acceptance/TestReporter.h new file mode 100644 index 00000000..10266b77 --- /dev/null +++ b/source/acceptance/TestReporter.h @@ -0,0 +1,73 @@ +/*============================================================================== + + Copyright 2018 by Tracktion Corporation. + For more information visit www.tracktion.com + + You may also use this code under the terms of the GPL v3 (see + www.gnu.org/licenses). + + pluginval IS PROVIDED "AS IS" WITHOUT ANY WARRANTY, AND ALL WARRANTIES, WHETHER + EXPRESSED OR IMPLIED, INCLUDING MERCHANTABILITY AND FITNESS FOR PURPOSE, ARE + DISCLAIMED. + + ==============================================================================*/ + +#pragma once + +#include "ReferenceComparator.h" + +#include +#include + +#include +#include + +namespace acceptance +{ + +//============================================================================== +/** The result of running one acceptance test. */ +struct TestResult +{ + enum class Outcome + { + referenceCreated, /**< No reference existed, so one was recorded. Treated as success. */ + passed, /**< Reference existed and every comparator passed. */ + failed, /**< Reference existed and at least one comparator failed. */ + error /**< Setup failed (plugin load, render, IO, bad config, ...). */ + }; + + Outcome outcome = Outcome::error; + juce::String name; + juce::String message; /**< Error text, or an overall summary. */ + + std::vector> comparisons; /**< method name -> result. */ + + juce::File referenceFile; + juce::File diffFile; /**< Written only on failure. */ + + //============================================================================== + static TestResult makeError (const juce::String& name, const juce::String& message) + { + TestResult r; + r.outcome = Outcome::error; + r.name = name; + r.message = message; + return r; + } +}; + +//============================================================================== +namespace reporter +{ + /** Structured, machine-readable representation of a result. */ + nlohmann::json toJson (const TestResult&); + + /** Prints a human-readable summary followed by the JSON to stdout. */ + void report (const TestResult&); + + /** 0 for referenceCreated / passed, 1 for failed / error. */ + int exitCode (const TestResult&); +} + +} // namespace acceptance diff --git a/tests/acceptance/Acceptance testing design.md b/tests/acceptance/Acceptance testing design.md new file mode 100644 index 00000000..ee546388 --- /dev/null +++ b/tests/acceptance/Acceptance testing design.md @@ -0,0 +1,390 @@ +# Acceptance testing (design) + +> Status: **Phase 1 implemented.** The `pluginval test ` mode +> described here is built (`source/acceptance/`, wired into the CLI dispatcher) +> with the `sample` comparator, record-or-compare, float-WAV + JSON-sidecar +> references, a dogfood tone-generator plugin (`tests/test_plugins/tone_generator/`, +> behind `PLUGINVAL_BUILD_TEST_PLUGINS`) and CTest self-tests +> (`tests/acceptance/`). The Phase 2 items in §10 remain stubs/notes. This +> document is the reference spec for both the implementation and future extension. + +## 1. Overview + +The existing `validate` mode is a graded **unit-test suite** (`PluginTests : +juce::UnitTest`, self-registering `PluginTest` instances, pass/fail via +`expect`). It answers "does this plugin conform to the host API and behave +safely?" + +Acceptance testing answers a different question: **"does this plugin produce the +output I expect for a known input and state?"** It is a deterministic +*render + golden-file comparison*: + +1. Load a plugin, apply a known state / parameter set. +2. Feed a known input (audio and/or MIDI, or silence). +3. Render a fixed duration of audio. +4. If no reference exists, **record** one. If a reference exists, **compare** + the freshly rendered output against it and emit a pass/fail verdict. + +Because it is a fundamentally different activity from `validate`, it is built as +a **parallel subsystem** rather than as another `PluginTest`. It reuses the +plugin-loading, lifecycle, and render infrastructure but has its own CLI +command, config schema, and result model. + +### Scope and honest caveats + +- Acceptance testing is only meaningful for plugins that are **deterministic** + given a fixed input + state. A plugin with free-running internal randomness + cannot be golden-tested reliably. The planned playhead / seed controls help, + but cannot make a non-deterministic plugin deterministic. +- The **cross-platform / cross-version portability** goal (matching a reference + produced on a different OS, CPU, or plugin version) is realistic only with + tolerant comparison methods. Exact / per-sample comparison rarely survives + SIMD and platform floating-point differences. This is precisely why the + comparator is a pluggable abstraction (see §5): spectrum, cross-correlation + and fingerprint methods are what make portable references viable. + +## 2. Command-line interface + +A new subcommand sits alongside the existing verbs: + +``` +pluginval test +``` + +- `` is the **positional** acceptance-test definition (see §4). It + is parsed by its own loader. +- If the config's reference file does not exist, it is **created** (record + mode) and the command reports success. +- If the reference exists, the rendered output is **compared** against it and + the command exits `0` (match) or `1` (mismatch), consistent with `validate`. + +### Relationship to `--config` (important) + +Do **not** confuse the acceptance-test config with the existing `--config` +flag. They are unrelated: + +| | `--config file.json` | `pluginval test file.json` | +|---|---|---| +| Purpose | A *settings layer* for a **validate** run | The full **acceptance-test definition** | +| Schema | `PluginvalSettings` (strictness, timeouts, sample-rate lists, …) | Plugin + input + reference + state + comparison | +| Merge behaviour | `merge_patch`ed into the settings layering | Loaded standalone | +| How supplied | `--config` option | Positional argument to the `test` verb | + +The acceptance-test definition is **never** routed through the +`PluginvalSettings` / `--config` layering. + +## 3. Integration with the CLI pipeline + +The CLI was rewritten (PRs #175 and #176) onto **CLI11** + an **nlohmann/json** +settings pipeline with a subcommand dispatcher. Acceptance testing plugs into +that dispatcher; it does **not** touch the validate settings-layering at all. + +Required edits (small and localised): + +| Location | Edit | +|---|---| +| `SettingsParser.h` — `enum class Command` | add `test` | +| `SettingsParser.cpp` — `dispatch()` | recognise the verb `test`, peel it, capture the positional config path(s) | +| `SettingsParser.cpp` — `isCommandLine()` | recognise `test` so CLI mode is triggered | +| `SettingsParser.cpp` — `getFooterText()` | add `test` to the `Commands:` help block | +| `CommandLine.cpp` — `performCommandLine()` | add a `Command::test` branch that runs the acceptance runner and async-quits | + +Because `test` is brand new there are **no deprecated-alias** concerns. + +### Process isolation + +For v1, acceptance tests run **in-process**: record and compare both need the +rendered buffer back in the calling process, so in-process is the natural +choice. Crash isolation can be layered on later by mirroring the validate +child-process handoff (`createChildProcessCommandLine` in `SettingsParser.cpp`, +which passes an authoritative base64-JSON blob via `--config-base64`); the +equivalent would be `test --config-base64 ` with the child writing the +reference / diff to disk and returning an exit code. + +## 4. Config schema + +The config is a plain, std-typed struct that follows the same **nlohmann/json +(de)serialisation pattern** as `PluginvalSettings.h`, plus a `toX()` boundary +conversion to JUCE types. **JSON keys are `snake_case`** (e.g. `sample_rate`). +Because the C++ members stay JUCE-style `camelCase`, the snake_case keys are +mapped explicitly via per-struct `to_json` / `from_json` (rather than the bare +`NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE_WITH_DEFAULT` macro, which would emit the +member names verbatim). Missing keys still fall back to the defaults. + +```jsonc +{ + "name": "myReverb-default", + "plugin": "/path/to/Plugin.vst3", // path or AU identifier string + "input": { "audio": "in.wav", "midi": "in.mid" }, // either / both / omitted -> silence + "reference": "refs/myReverb-default.wav", // optional; default derived from "name" + "state": { "file": "preset.state" }, // OR "parameters": { "Mix": 0.5, "3": 1.0 } + "sample_rate": 48000, + "block_size": 512, + "render_duration": 2.0, // seconds; omitted -> use input length + "comparison": { "sample": 1e-6 }, // omitted -> default 1/32768 (16-bit LSB) + "playhead": { "bpm": 120 }, // future + "automation": [ /* phase 2 */ ] +} +``` + +### Field reference + +| Field | Type | Default | Notes | +|---|---|---|---| +| `name` | string | basename of config file | Used to derive the default reference path and to label results. | +| `plugin` | string | — (required) | Absolute/relative path, or an AU identifier string. Relative paths resolve against the working directory. | +| `input.audio` | string | none | Path to an input audio file (WAV/AIFF/FLAC via `AudioFormatManager`). Used for effects. | +| `input.midi` | string | none | Path to a `.mid` file (`juce::MidiFile`). Used for instruments. | +| `reference` | string | `/.wav` | The golden file. Absent -> record mode; present -> compare mode. | +| `state.file` | string | none | Binary blob from `getStateInformation`, applied via the VST3-safe helper. | +| `state.parameters` | object | none | `name-or-index -> normalised value`. Applied after `state.file` if both are given. | +| `sample_rate` | number | 44100 | Single value (not a list, unlike validate). | +| `block_size` | number | 512 | Single value. | +| `render_duration` | number | input length | Seconds. `num_samples = round(duration * sample_rate)`. Input shorter than the duration is padded with silence; longer is truncated. | +| `comparison` | object | `{ "sample": 0.0000305 }` | Map of comparator name -> its sub-config. See §5. | +| `playhead` | object | none | **Future.** Fixed tempo / time signature for time-dependent plugins. | +| `automation` | array | none | **Phase 2.** Parameter changes scheduled at sample positions. | + +### State precedence + +If both `state.file` and `state.parameters` are present, the binary state is +applied **first** (it represents the full plugin state), then the parameter map +is applied on top as overrides. + +### Multiplexing (future) + +The loader accepts **either a single object or a top-level array** of configs. +v1 may execute only the single / first entry; phase 2 iterates the array to +multiplex many test cases from one file. Designing the parser for arrays now +costs nothing and avoids a schema break later. Each array entry should carry a +`name` so its reference path and result are uniquely identifiable. + +## 5. Comparator abstraction (the core extension point) + +Comparison is **pluggable**. The config selects one or more methods; v1 ships +only `sample`, and additional methods register later with **zero** changes to +config parsing or the runner. + +```cpp +struct ComparisonResult +{ + bool passed = false; + double score = 0.0; // method-specific metric (e.g. max abs diff) + juce::String summary; // human-readable + nlohmann::json details; // structured, for the machine-readable report +}; + +struct Comparator +{ + virtual ~Comparator() = default; + virtual juce::String getName() const = 0; // "sample", "spectrum", ... + virtual ComparisonResult compare (const juce::AudioBuffer& reference, + const juce::AudioBuffer& output, + const nlohmann::json& config) = 0; // value under comparison[name] +}; +``` + +- A small **registry** maps method name -> factory. +- `config` is whatever sits under the comparator's key — a bare number + (`1e-6`) for `sample`, or an object for richer methods + (e.g. `"spectrum": { "tolerance": 0.05, "fftSize": 2048 }`). +- When multiple methods are listed, **all must pass** (logical AND); each is + reported separately. + +### `comparison` defaults + +When `comparison` is omitted, the default is: + +```json +"comparison": { "sample": 0.0000305 } // 1.0 / 32768.0, i.e. one 16-bit LSB +``` + +### Roadmap of comparators + +| Method | Phase | Description | +|---|---|---| +| `sample` | **v1** | Per-sample absolute-difference tolerance (`0` = bit-exact) plus a length check. | +| `peakrms` | future | Peak and RMS difference thresholds. | +| `spectrum` | future | FFT-magnitude comparison with tolerance; robust to small phase/SIMD differences. | +| `crosscorr` | future | Cross-correlation, tolerant to small latency offsets. | +| `fingerprint` | future | Acoustic fingerprint match; most robust for cross-platform/version. | + +The `spectrum` / `crosscorr` / `fingerprint` methods are what make the +**cross-platform-portable** reference goal practical. + +## 6. Reference artifact format + +A reference is **two files**: + +1. **`.wav`** — the rendered output as a **32-bit float WAV** (preserves + full precision; it is the source of truth for comparison). +2. **`.wav.json`** — a sidecar manifest: + +```jsonc +{ + "plugin": { "name": "...", "manufacturer": "...", "version": "...", + "format": "VST3", "uid": "..." }, + "render": { "sample_rate": 48000, "block_size": 512, "num_channels": 2, + "num_samples": 96000, "length_seconds": 2.0 }, + "pluginval_version": "2.0.0", + "config_hash": "…", // hash of the resolved config (excluding the reference path) + "created_on": { "os": "macOS", "arch": "arm64", "date": "2026-…" } // informational only +} +``` + +- `created_on` is **informational only** — it is never used for matching, because + references are meant to be portable and shared. +- `config_hash` lets the runner detect a **stale** reference (one produced from a + different config than the one now being run) and warn. +- The default reference path deliberately contains **no platform/arch**, since + references are intended to be portable and checked into a repo. + +### Diff output on failure + +When a comparison fails, the runner also writes a **diff WAV** (`output − reference`) +next to the result to aid debugging. + +## 7. Module layout + +``` +source/acceptance/ + TestConfig.h/.cpp // std-typed struct + NLOHMANN_DEFINE_TYPE..._WITH_DEFAULT + // + toRenderSpec() boundary conversion (mirrors PluginvalSettings.h) + AcceptanceTest.h/.cpp // resolve + load plugin, apply state, feed input, render -> buffer + ReferenceComparator.h/.cpp // Comparator interface + registry + SampleComparator; record-or-compare + TestReporter.h/.cpp // text + JSON result, exit code +``` + +Reused infrastructure: + +- Plugin loading / scanning: `AudioPluginFormatManager::createPluginInstance` + and `KnownPluginList` (as in `PluginTests.cpp`). +- VST3-safe lifecycle helpers in `TestUtilities.h`: + `callPrepareToPlayOnMessageThreadIfVST3`, + `callSetStateInformationOnMessageThreadIfVST3`, etc. (VST3 requires several + operations on the message thread). +- The render-loop shape from the `AudioProcessingTest` in + `source/tests/BasicTests.cpp` (`prepareToPlay` -> `processBlock` loop with + `AudioBuffer` + `MidiBuffer`). + +Build wiring: add the new `.cpp` files to the `SourceFiles` list in +`CMakeLists.txt`, and add acceptance-mode cases to `source/CommandLineTests.cpp` +following the existing test style. + +## 8. Dogfood test plugin (tone generator) + +To develop and self-test the acceptance feature we need a **device under test +that is fully deterministic** — something whose output is known exactly, doesn't +depend on a third-party binary, and is stable across runs and platforms. We +build a tiny in-repo **tone-generator plugin** for this purpose ("dogfooding" +the feature with our own plugin). + +### What it is + +A minimal JUCE plugin (built with `juce_add_plugin`) that synthesises a simple, +deterministic tone: + +- **Parameters** (so we can exercise state / parameter application): + - `waveform` — choice: `sine`, `square` (extendable to `saw`, `triangle`). + - `frequency` — Hz (e.g. 20–20000, default 440). + - `gain` — linear or dB (default −6 dB). +- **Determinism**: the oscillator **phase resets to 0 on `prepareToPlay`** and + the waveform is computed in closed form from the sample index, so the same + config always renders the identical buffer (ideal for the `sample` + comparator, including bit-exact). No randomness, no denormal-sensitive + feedback paths. +- **No audio input required**: it is a generator, so acceptance configs for it + use silence input (or omit `input`) and a fixed `render_duration`. +- Produces a VST3 on all platforms (and an AU on macOS) so the same plugin + exercises both formats. + +### Where it lives + +A new CMake target alongside the existing `tests/test_plugins/`, e.g. +`pluginval_tone_generator`, built on demand (guarded behind an option such as +`PLUGINVAL_BUILD_TEST_PLUGINS`, off for normal release builds). Suggested +layout: + +``` +tests/test_plugins/tone_generator/ + CMakeLists.txt // juce_add_plugin target + ToneGeneratorPlugin.h/.cpp +``` + +### How it self-tests the acceptance feature + +Check in a handful of acceptance configs plus their recorded reference WAVs and +wire them into CTest, so CI both validates the tone generator's stability and +exercises the full record/compare path: + +``` +tests/acceptance/ + sine-440.json.in // tone gen, waveform=sine, state.parameters + square-220.json.in // tone gen, waveform=square, state.parameters, bit-exact + square-state.json.in // tone gen, state.file blob + a gain parameter override + gain-half.json.in // gain effect, input.audio = a full-height sine, bit-exact + inputs/sine-full.wav // checked-in input for gain-half (±1.0 sine) + refs/.wav (+ .wav.json) // checked-in references + sidecar manifests + refs/square-state.state // checked-in getStateInformation blob +``` + +The configs are checked-in **templates** (`*.json.in`): the plugin artefact +paths and the input/reference directories are substituted at CMake configure +time (the tests can't hardcode a build-tree plugin path). Each `pluginval test +` CTest case asserts exit code `0` against the checked-in reference. +Because the dogfood plugins are closed-form deterministic, these references are +stable enough to commit — a first smoke test of cross-platform portability for +the `sample` comparator and the baseline for future comparators (`spectrum`, +`crosscorr`, …). + +The four cases cover the distinct render paths: + +- **`sine-440` / `square-220`** — the `state.parameters` (name/index → normalised + value) path. `square-220` compares bit-exact (`"sample": 0`). +- **`square-state`** — the binary `state.file` (`setStateInformation`) path, + plus a `state.parameters` `gain` override on top, exercising the + state-then-parameters precedence of §4. The blob is captured once and checked + in alongside its reference WAV. +- **`gain-half`** — the **`input.audio`** effect path: a second dogfood plugin + (`tests/test_plugins/gain/`, a deterministic gain effect) gains a checked-in + full-height (±1.0) sine by 0.5 and is compared bit-exact (0.5 is exact in + float). It also omits `render_duration`, so the render length is derived from + the input file. + +## 9. Execution flow + +1. Parse `pluginval test ` into one or more `TestConfig`. +2. Resolve the plugin (path or ID); create the instance at the config's + `sample_rate` / `block_size`. +3. Apply `state.file` then `state.parameters`. +4. Load `input.audio` and/or `input.midi`; otherwise use silence. +5. `prepareToPlay`, render `render_duration` worth of blocks, accumulating the + output into a single buffer. +6. **No reference exists** -> write the float WAV + manifest (record mode); + report "reference created". +7. **Reference exists** -> run each configured comparator; report each verdict + and the overall pass/fail; on failure write a diff WAV; exit `0` / `1`. + +## 10. Phasing + +**Phase 1 (initial implementation)** + +- `pluginval test ` subcommand wired into `settings_parser`. +- Single-config (object) execution; parser already accepts arrays. +- Plugin load + state (file and/or parameter map) + file-based audio/MIDI input + (or silence). +- Fixed-duration render in-process. +- Record-or-compare with float-WAV + JSON-sidecar references. +- `sample` comparator only (default tolerance = one 16-bit LSB). +- Text + JSON result reporting; diff WAV on failure. +- **Dogfood tone-generator plugin** (§8) plus a CTest self-test that runs the + full record/compare path against checked-in references. + +**Phase 2 and beyond** + +- Multiplexed execution of config arrays. +- Parameter `automation` timelines. +- Fixed `playhead` (tempo / time signature) for time-dependent plugins. +- Additional comparators: `peakrms`, `spectrum`, `crosscorr`, `fingerprint`. +- Synthesised inputs (`input.generator`: noise / sine, with seed). +- Optional child-process isolation mirroring the validate handoff. diff --git a/tests/acceptance/CMakeLists.txt b/tests/acceptance/CMakeLists.txt new file mode 100644 index 00000000..cfb76076 --- /dev/null +++ b/tests/acceptance/CMakeLists.txt @@ -0,0 +1,36 @@ +# Acceptance self-tests: drive the dogfood plugins through the full record/compare +# path against checked-in reference WAVs. +# +# The configs are checked-in templates (*.json.in). At configure time the dogfood +# plugin artefact paths, the input directory and the (source-tree) reference +# directory are substituted in. An artefact path may contain a generator +# expression (e.g. $ on multi-config generators), so we resolve it via +# file(GENERATE). + +get_target_property(tone_artefact pluginval_tone_generator_VST3 JUCE_PLUGIN_ARTEFACT_FILE) +get_target_property(gain_artefact pluginval_gain_VST3 JUCE_PLUGIN_ARTEFACT_FILE) + +set(ACCEPTANCE_REF_DIR "${CMAKE_CURRENT_SOURCE_DIR}/refs") +set(ACCEPTANCE_INPUT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/inputs") +set(TONE_GENERATOR_PLUGIN "${tone_artefact}") +set(GAIN_PLUGIN "${gain_artefact}") + +set(acceptance_cases + sine-440 # state.parameters path (waveform/frequency/gain by name) + square-220 # state.parameters path, bit-exact square wave + square-state # state.file (setStateInformation) + parameter override precedence + gain-half) # input.audio effect path: gain a full-height sine, length from input + +foreach(case ${acceptance_cases}) + # Step 1: @-substitute the reference dir (and the possibly-genex plugin path). + configure_file("${CMAKE_CURRENT_SOURCE_DIR}/${case}.json.in" + "${CMAKE_CURRENT_BINARY_DIR}/${case}.json.gen" @ONLY) + + # Step 2: resolve any generator expression left in the plugin path. The output + # path is per-config so multi-config generators don't collide on one file. + file(GENERATE OUTPUT "${CMAKE_CURRENT_BINARY_DIR}/$/${case}.json" + INPUT "${CMAKE_CURRENT_BINARY_DIR}/${case}.json.gen") + + add_test(NAME "pluginval.acceptance.${case}" + COMMAND pluginval test "${CMAKE_CURRENT_BINARY_DIR}/$/${case}.json") +endforeach() diff --git a/tests/acceptance/gain-half.json.in b/tests/acceptance/gain-half.json.in new file mode 100644 index 00000000..1fa0970f --- /dev/null +++ b/tests/acceptance/gain-half.json.in @@ -0,0 +1,16 @@ +{ + "name": "gain-half", + "plugin": "@GAIN_PLUGIN@", + "input": { "audio": "@ACCEPTANCE_INPUT_DIR@/sine-full.wav" }, + "reference": "@ACCEPTANCE_REF_DIR@/gain-half.wav", + "state": { + "parameters": { + "gain": 0.5 + } + }, + "sample_rate": 48000, + "block_size": 512, + "comparison": { + "sample": 0.0 + } +} diff --git a/tests/acceptance/inputs/sine-full.wav b/tests/acceptance/inputs/sine-full.wav new file mode 100644 index 0000000000000000000000000000000000000000..6cf57322d57626238ad44da8143b4432e88f20bb GIT binary patch literal 96104 zcmeI*``2c5dEfCtLsS$`iHTw{R3M<%69E*MXMYhR2JnP}$BIf(M-Y{R48y<-StgA} zl6q=1^$5fL0isx*@Z9Ll0V=7o$$GHHcF2=>;JYMd*8=Rr7wQ@RjrH8e(O}a z-)leGdet{SHhD|Ef8-ZaY5TeN9;HYB@vpa@_?Ux7>C;bsOzWwiec~wn_=6|3 z&ivjfqx7+zr?uYsf`2(m=Un=)S_gjPRipIe9j|RY_@oO*>5V5{+mC`n|v2JxaG-yQ_8R*Df8U5B|m_tuMXxqEY(55$Cu5*S_bD(wFb|%GRFK zUph)ZyYcz0Cw$~tqjd7;pW51-rcwIn1CDH6cgiD1>HfDJ*gENe2aeMA>-TLv`bRfU zrMtcDtF13xa@|z=v(H@Fy6FwSH)i zhc`=S|IYnarN4N^$*s5iFymjz_zcrV(KYydO-z_K3rB8h8nXPXg`uw@{rjNa(b@I+v%%wBl@aonF zu0L-sU47(5Yu&J8zB+oxo=aQ5c-OAEG(NSvwe9D-=hDWuwT}DQWpn9WpWN9x;XN16 zrN6!U!qz8me(hZP`nI!M`#j{VxwPvZFK(T7?CEpq;eYkC*1>Ol^4z-SnEBvn=k(at zy^lU*E*-Jw!L94BxX)b5{n0vP|8K8KKY8vKT93T-W2@3nfA;p))9>@zRp~EJKdSYN z{chVVo%*iJTNk|T!`GIMJ>$6>r8BSlX6xL~K5{Bu{qD0`|M=BEo=Vq!_d~7ow*B=~ z`o#FV*5>nmHkBU#^!-NZ!8?Afb^M_ZAEnh(AKg0Wt4|oE*L?m-ts@_D$|(KOwa;lC z@}U=v(#AK=Z2gB*Uo}bx-SL{%pZ(nhqx67(+TOb3sGXy9)xW>2b^B4fN9oF&*ZTSY z*gZrGF47b1>F(EjsdeyCKUkH{+kW@Cbj`Q!*ZTZdA3B%be&S)R z=N)#$T)OdfTU%!xaN=CreCRV<$DjMWx%7>DyrgyTKfHV{9dpMyty`aU-dy_P4Xc?N)+VSJl=hFGV|Mb=m&-(4T^cU|vwzba_A2*jiwe`@}A)k22 zTzcS#_iugr@O|dedFOq1ReH#O{hQXkF1}_}y6}oWX&rdNg{#sn?>@HmqUUW{mHyo+ zZ{93@?X8cxu5`_hj^8Nl`zv2*?bvzfRJ!@4FKM0m+gD7bqhGeS^`IYqW-7hsE8l3{ zvgP)v^rJ`JYn1-!%m=j&dFVeMr7KSU7p<$mbMz>^@q5R&{`9FQkJ1lse|GDz+g~_J z+ov;IAOFm@Q9AVvuW6lk@dcxF2 zr{3fEQM&cEqgsbP?NOujo%quhwH`l<9c#^x!zoVu1D9W>(%w^ zdUk!g-d+Fh2lt2j#r@-ca(}ts+<)#z_ow^S{p)^qf4kq^|LTGIpkAmS>WTWI-l#w7 zk@}=wsbA`u`ljBgf9j$7s9vg{>Z$sw-m1UqvHGlDtKaIm`mWxq|N4RcpkL@8`icIc z-{?R3k^ZD#>0kPp{-)pQfBK>Rs9)-z`l)-mh{;uEa|K-;-E&)@U={68Lm58ws(0iJ*_;0^c#9)VBb75D|7fp6d)_y-<> zkKiTv37&$l;4SzI9)r)|HTVslgYV!y_zxb058*}l5uSuE;Z67x9)(ZgRgbypVON#% z&zq(E`&RzrCiUgT;How zuJ=~1|5YjXLo4^ks+9YsmHTH^%Kfxi%Kg>K{kFA~`){L^`*AAe{%qxbol3cXTe+X7 zQtt0o?)Ryb`+t;D4_c`Yqm+8lO8pq6)RR`~%P6JZv{HXYDfOt8`ZP+ZSFO~qQ3}6W z>xLa8^=+iywNn2^DfO_G`Z!9dm#x&#QA#~+rM`|*>TN6aca&0(TdB{ZlzQDt{T`*% z^H$i^XkBy6NWCAa|E=_cQA&SkrC*Fv`bR7MWGbb;QuEEsN^SPA%-b%lpOX>fu%mZ^N^Fb@~!d%My(8@e9moi_pGH=YK%pa}HBXcS9Nh|Zp zT*~~?$~+Sua?zamrj>bTE@l2{WgePKnU7kTm*!IDr&i{vxs>^;m3eC}W&Ubq9-B*< z&sv$+R;A2ut;}<)Qs%o>=Dk%Z^It3T;Hs4QaI=(ov6cC8Ybo>OMk({B%KSN% zGLN=0pH8LBtF6qhQz`T8C}qBFW!@d7%)hP7!=se>xRrT%lrlfJGEa|E=Id7G?NQ47 z-O4;ZN}11Fnb$`t^Ls1v{3vCQ1KWN1-gg0J1;vZV^6QdM=(Td*~ zrTC9l{KzQ9pS0pvMk)TK6+bgd@i(pbol%PaX~hqXQvA_WieGBQKTW0hsaE{eREpnf z#eYqu__2*r{Mpu0{Mu$I{;d^1w<^Wowc__yrTD*A{NSn-f7ps&T$SP<=TiJ+EB>=y$7U(~qZJ;qwG=+GQ3@}aO5rE1@RX?( zzS0VBnM&a=t?-zs6h1Rb;We%Bn^6kSX@&2MQg}}*{AZNHgIeK3qZD4$3O^d9@T6Aw z(kO*DwZfl9DLkqbJ~c|=SFNzC^pDH$`e*;%|K5M^&-3T|`~3ZW9zUO-*U#_w@%#CG z{r)}=pO4SW=jZeE`TD$l{=N_2kMGO(=lk^i`o4Ytt_RnL>&5lsdUAca-dumKN7tw8 z)%ELoc740vUH|R}_lNt%{o{Uef4Se>f9^;3r~B3Y>wb2ByWida>Vf*8UZ@}HiTa}6 zs6XnF`lMc|U+S6qrrxQ4>Y@6mUaFt!srstks=w;7`mA27-|D&guHLKv`hotSU+5qD ziTc9H2{;Xf?-}<@!uHWnb<^l78 zdBOZ(o-kjSH_RXA5%Y<8#r$HPG2fVX%s=KK^O1SU{A8XoUzxYeU*<9MnR(6pW}Y+O znfJ_p=0WqJdC~l6o-|*YH_e~sQS+&J)%Y+ievN7A3w++@{9Zro`5gl4fq2dfluHS_ywMUZ{QvH z2Offt;3fD8o`SF7E%*x_gU{eK_zj+e@8CW74<3XM;YIino`f&qP52WYg-_vCkGbi6 zhn4Uwd<*Zwzwj`83@^ja@HBi4Z^Pg4ID8JT!|(7sd=Kx#|L{P35HG|J@kD$PZ^R$* zNPH5n#4qtod=u})Kk-m}6feb3@l<>jZ^d8nSbP?*#c%Okd>8Mik^zwvN<952VuQ;(MX)snwj@>fg#YRO+M`Ku*=wdAilZ@1*H z*7?gNf3?ozF8Ql-;-E&)@U={68Lm58ws(0iJ*_;0^c#9)VBb75D|7fp6d) z_y-<>kKiTv37&$l;4SzI9)r)|HTVslgYV!y_zxb058*}l5uSuE;Z67x9)(ZgReO%u zzPE&D;ahkY{)LC(V|W>UhNt0ccpLtP$Ki8$9e#)B;d^)={)Y$RgLol+h$rHUcq9IZ zN8*!sC4Px#;+uFU{)va;qj)KPil^eMcq{&j$KtbiEq;sV;=6b+{)-3W!+0@%j3?vE zcr*TtN8{6YHGYj}llr_VxDn_WAbv z_Wkz%@&NJy@&fV$@&xh)@&@t;@(A(?@(S_`@(l6~@(%J3@(}V7@)GhB@)YtF@)q(J z@)+_N@*46R@*MIV@*eUZ@*wgd@*?sh@+9&l@+R^p@+k5t@+$Hx@+|T#@-Ff(@-Xr- z@-p%>@-*@_@;34}@;LH2@;dT6@;vfA@;>rE@<8%I@23s@>KFwOa5w|xAXnG9$X);7uS#L$@S%WbN#sWg}#{-{Ul zlX|6osb}h&dZ+%Whw7txseY=b>Z^LI{;J37vwE$5tLN(blD}H#t(W}OI)A_9uh!3F zEcvVT^Evz?|Hx1Bm;5IG$&d1<{3`#-&+@nYF8|99^T+%$|IAPG*ZemB&5!fv{5t>6 z&-3^EKL3vg-~)I8et;+73wQ(mfJfjHcm;leXW$!n2mXPF;3IemeuAgqD|ie3g2&)9 zcnyAo=iobd5B`G(;X`;4euO9COL!Chgh%01cvYSqJiLTw;ahkY{)LC(V|W>UhNt0c zcpLtP$Ki8$9e#)B;d^)={)Y$RgLol+h$rHUcq9IZN8*!sC4Px#;+uFU{)va;qj)KP zil^eMcq{&j$KtbiEq;sV;=6b+{)-3W!+0@%j3?vEcr*TtN8{6YHGYj}U*!y}%&*J&S2+VKer7rUDraED-!A80E$3g|bzWBZ zSD&A?oPV`GPc8ha&sPh->hspZuloG8@T)$LE&Qs_XA8gT^V-6%`uw)=t3J;y{Ho7) z3%}~~-omf?{I~F{J`XPZs?UcDzv}bi!ms-LxbUk!PcHnb&zB3o>NDrUuloGC@T)$L zF8r#`rwhLtrSPj(_*I{0=l%}A>htcxuloGEC4aT#uj&k}<@~Gl`Kaba^JC4DrRDsq z_4&HX`B&@nmibS9lt1NH`B#3HzvXxNUw)WB=9l?rewx4LxA|{=oImH+`FDPvzvuV) ze>?ymzzgsLJON+88}J7_0-wMu@C!Tx-@rTY4?F}P!AtNHJOy9DTksb=2A{!e@Ebe_ z-@$wEA3O*j!i(@DJPBXIoA4(*3ZKHOE_lu#T~)%f@GZOx|H8xYF}w^v!_)9JybXWD zaGrwm6`Z%=`~~MRIG@3J4bE?Ho`drpocG}T z2j@XJAHsPN&W~`Og!3hwH{tvV=TSJH!g&?WuW+7)^DUfrVgKqp4CiAwFT?p6&eL$d zhVwR@zu`O%=W{r(!}%S~^Kib0^FEya;XDxMgE%k5`612|alVN2Mw~z5JQC-VIIqO{ zCC)Q(zKQcroPS~;Z$EEeZ+~x}Z@+KfZ~re3ARizvAU_~aAYUMFAb%i_AfF(wAip5b zAm1SGApal_As-SeIA^#x{A|E0zB0nNeB3~kJ zB7Y)}BA+6!BEKTfBHtqKBL5-}BOfC#BR?ZgBVQwLBY(5xuh!>b$!E!H$#2PX$#=w0#5yWU;@?g#gW`^Ej^esX`g-`s!hNB5`u)&1*! zc7MCy-T&%=`k-E@AL@zvqTZ-K>XG`SUa4Q|nfj*QsekIB`lw#2pX#als@|%<>aqH) zUaQ~gx%#f&tN;3e{-9szANq;@qTlF0`jP&mU+G`^nf|8V>3{m6{-|H-pZcl(s^99r z`mz43U+drcx&E%->;L8f^MQH6{9v9iUzj({ALbGBiFw8RVxBSIn0L%S<{|TudCB}_ zo-$vVx6EJWG4q*u&HQGbGvAr_%zx&=<@~F)KUmJcTA!E9Kk}3OCBMmk@}vAIzskSz zv-~Z;%m4Dj{4u}GKl9W4HNVY&^W*$Ezs|q&^ZY%(&;R2A_yAsjAK(f20^Wc>;1T!) zUV&fW8Tba?fq&p3_y}HtpWrF@3f_Xh;4%0NUW4D@Irt9Vga6<`_z+%%AK^*(65fPA z;ZgV$UbW|lA0J-Av+ymv3;)8y@G-m$Kf}}THM|Xf!{hKdybiy^^YA^q5C6jh@j<*0 zKg1L9MZ6Jz#3S)Zyb`~}Gx1Hl6aU0R@lm`KKgCn=RlF5{#bfbVycWO3bMalg7yreB z@nO6eKgN^sWxN@G#-s6Ryc)m8v+-@b8~zy$6WA0G%`w+vg`u&LESN*=k@T-1* zV)#|RPci(e->(>c)$dyjzv}ldhF|sj7{jmn{fyyP{l3QVtA2lD_*K8pG5o6E?-+j7 z?|Tft>i0i}U-kPS!>{`Nkl|PTzR2*aet%^6)kZ1&s^2deezlx`l?;~qf5~4h`Ku*= zb=h6#qna1ZkLF49WzCwU<^6-}_n$BCA3S_6Ps*S2tNbfJ%ir?5{4YPuAM?xnGe6B= z^V|G4KhB@?>-;-E&)@U={68Lm58ws(0iJ*_;0^c#9)VBb75D|7fp6d)_y-<>kKiTv z37&$l;4SzI9)r)|HTVslgYV!y_zxb058*}l5uSuE;Z67x9)(ZgRom|Wg}o&_3*W-K z@Gm?JAH&P=GdvAn!`tvTJPx13>+m}~58uQ4@IO2dAH)msLp%{*#2fKPJQAP8EAdM_ z6W_!;@lQMyAH_@YQ#=)4#ar=LJQknDYw=q=7vIHu@n1X`AI6LEV>}sO#+&hHJQ|-0{pQR22iNbr=Rf&T{*+(kU-?=7 zmfz)n`C;<@-P-i!a@ z!T2ybOL@QUMk(*xZRP#DQz`G`ZRP#EQz`H3ZRP#FQz`HBZRP#G zQz`HJ9i_bgx0Uw+k5b+b+{*icM=9?QZsmQ#qm=gxxAMN>QOf&=TX`SxDCPact-P;z zl$v5SBz0N#U_NkQT`C56tZ!S5n%K24!9^88>c&kxR}JU`gV^Mi9K&kwfp{NP;5^MkECKRB22{9r5356-1LKiJCigL5g* z54Q6B;9Sb{gRMM2IG6JLU@OlL&ZRs**vj*Rb1Bacw(|VoT*~ux5{UTvk8>R58R@l{XrLe2bQutLX>}pjCziNeDtxDlnt+1h@4O4ydvipInT)XM$S8uzi}Rt^O3?k zHcQS=a-Nd&m7KTa{3YixIiJaSP0nv}o|E&PocHAXC+9&qAIf=A&X01Ql=G#WHx+&~ zO3tHlK9%#ToL}WUE9YA+?;l*=KUnv?`(Hg!AJhx=Lp@Pn)Eo6jJyM_4EA>k~Q{U7( z^-n!iAJt3sQ$1B*)m!yfJyxI9YxP?_SKrlp^k=!Ef*!d*ZQKBJV+tCi1hl=6AD^7)QZKJQjO|53{K z(aQHTO8LH8`Tj;J-)Ae|?{`X_?uSz&RmNBX~hrC zrTC*&DSoLH|FkN_PqpH&R;Bo@R{Ynh6hF3Eia)!q6u-7nihpax&rPNHyH@<(REqy= z#Sc!U_`_EG;#7)%9HsclR{Z5C#c#IaKSwEkv=x6kN=yFg&!;e~Dg3G>k0YNWuOq)B z&m-R>?<4;s4bp`NHO>W%uN9;r|2mHMTgsc-6?`llYMkLsoRsh+B@>aF^# z9;?slwfe1|tMBT)`mZ185Bi1vp`Yk4`i=ghAL&o}mHwrl>2La-{-+=6kNTzlsh{ev z`mO$}AM4Nhwf?Q2>+kx#{%;;IAD9=+59SH;Ma>$eoZo(3Dd)FulyZK1E9bXQrJUd1 z%K7b6Dd)Gha(??%%K7cBoZmi`a(??L<^1+m&Tk*3oZsHc`R${W^V?fFzkQT)etRqD zw~tcJZ*S%N_EF0D?X8^OK1w;iy_NIZM=9sGx8(EX_2u{F`Q`h~oAUpj2k?AAcCG!q zfaeE1PvH3i&l`CD!1D;6Pw>2g=NCNB;Q0p6J9z%V^AMho@VtcQCp=H#`3lclc>cok z7)$=@6?5M4HRrsUR?eT9OWtSi{r1jV=080T zd;+h)FYpX}1Mk2;@DO|iFTqdn6nq75!C&wgd-O#T@AmQb^Y-=j_xAbr`}Y0z|MCFx0rCR!1M&p&1@Z>+2l5E= z3Gxc^3-S!|4e}215AqQ4mvc#8LViM?LcT)YLjGdOU#;^vOa5xtsEn4pmi(4HmwcDJ zm;9GJn0%PLnEaSLnS7bNnf#eNntYnPn*5qPn|zzRoBW$RoP3*SE8XwzfAv6pP%qRE^+bJ9Z`2?4NPSYT)Gzf+eN*q$KlM<3R4>&}^;CUT zZ`EJ*SbbKn)o=A&eOK?*fBisz&@c24{X~D!Z}cDiNPp6=^e_EPf79>uKmAaD)Gzf< z{ZxO|Z}ngOSbx^9^>6)Lf7kEzfAfI(z`S67Sl&OloPSjrtR;VS;wX>lc^}XJcpk{} zL7o@#{E+8~JYVE_BhMds9?A1bo>%hxlINK`-{g5G&p&w{%JWg4m-76S=cznj<#{X5 zUwIzO^I4wP^8A+Pxjf&6H{cI=1U`XR;1_rXzJYh(A9x5pf|uYYcnZFPx8N^$3_gR` z;5T>H{;KEG(L@2ytOXb}oH=+u5ys9&*-P+I5c?w@y3u^ttr#zj|8h z;5R;bZe4TCd~mdLdTi_7M;|hmj@a|ymgmhqf9`p7&!?x?^ch&5U-vw_{fd2y{fm8! z{fvE${f&K&{f>Q){f~W+{g8c;{gHi={gQo?{gZu^{gi!`{gr)|{g!=~{g-{1{g{23 z{h585{hEE7{hNK9{hWQB{hfWD{hocF{hxiH{h)oJ{h@uL{i1!N{iA)P{iJ=R{iS`T z{ic1V{il7X{iuDZ{i%Jb{i=Pd{i}Vf{j7bh{jGhj{jPnl{jYtn{jhzp{jq(r{jz{k?s@{l0y_{l7eb ze1N=w{D3@xe1W`y{DC}ze1g1!{DM4#e1p7${DVA%e1yD&{DeG(e1*J){6%FkO3V8P z*YBrW@>lEgvgEhqx#YX#z2v{-!Q{i_#pK82$>huA&E(JI(d5(Q)#TUY+2q^g-Q?fo z;pF4w<>cplpYnC`cJg^@{ZN0@FZECTRDacP z^@B;h* zPrw)O2K)h!z$fqu`~uIwH}DSp0}sJR@Dlt4Pr+C47W@T|!DsLq{07g#ckmwk2M@xB z@FM&OPr{e*Cj1GH!l&@6uiSFmVI@2Z-@?1_FFXt%!^`k9JPlvN+weC$4xhv8@H;#Y z-^2UxKRgg0#0&95JP}{S8}Uax5}(8?@k=}t-^4rdPdpSK#Y^#1JQZKXTk%&s7N5my z@mo9>-^F|JUpyEe#*6V|JQ-icoAGBn8lT3i@oPLA-^RP~Z#;a-UoH8oC4aSFXRnt0 z)mg*-+&C%D)pGvTL95DOE%~c6=epWTWI-l#w7k@}=wsbA`u`ljBgf9j$7s9vg{>Z$sw-m1UqvHGlDtKaIm z`mWxq|N4RcpkL@8Oa5w|=MKM`O5s-#Wq0^Hb&dYql(fU$w%o`ux>8i?!r@*6^!7uhsdjb(U+%`L1== zYsvYqbrx*N`LK0XY{~htb(UiHuKYj~eMf5~t1pZq9)%CGXT{49UV@AAL=Fn`Q1^UwS=f6Z_6 z-~2d#&ad;<@-P-i!a@!T2yzc2Z#<^6-Tle3?*ud~0i&$HjN@3a52540b&FSI|j zPqbgOZ?u23kF=k(ue86k&$QpP@3jB4549h)FSS3lPqkmQZ?%85kF}q*ueHCm&$ZvR z@3sH654In+FSb9nPqtsSZ?=E7kG7w-ueQIo&$i#T@3#N854Rt;FSkFpPq$yUZ?}K9 zkGG$e2Kh?{E0k@e2Tn^ z{E9q_e2ct`{EIw{e2lz|{ER$}e2u(~{Ea-0e2%=1{Ej@2e2=`3{Es}4e2~15{E$46 ze3877{E2~x!>G>?nn2h`_=vHes+Jm-`)S} zf%>3cs2}Qy`l8;bKkAYCq+Y3C>Y4hc-l>1;q57y^s-Nnq`l{Zlzv{92tX`|%>bd%^ z-mCxmf&QRh=pXut{-WRLKl+jWq+jV@`kDTw-|2t)q5h~}>Yw_l{;J>VzxuKMtY7Qj z`nmqD-|PS80rP=*!TeyJFkhHA%pc|v^ND%I{9>Ll-;1T!)UV&fW8Tba? zfq&p3_y}HtpWrF@3f_Xh;4%0NUW4D@Irt9Vga6<`_z+%%AK^*(65fPA;ZgV$UUkYH zM}Dw`XW?6T7ygBZ;bV9ieuk&vYj_*}hR5M^cpZL+=iz&JAO432;)8f0euyXHi+Cgc zh)3d+cqM*`XX2Z9C;o|t;-h#eeu}5!t9UE^ipS!!crAX5=i<9~FaC=MuQ7iRnlv1x+;a8&+ezn#OJ4W)QBYD$S z@~5MeJZdZX)KN-awUzwpC?(I@O1^cJl6P$-|2j&^!?u!-9i`-DTglIkQu4H|JUcjA z*BmpFw;jpfwvxvkrQ~y4$?J|%^1ChX1Mz&i_l4M>*r(X9*tgif*vHt<*w@(K*yq^q z*!S4~*az7U*%#R#*(cdA**Dog*+~B0*@xMW*_YX$*{9jB*|*uh z*~i(>+1J_M+2`5s+4tH1*$3JW+85d%+9%pC+Be!i+DFolIN1|lJ}DTk_VFylNXa8lP8le zlQ)w;lSh+JlUI{plV_7}lXsJUTh6~)=ON`IOC{1X4fPw`j$ z7XQVM@n`%R|HjXiw`<+$_xL}4kU!)X`A2?|zvMUhPkxj?i0?D>v;Q;zq)+0b};E@*Ot=XHcFnCPyd@r zo~KWLoJyXzPyd`sp2tsrol5DqqvZMh^y5+Te1H1&D0%)r{d|<150HL8O3n{(o`CZO zGB1ph^9P(qkojVioLAudg3KeMq$XCc)$Y01~$Y;oF$ZyDV$alzl$bZO#$cM;_$dAaA$d|~Q$e+lg$fwAw z$gjw=$hXM5$iK+L$j8Xb$j`{r$k)i*$lu80$mcBitL6Nw<@~GT``x8gL(~)XMZHmf z)Fbsty;8r_GxbfqQ~%UM^-;Z4Kh;z9RlQYz)noNpy;i@~bM;-lSO4_`{XxIbKlBs* zMZeL1^dtRAztX?-GyP4!)Bp5C{ZYTvKlM}nRln7L^<({6zt+F?bNyYv*Z<7}<^%JB z`N2G4zA$f?Kg=WM6Z4As#XMuaG4Ggv%tPiQ^OE_=JY~KzZ<)W$W9BpSn)%HD z!B_AW`~{D}XYd;Q2G7BF@E-gJ55kA=BK!zX!k6$S{0Wc3r|_x=eEpsuEa6%B7T$$_ z;bHg~UWT9HY4{r6hQHx)_#9q`-{E=q9^Qxl;eq%dUWgy!iTEPkh(F?y_#|G5U*ehg zCfG(R{j=wMYtM&8n_A~Z1_BZx9_B-}H_CNMP_CxkX_DA+f z_Dl9n_D}Xv_EYv%_E+{<_FMK{_Fwj4_G9*C_Gk8K_G|WS_HXua_H*`i_ILJq_Ivhy z_J8()_Jj6?_J{U~_KWt7_K)_F_LKIN_LugV_M7&d_Mi5l_M`Tt_NVr#_N(@-_OJG_ z_Otf2_P6%A_Ph4I_P_SQ_QUqY_Q&?g_RIFo_Rsdw_S5#&_Sg2=_S^Q|_TTp5_T#lH zFU5bf;zvd){-hPZGD`6;t@xQyioa>a?~GFXPb+?Cl;V%3Qv6aY{%I=3PqpH&rc(S? zEB2;?K60;@37y@o%m8xm79tt`)zxD#ib`;s;ly_`_EG;;Iz?IG5rlTk)66 z`v*_?)j9mC6@IlUg+;W&uU4h-t5*2cs^mPRIvc4JezjTh{yOio^L{&dANe17Ao(D9 zA^9PBBKabDBl#nFB>5zHCHWg}1iCU*}SIY%6?rE``^&!f)qNcy23v zcP@qZw!(ksQh0DHe0VN}7q`NX=TdlbYsp`&>$7^Teyiu|yLzww>j(OSexZNpC;E$i zqyOkf`jdX8f9YrXn|`PN>4*BGeyM-zr~0dYtN-f9`m=tmf9vP^yMC|#n+MDX<^}VE zdBS{Q-Y|cdN6aVY74wUE#(ZPmG5?r{%tz)W^OJeXd}ZD;f0@V3XXZ8Yn|aQBXWlda znFq~>=0)?PdD47o-ZX!jN6n|^Rr9NP)_iN;HUFB2&Bx|t^Rs!{d~Mz~f1Ah6=jL_u zyLsMxzvQph&)e{S{2+hGFY=H4B!9_o@}K-Df6A}&uly{3%kT2P{4jsaFZ0j*G=I%+ z^WXe9f6lM-@BBP}&+qg9cmO_t7vKkY0=|GZ;175NK7m)@7kCD~fp_2^cnCg%m*6LO z3ciB3;4gR#K7-fbH+T-dgZJP+co05>7vV>E6262t;ZJxJK806(=bis>cnQzKx9~3f z3lGD`@G|@iPs7*nHvA2b!{_ii{0`5<_wYXa4-do#@k0C%PsA7TM*IMJNrEQJ^McUKl?!YLHk1c zL;FPgMf*nkNBca=w=Hww%A^JTB*RIj_t4 zUC#4zzL)d9od4xKFz16gFUW_M)KB-sgmwKkYsdwt1dZ<3Cm+Gf_s=lhX z>aTjNKC9R2w|cI=tM}@^exN_-7y5^OqQB@j`j393Kj~Ncmwu+d>38~{eyBg{m-?rE zs=w;D`mcVhKkL`}w|=g_>-YMH}d?E=aD?0k=!Ef*!dU zhNt0ccpLtP$Ki8$9e#)B;d^)={)Y$RgLol+h$rHUcq9IZN8*!sC4Px#;+uFU{)va; zqj)KPil^eMcq{&j$KtbiEq;sV;=6b+{)-3W!+0@%j3?vEcr*TtN8{6YHGYj}L=j`k3@9gvJ_w4)Z|LgH1aj_Hu5*}IPy92I`TX6Jn}vAKJq{EK=MKILh?iMMDj)QM)F7UNb*VYO7ctc zO!7_gPV!IkQ1VgoQu0&sRPt5wR!jbBoxgKExISDjt{>Nv>&x}#`g1+HK3%V_U)Qti z+x71HcR#p4+%N7Q_mlg}{pS92KQ8&JC4W^tTEEu6^>h7Qzt{iG1Lgzsg89KbVZJbL zm_N)T<`eUZ`NceAzA^8Zf6PPXBlD8^$vkDgGH;o`%wy&=^P2h1JZHXJ@>k3I2g~#E zfBYbS$S?Ab{3L(LZ}Ok~D1XYY@~`|Xf6MRkzx*(N%rEoL{4{^fZ}Z>$IDgKs^Y8pT zf6wpp|9AjCfEVBgcmlqFH{cI=1U`XR;1_rXzJYh(A9x5pf|uYYcnZFPx8N^$3_gR` h;5T>V9ufTZkf#CQ;*4tQw(tM}h< zzYTA^B>&HK57_W8Hyp6xl`lW#C{(0 zY?L1In~!h(w;O(Ol%9J2e{Nm+vHw0wr=9fH*5m)+J)`ssPxxT#Gtb&OO2=OIm#w=# zeCa5?;+8MAEqAfZ(I|cG0Y7M6`H-7N zY4ZzjXzjZ0E2DJe$NsVPoXh`al-9>=UGT6^j?%W{KHPffo#&3yv-f{T>$FSH9Hqzq z=;YS@-tpp5`ts{Hw*K@B2aM9KZ~Cd$>E~auR=WJ%f7Cki!AGo>&e`(Djim>j{Gx58 z=lrL`)=RJc#kaRU{;JE@OK<=FKC|@tBOlXx*zTv!(xs1mVe5aq>o;fV#B+bIwPC|K zv-J9#-rt(f-ZD!EKKc`_(|&b*mL9bIuUi-2c*QK;aPU`JZ@%ywv-F%ZzSBD4P2ZoT z4_$hDYwO{6&el~s=ho<5n|HVNy!YN&I_lGVTbp+8ou#!+D?NSJo>_YDr*^fT`To0S z>DsM3TA#Z8wpqG<(|23@Jo3g_dcy;+Z~e;Cu9~IC{^j4d4m|4%vvtvD=7G@}HF6_c$Bu?{J7T1n~oW! zi;sGFYyHJ%jM9^je@p8Tr@woYjy`yEYaV;SC>?Okr(3^q`CpCF;g9@c>us0((n->H(@}cszkg@zo!|NWQ99&#r?lqYmyFUg?jEhp z&pU9GPT2o}t$+B=m20KP-1sN0>n=HZt@P=C{qBvW-}=MDwv`@t;Q{NVf3fo|t=AuP z*?Q^iTkdT=^&^j(rG0+q$kzQY`lVSq@aIozoqXDxX6d3E|FCuWHSe3HcOG+o>&1s$ zG)ve0*0$Cw_y6lzT7T3Pt)ox6W|qGGz;Cn;{PwqJ>8X3a*V^&?+h^&DtL|uRfAh{+ z+W6|-t*4!G?<_s>Lwj5M@7+5~Ti>wKuO58wER7wzS{r|G_blD`vv;)4*!06$y8HI; zw)TAD##y@N6W6y+zw4@5dd->t(7NrF|7Vsye(q;m`~1SkXX(=$|Dtu!#eY0YKl{JqOKRNEFTVFlr zqnDH}`qAe%mLBl%7p;{}KjSZ2x4-NgYo%BGho2dxBVK-J>*sEJ)+oLI>JwVqH=I67 zKRWEqtvg=vN27GmqyDUQ_H#ZqN?ULGRO_r;K0iu-eC*||+jf3=ln&W>ZR@o0tx@{K z7jAC7;7zxV(#fye-n#Asca741+wW;T`89h+>CpZ6w%&Tn-ckDSH&!}(_r0UEb^XzK7pDO8cMt&DMFxTsulvAMxeZJ5Tw$Q9Aa4pKtBB z^WsrD__-f#-Sm+2M(MKS-ql*)_pDJmam%T#?I-@)D81yAXS9CzD+iC#br!mGQ&TRej4WC;t9eL``)(ww(GXpQoR% zpSPdC--q9i-2WN-?!ht&x6m0&x_BG&y&xW&zsMm&!f+$&#TX`&$G|B&%4jR z?}P7$?~Ct`@00JB@0;(R@1yUh@2l^x@3Zf>@4N56dZ0e27wU(4qQ0m%>W_M)KB-sg zmwKkYsdwt1dZ<3Cm+Gf_s=lhX>aTjNKC9R2w|cI=tM}@^exN_-7y5^OqQB@j`j393 zKj~Ncmwu+d>38~{eyBg{m-?rEs=w;D`mcVhKkL`}w|=g_>-YM1QHNTo?&9~-V^RIc>d~9AeKbxn`*XC{Ww|U%rZeBOPo9E5< z=6&_he=`;z_1K4rhMZ`r@> zWA-!qn*Gf_SG!&7zJ1UBXCJg5+86DQ_DTDtebfGFAGM#_SM9I%S^KSh*ZylCwjbM< z?a%gU`?Y=B{%s$(pWD~%@Ai57y?x*Qj|bobcmaNZC*TWs1O9+V;1hTSet~D;8+Zr) zfrsEDcnN-jr{F7i3;u$~;4^p)euL-WJ9rQNg9qV5coBYtC*ezY6aIup;Zu0k6K?&$ zhf4X^+e-QO*Gu`&xANa#FXf*%OZn%w^1nAr`QLBl&ofK;^R@Elou&NwTlxFUQvQCe z{C#IBfB#l~9%YV&wG~g^Ka$%F-!UVwDS9!rTqR{`F+k( ze!s2!zGo@F|5iQ^vy{(AE1#EH%IBw*&(kdB^VQ1dZI<%+YvuDeOZj}Z@_C)5e12Q` zJkL@-->rP!XDOfmR=y9jlrP#M=9U`R_ehhr9QM$FGeZ#qm_CxN~tfc)SFRC{b{8hjZ*4U zEA?uWQomYZSEF^+&XM{yQtw)+f1{Lo*h+mIrPRw->gOn>p0-k7M=ABTmHInMsmHC< z=TS<%Zl!*YQtEjt>}s?w`pihZAF2PX^n+1Ke`uv&j8ghXEB$1Y(qCHXH=~sP(@H;D zE2TfR(y!J^>0hn%v!|ER-?o*~@77D{f35Vx^-}udETvzz(m!V@{j`<-I!o!dt@Phn zNF=|Ye&0&}pQX$Lt;`3rlzE|*`C*naPqZ>$%u?o!R_2dc z$~@A_d@@U!S6Z20W-0SbEA!1PW!`CJ{+XrBL#@n5vy^$MmHBCwGEcQKU(HhHtybo* zS;{=t%6v9Unb%sG-)1TETr2b4EM?wnW&T?)Wgcv0K3p$lUffp7{8$WYW9G@VQs&E6 z=FPQI=Fe8<(NW5L+RD5-N||3 zw=!>!Qs(bg=J8R=eBR2uK1!M2TbbuaDf4|R^ZqDh{%>U;7^Un7t?Ucojdzdi53TGI zqm=!km3?EBvVXL)kBn0GlUDYXQOf?(%04qn*>76ecSb4uPb>S-C}lrtWnUVl>`$%i zQ=^pqYOR!gtCjt0t(1MNmHq7LrR;0lO4;AmOWEgI+3(g%+4p8C`(G>j;4Ec7Y-L}Z zrRNGueS4O&f48!a&r&8Z@%Quh_4oJl@bmHW^7Hfa^z-%e_Vf4q@cZ%m^854q^!xSu_WSpF@cHn0 z@%iz2^7-<4^ZE06^!fC8_4)OA_WAaC_xbmI@cr<8@%{0A^8NCC^ZoOE^!@aG_5JmI z_WkyK_x)E7)CcuK{ZLQT7xhN{QIFIo^-BFx&(t^dPW@94)kpPG{ZvoYSM^r?RgcwY z^;-Q_&((MJUj5e(^auSy|IknL7yU;6(U0^e{YwAR&-6F_PXE&n^+)|u|I|^;{xpx8PtB|5SM#j-*1T)} zH4mGQ&CBLz^R)TeylwtAkDJfU>*ja!y!qa|Z~nIr*bnRr_6Pff{ldOs|FDnPPwXr9 zm)d1o_w6_K9s7@c$bMvBvOn3U>{s?J`f$)Ds` z@-O+BnGcuzRWexoZvHnvoIlPl&;GIGuad!%r;x9Zw~)V($B@sE*O1?k=aBEnzP#kG zlEISykp~K2S@KuOhb{T5RbFt(U#;?^Oa5w=CtmVbt9-e6$NXa+G9Q_j%unVi^Obqa z{AC_9pPARpZ{|7koq5mvXC5>knitKF=E<5ZOH2N0b>3siU#-sn*hlTB_Er0gN zfnVSm_y*pAf8Zhb2wsAp;3@bD-h#j2G58E#gWupe_zvEK|KLIR5MG2I;Ys)s-h@Bl zQTP;IwfQM`onOMU@GZOx|H8xYF}w^v!_)9JybXWDfg#sysRV9KVi# z$Is*M@%#9H{6PL7zmR{(PvkH18~Km?Nd6?hl7Gq1^@{ZN0@FZECTRDacP^B1^{%W7K-`aQWzxHAKv3=S8 zY@fDY+qdoC_Hp~Ueck?UpSR!J_wE0906u^h;0JgDzJNF24|oJVfmh%ccm}?KciL2GzKM6@pLi%fikIT2cq+b%x8kpOEIy0Z; zH{;KEG(L@2OQ&U{;Spf zg5g(v-(dJv-#-|B)%OvGU-kWj;a7cMVfa$5_Kih}x$M$9WvwhlrZQr(k+sEza_I3Naecpa=-?#tc0r&u3fFIxq_yXR5 zKj0Dg#FD>S-PiuwU*7Nh5}t)`;a&I_9)^$MW%wDMhOgmm_!}OF&*63W9iE5p;eGfY z9*7U(h4>+!h%e%e_#+;PPvVvMC7y|I;+^;>9*U3RrT8hHim&3W_$wZZ&*HWCEuM?- z;=TAU9*hs;#rQFvj4$KO_%j}jPvh12HJ*)c5y2lI#d#r$J_GJl!h%zx%b^QZaM{A+$Tf1BUU z|K^AD$NA;_bACF1o!`!X=g0Hs`StvJem;Mn-_QS-2apet7my#2Cy+0YH;_M&N03jD zSCC(jXOM4@caVROhmenumyn;3r;x8$@>k3KSCzq9@>fg#sxp}Bf%>3cs2}Qy`l8;b zKkAYCq+Y3C>Y4hc-l>1;q57y^s-Nnq`l{Zlzv{92tX`|%>bd%^-mCxmf&QRh=pXut z{-WRLKl+jWq+jV@`kDTw-|2t)q5h~}>Yw_l{;J>VzxuKMtY7Qj`nmqD-|PS80rP=* z!TeyJFkhHA%pc|v^ND%I{9>Ll- z_iFYlE%#ro?!#T~zgpdoY+tlL+9&Oo_D%bzebjzxU$wv5XYIH4UHh+n*nVtZwm;ja z?br5g`?r1Eer{j4zuV{S_x64JKOTS&;05>ro`5gl4fq2dfluHS_ywMUZ{QvH2Offt z;3fD8o`SF7E%*x_gU{eK_zj+e@8CW74<3XM;YIino`f&qP52WYg-_vCo1b#pp{4xm z<^91Wf3>_nxV%5Oyg#_SKe*g~wY)#Lyg!&X2y=dLd4DkPKQ8YNuI{6AKb`yP++XKD zJNMhU@6P>q?!ybe>i0>*ulmls@T-2mH2kXHGY!A$_f5mE`n}WetA779{A!fKuUg?( z{a$MLRllDae%0@(R<^Vhe$@)U>i1W}ulhaK@T-2GHTl@HeqT8Js^1$9zv}mg!>{^1 z;_$0}pE&%g-zyHk>i3JoulhaX@T-2`IQ**LI}X3<_m9J``aR_EtG=Hz{Hote4!`R6 zlf$q2J!ScFd2{)5d35=7d3E`9d3O1Bd3X7Dd3gDFd3pJHd3yPJd3*VLd3^bNd42hP zd4BnRdH>4(mzMVjSMQ-O?+?DecU0$voF8(Y$oV4YjhsJP@>hFD7}f~CYK32oQutLX z{A!fKuUg?(qZEGC3cnhq@T*q%)hLBuwZgAPDg3Gxel<$rSFP}?Q3}6mgsH$Q#HX$Ro%n$ScS%$TP?{$UDeC$V13S$VUR`KvmUV?VSn+8^zc_DlPw{nI{bKeeygU+uH@ zTl=p4*FJ1NUh-EfUV@+CDfkNBg1_J~_zYfy-{3j;4&HxgNu4is-qiV1=TV(cbzar^Rp(iqZ*|_)`B&#* zosV^1*7;fIX`Qci-q!hB=W(6Sbzax`UFUi8{ai2M&G<7OjZfp%_%)uL`p|qk^}o5{%oJNU)#6s-}Z6)xqaRKZlAZ`+xPAN zcmO_t7vKkY0=|GZ;175NK7m)@7kCD~fp_2^cnCg%m*6LO3ciB3;4gR#K7-fbH+T-d zgZJP+co05>7vV>E6262t;ZJxJK807^^qy~@U&6ESExZf=!o%<}ybM3X)9^LC4S&Pq z@HxB=zr*wJJ-iS9!vpa_ybwQJ@>g$PE2fMuALFrf1lsa|Ca}l z50Dp-ACM=IFOWBoKafX|PmouTUyx^zZ;*G8e~^cekC2y;pOB|m@>kF3jE(o^!meg1 z>}tIfe$@)QS}%oNZ7YRc6~o#XcC}UtziKV-4}N>p8;IUR^gg2Z61|`3Jw@*;dT-JD zi{4}OKBM;2La-{-+=6kNTzlsh{ev`mO$}AM4Nhwf?Q2>+kx#{%;;IAD9=+ z59SH;g?YpLVIDD`m{-g%<{9&idB^-?9x@-9m&{M*Df5+i%lu^?GoP8)%x~s7^WBoa zTHYV5H|O1dY#+2A+86DQ_DTDtebfGFAGM#_SM9I%S^KSh*ZylCwjVF~tDlVg)oXwG zZHJWbEPM;^!oTn^d<-wc&+s&S4R6EW@Hl)9ufy-~JbVxD!~gI=d=M|h5Aj5N5pTpF z@ko3Uuf#9$OnejX#6R&+d=xLmPw`ZI6>r5~@mPEouf=ciTznVr#eeZ&d>AjrkMU%D z8E?j)m;BYrk2vtGFU;0OpP2_nXN(J5KYhfX&yw@ym!7>kZ@yu}^8VoR{@`KTGEeq> z6qzskeu^c3wdAkr&KmdExX;G@HtxG||Bd@_+>hhF9QWtAPsjZ_PkznnzMVt&-#bcg zy=5iuqn*8b^i4mUotamWP&)mY0^FmZz4lmbaF_mdBRQ zme-cwHcyx@%p2iX%l%i&{a4dIVn4C3*k5XwY2CNqEcvUa&Lw|!?~l));UV}4UV@+C zDfkNBg1_J~_zYfy-{3j;4&H--N>AUlXO!Oisa>sSzW?q~x_0Z1)~9a2ZIrIx^xf7zkGye| z-tfTdTX{crlv2-IVOOJd(Pu{L{Yd?9r5}t^`a>)IVwA$KS_kd-s!_W8MaQ)sx8sSU zbkDAbw~qgr+ty0n7kS=;mi$%SdEov7ejI;}U&p`W=kfRWef&RuAb*fw$Uo#K@)!Ay z{6~Hyf0AFxzvO4~H~F3XPktzWlwZm}<)`vj`K|m{ek^~MU(3Jc=kj;?z5HK(Fn^d| z%s=KQ^OyO}{AYeNf0|#-zvgH2xB1=tZ+Yx&LZ;e{gw!@Y?ky ze|7DwJMh#4`Am6D`AvCF`A&IH`A>OJ`A~UL`B8aN`BHgP`BQmR`BZsT`BiyV`Br&X z`B!;Z`B-^b`B`~d`C55f`CEBh{XxIbKlBs*MZeL1^dtRAztX?-GyP4!)Bp5C{ZYTv zKlM}nRln7L^<({6zt+F?bNyYv*Z<7}<^%JB`N2G4zA$f?Kg=WM6Z4As#XMuaG4Ggv z%tPiQ^OE_=JY~KzZ<)W$W9BpSn)%HHo%jG=fFIxq_yXR5 zKj0DgM6rs}^8Vmmd4KS(<^HR@Jy;p6C4aTte^nj_zhlW?t=@m}9*p;4ycgsB81Kn= zU&ebgnO{aJ^GqxA%_wEwX=VNyrOZRE%txb?d8w88X_PWgwK89gQs%8z=C4u8Jl4v5 zHcFY-TAANQDf3(_^W7+A-fLz4>-T#y5B7ULnGgGYpUjK>-VgtiAIcx)m-0{fsr*%b zEB}=r%b(@f@^AUM{9S%8|Cb-kALbYHkNL^`WqvdNnIFxc=2!Et`Puw!emDP{AD;bU zx&JC}gvm?DPsmfqSIAq)U&v#~XUJ>FZ^(1VcgTClf5?N#hscX8`K#y8oPnD;57)~1 zxLL}1xmM24%~Hc{%CeyxA&=lZ*T zum76|%m?NL^MiT9d|}=&f0#$iC*~FNi+RR;W8N|Un1{?q<|Xr!dCGic-ZFog$INHu zHS?Q!&U|OyGyjLH~if3@VVDucD;ua^AP0ef$qd9S#Y`C*naPqZ>$%u?o!R_2dc$~@Al zw+EN^2ag>5R{kqLmOsm{<=^si`MdmH{x3h6Kg=)YAM=y>%lu~kGe4R?&9CNP^RxNe z{BHg?Kb$|#FXx~0)A{TCcK$m*o#U zTH#l-6n@nTznZ1+t5*2cEam)QD`y91Ddz`UIXgH@IX~FS*}+-L`N7uJM|^pf-g(O3 zwT^w@=Vxihofo$be(pzS>86LA*ShSucg@oJzGt;g+;ZwH<^J}T`;FXpB!A;RB=;k^ zFUkE$?o)EVlKYn2zvMn9_cOV#$^A|4b8^3v`<~qY`*m-K`6bzju@>d$r`Rmi(3Pzj~lPs2A#odZNCl zH|mdiq&}%v>X&+^zNvTWpL(c1s+a1gdaAyvx9YEYtUjyP>bH8XzN`1@zkZ-U=ok8j zexkqVH~No$q(A9b`j>vDzv*}SpMI!6>X-VbeyYFfxB9PstUv45`nP_rzw7t6DJZZi(Z<;^NqcxkBKKp`OT5Dhb)-2`j_Ezq0pQXbe`Nh_6T>e+Hbig&A zZp~van5Cl+-rRb`>F=JUCm;Wo*7}Rjn5B!4dU@;QO~=gAwwoW<`u;T!o~5ntyK%jA z(q$iMy<*$T)=N)2;p%NA_uabx)_Dx)Go06Oe#3bV=R2JDaQ?%25a&aj7jb^1&XTn5 zJ740wiSsAUqd1@9yo&QH&a*h*;=GIVFV4d_ALG1?^E1xV)Y+QWedleQzi}SN`5fnU zoZoSt$N3)TeVqSs9>{*`ypZ!l&J#IbiSI%QC`K!0ix+C3v>F!T=pSt_i-M8-kb@#EmpB=B-^w2LJT5_Jy z`9kLnoj-IQ(fLH@6`fynp3(V6=N+AYbRN?ANarP;pLCwm`AX+4oxgM*)A>y2HJ#sd zp40hGyb*uIBk@VR62HVVm;BX|zgqHFw~ovUE&dljj6cRNp{@)P-s{6_vGKaxMmujF6yGx?kRPW~r9lt0QZ<)89X`K$a^ z{wqJ0Kg+M>-|}<$yZm1MFF%++%r9Q@S8rU;4z`|std;%jl2Z1yjiu~wYo+XSt?YMe zrR;m7l>M)jeQ=brAGWeDj#Bo=R`$tJazD+IzdCD#VU4h>*7E+~?iq$P!>(FkSF^NX z!-mh5{A>Sy{bx&IU#;-4Sqd9#g_q4z*ja0NfAHcFraHn_Tj8yv6!zM3pPT#L-1p}G zH}}E0AI^Po@{96}@{RJ2@{jV6@{#hA@{{tE@|E(I@|W_M@|p6Q@|)_3`l8;bKkAYC zq+Y3C>Y4hc-l>1;q57y^R_!c>U$w%o)=S}6vlM>S3cs4A@T*q%)hvZywZgAvDg3Gx zel<(sSFP}?Sqi^ugU@iAw%`mLK`)axW>L>sHtK;^5a!U!%!ng1){0k4m$M7=z3{NYz zR$AU4Ja-mz#b5DQd={_8Z}D7w7w^S?@nC!yFUF7YWPBNK#-H(Md>XIDukmbr8}G)y z@o;<`FUQaEbbK9e$KRLxua@@*GtH?7P&vy}O#m3e5EG9R@vFU?Zsr&i{vS;~CX z%Dgp8nZH_@$7U(>Su69}EMud& z@>=p+@?7#=@?P>^@?i2|@?!F1@?`R5@@Dd9@@VpD@@n#H@@(>L@^12P@^JET@^bQX z@^tcb@^TD{erw;g|JsM`$M$9WvwhlrZQr(k+sEza_I3Na zecpa=-?#rS`K$Ld6T*k^BK!zX!k6$S{0Wc3r|_yL-1>n-OH2N0x&JCPVWvK`QZHsH z^`n(~GE1p1t<;-YO6_TdU(HhZRV(~zmcp-E;a9VD)y|oG=}g|VmHg=}C6C%lK6RFo zS8XM~I!np3wvun1rQ}^($-mB0^02MsV`nLO*;ewiv*bMv{zSjkvAjRn-c|Wm{4D+! zzl;CH595#V%lK#fH2xaDjsM1vs~{vN-N|HlvH5AqB7hx|nTBEOOU$dBYt z@+rf1lsa|Ca}l z50DqA>_ADLK)yiUK>k1;K|VoVL4HA=LB2uWLHwe$Wa}7^U!q zR%Nf2{M8*l&QHlx$ydo+E%~c`M;O)!ziNeFjgq{c{GQK~&zH}e&!5ku&!^9;&#%w3 z&$rLJ&%f`3?}zV;?~m`3@0ahJ@1O6Z@2Bsp@2~H(@3-%}@4tGWKByP!hkByEs5k15 zdZa$7SL&B~roO3n>YsY3KB|}Mr+TWss<-N|daORH*Xp-=uD+}H>c4)VKj;_whkl~J z=r{V0ezfGTR`ZE@#r$HPG2fVX%s=KK^O1SU{A8XoUzxYeU*<9MnR(6pW}Y+OnfJ_p z=0WqJdC~l6o-|*YH_e~sQS+&Jb;)0?-cPsB*l+AR_86uk2g)FZ-DN z%)VxSv(MFT*Sc@tv;Wx#?T7Y7`=fo*ereydf7(awr?sn=!mnE4SF=>vt0jN6dpK|A z{F(D;&ZjxA=KPxTY|ghi@8hfZ%@5~~^UJe;j1oVczs_&xzw_hy^Za`LJwKnn&+q5|%LB*<$P361$P>sH z$Q#HX$Ro%n$ScS%$TP?{$UDeC$V13S$VC%Dm3&oY ztxDlnN0t0*|Gxjc|Gs~of4=`c|NH(t{(Szt{`~$v{(kS3cs4A@T*q%)hvZyweTQ(2rt5q@FaW*Z^EDOD0~X9y6HXt<T_q_!l0AkKtwb8J>o(;cfUE9*582b@&~ghwtHi_#YmK58{RRA)bgY;*Izt9=Y6q zwc^uwHGYj}-Et?;W+3cqTFUyV}uRV(~zl)|rC;a8&+e$@)U8l~{7R`}H@xi5f!#n0kz@w@n6 z{4o9)zl?vzPvfug+xTz%IQ|^Jj(^9` z$*wv~AAYr7%K5=o&JS)Y<^15rQqB*qm2!TtmGgsZrOI9{ z`KxP(dZ0e27wU(4qQ0m%>W_M)KB-sgmwKkYsdwt1dZ<3Cm+Gf_s=lhX>aTjNKC9R2 zw|cI=tM}@^exN_-7y5^OqQB@j`j39J+=cp-j>C*q5EBmRg-;*)q~?r(2?nfu#kDfhRxu6)Q%v$XkzH?($L z_mx>X@?-zldd}s4GfV4Zwk~+sCuhm~0?xZS|LQ!f^Rdp$IzQ_?t@E|c+d6;iJg)P( z&g(kA>pZXXz0UhO|LZ)k^TGMPj}m@eEW4EY&!Tk&FV{ku%`x@Ne;64ZUJGk$` z{SV<+%l%iU+&kP4;=T~~hqzB9`@tyjyZPVzaQ-;IoPW+w=dbhI`S1LA{ye{)f6ve7 z?=ShQC4W_!ANe17Ao(D9A^9PBBKabDBl#nFB>5zHCHW1JS&qapDC{?zbVft-zo1Y|0xeDA1W^@KPpcuUn*}Ze=3hE zpDM2^zbelv-zx7a|EeCV&+4`Mt)8pz>b?4}ALtMIh5n(R=r8(>{-Yo1Px_VqrJw0< z%l%iYdB^-?9x@-9m&{M*Df5+i%lu^?GoP8)%x~s7^IhG2Ra)-9TAjbJ4=nEwerME< zXkWBH+9&Oo_D%bzebjzxU$wv5XYIH4UHh+n*nVtZwm;ja?br5g`?r1Eer{j4zuV{S z_x64JKOTS&;05>ro`5gl4fq2dfluHS_ywMUZ{QvH2Offt;3fD8o`SF7E%*x_gU{eK h_zj+e@8CW74<3XM;YIino`f&qP52WYg-_vC{}0KKtDXP= literal 0 HcmV?d00001 diff --git a/tests/acceptance/refs/gain-half.wav.json b/tests/acceptance/refs/gain-half.wav.json new file mode 100644 index 00000000..303838c4 --- /dev/null +++ b/tests/acceptance/refs/gain-half.wav.json @@ -0,0 +1,23 @@ +{ + "config_hash": "e1622f157cf37917", + "created_on": { + "arch": "arm64", + "date": "2026-06-16T14:22:49.718+01:00", + "os": "Mac OSX 26.2" + }, + "plugin": { + "format": "VST3", + "manufacturer": "Tracktion", + "name": "pluginval Gain", + "uid": "VST3-pluginval Gain-327cfb94-d8aa82c7", + "version": "1.0.4" + }, + "pluginval_version": "1.0.4", + "render": { + "block_size": 512, + "length_seconds": 0.25, + "num_channels": 2, + "num_samples": 12000, + "sample_rate": 48000.0 + } +} \ No newline at end of file diff --git a/tests/acceptance/refs/sine-440.wav b/tests/acceptance/refs/sine-440.wav new file mode 100644 index 0000000000000000000000000000000000000000..8d419d31934853091f74eb08b088b3abd71cfdf4 GIT binary patch literal 96104 zcmeI*`}5~zectg!V=X3lh?7=vw2VNZG9F7H>V9ufTZkf#CQ;*4tQw(tM}h< zzYTA^B>&HK57_W8Hyp6xl`lW#C{(0 zY?L1In~!h(w;O(Ol%9J2e{Nm+vHw0wr=9fH*5m)+J)`ssPxxT#Gtb&OO2=OIm#w=# zeCa5?;+8MAEqAfZ(I|cG0Y7M6`H-7N zY4ZzjXzjZ0E2DJe$NsVPoXh`al-9>=UGT6^j?%W{KHPffo#&3yv-f{T>$FSH9Hqzq z=;YS@-tpp5`ts{Hw*K@B2aM9KZ~Cd$>E~auR=WJ%f7Cki!AGo>&e`(Djim>j{Gx58 z=lrL`)=RJc#kaRU{;JE@OK<=FKC|@tBOlXx*zTv!(xs1mVe5aq>o;fV#B+bIwPC|K zv-J9#-rt(f-ZD!EKKc`_(|&b*mL9bIuUi-2c*QK;aPU`JZ@%ywv-F%ZzSBD4P2ZoT z4_$hDYwO{6&el~s=ho<5n|HVNy!YN&I_lGVTbp+8ou#!+D?NSJo>_YDr*^fT`To0S z>DsM3TA#Z8wpqG<(|23@Jo3g_dcy;+Z~e;Cu9~IC{^j4d4m|4%vvtvD=7G@}HF6_c$Bu?{J7T1n~oW! zi;sGFYyHJ%jM9^je@p8Tr@woYjy`yEYaV;SC>?Okr(3^q`CpCF;g9@c>us0((n->H(@}cszkg@zo!|NWQ99&#r?lqYmyFUg?jEhp z&pU9GPT2o}t$+B=m20KP-1sN0>n=HZt@P=C{qBvW-}=MDwv`@t;Q{NVf3fo|t=AuP z*?Q^iTkdT=^&^j(rG0+q$kzQY`lVSq@aIozoqXDxX6d3E|FCuWHSe3HcOG+o>&1s$ zG)ve0*0$Cw_y6lzT7T3Pt)ox6W|qGGz;Cn;{PwqJ>8X3a*V^&?+h^&DtL|uRfAh{+ z+W6|-t*4!G?<_s>Lwj5M@7+5~Ti>wKuO58wER7wzS{r|G_blD`vv;)4*!06$y8HI; zw)TAD##y@N6W6y+zw4@5dd->t(7NrF|7Vsye(q;m`~1SkXX(=$|Dtu!#eY0YKl{JqOKRNEFTVFlr zqnDH}`qAe%mLBl%7p;{}KjSZ2x4-NgYo%BGho2dxBVK-J>*sEJ)+oLI>JwVqH=I67 zKRWEqtvg=vN27GmqyDUQ_H#ZqN?ULGRO_r;K0iu-eC*||+jf3=ln&W>ZR@o0tx@{K z7jAC7;7zxV(#fye-n#Asca741+wW;T`89h+>CpZ6w%&Tn-ckDSH&!}(_r0UEb^XzK7pDO8cMt&DMFxTsulvAMxeZJ5Tw$Q9Aa4pKtBB z^WsrD__-f#-Sm+2M(MKS-ql*)_pDJmam%T#?I-@)D81yAXS9CzD+iC#br!mGQ&TRej4WC;t9eL``)(ww(GXpQoR% zpSPdC--q9i-2WN-?!ht&x6m0&x_BG&y&xW&zsMm&!f+$&#TX`&$G|B&%4jR z?}P7$?~Ct`@00JB@0;(R@1yUh@2l^x@3Zf>@4N56dZ0e27wU(4qQ0m%>W_M)KB-sg zmwKkYsdwt1dZ<3Cm+Gf_s=lhX>aTjNKC9R2w|cI=tM}@^exN_-7y5^OqQB@j`j393 zKj~Ncmwu+d>38~{eyBg{m-?rEs=w;D`mcVhKkL`}w|=g_>-YM1QHNTo?&9~-V^RIc>d~9AeKbxn`*XC{Ww|U%rZeBOPo9E5< z=6&_he=`;z_1K4rhMZ`r@> zWA-!qn*Gf_SG!&7zJ1UBXCJg5+86DQ_DTDtebfGFAGM#_SM9I%S^KSh*ZylCwjbM< z?a%gU`?Y=B{%s$(pWD~%@Ai57y?x*Qj|bobcmaNZC*TWs1O9+V;1hTSet~D;8+Zr) zfrsEDcnN-jr{F7i3;u$~;4^p)euL-WJ9rQNg9qV5coBYtC*ezY6aIup;Zu0k6K?&$ zhf4X^+e-QO*Gu`&xANa#FXf*%OZn%w^1nAr`QLBl&ofK;^R@Elou&NwTlxFUQvQCe z{C#IBfB#l~9%YV&wG~g^Ka$%F-!UVwDS9!rTqR{`F+k( ze!s2!zGo@F|5iQ^vy{(AE1#EH%IBw*&(kdB^VQ1dZI<%+YvuDeOZj}Z@_C)5e12Q` zJkL@-->rP!XDOfmR=y9jlrP#M=9U`R_ehhr9QM$FGeZ#qm_CxN~tfc)SFRC{b{8hjZ*4U zEA?uWQomYZSEF^+&XM{yQtw)+f1{Lo*h+mIrPRw->gOn>p0-k7M=ABTmHInMsmHC< z=TS<%Zl!*YQtEjt>}s?w`pihZAF2PX^n+1Ke`uv&j8ghXEB$1Y(qCHXH=~sP(@H;D zE2TfR(y!J^>0hn%v!|ER-?o*~@77D{f35Vx^-}udETvzz(m!V@{j`<-I!o!dt@Phn zNF=|Ye&0&}pQX$Lt;`3rlzE|*`C*naPqZ>$%u?o!R_2dc z$~@A_d@@U!S6Z20W-0SbEA!1PW!`CJ{+XrBL#@n5vy^$MmHBCwGEcQKU(HhHtybo* zS;{=t%6v9Unb%sG-)1TETr2b4EM?wnW&T?)Wgcv0K3p$lUffp7{8$WYW9G@VQs&E6 z=FPQI=Fe8<(NW5L+RD5-N||3 zw=!>!Qs(bg=J8R=eBR2uK1!M2TbbuaDf4|R^ZqDh{%>U;7^Un7t?Ucojdzdi53TGI zqm=!km3?EBvVXL)kBn0GlUDYXQOf?(%04qn*>76ecSb4uPb>S-C}lrtWnUVl>`$%i zQ=^pqYOR!gtCjt0t(1MNmHq7LrR;0lO4;AmOWEgI+3(g%+4p8C`(G>j;4Ec7Y-L}Z zrRNGueS4O&f48!a&r&8Z@%Quh_4oJl@bmHW^7Hfa^z-%e_Vf4q@cZ%m^854q^!xSu_WSpF@cHn0 z@%iz2^7-<4^ZE06^!fC8_4)OA_WAaC_xbmI@cr<8@%{0A^8NCC^ZoOE^!@aG_5JmI z_WkyK_x)E7)CcuK{ZLQT7xhN{QIFIo^-BFx&(t^dPW@94)kpPG{ZvoYSM^r?RgcwY z^;-Q_&((MJUj5e(^auSy|IknL7yU;6(U0^e{YwAR&-6F_PXE&n^+)|u|I|^;{xpx8PtB|5SM#j-*1T)} zH4mGQ&CBLz^R)TeylwtAkDJfU>*ja!y!qa|Z~nIr*bnRr_6Pff{ldOs|FDnPPwXr9 zm)d1o_w6_K9s7@c$bMvBvOn3U>{s?J`f$)Ds` z@-O+BnGcuzRWexoZvHnvoIlPl&;GIGuad!%r;x9Zw~)V($B@sE*O1?k=aBEnzP#kG zlEISykp~K2S@KuOhb{T5RbFt(U#;?^Oa5w=CtmVbt9-e6$NXa+G9Q_j%unVi^Obqa z{AC_9pPARpZ{|7koq5mvXC5>knitKF=E<5ZOH2N0b>3siU#-sn*hlTB_Er0gN zfnVSm_y*pAf8Zhb2wsAp;3@bD-h#j2G58E#gWupe_zvEK|KLIR5MG2I;Ys)s-h@Bl zQTP;IwfQM`onOMU@GZOx|H8xYF}w^v!_)9JybXWDfg#sysRV9KVi# z$Is*M@%#9H{6PL7zmR{(PvkH18~Km?Nd6?hl7Gq1^@{ZN0@FZECTRDacP^B1^{%W7K-`aQWzxHAKv3=S8 zY@fDY+qdoC_Hp~Ueck?UpSR!J_wE0906u^h;0JgDzJNF24|oJVfmh%ccm}?KciL2GzKM6@pLi%fikIT2cq+b%x8kpOEIy0Z; zH{;KEG(L@2OQ&U{;Spf zg5g(v-(dJv-#-|B)%OvGU-kWj;a7cMVfa$5_Kih}x$M$9WvwhlrZQr(k+sEza_I3Naecpa=-?#tc0r&u3fFIxq_yXR5 zKj0Dg#FD>S-PiuwU*7Nh5}t)`;a&I_9)^$MW%wDMhOgmm_!}OF&*63W9iE5p;eGfY z9*7U(h4>+!h%e%e_#+;PPvVvMC7y|I;+^;>9*U3RrT8hHim&3W_$wZZ&*HWCEuM?- z;=TAU9*hs;#rQFvj4$KO_%j}jPvh12HJ*)c5y2lI#d#r$J_GJl!h%zx%b^QZaM{A+$Tf1BUU z|K^AD$NA;_bACF1o!`!X=g0Hs`StvJem;Mn-_QS-2apet7my#2Cy+0YH;_M&N03jD zSCC(jXOM4@caVROhmenumyn;3r;x8$@>k3KSCzq9@>fg#sxp}Bf%>3cs2}Qy`l8;b zKkAYCq+Y3C>Y4hc-l>1;q57y^s-Nnq`l{Zlzv{92tX`|%>bd%^-mCxmf&QRh=pXut z{-WRLKl+jWq+jV@`kDTw-|2t)q5h~}>Yw_l{;J>VzxuKMtY7Qj`nmqD-|PS80rP=* z!TeyJFkhHA%pc|v^ND%I{9>Ll- z_iFYlE%#ro?!#T~zgpdoY+tlL+9&Oo_D%bzebjzxU$wv5XYIH4UHh+n*nVtZwm;ja z?br5g`?r1Eer{j4zuV{S_x64JKOTS&;05>ro`5gl4fq2dfluHS_ywMUZ{QvH2Offt z;3fD8o`SF7E%*x_gU{eK_zj+e@8CW74<3XM;YIino`f&qP52WYg-_vCo1b#pp{4xm z<^91Wf3>_nxV%5Oyg#_SKe*g~wY)#Lyg!&X2y=dLd4DkPKQ8YNuI{6AKb`yP++XKD zJNMhU@6P>q?!ybe>i0>*ulmls@T-2mH2kXHGY!A$_f5mE`n}WetA779{A!fKuUg?( z{a$MLRllDae%0@(R<^Vhe$@)U>i1W}ulhaK@T-2GHTl@HeqT8Js^1$9zv}mg!>{^1 z;_$0}pE&%g-zyHk>i3JoulhaX@T-2`IQ**LI}X3<_m9J``aR_EtG=Hz{Hote4!`R6 zlf$q2J!ScFd2{)5d35=7d3E`9d3O1Bd3X7Dd3gDFd3pJHd3yPJd3*VLd3^bNd42hP zd4BnRdH>4(mzMVjSMQ-O?+?DecU0$voF8(Y$oV4YjhsJP@>hFD7}f~CYK32oQutLX z{A!fKuUg?(qZEGC3cnhq@T*q%)hLBuwZgAPDg3Gxel<$rSFP}?Q3}6mgsH$Q#HX$Ro%n$ScS%$TP?{$UDeC$V13S$VUR`KvmUV?VSn+8^zc_DlPw{nI{bKeeygU+uH@ zTl=p4*FJ1NUh-EfUV@+CDfkNBg1_J~_zYfy-{3j;4&HxgNu4is-qiV1=TV(cbzar^Rp(iqZ*|_)`B&#* zosV^1*7;fIX`Qci-q!hB=W(6Sbzax`UFUi8{ai2M&G<7OjZfp%_%)uL`p|qk^}o5{%oJNU)#6s-}Z6)xqaRKZlAZ`+xPAN zcmO_t7vKkY0=|GZ;175NK7m)@7kCD~fp_2^cnCg%m*6LO3ciB3;4gR#K7-fbH+T-d zgZJP+co05>7vV>E6262t;ZJxJK807^^qy~@U&6ESExZf=!o%<}ybM3X)9^LC4S&Pq z@HxB=zr*wJJ-iS9!vpa_ybwQJ@>g$PE2fMuALFrf1lsa|Ca}l z50Dp-ACM=IFOWBoKafX|PmouTUyx^zZ;*G8e~^cekC2y;pOB|m@>kF3jE(o^!meg1 z>}tIfe$@)QS}%oNZ7YRc6~o#XcC}UtziKV-4}N>p8;IUR^gg2Z61|`3Jw@*;dT-JD zi{4}OKBM;2La-{-+=6kNTzlsh{ev`mO$}AM4Nhwf?Q2>+kx#{%;;IAD9=+ z59SH;g?YpLVIDD`m{-g%<{9&idB^-?9x@-9m&{M*Df5+i%lu^?GoP8)%x~s7^WBoa zTHYV5H|O1dY#+2A+86DQ_DTDtebfGFAGM#_SM9I%S^KSh*ZylCwjVF~tDlVg)oXwG zZHJWbEPM;^!oTn^d<-wc&+s&S4R6EW@Hl)9ufy-~JbVxD!~gI=d=M|h5Aj5N5pTpF z@ko3Uuf#9$OnejX#6R&+d=xLmPw`ZI6>r5~@mPEouf=ciTznVr#eeZ&d>AjrkMU%D z8E?j)m;BYrk2vtGFU;0OpP2_nXN(J5KYhfX&yw@ym!7>kZ@yu}^8VoR{@`KTGEeq> z6qzskeu^c3wdAkr&KmdExX;G@HtxG||Bd@_+>hhF9QWtAPsjZ_PkznnzMVt&-#bcg zy=5iuqn*8b^i4mUotamWP&)mY0^FmZz4lmbaF_mdBRQ zme-cwHcyx@%p2iX%l%i&{a4dIVn4C3*k5XwY2CNqEcvUa&Lw|!?~l));UV}4UV@+C zDfkNBg1_J~_zYfy-{3j;4&H--N>AUlXO!Oisa>sSzW?q~x_0Z1)~9a2ZIrIx^xf7zkGye| z-tfTdTX{crlv2-IVOOJd(Pu{L{Yd?9r5}t^`a>)IVwA$KS_kd-s!_W8MaQ)sx8sSU zbkDAbw~qgr+ty0n7kS=;mi$%SdEov7ejI;}U&p`W=kfRWef&RuAb*fw$Uo#K@)!Ay z{6~Hyf0AFxzvO4~H~F3XPktzWlwZm}<)`vj`K|m{ek^~MU(3Jc=kj;?z5HK(Fn^d| z%s=KQ^OyO}{AYeNf0|#-zvgH2xB1=tZ+Yx&LZ;e{gw!@Y?ky ze|7DwJMh#4`Am6D`AvCF`A&IH`A>OJ`A~UL`B8aN`BHgP`BQmR`BZsT`BiyV`Br&X z`B!;Z`B-^b`B`~d`C55f`CEBh{XxIbKlBs*MZeL1^dtRAztX?-GyP4!)Bp5C{ZYTv zKlM}nRln7L^<({6zt+F?bNyYv*Z<7}<^%JB`N2G4zA$f?Kg=WM6Z4As#XMuaG4Ggv z%tPiQ^OE_=JY~KzZ<)W$W9BpSn)%HHo%jG=fFIxq_yXR5 zKj0DgM6rs}^8Vmmd4KS(<^HR@Jy;p6C4aTte^nj_zhlW?t=@m}9*p;4ycgsB81Kn= zU&ebgnO{aJ^GqxA%_wEwX=VNyrOZRE%txb?d8w88X_PWgwK89gQs%8z=C4u8Jl4v5 zHcFY-TAANQDf3(_^W7+A-fLz4>-T#y5B7ULnGgGYpUjK>-VgtiAIcx)m-0{fsr*%b zEB}=r%b(@f@^AUM{9S%8|Cb-kALbYHkNL^`WqvdNnIFxc=2!Et`Puw!emDP{AD;bU zx&JC}gvm?DPsmfqSIAq)U&v#~XUJ>FZ^(1VcgTClf5?N#hscX8`K#y8oPnD;57)~1 zxLL}1xmM24%~Hc{%CeyxA&=lZ*T zum76|%m?NL^MiT9d|}=&f0#$iC*~FNi+RR;W8N|Un1{?q<|Xr!dCGic-ZFog$INHu zHS?Q!&U|OyGyjLH~if3@VVDucD;ua^AP0ef$qd9S#Y`C*naPqZ>$%u?o!R_2dc$~@Al zw+EN^2ag>5R{kqLmOsm{<=^si`MdmH{x3h6Kg=)YAM=y>%lu~kGe4R?&9CNP^RxNe z{BHg?Kb$|#FXx~0)A{TCcK$m*o#U zTH#l-6n@nTznZ1+t5*2cEam)QD`y91Ddz`UIXgH@IX~FS*}+-L`N7uJM|^pf-g(O3 zwT^w@=Vxihofo$be(pzS>86LA*ShSucg@oJzGt;g+;ZwH<^J}T`;FXpB!A;RB=;k^ zFUkE$?o)EVlKYn2zvMn9_cOV#$^A|4b8^3v`<~qY`*m-K`6bzju@>d$r`Rmi(3Pzj~lPs2A#odZNCl zH|mdiq&}%v>X&+^zNvTWpL(c1s+a1gdaAyvx9YEYtUjyP>bH8XzN`1@zkZ-U=ok8j zexkqVH~No$q(A9b`j>vDzv*}SpMI!6>X-VbeyYFfxB9PstUv45`nP_rzw7t6DJZZi(Z<;^NqcxkBKKp`OT5Dhb)-2`j_Ezq0pQXbe`Nh_6T>e+Hbig&A zZp~van5Cl+-rRb`>F=JUCm;Wo*7}Rjn5B!4dU@;QO~=gAwwoW<`u;T!o~5ntyK%jA z(q$iMy<*$T)=N)2;p%NA_uabx)_Dx)Go06Oe#3bV=R2JDaQ?%25a&aj7jb^1&XTn5 zJ740wiSsAUqd1@9yo&QH&a*h*;=GIVFV4d_ALG1?^E1xV)Y+QWedleQzi}SN`5fnU zoZoSt$N3)TeVqSs9>{*`ypZ!l&J#IbiSI%QC`K!0ix+C3v>F!T=pSt_i-M8-kb@#EmpB=B-^w2LJT5_Jy z`9kLnoj-IQ(fLH@6`fynp3(V6=N+AYbRN?ANarP;pLCwm`AX+4oxgM*)A>y2HJ#sd zp40hGyb*uIBk@VR62HVVm;BX|zgqHFw~ovUE&dljj6cRNp{@)P-s{6_vGKaxMmujF6yGx?kRPW~r9lt0QZ<)89X`K$a^ z{wqJ0Kg+M>-|}<$yZm1MFF%++%r9Q@S8rU;4z`|std;%jl2Z1yjiu~wYo+XSt?YMe zrR;m7l>M)jeQ=brAGWeDj#Bo=R`$tJazD+IzdCD#VU4h>*7E+~?iq$P!>(FkSF^NX z!-mh5{A>Sy{bx&IU#;-4Sqd9#g_q4z*ja0NfAHcFraHn_Tj8yv6!zM3pPT#L-1p}G zH}}E0AI^Po@{96}@{RJ2@{jV6@{#hA@{{tE@|E(I@|W_M@|p6Q@|)_3`l8;bKkAYC zq+Y3C>Y4hc-l>1;q57y^R_!c>U$w%o)=S}6vlM>S3cs4A@T*q%)hvZywZgAvDg3Gx zel<(sSFP}?Sqi^ugU@iAw%`mLK`)axW>L>sHtK;^5a!U!%!ng1){0k4m$M7=z3{NYz zR$AU4Ja-mz#b5DQd={_8Z}D7w7w^S?@nC!yFUF7YWPBNK#-H(Md>XIDukmbr8}G)y z@o;<`FUQaEbbK9e$KRLxua@@*GtH?7P&vy}O#m3e5EG9R@vFU?Zsr&i{vS;~CX z%Dgp8nZH_@$7U(>Su69}EMud& z@>=p+@?7#=@?P>^@?i2|@?!F1@?`R5@@Dd9@@VpD@@n#H@@(>L@^12P@^JET@^bQX z@^tcb@^TD{erw;g|JsM`$M$9WvwhlrZQr(k+sEza_I3Na zecpa=-?#rS`K$Ld6T*k^BK!zX!k6$S{0Wc3r|_yL-1>n-OH2N0x&JCPVWvK`QZHsH z^`n(~GE1p1t<;-YO6_TdU(HhZRV(~zmcp-E;a9VD)y|oG=}g|VmHg=}C6C%lK6RFo zS8XM~I!np3wvun1rQ}^($-mB0^02MsV`nLO*;ewiv*bMv{zSjkvAjRn-c|Wm{4D+! zzl;CH595#V%lK#fH2xaDjsM1vs~{vN-N|HlvH5AqB7hx|nTBEOOU$dBYt z@+rf1lsa|Ca}l z50DqA>_ADLK)yiUK>k1;K|VoVL4HA=LB2uWLHwe$Wa}7^U!q zR%Nf2{M8*l&QHlx$ydo+E%~c`M;O)!ziNeFjgq{c{GQK~&zH}e&!5ku&!^9;&#%w3 z&$rLJ&%f`3?}zV;?~m`3@0ahJ@1O6Z@2Bsp@2~H(@3-%}@4tGWKByP!hkByEs5k15 zdZa$7SL&B~roO3n>YsY3KB|}Mr+TWss<-N|daORH*Xp-=uD+}H>c4)VKj;_whkl~J z=r{V0ezfGTR`ZE@#r$HPG2fVX%s=KK^O1SU{A8XoUzxYeU*<9MnR(6pW}Y+OnfJ_p z=0WqJdC~l6o-|*YH_e~sQS+&Jb;)0?-cPsB*l+AR_86uk2g)FZ-DN z%)VxSv(MFT*Sc@tv;Wx#?T7Y7`=fo*ereydf7(awr?sn=!mnE4SF=>vt0jN6dpK|A z{F(D;&ZjxA=KPxTY|ghi@8hfZ%@5~~^UJe;j1oVczs_&xzw_hy^Za`LJwKnn&+q5|%LB*<$P361$P>sH z$Q#HX$Ro%n$ScS%$TP?{$UDeC$V13S$VC%Dm3&oY ztxDlnN0t0*|Gxjc|Gs~of4=`c|NH(t{(Szt{`~$v{(kS3cs4A@T*q%)hvZyweTQ(2rt5q@FaW*Z^EDOD0~X9y6HXt<T_q_!l0AkKtwb8J>o(;cfUE9*582b@&~ghwtHi_#YmK58{RRA)bgY;*Izt9=Y6q zwc^uwHGYj}-Et?;W+3cqTFUyV}uRV(~zl)|rC;a8&+e$@)U8l~{7R`}H@xi5f!#n0kz@w@n6 z{4o9)zl?vzPvfug+xTz%IQ|^Jj(^9` z$*wv~AAYr7%K5=o&JS)Y<^15rQqB*qm2!TtmGgsZrOI9{ z`KxP(dZ0e27wU(4qQ0m%>W_M)KB-sgmwKkYsdwt1dZ<3Cm+Gf_s=lhX>aTjNKC9R2 zw|cI=tM}@^exN_-7y5^OqQB@j`j39J+=cp-j>C*q5EBmRg-;*)q~?r(2?nfu#kDfhRxu6)Q%v$XkzH?($L z_mx>X@?-zldd}s4GfV4Zwk~+sCuhm~0?xZS|LQ!f^Rdp$IzQ_?t@E|c+d6;iJg)P( z&g(kA>pZXXz0UhO|LZ)k^TGMPj}m@eEW4EY&!Tk&FV{ku%`x@Ne;64ZUJGk$` z{SV<+%l%iU+&kP4;=T~~hqzB9`@tyjyZPVzaQ-;IoPW+w=dbhI`S1LA{ye{)f6ve7 z?=ShQC4W_!ANe17Ao(D9A^9PBBKabDBl#nFB>5zHCHW1JS&qapDC{?zbVft-zo1Y|0xeDA1W^@KPpcuUn*}Ze=3hE zpDM2^zbelv-zx7a|EeCV&+4`Mt)8pz>b?4}ALtMIh5n(R=r8(>{-Yo1Px_VqrJw0< z%l%iYdB^-?9x@-9m&{M*Df5+i%lu^?GoP8)%x~s7^IhG2Ra)-9TAjbJ4=nEwerME< zXkWBH+9&Oo_D%bzebjzxU$wv5XYIH4UHh+n*nVtZwm;ja?br5g`?r1Eer{j4zuV{S z_x64JKOTS&;05>ro`5gl4fq2dfluHS_ywMUZ{QvH2Offt;3fD8o`SF7E%*x_gU{eK h_zj+e@8CW74<3XM;YIino`f&qP52WYg-_vC{}0KKtDXP= literal 0 HcmV?d00001 diff --git a/tests/acceptance/refs/sine-440.wav.json b/tests/acceptance/refs/sine-440.wav.json new file mode 100644 index 00000000..cf674ea2 --- /dev/null +++ b/tests/acceptance/refs/sine-440.wav.json @@ -0,0 +1,23 @@ +{ + "config_hash": "815cbbe305c6d5b0", + "created_on": { + "arch": "arm64", + "date": "2026-06-16T13:10:20.117+01:00", + "os": "Mac OSX 26.2" + }, + "plugin": { + "format": "VST3", + "manufacturer": "Tracktion", + "name": "pluginval Tone Generator", + "uid": "VST3-pluginval Tone Generator-d86456ee-d8b77bc7", + "version": "1.0.4" + }, + "pluginval_version": "1.0.4", + "render": { + "block_size": 512, + "length_seconds": 0.25, + "num_channels": 2, + "num_samples": 12000, + "sample_rate": 48000.0 + } +} \ No newline at end of file diff --git a/tests/acceptance/refs/square-220.wav b/tests/acceptance/refs/square-220.wav new file mode 100644 index 0000000000000000000000000000000000000000..4032e1dfc30d2fcc5f5da877575beef0961ab2ce GIT binary patch literal 96104 zcmeIzF=|y|7)8;mL9lcP8AN)qFc1S4foN+W*x3baotK3ZaWJ_SX99t3wE~YnO;Q~8 zx8B{&_4T)(ua2)DzI^<2|M~Xnc-hbQA5WL>&Og04ULU`HpTGQlbG$t+kH?3nhvWSJ z=g)f|8))FaHSp)$+Q$YOXdCdo=DS@3eFGe89kZ`}|24oe8)%?!fVaHWK;Hn%ddqBV zW0v)n+1SRcWvvGK23X5l4fGALthda@HfC9GnT>6{2AIoS4fGA{-n{J9K-U1vddqBV zV}^Bx+19pefVHgEK;OXb&C6a5bPceqx6H;iW>{yKZEd>-Sj$=s^bPFZyzJFL*8t0U z%WQ1pHNdjYGTYjA4eZ{$?A1Wm!0yex9?#;_y4Rj6c z-n{J9K-U1nI>T&h+cm(j&M@2Bb`9*_yzJFL*TC-0%U%ui4X~`U%(k{&11#$;v$2hL zZ(jClplg7&tkpo@0K+=NY-`&!z_Q*l8{2sI=4G!2x&~OwS`G9KFsw7owzgdZEbA?^ zv5j|cUiNCBZ-BYX)j;0>%X-UfY-5)7mf6_GtYxhR`UY6bS`G9Ku&lSt#x`bIZ<&p4 z%v#oJpl^V+tkpo@0LyyIY;0qe^_JP##%qAN%+)~O!0yenyXaZP&o=&C6a5bPep@yzJFL*8syh!)$BYHNddWFx%R84eZ{$?A1Wm!0yex9?#;_y4Rj6c-n{J9K;Hn%I?HTp+cm(l-ZC58c=zUIuLimX zSj$=s^bIhqGt9QOT>~uZEwizWcW++yYM^U?wXD@Z-vGlp!)$BYHNdjoG8@}?_vU4< z2Kok=%Ulig4X~`Y%*HlmS#Oz*ZOmHMYM^g`wXD@Z-vG;c%WQ08mi3m|*v717tp@rA zSj$=s^bN4Ax6H;iW?65UjcvRJn9E!Z^bPFZyzJFL*8t0U%WQ08hINM7*0yVawXD@Z p-@xw8%U%t14X~`Y%*HlmSZA1RZMz0o%UTWe4eZ{$?A1Wjz%v^Sk7ED; literal 0 HcmV?d00001 diff --git a/tests/acceptance/refs/square-220.wav.json b/tests/acceptance/refs/square-220.wav.json new file mode 100644 index 00000000..9168e12d --- /dev/null +++ b/tests/acceptance/refs/square-220.wav.json @@ -0,0 +1,23 @@ +{ + "config_hash": "a1c34639b14ec668", + "created_on": { + "arch": "arm64", + "date": "2026-06-16T13:10:20.377+01:00", + "os": "Mac OSX 26.2" + }, + "plugin": { + "format": "VST3", + "manufacturer": "Tracktion", + "name": "pluginval Tone Generator", + "uid": "VST3-pluginval Tone Generator-d86456ee-d8b77bc7", + "version": "1.0.4" + }, + "pluginval_version": "1.0.4", + "render": { + "block_size": 512, + "length_seconds": 0.25, + "num_channels": 2, + "num_samples": 12000, + "sample_rate": 48000.0 + } +} \ No newline at end of file diff --git a/tests/acceptance/refs/square-state.state b/tests/acceptance/refs/square-state.state new file mode 100644 index 0000000000000000000000000000000000000000..5a668c9468c58530b6e31e900cc5c3bf285984fb GIT binary patch literal 12 TcmZQzXs~BBVYX#pV6X=O3RwYo literal 0 HcmV?d00001 diff --git a/tests/acceptance/refs/square-state.wav b/tests/acceptance/refs/square-state.wav new file mode 100644 index 0000000000000000000000000000000000000000..9dcab4c27142462a6ae386160a7c94555364572c GIT binary patch literal 96104 zcmeI*`}6H(dEW7@hIk~3sfl7WR6vMEMFJ@BU5}^{19(EgV@0K?D54@;U~|};jHb~@ ztV$Y9jmF{>@B|q=McM1q7RS_TtQ{*-Q#9BZPxUYeLGdv3y6?ryKOjFpnYm`-0DFJG zYkfZV;eB7v{XG4cqmO>!*>~9Xtj9j{3CEuJl;aNB_Rssjmz=(RpL^v$z0Rqx6Z#KfLwCPd|E;e*C`2wNC!d zaietQc_*~q{+xe3N~fLwZ(0xd+RH}i@jG7Cy6-XDN9hg6oYQ*P+2@VYiC?&&we4xU zM(Mjh-stszvul)Yxq4@7|F4`sO7Huvb6a10>sh08>A|mQ{hvKfAEhtd^QEmzPJH1g z{p|W@w;uJur;XCFpM7F$J;o?~Xs<(ByN~~;_q|8y>}&RDJ?uv}u1a@& z+jXrkp1XTh`pZwfr}cx^y?#~NaoJN_pZ&A@tV)miwU2F;-g^1A*4@td^3|ndU+{t5 zrI-Kyz1F3_ddab^w|wsp)}?bF_3qYQKmWwKbj^dVZSDHgAFfNMz3W$&(zb1Twa)$4 zzg$YoF$cExeZcQ7rL&*CYTfOoqnFZ0zwy-84f{WPDZTN^7qpH&?FZ)L4jYqWZ ze%O9X>EKK5+q&kWdn~2AAFciN{N}p!lV^Ue^^lvdT$g_Ovx{3#zQ?Q9rH`C=XzM9= zyJdIj_;*~`I`eHG_+;sblb*R%I(hdEtyY~$H%fnU^)p)g zz5jWmwEEh~txq2RvQgUSwpX_P^6O`g(q8{`cI&o7&l{!7{^JF$TMykeO7FRGqo4n; zU8D5iukLL9aL)@y>Hog;+}76j&l;th-t(H)hyLy9qx7FId1>pnfBM2vdh6Ss(|Y`C zjvu85z32B^SM2+kQM&0@4sE^hNe>yNKfcp%wchl*zdlMAfAd#c2R!BGRcX2HE3L== zWLcFidCGfRC++=5tJ3Gcc|z-f-`IOqy8i0_vQ_%{Z~jK>)&K4TSC<}g<$HFQe)ZP7 ztxM9sd4zw(fBL$JeDhUG>G*zK4E)T{`3JJ1wQFzH!giXRo{eQo8u) z16t2I;NYcn{j0ZHr|f<7Qd-~tsjZ_w{yp#^V?X=d-Pdj5Nec{^I zwr+akj-|Bq^7C6qY~Q(*9&zce*50@6T1r>!-00c+?_5gb=JQ%x-#=$5edFG*ZJl@O ztC!MGZhU#`7au-lDP8yB=eKtJ_{633nm>GU>xZZO-ctIjcOB8X!=n#cN*~|azqQ{- z@3)lh{eeAOpE&RiOX-X=zP&Ep?{7cfy6ZVttxMZ4`m@#p9=Cm6y6K%qw4V2@ZR^s1 zIPOimOJ8~GL$5Af^`oPl|+kaKM@r5sFo&0+jtxAWz=4^FLb;_|kXRr9Xb!9hcI{pZT@c9sk?jOX-NmJ-GFx zgC4n*e(*;}ww`y?6PD8P$3La@fDb=&DZTfklUgS|`Nd1=6+2FAz2&Q~UP_0(_O-2Z zwstI~eGWRm^`R&4TuLWgysNcs+l9#=7yj~p`{(|B|GhuYpYQMU_xpMLe12X(zu(92 z=lAvd`#gL;J};l2&(r7Y^Y;1sK72pEFW;Z<)A#H9_WgSvJRhDH&yVNH^W}N-{COTd zpPpCGujkqG?Roe7dmp?X-WTtW_sRR^ee?c#AHAR6SMRU)+57E%_x_6q;)8f0euyXH zi+Cgch)3d+cqM*`XX2Z9C;o|t;-h#eeu}5!t9UE^ipS!!crAX5=i<9~FaFC1@`HRK zf5<2Di+m&h$Vc*%d?kO$XY!kTC;!QZ@}qnyf6Axwt9&c}%E$7vd@X;==kmLJFaN6t z)CcMX^@DmseWBh^f2c>)C+ZdTi+VfsE5=?>LvA)dP;qz-co<5$JA%)HT9c% zPJO4|Q~#+4)raau^`m-HeW~75f2v2-r|MPpt9n*_tKL=rs)yCb>Sgt_dRl$0-d2CB z$JOWRb@jV?UVX3LSO4n=^auI{{eyl&f1%&df9OZ_C;AorOZ76X+xr{+j{ZkKq(9Ox z>7Vpd`YZjG{!2fmKhv-2-}H0U+qG`*_w;}ILH(hAQU9o))L-g1^`H7t{i%Lc|EizW z-|BbuzxrYQv3^$mma`f>faeqH~rpV!~(_x1mH06u^h;0JgDzJNF24|oJV zfmh%ccm}?KcifcElz-mJzrQZ!zi;KwTbJ_ZxAOO`OZod-`FYl*{CrC(KW{5P z|5D2D)5`C+l=Azw^7}8Pd>*ZQK1(T|S1X_2Qp)Gq%ICY3@_D!N`7fn>AFX^pODW%1 zE8pKz%J7# zmGXYJ^1iN0d4F4ZpI4>4->tmwt5V+oQA#{$B|eN&;zcX*W0Vq4T8S^Clz7uh{28Uh zqgLY6C?#IC62C?%{A#0XcZ|fhk$Bfi{2Qgj!&c(sC?#ID5N@wAorI!cMRt;F9^ zN<3~QK95r3bu00EloHQdVOOJd)!`%YekA_4k`G2H`Jt73F-plFt>lwcDfy+9e6uPg z|Fn{iR;A>pR`S(WDfz3Fe0D@B`E7S8`L31xw=N|gwvr#$rR2+2^5?pgeA-HWU6+z? zms0X?EBSaSB|o>4ua{EtcPsgPDJ8$RlJA#N@_#G!z*0(m&`Q0qlu|#mQcofKRF{o6`CJW8pLTd9{v zDfM$J_4FvEzHX)79;MXZt<>YAl={4tdVQ2qzqeA)k5cOUR_gsxO8wtTKQKz^4_fIL z!W+*S=^t9@Cq^m#MJxTrD5d{sr5_ok^e3(KE2EVDrImhWl+xd{((jB?`kz+%p;1bI zv?`@vYNdZ#mC{eO(qFAg>9<ew z=?B-P^oOnVi|bPQ$EB2hvX%aFDW%_RrT<(?=|@}XPnS~q)mHk~rIdcQmHu`qrQdC( z|6NMyhg<27ms0xWR{H0qlzzIE{(32;-)^P9fVO5p*m@PVZiUeF3ZSW4jut?-4V6yDGZe^{5oBU<4T>r!|{EBs&q< z9lJ~6AFc3^BTL~UTcz-lRVn2TUMp;msWVpsuVsmO5ruF@S9Nz&uNA4 zj8b?{EBt4a!h>4jL!%U4)CxZurSPOy_|hnaH?_i_Mkze16+Sge;a9D&tK^RhfBC=t zbN{~o-k;~s_xJhx{XBj?Kd+zP@8kFL`}+NT9zGwRm(S1V>GSn@`}}<$z8~M0@6Y$? z`}KYM{yh(#56_F|$MfX*^1ONeJdd7F&#ULx^X&QdynFt=58e;&i}%O-^@{pMJ)^!+@2G#& zL+T^-lKM$KrM^;cslU`?>NEA4`b|BjzEkh1|I~x(L-nHiQ9Y@?RBx(3)uZZD^{V<+ zJ*&P|@2Y>*!|G%8viey)t-e-otH0Ib>T~tF`dvM*zE|(7|Mdg<1O0;jK|i6t&~NBJ z^dtHc{fhpjdYRVk{f&M{|Dzw$AL*C$Px>kSm3~YAr61Fu>DTmc`nl@uTDSLm`ak`k z{!qWDf7DOvFZG-HPyML=RKKc!)z9j0^}G6C{jmO6zpQ`OPwTJs+xl<)xc*$fu7B6h z>+kjZ`hPqCAHWOn13Up=z#H%fJOZDA#x(tLeX*{;T$0l}CVIfPa9WfWLs>fd7CW zfj@y?fq#LYfxm&@f&YOYf^_g8zaagFk~`gMWjcgTEvF^7LO#|5bax z%7ZoiR~tXz^j~fKq2iVJC7y|I;+^;>9*U3RrT8hHR%|Uz|JBCtJpET2|NHb`ZO%hX z|JCMv%=BMv&Pz@I)#m&ho`dh;J@^kEgb(3G_z|9jFX2u26CQ<6;Z>I$eD-A}JPY5# zyYMeO3?IYG@H0FOU&Gt*H#`oX!|U)nJP+T)`|v+J5Ff+~@k2ZjU&I^nM?4ar#4GVj zJQLr5`Gi@6Mhu_6n+){6@C`}7Je807k(K27=9W48Gai6 z8h#u88-5)A9DW`C9ey7E9)2JGAATVIAbuhKA$}tMB7P(OBYq_QBz`6SC4MIUCVnUW zCw?gYD1IsaDSj&cs_DPl`0afEo(Io|=f(5mdGdUD-aLPvN6)9{)${9l_I!KZJ^$VZ z?}zur`{RA`etF-#f8Iy$r}x$S>wWfqd*8kP;(_=eUWgy!iTEPkh(F?y_#|G5U*ehg zCftA190tKZfC>WB5m`epsIep-L6-`0QY$Mxs>b^W`3UVpFO z*Z<=I_yAsjAK(f20^Wc>;1T!)UV&fW8Tba?fq&p3_y}HtpWrF@3f_Xh;4%0NUW4D@ zIrt9Vga6<`_z+%%AK^*(65fPA;ZgV$UX`bN|)cf0g}M zbN|(5Ut0K8?@tTA8l~{7R`^x#TMNJH{cGV@y^n4BucrU1c3{o@SDSsE>KFBl`bNE@ z{!tI9kJL-*r>dn&bN|(5zwF$9wb?hU|I&}?&-826yOrkttLs)_SiRdi3~LpB)%&i) zuX_J=_*L)24!`RC*x^^bFFX9I_h*M+^*-(JtKP32e%1T7!>@Y(cKFpOg@W@c=%QC4-dcUed6I)ycL^Ks70;Wu~=zJvGRKX?#6gcspQcoM#ZH{nls6h4Jlo%xJExwM35;ahkY z{)LC(V|W>UhNt0ccpLtP$Ki8$9e#)B;d^)={)Y$RgLol+h$rHUcq9IZN8*!sC4Px# z;+uFU{)va;qj)KPil^eMcq{&j$KtbiEq;sV;=6b+{)-3W!+0@%j3?vEcr*TtN8{6Y zHGYj}_4y%!F~k$66{Z~Pr-f#`xfk9u#drh z2KyT9Z?Mn7eh2#=?0>Ki!hQ(*BJ7W_Pr`l)`zGw4u#dui3i~STudvU;ehd39%wO%p zuph&|4Er?5(C#J&>yOYAeT-^9KX`%lc{&F9VQ&F{_g&G*gw&HwoU_yhO__y_n2_zU>^E`S!J+Gc$&$H*-^X~cgK6pR8FWw*TllRN}=Kb?N zdOy9d-e2#t_uKpK{TC0!2k}Du5KqJx@kaa+kHjbOO8gSf#5eIy{1XqwNAXhp6i>xh z@mBm5kHu&4TKpEz#dq;u{Fe{p2l+z&kWb_n`9}VckK`x$O8%11oNBL6z zluzYX`BwgwkL73iTK<;L<#+jB{#OsE57Z0l2la&dLcO8>P>-lj)GO*2^^E#Py`%n7 z52=sTOX?@}l=@1&rT$Wnsn67F>NoYA`cA#4{!`T@^>L>M=`c3_( zepG*|U)8_rXZ5%GUHz|qSbwZv)<5f~_1F4s{kMKxf39EGzw77q_xgSPKOTS&;05>r zo`5gl4fq2dfluHS_ywMUZ{QvH2Offt;3fD8o`SF7E%*x_gU{eK_zj+e@8CW74<3XM z;YIino`f&qP52WYg-_vCmmK`#14?)nzJ+(;Uw9ZkhL_=IcpAQjx8ZMi96pEF;dgi* zzK8eWe|R81h!^6Acp|=tH{y?YBtD5(;+J?PzKM6@pLi%fikIT2cq+b%x8kpOY_Zu= zvD#9e|9Su5^k2>U2j~5R^Zvnk|6t!em>#)zU!`a6y9d)l&-(}G{e!&&EBvbOM-0E} z`x3*i`u@c5tG-V${HpI)48Q987Q?Ul{>AXCzK=2ds_$nEzv}xM!>{`O#_+4Y&oTU} z?{^Hp>iZtUuloMS@TU-kW{;a7cMYWP*( zpBjGE_o;?o_5G^hSAE}V_*LJ(8h+LHv4&su{jA|veP3(%Ro~wle$_h#!>{K4tK9J$ ze$|@$ujc-%x`S}uKe)LMe)_LozY4=zg-^F|J zUpyEe#*6V|JQ-icoAGBn8lT3i@oPLA-^RP~Z#*0y$IJ2a>A#xy56=4s-`RH$){ND> z)%?{w)_m5y*8J8y*L>H!*ZkK!*nHT$*!fd7CWfj@y? zfq#LYfxm&@f&YOYf^_g8zaagFk~`gMWjcgTI5{ga3mcgg=B|gnxve zgujH}g#Uyeg+GN~g@1*gg};U0h5v;ghCha1hJS{ihQEg2hW~~ihd+m3hku8khrfs4 zhyRBkh(Cy5h<}Kmh`)&6i2sNmi9d;7iGPWoiNA^8iT{Zoia&~9ihqiqioa^^znc56 z>YexgiwEL^cp-j>C*q5EBmRg-;*)qKeu-z|n|LSwiHG8&cqx90r{b%4EB=bd;PIs|Tk4YI8sJynk@hFU*w|N`hERB9)J(v1^5A;fG^+;_yZn+Pv8~! z1)hO#;2rn}9)geHCHM)Rg0J8$_zNC`&)_xq4W5JV;63;c9)u6!Mfee(gfHPu_!Az5 zPvKR$docU2-aY+Sx1T4)H}OvV6A#5l@lyO0PsLa9R{Rx@#b@za{1(qGwp+^k(#rd@ zD&>9JD&_rZ<$b%ll=p9UDevRDl=rih_jO&$``gOUoEBZt5(hrE~V^mZ)Jb`Qp*1Jjjr9X>|)lv$-YHizgVe;yQzx?0+xqsh(@6Yq+`}_R;ejY!cpV!at_woDref|DE z51)_E%jf6w^!fU{eg3`=-;eLh_vicc{rbLr|DFfWhv&uf<9YIYdEPvKo=4B8=hgG; zdG>sJ-aY@`2k(dX#rxxZs&}gtUfe40W2wB5rSe9W!kb&+&!ZF`-3p%`rLe13_|+(d zU$w%oMk)NN6@E2J(|>i!5{9*eU$w%omQwgtEBtCHgA%|4Tk0?MnEFh;rhZe;sqfT#>Ob|M`cS>7 zepFAYFV&msPxYw!RK2QxRnMw#)w}9n^|1O_y{vv#Pphxz{;RqFs`_pHw|-oIu3y)` z>*w|N`hERB9)J(v1^5A;fG^+;_yZn+PfY*SXxh@mBm5kHu&4TKpEz#dq;u{1*?#hw)4fZ$K=U~5seGm3O*au-hgnbeAN7yG}zl41g_D|SHVLyd^74}!yXJNi- z-fI489&0{pUTc19o@>5q-fRAA9&A2rUTl7Bo@~Bs-faGC9&J8tUTuDDo^8Hu-fjME z9&SEvUT%JFo^HNw-fsSG9&bKxUT=PHo^QTy-f#ZT55OP5FTg**PrzTmZ@_=RkHDY6 zufV^+&%ocn@4)}S55XV7FTp>-Pr+ZoZ^3^t{Z|`5&h%gH9OcpC*W%ye=i=|;_u~KJ z2jdUp7vmq}C*v>UH{(C!N8?Z9SL0vfXX9_t+wtG=+$dL^YQob`|}G8}c9WBl0KmEAlV$Gx9g`JMuqW#0&95JP}{S8}Uax5}(8?@k=}t-^4rdPdpSK#Y^#1JQZKXTk%&s z7N5my@mo9>-^F|JUp|l@_Pd=0%MQk@`b#~gK2xu$-_&#JyQ=j{bN|(*{?`xa5A+Mue>MGA#SriWd;xF3AMgl# z0iz8G9a3Dwb8sKX2vVUzPITxANz$O8N6!`TJI-{Qa%`JgZWEzER50+se;B zO8I?S`Ta&Izi%tQ|0w12Xyx-6rF>qke14;p&$E@!xA)`c^X`55`TTo-e!h?1r=Rbq z_v`2T>V5n9{(Ap@zR%vrpYON#^XL2Sef|0Vdw)NkiErYa_$MBUkK(2HDV~b2;;r~A z9*fW7wfHTbi|^vS`0w;zZRP><{=wY`$#XZ2oK>Z9Z*YZGLT@ zZN6>ZZT@W@Za!YK@>2SbR{D{pl>VfZeq|}8e`%$k={pG1-}D`X>35b=`kz+%p{11m zXkALb)Jp%fE~TGprN3$qR{E{p*OLCL_qU`U>wPZi&w9U0`nBHolK!ptzoeh*eK6_o zdOuA1z1|m-{;&7Pq#x{kGU*R{zfAhY-Zzu}vG>oUpX_}!=`VXf4Zj5c1V06T1-}LV z1wRIV2EPXX20sUXM|$PkogYbv&i|~)|lkk`DoA96TqwuHjtMISzv+%d@yYRp8 z!|=!O%ka+tXJ^YHiZ`|$tp1Mvs(3-J%}6Y&@E8}T3UBk?Ek zEAcP!Gx0a^JMlm9L-9xPOYu+fQ}I_#|JB@oRqwp_Upx>W#0&95JP}{S8}Uax5}(8? z@k=}t-^4rdPdpSK#Y^#1JQZKXTk%&s7N5my@mo9>-^F|JUp|l@_Pd=0%U z|Hw!3lYAwA$!GGLd?)|Ohw`I*DSyhR@~eC+|H{YmvwSUo%jfdDd@uj22h<1Z1@(h^ zLVcm$P=BaL)FNWM7dQN?(-c$dn z2i1pFE0%J8u$A+JODX3ETRA_tlyZKsmGgs3Ddz`UIX}3Ra(=Lt^Mgw%=LcIkKe&`~ zez2AEgG(vr2U|HmxRi2!uoZT-l)|sprJNsZWq13!l>P0k?_Kxib?J(Cp3*w~(+^pf zPTqaP?$Qw_J+tM!h4UBFfAxxyoifAy^!i2Tcbw;OzQ=hV=YO0Baz4m;A?JsjCvv{X zc_ZhKoJVp#$$2H`mz-yEzR7tf=bxO1az4s=Dd(r0r*gi^c`N6yoX2uL%XuyQ08hXd z@CN(=kH9DJ3j6}kz&G#?`~wfcNAME-1W&T_q_!l0AkKtwb8J>o(;cfUE9*56O|J8=qPXE=; zr5HY*kMHCC_`i9;^j}T?)%0Jz;FtS4%#Y2J&6mxa&7aMq&8N+)&9BX~&9}|F&A-jV z&Bx8l&Cku#&DYJ_&EL)A&F9VQ&F{_g&G*gw&HwoU_yhO__y_n2_zU}Aip60AU`30A-^I2AwMF2BEKU4B0nR4Bflg6V|eo1e-%%x z_$*$F-{QIWF5Zj(@`3yyU&tTwiTonp$UpLt{3Kt=U-Fs!Cf~__@}c}FU&^1;e>MGA z(|@&n(O>De^k4ch{h5Ak`mc7cs)u);%K0kit(?Dd9?SVG=e3;Qa-PfiF6X_R|8gG8 z`7r0joF8+Z%=t3s&741T9?khQ=hd8FbDqulHs{@(e{&wr`8en0oS(yU@EyDd|G|Us zA-o7b!jteNya|88qwp!b>dQCX^57Dlg>T_q_!l0AkKtwb8J>o(;cfUE9*582b@&~g zhwtHi_#YmK58{RRA)bgY;*Izt9*IxlmG~u|iErYa_$MBUkK(2HDV~b2;;r~A9*fW7 zwfHTbi|^vS_%9xe597u7F`kSsH!+ucP#LS8Q*6?8aA((pOJ? zdFu}MJ7tu1-ud~h6OK4>lpgfgPipP^hR2WARfms#qw~fiT6aHezfn5)lKZxd6)T@d6@Z_d71f{d7Al}d7Js0d7Sy2d7b&4 zd7k;6d7t^8d7$~Ad7=5Cd7}BEd87HGd8GNId8PTKd8YZMd8hfOd8qlQd8zrSd8+xU zd8_%Wd93-Yd9C@ad9L}cd9V4ed9eAgd9nGid9wMkd9(Smd9?YodA0eqdA9ksdAIqu zdARwwdAa$ydAj+!dAs?$dA#|&dA<3)dA|9+dB6ETKLCFKzX1OLKLLLMzXAUNKLURO zzXJaPKLdXQzXSgRKLmdSzXbmTKLvjUzXkur^k2>WSLMm#&*InO-{R-u@8b93|KbPZ z591f(ALA$EFXK1kKjTN^PvckPU*l)vZ{v64f8&SakK>o)pW~+1?^DFUL zycWO3bMalg7yso0`9Z#rKjahnMZS@LzLLM>Gx<%vlmFyH`BA=I3zH`awORzEE$dKhz`Y6ZMMvMLnawQSYdK)I;hc^^*EY zJ*B=hi3W9l>Yn)*#Wr@m9~ssGf2>O=LS`cXZpzEp3jKh>k^Q}wF)RXwY|Rqv{Q z)x+vz^|Ja|J*~c0Z>zu6M!-1`cM6+{#3uJf7Q?GZ}q$SU;VKDSih`))=%rN_1pSy{kZ;Izpj7R&+G5?`}%)8 z03W~$@B=&nU%(si2Rs6wz$@?zJOkgrJMa%Y1Rudm@Dn@*U%^}O7d!@^!E5jvJO|&w zd+;AT2p__W@FP44U&5R4Cp-$D!mCcb*Kr4y@GN`_@4~I67QG9|5ozBQc8Yk zC0{J147MYIiC5tCf8A$x`y$Rw?`;ezm(4esy&z{A#Nde$@)UT9v}DTH#l#QutLX{AyJSziNeFtxDlnqZEGC z3cnhq@T*pN+e_hBt?;Y9|2h0>l)|rC;a7cMbof=@A02+x_eqCe_5IS}SECeu)e67r z`>4aO`hM#0tG=&#-aokMAM_LY3;l-vLqDQF(XZ%Vs+Vcq-rs~@Ev4|QR`}IY3cqTF zUoEBZt5*2cQkwp&ZChbjTlP)cKW!hi{nYkVJ3lr3SDSt2_zvEK|KLIR5MG2I;Ys)s z-h@BlQTP;Ib=+--TwcPn@GZOx|H8xYF}w^v!_)9JybXWDQ}H_bcEKg~nUN6kyk zPt8-!SIt|^U(I99XU%KPZ_RVfcg=gvf6arL}C&HYyg4e?(-kRRj= z`9nUDU*sG4M?R9DI?OT`a?aUK2fizU(_?|8}*L*M?IuIQZK2W)Kls!^_KceJ*GZWuc_bE zbLzY4zuMIM(|@%&f1@ANALR0uz`dR(0epmmiAJ!l1m-Wy3 zY5lc+TmP*e*PrXx_3!$5{k?u)|BnaY19$;`fG6M!cmw``N8l591%82N;2U@c{(*geT!kcoY7FN8wX=)x&@AzynKo z7QTgd;a_+dK8Ba!XLuUEhPUBwcpN@A{a17URoz+P{Iv7b&R08c?fkX#*yaP~1?C6l z3FZst4dxH#5z~M5j1iCZh~K&u|Me)vkKKwtdz9kWZpFVnO7U~I;_n`%_`O@^XXa_< zYvyg{Z{~64bLMsCcjkHKd**%Sf98SagXV?ihvtdqi{_2ykLHo)ljfD?m*$z~o93P7 zpXQM_yp{TVDWzU-rG8&Zspng%@0U{Q{Z{J#rIdc4mHuEUrC(^Je^^TCCtB$* zmQwnSdH>+Czcc3kt6NsJ!^l1(`;F{7vj50FB>o)xlI%~iPsx5I`NWM7dQN?(-c$dn2i1q_MfIb4Qhll3RDY^R)u-xJ^{aZeYTHuI z54LiCaFlX>u$A+Jqm=W5t(+emrP_Tp_g_u_Re7(b|7!bi-p%b6(E*IXnm7 z!F%u@JP04ci|`{n317mS@FzS9pTev5`s!USFX37E7T$$_;bHg~UWT9HY4{r6hQHx) z_#9q`-{E=q9^Qxl;eq%dUWgy!iTEPkh(F?y_#|G5U*ehgCfG(R{KK)l0Zq*FNd3fjJotJlh-g$cG>z%iE{@!_f=kuM{cYfb_e&_q0_jmr^J^=dx z>EJk@;Fyw&{GJl1^Hyw?2I zJlA~Jyx08KJlK5Lyx9EMJlTBNyxIKOJlcHPyxRQQJllNRyxaWSJluTTyxjcUJl%ZV zyxshL`mau2%8xPcAKYhMJCf{6vOmc_CHs}^Tbll>7mxCQ@q_V)@r&_~@sshF@tg6V z@uTsl@vHH#@w4%_@w@r_eILFb-%czn*=2 z_V3xpXFs2PefIa+=V!m4eSh}<*#~GppnZY%2ihlSzo31C_7C#I^2hSa^3U?q^4E&@ z;=g<#KgbvIhkPQx$T#wjd?Y`~SMry9Ccnvd@}GPtKgyT#r+g~E%D3{bd@Mi9*YdY~ zF2Bq7^1pgOeV|@YKd2|v7wQf5hk8VPqFzzIsAtqS>K*ludPsewUQ$1)r_@*KE%ldr zOns(aQ@^R_)OYGV^`ClBeW+ekKdL9ym+DRRr+QRRt7(dRTp|URFP= zr`6Z$ZS}W$Tz#%ySHG+0)%WWC>A#x(t6~6n06u^h;0JgDzJNF24|oJVfmh%ccm}?K zciGv71sGygLWG#@lCG(R*?G+#7tG=DUYG@mrDG`}>@G~YDuH2*XY zH6JxEH9s{^HD5JvHGegaHJ>%FHNQ2_o&KwD_Z?T}{pSDt0Q>>`0{jE~1pEd32K)#7 z2>c2B3j7QF4EzoJ4*U=N5d0DR64QS*{a3XEYwo|A`>&3F-7AOtMZ>E5zESs&x{uWT zr0y#fKg1L9MZ6Jz#3S)Zyb`~}Gx1Hl6aU0R@lm`KKgCn=RlF5{#pCI}n*OV*(bQ|| zH}#zQPQ9o8QxB>S)r;y!^`!bzy{Z0GkE&1AtLj(v?DSu4_WL?-;{1v8D9)!iuj2fQ z^DNG{IPc>8i}Ntf$2c$J{7jvtDP?#2Mu(iWWM|A0|3WK%hNTpLLo0rVr4;`|D}IQj z6n{i3eu?Pi7iT|P%KgPNge?}{QjddyhjaK{|>r(t3t@u6G zrT9N~m*NMxx-|V)8@__K;4gR#K7-fbH+T-dgZJP+co05>7vV>E6262t;ZJxJK806( z=(V4?tb}LbTX+}#g@@r|co}|%r{QaO8~%pJ;d6K$euwAbdw3uIhX>+=cp-j>C*q5E zBmRg-;*)qKeu-z|n|LSwiHG8&cqx90r{b%4>-1mE`v<50>fD?~JF8_rXI^K1XP#%i zXWnQ2XC7!iXkKW3Xr5@kXx?c4XdY=kXmP6z^}l+ zz|X+n!0*8Szz@M6!7sr-!B4?o!EeETQQoWRzj|(abEf}l=ddr!{w({n?ANky%l@tD zzdCfOI|%3fgI`-$j1sTJFY!!#6Ys=7@lbpeFU3#sRD2b0#b5DQd={_8Z}D7w7w^S? z`9OY?7y`X+jPpB`{8|n}Bi26jmqJB}&sBhFe>L2xx`bfQ`eo{}Vuhd)WFZG!E zOueRlQ_rdI)O*u^wK;z=?;qUUNAG-&^E%G&IM3sJkMlmx|2Plie30`(&JQ_H:AU> + IS_SYNTH FALSE + NEEDS_MIDI_INPUT FALSE + COPY_PLUGIN_AFTER_BUILD FALSE) + +target_sources(pluginval_gain PRIVATE + GainPlugin.cpp + GainPlugin.h) + +target_compile_features(pluginval_gain PRIVATE cxx_std_20) + +target_compile_definitions(pluginval_gain PRIVATE + JUCE_WEB_BROWSER=0 + JUCE_USE_CURL=0 + JUCE_VST3_CAN_REPLACE_VST2=0) + +target_link_libraries(pluginval_gain PRIVATE + juce::juce_audio_utils + juce::juce_audio_plugin_client + juce::juce_recommended_config_flags + juce::juce_recommended_warning_flags) diff --git a/tests/test_plugins/gain/GainPlugin.cpp b/tests/test_plugins/gain/GainPlugin.cpp new file mode 100644 index 00000000..fb3af1de --- /dev/null +++ b/tests/test_plugins/gain/GainPlugin.cpp @@ -0,0 +1,56 @@ +/*============================================================================== + + Copyright 2018 by Tracktion Corporation. + For more information visit www.tracktion.com + + You may also use this code under the terms of the GPL v3 (see + www.gnu.org/licenses). + + pluginval IS PROVIDED "AS IS" WITHOUT ANY WARRANTY, AND ALL WARRANTIES, WHETHER + EXPRESSED OR IMPLIED, INCLUDING MERCHANTABILITY AND FITNESS FOR PURPOSE, ARE + DISCLAIMED. + + ==============================================================================*/ + +#include "GainPlugin.h" + +//============================================================================== +GainProcessor::GainProcessor() + : juce::AudioProcessor (BusesProperties().withInput ("Input", juce::AudioChannelSet::stereo(), true) + .withOutput ("Output", juce::AudioChannelSet::stereo(), true)) +{ + // Default gain of 1.0 passes the input through unchanged. + addParameter (gain = new juce::AudioParameterFloat ("gain", "Gain", + juce::NormalisableRange (0.0f, 1.0f), 1.0f)); +} + +//============================================================================== +void GainProcessor::processBlock (juce::AudioBuffer& buffer, juce::MidiBuffer&) +{ + const float g = gain->get(); + + for (int c = 0; c < buffer.getNumChannels(); ++c) + buffer.applyGain (c, 0, buffer.getNumSamples(), g); +} + +//============================================================================== +void GainProcessor::getStateInformation (juce::MemoryBlock& destData) +{ + // getValue() is a private override on the concrete type, so go via the base. + juce::MemoryOutputStream mos (destData, false); + mos.writeFloat (static_cast (gain)->getValue()); +} + +void GainProcessor::setStateInformation (const void* data, int sizeInBytes) +{ + juce::MemoryInputStream mis (data, (size_t) sizeInBytes, false); + + if (mis.getNumBytesRemaining() >= (int) sizeof (float)) + gain->setValueNotifyingHost (mis.readFloat()); +} + +//============================================================================== +juce::AudioProcessor* JUCE_CALLTYPE createPluginFilter() +{ + return new GainProcessor(); +} diff --git a/tests/test_plugins/gain/GainPlugin.h b/tests/test_plugins/gain/GainPlugin.h new file mode 100644 index 00000000..27446fac --- /dev/null +++ b/tests/test_plugins/gain/GainPlugin.h @@ -0,0 +1,68 @@ +/*============================================================================== + + Copyright 2018 by Tracktion Corporation. + For more information visit www.tracktion.com + + You may also use this code under the terms of the GPL v3 (see + www.gnu.org/licenses). + + pluginval IS PROVIDED "AS IS" WITHOUT ANY WARRANTY, AND ALL WARRANTIES, WHETHER + EXPRESSED OR IMPLIED, INCLUDING MERCHANTABILITY AND FITNESS FOR PURPOSE, ARE + DISCLAIMED. + + ==============================================================================*/ + +#pragma once + +#include + +//============================================================================== +/** + A minimal, fully deterministic gain effect used to dogfood the acceptance + feature's audio-input path. + + Unlike the tone generator (a pure generator), this is an effect: it + multiplies its input by a single linear gain parameter. Feeding it a known + input file and comparing the gained output exercises the input.audio render + path. Multiplying by an exact-in-float gain (e.g. 0.5) keeps it bit-exact. +*/ +class GainProcessor : public juce::AudioProcessor +{ +public: + //============================================================================== + GainProcessor(); + ~GainProcessor() override = default; + + //============================================================================== + void prepareToPlay (double, int) override {} + void releaseResources() override {} + void processBlock (juce::AudioBuffer&, juce::MidiBuffer&) override; + + //============================================================================== + juce::AudioProcessorEditor* createEditor() override { return nullptr; } + bool hasEditor() const override { return false; } + + //============================================================================== + const juce::String getName() const override { return "pluginval Gain"; } + bool acceptsMidi() const override { return false; } + bool producesMidi() const override { return false; } + bool isMidiEffect() const override { return false; } + double getTailLengthSeconds() const override { return 0.0; } + + //============================================================================== + int getNumPrograms() override { return 1; } + int getCurrentProgram() override { return 0; } + void setCurrentProgram (int) override {} + const juce::String getProgramName (int) override { return "Default"; } + void changeProgramName (int, const juce::String&) override {} + + //============================================================================== + void getStateInformation (juce::MemoryBlock&) override; + void setStateInformation (const void*, int) override; + +private: + //============================================================================== + juce::AudioParameterFloat* gain = nullptr; + + JUCE_DECLARE_NON_COPYABLE_WITH_LEAK_DETECTOR (GainProcessor) +}; diff --git a/tests/test_plugins/tone_generator/CMakeLists.txt b/tests/test_plugins/tone_generator/CMakeLists.txt new file mode 100644 index 00000000..16a1e7c8 --- /dev/null +++ b/tests/test_plugins/tone_generator/CMakeLists.txt @@ -0,0 +1,32 @@ +# A minimal, fully deterministic tone-generator plugin used to dogfood the +# acceptance-testing feature. Built only when PLUGINVAL_BUILD_TEST_PLUGINS is ON. + +juce_add_plugin(pluginval_tone_generator + PRODUCT_NAME "pluginval Tone Generator" + COMPANY_NAME Tracktion + BUNDLE_ID com.Tracktion.pluginvalToneGenerator + PLUGIN_MANUFACTURER_CODE Trkt + PLUGIN_CODE Ptg1 + FORMATS VST3 $<$:AU> + IS_SYNTH FALSE + NEEDS_MIDI_INPUT FALSE + VST3_CATEGORIES Generator + AU_MAIN_TYPE kAudioUnitType_Generator + COPY_PLUGIN_AFTER_BUILD FALSE) + +target_sources(pluginval_tone_generator PRIVATE + ToneGeneratorPlugin.cpp + ToneGeneratorPlugin.h) + +target_compile_features(pluginval_tone_generator PRIVATE cxx_std_20) + +target_compile_definitions(pluginval_tone_generator PRIVATE + JUCE_WEB_BROWSER=0 + JUCE_USE_CURL=0 + JUCE_VST3_CAN_REPLACE_VST2=0) + +target_link_libraries(pluginval_tone_generator PRIVATE + juce::juce_audio_utils + juce::juce_audio_plugin_client + juce::juce_recommended_config_flags + juce::juce_recommended_warning_flags) diff --git a/tests/test_plugins/tone_generator/ToneGeneratorPlugin.cpp b/tests/test_plugins/tone_generator/ToneGeneratorPlugin.cpp new file mode 100644 index 00000000..ecc0565e --- /dev/null +++ b/tests/test_plugins/tone_generator/ToneGeneratorPlugin.cpp @@ -0,0 +1,114 @@ +/*============================================================================== + + Copyright 2018 by Tracktion Corporation. + For more information visit www.tracktion.com + + You may also use this code under the terms of the GPL v3 (see + www.gnu.org/licenses). + + pluginval IS PROVIDED "AS IS" WITHOUT ANY WARRANTY, AND ALL WARRANTIES, WHETHER + EXPRESSED OR IMPLIED, INCLUDING MERCHANTABILITY AND FITNESS FOR PURPOSE, ARE + DISCLAIMED. + + ==============================================================================*/ + +#include "ToneGeneratorPlugin.h" + +namespace +{ + constexpr double twoPi = 2.0 * 3.14159265358979323846; +} + +//============================================================================== +ToneGeneratorProcessor::ToneGeneratorProcessor() + : juce::AudioProcessor (BusesProperties().withOutput ("Output", juce::AudioChannelSet::stereo(), true)) +{ + // Parameter indices are stable (0, 1, 2) so acceptance configs can address + // them by index as well as by name. + addParameter (waveform = new juce::AudioParameterChoice ("waveform", "Waveform", + juce::StringArray { "sine", "square" }, 0)); + addParameter (frequency = new juce::AudioParameterFloat ("frequency", "Frequency", + juce::NormalisableRange (20.0f, 20000.0f), 440.0f)); + addParameter (gain = new juce::AudioParameterFloat ("gain", "Gain", + juce::NormalisableRange (0.0f, 1.0f), 0.5f)); +} + +//============================================================================== +void ToneGeneratorProcessor::prepareToPlay (double sampleRate, int) +{ + currentSampleRate = sampleRate; + sampleIndex = 0; // phase resets so renders are reproducible +} + +void ToneGeneratorProcessor::processBlock (juce::AudioBuffer& buffer, juce::MidiBuffer&) +{ + buffer.clear(); + + const auto numSamples = buffer.getNumSamples(); + const auto numChannels = buffer.getNumChannels(); + const bool isSquare = waveform->getIndex() == (int) Waveform::square; + const double freq = (double) frequency->get(); + const float g = gain->get(); + + if (numChannels > 0) + { + auto* channel0 = buffer.getWritePointer (0); + + for (int s = 0; s < numSamples; ++s) + { + const double phase = twoPi * freq * (double) (sampleIndex + s) / currentSampleRate; + + float value; + if (isSquare) + { + // Closed-form square: +1 for the first half of each cycle, -1 for the second. + const double fractional = phase / twoPi - std::floor (phase / twoPi); + value = fractional < 0.5 ? 1.0f : -1.0f; + } + else + { + value = (float) std::sin (phase); + } + + channel0[s] = value * g; + } + + // The tone is mono; copy it to every other output channel. + for (int c = 1; c < numChannels; ++c) + buffer.copyFrom (c, 0, buffer, 0, 0, numSamples); + } + + sampleIndex += numSamples; +} + +//============================================================================== +void ToneGeneratorProcessor::getStateInformation (juce::MemoryBlock& destData) +{ + // A tiny, explicit binary state: the three parameters' normalised values. + // This is the blob that a state.file acceptance test restores. getValue() is + // a private override on the concrete parameter types, so go via the base. + const auto normalised = [] (juce::AudioProcessorParameter* p) { return p->getValue(); }; + + juce::MemoryOutputStream mos (destData, false); + mos.writeFloat (normalised (waveform)); + mos.writeFloat (normalised (frequency)); + mos.writeFloat (normalised (gain)); +} + +void ToneGeneratorProcessor::setStateInformation (const void* data, int sizeInBytes) +{ + juce::MemoryInputStream mis (data, (size_t) sizeInBytes, false); + + if (mis.getNumBytesRemaining() >= (int) (3 * sizeof (float))) + { + waveform->setValueNotifyingHost (mis.readFloat()); + frequency->setValueNotifyingHost (mis.readFloat()); + gain->setValueNotifyingHost (mis.readFloat()); + } +} + +//============================================================================== +juce::AudioProcessor* JUCE_CALLTYPE createPluginFilter() +{ + return new ToneGeneratorProcessor(); +} diff --git a/tests/test_plugins/tone_generator/ToneGeneratorPlugin.h b/tests/test_plugins/tone_generator/ToneGeneratorPlugin.h new file mode 100644 index 00000000..04a6033d --- /dev/null +++ b/tests/test_plugins/tone_generator/ToneGeneratorPlugin.h @@ -0,0 +1,76 @@ +/*============================================================================== + + Copyright 2018 by Tracktion Corporation. + For more information visit www.tracktion.com + + You may also use this code under the terms of the GPL v3 (see + www.gnu.org/licenses). + + pluginval IS PROVIDED "AS IS" WITHOUT ANY WARRANTY, AND ALL WARRANTIES, WHETHER + EXPRESSED OR IMPLIED, INCLUDING MERCHANTABILITY AND FITNESS FOR PURPOSE, ARE + DISCLAIMED. + + ==============================================================================*/ + +#pragma once + +#include + +//============================================================================== +/** + A minimal, fully deterministic tone-generator plugin used to dogfood the + acceptance-testing feature. + + The output is computed in closed form from a sample index that resets to 0 + on prepareToPlay(), so the same configuration always renders the identical + buffer (ideal for the bit-exact / sample comparator). There is no randomness + and no denormal-sensitive feedback path. Audio input and MIDI are ignored - + it is a pure generator. +*/ +class ToneGeneratorProcessor : public juce::AudioProcessor +{ +public: + //============================================================================== + enum class Waveform { sine = 0, square = 1 }; + + ToneGeneratorProcessor(); + ~ToneGeneratorProcessor() override = default; + + //============================================================================== + void prepareToPlay (double sampleRate, int maximumExpectedSamplesPerBlock) override; + void releaseResources() override {} + void processBlock (juce::AudioBuffer&, juce::MidiBuffer&) override; + + //============================================================================== + juce::AudioProcessorEditor* createEditor() override { return nullptr; } + bool hasEditor() const override { return false; } + + //============================================================================== + const juce::String getName() const override { return "pluginval Tone Generator"; } + bool acceptsMidi() const override { return false; } + bool producesMidi() const override { return false; } + bool isMidiEffect() const override { return false; } + double getTailLengthSeconds() const override { return 0.0; } + + //============================================================================== + int getNumPrograms() override { return 1; } + int getCurrentProgram() override { return 0; } + void setCurrentProgram (int) override {} + const juce::String getProgramName (int) override { return "Default"; } + void changeProgramName (int, const juce::String&) override {} + + //============================================================================== + void getStateInformation (juce::MemoryBlock&) override; + void setStateInformation (const void*, int) override; + +private: + //============================================================================== + juce::AudioParameterChoice* waveform = nullptr; + juce::AudioParameterFloat* frequency = nullptr; + juce::AudioParameterFloat* gain = nullptr; + + double currentSampleRate = 44100.0; + juce::int64 sampleIndex = 0; // resets to 0 on prepareToPlay for determinism + + JUCE_DECLARE_NON_COPYABLE_WITH_LEAK_DETECTOR (ToneGeneratorProcessor) +}; From cb4d4cee854770cace8446923bf28a0ed1669bc6 Mon Sep 17 00:00:00 2001 From: David Rowland Date: Tue, 16 Jun 2026 16:01:49 +0100 Subject: [PATCH 06/21] Acceptance testing: Added playhead support --- CLAUDE.md | 33 +++++---- CMakeLists.txt | 1 + docs/Acceptance testing.md | 1 + source/CommandLineTests.cpp | 49 +++++++++++++ source/acceptance/AcceptanceTest.cpp | 53 +++++++++++++- source/acceptance/TestConfig.cpp | 28 +++++++ source/acceptance/TestConfig.h | 12 +++ tests/acceptance/Acceptance testing design.md | 26 ++++--- tests/acceptance/CMakeLists.txt | 9 ++- tests/acceptance/playhead-120.json.in | 15 ++++ tests/acceptance/refs/playhead-120.wav | Bin 0 -> 96104 bytes tests/acceptance/refs/playhead-120.wav.json | 23 ++++++ .../playhead_probe/CMakeLists.txt | 33 +++++++++ .../playhead_probe/PlayheadProbePlugin.cpp | 55 ++++++++++++++ .../playhead_probe/PlayheadProbePlugin.h | 69 ++++++++++++++++++ 15 files changed, 379 insertions(+), 28 deletions(-) create mode 100644 tests/acceptance/playhead-120.json.in create mode 100644 tests/acceptance/refs/playhead-120.wav create mode 100644 tests/acceptance/refs/playhead-120.wav.json create mode 100644 tests/test_plugins/playhead_probe/CMakeLists.txt create mode 100644 tests/test_plugins/playhead_probe/PlayheadProbePlugin.cpp create mode 100644 tests/test_plugins/playhead_probe/PlayheadProbePlugin.h diff --git a/CLAUDE.md b/CLAUDE.md index 58b11511..d6f27fef 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -114,7 +114,8 @@ pluginval/ │ ├── AddPluginvalTests.cmake # CMake module for CTest integration │ ├── test_plugins/ # Test plugin files │ │ ├── tone_generator/ # Deterministic dogfood generator (PLUGINVAL_BUILD_TEST_PLUGINS) -│ │ └── gain/ # Deterministic dogfood gain effect (for the input.audio path) +│ │ ├── gain/ # Deterministic dogfood gain effect (for the input.audio path) +│ │ └── playhead_probe/ # Writes the host transport to output (for the playhead path) │ ├── acceptance/ # Acceptance self-tests: configs (*.json.in), inputs/, checked-in refs/ WAVs │ ├── mac_tests/ # macOS-specific tests │ └── windows_tests.bat # Windows test scripts @@ -286,25 +287,27 @@ spec: `tests/acceptance/Acceptance testing design.md`; end-user guide: layering: the test config is a positional argument loaded standalone. - **Flow** (`AcceptanceTest.cpp`): load plugin → apply `state.file` then `state.parameters` (normalised, matched by index / case-insensitive name or - paramID) → feed `input.audio`/`input.midi` or silence → render a fixed - duration block-by-block (reusing the `AudioProcessingTest` shape + the - VST3-safe helpers in `TestUtilities.h`). If no reference exists it **records** - one (32-bit float WAV + `.wav.json` sidecar manifest); otherwise it - **compares** and writes a diff WAV on failure. Exit `0`/`1`. + paramID) → feed `input.audio`/`input.midi` or silence → if a `playhead` is + configured, point a fixed-tempo transport (`FixedPlayHead`, position advances + per block) at the plugin → render a fixed duration block-by-block (reusing the + `AudioProcessingTest` shape + the VST3-safe helpers in `TestUtilities.h`). If + no reference exists it **records** one (32-bit float WAV + `.wav.json` + sidecar manifest); otherwise it **compares** and writes a diff WAV on failure. + Exit `0`/`1`. - **Comparators** (`ReferenceComparator.cpp/h`): pluggable `Comparator` + `createComparator(name)` registry. v1 ships only `sample` (per-sample abs-diff tolerance, default one 16-bit LSB = `1/32768`; `0` = bit-exact). Adding `spectrum`/`crosscorr`/etc. is one registry entry, no config/runner changes. -- **Dogfood + self-tests**: two minimal deterministic `juce_add_plugin` targets +- **Dogfood + self-tests**: three minimal deterministic `juce_add_plugin` targets behind `PLUGINVAL_BUILD_TEST_PLUGINS` — `tests/test_plugins/tone_generator/` - (closed-form sine/square generator, phase resets on `prepareToPlay`) and - `tests/test_plugins/gain/` (a gain effect, used to dogfood the `input.audio` - path: it gains a checked-in full-height sine and is compared bit-exact). - `tests/acceptance/` holds checked-in configs (`*.json.in`, the plugin paths + - input dir substituted at configure time), `inputs/` and reference WAVs, run via - CTest (`pluginval.acceptance.*`: sine-440, square-220, square-state, gain-half). - Phase 2 items (config-array multiplexing, - automation, playhead, extra comparators, child-process isolation) are notes + (closed-form sine/square generator, phase resets on `prepareToPlay`), + `tests/test_plugins/gain/` (a gain effect, dogfoods the `input.audio` path) and + `tests/test_plugins/playhead_probe/` (writes the host transport to its output, + dogfoods the `playhead` path). `tests/acceptance/` holds checked-in configs + (`*.json.in`, the plugin paths + input dir substituted at configure time), + `inputs/` and reference WAVs, run via CTest (`pluginval.acceptance.*`: sine-440, + square-220, square-state, gain-half, playhead-120). Phase 2 items (config-array + multiplexing, automation, extra comparators, child-process isolation) are notes only. ### Test Framework diff --git a/CMakeLists.txt b/CMakeLists.txt index 6a333660..c1e7c12a 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -274,5 +274,6 @@ if (PLUGINVAL_BUILD_TEST_PLUGINS) enable_testing() add_subdirectory(tests/test_plugins/tone_generator) add_subdirectory(tests/test_plugins/gain) + add_subdirectory(tests/test_plugins/playhead_probe) add_subdirectory(tests/acceptance) endif() diff --git a/docs/Acceptance testing.md b/docs/Acceptance testing.md index 95c0a6a7..d3037cad 100644 --- a/docs/Acceptance testing.md +++ b/docs/Acceptance testing.md @@ -48,6 +48,7 @@ JSON keys are `snake_case`. The most useful fields: | `state.file` | A binary `getStateInformation` blob to restore first (e.g. a captured preset). Applied before `state.parameters`. | | `reference` | The golden `.wav`. Defaults to `.wav` next to the config. | | `render_duration` | Seconds to render. If omitted, the input audio's length is used. | +| `playhead` | A fixed transport for tempo-dependent plugins: `{ "bpm": 120, "time_signature": { "numerator": 4, "denominator": 4 } }`. Omit it and the plugin gets no playhead. The position advances with the render. | | `comparison` | How to compare. `{ "sample": }` is a per-sample absolute-difference tolerance (`0` = bit-exact); the default is one 16-bit LSB. | ### Determinism matters diff --git a/source/CommandLineTests.cpp b/source/CommandLineTests.cpp index 237007e0..e90772ad 100644 --- a/source/CommandLineTests.cpp +++ b/source/CommandLineTests.cpp @@ -506,6 +506,55 @@ struct CommandLineTests : public juce::UnitTest expect (! config.renderDuration.has_value()); expectEquals (config.getComparison()["sample"].get(), 0.0); } + + beginTest ("Acceptance TestConfig playhead (object time signature)"); + { + // Absent playhead -> unset (no transport supplied to the plugin). + { + const auto config = nlohmann::json::parse (R"({ "plugin": "P.vst3" })").get(); + expect (! config.playhead.has_value()); + } + + // Present playhead -> parsed, with time_signature as an object. + { + const auto json = R"({ + "plugin": "P.vst3", + "playhead": { + "bpm": 90, + "time_signature": { "numerator": 6, "denominator": 8 }, + "start_ppq": 4.0 + } + })"; + + const auto config = nlohmann::json::parse (json).get(); + expect (config.playhead.has_value()); + expectEquals (config.playhead->bpm, 90.0); + expectEquals (config.playhead->timeSigNumerator, 6); + expectEquals (config.playhead->timeSigDenominator, 8); + expectEquals (config.playhead->startPpq, 4.0); + } + + // Time signature defaults to 4/4 when omitted. + { + const auto config = nlohmann::json::parse (R"({ "plugin": "P.vst3", "playhead": { "bpm": 100 } })") + .get(); + expect (config.playhead.has_value()); + expectEquals (config.playhead->timeSigNumerator, 4); + expectEquals (config.playhead->timeSigDenominator, 4); + } + + // An invalid (zero) denominator is rejected. + { + bool threw = false; + try + { + nlohmann::json::parse (R"({ "plugin": "P.vst3", "playhead": { "bpm": 120, "time_signature": { "numerator": 4, "denominator": 0 } } })") + .get(); + } + catch (const std::exception&) { threw = true; } + expect (threw, "expected a zero denominator to be rejected"); + } + } } }; diff --git a/source/acceptance/AcceptanceTest.cpp b/source/acceptance/AcceptanceTest.cpp index dc907660..c05ed61c 100644 --- a/source/acceptance/AcceptanceTest.cpp +++ b/source/acceptance/AcceptanceTest.cpp @@ -38,6 +38,45 @@ namespace #endif } + //============================================================================== + /** A fixed-tempo transport whose position advances with the render. Constructed + from the config's playhead block and pointed at the plugin for the duration + of the render; setSamplePosition() is called before each processBlock. */ + class FixedPlayHead : public juce::AudioPlayHead + { + public: + FixedPlayHead (const TestConfig::PlayheadConfig& config, double sr) + : cfg (config), sampleRate (sr) + { + } + + void setSamplePosition (juce::int64 sample) noexcept { currentSample = sample; } + + juce::Optional getPosition() const override + { + const double seconds = (double) currentSample / sampleRate; + + // PPQ is measured in quarter notes; a bar is numerator * (4/denominator) of them. + const double quarterNotesPerBar = cfg.timeSigNumerator * 4.0 / cfg.timeSigDenominator; + const double ppq = cfg.startPpq + seconds * (cfg.bpm / 60.0); + + PositionInfo info; + info.setBpm (cfg.bpm); + info.setTimeSignature (TimeSignature { cfg.timeSigNumerator, cfg.timeSigDenominator }); + info.setTimeInSamples (currentSample); + info.setTimeInSeconds (seconds); + info.setPpqPosition (ppq); + info.setPpqPositionOfLastBarStart (std::floor (ppq / quarterNotesPerBar) * quarterNotesPerBar); + info.setIsPlaying (true); + return info; + } + + private: + const TestConfig::PlayheadConfig cfg; + const double sampleRate; + juce::int64 currentSample = 0; + }; + //============================================================================== std::unique_ptr loadPlugin (juce::AudioPluginFormatManager& formatManager, const juce::String& pathOrID, @@ -254,7 +293,15 @@ RenderedAudio renderPlugin (const TestConfig& config) if (numSamples <= 0) throw std::runtime_error ("render_duration is required when there is no input.audio"); - // 4. Prepare and render block by block. + // 4. Optional fixed transport for time-dependent plugins. + std::unique_ptr playHead; + if (config.playhead) + { + playHead = std::make_unique (*config.playhead, sampleRate); + instance->setPlayHead (playHead.get()); + } + + // 5. Prepare and render block by block. callPrepareToPlayOnMessageThreadIfVST3 (*instance, sampleRate, blockSize); const int numInputChannels = instance->getTotalNumInputChannels(); @@ -270,6 +317,9 @@ RenderedAudio renderPlugin (const TestConfig& config) { const int thisBlock = juce::jmin (blockSize, numSamples - pos); + if (playHead != nullptr) + playHead->setSamplePosition (pos); + block.clear(); // Copy the input audio slice into the block (silence past its end). @@ -290,6 +340,7 @@ RenderedAudio renderPlugin (const TestConfig& config) output.copyFrom (c, pos, proc, c, 0, thisBlock); } + instance->setPlayHead (nullptr); // playHead is about to be destroyed callReleaseResourcesOnMessageThreadIfVST3 (*instance); instance.reset(); diff --git a/source/acceptance/TestConfig.cpp b/source/acceptance/TestConfig.cpp index a3292c29..94b69042 100644 --- a/source/acceptance/TestConfig.cpp +++ b/source/acceptance/TestConfig.cpp @@ -73,6 +73,16 @@ void to_json (nlohmann::json& j, const TestConfig& c) if (! c.comparison.is_null()) j["comparison"] = c.comparison; + + if (c.playhead) + { + j["playhead"] = { + { "bpm", c.playhead->bpm }, + { "time_signature", { { "numerator", c.playhead->timeSigNumerator }, + { "denominator", c.playhead->timeSigDenominator } } }, + { "start_ppq", c.playhead->startPpq } + }; + } } void from_json (const nlohmann::json& j, TestConfig& c) @@ -98,6 +108,24 @@ void from_json (const nlohmann::json& j, TestConfig& c) if (auto comp = j.find ("comparison"); comp != j.end() && ! comp->is_null()) c.comparison = *comp; + + if (auto ph = j.find ("playhead"); ph != j.end() && ph->is_object()) + { + TestConfig::PlayheadConfig p; + p.bpm = ph->value ("bpm", 120.0); + p.startPpq = ph->value ("start_ppq", 0.0); + + if (auto ts = ph->find ("time_signature"); ts != ph->end() && ts->is_object()) + { + p.timeSigNumerator = ts->value ("numerator", 4); + p.timeSigDenominator = ts->value ("denominator", 4); + } + + if (p.timeSigNumerator <= 0 || p.timeSigDenominator <= 0) + throw std::runtime_error ("playhead.time_signature numerator and denominator must be positive"); + + c.playhead = p; + } } //============================================================================== diff --git a/source/acceptance/TestConfig.h b/source/acceptance/TestConfig.h index fdf0dae7..3db74b3f 100644 --- a/source/acceptance/TestConfig.h +++ b/source/acceptance/TestConfig.h @@ -40,6 +40,17 @@ namespace acceptance */ struct TestConfig { + /** A fixed transport supplied to the plugin during the render, for + time-dependent plugins (tempo-synced LFOs, arpeggiators, ...). The tempo + and time signature are constant; the position advances with the render. */ + struct PlayheadConfig + { + double bpm = 120.0; + int timeSigNumerator = 4; + int timeSigDenominator = 4; + double startPpq = 0.0; /**< Transport position (in quarter notes) at sample 0. */ + }; + std::string name; /**< Labels results; derives the default reference path. */ std::string plugin; /**< Plugin path or AU id. Required. */ std::string inputAudio; /**< input.audio: path to an input audio file, or empty for silence. */ @@ -51,6 +62,7 @@ struct TestConfig int blockSize = 512; std::optional renderDuration; /**< Seconds. Unset -> derive from the input length. */ nlohmann::json comparison; /**< Map of comparator name -> sub-config. Empty -> default. */ + std::optional playhead; /**< Fixed transport. Unset -> no playhead supplied to the plugin. */ //============================================================================== /** The directory the config was loaded from. Relative reference / input / diff --git a/tests/acceptance/Acceptance testing design.md b/tests/acceptance/Acceptance testing design.md index ee546388..0103cc73 100644 --- a/tests/acceptance/Acceptance testing design.md +++ b/tests/acceptance/Acceptance testing design.md @@ -122,7 +122,7 @@ member names verbatim). Missing keys still fall back to the defaults. "block_size": 512, "render_duration": 2.0, // seconds; omitted -> use input length "comparison": { "sample": 1e-6 }, // omitted -> default 1/32768 (16-bit LSB) - "playhead": { "bpm": 120 }, // future + "playhead": { "bpm": 120, "time_signature": { "numerator": 4, "denominator": 4 } }, "automation": [ /* phase 2 */ ] } ``` @@ -142,7 +142,7 @@ member names verbatim). Missing keys still fall back to the defaults. | `block_size` | number | 512 | Single value. | | `render_duration` | number | input length | Seconds. `num_samples = round(duration * sample_rate)`. Input shorter than the duration is padded with silence; longer is truncated. | | `comparison` | object | `{ "sample": 0.0000305 }` | Map of comparator name -> its sub-config. See §5. | -| `playhead` | object | none | **Future.** Fixed tempo / time signature for time-dependent plugins. | +| `playhead` | object | none | Fixed transport for time-dependent plugins. `{ "bpm": , "time_signature": { "numerator": N, "denominator": D }, "start_ppq": }`. `time_signature` defaults to 4/4, `start_ppq` to 0. Omitted -> no playhead is set (the plugin sees `getPlayHead() == nullptr`). The tempo / time signature are constant; the position advances with the render. | | `automation` | array | none | **Phase 2.** Parameter changes scheduled at sample positions. | ### State precedence @@ -323,6 +323,7 @@ tests/acceptance/ square-220.json.in // tone gen, waveform=square, state.parameters, bit-exact square-state.json.in // tone gen, state.file blob + a gain parameter override gain-half.json.in // gain effect, input.audio = a full-height sine, bit-exact + playhead-120.json.in // playhead probe, fixed transport (bpm 120, 4/4), bit-exact inputs/sine-full.wav // checked-in input for gain-half (±1.0 sine) refs/.wav (+ .wav.json) // checked-in references + sidecar manifests refs/square-state.state // checked-in getStateInformation blob @@ -337,7 +338,7 @@ stable enough to commit — a first smoke test of cross-platform portability for the `sample` comparator and the baseline for future comparators (`spectrum`, `crosscorr`, …). -The four cases cover the distinct render paths: +The five cases cover the distinct render paths: - **`sine-440` / `square-220`** — the `state.parameters` (name/index → normalised value) path. `square-220` compares bit-exact (`"sample": 0`). @@ -350,6 +351,11 @@ The four cases cover the distinct render paths: full-height (±1.0) sine by 0.5 and is compared bit-exact (0.5 is exact in float). It also omits `render_duration`, so the render length is derived from the input file. +- **`playhead-120`** — the **`playhead`** path: a third dogfood plugin + (`tests/test_plugins/playhead_probe/`) writes the host transport into its + output (channel 0 = `ppqPosition`, channel 1 = tempo), so the recorded + reference is a direct check that the fixed transport reached the plugin. If the + playhead regressed, the probe would output silence and the compare would fail. ## 9. Execution flow @@ -358,11 +364,13 @@ The four cases cover the distinct render paths: `sample_rate` / `block_size`. 3. Apply `state.file` then `state.parameters`. 4. Load `input.audio` and/or `input.midi`; otherwise use silence. -5. `prepareToPlay`, render `render_duration` worth of blocks, accumulating the +5. If a `playhead` is configured, point a fixed-tempo transport at the plugin + (its position advances each block). +6. `prepareToPlay`, render `render_duration` worth of blocks, accumulating the output into a single buffer. -6. **No reference exists** -> write the float WAV + manifest (record mode); +7. **No reference exists** -> write the float WAV + manifest (record mode); report "reference created". -7. **Reference exists** -> run each configured comparator; report each verdict +8. **Reference exists** -> run each configured comparator; report each verdict and the overall pass/fail; on failure write a diff WAV; exit `0` / `1`. ## 10. Phasing @@ -377,14 +385,14 @@ The four cases cover the distinct render paths: - Record-or-compare with float-WAV + JSON-sidecar references. - `sample` comparator only (default tolerance = one 16-bit LSB). - Text + JSON result reporting; diff WAV on failure. -- **Dogfood tone-generator plugin** (§8) plus a CTest self-test that runs the - full record/compare path against checked-in references. +- Fixed `playhead` (tempo / time signature) for time-dependent plugins. +- **Dogfood test plugins** (§8) plus CTest self-tests that run the full + record/compare path against checked-in references. **Phase 2 and beyond** - Multiplexed execution of config arrays. - Parameter `automation` timelines. -- Fixed `playhead` (tempo / time signature) for time-dependent plugins. - Additional comparators: `peakrms`, `spectrum`, `crosscorr`, `fingerprint`. - Synthesised inputs (`input.generator`: noise / sine, with seed). - Optional child-process isolation mirroring the validate handoff. diff --git a/tests/acceptance/CMakeLists.txt b/tests/acceptance/CMakeLists.txt index cfb76076..5d1f350c 100644 --- a/tests/acceptance/CMakeLists.txt +++ b/tests/acceptance/CMakeLists.txt @@ -7,19 +7,22 @@ # expression (e.g. $ on multi-config generators), so we resolve it via # file(GENERATE). -get_target_property(tone_artefact pluginval_tone_generator_VST3 JUCE_PLUGIN_ARTEFACT_FILE) -get_target_property(gain_artefact pluginval_gain_VST3 JUCE_PLUGIN_ARTEFACT_FILE) +get_target_property(tone_artefact pluginval_tone_generator_VST3 JUCE_PLUGIN_ARTEFACT_FILE) +get_target_property(gain_artefact pluginval_gain_VST3 JUCE_PLUGIN_ARTEFACT_FILE) +get_target_property(probe_artefact pluginval_playhead_probe_VST3 JUCE_PLUGIN_ARTEFACT_FILE) set(ACCEPTANCE_REF_DIR "${CMAKE_CURRENT_SOURCE_DIR}/refs") set(ACCEPTANCE_INPUT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/inputs") set(TONE_GENERATOR_PLUGIN "${tone_artefact}") set(GAIN_PLUGIN "${gain_artefact}") +set(PLAYHEAD_PROBE_PLUGIN "${probe_artefact}") set(acceptance_cases sine-440 # state.parameters path (waveform/frequency/gain by name) square-220 # state.parameters path, bit-exact square wave square-state # state.file (setStateInformation) + parameter override precedence - gain-half) # input.audio effect path: gain a full-height sine, length from input + gain-half # input.audio effect path: gain a full-height sine, length from input + playhead-120) # fixed-playhead path: probe writes the host transport into the output foreach(case ${acceptance_cases}) # Step 1: @-substitute the reference dir (and the possibly-genex plugin path). diff --git a/tests/acceptance/playhead-120.json.in b/tests/acceptance/playhead-120.json.in new file mode 100644 index 00000000..d8aacd6f --- /dev/null +++ b/tests/acceptance/playhead-120.json.in @@ -0,0 +1,15 @@ +{ + "name": "playhead-120", + "plugin": "@PLAYHEAD_PROBE_PLUGIN@", + "reference": "@ACCEPTANCE_REF_DIR@/playhead-120.wav", + "sample_rate": 48000, + "block_size": 512, + "render_duration": 0.25, + "playhead": { + "bpm": 120, + "time_signature": { "numerator": 4, "denominator": 4 } + }, + "comparison": { + "sample": 0.0 + } +} diff --git a/tests/acceptance/refs/playhead-120.wav b/tests/acceptance/refs/playhead-120.wav new file mode 100644 index 0000000000000000000000000000000000000000..a98a876470708e98a965f8c437434b3d168e18e9 GIT binary patch literal 96104 zcmeI)uS=bA6u{xv$zUv8~9-~XH8oPFp(2kw9a z9N+*4IKTl8aDW3GsOUhi9p(Gqw1?RGY=s*YVfCC)h00%h00S<7012r9(TffftziAKQybg4r19!jy4sd`29N+*4IKY9L z4(zSm=KJ5Yhj3m8I?#bT-~b0WzyS_$fCC)hKurfu-~7$@ziAKQybg4r19$MhIB*aA CLl;~C literal 0 HcmV?d00001 diff --git a/tests/acceptance/refs/playhead-120.wav.json b/tests/acceptance/refs/playhead-120.wav.json new file mode 100644 index 00000000..680f2bce --- /dev/null +++ b/tests/acceptance/refs/playhead-120.wav.json @@ -0,0 +1,23 @@ +{ + "config_hash": "6c38c7c43e1f8520", + "created_on": { + "arch": "arm64", + "date": "2026-06-16T15:51:01.553+01:00", + "os": "Mac OSX 26.2" + }, + "plugin": { + "format": "VST3", + "manufacturer": "Tracktion", + "name": "pluginval Playhead Probe", + "uid": "VST3-pluginval Playhead Probe-f5ce8b6f-d8b37cc7", + "version": "1.0.4" + }, + "pluginval_version": "1.0.4", + "render": { + "block_size": 512, + "length_seconds": 0.25, + "num_channels": 2, + "num_samples": 12000, + "sample_rate": 48000.0 + } +} \ No newline at end of file diff --git a/tests/test_plugins/playhead_probe/CMakeLists.txt b/tests/test_plugins/playhead_probe/CMakeLists.txt new file mode 100644 index 00000000..c8c3660d --- /dev/null +++ b/tests/test_plugins/playhead_probe/CMakeLists.txt @@ -0,0 +1,33 @@ +# A minimal, deterministic plugin that writes the host transport into its output, +# used to dogfood the acceptance feature's fixed-playhead support. Built only when +# PLUGINVAL_BUILD_TEST_PLUGINS is ON. + +juce_add_plugin(pluginval_playhead_probe + PRODUCT_NAME "pluginval Playhead Probe" + COMPANY_NAME Tracktion + BUNDLE_ID com.Tracktion.pluginvalPlayheadProbe + PLUGIN_MANUFACTURER_CODE Trkt + PLUGIN_CODE Pph1 + FORMATS VST3 $<$:AU> + IS_SYNTH FALSE + NEEDS_MIDI_INPUT FALSE + VST3_CATEGORIES Generator + AU_MAIN_TYPE kAudioUnitType_Generator + COPY_PLUGIN_AFTER_BUILD FALSE) + +target_sources(pluginval_playhead_probe PRIVATE + PlayheadProbePlugin.cpp + PlayheadProbePlugin.h) + +target_compile_features(pluginval_playhead_probe PRIVATE cxx_std_20) + +target_compile_definitions(pluginval_playhead_probe PRIVATE + JUCE_WEB_BROWSER=0 + JUCE_USE_CURL=0 + JUCE_VST3_CAN_REPLACE_VST2=0) + +target_link_libraries(pluginval_playhead_probe PRIVATE + juce::juce_audio_utils + juce::juce_audio_plugin_client + juce::juce_recommended_config_flags + juce::juce_recommended_warning_flags) diff --git a/tests/test_plugins/playhead_probe/PlayheadProbePlugin.cpp b/tests/test_plugins/playhead_probe/PlayheadProbePlugin.cpp new file mode 100644 index 00000000..b94eb2b9 --- /dev/null +++ b/tests/test_plugins/playhead_probe/PlayheadProbePlugin.cpp @@ -0,0 +1,55 @@ +/*============================================================================== + + Copyright 2018 by Tracktion Corporation. + For more information visit www.tracktion.com + + You may also use this code under the terms of the GPL v3 (see + www.gnu.org/licenses). + + pluginval IS PROVIDED "AS IS" WITHOUT ANY WARRANTY, AND ALL WARRANTIES, WHETHER + EXPRESSED OR IMPLIED, INCLUDING MERCHANTABILITY AND FITNESS FOR PURPOSE, ARE + DISCLAIMED. + + ==============================================================================*/ + +#include "PlayheadProbePlugin.h" + +//============================================================================== +void PlayheadProbeProcessor::processBlock (juce::AudioBuffer& buffer, juce::MidiBuffer&) +{ + buffer.clear(); + + auto* playHead = getPlayHead(); + + if (playHead == nullptr) + return; + + const auto position = playHead->getPosition(); + + if (! position.hasValue()) + return; + + double ppq = 0.0, bpm = 0.0; + + if (const auto v = position->getPpqPosition()) + ppq = *v; + + if (const auto v = position->getBpm()) + bpm = *v; + + const int numSamples = buffer.getNumSamples(); + + // Channel 0: the block's ppqPosition (a per-block staircase that advances with + // the transport). Channel 1: the tempo, scaled into a sane range. + if (buffer.getNumChannels() > 0) + juce::FloatVectorOperations::fill (buffer.getWritePointer (0), (float) ppq, numSamples); + + if (buffer.getNumChannels() > 1) + juce::FloatVectorOperations::fill (buffer.getWritePointer (1), (float) (bpm / 1000.0), numSamples); +} + +//============================================================================== +juce::AudioProcessor* JUCE_CALLTYPE createPluginFilter() +{ + return new PlayheadProbeProcessor(); +} diff --git a/tests/test_plugins/playhead_probe/PlayheadProbePlugin.h b/tests/test_plugins/playhead_probe/PlayheadProbePlugin.h new file mode 100644 index 00000000..d3f03781 --- /dev/null +++ b/tests/test_plugins/playhead_probe/PlayheadProbePlugin.h @@ -0,0 +1,69 @@ +/*============================================================================== + + Copyright 2018 by Tracktion Corporation. + For more information visit www.tracktion.com + + You may also use this code under the terms of the GPL v3 (see + www.gnu.org/licenses). + + pluginval IS PROVIDED "AS IS" WITHOUT ANY WARRANTY, AND ALL WARRANTIES, WHETHER + EXPRESSED OR IMPLIED, INCLUDING MERCHANTABILITY AND FITNESS FOR PURPOSE, ARE + DISCLAIMED. + + ==============================================================================*/ + +#pragma once + +#include + +//============================================================================== +/** + A minimal, deterministic plugin that writes the host transport into its + output so the acceptance feature's fixed-playhead support can be verified. + + Each block it reads getPlayHead()->getPosition() and writes the block's + ppqPosition into channel 0 and the tempo (scaled) into channel 1. With no + playhead (or no position) it outputs silence - so a recorded reference is a + direct check that the transport actually reached the plugin. +*/ +class PlayheadProbeProcessor : public juce::AudioProcessor +{ +public: + //============================================================================== + PlayheadProbeProcessor() + : juce::AudioProcessor (BusesProperties().withOutput ("Output", juce::AudioChannelSet::stereo(), true)) + { + } + + ~PlayheadProbeProcessor() override = default; + + //============================================================================== + void prepareToPlay (double, int) override {} + void releaseResources() override {} + void processBlock (juce::AudioBuffer&, juce::MidiBuffer&) override; + + //============================================================================== + juce::AudioProcessorEditor* createEditor() override { return nullptr; } + bool hasEditor() const override { return false; } + + //============================================================================== + const juce::String getName() const override { return "pluginval Playhead Probe"; } + bool acceptsMidi() const override { return false; } + bool producesMidi() const override { return false; } + bool isMidiEffect() const override { return false; } + double getTailLengthSeconds() const override { return 0.0; } + + //============================================================================== + int getNumPrograms() override { return 1; } + int getCurrentProgram() override { return 0; } + void setCurrentProgram (int) override {} + const juce::String getProgramName (int) override { return "Default"; } + void changeProgramName (int, const juce::String&) override {} + + //============================================================================== + void getStateInformation (juce::MemoryBlock&) override {} + void setStateInformation (const void*, int) override {} + +private: + JUCE_DECLARE_NON_COPYABLE_WITH_LEAK_DETECTOR (PlayheadProbeProcessor) +}; From 2964105814563c3941cd8d82dddb7c7f0ad683ab Mon Sep 17 00:00:00 2001 From: David Rowland Date: Tue, 16 Jun 2026 16:06:01 +0100 Subject: [PATCH 07/21] Acceptance testing: Added a missing dependency for Linux --- .github/workflows/acceptance.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/acceptance.yml b/.github/workflows/acceptance.yml index a216337b..80b1f18a 100644 --- a/.github/workflows/acceptance.yml +++ b/.github/workflows/acceptance.yml @@ -45,7 +45,7 @@ jobs: sudo apt-get install -y \ libasound2-dev libx11-dev libxext-dev libxinerama-dev libxrandr-dev \ libxcursor-dev libxcomposite-dev libfreetype6-dev libfontconfig1-dev \ - libgl1-mesa-dev libcurl4-openssl-dev ninja-build xvfb + libgl1-mesa-dev libcurl4-openssl-dev ladspa-sdk ninja-build xvfb # The acceptance tests host the dogfood plugins via JUCE's own VST3 hosting, # so the embedded VST3 validator (and its heavy SDK build) isn't needed here. From 64faeda7e8f4b98ac111ca93cacca0ce884e7d03 Mon Sep 17 00:00:00 2001 From: David Rowland Date: Wed, 7 Oct 2026 22:06:50 +0100 Subject: [PATCH 08/21] Acceptance testing: Fail on missing reference, render off the message thread, XXH3 config hash - A missing reference is now an error; --record-missing (or RECORD_MISSING=1) records it and passes, for CIs that check new references in. - The render runs on a background TestRunner thread so the message thread is free for the plugin; the plugin is released and deleted on the message thread, including when the render throws. --timeout-ms (default 30000, also TIMEOUT_MS) is enforced by a watchdog that reports and exits 1. - config_hash is now XXH3 (xxHash via CPM, header-only) of the config without the plugin/reference paths, with input/state files hashed by content, so checked-in references no longer look stale on other machines. Regenerated the self-test sidecars. --- CLAUDE.md | 17 +- CMakeLists.txt | 9 + docs/Acceptance testing.md | 31 ++- docs/Command line options.md | 10 +- source/CommandLine.cpp | 24 ++- source/CommandLine.h | 3 + source/CommandLineTests.cpp | 23 ++ source/SettingsParser.cpp | 32 ++- source/SettingsParser.h | 2 + source/acceptance/AcceptanceTest.cpp | 201 ++++++++++++++++-- source/acceptance/AcceptanceTest.h | 49 ++++- tests/acceptance/Acceptance testing design.md | 28 ++- tests/acceptance/refs/gain-half.wav.json | 2 +- tests/acceptance/refs/playhead-120.wav.json | 2 +- tests/acceptance/refs/sine-440.wav.json | 2 +- tests/acceptance/refs/square-220.wav.json | 2 +- tests/acceptance/refs/square-state.wav.json | 2 +- 17 files changed, 372 insertions(+), 67 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index d6f27fef..bc5d471e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -278,9 +278,13 @@ spec: `tests/acceptance/Acceptance testing design.md`; end-user guide: - **CLI**: `pluginval test `. A new `Command::test` is recognised by `settings_parser::dispatch()` (captures the positional config path into - `DispatchResult::testConfigPath`), `isCommandLine()` and `getFooterText()`; - `CommandLine.cpp`'s `performCommandLine()` has a `Command::test` branch that - runs the acceptance runner **synchronously on the message thread** and quits. + `DispatchResult::testConfigPath`, plus `recordMissing` / `timeoutMs` from + `--record-missing` / `--timeout-ms` or `RECORD_MISSING` / `TIMEOUT_MS`), + `isCommandLine()` and `getFooterText()`; `CommandLine.cpp`'s + `performCommandLine()` has a `Command::test` branch that starts an + `acceptance::TestRunner`, which renders on a **background thread** (message + thread left free; plugin deleted on it), with a watchdog for the timeout, and + quits via `callAsync` when done. - **Config**: `acceptance::TestConfig` (`TestConfig.cpp/h`) — std-typed struct, **snake_case JSON keys** mapped via explicit `to_json`/`from_json` (members stay camelCase). It is **independent** of `PluginvalSettings` / the `--config` @@ -291,9 +295,10 @@ spec: `tests/acceptance/Acceptance testing design.md`; end-user guide: configured, point a fixed-tempo transport (`FixedPlayHead`, position advances per block) at the plugin → render a fixed duration block-by-block (reusing the `AudioProcessingTest` shape + the VST3-safe helpers in `TestUtilities.h`). If - no reference exists it **records** one (32-bit float WAV + `.wav.json` - sidecar manifest); otherwise it **compares** and writes a diff WAV on failure. - Exit `0`/`1`. + no reference exists it fails, or with `--record-missing` **records** one + (32-bit float WAV + `.wav.json` sidecar manifest, whose `config_hash` is + XXH3 of the config minus paths, with input/state files hashed by content); + otherwise it **compares** and writes a diff WAV on failure. Exit `0`/`1`. - **Comparators** (`ReferenceComparator.cpp/h`): pluggable `Comparator` + `createComparator(name)` registry. v1 ships only `sample` (per-sample abs-diff tolerance, default one 16-bit LSB = `1/32768`; `0` = bit-exact). Adding diff --git a/CMakeLists.txt b/CMakeLists.txt index c1e7c12a..714fb917 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -68,6 +68,13 @@ CPMAddPackage("gh:Neargye/magic_enum#v0.9.7") CPMAddPackage("gh:CLIUtils/CLI11@2.6.2") CPMAddPackage("gh:nlohmann/json@3.12.0") +# xxHash is used header-only (XXH_INLINE_ALL) for the acceptance-test config hash +CPMAddPackage( + NAME xxHash + GITHUB_REPOSITORY Cyan4973/xxHash + VERSION 0.8.3 + DOWNLOAD_ONLY YES) + if(PLUGINVAL_ENABLE_RTCHECK) CPMAddPackage("gh:Tracktion/rtcheck#main") endif() @@ -201,6 +208,8 @@ target_link_libraries(pluginval PRIVATE CLI11::CLI11 nlohmann_json::nlohmann_json) +target_include_directories(pluginval PRIVATE "${xxHash_SOURCE_DIR}") + if (${CMAKE_SYSTEM_NAME} MATCHES "Linux") target_link_libraries(pluginval PRIVATE -static-libstdc++) diff --git a/docs/Acceptance testing.md b/docs/Acceptance testing.md index d3037cad..cb998483 100644 --- a/docs/Acceptance testing.md +++ b/docs/Acceptance testing.md @@ -2,7 +2,7 @@ Acceptance testing answers a different question from normal validation. Instead of "does this plugin conform to the host API and behave safely?" it asks **"does this -plugin still produce the output I expect for a known input and state?"** — a +plugin still produce the output I expect for a known input and state?"** - a deterministic *render + golden-file comparison*. It is ideal for catching accidental DSP changes (a wrong coefficient, a refactor that shifts the output) in your own plugins from CI. @@ -13,13 +13,16 @@ It is driven by a small JSON config and a dedicated command: pluginval test myPlugin-default.json ``` -- The **first** run renders the plugin and, finding no reference, **records** one - (a `.wav` plus a `.wav.json` manifest) next to the config and reports success. +- The **first** run, with `--record-missing`, renders the plugin and, finding no + reference, **records** one (a `.wav` plus a `.wav.json` manifest) next to the + config and reports success. Without `--record-missing` a missing reference is an + error, so a golden file that is deleted or mis-pathed can't silently pass in CI. - **Subsequent** runs render again and **compare** against that reference, exiting `0` on a match and `1` on a mismatch (writing a diff `.wav` to help you see what changed). -So the workflow is: write a config, run once to record the reference, **commit the +So the workflow is: write a config, run once with `--record-missing` to record the +reference, **commit the config and the reference**, then let CI run the same command on every change. ### A minimal config @@ -44,7 +47,7 @@ JSON keys are `snake_case`. The most useful fields: |---|---| | `plugin` | Path to the plugin (or an AU identifier). **Required.** | | `input.audio` / `input.midi` | Input files to feed it. Omit both for silence (e.g. instruments driven only by MIDI, or generators). | -| `state.parameters` | A map of parameter name (or index) → **normalised** value (`0`–`1`), applied before rendering. | +| `state.parameters` | A map of parameter name (or index) → **normalised** value (`0` to `1`), applied before rendering. | | `state.file` | A binary `getStateInformation` blob to restore first (e.g. a captured preset). Applied before `state.parameters`. | | `reference` | The golden `.wav`. Defaults to `.wav` next to the config. | | `render_duration` | Seconds to render. If omitted, the input audio's length is used. | @@ -56,7 +59,7 @@ JSON keys are `snake_case`. The most useful fields: Acceptance testing only works for output that is reproducible from a fixed input and state. A plugin with free-running randomness can't be golden-tested reliably. For the same reason, references are only safely **portable across platforms** when -you allow a tolerance — exact per-sample matches rarely survive different CPUs and +you allow a tolerance - exact per-sample matches rarely survive different CPUs and floating-point libraries. If a reference recorded on one OS fails on another, raise the `sample` tolerance (or use a more tolerant comparison method as they are added). @@ -72,5 +75,21 @@ pluginval test tests/acceptance/myReverb-default.json A non-zero exit fails the build. Commit the config and its reference `.wav` alongside your project so every run compares against the same golden file. +Options: + +- `--record-missing` - record a missing reference and pass instead of failing. + Useful if your CI records new references and checks them in. Also settable as + `RECORD_MISSING=1`. +- `--timeout-ms ` - fail (exit `1`) if the run takes longer than this + (default `30000`, `-1` to never time out). Also settable as `TIMEOUT_MS`. + +The render runs on a background thread, as a host's audio thread would, leaving +the message thread free for the plugin. + +Each reference's `.wav.json` manifest stores a `config_hash`. If the config (or the +contents of its input / state files) changes after the reference was recorded, the +run warns that the reference may be stale. Paths and the plugin binary aren't part +of the hash, so a checked-in reference doesn't warn on other machines. + For the complete schema, the comparator design and the planned roadmap, see the [design document](<../tests/acceptance/Acceptance testing design.md>). diff --git a/docs/Command line options.md b/docs/Command line options.md index 6c4c66a5..ff15ab50 100644 --- a/docs/Command line options.md +++ b/docs/Command line options.md @@ -11,9 +11,13 @@ COMMANDS: "pluginval [options] " also work. run-tests Run the internal unit tests. strictness-help [level] List all tests that run at the given strictness level. - test Run a deterministic acceptance (golden-file) test - from a config. Records a reference on first run, - compares against it afterwards (exit 0/1). + test [options] Run a deterministic acceptance (golden-file) test + from a config. Compares against the reference + (exit 0/1); a missing reference is an error. + --record-missing Record a missing reference and + pass instead. + --timeout-ms Fail if the run takes longer + (default 30000, -1 = never). The flat flags --validate , --run-tests and --strictness-help [level] are deprecated aliases for the commands above and will be removed in a future diff --git a/source/CommandLine.cpp b/source/CommandLine.cpp index 7b46af1c..ef7fb512 100644 --- a/source/CommandLine.cpp +++ b/source/CommandLine.cpp @@ -119,6 +119,19 @@ void CommandLineValidator::validate (const juce::String& fileOrID, PluginTests:: }); } +void CommandLineValidator::runAcceptanceTest (const juce::File& configFile, const acceptance::RunOptions& options) +{ + // Runs on a background thread so the message thread stays free for the + // plugin; quits itself when done. + testRunner = std::make_unique (configFile, options, + [] (int exitCode) + { + auto& app = *juce::JUCEApplication::getInstance(); + app.setApplicationReturnValue (exitCode); + app.quit(); + }); +} + //============================================================================== static void printStrictnessHelp (int level) { @@ -245,11 +258,12 @@ void performCommandLine (CommandLineValidator& validator, const juce::String& co return; } - // The acceptance runner needs the message thread for the VST3-safe - // lifecycle helpers, and we are already on it here, so run synchronously. - const auto configFile = juce::File::getCurrentWorkingDirectory().getChildFile (routed.testConfigPath); - app.setApplicationReturnValue (acceptance::runTestFile (configFile)); - app.quit(); + acceptance::RunOptions options; + options.recordMissing = routed.recordMissing; + options.timeoutMs = routed.timeoutMs; + + // Runs async so will quit itself when done + validator.runAcceptanceTest (juce::File::getCurrentWorkingDirectory().getChildFile (routed.testConfigPath), options); return; } diff --git a/source/CommandLine.h b/source/CommandLine.h index c19f69f1..1a4f6061 100644 --- a/source/CommandLine.h +++ b/source/CommandLine.h @@ -16,6 +16,7 @@ #include "juce_core/juce_core.h" #include "Validator.h" +#include "acceptance/AcceptanceTest.h" //============================================================================== struct CommandLineValidator @@ -24,9 +25,11 @@ struct CommandLineValidator ~CommandLineValidator(); void validate (const juce::String&, PluginTests::Options); + void runAcceptanceTest (const juce::File& configFile, const acceptance::RunOptions&); private: std::unique_ptr validator; + std::unique_ptr testRunner; }; //============================================================================== diff --git a/source/CommandLineTests.cpp b/source/CommandLineTests.cpp index e90772ad..de228bcf 100644 --- a/source/CommandLineTests.cpp +++ b/source/CommandLineTests.cpp @@ -463,6 +463,29 @@ struct CommandLineTests : public juce::UnitTest } } + beginTest ("Subcommand: test options (--record-missing, --timeout-ms)"); + { + using settings_parser::Command; + + { + const auto d = settings_parser::dispatch (settings_parser::tokenise ("test --record-missing --timeout-ms 500 config.json")); + expect (d.command == Command::test); + expect (d.recordMissing); + expectEquals (d.timeoutMs, (juce::int64) 500); + expectEquals (d.testConfigPath, juce::String ("config.json")); + } + { + // Options after the positional; the timeout value isn't taken as the config. + const auto d = settings_parser::dispatch (settings_parser::tokenise ("test config.json --timeout-ms -1")); + expectEquals (d.testConfigPath, juce::String ("config.json")); + expectEquals (d.timeoutMs, (juce::int64) -1); + } + { + const auto d = settings_parser::dispatch (settings_parser::tokenise ("test --timeout-ms 500 config.json")); + expectEquals (d.testConfigPath, juce::String ("config.json")); + } + } + beginTest ("Acceptance TestConfig JSON parsing (snake_case keys, defaults)"); { const auto json = R"({ diff --git a/source/SettingsParser.cpp b/source/SettingsParser.cpp index 61daea8d..0ae5ae06 100644 --- a/source/SettingsParser.cpp +++ b/source/SettingsParser.cpp @@ -130,7 +130,11 @@ R"(Commands: validate [options] Validate the plugin at the given path or AU id (the default). run-tests Run the internal unit tests. strictness-help [level] List all tests that run at the given strictness level. - test Run a deterministic acceptance (golden-file) test from a config. + test [options] Run a deterministic acceptance (golden-file) test from a config. + --record-missing Record a missing reference and pass + (default: a missing reference fails). + --timeout-ms Fail if the run takes longer than this + (default 30000, -1 to never time out). The flat flags --validate , --run-tests and --strictness-help [level] are deprecated aliases for the commands above and will be removed in a future version. @@ -371,13 +375,31 @@ Precedence (lowest to highest): defaults, environment variables, --config, comma { result.command = Command::test; - // The positional acceptance-test config (first non-option token after the verb). + // Environment variables first, so the command line wins (matching validate). + if (const auto env = juce::SystemStats::getEnvironmentVariable ("RECORD_MISSING", {}); env.isNotEmpty()) + result.recordMissing = env.getIntValue() != 0 || env.equalsIgnoreCase ("true"); + + if (const auto env = juce::SystemStats::getEnvironmentVariable ("TIMEOUT_MS", {}); env.isNotEmpty()) + result.timeoutMs = env.getLargeIntValue(); + + // The options, plus the positional acceptance-test config (the first + // non-option token after the verb). for (int i = 1; i < tokensIn.size(); ++i) { - if (! tokensIn.getReference (i).startsWith ("-")) + const auto& token = tokensIn.getReference (i); + + if (token == "--record-missing") + { + result.recordMissing = true; + } + else if (token == "--timeout-ms") + { + if (i + 1 < tokensIn.size()) + result.timeoutMs = tokensIn.getReference (++i).getLargeIntValue(); + } + else if (! token.startsWith ("-") && result.testConfigPath.isEmpty()) { - result.testConfigPath = tokensIn.getReference (i); - break; + result.testConfigPath = token; } } diff --git a/source/SettingsParser.h b/source/SettingsParser.h index 1898fb4b..98f1705a 100644 --- a/source/SettingsParser.h +++ b/source/SettingsParser.h @@ -81,6 +81,8 @@ namespace settings_parser bool deprecatedAlias = false; /**< A legacy flat flag (--validate/--run-tests/--strictness-help) was used. */ int strictnessLevel = 5; /**< The level for the strictnessHelp command. */ juce::String testConfigPath; /**< The positional config.json for the test command. */ + bool recordMissing = false; /**< test: record a missing reference (and pass) instead of failing. */ + juce::int64 timeoutMs = 30000; /**< test: abort the run after this long (-1 to never time out). */ }; /** Routes tokenised input to a subcommand, peeling the leading verb. The diff --git a/source/acceptance/AcceptanceTest.cpp b/source/acceptance/AcceptanceTest.cpp index c05ed61c..8d29053d 100644 --- a/source/acceptance/AcceptanceTest.cpp +++ b/source/acceptance/AcceptanceTest.cpp @@ -17,8 +17,13 @@ #include +#define XXH_INLINE_ALL +#include + #include #include +#include +#include #include namespace acceptance @@ -100,6 +105,47 @@ namespace return instance; } + //============================================================================== + /** Deletes the plugin on the message thread, as hosts are expected to. */ + void deleteOnMessageThread (std::unique_ptr instance) + { + if (juce::MessageManager::getInstance()->isThisTheMessageThread()) + { + instance.reset(); + return; + } + + juce::WaitableEvent deleted; + juce::MessageManager::callAsync ([&] + { + instance.reset(); + deleted.signal(); + }); + deleted.wait(); + } + + /** Owns the plugin for the duration of the render. On the way out (including + when an exception unwinds the render) it detaches the playhead, releases + resources if the plugin was prepared and deletes it on the message thread. */ + struct PluginLifetime + { + ~PluginLifetime() + { + if (instance == nullptr) + return; + + instance->setPlayHead (nullptr); + + if (prepared) + callReleaseResourcesOnMessageThreadIfVST3 (*instance); + + deleteOnMessageThread (std::move (instance)); + } + + std::unique_ptr instance; + bool prepared = false; + }; + //============================================================================== void applyState (juce::AudioPluginInstance& instance, const TestConfig& config) { @@ -220,25 +266,44 @@ namespace } //============================================================================== + juce::String toHex (XXH64_hash_t hash) + { + return juce::String::toHexString ((juce::int64) hash).paddedLeft ('0', 16); + } + + /** XXH3 of a file's contents, or "missing" if it can't be read. */ + juce::String hashFile (const juce::File& file) + { + juce::MemoryBlock data; + + if (! file.loadFileAsData (data)) + return "missing"; + + return toHex (XXH3_64bits (data.getData(), data.getSize())); + } + + /** A hash of everything that determines the render apart from the plugin + binary itself (which is what's under test). Paths are machine-specific, + so they are left out: the input and state files are hashed by content + instead, and the plugin and reference locations are dropped. */ juce::String configHash (const TestConfig& config) { auto j = nlohmann::json {}; to_json (j, config); - j.erase ("reference"); // the hash must be independent of where the reference lives - - // A small, stable (cross-platform) FNV-1a 64-bit hash. The hash is only - // used for an informational "stale reference" warning, so juce_cryptography - // (MD5) isn't worth pulling in. - const auto dump = j.dump(); - std::uint64_t hash = 1469598103934665603ull; + j.erase ("plugin"); + j.erase ("reference"); - for (const auto c : dump) + if (auto input = j.find ("input"); input != j.end()) { - hash ^= (std::uint64_t) (unsigned char) c; - hash *= 1099511628211ull; + if (input->contains ("audio")) (*input)["audio"] = hashFile (config.getInputAudioFile()).toStdString(); + if (input->contains ("midi")) (*input)["midi"] = hashFile (config.getInputMidiFile()).toStdString(); } - return juce::String::toHexString ((juce::int64) hash); + if (auto state = j.find ("state"); state != j.end() && state->contains ("file")) + (*state)["file"] = hashFile (config.getStateFile()).toStdString(); + + const auto dump = j.dump(); + return toHex (XXH3_64bits (dump.data(), dump.size())); } } @@ -255,7 +320,12 @@ RenderedAudio renderPlugin (const TestConfig& config) formatManager.addDefaultFormats(); #endif - auto instance = loadPlugin (formatManager, config.getPluginPathOrID(), sampleRate, blockSize); + // Declared before the plugin so it outlives it. + std::unique_ptr playHead; + + PluginLifetime plugin; + plugin.instance = loadPlugin (formatManager, config.getPluginPathOrID(), sampleRate, blockSize); + auto& instance = plugin.instance; const auto description = instance->getPluginDescription(); // 1. State (file then parameter overrides), applied before prepareToPlay. @@ -294,7 +364,6 @@ RenderedAudio renderPlugin (const TestConfig& config) throw std::runtime_error ("render_duration is required when there is no input.audio"); // 4. Optional fixed transport for time-dependent plugins. - std::unique_ptr playHead; if (config.playhead) { playHead = std::make_unique (*config.playhead, sampleRate); @@ -303,6 +372,7 @@ RenderedAudio renderPlugin (const TestConfig& config) // 5. Prepare and render block by block. callPrepareToPlayOnMessageThreadIfVST3 (*instance, sampleRate, blockSize); + plugin.prepared = true; const int numInputChannels = instance->getTotalNumInputChannels(); const int numOutputChannels = juce::jmax (1, instance->getTotalNumOutputChannels()); @@ -340,23 +410,30 @@ RenderedAudio renderPlugin (const TestConfig& config) output.copyFrom (c, pos, proc, c, 0, thisBlock); } - instance->setPlayHead (nullptr); // playHead is about to be destroyed - callReleaseResourcesOnMessageThreadIfVST3 (*instance); - instance.reset(); - return { std::move (output), sampleRate, blockSize, description }; } //============================================================================== -TestResult runTest (const TestConfig& config) +TestResult runTest (const TestConfig& config, const RunOptions& options) { const auto name = config.getName(); try { - const auto rendered = renderPlugin (config); const auto referenceFile = config.getReferenceFile(); + // Fail fast: without --record-missing, a missing reference is an error + // (e.g. a golden file deleted or mis-pathed in CI must not silently pass). + if (! referenceFile.existsAsFile() && ! options.recordMissing) + { + auto result = TestResult::makeError (name, "reference not found: " + referenceFile.getFullPathName() + + " (run with --record-missing to record it)"); + result.referenceFile = referenceFile; + return result; + } + + const auto rendered = renderPlugin (config); + TestResult result; result.name = name; result.referenceFile = referenceFile; @@ -401,11 +478,14 @@ TestResult runTest (const TestConfig& config) try { const auto manifest = nlohmann::json::parse (sidecar.loadFileAsString().toStdString()); + const auto currentHash = configHash (config).toStdString(); + if (manifest.contains ("config_hash") - && manifest["config_hash"].get() != configHash (config).toStdString()) + && manifest["config_hash"].get() != currentHash) { std::cout << " WARNING: reference was recorded from a different config (stale?): " - << sidecar.getFullPathName() << std::endl; + << sidecar.getFullPathName() << " (recorded " << manifest["config_hash"].get() + << ", current " << currentHash << ")" << std::endl; } } catch (const std::exception&) { /* a malformed sidecar is non-fatal */ } @@ -462,10 +542,14 @@ TestResult runTest (const TestConfig& config) { return TestResult::makeError (name, e.what()); } + catch (...) + { + return TestResult::makeError (name, "unknown exception"); + } } //============================================================================== -int runTestFile (const juce::File& configFile) +int runTestFile (const juce::File& configFile, const RunOptions& options) { std::vector configs; @@ -492,9 +576,80 @@ int runTestFile (const juce::File& configFile) if (configs.size() > 1) std::cout << "Note: " << configs.size() << " configs found; v1 runs the first only." << std::endl; - const auto result = runTest (configs.front()); + const auto result = runTest (configs.front(), options); reporter::report (result); return reporter::exitCode (result); } +//============================================================================== +struct TestRunner::Watchdog : private juce::Thread +{ + Watchdog (const juce::File& file, juce::int64 timeout) + : Thread ("pluginval test watchdog"), configFile (file), timeoutMs (timeout) + { + startThread (juce::Thread::Priority::low); + } + + ~Watchdog() override + { + markFinished(); + stopThread (5000); + } + + void markFinished() + { + finished.signal(); + } + +private: + const juce::File configFile; + const juce::int64 timeoutMs; + juce::WaitableEvent finished; + + void run() override + { + if (finished.wait ((double) timeoutMs)) + return; + + // The worker is stuck in a plugin call that can't be interrupted, so + // report the failure and end the process. + const auto result = TestResult::makeError (configFile.getFileNameWithoutExtension(), + "timed out after " + juce::RelativeTime::milliseconds (timeoutMs).getDescription()); + reporter::report (result); + std::cout << std::flush; + std::_Exit (reporter::exitCode (result)); + } +}; + +//============================================================================== +TestRunner::TestRunner (const juce::File& file, const RunOptions& opts, std::function callback) + : Thread ("pluginval test"), + configFile (file), + options (opts), + onComplete (std::move (callback)) +{ + jassert (onComplete); + + if (options.timeoutMs > 0) + watchdog = std::make_unique (configFile, options.timeoutMs); + + startThread(); +} + +TestRunner::~TestRunner() +{ + stopThread (5000); + watchdog.reset(); +} + +void TestRunner::run() +{ + const auto exitCode = runTestFile (configFile, options); + + if (watchdog != nullptr) + watchdog->markFinished(); + + juce::MessageManager::callAsync ([callback = onComplete, exitCode] { callback (exitCode); }); +} + } // namespace acceptance diff --git a/source/acceptance/AcceptanceTest.h b/source/acceptance/AcceptanceTest.h index fe666619..65ea6bea 100644 --- a/source/acceptance/AcceptanceTest.h +++ b/source/acceptance/AcceptanceTest.h @@ -19,6 +19,9 @@ #include +#include +#include + namespace acceptance { @@ -36,21 +39,53 @@ struct RenderedAudio /** Loads the plugin, applies state (file then parameters), feeds the input (audio / MIDI / silence) and renders a fixed duration into a single buffer. - Must be called on the message thread (it uses the VST3-safe lifecycle - helpers and creates the plugin instance directly). Throws std::runtime_error - on any setup / render failure. + Intended to be called off the message thread (see TestRunner): VST3 lifecycle + calls and the plugin's destruction are marshalled onto the message thread, + which must be free to service them. Throws std::runtime_error on any setup / + render failure. */ RenderedAudio renderPlugin (const TestConfig&); +//============================================================================== +/** Options from the "pluginval test" command line. */ +struct RunOptions +{ + bool recordMissing = false; /**< Record a missing reference and pass, rather than failing. */ + juce::int64 timeoutMs = 30000; /**< Abort the whole run after this long. <= 0 never times out. */ +}; + //============================================================================== /** Runs a single resolved config end-to-end: render, then record-or-compare, producing a TestResult. Never throws - setup failures become an error result. */ -TestResult runTest (const TestConfig&); +TestResult runTest (const TestConfig&, const RunOptions&); /** Loads the config file (first entry only for v1), runs it, reports the result - to stdout and returns the process exit code (0 success, 1 failure). + to stdout and returns the process exit code (0 success, 1 failure). */ +int runTestFile (const juce::File& configFile, const RunOptions&); + +//============================================================================== +/** Runs runTestFile() on a background thread so the message thread stays free + to service the plugin (VST3 lifecycle calls, async creation, posted work). - Must be called on the message thread. */ -int runTestFile (const juce::File& configFile); + onComplete is called on the message thread with the exit code. If the run + exceeds the timeout, a watchdog reports the failure and terminates the + process with exit code 1, since a hung plugin call can't be interrupted. +*/ +class TestRunner : private juce::Thread +{ +public: + TestRunner (const juce::File& configFile, const RunOptions&, std::function onComplete); + ~TestRunner() override; + +private: + struct Watchdog; + + const juce::File configFile; + const RunOptions options; + std::function onComplete; + std::unique_ptr watchdog; + + void run() override; +}; } // namespace acceptance diff --git a/tests/acceptance/Acceptance testing design.md b/tests/acceptance/Acceptance testing design.md index 0103cc73..fb128fae 100644 --- a/tests/acceptance/Acceptance testing design.md +++ b/tests/acceptance/Acceptance testing design.md @@ -53,8 +53,13 @@ pluginval test - `` is the **positional** acceptance-test definition (see §4). It is parsed by its own loader. -- If the config's reference file does not exist, it is **created** (record - mode) and the command reports success. +- If the config's reference file does not exist, the command fails, unless + `--record-missing` (or `RECORD_MISSING=1`) is given, in which case it is + **created** (record mode) and the command reports success. Failing by default + stops a deleted or mis-pathed reference from silently passing in CI. +- `--timeout-ms ` (default 30000, `-1` = never; also `TIMEOUT_MS`) bounds the + whole run. A watchdog reports the timeout and exits `1`, since a hung plugin + call can't be interrupted. - If the reference exists, the rendered output is **compared** against it and the command exits `0` (match) or `1` (mismatch), consistent with `validate`. @@ -95,7 +100,10 @@ Because `test` is brand new there are **no deprecated-alias** concerns. For v1, acceptance tests run **in-process**: record and compare both need the rendered buffer back in the calling process, so in-process is the natural -choice. Crash isolation can be layered on later by mirroring the validate +choice. The render runs on a background thread (`acceptance::TestRunner`) so the +message thread stays free to service the plugin (VST3 lifecycle calls, +async-only creation, work the plugin posts to the message thread); the plugin is +deleted on the message thread. Crash isolation can be layered on later by mirroring the validate child-process handoff (`createChildProcessCommandLine` in `SettingsParser.cpp`, which passes an authoritative base64-JSON blob via `--config-base64`); the equivalent would be `test --config-base64 ` with the child writing the @@ -227,7 +235,7 @@ A reference is **two files**: "render": { "sample_rate": 48000, "block_size": 512, "num_channels": 2, "num_samples": 96000, "length_seconds": 2.0 }, "pluginval_version": "2.0.0", - "config_hash": "…", // hash of the resolved config (excluding the reference path) + "config_hash": "…", // XXH3 of the config, paths excluded (see below) "created_on": { "os": "macOS", "arch": "arm64", "date": "2026-…" } // informational only } ``` @@ -235,7 +243,12 @@ A reference is **two files**: - `created_on` is **informational only** — it is never used for matching, because references are meant to be portable and shared. - `config_hash` lets the runner detect a **stale** reference (one produced from a - different config than the one now being run) and warn. + different config than the one now being run) and warn. It is an XXH3-64 hash + (xxHash, header-only via CPM) of the config with the `plugin` and `reference` + paths removed and the `input.audio` / `input.midi` / `state.file` paths replaced + by XXH3 hashes of the files' contents. Paths are machine-specific, so hashing + them would make every checked-in reference look stale elsewhere; the plugin + binary is what's under test, so it is deliberately not part of the hash. - The default reference path deliberately contains **no platform/arch**, since references are intended to be portable and checked into a repo. @@ -368,8 +381,9 @@ The five cases cover the distinct render paths: (its position advances each block). 6. `prepareToPlay`, render `render_duration` worth of blocks, accumulating the output into a single buffer. -7. **No reference exists** -> write the float WAV + manifest (record mode); - report "reference created". +7. **No reference exists** -> without `--record-missing`, fail before rendering; + with it, write the float WAV + manifest (record mode) and report + "reference created". 8. **Reference exists** -> run each configured comparator; report each verdict and the overall pass/fail; on failure write a diff WAV; exit `0` / `1`. diff --git a/tests/acceptance/refs/gain-half.wav.json b/tests/acceptance/refs/gain-half.wav.json index 303838c4..e2704a33 100644 --- a/tests/acceptance/refs/gain-half.wav.json +++ b/tests/acceptance/refs/gain-half.wav.json @@ -1,5 +1,5 @@ { - "config_hash": "e1622f157cf37917", + "config_hash": "4fb4289c542534c8", "created_on": { "arch": "arm64", "date": "2026-06-16T14:22:49.718+01:00", diff --git a/tests/acceptance/refs/playhead-120.wav.json b/tests/acceptance/refs/playhead-120.wav.json index 680f2bce..85ddde3a 100644 --- a/tests/acceptance/refs/playhead-120.wav.json +++ b/tests/acceptance/refs/playhead-120.wav.json @@ -1,5 +1,5 @@ { - "config_hash": "6c38c7c43e1f8520", + "config_hash": "1f49c0804e8fa76f", "created_on": { "arch": "arm64", "date": "2026-06-16T15:51:01.553+01:00", diff --git a/tests/acceptance/refs/sine-440.wav.json b/tests/acceptance/refs/sine-440.wav.json index cf674ea2..857f0486 100644 --- a/tests/acceptance/refs/sine-440.wav.json +++ b/tests/acceptance/refs/sine-440.wav.json @@ -1,5 +1,5 @@ { - "config_hash": "815cbbe305c6d5b0", + "config_hash": "b8d9bce1db94737b", "created_on": { "arch": "arm64", "date": "2026-06-16T13:10:20.117+01:00", diff --git a/tests/acceptance/refs/square-220.wav.json b/tests/acceptance/refs/square-220.wav.json index 9168e12d..69e7dc43 100644 --- a/tests/acceptance/refs/square-220.wav.json +++ b/tests/acceptance/refs/square-220.wav.json @@ -1,5 +1,5 @@ { - "config_hash": "a1c34639b14ec668", + "config_hash": "8a36b10bbd3c27a3", "created_on": { "arch": "arm64", "date": "2026-06-16T13:10:20.377+01:00", diff --git a/tests/acceptance/refs/square-state.wav.json b/tests/acceptance/refs/square-state.wav.json index 787b1083..5e49c1be 100644 --- a/tests/acceptance/refs/square-state.wav.json +++ b/tests/acceptance/refs/square-state.wav.json @@ -1,5 +1,5 @@ { - "config_hash": "bb5f4e275caed1db", + "config_hash": "1cdcebfe70935c8f", "created_on": { "arch": "arm64", "date": "2026-06-16T13:10:20.606+01:00", From e014cb32b4125f51f6bf40b9dbd70acbd5881f26 Mon Sep 17 00:00:00 2001 From: David Rowland Date: Wed, 7 Oct 2026 22:13:27 +0100 Subject: [PATCH 09/21] Acceptance testing: Validate configs, fail on NaN, apply parameters in file order - The sample comparator fails on any NaN / inf difference (NaN previously compared false both ways and passed). - from_json rejects an empty or non-object comparison, a missing plugin, a missing name and reference, a non-positive sample_rate, a block_size outside 1-65536 and a non-positive or int-overflowing render_duration. - Input audio and reference WAV sample rates must match the render; WAV lengths over INT_MAX and failed reads are errors. - state.parameters is parsed with ordered_json and applied in file order; parameter IDs are matched via HostedParameter::getParameterID(). - MIDI meta events are no longer sent to processBlock. - A stale -diff.wav is removed before comparing; *-diff.wav is ignored. - The test verb rejects unknown options, a bad --timeout-ms value and extra positional arguments. - Mono input still feeds channel 0 only; now documented. - Regenerated the self-test sidecar config hashes (key order changed). --- .gitignore | 4 +- CLAUDE.md | 4 +- docs/Acceptance testing.md | 19 +++- docs/Command line options.md | 1 + source/CommandLine.cpp | 6 ++ source/CommandLineTests.cpp | 92 +++++++++++++++++-- source/SettingsParser.cpp | 24 ++++- source/SettingsParser.h | 1 + source/acceptance/AcceptanceTest.cpp | 68 +++++++++++--- source/acceptance/ReferenceComparator.cpp | 30 ++++-- source/acceptance/ReferenceComparator.h | 2 +- source/acceptance/TestConfig.cpp | 83 +++++++++++++---- source/acceptance/TestConfig.h | 17 ++-- tests/acceptance/Acceptance testing design.md | 16 ++-- tests/acceptance/refs/gain-half.wav.json | 2 +- tests/acceptance/refs/playhead-120.wav.json | 2 +- tests/acceptance/refs/sine-440.wav.json | 2 +- tests/acceptance/refs/square-220.wav.json | 2 +- tests/acceptance/refs/square-state.wav.json | 2 +- 19 files changed, 300 insertions(+), 77 deletions(-) diff --git a/.gitignore b/.gitignore index e2c58172..355bd37e 100644 --- a/.gitignore +++ b/.gitignore @@ -79,4 +79,6 @@ pluginval.build/ pluginval.xcodeproj/ pluginval_artefacts/ -.claude/*local* \ No newline at end of file +.claude/*local* +# Acceptance-test failure diffs (written next to the reference) +*-diff.wav diff --git a/CLAUDE.md b/CLAUDE.md index bc5d471e..dd355de5 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -290,8 +290,8 @@ spec: `tests/acceptance/Acceptance testing design.md`; end-user guide: stay camelCase). It is **independent** of `PluginvalSettings` / the `--config` layering: the test config is a positional argument loaded standalone. - **Flow** (`AcceptanceTest.cpp`): load plugin → apply `state.file` then - `state.parameters` (normalised, matched by index / case-insensitive name or - paramID) → feed `input.audio`/`input.midi` or silence → if a `playhead` is + `state.parameters` (normalised, applied in file order via `ordered_json`, + matched by index / case-insensitive `HostedParameter` ID or name) → feed `input.audio`/`input.midi` or silence → if a `playhead` is configured, point a fixed-tempo transport (`FixedPlayHead`, position advances per block) at the plugin → render a fixed duration block-by-block (reusing the `AudioProcessingTest` shape + the VST3-safe helpers in `TestUtilities.h`). If diff --git a/docs/Acceptance testing.md b/docs/Acceptance testing.md index cb998483..6352c71d 100644 --- a/docs/Acceptance testing.md +++ b/docs/Acceptance testing.md @@ -45,14 +45,19 @@ JSON keys are `snake_case`. The most useful fields: | Field | Notes | |---|---| +| `name` | Labels the result and gives the default reference path. **Required** unless `reference` is given. | | `plugin` | Path to the plugin (or an AU identifier). **Required.** | -| `input.audio` / `input.midi` | Input files to feed it. Omit both for silence (e.g. instruments driven only by MIDI, or generators). | -| `state.parameters` | A map of parameter name (or index) → **normalised** value (`0` to `1`), applied before rendering. | +| `input.audio` / `input.midi` | Input files to feed it. Omit both for silence (e.g. instruments driven only by MIDI, or generators). The audio file's sample rate must match `sample_rate`. Each file channel feeds the plugin input channel with the same index: a mono file feeds channel 0 only (it isn't duplicated), and extra channels on either side are silent / ignored. MIDI meta events (tempo, track names) aren't sent to the plugin. | +| `state.parameters` | A map of parameter → **normalised** value (`0` to `1`), applied before rendering, **in file order**. A key of digits only is a parameter index; otherwise it matches the format's parameter ID, then the display name (case-insensitively). | | `state.file` | A binary `getStateInformation` blob to restore first (e.g. a captured preset). Applied before `state.parameters`. | | `reference` | The golden `.wav`. Defaults to `.wav` next to the config. | -| `render_duration` | Seconds to render. If omitted, the input audio's length is used. | +| `sample_rate` / `block_size` | Default `44100` / `512`. `sample_rate` must be positive; `block_size` must be 1 to 65536. | +| `render_duration` | Seconds to render (positive). If omitted, the input audio's length is used. | | `playhead` | A fixed transport for tempo-dependent plugins: `{ "bpm": 120, "time_signature": { "numerator": 4, "denominator": 4 } }`. Omit it and the plugin gets no playhead. The position advances with the render. | -| `comparison` | How to compare. `{ "sample": }` is a per-sample absolute-difference tolerance (`0` = bit-exact); the default is one 16-bit LSB. | +| `comparison` | How to compare. `{ "sample": }` is a per-sample absolute-difference tolerance (`0` = bit-exact); the default is one 16-bit LSB. Any NaN / inf difference fails. Omit it for the default; an empty `{}` is an error. | + +An invalid config (a missing required field or an out-of-range value) is reported +as an error (exit `1`) without rendering. ### Determinism matters @@ -77,6 +82,8 @@ alongside your project so every run compares against the same golden file. Options: +Any other option is an error. + - `--record-missing` - record a missing reference and pass instead of failing. Useful if your CI records new references and checks them in. Also settable as `RECORD_MISSING=1`. @@ -91,5 +98,9 @@ contents of its input / state files) changes after the reference was recorded, t run warns that the reference may be stale. Paths and the plugin binary aren't part of the hash, so a checked-in reference doesn't warn on other machines. +On a mismatch a `-diff.wav` (output minus reference) is written next to the +reference; it's removed again on the next passing run. Add `*-diff.wav` to your +`.gitignore`. + For the complete schema, the comparator design and the planned roadmap, see the [design document](<../tests/acceptance/Acceptance testing design.md>). diff --git a/docs/Command line options.md b/docs/Command line options.md index ff15ab50..4d1bc568 100644 --- a/docs/Command line options.md +++ b/docs/Command line options.md @@ -18,6 +18,7 @@ COMMANDS: pass instead. --timeout-ms Fail if the run takes longer (default 30000, -1 = never). + Any other option is an error. The flat flags --validate , --run-tests and --strictness-help [level] are deprecated aliases for the commands above and will be removed in a future diff --git a/source/CommandLine.cpp b/source/CommandLine.cpp index ef7fb512..4eeb5d58 100644 --- a/source/CommandLine.cpp +++ b/source/CommandLine.cpp @@ -252,6 +252,12 @@ void performCommandLine (CommandLineValidator& validator, const juce::String& co if (routed.command == settings_parser::Command::test) { + if (routed.error.isNotEmpty()) + { + exitWithError ("*** FAILED: " + routed.error); + return; + } + if (routed.testConfigPath.isEmpty()) { exitWithError ("*** FAILED: No acceptance-test config specified (usage: pluginval test )"); diff --git a/source/CommandLineTests.cpp b/source/CommandLineTests.cpp index de228bcf..70b5d510 100644 --- a/source/CommandLineTests.cpp +++ b/source/CommandLineTests.cpp @@ -12,6 +12,7 @@ ==============================================================================*/ +#include #include struct CommandLineTests : public juce::UnitTest @@ -483,6 +484,25 @@ struct CommandLineTests : public juce::UnitTest { const auto d = settings_parser::dispatch (settings_parser::tokenise ("test --timeout-ms 500 config.json")); expectEquals (d.testConfigPath, juce::String ("config.json")); + expect (d.error.isEmpty()); + } + { + // Unknown options are errors, not silently skipped (their value + // would otherwise be taken as the config path). + const auto d = settings_parser::dispatch (settings_parser::tokenise ("test --output-dir out config.json")); + expect (d.error.contains ("--output-dir")); + } + { + const auto d = settings_parser::dispatch (settings_parser::tokenise ("test config.json --timeout-ms abc")); + expect (d.error.contains ("--timeout-ms")); + } + { + const auto d = settings_parser::dispatch (settings_parser::tokenise ("test config.json --timeout-ms")); + expect (d.error.contains ("--timeout-ms")); + } + { + const auto d = settings_parser::dispatch (settings_parser::tokenise ("test a.json b.json")); + expect (d.error.contains ("b.json")); } } @@ -498,7 +518,7 @@ struct CommandLineTests : public juce::UnitTest "render_duration": 2.0 })"; - const auto config = nlohmann::json::parse (json).get(); + const auto config = nlohmann::ordered_json::parse (json).get(); expectEquals (config.getName(), juce::String ("myReverb-default")); expectEquals (juce::String (config.plugin), juce::String ("/path/to/Plugin.vst3")); @@ -508,9 +528,12 @@ struct CommandLineTests : public juce::UnitTest expectEquals (config.blockSize, 256); expect (config.renderDuration.has_value()); expectEquals (*config.renderDuration, 2.0); + // state.parameters keeps file order ("Mix" sorts after "3"). expectEquals ((int) config.stateParameters.size(), 2); - expectEquals (config.stateParameters.at ("Mix"), 0.5); - expectEquals (config.stateParameters.at ("3"), 1.0); + expectEquals (juce::String (config.stateParameters[0].first), juce::String ("Mix")); + expectEquals (config.stateParameters[0].second, 0.5); + expectEquals (juce::String (config.stateParameters[1].first), juce::String ("3")); + expectEquals (config.stateParameters[1].second, 1.0); // Omitted comparison falls back to one 16-bit LSB. const auto comparison = config.getComparison(); @@ -521,11 +544,12 @@ struct CommandLineTests : public juce::UnitTest beginTest ("Acceptance TestConfig omitted render_duration and explicit comparison"); { const auto json = R"({ + "name": "t", "plugin": "Plugin.vst3", "comparison": { "sample": 0.0 } })"; - const auto config = nlohmann::json::parse (json).get(); + const auto config = nlohmann::ordered_json::parse (json).get(); expect (! config.renderDuration.has_value()); expectEquals (config.getComparison()["sample"].get(), 0.0); } @@ -534,13 +558,14 @@ struct CommandLineTests : public juce::UnitTest { // Absent playhead -> unset (no transport supplied to the plugin). { - const auto config = nlohmann::json::parse (R"({ "plugin": "P.vst3" })").get(); + const auto config = nlohmann::ordered_json::parse (R"({ "name": "t", "plugin": "P.vst3" })").get(); expect (! config.playhead.has_value()); } // Present playhead -> parsed, with time_signature as an object. { const auto json = R"({ + "name": "t", "plugin": "P.vst3", "playhead": { "bpm": 90, @@ -549,7 +574,7 @@ struct CommandLineTests : public juce::UnitTest } })"; - const auto config = nlohmann::json::parse (json).get(); + const auto config = nlohmann::ordered_json::parse (json).get(); expect (config.playhead.has_value()); expectEquals (config.playhead->bpm, 90.0); expectEquals (config.playhead->timeSigNumerator, 6); @@ -559,7 +584,7 @@ struct CommandLineTests : public juce::UnitTest // Time signature defaults to 4/4 when omitted. { - const auto config = nlohmann::json::parse (R"({ "plugin": "P.vst3", "playhead": { "bpm": 100 } })") + const auto config = nlohmann::ordered_json::parse (R"({ "name": "t", "plugin": "P.vst3", "playhead": { "bpm": 100 } })") .get(); expect (config.playhead.has_value()); expectEquals (config.playhead->timeSigNumerator, 4); @@ -571,13 +596,64 @@ struct CommandLineTests : public juce::UnitTest bool threw = false; try { - nlohmann::json::parse (R"({ "plugin": "P.vst3", "playhead": { "bpm": 120, "time_signature": { "numerator": 4, "denominator": 0 } } })") + nlohmann::ordered_json::parse (R"({ "name": "t", "plugin": "P.vst3", "playhead": { "bpm": 120, "time_signature": { "numerator": 4, "denominator": 0 } } })") .get(); } catch (const std::exception&) { threw = true; } expect (threw, "expected a zero denominator to be rejected"); } } + + beginTest ("Acceptance TestConfig validation"); + { + const auto rejects = [] (const char* json) + { + try + { + nlohmann::ordered_json::parse (json).get(); + } + catch (const std::exception&) + { + return true; + } + + return false; + }; + + expect (! rejects (R"({ "name": "t", "plugin": "P.vst3" })")); + expect (! rejects (R"({ "reference": "r.wav", "plugin": "P.vst3" })"), "reference alone names the test"); + + expect (rejects (R"({ "name": "t" })"), "plugin is required"); + expect (rejects (R"({ "plugin": "P.vst3" })"), "name or reference is required"); + expect (rejects (R"({ "name": "t", "plugin": "P.vst3", "comparison": {} })"), "empty comparison"); + expect (rejects (R"({ "name": "t", "plugin": "P.vst3", "comparison": 1 })"), "non-object comparison"); + expect (rejects (R"({ "name": "t", "plugin": "P.vst3", "sample_rate": 0 })"), "zero sample_rate"); + expect (rejects (R"({ "name": "t", "plugin": "P.vst3", "sample_rate": -48000 })"), "negative sample_rate"); + expect (rejects (R"({ "name": "t", "plugin": "P.vst3", "block_size": 0 })"), "zero block_size"); + expect (rejects (R"({ "name": "t", "plugin": "P.vst3", "block_size": 65537 })"), "huge block_size"); + expect (rejects (R"({ "name": "t", "plugin": "P.vst3", "render_duration": 0 })"), "zero render_duration"); + expect (rejects (R"({ "name": "t", "plugin": "P.vst3", "render_duration": "1" })"), "string render_duration"); + expect (rejects (R"({ "name": "t", "plugin": "P.vst3", "render_duration": 1e6, "sample_rate": 48000 })"), + "render_duration * sample_rate overflowing int"); + } + + beginTest ("Acceptance sample comparator fails on NaN"); + { + auto comparator = acceptance::createComparator ("sample"); + expect (comparator != nullptr); + + juce::AudioBuffer reference (1, 4), output (1, 4); + reference.clear(); + output.clear(); + + expect (comparator->compare (reference, output, nlohmann::ordered_json (0.0)).passed); + + output.setSample (0, 2, std::numeric_limits::quiet_NaN()); + const auto r = comparator->compare (reference, output, nlohmann::ordered_json (1.0)); + expect (! r.passed, "NaN must not pass, whatever the tolerance"); + expectEquals (r.details["first_fail_channel"].get(), 0); + expectEquals (r.details["first_fail_sample"].get(), 2); + } } }; diff --git a/source/SettingsParser.cpp b/source/SettingsParser.cpp index 0ae5ae06..45b61356 100644 --- a/source/SettingsParser.cpp +++ b/source/SettingsParser.cpp @@ -394,13 +394,31 @@ Precedence (lowest to highest): defaults, environment variables, --config, comma } else if (token == "--timeout-ms") { - if (i + 1 < tokensIn.size()) - result.timeoutMs = tokensIn.getReference (++i).getLargeIntValue(); + const auto value = i + 1 < tokensIn.size() ? tokensIn.getReference (++i) : juce::String(); + const auto digits = value.startsWith ("-") ? value.substring (1) : value; + + if (digits.isEmpty() || ! digits.containsOnly ("0123456789")) + { + result.error = "--timeout-ms needs a whole number of milliseconds (got \"" + value + "\")"; + break; + } + + result.timeoutMs = value.getLargeIntValue(); + } + else if (token.startsWith ("-")) + { + result.error = "unknown option for test: " + token; + break; } - else if (! token.startsWith ("-") && result.testConfigPath.isEmpty()) + else if (result.testConfigPath.isEmpty()) { result.testConfigPath = token; } + else + { + result.error = "unexpected argument for test: " + token + " (only one config can be given)"; + break; + } } return result; diff --git a/source/SettingsParser.h b/source/SettingsParser.h index 98f1705a..5d5ae2a7 100644 --- a/source/SettingsParser.h +++ b/source/SettingsParser.h @@ -83,6 +83,7 @@ namespace settings_parser juce::String testConfigPath; /**< The positional config.json for the test command. */ bool recordMissing = false; /**< test: record a missing reference (and pass) instead of failing. */ juce::int64 timeoutMs = 30000; /**< test: abort the run after this long (-1 to never time out). */ + juce::String error; /**< test: an invalid option / argument, to report instead of running. */ }; /** Routes tokenised input to a subcommand, peeling the leading verb. The diff --git a/source/acceptance/AcceptanceTest.cpp b/source/acceptance/AcceptanceTest.cpp index 8d29053d..26d67a80 100644 --- a/source/acceptance/AcceptanceTest.cpp +++ b/source/acceptance/AcceptanceTest.cpp @@ -24,6 +24,7 @@ #include #include #include +#include #include namespace acceptance @@ -172,12 +173,13 @@ namespace } else { - // Match the display name (case-insensitively), or the JUCE paramID - // when the host exposes parameters as AudioProcessorParameterWithID. + // Match the format's parameter ID, then the display name (both + // case-insensitively). Hosted parameters are HostedParameters, whose + // ID is the format's own (e.g. an LV2 symbol), not the JUCE paramID. for (auto* p : instance.getParameters()) { - if (auto* withID = dynamic_cast (p)) - if (withID->paramID.equalsIgnoreCase (name)) + if (auto* hosted = dynamic_cast (p)) + if (hosted->getParameterID().equalsIgnoreCase (name)) { match = p; break; @@ -199,16 +201,29 @@ namespace } //============================================================================== - juce::AudioBuffer readWav (juce::AudioFormatManager& formatManager, const juce::File& file) + struct AudioFileData + { + juce::AudioBuffer buffer; + double sampleRate = 0.0; + }; + + AudioFileData readWav (juce::AudioFormatManager& formatManager, const juce::File& file) { std::unique_ptr reader (formatManager.createReaderFor (file)); if (reader == nullptr) throw std::runtime_error (("could not read audio file: " + file.getFullPathName()).toStdString()); - juce::AudioBuffer buffer ((int) reader->numChannels, (int) reader->lengthInSamples); - reader->read (&buffer, 0, (int) reader->lengthInSamples, 0, true, true); - return buffer; + if (reader->lengthInSamples > std::numeric_limits::max()) + throw std::runtime_error (("audio file is too long: " + file.getFullPathName()).toStdString()); + + const auto numSamples = (int) reader->lengthInSamples; + AudioFileData data { juce::AudioBuffer ((int) reader->numChannels, numSamples), reader->sampleRate }; + + if (! reader->read (&data.buffer, 0, numSamples, 0, true, true)) + throw std::runtime_error (("failed reading audio file: " + file.getFullPathName()).toStdString()); + + return data; } void writeFloatWav (const juce::File& file, const juce::AudioBuffer& buffer, double sampleRate) @@ -258,6 +273,11 @@ namespace if (const auto* track = midiFile.getTrack (t)) for (const auto* event : *track) { + // Meta events (tempo, track name, end of track...) are file + // structure, not MIDI to send to the plugin. + if (event->message.isMetaEvent()) + continue; + const auto sample = (int) std::lround (event->message.getTimeStamp() * sampleRate); buffer.addEvent (event->message, sample); } @@ -288,7 +308,7 @@ namespace instead, and the plugin and reference locations are dropped. */ juce::String configHash (const TestConfig& config) { - auto j = nlohmann::json {}; + auto j = nlohmann::ordered_json {}; to_json (j, config); j.erase ("plugin"); j.erase ("reference"); @@ -311,7 +331,7 @@ namespace RenderedAudio renderPlugin (const TestConfig& config) { const auto sampleRate = config.sampleRate; - const auto blockSize = juce::jmax (1, config.blockSize); + const auto blockSize = config.blockSize; // validated by from_json juce::AudioPluginFormatManager formatManager; #if JUCE_VERSION >= 0x08000B @@ -341,7 +361,13 @@ RenderedAudio renderPlugin (const TestConfig& config) if (! audioFile.existsAsFile()) throw std::runtime_error (("input.audio not found: " + audioFile.getFullPathName()).toStdString()); - inputAudio = readWav (audioFormats, audioFile); + auto data = readWav (audioFormats, audioFile); + + if (data.sampleRate != sampleRate) + throw std::runtime_error (("input.audio sample rate (" + juce::String (data.sampleRate) + ") doesn't match sample_rate (" + + juce::String (sampleRate) + "): " + audioFile.getFullPathName()).toStdString()); + + inputAudio = std::move (data.buffer); } juce::MidiBuffer midi; @@ -356,7 +382,14 @@ RenderedAudio renderPlugin (const TestConfig& config) // 3. Render length. int numSamples = 0; if (config.renderDuration) - numSamples = (int) std::lround (*config.renderDuration * sampleRate); + { + const auto samples = std::llround (*config.renderDuration * sampleRate); + + if (samples > std::numeric_limits::max()) + throw std::runtime_error ("render_duration * sample_rate is too many samples"); + + numSamples = (int) samples; + } else if (inputAudio.getNumSamples() > 0) numSamples = inputAudio.getNumSamples(); @@ -493,7 +526,13 @@ TestResult runTest (const TestConfig& config, const RunOptions& options) juce::AudioFormatManager formats; formats.registerBasicFormats(); - const auto reference = readWav (formats, referenceFile); + const auto referenceData = readWav (formats, referenceFile); + const auto& reference = referenceData.buffer; + + if (referenceData.sampleRate != rendered.sampleRate) + return TestResult::makeError (name, "reference sample rate (" + juce::String (referenceData.sampleRate) + + ") doesn't match the render (" + juce::String (rendered.sampleRate) + "): " + + referenceFile.getFullPathName()); bool allPassed = true; @@ -514,6 +553,9 @@ TestResult runTest (const TestConfig& config, const RunOptions& options) result.outcome = allPassed ? TestResult::Outcome::passed : TestResult::Outcome::failed; + // Never leave a diff from an earlier failure behind. + config.getDiffFile().deleteFile(); + if (! allPassed) { // Write a diff WAV (output - reference) over the overlapping region. diff --git a/source/acceptance/ReferenceComparator.cpp b/source/acceptance/ReferenceComparator.cpp index 9853267c..a036898f 100644 --- a/source/acceptance/ReferenceComparator.cpp +++ b/source/acceptance/ReferenceComparator.cpp @@ -15,6 +15,7 @@ #include "ReferenceComparator.h" #include +#include namespace acceptance { @@ -25,13 +26,14 @@ namespace acceptance config is the value under the "sample" key: a bare number giving the tolerance (0 == bit-exact), or an object { "tolerance": }. Channel - and length mismatches fail outright. + and length mismatches fail outright, as does any non-finite (NaN / inf) + difference. */ struct SampleComparator : public Comparator { juce::String getName() const override { return "sample"; } - static double toleranceFrom (const nlohmann::json& config) + static double toleranceFrom (const nlohmann::ordered_json& config) { if (config.is_number()) return config.get(); @@ -45,7 +47,7 @@ struct SampleComparator : public Comparator ComparisonResult compare (const juce::AudioBuffer& reference, const juce::AudioBuffer& output, - const nlohmann::json& config) override + const nlohmann::ordered_json& config) override { const double tolerance = toleranceFrom (config); @@ -67,6 +69,7 @@ struct SampleComparator : public Comparator } double maxAbsDiff = 0.0; + bool nonFinite = false; int firstFailChannel = -1, firstFailSample = -1; for (int c = 0; c < reference.getNumChannels(); ++c) @@ -78,10 +81,14 @@ struct SampleComparator : public Comparator { const double diff = std::abs ((double) out[s] - (double) ref[s]); - if (diff > maxAbsDiff) + // NaN compares false both ways, so test for failure with + // ! (diff <= tolerance) and track non-finite diffs separately. + if (! std::isfinite (diff)) + nonFinite = true; + else if (diff > maxAbsDiff) maxAbsDiff = diff; - if (diff > tolerance && firstFailChannel < 0) + if (! (diff <= tolerance) && firstFailChannel < 0) { firstFailChannel = c; firstFailSample = s; @@ -89,9 +96,10 @@ struct SampleComparator : public Comparator } } - r.score = maxAbsDiff; - r.passed = maxAbsDiff <= tolerance; + r.score = nonFinite ? std::numeric_limits::infinity() : maxAbsDiff; + r.passed = firstFailChannel < 0; r.details["max_abs_diff"] = maxAbsDiff; + r.details["non_finite"] = nonFinite; if (r.passed) { @@ -99,8 +107,12 @@ struct SampleComparator : public Comparator } else { - r.summary = "max abs diff " + juce::String (maxAbsDiff) + " > tolerance " + juce::String (tolerance) - + " (first at channel " + juce::String (firstFailChannel) + ", sample " + juce::String (firstFailSample) + ")"; + const auto firstFail = " (first at channel " + juce::String (firstFailChannel) + ", sample " + juce::String (firstFailSample) + ")"; + + if (nonFinite) + r.summary = "non-finite (NaN / inf) difference; finite max abs diff " + juce::String (maxAbsDiff) + firstFail; + else + r.summary = "max abs diff " + juce::String (maxAbsDiff) + " > tolerance " + juce::String (tolerance) + firstFail; r.details["first_fail_channel"] = firstFailChannel; r.details["first_fail_sample"] = firstFailSample; } diff --git a/source/acceptance/ReferenceComparator.h b/source/acceptance/ReferenceComparator.h index 680c9d45..4f169ace 100644 --- a/source/acceptance/ReferenceComparator.h +++ b/source/acceptance/ReferenceComparator.h @@ -50,7 +50,7 @@ struct Comparator map (a bare number for "sample", or an object for richer methods). */ virtual ComparisonResult compare (const juce::AudioBuffer& reference, const juce::AudioBuffer& output, - const nlohmann::json& config) = 0; + const nlohmann::ordered_json& config) = 0; }; //============================================================================== diff --git a/source/acceptance/TestConfig.cpp b/source/acceptance/TestConfig.cpp index 94b69042..bdb25d0a 100644 --- a/source/acceptance/TestConfig.cpp +++ b/source/acceptance/TestConfig.cpp @@ -14,6 +14,8 @@ #include "TestConfig.h" +#include +#include #include namespace acceptance @@ -23,13 +25,13 @@ namespace acceptance namespace { /** The default comparison when none is supplied: one 16-bit LSB. */ - nlohmann::json defaultComparison() + nlohmann::ordered_json defaultComparison() { - return nlohmann::json { { "sample", 1.0 / 32768.0 } }; + return nlohmann::ordered_json { { "sample", 1.0 / 32768.0 } }; } /** Reads a string member from a (possibly absent) nested object. */ - std::string getNestedString (const nlohmann::json& j, const char* outer, const char* inner) + std::string getNestedString (const nlohmann::ordered_json& j, const char* outer, const char* inner) { if (auto o = j.find (outer); o != j.end() && o->is_object()) if (auto i = o->find (inner); i != o->end() && i->is_string()) @@ -40,15 +42,15 @@ namespace } //============================================================================== -void to_json (nlohmann::json& j, const TestConfig& c) +void to_json (nlohmann::ordered_json& j, const TestConfig& c) { - j = nlohmann::json::object(); + j = nlohmann::ordered_json::object(); j["name"] = c.name; j["plugin"] = c.plugin; if (! c.inputAudio.empty() || ! c.inputMidi.empty()) { - auto input = nlohmann::json::object(); + auto input = nlohmann::ordered_json::object(); if (! c.inputAudio.empty()) input["audio"] = c.inputAudio; if (! c.inputMidi.empty()) input["midi"] = c.inputMidi; j["input"] = input; @@ -59,9 +61,19 @@ void to_json (nlohmann::json& j, const TestConfig& c) if (! c.stateFile.empty() || ! c.stateParameters.empty()) { - auto state = nlohmann::json::object(); - if (! c.stateFile.empty()) state["file"] = c.stateFile; - if (! c.stateParameters.empty()) state["parameters"] = c.stateParameters; + auto state = nlohmann::ordered_json::object(); + if (! c.stateFile.empty()) state["file"] = c.stateFile; + + if (! c.stateParameters.empty()) + { + auto params = nlohmann::ordered_json::object(); + + for (const auto& [key, value] : c.stateParameters) + params[key] = value; + + state["parameters"] = params; + } + j["state"] = state; } @@ -85,8 +97,10 @@ void to_json (nlohmann::json& j, const TestConfig& c) } } -void from_json (const nlohmann::json& j, TestConfig& c) +void from_json (const nlohmann::ordered_json& j, TestConfig& c) { + const auto fail = [] (const std::string& message) { throw std::runtime_error (message); }; + c.name = j.value ("name", std::string()); c.plugin = j.value ("plugin", std::string()); c.reference = j.value ("reference", std::string()); @@ -98,16 +112,51 @@ void from_json (const nlohmann::json& j, TestConfig& c) if (auto state = j.find ("state"); state != j.end() && state->is_object()) if (auto params = state->find ("parameters"); params != state->end() && params->is_object()) - c.stateParameters = params->get>(); + for (const auto& [key, value] : params->items()) + c.stateParameters.emplace_back (key, value.get()); c.sampleRate = j.value ("sample_rate", 44100.0); c.blockSize = j.value ("block_size", 512); - if (auto d = j.find ("render_duration"); d != j.end() && d->is_number()) + if (auto d = j.find ("render_duration"); d != j.end() && ! d->is_null()) + { + if (! d->is_number()) + fail ("render_duration must be a number of seconds"); + c.renderDuration = d->get(); + } if (auto comp = j.find ("comparison"); comp != j.end() && ! comp->is_null()) + { + if (! comp->is_object() || comp->empty()) + fail ("comparison must be a non-empty object of method -> settings (omit it for the default)"); + c.comparison = *comp; + } + + // Validation. + if (c.plugin.empty()) + fail ("plugin is required"); + + if (c.name.empty() && c.reference.empty()) + fail ("name or reference is required"); + + if (! std::isfinite (c.sampleRate) || c.sampleRate <= 0.0) + fail ("sample_rate must be a positive number"); + + if (c.blockSize < 1 || c.blockSize > 65536) + fail ("block_size must be between 1 and 65536"); + + if (c.renderDuration) + { + const auto duration = *c.renderDuration; + + if (! std::isfinite (duration) || duration <= 0.0) + fail ("render_duration must be a positive number of seconds"); + + if (duration * c.sampleRate > (double) std::numeric_limits::max()) + fail ("render_duration * sample_rate is too many samples"); + } if (auto ph = j.find ("playhead"); ph != j.end() && ph->is_object()) { @@ -129,7 +178,7 @@ void from_json (const nlohmann::json& j, TestConfig& c) } //============================================================================== -nlohmann::json TestConfig::getComparison() const +nlohmann::ordered_json TestConfig::getComparison() const { return comparison.is_null() ? defaultComparison() : comparison; } @@ -200,11 +249,13 @@ std::vector TestConfig::loadFromFile (const juce::File& file) if (! file.existsAsFile()) throw std::runtime_error (("test config not found: " + file.getFullPathName()).toStdString()); - nlohmann::json j; + // ordered_json keeps object keys in file order (state.parameters is applied + // in that order). + nlohmann::ordered_json j; try { - j = nlohmann::json::parse (file.loadFileAsString().toStdString()); + j = nlohmann::ordered_json::parse (file.loadFileAsString().toStdString()); } catch (const std::exception& e) { @@ -213,7 +264,7 @@ std::vector TestConfig::loadFromFile (const juce::File& file) std::vector configs; - const auto addOne = [&] (const nlohmann::json& entry) + const auto addOne = [&] (const nlohmann::ordered_json& entry) { auto c = entry.get(); c.configDir = file.getParentDirectory(); diff --git a/source/acceptance/TestConfig.h b/source/acceptance/TestConfig.h index 3db74b3f..556a5823 100644 --- a/source/acceptance/TestConfig.h +++ b/source/acceptance/TestConfig.h @@ -17,9 +17,9 @@ #include #include -#include #include #include +#include #include namespace acceptance @@ -33,7 +33,8 @@ namespace acceptance of the "pluginval test" command. JSON keys are snake_case (e.g. sample_rate); the C++ members stay JUCE-style camelCase, so the mapping is done explicitly in the per-struct to_json / from_json (see TestConfig.cpp) - rather than the bare NLOHMANN macro. + rather than the bare NLOHMANN macro. It uses nlohmann::ordered_json so object + keys keep their file order (state.parameters is applied in that order). This config is completely independent of PluginvalSettings / the --config settings layering used by the validate command. @@ -57,11 +58,11 @@ struct TestConfig std::string inputMidi; /**< input.midi: path to a .mid file, or empty for none. */ std::string reference; /**< The golden file. Empty -> derived from name. */ std::string stateFile; /**< state.file: binary getStateInformation blob. */ - std::map stateParameters; /**< state.parameters: name-or-index -> normalised value. */ + std::vector> stateParameters; /**< state.parameters: name-or-index -> normalised value, in file order. */ double sampleRate = 44100.0; int blockSize = 512; std::optional renderDuration; /**< Seconds. Unset -> derive from the input length. */ - nlohmann::json comparison; /**< Map of comparator name -> sub-config. Empty -> default. */ + nlohmann::ordered_json comparison; /**< Map of comparator name -> sub-config. Null -> default. */ std::optional playhead; /**< Fixed transport. Unset -> no playhead supplied to the plugin. */ //============================================================================== @@ -73,7 +74,7 @@ struct TestConfig //============================================================================== /** Returns the resolved comparison map, substituting the default ({ "sample": 1/32768 }) when none was supplied. */ - nlohmann::json getComparison() const; + nlohmann::ordered_json getComparison() const; /** The plugin path or AU id, resolved against the working directory. */ juce::String getPluginPathOrID() const; @@ -110,7 +111,9 @@ struct TestConfig }; //============================================================================== -void to_json (nlohmann::json&, const TestConfig&); -void from_json (const nlohmann::json&, TestConfig&); +void to_json (nlohmann::ordered_json&, const TestConfig&); + +/** Throws std::runtime_error (or a nlohmann type error) on an invalid config. */ +void from_json (const nlohmann::ordered_json&, TestConfig&); } // namespace acceptance diff --git a/tests/acceptance/Acceptance testing design.md b/tests/acceptance/Acceptance testing design.md index fb128fae..836a8051 100644 --- a/tests/acceptance/Acceptance testing design.md +++ b/tests/acceptance/Acceptance testing design.md @@ -139,17 +139,17 @@ member names verbatim). Missing keys still fall back to the defaults. | Field | Type | Default | Notes | |---|---|---|---| -| `name` | string | basename of config file | Used to derive the default reference path and to label results. | -| `plugin` | string | — (required) | Absolute/relative path, or an AU identifier string. Relative paths resolve against the working directory. | -| `input.audio` | string | none | Path to an input audio file (WAV/AIFF/FLAC via `AudioFormatManager`). Used for effects. | -| `input.midi` | string | none | Path to a `.mid` file (`juce::MidiFile`). Used for instruments. | +| `name` | string | reference's basename | Used to derive the default reference path and to label results. `name` or `reference` is required. | +| `plugin` | string | (required) | Absolute/relative path, or an AU identifier string. Relative paths resolve against the working directory. | +| `input.audio` | string | none | Path to an input audio file (WAV/AIFF/FLAC via `AudioFormatManager`). Used for effects. Its sample rate must equal `sample_rate`. File channel N feeds plugin input channel N; a mono file feeds channel 0 only (deliberately not duplicated). | +| `input.midi` | string | none | Path to a `.mid` file (`juce::MidiFile`). Used for instruments. Meta events are skipped. | | `reference` | string | `/.wav` | The golden file. Absent -> record mode; present -> compare mode. | | `state.file` | string | none | Binary blob from `getStateInformation`, applied via the VST3-safe helper. | -| `state.parameters` | object | none | `name-or-index -> normalised value`. Applied after `state.file` if both are given. | -| `sample_rate` | number | 44100 | Single value (not a list, unlike validate). | -| `block_size` | number | 512 | Single value. | +| `state.parameters` | object | none | `name-or-index -> normalised value`, applied in file order (parsed with `nlohmann::ordered_json`). Applied after `state.file` if both are given. Digits-only keys are indices; otherwise the format's parameter ID, then the display name, case-insensitively. | +| `sample_rate` | number | 44100 | Single value (not a list, unlike validate). Must be positive and finite. | +| `block_size` | number | 512 | Single value, 1 to 65536. | | `render_duration` | number | input length | Seconds. `num_samples = round(duration * sample_rate)`. Input shorter than the duration is padded with silence; longer is truncated. | -| `comparison` | object | `{ "sample": 0.0000305 }` | Map of comparator name -> its sub-config. See §5. | +| `comparison` | object | `{ "sample": 0.0000305 }` | Map of comparator name -> its sub-config. See §5. An empty object is rejected. The `sample` comparator fails on any NaN / inf difference. | | `playhead` | object | none | Fixed transport for time-dependent plugins. `{ "bpm": , "time_signature": { "numerator": N, "denominator": D }, "start_ppq": }`. `time_signature` defaults to 4/4, `start_ppq` to 0. Omitted -> no playhead is set (the plugin sees `getPlayHead() == nullptr`). The tempo / time signature are constant; the position advances with the render. | | `automation` | array | none | **Phase 2.** Parameter changes scheduled at sample positions. | diff --git a/tests/acceptance/refs/gain-half.wav.json b/tests/acceptance/refs/gain-half.wav.json index e2704a33..ad69ab70 100644 --- a/tests/acceptance/refs/gain-half.wav.json +++ b/tests/acceptance/refs/gain-half.wav.json @@ -1,5 +1,5 @@ { - "config_hash": "4fb4289c542534c8", + "config_hash": "19abbba838eff61e", "created_on": { "arch": "arm64", "date": "2026-06-16T14:22:49.718+01:00", diff --git a/tests/acceptance/refs/playhead-120.wav.json b/tests/acceptance/refs/playhead-120.wav.json index 85ddde3a..7a25f3b1 100644 --- a/tests/acceptance/refs/playhead-120.wav.json +++ b/tests/acceptance/refs/playhead-120.wav.json @@ -1,5 +1,5 @@ { - "config_hash": "1f49c0804e8fa76f", + "config_hash": "2756114d8a1871ff", "created_on": { "arch": "arm64", "date": "2026-06-16T15:51:01.553+01:00", diff --git a/tests/acceptance/refs/sine-440.wav.json b/tests/acceptance/refs/sine-440.wav.json index 857f0486..07ab0488 100644 --- a/tests/acceptance/refs/sine-440.wav.json +++ b/tests/acceptance/refs/sine-440.wav.json @@ -1,5 +1,5 @@ { - "config_hash": "b8d9bce1db94737b", + "config_hash": "cfada3428594ae2a", "created_on": { "arch": "arm64", "date": "2026-06-16T13:10:20.117+01:00", diff --git a/tests/acceptance/refs/square-220.wav.json b/tests/acceptance/refs/square-220.wav.json index 69e7dc43..edc05288 100644 --- a/tests/acceptance/refs/square-220.wav.json +++ b/tests/acceptance/refs/square-220.wav.json @@ -1,5 +1,5 @@ { - "config_hash": "8a36b10bbd3c27a3", + "config_hash": "92e8e5abbb73ec76", "created_on": { "arch": "arm64", "date": "2026-06-16T13:10:20.377+01:00", diff --git a/tests/acceptance/refs/square-state.wav.json b/tests/acceptance/refs/square-state.wav.json index 5e49c1be..38823684 100644 --- a/tests/acceptance/refs/square-state.wav.json +++ b/tests/acceptance/refs/square-state.wav.json @@ -1,5 +1,5 @@ { - "config_hash": "1cdcebfe70935c8f", + "config_hash": "2ccce4612b4db9fc", "created_on": { "arch": "arm64", "date": "2026-06-16T13:10:20.606+01:00", From aca9956075963895d6bc2d4be18ec649a7c8e72d Mon Sep 17 00:00:00 2001 From: David Rowland Date: Thu, 8 Oct 2026 12:03:58 +0100 Subject: [PATCH 10/21] Version 2.0.0 Bumped VERSION to 2.0.0 and added acceptance testing to the 2.0.0 changelist. --- CHANGELIST.md | 1 + CLAUDE.md | 2 +- VERSION | 2 +- 3 files changed, 3 insertions(+), 2 deletions(-) diff --git a/CHANGELIST.md b/CHANGELIST.md index 6afdf839..4154f08a 100644 --- a/CHANGELIST.md +++ b/CHANGELIST.md @@ -1,6 +1,7 @@ # pluginval Change List ### 2.0.0 +- Added acceptance testing: `pluginval test ` renders a plugin with a known state, input (audio/MIDI) and optional playhead, then compares the output against a checked-in reference WAV, exiting `0` on a match and `1` on a mismatch (with a diff WAV). Use `--record-missing` to record new references. See `docs/Acceptance testing.md` - Restructured the command line into subcommands: `pluginval validate [options] ` (the default), `pluginval run-tests` and `pluginval strictness-help [level]`. The plugin path is now a positional argument of `validate` - **Deprecated:** the flat flags `--validate `, `--run-tests` and `--strictness-help [level]` still work as aliases (with a one-line notice) but will be removed in a future version. `pluginval ` remains a silent shorthand for `validate` - Replaced the hand-rolled command-line parser with CLI11 and a single JSON-based settings pipeline diff --git a/CLAUDE.md b/CLAUDE.md index dd355de5..4a168982 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -4,7 +4,7 @@ **pluginval** is a cross-platform audio plugin validator and tester application developed by Tracktion Corporation. It tests VST, VST3, AU (Audio Unit), LV2, and LADSPA plugins for compatibility and stability with host applications. -- **Version**: 1.0.4 (see `VERSION` file; a 2.0.0 entry is staged in `CHANGELIST.md` but `VERSION` is not yet bumped) +- **Version**: 2.0.0 (see `VERSION` file) - **License**: GPLv3 - **Framework**: Built on JUCE (v8.0.x) - **Language**: C++20 diff --git a/VERSION b/VERSION index ee90284c..227cea21 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.0.4 +2.0.0 From 031b77832c421829ef1514efd5c76d3a39dd5b02 Mon Sep 17 00:00:00 2001 From: Sudara Date: Sat, 18 Jul 2026 23:37:06 +0200 Subject: [PATCH 11/21] Skip the pluginval_cli target when the consuming project has no matching plugin target --- CMakeLists.txt | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index 714fb917..fdda65b7 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -266,16 +266,19 @@ else() set(PLUGINVAL_TARGET "${CMAKE_PROJECT_NAME}_VST3") endif() - get_target_property(artefact ${PLUGINVAL_TARGET} JUCE_PLUGIN_ARTEFACT_FILE) - - # TODO: This doesn't populate the executable in clion - add_custom_target(${CMAKE_PROJECT_NAME}_pluginval_cli - COMMAND $ - validate - --strictness-level 10 - ${artefact} - DEPENDS pluginval ${PLUGINVAL_TARGET} - COMMENT "Run pluginval CLI with strict validation") + # An umbrella/multi-product project has no ${CMAKE_PROJECT_NAME}_AU target, so skip instead of hard-erroring. + if (TARGET ${PLUGINVAL_TARGET}) + get_target_property(artefact ${PLUGINVAL_TARGET} JUCE_PLUGIN_ARTEFACT_FILE) + + # TODO: This doesn't populate the executable in clion + add_custom_target(${CMAKE_PROJECT_NAME}_pluginval_cli + COMMAND $ + validate + --strictness-level 10 + ${artefact} + DEPENDS pluginval ${PLUGINVAL_TARGET} + COMMENT "Run pluginval CLI with strict validation") + endif() endif() # In-repo dogfood plugins + acceptance self-tests. From d841ba663b6fe21b4435f807e1d023075a2d4979 Mon Sep 17 00:00:00 2001 From: David Rowland Date: Thu, 8 Oct 2026 12:10:13 +0100 Subject: [PATCH 12/21] Bump JUCE to 9.0.3 --- CMakeLists.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index fdda65b7..e682e9c6 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -81,7 +81,7 @@ endif() # Only fetch JUCE when top-level (when used as dependency, JUCE is already available) if (pluginval_IS_TOP_LEVEL) - CPMAddPackage("gh:juce-framework/juce#8.0.13") + CPMAddPackage("gh:juce-framework/juce#9.0.3") endif() From 08ffb973393a67e59c19ddf2a6c6ff56a3cf7f5a Mon Sep 17 00:00:00 2001 From: David Rowland Date: Thu, 8 Oct 2026 12:13:48 +0100 Subject: [PATCH 13/21] Test as-dependency against JUCE 9.0.3 and add libxi-dev to Linux CI deps --- .github/workflows/build.yaml | 6 +++--- tests/pluginval_as_dependency/CMakeLists.txt | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index 02725283..fbb9dcb0 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -67,7 +67,7 @@ jobs: run: | sudo add-apt-repository ppa:ubuntu-toolchain-r/test sudo apt-get update - sudo apt-get install -y freeglut3-dev g++ libasound2-dev libcurl4-openssl-dev libfreetype6-dev libjack-jackd2-dev libx11-dev libxcomposite-dev libxcursor-dev libxinerama-dev libxrandr-dev mesa-common-dev ladspa-sdk libwebkit2gtk-4.1-dev xvfb gdb + sudo apt-get install -y freeglut3-dev g++ libasound2-dev libcurl4-openssl-dev libfreetype6-dev libjack-jackd2-dev libx11-dev libxi-dev libxcomposite-dev libxcursor-dev libxinerama-dev libxrandr-dev mesa-common-dev ladspa-sdk libwebkit2gtk-4.1-dev xvfb gdb sudo /usr/bin/Xvfb $DISPLAY & - name: Make VST2 SDK available @@ -177,7 +177,7 @@ jobs: fail-fast: false matrix: os: [ubuntu-latest, macos-15, windows-latest] - juce: ["8.0.13"] + juce: ["8.0.13", "9.0.3"] runs-on: ${{ matrix.os }} steps: - uses: actions/checkout@v4 @@ -186,7 +186,7 @@ jobs: if: runner.os == 'Linux' run: | sudo apt-get update - sudo apt-get install -y libasound2-dev libfreetype6-dev libx11-dev libxcomposite-dev libxcursor-dev libxinerama-dev libxrandr-dev mesa-common-dev libwebkit2gtk-4.1-dev ladspa-sdk + sudo apt-get install -y libasound2-dev libfreetype6-dev libx11-dev libxi-dev libxcomposite-dev libxcursor-dev libxinerama-dev libxrandr-dev mesa-common-dev libwebkit2gtk-4.1-dev ladspa-sdk - name: Configure run: cmake -B build -S tests/pluginval_as_dependency -DCMAKE_BUILD_TYPE=${{ env.BUILD_TYPE}} -DJUCE_VERSION="${{ matrix.juce }}" diff --git a/tests/pluginval_as_dependency/CMakeLists.txt b/tests/pluginval_as_dependency/CMakeLists.txt index 0ea3743e..01d22b77 100644 --- a/tests/pluginval_as_dependency/CMakeLists.txt +++ b/tests/pluginval_as_dependency/CMakeLists.txt @@ -1,7 +1,7 @@ cmake_minimum_required(VERSION 3.22) project(PluginvalAsDependency VERSION 1.0.0) -set(JUCE_VERSION "8.0.3" CACHE STRING "JUCE version to test against") +set(JUCE_VERSION "9.0.3" CACHE STRING "JUCE version to test against") include(${CMAKE_CURRENT_LIST_DIR}/../../cmake/CPM.cmake) From 05e169bed161e01dc93b433724f137f6d25a3913 Mon Sep 17 00:00:00 2001 From: David Rowland Date: Thu, 8 Oct 2026 12:13:51 +0100 Subject: [PATCH 14/21] Add JUCE 9.0.3 update to the change list --- CHANGELIST.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELIST.md b/CHANGELIST.md index 4154f08a..edd972ae 100644 --- a/CHANGELIST.md +++ b/CHANGELIST.md @@ -1,6 +1,7 @@ # pluginval Change List ### 2.0.0 +- Updated JUCE to 9.0.3 - Added acceptance testing: `pluginval test ` renders a plugin with a known state, input (audio/MIDI) and optional playhead, then compares the output against a checked-in reference WAV, exiting `0` on a match and `1` on a mismatch (with a diff WAV). Use `--record-missing` to record new references. See `docs/Acceptance testing.md` - Restructured the command line into subcommands: `pluginval validate [options] ` (the default), `pluginval run-tests` and `pluginval strictness-help [level]`. The plugin path is now a positional argument of `validate` - **Deprecated:** the flat flags `--validate `, `--run-tests` and `--strictness-help [level]` still work as aliases (with a one-line notice) but will be removed in a future version. `pluginval ` remains a silent shorthand for `validate` From 8504562c7fdad6d221846ed80e45981846fb11d2 Mon Sep 17 00:00:00 2001 From: David Rowland Date: Thu, 8 Oct 2026 12:13:51 +0100 Subject: [PATCH 15/21] Update JUCE version mentions in docs --- CLAUDE.md | 2 +- README.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 4a168982..56ba3156 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ - **Version**: 2.0.0 (see `VERSION` file) - **License**: GPLv3 -- **Framework**: Built on JUCE (v8.0.x) +- **Framework**: Built on JUCE (v9.0.x) - **Language**: C++20 ### Key Features diff --git a/README.md b/README.md index 144a6bc9..4daf1471 100644 --- a/README.md +++ b/README.md @@ -49,7 +49,7 @@ CPMAddPackage("gh:tracktion/pluginval#develop") Then all you need to do is call `add_subdirectory ("modules/pluginval")` in your `CMakeLists.txt`. This should be done **after** your call to `juce_add_plugin`. -Note that only JUCE 8 is currently supported/tested for this method. +Note that only JUCE 8 and 9 are currently supported/tested for this method. ### Third-party Installation ###### _Chocolatey (Windows):_ From 09a7695fc7ff1386e7213433422229eb3834b058 Mon Sep 17 00:00:00 2001 From: David Rowland Date: Thu, 8 Oct 2026 12:43:46 +0100 Subject: [PATCH 16/21] Add libxi-dev to the acceptance workflow's Linux deps --- .github/workflows/acceptance.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/acceptance.yml b/.github/workflows/acceptance.yml index 80b1f18a..f7648647 100644 --- a/.github/workflows/acceptance.yml +++ b/.github/workflows/acceptance.yml @@ -43,7 +43,7 @@ jobs: run: | sudo apt-get update sudo apt-get install -y \ - libasound2-dev libx11-dev libxext-dev libxinerama-dev libxrandr-dev \ + libasound2-dev libx11-dev libxext-dev libxi-dev libxinerama-dev libxrandr-dev \ libxcursor-dev libxcomposite-dev libfreetype6-dev libfontconfig1-dev \ libgl1-mesa-dev libcurl4-openssl-dev ladspa-sdk ninja-build xvfb From 8f3b4a5312266345ff85310aad90765c3e333537 Mon Sep 17 00:00:00 2001 From: David Rowland Date: Mon, 9 Dec 2024 13:21:40 +0700 Subject: [PATCH 17/21] Refactored PR #144 --- source/CommandLine.cpp | 8 -- source/CommandLine.h | 4 +- source/CrashHandler.cpp | 113 +++------------------- source/Validator.cpp | 2 +- source/native/pluginval_native_posix.h | 125 +++++++++++++++++++++++++ 5 files changed, 143 insertions(+), 109 deletions(-) create mode 100644 source/native/pluginval_native_posix.h diff --git a/source/CommandLine.cpp b/source/CommandLine.cpp index 031c5649..f317557b 100644 --- a/source/CommandLine.cpp +++ b/source/CommandLine.cpp @@ -63,14 +63,6 @@ inline void logAndFlush (const juce::String& m) //============================================================================== //============================================================================== -CommandLineValidator::CommandLineValidator() -{ -} - -CommandLineValidator::~CommandLineValidator() -{ -} - void CommandLineValidator::validate (const juce::String& fileOrID, PluginTests::Options options) { validator = std::make_unique (fileOrID, options, ValidationType::inProcess, diff --git a/source/CommandLine.h b/source/CommandLine.h index 1a4f6061..ce6d153b 100644 --- a/source/CommandLine.h +++ b/source/CommandLine.h @@ -21,8 +21,8 @@ //============================================================================== struct CommandLineValidator { - CommandLineValidator(); - ~CommandLineValidator(); + CommandLineValidator() = default; + ~CommandLineValidator() = default; void validate (const juce::String&, PluginTests::Options); void runAcceptanceTest (const juce::File& configFile, const acceptance::RunOptions&); diff --git a/source/CrashHandler.cpp b/source/CrashHandler.cpp index 50ba8cf9..931d7ee1 100644 --- a/source/CrashHandler.cpp +++ b/source/CrashHandler.cpp @@ -15,14 +15,9 @@ #include "juce_core/juce_core.h" #include "CrashHandler.h" -#if !JUCE_WINDOWS - #include - #include - #include - #include - #include - #include - #define CRASH_LOG "/tmp/pluginval_crash.txt" +#if ! JUCE_WINDOWS + #include "native/pluginval_native_posix.h" + static const auto crashLogPath = "/tmp/pluginval_crash.txt"; #endif namespace @@ -34,105 +29,27 @@ namespace static juce::File getCrashTraceFile() { - #if JUCE_WINDOWS + #if JUCE_WINDOWS return juce::File::getSpecialLocation (juce::File::tempDirectory).getChildFile ("pluginval_crash.txt"); - #else - return juce::File(CRASH_LOG); - #endif + #else + return juce::File (crashLogPath); + #endif } - #if JUCE_WINDOWS - static void handleCrash(void*) + #if JUCE_WINDOWS + static void handleCrash (void*) { const auto log = getCrashLogContents(); std::cout << "\n*** FAILED: VALIDATION CRASHED\n" << log << std::endl; getCrashTraceFile().replaceWithText (log); } - #else - #ifdef __printflike - __printflike(2, 3) - #endif - static void writeToCrashLog(int fd, const char* fmt, ...) + #else + static void handleCrash (void*) { - char buf[1024]; - va_list args; - va_start(args, fmt); - // Warning: printf is not 100% async-signal-safe, but should be ok for locale-independent arguments like - // integers, strings, hex... floating point is locale-dependent and not safe to use here. - vsnprintf(buf, sizeof(buf), fmt, args); - va_end(args); - auto len = strlen(buf); - write(STDERR_FILENO, buf, len); - if (fd != -1) - write(fd, buf, len); - } - - static void handleCrash(void*) - { - // On Linux & Mac this is a signal handler, and therefore only "async-signal-safe" functions should be used. - // This means nothing that uses malloc (juce::File, juce::String, std::string, std::vector etc.) or buffered I/O. - - int fd = open(CRASH_LOG, O_RDWR | O_CREAT | O_TRUNC); - - const char *message = "\n*** FAILED: VALIDATION CRASHED\n"; - write(STDERR_FILENO, message, strlen(message)); - - // Recreate the output of backtrace_symbols(), which cannot be used in a signal handler because it uses malloc - static const int kMaxStacks = 128; - void *stacktrace[kMaxStacks] {}; - int stackCount = backtrace(stacktrace, kMaxStacks); - - static const int kMaxImages = 64; - const void *imageAddresses[kMaxImages] {}; - const char *imageNames[kMaxImages] {}; - int imageCount = 0; - - int skip = 2; // Skip handleCrash and juce::handleCrash) - for (int i = skip; i < stackCount; i++) - { - Dl_info info {}; - // Warning: dladdr can deadlock under rare conditions on macOS - if dyld is adding an image to its list - if (!dladdr(stacktrace[i], &info)) - { - writeToCrashLog(fd, "%-3d %-35s %p\n", i - skip, "", stacktrace[i]); - continue; - } - - const char *imageName = info.dli_fname ? strrchr(info.dli_fname, '/') : nullptr; - if (imageName) - { - imageName++; - - auto it = std::find(std::begin(imageAddresses), std::end(imageAddresses), info.dli_fbase); - if (it == std::end(imageAddresses) && imageCount < kMaxImages) - { - imageAddresses[imageCount] = info.dli_fbase; - imageNames[imageCount] = imageName; - imageCount++; - } - } - - if (info.dli_saddr) - { - ptrdiff_t offset = (char *)stacktrace[i] - (char *)info.dli_saddr; - writeToCrashLog(fd, "%-3d %-35s %p %s + %ld\n", i - skip, imageName, stacktrace[i], info.dli_sname, offset); - } - else - { - writeToCrashLog(fd, "%-3d %-35s %p\n", i - skip, imageName, stacktrace[i]); - } - } - - if (imageCount) - { - writeToCrashLog(fd, "\nBinary Images:"); - for (int i = 0; i < imageCount; i++) - writeToCrashLog(fd, "\n%p %s", imageAddresses[i], imageNames[i]); - writeToCrashLog(fd, "\n"); - } + const char* header = "\n*** FAILED: VALIDATION CRASHED\n"; + write (STDERR_FILENO, header, strlen (header)); - if (fd != -1) - close(fd); + writeStackTrace (crashLogPath, 2); // Skip handleCrash and juce::handleCrash) // Terminate normally to work around a bug in juce::ChildProcess::ActiveProcess::getExitStatus() // which returns 0 (a "pass" in the host process) if the child process terminates abnormally. @@ -140,7 +57,7 @@ namespace // - https://forum.juce.com/t/killed-childprocess-activeprocess-exit-code/61645/3 // Use _Exit() instead of exit() so that static destructors don't run (they may not be async-signal-safe). // FIXME: exiting here prevents Apple's Crash Reporter from creating reports. - std::_Exit(SIGKILL); + std::_Exit (SIGKILL); } #endif } diff --git a/source/Validator.cpp b/source/Validator.cpp index 529866f8..e9a6fdfa 100644 --- a/source/Validator.cpp +++ b/source/Validator.cpp @@ -22,7 +22,7 @@ //============================================================================== struct PluginsUnitTestRunner : public juce::UnitTestRunner, - private juce::Thread + private juce::Thread { PluginsUnitTestRunner (std::function logCallback, std::unique_ptr logDestination, juce::int64 timeoutInMs) : Thread ("TimoutThread"), diff --git a/source/native/pluginval_native_posix.h b/source/native/pluginval_native_posix.h new file mode 100644 index 00000000..b13c93b0 --- /dev/null +++ b/source/native/pluginval_native_posix.h @@ -0,0 +1,125 @@ +/*============================================================================== + + Copyright 2018 by Tracktion Corporation. + For more information visit www.tracktion.com + + You may also use this code under the terms of the GPL v3 (see + www.gnu.org/licenses). + + pluginval IS PROVIDED "AS IS" WITHOUT ANY WARRANTY, AND ALL WARRANTIES, WHETHER + EXPRESSED OR IMPLIED, INCLUDING MERCHANTABILITY AND FITNESS FOR PURPOSE, ARE + DISCLAIMED. + + ==============================================================================*/ + +#pragma once + +#if __has_include() + +#include +#include +#include +#include +#include +#include + +#ifdef __printflike + __printflike(2, 3) +#endif + +/** Writes a string to a file descriptor in a async-signal-safe way. I.e. no allocations etc. */ +inline void writeAsyncSignalSafe (int fd, const char* fmt, ...) +{ + char buf[1024]; + + va_list args; + va_start (args, fmt); + // Warning: printf is not 100% async-signal-safe, but should be ok for locale-independent arguments like + // integers, strings, hex... floating point is locale-dependent and not safe to use here. + vsnprintf (buf, sizeof (buf), fmt, args); + va_end (args); + + auto len = strlen (buf); + write (STDERR_FILENO, buf, len); + + if (fd != -1) + write (fd, buf, len); +} + +/** Writes the current stack trace and images to a given filepath and stderr. +*/ +inline void writeStackTrace (const char* filePath, const int numLinesToSkip = 0) +{ + // On Linux & Mac this is a signal handler, and therefore only "async-signal-safe" functions should be used. + // This means nothing that uses malloc (juce::File, juce::String, std::string, std::vector etc.) or buffered I/O. + int fd = open (filePath, O_RDWR | O_CREAT | O_TRUNC); + + // Write stack traces and images + { + // Recreate the output of backtrace_symbols(), which cannot be used in a + // signal handler because it uses malloc + constexpr int maxNumStacks = 128; + void *stacktrace[maxNumStacks]{}; + int stackCount = backtrace(stacktrace, maxNumStacks); + + constexpr int maxNumImages = 64; + const void *imageAddresses[maxNumImages]{}; + const char *imageNames[maxNumImages]{}; + int imageCount = 0; + + for (int i = numLinesToSkip; i < stackCount; ++i) + { + Dl_info info{}; + + // Warning: dladdr can deadlock under rare conditions on macOS - if dyld + // is adding an image to its list + if (! dladdr (stacktrace[i], &info)) + { + writeAsyncSignalSafe (fd, "%-3d %-35s %p\n", i - numLinesToSkip, "", stacktrace[i]); + continue; + } + + const char* imageName = info.dli_fname ? strrchr (info.dli_fname, '/') : nullptr; + + if (imageName) + { + ++imageName; + + auto it = std::find (std::begin (imageAddresses), std::end (imageAddresses), info.dli_fbase); + + if (it == std::end (imageAddresses) && imageCount < maxNumImages) + { + imageAddresses[imageCount] = info.dli_fbase; + imageNames[imageCount] = imageName; + ++imageCount; + } + } + + if (info.dli_saddr) + { + ptrdiff_t offset = static_cast (stacktrace[i]) - static_cast (info.dli_saddr); + writeAsyncSignalSafe (fd, "%-3d %-35s %p %s + %ld\n", i - numLinesToSkip, imageName, stacktrace[i], info.dli_sname, + offset); + } + else + { + writeAsyncSignalSafe (fd, "%-3d %-35s %p\n", i - numLinesToSkip, imageName, stacktrace[i]); + } + } + + if (imageCount > 0) + { + writeAsyncSignalSafe (fd, "\nBinary Images:"); + + for (int i = 0; i < imageCount; i++) + writeAsyncSignalSafe (fd, "\n%p %s", imageAddresses[i], imageNames[i]); + + writeAsyncSignalSafe (fd, "\n"); + } + } + + if (fd != -1) + close (fd); +} + + #endif //__has_include() \ No newline at end of file From 6b9855ebb828e06f2a3bcf55244f06ff5a75e3dc Mon Sep 17 00:00:00 2001 From: David Rowland Date: Mon, 9 Dec 2024 13:38:19 +0700 Subject: [PATCH 18/21] Fixed unused results --- source/CrashHandler.cpp | 2 +- source/native/pluginval_native_posix.h | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/source/CrashHandler.cpp b/source/CrashHandler.cpp index 931d7ee1..a0cd4190 100644 --- a/source/CrashHandler.cpp +++ b/source/CrashHandler.cpp @@ -47,7 +47,7 @@ namespace static void handleCrash (void*) { const char* header = "\n*** FAILED: VALIDATION CRASHED\n"; - write (STDERR_FILENO, header, strlen (header)); + [[ maybe_unused]] auto r = write (STDERR_FILENO, header, strlen (header)); writeStackTrace (crashLogPath, 2); // Skip handleCrash and juce::handleCrash) diff --git a/source/native/pluginval_native_posix.h b/source/native/pluginval_native_posix.h index b13c93b0..f266c91e 100644 --- a/source/native/pluginval_native_posix.h +++ b/source/native/pluginval_native_posix.h @@ -40,10 +40,10 @@ inline void writeAsyncSignalSafe (int fd, const char* fmt, ...) va_end (args); auto len = strlen (buf); - write (STDERR_FILENO, buf, len); + [[ maybe_unused]] auto r = write (STDERR_FILENO, buf, len); if (fd != -1) - write (fd, buf, len); + [[ maybe_unused]] auto r2 = write (fd, buf, len); } /** Writes the current stack trace and images to a given filepath and stderr. From 47b1368cc8d581c1a9648d92d25ec28403ec46b4 Mon Sep 17 00:00:00 2001 From: David Rowland Date: Thu, 8 Oct 2026 12:12:45 +0100 Subject: [PATCH 19/21] Fixed a linux build error by adding permissions to open --- source/native/pluginval_native_posix.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/native/pluginval_native_posix.h b/source/native/pluginval_native_posix.h index f266c91e..f2234407 100644 --- a/source/native/pluginval_native_posix.h +++ b/source/native/pluginval_native_posix.h @@ -52,7 +52,7 @@ inline void writeStackTrace (const char* filePath, const int numLinesToSkip = 0) { // On Linux & Mac this is a signal handler, and therefore only "async-signal-safe" functions should be used. // This means nothing that uses malloc (juce::File, juce::String, std::string, std::vector etc.) or buffered I/O. - int fd = open (filePath, O_RDWR | O_CREAT | O_TRUNC); + int fd = open (filePath, O_RDWR | O_CREAT | O_TRUNC, 00644); // Write stack traces and images { From afe42522b058c3d998d36a765254aee3b23ada86 Mon Sep 17 00:00:00 2001 From: David Rowland Date: Mon, 16 Dec 2024 08:02:02 +0000 Subject: [PATCH 20/21] Removed numLinesToSkip as this probably won't work now it's been broken out in to functions --- source/CrashHandler.cpp | 2 +- source/native/pluginval_native_posix.h | 10 +++++----- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/source/CrashHandler.cpp b/source/CrashHandler.cpp index a0cd4190..d56c8937 100644 --- a/source/CrashHandler.cpp +++ b/source/CrashHandler.cpp @@ -49,7 +49,7 @@ namespace const char* header = "\n*** FAILED: VALIDATION CRASHED\n"; [[ maybe_unused]] auto r = write (STDERR_FILENO, header, strlen (header)); - writeStackTrace (crashLogPath, 2); // Skip handleCrash and juce::handleCrash) + writeStackTrace (crashLogPath); // Terminate normally to work around a bug in juce::ChildProcess::ActiveProcess::getExitStatus() // which returns 0 (a "pass" in the host process) if the child process terminates abnormally. diff --git a/source/native/pluginval_native_posix.h b/source/native/pluginval_native_posix.h index f2234407..1e5cbd40 100644 --- a/source/native/pluginval_native_posix.h +++ b/source/native/pluginval_native_posix.h @@ -48,7 +48,7 @@ inline void writeAsyncSignalSafe (int fd, const char* fmt, ...) /** Writes the current stack trace and images to a given filepath and stderr. */ -inline void writeStackTrace (const char* filePath, const int numLinesToSkip = 0) +inline void writeStackTrace (const char* filePath) { // On Linux & Mac this is a signal handler, and therefore only "async-signal-safe" functions should be used. // This means nothing that uses malloc (juce::File, juce::String, std::string, std::vector etc.) or buffered I/O. @@ -67,7 +67,7 @@ inline void writeStackTrace (const char* filePath, const int numLinesToSkip = 0) const char *imageNames[maxNumImages]{}; int imageCount = 0; - for (int i = numLinesToSkip; i < stackCount; ++i) + for (int i = 0; i < stackCount; ++i) { Dl_info info{}; @@ -75,7 +75,7 @@ inline void writeStackTrace (const char* filePath, const int numLinesToSkip = 0) // is adding an image to its list if (! dladdr (stacktrace[i], &info)) { - writeAsyncSignalSafe (fd, "%-3d %-35s %p\n", i - numLinesToSkip, "", stacktrace[i]); + writeAsyncSignalSafe (fd, "%-3d %-35s %p\n", i, "", stacktrace[i]); continue; } @@ -98,12 +98,12 @@ inline void writeStackTrace (const char* filePath, const int numLinesToSkip = 0) if (info.dli_saddr) { ptrdiff_t offset = static_cast (stacktrace[i]) - static_cast (info.dli_saddr); - writeAsyncSignalSafe (fd, "%-3d %-35s %p %s + %ld\n", i - numLinesToSkip, imageName, stacktrace[i], info.dli_sname, + writeAsyncSignalSafe (fd, "%-3d %-35s %p %s + %ld\n", i, imageName, stacktrace[i], info.dli_sname, offset); } else { - writeAsyncSignalSafe (fd, "%-3d %-35s %p\n", i - numLinesToSkip, imageName, stacktrace[i]); + writeAsyncSignalSafe (fd, "%-3d %-35s %p\n", i, imageName, stacktrace[i]); } } From 1f1e34a4e223624e9bf7975b57e894259a195770 Mon Sep 17 00:00:00 2001 From: David Rowland Date: Mon, 12 Oct 2026 16:54:26 +0100 Subject: [PATCH 21/21] Crash handler: Updated change list and docs --- CHANGELIST.md | 1 + CLAUDE.md | 1 + 2 files changed, 2 insertions(+) diff --git a/CHANGELIST.md b/CHANGELIST.md index edd972ae..b53db341 100644 --- a/CHANGELIST.md +++ b/CHANGELIST.md @@ -2,6 +2,7 @@ ### 2.0.0 - Updated JUCE to 9.0.3 +- Fixed the crash handler so a backtrace is shown again when a plugin crashes, and made it async-signal-safe [#144] - Added acceptance testing: `pluginval test ` renders a plugin with a known state, input (audio/MIDI) and optional playhead, then compares the output against a checked-in reference WAV, exiting `0` on a match and `1` on a mismatch (with a diff WAV). Use `--record-missing` to record new references. See `docs/Acceptance testing.md` - Restructured the command line into subcommands: `pluginval validate [options] ` (the default), `pluginval run-tests` and `pluginval strictness-help [level]`. The plugin path is now a positional argument of `validate` - **Deprecated:** the flat flags `--validate `, `--run-tests` and `--strictness-help [level]` still work as aliases (with a one-line notice) but will be removed in a future version. `pluginval ` remains a silent shorthand for `validate` diff --git a/CLAUDE.md b/CLAUDE.md index 56ba3156..ff21a61e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -90,6 +90,7 @@ pluginval/ │ ├── PluginvalLookAndFeel.h # Custom UI styling │ ├── StrictnessInfoPopup.h # Strictness level info UI │ ├── binarydata/ # Binary resources (icons) +│ ├── native/ # Platform helpers (pluginval_native_posix.h: async-signal-safe stack trace writing) │ ├── vst3validator/ # Embedded VST3 validator integration │ │ ├── VST3ValidatorRunner.h │ │ └── VST3ValidatorRunner.cpp