Assert every command placeholder names a command that exists - #4249
Open
Hermann-web wants to merge 1 commit into
Open
Assert every command placeholder names a command that exists#4249Hermann-web wants to merge 1 commit into
Hermann-web wants to merge 1 commit into
Conversation
`resolve_command_refs` rewrites `__SPECKIT_COMMAND_<NAME>__` with a regex and never consults a list of commands, so a misspelled placeholder does not fail. `__SPECKIT_COMMAND_ASSES_SHAPE__` becomes `/speckit.asses.shape` and ships into agent instructions as a plausible command no agent can run. Existing coverage asserts the rewrite uses the right separator and prefix, which is the mechanism rather than the target. Nothing checks that the target exists. The command set is derived from `templates/commands/*.md` and from the `provides.commands` entries of every extension and preset manifest, rather than restated in the test, so a command added or renamed is covered without editing this module. A second assertion keeps core templates from referencing a command only an extension provides, since a core template has to work when that extension is not installed. Both assertions were confirmed to fail on an injected typo and an injected extension reference before being committed. No production code changes.
Hermann-web
force-pushed
the
chore/command-placeholder-target-coverage
branch
from
August 21, 2026 13:52
e8fd5f4 to
5468ad7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
IntegrationBase.resolve_command_refsrewrites__SPECKIT_COMMAND_<NAME>__with a bare regex and never consults a list of commands:
Any name that matches the pattern is rewritten. A typo such as
__SPECKIT_COMMAND_ASSES_SHAPE__becomes/speckit.asses.shape, which iswritten into a user's agent instructions as a plausible-looking command that no
agent can run. There is no error, and the failure surfaces only when someone
tries the command.
Existing coverage checks the mechanism, not the target:
tests/integrations/test_base.pyasserts the separator variants are produced;tests/test_agent_config_consistency.pyasserts extension tokens resolve withthe correct separator.
Neither asserts that the resolved command exists. This matters more now that
extensions/catalog.community.jsoncarries 157 community extensions whoseauthors are told to use these placeholders.
Change
One new test module,
tests/test_command_placeholder_targets.py. No productioncode changes.
The command set is derived rather than restated:
speckit.<stem>for everytemplates/commands/*.md;provides.commands[].nameandaliasesfrom eachextensions/*/extension.ymlandpresets/*/preset.yml.A command added or renamed is therefore covered without editing the test.
It asserts:
templates/,extensions/,presets/) resolves to a command that exists.core template has to work when that extension is not installed.
shipped directories fails loudly instead of scanning nothing and passing.
resolve_command_refsimplements,checked against the real function rather than reimplemented on faith.
Verification
Passes on current
main(95efce4): the repository is clean today, with all 17placeholders resolving and no core template leaking an extension reference.
Both assertions were confirmed to fail before committing, by appending
__SPECKIT_COMMAND_ASSES_SHAPE__and__SPECKIT_COMMAND_BUG_FIX__totemplates/commands/plan.md:The injection was reverted; the diff is the new file only.
pytest tests/test_command_placeholder_targets.pyis 6 passed.pytest tests/test_agent_config_consistency.py tests/integrations/test_base.py tests/test_commands_package.py tests/test_command_placeholder_targets.pyis125 passed.
The full suite was also run: 6958 passed, 155 skipped, and 26 failures all in
test_ps_*PowerShell parity tests becausepwshis not installed on themachine used. No non-PowerShell failures.
Scope
Deliberately small and additive, per CONTRIBUTING's note about large unagreed
changes. It adds coverage for an existing behaviour and changes nothing that
ships to users.