feat(extensions): let extensions contribute always-on instructions - #4259
Open
TheovanKraay wants to merge 1 commit into
Open
feat(extensions): let extensions contribute always-on instructions#4259TheovanKraay wants to merge 1 commit into
TheovanKraay wants to merge 1 commit into
Conversation
…ithub#4200) Adds a provides.instructions capability so an extension can ship a compact always-on rule block that reaches the agent without any command/hook invocation. Ownership per maintainer decision: core validates the metadata only; the opt-in agent-context extension composes and owns the agent-file writes (namespaced <!-- SPECKIT EXT:<id> START/END --> blocks, per-agent routing, enable/disable/remove lifecycle). No agent-file writes when agent-context is not installed. core: accept+validate provides.instructions (path-safe, instructions-only extension allowed), expose .instructions. agent-context: compose enabled extensions' instruction blocks into the routed context file; bash/ps1 twins delegate to the python twin's --emit-extension-blocks for byte-identical output. Tests: tests/extensions/test_extension_instructions.py (13). Evidence: extensions/agent-context/INSTRUCTIONS-POC-EVIDENCE.md.
TheovanKraay
changed the title
Contributor
There was a problem hiding this comment.
Pull request overview
Adds opt-in, always-on extension instructions composed through agent-context.
Changes:
- Validates
provides.instructionsmanifests. - Composes enabled extensions’ instruction blocks across script variants.
- Adds validation/composition tests and supporting evidence.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
src/specify_cli/extensions/__init__.py |
Adds manifest validation and accessor. |
extensions/agent-context/scripts/python/update_agent_context.py |
Collects and renders instruction blocks. |
extensions/agent-context/scripts/bash/update-agent-context.sh |
Delegates block rendering to Python. |
extensions/agent-context/scripts/powershell/update-agent-context.ps1 |
Delegates block rendering to Python. |
tests/extensions/test_extension_instructions.py |
Tests validation and composition. |
extensions/agent-context/INSTRUCTIONS-POC-EVIDENCE.md |
Documents the prototype and evidence. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+204
to
+205
| def _collect_extension_instruction_blocks(project_root: str) -> list[tuple[str, str]]: | ||
| """Collect always-on instruction blocks from installed + enabled extensions. |
Comment on lines
+303
to
+307
| for ext_id, content in _collect_extension_instruction_blocks(project_root): | ||
| lines.append("") | ||
| lines.append(f"<!-- SPECKIT EXT:{ext_id} START -->") | ||
| lines.append(content) | ||
| lines.append(f"<!-- SPECKIT EXT:{ext_id} END -->") |
Comment on lines
+267
to
+270
| try: | ||
| parts.append(target.read_text(encoding="utf-8").strip()) | ||
| except OSError: | ||
| continue |
Comment on lines
+34
to
+39
| The delivered payload is the **same rule block** measured in the delivery A/B. Installed | ||
| via this path, the block written to `.github/copilot-instructions.md` is **byte-identical** | ||
| to the always-on rule block that scored **+0.142 mean** best-practice conformance over bare | ||
| (vs +0.10 for the same content as on-demand commands), across 2 models × 4 languages × | ||
| 3 complexity levels. Because the payload is identical, the measured lift carries over by | ||
| construction — this change is about **delivery/reachability**, not content or instruction |
| # Verify the candidate is a real, runnable Python 3 (skips the Windows Store | ||
| # 'python3' alias stub, mirroring the config-parse detection above). | ||
| try { | ||
| & $candidate -c "import sys; sys.exit(0 if sys.version_info[0] == 3 else 1)" 2>$null | Out-Null |
Comment on lines
+12
to
+15
| - **Core (`src/specify_cli/extensions/__init__.py`)** — accepts and validates a new | ||
| `provides: instructions:` capability (list of `{ file, description? }`), path-safe via | ||
| the existing `relative_extension_path_violation` guard, exposed as `.instructions`. | ||
| Core performs **no** agent-file writes. An instructions-only extension is valid. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Part of #4200.
Adds a
provides.instructionscapability so a Spec Kit extension can ship a compact,always-on rule block that reaches the agent without any command or hook being invoked —
the reachability gap discussed in #4200. Ownership follows what we agreed on in the thread:
agent-contextextension composes enabled extensions' blocks into the routedagent file (
.github/copilot-instructions.md,CLAUDE.md, …) as namespaced<!-- SPECKIT EXT:<id> START/END -->blocks, and owns routing, refresh, update, and removal.agent-contextisn't installed, nothing touches any agent file. The key is additive anda no-op on cores without this change.
Why it's needed: today nothing delivers an extension's guidance automatically on install in a way
that survives a hands-off agent run. This adds that always-on channel, scoped to an opt-in extension.
Efficacy (same rule block + conformance metric throughout):
0.843, B always-on0.986,C rules in an invoked command
0.989, D same command never invoked0.831.C − B = +0.003,payload present for B/C and absent for D — so the effect is reachability (present vs absent), not
obedience.
22 wins / 0 ties / 2 losses. The block that lands in
copilot-instructions.mdon a real installis byte-identical to the one measured.
One test not completed (for transparency): a behavioral A/B on an internal at-scale agent-execution
harness is currently inconclusive due to an agent-runtime regression unrelated to this change — the
agent crashed on startup on every attempt (both arms, incl. bare), confirmed by an untouched baseline
image failing identically. In-container delivery of the block was verified separately. That datapoint
is pending a healthy agent build and is not reported as a result.
Testing
uv run specify --helpuv sync && uv run pytestNew tests
tests/extensions/test_extension_instructions.py: 13 passed (validation accept/reject,composition, disable-cleanup, multi-extension ordering, path-unsafe skip, no-op without
agent-context,emit mode). Full suite on this branch, rebased on current
main: 6916 passed, 415 skipped, 0 failed.bash/powershell/pythonemit paths verified byte-identical. End-to-end verified in a sampleproject: after
specify extension add <ext> --dev+specify extension add /agent-context --dev, thenamespaced block appears in
.github/copilot-instructions.md, and enable/disable/remove clean up correctly.AI Disclosure
Implemented with GitHub Copilot (agentic) — the core validation, the
agent-contextcompose logic andits bash/powershell/python twins, and the test suite were written with AI assistance and reviewed by me.
The empirical evaluation (the pre-check and the +0.123 conformance measurement) was also run and analyzed
with AI assistance.