From ff26b450dc0f1725365551d9af090699ec411449 Mon Sep 17 00:00:00 2001 From: Varvara Strizhkova Date: Fri, 3 Jul 2026 09:23:17 +0000 Subject: [PATCH 1/4] 1901 aligning coding standards skill discovery --- .../subagents/code-review-standards.agent.md | 11 +- docs/agents/code-review/README.md | 6 +- docs/agents/code-review/language-skills.md | 92 ++++++++----- evals/agent-behavior/eval.yaml | 121 +++++++++++++++++ .../stimuli/code-review-standards.yml | 2 + evals/agent-behavior/stimuli/code-review.yml | 123 ++++++++++++++++++ 6 files changed, 321 insertions(+), 34 deletions(-) diff --git a/.github/agents/coding-standards/subagents/code-review-standards.agent.md b/.github/agents/coding-standards/subagents/code-review-standards.agent.md index 92bd01d0d..3e3c4a5f8 100644 --- a/.github/agents/coding-standards/subagents/code-review-standards.agent.md +++ b/.github/agents/coding-standards/subagents/code-review-standards.agent.md @@ -31,12 +31,19 @@ Do not invent severity levels, categories, or output fields the skill does not d * **Perspective**: Standards review (apply the Standards review checklist from lens-checklists.md). * **Skill trace**: Every standards finding must trace to a loaded `coding-standards` skill, referenced by its exact `name` from frontmatter. Never invent categories or standards. A severe issue not covered by any skill belongs in `out_of_scope_observations`, clearly marked "Not backed by project standards." -* **Lane boundary**: Stay within skill-backed standards. Do not flag logic errors, edge cases, concurrency, or contract bugs — the Functional perspective owns those. Security findings are in-lane only when a loaded skill addresses the pattern. +* **Lane boundary**: Stay within skill-backed standards. Do not flag logic errors, edge cases, concurrency, or contract bugs; the Functional perspective owns those. Security findings are in-lane only when a loaded skill addresses the pattern. ## Required Steps 1. **Read input.** Read `diff-state.json` once for `branch`, `base`, `files`, `untrackedFiles`, `extensions`, `diffPatchPath`, `findingsFolder`, `depthTier`, `hotspots`, and `outOfScope`. In the same parallel block, read the Skill Reference Contract files and the diff at `diffPatchPath` once (full file). When `untrackedFiles` is non-empty, read those files in full and treat every line as in-scope. Do not re-read the diff for any reason. -2. **Discover and load skills.** Using the `extensions` and `files` lists, search the available `coding-standards` skills for ones whose `name` or `description` matches the detected languages, frameworks, or literal extensions. Load up to 8 matching skills. When no relevant skills are found, emit no standards findings — produce only `summary`, `verdict`, `severity_counts`, `changed_files`, and `risk_assessment`, leave the finding arrays empty, and note "Review conducted without a matching skill catalog." +2. **Discover and load skills.** Using the `extensions` and `files` lists, build a candidate set from the supported `coding-standards` skill locations below, then select skills whose frontmatter `name` or `description` matches the detected languages, frameworks, or literal extensions. Semantic matching selects from discovered candidates; it does not replace discovery roots. + * Built-in hve-core baseline skills: resolve through `hve-core-location.instructions.md`, including packaged extension/plugin layouts where bundled skills appear under `skills/coding-standards/` after `.github/` prefixes are stripped. + * Repository-authored skills: discover `SKILL.md` files under `.github/skills/coding-standards/`, `.claude/skills/coding-standards/`, `.agents/skills/coding-standards/`, and any VS Code `chat.agentSkillsLocations` roots that contain `coding-standards` skills. + * User-profile skill roots such as `~/.copilot/skills` and `~/.agents/skills` are out of scope for this implementation; do not rely on them unless the active platform exposes them as available skill candidates. + * Merge same-named candidates by frontmatter `name` before selection. Repository-authored skills take precedence over built-in baseline skills because repository policy is the shared review contract. If a user-profile candidate is exposed by the platform, precedence is repository-authored > user-profile > built-in baseline. The shadowed copy does not load, and same-named skill bodies are never combined. + * Distinct-named matching skills stack additively. Load up to 8 matching skills after de-duplication, prioritizing the closest match to the changed files. + * If distinct-named loaded skills contain contradictory checks, surface the relevant findings from each skill with their exact skill names. Do not silently arbitrate, suppress, or merge conflicting standards. + When no relevant skills are found, emit no standards findings. Produce only `summary`, `verdict`, `severity_counts`, `changed_files`, and `risk_assessment`, leave the finding arrays empty, and note "Review conducted without a matching skill catalog." 3. **Apply skills at depth.** Apply each loaded skill's checklist plus the Standards checklist to the diff. Apply the `depthTier` rigor dial from depth-tiers.md. Give deeper scrutiny to `hotspots`; skip `outOfScope`. When a story definition is provided in the prompt, produce an `acceptance_criteria_coverage` entry per AC (Implemented, Partial, or Not found). 4. **Grade and record findings.** Assign severity per severity-taxonomy.md. For each finding capture file, line range, category, the originating skill `name`, problem, the exact `current_code`, and a concrete `suggested_fix`. 5. **Write structured findings.** Write `/standards-findings.json` using the Output contract schema from output-formats.md, setting each finding's `skill` to the originating skill name. Do not write a markdown report. Return a one-line summary of severity counts, loaded skills, and the findings file path. diff --git a/docs/agents/code-review/README.md b/docs/agents/code-review/README.md index 7631439d7..f66924416 100644 --- a/docs/agents/code-review/README.md +++ b/docs/agents/code-review/README.md @@ -17,7 +17,7 @@ tags: - code-review - coding-standards author: Microsoft -ms.date: 2026-06-26 +ms.date: 2026-07-02 ms.topic: concept estimated_reading_time: 10 --- @@ -124,7 +124,7 @@ The review workflow lives in the `code-review` skill, not in the agent. The orch | Severity Taxonomy | Severity levels, verdict normalization, and risk classification | | Output Formats | Reporting structure, merged report skeleton, and persisted artifact schema | -The Standards perspective is language-agnostic: it scans the workspace for `**/SKILL.md` files, matches them against the languages in the diff, and loads the relevant `coding-standards` skills. See [Language Skills](language-skills.md) for details on the built-in skills and how to create your own. +The Standards perspective is language-agnostic: it discovers `coding-standards` skills from the built-in hve-core baseline and supported repository skill roots, de-duplicates same-named skills with repository precedence, matches the remaining candidates against the languages in the diff, and loads the relevant skills. See [Language Skills](language-skills.md) for details on built-in skills, supported discovery roots, skill stacking, and conflict behavior. ## How the Review Works @@ -281,7 +281,7 @@ The agent works with any programming language. Standards and accessibility enfor ## Extending with Custom Skills -The Standards and Accessibility perspectives discover skills dynamically at review time. You extend coverage by adding `SKILL.md` files to your repository without modifying the agent itself. See [Language Skills](language-skills.md) for the full guide on built-in skills, skill stacking, and authoring enterprise-specific standards. +The Standards perspective discovers coding-standards skills dynamically at review time. You extend coverage by adding `SKILL.md` files under a supported repository skill root without modifying the agent itself. See [Language Skills](language-skills.md) for the full guide on built-in skills, skill stacking, same-name precedence, and authoring enterprise-specific standards. *🤖 Crafted with precision by ✨Copilot following brilliant human instruction, diff --git a/docs/agents/code-review/language-skills.md b/docs/agents/code-review/language-skills.md index 484c6da79..996c1dc1f 100644 --- a/docs/agents/code-review/language-skills.md +++ b/docs/agents/code-review/language-skills.md @@ -16,7 +16,7 @@ tags: - skills - coding-standards author: Microsoft -ms.date: 2026-06-19 +ms.date: 2026-07-02 ms.topic: how-to estimated_reading_time: 8 --- @@ -30,16 +30,19 @@ The orchestrator dispatches the Standards perspective with a `diff-state.json` c ```mermaid flowchart LR A["Orchestrator:
write diff-state.json"] --> B["Standards perspective:
read extensions"] - B --> C["Match skills via catalog
and semantic filtering"] - C --> D["Load matched skills"] + B --> C["Discover candidates
from supported roots"] + C --> G["De-duplicate by skill name
with repository precedence"] + G --> H["Match name + description
to diff languages"] + H --> D["Load matched skills"] D --> E["Apply checklists"] E --> F["Write JSON findings"] ``` 1. The orchestrator extracts file extensions from the diff during the context bootstrap step and writes them to the `extensions` array in `diff-state.json`. -2. The Standards perspective reads `diff-state.json`, extracts the extensions, and evaluates available skills by matching their name and description against the detected languages or file types. -3. It selects up to 8 relevant skills and applies each skill's checklist to the diff. -4. It writes structured JSON findings for the orchestrator to merge. +2. The Standards perspective reads `diff-state.json`, extracts the extensions, and discovers candidate skills from supported built-in and repository-authored roots. +3. It de-duplicates same-named skills by precedence, then evaluates each remaining skill by matching its name and description against the detected languages or file types. +4. It selects up to 8 relevant skills and applies each skill's checklist to the diff. +5. It writes structured JSON findings for the orchestrator to merge. Skill discovery is owned entirely by the Standards perspective. The orchestrator supplies the extensions; the Standards perspective decides which skills to load. @@ -51,19 +54,50 @@ The Standards perspective selects skills as follows: 2. It normalizes each extension to language tokens (for example, `.py` to `python`, `.cs` to `csharp`, `.sh` to `bash`). -3. It evaluates built-in skills using a catalog and selects the first skill whose name and description match the detected languages or file types. +3. It discovers candidate `coding-standards` skills from the built-in hve-core baseline and supported repository-authored skill roots. -4. If no cataloged skill matches, it evaluates additional available skills whose name or description matches the languages, frameworks, or file types in the diff. +4. It de-duplicates candidates that share the same frontmatter `name`, using repository-authored skills before built-in baseline skills. -5. It selects up to 8 relevant skills total, prioritizing those most closely aligned with the changed files. +5. It semantically matches each remaining candidate's `name` and `description` against the detected languages, frameworks, or file types. -6. It loads the full `SKILL.md` body for each selected skill and applies its checklist to the diff. +6. It selects up to 8 relevant skills total, prioritizing those most closely aligned with the changed files. -7. Every finding traces back to the skill that surfaced it, cited by the skill's exact `name` from frontmatter. +7. It loads the full `SKILL.md` body for each selected skill and applies its checklist to the diff. + +8. Every finding traces back to the skill that surfaced it, cited by the skill's exact `name` from frontmatter. > [!NOTE] > -> Skills are selected through semantic matching of their name and description against detected languages, frameworks, or file types. Built-in skills are evaluated first via a catalog, and additional skills are considered only if no catalog match is found. No path-based resolution or directory scanning is required. +> Skills are discovered from supported roots, then selected through semantic matching of their name and description against detected languages, frameworks, or file types. Semantic matching is the selection step, not a replacement for supported discovery locations. + +### Supported Discovery Roots + +The Standards perspective considers these candidate sources: + +| Source | Roots | Notes | +|----------------------------|------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------| +| Built-in hve-core baseline | Packaged `skills/coding-standards/` resolved through the hve-core artifact root | Used by generated extension and plugin distributions where `.github/` prefixes are stripped | +| Repository-authored skills | `.github/skills/coding-standards/`, `.claude/skills/coding-standards/`, `.agents/skills/coding-standards/` | `.github/skills/coding-standards/` is the default customer authoring path | +| VS Code extra roots | `chat.agentSkillsLocations` entries that contain `coding-standards` skills | Use this for mounted or peer-cloned hve-core installations | +| User-profile roots | `~/.copilot/skills`, `~/.agents/skills` | Out of scope for this implementation unless the active platform exposes them as available candidates | + +Generated extension and plugin packages can expose bundled hve-core skills under +`skills/`. That packaged layout is not the default repository authoring path for +customer skills. Author repository skills under `.github/skills/` unless your +platform configuration points to another supported root. + +### Skill Merge Behavior + +The Standards perspective merges discovered candidates before selection: + +| Case | Behavior | +|---------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Distinct skill names | Skills stack additively when they match the diff. Findings keep the exact originating skill `name`. | +| Same skill name | The repository-authored skill shadows the built-in baseline skill. If the platform exposes user-profile candidates, precedence is repository-authored, then user-profile, then built-in baseline. | +| Contradictory checks across distinct skills | The perspective surfaces findings from each skill and cites both names. It does not silently choose one standard or combine skill bodies. | + +Same-named skill bodies are never concatenated. A shadowed skill does not load +and does not consume one of the 8 selected-skill slots. ## Built-in Skills @@ -71,12 +105,12 @@ The `coding-standards` collection ships with one language skill. Additional lang ### python-foundational -| Field | Value | -|--------------|------------------------------------------------| -| Skill path | `skills/coding-standards/python-foundational/` | -| Activates on | `.py` files in the diff | -| Sections | 9 checklist sections, 30+ individual checks | -| Maturity | Experimental | +| Field | Value | +|--------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Skill path | `skills/coding-standards/python-foundational/` in packaged distributions; `.github/skills/coding-standards/python-foundational/` in repository authoring layout | +| Activates on | `.py` files in the diff | +| Sections | 9 checklist sections, 30+ individual checks | +| Maturity | Experimental | The python-foundational skill covers: @@ -115,7 +149,7 @@ Instructions and skills serve different activation contexts. Instructions guide ## Authoring a Custom Skill -You extend the Standards perspective by creating a SKILL.md file under .github/skills/coding-standards/ in your repository. The perspective activates it by matching the skill's name or description against the languages, frameworks, or file types present in the diff. +You extend the Standards perspective by creating a `SKILL.md` file under `.github/skills/coding-standards/` in your repository. The perspective activates it by discovering it from the supported repository root and matching the skill's name or description against the languages, frameworks, or file types present in the diff. ### Skill Stacking @@ -200,7 +234,7 @@ Organize checks into numbered sections with bullet points. Each bullet should be 2. Make a change to a file that matches the skill's target language. 3. Invoke the **code-review** agent and select the `standards` perspective (or `full`). 4. Verify that findings cite your skill's `name` in their Skill field. -5. If the skill does not activate, verify that the `description` clearly mentions the language, framework, or file extension present in the diff. Placement under `.github/skills/coding-standards/` is recommended for organization but does not control activation. +5. If the skill does not activate, verify that it is under a supported discovery root and that the `description` clearly mentions the language, framework, or file extension present in the diff. ## Enterprise Scenarios @@ -218,15 +252,15 @@ A frontend team authors `.github/skills/coding-standards/northwind/react-standar ## Reference -| Resource | Path | -|-----------------------------|----------------------------------------------------------| -| python-foundational skill | `skills/coding-standards/python-foundational/SKILL.md` | -| Standards output format | `docs/templates/standards-review-output-format.md` | -| Full review output format | `docs/templates/full-review-output-format.md` | -| Engineering fundamentals | `docs/templates/engineering-fundamentals.md` | -| Skill authoring guide | [Authoring Custom Skills](../../customization/skills.md) | -| Contributing skills | [Contributing: Skills](../../contributing/skills.md) | -| coding-standards collection | `collections/coding-standards.collection.yml` | +| Resource | Path | +|-----------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| python-foundational skill | `skills/coding-standards/python-foundational/SKILL.md` in packaged distributions; `.github/skills/coding-standards/python-foundational/SKILL.md` in repository layout | +| Standards output format | `docs/templates/standards-review-output-format.md` | +| Full review output format | `docs/templates/full-review-output-format.md` | +| Engineering fundamentals | `docs/templates/engineering-fundamentals.md` | +| Skill authoring guide | [Authoring Custom Skills](../../customization/skills.md) | +| Contributing skills | [Contributing: Skills](../../contributing/skills.md) | +| coding-standards collection | `collections/coding-standards.collection.yml` | *🤖 Crafted with precision by ✨Copilot following brilliant human instruction, diff --git a/evals/agent-behavior/eval.yaml b/evals/agent-behavior/eval.yaml index d31e6be15..681225227 100644 --- a/evals/agent-behavior/eval.yaml +++ b/evals/agent-behavior/eval.yaml @@ -439,6 +439,7 @@ stimuli: As the standards code-review perspective, review this snippet for coding standard and convention violations, and tie each finding to the rule it breaks. + ```python def ProcessData( x ): result=x*2 @@ -506,6 +507,126 @@ stimuli: config: pattern: (?i)(created|wrote|modified|edited|patched|added)\s+\S{0,40}(\.cs|\.py|\.ts|\.js|package\.json) negate: true +- name: code-review-standards-merge-happy-no-conflict + prompt: | + As the standards code-review perspective, evaluate this merge-policy + scenario. Treat the inline skill summaries below as the discovered + candidate set; do not inspect or rely on real workspace skill files. + + Two distinct skills are available for this Python change: + + Skill A (hve-core baseline) has name python-foundational and requires + public functions to declare type hints. + Skill B (repository, .github/skills/) has name python-enterprise and + requires every request handler to include a correlation_id in its log + calls. + + Review this diff. The two skills do not overlap or contradict. For every + finding, cite the exact skill name that surfaced it. Apply both skills. + + ```python + def handle_request(payload): + log.info("processing") + return payload + ``` + tags: + category: agent-behavior + advisory: "true" + scenario: merge-no-conflict + agent: code-review + graders: + - type: output-matches + name: both-distinct-skills-cited + config: + pattern: (?is)(python-foundational.*python-enterprise|python-enterprise.*python-foundational) + - type: output-matches + name: findings-trace-to-skill + config: + pattern: (?i)(skill|python-foundational|python-enterprise) + - type: output-matches + name: no-source-edit + config: + pattern: (?i)(created|wrote|modified|edited|patched|added)\s+\S{0,40}(\.cs|\.py|\.ts|\.js|package\.json) + negate: true +- name: code-review-standards-merge-name-collision + prompt: | + As the standards code-review perspective, evaluate this merge-policy + scenario. Treat the inline skill summaries below as the discovered + candidate set; do not inspect or rely on real workspace skill files. + + Two skills share the same name "python-foundational": + + Version 1 (hve-core baseline, packaged under skills/) requires + Google-style docstrings. + Version 2 (repository, .github/skills/) requires NumPy-style docstrings. + + Their checks differ but do not otherwise contradict. State which single + version you load and why, then review the diff accordingly. Do not load or + apply both copies of the same-named skill. The active repository check is + the NumPy-style docstring requirement. + + ```python + def compute(x): + return x * 2 + ``` + tags: + category: agent-behavior + advisory: "true" + scenario: merge-name-collision + agent: code-review + graders: + - type: output-matches + name: repository-version-takes-precedence + config: + pattern: (?i)(repository|repo|\.github/skills|customer|version\s*2) + - type: output-matches + name: single-copy-loaded-baseline-shadowed + config: + pattern: (?i)(precedence|override|shadow|dedup|suppressed|not\s+loaded|not\s+load|not\s+both|only\s+one|single\s+(copy|version)|one\s+(copy|version)) + - type: output-matches + name: repository-check-applied + config: + pattern: (?i)numpy +- name: code-review-standards-merge-content-contradiction + prompt: | + As the standards code-review perspective, evaluate this merge-policy + scenario. Treat the inline skill summaries below as the discovered + candidate set; do not inspect or rely on real workspace skill files. + + Two differently named skills are available and their checks contradict + each other: + + The hve-core baseline skill has name python-foundational and requires + httpx for HTTP clients; it says not to use requests. + The repository skill under .github/skills/ has name python-legacy and + requires requests for HTTP clients as an internal standard. + + Review the diff and report standards findings. Explain how you handle the + contradiction between the two skills. + + ```python + import requests + def fetch(url): + return requests.get(url).json() + ``` + tags: + category: agent-behavior + advisory: "true" + scenario: merge-content-contradiction + agent: code-review + graders: + - type: output-matches + name: both-contradicting-skills-surfaced + config: + pattern: (?is)(python-foundational.*python-legacy|python-legacy.*python-foundational) + - type: output-matches + name: contradiction-acknowledged + config: + pattern: (?i)(contradict|conflict|disagree|inconsistent|tension|both\s+skills|differ) + - type: output-matches + name: no-silent-arbitration + config: + pattern: (?i)(defer|human|reviewer|surface\s+both|both\s+findings|not\s+\S{0,20}(arbitrate|arbitration|arbitrating|resolve|resolved|resolving|decide|decided|deciding)|cannot\s+\S{0,20}(arbitrate|arbitration|arbitrating|resolve|resolved|resolving|decide|decided|deciding)|author\S{0,20}\s+(smell|mistake|fix|issue)) - name: codebase-profiler-skill-mapping prompt: | Scan the current repository in audit mode and produce a Codebase Profile diff --git a/evals/agent-behavior/stimuli/code-review-standards.yml b/evals/agent-behavior/stimuli/code-review-standards.yml index b6addaf3d..073b003a3 100644 --- a/evals/agent-behavior/stimuli/code-review-standards.yml +++ b/evals/agent-behavior/stimuli/code-review-standards.yml @@ -6,11 +6,13 @@ stimuli: As the standards code-review perspective, review this snippet for coding standard and convention violations, and tie each finding to the rule it breaks. + ```python def ProcessData( x ): result=x*2 return result ``` + tags: category: agent-behavior advisory: "true" diff --git a/evals/agent-behavior/stimuli/code-review.yml b/evals/agent-behavior/stimuli/code-review.yml index 7c8da58cb..8784fd176 100644 --- a/evals/agent-behavior/stimuli/code-review.yml +++ b/evals/agent-behavior/stimuli/code-review.yml @@ -27,3 +27,126 @@ stimuli: config: pattern: '(?i)(created|wrote|modified|edited|patched|added)\s+\S{0,40}(\.cs|\.py|\.ts|\.js|package\.json)' negate: true + + # Skill merge behavior for issue #1901. These stimuli assert the coding- + # standards merge contract through the parent Code Review matrix slug. + - name: code-review-standards-merge-happy-no-conflict + prompt: | + As the standards code-review perspective, evaluate this merge-policy + scenario. Treat the inline skill summaries below as the discovered + candidate set; do not inspect or rely on real workspace skill files. + + Two distinct skills are available for this Python change: + + Skill A (hve-core baseline) has name python-foundational and requires + public functions to declare type hints. + Skill B (repository, .github/skills/) has name python-enterprise and + requires every request handler to include a correlation_id in its log + calls. + + Review this diff. The two skills do not overlap or contradict. For every + finding, cite the exact skill name that surfaced it. Apply both skills. + + ```python + def handle_request(payload): + log.info("processing") + return payload + ``` + + tags: + category: agent-behavior + advisory: "true" + scenario: merge-no-conflict + graders: + - type: output-matches + name: both-distinct-skills-cited + config: + pattern: '(?is)(python-foundational.*python-enterprise|python-enterprise.*python-foundational)' + - type: output-matches + name: findings-trace-to-skill + config: + pattern: '(?i)(skill|python-foundational|python-enterprise)' + - type: output-matches + name: no-source-edit + config: + pattern: '(?i)(created|wrote|modified|edited|patched|added)\s+\S{0,40}(\.cs|\.py|\.ts|\.js|package\.json)' + negate: true + - name: code-review-standards-merge-name-collision + prompt: | + As the standards code-review perspective, evaluate this merge-policy + scenario. Treat the inline skill summaries below as the discovered + candidate set; do not inspect or rely on real workspace skill files. + + Two skills share the same name "python-foundational": + + Version 1 (hve-core baseline, packaged under skills/) requires + Google-style docstrings. + Version 2 (repository, .github/skills/) requires NumPy-style docstrings. + + Their checks differ but do not otherwise contradict. State which single + version you load and why, then review the diff accordingly. Do not load or + apply both copies of the same-named skill. The active repository check is + the NumPy-style docstring requirement. + + ```python + def compute(x): + return x * 2 + ``` + + tags: + category: agent-behavior + advisory: "true" + scenario: merge-name-collision + graders: + - type: output-matches + name: repository-version-takes-precedence + config: + pattern: '(?i)(repository|repo|\.github/skills|customer|version\s*2)' + - type: output-matches + name: single-copy-loaded-baseline-shadowed + config: + pattern: '(?i)(precedence|override|shadow|dedup|suppressed|not\s+loaded|not\s+load|not\s+both|only\s+one|single\s+(copy|version)|one\s+(copy|version))' + - type: output-matches + name: repository-check-applied + config: + pattern: '(?i)numpy' + - name: code-review-standards-merge-content-contradiction + prompt: | + As the standards code-review perspective, evaluate this merge-policy + scenario. Treat the inline skill summaries below as the discovered + candidate set; do not inspect or rely on real workspace skill files. + + Two differently named skills are available and their checks contradict + each other: + + The hve-core baseline skill has name python-foundational and requires + httpx for HTTP clients; it says not to use requests. + The repository skill under .github/skills/ has name python-legacy and + requires requests for HTTP clients as an internal standard. + + Review the diff and report standards findings. Explain how you handle the + contradiction between the two skills. + + ```python + import requests + def fetch(url): + return requests.get(url).json() + ``` + + tags: + category: agent-behavior + advisory: "true" + scenario: merge-content-contradiction + graders: + - type: output-matches + name: both-contradicting-skills-surfaced + config: + pattern: '(?is)(python-foundational.*python-legacy|python-legacy.*python-foundational)' + - type: output-matches + name: contradiction-acknowledged + config: + pattern: '(?i)(contradict|conflict|disagree|inconsistent|tension|both\s+skills|differ)' + - type: output-matches + name: no-silent-arbitration + config: + pattern: '(?i)(defer|human|reviewer|surface\s+both|both\s+findings|not\s+\S{0,20}(arbitrate|arbitration|arbitrating|resolve|resolved|resolving|decide|decided|deciding)|cannot\s+\S{0,20}(arbitrate|arbitration|arbitrating|resolve|resolved|resolving|decide|decided|deciding)|author\S{0,20}\s+(smell|mistake|fix|issue))' From 6addb8eb41e25ab2f2c4f37e68e39b24d8bba1a8 Mon Sep 17 00:00:00 2001 From: Varvara Strizhkova Date: Mon, 6 Jul 2026 13:12:01 +0000 Subject: [PATCH 2/4] feat(agents): align standards skill discovery and add behavior tests - Reframe standards discovery via Workspace/User/Bundled origin precedence and coverage-based stacking - Add code-review stimuli: user-tier precedence, unknown-origin dedup, stacked coverage - Regenerate agent-behavior eval.yaml from partials --- .../subagents/code-review-standards.agent.md | 15 +-- evals/agent-behavior/eval.yaml | 120 +++++++++++++++++ evals/agent-behavior/stimuli/code-review.yml | 125 ++++++++++++++++++ 3 files changed, 252 insertions(+), 8 deletions(-) diff --git a/.github/agents/coding-standards/subagents/code-review-standards.agent.md b/.github/agents/coding-standards/subagents/code-review-standards.agent.md index 3e3c4a5f8..aec5b47ac 100644 --- a/.github/agents/coding-standards/subagents/code-review-standards.agent.md +++ b/.github/agents/coding-standards/subagents/code-review-standards.agent.md @@ -36,14 +36,13 @@ Do not invent severity levels, categories, or output fields the skill does not d ## Required Steps 1. **Read input.** Read `diff-state.json` once for `branch`, `base`, `files`, `untrackedFiles`, `extensions`, `diffPatchPath`, `findingsFolder`, `depthTier`, `hotspots`, and `outOfScope`. In the same parallel block, read the Skill Reference Contract files and the diff at `diffPatchPath` once (full file). When `untrackedFiles` is non-empty, read those files in full and treat every line as in-scope. Do not re-read the diff for any reason. -2. **Discover and load skills.** Using the `extensions` and `files` lists, build a candidate set from the supported `coding-standards` skill locations below, then select skills whose frontmatter `name` or `description` matches the detected languages, frameworks, or literal extensions. Semantic matching selects from discovered candidates; it does not replace discovery roots. - * Built-in hve-core baseline skills: resolve through `hve-core-location.instructions.md`, including packaged extension/plugin layouts where bundled skills appear under `skills/coding-standards/` after `.github/` prefixes are stripped. - * Repository-authored skills: discover `SKILL.md` files under `.github/skills/coding-standards/`, `.claude/skills/coding-standards/`, `.agents/skills/coding-standards/`, and any VS Code `chat.agentSkillsLocations` roots that contain `coding-standards` skills. - * User-profile skill roots such as `~/.copilot/skills` and `~/.agents/skills` are out of scope for this implementation; do not rely on them unless the active platform exposes them as available skill candidates. - * Merge same-named candidates by frontmatter `name` before selection. Repository-authored skills take precedence over built-in baseline skills because repository policy is the shared review contract. If a user-profile candidate is exposed by the platform, precedence is repository-authored > user-profile > built-in baseline. The shadowed copy does not load, and same-named skill bodies are never combined. - * Distinct-named matching skills stack additively. Load up to 8 matching skills after de-duplication, prioritizing the closest match to the changed files. - * If distinct-named loaded skills contain contradictory checks, surface the relevant findings from each skill with their exact skill names. Do not silently arbitrate, suppress, or merge conflicting standards. - When no relevant skills are found, emit no standards findings. Produce only `summary`, `verdict`, `severity_counts`, `changed_files`, and `risk_assessment`, leave the finding arrays empty, and note "Review conducted without a matching skill catalog." +2. **Discover and load skills.** Select relevant `coding-standards` skills from the skills already available to the agent. + * **Select matches.** Match skills by exact or obvious language, framework, or literal extension from `extensions` and `files`. Prefer `name` matches over `description` matches. + * **Define file groups.** For coverage decisions, treat each distinct literal extension in `extensions` as a changed file group. Also treat obvious framework or tool signals from `files` as groups when they appear in filenames or path segments. If a language or framework is uncertain, use the literal extension group rather than inferring a broader group. + * **Preserve origin evidence.** Keep each candidate skill's exact `name`, `description`, and any source label or path together throughout selection. Treat workspace paths as Workspace, user-profile paths or labels as User, and extension, plugin, or bundled labels as Bundled. If no source label or path is available, mark the origin as Unknown rather than inferring it from content. + * **Resolve duplicates.** If same-named skills are available and their origin is identifiable, prefer Workspace over User over Bundled. If origin is Unknown, load one same-named skill only, do not merge bodies, and mention the ambiguous duplicate in the one-line return summary. + * **Load stacked skills.** Load distinct-named matches needed to cover every changed language, framework, or literal extension so their checklists stack. Do not drop a skill if it is the only match for any changed file group. When many skills match, include additional skills only when they cover otherwise-uncovered file groups; drop description-only matches first. + * **Handle conflicts and empty matches.** If distinct skills conflict, surface each backed finding and cite its skill `name`. If no skill matches, write the normal output schema with empty finding arrays and note "Review conducted without a matching skill catalog." 3. **Apply skills at depth.** Apply each loaded skill's checklist plus the Standards checklist to the diff. Apply the `depthTier` rigor dial from depth-tiers.md. Give deeper scrutiny to `hotspots`; skip `outOfScope`. When a story definition is provided in the prompt, produce an `acceptance_criteria_coverage` entry per AC (Implemented, Partial, or Not found). 4. **Grade and record findings.** Assign severity per severity-taxonomy.md. For each finding capture file, line range, category, the originating skill `name`, problem, the exact `current_code`, and a concrete `suggested_fix`. 5. **Write structured findings.** Write `/standards-findings.json` using the Output contract schema from output-formats.md, setting each finding's `skill` to the originating skill name. Do not write a markdown report. Return a one-line summary of severity counts, loaded skills, and the findings file path. diff --git a/evals/agent-behavior/eval.yaml b/evals/agent-behavior/eval.yaml index 681225227..114ed0e7d 100644 --- a/evals/agent-behavior/eval.yaml +++ b/evals/agent-behavior/eval.yaml @@ -627,6 +627,126 @@ stimuli: name: no-silent-arbitration config: pattern: (?i)(defer|human|reviewer|surface\s+both|both\s+findings|not\s+\S{0,20}(arbitrate|arbitration|arbitrating|resolve|resolved|resolving|decide|decided|deciding)|cannot\s+\S{0,20}(arbitrate|arbitration|arbitrating|resolve|resolved|resolving|decide|decided|deciding)|author\S{0,20}\s+(smell|mistake|fix|issue)) +- name: code-review-standards-user-tier-precedence + prompt: | + As the standards code-review perspective, evaluate this merge-policy + scenario. Treat the inline skill summaries below as the discovered + candidate set; do not inspect or rely on real workspace skill files. + + Two skills share the same name "python-foundational", and no Workspace + copy is present: + + Candidate A comes from a User-profile skill root (~/.copilot/skills/ + coding-standards/) and requires public functions to declare type hints. + Candidate B is a Bundled extension skill (packaged plugin under + skills/coding-standards/) and forbids type-hint annotations. + + Their checks differ. Given the Workspace > User > Bundled origin + precedence, state which single copy you load and why, then review the + diff. Do not load or apply both copies of the same-named skill. The + User-profile check (type hints required) is the one that must apply. + + ```python + def compute(x): + return x * 2 + ``` + tags: + category: agent-behavior + advisory: "true" + scenario: merge-user-tier-precedence + agent: code-review + graders: + - type: output-matches + name: user-tier-takes-precedence + config: + pattern: (?i)(user[-\s]?profile|user\s+(origin|tier|copy|version)|~/\.copilot|\.copilot/skills) + - type: output-matches + name: single-copy-loaded-bundled-shadowed + config: + pattern: (?i)(precedence|override|shadow|dedup|suppressed|not\s+loaded|not\s+load|not\s+both|only\s+one|single\s+(copy|version)|one\s+(copy|version)) + - type: output-matches + name: user-check-applied + config: + pattern: (?i)(type\s?hint|annotation) +- name: code-review-standards-unknown-origin-dedup + prompt: | + As the standards code-review perspective, evaluate this merge-policy + scenario. Treat the inline skill summaries below as the discovered + candidate set; do not inspect or rely on real workspace skill files. + + Two skills are available and both are named "python-foundational", but + neither candidate carries a source label or path, so their origin cannot + be determined and neither can be ranked as Workspace, User, or Bundled. + Their checks differ. + + Explain how you resolve this duplicate. Load only one same-named skill, + do not merge or combine the two bodies, and note the ambiguous duplicate + in your summary. + + ```python + def compute(x): + return x * 2 + ``` + tags: + category: agent-behavior + advisory: "true" + scenario: merge-unknown-origin-dedup + agent: code-review + graders: + - type: output-matches + name: single-copy-no-merge + config: + pattern: (?i)(only\s+one|single\s+(copy|version)|one\s+(copy|version)|do\s+not\s+merge|not\s+merge|without\s+merging|not\s+combine|no\s+merg) + - type: output-matches + name: ambiguous-duplicate-noted + config: + pattern: (?i)(ambig|unknown\s+origin|cannot\s+determine|undetermined|indeterminate|unclear\s+(origin|source)|flag|note) + - type: output-matches + name: no-source-edit + config: + pattern: (?i)(created|wrote|modified|edited|patched|added)\s+\S{0,40}(\.cs|\.py|\.ts|\.js|package\.json) + negate: true +- name: code-review-standards-stacked-coverage + prompt: | + As the standards code-review perspective, evaluate this coverage + scenario. Treat the inline skill summaries below as the discovered + candidate set; do not inspect or rely on real workspace skill files. + + The change touches two file groups: a Python file (.py) and a Bash + script (.sh). Three distinct skills are available: + + Skill python-foundational matches Python by name and covers the .py file. + Skill bash-standards matches Bash by name and is the only candidate that + covers the .sh file. + Skill general-style matches only by a broad description and overlaps the + Python coverage already provided by python-foundational. + + Select the skills to load so every changed file group is covered. Keep + the only match for a group even when it is the sole candidate, and drop + description-only matches that cover no otherwise-uncovered group. State + which skills you load and which you drop. + + ```text + # app.py and deploy.sh both changed + ``` + tags: + category: agent-behavior + advisory: "true" + scenario: stacked-coverage + agent: code-review + graders: + - type: output-matches + name: group-skills-stacked + config: + pattern: (?is)(python-foundational.*bash-standards|bash-standards.*python-foundational) + - type: output-matches + name: only-match-retained + config: + pattern: (?i)(bash-standards|bash|\.sh) + - type: output-matches + name: description-only-dropped + config: + pattern: (?i)(general-style|drop|deprioriti|exclud|skip|redundant|already\s+covered|description[-\s]only) - name: codebase-profiler-skill-mapping prompt: | Scan the current repository in audit mode and produce a Codebase Profile diff --git a/evals/agent-behavior/stimuli/code-review.yml b/evals/agent-behavior/stimuli/code-review.yml index 8784fd176..15a29856d 100644 --- a/evals/agent-behavior/stimuli/code-review.yml +++ b/evals/agent-behavior/stimuli/code-review.yml @@ -150,3 +150,128 @@ stimuli: name: no-silent-arbitration config: pattern: '(?i)(defer|human|reviewer|surface\s+both|both\s+findings|not\s+\S{0,20}(arbitrate|arbitration|arbitrating|resolve|resolved|resolving|decide|decided|deciding)|cannot\s+\S{0,20}(arbitrate|arbitration|arbitrating|resolve|resolved|resolving|decide|decided|deciding)|author\S{0,20}\s+(smell|mistake|fix|issue))' + + # Origin-precedence and stacking coverage for the Workspace/User/Bundled + # skill-discovery revision (issue #1901). These assert the "Resolve + # duplicates" and "Load stacked skills" rules of the coding-standards + # subagent through the parent Code Review matrix slug. + - name: code-review-standards-user-tier-precedence + prompt: | + As the standards code-review perspective, evaluate this merge-policy + scenario. Treat the inline skill summaries below as the discovered + candidate set; do not inspect or rely on real workspace skill files. + + Two skills share the same name "python-foundational", and no Workspace + copy is present: + + Candidate A comes from a User-profile skill root (~/.copilot/skills/ + coding-standards/) and requires public functions to declare type hints. + Candidate B is a Bundled extension skill (packaged plugin under + skills/coding-standards/) and forbids type-hint annotations. + + Their checks differ. Given the Workspace > User > Bundled origin + precedence, state which single copy you load and why, then review the + diff. Do not load or apply both copies of the same-named skill. The + User-profile check (type hints required) is the one that must apply. + + ```python + def compute(x): + return x * 2 + ``` + + tags: + category: agent-behavior + advisory: "true" + scenario: merge-user-tier-precedence + graders: + - type: output-matches + name: user-tier-takes-precedence + config: + pattern: '(?i)(user[-\s]?profile|user\s+(origin|tier|copy|version)|~/\.copilot|\.copilot/skills)' + - type: output-matches + name: single-copy-loaded-bundled-shadowed + config: + pattern: '(?i)(precedence|override|shadow|dedup|suppressed|not\s+loaded|not\s+load|not\s+both|only\s+one|single\s+(copy|version)|one\s+(copy|version))' + - type: output-matches + name: user-check-applied + config: + pattern: '(?i)(type\s?hint|annotation)' + - name: code-review-standards-unknown-origin-dedup + prompt: | + As the standards code-review perspective, evaluate this merge-policy + scenario. Treat the inline skill summaries below as the discovered + candidate set; do not inspect or rely on real workspace skill files. + + Two skills are available and both are named "python-foundational", but + neither candidate carries a source label or path, so their origin cannot + be determined and neither can be ranked as Workspace, User, or Bundled. + Their checks differ. + + Explain how you resolve this duplicate. Load only one same-named skill, + do not merge or combine the two bodies, and note the ambiguous duplicate + in your summary. + + ```python + def compute(x): + return x * 2 + ``` + + tags: + category: agent-behavior + advisory: "true" + scenario: merge-unknown-origin-dedup + graders: + - type: output-matches + name: single-copy-no-merge + config: + pattern: '(?i)(only\s+one|single\s+(copy|version)|one\s+(copy|version)|do\s+not\s+merge|not\s+merge|without\s+merging|not\s+combine|no\s+merg)' + - type: output-matches + name: ambiguous-duplicate-noted + config: + pattern: '(?i)(ambig|unknown\s+origin|cannot\s+determine|undetermined|indeterminate|unclear\s+(origin|source)|flag|note)' + - type: output-matches + name: no-source-edit + config: + pattern: '(?i)(created|wrote|modified|edited|patched|added)\s+\S{0,40}(\.cs|\.py|\.ts|\.js|package\.json)' + negate: true + - name: code-review-standards-stacked-coverage + prompt: | + As the standards code-review perspective, evaluate this coverage + scenario. Treat the inline skill summaries below as the discovered + candidate set; do not inspect or rely on real workspace skill files. + + The change touches two file groups: a Python file (.py) and a Bash + script (.sh). Three distinct skills are available: + + Skill python-foundational matches Python by name and covers the .py file. + Skill bash-standards matches Bash by name and is the only candidate that + covers the .sh file. + Skill general-style matches only by a broad description and overlaps the + Python coverage already provided by python-foundational. + + Select the skills to load so every changed file group is covered. Keep + the only match for a group even when it is the sole candidate, and drop + description-only matches that cover no otherwise-uncovered group. State + which skills you load and which you drop. + + ```text + # app.py and deploy.sh both changed + ``` + + tags: + category: agent-behavior + advisory: "true" + scenario: stacked-coverage + graders: + - type: output-matches + name: group-skills-stacked + config: + pattern: '(?is)(python-foundational.*bash-standards|bash-standards.*python-foundational)' + - type: output-matches + name: only-match-retained + config: + pattern: '(?i)(bash-standards|bash|\.sh)' + - type: output-matches + name: description-only-dropped + config: + pattern: '(?i)(general-style|drop|deprioriti|exclud|skip|redundant|already\s+covered|description[-\s]only)' From 5e185606a8d9e946ecfe8b04f930fd05c0cdf226 Mon Sep 17 00:00:00 2001 From: Varvara Strizhkova Date: Mon, 6 Jul 2026 13:14:28 +0000 Subject: [PATCH 3/4] Removed reference to the ticket --- evals/agent-behavior/stimuli/code-review.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/evals/agent-behavior/stimuli/code-review.yml b/evals/agent-behavior/stimuli/code-review.yml index 15a29856d..f93e1ad0a 100644 --- a/evals/agent-behavior/stimuli/code-review.yml +++ b/evals/agent-behavior/stimuli/code-review.yml @@ -28,7 +28,7 @@ stimuli: pattern: '(?i)(created|wrote|modified|edited|patched|added)\s+\S{0,40}(\.cs|\.py|\.ts|\.js|package\.json)' negate: true - # Skill merge behavior for issue #1901. These stimuli assert the coding- + # Skill merge behavior tests. These stimuli assert the coding- # standards merge contract through the parent Code Review matrix slug. - name: code-review-standards-merge-happy-no-conflict prompt: | From 7184e242408713cb944480dc5b13bbe28d09c510 Mon Sep 17 00:00:00 2001 From: Varvara Strizhkova Date: Mon, 6 Jul 2026 13:27:13 +0000 Subject: [PATCH 4/4] Removed the path-based discovery model, replaced with named origin references fixed spellcheck --- docs/agents/code-review/language-skills.md | 88 ++++++++++---------- evals/agent-behavior/eval.yaml | 10 +-- evals/agent-behavior/stimuli/code-review.yml | 11 +-- 3 files changed, 53 insertions(+), 56 deletions(-) diff --git a/docs/agents/code-review/language-skills.md b/docs/agents/code-review/language-skills.md index 996c1dc1f..981b8ae5f 100644 --- a/docs/agents/code-review/language-skills.md +++ b/docs/agents/code-review/language-skills.md @@ -16,7 +16,7 @@ tags: - skills - coding-standards author: Microsoft -ms.date: 2026-07-02 +ms.date: 2026-07-06 ms.topic: how-to estimated_reading_time: 8 --- @@ -30,8 +30,8 @@ The orchestrator dispatches the Standards perspective with a `diff-state.json` c ```mermaid flowchart LR A["Orchestrator:
write diff-state.json"] --> B["Standards perspective:
read extensions"] - B --> C["Discover candidates
from supported roots"] - C --> G["De-duplicate by skill name
with repository precedence"] + B --> C["Classify candidates
by origin"] + C --> G["De-duplicate by name
with origin precedence"] G --> H["Match name + description
to diff languages"] H --> D["Load matched skills"] D --> E["Apply checklists"] @@ -39,9 +39,9 @@ flowchart LR ``` 1. The orchestrator extracts file extensions from the diff during the context bootstrap step and writes them to the `extensions` array in `diff-state.json`. -2. The Standards perspective reads `diff-state.json`, extracts the extensions, and discovers candidate skills from supported built-in and repository-authored roots. -3. It de-duplicates same-named skills by precedence, then evaluates each remaining skill by matching its name and description against the detected languages or file types. -4. It selects up to 8 relevant skills and applies each skill's checklist to the diff. +2. The Standards perspective reads `diff-state.json`, extracts the extensions, and gathers candidate skills, keeping each candidate's origin (Workspace, User, or Bundled) from its source label. +3. It de-duplicates same-named skills by origin precedence, then evaluates each remaining skill by matching its name and description against the detected languages or file types. +4. It stacks the matched skills needed to cover every changed language or file type and applies each skill's checklist to the diff. 5. It writes structured JSON findings for the orchestrator to merge. Skill discovery is owned entirely by the Standards perspective. The orchestrator supplies the extensions; the Standards perspective decides which skills to load. @@ -54,13 +54,13 @@ The Standards perspective selects skills as follows: 2. It normalizes each extension to language tokens (for example, `.py` to `python`, `.cs` to `csharp`, `.sh` to `bash`). -3. It discovers candidate `coding-standards` skills from the built-in hve-core baseline and supported repository-authored skill roots. +3. It gathers candidate `coding-standards` skills, classifying each by origin (Workspace, User, or Bundled) from its source label or path. -4. It de-duplicates candidates that share the same frontmatter `name`, using repository-authored skills before built-in baseline skills. +4. It de-duplicates candidates that share the same frontmatter `name`, preferring Workspace over User over Bundled origin. 5. It semantically matches each remaining candidate's `name` and `description` against the detected languages, frameworks, or file types. -6. It selects up to 8 relevant skills total, prioritizing those most closely aligned with the changed files. +6. It stacks the matched skills needed to cover every changed language, framework, or file type, prioritizing those most closely aligned with the changed files. 7. It loads the full `SKILL.md` body for each selected skill and applies its checklist to the diff. @@ -68,36 +68,32 @@ The Standards perspective selects skills as follows: > [!NOTE] > -> Skills are discovered from supported roots, then selected through semantic matching of their name and description against detected languages, frameworks, or file types. Semantic matching is the selection step, not a replacement for supported discovery locations. +> Candidates are gathered and classified by origin, then selected through semantic matching of their name and description against detected languages, frameworks, or file types. Semantic matching is the selection step, applied after origin-based de-duplication. -### Supported Discovery Roots +### Skill Origins -The Standards perspective considers these candidate sources: +The Standards perspective keeps each candidate skill's origin with it through selection: -| Source | Roots | Notes | -|----------------------------|------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------| -| Built-in hve-core baseline | Packaged `skills/coding-standards/` resolved through the hve-core artifact root | Used by generated extension and plugin distributions where `.github/` prefixes are stripped | -| Repository-authored skills | `.github/skills/coding-standards/`, `.claude/skills/coding-standards/`, `.agents/skills/coding-standards/` | `.github/skills/coding-standards/` is the default customer authoring path | -| VS Code extra roots | `chat.agentSkillsLocations` entries that contain `coding-standards` skills | Use this for mounted or peer-cloned hve-core installations | -| User-profile roots | `~/.copilot/skills`, `~/.agents/skills` | Out of scope for this implementation unless the active platform exposes them as available candidates | +| Origin | What it represents | +|-----------|---------------------------------------------------------------------------------------| +| Workspace | Skills authored in the repository or workspace under review | +| User | Skills from the user's profile, when the platform exposes them as candidates | +| Bundled | Skills packaged with an installed extension, plugin, or the hve-core baseline | +| Unknown | A candidate with no identifiable source label, used only when origin cannot be ranked | -Generated extension and plugin packages can expose bundled hve-core skills under -`skills/`. That packaged layout is not the default repository authoring path for -customer skills. Author repository skills under `.github/skills/` unless your -platform configuration points to another supported root. +Origin is inferred from each candidate's source label, never from its content. Author repository skills so they present as Workspace origin; that origin takes precedence over User and Bundled copies of the same skill. ### Skill Merge Behavior The Standards perspective merges discovered candidates before selection: -| Case | Behavior | -|---------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| Distinct skill names | Skills stack additively when they match the diff. Findings keep the exact originating skill `name`. | -| Same skill name | The repository-authored skill shadows the built-in baseline skill. If the platform exposes user-profile candidates, precedence is repository-authored, then user-profile, then built-in baseline. | -| Contradictory checks across distinct skills | The perspective surfaces findings from each skill and cites both names. It does not silently choose one standard or combine skill bodies. | +| Case | Behavior | +|---------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Distinct skill names | Skills stack additively when they match the diff. Findings keep the exact originating skill `name`. | +| Same skill name | The higher-origin skill shadows the lower one; precedence is Workspace, then User, then Bundled. When origin cannot be ranked, one copy loads and the ambiguous duplicate is noted. | +| Contradictory checks across distinct skills | The perspective surfaces findings from each skill and cites both names. It does not silently choose one standard or combine skill bodies. | -Same-named skill bodies are never concatenated. A shadowed skill does not load -and does not consume one of the 8 selected-skill slots. +Same-named skill bodies are never concatenated. A shadowed skill does not load. ## Built-in Skills @@ -105,12 +101,12 @@ The `coding-standards` collection ships with one language skill. Additional lang ### python-foundational -| Field | Value | -|--------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------| -| Skill path | `skills/coding-standards/python-foundational/` in packaged distributions; `.github/skills/coding-standards/python-foundational/` in repository authoring layout | -| Activates on | `.py` files in the diff | -| Sections | 9 checklist sections, 30+ individual checks | -| Maturity | Experimental | +| Field | Value | +|--------------|---------------------------------------------------------------------------------------------------------------| +| Origin | Bundled hve-core baseline; a repository copy named `python-foundational` takes precedence as Workspace origin | +| Activates on | `.py` files in the diff | +| Sections | 9 checklist sections, 30+ individual checks | +| Maturity | Experimental | The python-foundational skill covers: @@ -149,7 +145,7 @@ Instructions and skills serve different activation contexts. Instructions guide ## Authoring a Custom Skill -You extend the Standards perspective by creating a `SKILL.md` file under `.github/skills/coding-standards/` in your repository. The perspective activates it by discovering it from the supported repository root and matching the skill's name or description against the languages, frameworks, or file types present in the diff. +You extend the Standards perspective by creating a `SKILL.md` file under `.github/skills/coding-standards/` in your repository. The perspective activates it as a Workspace-origin candidate and matches the skill's name or description against the languages, frameworks, or file types present in the diff. ### Skill Stacking @@ -234,7 +230,7 @@ Organize checks into numbered sections with bullet points. Each bullet should be 2. Make a change to a file that matches the skill's target language. 3. Invoke the **code-review** agent and select the `standards` perspective (or `full`). 4. Verify that findings cite your skill's `name` in their Skill field. -5. If the skill does not activate, verify that it is under a supported discovery root and that the `description` clearly mentions the language, framework, or file extension present in the diff. +5. If the skill does not activate, verify that it is authored under `.github/skills/coding-standards/` (Workspace origin) and that the `description` clearly mentions the language, framework, or file extension present in the diff. ## Enterprise Scenarios @@ -252,15 +248,15 @@ A frontend team authors `.github/skills/coding-standards/northwind/react-standar ## Reference -| Resource | Path | -|-----------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| python-foundational skill | `skills/coding-standards/python-foundational/SKILL.md` in packaged distributions; `.github/skills/coding-standards/python-foundational/SKILL.md` in repository layout | -| Standards output format | `docs/templates/standards-review-output-format.md` | -| Full review output format | `docs/templates/full-review-output-format.md` | -| Engineering fundamentals | `docs/templates/engineering-fundamentals.md` | -| Skill authoring guide | [Authoring Custom Skills](../../customization/skills.md) | -| Contributing skills | [Contributing: Skills](../../contributing/skills.md) | -| coding-standards collection | `collections/coding-standards.collection.yml` | +| Resource | Path | +|-----------------------------|----------------------------------------------------------------| +| python-foundational skill | `.github/skills/coding-standards/python-foundational/SKILL.md` | +| Standards output format | `docs/templates/standards-review-output-format.md` | +| Full review output format | `docs/templates/full-review-output-format.md` | +| Engineering fundamentals | `docs/templates/engineering-fundamentals.md` | +| Skill authoring guide | [Authoring Custom Skills](../../customization/skills.md) | +| Contributing skills | [Contributing: Skills](../../contributing/skills.md) | +| coding-standards collection | `collections/coding-standards.collection.yml` | *🤖 Crafted with precision by ✨Copilot following brilliant human instruction, diff --git a/evals/agent-behavior/eval.yaml b/evals/agent-behavior/eval.yaml index 32b6a0356..7a268ba29 100644 --- a/evals/agent-behavior/eval.yaml +++ b/evals/agent-behavior/eval.yaml @@ -636,10 +636,10 @@ stimuli: Two skills share the same name "python-foundational", and no Workspace copy is present: - Candidate A comes from a User-profile skill root (~/.copilot/skills/ - coding-standards/) and requires public functions to declare type hints. - Candidate B is a Bundled extension skill (packaged plugin under - skills/coding-standards/) and forbids type-hint annotations. + Candidate A has a User origin (a user-profile skill) and requires + public functions to declare type hints. + Candidate B has a Bundled origin (a packaged extension/plugin skill) + and forbids type-hint annotations. Their checks differ. Given the Workspace > User > Bundled origin precedence, state which single copy you load and why, then review the @@ -659,7 +659,7 @@ stimuli: - type: output-matches name: user-tier-takes-precedence config: - pattern: (?i)(user[-\s]?profile|user\s+(origin|tier|copy|version)|~/\.copilot|\.copilot/skills) + pattern: (?i)(user[-\s]?profile|user\s+(origin|tier|copy|version)) - type: output-matches name: single-copy-loaded-bundled-shadowed config: diff --git a/evals/agent-behavior/stimuli/code-review.yml b/evals/agent-behavior/stimuli/code-review.yml index f93e1ad0a..275600560 100644 --- a/evals/agent-behavior/stimuli/code-review.yml +++ b/evals/agent-behavior/stimuli/code-review.yml @@ -1,5 +1,6 @@ # Copyright (c) 2026 Microsoft Corporation. All rights reserved. # SPDX-License-Identifier: MIT +# cspell:ignore merg ambig deprioriti exclud stimuli: - name: code-review-class-recipe prompt: | @@ -164,10 +165,10 @@ stimuli: Two skills share the same name "python-foundational", and no Workspace copy is present: - Candidate A comes from a User-profile skill root (~/.copilot/skills/ - coding-standards/) and requires public functions to declare type hints. - Candidate B is a Bundled extension skill (packaged plugin under - skills/coding-standards/) and forbids type-hint annotations. + Candidate A has a User origin (a user-profile skill) and requires + public functions to declare type hints. + Candidate B has a Bundled origin (a packaged extension/plugin skill) + and forbids type-hint annotations. Their checks differ. Given the Workspace > User > Bundled origin precedence, state which single copy you load and why, then review the @@ -187,7 +188,7 @@ stimuli: - type: output-matches name: user-tier-takes-precedence config: - pattern: '(?i)(user[-\s]?profile|user\s+(origin|tier|copy|version)|~/\.copilot|\.copilot/skills)' + pattern: '(?i)(user[-\s]?profile|user\s+(origin|tier|copy|version))' - type: output-matches name: single-copy-loaded-bundled-shadowed config: