diff --git a/docs/architecture/agentic-workflows.md b/docs/architecture/agentic-workflows.md index bd474abc0..46595a08c 100644 --- a/docs/architecture/agentic-workflows.md +++ b/docs/architecture/agentic-workflows.md @@ -2,7 +2,7 @@ title: Agentic Workflows description: End-to-end process flow for AI-driven issue triage, implementation, and review workflows in hve-core author: HVE Core Team -ms.date: 2026-07-16 +ms.date: 2026-07-31 ms.topic: concept sidebar_position: 4 keywords: @@ -32,8 +32,6 @@ flowchart TD C["Classify by type
and component"] D["Detect duplicates
via keyword search"] E["Assess issue quality"] - F{"Scope too broad
for single deliverable?"} - G["Decompose into
sub-issues"] H{"Passes all
agent-ready criteria?"} I["Apply labels,
remove needs-triage"] J["Add agent-ready label"] @@ -66,10 +64,7 @@ flowchart TD B --> C C --> D D --> E - E --> F - F -- Yes --> G - G --> I - F -- No --> H + E --> H H -- Yes --> I I --> J H -- No --> I @@ -94,7 +89,7 @@ flowchart TD | Workflow | Trigger | Execution Owner | Key Actions | |----------------------|---------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------| -| Issue Triage | Issue opened or labeled `needs-triage` | [Issue Triage Agent](https://github.com/microsoft/hve-core/blob/main/.github/agents/issue-triage.agent.md) | Classify, detect duplicates, assess quality, decompose, label, evaluate readiness | +| Issue Triage | Issue opened or labeled `needs-triage` | [Issue Triage Agent](https://github.com/microsoft/hve-core/blob/main/.github/agents/issue-triage.agent.md) | Classify, detect duplicates, assess quality, label, evaluate readiness | | Issue Implementation | Issue labeled `agent-ready` | Workflow-owned procedure in `.github/workflows/issue-implement.md` | Research codebase, plan changes, implement, open PR | | PR Review | PR opened or marked ready for review | [Code Review Agent](https://github.com/microsoft/hve-core/blob/main/.github/agents/coding-standards/code-review.agent.md) | Review correctness, conventions, security; label `review-passed` or `needs-revision` for non-maintainer PRs, advisory `COMMENT` only for maintainer PRs | | Dependabot PR Review | Dependabot PR opened or updated | [Dependency Reviewer Agent](https://github.com/microsoft/hve-core/blob/main/.github/agents/dependency-reviewer.agent.md) | Validate licensing, SHA pinning, environment sync; approve safe bumps | @@ -102,7 +97,7 @@ flowchart TD | VEX Draft | `workflow_run` after VEX Detection succeeds, or `workflow_dispatch` | [SSSC Reviewer](https://github.com/microsoft/hve-core/blob/main/.github/agents/security/sssc-reviewer.agent.md) | Enrich CVEs, analyze reachability, open one PR with OpenVEX draft statements for human review | > [!TIP] -> The triage agent only classifies, labels, and optionally decomposes issues. It does not close issues, assign users, or modify issue titles. +> The triage agent classifies issues, applies type, area, and priority labels, detects duplicates, assesses quality, and marks qualifying issues `agent-ready`. It does not create sub-issues, close issues, assign users, or modify issue titles. @@ -131,6 +126,22 @@ All six workflows are defined as GitHub Agentic Workflow markdown files under `. Each workflow file declares permissions, safe output limits, and activation guards that prevent unintended execution. +### Lock File Ownership + +The `*.lock.yml` files under `.github/workflows/` are generated outputs of `gh aw compile`. Editing them directly is not supported: the next compile overwrites the change. Edit the source `.md` workflow instead, then recompile. + +Because these files are generated, Dependabot is configured to leave them alone. `.github/dependabot.yml` excludes the `.github/workflows/*.lock.yml` path and the `github/gh-aw-actions/*` action family, so action bumps inside a lock file never arrive as a pull request. + +### Upgrading gh-aw-actions + +The `gh-aw-actions` version is pinned in `.github/aw/actions-lock.json`, which maps each action reference to a resolved commit SHA. This file, not Dependabot, is the upgrade path: + +1. Upgrade the `gh aw` CLI/compiler (the resolved `gh-aw-actions` version is tied to the compiler release, not set independently). +2. Run `gh aw compile` (or `gh aw update-actions`) with API access to re-resolve actions and regenerate `actions-lock.json` plus every `*.lock.yml` file. +3. Commit `actions-lock.json` and the regenerated lock files together. + +The pinned version is version-locked to the `gh aw` compiler that produces the lock files, so the bump and the recompile belong in the same change. + ## Label-Driven Handoffs Labels serve as the event bus connecting workflows. Each label transition triggers the next stage: diff --git a/docs/contributing/evals-ci.md b/docs/contributing/evals-ci.md index 1bea35d75..a2b9b415a 100644 --- a/docs/contributing/evals-ci.md +++ b/docs/contributing/evals-ci.md @@ -3,7 +3,7 @@ title: Evals in CI description: Auth contract, fork-PR policy, and how to add a new eval spec for the hve-core vally pipeline sidebar_position: 11 author: Microsoft -ms.date: 2026-07-17 +ms.date: 2026-07-31 ms.topic: how-to keywords: - evals @@ -184,12 +184,12 @@ The CI-owned eval-validation workflow runs the static eval-lint lanes. They are not part of `validate:local`; see [Validation Commands and CI-Owned Lanes](validation) for local reproduction prerequisites and output handling. -| Script | Tool | Purpose | -|-----------------------|----------------------------|------------------------------------------------------------------| -| `ci:eval:lint:vally` | `vally lint --eval evals/` | Spec validation via the upstream CLI | -| `ci:eval:lint:schema` | `Test-EvalSpec.ps1` | hve-core schema lint (graders, executor, `moderation.threshold`) | -| `ci:eval:lint:text` | `Test-EvalSpecText.ps1` | retext-profanities + alex.js gate on the AI-artifact corpus | -| `ci:eval:lint:safety` | `Test-VallyTestSafety.ps1` | Safety validation for eval stimuli | +| Script | Tool | Purpose | +|-----------------------|---------------------------------|---------------------------------------------------------------------| +| `ci:eval:lint:vally` | `vally lint --eval-spec evals/` | Spec validation via the upstream CLI | +| `ci:eval:lint:schema` | `Test-EvalSpec.ps1` | Schema lint, agent-behavior coverage, and orphaned-tag reachability | +| `ci:eval:lint:text` | `Test-EvalSpecText.ps1` | retext-profanities + alex.js gate on the AI-artifact corpus | +| `ci:eval:lint:safety` | `Test-VallyTestSafety.ps1` | Safety validation for eval stimuli | `ci:eval:lint:text` scans `.github/{agents,prompts,instructions,skills}/**/*.md` and `docs/**/*.md`. By default `retext-profanities` findings flip the exit code (errors) and `alex` findings emit `::warning` annotations only. Pass `-FailOnAlex` to promote alex findings to errors for local hardening: @@ -207,18 +207,68 @@ False-positive lexical matches (e.g., `penetration test`, `attack surface`, `tok | 1 | At least one `retext-profanities` finding, or any alex.js finding when `-FailOnAlex` is set. | | 2 | Setup failure (corpus expansion failed, Node shim missing, or `node` not on PATH). | +### Schema lint, coverage, and reachability + +`ci:eval:lint:schema` runs three checks in a single pass, and any one of them can fail the lane: + +* Schema validation of every spec under `evals/`, covering required keys, the executor whitelist, `moderation.threshold`, and stimulus backlink tags. +* Agent-behavior coverage. Every parent (user-invocable) agent under `.github/agents/` must have a stimulus partial at `evals/agent-behavior/stimuli/.yml`. +* Orphaned-tag reachability. Every `agent=` and `scenario=` tag in `evals/agent-behavior/eval.yaml` must resolve to a slug present in the agent inventory. Unresolvable tags emit `::error` annotations and hard-fail the lane. + +`Test-EvalSpec.ps1` exit codes: + +| Exit | Meaning | +|------|---------------------------------------------------------------------------------------------------------------------------------| +| 0 | Every spec is valid, every parent agent is covered, and no tag is orphaned. | +| 1 | A spec failed schema validation, a parent agent lacked a stimulus partial, the inventory was unreadable, or a tag was orphaned. | +| 2 | Setup failure (the `powershell-yaml` module is not installed). | + +Use `-SkipAgentCoverage` for fixture-only runs, or `-NewAgentsOnly` with `-BaseRef` to enforce coverage incrementally on newly added agents. + +### Agent inventory + +`evals/agent-behavior/AGENTS.yml` is the source of truth for the orphaned-tag gate. It is generated rather than hand-edited: + +```pwsh +pwsh scripts/evals/Build-AgentInventory.ps1 -Force +``` + +Enrollment follows two rules: + +* Parent agents are always inventoried. An agent file with no `user-invocable` key is treated as a parent. +* Subagents (`user-invocable: false`) are inventoried only when a matching stimulus partial exists at `evals/agent-behavior/stimuli/.yml`. + +When the lane reports an orphaned tag, either the tag is misspelled or the agent it names is not enrolled. Add the agent's stimulus partial when the agent is a subagent, regenerate the inventory, and commit the regenerated `AGENTS.yml` alongside the change. + ### Baseline-equivalence specs -`ci:eval:lint:vally` runs `vally lint --eval evals/`, which validates the eval YAML files immediately under `evals/` but does not recurse into nested subdirectories. The baseline-equivalence suite under [evals/baseline-equivalence/](https://github.com/microsoft/hve-core/blob/main/evals/baseline-equivalence/README.md) ships nested specs (`baseline/eval.yaml`, `customized/eval.yaml`, and `compare.eval.yml`) that need explicit per-file lint invocations: +`ci:eval:lint:vally` runs `vally lint --eval-spec evals/`, which validates the eval YAML files immediately under `evals/` but does not recurse into nested subdirectories. The baseline-equivalence suite under [evals/baseline-equivalence/](https://github.com/microsoft/hve-core/blob/main/evals/baseline-equivalence/README.md) ships nested specs (`baseline/eval.yaml`, `customized/eval.yaml`, and `compare.eval.yml`) that need explicit per-file lint invocations: ```pwsh -vally lint --eval evals/baseline-equivalence/baseline/eval.yaml -vally lint --eval evals/baseline-equivalence/customized/eval.yaml -vally lint --eval evals/baseline-equivalence/compare.eval.yml +vally lint --eval-spec evals/baseline-equivalence/baseline/eval.yaml +vally lint --eval-spec evals/baseline-equivalence/customized/eval.yaml +vally lint --eval-spec evals/baseline-equivalence/compare.eval.yml ``` [scripts/evals/Invoke-BaselineEquivalence.ps1](../../scripts/evals/Invoke-BaselineEquivalence.ps1) runs all three implicitly during `npm run ci:eval:run:equivalence`. See [evals/baseline-equivalence/README.md](https://github.com/microsoft/hve-core/blob/main/evals/baseline-equivalence/README.md) for the suite operator guide and driver-output contract. +## Matrix, Moderation, and Dashboard Scripts + +Beyond the lint lanes, `scripts/evals/` holds the scripts that scope runs, moderate artifacts, and render results. + +| Script | Invoked by | Purpose | +|-------------------------------------------|-----------------------------------------------------------------|---------------------------------------------------------------------------------------------| +| `Invoke-ArtifactModeration.ps1` | `ci:eval:moderate:artifacts` | Moderates all eval specs plus changed AI artifacts from the changed-artifact manifest | +| `New-AgentMatrixDashboard.ps1` | `ci:eval:agent:dashboard`, `ci:eval:agent:report` | Renders a self-contained HTML matrix dashboard, one row per inventory agent | +| `New-EquivalenceDashboard.ps1` | `ci:eval:dashboard` | Renders a self-contained HTML dashboard for a baseline-equivalence run | +| `New-AgentSurfaceSignatures.ps1` | `Build-AgentBehaviorSpec.ps1`, `Invoke-BaselineEquivalence.ps1` | Generates the per-agent surface signature YAML used by baseline equivalence | +| `Get-AgentDependencyMap.ps1` | Run directly | Builds a JSON map of agent dependencies for the baseline-equivalence dispatcher | +| `Update-AgentMatrixSummariesFromLogs.ps1` | Run directly | Rebuilds per-agent matrix summaries from existing vally logs without re-running `npx vally` | + +`Invoke-ArtifactModeration.ps1` and `Invoke-CorpusModeration.ps1` are distinct lanes over the same changed-artifact manifest. Corpus moderation scores stimulus text inside eval specs; artifact moderation covers the specs plus the changed AI artifacts themselves, writing to a separate output file. + +`ci:eval:agent:report` is the end-to-end convenience command: it runs `ci:eval:agent:matrix` and then `ci:eval:agent:dashboard`. + ## Running Pester Tests Locally `npm run test:ps` wraps `scripts/tests/Invoke-PesterTests.ps1`. The default invocation applies `ExcludeTag=@('Integration','Slow')`: diff --git a/docs/security/security-model.md b/docs/security/security-model.md index 47a74474e..beeb7d4e5 100644 --- a/docs/security/security-model.md +++ b/docs/security/security-model.md @@ -3,7 +3,7 @@ title: Security Assurance Case and Security Model description: Comprehensive security model and security assurance documentation demonstrating enterprise security practices sidebar_position: 2 author: Microsoft -ms.date: 2026-07-28 +ms.date: 2026-07-31 ms.topic: reference keywords: - security @@ -1393,21 +1393,21 @@ Its headline residual is speaker-notes content egress to the Azure region; input ### Supply Chain Security Controls -| ID | Control | Implementation | Validates Against | -|------|------------------------------------------|---------------------------------------------------------------------------|-------------------| -| SC-1 | Dependency Pinning Validation | Test-DependencyPinning.ps1 | S-1, S-2 | -| SC-2 | SHA Staleness Monitoring | Test-SHAStaleness.ps1 | S-1 | -| SC-3 | Dependency Review | dependency-review.yml | S-2, AI-5 | -| SC-4 | npm Security Audit | npm audit in pr-validation.yml | S-2 | -| SC-5 | Dependabot Updates | dependabot.yml | S-1, S-2 | -| SC-6 | Tool Checksum Verification | scripts/security/tool-checksums.json | S-1 | -| SC-7 | SBOM Generation and Attestation | anchore/sbom-action, actions/attest in main.yml | S-1, S-2 | -| SC-8 | SBOM Dependency Diff | sbom-diff job in main.yml | S-1, S-2 | -| SC-9 | VEX Vulnerability Triage and Attestation | vex-detect.yml, vex-draft.md, attest-and-upload-vex in release-stable.yml | S-1, S-2 | +| ID | Control | Implementation | Validates Against | +|------|------------------------------------------|--------------------------------------------------------------------|-------------------| +| SC-1 | Dependency Pinning Validation | Test-DependencyPinning.ps1 | S-1, S-2 | +| SC-2 | SHA Staleness Monitoring | Test-SHAStaleness.ps1 | S-1 | +| SC-3 | Dependency Review | dependency-review.yml | S-2, AI-5 | +| SC-4 | npm Security Audit | npm audit in pr-validation.yml | S-2 | +| SC-5 | Dependabot Updates | dependabot.yml | S-1, S-2 | +| SC-6 | Tool Checksum Verification | scripts/security/tool-checksums.json | S-1 | +| SC-7 | SBOM Generation and Attestation | anchore/sbom-action, actions/attest in release-stable.yml | S-1, S-2 | +| SC-8 | SBOM Dependency Diff | sbom-diff job in release-stable.yml | S-1, S-2 | +| SC-9 | VEX Vulnerability Triage and Attestation | vex-detect.yml, vex-draft.md, vex-attest job in release-stable.yml | S-1, S-2 | #### SC-8: SBOM Dependency Diff Implementation -The `sbom-diff` job in `main.yml` runs during each release to surface supply chain changes between consecutive versions. It compares the current dependency SBOM against the previous release, generating a structured `dependency-diff.md` report that is uploaded to the GitHub Release. +The `sbom-diff` job in `release-stable.yml` runs during each release to surface supply chain changes between consecutive versions. It compares the current dependency SBOM against the previous release, generating a structured `dependency-diff.md` report that is uploaded to the GitHub Release. | Field | Value | |------------------|----------------------------------------------------------------------------| @@ -1425,6 +1425,30 @@ The diff script parses SPDX JSON packages, excludes root document entries, and c When no previous release exists or the prior release lacks a dependency SBOM, the job exits cleanly without producing a diff. This graceful degradation ensures the first release in a repository proceeds without error. +#### SC-9: VEX Vulnerability Triage and Attestation Implementation + +SC-9 spans three workflows: detection finds untriaged vulnerabilities, drafting proposes OpenVEX status updates for human review, and the release pipeline attests the resulting document. The canonical VEX document is `security/vex/hve-core.openvex.json`. + +| Field | Value | +|---------------------------|----------------------------------------------------------------------------------------| +| **Detection Trigger** | Tuesdays 08:00 UTC, after a successful Stable Release Pipeline run, or manual dispatch | +| **Detection Workflow** | `vex-detect.yml` runs OSV-Scanner and files or updates a single triage issue | +| **Detection Permissions** | `contents: read`, `issues: write` | +| **Drafting Trigger** | `workflow_run` from VEX Detection, plus manual dispatch | +| **Drafting Workflow** | `vex-draft.md` invokes the SSSC Reviewer agent and opens one pull request | +| **Drafting Permissions** | `contents: read`, `issues: read` | +| **Release Attestation** | `vex-attest` job in `release-stable.yml`, via the reusable `vex-attest.yml` | +| **Attestations** | Build provenance over the VEX document, plus VEX as predicate over the SBOM subject | +| **Human Review Gate** | AI drafts; a CODEOWNERS-required human reviews and merges the pull request | + +Detection performs no AI drafting. It compares OSV-Scanner findings against the VEX document and reports divergence as a triage issue. + +Drafting is gated twice so it consumes no model budget when there is nothing to do. The first gate skips while a VEX draft pull request is already open. The second gate skips when every finding already carries a terminal VEX status. The resulting pull request is restricted to the VEX document and is labeled `security`, `automated`, and `needs-triage`. + +The release attestation produces two artifacts: a build-provenance attestation whose subject is the VEX document, and an in-toto attestation that binds the VEX document as an OpenVEX predicate over the dependency SBOM subject. The VEX document is also uploaded to the GitHub Release. + +The merge commit author is the accountable author of record, never the agent. + ### Code Quality Controls | ID | Control | Implementation | Validates Against |