Skip to content

docs: fix stale workflow, job, and script references across three guides #2533

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
WilliamBerryiii merged 7 commits into from
Jul 31, 2026
+118 −33
Merged

docs: fix stale workflow, job, and script references across three guides #2533

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
a575687
update docs
jkim323 Jul 28, 2026
d4d7fc1
Merge branch 'main' into docs/stale-workflow-job-script-refs
WilliamBerryiii Jul 28, 2026
78243ec
Merge branch 'main' into docs/stale-workflow-job-script-refs
jkim323 Jul 29, 2026
457c38c
Merge branch 'main' into docs/stale-workflow-job-script-refs
jkim323 Jul 30, 2026
beafcaf
Merge branch 'main' into docs/stale-workflow-job-script-refs
jkim323 Jul 31, 2026
c3e67db
update docs + msdate
jkim323 Jul 31, 2026
4913895
fix date
jkim323 Jul 31, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 20 additions & 9 deletions docs/architecture/agentic-workflows.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: Agentic Workflows
description: End-to-end process flow for AI-driven issue triage, implementation, and review workflows in hve-core
author: HVE Core Team
ms.date: 2026-07-16
ms.date: 2026-07-31
ms.topic: concept
sidebar_position: 4
keywords:
Expand Down Expand Up @@ -32,8 +32,6 @@ flowchart TD
C["Classify by type<br/>and component"]
D["Detect duplicates<br/>via keyword search"]
E["Assess issue quality"]
F{"Scope too broad<br/>for single deliverable?"}
G["Decompose into<br/>sub-issues"]
H{"Passes all<br/>agent-ready criteria?"}
I["Apply labels,<br/>remove needs-triage"]
J["Add agent-ready label"]
Expand Down Expand Up @@ -66,10 +64,7 @@ flowchart TD
B --> C
C --> D
D --> E
E --> F
F -- Yes --> G
G --> I
F -- No --> H
E --> H
H -- Yes --> I
I --> J
H -- No --> I
Expand All @@ -94,15 +89,15 @@ flowchart TD

| Workflow | Trigger | Execution Owner | Key Actions |
|----------------------|---------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------|
| Issue Triage | Issue opened or labeled `needs-triage` | [Issue Triage Agent](https://github.com/microsoft/hve-core/blob/main/.github/agents/issue-triage.agent.md) | Classify, detect duplicates, assess quality, decompose, label, evaluate readiness |
| Issue Triage | Issue opened or labeled `needs-triage` | [Issue Triage Agent](https://github.com/microsoft/hve-core/blob/main/.github/agents/issue-triage.agent.md) | Classify, detect duplicates, assess quality, label, evaluate readiness |
| Issue Implementation | Issue labeled `agent-ready` | Workflow-owned procedure in `.github/workflows/issue-implement.md` | Research codebase, plan changes, implement, open PR |
| PR Review | PR opened or marked ready for review | [Code Review Agent](https://github.com/microsoft/hve-core/blob/main/.github/agents/coding-standards/code-review.agent.md) | Review correctness, conventions, security; label `review-passed` or `needs-revision` for non-maintainer PRs, advisory `COMMENT` only for maintainer PRs |
| Dependabot PR Review | Dependabot PR opened or updated | [Dependency Reviewer Agent](https://github.com/microsoft/hve-core/blob/main/.github/agents/dependency-reviewer.agent.md) | Validate licensing, SHA pinning, environment sync; approve safe bumps |
| Documentation Drift | Push to main | [Documentation Agent](https://github.com/microsoft/hve-core/blob/main/.github/agents/hve-core/documentation.agent.md) (drift mode) | Map code changes to docs, flag stale documentation for follow-up |
| VEX Draft | `workflow_run` after VEX Detection succeeds, or `workflow_dispatch` | [SSSC Reviewer](https://github.com/microsoft/hve-core/blob/main/.github/agents/security/sssc-reviewer.agent.md) | Enrich CVEs, analyze reachability, open one PR with OpenVEX draft statements for human review |

> [!TIP]
> The triage agent only classifies, labels, and optionally decomposes issues. It does not close issues, assign users, or modify issue titles.
> The triage agent classifies issues, applies type, area, and priority labels, detects duplicates, assesses quality, and marks qualifying issues `agent-ready`. It does not create sub-issues, close issues, assign users, or modify issue titles.

<!-- markdownlint-disable-next-line MD028 -->

Expand Down Expand Up @@ -131,6 +126,22 @@ All six workflows are defined as GitHub Agentic Workflow markdown files under `.

Each workflow file declares permissions, safe output limits, and activation guards that prevent unintended execution.

### Lock File Ownership

The `*.lock.yml` files under `.github/workflows/` are generated outputs of `gh aw compile`. Editing them directly is not supported: the next compile overwrites the change. Edit the source `.md` workflow instead, then recompile.

Because these files are generated, Dependabot is configured to leave them alone. `.github/dependabot.yml` excludes the `.github/workflows/*.lock.yml` path and the `github/gh-aw-actions/*` action family, so action bumps inside a lock file never arrive as a pull request.

### Upgrading gh-aw-actions

The `gh-aw-actions` version is pinned in `.github/aw/actions-lock.json`, which maps each action reference to a resolved commit SHA. This file, not Dependabot, is the upgrade path:

1. Upgrade the `gh aw` CLI/compiler (the resolved `gh-aw-actions` version is tied to the compiler release, not set independently).
2. Run `gh aw compile` (or `gh aw update-actions`) with API access to re-resolve actions and regenerate `actions-lock.json` plus every `*.lock.yml` file.
3. Commit `actions-lock.json` and the regenerated lock files together.

The pinned version is version-locked to the `gh aw` compiler that produces the lock files, so the bump and the recompile belong in the same change.

## Label-Driven Handoffs

Labels serve as the event bus connecting workflows. Each label transition triggers the next stage:
Expand Down
72 changes: 61 additions & 11 deletions docs/contributing/evals-ci.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ title: Evals in CI
description: Auth contract, fork-PR policy, and how to add a new eval spec for the hve-core vally pipeline
sidebar_position: 11
author: Microsoft
ms.date: 2026-07-17
ms.date: 2026-07-31
ms.topic: how-to
keywords:
- evals
Expand Down Expand Up @@ -184,12 +184,12 @@ The CI-owned eval-validation workflow runs the static eval-lint lanes. They are
not part of `validate:local`; see [Validation Commands and CI-Owned Lanes](validation)
for local reproduction prerequisites and output handling.

| Script | Tool | Purpose |
|-----------------------|----------------------------|------------------------------------------------------------------|
| `ci:eval:lint:vally` | `vally lint --eval evals/` | Spec validation via the upstream CLI |
| `ci:eval:lint:schema` | `Test-EvalSpec.ps1` | hve-core schema lint (graders, executor, `moderation.threshold`) |
| `ci:eval:lint:text` | `Test-EvalSpecText.ps1` | retext-profanities + alex.js gate on the AI-artifact corpus |
| `ci:eval:lint:safety` | `Test-VallyTestSafety.ps1` | Safety validation for eval stimuli |
| Script | Tool | Purpose |
|-----------------------|---------------------------------|---------------------------------------------------------------------|
| `ci:eval:lint:vally` | `vally lint --eval-spec evals/` | Spec validation via the upstream CLI |
| `ci:eval:lint:schema` | `Test-EvalSpec.ps1` | Schema lint, agent-behavior coverage, and orphaned-tag reachability |
| `ci:eval:lint:text` | `Test-EvalSpecText.ps1` | retext-profanities + alex.js gate on the AI-artifact corpus |
| `ci:eval:lint:safety` | `Test-VallyTestSafety.ps1` | Safety validation for eval stimuli |

`ci:eval:lint:text` scans `.github/{agents,prompts,instructions,skills}/**/*.md` and `docs/**/*.md`. By default `retext-profanities` findings flip the exit code (errors) and `alex` findings emit `::warning` annotations only. Pass `-FailOnAlex` to promote alex findings to errors for local hardening:

Expand All @@ -207,18 +207,68 @@ False-positive lexical matches (e.g., `penetration test`, `attack surface`, `tok
| 1 | At least one `retext-profanities` finding, or any alex.js finding when `-FailOnAlex` is set. |
| 2 | Setup failure (corpus expansion failed, Node shim missing, or `node` not on PATH). |

### Schema lint, coverage, and reachability

`ci:eval:lint:schema` runs three checks in a single pass, and any one of them can fail the lane:

* Schema validation of every spec under `evals/`, covering required keys, the executor whitelist, `moderation.threshold`, and stimulus backlink tags.
* Agent-behavior coverage. Every parent (user-invocable) agent under `.github/agents/` must have a stimulus partial at `evals/agent-behavior/stimuli/<slug>.yml`.
* Orphaned-tag reachability. Every `agent=` and `scenario=` tag in `evals/agent-behavior/eval.yaml` must resolve to a slug present in the agent inventory. Unresolvable tags emit `::error` annotations and hard-fail the lane.

`Test-EvalSpec.ps1` exit codes:

| Exit | Meaning |
|------|---------------------------------------------------------------------------------------------------------------------------------|
| 0 | Every spec is valid, every parent agent is covered, and no tag is orphaned. |
| 1 | A spec failed schema validation, a parent agent lacked a stimulus partial, the inventory was unreadable, or a tag was orphaned. |
| 2 | Setup failure (the `powershell-yaml` module is not installed). |

Use `-SkipAgentCoverage` for fixture-only runs, or `-NewAgentsOnly` with `-BaseRef` to enforce coverage incrementally on newly added agents.

### Agent inventory

`evals/agent-behavior/AGENTS.yml` is the source of truth for the orphaned-tag gate. It is generated rather than hand-edited:

```pwsh
pwsh scripts/evals/Build-AgentInventory.ps1 -Force
```

Enrollment follows two rules:

* Parent agents are always inventoried. An agent file with no `user-invocable` key is treated as a parent.
* Subagents (`user-invocable: false`) are inventoried only when a matching stimulus partial exists at `evals/agent-behavior/stimuli/<slug>.yml`.

When the lane reports an orphaned tag, either the tag is misspelled or the agent it names is not enrolled. Add the agent's stimulus partial when the agent is a subagent, regenerate the inventory, and commit the regenerated `AGENTS.yml` alongside the change.

### Baseline-equivalence specs

`ci:eval:lint:vally` runs `vally lint --eval evals/`, which validates the eval YAML files immediately under `evals/` but does not recurse into nested subdirectories. The baseline-equivalence suite under [evals/baseline-equivalence/](https://github.com/microsoft/hve-core/blob/main/evals/baseline-equivalence/README.md) ships nested specs (`baseline/eval.yaml`, `customized/eval.yaml`, and `compare.eval.yml`) that need explicit per-file lint invocations:
`ci:eval:lint:vally` runs `vally lint --eval-spec evals/`, which validates the eval YAML files immediately under `evals/` but does not recurse into nested subdirectories. The baseline-equivalence suite under [evals/baseline-equivalence/](https://github.com/microsoft/hve-core/blob/main/evals/baseline-equivalence/README.md) ships nested specs (`baseline/eval.yaml`, `customized/eval.yaml`, and `compare.eval.yml`) that need explicit per-file lint invocations:

```pwsh
vally lint --eval evals/baseline-equivalence/baseline/eval.yaml
vally lint --eval evals/baseline-equivalence/customized/eval.yaml
vally lint --eval evals/baseline-equivalence/compare.eval.yml
vally lint --eval-spec evals/baseline-equivalence/baseline/eval.yaml
vally lint --eval-spec evals/baseline-equivalence/customized/eval.yaml
vally lint --eval-spec evals/baseline-equivalence/compare.eval.yml
```

[scripts/evals/Invoke-BaselineEquivalence.ps1](../../scripts/evals/Invoke-BaselineEquivalence.ps1) runs all three implicitly during `npm run ci:eval:run:equivalence`. See [evals/baseline-equivalence/README.md](https://github.com/microsoft/hve-core/blob/main/evals/baseline-equivalence/README.md) for the suite operator guide and driver-output contract.

## Matrix, Moderation, and Dashboard Scripts

Beyond the lint lanes, `scripts/evals/` holds the scripts that scope runs, moderate artifacts, and render results.

| Script | Invoked by | Purpose |
|-------------------------------------------|-----------------------------------------------------------------|---------------------------------------------------------------------------------------------|
| `Invoke-ArtifactModeration.ps1` | `ci:eval:moderate:artifacts` | Moderates all eval specs plus changed AI artifacts from the changed-artifact manifest |
| `New-AgentMatrixDashboard.ps1` | `ci:eval:agent:dashboard`, `ci:eval:agent:report` | Renders a self-contained HTML matrix dashboard, one row per inventory agent |
| `New-EquivalenceDashboard.ps1` | `ci:eval:dashboard` | Renders a self-contained HTML dashboard for a baseline-equivalence run |
| `New-AgentSurfaceSignatures.ps1` | `Build-AgentBehaviorSpec.ps1`, `Invoke-BaselineEquivalence.ps1` | Generates the per-agent surface signature YAML used by baseline equivalence |
| `Get-AgentDependencyMap.ps1` | Run directly | Builds a JSON map of agent dependencies for the baseline-equivalence dispatcher |
| `Update-AgentMatrixSummariesFromLogs.ps1` | Run directly | Rebuilds per-agent matrix summaries from existing vally logs without re-running `npx vally` |

`Invoke-ArtifactModeration.ps1` and `Invoke-CorpusModeration.ps1` are distinct lanes over the same changed-artifact manifest. Corpus moderation scores stimulus text inside eval specs; artifact moderation covers the specs plus the changed AI artifacts themselves, writing to a separate output file.

`ci:eval:agent:report` is the end-to-end convenience command: it runs `ci:eval:agent:matrix` and then `ci:eval:agent:dashboard`.

## Running Pester Tests Locally

`npm run test:ps` wraps `scripts/tests/Invoke-PesterTests.ps1`. The default invocation applies `ExcludeTag=@('Integration','Slow')`:
Expand Down
Loading
Loading