Skip to content
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 19 additions & 8 deletions docs/architecture/agentic-workflows.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,6 @@ flowchart TD
C["Classify by type<br/>and component"]
D["Detect duplicates<br/>via keyword search"]
E["Assess issue quality"]
F{"Scope too broad<br/>for single deliverable?"}
G["Decompose into<br/>sub-issues"]
H{"Passes all<br/>agent-ready criteria?"}
I["Apply labels,<br/>remove needs-triage"]
J["Add agent-ready label"]
Expand Down Expand Up @@ -66,10 +64,7 @@ flowchart TD
B --> C
C --> D
D --> E
E --> F
F -- Yes --> G
G --> I
F -- No --> H
E --> H
H -- Yes --> I
I --> J
H -- No --> I
Expand All @@ -94,15 +89,15 @@ flowchart TD

| Workflow | Trigger | Execution Owner | Key Actions |
|----------------------|---------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------|
| Issue Triage | Issue opened or labeled `needs-triage` | [Issue Triage Agent](https://github.com/microsoft/hve-core/blob/main/.github/agents/issue-triage.agent.md) | Classify, detect duplicates, assess quality, decompose, label, evaluate readiness |
| Issue Triage | Issue opened or labeled `needs-triage` | [Issue Triage Agent](https://github.com/microsoft/hve-core/blob/main/.github/agents/issue-triage.agent.md) | Classify, detect duplicates, assess quality, label, evaluate readiness |
| Issue Implementation | Issue labeled `agent-ready` | Workflow-owned procedure in `.github/workflows/issue-implement.md` | Research codebase, plan changes, implement, open PR |
| PR Review | PR opened or marked ready for review | [Code Review Agent](https://github.com/microsoft/hve-core/blob/main/.github/agents/coding-standards/code-review.agent.md) | Review correctness, conventions, security; label `review-passed` or `needs-revision` for non-maintainer PRs, advisory `COMMENT` only for maintainer PRs |
| Dependabot PR Review | Dependabot PR opened or updated | [Dependency Reviewer Agent](https://github.com/microsoft/hve-core/blob/main/.github/agents/dependency-reviewer.agent.md) | Validate licensing, SHA pinning, environment sync; approve safe bumps |
| Documentation Drift | Push to main | [Documentation Agent](https://github.com/microsoft/hve-core/blob/main/.github/agents/hve-core/documentation.agent.md) (drift mode) | Map code changes to docs, flag stale documentation for follow-up |
| VEX Draft | `workflow_run` after VEX Detection succeeds, or `workflow_dispatch` | [SSSC Reviewer](https://github.com/microsoft/hve-core/blob/main/.github/agents/security/sssc-reviewer.agent.md) | Enrich CVEs, analyze reachability, open one PR with OpenVEX draft statements for human review |

> [!TIP]
> The triage agent only classifies, labels, and optionally decomposes issues. It does not close issues, assign users, or modify issue titles.
> The triage agent classifies issues, applies type, area, and priority labels, detects duplicates, assesses quality, and marks qualifying issues `agent-ready`. It does not create sub-issues, close issues, assign users, or modify issue titles.

<!-- markdownlint-disable-next-line MD028 -->

Expand Down Expand Up @@ -131,6 +126,22 @@ All six workflows are defined as GitHub Agentic Workflow markdown files under `.

Each workflow file declares permissions, safe output limits, and activation guards that prevent unintended execution.

### Lock File Ownership

The `*.lock.yml` files under `.github/workflows/` are generated outputs of `gh aw compile`. Editing them directly is not supported: the next compile overwrites the change. Edit the source `.md` workflow instead, then recompile.

Because these files are generated, Dependabot is configured to leave them alone. `.github/dependabot.yml` excludes the `.github/workflows/*.lock.yml` path and the `github/gh-aw-actions/*` action family, so action bumps inside a lock file never arrive as a pull request.

### Upgrading gh-aw-actions

The `gh-aw-actions` version is pinned in `.github/aw/actions-lock.json`, which maps each action reference to a resolved commit SHA. This file, not Dependabot, is the upgrade path:

1. Bump the `gh-aw-actions` version in `.github/aw/actions-lock.json`.
2. Run `gh aw compile` to regenerate every `*.lock.yml` file against the new version.
3. Commit the updated `actions-lock.json` and the regenerated lock files together.
Comment thread
jkim323 marked this conversation as resolved.
Outdated

The pinned version is version-locked to the `gh aw` compiler that produces the lock files, so the bump and the recompile belong in the same change.

## Label-Driven Handoffs

Labels serve as the event bus connecting workflows. Each label transition triggers the next stage:
Expand Down
70 changes: 60 additions & 10 deletions docs/contributing/evals-ci.md
Original file line number Diff line number Diff line change
Expand Up @@ -184,12 +184,12 @@ The CI-owned eval-validation workflow runs the static eval-lint lanes. They are
not part of `validate:local`; see [Validation Commands and CI-Owned Lanes](validation)
for local reproduction prerequisites and output handling.

| Script | Tool | Purpose |
|-----------------------|----------------------------|------------------------------------------------------------------|
| `ci:eval:lint:vally` | `vally lint --eval evals/` | Spec validation via the upstream CLI |
| `ci:eval:lint:schema` | `Test-EvalSpec.ps1` | hve-core schema lint (graders, executor, `moderation.threshold`) |
| `ci:eval:lint:text` | `Test-EvalSpecText.ps1` | retext-profanities + alex.js gate on the AI-artifact corpus |
| `ci:eval:lint:safety` | `Test-VallyTestSafety.ps1` | Safety validation for eval stimuli |
| Script | Tool | Purpose |
|-----------------------|---------------------------------|---------------------------------------------------------------------|
| `ci:eval:lint:vally` | `vally lint --eval-spec evals/` | Spec validation via the upstream CLI |
| `ci:eval:lint:schema` | `Test-EvalSpec.ps1` | Schema lint, agent-behavior coverage, and orphaned-tag reachability |
| `ci:eval:lint:text` | `Test-EvalSpecText.ps1` | retext-profanities + alex.js gate on the AI-artifact corpus |
| `ci:eval:lint:safety` | `Test-VallyTestSafety.ps1` | Safety validation for eval stimuli |

`ci:eval:lint:text` scans `.github/{agents,prompts,instructions,skills}/**/*.md` and `docs/**/*.md`. By default `retext-profanities` findings flip the exit code (errors) and `alex` findings emit `::warning` annotations only. Pass `-FailOnAlex` to promote alex findings to errors for local hardening:

Expand All @@ -207,18 +207,68 @@ False-positive lexical matches (e.g., `penetration test`, `attack surface`, `tok
| 1 | At least one `retext-profanities` finding, or any alex.js finding when `-FailOnAlex` is set. |
| 2 | Setup failure (corpus expansion failed, Node shim missing, or `node` not on PATH). |

### Schema lint, coverage, and reachability

`ci:eval:lint:schema` runs three checks in a single pass, and any one of them can fail the lane:

* Schema validation of every spec under `evals/`, covering required keys, the executor whitelist, `moderation.threshold`, and stimulus backlink tags.
* Agent-behavior coverage. Every parent (user-invocable) agent under `.github/agents/` must have a stimulus partial at `evals/agent-behavior/stimuli/<slug>.yml`.
* Orphaned-tag reachability. Every `agent=` and `scenario=` tag in `evals/agent-behavior/eval.yaml` must resolve to a slug present in the agent inventory. Unresolvable tags emit `::error` annotations and hard-fail the lane.

`Test-EvalSpec.ps1` exit codes:

| Exit | Meaning |
|------|---------------------------------------------------------------------------------------------------------------------------------|
| 0 | Every spec is valid, every parent agent is covered, and no tag is orphaned. |
| 1 | A spec failed schema validation, a parent agent lacked a stimulus partial, the inventory was unreadable, or a tag was orphaned. |
| 2 | Setup failure (the `powershell-yaml` module is not installed). |

Use `-SkipAgentCoverage` for fixture-only runs, or `-NewAgentsOnly` with `-BaseRef` to enforce coverage incrementally on newly added agents.

### Agent inventory

`evals/agent-behavior/AGENTS.yml` is the source of truth for the orphaned-tag gate. It is generated rather than hand-edited:

```pwsh
pwsh scripts/evals/Build-AgentInventory.ps1 -Force
```

Enrollment follows two rules:

* Parent agents are always inventoried. An agent file with no `user-invocable` key is treated as a parent.
* Subagents (`user-invocable: false`) are inventoried only when a matching stimulus partial exists at `evals/agent-behavior/stimuli/<slug>.yml`.

When the lane reports an orphaned tag, either the tag is misspelled or the agent it names is not enrolled. Add the agent's stimulus partial when the agent is a subagent, regenerate the inventory, and commit the regenerated `AGENTS.yml` alongside the change.

### Baseline-equivalence specs

`ci:eval:lint:vally` runs `vally lint --eval evals/`, which validates the eval YAML files immediately under `evals/` but does not recurse into nested subdirectories. The baseline-equivalence suite under [evals/baseline-equivalence/](https://github.com/microsoft/hve-core/blob/main/evals/baseline-equivalence/README.md) ships nested specs (`baseline/eval.yaml`, `customized/eval.yaml`, and `compare.eval.yml`) that need explicit per-file lint invocations:
`ci:eval:lint:vally` runs `vally lint --eval-spec evals/`, which validates the eval YAML files immediately under `evals/` but does not recurse into nested subdirectories. The baseline-equivalence suite under [evals/baseline-equivalence/](https://github.com/microsoft/hve-core/blob/main/evals/baseline-equivalence/README.md) ships nested specs (`baseline/eval.yaml`, `customized/eval.yaml`, and `compare.eval.yml`) that need explicit per-file lint invocations:

```pwsh
vally lint --eval evals/baseline-equivalence/baseline/eval.yaml
vally lint --eval evals/baseline-equivalence/customized/eval.yaml
vally lint --eval evals/baseline-equivalence/compare.eval.yml
vally lint --eval-spec evals/baseline-equivalence/baseline/eval.yaml
vally lint --eval-spec evals/baseline-equivalence/customized/eval.yaml
vally lint --eval-spec evals/baseline-equivalence/compare.eval.yml
```

[scripts/evals/Invoke-BaselineEquivalence.ps1](../../scripts/evals/Invoke-BaselineEquivalence.ps1) runs all three implicitly during `npm run ci:eval:run:equivalence`. See [evals/baseline-equivalence/README.md](https://github.com/microsoft/hve-core/blob/main/evals/baseline-equivalence/README.md) for the suite operator guide and driver-output contract.

## Matrix, Moderation, and Dashboard Scripts

Beyond the lint lanes, `scripts/evals/` holds the scripts that scope runs, moderate artifacts, and render results.

| Script | Invoked by | Purpose |
|-------------------------------------------|-----------------------------------------------------------------|---------------------------------------------------------------------------------------------|
| `Invoke-ArtifactModeration.ps1` | `ci:eval:moderate:artifacts` | Moderates all eval specs plus changed AI artifacts from the changed-artifact manifest |
| `New-AgentMatrixDashboard.ps1` | `ci:eval:agent:dashboard`, `ci:eval:agent:report` | Renders a self-contained HTML matrix dashboard, one row per inventory agent |
| `New-EquivalenceDashboard.ps1` | `ci:eval:dashboard` | Renders a self-contained HTML dashboard for a baseline-equivalence run |
| `New-AgentSurfaceSignatures.ps1` | `Build-AgentBehaviorSpec.ps1`, `Invoke-BaselineEquivalence.ps1` | Generates the per-agent surface signature YAML used by baseline equivalence |
| `Get-AgentDependencyMap.ps1` | Run directly | Builds a JSON map of agent dependencies for the baseline-equivalence dispatcher |
| `Update-AgentMatrixSummariesFromLogs.ps1` | Run directly | Rebuilds per-agent matrix summaries from existing vally logs without re-running `npx vally` |

`Invoke-ArtifactModeration.ps1` and `Invoke-CorpusModeration.ps1` are distinct lanes over the same changed-artifact manifest. Corpus moderation scores stimulus text inside eval specs; artifact moderation covers the specs plus the changed AI artifacts themselves, writing to a separate output file.

`ci:eval:agent:report` is the end-to-end convenience command: it runs `ci:eval:agent:matrix` and then `ci:eval:agent:dashboard`.

## Running Pester Tests Locally

`npm run test:ps` wraps `scripts/tests/Invoke-PesterTests.ps1`. The default invocation applies `ExcludeTag=@('Integration','Slow')`:
Expand Down
Loading
Loading