chore(evals): migrate agent behavioral conformance from beval to Vally

[vyta]

Pull Request

Description

The beval harness was the only evaluation system in this repository that did not run on Vally. It carried its own package.json and lockfile under evals/beval/, its own workflow, and its own assertion vocabulary. This PR retires it and rebuilds the behavioral conformance coverage as six native Vally suites, so agent conformance, skill quality, agent behavior, and script validation all share one runner, one spec schema, and one reporting path.

What lands

Six new suites under evals/agent-conformance/, one per planner agent (accessibility, dt-coach, privacy, rai, security, sssc), totaling 80 stimuli. Each stimulus is two turns: turn 0 launches the agent artifact, turn 1 delivers the case, and a model judge grades turn 1 against a per-stimulus rubric. A wall-time grader carries an advisory latency budget, and four dt-coach-family suites add an output-contains grader.

.github/workflows/agent-conformance.yml runs the suites as a fail-fast: false matrix, invoked by weekly-validation.yml on the Monday schedule. evals/beval/** and .github/workflows/beval.yml are deleted.

Scoring semantics were corrected, not carried over

This is the substantive fix in the migration and worth reviewer attention.

The beval specs used case_pass: 0.5, which meant the fraction of assertions within a case that must pass — a meaningful number when a case carried several assertions. The first draft of these Vally specs carried 0.5 forward into scoring.threshold, but in Vally threshold is the aggregate score across graders for a single stimulus.

Vally 0.9.0's computeWeightedScore equal-averages when scoring.weights is absent, and resolveGradePass is score >= threshold. With two graders, a judge score of 0 plus a wall-time pass yields (0 + 1) / 2 = 0.5, which clears a 0.5 threshold. Every one of the 80 stimuli would have passed simply by responding within 120 seconds, regardless of what the agent said.

Each suite now declares explicit weights that sum to 1.0, with the threshold set so the judge alone is decisive:

scoring:
  weights:
    prompt: 0.85
    wall-time: 0.15
  threshold: 0.85

The three-grader suites use prompt: 0.7 / wall-time: 0.15 / output-contains: 0.15 with threshold: 0.7. Verification of this behavior is described under Testing.

Supporting cleanup

• package.json — removed the audit:npm:beval script, which pointed at the deleted evals/beval/ package root and would have failed on the next npm run audit:npm (invoked by pr-validation.yml).
• .github/instructions/ci-owned-validation.instructions.md — retargeted applyTo from the deleted beval.yml to agent-conformance.yml and rewrote the corresponding body rule.
• docs/contributing/validation.md — dropped the evals/beval package-root row, added an agent-conformance lane row, and replaced the ## Beval workflow section with ## Agent conformance workflow.
• evals/README.md — added agent-conformance/ to the architecture tree and the executor table, plus run commands.
• docs/planning/adrs/0002-adopt-vally-as-agent-and-skill-behavior-evaluation-framework.md — ms.date refreshed.

Related Issue(s)

Fixes #2479

Type of Change

Select all that apply:

Code & Documentation:

• Bug fix (non-breaking change fixing an issue)
• New feature (non-breaking change adding functionality)
• Breaking change (fix or feature causing existing functionality to change)
• Documentation update

Infrastructure & Configuration:

• GitHub Actions workflow
• Linting configuration (markdown, PowerShell, etc.)
• Security configuration
• DevContainer configuration
• Dependency update

AI Artifacts:

• Reviewed contribution with hve-builder and addressed all actionable findings
• Copilot instructions (.github/instructions/*.instructions.md)
• Copilot prompt (.github/prompts/*.prompt.md)
• Copilot agent (.github/agents/*.agent.md)
• Copilot skill (.github/skills/*/SKILL.md)
• Copilot hook (.github/hooks/*/*.json)
• Eval spec added/updated for changed AI artifacts (evals/)

Note for AI Artifact Contributors:

• Agents: Research, indexing/referencing other project (using standard VS Code GitHub Copilot/MCP tools), planning, and general implementation agents likely already exist. Review .github/agents/ before creating new ones.
• Skills: Must include both bash and PowerShell scripts. See Skills.
• Model Versions: Only contributions targeting the latest Anthropic and OpenAI models will be accepted. Older model versions (e.g., GPT-3.5, Claude 3) will be rejected.
• See Agents Not Accepted and Model Version Requirements.

Other:

• Script/automation (.ps1, .sh, .py)
• Other (please describe):

Testing

All commands below ran in throwaway containers with no network access.

Static validation — passing:

Check	Result
npx vally lint --eval-spec evals/	Exit 0. Only 6 regression-without-baseline warnings (see Additional Notes). No scoring-weight-sum error and no scoring-weight-coverage warning, which is Vally's own validator confirming the weights sum to 1.0 and that the declared grader types match those actually used.
scripts/evals/Test-EvalSpec.ps1	Validated 16 eval specs, 0 failed. 75 tags checked, 0 orphaned. 40/40 agent coverage.
markdownlint-cli2 on changed docs	0 errors.
Validate-MarkdownFrontmatter.ps1 -WarningsAsErrors -EnableSchemaValidation	814 files validated, 0 errors, 0 warnings.
markdown-table-formatter	Clean after reformatting docs/contributing/validation.md.

Scoring verification — executed, not inferred:

Because the scoring defect above was silent under static analysis, it was verified by execution rather than by reading. A throwaway script imported the installed Vally 0.9.0 gradeTrajectory, createGraderRegistry, and resolveGradePass, registered stub graders with scripted scores, and read the real weights and threshold out of each committed eval.yaml. Across all 80 stimuli:

• Judge passes, others fail → PASS (0.850 / 0.824 depending on suite)
• Judge fails, others pass → FAIL (0.150 / 0.176)
• Pre-fix configuration (no weights, threshold: 0.5) → PASS at 0.500 even with a failing judge

The judge is decisive for every stimulus in the suite. The script was deleted afterward and is not part of this PR.

Checklist

Required Checks

• Documentation is updated (if applicable)
• Files follow existing naming conventions
• Changes are backwards compatible (if applicable)
• Tests added for new functionality (if applicable)

AI Artifact Contributions

• Used hve-builder review mode to review contribution
• Addressed all actionable findings from the hve-builder review
• Verified contribution follows common standards and type-specific requirements

Required Local Checks

The following local-safe validation commands must pass before merging:

• Local validation aggregate: npm run validate:local
• Documentation validation (if docs changed): npm run validate:docs
• Spell checking: npm run spell-check
• Link validation: npm run lint:md-links

Security Considerations

• This PR does not contain any sensitive or NDA information
• Any new dependencies have been reviewed for security issues — (N/A — no dependencies added; evals/beval/package.json and its lockfile are removed)
• Security-related scripts follow the principle of least privilege — (N/A — no scripts/security/ changes)

Additional Notes

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.74%. Comparing base (2d1d25c) to head (3e11622).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2552      +/-   ##
==========================================
- Coverage   82.75%   82.74%   -0.01%     
==========================================
  Files         155      155              
  Lines       20960    20960              
  Branches       13       13              
==========================================
- Hits        17345    17344       -1     
- Misses       3613     3614       +1     
  Partials        2        2              

Flag	Coverage Δ
docusaurus	94.44% <ø> (ø)
pester	86.18% <ø> (-0.02%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.