Consolidate Evaluation Dataset Creator supporting docs into one eval guide and remediate Responsible AI gaps

[WilliamBerryiii]

Summary

The Evaluation Dataset Creator agent currently emits 5 files per run: 2 dataset serializations (.json + .csv) plus 3 separate Markdown documents ({agent-name}-curation-notes.md, {agent-name}-metric-selection.md, {agent-name}-tool-recommendations.md).

This is the third wave of the agent-output asset-reduction effort, following the DS Gen Data Spec consolidation (4 files to 3) and the RAI Planner consolidation (8 phase files to one plan).

A Responsible AI review of the same agent found six additive gaps, and the scope was expanded to close them on the same branch because both bodies of work edit the same two files.

Proposed change

Part one: consolidate the supporting documents

Merge the three supporting documents into a single sectioned guide, data/evaluation/docs/{agent-name}-eval-guide.md, with three top-level sections:

• ## Curation Notes
• ## Metric Selection
• ## Tool Recommendations

Every existing subsection, table, and checklist is preserved; only the packaging changes. The dataset .json + .csv dual-format output is unchanged.

Result: 5 files per run becomes 3.

Part two: Responsible AI remediation

The agent builds a measuring instrument, so every gap lands in the NIST AI RMF Measure function, where a defect yields false assurance about some other system rather than an obviously wrong answer. The remediation is additive:

• Route the Responsible AI interview answer (Question 5) into a ### Responsible AI Risks mapping in the guide's Metric Selection section, so a captured risk reaches a deliverable.
• Add the missing #### Responsibility and Safety Metrics table (Fairness, Harmful Content, Groundedness under adversarial input) with matching definition entries, closing the gap where the agent advertised Azure AI Foundry Risk/Safety metrics with no way to select one.
• Give representativeness a real schema axis: metadata.population_coverage as a sibling of distribution, plus a generation rule. Name each distinct user population in Question 6.
• Record provenance with metadata.validation_status and metadata.generation_method, state that the dataset is provisional while the status is ai-generated, and tie the Domain Expert Review checkboxes to updating that status.
• Bound generated content: pairs are synthesized rather than reproduced from real records, and safety and negative pairs record the disallowed request category and the expected refusal rather than the prohibited content itself.

The graded difficulty enum and the distribution keys are deliberately untouched, so datasets already produced under the current contract stay valid.

Edit surface

• .github/agents/data-science/eval-dataset-creator.agent.md — Output Artifacts tree, Phase 7 documentation prose and templates, Phase 1 interview wording, Dataset Requirements, the JSON schema block, the metric tables, and one appended Required Protocol item.
• evals/agent-behavior/expectations/eval-dataset-creator.expectations.yml — the three-supporting-docs, persona-driven-tool-recommendation, and output-path-under-data-evaluation expectations are the only filename-keyed consumers of the three docs, plus one new rai-metadata-completeness expectation.

No script, downstream agent, or CI schema reads the generated documents; they are terminal user deliverables.

Acceptance

• The agent file lists one guide file and no longer references the three separate document filenames.
• The eval spec asserts the single guide and its three sections, with no residual three-document assertion.
• metadata carries population_coverage, validation_status, and generation_method, while distribution keeps exactly its five difficulty keys and difficulty keeps exactly its five permitted values.
• Exactly one new eval expectation is added; every pre-existing expectation other than the three named above is byte-unchanged.
• Markdown, table, YAML, frontmatter, generated-docs, and asset-docs checks pass.