diff --git a/Docs/provenance_forge_pointers.md b/Docs/provenance_forge_pointers.md new file mode 100644 index 0000000..d4198d0 --- /dev/null +++ b/Docs/provenance_forge_pointers.md @@ -0,0 +1,109 @@ +# Forge pointers and harness snapshot join contract + +> Related: APPENG-4985 (persistence), APPENG-5370 Phase C (MLflow observer), +> APPENG-5300 (eval engine intake). +> +> **This document is the cross-project join contract.** Agent runtimes, eval +> engines, and certification pipelines link here. Do not invent a parallel +> schema elsewhere. Eval engines that already use “provenance” for +> *case-generation source* must keep that meaning separate from these forge +> join keys. + +## Purpose + +Three observability streams stay separate; **forge / harness join pointers** +correlate them: + +| Stream | Typical artifact | Writer role | +|--------|------------------|-------------| +| Harness snapshot | `harness-snapshot.json` | Agent runtime (run start) | +| Runtime trace | OTLP / local telemetry JSONL; and/or MLflow GenAI traces | Agent runtime and/or eval engine | +| Eval verdict | Scorecard / MLflow metrics + judge feedback | Eval / certification pipeline | + +`abevalflow.report.Provenance` is the **canonical typed join record** on an +analysis report (fields below). The harness snapshot JSON and MLflow tags use +the same semantic names (with the aliases noted in the field table). + +## Ownership matrix + +| Artifact | Writer | Canonical store | Reader discovery | +|----------|--------|-----------------|------------------| +| Harness snapshot | Agent runtime | Runtime run output directory; mirrored on OTel root-span attributes | File name `harness-snapshot.json` beside Level-1 telemetry | +| Snapshot on eval run | Eval engine MLflow logger | MLflow run **artifact** `harness-snapshot.json` + projected **tags** | Experiment + run name / `eval_run_id` (same join as `inputs/` artifacts) | +| Runtime OTel spans | Agent runtime | OTLP backend and/or `run-telemetry.jsonl` | Existing runtime tracing config | +| MLflow GenAI traces | Eval engine | MLflow experiment traces | `tracking_uri` + experiment; tag `eval_run_id` | +| `Provenance` on report | Certification / analyze pipeline | Analysis result / scorecard | ABEvalFlow report APIs | + +**Disk under the eval run tree is only the handoff into the MLflow logger.** +After logging, **canonical read for join keys is MLflow** (artifact + tags), +not re-scraping CI environment variables. + +## Field table + +| Provenance / MLflow tag | Snapshot JSON (when present) | Required | OTel (when on spans) | Notes | +|-------------------------|------------------------------|----------|----------------------|-------| +| `generated_at` | `recorded_at` | yes (default now) | — | Report timestamp vs snapshot write time | +| `commit_sha` | `ref_revision` | no | `vcs.ref.head.revision` | Same semantic | +| `pipeline_run_id` | `pipeline_run_id` | no | `cicd.pipeline.run.id` | | +| `pipeline_run_url` | `pipeline_run_url` | no | — | Optional convenience URL | +| `repository_url` | `repository_url` | no | `vcs.repository.url.full` | | +| `change_id` | `change_id` | no | `vcs.change.id` | PR/MR number | +| `ref_name` | `ref_name` | no | `vcs.ref.head.name` | Branch/tag short name | +| `trace_id` | `trace_id` | no | trace id on root span | W3C / backend id | +| `session_id` | — | no | `gen_ai.conversation.id` | | +| `eval_run_id` | — | no | — | Eval batch/case id; MLflow run name / tag | +| `harness_fingerprint` | `harness_content_sha` | no | `fullsend.harness.content_sha` (runtime-specific attr ok) | Config content hash | +| `forge_platform` | `forge_platform` | no | runtime-specific | `github` \| `gitlab` \| `bitbucket` \| `forgejo` \| … | +| `eval_engine` | — | yes on Provenance (default `harbor`) | — | `harbor` \| `ase` \| `a2a` \| `aeh` \| `mcpchecker` \| `both` | +| `treatment_image_ref` | — | no | — | Image digest refs | +| `control_image_ref` | — | no | — | | +| `harbor_fork_revision` | — | no | — | | + +Snapshot-only identity fields (not required on `Provenance`): `schema_version`, +`agent`, `role`, `slug`, `model`, `harness_path`, `skills`, `traceparent`. + +Existing consumers that only set `commit_sha` / `pipeline_run_id` remain valid. + +## Harness snapshot lifecycle + +1. **Write (agent runtime):** At run start, after the root span exists, write + `harness-snapshot.json` into the run output directory (next to Level-1 + telemetry). Fill forge/CI fields from process env / explicit overrides at + write time. Mirror join keys on the root span attributes. +2. **Handoff (eval case output):** Opaque / skill runners copy the file to a + stable path under the case or run output tree (same convention as other + case artifacts such as metrics). +3. **Log (eval engine):** If the file is present when logging results, upload + it as MLflow artifact **`harness-snapshot.json`** and project non-empty join + fields to MLflow tags (names in the field table; use `commit_sha` for + `ref_revision`). +4. **Read (later consumers):** Resolve experiment → MLflow run by + `tags.mlflow.runName = ''` (or equivalent) → + `download_artifacts(..., "harness-snapshot.json")`. Prefer tags when only + scalars are needed. Do **not** treat ambient CI env as the source of truth + when the artifact or tags exist. + +## MLflow tag mapping (Phase C observer) + +When `MLflowObserver` (or an eval engine logger) logs a run, prefer these +tag names (match Provenance fields): + +`commit_sha`, `pipeline_run_id`, `repository_url`, `change_id`, `trace_id`, +`session_id`, `eval_run_id`, `harness_fingerprint`, `forge_platform`, +`eval_engine`. + +Eval engines may also emit `ref_revision` alongside `commit_sha` when mirroring +snapshot JSON; prefer `commit_sha` for Provenance. + +## Naming hygiene for downstream PRs + +- Contract and ADR **Decision** text describe **roles** (agent runtime, eval + engine MLflow logger), not other products’ module paths. +- Eval engine public docs must not claim ownership of this schema. +- Agent-runtime ADRs link here under Related; they do not redefine the field + list. + +## Scorecard + +`Scorecard.provenance` remains a free-form `dict` for extras. Prefer copying +known keys from `AnalysisResult.provenance` so Compass/publish stay consistent. diff --git a/abevalflow/report.py b/abevalflow/report.py index 1395f9e..b9e5fd4 100644 --- a/abevalflow/report.py +++ b/abevalflow/report.py @@ -110,17 +110,56 @@ class VariantSummary(BaseModel): class Provenance(BaseModel): - """Run provenance metadata for reproducibility.""" + """Run provenance metadata for reproducibility and forge/trace join keys. + + See Docs/provenance_forge_pointers.md for the cross-project join contract. + """ generated_at: datetime = Field(default_factory=lambda: datetime.now(UTC)) commit_sha: str | None = None pipeline_run_id: str | None = None + pipeline_run_url: str | None = Field( + default=None, + description="CI pipeline run URL (convenience; complements pipeline_run_id)", + ) + ref_name: str | None = Field( + default=None, + description="Branch or tag short name (OTel vcs.ref.head.name)", + ) treatment_image_ref: str | None = None control_image_ref: str | None = None harbor_fork_revision: str | None = None eval_engine: str = Field( default="harbor", - description="Evaluation engine used: 'harbor', 'ase', or 'both'", + description="Evaluation engine used: 'harbor', 'ase', 'a2a', 'aeh', 'mcpchecker', or 'both'", + ) + repository_url: str | None = Field( + default=None, + description="Canonical repo URL (OTel vcs.repository.url.full)", + ) + change_id: str | None = Field( + default=None, + description="PR/MR number (OTel vcs.change.id)", + ) + trace_id: str | None = Field( + default=None, + description="W3C or MLflow trace id for the evaluation run", + ) + session_id: str | None = Field( + default=None, + description="Session / conversation id (OTel gen_ai.conversation.id)", + ) + eval_run_id: str | None = Field( + default=None, + description="Eval engine batch/case run id when applicable", + ) + harness_fingerprint: str | None = Field( + default=None, + description="Harness/config content hash for reproducibility", + ) + forge_platform: str | None = Field( + default=None, + description="Git hosting platform: github | gitlab | bitbucket | forgejo", ) diff --git a/scripts/analyze.py b/scripts/analyze.py index 98a71dc..b096abf 100644 --- a/scripts/analyze.py +++ b/scripts/analyze.py @@ -633,13 +633,42 @@ def main(argv: list[str] | None = None) -> int: ) parser.add_argument("--commit-sha", default=None) parser.add_argument("--pipeline-run-id", default=None) + parser.add_argument("--pipeline-run-url", default=None, help="CI pipeline run URL") + parser.add_argument("--ref-name", default=None, help="Branch or tag short name") parser.add_argument("--treatment-image-ref", default=None) parser.add_argument("--control-image-ref", default=None) parser.add_argument("--harbor-fork-revision", default=None) + parser.add_argument( + "--repository-url", + default=None, + help="Canonical repository URL (forge pointer)", + ) + parser.add_argument( + "--change-id", + default=None, + help="PR/MR number (forge pointer)", + ) + parser.add_argument("--trace-id", default=None, help="W3C/MLflow trace id") + parser.add_argument("--session-id", default=None, help="Session/conversation id") + parser.add_argument( + "--eval-run-id", + default=None, + help="Eval engine run / case id", + ) + parser.add_argument( + "--harness-fingerprint", + default=None, + help="Harness/config content hash", + ) + parser.add_argument( + "--forge-platform", + default=None, + help="github | gitlab | bitbucket | forgejo", + ) parser.add_argument( "--eval-engine", type=str, - choices=["harbor", "ase", "both", "a2a", "aeh"], + choices=["harbor", "ase", "both", "a2a", "aeh", "mcpchecker"], default="harbor", help="Evaluation engine used (for provenance tagging and analysis path)", ) @@ -688,10 +717,19 @@ def main(argv: list[str] | None = None) -> int: provenance = Provenance( commit_sha=args.commit_sha, pipeline_run_id=args.pipeline_run_id, + pipeline_run_url=args.pipeline_run_url, + ref_name=args.ref_name, treatment_image_ref=args.treatment_image_ref, control_image_ref=args.control_image_ref, harbor_fork_revision=args.harbor_fork_revision, eval_engine=args.eval_engine, + repository_url=args.repository_url, + change_id=args.change_id, + trace_id=args.trace_id, + session_id=args.session_id, + eval_run_id=args.eval_run_id, + harness_fingerprint=args.harness_fingerprint, + forge_platform=args.forge_platform, ) result = build_analysis( diff --git a/tests/test_analyze.py b/tests/test_analyze.py index 58514c9..1bf2244 100644 --- a/tests/test_analyze.py +++ b/tests/test_analyze.py @@ -354,6 +354,26 @@ def test_provenance_passthrough(self, results_dir: Path): assert result.provenance.commit_sha == "abc123" assert result.provenance.pipeline_run_id == "run-42" + def test_provenance_forge_join_fields(self, results_dir: Path): + prov = Provenance( + commit_sha="abc123", + pipeline_run_url="https://ci.example.com/runs/42", + ref_name="feat/thing", + repository_url="https://github.com/o/r", + change_id="7", + trace_id="4bf92a", + eval_run_id="20260722-1", + harness_fingerprint="deadbeef", + forge_platform="github", + ) + result = build_analysis(results_dir, "my-submission", provenance=prov) + assert result.provenance.repository_url == "https://github.com/o/r" + assert result.provenance.change_id == "7" + assert result.provenance.pipeline_run_url == "https://ci.example.com/runs/42" + assert result.provenance.ref_name == "feat/thing" + assert result.provenance.harness_fingerprint == "deadbeef" + assert result.provenance.forge_platform == "github" + def test_trials_included(self, results_dir: Path): result = build_analysis(results_dir, "my-submission") assert len(result.trials["treatment"]) == 5