docs: freeze forge pointer and harness snapshot join contract

[HofniGartner]

Summary

Freeze the cross-project join contract for harness snapshot / forge pointers, and extend Provenance so the typed report matches that field list.

Changes

• Add Docs/provenance_forge_pointers.md (ownership matrix, field mapping, snapshot lifecycle, MLflow-first later read)
• Extend Provenance with optional join fields (repository_url, change_id, trace_id, session_id, eval_run_id, harness_fingerprint, forge_platform)
• Wire matching --* flags in scripts/analyze.py
• Add passthrough test for forge join fields

Test plan

• uv run pytest tests/test_analyze.py -k provenance (or project venv equivalent)
• Review Docs/provenance_forge_pointers.md for field/ownership accuracy

Related

• APPENG-4985 / APPENG-5370 (join / MLflow observer context)
• Follow-up: agent runtime ADR + eval-engine MLflow consumer (separate repos)

Made with Cursor

[GuyZivRH]

great direction! do you extend this PR or open a following PR with the rest of the work?

[HofniGartner]

Thanks Guy!

This PR is intentionally the schema / contract slice only:

• freeze Docs/provenance_forge_pointers.md (ownership, fields, snapshot lifecycle)
• extend typed Provenance + analyze CLI flags to match

Following PRs (separate repos / issues), not stacked here:

• fullsend — ADR 0071 accepted as docs in docs(adr): accept ADR 0075 harness snapshot for agent traces fullsend-ai/fullsend#5524; runtime producer (harness-snapshot.json + root-span attrs) tracked in Implement ADR 0071: harness-snapshot.json + root-span forge pointers fullsend-ai/fullsend#5449
• agent-eval-harness — thin MLflow consumer (disk handoff at log_results, later fetch like inputs/) in feat(mlflow): load harness-snapshot from disk and MLflow opendatahub-io/agent-eval-harness#157

Happy to open a follow-up in ABEvalFlow for Phase C observer wiring once the producer + AEH tags/artifact path are in — that should stay out of this PR so the join contract can land first.

[HofniGartner]

Refreshed this branch onto current main (2026-08-02). Still intentionally the schema/contract slice only — follow-on consumer wiring stays in a later PR. Ready for another look when you have a moment @GuyZivRH.

[GuyZivRH]

PR #57 Consolidated Review — Forge pointer / harness snapshot join contract

PR: #57
Author: HofniGartner
Branch: HofniGartner:feat/join-contract-doc → main (cross-fork)
HEAD: 34ab13f
Diff: +192 / −2 · 4 files · 3 commits
Tickets: APPENG-4985, APPENG-5370, APPENG-5300
CI: no check runs on HEAD
Merge state: BEHIND (~24 commits on main)
Consolidated: 2026-08-06 · prefix mzbqx_
Sources folded in: hvtmq_, xkqre_, wfjkr_, qplmx_, rklwm_

---

Final verdict

Approve with minor fixes — right contract-first slice; merge after rebase, green CI, and doc↔Provenance field alignment.

One prior pass said Request changes for the same schema mismatch (pipeline_run_url / ref_name). Treat that as a must-fix before merge, not as “block forever” or “wire producers in this PR.” Author already scoped this as schema/contract only (fullsend producer + AEH consumer elsewhere).

---

What shipped

Freezes a cross-project join schema so three streams can correlate:

Stream	Artifact	Writer
Harness snapshot	harness-snapshot.json	Agent runtime
Runtime trace	OTel / MLflow GenAI	Runtime / eval engine
Eval verdict	Scorecard / report	ABEvalFlow analyze/store

File	Change
Docs/provenance_forge_pointers.md	Ownership matrix, field table, lifecycle, MLflow-first read
abevalflow/report.py	7 optional join fields on Provenance; eval_engine desc adds aeh
scripts/analyze.py	Matching --* CLI flags
tests/test_analyze.py	test_provenance_forge_join_fields passthrough

New fields: repository_url, change_id, trace_id, session_id, eval_run_id, harness_fingerprint, forge_platform.

---

Consensus strengths

1. Correct owner and problem — ABEvalFlow as typed join-record home; “do not invent a parallel schema.”
2. Clear ownership + write → handoff → log → read lifecycle; MLflow artifact/tags as canonical read after logging.
3. Backward compatible — all new fields optional; existing commit_sha / pipeline_run_id consumers stay valid.
4. OTel-aligned field descriptions (vcs.*, gen_ai.conversation.id).
5. Small, reviewable contract slice; producer/consumer wiring intentionally deferred.
6. Local provenance tests pass (pytest … -k provenance → 4 passed on this HEAD).
7. Full Provenance still lands in Postgres via report_json blob even without new columns.

---

Must fix before merge

1. Rebase + CI

Branch is stale (~24 commits behind). No GitHub check runs on fork HEAD. Rebase on main, ensure test workflow runs (may need maintainer approval for first-time fork contributor), merge only on green.

2. Doc field table vs typed Provenance

Contract table lists under “Provenance / MLflow tag” but code does not add:

• pipeline_run_url
• ref_name

Pick one:

• A: Add both to Provenance + analyze CLI + test, or
• B: Mark them snapshot / MLflow-only (not on Provenance), like other snapshot-only identity fields.

A freeze that over-claims typed fields will mislead fullsend/AEH consumers.

3. eval_engine description drift

Description adds aeh but still omits engines already allowed on main (e.g. a2a, and project-wide values such as mcpchecker where applicable). Align string with analyze choices / EvalEngine / observability tables.

---

Should fix / nice-to-have (non-blocking)

Item	Notes
Test completeness	Assert trace_id / eval_run_id / session_id; optional CLI → report.json round-trip
Tekton wiring	Analyze tasks still only pass commit-sha / pipeline-run-id — follow-up
Other entry points	aggregate_ase / aggregate_mcpchecker narrower Provenance — OK via defaults
MLflowObserver (#64)	Doc lists preferred forge tags; observer does not yet emit full set — follow-up
Cross-link #56	After rebase: link observability_tables.md ↔ forge pointers; note join keys in report_json
CRLF on new doc	Normalize to LF if present
forge_platform	Optional Literal / enum later
Cursor co-author trailers	Prefer not to advertise AI co-authors on future commits

---

Related work

Work	Relationship
#56 observability tables	Postgres column map; join keys mostly in report_json today
#64 MLflow Observer	Should adopt tag names from this contract
#61 AEH MLflow evaluate	Trial-level; separate from fleet forge join keys
fullsend / AEH PRs	Producer + MLflow consumer (out of this PR)

---

Suggested GitHub action

1. Ask author to rebase, fix field-table + eval_engine wording, push.
2. Confirm CI green.
3. Approve and merge.
4. Track follow-ups: Tekton param wiring, observer forge tags, optional hot-join DB columns.

---

Checklist

• Scope matches stated “contract slice”
• Additive optional fields (backward compatible)
• Local provenance tests pass
• Rebased on current main
• Field table matches typed model (pipeline_run_url / ref_name)
• eval_engine description matches allowed engines
• CI green on HEAD

[HofniGartner]

@GuyZivRH — Thanks for the thorough consolidated review. Addressed all must-fix items:

✅ 1. Rebase + CI

Rebased on current main (was ~28 commits behind). Branch is up to date; awaiting CI green (first-time fork contributor approval may be needed).

✅ 2. Doc field table vs typed Provenance

Went with Option A — added both pipeline_run_url and ref_name to:

• Provenance model (abevalflow/report.py)
• CLI flags (scripts/analyze.py)
• Test assertions (tests/test_analyze.py)

Contract doc and typed model now match 1:1.

✅ 3. eval_engine description drift

Updated Provenance.eval_engine description and the doc field table to list all engines currently on main: harbor | ase | a2a | aeh | mcpchecker | both.

Bonus: CRLF → LF

Normalized Docs/provenance_forge_pointers.md line endings.

---

All 4 provenance tests pass locally (pytest -k provenance → 4 passed). Ready for re-review once CI is green.

[GuyZivRH]

make sure CI is green and rebase