RFC: Provenance as a derived view over existing stores (Roadmap Phase 2 — artifact lineage without a parallel capture log)

[imjszhang]

Preflight checklist

• I searched existing issues and discussions and this hasn't been proposed yet.

What problem does this solve?

The Roadmap names Phase 2's provenance chain as the project's core differentiation:

⬜ Artifact versioning and a provenance chain (code + execution log + dependency graph + environment snapshot + conversation context) tied to every output

There is currently no tracked issue for it, so this RFC proposes an implementation direction to help scope that milestone before any code is written.

The obvious way to build provenance — and the way the main competing project built it — is a parallel capture log: listen to tool events and append records to a dedicated provenance file, alongside the data the app already persists. Having studied that implementation, its costs are visible in practice: a second source of truth that drifts from the first, duplicated schemas across layers, O(n) rewrite-time reads on a hot write path, remote runs that are only captured if the agent voluntarily calls a recording script, failed runs that vanish from lineage, and no way to backfill history that predates the log.

This RFC argues that this codebase is in an unusually good position to not take that path.

Key observation: the provenance data already exists here

For any completed session, the lineage of an artifact is already recorded — just across three stores that are never joined:

Store	What it already holds
sessions/<project>/<session>.json	Full conversation, PersistedToolActivity (raw input/output, terminal output), message ↔ artifactIds links
notebooks/<project>/<session>/run.json	Every execution: runId, script, outputs, status, environment, runtimeBindings (incl. EnvProvenance classification), workingFiles[].createdByRunId
artifacts/<project>/<session>/	Ownership encoded in the id (sessionId:ownerId:filename), .runs/<runId>.json finalize markers, .pending/current-run.json turn context

Several forward-looking anchors are also already in place: ArtifactReference.versionId (reserved, commented "future version switcher"), PersistedArtifact.sha256 (typed but never computed), NotebookRunRecord.artifacts[] (typed but always empty), and the metadata.open_science.runId projection in the ipynb exporter (#293 / #296).

What is missing is not capture — it is a read layer that joins what is already captured.

Proposed direction: provenance as a derived view, not a second system

Phase A — read-only lineage query (zero new writes, zero risk)

• A main-process provenance query module: given an artifact id/path, reverse-join the three stores and answer "which message and which run produced this file, with what script, in which environment, in which conversation."
• Surface it as a "History / Source" panel on artifact details (and the session notebook view).
• No new files, no schema changes, no migration. Old sessions are covered automatically because the query reads the stores they already live in.
• Crucially, this phase is also a measurement instrument: wherever the join fails, we learn precisely which anchor is missing — so the schema evolves from observed gaps rather than up-front design.

Phase B — fill only the joins that Phase A proves broken

Known candidates (all are field additions to existing records, not new stores):

1. Compute sha256 at artifact finalize time (the field already exists in PersistedArtifact).
2. Add notebookRunId / turnMessageId to .pending/current-run.json so agent turns and notebook runs get a structural cross-reference (today they are only weakly related via a shared session id).
3. Actually populate NotebookRunRecord.artifacts[].

Environment capture stays at the session/binding granularity that runtimeBindings already provides, recorded incrementally on change — not a per-write pip freeze on the hot path.

Phase C — artifact versioning, pulled by demand

Activate the reserved versionId only when users demonstrably overwrite same-named artifacts and need to see prior versions. The simplest storage that works: finalize no longer overwrites, it keeps the previous version's directory. Still no graph database.

Phase D — standard projections, not proprietary formats

The ipynb metadata.open_science namespace is already the provenance dialect for notebooks (#293). Further export targets (RO-Crate, W3C PROV) are projections generated from the query layer on demand — the internal stores never change to accommodate them.

Why this ordering (design rationale)

• Repair existing tools rather than replace them. The three stores are the system of record; provenance should read them, not compete with them. A parallel log is a second truth that must be kept consistent forever.
• Zero migration. A derived view covers historical sessions on day one. A capture log only knows about events after it shipped, and backfill is a permanent asterisk.
• Optimize the moment that matters. Researchers need provenance at read time ("a reviewer asked where this number came from", "I want to reproduce this from three months ago"), not at write time. So the budget should go to query completeness and a one-click reproduce affordance — and capture must never sit synchronously on the agent's execution path.
• Compatibility over elegance. A content-addressed provenance graph store is the elegant end-state and explicitly not the starting point: it brings schema versioning, migration and dual-write phases before it answers a single user question. Phases A–C answer the user question first and leave that option open.
• Consistent with the public-API separation from feat(cli): add headless task SDK and CLI #316. The query layer starts as internal renderer RPC; whether to expose e.g. GET /api/v1/artifacts/:id/lineage on the versioned task API is a later, deliberate decision, keeping the public surface intentionally small.

Non-goals

• No new append-only provenance log file, no .provenance.jsonl equivalent.
• No provenance graph database or content-addressed store in the first iterations.
• No per-write environment snapshotting on the execution hot path.
• Environment snapshot/export tooling (the other half of Phase 2) is complementary but separable.

Relationship to existing work

• [Feature]: Standard .ipynb interoperability — export session notebooks to nbformat (and later import / open in JupyterLab) #293 / feat(notebook): export sessions as ipynb #296 / feat(notebook): import ipynb sessions #309: the ipynb runId anchor is the model for how provenance surfaces externally.
• Reviewer TurnScope already performs turn-level artifact scoping with content digests — an existing in-codebase precedent for deriving lineage from the stores rather than logging it separately.

Each phase is independently shippable and reviewable. I'm happy to contribute, starting with Phase A, if the maintainers agree with the direction — and equally happy for this to just serve as decision input for scoping the Phase 2 milestone.

[ewen-poch]

Thanks for the exceptionally thorough write-up — the observation that the lineage data already exists across the three stores and just needs a join layer is exactly the right framing, and it matches how we see it internally.

To be transparent: this milestone is already under active design on our side. Because provenance touches the session store, the notebook runtime, artifact finalize, the ipynb projection, and the public task API all at once, the internal design has some constraints and moving parts that are not fully visible from the outside yet, and the direction we land on may differ from the phasing proposed here in ways that would be hard to reconcile after the fact.

For that reason we would advise against independently building this out as a standalone contribution for now — not because the direction is wrong (much of it aligns with our thinking), but because it is a deeply cross-cutting piece with a lot of interdependencies, and parallel work would likely collide with the in-flight design.

We would very much like to keep this issue open as decision input for scoping the Phase 2 milestone — the analysis of existing anchors (versionId, sha256, NotebookRunRecord.artifacts[], the metadata.open_science projection) and the read-vs-write timing argument are genuinely useful to us. Once the internal design firms up we will share it here so there is a concrete surface to collaborate on, and we would welcome your involvement at that point.

Really appreciate the depth here.

[ewen-poch]

Thanks again for the thoughtful RFC. Provenance has now landed on main, primarily in #481, with the environment-history follow-up in #489.

The shipped implementation covers the core user outcome described here:

• immutable, session-scoped Artifact lineages and Versions;
• SHA-256 checksums for Artifact bytes and immutable evidence;
• producer code, execution logs, exact input Versions, environment inventory, conversation/Branch context, and version-scoped review evidence;
• a Provenance UI with version navigation and Code, Execution Log, Messages, Environment, and Review views;
• structural links between the producing run and the prompt/message Branch, plus recovery and migration validation.

We did make a different architectural choice from the RFC's proposed zero-write derived view. The implementation uses SQLite authority records plus immutable evidence sidecars so that the audit record can survive Session deletion, Branch edits, restarts, and data-root migration, while failing closed when ownership or bytes cannot be verified.

There are still several items mentioned here that are not implemented on main:

• Pre-provenance historical Artifacts are not backfilled automatically. The UI explicitly reports that Provenance is unavailable for a legacy file.
• Deterministic replay / one-click reconstruction, portable environment lock export and restore, and full-fidelity Session replay remain future work. The current environment evidence is an observed inventory, not a solver lock.
• RO-Crate and W3C PROV projections are not implemented; the existing .ipynb projection remains the only standards-oriented export discussed here.
• Provenance queries are currently internal renderer IPC. There is no public GET /api/v1/artifacts/:id/lineage (or equivalent SDK method) yet.
• NotebookRunRecord.artifacts[] is still initialized empty for native runs. The shipped design instead makes the immutable Artifact Version's producerRunId and evidence record authoritative, so the lineage use case is covered, but that literal Phase B candidate was not implemented.

So Phases A-C are substantially delivered from a product perspective, with a stronger durable evidence model replacing some of the proposed joins, while historical backfill, deterministic reconstruction, portable environment restoration, and standard/public projections remain open follow-up areas.