API v2: report validations as an append-only, provenance-keyed resource on reports

[PavelMakarchuk]

@anth-volk — raising this now so the v2 reports stage can be designed with it in mind, rather than retrofitting it.

Context

policyengine-app-v2 now validates every flagship report on two data sources: the calibration dashboard (how well the microdata fits administrative targets for the variables a reform moves) and the scorecard (PolicyEngine versus external measurement for the programs it touches). The variables come from a dependency map traced from the model (PolicyEngine/policyengine-core#543); the matching is in PolicyEngine/policyengine-app-v2#1180 and #1184. On first view the app pins what it matched against — populace release, model version, per-variable fit, scorecard programs — and later views report drift against that pin (#1185).

Today the pin lives in the browser's localStorage, because the app's own reports table was never connected to a database, deliberately, since v2 is where user-owned report data is going. The validation snapshot is a fact about the base report, not about one user's view of it, so it should live with the report in v2 rather than in the app or on a user association.

Proposal: report_validations

One row per validation run, never overwritten, keyed to the base report:

column	type	purpose
id	uuid
report_id	fk → reports, indexed	the base API report
computed_at	timestamptz	when the run happened
model_version	text	policyengine-us the report was computed with
dependency_map_fingerprint	text	which traced map resolved paths to variables (core#543 stamps this)
populace_release_id	text, nullable	which calibration release was read
scorecard_build	text, nullable	which scorecard file was read
calibration	jsonb, nullable	geography, reached count, per-variable ring/depth/fit/worst target
scorecard	jsonb, nullable	programs (and later rows)

Endpoints:

• GET /v2/reports/{id}/validations → latest plus history, newest first
• POST /v2/reports/{id}/validations → append a run (write-once per run; never update)

Drift is then a diff between the latest row's populace_release_id / model_version and the current ones, which is how the tracker already treats bill validations.

JSON the app writes today

This is the shape already in production behind the app's localStorage fallback; calibration and scorecard map onto it directly, and the top-level keys become columns:

{
  "matched_at": "2026-09-07T08:55:20.772Z",
  "map_model_version": "1.808.0",
  "calibration": {
    "release_id": "populace-us-2024-buildp-sparse-rmloss100-cae8640-20260728T011454Z",
    "geography": "US",
    "reached_count": 66,
    "matches": [
      {
        "variable": "refundable_ctc",
        "depth": 2,
        "ring": "mechanism",
        "target_count": 2,
        "mean_abs_relative_error": 0.008,
        "worst": { "name": "irs_soi.ty2022.historic_table_2.us.all.actc_amount@2024", "source": "irs_soi", "geography": "US", "relative_error": -0.008 }
      }
    ]
  },
  "scorecard": { "programs": ["ctc_refund"] }
}

ring is one of primary (0–1 hops from the parameter), mechanism (2–4), downstream (5+). Errors are fractions.

Staged path

1. Table and endpoints land with the reports stage. Contained addition; the app changes only the store it posts to.
2. The app posts its snapshot on the first settled view instead of writing localStorage. Durable and shared across viewers from that day.
3. Computation moves into the API worker at report completion, once core#543 releases and the dependency map ships as a release asset per model version. The worker knows the exact model version it just used, so provenance stops being inferred.
4. A scheduled job re-validates reports whose pinned release or model is behind, appending rows. Drift becomes something the system records rather than something a browser notices.

Steps 1–2 need nothing from core; 3–4 wait on core#543.

Questions for you

• Does a separate table fit the reports stage, or would you rather hang it off the base report resource? Separate keeps history and avoids widening reports.
• Should POST be open to the app (anonymous trust model, like today's association writes) or restricted to the worker once step 3 lands?
• Anything in the v2 conventions (naming, envelope, pagination) the endpoint shapes above should follow?

References: PolicyEngine/policyengine-app-v2#1180, #1184, #1185, #1186; PolicyEngine/policyengine-core#543.