feat: intelligence layer, capture depth, schema-first adapters, eval harness, viewer polish

.github/workflows/witness-trace.yml

@@ -0,0 +1,98 @@
+# Reusable workflow: post a witness trace summary as a PR comment.
+#
+# A caller workflow runs an agent task that produces a witness trace (stored in
+# WITNESS_DIR), then calls this workflow. It renders a compact markdown summary
+# for the chosen trace (or the latest one) and posts it as a sticky PR comment.
+#
+# Example caller:
+#
+#   jobs:
+#     trace-summary:
+#       uses: ./.github/workflows/witness-trace.yml
+#       with:
+#         trace_id: ""          # empty -> use the latest trace
+#         witness_dir: .witness # where the trace db/blobs live
+#
+name: witness-trace
+
+on:
+  workflow_call:
+    inputs:
+      trace_id:
+        description: "Trace id to summarize. Leave empty to use the latest trace."
+        required: false
+        type: string
+        default: ""
+      witness_dir:
+        description: "Directory holding the witness database and trace blobs."
+        required: false
+        type: string
+        default: ".witness"
+      python_version:
+        description: "Python version to run with."
+        required: false
+        type: string
+        default: "3.12"
+
+# Needed so the job can write a comment back onto the pull request.
+permissions:
+  contents: read
+  pull-requests: write
+
+jobs:
+  comment:
+    runs-on: ubuntu-latest
+    env:
+      # storage.py reads WITNESS_DIR to locate the trace database and blobs.
+      WITNESS_DIR: ${{ inputs.witness_dir }}
+    steps:
+      - name: Check out repository
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ inputs.python_version }}
+
+      - name: Install witness
+        run: python -m pip install --upgrade pip && pip install -e .
+
+      # Render the markdown summary. scripts/witness_pr_comment.py prints it to
+      # stdout; we stash it in a file so the comment step can read it verbatim.
+      - name: Render trace summary
+        id: summary
+        run: |
+          python scripts/witness_pr_comment.py "${{ inputs.trace_id }}" > witness-summary.md
+          cat witness-summary.md
+
+      # Only post when this run is attached to a pull request.
+      - name: Post PR comment
+        if: ${{ github.event_name == 'pull_request' || github.event.pull_request != null }}
+        uses: actions/github-script@v7
+        with:
+          script: |
+            const fs = require('fs');
+            const body = fs.readFileSync('witness-summary.md', 'utf8');
+            const marker = '<!-- witness-trace-summary -->';
+            const issue_number = context.payload.pull_request.number;
+            const { owner, repo } = context.repo;
+
+            // Reuse a single sticky comment instead of stacking new ones.
+            const existing = await github.paginate(
+              github.rest.issues.listComments,
+              { owner, repo, issue_number }
+            );
+            const mine = existing.find(
+              (c) => c.body && c.body.includes(marker)
+            );
+            const fullBody = `${marker}\n${body}`;
+
+            if (mine) {
+              await github.rest.issues.updateComment({
+                owner, repo, comment_id: mine.id, body: fullBody,
+              });
+            } else {
+              await github.rest.issues.createComment({
+                owner, repo, issue_number, body: fullBody,
+              });
+            }

BACKLOG.md

@@ -1,35 +1,57 @@
 # Backlog
 
-Features intentionally not built for v0. In rough priority order.
+What has shipped and what is still intentionally unbuilt.
+
+## Shipped
+
+- **Share links** - `witness share <trace_id>` uploads a trace to a hosted
+  viewer and returns a public URL.
+- **Regression / divergence detection** - `witness diff trace_a trace_b`
+  compares two runs of the same task step-by-step and surfaces metric deltas and
+  the first point of trajectory divergence. The bench harness computes
+  divergences across a whole suite.
+- **Cost dashboards** - `witness stats` aggregates spend and tokens overall, by
+  model, and by day.
+- **Intelligence layer** - prompt-injection detection, trajectory health,
+  outcome / risk prediction, and PII/secret exfiltration detection, surfaced as
+  findings in the viewer, the API, and `witness analyze`. See
+  [docs/analysis.md](docs/analysis.md).
+- **Reliability harness** - `witness bench` and the WitnessBench reference tasks
+  report success rate, flakiness, and metric distributions. See
+  [docs/bench.md](docs/bench.md).
+- **HTML report exporter + GitHub Action** - `witness report` writes a
+  self-contained HTML report, and a reusable workflow posts a trace summary as a
+  sticky PR comment.
+- **OTLP export** - opt-in span forwarding to Jaeger / Grafana / Honeycomb via
+  `WITNESS_OTLP_ENDPOINT`, with GenAI semantic-convention attributes. See
+  [docs/otel-export.md](docs/otel-export.md).
+- **Schema-first capture adapters** - a versioned canonical schema
+  (`witness/schema.py`) plus an adapter registry. browser_use is fully
+  instrumented; Playwright detection is scaffolded. See
+  [docs/schema.md](docs/schema.md).
+
+## Still unbuilt
+
+### More frameworks
+
+- Playwright agents (detection exists; instrumentation still to wire up).
+- Claude in Chrome event stream subscription.
+- Stagehand, Skyvern, Manus (as users request).
+
+### Real-time streaming viewer
 
-## Share links
-
-`witness share <trace_id>` → uploads trace to hosted viewer, returns a public URL.
-Closed-source hosted service, free tier with sensible limits.
-
-## More frameworks
-
-- Playwright agents (via Agent class detection)
-- Claude in Chrome event stream subscription
-- Stagehand, Skyvern, Manus (as users request)
-
-## Regression detection
-
-`witness diff trace_a trace_b` — compare two runs of the same task.
-Eventually: scheduled runs with alerts on drift.
-
-## Cost dashboards
-
-Aggregate views: spend per day, per model, per task pattern.
+Watch a running agent step-by-step as it executes.
 
-## Real-time streaming viewer
+### OpenAI / Bedrock / Gemini full support
 
-Watch a running agent step-by-step as it executes.
+OpenLLMetry already instruments these; the remaining work is pricing entries and
+testing.
 
-## OpenAI / Bedrock / Gemini full support
+### Learned outcome model out of the box
 
-OpenLLMetry already instruments these; we just need pricing entries and testing.
+The training harness exists behind the optional `[ml]` extra. A shipped,
+pre-trained outcome model (and the calibration to back it) is still open.
 
-## Team features
+### Team features
 
-Workspaces, shared dashboards, RBAC — cloud only, paid.
+Workspaces, shared dashboards, RBAC. Cloud only, paid.

README.md

@@ -73,6 +73,51 @@ Everything is stored locally in SQLite + flat files under `~/.witness/`. Nothing
 
 ---
 
+## Intelligence layer
+
+Capture tells you what the agent did. The intelligence layer tells you whether it
+was safe and on-track. Run `witness analyze <trace_id>` (or `--all`) and the
+findings show up in the viewer, over the API, and in the terminal.
+
+- **Prompt-injection detection** flags indirect prompt injection: it fires on the
+  conjunction of injection-shaped content in the captured DOM (instruction text
+  aimed at an agent, hidden or offscreen text, exfiltration-shaped URLs) and a
+  deviating next action that obeys the injection or drifts off task. Benign pages
+  do not produce false highs.
+- **Trajectory health** scores the step sequence for loops/thrash, task drift
+  (stdlib lexical similarity, no embeddings), wasted/dead-end steps, and
+  backtracking, then rolls them into one health badge per trace.
+- **Outcome / risk prediction** estimates the probability a (possibly partial)
+  run will succeed using a transparent heuristic over cheap features, with an
+  optional ML training harness behind the `[ml]` extra to learn a real classifier
+  from labelled runs.
+- **PII / secret exfiltration detection** catches credentials and PII typed into
+  page fields and DOM-to-action carry-over, with every matched value masked in
+  the stored evidence.
+
+Findings are stored as a versioned `Finding` model (kind, severity, score, title,
+detail, evidence) and served at `GET /api/traces/{id}/findings` and `POST
+/api/traces/{id}/analyze`. Details in [`docs/analysis.md`](docs/analysis.md).
+
+Around the analyzers:
+
+- **Schema-first capture adapters** - a single versioned capture schema
+  (`witness/schema.py`, `SCHEMA_VERSION`) plus an adapter registry under
+  `witness/adapters/` so new frameworks plug in without touching storage, the
+  analyzers, or the viewer. browser_use is fully instrumented; Playwright is
+  scaffolded. See [`docs/schema.md`](docs/schema.md).
+- **Reliability harness** - `witness bench` and the WitnessBench reference tasks
+  run a task repeatedly and report success rate, flakiness, metric distributions,
+  and trajectory divergence. See [`docs/bench.md`](docs/bench.md).
+- **HTML report exporter + GitHub Action** - `witness report` writes a
+  self-contained HTML report for a trace, and a reusable workflow posts a trace
+  summary as a sticky PR comment.
+- **OTLP export** - set `WITNESS_OTLP_ENDPOINT` to forward LLM spans to Jaeger,
+  Grafana, or Honeycomb with OpenTelemetry GenAI semantic-convention attributes,
+  while local capture keeps working. See [`docs/otel-export.md`](docs/otel-export.md).
+
+---
+
 ## Screenshots
 
 ![trace list](docs/screenshots/01_list.png)
@@ -114,6 +159,12 @@ witness view              # open the viewer at localhost:7842
 witness ls                # list recent traces in the terminal
 witness rm <trace_id>     # delete a single trace
 witness rm --all          # delete every trace (asks first)
+witness analyze <id>      # run the analyzers and store findings (--all for every trace)
+witness bench [suite]     # reliability report across captured runs
+witness report <id>       # write a self-contained HTML report for a trace
+witness diff a b          # compare two traces step-by-step
+witness stats             # aggregate cost and token usage
+witness share <id>        # upload a trace to a hosted viewer
 witness config            # show config path and current settings
 ```
 
@@ -125,10 +176,10 @@ Telemetry is off by default and there is no toggle to turn it on. Witness never
 
 Short list of what's coming, roughly in order:
 
-- **Share links** — `witness share <trace_id>` uploads a trace to a hosted viewer so you can paste it into a bug report or a Slack thread
-- **More frameworks** — Playwright-based agents first, then Claude in Chrome event streams
-- **Regression detection** — run the same task against two model versions, diff the traces, alert on drift
-- **Cost dashboards** — aggregate spend across traces and tasks
+- **More frameworks** - finish Playwright instrumentation, then Claude in Chrome event streams
+- **Real-time streaming viewer** - watch a running agent step-by-step as it executes
+- **Pre-trained outcome model** - ship a learned outcome predictor, not just the heuristic and training harness
+- **OpenAI / Bedrock / Gemini** - full pricing and testing for the providers OpenLLMetry already instruments
 
 Full backlog: [`BACKLOG.md`](BACKLOG.md)