feat: intelligence layer, capture depth, schema-first adapters, eval harness, viewer polish#2
Open
EricFinland wants to merge 8 commits into
Open
feat: intelligence layer, capture depth, schema-first adapters, eval harness, viewer polish#2EricFinland wants to merge 8 commits into
EricFinland wants to merge 8 commits into
Conversation
… schema Add the shared contract the intelligence layer builds on: a Finding SQLModel table, the witness.analysis package (analyzer protocol in base.py plus an idempotent runner), a canonical versioned capture schema (witness/schema.py, SCHEMA_VERSION 1.0), findings API endpoints, the analyze/bench/report CLI commands, and the viewer Finding types/api.
…come) - Prompt-injection detection: flags the conjunction of injection-shaped DOM content and a deviating post-injection action. - Trajectory health: loop/thrash, task drift, wasted steps, backtracking, plus a single trace-level health score. - Secret/PII exfiltration: masked-evidence detection of credentials typed into fields and DOM secrets carried into navigations or form posts. - Outcome/risk prediction: heuristic score now, with an optional scikit-learn training harness that mines labels from captured runs.
When WITNESS_OTLP_ENDPOINT is set, additionally export spans to an OTLP collector (Jaeger/Grafana/Honeycomb) while keeping local-first as the default. Normalize spans to gen_ai.* GenAI conventions so exported data is standards-compliant.
Refactor the capture path into a CaptureAdapter registry (browser_use adapter plus a scaffolded Playwright adapter); witness.instrument stays backward compatible. Best-effort, fully guarded CDP capture of network requests, console errors, and the accessibility tree per step.
Run a task N times and report success rate, step/cost/latency distributions, flakiness, and trajectory divergence. Add a small named WitnessBench set of reference browser tasks under examples/bench.
witness report writes a single-file HTML report (summary, timeline, findings, LLM calls) and a compact markdown summary; a reusable workflow posts the summary as a PR comment.
Surface findings grouped by kind with severity badges and evidence, trajectory-health and risk badges, per-step injection/exfil warning icons, trace-list filters and full-text search, a timeline scrubber, and a re-analyze button.
Add an intelligence-layer section to the README, reconcile the backlog with what now ships, add docs for analysis, schema/adapters, the bench harness, and OTLP export, and declare the optional [ml] and [otlp] extras.
EricFinland
force-pushed
the
claude/all-tracks
branch
from
June 27, 2026 15:06
f1dbfe2 to
4f0d2a5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds out all five roadmap tracks: the intelligence layer, capture depth, schema-first adapters, a reliability harness, and viewer polish.
Track A: Intelligence layer
witness/analysis/injection.py): flags the conjunction of injection-shaped DOM content (instruction-override phrasing, hidden/offscreen text, suspicious links) and a deviating post-injection action. Weak patterns only escalate to high when paired with a strong signal and a navigation/submit, so benign FAQ copy does not produce false highs.trajectory.py): loops/thrash, task drift, wasted/dead-end steps, and backtracking, plus a single trace-level health score.exfil.py): masked-evidence detection of credentials typed into fields and DOM secrets carried into navigations or form posts. Raw secrets are never persisted.outcome.py): heuristic risk score now, with an optional scikit-learn training harness (training.py,[ml]extra) that mines success/failure labels from captured runs.Findingmodel, an idempotent analysis runner,witness analyze [--all],GET /api/traces/{id}/findings, andPOST /api/traces/{id}/analyze.Track B: Capture depth
WITNESS_OTLP_ENDPOINT(Jaeger/Grafana/Honeycomb), normalized to OpenTelemetry GenAI semantic conventions. Local-first remains the default.Track C: Schema-first, multi-framework
witness/schema.py, SCHEMA_VERSION 1.0) and aCaptureAdapterregistry (witness/adapters/): browser_use full, Playwright scaffolded.witness.instrumentstays backward compatible.Track D: Reliability / eval harness
witness benchruns a task N times and reports success rate, step/cost/latency distributions, flakiness, and trajectory divergence. WitnessBench reference tasks live inexamples/bench/.Track E: Distribution and polish
witness reportwrites a self-contained HTML report plus a markdown PR summary; a reusable GitHub Action posts the summary on PRs.Quality
An adversarial review pass caught and fixed three high-severity issues before commit: the injection false-positive escalation, a cookie-banner exfil false positive, and a per-step Playwright listener leak. Each fix shipped with regression tests.
Tests
187 passed, 0 failed. 50 files changed, roughly +8000/-340.
Notes
pip install usewitness[ml]).