v0.7.0 — ETL pipeline + handoff

v0.7.0 — proper ETL pipeline.

The dashboard's per-request aggregator passes are gone. Every cost / dashboard / compare / yield / optimize / messages-summary route now reads from indexed materialized marts; a filesystem watcher syncs marts within ~400 ms of any source-file write. End-to-end on a 247K-message store: dashboard cold-load went from 2.5 s → <50 ms warm.

Architecture: three layers, one watcher.

messages → usage_events → 5 marts (daily/session/project/provider_day/model_day)
                            ↑
                       filesystem watcher (200 ms debounce)

See docs/specs/etl-architecture.md for the design contract and docs/HANDOFF.md for the state-of-the-codebase walkthrough.

Numbers (real maintainer store)

• Backfilled 150,337 events in 226 s (idempotent re-runs ~29 s)
• Marts: daily 940, session 841, project 151, provider_day 146, model_day 184
• Watermarks all aligned to 150,337
• Dashboard route latencies (warm, dev box): cost-by-provider 1 ms · compare 2 ms · yield 1 ms · projects 6 ms · dashboard-data 7 ms · cost-data 12 ms · optimize 100 ms

What's new

• 11 PRs across 4 waves shipped: foundation (#72), 4 default normalizers (#73), 5 mart builders (#74), filesystem watcher (#75), hot-path route migration (#76), 12 beta normalizers (#79), analytical-route migration (#81), real backfill + writer hook (#80), /api/etl/status + CLI (#78), UI status badge + backfill button (#77), real-data e2e + perf regression (#82)
• 16/16 codeburn-catalog providers now have Normalizer subclasses
• New CLI: stackunderflow etl status, stackunderflow etl backfill [--force], stackunderflow start --no-watcher
• New API: GET /api/etl/status
• New UI: status badge in dashboard header, "Backfill now" on /settings

Tests

1598 passing, 2 skipped, 11 deselected (slow suite — pytest -m slow)

Migration

Schema bumps 5 → 6 via v006_etl_layer.sql (additive — existing tables and routes keep working). First-run cost: a one-time backfill (~3 minutes per 100K messages) populates the marts; subsequent runs are watcher-driven and incremental.

Handoff

docs/HANDOFF.md walks an incoming agent through the architecture, recent history, gotchas, real-data state, and what's left.