test(e2e): capture real agent TUI sessions and replays

[konard]

Fixes #841.

Summary

• captures the real OpenCode, Claude Code, and Codex fullscreen TUIs in a PTY and asserts the unrolled, scrollback-aware text that users actually saw;
• saves settled frames, an asciicast v2 recording, a static SVG, and an animated SVG for every captured run;
• drives typed text, control keys, Enter, and an explicit terminal resize, covering the repaint failure from Formal AI: Find hive-mind-control center folder on my desktop #819;
• verifies Formal AI: Find hive-mind on my desktop #838's multiselect report flow through the TUI, including the generated issue body and executed report actions;
• adds a TUI mode to Formal AI: Find hive-mind-control center folder on my desktop + Что такое фуфломицин? + ФБС vs ФБО — highest-standard answers via simple verified steps #840's task ladder and runs a representative node in CI;
• uploads complete or partial replay bundles with if: always(), so a failing run retains its visual evidence;
• consumes published agent-commander 0.9.1 (and transitive command-stream 0.15.0); Formal AI contains no private PTY, VT parser, terminal emulator, or animation renderer.

The permanent issue #841 case study contains the reproduction commands, transcripts, recordings, test logs, and self-hosting evidence.

Upstream work and releases

Responsibility	JavaScript	Rust	Merged work
Generic PTY execution, resize, VT rendering, scrollback, settled/unrolled frames, asciicast, SVG, and timeout artifacts	command-stream 0.15.0	command-stream 0.13.1	#176, #177, #179
Agent-specific builders, startup interactions, semantic events, and replay bundles	agent-commander 0.9.1	agent-commander 0.2.7	#44, #45

Both JavaScript releases are published on npm, and both Rust releases are published on crates.io. The original Formal AI prototype was removed after the released packages were consumed.

Reproduction and regression coverage

Before this change, the E2E suite asserted JSON exchanges while the agent CLI used a pipe. It could not observe terminal repainting, scrolled-off history, or the result of a resize. The focused regression uses a two-row terminal, proves that scrolled-off states appear exactly once and in order in the unrolled transcript, and requires the child PTY to observe an explicit 24×6 → 40×10 resize before succeeding.

The live matrix passed locally:

Agent direct
OpenCode direct
Claude Code direct
Codex direct
OpenCode TUI + replay
Claude Code TUI + replay
Codex TUI + replay

Focused and scenario commands:

cd experiments/agent_cli_e2e/issue_819_tui
bun install --frozen-lockfile
bun test

ARTIFACT_DIR=/tmp/formal-ai-tui-artifacts/path-discovery \
  experiments/agent_cli_e2e/run_issue_819.sh

ARTIFACT_DIR=/tmp/formal-ai-tui-artifacts/report-flow \
  experiments/agent_cli_e2e/run_issue_819_report_tui.sh

MODE=tui ONLY=838.L1 REQUIRE_ALL_PASS=1 \
  experiments/issue_840_task_ladder/run_ladder.sh

The repository-wide Rust suite was also exercised: 2,721 runnable tests passed in aggregate (the generated census was regenerated after the initial snapshot mismatch and its exact test then passed), along with clippy for all targets/features, formatting, doctests, release build, workflow/shell syntax, closure and census audits, language parity/coverage, and the self-hosting evidence ratchet.

Visual result

• OpenCode animation
• Claude Code animation
• Codex animation
• #838 resulting issue body
• #840 TUI ladder result

CI verification

• CI/CD run 30140861364: all lint, coverage, secrets, Docker, Rust test, agent E2E, web E2E, and package checks passed at f934f2f.
• Desktop build run 30140861365: Linux, macOS, and Windows arm64/x64 packages plus the VSIX passed.
• Successful real-TUI replay artifact: 74 files, including all five replay formats for OpenCode, Claude Code, Codex, and the report flow. The downloaded bundle was parsed and scanned for known token patterns.

Failure-path evidence

CI run 30138096256 failed before the final corrections, yet its agent-cli-tui-replays artifact retained the partial Codex frames.json, recording.svg, session.cast, snapshot.svg, and transcript.txt. That proves the upload and partial-artifact preservation path works on the condition it exists for.

Acceptance

• OpenCode, Claude Code, and Codex launch against Formal AI in real PTYs.
• Unrolled transcripts retain distinct states once and in order, including scrollback.
• Every run produces replay artifacts, uploaded even on CI failure.
• Text, control-key, Enter, and resize input are driven explicitly.
• The Formal AI: Find hive-mind-control center folder on my desktop #819 resize/repaint regression passes.
• The Formal AI: Find hive-mind on my desktop #838 multiselect and resulting issue body are asserted at TUI level.
• The Formal AI: Find hive-mind-control center folder on my desktop + Что такое фуфломицин? + ФБС vs ФБО — highest-standard answers via simple verified steps #840 task ladder runs through a real TUI.
• Formal AI consumes published upstream packages and owns no local terminal stack.

[konard]

Working session summary

Implemented and finalized PR #851; it is ready for review with a clean merge state and all checks passing.

Key results:

• Released reusable PTY/TUI support through command-stream and agent-commander.
• Added real OpenCode, Claude Code, and Codex TUI capture with input, resize, scrollback, transcripts, asciicasts, and SVG replays.
• Covered regressions Formal AI: Find hive-mind-control center folder on my desktop #819, Formal AI: Find hive-mind on my desktop #838, and Formal AI: Find hive-mind-control center folder on my desktop + Что такое фуфломицин? + ФБС vs ФБО — highest-standard answers via simple verified steps #840.
• Removed Formal AI’s local terminal prototype and consumed the published packages.
• Added failure-path artifact preservation, self-hosting evidence, and a permanent case study.
• Validated the 74-file CI replay artifact.

Verification:

• CI/CD run: passed.
• Desktop matrix: passed.
• 2,721 runnable Rust tests passed locally, plus clippy, formatting, doctests, release build, language/closure/census audits, and real-client scenarios.
• Branch is clean, pushed, 16 commits ahead and zero behind origin/main.

---

This summary was automatically extracted from the AI working session output.

[konard]

🤖 Solution Draft Log

This log file contains the complete execution trace of the AI solution draft process.

💰 Cost estimation:

• Model: GPT-5.6 Sol
• Provider: OpenAI
• Public pricing estimate: $176.391390

📊 Context and tokens usage:

• 3.1M / 200K (1556%) input tokens, 344.2K / 128K (269%) output tokens

Total: (3.1M + 129.8M cached) input tokens, 344.2K output tokens, $176.391390 cost

🤖 Models used:

• Tool: OpenAI Codex
• Requested: gpt-5.6-sol
• Thinking level: xhigh (~31999 tokens)
• Model: GPT 5.6 Sol (gpt-5.6-sol)

📎 Log file uploaded as Repository (35669KB)

• View complete solution draft log

---

Now working session is ended, feel free to review and add any feedback on the solution draft.

[konard]

✅ Ready to merge

This pull request is now ready to be merged:

• All CI checks have passed
• No merge conflicts
• No pending changes

---

Monitored by hive-mind with --auto-restart-until-mergeable flag

[konard]

Reviewed the visual artifacts. The capture is genuine — session.cast is valid asciicast v2 carrying full truecolor ANSI, and the PTY/resize/scrollback work holds up. But the SVG renderer discards nearly all of it, so the published images do not resemble a real terminal. Details and fixes filed upstream: command-stream#180 and agent-commander#46.

No GIF was delivered. There is no GIF/APNG/WebM encoder anywhere in command-stream@0.15.0 or agent-commander@0.9.1; outputs are only frames.json, session.cast, snapshot.svg, recording.svg, transcript.txt. The description links SVGs but never states this, so a reader looking for an animation finds only links. Worth saying explicitly, and worth embedding the animation inline once it renders correctly.

Verified defects

Renderer is command-stream/src/terminal-artifacts.mjs:80-137 — 58 lines total.

1. Colour and styling are entirely lost. frames.json stores lines as plain strings (889/889 typed str, no cell attributes), so svgText() emits bare <text> under one inherited fill="#e5e7eb". No fg/bg, bold, italic, underline, dim or reverse survives. This — not fonts — is why it doesn't match macOS Terminal or Warp: the whole TUI is one grey.

2. Spaces collapse, so columns cannot align. No xml:space="preserve", no per-cell x, no textLength. Measured in Chromium via the SVG DOM:

source (19 chars)	rendered glyphs
Report Confirm	14 — five spaces dropped
XXXReportXXXConfirm	19 — identical but for spaces

A box table's top border ends at x=172.15 while its own cell row ends at x=155.29 — 16.86px short, so borders detach from content. That is the "buggy rendering" visible in the artifacts.

3. Timing is fabricated. duration = frameCount * 0.35 with uniform keyTimes gives every frame an identical 0.35s. In the report-flow capture, a real 8.61s span plays as 14.70s; real gaps span 0.038–0.955s, so the longest pause is compressed 2.7× and the shortest stretched 9×. The true value is already in frame.time and is ignored.

4. Aspect ratio. capture-client.mjs hardcodes cols: 120, rows: 12 → 1104×234px = 4.72:1. For 4:3, cols/rows ≈ 2.67; 80×30 hits it exactly.

5. Rounded corners are hardcoded rx="6" in svgShell with no option to disable.

6. Font is a name list only (ui-monospace, SFMono-Regular, Menlo, Consolas) — none exist on GitHub's Linux renderers, so metrics differ per viewer and compound defect 2.

On the SVG approach

SVG is the right call, but the technique should change. GitHub does animate SVG in markdown when the animation is CSS and the file is committed to the repo — verified byte-level:

$ curl -sIL .../termtosvg/develop/docs/examples/awesome_window_frame_powershell.svg
content-type: image/svg+xml
content-security-policy: default-src 'none'; style-src 'unsafe-inline'; sandbox

The response still contains @keyframes, steps(1,end) and textLength — no sanitizer runs on that path. style-src 'unsafe-inline' permits CSS animation; script-src is absent, so JS/WAAPI animation is blocked. This PR's SMIL <animate> approach is also O(frames × lines) — 90KB for 42 frames of 80×24 — where every maintained tool in this space uses CSS @keyframes with steps(1, end) over a film strip.

Suggestion

The upstream fixes are substantial and shouldn't block this PR's real contribution (PTY capture, resize, scrollback, unrolled transcripts, artifact preservation on failure). I'd suggest landing that, and treating the renderer as follow-up work tracked by the two issues above — with the PR description noting that the current SVGs are known-lossy rather than presenting them as faithful visual confirmation, since the stated purpose is visual confirmation of e2e runs and a monochrome mis-aligned render can't serve that yet.