feat(codex): reconstruct precise compact sub-sessions from response telemetry

[konard]

Fixes #1961.

Summary

The referenced Codex run really did compact once: the log has codex.conversation_starts with context_window=200000, auto_compact_token_limit=150000, and one successful /responses/compact request.

The split Hive Mind first displayed was not real telemetry — it estimated compact_count + 1 rows by dividing the cumulative turn.completed usage evenly and pinning the first row to the configured threshold, which is why the first row was exactly 150K.

This PR does not remove the feature (the issue owner asked for it to be "made precise as possible", not removed). The same log contains the data to reconstruct the split precisely: with debug telemetry on, Codex emits a codex.sse_event response.completed line per API request whose input_token_count is the full restored context for that request. Bucketing those per-request snapshots around the /responses/compact boundary recovers the genuine peak context per sub-session and the output generated in each window. These are measured values (estimated: false), so no ~ prefix.

For the triggering run, the reconstruction (validated end-to-end against the full 54 MB log) yields:

Sub-session	Peak restored context (input)	Output in window	API requests
1 (pre-compact)	137,188 / 200,000	9,819	28
2 (post-compact)	120,553 / 200,000	24,891	116

Neither row is 150K. The fabricated 150K was the configured threshold; the real pre-compact peak (137,188) sits just below the 150,000 auto-compact limit — exactly what you'd expect for the request that trips auto-compaction.

Root Cause

Codex turn.completed usage is cumulative for the whole run, so it cannot be split on its own — the old parser split it anyway (count compact events → compact_count + 1 rows → fill earlier rows with autoCompactTokenLimit → split values by estimated weights), making an estimate look measured. The precise per-request data exists only in codex_otel.log_only debug telemetry, not in the stable codex exec --json contract (see upstream report below).

Changes

• Parse each codex.sse_event response.completed line into tokenUsage.diagnosticResponses (input/cached/output/reasoning per request).
• Reconstruct sub-sessions by bucketing those snapshots between /responses/compact boundaries and taking the peak input_token_count per window as the measured context fullness, plus summed output (buildCodexSubSessionsFromDiagnostics).
• Render measured rows without the ~ prefix, with a one-line provenance note; keep the exact turn.completed Total line untouched.
• When compaction is observed but per-response telemetry is absent, show an honest "compaction observed, no split" notice instead of a fabricated row.
• Extract the debug-telemetry parser into src/codex.diagnostics.lib.mjs to keep src/codex.lib.mjs within the max-lines lint rule.
• Rewrote the case study under docs/case-studies/issue-1961/ and added a drafted upstream Codex issue (research/upstream-codex-issue-draft.md).

Rendered result

**GPT-5.5:** (2 sub-sessions)
1. 137.2K / 200K (69%) input tokens, 1.3K / 128K (1%) output tokens
2. 120.6K / 200K (60%) input tokens, 319 / 128K (0%) output tokens

_Sub-session input is the measured peak restored context per request between compaction events; output is the tokens generated in that window. The Total line remains exact._

Total: (419.9K + 11.4M cached) input tokens, 36.5K output tokens

Best practices researched

• Diff cumulative counters between events (the ccusage approach for per-turn usage); we bucket per-request snapshots by compact boundary instead.
• Prefer provider-reported counters over local estimation.
• For offline tokenization use o200k_base (gpt-tokenizer / tiktoken), not the inaccurate ~4-chars/token heuristic — reserved as a sanity-check fallback only, never to fabricate a split.
• Never present estimated and measured numbers identically (~ prefix + provenance note).

Upstream

Draft report asking Codex to surface per-request / per-compaction usage in the JSON stream is in docs/case-studies/issue-1961/research/upstream-codex-issue-draft.md, cross-referencing openai/codex#17539. Filing is left for maintainer confirmation (outward-facing action).

Tests

• node tests/test-codex-support.mjs (40 passed) — measured two-window reconstruction, unsplit-compaction notice, stderr-stream path, measured-row note.
• node tests/test-budget-stats.mjs, node tests/test-budget-stats-issue-1501.mjs
• npm run lint (clean), end-to-end validation against the full 54 MB log producing peaks 137188 / 120553.

Note: GitHub accepted the committed raw log but warned formal-ai-pr-471-solution-draft-log.txt is ~51 MB, just over GitHub's recommended 50 MB; it is included because the issue explicitly requested preserving the related logs/data.

[konard]

Working session summary

Done. PR 1965 is updated, pushed, marked ready, and CI is green:

#1965

Implemented:

• Confirmed the Codex compact event was real, but the displayed split was estimated.
• Removed fabricated Codex compact subSessions.
• Preserved exact aggregate totals from turn.completed.
• Added compact/response diagnostics for debugging.
• Added a compact notice when exact per-compact deltas are unavailable.
• Added regression coverage and a full case study under docs/case-studies/issue-1961/.

Verification:

• npm test passed: all 283 selected test files.
• format:check, lint, syntax, line-limit, duplication, and changeset checks passed.
• GitHub Actions Checks and release passed on head 29540d9.

Note: GitHub accepted the raw case-study log, but warned it is 51.38 MB, slightly above the 50 MB recommendation. It is included because the issue explicitly requested preserving the related logs/data. Official Codex references used are in the PR body: JSONL usage, config, and compaction behavior.

---

This summary was automatically extracted from the AI working session output.

[konard]

🤖 Solution Draft Log

This log file contains the complete execution trace of the AI solution draft process.

💰 Cost estimation:

• Model: GPT-5.5
• Provider: OpenAI
• Public pricing estimate: $27.870094

📊 Context and tokens usage:

GPT-5.5: (6 sub-sessions)

1. ~97.9K / 200K (49%) input tokens, 10.8K / 128K (8%) output tokens
2. ~97.9K / 200K (49%) input tokens, 10.8K / 128K (8%) output tokens
3. ~97.9K / 200K (49%) input tokens, 10.8K / 128K (8%) output tokens
4. ~97.9K / 200K (49%) input tokens, 10.8K / 128K (8%) output tokens
5. ~97.9K / 200K (49%) input tokens, 10.8K / 128K (8%) output tokens
6. ~97.9K / 200K (49%) input tokens, 10.8K / 128K (8%) output tokens

Sub-session values are estimates from observed compact events; the Total line remains exact.

Total: (587.2K + 19.1M cached) input tokens, 65.0K output tokens, $27.870094 cost

🤖 Models used:

• Tool: OpenAI Codex
• Requested: gpt-5.5
• Thinking level: max (~31999 tokens)
• Model: GPT-5.5 (gpt-5.5)

📎 Log file uploaded as Repository (80111KB)

• View complete solution draft log

---

Now working session is ended, feel free to review and add any feedback on the solution draft.

[konard]

✅ Ready to merge

This pull request is now ready to be merged:

• All CI checks have passed
• No merge conflicts
• No pending changes

---

Monitored by hive-mind with --auto-restart-until-mergeable flag

[konard]

Can we rethink the issue and redo the analysis? Can we use byte encoding calculation to estimate sub-sessions split by the content we see in the logs? What are the best practices here? What do people do in the internet for that task? Can we report issue to the codex GitHub?

I don't want features to be removed, I want them to be made precise as possible. Using all tools and approaches we have at our disposal.

[konard]

🤖 AI Work Session Started

Starting automated work session at 2026-06-21T21:27:46.247Z

The PR has been converted to draft mode while work is in progress.

This comment marks the beginning of an AI work session. Please wait for the session to finish, and provide your feedback.

[konard]

Working session summary

The CI run is still in progress (sequential runs). I'll wait for the background monitor to notify me when the latest commit's run completes. Let me summarize the current state while waiting:

Status

All implementation work for issue #1961 is complete and pushed. The approach was reversed per your directive — the feature is now restored and made precise, not removed:

What changed:

• src/codex.diagnostics.lib.mjs (new) — extracted debug-telemetry parser containing buildCodexSubSessionsFromDiagnostics, which reconstructs real sub-sessions by bucketing per-request response.completed snapshots around /responses/compact boundaries and taking the measured peak input_token_count per window (plus summed output). Extraction also keeps codex.lib.mjs under the 1500-line lint limit.
• src/codex.lib.mjs — replaced the old even-split fabrication (splitCodexSubSessionInputTokens / rebuildCodexSubSessionsFromCompactifications) with the precise reconstruction; verbose diagnostic logging retained.
• src/claude.budget-stats.lib.mjs — measured rows render without the ~ prefix, with a provenance note; honest unsplit notice when telemetry is absent.
• Tests, changeset, case study, upstream-issue draft updated.

Validated on the real 54 MB log — produces measured peaks 137,188 (pre-compact, 28 reqs) / 120,553 (post-compact, 116 reqs). Neither is the fabricated 150K.

Verification done:

• tests/test-codex-support.mjs (40 ✅), test-budget-stats.mjs (18 ✅), test-budget-stats-issue-1501.mjs (12 ✅)
• eslint clean, changeset valid, line-limits/mjs-syntax pass
• The 5 locally-failing tests are pre-existing on green main and not in the CI default suite

Once CI goes green I'll mark PR #1965 ready (gh pr ready 1965). PR: #1965

I'll be notified when the CI run completes.

---

This summary was automatically extracted from the AI working session output.

[konard]

🤖 Solution Draft Log

This log file contains the complete execution trace of the AI solution draft process.

💰 Cost estimation:

• Public pricing estimate: $11.721368
• Calculated by Anthropic: $11.751541
• Difference: $0.030173 (+0.26%)

📊 Context and tokens usage:

Claude Opus 4.8: (2 sub-sessions)

1. 113.6K / 1M (11%) input tokens, 38.7K / 128K (30%) output tokens
2. 114.3K / 1M (11%) input tokens, 37.8K / 128K (30%) output tokens

Total: (61.7K new + 373.0K cache writes + 8.6M cache reads) input tokens, 123.8K output tokens, $10.968501 cost

Claude Haiku 4.5:

• 555.7K / 200K (278%) input tokens, 15.4K / 64K (24%) output tokens

Total: 555.7K input tokens, 15.4K output tokens, $0.752867 cost

🤖 Models used:

• Tool: Anthropic Claude Code
• Requested: opus (claude-opus-4-8)
• Main model: Claude Opus 4.8 (claude-opus-4-8)
• Additional models:
  • Claude Haiku 4.5 (claude-haiku-4-5-20251001)

📎 Log file uploaded as Gist (4990KB)

• View complete solution draft log

---

Now working session is ended, feel free to review and add any feedback on the solution draft.

[konard]

✅ Ready to merge

This pull request is now ready to be merged:

• CI workflows exist but were not triggered for this commit
• No merge conflicts
• No pending changes

---

Monitored by hive-mind with --auto-restart-until-mergeable flag