Detect orphaned run_in_background watchers (Issue #1739)#1740
Open
konard wants to merge 3 commits into
Open
Conversation
Adding .gitkeep for PR creation (default mode). This file will be removed when the task is complete. Issue: #1739
…unbounded poll loops (#1739) Issue #1739 reported a Claude session that emitted `until [ "$(gh run view ...)" = "completed" ]; do sleep 20; done` with `run_in_background: true`, then ended its turn while the watcher kept polling. The harness had no awareness of live background tasks, so it accepted `stop_reason: end_turn` / `terminal_reason: completed` as a clean session — the watcher was orphaned and the run silently hung for 2 h 54 min before manual CTRL+C. Changes: - src/claude.background-tasks.lib.mjs: BackgroundTaskTracker that maps `system.task_started` / `system.task_completed` events; `looksLikePassiveWaitText` heuristic; `classifyStuckWatch` combines them; thin `createStuckWatchSessionState`/`observeSessionEvent`/`reportStuckWatchAtResult` helpers keep the call site in claude.lib.mjs to two added lines. - src/claude.lib.mjs: wire the tracker into the JSONL parser; at the result event, log `🔎 Background tasks: ...` (clean or live) and `🛑 STUCK-WATCH DETECTED` when a passive final message coincides with surviving tasks. - src/claude.prompts.lib.mjs: forbid hand-rolled `until/while ... do sleep N; done` watchers in the system prompt; recommend `gh run watch <run-id> --exit-status` and `timeout T ...` instead. - tests/test-stuck-watch-detection-1739.mjs: replays the exact issue-1739 event sequence; covers passive-text heuristic edge cases and `formatForLog` output. - docs/case-studies/issue-1739/: full case study (README, timeline, requirements, root causes, solution plan, external research) plus the original 4.4 MB forensic log preserved via a narrow `.gitignore` exception. - .changeset/: patch release note. The classifier is diagnostic-only in this PR — no behavior change beyond log lines. A follow-up can gate `--abort-on-stuck-watch` on the same signal.
konard
changed the title
konard
marked this pull request as ready for review
May 1, 2026 14:53
Contributor
Author
Working session summaryCI is queued for the new commit. Done with the immediate work — let me confirm the PR contents are visible end-to-end. What landed:
Deferred to follow-ups (documented in case study): cost ceilings on auto-restart (S-3), CI is queued on commit Want me to /schedule an agent in 1 week to file the upstream This summary was automatically extracted from the AI working session output. |
Contributor
Author
🤖 Solution Draft LogThis log file contains the complete execution trace of the AI solution draft process. 💰 Cost: $8.124249📊 Context and tokens usage:Claude Opus 4.7: (3 sub-sessions)
Total: (3.8K new + 287.4K cache writes + 9.1M cache reads) input tokens, 69.6K output tokens, $8.124249 cost 🤖 Models used:
📎 Log file uploaded as Gist (4528KB)Now working session is ended, feel free to review and add any feedback on the solution draft. |
Contributor
Author
✅ Ready to mergeThis pull request is now ready to be merged:
Monitored by hive-mind with --auto-restart-until-mergeable flag |
This reverts commit f12b751.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1739.
Summary
Issue #1739 captured a Claude session that appeared "complete" while a background watcher kept polling forever. The model emitted
with
run_in_background: true, then ended its turn (stop_reason: end_turn,terminal_reason: completed). The hive-mind harness had no awareness of live background tasks, so it accepted the result as a clean session — the watcher was orphaned and the run hung silently for 2 h 54 min until manual CTRL+C. Total wasted spend: $12.87 across 242 turns. Seedocs/case-studies/issue-1739/README.mdfor the full forensic write-up.This PR addresses the root cause on three fronts:
BackgroundTaskTracker(src/claude.background-tasks.lib.mjs) that mapssystem.task_started/system.task_completed. At the result event the harness now logs how many background tasks survived end-of-turn (🔎 Background tasks: clean (0 alive at result event)for healthy sessions) and emits🛑 STUCK-WATCH DETECTEDwhen a passive final assistant message ("I'll wait for…", "Once it completes…") coincides with surviving tasks.src/claude.prompts.lib.mjs) now explicitly forbids hand-rolleduntil ... do sleep N; doneandwhile ... do sleep N; donewatchers and steers the model towardgh run watch <run-id> --exit-statusandtimeout T ...as finite-timeout alternatives.docs/case-studies/issue-1739/(timeline, requirements, root causes, solution plan, external research, draft upstream issue body foranthropics/claude-code) along with the original 4.4 MB forensic log, preserved via a narrow.gitignoreexception.The classifier is diagnostic-only in this PR — no behavior change beyond log lines. A follow-up can gate
--abort-on-stuck-watchon the same signal.Files
src/claude.background-tasks.lib.mjsBackgroundTaskTracker,looksLikePassiveWaitText,classifyStuckWatch, plus thincreateStuckWatchSessionState/observeSessionEvent/reportStuckWatchAtResulthelperssrc/claude.lib.mjsobserveSessionEvent(...)per event andreportStuckWatchAtResult(...)at the result eventsrc/claude.prompts.lib.mjstests/test-stuck-watch-detection-1739.mjsformatForLogoutputdocs/case-studies/issue-1739/logs/original.log(forensic artifact).changeset/stuck-watch-detection-1739.mdTest plan
node tests/test-stuck-watch-detection-1739.mjs— passes; replays the exact issue-1739 task IDs / passive text and assertsclassifyStuckWatchreturns{ stuck: true }npm run lint— clean across the whole repo (claude.lib.mjsstays under the 1500-line cap by extracting helpers into the new lib)npx prettier --checkon all touched files — cleannode scripts/run-tests.mjs --suite default— 47 passed, 3 pre-existing failures intest-issue-1706-sub-session-size.mjs(unrelated; reproduce onorigin/main)🔎 Background tasks: cleanappears in--verboselogs of any normal session, and🛑 STUCK-WATCH DETECTEDappears when the antipattern is provokedanthropics/claude-code(draft ready indocs/case-studies/issue-1739/external-research.md)Mapping to issue requirements
The issue body lists 12 explicit asks. Status table is in
docs/case-studies/issue-1739/requirements.md. In short:root-causes.md)docs/case-studies/issue-1739/external-research.md)🔎 Background tasks: …log line)