Skip to content

Fix high CPU/RAM load handling and EPIPE errors#1322

Open
konard wants to merge 4 commits into
mainfrom
issue-1317-00f03cde548b
Open

Fix high CPU/RAM load handling and EPIPE errors#1322
konard wants to merge 4 commits into
mainfrom
issue-1317-00f03cde548b

Conversation

@konard

@konard konard commented Feb 17, 2026

Copy link
Copy Markdown
Contributor

Summary

This PR addresses issues discovered during investigation of #1317 ("Something strange happening on high CPU/RAM load").

Root Cause Analysis

From the log file analysis:

System Resources at Failure Time:

  • Memory: MemFree: 117072 kB (~114 MB free - CRITICALLY LOW!)
  • Load Average: 332.51 341.45 350.83 (EXTREMELY HIGH - system completely overloaded)

Compared to Normal Operation (earlier in the same session):

  • Memory: MemFree: 5996068 kB (~5.7 GB free)
  • Load Average: 1.18 0.39 0.17

What Happened:

  1. The system was under extreme resource pressure during the auto-restart phase
  2. The Claude process was killed by the Linux OOM killer (exit code 137 = 128 + SIGKILL)
  3. Network operations timed out due to high system load
  4. gh-upload-log failed to get the GitHub user, creating an invalid URL (https://github.com//repo)
  5. EPIPE errors occurred when trying to write to pipes whose readers (killed processes) were gone

Why solve Failed but Manual gh-upload-log Succeeded

The difference was timing:

  • solve ran during the high-load period → network timeouts, OOM kills
  • Manual gh-upload-log ran later after system load subsided → success

Bug in gh-upload-log

Found a bug in gh-upload-log@0.6.0 where network failures during user lookup cause malformed URLs. This should be reported to https://github.com/link-foundation/gh-upload-log separately.

Changes Made

1. EPIPE Error Handling (src/exit-handler.lib.mjs)

  • Handle EPIPE errors specially with graceful shutdown
  • Add SIGPIPE signal handler
  • These errors occur when writing to closed pipes (e.g., child killed by OOM)

2. URL Validation (src/log-upload.lib.mjs)

  • Detect malformed URLs from gh-upload-log (like github.com//repo)
  • Flag potentially invalid URLs for debugging

3. Resource Checking Before Auto-restart (src/solve.watch.lib.mjs)

  • Check RAM availability (512MB minimum) before starting auto-restart
  • Skip iteration if resources are too low to prevent OOM kills
  • Log resource state when skipping
  • Wait 30 seconds before next check to allow system recovery

Test Plan

  • Lint passes for all changed files
  • Verify EPIPE handling works with: node -e "process.stdout.write('test'); process.exit(0)" | head -c 1
  • Test auto-restart with low memory simulation
  • Verify URL validation catches malformed URLs

Fixes #1317


This PR was created with assistance from the AI issue solver

Adding CLAUDE.md with task information for AI processing.
This file will be removed when the task is complete.

Issue: #1317
@konard konard self-assigned this Feb 17, 2026
konard and others added 2 commits February 17, 2026 19:52
This commit addresses issues discovered during investigation of #1317:

1. Add EPIPE error handling in exit-handler.lib.mjs:
   - Properly handle EPIPE errors that occur when writing to closed pipes
   - Add SIGPIPE signal handler for graceful shutdown on pipe breaks
   - These errors typically happen when child processes are killed by
     the OOM killer under high memory pressure

2. Add URL validation in log-upload.lib.mjs:
   - Detect malformed URLs returned by gh-upload-log (like github.com//repo)
   - This catches cases where network issues cause gh-upload-log to fail
     to get the GitHub username, resulting in invalid repository URLs

3. Add resource checking before auto-restart in solve.watch.lib.mjs:
   - Check RAM availability (512MB minimum) before starting auto-restart
   - Skip iteration if resources are too low to prevent OOM kills
   - Log resource state (memory, load) when skipping
   - Wait 30 seconds before next check to allow system recovery

Root cause analysis:
- System had 117MB free RAM with load average of 332
- Claude process was killed by OOM killer (exit code 137)
- Network operations timed out due to high system load
- gh-upload-log failed to get GitHub user, creating invalid URLs
- EPIPE errors occurred when trying to write to killed processes

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@konard konard changed the title [WIP] Something strange happening on high CPU/RAM load Fix high CPU/RAM load handling and EPIPE errors Feb 17, 2026
@konard
konard marked this pull request as ready for review February 17, 2026 19:52
@konard

konard commented Feb 17, 2026

Copy link
Copy Markdown
Contributor Author

🤖 Solution Draft Log

This log file contains the complete execution trace of the AI solution draft process.

💰 Cost estimation:

  • Public pricing estimate: $7.139250
  • Calculated by Anthropic: $4.535280 USD
  • Difference: $-2.603970 (-36.47%)
    📎 Log file uploaded as Gist (781KB)
    🔗 View complete solution draft log

Now working session is ended, feel free to review and add any feedback on the solution draft.

@konard

konard commented Feb 17, 2026

Copy link
Copy Markdown
Contributor Author

✅ Ready to merge

This pull request is now ready to be merged:

  • All CI checks have passed
  • No merge conflicts
  • No pending changes

Monitored by hive-mind with --auto-restart-until-mergeable flag

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Something strange happening on high CPU/RAM load

1 participant