247-agents: Long-Running Horizon Agent Test Harness

A research-backed test harness for validating autonomous agents that run indefinitely on servers, managed by PM2.

Research Foundation

This project is built on findings from:

• Anthropic (2026): Decoupled agent architecture — Session + Orchestration Loop + Sandbox ¹
• OpenHands (2026): Event-sourced state model for reproducibility and fault recovery ²
• OpenAI / MSR (2026): Harness engineering as the primary driver of agent robustness ³
• Shunyu Yao (OpenAI): The shift from model capability to agent infrastructure ⁴
• Fault Tolerance Literature: Checkpoint/restart patterns for long-running distributed systems ⁵[^7]

See docs/research/ for full BFS, DFS, and bidirectional analysis.

Architecture

┌─────────────────────────────────────────────┐
│  PM2 Process Manager (fork mode)            │
│  - Auto-restart with circuit breaker        │
│  - Memory-limit restart (leak mitigation)   │
│  - Graceful shutdown (SIGTERM → checkpoint) │
├─────────────────────────────────────────────┤
│  Agents (stateful, long-running loops)      │
│  - search-agent (×1)                        │
│  - extractor-agent (×2)                     │
│  - research-coordinator (×1)                │
│  - knowledge-agent (×1) — file scanner      │
│  - agent-gateway (×1) — HTTP + WebSocket    │
├─────────────────────────────────────────────┤
│  Checkpoint Store (filesystem)              │
│  - Survives process restarts                │
│  - Enables state rehydration                │
├─────────────────────────────────────────────┤
│  Structured JSON Logs                       │
│  - Machine-parseable for test validation    │
└─────────────────────────────────────────────┘

Quick Start

# Start all agents
npm run start

# Monitor in real-time
npm run monit

# View logs
npm run logs

# Run 30-second uptime test
npm run test:agents

# Run 5-minute endurance test
npm run test:agents:long

# Stop all agents
npm run stop

Agent Design Patterns

Each agent implements:

1. Structured JSON logging via Logger utility
2. Checkpointing on SIGTERM and periodic intervals
3. State rehydration on startup from .checkpoints/
4. Heartbeat emission for health monitoring
5. Graceful shutdown with queue draining

Test Harness

The test/uptime-test.js script validates:

• All expected processes are online
• No process exceeds restart threshold
• Memory usage stays within bounds
• Log files contain valid JSON
• Generates test-report-[timestamp].json

Directory Structure

├── docs/
│   ├── research/
│   │   ├── bfs/
│   │   ├── dfs/
│   │   └── bidirectional/
│   └── adrs/
├── src/
│   ├── utils/
│   │   ├── logger.js
│   │   └── checkpoint.js
│   └── agents/
│       ├── search-agent.js
│       ├── extractor-agent.js
│       ├── research-coordinator.js
│       ├── knowledge-agent.js
│       └── agent-gateway.js
│   └── gateway/
│       ├── agent-gateway.js
│       └── dashboard.html
├── test/
│   └── uptime-test.js
├── ecosystem.config.js
└── logs/

Knowledge Agent & RAG Chat

The knowledge-agent continuously scans ~/ for documents, chunks them, and generates OpenAI embeddings stored in SQLite (.knowledge/knowledge.db). The agent-gateway exposes:

• GET / — Dashboard with agent status, logs, and chat UI
• GET /api/knowledge/stats — Index statistics
• GET /api/knowledge/search?q= — Hybrid search (FTS5 + vector similarity)
• POST /api/chat — Non-streaming RAG chat
• WS /chatws — Streaming RAG chat with source citations

Content-Aware Compression

Before sending retrieved chunks to the LLM, the RAG pipeline applies content-aware compression inspired by Headroom. This reduces token usage by 30–70% depending on content type:

Content Type	Strategy	Typical Savings
JSON arrays	Deduplicate + sample rows	40–60%
Source code	Preserve signatures, compress bodies	30–50%
Logs	Keep errors/warnings, drop INFO noise	50–80%
Search results	Rank + keep top matches per file	30–50%
Git diffs	Trim context lines	40–60%
HTML	Strip tags/scripts/styles	60–80%
Plain text	Query-aware sentence extraction	20–40%

Compression is applied automatically with adaptive budgets: earlier (more relevant) chunks receive larger budgets. Metrics are logged to the gateway output:

[RAG] Compressed 8 chunks: 4433→2119 tokens (48% of original), strategies: [code, json, logs]

Configuration

Edit ecosystem.config.js to adjust:

• instances: Number of agent replicas
• max_memory_restart: Memory limit before restart
• restart_delay / max_restarts: Circuit breaker tuning
• SCAN_ROOT: Directory to scan for documents
• SCAN_INTERVAL_MS: How often to re-scan
• Environment variables per agent

References

Footnotes

1. Anthropic (2026). "Managed Agents Add Memory." ↩

2. OpenHands Team (2026). "OpenHands Software Agent SDK." arXiv:2511.03690. ↩

3. OpenAI / MSR (2026). "Natural-Language Agent Harnesses." arXiv:2603.25723. ↩

4. Yao, S. (2025). Language Agents: From Next-Token Prediction to Digital Automation. PhD Thesis. ↩

5. Alamleh & El Emam (2024). "A Survey of Fault Tolerance Techniques in Distributed Systems." ↩