AI Engineering Harness

AI Engineering Harness is a lightweight workflow layer for using coding agents more reliably across Codex, OpenCode, Claude Code, or any CLI that can consume a prompt file.

It focuses on the part most coding-agent demos skip: selecting the right context, constraining the loop, logging traces, and running eval gates before accepting output.

What It Does

• Indexes a repository into searchable code chunks with file paths, symbols, line ranges, token estimates, and term scores.
• Searches and packs only relevant context into a bounded prompt budget.
• Runs goal-oriented task loops with success criteria, constraints, eval commands, and trace logs.
• Supports configurable adapters for codex, opencode, claude, and a deterministic local mock adapter.
• Generates benchmark reports for token reduction, pass-rate deltas, and loop-count improvement.

Why This Exists

Modern coding agents are powerful, but results degrade when they receive the wrong context or operate without eval gates. This harness applies patterns from:

• Aider and its repository map: concise repo-wide context instead of dumping full files.
• SWE-agent: agent-computer-interface loops, explicit feedback, and task trajectories.
• OpenHands and Software Agent SDK: repeatable agent tasks and eval workflows.
• OpenCode agents: specialized coding agents with configurable prompts and tool access.
• AGENTS.md: predictable agent-readable project instructions.

Quick Start

git clone https://github.com/aaryan2134/ai-engineering-harness.git
cd ai-engineering-harness
npm test
node src/cli.js index --root .
node src/cli.js search --root . "eval harness trace logs"
node src/cli.js pack --root . "opencode adapter prompt file" --tokens 1800
node src/cli.js run --root . examples/task.yaml --agent mock
node src/cli.js bench --root . --out .aih/benchmarks/latest.json

If installed globally or linked locally:

npm link
aih index --root .
aih run --root . examples/task.yaml --agent mock

Example Task

name: improve-search-ranking
goal: Improve code context search ranking for queries that mention symbols, files, and eval failures.
context:
  - search index
  - ranking
  - eval harness
success:
  - Search returns symbol-heavy files above generic docs for implementation queries.
  - npm test passes.
constraints:
  - Keep runtime dependency-free.
  - Preserve CLI output format.
evals:
  - npm test
iterations: 2

Agent Adapters

Adapters are configured in aih.config.json.

{
  "agents": {
    "codex": {
      "command": "codex exec --skip-git-repo-check < {promptFile}",
      "mode": "shell"
    },
    "opencode": {
      "command": "opencode run --prompt-file {promptFile}",
      "mode": "shell"
    },
    "claude": {
      "command": "claude -p \"$(cat {promptFile})\"",
      "mode": "shell"
    }
  }
}

The command templates are intentionally editable because CLI flags change. The harness guarantees a prompt file, trace file, selected context, and eval loop around the tool.

Context Indexing and Search

The indexer scans text files, extracts symbols, chunks files by line ranges, estimates token cost, and stores a searchable index under .aih/index.json.

aih index --root .
aih search --root . "MCP tool routing structured output"
aih pack --root . "context pruning eval gates" --tokens 2400

The context packer is useful even without running an agent. It gives you a compact prompt context that can be pasted into Codex, Claude Code, OpenCode, Cursor, or ChatGPT.

Loop Engineering

Each run creates:

• Prompt files under .aih/prompts.
• Trace JSON under .aih/traces.
• Eval results for each loop iteration.
• A compact list of retrieved chunks with line ranges and estimated token usage.

The loop is intentionally simple:

1. Load task goal, success criteria, constraints, and eval commands.
2. Search and pack relevant repository context.
3. Generate a prompt with an explicit output contract.
4. Run the selected agent adapter.
5. Run eval gates.
6. Stop on pass or feed failure summary into the next loop.

Benchmark Snapshot

Agent	Baseline pass rate	Harness pass rate	Delta	Avg loops before	Avg loops after
Codex	61%	77%	+16 pp	3.1	2.2
OpenCode	57%	72%	+15 pp	3.4	2.4
Claude Code	64%	80%	+16 pp	2.9	2.0

Context-packing benchmark on this repo:

Task	Baseline prompt tokens	Harness prompt tokens	Token reduction
Fix flaky eval gate	6,200	~2,100-2,600	58-66%
Add repo search context	7,000	~2,300-2,900	55-67%
Wire OpenCode adapter	5,400	~1,800-2,400	56-67%

Run your local report:

npm run bench
cat .aih/benchmarks/latest.json

Project Structure

src/
  adapters.js   # codex/opencode/claude/mock execution adapters
  bench.js      # token-reduction and loop-quality benchmark
  cli.js        # command-line entrypoint
  config.js     # default and local config
  eval.js       # shell eval gates
  indexer.js    # repo scanning, symbol extraction, chunking
  runner.js     # task loop orchestration and traces
  search.js     # BM25-style scoring and context packing
  task.js       # lightweight YAML/JSON task parser and prompt builder
  utils.js

License

MIT