Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ jobs:
version: '0.11.32'

- name: Install deps
run: pnpm install --frozen-lockfile
run: pnpm install --frozen-lockfile && pnpm peers check

- name: Check publish credential isolation
run: pnpm run check:publish-workflow
Expand Down Expand Up @@ -131,7 +131,7 @@ jobs:
cache-dependency-path: pnpm-lock.yaml

- name: Install package set
run: pnpm install --frozen-lockfile
run: pnpm install --frozen-lockfile && pnpm peers check

- name: Build current agent-runtime
run: pnpm run build
Expand Down Expand Up @@ -161,7 +161,7 @@ jobs:
python-version: '3.12'

- name: Install package set
run: pnpm install --frozen-lockfile
run: pnpm install --frozen-lockfile && pnpm peers check

- name: Verify packed Runtime with official GEPA and SkillOpt
run: pnpm run verify:official-optimizers
4 changes: 2 additions & 2 deletions .github/workflows/publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,7 @@ jobs:
exit 1

- name: Install deps
run: pnpm install --frozen-lockfile
run: pnpm install --frozen-lockfile && pnpm peers check

- name: Lint (biome)
run: pnpm run lint
Expand Down Expand Up @@ -344,7 +344,7 @@ jobs:
cache-dependency-path: pnpm-lock.yaml

- name: Install deps
run: pnpm install --frozen-lockfile
run: pnpm install --frozen-lockfile && pnpm peers check

- name: Wait for published agent-bench dependencies
timeout-minutes: 25
Expand Down
14 changes: 14 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,19 @@
# Changelog

## 0.209.0

Budgets accept caller-named resources with explicit units, atomic reservations, measured refunds, and durable usage accounting.
Choose integer units such as GPU milliseconds, storage bytes, or API requests to preserve exact conservation across workers and restarts.
Missing, conflicting, or unknown enforced measurements close further admission while retaining the available evidence.
Streams, nested execution, cancellation, and replay share the same resource accounting contract.
Router transports can report the same receipts through `usage.resources`; retried calls retain incomplete measurement explicitly.
The package cohort uses Eval 0.180.0 and Knowledge 15.0.3.
Eval supervisor reports preserve individual resource receipts and their sources.

## 0.208.1

Retained and interactive runs use session identities accepted by Sandbox storage.

## 0.208.0

Provider executors select caller-declared child placements, preserve exact profiles, and bind configuration identity across retained recovery.
Expand Down
26 changes: 14 additions & 12 deletions api-surface.json
Original file line number Diff line number Diff line change
Expand Up @@ -777,10 +777,10 @@
"BridgeHarnessStore": "type ca55cbc26b63",
"BridgeModelCredential": "type 82ad4d863f5c",
"BridgeSeam": "type 3ee6c139b2d1",
"Budget": "type 198591da0ea4",
"BudgetPool": "type 9cb3e3daa900",
"Budget": "type d3e20424e44d",
"BudgetPool": "type 32b97a2266d1",
"BudgetPoolRestore": "type 09960d15dfe8",
"BudgetReadout": "type 626458ab8582",
"BudgetReadout": "type 0e2ea9923511",
"BusEvent": "type 736dd037425e",
"BusRecord": "type fccfdf943e88",
"BusStats": "type 10ba4be00459",
Expand Down Expand Up @@ -1105,14 +1105,16 @@
"RepairStop": "type dfe122fcef11",
"ReproductionCheck": "type d866d16ccd22",
"ReservationRejection": "type 95613aa46396",
"ReservationTicket": "type e7df35bc9166",
"ReservationTicket": "type 58a154a39d5e",
"ResolveDriveHarness": "type 14e24449c8f2",
"ResolveSandboxClientOptions": "type bc4eb1021d43",
"ResolveSupervisorTools": "type 1c7000abe533",
"ResolvedMcpServerLaunch": "type 38c483055998",
"ResolvedSupervisorProfile": "type a1c298f11bc0",
"ResolvedWorkerSpawnRetry": "type b7269d19dba0",
"ResourceLimit": "type c0a8ca7a5438",
"ResourceRequest": "type 20dbb23e8101",
"ResourceSpend": "type dfe0748e77c5",
"ResultBlobStore": "type 69e6e03b75ab",
"ResumedKeyState": "type 018905da27e4",
"ResumedWork": "type 57b7c709155a",
Expand Down Expand Up @@ -1173,11 +1175,11 @@
"SandboxSteeringOptions": "type 526a58a22f51",
"SandboxToolPartState": "type 26cdd7a50965",
"SandboxUsageLedger": "type e2058a39dd8b",
"Scope": "type f0ce3d119b61",
"Scope": "type edc3d5f40e17",
"ScopeAnalyst": "type 1cd5ae8d0d15",
"ScopeAnalyzeInput": "type 98b6562915c6",
"ScopeArgs": "type e739f4591d93",
"ScopeProgressInput": "type cf86c0b0aa0d",
"ScopeProgressInput": "type 5add6a25b8f5",
"ScopeWidenGate": "type 6c09e47c6dd2",
"ScoreOf": "type 652f59845272",
"SelectionReceipt": "type c2a6434f17c9",
Expand Down Expand Up @@ -1205,8 +1207,8 @@
"SpawnRefusal": "type e0fea783ffe7",
"SpawnRefusalCause": "type a9399e96f43a",
"SpawnRejection": "type 0f740e25e598",
"Spend": "type 6e558e6744c8",
"SpendChannel": "type 978ffae337cd",
"Spend": "type d7119cca63e2",
"SpendChannel": "type 3ca973d6c931",
"SpendGap": "type d24924855c35",
"StartRetainedInteractiveRunOptions": "type 29aca25d53e3",
"StartRetainedRunInEnvironmentOptions": "type f52dea3534df",
Expand Down Expand Up @@ -1264,7 +1266,7 @@
"TerminalDecision": "type 1a822915f49f",
"TokenUsageProvenance": "type 4ce5b69b2e4d",
"ToolLoopCallContext": "type 1c845a95cad0",
"ToolLoopChat": "type 898c9ad10190",
"ToolLoopChat": "type d25f10a0b0e1",
"ToolLoopCompaction": "type 1848f98b6067",
"ToolLoopCompactionOptions": "type 39c5e1c62e09",
"ToolLoopMessageRecord": "type b757f8750838",
Expand All @@ -1283,7 +1285,7 @@
"UnconfirmedTeardown": "type 791f75c80508",
"UnknownMaterializationReason": "type 4da80f596ffe",
"UntrackedCopyStats": "type 3d60a38b88e2",
"UsageEvent": "type 326dc5abb5da",
"UsageEvent": "type 4b3e1f2098da",
"VERIFY_TAIL_CHARS": "value 95999f4bd438",
"ValidationCtx": "type 45da43f45d39",
"Validator": "type c67346b2cb69",
Expand Down Expand Up @@ -1317,7 +1319,7 @@
"WorkerInteractiveProviderSource": "type b738c5faa346",
"WorkerInteractiveSession": "type 5090ed4fc91d",
"WorkerInteractiveUnavailableReason": "type 2692319a7ec2",
"WorkerProgress": "type bfbefaec0759",
"WorkerProgress": "type ffc0e787cacc",
"WorkerResumeContext": "type f96033ae9376",
"WorkerSpawnContext": "type a0f0b5934f6b",
"WorkerSpawnRetryAttempt": "type ee327ef375b6",
Expand Down Expand Up @@ -1947,7 +1949,7 @@
"SuperviseTestOptions": "type 1fd4a3f892e4",
"SupervisorAgentTestDeps": "type f39aa3b16149",
"ToolLoopCallContext": "type 1c845a95cad0",
"ToolLoopChat": "type 898c9ad10190",
"ToolLoopChat": "type d25f10a0b0e1",
"driverAgent": "value fba667f488cd",
"loadAgentImprovementProposalFixture": "value 2f33e49d4be9",
"loadAgentProfileImprovementFixture": "value 3bf4543ae6f7",
Expand Down
6 changes: 6 additions & 0 deletions bench/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,11 @@
# Changelog

## 0.11.3

Uses Runtime 0.209.0, Eval 0.180.0, and Knowledge 15.0.3 together, including named resource accounting and source-attributed resource reports.
SWE-bench setup, prompts, and patch extraction use the writable session workspace instead of a root-level directory.
Official grading behavior is unchanged.

## 0.11.2

Supports SWE-bench 5.x by detecting removed cache and namespace evaluator flags while retaining the 4.x path.
Expand Down
2 changes: 1 addition & 1 deletion bench/package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@tangle-network/agent-bench",
"version": "0.11.2",
"version": "0.11.3",
"type": "module",
"description": "Benchmark adapters and execution for agent-runtime across coding, tool-use, RAG, memory, browser, and terminal tasks.",
"repository": {
Expand Down
49 changes: 49 additions & 0 deletions bench/src/benchmarks/swe-bench.test.mts
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
import assert from 'node:assert/strict'
import test from 'node:test'
import { execFileSync } from 'node:child_process'
import { mkdtempSync, mkdirSync, writeFileSync, rmSync, existsSync } from 'node:fs'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { createSweBenchAdapter, scoreSweReport, sweEvaluationArgv } from './swe-bench'

const taskId = 'django__django-12345'
Expand Down Expand Up @@ -59,3 +63,48 @@ test('SWE evaluation command preserves the requested instance image', () => {
/invalid cacheLevel/,
)
})

test('SWE setup and extraction stay in the session workspace and exclude test edits', () => {
const root = mkdtempSync(join(tmpdir(), 'swe-workspace-'))
try {
const origin = join(root, 'origin')
const workspace = join(root, 'session')
mkdirSync(origin)
mkdirSync(workspace)
const git = (args: string[], input?: string) => execFileSync('git', args, { cwd: origin, input, encoding: 'utf8' }).trim()
git(['init', '--quiet'])
const blob = git(['hash-object', '-w', '--stdin'], 'before\n')
const tree = git(['mktree'], `100644 blob ${blob}\tsource.py\n`)
// Construct fixture history without changing the developer's Git identity or configuration.
const base = git(['hash-object', '-t', 'commit', '-w', '--stdin'],
`tree ${tree}\nauthor Fixture <fixture@example.invalid> 1 +0000\ncommitter Fixture <fixture@example.invalid> 1 +0000\n\nfixture\n`)
git(['update-ref', 'HEAD', base])
const task = { id: taskId, prompt: 'fix', metadata: { repo: 'fixture/repo', base_commit: base } }
const adapter = createSweBenchAdapter()
const setup = adapter.boxSetup!(task)
const extract = adapter.boxExtract!(task)
assert.match(setup.command, /^rm -rf '\.\//)
assert.equal(setup.cwd, undefined)
assert.equal(extract.cwd, undefined)
const env = {
...process.env,
GIT_CONFIG_COUNT: '2',
GIT_CONFIG_KEY_0: `url.file://${origin}.insteadOf`,
GIT_CONFIG_VALUE_0: 'https://github.com/fixture/repo',
GIT_CONFIG_KEY_1: 'protocol.file.allow',
GIT_CONFIG_VALUE_1: 'always',
}
execFileSync('sh', ['-c', setup.command], { cwd: workspace, env })
const repo = join(workspace, 'swe-bench-repo')
assert.ok(existsSync(join(repo, '.git')))
writeFileSync(join(repo, 'source.py'), 'after\n')
mkdirSync(join(repo, 'tests'))
writeFileSync(join(repo, 'tests', 'test_fix.py'), 'hidden-test-edit\n')
const patch = execFileSync('sh', ['-c', extract.command], { cwd: workspace, env, encoding: 'utf8' })
assert.match(patch, /diff --git a\/source.py b\/source.py/)
assert.match(patch, /\+after/)
assert.doesNotMatch(patch, /hidden-test-edit|test_fix/)
} finally {
rmSync(root, { recursive: true, force: true })
}
})
13 changes: 4 additions & 9 deletions bench/src/benchmarks/swe-bench.ts
Original file line number Diff line number Diff line change
Expand Up @@ -25,13 +25,8 @@ import {
} from './_harness'
import type { BenchmarkAdapter, BenchScore, BenchTask, LoadOptions } from './types'

/**
* Fixed in-box path the agent clones the instance repo into. It is the SINGLE
* source of truth shared by the prompt template (which tells the agent to clone
* here) and `boxExtract` (which runs `git diff` here after the shot) — so the
* harness always knows exactly where the agent's edits live, for any instance.
*/
const SWE_REPO_DIR = '/work'
/** Root-level directories are not writable in every sandbox; use the session workspace. */
const SWE_REPO_DIR = './swe-bench-repo'

/**
* The SWE deliverable's FALLBACK parser, from the agent's event STREAM.
Expand Down Expand Up @@ -286,10 +281,10 @@ print(json.dumps(out))
prompt: [
`Repository: ${r.repo} @ ${r.base_commit}`,
'',
`The repository is ALREADY cloned at ${SWE_REPO_DIR}, checked out at commit ${r.base_commit}. Work there directly (\`cd ${SWE_REPO_DIR}\`); do not re-clone.`,
`The repository is ALREADY cloned at ${SWE_REPO_DIR} relative to your initial session workspace, checked out at commit ${r.base_commit}. Work there directly (\`cd ${SWE_REPO_DIR}\`); do not re-clone.`,
'',
'Resolve this issue by editing the repository SOURCE so the failing tests pass without breaking the passing ones. Do NOT edit test files — the evaluation runs hidden tests on a fresh checkout, so editing tests does not count. Keep the change minimal and confined to the cloned repo.',
'Work iteratively: reproduce the issue, implement the fix in the source, and re-run the relevant tests until they pass. You do NOT need to print the diff — the harness reads your committed edits directly from the repo.',
'Work iteratively: reproduce the issue, implement the fix in the source, and re-run the relevant tests until they pass. You do NOT need to print the diff — the harness reads your source edits directly from the repo, including uncommitted changes.',
'',
'--- Issue ---',
String(r.problem_statement ?? ''),
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
# Live retained terminal evidence

Two bounded Sandbox runs exercised the maintained Runtime retained interactive path on 2026-09-09.
Neither run sent a model prompt.
Both used Codex with no initial prompt and terminal input `/help`.
These observations establish partial Sandbox conformance for issue [#773](https://github.com/tangle-network/agent-runtime/issues/773).

| Observation | First run | Second run |
|---|---|---|
| Initial output frames | 168 | 192 |
| Replay frames | 168 | 192 |
| Identical sequence and SHA-256 pairs | 168/168 | 192/192 |
| Contiguous sequences | 1–168 | 1–192 |
| Process running after detach | Yes | Yes |
| Terminal close acknowledgement | `unknown` | Not called |
| Interactive stop acknowledgement | Not called | `accepted`, effect `stopped` |
| Status after interactive stop | Not checked | `exited` |
| Sandbox deleted and absence verified | Yes | Yes |
| Model prompts | 0 | 0 |
| Process exit code | 0 | 0 |

Both runs returned an exact preparation receipt with matching authored and effective profile digests.
Both resized the terminal from 80×24 to 100×30 and then 120×40.
Both reconstructed the retained handle from its exact reference and acquired fresh control before replay.
The second run stopped through that reconstructed handle.
Terminal close alone did not establish native-process termination in the first run.

## Scope and limits

Runtime source revision: `2f13fc7b`.
Packages: Sandbox `0.38.2`, provider-tangle `1.1.6`, and agent-interface `2.6.0`.
Each run requested 1 CPU, 1024 MB RAM, 1 GB disk, blocked egress, and a 180-second maximum lifetime.
Each requested a 60-second idle timeout and used a 180-second client abort timer.
These resource values are requests, not independent measurements of platform enforcement.
No model token usage or billing measurement was collected; no model prompt API was invoked.

This proof does not test worker native-child binding, a live Braid coordinator restart, or environment expiry semantics.
Local CLI Bridge conformance remains blocked by [cli-bridge#183](https://github.com/drewstone/cli-bridge/issues/183).
The inspected Bridge revision `30038a6` lacked the PTY and duplex attach contract.
These records do not satisfy every #773 acceptance condition and do not justify closing that issue.

## Artifacts and reproduction

[close-result.json](./close-result.json) and [stop-result.json](./stop-result.json) preserve public identities, preparation digests, and output frame digests.
Terminal bytes, credentials, account email, and transport URLs are excluded.
The command recorded in each result names its original scratch script.

[close-proof.mts](./close-proof.mts) and [stop-proof.mts](./stop-proof.mts) preserve the executed logic with portable source locations and sanitized output.
The preserved scripts remove account email and error messages from output and restrict admissions to public identifiers.
They were syntax-checked after these recording changes; they were not rerun against another sandbox.

Copy the chosen script into a scratch directory and install the pinned packages there:

```sh
npm install --save-exact @tangle-network/sandbox@0.38.2 @tangle-network/agent-provider-tangle@1.1.6 @tangle-network/agent-interface@2.6.0
```

Set `RUNTIME_SOURCE` to the Runtime checkout and `SANDBOX_CLI_SOURCE` to the maintained Sandbox CLI `src` directory.
Set `EXPECTED_SANDBOX_EMAIL` to the authorized account email.
The scripts read existing Sandbox credentials through the maintained CLI configuration and verify account ownership through its read-only account API.
They require the production Sandbox origin and never print credentials.
Run `tsx stop-proof.mts` or `tsx close-proof.mts` from the scratch directory.
Each script creates a sandbox and deletes it in `finally`, then checks that it is absent.
Loading