feat(ctl): add Terminal-Bench chat runner - #1175
Conversation
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
|
|
||
| MOLTIS_CTL_BIN="${MOLTIS_CTL_BIN:-moltis-ctl}" | ||
| MOLTIS_GATEWAY_URL="${MOLTIS_GATEWAY_URL:-http://host.docker.internal:13131}" | ||
| MOLTIS_SESSION_KEY="${MOLTIS_SESSION_KEY:-terminal-bench:${HARBOR_TASK_ID:-task}}" |
There was a problem hiding this comment.
When two benchmark invocations omit both MOLTIS_SESSION_KEY and HARBOR_TASK_ID, this fallback gives both terminal-bench:task; the gateway then reads and appends the same persisted conversation, leaking prior instructions and results into the later task's model context.
Knowledge Base Used: CLI and Ctl: moltis binary and moltis-ctl gateway control
| pub async fn send(client: &mut CtlClient, text: &str, session_key: &str) -> anyhow::Result<Value> { | ||
| client | ||
| .call( | ||
| "chat.send", | ||
| json!({ | ||
| "text": text, | ||
| "_session_key": session_key, | ||
| }), | ||
| ) | ||
| .await |
There was a problem hiding this comment.
When a benchmark's model and tool loop takes longer than 120 seconds, chat.send remains subject to CtlClient::call's fixed deadline, causing moltis-ctl to exit with a timeout before the documented wait-for-completion behavior finishes.
Knowledge Base Used: CLI and Ctl: moltis binary and moltis-ctl gateway control
| exec "$MOLTIS_CTL_BIN" \ | ||
| --gateway-url "$MOLTIS_GATEWAY_URL" \ | ||
| --api-key "$MOLTIS_API_KEY" \ | ||
| chat \ | ||
| --session-key "$MOLTIS_SESSION_KEY" \ | ||
| "$instruction" |
There was a problem hiding this comment.
A non-empty task instruction beginning with a hyphen is passed without an option terminator, so clap interprets it as an option rather than the text positional and exits before sending chat.send.
| exec "$MOLTIS_CTL_BIN" \ | |
| --gateway-url "$MOLTIS_GATEWAY_URL" \ | |
| --api-key "$MOLTIS_API_KEY" \ | |
| chat \ | |
| --session-key "$MOLTIS_SESSION_KEY" \ | |
| "$instruction" | |
| exec "$MOLTIS_CTL_BIN" \ | |
| --gateway-url "$MOLTIS_GATEWAY_URL" \ | |
| --api-key "$MOLTIS_API_KEY" \ | |
| chat \ | |
| --session-key "$MOLTIS_SESSION_KEY" \ | |
| -- \ | |
| "$instruction" |
Knowledge Base Used: CLI and Ctl: moltis binary and moltis-ctl gateway control
Greptile SummaryAdds authenticated moltis-ctl chat commands and a Terminal-Bench installed-agent wrapper.
Confidence Score: 2/5This PR should not merge until task sessions remain isolated without HARBOR_TASK_ID and long-running chat turns can outlive the generic RPC deadline. The default wrapper state reuses one persisted conversation across tasks, while the synchronous benchmark command can terminate before valid model and tool work completes; leading-hyphen instructions also fail before reaching the gateway. Files Needing Attention: tools/terminal_bench/run_moltis_agent.sh, crates/ctl/src/commands/chat.rs
|
| Filename | Overview |
|---|---|
| crates/ctl/src/commands/chat.rs | Adds correctly shaped chat RPC wrappers, but synchronous agent turns remain constrained by the generic fixed RPC timeout. |
| crates/ctl/src/main.rs | Adds chat command dispatch; the text positional does not accept hyphen-prefixed task instructions. |
| tools/terminal_bench/run_moltis_agent.sh | Adds the installed-agent wrapper, but its constant session fallback can merge task histories and its argument boundary rejects leading-hyphen instructions. |
| tools/terminal_bench/test_run_moltis_agent.sh | Covers normal argument, stdin, session, and missing-key behavior but not isolation across missing task IDs or hyphen-prefixed instructions. |
| tools/terminal_bench/README.md | Documents the workflow and isolation requirement, although the wrapper fallback and fixed client timeout do not uphold those guarantees. |
Sequence Diagram
sequenceDiagram
participant Harbor
participant Wrapper as run_moltis_agent.sh
participant Ctl as moltis-ctl
participant Gateway
participant Agent
participant Sessions
Harbor->>Wrapper: task instruction
Wrapper->>Ctl: chat --session-key key instruction
Ctl->>Gateway: chat.send(text, _session_key)
Gateway->>Sessions: load session history
Gateway->>Agent: run model/tool loop
Agent->>Sessions: persist messages
Agent-->>Gateway: final response
Gateway-->>Ctl: RPC response
Ctl-->>Harbor: JSON result
Reviews (1): Last reviewed commit: "feat(ctl): add terminal bench chat runne..." | Re-trigger Greptile
There was a problem hiding this comment.
Pull request overview
This PR adds chat runner capabilities to moltis-ctl (authenticated gateway RPC over WebSocket) and introduces a Terminal-Bench/Harbor “installed-agent” wrapper that invokes moltis-ctl chat with per-task session separation, plus documentation and a shell contract test.
Changes:
- Add
moltis-ctl chatandmoltis-ctl chat-historysubcommands backed by gateway RPC (chat.send/chat.history). - Add
tools/terminal_bench/run_moltis_agent.shwrapper to run Moltis as a Harbor/Terminal-Bench installed agent with a session key. - Add wrapper documentation and a contract-style shell test for the invocation contract.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tools/terminal_bench/test_run_moltis_agent.sh | Adds a shell contract test validating how the wrapper invokes moltis-ctl. |
| tools/terminal_bench/run_moltis_agent.sh | Adds the installed-agent wrapper that builds an instruction and execs moltis-ctl chat. |
| tools/terminal_bench/README.md | Documents how to build/configure Terminal-Bench/Harbor runner and how to smoke test it. |
| crates/ctl/src/main.rs | Adds Chat and ChatHistory clap subcommands and routes them to new command handlers. |
| crates/ctl/src/commands/mod.rs | Exposes the new chat subcommand module. |
| crates/ctl/src/commands/chat.rs | Implements the chat.send / chat.history RPC calls and adds a small unit test. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| exec "$MOLTIS_CTL_BIN" \ | ||
| --gateway-url "$MOLTIS_GATEWAY_URL" \ | ||
| --api-key "$MOLTIS_API_KEY" \ | ||
| chat \ |
| MOLTIS_CTL_BIN="${MOLTIS_CTL_BIN:-moltis-ctl}" | ||
| MOLTIS_GATEWAY_URL="${MOLTIS_GATEWAY_URL:-http://host.docker.internal:13131}" | ||
| MOLTIS_SESSION_KEY="${MOLTIS_SESSION_KEY:-terminal-bench:${HARBOR_TASK_ID:-task}}" | ||
|
|
| pub async fn send(client: &mut CtlClient, text: &str, session_key: &str) -> anyhow::Result<Value> { | ||
| client | ||
| .call( | ||
| "chat.send", | ||
| json!({ | ||
| "text": text, | ||
| "_session_key": session_key, | ||
| }), | ||
| ) | ||
| .await | ||
| .map_err(Into::into) | ||
| } |
Summary
moltis-ctl chatandchat-historycommands over authenticated gateway RPCVerification
cargo fmt --all -- --checkcargo test -p moltis-ctl(2 passed)cargo clippy -p moltis-ctl --all-targets -- -D warningstools/terminal_bench/test_run_moltis_agent.shchat.send; the deployed gateway did not return before the existing 10-second ctl timeout, so model-backed benchmark completion remains environment-blockedNo credentials are included.