Skip to content

feat(agent-server): realtime voice broker for Insider controller conversations - #5180

Draft
enyst wants to merge 4 commits into
mainfrom
feat/insider-voice-realtime
Draft

enyst wants to merge 4 commits into
mainfrom
feat/insider-voice-realtime

Conversation

@enyst

@enyst enyst commented Sep 18, 2026

Copy link
Copy Markdown
Member

HUMAN:


AGENT:

Why

Insider Cat needs Voice to remain attached to its saved controller conversation, with credentials held by Agent Server. A real Codex call could hear the user but lose the request when its intermediate reasoning turn completed without selecting the forwarding tool. Dispatch explicit realtime handoffs directly so the Cat receives the request and the caller hears its saved result.

Summary

  • Add conversation-bound Voice availability, WebRTC setup, call status, and teardown endpoints for tagged Insider controllers. OpenAI API credentials remain on the server.
  • Add an opt-in Codex transport using a dedicated app-server home and sign-in. Validate handoff_request.input_transcript, reserve each handoff_id before dispatch, and forward at most once per call. No Codex tool call chooses the destination or supplies the answer.
  • Wait for the saved Cat run, including stop hooks, to finish before speaking its verified answer through appendSpeech. Replays, conflicts, concurrent requests, disconnects, and uncertain outcomes preserve accepted work and avoid automatic resubmission.
  • Add optional fixed error_code values to call status so clients distinguish requests that were not sent, uncertain saved results, and connection failures without displaying provider error text.
  • Make explicit LocalConversation.condense() invoke the condenser without executing pending unconfirmed tools; return 409 when no compatible condenser exists.

REST API contract changes

Compared with base OpenAPI 28e8ed273617 for public /api/** paths.

--- base public OpenAPI
+++ head public OpenAPI
@@ -5,0 +6 @@
+operation DELETE /api/conversations/{conversation_id}/voice/realtime/{call_id} operationId=end_realtime_call_api_conversations__conversation_id__voice_realtime__call_id__delete
@@ -32,0 +34,2 @@
+operation GET /api/conversations/{conversation_id}/voice operationId=voice_availability_api_conversations__conversation_id__voice_get
+operation GET /api/conversations/{conversation_id}/voice/realtime/{call_id} operationId=realtime_call_status_api_conversations__conversation_id__voice_realtime__call_id__get
@@ -116,0 +120 @@
+operation POST /api/conversations/{conversation_id}/voice/realtime operationId=create_realtime_call_api_conversations__conversation_id__voice_realtime_post
@@ -150,0 +155,2 @@
+parameter DELETE /api/conversations/{conversation_id}/voice/realtime/{call_id} path:call_id required=true schema=type="string" pattern="^[A-Za-z0-9_-]{1,128}$"
+parameter DELETE /api/conversations/{conversation_id}/voice/realtime/{call_id} path:conversation_id required=true schema=type="string" format="uuid"
@@ -201,0 +208,3 @@
+parameter GET /api/conversations/{conversation_id}/voice path:conversation_id required=true schema=type="string" format="uuid"
+parameter GET /api/conversations/{conversation_id}/voice/realtime/{call_id} path:call_id required=true schema=type="string" pattern="^[A-Za-z0-9_-]{1,128}$"
+parameter GET /api/conversations/{conversation_id}/voice/realtime/{call_id} path:conversation_id required=true schema=type="string" format="uuid"
@@ -306,0 +316 @@
+parameter POST /api/conversations/{conversation_id}/voice/realtime path:conversation_id required=true schema=type="string" format="uuid"
@@ -351,0 +362 @@
+requestBody POST /api/conversations/{conversation_id}/voice/realtime application/json required=true schema=RealtimeOffer
@@ -384,0 +396,2 @@
+response DELETE /api/conversations/{conversation_id}/voice/realtime/{call_id} 200 application/json schema=Success
+response DELETE /api/conversations/{conversation_id}/voice/realtime/{call_id} 422 application/json schema=HTTPValidationError
@@ -446,0 +460,4 @@
+response GET /api/conversations/{conversation_id}/voice 200 application/json schema=VoiceAvailability
+response GET /api/conversations/{conversation_id}/voice 422 application/json schema=HTTPValidationError
+response GET /api/conversations/{conversation_id}/voice/realtime/{call_id} 200 application/json schema=CodexVoiceStatus
+response GET /api/conversations/{conversation_id}/voice/realtime/{call_id} 422 application/json schema=HTTPValidationError
@@ -576,0 +594 @@
+response POST /api/conversations/{conversation_id}/condense 409 no-content
@@ -637,0 +656,2 @@
+response POST /api/conversations/{conversation_id}/voice/realtime 200 application/json schema=RealtimeAnswer
+response POST /api/conversations/{conversation_id}/voice/realtime 422 application/json schema=HTTPValidationError
@@ -1255,0 +1276,7 @@
+schema CodexVoiceErrorCode type="string" enum=["request_not_sent","relay_failed","connection_failed"]
+schema CodexVoiceStatus property error optional schema=anyOf=[type="string",type="null"]
+schema CodexVoiceStatus property error_code optional schema=anyOf=[CodexVoiceErrorCode,type="null"]
+schema CodexVoiceStatus property provider optional schema=type="string" const="codex" default="codex"
+schema CodexVoiceStatus property status required schema=type="string" enum=["listening","thinking","speaking","closed","error"]
+schema CodexVoiceStatus property transcripts required schema=type="array" items=VoiceTranscript
+schema CodexVoiceStatus type="object"
@@ -2331,0 +2359,8 @@
+schema RealtimeAnswer property call_id optional schema=anyOf=[type="string",type="null"]
+schema RealtimeAnswer property delegation optional schema=type="string" enum=["client","server"] default="client"
+schema RealtimeAnswer property model optional schema=type="string" default="gpt-realtime-2.1"
+schema RealtimeAnswer property provider optional schema=type="string" enum=["openai","codex"] default="openai"
+schema RealtimeAnswer property sdp required schema=type="string"
+schema RealtimeAnswer type="object"
+schema RealtimeOffer property sdp required schema=type="string" minLength=1 maxLength=200000
+schema RealtimeOffer type="object" additionalProperties=false
@@ -2855,0 +2891,12 @@
+schema VoiceAvailability property available required schema=type="boolean"
+schema VoiceAvailability property delegation optional schema=type="string" enum=["client","server"] default="client"
+schema VoiceAvailability property execution_status required schema=ConversationExecutionStatus
+schema VoiceAvailability property model optional schema=type="string" default="gpt-realtime-2.1"
+schema VoiceAvailability property provider optional schema=type="string" enum=["openai","codex"] default="openai"
+schema VoiceAvailability property reason optional schema=anyOf=[type="string" enum=["missing_openai_api_key","codex_not_installed","codex_not_signed_in","codex_unavailable"],type="null"]
+schema VoiceAvailability property run_active required schema=type="boolean"
+schema VoiceAvailability type="object"
+schema VoiceTranscript property id required schema=type="string"
+schema VoiceTranscript property role required schema=type="string" enum=["user","assistant"]
+schema VoiceTranscript property text required schema=type="string"
+schema VoiceTranscript type="object"

Issue Number

None.

How to Test

From the repository root:

uv run pytest tests/agent_server/test_voice_router.py tests/agent_server/test_codex_voice.py tests/agent_server/test_codex_voice_protocol.py tests/cross/test_remote_conversation_live_server.py::test_insider_voice_missing_key_and_condense_are_safe_over_http

58 passed on publication commit 02a51ae46, in its own frozen environment. All required pre-commit hooks passed on the three files changed by the handoff repair. The HTTP regression checks authentication, missing-key availability, safe condensation, conversation-bound status, fixed error codes, and no-store caching.

Real Codex WebRTC verification, September 19–20, 2026: generated speech with an existing Codex sign-in and a Luna/high saved Cat produced a saved request and answer and returned the answer as audio. A fresh call recalled the same saved test word, saved its new question and answer, and spoke the correct word. Temporary calls and the test conversation were removed afterward. This verifies actual provider round trips; physical iPad microphone/speaker behavior and the API-key transport still need live device/provider validation.

Video/Screenshots

No new browser capture; real audio and saved-event checks were run through WebRTC.

Type

  • Bug fix
  • Feature
  • Refactor
  • Breaking change
  • Docs / chore

Notes


🐳 Agent Server images for this PR — GHCR package, pull/run commands, and all pushed tags (click to expand)

GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server

Variants & Base Images

Variant Architectures Base Image Docs / Tags
java amd64, arm64 eclipse-temurin:17-jdk Link
python-slim amd64, arm64 python-node-runtime Link
python-minimal amd64, arm64 python-node-runtime Link
python amd64, arm64 python-node-runtime Link
golang amd64, arm64 golang:1.21-bookworm Link

Pull (multi-arch manifest)

# Each variant is a multi-arch manifest supporting both amd64 and arm64
docker pull ghcr.io/openhands/agent-server:02a51ae-python

Run

docker run -it --rm \
  -p 8000:8000 \
  --name agent-server-02a51ae-python \
  ghcr.io/openhands/agent-server:02a51ae-python

All tags pushed for this build

ghcr.io/openhands/agent-server:02a51ae-golang-amd64
ghcr.io/openhands/agent-server:02a51ae46744aa5a7c27067179502ceffb404610-golang-amd64
ghcr.io/openhands/agent-server:feat-insider-voice-realtime-golang-amd64
ghcr.io/openhands/agent-server:02a51ae-golang_tag_1.21-bookworm-amd64
ghcr.io/openhands/agent-server:02a51ae-golang-arm64
ghcr.io/openhands/agent-server:02a51ae46744aa5a7c27067179502ceffb404610-golang-arm64
ghcr.io/openhands/agent-server:feat-insider-voice-realtime-golang-arm64
ghcr.io/openhands/agent-server:02a51ae-golang_tag_1.21-bookworm-arm64
ghcr.io/openhands/agent-server:02a51ae-java-amd64
ghcr.io/openhands/agent-server:02a51ae46744aa5a7c27067179502ceffb404610-java-amd64
ghcr.io/openhands/agent-server:feat-insider-voice-realtime-java-amd64
ghcr.io/openhands/agent-server:02a51ae-eclipse-temurin_tag_17-jdk-amd64
ghcr.io/openhands/agent-server:02a51ae-java-arm64
ghcr.io/openhands/agent-server:02a51ae46744aa5a7c27067179502ceffb404610-java-arm64
ghcr.io/openhands/agent-server:feat-insider-voice-realtime-java-arm64
ghcr.io/openhands/agent-server:02a51ae-eclipse-temurin_tag_17-jdk-arm64
ghcr.io/openhands/agent-server:02a51ae-python-amd64
ghcr.io/openhands/agent-server:02a51ae46744aa5a7c27067179502ceffb404610-python-amd64
ghcr.io/openhands/agent-server:feat-insider-voice-realtime-python-amd64
ghcr.io/openhands/agent-server:02a51ae-python-node-runtime-amd64
ghcr.io/openhands/agent-server:02a51ae-python-arm64
ghcr.io/openhands/agent-server:02a51ae46744aa5a7c27067179502ceffb404610-python-arm64
ghcr.io/openhands/agent-server:feat-insider-voice-realtime-python-arm64
ghcr.io/openhands/agent-server:02a51ae-python-node-runtime-arm64
ghcr.io/openhands/agent-server:02a51ae-python-minimal-amd64
ghcr.io/openhands/agent-server:02a51ae46744aa5a7c27067179502ceffb404610-python-minimal-amd64
ghcr.io/openhands/agent-server:feat-insider-voice-realtime-python-minimal-amd64
ghcr.io/openhands/agent-server:02a51ae-python-node-runtime-minimal-amd64
ghcr.io/openhands/agent-server:02a51ae-python-minimal-arm64
ghcr.io/openhands/agent-server:02a51ae46744aa5a7c27067179502ceffb404610-python-minimal-arm64
ghcr.io/openhands/agent-server:feat-insider-voice-realtime-python-minimal-arm64
ghcr.io/openhands/agent-server:02a51ae-python-node-runtime-minimal-arm64
ghcr.io/openhands/agent-server:02a51ae-python-slim-amd64
ghcr.io/openhands/agent-server:02a51ae46744aa5a7c27067179502ceffb404610-python-slim-amd64
ghcr.io/openhands/agent-server:feat-insider-voice-realtime-python-slim-amd64
ghcr.io/openhands/agent-server:02a51ae-python-node-runtime-slim-amd64
ghcr.io/openhands/agent-server:02a51ae-python-slim-arm64
ghcr.io/openhands/agent-server:02a51ae46744aa5a7c27067179502ceffb404610-python-slim-arm64
ghcr.io/openhands/agent-server:feat-insider-voice-realtime-python-slim-arm64
ghcr.io/openhands/agent-server:02a51ae-python-node-runtime-slim-arm64
ghcr.io/openhands/agent-server:02a51ae-golang
ghcr.io/openhands/agent-server:02a51ae46744aa5a7c27067179502ceffb404610-golang
ghcr.io/openhands/agent-server:feat-insider-voice-realtime-golang
ghcr.io/openhands/agent-server:02a51ae-golang_tag_1.21-bookworm
ghcr.io/openhands/agent-server:02a51ae-java
ghcr.io/openhands/agent-server:02a51ae46744aa5a7c27067179502ceffb404610-java
ghcr.io/openhands/agent-server:feat-insider-voice-realtime-java
ghcr.io/openhands/agent-server:02a51ae-eclipse-temurin_tag_17-jdk
ghcr.io/openhands/agent-server:02a51ae-python-minimal
ghcr.io/openhands/agent-server:02a51ae46744aa5a7c27067179502ceffb404610-python-minimal
ghcr.io/openhands/agent-server:feat-insider-voice-realtime-python-minimal
ghcr.io/openhands/agent-server:02a51ae-python-node-runtime-minimal
ghcr.io/openhands/agent-server:02a51ae-python-slim
ghcr.io/openhands/agent-server:02a51ae46744aa5a7c27067179502ceffb404610-python-slim
ghcr.io/openhands/agent-server:feat-insider-voice-realtime-python-slim
ghcr.io/openhands/agent-server:02a51ae-python-node-runtime-slim
ghcr.io/openhands/agent-server:02a51ae-python
ghcr.io/openhands/agent-server:02a51ae46744aa5a7c27067179502ceffb404610-python
ghcr.io/openhands/agent-server:feat-insider-voice-realtime-python
ghcr.io/openhands/agent-server:02a51ae-python-node-runtime

About Multi-Architecture Support

  • Each variant tag (e.g., 02a51ae-python) is a multi-arch manifest supporting both amd64 and arm64
  • Docker automatically pulls the correct architecture for your platform
  • Individual architecture tags (e.g., 02a51ae-python-amd64) are also available if needed

enyst and others added 2 commits September 19, 2026 00:32
LocalConversation.condense() used to process the condensation request by
calling agent.step(), which also executes pending actions, including tool
calls still waiting for user confirmation. Invoke the condenser directly
instead so an explicit compaction only compacts history.

The condense endpoint now maps the ValueError raised when no compatible
condenser exists to HTTP 409 instead of a 500.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ersations

Add a conversation-scoped voice API under /api/conversations/{id}/voice,
restricted to conversations tagged as Insider controllers:

- GET reports availability, provider, model, and whether a run is active.
- POST /realtime brokers a WebRTC offer. On the default OpenAI provider the
  server builds the Realtime session (instructions, bounded saved context,
  send_to_insider and end_voice_call tools) and exchanges the SDP through
  OpenAI's calls endpoint with the server-held API key.
- GET and DELETE /realtime/{call_id} poll and end a call, with call IDs
  bound to their conversation.

Add an opt-in Codex Voice relay (voice_provider = "codex") that delegates
the call to a Codex app-server sign-in kept in a dedicated home directory,
configured through codex_voice_home.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

REST API breakage checks (OpenAPI) — ✅ PASSED

Result:PASSED

Action log

Build lifecycle test state with the Codex Voice manager created by create_app and verify it is awaited during shutdown.

Co-authored-by: openhands <openhands@all-hands.dev>
@smolpaws

Copy link
Copy Markdown
Contributor

Traceability and validation update (2026-09-19): the original tested SDK worktree is now preserved at enyst/agent-sdk@7c806a919, on gpt/insider-ui-voice. Its 11 feature-file changes match this draft's original head 134f57d06: all newly added files are byte-identical, and the existing files have identical feature additions/removals. That snapshot retains its older base; this draft remains the upstream review target.

The six failing lifecycle cases were missing the Voice manager in their fake application state. The test-only follow-up adds that manager and verifies shutdown awaits it. Production behavior is unchanged. The corresponding fix is also preserved on the original tested branch at d5dc75ee6.

Validation:

  • Original tested snapshot: 148 focused tests passed (44 Voice, 13 condensation, 91 conversation-router), plus the real HTTP regression test_insider_voice_missing_key_and_condense_are_safe_over_http.
  • This draft's newer base with the fixture fix: all six previously failing lifecycle cases passed with LITELLM_LOCAL_MODEL_COST_MAP=True OPENHANDS_SUPPRESS_BANNER=1 .venv/bin/python -m pytest tests/agent_server/test_api.py::TestServiceParallelization tests/agent_server/test_init_router.py::test_lifespan_teardown_releases_conversation_service_after_init -q.
  • Required pre-commit checks passed on both checkouts, including Pyright; no model or provider calls were needed for these checks.

The HTTP regression verifies authentication, missing-provider availability, and safe rejection of condensation without a condenser. Pending-approval preservation is covered by the separate condensation unit test. Full CI still needs to complete on the new head; this update does not claim that the earlier sdk-tests failure is resolved.

@github-actions

github-actions Bot commented Sep 19, 2026

Copy link
Copy Markdown
Contributor

Coverage

Coverage Report •
FileStmtsMissCoverMissing
openhands-agent-server/openhands/agent_server
   api.py3142692%141, 143–148, 150, 152, 154, 199, 213, 219, 275, 280, 290–292, 323, 329, 333, 354–355, 607, 610, 616
   codex_voice.py3985586%120–130, 135–137, 154–157, 217, 248, 275, 283, 298, 319, 332, 367, 376–379, 387, 394–396, 403–405, 407–408, 422–423, 489–490, 539, 569, 583, 600, 605, 641–644, 672–673, 687
   config.py132199%457
   conversation_router.py2721495%193, 320, 402, 448, 508, 678–681, 704–706, 746, 784
   voice_router.py158994%129, 154, 193, 204–205, 321, 330–331, 333
openhands-sdk/openhands/sdk/conversation/impl
   local_conversation.py10988492%167, 345, 411, 678–679, 721, 763, 1115–1116, 1121, 1136, 1138, 1303, 1329, 1409, 1413–1419, 1472, 1492–1494, 1543, 1562–1564, 1875–1876, 2114, 2117–2118, 2143, 2174, 2180, 2315, 2319, 2322, 2325, 2329–2330, 2334–2335, 2338, 2345, 2365, 2369, 2372, 2391, 2443, 2446, 2478, 2485–2486, 2494, 2498–2500, 2507, 2544–2550, 2553, 2556, 2563, 2664, 2669, 2789–2790, 2808–2809, 2842, 3046, 3050, 3120, 3127–3128
TOTAL44948820382% 

Validate and deduplicate realtime handoffs before dispatch, speak only saved results, and expose safe error codes for connection and request failures.

Co-authored-by: openhands <openhands@all-hands.dev>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants