feat(voice): port AvatarSession base class and transcript sync asymmetric detach warning#1280
feat(voice): port AvatarSession base class and transcript sync asymmetric detach warning#1280toubatbrian wants to merge 1 commit intomainfrom
Conversation
…tric detach warning Ports livekit/agents#5499 from the Python agents repo. - Add `voice.AvatarSession` base class that registers `aclose` as a job shutdown callback and warns when started after `AgentSession.start()` has already wired an audio output. - Port the `TranscriptSynchronizer` one-shot warning for asymmetric audio/text detach: track `_audioAttached` / `_textAttached` via `onAttached` / `onDetached`, reset the warn flag on re-enable, and log when only one of audio/text is detached. - Wire avatar plugins (anam, bey, lemonslice, trugen) to inherit from the new base class and call `super.start(agentSession, room)` first. https://claude.ai/code/session_01HwutyuE78oUGoYwUV69pF8
🦋 Changeset detectedLatest commit: cc45c7c The changes in this PR will be included in the next version bump. This PR includes changesets to release 26 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| const _AVATAR_NAME = 'anam-avatar-agent'; | ||
|
|
||
| export class AvatarSession { | ||
| export class AvatarSession extends voice.AvatarSession { |
There was a problem hiding this comment.
🟡 Missing // Ref comment on extends voice.AvatarSession class declaration (anam)
Per the CLAUDE.md porting rule, every JS change that corresponds to a Python change must carry an inline // Ref: python <path> - <line-range> lines comment directly above the relevant line(s). The class declaration change to extends voice.AvatarSession is a direct port from the Python plugin (where the class similarly extends the base AvatarSession), but it lacks the required reference comment. The super() call at plugins/anam/src/avatar.ts:53 is also a ported change without a // Ref.
Was this helpful? React with 👍 or 👎 to provide feedback.
| * ``` | ||
| */ | ||
| export class AvatarSession { | ||
| export class AvatarSession extends voice.AvatarSession { |
There was a problem hiding this comment.
🟡 Missing // Ref comment on extends voice.AvatarSession class declaration (bey)
Same issue as in the anam plugin. The class declaration change to extends voice.AvatarSession at plugins/bey/src/avatar.ts:99 is a direct port from the Python plugin but lacks the required // Ref comment per CLAUDE.md. The super() call at plugins/bey/src/avatar.ts:116 also lacks a // Ref.
Was this helpful? React with 👍 or 👎 to provide feedback.
| * ``` | ||
| */ | ||
| export class AvatarSession { | ||
| export class AvatarSession extends voice.AvatarSession { |
There was a problem hiding this comment.
🟡 Missing // Ref comment on extends voice.AvatarSession class declaration (lemonslice)
Same issue as in the anam plugin. The class declaration change to extends voice.AvatarSession at plugins/lemonslice/src/avatar.ts:125 is a direct port from the Python plugin but lacks the required // Ref comment per CLAUDE.md. The super() call at plugins/lemonslice/src/avatar.ts:146 also lacks a // Ref.
Was this helpful? React with 👍 or 👎 to provide feedback.
| * @public | ||
| */ | ||
| export class AvatarSession { | ||
| export class AvatarSession extends voice.AvatarSession { |
There was a problem hiding this comment.
🟡 Missing // Ref comment on extends voice.AvatarSession class declaration (trugen)
Same issue as in the anam plugin. The class declaration change to extends voice.AvatarSession at plugins/trugen/src/avatar.ts:67 is a direct port from the Python plugin but lacks the required // Ref comment per CLAUDE.md. The super() call at plugins/trugen/src/avatar.ts:84 also lacks a // Ref.
Was this helpful? React with 👍 or 👎 to provide feedback.
|
|
3 participants
Summary
Automated port of livekit/agents#5499 (
feat(avatar): add AvatarSession base class, warn on sync mis-wire) into agents-js.cc @toubatbrian @livekit/agent-devs — please review.
Ported Features
1.
voice.AvatarSessionbase class (new)New file:
agents/src/voice/avatar/avatar_session.ts— re-exported fromvoice/avatar/index.tsso it lands onvoice.AvatarSession.Behavior mirrors the Python base at
livekit-agents/livekit/agents/voice/avatar/_types.py:start(agentSession, room):getJobContext(false)and, if present, registers() => this.aclose()viajobCtx.addShutdownCallback(...). If no job context is active, logs a debug message telling the caller toaclose()manually — same message as Python.agentSession.output.audioand, when the session has already been started, logs a warning that the existing audio output will be replaced. In JS there is no.labelonAudioOutput, so the warning attachesaudioOutput.constructor.nameinstead oflabel.aclose(): default no-op, overridable by subclasses.2.
AgentSession._startedinternal getterAgentSessionexposes an internal_startedgetter that reads the privatestartedfield. This is the JS analog of Python'sagent_session._startedattribute access inside the baseAvatarSession.start. Marked@internalwith a// Ref:comment pointing at the Python source.3.
TranscriptionSynchronizerasymmetric-detach warningIn
agents/src/voice/transcription/synchronizer.ts:_audioAttached,_textAttached, and_warnedAsymmetricDetachinternal flags toTranscriptionSynchronizer(defaultstrue,true,false— matches Python)._onAttachmentChanged({ audioAttached?, textAttached? })which updates the attached flags and callsthis.enabled = this._audioAttached && this._textAttached.set enabled()now resets_warnedAsymmetricDetach = falseon the disabled → enabled transition (matches Pythonset_enabled).SyncedAudioOutputandSyncedTextOutputnow overrideonAttached()/onDetached()to forward the event into_onAttachmentChanged. This is the JS equivalent of Python'son_attached/on_detachedoverrides in_SyncedAudioOutput/_SyncedTextOutput.SyncedAudioOutput.captureFrame(when the synchronizer is disabled) andSyncedTextOutput.captureText(same), we now emit a one-shot warning when the synchronizer finds itself with an asymmetric (audio attached / text detached, or vice versa) state. Warning strings match Python verbatim.4. Avatar plugin updates
Updated plugins to extend
voice.AvatarSessionand callsuper.start(agentSession, room)at the top ofstart():plugins/anam/src/avatar.tsplugins/bey/src/avatar.tsplugins/lemonslice/src/avatar.tsplugins/trugen/src/avatar.tsEach call site carries a
// Ref: python ...comment pointing at the line in the corresponding Python plugin.plugins/hedrais already deprecated (the ctor throws), so it was intentionally left unchanged.plugins/runwayis TTS-only in agents-js and has no avatar, so it was skipped.Implementation Notes (language-level differences)
AvatarSession.start: Python usesNone, and some plugins (e.g. lemonslice) return a session id. Python's type system lets them override with# type: ignore[override]. TypeScript rejects wideningPromise<void>toPromise<string>in an override, so the base class's return type isPromise<unknown>instead ofPromise<void>. Subclasses are free to returnvoidor any other value; the base implementation explicitlyreturn undefined.labelonAudioOutput: The PythonAudioOutputbase has alabelproperty that's surfaced in the "audio output will be replaced" warning.AudioOutputin agents-js does not exposelabel, so the JS warning usesaudioOutput.constructor.nameinstead. Same signal, slightly different shape.on_attached/on_detachedon_SyncedAudioOutput/_SyncedTextOutputvia the baseio.AudioOutput/io.TextOutputhooks, which are already called onsession.output.audio = .../session.output.transcription = ...assignment inAgentOutput. The JSAudioOutput/TextOutputbase classes already haveonAttached()/onDetached()with the same call-site wiring inAgentOutput(seeagents/src/voice/io.ts), so the JS port simply overrides these methods in the synced subclasses — no new plumbing was required.aclose: The Python PR also migrates bithuman's local-mode runtime cleanup from an inlinejob_ctx.add_shutdown_callbackintoAvatarSession.aclose(). agents-js has no bithuman plugin, so this piece is not ported.Tests
pnpm buildpasses (all packages).pnpm test agents/src/voice/transcriptionpasses (19/19 existing synchronizer tests unaffected).pnpm test plugins/lemonslicepasses (2/2).pnpm lintpasses (only pre-existing warnings in unrelated files).Test plan
AvatarSession.start()registersacloseon the job shutdown callback (anam, bey, lemonslice, trugen).session.start({ agent, room }).session.output.audio = nullorsession.output.transcription = nullwhile the other side is still attached.https://claude.ai/code/session_01HwutyuE78oUGoYwUV69pF8