Add support for local LLM provider#3
Open
ereish64 wants to merge 1 commit into
Open
Conversation
Introduce configurable local LLM support alongside Groq. add LLM_PROVIDER, LOCAL_LLM_BASE_URL, LOCAL_LLM_API_KEY and update default LLM_MODEL to .env.example This enables using a local OpenAI-compatible endpoint for LLM streaming while preserving Groq/OpenAI compatibility and updates docs and validation accordingly.
Contributor
Author
|
VLLM3_5 14B running on 4x V100 GPUs had a first token latency of <100ms. Perhaps this PR is out of the scope of what you're looking to build, but dedicated will always be faster than public APIs (if you have the hardware). Hope this helps! |
revl
added a commit
to revl/shuo
that referenced
this pull request
May 13, 2026
This fixes delayed assistant turn completion when using ElevenLabs realtime websocket TTS. shuo currently treats `flush()` as both: - "generate buffered audio now" - "end the TTS input stream" For ElevenLabs, these are not equivalent behaviors. As a result, after assistant speech is synthesized, the websocket remains logically open for additional text input until ElevenLabs times out (~20 seconds), producing: TTS error: code=1008 error='input_timeout_exceeded' message= 'Have not received a new text input within the timeout of 20 seconds. Streaming input terminated. Please make sure to either feed the input in the timely manner, or to send the end of input text (empty string "") when done.' During that interval, the agent turn remains active, which can interfere with caller speech / turn-taking. Fix: Explicitly send the ElevenLabs end-of-input message. This allows the assistant turn to complete immediately after synthesis finishes. PR NickTikhonov#3
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Introduce configurable local LLM support alongside Groq. add LLM_PROVIDER, LOCAL_LLM_BASE_URL, LOCAL_LLM_API_KEY and update default LLM_MODEL to .env.example
This enables using a local OpenAI-compatible endpoint for LLM streaming while preserving Groq compatibility.