feat: add submodule of worker-vllm, updated fastapi endpoints - #6
Open
velaraptor-runpod wants to merge 13 commits into
Open
feat: add submodule of worker-vllm, updated fastapi endpoints#6velaraptor-runpod wants to merge 13 commits into
velaraptor-runpod wants to merge 13 commits into
Conversation
TimPietruskyRunPod
suggested changes
Mar 26, 2026
TimPietruskyRunPod
left a comment
TimPietruskyRunPod
left a comment
There was a problem hiding this comment.
A few items to address before merge — mostly small fixes. The core rewrite using vLLM's native serving classes is solid.
Yhlong00
approved these changes
May 1, 2026
There was a problem hiding this comment.
Pull request overview
This PR migrates the project from a custom FastAPI + vLLM wrapper (src/handler.py + local request/response models) to a new load-balancer-oriented HTTP server (handler_lb.py) built on top of the worker-vllm submodule and vLLM’s OpenAI/Anthropic entrypoints, while updating documentation and RunPod Hub metadata accordingly.
Changes:
- Replaced the legacy
src/handler/models/utils withhandler_lb.py, exposing/ping,/v1/models, OpenAI Completions/Chat/Responses, and Anthropic Messages endpoints. - Updated the Docker image to CUDA 12.9 +
vllm[flashinfer], and switched the container layout to copyworker-vllm/srcinto/src. - Added RunPod Hub assets (
.runpod/*) and theworker-vllmgit submodule; removed the previous example script and CI workflow files.
Reviewed changes
Copilot reviewed 16 out of 16 changed files in this pull request and generated 15 comments.
Show a summary per file
| File | Description |
|---|---|
handler_lb.py |
New FastAPI server implementing RunPod LB health semantics and OpenAI/Anthropic-compatible endpoints via vLLM serving classes. |
Dockerfile |
Moves to CUDA 12.9 base, installs uv + vLLM/FlashInfer, and copies worker-vllm sources for engine/model download. |
README.md |
Updated documentation for the new architecture, endpoints, and environment variables (worker-vllm based). |
.runpod/README.md |
Added RunPod template documentation for configuration and SDK usage. |
.runpod/hub.json |
Added RunPod Hub metadata/configuration for LB endpoint deployment. |
.gitmodules |
Adds worker-vllm as a git submodule dependency. |
src/handler.py |
Removed legacy custom FastAPI handler implementation. |
src/models.py |
Removed legacy Pydantic request/response models. |
src/utils.py |
Removed legacy tokenizer/prompt formatting helpers. |
example.py |
Removed legacy manual test script. |
builder/setup.sh |
Removed legacy optional setup script. |
.github/workflows/CI-test_handler.yml |
Removed legacy CI workflow. |
.github/workflows/CI-test_e2e.yml |
Removed legacy CI workflow. |
.github/workflows/CI-runpod_dep.yml |
Removed legacy CI workflow. |
.github/workflows/build-test-release.yml |
Removed legacy CI workflow. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| return StreamingResponse(response, media_type="text/event-stream") | ||
|
|
||
|
|
||
| if __name__ == "__main__" or multiprocessing.current_process().name == "MainProcess": |
Comment on lines
+198
to
+206
| @app.post("/v1/chat/completions") | ||
| async def chat_completions(request: Request): | ||
| body = await request.json() | ||
|
|
||
| try: | ||
| req = ChatCompletionRequest(**body) | ||
| except Exception as e: | ||
| log.warning(f"Chat completions validation error: {e}") | ||
| return JSONResponse( |
|
|
||
|
|
||
| @app.get("/v1/models") | ||
| async def list_models(): |
Comment on lines
+211
to
+212
| response = await _chat_engine.create_chat_completion(req, raw_request=request) | ||
|
|
Comment on lines
+15
to
+18
| # Install additional Python dependencies (after vLLM to avoid PyTorch version conflicts) | ||
| COPY worker-vllm/builder/requirements.txt /requirements.txt | ||
| RUN --mount=type=cache,target=/root/.cache/uv \ | ||
| uv pip install --system -r /requirements.txt |
Comment on lines
+240
to
+241
| response = await _completion_engine.create_completion(req, raw_request=request) | ||
|
|
Comment on lines
+269
to
+270
| response = await _responses_engine.create_responses(req, raw_request=request) | ||
|
|
Comment on lines
+350
to
+351
| response = await _messages_engine.create_messages(req, raw_request=request) | ||
|
|
Comment on lines
+293
to
+296
| try: | ||
| response = await _responses_engine.retrieve_responses( | ||
| response_id, starting_after=starting_after, stream=stream | ||
| ) |
Comment on lines
+321
to
+322
| try: | ||
| response = await _responses_engine.cancel_responses(response_id) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.