Skip to content
BranchesTags

Repository files navigation

SGLang worker banner

Run LLMs and VLMs using SGLang

RunPod

Deploy

From the Runpod Hub

Open the Hub listing, click Deploy, and fill in:

Field Required Notes
Model (MODEL_NAME) Yes Hugging Face repo ID, e.g. HuggingFaceTB/SmolLM2-1.7B-Instruct. There is no default — the worker will not start without it.
Hugging Face Token (HF_TOKEN) Only for gated or private models Needed for anything behind a licence click-through, such as meta-llama/*.

Everything else is optional and listed under Endpoint Configuration.

Two things to check before deploying a larger model:

  • Container disk defaults to 100 GB and must hold the image plus the model weights. Raise it for models above roughly 60 GB of weights.
  • GPU selection must have enough VRAM for the model. The default pool set starts at 24 GB; narrow it to the larger pools for bigger models.

Deploy the image yourself

runpod/worker-sglang:<tag>

Set MODEL_NAME (and HF_TOKEN if the model is gated), and constrain the endpoint to CUDA 12.9 or newer — see Hardware requirements. Without that constraint the endpoint can be placed on a host the image cannot start on, and jobs will queue forever against a silently restarting worker.

Run it locally

echo "HF_TOKEN=your_huggingface_token_here" > .env
docker compose up

See .github/CONTRIBUTING.md for the full local workflow.

Hardware requirements

This worker is built on lmsysorg/sglang:v0.5.17-cu129.

  • The host driver must provide CUDA 12.9 or newer. The base image declares cuda>=12.9. On an older host the container never starts: it fails in the NVIDIA runtime hook before the handler runs, and the worker restarts in a loop without being reported as unhealthy, so requests simply sit in the queue. The Hub listing pins allowedCudaVersions to ["13.0", "12.9"] for this reason. If you deploy the image directly instead of from the Hub, apply the same constraint yourself.
  • Blackwell is supported. The cu129 build ships sm_120 kernels, so RTX PRO 6000 Blackwell works, including the MIG slices. Verified on RTX PRO 6000 Blackwell Server Edition and on RTX 4090.
  • The listing defaults to a wide GPU set — ADA_24, AMPERE_24, ADA_32_PRO, ADA_48_PRO, AMPERE_48, ADA_80_PRO, AMPERE_80 — spanning 24GB to 80GB so the scheduler has somewhere to place a worker. Narrow it if you want to control cost, and widen it to BLACKWELL_96 if you need a 96GB card. Note that the Blackwell MIG slices are not in Blackwell-named pools: 1g.24gb sits in AMPERE_24 and 2g.48gb in ADA_48_PRO.

Endpoint Configuration

All behaviour is controlled through environment variables:

Environment Variable Description Default Options
MODEL_NAME Hugging Face model name or local path (required) Hugging Face repo ID or local folder path
HF_TOKEN HuggingFace access token for gated/private models Your HuggingFace access token
TOKENIZER_PATH Path of the tokenizer
TOKENIZER_MODE Tokenizer mode "auto" "auto", "slow"
LOAD_FORMAT Format of model weights to load "auto" "auto", "pt", "safetensors", "npcache", "dummy"
DTYPE Data type for weights and activations "auto" "auto", "half", "float16", "bfloat16", "float", "float32"
CONTEXT_LENGTH Model's maximum context length
QUANTIZATION Quantization method "awq", "awq_marlin", "bitsandbytes", "compressed-tensors", "fp8", "gguf", "gptq", "gptq_marlin", "marlin", "modelopt_fp4", "modelopt_fp8", "mxfp4", "w8a8_fp8", "w8a8_int8"
SERVED_MODEL_NAME Override model name in API
CHAT_TEMPLATE Chat template name or path
MEM_FRACTION_STATIC Fraction of memory for static allocation
MAX_RUNNING_REQUESTS Maximum number of running requests
MAX_TOTAL_TOKENS Maximum tokens in memory pool
CHUNKED_PREFILL_SIZE Max tokens in chunk for chunked prefill
MAX_PREFILL_TOKENS Max tokens in prefill batch 16384
SCHEDULE_POLICY Request scheduling policy "fcfs" "lpm", "random", "fcfs", "dfs-weight"
SCHEDULE_CONSERVATIVENESS Conservativeness of schedule policy 1.0
TENSOR_PARALLEL_SIZE Tensor parallelism size 1
STREAM_INTERVAL Streaming interval in token length 1
RANDOM_SEED Random seed
LOG_LEVEL Logging level for all loggers "info"
LOG_LEVEL_HTTP Logging level for HTTP server
API_KEY API key for the server
FILE_STORAGE_PATH Directory for storing uploaded/generated files "sglang_storage"
DATA_PARALLEL_SIZE Data parallelism size 1
LOAD_BALANCE_METHOD Load balancing strategy "round_robin" "round_robin", "shortest_queue"
SKIP_TOKENIZER_INIT Skip tokenizer init false boolean (true or false)
TRUST_REMOTE_CODE Allow custom models from Hub false boolean (true or false)
LOG_REQUESTS Log inputs and outputs of requests false boolean (true or false)
SHOW_TIME_COST Show time cost of custom marks false boolean (true or false)
DISABLE_RADIX_CACHE Disable RadixAttention for prefix caching false boolean (true or false)
DISABLE_CUDA_GRAPH Disable CUDA Graph false boolean (true or false)
DISABLE_OUTLINES_DISK_CACHE Disable disk cache for Outlines grammar false boolean (true or false)
ENABLE_TORCH_COMPILE Optimize model with torch.compile false boolean (true or false)
ENABLE_P2P_CHECK Enable P2P check for GPU access false boolean (true or false)
TRITON_ATTENTION_REDUCE_IN_FP32 Cast Triton attention reduce op to FP32 false boolean (true or false)
TOOL_CALL_PARSER Defines the parser used to interpret responses "apertus2509", "cohere_command4", "deepseekv3", "deepseekv31", "deepseekv32", "deepseekv4", "gemma4", "gigachat3", "glm", "glm45", "glm47", "gpt-oss", "hermes", "hunyuan", "inkling", "interns1", "kimi_k2", "kimi_k3", "lfm2", "llama3", "mimo", "minicpm5", "minimax-m2", "minimax-m3", "mistral", "poolside_v1", "pythonic", "qwen", "qwen25", "qwen3_coder", "step3", "step3p5", "trinity"
REASONING_PARSER Defines the parser used for reasoning traces "apertus2509", "cohere_command4", "deepseek-r1", "deepseek-v3", "deepseek-v4", "gemma4", "glm45", "gpt-oss", "hunyuan", "inkling", "interns1", "kimi", "kimi_k2", "kimi_k3", "mimo", "minimax", "minimax-append-think", "minimax-m3", "mistral", "nemotron_3", "poolside_v1", "qwen3", "qwen3-thinking", "step3", "step3p5"

Tool/Function Calling and Reasoning

  • Tool/Function calling: Set the TOOL_CALL_PARSER environment variable to match your model family. See the TOOL_CALL_PARSER row above for the full list of supported values. If unset, this worker does not pass --tool-call-parser to SGLang.

    • Example (docker-compose): add TOOL_CALL_PARSER=llama3 under environment:.
    • Example (RunPod Hub): set the TOOL_CALL_PARSER env var in the UI.

  • Reasoning: Set the REASONING_PARSER environment variable to match your model family if you want to enable reasoning traces parsing. See the REASONING_PARSER row above for the full list of supported values. If unset, this worker does not pass --reasoning-parser to SGLang.

    • Example (docker-compose): add REASONING_PARSER=qwen3 under environment:.
    • Example (RunPod Hub): set the REASONING_PARSER env var in the UI.

API Usage

This worker supports two API formats: RunPod native and OpenAI-compatible.

RunPod Native API

For testing directly in the RunPod UI, use these examples in your endpoint's request tab.

Chat Completions

{
  "input": {
    "messages": [
      { "role": "system", "content": "You are a helpful assistant." },
      { "role": "user", "content": "What is the capital of France?" }
    ],
    "max_tokens": 100,
    "temperature": 0.7
  }
}

Chat Completions (Streaming)

{
  "input": {
    "messages": [
      { "role": "user", "content": "Write a short story about a robot." }
    ],
    "max_tokens": 500,
    "temperature": 0.8,
    "stream": true
  }
}

Native Text Generation

For direct SGLang text generation without OpenAI chat format:

{
  "input": {
    "text": "The capital of France is",
    "sampling_params": {
      "max_new_tokens": 64,
      "temperature": 0.0
    }
  }
}

List Models

{
  "input": {
    "openai_route": "/v1/models"
  }
}

Choosing the HTTP verb

With openai_route, the verb is inferred from the payload: a request carrying openai_input is sent as a POST, one without it as a GET. That way read-only routes such as /v1/models and /health, which SGLang registers as GET-only, work without spelling anything out.

Pass method to override it:

{
  "input": {
    "openai_route": "/v1/some-route",
    "method": "POST",
    "openai_input": {}
  }
}

OpenAI-Compatible API

For external clients and SDKs, use the /openai/v1 path prefix with your RunPod API key.

Chat Completions

Path: /openai/v1/chat/completions

{
  "model": "meta-llama/Meta-Llama-3-8B-Instruct",
  "messages": [
    { "role": "system", "content": "You are a helpful assistant." },
    { "role": "user", "content": "What is the capital of France?" }
  ],
  "max_tokens": 100,
  "temperature": 0.7
}

Chat Completions (Streaming)

{
  "model": "meta-llama/Meta-Llama-3-8B-Instruct",
  "messages": [
    { "role": "user", "content": "Write a short story about a robot." }
  ],
  "max_tokens": 500,
  "temperature": 0.8,
  "stream": true
}

List Models

Path: /openai/v1/models

{}

Response Format

Both APIs return the same OpenAI-shaped completion. The queue API (/run, /runsync) wraps it in the standard job envelope, under output:

{
  "id": "...",
  "status": "COMPLETED",
  "output": [
    {
      "choices": [
        {
          "index": 0,
          "message": { "role": "assistant", "content": "Paris." },
          "finish_reason": "stop"
        }
      ],
      "usage": {
        "prompt_tokens": 9,
        "completion_tokens": 1,
        "total_tokens": 10
      }
    }
  ]
}

The OpenAI-compatible path (/openai/v1/...) returns the completion directly, so the official openai SDK can consume it unchanged.

If the request fails upstream, the job is marked FAILED and carries the SGLang status code and body:

{
  "status": "FAILED",
  "error": "SGLang returned HTTP 400: temperature must be a non-negative finite number, got -99.0."
}

Usage

Below are minimal python snippets so you can copy-paste to get started quickly.

Replace <ENDPOINT_ID> with your endpoint ID and <API_KEY> with a RunPod API key.

OpenAI compatible API

Minimal Python example using the official openai SDK:

from openai import OpenAI
import os

# Initialize the OpenAI Client with your RunPod API Key and Endpoint URL
client = OpenAI(
    api_key=os.getenv("RUNPOD_API_KEY"),
    base_url=f"https://api.runpod.ai/v2/<ENDPOINT_ID>/openai/v1",
)

Chat Completions (Non-Streaming)

response = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3-8B-Instruct",
    messages=[{"role": "user", "content": "Give a two lines on Planet Earth ?"}],
    temperature=0,
    max_tokens=100,

)
print(f"Response: {response}")

Chat Completions (Streaming)

response_stream = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3-8B-Instruct",
    messages=[{"role": "user", "content": "Give a two lines on Planet Earth ?"}],
    temperature=0,
    max_tokens=100,
    stream=True

)
for response in response_stream:
    print(response.choices[0].delta.content or "", end="", flush=True)

Compatibility

Anything not recognized by worker-sglang is forwarded verbatim to /generate, so advanced options in the SGLang docs (logprobs, sessions, images, etc.) also work.

About

SGLang is fast serving framework for large language models and vision language models.

Resources

Readme
MIT license

Contributing

Contributing
Activity
Custom properties

Stars

37 stars

Watchers

0 watching

Forks

34 forks
Report repository

Releases

Packages

Used by

Contributors

Languages

Generated from runpod-workers/worker-template