Streamweld

Durable token streams for self-hosted LLM inference.

Website | Documentation | Get started | Architecture | Discussions

Streamweld is an OpenAI-compatible durability layer for self-hosted LLM inference. It gives each streaming generation an identity, an append-only journal, and an exact resume cursor that can outlive a reader connection or backend attempt.

Important

The current release is v0.1.0. APIs and v1alpha1 Kubernetes resources may change during the 0.x release series.

Install

Install the proxy and operator into Kubernetes with Helm:

helm upgrade --install streamweld oci://ghcr.io/satwiksps/charts/streamweld \
  --namespace streamweld-system \
  --create-namespace \
  --version 0.1.0 \
  --wait --timeout 3m

Install the dependency-free TypeScript client, or the Vercel AI SDK v5 adapter:

npm install @streamweld/client
npm install @streamweld/ai-sdk ai@^5

Prebuilt streamweld-proxy, streamweld-operator, and streamweldctl archives for Linux, macOS, and Windows are available from GitHub Releases.

Why Streamweld

Long LLM responses are vulnerable to pod failures, rolling updates, reclaimed nodes, proxy timeouts, and client network changes. A normal retry restarts the request and can discard generated output. Streamweld makes three operations explicit:

• Reader resume: replay after an exact Last-Event-ID, then join the live tail.
• Producer migration: continue on a compatible backend only when every safety gate passes.
• User stop: cancel generation explicitly; a disconnected reader is not treated as stop.

Architecture

flowchart LR
    Client[OpenAI-compatible client] -->|HTTP + SSE · exact resume cursor| Proxy[Streamweld proxy]
    Proxy -->|OpenAI-compatible request| Pool[Inference pool<br/>vLLM · SGLang · TGI]
    Proxy -->|commit and replay| Journal[(Memory or Redis journal)]
    Operator[Kubernetes operator<br/>routes · policy · drain] -->|route snapshots| Proxy

Loading

The Go proxy owns the request and streaming data path. The journal owns ordered events, replay, and terminal state. The Kubernetes operator manages eligible backends and rollout draining without reading prompts or generated text.

Run from source

Requirements: Go 1.25+, Node.js 20.19+, and pnpm 11.19.

git clone https://github.com/satwiksps/streamweld.git
cd streamweld
make bootstrap
make test

Run the CPU-only Kubernetes end-to-end path with Docker, kind, kubectl, and Helm installed:

make e2e

See the ten-minute guide for a complete installation and recovery walkthrough.

Safety boundary

Migration is deliberately conservative. The target must pass model, tokenizer, chat-template, request-shape, token-budget, structured-output, tool-call, and terminal-state checks. Unsafe continuation is refused instead of silently returning a corrupted answer.

Use streamweldctl doctor to probe a specific immutable backend/model/tokenizer tuple before enabling migration:

streamweldctl doctor --backend http://127.0.0.1:8000 --model MODEL --json

Evidence

The committed deterministic-local profile checks complete output across 9 deterministic fault scenarios. It is an in-process correctness model, not a Kubernetes or GPU benchmark. The scheduled kind matrix is the physical failure gate. See the results and methodology.

Documentation

• Durability guarantees
• Architecture
• Resume and stop protocol
• Production operations
• TypeScript integration

Contributing

Read CONTRIBUTING.md before opening a pull request. Security issues should follow SECURITY.md, not the public issue tracker.

Apache License 2.0. See LICENSE.