Skip to content

Pin ComfyUI to 0.29.0, build with CUDA 12.8, expand allowed CUDA versions to 12.8–13.3 - #231

Merged
Madiator2011Work merged 4 commits into
runpod-workers:mainfrom
lukepiette:pin-comfyui-cuda-12.8
Aug 5, 2026
Merged

Pin ComfyUI to 0.29.0, build with CUDA 12.8, expand allowed CUDA versions to 12.8–13.3#231
Madiator2011Work merged 4 commits into
runpod-workers:mainfrom
lukepiette:pin-comfyui-cuda-12.8

Conversation

@lukepiette

@lukepiette lukepiette commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

What

  • Pin ComfyUI to 0.29.0 (latest stable release, 2026-07-29) instead of latest, so builds are reproducible and don't silently change between deploys. comfy-cli is also pinned (1.13.0) for the same reason.
  • Build with CUDA 12.8: default base image moves from nvidia/cuda:12.6.3-cudnn-runtime-ubuntu24.04 to nvidia/cuda:12.8.1-cudnn-runtime-ubuntu24.04 (Dockerfile defaults + docker-bake.hcl globals).
  • Pin the RUNTIME PyTorch to 2.11.0+cu128: the venv that actually runs ComfyUI (/opt/venv) previously got its torch from default PyPI via the requirements mirror step — and PyPI torch ≥ 2.11 ships CUDA 13 builds (nvidia-*-cu13 deps) that require driver ≥ 580. That's incompatible with hosts advertising CUDA 12.8/12.9 (driver 570/575) and is likely the source of the current torch-related worker failures. Installing torch==2.11.0+cu128 before the mirror step satisfies ComfyUI's bare torch requirement, so the runtime torch is now a known cu128 build that runs on driver ≥ 570.
  • .runpod/hub.json: allowedCudaVersions updated from ["12.7", "12.6"] to ["12.8", "12.9", "13.0"]. The disabled .runpod/tests_.json is synced to the same list.
  • Changeset added (minor) so the change ships with the next release cut.

Validated on real hosts (2026-07-30)

This exact build (base target, PR Dockerfile defaults) was pushed to a test registry and run as a serverless endpoint pinned to each CUDA version, one job per host type:

Host CUDA GPU Result
12.8 RTX 4090 ✅ COMPLETED — worker healthy (GPU pre-flight passed), workflow executed, image returned (exec 3.0s)
12.9 RTX 5090 / H200 ⚠️ Could not schedule — no 12.9 host became available during the test window (capacity, not compatibility; cu128 requires driver ≥ 570, which 12.9 hosts satisfy by definition). Kept in the allowlist — harmless, and correct when 12.9 hosts appear.
13.0 RTX 4090 ✅ COMPLETED — worker healthy, workflow executed, image returned (exec 3.1s)

Additional validation notes:

  • The allowlist ceiling of 13.0 matches Runpod's API schema exactly: the platform's CUDA version enum currently ends at "13.0" — values above it don't exist yet, so this list covers every currently-valid version ≥ 12.8. When the platform adds 13.1+, extend the list (or switch to minCudaVersion if hub.json gains support for it — floor-only semantics avoid this ceiling maintenance entirely).
  • The old 12.6/12.7 list is nearly dead capacity-wise (zero community-cloud 4090 stock, Low secure on 12.7 only); current ADA_24 stock is concentrated on 12.8 and 13.0 hosts, so this change is a strict capacity improvement and unlocks community cloud.
  • CUDA 12.8 + cu128 torch also unlocks Blackwell (sm_120 / RTX 5090) support.

Known limitations / follow-ups

  • comfy --install's workspace venv (/comfyui/.venv) still receives its own unused torch (~5–7 GB installed). Removing it is a follow-up image-size win, left out here to keep this change low-risk.
  • The base-cuda12-8-1 bake target now produces nearly the same image as base (both end with a cu128 runtime torch). Left untouched since release.yml/manual-build-all.yml reference it by name; consolidating is a follow-up.
  • Existing Hub endpoints keep the old release and its exhausted 12.6/12.7 host pool — release notes should tell users to update their endpoint.

Verification (build-time)

  • torch==2.11.0+cu128 / torchvision==0.26.0+cu128 / torchaudio==2.11.0+cu128 cp312 manylinux wheels confirmed present on the cu128 index (it caps at 2.11.0 — newer torch is cu129/cu13x only).
  • ComfyUI 0.29.0's requirements verified compatible with the existing transformers>=4.50.3,<5 / huggingface-hub<1.0 pins (DR-1170); --quick-test-for-ci --cpu flags still exist in 0.29.0, so the build-time smoke test keeps catching startup-breaking deps.
  • Pinning --version 0.29.0 resolves the tag by string construction in comfy-cli — no GitHub API call, removing the rate-limit risk that latest resolution carries.

🤖 Generated with Claude Code

lukepiette and others added 3 commits July 30, 2026 17:22
…ions

- Pin COMFYUI_VERSION to 0.29.0 (latest stable) instead of latest for reproducible builds
- Move default base image to nvidia/cuda:12.8.1-cudnn-runtime-ubuntu24.04
- Install ComfyUI with --cuda-version 12.8 so comfy-cli pulls current PyTorch from the cu128 index
- Allow CUDA 12.8-13.3 hosts in hub.json (driver >= 570 required by cu128 wheels; newer drivers are backward compatible)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…, add changeset

The venv that actually runs ComfyUI (/opt/venv) got torch from default
PyPI via the requirements mirror step - and PyPI torch >= 2.11 ships CUDA 13
builds (nvidia-*-cu13 deps) requiring driver >= 580. That crashes on hosts
advertising CUDA 12.8/12.9 (driver 570/575), which hub.json now allows.
Installing torch 2.11.0+cu128 first satisfies ComfyUI's bare torch
requirement so the PyPI pass leaves it alone; cu128 runs on driver >= 570,
i.e. every allowed host.

Also pin comfy-cli (reproducibility), sync the disabled tests_.json CUDA
allowlist so it can't reintroduce the dead 12.6/12.7 pool if re-enabled,
and add a changeset since releases are cut via changesets.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
12.8 and 13.0 are where ADA_24 stock actually is today, and 13.0 is the
highest version we can empirically validate. Extend the list as newer
host CUDA versions are verified.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@Chmokachka Chmokachka left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Madiator2011Work
Madiator2011Work merged commit 8ec7aea into runpod-workers:main Aug 5, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants