feat(holo-3.1): 256K long-context serving recipes (35B-A3B NVFP4 + 9B dense)#1
Draft
rsafier wants to merge 6 commits into
Draft
feat(holo-3.1): 256K long-context serving recipes (35B-A3B NVFP4 + 9B dense)#1rsafier wants to merge 6 commits into
rsafier wants to merge 6 commits into
Conversation
… dense) Best throughput/latency at a 256K context window on a single GB10 (sm_121), live-validated on gx10 2026-07-11 against a tip Atlas build. - holo-3.1-35b-a3b-nvfp4: qwen3_5_moe, kernel target holo-3.1-35b-a3b. bf16 KV (1.4-2x faster prefill than fp8, coherent at 256K depth), ATLAS_KV_OVERCOMMIT=1 (on-demand paged KV; required at 256K), gpu-mem-util 0.72 (~996K KV tokens w/ ~27GB co-tenants), SLAI @100Ms TBT, ssm-cache-slots 256, lm-head left default (checkpoint ships NVFP4 head). - holo-3.1-9b-bf16: dense qwen3_5 (hidden 4096) served by the ornith-1.0-9b kernel target (no holo-3.1-9b kernel dir); bf16 weights + bf16 KV; 256K fits within its 262144 max_position_embeddings; dense => no SSM slots. Mirrors the qwen3.6 recipe_version:2 declarative structure. Build the serve-anything artifact with ATLAS_TARGET_MODEL='*' and --no-default-features --features cuda (drop default nccl on single GB10); required env + build/run flags documented in the description prose, as recipe_version:2 has no declarative env slot.
Add the complete prefill-accelerator flag cluster validated live 2026-07-11 on GB10 (gx10), Holo-3.1-35B-A3B-NVFP4, single-request WARM prefill: 4K=4975, 8K=5489, 16K=5208 tok/s (~10x the ~500-600 the partial config delivered). - 35B: full ATLAS_HOLO_* MoE + GDN/SSM + prefill-accelerator env stack, build-inside-container requirement (build.rs gates fused-MoE kernel on CUTLASS_HOME), and declarative defaults (max_num_seqs/max_batch_size/ max_prefill_tokens 16384, ssm_cache_slots 128, checkpoint_interval 16, max_model_len 262144). - 9B: same generic prefill + GDN/SSM + KV/GEMM env stack minus the Holo-only MoE flags (dense), kernel target ornith-1.0-9b.
FlashInfer GDN (=1) corrupts >=128K context: 7/8 needle FAIL at 200K per MonumentalSystems/atlas PR #22 LONGCTX-NEEDLE-BENCHMARK (F32 output/inverse do not fix it). FLA in-tree GDN (=0) is 8/8 PASS. These are 256K recipes, so correctness wins. - Flip ATLAS_GDN_FLASHINFER 1 -> 0 in env list + docker run for 35B and 9B. - Add long-context correctness note to both metadata.description blocks. - 35B headline perf updated to the =0 (correct) values: 4K=3518, 8K=3759, 16K=3569 tok/s; keep =1 (4975/5489/5208) as a labeled short-context-only fast variant.
…NGLE_SCALE Hcompany/Holo-3.1-35B-A3B-FP8 is compressed-tensors float-quantized, strategy:block (weight_scale [N/128,K/128]). Loads native (Fp8BlockScaled, native FP8 MoE). CRITICAL: ATLAS_FP8_SINGLE_SCALE=1 disables block-scaled prefill -> 14x drift -> agentic write-path drift (0/2 webserver_ok w/ flag, path_drift=0 and passing without). NVFP4-only flag; drop it here. Validated 2026-07-11 GB10, ~73 tok/s. lm-head default, no speculative (no mtp).
…r_ok+followed_dir) Holo-3.1-35B-A3B-FP8 + --lm-head-dtype fp8 (per-row calib, REQUIRES main #301) + ATLAS_LOOP_NO_SUPPRESS=1 + no ATLAS_FP8_SINGLE_SCALE + memory-tuned. Scored PERFECT 10/10 webserver_ok AND 10/10 followed_directions on the opencode agentic harness (2026-07-12 GB10), beating NVFP4-head, bf16-head, pre-#301 fp8-head, and vLLM+DFlash (7/10) on task completion. ~70 tok/s decode, ~1150-1780 tok/s cold prefill (memory-tuned chunk), ~84 GB stable.
…, not FP8 A/B on identical stack (same 8K chunk): NVFP4 5547/5637/4704 vs FP8 block-scaled 1150/1220/1780 tok/s. Block-scaled FP8 MoE prefill is ~3-5x slower; raising the chunk does NOT reach NVFP4 levels. FP8 edge is fp8-head fidelity, not speed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Holo-3.1 serving recipes — best throughput/latency at 256K context
Adds
recipes/holo-3.1/with two single-GB10 (sm_121) recipes, modeled on theexisting
qwen3.6recipe_version: "2"declarative templates. Live-validated ongx10 2026-07-11 against a tip Atlas build in
atlas-gb10:b12x-ready.Recipes
holo-3.1-35b-a3b-nvfp4Hcompany/Holo-3.1-35B-A3B-NVFP4holo-3.1-35b-a3bholo-3.1-9b-bf16Hcompany/Holo-3.1-9Bornith-1.0-9bThe 256K long-context config (validated)
--max-seq-len 256000— both models reportmax_position_embeddings=262144, so 256K fits with headroom (9B needs no cap).--kv-cache-dtype bf16— 1.4-2x faster prefill than fp8 and stays coherent at depth; fp8/nvfp4 KV degrade long-context. Mandatory for these recipes.ATLAS_KV_OVERCOMMIT=1(env) — required at 256K. A 256K seq needs ~16000 KV blocks; without overcommit the server refuses to start unless the pool can reservemax_batch_size × 256K. Overcommit does on-demand paged allocation, somax_batch_size 8boots while a seq still grows to 256K.--gpu-memory-utilization 0.72— safe with ~27 GB co-tenants (~19 GB KV pool → ~996K KV tokens, ~3.9 concurrent 256K seqs). Raise to 0.80-0.85 on a dedicated box.--scheduling-policy slai --tbt-deadline-ms 100,--ssm-cache-slots 256(35B only; 9B dense has no SSM snapshot),--bind 0.0.0.0 --port 8888.--lm-head-dtype bf16is ignored.Build (serve-anything artifact)
ATLAS_TARGET_MODEL='*'builds all kernels so onesparkbinary serves any Holo-3.1 combo (a single-target build is "missing more kernels than expected"). Drop the default-onncclfeature on a single GB10 (multi-GPU EP/TP only; libnccl not in the single-node link path).Structure fidelity
Mirrors the qwen3.6
recipe_version: "2"schema exactly (model/runtime/container/max_nodes+metadata+ declarativedefaults).recipe_version: "2"has no declarative env slot (no v2 recipe uses one;qwen3.6-27b-nvfp4-prefill-recorddocuments its env stack in prose), so the requiredATLAS_KV_OVERCOMMIT=1,HF_HUB_CACHE,--tbt-deadline-ms, and the build/run flags are documented in themetadata.descriptionprose the same way.containeris set toatlas-gb10:b12x-ready(the validated image on the box) rather than a publicavarok/atlas-gb10:*tag.Draft: opening for review of the recipe shape before we mark ready.