refactor(loader): gate batched-prefill MLA exclusion on capability, not model_type#285
refactor(loader): gate batched-prefill MLA exclusion on capability, not model_type#285rsafier wants to merge 1 commit into
Conversation
[BETA] Automated PR review — SeedSource AI pipelineAdvisory only, not a merge gate. AI-generated (internal risk-triage, Atlas serve gate, GLM-5.2 review), attributed per Atlas convention. Triage
Serve gate
GLM-5.2 code reviewCorrectness: Sound. Risk: Low. The refactor is narrowly scoped to one eligibility gate. The only latent risk: if a future MLA model has Quality concerns:
Verdict: Approve with the test gap addressed. Pipeline: build+serve on GB10, GLM-5.2 via NVIDIA NIM, internal risk-triage classifier. External diff treated as untrusted input. Public repo. |
cb79d49 to
fdc6bba
Compare
dbec4e6 to
63ca929
Compare
fdc6bba to
6a48210
Compare
…ot model_type
check_kernel_batched_eligible() blocked batched-prefill attention for MLA models
via `config.model_type == "mistral"` — a hardcoded proxy the author flagged as
conservative ("mistral is the only MLA model in Atlas today"). Key it on the
actual MLA capability instead: `config.kv_lora_rank > 0` (== AttentionType::Mla).
- signature: `model_type: &str` -> `is_mla: bool`; wrapper passes
`self.config.kv_lora_rank > 0` (the exact derivation of
capabilities().attention_type == Mla; read directly to avoid rebuilding the
full ModelCapabilities struct on this per-prefill-batch hot path).
- tests updated to pass the bool.
Behavior-preserving on every shipped model: kv_lora_rank>0 == {mistral,
deepseek_v4}; all other model_types have kv_lora_rank=0 -> is_mla=false ->
unchanged. deepseek_v4 is the only newly-matched model, but it is ALREADY
rejected one check later by `head_dim > 256` (head_dim=512), so its outcome
(not batched-eligible) is identical. Also future-proofs any MLA checkpoint
shipped with head_dim<=256. Unit tests (batch_kernel_tests) pass.
Stacked on #260 (fix/generic-nvfp4-moe-gate). NOTE: an exhaustive adversarial
pass over the OTHER model_type gates found this is the ONLY additional safe
architecture-generic conversion — notably the MTP allowlist (preflight.rs) must
NOT be converted to has_mtp (broader than intent: would re-enable the
MiniMax/DeepSeek per-module-MTP path the guard rejects). See PR description.
63ca929 to
9343daf
Compare
Stacked on #260 (base
fix/generic-nvfp4-moe-gate). Unit tests green; the e2e co-dispatch smoke is still outstanding.What
check_kernel_batched_eligible()blocked batched-prefill attention for MLA models viaconfig.model_type == "mistral"— a hardcoded proxy the original author flagged as conservative ("mistral is the only MLA model in Atlas today"). This keys it on the actual capability:config.kv_lora_rank > 0(==AttentionType::Mla).Behavior preservation
kv_lora_rank > 0=={mistral, deepseek_v4}; every other model_type haskv_lora_rank == 0→is_mla=false→ unchanged. deepseek_v4 is the only newly-matched model, but it is already rejected one check later byhead_dim > 256(head_dim=512) — so its outcome (not batched-eligible) is identical. Net: bit-identical on every shipped model, plus it future-proofs any MLA checkpoint shipped withhead_dim <= 256.batch_kernel_testspass (12/12) — they directly exercise the converted predicate (MLA reject, head_dim reject, non-MLA accept).Why this is the only additional conversion
This PR is the output of an exhaustive adversarial pass over every
model_typestring gate in the tree (stacking on #260, which already did the twoqwen35/load_layers.rsMoE gates). The analysis found this is the only other safe architecture-generic conversion. In particular it rejected several tempting ones:preflight.rsMTP_SUPPORTED_MODEL_TYPES) → must NOT becomecapabilities().has_mtp:has_mtp(mtp_num_hidden_layers>0) is broader than the allowlist and is true for deepseek_v4 / minimax_m2 / step3p7 — converting would re-enable exactly the unconsumable per-module-MTP loader path (MiniMax) the guard exists to reject.is_qwen35*predicates → loader sub-dispatch (protected; no capability expresses "use dense vs VL loader").skip_lm_head_quantization/ softcap, minimax/step3p7 setup, thedispatch.rsmodel_type→kernel-dir rewrites → genuinely checkpoint/family-specific, kept.No kernels move; no loader dispatch changes. The refactor is deliberately minimal — the value is as much the documented boundary of what stays model_type-gated as the one conversion itself.