feat: add tract_moe_ffn operator for Mixture-of-Experts FFN by JulienBalianSonos · Pull Request #2084 · sonos/tract

JulienBalianSonos · 2026-03-30T08:56:16Z

Implements the tract_moe_ffn operator in the tract_transformers extension, enabling inference of routed Mixture-of-Experts FFN blocks exported via torch_to_nnef.

The operator encapsulates the full MoE FFN block:

Router: x @ wg.T followed by top-k expert selection.
Gating modes:
- softmax over the selected top-k experts
- softmax over all experts, then gather top-k without renormalization
- raw / already-normalized top-k weights
Token grouping: batch selected tokens per expert for efficient GEMM.
Expert FFN: activation(x @ w1) * (x @ w3) followed by @ w2 for SwiGLU-style experts.
Weighted scatter-add of expert outputs.

Real conditional compute: unused experts are fully skipped. The op handles both 2D [T,D] and 3D [B,S,D] input shapes.

Model Coverage

The implemented semantics cover the routed FFN variants used by:

Granite / Mixtral-style MoE: top-k routing, softmax-over-top-k gating, SwiGLU experts.
OLMoE-style MoE: router softmax over all experts, gather top-k, no top-k renormalization.
OpenAI GPT-OSS-style MoE: optional router/expert biases and clamped SwiGLU.
Qwen3 / Qwen3.5-style MoE: top-k routing with configurable top-k probability normalization.

Current validation:

Bit-exact against PyTorch on TitanML/tiny-mixtral (8 experts, top-2, 246M params).
End-to-end NNEF regression on a sanitized Qwen3 tiny MoE graph.
Full-model local validation on a Granite MoE GPTQ/Q40 export with Q40 experts and embeddings, keeping routing in full precision.

Granite MoE Benchmark

Local release measurements on Apple Silicon. The benchmark artifact is not included in this PR.

Model / path	Backend	Prompt / decode	KV path	Throughput
Granite MoE GPTQ/Q40 experts + embeddings	Metal	5 / 64	unfolded	`77.68 tok/s`
Granite MoE GPTQ/Q40 experts + embeddings	Metal	5 / 64	stateful	`78.66 tok/s`
Granite MoE GPTQ/Q40 experts + embeddings	Metal	51 / 128	unfolded	`73.86 tok/s`
Granite MoE GPTQ/Q40 experts + embeddings	Metal	51 / 128	stateful	`74.47 tok/s`
Granite MoE GPTQ/Q40 experts + embeddings	Metal	51 / 512	unfolded	`65.04 tok/s`
Granite MoE GPTQ/Q40 experts + embeddings	Metal	51 / 512	stateful	`65.35 tok/s`
Granite MoE GPTQ/Q40 experts + embeddings	CPU	51 / 128	unfolded	`40.63 tok/s`

Additional steady-state Metal harness results on the same model family:

Variant	PP512	TG128
Q40 MoE no-SDPA reference	`1280.3 tok/s`	`123.5 tok/s`
mixed SDPA, f16 input / f32 accumulation	`1254.6 tok/s`	`122.0 tok/s`
old SDPA f32-boundary casts	`1128.2 tok/s`	`119.4 tok/s`

Implements the tract_moe_ffn operator in the tract_transformers extension, enabling inference of MoE-based models (Mixtral, GPT-OSS, Qwen MoE) exported via torch_to_nnef. The operator encapsulates the full MoE FFN block: - Router: x @ wg.T -> top-k expert selection with softmax gating - Token grouping: batch tokens per expert for efficient GEMM - Expert FFN: SwiGLU (silu(x@w1) * (x@w3)) @ w2 with BLAS-backed matmul - Weighted scatter-add of expert outputs Real conditional compute: unused experts are fully skipped. Handles both 2D [T,D] and 3D [B,S,D] input shapes. Verified bit-exact against PyTorch on TitanML/tiny-mixtral (8 experts, top-2, 246M params).

…clamped-swiglu act_alpha/act_limit; codegen keeps biased/clamped path on reference eval)

…n (no router_logits, no rank-align unsqueeze)

…ax_all|sigmoid|raw); fix stale op unit tests (single-output + gate); regen harness asset

… native dtype, cast output back); compute router in f32 to stabilize top-k selection vs PyTorch; f16-faithful weighted scatter

…rch CPU f32 matmul accumulation; native-f16 accumulation derailed greedy decoding

github-actions · 2026-06-30T11:04:41Z

🔴 Bench vs main — 1 speed regression(s)

Reference: main nightly, latest 2026-07-09 (0d old) · PR dfd2901db · ran on apple-m1-max, i9-11900kb_rtx-4060, jetson-orin-nx · 791 metrics compared

Speed — evaltime · prefill · decode

Δ	metric	device	main → PR
🔴 -4.8%	openelm_270M_q40ef16_541 _{decode · cpu}	`apple-m1-max`	79.54 tok/s → 75.75 tok/s

lower is better except prefill/decode (tok/s) · adaptive thresholds (max(floor, k×noise) vs the series' own history) · single-shot vs nightly reference · full report → run

The routed Q40 path handles the fully block-quantized case; models that fall back to per-expert subplans still went through them one at a time, both when building and when evaluating. Build: each plan slices its own expert out of the shared weight tensors and optimizes an independent sub-model, which includes prepacking that expert's weights. Serially that is layers x experts optimizations on the model-load critical path, 768 of them for a 24-layer 32-expert model (measured 62s). Spread them over rayon's global pool, as FlashSdpa already does for its heads. Eval: a prompt pass with top-k routing lights up nearly every expert in every layer, and each expert's matmuls are too narrow to fill the machine alone. Run the active experts concurrently and keep the weighted scatter serial and in fixed expert order so the reduction stays deterministic. `OpState` is deliberately not `Send`, so experts can no longer share long-lived pre-spawned states. Expert sub-models are stateless matmuls, so spawn per eval and drop the per-expert states from the op state entirely; only the router keeps one. Measured on a 20B gpt-oss-style export on the subplan path: prepare 62s -> 13s, decode 4.9 -> 7.3 tok/s, output byte-identical.

build_expert_plan upcast every non-block-quant expert weight to f32. For a mixed-precision export that weight is the bulk of the model: gpt-oss keeps the down projection in f16, about 12.7GB of a 22GB export, so the upcast doubled it and cost both prepare time and decode bandwidth. Keep f16 weights as they were exported; the matmul still accumulates in f32. Measured on a 20B gpt-oss export, subplan path: prepare 20.9s -> 12.1s, prefill 7.0s -> 2.8s, decode 5.6 -> 9.0 tok/s, output unchanged.

github-actions · 2026-07-28T17:45:41Z

🔴 Bench vs main — 40 speed regression(s) · 12 load/memory

Reference: main nightly, latest 2026-07-28 (0d old) · PR 0343bb678 · ran on apple-m1-max, i9-11900kb_rtx-4060, jetson-orin-nx · 727 metrics compared

Speed — evaltime · prefill · decode

Δ	metric	device	main → PR
🔴 +379.7%	parakeet_tdt_600m_v3_f32f32_preprocessor_1s _{evaltime · cpu}	`apple-m1-max`	1.06 ms → 5.09 ms
🔴 +370.3%	parakeet_tdt_600m_v3_f32f32_preprocessor_1s _{evaltime · cpu}	`i9-11900kb_rtx-4060`	0.806 ms → 3.79 ms
🔴 +215.5%	en_tdnn_pyt_15M _{evaltime · pulse_120ms}	`i9-11900kb_rtx-4060`	2.95 ms → 9.29 ms
🔴 +189.9%	parakeet_tdt_600m_v3_f32f32_encoder_1s _{evaltime · cpu}	`i9-11900kb_rtx-4060`	241 ms → 698 ms
🔴 +142.3%	en_tdnn_pyt_15M _{evaltime · pulse_120ms}	`apple-m1-max`	1.69 ms → 4.1 ms
🔴 +102.2%	en_tdnn_15M _{evaltime · 2600ms}	`i9-11900kb_rtx-4060`	46.9 ms → 94.9 ms
🔴 +73.5%	parakeet_tdt_600m_v3_f32f32_encoder_1s _{evaltime · cpu}	`apple-m1-max`	124 ms → 216 ms
🔴 +59.6%	en_tdnn_15M _{evaltime · pulse_120ms}	`apple-m1-max`	2.36 ms → 3.76 ms
🔴 +59.3%	dummy_conmer_12M _{evaltime · pulse_120ms}	`apple-m1-max`	1.97 ms → 3.13 ms
🔴 +51.0%	en_tdnn_8M _{evaltime · pulse_120ms}	`apple-m1-max`	1.2 ms → 1.82 ms
🔴 +50.1%	en_tdnn_8M _{evaltime · 2600ms}	`i9-11900kb_rtx-4060`	22.9 ms → 34.3 ms
🔴 +47.8%	trunet _{evaltime · pulse1_f16}	`apple-m1-max`	0.291 ms → 0.429 ms
🔴 +46.9%	hey_snips_v4_model17 _{evaltime · 2sec}	`apple-m1-max`	0.981 ms → 1.44 ms
🔴 +38.1%	mobilenet_v1_1 _{evaltime · pass}	`i9-11900kb_rtx-4060`	35.7 ms → 49.2 ms
🔴 +31.6%	en_tdnn_8M _{evaltime · pulse_120ms}	`i9-11900kb_rtx-4060`	1.94 ms → 2.55 ms
🔴 +28.3%	en_tdnn_15M _{evaltime · pulse_240ms}	`i9-11900kb_rtx-4060`	6.01 ms → 7.71 ms
🔴 +28.1%	en_tdnn_15M_nnef _{evaltime · pulse_240ms}	`i9-11900kb_rtx-4060`	5.9 ms → 7.55 ms
🔴 +27.7%	en_tdnn_8M _{evaltime · pulse_180ms}	`apple-m1-max`	1.7 ms → 2.17 ms
🔴 +25.3%	inceptionv3 _{evaltime · pass}	`i9-11900kb_rtx-4060`	154 ms → 192 ms
🔴 +20.2%	mobilenet_v2_1 _{evaltime · pass}	`i9-11900kb_rtx-4060`	54.6 ms → 65.6 ms
🔴 +19.8%	arm_ml_kws_cnn_m _{evaltime · pass}	`apple-m1-max`	0.123 ms → 0.148 ms
🔴 +19.5%	en_tdnn_15M _{evaltime · pulse_240ms}	`apple-m1-max`	4.04 ms → 4.82 ms
🔴 +18.2%	trunet _{evaltime · pulse1_f32}	`apple-m1-max`	0.164 ms → 0.194 ms
🔴 +16.5%	speaker_id _{evaltime · pulse8}	`i9-11900kb_rtx-4060`	0.0624 ms → 0.0727 ms
🔴 +15.4%	hey_snips_v1 _{evaltime · 400ms}	`i9-11900kb_rtx-4060`	0.184 ms → 0.213 ms
🔴 +15.0%	en_tdnn_15M_nnef _{evaltime · pulse_240ms}	`apple-m1-max`	4.08 ms → 4.7 ms
🔴 +11.6%	speaker_id _{evaltime · pulse8}	`apple-m1-max`	0.0596 ms → 0.0666 ms
🔴 +10.2%	parakeet_tdt_600m_v3_f32f32_joint_pass _{evaltime · cuda}	`i9-11900kb_rtx-4060`	0.0858 ms → 0.0946 ms
🔴 +9.9%	mdl_en_2019_Q3_librispeech_onnx _{evaltime · pulse_240ms}	`apple-m1-max`	0.733 ms → 0.805 ms
🔴 -9.9%	llama_3_2_1B_instruct_q40ef16_541 _{prefill · cpu}	`apple-m1-max`	84.63 tok/s → 76.25 tok/s
🔴 +9.6%	voicecom_fake_quant _{evaltime · 2sec}	`apple-m1-max`	1.5 ms → 1.64 ms
🔴 +9.6%	hey_snips_v4_model17 _{evaltime · pulse8}	`i9-11900kb_rtx-4060`	0.131 ms → 0.144 ms
🔴 +8.5%	parakeet_tdt_600m_v3_f32f32_decoder_pass _{evaltime · cuda}	`i9-11900kb_rtx-4060`	0.24 ms → 0.26 ms
🔴 +6.3%	parakeet_tdt_600m_v3_f32f32_joint_pass _{evaltime · metal}	`apple-m1-max`	0.375 ms → 0.399 ms
🔴 +6.2%	voicecom_float _{evaltime · 2sec}	`apple-m1-max`	1.3 ms → 1.38 ms
🔴 +5.9%	en_tdnn_8M _{evaltime · pulse_240ms}	`apple-m1-max`	2.21 ms → 2.33 ms
🔴 -5.8%	openelm_270M_q40ef16_516 _{prefill · cpu}	`apple-m1-max`	300.8 tok/s → 283.3 tok/s
🔴 -5.8%	openelm_270M_q40ef16_541 _{prefill · cpu}	`apple-m1-max`	296.8 tok/s → 279.7 tok/s
🔴 -4.1%	openelm_270M_q40ef16_541 _{decode · metal}	`apple-m1-max`	219.8 tok/s → 210.7 tok/s
🔴 -3.9%	llama_3_2_1B_q40ef32_516 _{prefill · cpu}	`apple-m1-max`	46.44 tok/s → 44.65 tok/s

Load & memory (worst first)

Δ	metric	device	main → PR
🔴 +14.9%	hey_snips_v4_model17 _{heap @ ready · 2sec}	`apple-m1-max`	1.93 MB → 2.21 MB
🔴 +14.6%	hey_snips_v4_model17 _{heap @ ready · pulse8}	`apple-m1-max`	1.84 MB → 2.11 MB
🔴 +14.5%	hey_snips_v4_model17_nnef _{heap @ ready · pulse8}	`apple-m1-max`	1.84 MB → 2.11 MB
🔴 +12.2%	hey_snips_v4_model17 _{heap @ ready · 2sec}	`i9-11900kb_rtx-4060`	2.13 MB → 2.39 MB
🔴 +12.1%	hey_snips_v4_model17 _{heap @ ready · pulse8}	`i9-11900kb_rtx-4060`	2.15 MB → 2.41 MB
🔴 +12.1%	hey_snips_v4_model17_nnef _{heap @ ready · pulse8}	`i9-11900kb_rtx-4060`	2.15 MB → 2.42 MB
🔴 +6.2%	openelm_270M_q40ef16_516 _{load · cuda}	`i9-11900kb_rtx-4060`	741 ms → 787 ms
🔴 +5.9%	openelm_270M_q40ef16_541 _{load · cuda}	`i9-11900kb_rtx-4060`	623 ms → 660 ms
🔴 +5.8%	speaker_id _{heap @ ready · pulse8}	`apple-m1-max`	1.06 MB → 1.13 MB
🔴 +3.9%	trunet _{heap @ ready · pulse1_f16}	`apple-m1-max`	2.41 MB → 2.5 MB
🔴 +3.4%	trunet _{heap @ ready · pulse1_f16}	`i9-11900kb_rtx-4060`	2.43 MB → 2.51 MB
🔴 +2.3%	trunet _{heap @ ready · pulse1_f32}	`i9-11900kb_rtx-4060`	4.21 MB → 4.3 MB

🟢 19 improvement(s)

Δ	metric	device	main → PR
🟢 -6.1%	parakeet_tdt_600m_v3_f32f32_encoder_1s _{evaltime · metal}	`apple-m1-max`	54.3 ms → 51 ms
🟢 -5.7%	hey_snips_v1 _{evaltime · 400ms}	`apple-m1-max`	0.13 ms → 0.123 ms
🟢 -5.0%	mdl_en_2019_Q3_librispeech_onnx _{evaltime · 2600ms}	`apple-m1-max`	5.52 ms → 5.24 ms
🟢 +3.3%	openelm_270M_q40ef16_516 _{decode · metal}	`apple-m1-max`	224.7 tok/s → 232 tok/s
🟢 -11.1%	hey_snips_v4_model17_nnef _{load+optimize · pulse8}	`apple-m1-max`	54 ms → 48 ms
🟢 -7.8%	en_tdnn_pyt_15M _{RSS @ load · pulse_120ms}	`i9-11900kb_rtx-4060`	84.5 MB → 78 MB
🟢 -7.7%	en_tdnn_15M_nnef _{load+optimize · pulse_240ms}	`apple-m1-max`	52 ms → 48 ms
🟢 -6.5%	en_tdnn_pyt_15M _{RSS @ ready · pulse_120ms}	`apple-m1-max`	118 MB → 110 MB
🟢 -6.3%	hey_snips_v1 _{RSS @ ready · 400ms}	`apple-m1-max`	22.1 MB → 20.7 MB
🟢 -6.0%	parakeet_tdt_600m_v3_f32f32_preprocessor_1s _{heap @ ready · cpu}	`i9-11900kb_rtx-4060`	675 kB → 634 kB
🟢 -5.8%	arm_ml_kws_cnn_m _{RSS @ ready · pass}	`apple-m1-max`	21.6 MB → 20.4 MB
🟢 -5.8%	parakeet_tdt_600m_v3_f32f32_encoder_1s _{heap @ ready · cpu}	`i9-11900kb_rtx-4060`	2.5 GB → 2.35 GB
🟢 -4.1%	parakeet_tdt_600m_v3_f32f32_encoder_1s _{heap @ ready · cpu}	`apple-m1-max`	2.45 GB → 2.35 GB
🟢 -3.5%	hey_snips_v1 _{heap @ ready · 400ms}	`i9-11900kb_rtx-4060`	1.19 MB → 1.15 MB
🟢 -2.7%	parakeet_tdt_600m_v3_f32f32_preprocessor_1s _{heap @ ready · cpu}	`apple-m1-max`	662 kB → 644 kB
🟢 -2.5%	mdl_en_2019_Q3_librispeech_onnx _{heap @ ready · 2600ms}	`i9-11900kb_rtx-4060`	10.4 MB → 10.2 MB
🟢 -2.3%	mobilenet_v1_1 _{heap @ ready · pass}	`i9-11900kb_rtx-4060`	18.5 MB → 18 MB
🟢 -2.2%	arm_ml_kws_cnn_m _{heap @ ready · pass}	`i9-11900kb_rtx-4060`	966 kB → 945 kB
🟢 -2.0%	en_tdnn_8M _{heap @ ready · pulse_120ms}	`i9-11900kb_rtx-4060`	34.1 MB → 33.5 MB

lower is better except prefill/decode (tok/s) · adaptive thresholds (max(floor, k×noise) vs the series' own history) · single-shot vs nightly reference · full report → run

# Conflicts: # metal/src/kernels/mod.rs # transformers/src/lib.rs # transformers/src/ops/mod.rs

…ption

github-actions · 2026-08-04T09:31:24Z

🔴 Bench vs main — 1 speed regression(s) · ⚠️ 3 secondary

Reference: 2026-08-04 morning nightly run (0d old) · full report → run

Speed — evaltime · prefill · decode

Δ	metric	device	main → PR
🔴 +3.8%	openelm_270M_q40ef16_541 _{decode · metal}	`apple-m1-max`	4.55 ms/tok _{219.8 tok/s} → 4.72 ms/tok _{211.7 tok/s}

⚠️ 3 secondary regression(s)

Δ	metric	device	main → PR
⚠️ +44.6%	arm_ml_kws_cnn_m _{load · pass}	`cortex-a9`	83 ms → 120 ms
⚠️ +33.1%	arm_ml_kws_cnn_m _{load+optimize · pass}	`cortex-a9`	127 ms → 169 ms
⚠️ +21.4%	speaker_id _{RSS @ ready · pulse8}	`cortex-a7`	17.7 MB → 21.5 MB

…place KV

github-actions · 2026-08-05T06:16:35Z

🔴 Bench vs main — 1 speed regression(s) · ⚠️ 6 secondary

Reference: 2026-08-05 morning nightly run (0d old) · full report → run

Speed — evaltime · prefill · decode

Δ	metric	device	main → PR
🔴 +9.5%	arm_ml_kws_cnn_m _{evaltime · pass}	`cortex-a9`	9.25 ms → 10.1 ms

⚠️ 6 secondary regression(s)

Δ	metric	device	main → PR
⚠️ +21.4%	arm_ml_kws_cnn_m _{load · pass}	`cortex-a9`	84 ms → 102 ms
⚠️ +16.2%	arm_ml_kws_cnn_m _{load+optimize · pass}	`cortex-a9`	130 ms → 151 ms
⚠️ +12.8%	arm_ml_kws_cnn_m _{load · pass}	`cortex-a53`	47 ms → 53 ms
⚠️ +11.2%	arm_ml_kws_cnn_m _{load · pass}	`cortex-a7`	80 ms → 89 ms
⚠️ +10.0%	hey_snips_v1 _{load · 400ms}	`cortex-a7`	70 ms → 77 ms
⚠️ +9.6%	arm_ml_kws_cnn_m _{load+optimize · pass}	`cortex-a53`	73 ms → 80 ms

github-actions · 2026-08-05T06:44:45Z

🔴 Bench vs main — 1 speed regression(s) · ⚠️ 4 secondary

Reference: 2026-08-05 morning nightly run (0d old) · full report → run

Speed — evaltime · prefill · decode

Δ	metric	device	main → PR
🔴 +5.4%	openelm_270M_q40ef16_541 _{decode · metal}	`apple-m1-max`	4.53 ms/tok _{220.8 tok/s} → 4.78 ms/tok _{209.4 tok/s}

⚠️ 4 secondary regression(s)

Δ	metric	device	main → PR
⚠️ +17.9%	arm_ml_kws_cnn_m _{load · pass}	`cortex-a9`	84 ms → 99 ms
⚠️ +13.1%	arm_ml_kws_cnn_m _{load+optimize · pass}	`cortex-a9`	130 ms → 147 ms
⚠️ +11.2%	arm_ml_kws_cnn_m _{load · pass}	`cortex-a7`	80 ms → 89 ms
⚠️ +11.0%	mobilenet_v1_1 _{RSS @ ready · pass}	`cortex-a53`	48.1 MB → 53.4 MB

github-actions · 2026-08-05T07:04:49Z

🔴 Bench vs main — 2 speed regression(s) · ⚠️ 7 secondary

Reference: 2026-08-05 morning nightly run (0d old) · full report → run

Speed — evaltime · prefill · decode

Δ	metric	device	main → PR
🔴 +9.6%	arm_ml_kws_cnn_m _{evaltime · pass}	`cortex-a9`	9.25 ms → 10.1 ms
🔴 +4.3%	openelm_270M_q40ef16_541 _{decode · metal}	`apple-m1-max`	4.53 ms/tok _{220.8 tok/s} → 4.72 ms/tok _{211.6 tok/s}

⚠️ 7 secondary regression(s)

Δ	metric	device	main → PR
⚠️ +34.5%	arm_ml_kws_cnn_m _{load · pass}	`cortex-a9`	84 ms → 113 ms
⚠️ +30.8%	arm_ml_kws_cnn_m _{load+optimize · pass}	`cortex-a9`	130 ms → 170 ms
⚠️ +11.2%	hey_snips_v4_model17 _{load+optimize · 2sec}	`cortex-a7`	5.12 s → 5.7 s
⚠️ +10.2%	hey_snips_v4_model17 _{load · 2sec}	`cortex-a7`	4.39 s → 4.84 s
⚠️ +9.7%	mobilenet_v1_1 _{RSS @ ready · pass}	`cortex-a53`	48.1 MB → 52.7 MB
⚠️ +7.6%	mobilenet_v1_1 _{load · pass}	`cortex-a9`	2.09 s → 2.25 s
⚠️ +7.1%	mobilenet_v1_1 _{load+optimize · pass}	`cortex-a9`	2.37 s → 2.54 s

github-actions · 2026-08-05T09:35:43Z

🔴 Bench vs main — 2 speed regression(s) · ⚠️ 5 secondary

Reference: 2026-08-05 morning nightly run (0d old) · full report → run

Speed — evaltime · prefill · decode

Δ	metric	device	main → PR
🔴 +7.7%	arm_ml_kws_cnn_m _{evaltime · pass}	`cortex-a9`	9.25 ms → 9.96 ms
🔴 +4.9%	openelm_270M_q40ef16_541 _{decode · metal}	`apple-m1-max`	4.53 ms/tok _{220.8 tok/s} → 4.75 ms/tok _{210.4 tok/s}

⚠️ 5 secondary regression(s)

Δ	metric	device	main → PR
⚠️ +44.0%	arm_ml_kws_cnn_m _{load · pass}	`cortex-a9`	84 ms → 121 ms
⚠️ +37.7%	arm_ml_kws_cnn_m _{load+optimize · pass}	`cortex-a9`	130 ms → 179 ms
⚠️ +10.8%	mobilenet_v1_1 _{RSS @ ready · pass}	`cortex-a53`	48.1 MB → 53.2 MB
⚠️ +6.5%	en_tdnn_15M _{RSS @ ready · pulse_120ms}	`cortex-a55`	112 MB → 120 MB
⚠️ +6.4%	arm_ml_kws_cnn_m _{load · pass}	`cortex-a53`	47 ms → 50 ms

github-actions · 2026-08-05T12:35:57Z

🔴 Bench vs main — 3 speed regression(s) · ⚠️ 12 secondary

Reference: 2026-08-05 morning nightly run (0d old) · full report → run

Speed — evaltime · prefill · decode

Δ	metric	device	main → PR
🔴 +14.9%	hey_snips_v1 _{evaltime · 400ms}	`cortex-a9`	9.29 ms → 10.7 ms
🔴 +5.7%	inceptionv1q _{evaltime · pass}	`apple-m1-max`	39.5 ms → 41.7 ms
🔴 +5.7%	openelm_270M_q40ef16_541 _{decode · metal}	`apple-m1-max`	4.53 ms/tok _{220.8 tok/s} → 4.79 ms/tok _{209 tok/s}

⚠️ 12 secondary regression(s)

Δ	metric	device	main → PR
⚠️ +39.3%	arm_ml_kws_cnn_m _{load · pass}	`cortex-a9`	84 ms → 117 ms
⚠️ +29.2%	arm_ml_kws_cnn_m _{load+optimize · pass}	`cortex-a9`	130 ms → 168 ms
⚠️ +19.5%	hey_snips_v1 _{load · 400ms}	`cortex-a9`	77 ms → 92 ms
⚠️ +17.0%	hey_snips_v1 _{load+optimize · 400ms}	`cortex-a9`	147 ms → 172 ms
⚠️ +13.0%	hey_snips_v4_model17_nnef _{load · pulse8}	`cortex-a7`	876 ms → 990 ms
⚠️ +9.1%	mobilenet_v1_1 _{load · pass}	`cortex-a9`	2.09 s → 2.28 s
⚠️ +8.7%	hey_snips_v4_model17_nnef _{load+optimize · pulse8}	`cortex-a7`	1.55 s → 1.69 s
⚠️ +8.5%	mobilenet_v1_1 _{load+optimize · pass}	`cortex-a9`	2.37 s → 2.58 s
⚠️ +6.0%	inceptionv3 _{load+optimize · pass}	`apple-m1-max`	350 ms → 371 ms
⚠️ +5.5%	hey_snips_v31 _{load+optimize · 400ms}	`cortex-a9`	344 ms → 363 ms
⚠️ +5.5%	inceptionv3 _{load · pass}	`apple-m1-max`	237 ms → 250 ms
⚠️ +5.1%	en_tdnn_15M _{RSS @ ready · 2600ms}	`cortex-a55`	112 MB → 118 MB

github-actions · 2026-08-05T13:43:02Z

🔴 Bench vs main — 2 speed regression(s) · ⚠️ 5 secondary

Reference: 2026-08-05 morning nightly run (0d old) · full report → run

Speed — evaltime · prefill · decode

Δ	metric	device	main → PR
🔴 +7.8%	en_tdnn_8M_nnef _{evaltime · pulse_240ms}	`cortex-a55`	29.8 ms → 32.1 ms
🔴 +5.8%	openelm_270M_q40ef16_541 _{decode · metal}	`apple-m1-max`	4.53 ms/tok _{220.8 tok/s} → 4.79 ms/tok _{208.6 tok/s}

⚠️ 5 secondary regression(s)

Δ	metric	device	main → PR
⚠️ +17.9%	arm_ml_kws_cnn_m _{load · pass}	`cortex-a9`	84 ms → 99 ms
⚠️ +12.3%	arm_ml_kws_cnn_m _{load+optimize · pass}	`cortex-a9`	130 ms → 146 ms
⚠️ +11.2%	mobilenet_v1_1 _{RSS @ ready · pass}	`cortex-a53`	48.1 MB → 53.4 MB
⚠️ +9.1%	hey_snips_v4_model17 _{load+optimize · pulse8}	`cortex-a7`	7.52 s → 8.21 s
⚠️ +7.6%	hey_snips_v4_model17 _{load · pulse8}	`cortex-a7`	6.78 s → 7.3 s

github-actions · 2026-08-05T17:44:31Z

⚠️ Bench vs main — no speed regressions · 7 secondary regression(s)

Reference: 2026-08-05 morning nightly run (0d old) · full report → run

Speed — evaltime · prefill · decode

no inference-speed regressions

Improvements

Δ	metric	device	main → PR
🟢 -16.5%	parakeet_tdt_600m_v3_f32f32_preprocessor_1s _{evaltime · metal}	`apple-m1-max`	1.59 ms → 1.33 ms
🟢 -8.6%	parakeet_tdt_600m_v3_f32f32_joint_pass _{evaltime · metal}	`apple-m1-max`	0.395 ms → 0.361 ms
🟢 -6.5%	parakeet_tdt_600m_v3_f32f32_encoder_1s _{evaltime · metal}	`apple-m1-max`	51.5 ms → 48.1 ms
🟢 -4.2%	llama_3_2_1B_instruct_q40ef16_541 _{decode · metal}	`apple-m1-max`	5.74 ms/tok _{174.1 tok/s} → 5.5 ms/tok _{181.7 tok/s}

⚠️ 7 secondary regression(s)

Δ	metric	device	main → PR
⚠️ +50.0%	hey_snips_v1 _{load · 400ms}	`cortex-a7`	70 ms → 105 ms
⚠️ +31.6%	hey_snips_v1 _{load+optimize · 400ms}	`cortex-a7`	117 ms → 154 ms
⚠️ +16.7%	arm_ml_kws_cnn_m _{load · pass}	`cortex-a9`	84 ms → 98 ms
⚠️ +12.1%	hey_snips_v31 _{load+optimize · 400ms}	`cortex-a7`	390 ms → 437 ms
⚠️ +11.7%	hey_snips_v31 _{load · 400ms}	`cortex-a7`	342 ms → 382 ms
⚠️ +11.5%	arm_ml_kws_cnn_m _{load+optimize · pass}	`cortex-a9`	130 ms → 145 ms
⚠️ +5.2%	hey_snips_v31 _{load+optimize · 400ms}	`cortex-a9`	344 ms → 362 ms

github-actions · 2026-08-05T18:31:24Z

⚠️ Bench vs main — no speed regressions · 6 secondary regression(s)

Reference: 2026-08-05 morning nightly run (0d old) · full report → run

Speed — evaltime · prefill · decode

no inference-speed regressions

Improvements

Δ	metric	device	main → PR
🟢 -16.5%	parakeet_tdt_600m_v3_f32f32_preprocessor_1s _{evaltime · metal}	`apple-m1-max`	1.59 ms → 1.33 ms
🟢 -8.3%	parakeet_tdt_600m_v3_f32f32_joint_pass _{evaltime · metal}	`apple-m1-max`	0.395 ms → 0.363 ms
🟢 -6.9%	parakeet_tdt_600m_v3_f32f32_encoder_1s _{evaltime · metal}	`apple-m1-max`	51.5 ms → 47.9 ms

⚠️ 6 secondary regression(s)

Δ	metric	device	main → PR
⚠️ +16.7%	arm_ml_kws_cnn_m _{load · pass}	`cortex-a9`	84 ms → 98 ms
⚠️ +15.2%	hey_snips_v31 _{load · 400ms}	`cortex-a7`	342 ms → 394 ms
⚠️ +14.9%	hey_snips_v31 _{load+optimize · 400ms}	`cortex-a7`	390 ms → 448 ms
⚠️ +10.8%	arm_ml_kws_cnn_m _{load+optimize · pass}	`cortex-a9`	130 ms → 144 ms
⚠️ +10.6%	mobilenet_v1_1 _{RSS @ ready · pass}	`cortex-a53`	48.1 MB → 53.2 MB
⚠️ +6.4%	arm_ml_kws_cnn_m _{load · pass}	`cortex-a53`	47 ms → 50 ms

github-actions · 2026-08-05T20:01:40Z

⚠️ Bench vs main — no speed regressions · 8 secondary regression(s)

Reference: 2026-08-05 morning nightly run (0d old) · full report → run

Speed — evaltime · prefill · decode

no inference-speed regressions

Improvements

Δ	metric	device	main → PR
🟢 -16.4%	parakeet_tdt_600m_v3_f32f32_preprocessor_1s _{evaltime · metal}	`apple-m1-max`	1.59 ms → 1.33 ms
🟢 -7.3%	parakeet_tdt_600m_v3_f32f32_joint_pass _{evaltime · metal}	`apple-m1-max`	0.395 ms → 0.366 ms
🟢 -6.5%	parakeet_tdt_600m_v3_f32f32_encoder_1s _{evaltime · metal}	`apple-m1-max`	51.5 ms → 48.2 ms

⚠️ 8 secondary regression(s)

Δ	metric	device	main → PR
⚠️ +15.5%	arm_ml_kws_cnn_m _{load · pass}	`cortex-a9`	84 ms → 97 ms
⚠️ +11.1%	hey_snips_v4_model17_nnef _{load+optimize · pulse8}	`cortex-a7`	1.55 s → 1.73 s
⚠️ +10.0%	arm_ml_kws_cnn_m _{load+optimize · pass}	`cortex-a9`	130 ms → 143 ms
⚠️ +7.5%	mobilenet_v1_1 _{load · pass}	`cortex-a9`	2.09 s → 2.25 s
⚠️ +6.9%	mobilenet_v1_1 _{load+optimize · pass}	`cortex-a9`	2.37 s → 2.54 s
⚠️ +6.4%	arm_ml_kws_cnn_m _{load · pass}	`cortex-a53`	47 ms → 50 ms
⚠️ +5.3%	en_tdnn_15M _{RSS @ ready · 2600ms}	`cortex-a55`	112 MB → 118 MB
⚠️ +5.2%	hey_snips_v31 _{load+optimize · 400ms}	`cortex-a9`	344 ms → 362 ms

github-actions · 2026-08-05T20:33:46Z

⚠️ Bench vs main — no speed regressions · 10 secondary regression(s)

Reference: 2026-08-05 morning nightly run (0d old) · full report → run

Speed — evaltime · prefill · decode

no inference-speed regressions

Improvements

Δ	metric	device	main → PR
🟢 -16.3%	parakeet_tdt_600m_v3_f32f32_preprocessor_1s _{evaltime · metal}	`apple-m1-max`	1.59 ms → 1.33 ms
🟢 -7.8%	parakeet_tdt_600m_v3_f32f32_joint_pass _{evaltime · metal}	`apple-m1-max`	0.395 ms → 0.364 ms
🟢 -6.7%	parakeet_tdt_600m_v3_f32f32_encoder_1s _{evaltime · metal}	`apple-m1-max`	51.5 ms → 48.1 ms

⚠️ 10 secondary regression(s)

Δ	metric	device	main → PR
⚠️ +16.7%	arm_ml_kws_cnn_m _{load · pass}	`cortex-a9`	84 ms → 98 ms
⚠️ +12.9%	hey_snips_v1 _{load · 400ms}	`cortex-a7`	70 ms → 79 ms
⚠️ +12.0%	hey_snips_v31 _{load · 400ms}	`cortex-a7`	342 ms → 383 ms
⚠️ +11.8%	hey_snips_v31 _{load+optimize · 400ms}	`cortex-a7`	390 ms → 436 ms
⚠️ +11.6%	inceptionv1q _{load+optimize · pass}	`cortex-a55`	3.89 s → 4.33 s
⚠️ +11.5%	arm_ml_kws_cnn_m _{load+optimize · pass}	`cortex-a9`	130 ms → 145 ms
⚠️ +11.1%	mobilenet_v1_1 _{RSS @ ready · pass}	`cortex-a53`	48.1 MB → 53.4 MB
⚠️ +11.1%	hey_snips_v1 _{load+optimize · 400ms}	`cortex-a7`	117 ms → 130 ms
⚠️ +8.5%	arm_ml_kws_cnn_m _{load · pass}	`cortex-a53`	47 ms → 51 ms
⚠️ +5.2%	hey_snips_v31 _{load+optimize · 400ms}	`cortex-a9`	344 ms → 362 ms

github-actions · 2026-08-06T10:09:54Z

🔴 Bench vs main — 1 speed regression(s) · ⚠️ 2 secondary

Reference: 2026-08-06 morning nightly run (0d old) · full report → run

Speed — evaltime · prefill · decode

Δ	metric	device	main → PR
🔴 +6.2%	hey_snips_v1 _{evaltime · 400ms}	`cortex-a55`	2.22 ms → 2.36 ms

Improvements

Δ	metric	device	main → PR
🟢 -19.2%	parakeet_tdt_600m_v3_f32f32_preprocessor_1s _{evaltime · metal}	`apple-m1-max`	1.59 ms → 1.28 ms
🟢 -8.4%	parakeet_tdt_600m_v3_f32f32_joint_pass _{evaltime · metal}	`apple-m1-max`	0.395 ms → 0.362 ms
🟢 -5.9%	parakeet_tdt_600m_v3_f32f32_encoder_1s _{evaltime · metal}	`apple-m1-max`	51.5 ms → 48.4 ms

⚠️ 2 secondary regression(s)

Δ	metric	device	main → PR
⚠️ +27.7%	arm_ml_kws_cnn_m _{load · pass}	`cortex-a9`	83 ms → 106 ms
⚠️ +21.7%	arm_ml_kws_cnn_m _{load+optimize · pass}	`cortex-a9`	129 ms → 157 ms

… matmul

github-actions · 2026-08-06T11:40:25Z

🔴 Bench vs main — 1 speed regression(s) · ⚠️ 9 secondary

Reference: 2026-08-06 morning nightly run (0d old) · full report → run

Speed — evaltime · prefill · decode

Δ	metric	device	main → PR
🔴 +18.1%	arm_ml_kws_cnn_m _{evaltime · pass}	`cortex-a9`	9.23 ms → 10.9 ms

Improvements

Δ	metric	device	main → PR
🟢 -16.8%	parakeet_tdt_600m_v3_f32f32_preprocessor_1s _{evaltime · metal}	`apple-m1-max`	1.59 ms → 1.32 ms
🟢 -7.3%	parakeet_tdt_600m_v3_f32f32_joint_pass _{evaltime · metal}	`apple-m1-max`	0.395 ms → 0.366 ms
🟢 -5.8%	parakeet_tdt_600m_v3_f32f32_encoder_1s _{evaltime · metal}	`apple-m1-max`	51.5 ms → 48.5 ms

⚠️ 9 secondary regression(s)

Δ	metric	device	main → PR
⚠️ +43.4%	arm_ml_kws_cnn_m _{load · pass}	`cortex-a9`	83 ms → 119 ms
⚠️ +36.4%	arm_ml_kws_cnn_m _{load+optimize · pass}	`cortex-a9`	129 ms → 176 ms
⚠️ +22.4%	hey_snips_v1 _{load · 400ms}	`cortex-a9`	76 ms → 93 ms
⚠️ +20.8%	hey_snips_v1 _{load+optimize · 400ms}	`cortex-a9`	144 ms → 174 ms
⚠️ +10.4%	arm_ml_kws_cnn_m _{load · pass}	`cortex-a53`	48 ms → 53 ms
⚠️ +9.5%	arm_ml_kws_cnn_m _{load+optimize · pass}	`cortex-a53`	74 ms → 81 ms
⚠️ +9.1%	hey_snips_v4_model17 _{load · pulse8}	`cortex-a7`	6.74 s → 7.36 s
⚠️ +8.2%	hey_snips_v4_model17 _{load+optimize · pulse8}	`cortex-a7`	7.48 s → 8.09 s
⚠️ +7.2%	hey_snips_v4_model17 _{load+optimize · 2sec}	`cortex-a7`	5.04 s → 5.4 s

github-actions · 2026-08-06T23:37:59Z

⚠️ Bench vs main — no speed regressions · 6 secondary regression(s)

Reference: 2026-08-06 morning nightly run (0d old) · full report → run

Speed — evaltime · prefill · decode

no inference-speed regressions

Improvements

Δ	metric	device	main → PR
🟢 -17.3%	parakeet_tdt_600m_v3_f32f32_preprocessor_1s _{evaltime · metal}	`apple-m1-max`	1.59 ms → 1.31 ms
🟢 -7.9%	parakeet_tdt_600m_v3_f32f32_joint_pass _{evaltime · metal}	`apple-m1-max`	0.395 ms → 0.364 ms
🟢 -6.9%	parakeet_tdt_600m_v3_f32f32_encoder_1s _{evaltime · metal}	`apple-m1-max`	51.5 ms → 47.9 ms

⚠️ 6 secondary regression(s)

Δ	metric	device	main → PR
⚠️ +27.7%	arm_ml_kws_cnn_m _{load · pass}	`cortex-a9`	83 ms → 106 ms
⚠️ +20.2%	arm_ml_kws_cnn_m _{load+optimize · pass}	`cortex-a9`	129 ms → 155 ms
⚠️ +13.8%	hey_snips_v4_model17 _{load · pulse8}	`cortex-a7`	6.74 s → 7.67 s
⚠️ +12.7%	hey_snips_v4_model17 _{load+optimize · pulse8}	`cortex-a7`	7.48 s → 8.43 s
⚠️ +8.2%	llama_3_2_3B_instruct_q40ef16_541 _{load+optimize · cuda}	`jetson-orin-nx`	4.38 s → 4.73 s
⚠️ +6.1%	llama_3_2_3B_instruct_q40ef16_541 _{load · cuda}	`jetson-orin-nx`	3.15 s → 3.34 s

JulienBalianSonos mentioned this pull request Apr 8, 2026

[RDY] feat: add basic MoE support sonos/torch-to-nnef#55

Open

JulienBalianSonos force-pushed the feat/moe-ffn-operator branch 2 times, most recently from 9f96279 to 4e1671e Compare June 12, 2026 10:13

JulienBalianSonos added 8 commits June 30, 2026 12:40

fix: add moe revamp with tests/cleaner ops

a679b3e

feat: tract_moe_ffn supports gpt-oss (optional router/expert biases, …

5ea9fc4

…clamped-swiglu act_alpha/act_limit; codegen keeps biased/clamped path on reference eval)

test: regenerate moe-ffn harness asset for single-output tract_moe_ff…

08cdaed

…n (no router_logits, no rank-align unsqueeze)

feat: replace normalize_gates bool with gate enum (softmax_topk|softm…

21b1a97

…ax_all|sigmoid|raw); fix stale op unit tests (single-output + gate); regen harness asset

fix: f16/bf16 support in MoE op (cast small glue to f32, keep experts…

c58e129

… native dtype, cast output back); compute router in f32 to stabilize top-k selection vs PyTorch; f16-faithful weighted scatter

fix: run MoE experts in f32 (upcast weights at codegen) to match PyTo…

99dc585

…rch CPU f32 matmul accumulation; native-f16 accumulation derailed greedy decoding

fix: allow Q40 expert weights in MoeFfn fallback

c547fe8

JulienBalianSonos force-pushed the feat/moe-ffn-operator branch from dccfa06 to c547fe8 Compare June 30, 2026 10:42

JulienBalianSonos added 9 commits July 6, 2026 11:00

perf(moe): fold routed q40 cpu and metal optimizations

ecee6ca

fix(metal): fallback to ggml for f16 matmul with f32 output

29243a6

feat(metal): support GPT-OSS Q40 MoE lowering

9d4091f

metal: select the GEMM implementation via TRACT_METAL_GEMM_IMPL

2492810

feat(moe): allow mixed-precision expert projections on the subplan path

9d50072

feat(moe): lower biased and clamped-activation experts into subplans

eac0107

test(moe): cover square mixed-precision experts over a long sequence

0343bb6

JulienBalianSonos added 6 commits August 3, 2026 22:42

Merge remote-tracking branch 'origin/main' into feat/moe-ffn-operator

ef4ab6d

# Conflicts: # metal/src/kernels/mod.rs # transformers/src/lib.rs # transformers/src/ops/mod.rs

fix(metal): sync after routed MoE matmuls to avoid long-context corru…

4bdb3eb

…ption

perf(metal): split MoE sync into non-blocking command-buffer commits

a869fda

perf(metal): bound in-flight MoE commits to keep transient memory flat

fe6980d

perf(metal): restrict MoE commit splits to prefill-sized batches

c75e37d

debug(metal): env-gated per-buffer GPU timing and MoE commit threshold

42b4796

JulienBalianSonos added 5 commits August 4, 2026 22:33

feat(metal): sinks softmax kernel and device view constructor for in-…

c848f33

…place KV

feat(metal): device-resident in-place KV attention for GPT-OSS

69cf430

fix(metal): correct multi-output shunting in fuse_axis_op split path

245c230

feat(metal): keep in-place KV cache outputs device-resident

f78acc0

fix(metal): in-place KV in-graph correctness and debug instrumentation

2dcf147

feat(gpu): metadata-only slicing for device KV cache views

69754a8

feat(metal): batch GPT-OSS attention gemms across kv heads

72ae32a

feat(metal): flash-attention decode kernel for GPT-OSS, env-gated

e772ae2

feat(metal): env-gated copy_nd and kv-rebuild diagnostics

6e709db

feat(metal): recycle device buffers through an exact-shape pool

03e0845

fix(metal): cap pooled buffer size and add pool disable gate

0c15008

feat(metal): opt-in q8_0 KV shadow for GPT-OSS decode attention

6e06119

feat(metal): split-k decode AV gemv for long contexts

e371828

feat(transformers,metal): clamp GPT-OSS sliding-window attention reads

38e1d87

feat(metal): token-major MoE combine fast path, opt-in grouped routed…

cce6af1

… matmul

feat(metal): expert-sorted tiled mm for prefill MoE, default on

e188168

feat(metal): fused per-route expert bias add

061a4a0

Uh oh!

Conversation

JulienBalianSonos commented Mar 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Model Coverage

Granite MoE Benchmark

Uh oh!

github-actions Bot commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 28, 2026

Uh oh!

github-actions Bot commented Aug 4, 2026

Uh oh!

github-actions Bot commented Aug 5, 2026

Uh oh!

github-actions Bot commented Aug 5, 2026

Uh oh!

github-actions Bot commented Aug 5, 2026

Uh oh!

github-actions Bot commented Aug 5, 2026

Uh oh!

github-actions Bot commented Aug 5, 2026

Uh oh!

github-actions Bot commented Aug 5, 2026

Uh oh!

github-actions Bot commented Aug 5, 2026

Uh oh!

github-actions Bot commented Aug 5, 2026

Uh oh!

github-actions Bot commented Aug 5, 2026

Uh oh!

github-actions Bot commented Aug 5, 2026

Uh oh!

github-actions Bot commented Aug 6, 2026

Uh oh!

github-actions Bot commented Aug 6, 2026

Uh oh!

github-actions Bot commented Aug 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

JulienBalianSonos commented Mar 30, 2026 •

edited

Loading

github-actions Bot commented Jun 30, 2026 •

edited

Loading