Skip to content

feat(linalg): add AVX2+FMA SiLU f32 kernel - #2560

Merged
kali merged 1 commit into
sonos:mainfrom
cverrier:feat/add-avx2-fma-silu-f32-kernel
Aug 8, 2026
Merged

feat(linalg): add AVX2+FMA SiLU f32 kernel#2560
kali merged 1 commit into
sonos:mainfrom
cverrier:feat/add-avx2-fma-silu-f32-kernel

Conversation

@cverrier

@cverrier cverrier commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

x86 hosts with AVX2+FMA but no AVX-512 ran SiLU through the generic scalar kernel; they now get a vectorized one, which improves the throughput. Accuracy follows the FMA sigmoid f32 kernel.

Also, I benched the different SiLU f32 kernels on an AVX-512 machine:

cargo bench -p tract-linalg --bench silu -- silu_f32

Throughput results:

silu_f32/linalg         thrpt:  [1.8691 Gelem/s 1.8697 Gelem/s 1.8703 Gelem/s]

silu_f32/linalg-asm-fused
                        thrpt:  [3.0071 Gelem/s 3.0179 Gelem/s 3.0314 Gelem/s]

which gives a nice ~61% increase in throughput: this was expected because the x86_64_avx512_silu_f32_16n kernel is composed, not fused — it calls avx512_sigmoid_f32 directly. Therefore, on AVX-512 — which is backward compatible with AVX2+FMA — we can fall back to this new AVX2+FMA SiLU kernel for now; see #2567, where this change will be implemented.

Later, I will also implement a fused version for AVX-512.

@cverrier
cverrier marked this pull request as ready for review August 3, 2026 12:59
@cverrier
cverrier force-pushed the feat/add-avx2-fma-silu-f32-kernel branch from 06fa421 to 2911709 Compare August 3, 2026 13:00
@github-actions

github-actions Bot commented Aug 3, 2026

Copy link
Copy Markdown

⚠️ Bench vs main — no speed regressions · 2 secondary regression(s)

Reference: 2026-08-02 morning nightly run (1d old) · full report → run

Speed — evaltime · prefill · decode

no inference-speed regressions

Improvements

Δ metric device main → PR
🟢 -19.2% parakeet_tdt_600m_v3_f32f32_decoder_pass
evaltime · cpu
i9-11900kb_rtx-4060 0.838 ms → 0.677 ms
⚠️ 2 secondary regression(s)
Δ metric device main → PR
⚠️ +8.2% llama_3_2_3B_instruct_q40ef16_541
load+optimize · cuda
jetson-orin-nx 4.14 s → 4.49 s
⚠️ +5.9% qwen3_1_7B_q40ef16_541
load+optimize · cuda
jetson-orin-nx 3.77 s → 4 s

@cverrier

cverrier commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

@kali A few checks got cancelled, looks unrelated to the PR itself. Could you re-run them when you get a chance?

x86 hosts with AVX2+FMA but no AVX-512 ran SiLU through the generic
scalar kernel; they now get a vectorized one, which improves the
throughput. Accuracy follows the FMA sigmoid f32 kernel.
@kali

kali commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

⚠️⚠️⚠️ Just rebased! ⚠️⚠️⚠️

@kali
kali force-pushed the feat/add-avx2-fma-silu-f32-kernel branch from 2911709 to d6199af Compare August 4, 2026 11:52
@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown

⚠️ Bench vs main — no speed regressions · 5 secondary regression(s)

Reference: 2026-08-04 morning nightly run (0d old) · full report → run

Speed — evaltime · prefill · decode

no inference-speed regressions

⚠️ 5 secondary regression(s)
Δ metric device main → PR
⚠️ +20.9% speaker_id
RSS @ ready · pulse8
cortex-a7 17.7 MB → 21.4 MB
⚠️ +19.4% arm_ml_kws_cnn_m
load+optimize · pass
cortex-a7 124 ms → 148 ms
⚠️ +19.3% arm_ml_kws_cnn_m
load · pass
cortex-a7 83 ms → 99 ms
⚠️ +19.3% arm_ml_kws_cnn_m
load · pass
cortex-a9 83 ms → 99 ms
⚠️ +14.2% arm_ml_kws_cnn_m
load+optimize · pass
cortex-a9 127 ms → 145 ms

@cverrier

cverrier commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

@kali Is this work ready to be merged?

@kali
kali merged commit 0c9b292 into sonos:main Aug 8, 2026
58 checks passed
@cverrier
cverrier deleted the feat/add-avx2-fma-silu-f32-kernel branch August 8, 2026 16:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants