feat(linalg): add AVX2+FMA SiLU f32 kernel - #2560
Merged
Merged
Conversation
cverrier
marked this pull request as ready for review
August 3, 2026 12:59
cverrier
force-pushed
the
feat/add-avx2-fma-silu-f32-kernel
branch
from
August 3, 2026 13:00
06fa421 to
2911709
Compare
|
Reference: 2026-08-02 morning nightly run (1d old) · full report → run Speed — evaltime · prefill · decode no inference-speed regressions Improvements
|
| Δ | metric | device | main → PR |
|---|---|---|---|
| llama_3_2_3B_instruct_q40ef16_541 load+optimize · cuda |
jetson-orin-nx |
4.14 s → 4.49 s | |
| qwen3_1_7B_q40ef16_541 load+optimize · cuda |
jetson-orin-nx |
3.77 s → 4 s |
Contributor
Author
|
@kali A few checks got cancelled, looks unrelated to the PR itself. Could you re-run them when you get a chance? |
x86 hosts with AVX2+FMA but no AVX-512 ran SiLU through the generic scalar kernel; they now get a vectorized one, which improves the throughput. Accuracy follows the FMA sigmoid f32 kernel.
Collaborator
|
|
kali
force-pushed
the
feat/add-avx2-fma-silu-f32-kernel
branch
from
August 4, 2026 11:52
2911709 to
d6199af
Compare
|
Reference: 2026-08-04 morning nightly run (0d old) · full report → run Speed — evaltime · prefill · decode no inference-speed regressions
|
| Δ | metric | device | main → PR |
|---|---|---|---|
| speaker_id RSS @ ready · pulse8 |
cortex-a7 |
17.7 MB → 21.4 MB | |
| arm_ml_kws_cnn_m load+optimize · pass |
cortex-a7 |
124 ms → 148 ms | |
| arm_ml_kws_cnn_m load · pass |
cortex-a7 |
83 ms → 99 ms | |
| arm_ml_kws_cnn_m load · pass |
cortex-a9 |
83 ms → 99 ms | |
| arm_ml_kws_cnn_m load+optimize · pass |
cortex-a9 |
127 ms → 145 ms |
Contributor
Author
|
@kali Is this work ready to be merged? |
kali
approved these changes
Aug 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
x86 hosts with AVX2+FMA but no AVX-512 ran SiLU through the generic scalar kernel; they now get a vectorized one, which improves the throughput. Accuracy follows the FMA sigmoid f32 kernel.
Also, I benched the different SiLU f32 kernels on an AVX-512 machine:
Throughput results:
which gives a nice ~61% increase in throughput: this was expected because the
x86_64_avx512_silu_f32_16nkernel is composed, not fused — it callsavx512_sigmoid_f32directly. Therefore, on AVX-512 — which is backward compatible with AVX2+FMA — we can fall back to this new AVX2+FMA SiLU kernel for now; see #2567, where this change will be implemented.Later, I will also implement a fused version for AVX-512.