linalg,core: zero the padding lanes the mmm kernels compute on - #2596
linalg,core: zero the padding lanes the mmm kernels compute on#2596czoli1976 wants to merge 1 commit into
Conversation
Packed-operand buffers (pack_tensor/pack_tensor_view, im2col's PackedFormat panels) and the border-tile scratch buffers were zeroed only under debug_assertions, assuming arithmetic on discarded padding lanes is free. Uninitialized bytes there frequently decode to denormal floats, and every kernel op touching such a lane takes a microcode assist — measured 14x on partial tiles, and unfixable downstream under WASM where FTZ cannot be set. Zero the partial last panel at pack time and the border scratch buffers unconditionally (also fixing AddRowColProducts' col-tail zeroing hitting the rows buffer). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Additional measurements on scope. The touched code is arch-independent (generic packers, shared tile driver, im2col), so every CPU backend runs through it; whether the zeroing shows up as speed depends on the hardware's denormal cost. Native x86-64 (AVX/FMA kernels, same pulsed DeepFilterNet3 submodels, 100 frames/iter):
Native x86 is hit harder than wasm: the wider native tiles carry more padding lanes per border tile, and neither tract nor Rust sets FTZ/DAZ (MXCSR is never touched), so every op on a garbage lane pays the microcode assist — on main, native df_dec actually ran slower than the wasm build of the same model (80 ms vs 49 ms). aarch64 should be roughly neutral on speed (modern cores handle denormals at full rate; armv7 NEON is architecturally flush-to-zero), but every platform gains determinism: previously the padding content of packed buffers — and therefore timing — depended on allocator history, which is exactly the kind of run-to-run noise CI bench thresholds hate. Outputs are unaffected everywhere: padding lanes never feed valid results. Verified bit-identical enhanced audio through the full DeepFilterNet3 pipeline (STFT → three models → synthesis), main vs this PR. |
|
@kali really curious to see what this does to the production workloads and hey snips |
1 participant
In release builds the packed-operand allocators (pack_tensor/pack_tensor_view, im2col's PackedFormat buffers) and the border-tile scratch buffers leave their padding lanes uninitialized; the mmm kernels compute on those lanes before discarding them, and garbage bytes there frequently decode to denormal f32s, stalling partial tiles ~14x on x86 via microcode assists (with no FTZ escape hatch under WASM). Zeroing the partial last panel at pack time and the border scratch unconditionally fixes this — DeepFilterNet3's streaming decoders come out ~11% faster under wasmtime on x86-64 with bit-identical output — and also corrects AddRowColProducts' col-tail zeroing, which wrote to the rows buffer.
🍍
🤖 Generated with Claude Code