riscv64: 3 of 4: 8 and i32 RVV matmul kernels - #2601
Open
czoli1976 wants to merge 5 commits into
Open
Conversation
added 5 commits
August 7, 2026 09:26
tract had no RISC-V backend, so rv64gc ran the generic Rust kernels for every matmul. Add an RVV 1.0 f32 mmm tier, detecting V from the AT_HWCAP bit that Linux never sets for the incompatible 0.7.1 draft and VLEN from the vlenb CSR. Because VLEN is a runtime property while MR must be a const generic, each kernel fixes (MR, NR, LMUL) and pins vl to MR, which is correct wherever VLMAX >= MR and short below it, so dispatch is gated on the hart's VLMAX reaching MR and the kernel re-checks the granted vl before running. An assembler probe keeps toolchains predating RVV 1.0 on the generic fallback.
hwbench's load_a_slice has hand-written versions for x86_64, aarch64 and arm only, so tract-linalg fails to compile with the hwbench feature on any other target -- which includes riscv64, and so blocks tract-cli entirely, since it always enables that feature. Add a portable version reading one volatile word per cache line, which moves the same traffic; with no wide loads there are fewer misses in flight, so an out-of-order core may not saturate and the figure reads low rather than wrong.
The RVV kernels are gated on the hart's vector length, so a single emulated width would leave half the kernel set untested. Add riscv64gc to the qemu cross-test platforms twice, at VLEN 256 and 128, which select disjoint halves. -cpu max rather than a profile model because the generic rv64 model cannot run Debian's riscv64 glibc at all.
The riscv64 tier covered f32 only, so f16 matmul fell back to the generic kernels even on parts with native half-precision vectors, which includes the SpacemiT X60 in the K1. Add an f16 mmm tier from the same template at SEW=16, where VLMAX doubles and so does every tile height for a given LMUL and VLEN. Zvfh is read from the /proc/cpuinfo isa line, since RVA23 mandates only Zvfhmin and that cannot hold an f16 accumulator, and a second assembler probe keeps toolchains predating Zvfh on the f32 tier alone.
The riscv64 tier had no integer kernels, so quantised models fell back to the generic ones for every matmul. Add an i32 accumulator tier handling both packings the frame offers: i32 x i32 at e32, and i8 x i8 through a loop at e16 where vle8.v picks EEW=8 off the instruction and vwmacc.vx widens straight into the e32 accumulators, which keeps VLMAX identical between the two so one vl serves both. QScale, RoundingShiftRight and ShiftLeft follow the sign-magnitude reference in generic/rounding.rs, widening to e64 for the multiply; the negative-value mask taken before the magnitude overwrites it also serves the MinusInf and PlusInf nudges, which differ only where the result is zero anyway.
czoli1976
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on the f32 and f16 tiers. Adds four i32-accumulator kernels — 8x8 and
16x8 GEMM, 16x1 and 32x1 GEMV — covering both packings the frame offers:
i32 x i32, and i8 x i8 accumulating into i32.
The i8 inner loop
vtype sits at e16 for that loop, which buys two things at once.
vle8.vtakesEEW=8 from the instruction rather than from vtype, so the packed A column loads
as bytes with no separate widening step beyond one
vsext.vf2; andvwmacc.vxwidens on the way in, landing straight in the e32 accumulators.Because VLMAX at e16/mN equals VLMAX at e32/m2N, a single
vlof MR is validin both states, so the loop needs one
vsetvlion entry and one on exit ratherthan a pair per K step.
Zvqdotqwould collapse this further, but it is unratified and no hardwareimplements it yet.
Quantised ops
QScale,RoundingShiftRightandShiftLeftfollow the sign-magnitudereference in
generic/rounding.rsrather than an arithmetic shift, so the sixrounding policies differ only in the nudge term. Two things make that cheaper
than it looks:
serves the MinusInf and PlusInf nudges as well. PlusInf wants
v <= 0ratherthan
v < 0, but those differ only atv == 0, wherehalf + nudgeneversurvives the shift and the result is 0 under every policy.
signumneeds no zero case.QScalewidens to e64 for the i64 product, rounds there, and truncates back,matching the reference which computes in i64 and casts.
Testing
test_mmm_kernel_i32!pulls inmmm_q_scale_tests!, so every policy isproptested against the reference — that is what these were developed against.
2683 tract-linalg tests and tract-core's 270 pass at VLEN 128, 256, 512 and
1024, on stock RVA23, and with V absent, under both qemu 10.2.1 and 11.0.3.
No CI change needed; the entries the f32 PR added cover this tier.
Same caveat as the earlier two: correctness under emulation, no hardware
numbers.
🍍