riscv64: 4 of 4: RVV element-wise and reduction kernels - #2602
Open
czoli1976 wants to merge 6 commits into
Open
Conversation
added 6 commits
August 7, 2026 09:26
tract had no RISC-V backend, so rv64gc ran the generic Rust kernels for every matmul. Add an RVV 1.0 f32 mmm tier, detecting V from the AT_HWCAP bit that Linux never sets for the incompatible 0.7.1 draft and VLEN from the vlenb CSR. Because VLEN is a runtime property while MR must be a const generic, each kernel fixes (MR, NR, LMUL) and pins vl to MR, which is correct wherever VLMAX >= MR and short below it, so dispatch is gated on the hart's VLMAX reaching MR and the kernel re-checks the granted vl before running. An assembler probe keeps toolchains predating RVV 1.0 on the generic fallback.
hwbench's load_a_slice has hand-written versions for x86_64, aarch64 and arm only, so tract-linalg fails to compile with the hwbench feature on any other target -- which includes riscv64, and so blocks tract-cli entirely, since it always enables that feature. Add a portable version reading one volatile word per cache line, which moves the same traffic; with no wide loads there are fewer misses in flight, so an out-of-order core may not saturate and the figure reads low rather than wrong.
The RVV kernels are gated on the hart's vector length, so a single emulated width would leave half the kernel set untested. Add riscv64gc to the qemu cross-test platforms twice, at VLEN 256 and 128, which select disjoint halves. -cpu max rather than a profile model because the generic rv64 model cannot run Debian's riscv64 glibc at all.
The riscv64 tier covered f32 only, so f16 matmul fell back to the generic kernels even on parts with native half-precision vectors, which includes the SpacemiT X60 in the K1. Add an f16 mmm tier from the same template at SEW=16, where VLMAX doubles and so does every tile height for a given LMUL and VLEN. Zvfh is read from the /proc/cpuinfo isa line, since RVA23 mandates only Zvfhmin and that cannot hold an f16 accumulator, and a second assembler probe keeps toolchains predating Zvfh on the f32 tier alone.
The riscv64 tier had no integer kernels, so quantised models fell back to the generic ones for every matmul. Add an i32 accumulator tier handling both packings the frame offers: i32 x i32 at e32, and i8 x i8 through a loop at e16 where vle8.v picks EEW=8 off the instruction and vwmacc.vx widens straight into the e32 accumulators, which keeps VLMAX identical between the two so one vl serves both. QScale, RoundingShiftRight and ShiftLeft follow the sign-magnitude reference in generic/rounding.rs, widening to e64 for the multiply; the negative-value mask taken before the magnitude overwrites it also serves the MinusInf and PlusInf nudges, which differ only where the result is zero anyway.
Every element-wise binary op and every f32 reduction still ran the generic kernels on riscv64. Add RVV versions of by-scalar and unicast mul, add, sub, subf, min and max, plus the max, min and sum reductions. These strip-mine on vsetvli rather than fixing a tile, so unlike the matmul kernels they are vector-length agnostic and need no VLEN predicate, and being Rust asm! blocks rather than .S files they need only rustc's own assembler, so they sit outside the tract_rvv cfg that records whether an external one could encode RVV.
Contributor
Author
|
@kali these PRs all use rafitied and stable ISA, no vendor specific extensions of any kind (it is quite fragmented with competing proposals and implementations for various extensions) so these PRs add Foundational Risc-V to extend the reach of TRACT to this platform and possibily invite other contributions from the community as the ISA standardisation evolves and solidifies. from a CI point of view QEMU is plenty to verify correctness, I have used both 10.X and 11.X). the fact it runs InceptionV3 executing end-to-end is quite remarkable. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on the matmul tiers. Adds fifteen kernels: by-scalar and unicast
mul,add,sub,subf,min,maxover f32, and themax,minandsumreductions — filling theregister_all_by_scalar/register_all_unicastregistries and the
mul_by_scalar_f32,max_f32,min_f32,sum_f32slots.Two ways in which these differ from the matmul kernels
No VLEN predicate. They strip-mine on
vsetvliinstead of pinningvltoa tile height, so
vlis whatever the hart grants and the tail falls out of theloop condition. That makes them vector-length agnostic outright — nothing to
gate beyond
has_rvv().nris only the frame's chunking granularity here, nota tile width.
No external assembler. They are Rust
asm!blocks, as the correspondingarm64 kernels are, so
.option arch, +vgoes through rustc's own LLVM and nobinutils version can refuse them. They therefore sit outside the
tract_rvvcfg, which records only whether an external assembler could handle the
.Smatmul kernels. A toolchain too old for those still gets these.
The reductions keep the running result in element 0 of
v1and feed it back asthe reduction's scalar operand each round, so strip-mining needs no separate
accumulator vector and no final horizontal step — reduction operands are LMUL=1
whatever vtype says.
vfredusumis the unordered sum, reassociating as thearm64 kernels also do.
Testing
The frames' own proptests, with
has_rvv()as the condition so they staymeaningful on a hart without V. 2709 tract-linalg tests and tract-core's 270
pass at VLEN 128, 256 and 512, on stock RVA23, and with V absent, under qemu
10.2.1 and 11.0.3.
No CI change needed. Same caveat as the earlier PRs: correctness under
emulation, no hardware numbers.
🍍