Some more optimizations for q40, q80, q6k, q5k

[liangliangchang]

Nibble-split, Batch J=128 WMMAs, X prefetch, Raise MMQ N cutoff

Q4_0: Latency and throughput

MUL_MAT shape (m × n × k)	gfx11 (µs)	q4-0-opt (µs)	gfx11 TFLOPS	q4-0-opt TFLOPS	Δ TFLOPS
1024 × 128 × 4096	79.64	69.60	13.52	15.43	+14.1%
12288 × 32 × 4096	248.93	235.95	12.94	13.66	+5.5%
4096 × 32 × 12288	297.90	289.49	10.83	11.12	+2.7%
64 × 128 × 4096	66.91	65.53	1.00	1.02	+2.0%
4096 × 4 × 4096	29.47	29.24	4.55	4.59	+0.8%
4096 × 32 × 4096	103.48	103.62	10.38	10.37	−0.1%
4096 × 512 × 14336	2455.40	2457.42	24.49	24.47	−0.1%
4096 × 8 × 14336	165.00	165.21	5.70	5.69	−0.2%
4096 × 128 × 12288	616.05	619.37	20.92	20.81	−0.5%
4096 × 128 × 4096	198.57	199.85	21.63	21.49	−0.6%
8192 × 128 × 4096	340.09	342.86	25.26	25.05	−0.8%
12288 × 128 × 4096	498.37	502.49	25.86	25.64	−0.8%
32 × 128 × 4096	68.96	71.56	0.49	0.47	−3.4%

Q8_0 Latency and throughput

MUL_MAT shape (m × n × k)	gfx11 (µs)	q8-0-opt (µs)	gfx11 TFLOPS	q8-0-opt TFLOPS	Δ TFLOPS
8 × 128 × 4096	94.39	61.67	0.089	0.136	+52.9%
16 × 128 × 4096	92.45	62.75	0.181	0.267	+47.3%
32 × 128 × 4096	93.27	62.75	0.360	0.535	+48.6%
64 × 128 × 4096	69.48	61.22	0.97	1.10	+13.2%
1024 × 128 × 4096	69.92	67.06	15.36	16.01	+4.2%
12288 × 128 × 4096	512.75	497.07	25.13	25.92	+3.2%
4096 × 512 × 14336	2462.20	2405.64	24.42	25.00	+2.4%
8192 × 128 × 4096	355.01	353.58	24.20	24.30	+0.4%
4096 × 4 × 4096	35.75	35.60	3.76	3.77	+0.4%
4096 × 128 × 4096	191.53	190.98	22.43	22.49	+0.3%
4096 × 8 × 14336	303.66	304.52	3.09	3.09	−0.2%

Q5_k Latency and throughput

MUL_MAT shape (m × n × k)	gfx11 (µs)	final (µs)	paired speedup
8192 × 128 × 2560	245.00	224.60	+9.03%
8192 × 512 × 2560	914.88	844.89	+8.16%
2560 × 128 × 4096	123.31	119.88	+2.85%
1024 × 128 × 4096	68.85	68.12	+1.06%
4096 × 128 × 4096	212.24	205.76	+3.14%