dspqueue-based llama.cpp hexagon backend hangs at the first compute round-trip on mainline fastrpc (glymur / X2 Elite)

[DerekParks]

FastRPC version

v1.0.7 (git 228d98b)

DSP firmware version

CDSP.HT.3.3-00304-GLYMUR-1

SoC and device name

Snapdragon X2 Elite (glymur), ASUS Zenbook A16

Kernel version

7.2.0-rc6 (linux-next based, mainline drivers/misc/fastrpc.c)

What happened?

Possibly related to #342

Process bring-up is fine. The unsigned user PD is created, all weight tensors static-map, the HMX clock comes up, and the command queue imports:

dspqueue_import: Queue Monitor supported. Importing Queue 0x38500000000 requires packet_callback

Like #342 I also see the method-3 EUNSUPPORTED on libdspqueue_rpc_skel.so:

Error 0x80000414: remote_handle64_invoke failed for module libdspqueue_rpc_skel.so, ... method 3 on domain 3 (sc 0x3010100)

The hang is one step later, at the first compute round-trip. The host writes the op packet, bumps the request counter, signals the DSP, then blocks in dspqueue_read waiting for RESP_PACKET and never gets it:

#4 wait_signal_locked (signal_no=2, timeout=...) at dspqueue/dspqueue_cpu.c:1335
#5 dspqueue_read (timeout_us=1000000)             at dspqueue/dspqueue_cpu.c:2401
#6 ggml_hexagon_session::flush_pending(bool)      in libggml-hexagon.so
#7 ggml_backend_hexagon_graph_compute
#8 llama_context::graph_compute

The DSP never consumes the request. With FARF turned up (logmask 0x001f001f001f001f) the DSP-side receive path comes up fully and then goes idle:

Initialize dspsignal session ...
Opening signal MQ "sigs5"
dspsignal_create: Signal 0 created ... Signal 3 created
dspsignal_mq_thread: <id>: Reading from MQ 8

After that it just hangs.

Things we (okay mainly claude but I provided moral support) ruled out:

• Unmodified driver: this is stock drivers/misc/fastrpc.c (nothing added), so it's not a local patch of mine.

• Poll mode: I enabled it (FASTRPC_IOCTL_SET_OPTION / FASTRPC_POLL_MODE succeeds, kaanapali has poll_mode_supported). Same hang; it only changes the host wait from a timeout loop to a poll spin.

• Driver signaling: I also backported the dspsignal ioctls to try driver signaling. The 8-byte signal is glink-delivered (rpmsg_send returns 0) but the DSP still doesn't consume the request. Same hang as RPC signaling.

• Not my build: GenieX bundles its own newer llama.cpp hexagon build (Hexagon Tools 6.6.0.0) and hangs identically on the same kernel:

  #4 wait_signal_locked (signal_no=2)   at dspqueue/dspqueue_cpu.c:1335
  #5 dspqueue_read (timeout_us=1000000) at dspqueue/dspqueue_cpu.c:2401
  #6 ggml_hexagon_session::flush_pending(bool)  in <geniex>/llama_cpp/libggml-hexagon.so
  geniex::LlamaLlm::generate
  

mainline drivers/misc/fastrpc.c on glymur seems to be the issue

I looked at the coherency angle since the Windows driver for this part (qcadsprpc8480.sys) manages the buffer cache in software - it flushes the FastRPC DMA buffers explicitly (KeFlushIoBuffers) and allocates them with an explicit cache type (MmAllocateContiguousMemorySpecifyCache) rather than relying on hardware coherency. glymur.dtsi marks the CDSP compute-cb nodes dma-coherent, so the Linux driver skips cache maintenance on these buffers. But I tested this: I rebuilt the DTB with dma-coherent removed from the CDSP compute-cb nodes. The buffers then map non-cached (confirmed the DSP reads them uncached from DRAM), and it still hangs at the same place. So it isn't a simple stale-read/coherency problem - the DSP monitor doesn't consume the request even when its view of the counter is guaranteed fresh.

Questions:

1. What does the BSP fastrpc driver do for the dspqueue path on glymur specifically (as opposed to the qcs Dragonwing parts) that mainline drivers/misc/fastrpc.c doesn't? I've matched the invoke and mem_map paths against the qcom-6.18.y driver and they look equivalent for this case, so I suspect the missing piece is glymur-specific.
2. Is there something beyond the RPC signal / driver signal that the DSP-side queue monitor needs to be armed on glymur (a notification/registration the mainline driver doesn't do)?
3. Is accel/qda the intended driver for running this on glymur under Linux, or is mainline fastrpc expected to work?

Happy to collect more DSP logs or test patches.

Steps to reproduce

Build fedora 7.2.0-rc6 w/ fastrpc.c on glymur and try to run llama.cpp

GGML_HEXAGON_VERBOSE=1 ./bin/llama-cli -m Llama-3.2-3B-Instruct-Q4_0.gguf --device HTP0 -ngl 99 -p "what is the most popular cookie in the world?" -n 128 -no-cnv

Relevant log output

[DerekParks]

Attached gdb to the hung process (RPC-signaling path, stock mainline fastrpc.c) and read the struct dspqueue header at the point where it's blocked in wait_signal_locked(RESP_PACKET).

The CPU wrote the request packet into the REQ ring (write position moved to 56, packet_count 1) and is parked waiting on RESP (read wait_count 1). The DSP's REQ read pointer never moved — position 0, packet_count 0. It never dequeued the request. It never produced a response.

The host is ringing the doorbell, and the DSP acks it:

PATCHDBG: calling dspqueue_rpc_signal
PATCHDBG: dspqueue_rpc_signal returned 0
dspqueue/dspqueue_cpu.c:2401::error: 12: (nErr = wait_signal_locked(q, DSPQUEUE_SIGNAL_RESP_PACKET, timeout_ts)) == 0 (repeats every 1s)
dspqueue_rpc_signal returns 0 (the DSP receives it), but the DSP-side queue monitor never advances its REQ read position to consume the packet.

[DerekParks]

Following up:

I ran the dspqueue test suite from PR #376 and 95/117 pass, 22/117 fail. Every failure is a test that requires the DSP to consume a queue packet. Every host-side and queue-lifecycle test passes.

I also ran Hexagon SDK's examples/dspqueue sample: fails on the first dspqueue_write:
dspqueue_cpu.c:1649: error 12: wait_signal_locked(q, DSPQUEUE_SIGNAL_REQ_SPACE, timeout_ts) == 0
dspqueue_cpu.c:1666: Error 0xc: dspqueue_write failed ... (message_length 8)

Any ideas on how to get the dsp to consume the request packet on glymur?

[UnsignedChad]

Same hang on X1E80100 (Snapdragon X Elite, not just glymur/X2) — Lenovo ThinkPad
T14s Gen 6 (LENOVO 21N1, 30 GB), so this is not X2-specific.

Firmware: cDSP CDSP.HT.2.9.c1-00078-HAMOA-1 (from Lenovo's Windows driver package
qcnspmcdm_ext_cdsp8380), shells from the same build. Note: the linux-firmware
qcom/x1e80100/LENOVO/21N1/qccdsp8380.mbn is CDSP.HT.2.9-00447-HAMOA-2 and matches
NO available shell (INIT_CREATE just times out with it) — separate linux-firmware bug
worth filing. The Qualcomm generic cdsp.mbn (CDSP.HT.2.9.c1-00069-HAMOA-1) is rejected
by the Lenovo PAS (-EINVAL at remoteproc start), so firmware swaps are limited to
Lenovo-signed builds.

Kernels: ubuntu-concept 7.2.0-3-qcom-x1e and jg 7.2.0-0sp11v7-qcom-x1e
(jglathe line, with the GLINK framing fix). Identical behavior on both.
Userspace: fastrpc development branch, v1.0.7, v1.0.2 — identical.

What works (direct invoke):

fastrpc_test -d 3 -U 1 -a v68 calculator: PASS (Sum=499500).

QAIRT 2.48 qnn-platform-validator --backend dsp --testBackend: PASS
("QNN is supported for backend DSP on the device").

What hangs: llama.cpp ggml-hexagon (--device HTP0). Session opens, buffers map
(MEM_MAP ok), the dspqueue skel open returns the expected method-3 0x80000414
EUNSUPPORTED (matches #342; per Ekansh Gupta (@ekanshibu) that's non-fatal), then the first compute
round-trip never returns. Thread stacks: main in futex wait; two workers in
fastrpc_internal_invoke. Kernel log: compute-cb@2: Error: Invoke Failed -110
(-ETIMEDOUT), and on teardown No context ID matches response.

This matches #377's packet-level finding exactly (host writes REQ, DSP acks the
doorbell, DSP-side queue monitor never advances the REQ read pointer). Also worth
noting for the kernel side: on pre-fix kernels the teardown race after such a hang
produced a hard wedge — a list-poison deref (0xdead000000000122) in the fastrpc
release path when a process was killed with invokes in flight. With the June fixes
(7.0.0-31+) that UAF is gone; the dspqueue hang remains.

Happy to collect DSP-side (FARF/mini-dm) logs if someone can point me at the right
knob on this stack — remotectl1 is "not supported for domain 3" here, so the usual
log-level control path is absent. Can also test patches.

[UnsignedChad]

Update: got LLMs running on the NPU on this machine (the dspqueue hang isn't the
whole story — a different stack works).

Root cause of our specific situation turned out to be a firmware/shell branch
mismatch: linux-firmware qcom/x1e80100/LENOVO/21N1/qccdsp8380.mbn is
CDSP.HT.2.9-00447-HAMOA-2, but every fastrpc shell is CDSP.HT.2.9.c1-HAMOA-1.
Booting the cDSP with Lenovo's matched -00078 build (from its own Windows driver
package) fixed session creation entirely. So two distinct issues were stacked: the
firmware mismatch (ours) and the dspqueue-consume hang (this issue).

With the matched firmware, the following all work on the NPU, no hangs:

• fastrpc calculator (direct invoke) — PASS.
• QAIRT 2.48 qnn-platform-validator --backend dsp --testBackend — PASS.
• Genie genie-t2t-run (QnnHtp engine) — real LLM inference: Llama-3.2-1B Q4_0
  at 39.4 tok/s prompt / 11.65 tok/s gen, Qwen2.5-Coder-3B Q4_0 at 26.0 / 6.42 tok/s,
  coherent output, reproducible.

Two things that made the Genie path work, in case it helps others here:

1. The runtime that supports SC8380XP is lib-safe/aarch64-oe-linux-gcc11.2
   (SoC model name is SC8380XP in logs). The aarch64-ubuntu-gcc9.4 builds of
   libQnnHtp.so/libQairtHtp.so reject SoC model 60 (QnnDevice_create → 14001,
   or crash in libQnnHtpPrepare.so's supertile builder with an empty soc_type).
   The ubuntu builds just aren't wired for X Elite in 2.48.0.260626.
2. LLM ctx-bins must be produced on an x86 host (qairt.gen_ai_api
GGUFBuilderHTP, targets chipset:SC8380XP, SHA graph mode) — the aarch64 SDK
   can't emit them.

So the dspqueue gap (llama.cpp/ggml-hexagon) stands as reported here, but QNN/Genie's
own runtime drives the cDSP fine on this firmware. Writeup:
https://github.com/UnsignedChad/t14s-x1e-npu