MOSS-TTS-Local v1.5 BF16 GGUF is significantly slower than the safetensors model on Windows (CUDA + RTX 5070 Ti)

[cinetube]

Hi,

I observed a consistent performance difference between the safetensors and BF16 GGUF versions of MOSS-TTS-Local v1.5.

Environment

• Latest audio.cpp prebuilt binaries
• CUDA backend
• RTX 5070 Ti (16 GB)
• Windows
• Same input text and reference audio
• Warm runs (CUDA warm-up excluded)

Safetensors

• 1:32 audio → 31.56 s
• 1:47 audio → 34.58 s

BF16 GGUF

• 1:09 audio → 46.01 s
• 1:34 audio → 1.11 min
• 1:40 audio → 1.13 min

The BF16 GGUF model is consistently much slower (roughly 2× in my testing) than the equivalent safetensors model.

I also noticed that the BF16 GGUF occupies almost the full 16 GB of VRAM, whereas the safetensors model uses noticeably less VRAM.

Is this expected with the current GGUF implementation for MOSS-TTS-Local, or could there be an optimization opportunity in the GGUF inference path?

I'd be happy to run additional tests if needed.

[0xShug0]

@cinetube GGUF and SafeTensors are just weight storage formats; they use the same inference code.

I tested this on Linux. The differences are within normal measurement noise, since I was also running other tests on the GPU. I changed the title to make the issue clearer in case anyone else runs into the same problem. I’ll test it on Windows once my current tests are finished.

model	avg session.wall_ms	avg wav duration	avg RTF	avg ms/generated-frame
safetensors	4905.441 ms	22.907s	0.2147	17.177
BF16 GGUF	4856.212 ms	22.240s	0.2188	17.503

RTF per run

model	run1	run2	run3
safetensors	0.2091	0.2131	0.2219
BF16 GGUF	0.2113	0.2282	0.2168

@mirek190 Do you have any thoughts on what might be causing this on Windows?

[0xShug0]

@cinetube Tested on Windows. I cannot reproduce the issue. Could you share the cmds, assets and logs (with --metrics enabled) for the safetensor and gguf runs?

model	avg session.wall_ms	avg wav duration	avg RTF	avg ms/generated-frame
safetensors	2500.451 ms	10.907s	0.2303	18.421
BF16 GGUF	2338.401 ms	9.787s	0.2389	19.111

model	run1	run2	run3
safetensors	0.2281	0.2218	0.2408
BF16 GGUF	0.2399	0.2388	0.2379

[mirek190]

0xShug0

For me is also similar speed between sefetensor and gguf 16.
I suspect user has not enough vram.

I think for gguf we should have Q8 models as default otherwise full fp16 gguf.
Q8 are a mixture of int8 and fp16 layers.

Also after a webui I check if we can handle gguf fp16 more efficient way as I was only optimizing Q8.
Anyway we need logs!