Feature Request: Support for Bonsai-8B 1-bit Quantized Model

[kamjin3086]

Feature Request: Support for Bonsai-8B 1-bit Quantized Model

Hello,

I'd like to request support for Prism-ML's Bonsai-8B, a groundbreaking 1-bit quantized model available on Hugging Face:

Model URL: https://huggingface.co/prism-ml/Bonsai-8B-gguf

Why This Matters

Bonsai-8B represents a significant advancement in efficient LLM deployment:

• Compact Size: An 8B parameter model compressed to just ~1.15 GB
• Impressive Performance: Achieves an MMLU-R of 65.7, competitive with larger models
• Energy Efficient: Reported to be 5x more energy efficient than comparable 8B models
• Dramatically Smaller: 14x smaller footprint than typical 8B models

NPU Potential

The 1-bit quantization approach makes Bonsai-8B an ideal candidate for NPU acceleration. The dramatically reduced memory footprint and simplified computations could enable:

• Smooth inference on edge devices with NPUs
• Significantly reduced power consumption
• Real-time AI capabilities without cloud dependency
• Privacy-preserving local inference

Request

I'd love to see FastFlowLM add support for this model. Given its efficiency characteristics, it seems like a perfect fit for demonstrating NPU performance advantages with quantized models.

Thank you for considering this request!

Best regards

[twflm]

Feature Request: Support for Bonsai-8B 1-bit Quantized Model

Hello,

I'd like to request support for Prism-ML's Bonsai-8B, a groundbreaking 1-bit quantized model available on Hugging Face:

Model URL: https://huggingface.co/prism-ml/Bonsai-8B-gguf

Why This Matters

Bonsai-8B represents a significant advancement in efficient LLM deployment:

• Compact Size: An 8B parameter model compressed to just ~1.15 GB
• Impressive Performance: Achieves an MMLU-R of 65.7, competitive with larger models
• Energy Efficient: Reported to be 5x more energy efficient than comparable 8B models
• Dramatically Smaller: 14x smaller footprint than typical 8B models

NPU Potential

The 1-bit quantization approach makes Bonsai-8B an ideal candidate for NPU acceleration. The dramatically reduced memory footprint and simplified computations could enable:

• Smooth inference on edge devices with NPUs
• Significantly reduced power consumption
• Real-time AI capabilities without cloud dependency
• Privacy-preserving local inference

Request

I'd love to see FastFlowLM add support for this model. Given its efficiency characteristics, it seems like a perfect fit for demonstrating NPU performance advantages with quantized models.

Thank you for considering this request!

Best regards

Thanks for bringing this up. It seems very interesting! However, if take the a second look, the compression is not that impressive, since 14x is with respect to bf16 format. Edge LLM often uses q4 format, making the compression rate at around 4x.

Also, from figure 4 of https://prismml.com/news/bonsai-8b , the accuracy is slightly better than 1.7b ver (which, in q4, has a smaller footprint than bonsai-8b).

that said, 1 bit or 1.58 bit (log3_base_2, aka 3-state weight model or -1, 0, 1 weight model) is super cool, and solves the most critical bottleneck of HW, especially NPUs -- limited mem BW. We will look back when they mature up a bit.

Thank you 🙏

[xxxn3m3s1sxxx]

Related Work

This aligns with our ATLAS Ternary Packer research. For 1-bit models like BitNet/Bonsai, we use 2-bit packing:

• Current: 2-bit per weight
• Proposed: Pack 4 weights into 1 byte (16x from float32)

For Bonsai-8B (1.15GB):

• Float32 would be ~32GB
• Our 2-bit packing: ~2GB
• Savings: ~93%

Reference: microsoft/BitNet#548 with implementation details.

The 5x energy efficiency you mention aligns with our findings - bit-packing reduces memory bandwidth significantly.