Issue: Group quantization currently expands scales/zeros from compact format [out_features, num_groups, 1] to full weight shape [out_features, in_features], causing massive memory overhead - up to 18GB+ just for scale storage in large models like LLaMA-3 8B.
Thought: Keep scales in their natural compact grouped format and update quantization operations to handle reshaping automatically, reducing scale storage memory by 99%+ while maintaining identical quantization behavior. But will this result any impact while finding the optimized orthogonal rotation matrix?
Issue: Group quantization currently expands scales/zeros from compact format [out_features, num_groups, 1] to full weight shape [out_features, in_features], causing massive memory overhead - up to 18GB+ just for scale storage in large models like LLaMA-3 8B.
Thought: Keep scales in their natural compact grouped format and update quantization operations to handle reshaping automatically, reducing scale storage memory by 99%+ while maintaining identical quantization behavior. But will this result any impact while finding the optimized orthogonal rotation matrix?