Quantization Eats the Long Tail
·40 words·1 min
Great finding that I think is probably more generalizable: rarer domains gets way more impacted from quantization. This is loss of long tail for quantized models.
Many quantization methods involve fine-tuning but rarer domains won’t get fully restored. https://x.com/cheeesio/status/1846913815624982802