The study examines the pervasive phenomenon of parameter heterogeneity in large language models (LLMs), where a select few parameters, termed 'cherry' parameters, have a major impact on performance. In response, the authors introduce CherryQ, a quantization technique that optimizes mixed-precision parameters, maintaining high precision for critical parameters while lowering precision for less impactful ones. Experimental results indicate CherryQ outperforms existing methods, achieving notable performance even with a 3-bit quantized model, thus promoting efficient LLM deployment by leveraging parameter diversity.
This paper reveals the phenomenon of parameter heterogeneity in large language models (LLMs), identifying 'cherry' parameters that disproportionately influence model performance.
CherryQ, our novel quantization method, unifies mixed-precision optimization by preserving critical parameters in high precision while aggressively quantizing others to low precision.
Experiments show CherryQ's effectiveness, achieving competitive performance with a 3-bit quantized Vicuna-1.5 compared to traditional 16-bit models.
The findings highlight CherryQ's potential for enabling the efficient deployment of LLMs, leveraging the inherent parameter heterogeneity within these models.
Collection
[
|
...
]