MobileLLM aims to demonstrate that for smaller models, quality is influenced more by architecture design rather than the sheer number of parameters.
Our results show that, especially for smaller models, going deeper in the architecture yields better performance improvements than simply increasing width.
Embedding sharing helps reduce total parameters in smaller models, proving effective as they can account for a significant percentage of total weights.
Techniques like immediate block-wise weight sharing further emphasize the importance of efficient weight utilization in smaller models for maximizing performance.
Collection
[
|
...
]