The article discusses the training of multiple AI language models at AWS AI Labs, ranging from 125 million to 13 billion parameters. It highlights the methodologies behind model configurations, including attention variations and specific training hyperparameters. Through extensive experiments, the study compares the capabilities and latencies of different attention mechanisms, such as multi-query and multi-head attention, revealing their implications for improving inference efficiency. The findings also emphasize the importance of consistent settings across model configurations to validate results effectively in the realm of language modeling.
We trained multiple models with varying sizes, ranging from 125 million parameters to 13 billion parameters, using code data with a context size of 2048.
For our largest model family, the 13 billion parameter model, we used a global batch size of 1024, which approximately translates to 2 million tokens per batch.
The settings for each model within each model-size family were kept consistent to ensure valid comparisons of capabilities and latencies during experiments.
Experiments demonstrated the strengths of multi-query and multi-head attention mechanisms, revealing insights into better performance and efficiency in language model inference.
Collection
[
|
...
]