Charlotte Qi from Meta outlines the significant challenges involved in developing and scaling LLM (Large Language Model) serving infrastructure. With AI advancements accelerating, the demand for compute resources has surged, driven by the rise of big models and more extended input contexts. she emphasizes that effectively serving LLMs requires a holistic approach that includes optimizing models, products, and systems in unison. Her current work focuses on making LLM infrastructure efficient and powerful, especially within Meta's AI initiatives. The complexities include handling immense internal traffic during model training and refining processes.
Scaling LLM serving resembles building a distributed operating system, as the demand for compute resources grows due to larger models and longer context.
The best optimization for model serving requires a comprehensive approach that addresses joint optimization between model, product, and system.
Collection
[
|
...
]