Microsoft has introduced BitNet b1.58 2B4T, a groundbreaking large language model (LLM) that is natively trained with 1-bit weights instead of using quantization from floating point weights. This new approach not only reduces computational and hardware demands but also matches the performance of full-precision models. The extensive training on a 4 trillion token dataset exemplifies Microsoft's innovative efforts to enhance LLM usability for edge devices and in real-time applications, showing advantages in memory footprint, inference latency, and energy consumption over other models.
Microsoft's BitNet b1.58 2B4T, the first natively trained with 1-bit weights, achieves performance akin to full-precision models while drastically reducing needs for computation and resources.
This model was trained from scratch on a massive dataset, demonstrating an innovative approach to building LLMs that mitigate the typical precision loss found in quantization.
Collection
[
|
...
]