Microsoft researchers have introduced BitNet b1.58 2B4T, the first large language model operating on a 1-bit format with 2 billion parameters. This model, designed to run on commercial CPUs like Apple's M2, is trained on an extensive corpus of 4 trillion tokens. Unlike traditional models that use higher bit formats, BitNet's unique 1-bit structure significantly reduces memory needs, all while maintaining competitive performance levels. Measuring only 400MB, this innovative approach marks a key advancement in the efficiency of language models, enabling faster computations with lower energy consumption, and outpacing competitors like Meta's Llama and Google's Gemma in benchmarks.
Microsoft researchers claim to have developed the first 1-bit large language model, BitNet b1.58 2B4T, which can run on standard CPUs like the Apple M2.
Trained on 4 trillion tokens, this model illustrates how 1-bit LLMs can reach performance equivalent to full-precision models, while being more efficient.
BitNet b1.58 2B4T differs from other models as it uses a unique 1-bit format, resulting in drastically lower memory requirements without sacrificing performance.
With a size of only 400MB, BitNet b1.58 2B4T showcases the potential of native 1-bit LLMs trained at scale, outshining competitors in various benchmarks.
Collection
[
|
...
]