Trained on the same data set for long enough, pretty much every model converges to the same point.
It does seem like the performance gains are coming from data, at least once you have a stable training setup.
Llama 3 was trained on significantly more data than OLMo, which Lo believes explains its superiority on many popular AI benchmarks.
[
Collection
]
[
|
...
]