CulturaX is a novel multilingual dataset with text data for 167 languages, producing 6.3 trillion tokens, facilitating high-performing LLMs for multilingual learning.
Our comprehensive pipeline has ensured the dataset is cleaned and deduplicated, which enhances the quality and utility of the data for researchers.
Open accessibility of CulturaX aims to promote research and practical applications in multilingual machine learning, addressing the growing need for diverse language understanding.
By providing such a vast and organized dataset, we hope to empower developers and researchers to advance multilingual AI technologies effectively.
Collection
[
|
...
]