Hugging Face Introduces RTEB, a New Benchmark for Evaluating Retrieval Models

"Hugging Face introduced the Retrieval Embedding Benchmark (RTEB), a new evaluation framework designed to more accurately measure how well embedding models generalize in real-world retrieval tasks. The beta benchmark aims to establish a community standard for evaluating retrieval accuracy in both open and private datasets. Retrieval quality is crucial for various AI systems, such as RAG, intelligent agents, enterprise search, and recommendation engines. However, existing benchmarks often do not represent real-world performance accurately."

"It combines open datasets, which are public and reproducible, with private datasets that remain accessible only to the MTEB maintainers, ensuring that results reflect genuine generalization rather than memorization. For each private dataset, only descriptive statistics and sample examples are released, maintaining transparency while preventing data leakage. In addition to its methodological improvements, RTEB focuses on real-world applicability. It includes datasets across critical domains such as law, healthcare, finance, and code, covering 20 languages from English and Japanese to Bengali and Finnish."

RTEB introduces a hybrid evaluation framework that combines public and private datasets to measure embedding model generalization for retrieval tasks. The benchmark uses open datasets for reproducibility and private datasets accessible only to maintainers, releasing only descriptive statistics and samples to prevent leakage. RTEB targets real-world applicability by including domains such as law, healthcare, finance, and code, and supports 20 languages including English, Japanese, Bengali, and Finnish. Datasets are sized to balance meaningful results with efficient evaluation. RTEB aims to reduce the generalization gap observed when models perform well on public benchmarks but fail in production.

Read at InfoQ