The pretrained Mistral-7b checkpoint undergoes fine-tuning for one epoch, employing techniques like LoRA and gradient checkpointing to optimize GPU memory use during training.
In our training, we incorporated approximately 1.8 million examples, integrating both synthetic data and data from 13 public datasets, facilitating a comprehensive evaluation framework.
Our evaluation on the MTEB benchmark, particularly the retrieval category aligned with the BEIR benchmark, emphasizes the extensive computational resources required for model assessments.
Despite accommodating sequences longer than 512, our evaluation focuses specifically on certain conditions, reinforcing the importance of targeted metrics in performance analysis.
Collection
[
|
...
]