LoRA (Low Rank Adaptation) is a technique for fine-tuning Large Language Models (LLMs) efficiently by training only a minimal number of low-rank adapter matrices while keeping the majority of model weights frozen. This method significantly reduces memory and computational requirements. As of 2024, several new fine-tuning methods have emerged, often utilizing Singular Value Decomposition (SVD) for optimal parameter selection. These developments signal a growing interest in enhancing parameter-efficient training strategies within the machine learning community.
The original LoRA insight is that fine-tuning all the weights of a model is overkill. Instead, LoRA freezes the model and only trains a small pair of low-rank adapter matrices.
LoRA is very popular. It has even made its way into mainstream ML frameworks like Keras, allowing users to easily implement this technique.
In 2024, there has been an explosion of new parameter-efficient fine-tuning techniques, providing alternatives to LoRA, all leveraging concepts like Singular Value Decomposition.
The SVD is a great tool for understanding the structure of matrices and is utilized in several new fine-tuning approaches to optimize parameter selection.
Collection
[
|
...
]