LoRA: Low-Rank Adaptation of Large Language Models

Freezes the pre-trained weights and trains a pair of small low-rank matrices beside selected layers. This cuts trainable parameters and optimizer memory by orders of magnitude while staying close to full fine-tuning quality.

Why it matters

The update can be merged back into the weights, so inference is no slower. It is now the default way to customise large models on modest hardware.

Read the paper: arXiv:2106.09685

Plain-language summary written by SolxLab. Read the full paper at the official link; all rights remain with the authors.