LoRA: Low-Rank Adaptation of Large Language Models
ReadFine-tunes a frozen model by learning low-rank updates to its weight matrices. It is how most people adapt a large model on one GPU.
My notes
I had used LoRA via PEFT before reading this. The paper made the rank choice feel much less like a magic number.
The Update
W = W₀ + ΔW = W₀ + B·A, rank(B·A) ≤ r ≪ min(d, k)
Practical Notes
- ~10,000× fewer trainable parameters than full fine-tuning on GPT-3 in their experiments, with no inference latency once merged.
- Applying it to attention projections (W_q, W_v) gave most of the gain.
- Swap adapters per task on one base model — which is how I think about serving multiple variants of a reviewer model for Code-Eagle AI.
- QLoRA (already on my list) = LoRA on top of a 4-bit quantized base.