AI Engineer Roadmap All lessons Interactive Quiz MernStackDev GitHub

LoRA in Depth

Intermediate · 11 min read · Updated · By Saurabh Pathak

In short: LoRA freezes a weight matrix W and learns two thin matrices B and A whose scaled product is added to it, W' = W + (alpha/r) B A, so you train a fraction of a percent of the parameters, can merge the result back for free, and choose rank, alpha and target layers to trade capacity for cost.

Play the interactive lesson: LoRA in depthRead on GitHub
On this page
  1. Explain it like I'm new
  2. Watch it
  3. How it works
  4. Code example
  5. When to use it / when not to
  6. Common mistakes
  7. Recap
  8. Interview questions
  9. Go deeper

Level: Intermediate · Reading time: 11 min · Phase: 8 · Fine-tuning

Before you start: LoRA and PEFT (the short version, with a 4x4 example) and Supervised and Instruction Tuning. This lesson goes deeper: rectangular matrices, whole-model counts, a rank proof, and how to choose settings.

Words to know:

Explain it like I'm new

A huge printed atlas is too expensive to reprint, so you stick a thin transparent overlay on certain pages with your corrections. The overlay is small, cheap to make, can be peeled off, and you can keep a different overlay per customer. Once you are happy, you can print the corrections into the page permanently (merging).

The reason a thin overlay can work is an empirical observation from the LoRA paper: the change needed to adapt a pretrained model to a new task tends to have low "intrinsic rank". You do not need to move every weight independently; a handful of directions of change capture most of what matters. LoRA hard-wires that assumption: instead of learning a full d_out by d_in change, it learns it as a product of a tall thin matrix and a short wide one.

Watch it

LoRA in depth: Freeze W, train a small B and A, then optionally merge them.
LoRA in depth: Freeze W, train a small B and A, then optionally merge them.

It plays on a loop. Read the caption in the box, one step at a time. Want to go at your own pace? Step through it with Previous and Next.

How it works

The formula

For a frozen weight W of shape d_out by d_in, LoRA adds two trainable matrices: A (r by d_in) and B (d_out by r). The layer computes:

h  =  W x  +  (alpha / r) * B (A x)
W' =  W    +  (alpha / r) * B A          (the merged weight)

A is initialised with small random numbers and B starts at zero, so at step 0 the update is exactly zero and the model behaves like the base model. Training then moves only A and B.

Counting parameters

Full fine-tuning of one matrix trains d_out x d_in numbers. LoRA trains r x (d_in + d_out):

full  = d_in * d_out
LoRA  = r * (d_in + d_out)

From ft_lora_params.py:

For a whole model we use a toy 7B-class layout: 32 layers, d_model 4096, feed-forward width 11008 with three matrices (gate, up, down), 32,000-word vocabulary. That totals 6,738,149,376 parameters. Trainable parameters by choice of layers and rank:

Target modulesRankTrainableShare of the model
q, v84,194,3040.062%
q, v168,388,6080.124%
q, k, v, o1616,777,2160.249%
all seven linear layers1639,976,9600.593%

Trainable count is linear in rank: doubling r from 8 to 16 exactly doubles it. (The earlier lesson's 4,194,304 appears again here as the q, v, r = 8 row; the earlier lesson quotes a rounded 7B total, so its memory figure differs slightly from ours.)

Why the update has rank at most r

B A is a (d_out by r) times (r by d_in) product, so every column of B A is a combination of the r columns of B. It cannot have more than r independent directions. The demo checks this with a pure-Python rank function (Gaussian elimination with partial pivoting) on random Gaussian matrices with d = 12:

r=1: rank(BA) = 1      r=2: rank(BA) = 2
r=4: rank(BA) = 4      r=8: rank(BA) = 8
a random 12x12 matrix: rank 12

So the update is a rank-r correction to a (usually) full-rank W. It is a strong restriction, and that is both LoRA's saving and its limit.

Choosing rank and alpha

These are starting points, not laws. Treat them as hyperparameters and compare on a validation set (lesson 6).

Which layers?

The original paper adapted only the attention query and value projections. Many practitioners now adapt all linear layers (attention and feed-forward), which costs more parameters (0.593% above versus 0.062%) but often fits harder tasks better; the QLoRA paper found that adapting all linear layers mattered for matching full fine-tuning quality. Embeddings and norm layers are usually left alone. Try attention-only first for a narrow style task, and widen if validation loss stalls.

Merging, and not merging

Because the update is linear, W x + s B (A x) equals (W + s B A) x where s = alpha / r. The demo verifies this on random matrices with d = 16, r = 4, alpha = 8: the largest difference between the two outputs is 3.55e-15, which is rounding noise.

Worked example

Take d = 4096 for q and v in all 32 layers at r = 8.

  1. One matrix: 8 x (4096 + 4096) = 65,536 trainable numbers.
  2. Two matrices per layer, 32 layers: 64 matrices x 65,536 = 4,194,304.
  3. Against 6,738,149,376 total: 4,194,304 / 6,738,149,376 = 0.062%.
  4. Gradients and optimizer state exist only for those 4.19 million numbers, which is why memory drops so much (next lesson counts it).
  5. Raise the rank to 16 and the count doubles to 8,388,608, still only 0.124%.
flowchart LR
    X[Input x] --> W[Frozen W]
    X --> A[A: r x d_in, trained]
    A --> B[B: d_out x r, trained, starts at 0]
    B --> S["Scale alpha / r"]
    W --> P((Add))
    S --> P
    P --> Y[Output h]

Code example

From ft_lora_params.py (run python 10-fine-tuning/code/ft_lora_params.py):

import random
import ft_lora_params as lp

print(lp.lora_params(4096, 4096, 8))                       # -> 65536
m = lp.ToyTransformer()
print(m.total_params())                                    # -> 6738149376
print(m.lora_trainable(["q", "v"], 8))                     # -> 4194304

rng = random.Random(0)
BA = lp.matmul(lp.random_matrix(12, 4, rng), lp.random_matrix(4, 12, rng))
print(lp.rank(BA))                                         # -> 4

The tests in test_ft_lora_params.py check every count above, that the rank of B A equals r for r in 1, 2, 4 and 8, that merge equals the separate adapter to 1e-9, that a zero B leaves the weights unchanged, and that doubling the rank doubles the trainable count.

In real projects use the Hugging Face PEFT library, which exposes rank, alpha, dropout and target modules as configuration, and can merge adapters. Its API changes, so read the current docs. The method is in the LoRA paper (Hu et al., 2021).

When to use it / when not to

SituationLoRA?
Full fine-tuning does not fit in memoryYes
Many tasks or customers on one base modelYes, one small adapter each
Narrow style or format changeYes, start with r = 8 to 16 on attention layers
A task far from the base model's abilities, or needing broad new skillsMaybe not. Try a higher rank, all linear layers, or a different base
You need new reliable factsNo. RAG

Common mistakes

Recap

If you remember only three things:

Interview questions

1. Derive the number of trainable parameters for a LoRA adapter on a d_out x d_in matrix. <details><summary>Answer</summary> A is r x d_in and B is d_out x r, so r x d_in + d_out x r = r x (d_in + d_out). For 4096 x 4096 at r = 8 that is 8 x 8192 = 65,536, against 16,777,216 for the full matrix. </details>

2. Why does B start at zero, and why does it matter? <details><summary>Answer</summary> With B = 0 the product B A is zero, so the adapted model equals the base model at step 0. Training starts from known-good behaviour and moves away gradually, instead of starting from a random perturbation of the weights. </details>

3. How do alpha and r interact, and when is merging a bad idea? <details><summary>Answer</summary> The update is scaled by alpha / r, so changing r with alpha fixed changes the update's strength; tune them together with the learning rate. Merging is exact in full precision, but merging into a quantized base re-rounds the weights and can lose quality, so merge in 16-bit, re-quantize and re-test, or keep the adapter separate. </details>

Go deeper

Prev: Dataset Preparation · Next: QLoRA and Quantized Training · Back to phase: Phase 8