LoRA and QLoRA Explained: Fine-Tuning a 7B Model Without 100 GB of VRAM

LoRA only makes sense once you see where it sits between pre-training and fine-tuning, so start there.

Pre-training versus fine-tuning

Pre-training happens when a model is first built. It learns from sequences of text on its own, with no labels: the training text itself supplies both inputs and targets, and the result is a general-purpose base model.

That base model has plenty of knowledge, but it doesn't follow a particular format, pattern or instruction. Quite often it just continues whatever you typed and tacks more words onto the end. Fine-tuning fixes this. You train it on labeled examples made of an instruction, an input and a desired response, so it learns what good answers look like.

Suppose you want a model for legal queries. You wouldn't train from scratch when a pre-trained model with plenty of domain knowledge already exists. You fine-tune it so it replies efficiently.

Why full fine-tuning is painful

Updating every parameter eats memory. Take a 7B Llama model with this setup:

Model size: 7 billion parameters
Training dtype: BF16 / FP16
Optimizer: AdamW
Sequence length: 2048

Just loading the base model takes 14 GB. In full fine-tuning every parameter is trainable, so each one needs a gradient, and if gradients are stored in BF16 or FP16 that's another 14 GB. The AdamW optimizer then asks for roughly 56 GB more, and it may keep an FP32 copy of the parameters, adding around 28 GB. Activations take some too. Altogether you're looking at 80 to 100 GB, far too much for a narrow, task-specific job that isn't even very complicated.

What LoRA does

LoRA stands for Low-Rank Adaptation. It fine-tunes a model without updating every weight individually. All original weights stay frozen, and a pair of small matrices does the learning.

How can small matrices steer a big model? Think about what a pre-trained model already knows: English, grammar, reasoning patterns, programming, facts. If you now want it to answer legal questions in a particular style, it doesn't have to relearn language. You only need to steer what's there. LoRA bets that such a change can usually be captured in a lower-dimensional structure:

ΔW = AB

So the billions of weights the model has learned stay untouched. LoRA leaves W alone and adds a correction on top:

W_new = W + ΔW

Working through the numbers

Imagine a transformer with one linear layer:

X -> W -> Output

Give W the shape 1000 by 1000. It then contains:

1000 x 1000 = 1,000,000 parameters

Full fine-tuning trains every one of those, and real LLMs have matrices like this everywhere, so a 7B model has about 7B parameters to train.

With LoRA you build two skinny matrices, A and B, and choose a rank, for example r = 8:

A = [1000 x 8]
b = [8 x 1000]

Their product has the same shape as W, so the sum is valid:

W_new = W + AB

[1000×1000] + [1000×1000]

The saving comes from how few parameters the two skinny matrices hold:

A: 1000 × 8 = 8,000
B: 8 × 1000 = 8,000

Total = 16,000 parameters

Compare the two approaches:

Full fine-tuning:
1,000,000 parameters

LoRA:
16,000 parameters

For this layer that's just 1.6% as many trainable parameters.

What rank means

Rank is one of LoRA's main hyperparameters, and it sets how much capacity the adapter has. A low rank such as 4 or 8 gives a smaller adapter that uses less memory but is less flexible. Complex tasks may need 32, 64 or 128. Higher rank costs more memory, and it doesn't automatically mean better quality, since it can add noise. Try several ranks and keep whichever works best.

Where the adapters go

LoRA attaches small trainable matrices to a frozen transformer, but a transformer contains many matrices. A simplified decoder-only block holds several, and a model may stack that block 32, 40, 80 or more times. The question LoRA asks is which of those W matrices deserve a trainable BA correction. Three common strategies:

Strategy 1: q_proj and v_proj only. An adapter on q_proj can change how representations form queries for relevant context, while one on v_proj changes the information passed along from attended tokens.

Strategy 2: every attention projection. Add adapters to q_proj, k_proj, v_proj and o_proj, and LoRA can modify all the main learned linear transformations in attention. You get more adaptability and more trainable parameters.

Strategy 3: attention projections plus MLP layers. This is the larger, strong general-purpose setup:

ATTENTION

q_proj       ✅
k_proj       ✅
v_proj       ✅
o_proj       ✅


MLP

gate_proj    ✅
up_proj      ✅
down_proj    ✅

Transformers aren't only attention. The MLP does a huge amount of learned feature transformation, so covering it gives fine-tuning noticeably more freedom.

Did LoRA fix the memory problem?

No, not completely. It cuts trainable parameters by around 99%, but GPU memory doesn't fall by 99%, since the base model still has to be loaded and the forward pass still runs, with a little extra computation on top. The real win is in gradients and optimizer states. You no longer have to keep gradients for 7B parameters.

QLoRA

QLoRA stands for quantized low-rank adaptation and takes the idea a step further. Because the base weights are frozen anyway, they can be quantized to 4-bit. A 7B model that needs 14 GB in BF16 shrinks to roughly 3.5 to 5 GB. This is why a 7B model can be tuned with QLoRA on one gaming-class graphics card, a job that plain full fine-tuning would spread across several high-memory GPUs.