LoRA & QLoRA: Low-Rank Intrinsic Dimension and SVD Decomposed for Engineers

1. The True Memory Cost of Full Fine-Tuning

Why can't engineers simply fine-tune large foundation models with standard backpropagation on consumer hardware?

Most people assume the memory requirement is just the size of the model weights. In reality, full training requires:

1. Model Weights: $2 \text{ bytes / parameter}$ (in FP16).
2. Gradients: $2 \text{ bytes / parameter}$.
3. Adam Optimizer States:
   • First momentum vector $m_t$: $4 \text{ bytes / parameter}$ (in FP32).
   • Second variance vector $v_t$: $4 \text{ bytes / parameter}$ (in FP32).
   • Master FP32 weights copy: $4 \text{ bytes / parameter}$.

For a 70B parameter model: $\text{Total VRAM} = 70 \times (2 + 2 + 4 + 4 + 4) = 70 \times 16 \text{ GB} = \mathbf{1,120 \text{ GB of VRAM}}$

You would need a cluster of fourteen $80\text{GB}$ NVIDIA A100 GPUs just to update weights!

2. The Low-Rank Intrinsic Dimension Hypothesis

In 2020, Aghajanyan et al. published an astonishing empirical finding: although pre-trained foundation models contain billions of weights, the parameter updates $\Delta W$ necessary to adapt to a downstream task reside in a subspace of extremely low intrinsic dimensionality.

In linear algebraic terms: a dense transformation matrix $W_0 \in \mathbb{R}^{d \times k}$ may have full rank $\min(d, k)$, but the task-specific update matrix $\Delta W$ satisfies:

$\text{rank}(\Delta W) \ll \min(d, k)$

Often, an intrinsic rank of $r = 4$ or $r = 8$ captures over $99\%$ of the performance of a full rank update!

3. Mathematical Decomposition: $\Delta W = B \times A$

Instead of freezing and directly optimizing the full $d \times k$ parameters of $\Delta W$, Low-Rank Adaptation (LoRA) factorizes $\Delta W$ into two low-rank matrices:

$\Delta W = B \times A$

Where:

• $W_0 \in \mathbb{R}^{d \times k}$ is the frozen pre-trained weight matrix.
• $B \in \mathbb{R}^{d \times r}$ is initialized to all zeros.
• $A \in \mathbb{R}^{r \times k}$ is initialized with Gaussian random noise $\mathcal{N}(0, \sigma^2)$.
• $r \ll \min(d, k)$ is the low-rank rank bottleneck.

The modified forward pass is:

$h = W_0 \mathbf{x} + \Delta W \mathbf{x} = W_0 \mathbf{x} + \frac{\alpha}{r} B A \mathbf{x}$

The Parameter Reduction Ratio

For a matrix of size $4,096 \times 4,096$:

• Full rank parameters: $4096 \times 4096 = \mathbf{16,777,216}$
• Rank $r = 8$ LoRA parameters: $(4096 \times 8) + (8 \times 4096) = \mathbf{65,536}$

That is a $256\times$ reduction in trainable parameters!

Because $B$ is initialized to zero, at step $t = 0$: $\Delta W = 0 \times A = 0$ The model begins training in the exact identical state as the pre-trained model with no sudden training perturbation.

4. The Scaling Factor $\frac{\alpha}{r}$

Why does LoRA include the scalar multiplier $\frac{\alpha}{r}$?

When tuning hyperparameters, you might experiment with varying rank $r$ (e.g. comparing $r=4$, $r=8$, $r=16$).

Without normalization, increasing $r$ increases the scale of the adapter output $B A \mathbf{x}$, which would force you to re-tune your learning rate every time you changed $r$.

By setting $\alpha$ as a constant (e.g. $\alpha = 16$ or $2r$), the scale of adapter updates remains constant regardless of the rank choice, stabilizing the optimization dynamics.

5. Zero-Latency Weight Merging in Production

The most celebrated practical feature of LoRA is zero inference latency overhead.

In production serving, you do not need to run two parallel forward branches. Because matrix multiplication is distributive:

$h = W_0 \mathbf{x} + \left( \frac{\alpha}{r} B A \right) \mathbf{x} = \left( W_0 + \frac{\alpha}{r} B A \right) \mathbf{x} = W_{\text{merged}} \mathbf{x}$

We can literally calculate $W_{\text{merged}} = W_0 + \frac{\alpha}{r} B A$ once at deployment time and store the merged weights. The inference code runs at the exact same speed as the original model!