Introduction
Fine-tuning a model with 100 million parameters is easy on one GPU. Fine-tuning a model with 7 billion or 70 billion parameters is a different story. Full fine-tuning updates every weight, so you need memory for the weights, their gradients and the optimizer's state (about 4× the model size with AdamW, topic 2.2). And if you have 20 different tasks, you'd store 20 full copies of a huge model.
Parameter-efficient fine-tuning (PEFT) keeps the pretrained model frozen and trains only a small number of new parameters (often less than 1% of the total). Memory and storage needs drop dramatically, and results are often close to full fine-tuning.
Think of a professional camera. Instead of rebuilding the camera for every kind of photo, you clip on a small lens filter or attachment. The camera stays the same; the cheap attachment adapts it to the job. PEFT methods are those attachments.