Introduction
Imagine you're lost on a foggy mountain and want to reach the valley floor. You can't see far, but you can feel which way the ground slopes under your feet. A sensible strategy: take a step in the steepest downhill direction, then feel again, and repeat. Eventually you reach the bottom.
That's gradient descent — the most important optimisation algorithm in machine learning. It finds good parameters for linear regression, logistic regression, SVMs and every neural network.
The "mountain" is the cost function (e.g. MSE): its height at each point is how bad the model is with those parameter values. The goal is the lowest point.