Introduction
Some optimization problems are easy and some are hopeless. The dividing line is usually convexity. For convex problems, every local minimum is a global minimum, so gradient descent can't get stuck in a bad valley and solvers can certify optimality. Linear regression, logistic regression, SVMs and lasso are all convex — which is why they are so reliable.