Introduction
After each neuron computes its weighted sum, it passes the result through an activation function. This small step is what gives neural networks their power. Without it, a network of any depth would be no more capable than a single straight line.
Think of activation functions as the bends in a pipe cleaner. A straight pipe cleaner can only make a straight line. Add bends, and you can shape it into anything — a star, a letter, a spiral. Each activation adds a "bend", and many bends together can form any shape.