Introduction

Introduction

ReLU, $\max(0, z)$, was a breakthrough. It doesn't saturate for positive inputs, it's cheap to compute, and it made deep networks trainable. But it has a weak spot: for negative inputs its output is 0 and its slope is 0. A neuron stuck there gets no gradient and can't learn its way out.

Think of a light switch with a sticky spring. Push it down too far and it stays stuck off, and no amount of gentle pressure brings it back. The activations in this topic are switches with a "gentler spring": they let a little signal through even in the "off" position, so neurons can always recover.