Introduction
Choosing the output layer and the loss is the most important design decision after picking the architecture. Get it wrong and the network optimises the wrong thing — sometimes without any error message, just mysteriously bad results.
The loss function is the network's definition of success. It's like the scoring rules of a game: change the rules and players change their strategy. Choose a loss that rewards exactly what you care about.