Introduction
Ridge and Lasso each have a weakness:
- Ridge never removes features, so with many irrelevant features the model stays cluttered.
- Lasso can behave erratically when features are strongly correlated, or when there are more features than examples — it may keep one feature from a group at random, and change its choice when the data change slightly.
Elastic Net simply mixes both penalties. It's like a recipe that's half Ridge, half Lasso — and you can turn a dial to choose the proportions.