Introduction

Introduction

Gradient boosting is arguably the most powerful classic ML algorithm for tabular data (spreadsheets, databases). Popular implementations — XGBoost, LightGBM, CatBoost and scikit-learn's HistGradientBoosting — win a huge share of competitions and power many real systems: credit scoring, ranking search results, predicting delivery times.

Like AdaBoost, it adds models one after another, each correcting the previous ones. But instead of re-weighting examples, each new model directly learns the errors that are still left.

Imagine estimating a friend's height by guessing in rounds:

1. "I guess 170 cm." — The truth is 182 cm; you're 12 cm short.
2. Next round you don't guess the height again — you guess the remaining error: "+10 cm".
3. Now you're 2 cm short, so the next guess is "+2 cm"…

Each round corrects what's left over. That's gradient boosting.