Introduction
Before you build a model, get to know your data the way a detective studies a crime scene. Which values are typical? Which are strange? What seems to go with what? This stage is called exploratory data analysis (EDA).
Good exploration:
- reveals data problems (errors, caps, odd values) before they confuse the model;
- suggests useful features and transformations;
- gives you a feel for what a reasonable prediction looks like.
Work on a copy of the training set only — the test set stays locked away (topic 4.2). If the training set is huge, explore a sample to keep things fast.
PYTHON
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split
df = fetch_california_housing(as_frame=True).frame
train_set, test_set = train_test_split(df, test_size=0.2, random_state=42)
housing = train_set.copy() # explore a copy, keep the original clean