Introduction
Neural networks work with numbers, not letters. Before a model can read a sentence, the text must be split into small units called tokens, and each token mapped to an integer ID from a fixed vocabulary. This step is tokenization, and it shapes everything that follows: how long sequences are, how big the model's vocabulary is, and whether it can handle words it has never seen.
Think of Lego. You can build any model from a fixed set of brick types. If your bricks are huge (whole words), you need a gigantic catalogue and you're stuck when a new shape appears. If they're tiny (single letters), you can build anything, but every model needs thousands of bricks. Subword tokens are the medium-sized bricks: a manageable catalogue that can still build any word.