Introduction
Some models take weeks to train on one GPU, and some are simply too big to fit on one. Distributed training spreads the work across several GPUs, often across several machines.
There are two basic ways to split the work. Think of a team writing a large report:
- data parallelism: everyone has a full copy of the report template and works on a different chapter of research data, then they merge their notes. Each GPU has a full copy of the model and processes a different slice of the batch;
- model parallelism: the report is so huge that each person is responsible for different sections of it. Each GPU holds part of the model.
Most training uses data parallelism; very large models combine both.