Introduction
Bigger models generally perform better, but in a normal ("dense") network, every parameter is used for every input. Doubling the parameters doubles the computation for every token. That gets very expensive.
A mixture of experts (MoE) breaks this link. The model contains many expert sub-networks, but for each input, a small router picks only a few of them to run. The model can have a huge total number of parameters (lots of knowledge), while each token only pays for the few experts it uses.
Think of a large hospital. It employs hundreds of specialists, but each patient sees only one or two of them, chosen by a triage nurse at the entrance. The hospital as a whole knows a lot; each visit is still quick. The specialists are the experts, the triage nurse is the router.
MoE layers are used in several of the largest language models (e.g. Mixtral, DeepSeek-V3, and reportedly some proprietary frontier models), and also in vision and multimodal models.