An interactive explainer

47 billion parameters. 13 billion used. That gap is the whole trick.

A dense model uses every parameter for every token, so making it twice as big makes every token twice as expensive. Mixture of Experts breaks that chain. A tiny router sends each token to only a few of the model's sub-networks, so the model can know a lot while each token uses a little. Watch the router work, break it on purpose, and find out that the 'experts' are not what the name suggests.

Mixture of experts · Sparse models · Token routing

AI & Machine Learning · Advanced · 11 min · How mixture of experts makes giant models affordable — an interactive explainer on ViLearn, with a simulation and a knowledge check. All explainers.

Related: How an LLM picks the next word · How gradient descent makes machines learn · Why sending the model more can make the answer worse