Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
ReadRoutes each token to a single expert feed-forward block, so parameter count grows without growing compute per token. The simplified mixture-of-experts behind many current large models.
My notes
The simplification is the contribution: top-1 routing instead of top-k. Made MoE something normal teams could actually train.
Details I Had to Reread
- Capacity factor: each expert has a fixed buffer. Overflowing tokens skip the layer via the residual — "dropped" tokens.
- An auxiliary load-balancing loss stops the router from sending everything to one expert.
- Router computations in float32 while the rest runs in bfloat16 — a small stability trick that mattered a lot.
- Parameters ≠ compute. A 1T-parameter MoE can be cheaper per token than a dense 100B model.