NOTES / INFERENCE AND SERVING

Mixture of experts

Sparse models route each token to a few expert sub-networks, so compute per token is lower than total size suggests, while memory still holds every expert.

UPDATED 18 NOV 2026#inference#mixture-of-experts#sparse-models#mixtral

The idea

In a standard (dense) transformer, every token passes through every parameter. In a sparse mixture-of-experts (MoE) model, some layers contain several parallel feed-forward blocks called experts, and a small router chooses which few to use for each token.

Mixtral 8x7B is the standard worked example. It has the same decoder-only design as Mistral 7B, but each layer has 8 feed-forward blocks. For every token at every layer, a router network picks two of them and combines their outputs as a weighted sum1.

Diagram: a token goes to a router that picks two of eight experts, whose outputs are combined by a weighted sum

Diagram: my own.

Total versus active parameters

Because only two experts run per token, the model touches only part of its parameters each time. The paper puts Mixtral at 47 billion parameters in total, of which about 13 billion are active per token1. So its per-token compute resembles a much smaller dense model, while its capacity resembles a larger one. The authors report that it matches or beats Llama 2 70B and GPT-3.5 on most benchmarks they tested1; that comparison is theirs and dates from early 2024.

The idea is older than Mixtral. Google’s Switch Transformer simplified routing and scaled sparse models to over a trillion parameters, with the sparsity coming from activating a subset of weights for each example3.

What it means for serving

  • Compute per token is lower. Mixtral’s authors say sparsity allows faster inference at low batch sizes and higher throughput at large ones1.
  • Memory is not. All experts must be loaded even though few run. Hugging Face notes that serving Mixtral needs enough VRAM to hold roughly a 47-billion-parameter model, not a 13-billion one2.
  • Batches touch many experts. With enough tokens in a batch, most experts are used each step, so the bandwidth saving shrinks at large batch sizes (my understanding; the Mixtral paper’s own statement about batch sizes is more favourable, so measure).
  • Multi-GPU traffic. Large MoE models are usually spread across devices, with tokens sent to the GPUs that hold their experts, which adds communication cost (my summary of common practice; see the engine docs).
  • Load balance matters. If the router favours a few experts, those become bottlenecks. Training typically adds a loss term to encourage even use, and newer designs use other balancing schemes4.

Diagram: a trade-off between lower compute per token and the need to hold all experts in memory

Diagram: my own, reusing the memory arithmetic from Quantisation. An MoE needs memory for its total size, as a dense model of that size would.

Why this matters to you

Most application developers meet MoE as a property of the model they pick, not something they build.

  1. Read both numbers. A listing like “47B total, 13B active” tells you memory needs and compute per token respectively.
  2. Expect different cost profiles. Per-token API prices for MoE models can be lower than their size suggests, but you cannot infer prices from parameters; check the pricing page.
  3. Self-hosting needs memory planning for the total size, plus the KV cache (Serving stacks).
  4. Quantise carefully. Experts hold most of the parameters, so they are the main target for quantisation; test for quality loss.
  5. Fine-tuning is harder than for dense models, according to Hugging Face’s overview, though recent work on instruction-tuning MoEs is promising2 (When to fine-tune).

Do not assume bigger total size means better results on your task. Compare candidate models on your own eval set.

Sources

  1. [1]Mixtral of Experts — Jiang et al., Mistral AI · 2024PARAPHRASED
  2. [2]Mixture of Experts Explained — Hugging Face · 2023PARAPHRASED
  3. [3]Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity — Fedus, Zoph, Shazeer · 2021PARAPHRASED
  4. [4]Mixture of Experts: many experts, only a few wake up — TechJack SolutionsLOAD BALANCING AND ROUTING VARIANTS, AS SUMMARISED BY A THIRD PARTY

© Ashish Kumar. All rights reserved. Quoted and cited material belongs to its owners. How I attribute sources.

Product, company and framework names are trademarks of their respective owners. Mentioning them does not imply affiliation or endorsement.