Sovereignty · 2026-09-16 · 1:42

Why a sparse model can outrun a bigger one on a small GPU

Name: Why a sparse model can outrun a bigger one on a small GPU
Uploaded: 2026-09-16
Duration: 1 min 42 s
Description: Two honest paths for a modest local box: a dense model with an external GPU, or a sparse mixture-of-experts that stays fast on the integrated one. Match the architecture to the bottleneck.

On integrated GPUs, generation speed is limited by memory bandwidth: every token streams the active weights past the chip. That is why a mixture-of-experts model that activates only a few billion parameters per token can feel faster than a smaller dense one.

In favour

Sparse (mixture-of-experts) models with few active parameters suit bandwidth-bound hardware well.
It gives budget-friendly local setups a path to larger models without a discrete GPU.

Worth watching

Total model size still has to fit in memory, even if only part is active per token.
Benchmark claims move fast; verify a specific model on your own workload before trusting the numbers.

Our takeTwo honest paths for a modest local box: a dense model with an external GPU, or a sparse mixture-of-experts that stays fast on the integrated one. Match the architecture to the bottleneck.

Source: On mixture-of-experts and memory-bound hardware

← All Focus posts

AI-narrated · AI-generated content

Transcript

On a small integrated GPU, a larger model can feel faster than a smaller one. That sounds backwards. So what actually decides the speed? On integrated GPUs, generation speed is limited by memory bandwidth. Every token streams the active weights past the chip. So what matters is how much has to move. That is why a mixture-of-experts model can feel faster than a smaller dense one. It activates only a few billion parameters per token, so less has to stream past the chip. The case in favour. Sparse models, with few active parameters, suit bandwidth-bound hardware well. The less each token has to move, the better they fit the limit. And for budget-friendly local setups, it opens a path to larger models, without a discrete GPU in the box. Two things are worth watching. First, the total model size still has to fit in memory. Only part of it is active per token, but all of it has to be there. Second, benchmark claims move fast. Before you trust the numbers, verify a specific model on your own workload. Our take. There are two honest paths for a modest local box. A dense model with an external GPU. Or a sparse mixture-of-experts that stays fast on the integrated one. Match the architecture to the bottleneck.