Sovereignty · 2026-09-16 · 1:40

Running a 27B model locally: it's the memory pipe, not the chip

Name: Running a 27B model locally: it's the memory pipe, not the chip
Uploaded: 2026-09-16
Duration: 1 min 40 s
Description: For local, sovereign LLMs the rule is blunt: buy VRAM, not bandwidth. Last year's model that you own at zero cost per token often beats this year's that you rent.

A hands-on test of a 27B model on integrated graphics shows the real bottleneck: on shared memory, every token streams all the weights past the chip, so speed is bound by memory bandwidth, not raw compute.

In favour

Free wins that actually help: a tighter quantisation that fits in VRAM, plus speculative decoding, roughly double throughput.
An external GPU over USB4/Thunderbolt is enough for inference — text generation barely uses the link once weights are resident.

Worth watching

Integrated GPUs stay slow for large dense models however you tune them.
Blindly offloading all layers when the model does not fit makes it dramatically slower, not faster.

Our takeFor local, sovereign LLMs the rule is blunt: buy VRAM, not bandwidth. Last year's model that you own at zero cost per token often beats this year's that you rent.

Source: «I ran a 27B model on a hand-sized PC» (hardware review)

← All Focus posts

AI-narrated · AI-generated content

Transcript

A 27B parameter model, running on integrated graphics. No separate graphics card. So what actually sets how fast it can answer? A hands-on test put exactly that setup to work. Integrated graphics have no memory of their own. They share the system's memory, and the model's weights live there too. And that is the real bottleneck. For every token, all of the model's weights stream past the chip. So speed is bound by memory bandwidth, not by raw compute. There are free wins that actually help. A tighter quantisation that fits in video memory, plus speculative decoding. Together, they roughly double throughput. And an external graphics card, plugged in over a USB4 connection, is enough for inference. Once the weights are resident, text generation barely uses the link. Two things to keep straight. First, integrated graphics stay slow for large, dense models. And that holds however you tune them. Second, when the model does not fit, blindly offloading all of its layers does not help. It makes it dramatically slower, not faster. So for local, sovereign language models, the rule is blunt. Buy video memory, not bandwidth. Last year's model that you own, at zero cost per token, often beats this year's that you rent.