Running a private AI assistant on a Raspberry Pi 5

PHNTM One's whole promise is one sentence: nothing you say leaves the box. That sentence has a price, and the price is that everything — the language model, the embedding model, the assistant's memory, the touchscreen UI, and the voice — has to fit and stay responsive on a Raspberry Pi 5 with 8 GB of RAM. This post is what that actually looks like.

The budget

8 GB sounds like plenty until you write the ledger down. The OS, the display stack, and the assistant's own services want their gigabyte. A quantized small language model in the 7–8B class wants four to five more. The embedding model that powers memory recall needs its share, and then you still need real headroom for context, audio, and the browser engine that drives the touchscreen. There is no slack — every resident megabyte is a decision.

The eviction trap

The sneakiest failure wasn't a crash — it was the model runner helpfully evicting the language model to make room for an embedding batch, then reloading it on the next question. Everything still "worked." It just meant the assistant sometimes took an agonizing pause while gigabytes reloaded from storage. The fix was unglamorous: pin what must stay warm, give memory-recall its own bounded slice, and add a warm-keeper service whose only job is making sure the first question of the morning isn't the slow one.

What a small model is honestly good for

A 7–8B-class model on a Pi is not a frontier model, and pretending otherwise would sink the product. What it is good for: conversation, questions about your own stuff (backed by local memory and retrieval), summarizing, drafting, being a companion with real context about your world. The device always shows which brain answered and how it's running. And for the tasks where you genuinely want a big model, there's an optional bring-your-own-key mode — clearly labeled, off by default, honest that those prompts leave the box.

The trade is: a beat slower and a notch less brilliant, but completely, verifiably yours. It turns out a lot of people want exactly that trade.

Things that bit us (so they don't bite you)

• Request-time context settings on some local runners are silently ignored — bake the context length into the model configuration itself, then verify it.
• Audio devices renumber across reboots and kernel updates. Resolve sound hardware by what it is, never by card number.
• A model runner can die in ways that leave its API answering health checks while every actual request hangs. Health-check the work, not the port.
• Burn-in matters. We run every build for 72 hours straight — memory, thermals, restarts — before we trust it. Boring, and it catches things nothing else does.

Why bother?

Because the alternative is a microphone in your house that streams your life to a data center, and "we promise we delete it" as a business model. Making a private assistant work on $150 of compute is a genuine engineering fight — but it's winnable, and once it's won, the privacy isn't a policy. It's physics: the wire isn't there.

PHNTM One is a private AI appliance being built by hand — see the specs or join the waitlist. Questions? Email me — I answer everything myself.