Running a private AI assistant on a Raspberry Pi 5
Hardware note, October 9, 2026: development drawings described a 256 GB NVMe M.2 SSD; current units use a 256 GB microSD card. Historical benchmark results below are unchanged.
PHNTM One is a Raspberry Pi 5 AI assistant that runs its conversation model, memory, document retrieval and voice on the device. In its default Private mode, your conversation is processed locally. Optional cloud modes are separate choices and send data to the services you enable.
The engineering challenge is fitting all of that beside a touchscreen interface in 8 GB of RAM. Here is the hardware, the software stack, the failures that shaped it, and the evidence you can inspect before ordering one.
Try the interactive PHNTM One demo for the product experience, or use our offline AI guide to compare a desktop runner, a DIY build and an appliance. The browser demo simulates the interface.
Can a Raspberry Pi run an AI assistant?
Yes. A Raspberry Pi 5 with 8 GB of RAM can run a small language model locally, so a Raspberry Pi AI assistant can hold a conversation, answer questions about your own notes and documents, and talk back without a cloud AI service. What it cannot do is match a large cloud model on speed or difficult reasoning: a short answer takes tens of seconds, not one or two.
A useful Raspberry Pi personal assistant needs more than a model runner:
- A model that fits: a 4B-class model such as Gemma 3 4B, sized to leave room for everything else.
- Memory: an embedding model and local search, so it can recall what you told it and find passages in your documents.
- Voice: speech-to-text and text-to-speech that also run on the device.
- An interface: a screen, keyboard or phone page you can actually use every day.
- Recovery: something that notices when a worker stops doing useful work and fixes it.
The rest of this post covers each of those on PHNTM One and the trade-offs that shaped them. If you are comparing a Pi build with other private AI devices, the private AI appliance buyer's guide lists what to check before you buy.
The Raspberry Pi AI assistant hardware
- Compute: Raspberry Pi 5, 8 GB RAM, quad-core Cortex-A76.
- Interface: 10.1-inch 1280×800 IPS touchscreen in an aluminium case, USB speaker, USB microphone and mini wireless keyboard.
- Storage and power: 256 GB microSD card, installed, passive cooling through the aluminium case and a 27 W USB-C power supply (100–240 V, US plug) that plugs into the bottom edge of the display, which powers the Raspberry Pi 5. Older prototype measurements remain separate.
The full specifications distinguish the supplied configuration from upgrades. The bill of materials lists the parts and their total cost.
Local model, memory and voice stack
The published stack uses Debian 13, Ollama / llama.cpp for inference, Gemma 3 4B with a Llama 3.2 3B fallback, and bge-m3 for memory and document retrieval. whisper.cpp transcribes a tap-to-talk request locally; Piper generates the spoken reply. Voice processing shares the same memory budget as chat.
A model runner is one component. A usable assistant also needs document ingestion, remembered context, a clear indication of the answering model, and recovery when a worker stops doing useful work. Those are the parts that make this an appliance.
The budget
8 GB sounds like plenty until you write the ledger down. The OS, the display stack, and the assistant's own services want their gigabyte. The quantized 4B model (Gemma 3 4B, Q4_K_M) has model files of a few gigabytes, but its runtime memory also includes context buffers and runner overhead; anything in the 7–8B class wants four to five, and that is the line that doesn't fit next to everything else. The embedding model that powers memory recall needs its share, and then you still need real headroom for context, audio, and the browser engine that drives the touchscreen. There is no slack — every resident megabyte is a decision.
Treat RAM like a spreadsheet, not a vibe. Every model that stays warm is a line item with a number next to it. The one time we let two things "temporarily" coexist without doing the math, the out-of-memory killer did the math for us.
The eviction trap
The sneakiest failure wasn't a crash — it was the model runner helpfully evicting the language model to make room for an embedding batch, then reloading it on the next question. Everything still "worked." It just meant the assistant sometimes took an agonizing pause while gigabytes reloaded from storage. The fix was unglamorous: set model residency in exactly one place instead of letting each code path decide for itself, give memory-recall its own bounded slice, and warm the model at boot.
Update, September 2026: we later chose to let an idle box release the model after about four idle hours rather than hold two thirds of its RAM forever. The honest cost: the first answer after a long idle spell is slower while it reloads, then it's back to its usual pace. More in the FAQ.
Update, November 2026: there is no fixed idle time any more. The device tries to keep the model ready, but residency is not permanent: after a long quiet spell the first answer can take about two minutes while the model reloads, and later answers are faster.
What a small model is honestly good for
A 4B-class model on a Pi — we ship Gemma 3 4B — is not a frontier model, and pretending otherwise would sink the product. What it is good for: conversation, questions about your own stuff (backed by local memory and retrieval), summarizing, drafting, being a companion with real context about your world. The device always shows which brain answered and how it's running. And for the tasks where you genuinely want a big model, there's an optional bring-your-own-key mode — clearly labeled, off by default, honest that those prompts leave the box.
The trade is: a beat slower and a notch less brilliant, but completely, verifiably yours. It turns out a lot of people want exactly that trade.
Things that bit us (so they don't bite you)
- Request-time context settings on some local runners are silently ignored — bake the context length into the model configuration itself, then verify it.
- Audio devices renumber across reboots and kernel updates. Resolve sound hardware by what it is, never by card number.
- A model runner can die in ways that leave its API answering health checks while every actual request hangs. Health-check the work, not the port.
- Burn-in matters. Each shipping unit is scheduled for automated burn-in on its accepted build — memory, thermals, restarts; when available, its certificate is on the device at /docs/certificate. Boring, and it catches things nothing else does.
How fast is AI on a Raspberry Pi 5?
The proof page keeps a historical figure of about 2–3 tokens per second, measured on software 0.8; the current manual gives answer times instead. The August 2026 NVMe prototype measured 3.3–3.8 tokens per second. Those are different configurations, not interchangeable promises for the current shipping build.
A short response usually takes tens of seconds; a long or cold-model answer can take longer than a minute. On November 17, 2026, three local replies in an ongoing review-unit chat after a meeting showed their first sentence in 23–30 s and finished in 30–37 s. These are dated measurements, not a guarantee for every unit. Prompt length, output length, the selected model and other work affect latency; a cold model adds loading time. For useful comparisons, record the hardware, storage, model, software version and whether the model was already loaded.
Real hardware and checks you can repeat
To check disconnected operation, select Private mode, disconnect external integrations and remove the device's internet connection while keeping it powered. Ask a new question, recall a saved memory and inspect a cited local document. Use the verification guide for the connected packet-capture check. Read unedited transcripts to see actual answers, including mistakes.
DIY Raspberry Pi AI assistant vs. PHNTM One
A DIY build lets you choose every component and software project. PHNTM One combines the hardware and assistant software in a device built and supported by PHNTM. Both approaches still face the memory and speed limits of a Raspberry Pi 5.
On a small screen, scroll the table sideways to compare both options.
| What to compare | DIY Raspberry Pi build | PHNTM One |
|---|---|---|
| Parts | Choose and source the Pi, screen or other interface, storage, power, cooling and audio hardware. | Raspberry Pi 5 with 8 GB RAM, 10.1-inch touchscreen, 256 GB microSD card, passively cooled aluminium case, power supply, USB speaker, USB mic and mini keyboard. See what is included. |
| Setup work | Assemble the hardware, install the OS and models, configure audio and connect your chosen services. | Assembly and software integration are part of the product. Complete the owner setup when your unit arrives. |
| Assistant software | Your chosen projects determine which chat, memory, document and voice features are available and how they work together. | A touchscreen interface with local chat, memory, document search and tap-to-talk voice. Explore the included features. |
| Maintenance | You select, test and maintain the OS, models and software integrations. | Integrated recovery and owner-approved software updates. See how device care works and the help center. |
| Support | Hardware vendors and the maintainers or communities of your chosen software, under their respective terms. | Direct contact with the builder and a 1-year limited hardware warranty under the published terms. |
| Cost | Depends on the parts you already own and the configuration you choose. Our published BOM records $409.95 in parts and packaging as paid in August 2026; that excludes your setup time and is not a current DIY quote. | $799 one-time, plus applicable sales tax. No required subscription for local use. Optional cloud services use your own provider account and may add charges. |
Choose DIY if you want to build and maintain your own system. Choose PHNTM One if you want the integrated device and support from its builder. The hardware does not become faster simply because it is sold as an appliance; use the performance figures above to set expectations.
Made-to-order purchasing is open at $799 plus applicable tax, with free US shipping. Units ship within 6–8 weeks after payment; EU orders are pre-orders with no ship date yet. The interactive demo simulates the interface so you can explore before deciding.
Why bother?
Because people should be able to choose where their conversations are processed and keep using their assistant when an internet service is unavailable. A dedicated local device makes that choice tangible. The trade is smaller models and slower answers on affordable hardware, with the files and operation available for inspection.
Technical references
- Raspberry Pi 5 hardware
- Ollama: model residency, memory and local processing
- Google: Gemma models
- whisper.cpp speech recognition
PHNTM One is a private AI appliance being built by hand — see the specs or explore the demo or order PHNTM One. Questions? Email me — I answer everything myself.
Own your AI. Keep your words.
Order your PHNTM One. Each unit must complete an automated burn-in on its shipping build before shipment. $799 once, with no required subscription for local use.
$799 one-time · ships within 6–8 weeks after payment (EU: pre-order) · free US shipping · ships worldwide · 30-day returns