What Is an NPU? Neural Processing Units Explained
An NPU (neural processing unit) is a processor built specifically to run trained neural networks, the math behind AI tasks like object detection and speech recognition. It executes billions of small multiply-accumulate operations in parallel at low precision, which lets it deliver AI performance at a fraction of the power a CPU or GPU would need for the same job. Inference is the process of running a trained model to make predictions; training is the separate, more compute-heavy process of building the model in the first place.
That is the whole idea. Everything else about NPUs is detail on top of those two sentences.
What is an NPU used for?
Anywhere AI has to run on the device itself rather than in a cloud data center: phones (photo processing, voice), laptops (Windows Copilot+ features), cars (driver assistance), and edge devices such as smart cameras and industrial sensors. On a Raspberry Pi 5, an NPU add-on board turns an $80 single-board computer into a machine that runs real-time computer vision locally, with no video ever leaving the device.
NPU vs GPU vs TPU: what is the difference?
| CPU | GPU | NPU | TPU (Tensor Processing Unit) | |
|---|---|---|---|---|
| Built for | General computing | Parallel graphics and compute | Neural network inference | Neural network training and inference (Google’s chip) |
| AI inference speed | Slowest | Fast for large models | Fast per watt for small models | Fast, mostly in Google Cloud |
| Typical power for edge inference | Edge CPUs 5 to 15 W; desktop/65 to 200+ W | Edge GPUs 10 to 60 W; desktop/75 to 450 W | 1 to 12 W typical (23 W peak burst) | Edge version (Coral) about 2 W, discontinued ecosystem |
| Precision it favors | FP64, FP32, FP16, BF16, INT8, INT4 | FP32, FP16, BF16, TF32, INT8, FP8, (INT4/FP4 on new gen GPUs) | INT8, INT4, FP16, BF16 (sometimes) | BF16, FP32, (INT8 on new gen TPUs) |
| Where you meet it | Every computer | Desktops, servers | Phones, laptops, edge add-on boards | Google Cloud; the old Coral edge modules |
What does an AI accelerator do?
An AI accelerator is any hardware designed to speed up AI and machine learning workloads. That includes GPUs, NPUs, TPUs, FPGAs, and custom ASICs. The term is broad; the chip inside your phone that handles voice recognition and the GPU in a data center training a large language model are both AI accelerators.
Is an AI accelerator the same as a GPU?
No, but a GPU is one type of AI accelerator. A GPU is a general parallel processor that happens to be good at AI; an NPU is a specialized AI accelerator purpose-built for neural network math and does almost nothing else. That specialization is why an NPU can match GPU-class inference on small models while drawing single-digit watts. IBM’s explainer puts it plainly: “An AI accelerator is any piece of hardware, including a graphics processing unit (GPU), used to speed up machine learning and deep learning models”.
What does TOPS actually mean?
TOPS is trillions of operations per second. It is the standard NPU headline number, and it is only meaningful with two qualifiers:
Precision. A TOPS figure at INT8 (8-bit integers, the standard for edge inference) is not comparable to one at INT4 or FP16. Marketing numbers sometimes quote the lowest precision the chip supports because it doubles the figure. The Raspberry Pi AI HAT+ 2 illustrates this: its Hailo-10H NPU is rated at 40 TOPS at INT4 but only 20 TOPS at INT8. Same chip, half the headline number depending on precision. Always ask: TOPS at what precision?
The rest of the system. Memory size and bandwidth, the host interface, and how well your model compiles for the chip decide real throughput. A model that does not fit in the NPU’s memory or does not quantize cleanly will not hit the headline number.
Concrete example: the 25 TOPS (INT8) DEEPX DX-M1 on the Sixfab AI HAT+ runs YOLOv8l object detection at 50.1 FPS on a Raspberry Pi 5, drawing about 3 W. That is what a TOPS number looks like when it touches reality: enough for real-time detection on multiple camera streams, not a supercomputer. DEEPX specifies the DX-M1 at 25 TOPS INT8 and 1–5 W power consumption.
At the server level, recent benchmarks show NPUs matching or exceeding GPU throughput for many inference workloads while consuming 35–70% less power.
For scale, Microsoft’s Copilot+ PC program requires an NPU of 40+ TOPS, and Apple’s M4 neural engine is rated at 38 TOPS. Edge add-on NPUs for the Raspberry Pi span roughly 13 to 40 TOPS today.
Do I need an NPU on a Raspberry Pi?
Honest answer: not always. The Pi 5’s CPU handles some AI workloads fine on its own: occasional image classification, small audio models, low-frame-rate detection where a result every second or two is acceptable.
You need an NPU when any of these are true: you want real-time video inference (object detection at 25+ FPS), you run more than one camera or model at once, you need the CPU free for your actual application, or the device runs on a power budget where a spinning, throttling CPU is not acceptable. As a rule of thumb, YOLO-class detection on the Pi 5 CPU runs at a few frames per second at high CPU load; the same model on a 13 to 26 TOPS NPU runs an order of magnitude faster at a few watts. We published measured numbers in our DEEPX vs Hailo benchmark.
What NPUs can you actually buy for a Raspberry Pi 5?
| Option | NPU | TOPS (INT8) | Typical power |
|---|---|---|---|
| Sixfab AI HAT+ | DEEPX DX-M1 family | 13 or 25 | ~3 W |
| Raspberry Pi AI HAT+ | Hailo-8L / Hailo-8 | 13 / 26 | ~2 to 5 W |
| Raspberry Pi AI HAT+ 2 | Hailo-10H | 20 (40 at INT4) | ~5 to 8 W |
| M.2 modules on a carrier (e.g. Sixfab Edge AI Expansion Board) | DEEPX, Hailo, MemryX and others | 6 to 26 | ~1 to 5 W |
For how to put one to work end to end, see the Sixfab AI HAT+ Raspberry Pi 5 Quickstart and Sixfab Edge AI Expansion Board Quickstart.
FAQ
What is an NPU in simple terms?
A chip that does one thing well: running trained neural networks fast while using very little power. It is to AI inference what a graphics card is to rendering.
Is an NPU better than a GPU for AI?
For running (not training) small and mid-size models on a device, usually yes per watt. For training models or running very large ones, GPUs remain the tool.
What is a good TOPS number?
Depends on the job, and only at a stated precision. 10 to 30 TOPS at INT8 covers real-time vision on edge devices; Microsoft sets 40+ TOPS as its Copilot+ laptop bar. More TOPS than your model can use buys nothing.
Does the Raspberry Pi 5 have a built-in NPU?
No. The Pi 5 has no NPU on board; you add one via a HAT or an M.2 module. Its CPU can run light AI workloads without one.
Do NPUs work without the cloud?
Yes, that is the point. Inference happens on the device, which also means camera footage can stay on the device, a growing requirement in privacy-sensitive deployments.
Do NPUs make sense for commercial edge deployments?
Yes, when you are running real-time inference at scale (multiple cameras, 24/7 operation) or when data privacy requires on-device processing. Edge NPUs for the Pi 5 start at $63 (13 TOPS, Sixfab AI HAT+) and $70 (13 TOPS, Raspberry Pi AI HAT+); the 40 TOPS Raspberry Pi AI HAT+ 2 is $200. Whichever you pick, the NPU frees the CPU for application logic.