No. Phone NPUs, such as Apple’s Neural Engine, Qualcomm’s Hexagon and the TPU in Google’s Tensor chips, mostly run the phone maker’s own AI features. Apps that run open language models generally use the GPU, or the CPU when there’s no supported GPU. What decides whether a phone runs a local chatbot well is RAM first, then memory bandwidth and GPU support. The TOPS figure on the spec sheet barely enters into it.
What is an NPU? #
A neural processing unit is a block of the phone’s chip built for the math neural networks do most: multiplying large grids of numbers at low precision. Doing that on dedicated silicon uses far less power than doing it on the main processor, which is why phones have had NPUs for years.
| Chip | What it’s built for | Role in running a chatbot |
|---|---|---|
| CPU | General tasks, a few things at a time, quickly | Fallback when no GPU support exists; slower |
| GPU | Thousands of small calculations in parallel | The main engine in most local chat apps |
| NPU | Neural network math at low power and low precision | Runs the maker’s own AI features; awkward for third-party apps |
Every vendor brands its own: Apple’s Neural Engine in A-series chips, Qualcomm’s Hexagon in Snapdragon, Google’s TPU in Tensor, Samsung’s NPU in Exynos, MediaTek’s APU in Dimensity. Performance is quoted in TOPS, trillions of operations per second.
What uses the NPU on your phone? #
Mostly the features built in by the phone maker:
- Camera processing: portrait blur, night mode, scene detection.
- Face recognition and other biometrics.
- Voice and dictation, where speech recognition runs on the device.
- Photo search and organization, like finding “dog” or grouping faces.
- Built-in AI features, such as the models behind Apple Intelligence or Gemini Nano.
These are tuned by companies that know their own silicon intimately and can convert their models into whatever format the NPU expects. They’re also small models that fit in fast memory, which is exactly the case NPUs were designed for.
Why do most local LLM apps use the GPU or CPU instead? #
Three reasons.
Generating text is limited by memory, not math. To produce each token, the runtime reads essentially the entire model out of memory. A 2.7 GB model moves about 2.7 GB through the memory system per token. The rough ceiling is:
tokens per second ≈ memory bandwidth ÷ model size
The NPU, GPU and CPU all share the same memory, so a faster calculator doesn’t help much when everything is waiting on the same data. That’s why a 5.7 GB model runs at roughly half the speed of a 2.7 GB one on the same phone, and why shrinking a model speeds it up. Reading your prompt is the opposite case: that phase processes many tokens at once and is compute-heavy, which is where extra compute does help, and it’s what you feel when you attach a long document or a photo.
NPUs are hard to program for arbitrary models. Each vendor has its own toolkit, and models usually have to be converted into a vendor-specific format, often with fixed input sizes. A model that runs on one company’s NPU won’t run on another’s without separate work. That’s manageable for a phone maker shipping one model; it’s a burden for an app that lets you load dozens.
The GPU path is mature and keeps up. llama.cpp, the open-source engine behind many local AI apps, supports Apple’s Metal, OpenCL for Qualcomm’s Adreno GPUs, Vulkan and plain CPU execution, and it runs anything distributed as a GGUF file. When a new model family ships with a different design, the GPU and CPU code paths are usually updated within days or weeks, while vendor NPU toolchains lag. llama.cpp has added a Hexagon backend for Snapdragon NPUs, but GPU and CPU remain the broad, reliable paths.
What does your phone’s chip actually run? #
| Your phone’s chip | What llama.cpp-based chat apps typically use |
|---|---|
| Apple A-series (any iPhone on iOS 15.1+) | GPU via Metal |
| Snapdragon with Adreno 700-series or newer GPU | GPU via OpenCL |
| Google Tensor (Pixel) | CPU in most such apps |
| Most MediaTek and Exynos chips, including Exynos Galaxy phones | CPU in most such apps |
An “AI phone” badge therefore says little about how well the phone runs a chat app. An Apple Intelligence iPhone uses its Neural Engine for Apple’s features while a third-party model on the same phone runs on the GPU.
Which processor does Personal LLM use? #
Personal LLM runs its models through llama.cpp. On iPhone it uses the GPU through Metal. On Android it uses OpenCL on Snapdragon phones with Adreno 700-series and newer GPUs, and falls back to the CPU on everything else. It doesn’t use the NPU.
That’s why the app’s device check reads your RAM rather than your NPU. Each model card shows a “Fits your device” badge based on your phone’s actual memory before you download, and the built-in benchmark reports how fast a model generates on your hardware and whether it ran on the GPU or the CPU.
Do NPUs help language models at all? #
They can. NPUs are efficient at the compute-heavy prompt-processing phase, and chip makers demonstrate language models running on their NPUs with their own toolkits. Phone makers’ built-in assistants lean on them heavily. As tooling matures, more open-source engines may use NPUs for parts of the work.
For a buyer, the takeaway is simple: a strong NPU doesn’t hurt, but it isn’t the spec that decides whether a chat app runs a 4B or a 9B model well.
What matters more than the NPU? #
| Spec | Why it matters for local AI | Where to find it |
|---|---|---|
| RAM | Decides which models fit at all | Spec sheets for Android; reviews for iPhone, since Apple doesn’t publish it |
| Memory bandwidth | Sets how fast tokens are generated | Newer chips with newer memory are faster |
| A GPU the app supports | Speeds up both prompt processing and generation | Metal on every iPhone; Adreno 700-series and newer on Snapdragon |
| Cooling | Long sessions slow down as the phone warms | Reviews; larger phones usually hold speed longer |
| Free storage | Models are 0.8 to 6 GB each | Settings > Storage |
TOPS is missing from that list on purpose. It’s a peak figure at low precision, measured differently by each vendor, and it describes compute when generation is limited by memory. A phone with a smaller TOPS number and more RAM is usually the better local AI phone. What TOPS means in AI phone specs takes that number apart properly.
For applying all this to real hardware, see the best phones for running a local LLM, and for the memory side, how much RAM you need to run an LLM on a phone.
Why does my phone get hot and slow down? #
Generating text keeps the GPU and memory busy continuously, and phones have no fans, so after a few minutes of heavy use they lower clock speeds to stay cool and tokens per second drop. A quick benchmark shows your phone at its best; a long session shows it warm. Why your phone gets hot running AI has the fixes.
Does using the GPU instead of the NPU drain more battery? #
Running a language model is demanding whichever block does it, and the GPU does draw more power than an NPU would for equivalent work. In practice the model’s size and the length of the reply matter more than which processor runs it. Does local AI drain your phone battery shows how to measure it.
Frequently asked questions #
Does llama.cpp use the NPU? #
It has a Hexagon backend for Snapdragon NPUs, but most apps built on llama.cpp run models on the GPU (Metal on iPhone, OpenCL on Adreno) or the CPU. Those paths cover far more phones and far more models.
Is Apple’s Neural Engine used for AI chat apps? #
Apple’s own features use it as part of Apple Intelligence, and apps built with Core ML can reach it. Third-party chat apps running open GGUF models through llama.cpp use the iPhone’s GPU through Metal instead.
Do I need an “AI phone” to run a local LLM? #
No. Any phone with enough RAM can run a small model, and GPU support matters more than an NPU for chat apps. A recent iPhone, or a Snapdragon phone with 6 GB of RAM or more, runs 2B to 4B models comfortably.
Is a phone with a better NPU faster at local AI chat? #
Not necessarily. Chat speed depends mostly on memory bandwidth, the GPU the app can use and the model’s size, and whether a model runs at all depends on RAM. Check RAM and chip generation first.