TOPS stands for trillions of operations per second. It’s a peak throughput figure for a chip’s neural processing unit, measured under ideal conditions at low numeric precision. A phone advertised at 45 TOPS can, in theory, perform 45 trillion low-precision multiply-and-add operations every second on that one block of silicon.
What it tells you about running an AI chatbot on that phone is close to nothing. Whether a model runs at all is decided by RAM. How fast it feels is mostly decided by memory bandwidth. TOPS is the most prominently advertised of the three and the least useful for this particular job.
What the number actually counts #
Three things get lost when TOPS appears on a spec sheet.
It’s a peak, not a sustained rate. The figure is the silicon’s theoretical maximum with everything perfectly fed. Real workloads stall waiting for memory, and phones throttle when they get warm.
It depends on precision. Operations at 8-bit integer precision are cheaper than at 16-bit, and 4-bit cheaper again. The same chip can honestly claim very different TOPS figures depending on which precision the vendor quotes, and vendors don’t always say. Doubling a TOPS number by switching precision is a spec-sheet change, not a hardware improvement.
It usually covers one block. Some vendors quote NPU TOPS, others a combined figure across NPU, GPU and CPU. Two products both claiming “50 TOPS” may not be counting the same silicon.
Because there’s no shared measurement standard, comparing TOPS across brands is unreliable in a way that comparing megapixels or gigabytes is not.
Where you see the number used seriously #
The clearest use is Microsoft’s hardware bar for Windows AI features. Its Copilot+ PC developer guidance defines the class as machines with an NPU that “can perform more than 40 trillion operations per second (TOPS)”, and says many of the new Windows AI features require an NPU capable of 40+ TOPS. The qualifying hardware spans Snapdragon X Elite machines along with Copilot+ PCs built on AMD Ryzen AI 300 series and Intel Core Ultra 200V series silicon.
That threshold does a specific job: it guarantees a floor for the always-on, low-power features Windows wants to ship. It isn’t a claim about how well a large language model will run, and it doesn’t transfer to phones, where makers publish these figures inconsistently if at all.
Why TOPS doesn’t predict local AI chat speed #
RAM decides whether the model runs #
A language model has to be held in memory while it generates. That’s the hard gate, and no amount of NPU throughput moves it. In a typical phone catalog, the smallest chat model is about 0.81 GB on disk and wants at least 2 GB of RAM free, a mid-sized 4B model is around 2.74 GB and wants 3 GB or more, and a 9B model at roughly 5.7 GB wants 6 to 8 GB. A phone with a headline NPU figure and 4 GB of RAM simply cannot load the larger model. How much RAM you need to run an LLM on a phone goes through the thresholds.
Generating text is memory-bound, not compute-bound #
Here’s the part the spec sheet hides. To produce each token, the runtime reads essentially the entire model out of memory. A 3 GB model produces a token by moving 3 GB through the memory system. Do that ten times a second and you need 30 GB/s of bandwidth, and compute capacity is not what’s holding you back.
This is why two phones with similar TOPS can differ substantially in tokens per second, and why memory generation and bus width matter more than the marketing number. It’s also why prompt processing, which is compute-heavy, behaves differently from generation, which is bandwidth-heavy. The same chip can look fast at one and ordinary at the other. Local LLM speed on your phone covers both numbers.
Most phone LLM apps don’t use the NPU anyway #
Popular on-device chat apps are built on runtimes like llama.cpp, which accelerate with the GPU and fall back to the CPU. So the block your TOPS figure describes is often idle while you chat. Whether you need an NPU to run AI covers the reasons, and what each phone chip actually runs.
Heat caps the peak anyway #
Sustained generation warms a phone, and warm phones throttle. Peak TOPS describes a burst. A long answer is not a burst. This is why a phone gets hot running AI.
What NPUs are genuinely good at #
None of this makes NPUs pointless. They’re excellent at what they were designed for: small models running constantly at very low power. Wake-word detection, camera pipeline work like segmentation and noise reduction, live captions, on-device text recognition, photo categorization, sensor classification. Those models are small enough to sit in fast memory, so the bandwidth problem doesn’t bite, and running them on an NPU rather than a CPU is the difference between all-day battery and none.
Language models are the outlier case, not the typical one. A phone maker’s own assistant leans on the NPU; a chat app running a 4 GB open model does not.
What to look at instead #
If your goal is running a chat model on your phone, check in this order:
- Total RAM. It sets the ceiling on model size. 6 GB opens up 9B models; 3 to 4 GB keeps you in the 2B to 4B range.
- Memory bandwidth, which usually tracks how recent the memory generation is. This is what you feel as tokens per second.
- Free storage. Models are 1 to 6 GB each, plus extra for image support.
- GPU acceleration support in the app you plan to use, which matters far more than the NPU figure.
Best phones for running a local LLM applies that ordering to real hardware.
The only benchmark that settles it #
Run the model you want on the phone you have and read the tokens per second. Nothing else transfers reliably across chips, quantizations and runtimes.
Personal LLM makes that easy to check: a live tokens-per-second readout on every reply, a built-in benchmark for any downloaded model, and a “Fits your device” badge on each model card based on your phone’s actual RAM before you download anything. It runs open models such as Qwen 3.5 and Gemma 4 on the phone itself, with Metal on iPhone and OpenCL on Snapdragon Adreno hardware and a CPU fallback. It’s free on iOS and Android and needs no account. Two minutes with it tells you more about your phone’s AI performance than any TOPS figure.
Frequently asked questions #
How many TOPS do I need to run an AI chatbot on my phone? #
There’s no threshold, because the NPU usually isn’t what’s running it. Phones with 3 GB of RAM run small models, 6 GB or more runs the larger ones, and speed follows memory bandwidth. Buy for RAM, not for TOPS.
Is a 45 TOPS phone twice as fast as a 22 TOPS phone? #
Not for text generation. Even in workloads that do use the NPU, peak figures rarely translate into proportional real-world gains, and for language models the bottleneck is memory rather than compute. Treat the ratio as marketing.
Does my phone’s NPU run local LLMs? #
Often not. Most on-device chat apps use the GPU or CPU, because mobile NPUs are built around fixed operations and specific quantization formats. Some vendor-specific apps do target the NPU, and their model choice is usually restricted as a result.
Why do manufacturers quote different TOPS for the same chip? #
Because the measurement isn’t standardized. Numbers vary by numeric precision, by whether GPU and CPU are included, and by whether the figure is a theoretical peak or a benchmark result. Check what the footnote says is being counted.
Does a higher TOPS number at least mean better battery life for AI features? #
For small always-on features, broadly yes: a capable NPU does that work at a fraction of the power a CPU would use. For a sustained chat session with a multi-gigabyte model, battery drain tracks how long the whole system stays busy, which whether local AI drains your battery covers.