On a recent phone, a 4B model usually writes at reading pace or faster, and a 9B model is noticeably slower but smarter. The number that settles it is tokens per second on your own phone: above about 12 feels fast, 6 to 12 feels comfortable, 3 to 6 is slow, and under 3 feels like waiting. Most people read around 4 words a second, which is roughly 5 to 6 tokens.
If yours is slower than that, the cause is almost always one of four things: the model is too big for the phone, the app is running on the CPU instead of the GPU, the conversation has grown long, or the phone is hot and throttling. Here’s how to measure it, and what to change.
What does tokens per second mean? #
A token is a chunk of text, often a short word or part of one. In English, 100 tokens is roughly 75 words. Tokens per second is how fast the model writes: a 300-token answer at 10 tokens per second takes about 30 seconds to finish, though you can start reading as soon as the first words appear.
There are two speeds, and they’re limited by different things:
- Prompt speed (prefill): how fast the model reads your message, any pasted text and any images. This sets how long you wait before the first word. It’s compute-heavy, so a good GPU helps most here.
- Generation speed (decode): how fast it writes the answer, one token at a time. Each token means reading essentially the whole model out of memory, so this is limited mostly by memory bandwidth, not raw compute.
That second point is the one that explains nearly everything about phone AI speed. The size of the model file roughly sets the speed ceiling. Halving the file size roughly doubles how fast the same phone can generate.
What’s a good speed for a phone? #
These bands match the labels Personal LLM shows next to benchmark results:
| Generation speed | Label | What it feels like |
|---|---|---|
| 25+ tokens/s | Very fast | Text appears faster than you can read |
| 12 to 25 | Fast | Comfortably ahead of your reading |
| 6 to 12 | Comfortable | About reading pace |
| 3 to 6 | Slow | You catch up and wait |
| Under 3 | Very slow | Fine for short answers only |
A 2B to 4B model on a recent phone with GPU acceleration often lands in the top two bands. The same phone running a 9B model might manage a third of that.
How to benchmark a model on your phone #
For a fair comparison between models or between phones:
- Cool the phone down. Let it sit a few minutes, out of the sun and off the charger.
- Turn off Low Power Mode and close heavy apps like games, the camera or a browser with fifty tabs.
- Run the benchmark. In Personal LLM, open a downloaded model’s details and tap Run benchmark in the On-device benchmark section. It loads the model, generates a fixed answer, and reports generation speed, prompt speed, load time, the context size used, and whether it ran on the GPU or the CPU.
- Run it two or three times and use the typical result, since the first run includes one-off loading costs.
- Compare like with like: same phone temperature, same settings, similar model sizes.
The live tokens-per-second readout under each chat reply is the real-world check. It shows speed changing as a conversation grows and the phone warms up, which a one-shot benchmark never will.
What makes a local AI slow? #
| Symptom | Likely cause | Fix |
|---|---|---|
| Slow from the very first reply | Model too big for the phone | Use a smaller model |
| Long pause before any text appears | Long prompt, big document or an attached photo | Shorter prompt, fewer pages |
| Starts fast, then crawls | Phone heating and throttling | Let it cool, take the case off |
| Gets slower as the chat goes on | Context filling up | Start a new chat |
| Long wait, then a short answer | Thinking mode is on | Turn it off for simple questions |
| Slow on Android specifically | Running on the CPU | Smaller model; check GPU support |
| Slow while the battery is low | Low Power Mode is capping performance | Turn it off for the session |
Eight ways to speed up AI on your phone #
1. Use a smaller model #
The biggest lever by far. A 4B model can be twice as fast as a 9B, and for rewriting, summarizing and everyday questions the quality difference is modest. Qwen 3.5 4B is the recommended starting point in Personal LLM, and Qwen 3.5 0.8B is the fastest option for weak phones.
2. Switch models mid-conversation #
You don’t have to commit to one. Start on a fast model and hand the genuinely hard question to the 9B. Personal LLM lets you switch from the chat header without losing the conversation.
3. Turn off thinking mode #
Thinking mode makes the model write out step-by-step reasoning before its answer. That helps with math and logic and multiplies the tokens generated for everything else. It’s off by default for exactly this reason; use the Thinking preset deliberately.
4. Start a new chat #
Every reply takes the whole conversation into account, so each answer in a long chat costs more work and more memory than the last. When the topic changes, start fresh. The chat header shows how much of the context window is in use.
5. Lower context size and max tokens #
A smaller context window means less memory and faster processing. A lower max-tokens setting stops rambling answers early. Both are in the model settings.
6. Keep the phone cool #
Phones throttle when they get hot, and long AI sessions, direct sun, a thick case or charging at the same time all add heat. If speed drops partway through a session, heat is the likely reason; why your phone gets hot running AI has more.
7. Turn off Low Power Mode or battery saver #
Power-saving modes cap performance. Worth keeping on when the battery is nearly flat, but expect slower answers.
8. Close heavy apps #
Games, the camera and a loaded browser compete for memory. If the system runs short, it may unload the model, which means a slow reload on your next message.
Why is the first reply slow? #
Before the first answer, the app loads the model from storage into memory. The benchmark reports this as load time, and it can be a couple of seconds for a small model and much longer for a 6 GB one. After that, replies start quickly until the model is unloaded, for example when the operating system reclaims memory while the app is in the background.
Images have their own cost: each photo becomes hundreds of tokens or more that the model reads before answering.
Does the GPU make a difference? #
A big one, often several times. GPU acceleration uses Metal on iPhone and OpenCL on Snapdragon phones with Adreno 700-series or newer graphics. Other phones, including Exynos-based Galaxy models and most Pixels, fall back to the CPU in llama.cpp-based apps. The benchmark tells you which one your phone used.
On a phone without GPU support, stay with smaller models. Running AI on an old phone covers which ones are worth trying, and iPhone vs Android for local AI compares the platforms.
Is local AI slower than ChatGPT? #
In raw generation speed, often yes. Cloud models run on racks of data-centre GPUs and stream quickly when the service isn’t busy. But a local model has no network round trip, no queue at peak hours, and runs the same with one bar of signal or none. For short everyday tasks on a recent phone the difference is small; for long, hard answers the cloud is faster and smarter.
Cloud services have their own slow days, which is a separate problem with separate fixes: see why ChatGPT is slow.
Does more RAM make it faster? #
Not directly. RAM decides which models fit; memory bandwidth and the GPU decide speed. Extra RAM does leave headroom, so the model isn’t pushed out of memory and reloaded, which avoids long pauses. How much RAM you need to run an LLM on a phone works through the sizing, and whether you need an NPU explains why the TOPS figure on the box doesn’t predict any of this.
Does a faster model mean worse answers? #
Usually a smaller model means somewhat weaker answers, especially on multi-step reasoning and rare facts. But speed changes how you use it. A fast model you actually reach for beats a slow one you give up on halfway through. Pick the smallest model that gets your task right, keep a bigger one for the questions that need it, and check anything factual either way.
Frequently asked questions #
What is a good tokens per second for a phone LLM? #
Around 6 to 12 tokens per second feels like reading pace, and above 12 feels fast. Small 2B to 4B models on recent phones with GPU acceleration often reach the faster range. Under 3 tokens per second is usable only for short replies.
Why is my first message so slow? #
The app has to load the model from storage into memory before it can answer, which takes seconds for a multi-gigabyte file. Long prompts and attached documents or photos also have to be read before the first word appears.
Why does my local AI get slower over time? #
Long conversations add context that gets processed with every new token, and phones throttle as they warm up. Starting a new chat and letting the phone cool usually restores the original speed.
Is on-device AI faster than cloud AI? #
It starts without network delay and never waits in a server queue, so short answers often feel just as quick. For long answers from large models, cloud hardware generates text faster.