A local LLM is a large language model that runs on your own device, your phone or laptop, instead of on a company’s servers. You download the model once as a file, and from then on your device does all the work: it reads your message and writes the reply without sending anything over the internet. That’s why local LLMs work offline and why nobody else sees your conversations.
ChatGPT, Gemini and Claude are the opposite: cloud LLMs. The model lives in a data center and your device only sends text and displays the answer.
How does a local LLM work? #
A language model is, at heart, a very large table of numbers called weights (or parameters), learned from reading enormous amounts of text. When you type a question, the model uses those weights to predict the most likely next word, then the next, and so on. That’s the reply you see streaming in.
To run locally, three pieces sit on your device:
- The model file. The weights, stored as a file. Phone-sized models are roughly 1 to 6 GB.
- An inference engine. Software that loads the file into memory and does the math. The most widely used open-source engine for phones and laptops is llama.cpp, which can use a phone’s GPU (Metal on iPhone, for example) to speed things up.
- An app. The chat screen, history, settings and file handling.
When you press send, your phone’s chip performs billions of calculations per word. That’s why local AI makes phones warm during long answers.
What do “4B,” “Q4” and “GGUF” mean? #
Local LLMs come with a lot of labels. The ones you’ll see most:
| Term | What it means | Why you care |
|---|---|---|
| Parameters (0.8B, 4B, 9B) | Number of weights, in billions | Bigger is usually smarter, but slower and needs more RAM |
| Quantization (Q4, Q8) | How many bits store each weight | Fewer bits means a smaller file with a small loss in quality |
| GGUF | A file format for quantized models used by llama.cpp | The format most phone apps load |
| Context window | How much text the model can consider at once | Limits how long a conversation or document it can handle |
| Tokens per second | Generation speed; a token is roughly three-quarters of an English word | Tells you how fast replies will stream |
| Open weights | The model file is published for anyone to download | What makes local LLMs possible |
A rough rule for quantized models: a 4B model at about 4 bits per weight is a file of around 2.5 GB, and it needs a bit more than that in free memory to run. We go through the maths for each size in how much RAM you need to run an LLM on a phone.
Which LLMs can you run locally? #
Only open-weight models, whose makers publish the files. The current phone-friendly families include:
- Qwen 3.5 from Alibaba’s Qwen team (0.8B, 4B and 9B), released in 2026 under the Apache 2.0 license and covering 201 languages and dialects.
- Gemma 4 from Google DeepMind, whose E2B and E4B versions are built for phones.
- Ministral 3 from Mistral AI, a compact 3B model with a 256k context window.
- GLM 4.6V Flash from Zhipu AI, a 9B vision model for high-end phones.
You can’t run ChatGPT, Gemini or Claude locally: their weights aren’t published. For a closer comparison of the open options, see the best open-source LLMs for phones.
What do you need to run one? #
- RAM. The deciding spec. Phones with 3 to 4 GB can run the smallest models (under 1B to about 3B). Phones with 6 GB handle 4B models comfortably. Phones with 8 GB or more, such as the iPhone 15 Pro and later and recent Android flagships, can run 9B models.
- Storage. Each model takes 1 to 6 GB, plus room for any vision add-on.
- An app. On phones, dedicated apps handle downloading, loading and chatting. On computers, tools such as Ollama and LM Studio do the same job.
- Wi-Fi, once. Model files are big, so download them on Wi-Fi. After that, no connection is needed.
Why would you use a local LLM? #
- Privacy. Your conversations never reach a company’s server, so they can’t be stored, reviewed, used for training or targeted with ads. See AI chatbot privacy myths for what the cloud services do with chats.
- Offline use. Planes, trains, remote trips, basements, anywhere without a signal.
- No subscription. Open models are free; there are no message caps.
- No account. Nothing links your questions to your identity.
- Control. You pick the model, and it doesn’t change unless you change it.
What are the downsides? #
- Less capable than flagship cloud models, especially on rare facts and hard reasoning.
- No web access. It knows only what it learned in training, so it can’t tell you today’s news.
- Battery and heat during long sessions.
- Storage used by model files.
For a full list of what works and what doesn’t, see what on-device AI can and can’t do.
How do you start using a local LLM on your phone? #
- Install a local AI app from the App Store or Google Play.
- Pick a model that fits your phone’s RAM. A 4B model is the usual sweet spot.
- Download it on Wi-Fi.
- Turn on airplane mode and ask it something, to confirm it’s running locally.
Personal LLM is built to make that first run easy. It reads your phone’s RAM and badges each model in its catalog as “Fits your device,” “Should run” or too big, before you download anything. Qwen 3.5 4B is the recommended first download: about 2.7 GB, and it runs on phones with 3 GB of RAM or more. The app uses llama.cpp with GPU acceleration (Metal on iPhone, OpenCL on Snapdragon Adreno 700 series and newer, CPU elsewhere), shows a live tokens-per-second readout on every reply, and has no account or server. Other free options include PocketPal AI, an open-source app for iOS and Android, and Google’s AI Edge Gallery, which runs Gemma models on Android and iOS.
Frequently asked questions #
Is a local LLM really private? #
The model itself runs on your device, so your conversation doesn’t need to go anywhere. Whether the app sends anything else, such as analytics or ads, depends on the app. Check its privacy label and test it in airplane mode.
Can I run ChatGPT locally? #
No. OpenAI doesn’t publish the weights for the models behind ChatGPT, so they only run on OpenAI’s servers. Open-weight models such as Qwen 3.5 and Gemma 4 are the local alternatives.
Is a local LLM free? #
The models are usually free under open licenses such as Apache 2.0 or MIT. Many phone apps that run them are free too, sometimes supported by ads. Your costs are storage space and battery.
How big is a local LLM? #
Phone-sized models range from under 1 GB for a sub-1B model to about 6 GB for a 9B model at typical compression. Desktop models can be tens of gigabytes.