The code in a GGUF filename, such as Q4_K_M or Q8_0, says how much the model’s weights were compressed. Q8_0 uses about 8.5 bits per weight and is very close to the original; Q4_K_M uses about 4.9 bits, cutting the size by roughly 70% compared with 16-bit, and is the usual default. On a phone, start with Q4_K_M and only go lower if the model otherwise won’t fit.
One model on Hugging Face often has a dozen of these files side by side. Picking between them is the whole decision, and it changes the download size, the RAM you need and how fast replies appear. (For the format itself, see what is a GGUF file.)
What does quantization do? #
A model’s knowledge lives in billions of numbers called weights. During training they’re stored at 16 or 32 bits each. Quantization stores them with fewer bits: weights are grouped into small blocks that share a scale factor, and each weight is rounded to one of a small set of values within its block.
It’s a bit like saving a photo as a smaller JPEG. At moderate settings you can barely tell. Push it too far and artifacts appear, though for language models the artifacts are subtler: slightly worse word choices, more factual slips, weaker reasoning.
Three things improve when you quantize:
- File size and download time drop in proportion to the bits.
- RAM needed drops the same way, which decides whether a model fits on your phone at all.
- Generation speed rises, because producing each word means reading all the weights from memory, and fewer bits read faster.
What do Q4_K_M, Q5_K_M and Q8_0 mean? #
The naming looks cryptic but follows a pattern:
- Q plus a number is the rough bits per weight: Q4 is about 4 bits, Q8 about 8.
- _0 and _1 are the older, simpler schemes. Q8_0 survives because 8 bits needs no clever tricks.
- K marks “k-quants”, a smarter scheme that groups weights better and keeps some sensitive parts of the model at higher precision.
- _S, _M, _L (small, medium, large) say how many of those sensitive parts get extra bits. M is the common middle ground.
- IQ marks “i-quants”, newer formats that hold up better at very low bit counts, often built with an importance matrix that measures which weights matter most. They can run slower on some hardware.
- F16 and BF16 are unquantized 16-bit weights.
Here are figures from llama.cpp’s own quantization documentation, measured on an 8-billion-parameter model:
| Type | Bits per weight | Size for an 8B model | In practice |
|---|---|---|---|
| IQ2_M | 2.93 | 2.74 GiB | Last resort; noticeable quality loss |
| Q2_K | 3.16 | 2.95 GiB | Last resort |
| IQ3_M | 3.76 | 3.52 GiB | Usable when space is tight |
| Q3_K_M | 3.99 | 3.74 GiB | Usable when space is tight |
| IQ4_XS | 4.46 | 4.17 GiB | Good, slightly smaller than Q4_K_M |
| Q4_K_M | 4.89 | 4.58 GiB | The usual default |
| Q5_K_M | 5.70 | 5.33 GiB | A little better, a little bigger |
| Q6_K | 6.56 | 6.14 GiB | Very close to the original |
| Q8_0 | 8.50 | 7.95 GiB | Near-lossless; rarely worth it on a phone |
| F16 | 16.0 | 14.96 GiB | Unquantized; too big for most phones |
How do you estimate a GGUF file’s size? #
Multiply the parameter count in billions by the bits per weight and divide by 8. The answer is roughly the size in gigabytes.
- A 4B model at Q4_K_M: 4 × 4.89 ÷ 8 ≈ 2.4 GB
- A 9B model at Q4_K_M: 9 × 4.89 ÷ 8 ≈ 5.5 GB
- A 4B model at Q8_0: 4 × 8.5 ÷ 8 ≈ 4.3 GB
Real files run a little larger because of embeddings and metadata. For comparison, the Qwen 3.5 9B build in Personal LLM’s catalog is 5.68 GB, in line with that math for a 4-bit-class file.
To run comfortably you need the file size plus room for the conversation’s working memory and the rest of the phone. Personal LLM marks a model “Fits your device” when your RAM is at least 2 GB above its minimum, which is a sensible target for your own models too. How much RAM you need to run an LLM on a phone works through the arithmetic.
Which quantization should you pick for a phone? #
| Your situation | Pick | Why |
|---|---|---|
| Most models on most phones | Q4_K_M | Best balance of size, speed and quality |
| RAM to spare and you want quality | Q5_K_M or Q6_K | Small quality gain for 15 to 35% more size |
| A tiny model (1B or smaller) | Q8_0 | Small models lose more from compression, and the file is small anyway |
| The model barely doesn’t fit | IQ4_XS or Q3_K_M, or a smaller model | Try the next size down before going below 3 bits |
| Maximum speed | The smallest quant that still answers well | Generation speed tracks file size |
A rule of thumb from the local AI community: a larger model at 4 bits usually beats a smaller model at 8 bits of similar file size, but below about 3 bits quality falls off quickly. So a 9B at Q4_K_M generally makes a better assistant than a 4B at Q8_0, if your phone can hold it. Test on your own questions, since results vary by model and by task.
Does quantization hurt some tasks more than others? #
Yes. Chat, rewriting and summarizing hold up well at 4 bits. Arithmetic, code and long multi-step reasoning are the first things to degrade, and small models degrade faster than large ones at the same bit depth. If a model has started rambling, repeating or slipping into the wrong language, an over-aggressive quant is one of the usual suspects; why local AI repeats itself lists the others.
One exception is worth knowing: quantization-aware training, where a model is taught during training to cope with low precision. Google’s Gemma 4 builds for phones are trained this way, which is why they hold quality at sizes that would hurt an ordinary model.
Why is the vision projector a separate file? #
Models that understand images come in two parts. The language model is the main GGUF. The vision part, the projector or mmproj file, is an image encoder that turns a photo into tokens the language model can read. It’s distributed as its own GGUF and usually kept at higher precision, since it’s relatively small and sensitive to compression.
That’s why photo support is a separate download in phone apps: adding image support to a catalog model in Personal LLM is a 195 MB to 1 GB download, and when you add your own model by URL you can supply a matching projector URL too. The projector must be built for that exact model.
Where do you find GGUF files? #
On Hugging Face, search for the model name plus “GGUF”. Model makers sometimes publish official builds, and well-known community converters handle popular models within days of release. Check the model card for the license, the recommended settings and which quantizations are offered. Our step-by-step guide to running a custom GGUF model on your phone covers downloading one and loading it by URL.
Frequently asked questions #
What does Q4_K_M mean? #
It’s a llama.cpp quantization type: about 4 bits per weight (“Q4”) using the k-quant scheme (“K”) at the medium setting (“M”), which keeps some sensitive parts of the model at higher precision. It averages about 4.9 bits per weight and is the most popular default.
Is Q4 or Q8 better? #
Q8 is closer to the original model, but it’s nearly twice the size and correspondingly slower to generate. On a phone, Q4_K_M usually gives the better overall experience, and the RAM you save can go toward a larger, smarter model.
What is an IQ or imatrix quant? #
IQ types are newer llama.cpp formats that preserve quality better at very low bit counts, often using an importance matrix computed from sample text to decide which weights need more precision. They’re most useful at 2 to 4 bits, and can be slower than k-quants on some hardware.
How big is a 7B model at 4-bit? #
About 4 to 4.5 GB. At Q4_K_M, 7 billion × 4.89 bits ÷ 8 comes to roughly 4.3 GB, plus a little for embeddings and metadata. That wants a phone with around 6 GB of RAM or more to run well.
Does quantization make a model dumber? #
A little, and unevenly. At 5 to 8 bits the difference is hard to notice; at 4 bits it’s small for most everyday tasks; below 3 bits small models degrade clearly. Choosing a bigger model at 4 bits usually beats a smaller one at higher precision.