GGUF is the single-file format that llama.cpp, and the phone and desktop apps built on it, use for language models. One .gguf file holds the model’s weights, its tokenizer, its chat template and its settings, so you can download it once and run it offline with nothing else installed. If you’ve ever pasted a link into a phone AI app and watched a model appear in the catalog, that file was a GGUF.
The format is why local AI works the way it does: no dependencies, no conversion step, no framework to install. Here’s what’s actually in one.
What’s inside a GGUF file? #
Three parts:
- A header identifying the format and its version.
- Metadata: the model’s architecture, its context length, the tokenizer vocabulary, the chat template, and details such as who made it and under which license.
- Tensors: the model’s weights, stored at whatever precision the file was built with.
That metadata block is the interesting part. The chat template tells the app how to wrap your messages in the special tokens the model was trained on, which is why a GGUF usually behaves correctly in an app that has never heard of that specific model. When a model produces strange tags or won’t stop talking, a missing or mismatched template is a common cause.
GGUF replaced llama.cpp’s older GGML file format in 2023 and has become the standard way to share models for local use. New open models typically get GGUF builds within days of release, which is why local apps can keep up with a fast-moving field.
Why does one file matter? #
Because it makes “add a model by URL” possible. There’s no folder of shards, no separate tokenizer download, no config file to keep in sync. The app fetches one file, reads the metadata, and loads the weights.
It also makes models portable. The same GGUF runs in a phone app, a desktop tool and a command-line program, because they all use the same engine underneath. See run a custom GGUF model on your phone for the practical steps.
Is a GGUF file safe to open? #
A GGUF holds data, not a program. That makes it much safer than the old pickle-based PyTorch checkpoints, which could execute code when loaded. The residual risks are a malformed file targeting a bug in the software that reads it, and an uploader who published something other than what the name claims. Both are handled the same way: download from the model’s own publisher or a well-known converter. Are AI model files safe to download covers this properly.
What do the quantization codes in the filename mean? #
A name like Qwen3.5-4B-Q4_K_M.gguf tells you the model, its size and how heavily its weights were compressed. Roughly:
- Q plus a number is the approximate bits per weight. Q4 is about 4 bits, Q8 about 8.
- K marks the k-quant scheme, and S, M, L say how many sensitive layers keep extra precision.
- IQ marks the newer i-quants, which hold up better at very low bit counts.
- F16 or BF16 means no quantization at all, which is too big for a phone.
Q4_K_M is the default for phones: roughly a third of the original size, with a modest quality cost. For the full table with measured bits per weight and what to pick for your RAM, see GGUF quantization explained.
What is an mmproj file? #
For models that understand images, the vision half often ships as a separate GGUF with mmproj in the name. It’s the projector that turns a photo into tokens the language model can read. It has to match the model family and size it was built for, and without it the model handles text only.
In phone apps this usually appears as “image support,” a separate download of roughly 195 MB to 1 GB per model.
How does GGUF compare with other model formats? #
| Format | Used by | Typical use |
|---|---|---|
| GGUF | llama.cpp, LM Studio, Ollama, phone apps | Running quantized LLMs locally |
| Safetensors | Hugging Face Transformers, most training tools | Sharing original weights, fine-tuning |
| .tflite (LiteRT) | Google’s on-device runtime | Mostly smaller vision and audio models in apps |
| Core ML (.mlpackage) | Apple’s frameworks | Models built into iPhone and Mac apps |
| ONNX | Cross-platform runtimes | Portable models, often for vision |
Google renamed TensorFlow Lite to LiteRT in September 2024 and broadened it to run models from PyTorch and JAX as well. It powers a lot of built-in phone features. For chatting with open language models, though, GGUF with llama.cpp is what most apps use.
Safetensors is worth understanding as the other half of the pair: model makers publish safetensors, and the community converts them to GGUF. If a model exists only as safetensors, it can’t be loaded by a phone app until someone converts it.
Where do you get GGUF files? #
Hugging Face hosts almost all of them. Look first in the model maker’s own organization, since some publish official GGUF builds, then at established community converters. Search for the model name plus “GGUF”, open the Files and versions tab, and pick a file whose size leaves your phone about 2 GB of headroom; how much RAM you need to run an LLM on a phone explains that margin.
Personal LLM runs GGUF models through llama.cpp on the phone itself. Its catalog makes the choice for you with ready-made builds of Qwen 3.5, Gemma 4, GLM 4.6V Flash and Ministral 3, and you can add any other GGUF by pasting its link, with an optional mmproj for images.
Frequently asked questions #
Is GGUF the same as GGML? #
GGUF is the successor to GGML’s old file format, introduced in 2023 so that a single file could carry the tokenizer, chat template and other metadata, and stay compatible as the format evolved. GGML is still the name of the underlying tensor library.
Can I run a GGUF file on my phone? #
Yes, in an app built on llama.cpp that accepts custom models. Pick a model and quantization whose file is comfortably smaller than your phone’s RAM, usually a 4-bit build of a 1B to 9B model.
Can I open a GGUF file to see what’s in it? #
Not usefully in a text editor; it’s binary. Hugging Face shows the metadata of GGUF files in its web interface, including the architecture, context length and quantization type, which is the quickest way to inspect one before downloading.
Why won’t my GGUF load in the app? #
Usually because the file is too big for the phone’s free RAM, it’s one part of a split multi-file model, or its architecture is newer than the app’s engine. A single-file 4-bit build of an established model family is the safest choice.