Run a Custom GGUF Model on iPhone or Android

You don’t need a computer for this. Find the model on Hugging Face, pick the GGUF file that fits comfortably in your phone’s RAM (usually the Q4_K_M one), copy that file’s direct download link, and paste it into a phone app that accepts models by URL. The file downloads once, and from then on the model runs offline like any built-in one, on Android or iPhone.

Two details trip most people up. The link has to be the /resolve/ download URL rather than the /blob/ page you see in your browser, and the file has to leave room for the rest of your phone, not just fit in it.

What you need before you start #

• A phone that can run local models. Personal LLM, for example, needs iOS 15.1 or later, or Android 7 or later.
• Enough RAM for the model you want. Sizing is the next section.
• Free storage for the file, plus margin. Model files are typically 1 to 6 GB.
• Wi-Fi, unless you enjoy watching a 5 GB file crawl over cellular.
• A model llama.cpp supports. Phone apps run GGUF files through llama.cpp, so the model’s architecture has to be one llama.cpp knows. Popular open families are supported within days of release; a brand-new architecture may not load until the app’s engine is updated.

Step 1: Pick a model and quantization that fit your RAM #

The whole model sits in memory while it answers, alongside the operating system and everything else you have open. A good target is a file at least 2 GB smaller than your phone’s total RAM, which is the same headroom rule the app uses before it calls a model a good fit.

The quantization code in the filename sets that size. Lower means smaller and faster, with some loss of quality:

“K” means the k-quant scheme, and S, M and L say how many sensitive layers keep extra precision. M is the usual balance. When in doubt, take Q4_K_M. GGUF quantization explained has the full table with measured bits per weight, and how much RAM you need to run an LLM on a phone covers the memory side.

One rule worth internalizing: a bigger model at 4 bits usually beats a smaller model at 8 bits of the same file size. Spend your memory on parameters before precision.

If the model you want is already in your app’s catalog, use the catalog build instead. It’s been sized for phones and its image support installs in one tap. Custom URLs are for everything else: a model tuned for one language, a coding model, a fine-tune of your own, or something released last week.

Step 2: Find the GGUF on Hugging Face #

1. Search Hugging Face for the model name plus “GGUF”.
2. Prefer, in order: the model maker’s own repository, an established converter such as the llama.cpp team’s ggml-org or unsloth, then other uploaders with a clear model card, many downloads and recent activity.
3. Open the model card and read the license, the recommended settings, and which quantizations exist.
4. Skip repositories with no model card, a handful of downloads, or a name that imitates a popular model.

A GGUF file holds weights rather than a program, which makes it far lower-risk than the old pickle-based formats, but a malformed file can still poke at bugs in the software that loads it. Are AI model files safe to download goes through what can and can’t go wrong.

Gated repositories ask you to log in and accept a license before downloading. A plain URL won’t work for those, so look for an ungated re-upload of the same model.

Step 3: Copy the direct download link #

Open the Files and versions tab and click the file you want. You need the link that downloads the file itself, not the page describing it:

• Download link: https://huggingface.co/<owner>/<repo>/resolve/main/<file>.gguf
• Page link: https://huggingface.co/<owner>/<repo>/blob/main/<file>.gguf

If you copied a page link, change /blob/ to /resolve/. You can also long-press the download button and copy its link.

Choose a single-file GGUF. Large models are sometimes split into parts named like -00001-of-00003.gguf. Those need extra handling that most phone apps don’t do, so pick a quantization that ships as one file.

Step 4: Add the model by URL #

1. Open the Models tab and choose the option to add a model by URL.
2. Paste the .gguf link.
3. Give it a name that includes the size and quantization, so you can tell versions apart later.
4. Start the download on Wi-Fi. A good app checks your free space first, and lets you pause and resume if the connection drops.

In Personal LLM this is Add model by URL on the Models tab, with the link going in the Model URL (.gguf) field. The model then joins your catalog next to the built-in ones and can be picked for any chat.

Once it’s downloaded, test it in airplane mode. If it answers, nothing about it needs the network again.

Step 5: Add image support with an mmproj file #

This step is optional and only applies to models that can read images. Those ship in two parts: the language model itself, and a vision projector, usually a separate GGUF with mmproj in its name in the same repository. The projector turns a photo into tokens the language model can read.

Paste the projector’s /resolve/ link into the optional projector field when you add the model. It has to be the projector built for that exact model and size; one borrowed from another model won’t work. Without it, the model handles text only.

Step 6: Tune the settings #

Custom models arrive with generic defaults, and a few adjustments usually help:

• Temperature: around 0.3 for factual work, 0.7 for general chat, higher for fiction. Presets named Precise, Balanced and Creative set this for you.
• Context size: smaller uses less memory and loads faster. Start modest and raise it only if you need long chats.
• Max tokens: caps how long each reply can run.
• Thinking mode: switch it on for reasoning models when you want the step-by-step working, off for everyday questions.
• System prompt: many models follow instructions far better with a clear one, and some model cards recommend a specific wording.

Start from whatever the model card suggests, then adjust once you’ve seen a few answers.

Step 7: Benchmark it #

Before you make a custom model your daily driver, measure it. A built-in benchmark reports generation speed in tokens per second, prompt-reading speed, load time, and whether the model ran on the GPU or the CPU. Compare it against a catalog model of similar size that you already know.

If it’s much slower, the file is probably too big for the phone, or the phone is falling back to the CPU. GPU acceleration uses Metal on iPhone and OpenCL on Snapdragon phones with Adreno 700-series or newer graphics; other phones, including Exynos Galaxy models, run on the CPU. Local LLM speed on your phone explains what good numbers look like.

Troubleshooting custom GGUF models #

Base versus instruct. A base model only continues text. For chat you want the build labeled instruct, it, or chat. This is the single most common reason a technically working model feels broken.

More detail lives in why a local AI app crashes or won’t load a model, AI model download not working and why local AI repeats itself.

How do you update a custom model? #

Local model files never update themselves, which is mostly a feature: a model you’ve tuned your prompts around won’t change behavior overnight. When a new version appears:

1. Read the release notes on Hugging Face and decide whether the change matters for what you use it for.
2. Add the new version as a separate model rather than replacing the old one.
3. Ask both the same five questions you actually care about.
4. Delete the loser to free the storage.

Your chats stay where they are. If the app lets you switch models mid-conversation, you can even continue an old chat with the new model and compare directly.

Do model licenses matter? #

They do, particularly for work. Licenses vary between families: some open models ship under Apache 2.0 or MIT, others carry their own terms with restrictions on commercial use or on what you may do with the outputs. The license is on the model card, and it’s worth two minutes before you build anything on top of a model.

Frequently asked questions #

Can I run any Hugging Face model on my phone? #

Only models published as GGUF, whose architecture your app’s llama.cpp version supports, and whose file is small enough for your RAM. Models released only as safetensors need converting first, though for popular models someone in the community has usually already done it.

What is an mmproj file? #

It’s the vision projector: a separate GGUF that turns an image into tokens the language model can read. Download it from the same repository as the model, add it alongside the main file, and the model can answer questions about photos. Without it, the model is text-only.

Why won’t my custom GGUF model load? #

The usual causes are a file too large for your phone’s free RAM, a split multi-part download, a gated repository, or an architecture the app’s engine doesn’t support yet. Try a smaller quantization, a single-file build, or a more established model.

Do I need a Hugging Face account to download models? #

Not for public, ungated repositories, which covers most GGUF uploads. Gated models require logging in and accepting a license, so a direct link alone won’t fetch them.

Can I load a Llama or Mistral GGUF this way? #

Yes. Anything with a phone-sized GGUF build can be added by URL, whether or not it’s in the app’s own catalog. Check the model’s license terms for your use, and pick a quantization that leaves your phone about 2 GB of headroom.