The biggest improvements to a local AI’s answers come from three things: using the largest model your phone runs comfortably, giving it a clear system prompt, and matching the temperature to the job (around 0.3 for facts, 0.7 for everyday chat, 1.0 or more for creative writing). Turn on thinking mode for math, logic and code. Everything else is fine-tuning.
Small models are more sensitive to settings and phrasing than big cloud models, so a few minutes here pays off more than it would with ChatGPT.
What does each setting do? #
| Setting | What it controls | Typical values |
|---|---|---|
| Temperature | How adventurous word choices are | 0.3 precise, 0.7 balanced, 1.2 creative |
| Top-p | Only words within the top share of probability are considered | 0.9 to 0.95 |
| Top-k | Only the k most likely next words are considered | 40 |
| Max tokens | The longest reply the model can write | 512 to 2,048 |
| Context size | How much conversation or document the model can hold in view | As large as your phone handles smoothly |
| Thinking mode | Whether the model reasons step by step before answering | On for math and logic, off for chat |
| System prompt | Standing instructions for the whole chat | A sentence or two about role and style |
Temperature #
A language model picks each word from a list of likely candidates. Temperature controls how far down that list it’s willing to go. Low values make it pick the most likely word almost every time, which gives consistent, conservative answers. High values let it take more chances, which gives variety and occasionally nonsense.
- 0.2 to 0.4: facts, summaries, extracting details from documents, code
- 0.6 to 0.8: general chat, explanations, emails
- 1.0 to 1.3: brainstorming, stories, poems, alternative phrasings
If a small model starts inventing details, lower the temperature first.
Top-p and top-k #
Both trim the list of candidate words before temperature applies. Top-k keeps a fixed number of the most likely words; top-p keeps however many words make up a set share of the probability. You rarely need to touch them. If replies wander or go off-topic at higher temperatures, lowering top-p to around 0.85 reins them in.
Max tokens and context size #
Max tokens caps the length of each reply. A token is roughly three-quarters of an English word, so 1,024 tokens is around 750 words. Context size is the model’s working memory. A bigger context lets it remember more of a long conversation or document, but uses more RAM and slows each reply. When a chat gets long and the model starts forgetting early details, start a new chat with a short summary of what matters.
Thinking mode #
With thinking mode on, reasoning models such as Qwen 3.5 work through the problem step by step before giving the final answer. It’s noticeably better for math, logic puzzles, planning and code, and noticeably slower. Leave it off for everyday chat.
Prompt habits that help small models #
- Be specific about the output. “Give me five bullet points, under 15 words each” beats “summarize this.”
- Give the model the material. Paste the text, attach the file or share the photo. Small models are far more reliable working from what’s in front of them than from memory.
- Show an example. One example of the format you want (a sample line, a sample email) steers small models better than a paragraph of instructions. We cover this in zero-shot vs few-shot prompting.
- Split big tasks. Ask for an outline, then each section, then the edit, rather than everything at once.
- Ask it to say when it doesn’t know. “If you’re not sure, say so” measurably reduces invented answers in practice.
- Regenerate. If the first answer misses, regenerating or rewording your question is often faster than arguing with it.
What makes a good system prompt? #
A system prompt is a standing instruction that applies to every message in a chat. Keep it short and concrete:
- “You are a concise assistant. Answer in plain English. Use bullet points for lists.”
- “You’re an editor. Fix grammar and clarity. Keep my voice. Show only the edited text.”
- “You’re a patient tutor. Explain step by step and check my understanding with a question.”
Different chats can have different system prompts, so your recipe helper doesn’t have to sound like your code reviewer. For more, see 12 system prompt examples for your AI chat app.
Pick the right model for the job #
Settings can’t make a small model into a big one. Rough guide:
- Quick chat, rewrites, simple questions: a 3B to 4B model
- Careful reasoning, longer writing, tricky facts: the largest model your phone runs, such as a 9B on an 8 GB phone
- Photos and documents: a vision-capable model with its image support added
Which model fits your phone covers the RAM side, and what on-device AI can and can’t do covers the limits no setting will fix.
Presets: a shortcut #
If you don’t want to think about numbers, presets bundle sensible values. Personal LLM has four one-tap presets: Creative (temperature 1.2), Balanced (0.7), Precise (0.3) and Thinking (reasoning on). You can also set temperature, top-p, top-k, context size, max tokens and thinking mode yourself, write a system prompt per chat, or set a default system prompt for every new chat. Thinking mode shows the model’s reasoning in a collapsible panel above the answer, so you can check the working. The chat header shows how much of the context you’ve used, and you can switch to a bigger model from the header mid-conversation when a question needs it. Replies can be edited, regenerated or stopped, and a stopped reply keeps what it had.
Frequently asked questions #
What temperature should I use for an LLM? #
Around 0.3 for factual answers, summaries and code; around 0.7 for everyday chat; and 1.0 or higher for creative writing and brainstorming. If the model starts making things up, lower it.
What is the difference between top-p and top-k? #
Top-k limits the model to a fixed number of the most likely next words. Top-p limits it to the smallest group of words whose combined probability reaches a threshold, such as 90%. Most people can leave both at their defaults.
Does thinking mode make AI smarter? #
It makes reasoning models more accurate on multi-step problems such as math, logic and code, because they work through the steps before answering. It’s slower, and it doesn’t add knowledge the model doesn’t have.
Why does my local AI forget earlier parts of the conversation? #
The conversation has outgrown the model’s context size. Increase the context if your phone has room, or start a new chat and paste in a short summary of what matters.