Zero-Shot vs Few-Shot Prompting for Small On-Device LLMs

Zero-shot prompting means asking a model to do a task with instructions only. Few-shot prompting adds two or three worked examples of what you want. Small models that run on your phone, in the 1B to 9B range, handle simple tasks zero-shot, but they benefit much more than big cloud models from examples, a fixed output format and a clear system prompt.

That difference is most of the skill of getting good answers from local AI.

What is zero-shot learning? #

The term comes from machine learning, where “zero-shot” meant recognizing a category the model never saw labeled examples of, using only a description. A model that had never seen a zebra could identify one from “a horse with black and white stripes.”

With language models it means something simpler: you describe the task and the model does it, with no examples in the prompt. “Summarize this in three bullet points” is zero-shot. The model relies entirely on what it learned in training.

Zero-shot vs few-shot vs step-by-step #

On a big cloud model, zero-shot works for almost everything. On a 4B phone model, few-shot often turns a mediocre answer into a reliable one.

Why do small models need more help? #

• Less knowledge packed in. A 4B model has seen the same internet as a big one but has far fewer parameters to store it. It guesses more.
• More sensitive to phrasing. Small changes in wording shift the answer more.
• Format drift. Ask for JSON or a table and a small model may add chatty preambles or change the columns. An example fixes that better than an instruction.
• Fact recall is weak. Artificial Analysis measured Qwen 3.5 4B answering only about 13% of hard knowledge questions correctly. Give the model the facts instead of asking it to remember them.

Prompt templates that work on a phone #

Sorting and labeling (few-shot) #

Categorize each expense as Food, Transport, Bills or Other.

Coffee at Blue Bottle $6.50 -> Food
Uber to airport $38 -> Transport
Electric bill $92 -> Bills

Parking meter $4 ->
Groceries at Trader Joe's $61 ->

Two or three examples in the exact format you want is the sweet spot. More examples use up context without helping much.

Pulling details out of text (few-shot plus format) #

Extract the name, date and amount from each message.
Reply only with a table: Name | Date | Amount.

Example:
"Hi, it's Dana, I sent $40 on March 3 for the tickets."
Dana | March 3 | $40

Messages:
[paste your messages here]

Rewriting tone (zero-shot is fine) #

Rewrite this to be polite but firm, under 80 words:
[paste your draft]

Rewriting is something every model has seen thousands of times, so examples rarely help.

Math and logic (step-by-step) #

Turn on a reasoning mode, or ask the model to work through it before answering. Reasoning models write out their working first, which improves accuracy on multi-step problems but makes replies longer and slower.

Put repeated instructions in the system prompt #

If you give the same instructions every time, move them into the system prompt, the standing instructions the model reads before every message. That keeps your actual prompts short and leaves more room in the context window.

In Personal LLM, each chat can have its own system prompt, so a “Budget sorter” chat can hold the categories and examples above while your recipe chat holds something else entirely. You can also set a default system prompt for every new chat under Settings, Behavior. The one-tap presets pair well with prompt styles: Precise (temperature 0.3) for extraction and sorting, Balanced (0.7) for everyday writing, Creative (1.2) for brainstorming, and Thinking to switch reasoning on for supported models. We cover system prompts in more depth in how to personalize an AI chatbot without sharing your data.

Common mistakes with small models #

• Too many examples. Ten examples fill the context window and slow the first response. Two or three is usually enough.
• Asking for facts instead of giving them. “When does this store close?” invites a made-up answer. Paste the store’s hours or attach the document, then ask. We explain why models don’t know recent facts in do offline AI models get updated.
• Vague output instructions. “Make a list” gives you any kind of list. “Five bullet points, under 10 words each” gives you what you wanted.
• Using a model that’s too small. If a 0.8B model keeps missing the point, a 4B model usually gets it zero-shot. Which AI models can run on a phone helps you pick.
• High temperature for precise tasks. Creativity settings make extraction sloppy. Use a low temperature when you want the same answer every time.

Frequently asked questions #

What is zero-shot prompting? #

It’s giving a model a task with instructions but no examples, like “Translate this into Spanish.” The model relies on what it learned during training. It works well for common tasks and less reliably for unusual formats.

When should I use few-shot prompting? #

Use it when the output needs a specific format, when you’re sorting or labeling things into your own categories, or when a zero-shot answer keeps coming out wrong. Two or three examples that match the exact format you want work best.

Does few-shot prompting slow down a local model? #

Slightly. The phone has to read the extra example text before it starts writing, which adds a moment to the first reply. With two or three short examples the delay is small, and the better answer is usually worth it.

What temperature should I use for a local LLM? #

Around 0.2 to 0.4 for extraction, sorting and factual summaries, around 0.7 for everyday writing, and 1.0 or higher for brainstorming and creative writing. Lower temperature makes answers more consistent; higher makes them more varied.