Why Does AI Forget What I Said? Context Windows Explained

An AI chatbot forgets what you said earlier because it doesn’t remember anything between messages. Every time you hit send, the app hands the model your instructions plus as much of the conversation as fits in a fixed budget of text, called the context window. When a chat outgrows that budget, the oldest messages are dropped or squeezed into a summary, and whatever was in them is gone.

That’s why a long chat suddenly loses your name, the rules of a game you set up, or the first half of a document. Phone models work with smaller windows than cloud chatbots, so you’ll meet the limit sooner on a local AI. The fixes are straightforward once you know what’s happening.

What is a context window? #

A context window is the maximum amount of text a language model can consider at once, counted in tokens. A token is a chunk of text, often a short word or part of a longer one. In English, 1,000 tokens is roughly 750 words.

Everything in a turn shares that one window:

That last row surprises people. If the window is 2,048 tokens and the app reserves 512 for the answer, the instructions and conversation share about 1,500 tokens, or roughly 1,100 words.

If a message isn’t in the window, then as far as the model is concerned it never happened.

What does forgetting look like? #

• It asks for something you already told it, like your name or where you’re traveling.
• It stops following an instruction from the start of the chat, such as “answer in bullet points.”
• A character in a story changes personality halfway through.
• It contradicts a decision you made together ten messages ago.

All of these mean the relevant message has fallen outside the window.

Why does a chatbot forget earlier messages? #

When the window fills, the app has to leave something out. Three strategies are common, and you’ll meet all of them:

1. Drop the oldest messages. The simplest approach, and what most local apps do. The front of the chat falls off.
2. Summarize the old part. Early messages are replaced by a short summary, and details the summary judged unimportant disappear.
3. Keep everything but pay less attention to the middle. Even when text technically fits, models use information near the start and end of a long prompt better than the middle. Researchers documented this in the 2023 paper Lost in the Middle.

So forgetting can start well before the window is completely full, especially with long pasted documents.

Is this the same as ChatGPT’s memory? #

No. A memory feature saves selected facts about you (your job, your kids’ names, how you like answers formatted) to a separate store and adds them to future chats. That’s memory across conversations, and it works by keeping notes about you on the company’s servers.

The context window is memory within one conversation. A chatbot can have a saved memory of your name and still lose the plot of the story you started 40 messages ago. Our guide to personalizing a chatbot without sharing your data covers the cross-chat kind, and how to get the same effect with a system prompt that stays on your phone.

Why are context windows smaller on a phone? #

The context a model supports and the context an app actually uses are different numbers.

Qwen 3.5 supports a very large window natively, and Ministral 3 3B was trained for 256k tokens. But holding a long context costs working memory: the model keeps a cache for every token in the window, and that cache grows as the window grows. On a server with hundreds of gigabytes, that’s affordable. On a phone with 6 or 8 GB shared between the system and every other app, it isn’t.

So on-device apps run models with a much smaller window, typically a few thousand tokens, and let you raise it if your phone can take it. A bigger window also delays the first word of each reply, because there’s more text to read before the model can start. How much RAM you need to run an LLM on a phone explains where that memory goes.

How Personal LLM handles it #

In Personal LLM, which runs open models like Qwen 3.5 and Gemma 4 entirely on the phone, the context size defaults to 2,048 tokens and can be set anywhere from 512 to 8,192. The app warns you before you go above 2,048, since larger windows run out of memory on some phones, and chats with an attached image are capped at 4,096 because the photo takes memory too. The chat header shows how much of the window the conversation is using.

When a chat outgrows the window, the app drops the oldest messages first and shows a banner saying how many were left out, so the forgetting is never silent. Your system prompt isn’t dropped with them, which means rules you put there hold for the whole chat. The reply shares the window as well: a high max-tokens setting reserves more room for the answer and leaves less for history.

How to stop an AI from forgetting #

These work in any chatbot, cloud or local.

1. Put anything that must stick in the system prompt. Instructions, your preferences, a character’s personality, a project’s rules. It survives trimming, while instructions buried in message 3 are the first to go. Our system prompt examples are a good starting point.
2. Start a new chat for a new topic. One endless conversation about everything is the biggest single cause of forgetting.
3. Carry a summary forward. Before a long chat fills up, ask: “Summarize the key facts and decisions so far in 10 bullet points.” Paste that into a fresh chat, or into its system prompt.
4. Restate the detail that matters. “Remember, the budget is $500” costs a handful of tokens and saves a wrong answer.
5. Paste less. Share the paragraph or the function that matters, not the whole document.
6. Attach documents instead of pasting them. When you attach a PDF, text or Markdown file in Personal LLM, the app searches it on the phone and gives the model only the passages that match your question, then shows which ones it used. That handles far more text than would ever fit in the window. The walkthrough is in how to chat with a PDF offline.
7. Keep replies short. A lower max-tokens setting, or just “answer in under 100 words,” leaves more of the window for history.
8. Raise the context size if your phone can take it. Going from 2,048 to 4,096 doubles what the model can see. It costs memory and speed, so it suits phones with 8 GB of RAM or more. If the app slows badly or closes, go back down; why a local AI app crashes covers what that looks like.

Story games are where this bites hardest. Our offline text adventure guide shows how to keep a running “story so far” note that survives trimming.

Does a bigger context window mean better answers? #

Not automatically. A larger window lets the model see more, but:

• replies take longer to start, because the whole window is read first
• it uses more memory, which on a phone can mean a crash or a forced downgrade to a smaller model
• the model still pays less attention to the middle of very long inputs

For everyday chats, a modest window with a clean, focused conversation beats a huge window full of old tangents.

Frequently asked questions #

Why does ChatGPT forget what I told it earlier? #

The conversation has outgrown the model’s context window, or the app has summarized the older part to save room. Long chats that wander across many topics are the usual cause. Start a new chat, or ask for a summary and continue from that.

How much can an AI remember in one conversation? #

It depends on the context size the app uses, measured in tokens. Large cloud models can handle hundreds of thousands. Apps that run models on a phone usually use 2,000 to 8,000 tokens, which is roughly 1,500 to 6,000 words of conversation.

Does AI remember previous chats? #

Not by default. Each chat starts empty. Some cloud chatbots have a memory feature that saves facts about you and adds them to new chats, stored on their servers. Local apps generally don’t, though a reusable system prompt gives a similar result without a profile living anywhere.

What happens when the context window is full? #

The app drops or summarizes the oldest messages to make room, so early details are lost. Some apps tell you when this happens; others do it silently. Instructions in a system prompt are usually kept.

Why does my AI get slower as the chat gets longer? #

Every reply has to process the whole conversation in the window, so a longer chat means more work per word. Starting a new chat, or carrying a short summary into one, brings the speed back.