RAG stands for retrieval-augmented generation. Before the AI answers, the app searches your document for the passages that match your question and hands only those to the model, which writes its answer from them. It’s how every “chat with your PDF” feature works, because a language model can’t read a long document in one go.
Knowing how it works explains most of the frustrating moments, like the AI saying something isn’t in a file when it clearly is.
Why can’t the AI just read the whole document? #
A model can only consider a limited amount of text at once, called its context window. A 100-page contract is roughly 50,000 tokens (word pieces). Cloud models can take in a lot, but on a phone, context uses memory and slows every reply, so local apps work with a few thousand tokens at a time.
RAG gets around that. Instead of feeding in the whole document, the app finds the few paragraphs that matter for this question and feeds in those.
How does RAG work, step by step? #
- Extract the text. The app pulls the text out of the PDF or file.
- Split it into chunks. The text is cut into pieces of a few hundred words, ideally at paragraph or sentence boundaries.
- Index the chunks. The app builds a search index so it can find chunks quickly.
- Search with your question. When you ask something, your question is matched against the index and the chunks are ranked.
- Add the best chunks to the prompt. The top passages are inserted into the model’s instructions, as much as fits.
- Generate the answer. The model answers using those passages, and a good app shows you which ones it used.
Keyword search vs embeddings: what’s the difference? #
Step 4 can work in two main ways.
Keyword search ranks chunks by the words they share with your question, giving more weight to rare words and to chunks where they appear often. The standard method is called BM25. It’s fast, needs no extra model, and is excellent for “where does it say X” questions. Its weakness is synonyms: ask about “ending the lease” when the document says “termination,” and it may miss.
Embedding search turns each chunk and your question into a list of numbers that represents meaning (a vector). Chunks with similar meaning end up close together even if they share no words, so “car” can match “automobile.” It catches paraphrases, but it needs a second AI model to create the vectors, which costs time, storage and memory.
| Keyword search (BM25) | Embedding search | |
|---|---|---|
| Finds exact terms, names, numbers | Very well | Sometimes misses them |
| Finds paraphrases and synonyms | Poorly | Well |
| Extra model needed | No | Yes |
| Memory and battery cost on a phone | Tiny | Noticeable |
| Speed | Near instant | Slower to index |
Many large systems combine both, which is called hybrid search.
How does chat-with-PDF work in Personal LLM? #
Personal LLM takes the keyword approach, entirely on the phone. When you attach a PDF, text or Markdown file to a chat, the app extracts the text, splits it into chunks of about 300 tokens, and ranks them against each question with BM25. The best excerpts are added to the prompt, and the answer tells you which passages it used, so you can check them. There’s no embedding model, no second engine, and no extra RAM, and the file is never uploaded.
The trade-off is the one in the table: it’s precise for questions that use the document’s own words, and weaker when your wording differs from the document’s. Our step-by-step guide to chatting with a PDF offline shows the workflow.
Why does RAG miss things? #
Most failures come from the retrieval step, not the model.
- Different words. You ask about “pets,” the lease says “animals.” Keyword search can’t connect them.
- The answer is spread out. If the rule is on page 3 and the exception on page 40, the app may only retrieve one of them.
- Summary questions. “Summarize this whole report” doesn’t match any particular passage, so the model sees a few random-looking chunks instead of the whole thing.
- No text layer. A scanned PDF that’s just images of pages has no text to extract. Take photos of the pages and ask about them with a vision model instead.
- Tables and columns. Text extraction can scramble multi-column layouts and tables, so numbers end up next to the wrong labels.
How to get better answers from your documents #
- Use the document’s vocabulary. If you know it says “termination,” ask about termination.
- Ask narrow questions. “What’s the notice period for ending early?” works better than “Tell me about ending the lease.”
- Ask for the quote. “Quote the exact sentence” makes it easy to verify.
- Break big questions up. Ask about each section, then ask the model to combine the answers you got.
- Check the cited passages. If the answer and the passage don’t match, trust the passage.
- Follow up with synonyms when it says it can’t find something: “It might be called ‘animals’ or ’livestock.’”
Does RAG stop AI from making things up? #
It reduces it, because the model has the right text in front of it. It doesn’t eliminate it. A model can still misread a passage, mix two passages together, or fill a gap from its general knowledge. Small on-device models are more prone to this than large cloud ones. See why AI makes things up for ways to reduce it.
Is RAG private? #
That depends on where the search and the model run. Most “chat with PDF” websites and cloud chatbots upload your file to their servers to do both steps. When the extraction, the index and the model all run on your phone, the document never leaves it. That matters for contracts, medical records and anything with account numbers.
Frequently asked questions #
What does RAG stand for? #
Retrieval-augmented generation. The model’s answer (generation) is improved (augmented) by first fetching relevant text (retrieval) from your documents or another source.
What’s the difference between RAG and fine-tuning? #
RAG gives a model the right information at the moment you ask, without changing the model. Fine-tuning retrains the model’s parameters on new data. For “answer from this document,” RAG is simpler, cheaper and easy to update.
Can RAG work offline? #
Yes. If the app extracts text, searches it and runs the model on your device, the whole process works without a connection. Keyword search is especially light, so it runs well on phones.
Why does my AI say something isn’t in the PDF when it is? #
Usually the search didn’t retrieve that passage, often because your question uses different words from the document, or the text couldn’t be extracted. Rephrase with the document’s terms, ask about a specific section, or check that the PDF has selectable text.
Can RAG read a scanned document? #
Only if the scan has a text layer, which some scanner apps add. A plain image of a page has no text for the app to search. In that case, photograph the page and ask a vision model to read it.