RAG explained simply: how to make an AI read your documents before it answers

Ask a chatbot about your company’s leave policy and one of two things happens. It says it doesn’t know, or it confidently makes something up. RAG is the fix for both, and it’s much simpler than its name suggests.

This post is for someone meeting RAG for the first time. No maths and no AI background needed. Every idea gets a picture and a plain explanation, and we put the pieces together at the end.

The problem: an AI taking a closed-book exam

A large language model (LLM) like GPT learned from a huge pile of text during training. Whatever it learned is stored inside the model itself, a bit like a student who memorised their notes before an exam. Ask about something outside those notes and three things can go wrong:

Why not just retrain the model?

The obvious idea is fine-tuning: keep training the model on your own data until it knows it. It works, but it has two big drawbacks:

• It’s expensive. Training a large model needs serious computing power and money.
• It doesn’t stay up to date. Add a new course or publish a new article and you have to train again.

There’s a smarter door into the model. Researchers noticed that big models can learn a task from examples written inside the prompt itself, with no retraining at all. That is called few-shot prompting. RAG uses the same door, but instead of sending examples it sends facts.

The idea: turn it into an open-book exam

With RAG, the AI gets to look at the relevant pages before it answers. We don’t change the model. We change what we hand it.

The name is just three jobs in a row:

• Retrieval: find the pieces of your data that matter for the question.
• Augmented: add those pieces to the prompt.
• Generation: let the LLM write the answer using them.

Because the answer comes from text you supplied, it is far less likely to be invented. It can also use private or brand-new information, since you can swap the data any time without touching the model.

One idea you need first: embeddings

There’s one question left: how does the system find the relevant pieces? Searching for matching keywords isn’t enough. “How much do I get paid?” shares no words with a document that says “salary”, yet they mean the same thing.

The trick is to turn text into a list of numbers called an embedding. Texts with similar meaning get similar numbers. The easiest way to picture it is a map where meaning decides the location:

Finding the closest points is called similarity search (usually measured with cosine similarity). It finds text that is conceptually close even when no words match.

The big picture: two phases

Every RAG system does its work in two phases. The first runs once, ahead of time. The second runs every time someone asks a question.

Phase 1: Indexing (once, ahead of time)

Loadread your files

→

Splitcut into chunks

→

Embedtext to numbers

→

Storevector database

↓ the answering phase searches what was stored above ↓

Phase 2: Answering (every question)

Questionfrom the user

→

Retrievefind best chunks

→

Augmentadd to prompt

→

GenerateLLM writes answer

Phase 1: Indexing, preparing the library

Step 1: Load your documents

Your data lives in PDFs, text files, web pages and spreadsheets. A document loader reads each source and converts it into the same standard shape, so everything after this step doesn’t care where the data came from. In LangChain that shape is a Document with two parts:

• page_content: the actual text.
• metadata: extra details such as the file name, page number or author.

There’s a loader for most common sources:

Loader	What it does
TextLoader	Reads plain text files such as logs, transcripts and notes.
PyPDFLoader	Reads a PDF page by page, one Document per page. Best for text-heavy PDFs and weaker on scanned images or complex layouts.
DirectoryLoader	Loads many files from a folder. You can filter by file type with a pattern like *.pdf.
WebBaseLoader	Pulls text from web pages. Good for static sites, but pages that depend heavily on JavaScript need other tools.
CSVLoader	Treats every row of a CSV file as its own Document.

Two more things worth knowing. .load() reads everything into memory at once, which is fine for small data. .lazy_load() hands you one document at a time, which is far safer for large data. And if no loader fits your source, you can write your own by extending BaseLoader.

Step 2: Split into chunks

Why not feed the whole document to the model? Two reasons. Models can only read a limited amount of text at once, and a small, focused chunk is much easier to match to a question than a 200-page PDF. So we cut documents into smaller pieces.

There are two common ways to cut. Length-based splitting chops every N characters or tokens. It’s fast and simple, but it can slice through the middle of a word or sentence and hurt search quality. Structure-based splitting, with the Recursive Character Text Splitter, is the usual choice. It tries the biggest natural break first (paragraphs), then lines, then words, and only goes smaller when a piece is still too long. Chunks end up as complete thoughts.

Either way you can set a chunk overlap: a small amount of text repeated at the edge of neighbouring chunks, so an idea that straddles a cut isn’t lost.

Step 3: Turn chunks into embeddings

Each chunk now goes through an embedding model, the same “text to meaning-numbers” step from the map earlier. Each chunk comes out as one vector that captures what it is about.

Step 4: Save them in a vector store

Those vectors need a home. Regular SQL databases aren’t built to search through thousands of long number lists by closeness, and comparing your question against every single vector would be slow. A vector store is built for exactly this. It keeps the vectors with their metadata (file name, page and so on), finds the nearest ones quickly using smart indexing, and still lets you add, read, update and delete records.

You’ll hear both “vector store” and “vector database”. A vector store is the lighter idea, focused on storing vectors and searching them (FAISS is an example). A vector database adds the heavy-duty extras such as running across many machines, saving data permanently and access control (Pinecone and Milvus are examples). Every vector database is a vector store, but not the other way round.

For learning and local projects, Chroma is a popular choice: lightweight, open source and easy to run on your own machine. LangChain wraps all of these behind the same methods, such as add_documents and similarity_search, so you can switch providers later without rewriting your app.

That completes Phase 1. The library is organised, and it only needs redoing when your data changes.

Phase 2: Answering a question

Step 1: Retrieve

A retriever is the librarian. It takes the user’s question and returns the most relevant documents from your data source. In LangChain it works as a building block that plugs straight into a chain. The simplest version converts the question into an embedding, then asks the vector store for the closest chunks. There are also retrievers for other sources, like Wikipedia or arXiv.

On top of that, you can pick a smarter search strategy:

Strategy	The idea
Similarity search	Return the chunks closest to the question. Simple and the default.
MMR	Maximum Marginal Relevance. Balance relevance with variety so you don’t get five copies of the same fact.
Multi-query	An LLM rewrites a vague question several ways, runs them all and merges the results without duplicates.
Contextual compression	Fetch documents first, then trim each one down to only the lines that matter for this question.

MMR is easiest to see side by side:

All three on the left are relevant, but they say one thing. MMR picks the best match first, then prefers chunks that are relevant but different from what it already has, so the model sees more of the picture.

Step 2: Augment

Now we build the prompt. This step is just text assembly: the retrieved chunks and the user’s question are placed in one template.

Answer the question using only the context below.
If the answer is not in the context, say "I don't know".

Context:
{the retrieved chunks go here}

Question:
{the user's question goes here}

That “say I don’t know” line matters. It gives the model permission to admit gaps instead of guessing, which is how RAG cuts down on hallucination.

Step 3: Generate

The LLM reads the filled-in prompt and writes an answer grounded in the context you handed it. The model only ever sees the few chunks that were retrieved, not your whole library.

Putting it all together

Stage	What happens	Library analogy
Load	Read files into Document objects	Collecting the books
Split	Cut into small chunks with overlap	Splitting books into pages
Embed + store	Chunks become vectors saved in a vector store	Shelving pages by topic
Retrieve	Find the chunks closest to the question	Librarian fetches pages
Augment	Put chunks + question in one prompt	Handing over the pages with the question
Generate	LLM answers from that context	Student writes with the book open

A last word on RAG versus fine-tuning, since the two get mixed up. Fine-tuning changes the model itself, which is useful when you want it to behave or write in a certain way. RAG leaves the model alone and gives it the right facts at question time, which is why it suits data that is private or changes often. Many real products use both.

If you remember one thing, make it this: RAG doesn’t make the model smarter. It makes sure the model has the right pages open when it answers.

I’m still learning this myself and writing it down helps me. If something here is unclear or wrong, tell me on X and I’ll fix it.