RAG and Embeddings

A support chatbot gets asked, "Can I return shoes I bought three weeks ago?" It answers, confidently, "Returns are accepted within 14 days of purchase." The real policy is 30 days. Nobody wrote 14 anywhere. The model made it up.

This is not a bug in the usual sense. As how-language-models-work laid out, the model predicts the next token from patterns it absorbed during training, and it never read this company's handbook. The handbook is private, or it was written after the model's training cutoff, or both. Asked about it anyway, the model produces text that sounds like a return policy, because sounding right is the only thing it was trained to do. It has no way to know the real number.

Retraining the model on the handbook is possible but slow, costly, and stale the day the policy changes. There is a cheaper move. Find the exact passage of the handbook that answers the question, and paste it into the prompt right before asking. The model reads the real policy and answers from it. That trick, done properly, is called retrieval-augmented generation, or RAG, and it is how most real AI products give a model knowledge it was never trained on.

Why not just paste everything in

The obvious version of this idea is to skip the "find the passage" step and drop the whole handbook into the prompt. building-on-the-api showed how code sends a prompt to the model and gets a response, so pasting 200 pages of text ahead of the question is one string away. For a single short document, that is genuinely the right call, and it is simpler than everything below. The trouble starts when the knowledge is large.

Every model reads a bounded amount of text at once, the context window from how-language-models-work, measured in tokens. A company wiki, a codebase, or a year of support tickets is far larger than any window, even the large ones that hold hundreds of thousands of tokens. It simply does not fit.

Cost and speed are the next wall. The API charges per token on every single call, and a bigger prompt takes longer to process. Sending 200 pages with every question means paying for 200 pages every question, and waiting longer for each answer, to use one paragraph of it.

The quiet problem is quality. A model does not attend evenly across a very long prompt. A relevant fact buried in the middle of 100,000 tokens of noise gets found less reliably than the same fact sitting alone at the top. Researchers call this the "lost in the middle" effect, and it means more context can make an answer worse, not better. So the better move is to send the model only the few passages that actually bear on the question. Everything in this lesson is machinery for finding those passages.

An embedding turns meaning into numbers

Finding "the passages that bear on the question" is the hard part, because the right passage often shares no words with the question. Someone asks "how do I get my money back," and the answer lives under a heading that says "Refunds." A plain keyword search, matching identical words, misses it. What is needed is a way to match on meaning.

An embedding is how a machine measures meaning. An embedding model, a neural network that is a close cousin of the language model, reads a piece of text and outputs a fixed-length list of numbers called a vector. The same text always produces the same vector. "Refund policy" might come out as a list starting [0.02, -0.19, 0.44, ...], and it is the same list every time.

The point of those numbers is that they encode meaning. Two texts that mean similar things get vectors that sit close together, and two unrelated texts get vectors far apart. "How do I get my money back" and "Refunds are issued to the original card" land near each other even with no shared words, because the embedding model learned, from enormous amounts of text, that those phrases travel in the same company.

A real embedding vector is not three numbers, it is a few hundred to a few thousand: common models output 384, 1,536, or 3,072 of them. Each number is one coordinate, so the vector is a single point in a space with that many dimensions. Nobody can picture 1,536 dimensions, and nobody needs to. The only thing that matters is a measure of how close two points are.

The usual measure is cosine similarity. It compares the direction two vectors point, ignoring their length, and returns a number from 1 down to -1. A score of 1 means the two texts point the same way, meaning nearly the same thing. A score near 0 means unrelated. Negative means opposite. Most retrieval systems rank passages by cosine similarity to the question and take the highest scorers.

The whole mechanic fits in a few lines of Python. Real embeddings would come from a model, but shrinking each text to three hand-picked numbers keeps the math visible.

import math

def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    mag_a = math.sqrt(sum(x * x for x in a))
    mag_b = math.sqrt(sum(y * y for y in b))
    return dot / (mag_a * mag_b)

# Pretend an embedding model turned each sentence into 3 numbers.
# Real embeddings use hundreds or thousands; 3 keeps it readable.
question = [0.91, 0.10, 0.05]

chunks = {
    "Returns are accepted within 30 days of delivery.": [0.83, 0.22, 0.11],
    "The office is closed on all federal holidays.":     [0.09, 0.90, 0.16],
    "Refunds are issued to the original payment card.":  [0.62, 0.20, 0.55],
}

scored = sorted(chunks, key=lambda text: cosine(question, chunks[text]), reverse=True)
for text in scored:
    print(round(cosine(question, chunks[text]), 3), "|", text)

The returns passage scores highest, the refund passage is a close second, and the holidays passage sits far down where it belongs. That ranking is the entire job of vector search: turn the question into a vector, compare it against every stored vector, and hand back the closest. Swap in real embeddings with 1,536 numbers each and the code is the same, only the vectors get longer and the meanings get sharper.

The retrieval-augmented-generation pipeline

RAG has two phases. One runs once, ahead of time, to build a searchable index of the documents. The other runs on every question. Keeping them separate is what makes the system fast at answer time.

The build phase has three steps. First, chunk the documents. A whole document is too big to be a good unit of retrieval, and it mixes many topics, so it gets split into passages of a few hundred tokens each, roughly a paragraph or two. Chunks often overlap by a sentence or so, so an idea split across a boundary still lands whole inside one chunk. Second, embed every chunk by running it through the embedding model, producing one vector per chunk. Third, store the vectors, alongside the original text, in a vector store: a database built for nearest- neighbor search, such as pgvector, FAISS, or Chroma. It indexes the vectors so a search does not have to compare against every one by brute force.

The query phase runs when a question arrives. The question goes through the same embedding model to become a vector. The vector store returns the top matches, usually the closest three to eight chunks, by cosine similarity. Those chunks get pasted into the prompt as context, above the question, with an instruction like "answer using only the context below." That full prompt goes to the model through the API, and the model writes an answer grounded in the retrieved text. Retrieval finds the facts, generation writes the reply. That is the name.

In code, the query phase is short. The embedding model and the store are shown here as functions to keep the shape clear.

# Sketch: embed_text() and store are provided by an embedding
# model and a vector store, not shown here.

def answer(question, store, client):
    q_vector = embed_text(question)
    top_chunks = store.search(q_vector, k=4)   # 4 closest passages

    context = "\n\n".join(top_chunks)
    prompt = (
        "Answer using only the context below. "
        "If the answer is not in it, say you do not know.\n\n"
        f"Context:\n{context}\n\n"
        f"Question: {question}"
    )
    return client.generate(prompt)

Trace the support bot through it. The handbook gets chopped into 400 chunks, each embedded and stored, once. A user asks, "Can I return shoes I bought three weeks ago?" That question becomes a vector, the store returns the four nearest chunks, and the top one is "Returns are accepted within 30 days of delivery." The prompt now carries the real policy, so the model answers, "Yes, the window is 30 days and three weeks is inside it," and it can point at the passage it used. The model never memorized the policy. It read it at answer time.

RAG gives a model the right facts, but the model still only writes an answer. It does not check the order status, book the return, or run a search when the first retrieval comes up empty. The next lesson, ai-agents-and-tools, is about a model that does not just answer from what it is handed but decides what to do next and takes the action itself.