Module 3 — RAG Basics

3.1 Why Retrieval: The Knowledge Cutoff & Hallucination Problem

Learning objective: explain why retrieval-augmented generation exists as a pattern and what problem it actually solves.

Every model is trained on a fixed snapshot of data up to some cutoff date, and it has no built-in access to your private documents, your company’s internal wiki, or anything published after training. Ask it about something outside that knowledge, and it doesn’t reliably say “I don’t know” — it often produces a fluent, confident-sounding answer that’s simply wrong. That failure mode is hallucination, and it’s not a bug you can prompt your way out of; it’s a structural consequence of how these models generate text (Module 1).

Retrieval-Augmented Generation (RAG) sidesteps this by not asking the model to recall facts from memory at all. Instead, at query time, you fetch relevant real documents from an external source and hand them to the model as context, then ask it to answer using only what’s in front of it. The model’s job shifts from “remember the fact” to “read this and summarize/answer accurately” — a task models are considerably more reliable at.

3.2 Embeddings and Semantic Similarity, in Plain Terms

Learning objective: explain what an embedding is and why it enables “search by meaning” rather than keyword matching.

An embedding is a numerical vector representation of a piece of text, produced by a model trained so that texts with similar meaning end up as vectors that are close together in that vector space — even if they don’t share any of the same words.

flowchart TB
    subgraph Close in embedding space
    A["'How do I reset my password'"]
    B["'I forgot my login credentials'"]
    end
    subgraph Far in embedding space
    C["'How do I bake bread'"]
    end
    A -.->|"small distance"| B
    A -.->|"large distance"| C

This is what makes semantic search possible: instead of matching keywords, you embed the user’s query and find the stored document chunks whose embeddings are nearest to it (typically via cosine similarity):

import numpy as np

def cosine_similarity(a: np.ndarray, b: np.ndarray) -> float:
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

def top_k_chunks(query_embedding, chunk_embeddings: list[tuple[str, np.ndarray]], k=3):
    scored = [
        (text, cosine_similarity(query_embedding, emb))
        for text, emb in chunk_embeddings
    ]
    scored.sort(key=lambda pair: pair[1], reverse=True)
    return scored[:k]

It’s approximate and probabilistic, not exact — which is a tradeoff worth sitting with, because it’s the source of most RAG quality problems in practice: a document semantically close to your query isn’t guaranteed to be the correct source for it.

3.3 Chunking Strategies and Why Chunk Size Matters

Learning objective: explain the tradeoffs in choosing how documents are split before being embedded and stored.

You can’t usefully embed an entire 50-page document as a single vector — too much gets averaged away, and it won’t fit usefully into a model’s context anyway. So documents get split into chunks before embedding: paragraphs, fixed-size windows, or semantically-aware splits (e.g., breaking at section headers).

def chunk_text(text: str, chunk_size: int = 400, overlap: int = 50) -> list[str]:
    """Naive fixed-size chunker with overlap, operating on whitespace-split words.
    A real system would chunk on tokens, not words, and often on semantic
    boundaries (headers, paragraphs) rather than a blind fixed window."""
    words = text.split()
    chunks = []
    start = 0
    while start < len(words):
        end = start + chunk_size
        chunks.append(" ".join(words[start:end]))
        start = end - overlap  # step back by `overlap` so ideas spanning
                                # the boundary appear in both chunks
    return chunks

The size tradeoff is real in both directions:

  • Chunks too small lose surrounding context — a sentence fragment about “the fee” with no indication of which fee is nearly useless once retrieved on its own.
  • Chunks too large dilute the embedding (mixing multiple topics into one vector, making retrieval less precise) and waste context budget once retrieved.

A common practical starting point is a few hundred tokens per chunk with some overlap between consecutive chunks, so an idea that spans a chunk boundary isn’t cut off entirely in either piece. There’s no universally correct number — the right size depends on your document structure and how granular your queries tend to be.

3.4 The RAG Pipeline End to End

Learning objective: describe the full RAG pipeline as a sequence of concrete steps.

Putting it together, a RAG system runs in two phases:

flowchart LR
    subgraph Indexing [Indexing — done ahead of time]
    D["Source documents"] --> CH["Chunk"]
    CH --> EM1["Embed each chunk"]
    EM1 --> VS[("Vector store")]
    end

    subgraph Query [Query time — done per question]
    Q["User query"] --> EM2["Embed query"]
    EM2 --> R["Retrieve top-k\nnearest chunks"]
    VS --> R
    R --> AUG["Augment prompt\nwith retrieved chunks"]
    AUG --> GEN["Generate answer\n(grounded in context)"]
    end

Indexing (done ahead of time, whenever documents change):

  1. Split source documents into chunks
  2. Embed each chunk into a vector
  3. Store each vector alongside its original text in a vector database

Query time (done for every user question):

  1. Embed the incoming query using the same embedding model
  2. Retrieve the top-k nearest chunks by vector similarity
  3. Augment the model’s context with those retrieved chunks
  4. Generate an answer, instructed to rely on the retrieved context rather than prior knowledge

A minimal version of the query-time half, tying together 3.2’s similarity function and 3.3’s chunker:

def answer_question(question: str, indexed_chunks, embed_fn, llm_call, k=3):
    query_embedding = embed_fn(question)
    top_chunks = top_k_chunks(query_embedding, indexed_chunks, k=k)
    context = "\n\n---\n\n".join(text for text, _score in top_chunks)

    system_prompt = (
        "Answer the user's question using ONLY the context below. "
        "If the context doesn't contain the answer, say so explicitly "
        "rather than guessing.\n\nContext:\n" + context
    )
    return llm_call(system_prompt=system_prompt, user_message=question)

Every stage is a place quality can silently degrade — bad chunking, a query phrased differently than the source material, or a retrieval count (k) that’s too low to capture the right chunk. When a RAG system gives a wrong answer, the first debugging question is always “did we even retrieve the right chunk?” before blaming the model’s generation.

Exercise

Manually “retrieve” the right chunk from a small doc set (5-10 short paragraphs on different topics) for 3 sample queries — by reading and judging similarity yourself rather than using an embedding model. Then reflect: were there queries where the literally correct chunk didn’t share obvious keywords with the query? Those are the cases embeddings are built to catch.

Quiz

  1. Why doesn’t asking a model to “just be more careful” fix hallucination on out-of-knowledge questions?
  2. In your own words, what does it mean for two pieces of text to be “close” in embedding space?
  3. Name one failure mode from chunks that are too small, and one from chunks that are too large.
  4. List the four query-time steps of a RAG pipeline in order.

Answer key: TODO — write once the module has been through at least one test learner.

← Back to Track 1 · Next: Module 4 — Model Capabilities & Limitations →