Lab 19: Retrieval-Augmented Generation (RAG)¶

Lab 18 built a retriever that finds relevant passages. This lab hands those passages to a language model and asks it to answer — which is the entire idea behind Retrieval-Augmented Generation.

RAG is the most common way to make a general-purpose language model useful on your data, and it needs no training at all. Everything here runs locally on CPU.

We will cover the following topics:

• What RAG fixes, and when to reach for it instead of fine-tuning
• Building the full pipeline: chunk, embed, index, retrieve, prompt, generate
• Closed-book versus grounded answers, measured side by side
• Making the model say "I don't know" instead of inventing an answer
• Citations, and why they are the main reason to use RAG at all
• The knobs that matter: chunk size, top-k, and the retrieval threshold

1. Why Retrieval-Augmented Generation?¶

A language model knows what was in its training data, frozen at the moment training stopped. That causes three specific problems:

1. It does not know your data. Your internal documents, your codebase, last week's meeting notes — none of it was in the training set.
2. It hallucinates. Asked something it does not know, a language model does not fall silent. It produces the most plausible-sounding continuation, which is often a confident, fluent, wrong answer.
3. It cannot cite. The knowledge is smeared across billions of weights, so there is nothing to point at when someone asks "where did that come from?"

RAG addresses all three by retrieving relevant text and putting it in the prompt. The model is never retrained; it is simply given the reference material at the moment of the question.

	Fine-tuning	RAG
Adding new facts	retrain	add a document to the index
Removing a fact	retrain, and hope	delete the row
Citations	not really possible	built in
Up-front cost	GPU hours	an embedding pass
Per-query cost	normal	plus retrieval and a longer prompt
Best for	style, format, behaviour	facts, documents, anything that changes

The rule of thumb: fine-tune to change how a model behaves, retrieve to change what it knows.

In [1]:

import torch
import torch.nn.functional as F
import numpy as np
import matplotlib.pyplot as plt
import textwrap

torch.manual_seed(0)
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print('PyTorch version:', torch.__version__, '| device:', device)

import torch import torch.nn.functional as F import numpy as np import matplotlib.pyplot as plt import textwrap torch.manual_seed(0) device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') print('PyTorch version:', torch.__version__, '| device:', device)

PyTorch version: 2.13.0+cpu | device: cpu

2. The Knowledge Base¶

RAG needs a corpus the model demonstrably does not already know. We will use documentation of this repository — the labs you have just worked through. A small language model has never seen it, so any correct answer must have come from retrieval.

Each document carries a title, which is what makes citation possible later.

In [2]:

DOCUMENTS = [
    ("Lab 06 - Neurons and Activation Functions",
     "A neuron computes a weighted sum of its inputs, adds a bias, and applies an activation "
     "function. Without a non-linear activation, stacking layers would collapse into a single "
     "linear transformation. The sigmoid squashes values into the range 0 to 1 but saturates "
     "at the extremes, where its gradient nearly vanishes. ReLU returns zero for negative "
     "inputs and the input itself otherwise, which is cheap and does not saturate on the "
     "positive side, and it is the default choice for hidden layers."),

    ("Lab 08 - Dense Neural Networks",
     "A dense or fully connected network connects every neuron in one layer to every neuron "
     "in the next. Lab 08 trains one on the make_moons dataset from scikit-learn using binary "
     "cross-entropy loss and the Adam optimizer, reaching high accuracy on a problem that no "
     "straight line can separate. Dense layers are the building block that every other "
     "architecture eventually falls back on for its final classification head."),

    ("Lab 09 - Convolutional Neural Networks",
     "A convolutional layer slides a small learned filter across an image and computes a dot "
     "product at every position, producing a feature map. Pooling then downsamples the feature "
     "map, which reduces computation and makes the network tolerant of small shifts. Lab 09 "
     "trains a CNN with two convolutional blocks on the MNIST handwritten digit dataset. "
     "Before the fully connected head, the feature maps must be flattened with x.view(x.size(0), -1)."),

    ("Lab 10 - Recurrent Neural Networks",
     "A recurrent network processes a sequence one element at a time while carrying a hidden "
     "state that summarises everything seen so far. The same weights are reused at every step, "
     "which is called parameter sharing and lets one network handle sequences of any length. "
     "Lab 10 forecasts a noisy sine wave and measures the vanishing gradient problem: the "
     "gradient reaching the first time step shrinks by orders of magnitude as sequences grow, "
     "so a vanilla RNN cannot learn long-range dependencies."),

    ("Lab 11 - Long Short-Term Memory",
     "An LSTM adds a cell state that runs through the whole sequence touched only by an "
     "elementwise multiplication and an addition, so information and gradients can travel a "
     "long way unchanged. Three gates control it: the forget gate decides what to erase, the "
     "input gate decides what to write, and the output gate decides what to expose. In "
     "PyTorch the four gate weight blocks are packed in the order input, forget, candidate, "
     "output. A GRU is a lighter variant with two gates and about 25 percent fewer parameters."),

    ("Lab 12 - Transfer Learning",
     "Transfer learning reuses a network trained on a large dataset such as ImageNet and "
     "replaces only its final classification layer. Early layers learn edges and textures, "
     "which are useful for almost any image task, so only the last layers need to change. "
     "Feature extraction freezes the backbone with requires_grad = False and trains just the "
     "new head. Fine-tuning then unfreezes the last block at a learning rate about ten times "
     "smaller. Always train the head first, or large early gradients will damage the "
     "pretrained weights."),

    ("Lab 13 - Regularization and Optimization",
     "Overfitting shows up as a widening gap between training and validation loss. Dropout "
     "randomly switches off units during training so no neuron can be relied upon. Weight "
     "decay adds the squared norm of the parameters to the loss, keeping weights small. Batch "
     "normalization standardises layer inputs and mainly speeds up convergence. Data "
     "augmentation is usually the most effective regularizer for images. Early stopping halts "
     "training when validation loss stops improving and restores the best weights. Every one "
     "of these has a sweet spot: too little overfits, too much underfits."),

    ("Lab 14 - Attention and Transformers",
     "Attention compares a query vector against every key vector, converts the scores into "
     "weights with a softmax, and returns a weighted average of the value vectors. The scores "
     "are divided by the square root of the key dimension, because without that scaling the "
     "dot products grow with dimension and push the softmax into a saturated regime with "
     "almost no gradient. Multi-head attention runs several attention operations in parallel "
     "on different projections at no extra cost. Attention is permutation equivariant, so "
     "positional encoding must be added to represent order."),

    ("Lab 15 - Autoencoders and VAEs",
     "An autoencoder compresses its input through a narrow bottleneck and is trained to "
     "reconstruct the original, so it learns without labels. Its latent space contains holes, "
     "which is why decoding a random vector produces nonsense. A variational autoencoder makes "
     "the encoder output a mean and a log-variance instead of a point, and adds a KL "
     "divergence term pulling those distributions towards a standard normal. The "
     "reparameterization trick writes z as mu plus sigma times epsilon, moving the randomness "
     "into an input so gradients can flow through it."),

    ("Lab 16 - Generative Adversarial Networks",
     "A GAN trains a generator against a discriminator. The generator turns random noise into "
     "fake images and never sees real data; the discriminator judges whether an image is real. "
     "In practice the generator maximises log D(G(z)) rather than minimising log(1 - D(G(z))), "
     "because the original form saturates early in training. Loss curves mean little here: the "
     "useful diagnostic is whether D(real) and D(fake) both approach 0.5. Mode collapse, where "
     "the generator produces only a few outputs, is the characteristic failure."),

    ("Lab 17 - Deployment and Model Optimization",
     "Save a model with torch.save(model.state_dict(), path) rather than pickling the whole "
     "object, because pickle stores a reference to your class and breaks when you refactor. "
     "Load with map_location='cpu' and weights_only=True, then call model.eval(). TorchScript "
     "tracing records only the operations that ran, so it silently discards data-dependent "
     "control flow; use torch.jit.script when the forward pass branches. Dynamic quantization "
     "stores weights as 8-bit integers, giving roughly a fourfold size reduction for a "
     "fraction of a point of accuracy."),

    ("Lab 18 - Embeddings and Semantic Search",
     "An embedding maps text to a vector so that similar meanings land close together. A "
     "sentence vector is produced by mean pooling the token vectors while ignoring padding "
     "positions. Normalising the vectors to unit length makes the dot product equal to the "
     "cosine similarity, so an entire search becomes one matrix multiplication. Retrieval "
     "quality is measured with recall@k and mean reciprocal rank. A similarity search always "
     "returns its nearest neighbour, so a score threshold is the only way to express that "
     "nothing relevant was found."),

    ("Environment setup",
     "The labs run in a conda environment. Create it with conda create -n envNeural python=3.12 "
     "and install PyTorch, torchvision, numpy, matplotlib, scikit-learn and transformers. "
     "The labs are CPU-friendly, though Lab 12 and Lab 16 are noticeably faster on a GPU. "
     "Datasets download automatically into a local data directory on first run."),

    ("Repository layout",
     "Each lab is a numbered Jupyter notebook in the repository root, from Lab-01 through "
     "Lab-19. Shared images, diagrams and animations live in the Misc folder and are "
     "referenced from the notebooks by relative path. The README lists every lab with a short "
     "description and links to further reading."),
]

print(f'{len(DOCUMENTS)} documents, '
      f'{sum(len(text.split()) for _, text in DOCUMENTS)} words total')

DOCUMENTS = [ ("Lab 06 - Neurons and Activation Functions", "A neuron computes a weighted sum of its inputs, adds a bias, and applies an activation " "function. Without a non-linear activation, stacking layers would collapse into a single " "linear transformation. The sigmoid squashes values into the range 0 to 1 but saturates " "at the extremes, where its gradient nearly vanishes. ReLU returns zero for negative " "inputs and the input itself otherwise, which is cheap and does not saturate on the " "positive side, and it is the default choice for hidden layers."), ("Lab 08 - Dense Neural Networks", "A dense or fully connected network connects every neuron in one layer to every neuron " "in the next. Lab 08 trains one on the make_moons dataset from scikit-learn using binary " "cross-entropy loss and the Adam optimizer, reaching high accuracy on a problem that no " "straight line can separate. Dense layers are the building block that every other " "architecture eventually falls back on for its final classification head."), ("Lab 09 - Convolutional Neural Networks", "A convolutional layer slides a small learned filter across an image and computes a dot " "product at every position, producing a feature map. Pooling then downsamples the feature " "map, which reduces computation and makes the network tolerant of small shifts. Lab 09 " "trains a CNN with two convolutional blocks on the MNIST handwritten digit dataset. " "Before the fully connected head, the feature maps must be flattened with x.view(x.size(0), -1)."), ("Lab 10 - Recurrent Neural Networks", "A recurrent network processes a sequence one element at a time while carrying a hidden " "state that summarises everything seen so far. The same weights are reused at every step, " "which is called parameter sharing and lets one network handle sequences of any length. " "Lab 10 forecasts a noisy sine wave and measures the vanishing gradient problem: the " "gradient reaching the first time step shrinks by orders of magnitude as sequences grow, " "so a vanilla RNN cannot learn long-range dependencies."), ("Lab 11 - Long Short-Term Memory", "An LSTM adds a cell state that runs through the whole sequence touched only by an " "elementwise multiplication and an addition, so information and gradients can travel a " "long way unchanged. Three gates control it: the forget gate decides what to erase, the " "input gate decides what to write, and the output gate decides what to expose. In " "PyTorch the four gate weight blocks are packed in the order input, forget, candidate, " "output. A GRU is a lighter variant with two gates and about 25 percent fewer parameters."), ("Lab 12 - Transfer Learning", "Transfer learning reuses a network trained on a large dataset such as ImageNet and " "replaces only its final classification layer. Early layers learn edges and textures, " "which are useful for almost any image task, so only the last layers need to change. " "Feature extraction freezes the backbone with requires_grad = False and trains just the " "new head. Fine-tuning then unfreezes the last block at a learning rate about ten times " "smaller. Always train the head first, or large early gradients will damage the " "pretrained weights."), ("Lab 13 - Regularization and Optimization", "Overfitting shows up as a widening gap between training and validation loss. Dropout " "randomly switches off units during training so no neuron can be relied upon. Weight " "decay adds the squared norm of the parameters to the loss, keeping weights small. Batch " "normalization standardises layer inputs and mainly speeds up convergence. Data " "augmentation is usually the most effective regularizer for images. Early stopping halts " "training when validation loss stops improving and restores the best weights. Every one " "of these has a sweet spot: too little overfits, too much underfits."), ("Lab 14 - Attention and Transformers", "Attention compares a query vector against every key vector, converts the scores into " "weights with a softmax, and returns a weighted average of the value vectors. The scores " "are divided by the square root of the key dimension, because without that scaling the " "dot products grow with dimension and push the softmax into a saturated regime with " "almost no gradient. Multi-head attention runs several attention operations in parallel " "on different projections at no extra cost. Attention is permutation equivariant, so " "positional encoding must be added to represent order."), ("Lab 15 - Autoencoders and VAEs", "An autoencoder compresses its input through a narrow bottleneck and is trained to " "reconstruct the original, so it learns without labels. Its latent space contains holes, " "which is why decoding a random vector produces nonsense. A variational autoencoder makes " "the encoder output a mean and a log-variance instead of a point, and adds a KL " "divergence term pulling those distributions towards a standard normal. The " "reparameterization trick writes z as mu plus sigma times epsilon, moving the randomness " "into an input so gradients can flow through it."), ("Lab 16 - Generative Adversarial Networks", "A GAN trains a generator against a discriminator. The generator turns random noise into " "fake images and never sees real data; the discriminator judges whether an image is real. " "In practice the generator maximises log D(G(z)) rather than minimising log(1 - D(G(z))), " "because the original form saturates early in training. Loss curves mean little here: the " "useful diagnostic is whether D(real) and D(fake) both approach 0.5. Mode collapse, where " "the generator produces only a few outputs, is the characteristic failure."), ("Lab 17 - Deployment and Model Optimization", "Save a model with torch.save(model.state_dict(), path) rather than pickling the whole " "object, because pickle stores a reference to your class and breaks when you refactor. " "Load with map_location='cpu' and weights_only=True, then call model.eval(). TorchScript " "tracing records only the operations that ran, so it silently discards data-dependent " "control flow; use torch.jit.script when the forward pass branches. Dynamic quantization " "stores weights as 8-bit integers, giving roughly a fourfold size reduction for a " "fraction of a point of accuracy."), ("Lab 18 - Embeddings and Semantic Search", "An embedding maps text to a vector so that similar meanings land close together. A " "sentence vector is produced by mean pooling the token vectors while ignoring padding " "positions. Normalising the vectors to unit length makes the dot product equal to the " "cosine similarity, so an entire search becomes one matrix multiplication. Retrieval " "quality is measured with recall@k and mean reciprocal rank. A similarity search always " "returns its nearest neighbour, so a score threshold is the only way to express that " "nothing relevant was found."), ("Environment setup", "The labs run in a conda environment. Create it with conda create -n envNeural python=3.12 " "and install PyTorch, torchvision, numpy, matplotlib, scikit-learn and transformers. " "The labs are CPU-friendly, though Lab 12 and Lab 16 are noticeably faster on a GPU. " "Datasets download automatically into a local data directory on first run."), ("Repository layout", "Each lab is a numbered Jupyter notebook in the repository root, from Lab-01 through " "Lab-19. Shared images, diagrams and animations live in the Misc folder and are " "referenced from the notebooks by relative path. The README lists every lab with a short " "description and links to further reading."), ] print(f'{len(DOCUMENTS)} documents, ' f'{sum(len(text.split()) for _, text in DOCUMENTS)} words total')

14 documents, 1074 words total

3. Chunk, Embed, Index¶

The same three steps as Lab 18, with one addition: each chunk keeps a reference to the document it came from, so we can cite it later. In a production system this is "metadata", and you would also store timestamps, permissions and a URL.

In [3]:

from transformers import AutoTokenizer, AutoModel

EMBED_MODEL = 'sentence-transformers/all-MiniLM-L6-v2'
embed_tokenizer = AutoTokenizer.from_pretrained(EMBED_MODEL)
embed_model = AutoModel.from_pretrained(EMBED_MODEL).to(device)
embed_model.eval()
print(f'Embedding model loaded: {sum(p.numel() for p in embed_model.parameters()):,} parameters')

def mean_pool(token_embeddings, attention_mask):
    mask = attention_mask.unsqueeze(-1).float()
    return (token_embeddings * mask).sum(1) / mask.sum(1).clamp(min=1e-9)

@torch.inference_mode()
def embed(texts, batch_size=32):
    if isinstance(texts, str):
        texts = [texts]
    out = []
    for start in range(0, len(texts), batch_size):
        tokens = embed_tokenizer(texts[start:start + batch_size], padding=True,
                                 truncation=True, max_length=256,
                                 return_tensors='pt').to(device)
        hidden = embed_model(**tokens).last_hidden_state
        out.append(F.normalize(mean_pool(hidden, tokens['attention_mask']), p=2, dim=1).cpu())
    return torch.cat(out)

from transformers import AutoTokenizer, AutoModel EMBED_MODEL = 'sentence-transformers/all-MiniLM-L6-v2' embed_tokenizer = AutoTokenizer.from_pretrained(EMBED_MODEL) embed_model = AutoModel.from_pretrained(EMBED_MODEL).to(device) embed_model.eval() print(f'Embedding model loaded: {sum(p.numel() for p in embed_model.parameters()):,} parameters') def mean_pool(token_embeddings, attention_mask): mask = attention_mask.unsqueeze(-1).float() return (token_embeddings * mask).sum(1) / mask.sum(1).clamp(min=1e-9) @torch.inference_mode() def embed(texts, batch_size=32): if isinstance(texts, str): texts = [texts] out = [] for start in range(0, len(texts), batch_size): tokens = embed_tokenizer(texts[start:start + batch_size], padding=True, truncation=True, max_length=256, return_tensors='pt').to(device) hidden = embed_model(**tokens).last_hidden_state out.append(F.normalize(mean_pool(hidden, tokens['attention_mask']), p=2, dim=1).cpu()) return torch.cat(out)

Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.

Embedding model loaded: 22,713,216 parameters

In [4]:

def chunk_text(text, chunk_words=60, overlap=15):
    words = text.split()
    step = max(chunk_words - overlap, 1)
    chunks = []
    for start in range(0, max(len(words) - overlap, 1), step):
        chunk = words[start:start + chunk_words]
        if chunk:
            chunks.append(' '.join(chunk))
        if start + chunk_words >= len(words):
            break
    return chunks

def build_index(documents, chunk_words=60, overlap=15):
    """Split every document into chunks, embed them, and keep the source title."""
    chunks, sources = [], []
    for title, body in documents:
        for chunk in chunk_text(body, chunk_words, overlap):
            chunks.append(chunk)
            sources.append(title)
    return {'chunks': chunks, 'sources': sources, 'vectors': embed(chunks)}

index = build_index(DOCUMENTS)
print(f"{len(index['chunks'])} chunks from {len(DOCUMENTS)} documents")
print('Vector matrix:', tuple(index['vectors'].shape))
print(f"\nExample chunk (from '{index['sources'][0]}'):")
print(' ', index['chunks'][0][:180], '...')

def chunk_text(text, chunk_words=60, overlap=15): words = text.split() step = max(chunk_words - overlap, 1) chunks = [] for start in range(0, max(len(words) - overlap, 1), step): chunk = words[start:start + chunk_words] if chunk: chunks.append(' '.join(chunk)) if start + chunk_words >= len(words): break return chunks def build_index(documents, chunk_words=60, overlap=15): """Split every document into chunks, embed them, and keep the source title.""" chunks, sources = [], [] for title, body in documents: for chunk in chunk_text(body, chunk_words, overlap): chunks.append(chunk) sources.append(title) return {'chunks': chunks, 'sources': sources, 'vectors': embed(chunks)} index = build_index(DOCUMENTS) print(f"{len(index['chunks'])} chunks from {len(DOCUMENTS)} documents") print('Vector matrix:', tuple(index['vectors'].shape)) print(f"\nExample chunk (from '{index['sources'][0]}'):") print(' ', index['chunks'][0][:180], '...')

26 chunks from 14 documents
Vector matrix: (26, 384)

Example chunk (from 'Lab 06 - Neurons and Activation Functions'):
  A neuron computes a weighted sum of its inputs, adds a bias, and applies an activation function. Without a non-linear activation, stacking layers would collapse into a single linea ...

In [5]:

def retrieve(question, index, k=3):
    """Return the k best-matching chunks, with their source titles and scores."""
    q = embed(question)
    scores = (index['vectors'] @ q.T).squeeze(1)
    top = scores.topk(min(k, len(index['chunks'])))
    return [{'chunk': index['chunks'][i], 'source': index['sources'][i], 'score': s.item()}
            for s, i in zip(top.values, top.indices)]

QUESTION = 'why is the attention score divided by the square root of the key dimension?'
for hit in retrieve(QUESTION, index, k=3):
    print(f"{hit['score']:.3f}  [{hit['source']}]")
    print('        ', hit['chunk'][:120], '...')

def retrieve(question, index, k=3): """Return the k best-matching chunks, with their source titles and scores.""" q = embed(question) scores = (index['vectors'] @ q.T).squeeze(1) top = scores.topk(min(k, len(index['chunks']))) return [{'chunk': index['chunks'][i], 'source': index['sources'][i], 'score': s.item()} for s, i in zip(top.values, top.indices)] QUESTION = 'why is the attention score divided by the square root of the key dimension?' for hit in retrieve(QUESTION, index, k=3): print(f"{hit['score']:.3f} [{hit['source']}]") print(' ', hit['chunk'][:120], '...')

0.758  [Lab 14 - Attention and Transformers]
         Attention compares a query vector against every key vector, converts the scores into weights with a softmax, and returns ...
0.494  [Lab 14 - Attention and Transformers]
         grow with dimension and push the softmax into a saturated regime with almost no gradient. Multi-head attention runs seve ...
0.375  [Lab 18 - Embeddings and Semantic Search]
         an entire search becomes one matrix multiplication. Retrieval quality is measured with recall@k and mean reciprocal rank ...

4. The Generator¶

For generation we use FLAN-T5, an instruction-tuned encoder-decoder model. It is small enough to run on CPU and it follows instructions like "answer using only the context below", which is exactly the behaviour RAG depends on.

The download is roughly 1 GB the first time. If that is too much, google/flan-t5-small works with the same code — its answers are noticeably worse, which is itself informative.

In [6]:

from transformers import AutoTokenizer as GenTokenizer, AutoModelForSeq2SeqLM

GEN_MODEL = 'google/flan-t5-base'          # or 'google/flan-t5-small' for a 300 MB download
gen_tokenizer = GenTokenizer.from_pretrained(GEN_MODEL)
generator = AutoModelForSeq2SeqLM.from_pretrained(GEN_MODEL).to(device)
generator.eval()
print(f'Generator loaded: {sum(p.numel() for p in generator.parameters()):,} parameters')

@torch.inference_mode()
def generate(prompt, max_new_tokens=96, min_new_tokens=0):
    tokens = gen_tokenizer(prompt, return_tensors='pt', truncation=True,
                           max_length=512).to(device)
    output = generator.generate(**tokens, max_new_tokens=max_new_tokens,
                                min_new_tokens=min_new_tokens,
                                num_beams=4, early_stopping=True)
    return gen_tokenizer.decode(output[0], skip_special_tokens=True)

# The same well-known fact, asked three ways, with no context supplied
print('\nClosed-book, three phrasings of one question:')
for phrasing in ['What is the capital of France?',
                 'Q: What is the capital of France? A:',
                 'Answer the question.\n\nQuestion: What is the capital of France?\nAnswer:']:
    label = phrasing.splitlines()[0][:44]
    print(f'  {generate(phrasing)!r:12s}  <- {label!r}')

from transformers import AutoTokenizer as GenTokenizer, AutoModelForSeq2SeqLM GEN_MODEL = 'google/flan-t5-base' # or 'google/flan-t5-small' for a 300 MB download gen_tokenizer = GenTokenizer.from_pretrained(GEN_MODEL) generator = AutoModelForSeq2SeqLM.from_pretrained(GEN_MODEL).to(device) generator.eval() print(f'Generator loaded: {sum(p.numel() for p in generator.parameters()):,} parameters') @torch.inference_mode() def generate(prompt, max_new_tokens=96, min_new_tokens=0): tokens = gen_tokenizer(prompt, return_tensors='pt', truncation=True, max_length=512).to(device) output = generator.generate(**tokens, max_new_tokens=max_new_tokens, min_new_tokens=min_new_tokens, num_beams=4, early_stopping=True) return gen_tokenizer.decode(output[0], skip_special_tokens=True) # The same well-known fact, asked three ways, with no context supplied print('\nClosed-book, three phrasings of one question:') for phrasing in ['What is the capital of France?', 'Q: What is the capital of France? A:', 'Answer the question.\n\nQuestion: What is the capital of France?\nAnswer:']: label = phrasing.splitlines()[0][:44] print(f' {generate(phrasing)!r:12s} <- {label!r}')

[transformers] The tied weights mapping and config for this model specifies to tie shared.weight to lm_head.weight, but both are present in the checkpoints with different values, so we will NOT tie them. You should update the config with `tie_word_embeddings=False` to silence this warning.

Generator loaded: 247,577,856 parameters

Closed-book, three phrasings of one question:

  'london'      <- 'What is the capital of France?'

  'Rouen'       <- 'Q: What is the capital of France? A:'

  'paris'       <- 'Answer the question.'

5. Closed-Book vs Grounded¶

The moment of truth. First ask the model directly, with no context — the "closed-book" setting. Then ask it again with the retrieved passages in the prompt.

In [7]:

CLOSED_BOOK = 'Answer the question.\n\nQuestion: {question}\nAnswer:'

# "Give a complete, self-contained explanation" earns its place: without it this model
# answers in fragments lifted straight out of the context ("without that scaling").
RAG_PROMPT = (
    'You are answering from the context below. Give a complete, self-contained '
    'explanation.\n\n'
    'Context:\n{context}\n\n'
    'Question: {question}\nAnswer:'
)

def format_context(hits):
    return '\n\n'.join(f"[{i + 1}] {hit['chunk']}" for i, hit in enumerate(hits))

def ask(question, index, k=3, prompt_template=RAG_PROMPT, show_prompt=False):
    hits = retrieve(question, index, k=k)
    prompt = prompt_template.format(context=format_context(hits), question=question)
    if show_prompt:
        print('-' * 78)
        print(prompt)
        print('-' * 78)
    return generate(prompt), hits

CLOSED_BOOK = 'Answer the question.\n\nQuestion: {question}\nAnswer:' # "Give a complete, self-contained explanation" earns its place: without it this model # answers in fragments lifted straight out of the context ("without that scaling"). RAG_PROMPT = ( 'You are answering from the context below. Give a complete, self-contained ' 'explanation.\n\n' 'Context:\n{context}\n\n' 'Question: {question}\nAnswer:' ) def format_context(hits): return '\n\n'.join(f"[{i + 1}] {hit['chunk']}" for i, hit in enumerate(hits)) def ask(question, index, k=3, prompt_template=RAG_PROMPT, show_prompt=False): hits = retrieve(question, index, k=k) prompt = prompt_template.format(context=format_context(hits), question=question) if show_prompt: print('-' * 78) print(prompt) print('-' * 78) return generate(prompt), hits

In [8]:

QUESTIONS = [
    'why is the attention score divided by the square root of the key dimension?',
    'in what order does PyTorch pack the LSTM gate weights?',
    'what should I use instead of pickling a whole model?',
    'which regularizer is usually the most effective for images?',
    'what does the reparameterization trick do?',
]

for question in QUESTIONS:
    closed = generate(CLOSED_BOOK.format(question=question))
    grounded, hits = ask(question, index, k=3)
    print(f'Q: {question}')
    print(f'   closed book : {closed}')
    print(f'   with RAG    : {grounded}')
    print(f'   retrieved   : {hits[0]["source"]} ({hits[0]["score"]:.2f})\n')

QUESTIONS = [ 'why is the attention score divided by the square root of the key dimension?', 'in what order does PyTorch pack the LSTM gate weights?', 'what should I use instead of pickling a whole model?', 'which regularizer is usually the most effective for images?', 'what does the reparameterization trick do?', ] for question in QUESTIONS: closed = generate(CLOSED_BOOK.format(question=question)) grounded, hits = ask(question, index, k=3) print(f'Q: {question}') print(f' closed book : {closed}') print(f' with RAG : {grounded}') print(f' retrieved : {hits[0]["source"]} ({hits[0]["score"]:.2f})\n')

Q: why is the attention score divided by the square root of the key dimension?
   closed book : the square root of the key dimension
   with RAG    : Without that scaling the dot products grow with dimension and push the softmax into a saturated regime with almost no gradient.
   retrieved   : Lab 14 - Attention and Transformers (0.76)

Q: in what order does PyTorch pack the LSTM gate weights?
   closed book : first and second
   with RAG    : input, forget, candidate, output
   retrieved   : Lab 11 - Long Short-Term Memory (0.64)

Q: what should I use instead of pickling a whole model?
   closed book : a tin can
   with RAG    : torch.save(model.state_dict(), path)
   retrieved   : Lab 17 - Deployment and Model Optimization (0.37)

Q: which regularizer is usually the most effective for images?
   closed book : Adobe Photoshop
   with RAG    : Data augmentation
   retrieved   : Lab 13 - Regularization and Optimization (0.73)

Q: what does the reparameterization trick do?
   closed book : reduces the amount of time it takes to repair a broken circuit
   with RAG    : moves the randomness into an input so gradients can flow through it.
   retrieved   : Lab 15 - Autoencoders and VAEs (0.52)

The closed-book answers are vague, generic, or simply wrong — the model is guessing from general knowledge. With the retrieved context, the answers become specific and correct, because the facts were in the prompt.

Nothing about the model changed between those two columns. Only the prompt did.

In [9]:

# What the model actually receives
_ = ask('in what order does PyTorch pack the LSTM gate weights?', index, k=2, show_prompt=True)

# What the model actually receives _ = ask('in what order does PyTorch pack the LSTM gate weights?', index, k=2, show_prompt=True)

------------------------------------------------------------------------------
You are answering from the context below. Give a complete, self-contained explanation.

Context:
[1] decides what to write, and the output gate decides what to expose. In PyTorch the four gate weight blocks are packed in the order input, forget, candidate, output. A GRU is a lighter variant with two gates and about 25 percent fewer parameters.

[2] An LSTM adds a cell state that runs through the whole sequence touched only by an elementwise multiplication and an addition, so information and gradients can travel a long way unchanged. Three gates control it: the forget gate decides what to erase, the input gate decides what to write, and the output gate decides what to expose. In PyTorch the

Question: in what order does PyTorch pack the LSTM gate weights?
Answer:
------------------------------------------------------------------------------

6. Making It Say "I Don't Know"¶

Lab 18 ended on a warning: a similarity search always returns its nearest neighbour, no matter how irrelevant. Feed that irrelevant chunk to a generator and you get a fluent answer built on unrelated text — a hallucination caused by retrieval.

Two defences, and you want both:

1. Instruct the model that "not in the context" is an acceptable answer.
2. Threshold the retrieval score, and do not call the generator at all when nothing scores well enough.

The second is the reliable one. Instructions help; a threshold decides.

In [10]:

STRICT_PROMPT = (
    'You are answering from the context below. Give a complete, self-contained '
    'explanation. If the context does not contain the answer, reply exactly: '
    'I do not know.\n\n'
    'Context:\n{context}\n\n'
    'Question: {question}\nAnswer:'
)

OFF_TOPIC = [
    'what is the boiling point of water?',
    'who won the world cup in 1998?',
    'how do I make a good espresso?',
]

print(f'{"score":>6}  {"lenient prompt":<32} {"strict prompt":<28}')
print('-' * 78)
for question in OFF_TOPIC:
    hits = retrieve(question, index, k=3)
    lenient = generate(RAG_PROMPT.format(context=format_context(hits), question=question))
    strict = generate(STRICT_PROMPT.format(context=format_context(hits), question=question))
    print(f'{hits[0]["score"]:>6.3f}  {lenient[:30]:<32} {strict[:26]:<28}')
    print(f'        (best match: {hits[0]["source"]})')

STRICT_PROMPT = ( 'You are answering from the context below. Give a complete, self-contained ' 'explanation. If the context does not contain the answer, reply exactly: ' 'I do not know.\n\n' 'Context:\n{context}\n\n' 'Question: {question}\nAnswer:' ) OFF_TOPIC = [ 'what is the boiling point of water?', 'who won the world cup in 1998?', 'how do I make a good espresso?', ] print(f'{"score":>6} {"lenient prompt":<32} {"strict prompt":<28}') print('-' * 78) for question in OFF_TOPIC: hits = retrieve(question, index, k=3) lenient = generate(RAG_PROMPT.format(context=format_context(hits), question=question)) strict = generate(STRICT_PROMPT.format(context=format_context(hits), question=question)) print(f'{hits[0]["score"]:>6.3f} {lenient[:30]:<32} {strict[:26]:<28}') print(f' (best match: {hits[0]["source"]})')

 score  lenient prompt                   strict prompt               
------------------------------------------------------------------------------

 0.116  The boiling point of water is    I do not know.              
        (best match: Lab 11 - Long Short-Term Memory)

 0.053  the United States                I do not know               
        (best match: Repository layout)

 0.105  a log-variance instead of a po   I do not know.              
        (best match: Lab 15 - Autoencoders and VAEs)

A trap: forcing the model to talk¶

Small models answer tersely, and the usual reflex is to force longer output with min_new_tokens. Watch what that does to the refusal.

The instruction says "reply exactly: I do not know" — four tokens. A minimum length of 16 makes that reply impossible to produce, so the decoder must find something else to say, and what it finds is an invention.

In [11]:

off_topic_hits = retrieve('who won the world cup in 1998?', index, k=3)
prompt = STRICT_PROMPT.format(context=format_context(off_topic_hits),
                              question='who won the world cup in 1998?')

print('min_new_tokens=0  ->', generate(prompt))
print('min_new_tokens=16 ->', generate(prompt, min_new_tokens=16))

off_topic_hits = retrieve('who won the world cup in 1998?', index, k=3) prompt = STRICT_PROMPT.format(context=format_context(off_topic_hits), question='who won the world cup in 1998?') print('min_new_tokens=0 ->', generate(prompt)) print('min_new_tokens=16 ->', generate(prompt, min_new_tokens=16))

min_new_tokens=0  -> I do not know

min_new_tokens=16 -> The 1998 World Cup was the fourth edition of the FIFA World Cup, which was held in Paris, France.

Nothing about the prompt, the model or the retrieved context changed between those two lines. A generation setting that looks purely cosmetic silently overrode a safety instruction and produced a confident fabrication.

This is worth generalising: any constraint that makes the safe answer unrepresentable will produce an unsafe one. Minimum lengths, output schemas that have no "unknown" field, a JSON format with a required answer key — each removes the model's ability to decline, and none of them look like a safety decision at the time you make them.

In [12]:

THRESHOLD = 0.35     # tuned on your own eval set - this value is corpus-specific

def ask_safely(question, index, k=3, threshold=THRESHOLD):
    hits = retrieve(question, index, k=k)
    if hits[0]['score'] < threshold:
        return ('I could not find anything relevant in the knowledge base.', hits)
    prompt = STRICT_PROMPT.format(context=format_context(hits), question=question)
    return generate(prompt), hits

for question in QUESTIONS[:2] + OFF_TOPIC[:2]:
    answer, hits = ask_safely(question, index)
    status = 'answered' if hits[0]['score'] >= THRESHOLD else 'REFUSED  '
    print(f'[{status}] {hits[0]["score"]:.3f}  {question}')
    print(f'            {answer}\n')

THRESHOLD = 0.35 # tuned on your own eval set - this value is corpus-specific def ask_safely(question, index, k=3, threshold=THRESHOLD): hits = retrieve(question, index, k=k) if hits[0]['score'] < threshold: return ('I could not find anything relevant in the knowledge base.', hits) prompt = STRICT_PROMPT.format(context=format_context(hits), question=question) return generate(prompt), hits for question in QUESTIONS[:2] + OFF_TOPIC[:2]: answer, hits = ask_safely(question, index) status = 'answered' if hits[0]['score'] >= THRESHOLD else 'REFUSED ' print(f'[{status}] {hits[0]["score"]:.3f} {question}') print(f' {answer}\n')

[answered] 0.758  why is the attention score divided by the square root of the key dimension?
            Without that scaling the dot products grow with dimension and push the softmax into a saturated regime with almost no gradient.

[answered] 0.635  in what order does PyTorch pack the LSTM gate weights?
            input, forget, candidate, output

[REFUSED  ] 0.116  what is the boiling point of water?
            I could not find anything relevant in the knowledge base.

[REFUSED  ] 0.053  who won the world cup in 1998?
            I could not find anything relevant in the knowledge base.

Choosing the threshold. There is no universal value — it depends on the encoder and the corpus. Collect questions you can answer and questions you cannot, plot the two score distributions, and pick the point that separates them. That is a twenty-minute job that prevents the most embarrassing class of RAG failure.

In [13]:

answerable = [retrieve(q, index, k=1)[0]['score'] for q in QUESTIONS]
unanswerable = [retrieve(q, index, k=1)[0]['score'] for q in OFF_TOPIC]

fig, ax = plt.subplots(figsize=(8, 3.4))
ax.scatter(answerable, [1] * len(answerable), s=90, c='#55A868', label='in the corpus')
ax.scatter(unanswerable, [1] * len(unanswerable), s=90, c='#C44E52', label='not in the corpus')
ax.axvline(THRESHOLD, color='gray', linestyle='--', label=f'threshold = {THRESHOLD}')
ax.set_yticks([])
ax.set_xlabel('top retrieval score')
ax.set_title('Where to draw the line')
ax.legend(fontsize=9)
plt.tight_layout()
plt.show()

answerable = [retrieve(q, index, k=1)[0]['score'] for q in QUESTIONS] unanswerable = [retrieve(q, index, k=1)[0]['score'] for q in OFF_TOPIC] fig, ax = plt.subplots(figsize=(8, 3.4)) ax.scatter(answerable, [1] * len(answerable), s=90, c='#55A868', label='in the corpus') ax.scatter(unanswerable, [1] * len(unanswerable), s=90, c='#C44E52', label='not in the corpus') ax.axvline(THRESHOLD, color='gray', linestyle='--', label=f'threshold = {THRESHOLD}') ax.set_yticks([]) ax.set_xlabel('top retrieval score') ax.set_title('Where to draw the line') ax.legend(fontsize=9) plt.tight_layout() plt.show()

7. Citations¶

This is the feature that makes RAG worth the trouble in any serious setting. Because we know exactly which chunks went into the prompt, every answer can point at its sources — and a reader can check them.

An answer nobody can verify is worth much less than one they can.

In [14]:

def answer_with_citations(question, index, k=3, threshold=THRESHOLD, width=76):
    hits = retrieve(question, index, k=k)
    print(f'Q: {question}\n')
    if hits[0]['score'] < threshold:
        print('A: I could not find anything relevant in the knowledge base.')
        print(f"   (best match scored {hits[0]['score']:.3f}, below the {threshold} threshold)")
        return
    answer = generate(STRICT_PROMPT.format(context=format_context(hits), question=question))
    print('A:', textwrap.fill(answer, width))
    print('\n   Sources:')
    for i, hit in enumerate(hits, start=1):
        print(f"   [{i}] {hit['source']}  (score {hit['score']:.3f})")
        print(textwrap.fill(hit['chunk'][:150] + '...', width,
                            initial_indent='       ', subsequent_indent='       '))

answer_with_citations('how do I keep a pretrained backbone from being updated?', index)
print('\n' + '=' * 78 + '\n')
answer_with_citations('what is mode collapse?', index)

def answer_with_citations(question, index, k=3, threshold=THRESHOLD, width=76): hits = retrieve(question, index, k=k) print(f'Q: {question}\n') if hits[0]['score'] < threshold: print('A: I could not find anything relevant in the knowledge base.') print(f" (best match scored {hits[0]['score']:.3f}, below the {threshold} threshold)") return answer = generate(STRICT_PROMPT.format(context=format_context(hits), question=question)) print('A:', textwrap.fill(answer, width)) print('\n Sources:') for i, hit in enumerate(hits, start=1): print(f" [{i}] {hit['source']} (score {hit['score']:.3f})") print(textwrap.fill(hit['chunk'][:150] + '...', width, initial_indent=' ', subsequent_indent=' ')) answer_with_citations('how do I keep a pretrained backbone from being updated?', index) print('\n' + '=' * 78 + '\n') answer_with_citations('what is mode collapse?', index)

Q: how do I keep a pretrained backbone from being updated?

A: Always train the head first, or large early gradients will damage the
pretrained weights.

   Sources:
   [1] Lab 12 - Transfer Learning  (score 0.593)
       the backbone with requires_grad = False and trains just the new head.
       Fine-tuning then unfreezes the last block at a learning rate about
       ten times sma...
   [2] Lab 12 - Transfer Learning  (score 0.349)
       Transfer learning reuses a network trained on a large dataset such as
       ImageNet and replaces only its final classification layer. Early
       layers learn ed...
   [3] Lab 17 - Deployment and Model Optimization  (score 0.269)
       Save a model with torch.save(model.state_dict(), path) rather than
       pickling the whole object, because pickle stores a reference to your
       class and brea...

==============================================================================

Q: what is mode collapse?

A: I could not find anything relevant in the knowledge base.
   (best match scored 0.307, below the 0.35 threshold)

8. The Knobs That Matter¶

Two hyperparameters do most of the work, and both have a genuine optimum rather than "more is better".

top-k. Retrieve too few chunks and the answer may not be among them. Retrieve too many and the real answer is buried in irrelevant text — models demonstrably get worse at using context as it grows, and every extra chunk costs tokens.

Chunk size. Small chunks retrieve precisely but may cut an answer in half. Large chunks carry full context but produce vague, diluted vectors.

Rather than guessing, measure — but measure the right thing. The obvious metric is "did we retrieve the correct document", and it is too generous: a document counts as a hit even when the sentence carrying the answer was split down the middle by chunking. So we check something stricter: does a retrieved chunk actually contain the answer?

In [15]:

# Question -> a phrase that MUST appear in a retrieved chunk for the answer to be present.
# Checking the document title is too generous: a whole document always "matches", even
# when the specific sentence carrying the answer was chopped in half.
EVAL_SET = [
    ('why is the attention score divided by the square root of the key dimension?',
     'saturated regime'),
    ('in what order does PyTorch pack the LSTM gate weights?',
     'input, forget, candidate, output'),
    ('what should I use instead of pickling a whole model?',
     'state_dict'),
    ('which regularizer is usually the most effective for images?',
     'most effective regularizer'),
    ('what does the reparameterization trick do?',
     'moving the randomness'),
    ('how do I freeze a pretrained backbone?',
     'requires_grad = False'),
    ('why does a vanilla RNN forget?',
     'vanishing gradient'),
    ('how do I flatten feature maps before the classifier?',
     'x.view(x.size(0), -1)'),
    ('what makes the dot product equal to cosine similarity?',
     'unit length'),
    ('how do I create the conda environment?',
     'conda create -n envNeural'),
]

def answer_recall_at_k(index, eval_set, k):
    """Fraction of questions where one of the top k chunks actually contains the answer."""
    hits = 0
    for question, phrase in eval_set:
        retrieved = retrieve(question, index, k=k)
        hits += any(phrase.lower() in hit['chunk'].lower() for hit in retrieved)
    return hits / len(eval_set)

chunk_settings = [10, 20, 40, 60, 100, 200]
curves = {k: [] for k in (1, 3, 5)}

for size in chunk_settings:
    trial = build_index(DOCUMENTS, chunk_words=size, overlap=max(size // 4, 2))
    for k in curves:
        curves[k].append(answer_recall_at_k(trial, EVAL_SET, k))
    print(f'chunk_words={size:3d}  ({len(trial["chunks"]):3d} chunks)  '
          + '  '.join(f'answer recall@{k}={curves[k][-1]:.2f}' for k in curves))

# Question -> a phrase that MUST appear in a retrieved chunk for the answer to be present. # Checking the document title is too generous: a whole document always "matches", even # when the specific sentence carrying the answer was chopped in half. EVAL_SET = [ ('why is the attention score divided by the square root of the key dimension?', 'saturated regime'), ('in what order does PyTorch pack the LSTM gate weights?', 'input, forget, candidate, output'), ('what should I use instead of pickling a whole model?', 'state_dict'), ('which regularizer is usually the most effective for images?', 'most effective regularizer'), ('what does the reparameterization trick do?', 'moving the randomness'), ('how do I freeze a pretrained backbone?', 'requires_grad = False'), ('why does a vanilla RNN forget?', 'vanishing gradient'), ('how do I flatten feature maps before the classifier?', 'x.view(x.size(0), -1)'), ('what makes the dot product equal to cosine similarity?', 'unit length'), ('how do I create the conda environment?', 'conda create -n envNeural'), ] def answer_recall_at_k(index, eval_set, k): """Fraction of questions where one of the top k chunks actually contains the answer.""" hits = 0 for question, phrase in eval_set: retrieved = retrieve(question, index, k=k) hits += any(phrase.lower() in hit['chunk'].lower() for hit in retrieved) return hits / len(eval_set) chunk_settings = [10, 20, 40, 60, 100, 200] curves = {k: [] for k in (1, 3, 5)} for size in chunk_settings: trial = build_index(DOCUMENTS, chunk_words=size, overlap=max(size // 4, 2)) for k in curves: curves[k].append(answer_recall_at_k(trial, EVAL_SET, k)) print(f'chunk_words={size:3d} ({len(trial["chunks"]):3d} chunks) ' + ' '.join(f'answer recall@{k}={curves[k][-1]:.2f}' for k in curves))

chunk_words= 10  (137 chunks)  answer recall@1=0.20  answer recall@3=0.40  answer recall@5=0.50

chunk_words= 20  ( 74 chunks)  answer recall@1=0.80  answer recall@3=0.90  answer recall@5=0.90

chunk_words= 40  ( 38 chunks)  answer recall@1=0.70  answer recall@3=0.90  answer recall@5=0.90

chunk_words= 60  ( 26 chunks)  answer recall@1=1.00  answer recall@3=1.00  answer recall@5=1.00

chunk_words=100  ( 14 chunks)  answer recall@1=1.00  answer recall@3=1.00  answer recall@5=1.00

chunk_words=200  ( 14 chunks)  answer recall@1=1.00  answer recall@3=1.00  answer recall@5=1.00

In [16]:

fig, ax = plt.subplots(figsize=(7.5, 4.2))
for k, values in curves.items():
    ax.plot(chunk_settings, values, marker='o', label=f'answer recall@{k}')
ax.set_xlabel('chunk size (words)')
ax.set_ylabel('fraction of questions whose answer was retrieved')
ax.set_ylim(0, 1.05)
ax.set_title('Chunk size is a tunable, not a default')
ax.legend()
ax.grid(alpha=0.3)
plt.tight_layout()
plt.show()

best = chunk_settings[int(np.argmax(curves[3]))]
print(f'Best chunk size for recall@3 on this corpus: {best} words')
print('Expect a different answer on your own data. That is the point of measuring.')

fig, ax = plt.subplots(figsize=(7.5, 4.2)) for k, values in curves.items(): ax.plot(chunk_settings, values, marker='o', label=f'answer recall@{k}') ax.set_xlabel('chunk size (words)') ax.set_ylabel('fraction of questions whose answer was retrieved') ax.set_ylim(0, 1.05) ax.set_title('Chunk size is a tunable, not a default') ax.legend() ax.grid(alpha=0.3) plt.tight_layout() plt.show() best = chunk_settings[int(np.argmax(curves[3]))] print(f'Best chunk size for recall@3 on this corpus: {best} words') print('Expect a different answer on your own data. That is the point of measuring.')

Best chunk size for recall@3 on this corpus: 60 words
Expect a different answer on your own data. That is the point of measuring.

Watching the pipeline run¶

The animation below runs the whole pipeline over each evaluation question: the bars are the similarity scores of every chunk in the index, the highlighted ones are what got retrieved, and the title is the answer the model produced from them.

In [17]:

from matplotlib import animation
import os

os.makedirs('Misc', exist_ok=True)

def save_animation(ani, filename, fps=8, dpi=80):
    """Render an animation to a GIF under Misc/ and report where it went."""
    path = os.path.join('Misc', filename)
    ani.save(path, writer='pillow', fps=fps, dpi=dpi)
    print(f'saved {path}  ({os.path.getsize(path) / 1e6:.1f} MB)')
    return path

TOP_K = 3
frames_data = []
for question, gold in EVAL_SET[:6]:
    q = embed(question)
    scores = (index['vectors'] @ q.T).squeeze(1).numpy()
    chosen = scores.argsort()[::-1][:TOP_K]
    answer, _ = ask_safely(question, index, k=TOP_K)
    frames_data.append((question, scores, chosen, answer))
    print(f'{question[:52]:54s} -> {answer[:44]}')

from matplotlib import animation import os os.makedirs('Misc', exist_ok=True) def save_animation(ani, filename, fps=8, dpi=80): """Render an animation to a GIF under Misc/ and report where it went.""" path = os.path.join('Misc', filename) ani.save(path, writer='pillow', fps=fps, dpi=dpi) print(f'saved {path} ({os.path.getsize(path) / 1e6:.1f} MB)') return path TOP_K = 3 frames_data = [] for question, gold in EVAL_SET[:6]: q = embed(question) scores = (index['vectors'] @ q.T).squeeze(1).numpy() chosen = scores.argsort()[::-1][:TOP_K] answer, _ = ask_safely(question, index, k=TOP_K) frames_data.append((question, scores, chosen, answer)) print(f'{question[:52]:54s} -> {answer[:44]}')

why is the attention score divided by the square roo   -> Without that scaling the dot products grow w

in what order does PyTorch pack the LSTM gate weight   -> input, forget, candidate, output

what should I use instead of pickling a whole model?   -> torch.save(model.state_dict(), path)

which regularizer is usually the most effective for    -> Data augmentation is usually the most effect

what does the reparameterization trick do?             -> moves the randomness into an input so gradie

how do I freeze a pretrained backbone?                 -> Fine-tuning the last block at a learning rat

In [18]:

n_chunks = len(index['chunks'])
fig, ax = plt.subplots(figsize=(10, 4.2))
bars = ax.bar(range(n_chunks), np.zeros(n_chunks), color='#c9c9c9')
ax.set_ylim(0, 1)
ax.set_xlabel('chunk in the index')
ax.set_ylabel('cosine similarity')
ax.axhline(THRESHOLD, color='gray', linestyle=':', lw=1)

def update(k):
    question, scores, chosen, answer = frames_data[k]
    for i, (bar, score) in enumerate(zip(bars, scores)):
        bar.set_height(score)
        bar.set_color('#DD8452' if i in chosen else '#c9c9c9')
    ax.set_title('\n'.join(textwrap.wrap(f'Q: {question}', 92)
                           + textwrap.wrap(f'A: {answer}', 92)),
                 fontsize=10, loc='left')
    return list(bars)

fig.tight_layout()
ani = animation.FuncAnimation(fig, update, frames=len(frames_data), interval=2200, blit=False)
save_animation(ani, 'lab19_rag.gif', fps=1, dpi=90)
plt.close(fig)

n_chunks = len(index['chunks']) fig, ax = plt.subplots(figsize=(10, 4.2)) bars = ax.bar(range(n_chunks), np.zeros(n_chunks), color='#c9c9c9') ax.set_ylim(0, 1) ax.set_xlabel('chunk in the index') ax.set_ylabel('cosine similarity') ax.axhline(THRESHOLD, color='gray', linestyle=':', lw=1) def update(k): question, scores, chosen, answer = frames_data[k] for i, (bar, score) in enumerate(zip(bars, scores)): bar.set_height(score) bar.set_color('#DD8452' if i in chosen else '#c9c9c9') ax.set_title('\n'.join(textwrap.wrap(f'Q: {question}', 92) + textwrap.wrap(f'A: {answer}', 92)), fontsize=10, loc='left') return list(bars) fig.tight_layout() ani = animation.FuncAnimation(fig, update, frames=len(frames_data), interval=2200, blit=False) save_animation(ani, 'lab19_rag.gif', fps=1, dpi=90) plt.close(fig)

saved Misc\lab19_rag.gif  (0.1 MB)

One frame per question: similarity against every chunk, the top-3 highlighted, and the answer generated from them.

9. Where RAG Goes Wrong¶

Almost every RAG failure is a retrieval failure wearing a generation costume. If the right passage never reaches the prompt, no amount of model quality will save the answer — so debug the retriever first, always.

Symptom	Usual cause	Fix
Confident but wrong answer	An irrelevant chunk was retrieved and used	Threshold the score; instruct the model to refuse
Answer is half right	The chunk was cut mid-explanation	Larger chunks, or more overlap
Right document, wrong detail	Too many chunks diluted the prompt	Lower top-k; add a reranker
Fails on exact codes or names	Embeddings blur rare tokens	Hybrid search: combine keyword and vector scores
Was correct last month	The index is stale	Re-index on document change
Slow at scale	Brute-force search	An ANN index (FAISS, HNSW)

Beyond the basics, in roughly the order they are worth adding:

1. Reranking. Retrieve 20 candidates with the fast bi-encoder, then rescore them with a cross-encoder that reads the query and passage together. Much more accurate, too slow to run over the whole corpus — which is exactly why it goes second.
2. Query rewriting. Expand or rephrase the question before searching, especially in a conversation where "what about the second one?" means nothing on its own.
3. Metadata filters. Restrict by date, author or permission before the vector search. Access control belongs here, not in the prompt.
4. An evaluation set. Thirty real questions with known answers will tell you more than any amount of prompt tinkering.

10. Experiment: Try Different Settings¶

You can experiment with:

• Sweeping k from 1 to 8 and watching answer quality rise and then fall
• Switching GEN_MODEL to google/flan-t5-small — the retrieval is unchanged, so every difference is the generator
• Removing the "reply exactly: I do not know" instruction and re-running the off-topic questions
• Replacing DOCUMENTS with your own notes, a README, or documentation you actually use
• Adding a cross-encoder reranker with cross-encoder/ms-marco-MiniLM-L-6-v2
• Implementing hybrid retrieval: normalise the keyword and embedding scores and blend them
• Deliberately putting a contradiction in two documents and seeing which one the model believes
• Measuring end-to-end accuracy, not just recall: does a retrieved-correct question actually get answered correctly?

Try modifying the code above and observe how the answers change.

11. Conclusion¶

In this lab, you built a complete RAG system from parts you had already made: the Transformer encoder from Lab 14, the semantic index from Lab 18, and a pretrained generator. No training was involved anywhere.

Key takeaways:

• Fine-tune to change how a model behaves; retrieve to change what it knows.
• The generator is only as good as the passages it is given — debug retrieval first.
• A vector search never says "not found", so threshold the score yourself.
• Instructions to refuse help; a retrieval threshold decides.
• Citations are the main reason to prefer RAG over fine-tuning for factual work.
• Chunk size and top-k are tunables with a real optimum. Measure them on your own data.

That closes the series. You started with a dot product and a derivative, and you have now built, trained, regularised, inspected, generated with, deployed and retrieved from neural networks — every mechanism implemented by hand at least once before reaching for the library version.

Go find a dataset you care about.