Retrieval-Augmented Generation (RAG)

RAG is a way to give a language model knowledge beyond what it saw during training — without needing to retrain the model.

What Is RAG?

Retrieval-Augmented Generation means exactly that: instead of relying only on the model's internal knowledge, we first retrieve documents relevant to the user's question from an external knowledge base, then give them to the model as added context alongside the original question so it can generate an answer grounded in them.

Why RAG?

• Reduced hallucination — the answer is grounded in a real document, not just the model's parametric memory
• Up-to-date knowledge — no re-training needed; just add the new document to the knowledge base
• Proprietary knowledge — a general-purpose model can answer about your organization's private data (which it never saw during general training)
• Citable — you can show the exact source of every answer

The RAG Pipeline

Chunking & Embedding

Large documents are first broken into smaller pieces (chunks) so each piece is focused enough and fits within the context window. Each chunk is then converted by an embedding model into a numeric vector (a semantic representation); texts with close meaning have vectors that are close together.

Vector Databases

These vectors are stored in a vector database optimized for nearest-neighbor search. Once the user's query is also converted to a vector, the database can find the closest matches among millions of documents in a few milliseconds.

Hybrid Search & Re-ranking

Vector search alone isn't always the best choice: for exact terms like a model name or a product code (e.g. SKU-4471), traditional keyword search (BM25) often performs more precisely than semantic search, because vectors are optimized for "conceptual similarity," not exact matching.

• Hybrid Search — results from both vector search and keyword search are combined with a weighting formula; this combination usually improves accuracy over either alone.
• Re-ranking — an optional second stage: after an initial retrieval of N results (e.g. 50 documents), a smaller, more precise model (a Cross-Encoder) re-scores these results so only the top K (e.g. 4 documents) are actually given to the main language model. This step is slower than vector search, which is why it only runs on the reduced set, not the whole database.

Code Sample: A Simple Pipeline

def answer_with_rag(question, vector_db, llm):
    query_vector = embed(question)
    top_chunks = vector_db.similarity_search(query_vector, k=4)

    context = "\n\n".join(chunk.text for chunk in top_chunks)
    prompt = f"Answer the question using the text below:\n\n{context}\n\nQuestion: {question}"

    return llm.generate(prompt)

Limitations

• Answer quality depends entirely on retrieval quality; if the relevant document isn't found, the model either guesses or gets it wrong
• Simple RAG only performs one search per turn; it's not enough for multi-step questions — which is why Agentic RAG came about

FAQ