AI & Machine Learning7 min read

Retrieval-Augmented Generation (RAG)

Grounds LLM responses in your documents by retrieving relevant chunks at query time - the standard fix for hallucination on private or recent information without retraining

LLMAIVector SearchEmbeddingspgvector
---
name: rag
description: >
  You are a RAG architect. You design retrieval-augmented generation pipelines: chunking, embedding, hybrid search, re-ranking, context assembly, and RAGAS evaluation.
---

RAG grounds LLM responses in your documents by retrieving relevant chunks at query time and injecting them into context - the standard fix for hallucination on private or recent knowledge. Choose RAG over fine-tuning when the knowledge changes frequently or when exact citations matter; choose fine-tuning when you need a behavior or style change, not a knowledge change. A RAG pipeline without an eval set is a hypothesis, not a system.

Workflow

  1. Analyze your corpus with real documents before choosing a chunking strategyFixed-size chunking at 512 tokens destroys semantic units in technical docs; semantic chunking on narrative prose produces tiny fragments - the right strategy depends on your document type and cannot be decided without sampling
  2. Select embedding model based on your latency and quality requirements, then validate on your domaintext-embedding-3-small is 5x cheaper than text-embedding-3-large and matches it on many domains; switching embedding models requires re-indexing the entire corpus, so measuring first saves cost
  3. Implement hybrid search (dense vector similarity + BM25 sparse keyword) and fuse rankings with RRFPure vector search misses exact product names, error codes, and technical identifiers that users search for literally; hybrid search adds BM25 recall with no accuracy regression on semantic queries
  4. Add a cross-encoder re-ranker on the top-k candidates before injecting into contextBi-encoder retrieval is optimized for recall (cast a wide net); cross-encoder re-ranking is optimized for precision (pick the most relevant) - the combination consistently outperforms either alone on NDCG@k
  5. Measure retrieval quality (precision@k, recall@k, MRR) and generation quality (faithfulness, answer relevance) separately with RAGASGeneration failures and retrieval failures have different root causes - a hallucinating LLM needs a better prompt or model; a low-recall retrieval needs better chunking or embedding - you cannot fix one by tuning the other
  6. Run chunking and embedding changes against the RAGAS eval set before re-indexing the full corpusRe-indexing large corpora is expensive (time + API cost); a change that looks better in informal testing can silently reduce precision on rare but important queries

Output Format

A RAG pipeline has two phases with distinct failure modes. Document each phase separately: indexing (chunking strategy, embedding model, index type) and retrieval (top-k, re-ranking, context assembly, prompt template).

Indexing: chunk_strategy=semantic, chunk_size=512, overlap=64, embedding=text-embedding-3-small, index=HNSW
Retrieval: top_k=20, rerank=cross-encoder/ms-marco-MiniLM-L-6-v2, final_k=5
Context: inject top-5 chunks with source metadata into system prompt before user query

Examples

Parent-child chunking for technical documentation

Input

API reference docs where each function has a 3-line signature and a 2-paragraph description

Output

Child chunks: one per function signature (precise retrieval). Parent chunks: function + full description (full context injection). Retrieve children, inject parents - 40% precision improvement over flat 512-token chunks in benchmarks

Wrong: flat fixed-size chunks on a code repository

Input

GitHub repo with Python files chunked at 512 tokens with no respect for function or class boundaries

Output

Functions split mid-body, class methods separated from their class definition - retrieval returns partial context that confuses the LLM and produces hallucinated completions

Right: AST-aware chunking for code

Input

Same Python repo

Output

Each function/class is one chunk with its full docstring; imports are prepended to each chunk. Retrieval returns complete, self-contained code units - LLM can reason about them without missing context

Edge Cases & Gotchas

  • Empty retrieval - query finds no relevant chunks because the question is out of the indexed domain. Fix: detect low similarity scores (cosine < 0.6) and respond with 'I don't have information about this' rather than hallucinating from model weights
  • Context window overflow - retrieving top-20 and re-ranking to top-5 still exceeds the model context when chunks are large. Fix: measure chunk token count during indexing and store it; enforce a hard token budget during context assembly
  • Stale index - source documents updated but embedding index not re-indexed. Fix: store document hash at index time; run a nightly diff against the source and re-index only changed documents
  • Re-ranker latency - cross-encoder adds 200-400ms on CPU for top-20 candidates. Fix: run re-ranker on GPU or use a distilled model (MiniLM-L-6 vs L-12); accept the latency trade-off for precision-critical use cases

When to Break These Rules

  • Skip re-ranking for simple FAQ bots where the corpus is small (<1000 chunks) and top-1 retrieval is almost always correct - the latency cost is not worth the marginal precision gain
  • Use fine-tuning instead of RAG when the knowledge base is stable, small (<500 facts), and the LLM needs to internalize behavior patterns rather than look up facts

Reference Files

  • src/data/skills/index.js - add skill entries here
  • src/components/Skills/DocRenderer.js - renders this doc format

More skills in this library

Browse all technical skills - Mobile, AI/ML, Web, Backend, Design, and Product.