Retrieval-Augmented Generation (RAG)

RAG is a pattern, not a single algorithm. The shape is:

1. Offline indexing. Documents are split into chunks, each chunk is embedded by an encoder model, and the resulting vectors are stored in a vector index (e.g. HNSW, IVF) along with the original text and metadata.

1. Online query. The user query is embedded with the same encoder. The index returns the top-k chunks by cosine similarity (or another metric). Optional rerankers refine the ordering.

1. Generation. The retrieved chunks are stuffed into the model's context window with the user query, and the model produces an answer that is conditioned on the retrieved evidence. Citations are typically included by referring to chunk IDs or document names.

Why RAG works:

• Freshness: new documents become retrievable after ingestion and indexing complete — no retraining.
• Attribution: the prompt contains the source, so the model can be instructed to cite it.
• Cost: far cheaper than continual pretraining or frequent fine-tuning.
• Privacy control: source documents can stay in your index and are not added to base-model training, but retrieved chunks still reach the inference endpoint unless the model runs inside your own security boundary.

Failure modes:

• Retrieval miss: the right chunk is not retrieved; the model may answer incorrectly unless it can abstain.
• Lost in the middle: LLMs attend less faithfully to mid-context information; placing the answer near the start or end of the prompt helps.
• Stale or noisy chunks: garbage in, garbage out. Quality of the corpus and the chunking strategy matter.

RAG remains a standard production pattern for "chat with your data" features.