ConceptDBISAI-written · not yet proofread

Retrieval-Augmented Generation

Also RAG

Retrieval-augmented generation (RAG) answers a query over a text larger than an LLM's context size by splitting the text into chunks, embedding them, retrieving the k chunks most similar to the query embedding, and passing only their text to the LLM with a suitable prompt.

Know these firstEmbedding Cosine Similarity Large Language Model

Definition

Retrieval-augmented generation (RAG) lets an LLM answer a query over a text that exceeds its context size: based on the query, extract a small subset of the text and pass only that to the LLM, together with a suitable prompt.

Preprocessing (once per text):

Chunking: split the text into (possibly overlapping) chunks; extend each chunk to complete sentences by searching for the nearest sentence delimiter to the left and to the right.
Embedding: compute an embedding $e_i \in \mathbb{R}^d$ of each chunk $i$ .

Query processing:

Compute the embedding $e_q$ of the query, with the same embedding method as for the chunks.
Find the $k$ chunks with the highest cosine similarity $\text{cos-sim}(e_q, e_i)$ .
Pass the text of these $k$ chunks to the LLM, together with a suitable system prompt.

Properties#

Motivation: context sizes are large (400k tokens for GPT-5, 1M for Gemini Pro), but many applications query texts larger than that. ES12: the transcripts of all lecture recordings exceed the context size of the provided LLM. Since only $k$ chunks enter the prompt, the text can be arbitrarily large.
Embedding method: a sum of word embeddings (ES11) is simplistic, because not every word is equally relevant. ES12 uses an embedding API (mixedbread-ai/mxbai-embed-large-v1) that computes one embedding per text (a dedicated embedding model, such as a sentence transformer).
Retrieval: a linear scan over all chunk embeddings suffices for ES12; real applications precompute a special data structure. With normalized embeddings, all similarities are one matrix-vector product (Cosine Similarity > Formula).
Tuning knobs: chunk size, overlap, the embedding model, $k$ and the system prompt (ES12: experiment with $k$ , chunk sizes and system prompts).
ES12 details: one chunk per subtitle paragraph (one paragraph per timestamp); remove the timestamps before embedding, as they carry no useful information.

Example#

ExampleToy retrieval with $k = 1$ (illustrative 2D embeddings)

Chunks: $C_1$ “B+ trees store keys in sorted leaves” $e_1 = (0.9, 0.1)$ , $C_2$ “BM25 ranks documents” $e_2 = (0.2, 0.8)$ , $C_3$ “Embeddings map words to vectors” $e_3 = (0.1, 1.0)$ . Query “How does BM25 ranking work?” $e_q = (0.3, 0.9)$ .

\text{cos-sim}(e_q, e_1) = \tfrac{0.36}{0.949 \cdot 0.906} \approx 0.42,\quad \text{cos-sim}(e_q, e_2) = \tfrac{0.78}{0.949 \cdot 0.825} \approx 1.00,\quad \text{cos-sim}(e_q, e_3) = \tfrac{0.93}{0.949 \cdot 1.005} \approx 0.98.

The top-1 chunk is $C_2$ ; the LLM gets the system prompt, the text of $C_2$ and the query.

Sources#

Slides 12, p. 14 (motivation), Slides 12, p. 15 (preprocessing), Slides 12, p. 16 (embedding API), Slides 12, p. 17 (query processing)
DBIS_VL12 > Retrieval-Augmented Generation, DBIS_VL12 > Preprocessing, DBIS_VL12 > Computing the embeddings, DBIS_VL12 > Processing a query

Connections

Mentioned in inline links5

Embedding
Uses: similarity search over words or documents (ES11) and retrieving the chunks most similar to a query in Retrieval-Augmented Generation (ES12). §
Language Model
Long prompts: if the prompt has more than $n$ tokens, only the last $n$ are considered, or a subset of the text is selected via Retrieval-Augmented Generation. §
Large Language Model
Prompt: the sequence of previous tokens. If it is longer than $n$ , only the last $n$ tokens count, or RAG selects the relevant part of a large text. §
VL12 – N-gram Models, LLMs, RAG
Context sizes of current LLMs are large (400k tokens for GPT-5, 1M tokens for Gemini Pro), but many applications need to query a text larger than that. For ES12 the input is the transcripts of all lecture recordings of this course, which e… §
Retrieval-Augmented Generation — chunk, embed, retrieve top- $k$ by cos-sim, prompt the LLM §
Word Embeddings and Language Models
Retrieval-Augmented Generation — chunk, embed, retrieve top- $k$ by cos-sim, prompt the LLM §
When the text exceeds even an LLM’s context size (ES12: all lecture transcripts), RAG uses the first half of the topic to choose the input for the second: split the text into sentence-complete chunks, embed them, embed the query with the s… §

Search in these mentions →