Getting Started with Retrieval-Augmented Generation

0. Step: Why Do We Need RAG?

When we press Enter in ChatBox and send a query like “Can the excess credits earned beyond the required core general education courses at the University of Electronic Science and Technology of China be converted into autonomous elective credits?” the predictable result is that we won’t get a useful answer: in ChatGPT Web, GPT 5.2 Thinking uses its powerful search ability to look through school documents and campus forum posts, and finally concludes “No”; DeepSeek in ChatBox just makes something up.

Another example: ask the AI “the original sentence or paragraph in which Guanyin tells Sun Wukong that someone will save him, and mark the chapter title.” This is a very specific question: in ChatGPT Web, it thinks for over two minutes, checks 106 web pages, and answers off the point, as if it didn’t understand the question.

What we want is the passage in “Chapter 8, The Buddha Creates the Scriptures and Spreads Bliss; Guanyin Goes to Chang'an by Imperial Decree”: “The bodhisattva heard this and was overjoyed. She said to the Great Sage, ‘The holy scripture says: “If your words are good, they will be echoed from a thousand li away; if your words are not good, they will be opposed from a thousand li away.” Since you have this intention, when I reach the great Tang in the east and find a scripture pilgrim, I will have him save you. You can be his disciple, uphold the faith, enter our Buddhist gate, and cultivate the right fruition again—how about it?’ The Great Sage said again and again, ‘I will go! I will go!’”

Books like Journey to the West are commonly included in training data, yet even so, the model still cannot accurately provide exact information. And relying on web search doesn’t necessarily handle factual answers well either.

Introducing RAG can help solve such problems:
A lack of domain-specific or private knowledge: some specialized knowledge was never included in training, or it lives in an internal knowledge base that the model cannot access.
Knowledge has a cutoff date: the model cannot get the latest knowledge.

We only need the school’s official curriculum manual to supply credit-related knowledge; we only need the complete novel to retrieve semantically similar passages.

1. Intro: What Is RAG?

RAG stands for Retrieval-Augmented Generation, a technique that combines information retrieval with LLMs. It retrieves relevant information from a knowledge base—such as internal documents, databases, and so on—to “augment” the LLM’s generation ability, making its answers more accurate, timely, and relevant to specific domains or private data, without retraining the entire model. This effectively addresses the knowledge limitations and “hallucination” problems of large models.

As the name suggests, it is divided into three stages: retrieval, augmentation, and generation.
Retrieval: when the user asks a question, the system searches the knowledge base for relevant information fragments.
Augmentation: the retrieved information is combined with the user’s original statement to form an augmented prompt.
Generation: this augmented input prompt is handed to the LLM to generate a more accurate, better-supported answer.

2. A Simple RAG

The figure above shows the most basic RAG pipeline.

It is divided into three stages:
Indexing stage: split the original knowledge documents into chunks and convert them into vector format.
Retrieval stage: convert the user query into vector format and look up the most relevant document chunks.
Generation stage: produce an “accurate” answer based on the retrieved information.

2.1 Indexing Stage

The indexing stage splits the original knowledge document into chunks and converts them into vector format through an embedding model, then stores them in a vector database for retrieval.

2.1.1 Splitting Documents

Splitting documents divides the original knowledge document into text chunks using different splitting strategies.

The core requirement is semantic coherence.

Common splitting strategies include:

1. Split by a fixed number of characters or tokens
2. Split by punctuation
3. Split by sentences
4. Split by paragraphs
5. Split by semantics
6. Overlapping split

In short, there are many options; the main point is to split the document appropriately into small chunks according to its type, while keeping the semantics coherent.

2.1.2 Text Embedding

The purpose of text embedding is to convert the split text chunks into a numerical representation that is easy for computers to compute with—that is, a vector.

Simple methods include: binary bag-of-words, count-based bag-of-words, and TF-IDF.

The bag-of-words method first extracts tokens from the corpus and treats each token as a dimension.
Binary bag-of-words: if the text contains the token, the dimension corresponding to that token is recorded as 1; otherwise it is 0.
Count-based bag-of-words: the dimension for each token records how many times that token appears in the text.

TF-IDF: on the basis of term frequency, inverse document frequency is introduced for weighting. Common words with low discriminative power are downweighted, while words that better reflect differences between texts are given higher weights.

But in any case, their use of dimensions is too sparse. For a novel, there may be hundreds of thousands of tokens—that is, a vector may have hundreds of thousands of dimensions. But for a particular chunk, most of the dimensions are 0 and only a few have values. Hence dense vectors came into being.

Dense vectors—which are what the embedding models we use today produce—compress the meaning of a text into a numerical space with fixed dimensions. Each dimension no longer corresponds to a specific token; it simply uses fewer dimensions to express semantic features. They can capture that “happy” and “glad” are synonyms, and know the relationship between Beijing and China, or Paris and France.

In our use, we only need to input the text into an embedding model to get the corresponding text vector.
You can choose a suitable model on the HuggingFace leaderboard, such as:
Qwen3-Embedding-8B
gemini-embedding-001

2.1.3 Vector Storage

After obtaining the text vectors, we also need a place to store them, and a corresponding retrieval method. So we usually use a vector database to store these vectors.

A vector database supports storing vectors and provides similarity-based vector retrieval, such as TopK nearest-neighbor queries, to quickly find the content most relevant to the query from a huge number of chunks.

A vector database generally stores three types of information:

1. The vector itself: the embedding vector of the chunk
2. The original content: the chunk, used to be directly concatenated into the augmented prompt
3. Metadata: stores data such as the source document, chapter, table of contents, page number, offset, and time labels

There are many vector databases too; you can search for suitable ones, such as:
Chroma
Qdrant

2.2 Retrieval Stage

The retrieval stage converts the user’s query into a text vector and looks up the most relevant document chunks.

Since we have already encoded text semantics into vectors, retrieval based on text similarity is essentially about how close two vectors are. Common methods are mainly: cosine similarity, Euclidean distance, and dot product.

Cosine similarity focuses on whether the directions of two vectors are aligned, namely their cosine value, with a range of [-1, 1]. The closer it is to 1, the more aligned the directions are.
Euclidean distance focuses on the straight-line distance between two vectors in space; the smaller the distance, the more similar they are.
Dot product is affected by both direction and length: the more aligned the directions and the longer the length, the larger the dot product tends to be.

In addition, there is keyword-based retrieval, which extracts keywords from the user’s query and matches corresponding documents. It is more effective in some scenarios, but we won’t go into it here.

2.3 Generation Stage

The generation stage concatenates the context retrieved in the previous step with the user’s query to form an augmented prompt, and hands it to the LLM so that it can produce an “accurate” answer with evidence.

There is not much to say about this stage; the main task is to select and organize the context, that is, concatenate the retrieved chunks with the user’s query, and finally hand it to the LLM.

But it also involves some optimizations, for example:

1. When different retrieval methods produce different topK chunk results, how to filter more appropriate topK chunks
2. How to make the prompt template clearer and more specific
3. How to get the LLM to correctly cite relevant evidence when generating an answer

2.4 Hands-On Practice

We have finished the theory of simple RAG. Next, let’s build a simple novel-Q&A RAG from scratch, using the Journey to the West question from the beginning as an example.

I used LM Studio to run the embedding and LLM services locally; it provides an OpenAI-compatible API.

I used text-embedding-qwen3-embedding-0.6b and qwen/qwen3-4b-2507 as the models for this hands-on practice.

Install the following dependencies: pip install openai chromadb

Among them, chromadb is an open-source vector database, and openai is added for convenient API access.

Indexing stage

For chunking, first use the heading format “第 xx 回” to split the novel into chapters, and then apply overlapping splitting within each chapter: split by CHUNK_SIZE, but the starting point moves forward by OVERLAP each time. That is, if a chapter has 2500 characters, the chunks are [0, 800), [650, 1450), [1300, 2100), [1950, 2750), [2600, none), and the last chunk is empty and filtered out.

Then vectorize each chunk and add it to the vector database; meta_data can carry the chapter title.

Distance metrics include: cosine, l2, ip—cosine similarity, Euclidean distance, and dot product.

Here we choose cosine similarity. chromadb requires you to decide the metric when creating the collection.

Retrieval stage

Vectorize the user query and look up the topK most similar results in the vector database.

Generation stage

Concatenate the user message and the retrieved context to form an augmented prompt, and hand it to the LLM to generate an answer.

"""
简易小说 RAG
三个核心阶段：索引、检索、生成
"""
import os
import re
from openai import OpenAI
import chromadb

# ============================================================
# 配置
# ============================================================∫

NOVEL_PATH = "/Users/chanler/Downloads/西游记.txt" # 小说路径
CHUNK_SIZE = 800 # 每个分块的字符数
OVERLAP = 150 # 相邻分块的重叠字符数(向前重叠)
DISTANCE_METRIC = "cosine"  # 距离度量：cosine / l2 / ip
CHROMA_DB_PATH = "./chroma_db"  # ChromaDB 存储路径

# 自动生成 collection 名称：文件名_度量方式
NOVEL_NAME = "xiyouji"
COLLECTION_NAME = f"{NOVEL_NAME}_{DISTANCE_METRIC}"

# OpenAI 客户端
client = OpenAI(
    base_url="http://127.0.0.1:1234/v1",
    api_key="lm-studio"
)
EMBEDDING_MODEL = "text-embedding-qwen3-embedding-0.6b"
LLM_MODEL = "qwen3-8b"


# ============================================================
# 第一阶段：索引 (Indexing)
# ============================================================

def load_and_split(path, max_chapters=100):
    """
    读取小说并切分为 chunks
    切分策略：先按回目划分章节，章节内重叠切分
    """
    with open(path, 'r', encoding='utf-8') as f:
        text = f.read()

    # 匹配回目标题
    chapter_pattern = r'(第[一二三四五六七八九十百零\d]+回\s+[^\n]+)'
    matches = list(re.finditer(chapter_pattern, text))

    chunks = []
    for i, match in enumerate(matches[:max_chapters]):
        chapter_title = match.group(1).strip()
        start = match.end()
        end = matches[i + 1].start() if i + 1 < len(matches) else len(text)
        content = text[start:end].strip()

        # 滑动窗口切分
        pos = 0
        while pos < len(content):
            chunk_text = content[pos:pos + CHUNK_SIZE]
            if chunk_text.strip():
                chunks.append((chunk_text.strip(), chapter_title))
            pos += CHUNK_SIZE - OVERLAP

    return chunks


def get_embedding(text):
    """获取文本向量"""
    response = client.embeddings.create(
        model=EMBEDDING_MODEL,
        input=text
    )
    return response.data[0].embedding


def build_index(chunks):
    """构建向量索引"""
    db = chromadb.PersistentClient(path=CHROMA_DB_PATH)

    try:
        db.delete_collection(COLLECTION_NAME)
    except:
        pass

    collection = db.create_collection(
        name=COLLECTION_NAME,
        metadata={"hnsw:space": DISTANCE_METRIC}
    )

    print(f"索引 {len(chunks)} 个文本块...")
    for i, (text, chapter) in enumerate(chunks):
        if i % 100 == 0:
            print(f"  进度: {i}/{len(chunks)}")

        collection.add(
            ids=[f"chunk_{i}"],
            embeddings=[get_embedding(text)],
            documents=[text],
            metadatas=[{"chapter": chapter}]
        )

    print(f"完成，共 {collection.count()} 条")
    return collection


# ============================================================
# 第二阶段：检索 (Retrieval)
# ============================================================

def retrieve(collection, query, top_k=3):
    """检索最相关的文本块"""
    results = collection.query(
        query_embeddings=[get_embedding(query)],
        n_results=top_k,
        include=["documents", "metadatas", "distances"]
    )
    return results


# ============================================================
# 第三阶段：生成 (Generation)
# ============================================================

def build_prompt(query, contexts):
    """构建增强提示词"""
    context_text = "\n\n".join(
        f"【{meta['chapter']}】\n{doc}"
        for doc, meta in contexts
    )

    return f"""你是一个小说问答助手。请根据以下参考内容回答问题。

要求：
1. 只根据参考内容作答，不要编造
2. 如果没有相关信息，明确说明
3. 标注出处（回目）

参考内容：
{context_text}

问题：{query}

回答："""


def generate(prompt):
    """调用 LLM 生成回答"""
    response = client.chat.completions.create(
        model=LLM_MODEL,
        messages=[{"role": "user", "content": prompt}],
        temperature=0.7,
        max_tokens=1000
    )
    return response.choices[0].message.content


def rag_query(collection, query, top_k=1):
    """完整 RAG 流程：检索 -> 增强 -> 生成"""
    results = retrieve(collection, query, top_k)

    contexts = list(zip(
        results["documents"][0],
        results["metadatas"][0]
    ))

    prompt = build_prompt(query, contexts)
    print(f"\n{'='*50}\n【完整提示词】\n{'='*50}\n{prompt}\n")
    answer = generate(prompt)

    return {
        "answer": answer,
        "sources": results["metadatas"][0],
        "distances": results["distances"][0]
    }


# ============================================================
# 索引管理
# ============================================================

def get_collection():
    """获取已有索引"""
    db = chromadb.PersistentClient(path=CHROMA_DB_PATH)
    return db.get_collection(COLLECTION_NAME)


def index_exists():
    """检查索引是否存在"""
    db = chromadb.PersistentClient(path=CHROMA_DB_PATH)
    try:
        return db.get_collection(COLLECTION_NAME).count() > 0
    except:
        return False


# ============================================================
# 主流程
# ============================================================

if __name__ == "__main__":
    import sys

    force_index = "--index" in sys.argv

    print("=" * 50)
    print("简易小说 RAG")
    print("=" * 50)
    print(f"Collection: {COLLECTION_NAME}")

    if force_index or not index_exists():
        print("\n[索引阶段]")
        chunks = load_and_split(NOVEL_PATH)
        print(f"切分为 {len(chunks)} 个文本块")
        collection = build_index(chunks)
    else:
        print("\n[加载已有索引]")
        collection = get_collection()
        print(f"已加载 {collection.count()} 条记录")

    print("\n[检索与生成]")
    query = "菩萨告诉孙悟空有人救他的原文句子或段落，并标注回目"
    print(f"查询: {query}\n")

    result = rag_query(collection, query, top_k=2)

    print("="*50)
    print("【参考】")
    for src, dist in zip(result["sources"], result["distances"]):
        print(f" {src['chapter']} (相似度: {1-dist:.1%})")
    print("="*50)
    print("【回答】")
    print(result["answer"])
    print("="*50)

I tested the answers with topK values of 1, 2, and 3, and they were basically usable. In fact, the correct original passage was the top1 result. In the top2 case, the model probably didn’t understand what “someone will save him” meant; using the deepseek-chat API gave the answer I expected.

top2 case

The original sentence in which Guanyin tells Sun Wukong that someone will save him is: “I am following the Buddha’s decree to go to the Eastern Land to find a scripture pilgrim, and I specially left a lingering step to see you.”
Source: Chapter 8, The Buddha Creates the Scriptures and Spreads Bliss; Guanyin Goes to Chang'an by Imperial Decree

References:
Chapter 8, The Buddha Creates the Scriptures and Spreads Bliss; Guanyin Goes to Chang'an by Imperial Decree (similarity: 70.0%)
Chapter 14, The Mind Monkey Returns to the Right; The Six Robbers Vanish (similarity: 69.9%)

top3 case

The original sentence in which Guanyin tells Sun Wukong that someone will save him is: “Since you have this intention, when I reach the great Tang in the east and find a scripture pilgrim, I will have him save you. You can be his disciple, uphold the faith, enter our Buddhist gate, and cultivate the right fruition again—how about it?”
Marked chapter: Chapter 8, The Buddha Creates the Scriptures and Spreads Bliss; Guanyin Goes to Chang'an by Imperial Decree

References:
Chapter 8, The Buddha Creates the Scriptures and Spreads Bliss; Guanyin Goes to Chang'an by Imperial Decree (similarity: 70.0%)
Chapter 14, The Mind Monkey Returns to the Right; The Six Robbers Vanish (similarity: 69.9%)
Chapter 21, The Protector Sets Up a Manor to Keep the Great Sage; Lingji of Sumeru Subdues the Wind Demon (similarity: 69.9%)

Note: the similarity values here are because chromaDB returns 1 - similarity as the distance for cosine; I’ve simply converted them back.

2.5 Summary

At this point, we have run through the simple RAG pipeline: offline indexing (splitting, vectorization, storage) + online Q&A (retrieval, augmented prompt, generation).

At the same time, through a series of hands-on practices, we can see many shortcomings. In fact, there are many places that can be improved, and this is exactly what “Advanced RAG” needs to solve:

• Chunking: how to split chunks to ensure semantic coherence (semantic splitting)
• Retrieval: how to improve retrieval precision and recall (multi-path recall, reranking, etc.)
• Query: make the query used for matching clearer and more searchable (query rewriting, coreference resolution)
• Generation: how to write prompts and organize context (prompt templates, citations)
• Evaluation: how to evaluate RAG with metrics (precision, recall, faithfulness)