What are the four stages of a RAG pipeline?

Ingestion (steps 1–4), indexing (steps 5–6), retrieval (steps 7–9), and generation (steps 10–12). Each stage has distinct failure modes.

Where do most RAG deployments fail?

Retrieval, not the LLM. Common causes: bad chunking, missing metadata, returning volume instead of relevance, and no evaluation after generation.

What is hybrid search in RAG?

Combining semantic vector search with keyword search (BM25). Pure vector search misses exact codes and error strings; pure keyword search misses paraphrases.

When should I add agentic RAG?

After you have a working 12-step pipeline with reranking. Agentic loops re-query when context is insufficient; they do not fix bad ingestion or indexing.

How do I evaluate RAG quality?

Step 12: measure faithfulness (answer grounded in retrieved chunks), relevance, and answer quality on a real user question set. Run evals before and after every pipeline change.

RAG or CAG for stable policies?

CAG when the knowledge is stable, fits in context, and the agent needs all of it every time. RAG when the corpus is large and changes often.

All notes

In this note (7 sections)

Architecture Jul 1, 2026Updated Jul 6, 2026 11 min

The complete RAG pipeline: 12 steps behind every reliable agent.

Last updated on Jul 6, 2026

Ingestion, indexing, retrieval, generation. Four stages. Twelve steps. Four failure points that kill most deployments.

Introduction

I have walked into more RAG postmortems than I can count where the team spent three weeks tuning the model while the retrieval layer was returning the wrong documents. The pattern is always the same: a demo that looked fine on ten PDFs, then quiet hallucinations in production once the corpus grew.

This note is the 12-step map I draw on whiteboards during engagements. Four stages, twelve steps, four failure points that kill most deployments. If you are comparing vanilla RAG to agentic RAG with sufficient context loops, this is the foundation those loops assume you already built.

Stage 1: Ingestion (steps 1–4)

Garbage in, garbage out. Ingestion is where most teams rush because it feels like plumbing. It is also where you poison every downstream step if you get it wrong.

01
Step 1: Data sources
Collect from every source the agent will need: Confluence, SharePoint, tickets, product docs, Slack exports. Miss a source and the agent will confidently answer from what it has, which is worse than saying it does not know.
02
Step 2: Document loading
Ingest into a pipeline that preserves structure. Tables, headers, and code blocks matter. A loader that flattens everything into one text blob destroys the semantics your chunker needs.
03
Step 3: Meaningful chunking
Chunk for meaning, not for token count. Fixed 512-token splits are fast to implement and expensive to debug. I prefer semantic boundaries: section headers, paragraph breaks, logical units. Overlap helps, but coherent units help more.
04
Step 4: Metadata extraction
Extract author, date, topic tags, product area, access level. Skipping metadata to save time is the second most common mistake I see. Retrieval without filters returns volume, not relevance.

Stage 2: Indexing (steps 5–6)

Indexing turns chunks into searchable knowledge. The embedding model and chunk boundaries matter more than which vector database logo is on the slide.

Step 5: Embeddings. Pick an embedding model matched to your domain and language mix. Re-embed when you change chunking strategy.
Step 6: Vector database. Store embeddings for semantic search. Pinecone, Weaviate, pgvector, Chroma: the choice is ops and cost, not magic. The output is indexed knowledge ready to query.

Stage 3: Retrieval (steps 7–9)

Retrieval is the make-or-break stage. I have seen teams burn Opus tokens on perfect prompts while step 8 returned forty chunks of noise. Fix retrieval first.

01
Step 7: Query rewriting
Expand intent, clarify meaning, rewrite the user query to match how content was indexed. A question about "refund policy" may need to match "returns and cancellations" in your docs.
02
Step 8: Hybrid search
Combine semantic search with keyword search (BM25). Pure vector search misses exact product codes and error strings. Pure keyword search misses paraphrases. You need both.
03
Step 9: Reranking
Reorder results by relevance, context, and intent. Returning top-k by cosine similarity alone is how you flood the LLM with irrelevant chunks. A cross-encoder reranker or LLM rerank step pays for itself quickly.

Four failure points that kill RAG

Failure	Symptom	Fix first
Chunks optimized for speed	Right topic, wrong detail	Semantic chunking with metadata
Skipped metadata	Cross-product contamination	Extract and filter by tags
Volume not relevance	Long prompts, wrong answers	Hybrid search + reranking
No evaluation	Quiet hallucinations in prod	Faithfulness eval on step 12

Stage 4: Generation (steps 10–12)

Generation only works if you close the loop. Steps 10–12 assemble context, produce a grounded answer, and measure whether you should ship it.

01
Step 10: Context assembly
Organize relevant chunks into a coherent prompt. Order matters: most relevant first, citations traceable, token budget respected.
02
Step 11: LLM generation
Produce an answer grounded in provided context. Instruct the model to cite sources and refuse when context is insufficient.
03
Step 12: Evaluation
Measure faithfulness, relevance, and quality. Without step 12 you ship a demo that hallucinates quietly. Build an eval set from real user questions, not synthetic trivia.

Where agentic RAG fits

Agentic RAG adds a re-search loop when context is insufficient. The agent queries again with a refined question until it has enough to answer. That loop sits after step 9 (reranking), not instead of steps 1–8. Read the full comparison in agentic RAG vs vanilla RAG.

Memory type matters too. Large changing corpora belong in RAG (steps 5–6). Stable knowledge that fits in context belongs in CAG. See wrong memory, dead agent for the four-type decision rule.

Conclusion

Before you tune the model, audit the pipeline. Draw all twelve steps on a whiteboard, mark where your eval set fails, and fix the earliest broken step. RAG is infrastructure. Treat it that way and your agents stop guessing.

Key takeaways

1RAG is a 12-step pipeline across four stages, not a vector database plus a prompt. Ingestion, indexing, retrieval, and generation each have distinct failure modes. Most production RAG breaks in retrieval, not in the LLM.
2Ingestion (steps 1–4): collect data sources, load documents, chunk for meaning (not speed), and extract metadata (author, date, topic tags). Garbage in, garbage out. Chunks optimized for token count instead of semantic coherence poison every downstream step.
3Indexing (steps 5–6): embed chunks into vectors and store them in a vector database for semantic search. The output is indexed knowledge ready to query. Embedding model choice and chunk boundaries matter more than which vector DB logo is on the slide.
4Retrieval (steps 7–9) is the make-or-break stage. Rewrite the query to match indexed content, run hybrid search (semantic plus keyword), then rerank by relevance and intent. Returning volume instead of relevance is the most common mistake I see in failed RAG deployments.
5Generation (steps 10–12): assemble context into a coherent prompt, generate a grounded answer from the LLM, then evaluate faithfulness and relevance. Without step 12, you ship a demo that hallucinates quietly in production.
6Four failure points that kill RAG: chunks optimized for speed instead of meaning, skipping metadata extraction, flooding the LLM with irrelevant retrieved chunks, and no evaluation after generation. Fix any one of these before you tune the model.
7Agentic RAG (re-search until context is sufficient) sits on top of this pipeline, not instead of it. See /blog/agentic-rag-sufficient-context-vs-vanilla-rag for when to add a Sufficient Context Agent loop after step 9.
8Match memory type to the job: large changing corpora belong in RAG (step 5–6), stable knowledge that fits in context belongs in CAG. See /notes/wrong-memory-dead-agent for the four-type decision rule.

Frequently asked questions

Get the visual notes by email

New agentic AI notes and breakdowns, plus what I am shipping for clients, one email on Thursdays.