Five RAG Architectures and When Each One Fits in Production
A practical tour of naive, hybrid, graph, advanced and agentic RAG with LlamaIndex and LangGraph, covering failure modes, costs and how to stack them.
This article is published in English.
Sources: The 5 RAG Architectures and Exactly When to Use Each One in Production
The word "RAG" now covers a family of quite different architectures, each built for a different class of retrieval problem. Choosing the wrong one isn't merely a performance matter. It can be the difference between a system that works and one that quietly hands users confident, wrong answers at scale.
This is a map of the whole family. The goal is to be able to look at any retrieval problem and say which architecture it calls for, and how to build it with LangGraph for orchestration and LlamaIndex for the data layer. The five architectures run from simplest to most complex: naive, hybrid, graph, advanced and agentic. They aren't rivals. They're layers added progressively as a problem demands more, and most production systems combine at least two.
The shared problem
Underneath, each RAG setup tackles a single puzzle: a language model needs facts it never saw in training, delivered exactly when it is answering, and shaped so it can reason about them.
Models stop learning at a cutoff date. They have never read a firm's internal wiki, a spec sheet, or anything published after training ended. Fine-tuning on such content is slow and pricey, and the tuned model still goes stale whenever the documents change.
RAG sidesteps all of that. Rather than teaching the model new facts, the system retrieves relevant knowledge at query time and places it in the prompt as context. The model never has to memorise anything, it only has to read it when it matters. The five architectures below are five answers to how to retrieve well, and each suits a different problem.
Architecture 1: naive RAG
Naive RAG is the starting point, and plenty of real workloads never need more: internal policy bots, FAQ assistants, documentation search, onboarding helpers. Despite the name, it is a mature pattern that runs at scale. You load documents, build a LlamaIndex VectorStoreIndex, expose its query engine as a @tool, and give that tool to a LangGraph agent.
Its five sequential steps map directly onto what LlamaIndex offers out of the box. The first four happen when the index is built, and the fifth happens for every question:
Stages 1–4: Indexing time (run once)
Stage 5: Query time (run on every user question)
The most important distinction in all of RAG is indexing time versus query time. The embedding cost is paid once, up front. At query time the system pays only for a single similarity search and one LLM call.
Why it works
A user's question is embedded into the same vector space as the document chunks, and the chunks whose vectors lie geometrically closest to the question vector get returned. The underlying assumption is that closeness in vector space reflects closeness in meaning. For clean, factual corpora, that assumption holds up surprisingly well.
The code
A complete template follows, covering imports and configuration, state, a persisted knowledge base, the tool, nodes, routing and graph assembly:
# ============================================================
# NAIVE RAG — COMPLETE TEMPLATE
# ============================================================
# ── MODULE 1: IMPORTS & CONFIGURATION ───────────────────────
import os
from typing import Literal
from langchain_openai import ChatOpenAI
from langchain_core.messages import HumanMessage, SystemMessage
from langchain_core.tools import tool
from langgraph.graph import StateGraph, MessagesState, START, END
from langgraph.prebuilt import ToolNode
from langgraph.checkpoint.memory import MemorySaver
from llama_index.core import (
Settings, SimpleDirectoryReader, VectorStoreIndex,
StorageContext, load_index_from_storage
)
from llama_index.llms.openai import OpenAI as LlamaOpenAI
from llama_index.embeddings.openai import OpenAIEmbedding
# LangGraph's reasoning model
llm = ChatOpenAI(model="gpt-4o", temperature=0)
# LlamaIndex's internal model - separate from the above
# These two model configs do NOT share state or conversation history
Settings.llm = LlamaOpenAI(model="gpt-4o-mini", temperature=0.1)
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")
Settings.chunk_size = 512
Settings.chunk_overlap = 50
# ── MODULE 2: STATE ──────────────────────────────────────────
class State(MessagesState):
pass # messages list inherited; extend if you need more
# ── MODULE 3: KNOWLEDGE BASE (Build once, persist, reload) ───
PERSIST_DIR = "./storage/naive"
if os.path.exists(PERSIST_DIR):
storage_context = StorageContext.from_defaults(persist_dir=PERSIST_DIR)
index = load_index_from_storage(storage_context)
else:
documents = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(documents, show_progress=True)
index.storage_context.persist(persist_dir=PERSIST_DIR)
query_engine = index.as_query_engine(similarity_top_k=3)
# ── MODULE 3 (cont.): TOOL ───────────────────────────────────
@tool
def search_knowledge_base(query: str) -> str:
"""Search internal company documents for policies, product specs,
and procedures. Use this for any question requiring domain-specific
knowledge rather than general world knowledge."""
response = query_engine.query(query)
return str(response)
tools = [search_knowledge_base]
llm_with_tools = llm.bind_tools(tools)
tool_node = ToolNode(tools)
# ── MODULE 4: NODES ──────────────────────────────────────────
def agent_node(state: State) -> dict:
system_prompt = SystemMessage(content=(
"You are a helpful assistant with access to an internal knowledge base. "
"Use search_knowledge_base for company-specific questions. "
"Answer general questions directly."
))
response = llm_with_tools.invoke([system_prompt] + state["messages"])
return {"messages": [response]}
# ── MODULE 5: ROUTING ────────────────────────────────────────
def should_continue(state: State) -> Literal["tools", "__end__"]:
last = state["messages"][-1]
if hasattr(last, "tool_calls") and last.tool_calls:
return "tools"
return "__end__"
# ── MODULE 6: GRAPH ASSEMBLY ─────────────────────────────────
builder = StateGraph(State)
builder.add_node("agent", agent_node)
builder.add_node("tools", tool_node)
builder.add_edge(START, "agent")
builder.add_conditional_edges("agent", should_continue,
{"tools": "tools", "__end__": END})
builder.add_edge("tools", "agent")
graph = builder.compile(checkpointer=MemorySaver())
Where it breaks
Naive RAG rests on one assumption, and that assumption fails in two common situations.
The first is vocabulary mismatch. A user asks for the SLA that applies to tier-1 clients, but the document promises Gold-tier customers 99.9% uptime. SLA, tier-1 and Gold-tier mean related things without matching, so similarity scoring may rank that chunk too low and the answer slips through.
The second is relational questions. Asking which products are affected if a particular supplier goes offline requires following a chain of relationships across several documents. No single chunk holds the answer, and naive RAG returns pieces from different documents with no means of connecting them.
Those two failure modes are what the next two architectures address.
Architecture 2: hybrid RAG
Hybrid RAG reflects something practitioners learned in production: semantic similarity and exact keyword matching are complementary signals of relevance, and neither alone is enough.
Dense retrieval, which is vector search, handles paraphrase well: ask about an SLA and it can still find the Gold-tier uptime passage. It falters on proper nouns, product codes, clinical terms, legal citations and other specialist wording where the exact string carries the meaning.
Sparse retrieval (BM25, or keyword search) is the mirror image. It's superb at exact term matching and will always find "SKU-4829" for a search on "SKU-4829", but it has no notion of semantic equivalence, so it won't find "uptime guarantee" for "SLA".
Hybrid RAG runs both searches in parallel and then uses a reranker to turn the merged results into one ranked list. The reranker is the key component. Embedding models compare a query and a chunk independently, whereas a cross-encoder reranker reads the query and each candidate together and produces a score for their relationship directly. It's slower than similarity search, but it only touches the already reduced candidate set, so latency stays manageable.
The code
The LlamaIndex version combines a dense retriever, a BM25 retriever and reciprocal rank fusion:
# ============================================================
# HYBRID RAG — LLAMAINDEX IMPLEMENTATION
# pip install llama-index-retrievers-bm25 rank-bm25
# ============================================================
from llama_index.core import SimpleDirectoryReader, VectorStoreIndex
from llama_index.core.retrievers import VectorIndexRetriever
from llama_index.retrievers.bm25 import BM25Retriever
from llama_index.core.retrievers import QueryFusionRetriever
from llama_index.core.query_engine import RetrieverQueryEngine
from langchain_core.tools import tool
# Build document store and index as before
documents = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(documents)
# ── DENSE RETRIEVER (vector similarity) ──────────────────────
vector_retriever = VectorIndexRetriever(
index=index,
similarity_top_k=5, # Fetch more candidates pre-rerank
)
# ── SPARSE RETRIEVER (BM25 keyword match) ────────────────────
# BM25Retriever builds directly from the index's nodes
bm25_retriever = BM25Retriever.from_defaults(
index=index,
similarity_top_k=5,
)
# ── FUSION: Merge both retrievers with Reciprocal Rank Fusion ─
# QueryFusionRetriever is LlamaIndex's built-in hybrid combiner
# mode="reciprocal_rerank" implements the RRF algorithm:
# it combines ranked lists without needing score calibration
hybrid_retriever = QueryFusionRetriever(
retrievers=[vector_retriever, bm25_retriever],
similarity_top_k=3, # Final top-k after fusion
num_queries=1, # No query expansion in this mode
mode="reciprocal_rerank", # The fusion algorithm
use_async=True, # Run both retrievers in parallel
)
# Build a QueryEngine on top of the hybrid retriever
hybrid_query_engine = RetrieverQueryEngine.from_args(
retriever=hybrid_retriever,
)
# ── LANGGRAPH TOOL ────────────────────────────────────────────
@tool
def search_hybrid(query: str) -> str:
"""Search internal knowledge base using both semantic similarity
and keyword matching. More precise than pure vector search,
especially for technical terms, product codes, and exact names."""
response = hybrid_query_engine.query(query)
return str(response)
# The LangGraph graph structure is identical to Naive RAG.
# You are only swapping the tool. Everything from Module 4 onward
# stays exactly the same.
Chunk size under hybrid retrieval
With hybrid search you will usually index smaller chunks than you would for plain vector search. BM25 gets a sharper signal from a repeated term in a short chunk than in a long one. Around 256 tokens with 30 tokens of overlap makes a sensible first setting once BM25 joins the pipeline.
When hybrid is the right call
Use it whenever documents mix free-form prose with structured terminology, which covers nearly every serious enterprise use case. Examples include legal review, where citation forms must match exactly, medical records full of precise drug names, dosage codes and ICD codes, financial analysis with ticker symbols and clause identifiers, and technical documentation with error codes, API method names and version numbers.
Architecture 3: graph RAG
Graph RAG redefines the unit of knowledge. Under naive and hybrid RAG, a document is just text and retrieval picks the passages that read most like the query. Here, documents become entities linked by relationships, and retrieval walks along paths in that network.
Consider the question of which enterprise clients would be affected if the legacy authentication module were deprecated. A naive system would look for chunks mentioning "enterprise clients" and "authentication module" together and might find a few. What's actually needed is a chain: from the module to the products that depend on it, from those products to the clients who use them. No single chunk contains that answer, which emerges from the structure of the knowledge graph. Graph RAG targets exactly this situation, whether the task is hopping across several links, tracing a relationship, or answering something that needs facts from far-apart parts of the corpus.
How the index gets built
Instead of chunking documents and embedding the chunks, graph RAG begins with an entity extraction pass over every document. It identifies named entities (products, people, organisations, concepts) and the relationships between them, such as "depends on", "is a client of", "is authored by" or "was superseded by". These become the nodes and edges of a knowledge graph. Graph clustering algorithms then group closely related entities into communities, and an LLM writes a summary for each community. At query time the system searches the community summaries and traverses the graph to reach relevant entities.
The code
LlamaIndex's property graph index handles the extraction pipeline:
# ============================================================
# GRAPH RAG — LLAMAINDEX PROPERTY GRAPH INDEX
# pip install llama-index-core
# ============================================================
from llama_index.core import SimpleDirectoryReader, PropertyGraphIndex
from llama_index.core.indices.property_graph import (
ImplicitPathExtractor,
SimpleLLMPathExtractor,
)
from langchain_core.tools import tool
# Load documents
documents = SimpleDirectoryReader("./data").load_data()
# ── BUILD THE PROPERTY GRAPH INDEX ───────────────────────────
# LlamaIndex's PropertyGraphIndex handles the full extraction pipeline.
# SimpleLLMPathExtractor uses an LLM to extract (subject, relation, object)
# triples from each chunk - these become graph edges.
# ImplicitPathExtractor uses fast heuristics (cheaper, less precise).
index = PropertyGraphIndex.from_documents(
documents,
kg_extractors=[
# LLM-based extraction: higher quality, higher cost
SimpleLLMPathExtractor(
llm=Settings.llm,
max_paths_per_chunk=10,
),
# Heuristic extraction: fast fallback
ImplicitPathExtractor(),
],
show_progress=True,
)
# ── CREATE A GRAPH-AWARE RETRIEVER ────────────────────────────
# This retriever traverses the graph rather than searching flat vectors.
kg_retriever = index.as_retriever(
include_text=True, # Include surrounding text with each entity
retriever_mode="hybrid", # Combines keyword + embedding on the graph
similarity_top_k=3,
)
from llama_index.core.query_engine import RetrieverQueryEngine
graph_query_engine = RetrieverQueryEngine.from_args(retriever=kg_retriever)
# ── LANGGRAPH TOOL ────────────────────────────────────────────
@tool
def search_knowledge_graph(query: str) -> str:
"""Search the knowledge graph for questions involving relationships
between entities - dependencies, organizational hierarchies, impact
analysis, and multi-hop reasoning across connected information.
Use this when the answer requires tracing a chain of relationships
rather than finding a single relevant document."""
response = graph_query_engine.query(query)
return str(response)
# ── COMBINING WITH NAIVE RAG: DUAL-TOOL AGENT ─────────────────
# In practice, Graph RAG and Naive RAG are often combined.
# The agent's LLM decides which tool fits the query.
@tool
def search_knowledge_base(query: str) -> str:
"""Search for factual information in company documents.
Best for direct questions with answers in a single document."""
response = query_engine.query(query) # The naive query engine
return str(response)
# Give the LangGraph agent BOTH tools.
# It will route to the right one based on the question type.
tools = [search_knowledge_base, search_knowledge_graph]
llm_with_tools = llm.bind_tools(tools)
tool_node = ToolNode(tools)
# Graph assembly remains identical - only the tool list changes.
A warning about cost
Entity extraction runs an LLM call over every chunk, so a large document set means thousands of calls at indexing time. That's expensive and slow by design: the price buys a much richer data structure, once. A graph RAG index must never be rebuilt at every startup. Persist the graph and reload it, exactly as with the naive version.
When to choose it
Pick graph RAG when questions require following chains of relationships. Typical cases are compliance and risk analysis (which processes does a given regulation affect), supply chain intelligence (which products depend on a supplier), organisational knowledge (who owns what and how those ownership chains connect) and software dependency mapping (what breaks if a module is removed).
Architecture 4: advanced RAG
Advanced RAG isn't a single new technique. It's a structured set of improvements layered on top of whatever base retrieval you already have. Naive RAG trusts its first retrieval pass, while advanced RAG questions, refines and validates it. The improvements fall into three groups that slot into different points of the pipeline: before retrieval, during query handling, and after retrieval.
Before retrieval: query rewriting and HyDE
What a user types is rarely the best search query. A casual question about the returns policy for enterprise customers is perfectly natural for a human but retrieves worse results than a phrase like "enterprise customer return and refund policy procedures". Query rewriting has the LLM transform the question into a better search query before it reaches the index.
HyDE, short for Hypothetical Document Embedding, takes another route. Instead of searching with the question, it asks the LLM to write a hypothetical document that would answer it, then embeds that document for the search. The insight is that answer-shaped text sits closer in vector space to other answer-shaped text than question-shaped text does.
Query decomposition
Multi-step questions defeat naive RAG because they need several retrievals. Advanced RAG breaks them apart first. Asking to compare the refund policy for enterprise and retail customers and summarise the main differences is really three tasks: retrieve the enterprise policy, retrieve the retail policy, compare them. Decomposition splits it into sub-queries, retrieves against each and merges the results before synthesis.
After retrieval: reranking and CRAG
Reranking, already covered under hybrid search, fits in again at this stage: a cross-encoder rescoring whatever chunks were fetched, even if retrieval was purely dense.
CRAG (Corrective RAG) is the most elaborate post-retrieval technique. After fetching, it runs a quick relevance check on the chunks. If they look inadequate, it turns to a different source, say a web search or a wider index, so the LLM is not pushed to answer from weak context.
The code
One implementation brings several of these techniques together:
# ============================================================
# ADVANCED RAG — MULTI-TECHNIQUE IMPLEMENTATION
# ============================================================
from llama_index.core import SimpleDirectoryReader, VectorStoreIndex
from llama_index.core.query_engine import (
SubQuestionQueryEngine, # Handles query decomposition
RetrieverQueryEngine,
)
from llama_index.core.tools import QueryEngineTool, ToolMetadata
from llama_index.core.postprocessor import SentenceTransformerRerank
from langchain_core.tools import tool
documents = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(documents)
# ── RERANKER: Post-retrieval cross-encoder scoring ────────────
# Fetch 8 candidates, rerank down to 3
# This is the most impactful single improvement in Advanced RAG
reranker = SentenceTransformerRerank(
model="cross-encoder/ms-marco-MiniLM-L-2-v2",
top_n=3,
)
# ── RETRIEVER WITH RERANKING ──────────────────────────────────
base_retriever = index.as_retriever(similarity_top_k=8)
reranked_engine = RetrieverQueryEngine.from_args(
retriever=base_retriever,
node_postprocessors=[reranker], # Applied after retrieval
)
# ── QUERY DECOMPOSITION with SubQuestionQueryEngine ───────────
# Wrap the base engine as a "tool" that the decomposer can call
engine_tools = [
QueryEngineTool(
query_engine=reranked_engine,
metadata=ToolMetadata(
name="company_knowledge_base",
description=(
"Searches company documents for policies, procedures, "
"and product information."
),
),
)
]
# SubQuestionQueryEngine decomposes complex questions into
# sub-questions, runs each against the available tools,
# and synthesizes a final answer from all sub-answers
decomposed_engine = SubQuestionQueryEngine.from_defaults(
query_engine_tools=engine_tools,
use_async=True, # Sub-queries run in parallel when possible
)
# ── HYDE: Hypothetical Document Embedding ─────────────────────
from llama_index.core.indices.query.query_transform.base import (
HyDEQueryTransform,
)
from llama_index.core.query_engine import TransformQueryEngine
hyde_transform = HyDEQueryTransform(include_original=True)
hyde_engine = TransformQueryEngine(
query_engine=reranked_engine,
query_transform=hyde_transform,
)
# ── LANGGRAPH TOOLS: Different engines for different patterns ──
@tool
def search_with_decomposition(query: str) -> str:
"""Search for answers to complex questions that may require
combining information from multiple documents. Automatically
breaks the question into sub-questions and merges the results.
Best for comparison questions, multi-part questions, and anything
requiring synthesis across different topics."""
response = decomposed_engine.query(query)
return str(response)
@tool
def search_with_hyde(query: str) -> str:
"""Search the knowledge base using hypothetical document embedding.
More effective than standard search for abstract or exploratory
questions where the exact terminology in the answer differs from
the terminology in the question."""
response = hyde_engine.query(query)
return str(response)
# The LangGraph agent now has specialized retrieval tools
# and its LLM decides which retrieval strategy each question needs.
tools = [search_with_decomposition, search_with_hyde]
The rule to follow
Resist adding everything together. Pick the one change most likely to cure the failure you actually see. Reranking is usually the best first move, decomposition follows, and HyDE is a fair third choice for abstract or conceptual collections. Build up gradually and check recall after every step.
Architecture 5: agentic RAG
Agentic RAG goes beyond better retrieval: it hands the retrieval decisions to someone else. The earlier four are pipelines, fixed sequences that behave identically on every run. Here retrieval becomes a loop in which an LLM agent picks what to search, judges the results, chooses whether to search again, and stops when it has enough, or decides the question has no answer.
That is exactly what LangGraph was designed for. The agent node, the tool node and the conditional edge form an agentic RAG scaffold already. What changes is the richness of the tool suite and the sophistication of the routing logic built around it.
Several retrieval tools at once
The real strength of agentic RAG in LangGraph is that the agent can be given every retrieval strategy described above at the same time, and its LLM chooses the tool for each sub-question. This scaffold makes all five architectures available to a single agent:
# ============================================================
# AGENTIC RAG — COMPLETE MULTI-TOOL TEMPLATE
# This is the full production scaffold: all five architectures
# available to a single agent, which selects dynamically.
# ============================================================
# ── MODULE 1: IMPORTS & CONFIGURATION ───────────────────────
import os
from typing import Literal, Annotated
from langchain_openai import ChatOpenAI
from langchain_core.messages import HumanMessage, SystemMessage, BaseMessage
from langchain_core.tools import tool
from langgraph.graph import StateGraph, MessagesState, START, END
from langgraph.prebuilt import ToolNode
from langgraph.checkpoint.memory import MemorySaver
from llama_index.core import (
Settings, SimpleDirectoryReader, VectorStoreIndex,
StorageContext, load_index_from_storage, PropertyGraphIndex
)
from llama_index.core.indices.property_graph import SimpleLLMPathExtractor
from llama_index.core.postprocessor import SentenceTransformerRerank
from llama_index.core.query_engine import RetrieverQueryEngine, SubQuestionQueryEngine
from llama_index.core.tools import QueryEngineTool, ToolMetadata
from llama_index.llms.openai import OpenAI as LlamaOpenAI
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.retrievers.bm25 import BM25Retriever
from llama_index.core.retrievers import QueryFusionRetriever, VectorIndexRetriever
# The agent's main reasoning model (LangGraph)
llm = ChatOpenAI(model="gpt-4o", temperature=0)
# LlamaIndex internal configuration
Settings.llm = LlamaOpenAI(model="gpt-4o-mini", temperature=0.1)
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")
Settings.chunk_size = 512
Settings.chunk_overlap = 50
# ── MODULE 2: STATE ──────────────────────────────────────────
class AgentState(MessagesState):
# Extend with retrieval metadata if you need observability
retrieval_count: int = 0 # Track how many retrieval calls were made
# ── MODULE 3: KNOWLEDGE BASES (all built at startup) ──────────
VECTOR_DIR = "./storage/vector"
GRAPH_DIR = "./storage/graph"
documents = SimpleDirectoryReader("./data").load_data()
# Vector index (for Naive, Hybrid, Advanced)
if os.path.exists(VECTOR_DIR):
ctx = StorageContext.from_defaults(persist_dir=VECTOR_DIR)
vector_index = load_index_from_storage(ctx)
else:
vector_index = VectorStoreIndex.from_documents(documents, show_progress=True)
vector_index.storage_context.persist(persist_dir=VECTOR_DIR)
# Property Graph index (for Graph RAG)
if os.path.exists(GRAPH_DIR):
ctx = StorageContext.from_defaults(persist_dir=GRAPH_DIR)
graph_index = load_index_from_storage(ctx)
else:
graph_index = PropertyGraphIndex.from_documents(
documents,
kg_extractors=[SimpleLLMPathExtractor(llm=Settings.llm)],
show_progress=True,
)
graph_index.storage_context.persist(persist_dir=GRAPH_DIR)
# ── Set up individual engines ─────────────────────────────────
reranker = SentenceTransformerRerank(
model="cross-encoder/ms-marco-MiniLM-L-2-v2", top_n=3
)
# Tool 1: Naive / vector search
vector_engine = vector_index.as_query_engine(
similarity_top_k=3,
node_postprocessors=[reranker],
)
# Tool 2: Hybrid (vector + BM25)
hybrid_retriever = QueryFusionRetriever(
retrievers=[
VectorIndexRetriever(index=vector_index, similarity_top_k=5),
BM25Retriever.from_defaults(index=vector_index, similarity_top_k=5),
],
similarity_top_k=3,
mode="reciprocal_rerank",
use_async=True,
)
hybrid_engine = RetrieverQueryEngine.from_args(
retriever=hybrid_retriever,
node_postprocessors=[reranker],
)
# Tool 3: Graph traversal
graph_engine = graph_index.as_query_engine(
include_text=True,
retriever_mode="hybrid",
similarity_top_k=3,
)
# Tool 4: Decomposed (for complex multi-part questions)
decomposed_engine = SubQuestionQueryEngine.from_defaults(
query_engine_tools=[
QueryEngineTool(
query_engine=vector_engine,
metadata=ToolMetadata(
name="docs",
description="Company documents and policies."
)
)
],
use_async=True,
)
# ── MODULE 3 (cont.): ALL TOOLS ───────────────────────────────
@tool
def search_documents(query: str) -> str:
"""Search company documents by semantic meaning. Best for
conceptual questions where the exact wording in the answer
may differ from the question."""
return str(vector_engine.query(query))
@tool
def search_exact_terms(query: str) -> str:
"""Search using both keyword and semantic matching. Best when
the query contains specific terminology, product codes, names,
or exact phrases that must appear in the result."""
return str(hybrid_engine.query(query))
@tool
def search_relationships(query: str) -> str:
"""Search the knowledge graph for questions about how things
connect: dependencies, impact chains, organizational links,
and multi-hop reasoning. Use when the answer requires tracing
a relationship across multiple entities."""
return str(graph_engine.query(query))
@tool
def search_complex_question(query: str) -> str:
"""For multi-part questions requiring synthesis across several
topics. Automatically decomposes the question into sub-queries,
retrieves each independently, and combines the results."""
return str(decomposed_engine.query(query))
tools = [
search_documents,
search_exact_terms,
search_relationships,
search_complex_question,
]
llm_with_tools = llm.bind_tools(tools)
tool_node = ToolNode(tools)
# ── MODULE 4: NODES ──────────────────────────────────────────
def agent_node(state: AgentState) -> dict:
system_prompt = SystemMessage(content=(
"You are a precise research assistant with access to four retrieval tools:\n\n"
"1. search_documents - semantic search over company documents\n"
"2. search_exact_terms - hybrid semantic + keyword search\n"
"3. search_relationships - graph traversal for relationship questions\n"
"4. search_complex_question - decomposed retrieval for multi-part questions\n\n"
"Think step by step. Use the tool that best fits the question type. "
"You may call multiple tools if a question has multiple parts. "
"Only answer when you have retrieved sufficient evidence."
))
response = llm_with_tools.invoke([system_prompt] + state["messages"])
return {
"messages": [response],
"retrieval_count": state.get("retrieval_count", 0),
}
# ── MODULE 5: ROUTING ────────────────────────────────────────
def should_continue(state: AgentState) -> Literal["tools", "__end__"]:
last = state["messages"][-1]
if hasattr(last, "tool_calls") and last.tool_calls:
return "tools"
return "__end__"
# ── MODULE 6: GRAPH ASSEMBLY ─────────────────────────────────
builder = StateGraph(AgentState)
builder.add_node("agent", agent_node)
builder.add_node("tools", tool_node)
builder.add_edge(START, "agent")
builder.add_conditional_edges("agent", should_continue,
{"tools": "tools", "__end__": END})
builder.add_edge("tools", "agent")
graph = builder.compile(checkpointer=MemorySaver())
# ── MODULE 7: ENTRYPOINT ──────────────────────────────────────
if __name__ == "__main__":
config = {"configurable": {"thread_id": "agentic-session-001"}}
print("Agentic RAG ready. Ask anything.\n")
while True:
user_text = input("You: ").strip()
if not user_text or user_text.lower() in ("exit", "quit"):
break
response = graph.invoke(
{"messages": [HumanMessage(content=user_text)]},
config=config,
)
print(f"\nAgent: {response['messages'][-1].content}\n")
What pipelines can't do
What sets this apart from the rest is self-correction. A pipeline has no way to notice it fetched the wrong material; an agent does. When early results are thin, its next reasoning step rewrites the query with new terms. When a question hides a dependency, the agent spots it halfway through and fetches more. When the question is vague, it can ask the user before searching.
Reach for this architecture when a wrong answer is costly, as in compliance, legal, financial or medical settings, because verification steps, confidence thresholds and human-in-the-loop checkpoints can be added straight into the agent graph.
The price in latency
There is no hiding the slowdown. One-tool pipelines answer in roughly 200 to 500 milliseconds; an agent making three retrieval calls first can take 8 to 12 seconds, too slow for the main path of many real-time interfaces. Two patterns soften it: stream intermediate steps so users watch progress rather than wait in silence, and run agentic retrieval in the background to pre-fetch likely follow-ups.
Choosing and combining
Every retrieval problem has a right answer among these five. As a rough decision path: start with naive RAG for clean, factual document Q&A; move to hybrid when terminology is specialised or exact identifiers matter; reach for graph RAG when answers depend on relationships across documents; layer in advanced techniques when retrieval quality needs to improve; and adopt an agent when the task is open-ended, high-stakes or needs self-correction.
The architectures are also composable, so there's no need to pick one and discard the rest. The most common production pattern is a stack. In LangGraph that stack translates directly into the tool list. An agent with a naive tool, a hybrid tool, a graph tool and a decomposed-query tool is exactly the five-architecture stack, with the agent as the top layer selecting from and orchestrating the others on every turn.
A keyword reference card
This card lists the main classes and settings for each layer:
NAIVE RAG
SimpleDirectoryReader Load files into Document objects
VectorStoreIndex.from_documents Build the embed-and-store index
index.as_query_engine() Full retrieve-and-answer pipeline
index.as_retriever() Retrieve only (no answer generation)
Settings.chunk_size Token size per Node
Settings.chunk_overlap Token overlap between adjacent Nodes
HYBRID RAG
BM25Retriever Keyword-based sparse retriever
VectorIndexRetriever Dense embedding retriever
QueryFusionRetriever Merges multiple retrievers (RRF algorithm)
SentenceTransformerRerank Cross-encoder reranker for post-retrieval
GRAPH RAG
PropertyGraphIndex Builds a knowledge graph from documents
SimpleLLMPathExtractor LLM-based entity and relation extraction
ImplicitPathExtractor Heuristic-based entity extraction (fast)
ADVANCED RAG
SubQuestionQueryEngine Decomposes complex queries into sub-queries
HyDEQueryTransform Hypothetical Document Embedding transform
TransformQueryEngine Wraps any engine with a query transform
node_postprocessors Where rerankers and filters attach
AGENTIC RAG (LANGGRAPH LAYER)
@tool The bridge - every LlamaIndex engine becomes
a LangGraph tool through this decorator
ToolNode Executes whatever tool the agent selects
bind_tools() Gives the agent LLM its tool registry
MemorySaver / SqliteSaver Thread-level memory across turns (Part 2)
interrupt() Human approval checkpoint before retrieval (Part 3)
Retrieval as infrastructure
These five aren't five ways of doing the same thing. They're answers to five different retrieval problems, in a clean progression from simple to sophisticated.
- Naive RAG: quick, inexpensive, and the right fit for most plain document Q&A.
- Hybrid RAG: the sensible default whenever the vocabulary is specialised.
- Graph RAG: the pick once links between facts matter more than any single document.
- Advanced RAG is the pattern for lifting accuracy when the weak point is retrieval quality.
- Agentic RAG: for open-ended, high-stakes tasks where the system must reason on its own.
Put LlamaIndex on the data layer and LangGraph on orchestration, and these patterns meet nearly everything a production retrieval application will need.