LlamaIndex Plus LangGraph: A Clean Pattern for RAG Agents

The blind spot, and a division of labor

A LangGraph agent only knows what its LLM picked up in training. Questions about an internal pricing policy, last quarter's financial report or a manual finished last week get either an admission of ignorance or, worse, a confident fabrication.

Retrieval-Augmented Generation (RAG) closes that gap. At query time it retrieves relevant snippets from your own documents and passes them to the LLM as context before it answers, so the model reads instead of guessing. RAG is a capability added to a single agent. It isn't a new orchestration pattern, so supervisors, subgraphs and Send stay out of this picture. All that's assumed is some familiarity with a LangGraph agent's structure and how to give it a tool.

Building the pipeline by hand inside LangGraph is possible: write your own chunker, embedding calls, vector index and retrieval code. Many teams do, and it's plenty of plumbing for something that's already a solved problem. LlamaIndex was made to absorb that work. It's easily the most mature and popular option for an AI application's data side, which covers loading, chunking, embedding, indexing and retrieval. The orchestration side, meaning state, control flow and multi-step reasoning, belongs to LangGraph.

From that division comes the layout most production RAG systems use: LlamaIndex creates and serves the knowledge base. LangGraph treats it as a tool and chooses when to invoke it.

A mental model for LlamaIndex

Think of LlamaIndex as a librarian for your LLM. The LLM resembles a brilliant research assistant who has read much of the internet but never opened your company's files, and nobody can ask them to memorize a 10,000-page binder. So the librarian steps in: it reads every document, breaks it down into small searchable index cards (chunks), files those cards where topic lookup is fast (a vector index), and fetches only the 3 to 5 most relevant cards whenever the assistant asks something. The assistant never memorizes anything. A few snippets arrive and the reasoning happens on them.

The five-stage pipeline

Every RAG system, whatever the framework, goes through the same five stages, and LlamaIndex has a clean abstraction for each.

1. LOAD       → Read raw files (PDF, Word, web pages, databases) into Documents
2. CHUNK      → Split Documents into small, retrievable Nodes
3. EMBED      → Convert each Node's text into a vector (a list of numbers
                representing meaning)
4. STORE      → Save those vectors in a Vector Index for fast lookup
5. RETRIEVE   → At query time, embed the user's question, find the most
                  similar Nodes, and return them as context

The first four stages run once, when the knowledge base is built, which is called indexing. The fifth runs whenever a user asks something, which is called querying. Indexing time versus query time is probably the most important mental model in RAG, and it's worth holding onto.

The vocabulary

• Document is a raw piece of source content, such as one PDF, one web page or one database row, plus its metadata. It's the unprocessed, full-length unit.
• Node is a chunk cut from a Document. LLMs and embedding models work better on focused, bite-sized text than on whole documents, so LlamaIndex splits each Document into many Node objects. Nodes carry metadata back to their parent, so any chunk can be traced to its source.
• SimpleDirectoryReader is the usual loader: point it at a folder and it reads PDFs, text, Word files and other formats, producing one or more Document objects per file.
• VectorStoreIndex is the main indexing class. Given Document or Node objects, it embeds them and assembles a searchable vector index, either in memory or inside an external vector database when one is set up.
• Settings is a global configuration object. The LLM and embedding model are set on it once, and every component picks them up, so they needn't be passed into each call.
• QueryEngine is what you actually ask questions of. It comes from .as_query_engine() on an index, and a single call embeds the query, runs similarity search and feeds the retrieved context to the LLM for a final answer.
• Retriever comes from .as_retriever() on an index and handles only the search step. It returns the raw matching Node objects and generates no answer, which suits cases where retrieval and generation should be separate, controllable steps. That's exactly the situation once LangGraph enters.

The pipeline gets built on its own first, with no LangGraph.

A working pipeline in about ten lines

Installation comes first.

# Core package + OpenAI LLM and embedding integrations (the common starting setup)
pip install llama-index-core llama-index-llms-openai llama-index-embeddings-openai

# Readers for common file types (PDF, Word, etc.)
pip install llama-index-readers-file pypdf

LlamaIndex ships as many small packages on purpose. llama-index-core carries the framework logic, while the choice of LLM, embedding model, vector database and file reader each lives in a separate package that's installed only if needed. That keeps installations light.

A small configuration file sets the global defaults.

# ── llamaindex_config.py ─────────────────────────────────────
import os
from llama_index.core import Settings
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding

# Settings is global - configure once, used everywhere in LlamaIndex
Settings.llm = OpenAI(
    model="gpt-4o-mini",       # Used for generating final answers from retrieved context
    temperature=0.1,            # Low temperature: factual, not creative
)
Settings.embed_model = OpenAIEmbedding(
    model="text-embedding-3-small",   # Used to convert text into vectors
)
# Controls how documents are split into Nodes (chunks)
Settings.chunk_size = 512        # Max tokens per chunk
Settings.chunk_overlap = 50      # Overlap between consecutive chunks, to preserve context across boundaries

On chunk_size: oversized chunks hold so much text that retrieval gets fuzzy, because a fetched chunk might be only 10% relevant. Undersized chunks drop the surrounding context the LLM needs to make sense of a fact. Starting at 512 tokens with a 50-token overlap is common and sensible, then adjusting for the documents involved.

Then the knowledge base itself:

# ── build_knowledge_base.py ──────────────────────────────────
from llama_index.core import SimpleDirectoryReader, VectorStoreIndex

# ── LOAD: Read all files in a folder into Document objects ──
documents = SimpleDirectoryReader("./data").load_data()
print(f"Loaded {len(documents)} documents")
# ── CHUNK + EMBED + STORE: all three happen inside this one call ──
# VectorStoreIndex automatically:
#   1. Splits each Document into Nodes (using Settings.chunk_size)
#   2. Embeds each Node (using Settings.embed_model)
#   3. Stores the vectors in an in-memory index
index = VectorStoreIndex.from_documents(documents, show_progress=True)
# ── RETRIEVE + GENERATE: ask a question ──────────────────────
query_engine = index.as_query_engine(
    similarity_top_k=3,   # Retrieve the 3 most relevant chunks for each query
)
response = query_engine.query("What is our refund policy for enterprise customers?")
print(response)

Roughly ten lines produce a working RAG system, which is the framework's "hello world" and a decent case for why it exists. Doing it by hand would mean a custom splitter, a loop calling the embedding model, a nearest-neighbor lookup and a prompt template.

Persisting the index

Every chunk goes through the embedding model while the index is built, which costs money and time, so rebuilding at every app start is wasteful. Saving the index to disk and loading it later avoids that.

# ── Save the index after building it ─────────────────────────
index.storage_context.persist(persist_dir="./storage")

# ── Load it back later without re-embedding anything ──────────
from llama_index.core import StorageContext, load_index_from_storage
storage_context = StorageContext.from_defaults(persist_dir="./storage")
index = load_index_from_storage(storage_context)

storage_context.persist() stores the vector data, node metadata and document store in a local folder, and load_index_from_storage() recreates the identical index from those files with no re-embedding and no API calls. It follows the same build-once, persist, reload-cheaply idea as a LangGraph checkpointer.

A real vector database

An in-memory index is fine for prototypes. Production usually calls for a dedicated vector database designed for millions of vectors, concurrent reads and metadata filtering at scale. Chroma is the example here.

# ── Using Chroma as a persistent, production-grade vector store ──
# pip install llama-index-vector-stores-chroma chromadb

import chromadb
from llama_index.vector_stores.chroma import ChromaVectorStore
from llama_index.core import StorageContext, VectorStoreIndex
chroma_client = chromadb.PersistentClient(path="./chroma_db")
chroma_collection = chroma_client.get_or_create_collection("my_knowledge_base")
vector_store = ChromaVectorStore(chroma_collection=chroma_collection)
storage_context = StorageContext.from_defaults(vector_store=vector_store)
# Build the index directly into Chroma
index = VectorStoreIndex.from_documents(
    documents,
    storage_context=storage_context
)
# Later, in a different process, reconnect without re-indexing:
index = VectorStoreIndex.from_vector_store(vector_store=vector_store)

The swap resembles checkpointers. Exchange a single object (the in-memory store for ChromaVectorStore, just as MemorySaver gives way to SqliteSaver) and the query engine and retrieval logic downstream stay as they were. Pinecone, Qdrant, Weaviate and most other vector databases LlamaIndex supports work the same way.

Bridging the two frameworks

Here's the heart of it. The frameworks speak different languages: a LlamaIndex QueryEngine isn't a LangGraph tool, because a LangGraph agent expects tools built with LangChain's @tool decorator.

The bridge is simple. Define a regular Python function, put @tool on it, and have its body call query_engine.query(...). That's all the integration there is. LangGraph needs to know nothing about LlamaIndex internals, because it sees a tool taking a string and returning a string, like every other tool.

# ── MODULE 3: TOOLS (LlamaIndex-backed) ─────────────────────
from langchain_core.tools import tool

# The query_engine built in Part B - created once, at startup
# (In a real app, you'd load this from persisted storage, not rebuild it every time)
@tool
def search_knowledge_base(query: str) -> str:
    """Search the internal knowledge base for company policies, product
    documentation, and internal procedures. Use this whenever the user asks
    a question that might be answered by internal company documents rather
    than general knowledge.

    Args:
        query: A natural-language question to search for.

    Returns:
        A synthesized answer based on the most relevant retrieved documents.
    """
    response = query_engine.query(query)
    return str(response)

The docstring matters just as much as it does for any tool. The agent's LLM reads it to decide when to call this tool, as opposed to answering from its own knowledge or calling another tool.

The complete agent

Here's the full agent template with a LlamaIndex-backed RAG tool slotted into the tools module. Nothing else changes.

# ============================================================
# LANGGRAPH + LLAMAINDEX RAG AGENT — COMPLETE TEMPLATE
# Extends: Part 1 (core structure)
# ============================================================

# ── MODULE 1: IMPORTS & CONFIGURATION ───────────────────────
import os
from typing import Literal
# LangChain / LangGraph imports (the orchestration layer)
from langchain_openai import ChatOpenAI
from langchain_core.messages import HumanMessage, SystemMessage, BaseMessage
from langchain_core.tools import tool
from langgraph.graph import StateGraph, MessagesState, START, END
from langgraph.prebuilt import ToolNode
from langgraph.checkpoint.memory import MemorySaver
# LlamaIndex imports (the retrieval / data layer)
from llama_index.core import (1
    Settings, SimpleDirectoryReader, VectorStoreIndex,
    StorageContext, load_index_from_storage
)
from llama_index.llms.openai import OpenAI as LlamaOpenAI
from llama_index.embeddings.openai import OpenAIEmbedding
# LangGraph's chat model - used by the agent's reasoning
llm = ChatOpenAI(model="gpt-4o", temperature=0)
# LlamaIndex's model config - used internally by the query engine
# Note: these are SEPARATE from the LangGraph llm above. Each framework
# manages its own model instances; they don't share state.
Settings.llm = LlamaOpenAI(model="gpt-4o-mini", temperature=0.1)
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")
Settings.chunk_size = 512

# ── MODULE 2: STATE ──────────────────────────────────────────
class State(MessagesState):
    pass  # messages field inherited; extend if your agent needs more

# ── MODULE 3: TOOLS (RAG-backed) ─────────────────────────────
# Build or load the LlamaIndex knowledge base ONCE, at startup
PERSIST_DIR = "./storage"
if os.path.exists(PERSIST_DIR):
    # Reload existing index - no re-embedding, fast startup
    storage_context = StorageContext.from_defaults(persist_dir=PERSIST_DIR)
    index = load_index_from_storage(storage_context)
else:
    # First run - build the index and persist it
    documents = SimpleDirectoryReader("./data").load_data()
    index = VectorStoreIndex.from_documents(documents, show_progress=True)
    index.storage_context.persist(persist_dir=PERSIST_DIR)
query_engine = index.as_query_engine(similarity_top_k=3)

@tool
def search_knowledge_base(query: str) -> str:
    """Search internal company documents for policies, product specs,
    procedures, and other domain-specific information. Use this for any
    question that requires knowledge specific to this organization rather
    than general world knowledge."""
    response = query_engine.query(query)
    return str(response)

tools = [search_knowledge_base]
llm_with_tools = llm.bind_tools(tools)
tool_node = ToolNode(tools)

# ── MODULE 4: NODES ──────────────────────────────────────────
def agent_node(state: State) -> dict:
    """The reasoning node. Decides whether to answer directly or
    search the knowledge base first."""
    system_prompt = SystemMessage(content=(
        "You are a helpful assistant with access to an internal knowledge base. "
        "Use the search_knowledge_base tool when the user asks about company-specific "
        "information. For general questions, answer directly."
    ))
    messages = [system_prompt] + state["messages"]
    response = llm_with_tools.invoke(messages)
    return {"messages": [response]}

# ── MODULE 5: ROUTING ────────────────────────────────────────
def should_continue(state: State) -> Literal["tools", "__end__"]:
    last_message = state["messages"][-1]
    if hasattr(last_message, "tool_calls") and last_message.tool_calls:
        return "tools"
    return "__end__"

# ── MODULE 6: GRAPH ASSEMBLY ─────────────────────────────────
graph_builder = StateGraph(State)
graph_builder.add_node("agent", agent_node)
graph_builder.add_node("tools", tool_node)
graph_builder.add_edge(START, "agent")
graph_builder.add_conditional_edges(
    "agent", should_continue,
    {"tools": "tools", "__end__": END}
)
graph_builder.add_edge("tools", "agent")
graph = graph_builder.compile(checkpointer=MemorySaver())

# ── MODULE 7: ENTRYPOINT ──────────────────────────────────────
if __name__ == "__main__":
    config = {"configurable": {"thread_id": "session-001"}}

    print("RAG agent ready. Ask about your documents, or anything else.\n")

    while True:
        user_text = input("You: ").strip()
        if not user_text or user_text.lower() == "exit":
            break

        response = graph.invoke(
            {"messages": [HumanMessage(content=user_text)]},
            config=config
        )
        print(f"Agent: {response['messages'][-1].content}\n")

What happens on one query

Tracing a single request makes the division of labor clear.

User: "What's our policy on remote work?"
        ↓
[agent_node] — LangGraph's LLM reads the message, recognizes this needs
               internal info, decides to call search_knowledge_base
        ↓
[tools] — ToolNode executes search_knowledge_base("What's our policy on remote work?")
        ↓
        Inside the tool: query_engine.query(...) runs —
        this is 100% LlamaIndex, invisible to LangGraph:
          1. Embeds the query
          2. Searches the vector index for the 3 closest chunks
          3. Feeds those chunks + the question to Settings.llm
          4. Returns a synthesized answer string
        ↓
[agent_node] — LangGraph's LLM receives the tool's string result,
               and crafts the final response shown to the user
        ↓
Response to user

The boundary is tidy. LangGraph never encounters a Node, an embedding or a vector, only a tool that returns text, no different from search_web or calculate. LlamaIndex never encounters messages, state or a thread_id, just a query string coming in and an answer going out. Each framework sticks to its strength, and one function call forms the seam.

Retriever-only mode for more control

The query_engine.query() pattern is the simplest integration and fits most cases. Sometimes more control helps, for instance when the LangGraph agent's own LLM should write the final answer, using the existing system prompt and conversation context, and not LlamaIndex's internal Settings.llm, which generates a separate, disconnected answer.

In that case, use a Retriever in place of a QueryEngine. It performs only the search step and returns raw chunks, and what happens next is up to the agent.

# ── Retriever-only tool: returns raw chunks, not a synthesized answer ──
retriever = index.as_retriever(similarity_top_k=3)
@tool
def retrieve_documents(query: str) -> str:
    """Retrieve relevant document excerpts from the internal knowledge base.
    Returns raw excerpts for you to read and reason over yourself -
    use this when you need to cite specific sources or combine information
    from multiple documents."""

    nodes = retriever.retrieve(query)

    # Format each retrieved chunk with its source for transparency
    formatted_chunks = []
    for i, node in enumerate(nodes):
        source = node.metadata.get("file_name", "unknown source")
        formatted_chunks.append(f"[Excerpt {i+1} from {source}]\n{node.text}")

    return "\n\n---\n\n".join(formatted_chunks)

With this setup, the agent_node's own LLM call, using llm_with_tools plus the full conversation history and system prompt, reads the raw excerpts and writes the answer. That has real payoffs: the agent can name the document a fact came from, it can mix retrieved excerpts with earlier conversation, and users hear one consistent voice and not two LLMs in different styles.

Which one to pick

Choose QueryEngine for the simplest integration when it's fine for LlamaIndex's LLM to produce a retrieval-based answer as a self-contained string. That suits most chatbots and assistants.

Choose Retriever when source citations are needed, when the agent's main LLM should keep a steady voice and reasoning style across retrieved and non-retrieved answers, or when retrieved chunks must be combined with other tool results before the final answer.

Keyword reference

Loading and indexing. Document is a loaded source, one file or one page. Node is a chunk of it, the retrievable unit, with metadata pointing back to its origin. SimpleDirectoryReader("./path").load_data() loads every supported file in a folder as Document objects. Settings is the global configuration singleton, and Settings.llm, Settings.embed_model and Settings.chunk_size get set once for all components. VectorStoreIndex.from_documents(documents) chunks, embeds and stores in one call.

Querying. index.as_query_engine(similarity_top_k=3) yields a QueryEngine that fetches the top-k chunks and also writes a synthesized answer. index.as_retriever(similarity_top_k=3) yields a Retriever that only fetches the top-k chunks, which suits setups where your own LLM does the synthesis. query_engine.query("...") executes the complete retrieve-then-generate flow and returns a Response object, and str(response) extracts the text. retriever.retrieve("...") runs retrieval only and returns Node objects exposing .text and .metadata.

Persistence. index.storage_context.persist(persist_dir="./storage") saves the index so restarts needn't re-embed. StorageContext.from_defaults(persist_dir="./storage") combined with load_index_from_storage(...) reloads it. ChromaVectorStore, PineconeVectorStore and QdrantVectorStore are external backends that replace the default in-memory store, in the same spirit as LangGraph's checkpointers.

The bridge. A @tool function whose body calls query_engine.query(...) is the whole integration: LangGraph sees an ordinary tool and LlamaIndex stays encapsulated. Two separate LLM configurations exist as well. Settings.llm is LlamaIndex's internal model, used only within the query engine, and the agent's LangGraph llm handles main reasoning. They run independently and share no history or state.

Do you need this?

• If the agent needs only general knowledge, math or live web search, LlamaIndex isn't required, and the basic template with standard tools is enough.
• If it must answer from a fixed set of documents such as policies, manuals or reports that rarely change, wrapping a QueryEngine as a tool is exactly right.
• If it has to cite sources, or combine retrieved facts with reasoning across a multi-turn conversation, use the Retriever pattern.
• If the knowledge base is large (100,000+ documents), updates often or must serve many concurrent users, use an external vector database such as Chroma, Pinecone or Qdrant in place of the in-memory index.
• If retrieval itself has to be reviewable and pausable, say when compliance demands human approval of what gets searched, pair this with LangGraph's interrupt() and put the retrieval call in a node that waits for approval, as with any sensitive tool.

Two frameworks and one seam

The lesson underneath it all is that giving an agent real-world knowledge isn't a LangGraph problem. It's a data problem, and a data framework is the right tool for it. A complex multi-agent system is really several single-agent graphs composed together, and knowledge access follows the same modular spirit.

LlamaIndex handles indexing and retrieval, and LangGraph handles reasoning and orchestration. The whole connection is one @tool-decorated function. That's not a limitation but the right amount of coupling, since each framework stays in its lane and either side can be upgraded independently. The vector database can be swapped without touching the graph, and a supervisor pattern can be added without touching retrieval logic.