Practical notes: RAG Without the Guesswork: A Standardized LangGraph +

This walkthrough rebuilds the path from raw materials to a working system for: RAG Without the Guesswork: A Standardized LangGraph + LlamaIndex Pattern.. The focus is operable steps, explicit checks, and code that you can drop into a repo without guessing intent. For Overview, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish.

Why This Article Exists

When working through Why This Article Exists, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline. Measure recall on a fixed question set before tuning prompts. Prompt churn rarely fixes a weak retrieval surface.

Part A: Understanding LlamaIndex (The Concept First)

When working through Part A: Understanding LlamaIndex (The Concept First), write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion. Measure recall on a fixed question set before tuning prompts. Prompt churn rarely fixes a weak retrieval surface.

What LlamaIndex Actually Does

When working through What LlamaIndex Actually Does, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments. Measure recall on a fixed question set before tuning prompts. Prompt churn rarely fixes a weak retrieval surface. When working through What LlamaIndex Actually Does, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish.

The Five-Stage RAG Pipeline

The Five-Stage RAG Pipeline works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline. Separate chunking policy from retrieval policy. Changing one should not force a rewrite of the other when quality metrics move.

1. LOAD       → Read raw files (PDF, Word, web pages, databases) into Documents
2. CHUNK      → Split Documents into small, retrievable Nodes
3. EMBED      → Convert each Node's text into a vector (a list of numbers
                representing meaning)
4. STORE      → Save those vectors in a Vector Index for fast lookup
5. RETRIEVE   → At query time, embed the user's question, find the most
                  similar Nodes, and return them as context

The Core Keywords You Need to Know

The Core Keywords You Need to Know works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion. Separate chunking policy from retrieval policy. Changing one should not force a rewrite of the other when quality metrics move.

Part B: Building a Standalone LlamaIndex Knowledge Base

Part B: Building a Standalone LlamaIndex Knowledge Base works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments. Separate chunking policy from retrieval policy. Changing one should not force a rewrite of the other when quality metrics move. Part B: Building a Standalone LlamaIndex Knowledge Base works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish.

Step 1: Installation

For Step 1: Installation, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline. Cite the passages that actually grounded the answer. Without citations, operators cannot tell hallucination from an indexing gap.

# Core package + OpenAI LLM and embedding integrations (the common starting setup)
pip install llama-index-core llama-index-llms-openai llama-index-embeddings-openai

# Readers for common file types (PDF, Word, etc.)
pip install llama-index-readers-file pypdf

Step 2: Global Configuration with Settings

For Step 2: Global Configuration with Settings, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion. Cite the passages that actually grounded the answer. Without citations, operators cannot tell hallucination from an indexing gap.

# ── llamaindex_config.py ─────────────────────────────────────
import os
from llama_index.core import Settings
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding

# Settings is global - configure once, used everywhere in LlamaIndex
Settings.llm = OpenAI(
    model="gpt-4o-mini",       # Used for generating final answers from retrieved context
    temperature=0.1,            # Low temperature: factual, not creative
)
Settings.embed_model = OpenAIEmbedding(
    model="text-embedding-3-small",   # Used to convert text into vectors
)
# Controls how documents are split into Nodes (chunks)
Settings.chunk_size = 512        # Max tokens per chunk
Settings.chunk_overlap = 50      # Overlap between consecutive chunks, to preserve context across boundaries

Step 3: Load → Index → Query (The Standalone Pipeline)

For Step 3: Load → Index → Query (The Standalone Pipeline), define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments. Cite the passages that actually grounded the answer. Without citations, operators cannot tell hallucination from an indexing gap.

# ── build_knowledge_base.py ──────────────────────────────────
from llama_index.core import SimpleDirectoryReader, VectorStoreIndex

# ── LOAD: Read all files in a folder into Document objects ──
documents = SimpleDirectoryReader("./data").load_data()
print(f"Loaded {len(documents)} documents")
# ── CHUNK + EMBED + STORE: all three happen inside this one call ──
# VectorStoreIndex automatically:
#   1. Splits each Document into Nodes (using Settings.chunk_size)
#   2. Embeds each Node (using Settings.embed_model)
#   3. Stores the vectors in an in-memory index
index = VectorStoreIndex.from_documents(documents, show_progress=True)
# ── RETRIEVE + GENERATE: ask a question ──────────────────────
query_engine = index.as_query_engine(
    similarity_top_k=3,   # Retrieve the 3 most relevant chunks for each query
)
response = query_engine.query("What is our refund policy for enterprise customers?")
print(response)

For Step 3: Load → Index → Query (The Standalone Pipeline), define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish.

Step 4: Persisting the Index (Don’t Re-Embed Every Time)

When working through Step 4: Persisting the Index (Don’t Re-Embed Every Time), write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline. Measure recall on a fixed question set before tuning prompts. Prompt churn rarely fixes a weak retrieval surface.

# ── Save the index after building it ─────────────────────────
index.storage_context.persist(persist_dir="./storage")

# ── Load it back later without re-embedding anything ──────────
from llama_index.core import StorageContext, load_index_from_storage
storage_context = StorageContext.from_defaults(persist_dir="./storage")
index = load_index_from_storage(storage_context)

Step 5: Using an External Vector Database (Chroma)

When working through Step 5: Using an External Vector Database (Chroma), write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion. Measure recall on a fixed question set before tuning prompts. Prompt churn rarely fixes a weak retrieval surface.

# ── Using Chroma as a persistent, production-grade vector store ──
# pip install llama-index-vector-stores-chroma chromadb

import chromadb
from llama_index.vector_stores.chroma import ChromaVectorStore
from llama_index.core import StorageContext, VectorStoreIndex
chroma_client = chromadb.PersistentClient(path="./chroma_db")
chroma_collection = chroma_client.get_or_create_collection("my_knowledge_base")
vector_store = ChromaVectorStore(chroma_collection=chroma_collection)
storage_context = StorageContext.from_defaults(vector_store=vector_store)
# Build the index directly into Chroma
index = VectorStoreIndex.from_documents(
    documents,
    storage_context=storage_context
)
# Later, in a different process, reconnect without re-indexing:
index = VectorStoreIndex.from_vector_store(vector_store=vector_store)

Part C: Connecting LlamaIndex to LangGraph

When working through Part C: Connecting LlamaIndex to LangGraph, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments. Measure recall on a fixed question set before tuning prompts. Prompt churn rarely fixes a weak retrieval surface. When working through Part C: Connecting LlamaIndex to LangGraph, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish.

The Bridge: Wrap the Query Engine as a Tool

The Bridge: Wrap the Query Engine as a Tool works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline. Separate chunking policy from retrieval policy. Changing one should not force a rewrite of the other when quality metrics move.

# ── MODULE 3: TOOLS (LlamaIndex-backed) ─────────────────────
from langchain_core.tools import tool

# The query_engine built in Part B - created once, at startup
# (In a real app, you'd load this from persisted storage, not rebuild it every time)
@tool
def search_knowledge_base(query: str) -> str:
    """Search the internal knowledge base for company policies, product
    documentation, and internal procedures. Use this whenever the user asks
    a question that might be answered by internal company documents rather
    than general knowledge.

    Args:
        query: A natural-language question to search for.

    Returns:
        A synthesized answer based on the most relevant retrieved documents.
    """
    response = query_engine.query(query)
    return str(response)

The Complete Integration: Modules 1 Through 7

The Complete Integration: Modules 1 Through 7 works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion. Separate chunking policy from retrieval policy. Changing one should not force a rewrite of the other when quality metrics move.

# ============================================================
# LANGGRAPH + LLAMAINDEX RAG AGENT — COMPLETE TEMPLATE
# Extends: Part 1 (core structure)
# ============================================================

# ── MODULE 1: IMPORTS & CONFIGURATION ───────────────────────
import os
from typing import Literal
# LangChain / LangGraph imports (the orchestration layer)
from langchain_openai import ChatOpenAI
from langchain_core.messages import HumanMessage, SystemMessage, BaseMessage
from langchain_core.tools import tool
from langgraph.graph import StateGraph, MessagesState, START, END
from langgraph.prebuilt import ToolNode
from langgraph.checkpoint.memory import MemorySaver
# LlamaIndex imports (the retrieval / data layer)
from llama_index.core import (1
    Settings, SimpleDirectoryReader, VectorStoreIndex,
    StorageContext, load_index_from_storage
)
from llama_index.llms.openai import OpenAI as LlamaOpenAI
from llama_index.embeddings.openai import OpenAIEmbedding
# LangGraph's chat model - used by the agent's reasoning
llm = ChatOpenAI(model="gpt-4o", temperature=0)
# LlamaIndex's model config - used internally by the query engine
# Note: these are SEPARATE from the LangGraph llm above. Each framework
# manages its own model instances; they don't share state.
Settings.llm = LlamaOpenAI(model="gpt-4o-mini", temperature=0.1)
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")
Settings.chunk_size = 512

# ── MODULE 2: STATE ──────────────────────────────────────────
class State(MessagesState):
    pass  # messages field inherited; extend if your agent needs more

# ── MODULE 3: TOOLS (RAG-backed) ─────────────────────────────
# Build or load the LlamaIndex knowledge base ONCE, at startup
PERSIST_DIR = "./storage"
if os.path.exists(PERSIST_DIR):
    # Reload existing index - no re-embedding, fast startup
    storage_context = StorageContext.from_defaults(persist_dir=PERSIST_DIR)
    index = load_index_from_storage(storage_context)
else:
    # First run - build the index and persist it
    documents = SimpleDirectoryReader("./data").load_data()
    index = VectorStoreIndex.from_documents(documents, show_progress=True)
    index.storage_context.persist(persist_dir=PERSIST_DIR)
query_engine = index.as_query_engine(similarity_top_k=3)

@tool
def search_knowledge_base(query: str) -> str:
    """Search internal company documents for policies, product specs,
    procedures, and other domain-specific information. Use this for any
    question that requires knowledge specific to this organization rather
    than general world knowledge."""
    response = query_engine.query(query)
    return str(response)

tools = [search_knowledge_base]
llm_with_tools = llm.bind_tools(tools)
tool_node = ToolNode(tools)

# ── MODULE 4: NODES ──────────────────────────────────────────
def agent_node(state: State) -> dict:
    """The reasoning node. Decides whether to answer directly or
    search the knowledge base first."""
    system_prompt = SystemMessage(content=(
        "You are a helpful assistant with access to an internal knowledge base. "
        "Use the search_knowledge_base tool when the user asks about company-specific "
        "information. For general questions, answer directly."
    ))
    messages = [system_prompt] + state["messages"]
    response = llm_with_tools.invoke(messages)
    return {"messages": [response]}

# ── MODULE 5: ROUTING ────────────────────────────────────────
def should_continue(state: State) -> Literal["tools", "__end__"]:
    last_message = state["messages"][-1]
    if hasattr(last_message, "tool_calls") and last_message.tool_calls:
        return "tools"
    return "__end__"

# ── MODULE 6: GRAPH ASSEMBLY ─────────────────────────────────
graph_builder = StateGraph(State)
graph_builder.add_node("agent", agent_node)
graph_builder.add_node("tools", tool_node)
graph_builder.add_edge(START, "agent")
graph_builder.add_conditional_edges(
    "agent", should_continue,
    {"tools": "tools", "__end__": END}
)
graph_builder.add_edge("tools", "agent")
graph = graph_builder.compile(checkpointer=MemorySaver())

# ── MODULE 7: ENTRYPOINT ──────────────────────────────────────
if __name__ == "__main__":
    config = {"configurable": {"thread_id": "session-001"}}

    print("RAG agent ready. Ask about your documents, or anything else.\n")

    while True:
        user_text = input("You: ").strip()
        if not user_text or user_text.lower() == "exit":
            break

        response = graph.invoke(
            {"messages": [HumanMessage(content=user_text)]},
            config=config
        )
        print(f"Agent: {response['messages'][-1].content}\n")

What Actually Happens When You Run This

What Actually Happens When You Run This works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments. Separate chunking policy from retrieval policy. Changing one should not force a rewrite of the other when quality metrics move. What Actually Happens When You Run This works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish.

User: "What's our policy on remote work?"
        ↓
[agent_node] — LangGraph's LLM reads the message, recognizes this needs
               internal info, decides to call search_knowledge_base
        ↓
[tools] — ToolNode executes search_knowledge_base("What's our policy on remote work?")
        ↓
        Inside the tool: query_engine.query(...) runs —
        this is 100% LlamaIndex, invisible to LangGraph:
          1. Embeds the query
          2. Searches the vector index for the 3 closest chunks
          3. Feeds those chunks + the question to Settings.llm
          4. Returns a synthesized answer string
        ↓
[agent_node] — LangGraph's LLM receives the tool's string result,
               and crafts the final response shown to the user
        ↓
Response to user

Part D: One Level Deeper — Retriever-Only Mode (More Control)

For Part D: One Level Deeper — Retriever-Only Mode (More Control), define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline. Cite the passages that actually grounded the answer. Without citations, operators cannot tell hallucination from an indexing gap.

# ── Retriever-only tool: returns raw chunks, not a synthesized answer ──
retriever = index.as_retriever(similarity_top_k=3)
@tool
def retrieve_documents(query: str) -> str:
    """Retrieve relevant document excerpts from the internal knowledge base.
    Returns raw excerpts for you to read and reason over yourself -
    use this when you need to cite specific sources or combine information
    from multiple documents."""

    nodes = retriever.retrieve(query)

    # Format each retrieved chunk with its source for transparency
    formatted_chunks = []
    for i, node in enumerate(nodes):
        source = node.metadata.get("file_name", "unknown source")
        formatted_chunks.append(f"[Excerpt {i+1} from {source}]\n{node.text}")

    return "\n\n---\n\n".join(formatted_chunks)

When to Use QueryEngine vs. Retriever

For When to Use QueryEngine vs. Retriever, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion. Cite the passages that actually grounded the answer. Without citations, operators cannot tell hallucination from an indexing gap.

The Updated Keyword Reference Card

For The Updated Keyword Reference Card, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments. Cite the passages that actually grounded the answer. Without citations, operators cannot tell hallucination from an indexing gap. For The Updated Keyword Reference Card, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish.

The Decision Guide: When Do You Actually Need This?

When working through The Decision Guide: When Do You Actually Need This?, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline. Measure recall on a fixed question set before tuning prompts. Prompt churn rarely fixes a weak retrieval surface.

Conclusion: Two Frameworks, One Seam

When working through Conclusion: Two Frameworks, One Seam, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion. Measure recall on a fixed question set before tuning prompts. Prompt churn rarely fixes a weak retrieval surface.

Operational checklist

Operational checklist works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope.

Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph.

Separate chunking policy from retrieval policy. Changing one should not force a rewrite of the other when quality metrics move.

Put human approval on edges that spend money or change production data. Compile-time wiring does not equal business completeness.

Write a short runbook: how to rotate keys, how to drain the queue, how to roll back the last ingest.

Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion.

Before promoting the stack, freeze versions, capture a golden transcript for the critical path, and confirm rollback steps. Shared environments need rate limits, tenancy checks, and a clear owner for secret rotation. Prefer boring reliability over clever one-off demos.

Batch note for bcaf14f9c811: keep provider keys out of the repo, set a per-session token ceiling, and store transcripts next to the eval fixtures so later model swaps stay comparable.