Building a RAG Documentation Assistant for an API With LlamaIndex
Build an API documentation assistant with LlamaIndex: load Markdown docs, chunk with SentenceSplitter, embed, store in Chroma and query with a RAG QueryEngine.
This article is published in English.
Sources: Building a RAG-Powered API Documentation Assistant with LlamaIndex (Part 1: Components)
You're wiring up a new payments API. You open the docs, press Ctrl+F, type "refund" and get twelve hits spread over four pages, half of them irrelevant because they merely mention the word. A quarter of an hour later you've found the endpoint, and you've lost the thread of what you were doing.
Now picture typing "How do I issue a refund?" and getting a clear answer drawn straight from the documentation. That's the goal here: a small assistant for a made-up e-commerce API called ShopSphere. It doesn't lean on the language model's general knowledge or invent an answer. It searches the real ShopSphere docs and answers from them. LlamaIndex does the heavy lifting, and each piece is explained along the way, including the spots where the framework can feel like magic.
The problem being solved
Language models know a lot about the world but nothing about your company's API reference. Pasting hundreds of pages into every prompt would be slow, costly and noisy. A better plan is to find the relevant passages first and then ask the model to answer from them. Someone asks how to create an order, the system locates the section on creating orders, and that text goes to the model together with the question.
The technique is Retrieval-Augmented Generation (RAG): retrieve, then generate. Everything below is about building the retrieval half and eventually letting an agent choose which information or tool to use.
Four ideas in LlamaIndex
The framework organizes everything around four concepts, and once they click it stops looking like a maze of class names. A small restaurant kitchen makes a handy comparison:
- Components are ingredients and appliances: the oven, the fridge, the flour. They're raw capabilities.
- Tools are recipes taped to the appliances, written so a new cook could follow them without asking questions.
- Agents are the chef, deciding order by order which recipe to use and in what sequence, based on what the customer wanted.
- Workflows are the printed recipe card, used when you don't want improvisation and need a fixed sequence followed every time.
The assistant gets built by stacking these four in that order.
Components: turning Markdown docs into a searchable knowledge base
This is the prep stage. Before any reasoning happens, five documentation files (authentication, products, orders, errors and webhooks) need to be in a form a computer can search by meaning and not only by keyword. It takes five steps.
Step 1: loading
from llama_index.core import SimpleDirectoryReader
def load_documents():
reader = SimpleDirectoryReader(input_dir="data", required_exts=[".md"])
documents = reader.load_data()
return documents
The reader opens every Markdown file in the data/ folder and wraps each one in a Document, LlamaIndex's representation of a text plus information about its origin. The origin part is more useful than it looks. The original filename is kept as metadata, so when the assistant later answers, it can say the information came from something like 03-cart-orders.md without any manual bookkeeping.
Step 2: chunking
One problem: 03-cart-orders.md covers carts, checkout, orders, refunds and cancellations all in one file. Treated as a single searchable unit, it could answer a refund question with unrelated cancellation details. So each document is cut into smaller pieces that LlamaIndex calls nodes, much like turning a book into a stack of index cards that each remember their book.
from llama_index.core.node_parser import SentenceSplitter
splitter = SentenceSplitter(chunk_size=400, chunk_overlap=40)
Cutting blindly every 400 characters could split a sentence in half or, worse, slice through a curl command. SentenceSplitter looks for natural sentence boundaries instead. A small overlap of 40 characters also repeats the tail of one chunk at the start of the next, which keeps related material together, such as a code sample and the sentence explaining it.
Step 3: embedding
Searching by meaning depends on this step. Each chunk goes through an embedding model, which maps the text to a long list of numbers, known as a vector. Passages that mean similar things land on similar vectors, even when the words used are completely different.
A user might ask, "How do I get my money back?" while the docs have a section titled "Issue a Refund." The two phrases share no important words but express the same idea. The embedding model captures that link, so semantic search can surface the refund documentation.
from llama_index.embeddings.huggingface import HuggingFaceInferenceAPIEmbedding
embed_model = HuggingFaceEmbedding(
model_name="BAAI/bge-small-en-v1.5",
)
Step 4: storing the embeddings
Embedding every chunk takes time and may cost money through API calls, and nobody wants to repeat it at each startup. So the vectors go into a vector database that can be reused. Here that's Chroma, a free local database that makes persisting and searching embeddings easy.
import chromadb
from llama_index.vector_stores.chroma import ChromaVectorStore
db = chromadb.PersistentClient(path="./chroma_db")
chroma_collection = db.get_or_create_collection("shopsphere_api_docs")
vector_store = ChromaVectorStore(chroma_collection=chroma_collection)
LlamaIndex can fold steps 2 to 4 into a single process, the IngestionPipeline. Define the stages once, and every document passes through them automatically, being split, embedded and then written to Chroma.
from llama_index.core.ingestion import IngestionPipeline
pipeline = IngestionPipeline(
transformations=[
SentenceSplitter(chunk_size=400, chunk_overlap=20),
embed_model
],
vector_store=vector_store,
)
Run it once and a durable, searchable index sits in the chroma_db folder. It only needs redoing when the docs change.
Step 5: querying
Now comes the payoff. The stored vectors are wrapped in a VectorStoreIndex, which becomes a QueryEngine, the object you actually put questions to:
from llama_index.core import VectorStoreIndex
index = VectorStoreIndex.from_vector_store(vector_store, embed_model=embed_model)
query_engine = index.as_query_engine(llm=llm, similarity_top_k=3)
query_engine.query("How do I create a new order?")
Three things happen when a question arrives:
- The question becomes a vector, using the same embedding model as ingestion. That matters because question and chunks must live in the same vector space. It's like measuring everything in one unit, since mixing inches and centimeters makes comparison meaningless.
- The most similar chunks are retrieved from the vector store. Here that means the three chunks whose meaning sits closest to the question.
- Those chunks go to the LLM as context, and the model writes its answer from them instead of relying only on training data.
That's a complete RAG pipeline, and on its own it's already handy. It can only do one thing, though: retrieve, then answer. It has no way to conclude that a particular question calls for a different lookup. That takes tools and an agent.
Running it yourself
The finished project can be cloned and run locally. Set up a virtual environment, install the requirements and paste a Hugging Face token into the .env file:
git clone https://github.com/H11199/API-Documentation-Agent.git
cd API-Documentation-Agent
python3 -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\Activate.ps1
pip install -r requirements.txt
cp .env.example .env # then paste your HF token in as HF_TOKEN=hf_...
python ingestion_pipeline.py # builds the index from the sample docs in data/
python app.py # chat with the assistant
What comes next
At this point there's a working search engine for the documentation, but it only retrieves and answers. The next stage gives the assistant tools and lets an agent decide which to use. It also uncovers a part of LlamaIndex Workflows that feels like magic at first: how a workflow knows which step should run next.