Vectorless RAG: How PageIndex Reads Documents Without Embeddings

Ask someone what happened in chapter 7 and they won't reread the whole book. They'll open the table of contents, jump to chapter 7 and read that part. It's quick, and it's accurate.

Most AI document search works nothing like that. It chops the text into arbitrary pieces, turns each piece into numbers, and matches the question against whichever numbers sit closest. Sometimes that's fine. Sometimes the answer straddles a chunk boundary and the system never sees it. A newer approach called vectorless RAG drops the numbers game and has the model read a document's structure the way a person would. It reportedly hit 98.7% on a tough financial benchmark where conventional pipelines barely reached 50%. What follows builds up to that result from the basics.

What RAG does

RAG stands for retrieval-augmented generation. In practice it means handing a model access to your own documents so its answers come from your data and not only from training.

That matters because models like GPT or Claude have a knowledge cutoff. They've never seen the internal reports, legal contracts or medical records of any particular organization. RAG fills the gap by fetching relevant passages and placing them in front of the model just before it answers. The promise is simple: ask, and get an answer grounded in your material. The reality gets messier.

The brute-force version

The most obvious design pastes the whole document into the prompt alongside the question and lets the model sort it out. No databases, no clever machinery.

from openai import OpenAI
import PyPDF2

client = OpenAI(api_key="your-api-key")

# Extract all text from a PDF
def extract_pdf_text(pdf_path):
    reader = PyPDF2.PdfReader(pdf_path)
    full_text = ""
    for page in reader.pages:
        full_text += page.extract_text() + "\n"
    return full_text

document_text = extract_pdf_text("annual_report.pdf")
user_question = "What was the total revenue in 2024?"

response = client.chat.completions.create(
    model="gpt-4.1",
    messages=[
        {"role": "system", "content": "Answer based on the provided document."},
        {"role": "user", "content": f"Document:\n{document_text}\n\nQuestion: {user_question}"}
    ]
)

print(response.choices[0].message.content)

That's fine for a five-page PDF. A 200-page annual report runs into three walls fast.

• Context limits. Models accept only so much text at once, and a 200-page filing can run to 300,000 tokens, past what even the largest models take.
• Quality loss. Even if everything fit, answers go vague. Asking a model to find one needle while it holds the entire haystack makes it lose focus.
• Cost. Providers bill per token, so sending 300,000 of them for each question, even one answered by a single page, adds up quickly.

RAG was invented to get around exactly those walls.

How vector RAG works

The idea is neat. Process the document once, store smart representations of it in a database, and at question time pull back only the relevant pieces. The pipeline has five stages:

1. Chunking. Cut the document into smaller pieces, perhaps every 500 words or one paragraph at a time.
2. Embedding. A specialized model converts each chunk into a list of numbers, a vector, that captures its meaning. Similar ideas land on similar numbers.
3. Storage. The vectors go into a purpose-built database such as ChromaDB, Pinecone or Weaviate. Milvus, Qdrant and the PGVector extension for PostgreSQL are other common options.
4. Search. The question is embedded too, and the database returns the chunks whose vectors lie closest. That's similarity search.
5. Generation. The retrieved chunks and the question go to the model together, so it works from focused context.

A complete working version using ChromaDB and OpenAI looks like this:

from openai import OpenAI
import chromadb
import PyPDF2

client = OpenAI(api_key="your-api-key")

# Step 1: Extract and chunk the document
def extract_and_chunk(pdf_path, chunk_size=500):
    reader = PyPDF2.PdfReader(pdf_path)
    full_text = ""
    for page in reader.pages:
        full_text += page.extract_text() + "\n"

    words = full_text.split()
    chunks = []
    for i in range(0, len(words), chunk_size):
        chunk = " ".join(words[i:i + chunk_size])
        chunks.append(chunk)
    return chunks

chunks = extract_and_chunk("annual_report.pdf")

# Step 2 and 3: Embed and store in ChromaDB
chroma_client = chromadb.Client()
collection = chroma_client.create_collection("my_documents")

collection.add(
    documents=chunks,
    ids=[f"chunk_{i}" for i in range(len(chunks))]
)

# Step 4: Search for relevant chunks
user_question = "What was the total revenue in 2024?"

results = collection.query(
    query_texts=[user_question],
    n_results=5
)

relevant_chunks = "\n\n".join(results["documents"][0])

# Step 5: Generate answer with focused context
response = client.chat.completions.create(
    model="gpt-4.1",
    messages=[
        {"role": "system", "content": "Answer based only on the provided context."},
        {"role": "user", "content": f"Context:\n{relevant_chunks}\n\nQuestion: {user_question}"}
    ]
)

print(response.choices[0].message.content)

Most companies run something close to it today. It copes well with straightforward questions about simple documents, and then it hits blind spots once documents get complicated.

Three ways it fails

Chunks that cut an idea in half

Imagine a report paragraph saying that Q3 operating income reached $4.2 billion, up 12% on the previous year, mostly because cloud services brought in $2.8 billion of recurring revenue (details in Appendix C). If a 500-word window happens to end in the middle of that sentence, one chunk keeps the figure and loses the explanation while the next keeps the explanation and loses the figure. Neither tells the whole story.

The failure is easy to reproduce:

paragraph = """The company's Q3 operating income was $4.2 billion,
representing a 12% increase over the prior year period. This growth
was primarily driven by the expansion of cloud services, which
contributed $2.8 billion in recurring revenue as detailed in the
segment breakdown in Appendix C."""

# Simulating a chunk boundary at word 20
words = paragraph.split()
chunk_1 = " ".join(words[:20])
chunk_2 = " ".join(words[20:])

print("Chunk 1:", chunk_1)
print("---")
print("Chunk 2:", chunk_2)

Ask what drove the increase in Q3 operating income, and the search may return the chunk about cloud services with no figures, or the chunk with the figures and no cause. Either way the answer is incomplete.

Splitting by paragraph doesn't fix it. Some paragraphs are two sentences long and others fill half a page, and three consecutive paragraphs may tell one story that only works read together.

References that go nowhere

Contracts and financial filings lean on pointers like Section 4.2.1 governs this or detailed tables are in Appendix G. A question about something on page 12 that refers to data on page 87 will likely land on page 12, because the keywords match, but nothing in vector search follows the pointer to page 87 where the figures live.

A quick simulation shows the problem:

# Simulating a cross-reference failure
documents = {
    "page_12": "The company's debt-to-equity ratio improved significantly in 2024. For a full breakdown of long-term obligations, see Appendix G on page 87.",
    "page_45": "Marketing expenses increased by 15% due to new campaigns.",
    "page_87": "Appendix G: Long-term debt stands at $12.4B. Senior notes: $8.1B. Credit facility: $4.3B. Maturity schedule: 2026-2034."
}

# Vector RAG would likely return page_12 for a debt question
# but miss page_87 where the actual numbers live
# because "debt-to-equity ratio" is more similar to the query
# than "Senior notes" and "Credit facility"

This is common in contracts, regulatory filings and academic papers. The answer often lives in two places at once, and a system that can't connect them will always come back incomplete.

Vocabulary that doesn't match

Vector similarity leans heavily on the user's wording resembling the document's. If a report says non-GAAP adjusted EBITDA and the user asks about adjusted earnings, the embeddings may not line up well enough to surface the right chunk. People don't know a document's exact terminology, and they shouldn't need to.

It gets worse in technical fields. A doctor searching for heart attack treatment protocols might miss chunks written as myocardial infarction management pathways. The meaning is identical, but the vectors can sit far enough apart that the right chunks never appear.

These aren't rare edge cases. On long, structured documents in production they happen routinely.

Enter vectorless RAG

In September 2025, a team at VectifyAI released a framework called PageIndex that takes a different route. It doesn't slice documents and match numbers. It builds a structured map of the document and lets an AI model reason its way around it.

The inspiration is how experts actually work. An analyst handed a 200-page report, who needs one figure, doesn't read every page. They check the table of contents, pick the section most likely to hold the answer, and read it closely. If it says see Appendix G, they turn to Appendix G.

PageIndex makes a model behave the same way. The key insight is that the most similar text isn't always the most relevant text. A vector database returns whatever looks most like the question, yet the answer sometimes sits in a section that looks nothing like it. What's needed is reasoning about where the answer should be, not pattern matching.

The system works in two steps.

Step one: a tree index

Instead of chunking and embedding, the document is handed to a reasoning model that outlines it as a tree. Think of a sharp intern who reads the whole report and drafts a multi-level table of contents, where each entry holds a title, a brief summary, a page range and links to its sub-entries. A financial report might come out like this:

Root: Annual Report 2024
  |-- Executive Summary (pages 1-3)
  |     Summary: "Overview of company performance, key metrics..."
  |-- Financial Statements (pages 15-45)
  |     |-- Income Statement (pages 15-20)
  |     |-- Balance Sheet (pages 21-30)
  |     +-- Cash Flow Statement (pages 31-45)
  |-- Risk Factors (pages 46-60)
  +-- Appendices (pages 80-120)
        |-- Appendix A: Segment Data (pages 80-95)
        +-- Appendix G: Detailed Tables (pages 96-120)

No vectors and no embeddings, only a logical structure matching how the document is organized. The model uses its reasoning to detect natural boundaries such as chapter breaks, topic shifts and new sections, instead of cutting at a fixed word count.

Step two: reasoning-based search

At query time the model is shown only the tree, never the whole document. It scans the titles and summaries, brings its own domain knowledge to bear, and chooses the nodes that look most likely to contain the answer.

Consider a question about the year-over-year change in operating margin. The model's reasoning goes roughly like this: margins are a financial metric, so the income statement matters, but a year-over-year comparison may also need earlier figures, perhaps in the appendix. It answers with the node IDs of both sections, the system pulls the original text for those two sections only, and the final answer is written from that narrow context.

A working demonstration of the concept needs nothing beyond plain OpenAI calls:

import json
from openai import OpenAI
import PyPDF2

client = OpenAI(api_key="your-api-key")

# Extract text page by page
def extract_pages(pdf_path):
    reader = PyPDF2.PdfReader(pdf_path)
    pages = {}
    for i, page in enumerate(reader.pages):
        pages[i + 1] = page.extract_text()
    return pages

pages = extract_pages("annual_report.pdf")
all_text = "\n".join([f"--- Page {k} ---\n{v}" for k, v in pages.items()])

# Step 1: Build the tree index using AI reasoning
tree_prompt = f"""You are a document analyst. Read this document and create
a hierarchical table of contents as a JSON tree. Each node should have:
- "title": section name
- "summary": 2-3 sentence description of what this section covers
- "pages": [start_page, end_page]
- "children": array of child nodes (or empty array)

Document:
{all_text[:80000]}

Return ONLY valid JSON. No other text."""

tree_response = client.chat.completions.create(
    model="gpt-4.1",
    messages=[{"role": "user", "content": tree_prompt}],
    response_format={"type": "json_object"}
)

document_tree = json.loads(tree_response.choices[0].message.content)
print("Document tree built successfully!")
print(json.dumps(document_tree, indent=2)[:500])

# Step 2: Reasoning-based search over the tree
user_question = "What was the year-over-year change in operating margin?"

search_prompt = f"""You are a retrieval expert. Given a user question and
a document's table of contents tree, identify which sections are most
likely to contain the answer.

Think step by step:
1. What kind of information does the question ask for?
2. Which sections would a human expert check first?
3. Are there sections that might cross-reference each other?

Document tree:
{json.dumps(document_tree, indent=2)}

Question: {user_question}

Return JSON with:
- "reasoning": your step-by-step thought process
- "relevant_pages": list of page numbers to retrieve"""

search_response = client.chat.completions.create(
    model="gpt-4.1",
    messages=[{"role": "user", "content": search_prompt}],
    response_format={"type": "json_object"}
)

search_result = json.loads(search_response.choices[0].message.content)
print("Reasoning:", search_result["reasoning"])

# Step 3: Fetch only relevant pages and generate answer
relevant_text = ""
for page_num in search_result["relevant_pages"]:
    if page_num in pages:
        relevant_text += f"\n--- Page {page_num} ---\n{pages[page_num]}"

answer_response = client.chat.completions.create(
    model="gpt-4.1",
    messages=[
        {"role": "system", "content": "Answer precisely based on the document context provided."},
        {"role": "user", "content": f"Context:{relevant_text}\n\nQuestion: {user_question}"}
    ]
)

print("\nAnswer:", answer_response.choices[0].message.content)

So the whole trick fits in a sentence: skip the vector store, skip embeddings, skip chunk-size tuning, and let the model study the outline, decide where the answer should live and fetch only those sections.

Using the PageIndex framework

The do-it-yourself version is good for learning. For real projects, PageIndex does the heavy lifting: it produces optimized trees, supports more advanced search strategies, and comes as a cloud service or a self-hosted option. Basic usage looks like this:

from pageindex import PageIndexClient
import time

# Initialize the client
pi_client = PageIndexClient(api_key="your-pageindex-api-key")

# Upload and process your document
result = pi_client.submit_document("./annual_report.pdf")
doc_id = result["doc_id"]

# Wait for processing to complete
while True:
    status = pi_client.get_document(doc_id)["status"]
    if status == "completed":
        print("Document processed!")
        break
    time.sleep(5)

# Inspect the generated tree structure
tree_result = pi_client.get_tree(doc_id)
if tree_result.get("status") == "completed":
    tree = tree_result["result"]
    for node in tree:
        print(f"[{node['node_id']}] {node['title']} - Page {node['page_index']}")

# Ask questions using the Chat API
response = pi_client.chat_completions(
    messages=[{"role": "user", "content": "What was total revenue in FY2024 vs FY2023?"}],
    doc_id=doc_id
)

print(response["choices"][0]["message"]["content"])

The generated tree has a shape like this:

{
  "title": "Financial Stability",
  "node_id": "0006",
  "page_index": 21,
  "text": "The Federal Reserve maintains financial stability...",
  "nodes": [
    {
      "title": "Monitoring Financial Vulnerabilities",
      "node_id": "0007",
      "page_index": 22,
      "text": "The Federal Reserve's monitoring focuses on..."
    },
    {
      "title": "Domestic and International Cooperation",
      "node_id": "0008",
      "page_index": 28,
      "text": "In 2023, the Federal Reserve collaborated..."
    }
  ]
}

Every node has a unique ID pointing back to the original document, a title, the text itself and child nodes for subsections. The model walks the tree by reading summaries and deciding which branches to open further.

Several documents can be compared in one query:

response = pi_client.chat_completions(
    messages=[{"role": "user", "content": "Compare the results across these two reports."}],
    doc_id=["pi-abc123def456", "pi-abc123ghi789"]
)

Developers building agents with LangChain, CrewAI or the OpenAI Agents SDK can connect PageIndex through the Model Context Protocol (MCP), which lets an agent call it as a retrieval tool without custom integration code.

The benchmark behind the claim

FinanceBench is the benchmark that brought vectorless RAG attention. Its questions come from genuine SEC filings (10-K and 10-Q reports) and expect precise answers: figures lifted from tables, calculations with several steps, and cross-references chased from one section to another. Nothing about it is a toy.

Conventional vector RAG scores roughly 30% to 50% on it. PageIndex, driven by its Mafin 2.5 reasoning engine, scored 98.7%, which is a different league and not a small gain. Three reasons keep coming up:

• Following cross-references. When a filing says see Appendix G, the reasoning model navigates the tree to Appendix G and pulls that data. A vector store has no idea it should follow the pointer.
• Keeping layout intact. A financial table carries headers, nested headers, footnotes and links between cells. Tree sections keep those together under clear boundaries, while fixed-size chunking pulls them apart.
• Multi-hop reasoning. Asking for the year-over-year change in operating margin requires numbers from two sections plus a calculation. The reasoning model works out that it needs both and retrieves them together, where similarity search would probably grab only the chunk closest to the question's wording.

One honest caveat: as of early 2026, the PageIndex team hadn't published detailed speed or throughput figures. Each query needs several sequential model calls for traversal and reasoning, so it's slower and costlier per query than a single vector lookup. For high-volume, low-latency uses, that tradeoff may not work.

Choosing between them

Vectorless RAG isn't a universal replacement for vector databases. It's a specialized tool.

Vector RAG fits when responses must arrive in under a second, when the corpus is many short documents like a FAQ base or collection of articles, when cost per query outweighs perfect accuracy, or when a consumer chatbot can get by with good-enough answers.

Vectorless RAG fits long, structured professional documents: annual reports, contracts, regulatory filings and technical manuals. It also suits cases where accuracy comes first, where queries must follow cross-references, or where a clear audit trail should show where each part of an answer came from.

A hybrid is worth considering for production. Lightweight metadata or keyword filters can narrow which documents are relevant, and then tree-based reasoning finds the exact sections inside them. That gives the speed of traditional search on the first pass and the accuracy of reasoning on the second.

The cost tradeoff is real. Reasoning models cost more per call than embedding models, and tree traversal means multiple sequential calls and higher latency. Speed and price are being exchanged for accuracy and explainability. In regulated fields like finance, law and healthcare, where wrong answers have consequences, that exchange usually pays off.

Where this is heading

The move from similarity-based to reasoning-based retrieval is young, but the direction is clear. PageIndex already supports a vision-based variant that works directly on PDF page images, with no text extraction or OCR. Page images go in, and the model reasons over what it sees. That helps with complex layouts, scanned pages and embedded charts, which text extraction handles poorly.

Work on agentic retrieval is also underway. In place of a single pass, the model keeps exploring: it follows references, compares sections, poses follow-up questions to itself and sharpens its answer across several reasoning rounds, which is the gap between skimming a document and studying it.

Anyone who wants to experiment can use the open-source PageIndex repository on GitHub, which has runnable Jupyter notebooks for both the standard and the vision-based pipelines. The project passed 20,000 stars after its September 2025 launch, and its Discord is active.

Vector databases will stay, and they handle plenty of workloads well. Yet applications that depend on a deep read of long, intricate documents are multiplying, and cutting text into fixed-size slices then hoping a similarity search surfaces the right slice is wearing thin there. Contracts, filings and technical specifications are the very documents a business values most, and they're where reasoning-based retrieval pays off most.

The sensible way to judge it is to run the do-it-yourself version on a document that's already well understood, then try PageIndex on the same file and see how the tree captures the organization chunking destroys. The best retrieval system isn't the one with the fanciest technology. It's the one that finds the right answer.

A path to try it

• Vector RAG, good for fundamentals: install chromadb and openai, run the ChromaDB example on a short document, then try a longer one and watch where answers get shaky.
• DIY vectorless, good for grasping the concept: run the plain OpenAI tree-and-reason example on the same document and compare answers with the vector version.
• Production vectorless: get a free PageIndex API key from the docs site, install the pageindex Python package and follow the SDK example. Upload a real document and ask questions whose answers are already known, which is the quickest way to judge fit.

All examples use Python 3.10 or later with the OpenAI SDK, ChromaDB and PyPDF2, installed through pip install openai chromadb PyPDF2 pageindex. An OpenAI API key is needed, plus a PageIndex key for the PageIndex examples.