Reasoning Over a Document Tree Instead of Searching Vectors

Few things are as embarrassing as a model confidently quoting the wrong section of a 200-page annual report in the middle of a live demo. A familiar sequel follows: the team "fixes" the cause with better chunking, smarter overlap and a more carefully tuned embedding model, the usual upgrade cycle every RAG engineer goes through, convinced that the next tweak will finally work. Then the system answers a question about 2023 revenue projections with a paragraph from the 2021 capital expenditure plan. The two sections used similar financial language, so the embeddings judged them related. They weren't.

That kind of failure raises a fair question about whether the whole approach is broken at its foundation.

Why similarity falls short

Every RAG tutorial teaches the same recipe. Cut the document into 500-word pieces, embed them, find the nearest vectors for each question and give those chunks to an LLM. It sounds elegant, semantic search paired with language generation.

What the recipe leaves out is that similarity isn't relevance. A vector database finds text that's semantically close to the question and not text that answers it. A paragraph on a drug's risk factors and one on its efficacy sit near each other in embedding space, yet only one of them addresses a question about the company's financial risks.

Chunking creates a second problem. Cut a 200-page legal document into 500-token windows and everything its author built with care disappears: sections, cross-references, logical flow. What remains is a heap of fragments with no sense of where they came from or how they relate. Better chunking eases it a little but never fixes the root issue, since structure that has been discarded can't be recovered.

A human analyst would never read like that. They'd skim the table of contents, jump to the right section, read it knowing the chapter around it and check the footnotes. They navigate the document as a document.

A tree instead of vectors

The VectifyAI team asked what a RAG system would look like if it worked that way, and built PageIndex. Once spoken aloud, the key idea seems almost uncomfortably obvious. Skip turning documents into vectors and searching by similarity. Instead, construct a hierarchical tree index of the document, in effect a machine-readable table of contents, and let an LLM reason along that tree toward the relevant material.

That means no vector database, no chunking and no similarity scores, only structured indexing and intelligent navigation.

Step one: building the index

PageIndex doesn't fragment a document it's given. It analyzes the natural structure, meaning sections, subsections, headings and logical groupings, and produces a hierarchy resembling how an expert would organize the content. Picture an automatically generated table of contents, only richer. Every node holds more than a title: a summary of what its branch covers, the questions it could plausibly answer and its relation to sibling and parent nodes.

A 100-page technical specification could produce a tree three or four levels deep, with major functional areas at the top and narrower topics in the sub-nodes beneath. The document's logical hierarchy stays as its author arranged it. The step runs once per document, upfront, and the index gets stored while the original content remains intact and reachable.

Step two: reasoning through the tree

When a query arrives, PageIndex doesn't embed it or run a nearest-neighbor search. An LLM navigates the index tree instead. It reads the top-level nodes and reasons about which branch most likely holds the answer, follows that branch, reads the next level and reasons again. It keeps descending until it has pinpointed the relevant sections.

That's the part that changes things. It then fetches those complete sections from the original document, with no fragmentation, and uses them as context for the answer.

The retrieval is explainable by design. It's possible to see exactly which nodes the LLM traversed and which pages of the original it landed on. There's no black-box cosine score, and the reasoning chain is visible.

Trying it

The setup is small. Start by cloning the repository and installing its dependencies:

git clone https://github.com/VectifyAI/PageIndex.git
cd PageIndex
pip3 install --upgrade -r requirements.txt

Then create a .env file in the root directory holding the API key:

CHATGPT_API_KEY=your_openai_api_key_here

To index a PDF, point the script at it. A financial report makes a good example, since that's where the tool does well:

python3 run_pageindex.py --pdf_path /path/to/annual_report.pdf

Markdown documents use a different flag:

python3 run_pageindex.py --md_path /path/to/technical_spec.md

That completes the setup. No vector database to configure, no embedding model to pick, no chunking parameters to agonize over.

Processing a document produces an index structure along these lines, simplified here:

{
  "document": "Apple Inc. Annual Report 2023",
  "index": {
    "title": "Apple Inc. Annual Report 2023",
    "summary": "Comprehensive financial and operational report covering revenue, product segments, risks, and strategic outlook",
    "children": [
      {
        "title": "Business Overview",
        "summary": "Company description, product lines, and market position",
        "pages": [1, 8],
        "children": [...]
      },
      {
        "title": "Financial Results",
        "summary": "Revenue, operating income, EPS, and segment performance for fiscal 2023",
        "pages": [45, 72],
        "children": [
          {
            "title": "Revenue by Product Category",
            "summary": "iPhone, Mac, iPad, Wearables, and Services revenue breakdown",
            "pages": [46, 52]
          },
          {
            "title": "Geographic Revenue Distribution",
            "summary": "Americas, Europe, Greater China, Japan, Rest of Asia Pacific",
            "pages": [53, 58]
          }
        ]
      },
      {
        "title": "Risk Factors",
        "summary": "Operational, market, regulatory, and competitive risks",
        "pages": [89, 110]
      }
    ]
  }
}

Imagine an analyst asking for iPhone revenue in fiscal 2023. Instead of hunting for similar text fragments, the LLM reads the index, concludes that "Financial Results > Revenue by Product Category" is the branch to follow, goes there and retrieves pages 46 to 52 in full. The answer arrives with an exact page citation, and since the model never saw the Risk Factors section, nothing noisy muddies it.

A simplified version of the reasoning-driven retrieval looks like this in code:

import json
from openai import OpenAI

client = OpenAI()
def navigate_index(query: str, index_node: dict, depth: int = 0) -> list[dict]:
    """
    Recursively navigate the document index using LLM reasoning.
    Returns list of relevant leaf nodes with page references.
    """
    children = index_node.get("children", [])
    if not children:
        # Leaf node: return this section as relevant
        return [index_node]
    # Ask the LLM which branches are relevant to the query
    children_summary = "\n".join([
        f"[{i}] {child['title']}: {child['summary']}"
        for i, child in enumerate(children)
    ])
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {
                "role": "system",
                "content": (
                    "You are navigating a document index to find sections relevant "
                    "to a query. Select the index numbers of sections that are likely "
                    "to contain the answer. Return a JSON array of selected indices."
                )
            },
            {
                "role": "user",
                "content": (
                    f"Query: {query}\n\n"
                    f"Available sections:\n{children_summary}\n\n"
                    f"Which sections should I look into? Return JSON array of indices only."
                )
            }
        ],
        temperature=0,
        response_format={"type": "json_object"}
    )
    selected = json.loads(response.choices[0].message.content).get("indices", [])
    relevant_nodes = []
    for idx in selected:
        if idx             # Recurse into selected branches
            relevant_nodes.extend(
                navigate_index(query, children[idx], depth + 1)
            )
    return relevant_nodes

def answer_with_pageindex(query: str, index: dict, document_pages: dict) -> str:
    """
    Full PageIndex retrieval and answer generation.
    """
    # Navigate the index to find relevant sections
    relevant_nodes = navigate_index(query, index)
    # Retrieve full text from identified pages
    context_parts = []
    citations = []
    for node in relevant_nodes:
        pages = node.get("pages", [])
        if pages:
            page_start, page_end = pages[0], pages[1]
            for page_num in range(page_start, page_end + 1):
                if page_num in document_pages:
                    context_parts.append(document_pages[page_num])
                    citations.append(f"p.{page_num}")
    context = "\n\n".join(context_parts)
    # Generate answer with full, unchunked context
    answer_response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {
                "role": "system",
                "content": (
                    "Answer the question based on the provided document sections. "
                    "Be precise. If the answer involves numbers or dates, quote them exactly."
                )
            },
            {
                "role": "user",
                "content": f"Document sections:\n{context}\n\nQuestion: {query}"
            }
        ],
        temperature=0
    )
    answer = answer_response.choices[0].message.content
    citation_str = ", ".join(set(citations))
    return f"{answer}\n\n**Source:** {citation_str}"

Look at what's absent: embeddings, cosine similarity, vector databases and chunking logic. Structured reasoning over a document index replaces the whole retrieval pipeline.

Evidence from a finance benchmark

FinanceBench collects questions about genuine financial documents such as SEC filings, annual reports and earnings calls. Answers must be numerically precise, sections must be cross-referenced exactly and footnotes read carefully. Vector-based RAG tends to struggle here, because financial documents are packed with similar-sounding language on different subjects.

Mafin 2.5, built on PageIndex, reached 98.7% accuracy on FinanceBench. Typical vector RAG systems on comparable financial benchmarks land around 65 to 80%, depending on configuration. That gap isn't marginal. It separates a tool an investment analyst could actually trust from one that produces plausible nonsense to be verified line by line.

Structure explains it. Financial documents are exactly where hierarchy counts most. A consolidated balance sheet, the notes to the statements and the MD&A all follow a clear, deliberate layout that analysts rely on all the time. PageIndex preserves and uses that layout, and vector chunking wrecks it.

When to choose which

PageIndex doesn't universally replace vector RAG. The right choice depends on the project.

It does well when:

• Documents have a clear hierarchy, as with reports, manuals, contracts and specifications.
• Accuracy and exact citations are non-negotiable.
• The domain is specialized, such as finance, legal, medical or regulatory work.
• Questions need cross-references between several sections of a single document.
• Explainability matters and the system has to show why it returned what it did.

Traditional vector RAG still makes sense when:

• The corpus is thousands of short, independent documents, such as support tickets or product reviews.
• Documents have no consistent structure.
• Approximate semantic search across a broad corpus is the goal.
• Latency is the main concern and chunking is already optimized.

Most enterprise document use cases, where a system must be truly useful and not merely impressive in a demo, resemble the first list much more than the second.

For production use without the infrastructure burden, PageIndex also offers a REST API and MCP integration. The developer setup looks like this:

import requests

PAGEINDEX_API_KEY = "your_api_key"
BASE_URL = "https://api.pageindex.ai/v1"
def upload_document(file_path: str) -> str:
    """Upload a document and get back a document_id."""
    with open(file_path, "rb") as f:
        response = requests.post(
            f"{BASE_URL}/documents",
            headers={"Authorization": f"Bearer {PAGEINDEX_API_KEY}"},
            files={"file": f}
        )
    return response.json()["document_id"]

def query_document(document_id: str, question: str) -> dict:
    """Query an indexed document and get a cited answer."""
    response = requests.post(
        f"{BASE_URL}/query",
        headers={
            "Authorization": f"Bearer {PAGEINDEX_API_KEY}",
            "Content-Type": "application/json"
        },
        json={
            "document_id": document_id,
            "question": question
        }
    )
    return response.json()

# Example usage
doc_id = upload_document("q3_earnings_report.pdf")
result = query_document(doc_id, "What was total revenue in Q3?")
print(result["answer"])
print(f"Sources: {result['citations']}")

On upload the API builds the index automatically, and when queried it returns structured answers with page citations that can be traced.

The bigger idea

Beyond the benchmark scores, something deeper is going on. Standard vector RAG sees a document as a bag of related text fragments and disregards the structure its author built in. Each heading, section break and cross-reference is a signal, the writer's way of showing readers where to go, and chunking throws all of it away before trying to reconstruct relevance from nothing. PageIndex handles a document as what it is: a structured artifact intended for navigation by a capable reader.

A senior analyst with an annual report doesn't read from page one onward. They check the table of contents, jump to the part they care about, verify a figure against the notes and flip back to the MD&A. The LLM inside PageIndex does the same thing.

That's also why it carries over to specialized areas. A pharmaceutical trial report has a different hierarchy from a legal contract or a technical specification, yet in each one the author followed an internal logic that expert readers take advantage of. PageIndex captures that logic, and retrieval reasons through it.

A different diagnosis

The sobering conclusion is that years of effort may have targeted the wrong problem. Attention went to chunking strategies while the actual flaw was wrecking structure at the outset. Embedding models were tuned even though semantic similarity was never the right retrieval signal. Rerankers were bolted on to rescue retrieval quality when the retrieval method itself was the bottleneck.

The RAG ecosystem assembled intricate machinery to make up for chunking's original sin. PageIndex proposes an alternative: no chunks, no embeddings, no similarity search. Build an index mirroring the document's real structure and move through it by reasoning.

It doesn't answer everything. Vector search still wins for semantic search across millions of short, unstructured documents. Yet for the broad family of questions about long, structured documents, which covers most enterprise needs, a hierarchical tree deserves serious attention. Rerun the demo that failed twice, asking about 2023 revenue projections, and the system now heads to the right section, pulls the right pages and replies with exact citations. No wrong year, no wrong section, no misplaced confidence.

PageIndex is open source and self-hostable on GitHub, and a hosted chat interface and API documentation with REST and MCP references are available for trying it on your own documents.