Home / Articles

KohakuRAG: Climbing the Document Tree to Find the Right Evidence

KohakuRAG climbs a four-level document tree to find evidence: multi-query retrieval, retries on blank answers and voting, as tested on the WattBot 2025 task.

Publisher: REACTAPP.TOP5 min read1122 words

This article is published in English.

Sources: KohakuRAG: Climb the Document Tree, Find the Right Evidence

Retrieval-augmented generation often looks great in a demo and then drifts apart on real questions. KohakuRAG, a framework described in a 2026 arXiv paper, goes after that gap with plain engineering choices, and it's worth a close look.

Where ordinary RAG falls short

Tasks like the WattBot 2025 Challenge expose the weak spots. It demands numeric answers accurate to within 0.1 percent along with exact source attribution. Three habits of standard pipelines hurt there:

  • Cutting text into flat chunks ignores sections and document structure, so tracing an answer to its source gets harder and citations earn less trust.
  • One query can easily miss good evidence when the question and the document word the same idea differently.
  • A single generation pass is unstable. Answers and citations change between runs, and a model afraid of hallucinating may return nothing even though the evidence sits right in its context.

From flat chunks to a tree

KohakuRAG is a compact framework built on a four-level index: document, section, paragraph, sentence. Structure survives because embeddings are combined from the bottom up. Query planning by an LLM, together with reranking across queries, widens retrieval coverage. Answers are steadied by voting across several generations, with blank outputs handled specially.

Offline: building the index

Parsing comes first. Each document becomes a tree of those four levels. Images and tables are treated as special paragraph nodes with generated captions. PyMuPDF pulls out the images, and a vision-language model (Qwen3-VL) writes captions that focus on technical content, numbers and axis labels.

Most of the parsing is rule-based. PyMuPDF extracts text and layout, then heuristics based on font size, spacing and formatting recover the hierarchy:

  • Sections: blocks whose font is at least 1.2 times the median size count as candidate titles, and numbering such as 1.2 or A.3 helps confirm a boundary.
  • Paragraphs: neighboring blocks separated by less than 1.5 line heights are merged, and larger gaps start a new paragraph.
  • Sentences: NLTK's sent_tokenize does the splitting, plus extra rules for technical abbreviations like Fig., et al. and i.e.

Embeddings come next. Sentences, the leaves, are embedded directly. Each parent is the average of its children, weighted by token count, so longer pieces contribute more. Two backends are supported: Jina v3 for text (768 dimensions, up to 8K context) and Jina v4 for multimodal retrieval, with Matryoshka dimensions from 128 to 2048 and direct handling of both text and images. Storage uses sqlite-vec, a light vector backend.

Online: multi-query retrieval

At question time, an LLM expands the original question into several related queries covering rephrasings, acronym expansions and sub-questions. Each query runs a cosine-similarity search restricted to sentence and paragraph nodes. An optional hybrid step adds BM25 sparse retrieval on the paragraph index only, with those results appended after the dense ones.

Results are then reranked across queries using frequency, score or a combination, so a node matched by several queries ranks higher. The system then walks the tree and expands context: for every hit, it brings in parent and sibling nodes. A matched sentence pulls in its paragraph, and a matched paragraph pulls in its section. Section-level hits themselves are dropped as too coarse. An optional deduplication step can remove ancestor and descendant overlap to save context window, though the competition runs did not use it.

Producing the answer

Prompts put the retrieved context ahead of the question, which counters the lost-in-the-middle effect, and citation markers make attribution easier for the model. Output is structured JSON containing the answer, the extracted value, citation IDs, an explanation and a flag for a blank result. When evidence is thin, the model is expected to abstain explicitly. Note that citations are judged at the document-ID level, not by page or span, which matters when interpreting what the system optimizes.

A retry loop covers blanks. If the model reports insufficient evidence, the system raises top-k, retrieves again, and repeats until it gets a valid answer or hits the retry limit.

Several generations, then a vote

Rather than request one answer, the system produces several independently. Blank answers are removed before the vote when at least one real answer exists. If every run is blank, it abstains. Voting steadies both the answers and the citations.

Results

The framework was built for the WattBot 2025 Challenge, so results come mainly from that benchmark: 32 technical documents on AI energy use, including papers, reports and slide decks, roughly 500K tokens in all. Topics cover power consumption, water use, carbon footprint and model performance characteristics.

KohakuRAG scored 0.902 on the public leaderboard and 0.861 on the private one, and it was the only method to hold first place on both.

Two findings stand out. Prompt order matters enormously: putting the question after the retrieved context, not before it, gave an 80 percent relative improvement, supporting the lost-in-the-middle idea. Sometimes the biggest gain comes from the simplest change. And dense retrieval alone may be enough. Hybrid search with BM25 is usually treated as a strong baseline, yet a well-structured hierarchical dense setup was highly competitive, and adding BM25 contributed only about 3.1 percent.

Assessment

No single component here is radically new. The strength is in how a handful of known techniques (a layered index, several queries per question, retry when the model abstains, and majority voting) are tied together around one demanding goal: accurate answers, precise citations and a willingness to abstain when the evidence is weak.

It also touches one of the hardest parts of enterprise question answering, where traceability matters even more than being right. Seen that way, the tree isn't a fancier way to chunk text. It's infrastructure for citations, audits and error analysis.

Two concerns remain. The first is sensitivity to parsing quality, and probably to the quality of query planning. Real enterprise content is full of messy PDFs and inconsistent layouts, and rule-based section detection can fail at the foundation. The second is cost and latency. Ensembles raise cost roughly linearly, and retries can worsen tail latency, so this looks like a high-reliability, low-throughput design and not a natural fit for high-concurrency online question answering.

The underlying paper is titled "KohakuRAG: A simple RAG framework with hierarchical document indexing" and is available on arXiv.