MultiDocFusion: Hierarchy-Aware Chunking for RAG on Long PDFs
MultiDocFusion keeps section context in every chunk of a long PDF: layout detection, OCR, a header-tree LLM and DFS grouping, for 8-15% better RAG retrieval.
This article is published in English.
Sources: MultiDocFusion: From Flat Chunks to Hierarchy-Aware RAG
Most RAG setups start from the same barely noticed assumption: split the document into chunks, embed them, fetch the top matches and trust the rest to work out. That holds until a long industrial PDF arrives and makes it obvious that documents were never meant to be read as confetti.
A typical system retrieves relevant chunks and asks an LLM to answer from them. To make longer documents searchable, they have to be cut into smaller pieces, and the usual cutting methods are crude: a fixed number of words, or a split driven purely by semantic similarity. Real documents are more than plain text. They carry:
- Nested headings and subheadings, several layers deep
- Tables, figures and other elements that depend on page layout
- Passages that continue from one page onto the next
- Scanned PDFs that need OCR
Splitting by length is a bit like chopping up a textbook at random. Headings get cut off from their sections, paragraphs lose the tables that explain them and key context disappears. Retrieval then returns disjointed evidence and answer quality drops. The damage is worst with industrial documents, financial reports, legal contracts and scans.
Three signals, fused
Faithful RAG over long documents needs three kinds of signal modeled together:
- Visual layout, meaning what the page looks like
- Textual content, meaning what it says
- Structural hierarchy, meaning how it's organized
MultiDocFusion is built around exactly this combination. "Fusion" refers to blending layout, text and hierarchy, while "MultiDoc" signals support for varied document formats and corpus-level, multi-document RAG.
The pipeline in the figure runs as follows: layout extraction by a document parser, OCR to read and annotate text, a hierarchical tree built from detected section headers and general nodes by DSHP-LLM, and then a DFS-based grouping step that forms coherent hierarchical chunks. Color coding distinguishes the root and title, section headers, and general nodes such as tables, figures and text blocks.
A fixed-length splitter cuts a book at token boundaries. MultiDocFusion reads more like a thoughtful person. It rebuilds the document's structural tree on the fly, a bit like generating a table of contents, and groups together material that belongs together. That makes it a good fit for long, complex documents.
The four steps
1. Layout detection
Every page is first looked at as an image. The system finds where titles, subtitles, body text, tables and pictures sit, which is layout recognition. DETR, DiT and VGT are used, all fine-tuned on the DocLayNet dataset. The output isn't merely text but boxes with positions, each recording the page, its coordinates and a type such as title or table.
2. OCR
After regions are found, OCR reads the text in every box. EasyOCR, Tesseract and TrOCR were compared under various settings, with the caveat that results still depend on scan quality, fonts and language. The system now knows both that a box exists and what it contains, for instance a heading such as "2. Method" in one box and table data in another.
3. Building the hierarchy
This step is the central contribution. A specialized LLM is trained to grasp hierarchical relationships inside the document. A heading like "1 Introduction" may sit at the top level, "1.1 Background" under it at the second level, and specific paragraphs or tables get matched with the right headings.
Rather than only extracting text, the system rebuilds a logical table of contents, uncovering the hidden outline of the document.
After comparing a handful of open-weight models, among them Llama-3.2 and Qwen-2.5, the team picked Mistral-8B as the strongest backbone. It was fine-tuned using LoRA together with 4-bit quantization (the QLoRA technique). Training and evaluation data for DSHP-LLM came from DocHieNet, 1,673 documents, and HRDH, around 1,500 academic papers. The model's only input is the structured HeaderList, from which it builds the skeleton of the tree. Once the LLM has rebuilt that header tree, the system walks through every segment, text blocks and tables alike, and hangs each one under the right header according to its spatial coordinates.
Why a specialized LLM and not plain rules? Real documents are complicated. Heuristics like "bigger font means title" or "section 1.2 must sit under section 1" often fail on scans, older PDFs and tricky layouts. Many existing methods analyze only the text or only the visual layout, which makes accurate hierarchy recovery hard, especially for scanned or irregular documents.
The LLM infers parent-child relations among section headers. Other content blocks get attached afterward by spatial ordering and aren't predicted jointly by the LLM within the same hierarchy task.
4. Hierarchical chunking
Armed with the tree, the system groups content by logic and doesn't cut it arbitrarily. A section's titles, text and tables remain together and are divided only when length demands it. Every chunk carries its lineage, for example "Document Title → Chapter 1 → Section 1.1 → specific paragraph". Retrieval consequently returns segments with context attached, not stray snippets, and holding the hierarchy inside each chunk cuts down on lost context.
As an implementation detail, the finished chunks write their hierarchy out as Markdown headers (#, ## and so on), putting structure into plain text that retrieval and generation can use.
What the numbers show
Across most datasets, MultiDocFusion achieves the best retrieval performance, with especially clear gains on more complex sets such as DUDE, MPVQA and MOAMOB. Retrieval precision rose by roughly 8% to 15%. Earlier systems often pulled back incomplete snippets scattered across pages. Hierarchy-aware chunking is likelier to return structurally coherent pieces that keep the section context, even when a whole section still has to be divided for length reasons.
On question answering, the results also improve overall, with ANLS up by about 2% to 3%. That fits the idea that better-structured context supports better answers, though the study doesn't single out retrieval quality as the only cause of the QA gains.
The real insight is turning long documents into a retrieval-friendly "tree-path representation". DSHP-LLM finds relationships among headers, paragraphs and tables are attached by spatial order, and DFS then pulls ancestor headers into every chunk. In effect it performs a structured rewrite at indexing time.
Two concerns
- A questionable assumption. The method treats every node between one header and the next as belonging under the current header. That isn't always true, notably for two-column designs, sidebars, floating tables or images, footnotes spanning pages and cross-referenced appendices. Two items sitting next to each other on the page aren't necessarily related in meaning.
- Index bloat. Repeatedly copying ancestor headers into child chunks produces a large number of chunks, 20,773 in total at an average of 521.65 tokens. Retrieval might then lean on section titles more than on the evidence in the text, especially for enterprise template documents whose files share similar headers. The study itself admits this trade-off under its limitations, noting that duplicating parent context raises index size, storage cost and retrieval latency. Budget-aware chunking or node-level deduplication would be needed at large scale.
A parallel with MoDora
MoDora, a related approach to turning broken OCR chunks into a living document tree, shares the core idea: reconstruct the internal structure first, and only afterward hand things to retrieval and QA.
- MultiDocFusion moves from parsing and OCR, to a hierarchical tree through DSHP-LLM, to hierarchical chunks through DFS. It's structure-aware modeling and chunk creation.
- MoDora gathers OCR elements into layout-aware components first, then builds a CCTree. It models component hierarchies, layout differences and relations between components before retrieval and analysis.
Neither treats a document as a flat run of text. Each represents the hierarchy explicitly as a middle step before retrieval.