DocSeeker: Teaching a Vision Model to Cite Pages in Long Documents

Anyone who has put a question to a model about a long report and got back a confident answer they couldn't verify knows how annoying that is. Handling long documents isn't just a matter of reading more pages. The harder part is keeping the answer trustworthy from the first page to the last. DocSeeker, a recent paper (arXiv 2604.12812), goes after exactly that by making the model show which pages it relied on.

Where long-document models break

Existing approaches to long-document question answering fall into three camps, and each fails in its own way.

OCR and parsing pipelines. These pull out text and layout with OCR or PDF parsers, then hand the result to a language model. LongFormer, LayTokenLLM, HiVT5, DocLLM and DocVLM belong here. They stumble on OCR mistakes and can lose layout and visual cues along the way.

Visual RAG. A visual retriever picks the top-K most relevant pages, and only those go to a multimodal LLM. ColPali and DSE are typical retrievers, and VisRAG, SV-RAG, VDocRAG and M3DocRAG are typical full systems. The sticking point is K. Set it too low and you miss the key evidence. Set it too high and you add noise.

End-to-end multimodal models. These feed all the page images straight into the model. mPLUG-DocOwl2, Qwen2.5-VL and InternVL3 are examples. The huge number of visual tokens in a long file produces a poor signal-to-noise ratio, with the key evidence buried in a mass of irrelevant content, which hurts both speed and quality.

There's a data problem too. Most datasets supply only a short final answer. Training on those labels invites brittle shortcuts and dataset-specific tricks, and it never teaches the model to locate evidence or reason transparently.

DocSeeker's three-step approach

The core idea is simple: split the job into three explicit steps. First analyze the question, then find the evidence, then reason from it. That leaves page-level traces a person can actually check.

It starts from Qwen2.5-VL-7B-Instruct and adds no extra modules. All the effort goes into training the model itself to find its way around a long document and reason over it.

One input trick makes this possible. Before processing, each page's visual tokens get a text page ID such as "Page 1", an anchor that lets the model tie its reasoning to specific pages.

The framework is called ALR, for Analysis, Localization and Reasoning:

1. Analysis: the model restates the user's question and works out exactly what's being asked.
2. Localization: it names the pages that hold the evidence.
3. Reasoning: it draws conclusions from that evidence and outputs both the pages and the final answer.

So the model doesn't leap to an answer. It's trained to emit a structured ALR trace. A simplified sample from the distillation prompt looks like this:

<think>
\boxed{Question Analysis}
The user asks for bracketed text in paragraph 2 on page 5 that is
also bolded.
\boxed{Evidence Localization}
Page 5 contains bracketed text but no bolded text.
\boxed{Reasoning Process}
No text meets all required conditions; therefore the question cannot
be answered.
</think>
<answer>
{"evidence_pages": [5], "answer": "Not answerable"}
</answer>

Training: distill first, then reward

Stage one: supervised fine-tuning

Gemini-2.5-Flash acted as the teacher. It saw a minimal context made of the ground-truth evidence pages, a few distractor pages, the page IDs and the question, and it generated ALR-format traces from that. The student, though, was fine-tuned on the full document with every page and page ID. Giving the teacher less keeps distillation cheaper and cleaner, while the student still practices on realistic input.

The traces were filtered with exact-match checks on answers and evidence pages, plus GPT-4o for semantic validation. What survived was a set of 13,986 ALR examples: MP-DocVQA supplied 8,676 of them and DUDE the other 5,310.

Stage two: EviGRPO

Plain GRPO mostly rewards correct answers. DocSeeker uses three rewards instead: format, evidence, and answer, weighted 0.1, 0.3 and 0.6.

• The evidence reward is a weighted F1 with beta set to 2.0. That favors recall, so skipping an evidence page costs the model more than including an extra one.
• The answer reward uses ANLS instead of strict exact match, which is standard in document VQA since correct answers can differ slightly in formatting.

Cutting visual-token cost with EGRA

To keep training affordable, EGRA leaves evidence pages at high resolution and randomly shrinks 70% of the non-evidence pages from ri = 1024 to ri = 256. The other 30% stay at full resolution. This only applies in training. At inference every page is processed at high resolution, and reported evaluation uses 1024 by 784 per page.

Results

DocSeeker beats the baseline on all five benchmarks:

• DUDE: 35.2 up to 57.4, the biggest gain at 22.2 points.
• MPDocVQA: 70.1 up to 86.2.
• SlideVQA: 59.8 up to 77.1.
• LongDocURL: 37.8 up to 51.7.
• MMLongBench-doc: 25.4 up to 40.1.

It's the strongest open-source method in the reported comparison and holds up against closed models too. It passes GPT-4o on the two in-domain benchmarks, though it stays behind GPT-4o on MMLongBench-doc and LongDocURL.

What to make of it

The real contribution is turning a vague challenge into supervised sub-tasks: find the relevant pages, then reason over them. By emitting structured thoughts, page IDs and evidence pages, the model learns what you might call attention routing, and that has real practical appeal for industry use.

The direction looks right, but a few questions remain before broad production use:

• How reliable is localization beyond page-level evidence?
• What does it cost to generate ALR data for other domains?
• How do you control the extra reasoning latency?
• Can citations get finer than whole pages?

That last one matters most. Page-level evidence is a good start, but many real deployments will want quoted spans, table cells or visual regions, so that checking an answer takes seconds and leaves an audit trail.