From Naive to Production RAG on SEC 10-K Filings: Chunking, Hybrid Search

A pipeline that looked right and retrieved wrong

Imagine a RAG pipeline for SEC 10-K filings that took three weeks to build, with LangChain wiring, ChromaDB as the vector store, a clean Python ingestion script and a polished Streamlit front end. A simple first test asks which critical audit matters concern revenue recognition. The system answers, totally sure of itself, about executive compensation from another section entirely. Hallucination isn't quite the word. The text is real but comes from the wrong region of a 300-page document, and nothing in the system noticed.

Looking like a RAG system and working like one are different things. Below are the chunking decisions, the retrieval failures, the embedding experiments, the evaluation set born of necessity, and the version that finally cut manual audit review by 65%. Anyone building LLM features over real documents, not toy PDFs, can skip weeks of unproductive learning.

Why 10-Ks are hard to retrieve from

Treating a 10-K as an ordinary PDF underestimates it. A filing from a mid-sized or large public company is 150 to 400 pages long, with financial tables, footnotes citing other footnotes, legal boilerplate stretching across dozens of pages, narrative risk disclosures and management commentary. The retrieval target was Critical Audit Matters (CAMs), where auditors explain the most complex or judgment-heavy areas of their work.

What a reader wants to find, like risk language and audit judgments, is unevenly distributed. It's concentrated in certain sections, written in specialist vocabulary, and tied to figures that mean little out of context. Treat the document as a flat bag of chunks and retrieval picks the wrong ones most of the time, which is exactly what the first version did.

Two pipelines side by side

The distance between the versions holds all the lessons.

Version 1 was the textbook design: ingest the PDF, cut it into 500-token chunks with a 50-token overlap, embed with OpenAI's text-embedding-ada-002, store in ChromaDB, retrieve the top 5 chunks for a query, hand them to GPT-4 and generate a summary. It worked on simple queries against clean documents and fell apart on a dense 300-page filing in three specific ways.

Version 2 has many more stages: PDF ingestion, a section-detection layer, section-aware chunking with sizes that vary by section type, metadata tagging for every chunk, a domain-aware embedding model, ChromaDB storage with rich metadata, query rewriting, hybrid retrieval (semantic plus keyword), reranking, validation and finally generation. Each box exists for a reason, covered next.

Failure one: fixed chunks cut context apart

V1's most basic error was fixed 500-token chunks, cut wherever the count happened to land. A 10-K is naturally structured. A CAM consists of a heading, a description, the auditor's response and frequently a pointer to a financial note, and the unit can stretch to 600 or 800 tokens. Splitting at 500 often cut a CAM in half, with the description in one chunk and the response in the next, so neither was useful by itself.

The fix was a section-detection layer ahead of chunking. pdfplumber parsed the document structure to find section boundaries, headings, subheadings and table edges, and chunking then happened within sections and not across them, using sizes that respected natural breaks. Dense financial tables got small chunks of roughly 200 to 300 tokens to keep row-level detail searchable. Narrative sections like CAMs and risk disclosures got large ones, up to 800 to 900 tokens, to keep each logical unit intact. Overlap stayed at 10% of chunk size. This single change noticeably improved retrieval relevance before anything else was touched.

Failure two: embeddings that missed financial language

text-embedding-ada-002 is a strong general-purpose model, though it wasn't trained specifically on financial or legal text, and the gap shows with dense audit documents. A query about revenue recognition judgment areas pulled chunks about revenue and about judgment in a general sense. Some came from management discussion, some from footnotes and a few from unrelated risk disclosures, and the revenue recognition CAM itself wasn't reliably found.

Two alternatives were tried. OpenAI's newer text-embedding-3-large improved retrieval quality measurably. A hybrid approach combined dense vector search with sparse BM25 keyword search and merged the results, and it beat either method alone. The reasoning is simple: dense retrieval finds conceptually related content, while sparse retrieval finds exact terminology. In financial text, exact wording matters a great deal. "Critical audit matter" isn't the same thing as "significant audit issue" in legal language, and having both signals raised precision substantially. The sparse side used the Python library rank_bm25, and a simple reciprocal rank fusion function merged the two result lists before reranking.

Failure three: no validation, confident wrong answers

This was the quietest and most dangerous problem. In V1, bad chunks from retrieval still led the LLM to answer, and GPT-4 is very good at sounding sure of itself with thin context. Its answers weren't obviously wrong. They were plausible summaries of whatever chunks arrived, even when those chunks came from the wrong part of the document.

Only a small evaluation set exposed the problem: 30 question-answer pairs verified by hand against particular filings, each tagged with the exact source chunks that ought to come back. Measured against it, V1 retrieved the right source chunks about 55% of the time, and the other 45% produced fluent text built from irrelevant material.

Two changes fixed it. The first was a reranking stage: a HuggingFace cross-encoder named ms-marco-MiniLM-L-6-v2 rescored the ten retrieved chunks, and only the best three went on to the generator. Cross-encoders run slower than bi-encoders but rank far more accurately, because the query and a candidate chunk are read as one input instead of two separate ones.

The second was a validation gate before generation. It checked that the retrieved chunks carried the section identifiers expected from ingestion-time metadata. When a question about CAMs returned chunks from the compensation tables, for instance, validation failed, and the system rewrote the query and retried up to twice before labeling the response low-confidence. This separates a demo from a production system: the demo covers the happy path, while production deals with the 45% of cases where that path doesn't hold.

Metadata on every chunk

One design decision rarely appears in RAG tutorials: rich metadata attached to every chunk. Each chunk in the ChromaDB collection carried these fields:

• section_type, the part of the 10-K it came from, such as CAM, risk disclosure, financial statements, footnotes or management discussion
• section_title, the exact heading above the content
• page_range, the pages of the source document the chunk covered
• document_id, the filing, as company ticker plus fiscal year
• has_table, a boolean marking tabular data
• fiscal_year, the reporting period

The metadata did two jobs. It allowed filtered retrieval: for an audit-matter question the search could pre-filter to a section type of critical_audit_matter before vector search, which narrowed the space sharply. It also powered the UI's citation layer, since each summary referenced the pages and sections it drew from, and auditors could check the source immediately.

In financial audit tooling, citations are mandatory. An auditor unable to verify a summary's origin can't rely on it, and generating citations from metadata moved the system from curiosity to something its users could actually work with.

The final system

The complete pipeline that reduced audit review effort by 65% has three layers.

• Ingestion: pdfplumber, section detection, variable chunking, metadata extraction, embedding with text-embedding-3-large, then ChromaDB storage with full metadata.
• Query: query rewriting (an LLM reformulation for specificity), filtered retrieval with a metadata pre-filter by section type where applicable, hybrid dense plus BM25 retrieval to the top 10, cross-encoder reranking down to the top 3, a validation check, GPT-4 generation, citation attachment and UI display.
• Evaluation: the 30-item dataset, retrieval accuracy tracking and response quality scoring, using an LLM judge for relevance, faithfulness and completeness.

The Streamlit front end offered auditors a query box, the generated summary, source citations that expand on demand, and a confidence indicator taken from the validation step.

What to do differently

• Build the evaluation dataset first. It was assembled out of desperation once V1 failed, and it belongs on day one. Nothing improves without measurement, and "it seems better" isn't a metric.
• Design the metadata schema early. It took time to get right, yet every hour paid back several times over in retrieval quality and in the UI features it enabled. Metadata deserves first-class status in the architecture.
• Assume the PDF parser misses things. pdfplumber is excellent, yet merged cells, rotated headers or footnote markers inside cells in financial tables can still confuse it. Log each extraction anomaly and review that log on a schedule, since silent extraction failures are the hardest bugs to track down later.
• Question every default. Default chunk size, default embedding model and default top-k all exist because they work acceptably on average, and a document type may not be average. Test each against real data.

The result

After deployment, the system extracted and summarized Critical Audit Matters with 65% less manual review effort than the process it replaced. Acting as a first-pass reviewer, it located relevant sections and drafted structured summaries, so auditors shifted from extracting and synthesizing to verifying and judging.

That last point matters most. The goal was never to replace auditor judgment but to strip out the mechanical, time-consuming steps so that expert judgment lands where it counts. Well-designed LLM systems don't replace expertise. They redirect it.